Commit Graph
70 Commits
Author SHA1 Message Date
splitsec2 e575ff0fcb feat(activity): clearer cards, a Rejected view and Reopen for failed deliveries (#1618)
A batch of activity list changes from running it as an admin on my own
instance. They are separate commits that build on each other, so they
can be reviewed in order.

The cards read better. Every card gets a format pill (violet for audio,
orange for ebooks), the requester is named once, and finished cards say
how long a download took and how long ago it ended ("took 5m · 35m
ago"). A download that picked up work already under way shows a
Reattached pill.

Admins can tell what is automatic and what needs a person. A declined
request shows who declined it and when, anything with an `[auto]` note
reads "Declined automatically", and "Needs Review" becomes "No Match
Yet" with a line saying the search repeats on every sync.

There is a Rejected view next to History for admins, with Reopen on each
card and a way to hide the dead ones.

The part to read most carefully is the backend in `requests_service.py`
and `user_db.py`. An admin can now reopen a fulfilled request whose
download failed or was cancelled, and the reopen a failed download
triggers remembers which release failed, so the next search doesn't pick
it again.

I ran it on a scratch instance (when pulling them together to move from
my fork to upstream) as an admin with seeded requests and looked at the
pills, the durations, the wording and the Rejected view in a browser.
Reopening a fulfilled request I only exercised in the tests. The
frontend and Python suites, type check, ruff and basedpyright pass.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Admins can review rejected requests in a dedicated view, hide or
restore cards, and reopen requests.
* Fulfilled requests with failed or cancelled deliveries can be
reopened; the failed release is skipped in the next automatic search.
* Activity cards now show format badges, download timing, and when a
download resumes existing work.
* **Improvements**
* Activity views clarify automatic retries and actions, and show
reviewer names for admins.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-09 12:02:21 -06:00
Rupraj Datta ed0777a2e0 Spool large debrid downloads to disk (#1632)
Fixes #1630.

Debrid downloads currently buffer entire files in RAM. The four clients
now opt into a spooled temporary file: downloads up to 8 MiB stay in
memory, and larger downloads roll to disk. The clients copy from the
file in chunks and close it after use. Other download_url callers retain
the existing BytesIO behavior.

Temporary files are closed on unsuccessful attempts and cancellation;
successful resumed downloads remain open for their caller. Added offline
coverage for disk rollover, timeout cleanup, and resumed content.
Updated the Debrid-Link test doubles for the new opt-in argument.

Validation: 113 tests passed across test_http_download_url,
test_debridlink_client, and test_torbox_client; Ruff check and format
check passed for changed files; git diff --check passed. No live debrid
services were exercised. AI assistance was used in implementing this
change.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Improvements**
* Large downloads are handled more efficiently, reducing memory use
while preserving download and resume behavior.
* **Bug Fixes**
* Temporary download data is cleaned up when a download is cancelled or
fails, helping prevent unnecessary resource usage.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-01 22:54:46 -05:00
CaliBrain 947ce7bfe5 test(e2e): serve the mock Prowlarr .torrent from Prowlarr's own origin (#1602)
The nightly client-deluge profile has failed since 61c7ad3 with "Failed
to
add to deluge: Failed to fetch torrent file: torrent fetch refused:
address
is not public". Shelfmark now fetches a .torrent only from the
configured
Prowlarr/Newznab/Jackett origin or from a public address, and the mock
Prowlarr's release linked straight to http://mock-aa/payload.torrent, a
private address on another origin. qBittorrent and Transmission hid this
by
handing the URL to the client; Deluge needs the file and failed.

Real Prowlarr rewrites indexer links to its own /{id}/download endpoint.
The
mock now does the same: the torznab link uses the host Shelfmark reached
it
on, and /{id}/download fetches the .torrent from mock-aa. Every client
now
gets the file through Shelfmark's own fetch.

tests/download/test_prowlarr_mock_torrent_fetch.py runs the real
extract_torrent_info against the mock in normal CI, so the link cannot
drift
off the Prowlarr origin again unnoticed; it fails on the old mock with
the
same error.
2026-10-27 11:25:49 -04:00
splitsec2 e8b28d05e0 fix(sources): take the retry download URL from the release cache only (#1597)
Newznab and Jackett fell back to the download URL sent with the queue
request when the release was no longer in the cache. Prowlarr already
returns no URL in that case, so this makes the other two match it: the
retry URL comes from the cache and nothing else.

A request for a release that has expired from the cache now fails with
"Release not found in cache" instead of being queued with whatever URL
came along.

The new tests cover both handlers on a cache miss and with no source_id,
and they fail on main. Full suite, ruff and basedpyright pass.
2026-10-26 11:15:44 -06:00
splitsec2 ac14e4dd24 fix(downloads): a swarm note change is not activity for the stall timer (#1572)
The torrent stall timer can be deferred for as long as the swarm keeps
changing. Since #1494 the torrent handler appends a note to its progress
message (`0% · 2 seeds`, and `· 0 seeds, giving up in 7m` while the
no-seeders clock runs). `update_download_status` treats any changed
message as activity, so a seed count that goes 2, 3, 1 between polls, or
the give-up countdown ticking down every minute, refreshes
`_last_activity` and the torrent never stalls. A torrent whose seed
count stays constant still stalls, which is why this is easy to miss.

`shelfmark/download/activity.py` already says what counts: a changed
status event or a changed progress value, and a message that changes on
a timer proves nothing. So `update_download_status` now compares
messages through a small `activity_key()` that drops that trailing seed
note. The card still gets every message and shows the live seed count,
only the stall clock ignores a change in the note. A new progress
percentage, a different client message or a status change still counts
as activity.

The handler builds the note in `_seed_note` and `activity_key` has to
match it, so one test builds the note for a range of seed counts and
countdowns and checks the key strips all of them. That also covers the
gap #1494 left, where no test connected a handler's message to the
orchestrator: the other two drive `update_download_status` with real
notes and a fake clock. Without the change the note test fails and the
key tests cannot even import `activity_key`. The other orchestrator test
is a control that a progress or status change still counts. Full suite
passes, and each new test also passes when run alone.
2026-10-26 00:45:28 -06:00
splitsec2 9011319ade refactor: one lookup for the supported formats, public names for two shared helpers (#1576)
Six places read the Supported Book Formats and Supported Audiobook
Formats settings on their own: post-processing, the post-process scan,
IRC, slskd, Prowlarr and direct download. For ebooks they already
agreed, since they all fall back to `DEFAULT_EBOOK_FORMATS`. For
audiobooks they did not. The settings registry and slskd fall back to
every audio format plus zip and rar, while post-processing and IRC fell
back to m4b and mp3 only.

They now share `shelfmark/core/formats.py`, with
`get_supported_formats`, `get_supported_audiobook_formats` and a
content-type aware `get_supported_formats_for`, and a shared
`DEFAULT_AUDIOBOOK_FORMATS` in `core/utils.py` next to the ebook one.
`policy.py` re-exports the same functions, so nothing that imports them
from there changes. It is not in `policy.py` itself because importing
the `postprocess` package pulls in the router, which imports the release
sources, so a source could not import from it without a cycle.

One behaviour difference, and only in the fallback: with the audiobook
setting unset, or emptied, post-processing and IRC used to accept m4b
and mp3 and now accept the full default list like slskd and the settings
page already did. A configured list is read exactly as before. Direct
download's own helper used the raw configured list and now reads the
same normalized one, which is identical for any list the settings page
can save.

Also public now: `parse_size` in the Prowlarr source (slskd and Newznab
imported it with the underscore) and
`coerce_completed_path_timeout_seconds` in the client base handler
(slskd imported that too).

Not changed on purpose: slskd's `_coerce_int` looks like a copy of
`coerce_int` but is not. It reads `"12.0"` as 12 and `True` as 0, where
`coerce_int` gives 0 and 1. And the `_parse_size` in the IRC source is a
different function, string to bytes.

Tests: a new set checks that post-processing, IRC and slskd all fall
back to the shared defaults and that the settings default is the shared
one, and the scan-archive and IRC tests now patch the shared config
instead of the per-module copies. Full suite passes.
2026-10-24 16:31:07 -06:00
splitsec2 4f97fc9bec fix: keep server paths and URLs out of an error response and a debug log (#1558)
Two small leaks from going through the open CodeQL alerts (code scanning
alerts 100 and 123).

The local download route returned `str(e)` on an OSError, and for a
missing or unreadable file that text is the file's path on the server
(`[Errno 2] No such file or directory:
'/audiobooks/Author/Title.epub'`). #1422 cuts a non-admin's download
path down to the file name everywhere else, so this put it back. The
route now answers "Could not read the file" and the detail stays in the
log.

`should_bypass_proxy` logged the whole request URL at debug when it
could not be parsed. URLs can carry credentials, so it logs only the
exception now.

Each change has a test that fails without it. The full non-e2e suite,
ruff and basedpyright pass.
2026-10-23 01:52:39 -06:00
CaliBrain 1aa9846f22 fix(download): trust a pinned AA mirror and repair the full e2e profile (#1518)
The destination guard from the oceanofpdf/SSRF fix (f837506) trusts only
hosts in
the mirror lists and refuses any other host that resolves to a private
address.

The full e2e profile has failed on every PR since then: its AA mirror
was mock-aa,
but the slow-download links pointed at cf.mock.test (172.30.0.11). The
guard refused
the slow-download page before the internal Chrome bypasser ever started,
so the
download ended with "All download sources failed". Adding cf.mock.test
as a second
AA mirror would not help: AAMirrorSelector.rewrite would then move the
gated link
onto mock-aa, and the test would pass without Chrome solving anything.
The profile
now uses the gate as its only AA mirror. The cloudflare mock takes
CF_GATED_PATHS,
and the full profile challenges only /slow_download/, so search, detail
and the
final file pass through while the download still needs Chrome. The
unused
SLOW_DOWNLOAD_BASE indirection is gone.

The same allowlist missed a supported setup: Primary Mirror
(AA_BASE_URL) may pin a
URL that is not in the mirror list, and network uses it as the active AA
base. With
it on a LAN address, every AA request was refused. The guard now trusts
the pinned
URL too (mirrors.get_pinned_aa_base_url), and
has_aa_mirror_configuration reuses
the same helper.

download_url also ran the guard a second time outside its try block,
which repeated
the DNS lookup and could raise instead of returning None. The duplicate
call is gone.
2026-10-21 11:19:48 -04:00
splitsec2andCaliBrain f837506017 Merge commit from fork
A signed-in user picks the release fields of POST /api/releases/download, and a record
id with the "oceanofpdf:" prefix reached the provider without handles() or is_enabled().
The provider then fetched source_url, POSTed to the form action found on that page and
fetched the meta-refresh target, with no check on any of them.

The provider now refuses to run when it is switched off, and the book page, the form
action and every POST redirect hop must be on the configured mirror (same scheme, host
and port). The signed file link may sit on another host but must be http(s).

network.ensure_public_destination refuses a host that resolves to a loopback, private,
link-local or otherwise non-global address (IPv4-mapped IPv6 judged by its IPv4 address).
Hosts the operator configured as mirrors and names that do not resolve pass as before.
html_get_page and download_url call it on their first URL and check each redirect hop
through a response hook. The image cache uses the shared address check too.

Co-authored-by: CaliBrain <calibrain@l4n.xyz>
2026-10-21 02:06:31 -04:00
CaliBrain a00f3d28bf fix(slskd): refuse unsafe peer paths, keep queued downloads alive, report search outages (#1512)
Fixes four defects in the Soulseek (slskd)

A peer names its own files, and Shelfmark works out where slskd stored
one from the last folder and the file name. `remote_directory_name`
returned the whole directory when its last segment was empty. So
`../../../etc//passwd.epub` or `/abs/path//book.epub` resolved outside
the downloads folder, and Shelfmark imported whatever was there. For a
multi-file release it moved those files into staging (`STAGE_MOVE`).

- New `safe_remote_segments` refuses a name with an empty, `.` or `..`
segment (which covers absolute and UNC paths), a NUL, or a drive
anywhere but a Windows share root (`C:\Books\…`, which never reaches a
local path).
- Search (`iter_peer_files`, so `build_releases`) drops such files.
- `download` refuses them before anything is queued, in both modes. The
spec arrives with the queued release payload, so search filtering alone
is not enough.
- `local_path_for` also refuses a path that resolves outside the root
(`resolve()` + `is_relative_to`), for example through a symlinked
folder.

`_poll`, used when isolation is off and as the fallback for slskd <
0.26, never asked for an activity grace. A file "Queued, Remotely" sent
the same status and progress on every poll, so the orchestrator
cancelled it after 300s and `SLSKD_QUEUE_TIMEOUT_MINUTES` never applied.
It now requests, renews and releases the grace the same way isolated
mode does.

`search` caught every exception per query, so slskd being down or
rejecting the API key showed as "no releases". When no query gets an
answer it now raises `SourceUnavailableError`, as Prowlarr does, and the
error reaches the user. One failed query among answered ones is still
not an outage.

A retry reuses the task's folder, and slskd renames a new file when one
is already there. Files from a failed attempt therefore meant
`_folder_complete`'s exact count could never match again. Now:
- a failed or cancelled attempt removes its folder;
- a new attempt removes any finished files an earlier one left (for
example after a restart) before it queues the batch.

`tests/slskd/conftest.py` stubbed `flask_socketio` whenever it had not
been imported yet, even though it is installed. Running `pytest
tests/slskd tests/` therefore broke the Flask app tests in the same
worker (285 errors, also on 10710ab). It now stubs only when the module
is missing.

- 25 new regression tests. All of them fail on 10710ab.
- `make python-checks` passes.
- `uv run pytest tests/slskd tests/ -m "not integration and not e2e"`
passes: 4047 tests on e83bab9.
2026-10-21 01:58:13 -04:00
CaliBrain 49aeec97cd fix(download): stop deleting user data and losing completed downloads (#1511)
Fixes six defects. Each one breaks downloads or deletes user data, and
each has a regression test that fails on 10710ab and on 77b80eb (current
main).

- **Completed-path wait (#1464)**: `_wait_for_completed_path` only
returned a directory after two identical signatures of supported
formats.
- A directory holding only archives (a scene `book.rar` + `.nfo` +
`.sfv`) never got a signature and failed with "Completed download has no
stable supported files yet". Archives (`.zip`/`.rar`, via `is_archive`)
now count toward the signature, so post-processing extracts them again,
as in v1.4.0.
- With `DOWNLOAD_CLIENT_COMPLETED_PATH_TIMEOUT=0`, `max_attempts` is 1,
so every directory failed. One sample with book files or archives is now
accepted when no retry is possible.
- **Remove & Delete Files (#1401)**: `post_process_cleanup` called
`remove(id, delete_files=True)` on torrents matched by `find_existing`
that Shelfmark never added, deleting the user's seeding data.
`_cleanup_refs` now records `added_by_shelfmark`. A pre-existing torrent
(complete, or in progress and joined) is removed but keeps its files.
- **Cancelling an unstarted torrent (#1421)**:
`_remove_unstarted_torrent` deleted with files whenever `progress == 0`.
That included `DownloadStatus.error()`, which always reports 0, and
checking states that are re-verifying data already on disk. The torrent
must now be QUEUED, PAUSED or DOWNLOADING. rTorrent's
`remove(delete_files=True)` always returns False, so a new
`DownloadClient.can_delete_files` flag (False on rTorrent) makes it
remove without files instead.
- **Queue grace overwritten**: when a QUEUED torrent's progress rose on
the same poll, the 900 s queue grace was immediately replaced by a
shorter stall window, because the orchestrator keeps one deadline. The
stall window is now requested only when it is longer than what is left
of the queue grace.
- **`drop_ignored_files` (#1480)**: the suffix rule `w.endswith("/" +
rel)` dropped a kept root `Book.epub` when `Extras/Book.epub` was
skipped. Each skipped entry now drops at most one file: the exact path,
else the file sharing the most trailing components, else a unique
basename, else an AudiobookBay glued `"<folder> <file>"` name (the
fallback `match_plan_to_files` uses, matched on folder plus file so
chaptered `Book 2 01.mp3` picks the right folder). Ties drop nothing.
- **Email and BookLore outputs** ignored `task.ignored_files` and sent
every prepared file. Both now apply `drop_ignored_files` once after
`prepare_output_files`, and report "No book files found" if the user
skipped everything. It is applied in the outputs rather than in
`prepare` because the folder output already filters in
`transfer_book_files`, and filtering twice could match a kept file on
the second pass.

- `make python-checks`: ruff, ruff format, basedpyright (backend +
tests), vulture all pass.
- `uv run pytest tests/ -m "not integration and not e2e"`: 4054 passed
on the rebased branch.
- New tests run against unfixed 77b80eb and 10710ab: all 18 new
regression cases fail, and the 11 existing post-process-cleanup tests
also fail there because they now build the 4-tuple cleanup ref.
2026-10-21 01:56:19 -04:00
splitsec2 6f65d6e20a Merge commit from fork
A release format such as /../../x was appended to the staging file name
unchanged, so the direct-download and libgen handlers could write to, and on
cancel unlink, a path outside TMP_DIR. Normalise the format to a plain
extension at queue time and in build_filename, reject path-like source ids for
the sources that use them as file names, and make both handlers resolve the
staging path and refuse anything that is not a direct child of TMP_DIR.
2026-10-20 23:40:16 -04:00
splitsec2andCaliBrain 82c8c91c8b refactor(clients): read the stall window with a type check instead of a pyright ignore (#1498)
The only `pyright: ignore[reportArgumentType]` in `shelfmark/` is one I
added in #1426, and I was being lazy when I did. I hit the same error on
another PR and replaced the ignore there with a real check, so this does
the same for the original.

`_stall_window_seconds` reads the setting as an `object`. It now narrows
to int, float or str before calling `float()`, and a bad string falls
back to the default as before. A non-numeric value of another type
(None, a list) falls back to the default too, which used to happen
through a `TypeError`.

No behaviour change. The existing range test still passes and I added
`None` and a list to it. basedpyright is at 0 errors, ruff is clean and
the backend suite passes.

Co-authored-by: CaliBrain <calibrain@l4n.xyz>
2026-10-20 23:28:42 -04:00
splitsec2 e83bab9337 feat(clients): give up on a torrent with no seeders (#1494)
A torrent with no seeds won't finish. It runs on the pieces its few
peers hold and then sits still, and nothing says so until the stall
timer ends it. The card only reads "downloading". I run AudiobookBay
releases, which show no seed count in the listing, so the client is the
only place to find out.

Two new settings, both off until you turn them on:
`TORRENT_NO_SEEDERS_WAIT` (minutes, 0 turns it off) and
`TORRENT_MIN_SEEDERS` (default 1).

- A torrent that has fewer seeds than the minimum for the whole wait,
while nothing is downloading, is removed from the client with its data
and the task ends as an error.
- A torrent with no seed at all for the wait is given up on even while
it still moves.
- A torrent that met the minimum and lost seeds down to one, but is
still moving, keeps going.
- The card shows the seed count beside the progress, with a countdown
while it waits ("0 seeds, giving up in 4m").

qBittorrent only, because it reports connected and tracker seed counts.
Other clients report none and are left to the stall timer, and an
unknown count is not read as zero. A torrent that was already in the
client when the download started is never removed, and a queued torrent
isn't judged.

Tests cover the wait and countdown, the clock restarting on progress,
the tracker count, a higher minimum, off by default, a client with no
counts, a preexisting torrent, a queued torrent, and the moving cases.
Ruff and the backend suite pass.
2026-10-20 23:26:13 -04:00
splitsec2 06a67a7150 feat(download): a concurrency limit per source (#1495)
With one global limit, a few tasks that are only waiting hold every
slot. A Soulseek download sitting in a peer's queue is the example I
hit: three of those and a torrent that could start waits behind them.

`MAX_CONCURRENT_DOWNLOADS_PER_SOURCE` (0 keeps the single global limit,
which is the default). With it set, the coordinator starts a download
only when its source is below the limit and passes over tasks of a full
source without losing their place in the queue. `get_next` gets an
optional filter for this, and a cancelled task is still dropped when one
is given. The start-up stagger now applies only between downloads from
the same source, and the pool is sized so every source can reach its
limit.

The second commit: qBittorrent keeps its own queue of torrents waiting
for a free slot. The task polling one of those does no work, so it no
longer counts against its source's limit and the next task of that
source can start. The client does the pacing, as it does for its own
queue.

Tests cover the queue pass-over (order and priority kept, nothing
eligible, cancelled tasks), the coordinator (a full source doesn't block
the others, each source has its own limit, the global limit when unset)
and the client-queue case. Ruff and the backend suite pass.
2026-10-20 23:19:33 -04:00
CaliBrain 10710abd4e fix(download): send every file of a multi-file download to the browser (#1489)
Download to Browser only ever sent the first file of a multi-file
release
(audiobook chapters, packs): the folder output returned final_paths[0]
and
/api/localdownload served that one path.

The folder output now keeps every final path on the in-memory task
(download_paths), and /api/localdownload streams them as one
uncompressed
ZIP named after the book when there is more than one. The archive is
never
staged on disk or held in memory, and ZIP64 covers large audiobooks.
History still stores only the first path, so a re-download from history
after a dismiss or restart serves a single file, as before.

Also brings every linter back to green on all files:
- pre-commit: bump the ruff hook to v0.17.1 to match uv.lock;
ruff-format
  the Python examples in docs/dev and skills
- ruff: fix the remaining issues in skills/shelfmark/download_books.py;
  format base_handler.py
- basedpyright: Optional and attrs errors in databazeknih.py
- knip: drop 31 unused exports, reuse ColumnColorHint in colorMaps
- oxfmt 0.72: styles.css
- direct download: fix the "Welib, mirrors" typo and the test that still
  expected the pre-OceanofPDF message

Fixes #1467
2026-10-20 02:33:53 -04:00
amura11andClaude Sonnet 5.5 51f32b374d feat(packs): let users choose which files, books and formats to download (#1480)
## Summary
- **Skip files or books in a multi-book pack.** The pack review panel
gets a switch per file and a "Skip this book" switch per book, using the
shared `ToggleSwitch`. A partly skipped book shows e.g. "3 of 5 files"
in the Files column. A fully skipped book is greyed out.
- **Choose formats for a single book.** A new "Choose formats" header
toggle, next to the multi-book one, opens a format picker for a release
that has several book formats (EPUB, MOBI, PDF and so on). It has one
switch per format with file count and size. Releases with one format, or
that can't be inspected, download as before.
- **Skipped files are sent as `ignored_files`.** They are stored on the
task (including restart-safe retry) and removed before grouping and
transfer. This matters for packs because files missing from `book_plan`
would otherwise be filed by the heuristic fallback. Ignored entries
match by release-relative path, then path suffix, then unique basename.
- A pack that ends up with one book after skips keeps the titles you
reviewed instead of falling back to the searched title.

## Screenshots
<img width="1277" height="413" alt="Screenshot 2026-10-19 at 09-39-28
Shelfmark"
src="https://github.com/user-attachments/assets/c005ca4b-3dac-45dd-b2ff-d02fc0f9fcc9"
/>
<img width="1277" height="1210" alt="Screenshot 2026-10-19 at 09-40-23
Shelfmark"
src="https://github.com/user-attachments/assets/2145f3ba-e5fc-4635-9364-1790feb0c8b8"
/>
<img width="1277" height="748" alt="Screenshot 2026-10-19 at 09-39-03
Shelfmark"
src="https://github.com/user-attachments/assets/39023fac-70e3-46d8-af89-1160ebd6e415"
/>


## Test plan
- [x] `pytest` pack tests (new `drop_ignored_files` cases added)
- [x] `vitest`, `tsc`, `oxlint`, `oxfmt`, `ruff`
- [x] Manual check in the UI with a real multi-book pack

🤖 (mostly) Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-20 02:03:05 -04:00
splitsec2 0ade063db5 fix(download): extract RAR archives in place, not in the IO threadpool (#1459)
Fixes #1458.

A RAR download fails with `Download failed: TypeError`, and the log says
`child watchers are only available on the default loop`. rarfile
extracts by running `unrar` as a subprocess, and `extract_archive_files`
calls it through `run_blocking_io`, so it runs in gevent's IO
threadpool. gevent only lets a subprocess start on the thread that owns
the default loop. `run_blocking_io` already avoids the threadpool for
`subprocess.run`, but a RAR gets to the subprocess through rarfile, so
nothing catches it.

This extracts a RAR on the calling greenlet, where the patched
subprocess waits for `unrar` without blocking anything else. ZIP is pure
Python and stays in the threadpool. I checked it with a real RAR4 file
on a build of main with the stock image's `unrar`: from the threadpool
it raises the TypeError, in place it extracts all 7 files.

Two tests. One checks that a RAR doesn't go through `run_blocking_io`.
The other runs under real gevent patching with an extractor that starts
a process the way rarfile does. Both fail without the change, and the
full suite passes.

I keyed it on the `.rar` suffix to keep it small. Teaching
`run_blocking_io` about rarfile's subprocess would be the more general
fix, but that's a bigger change to a helper a lot of code goes through,
so I left it for you to say if you want it.
2026-10-20 01:12:10 -04:00
splitsec2 47707b827a feat(download): make the torrent stall window a setting (#1426)
#1420 gave a torrent 15 minutes to get going and 15 minutes after each
step forward. That suits my swarms, but other people will want it
shorter or longer, so this makes the number a setting.

`TORRENT_STALL_TIMEOUT_MINUTES` sits in Download Clients with the other
torrent options and takes 5 to 60. The default is 15, so nothing changes
unless someone sets it. 5 is the orchestrator's own stall timeout, so
choosing it asks for no extra time and gives the old behaviour. The
value can arrive as text from an env var, so anything that isn't a
number falls back to the default and anything out of range is clamped.

The orchestrator caps one activity grace at 16 minutes, so I raised that
to an hour to fit the top of the range. As far as I can tell the
protection bypass is the only caller that asks for more than a few
minutes, but I haven't audited every caller. If you'd rather keep the
cap, I can lower the maximum to 16 minutes instead.

Tests cover the default, 5 and 30, text values, out of range values and
NaN, the start and movement windows following the setting, and the
setting showing up in the Download Clients tab. I broke each rule on
purpose to check a test catches it. The full suite fails the same tests
with and without this change (the bypass and seleniumbase tests don't
run in my environment).
2026-10-20 00:58:24 -04:00
splitsec2 d99e9d7c4f fix(download): remove a cancelled torrent that never started (#1421)
Cancelling a torrent download leaves the torrent in the client.
`_safe_remove_download` documents the rule: "torrents: never remove or
delete client data (avoid breaking seeding)". That makes sense for a
torrent that has data. A torrent that is still at 0% has nothing to seed
and nothing to resume, and leaving it behind holds a download slot in
qBittorrent's queue or sits there as a dead entry.

I hit this on a qBittorrent that is shared with Sonarr, Radarr and
Lidarr and has a download limit. Shelfmark cancelled downloads that were
queued (#1420) and left them in the client. Three torrents that never
fetched metadata held every active slot, and each torrent added after
them waited behind them. Removing those by hand freed the slots.

This removes the torrent and its files when a download is cancelled,
Shelfmark added the torrent, and the client reports 0% progress. A
torrent with any data is left alone as before. So is one that was
already in the client when the download started, since Shelfmark didn't
add it. If the removal fails it is logged and the cancel carries on.

Tests cover removal at 0% for a queued and a downloading torrent,
leaving one that has data, leaving one the user already had, and a
failing removal. The full suite passes, and I broke each rule on purpose
to check a test catches it.

This changes a rule you wrote down, so I've kept it apart from the queue
fix. If you'd rather have it as a setting, or only remove a torrent that
is still queued or fetching metadata, I'm happy to rework it.
2026-10-02 22:12:51 -04:00
splitsec2 88f73e3bc3 fix(download): do not cancel a queued torrent as stalled (#1420)
qBittorrent only runs a few downloads at once (`max_active_downloads`
defaults to 3) and holds the rest as `queuedDL`. Shelfmark's stall timer
cancels a download after five minutes without a changed status or
progress. A queued torrent reports "Queued" on every poll, so it looks
idle, and Shelfmark cancels a download that was only waiting for a slot.

I run Shelfmark against the same qBittorrent that Sonarr, Radarr and
Lidarr use, which I assume is a common setup. Their torrents and
Shelfmark's share one limit, so anything that fills the slots makes
Shelfmark's torrent wait. In my case three torrents that never fetched
metadata held every slot. I'd expect a busy Sonarr queue to hold them
the same way, but I haven't reproduced that. Once the stall timer had
cancelled the waiting downloads they stayed in the client, still queued
behind the same blockers. On my instance 10 AudioBookBay downloads were
cancelled while qBittorrent still had them queued.

This changes the torrent poll loop. When the client reports a queued
torrent, it asks for an activity grace (the mechanism `html_get_page`
already uses for protection bypasses) and releases it as soon as the
torrent leaves the queue. The grace is renewed every ten minutes and
stops at two hours, so a queue that never moves still ends. A torrent
that is downloading and moving is timed as before until the new window
below applies.

The second commit sets the stall message before the cancel runs. The
terminal hook copies the message into the history row, and it was set
afterwards, so a download the timer ended was recorded with the last
poll's "Queued". That left no way to tell it apart from a person
cancelling.

The last two commits are about slow torrents. The timer only counts a
change in progress, so a torrent that needs more than five minutes to
fetch metadata or find its first peer is cancelled before it has any
progress to count, and one that moves a few megabytes at a time can be
cancelled between bursts. I hit this with a torrent that had a single
peer. A torrent now gets one 15 minute window the first time it is seen
outside the queue, and each step forward in progress renews it. One that
makes no progress for 15 minutes is still cancelled, so a dead torrent
now takes about 15 minutes to end instead of five. That trade is the
part I'm least sure about, so the number is easy to change.

Tests cover the grace being requested once, released when the torrent
starts, renewed, and capped, the order of the message and the cancel,
and the start and movement windows. The full suite passes, and I broke
each rule on purpose to check a test catches it.

The two hour ceiling is a guess. If you'd rather have a setting for it,
or a different number, I'm happy to change it.
2026-10-02 22:12:25 -04:00
vansh 41a4df01c4 fix: hand DiamWall 513 challenges to the bypasser (#1400)
## Summary
Recognize DiamWall interstitials in both the HTTP retry path and the
internal browser helper. HTTP 513 challenge responses go directly to the
configured bypasser, while ordinary 513 responses keep the existing
error behavior.

Fixes #1386

## Testing
- `uv run --frozen --extra browser pytest -q -n 0
tests/download/test_http_challenge_513.py
tests/bypass/test_internal_bypasser.py`: 27 passed
- Ruff lint and format checks passed
2026-10-02 16:26:04 -04:00
splitsec2 b690832659 fix(download): stream a completed book instead of buffering it in RAM - lowering memory needs significantly (#1378)
Credit where it is due: this defect was found and measured by **@DrNgo**
in
[DrNgo/shelfmark-fork@ae2185c](https://github.com/DrNgo/shelfmark-fork/commit/ae2185c8a83d537e55f1dc03f745ab844b5cdcdb).
He recorded a 493 MB audiobook peaking near 963 MB and OOM-killing a 1
GiB container, with the proxy access log showing a single `GET
/api/localdownload` returning 502 at the exact second of the kill. The
analysis is his; I am sending it because it is still open here.

Serving a completed book from the live queue calls `get_book_data`,
which reads the whole file into bytes so the route can wrap it in a
`BytesIO` for `send_file`. That is two copies of the book, with one
resident for the length of the client transfer, to hand over a file that
is already sitting on disk.

`get_book_path` returns the path the task already holds and `send_file`
streams it. The history fallback a few lines up in the same route has
always worked this way, so this makes the two paths consistent rather
than introducing anything new. `get_book_data` stays for callers that
genuinely want the bytes, now documented as the expensive option.

## The fetch side has the same problem, and this PR does not fix it

I should note: `download_url` in `shelfmark/download/http.py` still
builds the inbound file in a `BytesIO` and returns it, so a large
download is fully resident while it fetches. @DrNgo's commit fixes that
too, with a `tempfile.SpooledTemporaryFile(max_size=...)` so small
payloads stay in memory exactly as they do now and large ones spill to
disk.

I left it out deliberately. It changes the return type of `download_url`
from `BytesIO` to a file object, which touches several callers, and it
lands in a file that has been reworked around bypass handling, waiting
rooms and resume since his branch point. That deserves its own PR rather
than riding along with a two-function change. I may send it later; if
@DrNgo sends it first, his should win, and if you would rather have both
together say so and I will hold this one.

## Verification

- `tests/download/test_orchestrator_retry.py`: `get_book_path` returns
the path without opening the file (the test fails the run if it does),
and reports a missing file rather than handing back a dead path.
- The existing `/api/localdownload` tests still pass unchanged,
including the history fallback and the ownership checks.
- Full suite (3222), ruff, ruff format, basedpyright, vulture green.
2026-09-25 18:06:15 -04:00
splitsec2 bc03ad062e fix(http): keep the host of a protocol-relative download link (#1368)
`get_absolute_url()` replaced both `netloc` and `scheme` whenever either
one was missing. A protocol-relative href such as
`//cdn.example.org/f.epub`, scraped from a page on
`https://annas-archive.org/...`, parses with a netloc and an empty
scheme, so it came back pointing at the page's own host. The download
then 404s and the source is skipped.

Each field now falls back to the base URL only when the parsed URL does
not supply it. Plain relative paths resolve exactly as before, which the
control test covers.

This affects the Z-Library, welib and generic download link handling in
`release_sources/direct_download/annas_archive.py`.

## Verification

- New `tests/download/test_http_absolute_url.py`: a protocol-relative
link keeps its own host, and a plain `/path` still resolves against the
base. The first fails on current main and passes here.
- Full suite (3165), ruff, ruff format, basedpyright, vulture green.
2026-09-20 23:04:22 -04:00
Paul Rimmer d4619be69a Feature: Show the AA search result stats (#1362)
When doing a direct search for a book, show the stats of the AA results.
For example if we search for "The Great Gatsby", AA reports it has 240
hits and shows the first page of 50. Provide this stats info in the
shelfmark webUI via ResultsSection.tsx and ReleaseModal.tsx. This table
shows what should be displayed based on the total number of hits found:

|Total	|Display|
|-------|-------|
|1	|"Result 1 (1 Total)"|
|6	|"Results 1-6 (6 Total)"|
|144	|"Results 1-50 (144 Total)"|
|500+	|"Results 1-50 (500+ Total)"|

Currently, shelfmark also only shows us the first 50 search hits even if
there were more available from AA. This could be added later if
considered desirable.

As usual, a picture is worth a 1000 words:

<img width="1012" height="610" alt="direct-results-stat"
src="https://github.com/user-attachments/assets/8c0c688f-444e-482a-a9a3-2dee5a5563b8"
/>

<img width="1013" height="741" alt="universal-results-info"
src="https://github.com/user-attachments/assets/399bbb79-87f2-4432-a3d0-64937795f5f1"
/>

Coded with llama.cpp, opencode and 🤖
2026-09-20 13:03:32 -04:00
Marcelo Rodrigo b7002a6eca feat: Add TorBox client support and settings integration (#1342)
Add **TorBox** as a torrent download client for Prowlarr releases.

Users can select `TorBox` in the download client settings, configure it
with the new `TORBOX_API_KEY` environment variable, and verify their
credentials with the connection test button.

The integration supports both magnet links and `.torrent` files. It
tracks the torrent lifecycle through TorBox, downloads supported book
and audiobook files from the TorBox CDN, preserves safe nested file
paths, and cleans up remote and local download state.

Important: Shared HTTP download logs omit full download URLs and
URL-bearing exception text to avoid exposing credentials, following best
practices. This applies to all clients that use the shared
`download_url()` path; URLs remain available to the HTTP operations
themselves.

---
There is already related work in progress in #1173, which includes both
torrent and direct-download support for TorBox.

This PR is not intended to replace or compete with that contribution. It
offers the tested torrent client functionality as a smaller, focused
change that can make TorBox available to the community sooner. The
direct-download integration proposed in #1173 remains valuable and could
be reviewed or introduced separately.

Automated tests cover configuration, connection validation, magnet and
torrent-file submission, API errors, status and progress handling, file
retrieval, path traversal protection, cancellation, cleanup, and
sensitive URL redaction.

I also validated the complete flow locally with several magnet links and
`.torrent` downloads. TorBox processed the torrents and Shelfmark
downloaded the resulting files as expected.

AI was used to help with the implementation, with human validation. This
PR and long description? Took me some good minutes at night after work,
but gives me joy to open this PR to share with the community this
improvement.
2026-09-19 23:15:57 -04:00
CaliBrain e5dd34ae0e fix: unbreak main and follow up on the Blackhole handoff review (#1346)
DownloadHistoryService.record_download and updated the single production
caller, but not the eleven in the test suite, leaving main red with 32
failures. Pass None, which is what the pre-#1336 behaviour recorded.

For the Blackhole handoff (#1345): add_download publishes the torrent
before the cancel check runs, and BlackholeClient.remove() is a no-op,
so
the watcher picks the file up regardless. Reporting a bare "Cancelled"
hid
that from the user. Name the completed handoff in the cancellation
message
instead, drop the _handle_cancelled_download call whose usenet branch
cannot apply to a handoff-only client, and record why the orchestrator
no
longer verifies HandoffResult.path.

Finally, make tests/direct_download a package: test_libgen_extract.py
imports tests.libgen.sample_html across test directories, so without an
__init__.py pytest named its modules by bare basename and a same-named
module elsewhere would collide.
2026-09-17 16:27:53 -04:00
Atirna 8f608f2e64 fix(download): complete consumed Blackhole handoffs (#1345)
A Blackhole watcher can consume the torrent before Shelfmark checks it,
leaving the task in error even though the handoff succeeded. Complete
the handoff when `add_download` successfully publishes the file, and
stop requiring a `HandoffResult` path to remain present.

Follow-up to #1312.

## Verification

- A watcher that immediately reads and removes the torrent receives the
exact bytes. The task changes from ERROR before this fix to COMPLETE
afterward, without running book postprocessing.
- The consumed-file regression fails on current main and passes here.
Resident files, write failures, cancellation, magnet rejection and
normal downloads remain covered: 81 focused tests pass.
- Ruff lint and formatting pass for the changed files.
2026-09-17 15:51:20 -04:00
David Young 2bb84a17a2 Extract archives when zip/rar are enabled as supported formats (#1343)
The default audiobook formats include `zip` and `rar`.
`scan_directory_tree` checks the supported-format list before checking
for archives, so a downloaded archive lands in `book_files` and is
imported as-is. The extraction branch in `collect_directory_files` is
never reached.

This keeps archives out of `book_files`, so they always take the archive
path: extracted when extraction is allowed, imported as-is when it isn't
(unchanged).

Tests added in `tests/download/test_postprocess_scan_archives.py`; three
of the four fail without the change.
2026-09-17 15:50:52 -04:00
Atirna 265da07d7f feat(download): add Blackhole torrent handoff (#1312)
## Why

Blackhole users need Shelfmark to hand a torrent file to their existing
downloader instead of importing the downloaded book itself.

## Change

- add Blackhole as a torrent client with a configurable watched
directory
- prefer a fetched `.torrent` file for Blackhole while preserving magnet
preference for other clients
- complete the queue task after the handoff without invoking book
post-processing

## Verification

- `uv run pytest -q tests/prowlarr/test_blackhole_client.py
tests/prowlarr/test_handler.py tests/newznab/test_handler.py
tests/download/test_orchestrator_lifecycle.py`
- `uv run basedpyright shelfmark/download/clients/blackhole.py
shelfmark/download/clients/__init__.py
shelfmark/download/clients/base_handler.py
shelfmark/download/clients/settings.py
shelfmark/download/orchestrator.py shelfmark/release_sources/__init__.py
shelfmark/release_sources/prowlarr/utils.py
shelfmark/release_sources/prowlarr/handler.py
shelfmark/release_sources/newznab/handler.py
tests/prowlarr/test_blackhole_client.py tests/prowlarr/test_handler.py
tests/newznab/test_handler.py
tests/download/test_orchestrator_lifecycle.py`

Fixes #1229
2026-09-11 00:46:47 -04:00
Ama Senevirathneandamasen02 c18da92569 fix(postprocess): attach unmatched chaptered audio files to existing book group (#1176) (#1309)
### Summary
Fixes #1176

When downloading an audiobook with many chaptered tracks (e.g. 250+
`.flac` or `.mp3` files), indexer XML or release metadata often caps the
file list at ~100-110 entries. When the release extracts on disk,
`match_plan_to_files()` matched those first ~110 files to the planned
book group, while the remaining 140+ files fell into `unmatched` and
triggered fallback heuristic grouping.

Because heuristic grouping parsed the folder name (`Westwell - Hot &
Cold (2023)`) and stripped the series/author prefix, it generated a
second book titled `Hot & Cold` containing the remaining tracks,
resulting in two split book folders.

### Changes
- In `match_plan_to_files()`
(`shelfmark/download/postprocess/packs.py`), check `unmatched` files
before falling back to heuristic multi-book splitting.
- If an unmatched file is chaptered audio (`.flac`, `.mp3`, `.aac`,
etc.) and shares the directory with an existing book group, or if the
plan was a single-book plan, append it to that group instead of creating
a secondary book.
- Non-chaptered standalone books (e.g. `.m4b`, `.epub`) or files in
separate subfolders continue to fall back to heuristic grouping as
before.
- Added unit tests in `tests/download/test_packs.py` verifying:
1. Truncated track list in single folder properly appends remaining
chaptered tracks without splitting.
2. Single-book plan with multi-disc audio files (`CD1`/`CD2`) groups
together cleanly.
3. Multi-book packs with unmatched chaptered tracks route each track to
its respective book folder.

### Testing
Ran `uv run pytest tests/download/test_packs.py` and `uv run pytest
tests/core/test_processing_packs.py` (all passed cleanly). Checked type
annotations with `basedpyright` (0 errors) and formatting with `ruff`.

Co-authored-by: amasen02 <amasen02@users.noreply.github.com>
2026-09-04 11:01:55 -04:00
CaliBrain 9452ebc70d fix(bypass): stop handing solvers DDoS-Guard's ?check=1 probe URL (#1300)
html_get_page follows Anna's Archive redirects by hand, and DDoS-Guard's
gate
answers /search with a 302 to the same path plus `check=1`. The follower
walks
that handshake by reassigning `current_url`, so every downstream handoff
- the
403 branch, the 503-challenge branch, both redirect-loop rescues -
passed the
*probe* URL to the bypasser rather than the page we actually wanted.

A solver opens that in a fresh browser holding none of the cookies the
probe
exists to collect, so DDoS-Guard cannot verify it automatically and
serves the
manual CAPTCHA page that nothing can solve. The #1292 log is exactly
that: a 403
handed off on `&check=1`, FlareSolverr answering "Challenge solved!",
and a
4721-byte DDOS-GUARD captcha page coming back.

- `_solvable_url()` strips the probe parameter, applied at the single
choke point
in `_run_bypasser` so all four handoffs are covered. Scoped to the hosts
whose
redirects we follow manually; a URL without the parameter is returned by
  identity, so nothing else is re-encoded.

The same reports showed three further defects, all of which stand
whatever the
host was reacting to:

- The external bypasser logged that the solve had not cleared the
protection and
then returned the challenge page as a success. That skipped the one
recovery
left - get_bypassed_page's retry-and-rotate loop, where the next mirror
is a
different DDoS-Guard host - and filed the captcha page's own __ddg
cookies as
that host's clearance, to be replayed on every later request. It now
raises
  ChallengeNotSolvedError before storing anything.

- "Check that the bypasser is reachable and working" was the one piece
of advice
guaranteed to waste the reporter's time: it was reachable, it ran a full
solve,
and it returned a captcha. ChallengeNotSolvedError carries the marker so
the
  search layer can name the host as the cause instead of the bypasser.

- The untabled-page fingerprint logged `attempt_url`, which
html_get_page has
since rotated past. The #1298 bundle reported the page against
annas-archive.gl
when the body had come from .pk - the triage cost #1289 added the line
to
remove. The search now asks for the response URL and logs that. Its
give-up
shape is the tuple ("", url), which is truthy, so the exhaustion check
reads
  the body rather than the response.

Regression fixtures are built from the pages in the reports. The two
behavioural
handoff tests were checked against the unfixed code: both fail there,
reproducing
the reporter's log line verbatim.

Refs #1292
Refs #1298
2026-09-02 16:12:52 -04:00
CaliBrain 97e289ae13 fix: search, Prowlarr and qBittorrent follow-ups (#1276, #1283) (#1284) 2026-08-30 03:09:13 -04:00
CaliBrain b25acdb2ad fix(packs): don't disrupt normal downloads when inspecting for packs (#1274)
Follow-ups to the multi-book pack feature (#1270), which inspects every
release before download. Two behaviours leaked into the ordinary
single-book
flow and are corrected here:

- A flat folder of chaptered audio (`01 - Chapter.mp3`, `02 - ...`) was
detected as a pack, because each track name parses to a series position,
so
clicking download popped the review panel for one normal audiobook. Flat
folders are now split one-book-per-file only with real evidence of
distinct
books: two or more series positions, more than one title, and no
chaptered
audio (only the single-file m4b/m4a containers and ebook formats
qualify).
  Subfolder packs and flat m4b/m4a packs are unchanged.

- Every release that couldn't be inspected (usenet, magnet-only, sources
  without a list_files hook, ABB single-file) showed an info toast on
download. That is now a console.warn, so a normal download is silent
again.

Adds regression tests for the chaptered-mp3 cases.
2026-08-27 01:00:54 -04:00
Lance Marks f441b85da2 feat(packs): inspect multi-book releases and file each book separately (#1270)
## Multi-book packs: inspect a release before download and file each
book separately

Closes #576

### Problem

One queued release is always treated as one book. When a torrent is
actually a whole series
(`Series/Book 1 - Title/…`, or a flat folder of `Series 1.0 - Title.m4b`
files), post-processing
walks the whole tree, flattens every file into one list and renames them
`Title - 01…10` under the
searched book's `{Author}/{Title}`. Audiobookshelf then sees a single
10-file "book" and the user
has to re-file everything by hand.

### What this does

Most releases expose their file list *before* anything is downloaded, so
the split is decided up
front and approved by the user, then the download is fire-and-forget:

1. **Inspect** – clicking a release's download button now calls `POST
/api/releases/inspect`
first. A new optional `DownloadHandler.list_files(release_data)` hook
returns the release's
   files without downloading:
- **AudiobookBay** reads the torrent file table off the detail page it
already fetches (the
page is now cached for 120 s, so inspect + download cost ABB one
request).
- **Prowlarr** parses `info.files` from the `.torrent` it already
fetches (the existing 120 s
torrent-fetch cache is reused). Magnet-only and usenet releases report
"can't inspect".
   - Other sources default to `None`.
2. **Review** – if the plan contains more than one book, the Find
Releases modal swaps the list
for a review panel: one row per book with editable title / series
position / year, expandable
file lists, non-book sidecars (`.txt`, covers) shown as ignored, a
"Treat as a single book"
switch, and **Download N books**. Single-book releases queue
immediately, exactly as before.
3. **File** – the approved plan travels with the task
(`DownloadTask.book_plan`, retry-safe) and
post-processing files each book through the existing transfer code, one
book at a time
(`dataclasses.replace(task, title=…, series_position=…, year=…)`), so
organize/rename
templates, part numbering (now scoped per book), hardlinks, torrent
copy-preserve and usenet
   handling are unchanged. Status reads `Complete (N books, M files)`.
4. **Fallback** – when a release can't be inspected the user gets a
toast, and a small
"Multi-book pack" toggle in the modal header forces a heuristic split
(subfolder = book, or
   one book per file when the file names carry series positions).

Planning lives in `shelfmark/download/postprocess/packs.py` and is
shared by the inspect endpoint
and post-processing, so what the user approved is what gets filed. The
name parser strips
`Book 3 -`, `03 -`, `1.0 -`, `3.`, `[03]`, `#3`, a leading series name,
labels like
"An Expanse Novella -", repeated titles (`Gods of Risk 2.5 - Gods of
Risk`) and a trailing
`(Year)`; author and series name come from the book that was searched,
and the searched book's
own series position is never applied to its siblings.

### Files

- `shelfmark/download/postprocess/packs.py` (new) –
`PackFile/PackBook/PackPlan`, `plan_pack`,
`parse_pack_book_name`, `group_files_into_books`, `match_plan_to_files`
- `shelfmark/core/release_inspect_routes.py` (new) – `POST
/api/releases/inspect`
- `shelfmark/release_sources/__init__.py` – `DownloadHandler.list_files`
hook
- `shelfmark/release_sources/audiobookbay/{scraper,handler}.py` –
detail-page cache,
  `extract_file_list`, `list_files`
- `shelfmark/release_sources/prowlarr/handler.py`,
`download/clients/torrent_utils.py` –
  `extract_file_list_from_torrent`, `list_files`
- `shelfmark/core/models.py`, `download/orchestrator.py` – `multi_book`
/ `book_plan` fields,
  queue + retry serialization
- `shelfmark/download/postprocess/transfer.py`, `pipeline.py`,
`outputs/folder.py` – per-book
  transfer branch and status message
- `src/frontend`: `components/PackReviewPanel.tsx` (new),
`ReleaseModal.tsx`, `App.tsx`,
`services/api.ts`, `types/index.ts`, `utils/releasePayload.ts` (payload
builder moved out of
  `App.tsx`), `utils/packReview.ts`
- `docs/dev/release-sources-plugin-guide.md` – documents the
`list_files` hook

### Out of scope (follow-ups)

- Listing files from an NZB (Shelfmark already fetches the bytes; `<file
subject>` names are noisy)
- Inspecting magnet links via qBittorrent's files API after a paused add
- BookLore / email outputs (they ignore `book_plan`; noted in code)
- The combined ebook + audiobook flow

### Testing

**Automated** (`make checks`, `make python-test`, `make frontend-test`
all green; the only
failures on my machine are the pre-existing
`tests/config/test_entrypoint_permissions.py` cases,
which need bash ≥ 4 and fail identically on `main` under macOS bash
3.2):

- `tests/download/test_packs.py` – name parsing (markers, series name,
novella labels, repeated
titles, bare numeric titles like `1984`), nested / flat / mixed /
deeper-nested packs, single
wrapping folder not treated as a pack, plan-to-disk matching with
basename fallback
- `tests/core/test_processing_packs.py` – full `post_process_download`
runs on a real temp
filesystem: approved plan files each book under its own
`{Author}/{Title}`, heuristic split
of a nested pack, searched book's series position does not leak,
multi-file book inside a pack
keeps `- 01/- 02` per book, hardlinked torrent pack leaves the seeding
tree intact, no pack
fields ⇒ behaviour unchanged, single group degrades to the searched
title, status message
- `tests/core/test_release_inspect_routes.py` – plan response,
not-inspectable, handler errors
  never 500, unknown source / missing `source_id` ⇒ 400, login required
- `tests/audiobookbay/test_file_list.py` – file-table scraping from real
ABB markup (multi-file
and single-file pages), handler host validation, one page fetch shared
by magnet + file list
- `tests/prowlarr/test_torrent_file_list.py` – multi-file / single-file
`.torrent` parsing,
  handler behaviour for torrent URL vs magnet vs usenet vs cache miss
- `tests/download/test_orchestrator_pack_fields.py` – queue-time parsing
and retry round-trip
- Frontend: `releasePayload.test.ts`, `packReview.test.ts` (vitest)

**Manual, on a real deployment** (arm64 image built from this branch,
run as a side container
next to production with the same qBittorrent / Audiobookshelf setup,
`FILE_ORGANIZATION_AUDIOBOOK=organize`,
hardlinks on):

- AudiobookBay "The Expanse Complete 2.0" (7.87 GB, 36 files): clicking
download opened the review
panel in ~1 s showing **18 books · 18 files · 18 files ignored** (the
`.txt` sidecars), with
series positions 0.1–9.5 and years parsed from the file names; novella
labels stripped
("The Churn", "The Butcher of Anderson Station"). Editing a title in the
panel works.
Confirming queued one task; the magnet resolved from the cached page in
~30 ms; after the
download the task reported `Complete (18 books, 18 files)`, 18 hardlinks
landed as
`audiobooks/James S. A. Corey/<Title>/<Title>.m4b`, the torrent kept
seeding, and
Audiobookshelf scanned each folder as its own book (title, author,
embedded chapters).
- A second pack ("Expanse [01 - 9.5]", `Title N - Title` naming) was
inspected to verify the
  repeated-title rule and the Back button, without downloading.
- Single-book releases still queue immediately with no extra UI.
2026-08-27 00:40:08 -04:00
CaliBrain ddc26f01b6 fix(download): escalating per-host cooldown on HTTP 429 (#1263)
Anna's Archive 429-throttles the source IP after repeated automated
requests.
The bypasser could clear the DDoS-Guard challenge but not the 429, so
each retry
re-solved, re-spawned Chrome, and rotated mirrors that share the same IP
- a
costly loop that never converged.

Add a process-global, per-host cooldown that escalates 2 -> 5 -> 10 ->
15 -> 30
minutes each time a host 429s again after its window elapsed, resetting
after a
long clear gap. Mirror selection skips cooling hosts and the bypasser
refuses to
solve one, so a throttled host fails fast instead of storming the
solver.
2026-08-24 13:08:07 -04:00
CaliBrain 12d554a92f fix(download): hand a 503 carrying a challenge to the bypasser (#1238)
503 is in RETRYABLE_CODES, and the bypasser is only ever reached from
the 403
branch and the AA redirect-loop rescues. Once Z-Library re-serves its
DDoS-Guard
interstitial with the same cookie the #1188 handshake already echoed
back, the
request has nothing left to try and spends every attempt on the same
wall.

Gate the handoff on the response body rather than the status, so a
genuine
overloaded-origin 503 keeps its retry path, and on
allow_bypasser_fallback, so
best-effort fetches still fail fast. The challenge indicators move out
of
internal_bypasser into shelfmark/bypass/challenge.py so http.py can use
them
without importing SeleniumBase, which is lazily imported precisely
because it
is optional.

Refs #1233
2026-08-20 10:29:52 -04:00
CaliBrain 651096ed7b fix(bypass): reuse external bypasser clearance instead of re-solving (#1223)
Direct download was unusable behind an external bypasser (FlareSolverr /
Byparr): every request paid a 403 plus a full solve, and a search that
never ran was reported to the user as "No books found".

Clearance was discarded on the external path. get_cf_cookies_for_domain
and get_cf_user_agent_for_domain returned {} / None whenever
USING_EXTERNAL_BYPASSER was set, and _fetch_via_bypasser read only
solution.response - dropping solution.cookies and solution.userAgent,
which FlareSolverr-compatible services do return. A solve therefore
cleared the one request that paid for it and nothing else, and a file
download - which the solver cannot proxy, being binary - presented no
clearance at all. Diagnosed from a v1.3.9 debug bundle: ~35s in the
bypasser per search, on every search.

- Move the cookie jar out of internal_bypasser into bypass/cookie_store.
  internal_bypasser imports seleniumbase at module scope, which is the
  dependency an external-bypasser deployment is entitled not to have, so
  it cannot host a store the external path depends on.
- Harvest solution.cookies and solution.userAgent after a successful
  solve. The existing filtering applies unchanged, so the per-check
  __ddg8_/__ddg9_/__ddg10_ trio is still dropped and the external path
  cannot reintroduce the ?check=1 loop fixed in ebb833a. The UA matters
  as much as the cookies: Cloudflare ties cf_clearance to the UA that
  solved the challenge.
- Read cookie fields from either shape - CDP objects or JSON mappings.
  Both use the same field names, expires included.
- Point http.py's getters and _purge_clearance at the shared store, so
  either bypasser fills and drains the same jar.
- Give the Docker helper-subprocess handoff explicit export_store /
  import_store rather than reaching into module globals.

An unsolved challenge was also indistinguishable from an empty result.
_looks_like_aa_page() counted the challenge markers as "recognisably
AA", so _fetch_search_table handed a DDoS-Guard interstitial back as a
legitimate no-table response and the user was told their query found
nothing when the search never ran. Split challenge detection out and
raise SearchUnavailableError with the reason instead. The mirror is
still not quarantined - every mirror shares the same protection, so it
is not the mirror's fault.

Verified: 2531 unit tests pass; ruff, basedpyright and vulture clean;
e2e bypasser-external profile passes (5). Its mock FlareSolverr already
returned cookies and userAgent from /v1 - the contract was there,
shelfmark was not reading it.

Refs #1220. Deliberately not "Fixes": this removes the re-solve and
makes a failed solve legible, but if Byparr genuinely cannot clear AA's
current DDoS-Guard, the reporter now gets that as an error rather than a
silent "no books found". The download path may swallow interstitials the
same way; not audited here.
2026-08-16 12:08:45 -04:00
CaliBrain ebb833a82c fix(bypass): discard rejected DDoS-Guard cookies instead of replaying them (#1221)
A cookie that has been rejected was kept and presented again on every
later
request, so a single bad clearance could re-arm the challenge
indefinitely.

Cookie storage:
- Enforce expiry for every stored cookie, not just cf_clearance.
DDoS-Guard
domains have no cf_clearance, so the existing check never fired for them
and
  expired cookies were replayed forever.
- Stop storing the per-check cookies __ddg8_/__ddg9_/__ddg10_ and
ddg_last_challenge. Captured live from Anna's Archive, these carry the
client
IP and the timestamp the check was issued (~40 min), versus ~1 year for
the
  __ddg1_/__ddg2_/__ddgid_ clearance. Replaying an IP-bound token stops
describing the caller as soon as the egress IP changes, which is routine
  behind a VPN.

Failure handling — every path that is rejected while carrying cookies
now
purges them, not just the redirect loop:
- 403 returned while presenting cookies.
- Cached-cookie attempt rejected, whether by status or by redirect loop.
- Factored the purge into _purge_clearance, guarded on a non-empty
hostname
since clear_cf_cookies("") means "every host" and would wipe clearance
for
  sites that are working fine.

Also fix the search warm-up switches shipped inert in v1.3.8:
SEARCH_WARMUP_ENABLED and SEARCH_WARMUP_QUERY are not in the settings
registry, and config.get only consults the environment for keys it
knows, so
both always returned their defaults — the warm-up could not be turned
off or
retargeted. Read os.environ first.

Refs #1220. Deliberately not "Fixes": the reported failure could not be
reproduced on v1.3.8 from a stable IP (the reporter's own queries all
returned
200 on both the pre- and post-change builds), and the new purge paths
did not
fire in live testing because the failures arrive as redirect loops,
which were
already purged. These are correctness fixes with no measured effect on
that
issue. The underlying problem remains that Chrome-obtained cookies never
satisfy DDoS-Guard when replayed by requests, so every search still
re-solves.

Verified: 2542 unit tests pass; ruff, basedpyright and vulture clean;
e2e
platform baseline (10), full (6) and bypasser-external (5) all pass;
five
sequential live searches against Anna's Archive all returned 200 with
zero
"Exceeded 30 redirects".
2026-08-15 17:08:11 -04:00
CaliBrain b656f019be feat(download): add DoH wireformat support, mirror quarantine, and search warmup (#1218)
- Add RFC 8484 DNS wireformat codec and HTTP/2 support (httpx) for
Quad9/OpenDNS DoH providers.
- Quarantine dead, parked, or seized mirrors for the session on hard
failure (DNS errors, connection refused, 410/451, parked pages) while
preserving bypass clearance on live mirrors.
- Add background startup search warmup to prime DNS, elect mirrors, and
pre-solve protection challenges to eliminate cold-start search latency.
- Add comprehensive test suites for DoH wireformat, mirror quarantine,
parked domain detection, and search warmup.
2026-08-15 14:17:44 -04:00
3e2a7a48d5 fix: clear the DDoS-Guard cookie probe on AA search (#1209)
## Summary

Two failure modes on the same code path, both reported this week: Anna's
Archive `/search` is gated behind a DDoS-Guard cookie probe that the
manual redirect follower can never satisfy.

**#1202 — the cookie is dropped on every hop.** AA URLs set
`allow_redirects = False`, so `html_get_page` follows redirects by hand.
The 302 to `?check=1` carries a `Set-Cookie` (`__ddg*`) that has to come
back on the next request. Because cookies are passed per call and
`requests` keeps no jar across manual hops, it was discarded each time
and the server just re-issued the same redirect until `_MAX_REDIRECTS`
raised `TooManyRedirects`. The file already had the right helper —
`_new_cookies()` — but only the 503 Z-Library handshake branch called
it.

**#1204 — the loop never reaches the bypasser.** `TooManyRedirects`
isn't in `_is_retryable_error` and carries no status code, so the 403
rescue path (`status == _HTTP_STATUS_FORBIDDEN`) never fired and all
attempts repeated the identical failure — ~2.5 min, surfacing as the
misleading "Network restricted or mirrors are blocked".

These interact, which is why #1202's fix alone isn't enough. Requests
merge as `cookies={**handshake_cookies, **cookies}`, so **stale bypasser
cookies override the fresh handshake ones** — once `_cf_cookies` holds
an expired `__ddg*`, the probe can never clear no matter how faithfully
we echo. Hence one search per restart, exactly as #1204 describes.

## Changes

1. Harvest cookies in the same-host redirect branch, the way the 503
branch already does. `_new_cookies()` returns only *new* values, so a
server re-sending an identical cookie yields an empty dict and a genuine
redirect loop still terminates at `_MAX_REDIRECTS`.
2. Treat a redirect loop as a detected challenge: purge the stored
cookies for that host and switch to the bypasser, instead of burning the
retry budget. Gated on `allow_bypasser_fallback` and
`_is_cf_bypass_enabled()`, and skipped when already bypassing, so
AudiobookBay (`allow_bypasser_fallback=False`) and external-bypasser
setups are unaffected.

The broader point in #1204 stands — the fallback would be better gated
on "challenge detected" than on specific status codes, since DDoS-Guard
presents at least three faces (403 js-challenge, 429, and this redirect
loop). This PR fixes the two live exits without that refactor.

## Tests

Two regression tests, both failing before and passing after:

- `test_html_get_page_echoes_cookies_across_same_host_redirects` — the
fake server only returns results if `__ddg2_` comes back on the
`?check=1` hop.
- `test_html_get_page_redirect_loop_purges_cookies_and_bypasses` —
asserts the stored cookies are cleared, the bypasser runs, and the loop
is cut short rather than repeated per attempt.

`ruff check` and `ruff format` clean. `tests/download/` passes except
`test_download_url_ignores_zlib_cookie_refresh_failure`, which fails
identically on unmodified `main` in my environment (no `seleniumbase` —
the `browser` extra isn't installed).

## Verification

Applied on a live v1.3.7 install (Debian LXC, internal CDP bypasser).
Before: every search timed out through 10 retries with
`TooManyRedirects`, zero results. After:

```
http.py:455 - Redirect loop detected; switching to bypasser
internal_bypasser.py:756 - Bypass successful using _bypass_method_cdp_gui_click
internal_bypasser.py:322 - Extracted 9 protection cookies for annas-archive.pk
direct_download.py:1865 - Found 24 releases via ISBN
```

~25 s per search, results render. Note the second search still re-solves
the challenge, since the freshly stored cookies go stale immediately —
the design issue #1204 raises, left for the broader fix.

Fixes #1202
Fixes #1204

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_012Ln3yVj3sWHG2c6T78W1we

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: CaliBrain <calibrain@l4n.xyz>
2026-08-15 11:27:03 -04:00
CaliBrain 0a5256ecbb fix(download): reconcile the two AA redirect-loop rescues (#1213)
#1210 and #1212 both added a DDoS-Guard `?check=1` rescue, and #1212 was
branched before #1210 landed, so the merged result had two of them with
identical guards. #1212's inline handoff returns before the raise that
#1210's exception handler keys on, so the handler was shadowed and its
stale-cookie purge — the substance of #1210 — never ran. Its regression
test has been failing on main since the merge.

Fold both into one path:

- `_redirect_loop_handoff()` purges the host's stale clearance cookies,
  then bypasses, so the inline AA handoff and the exception handler
  cannot drift apart again.
- The exception handler keeps its own reason to exist: non-AA hosts run
with allow_redirects=True, so `requests` raises the loop itself and the
  manual AA follower never sees it. It now invokes the bypasser directly
  rather than setting a flag and continuing, which was a no-op at
  MAX_RETRY=1 for the same reason the 403 handoff was.
- An unrescuable loop returns empty instead of raising TooManyRedirects
into the retry path. That error is not retryable and carries no status,
  so `/dyn/md5/summary` (allow_bypasser_fallback=False) re-ran the full
  6-redirect loop on all 10 attempts: 60 requests to AA and ~30s of
  backoff, measured. Every AA mirror shares the challenge, so there is
  nothing to rotate to.
- `allow_bypasser_fallback` docs now describe what the flag actually
  gates; the old text predated #1198 and named the wrong callers.
2026-08-15 11:14:13 -04:00
Zoltán SzabóandKukkerem 056ddd372a Send DDoS-Guard's ?check=1 redirect loop to the bypasser (#1210)
Fixes #1204.

## Problem

#1198 sends a gated AA `/search` to the bypasser when the origin answers
403.
DDoS-Guard has a second response: when the clearance cookies from an
earlier
solve go stale, it serves an endless `?check=1` redirect instead.

`requests` follows that until `_raise_too_many_redirects`, and
`TooManyRedirects` carries no status code, so `status ==
_HTTP_STATUS_FORBIDDEN`
is false and the rescue never runs. All 10 retries re-send the same dead
cookies, then the search fails as `Unable to reach download source.
Network
restricted or mirrors are blocked.`

Direct-download search therefore works once per container start, and
stays dead
after the stored cookie ages out.

v1.3.7 (`sha256:520715f3…`), internal bypasser, mirrors `.gl/.pk/.gd`:

```
17:04:36 internal_bypasser.py:756 - Bypass successful using _bypass_method_cdp_gui_click
...
17:11:39 http.py:483 - Retry 1/10 for https://annas-archive.gl/search?...&check=1:
    TooManyRedirects: Too many redirects
17:12:12 http.py:493 - Giving up after 10 attempts
17:12:12 main.py:2870 - Release search failed for source direct_download:
    Unable to reach download source. Network restricted or mirrors are blocked.
```

The token is short-lived, which is what makes this reachable in normal
use:

```
$ curl -sD - 'https://annas-archive.gl/search?...&check=1'
HTTP/2 403
server: ddos-guard
set-cookie: __ddg8_=…; Expires=Fri, 14-Aug-2026 15:39:38 GMT   # issued 15:19:38, 20 min
```

## Fix

Handle the loop like the 403: drop the domain's stored cookies, then
retry
through the bypasser. The branch sits above the `status ==` ladder
because
`_get_status_code()` returns `None` for this exception.

Cookies are purged only for the internal bypasser; with an external one
`get_cf_cookies_for_domain()` already returns `{}`.

Related but not changed here: `get_cf_cookies_for_domain()` enforces
expiry for
`cf_clearance` only, so `__ddg*` cookies are never evicted on age, which
is why
they go stale. This patch makes the rescue fire whatever the reason the
cookies
stopped working.

## Verification

The regression test drives a real redirect loop through `html_get_page`
(302 to `&check=1`, exception raised by the production path rather than
faked)
and asserts the cookies are purged and the bypasser runs once.

- `pytest tests/download/test_http_bypasser_fallbacks.py`: 8 passed.
`test_download_url_ignores_zlib_cookie_refresh_failure` fails in my
checkout
  on a missing `seleniumbase`, unrelated to this change.
- `ruff check`, `ruff format --check`: clean.
- Running in production since 2026-08-14 on v1.3.7 with only this file
replaced:
six direct-download searches, five served, three books downloaded end to
end,
against one search per container start before. The rescue mid-download:

```
19:12:14 http.py:449 - Redirect loop detected; switching to bypasser:
    https://annas-archive.gl/md5/cb8fba7abae800ddbae1adfb8d7699d9?&check=1
19:12:38 internal_bypasser.py:756 - Bypass successful using _bypass_method_cdp_gui_click
19:14:36 direct_download.py:1142 - Resolved download URL [aa-slow-nowait]: …
19:14:47 orchestrator.py:735 - download finished; starting post-processing
```

## Separate issue this exposes

DDoS-Guard does not accept a solved cookie from plain `requests`
traffic, so
after this patch the rescue runs for nearly every AA URL.
`internal_bypasser.get()`
serializes all solves on one module-wide lock and builds a fresh Chrome
each
time: 11-16 s uncontended, 43-52 s under concurrent load, measured on
the host
above. Correctness is cheap here, latency is not. Happy to open a
separate PR
for a warm browser session if that direction is welcome.

Co-authored-by: Kukkerem <Kukkerem@users.noreply.github.com>
2026-08-15 10:59:41 -04:00
CaliBrain 03e219eb43 Let the bypasser solve bot challenges on Anna's Archive search (#1198)
Anna's Archive put a DDoS-Guard JS challenge in front of /search: the
homepage still returns 200, but /search and /md5/<id> answer 403 on
every mirror (.gl, .pk, .gd all confirmed). Search fetched both with
allow_bypasser_fallback=False, which rotates mirrors on a 403 instead of
invoking the bypasser, so it walked the whole mirror list, exhausted it,
and surfaced "Unable to reach download source. Network restricted or
mirrors are blocked." as a 503 on every query. Adding mirrors could not
help — they sit behind the same gate — and neither could USE_CF_BYPASS,
since search never reached that branch.
Fetch search and the detail page with allow_bypasser_fallback=True so a
403 hands over to the bypasser, which already detects this challenge
(DDOS_GUARD_INDICATORS matches the live page). Echoing the __ddg cookies
back does not clear it; it needs real JS execution. The download-count
fetch keeps allow_bypasser_fallback=False: it is decoration on the
details modal and not worth holding the modal open for a browser solve.
Fixes #1196
2026-08-13 13:06:11 -04:00
CaliBrain e320b7623d Fix LOG_LEVEL being ignored and Z-Library 503 cookie gate (#1188)
LOG_LEVEL never reached the app logger: env.py hardcoded the level to
DEBUG or INFO, so INFO lines kept appearing under LOG_LEVEL=error. Read
it from the env var and advanced settings, normalize unknown values to
INFO, and expose it as a setting. entrypoint.sh now normalizes
gunicorn's level too, so a typo falls back to info instead of stopping
the container from booting.

Z-Library gates the first hit on /md5/<hash> with a 503 whose only
payload is a Set-Cookie; echoing that cookie back returns the 302 to the
real page. html_get_page dropped it and re-ran the same rejected request
on every retry, ending in "No download URL resolved". Retry once with
the
cookies the 503 issued.

Fixes #1185
Fixes #1187
2026-08-11 23:27:44 -04:00
CaliBrain bb848f05bc fix/bypass stall watchdog (#1186)
- Fix protection bypass cancelled by stall detection at exactly 300s
- make fixes
- Try to fix Synology DELETE issues
2026-08-11 12:23:34 -04:00
CaliBrain cc1a95f965 Fix protection bypass cancelled by stall detection at exactly 300s (#1184)
A download that hits Cloudflare hung on "Bypassing protection..." for
five
minutes and then died, regardless of which bypasser was configured.

html_get_page() started a BypassHeartbeat thread to keep the download
marked
alive during a bypass, but the thread had no loop: it fired one status
event
and returned. Even with the loop restored it could not have worked,
because
update_download_status() dedupes identical (status, message) tuples and
returns before refreshing _last_activity, and the heartbeat re-sent the
byte-identical payload already emitted just above it.

So _last_activity was frozen for the whole bypass, while both bypassers
are
allowed to run longer than STALL_TIMEOUT (external FlareSolverr ~394s at
default settings, internal 420s per get() call). The watchdog always
won.
From a reporter's log: 403 at 07:04:33.390, cancelled at 07:09:33.987 -
exactly 300.000s, and 41s before the bypasser would have finished and
reported the real error, an HTTP 500 from FlareSolverr the user never
saw.

The regression is not one commit. 1f093de (#536) added the heartbeat and
the
dedup together and refreshed activity before the dedup return, so it
worked.
ff094be (#832) moved the refresh below that return while tightening
stall
detection for #823. 3a3a3ce (#845) then deleted the heartbeat's while
loop
to silence a B023 lint, removing the last evidence of intent.

The dedup itself is correct and stays: a keep-alive that ticks on a
timer
proves nothing about whether an operation is progressing, so letting it
refresh the stall clock would make a wedged download immortal. Split the
two
concerns instead.

Add shelfmark/download/activity.py. A long single-shot operation
declares its
own upper bound once, over a sentinel status carried on the existing
status_callback channel - so no new parameter has to be threaded through
every
handler, post-processor and output module. The orchestrator intercepts
the
sentinel in its per-task closure and records an absolute deadline in
_activity_grace, which stall detection honours alongside STALL_TIMEOUT.
The
grace never extends itself and is clamped to
_MAX_ACTIVITY_GRACE_SECONDS, so
an operation that overruns its own declared budget is still cancelled.

Each bypasser now reports max_duration_seconds() derived from its own
retry
and timeout settings, and http.py asks whichever is active, plus 30s of
slack
so the bypasser's own deadline expires first and the user sees its real
failure. On that path html_get_page() also emits
status_callback("error", ...)
rather than silently returning an empty page.

Three further fixes on the same code path:

- Extract the watchdog into _find_stalled_tasks() and
_cancel_stalled_task().
It was the only place holding _progress_lock across a call into
book_queue,
whose terminal-status hooks reach a sqlite write that gevent does not
patch,
blocking the hub and every download worker. It now holds the lock for
dict
  reads only.
- Bound _CDP_WORKER.run(), which waited with timeout=None while holding
the
module-wide LOCKED, so a single wedged in-process CDP session blocked
every
  subsequent bypass forever on non-Docker installs.
- Broaden the coordinator loop's except clause back to Exception, with
  escalating backoff. 8d98e12 (#868) narrowed it to a six-type tuple to
silence BLE001, which let gevent's LoopExit and similar kill the only
thread
driving the download queue - undoing #832's fix for #823 and resurfacing
it
  as #1166. GreenletExit and gevent.Timeout still propagate.

Fixes #1001
Refs #1166, #823
2026-08-11 02:25:28 -04:00
816a735cde Add a {Language} naming template variable, and consolidate language resolution (#1142)
Fixes #1138
Fixes #1141

## Problem

Two language editions of one book resolve to the same canonical title,
so they render to the same path and the second gets a `_1` collision
suffix. Audiobookshelf treats a folder as exactly one library item, so
the pair becomes a single book with both files as tracks and a summed
runtime.

Shelfmark already parses and displays the language. It just never
reached the template engine.

## `{Language}` template variable

A template like `{Author}/{Title}{ (Language)}/{Author} - {Title}` now
yields:

```
/library/J K Rowling/Harry Potter (sv)/J K Rowling - Harry Potter.m4b
/library/J K Rowling/Harry Potter/J K Rowling - Harry Potter.m4b
```

The untagged edition's path is byte-identical to today, so no existing
layout shifts.

Three details worth flagging:

**The value is casefolded.** On a case-insensitive filesystem `(SV)` and
`(sv)` would collapse back into one folder, reintroducing the exact
collision being fixed.

**Values meaning "we don't know" render nothing** rather than producing
`Project Hail Mary (unknown)` folders. Anna's Archive reports that
string literally (`direct_download.py`, `language = detected or
"unknown"`).

**The frontend wasn't sending the release language at all**, so the
token would have stayed empty for exactly the audiobook sources in the
report. Prowlarr and AudiobookBay do not put language in `extra` the way
`direct_download` does, hence the payload plumbing. It reads
`release.language`, never `book.language` — the latter is the provider's
canonical edition and would mislabel a translation, with a regression
test for that specifically.

Not gated to audiobooks: Calibre-Web-Automated stages ingested files by
basename and discards folder structure, so the rename (filename)
template is the only lever those users have. Verified that form works:
`J K Rowling - Harry Potter (sv).epub`.

## Language consolidation (#1141)

Three release sources each carried their own alias map, all resolving to
the same ISO 639-1 codes, alongside a bundled database that only one of
them used. Adding a language meant editing three places.

Aliases now live in `data/book-languages.json` beside the code and name
they belong to, and `shelfmark/core/languages.py` resolves any of them —
two-letter code, ISO 639-2 three-letter in either the bibliographic or
terminological form, or English name. Prowlarr and AudiobookBay drop
their tables. Direct Download keeps its own path-parsing heuristics,
including the ambiguous short codes that collide with English words
(`de`, `en`, `no`, `in`), and takes only the alias data.

This also closes a coverage gap. MyAnonamouse offers 62 languages;
Prowlarr mapped 37, and an unmapped code is *dropped* rather than passed
through, so the other 25 carried no language at all — leaving
`{Language}` empty and the collision unfixed for Latin, Farsi, Tamil,
Urdu and the rest. Seven languages MAM offers had no database entry at
all: Bosnian, Burmese, Estonian, Icelandic, Manx, Scottish Gaelic,
Sanskrit.

Also fixes the Traditional Chinese code, which used a U+2011
non-breaking hyphen. Nothing compares against the ASCII spelling today
so it was latent, but it would silently defeat the first thing that did.

## Validation

Verified end to end against a live Prowlarr and MyAnonamouse, not just
unit tests. A real search returning both an English and a Swedish
edition, through the actual `queue_release` → `DownloadTask` → naming
path:

```
STEP 1  real MAM search        -> 37 releases, languages: ['en', 'sv']
STEP 3  queue_release          -> task.language='sv'
STEP 4  build_metadata_dict    -> metadata['Language']='sv'
STEP 5  build_library_path     -> /library/J K Rowling/Harry Potter (sv)/...
two language editions resolve to DIFFERENT folders: True
```

The refactor is pinned by a snapshot of both per-source maps taken
*before* they were deleted. All 131 aliases are asserted to still
resolve to the same code, one parametrised test each, so a regression
names the specific alias.

Also verified: the filename-only template, the retry round-trip
(`serialize_task_for_retry` → `_restore_task_from_retry_payload`, plus a
legacy payload with no `language` key), and placeholder handling.

Added a `KNOWN_TOKENS` ordering invariant test — `find_placeholder()`
does a substring `.find()` in list order and nothing protected that
contract, so a future token in the wrong position could silently shadow
an existing one. And a lockstep guard on the frontend, since
`KNOWN_TOKENS` is hand-duplicated in TypeScript.

**One caveat worth stating.** Three MAM codes are confirmed by
observation (`ENG`→`en`, `SWE`→`sv`, `MAL`→`ml`, the last from a real
`[MAL / EPUB]` Tagore release). The remaining ~59 are derived from ISO
639-2 rather than observed, because MAM's catalogue is overwhelmingly
English — enabling 27 extra languages still yielded only one non-English
hit across 258 results. Mitigated rather than closed: both 639-2
variants are present for every language where they differ, and a wrong
alias is an unused entry while a missing one loses the language. Happy
to correct any code a maintainer knows differs.

## Test results

2056 Python tests pass (up from 1906). Frontend typecheck, lint, format
and 126 unit tests pass.

Pre-existing failures on my machine, unchanged by this branch and
unrelated: `tests/bypass/` needs `seleniumbase`, and
`tests/config/test_entrypoint_permissions.py` uses bash-4 syntax that
macOS bash 3.2 rejects.

---------

Co-authored-by: delize <4028612+delize@users.noreply.github.com>
Co-authored-by: CaliBrain <calibrain@l4n.xyz>
2026-07-28 15:19:59 -04:00
Aidan AbbottandAidan Abbott f8d3f990ca fix: Prowlarr expired download link refresh (#1109)
## Summary

Fixes Prowlarr downloads that fail after a queued torrent result’s
tracker download link expires.

Prowlarr torrent results can expose a `downloadUrl` that is only a
short-lived proxy to the upstream tracker. Some trackers, including MAM,
include expiring credentials in that URL. Shelfmark was persisting that
URL as retry data and later treating it as durable. If the in-memory
Prowlarr cache was gone, or if qBittorrent tried to add a stale URL,
Shelfmark could fail with a misleading torrent-hash error instead of
refreshing the release.

## What changed

- Stop persisting Prowlarr `downloadUrl` values as durable retry data.
- Persist only source context needed to refresh the release later.
- On a Prowlarr cache miss, re-query Prowlarr using the queued task
context.
- Accept refreshed results only when the stable identity matches the
original release:
  - `guid == task.task_id`
  - or `infoUrl == task.task_id`
- Cache the fresh raw Prowlarr result and build the download request
from its current `downloadUrl` / `magnetUrl`.
- If qBittorrent add fails with `Could not determine torrent hash from
URL`, remove the stale cached Prowlarr result, refresh once, and retry
with the fresh URL.
- Preserve existing magnet handling: torrent results continue to use
`magnetUrl` first, so the refresh path only targets Prowlarr proxy
`downloadUrl` values that can expire.
- Improve the user-facing failure when refresh cannot find the same
release:
`The indexer download link expired and the release could not be
refreshed. Search again for a fresh result.`

## Why this approach

The important constraint is avoiding accidental downloads of a different
edition or format after the original tracker link expires. Re-running a
search by title can return many plausible results, so the refresh path
deliberately requires an exact stable identity match before using any
new URL.

This treats Prowlarr `downloadUrl` as a short-lived hint, while still
allowing retries to recover when Prowlarr can find the same release
again. The one-shot retry after qBittorrent add failure handles the case
where Shelfmark still has a cached Prowlarr result, but that cached
result contains an expired proxy URL.

The refresh hook is source-specific and defaults to no-op for other
external download handlers, so Newznab and other sources keep their
existing retry behavior.

## Bug

Fixes #1012

Context:
https://github.com/calibrain/shelfmark/issues/1012#issuecomment-4917148398

## Validation

- `uv run pytest tests/prowlarr -q`
- `uv run pytest tests/download/test_orchestrator_user_output_mode.py
-q`
- `uv run pytest tests/newznab/test_handler.py
tests/audiobookbay/test_handler.py -q`
- `uv run ruff check shelfmark/core/models.py
shelfmark/download/orchestrator.py
shelfmark/download/clients/base_handler.py
shelfmark/release_sources/prowlarr/handler.py
tests/prowlarr/test_handler.py
tests/prowlarr/test_integration_handler.py
tests/prowlarr/test_failure_scenarios.py tests/prowlarr/test_source.py
tests/download/test_orchestrator_user_output_mode.py`
- after resolving conflicts with latest `main`: `uv run pytest
tests/prowlarr -q`
- after resolving conflicts with latest `main`: `uv run ruff check
shelfmark/release_sources/prowlarr/handler.py`
- after resolving conflicts with latest `main`: `uv run pytest
tests/prowlarr/test_handler.py tests/prowlarr/test_source.py -q`

Co-authored-by: Aidan Abbott <aidanabbott@Aidans-MacBook-Pro.local>
2026-07-14 14:22:29 -04:00
CaliBrain 1c19326bd0 e2e testing 2 (#1083) 2026-06-23 16:58:24 -04:00