`get_absolute_url()` replaced both `netloc` and `scheme` whenever either
one was missing. A protocol-relative href such as
`//cdn.example.org/f.epub`, scraped from a page on
`https://annas-archive.org/...`, parses with a netloc and an empty
scheme, so it came back pointing at the page's own host. The download
then 404s and the source is skipped.
Each field now falls back to the base URL only when the parsed URL does
not supply it. Plain relative paths resolve exactly as before, which the
control test covers.
This affects the Z-Library, welib and generic download link handling in
`release_sources/direct_download/annas_archive.py`.
## Verification
- New `tests/download/test_http_absolute_url.py`: a protocol-relative
link keeps its own host, and a plain `/path` still resolves against the
base. The first fails on current main and passes here.
- Full suite (3165), ruff, ruff format, basedpyright, vulture green.
When doing a direct search for a book, show the stats of the AA results.
For example if we search for "The Great Gatsby", AA reports it has 240
hits and shows the first page of 50. Provide this stats info in the
shelfmark webUI via ResultsSection.tsx and ReleaseModal.tsx. This table
shows what should be displayed based on the total number of hits found:
|Total |Display|
|-------|-------|
|1 |"Result 1 (1 Total)"|
|6 |"Results 1-6 (6 Total)"|
|144 |"Results 1-50 (144 Total)"|
|500+ |"Results 1-50 (500+ Total)"|
Currently, shelfmark also only shows us the first 50 search hits even if
there were more available from AA. This could be added later if
considered desirable.
As usual, a picture is worth a 1000 words:
<img width="1012" height="610" alt="direct-results-stat"
src="https://github.com/user-attachments/assets/8c0c688f-444e-482a-a9a3-2dee5a5563b8"
/>
<img width="1013" height="741" alt="universal-results-info"
src="https://github.com/user-attachments/assets/399bbb79-87f2-4432-a3d0-64937795f5f1"
/>
Coded with llama.cpp, opencode and 🤖
Add **TorBox** as a torrent download client for Prowlarr releases.
Users can select `TorBox` in the download client settings, configure it
with the new `TORBOX_API_KEY` environment variable, and verify their
credentials with the connection test button.
The integration supports both magnet links and `.torrent` files. It
tracks the torrent lifecycle through TorBox, downloads supported book
and audiobook files from the TorBox CDN, preserves safe nested file
paths, and cleans up remote and local download state.
Important: Shared HTTP download logs omit full download URLs and
URL-bearing exception text to avoid exposing credentials, following best
practices. This applies to all clients that use the shared
`download_url()` path; URLs remain available to the HTTP operations
themselves.
---
There is already related work in progress in #1173, which includes both
torrent and direct-download support for TorBox.
This PR is not intended to replace or compete with that contribution. It
offers the tested torrent client functionality as a smaller, focused
change that can make TorBox available to the community sooner. The
direct-download integration proposed in #1173 remains valuable and could
be reviewed or introduced separately.
Automated tests cover configuration, connection validation, magnet and
torrent-file submission, API errors, status and progress handling, file
retrieval, path traversal protection, cancellation, cleanup, and
sensitive URL redaction.
I also validated the complete flow locally with several magnet links and
`.torrent` downloads. TorBox processed the torrents and Shelfmark
downloaded the resulting files as expected.
AI was used to help with the implementation, with human validation. This
PR and long description? Took me some good minutes at night after work,
but gives me joy to open this PR to share with the community this
improvement.
DownloadHistoryService.record_download and updated the single production
caller, but not the eleven in the test suite, leaving main red with 32
failures. Pass None, which is what the pre-#1336 behaviour recorded.
For the Blackhole handoff (#1345): add_download publishes the torrent
before the cancel check runs, and BlackholeClient.remove() is a no-op,
so
the watcher picks the file up regardless. Reporting a bare "Cancelled"
hid
that from the user. Name the completed handoff in the cancellation
message
instead, drop the _handle_cancelled_download call whose usenet branch
cannot apply to a handoff-only client, and record why the orchestrator
no
longer verifies HandoffResult.path.
Finally, make tests/direct_download a package: test_libgen_extract.py
imports tests.libgen.sample_html across test directories, so without an
__init__.py pytest named its modules by bare basename and a same-named
module elsewhere would collide.
A Blackhole watcher can consume the torrent before Shelfmark checks it,
leaving the task in error even though the handoff succeeded. Complete
the handoff when `add_download` successfully publishes the file, and
stop requiring a `HandoffResult` path to remain present.
Follow-up to #1312.
## Verification
- A watcher that immediately reads and removes the torrent receives the
exact bytes. The task changes from ERROR before this fix to COMPLETE
afterward, without running book postprocessing.
- The consumed-file regression fails on current main and passes here.
Resident files, write failures, cancellation, magnet rejection and
normal downloads remain covered: 81 focused tests pass.
- Ruff lint and formatting pass for the changed files.
The default audiobook formats include `zip` and `rar`.
`scan_directory_tree` checks the supported-format list before checking
for archives, so a downloaded archive lands in `book_files` and is
imported as-is. The extraction branch in `collect_directory_files` is
never reached.
This keeps archives out of `book_files`, so they always take the archive
path: extracted when extraction is allowed, imported as-is when it isn't
(unchanged).
Tests added in `tests/download/test_postprocess_scan_archives.py`; three
of the four fail without the change.
## Why
Blackhole users need Shelfmark to hand a torrent file to their existing
downloader instead of importing the downloaded book itself.
## Change
- add Blackhole as a torrent client with a configurable watched
directory
- prefer a fetched `.torrent` file for Blackhole while preserving magnet
preference for other clients
- complete the queue task after the handoff without invoking book
post-processing
## Verification
- `uv run pytest -q tests/prowlarr/test_blackhole_client.py
tests/prowlarr/test_handler.py tests/newznab/test_handler.py
tests/download/test_orchestrator_lifecycle.py`
- `uv run basedpyright shelfmark/download/clients/blackhole.py
shelfmark/download/clients/__init__.py
shelfmark/download/clients/base_handler.py
shelfmark/download/clients/settings.py
shelfmark/download/orchestrator.py shelfmark/release_sources/__init__.py
shelfmark/release_sources/prowlarr/utils.py
shelfmark/release_sources/prowlarr/handler.py
shelfmark/release_sources/newznab/handler.py
tests/prowlarr/test_blackhole_client.py tests/prowlarr/test_handler.py
tests/newznab/test_handler.py
tests/download/test_orchestrator_lifecycle.py`
Fixes#1229
### Summary
Fixes#1176
When downloading an audiobook with many chaptered tracks (e.g. 250+
`.flac` or `.mp3` files), indexer XML or release metadata often caps the
file list at ~100-110 entries. When the release extracts on disk,
`match_plan_to_files()` matched those first ~110 files to the planned
book group, while the remaining 140+ files fell into `unmatched` and
triggered fallback heuristic grouping.
Because heuristic grouping parsed the folder name (`Westwell - Hot &
Cold (2023)`) and stripped the series/author prefix, it generated a
second book titled `Hot & Cold` containing the remaining tracks,
resulting in two split book folders.
### Changes
- In `match_plan_to_files()`
(`shelfmark/download/postprocess/packs.py`), check `unmatched` files
before falling back to heuristic multi-book splitting.
- If an unmatched file is chaptered audio (`.flac`, `.mp3`, `.aac`,
etc.) and shares the directory with an existing book group, or if the
plan was a single-book plan, append it to that group instead of creating
a secondary book.
- Non-chaptered standalone books (e.g. `.m4b`, `.epub`) or files in
separate subfolders continue to fall back to heuristic grouping as
before.
- Added unit tests in `tests/download/test_packs.py` verifying:
1. Truncated track list in single folder properly appends remaining
chaptered tracks without splitting.
2. Single-book plan with multi-disc audio files (`CD1`/`CD2`) groups
together cleanly.
3. Multi-book packs with unmatched chaptered tracks route each track to
its respective book folder.
### Testing
Ran `uv run pytest tests/download/test_packs.py` and `uv run pytest
tests/core/test_processing_packs.py` (all passed cleanly). Checked type
annotations with `basedpyright` (0 errors) and formatting with `ruff`.
Co-authored-by: amasen02 <amasen02@users.noreply.github.com>
html_get_page follows Anna's Archive redirects by hand, and DDoS-Guard's
gate
answers /search with a 302 to the same path plus `check=1`. The follower
walks
that handshake by reassigning `current_url`, so every downstream handoff
- the
403 branch, the 503-challenge branch, both redirect-loop rescues -
passed the
*probe* URL to the bypasser rather than the page we actually wanted.
A solver opens that in a fresh browser holding none of the cookies the
probe
exists to collect, so DDoS-Guard cannot verify it automatically and
serves the
manual CAPTCHA page that nothing can solve. The #1292 log is exactly
that: a 403
handed off on `&check=1`, FlareSolverr answering "Challenge solved!",
and a
4721-byte DDOS-GUARD captcha page coming back.
- `_solvable_url()` strips the probe parameter, applied at the single
choke point
in `_run_bypasser` so all four handoffs are covered. Scoped to the hosts
whose
redirects we follow manually; a URL without the parameter is returned by
identity, so nothing else is re-encoded.
The same reports showed three further defects, all of which stand
whatever the
host was reacting to:
- The external bypasser logged that the solve had not cleared the
protection and
then returned the challenge page as a success. That skipped the one
recovery
left - get_bypassed_page's retry-and-rotate loop, where the next mirror
is a
different DDoS-Guard host - and filed the captcha page's own __ddg
cookies as
that host's clearance, to be replayed on every later request. It now
raises
ChallengeNotSolvedError before storing anything.
- "Check that the bypasser is reachable and working" was the one piece
of advice
guaranteed to waste the reporter's time: it was reachable, it ran a full
solve,
and it returned a captcha. ChallengeNotSolvedError carries the marker so
the
search layer can name the host as the cause instead of the bypasser.
- The untabled-page fingerprint logged `attempt_url`, which
html_get_page has
since rotated past. The #1298 bundle reported the page against
annas-archive.gl
when the body had come from .pk - the triage cost #1289 added the line
to
remove. The search now asks for the response URL and logs that. Its
give-up
shape is the tuple ("", url), which is truthy, so the exhaustion check
reads
the body rather than the response.
Regression fixtures are built from the pages in the reports. The two
behavioural
handoff tests were checked against the unfixed code: both fail there,
reproducing
the reporter's log line verbatim.
Refs #1292
Refs #1298
Follow-ups to the multi-book pack feature (#1270), which inspects every
release before download. Two behaviours leaked into the ordinary
single-book
flow and are corrected here:
- A flat folder of chaptered audio (`01 - Chapter.mp3`, `02 - ...`) was
detected as a pack, because each track name parses to a series position,
so
clicking download popped the review panel for one normal audiobook. Flat
folders are now split one-book-per-file only with real evidence of
distinct
books: two or more series positions, more than one title, and no
chaptered
audio (only the single-file m4b/m4a containers and ebook formats
qualify).
Subfolder packs and flat m4b/m4a packs are unchanged.
- Every release that couldn't be inspected (usenet, magnet-only, sources
without a list_files hook, ABB single-file) showed an info toast on
download. That is now a console.warn, so a normal download is silent
again.
Adds regression tests for the chaptered-mp3 cases.
## Multi-book packs: inspect a release before download and file each
book separately
Closes#576
### Problem
One queued release is always treated as one book. When a torrent is
actually a whole series
(`Series/Book 1 - Title/…`, or a flat folder of `Series 1.0 - Title.m4b`
files), post-processing
walks the whole tree, flattens every file into one list and renames them
`Title - 01…10` under the
searched book's `{Author}/{Title}`. Audiobookshelf then sees a single
10-file "book" and the user
has to re-file everything by hand.
### What this does
Most releases expose their file list *before* anything is downloaded, so
the split is decided up
front and approved by the user, then the download is fire-and-forget:
1. **Inspect** – clicking a release's download button now calls `POST
/api/releases/inspect`
first. A new optional `DownloadHandler.list_files(release_data)` hook
returns the release's
files without downloading:
- **AudiobookBay** reads the torrent file table off the detail page it
already fetches (the
page is now cached for 120 s, so inspect + download cost ABB one
request).
- **Prowlarr** parses `info.files` from the `.torrent` it already
fetches (the existing 120 s
torrent-fetch cache is reused). Magnet-only and usenet releases report
"can't inspect".
- Other sources default to `None`.
2. **Review** – if the plan contains more than one book, the Find
Releases modal swaps the list
for a review panel: one row per book with editable title / series
position / year, expandable
file lists, non-book sidecars (`.txt`, covers) shown as ignored, a
"Treat as a single book"
switch, and **Download N books**. Single-book releases queue
immediately, exactly as before.
3. **File** – the approved plan travels with the task
(`DownloadTask.book_plan`, retry-safe) and
post-processing files each book through the existing transfer code, one
book at a time
(`dataclasses.replace(task, title=…, series_position=…, year=…)`), so
organize/rename
templates, part numbering (now scoped per book), hardlinks, torrent
copy-preserve and usenet
handling are unchanged. Status reads `Complete (N books, M files)`.
4. **Fallback** – when a release can't be inspected the user gets a
toast, and a small
"Multi-book pack" toggle in the modal header forces a heuristic split
(subfolder = book, or
one book per file when the file names carry series positions).
Planning lives in `shelfmark/download/postprocess/packs.py` and is
shared by the inspect endpoint
and post-processing, so what the user approved is what gets filed. The
name parser strips
`Book 3 -`, `03 -`, `1.0 -`, `3.`, `[03]`, `#3`, a leading series name,
labels like
"An Expanse Novella -", repeated titles (`Gods of Risk 2.5 - Gods of
Risk`) and a trailing
`(Year)`; author and series name come from the book that was searched,
and the searched book's
own series position is never applied to its siblings.
### Files
- `shelfmark/download/postprocess/packs.py` (new) –
`PackFile/PackBook/PackPlan`, `plan_pack`,
`parse_pack_book_name`, `group_files_into_books`, `match_plan_to_files`
- `shelfmark/core/release_inspect_routes.py` (new) – `POST
/api/releases/inspect`
- `shelfmark/release_sources/__init__.py` – `DownloadHandler.list_files`
hook
- `shelfmark/release_sources/audiobookbay/{scraper,handler}.py` –
detail-page cache,
`extract_file_list`, `list_files`
- `shelfmark/release_sources/prowlarr/handler.py`,
`download/clients/torrent_utils.py` –
`extract_file_list_from_torrent`, `list_files`
- `shelfmark/core/models.py`, `download/orchestrator.py` – `multi_book`
/ `book_plan` fields,
queue + retry serialization
- `shelfmark/download/postprocess/transfer.py`, `pipeline.py`,
`outputs/folder.py` – per-book
transfer branch and status message
- `src/frontend`: `components/PackReviewPanel.tsx` (new),
`ReleaseModal.tsx`, `App.tsx`,
`services/api.ts`, `types/index.ts`, `utils/releasePayload.ts` (payload
builder moved out of
`App.tsx`), `utils/packReview.ts`
- `docs/dev/release-sources-plugin-guide.md` – documents the
`list_files` hook
### Out of scope (follow-ups)
- Listing files from an NZB (Shelfmark already fetches the bytes; `<file
subject>` names are noisy)
- Inspecting magnet links via qBittorrent's files API after a paused add
- BookLore / email outputs (they ignore `book_plan`; noted in code)
- The combined ebook + audiobook flow
### Testing
**Automated** (`make checks`, `make python-test`, `make frontend-test`
all green; the only
failures on my machine are the pre-existing
`tests/config/test_entrypoint_permissions.py` cases,
which need bash ≥ 4 and fail identically on `main` under macOS bash
3.2):
- `tests/download/test_packs.py` – name parsing (markers, series name,
novella labels, repeated
titles, bare numeric titles like `1984`), nested / flat / mixed /
deeper-nested packs, single
wrapping folder not treated as a pack, plan-to-disk matching with
basename fallback
- `tests/core/test_processing_packs.py` – full `post_process_download`
runs on a real temp
filesystem: approved plan files each book under its own
`{Author}/{Title}`, heuristic split
of a nested pack, searched book's series position does not leak,
multi-file book inside a pack
keeps `- 01/- 02` per book, hardlinked torrent pack leaves the seeding
tree intact, no pack
fields ⇒ behaviour unchanged, single group degrades to the searched
title, status message
- `tests/core/test_release_inspect_routes.py` – plan response,
not-inspectable, handler errors
never 500, unknown source / missing `source_id` ⇒ 400, login required
- `tests/audiobookbay/test_file_list.py` – file-table scraping from real
ABB markup (multi-file
and single-file pages), handler host validation, one page fetch shared
by magnet + file list
- `tests/prowlarr/test_torrent_file_list.py` – multi-file / single-file
`.torrent` parsing,
handler behaviour for torrent URL vs magnet vs usenet vs cache miss
- `tests/download/test_orchestrator_pack_fields.py` – queue-time parsing
and retry round-trip
- Frontend: `releasePayload.test.ts`, `packReview.test.ts` (vitest)
**Manual, on a real deployment** (arm64 image built from this branch,
run as a side container
next to production with the same qBittorrent / Audiobookshelf setup,
`FILE_ORGANIZATION_AUDIOBOOK=organize`,
hardlinks on):
- AudiobookBay "The Expanse Complete 2.0" (7.87 GB, 36 files): clicking
download opened the review
panel in ~1 s showing **18 books · 18 files · 18 files ignored** (the
`.txt` sidecars), with
series positions 0.1–9.5 and years parsed from the file names; novella
labels stripped
("The Churn", "The Butcher of Anderson Station"). Editing a title in the
panel works.
Confirming queued one task; the magnet resolved from the cached page in
~30 ms; after the
download the task reported `Complete (18 books, 18 files)`, 18 hardlinks
landed as
`audiobooks/James S. A. Corey/<Title>/<Title>.m4b`, the torrent kept
seeding, and
Audiobookshelf scanned each folder as its own book (title, author,
embedded chapters).
- A second pack ("Expanse [01 - 9.5]", `Title N - Title` naming) was
inspected to verify the
repeated-title rule and the Back button, without downloading.
- Single-book releases still queue immediately with no extra UI.
Anna's Archive 429-throttles the source IP after repeated automated
requests.
The bypasser could clear the DDoS-Guard challenge but not the 429, so
each retry
re-solved, re-spawned Chrome, and rotated mirrors that share the same IP
- a
costly loop that never converged.
Add a process-global, per-host cooldown that escalates 2 -> 5 -> 10 ->
15 -> 30
minutes each time a host 429s again after its window elapsed, resetting
after a
long clear gap. Mirror selection skips cooling hosts and the bypasser
refuses to
solve one, so a throttled host fails fast instead of storming the
solver.
503 is in RETRYABLE_CODES, and the bypasser is only ever reached from
the 403
branch and the AA redirect-loop rescues. Once Z-Library re-serves its
DDoS-Guard
interstitial with the same cookie the #1188 handshake already echoed
back, the
request has nothing left to try and spends every attempt on the same
wall.
Gate the handoff on the response body rather than the status, so a
genuine
overloaded-origin 503 keeps its retry path, and on
allow_bypasser_fallback, so
best-effort fetches still fail fast. The challenge indicators move out
of
internal_bypasser into shelfmark/bypass/challenge.py so http.py can use
them
without importing SeleniumBase, which is lazily imported precisely
because it
is optional.
Refs #1233
Direct download was unusable behind an external bypasser (FlareSolverr /
Byparr): every request paid a 403 plus a full solve, and a search that
never ran was reported to the user as "No books found".
Clearance was discarded on the external path. get_cf_cookies_for_domain
and get_cf_user_agent_for_domain returned {} / None whenever
USING_EXTERNAL_BYPASSER was set, and _fetch_via_bypasser read only
solution.response - dropping solution.cookies and solution.userAgent,
which FlareSolverr-compatible services do return. A solve therefore
cleared the one request that paid for it and nothing else, and a file
download - which the solver cannot proxy, being binary - presented no
clearance at all. Diagnosed from a v1.3.9 debug bundle: ~35s in the
bypasser per search, on every search.
- Move the cookie jar out of internal_bypasser into bypass/cookie_store.
internal_bypasser imports seleniumbase at module scope, which is the
dependency an external-bypasser deployment is entitled not to have, so
it cannot host a store the external path depends on.
- Harvest solution.cookies and solution.userAgent after a successful
solve. The existing filtering applies unchanged, so the per-check
__ddg8_/__ddg9_/__ddg10_ trio is still dropped and the external path
cannot reintroduce the ?check=1 loop fixed in ebb833a. The UA matters
as much as the cookies: Cloudflare ties cf_clearance to the UA that
solved the challenge.
- Read cookie fields from either shape - CDP objects or JSON mappings.
Both use the same field names, expires included.
- Point http.py's getters and _purge_clearance at the shared store, so
either bypasser fills and drains the same jar.
- Give the Docker helper-subprocess handoff explicit export_store /
import_store rather than reaching into module globals.
An unsolved challenge was also indistinguishable from an empty result.
_looks_like_aa_page() counted the challenge markers as "recognisably
AA", so _fetch_search_table handed a DDoS-Guard interstitial back as a
legitimate no-table response and the user was told their query found
nothing when the search never ran. Split challenge detection out and
raise SearchUnavailableError with the reason instead. The mirror is
still not quarantined - every mirror shares the same protection, so it
is not the mirror's fault.
Verified: 2531 unit tests pass; ruff, basedpyright and vulture clean;
e2e bypasser-external profile passes (5). Its mock FlareSolverr already
returned cookies and userAgent from /v1 - the contract was there,
shelfmark was not reading it.
Refs #1220. Deliberately not "Fixes": this removes the re-solve and
makes a failed solve legible, but if Byparr genuinely cannot clear AA's
current DDoS-Guard, the reporter now gets that as an error rather than a
silent "no books found". The download path may swallow interstitials the
same way; not audited here.
A cookie that has been rejected was kept and presented again on every
later
request, so a single bad clearance could re-arm the challenge
indefinitely.
Cookie storage:
- Enforce expiry for every stored cookie, not just cf_clearance.
DDoS-Guard
domains have no cf_clearance, so the existing check never fired for them
and
expired cookies were replayed forever.
- Stop storing the per-check cookies __ddg8_/__ddg9_/__ddg10_ and
ddg_last_challenge. Captured live from Anna's Archive, these carry the
client
IP and the timestamp the check was issued (~40 min), versus ~1 year for
the
__ddg1_/__ddg2_/__ddgid_ clearance. Replaying an IP-bound token stops
describing the caller as soon as the egress IP changes, which is routine
behind a VPN.
Failure handling — every path that is rejected while carrying cookies
now
purges them, not just the redirect loop:
- 403 returned while presenting cookies.
- Cached-cookie attempt rejected, whether by status or by redirect loop.
- Factored the purge into _purge_clearance, guarded on a non-empty
hostname
since clear_cf_cookies("") means "every host" and would wipe clearance
for
sites that are working fine.
Also fix the search warm-up switches shipped inert in v1.3.8:
SEARCH_WARMUP_ENABLED and SEARCH_WARMUP_QUERY are not in the settings
registry, and config.get only consults the environment for keys it
knows, so
both always returned their defaults — the warm-up could not be turned
off or
retargeted. Read os.environ first.
Refs #1220. Deliberately not "Fixes": the reported failure could not be
reproduced on v1.3.8 from a stable IP (the reporter's own queries all
returned
200 on both the pre- and post-change builds), and the new purge paths
did not
fire in live testing because the failures arrive as redirect loops,
which were
already purged. These are correctness fixes with no measured effect on
that
issue. The underlying problem remains that Chrome-obtained cookies never
satisfy DDoS-Guard when replayed by requests, so every search still
re-solves.
Verified: 2542 unit tests pass; ruff, basedpyright and vulture clean;
e2e
platform baseline (10), full (6) and bypasser-external (5) all pass;
five
sequential live searches against Anna's Archive all returned 200 with
zero
"Exceeded 30 redirects".
- Add RFC 8484 DNS wireformat codec and HTTP/2 support (httpx) for
Quad9/OpenDNS DoH providers.
- Quarantine dead, parked, or seized mirrors for the session on hard
failure (DNS errors, connection refused, 410/451, parked pages) while
preserving bypass clearance on live mirrors.
- Add background startup search warmup to prime DNS, elect mirrors, and
pre-solve protection challenges to eliminate cold-start search latency.
- Add comprehensive test suites for DoH wireformat, mirror quarantine,
parked domain detection, and search warmup.
## Summary
Two failure modes on the same code path, both reported this week: Anna's
Archive `/search` is gated behind a DDoS-Guard cookie probe that the
manual redirect follower can never satisfy.
**#1202 — the cookie is dropped on every hop.** AA URLs set
`allow_redirects = False`, so `html_get_page` follows redirects by hand.
The 302 to `?check=1` carries a `Set-Cookie` (`__ddg*`) that has to come
back on the next request. Because cookies are passed per call and
`requests` keeps no jar across manual hops, it was discarded each time
and the server just re-issued the same redirect until `_MAX_REDIRECTS`
raised `TooManyRedirects`. The file already had the right helper —
`_new_cookies()` — but only the 503 Z-Library handshake branch called
it.
**#1204 — the loop never reaches the bypasser.** `TooManyRedirects`
isn't in `_is_retryable_error` and carries no status code, so the 403
rescue path (`status == _HTTP_STATUS_FORBIDDEN`) never fired and all
attempts repeated the identical failure — ~2.5 min, surfacing as the
misleading "Network restricted or mirrors are blocked".
These interact, which is why #1202's fix alone isn't enough. Requests
merge as `cookies={**handshake_cookies, **cookies}`, so **stale bypasser
cookies override the fresh handshake ones** — once `_cf_cookies` holds
an expired `__ddg*`, the probe can never clear no matter how faithfully
we echo. Hence one search per restart, exactly as #1204 describes.
## Changes
1. Harvest cookies in the same-host redirect branch, the way the 503
branch already does. `_new_cookies()` returns only *new* values, so a
server re-sending an identical cookie yields an empty dict and a genuine
redirect loop still terminates at `_MAX_REDIRECTS`.
2. Treat a redirect loop as a detected challenge: purge the stored
cookies for that host and switch to the bypasser, instead of burning the
retry budget. Gated on `allow_bypasser_fallback` and
`_is_cf_bypass_enabled()`, and skipped when already bypassing, so
AudiobookBay (`allow_bypasser_fallback=False`) and external-bypasser
setups are unaffected.
The broader point in #1204 stands — the fallback would be better gated
on "challenge detected" than on specific status codes, since DDoS-Guard
presents at least three faces (403 js-challenge, 429, and this redirect
loop). This PR fixes the two live exits without that refactor.
## Tests
Two regression tests, both failing before and passing after:
- `test_html_get_page_echoes_cookies_across_same_host_redirects` — the
fake server only returns results if `__ddg2_` comes back on the
`?check=1` hop.
- `test_html_get_page_redirect_loop_purges_cookies_and_bypasses` —
asserts the stored cookies are cleared, the bypasser runs, and the loop
is cut short rather than repeated per attempt.
`ruff check` and `ruff format` clean. `tests/download/` passes except
`test_download_url_ignores_zlib_cookie_refresh_failure`, which fails
identically on unmodified `main` in my environment (no `seleniumbase` —
the `browser` extra isn't installed).
## Verification
Applied on a live v1.3.7 install (Debian LXC, internal CDP bypasser).
Before: every search timed out through 10 retries with
`TooManyRedirects`, zero results. After:
```
http.py:455 - Redirect loop detected; switching to bypasser
internal_bypasser.py:756 - Bypass successful using _bypass_method_cdp_gui_click
internal_bypasser.py:322 - Extracted 9 protection cookies for annas-archive.pk
direct_download.py:1865 - Found 24 releases via ISBN
```
~25 s per search, results render. Note the second search still re-solves
the challenge, since the freshly stored cookies go stale immediately —
the design issue #1204 raises, left for the broader fix.
Fixes#1202Fixes#1204🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012Ln3yVj3sWHG2c6T78W1we
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: CaliBrain <calibrain@l4n.xyz>
#1210 and #1212 both added a DDoS-Guard `?check=1` rescue, and #1212 was
branched before #1210 landed, so the merged result had two of them with
identical guards. #1212's inline handoff returns before the raise that
#1210's exception handler keys on, so the handler was shadowed and its
stale-cookie purge — the substance of #1210 — never ran. Its regression
test has been failing on main since the merge.
Fold both into one path:
- `_redirect_loop_handoff()` purges the host's stale clearance cookies,
then bypasses, so the inline AA handoff and the exception handler
cannot drift apart again.
- The exception handler keeps its own reason to exist: non-AA hosts run
with allow_redirects=True, so `requests` raises the loop itself and the
manual AA follower never sees it. It now invokes the bypasser directly
rather than setting a flag and continuing, which was a no-op at
MAX_RETRY=1 for the same reason the 403 handoff was.
- An unrescuable loop returns empty instead of raising TooManyRedirects
into the retry path. That error is not retryable and carries no status,
so `/dyn/md5/summary` (allow_bypasser_fallback=False) re-ran the full
6-redirect loop on all 10 attempts: 60 requests to AA and ~30s of
backoff, measured. Every AA mirror shares the challenge, so there is
nothing to rotate to.
- `allow_bypasser_fallback` docs now describe what the flag actually
gates; the old text predated #1198 and named the wrong callers.
Fixes#1204.
## Problem
#1198 sends a gated AA `/search` to the bypasser when the origin answers
403.
DDoS-Guard has a second response: when the clearance cookies from an
earlier
solve go stale, it serves an endless `?check=1` redirect instead.
`requests` follows that until `_raise_too_many_redirects`, and
`TooManyRedirects` carries no status code, so `status ==
_HTTP_STATUS_FORBIDDEN`
is false and the rescue never runs. All 10 retries re-send the same dead
cookies, then the search fails as `Unable to reach download source.
Network
restricted or mirrors are blocked.`
Direct-download search therefore works once per container start, and
stays dead
after the stored cookie ages out.
v1.3.7 (`sha256:520715f3…`), internal bypasser, mirrors `.gl/.pk/.gd`:
```
17:04:36 internal_bypasser.py:756 - Bypass successful using _bypass_method_cdp_gui_click
...
17:11:39 http.py:483 - Retry 1/10 for https://annas-archive.gl/search?...&check=1:
TooManyRedirects: Too many redirects
17:12:12 http.py:493 - Giving up after 10 attempts
17:12:12 main.py:2870 - Release search failed for source direct_download:
Unable to reach download source. Network restricted or mirrors are blocked.
```
The token is short-lived, which is what makes this reachable in normal
use:
```
$ curl -sD - 'https://annas-archive.gl/search?...&check=1'
HTTP/2 403
server: ddos-guard
set-cookie: __ddg8_=…; Expires=Fri, 14-Aug-2026 15:39:38 GMT # issued 15:19:38, 20 min
```
## Fix
Handle the loop like the 403: drop the domain's stored cookies, then
retry
through the bypasser. The branch sits above the `status ==` ladder
because
`_get_status_code()` returns `None` for this exception.
Cookies are purged only for the internal bypasser; with an external one
`get_cf_cookies_for_domain()` already returns `{}`.
Related but not changed here: `get_cf_cookies_for_domain()` enforces
expiry for
`cf_clearance` only, so `__ddg*` cookies are never evicted on age, which
is why
they go stale. This patch makes the rescue fire whatever the reason the
cookies
stopped working.
## Verification
The regression test drives a real redirect loop through `html_get_page`
(302 to `&check=1`, exception raised by the production path rather than
faked)
and asserts the cookies are purged and the bypasser runs once.
- `pytest tests/download/test_http_bypasser_fallbacks.py`: 8 passed.
`test_download_url_ignores_zlib_cookie_refresh_failure` fails in my
checkout
on a missing `seleniumbase`, unrelated to this change.
- `ruff check`, `ruff format --check`: clean.
- Running in production since 2026-08-14 on v1.3.7 with only this file
replaced:
six direct-download searches, five served, three books downloaded end to
end,
against one search per container start before. The rescue mid-download:
```
19:12:14 http.py:449 - Redirect loop detected; switching to bypasser:
https://annas-archive.gl/md5/cb8fba7abae800ddbae1adfb8d7699d9?&check=1
19:12:38 internal_bypasser.py:756 - Bypass successful using _bypass_method_cdp_gui_click
19:14:36 direct_download.py:1142 - Resolved download URL [aa-slow-nowait]: …
19:14:47 orchestrator.py:735 - download finished; starting post-processing
```
## Separate issue this exposes
DDoS-Guard does not accept a solved cookie from plain `requests`
traffic, so
after this patch the rescue runs for nearly every AA URL.
`internal_bypasser.get()`
serializes all solves on one module-wide lock and builds a fresh Chrome
each
time: 11-16 s uncontended, 43-52 s under concurrent load, measured on
the host
above. Correctness is cheap here, latency is not. Happy to open a
separate PR
for a warm browser session if that direction is welcome.
Co-authored-by: Kukkerem <Kukkerem@users.noreply.github.com>
Anna's Archive put a DDoS-Guard JS challenge in front of /search: the
homepage still returns 200, but /search and /md5/<id> answer 403 on
every mirror (.gl, .pk, .gd all confirmed). Search fetched both with
allow_bypasser_fallback=False, which rotates mirrors on a 403 instead of
invoking the bypasser, so it walked the whole mirror list, exhausted it,
and surfaced "Unable to reach download source. Network restricted or
mirrors are blocked." as a 503 on every query. Adding mirrors could not
help — they sit behind the same gate — and neither could USE_CF_BYPASS,
since search never reached that branch.
Fetch search and the detail page with allow_bypasser_fallback=True so a
403 hands over to the bypasser, which already detects this challenge
(DDOS_GUARD_INDICATORS matches the live page). Echoing the __ddg cookies
back does not clear it; it needs real JS execution. The download-count
fetch keeps allow_bypasser_fallback=False: it is decoration on the
details modal and not worth holding the modal open for a browser solve.
Fixes#1196
LOG_LEVEL never reached the app logger: env.py hardcoded the level to
DEBUG or INFO, so INFO lines kept appearing under LOG_LEVEL=error. Read
it from the env var and advanced settings, normalize unknown values to
INFO, and expose it as a setting. entrypoint.sh now normalizes
gunicorn's level too, so a typo falls back to info instead of stopping
the container from booting.
Z-Library gates the first hit on /md5/<hash> with a 503 whose only
payload is a Set-Cookie; echoing that cookie back returns the 302 to the
real page. html_get_page dropped it and re-ran the same rejected request
on every retry, ending in "No download URL resolved". Retry once with
the
cookies the 503 issued.
Fixes#1185Fixes#1187
A download that hits Cloudflare hung on "Bypassing protection..." for
five
minutes and then died, regardless of which bypasser was configured.
html_get_page() started a BypassHeartbeat thread to keep the download
marked
alive during a bypass, but the thread had no loop: it fired one status
event
and returned. Even with the loop restored it could not have worked,
because
update_download_status() dedupes identical (status, message) tuples and
returns before refreshing _last_activity, and the heartbeat re-sent the
byte-identical payload already emitted just above it.
So _last_activity was frozen for the whole bypass, while both bypassers
are
allowed to run longer than STALL_TIMEOUT (external FlareSolverr ~394s at
default settings, internal 420s per get() call). The watchdog always
won.
From a reporter's log: 403 at 07:04:33.390, cancelled at 07:09:33.987 -
exactly 300.000s, and 41s before the bypasser would have finished and
reported the real error, an HTTP 500 from FlareSolverr the user never
saw.
The regression is not one commit. 1f093de (#536) added the heartbeat and
the
dedup together and refreshed activity before the dedup return, so it
worked.
ff094be (#832) moved the refresh below that return while tightening
stall
detection for #823. 3a3a3ce (#845) then deleted the heartbeat's while
loop
to silence a B023 lint, removing the last evidence of intent.
The dedup itself is correct and stays: a keep-alive that ticks on a
timer
proves nothing about whether an operation is progressing, so letting it
refresh the stall clock would make a wedged download immortal. Split the
two
concerns instead.
Add shelfmark/download/activity.py. A long single-shot operation
declares its
own upper bound once, over a sentinel status carried on the existing
status_callback channel - so no new parameter has to be threaded through
every
handler, post-processor and output module. The orchestrator intercepts
the
sentinel in its per-task closure and records an absolute deadline in
_activity_grace, which stall detection honours alongside STALL_TIMEOUT.
The
grace never extends itself and is clamped to
_MAX_ACTIVITY_GRACE_SECONDS, so
an operation that overruns its own declared budget is still cancelled.
Each bypasser now reports max_duration_seconds() derived from its own
retry
and timeout settings, and http.py asks whichever is active, plus 30s of
slack
so the bypasser's own deadline expires first and the user sees its real
failure. On that path html_get_page() also emits
status_callback("error", ...)
rather than silently returning an empty page.
Three further fixes on the same code path:
- Extract the watchdog into _find_stalled_tasks() and
_cancel_stalled_task().
It was the only place holding _progress_lock across a call into
book_queue,
whose terminal-status hooks reach a sqlite write that gevent does not
patch,
blocking the hub and every download worker. It now holds the lock for
dict
reads only.
- Bound _CDP_WORKER.run(), which waited with timeout=None while holding
the
module-wide LOCKED, so a single wedged in-process CDP session blocked
every
subsequent bypass forever on non-Docker installs.
- Broaden the coordinator loop's except clause back to Exception, with
escalating backoff. 8d98e12 (#868) narrowed it to a six-type tuple to
silence BLE001, which let gevent's LoopExit and similar kill the only
thread
driving the download queue - undoing #832's fix for #823 and resurfacing
it
as #1166. GreenletExit and gevent.Timeout still propagate.
Fixes#1001
Refs #1166, #823
Fixes#1138Fixes#1141
## Problem
Two language editions of one book resolve to the same canonical title,
so they render to the same path and the second gets a `_1` collision
suffix. Audiobookshelf treats a folder as exactly one library item, so
the pair becomes a single book with both files as tracks and a summed
runtime.
Shelfmark already parses and displays the language. It just never
reached the template engine.
## `{Language}` template variable
A template like `{Author}/{Title}{ (Language)}/{Author} - {Title}` now
yields:
```
/library/J K Rowling/Harry Potter (sv)/J K Rowling - Harry Potter.m4b
/library/J K Rowling/Harry Potter/J K Rowling - Harry Potter.m4b
```
The untagged edition's path is byte-identical to today, so no existing
layout shifts.
Three details worth flagging:
**The value is casefolded.** On a case-insensitive filesystem `(SV)` and
`(sv)` would collapse back into one folder, reintroducing the exact
collision being fixed.
**Values meaning "we don't know" render nothing** rather than producing
`Project Hail Mary (unknown)` folders. Anna's Archive reports that
string literally (`direct_download.py`, `language = detected or
"unknown"`).
**The frontend wasn't sending the release language at all**, so the
token would have stayed empty for exactly the audiobook sources in the
report. Prowlarr and AudiobookBay do not put language in `extra` the way
`direct_download` does, hence the payload plumbing. It reads
`release.language`, never `book.language` — the latter is the provider's
canonical edition and would mislabel a translation, with a regression
test for that specifically.
Not gated to audiobooks: Calibre-Web-Automated stages ingested files by
basename and discards folder structure, so the rename (filename)
template is the only lever those users have. Verified that form works:
`J K Rowling - Harry Potter (sv).epub`.
## Language consolidation (#1141)
Three release sources each carried their own alias map, all resolving to
the same ISO 639-1 codes, alongside a bundled database that only one of
them used. Adding a language meant editing three places.
Aliases now live in `data/book-languages.json` beside the code and name
they belong to, and `shelfmark/core/languages.py` resolves any of them —
two-letter code, ISO 639-2 three-letter in either the bibliographic or
terminological form, or English name. Prowlarr and AudiobookBay drop
their tables. Direct Download keeps its own path-parsing heuristics,
including the ambiguous short codes that collide with English words
(`de`, `en`, `no`, `in`), and takes only the alias data.
This also closes a coverage gap. MyAnonamouse offers 62 languages;
Prowlarr mapped 37, and an unmapped code is *dropped* rather than passed
through, so the other 25 carried no language at all — leaving
`{Language}` empty and the collision unfixed for Latin, Farsi, Tamil,
Urdu and the rest. Seven languages MAM offers had no database entry at
all: Bosnian, Burmese, Estonian, Icelandic, Manx, Scottish Gaelic,
Sanskrit.
Also fixes the Traditional Chinese code, which used a U+2011
non-breaking hyphen. Nothing compares against the ASCII spelling today
so it was latent, but it would silently defeat the first thing that did.
## Validation
Verified end to end against a live Prowlarr and MyAnonamouse, not just
unit tests. A real search returning both an English and a Swedish
edition, through the actual `queue_release` → `DownloadTask` → naming
path:
```
STEP 1 real MAM search -> 37 releases, languages: ['en', 'sv']
STEP 3 queue_release -> task.language='sv'
STEP 4 build_metadata_dict -> metadata['Language']='sv'
STEP 5 build_library_path -> /library/J K Rowling/Harry Potter (sv)/...
two language editions resolve to DIFFERENT folders: True
```
The refactor is pinned by a snapshot of both per-source maps taken
*before* they were deleted. All 131 aliases are asserted to still
resolve to the same code, one parametrised test each, so a regression
names the specific alias.
Also verified: the filename-only template, the retry round-trip
(`serialize_task_for_retry` → `_restore_task_from_retry_payload`, plus a
legacy payload with no `language` key), and placeholder handling.
Added a `KNOWN_TOKENS` ordering invariant test — `find_placeholder()`
does a substring `.find()` in list order and nothing protected that
contract, so a future token in the wrong position could silently shadow
an existing one. And a lockstep guard on the frontend, since
`KNOWN_TOKENS` is hand-duplicated in TypeScript.
**One caveat worth stating.** Three MAM codes are confirmed by
observation (`ENG`→`en`, `SWE`→`sv`, `MAL`→`ml`, the last from a real
`[MAL / EPUB]` Tagore release). The remaining ~59 are derived from ISO
639-2 rather than observed, because MAM's catalogue is overwhelmingly
English — enabling 27 extra languages still yielded only one non-English
hit across 258 results. Mitigated rather than closed: both 639-2
variants are present for every language where they differ, and a wrong
alias is an unused entry while a missing one loses the language. Happy
to correct any code a maintainer knows differs.
## Test results
2056 Python tests pass (up from 1906). Frontend typecheck, lint, format
and 126 unit tests pass.
Pre-existing failures on my machine, unchanged by this branch and
unrelated: `tests/bypass/` needs `seleniumbase`, and
`tests/config/test_entrypoint_permissions.py` uses bash-4 syntax that
macOS bash 3.2 rejects.
---------
Co-authored-by: delize <4028612+delize@users.noreply.github.com>
Co-authored-by: CaliBrain <calibrain@l4n.xyz>
## Summary
Fixes Prowlarr downloads that fail after a queued torrent result’s
tracker download link expires.
Prowlarr torrent results can expose a `downloadUrl` that is only a
short-lived proxy to the upstream tracker. Some trackers, including MAM,
include expiring credentials in that URL. Shelfmark was persisting that
URL as retry data and later treating it as durable. If the in-memory
Prowlarr cache was gone, or if qBittorrent tried to add a stale URL,
Shelfmark could fail with a misleading torrent-hash error instead of
refreshing the release.
## What changed
- Stop persisting Prowlarr `downloadUrl` values as durable retry data.
- Persist only source context needed to refresh the release later.
- On a Prowlarr cache miss, re-query Prowlarr using the queued task
context.
- Accept refreshed results only when the stable identity matches the
original release:
- `guid == task.task_id`
- or `infoUrl == task.task_id`
- Cache the fresh raw Prowlarr result and build the download request
from its current `downloadUrl` / `magnetUrl`.
- If qBittorrent add fails with `Could not determine torrent hash from
URL`, remove the stale cached Prowlarr result, refresh once, and retry
with the fresh URL.
- Preserve existing magnet handling: torrent results continue to use
`magnetUrl` first, so the refresh path only targets Prowlarr proxy
`downloadUrl` values that can expire.
- Improve the user-facing failure when refresh cannot find the same
release:
`The indexer download link expired and the release could not be
refreshed. Search again for a fresh result.`
## Why this approach
The important constraint is avoiding accidental downloads of a different
edition or format after the original tracker link expires. Re-running a
search by title can return many plausible results, so the refresh path
deliberately requires an exact stable identity match before using any
new URL.
This treats Prowlarr `downloadUrl` as a short-lived hint, while still
allowing retries to recover when Prowlarr can find the same release
again. The one-shot retry after qBittorrent add failure handles the case
where Shelfmark still has a cached Prowlarr result, but that cached
result contains an expired proxy URL.
The refresh hook is source-specific and defaults to no-op for other
external download handlers, so Newznab and other sources keep their
existing retry behavior.
## Bug
Fixes#1012
Context:
https://github.com/calibrain/shelfmark/issues/1012#issuecomment-4917148398
## Validation
- `uv run pytest tests/prowlarr -q`
- `uv run pytest tests/download/test_orchestrator_user_output_mode.py
-q`
- `uv run pytest tests/newznab/test_handler.py
tests/audiobookbay/test_handler.py -q`
- `uv run ruff check shelfmark/core/models.py
shelfmark/download/orchestrator.py
shelfmark/download/clients/base_handler.py
shelfmark/release_sources/prowlarr/handler.py
tests/prowlarr/test_handler.py
tests/prowlarr/test_integration_handler.py
tests/prowlarr/test_failure_scenarios.py tests/prowlarr/test_source.py
tests/download/test_orchestrator_user_output_mode.py`
- after resolving conflicts with latest `main`: `uv run pytest
tests/prowlarr -q`
- after resolving conflicts with latest `main`: `uv run ruff check
shelfmark/release_sources/prowlarr/handler.py`
- after resolving conflicts with latest `main`: `uv run pytest
tests/prowlarr/test_handler.py tests/prowlarr/test_source.py -q`
Co-authored-by: Aidan Abbott <aidanabbott@Aidans-MacBook-Pro.local>
Clears up seedtime logic to use user-specified seedtime only, ignore the
indexer-defaults.
Adds a toggle to enable the seedtime feature, disabled by default.
Fixes#955
- Updated mirror selection
- Removed built-in mirror options, users must provide their own
configurations
- Set Universal search to default, added ability to disable direct
source
- Updated documentation
- Updated makefile
- Clean up a few uses of config options that may miss the env variable
if this is set
- Add enhanced retry availability utilising the DB to persist download
errors / retries across restarts, request failures, and pass Prowlarr
detail through the download task to maintain retry data.
- Strip back entrypoint permissions for less intensive chown operations.
Fixes#796
- Refactored user and request code to avoid any database conflicts
- Fix threading behavior with custom script execution
- Harden the no_auth activity user filtering
- Add a hint to add local admin if none is created
- Added secret key to persist login states across updates / restarts
- Added the manual retry option for failed downloads
- Added the ability to retry failed post-processing using existing
downloaded file
- Added admin-visible "Download as" selector, admin chooses a user to
download on-behalf of - inherits their output preferences.
- Added search mode and default metadata provider / release source
options to User Preferences and My Account settings.
- Added sort by format option in release results
- Added {OriginalName} renaming field option, to retain the exact
downloaded filename
- Frontend dependency updates - fixes rollup vulnerability from this
week
Closes#662#656#649#562
- Add certificate validation setting
- Fix some OIDC providers not linking emails to local users
- Reintroduce sort by peers option for prowlarr results
- Fix "All languages" search query reverting to default language
- Fix download/request dismissal with multiple admin users
- Fix download / request behavior on details modal
- Moved backend OIDC functionality to external library Authlib to help
maintainability
- Separated User settings UI into individual components, allowing for
standard settings UI decorator components to be used.
- Added full support for reverse proxy and CWA users alongside local and
OIDC
- Added mapping and syncing functionality for OIDC, CWA and reverse
proxy users
- Added per-user settings into the app-wide config system. Each config
can be declared as user-overrideable, and app-wide functionality can now
receive user-specific options via standard config calls.
- Added per-user audiobook destination config
- Updated login modal UI for simplified login, plus custom labels for
OIDC login
- Added user visibility in header dropdown
- Unified "restrict settings to admin" to use app-wide user roles.
- Add new booklore API file formats
- Renamed cookie for better login persistence with reverse proxy
- Updated fs.py to try hardlink before atomic move from tmp dir
- Fix transmission URL parsing
- Fix scenario where file processing of huge files starves the
healthcheck
- Large enhancements to custom scripting, including passing JSON
download info, more consistent activation across output types,
decoupling from staging behavior, and added full documentation.
- Refreshed available AA URLs
- Fixed potential redirect from AA itself causing mirror cache errors
- Added fully customizable mirror list in UI
- Segmented rotation behavior to Auto mode only
Fixes#588
splitsec2
Paul Rimmer
Marcelo Rodrigo
CaliBrain
Atirna
David Young
Ama Senevirathneandamasen02
Lance Marks
Zoltán SzabóandKukkerem
Aidan AbbottandAidan Abbott
Alex
Alex