mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-10-03 13:25:47 +01:00
aafce2be1dabe891d30eabd8aa85b4672808ef2a
202
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 CaliBrain
|
e5dd34ae0e |
fix: unbreak main and follow up on the Blackhole handoff review (#1346)
DownloadHistoryService.record_download and updated the single production caller, but not the eleven in the test suite, leaving main red with 32 failures. Pass None, which is what the pre-#1336 behaviour recorded. For the Blackhole handoff (#1345): add_download publishes the torrent before the cancel check runs, and BlackholeClient.remove() is a no-op, so the watcher picks the file up regardless. Reporting a bare "Cancelled" hid that from the user. Name the completed handoff in the cancellation message instead, drop the _handle_cancelled_download call whose usenet branch cannot apply to a handoff-only client, and record why the orchestrator no longer verifies HandoffResult.path. Finally, make tests/direct_download a package: test_libgen_extract.py imports tests.libgen.sample_html across test directories, so without an __init__.py pytest named its modules by bare basename and a same-named module elsewhere would collide. |
||
 Vinicius Gabriel
|
c53545d9fe |
Add configurable word separator for naming templates (#1333)
Closes #1230 ## What Adds a "Word Separator" setting (Space / Dot / Underscore / Hyphen / Custom) that replaces internal whitespace in each naming-template placeholder's rendered value — e.g. `{Author}` renders "Arthur.Conan.Doyle" instead of "Arthur Conan Doyle" when Dot is selected. This follows option 2 from the issue rather than inventing new dotted-keyword template syntax (`{Author.}`), since it's a smaller surface: one setting applies uniformly across all four templates (books/audiobooks × rename/organize) instead of needing a parallel token for every existing one. ## How it works - Literal characters typed into the template itself (e.g. the `.` in `{Author}.-.{Title}`) are never touched — only whitespace *inside* a placeholder's resolved value is affected. - Default is "Space", which is a no-op: existing templates produce byte-identical output after this change (verified via the existing test suite, unmodified, still passing). ## Where - `shelfmark/core/naming.py` — `word_separator` param on `parse_naming_template` / `build_library_path`. - `shelfmark/download/postprocess/policy.py` — `get_word_separator()`, mirroring the existing `get_file_organization()` accessor. - `shelfmark/download/postprocess/transfer.py` — wires the resolved separator through the four existing template-rendering call sites. - `shelfmark/config/settings.py` — new `Word Separator` / `Custom Word Separator` fields next to the existing naming-template fields. - `src/frontend/.../namingTemplatePreview.ts` + `NamingTemplateField.tsx` — the settings UI has its own TS mirror of the Python renderer for the live preview; updated it in lockstep so the preview doesn't lie about what the separator will actually do. - Tests added on both sides (pytest + vitest). ## Testing - `uv run pytest tests/core/test_naming.py tests/core/test_destination_file_organization.py` — all pass, including new cases. - `uv run pytest` (full suite) — same pre-existing failures as on `main` before this change (browser/network-dependent bypass & e2e tests unrelated to this diff), everything else green. - `uv run ruff check` / `ruff format --check` / `basedpyright` — clean. - `npm run lint` / `format:check` / `typecheck` / `test:unit` (196 tests) — clean. |
||
 Atirna
|
8f608f2e64 |
fix(download): complete consumed Blackhole handoffs (#1345)
A Blackhole watcher can consume the torrent before Shelfmark checks it, leaving the task in error even though the handoff succeeded. Complete the handoff when `add_download` successfully publishes the file, and stop requiring a `HandoffResult` path to remain present. Follow-up to #1312. ## Verification - A watcher that immediately reads and removes the torrent receives the exact bytes. The task changes from ERROR before this fix to COMPLETE afterward, without running book postprocessing. - The consumed-file regression fails on current main and passes here. Resident files, write failures, cancellation, magnet rejection and normal downloads remain covered: 81 focused tests pass. - Ruff lint and formatting pass for the changed files. |
||
 David Young
|
2bb84a17a2 |
Extract archives when zip/rar are enabled as supported formats (#1343)
The default audiobook formats include `zip` and `rar`. `scan_directory_tree` checks the supported-format list before checking for archives, so a downloaded archive lands in `book_files` and is imported as-is. The extraction branch in `collect_directory_files` is never reached. This keeps archives out of `book_files`, so they always take the archive path: extracted when extraction is allowed, imported as-is when it isn't (unchanged). Tests added in `tests/download/test_postprocess_scan_archives.py`; three of the four fail without the change. |
||
 Alex Guerrieri
|
c6b70a6844 |
fix(sources): send a Referer when fetching libgen ads.php pages (#1340)
## What libgen.li's `ads.php?md5=` now returns an **empty `200`** to any request without a `Referer` — an anti-hotlinking check the mirrors added recently. Both libgen paths fetch it without one, so the page comes back blank and the download silently fails while **search keeps working** (which is exactly why it looks like rate-limiting or mirror drift rather than a bug). Same one-line cause, two call sites: the Libgen search source (`libgen/scraper.py:fetch_page`) and the AA-md5 → libgen fallback (`direct_download/annas_archive.py:_extract_libgen_download_url`). Fix: send a same-origin `Referer: <scheme>://<host>/` on the `ads.php` fetch in both. ## Worth a look in review - **The referer goes on the *resolution* fetch, not the download.** `download_url(..., referer=...)` was already correct — the blank page happens one step earlier, at the `ads.php` GET. - Reproduced against live mirrors: `ads.php` returns `Content-Length: 0` bare, the full page with a `Referer`, and resolvable files download valid bytes again. Regression tests in `tests/libgen/` and `tests/direct_download/` assert the header on both paths. Lint/format/typecheck clean. Follow-up to #1326. |
||
|
CaliBrain
|
35b89b0d78 |
fix(sources): restore Direct Download search errors and language matches (#1339)
Fixes two regressions from the provider-driven refactor (#1337). First, the composite search caught RuntimeError, TypeError, ValueError and request errors from each provider and returned an empty list, so a failed search looked like one with no hits. It now raises the first provider failure when no provider returned releases. Second, the shared parser re-matched every row's language locally, dropping rows Anna's Archive had already matched with &lang= (free-text cells like 'English, French' or 'unknown'). parse_search_items gains a filter_languages option, which AA turns off, so AA's own language-from-path filter is again the only local one. |
||
 Tom
|
a5cd9f0bfb |
refactor: make direct download provider-driven (#1337)
This is the refactor for the download handler |
||
|
Alex Guerrieri
|
af21d1da1f |
feat(sources): add Libgen as a direct catalogue search source (#1326)
## What Adds **Libgen as a search source**. Today Libgen is only a download mirror (reached by an Anna's Archive md5), so anything in Libgen but not in AA's search index is invisible — and that's where most of the CBZ/CBR comics and manga live. A Libgen search for *One Piece*, for instance, turns up ~99 volumes that AA search never shows. It's a self-contained `release_sources/libgen/` package (source + handler + settings) plus one line to register it. **No changes to `direct_download.py`** — it reuses the existing `ads.php → get.php` resolution and the mirrors already configured in `LIBGEN_MIRROR_URLS`. Plain HTTP, no bypasser needed (libgen.li isn't behind DDoS-Guard). Opt-in via a settings toggle. ## Worth a look in review - **`source_id` is `libgen:<md5>`, not the bare md5.** The download queue keys on `task_id` (= `source_id`), and `direct_download` already uses the bare md5. Since AA indexes a lot of Libgen, the same md5 shows up from both sources — a bare id would collide in the queue. The handler strips the prefix before downloading. - **Reachable like the other non-default sources** (Prowlarr, AudiobookBay, …): it appears in the per-book release search, not the free-text box (that stays wired to `direct_download`). Tests in `tests/libgen/` cover parsing (both row layouts), the source, the handler, and `get_record`. Lint/format/typecheck clean. |
||
|
CaliBrain
|
1b17fe179a |
fix(irc): rank a surname-only result as partial, not wrong (#1332) (#1334)
"David Petrie" as "D. Petrie", then ranked the answer by the full name to recover the precision the surname gave up. The two halves disagreed. author_affinity needs two agreeing tokens before it calls a name the same person, so "Petrie" - the name on the filenames a surname search exists to reach - matched one and came back AUTHOR_MISMATCH. It therefore sorted below "Unknown" and level with "Gordon Petrie", a different author who merely shares the surname. The widened query pulled those rows in and the ranker buried them. Falling short of agreement is now separated from disagreeing with it. A name whose every token fits the one asked for is an abbreviation of it and ranks AUTHOR_PARTIAL, between agreement and "no author reported"; a name carrying a token that fits nothing still ranks AUTHOR_MISMATCH. Nothing that agreed before changes tier - "Homer"/"Homer Simpson" is still a match, since the extra token must not demote a mononym that already met its one-token requirement - so Prowlarr's #1293 ordering is unchanged except that a tracker listing a bare surname stops being read as the wrong author. Measured on the issue's own case, wanted "David Petrie": before: D Petrie, Unknown, Petrie, Gordon Petrie after: D Petrie, Petrie, Unknown, Gordon Petrie Second fix, same release: a book with no title posted the surname on its own. _build_query fell back to book.search_title or book.title, which is empty on exactly the path where the plan has no title variants, so the line reaching the channel was "@search Petrie" - not a search for anything, and the kind of bare over-broad post is_available refuses unaddressed queries to avoid. It now returns "" and the existing "No search query could be built" guard takes it. Tested with make python-lint, python-format, python-dead-code, python-typecheck and python-test. |
||
 Zoltán Szabó
|
35037b35fd |
fix(irc): search by surname, and rank the answer by author (#1331) (#1332)
Fixes #1331. A search bot ANDs every term against a filename, so the given name is the term that empties the result set. Measured against irchighway's #ebooks: "Revelations David Petrie" is answered "no results", "Revelations Petrie" returns 9 matches, 6 of which parse, all filed as "D Petrie". The query now carries the title and the surname, read off the search variant so the ISBN fallback and a manual query - which set author="" on purpose - keep their current shape. Title-only, the shape #1295 settled on for Prowlarr, does not transfer: the bot caps an answer at 1000 matches, and a bare "Revelations" hits that cap with 923 parsed rows across 500 authors, so the cap itself can drop the wanted book. The surname is the token the two spellings share and it keeps the answer small. The full author then orders what comes back, reusing author_affinity from #1295, since a surname also matches a different author who shares it. It sits under server availability the way indexer priority does in #1295: a download addresses one named bot and waits 120s for it, so a match from a bot that has left the channel must not outrank a mismatch that can answer. Ranking runs on the way out rather than before the cache, because one query identity is shared by every book that produced that query. Two things found while testing: - The parser writes the literal "Unknown" when a filename has no " - " separator (parser.py:168). Ranked literally that sorts as a wrong author, so author_affinity's middle tier was unreachable here; it is now read as absent. 5 of those 923 rows are affected. - author_affinity moves to shelfmark/core/author_match.py, unchanged, so IRC does not import from the Prowlarr package. Prowlarr behaviour is untouched and its tests pass as they are. The three IRC assertions in the #1252 regression file move to the surname form. The invariant they pin - one contributor's name reaches the query, never the whole credit list - is unchanged. Tested with make python-lint, python-format, python-dead-code, python-typecheck and python-test, and end to end against irchighway with the patched source: it posts "Revelations Petrie" and returns 6 releases. |
||
 Austin Brogle
|
99e0cfde3d |
fix: prevent Anna's Archive download countdown resets by preserving browser sessions (#1325)
## Observed bug Anna's Archive slow-download pages can return a JavaScript countdown before a download link is available. The internal browser returns that waiting-room HTML and closes its incognito session. The downloader then sleeps and fetches the URL again, which can create a new queue session and **restart the countdown instead of reaching the download link**. ## Fix - **Preserve the queue session:** keep the original browser tab open while the site's own countdown and automatic navigation finish. HTTP 200 and cached-cookie waiting-room responses enter the same flow. - **Return a consistent page:** capture HTML and readiness together in one browser evaluation so navigation cannot pair a new page's status with stale protection-page HTML. Share cache validation and page-readiness rules across their callers. - **Keep waiting cancellable and bounded:** poll cancellation while a slow browser read remains pending, rather than repeatedly cancelling and reissuing it. Apply a **300-second waiting-room limit** within the existing browser watchdog. - **Report queue timeouts accurately:** preserve the timeout across the helper-process boundary and stop the solve without restarting the browser or rotating mirrors. Waiting-room detection is limited to Anna's Archive `/slow_download/` pages containing an actual `.js-partner-countdown` element. The site controls the countdown and refresh. External-bypasser behavior and file-transfer timeouts are unchanged; the PR adds no deployment configuration or dependencies. ## Validation Validated at `c81e0a2`: | Check | Result | | --- | --- | | Full Linux unit suite | **2,978 passed** on Python 3.14 in a non-root environment with entrypoint test stubs enabled | | Focused regression coverage | **30 passed**, covering countdown completion, zero timers, navigation, both cookie-cache paths, cancellation, stuck queues, slow reads, and timeout propagation | | Navigation-race regression | Fails against the previous PR implementation and passes with the fix | | Python static checks | Ruff lint/format, BasedPyright for backend and tests, and Vulture passed | | Real Chromium fixture | Queue cookie persisted through 1.5-second DOM reads and one automatic refresh; the CDP connection survived multiple polling intervals | | Live source check | Observed **19 → 14 → 9 → 4 → download link** while retaining the browser session; the patched browser path also completed the waiting room | Full unit-suite command: ```sh pytest tests/ -n 2 --tb=short -m "not integration and not e2e" ``` The live check validates waiting-room completion and link resolution. Remote file-host availability remains a separate concern. The unit suite emitted two existing Authlib deprecation warnings. |
||
 Jorge Lima
|
c45d342931 |
fix(prowlarr): skip indexers in Prowlarr failure back-off (#1324)
## What
Read `/api/v1/indexerstatus` once per search and skip indexers whose
`disabledTill` is still ahead. Skipped is neither attempted nor failed.
One client method, one counter on `_IndexerSearchOutcome`, ten tests.
## Why
Prowlarr's own search leaves out an indexer it has disabled after
repeated failures. Shelfmark queries each indexer through its Torznab
endpoint, which answers 429 instead:
```
Prowlarr Torznab error response: <error code="429" description="Indexer is disabled till 09/09/2026 15:00:34 due to recent failures." />
Prowlarr: 1 of 5 indexer searches failed (indexer 2 search failed: 429 Client Error: Too Many Requests ...)
Release search failed for source prowlarr: 1 of 5 indexer searches failed (...)
```
That counted as a failed search, so with one indexer in back-off and the
other four answering empty, `/api/releases?source=prowlarr` returned 503
for every book for the length of the back-off (one hour here).
`/api/v1/indexerstatus` on Prowlarr 2.5.2:
```json
[{"indexerId": 2, "disabledTill": "2026-09-09T15:00:34Z", "mostRecentFailure": "2026-09-09T14:00:34Z", "initialFailure": "2026-09-09T14:00:34Z"}]
```
## Behaviour
| indexers | before | after |
|---|---|---|
| 1 in back-off, 4 answer empty | 503 "1 of 5 indexer searches failed" |
"No releases found" |
| 1 in back-off, 1 answers with releases | releases | releases, one
Torznab call fewer |
| 1 in back-off, 1 times out, 3 answer empty | "1 of 5 failed" | "1 of 4
failed" |
| every indexer in back-off | 503 "5 of 5 failed" | "every indexer is
disabled by Prowlarr after recent failures (until ...)" |
| status endpoint unreachable | n/a | as before, nothing skipped |
Auto-expand no longer retries a pass in which nothing was asked.
## Tests
`uv run pytest tests/prowlarr`: 563 passed, 42 skipped. `ruff check` and
`ruff format` clean.
|
||
 oliver
|
1e3fd48b8b |
fix: share rotating log file handlers (#1316)
This patch shares (for each log file) the `RotatingFileHandler` for logging across all modules, reducing the number of open file descriptors from ~78 to 1. I had originally assumed this issue was a resource leak, but it seems to just be a large fixed number of file descriptors. So this change mostly just (1) shrinks the number of open file descriptors to a reasonable level and (2) prevents two modules in the same process competing to write to a log file. |
||
 Vinicius GabrielandClaude Sonnet 5
|
c576003319 |
feat(naming): add {FirstAuthor} template token (#1322)
Closes #930. ## What New `{FirstAuthor}` naming-template token. It renders only the first author when metadata lists several ("Author1, Author2, Author3"), so multi-author books can be filed alongside the rest of that author's work instead of getting their own "Author1, Author2, ..." folder. ``` {Author} -> Terry Pratchett, Neil Gaiman {FirstAuthor} -> Terry Pratchett ``` ## How - Added to `KNOWN_TOKENS` in `shelfmark/core/naming.py`, positioned before `author` so `{FirstAuthor}` isn't parsed as literal `First` + `{Author}`. - Derived inside `parse_naming_template` from the existing `Author` value (split on `,` / `;`), so every caller — folder transfer, rename, the settings preview — picks it up with no extra wiring. An explicit `FirstAuthor` key in the metadata still wins if one is ever passed. - `{Author}` behaviour is unchanged. - Frontend `namingTemplatePreview.ts` token list + `KNOWN_TOKENS` kept in lockstep (there's a test enforcing that), with a matching `firstAuthor` helper. - Settings field descriptions + `docs/environment-variables.md` list the new token. ## Known limitation A lone author written `Last, First` is split on the comma too and renders as `Last` — the source metadata doesn't mark which form it is. Called out in the token help text and covered by a test. `{Author}` remains available for anyone who wants the raw string. ## Checks - `make python-test` — 2963 passed - `make python-lint` / `make python-format` / `make python-typecheck` / vulture — clean - `make frontend-test` — 187 passed · `frontend-lint` / `frontend-format` / `frontend-typecheck` — clean 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Atirna
|
265da07d7f |
feat(download): add Blackhole torrent handoff (#1312)
## Why Blackhole users need Shelfmark to hand a torrent file to their existing downloader instead of importing the downloaded book itself. ## Change - add Blackhole as a torrent client with a configurable watched directory - prefer a fetched `.torrent` file for Blackhole while preserving magnet preference for other clients - complete the queue task after the handoff without invoking book post-processing ## Verification - `uv run pytest -q tests/prowlarr/test_blackhole_client.py tests/prowlarr/test_handler.py tests/newznab/test_handler.py tests/download/test_orchestrator_lifecycle.py` - `uv run basedpyright shelfmark/download/clients/blackhole.py shelfmark/download/clients/__init__.py shelfmark/download/clients/base_handler.py shelfmark/download/clients/settings.py shelfmark/download/orchestrator.py shelfmark/release_sources/__init__.py shelfmark/release_sources/prowlarr/utils.py shelfmark/release_sources/prowlarr/handler.py shelfmark/release_sources/newznab/handler.py tests/prowlarr/test_blackhole_client.py tests/prowlarr/test_handler.py tests/newznab/test_handler.py tests/download/test_orchestrator_lifecycle.py` Fixes #1229 |
||
  Ama Senevirathneandamasen02
|
c18da92569 |
fix(postprocess): attach unmatched chaptered audio files to existing book group (#1176) (#1309)
### Summary Fixes #1176 When downloading an audiobook with many chaptered tracks (e.g. 250+ `.flac` or `.mp3` files), indexer XML or release metadata often caps the file list at ~100-110 entries. When the release extracts on disk, `match_plan_to_files()` matched those first ~110 files to the planned book group, while the remaining 140+ files fell into `unmatched` and triggered fallback heuristic grouping. Because heuristic grouping parsed the folder name (`Westwell - Hot & Cold (2023)`) and stripped the series/author prefix, it generated a second book titled `Hot & Cold` containing the remaining tracks, resulting in two split book folders. ### Changes - In `match_plan_to_files()` (`shelfmark/download/postprocess/packs.py`), check `unmatched` files before falling back to heuristic multi-book splitting. - If an unmatched file is chaptered audio (`.flac`, `.mp3`, `.aac`, etc.) and shares the directory with an existing book group, or if the plan was a single-book plan, append it to that group instead of creating a secondary book. - Non-chaptered standalone books (e.g. `.m4b`, `.epub`) or files in separate subfolders continue to fall back to heuristic grouping as before. - Added unit tests in `tests/download/test_packs.py` verifying: 1. Truncated track list in single folder properly appends remaining chaptered tracks without splitting. 2. Single-book plan with multi-disc audio files (`CD1`/`CD2`) groups together cleanly. 3. Multi-book packs with unmatched chaptered tracks route each track to its respective book folder. ### Testing Ran `uv run pytest tests/download/test_packs.py` and `uv run pytest tests/core/test_processing_packs.py` (all passed cleanly). Checked type annotations with `basedpyright` (0 errors) and formatting with `ruff`. Co-authored-by: amasen02 <amasen02@users.noreply.github.com> |
||
|
Jorge Lima
|
97d1bb0df4 |
fix(bypass): keep Anna's Archive's aa_ddg_check so clearance replays (#1305)
## What Add `aa_ddg_check` to the cookie-store allowlist. One name, one test. ## Why Every replay of stored clearance ends in the `?check=1` redirect loop, so each search pays a fresh browser solve. On this instance (v1.3.15, WireGuard egress, 0 VPN restarts across the traces) not one replay was accepted in three days of DEBUG logs. The `__ddg*` cookies are stored and replayed correctly. Anna's Archive also sets a cookie of its own, `aa_ddg_check`, and its `?check=1` hop only answers with the page when that cookie is present too. The allowlist keeps `cf_*` and `__ddg*` names, so this one was never stored. ## Measured, same egress IP, cookies taken from one solve | replayed | plain `requests` | `curl_cffi`, Chrome TLS fingerprint | |---|---|---| | filtered `__ddg*` only (current behaviour) | 302 → 302 → 302 … loop | 302 → 302 → 302 … loop | | filtered + `__ddg8_/9_/10_` | loop | loop | | filtered + `aa_ddg_check` | **302 → 200, real search page** | 302 → 200 | | `aa_ddg_check` alone | 302 → 403 | — | So the TLS fingerprint is not the problem, the per-check trio is not the answer, and the cookie needs the `__ddg*` clearance next to it. Cookie attributes as issued: domain `.annas-archive.gl`, path `/`, expiry 90 days. It is not bound to the query, and it is accepted with a stock Python User-Agent. ## Through the real fetch path Same process, `html_get_page`, the name allowlisted, three different queries: ``` 1st: solve expected 25.8s bypass_calls=1 title='frankenstein shelley - search - an' md5=True 2nd: other query 9.5s bypass_calls=0 title='pride and prejudice austen - searc' md5=True 3rd: third query 4.8s bypass_calls=0 title='dracula stoker - search - anna's a' md5=True ``` ## Notes - `tests/bypass/test_ddg_cookie_reuse.py` gains `test_aa_check_cookie_is_stored`; its docstring table gains the row. The bypass tests need seleniumbase to import and do not run on my macOS host, so this leans on CI. `ruff check` and `ruff format --check` pass. The logic was checked directly against `cookie_store` with the settings registry stubbed. - `__ddgmark_` carries a 24 h expiry, so the store's clearance is good for about a day before the next solve, which is what a browser would see too. - Follow-up to #1286. Same instance, same method: DEBUG trace, then a probe script inside the container. |
||
  Matt Van HornandMatt Van Horn
|
9f11e83e1f |
fix: keep polling queued Real-Debrid torrents (#1303)
Add `queued` to the existing set of non-terminal Real-Debrid torrent states so `_handle_torrent_info` returns an in-progress `DownloadStatus` and leaves the mutable download state eligible for subsequent polling. Keep the change within the existing status-classification path rather than introducing a new helper or changing the broader handling of unknown statuses. The native Real-Debrid client currently treats the documented `queued` torrent status as a terminal error because it is absent from `_STATUS_DOWNLOADING`. This occurs after a torrent has been added and its files selected, particularly for uncached torrents that wait before downloading. A torrent-info payload with `status: queued`, zero progress, and a filename returns a non-complete `DownloadState.DOWNLOADING` result rather than `DownloadState.ERROR`; After handling `queued`, the internal download state remains non-terminal so a later status poll can be processed instead of returning a cached error. Fixes #1268 Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> |
||
|
CaliBrain
|
9452ebc70d |
fix(bypass): stop handing solvers DDoS-Guard's ?check=1 probe URL (#1300)
html_get_page follows Anna's Archive redirects by hand, and DDoS-Guard's gate answers /search with a 302 to the same path plus `check=1`. The follower walks that handshake by reassigning `current_url`, so every downstream handoff - the 403 branch, the 503-challenge branch, both redirect-loop rescues - passed the *probe* URL to the bypasser rather than the page we actually wanted. A solver opens that in a fresh browser holding none of the cookies the probe exists to collect, so DDoS-Guard cannot verify it automatically and serves the manual CAPTCHA page that nothing can solve. The #1292 log is exactly that: a 403 handed off on `&check=1`, FlareSolverr answering "Challenge solved!", and a 4721-byte DDOS-GUARD captcha page coming back. - `_solvable_url()` strips the probe parameter, applied at the single choke point in `_run_bypasser` so all four handoffs are covered. Scoped to the hosts whose redirects we follow manually; a URL without the parameter is returned by identity, so nothing else is re-encoded. The same reports showed three further defects, all of which stand whatever the host was reacting to: - The external bypasser logged that the solve had not cleared the protection and then returned the challenge page as a success. That skipped the one recovery left - get_bypassed_page's retry-and-rotate loop, where the next mirror is a different DDoS-Guard host - and filed the captcha page's own __ddg cookies as that host's clearance, to be replayed on every later request. It now raises ChallengeNotSolvedError before storing anything. - "Check that the bypasser is reachable and working" was the one piece of advice guaranteed to waste the reporter's time: it was reachable, it ran a full solve, and it returned a captcha. ChallengeNotSolvedError carries the marker so the search layer can name the host as the cause instead of the bypasser. - The untabled-page fingerprint logged `attempt_url`, which html_get_page has since rotated past. The #1298 bundle reported the page against annas-archive.gl when the body had come from .pk - the triage cost #1289 added the line to remove. The search now asks for the response URL and logs that. Its give-up shape is the tuple ("", url), which is truthy, so the exhaustion check reads the body rather than the response. Regression fixtures are built from the pages in the reports. The two behavioural handoff tests were checked against the unfixed code: both fail there, reproducing the reporter's log line verbatim. Refs #1292 Refs #1298 |
||
|
CaliBrain
|
cb690b45b8 |
fix(prowlarr): rank releases by author instead of querying for it (#1293) (#1295)
MyAnonamouse is the only indexer Shelfmark treats as enriched, and it
alone was sent {title} {author} while every other indexer got the title
on its own. MAM matches all search terms conjunctively, so whenever the
metadata provider spelled the author differently to the tracker -
Hardcover says Timothy Ferriss, MAM lists Tim Ferriss - the search came
back empty and the UI reported No releases found for this book, with the
release sitting on the tracker the whole time.
The enriched flag is a statement about responses: MAM returns clean
author and bookTitle attributes, which is why it earns format detection
and preferential ordering. Using that same flag to shape the request is
the actual defect, and it is why turning the flag off recovers the
search but takes format detection down with it.
So the query is title-only for every indexer now, and the author orders
the results rather than narrowing them. MAM already hands us its author
field, so agreement is judged on data we hold instead of by an AND we
cannot control. The ranking is three-way on purpose - agrees, no
metadata, disagrees - so an indexer reporting no author does not sort
below one reporting the wrong author.
A wrong verdict costs a release its position, never its visibility: a
transliteration such as Dostoevsky against Dostoyevsky sorts last
instead of vanishing. That is what makes the loose token comparison safe
to ship without a tuning knob.
Falling back to a title-only query on zero results was the alternative.
It only rescues total failure - if two of six editions happen to use the
provider's spelling, the search returns those two, no fallback fires,
and the user quietly gets a truncated list. It also spends a round trip
inside the search deadline and stacks a retry on an indexer that may
still be solving a challenge (#1249).
Manual queries skip author ranking: they are the user's own words and
should not be reordered against the metadata they were typed to
override.
|
||
|
CaliBrain
|
3d7ea40088 |
fix(search): reach the server's deadline, query one author (#1285, #1252) (#1287)
Two independent reasons a working search reported failure to the user. 1. The client gave up before the server did (#1285) `/api/releases` bounds one release search with RELEASE_SEARCH_TIMEOUT (default 300s) and answers a spent budget with a sentence naming the real cause - the machinery added for #1276. The frontend then aborted the direct_download search at a hard-coded 180s, so it always won the race: the user saw "Request timed out. Check your network connection or proxy configuration." instead, and raising RELEASE_SEARCH_TIMEOUT changed nothing they could observe, the 180s being baked into the hashed bundle inside the image. - /api/config reports the effective (clamped) budget, and the client derives its abort from it plus a margin, so the server always answers first. - Direct-mode search shows what the server actually said. Every non-auth failure was relabelled "Unable to reach download source. Network may be restricted or mirrors blocked.", which discarded the explanation and blamed the user's network. ApiResponseError now carries `serverMessage`, set only when the server explained itself, so the status-line placeholder still falls back. Two latency fixes for the cost that made the timeout reachable at all: - Fetch each distinct AA search URL once per search. The language-filter retry re-runs every title variant, and with DIRECT_DOWNLOAD_LANGUAGE_FROM_PATH on both passes build a byte-identical URL - behind DDoS-Guard each repeat is a fresh browser solve. - Drop the solve-only bypass method. `_bypass_method_cdp_gui_click` opens with exactly that call and returns the moment it works, so the entry ahead of it could only repeat the half that had already failed, plus the backoff before the method that does work started. Reported at 0/19 successes and ~5.5s of each ~26s solve against DDoS-Guard. 2. The query carried every contributor, not one author (#1252) `_pick_search_author` returned `book.search_author` verbatim while the authors[] fallback beside it deliberately narrowed to the first name before a comma. Both fields routinely arrive holding every contributor joined with ", ": the frontend builds `book.author` as `authors.join(', ')` for display (bookTransformers.ts) and the release modal sends that display string straight back as the `author` parameter, and `browse_record_to_book_metadata` and the manual-search branch both split the joined text into `authors` while still passing the unsplit string as `search_author`, so the split was never used. A book whose metadata lists translators was therefore searched for as Blindness Jose Saramago, Giovanni Pontiero, <persian translator> which matches nothing on Anna's Archive. The bypass succeeds, the search comes back empty, and the user is told the book has no releases. Narrowed in one place, `search_plan.first_author`, so the two branches cannot drift apart again, and applied to the IRC source, which built its query with the same verbatim preference. Hardcover is unaffected: it already sets `search_author` from `_simplify_author_for_search(authors[0])`, which resolves "Last, First" itself and never yields a multi-author string. |
||
|
CaliBrain
|
633004ecf0 |
fix(search): stop reading real Anna's Archive pages as unsolved challenges (#1294)
`_looks_like_challenge_page` substring-matched "ddos-guard"/"cloudflare" over the whole document. DDoS-Guard-fronted sites carry those strings on their own pages - Anna's Archive ships a `DDOS-GUARD` comment in the inline JS it serves on every page - so every real AA response that was not a results table was reported as an unsolved protection challenge, sending users off to fix a bypasser that had just succeeded. Measured against live pages: a served AA page (HTTP 200) is 182,685 bytes and matched the old detector; the real interstitial is 902 bytes. - `_looks_like_challenge_page` now delegates to the shared `challenge_marker()`, whose 64 KB cap is what separates a few-KB interstitial from the page behind it. `download/http.py` already used it; this module carried an unguarded private copy. - `_looks_like_aa_page` is checked ahead of the challenge branch. A genuine interstitial carries no AA markers, so nothing actually blocked leaks through. Also adds the diagnostics whose absence made #1289 guesswork: the debug bundle carries no response bodies, so "unsolved protection challenge" and FlareSolverr's "Challenge solved!" were indistinguishable after the fact. - `_log_untabled_search_page()` fingerprints the one ambiguous shape at INFO - size, size-cap verdict, AA markers, challenge marker - with a bounded 700-char head at DEBUG. Best-effort: it swallows its own errors. - The external bypasser records what it actually returned, and warns when it reports success while handing back a challenge page. Regression tests use fixtures built from the live pages rather than invented ones; the previous fixtures were two-line synthetic pages with no "ddos-guard" substring, which is why nothing caught this. Closes #1289 Closes #1292 |
||
 Nicholas Velten
|
69ff0d6a78 |
fix: trim a credit list in search_author to the first name (#1290)
Fixes #1252 for the case in the second report. `_pick_search_author` returns `search_author` untouched but trims `authors[0]` to its first comma-separated name. So the same credit list searches differently depending on which field carries it: ``` via authors[0] -> "Blindness Jose Saramago" via search_author -> "Blindness Jose Saramago, Giovanni Pontiero, Zohreh Eftekhari" ``` Anna's Archive answers the second one with nothing. That is the query in @theDoz12's log, and it explains the shape of the report: the bypass succeeds, the search runs, and the UI still says no releases. Nothing in the download path is broken, the query simply cannot match. Measured against live AA on 1.3.14, same book, same source, only the field carrying the author changed: | query | releases | | --- | --- | | `Blindness Jose Saramago, Giovanni Pontiero, Zohreh Eftekhari` | 0 | | `Blindness Jose Saramago` | 49 | | `Blindness` | 50 | With the patch the second form is produced from either field, and the same search returns 49. Three regression tests added, including one that asserts both fields yield the same query. On `tests/core/test_search_plan.py` the run goes from 5 failures to 3; the 3 that remain are the language tests, which fail identically with and without this change on my machine. Worth saying what this does not cover: the first report in that issue ends with `Found 2 releases via ISBN` and still shows nothing, so that one is a different fault further along. I could not reproduce it here. |
||
|
Jorge Lima
|
d7fe28595c |
fix(bypass): wait for the solved page before reading its source (#1286)
Follow-up to #1276 with a measurement from the instance I reported there. v1.3.13 solves the challenge again, but on my setup the solve was being thrown away immediately afterwards: ``` 19:26:08 Bypass successful using _bypass_method_cdp_gui_click 19:26:16 Bypass failed (attempt 1/10): TimeoutError: Time ran out while waiting for: {html} ``` `_get()` ends with `return await page.get_page_source()`, which is `find("html", timeout=1)` in SeleniumBase. One second is enough for a page that is already sitting on its content, but Anna's Archive answers a cleared check with a redirect to the real page, so the document is not there yet. The solve is discarded, the whole attempt restarts, and the extra requests are what earn the 429 that `note_rate_limited()` then parks the host for — 120 s, then 300 s. ## Change `_read_page_source()` waits for the document itself, with a `BYPASS_PAGE_SOURCE_TIMEOUT` setting (default 20 s, min 1, max 120) in Direct Download → Cloudflare Bypass, next to the existing bypasser timeouts. ## Measured on a live instance I patched the wait in the running container (`find("html", timeout=1)` → `timeout=20` in the installed seleniumbase, which is the same effect as this PR) and re-ran the same searches on the same host, k3s behind a Surfshark WireGuard exit, internal bypasser, v1.3.13: | | 1 s wait | 20 s wait | |---|---|---| | `Time ran out while waiting for: {html}` | one per solve | none | | 429 backoffs | 2 (120 s, then 300 s) | none | | Search for a book AA has | 199 s and 200 s, both errored | 61 s, 2 epub releases | A download after that took 5 s from LibGen, so the search was the whole cost. ## Tests Two tests in `tests/bypass/test_bypass_budgets.py`, the file already covering #1276: a page that needs longer than a second still yields its HTML, and `BYPASS_PAGE_SOURCE_TIMEOUT` overrides the default. `uv run pytest tests/ --ignore=tests/e2e`: 2848 passed, 47 skipped. Ruff check and format clean. The docs table is auto-generated, but running `scripts/generate_env_docs.py` here rewrote unrelated entries (Newznab, BOOK_LANGUAGE), so I added only the new entry by hand in the generator's format rather than commit that churn. One thing I could not judge from outside: whether 20 s is the right default for hosts other than AA. It only costs anything when a solve would otherwise be discarded, but I have measured it on one site. |
||
|
CaliBrain
|
97e289ae13 | fix: search, Prowlarr and qBittorrent follow-ups (#1276, #1283) (#1284) | ||
|
Jorge Lima
|
c95ee72ad5 |
fix(qbittorrent): keep magnets whose metadata is still pending (#1282)
## Problem
`QBittorrentClient.add_download()` waits 20 × 0.5 s for qBittorrent to
leave `metaDL`, then raises:
```
Failed to add to qbittorrent: Torrent metadata resolution was not confirmed within the visibility grace period
(response=TorrentsAddedMetadata({'added_torrent_ids': [], 'failure_count': 0, 'pending_count': 1, 'success_count': 0}))
```
The wait exists to learn qBittorrent's primary torrent ID, which for
hybrid torrents switches from the v1 hash to the truncated v2 hash once
metadata resolves. A magnet on a thin public swarm routinely needs
longer than 10 s to find a peer that will serve metadata, and the
download is then abandoned even though the add itself succeeded. The
torrent stays in qBittorrent (`base_handler` logs "leaving in
qbittorrent") and often completes minutes later with nobody watching it.
Seen on v1.3.12 with public indexers through Prowlarr: every magnet-only
release failed this way, while `.torrent` releases from a private
indexer were fine. qBittorrent showed the same torrents at `metaDL 0%
seeds=0/0`, and they resolved on their own well after shelfmark had
given up.
## Change
Return the info hash we already have instead of raising when the grace
period expires. Reads then resolve either identity:
- `get_status()` and `get_download_path()` use `_resolve_torrent()`
instead of `_get_torrent_info()`, so a v1 hash still matches after
qBittorrent re-keys the torrent to v2. `_torrent_matches_download_id`
already compares `hash`, `infohash_v1` and `infohash_v2`.
- `remove()` and `set_category()` address the torrent by its current
primary hash through a new `_current_hash()` helper, which falls back to
the ID it was given when the torrent cannot be resolved.
- The two magic numbers become `_METADATA_WAIT_POLLS` and
`_METADATA_WAIT_INTERVAL_SECONDS`.
The happy path does not change. When metadata resolves inside the grace
period the resolved primary hash comes back as before, and
`_resolve_torrent()` tries the exact-hash lookup first, so it costs no
extra request.
## Tests
`test_add_fails_when_metadata_never_resolves` asserted the old
behaviour, so it becomes
`test_add_keeps_torrent_when_metadata_never_resolves` and asserts the
info hash is returned.
`test_get_status_resolves_hash_after_metadata_switch` is new: it reads
status by the v1 hash after qBittorrent reports the torrent under its v2
hash.
`uv run pytest tests/ --ignore=tests/e2e` gives the same 55 failures
with and without this change (they are all in `tests/bypass/` and need
Chrome, which my machine has no headless setup for), and
`tests/prowlarr/` is green at 524 passed. Ruff check and format are
clean. I have not run this branch against a live qBittorrent, so a
second pair of eyes on the `remove()` path would help.
|
||
|
CaliBrain
|
b25acdb2ad |
fix(packs): don't disrupt normal downloads when inspecting for packs (#1274)
Follow-ups to the multi-book pack feature (#1270), which inspects every release before download. Two behaviours leaked into the ordinary single-book flow and are corrected here: - A flat folder of chaptered audio (`01 - Chapter.mp3`, `02 - ...`) was detected as a pack, because each track name parses to a series position, so clicking download popped the review panel for one normal audiobook. Flat folders are now split one-book-per-file only with real evidence of distinct books: two or more series positions, more than one title, and no chaptered audio (only the single-file m4b/m4a containers and ebook formats qualify). Subfolder packs and flat m4b/m4a packs are unchanged. - Every release that couldn't be inspected (usenet, magnet-only, sources without a list_files hook, ABB single-file) showed an info toast on download. That is now a console.warn, so a normal download is silent again. Adds regression tests for the chaptered-mp3 cases. |
||
 Lance Marks
|
f441b85da2 |
feat(packs): inspect multi-book releases and file each book separately (#1270)
## Multi-book packs: inspect a release before download and file each book separately Closes #576 ### Problem One queued release is always treated as one book. When a torrent is actually a whole series (`Series/Book 1 - Title/…`, or a flat folder of `Series 1.0 - Title.m4b` files), post-processing walks the whole tree, flattens every file into one list and renames them `Title - 01…10` under the searched book's `{Author}/{Title}`. Audiobookshelf then sees a single 10-file "book" and the user has to re-file everything by hand. ### What this does Most releases expose their file list *before* anything is downloaded, so the split is decided up front and approved by the user, then the download is fire-and-forget: 1. **Inspect** – clicking a release's download button now calls `POST /api/releases/inspect` first. A new optional `DownloadHandler.list_files(release_data)` hook returns the release's files without downloading: - **AudiobookBay** reads the torrent file table off the detail page it already fetches (the page is now cached for 120 s, so inspect + download cost ABB one request). - **Prowlarr** parses `info.files` from the `.torrent` it already fetches (the existing 120 s torrent-fetch cache is reused). Magnet-only and usenet releases report "can't inspect". - Other sources default to `None`. 2. **Review** – if the plan contains more than one book, the Find Releases modal swaps the list for a review panel: one row per book with editable title / series position / year, expandable file lists, non-book sidecars (`.txt`, covers) shown as ignored, a "Treat as a single book" switch, and **Download N books**. Single-book releases queue immediately, exactly as before. 3. **File** – the approved plan travels with the task (`DownloadTask.book_plan`, retry-safe) and post-processing files each book through the existing transfer code, one book at a time (`dataclasses.replace(task, title=…, series_position=…, year=…)`), so organize/rename templates, part numbering (now scoped per book), hardlinks, torrent copy-preserve and usenet handling are unchanged. Status reads `Complete (N books, M files)`. 4. **Fallback** – when a release can't be inspected the user gets a toast, and a small "Multi-book pack" toggle in the modal header forces a heuristic split (subfolder = book, or one book per file when the file names carry series positions). Planning lives in `shelfmark/download/postprocess/packs.py` and is shared by the inspect endpoint and post-processing, so what the user approved is what gets filed. The name parser strips `Book 3 -`, `03 -`, `1.0 -`, `3.`, `[03]`, `#3`, a leading series name, labels like "An Expanse Novella -", repeated titles (`Gods of Risk 2.5 - Gods of Risk`) and a trailing `(Year)`; author and series name come from the book that was searched, and the searched book's own series position is never applied to its siblings. ### Files - `shelfmark/download/postprocess/packs.py` (new) – `PackFile/PackBook/PackPlan`, `plan_pack`, `parse_pack_book_name`, `group_files_into_books`, `match_plan_to_files` - `shelfmark/core/release_inspect_routes.py` (new) – `POST /api/releases/inspect` - `shelfmark/release_sources/__init__.py` – `DownloadHandler.list_files` hook - `shelfmark/release_sources/audiobookbay/{scraper,handler}.py` – detail-page cache, `extract_file_list`, `list_files` - `shelfmark/release_sources/prowlarr/handler.py`, `download/clients/torrent_utils.py` – `extract_file_list_from_torrent`, `list_files` - `shelfmark/core/models.py`, `download/orchestrator.py` – `multi_book` / `book_plan` fields, queue + retry serialization - `shelfmark/download/postprocess/transfer.py`, `pipeline.py`, `outputs/folder.py` – per-book transfer branch and status message - `src/frontend`: `components/PackReviewPanel.tsx` (new), `ReleaseModal.tsx`, `App.tsx`, `services/api.ts`, `types/index.ts`, `utils/releasePayload.ts` (payload builder moved out of `App.tsx`), `utils/packReview.ts` - `docs/dev/release-sources-plugin-guide.md` – documents the `list_files` hook ### Out of scope (follow-ups) - Listing files from an NZB (Shelfmark already fetches the bytes; `<file subject>` names are noisy) - Inspecting magnet links via qBittorrent's files API after a paused add - BookLore / email outputs (they ignore `book_plan`; noted in code) - The combined ebook + audiobook flow ### Testing **Automated** (`make checks`, `make python-test`, `make frontend-test` all green; the only failures on my machine are the pre-existing `tests/config/test_entrypoint_permissions.py` cases, which need bash ≥ 4 and fail identically on `main` under macOS bash 3.2): - `tests/download/test_packs.py` – name parsing (markers, series name, novella labels, repeated titles, bare numeric titles like `1984`), nested / flat / mixed / deeper-nested packs, single wrapping folder not treated as a pack, plan-to-disk matching with basename fallback - `tests/core/test_processing_packs.py` – full `post_process_download` runs on a real temp filesystem: approved plan files each book under its own `{Author}/{Title}`, heuristic split of a nested pack, searched book's series position does not leak, multi-file book inside a pack keeps `- 01/- 02` per book, hardlinked torrent pack leaves the seeding tree intact, no pack fields ⇒ behaviour unchanged, single group degrades to the searched title, status message - `tests/core/test_release_inspect_routes.py` – plan response, not-inspectable, handler errors never 500, unknown source / missing `source_id` ⇒ 400, login required - `tests/audiobookbay/test_file_list.py` – file-table scraping from real ABB markup (multi-file and single-file pages), handler host validation, one page fetch shared by magnet + file list - `tests/prowlarr/test_torrent_file_list.py` – multi-file / single-file `.torrent` parsing, handler behaviour for torrent URL vs magnet vs usenet vs cache miss - `tests/download/test_orchestrator_pack_fields.py` – queue-time parsing and retry round-trip - Frontend: `releasePayload.test.ts`, `packReview.test.ts` (vitest) **Manual, on a real deployment** (arm64 image built from this branch, run as a side container next to production with the same qBittorrent / Audiobookshelf setup, `FILE_ORGANIZATION_AUDIOBOOK=organize`, hardlinks on): - AudiobookBay "The Expanse Complete 2.0" (7.87 GB, 36 files): clicking download opened the review panel in ~1 s showing **18 books · 18 files · 18 files ignored** (the `.txt` sidecars), with series positions 0.1–9.5 and years parsed from the file names; novella labels stripped ("The Churn", "The Butcher of Anderson Station"). Editing a title in the panel works. Confirming queued one task; the magnet resolved from the cached page in ~30 ms; after the download the task reported `Complete (18 books, 18 files)`, 18 hardlinks landed as `audiobooks/James S. A. Corey/<Title>/<Title>.m4b`, the torrent kept seeding, and Audiobookshelf scanned each folder as its own book (title, author, embedded chapters). - A second pack ("Expanse [01 - 9.5]", `Title N - Title` naming) was inspected to verify the repeated-title rule and the Back button, without downloading. - Single-book releases still queue immediately with no extra UI. |
||
  zab1996andRyan
|
02b7e9d958 |
feat(newznab): support multiple named indexers (#1271)
## Summary - add a named Newznab indexer table with per-indexer URL and API key settings - search every configured indexer and retain the originating indexer name on each result - namespace cached release IDs across connections and isolate individual indexer failures - preserve the legacy single-indexer settings as a fallback - support masked API-key cells and trusted SABnzbd prefetching for named indexers ## Validation - 121 Newznab and SABnzbd backend tests passed on Python 3.14 - Ruff passed for all changed Python files - frontend TypeScript and strict lint checks passed - all 134 frontend unit tests passed - frontend formatting check passed ## Compatibility Existing `NEWZNAB_URL` and `NEWZNAB_API_KEY` configurations continue to work whenever `NEWZNAB_INDEXERS` is empty. Co-authored-by: Ryan <zab1996@users.noreply.github.com> |
||
|
CaliBrain
|
ff06a1a581 |
fix(search): follow-ups to per-user book languages (#1267)
Review follow-ups to #1255, all in the code that PR touched. Drop the dead user_id from the Prowlarr retry path. ProwlarrSource.search never reads plan.languages, and _refresh_release builds a synthetic book with no titles_by_language, so the title variants came out identical with and without it. It also should not language-filter: it re-finds one exact release by its guid. Pin the tab move in tests. BOOK_LANGUAGE moved from the General tab to Search Mode with no migration, which only works because both tabs persist into the same settings.json. Nothing asserted that, so splitting the files later would silently reset every install to ["en"]. Covers the stored value, a fresh install, and ENV precedence. Stop the UI inventing a default language. An empty BOOK_LANGUAGE is a deliberate "no default filter" that the backend preserves, but the two frontend call sites replaced it with the first supported language, so the filter said English where the server filtered nothing. resolveDefaultLanguageCodes now falls back only when the value is absent. Keep the normalized value for every validated search key. validate_user_settings gated the write-back on a hand-maintained subset of the keys the search validator recognises, so METADATA_PROVIDER_COMBINED, SHOW_COMBINED_SELECTOR and FORCE_COMBINED_SEARCH were validated and then stored raw -- a padded provider name was accepted and persisted with its padding. Reuse the validator's own key set instead. Skip blank language entries rather than rejecting them, so "" and "en," mean the same as [] and ["en"] instead of erroring on an unnamed language. Extract resolveListOverride for the list-override detection that was copy-pasted between the two user-settings sections, and mention languages in the Search Preferences section description. |
||
|
463ef49ac3 |
feat(search): let each user pick their own default book languages (#1255)
## Why `BOOK_LANGUAGE` is a per-reader property, not a per-instance one. On a shared install one household member searches in German while another wants English and German — today whoever changes the setting changes it for everyone, and the only escape is re-picking languages in the filter on every single search. The per-user override machinery already carries `SEARCH_MODE`, the metadata providers and the default release sources, so the language default mostly had to opt into it. ## What changed **The field.** `BOOK_LANGUAGE` becomes `user_overridable` and moves from the **General** tab to **Search Mode**, next to the other user-overridable search defaults (per [review](https://github.com/calibrain/shelfmark/pull/1255#issuecomment-5391189094) — the first version had the Search section span two tabs, this one doesn't). Admins set it per user in the user editor, users set it in **My Account → Search Preferences**, and the Search Mode tab carries the usual "N users override this" summary. **No migration for the move.** `general` and `search_mode` both persist into `settings.json`, and a field's value is resolved through `load_config_file(tab)` for the tab it's declared on — so an install that already stores `BOOK_LANGUAGE` keeps its value. Checked against a `settings.json` written while the field still lived on General: the stored value resolves unchanged, a fresh install still gets `["en"]`, and `BOOK_LANGUAGE` in the environment still overrides both. **The two places the default is read.** - `/api/config` seeds the frontend's language filter, so it now resolves `BOOK_LANGUAGE` for the session user. - `build_release_search_plan` falls back to the default whenever a request carries no language filter — which is exactly what the filter's "Default" option sends. It takes an optional `user_id`, passed by `/api/releases` from the session and by the Prowlarr retry path from `task.user_id`, so a retry re-searches in the languages of whoever queued the download. **Validation.** Overrides go through `normalize_language()`, so `"German"`, `"ger"` and `"de"` all store as `de`, and an unknown language is rejected with a message naming it instead of being silently searched for. An empty list stays an empty list (a deliberate "no default filter"), `null` clears the override as everywhere else, and ENV still wins: with `BOOK_LANGUAGE` set in the environment the field reports `fromEnv` and overrides are ignored. **Scope.** Only the language default becomes overridable. The two format lists left behind under "Default Search Filters" stay admin-only — they describe what the library and its post-processing accept, not what a reader wants to read. There's a test pinning that. ## Verification - 2681 unit tests pass (2670 before, 11 added) - `ruff check`, `ruff format`, `basedpyright` over backend and tests, and `vulture` all clean; frontend lint, format, typecheck and 126 unit tests clean - `docs/environment-variables.md` regenerated via `scripts/generate_env_docs.py` (the `BOOK_LANGUAGE` row follows the field into the Search Mode section) - Manually against a two-user instance with builtin auth (first round, before the tab move): with user A on German and user B on English+German, `/api/config` returns each reader their own `default_language` and an unfiltered `/api/releases` plans the matching languages; an admin can set and read the same override for another user; clearing it falls back to the global value; a stray `"klingon"` is rejected; and `BOOK_LANGUAGE` in the environment overrides both users with the field marked `fromEnv` - After the tab move I re-ran the suites above plus the stored-value/fresh-install/ENV check described under "No migration for the move"; the behaviour it exercises is what the move could have broken Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
CaliBrain
|
a5595cf9f1 | Change test for fake extension that wont work (#1266) | ||
 jakesterpdxandClaude Fable 5
|
9bcf595111 |
feat(prowlarr): warn when an indexer declares a format Shelfmark can't process (#1265)
## Problem Companion to #1264, but general rather than mp4-specific. MyAnonamouse titles carry a structured `[LANG / FORMATS]` bracket that `_extract_mam_formats` parses. When every token in it is something Shelfmark doesn't know — e.g. `The Martian by Andy Weir [ENG / MP4]` — the release is rendered with **no format chip at all**, just the generic headphones/book icon with an "Audiobook" tooltip. To a user that looks like an ordinary result. It downloads fine and then fails post-processing with *"No book files found in download"*. The backend already *had* the signal (a format token it couldn't map); it just threw it away. ## Change **Backend** (`shelfmark/release_sources/prowlarr/source.py`) - `_split_mam_formats(raw_title) -> (recognized, unrecognized)` replaces the body of `_extract_mam_formats`, which is kept as a thin wrapper returning `recognized` so nothing else changes. - Releases gain `extra["unrecognized_formats"]` (list, or `None` when empty / when format detection is off). **Frontend** - `getUnrecognizedReleaseFormats(release)` in `utils/releaseFormats.ts` (normalised + deduped, same shape as `getReleaseFormats`). - `ReleaseCell` `format_content_type`: when there is **no** recognised format but the indexer named one, render an amber `MP4 Unsupported` badge (compact view: amber `MP4`) with tooltip *"Unsupported format (MP4) - Shelfmark cannot process this release"*. When a recognised format exists the existing badge is untouched, even if extra unknown tokens were present. Only the chip changes — the download button still works, so a user can still grab and hand-process the files if they want to. Happy to disable the button instead if you'd prefer. ## Tests - `tests/prowlarr/test_source.py`: `TestSplitMamFormats` (recognised / unrecognised / mixed / no bracket / wrapper compat) and `TestUnrecognizedFormatOnRelease` (lands in `extra`, empty when recognised, absent without format detection). - `src/frontend/src/tests/releaseFormats.test.ts`: 3 cases for the new helper. - `ruff check` clean; `pytest tests/prowlarr -m "not integration"` 511 passed; `tsc --noEmit`, `oxlint --deny warnings`, `vitest` all clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
jakesterpdxandClaude Fable 5
|
65e2e3be20 |
feat(audiobook): recognise .mp4 as an audiobook format (#1264)
## Problem Some trackers — MyAnonamouse in particular — distribute AAC audiobooks as per-chapter `.mp4` files. That's the same ISO-BMFF container as `.m4a`/`.m4b`, just with the generic extension (`ftyp isom`, audio-only). Today those releases: 1. show up in Prowlarr search results with **no format chip** — only the generic "Audiobook" icon, because no format could be inferred; 2. download successfully; then 3. fail post-processing with **"No book files found in download"**, because `.mp4` isn't in `AUDIOBOOK_FORMATS` (`shelfmark/core/utils.py`). Real example: MAM #627978, *The Martian* (Andy Weir, 2020 edition) — 142 files `0001 … 0142 Andy Weir (2020) The Martian.mp4` + `cover.jpg`, 305 MB. Every file is a valid AAC-in-MP4 chapter. Adding `mp4` to `SUPPORTED_AUDIOBOOK_FORMATS` in `settings.json` doesn't help since the hard-coded tuple is what post-processing scans against. ## Change - Add `"mp4"` to `AUDIOBOOK_FORMATS` (single source of truth — settings UI, Prowlarr parsing, IRC parser, archive extraction and post-download scan all derive from it), with a comment explaining why. - Add `".mp4"` to the two hand-maintained debrid `_BOOK_EXTENSIONS` lists (AllDebrid / Real-Debrid) so file selection matches. - Slot `mp4` into the IRC `AUDIOBOOK_FORMAT_PRIORITY` table right after `m4a` (same container family). - Update the documented default in `docs/environment-variables.md`. - New regression test `test_audiobook_multifile_mp4_chapters_are_book_files` modelled on the existing multi-file usenet test. ### Note for existing installs The legacy-default migration only widens configs that still hold the old `m4b,mp3` list, so users on the current widened default won't pick up `mp4` automatically — they'll need to tick it in Settings → Audiobook formats. New installs get it by default. Happy to extend the migration if you'd rather it be automatic. ## Testing - `ruff check` / `ruff format --check`: clean - `pytest tests/core tests/config tests/irc tests/prowlarr tests/download -m "not integration and not e2e"`: 2296 passed, new test + `test_audiobook_format_consistency.py` all green. The 10 failures in `test_entrypoint_permissions.py` / `test_orchestrator_stall.py` reproduce identically on untouched `main` on macOS and are unrelated. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
CaliBrain
|
ddc26f01b6 |
fix(download): escalating per-host cooldown on HTTP 429 (#1263)
Anna's Archive 429-throttles the source IP after repeated automated requests. The bypasser could clear the DDoS-Guard challenge but not the 429, so each retry re-solved, re-spawned Chrome, and rotated mirrors that share the same IP - a costly loop that never converged. Add a process-global, per-host cooldown that escalates 2 -> 5 -> 10 -> 15 -> 30 minutes each time a host 429s again after its window elapsed, resetting after a long clear gap. Mirror selection skips cooling hosts and the bypasser refuses to solve one, so a throttled host fails fast instead of storming the solver. |
||
|
CaliBrain
|
95e34670f7 |
fix/group archive extracted audiobooks (#1261)
- fix: group multi-file audiobooks that arrive as an archive - Surface the concrete reason when a direct-download fetch fails |
||
|
CaliBrain
|
7d56624ab6 |
fix: group multi-file audiobooks that arrive as an archive (#1254)
Follow-up to #1237. \`rename_and_group\` only grouped when the source root was a directory, so a multi-file audiobook delivered as a single archive fell through to the flat path: a \`Book.zip\` of twelve chapters landed loose in the destination root with its original chapter names — the layout #1181 is about. The \`is_dir()\` guard was there to keep \`Book.zip/\` from becoming the folder name, but skipping the file case gives up the grouping instead of naming it. A non-directory source can only produce several book files by having been extracted (\`collect_staged_files\` returns a single-element list for every other file shape), so the archive stem is the release name and the suffix is packaging: group under \`Book/\`. Also regenerates the env docs for the new option and gives it the same \"do not use with ingest folders\" caveat Rename and Organize carries, since both now create directories in the destination. Tested: reverting only the source fix makes both new tests fail and the \`rename\` control case pass, so grouping stays opt-in. Full non-e2e suite green (2653 passed). |
||
|
|
f4421ff189 |
fix: preserve multi-file audiobook folders (#1237)
Pass the effective source root from `process_folder_output` into `transfer_book_files`, and have the transfer layer select a sanitized child directory named after that source root when an audiobook has multiple files and its organization mode is `none` or `rename`. Create that grouping directory before applying the existing hardlink/copy/move logic so operation accounting, torrent seeding preservation, collision handling, cleanup, and custom-script final paths continue to use the established production path. Completed multi-file audiobook torrents arrive as a directory whose chapter filenames may not identify the book, but folder output currently sends every discovered chapter directly to the configured destination in `none` and `rename` modes. This flattens chapters from unrelated books together and causes directory-oriented consumers such as Audiobookshelf to interpret individual chapters as separate books. A multi-file audiobook torrent in the default `rename` mode copies or hardlinks all supported chapter files beneath `<destination>/<original torrent directory>/` with their original chapter filenames, and places no chapters directly in the destination root; A multi-file audiobook in `none` mode receives the same source-folder grouping without renaming its chapter files. Fixes #1181 --------- Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
CaliBrain
|
e7007865a4 |
fix(prowlarr): stop turning indexer failures into empty results and 404s (#1251)
Two independent bugs, both from an indexer that Prowlarr proxies rather
than answers for itself: the search never reported that it had failed,
and the grab never resolved what it was handed.
Search. A Torznab search is Prowlarr proxying a live request out to the
tracker, so for a Cloudflare-fronted indexer it waits on FlareSolverr.
The client gave it the 30s budget sized for Prowlarr's own JSON
endpoints, then swallowed every failure -- the timeout, the 429 Prowlarr
returns once it has disabled an indexer, a parse error -- into the same
empty list that means "this indexer has nothing". A cold challenge
routinely runs past a minute, so the UI said "No releases found for this
book" while FlareSolverr was still solving. That empty list also drove
the auto-expand retry, which fires on "no results with the category
filter". A timeout satisfies it, so Shelfmark sent a second search to an
indexer still busy with the first -- two Chromes at once, enough to take
FlareSolverr's down on a small host.
torznab_search now raises ProwlarrSearchError, and an empty list
strictly
means the indexer answered with no matches. The source records which
indexer searches failed: one dead indexer no longer sinks the others,
auto-expand runs only when every indexer genuinely answered, and zero
results with at least one failure raises SourceUnavailableError, which
the releases endpoint already turns into a 503 carrying a real message.
Prowlarr being unreachable was the same lie by another route -- the
indexer list came back empty, leaving nothing to query -- and now says
so.
Indexer searches also get their own timeout, PROWLARR_INDEXER_TIMEOUT,
defaulting to 90s and clamped to 5-300. Prowlarr's status and indexer
list keep 30s so Test Connection stays responsive, and the connect
timeout is split out at 10s so an unreachable Prowlarr fails fast rather
than hanging for the whole read budget. The overall per-request search
budget now scales to twice the indexer timeout, capped at 240s, so
raising the setting is not undone by the cap one level up while staying
under the 300s gunicorn worker timeout.
Grab. Prowlarr hands out a proxy download URL, with no magnetUrl and no
infoHash, for any indexer that only publishes torrent files. The native
Real-Debrid client built its magnet as "if not
url.startswith('magnet:') and expected_hash", so with no hash to work
from it left the URL alone and POSTed it to /torrents/addMagnet as the
magnet field. Real-Debrid answered 404 and the grab died on a raw HTTP
error. AllDebrid carried the same line and the same bug.
Both now resolve the URL first, through the extract_torrent_info path
the
torrent clients have used since #1108: pass a magnet through untouched,
follow a redirect or a response body that turns out to be a magnet,
otherwise upload the fetched .torrent, and fall back to a magnet built
from the infoHash only when the fetch failed. The file is preferred over
a synthesized urn:btih: magnet because it carries the tracker list; a
bare hash leaves the service to find the swarm on DHT alone. Fetches are
shared with the rest of the add path through the torrent fetch cache, so
resolving costs at most one request. Real-Debrid takes the file on PUT
/torrents/addTorrent with the raw bytes as the request body, AllDebrid
on
POST /magnet/upload/file as multipart files[]. A URL that resolves to
neither form now raises before any request reaches the service, so the
user reads why instead of a 404. Neither debrid client had any test
coverage; both have some now.
Fixes #1249
Fixes #1250
|
||
|
CaliBrain
|
5247ec6124 |
fix(bypass): close the gaps a helper that outlives its request opened (#1244)
assumptions the code around it still made were written for a helper that was killed after every request. A bypass that hits the child's deadline is cancelled from the calling thread, which returns the moment the cancellation is scheduled - so the helper went on to serve the next request while the abandoned one was still closing its browser, on the same loop, sharing the DISPLAY globals and one process group. The deadline now lives inside the loop, where asyncio.wait_for() waits for the unwind before it raises, with the calling thread keeping a bounded backstop in case the cleanup wedges too. Both budgets are set so the child still answers before the parent gives up on it. The helper's cookie store survived the request as well, and the whole of it is exported back to the parent on every answer - so clearance the parent had purged for one host came back the next time some other host was solved, the dead-cookie resurrection _redirect_loop_handoff purges to avoid. The child starts each request from an empty store again; the parent already runs the cached-cookie check against a superset of it. DNS config is compared against what the helper is actually resolving through rather than skipped whenever the parent reports "auto", so a user flipping CUSTOM_DNS back to auto - which applies live - reaches a warm helper instead of leaving it on an abandoned DoH resolver. The 15s exit grace is now asked only of a helper that can still read its stdin. One dropped mid-bypass never returns to that read, so the grace could only end in the kill - while a user cancelling a download, and every bypass queued behind them on LOCKED, waited it out. Result files are cleaned on the timeout and cancellation paths too, staging file included, rather than only when the answer was read. |
||
|
CaliBrain
|
bd21ec1257 |
fix(audiobookbay): search the ASCII punctuation ABB actually stores (#1242)
WordPress texturizes punctuation on output only, so a post stored as "The Stranger's Wife" renders as "The Stranger’s Wife". ABB's search matches the stored value and ANDs its terms, so one typographic character in the query empties the entire result set rather than merely ranking worse. Book metadata and mobile keyboards both hand us those characters. Map curly quotes, dashes and ellipses to ASCII before a query goes out, and on both sides of the relevance comparison, since scraped titles carry the rendered forms. Release titles are still stored and displayed exactly as ABB renders them; only matching normalizes. Also percent-encode the search query properly. The hand-rolled encoder only escaped double quotes and spaces, so a bare "&" started a new query parameter and silently truncated the search: "detective dan riley books 1 & 2 weatherley" reached ABB as "detective dan riley books 1" and returned six confident-looking results without the requested book among them. "%" and "+" were mangled too. |
||
 helgehelge123andhelgehelge123
|
7b9c416df8 |
perf(bypass): keep the helper subprocess alive between bypasses (#1222)
Every protected request spawns a fresh helper subprocess, paying interpreter start and imports before any work begins. Measured inside the container, five consecutive runs of `python -c "import shelfmark.bypass.internal_bypasser"`: ``` 3.53s 3.45s 3.55s 3.54s 3.46s ``` A single search issues several protected requests, so that is paid several times over per search. ## What changed The helper now serves one JSON request per line of stdin until the parent closes the pipe, and an idle timer (`BYPASS_BROWSER_IDLE_TIMEOUT`, default 180s) shuts it down once searching stops. Answers still travel by result file, but the file is now written to a `.part` path and renamed into place — the parent treats the file's existence as the answer, so it must never observe a half-written one. stdout and stderr stay attached to the parent's, so helper logs keep appearing in `docker logs` exactly as before. Failure handling, since a warm helper is exposed to more of it than a per-request one ever was: | Situation | Handling | | --- | --- | | Helper died between requests | Detected via `poll()`, respawned | | Pipe broken at write time (`poll()` can miss this) | One retry on a fresh process; a fresh one failing there is a real failure | | Helper exits without writing a result | `RuntimeError` naming the exit code | | Wedged past the timeout, or cancelled mid-bypass | Helper killed, then `_cleanup_orphan_processes` because a killed helper never got to close Chrome | | Idle reaper racing an arriving request | Re-checks the deadline under the lock and re-arms instead of killing a helper that just did work | The DNS config now travels with every request rather than only at spawn: a warm helper outlives changes the parent makes to its provider. ## `BYPASS_REUSE_BROWSER`, off by default This parks the CDP driver between bypasses. A driver's websockets are bound to the loop that opened them and cannot outlive their process, so the persistent helper is what makes this possible at all — and the warm path runs on `_CDP_WORKER`'s long-lived loop rather than `asyncio.run` for the same reason. The mechanism works. With it on, the browser start disappears from the second request onward: 0.7s from `Reusing warm Chrome browser` to the first bypass attempt, against roughly 16s cold. **It still ships off, because a matched-pair test shows it is a net loss against DDoS-Guard.** Each round primed with one cold bypass, waited 10s, then measured a second — identical timing in both arms, only the browser strategy differing, order balanced (fresh, warm, warm, fresh) so drift over the session cannot masquerade as an effect: | Arm | Measured request | | --- | --- | | fresh browser | 42.8s, 40.6s | | warm browser | 57.1s, 59.6s | Spread within each arm is 2.2s and 2.5s, against 16.7s between them. Reuse removes the ~15s browser start and then gives back roughly twice that in solving: a returning browser draws a harder challenge. Where the cold browser is through on the second bypass method, the warm one fails the first three and only `_bypass_method_humanlike` gets it, at ~30s for that method alone. Worth separating from a second effect I ran into while measuring: five back-to-back searches slow from ~32s to 51–98s with reuse **disabled** as well, so DDoS-Guard escalates on request rate independently of any of this. That is why the pairs above are timed identically rather than simply run in sequence. It is the larger of the two effects, but not something this project can patch around. Reuse is left available rather than dropped because Cloudflare sites may not respond the same way, and because the two concerns are independent: the helper start is pure overhead and always worth removing, the browser is not. ## Verification - 2559 unit tests pass (2542 before, 17 added in `tests/bypass/test_warm_browser.py`) - `ruff check`, `ruff format`, `basedpyright` over backend and tests, and `vulture` all clean - `docs/environment-variables.md` regenerated via `scripts/generate_env_docs.py` - Live against Anna's Archive on a warm helper: searches return their usual ~760KB and 667 results, the app's own search warm-up completes with 50 results, and the container is left with no orphan chrome/Xvfb/ffmpeg processes Happy to drop the `BYPASS_REUSE_BROWSER` half entirely if you would rather not carry a default-off path — the helper persistence stands on its own. Co-authored-by: helgehelge123 <helge.neumann@zollsoft.de> |
||
|
CaliBrain
|
646b531669 |
fix(hardcover): accept the short hc_pat_ keys Hardcover issues now (#1241)
Hardcover replaced its ~500 char JWTs with short opaque personal access
tokens ("hc_pat_..."), and the connection test rejected anything under
100 chars before a request ever left Shelfmark, so every newly created
key failed with "API key seems too short".
The length floor now applies only to keys without the hc_pat_ prefix; a
prefixed key goes straight to Hardcover, which is the authority on
whether it is valid. Also strip a pasted "bearer " prefix regardless of
casing -- Hardcover's docs tell users to paste the token into an
"authorization" header, so the prefix rides along on the copy, and the
old case-sensitive removeprefix() sent it through as part of the token.
The API key field now names the expected shape.
Note that Hardcover's PAT path currently answers every hc_pat_ token
with a 500, a fabricated one included, while non-PAT tokens still get a
clean 401. So a new key cannot connect yet regardless of this change --
that failure is server-side and not something this code can reach.
Refs #1240
|
||
|
CaliBrain
|
7193036626 |
fix(rtorrent): apply the audiobook label to audiobook downloads (#1239)
add_download() picks self._audiobook_label from a content_type kwarg, but the only call site never passed one, so is_audiobook was always False and every download got RTORRENT_LABEL. category does not fill the gap: _get_category_for_task() returns None for rTorrent, which has no category concept, leaving content_type as its only audiobook signal. Pass task.content_type through from base_handler, and match it with the shared is_audiobook() helper instead of == "audiobook". normalize_content_type() treats "book (audiobook)" as an audiobook, so the exact-string check would have mislabeled that value even once it arrived. The existing rTorrent tests passed content_type straight to the client, which is why nothing caught the missing wiring; the new handler test covers the call site itself. Post-processing was never affected: destination.py reads task.content_type directly, so files already landed in DESTINATION_AUDIOBOOK correctly. Fixes #1235 |
||
|
CaliBrain
|
12d554a92f |
fix(download): hand a 503 carrying a challenge to the bypasser (#1238)
503 is in RETRYABLE_CODES, and the bypasser is only ever reached from the 403 branch and the AA redirect-loop rescues. Once Z-Library re-serves its DDoS-Guard interstitial with the same cookie the #1188 handshake already echoed back, the request has nothing left to try and spends every attempt on the same wall. Gate the handoff on the response body rather than the status, so a genuine overloaded-origin 503 keeps its retry path, and on allow_bypasser_fallback, so best-effort fetches still fail fast. The challenge indicators move out of internal_bypasser into shelfmark/bypass/challenge.py so http.py can use them without importing SeleniumBase, which is lazily imported precisely because it is optional. Refs #1233 |
||
|
CaliBrain
|
fae6140c6a |
fix(bypass): scope browser cleanup to the calling session (#1232)
The orphan sweep ran a container-wide 'pkill -9 -f chrome|chromium|Xvfb|ffmpeg', so it also matched browsers another bypass was still driving. Scope it by process group: kill only our own group and groups whose leader has died. Spawn the helper with start_new_session so its browser tree is identifiable, tear the whole group down after every run (a timed-out helper used to leak its Chrome and Xvfb), and have an orphaned helper take its browser down with it. Fixes #1231. |
||
|
CaliBrain
|
4cd1091d16 |
fix(hardcover): send the field count Hardcover's Book search requires (#1224)
Advanced title search, advanced title+author search, and the title typeahead returned zero results every time, and the sort fallback added in #1183 blamed the sort value for it. Hardcover turns the `fields` search parameter into Typesense's `query_by` but keeps `num_typos` and `query_by_weights` as fixed-length presets per query_type. For query_type=Book the preset expects exactly five fields, so a shorter list is not searched loosely - the whole search is rejected with a null results body. Confirmed against the live API: 1, 2, 3, 4 and 6 fields are all rejected, only 5 works, and weights must match one-for-one when sent. Every Book-type list we sent was the wrong length - the title typeahead and advanced title search sent 2, title+author sent 3. - Send BOOK_SEARCH_FIELDS (the full five) for every narrowed Book search and express the intent through weights instead. Weights only bias ranking - a field weighted 0 still matches - so a title search now ranks titles first rather than restricting to them. That is the closest behaviour Hardcover still allows, and there is no client-side filter to restore the old precision. - Pin the field and weight counts in tests, since the failure mode is a silent zero results rather than an error. The sort fallback from #1183 also misread these rejections: - Select the `error` field on every search and log Hardcover's own explanation. The reason is only ever in that sibling field, so a rejection surfaced as "returned no result body" with nothing to act on. Reading it is what made the field-count rule findable. - Drop `sort` entirely on the retry instead of sending an empty string. An empty sort is a value like any other and can be rejected too. - Arm the 900s sticky window only after the sortless retry succeeds. It was armed before the retry and never rolled back, so one rejected typeahead disabled sorting process-wide for 15 minutes whatever the actual cause. Verified against the live Hardcover API: advanced title search 0 -> 84 results, title+author 0 -> 139, title typeahead 0 -> 84 with the exact title top. 2566 unit tests pass; ruff, basedpyright and vulture clean. Refs #1183. The sort_by regression #1183 was written for is gone from Hardcover's side - every sort value it rejected, including the one in the report, is accepted again today. Two plain-search rejections in that report (fields=None) remain unexplained: they could not be reproduced under any per_page, page depth, sort value or query shape, and are most likely transient upstream. They now self-report the reason if they recur. |
||
|
CaliBrain
|
651096ed7b |
fix(bypass): reuse external bypasser clearance instead of re-solving (#1223)
Direct download was unusable behind an external bypasser (FlareSolverr /
Byparr): every request paid a 403 plus a full solve, and a search that
never ran was reported to the user as "No books found".
Clearance was discarded on the external path. get_cf_cookies_for_domain
and get_cf_user_agent_for_domain returned {} / None whenever
USING_EXTERNAL_BYPASSER was set, and _fetch_via_bypasser read only
solution.response - dropping solution.cookies and solution.userAgent,
which FlareSolverr-compatible services do return. A solve therefore
cleared the one request that paid for it and nothing else, and a file
download - which the solver cannot proxy, being binary - presented no
clearance at all. Diagnosed from a v1.3.9 debug bundle: ~35s in the
bypasser per search, on every search.
- Move the cookie jar out of internal_bypasser into bypass/cookie_store.
internal_bypasser imports seleniumbase at module scope, which is the
dependency an external-bypasser deployment is entitled not to have, so
it cannot host a store the external path depends on.
- Harvest solution.cookies and solution.userAgent after a successful
solve. The existing filtering applies unchanged, so the per-check
__ddg8_/__ddg9_/__ddg10_ trio is still dropped and the external path
cannot reintroduce the ?check=1 loop fixed in
|
||
|
CaliBrain
|
ebb833a82c |
fix(bypass): discard rejected DDoS-Guard cookies instead of replaying them (#1221)
A cookie that has been rejected was kept and presented again on every
later
request, so a single bad clearance could re-arm the challenge
indefinitely.
Cookie storage:
- Enforce expiry for every stored cookie, not just cf_clearance.
DDoS-Guard
domains have no cf_clearance, so the existing check never fired for them
and
expired cookies were replayed forever.
- Stop storing the per-check cookies __ddg8_/__ddg9_/__ddg10_ and
ddg_last_challenge. Captured live from Anna's Archive, these carry the
client
IP and the timestamp the check was issued (~40 min), versus ~1 year for
the
__ddg1_/__ddg2_/__ddgid_ clearance. Replaying an IP-bound token stops
describing the caller as soon as the egress IP changes, which is routine
behind a VPN.
Failure handling — every path that is rejected while carrying cookies
now
purges them, not just the redirect loop:
- 403 returned while presenting cookies.
- Cached-cookie attempt rejected, whether by status or by redirect loop.
- Factored the purge into _purge_clearance, guarded on a non-empty
hostname
since clear_cf_cookies("") means "every host" and would wipe clearance
for
sites that are working fine.
Also fix the search warm-up switches shipped inert in v1.3.8:
SEARCH_WARMUP_ENABLED and SEARCH_WARMUP_QUERY are not in the settings
registry, and config.get only consults the environment for keys it
knows, so
both always returned their defaults — the warm-up could not be turned
off or
retargeted. Read os.environ first.
Refs #1220. Deliberately not "Fixes": the reported failure could not be
reproduced on v1.3.8 from a stable IP (the reporter's own queries all
returned
200 on both the pre- and post-change builds), and the new purge paths
did not
fire in live testing because the failures arrive as redirect loops,
which were
already purged. These are correctness fixes with no measured effect on
that
issue. The underlying problem remains that Chrome-obtained cookies never
satisfy DDoS-Guard when replayed by requests, so every search still
re-solves.
Verified: 2542 unit tests pass; ruff, basedpyright and vulture clean;
e2e
platform baseline (10), full (6) and bypasser-external (5) all pass;
five
sequential live searches against Anna's Archive all returned 200 with
zero
"Exceeded 30 redirects".
|
||
|
CaliBrain
|
b656f019be |
feat(download): add DoH wireformat support, mirror quarantine, and search warmup (#1218)
- Add RFC 8484 DNS wireformat codec and HTTP/2 support (httpx) for Quad9/OpenDNS DoH providers. - Quarantine dead, parked, or seized mirrors for the session on hard failure (DNS errors, connection refused, 410/451, parked pages) while preserving bypass clearance on live mirrors. - Add background startup search warmup to prime DNS, elect mirrors, and pre-solve protection challenges to eliminate cold-start search latency. - Add comprehensive test suites for DoH wireformat, mirror quarantine, parked domain detection, and search warmup. |