mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-09-24 20:10:28 +01:00
c95ee72ad50af437823e25ea1ad5511ab5815af1
177
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jorge Lima
|
c95ee72ad5 |
fix(qbittorrent): keep magnets whose metadata is still pending (#1282)
## Problem
`QBittorrentClient.add_download()` waits 20 × 0.5 s for qBittorrent to
leave `metaDL`, then raises:
```
Failed to add to qbittorrent: Torrent metadata resolution was not confirmed within the visibility grace period
(response=TorrentsAddedMetadata({'added_torrent_ids': [], 'failure_count': 0, 'pending_count': 1, 'success_count': 0}))
```
The wait exists to learn qBittorrent's primary torrent ID, which for
hybrid torrents switches from the v1 hash to the truncated v2 hash once
metadata resolves. A magnet on a thin public swarm routinely needs
longer than 10 s to find a peer that will serve metadata, and the
download is then abandoned even though the add itself succeeded. The
torrent stays in qBittorrent (`base_handler` logs "leaving in
qbittorrent") and often completes minutes later with nobody watching it.
Seen on v1.3.12 with public indexers through Prowlarr: every magnet-only
release failed this way, while `.torrent` releases from a private
indexer were fine. qBittorrent showed the same torrents at `metaDL 0%
seeds=0/0`, and they resolved on their own well after shelfmark had
given up.
## Change
Return the info hash we already have instead of raising when the grace
period expires. Reads then resolve either identity:
- `get_status()` and `get_download_path()` use `_resolve_torrent()`
instead of `_get_torrent_info()`, so a v1 hash still matches after
qBittorrent re-keys the torrent to v2. `_torrent_matches_download_id`
already compares `hash`, `infohash_v1` and `infohash_v2`.
- `remove()` and `set_category()` address the torrent by its current
primary hash through a new `_current_hash()` helper, which falls back to
the ID it was given when the torrent cannot be resolved.
- The two magic numbers become `_METADATA_WAIT_POLLS` and
`_METADATA_WAIT_INTERVAL_SECONDS`.
The happy path does not change. When metadata resolves inside the grace
period the resolved primary hash comes back as before, and
`_resolve_torrent()` tries the exact-hash lookup first, so it costs no
extra request.
## Tests
`test_add_fails_when_metadata_never_resolves` asserted the old
behaviour, so it becomes
`test_add_keeps_torrent_when_metadata_never_resolves` and asserts the
info hash is returned.
`test_get_status_resolves_hash_after_metadata_switch` is new: it reads
status by the v1 hash after qBittorrent reports the torrent under its v2
hash.
`uv run pytest tests/ --ignore=tests/e2e` gives the same 55 failures
with and without this change (they are all in `tests/bypass/` and need
Chrome, which my machine has no headless setup for), and
`tests/prowlarr/` is green at 524 passed. Ruff check and format are
clean. I have not run this branch against a live qBittorrent, so a
second pair of eyes on the `remove()` path would help.
|
||
 CaliBrain
|
b25acdb2ad |
fix(packs): don't disrupt normal downloads when inspecting for packs (#1274)
Follow-ups to the multi-book pack feature (#1270), which inspects every release before download. Two behaviours leaked into the ordinary single-book flow and are corrected here: - A flat folder of chaptered audio (`01 - Chapter.mp3`, `02 - ...`) was detected as a pack, because each track name parses to a series position, so clicking download popped the review panel for one normal audiobook. Flat folders are now split one-book-per-file only with real evidence of distinct books: two or more series positions, more than one title, and no chaptered audio (only the single-file m4b/m4a containers and ebook formats qualify). Subfolder packs and flat m4b/m4a packs are unchanged. - Every release that couldn't be inspected (usenet, magnet-only, sources without a list_files hook, ABB single-file) showed an info toast on download. That is now a console.warn, so a normal download is silent again. Adds regression tests for the chaptered-mp3 cases. |
||
 Lance Marks
|
f441b85da2 |
feat(packs): inspect multi-book releases and file each book separately (#1270)
## Multi-book packs: inspect a release before download and file each book separately Closes #576 ### Problem One queued release is always treated as one book. When a torrent is actually a whole series (`Series/Book 1 - Title/…`, or a flat folder of `Series 1.0 - Title.m4b` files), post-processing walks the whole tree, flattens every file into one list and renames them `Title - 01…10` under the searched book's `{Author}/{Title}`. Audiobookshelf then sees a single 10-file "book" and the user has to re-file everything by hand. ### What this does Most releases expose their file list *before* anything is downloaded, so the split is decided up front and approved by the user, then the download is fire-and-forget: 1. **Inspect** – clicking a release's download button now calls `POST /api/releases/inspect` first. A new optional `DownloadHandler.list_files(release_data)` hook returns the release's files without downloading: - **AudiobookBay** reads the torrent file table off the detail page it already fetches (the page is now cached for 120 s, so inspect + download cost ABB one request). - **Prowlarr** parses `info.files` from the `.torrent` it already fetches (the existing 120 s torrent-fetch cache is reused). Magnet-only and usenet releases report "can't inspect". - Other sources default to `None`. 2. **Review** – if the plan contains more than one book, the Find Releases modal swaps the list for a review panel: one row per book with editable title / series position / year, expandable file lists, non-book sidecars (`.txt`, covers) shown as ignored, a "Treat as a single book" switch, and **Download N books**. Single-book releases queue immediately, exactly as before. 3. **File** – the approved plan travels with the task (`DownloadTask.book_plan`, retry-safe) and post-processing files each book through the existing transfer code, one book at a time (`dataclasses.replace(task, title=…, series_position=…, year=…)`), so organize/rename templates, part numbering (now scoped per book), hardlinks, torrent copy-preserve and usenet handling are unchanged. Status reads `Complete (N books, M files)`. 4. **Fallback** – when a release can't be inspected the user gets a toast, and a small "Multi-book pack" toggle in the modal header forces a heuristic split (subfolder = book, or one book per file when the file names carry series positions). Planning lives in `shelfmark/download/postprocess/packs.py` and is shared by the inspect endpoint and post-processing, so what the user approved is what gets filed. The name parser strips `Book 3 -`, `03 -`, `1.0 -`, `3.`, `[03]`, `#3`, a leading series name, labels like "An Expanse Novella -", repeated titles (`Gods of Risk 2.5 - Gods of Risk`) and a trailing `(Year)`; author and series name come from the book that was searched, and the searched book's own series position is never applied to its siblings. ### Files - `shelfmark/download/postprocess/packs.py` (new) – `PackFile/PackBook/PackPlan`, `plan_pack`, `parse_pack_book_name`, `group_files_into_books`, `match_plan_to_files` - `shelfmark/core/release_inspect_routes.py` (new) – `POST /api/releases/inspect` - `shelfmark/release_sources/__init__.py` – `DownloadHandler.list_files` hook - `shelfmark/release_sources/audiobookbay/{scraper,handler}.py` – detail-page cache, `extract_file_list`, `list_files` - `shelfmark/release_sources/prowlarr/handler.py`, `download/clients/torrent_utils.py` – `extract_file_list_from_torrent`, `list_files` - `shelfmark/core/models.py`, `download/orchestrator.py` – `multi_book` / `book_plan` fields, queue + retry serialization - `shelfmark/download/postprocess/transfer.py`, `pipeline.py`, `outputs/folder.py` – per-book transfer branch and status message - `src/frontend`: `components/PackReviewPanel.tsx` (new), `ReleaseModal.tsx`, `App.tsx`, `services/api.ts`, `types/index.ts`, `utils/releasePayload.ts` (payload builder moved out of `App.tsx`), `utils/packReview.ts` - `docs/dev/release-sources-plugin-guide.md` – documents the `list_files` hook ### Out of scope (follow-ups) - Listing files from an NZB (Shelfmark already fetches the bytes; `<file subject>` names are noisy) - Inspecting magnet links via qBittorrent's files API after a paused add - BookLore / email outputs (they ignore `book_plan`; noted in code) - The combined ebook + audiobook flow ### Testing **Automated** (`make checks`, `make python-test`, `make frontend-test` all green; the only failures on my machine are the pre-existing `tests/config/test_entrypoint_permissions.py` cases, which need bash ≥ 4 and fail identically on `main` under macOS bash 3.2): - `tests/download/test_packs.py` – name parsing (markers, series name, novella labels, repeated titles, bare numeric titles like `1984`), nested / flat / mixed / deeper-nested packs, single wrapping folder not treated as a pack, plan-to-disk matching with basename fallback - `tests/core/test_processing_packs.py` – full `post_process_download` runs on a real temp filesystem: approved plan files each book under its own `{Author}/{Title}`, heuristic split of a nested pack, searched book's series position does not leak, multi-file book inside a pack keeps `- 01/- 02` per book, hardlinked torrent pack leaves the seeding tree intact, no pack fields ⇒ behaviour unchanged, single group degrades to the searched title, status message - `tests/core/test_release_inspect_routes.py` – plan response, not-inspectable, handler errors never 500, unknown source / missing `source_id` ⇒ 400, login required - `tests/audiobookbay/test_file_list.py` – file-table scraping from real ABB markup (multi-file and single-file pages), handler host validation, one page fetch shared by magnet + file list - `tests/prowlarr/test_torrent_file_list.py` – multi-file / single-file `.torrent` parsing, handler behaviour for torrent URL vs magnet vs usenet vs cache miss - `tests/download/test_orchestrator_pack_fields.py` – queue-time parsing and retry round-trip - Frontend: `releasePayload.test.ts`, `packReview.test.ts` (vitest) **Manual, on a real deployment** (arm64 image built from this branch, run as a side container next to production with the same qBittorrent / Audiobookshelf setup, `FILE_ORGANIZATION_AUDIOBOOK=organize`, hardlinks on): - AudiobookBay "The Expanse Complete 2.0" (7.87 GB, 36 files): clicking download opened the review panel in ~1 s showing **18 books · 18 files · 18 files ignored** (the `.txt` sidecars), with series positions 0.1–9.5 and years parsed from the file names; novella labels stripped ("The Churn", "The Butcher of Anderson Station"). Editing a title in the panel works. Confirming queued one task; the magnet resolved from the cached page in ~30 ms; after the download the task reported `Complete (18 books, 18 files)`, 18 hardlinks landed as `audiobooks/James S. A. Corey/<Title>/<Title>.m4b`, the torrent kept seeding, and Audiobookshelf scanned each folder as its own book (title, author, embedded chapters). - A second pack ("Expanse [01 - 9.5]", `Title N - Title` naming) was inspected to verify the repeated-title rule and the Back button, without downloading. - Single-book releases still queue immediately with no extra UI. |
||
  zab1996andRyan
|
02b7e9d958 |
feat(newznab): support multiple named indexers (#1271)
## Summary - add a named Newznab indexer table with per-indexer URL and API key settings - search every configured indexer and retain the originating indexer name on each result - namespace cached release IDs across connections and isolate individual indexer failures - preserve the legacy single-indexer settings as a fallback - support masked API-key cells and trusted SABnzbd prefetching for named indexers ## Validation - 121 Newznab and SABnzbd backend tests passed on Python 3.14 - Ruff passed for all changed Python files - frontend TypeScript and strict lint checks passed - all 134 frontend unit tests passed - frontend formatting check passed ## Compatibility Existing `NEWZNAB_URL` and `NEWZNAB_API_KEY` configurations continue to work whenever `NEWZNAB_INDEXERS` is empty. Co-authored-by: Ryan <zab1996@users.noreply.github.com> |
||
|
CaliBrain
|
ff06a1a581 |
fix(search): follow-ups to per-user book languages (#1267)
Review follow-ups to #1255, all in the code that PR touched. Drop the dead user_id from the Prowlarr retry path. ProwlarrSource.search never reads plan.languages, and _refresh_release builds a synthetic book with no titles_by_language, so the title variants came out identical with and without it. It also should not language-filter: it re-finds one exact release by its guid. Pin the tab move in tests. BOOK_LANGUAGE moved from the General tab to Search Mode with no migration, which only works because both tabs persist into the same settings.json. Nothing asserted that, so splitting the files later would silently reset every install to ["en"]. Covers the stored value, a fresh install, and ENV precedence. Stop the UI inventing a default language. An empty BOOK_LANGUAGE is a deliberate "no default filter" that the backend preserves, but the two frontend call sites replaced it with the first supported language, so the filter said English where the server filtered nothing. resolveDefaultLanguageCodes now falls back only when the value is absent. Keep the normalized value for every validated search key. validate_user_settings gated the write-back on a hand-maintained subset of the keys the search validator recognises, so METADATA_PROVIDER_COMBINED, SHOW_COMBINED_SELECTOR and FORCE_COMBINED_SEARCH were validated and then stored raw -- a padded provider name was accepted and persisted with its padding. Reuse the validator's own key set instead. Skip blank language entries rather than rejecting them, so "" and "en," mean the same as [] and ["en"] instead of erroring on an unnamed language. Extract resolveListOverride for the list-override detection that was copy-pasted between the two user-settings sections, and mention languages in the Search Preferences section description. |
||
 
|
463ef49ac3 |
feat(search): let each user pick their own default book languages (#1255)
## Why `BOOK_LANGUAGE` is a per-reader property, not a per-instance one. On a shared install one household member searches in German while another wants English and German — today whoever changes the setting changes it for everyone, and the only escape is re-picking languages in the filter on every single search. The per-user override machinery already carries `SEARCH_MODE`, the metadata providers and the default release sources, so the language default mostly had to opt into it. ## What changed **The field.** `BOOK_LANGUAGE` becomes `user_overridable` and moves from the **General** tab to **Search Mode**, next to the other user-overridable search defaults (per [review](https://github.com/calibrain/shelfmark/pull/1255#issuecomment-5391189094) — the first version had the Search section span two tabs, this one doesn't). Admins set it per user in the user editor, users set it in **My Account → Search Preferences**, and the Search Mode tab carries the usual "N users override this" summary. **No migration for the move.** `general` and `search_mode` both persist into `settings.json`, and a field's value is resolved through `load_config_file(tab)` for the tab it's declared on — so an install that already stores `BOOK_LANGUAGE` keeps its value. Checked against a `settings.json` written while the field still lived on General: the stored value resolves unchanged, a fresh install still gets `["en"]`, and `BOOK_LANGUAGE` in the environment still overrides both. **The two places the default is read.** - `/api/config` seeds the frontend's language filter, so it now resolves `BOOK_LANGUAGE` for the session user. - `build_release_search_plan` falls back to the default whenever a request carries no language filter — which is exactly what the filter's "Default" option sends. It takes an optional `user_id`, passed by `/api/releases` from the session and by the Prowlarr retry path from `task.user_id`, so a retry re-searches in the languages of whoever queued the download. **Validation.** Overrides go through `normalize_language()`, so `"German"`, `"ger"` and `"de"` all store as `de`, and an unknown language is rejected with a message naming it instead of being silently searched for. An empty list stays an empty list (a deliberate "no default filter"), `null` clears the override as everywhere else, and ENV still wins: with `BOOK_LANGUAGE` set in the environment the field reports `fromEnv` and overrides are ignored. **Scope.** Only the language default becomes overridable. The two format lists left behind under "Default Search Filters" stay admin-only — they describe what the library and its post-processing accept, not what a reader wants to read. There's a test pinning that. ## Verification - 2681 unit tests pass (2670 before, 11 added) - `ruff check`, `ruff format`, `basedpyright` over backend and tests, and `vulture` all clean; frontend lint, format, typecheck and 126 unit tests clean - `docs/environment-variables.md` regenerated via `scripts/generate_env_docs.py` (the `BOOK_LANGUAGE` row follows the field into the Search Mode section) - Manually against a two-user instance with builtin auth (first round, before the tab move): with user A on German and user B on English+German, `/api/config` returns each reader their own `default_language` and an unfiltered `/api/releases` plans the matching languages; an admin can set and read the same override for another user; clearing it falls back to the global value; a stray `"klingon"` is rejected; and `BOOK_LANGUAGE` in the environment overrides both users with the field marked `fromEnv` - After the tab move I re-ran the suites above plus the stored-value/fresh-install/ENV check described under "No migration for the move"; the behaviour it exercises is what the move could have broken Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
CaliBrain
|
a5595cf9f1 | Change test for fake extension that wont work (#1266) | ||
 jakesterpdxandClaude Fable 5
|
9bcf595111 |
feat(prowlarr): warn when an indexer declares a format Shelfmark can't process (#1265)
## Problem Companion to #1264, but general rather than mp4-specific. MyAnonamouse titles carry a structured `[LANG / FORMATS]` bracket that `_extract_mam_formats` parses. When every token in it is something Shelfmark doesn't know — e.g. `The Martian by Andy Weir [ENG / MP4]` — the release is rendered with **no format chip at all**, just the generic headphones/book icon with an "Audiobook" tooltip. To a user that looks like an ordinary result. It downloads fine and then fails post-processing with *"No book files found in download"*. The backend already *had* the signal (a format token it couldn't map); it just threw it away. ## Change **Backend** (`shelfmark/release_sources/prowlarr/source.py`) - `_split_mam_formats(raw_title) -> (recognized, unrecognized)` replaces the body of `_extract_mam_formats`, which is kept as a thin wrapper returning `recognized` so nothing else changes. - Releases gain `extra["unrecognized_formats"]` (list, or `None` when empty / when format detection is off). **Frontend** - `getUnrecognizedReleaseFormats(release)` in `utils/releaseFormats.ts` (normalised + deduped, same shape as `getReleaseFormats`). - `ReleaseCell` `format_content_type`: when there is **no** recognised format but the indexer named one, render an amber `MP4 Unsupported` badge (compact view: amber `MP4`) with tooltip *"Unsupported format (MP4) - Shelfmark cannot process this release"*. When a recognised format exists the existing badge is untouched, even if extra unknown tokens were present. Only the chip changes — the download button still works, so a user can still grab and hand-process the files if they want to. Happy to disable the button instead if you'd prefer. ## Tests - `tests/prowlarr/test_source.py`: `TestSplitMamFormats` (recognised / unrecognised / mixed / no bracket / wrapper compat) and `TestUnrecognizedFormatOnRelease` (lands in `extra`, empty when recognised, absent without format detection). - `src/frontend/src/tests/releaseFormats.test.ts`: 3 cases for the new helper. - `ruff check` clean; `pytest tests/prowlarr -m "not integration"` 511 passed; `tsc --noEmit`, `oxlint --deny warnings`, `vitest` all clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
jakesterpdxandClaude Fable 5
|
65e2e3be20 |
feat(audiobook): recognise .mp4 as an audiobook format (#1264)
## Problem Some trackers — MyAnonamouse in particular — distribute AAC audiobooks as per-chapter `.mp4` files. That's the same ISO-BMFF container as `.m4a`/`.m4b`, just with the generic extension (`ftyp isom`, audio-only). Today those releases: 1. show up in Prowlarr search results with **no format chip** — only the generic "Audiobook" icon, because no format could be inferred; 2. download successfully; then 3. fail post-processing with **"No book files found in download"**, because `.mp4` isn't in `AUDIOBOOK_FORMATS` (`shelfmark/core/utils.py`). Real example: MAM #627978, *The Martian* (Andy Weir, 2020 edition) — 142 files `0001 … 0142 Andy Weir (2020) The Martian.mp4` + `cover.jpg`, 305 MB. Every file is a valid AAC-in-MP4 chapter. Adding `mp4` to `SUPPORTED_AUDIOBOOK_FORMATS` in `settings.json` doesn't help since the hard-coded tuple is what post-processing scans against. ## Change - Add `"mp4"` to `AUDIOBOOK_FORMATS` (single source of truth — settings UI, Prowlarr parsing, IRC parser, archive extraction and post-download scan all derive from it), with a comment explaining why. - Add `".mp4"` to the two hand-maintained debrid `_BOOK_EXTENSIONS` lists (AllDebrid / Real-Debrid) so file selection matches. - Slot `mp4` into the IRC `AUDIOBOOK_FORMAT_PRIORITY` table right after `m4a` (same container family). - Update the documented default in `docs/environment-variables.md`. - New regression test `test_audiobook_multifile_mp4_chapters_are_book_files` modelled on the existing multi-file usenet test. ### Note for existing installs The legacy-default migration only widens configs that still hold the old `m4b,mp3` list, so users on the current widened default won't pick up `mp4` automatically — they'll need to tick it in Settings → Audiobook formats. New installs get it by default. Happy to extend the migration if you'd rather it be automatic. ## Testing - `ruff check` / `ruff format --check`: clean - `pytest tests/core tests/config tests/irc tests/prowlarr tests/download -m "not integration and not e2e"`: 2296 passed, new test + `test_audiobook_format_consistency.py` all green. The 10 failures in `test_entrypoint_permissions.py` / `test_orchestrator_stall.py` reproduce identically on untouched `main` on macOS and are unrelated. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
CaliBrain
|
ddc26f01b6 |
fix(download): escalating per-host cooldown on HTTP 429 (#1263)
Anna's Archive 429-throttles the source IP after repeated automated requests. The bypasser could clear the DDoS-Guard challenge but not the 429, so each retry re-solved, re-spawned Chrome, and rotated mirrors that share the same IP - a costly loop that never converged. Add a process-global, per-host cooldown that escalates 2 -> 5 -> 10 -> 15 -> 30 minutes each time a host 429s again after its window elapsed, resetting after a long clear gap. Mirror selection skips cooling hosts and the bypasser refuses to solve one, so a throttled host fails fast instead of storming the solver. |
||
|
CaliBrain
|
95e34670f7 |
fix/group archive extracted audiobooks (#1261)
- fix: group multi-file audiobooks that arrive as an archive - Surface the concrete reason when a direct-download fetch fails |
||
|
CaliBrain
|
7d56624ab6 |
fix: group multi-file audiobooks that arrive as an archive (#1254)
Follow-up to #1237. \`rename_and_group\` only grouped when the source root was a directory, so a multi-file audiobook delivered as a single archive fell through to the flat path: a \`Book.zip\` of twelve chapters landed loose in the destination root with its original chapter names — the layout #1181 is about. The \`is_dir()\` guard was there to keep \`Book.zip/\` from becoming the folder name, but skipping the file case gives up the grouping instead of naming it. A non-directory source can only produce several book files by having been extracted (\`collect_staged_files\` returns a single-element list for every other file shape), so the archive stem is the release name and the suffix is packaging: group under \`Book/\`. Also regenerates the env docs for the new option and gives it the same \"do not use with ingest folders\" caveat Rename and Organize carries, since both now create directories in the destination. Tested: reverting only the source fix makes both new tests fail and the \`rename\` control case pass, so grouping stays opt-in. Full non-e2e suite green (2653 passed). |
||
 
|
f4421ff189 |
fix: preserve multi-file audiobook folders (#1237)
Pass the effective source root from `process_folder_output` into `transfer_book_files`, and have the transfer layer select a sanitized child directory named after that source root when an audiobook has multiple files and its organization mode is `none` or `rename`. Create that grouping directory before applying the existing hardlink/copy/move logic so operation accounting, torrent seeding preservation, collision handling, cleanup, and custom-script final paths continue to use the established production path. Completed multi-file audiobook torrents arrive as a directory whose chapter filenames may not identify the book, but folder output currently sends every discovered chapter directly to the configured destination in `none` and `rename` modes. This flattens chapters from unrelated books together and causes directory-oriented consumers such as Audiobookshelf to interpret individual chapters as separate books. A multi-file audiobook torrent in the default `rename` mode copies or hardlinks all supported chapter files beneath `<destination>/<original torrent directory>/` with their original chapter filenames, and places no chapters directly in the destination root; A multi-file audiobook in `none` mode receives the same source-folder grouping without renaming its chapter files. Fixes #1181 --------- Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
CaliBrain
|
e7007865a4 |
fix(prowlarr): stop turning indexer failures into empty results and 404s (#1251)
Two independent bugs, both from an indexer that Prowlarr proxies rather
than answers for itself: the search never reported that it had failed,
and the grab never resolved what it was handed.
Search. A Torznab search is Prowlarr proxying a live request out to the
tracker, so for a Cloudflare-fronted indexer it waits on FlareSolverr.
The client gave it the 30s budget sized for Prowlarr's own JSON
endpoints, then swallowed every failure -- the timeout, the 429 Prowlarr
returns once it has disabled an indexer, a parse error -- into the same
empty list that means "this indexer has nothing". A cold challenge
routinely runs past a minute, so the UI said "No releases found for this
book" while FlareSolverr was still solving. That empty list also drove
the auto-expand retry, which fires on "no results with the category
filter". A timeout satisfies it, so Shelfmark sent a second search to an
indexer still busy with the first -- two Chromes at once, enough to take
FlareSolverr's down on a small host.
torznab_search now raises ProwlarrSearchError, and an empty list
strictly
means the indexer answered with no matches. The source records which
indexer searches failed: one dead indexer no longer sinks the others,
auto-expand runs only when every indexer genuinely answered, and zero
results with at least one failure raises SourceUnavailableError, which
the releases endpoint already turns into a 503 carrying a real message.
Prowlarr being unreachable was the same lie by another route -- the
indexer list came back empty, leaving nothing to query -- and now says
so.
Indexer searches also get their own timeout, PROWLARR_INDEXER_TIMEOUT,
defaulting to 90s and clamped to 5-300. Prowlarr's status and indexer
list keep 30s so Test Connection stays responsive, and the connect
timeout is split out at 10s so an unreachable Prowlarr fails fast rather
than hanging for the whole read budget. The overall per-request search
budget now scales to twice the indexer timeout, capped at 240s, so
raising the setting is not undone by the cap one level up while staying
under the 300s gunicorn worker timeout.
Grab. Prowlarr hands out a proxy download URL, with no magnetUrl and no
infoHash, for any indexer that only publishes torrent files. The native
Real-Debrid client built its magnet as "if not
url.startswith('magnet:') and expected_hash", so with no hash to work
from it left the URL alone and POSTed it to /torrents/addMagnet as the
magnet field. Real-Debrid answered 404 and the grab died on a raw HTTP
error. AllDebrid carried the same line and the same bug.
Both now resolve the URL first, through the extract_torrent_info path
the
torrent clients have used since #1108: pass a magnet through untouched,
follow a redirect or a response body that turns out to be a magnet,
otherwise upload the fetched .torrent, and fall back to a magnet built
from the infoHash only when the fetch failed. The file is preferred over
a synthesized urn:btih: magnet because it carries the tracker list; a
bare hash leaves the service to find the swarm on DHT alone. Fetches are
shared with the rest of the add path through the torrent fetch cache, so
resolving costs at most one request. Real-Debrid takes the file on PUT
/torrents/addTorrent with the raw bytes as the request body, AllDebrid
on
POST /magnet/upload/file as multipart files[]. A URL that resolves to
neither form now raises before any request reaches the service, so the
user reads why instead of a 404. Neither debrid client had any test
coverage; both have some now.
Fixes #1249
Fixes #1250
|
||
|
CaliBrain
|
5247ec6124 |
fix(bypass): close the gaps a helper that outlives its request opened (#1244)
assumptions the code around it still made were written for a helper that was killed after every request. A bypass that hits the child's deadline is cancelled from the calling thread, which returns the moment the cancellation is scheduled - so the helper went on to serve the next request while the abandoned one was still closing its browser, on the same loop, sharing the DISPLAY globals and one process group. The deadline now lives inside the loop, where asyncio.wait_for() waits for the unwind before it raises, with the calling thread keeping a bounded backstop in case the cleanup wedges too. Both budgets are set so the child still answers before the parent gives up on it. The helper's cookie store survived the request as well, and the whole of it is exported back to the parent on every answer - so clearance the parent had purged for one host came back the next time some other host was solved, the dead-cookie resurrection _redirect_loop_handoff purges to avoid. The child starts each request from an empty store again; the parent already runs the cached-cookie check against a superset of it. DNS config is compared against what the helper is actually resolving through rather than skipped whenever the parent reports "auto", so a user flipping CUSTOM_DNS back to auto - which applies live - reaches a warm helper instead of leaving it on an abandoned DoH resolver. The 15s exit grace is now asked only of a helper that can still read its stdin. One dropped mid-bypass never returns to that read, so the grace could only end in the kill - while a user cancelling a download, and every bypass queued behind them on LOCKED, waited it out. Result files are cleaned on the timeout and cancellation paths too, staging file included, rather than only when the answer was read. |
||
|
CaliBrain
|
bd21ec1257 |
fix(audiobookbay): search the ASCII punctuation ABB actually stores (#1242)
WordPress texturizes punctuation on output only, so a post stored as "The Stranger's Wife" renders as "The Stranger’s Wife". ABB's search matches the stored value and ANDs its terms, so one typographic character in the query empties the entire result set rather than merely ranking worse. Book metadata and mobile keyboards both hand us those characters. Map curly quotes, dashes and ellipses to ASCII before a query goes out, and on both sides of the relevance comparison, since scraped titles carry the rendered forms. Release titles are still stored and displayed exactly as ABB renders them; only matching normalizes. Also percent-encode the search query properly. The hand-rolled encoder only escaped double quotes and spaces, so a bare "&" started a new query parameter and silently truncated the search: "detective dan riley books 1 & 2 weatherley" reached ABB as "detective dan riley books 1" and returned six confident-looking results without the requested book among them. "%" and "+" were mangled too. |
||
 helgehelge123andhelgehelge123
|
7b9c416df8 |
perf(bypass): keep the helper subprocess alive between bypasses (#1222)
Every protected request spawns a fresh helper subprocess, paying interpreter start and imports before any work begins. Measured inside the container, five consecutive runs of `python -c "import shelfmark.bypass.internal_bypasser"`: ``` 3.53s 3.45s 3.55s 3.54s 3.46s ``` A single search issues several protected requests, so that is paid several times over per search. ## What changed The helper now serves one JSON request per line of stdin until the parent closes the pipe, and an idle timer (`BYPASS_BROWSER_IDLE_TIMEOUT`, default 180s) shuts it down once searching stops. Answers still travel by result file, but the file is now written to a `.part` path and renamed into place — the parent treats the file's existence as the answer, so it must never observe a half-written one. stdout and stderr stay attached to the parent's, so helper logs keep appearing in `docker logs` exactly as before. Failure handling, since a warm helper is exposed to more of it than a per-request one ever was: | Situation | Handling | | --- | --- | | Helper died between requests | Detected via `poll()`, respawned | | Pipe broken at write time (`poll()` can miss this) | One retry on a fresh process; a fresh one failing there is a real failure | | Helper exits without writing a result | `RuntimeError` naming the exit code | | Wedged past the timeout, or cancelled mid-bypass | Helper killed, then `_cleanup_orphan_processes` because a killed helper never got to close Chrome | | Idle reaper racing an arriving request | Re-checks the deadline under the lock and re-arms instead of killing a helper that just did work | The DNS config now travels with every request rather than only at spawn: a warm helper outlives changes the parent makes to its provider. ## `BYPASS_REUSE_BROWSER`, off by default This parks the CDP driver between bypasses. A driver's websockets are bound to the loop that opened them and cannot outlive their process, so the persistent helper is what makes this possible at all — and the warm path runs on `_CDP_WORKER`'s long-lived loop rather than `asyncio.run` for the same reason. The mechanism works. With it on, the browser start disappears from the second request onward: 0.7s from `Reusing warm Chrome browser` to the first bypass attempt, against roughly 16s cold. **It still ships off, because a matched-pair test shows it is a net loss against DDoS-Guard.** Each round primed with one cold bypass, waited 10s, then measured a second — identical timing in both arms, only the browser strategy differing, order balanced (fresh, warm, warm, fresh) so drift over the session cannot masquerade as an effect: | Arm | Measured request | | --- | --- | | fresh browser | 42.8s, 40.6s | | warm browser | 57.1s, 59.6s | Spread within each arm is 2.2s and 2.5s, against 16.7s between them. Reuse removes the ~15s browser start and then gives back roughly twice that in solving: a returning browser draws a harder challenge. Where the cold browser is through on the second bypass method, the warm one fails the first three and only `_bypass_method_humanlike` gets it, at ~30s for that method alone. Worth separating from a second effect I ran into while measuring: five back-to-back searches slow from ~32s to 51–98s with reuse **disabled** as well, so DDoS-Guard escalates on request rate independently of any of this. That is why the pairs above are timed identically rather than simply run in sequence. It is the larger of the two effects, but not something this project can patch around. Reuse is left available rather than dropped because Cloudflare sites may not respond the same way, and because the two concerns are independent: the helper start is pure overhead and always worth removing, the browser is not. ## Verification - 2559 unit tests pass (2542 before, 17 added in `tests/bypass/test_warm_browser.py`) - `ruff check`, `ruff format`, `basedpyright` over backend and tests, and `vulture` all clean - `docs/environment-variables.md` regenerated via `scripts/generate_env_docs.py` - Live against Anna's Archive on a warm helper: searches return their usual ~760KB and 667 results, the app's own search warm-up completes with 50 results, and the container is left with no orphan chrome/Xvfb/ffmpeg processes Happy to drop the `BYPASS_REUSE_BROWSER` half entirely if you would rather not carry a default-off path — the helper persistence stands on its own. Co-authored-by: helgehelge123 <helge.neumann@zollsoft.de> |
||
|
CaliBrain
|
646b531669 |
fix(hardcover): accept the short hc_pat_ keys Hardcover issues now (#1241)
Hardcover replaced its ~500 char JWTs with short opaque personal access
tokens ("hc_pat_..."), and the connection test rejected anything under
100 chars before a request ever left Shelfmark, so every newly created
key failed with "API key seems too short".
The length floor now applies only to keys without the hc_pat_ prefix; a
prefixed key goes straight to Hardcover, which is the authority on
whether it is valid. Also strip a pasted "bearer " prefix regardless of
casing -- Hardcover's docs tell users to paste the token into an
"authorization" header, so the prefix rides along on the copy, and the
old case-sensitive removeprefix() sent it through as part of the token.
The API key field now names the expected shape.
Note that Hardcover's PAT path currently answers every hc_pat_ token
with a 500, a fabricated one included, while non-PAT tokens still get a
clean 401. So a new key cannot connect yet regardless of this change --
that failure is server-side and not something this code can reach.
Refs #1240
|
||
|
CaliBrain
|
7193036626 |
fix(rtorrent): apply the audiobook label to audiobook downloads (#1239)
add_download() picks self._audiobook_label from a content_type kwarg, but the only call site never passed one, so is_audiobook was always False and every download got RTORRENT_LABEL. category does not fill the gap: _get_category_for_task() returns None for rTorrent, which has no category concept, leaving content_type as its only audiobook signal. Pass task.content_type through from base_handler, and match it with the shared is_audiobook() helper instead of == "audiobook". normalize_content_type() treats "book (audiobook)" as an audiobook, so the exact-string check would have mislabeled that value even once it arrived. The existing rTorrent tests passed content_type straight to the client, which is why nothing caught the missing wiring; the new handler test covers the call site itself. Post-processing was never affected: destination.py reads task.content_type directly, so files already landed in DESTINATION_AUDIOBOOK correctly. Fixes #1235 |
||
|
CaliBrain
|
12d554a92f |
fix(download): hand a 503 carrying a challenge to the bypasser (#1238)
503 is in RETRYABLE_CODES, and the bypasser is only ever reached from the 403 branch and the AA redirect-loop rescues. Once Z-Library re-serves its DDoS-Guard interstitial with the same cookie the #1188 handshake already echoed back, the request has nothing left to try and spends every attempt on the same wall. Gate the handoff on the response body rather than the status, so a genuine overloaded-origin 503 keeps its retry path, and on allow_bypasser_fallback, so best-effort fetches still fail fast. The challenge indicators move out of internal_bypasser into shelfmark/bypass/challenge.py so http.py can use them without importing SeleniumBase, which is lazily imported precisely because it is optional. Refs #1233 |
||
|
CaliBrain
|
fae6140c6a |
fix(bypass): scope browser cleanup to the calling session (#1232)
The orphan sweep ran a container-wide 'pkill -9 -f chrome|chromium|Xvfb|ffmpeg', so it also matched browsers another bypass was still driving. Scope it by process group: kill only our own group and groups whose leader has died. Spawn the helper with start_new_session so its browser tree is identifiable, tear the whole group down after every run (a timed-out helper used to leak its Chrome and Xvfb), and have an orphaned helper take its browser down with it. Fixes #1231. |
||
|
CaliBrain
|
4cd1091d16 |
fix(hardcover): send the field count Hardcover's Book search requires (#1224)
Advanced title search, advanced title+author search, and the title typeahead returned zero results every time, and the sort fallback added in #1183 blamed the sort value for it. Hardcover turns the `fields` search parameter into Typesense's `query_by` but keeps `num_typos` and `query_by_weights` as fixed-length presets per query_type. For query_type=Book the preset expects exactly five fields, so a shorter list is not searched loosely - the whole search is rejected with a null results body. Confirmed against the live API: 1, 2, 3, 4 and 6 fields are all rejected, only 5 works, and weights must match one-for-one when sent. Every Book-type list we sent was the wrong length - the title typeahead and advanced title search sent 2, title+author sent 3. - Send BOOK_SEARCH_FIELDS (the full five) for every narrowed Book search and express the intent through weights instead. Weights only bias ranking - a field weighted 0 still matches - so a title search now ranks titles first rather than restricting to them. That is the closest behaviour Hardcover still allows, and there is no client-side filter to restore the old precision. - Pin the field and weight counts in tests, since the failure mode is a silent zero results rather than an error. The sort fallback from #1183 also misread these rejections: - Select the `error` field on every search and log Hardcover's own explanation. The reason is only ever in that sibling field, so a rejection surfaced as "returned no result body" with nothing to act on. Reading it is what made the field-count rule findable. - Drop `sort` entirely on the retry instead of sending an empty string. An empty sort is a value like any other and can be rejected too. - Arm the 900s sticky window only after the sortless retry succeeds. It was armed before the retry and never rolled back, so one rejected typeahead disabled sorting process-wide for 15 minutes whatever the actual cause. Verified against the live Hardcover API: advanced title search 0 -> 84 results, title+author 0 -> 139, title typeahead 0 -> 84 with the exact title top. 2566 unit tests pass; ruff, basedpyright and vulture clean. Refs #1183. The sort_by regression #1183 was written for is gone from Hardcover's side - every sort value it rejected, including the one in the report, is accepted again today. Two plain-search rejections in that report (fields=None) remain unexplained: they could not be reproduced under any per_page, page depth, sort value or query shape, and are most likely transient upstream. They now self-report the reason if they recur. |
||
|
CaliBrain
|
651096ed7b |
fix(bypass): reuse external bypasser clearance instead of re-solving (#1223)
Direct download was unusable behind an external bypasser (FlareSolverr /
Byparr): every request paid a 403 plus a full solve, and a search that
never ran was reported to the user as "No books found".
Clearance was discarded on the external path. get_cf_cookies_for_domain
and get_cf_user_agent_for_domain returned {} / None whenever
USING_EXTERNAL_BYPASSER was set, and _fetch_via_bypasser read only
solution.response - dropping solution.cookies and solution.userAgent,
which FlareSolverr-compatible services do return. A solve therefore
cleared the one request that paid for it and nothing else, and a file
download - which the solver cannot proxy, being binary - presented no
clearance at all. Diagnosed from a v1.3.9 debug bundle: ~35s in the
bypasser per search, on every search.
- Move the cookie jar out of internal_bypasser into bypass/cookie_store.
internal_bypasser imports seleniumbase at module scope, which is the
dependency an external-bypasser deployment is entitled not to have, so
it cannot host a store the external path depends on.
- Harvest solution.cookies and solution.userAgent after a successful
solve. The existing filtering applies unchanged, so the per-check
__ddg8_/__ddg9_/__ddg10_ trio is still dropped and the external path
cannot reintroduce the ?check=1 loop fixed in
|
||
|
CaliBrain
|
ebb833a82c |
fix(bypass): discard rejected DDoS-Guard cookies instead of replaying them (#1221)
A cookie that has been rejected was kept and presented again on every
later
request, so a single bad clearance could re-arm the challenge
indefinitely.
Cookie storage:
- Enforce expiry for every stored cookie, not just cf_clearance.
DDoS-Guard
domains have no cf_clearance, so the existing check never fired for them
and
expired cookies were replayed forever.
- Stop storing the per-check cookies __ddg8_/__ddg9_/__ddg10_ and
ddg_last_challenge. Captured live from Anna's Archive, these carry the
client
IP and the timestamp the check was issued (~40 min), versus ~1 year for
the
__ddg1_/__ddg2_/__ddgid_ clearance. Replaying an IP-bound token stops
describing the caller as soon as the egress IP changes, which is routine
behind a VPN.
Failure handling — every path that is rejected while carrying cookies
now
purges them, not just the redirect loop:
- 403 returned while presenting cookies.
- Cached-cookie attempt rejected, whether by status or by redirect loop.
- Factored the purge into _purge_clearance, guarded on a non-empty
hostname
since clear_cf_cookies("") means "every host" and would wipe clearance
for
sites that are working fine.
Also fix the search warm-up switches shipped inert in v1.3.8:
SEARCH_WARMUP_ENABLED and SEARCH_WARMUP_QUERY are not in the settings
registry, and config.get only consults the environment for keys it
knows, so
both always returned their defaults — the warm-up could not be turned
off or
retargeted. Read os.environ first.
Refs #1220. Deliberately not "Fixes": the reported failure could not be
reproduced on v1.3.8 from a stable IP (the reporter's own queries all
returned
200 on both the pre- and post-change builds), and the new purge paths
did not
fire in live testing because the failures arrive as redirect loops,
which were
already purged. These are correctness fixes with no measured effect on
that
issue. The underlying problem remains that Chrome-obtained cookies never
satisfy DDoS-Guard when replayed by requests, so every search still
re-solves.
Verified: 2542 unit tests pass; ruff, basedpyright and vulture clean;
e2e
platform baseline (10), full (6) and bypasser-external (5) all pass;
five
sequential live searches against Anna's Archive all returned 200 with
zero
"Exceeded 30 redirects".
|
||
|
CaliBrain
|
b656f019be |
feat(download): add DoH wireformat support, mirror quarantine, and search warmup (#1218)
- Add RFC 8484 DNS wireformat codec and HTTP/2 support (httpx) for Quad9/OpenDNS DoH providers. - Quarantine dead, parked, or seized mirrors for the session on hard failure (DNS errors, connection refused, 410/451, parked pages) while preserving bypass clearance on live mirrors. - Add background startup search warmup to prime DNS, elect mirrors, and pre-solve protection challenges to eliminate cold-start search latency. - Add comprehensive test suites for DoH wireformat, mirror quarantine, parked domain detection, and search warmup. |
||
|
CaliBrain
|
2b8b35bb52 |
fix(newznab): make indexer book categories configurable (#1214)
Newznab searches hardcoded category 7000 for ebooks and 3030 for audiobooks, so indexers using custom IDs returned no results or the wrong ones. Add NEWZNAB_EBOOK_CATEGORIES and NEWZNAB_AUDIOBOOK_CATEGORIES (tag lists, defaulting to 7000 and 3030) and resolve the search categories from config. Values are parsed leniently — list or comma/whitespace separated, non-numeric entries skipped, duplicates dropped — and fall back to the standard IDs when empty, so a cleared field can't silently widen the search to every category. NEWZNAB_AUTO_EXPAND remains the way to do that on purpose. Results carrying a custom ID outside the standard 7000-7999 / 3030 ranges were typed as "other", which routed custom-category audiobooks as ebooks. Trust the searched content type when a result carries a category we explicitly asked for. Also drop the unused NEWZNAB_BOOKS / NEWZNAB_AUDIOBOOKS constants from api.py — a third copy of the same hardcoding. Closes #1208 |
||
 FlozeandCaliBrain
|
58a5b5ed27 |
fix: sync renamed CWA usernames safely (#1203)
## Summary - sync an existing CWA-backed user's username when CWA renames it - keep username collisions safe by assigning a stable `__cwa` alias instead of overwriting a local account - allow username updates through `UserDB` and cover rename/collision/repeat-sync behavior Fixes #1197. ## Testing - `uv run ruff check shelfmark tests` - `uv run ruff format --check shelfmark tests` - `uv run vulture shelfmark` - `uv run pytest tests/core/test_cwa_user_sync.py tests/core/test_user_db.py tests/core/test_admin_users_api.py tests/core/test_auth_api.py -k "cwa or update_user"` (36 passed) - `uv run pytest tests/ -x --tb=short -m "not integration and not e2e" --ignore=tests/config/test_entrypoint_permissions.py -q` (2445 passed, 5 skipped) The entrypoint permission tests were excluded locally because macOS ships Bash 3.2, which does not support the `${1,,}` expansion used by `entrypoint.sh`; the same failure reproduces on an unchanged checkout. `make python-typecheck` also currently reports the existing `settings.py:147` callback return-type mismatch on the unchanged base. Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
3e2a7a48d5 |
fix: clear the DDoS-Guard cookie probe on AA search (#1209)
## Summary Two failure modes on the same code path, both reported this week: Anna's Archive `/search` is gated behind a DDoS-Guard cookie probe that the manual redirect follower can never satisfy. **#1202 — the cookie is dropped on every hop.** AA URLs set `allow_redirects = False`, so `html_get_page` follows redirects by hand. The 302 to `?check=1` carries a `Set-Cookie` (`__ddg*`) that has to come back on the next request. Because cookies are passed per call and `requests` keeps no jar across manual hops, it was discarded each time and the server just re-issued the same redirect until `_MAX_REDIRECTS` raised `TooManyRedirects`. The file already had the right helper — `_new_cookies()` — but only the 503 Z-Library handshake branch called it. **#1204 — the loop never reaches the bypasser.** `TooManyRedirects` isn't in `_is_retryable_error` and carries no status code, so the 403 rescue path (`status == _HTTP_STATUS_FORBIDDEN`) never fired and all attempts repeated the identical failure — ~2.5 min, surfacing as the misleading "Network restricted or mirrors are blocked". These interact, which is why #1202's fix alone isn't enough. Requests merge as `cookies={**handshake_cookies, **cookies}`, so **stale bypasser cookies override the fresh handshake ones** — once `_cf_cookies` holds an expired `__ddg*`, the probe can never clear no matter how faithfully we echo. Hence one search per restart, exactly as #1204 describes. ## Changes 1. Harvest cookies in the same-host redirect branch, the way the 503 branch already does. `_new_cookies()` returns only *new* values, so a server re-sending an identical cookie yields an empty dict and a genuine redirect loop still terminates at `_MAX_REDIRECTS`. 2. Treat a redirect loop as a detected challenge: purge the stored cookies for that host and switch to the bypasser, instead of burning the retry budget. Gated on `allow_bypasser_fallback` and `_is_cf_bypass_enabled()`, and skipped when already bypassing, so AudiobookBay (`allow_bypasser_fallback=False`) and external-bypasser setups are unaffected. The broader point in #1204 stands — the fallback would be better gated on "challenge detected" than on specific status codes, since DDoS-Guard presents at least three faces (403 js-challenge, 429, and this redirect loop). This PR fixes the two live exits without that refactor. ## Tests Two regression tests, both failing before and passing after: - `test_html_get_page_echoes_cookies_across_same_host_redirects` — the fake server only returns results if `__ddg2_` comes back on the `?check=1` hop. - `test_html_get_page_redirect_loop_purges_cookies_and_bypasses` — asserts the stored cookies are cleared, the bypasser runs, and the loop is cut short rather than repeated per attempt. `ruff check` and `ruff format` clean. `tests/download/` passes except `test_download_url_ignores_zlib_cookie_refresh_failure`, which fails identically on unmodified `main` in my environment (no `seleniumbase` — the `browser` extra isn't installed). ## Verification Applied on a live v1.3.7 install (Debian LXC, internal CDP bypasser). Before: every search timed out through 10 retries with `TooManyRedirects`, zero results. After: ``` http.py:455 - Redirect loop detected; switching to bypasser internal_bypasser.py:756 - Bypass successful using _bypass_method_cdp_gui_click internal_bypasser.py:322 - Extracted 9 protection cookies for annas-archive.pk direct_download.py:1865 - Found 24 releases via ISBN ``` ~25 s per search, results render. Note the second search still re-solves the challenge, since the freshly stored cookies go stale immediately — the design issue #1204 raises, left for the broader fix. Fixes #1202 Fixes #1204 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012Ln3yVj3sWHG2c6T78W1we --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
CaliBrain
|
0a5256ecbb |
fix(download): reconcile the two AA redirect-loop rescues (#1213)
#1210 and #1212 both added a DDoS-Guard `?check=1` rescue, and #1212 was branched before #1210 landed, so the merged result had two of them with identical guards. #1212's inline handoff returns before the raise that #1210's exception handler keys on, so the handler was shadowed and its stale-cookie purge — the substance of #1210 — never ran. Its regression test has been failing on main since the merge. Fold both into one path: - `_redirect_loop_handoff()` purges the host's stale clearance cookies, then bypasses, so the inline AA handoff and the exception handler cannot drift apart again. - The exception handler keeps its own reason to exist: non-AA hosts run with allow_redirects=True, so `requests` raises the loop itself and the manual AA follower never sees it. It now invokes the bypasser directly rather than setting a flag and continuing, which was a no-op at MAX_RETRY=1 for the same reason the 403 handoff was. - An unrescuable loop returns empty instead of raising TooManyRedirects into the retry path. That error is not retryable and carries no status, so `/dyn/md5/summary` (allow_bypasser_fallback=False) re-ran the full 6-redirect loop on all 10 attempts: 60 requests to AA and ~30s of backoff, measured. Every AA mirror shares the challenge, so there is nothing to rotate to. - `allow_bypasser_fallback` docs now describe what the flag actually gates; the old text predated #1198 and named the wrong callers. |
||
  Zoltán SzabóandKukkerem
|
056ddd372a |
Send DDoS-Guard's ?check=1 redirect loop to the bypasser (#1210)
Fixes #1204. ## Problem #1198 sends a gated AA `/search` to the bypasser when the origin answers 403. DDoS-Guard has a second response: when the clearance cookies from an earlier solve go stale, it serves an endless `?check=1` redirect instead. `requests` follows that until `_raise_too_many_redirects`, and `TooManyRedirects` carries no status code, so `status == _HTTP_STATUS_FORBIDDEN` is false and the rescue never runs. All 10 retries re-send the same dead cookies, then the search fails as `Unable to reach download source. Network restricted or mirrors are blocked.` Direct-download search therefore works once per container start, and stays dead after the stored cookie ages out. v1.3.7 (`sha256:520715f3…`), internal bypasser, mirrors `.gl/.pk/.gd`: ``` 17:04:36 internal_bypasser.py:756 - Bypass successful using _bypass_method_cdp_gui_click ... 17:11:39 http.py:483 - Retry 1/10 for https://annas-archive.gl/search?...&check=1: TooManyRedirects: Too many redirects 17:12:12 http.py:493 - Giving up after 10 attempts 17:12:12 main.py:2870 - Release search failed for source direct_download: Unable to reach download source. Network restricted or mirrors are blocked. ``` The token is short-lived, which is what makes this reachable in normal use: ``` $ curl -sD - 'https://annas-archive.gl/search?...&check=1' HTTP/2 403 server: ddos-guard set-cookie: __ddg8_=…; Expires=Fri, 14-Aug-2026 15:39:38 GMT # issued 15:19:38, 20 min ``` ## Fix Handle the loop like the 403: drop the domain's stored cookies, then retry through the bypasser. The branch sits above the `status ==` ladder because `_get_status_code()` returns `None` for this exception. Cookies are purged only for the internal bypasser; with an external one `get_cf_cookies_for_domain()` already returns `{}`. Related but not changed here: `get_cf_cookies_for_domain()` enforces expiry for `cf_clearance` only, so `__ddg*` cookies are never evicted on age, which is why they go stale. This patch makes the rescue fire whatever the reason the cookies stopped working. ## Verification The regression test drives a real redirect loop through `html_get_page` (302 to `&check=1`, exception raised by the production path rather than faked) and asserts the cookies are purged and the bypasser runs once. - `pytest tests/download/test_http_bypasser_fallbacks.py`: 8 passed. `test_download_url_ignores_zlib_cookie_refresh_failure` fails in my checkout on a missing `seleniumbase`, unrelated to this change. - `ruff check`, `ruff format --check`: clean. - Running in production since 2026-08-14 on v1.3.7 with only this file replaced: six direct-download searches, five served, three books downloaded end to end, against one search per container start before. The rescue mid-download: ``` 19:12:14 http.py:449 - Redirect loop detected; switching to bypasser: https://annas-archive.gl/md5/cb8fba7abae800ddbae1adfb8d7699d9?&check=1 19:12:38 internal_bypasser.py:756 - Bypass successful using _bypass_method_cdp_gui_click 19:14:36 direct_download.py:1142 - Resolved download URL [aa-slow-nowait]: … 19:14:47 orchestrator.py:735 - download finished; starting post-processing ``` ## Separate issue this exposes DDoS-Guard does not accept a solved cookie from plain `requests` traffic, so after this patch the rescue runs for nearly every AA URL. `internal_bypasser.get()` serializes all solves on one module-wide lock and builds a fresh Chrome each time: 11-16 s uncontended, 43-52 s under concurrent load, measured on the host above. Correctness is cheap here, latency is not. Happy to open a separate PR for a warm browser session if that direction is welcome. Co-authored-by: Kukkerem <Kukkerem@users.noreply.github.com> |
||
|
CaliBrain
|
d0e008adde |
Stop dropping audiobook releases that are not m4b or mp3 (#1199)
An IRC audiobook search returned nothing while OpenBooks, reading the same @search answer from the same channel, listed results. Three separate defects were discarding them. The audiobook format list was maintained by hand in four places and had drifted. The settings UI offered only m4b/mp3/m4a/zip/rar, and that list is the only one a user's config can be built from, so flac, opus, ogg, aac, wav and wma were unreachable everywhere — even though the IRC parser recognized them, the IRC sorter ranked them (dead code that could never fire), archive extraction knew them and Prowlarr searched for them. A FLAC audiobook was invisible in search and, if it arrived anyway, rejected after download as "format not supported". AUDIOBOOK_FORMATS and ARCHIVE_FORMATS now live once in core.utils and every layer derives from them, which also restored the missing .opus in the post-download scan's trackable extensions. Widening the default alone would not have reached anyone already affected: initialize_default_configs() writes field defaults only when a tab has no config file yet, so an existing install keeps its persisted m4b/mp3 list forever. migrate_audiobook_formats rewrites a list that still matches the old default exactly and leaves every other value alone — re-enabling formats someone had deliberately turned off would be worse than leaving them narrow. The IRC parser filtered by file extension alone. Multi-file audiobooks ship as a .rar or .zip of MP3s, which matched neither SUPPORTED_FORMATS nor SUPPORTED_AUDIOBOOK_FORMATS, so they fell out of the ebook bucket and the audiobook bucket both. Results are now classified before the format filter is applied: an audio extension means audiobook, an ebook extension means ebook, and for a container — where the extension says nothing about the contents — the release name decides. An ebook archive stays out of audiobook results. RESULT_LINE_REGEX matched \w+ after any dot, so a line carrying no file extension parsed as format "5mb" out of "::INFO:: 620.5MB", taking the title and the size down with it and guaranteeing every downstream filter dropped it. Any decimal size did this. The extension is now matched against the known formats, so such a line falls through to the simple pattern and comes back as "unknown", which the rest of the parser already handles. ALL_RECOGNIZED_FORMATS became an ordered tuple in the process: it was a set, so which extension won for a line naming two of them depended on set iteration order and could vary between restarts. Refs #1129 |
||
|
CaliBrain
|
03e219eb43 |
Let the bypasser solve bot challenges on Anna's Archive search (#1198)
Anna's Archive put a DDoS-Guard JS challenge in front of /search: the homepage still returns 200, but /search and /md5/<id> answer 403 on every mirror (.gl, .pk, .gd all confirmed). Search fetched both with allow_bypasser_fallback=False, which rotates mirrors on a 403 instead of invoking the bypasser, so it walked the whole mirror list, exhausted it, and surfaced "Unable to reach download source. Network restricted or mirrors are blocked." as a 503 on every query. Adding mirrors could not help — they sit behind the same gate — and neither could USE_CF_BYPASS, since search never reached that branch. Fetch search and the detail page with allow_bypasser_fallback=True so a 403 hands over to the bypasser, which already detects this challenge (DDOS_GUARD_INDICATORS matches the live page). Echoing the __ddg cookies back does not clear it; it needs real JS execution. The download-count fetch keeps allow_bypasser_fallback=False: it is decoration on the details modal and not worth holding the modal open for a browser solve. Fixes #1196 |
||
|
CaliBrain
|
29ce83e274 |
Stop dependabot proposing Python pre-releases, bound the e2e health wait (#1189)
PR #1169 (python:3.14.6-slim -> python:3.15.0b3-slim) ran for 6h before GitHub's max job limit killed it, then did it again on re-run. Two independent defects. Dependabot proposed a beta at all: the config already excluded python from the docker digest group for dependabot-core#9496, but the comment claimed ungrouped python updates get their pre-release filtered. They don't. dependabot-core#13815 rewrote the Docker pre-release heuristic to catch PEP 440 tags (its tests cover 3.15.0a2 and 3.5.0b3), yet the suffixed real tag still got through seven months later. CPython spells pre-releases without a separator, so 3.15.0b3 parses as an ordinary version sorting above 3.14.6. Ignore python semver-minor/major instead of trusting the heuristic; patch and digest updates still flow. The run took hours rather than failing: the health wait looked bounded at 60 iterations x 2s, but bare `curl` has no timeout. The 3.15 image booted a container that bound 8084 without ever serving (greenlet has no 3.15 wheel, so the gevent gunicorn worker was wedged), so curl blocked on read forever and the loop never reached iteration 2. Every job's orphan process at cancellation was that curl. Bound each probe and switch to a wall-clock deadline, and add timeout-minutes so a hang can never reach 6h again. Verified against a socket that accepts and never responds: the old loop was still hung at 30s, the new one exits at 120s with HEALTHY=0 into the existing log-dump path, and a responsive endpoint is still detected immediately. |
||
|
CaliBrain
|
e320b7623d |
Fix LOG_LEVEL being ignored and Z-Library 503 cookie gate (#1188)
LOG_LEVEL never reached the app logger: env.py hardcoded the level to DEBUG or INFO, so INFO lines kept appearing under LOG_LEVEL=error. Read it from the env var and advanced settings, normalize unknown values to INFO, and expose it as a setting. entrypoint.sh now normalizes gunicorn's level too, so a typo falls back to info instead of stopping the container from booting. Z-Library gates the first hit on /md5/<hash> with a 503 whose only payload is a Set-Cookie; echoing that cookie back returns the 302 to the real page. html_get_page dropped it and re-ran the same rejected request on every retry, ending in "No download URL resolved". Retry once with the cookies the 503 issued. Fixes #1185 Fixes #1187 |
||
|
CaliBrain
|
bb848f05bc |
fix/bypass stall watchdog (#1186)
- Fix protection bypass cancelled by stall detection at exactly 300s - make fixes - Try to fix Synology DELETE issues |
||
|
CaliBrain
|
cc1a95f965 |
Fix protection bypass cancelled by stall detection at exactly 300s (#1184)
A download that hits Cloudflare hung on "Bypassing protection..." for five minutes and then died, regardless of which bypasser was configured. html_get_page() started a BypassHeartbeat thread to keep the download marked alive during a bypass, but the thread had no loop: it fired one status event and returned. Even with the loop restored it could not have worked, because update_download_status() dedupes identical (status, message) tuples and returns before refreshing _last_activity, and the heartbeat re-sent the byte-identical payload already emitted just above it. So _last_activity was frozen for the whole bypass, while both bypassers are allowed to run longer than STALL_TIMEOUT (external FlareSolverr ~394s at default settings, internal 420s per get() call). The watchdog always won. From a reporter's log: 403 at 07:04:33.390, cancelled at 07:09:33.987 - exactly 300.000s, and 41s before the bypasser would have finished and reported the real error, an HTTP 500 from FlareSolverr the user never saw. The regression is not one commit. |
||
|
CaliBrain
|
dfcd7c9b00 |
Fix silent Hardcover search failures on rejected sort values (#1183)
Hardcover forwards the `sort` argument to Typesense's `sort_by` and
rejects
the entire search if it dislikes the value -- an unknown field, a bare
field
name with no direction, or more than three sort keys. A rejected search
is
not a GraphQL error: it comes back as HTTP 200, no `errors` key, and a
null
`results` body.
_extract_typesense_hits() reads that null as `hits=[], found=0`, so a
failed
search was indistinguishable from one that matched nothing. Users saw
zero
results with a healthy container and no log line explaining why.
Add _execute_search_query(), used by the three sort-bearing call sites
(book
search, field typeahead, series resolution):
- Detect the rejection via the null `results` body. A search that
genuinely
matched nothing still returns a results object with `found: 0`, so empty
result sets are not mistaken for failures.
- Retry once with an empty sort, which Hardcover always accepts, so
searches
return results instead of nothing.
- Keep that fallback sticky for 15 minutes so every subsequent search
does
not pay for a request known to fail, and let it expire so sort order
comes
back on its own if the index is fixed upstream.
- Log rejections that no sort can explain, and retries that also fail,
at
ERROR instead of discarding them.
While the fallback is active, results fall back to Typesense's default
ordering regardless of the selected sort. Degraded ordering beats no
results,
and it is now logged rather than silent.
SORT_MAPPING itself is unchanged: all five of its values were verified
against
the live API and return results. The `sort: "relevance"` reported in
#1179 was
the raw SortOrder value sent by v1.3.5; the mapping already fixed that.
What
remained unfixed, and is fixed here, is that the failure was invisible.
Fixes #1179
|
||
 Guflly
|
3c51b7cfaa |
Fix OIDC redirects on custom ports (#1180)
Fixes #1175 Preserves the forwarded host and port when Shelfmark builds OIDC callback URLs. The reverse-proxy examples now retain custom ports as well. Tests: - `uv run pytest -n 0 -q tests/core/test_proxy_headers.py tests/core/test_oidc_routes.py` - `uv run ruff check shelfmark/main.py tests/core/test_proxy_headers.py` - `uv run ruff format --check shelfmark/main.py tests/core/test_proxy_headers.py` |
||
  FlyinPancakeandCopilot Autofix powered by AI
|
3a9cff9816 |
feat(metadata): add Moly.hu metadata provider (#1172)
First of all, I don't know if you even want to merge a scraper-based metadata provider. I made this just for my use-case. If you'd rather not, I completely understand it. An alternative would be adopting [Audiobookshelf's Metadata Provider API](https://audiobookshelf.org/docs/documentation/community/community-providers) which I contributed to it for exactly the reason to not have scrapers. ## What Adds [Moly.hu](https://moly.hu) — the Hungarian community book catalog — as a metadata provider, following the existing provider plugin architecture (`@register_provider` + settings tab with enable checkbox and Test Connection button, disabled by default). ## Why None of the current providers cover Hungarian editions well: Hardcover and Open Library rarely index them, and Google Books coverage is spotty. Moly.hu is the de-facto catalog for Hungarian books (local editions *and* Hungarian translations of foreign works). With this provider, Universal mode works end-to-end for Hungarian titles: moly search → localized title/author feed the release search → indexers that carry Hungarian content can actually match. Related pain points: #595 (books missing from metadata providers), #1035 (interest in niche sources). ## How - HTML scraping with BeautifulSoup (already a dependency), no API key needed - Scraping approach (search URL, page structure, language-tag mapping) adapted from the long-lived Calibre `Moly_hu` plugin (GPL v3, credited in the module docstring), with fallback selector chains inherited from it - Sliding-window rate limit (30 req/min) to stay polite to a small community site - Standard `@cacheable` decorators; fetch failures return `None` so they are not cached (same behavior as the Google Books provider) - Search results carry cover thumbnails, rating and series info as display fields; `get_book` parses title (zero-width chars stripped, nested series link excluded), authors, ISBN-13/10, publisher, publish year, description (spoiler-warning prefix stripped), tags/genres, cover, and language (from moly's language tags, defaulting to `hu`) - ISBN search resolves through moly's site search ## Testing - `tests/metadata/test_moly_parse.py`: offline tests with fixture HTML mirroring live moly.hu markup — search parsing/dedup, pagination guard, failure-not-cached behavior, book-page parsing, ISBN resolution, ISBN validation helper - `uv run pytest tests/metadata` green (45 passed), `ruff check` / `ruff format` clean - Verified live against moly.hu (search, get_book, ISBN lookup) and running in Docker alongside Hardcover --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
 CamGoed
|
e93fbd2a9b |
Fix issue with stale v1 hash queries to qbittorrent (#1162)
## Problem
Shelfmark can lose track of hybrid v1/v2 torrents after qBittorrent
completes their metadata download.
Shelfmark initially identifies the torrent by its v1 infohash. Once
metadata resolves, qBittorrent may switch the torrent’s primary `hash`
to the truncated v2 hash, causing lookups using the original v1 hash to
return nothing.
For example:
- v1: `edf46c7f938a3c678081734d7bff8b9c652ba5e5`
- qBittorrent `hash`: `0bed5f40753b342cb143e83c2b21924cc8474731`
- full v2:
`0bed5f40753b342cb143e83c2b21924cc847473134e44d1bd300bdc58c13010f`
At that point, querying `/api/v2/torrents/info` with the original v1
hash returns no records. Querying with the new primary hash works, and
the returned record still contains the original hash in `infohash_v1`.
`find_existing()` also ignored its provided category, so audiobook
torrents fall back to the default ebook category instead. This means the
fallback method for a mismatched download ID never occurs for
audiobooks, leading to a "failed" download that is actually successful
in qBittorrent. As a result, the downloaded files are not automatically
transferred/hardlinked to the output directory.
## Fix
- Match torrents against `hash`, `infohash_v1`, and `infohash_v2`.
- Wait for magnet metadata to finish downloading before returning the
torrent ID.
- Return qBittorrent’s current primary `hash`.
- Search the provided category first, then the configured default, and
finally the full torrent list.
Logs showing the issue:
```
2026-08-03 22:42:30,262 - shelfmark.release_sources.audiobookbay.scraper - DEBUG - scraper.py:450 - Generated Magnet Link: magnet:?xt=urn:btih:EDF46C7F938A3C678081734D7BFF8B9C652BA5E5&tr=...
2026-08-03 22:42:30,376 - shelfmark.download.clients.qbittorrent - DEBUG - qbittorrent.py:521 - qBittorrent add result: TorrentsAddedMetadata({'added_torrent_ids': ['edf46c7f938a3c678081734d7bff8b9c652ba5e5'], 'failure_count': 0, 'pending_count': 0, 'success_count': 1})
2026-08-03 22:42:30,427 - shelfmark.download.clients.qbittorrent - INFO - qbittorrent.py:545 - Added torrent: edf46c7f938a3c678081734d7bff8b9c652ba5e5
2026-08-03 22:42:30,427 - shelfmark.download.clients.base_handler - INFO - base_handler.py:868 - Added to qbittorrent: edf46c7f938a3c678081734d7bff8b9c652ba5e5 for 'Pathogenesis: A History of the World in Eight Plagues'
2026-08-03 22:42:30,427 - shelfmark.download.clients.base_handler - DEBUG - base_handler.py:906 - Starting poll for edf46c7f938a3c678081734d7bff8b9c652ba5e5 (content_type=audiobook)
2026-08-03 22:42:32,590 - shelfmark.download.clients.base_handler - DEBUG - base_handler.py:958 - Download edf46c7f938a3c678081734d7bff8b9c652ba5e5 not yet visible in client (attempt 1/15)
2026-08-03 22:43:02,345 - shelfmark.download.clients.base_handler - ERROR - base_handler.py:969 - Download edf46c7f938a3c678081734d7bff8b9c652ba5e5 not found after 15 attempts
2026-08-03 22:43:02,345 - shelfmark.download.clients.base_handler - INFO - base_handler.py:426 - Skipping download client cleanup for protocol=torrent after download error (client=qbittorrent id=edf46c7f938a3c678081734d7bff8b9c652ba5e5)
```
<br>
The successful torrent:
<br>
<img width="968" height="159" alt="image"
src="https://github.com/user-attachments/assets/d63c4444-6a4d-4214-97a0-732acc338970"
/>
<br>
v1 vs v2 hash:
<br>
<img width="749" height="212" alt="image"
src="https://github.com/user-attachments/assets/d985b78a-84fe-4a26-9697-126c787e7303"
/>
I ran some python queries from the shelfmark container that show the
mismatch:
```
qBittorrent URL: http://gluetun-mam:8081
Tracked hash: edf46c7f938a3c678081734d7bff8b9c652ba5e5
=== PROPERTIES LOOKUP USING SHELFMARK HASH ===
HTTP status: 404
Not Found
=== EXACT /torrents/info HASH LOOKUP ===
Returned torrents: 0
=== FIND VISIBLE PATHOGENESIS TORRENT ===
Matching visible torrents: 1
Name: Pathogenesis: A History of the World in Eight Plagues
Primary hash: 0bed5f40753b342cb143e83c2b21924cc8474731
Category: audiobooks
State: stalledUP
Progress: 1
Properties HTTP status: 200
Infohash v1: edf46c7f938a3c678081734d7bff8b9c652ba5e5
Infohash v2: 0bed5f40753b342cb143e83c2b21924cc847473134e44d1bd300bdc58c13010f
```
|
||
|
CaliBrainandClaude Opus 5
|
21f2b6b95c |
Fix torrent post-import category follow-ups and clear the lint backlog (#1154)
rTorrent set_category and remove now uppercase the info hash, which reaches us lowercase while rTorrent's XML-RPC lookups are case sensitive. remove() had this bug before #1148, making PROWLARR_TORRENT_ACTION=remove a silent no-op for rTorrent. Transmission set_category appends the post-import label instead of replacing the whole label list. The unsupported-client path in post_process_cleanup logs at debug instead of warning, so Real-Debrid and AllDebrid users stop seeing a warning on every successful import. The Real-Debrid and AllDebrid clients now follow the conventions used by the other clients (_raise_runtime_error helpers, narrow error tuples, ClassVar, Path.open), and register_client is generic over a TypeVar bound to DownloadClient so decorated classes keep their concrete type. make fix and the lint, format and typecheck targets all pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
CaliBrain
|
10554444a9 |
Create a Tor watchguard and make iptables non mandatory (#1152)
Fix https://github.com/calibrain/shelfmark/issues/1150 |
||
|
CaliBrain
|
ce026e2eb8 | IRC health checks (#1151) | ||
 Tilian B
|
ff770940ca |
Add torrent post-import category action (#1148)
## Summary - add a **Change Category** torrent completion action and conditionally show its post-import category/label setting - update kept torrents only after a successful library import - support qBittorrent categories, Transmission labels, Deluge's Label plugin, and rTorrent's `custom1` label - preserve existing Keep/Remove behavior and document the new environment setting ## Behavior Category changes happen from `post_process_cleanup`, after output transfer and post-processing complete. This keeps the existing hardlink flow unchanged. An empty post-import category is a no-op, and client API failures are logged without turning a successful library import into a failure. ## Validation - `pytest -n 0 tests/prowlarr/test_qbittorrent_client.py tests/prowlarr/test_transmission_client.py tests/prowlarr/test_deluge_client.py tests/prowlarr/test_rtorrent_client.py tests/prowlarr/test_handler.py tests/config/test_generate_env_docs.py` — 161 passed - `ruff check` on all changed Python files — passed - `basedpyright` on the changed client implementation files — passed - full `basedpyright shelfmark/download/clients` currently reports two pre-existing errors in the new debrid connection-test code at `settings.py:546` and `settings.py:563`, outside this PR's diff - multi-architecture Docker images built successfully for `linux/amd64` and `linux/arm64` |
||
 
|
816a735cde |
Add a {Language} naming template variable, and consolidate language resolution (#1142)
Fixes #1138 Fixes #1141 ## Problem Two language editions of one book resolve to the same canonical title, so they render to the same path and the second gets a `_1` collision suffix. Audiobookshelf treats a folder as exactly one library item, so the pair becomes a single book with both files as tracks and a summed runtime. Shelfmark already parses and displays the language. It just never reached the template engine. ## `{Language}` template variable A template like `{Author}/{Title}{ (Language)}/{Author} - {Title}` now yields: ``` /library/J K Rowling/Harry Potter (sv)/J K Rowling - Harry Potter.m4b /library/J K Rowling/Harry Potter/J K Rowling - Harry Potter.m4b ``` The untagged edition's path is byte-identical to today, so no existing layout shifts. Three details worth flagging: **The value is casefolded.** On a case-insensitive filesystem `(SV)` and `(sv)` would collapse back into one folder, reintroducing the exact collision being fixed. **Values meaning "we don't know" render nothing** rather than producing `Project Hail Mary (unknown)` folders. Anna's Archive reports that string literally (`direct_download.py`, `language = detected or "unknown"`). **The frontend wasn't sending the release language at all**, so the token would have stayed empty for exactly the audiobook sources in the report. Prowlarr and AudiobookBay do not put language in `extra` the way `direct_download` does, hence the payload plumbing. It reads `release.language`, never `book.language` — the latter is the provider's canonical edition and would mislabel a translation, with a regression test for that specifically. Not gated to audiobooks: Calibre-Web-Automated stages ingested files by basename and discards folder structure, so the rename (filename) template is the only lever those users have. Verified that form works: `J K Rowling - Harry Potter (sv).epub`. ## Language consolidation (#1141) Three release sources each carried their own alias map, all resolving to the same ISO 639-1 codes, alongside a bundled database that only one of them used. Adding a language meant editing three places. Aliases now live in `data/book-languages.json` beside the code and name they belong to, and `shelfmark/core/languages.py` resolves any of them — two-letter code, ISO 639-2 three-letter in either the bibliographic or terminological form, or English name. Prowlarr and AudiobookBay drop their tables. Direct Download keeps its own path-parsing heuristics, including the ambiguous short codes that collide with English words (`de`, `en`, `no`, `in`), and takes only the alias data. This also closes a coverage gap. MyAnonamouse offers 62 languages; Prowlarr mapped 37, and an unmapped code is *dropped* rather than passed through, so the other 25 carried no language at all — leaving `{Language}` empty and the collision unfixed for Latin, Farsi, Tamil, Urdu and the rest. Seven languages MAM offers had no database entry at all: Bosnian, Burmese, Estonian, Icelandic, Manx, Scottish Gaelic, Sanskrit. Also fixes the Traditional Chinese code, which used a U+2011 non-breaking hyphen. Nothing compares against the ASCII spelling today so it was latent, but it would silently defeat the first thing that did. ## Validation Verified end to end against a live Prowlarr and MyAnonamouse, not just unit tests. A real search returning both an English and a Swedish edition, through the actual `queue_release` → `DownloadTask` → naming path: ``` STEP 1 real MAM search -> 37 releases, languages: ['en', 'sv'] STEP 3 queue_release -> task.language='sv' STEP 4 build_metadata_dict -> metadata['Language']='sv' STEP 5 build_library_path -> /library/J K Rowling/Harry Potter (sv)/... two language editions resolve to DIFFERENT folders: True ``` The refactor is pinned by a snapshot of both per-source maps taken *before* they were deleted. All 131 aliases are asserted to still resolve to the same code, one parametrised test each, so a regression names the specific alias. Also verified: the filename-only template, the retry round-trip (`serialize_task_for_retry` → `_restore_task_from_retry_payload`, plus a legacy payload with no `language` key), and placeholder handling. Added a `KNOWN_TOKENS` ordering invariant test — `find_placeholder()` does a substring `.find()` in list order and nothing protected that contract, so a future token in the wrong position could silently shadow an existing one. And a lockstep guard on the frontend, since `KNOWN_TOKENS` is hand-duplicated in TypeScript. **One caveat worth stating.** Three MAM codes are confirmed by observation (`ENG`→`en`, `SWE`→`sv`, `MAL`→`ml`, the last from a real `[MAL / EPUB]` Tagore release). The remaining ~59 are derived from ISO 639-2 rather than observed, because MAM's catalogue is overwhelmingly English — enabling 27 extra languages still yielded only one non-English hit across 258 results. Mitigated rather than closed: both 639-2 variants are present for every language where they differ, and a wrong alias is an unused entry while a missing one loses the language. Happy to correct any code a maintainer knows differs. ## Test results 2056 Python tests pass (up from 1906). Frontend typecheck, lint, format and 126 unit tests pass. Pre-existing failures on my machine, unchanged by this branch and unrelated: `tests/bypass/` needs `seleniumbase`, and `tests/config/test_entrypoint_permissions.py` uses bash-4 syntax that macOS bash 3.2 rejects. --------- Co-authored-by: delize <4028612+delize@users.noreply.github.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
CaliBrain
|
a1367f431d |
Ship Prowlarr per-entry rows as opt-in, not the default (#1145)
#1140 fixed the guid-only dedup that hid results from filter-specific indexer entries, but shipped the new behaviour on by default: PROWLARR_COLLAPSE_DUPLICATES defaulted off, so every existing Prowlarr user got extra rows for any release that two indexer entries both returned, and the setting only let them opt back into what they already had. Default it on. The dedup key stays indexer-qualified, so the entries are still distinct internally; collapse then merges them back to one row, resolved by the Prowlarr priority rather than by query order as before. The visible result set matches what users had prior to #1140, and anyone who wants the per-entry rows (freeleech and the like) turns the setting off. Beyond the noisier list, the default mattered because split rows differ only by indexer name while sharing a title, size and peer count. Two of them have distinct source_ids, so the queue's duplicate guard does not fire, and the second grab's find_existing() matches the first by infohash and runs post-processing over the same download again, delivering the book twice. The search-side fallback in config.get(..., True) is flipped to agree with the field default. In production the field default governs, since the config cache is seeded from the registry and the fallback only applies to an unregistered key. The source tests monkeypatch config.get with a plain dict lookup, though, so the fallback is what they exercise: leaving it False would have kept every default-behaviour test asserting the opposite of what ships. A new test pins the two together. The deduplication tests now opt out explicitly, since they assert the split itself. test_collapse_off_by_default_keeps_both_rows becomes a pair, one for the untouched setting collapsing to a single row and one for opting out. docs/environment-variables.md is regenerated rather than hand-edited. It was already stale on main, so it also picks up RTORRENT_AUDIOBOOK_LABEL, DIRECT_DOWNLOAD_LANGUAGE_FROM_PATH, and reworded IRC_SEARCH_BOT and RTORRENT_LABEL text from earlier merges. The rest is fallout from the ruff 0.16.0 bump in #1139, which enabled a much larger default rule set and started formatting Python code blocks in Markdown: _find_existing_alias_user() uses min() instead of sorted()[0] (FURB192, currently failing Python Quality on main), and the two READMEs get their code blocks reformatted. |
||
|
Andrew Doeringanddelize
|
a4086f5e06 |
Keep Prowlarr results from distinct indexer entries separate (#1140)
Results were deduplicated on `guid` alone. When one tracker is configured in Prowlarr as several indexer entries differing only by a server-side search filter, all of them return the same guid for the same torrent, so every entry but the first was silently discarded. Freeleech and other filter-specific releases became invisible, replaced by the unfiltered entry's copy, and which copy survived depended on query ordering rather than user intent. Include the indexer id in the dedup key so the entries stay distinct. Release.source_id is qualified the same way. It keys the release cache and becomes the download task id, so rows sharing a guid would otherwise collide and a grab would route through whichever entry cached last, defeating the point of showing them separately. The handler's task matcher still accepts a bare guid or infoUrl so tasks queued before this change still resolve. Ordering now follows the priority already configured in Prowlarr (1-50, lower preferred) rather than a new setting, since users curate that ranking there and filtered entries are typically ranked ahead of their unfiltered counterparts. The enabled-indexer list is fetched once and reused for the enrichment check, so this costs no extra round trip. Releases carry extra.indexer_priority, and the sort dropdown gains an "Indexer priority" entry, ascending, alongside the existing alphabetical "Indexer" sort; SortOption grew a default_direction for that, defaulting to desc so "Peers" is unchanged. PROWLARR_COLLAPSE_DUPLICATES (default off) optionally collapses a release back to one row, resolved by the same Prowlarr priority. Left off, every entry that carried a release keeps its own row, which is what makes filtered results visible again. Identity handling is defensive about partial payloads: a result that cannot be identified is never dropped or merged, and collapse only merges on a strong identifier (guid/downloadUrl/magnetUrl/infoUrl) because merging on title alone would discard genuinely different releases that share a name. Fixes #1137 Co-authored-by: delize <4028612+delize@users.noreply.github.com> |
||
 Adam Vigneaux
|
0d02c6db47 |
Add qBittorrent API key authentication (#1143)
[qBittorrent 5.2.0](https://www.qbittorrent.org/news#sun-may-03rd-2026---qbittorrent-v5.2.0-release) (May 2026) added support for API key-based authentication in addition to the existing username/password-based authentication. This commit adds support for qBittorrent API key authentication to Shelfmark, configurable via environment variable or settings UI. If an API key is set at the same time as the username/password, API key will be preferred for authentication. Requires `qbittorrent-api` 2026.5.3, the version that added the `api_key` argument, or newer. `403 Forbidden` responses are not retried with API key authentication because a retry has no chance of succeeding. Tested end-to-end with my live qBittorrent 5.2.3 instance (WebAPI v2.15.1). <img width="785" height="616" alt="image" src="https://github.com/user-attachments/assets/8064e10e-9a7f-49f0-804e-4c441d23a1fc" /> |
||
|
CaliBrain
|
7fdaf3f67d | Add Audiobook support to IRC (#1136) | ||
|
CaliBrain
|
81a057c8a0 | Try and fix OIDC (#1127) |