Compare commits

...
12 Commits
Author SHA1 Message Date
CaliBrain 97e289ae13 fix: search, Prowlarr and qBittorrent follow-ups (#1276, #1283) (#1284) 2026-08-30 03:09:13 -04:00
Jorge Lima c95ee72ad5 fix(qbittorrent): keep magnets whose metadata is still pending (#1282)
## Problem

`QBittorrentClient.add_download()` waits 20 × 0.5 s for qBittorrent to
leave `metaDL`, then raises:

```
Failed to add to qbittorrent: Torrent metadata resolution was not confirmed within the visibility grace period
(response=TorrentsAddedMetadata({'added_torrent_ids': [], 'failure_count': 0, 'pending_count': 1, 'success_count': 0}))
```

The wait exists to learn qBittorrent's primary torrent ID, which for
hybrid torrents switches from the v1 hash to the truncated v2 hash once
metadata resolves. A magnet on a thin public swarm routinely needs
longer than 10 s to find a peer that will serve metadata, and the
download is then abandoned even though the add itself succeeded. The
torrent stays in qBittorrent (`base_handler` logs "leaving in
qbittorrent") and often completes minutes later with nobody watching it.

Seen on v1.3.12 with public indexers through Prowlarr: every magnet-only
release failed this way, while `.torrent` releases from a private
indexer were fine. qBittorrent showed the same torrents at `metaDL 0%
seeds=0/0`, and they resolved on their own well after shelfmark had
given up.

## Change

Return the info hash we already have instead of raising when the grace
period expires. Reads then resolve either identity:

- `get_status()` and `get_download_path()` use `_resolve_torrent()`
instead of `_get_torrent_info()`, so a v1 hash still matches after
qBittorrent re-keys the torrent to v2. `_torrent_matches_download_id`
already compares `hash`, `infohash_v1` and `infohash_v2`.
- `remove()` and `set_category()` address the torrent by its current
primary hash through a new `_current_hash()` helper, which falls back to
the ID it was given when the torrent cannot be resolved.
- The two magic numbers become `_METADATA_WAIT_POLLS` and
`_METADATA_WAIT_INTERVAL_SECONDS`.

The happy path does not change. When metadata resolves inside the grace
period the resolved primary hash comes back as before, and
`_resolve_torrent()` tries the exact-hash lookup first, so it costs no
extra request.

## Tests

`test_add_fails_when_metadata_never_resolves` asserted the old
behaviour, so it becomes
`test_add_keeps_torrent_when_metadata_never_resolves` and asserts the
info hash is returned.
`test_get_status_resolves_hash_after_metadata_switch` is new: it reads
status by the v1 hash after qBittorrent reports the torrent under its v2
hash.

`uv run pytest tests/ --ignore=tests/e2e` gives the same 55 failures
with and without this change (they are all in `tests/bypass/` and need
Chrome, which my machine has no headless setup for), and
`tests/prowlarr/` is green at 524 passed. Ruff check and format are
clean. I have not run this branch against a live qBittorrent, so a
second pair of eyes on the `remove()` path would help.
2026-08-30 02:00:42 -04:00
CaliBrain b25acdb2ad fix(packs): don't disrupt normal downloads when inspecting for packs (#1274)
Follow-ups to the multi-book pack feature (#1270), which inspects every
release before download. Two behaviours leaked into the ordinary
single-book
flow and are corrected here:

- A flat folder of chaptered audio (`01 - Chapter.mp3`, `02 - ...`) was
detected as a pack, because each track name parses to a series position,
so
clicking download popped the review panel for one normal audiobook. Flat
folders are now split one-book-per-file only with real evidence of
distinct
books: two or more series positions, more than one title, and no
chaptered
audio (only the single-file m4b/m4a containers and ebook formats
qualify).
  Subfolder packs and flat m4b/m4a packs are unchanged.

- Every release that couldn't be inspected (usenet, magnet-only, sources
  without a list_files hook, ABB single-file) showed an info toast on
download. That is now a console.warn, so a normal download is silent
again.

Adds regression tests for the chaptered-mp3 cases.
2026-08-27 01:00:54 -04:00
dependabot[bot] 7569aaecc5 build(deps): bump seleniumbase from 4.52.1 to 4.52.2 in the python-deps group (#1273)
Bumps the python-deps group with 1 update:
[seleniumbase](https://github.com/seleniumbase/SeleniumBase).

Updates `seleniumbase` from 4.52.1 to 4.52.2
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/seleniumbase/SeleniumBase/releases">seleniumbase's
releases</a>.</em></p>
<blockquote>
<h2>4.52.2 - MCP Server Support</h2>
<h2>MCP Server Support</h2>
<p><strong>If you love AI tools, this is one of the biggest releases
this year for SeleniumBase!</strong>
<strong>The new <code>seleniumbase-mcp</code> command starts the
&quot;Pure CDP Mode&quot; MCP Server.</strong>
<strong>(Be sure to install <code>seleniumbase[mcp]</code> to get
<code>mcp&gt;=2.0.0</code>!)</strong>
<strong>(To debug the MCP server from a <code>git clone</code> of
SeleniumBase, get <code>uv</code> as well before calling <code>mcp dev
server.py</code> from the <code>SeleniumBase/mcp_servers/</code>
folder.)</strong></p>
<ul>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/4014476e288d004a8de38eca582631c9c653fa22">Add
a SeleniumBase MCP server</a></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/ac7e449fbbc8a6e9347e65e4ced8176ba1655112">Add
.mcp.json to the root folder</a></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/32fac6becaabbb288be42e98d6528108bdb05e73">Update
setup.cfg files</a></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/168500f45b547c914539057f76e3cd31e0121461">Update
.gitignore</a></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/bb495cb23e405ab9a523068cdb64b48f487f22c5">Update
.dockerignore</a></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/f030eae7b492f41d1c74c731e775f21b853afbdf">Add
configuration for using the new MCP server</a></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/422bb2fc11bd6ef58758c0eff1456f033a94db1f">Refresh
Python dependencies</a></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/7b245c033dc47e356fdaa99b5db7ebf965329bfe">Update
ReadMe files</a></li>
</ul>
<p><strong>Note that you will need <code>mcp&gt;=2.0.0</code> for the
MCP Server to work!</strong></p>
<p>⚠️ Note: Due to a typing bug, (<a
href="https://redirect.github.com/seleniumbase/SeleniumBase/issues/4471">seleniumbase/SeleniumBase#4471</a>),
the MCP Server in this release only worked on Python 3.14+.
Upgrade to <a
href="https://github.com/seleniumbase/SeleniumBase/releases/tag/v4.52.3">https://github.com/seleniumbase/SeleniumBase/releases/tag/v4.52.3</a>
for the fix.</p>
<h2>What's Changed</h2>
<ul>
<li>MCP Server Support by <a
href="https://github.com/mdmintz"><code>@​mdmintz</code></a> in <a
href="https://redirect.github.com/seleniumbase/SeleniumBase/pull/4470">seleniumbase/SeleniumBase#4470</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/seleniumbase/SeleniumBase/compare/v4.52.1...v4.52.2">https://github.com/seleniumbase/SeleniumBase/compare/v4.52.1...v4.52.2</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/a28aa518e34c859d3a3f90daf03c1aa926940931"><code>a28aa51</code></a>
Merge pull request <a
href="https://redirect.github.com/seleniumbase/SeleniumBase/issues/4470">#4470</a>
from seleniumbase/mcp-server-support</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/7b245c033dc47e356fdaa99b5db7ebf965329bfe"><code>7b245c0</code></a>
Update ReadMe files</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/1e5ac8dd110ecb96add400a45f009735be5e8683"><code>1e5ac8d</code></a>
Version 4.52.2</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/422bb2fc11bd6ef58758c0eff1456f033a94db1f"><code>422bb2f</code></a>
Refresh Python dependencies</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/f030eae7b492f41d1c74c731e775f21b853afbdf"><code>f030eae</code></a>
Add configuration for using the new MCP server</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/bb495cb23e405ab9a523068cdb64b48f487f22c5"><code>bb495cb</code></a>
Update <code>.dockerignore</code></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/168500f45b547c914539057f76e3cd31e0121461"><code>168500f</code></a>
Update <code>.gitignore</code></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/32fac6becaabbb288be42e98d6528108bdb05e73"><code>32fac6b</code></a>
Update <code>setup.cfg</code> files</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/ac7e449fbbc8a6e9347e65e4ced8176ba1655112"><code>ac7e449</code></a>
Add <code>.mcp.json</code> to the root folder</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/4014476e288d004a8de38eca582631c9c653fa22"><code>4014476</code></a>
Add a SeleniumBase MCP server</li>
<li>See full diff in <a
href="https://github.com/seleniumbase/SeleniumBase/compare/v4.52.1...v4.52.2">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=seleniumbase&package-manager=uv&previous-version=4.52.1&new-version=4.52.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-27 00:41:57 -04:00
Lance Marks f441b85da2 feat(packs): inspect multi-book releases and file each book separately (#1270)
## Multi-book packs: inspect a release before download and file each
book separately

Closes #576

### Problem

One queued release is always treated as one book. When a torrent is
actually a whole series
(`Series/Book 1 - Title/…`, or a flat folder of `Series 1.0 - Title.m4b`
files), post-processing
walks the whole tree, flattens every file into one list and renames them
`Title - 01…10` under the
searched book's `{Author}/{Title}`. Audiobookshelf then sees a single
10-file "book" and the user
has to re-file everything by hand.

### What this does

Most releases expose their file list *before* anything is downloaded, so
the split is decided up
front and approved by the user, then the download is fire-and-forget:

1. **Inspect** – clicking a release's download button now calls `POST
/api/releases/inspect`
first. A new optional `DownloadHandler.list_files(release_data)` hook
returns the release's
   files without downloading:
- **AudiobookBay** reads the torrent file table off the detail page it
already fetches (the
page is now cached for 120 s, so inspect + download cost ABB one
request).
- **Prowlarr** parses `info.files` from the `.torrent` it already
fetches (the existing 120 s
torrent-fetch cache is reused). Magnet-only and usenet releases report
"can't inspect".
   - Other sources default to `None`.
2. **Review** – if the plan contains more than one book, the Find
Releases modal swaps the list
for a review panel: one row per book with editable title / series
position / year, expandable
file lists, non-book sidecars (`.txt`, covers) shown as ignored, a
"Treat as a single book"
switch, and **Download N books**. Single-book releases queue
immediately, exactly as before.
3. **File** – the approved plan travels with the task
(`DownloadTask.book_plan`, retry-safe) and
post-processing files each book through the existing transfer code, one
book at a time
(`dataclasses.replace(task, title=…, series_position=…, year=…)`), so
organize/rename
templates, part numbering (now scoped per book), hardlinks, torrent
copy-preserve and usenet
   handling are unchanged. Status reads `Complete (N books, M files)`.
4. **Fallback** – when a release can't be inspected the user gets a
toast, and a small
"Multi-book pack" toggle in the modal header forces a heuristic split
(subfolder = book, or
   one book per file when the file names carry series positions).

Planning lives in `shelfmark/download/postprocess/packs.py` and is
shared by the inspect endpoint
and post-processing, so what the user approved is what gets filed. The
name parser strips
`Book 3 -`, `03 -`, `1.0 -`, `3.`, `[03]`, `#3`, a leading series name,
labels like
"An Expanse Novella -", repeated titles (`Gods of Risk 2.5 - Gods of
Risk`) and a trailing
`(Year)`; author and series name come from the book that was searched,
and the searched book's
own series position is never applied to its siblings.

### Files

- `shelfmark/download/postprocess/packs.py` (new) –
`PackFile/PackBook/PackPlan`, `plan_pack`,
`parse_pack_book_name`, `group_files_into_books`, `match_plan_to_files`
- `shelfmark/core/release_inspect_routes.py` (new) – `POST
/api/releases/inspect`
- `shelfmark/release_sources/__init__.py` – `DownloadHandler.list_files`
hook
- `shelfmark/release_sources/audiobookbay/{scraper,handler}.py` –
detail-page cache,
  `extract_file_list`, `list_files`
- `shelfmark/release_sources/prowlarr/handler.py`,
`download/clients/torrent_utils.py` –
  `extract_file_list_from_torrent`, `list_files`
- `shelfmark/core/models.py`, `download/orchestrator.py` – `multi_book`
/ `book_plan` fields,
  queue + retry serialization
- `shelfmark/download/postprocess/transfer.py`, `pipeline.py`,
`outputs/folder.py` – per-book
  transfer branch and status message
- `src/frontend`: `components/PackReviewPanel.tsx` (new),
`ReleaseModal.tsx`, `App.tsx`,
`services/api.ts`, `types/index.ts`, `utils/releasePayload.ts` (payload
builder moved out of
  `App.tsx`), `utils/packReview.ts`
- `docs/dev/release-sources-plugin-guide.md` – documents the
`list_files` hook

### Out of scope (follow-ups)

- Listing files from an NZB (Shelfmark already fetches the bytes; `<file
subject>` names are noisy)
- Inspecting magnet links via qBittorrent's files API after a paused add
- BookLore / email outputs (they ignore `book_plan`; noted in code)
- The combined ebook + audiobook flow

### Testing

**Automated** (`make checks`, `make python-test`, `make frontend-test`
all green; the only
failures on my machine are the pre-existing
`tests/config/test_entrypoint_permissions.py` cases,
which need bash ≥ 4 and fail identically on `main` under macOS bash
3.2):

- `tests/download/test_packs.py` – name parsing (markers, series name,
novella labels, repeated
titles, bare numeric titles like `1984`), nested / flat / mixed /
deeper-nested packs, single
wrapping folder not treated as a pack, plan-to-disk matching with
basename fallback
- `tests/core/test_processing_packs.py` – full `post_process_download`
runs on a real temp
filesystem: approved plan files each book under its own
`{Author}/{Title}`, heuristic split
of a nested pack, searched book's series position does not leak,
multi-file book inside a pack
keeps `- 01/- 02` per book, hardlinked torrent pack leaves the seeding
tree intact, no pack
fields ⇒ behaviour unchanged, single group degrades to the searched
title, status message
- `tests/core/test_release_inspect_routes.py` – plan response,
not-inspectable, handler errors
  never 500, unknown source / missing `source_id` ⇒ 400, login required
- `tests/audiobookbay/test_file_list.py` – file-table scraping from real
ABB markup (multi-file
and single-file pages), handler host validation, one page fetch shared
by magnet + file list
- `tests/prowlarr/test_torrent_file_list.py` – multi-file / single-file
`.torrent` parsing,
  handler behaviour for torrent URL vs magnet vs usenet vs cache miss
- `tests/download/test_orchestrator_pack_fields.py` – queue-time parsing
and retry round-trip
- Frontend: `releasePayload.test.ts`, `packReview.test.ts` (vitest)

**Manual, on a real deployment** (arm64 image built from this branch,
run as a side container
next to production with the same qBittorrent / Audiobookshelf setup,
`FILE_ORGANIZATION_AUDIOBOOK=organize`,
hardlinks on):

- AudiobookBay "The Expanse Complete 2.0" (7.87 GB, 36 files): clicking
download opened the review
panel in ~1 s showing **18 books · 18 files · 18 files ignored** (the
`.txt` sidecars), with
series positions 0.1–9.5 and years parsed from the file names; novella
labels stripped
("The Churn", "The Butcher of Anderson Station"). Editing a title in the
panel works.
Confirming queued one task; the magnet resolved from the cached page in
~30 ms; after the
download the task reported `Complete (18 books, 18 files)`, 18 hardlinks
landed as
`audiobooks/James S. A. Corey/<Title>/<Title>.m4b`, the torrent kept
seeding, and
Audiobookshelf scanned each folder as its own book (title, author,
embedded chapters).
- A second pack ("Expanse [01 - 9.5]", `Title N - Title` naming) was
inspected to verify the
  repeated-title rule and the Back button, without downloading.
- Single-book releases still queue immediately with no extra UI.
2026-08-27 00:40:08 -04:00
zab1996andRyan 02b7e9d958 feat(newznab): support multiple named indexers (#1271)
## Summary

- add a named Newznab indexer table with per-indexer URL and API key
settings
- search every configured indexer and retain the originating indexer
name on each result
- namespace cached release IDs across connections and isolate individual
indexer failures
- preserve the legacy single-indexer settings as a fallback
- support masked API-key cells and trusted SABnzbd prefetching for named
indexers

## Validation

- 121 Newznab and SABnzbd backend tests passed on Python 3.14
- Ruff passed for all changed Python files
- frontend TypeScript and strict lint checks passed
- all 134 frontend unit tests passed
- frontend formatting check passed

## Compatibility

Existing `NEWZNAB_URL` and `NEWZNAB_API_KEY` configurations continue to
work whenever `NEWZNAB_INDEXERS` is empty.

Co-authored-by: Ryan <zab1996@users.noreply.github.com>
2026-08-27 00:29:58 -04:00
CaliBrain ff06a1a581 fix(search): follow-ups to per-user book languages (#1267)
Review follow-ups to #1255, all in the code that PR touched.

Drop the dead user_id from the Prowlarr retry path.
ProwlarrSource.search
never reads plan.languages, and _refresh_release builds a synthetic book
with no titles_by_language, so the title variants came out identical
with
and without it. It also should not language-filter: it re-finds one
exact
release by its guid.

Pin the tab move in tests. BOOK_LANGUAGE moved from the General tab to
Search Mode with no migration, which only works because both tabs
persist
into the same settings.json. Nothing asserted that, so splitting the
files
later would silently reset every install to ["en"]. Covers the stored
value, a fresh install, and ENV precedence.

Stop the UI inventing a default language. An empty BOOK_LANGUAGE is a
deliberate "no default filter" that the backend preserves, but the two
frontend call sites replaced it with the first supported language, so
the
filter said English where the server filtered nothing.
resolveDefaultLanguageCodes
now falls back only when the value is absent.

Keep the normalized value for every validated search key.
validate_user_settings
gated the write-back on a hand-maintained subset of the keys the search
validator recognises, so METADATA_PROVIDER_COMBINED,
SHOW_COMBINED_SELECTOR
and FORCE_COMBINED_SEARCH were validated and then stored raw -- a padded
provider name was accepted and persisted with its padding. Reuse the
validator's own key set instead.

Skip blank language entries rather than rejecting them, so "" and "en,"
mean the same as [] and ["en"] instead of erroring on an unnamed
language.

Extract resolveListOverride for the list-override detection that was
copy-pasted between the two user-settings sections, and mention
languages
in the Search Preferences section description.
2026-08-24 18:11:29 -04:00
463ef49ac3 feat(search): let each user pick their own default book languages (#1255)
## Why

`BOOK_LANGUAGE` is a per-reader property, not a per-instance one. On a
shared install one household member searches in German while another
wants English and German — today whoever changes the setting changes it
for everyone, and the only escape is re-picking languages in the filter
on every single search.

The per-user override machinery already carries `SEARCH_MODE`, the
metadata providers and the default release sources, so the language
default mostly had to opt into it.

## What changed

**The field.** `BOOK_LANGUAGE` becomes `user_overridable` and moves from
the **General** tab to **Search Mode**, next to the other
user-overridable search defaults (per
[review](https://github.com/calibrain/shelfmark/pull/1255#issuecomment-5391189094)
— the first version had the Search section span two tabs, this one
doesn't). Admins set it per user in the user editor, users set it in
**My Account → Search Preferences**, and the Search Mode tab carries the
usual "N users override this" summary.

**No migration for the move.** `general` and `search_mode` both persist
into `settings.json`, and a field's value is resolved through
`load_config_file(tab)` for the tab it's declared on — so an install
that already stores `BOOK_LANGUAGE` keeps its value. Checked against a
`settings.json` written while the field still lived on General: the
stored value resolves unchanged, a fresh install still gets `["en"]`,
and `BOOK_LANGUAGE` in the environment still overrides both.

**The two places the default is read.**

- `/api/config` seeds the frontend's language filter, so it now resolves
`BOOK_LANGUAGE` for the session user.
- `build_release_search_plan` falls back to the default whenever a
request carries no language filter — which is exactly what the filter's
"Default" option sends. It takes an optional `user_id`, passed by
`/api/releases` from the session and by the Prowlarr retry path from
`task.user_id`, so a retry re-searches in the languages of whoever
queued the download.

**Validation.** Overrides go through `normalize_language()`, so
`"German"`, `"ger"` and `"de"` all store as `de`, and an unknown
language is rejected with a message naming it instead of being silently
searched for. An empty list stays an empty list (a deliberate "no
default filter"), `null` clears the override as everywhere else, and ENV
still wins: with `BOOK_LANGUAGE` set in the environment the field
reports `fromEnv` and overrides are ignored.

**Scope.** Only the language default becomes overridable. The two format
lists left behind under "Default Search Filters" stay admin-only — they
describe what the library and its post-processing accept, not what a
reader wants to read. There's a test pinning that.

## Verification

- 2681 unit tests pass (2670 before, 11 added)
- `ruff check`, `ruff format`, `basedpyright` over backend and tests,
and `vulture` all clean; frontend lint, format, typecheck and 126 unit
tests clean
- `docs/environment-variables.md` regenerated via
`scripts/generate_env_docs.py` (the `BOOK_LANGUAGE` row follows the
field into the Search Mode section)
- Manually against a two-user instance with builtin auth (first round,
before the tab move): with user A on German and user B on
English+German, `/api/config` returns each reader their own
`default_language` and an unfiltered `/api/releases` plans the matching
languages; an admin can set and read the same override for another user;
clearing it falls back to the global value; a stray `"klingon"` is
rejected; and `BOOK_LANGUAGE` in the environment overrides both users
with the field marked `fromEnv`
- After the tab move I re-ran the suites above plus the
stored-value/fresh-install/ENV check described under "No migration for
the move"; the behaviour it exercises is what the move could have broken

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: CaliBrain <calibrain@l4n.xyz>
2026-08-24 17:57:21 -04:00
CaliBrain a5595cf9f1 Change test for fake extension that wont work (#1266) 2026-08-24 17:54:10 -04:00
jakesterpdxandClaude Fable 5 9bcf595111 feat(prowlarr): warn when an indexer declares a format Shelfmark can't process (#1265)
## Problem

Companion to #1264, but general rather than mp4-specific.

MyAnonamouse titles carry a structured `[LANG / FORMATS]` bracket that
`_extract_mam_formats` parses. When every token in it is something
Shelfmark doesn't know — e.g. `The Martian by Andy Weir [ENG / MP4]` —
the release is rendered with **no format chip at all**, just the generic
headphones/book icon with an "Audiobook" tooltip. To a user that looks
like an ordinary result. It downloads fine and then fails
post-processing with *"No book files found in download"*.

The backend already *had* the signal (a format token it couldn't map);
it just threw it away.

## Change

**Backend** (`shelfmark/release_sources/prowlarr/source.py`)
- `_split_mam_formats(raw_title) -> (recognized, unrecognized)` replaces
the body of `_extract_mam_formats`, which is kept as a thin wrapper
returning `recognized` so nothing else changes.
- Releases gain `extra["unrecognized_formats"]` (list, or `None` when
empty / when format detection is off).

**Frontend**
- `getUnrecognizedReleaseFormats(release)` in `utils/releaseFormats.ts`
(normalised + deduped, same shape as `getReleaseFormats`).
- `ReleaseCell` `format_content_type`: when there is **no** recognised
format but the indexer named one, render an amber `MP4 Unsupported`
badge (compact view: amber `MP4`) with tooltip *"Unsupported format
(MP4) - Shelfmark cannot process this release"*. When a recognised
format exists the existing badge is untouched, even if extra unknown
tokens were present.

Only the chip changes — the download button still works, so a user can
still grab and hand-process the files if they want to. Happy to disable
the button instead if you'd prefer.

## Tests

- `tests/prowlarr/test_source.py`: `TestSplitMamFormats` (recognised /
unrecognised / mixed / no bracket / wrapper compat) and
`TestUnrecognizedFormatOnRelease` (lands in `extra`, empty when
recognised, absent without format detection).
- `src/frontend/src/tests/releaseFormats.test.ts`: 3 cases for the new
helper.
- `ruff check` clean; `pytest tests/prowlarr -m "not integration"` 511
passed; `tsc --noEmit`, `oxlint --deny warnings`, `vitest` all clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-24 17:50:27 -04:00
jakesterpdxandClaude Fable 5 65e2e3be20 feat(audiobook): recognise .mp4 as an audiobook format (#1264)
## Problem

Some trackers — MyAnonamouse in particular — distribute AAC audiobooks
as per-chapter `.mp4` files. That's the same ISO-BMFF container as
`.m4a`/`.m4b`, just with the generic extension (`ftyp isom`,
audio-only).

Today those releases:
1. show up in Prowlarr search results with **no format chip** — only the
generic "Audiobook" icon, because no format could be inferred;
2. download successfully; then
3. fail post-processing with **"No book files found in download"**,
because `.mp4` isn't in `AUDIOBOOK_FORMATS` (`shelfmark/core/utils.py`).

Real example: MAM #627978, *The Martian* (Andy Weir, 2020 edition) — 142
files `0001 … 0142 Andy Weir (2020) The Martian.mp4` + `cover.jpg`, 305
MB. Every file is a valid AAC-in-MP4 chapter.

Adding `mp4` to `SUPPORTED_AUDIOBOOK_FORMATS` in `settings.json` doesn't
help since the hard-coded tuple is what post-processing scans against.

## Change

- Add `"mp4"` to `AUDIOBOOK_FORMATS` (single source of truth — settings
UI, Prowlarr parsing, IRC parser, archive extraction and post-download
scan all derive from it), with a comment explaining why.
- Add `".mp4"` to the two hand-maintained debrid `_BOOK_EXTENSIONS`
lists (AllDebrid / Real-Debrid) so file selection matches.
- Slot `mp4` into the IRC `AUDIOBOOK_FORMAT_PRIORITY` table right after
`m4a` (same container family).
- Update the documented default in `docs/environment-variables.md`.
- New regression test
`test_audiobook_multifile_mp4_chapters_are_book_files` modelled on the
existing multi-file usenet test.

### Note for existing installs

The legacy-default migration only widens configs that still hold the old
`m4b,mp3` list, so users on the current widened default won't pick up
`mp4` automatically — they'll need to tick it in Settings → Audiobook
formats. New installs get it by default. Happy to extend the migration
if you'd rather it be automatic.

## Testing

- `ruff check` / `ruff format --check`: clean
- `pytest tests/core tests/config tests/irc tests/prowlarr
tests/download -m "not integration and not e2e"`: 2296 passed, new test
+ `test_audiobook_format_consistency.py` all green. The 10 failures in
`test_entrypoint_permissions.py` / `test_orchestrator_stall.py`
reproduce identically on untouched `main` on macOS and are unrelated.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-24 17:45:54 -04:00
CaliBrain ddc26f01b6 fix(download): escalating per-host cooldown on HTTP 429 (#1263)
Anna's Archive 429-throttles the source IP after repeated automated
requests.
The bypasser could clear the DDoS-Guard challenge but not the 429, so
each retry
re-solved, re-spawned Chrome, and rotated mirrors that share the same IP
- a
costly loop that never converged.

Add a process-global, per-host cooldown that escalates 2 -> 5 -> 10 ->
15 -> 30
minutes each time a host 429s again after its window elapsed, resetting
after a
long clear gap. Mirror selection skips cooling hosts and the bypasser
refuses to
solve one, so a throttled host fails fast instead of storming the
solver.
2026-08-24 13:08:07 -04:00
97 changed files with 6364 additions and 294 deletions
+1
View File
@@ -236,6 +236,7 @@ pyrightconfig.json
*.local.*
AGENTS.md
.claude/
CLAUDE.md
.nvmrc
.playwright-mcp/
frontend-dist/
+21
View File
@@ -276,6 +276,27 @@ class DownloadHandler(ABC):
pass
```
### Optional: Listing Files Before Download
Some releases bundle several books (a whole-series torrent). Shelfmark inspects a
release before queueing it so the user can review how it will be split into books.
Override `list_files` when your source can enumerate a release's files without
downloading it; the default returns `None`, which the UI reports as "can't inspect":
```python
from shelfmark.download.postprocess.packs import PackFile
def list_files(self, release_data: dict[str, Any]) -> list[PackFile] | None:
"""Return the release's files (release-relative paths + sizes), or None."""
torrent_bytes = ... # e.g. fetch the .torrent, or scrape the indexer's detail page
return extract_file_list_from_torrent(torrent_bytes) # from download.clients.torrent_utils
```
`release_data` is the same payload the frontend sends to `/api/releases/download`
(`source_id`, `download_url`, `content_type`, `series_name`, ...). Built-in examples:
Prowlarr parses the `.torrent` it already fetches (magnet-only releases return
`None`), and AudiobookBay reads the file table off its detail page.
### Download Method Parameters
| Parameter | Type | Description |
+47 -18
View File
@@ -247,7 +247,7 @@ Seconds since the last WireGuard handshake before the healthcheck bounces the tu
| `CALIBRE_WEB_URL` | Adds a navigation button to your book library (Calibre-Web Automated, Grimmory, etc). | string | _none_ |
| `AUDIOBOOK_LIBRARY_URL` | Adds a separate navigation button for your audiobook library (Audiobookshelf, Plex, etc). When both URLs are set, icons are shown instead of text. | string | _none_ |
| `SUPPORTED_FORMATS` | Book formats to include in search results. ZIP/RAR archives are extracted automatically and book files are used if found. | string (comma-separated) | `epub,mobi,azw3,fb2,djvu,cbz,cbr` |
| `SUPPORTED_AUDIOBOOK_FORMATS` | Audiobook formats to include in search results. ZIP/RAR archives are extracted automatically and audiobook files are used if found. | string (comma-separated) | `m4b,mp3,m4a,flac,ogg,wma,aac,wav,opus,zip,rar` |
| `SUPPORTED_AUDIOBOOK_FORMATS` | Audiobook formats to include in search results. ZIP/RAR archives are extracted automatically and audiobook files are used if found. | string (comma-separated) | `m4b,mp3,m4a,mp4,flac,ogg,wma,aac,wav,opus,zip,rar` |
| `BOOK_LANGUAGE` | Default language filter for searches. | string (comma-separated) | `en` |
<details>
@@ -296,16 +296,7 @@ Book formats to include in search results. ZIP/RAR archives are extracted automa
Audiobook formats to include in search results. ZIP/RAR archives are extracted automatically and audiobook files are used if found.
- **Type:** string (comma-separated)
- **Default:** `m4b,mp3,m4a,flac,ogg,wma,aac,wav,opus,zip,rar`
#### `BOOK_LANGUAGE`
**Default Book Languages**
Default language filter for searches.
- **Type:** string (comma-separated)
- **Default:** `en`
- **Default:** `m4b,mp3,m4a,mp4,flac,ogg,wma,aac,wav,opus,zip,rar`
</details>
@@ -314,6 +305,7 @@ Default language filter for searches.
| Variable | Description | Type | Default |
|----------|-------------|------|---------|
| `SEARCH_MODE` | How you want to search for and download books. | string (choice) | `universal` |
| `BOOK_LANGUAGE` | Default language filter for searches. Users can override this for their own account. | string (comma-separated) | `en` |
| `AA_DEFAULT_SORT` | Default sort order for search results. | string (choice) | `relevance` |
| `SHOW_RELEASE_SOURCE_LINKS` | Show clickable release-source links in release and details modals. Metadata provider links stay enabled. | boolean | `true` |
| `SHOW_COMBINED_SELECTOR` | Show the option to search for and download both a book and audiobook together. | boolean | `true` |
@@ -337,6 +329,15 @@ How you want to search for and download books.
- **Default:** `universal`
- **Options:** `direct` (Direct), `universal` (Universal)
#### `BOOK_LANGUAGE`
**Default Book Languages**
Default language filter for searches. Users can override this for their own account.
- **Type:** string (comma-separated)
- **Default:** `en`
#### `AA_DEFAULT_SORT`
**Default Sort Order**
@@ -749,6 +750,7 @@ Automatically open the downloads sidebar when a new download is queued.
Automatically download completed files to your browser for the selected content types.
- **Type:** string (comma-separated)
- **Default:** _empty list_
#### `MAX_CONCURRENT_DOWNLOADS`
@@ -1313,8 +1315,9 @@ Apply per-indexer seed time and ratio preferences from Prowlarr when sending tor
| Variable | Description | Type | Default |
|----------|-------------|------|---------|
| `NEWZNAB_ENABLED` | Enable searching for books via a Newznab-compatible indexer | boolean | `false` |
| `NEWZNAB_URL` | Base URL of your Newznab indexer or aggregator | string | _none_ |
| `NEWZNAB_API_KEY` | Your Newznab API key (leave blank if not required) | string (secret) | _none_ |
| `NEWZNAB_INDEXERS` | Named Newznab connections. Each row accepts `name`, `url`, and `api_key`. | JSON array | `[]` |
| `NEWZNAB_URL` | Legacy single-indexer URL, used when `NEWZNAB_INDEXERS` is empty | string | _none_ |
| `NEWZNAB_API_KEY` | Legacy single-indexer API key | string (secret) | _none_ |
| `NEWZNAB_EBOOK_CATEGORIES` | Newznab category IDs searched for ebooks. Most indexers use the standard 7000, but some use custom IDs. Leave empty to use 7000. | string (comma-separated) | `7000` |
| `NEWZNAB_AUDIOBOOK_CATEGORIES` | Newznab category IDs searched for audiobooks. Most indexers use the standard 3030, but some use custom IDs. Leave empty to use 3030. | string (comma-separated) | `3030` |
| `NEWZNAB_AUTO_EXPAND` | Automatically retry search without category filtering if no results are found | boolean | `false` |
@@ -1331,21 +1334,36 @@ Enable searching for books via a Newznab-compatible indexer
- **Type:** boolean
- **Default:** `false`
#### `NEWZNAB_INDEXERS`
**Named Indexers**
Configure multiple named Newznab-compatible indexers. The name is shown beside each search result. For environment-based configuration, provide a JSON array:
```json
[
{"name":"NZBGeek","url":"https://api.nzbgeek.info","api_key":"..."},
{"name":"DrunkenSlug","url":"https://drunkenslug.com","api_key":"..."}
]
```
- **Type:** JSON array
- **Default:** `[]`
#### `NEWZNAB_URL`
**Newznab URL**
**Legacy Newznab URL**
Base URL of your Newznab indexer or aggregator
Single-indexer fallback used only when `NEWZNAB_INDEXERS` is empty.
- **Type:** string
- **Default:** _none_
- **Required:** Yes
#### `NEWZNAB_API_KEY`
**API Key**
**Legacy API Key**
Your Newznab API key (leave blank if not required)
API key for the legacy Newznab URL.
- **Type:** string (secret)
- **Default:** _none_
@@ -2161,6 +2179,7 @@ Enable Moly.hu as a metadata provider for book searches
| `SOURCE_PRIORITY` | Fallback sources, may have waiting. Requires bypasser. Drag to reorder. | JSON array | _see UI for defaults_ |
| `MAX_RETRY` | Maximum retry attempts for failed downloads. | number | `10` |
| `DEFAULT_SLEEP` | Wait time between download retry attempts. | number | `5` |
| `RELEASE_SEARCH_TIMEOUT` | How long one release search may run before it gives up and reports why. A first search on a cold start pays for a browser solve, so leave room for one. If you use a reverse proxy, its read timeout should be at least this high or it will cut the search off with a 504 first. | number | `300` |
| `AA_CONTENT_TYPE_ROUTING` | Override destination based on content type metadata. | boolean | `false` |
| `AA_CONTENT_TYPE_DIR_FICTION` | Fiction Books | string | _none_ |
| `AA_CONTENT_TYPE_DIR_NON_FICTION` | Non-Fiction Books | string | _none_ |
@@ -2239,6 +2258,16 @@ Wait time between download retry attempts.
- **Default:** `5`
- **Constraints:** min: 1, max: 60
#### `RELEASE_SEARCH_TIMEOUT`
**Release Search Timeout (seconds)**
How long one release search may run before it gives up and reports why. A first search on a cold start pays for a browser solve, so leave room for one. If you use a reverse proxy, its read timeout should be at least this high or it will cut the search off with a 504 first.
- **Type:** number
- **Default:** `300`
- **Constraints:** min: 30, max: 1800
#### `AA_CONTENT_TYPE_ROUTING`
**Enable Content-Type Routing**
+10 -1
View File
@@ -30,7 +30,7 @@ Requires mounting your Calibre-Web `app.db` to `/auth/app.db`.
Admins can configure per-user settings by editing a user in the user management panel. Non-admin users can also edit their own settings through **My Account** (accessible from the user menu). Admins control which sections are visible in My Account via the **Visible Self-Settings Sections** option.
There are three categories of per-user settings:
There are four categories of per-user settings:
### Delivery Preferences
@@ -42,6 +42,15 @@ Override where a user's downloads are sent. Options depend on the global output
- **BookLore library/path** — Per-user BookLore target (when using BookLore output mode)
- **Email recipient** — Per-user email address (when using Email output mode)
### Search Preferences
Override how a user searches, on top of the global search defaults:
- **Search mode** — Direct or Universal for this user
- **Default book languages** — The languages a user's searches fall back to when they don't pick one themselves. Useful for a shared instance where readers want different languages.
- **Metadata providers** — Book, audiobook, and combined-mode provider for this user
- **Default release sources** — The release tab opened first for books and audiobooks
### Notifications
Users can configure personal notification routes, separate from the global notification settings. Each route targets a URL (e.g. an Apprise-compatible endpoint) and can be scoped to specific event types or all events.
+1 -1
View File
@@ -32,7 +32,7 @@ dependencies = [
browser = [
"pyvirtualdisplay",
"pyautogui",
"seleniumbase==4.52.1",
"seleniumbase==4.52.2",
"python-xlib",
]
+1
View File
@@ -147,6 +147,7 @@ See the full [Environment Variables Reference](docs/environment-variables.md) fo
Some of the additional options available in Settings:
- **Prowlarr** - Configure indexers and download clients to download books and audiobooks
- **Additional audiobook sources** - Configure additional sources for audiobook discovery
- **Direct Download mirrors** - Supply your own Anna's Archive mirror URLs; Auto mode tries them in the order listed. The `annas-archive.is` domain does not currently work as a source — use `annas-archive.gl` instead (checked August 2026; mirror availability changes)
- **IRC** - Add details for IRC book sources and download directly from the UI. Most networks serve audiobooks from the same channel as ebooks (on `irc.irchighway.net` that's `#ebooks`, while `#bookz` is effectively inactive), so leave the separate audiobook channel blank unless your network actually indexes one. IRC audiobooks usually arrive as ZIP/RAR archives — keep those enabled under Supported Audiobook Formats or the releases are filtered out of results
- **Library Link** - Add a link to your Calibre-Web or Grimmory instance in the UI header
- **File processing** - Customiseable download paths, file renaming and directory creation with template-based renaming
+22 -1
View File
@@ -71,11 +71,18 @@ def _get_full_cookie_domains() -> set[str]:
return {_get_base_domain(domain) for domain in get_zlib_cookie_domains()}
def _replay_per_check_cookies() -> bool:
"""Whether the per-check trio is kept rather than dropped (see env.py)."""
from shelfmark.config import env
return env.DDG_REPLAY_PER_CHECK_COOKIES
def _should_extract_cookie(name: str, *, extract_all: bool) -> bool:
"""Determine if a cookie should be extracted based on its name."""
# Checked before extract_all: a per-check token is wrong to replay for every
# domain, including the full-session ones.
if name in DDG_EPHEMERAL_COOKIE_NAMES:
if name in DDG_EPHEMERAL_COOKIE_NAMES and not _replay_per_check_cookies():
return False
if extract_all:
return True
@@ -138,9 +145,11 @@ def store_extracted_cookies(
extract_all = base_domain in _get_full_cookie_domains()
cookies_found: dict[str, dict[str, Any]] = {}
dropped: list[str] = []
for cookie in cookies:
name = _cookie_field(cookie, "name") or ""
if not _should_extract_cookie(name, extract_all=extract_all):
dropped.append(name)
continue
secure = _cookie_field(cookie, "secure")
cookies_found[name] = {
@@ -152,6 +161,18 @@ def store_extracted_cookies(
"httpOnly": True,
}
# Names only, never values. Which cookies a solve won, and which of them were held
# back, is the evidence needed to settle what DDoS-Guard actually treats as clearance
# (issue #1276) - and without it a debug log shows a solve succeeding and the next
# request being challenged with nothing in between to explain why.
logger.debug(
"Solve on %s won %s; keeping %s; dropping %s",
base_domain,
sorted({_cookie_field(c, "name") or "" for c in cookies}),
sorted(cookies_found),
sorted(set(dropped)) or "nothing",
)
if not cookies_found:
return
+163 -13
View File
@@ -83,11 +83,14 @@ _HELPER_RESULT_POLL_SECONDS = 0.05
_HELPER_SHUTDOWN_GRACE_SECONDS = 15.0
_HELPER_IDLE_TIMEOUT_DEFAULT = 180.0
_PARENT_WATCHDOG_INTERVAL_SECONDS = 5.0
# How much of ffmpeg's stderr to quote when reporting that it died.
_FFMPEG_ERROR_TAIL_CHARS = 500
class _DisplayState(TypedDict):
ffmpeg: subprocess.Popen[bytes] | None
ffmpeg_output: Path | None
ffmpeg_error_log: Path | None
class _PageWithWindowRect(Protocol):
@@ -101,6 +104,7 @@ class _BrowserWithWindowRectPage(Protocol):
DISPLAY: _DisplayState = {
"ffmpeg": None,
"ffmpeg_output": None,
"ffmpeg_error_log": None,
}
LOCKED = threading.Lock()
_PROC_ROOT = Path("/proc")
@@ -618,6 +622,25 @@ BYPASS_METHODS = [
MAX_CONSECUTIVE_SAME_CHALLENGE = 3
# How many method attempts one _bypass() pass may make. Deliberately *not* MAX_RETRY:
# that value is already the outer page-load retry in _run_bypass_in_current_process, and
# reading it here too squared the budget - the default 10 meant 10 page loads x 4 methods
# = 40 solve attempts on one browser, which overruns the worker deadline and reports
# `TimeoutError` instead of a plain "bypass failed". One full pass through the methods
# plus a spare is all this loop can use anyway: the stuck-challenge guard below aborts at
# len(BYPASS_METHODS) + 1, so a larger number here only ever showed up in the logs.
_BYPASS_METHOD_ATTEMPTS = len(BYPASS_METHODS) + 1
# The undisturbed window a passive challenge gets before any method runs. Sized off the
# real thing: a desktop browser clears Anna's Archive's DDoS-Guard JS check in under 10s.
_PASSIVE_SOLVE_SECONDS = 15.0
_PASSIVE_SOLVE_POLL_SECONDS = 1.0
# Head-room the retry loop leaves itself so it can return a real failure rather than be
# cancelled at the worker deadline. Enough for the pass in flight to unwind and the
# browser to close.
_RESERVE_FOR_CLEAN_FAILURE_SECONDS = 60.0
def _check_cancellation(cancel_flag: Event | None, message: str) -> None:
"""Check if cancellation was requested and raise if so."""
@@ -627,13 +650,26 @@ def _check_cancellation(cancel_flag: Event | None, message: str) -> None:
raise BypassCancelledError(msg)
async def _wait_for_passive_solve(page: Any, cancel_flag: Event | None = None) -> bool:
"""Poll for a challenge that clears itself, without touching the page.
Returns True as soon as the page looks bypassed, False once the window is spent.
"""
logger.info("Waiting up to %.0fs for the challenge to clear itself...", _PASSIVE_SOLVE_SECONDS)
deadline = time.monotonic() + _PASSIVE_SOLVE_SECONDS
while time.monotonic() < deadline:
_check_cancellation(cancel_flag, "Bypass cancelled while waiting for a passive solve")
await asyncio.sleep(_PASSIVE_SOLVE_POLL_SECONDS)
if await _is_bypassed(page):
return True
return False
async def _bypass(
page: Any, max_retries: int | None = None, cancel_flag: Event | None = None
) -> bool:
"""Attempt to bypass Cloudflare/DDOS-Guard protection using multiple methods."""
max_retries = (
max_retries if max_retries is not None else _coerce_positive_int(app_config.MAX_RETRY, 10)
)
max_retries = max_retries if max_retries is not None else _BYPASS_METHOD_ATTEMPTS
last_challenge_type = None
consecutive_same_challenge = 0
@@ -651,6 +687,20 @@ async def _bypass(
challenge_type = await _detect_challenge_type(page)
logger.debug("Challenge detected: %s", challenge_type)
# Give a passive check the undisturbed window it needs before touching the page.
# DDoS-Guard's JS check on Anna's Archive has no click target: it runs, then
# navigates on its own - a desktop browser clears it in well under 15s. Every
# method below either clicks a selector that is not there or reloads, and a reload
# restarts an in-flight check (which DDoS-Guard also throttles), so going straight
# to them meant the one thing that actually solves this challenge was the one
# thing never tried. Costs one 15s window per solve against a minutes-long budget,
# and a challenge that needs interaction simply falls through to the methods.
if try_count == 0 and challenge_type != "none":
if await _wait_for_passive_solve(page, cancel_flag):
logger.info("Bypass successful: %s challenge cleared itself", challenge_type)
return True
logger.debug("Challenge did not clear on its own; trying bypass methods")
# No challenge detected but page doesn't look bypassed - wait and retry
if challenge_type == "none":
logger.info("No challenge detected, waiting for page to settle...")
@@ -810,14 +860,33 @@ async def _get(url: str, driver: Any, cancel_flag: Event | None = None) -> str:
def _run_bypass_in_current_process(url: str, retry: int, cancel_flag: Event | None = None) -> str:
"""Run the CDP bypass in the current process."""
timeout = (
_CHILD_BYPASS_TIMEOUT_SECONDS
if os.environ.get(_BYPASS_CHILD_ENV) == "1"
else _IN_PROCESS_BYPASS_TIMEOUT_SECONDS
)
async def _run_bypass() -> str:
driver = None
# Stop retrying while there is still time to say so. A challenge nothing can solve
# would otherwise spend every one of `retry` passes and be cut off mid-pass by the
# worker deadline, which surfaces to the caller as `RuntimeError: TimeoutError` -
# a message that says nothing about protection and sent users looking at their
# reverse proxy. Giving up a pass early returns the real "bypass failed" instead.
deadline = time.monotonic() + timeout - _RESERVE_FOR_CLEAN_FAILURE_SECONDS
try:
driver = await _create_cdp_browser(url)
for attempt in range(retry):
_check_cancellation(cancel_flag, "Bypass cancelled before attempt")
if attempt > 0 and time.monotonic() >= deadline:
logger.warning(
"Bypass budget spent after %s/%s attempts; giving up on %s",
attempt,
retry,
url,
)
break
try:
result = await _get(url, driver, cancel_flag)
@@ -838,7 +907,7 @@ def _run_bypass_in_current_process(url: str, retry: int, cancel_flag: Event | No
await _close_cdp_driver(driver)
driver = await _create_cdp_browser(url)
logger.error("Bypass failed after %s attempts", retry)
logger.error("Bypass failed for %s", url)
return ""
finally:
if driver:
@@ -852,12 +921,9 @@ def _run_bypass_in_current_process(url: str, retry: int, cancel_flag: Event | No
# one call and closes it on the way out, so a helper serving many requests would build
# and tear down a loop per bypass and would carry no deadline of its own. The worker's
# loop lives in a thread, outlives any single bypass, and cancels the coroutine when the
# deadline passes.
timeout = (
_CHILD_BYPASS_TIMEOUT_SECONDS
if os.environ.get(_BYPASS_CHILD_ENV) == "1"
else _IN_PROCESS_BYPASS_TIMEOUT_SECONDS
)
# deadline passes. `_run_bypass` aims to finish inside this same budget of its own
# accord, so reaching this deadline now means a wedged session rather than a stubborn
# challenge - which is the only case worth reporting as a timeout.
return _CDP_WORKER.run(_run_bypass(), timeout=timeout)
@@ -1155,6 +1221,20 @@ def get(url: str, retry: int | None = None, cancel_flag: Event | None = None) ->
if cached_result:
return cached_result
# Re-checked after the cached attempt, not just in get_bypassed_page: that check
# ran before the queue, and this call may have spent minutes holding for LOCKED
# while another request collected a 429 (or collected one itself, just above).
# A solve cannot clear a throttle - the challenge renders, the solve "succeeds",
# and the cleared request is refused again while the backoff is renewed.
remaining = network.host_cooldown_remaining(url)
if remaining > 0:
hostname = urlparse(url).hostname or url
msg = (
f"{hostname} is rate-limited (429); skipping bypass for ~{remaining:.0f}s "
"until the cooldown clears."
)
raise network.RateLimitedError(msg)
if env.DOCKERMODE and os.environ.get(_BYPASS_CHILD_ENV) != "1":
return _get_via_subprocess(url, retry, cancel_flag)
return _run_bypass_in_current_process(url, retry, cancel_flag)
@@ -1339,13 +1419,48 @@ def _start_ffmpeg_recording(display: str) -> None:
"-an",
output_file.as_posix(),
"-nostats",
# Was "0", which discards everything including the reason it could not start.
# Recordings have been arriving empty with no explanation anywhere: on issue
# #1276 all three of a session's recordings were gone and the log said only
# "FFmpeg already stopped", because ffmpeg exits before creating the file when
# it cannot open the X display. Errors only - this is a debug-mode recorder, not
# something to make chatty.
"-loglevel",
"0",
"error",
]
logger.debug("Starting FFmpeg recording to %s", output_file)
logger.debug_trace(f"FFmpeg command: {' '.join(ffmpeg_cmd)}")
DISPLAY["ffmpeg"] = subprocess.Popen(ffmpeg_cmd)
# Kept beside the recording so it travels in the debug bundle, which is the only
# place anyone will look for it. A file rather than a pipe: nothing here would drain
# a pipe, and a full one would wedge ffmpeg partway through a capture.
error_log = output_file.with_suffix(".ffmpeg.log")
try:
stderr_handle = error_log.open("wb")
except OSError as exc:
logger.debug("Could not open FFmpeg error log %s: %s", error_log, exc)
stderr_handle = None
DISPLAY["ffmpeg"] = subprocess.Popen(
ffmpeg_cmd, stderr=stderr_handle, stdout=subprocess.DEVNULL
)
if stderr_handle is not None:
# The child holds its own descriptor; this one has done its job.
stderr_handle.close()
DISPLAY["ffmpeg_output"] = output_file
DISPLAY["ffmpeg_error_log"] = error_log
def _ffmpeg_error_summary() -> str:
"""What ffmpeg wrote to stderr, for the log line that reports it died."""
error_log = DISPLAY.get("ffmpeg_error_log")
if not error_log:
return "No FFmpeg error log was captured."
try:
text = Path(error_log).read_text(encoding="utf-8", errors="replace").strip()
except OSError as exc:
return f"FFmpeg error log unreadable ({exc})."
if not text:
return f"FFmpeg logged nothing to {error_log}."
return f"FFmpeg said: {text[-_FFMPEG_ERROR_TAIL_CHARS:]}"
def _stop_ffmpeg_recording() -> None:
@@ -1357,9 +1472,17 @@ def _stop_ffmpeg_recording() -> None:
if not proc:
return
if proc.poll() is not None:
logger.debug("FFmpeg already stopped")
# Not "already stopped" - ffmpeg was asked to record until now and is gone, so
# the recording for this bypass does not exist. Say so, with the reason, rather
# than leaving an empty recording/ directory to be discovered later.
logger.warning(
"FFmpeg exited early (code %s); no recording for this bypass. %s",
proc.returncode,
_ffmpeg_error_summary(),
)
DISPLAY["ffmpeg"] = None
DISPLAY["ffmpeg_output"] = None
DISPLAY["ffmpeg_error_log"] = None
return
try:
proc.send_signal(signal.SIGINT)
@@ -1374,6 +1497,7 @@ def _stop_ffmpeg_recording() -> None:
proc.kill()
DISPLAY["ffmpeg"] = None
DISPLAY["ffmpeg_output"] = None
DISPLAY["ffmpeg_error_log"] = None
def _try_with_cached_cookies(url: str, hostname: str) -> str | None:
@@ -1400,6 +1524,20 @@ def _try_with_cached_cookies(url: str, hostname: str) -> str | None:
if response.status_code == HTTPStatus.OK:
logger.debug("Cached cookies worked, skipped Chrome bypass")
return response.text
if response.status_code == HTTPStatus.TOO_MANY_REQUESTS:
# Throttled, not challenged. The clearance is still good - the origin is
# rate-limiting this IP and would answer 429 to a browser holding the very
# same cookies. Discarding it here (as every other rejection does) meant a
# solve won seconds earlier was thrown away and the next query bought its
# own 20-60s browser solve, which is itself more traffic at a host that has
# just asked for less. Keep it, arm the backoff, and let the caller wait.
wait = network.note_rate_limited(url)
logger.debug(
"Cached cookies hit a 429 for %s; keeping them and backing off ~%.0fs",
url,
wait,
)
return None
logger.debug(
"Cached cookies rejected (%s) for %s; discarding them",
response.status_code,
@@ -1438,6 +1576,18 @@ def get_bypassed_page(
attempt_url = sel.rewrite(url)
hostname = urlparse(attempt_url).hostname or ""
# A 429 means the origin is throttling this IP; the challenge still renders, so a
# solve "succeeds" but the cleared request is rejected again and the throttle is only
# renewed. Never spend a minutes-long Chrome solve on a cooling-down host - fail fast
# so the caller waits the backoff out instead of looping the solve.
remaining = network.host_cooldown_remaining(attempt_url)
if remaining > 0:
msg = (
f"{hostname} is rate-limited (429); skipping bypass for ~{remaining:.0f}s "
"until the cooldown clears."
)
raise network.RateLimitedError(msg)
cached_result = _try_with_cached_cookies(attempt_url, hostname)
if cached_result:
return cached_result
+15
View File
@@ -203,6 +203,21 @@ ONBOARDING = string_to_bool(os.getenv("ONBOARDING", "true"))
_DEBUG_SKIP_SOURCES_RAW = os.getenv("DEBUG_SKIP_SOURCES", "").strip().lower()
DEBUG_SKIP_SOURCES = {s.strip() for s in _DEBUG_SKIP_SOURCES_RAW.split(",") if s.strip()}
# Debug: keep DDoS-Guard's __ddg8_/__ddg9_/__ddg10_ in the clearance store instead of
# dropping them after a solve.
#
# Which of DDoS-Guard's cookies actually *are* clearance is not settled. The store treats
# the trio as describing one check (client IP, timestamp, token) and drops them, on the
# reasoning that replaying a stale IP/timestamp is what re-arms the ?check=1 loop - see
# shelfmark.bypass.cookie_store. Field reports on issue #1276 point the other way: every
# request after a successful solve was challenged again, which is only consistent with
# what the store keeps not being sufficient clearance on its own.
#
# Deliberately env-only and off by default: this is a knob for reproducing the question
# against a live host, not a setting to offer users. Set it to true, solve once, and watch
# whether the next search still logs "Redirect loop detected".
DDG_REPLAY_PER_CHECK_COOKIES = string_to_bool(os.getenv("DDG_REPLAY_PER_CHECK_COOKIES", "false"))
# =============================================================================
# Legacy migration support - will be removed in future version
+24 -7
View File
@@ -430,13 +430,6 @@ def general_settings() -> list[SettingsField]:
options=_AUDIOBOOK_FORMAT_OPTIONS,
default=[*AUDIOBOOK_FORMATS, *ARCHIVE_FORMATS],
),
MultiSelectField(
key="BOOK_LANGUAGE",
label="Default Book Languages",
description="Default language filter for searches.",
options=_LANGUAGE_OPTIONS,
default=["en"],
),
]
@@ -474,6 +467,17 @@ def search_mode_settings() -> list[SettingsField]:
default="universal",
user_overridable=True,
),
MultiSelectField(
key="BOOK_LANGUAGE",
label="Default Book Languages",
description=(
"Default language filter for searches. Users can override this for their "
"own account."
),
options=_LANGUAGE_OPTIONS,
default=["en"],
user_overridable=True,
),
SelectField(
key="AA_DEFAULT_SORT",
label="Default Sort Order",
@@ -1556,6 +1560,19 @@ def download_source_settings() -> list[SettingsField]:
min_value=1,
max_value=60,
),
NumberField(
key="RELEASE_SEARCH_TIMEOUT",
label="Release Search Timeout (seconds)",
description=(
"How long one release search may run before it gives up and reports why. "
"A first search on a cold start pays for a browser solve, so leave room "
"for one. If you use a reverse proxy, its read timeout should be at least "
"this high or it will cut the search off with a 504 first."
),
default=300,
min_value=30,
max_value=1800,
),
HeadingField(
key="content_type_routing_heading",
title="Content-Type Routing",
+35 -4
View File
@@ -7,6 +7,7 @@ that talks to /api/admin/users endpoints.
from typing import Any
from shelfmark.core.languages import normalize_language
from shelfmark.core.request_policy import (
get_source_content_type_capabilities,
parse_policy_mode,
@@ -61,7 +62,7 @@ _SELF_SETTINGS_SECTION_OPTIONS = [
{
"value": "search",
"label": "Search Preferences",
"description": "Show personal search mode and provider settings.",
"description": "Show personal search mode, language, and provider settings.",
},
{
"value": "notifications",
@@ -77,8 +78,9 @@ _SEARCH_PREFERENCE_PROVIDER_KEYS = {
"METADATA_PROVIDER_AUDIOBOOK",
"METADATA_PROVIDER_COMBINED",
}
_SEARCH_PREFERENCE_VALIDATABLE_KEYS = {
SEARCH_PREFERENCE_VALIDATABLE_KEYS = {
"SEARCH_MODE",
"BOOK_LANGUAGE",
"DEFAULT_RELEASE_SOURCE",
"DEFAULT_RELEASE_SOURCE_AUDIOBOOK",
"SHOW_COMBINED_SELECTOR",
@@ -178,14 +180,43 @@ def _get_request_policy_rule_columns() -> list[dict[str, object]]:
]
def _validate_book_languages(value: Any) -> tuple[Any, str | None]:
"""Validate a per-user default language list against the known languages.
Accepts the list the settings UI sends as well as a comma-separated string, so an
API client can spell the value the way the env var does. Blank entries are skipped
rather than rejected, which makes "" and "en," mean the same as [] and ["en"]. An
empty result is a deliberate override meaning "no default language filter", so it
is kept as-is; ``None`` clears the override further up the chain.
"""
entries = value.split(",") if isinstance(value, str) else value
if not isinstance(entries, (list, tuple)):
return value, "BOOK_LANGUAGE must be a list of language codes"
normalized: list[str] = []
for entry in entries:
if entry is None or (isinstance(entry, str) and not entry.strip()):
continue
code = normalize_language(entry)
if code is None:
return value, f"BOOK_LANGUAGE contains an unsupported language: {entry}"
if code not in normalized:
normalized.append(code)
return normalized, None
def validate_search_preference_value(key: str, value: Any) -> tuple[Any, str | None]:
"""Validate and normalize a search preference value for user overrides."""
if key not in _SEARCH_PREFERENCE_VALIDATABLE_KEYS:
if key not in SEARCH_PREFERENCE_VALIDATABLE_KEYS:
return value, None
if value is None:
return None, None
if key == "BOOK_LANGUAGE":
return _validate_book_languages(value)
normalized_value = str(value).strip()
if key == "SEARCH_MODE":
@@ -298,7 +329,7 @@ def _on_save_users(values: dict[str, object]) -> dict[str, object]:
}
values["REQUEST_POLICY_RULES"] = normalized_rules
for key in _SEARCH_PREFERENCE_VALIDATABLE_KEYS:
for key in SEARCH_PREFERENCE_VALIDATABLE_KEYS:
if key not in values:
continue
normalized_value, validation_error = validate_search_preference_value(key, values[key])
+7 -8
View File
@@ -11,7 +11,10 @@ from shelfmark.config.notifications_settings import (
is_valid_notification_url,
normalize_notification_routes,
)
from shelfmark.config.users_settings import validate_search_preference_value
from shelfmark.config.users_settings import (
SEARCH_PREFERENCE_VALIDATABLE_KEYS,
validate_search_preference_value,
)
from shelfmark.core.config import config as app_config
from shelfmark.core.request_policy import parse_policy_mode, validate_policy_rules
from shelfmark.core.settings_registry import load_config_file
@@ -91,13 +94,9 @@ def validate_user_settings(
if search_validation_error:
errors.append(search_validation_error)
continue
if key in {
"SEARCH_MODE",
"METADATA_PROVIDER",
"METADATA_PROVIDER_AUDIOBOOK",
"DEFAULT_RELEASE_SOURCE",
"DEFAULT_RELEASE_SOURCE_AUDIOBOOK",
}:
# Every key the search validator recognises keeps its normalized value;
# a hand-maintained subset here silently dropped normalization for the rest.
if key in SEARCH_PREFERENCE_VALIDATABLE_KEYS:
valid[key] = normalized_search_value
continue
+6
View File
@@ -136,6 +136,12 @@ class DownloadTask:
default_factory=dict
) # Per-output parameters (e.g. email recipient)
# Multi-book packs: one release holding several books. `book_plan` is the split the
# user approved before download (list of {title, series_position, year, files});
# `multi_book` asks post-processing to split heuristically when no plan exists.
multi_book: bool = False
book_plan: list[dict[str, Any]] | None = None
# User association (multi-user support)
user_id: int | None = None # DB user ID who queued this download
username: str | None = None # Username for {User} template variable
+102
View File
@@ -0,0 +1,102 @@
"""Pre-download release inspection: list a release's files and plan a multi-book split."""
from __future__ import annotations
from typing import TYPE_CHECKING, Any
from flask import jsonify, request
from shelfmark.core.logger import setup_logger
from shelfmark.core.utils import is_audiobook
from shelfmark.download.postprocess.packs import PackFile, PackPlan, plan_pack
from shelfmark.download.postprocess.policy import (
get_supported_audiobook_formats,
get_supported_formats,
)
from shelfmark.release_sources import get_handler
if TYPE_CHECKING:
from collections.abc import Callable
from flask import Flask, Response
logger = setup_logger(__name__)
_INSPECT_ERRORS = (OSError, RuntimeError, ValueError, TypeError, KeyError, AttributeError)
NOT_INSPECTABLE_REASON = "This source cannot list the release's files before downloading"
def _serialize_plan(plan: PackPlan) -> dict[str, Any]:
return {
"is_pack": plan.is_pack,
"ignored": plan.ignored,
"books": [
{
"title": book.title,
"series_position": book.series_position,
"year": book.year,
"files": book.files,
}
for book in plan.books
],
}
def inspect_release(data: dict[str, Any]) -> dict[str, Any]:
"""Build the inspect response for a release payload (same shape as a download)."""
source = str(data["source"])
handler = get_handler(source)
try:
files: list[PackFile] | None = handler.list_files(data)
except _INSPECT_ERRORS as exc:
logger.warning(
"Could not list files for %s release %s: %s", source, data.get("source_id"), exc
)
return {"inspected": False, "reason": str(exc), "files": [], "plan": None}
if files is None:
return {"inspected": False, "reason": NOT_INSPECTABLE_REASON, "files": [], "plan": None}
content_type = data.get("content_type")
supported = (
get_supported_audiobook_formats()
if is_audiobook(content_type if isinstance(content_type, str) else None)
else get_supported_formats()
)
series_name = data.get("series_name")
author_name = data.get("author")
plan = plan_pack(
files,
supported_extensions=set(supported),
series_name=series_name if isinstance(series_name, str) else None,
author_name=author_name if isinstance(author_name, str) else None,
)
return {
"inspected": True,
"reason": None,
"files": [{"path": f.path, "size": f.size} for f in files],
"plan": _serialize_plan(plan),
}
def register_release_inspect_routes(
app: Flask,
login_required: Callable[..., Any],
) -> None:
"""Register POST /api/releases/inspect."""
@app.route("/api/releases/inspect", methods=["POST"])
@login_required
def api_inspect_release() -> Response | tuple[Response, int]:
data = request.get_json(silent=True)
if not isinstance(data, dict):
return jsonify({"error": "No data provided"}), 400
if not data.get("source_id"):
return jsonify({"error": "source_id is required"}), 400
if not data.get("source"):
return jsonify({"error": "source is required"}), 400
try:
get_handler(str(data["source"]))
except ValueError as exc:
return jsonify({"error": str(exc)}), 400
return jsonify(inspect_release(data))
+134
View File
@@ -0,0 +1,134 @@
"""A wall-clock budget for one release search, enforced through the existing cancel flag.
`/api/releases` is synchronous: the browser waits on it while the search runs. Nothing
bounded that wait, and the bypasser's own worst case is minutes long
(`internal_bypasser.max_duration_seconds()`), so a search that ran into an unsolvable
protection challenge outlived every reverse proxy in front of it. The user then saw
"Server unavailable (504)" - a gateway timeout that says nothing about what went wrong
and points the blame at their proxy config. See issue #1276.
The budget is expressed as the cancel flag the download path already understands: an
Event armed by a timer. `html_get_page`, the bypassers and the helper subprocess all poll
it, so an expired budget stops a solve already in flight rather than only refusing the
next one. When it trips, the search fails with a message that names the real cause.
Scoped to a context variable so it applies to the request that set it and to nothing else
- a queued download must keep its own, much longer, budget.
"""
from __future__ import annotations
import threading
import time
from contextlib import contextmanager
from contextvars import ContextVar
from typing import TYPE_CHECKING
from shelfmark.core.logger import setup_logger
if TYPE_CHECKING:
from collections.abc import Iterator
logger = setup_logger(__name__)
# What one search may spend. A first search on a cold start legitimately pays for a
# browser solve - jfmlima measured 60-120s for a successful one on Anna's Archive - so
# this cannot be as tight as a proxy's default read timeout without breaking working
# setups. It is instead well below the ~840s the bypass path could previously reach,
# which is what turned a failing challenge into a gateway timeout.
DEFAULT_SEARCH_BUDGET_SECONDS = 300.0
_MIN_SEARCH_BUDGET_SECONDS = 30.0
_MAX_SEARCH_BUDGET_SECONDS = 1800.0
# Raised to the caller when the budget runs out, so the API can say so plainly.
SEARCH_DEADLINE_MESSAGE = (
"The release search ran out of time (%.0fs). Anna's Archive is behind a protection "
"challenge the bypasser could not solve in that window. Raise the release search "
"timeout if your setup is simply slow."
)
class SearchDeadline:
"""A budget with an Event that trips when it expires."""
def __init__(self, budget_seconds: float) -> None:
self.budget_seconds = budget_seconds
self.expires_at = time.monotonic() + budget_seconds
# A plain threading.Event on purpose: this is handed on as a cancel flag, and
# that is the type the download path, the CDP worker thread and the bypass helper
# already poll.
self.event = threading.Event()
self._timer = threading.Timer(budget_seconds, self.event.set)
self._timer.daemon = True
def start(self) -> None:
self._timer.start()
def cancel(self) -> None:
self._timer.cancel()
@property
def remaining(self) -> float:
return max(0.0, self.expires_at - time.monotonic())
@property
def expired(self) -> bool:
return self.event.is_set() or self.remaining <= 0
_current: ContextVar[SearchDeadline | None] = ContextVar("search_deadline", default=None)
def budget_seconds() -> float:
"""The configured budget for one release search."""
from shelfmark.core.config import config as app_config
raw = app_config.get("RELEASE_SEARCH_TIMEOUT", DEFAULT_SEARCH_BUDGET_SECONDS)
if isinstance(raw, bool) or not isinstance(raw, int | float | str):
return DEFAULT_SEARCH_BUDGET_SECONDS
try:
value = float(raw)
except TypeError, ValueError:
return DEFAULT_SEARCH_BUDGET_SECONDS
if value <= 0:
return DEFAULT_SEARCH_BUDGET_SECONDS
return min(max(value, _MIN_SEARCH_BUDGET_SECONDS), _MAX_SEARCH_BUDGET_SECONDS)
@contextmanager
def search_deadline(budget: float | None = None) -> Iterator[SearchDeadline]:
"""Apply a budget to everything the calling context does."""
deadline = SearchDeadline(budget if budget is not None else budget_seconds())
token = _current.set(deadline)
deadline.start()
logger.debug("Release search budget: %.0fs", deadline.budget_seconds)
try:
yield deadline
finally:
deadline.cancel()
_current.reset(token)
def current() -> SearchDeadline | None:
"""The budget in force, or None outside a search."""
return _current.get()
def expired() -> bool:
"""Whether the budget in force has run out. False when there is no budget."""
deadline = _current.get()
return deadline is not None and deadline.expired
def cancel_event() -> threading.Event | None:
"""The Event that trips when the budget runs out, for use as a cancel flag."""
deadline = _current.get()
return deadline.event if deadline is not None else None
def deadline_message() -> str:
"""The failure to report when the budget has run out."""
deadline = _current.get()
budget = deadline.budget_seconds if deadline else DEFAULT_SEARCH_BUDGET_SECONDS
return SEARCH_DEADLINE_MESSAGE % budget
+58 -18
View File
@@ -7,6 +7,7 @@ from dataclasses import dataclass
from typing import TYPE_CHECKING
from shelfmark.core.config import config
from shelfmark.core.logger import setup_logger
from shelfmark.metadata_providers import (
BookMetadata,
build_localized_search_titles,
@@ -16,6 +17,8 @@ from shelfmark.metadata_providers import (
if TYPE_CHECKING:
from shelfmark.core.models import SearchFilters
logger = setup_logger(__name__)
MANUAL_QUERY_MAX_LEN = 256
@@ -52,30 +55,61 @@ class ReleaseSearchPlan:
return self.title_variants[0].query if self.title_variants else ""
def _normalize_languages(languages: list[str] | None) -> list[str] | None:
def _to_language_codes(values: Iterable[object], *, source: str) -> list[str] | None:
"""Resolve any spelling of a language to the ISO code the sources expect.
Anna's Archive matches `lang=` against ISO codes: `lang=english` is not a loose
spelling of `lang=en`, it is a facet value AA does not have, and it filters every
search down to nothing. Only the *per-user* override was normalised
(config.users_settings.validate), so a global BOOK_LANGUAGE=english - the spelling
the old docs used - reached the query verbatim and silently emptied every search
with no error anywhere. See issue #1276.
An entry that resolves to nothing is dropped with a warning rather than passed
through: searching unfiltered and saying so beats reporting "no results" for a book
the source is full of.
"""
from shelfmark.core.languages import normalize_language
codes: list[str] = []
unresolved: list[str] = []
for value in values:
text = str(value).strip() if value is not None else ""
if not text:
continue
if text.lower() == "all":
# An explicit "search every language", not a language.
return None
code = normalize_language(text)
if code is None:
unresolved.append(text)
continue
if code not in codes:
codes.append(code)
if unresolved:
logger.warning(
"Ignoring unrecognised language(s) in %s: %s. Use an ISO code such as 'en', "
"a three-letter code, or an English name like 'English'.",
source,
", ".join(unresolved),
)
return codes or None
def _normalize_languages(languages: list[str] | None, user_id: int | None) -> list[str] | None:
if not languages:
default = getattr(config, "BOOK_LANGUAGE", None)
default = config.get("BOOK_LANGUAGE", None, user_id=user_id)
if isinstance(default, str):
default_values: list[object] = [default]
elif isinstance(default, Iterable) and not isinstance(default, (bytes, bytearray, dict)):
default_values = list(default)
else:
return None
return [str(lang).strip() for lang in default_values if str(lang).strip()]
return _to_language_codes(default_values, source="BOOK_LANGUAGE")
normalized: list[str] = []
for lang in languages:
if not lang:
continue
s = str(lang).strip()
if not s:
continue
normalized.append(s)
if any(lang.lower() == "all" for lang in normalized):
return None
return normalized or None
return _to_language_codes(languages, source="the search request")
def _pick_search_author(book: BookMetadata) -> str:
@@ -102,9 +136,15 @@ def build_release_search_plan(
manual_query: str | None = None,
indexers: list[str] | None = None,
source_filters: SearchFilters | None = None,
user_id: int | None = None,
) -> ReleaseSearchPlan:
"""Build normalized search variants shared across release sources."""
resolved_languages = _normalize_languages(languages)
"""Build normalized search variants shared across release sources.
``user_id`` picks up that user's default languages when the caller does not
filter explicitly, so a search started without a language filter uses the
reader's own default rather than the instance-wide one.
"""
resolved_languages = _normalize_languages(languages, user_id)
resolved_manual_query = None
if manual_query:
+6 -1
View File
@@ -122,7 +122,12 @@ def is_audiobook(content_type: str | None) -> bool:
# had drifted apart: the settings UI only offered m4b/mp3/m4a, which meant a FLAC
# audiobook could never be enabled, was silently dropped from every search result, and
# was rejected after download as "format not supported".
AUDIOBOOK_FORMATS = ("m4b", "mp3", "m4a", "flac", "ogg", "wma", "aac", "wav", "opus")
#
# "mp4" is here because some trackers (MyAnonamouse in particular) ship AAC audiobooks
# as per-chapter .mp4 files - the same ISO-BMFF container as .m4a/.m4b, just with the
# generic extension. Without it those releases downloaded fine and then failed
# post-processing with "No book files found in download".
AUDIOBOOK_FORMATS = ("m4b", "mp3", "m4a", "mp4", "flac", "ogg", "wma", "aac", "wav", "opus")
# Multi-file audiobooks are almost always distributed as an archive. These are containers
# rather than formats: they are what a *release* looks like, and the formats above are
+1
View File
@@ -76,6 +76,7 @@ _BOOK_EXTENSIONS = (
".m4b",
".mobi",
".mp3",
".mp4",
".ogg",
".opus",
".pdf",
+105 -35
View File
@@ -45,6 +45,10 @@ _HASH_LENGTH_ED2K = 32
_HTTP_STATUS_FORBIDDEN = HTTPStatus.FORBIDDEN
_HTTP_STATUS_NOT_FOUND = HTTPStatus.NOT_FOUND
_METADATA_DOWNLOAD_STATES = {"forcedMetaDL", "metaDL"}
# How long add_download waits for magnet metadata before falling back to the info
# hash it already knows, rather than holding the download queue on a thin swarm.
_METADATA_WAIT_POLLS = 20
_METADATA_WAIT_INTERVAL_SECONDS = 0.5
_ONE_WEEK_IN_SECONDS = 604800
@@ -221,6 +225,9 @@ class QBittorrentClient(DownloadClient):
self._category = config_text(config.get("QBITTORRENT_CATEGORY", "books"))
self._download_dir = config_text(config.get("QBITTORRENT_DOWNLOAD_DIR", ""))
self._tags = _normalize_tags(config.get("QBITTORRENT_TAG", []))
# download_id -> qBittorrent's current primary hash, for identities that no
# longer match it directly. See _resolve_torrent().
self._primary_hashes: dict[str, str] = {}
@property
def _can_reauthenticate(self) -> bool:
@@ -307,13 +314,31 @@ class QBittorrentClient(DownloadClient):
params = {"category": category} if category else {}
return self._request_torrent_info_records(params)
def _remember_primary_hash(self, download_id: str, torrent: SimpleNamespace) -> None:
"""Note the primary hash a listing scan found, so later lookups skip the scan."""
torrent_hash = getattr(torrent, "hash", None)
if isinstance(torrent_hash, str) and torrent_hash:
self._primary_hashes[download_id.lower()] = torrent_hash.lower()
def _resolve_torrent(
self, download_id: str, category: str | None = None
) -> tuple[SimpleNamespace | None, str | None]:
"""Resolve any known torrent identity to its current qBittorrent record."""
torrent, error = self._get_torrent_info(download_id)
if error or torrent:
return torrent, error
"""Resolve any known torrent identity to its current qBittorrent record.
A hybrid torrent's primary hash switches from the v1 hash to the truncated v2
hash once metadata resolves, so a download tracked by its v1 hash misses the
`hashes=` lookup and falls through to a full listing. Since `get_status()`
polls every couple of seconds for the life of the download, remember the
primary hash a scan finds and try it first.
"""
cached = self._primary_hashes.get(download_id.lower())
for candidate in (item for item in dict.fromkeys((cached, download_id)) if item):
torrent, error = self._get_torrent_info(candidate)
if error:
return None, error
if torrent:
self._remember_primary_hash(download_id, torrent)
return torrent, None
categories = [candidate for candidate in (category, self._category) if candidate]
for candidate in dict.fromkeys(categories):
@@ -325,18 +350,41 @@ class QBittorrentClient(DownloadClient):
None,
)
if torrent:
self._remember_primary_hash(download_id, torrent)
return torrent, None
torrents, error = self._list_torrents_by_category(None)
if error:
return None, error
return (
next(
(item for item in torrents if _torrent_matches_download_id(item, download_id)),
None,
),
torrent = next(
(item for item in torrents if _torrent_matches_download_id(item, download_id)),
None,
)
if torrent:
self._remember_primary_hash(download_id, torrent)
else:
# The torrent is gone; drop the note so a re-add is not looked up by a
# hash that no longer exists.
self._primary_hashes.pop(download_id.lower(), None)
return torrent, None
def _current_hash(self, download_id: str) -> str:
"""qBittorrent's current primary hash for any identity we know the torrent by.
Falls back to the given ID when the torrent cannot be found, so callers
still address the hash they were handed and surface the client's error.
"""
try:
torrent, error = self._resolve_torrent(download_id)
except _QBITTORRENT_CLIENT_ERRORS as e:
logger.debug("Could not resolve current hash for %s: %s", download_id, e)
return download_id
if error or not torrent:
return download_id
torrent_hash = getattr(torrent, "hash", None)
if isinstance(torrent_hash, str) and torrent_hash:
return torrent_hash
return download_id
def _list_category_hashes(self, category: str | None) -> set[str] | None:
"""Snapshot the hashes qBittorrent currently reports for a category."""
@@ -495,9 +543,13 @@ class QBittorrentClient(DownloadClient):
message = f"{message} (torrent file fetch failed: {torrent_info.fetch_error})"
_raise_runtime_error(message)
# Wait until qBittorrent has resolved magnet metadata so the returned
# hash is its stable primary torrent ID, which may differ from the v1 hash.
for _ in range(20):
# Prefer qBittorrent's primary torrent ID, which for hybrid torrents
# switches from the v1 hash to the truncated v2 hash once metadata
# resolves. A magnet with few peers can take minutes to fetch metadata,
# and the torrent is worth keeping in the meantime: every lookup goes
# through `_resolve_torrent`, which still matches the v1 hash against
# `infohash_v1` after the primary ID has changed.
for _ in range(_METADATA_WAIT_POLLS):
torrent, error = self._resolve_torrent(expected_hash, category)
if error:
logger.debug("qBittorrent add_download: %s", error)
@@ -506,17 +558,18 @@ class QBittorrentClient(DownloadClient):
if isinstance(torrent_hash, str) and torrent_hash:
logger.info("Added torrent: %s", torrent_hash)
return torrent_hash.lower()
time.sleep(0.5)
time.sleep(_METADATA_WAIT_INTERVAL_SECONDS)
_raise_runtime_error(
"Torrent metadata resolution was not confirmed within the visibility grace period "
f"(response={result_text})"
logger.info(
"Added torrent %s; metadata still pending after %.0fs, tracking it by info hash",
expected_hash,
_METADATA_WAIT_POLLS * _METADATA_WAIT_INTERVAL_SECONDS,
)
except _QBITTORRENT_CLIENT_ERRORS:
logger.exception("qBittorrent add failed")
raise
else:
return expected_hash
return expected_hash.lower()
def get_status(self, download_id: str) -> DownloadStatus:
"""Get torrent status by hash.
@@ -529,7 +582,7 @@ class QBittorrentClient(DownloadClient):
"""
try:
torrent, error = self._get_torrent_info(download_id)
torrent, error = self._resolve_torrent(download_id)
if error:
return DownloadStatus.error(error)
if not torrent:
@@ -613,7 +666,9 @@ class QBittorrentClient(DownloadClient):
"""
try:
self._client.torrents_delete(torrent_hashes=download_id, delete_files=delete_files)
torrent_hash = self._current_hash(download_id)
self._client.torrents_delete(torrent_hashes=torrent_hash, delete_files=delete_files)
self._primary_hashes.pop(download_id.lower(), None)
logger.info(
"Removed torrent from qBittorrent: %s%s",
download_id,
@@ -635,7 +690,7 @@ class QBittorrentClient(DownloadClient):
logger.debug("Could not create category '%s': %s", category, e)
self._client.torrents_set_category(
torrent_hashes=download_id,
torrent_hashes=self._current_hash(download_id),
category=category,
)
logger.info("Set qBittorrent category for %s to '%s'", download_id, category)
@@ -657,7 +712,7 @@ class QBittorrentClient(DownloadClient):
- join `save_path` with the torrent's top-level directory
"""
try:
torrent, error = self._get_torrent_info(download_id)
torrent, error = self._resolve_torrent(download_id)
if error:
logger.debug("qBittorrent get_download_path: %s", error)
return None
@@ -758,6 +813,33 @@ class QBittorrentClient(DownloadClient):
)
return None
def _await_existing_torrent(
self, info_hash: str, category: str | None
) -> tuple[str, DownloadStatus] | None:
"""Report a torrent already in qBittorrent, waiting out magnet metadata first."""
for _ in range(_METADATA_WAIT_POLLS):
torrent, error = self._resolve_torrent(info_hash, category)
if error:
logger.debug("qBittorrent find_existing: %s", error)
return None
if not torrent:
return None
if getattr(torrent, "state", None) not in _METADATA_DOWNLOAD_STATES:
torrent_hash = getattr(torrent, "hash", None)
if isinstance(torrent_hash, str) and torrent_hash:
torrent_hash = torrent_hash.lower()
return (torrent_hash, self.get_status(torrent_hash))
time.sleep(_METADATA_WAIT_INTERVAL_SECONDS)
# Metadata is still pending, but the torrent is here and `add_download` keeps
# one in this state rather than giving up. Report it by info hash so the
# caller joins the download in progress instead of adding a duplicate.
logger.info(
"Existing torrent %s is still fetching metadata; joining it by info hash",
info_hash,
)
return (info_hash.lower(), self.get_status(info_hash))
def find_existing(
self, url: str, category: str | None = None
) -> tuple[str, DownloadStatus] | None:
@@ -767,21 +849,9 @@ class QBittorrentClient(DownloadClient):
if not torrent_info.info_hash:
return None
for _ in range(20):
torrent, error = self._resolve_torrent(torrent_info.info_hash, category)
if error:
logger.debug("qBittorrent find_existing: %s", error)
return None
if not torrent:
return None
if getattr(torrent, "state", None) not in _METADATA_DOWNLOAD_STATES:
torrent_hash = getattr(torrent, "hash", None)
if isinstance(torrent_hash, str) and torrent_hash:
torrent_hash = torrent_hash.lower()
return (torrent_hash, self.get_status(torrent_hash))
time.sleep(0.5)
existing = self._await_existing_torrent(torrent_info.info_hash, category)
except _QBITTORRENT_CLIENT_ERRORS as e:
logger.debug("Error checking for existing torrent: %s", e)
return None
else:
return None
return existing
+1
View File
@@ -80,6 +80,7 @@ _BOOK_EXTENSIONS = (
".m4b",
".mobi",
".mp3",
".mp4",
".ogg",
".opus",
".pdf",
+9
View File
@@ -248,6 +248,15 @@ class SABnzbdClient(DownloadClient):
if trusted_url and _url_origin(trusted_url) == target_origin:
return True
named_indexers = config.get("NEWZNAB_INDEXERS", [])
if isinstance(named_indexers, list):
for row in named_indexers:
if not isinstance(row, dict):
continue
trusted_url = normalize_http_config_url(row.get("url"))
if trusted_url and _url_origin(trusted_url) == target_origin:
return True
return False
def _get_prowlarr_headers(self, url: str) -> dict:
@@ -17,6 +17,7 @@ from shelfmark.core.config import config
from shelfmark.core.logger import setup_logger
from shelfmark.core.utils import normalize_http_url
from shelfmark.download.network import get_ssl_verify
from shelfmark.download.postprocess.packs import PackFile
logger = setup_logger(__name__)
@@ -433,6 +434,56 @@ def extract_info_hash_from_torrent(torrent_data: bytes) -> str | None:
return None
def _decode_torrent_text(value: object) -> str | None:
if isinstance(value, bytes):
return value.decode("utf-8", errors="replace")
if isinstance(value, str):
return value
return None
def extract_file_list_from_torrent(torrent_data: bytes) -> list[PackFile] | None:
"""List the files a .torrent describes, release-relative, without downloading it.
Multi-file torrents nest every path under the torrent name (which becomes the
client's save folder); single-file torrents are just the named file.
"""
try:
decoded, _ = bencode_decode(torrent_data)
except _TORRENT_PARSE_ERRORS as e:
logger.debug("Failed to parse torrent file list: %s", e)
return None
if not isinstance(decoded, dict):
return None
info = decoded.get(b"info")
if not isinstance(info, dict):
return None
name = _decode_torrent_text(info.get(b"name")) or ""
raw_files = info.get(b"files")
if not isinstance(raw_files, list):
length = info.get(b"length")
if not name:
return None
return [PackFile(name, length if isinstance(length, int) else None)]
files: list[PackFile] = []
for entry in raw_files:
if not isinstance(entry, dict):
continue
raw_path = entry.get(b"path")
if not isinstance(raw_path, list):
continue
segments = [seg for seg in (_decode_torrent_text(part) for part in raw_path) if seg]
if not segments:
continue
if name:
segments.insert(0, name)
length = entry.get(b"length")
files.append(PackFile("/".join(segments), length if isinstance(length, int) else None))
return files
def extract_hash_from_magnet(magnet_url: str) -> str | None:
"""Extract info_hash from a magnet URL."""
if not magnet_url.startswith("magnet:"):
+51 -2
View File
@@ -12,6 +12,7 @@ from tqdm import tqdm
from shelfmark.bypass import BypassCancelledError, cookie_store
from shelfmark.bypass.challenge import challenge_marker
from shelfmark.core import search_deadline
from shelfmark.core.config import config as app_config
from shelfmark.core.logger import setup_logger
from shelfmark.core.request_helpers import coerce_bool, normalize_positive_int
@@ -52,6 +53,7 @@ _BYPASSER_ERRORS = (
RuntimeError,
TypeError,
ValueError,
network.RateLimitedError,
requests.exceptions.RequestException,
)
@@ -337,6 +339,14 @@ def html_get_page(
# so it must be a concrete selector, not the Optional parameter.
selector = selector or network.AAMirrorSelector()
# A release search runs under a wall-clock budget (see shelfmark.core.search_deadline).
# Adopting it as the cancel flag is what makes the budget bite on a solve already in
# flight: the bypassers and the helper subprocess poll this flag but know nothing about
# deadlines. Only when the caller has no flag of its own - a queued download brings one
# and must keep it, and runs outside any search context anyway.
if cancel_flag is None:
cancel_flag = search_deadline.cancel_event()
def _result(html: str, response_url: str) -> str | tuple[str, str]:
if include_response_url:
return html, response_url
@@ -361,6 +371,13 @@ def html_get_page(
retry-loop branch above with `continue`, and with MAX_RETRY=1 there is no
later attempt for that branch to run on either.
"""
# Never start a minutes-long browser solve on a budget that has already run out:
# nothing downstream would get to report the real reason before the caller's
# deadline (or its reverse proxy) cut the request off.
if search_deadline.expired():
logger.info("Release search budget spent; not starting a bypass for %s", bypass_url)
return _fail(search_deadline.deadline_message(), bypass_url)
if status_callback:
status_callback("resolving", "Bypassing protection...")
try:
@@ -377,6 +394,17 @@ def html_get_page(
"not solved. Check that FlareSolverr/the CF bypasser is reachable.",
bypass_url,
)
except network.RateLimitedError as e:
# Not a bypasser malfunction: the host is throttling this IP and a solve
# cannot help. Surface the wait as a plain failure so the search ends cleanly
# instead of looping another minutes-long solve against a 429.
logger.info("Skipping bypass (rate-limited): %s", e)
if status_callback:
try:
status_callback("resolving", "Rate limited, try again shortly")
except _STATUS_CALLBACK_ERRORS:
logger.debug("Rate-limit status callback failed", exc_info=True)
return _fail(str(e), bypass_url)
except _BYPASSER_ERRORS as e:
logger.warning("Bypasser error: %s: %s", type(e).__name__, e)
# Surface the real reason. Without this the caller only sees an empty
@@ -388,6 +416,10 @@ def html_get_page(
except _STATUS_CALLBACK_ERRORS:
logger.debug("Bypass error status callback failed", exc_info=True)
if isinstance(e, BypassCancelledError):
# The budget trips the same cancel flag a user's cancel does, so tell them
# apart here - "cancelled" is a confusing thing to read when nobody did.
if search_deadline.expired():
return _fail(search_deadline.deadline_message(), bypass_url)
return _fail("The protection bypass was cancelled.", bypass_url)
return _fail(f"The protection bypasser failed: {type(e).__name__}: {e}", bypass_url)
finally:
@@ -444,6 +476,9 @@ def html_get_page(
for attempt in range(1, retry_limit + 1):
# Check for cancellation before each attempt
if cancel_flag and cancel_flag.is_set():
if search_deadline.expired():
logger.info("Release search budget spent before attempt %s", attempt)
return _fail(search_deadline.deadline_message(), current_url)
logger.info("html_get_page cancelled before attempt %s", attempt)
return _fail("The request was cancelled.", current_url)
@@ -472,8 +507,15 @@ def html_get_page(
current_url,
proxies=get_proxies(current_url),
timeout=REQUEST_TIMEOUT,
# Bypasser-derived cookies win: they came from a real solved challenge.
cookies={**handshake_cookies, **cookies},
# Handshake cookies win. They were issued by *this* exchange, so by
# definition they are fresher than anything the store holds, and the
# server is waiting to see them echoed back on the very next hop.
# Letting the store overwrite them meant a stored cookie of the same
# name (DDoS-Guard reuses __ddg1_/__ddg2_ for both) was replayed on
# every hop and the freshly issued value never left this process - the
# ?check=1 probe could then never terminate, so every request ended in
# the redirect-loop handoff and paid for a full browser solve.
cookies={**cookies, **handshake_cookies},
headers=headers,
allow_redirects=allow_redirects,
verify=get_ssl_verify(current_url),
@@ -698,6 +740,12 @@ def html_get_page(
f"Anna's Archive returned 404 Not Found for {current_url}.", current_url
)
# 429 = origin throttling this IP. Arm the per-host backoff so selection and
# the bypasser stop hammering it, then fall through to normal rotation onto a
# mirror that is not (yet) rate-limited.
if status == _HTTP_STATUS_RATE_LIMITED:
network.note_rate_limited(current_url)
# Try mirror/DNS rotation on retryable errors. A failure that proves the
# mirror is unusable also drops it from this process's rotation, so the
# next search does not pay for it again.
@@ -851,6 +899,7 @@ def download_url(
# Rate limited - skip to next source immediately
# (waiting doesn't help with concurrent downloads hitting the same server)
if status == _HTTP_STATUS_RATE_LIMITED:
network.note_rate_limited(current_url)
logger.info("Rate limited (429) - trying next source")
if status_callback:
status_callback("resolving", "Server busy, trying next")
+111 -3
View File
@@ -3,12 +3,13 @@
import fnmatch
import ipaddress
import socket
import time
import urllib.parse
import urllib.request
from datetime import UTC, datetime, timedelta
from http import HTTPStatus
from socket import AddressFamily, SocketKind
from typing import TYPE_CHECKING, Any, cast
from typing import TYPE_CHECKING, Any, NamedTuple, cast
import dns.resolver
import httpx
@@ -287,6 +288,108 @@ _dead_aa_urls: set[str] = set()
_dead_aa_urls_lock = _RLock()
# Per-host rate-limit backoff. A 429 is the origin throttling *this IP*, not a challenge:
# a DDoS-Guard/Cloudflare solve still renders, so the bypass "succeeds" yet the cleared
# request is rejected again and the throttle is only renewed. The single answer is to
# wait, so a 429 sidelines the host for a growing window - mirror selection and the
# bypasser both skip a cooling-down host until its deadline passes. The wait escalates
# 2 -> 5 -> 10 -> 15 -> 30 minutes each time the host throttles us again *after* we
# already waited a full window out; a host left clear for longer than the top step
# starts the ladder over. Keyed by host so every mirror and source shares one view;
# in-memory only, so a restart starts clean.
_RATE_LIMIT_COOLDOWN_LADDER_SECONDS: tuple[float, ...] = (120.0, 300.0, 600.0, 900.0, 1800.0)
# A host that has been clear this long is treated as a fresh episode: the next 429
# restarts the ladder at 2 minutes rather than resuming the escalation.
_RATE_LIMIT_RESET_AFTER_SECONDS = 1800.0
class _Cooldown(NamedTuple):
"""One host's active rate-limit window and how far up the ladder it has climbed."""
deadline: float # time.monotonic() value at which the wait expires
level: int # index into _RATE_LIMIT_COOLDOWN_LADDER_SECONDS
_host_cooldowns: dict[str, _Cooldown] = {}
_host_cooldowns_lock = _RLock()
class RateLimitedError(Exception):
"""Raised to abandon a request whose host is in a 429 cooldown.
Not a transport failure - nothing is wrong with the network, the origin is
throttling this IP and only time clears it. Callers surface it as a plain failure
rather than retrying or handing the URL to the bypasser.
"""
def _cooldown_key(url: str) -> str:
"""Host a cooldown is keyed by; '' when the URL carries none."""
return (urllib.parse.urlparse(url).hostname or "").lower()
def note_rate_limited(url: str) -> float:
"""Escalate a host's 429 backoff and (re)arm its cooldown; return the wait applied.
The step advances only when a fresh 429 arrives *after* the previous window already
elapsed - i.e. we waited it out and the host throttled us again. A 429 that lands
while the host is still cooling is the same episode: it neither escalates the level
nor shortens the wait. See the ladder note above.
"""
host = _cooldown_key(url)
if not host:
return 0.0
now = time.monotonic()
ladder = _RATE_LIMIT_COOLDOWN_LADDER_SECONDS
with _host_cooldowns_lock:
prev = _host_cooldowns.get(host)
if prev is not None and now < prev.deadline:
# Still inside the current window - same throttling episode, leave it be.
return prev.deadline - now
if prev is None or now - prev.deadline > _RATE_LIMIT_RESET_AFTER_SECONDS:
level = 0
else:
level = min(prev.level + 1, len(ladder) - 1)
wait = ladder[level]
_host_cooldowns[host] = _Cooldown(deadline=now + wait, level=level)
logger.info(
"Rate limited (429): backing off %s for %.0fs (step %d/%d)",
host,
wait,
level + 1,
len(ladder),
)
return wait
def host_cooldown_remaining(url: str) -> float:
"""Seconds left on a host's 429 cooldown; 0.0 when clear or expired.
Leaves an expired record in place: the ladder level it carries is what a later 429
escalates from (or resets, once the clear gap is long enough).
"""
host = _cooldown_key(url)
if not host:
return 0.0
now = time.monotonic()
with _host_cooldowns_lock:
rec = _host_cooldowns.get(host)
if rec is None or rec.deadline <= now:
return 0.0
return rec.deadline - now
def is_host_cooling_down(url: str) -> bool:
"""True while ``url``'s host is inside its 429 cooldown window."""
return host_cooldown_remaining(url) > 0.0
def clear_host_cooldowns() -> None:
"""Forget all rate-limit cooldowns (manual reset / tests)."""
with _host_cooldowns_lock:
_host_cooldowns.clear()
def _ensure_initialized() -> None:
"""Lazy guard so runtime setup happens once and late calls still work."""
global _initialized
@@ -1418,8 +1521,13 @@ def get_available_aa_urls() -> list[str]:
if not alive and _aa_urls:
logger.warning("All AA mirrors quarantined; retrying the full list")
_dead_aa_urls.clear()
return _aa_urls.copy()
return alive
alive = _aa_urls.copy()
# Prefer mirrors that are not serving a 429 cooldown so rotation stops hammering a
# throttled host. When every live mirror is cooling, keep the full live list rather
# than returning nothing: selection must never be left with nowhere to point, and
# the bypasser's fail-fast reports the "all rate-limited" case with a clear error.
breathing = [url for url in alive if not is_host_cooling_down(url)]
return breathing or alive
def _aa_base_for_url(url: str) -> str:
+35
View File
@@ -265,6 +265,8 @@ def queue_release(
series_position = release_data.get("series_position") or extra.get("series_position")
subtitle = release_data.get("subtitle") or extra.get("subtitle")
language = release_data.get("language") or extra.get("language")
multi_book = bool(release_data.get("multi_book") or extra.get("multi_book"))
book_plan = _normalize_book_plan(release_data.get("book_plan") or extra.get("book_plan"))
books_output_mode = (
str(config.get("BOOKS_OUTPUT_MODE", "folder", user_id=user_id) or "folder")
@@ -300,6 +302,8 @@ def queue_release(
series_position=series_position,
subtitle=subtitle,
language=language,
multi_book=multi_book or book_plan is not None,
book_plan=book_plan,
search_mode=search_mode,
output_mode=output_mode,
output_args=output_args,
@@ -408,6 +412,33 @@ def can_retry_download_task(
return _has_staged_retry_source(task)
def _normalize_book_plan(value: object) -> list[dict[str, Any]] | None:
"""Keep only well-formed pack books: a title plus a non-empty list of file paths."""
if not isinstance(value, list):
return None
books: list[dict[str, Any]] = []
for entry in value:
if not isinstance(entry, dict):
continue
title = normalize_optional_text(entry.get("title"))
raw_files = entry.get("files")
if title is None or not isinstance(raw_files, list):
continue
files = [f for f in raw_files if isinstance(f, str) and f.strip()]
if not files:
continue
year = entry.get("year")
books.append(
{
"title": title,
"series_position": _optional_number(entry.get("series_position")),
"year": year if isinstance(year, int) and not isinstance(year, bool) else None,
"files": files,
}
)
return books or None
def serialize_task_for_retry(task: DownloadTask) -> dict[str, Any]:
"""Serialize the task state needed for restart-safe retries."""
raw_search_mode = getattr(task, "search_mode", None)
@@ -437,6 +468,8 @@ def serialize_task_for_retry(task: DownloadTask) -> dict[str, Any]:
"subtitle": getattr(task, "subtitle", None),
"language": getattr(task, "language", None),
"search_mode": search_mode,
"multi_book": bool(getattr(task, "multi_book", False)),
"book_plan": _normalize_book_plan(getattr(task, "book_plan", None)),
"output_mode": getattr(task, "output_mode", None),
"output_args": dict(raw_output_args) if isinstance(raw_output_args, dict) else {},
"user_id": getattr(task, "user_id", None),
@@ -495,6 +528,8 @@ def _restore_task_from_retry_payload(payload: object) -> DownloadTask | None:
subtitle=normalize_optional_text(payload.get("subtitle")),
language=normalize_optional_text(payload.get("language")),
search_mode=search_mode,
multi_book=bool(payload.get("multi_book", False)),
book_plan=_normalize_book_plan(payload.get("book_plan")),
output_mode=normalize_optional_text(payload.get("output_mode")),
output_args=dict(output_args) if isinstance(output_args, dict) else {},
user_id=normalize_positive_int(payload.get("user_id")),
+10 -1
View File
@@ -105,6 +105,7 @@ def process_folder_output(
maybe_run_custom_script,
prepare_output_files,
record_step,
resolve_book_groups,
transfer_book_files,
)
@@ -260,7 +261,15 @@ def process_folder_output(
prepared.cleanup_paths,
)
message = "Complete" if len(final_paths) == 1 else f"Complete ({len(final_paths)} files)"
pack_groups = resolve_book_groups(
task, prepared.files, organization_mode=plan.organization_mode
)
if pack_groups is not None:
message = f"Complete ({len(pack_groups)} books, {len(final_paths)} files)"
elif len(final_paths) == 1:
message = "Complete"
else:
message = f"Complete ({len(final_paths)} files)"
status_callback("complete", message)
return str(final_paths[0])
+423
View File
@@ -0,0 +1,423 @@
"""Multi-book ("pack") release planning.
A pack is one release that contains several books: a whole-series torrent with one
subfolder per book, or a flat folder of `Series 1.0 - Title.m4b` files. The same
planning rules serve pre-download inspection (the file list comes from the release
source) and post-processing (the file list comes from disk), so what the user
approved in the modal is what gets filed.
"""
from __future__ import annotations
import os
import re
from dataclasses import dataclass
from pathlib import Path, PurePosixPath
from shelfmark.core.utils import AUDIOBOOK_FORMATS
# m4b/m4a hold a whole audiobook in one file; every other audio format (mp3, flac, ...) is
# chaptered - many files make up one book. Ebook formats are always one file per book, so
# only chaptered *audio* matters here. A flat folder is split one-book-per-file only when
# none of its files are chaptered audio: a bare list of `01 - Chapter.mp3` tracks is a
# single chaptered audiobook, not a pack of books.
_SINGLE_FILE_AUDIO_CONTAINERS = frozenset({"m4b", "m4a"})
_CHAPTERED_AUDIO_EXTENSIONS = frozenset(AUDIOBOOK_FORMATS) - _SINGLE_FILE_AUDIO_CONTAINERS
_YEAR_SUFFIX_RE = re.compile(r"\s*\(\s*(?P<year>\d{4})\s*\)\s*$")
_SERIES_MARKER_RE = re.compile(
r"""
^\s*
(?:
\[\s*\#?(?P<bracket>\d+(?:\.\d+)?)\s*\] # [03] / [#3]
| \#(?P<hash>\d+(?:\.\d+)?) # #3
| book\.?\s*(?P<book>\d+(?:\.\d+)?) # Book 3 / Book. 03
| (?P<plain>\d+(?:\.\d+)?)(?=[\s\-:.]) # 03 - / 1.0 - / 3.
)
\s*(?:[-:.]\s*)?
""",
re.IGNORECASE | re.VERBOSE,
)
_SEPARATOR_CHARS = " \t-_:."
# "Gods of Risk 2.5 - Gods of Risk": the title repeated on both sides of the position.
_REPEATED_TITLE_RE = re.compile(
r"^(?P<left>.+?)\s+(?P<position>\d+(?:\.\d+)?)\s*[-:\u2013]\s*(?P<right>.+)$"
)
_SERIES_LABEL_WORDS = r"(?:novella|novellas|short\s+story|short|story|novel)"
# "Uncrowned Cradle, Book 7" / "Reaper Cradle, Volume 10" / "Wintersteel (Cradle, Book 8)":
# an explicit word marks the position at the END of the name. A bare trailing number
# is deliberately not matched — "Title - 02" is a chapter, not a series position.
_TRAILING_MARKER_RE = re.compile(
r"""
[\s,\-:\u2013(]*
(?:book|volume|vol\.?)\s*\#?(?P<position>\d+(?:\.\d+)?)
\s*\)?\s*$
""",
re.IGNORECASE | re.VERBOSE,
)
# AudiobookBay renders a file inside a folder as "<folder> <file>" with no separator,
# so a pack row reads "Author - Title Series, Book 1 Title Series, Book 1".
_GLUED_FOLDER_RE = re.compile(
r"^(?P<prefix>.+?\s[-\u2013]\s)?(?P<core>.+?)\s+(?P=core)$", re.IGNORECASE
)
@dataclass(frozen=True)
class PackFile:
"""One file inside a release, path relative to the release root."""
path: str
size: int | None = None
@dataclass(frozen=True)
class PackBook:
"""One book split out of a pack, files as release-relative paths."""
title: str
series_position: float | None
year: int | None
files: list[str]
@dataclass(frozen=True)
class PackPlan:
books: list[PackBook]
ignored: list[str]
@property
def is_pack(self) -> bool:
return len(self.books) > 1
@dataclass(frozen=True)
class BookGroup:
"""One book's on-disk files, ready for transfer."""
title: str
series_position: float | None
year: int | None
files: list[Path]
def _strip_series_name(name: str, series_name: str | None) -> str:
if not series_name:
return name
prefix = series_name.strip()
if not prefix or not name.lower().startswith(prefix.lower()):
return name
remainder = name[len(prefix) :]
if remainder and remainder[0].isalnum():
return name
return remainder.lstrip(_SEPARATOR_CHARS)
def _strip_series_label(work: str, series_name: str | None) -> str:
"""Drop a leading "An <Series> Novella - " style label that some packs prepend."""
if not series_name:
return work
# "The Expanse" is labelled "An Expanse Novella", so match without the article.
core = re.sub(r"^(?:the|an?)\s+", "", series_name.strip(), flags=re.IGNORECASE)
if not core:
return work
pattern = re.compile(
rf"^(?:an?\s+|the\s+)?{re.escape(core)}\s+{_SERIES_LABEL_WORDS}\s*[-:\u2013]\s*",
re.IGNORECASE,
)
return pattern.sub("", work, count=1)
def _collapse_glued_folder(name: str) -> str:
match = _GLUED_FOLDER_RE.match(name)
if not match:
return name
prefix = match.group("prefix") or ""
core = match.group("core")
# "Author - X X" → "Author - X" (the folder carried the author, the file did not).
return (prefix + core).strip()
def _strip_author_name(name: str, author_name: str | None) -> str:
"""Drop a leading "Author - " (packs are often filed as `Author - Title`)."""
if not author_name:
return name
prefix = author_name.strip()
if not prefix or not name.lower().startswith(prefix.lower()):
return name
remainder = name[len(prefix) :]
stripped = remainder.lstrip(_SEPARATOR_CHARS + "\u2013")
if stripped == remainder: # no separator after the author: part of the title
return name
return stripped
def _strip_trailing_series_name(work: str, series_name: str | None) -> str:
"""Drop a trailing series name left behind by a trailing position marker."""
if not series_name:
return work
suffix = series_name.strip()
if not suffix or not work.lower().endswith(suffix.lower()):
return work
remainder = work[: -len(suffix)]
stripped = remainder.rstrip(_SEPARATOR_CHARS + ",(\u2013")
if not stripped or stripped == remainder:
return work
return stripped
def parse_pack_book_name(
name: str, *, series_name: str | None, author_name: str | None = None
) -> tuple[str, float | None, int | None]:
"""Split a book folder/file-stem name into (title, series position, year).
Strips a leading series name, a leading position marker (`Book 3 - `, `03 - `,
`1.0 - `, `3. `, `[03] `, `#3 `) and a trailing `(YYYY)`. Also understands a
trailing marker (`Title Series, Book 3`, `Title (Series, Volume 3)`), a leading
`Author - `, and AudiobookBay's glued `<folder> <file>` names. Returns the name
unchanged with no position/year when nothing would be left of the title.
"""
work = _collapse_glued_folder(name.strip())
work = _strip_author_name(work, author_name)
work = _strip_series_name(work, series_name)
year: int | None = None
year_match = _YEAR_SUFFIX_RE.search(work)
if year_match:
year = int(year_match.group("year"))
work = work[: year_match.start()]
position: float | None = None
repeated = _REPEATED_TITLE_RE.match(work.strip())
if (
repeated
and repeated.group("left").strip().lower() == repeated.group("right").strip().lower()
):
return repeated.group("right").strip(), float(repeated.group("position")), year
marker = _SERIES_MARKER_RE.match(work)
if marker:
raw = (
marker.group("bracket")
or marker.group("hash")
or marker.group("book")
or marker.group("plain")
)
position = float(raw)
work = work[marker.end() :]
else:
trailing = _TRAILING_MARKER_RE.search(work)
if trailing and trailing.start() > 0:
position = float(trailing.group("position"))
work = _strip_trailing_series_name(work[: trailing.start()], series_name)
work = _strip_series_label(work, series_name)
title = work.strip().strip(_SEPARATOR_CHARS).strip()
if not title:
return name, None, None
return title, position, year
def _book_from_name(
name: str, files: list[str], series_name: str | None, author_name: str | None = None
) -> PackBook:
title, position, year = parse_pack_book_name(
name, series_name=series_name, author_name=author_name
)
return PackBook(title=title, series_position=position, year=year, files=files)
def _common_root_parts(paths: list[PurePosixPath]) -> tuple[str, ...]:
parents = [p.parent.parts for p in paths]
common: list[str] = []
for parts in zip(*parents, strict=False):
if len(set(parts)) != 1:
break
common.append(parts[0])
return tuple(common)
def plan_pack(
files: list[PackFile],
*,
supported_extensions: set[str],
series_name: str | None,
author_name: str | None = None,
root_depth: int | None = None,
) -> PackPlan:
"""Group a release's file list into books.
Files in a subfolder (relative to the common root) group by that subfolder. Files
directly in the root split one-book-per-file only when at least two of them carry
a series position in their names; otherwise they are one book (a chaptered
audiobook, e.g. `01.mp3`, `02.mp3`). `root_depth` fixes how many leading path
components form the root instead of deriving it from the files' common parent.
"""
supported = {ext.lower().lstrip(".") for ext in supported_extensions}
book_files: list[PurePosixPath] = []
ignored: list[str] = []
for pack_file in files:
rel = PurePosixPath(pack_file.path.replace("\\", "/").lstrip("./"))
if rel.suffix.lower().lstrip(".") in supported:
book_files.append(rel)
else:
ignored.append(pack_file.path)
if not book_files:
return PackPlan(books=[], ignored=ignored)
root_parts = (
_common_root_parts(book_files) if root_depth is None else book_files[0].parts[:root_depth]
)
depth = len(root_parts)
root_files: list[PurePosixPath] = []
folders: dict[str, list[str]] = {}
for rel in book_files:
remainder = rel.parts[depth:]
if len(remainder) > 1:
folders.setdefault(remainder[0], []).append(str(rel))
else:
root_files.append(rel)
books: list[PackBook] = []
if root_files:
parsed = [
parse_pack_book_name(f.stem, series_name=series_name, author_name=author_name)
for f in root_files
]
positions = {p[1] for p in parsed if p[1] is not None}
titles = {p[0].strip().lower() for p in parsed if p[0]}
one_book_per_file = all(
rel.suffix.lower().lstrip(".") not in _CHAPTERED_AUDIO_EXTENSIONS for rel in root_files
)
# Split a flat folder into a book per file only with real evidence of distinct
# books: two or more series positions, more than one title, and no chaptered audio
# (a bare list of `01 - Chapter.mp3` tracks is one book, not a pack).
if len(positions) >= 2 and len(titles) >= 2 and one_book_per_file:
books.extend(
PackBook(title=title, series_position=position, year=year, files=[str(f)])
for f, (title, position, year) in zip(root_files, parsed, strict=True)
)
elif len(root_files) == 1:
books.append(
_book_from_name(root_files[0].stem, [str(root_files[0])], series_name, author_name)
)
else:
group_name = root_parts[-1] if root_parts else ""
books.append(
_book_from_name(group_name, [str(f) for f in root_files], series_name, author_name)
)
books.extend(
_book_from_name(folder, paths, series_name, author_name)
for folder, paths in folders.items()
)
return PackPlan(books=books, ignored=ignored)
def _relative_paths(
book_files: list[Path], root: Path | None = None
) -> tuple[Path, dict[Path, str]]:
if root is None:
root = Path(os.path.commonpath([str(f.parent) for f in book_files]))
return root, {f: f.relative_to(root).as_posix() for f in book_files}
def group_files_into_books(
book_files: list[Path],
*,
series_name: str | None,
author_name: str | None = None,
root: Path | None = None,
) -> list[BookGroup]:
"""Heuristically split on-disk files into books (see `plan_pack`).
`root` pins the release root when grouping a subset of a larger file set.
"""
if not book_files:
return []
_root, rel_by_path = _relative_paths(book_files, root)
path_by_rel = {rel: path for path, rel in rel_by_path.items()}
extensions = {f.suffix.lower().lstrip(".") for f in book_files}
plan = plan_pack(
[PackFile(rel) for rel in rel_by_path.values()],
supported_extensions=extensions,
series_name=series_name,
author_name=author_name,
root_depth=None if root is None else 0,
)
return [
BookGroup(
title=book.title,
series_position=book.series_position,
year=book.year,
files=[path_by_rel[rel] for rel in book.files],
)
for book in plan.books
]
def match_plan_to_files(
plan: list[PackBook],
book_files: list[Path],
*,
series_name: str | None = None,
author_name: str | None = None,
) -> list[BookGroup]:
"""Apply an approved plan to on-disk files.
Files match by release-relative path first, then by basename (archive extraction
and client save paths can shift the root), then by the on-disk basename being a
suffix of the planned name (sources that glue folder and file names together).
Book files the plan does not mention fall back to heuristic grouping so nothing
is silently dropped.
"""
if not book_files:
return []
root, rel_by_path = _relative_paths(book_files)
by_rel = {rel: path for path, rel in rel_by_path.items()}
by_name: dict[str, list[Path]] = {}
for path in book_files:
by_name.setdefault(path.name, []).append(path)
claimed: set[Path] = set()
groups: list[BookGroup] = []
for book in plan:
matched: list[Path] = []
for wanted in book.files:
wanted_rel = wanted.replace("\\", "/").lstrip("./")
candidate = by_rel.get(wanted_rel)
if candidate is None:
candidates = [
p for p in by_name.get(PurePosixPath(wanted_rel).name, []) if p not in claimed
]
candidate = candidates[0] if candidates else None
if candidate is None:
wanted_name = PurePosixPath(wanted_rel).name.lower()
candidates = [
p
for p in book_files
if p not in claimed and wanted_name.endswith(p.name.lower())
]
candidate = candidates[0] if len(candidates) == 1 else None
if candidate is not None and candidate not in claimed:
claimed.add(candidate)
matched.append(candidate)
if matched:
groups.append(
BookGroup(
title=book.title,
series_position=book.series_position,
year=book.year,
files=matched,
)
)
unmatched = [p for p in book_files if p not in claimed]
if unmatched:
groups.extend(
group_files_into_books(
unmatched, series_name=series_name, author_name=author_name, root=root
)
)
return groups
@@ -40,6 +40,7 @@ from .transfer import (
build_metadata_dict,
is_torrent_source,
process_directory,
resolve_book_groups,
resolve_hardlink_source,
should_hardlink,
transfer_book_files,
@@ -80,6 +81,7 @@ __all__ = [
"process_directory",
"record_step",
"resolve_custom_script_target",
"resolve_book_groups",
"resolve_hardlink_source",
"run_custom_script",
"safe_cleanup_path",
+110
View File
@@ -2,6 +2,7 @@
from __future__ import annotations
import dataclasses
import os
from pathlib import Path
from typing import TYPE_CHECKING
@@ -26,6 +27,7 @@ from shelfmark.download.fs import (
)
from shelfmark.download.postprocess.policy import get_file_organization, get_template
from .packs import BookGroup, PackBook, group_files_into_books, match_plan_to_files
from .scan import collect_directory_files, scan_directory_tree
from .types import TransferPlan
from .workspace import safe_cleanup_path
@@ -196,6 +198,19 @@ def transfer_book_files(
is_audiobook = check_audiobook(task.content_type)
organization_mode = organization_mode or get_file_organization(is_audiobook=is_audiobook)
groups = resolve_book_groups(task, book_files, organization_mode=organization_mode)
if groups is not None:
return _transfer_book_groups(
groups,
destination,
task,
use_hardlink=use_hardlink,
is_torrent=is_torrent,
preserve_source=preserve_source,
organization_mode=organization_mode,
)
max_attempts = _max_attempts_for_batch(len(book_files))
final_paths: list[Path] = []
@@ -299,6 +314,101 @@ def transfer_book_files(
return final_paths, None, op_counts
def resolve_book_groups(
task: DownloadTask,
book_files: list[Path],
*,
organization_mode: str,
) -> list[BookGroup] | None:
"""Split a multi-book pack into per-book groups, or None to file as one book.
An approved `book_plan` wins; a bare `multi_book` flag falls back to heuristic
grouping. Organization `none` keeps files as-is, and a split that yields a single
group is not a pack at all.
"""
if organization_mode == "none" or not (task.book_plan or task.multi_book):
return None
if task.book_plan:
plan = [
PackBook(
title=str(entry.get("title") or ""),
series_position=entry.get("series_position"),
year=entry.get("year"),
files=list(entry.get("files") or []),
)
for entry in task.book_plan
if isinstance(entry, dict)
]
groups = match_plan_to_files(
plan, book_files, series_name=task.series_name, author_name=task.author
)
else:
groups = group_files_into_books(
book_files, series_name=task.series_name, author_name=task.author
)
return groups if len(groups) > 1 else None
def _transfer_book_groups(
groups: list[BookGroup],
destination: Path,
task: DownloadTask,
*,
use_hardlink: bool,
is_torrent: bool,
preserve_source: bool,
organization_mode: str,
) -> tuple[list[Path], str | None, dict[str, int]]:
"""Transfer each book of a pack through the normal single-book path.
Each book gets an isolated task copy (the single-file path mutates `task.format`)
carrying its own title, position and year; the searched book's position must not
leak onto its siblings, while author and series name apply to all of them.
"""
all_paths: list[Path] = []
totals: dict[str, int] = {"hardlink": 0, "copy": 0, "move": 0}
errors: list[str] = []
for group in groups:
book_task = dataclasses.replace(
task,
title=group.title or task.title,
year=str(group.year) if group.year is not None else None,
subtitle=None,
series_position=group.series_position,
multi_book=False,
book_plan=None,
)
paths, error, op_counts = transfer_book_files(
group.files,
destination,
book_task,
use_hardlink=use_hardlink,
is_torrent=is_torrent,
preserve_source=preserve_source,
organization_mode=organization_mode,
source_root=group.files[0].parent,
)
for op, count in op_counts.items():
totals[op] = totals.get(op, 0) + count
if error:
errors.append(f"{group.title}: {error}")
logger.warning("Task %s: pack book %r failed: %s", task.task_id, group.title, error)
continue
all_paths.extend(paths)
if not all_paths:
return [], "; ".join(errors) or "No book files found", totals
if errors:
logger.warning(
"Task %s: pack filed with %d failed book(s): %s",
task.task_id,
len(errors),
"; ".join(errors),
)
return all_paths, None, totals
def process_directory(
directory: Path,
ingest_dir: Path,
+19
View File
@@ -32,6 +32,19 @@ _DEFAULT_QUERY = "The Great Gatsby"
_warmup_thread: threading.Thread | None = None
_warmup_lock = threading.Lock()
# Set as soon as a real release search starts. The warm-up exists to pay the cold path
# *before* the user does; once they have beaten it to the box there is nothing left to
# pre-solve, and running anyway is actively harmful - the bypasser serializes on one
# browser, so the warm-up's solve goes in front of the search the user is watching. In
# the bundle on issue #1276 that cost a full minute of a 2m27s wait, on a container 16
# seconds old, for a throwaway "The Great Gatsby" query nobody asked for.
_user_search_seen = threading.Event()
def note_user_search() -> None:
"""Record that a real search has run, so a pending warm-up stands down."""
_user_search_seen.set()
def _as_bool(value: object, *, default: bool) -> bool:
"""Coerce a config value that may arrive as a string, bool or None."""
@@ -85,6 +98,12 @@ def run_warmup() -> bool:
"""
from shelfmark.core.mirrors import has_aa_mirror_configuration
# Checked here rather than only at schedule time: the delay is what this races with,
# so the user's first search usually lands *during* the wait, not before it.
if _user_search_seen.is_set():
logger.info("Search warm-up skipped: a real search got there first")
return False
if not has_aa_mirror_configuration():
logger.debug("Search warm-up skipped: no Anna's Archive mirrors configured")
return False
+35 -9
View File
@@ -42,6 +42,7 @@ from shelfmark.config.settings import (
_SUPPORTED_BOOK_LANGUAGE,
migrate_audiobook_format_settings,
)
from shelfmark.core import search_deadline
from shelfmark.core.activity_view_state_service import ActivityViewStateService
from shelfmark.core.auth_modes import (
get_auth_check_admin_status,
@@ -62,6 +63,7 @@ from shelfmark.core.notifications import (
notify_user,
)
from shelfmark.core.prefix_middleware import PrefixMiddleware
from shelfmark.core.release_inspect_routes import register_release_inspect_routes
from shelfmark.core.request_helpers import (
coerce_bool,
emit_ws_event,
@@ -1024,6 +1026,9 @@ def _serialize_release(release: Release) -> dict:
return result
register_release_inspect_routes(app, login_required)
@app.route("/api/releases/download", methods=["POST"])
@login_required
def api_download_release() -> Response | tuple[Response, int]:
@@ -1150,7 +1155,7 @@ def api_config() -> Response | tuple[Response, int]:
"build_version": BUILD_VERSION,
"release_version": RELEASE_VERSION,
"book_languages": _SUPPORTED_BOOK_LANGUAGE,
"default_language": app_config.BOOK_LANGUAGE,
"default_language": app_config.get("BOOK_LANGUAGE", ["en"], user_id=db_user_id),
"supported_formats": app_config.SUPPORTED_FORMATS,
"supported_audiobook_formats": app_config.SUPPORTED_AUDIOBOOK_FORMATS,
"search_mode": search_mode,
@@ -2840,6 +2845,7 @@ def api_releases() -> Response | tuple[Response, int]:
manual_query=query_text if source_query_filters is not None else manual_query,
indexers=indexers,
source_filters=source_query_filters,
user_id=db_user_id,
)
if plan.source_filters is not None:
@@ -2892,6 +2898,8 @@ def api_releases() -> Response | tuple[Response, int]:
if languages_param
else None
)
# Without an explicit filter the plan falls back to this user's default languages.
db_user_id = get_session_db_user_id(session)
# Content type for audiobook vs ebook search
content_type = request.args.get("content_type", "ebook").strip()
@@ -2980,18 +2988,36 @@ def api_releases() -> Response | tuple[Response, int]:
# Search only enabled sources
sources_to_search = [src["name"] for src in list_available_sources() if src["enabled"]]
# Search each source for releases
# Search each source for releases.
#
# Under a wall-clock budget: this endpoint is synchronous, and the bypass path it
# can reach used to be allowed minutes per URL with nothing bounding the request
# as a whole. A search that ran into an unsolvable protection challenge therefore
# outlived every reverse proxy in front of it and surfaced to the user as
# "Server unavailable (504)" - a gateway timeout that blames their proxy for a
# challenge failure. The budget is shared across sources, so a stuck first source
# cannot spend the whole request on its own. See issue #1276.
all_releases = []
errors = []
source_instances = {} # Keep source instances for column config
for source_name in sources_to_search:
source, releases, error = _search_source_releases(source_name, book)
if source is not None:
source_instances[source_name] = source
all_releases.extend(releases)
if error is not None:
errors.append(error)
# A real search is under way, so a warm-up still sitting on its start-up delay
# should stand down rather than queue its throwaway solve in front of this one.
warmup.note_user_search()
with search_deadline.search_deadline():
for source_name in sources_to_search:
if search_deadline.expired():
logger.warning("Release search budget spent; %s not searched", source_name)
errors.append(f"{source_name}: {search_deadline.deadline_message()}")
continue
source, releases, error = _search_source_releases(source_name, book)
if source is not None:
source_instances[source_name] = source
all_releases.extend(releases)
if error is not None:
errors.append(error)
# Convert Release objects to dicts
releases_data = [_serialize_release(release) for release in all_releases]
+9
View File
@@ -13,6 +13,7 @@ if TYPE_CHECKING:
from shelfmark.core.models import DownloadTask
from shelfmark.core.search_plan import ReleaseSearchPlan
from shelfmark.download.postprocess.packs import PackFile
from shelfmark.metadata_providers import BookMetadata
@@ -400,6 +401,14 @@ class DownloadHandler(ABC):
"""Return private queue-time fields needed for restart-safe retry."""
return {}
def list_files(self, release_data: dict[str, Any]) -> list[PackFile] | None:
"""Return the release's file list without downloading it.
Lets the UI review a multi-book pack before queueing. Return None when the
source cannot know the files ahead of time (magnet links, usenet, ...).
"""
return None
@abstractmethod
def cancel(self, task_id: str) -> bool:
"""Cancel an in-progress download."""
@@ -1,6 +1,6 @@
"""AudiobookBay download handler - resolves magnet links and uses shared client lifecycle."""
from typing import TYPE_CHECKING
from typing import TYPE_CHECKING, Any
from urllib.parse import urlparse
from shelfmark.core.config import config
@@ -22,6 +22,7 @@ if TYPE_CHECKING:
from collections.abc import Callable
from shelfmark.core.models import DownloadTask
from shelfmark.download.postprocess.packs import PackFile
logger = setup_logger(__name__)
DEFAULT_ABB_HOSTNAME = "audiobookbay.lu"
@@ -68,6 +69,19 @@ class AudiobookBayHandler(ExternalClientHandler):
return task_id
return None
def list_files(self, release_data: dict[str, Any]) -> list[PackFile] | None:
"""Read the torrent's file list off the detail page, without downloading."""
raw_url = release_data.get("download_url") or release_data.get("source_url")
detail_url = raw_url.strip() if isinstance(raw_url, str) else ""
hostname = _resolve_allowed_detail_hostname()
if not detail_url or not _detail_url_matches_host(detail_url, hostname):
logger.debug("Cannot list files for AudiobookBay release without a valid detail URL")
return None
detail_html = scraper.fetch_detail_html(detail_url, hostname)
if not detail_html:
return None
return scraper.extract_file_list(detail_html)
def _get_client(self, protocol: str) -> DownloadClient | None:
"""Compatibility shim so module-level patching still works in tests."""
return get_client(protocol)
+107 -29
View File
@@ -2,6 +2,7 @@
import re
import time
from threading import Lock
from urllib.parse import quote, quote_plus
import requests
@@ -10,6 +11,7 @@ from bs4 import BeautifulSoup
from shelfmark.core.config import config
from shelfmark.core.logger import setup_logger
from shelfmark.download import http as downloader
from shelfmark.download.postprocess.packs import PackFile
from shelfmark.release_sources.audiobookbay.utils import normalize_search_punctuation
logger = setup_logger(__name__)
@@ -32,6 +34,13 @@ FIRST_PAGE_SESSION_REFRESH_ATTEMPTS = 2
# Legacy search parameter used by older ABB flows
LEGACY_CATEGORY_QUERY = "undefined%2Cundefined"
# Detail pages are fetched once and shared by inspection (file list) and download
# (magnet link) so a "review then download" round trip costs ABB a single request.
DETAIL_PAGE_CACHE_TTL_SECONDS = 120.0
DETAIL_PAGE_CACHE_MAX_ENTRIES = 8
_detail_page_cache: dict[str, tuple[float, str]] = {}
_detail_page_cache_lock = Lock()
# Precompiled patterns used while parsing result cards
LANGUAGE_PATTERN = re.compile(r"Language:\s*([A-Za-z]+)")
POSTED_PATTERN = re.compile(r"Posted:\s*(\d+\s+[A-Za-z]+\s+\d{4})")
@@ -39,6 +48,11 @@ FORMAT_PATTERN = re.compile(r"Format:\s*([A-Za-z0-9]+)")
BITRATE_PATTERN = re.compile(r"Bitrate:\s*([\d]+\s*[A-Za-z/]+)")
SIZE_PATTERN = re.compile(r"File Size:\s*([\d.]+)\s*([A-Za-z]+)")
INFO_HASH_LABEL_PATTERN = re.compile(r"Info Hash", re.IGNORECASE)
FILE_ROW_SIZE_PATTERN = re.compile(
r"^(?P<name>.+?)\s+(?P<size>\d+(?:\.\d+)?)\s*(?P<unit>Bytes?|KBs?|MBs?|GBs?|TBs?)$",
re.IGNORECASE,
)
_FILE_SIZE_MULTIPLIERS = {"b": 1, "k": 1024, "m": 1024**2, "g": 1024**3, "t": 1024**4}
def _coerce_non_negative_float(value: object, default: float) -> float:
@@ -348,6 +362,98 @@ def search_audiobookbay(
return results
def _get_cached_detail_page(details_url: str) -> str | None:
with _detail_page_cache_lock:
entry = _detail_page_cache.get(details_url)
if entry is None:
return None
fetched_at, html = entry
if time.monotonic() - fetched_at > DETAIL_PAGE_CACHE_TTL_SECONDS:
del _detail_page_cache[details_url]
return None
return html
def _store_cached_detail_page(details_url: str, html: str) -> None:
with _detail_page_cache_lock:
_detail_page_cache[details_url] = (time.monotonic(), html)
while len(_detail_page_cache) > DETAIL_PAGE_CACHE_MAX_ENTRIES:
oldest = min(_detail_page_cache, key=lambda key: _detail_page_cache[key][0])
del _detail_page_cache[oldest]
def clear_detail_page_cache() -> None:
"""Drop cached detail pages (used by tests)."""
with _detail_page_cache_lock:
_detail_page_cache.clear()
def _fetch_detail_page_once(details_url: str, hostname: str) -> str:
session = requests.Session()
_bootstrap_abb_session(hostname, session, DETAIL_PAGE_RETRY_ATTEMPTS)
return _coerce_markup_to_html(
downloader.html_get_page(
details_url,
retry=DETAIL_PAGE_RETRY_ATTEMPTS,
use_bypasser=False,
allow_bypasser_fallback=False,
success_delay=0,
session=session,
)
)
def fetch_detail_html(details_url: str, hostname: str = "audiobookbay.lu") -> str:
"""Fetch a detail page (one retry with a fresh session), cached briefly per URL."""
cached = _get_cached_detail_page(details_url)
if cached is not None:
logger.debug("Reusing recently fetched detail page: %s", details_url)
return cached
detail_html = _fetch_detail_page_once(details_url, hostname)
if not detail_html:
detail_html = _fetch_detail_page_once(details_url, hostname)
if detail_html:
_store_cached_detail_page(details_url, detail_html)
return detail_html
def _parse_file_row(text: str) -> PackFile | None:
match = FILE_ROW_SIZE_PATTERN.match(text.strip())
if not match:
return None
multiplier = _FILE_SIZE_MULTIPLIERS[match.group("unit")[0].lower()]
return PackFile(match.group("name"), int(float(match.group("size")) * multiplier))
def extract_file_list(detail_html: str) -> list[PackFile] | None:
"""Read the torrent file rows off a detail page.
ABB renders the torrent's file table as single-cell rows between the
"This is a Multifile Torrent" marker (absent for single-file torrents) and the
"Combined File Size" row. Returns None when the page has no such table.
"""
soup = BeautifulSoup(detail_html, "html.parser")
rows: list[PackFile] = []
for row in soup.find_all("tr"):
cells = row.find_all("td")
if not cells:
continue
label = cells[0].get_text(" ", strip=True)
if label.lower().startswith("combined file size"):
return rows or None
if len(cells) != 1:
rows = [] # a two-column metadata row means we're not in the file table yet
continue
text = cells[0].get_text(" ", strip=True)
if "multifile torrent" in text.lower():
rows = []
continue
parsed = _parse_file_row(text)
if parsed is not None:
rows.append(parsed)
return None
def extract_magnet_link(details_url: str, hostname: str = "audiobookbay.lu") -> str | None:
"""Extract info hash and trackers from book detail page, then construct magnet link.
@@ -360,35 +466,7 @@ def extract_magnet_link(details_url: str, hostname: str = "audiobookbay.lu") ->
"""
try:
session = requests.Session()
_bootstrap_abb_session(hostname, session, DETAIL_PAGE_RETRY_ATTEMPTS)
# Fetch detail page
detail_html = _coerce_markup_to_html(
downloader.html_get_page(
details_url,
retry=DETAIL_PAGE_RETRY_ATTEMPTS,
use_bypasser=False,
allow_bypasser_fallback=False,
success_delay=0,
session=session,
)
)
if not detail_html:
session = requests.Session()
_bootstrap_abb_session(hostname, session, DETAIL_PAGE_RETRY_ATTEMPTS)
detail_html = _coerce_markup_to_html(
downloader.html_get_page(
details_url,
retry=DETAIL_PAGE_RETRY_ATTEMPTS,
use_bypasser=False,
allow_bypasser_fallback=False,
success_delay=0,
session=session,
)
)
detail_html = fetch_detail_html(details_url, hostname)
if not detail_html:
logger.warning("Failed to fetch details page")
return None
+20 -1
View File
@@ -17,6 +17,7 @@ from bs4 import BeautifulSoup, Tag
from bs4.element import NavigableString
from shelfmark.config.env import DEBUG_SKIP_SOURCES, TMP_DIR
from shelfmark.core import search_deadline
from shelfmark.core.config import config
from shelfmark.core.languages import language_alias_map
from shelfmark.core.logger import setup_logger
@@ -586,6 +587,11 @@ def _fetch_search_table(url: str, selector: network.AAMirrorSelector) -> tuple[s
"""
attempt_url = url
for _ in range(len(network.get_available_aa_urls()) or 1):
# Every mirror shares the protection, so once the search budget is gone another
# mirror is another full solve nobody is still waiting for.
if search_deadline.expired():
raise SearchUnavailableError(search_deadline.deadline_message())
response = downloader.html_get_page(
attempt_url, selector=selector, allow_bypasser_fallback=True
)
@@ -1973,6 +1979,12 @@ class DirectDownloadSource(ReleaseSource):
query = f"{title} {author}".strip()
if not query:
continue
# `except Exception` below keeps this loop going past a failed variant, which
# is right for a parse error and wrong for a spent budget: without this the
# variants queue up behind each other and the request outlives the caller.
if search_deadline.expired():
logger.info("Release search budget spent; skipping remaining title variants")
break
logger.debug("Searching direct_download: title_author='%s', langs=%s", query, langs)
filters = SearchFilters(lang=langs if langs is not None else [])
@@ -1986,7 +1998,11 @@ class DirectDownloadSource(ReleaseSource):
except Exception:
logger.exception("Search error")
if not all_results and any(langs for _, langs in searches):
if (
not all_results
and any(langs for _, langs in searches)
and not search_deadline.expired()
):
logger.debug(
"No title+author results with language filter, retrying without language filter"
)
@@ -1994,6 +2010,9 @@ class DirectDownloadSource(ReleaseSource):
query = f"{title} {author}".strip()
if not query:
continue
if search_deadline.expired():
logger.info("Release search budget spent; skipping remaining retries")
break
logger.debug("Searching direct_download: title_author='%s', langs=[]", query)
try:
+9 -8
View File
@@ -428,14 +428,15 @@ class IRCReleaseSource(ReleaseSource):
"m4b": 0,
"mp3": 1,
"m4a": 2,
"flac": 3,
"opus": 4,
"ogg": 5,
"aac": 6,
"wav": 7,
"wma": 8,
"rar": 9,
"zip": 10,
"mp4": 3,
"flac": 4,
"opus": 5,
"ogg": 6,
"aac": 7,
"wav": 8,
"wma": 9,
"rar": 10,
"zip": 11,
}
def _convert_to_releases(
+68 -8
View File
@@ -8,6 +8,7 @@ from shelfmark.core.settings_registry import (
HeadingField,
PasswordField,
SettingsField,
TableField,
TagListField,
TextField,
register_settings,
@@ -16,12 +17,36 @@ from shelfmark.core.utils import normalize_http_url
def _test_newznab_connection(current_values: dict[str, Any] | None = None) -> dict[str, Any]:
"""Test the Newznab connection using current form values."""
"""Test all named Newznab connections, or the legacy connection as fallback."""
from shelfmark.core.config import config
from shelfmark.release_sources.newznab.api import NewznabClient
from shelfmark.release_sources.newznab.source import _parse_indexer_rows
current_values = current_values or {}
raw_indexers = current_values.get("NEWZNAB_INDEXERS")
if raw_indexers is None:
raw_indexers = config.get("NEWZNAB_INDEXERS", [])
indexers = _parse_indexer_rows(raw_indexers)
if indexers:
details: list[str] = []
all_successful = True
for name, url, api_key in indexers:
try:
success, message = NewznabClient(url, api_key).test_connection()
except Exception as e: # noqa: BLE001 — surface unexpected errors to the UI
success, message = False, f"Connection failed: {e!s}"
all_successful = all_successful and success
details.append(f"{name}: {message}")
summary = (
f"Connected to all {len(indexers)} indexers"
if all_successful
else "One or more Newznab indexers failed"
)
return {"success": all_successful, "message": summary, "details": details}
raw_url = str(current_values.get("NEWZNAB_URL") or config.get("NEWZNAB_URL", "") or "")
api_key = str(current_values.get("NEWZNAB_API_KEY") or config.get("NEWZNAB_API_KEY", "") or "")
@@ -64,25 +89,60 @@ def newznab_config_settings() -> list[SettingsField]:
default=False,
description="Enable searching for books via a Newznab-compatible indexer",
),
TableField(
key="NEWZNAB_INDEXERS",
label="Named Indexers",
description=(
"Add each Newznab-compatible indexer separately. The configured name is shown "
"beside every result from that indexer."
),
columns=[
{
"key": "name",
"label": "Name",
"type": "text",
"placeholder": "NZBGeek",
},
{
"key": "url",
"label": "URL",
"type": "text",
"placeholder": "https://api.nzbgeek.info",
},
{
"key": "api_key",
"label": "API Key",
"type": "password",
"placeholder": "Optional",
},
],
default=[],
add_label="Add Indexer",
empty_message=(
"No named indexers configured. The legacy single-indexer fields below are used "
"as a fallback."
),
show_when={"field": "NEWZNAB_ENABLED", "value": True},
),
TextField(
key="NEWZNAB_URL",
label="Newznab URL",
description="Base URL of your Newznab indexer or aggregator",
label="Legacy Newznab URL",
description="Used only when the named indexer list is empty",
placeholder="http://nzbhydra:5076",
required=True,
required=False,
show_when={"field": "NEWZNAB_ENABLED", "value": True},
),
PasswordField(
key="NEWZNAB_API_KEY",
label="API Key",
description="Your Newznab API key (leave blank if not required)",
label="Legacy API Key",
description="Used only with the legacy Newznab URL",
required=False,
show_when={"field": "NEWZNAB_ENABLED", "value": True},
),
ActionButton(
key="test_newznab",
label="Test Connection",
description="Verify your Newznab configuration",
label="Test Connections",
description="Verify every named indexer, or the legacy connection when the list is empty",
style="primary",
callback=_test_newznab_connection,
show_when={"field": "NEWZNAB_ENABLED", "value": True},
+125 -30
View File
@@ -4,7 +4,10 @@ from __future__ import annotations
import re
import time
from dataclasses import dataclass
from hashlib import sha256
from typing import TYPE_CHECKING, ClassVar
from urllib.parse import urlparse
if TYPE_CHECKING:
from shelfmark.core.search_plan import ReleaseSearchPlan
@@ -48,6 +51,50 @@ _DEFAULT_BOOK_CATS = [7000]
NEWZNAB_SEARCH_TIMEOUT_SECONDS = _SEARCH_TIMEOUT
@dataclass(frozen=True)
class _NamedClient:
"""A configured Newznab connection and its stable cache namespace."""
name: str
connection_id: str
client: NewznabClient
def _parse_indexer_rows(raw: object) -> list[tuple[str, str, str]]:
"""Normalize structured Newznab indexer settings.
Invalid/incomplete rows are ignored so one partially edited row cannot disable
the other configured indexers.
"""
if not isinstance(raw, list):
return []
indexers: list[tuple[str, str, str]] = []
seen_connections: set[tuple[str, str]] = set()
for row in raw:
if not isinstance(row, dict):
continue
raw_url = str(row.get("url") or "").strip()
url = normalize_http_url(raw_url)
if not url:
if raw_url:
logger.warning("Newznab: ignoring indexer row with invalid URL '%s'", raw_url)
continue
api_key = str(row.get("api_key") or "").strip()
connection_key = (url, api_key)
if connection_key in seen_connections:
continue
seen_connections.add(connection_key)
configured_name = str(row.get("name") or "").strip()
hostname = urlparse(url).hostname or ""
name = configured_name or hostname or "Newznab"
indexers.append((name, url, api_key))
return indexers
def _parse_category_ids(raw: object) -> list[int]:
"""Parse a configured category setting into Newznab category IDs.
@@ -146,8 +193,11 @@ def _newznab_result_to_release(
else None
)
# Build source_id from GUID
source_id = result.get("guid") or f"newznab:{hash(raw_title)}"
# Namespace IDs from named connections so identical GUIDs returned by two
# indexers cannot overwrite one another in the private release cache.
raw_source_id = result.get("guid") or f"newznab:{hash(raw_title)}"
connection_id = str(result.get("_newznab_connection_id") or "").strip()
source_id = f"newznab:{connection_id}:{raw_source_id}" if connection_id else raw_source_id
# Cache the raw result for the handler
cache_release(source_id, result)
@@ -272,6 +322,7 @@ class NewznabSource(ReleaseSource):
)
def _get_client(self) -> NewznabClient | None:
"""Build the legacy single-indexer client."""
raw_url = str(config.get("NEWZNAB_URL", "") or "")
api_key = str(config.get("NEWZNAB_API_KEY", "") or "")
@@ -284,6 +335,28 @@ class NewznabSource(ReleaseSource):
return NewznabClient(url, api_key or "")
def _get_clients(self) -> list[_NamedClient]:
"""Build named clients, falling back to the legacy single connection."""
configured = _parse_indexer_rows(config.get("NEWZNAB_INDEXERS", []))
if configured:
clients: list[_NamedClient] = []
for name, url, api_key in configured:
digest = sha256(f"{name}\0{url}\0{api_key}".encode()).hexdigest()[:16]
clients.append(
_NamedClient(
name=name,
connection_id=digest,
client=NewznabClient(url, api_key),
)
)
return clients
legacy_client = self._get_client()
if legacy_client is None:
return []
legacy_name = str(config.get("NEWZNAB_NAME", "") or "").strip() or "Newznab"
return [_NamedClient(name=legacy_name, connection_id="legacy", client=legacy_client)]
def search(
self,
book: BookMetadata,
@@ -293,8 +366,8 @@ class NewznabSource(ReleaseSource):
content_type: str = "ebook",
) -> list[Release]:
"""Search the Newznab indexer for releases matching the book."""
client = self._get_client()
if not client:
clients = self._get_clients()
if not clients:
logger.warning("Newznab not configured - skipping search")
return []
@@ -324,40 +397,60 @@ class NewznabSource(ReleaseSource):
all_results: list[dict] = []
try:
for idx, query in enumerate(queries, start=1):
_check_timeout()
if len(queries) > 1:
logger.debug("Newznab query %d/%d: '%s'", idx, len(queries), query)
for connection in clients:
try:
for idx, query in enumerate(queries, start=1):
_check_timeout()
if len(queries) > 1:
logger.debug(
"Newznab [%s] query %d/%d: '%s'",
connection.name,
idx,
len(queries),
query,
)
raw = client.search(query=query, categories=categories)
raw = connection.client.search(query=query, categories=categories)
# Auto-expand: retry without category filter if no results
if not raw and categories and auto_expand:
_check_timeout()
logger.info(
"Newznab: no results for '%s' with category filter, auto-expanding",
query,
)
raw = client.search(query=query, categories=None)
# Auto-expand: retry without category filter if no results
if not raw and categories and auto_expand:
_check_timeout()
logger.info(
"Newznab [%s]: no results for '%s' with category filter, "
"auto-expanding",
connection.name,
query,
)
raw = connection.client.search(query=query, categories=None)
for r in raw:
key = (
r.get("guid")
or r.get("downloadUrl")
or f"{r.get('indexer')}:{r.get('title')}"
)
if key in seen_keys:
continue
seen_keys.add(key)
all_results.append(r)
for raw_result in raw:
r = dict(raw_result)
# Aggregators can identify the underlying indexer. Plain feeds
# generally cannot, so use the user-configured connection name.
r["indexer"] = r.get("indexer") or connection.name
r["_newznab_connection_id"] = connection.connection_id
key = (
connection.connection_id,
r.get("guid")
or r.get("downloadUrl")
or f"{r.get('indexer')}:{r.get('title')}",
)
if key in seen_keys:
continue
seen_keys.add(key)
all_results.append(r)
except TimeoutError:
raise
except Exception:
logger.exception("Newznab search failed for %s", connection.name)
except TimeoutError as e:
logger.warning("Newznab search timed out: %s", e)
except Exception:
logger.exception("Newznab search failed")
return []
results = [_newznab_result_to_release(r, content_type, categories) for r in all_results]
if plan.indexers:
selected_indexers = set(plan.indexers)
results = [r for r in results if r.indexer in selected_indexers]
if results:
nzb_count = sum(1 for r in results if r.protocol == ReleaseProtocol.NZB)
@@ -379,5 +472,7 @@ class NewznabSource(ReleaseSource):
def is_available(self) -> bool:
if not config.get("NEWZNAB_ENABLED", False):
return False
if _parse_indexer_rows(config.get("NEWZNAB_INDEXERS", [])):
return True
url = normalize_http_url(str(config.get("NEWZNAB_URL", "") or ""))
return bool(url)
@@ -28,6 +28,10 @@ from shelfmark.download.clients.base_handler import (
DownloadRequest,
ExternalClientHandler,
)
from shelfmark.download.clients.torrent_utils import (
extract_file_list_from_torrent,
extract_torrent_info,
)
from shelfmark.metadata_providers import BookMetadata
from shelfmark.release_sources import register_handler
from shelfmark.release_sources.prowlarr.api import IndexerSeedSettings, ProwlarrClient
@@ -38,12 +42,14 @@ from shelfmark.release_sources.prowlarr.utils import (
coerce_int_like,
get_preferred_download_url,
get_protocol,
sanitize_download_url,
)
if TYPE_CHECKING:
from collections.abc import Callable
from shelfmark.core.models import DownloadTask
from shelfmark.download.postprocess.packs import PackFile
logger = setup_logger(__name__)
@@ -127,6 +133,24 @@ class ProwlarrHandler(ExternalClientHandler):
return settings.get(indexer_id)
def list_files(self, release_data: dict[str, Any]) -> list[PackFile] | None:
"""List a cached torrent release's files from its .torrent, without downloading.
Magnet-only and usenet releases cannot be listed ahead of time.
"""
source_id = str(release_data.get("source_id") or "")
prowlarr_result = get_release(source_id) if source_id else None
if not prowlarr_result or get_protocol(prowlarr_result) != "torrent":
return None
download_url = sanitize_download_url(str(prowlarr_result.get("downloadUrl") or "").strip())
if not download_url or download_url.startswith("magnet:"):
return None
expected_hash = str(prowlarr_result.get("infoHash") or "").strip() or None
info = extract_torrent_info(download_url, expected_hash=expected_hash)
if not info.torrent_data:
return None
return extract_file_list_from_torrent(info.torrent_data)
def _get_client(self, protocol: str) -> DownloadClient | None:
"""Compatibility shim so module-level patching still works in tests."""
return get_client(protocol)
@@ -297,6 +321,8 @@ class ProwlarrHandler(ExternalClientHandler):
search_title=title,
search_author=task.author,
)
# No language default here on purpose: this re-finds one exact release by its
# guid, and Prowlarr does not filter on plan.languages anyway.
plan = build_release_search_plan(
book,
indexers=[indexer] if indexer is not None else None,
+60 -8
View File
@@ -145,6 +145,36 @@ def _build_indexer_priority(indexers: list[dict]) -> dict[int, int]:
return priority
def _drop_unknown_indexer_ids(
selected_ids: list[int] | None, indexers: list[dict]
) -> list[int] | None:
"""Keep only selected indexer ids Prowlarr still serves.
An indexer removed or disabled in Prowlarr stays in the saved selection,
where settings can no longer show it - so it cannot be unselected, and every
search keeps querying an indexer that is gone (#1283). Dropping it here
keeps the saved selection intact for an indexer that comes back.
"""
if selected_ids is None:
return None
live_ids = {
indexer_id
for indexer in indexers
if (indexer_id := _coerce_indexer_id(indexer.get("id"))) is not None
}
kept = [indexer_id for indexer_id in selected_ids if indexer_id in live_ids]
stale = [indexer_id for indexer_id in selected_ids if indexer_id not in live_ids]
if stale:
logger.warning(
"Skipping selected Prowlarr indexers that are no longer enabled in Prowlarr: %s",
stale,
)
return kept
def _rank_for_indexer_id(indexer_id: object, priority: dict[int, int]) -> int:
"""Preference rank for an indexer id. Lower wins, unknown ranks last."""
coerced = _coerce_indexer_id(indexer_id)
@@ -317,19 +347,25 @@ def _extract_mam_language(raw_title: str) -> str | None:
return None
def _extract_mam_formats(raw_title: str) -> list[str]:
"""Extract a list of formats from MyAnonamouse titles.
def _split_mam_formats(raw_title: str) -> tuple[list[str], list[str]]:
"""Split the format tokens of a MyAnonamouse title into (recognized, unrecognized).
Prowlarr's MAM parser appends a structured bracket segment like:
[ENG / EPUB MOBI PDF]
We only trust this structured segment (and do not attempt generic title
heuristics for other indexers).
Tokens after the "/" that Shelfmark does not know as a book or audiobook format
(e.g. ``[ENG / AVI]``) are returned separately so the UI can warn that the release
will download but cannot be processed, instead of showing a bare content-type icon
that looks like an ordinary result.
"""
if not raw_title:
return []
return [], []
format_set = set(ALL_BOOK_FORMATS)
first_unrecognized: list[str] | None = None
for bracket in re.findall(r"\[([^\]]+)\]", raw_title):
if "/" not in bracket:
continue
@@ -338,15 +374,26 @@ def _extract_mam_formats(raw_title: str) -> list[str]:
tokens = re.findall(r"[A-Za-z0-9]+", after_slash)
formats: list[str] = []
unrecognized: list[str] = []
for token in tokens:
fmt = token.lower()
if fmt in format_set and fmt not in formats:
formats.append(fmt)
if fmt in format_set:
if fmt not in formats:
formats.append(fmt)
elif fmt not in unrecognized:
unrecognized.append(fmt)
if formats:
return formats
return formats, unrecognized
if unrecognized and first_unrecognized is None:
first_unrecognized = unrecognized
return []
return [], first_unrecognized or []
def _extract_mam_formats(raw_title: str) -> list[str]:
"""Extract the recognized formats from a MyAnonamouse title (see _split_mam_formats)."""
return _split_mam_formats(raw_title)[0]
def _formats_display(formats: list[str]) -> str | None:
@@ -485,6 +532,7 @@ def _prowlarr_result_to_release(
format_detected: str | None = None
formats: list[str] = []
unrecognized_formats: list[str] = []
formats_display: str | None = None
language_detected: str | None = None
if enable_format_detection:
@@ -492,7 +540,7 @@ def _prowlarr_result_to_release(
if book_title:
title = book_title
formats = _extract_mam_formats(str(raw_title or ""))
formats, unrecognized_formats = _split_mam_formats(str(raw_title or ""))
format_detected = formats[0] if formats else None
formats_display = _formats_display(formats)
language_detected = _extract_mam_language(str(raw_title or ""))
@@ -554,6 +602,9 @@ def _prowlarr_result_to_release(
"info_hash": result.get("infoHash"),
"formats": formats or None,
"formats_display": formats_display,
# Format tokens the indexer declared but Shelfmark can't process (e.g. a MAM
# "[ENG / AVI]"). Lets the UI warn instead of showing a bare content icon.
"unrecognized_formats": unrecognized_formats or None,
# Raw torznab attributes for rich tooltips (enriched indexers)
"torznab_attrs": result.get("torznabAttrs"),
},
@@ -940,6 +991,7 @@ class ProwlarrSource(ReleaseSource):
# found for this book" - the same lie as a swallowed timeout (#1249).
msg = f"could not reach Prowlarr: {e}"
raise SourceUnavailableError(msg) from e
indexer_ids = _drop_unknown_indexer_ids(indexer_ids, enabled_indexers)
indexer_priority = _build_indexer_priority(enabled_indexers)
# Some indexers benefit from title+author queries and extra format detection.
enriched_indexer_ids = client.get_enriched_indexer_ids(
+10 -43
View File
@@ -58,7 +58,6 @@ import {
isApiResponseError,
updateSelfUser,
setBookTargetState,
type DownloadReleasePayload,
} from './services/api';
import type {
Book,
@@ -87,11 +86,13 @@ import { bookSupportsTargets } from './utils/bookTargetLoader';
import { buildSearchQuery } from './utils/buildSearchQuery';
import { wasDownloadQueuedAfterResponseError } from './utils/downloadRecovery';
import { getDynamicOptionGroup } from './utils/dynamicFieldOptions';
import { resolveDefaultLanguageCodes } from './utils/languageFilters';
import { getConfiguredMetadataProviderForContentType } from './utils/metadataProviders';
import { getEffectiveMetadataSort } from './utils/metadataSort';
import { isRecord } from './utils/objectHelpers';
import { policyTrace } from './utils/policyTrace';
import { buildQueryTargets, getDefaultQueryTargetKey } from './utils/queryTargets';
import { buildReleaseDownloadPayload, type ReleaseDownloadOptions } from './utils/releasePayload';
import { applyRequestNoteToPayload } from './utils/requestConfirmation';
import { bookFromRequestData } from './utils/requestFulfil';
import {
@@ -218,6 +219,7 @@ type PendingOnBehalfDownload =
release: Release;
releaseContentType: ContentType;
actingAsUser: ActingAsUserSelection;
options?: ReleaseDownloadOptions;
}
| {
type: 'combined';
@@ -1071,41 +1073,6 @@ function App() {
[],
);
const buildReleaseDownloadPayload = useCallback(
(book: Book, release: Release, releaseContentType: ContentType): DownloadReleasePayload => {
const isManual = book.provider === 'manual';
const releasePreview =
typeof release.extra?.preview === 'string' ? release.extra.preview : undefined;
const releaseAuthor =
typeof release.extra?.author === 'string' ? release.extra.author : undefined;
return {
source: release.source,
source_id: release.source_id,
title: isManual ? release.title : book.title,
author: isManual ? releaseAuthor || '' : book.author,
year: book.year,
format: release.format,
size: release.size,
size_bytes: release.size_bytes,
download_url: release.download_url,
protocol: release.protocol,
indexer: release.indexer,
seeders: release.seeders,
extra: release.extra,
preview: isManual ? releasePreview || undefined : book.preview,
content_type: releaseContentType,
series_name: book.series_name,
series_position: book.series_position,
subtitle: book.subtitle,
// From the release, never the book: book.language is the provider's
// canonical edition, which would mislabel a translated release.
language: release.language ?? undefined,
};
},
[],
);
// When downloading a book while browsing a Hardcover list the user owns,
// automatically remove it from that list (fire-and-forget).
const searchFieldLabelsRef = useRef(searchFieldLabels);
@@ -1213,12 +1180,13 @@ function App() {
release: Release,
releaseContentType: ContentType,
onBehalfOfUserId?: number,
options?: ReleaseDownloadOptions,
): Promise<void> => {
const requestStartedAtSeconds = Date.now() / 1000;
try {
trackRelease(book.id, release.source_id);
await downloadRelease(
buildReleaseDownloadPayload(book, release, releaseContentType),
buildReleaseDownloadPayload(book, release, releaseContentType, options),
onBehalfOfUserId,
);
await fetchStatus();
@@ -1300,7 +1268,6 @@ function App() {
}
},
[
buildReleaseDownloadPayload,
fetchStatus,
openRequestConfirmation,
refreshRequestPolicy,
@@ -1415,6 +1382,7 @@ function App() {
effectivePendingOnBehalfDownload.release,
effectivePendingOnBehalfDownload.releaseContentType,
onBehalfOfUserId,
effectivePendingOnBehalfDownload.options,
);
}
setPendingOnBehalfDownload(null);
@@ -1639,6 +1607,7 @@ function App() {
book: Book,
release: Release,
releaseContentType: ContentType,
options?: ReleaseDownloadOptions,
) => {
policyTrace('release.action:start', {
bookId: book.id,
@@ -1654,11 +1623,12 @@ function App() {
release,
releaseContentType,
actingAsUser: effectiveActingAsUser,
options,
});
return;
}
await executeReleaseDownload(book, release, releaseContentType);
await executeReleaseDownload(book, release, releaseContentType, undefined, options);
};
const handleReleaseRequest = useCallback(
@@ -1927,10 +1897,7 @@ function App() {
);
const supportedFormats = config?.supported_formats || DEFAULT_SUPPORTED_FORMATS;
const defaultLanguageCodes = useMemo(
() =>
config?.default_language && config.default_language.length > 0
? config.default_language
: [bookLanguages[0]?.code || 'en'],
() => resolveDefaultLanguageCodes(config?.default_language, bookLanguages),
[config?.default_language, bookLanguages],
);
@@ -0,0 +1,193 @@
import { useState } from 'react';
import type { PackBook, PackPlan, Release } from '../types';
import {
describePackPlan,
parseSeriesPositionInput,
toBookPlanPayload,
updateReviewBook,
} from '../utils/packReview';
import { ToggleSwitch } from './shared/ToggleSwitch';
interface PackReviewPanelProps {
release: Release;
plan: PackPlan;
books: PackBook[];
onChange: (books: PackBook[]) => void;
onBack: () => void;
/** `null` means "treat the whole release as one book". */
onConfirm: (books: PackBook[] | null) => Promise<void>;
isSubmitting: boolean;
}
const inputClassName =
'w-full rounded-md border border-(--border-muted) bg-(--bg) px-2 py-1 text-sm text-(--text) focus:border-emerald-500 focus:outline-none';
export const PackReviewPanel = ({
release,
plan,
books,
onChange,
onBack,
onConfirm,
isSubmitting,
}: PackReviewPanelProps) => {
const [singleBook, setSingleBook] = useState(false);
const [expandedFiles, setExpandedFiles] = useState<number | null>(null);
const [showIgnored, setShowIgnored] = useState(false);
const payloadBooks = toBookPlanPayload(books);
const canConfirm = !isSubmitting && (singleBook || payloadBooks.length > 0);
const confirmLabel = singleBook
? 'Download as one book'
: `Download ${payloadBooks.length} ${payloadBooks.length === 1 ? 'book' : 'books'}`;
return (
<div className="flex flex-col gap-4 px-5 py-4" data-testid="pack-review-panel">
<div>
<h3 className="text-base font-semibold text-(--text)">
This release contains several books
</h3>
<p className="mt-1 text-sm text-zinc-500 dark:text-zinc-400">
<span className="font-medium text-(--text)">{release.title}</span> ·{' '}
{describePackPlan(books, plan.ignored)}
</p>
<p className="mt-1 text-xs text-zinc-500 dark:text-zinc-400">
Each book below is filed separately with its own title. Fix any titles before downloading
— the author and series come from the book you searched.
</p>
</div>
<div className="flex items-center justify-between rounded-lg border border-(--border-muted) px-3 py-2">
<div>
<p className="text-sm font-medium text-(--text)">Treat as a single book</p>
<p className="text-xs text-zinc-500 dark:text-zinc-400">
Use this if the split is wrong and the files are really one audiobook.
</p>
</div>
<ToggleSwitch
checked={singleBook}
onChange={setSingleBook}
color="emerald"
ariaLabel="Treat as a single book"
disabled={isSubmitting}
/>
</div>
<div
className={`flex flex-col divide-y divide-zinc-200/60 dark:divide-zinc-800/60 ${
singleBook ? 'pointer-events-none opacity-40' : ''
}`}
>
<div className="grid grid-cols-[minmax(0,1fr)_72px_72px_80px] gap-2 pb-1 text-xs font-medium tracking-wide text-zinc-500 uppercase dark:text-zinc-400">
<span>Title</span>
<span>Series #</span>
<span>Year</span>
<span className="text-right">Files</span>
</div>
{books.map((book, index) => (
<div key={book.files[0] ?? index} className="py-2">
<div className="grid grid-cols-[minmax(0,1fr)_72px_72px_80px] items-center gap-2">
<input
type="text"
value={book.title}
onChange={(e) =>
onChange(updateReviewBook(books, index, { title: e.target.value }))
}
aria-label={`Title for book ${index + 1}`}
className={inputClassName}
disabled={isSubmitting}
/>
<input
type="text"
inputMode="decimal"
value={book.series_position ?? ''}
onChange={(e) =>
onChange(
updateReviewBook(books, index, {
series_position: parseSeriesPositionInput(e.target.value),
}),
)
}
aria-label={`Series position for book ${index + 1}`}
className={inputClassName}
disabled={isSubmitting}
/>
<input
type="text"
inputMode="numeric"
value={book.year ?? ''}
onChange={(e) => {
const parsed = parseSeriesPositionInput(e.target.value);
onChange(
updateReviewBook(books, index, {
year: parsed === null ? null : Math.trunc(parsed),
}),
);
}}
aria-label={`Year for book ${index + 1}`}
className={inputClassName}
disabled={isSubmitting}
/>
<button
type="button"
onClick={() => setExpandedFiles(expandedFiles === index ? null : index)}
className="hover-surface rounded-md px-2 py-1 text-right text-sm text-zinc-500 transition-colors dark:text-zinc-400"
aria-expanded={expandedFiles === index}
>
{book.files.length} {book.files.length === 1 ? 'file' : 'files'}
</button>
</div>
{expandedFiles === index && (
<ul className="mt-2 max-h-40 overflow-y-auto rounded-md bg-(--bg-soft) px-3 py-2 font-mono text-xs break-all text-zinc-600 dark:text-zinc-300">
{book.files.map((file) => (
<li key={file}>{file}</li>
))}
</ul>
)}
</div>
))}
</div>
{plan.ignored.length > 0 && (
<div>
<button
type="button"
onClick={() => setShowIgnored(!showIgnored)}
className="text-xs text-zinc-500 underline-offset-2 hover:underline dark:text-zinc-400"
aria-expanded={showIgnored}
>
{plan.ignored.length} {plan.ignored.length === 1 ? 'file' : 'files'} ignored (not a book
format)
</button>
{showIgnored && (
<ul className="mt-2 max-h-32 overflow-y-auto rounded-md bg-(--bg-soft) px-3 py-2 font-mono text-xs break-all text-zinc-600 dark:text-zinc-300">
{plan.ignored.map((file) => (
<li key={file}>{file}</li>
))}
</ul>
)}
</div>
)}
<div className="flex items-center justify-end gap-3 border-t border-(--border-muted) pt-4">
<button
type="button"
onClick={onBack}
disabled={isSubmitting}
className="hover-surface rounded-lg px-3 py-1.5 text-sm font-medium text-(--text) transition-colors disabled:opacity-50"
>
&larr; Back
</button>
<button
type="button"
onClick={() => void onConfirm(singleBook ? null : payloadBooks)}
disabled={!canConfirm}
className="rounded-lg bg-emerald-600 px-4 py-1.5 text-sm font-medium text-white transition-colors hover:bg-emerald-700 disabled:cursor-not-allowed disabled:opacity-50"
>
{isSubmitting ? 'Queuing…' : confirmLabel}
</button>
</div>
</div>
);
};
@@ -8,6 +8,7 @@ import {
toStringArray,
toStringValue,
} from '../utils/objectHelpers';
import { getUnrecognizedReleaseFormats } from '../utils/releaseFormats';
import { Tooltip } from './shared/Tooltip';
interface ReleaseCellProps {
@@ -424,6 +425,38 @@ export const ReleaseCell = ({
const primaryFormat = formats?.[0] || null;
const additionalFormats = formats?.slice(1) || [];
// The indexer named a format Shelfmark can't process (e.g. MAM "[ENG / AVI]").
// Downloading it would only fail post-processing, so warn instead of showing the
// bare content-type icon that makes it look like any other result.
const unrecognizedFormats = primaryFormat ? [] : getUnrecognizedReleaseFormats(release);
if (unrecognizedFormats.length > 0) {
const unsupportedLabel = unrecognizedFormats.map((fmt) => fmt.toUpperCase()).join(', ');
const unsupportedTitle = `Unsupported format (${unsupportedLabel}) - Shelfmark cannot process this release`;
if (compact) {
return (
<span
className="font-semibold text-amber-600 dark:text-amber-400"
title={unsupportedTitle}
>
{unrecognizedFormats[0].toUpperCase()}
{unrecognizedFormats.length > 1 && ` +${unrecognizedFormats.length - 1}`}
</span>
);
}
return (
<div className="flex items-center justify-start" title={unsupportedTitle}>
<span className="inline-flex items-center gap-1">
<span className="w-13 rounded-lg bg-amber-500/20 py-0.5 text-center text-[10px] font-semibold tracking-wide whitespace-nowrap text-amber-700 sm:text-[11px] dark:text-amber-400">
{unrecognizedFormats[0].toUpperCase()}
</span>
<span className="text-[10px] font-medium whitespace-nowrap text-amber-700 sm:text-[11px] dark:text-amber-400">
Unsupported
</span>
</span>
</div>
);
}
// Use blue for book, violet for audiobook when no format specified
const noFormatStyle = isAudiobook
? { bg: 'bg-violet-500/20', text: 'text-violet-600 dark:text-violet-400' }
+117 -2
View File
@@ -7,6 +7,7 @@ import { useReleaseSearchSession } from '../hooks/releaseModal/useReleaseSearchS
import { useTabIndicator } from '../hooks/ui/useTabIndicator';
import { useBodyScrollLock } from '../hooks/useBodyScrollLock';
import { useEscapeKey } from '../hooks/useEscapeKey';
import { inspectRelease } from '../services/api';
import type {
Book,
Release,
@@ -18,6 +19,8 @@ import type {
LeadingCellConfig,
ContentType,
RequestPolicyMode,
PackBook,
PackPlan,
} from '../types';
import { isMetadataBook } from '../types';
import { bookSupportsTargets } from '../utils/bookTargetLoader';
@@ -29,7 +32,9 @@ import {
buildLanguageNormalizer,
} from '../utils/languageFilters';
import { getNestedValue, toComparableText, toStringValue } from '../utils/objectHelpers';
import { toBookPlanPayload } from '../utils/packReview';
import { getReleaseFormats } from '../utils/releaseFormats';
import { buildReleaseDownloadPayload, type ReleaseDownloadOptions } from '../utils/releasePayload';
import {
getBookTitleCandidates,
getBookAuthorCandidates,
@@ -50,6 +55,7 @@ import { BookTargetDropdown } from './BookTargetDropdown';
import { Dropdown } from './Dropdown';
import { DropdownList } from './DropdownList';
import { LanguageMultiSelect } from './LanguageMultiSelect';
import { PackReviewPanel } from './PackReviewPanel';
import { ReleaseCell } from './ReleaseCell';
// Combined mode configuration for the ReleaseModal
@@ -140,7 +146,12 @@ const DEFAULT_COLUMN_CONFIG: ReleaseColumnConfig = {
interface ReleaseModalProps {
book: Book | null;
onClose: () => void;
onDownload: (book: Book, release: Release, contentType: ContentType) => Promise<void>;
onDownload: (
book: Book,
release: Release,
contentType: ContentType,
options?: ReleaseDownloadOptions,
) => Promise<void>;
onRequestRelease?: (book: Book, release: Release, contentType: ContentType) => Promise<void>;
onRequestBook?: (book: Book, contentType: ContentType) => Promise<void>;
getPolicyModeForSource?: (source: string, contentType: ContentType) => RequestPolicyMode;
@@ -762,6 +773,15 @@ const ReleaseModalSession = ({
: supportedFormats;
const [isRequestingBook, setIsRequestingBook] = useState(false);
const [selectedRelease, setSelectedRelease] = useState<Release | null>(null);
// Multi-book packs: `multiBook` is the manual header toggle (heuristic split for
// releases we can't inspect); `packReview` holds an inspected pack awaiting approval.
const [multiBook, setMultiBook] = useState(false);
const [packReview, setPackReview] = useState<{
release: Release;
plan: PackPlan;
books: PackBook[];
} | null>(null);
const [packSubmitting, setPackSubmitting] = useState(false);
const isCombinedMode = combinedMode != null;
const combinedPhase = combinedMode?.phase ?? null;
const combinedStepLabel = combinedMode?.stepLabel ?? '';
@@ -1196,7 +1216,36 @@ const ReleaseModalSession = ({
const mode = getReleaseActionMode(release);
if (mode === 'download') {
await onDownload(book, release, contentType);
// Look at the release's files before queueing so a whole-series pack can be
// reviewed and filed as separate books instead of one mangled item.
let inspected = false;
let plan: PackPlan | null = null;
let reason: string | null = null;
try {
const inspection = await inspectRelease(
buildReleaseDownloadPayload(book, release, contentType),
);
inspected = inspection.inspected;
plan = inspection.plan;
reason = inspection.reason;
} catch (error) {
console.error('Release inspection failed:', error);
}
if (inspected && plan?.is_pack) {
setPackReview({ release, plan, books: plan.books });
return;
}
// Not a pack (or couldn't be inspected): queue exactly as before. A release we
// couldn't inspect might still be an unnoticed pack, so leave a console breadcrumb
// rather than interrupting the user; the multi-book toggle forces the split.
if (!inspected && !multiBook) {
console.warn(
`Could not inspect release "${release.title}" before download${
reason ? `: ${reason}` : ''
}. If it contains several books, enable the multi-book pack toggle.`,
);
}
await onDownload(book, release, contentType, multiBook ? { multiBook: true } : {});
handleClose();
return;
}
@@ -1215,9 +1264,31 @@ const ReleaseModalSession = ({
onRequestRelease,
contentType,
handleClose,
multiBook,
],
);
const handlePackConfirm = useCallback(
async (books: PackBook[] | null): Promise<void> => {
if (!book || !packReview) {
return;
}
setPackSubmitting(true);
try {
await onDownload(
book,
packReview.release,
contentType,
books ? { multiBook: true, bookPlan: toBookPlanPayload(books) } : {},
);
handleClose();
} finally {
setPackSubmitting(false);
}
},
[book, packReview, onDownload, contentType, handleClose],
);
const titleId = `release-modal-title-${book.id}`;
const providerDisplay =
book.provider_display_name ||
@@ -1720,6 +1791,37 @@ const ReleaseModalSession = ({
</div>
<div className="flex items-center gap-3 pr-1 pl-2">
{/* Multi-book pack toggle (fallback for releases that can't be inspected) */}
{!isCombinedMode && (
<button
type="button"
onClick={() => setMultiBook((prev) => !prev)}
className={`hover-surface relative rounded-full p-2.5 text-zinc-500 transition-colors dark:text-zinc-400 ${
multiBook ? 'text-emerald-600 dark:text-emerald-400' : ''
}`}
aria-label="Multi-book pack"
aria-pressed={multiBook}
title="Multi-book pack: file each subfolder (or each file) as a separate book. Only needed when a release can't be inspected before download."
>
<svg
className="h-4 w-4"
fill="none"
stroke="currentColor"
viewBox="0 0 24 24"
strokeWidth={1.5}
>
<path
strokeLinecap="round"
strokeLinejoin="round"
d="M6.429 9.75 2.25 12l4.179 2.25m0-4.5 5.571 3 5.571-3m-11.142 0L2.25 7.5 12 2.25l9.75 5.25-4.179 2.25m0 0L21.75 12l-4.179 2.25m0 0 4.179 2.25L12 21.75 2.25 16.5l4.179-2.25m11.142 0-5.571 3-5.571-3"
/>
</svg>
{multiBook && (
<span className="absolute top-1 right-1 h-2 w-2 rounded-full bg-emerald-500" />
)}
</button>
)}
{/* Manual query button */}
<button
type="button"
@@ -2120,6 +2222,19 @@ const ReleaseModalSession = ({
{/* Release list content */}
<div className="min-h-[200px]">
{(() => {
if (packReview) {
return (
<PackReviewPanel
release={packReview.release}
plan={packReview.plan}
books={packReview.books}
onChange={(books) => setPackReview({ ...packReview, books })}
onBack={() => setPackReview(null)}
onConfirm={handlePackConfirm}
isSubmitting={packSubmitting}
/>
);
}
if (sourcesLoading) {
return <ReleaseSkeleton />;
}
@@ -3,6 +3,7 @@ import type { Dispatch, SetStateAction } from 'react';
import { useMountEffect } from '@/hooks/useMountEffect';
import type { AppConfig, AdvancedFilterState, ContentType, SearchMode, SortOption } from '@/types';
import { buildSearchQuery } from '@/utils/buildSearchQuery';
import { resolveDefaultLanguageCodes } from '@/utils/languageFilters';
import { getEffectiveMetadataSort } from '@/utils/metadataSort';
import type { ParsedUrlSearch } from '@/utils/parseUrlSearchParams';
@@ -73,10 +74,10 @@ export const UrlSearchBootstrapMount = ({
}
const bookLanguages = config.book_languages || [];
const defaultLanguageCodes =
config.default_language && config.default_language.length > 0
? config.default_language
: [bookLanguages[0]?.code || 'en'];
const defaultLanguageCodes = resolveDefaultLanguageCodes(
config.default_language,
bookLanguages,
);
if (parsedParams.searchInput) {
setSearchInput(parsedParams.searchInput);
@@ -353,12 +353,12 @@ export const TableField = ({ field, value, onChange, disabled }: TableFieldProps
);
}
// text/path
// text/password/path
return (
<div key={col.key} className="flex min-w-0 flex-col gap-1">
{mobileLabel}
<input
type="text"
type={col.type === 'password' ? 'password' : 'text'}
value={toPrimitiveString(cellValue)}
onChange={(e) => updateCell(rowIndex, col.key, e.target.value)}
placeholder={col.placeholder}
@@ -7,7 +7,12 @@ import type {
} from '../../../types/settings';
import { HeadingField, MultiSelectField, SelectField, TextField } from '../fields';
import { FieldWrapper } from '../shared';
import { getFieldByKey, toNormalizedLowercaseTextValue, toTextValue } from './fieldHelpers';
import {
getFieldByKey,
resolveListOverride,
toNormalizedLowercaseTextValue,
toTextValue,
} from './fieldHelpers';
import type { PerUserSettings } from './types';
interface UserOverridesSectionProps {
@@ -175,16 +180,6 @@ export const UserOverridesSection = ({
label: 'Email Recipient',
description: 'Email address used for this user in Email output mode.',
};
const browserDownloadGlobalValue = Array.isArray(globalValues.DOWNLOAD_TO_BROWSER_CONTENT_TYPES)
? globalValues.DOWNLOAD_TO_BROWSER_CONTENT_TYPES.map((entry) => String(entry).trim()).filter(
(entry) => entry.length > 0,
)
: [];
const browserDownloadUserValue = Array.isArray(userSettings.DOWNLOAD_TO_BROWSER_CONTENT_TYPES)
? userSettings.DOWNLOAD_TO_BROWSER_CONTENT_TYPES.map((entry) => entry.trim()).filter(
(entry) => entry.length > 0,
)
: [];
const isOverridden = (key: DeliverySettingKey): boolean => {
if (
@@ -200,10 +195,12 @@ export const UserOverridesSection = ({
return userValue !== globalValue;
};
const isBrowserDownloadOverridden =
Object.prototype.hasOwnProperty.call(userSettings, 'DOWNLOAD_TO_BROWSER_CONTENT_TYPES') &&
userSettings.DOWNLOAD_TO_BROWSER_CONTENT_TYPES !== null &&
JSON.stringify(browserDownloadUserValue) !== JSON.stringify(browserDownloadGlobalValue);
const { value: browserDownloadContentTypes, isOverridden: isBrowserDownloadOverridden } =
resolveListOverride(
userSettings.DOWNLOAD_TO_BROWSER_CONTENT_TYPES,
globalValues.DOWNLOAD_TO_BROWSER_CONTENT_TYPES,
Object.prototype.hasOwnProperty.call(userSettings, 'DOWNLOAD_TO_BROWSER_CONTENT_TYPES'),
);
const resetKeys = (keys: DeliverySettingKey[]) => {
setUserSettings((prev) => {
@@ -232,9 +229,6 @@ export const UserOverridesSection = ({
const outputModeValue = readValue('BOOKS_OUTPUT_MODE', 'folder');
const effectiveOutputMode = normalizeMode(outputModeValue);
const browserDownloadContentTypes = isBrowserDownloadOverridden
? browserDownloadUserValue
: browserDownloadGlobalValue;
const destinationValue = readValue('DESTINATION');
const destinationAudiobookValue = readValue('DESTINATION_AUDIOBOOK');
const libraryValue = readValue('BOOKLORE_LIBRARY_ID');
@@ -1,8 +1,17 @@
import type { DeliveryPreferencesResponse } from '../../../services/api';
import type { HeadingFieldConfig, SelectFieldConfig } from '../../../types/settings';
import { HeadingField, SelectField } from '../fields';
import type {
HeadingFieldConfig,
MultiSelectFieldConfig,
SelectFieldConfig,
} from '../../../types/settings';
import { HeadingField, MultiSelectField, SelectField } from '../fields';
import { FieldWrapper } from '../shared';
import { getFieldByKey, toNormalizedLowercaseTextValue, toTextValue } from './fieldHelpers';
import {
getFieldByKey,
resolveListOverride,
toNormalizedLowercaseTextValue,
toTextValue,
} from './fieldHelpers';
import type { PerUserSettings } from './types';
interface UserSearchPreferencesSectionProps {
@@ -14,6 +23,7 @@ interface UserSearchPreferencesSectionProps {
type SearchSettingKey =
| 'SEARCH_MODE'
| 'BOOK_LANGUAGE'
| 'METADATA_PROVIDER'
| 'METADATA_PROVIDER_AUDIOBOOK'
| 'DEFAULT_RELEASE_SOURCE'
@@ -68,6 +78,15 @@ const fallbackDefaultAudiobookReleaseSourceField: SelectFieldConfig = {
options: [{ value: '', label: 'Use book release source' }],
};
const fallbackBookLanguageField: MultiSelectFieldConfig = {
type: 'MultiSelectField',
key: 'BOOK_LANGUAGE',
label: 'Default Book Languages',
description: 'Default language filter for searches.',
value: [],
options: [],
};
const searchHeading: HeadingFieldConfig = {
type: 'HeadingField',
key: 'search_preferences_heading',
@@ -120,6 +139,13 @@ export const UserSearchPreferencesSection = ({
'DEFAULT_RELEASE_SOURCE_AUDIOBOOK',
fallbackDefaultAudiobookReleaseSourceField,
);
const bookLanguageField = getFieldByKey(fields, 'BOOK_LANGUAGE', fallbackBookLanguageField);
const { value: bookLanguageValue, isOverridden: isBookLanguageOverridden } = resolveListOverride(
userSettings.BOOK_LANGUAGE,
globalValues.BOOK_LANGUAGE,
Object.prototype.hasOwnProperty.call(userSettings, 'BOOK_LANGUAGE'),
);
const isOverridden = (key: SearchSettingKey): boolean => {
if (
@@ -172,9 +198,12 @@ export const UserSearchPreferencesSection = ({
const canOverrideDefaultAudiobookReleaseSource =
isUserOverridable('DEFAULT_RELEASE_SOURCE_AUDIOBOOK') &&
preferenceKeySet.has('DEFAULT_RELEASE_SOURCE_AUDIOBOOK');
const canOverrideBookLanguage =
isUserOverridable('BOOK_LANGUAGE') && preferenceKeySet.has('BOOK_LANGUAGE');
if (
!canOverrideSearchMode &&
!canOverrideBookLanguage &&
!canOverrideMetadataProvider &&
!canOverrideAudiobookMetadataProvider &&
!canOverrideDefaultReleaseSource &&
@@ -208,6 +237,27 @@ export const UserSearchPreferencesSection = ({
</FieldWrapper>
)}
{canOverrideBookLanguage && (
<FieldWrapper
field={bookLanguageField}
resetAction={
isBookLanguageOverridden
? {
disabled: Boolean(bookLanguageField.fromEnv),
onClick: () => resetKeys(['BOOK_LANGUAGE']),
}
: undefined
}
>
<MultiSelectField
field={bookLanguageField}
value={bookLanguageValue}
onChange={(value) => setUserSettings((prev) => ({ ...prev, BOOK_LANGUAGE: value }))}
disabled={Boolean(bookLanguageField.fromEnv)}
/>
</FieldWrapper>
)}
{effectiveSearchMode === 'universal' && canOverrideMetadataProvider && (
<FieldWrapper
field={metadataProviderField}
@@ -31,6 +31,33 @@ export const toNormalizedLowercaseTextValue = (value: unknown): string => {
return toTrimmedTextValue(value).toLowerCase();
};
const toStringListValue = (value: unknown): string[] => {
if (!Array.isArray(value)) {
return [];
}
return value.map((entry) => toTrimmedTextValue(entry)).filter((entry) => entry.length > 0);
};
/**
* Resolve a list-valued per-user override against its global value.
*
* A key absent from userSettings, or set to null, is not an override. A stored list
* that matches the global one is treated as inherited, matching how
* buildUserSettingsPayload clears it on save.
*/
export const resolveListOverride = (
userValue: unknown,
globalValue: unknown,
hasUserKey: boolean,
): { value: string[]; isOverridden: boolean } => {
const globalList = toStringListValue(globalValue);
const userList = toStringListValue(userValue);
const isOverridden =
hasUserKey && userValue !== null && JSON.stringify(userList) !== JSON.stringify(globalList);
return { value: isOverridden ? userList : globalList, isOverridden };
};
export const toComparableValue = (value: unknown): string => {
if (value === null || value === undefined) {
return '';
@@ -10,6 +10,7 @@ export interface PerUserSettings {
EMAIL_RECIPIENT?: string;
DOWNLOAD_TO_BROWSER_CONTENT_TYPES?: string[];
SEARCH_MODE?: string;
BOOK_LANGUAGE?: string[];
METADATA_PROVIDER?: string;
METADATA_PROVIDER_AUDIOBOOK?: string;
DEFAULT_RELEASE_SOURCE?: string;
+14
View File
@@ -12,6 +12,8 @@ import type {
RequestSubmissionResult,
MetadataProvidersResponse,
MetadataSearchConfig,
PackBook,
InspectReleaseResponse,
} from '../types';
import type {
ActionResult,
@@ -510,6 +512,18 @@ export type DownloadReleasePayload = {
language?: string; // Release language code, for the {Language} naming variable
search_author?: string;
search_mode?: 'direct' | 'universal';
multi_book?: boolean; // Split a multi-book pack into one book per subfolder/file
book_plan?: PackBook[]; // The split the user approved before download
};
/** Inspect a release's file list before download (same body as downloadRelease). */
export const inspectRelease = async (
release: DownloadReleasePayload,
): Promise<InspectReleaseResponse> => {
return fetchJSON<InspectReleaseResponse>(`${API_BASE}/releases/inspect`, {
method: 'POST',
body: JSON.stringify(release),
});
};
export const downloadRelease = async (
@@ -7,6 +7,7 @@ import {
buildLanguageNormalizer,
getReleaseSearchLanguageParams,
releaseLanguageMatchesFilter,
resolveDefaultLanguageCodes,
} from '../utils/languageFilters';
const supportedLanguages: Language[] = [
@@ -67,3 +68,36 @@ describe('releaseLanguageMatchesFilter', () => {
expect(visibleLanguages).toHaveLength(48);
});
});
describe('resolveDefaultLanguageCodes', () => {
it('keeps an explicitly empty default as "no default filter"', () => {
// The backend stores [] to mean "do not filter"; substituting the first
// supported language here would make the UI filter where the server does not.
expect(resolveDefaultLanguageCodes([], supportedLanguages)).toEqual([]);
});
it('leaves a configured default untouched', () => {
expect(resolveDefaultLanguageCodes(['de', 'hu'], supportedLanguages)).toEqual(['de', 'hu']);
});
it('falls back to the first supported language only when nothing is configured', () => {
expect(resolveDefaultLanguageCodes(undefined, supportedLanguages)).toEqual(['en']);
expect(resolveDefaultLanguageCodes(null, [])).toEqual(['en']);
});
it('sends no language filter when an empty default is the whole selection', () => {
const defaults = resolveDefaultLanguageCodes([], supportedLanguages);
expect(
getReleaseSearchLanguageParams([LANGUAGE_OPTION_DEFAULT], supportedLanguages, defaults),
).toBe(undefined);
});
it('does not smuggle the first language into a Default+German selection', () => {
const defaults = resolveDefaultLanguageCodes([], supportedLanguages);
expect(
getReleaseSearchLanguageParams([LANGUAGE_OPTION_DEFAULT, 'de'], supportedLanguages, defaults),
).toEqual(['de']);
});
});
+53
View File
@@ -0,0 +1,53 @@
import { describe, it, expect } from 'vitest';
import type { PackBook } from '../types';
import {
describePackPlan,
parseSeriesPositionInput,
toBookPlanPayload,
updateReviewBook,
} from '../utils/packReview';
const books: PackBook[] = [
{ title: 'Leviathan Wakes', series_position: 1, year: 2011, files: ['a.m4b'] },
{ title: 'Caliban’s War', series_position: 2, year: 2012, files: ['b.m4b', 'b2.m4b'] },
];
describe('packReview.updateReviewBook', () => {
it('replaces one book without touching the others', () => {
const next = updateReviewBook(books, 1, { title: 'Caliban’s War (Unabridged)' });
expect(next[0]).toBe(books[0]);
expect(next[1]).toEqual({ ...books[1], title: 'Caliban’s War (Unabridged)' });
expect(books[1].title).toBe('Caliban’s War');
});
});
describe('packReview.parseSeriesPositionInput', () => {
it('accepts whole and fractional positions', () => {
expect(parseSeriesPositionInput('3')).toBe(3);
expect(parseSeriesPositionInput('2.5')).toBe(2.5);
});
it('treats blank or junk as no position', () => {
expect(parseSeriesPositionInput('')).toBeNull();
expect(parseSeriesPositionInput('abc')).toBeNull();
});
});
describe('packReview.toBookPlanPayload', () => {
it('trims titles, drops books without a title, and keeps file lists', () => {
const edited = updateReviewBook(books, 0, { title: ' ' });
expect(toBookPlanPayload(edited)).toEqual([
{ title: 'Caliban’s War', series_position: 2, year: 2012, files: ['b.m4b', 'b2.m4b'] },
]);
});
});
describe('packReview.describePackPlan', () => {
it('summarises books, files and ignored sidecars', () => {
expect(describePackPlan(books, ['a.txt', 'cover.jpg'])).toBe(
'2 books · 3 files · 2 files ignored',
);
expect(describePackPlan([books[0]], [])).toBe('1 book · 1 file');
});
});
+24 -1
View File
@@ -1,7 +1,7 @@
import { describe, it, expect } from 'vitest';
import type { Release } from '../types';
import { getReleaseFormats } from '../utils/releaseFormats';
import { getReleaseFormats, getUnrecognizedReleaseFormats } from '../utils/releaseFormats';
function buildRelease(overrides: Partial<Release>): Release {
return {
@@ -38,3 +38,26 @@ describe('releaseFormats.getReleaseFormats', () => {
expect(getReleaseFormats(release)).toEqual(['pdf']);
});
});
describe('releaseFormats.getUnrecognizedReleaseFormats', () => {
it('returns normalized, deduplicated unrecognized formats from extra', () => {
const release = buildRelease({
extra: { unrecognized_formats: ['AVI', ' avi ', 'WEBM'] },
});
expect(getUnrecognizedReleaseFormats(release)).toEqual(['avi', 'webm']);
});
it('accepts a single string value', () => {
const release = buildRelease({ extra: { unrecognized_formats: 'AVI' } });
expect(getUnrecognizedReleaseFormats(release)).toEqual(['avi']);
});
it('returns an empty list when nothing was flagged', () => {
expect(getUnrecognizedReleaseFormats(buildRelease({}))).toEqual([]);
expect(
getUnrecognizedReleaseFormats(buildRelease({ extra: { unrecognized_formats: null } })),
).toEqual([]);
});
});
@@ -0,0 +1,69 @@
import { describe, it, expect } from 'vitest';
import type { Book, Release } from '../types';
import { buildReleaseDownloadPayload } from '../utils/releasePayload';
const book: Book = {
id: 'hc-1',
title: 'Drive',
author: 'James S. A. Corey',
year: '2012',
preview: 'https://img/drive.jpg',
series_name: 'The Expanse',
series_position: 2.6,
subtitle: 'An Expanse Short Story',
provider: 'hardcover',
provider_id: 'hc-1',
source: 'direct_download',
};
const release: Release = {
source: 'audiobookbay',
source_id: 'abb-1',
title: 'James S. A. Corey - The Expanse Complete 2.0',
format: 'm4b',
language: 'en',
download_url: 'https://audiobookbay.lu/abss/expanse/',
};
describe('buildReleaseDownloadPayload', () => {
it('describes the searched book and the chosen release', () => {
const payload = buildReleaseDownloadPayload(book, release, 'audiobook');
expect(payload).toMatchObject({
source: 'audiobookbay',
source_id: 'abb-1',
title: 'Drive',
author: 'James S. A. Corey',
series_name: 'The Expanse',
series_position: 2.6,
language: 'en',
content_type: 'audiobook',
});
expect(payload.multi_book).toBeUndefined();
expect(payload.book_plan).toBeUndefined();
});
it('uses the release title and author for manual books', () => {
const manual: Book = { ...book, provider: 'manual', title: 'ignored' };
const withAuthor = { ...release, extra: { author: 'Release Author' } };
const payload = buildReleaseDownloadPayload(manual, withAuthor, 'audiobook');
expect(payload.title).toBe(release.title);
expect(payload.author).toBe('Release Author');
});
it('flags a manual multi-book pack', () => {
const payload = buildReleaseDownloadPayload(book, release, 'audiobook', { multiBook: true });
expect(payload.multi_book).toBe(true);
expect(payload.book_plan).toBeUndefined();
});
it('attaches the approved book plan', () => {
const plan = [{ title: 'Leviathan Wakes', series_position: 1, year: 2011, files: ['a.m4b'] }];
const payload = buildReleaseDownloadPayload(book, release, 'audiobook', {
multiBook: true,
bookPlan: plan,
});
expect(payload.multi_book).toBe(true);
expect(payload.book_plan).toEqual(plan);
});
});
+21
View File
@@ -460,6 +460,27 @@ export interface SourceSearchInfo {
}
// Response from /api/releases endpoint
/** One book split out of a multi-book pack release, files as release-relative paths. */
export interface PackBook {
title: string;
series_position: number | null;
year: number | null;
files: string[];
}
export interface PackPlan {
is_pack: boolean;
books: PackBook[];
ignored: string[];
}
export interface InspectReleaseResponse {
inspected: boolean;
reason: string | null;
files: { path: string; size: number | null }[];
plan: PackPlan | null;
}
export interface ReleasesResponse {
releases: Release[];
book: {
+7 -1
View File
@@ -146,7 +146,13 @@ export interface TableFieldColumnOption {
childOf?: string;
}
export type TableFieldColumnType = 'text' | 'select' | 'multiselect' | 'checkbox' | 'path';
export type TableFieldColumnType =
| 'text'
| 'password'
| 'select'
| 'multiselect'
| 'checkbox'
| 'path';
export interface TableFieldColumn {
key: string;
+18
View File
@@ -27,6 +27,24 @@ export const normalizeLanguageSelection = (selected: string[]): string[] => {
return unique.length ? unique : [LANGUAGE_OPTION_DEFAULT];
};
/**
* Resolve the language codes the "Default" filter option stands for.
*
* An explicitly empty list is a deliberate "no default filter" and is returned as-is;
* only a missing value falls back to the first supported language. Substituting a
* language for the empty list would make the UI filter by a language the backend
* does not apply.
*/
export const resolveDefaultLanguageCodes = (
configuredDefault: string[] | null | undefined,
supportedLanguages: Language[],
): string[] => {
if (Array.isArray(configuredDefault)) {
return configuredDefault;
}
return [supportedLanguages[0]?.code || 'en'];
};
export const getLanguageFilterValues = (
selection: string[],
supportedLanguages: Language[],
+39
View File
@@ -0,0 +1,39 @@
import type { PackBook } from '../types';
/** Return a copy of `books` with one entry patched; the input is not mutated. */
export function updateReviewBook(
books: PackBook[],
index: number,
patch: Partial<PackBook>,
): PackBook[] {
return books.map((book, i) => (i === index ? { ...book, ...patch } : book));
}
/** Parse a series-position text field: "3" → 3, "2.5" → 2.5, blank/junk → null. */
export function parseSeriesPositionInput(value: string): number | null {
const trimmed = value.trim();
if (!trimmed) return null;
const parsed = Number(trimmed);
return Number.isFinite(parsed) ? parsed : null;
}
/** The plan sent with the download: trimmed titles, untitled books dropped. */
export function toBookPlanPayload(books: PackBook[]): PackBook[] {
return books
.map((book) => ({ ...book, title: book.title.trim() }))
.filter((book) => book.title.length > 0 && book.files.length > 0);
}
function plural(count: number, noun: string): string {
return `${count} ${noun}${count === 1 ? '' : 's'}`;
}
/** "2 books · 3 files · 2 files ignored" */
export function describePackPlan(books: PackBook[], ignored: string[]): string {
const fileCount = books.reduce((sum, book) => sum + book.files.length, 0);
const parts = [plural(books.length, 'book'), plural(fileCount, 'file')];
if (ignored.length > 0) {
parts.push(`${plural(ignored.length, 'file')} ignored`);
}
return parts.join(' · ');
}
+24
View File
@@ -33,3 +33,27 @@ export function getReleaseFormats(release: Release): string[] {
return formats;
}
/**
* Format tokens the indexer declared but the backend could not map to a known
* book/audiobook format (e.g. MyAnonamouse "[ENG / AVI]"). Such a release will
* download but fail post-processing, so the UI warns instead of showing a bare
* content-type icon.
*/
export function getUnrecognizedReleaseFormats(release: Release): string[] {
const raw = release.extra?.unrecognized_formats;
const values = Array.isArray(raw) ? raw : [raw];
const formats: string[] = [];
const seen = new Set<string>();
values.forEach((value) => {
const normalized = normalizeFormatValue(value);
if (!normalized || seen.has(normalized)) {
return;
}
seen.add(normalized);
formats.push(normalized);
});
return formats;
}
+55
View File
@@ -0,0 +1,55 @@
import type { DownloadReleasePayload } from '../services/api';
import type { Book, ContentType, PackBook, Release } from '../types';
export interface ReleaseDownloadOptions {
/** Ask post-processing to split the release into one book per subfolder/file. */
multiBook?: boolean;
/** The split the user approved in the pack review panel. */
bookPlan?: PackBook[];
}
/** Build the body for /api/releases/download (and /api/releases/inspect). */
export function buildReleaseDownloadPayload(
book: Book,
release: Release,
releaseContentType: ContentType,
options: ReleaseDownloadOptions = {},
): DownloadReleasePayload {
const isManual = book.provider === 'manual';
const releasePreview =
typeof release.extra?.preview === 'string' ? release.extra.preview : undefined;
const releaseAuthor =
typeof release.extra?.author === 'string' ? release.extra.author : undefined;
const payload: DownloadReleasePayload = {
source: release.source,
source_id: release.source_id,
title: isManual ? release.title : book.title,
author: isManual ? releaseAuthor || '' : book.author,
year: book.year,
format: release.format,
size: release.size,
size_bytes: release.size_bytes,
download_url: release.download_url,
protocol: release.protocol,
indexer: release.indexer,
seeders: release.seeders,
extra: release.extra,
preview: isManual ? releasePreview || undefined : book.preview,
content_type: releaseContentType,
series_name: book.series_name,
series_position: book.series_position,
subtitle: book.subtitle,
// From the release, never the book: book.language is the provider's
// canonical edition, which would mislabel a translated release.
language: release.language ?? undefined,
};
if (options.multiBook || options.bookPlan) {
payload.multi_book = true;
}
if (options.bookPlan) {
payload.book_plan = options.bookPlan;
}
return payload;
}
+13
View File
@@ -0,0 +1,13 @@
"""AudiobookBay test fixtures."""
import pytest
from shelfmark.release_sources.audiobookbay import scraper
@pytest.fixture(autouse=True)
def _clear_detail_page_cache():
"""Detail pages are cached briefly in production; tests must not share them."""
scraper.clear_detail_page_cache()
yield
scraper.clear_detail_page_cache()
+102
View File
@@ -0,0 +1,102 @@
"""Tests for reading the torrent file list off an AudiobookBay detail page."""
from shelfmark.download.postprocess.packs import PackFile
from shelfmark.release_sources.audiobookbay import scraper
# Trimmed from a real detail page (2026-08): the file rows sit between the
# "Multifile Torrent" marker and the "Combined File Size" row.
MULTIFILE_DETAIL_HTML = """
<table>
<tr><td>Tracker:</td><td>udp://tracker.torrent.eu.org:451/announce</td></tr>
<tr><td>Creation Date:</td><td>Sun, 29 Mar 2026 21:09:39 +0200</td></tr>
<tr><td colspan='2'>This is a Multifile Torrent</td></tr>
<tr><td colspan='2'>The Expanse 9.0 - Leviathan Falls (2021).m4b 1.05 GBs</td></tr>
<tr><td colspan='2'>The Expanse 0.1 - An Expanse Novella - Drive (2012).txt 340 Bytes</td></tr>
<tr><td colspan='2'>The Expanse 0.2 - An Expanse Novella - The Churn (2014).m4b 125.72 MBs</td></tr>
<tr><td colspan='2'>The Expanse 2.0 - Caliban’s War (2012).m4b 578.97 MBs</td></tr>
<tr><td>Combined File Size:</td><td><span style='color:#00f;'>7.87</span> GBs</td></tr>
<tr><td>Info Hash:</td><td>e4a5538e26987ee58a43aa629ec2c4f2b2d46526</td></tr>
</table>
"""
SINGLE_FILE_DETAIL_HTML = """
<table>
<tr><td>Creation Date:</td><td>Sun, 29 Mar 2026 21:09:39 +0200</td></tr>
<tr><td colspan='2'>Drive.m4b 41.55 MBs</td></tr>
<tr><td>Combined File Size:</td><td><span style='color:#00f;'>41.55</span> MBs</td></tr>
<tr><td>Info Hash:</td><td>e4a5538e26987ee58a43aa629ec2c4f2b2d46526</td></tr>
</table>
"""
def test_extracts_multifile_rows_with_byte_sizes():
files = scraper.extract_file_list(MULTIFILE_DETAIL_HTML)
assert files == [
PackFile("The Expanse 9.0 - Leviathan Falls (2021).m4b", int(1.05 * 1024**3)),
PackFile("The Expanse 0.1 - An Expanse Novella - Drive (2012).txt", 340),
PackFile(
"The Expanse 0.2 - An Expanse Novella - The Churn (2014).m4b", int(125.72 * 1024**2)
),
PackFile("The Expanse 2.0 - Caliban’s War (2012).m4b", int(578.97 * 1024**2)),
]
def test_single_file_torrent_lists_the_row_before_combined_size():
assert scraper.extract_file_list(SINGLE_FILE_DETAIL_HTML) == [
PackFile("Drive.m4b", int(41.55 * 1024**2))
]
def test_page_without_file_table_returns_none():
assert scraper.extract_file_list("<html><body><p>nothing here</p></body></html>") is None
class TestHandlerListFiles:
def test_lists_files_from_detail_page(self):
from unittest.mock import patch
from shelfmark.release_sources.audiobookbay.handler import AudiobookBayHandler
with patch(
"shelfmark.release_sources.audiobookbay.handler.scraper.fetch_detail_html",
return_value=SINGLE_FILE_DETAIL_HTML,
) as fetch:
files = AudiobookBayHandler().list_files(
{"source_id": "abc", "download_url": "https://audiobookbay.lu/abss/drive/"}
)
assert files == [PackFile("Drive.m4b", int(41.55 * 1024**2))]
fetch.assert_called_once_with("https://audiobookbay.lu/abss/drive/", "audiobookbay.lu")
def test_rejects_detail_url_on_other_host(self):
from unittest.mock import patch
from shelfmark.release_sources.audiobookbay.handler import AudiobookBayHandler
with patch(
"shelfmark.release_sources.audiobookbay.handler.scraper.fetch_detail_html"
) as fetch:
files = AudiobookBayHandler().list_files(
{"source_id": "abc", "download_url": "https://evil.example/abss/drive/"}
)
assert files is None
fetch.assert_not_called()
def test_extract_magnet_link_and_file_list_share_one_page_fetch():
from unittest.mock import patch
page = MULTIFILE_DETAIL_HTML
with (
patch(
"shelfmark.release_sources.audiobookbay.scraper.downloader.html_get_page",
return_value=page,
) as get_page,
patch("shelfmark.release_sources.audiobookbay.scraper._bootstrap_abb_session"),
):
scraper.clear_detail_page_cache()
url = "https://audiobookbay.lu/abss/expanse/"
assert scraper.fetch_detail_html(url, "audiobookbay.lu") == page
magnet = scraper.extract_magnet_link(url, "audiobookbay.lu")
assert magnet is not None
assert "e4a5538e26987ee58a43aa629ec2c4f2b2d46526".upper() in magnet
assert get_page.call_count == 1
+268
View File
@@ -0,0 +1,268 @@
"""How long the internal bypasser is allowed to spend, and on what.
Issue #1276: MAX_RETRY drove *both* the outer page-load loop and the per-page method
loop, so the default of 10 meant ~40 solve attempts on one browser. That overran the
worker deadline, and the failure reached the user as `RuntimeError: TimeoutError` - a
message that says nothing about a protection challenge and sent people looking at their
reverse proxy instead.
Also covered: the undisturbed window a passive challenge gets before anything touches the
page. Anna's Archive's DDoS-Guard check has no click target and clears itself; going
straight to the click/reload methods meant the one thing that solves it was never tried.
"""
import asyncio
import pytest
@pytest.fixture
def bypass(monkeypatch):
"""internal_bypasser with sleeps and jitter removed."""
import shelfmark.bypass.internal_bypasser as internal_bypasser
async def _no_sleep(_seconds) -> None:
return None
monkeypatch.setattr(internal_bypasser.asyncio, "sleep", _no_sleep)
monkeypatch.setattr(internal_bypasser._RNG, "uniform", lambda _a, _b: 0)
return internal_bypasser
def _recording_methods(calls: list[str], count: int = 4):
def _make(name: str):
async def _method(_page) -> bool:
calls.append(name)
return False
_method.__name__ = name
return _method
return [_make(f"m{i}") for i in range(count)]
def _stub_page_state(monkeypatch, bypass, *, bypassed=False, challenge="ddos_guard"):
async def _is_bypassed(*_args, **_kwargs) -> bool:
return bypassed
async def _detect(*_args, **_kwargs) -> str:
return challenge
monkeypatch.setattr(bypass, "_is_bypassed", _is_bypassed)
monkeypatch.setattr(bypass, "_detect_challenge_type", _detect)
# --------------------------------------------------------------------------- #
# The method loop must not read MAX_RETRY
# --------------------------------------------------------------------------- #
def test_method_loop_budget_is_independent_of_max_retry(monkeypatch, bypass):
"""MAX_RETRY is the outer page-load retry; reading it here squared the budget.
Exercised against a challenge whose *type* keeps changing, because that is the case
where max_retries is what bounds the loop: the stuck-challenge guard only fires on a
run of the same type, so with a stable challenge it hid the real budget entirely.
"""
monkeypatch.setattr(type(bypass.app_config), "MAX_RETRY", 50, raising=False)
types = iter(["ddos_guard", "cloudflare"] * 100)
async def _alternating(*_args, **_kwargs) -> str:
return next(types)
calls: list[str] = []
monkeypatch.setattr(bypass, "BYPASS_METHODS", _recording_methods(calls))
monkeypatch.setattr(bypass, "_wait_for_passive_solve", _never_passes)
_stub_page_state(monkeypatch, bypass)
monkeypatch.setattr(bypass, "_detect_challenge_type", _alternating)
assert asyncio.run(bypass._bypass(object())) is False
assert len(calls) == bypass._BYPASS_METHOD_ATTEMPTS
assert len(calls) < 50, "MAX_RETRY must not reach the method loop"
def test_method_attempt_budget_is_reachable(bypass):
"""The number reported as `attempt N/X` must be a number the loop can reach.
It used to be MAX_RETRY (10) while the stuck-challenge guard capped the loop at 5,
so logs showed `4/10` and stopped, which reads like six lost attempts.
"""
assert bypass._BYPASS_METHOD_ATTEMPTS == len(bypass.BYPASS_METHODS) + 1
assert bypass._BYPASS_METHOD_ATTEMPTS >= (
max(bypass.MAX_CONSECUTIVE_SAME_CHALLENGE, len(bypass.BYPASS_METHODS) + 1)
)
# --------------------------------------------------------------------------- #
# A passive challenge gets an undisturbed window first
# --------------------------------------------------------------------------- #
async def _never_passes(*_args, **_kwargs) -> bool:
return False
def test_passive_challenge_is_given_time_before_any_method_runs(monkeypatch, bypass):
"""DDoS-Guard's JS check clears itself; nothing should click or reload first."""
calls: list[str] = []
monkeypatch.setattr(bypass, "BYPASS_METHODS", _recording_methods(calls))
_stub_page_state(monkeypatch, bypass)
async def _passes(*_args, **_kwargs) -> bool:
return True
monkeypatch.setattr(bypass, "_wait_for_passive_solve", _passes)
assert asyncio.run(bypass._bypass(object())) is True
assert calls == [], "the page must not be touched while the check can still pass"
def test_passive_wait_happens_once_not_before_every_method(monkeypatch, bypass):
"""It is a settling window, not a delay bolted onto each attempt."""
waits: list[int] = []
calls: list[str] = []
async def _count_wait(*_args, **_kwargs) -> bool:
waits.append(1)
return False
monkeypatch.setattr(bypass, "BYPASS_METHODS", _recording_methods(calls))
monkeypatch.setattr(bypass, "_wait_for_passive_solve", _count_wait)
_stub_page_state(monkeypatch, bypass)
asyncio.run(bypass._bypass(object()))
assert len(waits) == 1
assert calls == ["m0", "m1", "m2", "m3"]
def test_no_passive_wait_when_no_challenge_is_detected(monkeypatch, bypass):
"""The 'none' branch has its own settle-and-refresh handling."""
waits: list[int] = []
async def _count_wait(*_args, **_kwargs) -> bool:
waits.append(1)
return False
monkeypatch.setattr(bypass, "_wait_for_passive_solve", _count_wait)
_stub_page_state(monkeypatch, bypass, challenge="none")
class _Page:
async def reload(self, **_kwargs) -> None:
return None
asyncio.run(bypass._bypass(_Page(), max_retries=1))
assert waits == []
def test_wait_for_passive_solve_returns_as_soon_as_the_page_clears(monkeypatch, bypass):
polls = {"n": 0}
async def _is_bypassed(*_args, **_kwargs) -> bool:
polls["n"] += 1
return polls["n"] >= 3
monkeypatch.setattr(bypass, "_is_bypassed", _is_bypassed)
assert asyncio.run(bypass._wait_for_passive_solve(object())) is True
assert polls["n"] == 3
def test_wait_for_passive_solve_gives_up_at_the_window(monkeypatch, bypass):
"""It must not poll forever - the methods still need their share of the budget."""
clock = {"now": 0.0}
monkeypatch.setattr(bypass.time, "monotonic", lambda: clock["now"])
async def _tick(*_args, **_kwargs) -> bool:
clock["now"] += 1.0
return False
monkeypatch.setattr(bypass, "_is_bypassed", _tick)
assert asyncio.run(bypass._wait_for_passive_solve(object())) is False
assert clock["now"] >= bypass._PASSIVE_SOLVE_SECONDS
def test_wait_for_passive_solve_honours_cancellation(monkeypatch, bypass):
import threading
from shelfmark.bypass import BypassCancelledError
cancel = threading.Event()
cancel.set()
monkeypatch.setattr(bypass, "_is_bypassed", _never_passes)
with pytest.raises(BypassCancelledError):
asyncio.run(bypass._wait_for_passive_solve(object(), cancel))
# --------------------------------------------------------------------------- #
# The page-load loop stops while there is still time to report a real failure
# --------------------------------------------------------------------------- #
def test_page_load_loop_stops_before_the_worker_deadline(monkeypatch, bypass):
"""A stubborn challenge must produce "bypass failed", not a cancelled coroutine."""
clock = {"now": 0.0}
monkeypatch.setattr(bypass.time, "monotonic", lambda: clock["now"])
monkeypatch.delenv(bypass._BYPASS_CHILD_ENV, raising=False)
attempts = {"n": 0}
async def _create(_url):
return object()
async def _get(_url, _driver, _cancel=None) -> str:
attempts["n"] += 1
# Each pass eats a realistic slice of the budget.
clock["now"] += 120.0
return ""
async def _close(_driver) -> None:
return None
monkeypatch.setattr(bypass, "_create_cdp_browser", _create)
monkeypatch.setattr(bypass, "_get", _get)
monkeypatch.setattr(bypass, "_close_cdp_driver", _close)
class _RealWorker:
def run(self, coro, timeout=None):
return asyncio.run(coro)
monkeypatch.setattr(bypass, "_CDP_WORKER", _RealWorker())
result = bypass._run_bypass_in_current_process("https://example.com", 10)
assert result == ""
# Well short of the 10 it was asked for, and short of the deadline it had.
assert attempts["n"] < 10
budget = bypass._IN_PROCESS_BYPASS_TIMEOUT_SECONDS
assert clock["now"] < budget, "the loop must leave room to report the failure"
def test_page_load_loop_still_makes_one_attempt_on_a_spent_budget(monkeypatch, bypass):
"""The deadline check must never skip the request entirely."""
clock = {"now": 10_000.0}
monkeypatch.setattr(bypass.time, "monotonic", lambda: clock["now"])
monkeypatch.delenv(bypass._BYPASS_CHILD_ENV, raising=False)
attempts = {"n": 0}
async def _create(_url):
return object()
async def _get(_url, _driver, _cancel=None) -> str:
attempts["n"] += 1
return "<html>solved</html>"
async def _close(_driver) -> None:
return None
monkeypatch.setattr(bypass, "_create_cdp_browser", _create)
monkeypatch.setattr(bypass, "_get", _get)
monkeypatch.setattr(bypass, "_close_cdp_driver", _close)
class _RealWorker:
def run(self, coro, timeout=None):
return asyncio.run(coro)
monkeypatch.setattr(bypass, "_CDP_WORKER", _RealWorker())
assert bypass._run_bypass_in_current_process("https://example.com", 10) == "<html>solved</html>"
assert attempts["n"] == 1
+84
View File
@@ -43,6 +43,37 @@ def _store(cookies, url="https://annas-archive.gl/search"):
cs.store_extracted_cookies(url=url, cookies=cookies, user_agent="UA/1.0")
@pytest.fixture
def cookie_store_logs():
"""Collect cookie-store log messages.
The store's logger is built outside the standard hierarchy, so its records never
reach the root handler caplog installs.
"""
import logging
messages: list[str] = []
class _Capture(logging.Handler):
def emit(self, record: logging.LogRecord) -> None:
messages.append(record.getMessage())
handler = _Capture()
cs.logger.addHandler(handler)
previous = cs.logger.level
cs.logger.setLevel(logging.DEBUG)
# setup_logger builds its loggers with CustomLogger(name) rather than getLogger, so
# they are not in the manager's hierarchy - and Logger.setLevel only invalidates the
# is-enabled cache *through* the manager. Without this the logger keeps answering
# "DEBUG is off" from a cache entry made while it was at INFO.
cs.logger._cache.clear()
try:
yield messages
finally:
cs.logger.removeHandler(handler)
cs.logger.setLevel(previous)
# --------------------------------------------------------------------------- #
# Per-check cookies must not be persisted for replay
# --------------------------------------------------------------------------- #
@@ -220,3 +251,56 @@ def test_failure_only_clears_the_failing_host(monkeypatch):
assert ib.get_cf_cookies_for_domain("annas-archive.gl") == {}
assert ib.get_cf_cookies_for_domain("other-site.test") == {"__ddg1_": "other"}
# --------------------------------------------------------------------------- #
# Settling what DDoS-Guard actually treats as clearance (issue #1276)
# --------------------------------------------------------------------------- #
def test_per_check_cookies_can_be_kept_for_a_field_test(monkeypatch):
"""Which __ddg* cookies are clearance is not settled, so it has to be testable.
The store's premise - that __ddg8_/__ddg9_/__ddg10_ describe one check and must not
be replayed - is contradicted by the field reports on #1276, where every request
after a successful solve was challenged again. This env-only switch is how that gets
answered against a live host without building a branch.
"""
from shelfmark.config import env
monkeypatch.setattr(env, "DDG_REPLAY_PER_CHECK_COOKIES", True)
_store(
[
_Cookie("__ddg1_", "clearance"),
_Cookie("__ddg8_", "opaque"),
_Cookie("__ddg9_", "203.0.113.7"),
_Cookie("__ddg10_", "1786826304"),
]
)
stored = ib.get_cf_cookies_for_domain("annas-archive.gl")
assert set(stored) == {"__ddg1_", "__ddg8_", "__ddg9_", "__ddg10_"}
def test_dropping_per_check_cookies_is_the_default(monkeypatch):
"""The switch is for reproducing the question, not a behaviour change."""
from shelfmark.config import env
assert env.DDG_REPLAY_PER_CHECK_COOKIES is False
_store([_Cookie("__ddg1_", "clearance"), _Cookie("__ddg9_", "203.0.113.7")])
assert set(ib.get_cf_cookies_for_domain("annas-archive.gl")) == {"__ddg1_"}
def test_a_solve_logs_which_cookies_it_won_and_which_were_held_back(cookie_store_logs):
"""Without this, a debug log shows a solve succeed and the next request challenged,
with nothing in between to explain why."""
_store([_Cookie("__ddg1_", "clearance"), _Cookie("__ddg9_", "203.0.113.7")])
messages = cookie_store_logs
line = next((m for m in messages if "won" in m and "dropping" in m), None)
assert line is not None, messages
assert "__ddg1_" in line
assert "__ddg9_" in line
# Names only - a clearance cookie's value is a credential.
assert "clearance" not in line
assert "203.0.113.7" not in line
@@ -0,0 +1,110 @@
"""A recording that never happened must say why.
Issue #1276: the debug bundle's recording/ directory was empty, and the only trace was
three "FFmpeg already stopped" debug lines - one per bypass, each logged 20-56s after
the recorder was started, meaning ffmpeg had exited almost immediately every time. It ran
with `-loglevel 0` and no stderr capture, so nothing anywhere recorded the reason. The
screen recording is the single most useful artifact for diagnosing a bypass failure.
"""
import subprocess
import pytest
import shelfmark.bypass.internal_bypasser as ib
@pytest.fixture(autouse=True)
def _clean_display():
before = dict(ib.DISPLAY)
ib.DISPLAY["ffmpeg"] = None
ib.DISPLAY["ffmpeg_output"] = None
ib.DISPLAY["ffmpeg_error_log"] = None
yield
ib.DISPLAY.update(before)
class _Proc:
def __init__(self, returncode):
self.returncode = returncode
def poll(self):
return self.returncode
def test_ffmpeg_errors_are_captured_to_a_file_beside_the_recording(monkeypatch, tmp_path):
monkeypatch.setattr(ib, "RECORDING_DIR", tmp_path)
captured: dict[str, object] = {}
def fake_popen(cmd, **kwargs):
captured["cmd"] = cmd
captured["stderr"] = kwargs.get("stderr")
return _Proc(None)
monkeypatch.setattr(ib.subprocess, "Popen", fake_popen)
ib._start_ffmpeg_recording(display=":99")
cmd = captured["cmd"]
# Errors must not be thrown away any more.
assert "-loglevel" in cmd
assert cmd[cmd.index("-loglevel") + 1] == "error"
# stderr goes to a real file, not a pipe nothing would drain.
assert captured["stderr"] is not None
assert captured["stderr"] is not subprocess.PIPE
error_log = ib.DISPLAY["ffmpeg_error_log"]
assert error_log is not None
assert error_log.parent == tmp_path
# It sits beside the mp4, so it travels in the debug bundle.
assert error_log.name.startswith("screen_recording_")
def test_an_early_exit_is_reported_with_ffmpegs_own_reason(monkeypatch, tmp_path, caplog):
reason = "[x11grab @ 0x1] Cannot open display :99, error 1."
error_log = tmp_path / "screen_recording_x.ffmpeg.log"
error_log.write_text(reason, encoding="utf-8")
ib.DISPLAY["ffmpeg"] = _Proc(1)
ib.DISPLAY["ffmpeg_output"] = tmp_path / "screen_recording_x.mp4"
ib.DISPLAY["ffmpeg_error_log"] = error_log
messages: list[str] = []
class _Capture:
def emit(self, record):
messages.append(record.getMessage())
import logging
handler = logging.Handler()
handler.emit = _Capture().emit # type: ignore[method-assign]
ib.logger.addHandler(handler)
previous = ib.logger.level
ib.logger.setLevel(logging.DEBUG)
ib.logger._cache.clear()
try:
ib._stop_ffmpeg_recording()
finally:
ib.logger.removeHandler(handler)
ib.logger.setLevel(previous)
line = next((m for m in messages if "exited early" in m), None)
assert line is not None, messages
assert "code 1" in line
assert "Cannot open display" in line
assert ib.DISPLAY["ffmpeg"] is None
def test_summary_is_explicit_when_ffmpeg_logged_nothing(tmp_path):
empty = tmp_path / "screen_recording_y.ffmpeg.log"
empty.write_text("", encoding="utf-8")
ib.DISPLAY["ffmpeg_error_log"] = empty
assert "logged nothing" in ib._ffmpeg_error_summary()
def test_summary_survives_a_missing_log():
ib.DISPLAY["ffmpeg_error_log"] = None
assert "No FFmpeg error log" in ib._ffmpeg_error_summary()
+113
View File
@@ -0,0 +1,113 @@
"""A 429 is throttling, not a dead clearance cookie.
Issue #1276: every rejection of the cached cookies took the same exit, which cleared the
host's clearance. That is right for a 403 and for the ?check=1 redirect loop - being
challenged while presenting a cookie proves the cookie is dead - and wrong for a 429,
where the origin is rate-limiting the IP and would answer a real browser holding the very
same cookies identically.
The cost in the reported bundle: a solve completed at 13:41:23 and stored five cookies;
six seconds later a 429 threw them away, and the next query bought its own 56-second
browser solve. Reuse rate across the whole log was 0 of 2.
"""
import pytest
import shelfmark.bypass.cookie_store as cs
import shelfmark.bypass.internal_bypasser as ib
URL = "https://annas-archive.gl/search?q=dune"
HOST = "annas-archive.gl"
class _Resp:
def __init__(self, status_code, text="page"):
self.status_code = status_code
self.text = text
@pytest.fixture(autouse=True)
def _clean(monkeypatch):
monkeypatch.setattr(cs, "_cf_cookies", {})
monkeypatch.setattr(cs, "_cf_user_agents", {})
monkeypatch.setattr(cs, "_get_full_cookie_domains", set)
monkeypatch.setattr(ib, "get_proxies", lambda _url: None)
monkeypatch.setattr(ib, "get_ssl_verify", lambda _url: True)
cs._cf_cookies[HOST] = {
"__ddg1_": {"value": "clearance", "expiry": None},
"__ddg2_": {"value": "c2", "expiry": None},
}
def _cooldowns(monkeypatch):
"""Record note_rate_limited calls without arming the real per-host ladder."""
armed: list[str] = []
monkeypatch.setattr(ib.network, "note_rate_limited", lambda url: armed.append(url) or 120.0)
return armed
def test_429_keeps_the_clearance(monkeypatch):
armed = _cooldowns(monkeypatch)
monkeypatch.setattr(ib.requests, "get", lambda *a, **k: _Resp(429))
assert ib._try_with_cached_cookies(URL, HOST) is None
assert ib.get_cf_cookies_for_domain(HOST) == {"__ddg1_": "clearance", "__ddg2_": "c2"}
assert armed == [URL], "the backoff must still be armed"
def test_403_still_discards_the_clearance(monkeypatch):
"""The pre-existing behaviour for a genuine rejection must not regress."""
_cooldowns(monkeypatch)
monkeypatch.setattr(ib.requests, "get", lambda *a, **k: _Resp(403))
assert ib._try_with_cached_cookies(URL, HOST) is None
assert ib.get_cf_cookies_for_domain(HOST) == {}
def test_redirect_loop_still_discards_the_clearance(monkeypatch):
_cooldowns(monkeypatch)
def boom(*_a, **_k):
raise ib.requests.exceptions.TooManyRedirects("Exceeded 30 redirects")
monkeypatch.setattr(ib.requests, "get", boom)
assert ib._try_with_cached_cookies(URL, HOST) is None
assert ib.get_cf_cookies_for_domain(HOST) == {}
def test_a_throttled_host_is_not_handed_a_browser_solve(monkeypatch):
"""A solve cannot clear a throttle, and is itself more traffic at a host asking for
less. get_bypassed_page checks the cooldown before the queue; get() has to re-check
after it, because a request can hold for LOCKED while another collects the 429."""
monkeypatch.setattr(ib.requests, "get", lambda *a, **k: _Resp(429))
monkeypatch.setattr(ib.network, "note_rate_limited", lambda _url: 120.0)
monkeypatch.setattr(ib.network, "host_cooldown_remaining", lambda _url: 118.0)
solved: list[str] = []
monkeypatch.setattr(
ib, "_run_bypass_in_current_process", lambda url, *a, **k: solved.append(url) or "html"
)
monkeypatch.setattr(ib.env, "DOCKERMODE", False)
with pytest.raises(ib.network.RateLimitedError) as excinfo:
ib.get(URL, retry=1)
assert solved == [], "no browser should have been started"
assert "rate-limited" in str(excinfo.value)
# And the clearance survives, ready for when the cooldown clears.
assert ib.get_cf_cookies_for_domain(HOST) == {"__ddg1_": "clearance", "__ddg2_": "c2"}
def test_a_host_that_is_not_throttled_still_solves(monkeypatch):
monkeypatch.setattr(ib.requests, "get", lambda *a, **k: _Resp(403))
monkeypatch.setattr(ib.network, "host_cooldown_remaining", lambda _url: 0.0)
solved: list[str] = []
monkeypatch.setattr(
ib, "_run_bypass_in_current_process", lambda url, *a, **k: solved.append(url) or "html"
)
monkeypatch.setattr(ib.env, "DOCKERMODE", False)
assert ib.get(URL, retry=1) == "html"
assert solved == [URL]
+11
View File
@@ -0,0 +1,11 @@
"""Tests for general settings definitions."""
from shelfmark.config.settings import general_settings
def test_supported_formats_stay_admin_only():
"""The format lists describe the library, not a reader, so they are not overridable."""
fields = {field.key: field for field in general_settings() if hasattr(field, "key")}
assert fields["SUPPORTED_FORMATS"].user_overridable is False
assert fields["SUPPORTED_AUDIOBOOK_FORMATS"].user_overridable is False
+63
View File
@@ -1,8 +1,17 @@
"""Tests for search mode settings definitions."""
import json
import pytest
from shelfmark.config.settings import search_mode_settings
def _search_mode_field(key: str):
fields = {field.key: field for field in search_mode_settings() if hasattr(field, "key")}
return fields[key]
def test_search_mode_settings_include_release_source_links_toggle():
fields = {field.key: field for field in search_mode_settings() if hasattr(field, "key")}
@@ -11,3 +20,57 @@ def test_search_mode_settings_include_release_source_links_toggle():
assert field.label == "Show Release Source Links"
assert field.default is True
assert field.user_overridable is False
def test_book_language_is_user_overridable():
fields = {field.key: field for field in search_mode_settings() if hasattr(field, "key")}
field = fields["BOOK_LANGUAGE"]
assert field.label == "Default Book Languages"
assert field.default == ["en"]
assert field.user_overridable is True
def test_book_language_stored_under_the_general_tab_still_resolves(tmp_path):
"""BOOK_LANGUAGE moved from the General tab to Search Mode with no migration.
That is only safe because both tabs persist into the same settings.json, so an
install that stored the value while the field lived on General keeps it. If the
two tabs ever get separate files, every existing install silently falls back to
the ["en"] default instead.
"""
from shelfmark.core.settings_registry import _get_config_file_path, get_setting_value
(tmp_path / "settings.json").write_text(json.dumps({"BOOK_LANGUAGE": ["de", "fr"]}))
with pytest.MonkeyPatch.context() as monkeypatch:
monkeypatch.delenv("BOOK_LANGUAGE", raising=False)
monkeypatch.setattr("shelfmark.config.env.CONFIG_DIR", tmp_path)
assert _get_config_file_path("search_mode") == _get_config_file_path("general")
assert get_setting_value(_search_mode_field("BOOK_LANGUAGE"), "search_mode") == ["de", "fr"]
def test_book_language_uses_its_default_on_a_fresh_install(tmp_path):
from shelfmark.core.settings_registry import get_setting_value
with pytest.MonkeyPatch.context() as monkeypatch:
monkeypatch.delenv("BOOK_LANGUAGE", raising=False)
monkeypatch.setattr("shelfmark.config.env.CONFIG_DIR", tmp_path)
assert get_setting_value(_search_mode_field("BOOK_LANGUAGE"), "search_mode") == ["en"]
def test_book_language_env_var_beats_the_stored_value(tmp_path):
from shelfmark.core.settings_registry import get_setting_value, is_value_from_env
(tmp_path / "settings.json").write_text(json.dumps({"BOOK_LANGUAGE": ["de", "fr"]}))
field = _search_mode_field("BOOK_LANGUAGE")
with pytest.MonkeyPatch.context() as monkeypatch:
monkeypatch.setenv("BOOK_LANGUAGE", "es,it")
monkeypatch.setattr("shelfmark.config.env.CONFIG_DIR", tmp_path)
assert is_value_from_env(field) is True
assert get_setting_value(field, "search_mode") == ["es", "it"]
+1 -1
View File
@@ -90,7 +90,7 @@ def test_visible_self_settings_sections_field_defaults_and_options():
{
"value": "search",
"label": "Search Preferences",
"description": "Show personal search mode and provider settings.",
"description": "Show personal search mode, language, and provider settings.",
},
{
"value": "notifications",
+82
View File
@@ -508,6 +508,72 @@ class TestAdminUserUpdateEndpoint:
settings = user_db.get_user_settings(user["id"])
assert settings["DESTINATION_AUDIOBOOK"] == "/audiobooks/alice"
def test_update_user_settings_normalizes_book_languages(self, admin_client, user_db):
user = user_db.create_user(username="alice")
resp = admin_client.put(
f"/api/admin/users/{user['id']}",
json={"settings": {"BOOK_LANGUAGE": ["German", "de", " en "]}},
)
assert resp.status_code == 200
settings = user_db.get_user_settings(user["id"])
assert settings["BOOK_LANGUAGE"] == ["de", "en"]
def test_update_user_settings_rejects_unknown_book_language(self, admin_client, user_db):
user = user_db.create_user(username="alice")
resp = admin_client.put(
f"/api/admin/users/{user['id']}",
json={"settings": {"BOOK_LANGUAGE": ["de", "klingon"]}},
)
assert resp.status_code == 400
assert resp.json["error"] == "Invalid settings payload"
assert any(
"BOOK_LANGUAGE contains an unsupported language: klingon" in msg
for msg in resp.json["details"]
)
assert user_db.get_user_settings(user["id"]) == {}
def test_update_user_settings_skips_blank_book_languages(self, admin_client, user_db):
"""A trailing comma or a blank slot means "nothing there", not an unknown language."""
user = user_db.create_user(username="alice")
resp = admin_client.put(
f"/api/admin/users/{user['id']}",
json={"settings": {"BOOK_LANGUAGE": "en,"}},
)
assert resp.status_code == 200
assert user_db.get_user_settings(user["id"])["BOOK_LANGUAGE"] == ["en"]
def test_update_user_settings_normalizes_every_validated_search_key(
self, admin_client, user_db
):
"""Keys the search validator recognises keep their normalized value.
These three were validated but then stored raw, because the write-back was
gated on a hand-maintained subset of the validated keys. A padded provider
name was accepted and then persisted with its padding, so every later lookup
of it failed.
"""
user = user_db.create_user(username="alice")
resp = admin_client.put(
f"/api/admin/users/{user['id']}",
json={
"settings": {
"METADATA_PROVIDER_COMBINED": " openlibrary ",
"SHOW_COMBINED_SELECTOR": "yes",
"FORCE_COMBINED_SEARCH": "",
}
},
)
assert resp.status_code == 200
settings = user_db.get_user_settings(user["id"])
assert settings["METADATA_PROVIDER_COMBINED"] == "openlibrary"
assert settings["SHOW_COMBINED_SELECTOR"] is True
assert settings["FORCE_COMBINED_SEARCH"] is False
def test_update_user_settings_accepts_notification_overrides(self, admin_client, user_db):
user = user_db.create_user(username="alice")
@@ -1266,6 +1332,7 @@ class TestAdminSearchPreferences:
assert data["tab"] == "search_mode"
assert data["keys"] == [
"SEARCH_MODE",
"BOOK_LANGUAGE",
"SHOW_COMBINED_SELECTOR",
"FORCE_COMBINED_SEARCH",
"METADATA_PROVIDER",
@@ -1295,6 +1362,21 @@ class TestAdminSearchPreferences:
assert data["effective"]["DEFAULT_RELEASE_SOURCE_AUDIOBOOK"]["source"] == "user_override"
assert data["effective"]["DEFAULT_RELEASE_SOURCE_AUDIOBOOK"]["value"] == "audiobookbay"
def test_reports_book_language_override(self, admin_client, user_db):
user = user_db.create_user(username="alice")
user_db.set_user_settings(user["id"], {"BOOK_LANGUAGE": ["de", "en"]})
resp = admin_client.get(f"/api/admin/users/{user['id']}/search-preferences")
assert resp.status_code == 200
data = resp.json
assert data["userOverrides"]["BOOK_LANGUAGE"] == ["de", "en"]
assert data["effective"]["BOOK_LANGUAGE"] == {
"value": ["de", "en"],
"source": "user_override",
}
assert data["globalValues"]["BOOK_LANGUAGE"] == ["en"]
def test_returns_404_for_unknown_user(self, admin_client):
resp = admin_client.get("/api/admin/users/9999/search-preferences")
assert resp.status_code == 404
+3
View File
@@ -48,6 +48,7 @@ def test_config_endpoint_uses_user_scope_and_runtime_flags(main_module, client):
"DEFAULT_RELEASE_SOURCE": "prowlarr",
"DEFAULT_RELEASE_SOURCE_AUDIOBOOK": "audiobookbay",
"DOWNLOAD_TO_BROWSER_CONTENT_TYPES": ["book", "audiobook"],
"BOOK_LANGUAGE": ["de", "en"],
"AUTO_OPEN_DOWNLOADS_SIDEBAR": False,
"HARDCOVER_AUTO_REMOVE_ON_DOWNLOAD": True,
"AA_DEFAULT_SORT": "newest",
@@ -75,12 +76,14 @@ def test_config_endpoint_uses_user_scope_and_runtime_flags(main_module, client):
assert data["default_release_source"] == "prowlarr"
assert data["default_release_source_audiobook"] == "audiobookbay"
assert data["download_to_browser_content_types"] == ["book", "audiobook"]
assert data["default_language"] == ["de", "en"]
assert data["settings_enabled"] is True
assert data["metadata_default_sort"] == "relevance"
assert ("SHOW_RELEASE_SOURCE_LINKS", None) in calls
assert ("SHOW_COMBINED_SELECTOR", 42) in calls
assert ("DOWNLOAD_TO_BROWSER_CONTENT_TYPES", 42) in calls
assert ("BOOK_LANGUAGE", 42) in calls
def test_config_endpoint_falls_back_to_audiobook_metadata_provider(main_module, client):
+22
View File
@@ -143,3 +143,25 @@ def test_build_user_preferences_payload_reports_effective_sources(monkeypatch):
fields_by_key = {field["key"]: field for field in payload["fields"]}
assert fields_by_key["DESTINATION"]["fromEnv"] is False
assert fields_by_key["BOOKS_OUTPUT_MODE"]["fromEnv"] is True
def test_build_user_preferences_payload_carries_the_language_default():
"""BOOK_LANGUAGE rides along with the other search preferences on its tab."""
import shelfmark.config.settings # noqa: F401
from shelfmark.core.user_settings_overrides import build_user_preferences_payload
user_db = SimpleNamespace(get_user_settings=lambda user_id: {"BOOK_LANGUAGE": ["de", "en"]})
payload = build_user_preferences_payload(user_db, 7, "search_mode")
assert payload["tab"] == "search_mode"
assert {"SEARCH_MODE", "BOOK_LANGUAGE"}.issubset(payload["keys"])
assert payload["userOverrides"] == {"BOOK_LANGUAGE": ["de", "en"]}
assert payload["effective"]["BOOK_LANGUAGE"] == {
"value": ["de", "en"],
"source": "user_override",
}
fields_by_key = {field["key"]: field for field in payload["fields"]}
assert fields_by_key["BOOK_LANGUAGE"]["type"] == "MultiSelectField"
assert fields_by_key["BOOK_LANGUAGE"]["options"]
+7 -1
View File
@@ -6,7 +6,13 @@ class TestReleaseSearchPlanManualQuery:
def test_manual_query_overrides_plan(self, monkeypatch):
import shelfmark.core.search_plan as sp
monkeypatch.setattr(sp.config, "BOOK_LANGUAGE", ["en", "hu"], raising=False)
monkeypatch.setattr(
sp.config,
"get",
lambda key, default=None, user_id=None: (
["en", "hu"] if key == "BOOK_LANGUAGE" else default
),
)
book = BookMetadata(
provider="hardcover",
+44
View File
@@ -1458,3 +1458,47 @@ def test_external_directory_prefers_files_over_archives_and_keeps_source(
# TMP staging should be cleaned.
assert list(staging.iterdir()) == []
def test_audiobook_multifile_mp4_chapters_are_book_files(tmp_path):
"""Per-chapter .mp4 audiobooks (as MyAnonamouse ships AAC releases) must be
recognised as book files instead of failing with "No book files found"."""
from shelfmark.download.postprocess.router import (
post_process_download as _post_process_download,
)
source_dir = tmp_path / "downloads" / "Andy Weir (2020) The Martian"
source_dir.mkdir(parents=True)
for part in (1, 2):
(source_dir / f"{part:04d} Andy Weir (2020) The Martian.mp4").write_text(f"audio{part}")
(source_dir / "cover.jpg").write_text("jpg")
ingest = tmp_path / "ingest"
ingest.mkdir()
task = DownloadTask(
task_id="mp4-audio-grouped",
source="prowlarr",
title="The Martian",
author="Andy Weir",
format="mp4",
content_type="audiobook",
search_mode=SearchMode.UNIVERSAL,
)
with patch("shelfmark.core.config.config") as mock_config:
mock_config.get = _build_config(
ingest,
organization="rename_and_group",
supported_audiobook_formats=["mp4"],
)
mock_config.CUSTOM_SCRIPT = None
result = _post_process_download(source_dir, task, Event(), lambda *_args: None)
grouped_dir = ingest / "Andy Weir (2020) The Martian"
assert result is not None
assert Path(result).parent == grouped_dir
assert sorted(path.name for path in grouped_dir.glob("*.mp4")) == [
"0001 Andy Weir (2020) The Martian.mp4",
"0002 Andy Weir (2020) The Martian.mp4",
]
+212
View File
@@ -0,0 +1,212 @@
"""Multi-book packs are filed one book at a time through the normal pipeline."""
import os
from pathlib import Path
from threading import Event
from unittest.mock import patch
from shelfmark.core.models import DownloadTask, SearchMode
from tests.core.test_processing_integration import _build_config, _sync_config
def _run(temp_path: Path, task: DownloadTask, ingest: Path, staging: Path, **config_kwargs):
from shelfmark.download.postprocess.router import post_process_download
statuses: list[tuple[str, str | None]] = []
with (
patch("shelfmark.core.config.config") as mock_config,
patch("shelfmark.config.env.TMP_DIR", staging),
):
mock_config.get = _build_config(
ingest,
organization=config_kwargs.pop("organization", "organize"),
supported_audiobook_formats=["m4b", "mp3"],
audiobook_organize_template=config_kwargs.pop(
"audiobook_organize_template", "{Author}/{Title}/{Title}{ - PartNumber}"
),
**config_kwargs,
)
mock_config.CUSTOM_SCRIPT = None
_sync_config(mock_config, mock_config)
result = post_process_download(
temp_path, task, Event(), lambda s, m=None: statuses.append((s, m))
)
return result, statuses
def _nested_pack(root: Path) -> Path:
pack = root / "Sun Eater"
for folder, name in (
("Book 1 - Empire of Silence", "empire.m4b"),
("Book 2 - Howling Dark", "howling.m4b"),
):
(pack / folder).mkdir(parents=True)
(pack / folder / name).write_text(name)
(pack / "cover.jpg").write_text("img")
return pack
def _audiobook_task(**overrides) -> DownloadTask:
fields = {
"task_id": "pack-1",
"source": "direct_download",
"title": "Drive",
"author": "James S. A. Corey",
"content_type": "audiobook",
"series_name": "The Expanse",
"series_position": 2.6,
"search_mode": SearchMode.UNIVERSAL,
}
fields.update(overrides)
return DownloadTask(**fields)
def test_approved_plan_files_each_book_with_its_own_title(tmp_path):
staging = tmp_path / "staging"
ingest = tmp_path / "ingest"
staging.mkdir()
ingest.mkdir()
pack = staging / "Expanse"
pack.mkdir()
for name in (
"The Expanse 1.0 - Leviathan Wakes (2011).m4b",
"The Expanse 2.0 - Caliban's War (2012).m4b",
):
(pack / name).write_text(name)
(pack / "The Expanse 1.0 - Leviathan Wakes (2011).txt").write_text("notes")
task = _audiobook_task(
title="Sun Eater", # the searched book; must not name the pack's books
book_plan=[
{
"title": "Leviathan Wakes (edited)",
"series_position": 1.0,
"year": 2011,
"files": ["The Expanse 1.0 - Leviathan Wakes (2011).m4b"],
},
{
"title": "Caliban's War",
"series_position": 2.0,
"year": 2012,
"files": ["The Expanse 2.0 - Caliban's War (2012).m4b"],
},
],
)
result, statuses = _run(pack, task, ingest, staging)
assert result is not None
author_dir = ingest / "James S. A. Corey"
assert sorted(p.name for p in author_dir.iterdir()) == [
"Caliban's War",
"Leviathan Wakes (edited)",
]
assert (author_dir / "Leviathan Wakes (edited)" / "Leviathan Wakes (edited).m4b").exists()
assert (author_dir / "Caliban's War" / "Caliban's War.m4b").exists()
assert statuses[-1] == ("complete", "Complete (2 books, 2 files)")
def test_multi_book_flag_splits_nested_pack_heuristically(tmp_path):
staging = tmp_path / "staging"
ingest = tmp_path / "ingest"
staging.mkdir()
ingest.mkdir()
pack = _nested_pack(staging)
result, _ = _run(pack, _audiobook_task(multi_book=True), ingest, staging)
assert result is not None
author_dir = ingest / "James S. A. Corey"
assert (author_dir / "Empire of Silence" / "Empire of Silence.m4b").exists()
assert (author_dir / "Howling Dark" / "Howling Dark.m4b").exists()
def test_pack_book_series_position_does_not_leak_from_searched_book(tmp_path):
staging = tmp_path / "staging"
ingest = tmp_path / "ingest"
staging.mkdir()
ingest.mkdir()
pack = staging / "Two"
for folder in ("Alpha", "Beta"):
(pack / folder).mkdir(parents=True)
(pack / folder / f"{folder.lower()}.m4b").write_text(folder)
result, _ = _run(
pack,
_audiobook_task(multi_book=True),
ingest,
staging,
audiobook_organize_template="{Author}/{SeriesPosition - }{Title}/{Title}",
)
assert result is not None
assert sorted(p.name for p in (ingest / "James S. A. Corey").iterdir()) == ["Alpha", "Beta"]
def test_multifile_book_inside_pack_keeps_part_numbers_per_book(tmp_path):
staging = tmp_path / "staging"
ingest = tmp_path / "ingest"
staging.mkdir()
ingest.mkdir()
pack = staging / "Pack"
(pack / "Book 1 - One").mkdir(parents=True)
(pack / "Book 2 - Two").mkdir(parents=True)
for i in (1, 2, 3):
(pack / "Book 1 - One" / f"part{i}.mp3").write_text(str(i))
(pack / "Book 2 - Two" / "two.mp3").write_text("t")
result, statuses = _run(pack, _audiobook_task(multi_book=True), ingest, staging)
assert result is not None
one = ingest / "James S. A. Corey" / "One"
assert sorted(p.name for p in one.iterdir()) == ["One - 01.mp3", "One - 02.mp3", "One - 03.mp3"]
assert (ingest / "James S. A. Corey" / "Two" / "Two.mp3").exists()
assert statuses[-1] == ("complete", "Complete (2 books, 4 files)")
def test_hardlinked_torrent_pack_leaves_source_tree_intact(tmp_path):
downloads = tmp_path / "downloads"
ingest = tmp_path / "ingest"
downloads.mkdir()
ingest.mkdir()
pack = _nested_pack(downloads)
task = _audiobook_task(source="prowlarr", multi_book=True, original_download_path=str(pack))
result, _ = _run(pack, task, ingest, tmp_path / "staging", hardlink=True)
assert result is not None
empire_src = pack / "Book 1 - Empire of Silence" / "empire.m4b"
empire_dst = ingest / "James S. A. Corey" / "Empire of Silence" / "Empire of Silence.m4b"
assert empire_src.exists()
assert empire_dst.exists()
assert os.stat(empire_src).st_ino == os.stat(empire_dst).st_ino
def test_without_pack_fields_nested_pack_is_still_one_book(tmp_path):
staging = tmp_path / "staging"
ingest = tmp_path / "ingest"
staging.mkdir()
ingest.mkdir()
pack = _nested_pack(staging)
result, _ = _run(pack, _audiobook_task(), ingest, staging)
assert result is not None
drive = ingest / "James S. A. Corey" / "Drive"
assert sorted(p.name for p in drive.iterdir()) == ["Drive - 01.m4b", "Drive - 02.m4b"]
def test_single_group_with_multi_book_flag_uses_searched_title(tmp_path):
staging = tmp_path / "staging"
ingest = tmp_path / "ingest"
staging.mkdir()
ingest.mkdir()
pack = staging / "Series" / "Book 1 - Solo"
pack.mkdir(parents=True)
(pack / "solo.m4b").write_text("s")
result, statuses = _run(pack.parent, _audiobook_task(multi_book=True), ingest, staging)
assert result is not None
assert (ingest / "James S. A. Corey" / "Drive" / "Drive.m4b").exists()
assert statuses[-1] == ("complete", "Complete")
+134
View File
@@ -0,0 +1,134 @@
"""API tests for POST /api/releases/inspect."""
from __future__ import annotations
import importlib
from unittest.mock import patch
import pytest
from shelfmark.download.postprocess.packs import PackFile
@pytest.fixture(scope="module")
def main_module():
with patch("shelfmark.download.orchestrator.start"):
import shelfmark.main as main
importlib.reload(main)
return main
@pytest.fixture
def client(main_module):
client = main_module.app.test_client()
with client.session_transaction() as sess:
sess["user_id"] = "tester"
sess["is_admin"] = False
return client
class _Handler:
def __init__(self, files):
self._files = files
def list_files(self, release_data):
if isinstance(self._files, Exception):
raise self._files
return self._files
def _inspect(client, handler, payload=None):
body = {
"source": "audiobookbay",
"source_id": "abc",
"title": "Drive",
"content_type": "audiobook",
"series_name": "The Expanse",
**(payload or {}),
}
with patch("shelfmark.core.release_inspect_routes.get_handler", return_value=handler):
return client.post("/api/releases/inspect", json=body)
def test_pack_release_returns_a_plan(client):
files = [
PackFile("The Expanse 1.0 - Leviathan Wakes (2011).m4b", 100),
PackFile("The Expanse 1.0 - Leviathan Wakes (2011).txt", 1),
PackFile("The Expanse 2.0 - Caliban's War (2012).m4b", 100),
]
resp = _inspect(client, _Handler(files))
assert resp.status_code == 200
data = resp.get_json()
assert data["inspected"] is True
assert data["reason"] is None
assert data["files"] == [
{"path": "The Expanse 1.0 - Leviathan Wakes (2011).m4b", "size": 100},
{"path": "The Expanse 1.0 - Leviathan Wakes (2011).txt", "size": 1},
{"path": "The Expanse 2.0 - Caliban's War (2012).m4b", "size": 100},
]
assert data["plan"]["is_pack"] is True
assert data["plan"]["ignored"] == ["The Expanse 1.0 - Leviathan Wakes (2011).txt"]
assert data["plan"]["books"] == [
{
"title": "Leviathan Wakes",
"series_position": 1.0,
"year": 2011,
"files": ["The Expanse 1.0 - Leviathan Wakes (2011).m4b"],
},
{
"title": "Caliban's War",
"series_position": 2.0,
"year": 2012,
"files": ["The Expanse 2.0 - Caliban's War (2012).m4b"],
},
]
def test_single_book_release_is_not_a_pack(client):
resp = _inspect(client, _Handler([PackFile("Drive.m4b", 5)]))
data = resp.get_json()
assert data["inspected"] is True
assert data["plan"]["is_pack"] is False
assert len(data["plan"]["books"]) == 1
def test_handler_without_file_list_reports_not_inspected(client):
resp = _inspect(client, _Handler(None))
assert resp.status_code == 200
data = resp.get_json()
assert data["inspected"] is False
assert data["reason"]
assert data["plan"] is None
def test_handler_failure_reports_not_inspected_without_500(client):
resp = _inspect(client, _Handler(RuntimeError("boom")))
assert resp.status_code == 200
data = resp.get_json()
assert data["inspected"] is False
assert "boom" in data["reason"]
def test_unknown_source_is_a_client_error(client):
with patch(
"shelfmark.core.release_inspect_routes.get_handler", side_effect=ValueError("no source")
):
resp = client.post(
"/api/releases/inspect", json={"source": "nope", "source_id": "x", "title": "t"}
)
assert resp.status_code == 400
def test_missing_source_id_is_a_client_error(client):
resp = client.post("/api/releases/inspect", json={"source": "audiobookbay"})
assert resp.status_code == 400
def test_requires_login(main_module):
anonymous = main_module.app.test_client()
with patch.object(main_module, "load_active_auth_mode", return_value="builtin"):
resp = anonymous.post(
"/api/releases/inspect", json={"source": "audiobookbay", "source_id": "a"}
)
assert resp.status_code == 401
+139
View File
@@ -0,0 +1,139 @@
"""GET /api/releases answers within a budget instead of outliving the caller.
Issue #1276: the endpoint is synchronous and had no deadline, while the bypass path it
reaches was allowed ~840s per URL. A protection challenge nobody could solve therefore
ran until the reverse proxy in front of Shelfmark gave up, and the user was shown
"Server unavailable (504). If using a reverse proxy, check its configuration." - which
names the wrong thing entirely.
"""
from __future__ import annotations
import importlib
from unittest.mock import patch
import pytest
from shelfmark.core import search_deadline
@pytest.fixture(scope="module")
def main_module():
with patch("shelfmark.download.orchestrator.start"):
import shelfmark.main as main
importlib.reload(main)
return main
@pytest.fixture
def client(main_module):
return main_module.app.test_client()
def _authenticate(client) -> None:
with client.session_transaction() as sess:
sess["user_id"] = "alice"
sess["is_admin"] = False
sess["db_user_id"] = 7
def _request(client, main_module, sources, search_impl):
"""Drive /api/releases with a stubbed source list and search implementation."""
class _Source:
def search(self, book, plan, *, expand_search=False, content_type="ebook"):
return search_impl(book, plan)
def get_column_config(self):
from shelfmark.release_sources import _default_column_config
return _default_column_config()
with (
patch.object(main_module, "get_auth_mode", return_value="none"),
patch("shelfmark.release_sources.list_available_sources", return_value=sources),
patch("shelfmark.release_sources.get_source", return_value=_Source()),
patch("shelfmark.release_sources.source_results_are_releases", return_value=False),
):
return client.get(
"/api/releases",
query_string={"provider": "manual", "book_id": "abc", "title": "Dune"},
)
def test_a_search_runs_under_a_budget(client, main_module):
"""The handler must put a deadline in force for whatever the sources do."""
_authenticate(client)
observed: list[float | None] = []
def _search(_book, _plan):
deadline = search_deadline.current()
observed.append(deadline.budget_seconds if deadline else None)
return []
resp = _request(client, main_module, [{"name": "direct_download", "enabled": True}], _search)
assert resp.status_code == 200
assert observed and observed[0] is not None, "no budget was in force during the search"
def test_the_budget_is_shared_across_sources(client, main_module):
"""A stuck first source must not spend the whole request on its own."""
_authenticate(client)
searched: list[str] = []
def _search(book, _plan):
searched.append(book.title)
# Whatever the first source did, it ran the clock out.
deadline = search_deadline.current()
if deadline is not None:
deadline.event.set()
return []
sources = [
{"name": "direct_download", "enabled": True},
{"name": "prowlarr", "enabled": True},
]
resp = _request(client, main_module, sources, _search)
assert len(searched) == 1, "the second source should not have been started"
assert "ran out of time" in resp.get_json()["error"]
def test_the_failure_carries_a_message_the_frontend_will_show(client, main_module):
"""The whole point of the budget.
Shelfmark answers 503 when a search comes back empty with errors, and the frontend
only substitutes its "Server unavailable ... check your reverse proxy" text when the
body carries no message of its own. So the budget has to produce a body that names
the protection challenge - and has to trip before the proxy's own timeout, where
there would be no body at all.
"""
_authenticate(client)
def _search(_book, _plan):
deadline = search_deadline.current()
if deadline is not None:
deadline.event.set()
return []
sources = [
{"name": "direct_download", "enabled": True},
{"name": "prowlarr", "enabled": True},
]
resp = _request(client, main_module, sources, _search)
message = resp.get_json()["error"]
assert "protection challenge" in message
assert "reverse proxy" not in message
# The source prefix is stripped by the handler; the sentence must survive intact.
assert message.startswith("The release search ran out of time")
def test_no_budget_leaks_out_of_the_request(client, main_module):
"""A queued download later on must not inherit a search's deadline."""
_authenticate(client)
_request(client, main_module, [{"name": "direct_download", "enabled": True}], lambda *_: [])
assert search_deadline.current() is None
@@ -114,6 +114,41 @@ def test_releases_accepts_direct_download_provider(main_module, client):
assert all(call.args == ("direct_download",) for call in mock_get_source.call_args_list)
def test_releases_falls_back_to_the_session_users_default_languages(main_module, client):
"""A request without a language filter searches in the reader's own languages."""
import shelfmark.core.search_plan as search_plan
planned_languages: list[list[str] | None] = []
class _LanguageProbeSource(_FakeDirectSource):
def search(self, book, plan, expand_search=False, content_type="ebook"):
planned_languages.append(plan.languages)
return []
def fake_get(key, default=None, user_id=None):
if key == "BOOK_LANGUAGE":
return ["de"] if user_id == 42 else ["en"]
return default
with client.session_transaction() as session:
session["user_id"] = "reader"
session["db_user_id"] = 42
with patch.object(main_module, "get_auth_mode", return_value="none"):
with patch.object(search_plan, "config", SimpleNamespace(get=fake_get)):
with patch("shelfmark.release_sources.get_source", return_value=_LanguageProbeSource()):
resp = client.get(
"/api/releases",
query_string={
"provider": "direct_download",
"book_id": "md5-abc",
},
)
assert resp.status_code == 200
assert planned_languages == [["de"]]
def test_releases_direct_provider_returns_404_when_book_missing(main_module, client):
class _MissingDirectSource:
def get_record(self, record_id, *, fetch_download_count=True):
+257
View File
@@ -0,0 +1,257 @@
"""The wall-clock budget on a release search.
Issue #1276: `/api/releases` is synchronous and nothing bounded it, while the bypass path
it can reach was allowed ~840s per URL. A search that ran into an unsolvable protection
challenge therefore outlived every reverse proxy in front of it, and the user was shown
"Server unavailable (504)" - a gateway timeout that names their proxy rather than the
challenge that actually failed.
"""
import threading
import pytest
from shelfmark.core import search_deadline
@pytest.fixture(autouse=True)
def _no_ambient_deadline():
"""Each test starts outside any budget."""
token = search_deadline._current.set(None)
yield
search_deadline._current.reset(token)
# --------------------------------------------------------------------------- #
# The budget itself
# --------------------------------------------------------------------------- #
def test_no_budget_outside_a_search():
"""A queued download must not inherit a search's budget."""
assert search_deadline.current() is None
assert search_deadline.expired() is False
assert search_deadline.cancel_event() is None
def test_budget_applies_inside_the_context_and_not_after():
with search_deadline.search_deadline(60) as deadline:
assert search_deadline.current() is deadline
assert search_deadline.expired() is False
assert search_deadline.current() is None
def test_expiry_trips_the_cancel_event():
"""The Event is the mechanism: the bypassers poll it and know nothing of deadlines."""
with search_deadline.search_deadline(0.05):
event = search_deadline.cancel_event()
assert isinstance(event, threading.Event)
assert event.wait(timeout=5) is True
assert search_deadline.expired() is True
def test_timer_is_cancelled_on_exit():
"""A finished search must not leave a timer running to fire later."""
with search_deadline.search_deadline(3600) as deadline:
pass
assert deadline._timer.finished.is_set()
def test_message_names_the_challenge_not_the_proxy():
with search_deadline.search_deadline(120):
message = search_deadline.deadline_message()
assert "120s" in message
assert "protection challenge" in message
assert "504" not in message
# --------------------------------------------------------------------------- #
# Reading the setting
# --------------------------------------------------------------------------- #
@pytest.mark.parametrize(
("configured", "expected"),
[
(600, 600.0),
("450", 450.0),
(None, search_deadline.DEFAULT_SEARCH_BUDGET_SECONDS),
("nonsense", search_deadline.DEFAULT_SEARCH_BUDGET_SECONDS),
(0, search_deadline.DEFAULT_SEARCH_BUDGET_SECONDS),
(True, search_deadline.DEFAULT_SEARCH_BUDGET_SECONDS),
(5, 30.0), # clamped up: below this nothing can finish
(99999, 1800.0), # clamped down
],
)
def test_budget_seconds_coerces_and_clamps(monkeypatch, configured, expected):
from shelfmark.core.config import config as app_config
monkeypatch.setattr(
app_config,
"get",
lambda key, default=None: configured if key == "RELEASE_SEARCH_TIMEOUT" else default,
)
assert search_deadline.budget_seconds() == expected
# --------------------------------------------------------------------------- #
# What the search path does with it
# --------------------------------------------------------------------------- #
def test_html_get_page_will_not_start_a_bypass_on_a_spent_budget(monkeypatch):
"""A minutes-long solve nobody is still waiting for is worse than a clear failure."""
import shelfmark.download.http as http
started: list[str] = []
monkeypatch.setattr(
http, "get_bypassed_page", lambda *a, **k: started.append(a[0]) or "<html/>"
)
monkeypatch.setattr(http, "_is_cf_bypass_enabled", lambda: True)
monkeypatch.setattr(http, "_bypass_grace_seconds", lambda: 1.0)
monkeypatch.setattr(http.network, "should_rotate_dns_for_url", lambda _url: True)
monkeypatch.setattr(http, "get_proxies", lambda _url: {})
monkeypatch.setattr(http.time, "sleep", lambda _s: None)
class _Selector:
current_base = "https://annas-archive.gl"
attempts_this_dns = 0
last_failure = None
def rewrite(self, url):
return url
with search_deadline.search_deadline(60) as deadline:
deadline.event.set()
result = http.html_get_page(
"https://annas-archive.gl/search",
retry=1,
selector=_Selector(),
use_bypasser=True,
success_delay=0,
)
assert result == ""
assert started == [], "no solve should have been started"
def test_search_budget_becomes_the_cancel_flag(monkeypatch):
"""This is what makes the budget bite on a solve already running."""
import shelfmark.download.http as http
seen: list[object] = []
def fake_bypass(_url, _selector=None, cancel_flag=None):
seen.append(cancel_flag)
return "<html>solved</html>"
monkeypatch.setattr(http, "get_bypassed_page", fake_bypass)
monkeypatch.setattr(http, "_is_cf_bypass_enabled", lambda: True)
monkeypatch.setattr(http, "_bypass_grace_seconds", lambda: 1.0)
monkeypatch.setattr(http.network, "should_rotate_dns_for_url", lambda _url: True)
monkeypatch.setattr(http, "get_proxies", lambda _url: {})
monkeypatch.setattr(http.time, "sleep", lambda _s: None)
class _Selector:
current_base = "https://annas-archive.gl"
attempts_this_dns = 0
last_failure = None
def rewrite(self, url):
return url
with search_deadline.search_deadline(60) as deadline:
http.html_get_page(
"https://annas-archive.gl/search",
retry=1,
selector=_Selector(),
use_bypasser=True,
success_delay=0,
)
assert seen == [deadline.event]
def test_a_callers_own_cancel_flag_is_not_replaced(monkeypatch):
"""A queued download brings its own and must keep it."""
import shelfmark.download.http as http
seen: list[object] = []
def fake_bypass(_url, _selector=None, cancel_flag=None):
seen.append(cancel_flag)
return "<html>solved</html>"
monkeypatch.setattr(http, "get_bypassed_page", fake_bypass)
monkeypatch.setattr(http, "_is_cf_bypass_enabled", lambda: True)
monkeypatch.setattr(http, "_bypass_grace_seconds", lambda: 1.0)
monkeypatch.setattr(http.network, "should_rotate_dns_for_url", lambda _url: True)
monkeypatch.setattr(http, "get_proxies", lambda _url: {})
monkeypatch.setattr(http.time, "sleep", lambda _s: None)
class _Selector:
current_base = "https://annas-archive.gl"
attempts_this_dns = 0
last_failure = None
def rewrite(self, url):
return url
own_flag = threading.Event()
with search_deadline.search_deadline(60):
http.html_get_page(
"https://annas-archive.gl/search",
retry=1,
selector=_Selector(),
cancel_flag=own_flag,
use_bypasser=True,
success_delay=0,
)
assert seen == [own_flag]
def test_expired_budget_reports_the_challenge_not_a_cancellation(monkeypatch):
"""The budget trips the same flag a user's cancel does; the messages must differ."""
import shelfmark.download.http as http
from shelfmark.bypass import BypassCancelledError
def fake_bypass(*_a, **_k):
msg = "Bypass cancelled"
raise BypassCancelledError(msg)
monkeypatch.setattr(http, "get_bypassed_page", fake_bypass)
monkeypatch.setattr(http, "_is_cf_bypass_enabled", lambda: True)
monkeypatch.setattr(http, "_bypass_grace_seconds", lambda: 1.0)
monkeypatch.setattr(http.network, "should_rotate_dns_for_url", lambda _url: True)
monkeypatch.setattr(http, "get_proxies", lambda _url: {})
monkeypatch.setattr(http.time, "sleep", lambda _s: None)
failures: list[str] = []
class _Selector:
current_base = "https://annas-archive.gl"
attempts_this_dns = 0
def rewrite(self, url):
return url
@property
def last_failure(self):
return None
@last_failure.setter
def last_failure(self, value):
if value:
failures.append(value)
with search_deadline.search_deadline(60) as deadline:
# Expire mid-solve: the flag is set, but the caller reaches the handler by way of
# BypassCancelledError, which on its own reads as "someone cancelled this".
deadline.expires_at = 0.0
http.html_get_page(
"https://annas-archive.gl/search",
retry=1,
selector=_Selector(),
use_bypasser=True,
success_delay=0,
)
assert failures, "a give-up reason should have been recorded"
assert "ran out of time" in failures[-1]
assert "cancelled" not in failures[-1]
+57 -3
View File
@@ -2,12 +2,27 @@ from shelfmark.core.search_plan import build_release_search_plan
from shelfmark.metadata_providers import BookMetadata
def _book_language_config_get(
global_languages: list[str],
user_languages: list[str] | None = None,
):
"""Stand in for config.get(), answering BOOK_LANGUAGE per user."""
def _get(key: str, default: object = None, user_id: int | None = None) -> object:
if key != "BOOK_LANGUAGE":
return default
if user_id is not None and user_languages is not None:
return user_languages
return global_languages
return _get
class TestReleaseSearchPlan:
def test_uses_default_languages_when_none(self, monkeypatch):
# config.BOOK_LANGUAGE is a Config attribute; patch the instance.
import shelfmark.core.search_plan as sp
monkeypatch.setattr(sp.config, "BOOK_LANGUAGE", ["en", "hu"], raising=False)
monkeypatch.setattr(sp.config, "get", _book_language_config_get(["en", "hu"]))
book = BookMetadata(
provider="hardcover",
@@ -46,7 +61,7 @@ class TestReleaseSearchPlan:
def test_all_language_disables_grouping(self, monkeypatch):
import shelfmark.core.search_plan as sp
monkeypatch.setattr(sp.config, "BOOK_LANGUAGE", ["en"], raising=False)
monkeypatch.setattr(sp.config, "get", _book_language_config_get(["en"]))
book = BookMetadata(
provider="hardcover",
@@ -68,3 +83,42 @@ class TestReleaseSearchPlan:
assert [(v.title, v.languages) for v in plan.grouped_title_variants] == [
("The Lightning Thief", None),
]
def test_user_default_languages_beat_the_global_default(self, monkeypatch):
import shelfmark.core.search_plan as sp
monkeypatch.setattr(
sp.config,
"get",
_book_language_config_get(["en"], user_languages=["de", "en"]),
)
book = BookMetadata(
provider="hardcover",
provider_id="123",
title="The Final Empire",
authors=["Brandon Sanderson"],
)
assert build_release_search_plan(book).languages == ["en"]
assert build_release_search_plan(book, user_id=7).languages == ["de", "en"]
def test_explicit_languages_beat_the_user_default(self, monkeypatch):
import shelfmark.core.search_plan as sp
monkeypatch.setattr(
sp.config,
"get",
_book_language_config_get(["en"], user_languages=["de", "en"]),
)
book = BookMetadata(
provider="hardcover",
provider_id="123",
title="The Final Empire",
authors=["Brandon Sanderson"],
)
plan = build_release_search_plan(book, languages=["fr"], user_id=7)
assert plan.languages == ["fr"]
@@ -0,0 +1,148 @@
"""Language filters must reach the sources as ISO codes.
Issue #1276: a debug bundle showed `BOOK_LANGUAGE=english` arriving at Anna's Archive
verbatim as `&lang=english`. AA matches that parameter against ISO codes, so it is not a
loose spelling of `lang=en` - it is a facet value AA does not have, and it empties every
search. In that bundle a successful solve of an ISBN search for Philosopher's Stone
returned zero hits, while a warm-up query carrying no language filter hit the same host
in the same minute and returned 50.
Only the per-user override was normalised (config.users_settings.validate). The global
default - which is what the env var feeds - was passed through with nothing but a
.strip(), so anyone carrying `BOOK_LANGUAGE=english` from the old docs had every search
silently filtered to nothing, with no error anywhere.
"""
import logging
import pytest
import shelfmark.core.search_plan as sp
from shelfmark.metadata_providers import BookMetadata
def _config_get(global_languages):
def _get(key: str, default: object = None, user_id: int | None = None) -> object:
return global_languages if key == "BOOK_LANGUAGE" else default
return _get
def _book() -> BookMetadata:
return BookMetadata(
provider="hardcover",
provider_id="123",
title="Harry Potter and the Philosopher's Stone",
search_title="Harry Potter and the Philosopher's Stone",
search_author="J.K. Rowling",
authors=["J.K. Rowling"],
)
@pytest.fixture
def plan_logs():
"""Collect search_plan log records.
setup_logger builds its loggers outside the standard hierarchy, so caplog's root
handler never sees them, and Logger.setLevel cannot clear their is-enabled cache.
"""
messages: list[str] = []
class _Capture(logging.Handler):
def emit(self, record: logging.LogRecord) -> None:
messages.append(record.getMessage())
handler = _Capture()
sp.logger.addHandler(handler)
previous = sp.logger.level
sp.logger.setLevel(logging.DEBUG)
sp.logger._cache.clear()
try:
yield messages
finally:
sp.logger.removeHandler(handler)
sp.logger.setLevel(previous)
# --------------------------------------------------------------------------- #
# The global default (what BOOK_LANGUAGE feeds)
# --------------------------------------------------------------------------- #
@pytest.mark.parametrize(
("configured", "expected"),
[
(["english"], ["en"]), # the spelling from the old docs - the reported bug
("english", ["en"]), # env vars arrive as a bare string
(["English"], ["en"]),
(["eng"], ["en"]), # ISO 639-2
(["en"], ["en"]), # already a code, unchanged
(["english", "german"], ["en", "de"]),
(["english", "en", "English"], ["en"]), # collapses to one code
],
)
def test_default_languages_reach_the_plan_as_iso_codes(monkeypatch, configured, expected):
monkeypatch.setattr(sp.config, "get", _config_get(configured))
plan = sp.build_release_search_plan(_book(), languages=None)
assert plan.languages == expected
def test_unrecognised_default_searches_unfiltered_rather_than_empty(monkeypatch, plan_logs):
"""Dropping the filter is the safe failure: a warned-about unfiltered search beats
telling the user a book AA is full of does not exist."""
monkeypatch.setattr(sp.config, "get", _config_get(["klingon"]))
plan = sp.build_release_search_plan(_book(), languages=None)
assert plan.languages is None
assert any("klingon" in m and "BOOK_LANGUAGE" in m for m in plan_logs), plan_logs
def test_partly_unrecognised_default_keeps_what_resolved(monkeypatch, plan_logs):
monkeypatch.setattr(sp.config, "get", _config_get(["english", "klingon"]))
plan = sp.build_release_search_plan(_book(), languages=None)
assert plan.languages == ["en"]
assert any("klingon" in m for m in plan_logs)
def test_a_clean_default_logs_no_warning(monkeypatch, plan_logs):
monkeypatch.setattr(sp.config, "get", _config_get(["en", "de"]))
sp.build_release_search_plan(_book(), languages=None)
assert not [m for m in plan_logs if "Ignoring unrecognised" in m]
# --------------------------------------------------------------------------- #
# Explicit request languages go through the same door
# --------------------------------------------------------------------------- #
def test_explicit_languages_are_normalised_too(monkeypatch):
"""The request branch had the same bare .strip(), so an API client could reproduce
the bug even with BOOK_LANGUAGE set correctly."""
monkeypatch.setattr(sp.config, "get", _config_get(["en"]))
plan = sp.build_release_search_plan(_book(), languages=["english", "German"])
assert plan.languages == ["en", "de"]
def test_all_still_means_no_language_filter(monkeypatch):
monkeypatch.setattr(sp.config, "get", _config_get(["en"]))
assert sp.build_release_search_plan(_book(), languages=["all"]).languages is None
def test_blank_entries_are_skipped(monkeypatch):
monkeypatch.setattr(sp.config, "get", _config_get(["en"]))
plan = sp.build_release_search_plan(_book(), languages=["english", "", " ", None])
assert plan.languages == ["en"]
def test_no_configured_default_leaves_the_filter_off(monkeypatch):
monkeypatch.setattr(sp.config, "get", _config_get(None))
assert sp.build_release_search_plan(_book(), languages=None).languages is None
+43
View File
@@ -85,6 +85,7 @@ def test_users_me_edit_context_includes_search_preferences_when_visible(app, use
{
"SEARCH_MODE": "universal",
"METADATA_PROVIDER": "openlibrary",
"BOOK_LANGUAGE": ["de", "en"],
},
)
client = _authed_client_for_user(app, user)
@@ -102,8 +103,13 @@ def test_users_me_edit_context_includes_search_preferences_when_visible(app, use
assert resp.json["searchPreferences"]["tab"] == "search_mode"
assert resp.json["searchPreferences"]["effective"]["SEARCH_MODE"]["value"] == "universal"
assert resp.json["searchPreferences"]["effective"]["SEARCH_MODE"]["source"] == "user_override"
assert resp.json["searchPreferences"]["effective"]["BOOK_LANGUAGE"]["value"] == ["de", "en"]
assert resp.json["searchPreferences"]["effective"]["BOOK_LANGUAGE"]["source"] == (
"user_override"
)
assert "SEARCH_MODE" in resp.json["userOverridableKeys"]
assert "METADATA_PROVIDER" in resp.json["userOverridableKeys"]
assert "BOOK_LANGUAGE" in resp.json["userOverridableKeys"]
assert resp.json["notificationPreferences"] is None
assert resp.json["userOverridableKeys"] == sorted(resp.json["userOverridableKeys"])
@@ -170,6 +176,43 @@ def test_users_me_update_accepts_visible_section_settings(app, user_db):
assert resp.json["settings"]["DESTINATION"] == "/books/alice"
def test_users_me_update_accepts_book_language_when_search_section_visible(app, user_db):
user = user_db.create_user(username="alice")
client = _authed_client_for_user(app, user)
with patch("shelfmark.core.self_user_routes.load_active_auth_mode", return_value="builtin"):
with patch(
"shelfmark.core.self_user_routes.app_config.get",
side_effect=_visible_sections_config_get(["search"]),
):
resp = client.put(
"/api/users/me",
json={"settings": {"BOOK_LANGUAGE": ["German", "en"]}},
)
assert resp.status_code == 200
assert user_db.get_user_settings(user["id"])["BOOK_LANGUAGE"] == ["de", "en"]
def test_users_me_update_rejects_book_language_when_search_section_hidden(app, user_db):
user = user_db.create_user(username="alice")
client = _authed_client_for_user(app, user)
with patch("shelfmark.core.self_user_routes.load_active_auth_mode", return_value="builtin"):
with patch(
"shelfmark.core.self_user_routes.app_config.get",
side_effect=_visible_sections_config_get(["delivery"]),
):
resp = client.put(
"/api/users/me",
json={"settings": {"BOOK_LANGUAGE": ["de"]}},
)
assert resp.status_code == 400
assert resp.json["error"] == "Some settings are admin-only"
assert user_db.get_user_settings(user["id"]) == {}
def test_users_me_update_rejects_non_object_settings_payload(app, user_db):
user = user_db.create_user(username="alice")
client = _authed_client_for_user(app, user)
+144
View File
@@ -0,0 +1,144 @@
"""A spent search budget stops the search rather than starting the next attempt.
One release search fans out: a mirror loop inside `_fetch_search_table`, then a title
variant per grouped variant, then the whole set again without the language filter. Each
of those can reach the bypasser, and `except Exception` around the variant loops was
built to keep going past a parse failure. Applied to a spent budget that meant the
variants queued up behind each other long after anyone was still waiting - which is how
a challenge failure became a gateway timeout (issue #1276).
"""
import pytest
import shelfmark.release_sources.direct_download as dd
from shelfmark.core import search_deadline
@pytest.fixture(autouse=True)
def _no_ambient_deadline():
token = search_deadline._current.set(None)
yield
search_deadline._current.reset(token)
class _Selector:
current_base = "https://annas-archive.gl"
last_failure = None
def rewrite(self, url: str) -> str:
return url
def next_mirror_or_rotate_dns(self, *, fatal: bool = False, reason: str = ""):
return None, "exhausted"
def test_fetch_search_table_gives_up_when_the_budget_is_spent(monkeypatch):
"""Every mirror shares the protection, so another mirror is another wasted solve."""
fetches: list[str] = []
monkeypatch.setattr(dd.downloader, "html_get_page", lambda url, **_k: fetches.append(url) or "")
monkeypatch.setattr(dd.network, "get_available_aa_urls", lambda: ["https://annas-archive.gl"])
with search_deadline.search_deadline(60) as deadline:
deadline.event.set()
with pytest.raises(dd.SearchUnavailableError) as excinfo:
dd._fetch_search_table("https://annas-archive.gl/search?q=dune", _Selector())
assert fetches == [], "no fetch should have been attempted"
assert "ran out of time" in str(excinfo.value)
def test_fetch_search_table_runs_normally_within_budget(monkeypatch):
page = "<html><body><main><table><tbody></tbody></table></main></body></html>"
monkeypatch.setattr(dd.downloader, "html_get_page", lambda _url, **_k: page)
monkeypatch.setattr(dd.network, "get_available_aa_urls", lambda: ["https://annas-archive.gl"])
with search_deadline.search_deadline(60):
html, table = dd._fetch_search_table("https://annas-archive.gl/search?q=dune", _Selector())
assert table is not None
assert html == page
def _plan(titles):
"""A search plan with one grouped title variant per title."""
from shelfmark.core.search_plan import ReleaseSearchPlan, ReleaseSearchVariant
variants = [ReleaseSearchVariant(t, "Frank Herbert", ["en"]) for t in titles]
return ReleaseSearchPlan(
languages=["en"],
isbn_candidates=[],
author="Frank Herbert",
title_variants=variants,
grouped_title_variants=variants,
)
def _book():
from shelfmark.metadata_providers import BookMetadata
return BookMetadata(
provider="manual",
provider_id="x",
provider_display_name="Manual",
title="Dune",
search_title="Dune",
authors=["Frank Herbert"],
)
def test_title_variants_stop_once_the_budget_is_spent(monkeypatch):
"""`except Exception` keeps this loop going past a failure; a spent budget must not."""
queries: list[str] = []
def fake_search_books(query, _filters):
queries.append(query)
# The first variant is what spends the budget.
deadline = search_deadline.current()
if deadline is not None:
deadline.event.set()
return []
monkeypatch.setattr(dd, "search_books", fake_search_books)
monkeypatch.setattr(dd, "_ensure_direct_download_available", lambda: None)
source = dd.DirectDownloadSource()
with search_deadline.search_deadline(60):
releases = source.search(_book(), _plan(["Dune", "Duna", "Dünen"]))
assert releases == []
assert len(queries) == 1, f"only the first variant should have run, got {queries}"
def test_all_title_variants_run_within_budget(monkeypatch):
queries: list[str] = []
monkeypatch.setattr(dd, "search_books", lambda q, _f: queries.append(q) or [])
monkeypatch.setattr(dd, "_ensure_direct_download_available", lambda: None)
source = dd.DirectDownloadSource()
with search_deadline.search_deadline(60):
source.search(_book(), _plan(["Dune", "Duna"]))
# Two variants with a language filter, then both again without one.
assert len(queries) == 4
def test_language_filter_retry_is_skipped_on_a_spent_budget(monkeypatch):
"""The no-language sweep doubles the work; it must not start after the deadline."""
queries: list[str] = []
def fake_search_books(query, _filters):
queries.append(query)
if len(queries) == 2:
deadline = search_deadline.current()
if deadline is not None:
deadline.event.set()
return []
monkeypatch.setattr(dd, "search_books", fake_search_books)
monkeypatch.setattr(dd, "_ensure_direct_download_available", lambda: None)
source = dd.DirectDownloadSource()
with search_deadline.search_deadline(60):
source.search(_book(), _plan(["Dune", "Duna"]))
assert len(queries) == 2, f"the retry sweep should not have started, got {queries}"
+21
View File
@@ -0,0 +1,21 @@
"""Shared fixtures for the download tests."""
import pytest
@pytest.fixture(autouse=True)
def _clear_warmup_user_search_flag():
"""Start every download test with the warm-up's "a user searched" flag clear.
The flag is process-global by design - the warm-up is a one-shot per process, and
once a real search has run there is nothing left to pre-solve. That makes it leak
between tests: `/api/releases` sets it, so any API test sharing an xdist worker with
the warm-up tests would otherwise decide the warm-up for them. It surfaced as
test_search_warmup.py failing only on CI, where the workers divide up differently
than they happen to locally.
"""
from shelfmark.download import warmup
warmup._user_search_seen.clear()
yield
warmup._user_search_seen.clear()
@@ -0,0 +1,153 @@
"""The DDoS-Guard ?check=1 handshake must win over anything the clearance store holds.
DDoS-Guard reuses the same cookie names (__ddg1_/__ddg2_) for the probe it issues on the
302 and for what a solve leaves behind. When the store's copy was merged on top, the value
the server had just issued never left the process, the probe could never terminate, and
every request ended in the redirect-loop handoff and paid for a full browser solve - one
per query, which is exactly what was reported on issue #1276.
"""
import pytest
import requests
import shelfmark.bypass.cookie_store as cs
import shelfmark.download.http as http
URL = "https://annas-archive.gl/search?q=dune"
FRESH = {"__ddg1_": "FRESH1", "__ddg2_": "FRESH2"}
class _FakeResponse:
def __init__(self, status_code, *, headers=None, cookies=None, text="", url=URL):
self.status_code = status_code
self.headers = headers or {}
self.cookies = cookies or {}
self.text = text
self.url = url
@property
def is_redirect(self):
return self.status_code in (301, 302, 303, 307, 308) and "Location" in self.headers
def raise_for_status(self):
if self.status_code >= 400:
error = requests.exceptions.HTTPError(f"{self.status_code} Error")
error.response = self
raise error
class _DummySelector:
"""AA selector stub, so these tests never elect a real mirror."""
def __init__(self) -> None:
self.current_base = "https://annas-archive.gl"
self.attempts_this_dns = 0
self.last_failure: str | None = None
def rewrite(self, url: str) -> str:
return url
def next_mirror_or_rotate_dns(self, allow_dns=True, *, fatal=False, reason=""):
return None, "exhausted"
@pytest.fixture
def ddos_guard(monkeypatch):
"""An AA mirror that grants the page only once the client echoes what it issued."""
monkeypatch.setattr(cs, "_cf_cookies", {})
monkeypatch.setattr(cs, "_cf_user_agents", {})
# Keeps the store from importing the mirror registry (and its dependency graph)
# for a lookup these tests do not exercise.
monkeypatch.setattr(cs, "_get_full_cookie_domains", set)
monkeypatch.setattr(http.network, "should_rotate_dns_for_url", lambda _url: True)
monkeypatch.setattr(http.network, "is_aa_auto_mode", lambda: True)
monkeypatch.setattr(http, "get_proxies", lambda _url: {})
monkeypatch.setattr(http, "get_ssl_verify", lambda _url: True)
monkeypatch.setattr(http, "_is_cf_bypass_enabled", lambda: True)
monkeypatch.setattr(http.time, "sleep", lambda _seconds: None)
sent: list[dict[str, str]] = []
def fake_get(_url, **kwargs):
cookies = dict(kwargs.get("cookies") or {})
sent.append(cookies)
if all(cookies.get(name) == value for name, value in FRESH.items()):
return _FakeResponse(200, text="<html>the real search page</html>")
return _FakeResponse(
302, headers={"Location": "/search?q=dune&check=1"}, cookies=dict(FRESH)
)
monkeypatch.setattr(http.requests, "get", fake_get)
return sent
@pytest.fixture
def bypasser_calls(monkeypatch):
"""Record redirect-loop handoffs instead of starting a browser."""
calls: list[str] = []
def fake_get_bypassed_page(url, _selector=None, _cancel_flag=None):
calls.append(url)
return "<html>solved by the browser</html>"
monkeypatch.setattr(http, "get_bypassed_page", fake_get_bypassed_page)
monkeypatch.setattr(http, "_bypass_grace_seconds", lambda: 1.0)
return calls
def test_handshake_completes_with_an_empty_store(ddos_guard, bypasser_calls):
"""The baseline: echoing the issued cookies back clears the probe in two hops."""
html = http.html_get_page(URL, retry=1, selector=_DummySelector(), success_delay=0)
assert html == "<html>the real search page</html>"
assert ddos_guard == [{}, FRESH]
assert not bypasser_calls, "no browser solve should have been needed"
def test_stored_clearance_does_not_mask_the_issued_cookies(ddos_guard, bypasser_calls):
"""A solve leaves __ddg1_/__ddg2_ behind; the next probe must still be answerable.
This is the regression. With the store merged last, all six hops re-sent the stale
pair, the loop never terminated and the request fell through to a browser solve.
"""
cs._cf_cookies["annas-archive.gl"] = {
"__ddg1_": {"value": "STALE1", "expiry": None},
"__ddg2_": {"value": "STALE2", "expiry": None},
}
html = http.html_get_page(URL, retry=1, selector=_DummySelector(), success_delay=0)
assert html == "<html>the real search page</html>"
assert not bypasser_calls, "a stale cookie must not cost a browser solve"
# First hop presents what the store had; the second answers with what was just issued.
assert ddos_guard[0] == {"__ddg1_": "STALE1", "__ddg2_": "STALE2"}
assert ddos_guard[-1] == FRESH
def test_store_still_applies_when_the_server_issues_nothing(monkeypatch, bypasser_calls):
"""Handshake cookies winning must not stop stored clearance being presented."""
monkeypatch.setattr(cs, "_cf_cookies", {})
monkeypatch.setattr(cs, "_cf_user_agents", {})
# Keeps the store from importing the mirror registry (and its dependency graph)
# for a lookup these tests do not exercise.
monkeypatch.setattr(cs, "_get_full_cookie_domains", set)
monkeypatch.setattr(http.network, "should_rotate_dns_for_url", lambda _url: True)
monkeypatch.setattr(http, "get_proxies", lambda _url: {})
monkeypatch.setattr(http, "get_ssl_verify", lambda _url: True)
monkeypatch.setattr(http, "_is_cf_bypass_enabled", lambda: True)
monkeypatch.setattr(http.time, "sleep", lambda _seconds: None)
cs._cf_cookies["annas-archive.gl"] = {"__ddg1_": {"value": "CLEARANCE", "expiry": None}}
sent: list[dict[str, str]] = []
def fake_get(_url, **kwargs):
sent.append(dict(kwargs.get("cookies") or {}))
return _FakeResponse(200, text="<html>page</html>")
monkeypatch.setattr(http.requests, "get", fake_get)
assert (
http.html_get_page(URL, retry=1, selector=_DummySelector(), success_delay=0)
== "<html>page</html>"
)
assert sent == [{"__ddg1_": "CLEARANCE"}]
@@ -0,0 +1,123 @@
"""Tests for the per-host 429 backoff.
A 429 is the origin throttling this IP, which a challenge solve cannot clear - so the
host is sidelined for a growing window (2 -> 5 -> 10 -> 15 -> 30 min) that escalates
only when the host throttles us again *after* a full window has already elapsed. Mirror
selection skips a cooling-down host; the bypasser refuses to solve one. These guard that
the ladder climbs, caps, resets after a long clear gap, and stays per host.
"""
import shelfmark.download.network as network
MIRRORS = ["https://aa-one.test", "https://aa-two.test", "https://aa-three.test"]
LADDER = (120.0, 300.0, 600.0, 900.0, 1800.0)
def _fresh(monkeypatch, *, urls=None, start=1000.0):
"""Reset cooldown state and install a controllable monotonic clock.
Returns a ``clock`` list whose single element is the current fake time; mutate
``clock[0]`` to advance it.
"""
clock = [start]
monkeypatch.setattr(network, "_host_cooldowns", {})
monkeypatch.setattr(network.time, "monotonic", lambda: clock[0])
if urls is not None:
monkeypatch.setattr(network, "_initialized", True)
monkeypatch.setattr(network, "_aa_urls", list(urls))
monkeypatch.setattr(network, "_aa_base_url", urls[0])
monkeypatch.setattr(network, "_current_aa_url_index", 0)
monkeypatch.setattr(network, "_dead_aa_urls", set())
return clock
def test_first_429_arms_the_two_minute_step(monkeypatch):
_fresh(monkeypatch)
assert network.note_rate_limited("https://h.test/search?q=dune") == 120.0
# Keyed by host: any URL on the same host reads the same cooldown.
assert network.is_host_cooling_down("https://h.test/other") is True
assert network.host_cooldown_remaining("https://h.test") == 120.0
def test_cooldown_expires_after_the_window(monkeypatch):
clock = _fresh(monkeypatch)
network.note_rate_limited("https://h.test")
clock[0] += 121
assert network.is_host_cooling_down("https://h.test") is False
assert network.host_cooldown_remaining("https://h.test") == 0.0
def test_re_offense_after_expiry_climbs_the_ladder(monkeypatch):
clock = _fresh(monkeypatch)
for expected in LADDER:
assert network.note_rate_limited("https://h.test") == expected
# Wait the whole window out, then get throttled again -> next step.
clock[0] += expected + 1
# Top step holds: further re-offenses stay at 30 minutes, never beyond.
assert network.note_rate_limited("https://h.test") == 1800.0
def test_429_while_still_cooling_does_not_escalate(monkeypatch):
clock = _fresh(monkeypatch)
assert network.note_rate_limited("https://h.test") == 120.0
clock[0] += 30 # still inside the first window
# Same episode: keep the remaining wait, do not advance the ladder.
assert network.note_rate_limited("https://h.test") == 90.0
clock[0] += 91 # let the (unchanged) 2-min window lapse
# The next post-expiry 429 is step 2, proving the mid-window hit did not escalate.
assert network.note_rate_limited("https://h.test") == 300.0
def test_long_clear_gap_restarts_the_ladder(monkeypatch):
clock = _fresh(monkeypatch)
network.note_rate_limited("https://h.test") # step 1: 120s
clock[0] += 120 + 1801 # window lapses, then a gap longer than the reset threshold
assert network.note_rate_limited("https://h.test") == 120.0
def test_backoff_is_per_host(monkeypatch):
_fresh(monkeypatch)
network.note_rate_limited("https://a.test")
assert network.is_host_cooling_down("https://a.test") is True
assert network.is_host_cooling_down("https://b.test") is False
def test_available_mirrors_skip_cooling_hosts(monkeypatch):
_fresh(monkeypatch, urls=MIRRORS)
network.note_rate_limited(MIRRORS[1])
assert network.get_available_aa_urls() == [MIRRORS[0], MIRRORS[2]]
def test_all_mirrors_cooling_falls_back_to_full_list(monkeypatch):
_fresh(monkeypatch, urls=MIRRORS)
for mirror in MIRRORS:
network.note_rate_limited(mirror)
# Never leave selection with nowhere to point; the bypasser fail-fast handles this.
assert network.get_available_aa_urls() == MIRRORS
def test_clear_host_cooldowns_resets_everything(monkeypatch):
_fresh(monkeypatch)
network.note_rate_limited("https://h.test")
network.clear_host_cooldowns()
assert network.is_host_cooling_down("https://h.test") is False
def test_urls_without_a_host_are_ignored(monkeypatch):
_fresh(monkeypatch)
assert network.note_rate_limited("not-a-url") == 0.0
assert network.is_host_cooling_down("not-a-url") is False
@@ -0,0 +1,62 @@
"""Pack fields (multi_book / book_plan) survive queueing and restart-safe retry."""
from shelfmark.core.models import DownloadTask
from shelfmark.download import orchestrator
PLAN = [
{"title": "Leviathan Wakes", "series_position": 1.0, "year": 2011, "files": ["a.m4b"]},
{"title": "Caliban's War", "series_position": 2.0, "year": 2012, "files": ["b.m4b"]},
]
def test_task_defaults_to_single_book():
task = DownloadTask(task_id="t", source="prowlarr", title="T")
assert task.multi_book is False
assert task.book_plan is None
def test_retry_payload_round_trips_pack_fields():
task = DownloadTask(task_id="t", source="prowlarr", title="T", multi_book=True, book_plan=PLAN)
payload = orchestrator.serialize_task_for_retry(task)
restored = orchestrator._restore_task_from_retry_payload(payload)
assert restored is not None
assert restored.multi_book is True
assert restored.book_plan == PLAN
def test_retry_payload_drops_malformed_plan():
payload = orchestrator.serialize_task_for_retry(
DownloadTask(task_id="t", source="prowlarr", title="T")
)
payload["book_plan"] = "not a list"
restored = orchestrator._restore_task_from_retry_payload(payload)
assert restored is not None
assert restored.book_plan is None
def test_queue_release_reads_pack_fields(monkeypatch):
captured: dict[str, DownloadTask] = {}
def fake_add(task: DownloadTask) -> bool:
captured["task"] = task
return True
monkeypatch.setattr(orchestrator.config, "get", lambda _key, default=None, **_kw: default)
monkeypatch.setattr(orchestrator, "_source_unavailable_message", lambda _source: None)
monkeypatch.setattr(orchestrator.book_queue, "add", fake_add)
monkeypatch.setattr(orchestrator, "ws_manager", None)
ok, error = orchestrator.queue_release(
{
"source": "direct_download",
"source_id": "abc",
"title": "The Expanse",
"content_type": "audiobook",
"multi_book": True,
"book_plan": PLAN,
},
0,
)
assert ok, error
assert captured["task"].multi_book is True
assert captured["task"].book_plan == PLAN
+309
View File
@@ -0,0 +1,309 @@
"""Tests for multi-book pack planning (shelfmark.download.postprocess.packs)."""
from pathlib import Path
import pytest
from shelfmark.download.postprocess.packs import (
PackBook,
PackFile,
group_files_into_books,
match_plan_to_files,
parse_pack_book_name,
plan_pack,
)
AUDIO = {"m4b", "mp3"}
class TestParsePackBookName:
@pytest.mark.parametrize(
("name", "expected"),
[
("Book 3 - Howling Dark", ("Howling Dark", 3.0, None)),
("Book 03: Howling Dark", ("Howling Dark", 3.0, None)),
("03 - Empire of Silence", ("Empire of Silence", 3.0, None)),
("2.5 - Interlude", ("Interlude", 2.5, None)),
("[03] Empire of Silence", ("Empire of Silence", 3.0, None)),
("#3 Empire of Silence", ("Empire of Silence", 3.0, None)),
("3. Empire of Silence", ("Empire of Silence", 3.0, None)),
("Empire of Silence", ("Empire of Silence", None, None)),
("Empire of Silence (2018)", ("Empire of Silence", None, 2018)),
],
)
def test_strips_series_markers(self, name, expected):
assert parse_pack_book_name(name, series_name=None) == expected
def test_strips_leading_series_name_and_trailing_year(self):
assert parse_pack_book_name(
"The Expanse 1.0 - Leviathan Wakes (2011)", series_name="The Expanse"
) == ("Leviathan Wakes", 1.0, 2011)
def test_series_name_match_is_case_insensitive(self):
assert parse_pack_book_name(
"the expanse 2.5 - Gods of Risk", series_name="The Expanse"
) == (
"Gods of Risk",
2.5,
None,
)
@pytest.mark.parametrize(
"name",
[
"The Expanse 0.2 - An Expanse Novella - The Churn (2014)",
"The Expanse 0.2 - The Expanse Novella - The Churn (2014)",
"The Expanse 0.2 - An Expanse Short Story - The Churn (2014)",
],
)
def test_strips_series_novella_label(self, name):
assert parse_pack_book_name(name, series_name="The Expanse") == ("The Churn", 0.2, 2014)
@pytest.mark.parametrize(
("name", "expected"),
[
("Uncrowned Cradle, Book 7", ("Uncrowned", 7.0, None)),
("Reaper Cradle, Volume 10", ("Reaper", 10.0, None)),
("Soulsmith Cradle, Book 2", ("Soulsmith", 2.0, None)),
("Wintersteel - Cradle Book 8", ("Wintersteel", 8.0, None)),
("Wintersteel (Cradle, Book 8)", ("Wintersteel", 8.0, None)),
# A trailing bare number is a chapter/part, never a series position.
("Unsouled - 02", ("Unsouled - 02", None, None)),
],
)
def test_trailing_series_marker(self, name, expected):
assert parse_pack_book_name(name, series_name="Cradle") == expected
def test_trailing_series_name_only_stripped_with_a_marker(self):
# "Stories from Cradle" is the title; nothing marks a position, so keep it.
assert parse_pack_book_name("Threshold: Stories from Cradle", series_name="Cradle") == (
"Threshold: Stories from Cradle",
None,
None,
)
@pytest.mark.parametrize(
("name", "expected"),
[
# AudiobookBay renders "<folder> <file>" as one flat string.
(
"Will Wight - Unsouled Cradle, Book 1 Will Wight - Unsouled Cradle, Book 1",
("Unsouled", 1.0, None),
),
(
"Will Wight - Skysworn Cradle, Book 4 Skysworn Cradle, Book 4",
("Skysworn", 4.0, None),
),
("Will Wight - Bloodline Cradle, Book 9", ("Bloodline", 9.0, None)),
],
)
def test_strips_author_prefix_and_glued_folder_name(self, name, expected):
assert (
parse_pack_book_name(name, series_name="Cradle", author_name="Will Wight") == expected
)
@pytest.mark.parametrize(
("name", "expected"),
[
("Gods of Risk 2.5 - Gods of Risk", ("Gods of Risk", 2.5, None)),
("Cibola Burn 4 - Cibola Burn (2014)", ("Cibola Burn", 4.0, 2014)),
("cibola burn 4 - Cibola Burn", ("Cibola Burn", 4.0, None)),
# Different text on each side is a real "Series N - Title" name, not a repeat.
("Sun Eater 2 - Howling Dark", ("Sun Eater 2 - Howling Dark", None, None)),
],
)
def test_collapses_title_repeated_around_the_position(self, name, expected):
assert parse_pack_book_name(name, series_name=None) == expected
def test_bare_numeric_title_is_left_alone(self):
assert parse_pack_book_name("1984", series_name=None) == ("1984", None, None)
def test_marker_that_would_leave_nothing_is_left_alone(self):
assert parse_pack_book_name("Book 3", series_name=None) == ("Book 3", None, None)
class TestPlanPack:
def test_nested_subfolders_become_separate_books(self):
files = [
PackFile("Sun Eater/Book 1 - Empire of Silence/Empire of Silence.m4b", 10),
PackFile("Sun Eater/Book 2 - Howling Dark/Howling Dark.m4b", 20),
PackFile("Sun Eater/cover.jpg", 1),
]
plan = plan_pack(files, supported_extensions=AUDIO, series_name=None)
assert plan.is_pack
assert [b.title for b in plan.books] == ["Empire of Silence", "Howling Dark"]
assert [b.series_position for b in plan.books] == [1.0, 2.0]
assert plan.books[0].files == ["Sun Eater/Book 1 - Empire of Silence/Empire of Silence.m4b"]
assert plan.ignored == ["Sun Eater/cover.jpg"]
def test_flat_pack_becomes_one_book_per_file_and_ignores_sidecars(self):
files = [
PackFile("The Expanse 1.0 - Leviathan Wakes (2011).m4b", 100),
PackFile("The Expanse 1.0 - Leviathan Wakes (2011).txt", 1),
PackFile("The Expanse 2.0 - Caliban's War (2012).m4b", 100),
]
plan = plan_pack(files, supported_extensions=AUDIO, series_name="The Expanse")
assert plan.is_pack
assert [(b.title, b.series_position, b.year) for b in plan.books] == [
("Leviathan Wakes", 1.0, 2011),
("Caliban's War", 2.0, 2012),
]
assert plan.ignored == ["The Expanse 1.0 - Leviathan Wakes (2011).txt"]
def test_flat_chaptered_mp3_tracks_are_one_book_not_a_pack(self):
# A single audiobook whose chapters are named "01 - <chapter>.mp3" must not be
# split into one "book" per track just because each name carries a number.
files = [
PackFile("The Hobbit/01 - An Unexpected Party.mp3", 10),
PackFile("The Hobbit/02 - Roast Mutton.mp3", 10),
PackFile("The Hobbit/03 - A Short Rest.mp3", 10),
]
plan = plan_pack(files, supported_extensions=AUDIO, series_name=None)
assert not plan.is_pack
assert len(plan.books) == 1
assert plan.books[0].files == [
"The Hobbit/01 - An Unexpected Party.mp3",
"The Hobbit/02 - Roast Mutton.mp3",
"The Hobbit/03 - A Short Rest.mp3",
]
def test_flat_repeated_title_mp3_tracks_are_one_book(self):
# Same title on every track ("01 - The Hobbit.mp3") is a chaptered book too.
files = [
PackFile("01 - The Hobbit.mp3", 10),
PackFile("02 - The Hobbit.mp3", 10),
]
plan = plan_pack(files, supported_extensions=AUDIO, series_name=None)
assert not plan.is_pack
assert len(plan.books) == 1
def test_audiobookbay_flat_list_with_trailing_markers_is_a_pack(self):
# ABB's file table has no folder separators: "<folder> <file> <size>".
files = [
PackFile(
"Will Wight - Unsouled Cradle, Book 1 Will Wight - Unsouled Cradle, Book 1.sfv", 1
),
PackFile(
"Will Wight - Unsouled Cradle, Book 1 Will Wight - Unsouled Cradle, Book 1.m4a", 9
),
PackFile("Will Wight - Skysworn Cradle, Book 4 Skysworn Cradle, Book 4.m4b", 9),
PackFile("Uncrowned Cradle, Book 7.m4b", 9),
PackFile(
"Will Wight - Reaper Cradle, Volume 10 Will Wight - Reaper Cradle, Volume 10.m4b", 9
),
]
plan = plan_pack(
files,
supported_extensions={"m4a", "m4b"},
series_name="Cradle",
author_name="Will Wight",
)
assert plan.is_pack
assert [(b.title, b.series_position) for b in plan.books] == [
("Unsouled", 1.0),
("Skysworn", 4.0),
("Uncrowned", 7.0),
("Reaper", 10.0),
]
def test_deeper_nesting_collapses_onto_book_folder(self):
files = [
PackFile("Book 1/CD1/01.mp3"),
PackFile("Book 1/CD2/01.mp3"),
PackFile("Book 2/01.mp3"),
]
plan = plan_pack(files, supported_extensions=AUDIO, series_name=None)
assert [b.files for b in plan.books] == [
["Book 1/CD1/01.mp3", "Book 1/CD2/01.mp3"],
["Book 2/01.mp3"],
]
def test_single_wrapping_folder_is_not_a_book_boundary(self):
# A torrent named "Series" containing one multi-part book is a single book.
files = [PackFile("Series/Book 1/01.mp3"), PackFile("Series/Book 1/02.mp3")]
plan = plan_pack(files, supported_extensions=AUDIO, series_name=None)
assert not plan.is_pack
assert len(plan.books) == 1
def test_root_files_and_subfolders_coexist(self):
files = [PackFile("Novella.m4b"), PackFile("Book 1/a.m4b"), PackFile("Book 1/b.m4b")]
plan = plan_pack(files, supported_extensions=AUDIO, series_name=None)
assert [b.files for b in plan.books] == [["Novella.m4b"], ["Book 1/a.m4b", "Book 1/b.m4b"]]
def test_single_file_is_not_a_pack(self):
plan = plan_pack([PackFile("Book.m4b")], supported_extensions=AUDIO, series_name=None)
assert not plan.is_pack
assert plan.books[0].title == "Book"
def test_empty_input(self):
plan = plan_pack([], supported_extensions=AUDIO, series_name=None)
assert plan.books == []
assert not plan.is_pack
class TestGroupFilesIntoBooks:
def test_groups_on_disk_files_by_top_level_folder(self, tmp_path: Path):
a = tmp_path / "Book 1 - A" / "a.m4b"
b = tmp_path / "Book 2 - B" / "b.m4b"
for f in (a, b):
f.parent.mkdir(parents=True)
f.write_bytes(b"x")
groups = group_files_into_books([a, b], series_name=None)
assert [(g.title, g.series_position, g.files) for g in groups] == [
("A", 1.0, [a]),
("B", 2.0, [b]),
]
class TestMatchPlanToFiles:
def test_matches_by_relative_path_then_basename(self, tmp_path: Path):
root = tmp_path / "staging" / "Sun Eater"
a = root / "Book 1 - A" / "a.m4b"
b = root / "Book 2 - B" / "b.m4b"
for f in (a, b):
f.parent.mkdir(parents=True)
f.write_bytes(b"x")
plan = [
PackBook(title="Alpha", series_position=1.0, year=2001, files=["Book 1 - A/a.m4b"]),
PackBook(title="Beta", series_position=2.0, year=None, files=["b.m4b"]),
]
groups = match_plan_to_files(plan, [a, b])
assert [(g.title, g.series_position, g.year, g.files) for g in groups] == [
("Alpha", 1.0, 2001, [a]),
("Beta", 2.0, None, [b]),
]
def test_matches_glued_plan_path_by_basename_suffix(self, tmp_path: Path):
# The plan came from ABB's "<folder> <file>" strings; on disk the file sits in a folder.
root = tmp_path / "Cradle - Will Wight Books 1-10"
a = root / "Will Wight - Skysworn Cradle, Book 4" / "Skysworn Cradle, Book 4.m4b"
b = root / "Uncrowned Cradle, Book 7.m4b"
for f in (a, b):
f.parent.mkdir(parents=True, exist_ok=True)
f.write_bytes(b"x")
plan = [
PackBook(
title="Skysworn",
series_position=4.0,
year=None,
files=["Will Wight - Skysworn Cradle, Book 4 Skysworn Cradle, Book 4.m4b"],
),
PackBook(
title="Uncrowned",
series_position=7.0,
year=None,
files=["Uncrowned Cradle, Book 7.m4b"],
),
]
groups = match_plan_to_files(plan, [a, b])
assert [(g.title, g.files) for g in groups] == [("Skysworn", [a]), ("Uncrowned", [b])]
def test_unmatched_files_fall_back_to_heuristic_groups(self, tmp_path: Path):
a = tmp_path / "Book 1 - A" / "a.m4b"
c = tmp_path / "Book 3 - C" / "c.m4b"
for f in (a, c):
f.parent.mkdir(parents=True)
f.write_bytes(b"x")
plan = [PackBook(title="Alpha", series_position=1.0, year=None, files=["Book 1 - A/a.m4b"])]
groups = match_plan_to_files(plan, [a, c])
assert [(g.title, g.files) for g in groups] == [("Alpha", [a]), ("C", [c])]
@@ -0,0 +1,65 @@
"""The warm-up must not queue its throwaway solve in front of the user's first search.
Issue #1276: the warm-up fires 15s after boot, and the bundle showed a user clicking a
book 13 seconds in. Three seconds later the warm-up started anyway, and because the
bypasser serializes on one browser, the user's title+author search sat behind a solve for
"The Great Gatsby" from 13:41:26 to 13:42:26 - a full minute of a 2m27s wait, for a query
nobody asked for. Once the user has beaten the warm-up to it there is nothing left to
pre-solve.
"""
import shelfmark.download.warmup as warmup
# The flag is cleared around every test by an autouse fixture in conftest.py - it is
# process-global, and /api/releases sets it, so tests that read it cannot rely on
# whatever else shared their xdist worker.
def test_warmup_runs_when_nobody_has_searched(monkeypatch):
monkeypatch.setattr(
"shelfmark.core.mirrors.has_aa_mirror_configuration", lambda: True, raising=False
)
searched: list[str] = []
monkeypatch.setattr(
"shelfmark.release_sources.direct_download.search_books",
lambda query, _filters: searched.append(query) or ["a result"],
)
assert warmup.run_warmup() is True
assert searched == [warmup.warmup_query()]
def test_warmup_stands_down_once_a_real_search_has_started(monkeypatch):
searched: list[str] = []
monkeypatch.setattr(
"shelfmark.release_sources.direct_download.search_books",
lambda query, _filters: searched.append(query) or [],
)
warmup.note_user_search()
assert warmup.run_warmup() is False
assert searched == [], "the warm-up must not compete for the bypasser"
def test_the_check_happens_at_fire_time_not_schedule_time(monkeypatch):
"""The start-up delay is exactly what this races with, so a search that lands during
the wait has to count - checking only in start() would miss every real case."""
monkeypatch.setattr(
"shelfmark.core.mirrors.has_aa_mirror_configuration", lambda: True, raising=False
)
searched: list[str] = []
monkeypatch.setattr(
"shelfmark.release_sources.direct_download.search_books",
lambda query, _filters: searched.append(query) or [],
)
# Scheduling succeeds: at this point nothing has searched.
monkeypatch.setattr(warmup, "_setting", lambda _key, _default: True)
assert warmup.is_enabled() is True
# The user clicks while the timer is still pending.
warmup.note_user_search()
assert warmup.run_warmup() is False
assert searched == []
+78
View File
@@ -0,0 +1,78 @@
"""Tests for named Newznab indexer settings."""
from shelfmark.core.settings_registry import TableField
from shelfmark.release_sources.newznab.settings import (
_test_newznab_connection,
newznab_config_settings,
)
def test_settings_include_named_indexer_table():
field = next(
field
for field in newznab_config_settings()
if getattr(field, "key", None) == "NEWZNAB_INDEXERS"
)
assert isinstance(field, TableField)
assert [column["key"] for column in field.columns] == ["name", "url", "api_key"]
assert field.columns[2]["type"] == "password"
def test_connection_action_tests_every_named_indexer(monkeypatch):
import shelfmark.release_sources.newznab.api as api_module
tested: list[tuple[str, str]] = []
class FakeClient:
def __init__(self, url, api_key):
tested.append((url, api_key))
def test_connection(self):
return True, "Connected"
monkeypatch.setattr(api_module, "NewznabClient", FakeClient)
result = _test_newznab_connection(
{
"NEWZNAB_INDEXERS": [
{"name": "NZBGeek", "url": "https://geek.example", "api_key": "one"},
{"name": "DrunkenSlug", "url": "https://slug.example", "api_key": "two"},
]
}
)
assert result == {
"success": True,
"message": "Connected to all 2 indexers",
"details": ["NZBGeek: Connected", "DrunkenSlug: Connected"],
}
assert tested == [("https://geek.example", "one"), ("https://slug.example", "two")]
def test_connection_action_reports_each_failure(monkeypatch):
import shelfmark.release_sources.newznab.api as api_module
class FakeClient:
def __init__(self, url, _api_key):
self.url = url
def test_connection(self):
if "down" in self.url:
return False, "Could not connect"
return True, "Connected"
monkeypatch.setattr(api_module, "NewznabClient", FakeClient)
result = _test_newznab_connection(
{
"NEWZNAB_INDEXERS": [
{"name": "Working", "url": "https://working.example"},
{"name": "Unavailable", "url": "https://down.example"},
]
}
)
assert result["success"] is False
assert result["message"] == "One or more Newznab indexers failed"
assert result["details"] == ["Working: Connected", "Unavailable: Could not connect"]
+132
View File
@@ -1,5 +1,6 @@
"""Unit tests for the Newznab release source."""
from dataclasses import replace
from unittest.mock import MagicMock
import pytest
@@ -11,6 +12,7 @@ from shelfmark.release_sources.newznab.source import (
NewznabSource,
_newznab_result_to_release,
_parse_category_ids,
_parse_indexer_rows,
)
# ── fixtures / helpers ─────────────────────────────────────────────────────────
@@ -221,6 +223,19 @@ class TestIsAvailable:
monkeypatch.setattr(mod.config, "get", self._config(NEWZNAB_URL=""))
assert NewznabSource().is_available() is False
def test_available_with_named_indexer_and_no_legacy_url(self, monkeypatch):
import shelfmark.release_sources.newznab.source as mod
monkeypatch.setattr(
mod.config,
"get",
self._config(
NEWZNAB_URL="",
NEWZNAB_INDEXERS=[{"name": "NZBGeek", "url": "https://geek.example"}],
),
)
assert NewznabSource().is_available() is True
# ── category parsing ───────────────────────────────────────────────────────────
@@ -247,6 +262,42 @@ class TestParseCategoryIds:
assert _parse_category_ids(" ") == []
class TestParseIndexerRows:
def test_parses_named_connections(self):
assert _parse_indexer_rows(
[
{
"name": "NZBGeek",
"url": "https://api.nzbgeek.info/",
"api_key": "geek-key",
},
{
"name": "DrunkenSlug",
"url": "drunkenslug.com",
"api_key": "slug-key",
},
]
) == [
("NZBGeek", "https://api.nzbgeek.info", "geek-key"),
("DrunkenSlug", "http://drunkenslug.com", "slug-key"),
]
def test_uses_hostname_when_name_is_blank(self):
assert _parse_indexer_rows([{"url": "https://indexer.example.com"}]) == [
("indexer.example.com", "https://indexer.example.com", "")
]
def test_ignores_invalid_and_duplicate_connections(self):
assert _parse_indexer_rows(
[
None,
{"name": "Incomplete", "url": ""},
{"name": "First", "url": "https://indexer.example.com", "api_key": "key"},
{"name": "Duplicate", "url": "https://indexer.example.com", "api_key": "key"},
]
) == [("First", "https://indexer.example.com", "key")]
# ── NewznabSource.search ───────────────────────────────────────────────────────
@@ -449,6 +500,87 @@ class TestSearch:
call_kwargs = client.search.call_args
assert call_kwargs[1]["query"] == "9780441013593"
def test_searches_all_named_indexers_and_labels_plain_feed_results(self, monkeypatch):
import shelfmark.release_sources.newznab.source as mod
rows = [
{"name": "NZBGeek", "url": "https://geek.example", "api_key": "one"},
{"name": "DrunkenSlug", "url": "https://slug.example", "api_key": "two"},
]
monkeypatch.setattr(
mod.config,
"get",
self._fake_config(NEWZNAB_INDEXERS=rows),
)
clients = {}
def client_factory(url, api_key):
client = MagicMock()
client.search.return_value = [
_make_result(
guid="shared-guid",
downloadUrl=f"{url}/download?apikey={api_key}",
indexer=None,
)
]
clients[url] = client
return client
monkeypatch.setattr(mod, "NewznabClient", client_factory)
results = NewznabSource().search(_make_book(), _make_plan(_make_book()))
assert {release.indexer for release in results} == {"NZBGeek", "DrunkenSlug"}
assert len({release.source_id for release in results}) == 2
assert all(release.source_id.startswith("newznab:") for release in results)
assert set(clients) == {"https://geek.example", "https://slug.example"}
def test_indexer_filter_is_applied_to_named_results(self, monkeypatch):
import shelfmark.release_sources.newznab.source as mod
rows = [
{"name": "NZBGeek", "url": "https://geek.example"},
{"name": "DrunkenSlug", "url": "https://slug.example"},
]
monkeypatch.setattr(mod.config, "get", self._fake_config(NEWZNAB_INDEXERS=rows))
def client_factory(url, _api_key):
client = MagicMock()
client.search.return_value = [_make_result(guid=f"{url}/guid", indexer=None)]
return client
monkeypatch.setattr(mod, "NewznabClient", client_factory)
book = _make_book()
plan = replace(_make_plan(book), indexers=["DrunkenSlug"])
results = NewznabSource().search(book, plan)
assert [release.indexer for release in results] == ["DrunkenSlug"]
def test_one_named_indexer_failure_does_not_hide_other_results(self, monkeypatch):
import shelfmark.release_sources.newznab.source as mod
rows = [
{"name": "Unavailable", "url": "https://down.example"},
{"name": "Working", "url": "https://working.example"},
]
monkeypatch.setattr(mod.config, "get", self._fake_config(NEWZNAB_INDEXERS=rows))
def client_factory(url, _api_key):
client = MagicMock()
if "down" in url:
client.search.side_effect = RuntimeError("offline")
else:
client.search.return_value = [_make_result(indexer=None)]
return client
monkeypatch.setattr(mod, "NewznabClient", client_factory)
results = NewznabSource().search(_make_book(), _make_plan(_make_book()))
assert [release.indexer for release in results] == ["Working"]
def test_exception_in_client_returns_empty(self, monkeypatch):
client = MagicMock()
client.search.side_effect = RuntimeError("boom")
+190 -4
View File
@@ -919,8 +919,8 @@ class TestQBittorrentClientAddDownload:
{"category": "audiobooks"},
]
def test_add_fails_when_metadata_never_resolves(self, monkeypatch):
"""Fail rather than return a transitional hash after the metadata timeout."""
def test_add_keeps_torrent_when_metadata_never_resolves(self, monkeypatch):
"""Return the info hash rather than abandon a magnet whose metadata is slow."""
config_values = {
"QBITTORRENT_URL": "http://localhost:8080",
"QBITTORRENT_USERNAME": "admin",
@@ -957,8 +957,149 @@ class TestQBittorrentClientAddDownload:
client = qb_module.QBittorrentClient()
magnet = f"magnet:?xt=urn:btih:{v1_hash}&dn=test"
with pytest.raises(RuntimeError, match="metadata resolution was not confirmed"):
client.add_download(magnet, "Test Download")
assert client.add_download(magnet, "Test Download") == v1_hash
def test_get_status_resolves_hash_after_metadata_switch(self, monkeypatch):
"""Track a torrent by its v1 hash after qBittorrent re-keys it to v2."""
config_values = {
"QBITTORRENT_URL": "http://localhost:8080",
"QBITTORRENT_USERNAME": "admin",
"QBITTORRENT_PASSWORD": "password",
"QBITTORRENT_CATEGORY": "books",
}
monkeypatch.setattr(
"shelfmark.download.clients.qbittorrent.config.get",
lambda key, default="": config_values.get(key, default),
)
v1_hash = "edf46c7f938a3c678081734d7bff8b9c652ba5e5"
v2_hash = "0bed5f40753b342cb143e83c2b21924cc8474731"
full_v2_hash = "0bed5f40753b342cb143e83c2b21924cc847473134e44d1bd300bdc58c13010f"
resolved_torrent = MockTorrent(
hash_val=v2_hash,
state="downloading",
infohash_v1=v1_hash,
infohash_v2=full_v2_hash,
)
mock_client_instance = MagicMock()
mock_client_instance._session.get.side_effect = [
create_mock_session_response([]),
create_mock_session_response([resolved_torrent]),
]
mock_client_class = MagicMock(return_value=mock_client_instance)
with patch.dict("sys.modules", {"qbittorrentapi": MagicMock(Client=mock_client_class)}):
import importlib
import shelfmark.download.clients.qbittorrent as qb_module
importlib.reload(qb_module)
client = qb_module.QBittorrentClient()
status = client.get_status(v1_hash)
assert status.state.value == "downloading"
def test_status_polls_reuse_resolved_hash_after_metadata_switch(self, monkeypatch):
"""Scan for the re-keyed hash once, then poll it directly."""
config_values = {
"QBITTORRENT_URL": "http://localhost:8080",
"QBITTORRENT_USERNAME": "admin",
"QBITTORRENT_PASSWORD": "password",
"QBITTORRENT_CATEGORY": "books",
}
monkeypatch.setattr(
"shelfmark.download.clients.qbittorrent.config.get",
lambda key, default="": config_values.get(key, default),
)
v1_hash = "edf46c7f938a3c678081734d7bff8b9c652ba5e5"
v2_hash = "0bed5f40753b342cb143e83c2b21924cc8474731"
full_v2_hash = "0bed5f40753b342cb143e83c2b21924cc847473134e44d1bd300bdc58c13010f"
resolved_torrent = MockTorrent(
hash_val=v2_hash,
state="downloading",
infohash_v1=v1_hash,
infohash_v2=full_v2_hash,
)
mock_client_instance = MagicMock()
mock_client_instance._session.get.side_effect = [
create_mock_session_response([]),
create_mock_session_response([resolved_torrent]),
create_mock_session_response([resolved_torrent]),
]
mock_client_class = MagicMock(return_value=mock_client_instance)
with patch.dict("sys.modules", {"qbittorrentapi": MagicMock(Client=mock_client_class)}):
import importlib
import shelfmark.download.clients.qbittorrent as qb_module
importlib.reload(qb_module)
client = qb_module.QBittorrentClient()
assert client.get_status(v1_hash).state_value == "downloading"
assert client.get_status(v1_hash).state_value == "downloading"
# The second poll goes straight to the hash the first one resolved,
# rather than listing every torrent again.
assert [
call.kwargs["params"] for call in mock_client_instance._session.get.call_args_list
] == [
{"hashes": v1_hash},
{"category": "books"},
{"hashes": v2_hash},
]
def test_remove_forgets_resolved_hash(self, monkeypatch):
"""Drop the remembered hash on removal so a re-add is resolved afresh."""
config_values = {
"QBITTORRENT_URL": "http://localhost:8080",
"QBITTORRENT_USERNAME": "admin",
"QBITTORRENT_PASSWORD": "password",
"QBITTORRENT_CATEGORY": "books",
}
monkeypatch.setattr(
"shelfmark.download.clients.qbittorrent.config.get",
lambda key, default="": config_values.get(key, default),
)
v1_hash = "edf46c7f938a3c678081734d7bff8b9c652ba5e5"
v2_hash = "0bed5f40753b342cb143e83c2b21924cc8474731"
full_v2_hash = "0bed5f40753b342cb143e83c2b21924cc847473134e44d1bd300bdc58c13010f"
resolved_torrent = MockTorrent(
hash_val=v2_hash,
state="downloading",
infohash_v1=v1_hash,
infohash_v2=full_v2_hash,
)
mock_client_instance = MagicMock()
mock_client_instance._session.get.side_effect = [
create_mock_session_response([]),
create_mock_session_response([resolved_torrent]),
create_mock_session_response([resolved_torrent]),
]
mock_client_class = MagicMock(return_value=mock_client_instance)
with patch.dict("sys.modules", {"qbittorrentapi": MagicMock(Client=mock_client_class)}):
import importlib
import shelfmark.download.clients.qbittorrent as qb_module
importlib.reload(qb_module)
client = qb_module.QBittorrentClient()
client.get_status(v1_hash)
assert client.remove(v1_hash) is True
# The delete addressed the current primary hash, and the entry is gone.
assert (
mock_client_instance.torrents_delete.call_args.kwargs["torrent_hashes"] == v2_hash
)
assert client._primary_hashes == {}
def test_add_download_uses_expected_hash_without_fetch(self, monkeypatch):
"""Skip proxy fetch when expected hash is provided for URL torrents."""
@@ -1744,6 +1885,51 @@ class TestQBittorrentClientFindExisting:
{"hashes": v2_hash},
]
def test_find_existing_keeps_torrent_whose_metadata_is_pending(self, monkeypatch):
"""Join a magnet still fetching metadata instead of adding a duplicate."""
config_values = {
"QBITTORRENT_URL": "http://localhost:8080",
"QBITTORRENT_USERNAME": "admin",
"QBITTORRENT_PASSWORD": "password",
"QBITTORRENT_CATEGORY": "books",
}
monkeypatch.setattr(
"shelfmark.download.clients.qbittorrent.config.get",
lambda key, default="": config_values.get(key, default),
)
monkeypatch.setattr(
"shelfmark.download.clients.qbittorrent.time.sleep", lambda _seconds: None
)
v1_hash = "edf46c7f938a3c678081734d7bff8b9c652ba5e5"
metadata_torrent = MockTorrent(
hash_val=v1_hash,
state="metaDL",
infohash_v1=v1_hash,
)
mock_client_instance = MagicMock()
mock_client_instance._session.get.return_value = create_mock_session_response(
[metadata_torrent]
)
mock_client_class = MagicMock(return_value=mock_client_instance)
with patch.dict("sys.modules", {"qbittorrentapi": MagicMock(Client=mock_client_class)}):
import importlib
import shelfmark.download.clients.qbittorrent as qb_module
importlib.reload(qb_module)
client = qb_module.QBittorrentClient()
magnet = f"magnet:?xt=urn:btih:{v1_hash}&dn=test"
result = client.find_existing(magnet)
assert result is not None
download_id, status = result
assert download_id == v1_hash
assert status.state_value == "downloading"
assert status.message == "Fetching metadata"
def test_find_existing_not_found(self, monkeypatch):
"""Test finding non-existent torrent."""
config_values = {
+34
View File
@@ -654,6 +654,40 @@ class TestSABnzbdClientAddDownload:
assert "https://attacker.example/download.nzb" not in called_urls
assert called_urls == ["http://localhost:8080/api"]
def test_add_download_prefetches_named_newznab_indexer_url(self, monkeypatch):
"""Named Newznab origins should receive the same backend prefetch as legacy URLs."""
config_values = {
"SABNZBD_URL": "http://localhost:8080",
"SABNZBD_API_KEY": "abc123",
"SABNZBD_CATEGORY": "books",
"NEWZNAB_INDEXERS": [
{"name": "NZBGeek", "url": "https://geek.example", "api_key": "secret"}
],
}
monkeypatch.setattr(
"shelfmark.download.clients.sabnzbd.config.get",
lambda key, default="": config_values.get(key, default),
)
from shelfmark.download.clients.sabnzbd import SABnzbdClient
with (
patch.object(SABnzbdClient, "_fetch_nzb_content", return_value=b"nzbdata") as fetch,
patch.object(
SABnzbdClient,
"_api_post_file",
return_value={"status": True, "nzo_ids": ["SABnzbd_nzo_named"]},
),
):
client = SABnzbdClient()
result = client.add_download(
"https://geek.example/download.nzb?apikey=secret",
"Test Book",
)
assert result == "SABnzbd_nzo_named"
fetch.assert_called_once_with("https://geek.example/download.nzb?apikey=secret")
class TestSABnzbdClientRemove:
"""Tests for SABnzbdClient.remove()."""
+137
View File
@@ -17,6 +17,7 @@ from shelfmark.release_sources.prowlarr.source import (
_build_indexer_priority,
_collapse_duplicate_indexer_results,
_detect_content_type_from_categories,
_drop_unknown_indexer_ids,
_extract_format,
_extract_mam_language,
_fetch_indexer_seed_settings,
@@ -1398,3 +1399,139 @@ class TestSearchBudgetScalesWithIndexerTimeout:
def test_budget_stays_under_the_gunicorn_worker_timeout(self):
assert _search_budget_seconds(300) == 240.0
class TestSplitMamFormats:
"""MAM's structured "[LANG / FORMATS]" bracket, split into known vs unknown tokens."""
def test_recognized_only(self):
from shelfmark.release_sources.prowlarr.source import _split_mam_formats
assert _split_mam_formats("Title by Author [ENG / EPUB MOBI]") == (["epub", "mobi"], [])
def test_unrecognized_only_is_surfaced(self):
from shelfmark.release_sources.prowlarr.source import _split_mam_formats
assert _split_mam_formats("The Martian by Andy Weir [ENG / AVI]") == ([], ["avi"])
def test_mixed_keeps_both_sides(self):
from shelfmark.release_sources.prowlarr.source import _split_mam_formats
assert _split_mam_formats("Title [ENG / M4B AVI]") == (["m4b"], ["avi"])
def test_no_structured_bracket(self):
from shelfmark.release_sources.prowlarr.source import _split_mam_formats
assert _split_mam_formats("Title [VIP]") == ([], [])
assert _split_mam_formats("") == ([], [])
def test_extract_mam_formats_still_returns_recognized(self):
from shelfmark.release_sources.prowlarr.source import _extract_mam_formats
assert _extract_mam_formats("Title [ENG / MP3]") == ["mp3"]
assert _extract_mam_formats("Title [ENG / AVI]") == []
class TestUnrecognizedFormatOnRelease:
def _result(self, title: str) -> dict:
return {
"title": title,
"guid": "https://www.myanonamouse.net/t/627978",
"indexer": "MyAnonamouse",
"indexerId": 1,
"protocol": "torrent",
"size": 320000000,
"seeders": 800,
"leechers": 0,
"categories": [{"id": 3030}],
}
def test_unrecognized_format_lands_in_extra(self):
from shelfmark.release_sources.prowlarr.source import _prowlarr_result_to_release
release = _prowlarr_result_to_release(
self._result("The Martian by Andy Weir [ENG / AVI]"),
"audiobook",
enable_format_detection=True,
)
assert release.format is None
assert release.extra["formats"] is None
assert release.extra["unrecognized_formats"] == ["avi"]
def test_recognized_format_leaves_unrecognized_empty(self):
from shelfmark.release_sources.prowlarr.source import _prowlarr_result_to_release
release = _prowlarr_result_to_release(
self._result("The Martian by Andy Weir [ENG / M4B]"),
"audiobook",
enable_format_detection=True,
)
assert release.format == "m4b"
assert release.extra["unrecognized_formats"] is None
def test_not_populated_without_format_detection(self):
from shelfmark.release_sources.prowlarr.source import _prowlarr_result_to_release
release = _prowlarr_result_to_release(
self._result("The Martian by Andy Weir [ENG / AVI]"), "audiobook"
)
assert release.extra["unrecognized_formats"] is None
class TestProwlarrStaleIndexerSelection:
"""Indexers removed or disabled in Prowlarr must not be searched (#1283)."""
def test_drop_unknown_indexer_ids_keeps_only_live_indexers(self):
assert _drop_unknown_indexer_ids([1, 99], [{"id": 1}, {"id": 2}]) == [1]
def test_drop_unknown_indexer_ids_leaves_search_all_alone(self):
assert _drop_unknown_indexer_ids(None, [{"id": 1}]) is None
def _search_with_selection(self, monkeypatch, selection):
import shelfmark.release_sources.prowlarr.source as prowlarr_source
def fake_get(key: str, default=None):
values = {
"PROWLARR_INDEXERS": selection,
"PROWLARR_AUTO_EXPAND": False,
}
return values.get(key, default)
monkeypatch.setattr(prowlarr_source.config, "get", fake_get)
class RecordingClient(FakeTorznabClient):
def __init__(self):
super().__init__()
self.searched_indexer_ids: list[int] = []
def torznab_search(self, *, indexer_id: int, **kwargs):
self.searched_indexer_ids.append(indexer_id)
return super().torznab_search(indexer_id=indexer_id, **kwargs)
fake_client = RecordingClient()
source = ProwlarrSource()
monkeypatch.setattr(source, "_get_client", lambda: fake_client)
book = BookMetadata(
provider="hardcover",
provider_id="123",
title="Anything",
authors=["Someone"],
)
from shelfmark.core.search_plan import build_release_search_plan
plan = build_release_search_plan(book, languages=["en"], manual_query="my custom")
source.search(book, plan, content_type="ebook")
return fake_client
def test_search_skips_indexer_missing_from_prowlarr(self, monkeypatch):
# The fake Prowlarr only serves indexer 1; 99 was removed behind our back.
fake_client = self._search_with_selection(monkeypatch, [1, 99])
assert fake_client.searched_indexer_ids == [1]
def test_search_queries_nothing_when_every_selected_indexer_is_gone(self, monkeypatch):
fake_client = self._search_with_selection(monkeypatch, [98, 99])
assert fake_client.searched_indexer_ids == []
+98
View File
@@ -0,0 +1,98 @@
"""Tests for listing the files inside a .torrent without downloading it."""
from shelfmark.download.clients.torrent_utils import (
bencode_encode,
extract_file_list_from_torrent,
)
from shelfmark.download.postprocess.packs import PackFile
def _torrent(info: dict) -> bytes:
return bencode_encode({b"announce": b"http://t/announce", b"info": info})
def test_multi_file_torrent_lists_release_relative_paths():
data = _torrent(
{
b"name": b"Sun Eater",
b"piece length": 16384,
b"pieces": b"x" * 20,
b"files": [
{b"length": 10, b"path": [b"Book 1 - Empire of Silence", b"empire.m4b"]},
{b"length": 20, b"path": [b"Book 2 - Howling Dark", b"howling.m4b"]},
{b"length": 1, b"path": [b"cover.jpg"]},
],
}
)
assert extract_file_list_from_torrent(data) == [
PackFile("Sun Eater/Book 1 - Empire of Silence/empire.m4b", 10),
PackFile("Sun Eater/Book 2 - Howling Dark/howling.m4b", 20),
PackFile("Sun Eater/cover.jpg", 1),
]
def test_single_file_torrent_lists_its_one_file():
data = _torrent(
{b"name": b"Book.m4b", b"length": 42, b"piece length": 16384, b"pieces": b"x" * 20}
)
assert extract_file_list_from_torrent(data) == [PackFile("Book.m4b", 42)]
def test_unparseable_data_returns_none():
assert extract_file_list_from_torrent(b"not a torrent") is None
class TestProwlarrHandlerListFiles:
def _handler(self):
from shelfmark.release_sources.prowlarr.handler import ProwlarrHandler
return ProwlarrHandler()
def test_lists_files_from_torrent_url(self):
from unittest.mock import patch
from shelfmark.download.clients.torrent_utils import TorrentInfo
data = _torrent(
{b"name": b"Book.m4b", b"length": 42, b"piece length": 16384, b"pieces": b"x" * 20}
)
release = {
"protocol": "torrent",
"downloadUrl": "http://prowlarr/dl.torrent",
"magnetUrl": "magnet:?xt=urn:btih:abc",
"infoHash": "abc",
}
with (
patch("shelfmark.release_sources.prowlarr.handler.get_release", return_value=release),
patch(
"shelfmark.release_sources.prowlarr.handler.extract_torrent_info",
return_value=TorrentInfo(info_hash="abc", torrent_data=data, is_magnet=False),
) as extract,
):
files = self._handler().list_files({"source_id": "rel-1"})
assert files == [PackFile("Book.m4b", 42)]
extract.assert_called_once_with("http://prowlarr/dl.torrent", expected_hash="abc")
def test_magnet_only_release_cannot_be_listed(self):
from unittest.mock import patch
release = {"protocol": "torrent", "magnetUrl": "magnet:?xt=urn:btih:abc"}
with (
patch("shelfmark.release_sources.prowlarr.handler.get_release", return_value=release),
patch("shelfmark.release_sources.prowlarr.handler.extract_torrent_info") as extract,
):
assert self._handler().list_files({"source_id": "rel-1"}) is None
extract.assert_not_called()
def test_usenet_release_cannot_be_listed(self):
from unittest.mock import patch
release = {"protocol": "usenet", "downloadUrl": "http://prowlarr/dl.nzb"}
with patch("shelfmark.release_sources.prowlarr.handler.get_release", return_value=release):
assert self._handler().list_files({"source_id": "rel-1"}) is None
def test_unknown_release_cannot_be_listed(self):
from unittest.mock import patch
with patch("shelfmark.release_sources.prowlarr.handler.get_release", return_value=None):
assert self._handler().list_files({"source_id": "missing"}) is None
Generated
+4 -4
View File
@@ -1428,7 +1428,7 @@ wheels = [
[[package]]
name = "seleniumbase"
version = "4.52.1"
version = "4.52.2"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "attrs" },
@@ -1492,9 +1492,9 @@ dependencies = [
{ name = "wheel" },
{ name = "wsproto" },
]
sdist = { url = "https://files.pythonhosted.org/packages/fb/9e/a702bc91ac93ba4e4bc29ae3da16a5ac6b436f3d8885646f82555ebf0282/seleniumbase-4.52.1.tar.gz", hash = "sha256:b6c0d67c32465e888231a06839197e73165518dbb06165f7af6b42dfa925a362", size = 672812, upload-time = "2026-08-19T16:31:52.135Z" }
sdist = { url = "https://files.pythonhosted.org/packages/ef/0e/23b3f5232caf0cdadaa67b9e05ff83c9438839c20fe3161c5f0b98f19a8a/seleniumbase-4.52.2.tar.gz", hash = "sha256:261271b3c6d18d404acbe7b7efc661061ff02f0835cfc7cc6e9f52b13f1fa530", size = 677927, upload-time = "2026-08-23T23:36:25.076Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/f0/99/a42b49b356aff2682e1808ef7c6ec28479b96e07d40b5d79ae4858ee9235/seleniumbase-4.52.1-py3-none-any.whl", hash = "sha256:a21a9eb44cabf896a1d21edf7b4569f282e521f913efd69c1becd9202ee35e30", size = 677476, upload-time = "2026-08-19T16:31:48.9Z" },
{ url = "https://files.pythonhosted.org/packages/e9/36/d87a0f455bb80af763f99b0e3a1abea4f35700509338dbb80ce01d1a5422/seleniumbase-4.52.2-py3-none-any.whl", hash = "sha256:d7a7080767cf23ff9f4ace5589035f7eaf35f1ee9544dc038dbf0814f1a0701d", size = 682841, upload-time = "2026-08-23T23:36:21.471Z" },
]
[[package]]
@@ -1575,7 +1575,7 @@ requires-dist = [
{ name = "qbittorrent-api", specifier = ">=2026.8.1" },
{ name = "rarfile" },
{ name = "requests", extras = ["socks"] },
{ name = "seleniumbase", marker = "extra == 'browser'", specifier = "==4.52.1" },
{ name = "seleniumbase", marker = "extra == 'browser'", specifier = "==4.52.2" },
{ name = "tqdm" },
{ name = "transmission-rpc" },
]