Compare commits

..
21 Commits
Author SHA1 Message Date
CaliBrain 97e289ae13 fix: search, Prowlarr and qBittorrent follow-ups (#1276, #1283) (#1284) 2026-08-30 03:09:13 -04:00
Jorge Lima c95ee72ad5 fix(qbittorrent): keep magnets whose metadata is still pending (#1282)
## Problem

`QBittorrentClient.add_download()` waits 20 × 0.5 s for qBittorrent to
leave `metaDL`, then raises:

```
Failed to add to qbittorrent: Torrent metadata resolution was not confirmed within the visibility grace period
(response=TorrentsAddedMetadata({'added_torrent_ids': [], 'failure_count': 0, 'pending_count': 1, 'success_count': 0}))
```

The wait exists to learn qBittorrent's primary torrent ID, which for
hybrid torrents switches from the v1 hash to the truncated v2 hash once
metadata resolves. A magnet on a thin public swarm routinely needs
longer than 10 s to find a peer that will serve metadata, and the
download is then abandoned even though the add itself succeeded. The
torrent stays in qBittorrent (`base_handler` logs "leaving in
qbittorrent") and often completes minutes later with nobody watching it.

Seen on v1.3.12 with public indexers through Prowlarr: every magnet-only
release failed this way, while `.torrent` releases from a private
indexer were fine. qBittorrent showed the same torrents at `metaDL 0%
seeds=0/0`, and they resolved on their own well after shelfmark had
given up.

## Change

Return the info hash we already have instead of raising when the grace
period expires. Reads then resolve either identity:

- `get_status()` and `get_download_path()` use `_resolve_torrent()`
instead of `_get_torrent_info()`, so a v1 hash still matches after
qBittorrent re-keys the torrent to v2. `_torrent_matches_download_id`
already compares `hash`, `infohash_v1` and `infohash_v2`.
- `remove()` and `set_category()` address the torrent by its current
primary hash through a new `_current_hash()` helper, which falls back to
the ID it was given when the torrent cannot be resolved.
- The two magic numbers become `_METADATA_WAIT_POLLS` and
`_METADATA_WAIT_INTERVAL_SECONDS`.

The happy path does not change. When metadata resolves inside the grace
period the resolved primary hash comes back as before, and
`_resolve_torrent()` tries the exact-hash lookup first, so it costs no
extra request.

## Tests

`test_add_fails_when_metadata_never_resolves` asserted the old
behaviour, so it becomes
`test_add_keeps_torrent_when_metadata_never_resolves` and asserts the
info hash is returned.
`test_get_status_resolves_hash_after_metadata_switch` is new: it reads
status by the v1 hash after qBittorrent reports the torrent under its v2
hash.

`uv run pytest tests/ --ignore=tests/e2e` gives the same 55 failures
with and without this change (they are all in `tests/bypass/` and need
Chrome, which my machine has no headless setup for), and
`tests/prowlarr/` is green at 524 passed. Ruff check and format are
clean. I have not run this branch against a live qBittorrent, so a
second pair of eyes on the `remove()` path would help.
2026-08-30 02:00:42 -04:00
CaliBrain b25acdb2ad fix(packs): don't disrupt normal downloads when inspecting for packs (#1274)
Follow-ups to the multi-book pack feature (#1270), which inspects every
release before download. Two behaviours leaked into the ordinary
single-book
flow and are corrected here:

- A flat folder of chaptered audio (`01 - Chapter.mp3`, `02 - ...`) was
detected as a pack, because each track name parses to a series position,
so
clicking download popped the review panel for one normal audiobook. Flat
folders are now split one-book-per-file only with real evidence of
distinct
books: two or more series positions, more than one title, and no
chaptered
audio (only the single-file m4b/m4a containers and ebook formats
qualify).
  Subfolder packs and flat m4b/m4a packs are unchanged.

- Every release that couldn't be inspected (usenet, magnet-only, sources
  without a list_files hook, ABB single-file) showed an info toast on
download. That is now a console.warn, so a normal download is silent
again.

Adds regression tests for the chaptered-mp3 cases.
2026-08-27 01:00:54 -04:00
dependabot[bot] 7569aaecc5 build(deps): bump seleniumbase from 4.52.1 to 4.52.2 in the python-deps group (#1273)
Bumps the python-deps group with 1 update:
[seleniumbase](https://github.com/seleniumbase/SeleniumBase).

Updates `seleniumbase` from 4.52.1 to 4.52.2
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/seleniumbase/SeleniumBase/releases">seleniumbase's
releases</a>.</em></p>
<blockquote>
<h2>4.52.2 - MCP Server Support</h2>
<h2>MCP Server Support</h2>
<p><strong>If you love AI tools, this is one of the biggest releases
this year for SeleniumBase!</strong>
<strong>The new <code>seleniumbase-mcp</code> command starts the
&quot;Pure CDP Mode&quot; MCP Server.</strong>
<strong>(Be sure to install <code>seleniumbase[mcp]</code> to get
<code>mcp&gt;=2.0.0</code>!)</strong>
<strong>(To debug the MCP server from a <code>git clone</code> of
SeleniumBase, get <code>uv</code> as well before calling <code>mcp dev
server.py</code> from the <code>SeleniumBase/mcp_servers/</code>
folder.)</strong></p>
<ul>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/4014476e288d004a8de38eca582631c9c653fa22">Add
a SeleniumBase MCP server</a></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/ac7e449fbbc8a6e9347e65e4ced8176ba1655112">Add
.mcp.json to the root folder</a></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/32fac6becaabbb288be42e98d6528108bdb05e73">Update
setup.cfg files</a></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/168500f45b547c914539057f76e3cd31e0121461">Update
.gitignore</a></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/bb495cb23e405ab9a523068cdb64b48f487f22c5">Update
.dockerignore</a></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/f030eae7b492f41d1c74c731e775f21b853afbdf">Add
configuration for using the new MCP server</a></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/422bb2fc11bd6ef58758c0eff1456f033a94db1f">Refresh
Python dependencies</a></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/7b245c033dc47e356fdaa99b5db7ebf965329bfe">Update
ReadMe files</a></li>
</ul>
<p><strong>Note that you will need <code>mcp&gt;=2.0.0</code> for the
MCP Server to work!</strong></p>
<p>⚠️ Note: Due to a typing bug, (<a
href="https://redirect.github.com/seleniumbase/SeleniumBase/issues/4471">seleniumbase/SeleniumBase#4471</a>),
the MCP Server in this release only worked on Python 3.14+.
Upgrade to <a
href="https://github.com/seleniumbase/SeleniumBase/releases/tag/v4.52.3">https://github.com/seleniumbase/SeleniumBase/releases/tag/v4.52.3</a>
for the fix.</p>
<h2>What's Changed</h2>
<ul>
<li>MCP Server Support by <a
href="https://github.com/mdmintz"><code>@​mdmintz</code></a> in <a
href="https://redirect.github.com/seleniumbase/SeleniumBase/pull/4470">seleniumbase/SeleniumBase#4470</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/seleniumbase/SeleniumBase/compare/v4.52.1...v4.52.2">https://github.com/seleniumbase/SeleniumBase/compare/v4.52.1...v4.52.2</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/a28aa518e34c859d3a3f90daf03c1aa926940931"><code>a28aa51</code></a>
Merge pull request <a
href="https://redirect.github.com/seleniumbase/SeleniumBase/issues/4470">#4470</a>
from seleniumbase/mcp-server-support</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/7b245c033dc47e356fdaa99b5db7ebf965329bfe"><code>7b245c0</code></a>
Update ReadMe files</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/1e5ac8dd110ecb96add400a45f009735be5e8683"><code>1e5ac8d</code></a>
Version 4.52.2</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/422bb2fc11bd6ef58758c0eff1456f033a94db1f"><code>422bb2f</code></a>
Refresh Python dependencies</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/f030eae7b492f41d1c74c731e775f21b853afbdf"><code>f030eae</code></a>
Add configuration for using the new MCP server</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/bb495cb23e405ab9a523068cdb64b48f487f22c5"><code>bb495cb</code></a>
Update <code>.dockerignore</code></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/168500f45b547c914539057f76e3cd31e0121461"><code>168500f</code></a>
Update <code>.gitignore</code></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/32fac6becaabbb288be42e98d6528108bdb05e73"><code>32fac6b</code></a>
Update <code>setup.cfg</code> files</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/ac7e449fbbc8a6e9347e65e4ced8176ba1655112"><code>ac7e449</code></a>
Add <code>.mcp.json</code> to the root folder</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/4014476e288d004a8de38eca582631c9c653fa22"><code>4014476</code></a>
Add a SeleniumBase MCP server</li>
<li>See full diff in <a
href="https://github.com/seleniumbase/SeleniumBase/compare/v4.52.1...v4.52.2">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=seleniumbase&package-manager=uv&previous-version=4.52.1&new-version=4.52.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-27 00:41:57 -04:00
Lance Marks f441b85da2 feat(packs): inspect multi-book releases and file each book separately (#1270)
## Multi-book packs: inspect a release before download and file each
book separately

Closes #576

### Problem

One queued release is always treated as one book. When a torrent is
actually a whole series
(`Series/Book 1 - Title/…`, or a flat folder of `Series 1.0 - Title.m4b`
files), post-processing
walks the whole tree, flattens every file into one list and renames them
`Title - 01…10` under the
searched book's `{Author}/{Title}`. Audiobookshelf then sees a single
10-file "book" and the user
has to re-file everything by hand.

### What this does

Most releases expose their file list *before* anything is downloaded, so
the split is decided up
front and approved by the user, then the download is fire-and-forget:

1. **Inspect** – clicking a release's download button now calls `POST
/api/releases/inspect`
first. A new optional `DownloadHandler.list_files(release_data)` hook
returns the release's
   files without downloading:
- **AudiobookBay** reads the torrent file table off the detail page it
already fetches (the
page is now cached for 120 s, so inspect + download cost ABB one
request).
- **Prowlarr** parses `info.files` from the `.torrent` it already
fetches (the existing 120 s
torrent-fetch cache is reused). Magnet-only and usenet releases report
"can't inspect".
   - Other sources default to `None`.
2. **Review** – if the plan contains more than one book, the Find
Releases modal swaps the list
for a review panel: one row per book with editable title / series
position / year, expandable
file lists, non-book sidecars (`.txt`, covers) shown as ignored, a
"Treat as a single book"
switch, and **Download N books**. Single-book releases queue
immediately, exactly as before.
3. **File** – the approved plan travels with the task
(`DownloadTask.book_plan`, retry-safe) and
post-processing files each book through the existing transfer code, one
book at a time
(`dataclasses.replace(task, title=…, series_position=…, year=…)`), so
organize/rename
templates, part numbering (now scoped per book), hardlinks, torrent
copy-preserve and usenet
   handling are unchanged. Status reads `Complete (N books, M files)`.
4. **Fallback** – when a release can't be inspected the user gets a
toast, and a small
"Multi-book pack" toggle in the modal header forces a heuristic split
(subfolder = book, or
   one book per file when the file names carry series positions).

Planning lives in `shelfmark/download/postprocess/packs.py` and is
shared by the inspect endpoint
and post-processing, so what the user approved is what gets filed. The
name parser strips
`Book 3 -`, `03 -`, `1.0 -`, `3.`, `[03]`, `#3`, a leading series name,
labels like
"An Expanse Novella -", repeated titles (`Gods of Risk 2.5 - Gods of
Risk`) and a trailing
`(Year)`; author and series name come from the book that was searched,
and the searched book's
own series position is never applied to its siblings.

### Files

- `shelfmark/download/postprocess/packs.py` (new) –
`PackFile/PackBook/PackPlan`, `plan_pack`,
`parse_pack_book_name`, `group_files_into_books`, `match_plan_to_files`
- `shelfmark/core/release_inspect_routes.py` (new) – `POST
/api/releases/inspect`
- `shelfmark/release_sources/__init__.py` – `DownloadHandler.list_files`
hook
- `shelfmark/release_sources/audiobookbay/{scraper,handler}.py` –
detail-page cache,
  `extract_file_list`, `list_files`
- `shelfmark/release_sources/prowlarr/handler.py`,
`download/clients/torrent_utils.py` –
  `extract_file_list_from_torrent`, `list_files`
- `shelfmark/core/models.py`, `download/orchestrator.py` – `multi_book`
/ `book_plan` fields,
  queue + retry serialization
- `shelfmark/download/postprocess/transfer.py`, `pipeline.py`,
`outputs/folder.py` – per-book
  transfer branch and status message
- `src/frontend`: `components/PackReviewPanel.tsx` (new),
`ReleaseModal.tsx`, `App.tsx`,
`services/api.ts`, `types/index.ts`, `utils/releasePayload.ts` (payload
builder moved out of
  `App.tsx`), `utils/packReview.ts`
- `docs/dev/release-sources-plugin-guide.md` – documents the
`list_files` hook

### Out of scope (follow-ups)

- Listing files from an NZB (Shelfmark already fetches the bytes; `<file
subject>` names are noisy)
- Inspecting magnet links via qBittorrent's files API after a paused add
- BookLore / email outputs (they ignore `book_plan`; noted in code)
- The combined ebook + audiobook flow

### Testing

**Automated** (`make checks`, `make python-test`, `make frontend-test`
all green; the only
failures on my machine are the pre-existing
`tests/config/test_entrypoint_permissions.py` cases,
which need bash ≥ 4 and fail identically on `main` under macOS bash
3.2):

- `tests/download/test_packs.py` – name parsing (markers, series name,
novella labels, repeated
titles, bare numeric titles like `1984`), nested / flat / mixed /
deeper-nested packs, single
wrapping folder not treated as a pack, plan-to-disk matching with
basename fallback
- `tests/core/test_processing_packs.py` – full `post_process_download`
runs on a real temp
filesystem: approved plan files each book under its own
`{Author}/{Title}`, heuristic split
of a nested pack, searched book's series position does not leak,
multi-file book inside a pack
keeps `- 01/- 02` per book, hardlinked torrent pack leaves the seeding
tree intact, no pack
fields ⇒ behaviour unchanged, single group degrades to the searched
title, status message
- `tests/core/test_release_inspect_routes.py` – plan response,
not-inspectable, handler errors
  never 500, unknown source / missing `source_id` ⇒ 400, login required
- `tests/audiobookbay/test_file_list.py` – file-table scraping from real
ABB markup (multi-file
and single-file pages), handler host validation, one page fetch shared
by magnet + file list
- `tests/prowlarr/test_torrent_file_list.py` – multi-file / single-file
`.torrent` parsing,
  handler behaviour for torrent URL vs magnet vs usenet vs cache miss
- `tests/download/test_orchestrator_pack_fields.py` – queue-time parsing
and retry round-trip
- Frontend: `releasePayload.test.ts`, `packReview.test.ts` (vitest)

**Manual, on a real deployment** (arm64 image built from this branch,
run as a side container
next to production with the same qBittorrent / Audiobookshelf setup,
`FILE_ORGANIZATION_AUDIOBOOK=organize`,
hardlinks on):

- AudiobookBay "The Expanse Complete 2.0" (7.87 GB, 36 files): clicking
download opened the review
panel in ~1 s showing **18 books · 18 files · 18 files ignored** (the
`.txt` sidecars), with
series positions 0.1–9.5 and years parsed from the file names; novella
labels stripped
("The Churn", "The Butcher of Anderson Station"). Editing a title in the
panel works.
Confirming queued one task; the magnet resolved from the cached page in
~30 ms; after the
download the task reported `Complete (18 books, 18 files)`, 18 hardlinks
landed as
`audiobooks/James S. A. Corey/<Title>/<Title>.m4b`, the torrent kept
seeding, and
Audiobookshelf scanned each folder as its own book (title, author,
embedded chapters).
- A second pack ("Expanse [01 - 9.5]", `Title N - Title` naming) was
inspected to verify the
  repeated-title rule and the Back button, without downloading.
- Single-book releases still queue immediately with no extra UI.
2026-08-27 00:40:08 -04:00
zab1996andRyan 02b7e9d958 feat(newznab): support multiple named indexers (#1271)
## Summary

- add a named Newznab indexer table with per-indexer URL and API key
settings
- search every configured indexer and retain the originating indexer
name on each result
- namespace cached release IDs across connections and isolate individual
indexer failures
- preserve the legacy single-indexer settings as a fallback
- support masked API-key cells and trusted SABnzbd prefetching for named
indexers

## Validation

- 121 Newznab and SABnzbd backend tests passed on Python 3.14
- Ruff passed for all changed Python files
- frontend TypeScript and strict lint checks passed
- all 134 frontend unit tests passed
- frontend formatting check passed

## Compatibility

Existing `NEWZNAB_URL` and `NEWZNAB_API_KEY` configurations continue to
work whenever `NEWZNAB_INDEXERS` is empty.

Co-authored-by: Ryan <zab1996@users.noreply.github.com>
2026-08-27 00:29:58 -04:00
CaliBrain ff06a1a581 fix(search): follow-ups to per-user book languages (#1267)
Review follow-ups to #1255, all in the code that PR touched.

Drop the dead user_id from the Prowlarr retry path.
ProwlarrSource.search
never reads plan.languages, and _refresh_release builds a synthetic book
with no titles_by_language, so the title variants came out identical
with
and without it. It also should not language-filter: it re-finds one
exact
release by its guid.

Pin the tab move in tests. BOOK_LANGUAGE moved from the General tab to
Search Mode with no migration, which only works because both tabs
persist
into the same settings.json. Nothing asserted that, so splitting the
files
later would silently reset every install to ["en"]. Covers the stored
value, a fresh install, and ENV precedence.

Stop the UI inventing a default language. An empty BOOK_LANGUAGE is a
deliberate "no default filter" that the backend preserves, but the two
frontend call sites replaced it with the first supported language, so
the
filter said English where the server filtered nothing.
resolveDefaultLanguageCodes
now falls back only when the value is absent.

Keep the normalized value for every validated search key.
validate_user_settings
gated the write-back on a hand-maintained subset of the keys the search
validator recognises, so METADATA_PROVIDER_COMBINED,
SHOW_COMBINED_SELECTOR
and FORCE_COMBINED_SEARCH were validated and then stored raw -- a padded
provider name was accepted and persisted with its padding. Reuse the
validator's own key set instead.

Skip blank language entries rather than rejecting them, so "" and "en,"
mean the same as [] and ["en"] instead of erroring on an unnamed
language.

Extract resolveListOverride for the list-override detection that was
copy-pasted between the two user-settings sections, and mention
languages
in the Search Preferences section description.
2026-08-24 18:11:29 -04:00
463ef49ac3 feat(search): let each user pick their own default book languages (#1255)
## Why

`BOOK_LANGUAGE` is a per-reader property, not a per-instance one. On a
shared install one household member searches in German while another
wants English and German — today whoever changes the setting changes it
for everyone, and the only escape is re-picking languages in the filter
on every single search.

The per-user override machinery already carries `SEARCH_MODE`, the
metadata providers and the default release sources, so the language
default mostly had to opt into it.

## What changed

**The field.** `BOOK_LANGUAGE` becomes `user_overridable` and moves from
the **General** tab to **Search Mode**, next to the other
user-overridable search defaults (per
[review](https://github.com/calibrain/shelfmark/pull/1255#issuecomment-5391189094)
— the first version had the Search section span two tabs, this one
doesn't). Admins set it per user in the user editor, users set it in
**My Account → Search Preferences**, and the Search Mode tab carries the
usual "N users override this" summary.

**No migration for the move.** `general` and `search_mode` both persist
into `settings.json`, and a field's value is resolved through
`load_config_file(tab)` for the tab it's declared on — so an install
that already stores `BOOK_LANGUAGE` keeps its value. Checked against a
`settings.json` written while the field still lived on General: the
stored value resolves unchanged, a fresh install still gets `["en"]`,
and `BOOK_LANGUAGE` in the environment still overrides both.

**The two places the default is read.**

- `/api/config` seeds the frontend's language filter, so it now resolves
`BOOK_LANGUAGE` for the session user.
- `build_release_search_plan` falls back to the default whenever a
request carries no language filter — which is exactly what the filter's
"Default" option sends. It takes an optional `user_id`, passed by
`/api/releases` from the session and by the Prowlarr retry path from
`task.user_id`, so a retry re-searches in the languages of whoever
queued the download.

**Validation.** Overrides go through `normalize_language()`, so
`"German"`, `"ger"` and `"de"` all store as `de`, and an unknown
language is rejected with a message naming it instead of being silently
searched for. An empty list stays an empty list (a deliberate "no
default filter"), `null` clears the override as everywhere else, and ENV
still wins: with `BOOK_LANGUAGE` set in the environment the field
reports `fromEnv` and overrides are ignored.

**Scope.** Only the language default becomes overridable. The two format
lists left behind under "Default Search Filters" stay admin-only — they
describe what the library and its post-processing accept, not what a
reader wants to read. There's a test pinning that.

## Verification

- 2681 unit tests pass (2670 before, 11 added)
- `ruff check`, `ruff format`, `basedpyright` over backend and tests,
and `vulture` all clean; frontend lint, format, typecheck and 126 unit
tests clean
- `docs/environment-variables.md` regenerated via
`scripts/generate_env_docs.py` (the `BOOK_LANGUAGE` row follows the
field into the Search Mode section)
- Manually against a two-user instance with builtin auth (first round,
before the tab move): with user A on German and user B on
English+German, `/api/config` returns each reader their own
`default_language` and an unfiltered `/api/releases` plans the matching
languages; an admin can set and read the same override for another user;
clearing it falls back to the global value; a stray `"klingon"` is
rejected; and `BOOK_LANGUAGE` in the environment overrides both users
with the field marked `fromEnv`
- After the tab move I re-ran the suites above plus the
stored-value/fresh-install/ENV check described under "No migration for
the move"; the behaviour it exercises is what the move could have broken

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: CaliBrain <calibrain@l4n.xyz>
2026-08-24 17:57:21 -04:00
CaliBrain a5595cf9f1 Change test for fake extension that wont work (#1266) 2026-08-24 17:54:10 -04:00
jakesterpdxandClaude Fable 5 9bcf595111 feat(prowlarr): warn when an indexer declares a format Shelfmark can't process (#1265)
## Problem

Companion to #1264, but general rather than mp4-specific.

MyAnonamouse titles carry a structured `[LANG / FORMATS]` bracket that
`_extract_mam_formats` parses. When every token in it is something
Shelfmark doesn't know — e.g. `The Martian by Andy Weir [ENG / MP4]` —
the release is rendered with **no format chip at all**, just the generic
headphones/book icon with an "Audiobook" tooltip. To a user that looks
like an ordinary result. It downloads fine and then fails
post-processing with *"No book files found in download"*.

The backend already *had* the signal (a format token it couldn't map);
it just threw it away.

## Change

**Backend** (`shelfmark/release_sources/prowlarr/source.py`)
- `_split_mam_formats(raw_title) -> (recognized, unrecognized)` replaces
the body of `_extract_mam_formats`, which is kept as a thin wrapper
returning `recognized` so nothing else changes.
- Releases gain `extra["unrecognized_formats"]` (list, or `None` when
empty / when format detection is off).

**Frontend**
- `getUnrecognizedReleaseFormats(release)` in `utils/releaseFormats.ts`
(normalised + deduped, same shape as `getReleaseFormats`).
- `ReleaseCell` `format_content_type`: when there is **no** recognised
format but the indexer named one, render an amber `MP4 Unsupported`
badge (compact view: amber `MP4`) with tooltip *"Unsupported format
(MP4) - Shelfmark cannot process this release"*. When a recognised
format exists the existing badge is untouched, even if extra unknown
tokens were present.

Only the chip changes — the download button still works, so a user can
still grab and hand-process the files if they want to. Happy to disable
the button instead if you'd prefer.

## Tests

- `tests/prowlarr/test_source.py`: `TestSplitMamFormats` (recognised /
unrecognised / mixed / no bracket / wrapper compat) and
`TestUnrecognizedFormatOnRelease` (lands in `extra`, empty when
recognised, absent without format detection).
- `src/frontend/src/tests/releaseFormats.test.ts`: 3 cases for the new
helper.
- `ruff check` clean; `pytest tests/prowlarr -m "not integration"` 511
passed; `tsc --noEmit`, `oxlint --deny warnings`, `vitest` all clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-24 17:50:27 -04:00
jakesterpdxandClaude Fable 5 65e2e3be20 feat(audiobook): recognise .mp4 as an audiobook format (#1264)
## Problem

Some trackers — MyAnonamouse in particular — distribute AAC audiobooks
as per-chapter `.mp4` files. That's the same ISO-BMFF container as
`.m4a`/`.m4b`, just with the generic extension (`ftyp isom`,
audio-only).

Today those releases:
1. show up in Prowlarr search results with **no format chip** — only the
generic "Audiobook" icon, because no format could be inferred;
2. download successfully; then
3. fail post-processing with **"No book files found in download"**,
because `.mp4` isn't in `AUDIOBOOK_FORMATS` (`shelfmark/core/utils.py`).

Real example: MAM #627978, *The Martian* (Andy Weir, 2020 edition) — 142
files `0001 … 0142 Andy Weir (2020) The Martian.mp4` + `cover.jpg`, 305
MB. Every file is a valid AAC-in-MP4 chapter.

Adding `mp4` to `SUPPORTED_AUDIOBOOK_FORMATS` in `settings.json` doesn't
help since the hard-coded tuple is what post-processing scans against.

## Change

- Add `"mp4"` to `AUDIOBOOK_FORMATS` (single source of truth — settings
UI, Prowlarr parsing, IRC parser, archive extraction and post-download
scan all derive from it), with a comment explaining why.
- Add `".mp4"` to the two hand-maintained debrid `_BOOK_EXTENSIONS`
lists (AllDebrid / Real-Debrid) so file selection matches.
- Slot `mp4` into the IRC `AUDIOBOOK_FORMAT_PRIORITY` table right after
`m4a` (same container family).
- Update the documented default in `docs/environment-variables.md`.
- New regression test
`test_audiobook_multifile_mp4_chapters_are_book_files` modelled on the
existing multi-file usenet test.

### Note for existing installs

The legacy-default migration only widens configs that still hold the old
`m4b,mp3` list, so users on the current widened default won't pick up
`mp4` automatically — they'll need to tick it in Settings → Audiobook
formats. New installs get it by default. Happy to extend the migration
if you'd rather it be automatic.

## Testing

- `ruff check` / `ruff format --check`: clean
- `pytest tests/core tests/config tests/irc tests/prowlarr
tests/download -m "not integration and not e2e"`: 2296 passed, new test
+ `test_audiobook_format_consistency.py` all green. The 10 failures in
`test_entrypoint_permissions.py` / `test_orchestrator_stall.py`
reproduce identically on untouched `main` on macOS and are unrelated.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-24 17:45:54 -04:00
CaliBrain ddc26f01b6 fix(download): escalating per-host cooldown on HTTP 429 (#1263)
Anna's Archive 429-throttles the source IP after repeated automated
requests.
The bypasser could clear the DDoS-Guard challenge but not the 429, so
each retry
re-solved, re-spawned Chrome, and rotated mirrors that share the same IP
- a
costly loop that never converged.

Add a process-global, per-host cooldown that escalates 2 -> 5 -> 10 ->
15 -> 30
minutes each time a host 429s again after its window elapsed, resetting
after a
long clear gap. Mirror selection skips cooling hosts and the bypasser
refuses to
solve one, so a throttled host fails fast instead of storming the
solver.
2026-08-24 13:08:07 -04:00
CaliBrain 89104ae80f fix(search): let manual search switch media type under forced combined search (#1262)
Manual search browses release sources directly, one media type at a
time,
so the combined (both) flow never applied to it — yet
FORCE_COMBINED_SEARCH
locked the content-type selector onto both, pinning manual search to
ebook
sources with no way to reach audiobook sources (no Audiobay tab).

Treat a manual search target as combined-exempt in the search bar:
present a
plain, switchable Books/Audiobooks selector (unlocked, no combined
toggle),
even when combined search is forced on for metadata targets. Metadata
search
behavior is unchanged.

Fixes #1256
2026-08-24 01:57:42 -04:00
dependabot[bot] 1e45add4d5 build(deps-dev): bump knip from 6.32.1 to 6.32.2 in /src/frontend in the npm-deps group (#1246)
Bumps the npm-deps group in /src/frontend with 1 update:
[knip](https://github.com/webpro-nl/knip/tree/HEAD/packages/knip).

Updates `knip` from 6.32.1 to 6.32.2
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/webpro-nl/knip/releases">knip's
releases</a>.</em></p>
<blockquote>
<h2>Release 6.32.2</h2>
<ul>
<li>Support <code>oxfmt.config.mts</code> (<a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1933">#1933</a>)
(795900191dc75eec8d1e717b866bf57e1e2912cc) - thanks <a
href="https://github.com/joealden"><code>@​joealden</code></a>!</li>
<li>Support <code>oxlint.config.mts</code> (<a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1934">#1934</a>)
(531e2dc7c1d8bf31babea0068c34391182ec2d50) - thanks <a
href="https://github.com/joealden"><code>@​joealden</code></a>!</li>
<li>Fix Supported <code>lint-staged</code> Configs (<a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1935">#1935</a>)
(f9c755e414ed10baa4d01af8ddac6d04cb8d5617) - thanks <a
href="https://github.com/joealden"><code>@​joealden</code></a>!</li>
<li>Update dependencies (95f7c529f918dd9e1a84f92c68d064738977b825)</li>
<li>Update sentry snapshot
(ea7929fcbd6b323c8bdd9252ac57017feeb29ecf)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/webpro-nl/knip/commit/196a11a4e0cb5208c0dc8382a54341d5630af631"><code>196a11a</code></a>
Release knip@6.32.2</li>
<li><a
href="https://github.com/webpro-nl/knip/commit/95f7c529f918dd9e1a84f92c68d064738977b825"><code>95f7c52</code></a>
Update dependencies</li>
<li><a
href="https://github.com/webpro-nl/knip/commit/f9c755e414ed10baa4d01af8ddac6d04cb8d5617"><code>f9c755e</code></a>
Fix Supported <code>lint-staged</code> Configs (<a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1935">#1935</a>)</li>
<li><a
href="https://github.com/webpro-nl/knip/commit/531e2dc7c1d8bf31babea0068c34391182ec2d50"><code>531e2dc</code></a>
Support <code>oxlint.config.mts</code> (<a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1934">#1934</a>)</li>
<li><a
href="https://github.com/webpro-nl/knip/commit/795900191dc75eec8d1e717b866bf57e1e2912cc"><code>7959001</code></a>
Support <code>oxfmt.config.mts</code> (<a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1933">#1933</a>)</li>
<li>See full diff in <a
href="https://github.com/webpro-nl/knip/commits/knip@6.32.2/packages/knip">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=knip&package-manager=npm_and_yarn&previous-version=6.32.1&new-version=6.32.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-24 01:57:08 -04:00
dependabot[bot] cb3f6fee82 build(deps): bump astral-sh/uv from 0.11.3 to 0.12.5 in the docker-base-image-digests group (#1247)
Bumps the docker-base-image-digests group with 1 update:
[astral-sh/uv](https://github.com/astral-sh/uv).

Updates `astral-sh/uv` from 0.11.3 to 0.12.5
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/astral-sh/uv/releases">astral-sh/uv's
releases</a>.</em></p>
<blockquote>
<h2>0.12.5</h2>
<h2>Release Notes</h2>
<p>Released on 2026-08-14.</p>
<h3>Python</h3>
<ul>
<li>Add CPython 3.10.21, 3.11.16, and 3.12.14 (<a
href="https://redirect.github.com/astral-sh/uv/pull/21138">#21138</a>)</li>
<li>Prefer newer versions and standard variants when selecting between
equally prioritized Python interpreters (<a
href="https://redirect.github.com/astral-sh/uv/pull/21134">#21134</a>)</li>
</ul>
<h3>Enhancements</h3>
<ul>
<li>Simplify errors and hints for invalid editable requirements, and
redact credentials in requirement URLs (<a
href="https://redirect.github.com/astral-sh/uv/pull/21130">#21130</a>)</li>
</ul>
<h3>Preview features</h3>
<ul>
<li>Allow <code>--index</code> and <code>--default-index</code> to
select configured package indexes by name with the
<code>index-by-name</code> preview feature (<a
href="https://redirect.github.com/astral-sh/uv/pull/17455">#17455</a>)</li>
<li>Include distribution artifact URLs and hashes in CycloneDX SBOM
exports by default (<a
href="https://redirect.github.com/astral-sh/uv/pull/21131">#21131</a>)</li>
<li>Fall back to logical file sizes when using
<code>cache-physical-space</code> on filesystems that do not support
physical-space accounting (<a
href="https://redirect.github.com/astral-sh/uv/pull/21133">#21133</a>)</li>
</ul>
<h3>Bug fixes</h3>
<ul>
<li>Resolve relative package index paths in PEP 723 scripts against the
script directory (<a
href="https://redirect.github.com/astral-sh/uv/pull/21097">#21097</a>)</li>
</ul>
<h2>Install uv 0.12.5</h2>
<h3>Install prebuilt binaries via shell script</h3>
<pre lang="sh"><code>curl --proto '=https' --tlsv1.2 -LsSf
https://releases.astral.sh/github/uv/releases/download/0.12.5/uv-installer.sh
| sh
</code></pre>
<h3>Install prebuilt binaries via powershell script</h3>
<pre lang="sh"><code>powershell -ExecutionPolicy Bypass -c &quot;irm
https://releases.astral.sh/github/uv/releases/download/0.12.5/uv-installer.ps1
| iex&quot;
</code></pre>
<h2>Download uv 0.12.5</h2>
<table>
<thead>
<tr>
<th>File</th>
<th>Platform</th>
<th>Checksum</th>
</tr>
</thead>
<tbody>
<tr>
<td><a
href="https://releases.astral.sh/github/uv/releases/download/0.12.5/uv-aarch64-apple-darwin.tar.gz">uv-aarch64-apple-darwin.tar.gz</a></td>
<td>Apple Silicon macOS</td>
<td><a
href="https://releases.astral.sh/github/uv/releases/download/0.12.5/uv-aarch64-apple-darwin.tar.gz.sha256">checksum</a></td>
</tr>
<tr>
<td><a
href="https://releases.astral.sh/github/uv/releases/download/0.12.5/uv-x86_64-apple-darwin.tar.gz">uv-x86_64-apple-darwin.tar.gz</a></td>
<td>Intel macOS</td>
<td><a
href="https://releases.astral.sh/github/uv/releases/download/0.12.5/uv-x86_64-apple-darwin.tar.gz.sha256">checksum</a></td>
</tr>
<tr>
<td><a
href="https://releases.astral.sh/github/uv/releases/download/0.12.5/uv-aarch64-pc-windows-msvc.zip">uv-aarch64-pc-windows-msvc.zip</a></td>
<td>ARM64 Windows</td>
<td><a
href="https://releases.astral.sh/github/uv/releases/download/0.12.5/uv-aarch64-pc-windows-msvc.zip.sha256">checksum</a></td>
</tr>
<tr>
<td><a
href="https://releases.astral.sh/github/uv/releases/download/0.12.5/uv-i686-pc-windows-msvc.zip">uv-i686-pc-windows-msvc.zip</a></td>
<td>x86 Windows</td>
<td><a
href="https://releases.astral.sh/github/uv/releases/download/0.12.5/uv-i686-pc-windows-msvc.zip.sha256">checksum</a></td>
</tr>
<tr>
<td><a
href="https://releases.astral.sh/github/uv/releases/download/0.12.5/uv-x86_64-pc-windows-msvc.zip">uv-x86_64-pc-windows-msvc.zip</a></td>
<td>x64 Windows</td>
<td><a
href="https://releases.astral.sh/github/uv/releases/download/0.12.5/uv-x86_64-pc-windows-msvc.zip.sha256">checksum</a></td>
</tr>
<tr>
<td><a
href="https://releases.astral.sh/github/uv/releases/download/0.12.5/uv-aarch64-unknown-linux-gnu.tar.gz">uv-aarch64-unknown-linux-gnu.tar.gz</a></td>
<td>ARM64 Linux</td>
<td><a
href="https://releases.astral.sh/github/uv/releases/download/0.12.5/uv-aarch64-unknown-linux-gnu.tar.gz.sha256">checksum</a></td>
</tr>
<tr>
<td><a
href="https://releases.astral.sh/github/uv/releases/download/0.12.5/uv-i686-unknown-linux-gnu.tar.gz">uv-i686-unknown-linux-gnu.tar.gz</a></td>
<td>x86 Linux</td>
<td><a
href="https://releases.astral.sh/github/uv/releases/download/0.12.5/uv-i686-unknown-linux-gnu.tar.gz.sha256">checksum</a></td>
</tr>
<tr>
<td><a
href="https://releases.astral.sh/github/uv/releases/download/0.12.5/uv-powerpc64le-unknown-linux-gnu.tar.gz">uv-powerpc64le-unknown-linux-gnu.tar.gz</a></td>
<td>PPC64LE Linux</td>
<td><a
href="https://releases.astral.sh/github/uv/releases/download/0.12.5/uv-powerpc64le-unknown-linux-gnu.tar.gz.sha256">checksum</a></td>
</tr>
</tbody>
</table>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/astral-sh/uv/blob/main/CHANGELOG.md">astral-sh/uv's
changelog</a>.</em></p>
<blockquote>
<h2>0.12.5</h2>
<p>Released on 2026-08-14.</p>
<h3>Python</h3>
<ul>
<li>Add CPython 3.10.21, 3.11.16, and 3.12.14 (<a
href="https://redirect.github.com/astral-sh/uv/pull/21138">#21138</a>)</li>
<li>Prefer newer versions and standard variants when selecting between
equally prioritized Python interpreters (<a
href="https://redirect.github.com/astral-sh/uv/pull/21134">#21134</a>)</li>
</ul>
<h3>Enhancements</h3>
<ul>
<li>Simplify errors and hints for invalid editable requirements, and
redact credentials in requirement URLs (<a
href="https://redirect.github.com/astral-sh/uv/pull/21130">#21130</a>)</li>
</ul>
<h3>Preview features</h3>
<ul>
<li>Allow <code>--index</code> and <code>--default-index</code> to
select configured package indexes by name with the
<code>index-by-name</code> preview feature (<a
href="https://redirect.github.com/astral-sh/uv/pull/17455">#17455</a>)</li>
<li>Include distribution artifact URLs and hashes in CycloneDX SBOM
exports by default (<a
href="https://redirect.github.com/astral-sh/uv/pull/21131">#21131</a>)</li>
<li>Fall back to logical file sizes when using
<code>cache-physical-space</code> on filesystems that do not support
physical-space accounting (<a
href="https://redirect.github.com/astral-sh/uv/pull/21133">#21133</a>)</li>
</ul>
<h3>Bug fixes</h3>
<ul>
<li>Resolve relative package index paths in PEP 723 scripts against the
script directory (<a
href="https://redirect.github.com/astral-sh/uv/pull/21097">#21097</a>)</li>
</ul>
<h2>0.12.4</h2>
<p>Released on 2026-08-13.</p>
<h3>Enhancements</h3>
<ul>
<li>Prefer post-quantum key exchange and enable opt-in TLS diagnostics
(<a
href="https://redirect.github.com/astral-sh/uv/pull/21054">#21054</a>)</li>
<li>Accept whitespace before versions in noncompliant wildcard
comparisons such as <code>Requires-Python: &gt;= 3.5.*</code> (<a
href="https://redirect.github.com/astral-sh/uv/pull/21012">#21012</a>)</li>
<li>Report a specific error when a PEP 723 closing tag contains trailing
whitespace or other content (<a
href="https://redirect.github.com/astral-sh/uv/pull/20944">#20944</a>)</li>
<li>Omit source-span carets from diagnostics for empty PEP 508
requirements (<a
href="https://redirect.github.com/astral-sh/uv/pull/21094">#21094</a>)</li>
</ul>
<h3>Preview features</h3>
<ul>
<li>Add <code>uv check --no-install-project</code> and respect
<code>UV_NO_INSTALL_PROJECT</code> to install dependencies without
building or installing the project (<a
href="https://redirect.github.com/astral-sh/uv/pull/21085">#21085</a>)</li>
<li>Make the ty subprocess invoked by <code>uv check</code> honor uv's
color and progress settings, including quiet mode (<a
href="https://redirect.github.com/astral-sh/uv/pull/21086">#21086</a>)</li>
</ul>
<h3>Performance</h3>
<ul>
<li>Speed up resolutions with long runs of unavailable package versions
by coalescing gaps in the resolver's version ranges (<a
href="https://redirect.github.com/astral-sh/uv/pull/20804">#20804</a>)</li>
<li>Speed up Simple API parsing by deserializing PyPI and Pyx file
metadata directly (<a
href="https://redirect.github.com/astral-sh/uv/pull/21041">#21041</a>)</li>
</ul>
<h3>Bug fixes</h3>
<ul>
<li>Use windowed <code>pythonw.exe</code> launchers for virtual
environments created from managed Python minor-version links (<a
href="https://redirect.github.com/astral-sh/uv/pull/19235">#19235</a>)</li>
<li>Allow <code>uv lock</code> to proceed when <code>.venv</code> is an
unusable project environment (<a
href="https://redirect.github.com/astral-sh/uv/pull/21068">#21068</a>)</li>
<li>Respect <code>fork-strategy</code> when ordering forks created from
<code>environments</code> or existing lockfile
<code>resolution-markers</code> (<a
href="https://redirect.github.com/astral-sh/uv/pull/21000">#21000</a>)</li>
<li>Preserve consecutive wildcard Python minor-version exclusions such
as <code>!=3.11.*, !=3.12.*</code> in <code>uv.lock</code> (<a
href="https://redirect.github.com/astral-sh/uv/pull/21045">#21045</a>)</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/astral-sh/uv/commit/210d1f6785e95a8c8c0d53e284408c9be1134700"><code>210d1f6</code></a>
Bump version to 0.12.5 (<a
href="https://redirect.github.com/astral-sh/uv/issues/21140">#21140</a>)</li>
<li><a
href="https://github.com/astral-sh/uv/commit/802a9162a1153ddee799a0080c80a38d96009d91"><code>802a916</code></a>
Sync latest Python releases: 3.10.21, 3.11.16, 3.12.14 (<a
href="https://redirect.github.com/astral-sh/uv/issues/21138">#21138</a>)</li>
<li><a
href="https://github.com/astral-sh/uv/commit/a6904bbdabd3c6ac42b1051896ff597c8bd1895a"><code>a6904bb</code></a>
Order equal-priority Python installations by key (<a
href="https://redirect.github.com/astral-sh/uv/issues/21134">#21134</a>)</li>
<li><a
href="https://github.com/astral-sh/uv/commit/728a70d25c0889d9aaa53776256c3a843eea67a6"><code>728a70d</code></a>
Improve automated fixes for related bug manifestations (<a
href="https://redirect.github.com/astral-sh/uv/issues/21102">#21102</a>)</li>
<li><a
href="https://github.com/astral-sh/uv/commit/b82b0383fc3471227aa864f18d2c1c8efac8b9f1"><code>b82b038</code></a>
Include hashes in cyclonedx exports (<a
href="https://redirect.github.com/astral-sh/uv/issues/21131">#21131</a>)</li>
<li><a
href="https://github.com/astral-sh/uv/commit/8011778d41f2c96fb0b856fb3cbcb4ec8d28cb63"><code>8011778</code></a>
Simplify editable requirement errors and hints (<a
href="https://redirect.github.com/astral-sh/uv/issues/21130">#21130</a>)</li>
<li><a
href="https://github.com/astral-sh/uv/commit/dca33f5e6799f2aefee52729732a839a3b358740"><code>dca33f5</code></a>
Fall back to logical cache accounting on unsupported filesystems (<a
href="https://redirect.github.com/astral-sh/uv/issues/21133">#21133</a>)</li>
<li><a
href="https://github.com/astral-sh/uv/commit/3a76e496e36783371a6d91a8f4834478964d36ca"><code>3a76e49</code></a>
Get rid of <code>Lock::with_manifest</code>, make
<code>Lock::from_resolution</code> take the manif...</li>
<li><a
href="https://github.com/astral-sh/uv/commit/7e6caa45f62a0506fea1b8ecfa0558e5d62d520c"><code>7e6caa4</code></a>
Support referencing indexes by name via <code>--index</code> and
<code>--default-index</code> (<a
href="https://redirect.github.com/astral-sh/uv/issues/17455">#17455</a>)</li>
<li><a
href="https://github.com/astral-sh/uv/commit/298dda41bff49bbd0a2f397e1662ab09b9cd148e"><code>298dda4</code></a>
Fix relative indexes in PEP 723 scripts (<a
href="https://redirect.github.com/astral-sh/uv/issues/21097">#21097</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/astral-sh/uv/compare/0.11.3...0.12.5">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=astral-sh/uv&package-manager=docker&previous-version=0.11.3&new-version=0.12.5)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-24 01:57:00 -04:00
dependabot[bot] 0dc13c1ca4 build(deps): bump the gh-actions group with 5 updates (#1248)
Bumps the gh-actions group with 5 updates:

| Package | From | To |
| --- | --- | --- |
|
[docker/setup-buildx-action](https://github.com/docker/setup-buildx-action)
| `4.2.0` | `4.3.0` |
| [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) | `9.0.0`
| `10.0.1` |
| [github/codeql-action/init](https://github.com/github/codeql-action) |
`4.37.6` | `4.37.7` |
|
[github/codeql-action/autobuild](https://github.com/github/codeql-action)
| `4.37.6` | `4.37.7` |
|
[github/codeql-action/analyze](https://github.com/github/codeql-action)
| `4.37.6` | `4.37.7` |

Updates `docker/setup-buildx-action` from 4.2.0 to 4.3.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/docker/setup-buildx-action/releases">docker/setup-buildx-action's
releases</a>.</em></p>
<blockquote>
<h2>v4.3.0</h2>
<ul>
<li>Bump <code>@​docker/actions-toolkit</code> from 0.92.0 to 0.95.0 in
<a
href="https://redirect.github.com/docker/setup-buildx-action/pull/595">docker/setup-buildx-action#595</a></li>
<li>Bump brace-expansion from 1.1.13 to 1.1.18 in <a
href="https://redirect.github.com/docker/setup-buildx-action/pull/600">docker/setup-buildx-action#600</a></li>
<li>Bump js-yaml from 5.2.0 to 5.3.0 in <a
href="https://redirect.github.com/docker/setup-buildx-action/pull/585">docker/setup-buildx-action#585</a></li>
<li>Bump postcss from 8.5.10 to 8.5.25 in <a
href="https://redirect.github.com/docker/setup-buildx-action/pull/598">docker/setup-buildx-action#598</a></li>
<li>Bump undici from 6.27.0 to 6.28.0 in <a
href="https://redirect.github.com/docker/setup-buildx-action/pull/601">docker/setup-buildx-action#601</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/docker/setup-buildx-action/compare/v4.2.0...v4.3.0">https://github.com/docker/setup-buildx-action/compare/v4.2.0...v4.3.0</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/docker/setup-buildx-action/commit/37fe631027851001ddb9b187196cc803df7f5f0e"><code>37fe631</code></a>
Merge pull request <a
href="https://redirect.github.com/docker/setup-buildx-action/issues/595">#595</a>
from docker/dependabot/npm_and_yarn/docker/actions-to...</li>
<li><a
href="https://github.com/docker/setup-buildx-action/commit/b5c4f91922681cc7c58d15ab7838986951f09d19"><code>b5c4f91</code></a>
[dependabot skip] chore: update generated content</li>
<li><a
href="https://github.com/docker/setup-buildx-action/commit/3e93b637c6430ba8fa896fad44d3aa6821899d63"><code>3e93b63</code></a>
build(deps): bump <code>@​docker/actions-toolkit</code> from 0.92.0 to
0.95.0</li>
<li><a
href="https://github.com/docker/setup-buildx-action/commit/e527031b32c86649307d5d492506855f90470604"><code>e527031</code></a>
Merge pull request <a
href="https://redirect.github.com/docker/setup-buildx-action/issues/600">#600</a>
from docker/dependabot/npm_and_yarn/brace-expansion-1...</li>
<li><a
href="https://github.com/docker/setup-buildx-action/commit/c68814b33cb66f1f7538e546190d410ae557a640"><code>c68814b</code></a>
[dependabot skip] chore: update generated content</li>
<li><a
href="https://github.com/docker/setup-buildx-action/commit/3f891b01bd5012a434f582800366972569aa1886"><code>3f891b0</code></a>
build(deps): bump brace-expansion from 1.1.13 to 1.1.18</li>
<li><a
href="https://github.com/docker/setup-buildx-action/commit/787db26fcde8ddcabd49a81472318028f7113962"><code>787db26</code></a>
Merge pull request <a
href="https://redirect.github.com/docker/setup-buildx-action/issues/585">#585</a>
from docker/dependabot/npm_and_yarn/js-yaml-5.2.1</li>
<li><a
href="https://github.com/docker/setup-buildx-action/commit/f7793687c711790ca336bd4934f1b1bf5f778e17"><code>f779368</code></a>
[dependabot skip] chore: update generated content</li>
<li><a
href="https://github.com/docker/setup-buildx-action/commit/7d5e60413489a33d28077e11d71c668580cfaf8d"><code>7d5e604</code></a>
build(deps): bump js-yaml from 5.2.0 to 5.3.0</li>
<li><a
href="https://github.com/docker/setup-buildx-action/commit/292c2fb3837a12d3ac2d1e47bbc5c00712bad939"><code>292c2fb</code></a>
Merge pull request <a
href="https://redirect.github.com/docker/setup-buildx-action/issues/590">#590</a>
from docker/dependabot/github_actions/actions/setup-n...</li>
<li>Additional commits viewable in <a
href="https://github.com/docker/setup-buildx-action/compare/bb05f3f5519dd87d3ba754cc423b652a5edd6d2c...37fe631027851001ddb9b187196cc803df7f5f0e">compare
view</a></li>
</ul>
</details>
<br />

Updates `astral-sh/setup-uv` from 9.0.0 to 10.0.1
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/astral-sh/setup-uv/releases">astral-sh/setup-uv's
releases</a>.</em></p>
<blockquote>
<h2>v10.0.1 🌈 Tolerate transient manifest timeouts</h2>
<h2>Changes</h2>
<p>Thank you <a
href="https://github.com/arguile"><code>@​arguile</code></a>- for making
this action more resilient.</p>
<h2>🐛 Bug fixes</h2>
<ul>
<li>Tolerate transient manifest timeouts <a
href="https://github.com/arguile"><code>@​arguile</code></a>- (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/1016">#1016</a>)</li>
</ul>
<h2>🧰 Maintenance</h2>
<ul>
<li>chore: update known checksums for 0.12.4 @<a
href="https://github.com/apps/github-actions">github-actions[bot]</a>
(<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/1017">#1017</a>)</li>
</ul>
<h2>📚 Documentation</h2>
<ul>
<li>docs: update version references to v10.0.0 @<a
href="https://github.com/apps/github-actions">github-actions[bot]</a>
(<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/1014">#1014</a>)</li>
</ul>
<h2>v10.0.0 🌈 Disable automatic caching for sensitive events and new QOL
features</h2>
<h2>Changes</h2>
<p>Another breaking release, directly after v9.0.0 but we think the
added security justifies that.</p>
<h3>Extra security by default</h3>
<p>If you use the default <code>enable-cache: auto</code> this will now
<strong>DISABLE THE CACHE</strong> to protect against cache poisoning
for the following events:</p>
<ul>
<li><code>pull_request_target</code></li>
<li><code>workflow_run</code></li>
<li><code>release</code></li>
</ul>
<p>You can read the full reasoning in <a
href="https://redirect.github.com/astral-sh/setup-uv/issues/984">astral-sh/setup-uv#984</a></p>
<h3><code>version: latest-known</code></h3>
<pre lang="yaml"><code>- name: Install the latest version of uv known to
setup-uv
  uses: astral-sh/setup-uv@v10.0.0
  with:
    version: &quot;latest-known&quot;
</code></pre>
<p>This will now install the latest version with a checksum that is
known by this action. The <a
href="https://github.com/astral-sh/setup-uv/blob/4f6036f71cec78afb113b323f220c9185d983c12/src/download/checksum/known-checksums.ts">known
<code>uv</code> checksums</a> are automatically updated but will take a
release of this action to take effect. You won't be always using the
latest &amp; greatest but you will have an extra level of security.</p>
<h3>Read python version from <code>.tool-versions</code></h3>
<pre lang="yaml"><code>- name: Install uv based on the version defined
in .tool-versions and also set python
  uses: astral-sh/setup-uv@v10.0.0
  with:
    version-file: &quot;pyproject.toml&quot;
&lt;/tr&gt;&lt;/table&gt; 
</code></pre>
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/20cfd1bf945f4377ade1205e4dbc17946fc9a30d"><code>20cfd1b</code></a>
chore: update known checksums for 0.12.4 (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/1017">#1017</a>)</li>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/d73a0cab66a532d7afa440d9df4a67ea9fe65a30"><code>d73a0ca</code></a>
Tolerate transient manifest timeouts (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/1016">#1016</a>)</li>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/ae3b92d1bdb308a10adfe7b8f408e5cc8c30f3f6"><code>ae3b92d</code></a>
docs: update version references to v10.0.0 (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/1014">#1014</a>)</li>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/ae62891fec2bb8e7d6c99fc78c9fec3a63790f8d"><code>ae62891</code></a>
chore(deps): roll up Dependabot updates (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/1013">#1013</a>)</li>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/f9cdb47d487aee2be8925d1e57290177ad9e1ac2"><code>f9cdb47</code></a>
Reject paths in .tool-versions (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/1007">#1007</a>)</li>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/4f6036f71cec78afb113b323f220c9185d983c12"><code>4f6036f</code></a>
Require pull requests for Dependabot rollups (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/1005">#1005</a>)</li>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/8d6402c9b71205b2d8d0b82de531d8fed8430182"><code>8d6402c</code></a>
chore(deps): roll up Dependabot updates (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/1004">#1004</a>)</li>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/46f427bd47c794e99536b75ffaa9f27602425027"><code>46f427b</code></a>
Read Python version from .tool-versions (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/996">#996</a>)</li>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/8ed89c51143f65ea13eaba62db51dbb8ea52d0a3"><code>8ed89c5</code></a>
ci: pin Alpine container image (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/995">#995</a>)</li>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/8473c7fea42cdfd540f4b01317a17ac5f54126ae"><code>8473c7f</code></a>
chore(deps): roll up Dependabot updates (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/994">#994</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/astral-sh/setup-uv/compare/c771a70e6277c0a99b617c7a806ffedaca235ff9...20cfd1bf945f4377ade1205e4dbc17946fc9a30d">compare
view</a></li>
</ul>
</details>
<br />

Updates `github/codeql-action/init` from 4.37.6 to 4.37.7
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/github/codeql-action/releases">github/codeql-action/init's
releases</a>.</em></p>
<blockquote>
<h2>v4.37.7</h2>
<ul>
<li>Update default CodeQL bundle version to <a
href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.3">2.26.3</a>.
<a
href="https://redirect.github.com/github/codeql-action/pull/4085">#4085</a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/github/codeql-action/blob/main/CHANGELOG.md">github/codeql-action/init's
changelog</a>.</em></p>
<blockquote>
<h1>CodeQL Action Changelog</h1>
<p>See the <a
href="https://github.com/github/codeql-action/releases">releases
page</a> for the relevant changes to the CodeQL CLI and language
packs.</p>
<h2>[UNRELEASED]</h2>
<p>No user facing changes.</p>
<h2>4.37.7 - 13 Aug 2026</h2>
<ul>
<li>Update default CodeQL bundle version to <a
href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.3">2.26.3</a>.
<a
href="https://redirect.github.com/github/codeql-action/pull/4085">#4085</a></li>
</ul>
<h2>4.37.6 - 04 Aug 2026</h2>
<ul>
<li>Changed the default filepath for the new remote file address format
that was introduced in CodeQL Action 4.37.0 / 3.37.0 to
<code>.github/codeql-config.yml</code> to align it with the suggested
path that is used elsewhere. <a
href="https://redirect.github.com/github/codeql-action/pull/4070">#4070</a></li>
</ul>
<h2>4.37.5 - 03 Aug 2026</h2>
<ul>
<li>Fixed a bug where a network error while streaming the download of
the CodeQL bundle could terminate the <code>init</code> Action instead
of falling back to downloading the bundle before extracting it. <a
href="https://redirect.github.com/github/codeql-action/pull/4061">#4061</a></li>
</ul>
<h2>4.37.4 - 29 Jul 2026</h2>
<ul>
<li>This version of the CodeQL Action adds support for the
<code>tools</code> input for the <code>codeql-action/init</code> step to
be specified using a <code>github-codeql-tools</code> <a
href="https://docs.github.com/en/organizations/managing-organization-settings/managing-custom-properties-for-repositories-in-your-organization">repository
property</a>. This feature will gradually be rolled out following the
release of this version. Once rolled out, this allows for the CodeQL CLI
version that is used in GitHub-managed workflows, such as Default Setup,
to be set to a custom value. For example, customers who run into issues
with rate limits when a new CodeQL CLI version is released can set the
value to <code>toolcache</code> to always use the CodeQL CLI version
that is available in the runner toolcache. For Advanced Setup workflows,
the value provided for <code>tools</code> in the workflow definition
always takes precedence unless the value of the repository property
starts with <code>!</code>. <a
href="https://redirect.github.com/github/codeql-action/pull/4037">#4037</a></li>
<li>Update default CodeQL bundle version to <a
href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.2">2.26.2</a>.
<a
href="https://redirect.github.com/github/codeql-action/pull/4051">#4051</a></li>
</ul>
<h2>4.37.3 - 22 Jul 2026</h2>
<p>No user facing changes.</p>
<h2>4.37.2 - 21 Jul 2026</h2>
<ul>
<li>The new address format for the <code>config-file</code> input that
was introduced in CodeQL Action 4.37.0 is now enabled by default. In
addition to the format described there, the <code>remote=</code> prefix
can now be used to explicitly indicate that the input refers to a remote
file. All previous input formats continue to be accepted as well. <a
href="https://redirect.github.com/github/codeql-action/pull/4023">#4023</a></li>
<li>The CodeQL Action can now make use of <a
href="https://docs.github.com/en/code-security/how-tos/secure-at-scale/configure-organization-security/manage-usage-and-access/giving-org-access-private-registries">configured
private registries</a> in Default Setup to retrieve CodeQL configuration
files from remote repositories that require authentication. This will
allow customers to store their CodeQL configuration in a single
repository that can then be referenced by Default Setup workflows in
other repositories. We expect to roll this and other, related changes
out to everyone in July. <a
href="https://redirect.github.com/github/codeql-action/pull/4007">#4007</a></li>
</ul>
<h2>4.37.1 - 16 Jul 2026</h2>
<ul>
<li><em>Upcoming breaking change</em>: Add a deprecation warning for
customers using CodeQL version 2.20.6 and earlier. These versions of
CodeQL were discontinued on 1 July 2026 alongside GitHub Enterprise
Server 3.16, and will be unsupported by the next minor release of the
CodeQL Action. <a
href="https://redirect.github.com/github/codeql-action/pull/3956">#3956</a></li>
<li>Update default CodeQL bundle version to <a
href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.1">2.26.1</a>.
<a
href="https://redirect.github.com/github/codeql-action/pull/4019">#4019</a></li>
</ul>
<h2>4.37.0 - 08 Jul 2026</h2>
<ul>
<li>Update default CodeQL bundle version to <a
href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.0">2.26.0</a>.
<a
href="https://redirect.github.com/github/codeql-action/pull/3995">#3995</a></li>
<li>In addition to the existing input format, the
<code>config-file</code> input for the <code>codeql-action/init</code>
step will soon support a new <code>[owner/]repo[@ref][:path]</code>
format. All components except the repository name are optional. If
omitted, <code>owner</code> defaults to the same owner as the repository
the analysis is running for, <code>ref</code> to <code>main</code>, and
<code>path</code> to <code>.github/codeql-action.yaml</code>. Support
for this format ships in this version of the CodeQL Action, but will
only be enabled over the coming weeks. <a
href="https://redirect.github.com/github/codeql-action/pull/3973">#3973</a></li>
</ul>
<h2>4.36.3 - 01 Jul 2026</h2>
<p>No user facing changes.</p>
<h2>4.36.2 - 04 Jun 2026</h2>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/github/codeql-action/commit/ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd"><code>ff2f1c6</code></a>
Merge pull request <a
href="https://redirect.github.com/github/codeql-action/issues/4093">#4093</a>
from github/update-v4.37.7-be7a3dbb8</li>
<li><a
href="https://github.com/github/codeql-action/commit/951a133f96aa2114dd747e9e437305335d0bde16"><code>951a133</code></a>
Update changelog for v4.37.7</li>
<li><a
href="https://github.com/github/codeql-action/commit/be7a3dbb8147b82cd6d27e0707105b36aa190fc1"><code>be7a3db</code></a>
Merge pull request <a
href="https://redirect.github.com/github/codeql-action/issues/4087">#4087</a>
from github/dependabot/npm_and_yarn/npm-minor-0aa561...</li>
<li><a
href="https://github.com/github/codeql-action/commit/9310334b11405b305d9444edfa56cd86e2f1e4fe"><code>9310334</code></a>
Merge pull request <a
href="https://redirect.github.com/github/codeql-action/issues/4086">#4086</a>
from github/mbg/thread-action-state-to-codeql</li>
<li><a
href="https://github.com/github/codeql-action/commit/b4d8a54218a8792de9af2f6f32e33af899ca5212"><code>b4d8a54</code></a>
Rebuild</li>
<li><a
href="https://github.com/github/codeql-action/commit/ab5db2519c3344f2fa61c711fa2d6ad135829200"><code>ab5db25</code></a>
Bump the npm-minor group across 1 directory with 8 updates</li>
<li><a
href="https://github.com/github/codeql-action/commit/38055a3c3cf3979323eaf70fc6c73a8690250bde"><code>38055a3</code></a>
Drop <code>logger</code> from <code>databaseInitCluster</code> in
interface</li>
<li><a
href="https://github.com/github/codeql-action/commit/1f87aed5e66849f0c43ae147377cc77f2d98ac99"><code>1f87aed</code></a>
Merge pull request <a
href="https://redirect.github.com/github/codeql-action/issues/4085">#4085</a>
from github/update-bundle/codeql-bundle-v2.26.3</li>
<li><a
href="https://github.com/github/codeql-action/commit/dc1b98ad1c2f13ccf9fc33fb82f32fc76f944253"><code>dc1b98a</code></a>
Make <code>logger</code> available to <code>getCodeQLForCmd</code></li>
<li><a
href="https://github.com/github/codeql-action/commit/6f0220ee37121218af472efbde25f06907a4da4f"><code>6f0220e</code></a>
Merge pull request <a
href="https://redirect.github.com/github/codeql-action/issues/4084">#4084</a>
from github/navntoft/bump-undici</li>
<li>Additional commits viewable in <a
href="https://github.com/github/codeql-action/compare/5595ccaf912efad79be6eef63a5619ff05969be3...ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd">compare
view</a></li>
</ul>
</details>
<br />

Updates `github/codeql-action/autobuild` from 4.37.6 to 4.37.7
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/github/codeql-action/releases">github/codeql-action/autobuild's
releases</a>.</em></p>
<blockquote>
<h2>v4.37.7</h2>
<ul>
<li>Update default CodeQL bundle version to <a
href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.3">2.26.3</a>.
<a
href="https://redirect.github.com/github/codeql-action/pull/4085">#4085</a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/github/codeql-action/blob/main/CHANGELOG.md">github/codeql-action/autobuild's
changelog</a>.</em></p>
<blockquote>
<h1>CodeQL Action Changelog</h1>
<p>See the <a
href="https://github.com/github/codeql-action/releases">releases
page</a> for the relevant changes to the CodeQL CLI and language
packs.</p>
<h2>[UNRELEASED]</h2>
<p>No user facing changes.</p>
<h2>4.37.7 - 13 Aug 2026</h2>
<ul>
<li>Update default CodeQL bundle version to <a
href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.3">2.26.3</a>.
<a
href="https://redirect.github.com/github/codeql-action/pull/4085">#4085</a></li>
</ul>
<h2>4.37.6 - 04 Aug 2026</h2>
<ul>
<li>Changed the default filepath for the new remote file address format
that was introduced in CodeQL Action 4.37.0 / 3.37.0 to
<code>.github/codeql-config.yml</code> to align it with the suggested
path that is used elsewhere. <a
href="https://redirect.github.com/github/codeql-action/pull/4070">#4070</a></li>
</ul>
<h2>4.37.5 - 03 Aug 2026</h2>
<ul>
<li>Fixed a bug where a network error while streaming the download of
the CodeQL bundle could terminate the <code>init</code> Action instead
of falling back to downloading the bundle before extracting it. <a
href="https://redirect.github.com/github/codeql-action/pull/4061">#4061</a></li>
</ul>
<h2>4.37.4 - 29 Jul 2026</h2>
<ul>
<li>This version of the CodeQL Action adds support for the
<code>tools</code> input for the <code>codeql-action/init</code> step to
be specified using a <code>github-codeql-tools</code> <a
href="https://docs.github.com/en/organizations/managing-organization-settings/managing-custom-properties-for-repositories-in-your-organization">repository
property</a>. This feature will gradually be rolled out following the
release of this version. Once rolled out, this allows for the CodeQL CLI
version that is used in GitHub-managed workflows, such as Default Setup,
to be set to a custom value. For example, customers who run into issues
with rate limits when a new CodeQL CLI version is released can set the
value to <code>toolcache</code> to always use the CodeQL CLI version
that is available in the runner toolcache. For Advanced Setup workflows,
the value provided for <code>tools</code> in the workflow definition
always takes precedence unless the value of the repository property
starts with <code>!</code>. <a
href="https://redirect.github.com/github/codeql-action/pull/4037">#4037</a></li>
<li>Update default CodeQL bundle version to <a
href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.2">2.26.2</a>.
<a
href="https://redirect.github.com/github/codeql-action/pull/4051">#4051</a></li>
</ul>
<h2>4.37.3 - 22 Jul 2026</h2>
<p>No user facing changes.</p>
<h2>4.37.2 - 21 Jul 2026</h2>
<ul>
<li>The new address format for the <code>config-file</code> input that
was introduced in CodeQL Action 4.37.0 is now enabled by default. In
addition to the format described there, the <code>remote=</code> prefix
can now be used to explicitly indicate that the input refers to a remote
file. All previous input formats continue to be accepted as well. <a
href="https://redirect.github.com/github/codeql-action/pull/4023">#4023</a></li>
<li>The CodeQL Action can now make use of <a
href="https://docs.github.com/en/code-security/how-tos/secure-at-scale/configure-organization-security/manage-usage-and-access/giving-org-access-private-registries">configured
private registries</a> in Default Setup to retrieve CodeQL configuration
files from remote repositories that require authentication. This will
allow customers to store their CodeQL configuration in a single
repository that can then be referenced by Default Setup workflows in
other repositories. We expect to roll this and other, related changes
out to everyone in July. <a
href="https://redirect.github.com/github/codeql-action/pull/4007">#4007</a></li>
</ul>
<h2>4.37.1 - 16 Jul 2026</h2>
<ul>
<li><em>Upcoming breaking change</em>: Add a deprecation warning for
customers using CodeQL version 2.20.6 and earlier. These versions of
CodeQL were discontinued on 1 July 2026 alongside GitHub Enterprise
Server 3.16, and will be unsupported by the next minor release of the
CodeQL Action. <a
href="https://redirect.github.com/github/codeql-action/pull/3956">#3956</a></li>
<li>Update default CodeQL bundle version to <a
href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.1">2.26.1</a>.
<a
href="https://redirect.github.com/github/codeql-action/pull/4019">#4019</a></li>
</ul>
<h2>4.37.0 - 08 Jul 2026</h2>
<ul>
<li>Update default CodeQL bundle version to <a
href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.0">2.26.0</a>.
<a
href="https://redirect.github.com/github/codeql-action/pull/3995">#3995</a></li>
<li>In addition to the existing input format, the
<code>config-file</code> input for the <code>codeql-action/init</code>
step will soon support a new <code>[owner/]repo[@ref][:path]</code>
format. All components except the repository name are optional. If
omitted, <code>owner</code> defaults to the same owner as the repository
the analysis is running for, <code>ref</code> to <code>main</code>, and
<code>path</code> to <code>.github/codeql-action.yaml</code>. Support
for this format ships in this version of the CodeQL Action, but will
only be enabled over the coming weeks. <a
href="https://redirect.github.com/github/codeql-action/pull/3973">#3973</a></li>
</ul>
<h2>4.36.3 - 01 Jul 2026</h2>
<p>No user facing changes.</p>
<h2>4.36.2 - 04 Jun 2026</h2>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/github/codeql-action/commit/ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd"><code>ff2f1c6</code></a>
Merge pull request <a
href="https://redirect.github.com/github/codeql-action/issues/4093">#4093</a>
from github/update-v4.37.7-be7a3dbb8</li>
<li><a
href="https://github.com/github/codeql-action/commit/951a133f96aa2114dd747e9e437305335d0bde16"><code>951a133</code></a>
Update changelog for v4.37.7</li>
<li><a
href="https://github.com/github/codeql-action/commit/be7a3dbb8147b82cd6d27e0707105b36aa190fc1"><code>be7a3db</code></a>
Merge pull request <a
href="https://redirect.github.com/github/codeql-action/issues/4087">#4087</a>
from github/dependabot/npm_and_yarn/npm-minor-0aa561...</li>
<li><a
href="https://github.com/github/codeql-action/commit/9310334b11405b305d9444edfa56cd86e2f1e4fe"><code>9310334</code></a>
Merge pull request <a
href="https://redirect.github.com/github/codeql-action/issues/4086">#4086</a>
from github/mbg/thread-action-state-to-codeql</li>
<li><a
href="https://github.com/github/codeql-action/commit/b4d8a54218a8792de9af2f6f32e33af899ca5212"><code>b4d8a54</code></a>
Rebuild</li>
<li><a
href="https://github.com/github/codeql-action/commit/ab5db2519c3344f2fa61c711fa2d6ad135829200"><code>ab5db25</code></a>
Bump the npm-minor group across 1 directory with 8 updates</li>
<li><a
href="https://github.com/github/codeql-action/commit/38055a3c3cf3979323eaf70fc6c73a8690250bde"><code>38055a3</code></a>
Drop <code>logger</code> from <code>databaseInitCluster</code> in
interface</li>
<li><a
href="https://github.com/github/codeql-action/commit/1f87aed5e66849f0c43ae147377cc77f2d98ac99"><code>1f87aed</code></a>
Merge pull request <a
href="https://redirect.github.com/github/codeql-action/issues/4085">#4085</a>
from github/update-bundle/codeql-bundle-v2.26.3</li>
<li><a
href="https://github.com/github/codeql-action/commit/dc1b98ad1c2f13ccf9fc33fb82f32fc76f944253"><code>dc1b98a</code></a>
Make <code>logger</code> available to <code>getCodeQLForCmd</code></li>
<li><a
href="https://github.com/github/codeql-action/commit/6f0220ee37121218af472efbde25f06907a4da4f"><code>6f0220e</code></a>
Merge pull request <a
href="https://redirect.github.com/github/codeql-action/issues/4084">#4084</a>
from github/navntoft/bump-undici</li>
<li>Additional commits viewable in <a
href="https://github.com/github/codeql-action/compare/5595ccaf912efad79be6eef63a5619ff05969be3...ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd">compare
view</a></li>
</ul>
</details>
<br />

Updates `github/codeql-action/analyze` from 4.37.6 to 4.37.7
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/github/codeql-action/releases">github/codeql-action/analyze's
releases</a>.</em></p>
<blockquote>
<h2>v4.37.7</h2>
<ul>
<li>Update default CodeQL bundle version to <a
href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.3">2.26.3</a>.
<a
href="https://redirect.github.com/github/codeql-action/pull/4085">#4085</a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/github/codeql-action/blob/main/CHANGELOG.md">github/codeql-action/analyze's
changelog</a>.</em></p>
<blockquote>
<h1>CodeQL Action Changelog</h1>
<p>See the <a
href="https://github.com/github/codeql-action/releases">releases
page</a> for the relevant changes to the CodeQL CLI and language
packs.</p>
<h2>[UNRELEASED]</h2>
<p>No user facing changes.</p>
<h2>4.37.7 - 13 Aug 2026</h2>
<ul>
<li>Update default CodeQL bundle version to <a
href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.3">2.26.3</a>.
<a
href="https://redirect.github.com/github/codeql-action/pull/4085">#4085</a></li>
</ul>
<h2>4.37.6 - 04 Aug 2026</h2>
<ul>
<li>Changed the default filepath for the new remote file address format
that was introduced in CodeQL Action 4.37.0 / 3.37.0 to
<code>.github/codeql-config.yml</code> to align it with the suggested
path that is used elsewhere. <a
href="https://redirect.github.com/github/codeql-action/pull/4070">#4070</a></li>
</ul>
<h2>4.37.5 - 03 Aug 2026</h2>
<ul>
<li>Fixed a bug where a network error while streaming the download of
the CodeQL bundle could terminate the <code>init</code> Action instead
of falling back to downloading the bundle before extracting it. <a
href="https://redirect.github.com/github/codeql-action/pull/4061">#4061</a></li>
</ul>
<h2>4.37.4 - 29 Jul 2026</h2>
<ul>
<li>This version of the CodeQL Action adds support for the
<code>tools</code> input for the <code>codeql-action/init</code> step to
be specified using a <code>github-codeql-tools</code> <a
href="https://docs.github.com/en/organizations/managing-organization-settings/managing-custom-properties-for-repositories-in-your-organization">repository
property</a>. This feature will gradually be rolled out following the
release of this version. Once rolled out, this allows for the CodeQL CLI
version that is used in GitHub-managed workflows, such as Default Setup,
to be set to a custom value. For example, customers who run into issues
with rate limits when a new CodeQL CLI version is released can set the
value to <code>toolcache</code> to always use the CodeQL CLI version
that is available in the runner toolcache. For Advanced Setup workflows,
the value provided for <code>tools</code> in the workflow definition
always takes precedence unless the value of the repository property
starts with <code>!</code>. <a
href="https://redirect.github.com/github/codeql-action/pull/4037">#4037</a></li>
<li>Update default CodeQL bundle version to <a
href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.2">2.26.2</a>.
<a
href="https://redirect.github.com/github/codeql-action/pull/4051">#4051</a></li>
</ul>
<h2>4.37.3 - 22 Jul 2026</h2>
<p>No user facing changes.</p>
<h2>4.37.2 - 21 Jul 2026</h2>
<ul>
<li>The new address format for the <code>config-file</code> input that
was introduced in CodeQL Action 4.37.0 is now enabled by default. In
addition to the format described there, the <code>remote=</code> prefix
can now be used to explicitly indicate that the input refers to a remote
file. All previous input formats continue to be accepted as well. <a
href="https://redirect.github.com/github/codeql-action/pull/4023">#4023</a></li>
<li>The CodeQL Action can now make use of <a
href="https://docs.github.com/en/code-security/how-tos/secure-at-scale/configure-organization-security/manage-usage-and-access/giving-org-access-private-registries">configured
private registries</a> in Default Setup to retrieve CodeQL configuration
files from remote repositories that require authentication. This will
allow customers to store their CodeQL configuration in a single
repository that can then be referenced by Default Setup workflows in
other repositories. We expect to roll this and other, related changes
out to everyone in July. <a
href="https://redirect.github.com/github/codeql-action/pull/4007">#4007</a></li>
</ul>
<h2>4.37.1 - 16 Jul 2026</h2>
<ul>
<li><em>Upcoming breaking change</em>: Add a deprecation warning for
customers using CodeQL version 2.20.6 and earlier. These versions of
CodeQL were discontinued on 1 July 2026 alongside GitHub Enterprise
Server 3.16, and will be unsupported by the next minor release of the
CodeQL Action. <a
href="https://redirect.github.com/github/codeql-action/pull/3956">#3956</a></li>
<li>Update default CodeQL bundle version to <a
href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.1">2.26.1</a>.
<a
href="https://redirect.github.com/github/codeql-action/pull/4019">#4019</a></li>
</ul>
<h2>4.37.0 - 08 Jul 2026</h2>
<ul>
<li>Update default CodeQL bundle version to <a
href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.26.0">2.26.0</a>.
<a
href="https://redirect.github.com/github/codeql-action/pull/3995">#3995</a></li>
<li>In addition to the existing input format, the
<code>config-file</code> input for the <code>codeql-action/init</code>
step will soon support a new <code>[owner/]repo[@ref][:path]</code>
format. All components except the repository name are optional. If
omitted, <code>owner</code> defaults to the same owner as the repository
the analysis is running for, <code>ref</code> to <code>main</code>, and
<code>path</code> to <code>.github/codeql-action.yaml</code>. Support
for this format ships in this version of the CodeQL Action, but will
only be enabled over the coming weeks. <a
href="https://redirect.github.com/github/codeql-action/pull/3973">#3973</a></li>
</ul>
<h2>4.36.3 - 01 Jul 2026</h2>
<p>No user facing changes.</p>
<h2>4.36.2 - 04 Jun 2026</h2>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/github/codeql-action/commit/ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd"><code>ff2f1c6</code></a>
Merge pull request <a
href="https://redirect.github.com/github/codeql-action/issues/4093">#4093</a>
from github/update-v4.37.7-be7a3dbb8</li>
<li><a
href="https://github.com/github/codeql-action/commit/951a133f96aa2114dd747e9e437305335d0bde16"><code>951a133</code></a>
Update changelog for v4.37.7</li>
<li><a
href="https://github.com/github/codeql-action/commit/be7a3dbb8147b82cd6d27e0707105b36aa190fc1"><code>be7a3db</code></a>
Merge pull request <a
href="https://redirect.github.com/github/codeql-action/issues/4087">#4087</a>
from github/dependabot/npm_and_yarn/npm-minor-0aa561...</li>
<li><a
href="https://github.com/github/codeql-action/commit/9310334b11405b305d9444edfa56cd86e2f1e4fe"><code>9310334</code></a>
Merge pull request <a
href="https://redirect.github.com/github/codeql-action/issues/4086">#4086</a>
from github/mbg/thread-action-state-to-codeql</li>
<li><a
href="https://github.com/github/codeql-action/commit/b4d8a54218a8792de9af2f6f32e33af899ca5212"><code>b4d8a54</code></a>
Rebuild</li>
<li><a
href="https://github.com/github/codeql-action/commit/ab5db2519c3344f2fa61c711fa2d6ad135829200"><code>ab5db25</code></a>
Bump the npm-minor group across 1 directory with 8 updates</li>
<li><a
href="https://github.com/github/codeql-action/commit/38055a3c3cf3979323eaf70fc6c73a8690250bde"><code>38055a3</code></a>
Drop <code>logger</code> from <code>databaseInitCluster</code> in
interface</li>
<li><a
href="https://github.com/github/codeql-action/commit/1f87aed5e66849f0c43ae147377cc77f2d98ac99"><code>1f87aed</code></a>
Merge pull request <a
href="https://redirect.github.com/github/codeql-action/issues/4085">#4085</a>
from github/update-bundle/codeql-bundle-v2.26.3</li>
<li><a
href="https://github.com/github/codeql-action/commit/dc1b98ad1c2f13ccf9fc33fb82f32fc76f944253"><code>dc1b98a</code></a>
Make <code>logger</code> available to <code>getCodeQLForCmd</code></li>
<li><a
href="https://github.com/github/codeql-action/commit/6f0220ee37121218af472efbde25f06907a4da4f"><code>6f0220e</code></a>
Merge pull request <a
href="https://redirect.github.com/github/codeql-action/issues/4084">#4084</a>
from github/navntoft/bump-undici</li>
<li>Additional commits viewable in <a
href="https://github.com/github/codeql-action/compare/5595ccaf912efad79be6eef63a5619ff05969be3...ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd">compare
view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-24 01:56:52 -04:00
dependabot[bot] d1f8527089 build(deps): bump the python-deps group across 1 directory with 5 updates (#1260)
Bumps the python-deps group with 5 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [gunicorn](https://github.com/benoitc/gunicorn) | `26.0.0` | `26.1.0`
|
| [apprise](https://github.com/caronc/apprise) | `1.12.0` | `1.13.0` |
| [seleniumbase](https://github.com/seleniumbase/SeleniumBase) |
`4.51.12` | `4.52.1` |
| [prek](https://github.com/j178/prek) | `0.4.13` | `0.4.14` |
| [ruff](https://github.com/astral-sh/ruff) | `0.16.3` | `0.16.4` |


Updates `gunicorn` from 26.0.0 to 26.1.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/benoitc/gunicorn/releases">gunicorn's
releases</a>.</em></p>
<blockquote>
<h2>gunicorn 26.1.0</h2>
<h3>New Features</h3>
<ul>
<li><strong>Glob patterns in <code>reload_extra_files</code></strong>:
entries containing <code>*</code>, <code>?</code> or <code>[</code>
are treated as patterns, so <code>ui/*/config.json</code> watches every
view's config
without listing them one by one. Patterns are re-expanded on every
reload
check rather than once at startup, so a file created later starts being
watched without restarting gunicorn, and <code>**</code> recurses. A
pattern matching
nothing warns instead of failing, since with live expansion it may match
later
(<a
href="https://redirect.github.com/benoitc/gunicorn/issues/1643">#1643</a>,
<a
href="https://redirect.github.com/benoitc/gunicorn/pull/3662">#3662</a>).</li>
</ul>
<h3>Security</h3>
<ul>
<li><strong>Dependency floors raised past known advisories</strong>:
every declared floor was
checked against the advisory database. <code>tornado</code>,
<code>h2</code>, <code>setuptools</code> and
<code>pymdown-extensions</code> permitted vulnerable versions and now
require the first
clean release; <code>pytest</code> and <code>httpx</code> were unpinned
and now carry floors. The
<code>tornado</code> example pinned <code>tornado&lt;6</code>, which was
both the source of several
advisories and older than the <code>&gt;=6.5.0</code> the tornado worker
needs, so the
example could not run as pinned.</li>
</ul>
<h3>Bug Fixes</h3>
<ul>
<li>
<p><strong>SIGHUP did not reload the logger configuration</strong>:
<code>Arbiter.reload()</code>
re-read the configuration file but kept using the logger built at
startup,
calling only <code>reopen_files()</code> on its existing handlers.
Changes to
<code>logconfig</code>, <code>logconfig_dict</code>,
<code>logconfig_json</code> and <code>loglevel</code> were ignored
until a full restart, which in containers meant replacing the pod. The
existing logger now re-runs its setup on reload, so new handlers,
formats
and levels take effect while the process identity and its listeners are
preserved, and re-running the setup no longer stacks duplicate syslog
handlers. An invalid log configuration on reload is not fatal either:
the
error is reported on stderr, the previous working configuration is
restored
and the master keeps running with it
(<a
href="https://redirect.github.com/benoitc/gunicorn/issues/3353">#3353</a>).</p>
</li>
<li>
<p><strong>Truncated chunked bodies accepted</strong>: RFC 9112 section
7.1.2 ends a chunked
body with <code>0 CRLF CRLF</code>, the second CRLF being the mandatory
empty trailer
section. <code>ChunkedReader.parse_chunk_size()</code> swallowed the
<code>NoMoreData</code> raised
while scanning for it, so a body cut short right after the last chunk
line was
treated as complete instead of rejected. It now raises
<code>ChunkMissingTerminator</code>
(<a
href="https://redirect.github.com/benoitc/gunicorn/issues/3382">#3382</a>,
<a
href="https://redirect.github.com/benoitc/gunicorn/pull/3685">#3685</a>).</p>
</li>
<li>
<p><strong><code>--spew</code> crashed on dynamically generated
code</strong>: the trace hook indexed the
2-tuple returned by <code>inspect.getsourcelines()</code> by line number
rather than
indexing the list of lines, so a frame with no <code>__file__</code>
raised
<code>AttributeError: 'int' object has no attribute 'rstrip'</code> on
line 1 and</p>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/benoitc/gunicorn/commit/71b59a75820dd4a762dc42a3280124168b4e44a8"><code>71b59a7</code></a>
Merge pull request <a
href="https://redirect.github.com/benoitc/gunicorn/issues/3698">#3698</a>
from benoitc/fix/docker-health-check-readerror</li>
<li><a
href="https://github.com/benoitc/gunicorn/commit/48287de8d8360825c85ac981ddfe9c80dba3b418"><code>48287de</code></a>
test: catch every transport error in the docker health check</li>
<li><a
href="https://github.com/benoitc/gunicorn/commit/3110e8c37f716ca0ed63ca7f80c7c9ea0fbbdd50"><code>3110e8c</code></a>
Merge pull request <a
href="https://redirect.github.com/benoitc/gunicorn/issues/3696">#3696</a>
from benoitc/docs/roadmap</li>
<li><a
href="https://github.com/benoitc/gunicorn/commit/cc56c410b7103c1f2b877279f81cb3a14705315a"><code>cc56c41</code></a>
Merge pull request <a
href="https://redirect.github.com/benoitc/gunicorn/issues/3693">#3693</a>
from benoitc/release/26.1.0</li>
<li><a
href="https://github.com/benoitc/gunicorn/commit/5cf1f1651a40fa36afe13fde623e3437ca872463"><code>5cf1f16</code></a>
docs: surface the roadmap on the site home page</li>
<li><a
href="https://github.com/benoitc/gunicorn/commit/7e35f72d135056c37da321becf22b19aeec06937"><code>7e35f72</code></a>
docs: add FastCGI to the roadmap and point items at Ideas</li>
<li><a
href="https://github.com/benoitc/gunicorn/commit/18ddc586c9b9513b25e92c6ed6f818f66ff3abcd"><code>18ddc58</code></a>
docs: drop the framework and reverse-proxy non-goals from the
roadmap</li>
<li><a
href="https://github.com/benoitc/gunicorn/commit/1ecae56ebd096ee4f94d12c59a36f240ce348572"><code>1ecae56</code></a>
docs: add a roadmap and make the chat easy to find</li>
<li><a
href="https://github.com/benoitc/gunicorn/commit/ca412e3f7134bd9f0e91851778ae6d8dd222cf41"><code>ca412e3</code></a>
docs: sync the Latest changelog page with 26.1.0</li>
<li><a
href="https://github.com/benoitc/gunicorn/commit/640936fb29f7bdf5665d5b7f919783bb73e2f981"><code>640936f</code></a>
docs: note the dependency security work in 26.1.0</li>
<li>Additional commits viewable in <a
href="https://github.com/benoitc/gunicorn/compare/26.0.0...26.1.0">compare
view</a></li>
</ul>
</details>
<br />

Updates `apprise` from 1.12.0 to 1.13.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/caronc/apprise/releases">apprise's
releases</a>.</em></p>
<blockquote>
<h2>Release v1.13.0</h2>
<h2>What's Changed</h2>
<p>This update clears out our backlog of resolved bugs and introduces a
few new services. While it’s a standard maintenance release, we want to
give a massive shoutout to our community. The level of support we've
received for this update is truly incredible; there were more
commits/PRs from you than there were from me and that has never happend
before 🙂 . Your contributions are deeply
appreciated! Seriously!</p>
<p><strong>Note:</strong> ⚠️ <strong>This will be the last version of
Apprise v1.x</strong>. Future releases will be under Apprise v2.x. This
means: Developers, <em>please</em> make sure you pin your
<code>projects.toml</code> or <code>requirements.txt</code> to
<code>apprise &gt;=1.0.0, &lt;2.0.0</code>. v2.x will be awesome, but it
will introduce breaking changes your software may need to adapt to. I
will still continue supporting the v1.x branch should security issues
arise for at year or two, but eventually v2.x would be the way to
go.</p>
<h3>📣 New Notification Services:</h3>
<ul>
<li>Added Pinglet Support by <a
href="https://github.com/TheGlenn88"><code>@​TheGlenn88</code></a> in <a
href="https://redirect.github.com/caronc/apprise/pull/1680">caronc/apprise#1680</a></li>
<li>Add Trigv notification support by <a
href="https://github.com/hchouhan"><code>@​hchouhan</code></a> in <a
href="https://redirect.github.com/caronc/apprise/pull/1671">caronc/apprise#1671</a></li>
<li>Added Pingram (formerly NotificationApi) by <a
href="https://github.com/lipusal"><code>@​lipusal</code></a> in <a
href="https://redirect.github.com/caronc/apprise/pull/1665">caronc/apprise#1665</a></li>
</ul>
<h3>🐞  Bugfixes</h3>
<ul>
<li>Preserve x/y coordinate of 0 in dbus/glib url() (round-trip crash on
x=0) by <a href="https://github.com/gaoflow"><code>@​gaoflow</code></a>
in <a
href="https://redirect.github.com/caronc/apprise/pull/1663">caronc/apprise#1663</a></li>
<li>Allow RFC 3986 path characters that are safe to leave unencoded by
<a href="https://github.com/Sanjays2402"><code>@​Sanjays2402</code></a>
in <a
href="https://redirect.github.com/caronc/apprise/pull/1673">caronc/apprise#1673</a></li>
<li>small bugfix in splunk entity_id handling on url in <a
href="https://redirect.github.com/caronc/apprise/pull/1688">caronc/apprise#1688</a></li>
<li>Fix Microsoft Workflows CU-routed webhooks by <a
href="https://github.com/Sanjays2402"><code>@​Sanjays2402</code></a> in
<a
href="https://redirect.github.com/caronc/apprise/pull/1676">caronc/apprise#1676</a></li>
<li>Return None from parse_url() on malformed authority content (<a
href="https://redirect.github.com/caronc/apprise/issues/1693">#1693</a>)
by <a href="https://github.com/youdie006"><code>@​youdie006</code></a>
in <a
href="https://redirect.github.com/caronc/apprise/pull/1694">caronc/apprise#1694</a></li>
<li>fix issue causing in memory email attachments to fail by <a
href="https://github.com/kevinfeyrer"><code>@​kevinfeyrer</code></a> in
<a
href="https://redirect.github.com/caronc/apprise/pull/1696">caronc/apprise#1696</a></li>
</ul>
<h3>💡 Features</h3>
<ul>
<li>Improved handling matrix message splitting in <a
href="https://redirect.github.com/caronc/apprise/pull/1687">caronc/apprise#1687</a></li>
<li>feat(bark): add AES-GCM encryption by <a
href="https://github.com/IceCodeNew"><code>@​IceCodeNew</code></a> in <a
href="https://redirect.github.com/caronc/apprise/pull/1684">caronc/apprise#1684</a></li>
<li>fix(docs): CLI file attachments feature by <a
href="https://github.com/egvimo"><code>@​egvimo</code></a> in <a
href="https://redirect.github.com/caronc/apprise/pull/1678">caronc/apprise#1678</a></li>
<li>Support for Telegram rich message support in <a
href="https://redirect.github.com/caronc/apprise/pull/1690">caronc/apprise#1690</a></li>
</ul>
<h3>❤️ Life-Cycle Support</h3>
<ul>
<li>Fix grammar and tiny errors in README.md by <a
href="https://github.com/notrudyyy"><code>@​notrudyyy</code></a> in <a
href="https://redirect.github.com/caronc/apprise/pull/1674">caronc/apprise#1674</a></li>
<li>docs: quote pip extras install examples by <a
href="https://github.com/nyxst4ck"><code>@​nyxst4ck</code></a> in <a
href="https://redirect.github.com/caronc/apprise/pull/1679">caronc/apprise#1679</a></li>
<li>Decomissioned legacy service NotificationApi by <a
href="https://github.com/lipusal"><code>@​lipusal</code></a> in <a
href="https://redirect.github.com/caronc/apprise/pull/1665">caronc/apprise#1665</a></li>
<li>Fixed warnings emitted while running test suite in <a
href="https://redirect.github.com/caronc/apprise/pull/1689">caronc/apprise#1689</a></li>
<li>Proxy documentation added to man page in <a
href="https://redirect.github.com/caronc/apprise/pull/1691">caronc/apprise#1691</a></li>
<li>Updated github-action references to latest in <a
href="https://redirect.github.com/caronc/apprise/pull/1692">caronc/apprise#1692</a></li>
<li>Add Italian translation by <a
href="https://github.com/albanobattistella"><code>@​albanobattistella</code></a>
in <a
href="https://redirect.github.com/caronc/apprise/pull/1670">caronc/apprise#1670</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/gaoflow"><code>@​gaoflow</code></a> made
their first contribution in <a
href="https://redirect.github.com/caronc/apprise/pull/1663">caronc/apprise#1663</a></li>
<li><a
href="https://github.com/TheGlenn88"><code>@​TheGlenn88</code></a> made
their first contribution in <a
href="https://redirect.github.com/caronc/apprise/pull/1680">caronc/apprise#1680</a></li>
<li><a href="https://github.com/egvimo"><code>@​egvimo</code></a> made
their first contribution in <a
href="https://redirect.github.com/caronc/apprise/pull/1678">caronc/apprise#1678</a></li>
<li><a href="https://github.com/nyxst4ck"><code>@​nyxst4ck</code></a>
made their first contribution in <a
href="https://redirect.github.com/caronc/apprise/pull/1679">caronc/apprise#1679</a></li>
<li><a href="https://github.com/notrudyyy"><code>@​notrudyyy</code></a>
made their first contribution in <a
href="https://redirect.github.com/caronc/apprise/pull/1674">caronc/apprise#1674</a></li>
<li><a href="https://github.com/hchouhan"><code>@​hchouhan</code></a>
made their first contribution in <a
href="https://redirect.github.com/caronc/apprise/pull/1671">caronc/apprise#1671</a></li>
<li><a
href="https://github.com/IceCodeNew"><code>@​IceCodeNew</code></a> made
their first contribution in <a
href="https://redirect.github.com/caronc/apprise/pull/1684">caronc/apprise#1684</a></li>
<li><a
href="https://github.com/Sanjays2402"><code>@​Sanjays2402</code></a>
made their first contribution in <a
href="https://redirect.github.com/caronc/apprise/pull/1673">caronc/apprise#1673</a></li>
<li><a href="https://github.com/lipusal"><code>@​lipusal</code></a> made
their first contribution in <a
href="https://redirect.github.com/caronc/apprise/pull/1665">caronc/apprise#1665</a></li>
<li><a
href="https://github.com/albanobattistella"><code>@​albanobattistella</code></a>
made their first contribution in <a
href="https://redirect.github.com/caronc/apprise/pull/1670">caronc/apprise#1670</a></li>
<li><a href="https://github.com/youdie006"><code>@​youdie006</code></a>
made their first contribution in <a
href="https://redirect.github.com/caronc/apprise/pull/1694">caronc/apprise#1694</a></li>
<li><a
href="https://github.com/kevinfeyrer"><code>@​kevinfeyrer</code></a>
made their first contribution in <a
href="https://redirect.github.com/caronc/apprise/pull/1696">caronc/apprise#1696</a></li>
</ul>
<h2>Installation</h2>
<p>Apprise is available <a href="https://pypi.org/project/apprise/">on
PyPI</a> through <em>pip</em>:</p>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/caronc/apprise/commit/cf17bc9ecd2686c62b548b4577b6e06839331a60"><code>cf17bc9</code></a>
bumped version to v1.13.0</li>
<li><a
href="https://github.com/caronc/apprise/commit/c41f802190ff07a71edb98683c2094f04b5b9fc2"><code>c41f802</code></a>
in-memory email attachments no longer fail (<a
href="https://redirect.github.com/caronc/apprise/issues/1696">#1696</a>)</li>
<li><a
href="https://github.com/caronc/apprise/commit/6ebddd3a19a9076d1c0feca410f5e61bf9c927f5"><code>6ebddd3</code></a>
Return None from parse_url() on malformed authority content (<a
href="https://redirect.github.com/caronc/apprise/issues/1694">#1694</a>)</li>
<li><a
href="https://github.com/caronc/apprise/commit/6b8cb1364360c9af61190faa248d0ef4187daaa0"><code>6b8cb13</code></a>
Add Italian translation (<a
href="https://redirect.github.com/caronc/apprise/issues/1670">#1670</a>)</li>
<li><a
href="https://github.com/caronc/apprise/commit/468d26f3b49c4de8dd21a44f1db99288c79247f6"><code>468d26f</code></a>
Updated github-action plugins to latest supported versions (<a
href="https://redirect.github.com/caronc/apprise/issues/1692">#1692</a>)</li>
<li><a
href="https://github.com/caronc/apprise/commit/73c8512a5c518e4c601d6a32c53feabbc2a9e0fa"><code>73c8512</code></a>
Support for Telegram rich message support (<a
href="https://redirect.github.com/caronc/apprise/issues/1690">#1690</a>)</li>
<li><a
href="https://github.com/caronc/apprise/commit/fac5056b8cb50d725f83744ff26636ded60f9f46"><code>fac5056</code></a>
fixed issue causing time based xmpp tests to randomly fail</li>
<li><a
href="https://github.com/caronc/apprise/commit/2c48a87d1bfb5f5afa6a40775f6d3fb8fef2d2e4"><code>2c48a87</code></a>
Proxy documentation added to man page (<a
href="https://redirect.github.com/caronc/apprise/issues/1691">#1691</a>)</li>
<li><a
href="https://github.com/caronc/apprise/commit/094073af543726a52f29672f31ca060f2f8e31b0"><code>094073a</code></a>
relaxed and improved on unit-tests time restrictions</li>
<li><a
href="https://github.com/caronc/apprise/commit/cc2dffb3ccc6a6ee4cf0a6979c9881fe06b43b78"><code>cc2dffb</code></a>
Fixed warnings emitted while running test suite (<a
href="https://redirect.github.com/caronc/apprise/issues/1689">#1689</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/caronc/apprise/compare/v1.12.0...v1.13.0">compare
view</a></li>
</ul>
</details>
<br />

Updates `seleniumbase` from 4.51.12 to 4.52.1
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/seleniumbase/SeleniumBase/releases">seleniumbase's
releases</a>.</em></p>
<blockquote>
<h2>4.52.1 - Fix timeout issue with executing scripts</h2>
<h2>Fix timeout issue with executing scripts</h2>
<ul>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/e55bcc7a9f544cbb2ea9ad0c9d4fc358d36c055f">Fix
timeout issue with executing scripts in Selenium Mode</a>
--&gt; (See the next commit where the fix was moved to a
<code>finally</code> block)</li>
<li><strong><a
href="https://github.com/seleniumbase/SeleniumBase/commit/a5e37fd41813e92ed60cb7554781d5ba3aae6ef4">Improve
the earlier commit that fixes the script timeout issue</a></strong>
--&gt; (A method that changed the default timeout wasn't resetting it
back afterward)
--&gt; This resolves <a
href="https://redirect.github.com/seleniumbase/SeleniumBase/issues/4467">seleniumbase/SeleniumBase#4467</a></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/37ace079c94bb9e08d4604387fbce13876ac00fb">Update
examples</a>
--&gt; (Updates some examples after changes were made on the site being
tested)</li>
</ul>
<h2>What's Changed</h2>
<ul>
<li>Fix timeout issue with executing scripts by <a
href="https://github.com/mdmintz"><code>@​mdmintz</code></a> in <a
href="https://redirect.github.com/seleniumbase/SeleniumBase/pull/4468">seleniumbase/SeleniumBase#4468</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/seleniumbase/SeleniumBase/compare/v4.52.0...v4.52.1">https://github.com/seleniumbase/SeleniumBase/compare/v4.52.0...v4.52.1</a></p>
<h2>4.52.0 - CDP Mode: Patch 129</h2>
<h2>CDP Mode: Patch 129</h2>
<ul>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/814c21ed7d0e94d459778b32db8be1fe7eb85ba5">Update
CDP Mode</a></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/a3f0b5715fdb2d42e8ca53a043c173555f0ce097">Improve
output from driver downloads</a></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/17d723fb302ca7eb8a5604c05306a57688494499">Drop
support for Python 3.9</a></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/0b7cc37597bfdc1e008b6176a8cb793760da534a">Refresh
Python dependencies</a></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/99327cc58df60a9cce7cd0e50588e806d65e90c3">Add
scraping examples</a></li>
</ul>
<h3>Resolutions:</h3>
<ul>
<li>This resolves <a
href="https://redirect.github.com/seleniumbase/SeleniumBase/issues/4461">seleniumbase/SeleniumBase#4461</a></li>
<li>This resolves <a
href="https://redirect.github.com/seleniumbase/SeleniumBase/issues/4462">seleniumbase/SeleniumBase#4462</a></li>
<li>This resolves <a
href="https://redirect.github.com/seleniumbase/SeleniumBase/issues/4463">seleniumbase/SeleniumBase#4463</a></li>
<li>This resolves <a
href="https://redirect.github.com/seleniumbase/SeleniumBase/issues/4464">seleniumbase/SeleniumBase#4464</a></li>
<li>This resolves <a
href="https://redirect.github.com/seleniumbase/SeleniumBase/issues/4465">seleniumbase/SeleniumBase#4465</a></li>
</ul>
<h2>What's Changed</h2>
<ul>
<li>CDP Mode: Patch 129 by <a
href="https://github.com/mdmintz"><code>@​mdmintz</code></a> in <a
href="https://redirect.github.com/seleniumbase/SeleniumBase/pull/4466">seleniumbase/SeleniumBase#4466</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/seleniumbase/SeleniumBase/compare/v4.51.12...v4.52.0">https://github.com/seleniumbase/SeleniumBase/compare/v4.51.12...v4.52.0</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/c71edd581a72bfe46a21c1099a39f2ef8af5aa04"><code>c71edd5</code></a>
Merge pull request <a
href="https://redirect.github.com/seleniumbase/SeleniumBase/issues/4468">#4468</a>
from seleniumbase/fix-timeout-issue-with-executing-s...</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/a5e37fd41813e92ed60cb7554781d5ba3aae6ef4"><code>a5e37fd</code></a>
Improve the earlier commit that fixes the script timeout issue</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/cd7b8945ca0298be0f7a8f6578ad4313107ed7c9"><code>cd7b894</code></a>
Version 4.52.1</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/37ace079c94bb9e08d4604387fbce13876ac00fb"><code>37ace07</code></a>
Update examples</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/e55bcc7a9f544cbb2ea9ad0c9d4fc358d36c055f"><code>e55bcc7</code></a>
Fix timeout issue with executing scripts in Selenium mode</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/b2207cb2ff3f0849de5ece7b9e81402983f857d5"><code>b2207cb</code></a>
Merge pull request <a
href="https://redirect.github.com/seleniumbase/SeleniumBase/issues/4466">#4466</a>
from seleniumbase/cdp-mode-patch-129</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/0be0f562c918bef4c27ab19c5b3068a28aff8835"><code>0be0f56</code></a>
Version 4.52.0</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/44e9d8564ce6afd764941219916e464174badac3"><code>44e9d85</code></a>
Update the ReadMe</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/99327cc58df60a9cce7cd0e50588e806d65e90c3"><code>99327cc</code></a>
Add scraping examples</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/0b7cc37597bfdc1e008b6176a8cb793760da534a"><code>0b7cc37</code></a>
Refresh Python dependencies</li>
<li>Additional commits viewable in <a
href="https://github.com/seleniumbase/SeleniumBase/compare/v4.51.12...v4.52.1">compare
view</a></li>
</ul>
</details>
<br />

Updates `prek` from 0.4.13 to 0.4.14
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/j178/prek/releases">prek's
releases</a>.</em></p>
<blockquote>
<h2>0.4.14</h2>
<h2>Release Notes</h2>
<p>Released on 2026-08-17.</p>
<h3>Enhancements</h3>
<ul>
<li>Support check-yaml unsafe mode (<a
href="https://redirect.github.com/j178/prek/pull/2546">#2546</a>)</li>
</ul>
<h3>Performance</h3>
<ul>
<li>Reuse allocations in common filters (<a
href="https://redirect.github.com/j178/prek/pull/2548">#2548</a>)</li>
</ul>
<h3>Bug fixes</h3>
<ul>
<li>Accept non-finite floats in check-yaml (<a
href="https://redirect.github.com/j178/prek/pull/2545">#2545</a>)</li>
<li>Allow comment-heavy YAML in check-yaml (<a
href="https://redirect.github.com/j178/prek/pull/2554">#2554</a>)</li>
<li>Sanitize captured terminal output before replay (<a
href="https://redirect.github.com/j178/prek/pull/2552">#2552</a>)</li>
</ul>
<h3>Contributors</h3>
<ul>
<li><a href="https://github.com/j178"><code>@​j178</code></a></li>
</ul>
<h2>Install prek 0.4.14</h2>
<h3>Install prebuilt binaries via shell script</h3>
<pre lang="sh"><code>curl --proto '=https' --tlsv1.2 -LsSf
https://github.com/j178/prek/releases/download/v0.4.14/prek-installer.sh
| sh
</code></pre>
<h3>Install prebuilt binaries via powershell script</h3>
<pre lang="sh"><code>powershell -ExecutionPolicy Bypass -c &quot;irm
https://github.com/j178/prek/releases/download/v0.4.14/prek-installer.ps1
| iex&quot;
</code></pre>
<h3>Install prebuilt binaries via Homebrew</h3>
<pre lang="sh"><code>brew install prek
</code></pre>
<h2>Download prek 0.4.14</h2>
<table>
<thead>
<tr>
<th>File</th>
<th>Platform</th>
<th>Checksum</th>
</tr>
</thead>
<tbody>
<tr>
<td><a
href="https://github.com/j178/prek/releases/download/v0.4.14/prek-aarch64-apple-darwin.tar.gz">prek-aarch64-apple-darwin.tar.gz</a></td>
<td>Apple Silicon macOS</td>
<td><a
href="https://github.com/j178/prek/releases/download/v0.4.14/prek-aarch64-apple-darwin.tar.gz.sha256">checksum</a></td>
</tr>
<tr>
<td><a
href="https://github.com/j178/prek/releases/download/v0.4.14/prek-x86_64-apple-darwin.tar.gz">prek-x86_64-apple-darwin.tar.gz</a></td>
<td>Intel macOS</td>
<td><a
href="https://github.com/j178/prek/releases/download/v0.4.14/prek-x86_64-apple-darwin.tar.gz.sha256">checksum</a></td>
</tr>
<tr>
<td><a
href="https://github.com/j178/prek/releases/download/v0.4.14/prek-aarch64-pc-windows-msvc.zip">prek-aarch64-pc-windows-msvc.zip</a></td>
<td>ARM64 Windows</td>
<td><a
href="https://github.com/j178/prek/releases/download/v0.4.14/prek-aarch64-pc-windows-msvc.zip.sha256">checksum</a></td>
</tr>
</tbody>
</table>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/j178/prek/blob/master/CHANGELOG.md">prek's
changelog</a>.</em></p>
<blockquote>
<h2>0.4.14</h2>
<p>Released on 2026-08-17.</p>
<h3>Enhancements</h3>
<ul>
<li>Support check-yaml unsafe mode (<a
href="https://redirect.github.com/j178/prek/pull/2546">#2546</a>)</li>
</ul>
<h3>Performance</h3>
<ul>
<li>Reuse allocations in common filters (<a
href="https://redirect.github.com/j178/prek/pull/2548">#2548</a>)</li>
</ul>
<h3>Bug fixes</h3>
<ul>
<li>Accept non-finite floats in check-yaml (<a
href="https://redirect.github.com/j178/prek/pull/2545">#2545</a>)</li>
<li>Allow comment-heavy YAML in check-yaml (<a
href="https://redirect.github.com/j178/prek/pull/2554">#2554</a>)</li>
<li>Sanitize captured terminal output before replay (<a
href="https://redirect.github.com/j178/prek/pull/2552">#2552</a>)</li>
</ul>
<h3>Contributors</h3>
<ul>
<li><a href="https://github.com/j178"><code>@​j178</code></a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/j178/prek/commit/432eeb08a4d9fe760817ce8d3d75ed3ebf7d3943"><code>432eeb0</code></a>
Bump version to 0.4.14 (<a
href="https://redirect.github.com/j178/prek/issues/2571">#2571</a>)</li>
<li><a
href="https://github.com/j178/prek/commit/6b6d429319dabac1911b26ce6cb6f04bb814951a"><code>6b6d429</code></a>
Update Rust crate ignore to v0.4.33 (<a
href="https://redirect.github.com/j178/prek/issues/2564">#2564</a>)</li>
<li><a
href="https://github.com/j178/prek/commit/27d0cf8cb309f7b889d2b355254dc96957bbd60c"><code>27d0cf8</code></a>
Update prek hooks (<a
href="https://redirect.github.com/j178/prek/issues/2555">#2555</a>)</li>
<li><a
href="https://github.com/j178/prek/commit/aaf407bb098b624620eac3855e6818b3fa2c4ab0"><code>aaf407b</code></a>
Update Rust crate aws-lc-rs to v1.18.0 (<a
href="https://redirect.github.com/j178/prek/issues/2569">#2569</a>)</li>
<li><a
href="https://github.com/j178/prek/commit/3ddddfd494b87dc0ddee4e7694d1ead3ac4277b7"><code>3ddddfd</code></a>
Update Rust crate xml to v1.4.0 (<a
href="https://redirect.github.com/j178/prek/issues/2570">#2570</a>)</li>
<li><a
href="https://github.com/j178/prek/commit/fc90ec6c8a7b1519691b221bae5224064248a37d"><code>fc90ec6</code></a>
Update Rust crate similar to v3.1.2 (<a
href="https://redirect.github.com/j178/prek/issues/2567">#2567</a>)</li>
<li><a
href="https://github.com/j178/prek/commit/50e60d412350fcef8bc9e50755bd4ea357a45328"><code>50e60d4</code></a>
Update Rust crate thiserror to v2.0.20 (<a
href="https://redirect.github.com/j178/prek/issues/2568">#2568</a>)</li>
<li><a
href="https://github.com/j178/prek/commit/70a4c0635352987f55b995ecd768f1e8a2f9c9db"><code>70a4c06</code></a>
Update Rust crate liblzma to v0.4.8 (<a
href="https://redirect.github.com/j178/prek/issues/2565">#2565</a>)</li>
<li><a
href="https://github.com/j178/prek/commit/7b8dcaa4579449a9a656889398e7d380bd117b08"><code>7b8dcaa</code></a>
Update Rust crate globset to v0.4.20 (<a
href="https://redirect.github.com/j178/prek/issues/2563">#2563</a>)</li>
<li><a
href="https://github.com/j178/prek/commit/3e075d33a5584a8b0f467c39eac3833a8efd4da2"><code>3e075d3</code></a>
Update Rust crate clap_complete to v4.6.9 (<a
href="https://redirect.github.com/j178/prek/issues/2562">#2562</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/j178/prek/compare/v0.4.13...v0.4.14">compare
view</a></li>
</ul>
</details>
<br />

Updates `ruff` from 0.16.3 to 0.16.4
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/astral-sh/ruff/releases">ruff's
releases</a>.</em></p>
<blockquote>
<h2>0.16.4</h2>
<h2>Release Notes</h2>
<p>Released on 2026-08-20.</p>
<h3>Preview features</h3>
<ul>
<li>[<code>flake8-use-pathlib</code>] Add autofix for
<code>PTH116</code> (<a
href="https://redirect.github.com/astral-sh/ruff/pull/26460">#26460</a>)</li>
<li>[<code>refurb</code>] Restrict <code>delete-full-slice</code> to
lists (<code>FURB131</code>) (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27711">#27711</a>)</li>
<li>[<code>refurb</code>] Skip <code>FURB101</code> and
<code>FURB103</code> when the <code>open</code> argument is a file
descriptor (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27643">#27643</a>)</li>
</ul>
<h3>Bug fixes</h3>
<ul>
<li>Fix <code>InvalidInstruction</code> on Windows CPUs that do not
support <code>POPCNT</code> (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27803">#27803</a>)</li>
<li>[<code>pyflakes</code>] Emit semantic syntax errors in string type
definitions as <code>F722</code> (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27835">#27835</a>)</li>
<li>[<code>pylint</code>] Allow <code>os._exit</code> imports in
<code>import-private-name</code> (<code>PLC2701</code>) (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27738">#27738</a>)</li>
</ul>
<h3>Rule changes</h3>
<ul>
<li>[syntax-errors] Align mixed t-string/bytes error message with
CPython 3.14 (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27766">#27766</a>)</li>
<li>[<code>ruff</code>] Add <code>ctypes.LittleEndianStructure</code>
and related types to existing exception (<code>RUF012</code>) (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27753">#27753</a>)</li>
<li>[syntax-errors] Detect duplicate keyword arguments (<a
href="https://redirect.github.com/astral-sh/ruff/pull/17804">#17804</a>)</li>
<li>[syntax-errors] Detect parameters declared <code>nonlocal</code> (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27628">#27628</a>)</li>
</ul>
<h3>Server</h3>
<ul>
<li>Offer display-only fixes and mark safe fixes preferred (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27807">#27807</a>)</li>
<li>Support pull diagnostics for notebook cells (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27779">#27779</a>)</li>
</ul>
<h3>Documentation</h3>
<ul>
<li>Add default indicator to rules table (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27724">#27724</a>)</li>
<li>Fix broken link to Python docs (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27757">#27757</a>)</li>
</ul>
<h3>Other changes</h3>
<ul>
<li>Fix s390x stacker assembly in release builds (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27776">#27776</a>)</li>
<li>Guarantee minimum stack size when parsing a module, standalone
expression, and suites (<a
href="https://redirect.github.com/astral-sh/ruff/pull/25464">#25464</a>)</li>
<li>Reduce configuration deserialization code size (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27924">#27924</a>)</li>
<li>Check packed AST index bounds (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27849">#27849</a>)</li>
</ul>
<h3>Contributors</h3>
<ul>
<li><a
href="https://github.com/AbhinavMir"><code>@​AbhinavMir</code></a></li>
<li><a
href="https://github.com/eduardorittner"><code>@​eduardorittner</code></a></li>
<li><a href="https://github.com/royb3"><code>@​royb3</code></a></li>
<li><a
href="https://github.com/MichaReiser"><code>@​MichaReiser</code></a></li>
<li><a href="https://github.com/carljm"><code>@​carljm</code></a></li>
<li><a
href="https://github.com/rosstitmarsh"><code>@​rosstitmarsh</code></a></li>
<li><a href="https://github.com/ntBre"><code>@​ntBre</code></a></li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md">ruff's
changelog</a>.</em></p>
<blockquote>
<h2>0.16.4</h2>
<p>Released on 2026-08-20.</p>
<h3>Preview features</h3>
<ul>
<li>[<code>flake8-use-pathlib</code>] Add autofix for
<code>PTH116</code> (<a
href="https://redirect.github.com/astral-sh/ruff/pull/26460">#26460</a>)</li>
<li>[<code>refurb</code>] Restrict <code>delete-full-slice</code> to
lists (<code>FURB131</code>) (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27711">#27711</a>)</li>
<li>[<code>refurb</code>] Skip <code>FURB101</code> and
<code>FURB103</code> when the <code>open</code> argument is a file
descriptor (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27643">#27643</a>)</li>
</ul>
<h3>Bug fixes</h3>
<ul>
<li>Fix <code>InvalidInstruction</code> on Windows CPUs that do not
support <code>POPCNT</code> (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27803">#27803</a>)</li>
<li>[<code>pyflakes</code>] Emit semantic syntax errors in string type
definitions as <code>F722</code> (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27835">#27835</a>)</li>
<li>[<code>pylint</code>] Allow <code>os._exit</code> imports in
<code>import-private-name</code> (<code>PLC2701</code>) (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27738">#27738</a>)</li>
</ul>
<h3>Rule changes</h3>
<ul>
<li>[syntax-errors] Align mixed t-string/bytes error message with
CPython 3.14 (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27766">#27766</a>)</li>
<li>[<code>ruff</code>] Add <code>ctypes.LittleEndianStructure</code>
and related types to existing exception (<code>RUF012</code>) (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27753">#27753</a>)</li>
<li>[syntax-errors] Detect duplicate keyword arguments (<a
href="https://redirect.github.com/astral-sh/ruff/pull/17804">#17804</a>)</li>
<li>[syntax-errors] Detect parameters declared <code>nonlocal</code> (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27628">#27628</a>)</li>
</ul>
<h3>Server</h3>
<ul>
<li>Offer display-only fixes and mark safe fixes preferred (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27807">#27807</a>)</li>
<li>Support pull diagnostics for notebook cells (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27779">#27779</a>)</li>
</ul>
<h3>Documentation</h3>
<ul>
<li>Add default indicator to rules table (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27724">#27724</a>)</li>
<li>Fix broken link to Python docs (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27757">#27757</a>)</li>
</ul>
<h3>Other changes</h3>
<ul>
<li>Fix s390x stacker assembly in release builds (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27776">#27776</a>)</li>
<li>Guarantee minimum stack size when parsing a module, standalone
expression, and suites (<a
href="https://redirect.github.com/astral-sh/ruff/pull/25464">#25464</a>)</li>
<li>Reduce configuration deserialization code size (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27924">#27924</a>)</li>
<li>Check packed AST index bounds (<a
href="https://redirect.github.com/astral-sh/ruff/pull/27849">#27849</a>)</li>
</ul>
<h3>Contributors</h3>
<ul>
<li><a
href="https://github.com/AbhinavMir"><code>@​AbhinavMir</code></a></li>
<li><a
href="https://github.com/eduardorittner"><code>@​eduardorittner</code></a></li>
<li><a href="https://github.com/royb3"><code>@​royb3</code></a></li>
<li><a
href="https://github.com/MichaReiser"><code>@​MichaReiser</code></a></li>
<li><a href="https://github.com/carljm"><code>@​carljm</code></a></li>
<li><a
href="https://github.com/rosstitmarsh"><code>@​rosstitmarsh</code></a></li>
<li><a href="https://github.com/ntBre"><code>@​ntBre</code></a></li>
<li><a
href="https://github.com/zaniebot"><code>@​zaniebot</code></a></li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/astral-sh/ruff/commit/11c76bf48fdac06b2f240cba502eda96da4dce77"><code>11c76bf</code></a>
Bump 0.16.4 (<a
href="https://redirect.github.com/astral-sh/ruff/issues/27937">#27937</a>)</li>
<li><a
href="https://github.com/astral-sh/ruff/commit/d53c8c58662ca0576ddd502aa1a2979acf03832f"><code>d53c8c5</code></a>
Isolate playground builds from deployment credentials (<a
href="https://redirect.github.com/astral-sh/ruff/issues/27839">#27839</a>)</li>
<li><a
href="https://github.com/astral-sh/ruff/commit/cab001e5dec22f55653021f1f7be449e47c7d81e"><code>cab001e</code></a>
Disable uv preview for releases and pre-commit hooks (<a
href="https://redirect.github.com/astral-sh/ruff/issues/27939">#27939</a>)</li>
<li><a
href="https://github.com/astral-sh/ruff/commit/f8d575fedc97e75ea62c679d77afb14246afa88e"><code>f8d575f</code></a>
[ty] Clarify writing guidance for human readers (<a
href="https://redirect.github.com/astral-sh/ruff/issues/27912">#27912</a>)</li>
<li><a
href="https://github.com/astral-sh/ruff/commit/ca45faebb1750a213df19ed7f686ee5cf9277f93"><code>ca45fae</code></a>
Set <code>--preview</code> and <code>--default-index</code> for the
<code>uv-lock</code> hook (<a
href="https://redirect.github.com/astral-sh/ruff/issues/27935">#27935</a>)</li>
<li><a
href="https://github.com/astral-sh/ruff/commit/4827bf7cb449055e46fbfaf4b26e5125883a0569"><code>4827bf7</code></a>
Export <code>UV_DEFAULT_INDEX</code> in <code>release.sh</code> (<a
href="https://redirect.github.com/astral-sh/ruff/issues/27934">#27934</a>)</li>
<li><a
href="https://github.com/astral-sh/ruff/commit/d1087a4b9e03d253a88703f34e0869ee4b805456"><code>d1087a4</code></a>
[ty] Handle assignment expressions in string annotations (<a
href="https://redirect.github.com/astral-sh/ruff/issues/27921">#27921</a>)</li>
<li><a
href="https://github.com/astral-sh/ruff/commit/680cce48b6d89ab5b1566e4b797bd4847d861815"><code>680cce4</code></a>
[ty] Optimize inherited recursive protocol comparisons (<a
href="https://redirect.github.com/astral-sh/ruff/issues/27922">#27922</a>)</li>
<li><a
href="https://github.com/astral-sh/ruff/commit/974d3cbc04520c112843d6b92577844587402e04"><code>974d3cb</code></a>
Upgrade ecosystem-analyzer and mypy_primer to the latest upstream pins
(<a
href="https://redirect.github.com/astral-sh/ruff/issues/27932">#27932</a>)</li>
<li><a
href="https://github.com/astral-sh/ruff/commit/b169b402356d0676451f4a7bc6903da2644b31eb"><code>b169b40</code></a>
Install cargo tools locked (<a
href="https://redirect.github.com/astral-sh/ruff/issues/27929">#27929</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/astral-sh/ruff/compare/0.16.3...0.16.4">compare
view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-24 01:56:46 -04:00
CaliBrain 95e34670f7 fix/group archive extracted audiobooks (#1261)
- fix: group multi-file audiobooks that arrive as an archive
- Surface the concrete reason when a direct-download fetch fails
2026-08-24 01:25:00 -04:00
CaliBrain 7d56624ab6 fix: group multi-file audiobooks that arrive as an archive (#1254)
Follow-up to #1237. \`rename_and_group\` only grouped when the source
root was a directory, so a multi-file audiobook delivered as a single
archive fell through to the flat path: a \`Book.zip\` of twelve chapters
landed loose in the destination root with its original chapter names —
the layout #1181 is about.

The \`is_dir()\` guard was there to keep \`Book.zip/\` from becoming the
folder name, but skipping the file case gives up the grouping instead of
naming it. A non-directory source can only produce several book files by
having been extracted (\`collect_staged_files\` returns a single-element
list for every other file shape), so the archive stem is the release
name and the suffix is packaging: group under \`Book/\`.

Also regenerates the env docs for the new option and gives it the same
\"do not use with ingest folders\" caveat Rename and Organize carries,
since both now create directories in the destination.

Tested: reverting only the source fix makes both new tests fail and the
\`rename\` control case pass, so grouping stays opt-in. Full non-e2e
suite green (2653 passed).
2026-08-21 10:43:06 -04:00
f4421ff189 fix: preserve multi-file audiobook folders (#1237)
Pass the effective source root from `process_folder_output` into
`transfer_book_files`, and have the transfer layer select a sanitized
child directory named after that source root when an audiobook has
multiple files and its organization mode is `none` or `rename`. Create
that grouping directory before applying the existing hardlink/copy/move
logic so operation accounting, torrent seeding preservation, collision
handling, cleanup, and custom-script final paths continue to use the
established production path. Completed multi-file audiobook torrents
arrive as a directory whose chapter filenames may not identify the book,
but folder output currently sends every discovered chapter directly to
the configured destination in `none` and `rename` modes. This flattens
chapters from unrelated books together and causes directory-oriented
consumers such as Audiobookshelf to interpret individual chapters as
separate books.

A multi-file audiobook torrent in the default `rename` mode copies or
hardlinks all supported chapter files beneath `<destination>/<original
torrent directory>/` with their original chapter filenames, and places
no chapters directly in the destination root; A multi-file audiobook in
`none` mode receives the same source-folder grouping without renaming
its chapter files.

Fixes #1181

---------

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: CaliBrain <calibrain@l4n.xyz>
2026-08-21 09:44:08 -04:00
CaliBrain e7007865a4 fix(prowlarr): stop turning indexer failures into empty results and 404s (#1251)
Two independent bugs, both from an indexer that Prowlarr proxies rather
than answers for itself: the search never reported that it had failed,
and the grab never resolved what it was handed.

Search. A Torznab search is Prowlarr proxying a live request out to the
tracker, so for a Cloudflare-fronted indexer it waits on FlareSolverr.
The client gave it the 30s budget sized for Prowlarr's own JSON
endpoints, then swallowed every failure -- the timeout, the 429 Prowlarr
returns once it has disabled an indexer, a parse error -- into the same
empty list that means "this indexer has nothing". A cold challenge
routinely runs past a minute, so the UI said "No releases found for this
book" while FlareSolverr was still solving. That empty list also drove
the auto-expand retry, which fires on "no results with the category
filter". A timeout satisfies it, so Shelfmark sent a second search to an
indexer still busy with the first -- two Chromes at once, enough to take
FlareSolverr's down on a small host.

torznab_search now raises ProwlarrSearchError, and an empty list
strictly
means the indexer answered with no matches. The source records which
indexer searches failed: one dead indexer no longer sinks the others,
auto-expand runs only when every indexer genuinely answered, and zero
results with at least one failure raises SourceUnavailableError, which
the releases endpoint already turns into a 503 carrying a real message.
Prowlarr being unreachable was the same lie by another route -- the
indexer list came back empty, leaving nothing to query -- and now says
so.

Indexer searches also get their own timeout, PROWLARR_INDEXER_TIMEOUT,
defaulting to 90s and clamped to 5-300. Prowlarr's status and indexer
list keep 30s so Test Connection stays responsive, and the connect
timeout is split out at 10s so an unreachable Prowlarr fails fast rather
than hanging for the whole read budget. The overall per-request search
budget now scales to twice the indexer timeout, capped at 240s, so
raising the setting is not undone by the cap one level up while staying
under the 300s gunicorn worker timeout.

Grab. Prowlarr hands out a proxy download URL, with no magnetUrl and no
infoHash, for any indexer that only publishes torrent files. The native
Real-Debrid client built its magnet as "if not
url.startswith('magnet:') and expected_hash", so with no hash to work
from it left the URL alone and POSTed it to /torrents/addMagnet as the
magnet field. Real-Debrid answered 404 and the grab died on a raw HTTP
error. AllDebrid carried the same line and the same bug.

Both now resolve the URL first, through the extract_torrent_info path
the
torrent clients have used since #1108: pass a magnet through untouched,
follow a redirect or a response body that turns out to be a magnet,
otherwise upload the fetched .torrent, and fall back to a magnet built
from the infoHash only when the fetch failed. The file is preferred over
a synthesized urn:btih: magnet because it carries the tracker list; a
bare hash leaves the service to find the swarm on DHT alone. Fetches are
shared with the rest of the add path through the torrent fetch cache, so
resolving costs at most one request. Real-Debrid takes the file on PUT
/torrents/addTorrent with the raw bytes as the request body, AllDebrid
on
POST /magnet/upload/file as multipart files[]. A URL that resolves to
neither form now raises before any request reaches the service, so the
user reads why instead of a 404. Neither debrid client had any test
coverage; both have some now.

Fixes #1249
Fixes #1250
2026-08-21 09:07:03 -04:00
115 changed files with 7984 additions and 669 deletions
@@ -96,7 +96,7 @@ jobs:
type=ref,event=tag
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0
- name: Build and push ${{ matrix.target }} Docker image
id: push
@@ -141,7 +141,7 @@ jobs:
password: ${{ secrets.GITHUB_TOKEN }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0
- name: Create legacy aliases
run: |
+4 -4
View File
@@ -16,7 +16,7 @@ jobs:
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
- name: Install uv and Python
uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
version: "0.11.3"
python-version: "3.14"
@@ -42,7 +42,7 @@ jobs:
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
- name: Install uv and Python
uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
version: "0.11.3"
python-version: "3.14"
@@ -62,7 +62,7 @@ jobs:
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
- name: Install uv and Python
uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
version: "0.11.3"
python-version: "3.14"
@@ -81,7 +81,7 @@ jobs:
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0
- name: Build shelfmark-lite image
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
+3 -3
View File
@@ -25,14 +25,14 @@ jobs:
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
- name: Initialize CodeQL
uses: github/codeql-action/init@5595ccaf912efad79be6eef63a5619ff05969be3 # v3
uses: github/codeql-action/init@ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd # v3
with:
languages: ${{ matrix.language }}
- name: Autobuild
uses: github/codeql-action/autobuild@5595ccaf912efad79be6eef63a5619ff05969be3 # v3
uses: github/codeql-action/autobuild@ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd # v3
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@5595ccaf912efad79be6eef63a5619ff05969be3 # v3
uses: github/codeql-action/analyze@ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd # v3
with:
category: "/language:${{ matrix.language }}"
+2 -2
View File
@@ -72,7 +72,7 @@ jobs:
- name: Checkout
uses: actions/checkout@v7
- name: Install uv and Python
uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
python-version: "3.14"
enable-cache: true
@@ -97,7 +97,7 @@ jobs:
- name: Checkout
uses: actions/checkout@v7
- name: Install uv and Python
uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
python-version: "3.14"
enable-cache: true
+1
View File
@@ -236,6 +236,7 @@ pyrightconfig.json
*.local.*
AGENTS.md
.claude/
CLAUDE.md
.nvmrc
.playwright-mcp/
frontend-dist/
+9 -4
View File
@@ -28,7 +28,7 @@ RUN npm run build
# than copied into the image. A COPY here would land ~24 MB in a `base` layer that
# every published image inherits, and a later `rm` cannot take it back out again --
# a RUN adds a layer, it does not rewrite the one underneath.
FROM ghcr.io/astral-sh/uv:0.11.3@sha256:90bbb3c16635e9627f49eec6539f956d70746c409209041800a0280b93152823 AS uv
FROM ghcr.io/astral-sh/uv:0.12.5@sha256:e85be844203885286c60ffad8a858d48afb6c5a5c237ca0e67f12e74b8f174b1 AS uv
# Use python-slim as the base image
FROM python:3.14.7-slim@sha256:ce40764625a4ff50df3548277632e7f96c4e77fe75fa848aae9885476e7df5a4 AS base
@@ -152,9 +152,14 @@ RUN mkdir -p \
EXPOSE ${FLASK_PORT}
# Add healthcheck for container status
# Uses /api/health which doesn't require authentication
HEALTHCHECK --interval=60s --timeout=60s --start-period=60s --retries=3 \
CMD curl -s http://localhost:${FLASK_PORT}/api/health > /dev/null || exit 1
# Uses /api/health which doesn't require authentication.
# curl needs -f so an HTTP error status fails the probe instead of passing it:
# plain `curl -s` exits 0 on a 500, which reported a broken app as healthy.
# timeout stays well under interval so a hung probe cannot occupy a whole cycle.
# --start-interval matches the daemon default (5s), made explicit so startup
# probing does not depend on that default staying put.
HEALTHCHECK --interval=30s --timeout=10s --start-period=90s --start-interval=5s --retries=3 \
CMD curl -fsS http://localhost:${FLASK_PORT}/api/health > /dev/null || exit 1
# Use dumb-init as the entrypoint to handle signals properly
ENTRYPOINT ["/usr/bin/dumb-init", "--"]
+21
View File
@@ -276,6 +276,27 @@ class DownloadHandler(ABC):
pass
```
### Optional: Listing Files Before Download
Some releases bundle several books (a whole-series torrent). Shelfmark inspects a
release before queueing it so the user can review how it will be split into books.
Override `list_files` when your source can enumerate a release's files without
downloading it; the default returns `None`, which the UI reports as "can't inspect":
```python
from shelfmark.download.postprocess.packs import PackFile
def list_files(self, release_data: dict[str, Any]) -> list[PackFile] | None:
"""Return the release's files (release-relative paths + sizes), or None."""
torrent_bytes = ... # e.g. fetch the .torrent, or scrape the indexer's detail page
return extract_file_list_from_torrent(torrent_bytes) # from download.clients.torrent_utils
```
`release_data` is the same payload the frontend sends to `/api/releases/download`
(`source_id`, `download_url`, `content_type`, `series_name`, ...). Built-in examples:
Prowlarr parses the `.torrent` it already fetches (magnet-only releases return
`None`), and AudiobookBay reads the file table off its detail page.
### Download Method Parameters
| Parameter | Type | Description |
+59 -19
View File
@@ -247,7 +247,7 @@ Seconds since the last WireGuard handshake before the healthcheck bounces the tu
| `CALIBRE_WEB_URL` | Adds a navigation button to your book library (Calibre-Web Automated, Grimmory, etc). | string | _none_ |
| `AUDIOBOOK_LIBRARY_URL` | Adds a separate navigation button for your audiobook library (Audiobookshelf, Plex, etc). When both URLs are set, icons are shown instead of text. | string | _none_ |
| `SUPPORTED_FORMATS` | Book formats to include in search results. ZIP/RAR archives are extracted automatically and book files are used if found. | string (comma-separated) | `epub,mobi,azw3,fb2,djvu,cbz,cbr` |
| `SUPPORTED_AUDIOBOOK_FORMATS` | Audiobook formats to include in search results. ZIP/RAR archives are extracted automatically and audiobook files are used if found. | string (comma-separated) | `m4b,mp3,m4a,flac,ogg,wma,aac,wav,opus,zip,rar` |
| `SUPPORTED_AUDIOBOOK_FORMATS` | Audiobook formats to include in search results. ZIP/RAR archives are extracted automatically and audiobook files are used if found. | string (comma-separated) | `m4b,mp3,m4a,mp4,flac,ogg,wma,aac,wav,opus,zip,rar` |
| `BOOK_LANGUAGE` | Default language filter for searches. | string (comma-separated) | `en` |
<details>
@@ -296,16 +296,7 @@ Book formats to include in search results. ZIP/RAR archives are extracted automa
Audiobook formats to include in search results. ZIP/RAR archives are extracted automatically and audiobook files are used if found.
- **Type:** string (comma-separated)
- **Default:** `m4b,mp3,m4a,flac,ogg,wma,aac,wav,opus,zip,rar`
#### `BOOK_LANGUAGE`
**Default Book Languages**
Default language filter for searches.
- **Type:** string (comma-separated)
- **Default:** `en`
- **Default:** `m4b,mp3,m4a,mp4,flac,ogg,wma,aac,wav,opus,zip,rar`
</details>
@@ -314,6 +305,7 @@ Default language filter for searches.
| Variable | Description | Type | Default |
|----------|-------------|------|---------|
| `SEARCH_MODE` | How you want to search for and download books. | string (choice) | `universal` |
| `BOOK_LANGUAGE` | Default language filter for searches. Users can override this for their own account. | string (comma-separated) | `en` |
| `AA_DEFAULT_SORT` | Default sort order for search results. | string (choice) | `relevance` |
| `SHOW_RELEASE_SOURCE_LINKS` | Show clickable release-source links in release and details modals. Metadata provider links stay enabled. | boolean | `true` |
| `SHOW_COMBINED_SELECTOR` | Show the option to search for and download both a book and audiobook together. | boolean | `true` |
@@ -337,6 +329,15 @@ How you want to search for and download books.
- **Default:** `universal`
- **Options:** `direct` (Direct), `universal` (Universal)
#### `BOOK_LANGUAGE`
**Default Book Languages**
Default language filter for searches. Users can override this for their own account.
- **Type:** string (comma-separated)
- **Default:** `en`
#### `AA_DEFAULT_SORT`
**Default Sort Order**
@@ -704,7 +705,7 @@ Choose how downloaded audiobook files are named and organized.
- **Type:** string (choice)
- **Default:** `rename`
- **Options:** `none` (None), `rename` (Rename Only), `organize` (Rename and Organize)
- **Options:** `none` (None), `rename` (Rename Only), `organize` (Rename and Organize), `rename_and_group` (Rename and Group)
#### `TEMPLATE_AUDIOBOOK_RENAME`
@@ -749,6 +750,7 @@ Automatically open the downloads sidebar when a new download is queued.
Automatically download completed files to your browser for the selected content types.
- **Type:** string (comma-separated)
- **Default:** _empty list_
#### `MAX_CONCURRENT_DOWNLOADS`
@@ -1223,6 +1225,7 @@ How long to cache individual book details. Default: 600 (10 minutes). Max: 60480
| `PROWLARR_URL` | Base URL of your Prowlarr instance | string | _none_ |
| `PROWLARR_API_KEY` | Found in Prowlarr: Settings > General > API Key | string (secret) | _none_ |
| `PROWLARR_INDEXERS` | Select which indexers to search. 📚 = has book categories. Leave empty to search all. | string (comma-separated) | _empty list_ |
| `PROWLARR_INDEXER_TIMEOUT` | How long to wait for a single indexer to answer a search. Indexers behind FlareSolverr can need 90 seconds or more while a cold Cloudflare challenge is solved; raise this if searches come back empty and the Prowlarr log shows the search still running. | number | `90` |
| `PROWLARR_AUTO_EXPAND` | Automatically retry search without category filtering if no results are found | boolean | `false` |
| `PROWLARR_COLLAPSE_DUPLICATES` | Collapse a release that several indexer entries returned down to a single row, keeping the entry with the best Prowlarr priority. Turn this off to see every entry that carried it, which is what makes results from filter-specific entries (freeleech and the like) visible. | boolean | `true` |
| `PROWLARR_USE_SEED_PREFERENCES` | Apply per-indexer seed time and ratio preferences from Prowlarr when sending torrents to the download client | boolean | `false` |
@@ -1268,6 +1271,16 @@ Select which indexers to search. 📚 = has book categories. Leave empty to sear
- **Type:** string (comma-separated)
- **Default:** _empty list_
#### `PROWLARR_INDEXER_TIMEOUT`
**Indexer Search Timeout (seconds)**
How long to wait for a single indexer to answer a search. Indexers behind FlareSolverr can need 90 seconds or more while a cold Cloudflare challenge is solved; raise this if searches come back empty and the Prowlarr log shows the search still running.
- **Type:** number
- **Default:** `90`
- **Constraints:** min: 5, max: 300
#### `PROWLARR_AUTO_EXPAND`
**Auto-expand search on no results**
@@ -1302,8 +1315,9 @@ Apply per-indexer seed time and ratio preferences from Prowlarr when sending tor
| Variable | Description | Type | Default |
|----------|-------------|------|---------|
| `NEWZNAB_ENABLED` | Enable searching for books via a Newznab-compatible indexer | boolean | `false` |
| `NEWZNAB_URL` | Base URL of your Newznab indexer or aggregator | string | _none_ |
| `NEWZNAB_API_KEY` | Your Newznab API key (leave blank if not required) | string (secret) | _none_ |
| `NEWZNAB_INDEXERS` | Named Newznab connections. Each row accepts `name`, `url`, and `api_key`. | JSON array | `[]` |
| `NEWZNAB_URL` | Legacy single-indexer URL, used when `NEWZNAB_INDEXERS` is empty | string | _none_ |
| `NEWZNAB_API_KEY` | Legacy single-indexer API key | string (secret) | _none_ |
| `NEWZNAB_EBOOK_CATEGORIES` | Newznab category IDs searched for ebooks. Most indexers use the standard 7000, but some use custom IDs. Leave empty to use 7000. | string (comma-separated) | `7000` |
| `NEWZNAB_AUDIOBOOK_CATEGORIES` | Newznab category IDs searched for audiobooks. Most indexers use the standard 3030, but some use custom IDs. Leave empty to use 3030. | string (comma-separated) | `3030` |
| `NEWZNAB_AUTO_EXPAND` | Automatically retry search without category filtering if no results are found | boolean | `false` |
@@ -1320,21 +1334,36 @@ Enable searching for books via a Newznab-compatible indexer
- **Type:** boolean
- **Default:** `false`
#### `NEWZNAB_INDEXERS`
**Named Indexers**
Configure multiple named Newznab-compatible indexers. The name is shown beside each search result. For environment-based configuration, provide a JSON array:
```json
[
{"name":"NZBGeek","url":"https://api.nzbgeek.info","api_key":"..."},
{"name":"DrunkenSlug","url":"https://drunkenslug.com","api_key":"..."}
]
```
- **Type:** JSON array
- **Default:** `[]`
#### `NEWZNAB_URL`
**Newznab URL**
**Legacy Newznab URL**
Base URL of your Newznab indexer or aggregator
Single-indexer fallback used only when `NEWZNAB_INDEXERS` is empty.
- **Type:** string
- **Default:** _none_
- **Required:** Yes
#### `NEWZNAB_API_KEY`
**API Key**
**Legacy API Key**
Your Newznab API key (leave blank if not required)
API key for the legacy Newznab URL.
- **Type:** string (secret)
- **Default:** _none_
@@ -2150,6 +2179,7 @@ Enable Moly.hu as a metadata provider for book searches
| `SOURCE_PRIORITY` | Fallback sources, may have waiting. Requires bypasser. Drag to reorder. | JSON array | _see UI for defaults_ |
| `MAX_RETRY` | Maximum retry attempts for failed downloads. | number | `10` |
| `DEFAULT_SLEEP` | Wait time between download retry attempts. | number | `5` |
| `RELEASE_SEARCH_TIMEOUT` | How long one release search may run before it gives up and reports why. A first search on a cold start pays for a browser solve, so leave room for one. If you use a reverse proxy, its read timeout should be at least this high or it will cut the search off with a 504 first. | number | `300` |
| `AA_CONTENT_TYPE_ROUTING` | Override destination based on content type metadata. | boolean | `false` |
| `AA_CONTENT_TYPE_DIR_FICTION` | Fiction Books | string | _none_ |
| `AA_CONTENT_TYPE_DIR_NON_FICTION` | Non-Fiction Books | string | _none_ |
@@ -2228,6 +2258,16 @@ Wait time between download retry attempts.
- **Default:** `5`
- **Constraints:** min: 1, max: 60
#### `RELEASE_SEARCH_TIMEOUT`
**Release Search Timeout (seconds)**
How long one release search may run before it gives up and reports why. A first search on a cold start pays for a browser solve, so leave room for one. If you use a reverse proxy, its read timeout should be at least this high or it will cut the search off with a 504 first.
- **Type:** number
- **Default:** `300`
- **Constraints:** min: 30, max: 1800
#### `AA_CONTENT_TYPE_ROUTING`
**Enable Content-Type Routing**
+10 -1
View File
@@ -30,7 +30,7 @@ Requires mounting your Calibre-Web `app.db` to `/auth/app.db`.
Admins can configure per-user settings by editing a user in the user management panel. Non-admin users can also edit their own settings through **My Account** (accessible from the user menu). Admins control which sections are visible in My Account via the **Visible Self-Settings Sections** option.
There are three categories of per-user settings:
There are four categories of per-user settings:
### Delivery Preferences
@@ -42,6 +42,15 @@ Override where a user's downloads are sent. Options depend on the global output
- **BookLore library/path** — Per-user BookLore target (when using BookLore output mode)
- **Email recipient** — Per-user email address (when using Email output mode)
### Search Preferences
Override how a user searches, on top of the global search defaults:
- **Search mode** — Direct or Universal for this user
- **Default book languages** — The languages a user's searches fall back to when they don't pick one themselves. Useful for a shared instance where readers want different languages.
- **Metadata providers** — Book, audiobook, and combined-mode provider for this user
- **Default release sources** — The release tab opened first for books and audiobooks
### Notifications
Users can configure personal notification routes, separate from the global notification settings. Each route targets a URL (e.g. an Apprise-compatible endpoint) and can be scoped to specific event types or all events.
+3 -3
View File
@@ -22,7 +22,7 @@ dependencies = [
"qbittorrent-api>=2026.8.1",
"transmission-rpc",
"authlib>=1.7.2,<1.8",
"apprise>=1.12.0",
"apprise>=1.13.0",
# HTTP/2 client for RFC 8484 DoH: quad9 rejects HTTP/1.1 outright (505), which
# requests cannot speak. See shelfmark/download/doh_wireformat.py.
"httpx[http2]>=0.28.1",
@@ -32,7 +32,7 @@ dependencies = [
browser = [
"pyvirtualdisplay",
"pyautogui",
"seleniumbase==4.51.12",
"seleniumbase==4.52.2",
"python-xlib",
]
@@ -43,7 +43,7 @@ dev = [
"pytest",
"pytest-cov",
"pytest-xdist>=3.8.0",
"ruff==0.16.3",
"ruff==0.16.4",
"vulture>=2.14",
]
+1
View File
@@ -147,6 +147,7 @@ See the full [Environment Variables Reference](docs/environment-variables.md) fo
Some of the additional options available in Settings:
- **Prowlarr** - Configure indexers and download clients to download books and audiobooks
- **Additional audiobook sources** - Configure additional sources for audiobook discovery
- **Direct Download mirrors** - Supply your own Anna's Archive mirror URLs; Auto mode tries them in the order listed. The `annas-archive.is` domain does not currently work as a source — use `annas-archive.gl` instead (checked August 2026; mirror availability changes)
- **IRC** - Add details for IRC book sources and download directly from the UI. Most networks serve audiobooks from the same channel as ebooks (on `irc.irchighway.net` that's `#ebooks`, while `#bookz` is effectively inactive), so leave the separate audiobook channel blank unless your network actually indexes one. IRC audiobooks usually arrive as ZIP/RAR archives — keep those enabled under Supported Audiobook Formats or the releases are filtered out of results
- **Library Link** - Add a link to your Calibre-Web or Grimmory instance in the UI header
- **File processing** - Customiseable download paths, file renaming and directory creation with template-based renaming
+22 -1
View File
@@ -71,11 +71,18 @@ def _get_full_cookie_domains() -> set[str]:
return {_get_base_domain(domain) for domain in get_zlib_cookie_domains()}
def _replay_per_check_cookies() -> bool:
"""Whether the per-check trio is kept rather than dropped (see env.py)."""
from shelfmark.config import env
return env.DDG_REPLAY_PER_CHECK_COOKIES
def _should_extract_cookie(name: str, *, extract_all: bool) -> bool:
"""Determine if a cookie should be extracted based on its name."""
# Checked before extract_all: a per-check token is wrong to replay for every
# domain, including the full-session ones.
if name in DDG_EPHEMERAL_COOKIE_NAMES:
if name in DDG_EPHEMERAL_COOKIE_NAMES and not _replay_per_check_cookies():
return False
if extract_all:
return True
@@ -138,9 +145,11 @@ def store_extracted_cookies(
extract_all = base_domain in _get_full_cookie_domains()
cookies_found: dict[str, dict[str, Any]] = {}
dropped: list[str] = []
for cookie in cookies:
name = _cookie_field(cookie, "name") or ""
if not _should_extract_cookie(name, extract_all=extract_all):
dropped.append(name)
continue
secure = _cookie_field(cookie, "secure")
cookies_found[name] = {
@@ -152,6 +161,18 @@ def store_extracted_cookies(
"httpOnly": True,
}
# Names only, never values. Which cookies a solve won, and which of them were held
# back, is the evidence needed to settle what DDoS-Guard actually treats as clearance
# (issue #1276) - and without it a debug log shows a solve succeeding and the next
# request being challenged with nothing in between to explain why.
logger.debug(
"Solve on %s won %s; keeping %s; dropping %s",
base_domain,
sorted({_cookie_field(c, "name") or "" for c in cookies}),
sorted(cookies_found),
sorted(set(dropped)) or "nothing",
)
if not cookies_found:
return
+163 -13
View File
@@ -83,11 +83,14 @@ _HELPER_RESULT_POLL_SECONDS = 0.05
_HELPER_SHUTDOWN_GRACE_SECONDS = 15.0
_HELPER_IDLE_TIMEOUT_DEFAULT = 180.0
_PARENT_WATCHDOG_INTERVAL_SECONDS = 5.0
# How much of ffmpeg's stderr to quote when reporting that it died.
_FFMPEG_ERROR_TAIL_CHARS = 500
class _DisplayState(TypedDict):
ffmpeg: subprocess.Popen[bytes] | None
ffmpeg_output: Path | None
ffmpeg_error_log: Path | None
class _PageWithWindowRect(Protocol):
@@ -101,6 +104,7 @@ class _BrowserWithWindowRectPage(Protocol):
DISPLAY: _DisplayState = {
"ffmpeg": None,
"ffmpeg_output": None,
"ffmpeg_error_log": None,
}
LOCKED = threading.Lock()
_PROC_ROOT = Path("/proc")
@@ -618,6 +622,25 @@ BYPASS_METHODS = [
MAX_CONSECUTIVE_SAME_CHALLENGE = 3
# How many method attempts one _bypass() pass may make. Deliberately *not* MAX_RETRY:
# that value is already the outer page-load retry in _run_bypass_in_current_process, and
# reading it here too squared the budget - the default 10 meant 10 page loads x 4 methods
# = 40 solve attempts on one browser, which overruns the worker deadline and reports
# `TimeoutError` instead of a plain "bypass failed". One full pass through the methods
# plus a spare is all this loop can use anyway: the stuck-challenge guard below aborts at
# len(BYPASS_METHODS) + 1, so a larger number here only ever showed up in the logs.
_BYPASS_METHOD_ATTEMPTS = len(BYPASS_METHODS) + 1
# The undisturbed window a passive challenge gets before any method runs. Sized off the
# real thing: a desktop browser clears Anna's Archive's DDoS-Guard JS check in under 10s.
_PASSIVE_SOLVE_SECONDS = 15.0
_PASSIVE_SOLVE_POLL_SECONDS = 1.0
# Head-room the retry loop leaves itself so it can return a real failure rather than be
# cancelled at the worker deadline. Enough for the pass in flight to unwind and the
# browser to close.
_RESERVE_FOR_CLEAN_FAILURE_SECONDS = 60.0
def _check_cancellation(cancel_flag: Event | None, message: str) -> None:
"""Check if cancellation was requested and raise if so."""
@@ -627,13 +650,26 @@ def _check_cancellation(cancel_flag: Event | None, message: str) -> None:
raise BypassCancelledError(msg)
async def _wait_for_passive_solve(page: Any, cancel_flag: Event | None = None) -> bool:
"""Poll for a challenge that clears itself, without touching the page.
Returns True as soon as the page looks bypassed, False once the window is spent.
"""
logger.info("Waiting up to %.0fs for the challenge to clear itself...", _PASSIVE_SOLVE_SECONDS)
deadline = time.monotonic() + _PASSIVE_SOLVE_SECONDS
while time.monotonic() < deadline:
_check_cancellation(cancel_flag, "Bypass cancelled while waiting for a passive solve")
await asyncio.sleep(_PASSIVE_SOLVE_POLL_SECONDS)
if await _is_bypassed(page):
return True
return False
async def _bypass(
page: Any, max_retries: int | None = None, cancel_flag: Event | None = None
) -> bool:
"""Attempt to bypass Cloudflare/DDOS-Guard protection using multiple methods."""
max_retries = (
max_retries if max_retries is not None else _coerce_positive_int(app_config.MAX_RETRY, 10)
)
max_retries = max_retries if max_retries is not None else _BYPASS_METHOD_ATTEMPTS
last_challenge_type = None
consecutive_same_challenge = 0
@@ -651,6 +687,20 @@ async def _bypass(
challenge_type = await _detect_challenge_type(page)
logger.debug("Challenge detected: %s", challenge_type)
# Give a passive check the undisturbed window it needs before touching the page.
# DDoS-Guard's JS check on Anna's Archive has no click target: it runs, then
# navigates on its own - a desktop browser clears it in well under 15s. Every
# method below either clicks a selector that is not there or reloads, and a reload
# restarts an in-flight check (which DDoS-Guard also throttles), so going straight
# to them meant the one thing that actually solves this challenge was the one
# thing never tried. Costs one 15s window per solve against a minutes-long budget,
# and a challenge that needs interaction simply falls through to the methods.
if try_count == 0 and challenge_type != "none":
if await _wait_for_passive_solve(page, cancel_flag):
logger.info("Bypass successful: %s challenge cleared itself", challenge_type)
return True
logger.debug("Challenge did not clear on its own; trying bypass methods")
# No challenge detected but page doesn't look bypassed - wait and retry
if challenge_type == "none":
logger.info("No challenge detected, waiting for page to settle...")
@@ -810,14 +860,33 @@ async def _get(url: str, driver: Any, cancel_flag: Event | None = None) -> str:
def _run_bypass_in_current_process(url: str, retry: int, cancel_flag: Event | None = None) -> str:
"""Run the CDP bypass in the current process."""
timeout = (
_CHILD_BYPASS_TIMEOUT_SECONDS
if os.environ.get(_BYPASS_CHILD_ENV) == "1"
else _IN_PROCESS_BYPASS_TIMEOUT_SECONDS
)
async def _run_bypass() -> str:
driver = None
# Stop retrying while there is still time to say so. A challenge nothing can solve
# would otherwise spend every one of `retry` passes and be cut off mid-pass by the
# worker deadline, which surfaces to the caller as `RuntimeError: TimeoutError` -
# a message that says nothing about protection and sent users looking at their
# reverse proxy. Giving up a pass early returns the real "bypass failed" instead.
deadline = time.monotonic() + timeout - _RESERVE_FOR_CLEAN_FAILURE_SECONDS
try:
driver = await _create_cdp_browser(url)
for attempt in range(retry):
_check_cancellation(cancel_flag, "Bypass cancelled before attempt")
if attempt > 0 and time.monotonic() >= deadline:
logger.warning(
"Bypass budget spent after %s/%s attempts; giving up on %s",
attempt,
retry,
url,
)
break
try:
result = await _get(url, driver, cancel_flag)
@@ -838,7 +907,7 @@ def _run_bypass_in_current_process(url: str, retry: int, cancel_flag: Event | No
await _close_cdp_driver(driver)
driver = await _create_cdp_browser(url)
logger.error("Bypass failed after %s attempts", retry)
logger.error("Bypass failed for %s", url)
return ""
finally:
if driver:
@@ -852,12 +921,9 @@ def _run_bypass_in_current_process(url: str, retry: int, cancel_flag: Event | No
# one call and closes it on the way out, so a helper serving many requests would build
# and tear down a loop per bypass and would carry no deadline of its own. The worker's
# loop lives in a thread, outlives any single bypass, and cancels the coroutine when the
# deadline passes.
timeout = (
_CHILD_BYPASS_TIMEOUT_SECONDS
if os.environ.get(_BYPASS_CHILD_ENV) == "1"
else _IN_PROCESS_BYPASS_TIMEOUT_SECONDS
)
# deadline passes. `_run_bypass` aims to finish inside this same budget of its own
# accord, so reaching this deadline now means a wedged session rather than a stubborn
# challenge - which is the only case worth reporting as a timeout.
return _CDP_WORKER.run(_run_bypass(), timeout=timeout)
@@ -1155,6 +1221,20 @@ def get(url: str, retry: int | None = None, cancel_flag: Event | None = None) ->
if cached_result:
return cached_result
# Re-checked after the cached attempt, not just in get_bypassed_page: that check
# ran before the queue, and this call may have spent minutes holding for LOCKED
# while another request collected a 429 (or collected one itself, just above).
# A solve cannot clear a throttle - the challenge renders, the solve "succeeds",
# and the cleared request is refused again while the backoff is renewed.
remaining = network.host_cooldown_remaining(url)
if remaining > 0:
hostname = urlparse(url).hostname or url
msg = (
f"{hostname} is rate-limited (429); skipping bypass for ~{remaining:.0f}s "
"until the cooldown clears."
)
raise network.RateLimitedError(msg)
if env.DOCKERMODE and os.environ.get(_BYPASS_CHILD_ENV) != "1":
return _get_via_subprocess(url, retry, cancel_flag)
return _run_bypass_in_current_process(url, retry, cancel_flag)
@@ -1339,13 +1419,48 @@ def _start_ffmpeg_recording(display: str) -> None:
"-an",
output_file.as_posix(),
"-nostats",
# Was "0", which discards everything including the reason it could not start.
# Recordings have been arriving empty with no explanation anywhere: on issue
# #1276 all three of a session's recordings were gone and the log said only
# "FFmpeg already stopped", because ffmpeg exits before creating the file when
# it cannot open the X display. Errors only - this is a debug-mode recorder, not
# something to make chatty.
"-loglevel",
"0",
"error",
]
logger.debug("Starting FFmpeg recording to %s", output_file)
logger.debug_trace(f"FFmpeg command: {' '.join(ffmpeg_cmd)}")
DISPLAY["ffmpeg"] = subprocess.Popen(ffmpeg_cmd)
# Kept beside the recording so it travels in the debug bundle, which is the only
# place anyone will look for it. A file rather than a pipe: nothing here would drain
# a pipe, and a full one would wedge ffmpeg partway through a capture.
error_log = output_file.with_suffix(".ffmpeg.log")
try:
stderr_handle = error_log.open("wb")
except OSError as exc:
logger.debug("Could not open FFmpeg error log %s: %s", error_log, exc)
stderr_handle = None
DISPLAY["ffmpeg"] = subprocess.Popen(
ffmpeg_cmd, stderr=stderr_handle, stdout=subprocess.DEVNULL
)
if stderr_handle is not None:
# The child holds its own descriptor; this one has done its job.
stderr_handle.close()
DISPLAY["ffmpeg_output"] = output_file
DISPLAY["ffmpeg_error_log"] = error_log
def _ffmpeg_error_summary() -> str:
"""What ffmpeg wrote to stderr, for the log line that reports it died."""
error_log = DISPLAY.get("ffmpeg_error_log")
if not error_log:
return "No FFmpeg error log was captured."
try:
text = Path(error_log).read_text(encoding="utf-8", errors="replace").strip()
except OSError as exc:
return f"FFmpeg error log unreadable ({exc})."
if not text:
return f"FFmpeg logged nothing to {error_log}."
return f"FFmpeg said: {text[-_FFMPEG_ERROR_TAIL_CHARS:]}"
def _stop_ffmpeg_recording() -> None:
@@ -1357,9 +1472,17 @@ def _stop_ffmpeg_recording() -> None:
if not proc:
return
if proc.poll() is not None:
logger.debug("FFmpeg already stopped")
# Not "already stopped" - ffmpeg was asked to record until now and is gone, so
# the recording for this bypass does not exist. Say so, with the reason, rather
# than leaving an empty recording/ directory to be discovered later.
logger.warning(
"FFmpeg exited early (code %s); no recording for this bypass. %s",
proc.returncode,
_ffmpeg_error_summary(),
)
DISPLAY["ffmpeg"] = None
DISPLAY["ffmpeg_output"] = None
DISPLAY["ffmpeg_error_log"] = None
return
try:
proc.send_signal(signal.SIGINT)
@@ -1374,6 +1497,7 @@ def _stop_ffmpeg_recording() -> None:
proc.kill()
DISPLAY["ffmpeg"] = None
DISPLAY["ffmpeg_output"] = None
DISPLAY["ffmpeg_error_log"] = None
def _try_with_cached_cookies(url: str, hostname: str) -> str | None:
@@ -1400,6 +1524,20 @@ def _try_with_cached_cookies(url: str, hostname: str) -> str | None:
if response.status_code == HTTPStatus.OK:
logger.debug("Cached cookies worked, skipped Chrome bypass")
return response.text
if response.status_code == HTTPStatus.TOO_MANY_REQUESTS:
# Throttled, not challenged. The clearance is still good - the origin is
# rate-limiting this IP and would answer 429 to a browser holding the very
# same cookies. Discarding it here (as every other rejection does) meant a
# solve won seconds earlier was thrown away and the next query bought its
# own 20-60s browser solve, which is itself more traffic at a host that has
# just asked for less. Keep it, arm the backoff, and let the caller wait.
wait = network.note_rate_limited(url)
logger.debug(
"Cached cookies hit a 429 for %s; keeping them and backing off ~%.0fs",
url,
wait,
)
return None
logger.debug(
"Cached cookies rejected (%s) for %s; discarding them",
response.status_code,
@@ -1438,6 +1576,18 @@ def get_bypassed_page(
attempt_url = sel.rewrite(url)
hostname = urlparse(attempt_url).hostname or ""
# A 429 means the origin is throttling this IP; the challenge still renders, so a
# solve "succeeds" but the cleared request is rejected again and the throttle is only
# renewed. Never spend a minutes-long Chrome solve on a cooling-down host - fail fast
# so the caller waits the backoff out instead of looping the solve.
remaining = network.host_cooldown_remaining(attempt_url)
if remaining > 0:
msg = (
f"{hostname} is rate-limited (429); skipping bypass for ~{remaining:.0f}s "
"until the cooldown clears."
)
raise network.RateLimitedError(msg)
cached_result = _try_with_cached_cookies(attempt_url, hostname)
if cached_result:
return cached_result
+15
View File
@@ -203,6 +203,21 @@ ONBOARDING = string_to_bool(os.getenv("ONBOARDING", "true"))
_DEBUG_SKIP_SOURCES_RAW = os.getenv("DEBUG_SKIP_SOURCES", "").strip().lower()
DEBUG_SKIP_SOURCES = {s.strip() for s in _DEBUG_SKIP_SOURCES_RAW.split(",") if s.strip()}
# Debug: keep DDoS-Guard's __ddg8_/__ddg9_/__ddg10_ in the clearance store instead of
# dropping them after a solve.
#
# Which of DDoS-Guard's cookies actually *are* clearance is not settled. The store treats
# the trio as describing one check (client IP, timestamp, token) and drops them, on the
# reasoning that replaying a stale IP/timestamp is what re-arms the ?check=1 loop - see
# shelfmark.bypass.cookie_store. Field reports on issue #1276 point the other way: every
# request after a successful solve was challenged again, which is only consistent with
# what the store keeps not being sufficient clearance on its own.
#
# Deliberately env-only and off by default: this is a knob for reproducing the question
# against a live host, not a setting to offer users. Set it to true, solve once, and watch
# whether the next search still logs "Redirect loop detected".
DDG_REPLAY_PER_CHECK_COOKIES = string_to_bool(os.getenv("DDG_REPLAY_PER_CHECK_COOKIES", "false"))
# =============================================================================
# Legacy migration support - will be removed in future version
+37 -9
View File
@@ -430,13 +430,6 @@ def general_settings() -> list[SettingsField]:
options=_AUDIOBOOK_FORMAT_OPTIONS,
default=[*AUDIOBOOK_FORMATS, *ARCHIVE_FORMATS],
),
MultiSelectField(
key="BOOK_LANGUAGE",
label="Default Book Languages",
description="Default language filter for searches.",
options=_LANGUAGE_OPTIONS,
default=["en"],
),
]
@@ -474,6 +467,17 @@ def search_mode_settings() -> list[SettingsField]:
default="universal",
user_overridable=True,
),
MultiSelectField(
key="BOOK_LANGUAGE",
label="Default Book Languages",
description=(
"Default language filter for searches. Users can override this for their "
"own account."
),
options=_LANGUAGE_OPTIONS,
default=["en"],
user_overridable=True,
),
SelectField(
key="AA_DEFAULT_SORT",
label="Default Sort Order",
@@ -766,7 +770,10 @@ def _on_save_downloads(values: dict[str, Any]) -> dict[str, Any]:
}
# Audiobooks are always folder output.
if effective.get("FILE_ORGANIZATION_AUDIOBOOK", "rename") == "rename":
if effective.get("FILE_ORGANIZATION_AUDIOBOOK", "rename") in {
"rename",
"rename_and_group",
}:
template = effective.get("TEMPLATE_AUDIOBOOK_RENAME", "")
if _contains_path_separators(template):
return {
@@ -1294,6 +1301,11 @@ def download_settings() -> list[SettingsField]:
"label": "Rename and Organize",
"description": "Create folders and rename files using a template. Recommended for Audiobookshelf. Do not use with ingest folders.",
},
{
"value": "rename_and_group",
"label": "Rename and Group",
"description": "Rename single-file downloads; keep multi-file downloads grouped in their source folder. Do not use with ingest folders.",
},
],
default="rename",
universal_only=True,
@@ -1312,7 +1324,10 @@ def download_settings() -> list[SettingsField]:
),
default="{Author} - {Title}",
placeholder="{Author} - {Title}{ - Part }{PartNumber}",
show_when={"field": "FILE_ORGANIZATION_AUDIOBOOK", "value": "rename"},
show_when={
"field": "FILE_ORGANIZATION_AUDIOBOOK",
"value": ["rename", "rename_and_group"],
},
universal_only=True,
),
# Organize mode template - folders allowed
@@ -1545,6 +1560,19 @@ def download_source_settings() -> list[SettingsField]:
min_value=1,
max_value=60,
),
NumberField(
key="RELEASE_SEARCH_TIMEOUT",
label="Release Search Timeout (seconds)",
description=(
"How long one release search may run before it gives up and reports why. "
"A first search on a cold start pays for a browser solve, so leave room "
"for one. If you use a reverse proxy, its read timeout should be at least "
"this high or it will cut the search off with a 504 first."
),
default=300,
min_value=30,
max_value=1800,
),
HeadingField(
key="content_type_routing_heading",
title="Content-Type Routing",
+35 -4
View File
@@ -7,6 +7,7 @@ that talks to /api/admin/users endpoints.
from typing import Any
from shelfmark.core.languages import normalize_language
from shelfmark.core.request_policy import (
get_source_content_type_capabilities,
parse_policy_mode,
@@ -61,7 +62,7 @@ _SELF_SETTINGS_SECTION_OPTIONS = [
{
"value": "search",
"label": "Search Preferences",
"description": "Show personal search mode and provider settings.",
"description": "Show personal search mode, language, and provider settings.",
},
{
"value": "notifications",
@@ -77,8 +78,9 @@ _SEARCH_PREFERENCE_PROVIDER_KEYS = {
"METADATA_PROVIDER_AUDIOBOOK",
"METADATA_PROVIDER_COMBINED",
}
_SEARCH_PREFERENCE_VALIDATABLE_KEYS = {
SEARCH_PREFERENCE_VALIDATABLE_KEYS = {
"SEARCH_MODE",
"BOOK_LANGUAGE",
"DEFAULT_RELEASE_SOURCE",
"DEFAULT_RELEASE_SOURCE_AUDIOBOOK",
"SHOW_COMBINED_SELECTOR",
@@ -178,14 +180,43 @@ def _get_request_policy_rule_columns() -> list[dict[str, object]]:
]
def _validate_book_languages(value: Any) -> tuple[Any, str | None]:
"""Validate a per-user default language list against the known languages.
Accepts the list the settings UI sends as well as a comma-separated string, so an
API client can spell the value the way the env var does. Blank entries are skipped
rather than rejected, which makes "" and "en," mean the same as [] and ["en"]. An
empty result is a deliberate override meaning "no default language filter", so it
is kept as-is; ``None`` clears the override further up the chain.
"""
entries = value.split(",") if isinstance(value, str) else value
if not isinstance(entries, (list, tuple)):
return value, "BOOK_LANGUAGE must be a list of language codes"
normalized: list[str] = []
for entry in entries:
if entry is None or (isinstance(entry, str) and not entry.strip()):
continue
code = normalize_language(entry)
if code is None:
return value, f"BOOK_LANGUAGE contains an unsupported language: {entry}"
if code not in normalized:
normalized.append(code)
return normalized, None
def validate_search_preference_value(key: str, value: Any) -> tuple[Any, str | None]:
"""Validate and normalize a search preference value for user overrides."""
if key not in _SEARCH_PREFERENCE_VALIDATABLE_KEYS:
if key not in SEARCH_PREFERENCE_VALIDATABLE_KEYS:
return value, None
if value is None:
return None, None
if key == "BOOK_LANGUAGE":
return _validate_book_languages(value)
normalized_value = str(value).strip()
if key == "SEARCH_MODE":
@@ -298,7 +329,7 @@ def _on_save_users(values: dict[str, object]) -> dict[str, object]:
}
values["REQUEST_POLICY_RULES"] = normalized_rules
for key in _SEARCH_PREFERENCE_VALIDATABLE_KEYS:
for key in SEARCH_PREFERENCE_VALIDATABLE_KEYS:
if key not in values:
continue
normalized_value, validation_error = validate_search_preference_value(key, values[key])
+7 -8
View File
@@ -11,7 +11,10 @@ from shelfmark.config.notifications_settings import (
is_valid_notification_url,
normalize_notification_routes,
)
from shelfmark.config.users_settings import validate_search_preference_value
from shelfmark.config.users_settings import (
SEARCH_PREFERENCE_VALIDATABLE_KEYS,
validate_search_preference_value,
)
from shelfmark.core.config import config as app_config
from shelfmark.core.request_policy import parse_policy_mode, validate_policy_rules
from shelfmark.core.settings_registry import load_config_file
@@ -91,13 +94,9 @@ def validate_user_settings(
if search_validation_error:
errors.append(search_validation_error)
continue
if key in {
"SEARCH_MODE",
"METADATA_PROVIDER",
"METADATA_PROVIDER_AUDIOBOOK",
"DEFAULT_RELEASE_SOURCE",
"DEFAULT_RELEASE_SOURCE_AUDIOBOOK",
}:
# Every key the search validator recognises keeps its normalized value;
# a hand-maintained subset here silently dropped normalization for the rest.
if key in SEARCH_PREFERENCE_VALIDATABLE_KEYS:
valid[key] = normalized_search_value
continue
+6
View File
@@ -136,6 +136,12 @@ class DownloadTask:
default_factory=dict
) # Per-output parameters (e.g. email recipient)
# Multi-book packs: one release holding several books. `book_plan` is the split the
# user approved before download (list of {title, series_position, year, files});
# `multi_book` asks post-processing to split heuristically when no plan exists.
multi_book: bool = False
book_plan: list[dict[str, Any]] | None = None
# User association (multi-user support)
user_id: int | None = None # DB user ID who queued this download
username: str | None = None # Username for {User} template variable
+102
View File
@@ -0,0 +1,102 @@
"""Pre-download release inspection: list a release's files and plan a multi-book split."""
from __future__ import annotations
from typing import TYPE_CHECKING, Any
from flask import jsonify, request
from shelfmark.core.logger import setup_logger
from shelfmark.core.utils import is_audiobook
from shelfmark.download.postprocess.packs import PackFile, PackPlan, plan_pack
from shelfmark.download.postprocess.policy import (
get_supported_audiobook_formats,
get_supported_formats,
)
from shelfmark.release_sources import get_handler
if TYPE_CHECKING:
from collections.abc import Callable
from flask import Flask, Response
logger = setup_logger(__name__)
_INSPECT_ERRORS = (OSError, RuntimeError, ValueError, TypeError, KeyError, AttributeError)
NOT_INSPECTABLE_REASON = "This source cannot list the release's files before downloading"
def _serialize_plan(plan: PackPlan) -> dict[str, Any]:
return {
"is_pack": plan.is_pack,
"ignored": plan.ignored,
"books": [
{
"title": book.title,
"series_position": book.series_position,
"year": book.year,
"files": book.files,
}
for book in plan.books
],
}
def inspect_release(data: dict[str, Any]) -> dict[str, Any]:
"""Build the inspect response for a release payload (same shape as a download)."""
source = str(data["source"])
handler = get_handler(source)
try:
files: list[PackFile] | None = handler.list_files(data)
except _INSPECT_ERRORS as exc:
logger.warning(
"Could not list files for %s release %s: %s", source, data.get("source_id"), exc
)
return {"inspected": False, "reason": str(exc), "files": [], "plan": None}
if files is None:
return {"inspected": False, "reason": NOT_INSPECTABLE_REASON, "files": [], "plan": None}
content_type = data.get("content_type")
supported = (
get_supported_audiobook_formats()
if is_audiobook(content_type if isinstance(content_type, str) else None)
else get_supported_formats()
)
series_name = data.get("series_name")
author_name = data.get("author")
plan = plan_pack(
files,
supported_extensions=set(supported),
series_name=series_name if isinstance(series_name, str) else None,
author_name=author_name if isinstance(author_name, str) else None,
)
return {
"inspected": True,
"reason": None,
"files": [{"path": f.path, "size": f.size} for f in files],
"plan": _serialize_plan(plan),
}
def register_release_inspect_routes(
app: Flask,
login_required: Callable[..., Any],
) -> None:
"""Register POST /api/releases/inspect."""
@app.route("/api/releases/inspect", methods=["POST"])
@login_required
def api_inspect_release() -> Response | tuple[Response, int]:
data = request.get_json(silent=True)
if not isinstance(data, dict):
return jsonify({"error": "No data provided"}), 400
if not data.get("source_id"):
return jsonify({"error": "source_id is required"}), 400
if not data.get("source"):
return jsonify({"error": "source is required"}), 400
try:
get_handler(str(data["source"]))
except ValueError as exc:
return jsonify({"error": str(exc)}), 400
return jsonify(inspect_release(data))
+134
View File
@@ -0,0 +1,134 @@
"""A wall-clock budget for one release search, enforced through the existing cancel flag.
`/api/releases` is synchronous: the browser waits on it while the search runs. Nothing
bounded that wait, and the bypasser's own worst case is minutes long
(`internal_bypasser.max_duration_seconds()`), so a search that ran into an unsolvable
protection challenge outlived every reverse proxy in front of it. The user then saw
"Server unavailable (504)" - a gateway timeout that says nothing about what went wrong
and points the blame at their proxy config. See issue #1276.
The budget is expressed as the cancel flag the download path already understands: an
Event armed by a timer. `html_get_page`, the bypassers and the helper subprocess all poll
it, so an expired budget stops a solve already in flight rather than only refusing the
next one. When it trips, the search fails with a message that names the real cause.
Scoped to a context variable so it applies to the request that set it and to nothing else
- a queued download must keep its own, much longer, budget.
"""
from __future__ import annotations
import threading
import time
from contextlib import contextmanager
from contextvars import ContextVar
from typing import TYPE_CHECKING
from shelfmark.core.logger import setup_logger
if TYPE_CHECKING:
from collections.abc import Iterator
logger = setup_logger(__name__)
# What one search may spend. A first search on a cold start legitimately pays for a
# browser solve - jfmlima measured 60-120s for a successful one on Anna's Archive - so
# this cannot be as tight as a proxy's default read timeout without breaking working
# setups. It is instead well below the ~840s the bypass path could previously reach,
# which is what turned a failing challenge into a gateway timeout.
DEFAULT_SEARCH_BUDGET_SECONDS = 300.0
_MIN_SEARCH_BUDGET_SECONDS = 30.0
_MAX_SEARCH_BUDGET_SECONDS = 1800.0
# Raised to the caller when the budget runs out, so the API can say so plainly.
SEARCH_DEADLINE_MESSAGE = (
"The release search ran out of time (%.0fs). Anna's Archive is behind a protection "
"challenge the bypasser could not solve in that window. Raise the release search "
"timeout if your setup is simply slow."
)
class SearchDeadline:
"""A budget with an Event that trips when it expires."""
def __init__(self, budget_seconds: float) -> None:
self.budget_seconds = budget_seconds
self.expires_at = time.monotonic() + budget_seconds
# A plain threading.Event on purpose: this is handed on as a cancel flag, and
# that is the type the download path, the CDP worker thread and the bypass helper
# already poll.
self.event = threading.Event()
self._timer = threading.Timer(budget_seconds, self.event.set)
self._timer.daemon = True
def start(self) -> None:
self._timer.start()
def cancel(self) -> None:
self._timer.cancel()
@property
def remaining(self) -> float:
return max(0.0, self.expires_at - time.monotonic())
@property
def expired(self) -> bool:
return self.event.is_set() or self.remaining <= 0
_current: ContextVar[SearchDeadline | None] = ContextVar("search_deadline", default=None)
def budget_seconds() -> float:
"""The configured budget for one release search."""
from shelfmark.core.config import config as app_config
raw = app_config.get("RELEASE_SEARCH_TIMEOUT", DEFAULT_SEARCH_BUDGET_SECONDS)
if isinstance(raw, bool) or not isinstance(raw, int | float | str):
return DEFAULT_SEARCH_BUDGET_SECONDS
try:
value = float(raw)
except TypeError, ValueError:
return DEFAULT_SEARCH_BUDGET_SECONDS
if value <= 0:
return DEFAULT_SEARCH_BUDGET_SECONDS
return min(max(value, _MIN_SEARCH_BUDGET_SECONDS), _MAX_SEARCH_BUDGET_SECONDS)
@contextmanager
def search_deadline(budget: float | None = None) -> Iterator[SearchDeadline]:
"""Apply a budget to everything the calling context does."""
deadline = SearchDeadline(budget if budget is not None else budget_seconds())
token = _current.set(deadline)
deadline.start()
logger.debug("Release search budget: %.0fs", deadline.budget_seconds)
try:
yield deadline
finally:
deadline.cancel()
_current.reset(token)
def current() -> SearchDeadline | None:
"""The budget in force, or None outside a search."""
return _current.get()
def expired() -> bool:
"""Whether the budget in force has run out. False when there is no budget."""
deadline = _current.get()
return deadline is not None and deadline.expired
def cancel_event() -> threading.Event | None:
"""The Event that trips when the budget runs out, for use as a cancel flag."""
deadline = _current.get()
return deadline.event if deadline is not None else None
def deadline_message() -> str:
"""The failure to report when the budget has run out."""
deadline = _current.get()
budget = deadline.budget_seconds if deadline else DEFAULT_SEARCH_BUDGET_SECONDS
return SEARCH_DEADLINE_MESSAGE % budget
+58 -18
View File
@@ -7,6 +7,7 @@ from dataclasses import dataclass
from typing import TYPE_CHECKING
from shelfmark.core.config import config
from shelfmark.core.logger import setup_logger
from shelfmark.metadata_providers import (
BookMetadata,
build_localized_search_titles,
@@ -16,6 +17,8 @@ from shelfmark.metadata_providers import (
if TYPE_CHECKING:
from shelfmark.core.models import SearchFilters
logger = setup_logger(__name__)
MANUAL_QUERY_MAX_LEN = 256
@@ -52,30 +55,61 @@ class ReleaseSearchPlan:
return self.title_variants[0].query if self.title_variants else ""
def _normalize_languages(languages: list[str] | None) -> list[str] | None:
def _to_language_codes(values: Iterable[object], *, source: str) -> list[str] | None:
"""Resolve any spelling of a language to the ISO code the sources expect.
Anna's Archive matches `lang=` against ISO codes: `lang=english` is not a loose
spelling of `lang=en`, it is a facet value AA does not have, and it filters every
search down to nothing. Only the *per-user* override was normalised
(config.users_settings.validate), so a global BOOK_LANGUAGE=english - the spelling
the old docs used - reached the query verbatim and silently emptied every search
with no error anywhere. See issue #1276.
An entry that resolves to nothing is dropped with a warning rather than passed
through: searching unfiltered and saying so beats reporting "no results" for a book
the source is full of.
"""
from shelfmark.core.languages import normalize_language
codes: list[str] = []
unresolved: list[str] = []
for value in values:
text = str(value).strip() if value is not None else ""
if not text:
continue
if text.lower() == "all":
# An explicit "search every language", not a language.
return None
code = normalize_language(text)
if code is None:
unresolved.append(text)
continue
if code not in codes:
codes.append(code)
if unresolved:
logger.warning(
"Ignoring unrecognised language(s) in %s: %s. Use an ISO code such as 'en', "
"a three-letter code, or an English name like 'English'.",
source,
", ".join(unresolved),
)
return codes or None
def _normalize_languages(languages: list[str] | None, user_id: int | None) -> list[str] | None:
if not languages:
default = getattr(config, "BOOK_LANGUAGE", None)
default = config.get("BOOK_LANGUAGE", None, user_id=user_id)
if isinstance(default, str):
default_values: list[object] = [default]
elif isinstance(default, Iterable) and not isinstance(default, (bytes, bytearray, dict)):
default_values = list(default)
else:
return None
return [str(lang).strip() for lang in default_values if str(lang).strip()]
return _to_language_codes(default_values, source="BOOK_LANGUAGE")
normalized: list[str] = []
for lang in languages:
if not lang:
continue
s = str(lang).strip()
if not s:
continue
normalized.append(s)
if any(lang.lower() == "all" for lang in normalized):
return None
return normalized or None
return _to_language_codes(languages, source="the search request")
def _pick_search_author(book: BookMetadata) -> str:
@@ -102,9 +136,15 @@ def build_release_search_plan(
manual_query: str | None = None,
indexers: list[str] | None = None,
source_filters: SearchFilters | None = None,
user_id: int | None = None,
) -> ReleaseSearchPlan:
"""Build normalized search variants shared across release sources."""
resolved_languages = _normalize_languages(languages)
"""Build normalized search variants shared across release sources.
``user_id`` picks up that user's default languages when the caller does not
filter explicitly, so a search started without a language filter uses the
reader's own default rather than the instance-wide one.
"""
resolved_languages = _normalize_languages(languages, user_id)
resolved_manual_query = None
if manual_query:
+6 -1
View File
@@ -122,7 +122,12 @@ def is_audiobook(content_type: str | None) -> bool:
# had drifted apart: the settings UI only offered m4b/mp3/m4a, which meant a FLAC
# audiobook could never be enabled, was silently dropped from every search result, and
# was rejected after download as "format not supported".
AUDIOBOOK_FORMATS = ("m4b", "mp3", "m4a", "flac", "ogg", "wma", "aac", "wav", "opus")
#
# "mp4" is here because some trackers (MyAnonamouse in particular) ship AAC audiobooks
# as per-chapter .mp4 files - the same ISO-BMFF container as .m4a/.m4b, just with the
# generic extension. Without it those releases downloaded fine and then failed
# post-processing with "No book files found in download".
AUDIOBOOK_FORMATS = ("m4b", "mp3", "m4a", "mp4", "flac", "ogg", "wma", "aac", "wav", "opus")
# Multi-file audiobooks are almost always distributed as an archive. These are containers
# rather than formats: they are what a *release* looks like, and the formats above are
+68 -31
View File
@@ -26,6 +26,11 @@ from shelfmark.download.clients import (
register_client,
)
from shelfmark.download.clients._coercion import config_text
from shelfmark.download.clients.torrent_utils import (
DebridMagnet,
DebridUpload,
resolve_debrid_upload,
)
from shelfmark.download.http import download_url
from shelfmark.download.network import get_ssl_verify
@@ -71,6 +76,7 @@ _BOOK_EXTENSIONS = (
".m4b",
".mobi",
".mp3",
".mp4",
".ogg",
".opus",
".pdf",
@@ -202,41 +208,19 @@ class AllDebridClient(DownloadClient):
expected_hash: str | None = None,
**kwargs: object,
) -> str:
"""Upload a magnet link to AllDebrid and return the magnet ID."""
"""Send a torrent to AllDebrid and return the magnet ID.
Accepts a magnet link, a .torrent URL, or an indexer proxy URL; anything
that is not already a magnet is resolved first, since an HTTP URL posted
as a magnet is rejected rather than downloaded (#1250).
"""
if not self._api_key:
msg = "AllDebrid API key is not configured"
raise RuntimeError(msg)
magnet_link = url
if not magnet_link.startswith("magnet:") and expected_hash:
magnet_link = f"magnet:?xt=urn:btih:{expected_hash}"
api_url = f"{_API_BASE}/magnet/upload"
try:
resp = requests.post(
api_url,
headers=self._auth_headers(),
data={"magnets[]": magnet_link},
timeout=_API_TIMEOUT,
verify=get_ssl_verify(api_url),
)
resp.raise_for_status()
data = resp.json()
if data.get("status") != "success":
code = data.get("error", {}).get("code", "UNKNOWN")
msg = f"AllDebrid upload failed: {code}"
_raise_runtime_error(msg)
magnets = data.get("data", {}).get("magnets", [])
if not magnets:
msg = "No magnet returned from AllDebrid"
_raise_runtime_error(msg)
info = magnets[0]
if info.get("error"):
code = info["error"].get("code", "UNKNOWN")
msg = f"AllDebrid magnet error: {code}"
_raise_runtime_error(msg)
upload = resolve_debrid_upload(url, expected_hash=expected_hash)
info = self._send_torrent(upload)
magnet_id = str(info.get("id", ""))
if not magnet_id:
@@ -262,12 +246,65 @@ class AllDebridClient(DownloadClient):
)
except Exception:
logger.exception("Failed to upload magnet to AllDebrid")
logger.exception("Failed to add torrent to AllDebrid")
raise
else:
return magnet_id
def _send_torrent(self, upload: DebridUpload) -> dict[str, Any]:
"""Hand the torrent to AllDebrid, as a magnet or as a file upload.
Both endpoints answer with the same envelope and the same per-entry
error shape, differing only in which key holds the entries.
"""
if isinstance(upload, DebridMagnet):
api_url = f"{_API_BASE}/magnet/upload"
entries_key = "magnets"
resp = requests.post(
api_url,
headers=self._auth_headers(),
data={"magnets[]": upload.magnet_url},
timeout=_API_TIMEOUT,
verify=get_ssl_verify(api_url),
)
else:
api_url = f"{_API_BASE}/magnet/upload/file"
entries_key = "files"
resp = requests.post(
api_url,
headers=self._auth_headers(),
files={
"files[]": (
"release.torrent",
upload.torrent_data,
"application/x-bittorrent",
)
},
timeout=_API_TIMEOUT,
verify=get_ssl_verify(api_url),
)
resp.raise_for_status()
data = resp.json()
if data.get("status") != "success":
code = data.get("error", {}).get("code", "UNKNOWN")
msg = f"AllDebrid upload failed: {code}"
_raise_runtime_error(msg)
entries = data.get("data", {}).get(entries_key, [])
if not entries:
msg = "AllDebrid accepted the upload but returned no torrent"
_raise_runtime_error(msg)
info = entries[0]
if info.get("error"):
code = info["error"].get("code", "UNKNOWN")
msg = f"AllDebrid rejected the torrent: {code}"
_raise_runtime_error(msg)
return info
def get_status(self, download_id: str) -> DownloadStatus:
"""Poll AllDebrid for magnet status and drive the download."""
state = self._ensure_state(download_id)
+105 -35
View File
@@ -45,6 +45,10 @@ _HASH_LENGTH_ED2K = 32
_HTTP_STATUS_FORBIDDEN = HTTPStatus.FORBIDDEN
_HTTP_STATUS_NOT_FOUND = HTTPStatus.NOT_FOUND
_METADATA_DOWNLOAD_STATES = {"forcedMetaDL", "metaDL"}
# How long add_download waits for magnet metadata before falling back to the info
# hash it already knows, rather than holding the download queue on a thin swarm.
_METADATA_WAIT_POLLS = 20
_METADATA_WAIT_INTERVAL_SECONDS = 0.5
_ONE_WEEK_IN_SECONDS = 604800
@@ -221,6 +225,9 @@ class QBittorrentClient(DownloadClient):
self._category = config_text(config.get("QBITTORRENT_CATEGORY", "books"))
self._download_dir = config_text(config.get("QBITTORRENT_DOWNLOAD_DIR", ""))
self._tags = _normalize_tags(config.get("QBITTORRENT_TAG", []))
# download_id -> qBittorrent's current primary hash, for identities that no
# longer match it directly. See _resolve_torrent().
self._primary_hashes: dict[str, str] = {}
@property
def _can_reauthenticate(self) -> bool:
@@ -307,13 +314,31 @@ class QBittorrentClient(DownloadClient):
params = {"category": category} if category else {}
return self._request_torrent_info_records(params)
def _remember_primary_hash(self, download_id: str, torrent: SimpleNamespace) -> None:
"""Note the primary hash a listing scan found, so later lookups skip the scan."""
torrent_hash = getattr(torrent, "hash", None)
if isinstance(torrent_hash, str) and torrent_hash:
self._primary_hashes[download_id.lower()] = torrent_hash.lower()
def _resolve_torrent(
self, download_id: str, category: str | None = None
) -> tuple[SimpleNamespace | None, str | None]:
"""Resolve any known torrent identity to its current qBittorrent record."""
torrent, error = self._get_torrent_info(download_id)
if error or torrent:
return torrent, error
"""Resolve any known torrent identity to its current qBittorrent record.
A hybrid torrent's primary hash switches from the v1 hash to the truncated v2
hash once metadata resolves, so a download tracked by its v1 hash misses the
`hashes=` lookup and falls through to a full listing. Since `get_status()`
polls every couple of seconds for the life of the download, remember the
primary hash a scan finds and try it first.
"""
cached = self._primary_hashes.get(download_id.lower())
for candidate in (item for item in dict.fromkeys((cached, download_id)) if item):
torrent, error = self._get_torrent_info(candidate)
if error:
return None, error
if torrent:
self._remember_primary_hash(download_id, torrent)
return torrent, None
categories = [candidate for candidate in (category, self._category) if candidate]
for candidate in dict.fromkeys(categories):
@@ -325,18 +350,41 @@ class QBittorrentClient(DownloadClient):
None,
)
if torrent:
self._remember_primary_hash(download_id, torrent)
return torrent, None
torrents, error = self._list_torrents_by_category(None)
if error:
return None, error
return (
next(
(item for item in torrents if _torrent_matches_download_id(item, download_id)),
None,
),
torrent = next(
(item for item in torrents if _torrent_matches_download_id(item, download_id)),
None,
)
if torrent:
self._remember_primary_hash(download_id, torrent)
else:
# The torrent is gone; drop the note so a re-add is not looked up by a
# hash that no longer exists.
self._primary_hashes.pop(download_id.lower(), None)
return torrent, None
def _current_hash(self, download_id: str) -> str:
"""qBittorrent's current primary hash for any identity we know the torrent by.
Falls back to the given ID when the torrent cannot be found, so callers
still address the hash they were handed and surface the client's error.
"""
try:
torrent, error = self._resolve_torrent(download_id)
except _QBITTORRENT_CLIENT_ERRORS as e:
logger.debug("Could not resolve current hash for %s: %s", download_id, e)
return download_id
if error or not torrent:
return download_id
torrent_hash = getattr(torrent, "hash", None)
if isinstance(torrent_hash, str) and torrent_hash:
return torrent_hash
return download_id
def _list_category_hashes(self, category: str | None) -> set[str] | None:
"""Snapshot the hashes qBittorrent currently reports for a category."""
@@ -495,9 +543,13 @@ class QBittorrentClient(DownloadClient):
message = f"{message} (torrent file fetch failed: {torrent_info.fetch_error})"
_raise_runtime_error(message)
# Wait until qBittorrent has resolved magnet metadata so the returned
# hash is its stable primary torrent ID, which may differ from the v1 hash.
for _ in range(20):
# Prefer qBittorrent's primary torrent ID, which for hybrid torrents
# switches from the v1 hash to the truncated v2 hash once metadata
# resolves. A magnet with few peers can take minutes to fetch metadata,
# and the torrent is worth keeping in the meantime: every lookup goes
# through `_resolve_torrent`, which still matches the v1 hash against
# `infohash_v1` after the primary ID has changed.
for _ in range(_METADATA_WAIT_POLLS):
torrent, error = self._resolve_torrent(expected_hash, category)
if error:
logger.debug("qBittorrent add_download: %s", error)
@@ -506,17 +558,18 @@ class QBittorrentClient(DownloadClient):
if isinstance(torrent_hash, str) and torrent_hash:
logger.info("Added torrent: %s", torrent_hash)
return torrent_hash.lower()
time.sleep(0.5)
time.sleep(_METADATA_WAIT_INTERVAL_SECONDS)
_raise_runtime_error(
"Torrent metadata resolution was not confirmed within the visibility grace period "
f"(response={result_text})"
logger.info(
"Added torrent %s; metadata still pending after %.0fs, tracking it by info hash",
expected_hash,
_METADATA_WAIT_POLLS * _METADATA_WAIT_INTERVAL_SECONDS,
)
except _QBITTORRENT_CLIENT_ERRORS:
logger.exception("qBittorrent add failed")
raise
else:
return expected_hash
return expected_hash.lower()
def get_status(self, download_id: str) -> DownloadStatus:
"""Get torrent status by hash.
@@ -529,7 +582,7 @@ class QBittorrentClient(DownloadClient):
"""
try:
torrent, error = self._get_torrent_info(download_id)
torrent, error = self._resolve_torrent(download_id)
if error:
return DownloadStatus.error(error)
if not torrent:
@@ -613,7 +666,9 @@ class QBittorrentClient(DownloadClient):
"""
try:
self._client.torrents_delete(torrent_hashes=download_id, delete_files=delete_files)
torrent_hash = self._current_hash(download_id)
self._client.torrents_delete(torrent_hashes=torrent_hash, delete_files=delete_files)
self._primary_hashes.pop(download_id.lower(), None)
logger.info(
"Removed torrent from qBittorrent: %s%s",
download_id,
@@ -635,7 +690,7 @@ class QBittorrentClient(DownloadClient):
logger.debug("Could not create category '%s': %s", category, e)
self._client.torrents_set_category(
torrent_hashes=download_id,
torrent_hashes=self._current_hash(download_id),
category=category,
)
logger.info("Set qBittorrent category for %s to '%s'", download_id, category)
@@ -657,7 +712,7 @@ class QBittorrentClient(DownloadClient):
- join `save_path` with the torrent's top-level directory
"""
try:
torrent, error = self._get_torrent_info(download_id)
torrent, error = self._resolve_torrent(download_id)
if error:
logger.debug("qBittorrent get_download_path: %s", error)
return None
@@ -758,6 +813,33 @@ class QBittorrentClient(DownloadClient):
)
return None
def _await_existing_torrent(
self, info_hash: str, category: str | None
) -> tuple[str, DownloadStatus] | None:
"""Report a torrent already in qBittorrent, waiting out magnet metadata first."""
for _ in range(_METADATA_WAIT_POLLS):
torrent, error = self._resolve_torrent(info_hash, category)
if error:
logger.debug("qBittorrent find_existing: %s", error)
return None
if not torrent:
return None
if getattr(torrent, "state", None) not in _METADATA_DOWNLOAD_STATES:
torrent_hash = getattr(torrent, "hash", None)
if isinstance(torrent_hash, str) and torrent_hash:
torrent_hash = torrent_hash.lower()
return (torrent_hash, self.get_status(torrent_hash))
time.sleep(_METADATA_WAIT_INTERVAL_SECONDS)
# Metadata is still pending, but the torrent is here and `add_download` keeps
# one in this state rather than giving up. Report it by info hash so the
# caller joins the download in progress instead of adding a duplicate.
logger.info(
"Existing torrent %s is still fetching metadata; joining it by info hash",
info_hash,
)
return (info_hash.lower(), self.get_status(info_hash))
def find_existing(
self, url: str, category: str | None = None
) -> tuple[str, DownloadStatus] | None:
@@ -767,21 +849,9 @@ class QBittorrentClient(DownloadClient):
if not torrent_info.info_hash:
return None
for _ in range(20):
torrent, error = self._resolve_torrent(torrent_info.info_hash, category)
if error:
logger.debug("qBittorrent find_existing: %s", error)
return None
if not torrent:
return None
if getattr(torrent, "state", None) not in _METADATA_DOWNLOAD_STATES:
torrent_hash = getattr(torrent, "hash", None)
if isinstance(torrent_hash, str) and torrent_hash:
torrent_hash = torrent_hash.lower()
return (torrent_hash, self.get_status(torrent_hash))
time.sleep(0.5)
existing = self._await_existing_torrent(torrent_info.info_hash, category)
except _QBITTORRENT_CLIENT_ERRORS as e:
logger.debug("Error checking for existing torrent: %s", e)
return None
else:
return None
return existing
+44 -16
View File
@@ -24,6 +24,11 @@ from shelfmark.download.clients import (
register_client,
)
from shelfmark.download.clients._coercion import config_text
from shelfmark.download.clients.torrent_utils import (
DebridMagnet,
DebridUpload,
resolve_debrid_upload,
)
from shelfmark.download.http import download_url
from shelfmark.download.network import get_ssl_verify
@@ -75,6 +80,7 @@ _BOOK_EXTENSIONS = (
".m4b",
".mobi",
".mp3",
".mp4",
".ogg",
".opus",
".pdf",
@@ -173,26 +179,19 @@ class RealDebridClient(DownloadClient):
expected_hash: str | None = None,
**kwargs: object,
) -> str:
"""Upload a magnet link to Real-Debrid and select all files."""
"""Send a torrent to Real-Debrid and select all files.
Accepts a magnet link, a .torrent URL, or an indexer proxy URL; anything
that is not already a magnet is resolved first, because Real-Debrid
answers a non-magnet body on addMagnet with a bare 404 (#1250).
"""
if not self._api_key:
msg = "Real-Debrid API key is not configured"
raise RuntimeError(msg)
magnet_link = url
if not magnet_link.startswith("magnet:") and expected_hash:
magnet_link = f"magnet:?xt=urn:btih:{expected_hash}"
add_url = f"{_API_BASE}/torrents/addMagnet"
try:
resp = requests.post(
add_url,
headers=self._auth_headers(),
data={"magnet": magnet_link},
timeout=_API_TIMEOUT,
verify=get_ssl_verify(add_url),
)
resp.raise_for_status()
data = resp.json()
upload = resolve_debrid_upload(url, expected_hash=expected_hash)
data = self._send_torrent(upload)
torrent_id = str(data.get("id", ""))
if not torrent_id:
@@ -229,12 +228,41 @@ class RealDebridClient(DownloadClient):
)
except Exception:
logger.exception("Failed to upload magnet to Real-Debrid")
logger.exception("Failed to add torrent to Real-Debrid")
raise
else:
return torrent_id
def _send_torrent(self, upload: DebridUpload) -> dict[str, Any]:
"""Hand the torrent to Real-Debrid, as a magnet or as a file upload."""
if isinstance(upload, DebridMagnet):
add_url = f"{_API_BASE}/torrents/addMagnet"
resp = requests.post(
add_url,
headers=self._auth_headers(),
data={"magnet": upload.magnet_url},
timeout=_API_TIMEOUT,
verify=get_ssl_verify(add_url),
)
else:
# addTorrent is a PUT that takes the raw file as the request body,
# not a form field: https://api.real-debrid.com/
add_url = f"{_API_BASE}/torrents/addTorrent"
resp = requests.put(
add_url,
headers={
**self._auth_headers(),
"Content-Type": "application/x-bittorrent",
},
data=upload.torrent_data,
timeout=_API_TIMEOUT,
verify=get_ssl_verify(add_url),
)
resp.raise_for_status()
return resp.json()
def get_status(self, download_id: str) -> DownloadStatus:
"""Poll Real-Debrid for torrent status and drive the download."""
state = self._ensure_state(download_id)
+9
View File
@@ -248,6 +248,15 @@ class SABnzbdClient(DownloadClient):
if trusted_url and _url_origin(trusted_url) == target_origin:
return True
named_indexers = config.get("NEWZNAB_INDEXERS", [])
if isinstance(named_indexers, list):
for row in named_indexers:
if not isinstance(row, dict):
continue
trusted_url = normalize_http_config_url(row.get("url"))
if trusted_url and _url_origin(trusted_url) == target_origin:
return True
return False
def _get_prowlarr_headers(self, url: str) -> dict:
+105
View File
@@ -17,6 +17,7 @@ from shelfmark.core.config import config
from shelfmark.core.logger import setup_logger
from shelfmark.core.utils import normalize_http_url
from shelfmark.download.network import get_ssl_verify
from shelfmark.download.postprocess.packs import PackFile
logger = setup_logger(__name__)
@@ -82,6 +83,60 @@ class TorrentInfo:
return self
@dataclass
class DebridMagnet:
"""A magnet link, ready to hand to a debrid service as-is."""
magnet_url: str
@dataclass
class DebridTorrentFile:
"""Raw .torrent bytes, for a debrid service's file-upload endpoint."""
torrent_data: bytes
# A debrid service takes one or the other, never an indexer page or a proxy URL.
type DebridUpload = DebridMagnet | DebridTorrentFile
def resolve_debrid_upload(url: str, *, expected_hash: str | None = None) -> DebridUpload:
"""Resolve a release download URL into a magnet link or .torrent bytes.
Prowlarr hands out a proxy URL, with no magnetUrl and no infoHash, for any
indexer that only publishes torrent files - 1337x among them. Posting that
URL to a debrid service as if it were a magnet is what produced a bare 404
from the service instead of a download (#1250).
The torrent file is preferred over a synthesized `urn:btih:` magnet because
it carries the tracker list, which is how the service finds a swarm that is
not already cached. Fetches are shared with the rest of the add path through
the torrent fetch cache, so resolving here costs at most one request.
Raises:
ValueError: The URL resolved to neither form, so there is nothing to send.
"""
if url.startswith("magnet:"):
return DebridMagnet(magnet_url=url)
info = extract_torrent_info(url, expected_hash=expected_hash)
if info.is_magnet and info.magnet_url:
# The download URL redirected to, or returned, a magnet link.
return DebridMagnet(magnet_url=info.magnet_url)
if info.torrent_data:
return DebridTorrentFile(torrent_data=info.torrent_data)
if info.info_hash:
# No file to upload, but the hash alone still identifies the torrent.
return DebridMagnet(magnet_url=f"magnet:?xt=urn:btih:{info.info_hash}")
reason = info.fetch_error or "no magnet link, info hash, or torrent file was available"
msg = f"Could not resolve a torrent to send from {url[:120]} ({reason})"
raise ValueError(msg)
def extract_torrent_info(
url: str,
*,
@@ -379,6 +434,56 @@ def extract_info_hash_from_torrent(torrent_data: bytes) -> str | None:
return None
def _decode_torrent_text(value: object) -> str | None:
if isinstance(value, bytes):
return value.decode("utf-8", errors="replace")
if isinstance(value, str):
return value
return None
def extract_file_list_from_torrent(torrent_data: bytes) -> list[PackFile] | None:
"""List the files a .torrent describes, release-relative, without downloading it.
Multi-file torrents nest every path under the torrent name (which becomes the
client's save folder); single-file torrents are just the named file.
"""
try:
decoded, _ = bencode_decode(torrent_data)
except _TORRENT_PARSE_ERRORS as e:
logger.debug("Failed to parse torrent file list: %s", e)
return None
if not isinstance(decoded, dict):
return None
info = decoded.get(b"info")
if not isinstance(info, dict):
return None
name = _decode_torrent_text(info.get(b"name")) or ""
raw_files = info.get(b"files")
if not isinstance(raw_files, list):
length = info.get(b"length")
if not name:
return None
return [PackFile(name, length if isinstance(length, int) else None)]
files: list[PackFile] = []
for entry in raw_files:
if not isinstance(entry, dict):
continue
raw_path = entry.get(b"path")
if not isinstance(raw_path, list):
continue
segments = [seg for seg in (_decode_torrent_text(part) for part in raw_path) if seg]
if not segments:
continue
if name:
segments.insert(0, name)
length = entry.get(b"length")
files.append(PackFile("/".join(segments), length if isinstance(length, int) else None))
return files
def extract_hash_from_magnet(magnet_url: str) -> str | None:
"""Extract info_hash from a magnet URL."""
if not magnet_url.startswith("magnet:"):
+120 -13
View File
@@ -12,6 +12,7 @@ from tqdm import tqdm
from shelfmark.bypass import BypassCancelledError, cookie_store
from shelfmark.bypass.challenge import challenge_marker
from shelfmark.core import search_deadline
from shelfmark.core.config import config as app_config
from shelfmark.core.logger import setup_logger
from shelfmark.core.request_helpers import coerce_bool, normalize_positive_int
@@ -52,6 +53,7 @@ _BYPASSER_ERRORS = (
RuntimeError,
TypeError,
ValueError,
network.RateLimitedError,
requests.exceptions.RequestException,
)
@@ -333,11 +335,34 @@ def html_get_page(
"""
# Normalise before the closures below capture it: they touch selector.last_failure,
# so it must be a concrete selector, not the Optional parameter.
selector = selector or network.AAMirrorSelector()
# A release search runs under a wall-clock budget (see shelfmark.core.search_deadline).
# Adopting it as the cancel flag is what makes the budget bite on a solve already in
# flight: the bypassers and the helper subprocess poll this flag but know nothing about
# deadlines. Only when the caller has no flag of its own - a queued download brings one
# and must keep it, and runs outside any search context anyway.
if cancel_flag is None:
cancel_flag = search_deadline.cancel_event()
def _result(html: str, response_url: str) -> str | tuple[str, str]:
if include_response_url:
return html, response_url
return html
def _fail(reason: str, response_url: str) -> str | tuple[str, str]:
"""Record why the fetch is giving up, then return the empty result.
Every give-up path returns an empty page, which is all the caller used to
see. Stashing the concrete reason on the shared selector lets the caller
surface it (see release_sources.direct_download) rather than reporting the
same generic "network restricted or mirrors blocked" for every cause.
"""
selector.last_failure = reason
return _result("", response_url)
def _run_bypasser(bypass_url: str) -> str | tuple[str, str]:
"""Run the active bypasser for one URL and return its result.
@@ -346,6 +371,13 @@ def html_get_page(
retry-loop branch above with `continue`, and with MAX_RETRY=1 there is no
later attempt for that branch to run on either.
"""
# Never start a minutes-long browser solve on a budget that has already run out:
# nothing downstream would get to report the real reason before the caller's
# deadline (or its reverse proxy) cut the request off.
if search_deadline.expired():
logger.info("Release search budget spent; not starting a bypass for %s", bypass_url)
return _fail(search_deadline.deadline_message(), bypass_url)
if status_callback:
status_callback("resolving", "Bypassing protection...")
try:
@@ -355,7 +387,24 @@ def html_get_page(
# bypasser that fails to load is still reported as a bypasser error.
request_activity_grace(status_callback, _bypass_grace_seconds())
result = get_bypassed_page(bypass_url, selector, cancel_flag)
return _result(result or "", bypass_url)
if result:
return _result(result, bypass_url)
return _fail(
"The protection bypasser returned an empty page — the challenge was "
"not solved. Check that FlareSolverr/the CF bypasser is reachable.",
bypass_url,
)
except network.RateLimitedError as e:
# Not a bypasser malfunction: the host is throttling this IP and a solve
# cannot help. Surface the wait as a plain failure so the search ends cleanly
# instead of looping another minutes-long solve against a 429.
logger.info("Skipping bypass (rate-limited): %s", e)
if status_callback:
try:
status_callback("resolving", "Rate limited, try again shortly")
except _STATUS_CALLBACK_ERRORS:
logger.debug("Rate-limit status callback failed", exc_info=True)
return _fail(str(e), bypass_url)
except _BYPASSER_ERRORS as e:
logger.warning("Bypasser error: %s: %s", type(e).__name__, e)
# Surface the real reason. Without this the caller only sees an empty
@@ -366,7 +415,13 @@ def html_get_page(
status_callback("error", f"Bypass failed: {type(e).__name__}: {e}")
except _STATUS_CALLBACK_ERRORS:
logger.debug("Bypass error status callback failed", exc_info=True)
return _result("", bypass_url)
if isinstance(e, BypassCancelledError):
# The budget trips the same cancel flag a user's cancel does, so tell them
# apart here - "cancelled" is a confusing thing to read when nobody did.
if search_deadline.expired():
return _fail(search_deadline.deadline_message(), bypass_url)
return _fail("The protection bypass was cancelled.", bypass_url)
return _fail(f"The protection bypasser failed: {type(e).__name__}: {e}", bypass_url)
finally:
release_activity_grace(status_callback)
@@ -408,19 +463,24 @@ def html_get_page(
retry_limit = (
retry if retry is not None else (configured_retry if configured_retry is not None else 1)
)
selector = selector or network.AAMirrorSelector()
original_url = url
current_url = selector.rewrite(original_url)
use_bypasser_now = use_bypasser
# Survives across attempts so a cookie won once is still presented on later retries.
handshake_cookies: dict[str, str] = {}
handshake_retries = 0
# Last transport error seen, so the exhausted-retries path can name the real
# cause (timeout, connection refused, DNS, ...) instead of a generic message.
last_error: Exception | None = None
for attempt in range(1, retry_limit + 1):
# Check for cancellation before each attempt
if cancel_flag and cancel_flag.is_set():
if search_deadline.expired():
logger.info("Release search budget spent before attempt %s", attempt)
return _fail(search_deadline.deadline_message(), current_url)
logger.info("html_get_page cancelled before attempt %s", attempt)
return _result("", current_url)
return _fail("The request was cancelled.", current_url)
cookies: dict[str, str] = {}
try:
@@ -447,8 +507,15 @@ def html_get_page(
current_url,
proxies=get_proxies(current_url),
timeout=REQUEST_TIMEOUT,
# Bypasser-derived cookies win: they came from a real solved challenge.
cookies={**handshake_cookies, **cookies},
# Handshake cookies win. They were issued by *this* exchange, so by
# definition they are fresher than anything the store holds, and the
# server is waiting to see them echoed back on the very next hop.
# Letting the store overwrite them meant a stored cookie of the same
# name (DDoS-Guard reuses __ddg1_/__ddg2_ for both) was replayed on
# every hop and the freshly issued value never left this process - the
# ?check=1 probe could then never terminate, so every request ended in
# the redirect-loop handoff and paid for a full browser solve.
cookies={**cookies, **handshake_cookies},
headers=headers,
allow_redirects=allow_redirects,
verify=get_ssl_verify(current_url),
@@ -522,7 +589,12 @@ def html_get_page(
redirect_host,
current_url,
)
return _result("", current_url)
return _fail(
f"The configured mirror {current_host} redirected to "
f"{redirect_host}; it may be down or seized. Point MIRROR at "
"a working host or switch to auto mode.",
current_url,
)
new_url = _try_rotation(original_url, current_url, selector)
if new_url:
@@ -541,7 +613,11 @@ def html_get_page(
redirect_host,
current_url,
)
return _result("", current_url)
return _fail(
"Every Anna's Archive mirror redirected away to a dead host — "
"all configured mirrors are unreachable.",
current_url,
)
# Same-host redirect (relative or absolute) - follow manually.
# DDoS-Guard gates AA /search behind a cookie probe: the 302 to
@@ -572,7 +648,12 @@ def html_get_page(
logger.warning(
"Redirect loop and no bypasser available, giving up: %s", current_url
)
return _result("", current_url)
return _fail(
"Anna's Archive is behind a protection challenge (endless "
"redirect loop) and no bypasser is enabled to solve it. Enable "
"FlareSolverr/the CF bypasser.",
current_url,
)
current_url = redirect_url
continue
@@ -582,6 +663,7 @@ def html_get_page(
return _result(response.text, response.url)
except Exception as e:
last_error = e
status = _get_status_code(e)
# The same DDoS-Guard rescue, for the loops the manual AA follower above hands
@@ -608,7 +690,10 @@ def html_get_page(
current_url = new_url
continue
logger.warning("403 error, mirrors exhausted: %s", current_url)
return _result("", current_url)
return _fail(
"Anna's Archive returned 403 (blocked) and all mirrors are exhausted.",
current_url,
)
if _is_cf_bypass_enabled() and not use_bypasser_now:
# Before switching to bypasser, check if cookies have become available
@@ -642,12 +727,24 @@ def html_get_page(
# Same reasoning as the redirect-loop handoffs.
return _run_bypasser(current_url)
logger.warning("403 error, giving up: %s", current_url)
return _result("", current_url)
return _fail(
"Anna's Archive returned 403 (blocked) and no bypasser is enabled "
"to solve the protection challenge.",
current_url,
)
# 404 = Not found
if status == _HTTP_STATUS_NOT_FOUND:
logger.warning("404 error: %s", current_url)
return _result("", current_url)
return _fail(
f"Anna's Archive returned 404 Not Found for {current_url}.", current_url
)
# 429 = origin throttling this IP. Arm the per-host backoff so selection and
# the bypasser stop hammering it, then fall through to normal rotation onto a
# mirror that is not (yet) rate-limited.
if status == _HTTP_STATUS_RATE_LIMITED:
network.note_rate_limited(current_url)
# Try mirror/DNS rotation on retryable errors. A failure that proves the
# mirror is unusable also drops it from this process's rotation, so the
@@ -676,7 +773,16 @@ def html_get_page(
else:
logger.exception("Giving up after %s attempts: %s", retry_limit, current_url)
return _result("", current_url)
if last_error is not None:
return _fail(
f"Could not reach Anna's Archive after {retry_limit} attempt(s): "
f"{type(last_error).__name__}: {last_error}",
current_url,
)
return _fail(
"Could not reach Anna's Archive — all mirrors were exhausted without a usable response.",
current_url,
)
def download_url(
@@ -793,6 +899,7 @@ def download_url(
# Rate limited - skip to next source immediately
# (waiting doesn't help with concurrent downloads hitting the same server)
if status == _HTTP_STATUS_RATE_LIMITED:
network.note_rate_limited(current_url)
logger.info("Rate limited (429) - trying next source")
if status_callback:
status_callback("resolving", "Server busy, trying next")
+116 -3
View File
@@ -3,12 +3,13 @@
import fnmatch
import ipaddress
import socket
import time
import urllib.parse
import urllib.request
from datetime import UTC, datetime, timedelta
from http import HTTPStatus
from socket import AddressFamily, SocketKind
from typing import TYPE_CHECKING, Any, cast
from typing import TYPE_CHECKING, Any, NamedTuple, cast
import dns.resolver
import httpx
@@ -287,6 +288,108 @@ _dead_aa_urls: set[str] = set()
_dead_aa_urls_lock = _RLock()
# Per-host rate-limit backoff. A 429 is the origin throttling *this IP*, not a challenge:
# a DDoS-Guard/Cloudflare solve still renders, so the bypass "succeeds" yet the cleared
# request is rejected again and the throttle is only renewed. The single answer is to
# wait, so a 429 sidelines the host for a growing window - mirror selection and the
# bypasser both skip a cooling-down host until its deadline passes. The wait escalates
# 2 -> 5 -> 10 -> 15 -> 30 minutes each time the host throttles us again *after* we
# already waited a full window out; a host left clear for longer than the top step
# starts the ladder over. Keyed by host so every mirror and source shares one view;
# in-memory only, so a restart starts clean.
_RATE_LIMIT_COOLDOWN_LADDER_SECONDS: tuple[float, ...] = (120.0, 300.0, 600.0, 900.0, 1800.0)
# A host that has been clear this long is treated as a fresh episode: the next 429
# restarts the ladder at 2 minutes rather than resuming the escalation.
_RATE_LIMIT_RESET_AFTER_SECONDS = 1800.0
class _Cooldown(NamedTuple):
"""One host's active rate-limit window and how far up the ladder it has climbed."""
deadline: float # time.monotonic() value at which the wait expires
level: int # index into _RATE_LIMIT_COOLDOWN_LADDER_SECONDS
_host_cooldowns: dict[str, _Cooldown] = {}
_host_cooldowns_lock = _RLock()
class RateLimitedError(Exception):
"""Raised to abandon a request whose host is in a 429 cooldown.
Not a transport failure - nothing is wrong with the network, the origin is
throttling this IP and only time clears it. Callers surface it as a plain failure
rather than retrying or handing the URL to the bypasser.
"""
def _cooldown_key(url: str) -> str:
"""Host a cooldown is keyed by; '' when the URL carries none."""
return (urllib.parse.urlparse(url).hostname or "").lower()
def note_rate_limited(url: str) -> float:
"""Escalate a host's 429 backoff and (re)arm its cooldown; return the wait applied.
The step advances only when a fresh 429 arrives *after* the previous window already
elapsed - i.e. we waited it out and the host throttled us again. A 429 that lands
while the host is still cooling is the same episode: it neither escalates the level
nor shortens the wait. See the ladder note above.
"""
host = _cooldown_key(url)
if not host:
return 0.0
now = time.monotonic()
ladder = _RATE_LIMIT_COOLDOWN_LADDER_SECONDS
with _host_cooldowns_lock:
prev = _host_cooldowns.get(host)
if prev is not None and now < prev.deadline:
# Still inside the current window - same throttling episode, leave it be.
return prev.deadline - now
if prev is None or now - prev.deadline > _RATE_LIMIT_RESET_AFTER_SECONDS:
level = 0
else:
level = min(prev.level + 1, len(ladder) - 1)
wait = ladder[level]
_host_cooldowns[host] = _Cooldown(deadline=now + wait, level=level)
logger.info(
"Rate limited (429): backing off %s for %.0fs (step %d/%d)",
host,
wait,
level + 1,
len(ladder),
)
return wait
def host_cooldown_remaining(url: str) -> float:
"""Seconds left on a host's 429 cooldown; 0.0 when clear or expired.
Leaves an expired record in place: the ladder level it carries is what a later 429
escalates from (or resets, once the clear gap is long enough).
"""
host = _cooldown_key(url)
if not host:
return 0.0
now = time.monotonic()
with _host_cooldowns_lock:
rec = _host_cooldowns.get(host)
if rec is None or rec.deadline <= now:
return 0.0
return rec.deadline - now
def is_host_cooling_down(url: str) -> bool:
"""True while ``url``'s host is inside its 429 cooldown window."""
return host_cooldown_remaining(url) > 0.0
def clear_host_cooldowns() -> None:
"""Forget all rate-limit cooldowns (manual reset / tests)."""
with _host_cooldowns_lock:
_host_cooldowns.clear()
def _ensure_initialized() -> None:
"""Lazy guard so runtime setup happens once and late calls still work."""
global _initialized
@@ -1418,8 +1521,13 @@ def get_available_aa_urls() -> list[str]:
if not alive and _aa_urls:
logger.warning("All AA mirrors quarantined; retrying the full list")
_dead_aa_urls.clear()
return _aa_urls.copy()
return alive
alive = _aa_urls.copy()
# Prefer mirrors that are not serving a 429 cooldown so rotation stops hammering a
# throttled host. When every live mirror is cooling, keep the full live list rather
# than returning nothing: selection must never be left with nowhere to point, and
# the bypasser's fail-fast reports the "all rate-limited" case with a clear error.
breathing = [url for url in alive if not is_host_cooling_down(url)]
return breathing or alive
def _aa_base_for_url(url: str) -> str:
@@ -1486,6 +1594,11 @@ class AAMirrorSelector:
def __init__(self) -> None:
"""Initialize mirror state from the current AA configuration."""
# Set by html_get_page at each give-up path so a caller that only sees the
# returned empty page can still report *why* the fetch produced nothing
# (403, 404, redirect loop, bypasser error, mirrors exhausted, ...) instead
# of a blanket "network restricted" guess. None means "no failure recorded".
self.last_failure: str | None = None
self._ensure_fresh_state(reset_attempts=True)
def _ensure_fresh_state(self, *, reset_attempts: bool = False) -> None:
+35
View File
@@ -265,6 +265,8 @@ def queue_release(
series_position = release_data.get("series_position") or extra.get("series_position")
subtitle = release_data.get("subtitle") or extra.get("subtitle")
language = release_data.get("language") or extra.get("language")
multi_book = bool(release_data.get("multi_book") or extra.get("multi_book"))
book_plan = _normalize_book_plan(release_data.get("book_plan") or extra.get("book_plan"))
books_output_mode = (
str(config.get("BOOKS_OUTPUT_MODE", "folder", user_id=user_id) or "folder")
@@ -300,6 +302,8 @@ def queue_release(
series_position=series_position,
subtitle=subtitle,
language=language,
multi_book=multi_book or book_plan is not None,
book_plan=book_plan,
search_mode=search_mode,
output_mode=output_mode,
output_args=output_args,
@@ -408,6 +412,33 @@ def can_retry_download_task(
return _has_staged_retry_source(task)
def _normalize_book_plan(value: object) -> list[dict[str, Any]] | None:
"""Keep only well-formed pack books: a title plus a non-empty list of file paths."""
if not isinstance(value, list):
return None
books: list[dict[str, Any]] = []
for entry in value:
if not isinstance(entry, dict):
continue
title = normalize_optional_text(entry.get("title"))
raw_files = entry.get("files")
if title is None or not isinstance(raw_files, list):
continue
files = [f for f in raw_files if isinstance(f, str) and f.strip()]
if not files:
continue
year = entry.get("year")
books.append(
{
"title": title,
"series_position": _optional_number(entry.get("series_position")),
"year": year if isinstance(year, int) and not isinstance(year, bool) else None,
"files": files,
}
)
return books or None
def serialize_task_for_retry(task: DownloadTask) -> dict[str, Any]:
"""Serialize the task state needed for restart-safe retries."""
raw_search_mode = getattr(task, "search_mode", None)
@@ -437,6 +468,8 @@ def serialize_task_for_retry(task: DownloadTask) -> dict[str, Any]:
"subtitle": getattr(task, "subtitle", None),
"language": getattr(task, "language", None),
"search_mode": search_mode,
"multi_book": bool(getattr(task, "multi_book", False)),
"book_plan": _normalize_book_plan(getattr(task, "book_plan", None)),
"output_mode": getattr(task, "output_mode", None),
"output_args": dict(raw_output_args) if isinstance(raw_output_args, dict) else {},
"user_id": getattr(task, "user_id", None),
@@ -495,6 +528,8 @@ def _restore_task_from_retry_payload(payload: object) -> DownloadTask | None:
subtitle=normalize_optional_text(payload.get("subtitle")),
language=normalize_optional_text(payload.get("language")),
search_mode=search_mode,
multi_book=bool(payload.get("multi_book", False)),
book_plan=_normalize_book_plan(payload.get("book_plan")),
output_mode=normalize_optional_text(payload.get("output_mode")),
output_args=dict(output_args) if isinstance(output_args, dict) else {},
user_id=normalize_positive_int(payload.get("user_id")),
+11 -1
View File
@@ -105,6 +105,7 @@ def process_folder_output(
maybe_run_custom_script,
prepare_output_files,
record_step,
resolve_book_groups,
transfer_book_files,
)
@@ -205,6 +206,7 @@ def process_folder_output(
is_torrent=is_torrent,
preserve_source=preserve_source,
organization_mode=plan.organization_mode,
source_root=source_path,
)
if error:
@@ -259,7 +261,15 @@ def process_folder_output(
prepared.cleanup_paths,
)
message = "Complete" if len(final_paths) == 1 else f"Complete ({len(final_paths)} files)"
pack_groups = resolve_book_groups(
task, prepared.files, organization_mode=plan.organization_mode
)
if pack_groups is not None:
message = f"Complete ({len(pack_groups)} books, {len(final_paths)} files)"
elif len(final_paths) == 1:
message = "Complete"
else:
message = f"Complete ({len(final_paths)} files)"
status_callback("complete", message)
return str(final_paths[0])
+423
View File
@@ -0,0 +1,423 @@
"""Multi-book ("pack") release planning.
A pack is one release that contains several books: a whole-series torrent with one
subfolder per book, or a flat folder of `Series 1.0 - Title.m4b` files. The same
planning rules serve pre-download inspection (the file list comes from the release
source) and post-processing (the file list comes from disk), so what the user
approved in the modal is what gets filed.
"""
from __future__ import annotations
import os
import re
from dataclasses import dataclass
from pathlib import Path, PurePosixPath
from shelfmark.core.utils import AUDIOBOOK_FORMATS
# m4b/m4a hold a whole audiobook in one file; every other audio format (mp3, flac, ...) is
# chaptered - many files make up one book. Ebook formats are always one file per book, so
# only chaptered *audio* matters here. A flat folder is split one-book-per-file only when
# none of its files are chaptered audio: a bare list of `01 - Chapter.mp3` tracks is a
# single chaptered audiobook, not a pack of books.
_SINGLE_FILE_AUDIO_CONTAINERS = frozenset({"m4b", "m4a"})
_CHAPTERED_AUDIO_EXTENSIONS = frozenset(AUDIOBOOK_FORMATS) - _SINGLE_FILE_AUDIO_CONTAINERS
_YEAR_SUFFIX_RE = re.compile(r"\s*\(\s*(?P<year>\d{4})\s*\)\s*$")
_SERIES_MARKER_RE = re.compile(
r"""
^\s*
(?:
\[\s*\#?(?P<bracket>\d+(?:\.\d+)?)\s*\] # [03] / [#3]
| \#(?P<hash>\d+(?:\.\d+)?) # #3
| book\.?\s*(?P<book>\d+(?:\.\d+)?) # Book 3 / Book. 03
| (?P<plain>\d+(?:\.\d+)?)(?=[\s\-:.]) # 03 - / 1.0 - / 3.
)
\s*(?:[-:.]\s*)?
""",
re.IGNORECASE | re.VERBOSE,
)
_SEPARATOR_CHARS = " \t-_:."
# "Gods of Risk 2.5 - Gods of Risk": the title repeated on both sides of the position.
_REPEATED_TITLE_RE = re.compile(
r"^(?P<left>.+?)\s+(?P<position>\d+(?:\.\d+)?)\s*[-:\u2013]\s*(?P<right>.+)$"
)
_SERIES_LABEL_WORDS = r"(?:novella|novellas|short\s+story|short|story|novel)"
# "Uncrowned Cradle, Book 7" / "Reaper Cradle, Volume 10" / "Wintersteel (Cradle, Book 8)":
# an explicit word marks the position at the END of the name. A bare trailing number
# is deliberately not matched — "Title - 02" is a chapter, not a series position.
_TRAILING_MARKER_RE = re.compile(
r"""
[\s,\-:\u2013(]*
(?:book|volume|vol\.?)\s*\#?(?P<position>\d+(?:\.\d+)?)
\s*\)?\s*$
""",
re.IGNORECASE | re.VERBOSE,
)
# AudiobookBay renders a file inside a folder as "<folder> <file>" with no separator,
# so a pack row reads "Author - Title Series, Book 1 Title Series, Book 1".
_GLUED_FOLDER_RE = re.compile(
r"^(?P<prefix>.+?\s[-\u2013]\s)?(?P<core>.+?)\s+(?P=core)$", re.IGNORECASE
)
@dataclass(frozen=True)
class PackFile:
"""One file inside a release, path relative to the release root."""
path: str
size: int | None = None
@dataclass(frozen=True)
class PackBook:
"""One book split out of a pack, files as release-relative paths."""
title: str
series_position: float | None
year: int | None
files: list[str]
@dataclass(frozen=True)
class PackPlan:
books: list[PackBook]
ignored: list[str]
@property
def is_pack(self) -> bool:
return len(self.books) > 1
@dataclass(frozen=True)
class BookGroup:
"""One book's on-disk files, ready for transfer."""
title: str
series_position: float | None
year: int | None
files: list[Path]
def _strip_series_name(name: str, series_name: str | None) -> str:
if not series_name:
return name
prefix = series_name.strip()
if not prefix or not name.lower().startswith(prefix.lower()):
return name
remainder = name[len(prefix) :]
if remainder and remainder[0].isalnum():
return name
return remainder.lstrip(_SEPARATOR_CHARS)
def _strip_series_label(work: str, series_name: str | None) -> str:
"""Drop a leading "An <Series> Novella - " style label that some packs prepend."""
if not series_name:
return work
# "The Expanse" is labelled "An Expanse Novella", so match without the article.
core = re.sub(r"^(?:the|an?)\s+", "", series_name.strip(), flags=re.IGNORECASE)
if not core:
return work
pattern = re.compile(
rf"^(?:an?\s+|the\s+)?{re.escape(core)}\s+{_SERIES_LABEL_WORDS}\s*[-:\u2013]\s*",
re.IGNORECASE,
)
return pattern.sub("", work, count=1)
def _collapse_glued_folder(name: str) -> str:
match = _GLUED_FOLDER_RE.match(name)
if not match:
return name
prefix = match.group("prefix") or ""
core = match.group("core")
# "Author - X X" → "Author - X" (the folder carried the author, the file did not).
return (prefix + core).strip()
def _strip_author_name(name: str, author_name: str | None) -> str:
"""Drop a leading "Author - " (packs are often filed as `Author - Title`)."""
if not author_name:
return name
prefix = author_name.strip()
if not prefix or not name.lower().startswith(prefix.lower()):
return name
remainder = name[len(prefix) :]
stripped = remainder.lstrip(_SEPARATOR_CHARS + "\u2013")
if stripped == remainder: # no separator after the author: part of the title
return name
return stripped
def _strip_trailing_series_name(work: str, series_name: str | None) -> str:
"""Drop a trailing series name left behind by a trailing position marker."""
if not series_name:
return work
suffix = series_name.strip()
if not suffix or not work.lower().endswith(suffix.lower()):
return work
remainder = work[: -len(suffix)]
stripped = remainder.rstrip(_SEPARATOR_CHARS + ",(\u2013")
if not stripped or stripped == remainder:
return work
return stripped
def parse_pack_book_name(
name: str, *, series_name: str | None, author_name: str | None = None
) -> tuple[str, float | None, int | None]:
"""Split a book folder/file-stem name into (title, series position, year).
Strips a leading series name, a leading position marker (`Book 3 - `, `03 - `,
`1.0 - `, `3. `, `[03] `, `#3 `) and a trailing `(YYYY)`. Also understands a
trailing marker (`Title Series, Book 3`, `Title (Series, Volume 3)`), a leading
`Author - `, and AudiobookBay's glued `<folder> <file>` names. Returns the name
unchanged with no position/year when nothing would be left of the title.
"""
work = _collapse_glued_folder(name.strip())
work = _strip_author_name(work, author_name)
work = _strip_series_name(work, series_name)
year: int | None = None
year_match = _YEAR_SUFFIX_RE.search(work)
if year_match:
year = int(year_match.group("year"))
work = work[: year_match.start()]
position: float | None = None
repeated = _REPEATED_TITLE_RE.match(work.strip())
if (
repeated
and repeated.group("left").strip().lower() == repeated.group("right").strip().lower()
):
return repeated.group("right").strip(), float(repeated.group("position")), year
marker = _SERIES_MARKER_RE.match(work)
if marker:
raw = (
marker.group("bracket")
or marker.group("hash")
or marker.group("book")
or marker.group("plain")
)
position = float(raw)
work = work[marker.end() :]
else:
trailing = _TRAILING_MARKER_RE.search(work)
if trailing and trailing.start() > 0:
position = float(trailing.group("position"))
work = _strip_trailing_series_name(work[: trailing.start()], series_name)
work = _strip_series_label(work, series_name)
title = work.strip().strip(_SEPARATOR_CHARS).strip()
if not title:
return name, None, None
return title, position, year
def _book_from_name(
name: str, files: list[str], series_name: str | None, author_name: str | None = None
) -> PackBook:
title, position, year = parse_pack_book_name(
name, series_name=series_name, author_name=author_name
)
return PackBook(title=title, series_position=position, year=year, files=files)
def _common_root_parts(paths: list[PurePosixPath]) -> tuple[str, ...]:
parents = [p.parent.parts for p in paths]
common: list[str] = []
for parts in zip(*parents, strict=False):
if len(set(parts)) != 1:
break
common.append(parts[0])
return tuple(common)
def plan_pack(
files: list[PackFile],
*,
supported_extensions: set[str],
series_name: str | None,
author_name: str | None = None,
root_depth: int | None = None,
) -> PackPlan:
"""Group a release's file list into books.
Files in a subfolder (relative to the common root) group by that subfolder. Files
directly in the root split one-book-per-file only when at least two of them carry
a series position in their names; otherwise they are one book (a chaptered
audiobook, e.g. `01.mp3`, `02.mp3`). `root_depth` fixes how many leading path
components form the root instead of deriving it from the files' common parent.
"""
supported = {ext.lower().lstrip(".") for ext in supported_extensions}
book_files: list[PurePosixPath] = []
ignored: list[str] = []
for pack_file in files:
rel = PurePosixPath(pack_file.path.replace("\\", "/").lstrip("./"))
if rel.suffix.lower().lstrip(".") in supported:
book_files.append(rel)
else:
ignored.append(pack_file.path)
if not book_files:
return PackPlan(books=[], ignored=ignored)
root_parts = (
_common_root_parts(book_files) if root_depth is None else book_files[0].parts[:root_depth]
)
depth = len(root_parts)
root_files: list[PurePosixPath] = []
folders: dict[str, list[str]] = {}
for rel in book_files:
remainder = rel.parts[depth:]
if len(remainder) > 1:
folders.setdefault(remainder[0], []).append(str(rel))
else:
root_files.append(rel)
books: list[PackBook] = []
if root_files:
parsed = [
parse_pack_book_name(f.stem, series_name=series_name, author_name=author_name)
for f in root_files
]
positions = {p[1] for p in parsed if p[1] is not None}
titles = {p[0].strip().lower() for p in parsed if p[0]}
one_book_per_file = all(
rel.suffix.lower().lstrip(".") not in _CHAPTERED_AUDIO_EXTENSIONS for rel in root_files
)
# Split a flat folder into a book per file only with real evidence of distinct
# books: two or more series positions, more than one title, and no chaptered audio
# (a bare list of `01 - Chapter.mp3` tracks is one book, not a pack).
if len(positions) >= 2 and len(titles) >= 2 and one_book_per_file:
books.extend(
PackBook(title=title, series_position=position, year=year, files=[str(f)])
for f, (title, position, year) in zip(root_files, parsed, strict=True)
)
elif len(root_files) == 1:
books.append(
_book_from_name(root_files[0].stem, [str(root_files[0])], series_name, author_name)
)
else:
group_name = root_parts[-1] if root_parts else ""
books.append(
_book_from_name(group_name, [str(f) for f in root_files], series_name, author_name)
)
books.extend(
_book_from_name(folder, paths, series_name, author_name)
for folder, paths in folders.items()
)
return PackPlan(books=books, ignored=ignored)
def _relative_paths(
book_files: list[Path], root: Path | None = None
) -> tuple[Path, dict[Path, str]]:
if root is None:
root = Path(os.path.commonpath([str(f.parent) for f in book_files]))
return root, {f: f.relative_to(root).as_posix() for f in book_files}
def group_files_into_books(
book_files: list[Path],
*,
series_name: str | None,
author_name: str | None = None,
root: Path | None = None,
) -> list[BookGroup]:
"""Heuristically split on-disk files into books (see `plan_pack`).
`root` pins the release root when grouping a subset of a larger file set.
"""
if not book_files:
return []
_root, rel_by_path = _relative_paths(book_files, root)
path_by_rel = {rel: path for path, rel in rel_by_path.items()}
extensions = {f.suffix.lower().lstrip(".") for f in book_files}
plan = plan_pack(
[PackFile(rel) for rel in rel_by_path.values()],
supported_extensions=extensions,
series_name=series_name,
author_name=author_name,
root_depth=None if root is None else 0,
)
return [
BookGroup(
title=book.title,
series_position=book.series_position,
year=book.year,
files=[path_by_rel[rel] for rel in book.files],
)
for book in plan.books
]
def match_plan_to_files(
plan: list[PackBook],
book_files: list[Path],
*,
series_name: str | None = None,
author_name: str | None = None,
) -> list[BookGroup]:
"""Apply an approved plan to on-disk files.
Files match by release-relative path first, then by basename (archive extraction
and client save paths can shift the root), then by the on-disk basename being a
suffix of the planned name (sources that glue folder and file names together).
Book files the plan does not mention fall back to heuristic grouping so nothing
is silently dropped.
"""
if not book_files:
return []
root, rel_by_path = _relative_paths(book_files)
by_rel = {rel: path for path, rel in rel_by_path.items()}
by_name: dict[str, list[Path]] = {}
for path in book_files:
by_name.setdefault(path.name, []).append(path)
claimed: set[Path] = set()
groups: list[BookGroup] = []
for book in plan:
matched: list[Path] = []
for wanted in book.files:
wanted_rel = wanted.replace("\\", "/").lstrip("./")
candidate = by_rel.get(wanted_rel)
if candidate is None:
candidates = [
p for p in by_name.get(PurePosixPath(wanted_rel).name, []) if p not in claimed
]
candidate = candidates[0] if candidates else None
if candidate is None:
wanted_name = PurePosixPath(wanted_rel).name.lower()
candidates = [
p
for p in book_files
if p not in claimed and wanted_name.endswith(p.name.lower())
]
candidate = candidates[0] if len(candidates) == 1 else None
if candidate is not None and candidate not in claimed:
claimed.add(candidate)
matched.append(candidate)
if matched:
groups.append(
BookGroup(
title=book.title,
series_position=book.series_position,
year=book.year,
files=matched,
)
)
unmatched = [p for p in book_files if p not in claimed]
if unmatched:
groups.extend(
group_files_into_books(
unmatched, series_name=series_name, author_name=author_name, root=root
)
)
return groups
@@ -40,6 +40,7 @@ from .transfer import (
build_metadata_dict,
is_torrent_source,
process_directory,
resolve_book_groups,
resolve_hardlink_source,
should_hardlink,
transfer_book_files,
@@ -80,6 +81,7 @@ __all__ = [
"process_directory",
"record_step",
"resolve_custom_script_target",
"resolve_book_groups",
"resolve_hardlink_source",
"run_custom_script",
"safe_cleanup_path",
+1 -1
View File
@@ -52,7 +52,7 @@ def get_file_organization(*, is_audiobook: bool) -> str:
"""Get the file organization mode for the content type."""
key = "FILE_ORGANIZATION_AUDIOBOOK" if is_audiobook else "FILE_ORGANIZATION"
mode = _config_text(core_config.config.get(key, "rename")).strip().lower()
return mode if mode in ("none", "rename", "organize") else "rename"
return mode if mode in ("none", "rename", "rename_and_group", "organize") else "rename"
def get_template(*, is_audiobook: bool, organization_mode: str) -> str:
+138 -1
View File
@@ -2,6 +2,7 @@
from __future__ import annotations
import dataclasses
import os
from pathlib import Path
from typing import TYPE_CHECKING
@@ -17,6 +18,7 @@ from shelfmark.core.naming import (
sanitize_filename,
)
from shelfmark.core.utils import is_audiobook as check_audiobook
from shelfmark.download.archive import is_archive
from shelfmark.download.fs import (
atomic_copy,
atomic_hardlink,
@@ -25,6 +27,7 @@ from shelfmark.download.fs import (
)
from shelfmark.download.postprocess.policy import get_file_organization, get_template
from .packs import BookGroup, PackBook, group_files_into_books, match_plan_to_files
from .scan import collect_directory_files, scan_directory_tree
from .types import TransferPlan
from .workspace import safe_cleanup_path
@@ -160,6 +163,24 @@ def _transfer_single_file(
return atomic_move(source_path, dest_path, max_attempts=max_attempts), "move"
def _group_folder_name(source_root: Path | None) -> str:
"""Name the folder a grouped multi-file audiobook is transferred into.
A directory names the group directly. A file cannot hold several book files
on its own, so a non-directory source that produced more than one means
`collect_staged_files` extracted an archive: the stem is the release name and
the suffix is packaging, which is why `Book.zip` groups into `Book/` rather
than `Book.zip/` or, worse, not at all.
"""
if source_root is None:
return ""
if run_blocking_io(source_root.is_dir):
return sanitize_filename(source_root.name)
if is_archive(source_root):
return sanitize_filename(source_root.stem)
return ""
def transfer_book_files(
book_files: list[Path],
destination: Path,
@@ -169,6 +190,7 @@ def transfer_book_files(
is_torrent: bool,
preserve_source: bool = False,
organization_mode: str | None = None,
source_root: Path | None = None,
) -> tuple[list[Path], str | None, dict[str, int]]:
"""Transfer discovered book files into their final destination layout."""
if not book_files:
@@ -176,6 +198,19 @@ def transfer_book_files(
is_audiobook = check_audiobook(task.content_type)
organization_mode = organization_mode or get_file_organization(is_audiobook=is_audiobook)
groups = resolve_book_groups(task, book_files, organization_mode=organization_mode)
if groups is not None:
return _transfer_book_groups(
groups,
destination,
task,
use_hardlink=use_hardlink,
is_torrent=is_torrent,
preserve_source=preserve_source,
organization_mode=organization_mode,
)
max_attempts = _max_attempts_for_batch(len(book_files))
final_paths: list[Path] = []
@@ -238,6 +273,13 @@ def transfer_book_files(
return final_paths, None, op_counts
transfer_destination = destination
if is_audiobook and len(book_files) > 1 and organization_mode == "rename_and_group":
source_folder = _group_folder_name(source_root)
if source_folder:
transfer_destination = destination / source_folder
run_blocking_io(transfer_destination.mkdir, parents=True, exist_ok=True)
for book_file in book_files:
if len(book_files) == 1 and organization_mode != "none":
if not task.format:
@@ -256,7 +298,7 @@ def transfer_book_files(
else:
filename = book_file.name
dest_path = destination / filename
dest_path = transfer_destination / filename
final_path, op = _transfer_single_file(
book_file,
dest_path,
@@ -272,6 +314,101 @@ def transfer_book_files(
return final_paths, None, op_counts
def resolve_book_groups(
task: DownloadTask,
book_files: list[Path],
*,
organization_mode: str,
) -> list[BookGroup] | None:
"""Split a multi-book pack into per-book groups, or None to file as one book.
An approved `book_plan` wins; a bare `multi_book` flag falls back to heuristic
grouping. Organization `none` keeps files as-is, and a split that yields a single
group is not a pack at all.
"""
if organization_mode == "none" or not (task.book_plan or task.multi_book):
return None
if task.book_plan:
plan = [
PackBook(
title=str(entry.get("title") or ""),
series_position=entry.get("series_position"),
year=entry.get("year"),
files=list(entry.get("files") or []),
)
for entry in task.book_plan
if isinstance(entry, dict)
]
groups = match_plan_to_files(
plan, book_files, series_name=task.series_name, author_name=task.author
)
else:
groups = group_files_into_books(
book_files, series_name=task.series_name, author_name=task.author
)
return groups if len(groups) > 1 else None
def _transfer_book_groups(
groups: list[BookGroup],
destination: Path,
task: DownloadTask,
*,
use_hardlink: bool,
is_torrent: bool,
preserve_source: bool,
organization_mode: str,
) -> tuple[list[Path], str | None, dict[str, int]]:
"""Transfer each book of a pack through the normal single-book path.
Each book gets an isolated task copy (the single-file path mutates `task.format`)
carrying its own title, position and year; the searched book's position must not
leak onto its siblings, while author and series name apply to all of them.
"""
all_paths: list[Path] = []
totals: dict[str, int] = {"hardlink": 0, "copy": 0, "move": 0}
errors: list[str] = []
for group in groups:
book_task = dataclasses.replace(
task,
title=group.title or task.title,
year=str(group.year) if group.year is not None else None,
subtitle=None,
series_position=group.series_position,
multi_book=False,
book_plan=None,
)
paths, error, op_counts = transfer_book_files(
group.files,
destination,
book_task,
use_hardlink=use_hardlink,
is_torrent=is_torrent,
preserve_source=preserve_source,
organization_mode=organization_mode,
source_root=group.files[0].parent,
)
for op, count in op_counts.items():
totals[op] = totals.get(op, 0) + count
if error:
errors.append(f"{group.title}: {error}")
logger.warning("Task %s: pack book %r failed: %s", task.task_id, group.title, error)
continue
all_paths.extend(paths)
if not all_paths:
return [], "; ".join(errors) or "No book files found", totals
if errors:
logger.warning(
"Task %s: pack filed with %d failed book(s): %s",
task.task_id,
len(errors),
"; ".join(errors),
)
return all_paths, None, totals
def process_directory(
directory: Path,
ingest_dir: Path,
+19
View File
@@ -32,6 +32,19 @@ _DEFAULT_QUERY = "The Great Gatsby"
_warmup_thread: threading.Thread | None = None
_warmup_lock = threading.Lock()
# Set as soon as a real release search starts. The warm-up exists to pay the cold path
# *before* the user does; once they have beaten it to the box there is nothing left to
# pre-solve, and running anyway is actively harmful - the bypasser serializes on one
# browser, so the warm-up's solve goes in front of the search the user is watching. In
# the bundle on issue #1276 that cost a full minute of a 2m27s wait, on a container 16
# seconds old, for a throwaway "The Great Gatsby" query nobody asked for.
_user_search_seen = threading.Event()
def note_user_search() -> None:
"""Record that a real search has run, so a pending warm-up stands down."""
_user_search_seen.set()
def _as_bool(value: object, *, default: bool) -> bool:
"""Coerce a config value that may arrive as a string, bool or None."""
@@ -85,6 +98,12 @@ def run_warmup() -> bool:
"""
from shelfmark.core.mirrors import has_aa_mirror_configuration
# Checked here rather than only at schedule time: the delay is what this races with,
# so the user's first search usually lands *during* the wait, not before it.
if _user_search_seen.is_set():
logger.info("Search warm-up skipped: a real search got there first")
return False
if not has_aa_mirror_configuration():
logger.debug("Search warm-up skipped: no Anna's Archive mirrors configured")
return False
+35 -9
View File
@@ -42,6 +42,7 @@ from shelfmark.config.settings import (
_SUPPORTED_BOOK_LANGUAGE,
migrate_audiobook_format_settings,
)
from shelfmark.core import search_deadline
from shelfmark.core.activity_view_state_service import ActivityViewStateService
from shelfmark.core.auth_modes import (
get_auth_check_admin_status,
@@ -62,6 +63,7 @@ from shelfmark.core.notifications import (
notify_user,
)
from shelfmark.core.prefix_middleware import PrefixMiddleware
from shelfmark.core.release_inspect_routes import register_release_inspect_routes
from shelfmark.core.request_helpers import (
coerce_bool,
emit_ws_event,
@@ -1024,6 +1026,9 @@ def _serialize_release(release: Release) -> dict:
return result
register_release_inspect_routes(app, login_required)
@app.route("/api/releases/download", methods=["POST"])
@login_required
def api_download_release() -> Response | tuple[Response, int]:
@@ -1150,7 +1155,7 @@ def api_config() -> Response | tuple[Response, int]:
"build_version": BUILD_VERSION,
"release_version": RELEASE_VERSION,
"book_languages": _SUPPORTED_BOOK_LANGUAGE,
"default_language": app_config.BOOK_LANGUAGE,
"default_language": app_config.get("BOOK_LANGUAGE", ["en"], user_id=db_user_id),
"supported_formats": app_config.SUPPORTED_FORMATS,
"supported_audiobook_formats": app_config.SUPPORTED_AUDIOBOOK_FORMATS,
"search_mode": search_mode,
@@ -2840,6 +2845,7 @@ def api_releases() -> Response | tuple[Response, int]:
manual_query=query_text if source_query_filters is not None else manual_query,
indexers=indexers,
source_filters=source_query_filters,
user_id=db_user_id,
)
if plan.source_filters is not None:
@@ -2892,6 +2898,8 @@ def api_releases() -> Response | tuple[Response, int]:
if languages_param
else None
)
# Without an explicit filter the plan falls back to this user's default languages.
db_user_id = get_session_db_user_id(session)
# Content type for audiobook vs ebook search
content_type = request.args.get("content_type", "ebook").strip()
@@ -2980,18 +2988,36 @@ def api_releases() -> Response | tuple[Response, int]:
# Search only enabled sources
sources_to_search = [src["name"] for src in list_available_sources() if src["enabled"]]
# Search each source for releases
# Search each source for releases.
#
# Under a wall-clock budget: this endpoint is synchronous, and the bypass path it
# can reach used to be allowed minutes per URL with nothing bounding the request
# as a whole. A search that ran into an unsolvable protection challenge therefore
# outlived every reverse proxy in front of it and surfaced to the user as
# "Server unavailable (504)" - a gateway timeout that blames their proxy for a
# challenge failure. The budget is shared across sources, so a stuck first source
# cannot spend the whole request on its own. See issue #1276.
all_releases = []
errors = []
source_instances = {} # Keep source instances for column config
for source_name in sources_to_search:
source, releases, error = _search_source_releases(source_name, book)
if source is not None:
source_instances[source_name] = source
all_releases.extend(releases)
if error is not None:
errors.append(error)
# A real search is under way, so a warm-up still sitting on its start-up delay
# should stand down rather than queue its throwaway solve in front of this one.
warmup.note_user_search()
with search_deadline.search_deadline():
for source_name in sources_to_search:
if search_deadline.expired():
logger.warning("Release search budget spent; %s not searched", source_name)
errors.append(f"{source_name}: {search_deadline.deadline_message()}")
continue
source, releases, error = _search_source_releases(source_name, book)
if source is not None:
source_instances[source_name] = source
all_releases.extend(releases)
if error is not None:
errors.append(error)
# Convert Release objects to dicts
releases_data = [_serialize_release(release) for release in all_releases]
+9
View File
@@ -13,6 +13,7 @@ if TYPE_CHECKING:
from shelfmark.core.models import DownloadTask
from shelfmark.core.search_plan import ReleaseSearchPlan
from shelfmark.download.postprocess.packs import PackFile
from shelfmark.metadata_providers import BookMetadata
@@ -400,6 +401,14 @@ class DownloadHandler(ABC):
"""Return private queue-time fields needed for restart-safe retry."""
return {}
def list_files(self, release_data: dict[str, Any]) -> list[PackFile] | None:
"""Return the release's file list without downloading it.
Lets the UI review a multi-book pack before queueing. Return None when the
source cannot know the files ahead of time (magnet links, usenet, ...).
"""
return None
@abstractmethod
def cancel(self, task_id: str) -> bool:
"""Cancel an in-progress download."""
@@ -1,6 +1,6 @@
"""AudiobookBay download handler - resolves magnet links and uses shared client lifecycle."""
from typing import TYPE_CHECKING
from typing import TYPE_CHECKING, Any
from urllib.parse import urlparse
from shelfmark.core.config import config
@@ -22,6 +22,7 @@ if TYPE_CHECKING:
from collections.abc import Callable
from shelfmark.core.models import DownloadTask
from shelfmark.download.postprocess.packs import PackFile
logger = setup_logger(__name__)
DEFAULT_ABB_HOSTNAME = "audiobookbay.lu"
@@ -68,6 +69,19 @@ class AudiobookBayHandler(ExternalClientHandler):
return task_id
return None
def list_files(self, release_data: dict[str, Any]) -> list[PackFile] | None:
"""Read the torrent's file list off the detail page, without downloading."""
raw_url = release_data.get("download_url") or release_data.get("source_url")
detail_url = raw_url.strip() if isinstance(raw_url, str) else ""
hostname = _resolve_allowed_detail_hostname()
if not detail_url or not _detail_url_matches_host(detail_url, hostname):
logger.debug("Cannot list files for AudiobookBay release without a valid detail URL")
return None
detail_html = scraper.fetch_detail_html(detail_url, hostname)
if not detail_html:
return None
return scraper.extract_file_list(detail_html)
def _get_client(self, protocol: str) -> DownloadClient | None:
"""Compatibility shim so module-level patching still works in tests."""
return get_client(protocol)
+107 -29
View File
@@ -2,6 +2,7 @@
import re
import time
from threading import Lock
from urllib.parse import quote, quote_plus
import requests
@@ -10,6 +11,7 @@ from bs4 import BeautifulSoup
from shelfmark.core.config import config
from shelfmark.core.logger import setup_logger
from shelfmark.download import http as downloader
from shelfmark.download.postprocess.packs import PackFile
from shelfmark.release_sources.audiobookbay.utils import normalize_search_punctuation
logger = setup_logger(__name__)
@@ -32,6 +34,13 @@ FIRST_PAGE_SESSION_REFRESH_ATTEMPTS = 2
# Legacy search parameter used by older ABB flows
LEGACY_CATEGORY_QUERY = "undefined%2Cundefined"
# Detail pages are fetched once and shared by inspection (file list) and download
# (magnet link) so a "review then download" round trip costs ABB a single request.
DETAIL_PAGE_CACHE_TTL_SECONDS = 120.0
DETAIL_PAGE_CACHE_MAX_ENTRIES = 8
_detail_page_cache: dict[str, tuple[float, str]] = {}
_detail_page_cache_lock = Lock()
# Precompiled patterns used while parsing result cards
LANGUAGE_PATTERN = re.compile(r"Language:\s*([A-Za-z]+)")
POSTED_PATTERN = re.compile(r"Posted:\s*(\d+\s+[A-Za-z]+\s+\d{4})")
@@ -39,6 +48,11 @@ FORMAT_PATTERN = re.compile(r"Format:\s*([A-Za-z0-9]+)")
BITRATE_PATTERN = re.compile(r"Bitrate:\s*([\d]+\s*[A-Za-z/]+)")
SIZE_PATTERN = re.compile(r"File Size:\s*([\d.]+)\s*([A-Za-z]+)")
INFO_HASH_LABEL_PATTERN = re.compile(r"Info Hash", re.IGNORECASE)
FILE_ROW_SIZE_PATTERN = re.compile(
r"^(?P<name>.+?)\s+(?P<size>\d+(?:\.\d+)?)\s*(?P<unit>Bytes?|KBs?|MBs?|GBs?|TBs?)$",
re.IGNORECASE,
)
_FILE_SIZE_MULTIPLIERS = {"b": 1, "k": 1024, "m": 1024**2, "g": 1024**3, "t": 1024**4}
def _coerce_non_negative_float(value: object, default: float) -> float:
@@ -348,6 +362,98 @@ def search_audiobookbay(
return results
def _get_cached_detail_page(details_url: str) -> str | None:
with _detail_page_cache_lock:
entry = _detail_page_cache.get(details_url)
if entry is None:
return None
fetched_at, html = entry
if time.monotonic() - fetched_at > DETAIL_PAGE_CACHE_TTL_SECONDS:
del _detail_page_cache[details_url]
return None
return html
def _store_cached_detail_page(details_url: str, html: str) -> None:
with _detail_page_cache_lock:
_detail_page_cache[details_url] = (time.monotonic(), html)
while len(_detail_page_cache) > DETAIL_PAGE_CACHE_MAX_ENTRIES:
oldest = min(_detail_page_cache, key=lambda key: _detail_page_cache[key][0])
del _detail_page_cache[oldest]
def clear_detail_page_cache() -> None:
"""Drop cached detail pages (used by tests)."""
with _detail_page_cache_lock:
_detail_page_cache.clear()
def _fetch_detail_page_once(details_url: str, hostname: str) -> str:
session = requests.Session()
_bootstrap_abb_session(hostname, session, DETAIL_PAGE_RETRY_ATTEMPTS)
return _coerce_markup_to_html(
downloader.html_get_page(
details_url,
retry=DETAIL_PAGE_RETRY_ATTEMPTS,
use_bypasser=False,
allow_bypasser_fallback=False,
success_delay=0,
session=session,
)
)
def fetch_detail_html(details_url: str, hostname: str = "audiobookbay.lu") -> str:
"""Fetch a detail page (one retry with a fresh session), cached briefly per URL."""
cached = _get_cached_detail_page(details_url)
if cached is not None:
logger.debug("Reusing recently fetched detail page: %s", details_url)
return cached
detail_html = _fetch_detail_page_once(details_url, hostname)
if not detail_html:
detail_html = _fetch_detail_page_once(details_url, hostname)
if detail_html:
_store_cached_detail_page(details_url, detail_html)
return detail_html
def _parse_file_row(text: str) -> PackFile | None:
match = FILE_ROW_SIZE_PATTERN.match(text.strip())
if not match:
return None
multiplier = _FILE_SIZE_MULTIPLIERS[match.group("unit")[0].lower()]
return PackFile(match.group("name"), int(float(match.group("size")) * multiplier))
def extract_file_list(detail_html: str) -> list[PackFile] | None:
"""Read the torrent file rows off a detail page.
ABB renders the torrent's file table as single-cell rows between the
"This is a Multifile Torrent" marker (absent for single-file torrents) and the
"Combined File Size" row. Returns None when the page has no such table.
"""
soup = BeautifulSoup(detail_html, "html.parser")
rows: list[PackFile] = []
for row in soup.find_all("tr"):
cells = row.find_all("td")
if not cells:
continue
label = cells[0].get_text(" ", strip=True)
if label.lower().startswith("combined file size"):
return rows or None
if len(cells) != 1:
rows = [] # a two-column metadata row means we're not in the file table yet
continue
text = cells[0].get_text(" ", strip=True)
if "multifile torrent" in text.lower():
rows = []
continue
parsed = _parse_file_row(text)
if parsed is not None:
rows.append(parsed)
return None
def extract_magnet_link(details_url: str, hostname: str = "audiobookbay.lu") -> str | None:
"""Extract info hash and trackers from book detail page, then construct magnet link.
@@ -360,35 +466,7 @@ def extract_magnet_link(details_url: str, hostname: str = "audiobookbay.lu") ->
"""
try:
session = requests.Session()
_bootstrap_abb_session(hostname, session, DETAIL_PAGE_RETRY_ATTEMPTS)
# Fetch detail page
detail_html = _coerce_markup_to_html(
downloader.html_get_page(
details_url,
retry=DETAIL_PAGE_RETRY_ATTEMPTS,
use_bypasser=False,
allow_bypasser_fallback=False,
success_delay=0,
session=session,
)
)
if not detail_html:
session = requests.Session()
_bootstrap_abb_session(hostname, session, DETAIL_PAGE_RETRY_ATTEMPTS)
detail_html = _coerce_markup_to_html(
downloader.html_get_page(
details_url,
retry=DETAIL_PAGE_RETRY_ATTEMPTS,
use_bypasser=False,
allow_bypasser_fallback=False,
success_delay=0,
session=session,
)
)
detail_html = fetch_detail_html(details_url, hostname)
if not detail_html:
logger.warning("Failed to fetch details page")
return None
+31 -6
View File
@@ -17,6 +17,7 @@ from bs4 import BeautifulSoup, Tag
from bs4.element import NavigableString
from shelfmark.config.env import DEBUG_SKIP_SOURCES, TMP_DIR
from shelfmark.core import search_deadline
from shelfmark.core.config import config
from shelfmark.core.languages import language_alias_map
from shelfmark.core.logger import setup_logger
@@ -586,13 +587,22 @@ def _fetch_search_table(url: str, selector: network.AAMirrorSelector) -> tuple[s
"""
attempt_url = url
for _ in range(len(network.get_available_aa_urls()) or 1):
# Every mirror shares the protection, so once the search budget is gone another
# mirror is another full solve nobody is still waiting for.
if search_deadline.expired():
raise SearchUnavailableError(search_deadline.deadline_message())
response = downloader.html_get_page(
attempt_url, selector=selector, allow_bypasser_fallback=True
)
if not response:
# Network/mirror exhaustion path bubbles up so API can notify clients
msg = "Unable to reach download source. Network restricted or mirrors are blocked."
raise SearchUnavailableError(msg)
# Network/mirror exhaustion path bubbles up so API can notify clients.
# html_get_page records the concrete give-up reason on the selector; fall
# back to the generic line only if nothing was recorded.
detail = getattr(selector, "last_failure", None) or (
"Network restricted or mirrors are blocked."
)
raise SearchUnavailableError(f"Unable to reach download source. {detail}")
html = _html_response_text(response)
soup = BeautifulSoup(html, "html.parser")
@@ -747,8 +757,10 @@ def get_book_info(book_id: str, *, fetch_download_count: bool = True) -> BrowseR
html = downloader.html_get_page(url, selector=selector, allow_bypasser_fallback=True)
if not html:
msg = "Unable to reach download source. Network restricted or mirrors are blocked."
raise SearchUnavailableError(msg)
detail = getattr(selector, "last_failure", None) or (
"Network restricted or mirrors are blocked."
)
raise SearchUnavailableError(f"Unable to reach download source. {detail}")
soup = BeautifulSoup(_html_response_text(html), "html.parser")
@@ -1967,6 +1979,12 @@ class DirectDownloadSource(ReleaseSource):
query = f"{title} {author}".strip()
if not query:
continue
# `except Exception` below keeps this loop going past a failed variant, which
# is right for a parse error and wrong for a spent budget: without this the
# variants queue up behind each other and the request outlives the caller.
if search_deadline.expired():
logger.info("Release search budget spent; skipping remaining title variants")
break
logger.debug("Searching direct_download: title_author='%s', langs=%s", query, langs)
filters = SearchFilters(lang=langs if langs is not None else [])
@@ -1980,7 +1998,11 @@ class DirectDownloadSource(ReleaseSource):
except Exception:
logger.exception("Search error")
if not all_results and any(langs for _, langs in searches):
if (
not all_results
and any(langs for _, langs in searches)
and not search_deadline.expired()
):
logger.debug(
"No title+author results with language filter, retrying without language filter"
)
@@ -1988,6 +2010,9 @@ class DirectDownloadSource(ReleaseSource):
query = f"{title} {author}".strip()
if not query:
continue
if search_deadline.expired():
logger.info("Release search budget spent; skipping remaining retries")
break
logger.debug("Searching direct_download: title_author='%s', langs=[]", query)
try:
+9 -8
View File
@@ -428,14 +428,15 @@ class IRCReleaseSource(ReleaseSource):
"m4b": 0,
"mp3": 1,
"m4a": 2,
"flac": 3,
"opus": 4,
"ogg": 5,
"aac": 6,
"wav": 7,
"wma": 8,
"rar": 9,
"zip": 10,
"mp4": 3,
"flac": 4,
"opus": 5,
"ogg": 6,
"aac": 7,
"wav": 8,
"wma": 9,
"rar": 10,
"zip": 11,
}
def _convert_to_releases(
+68 -8
View File
@@ -8,6 +8,7 @@ from shelfmark.core.settings_registry import (
HeadingField,
PasswordField,
SettingsField,
TableField,
TagListField,
TextField,
register_settings,
@@ -16,12 +17,36 @@ from shelfmark.core.utils import normalize_http_url
def _test_newznab_connection(current_values: dict[str, Any] | None = None) -> dict[str, Any]:
"""Test the Newznab connection using current form values."""
"""Test all named Newznab connections, or the legacy connection as fallback."""
from shelfmark.core.config import config
from shelfmark.release_sources.newznab.api import NewznabClient
from shelfmark.release_sources.newznab.source import _parse_indexer_rows
current_values = current_values or {}
raw_indexers = current_values.get("NEWZNAB_INDEXERS")
if raw_indexers is None:
raw_indexers = config.get("NEWZNAB_INDEXERS", [])
indexers = _parse_indexer_rows(raw_indexers)
if indexers:
details: list[str] = []
all_successful = True
for name, url, api_key in indexers:
try:
success, message = NewznabClient(url, api_key).test_connection()
except Exception as e: # noqa: BLE001 — surface unexpected errors to the UI
success, message = False, f"Connection failed: {e!s}"
all_successful = all_successful and success
details.append(f"{name}: {message}")
summary = (
f"Connected to all {len(indexers)} indexers"
if all_successful
else "One or more Newznab indexers failed"
)
return {"success": all_successful, "message": summary, "details": details}
raw_url = str(current_values.get("NEWZNAB_URL") or config.get("NEWZNAB_URL", "") or "")
api_key = str(current_values.get("NEWZNAB_API_KEY") or config.get("NEWZNAB_API_KEY", "") or "")
@@ -64,25 +89,60 @@ def newznab_config_settings() -> list[SettingsField]:
default=False,
description="Enable searching for books via a Newznab-compatible indexer",
),
TableField(
key="NEWZNAB_INDEXERS",
label="Named Indexers",
description=(
"Add each Newznab-compatible indexer separately. The configured name is shown "
"beside every result from that indexer."
),
columns=[
{
"key": "name",
"label": "Name",
"type": "text",
"placeholder": "NZBGeek",
},
{
"key": "url",
"label": "URL",
"type": "text",
"placeholder": "https://api.nzbgeek.info",
},
{
"key": "api_key",
"label": "API Key",
"type": "password",
"placeholder": "Optional",
},
],
default=[],
add_label="Add Indexer",
empty_message=(
"No named indexers configured. The legacy single-indexer fields below are used "
"as a fallback."
),
show_when={"field": "NEWZNAB_ENABLED", "value": True},
),
TextField(
key="NEWZNAB_URL",
label="Newznab URL",
description="Base URL of your Newznab indexer or aggregator",
label="Legacy Newznab URL",
description="Used only when the named indexer list is empty",
placeholder="http://nzbhydra:5076",
required=True,
required=False,
show_when={"field": "NEWZNAB_ENABLED", "value": True},
),
PasswordField(
key="NEWZNAB_API_KEY",
label="API Key",
description="Your Newznab API key (leave blank if not required)",
label="Legacy API Key",
description="Used only with the legacy Newznab URL",
required=False,
show_when={"field": "NEWZNAB_ENABLED", "value": True},
),
ActionButton(
key="test_newznab",
label="Test Connection",
description="Verify your Newznab configuration",
label="Test Connections",
description="Verify every named indexer, or the legacy connection when the list is empty",
style="primary",
callback=_test_newznab_connection,
show_when={"field": "NEWZNAB_ENABLED", "value": True},
+125 -30
View File
@@ -4,7 +4,10 @@ from __future__ import annotations
import re
import time
from dataclasses import dataclass
from hashlib import sha256
from typing import TYPE_CHECKING, ClassVar
from urllib.parse import urlparse
if TYPE_CHECKING:
from shelfmark.core.search_plan import ReleaseSearchPlan
@@ -48,6 +51,50 @@ _DEFAULT_BOOK_CATS = [7000]
NEWZNAB_SEARCH_TIMEOUT_SECONDS = _SEARCH_TIMEOUT
@dataclass(frozen=True)
class _NamedClient:
"""A configured Newznab connection and its stable cache namespace."""
name: str
connection_id: str
client: NewznabClient
def _parse_indexer_rows(raw: object) -> list[tuple[str, str, str]]:
"""Normalize structured Newznab indexer settings.
Invalid/incomplete rows are ignored so one partially edited row cannot disable
the other configured indexers.
"""
if not isinstance(raw, list):
return []
indexers: list[tuple[str, str, str]] = []
seen_connections: set[tuple[str, str]] = set()
for row in raw:
if not isinstance(row, dict):
continue
raw_url = str(row.get("url") or "").strip()
url = normalize_http_url(raw_url)
if not url:
if raw_url:
logger.warning("Newznab: ignoring indexer row with invalid URL '%s'", raw_url)
continue
api_key = str(row.get("api_key") or "").strip()
connection_key = (url, api_key)
if connection_key in seen_connections:
continue
seen_connections.add(connection_key)
configured_name = str(row.get("name") or "").strip()
hostname = urlparse(url).hostname or ""
name = configured_name or hostname or "Newznab"
indexers.append((name, url, api_key))
return indexers
def _parse_category_ids(raw: object) -> list[int]:
"""Parse a configured category setting into Newznab category IDs.
@@ -146,8 +193,11 @@ def _newznab_result_to_release(
else None
)
# Build source_id from GUID
source_id = result.get("guid") or f"newznab:{hash(raw_title)}"
# Namespace IDs from named connections so identical GUIDs returned by two
# indexers cannot overwrite one another in the private release cache.
raw_source_id = result.get("guid") or f"newznab:{hash(raw_title)}"
connection_id = str(result.get("_newznab_connection_id") or "").strip()
source_id = f"newznab:{connection_id}:{raw_source_id}" if connection_id else raw_source_id
# Cache the raw result for the handler
cache_release(source_id, result)
@@ -272,6 +322,7 @@ class NewznabSource(ReleaseSource):
)
def _get_client(self) -> NewznabClient | None:
"""Build the legacy single-indexer client."""
raw_url = str(config.get("NEWZNAB_URL", "") or "")
api_key = str(config.get("NEWZNAB_API_KEY", "") or "")
@@ -284,6 +335,28 @@ class NewznabSource(ReleaseSource):
return NewznabClient(url, api_key or "")
def _get_clients(self) -> list[_NamedClient]:
"""Build named clients, falling back to the legacy single connection."""
configured = _parse_indexer_rows(config.get("NEWZNAB_INDEXERS", []))
if configured:
clients: list[_NamedClient] = []
for name, url, api_key in configured:
digest = sha256(f"{name}\0{url}\0{api_key}".encode()).hexdigest()[:16]
clients.append(
_NamedClient(
name=name,
connection_id=digest,
client=NewznabClient(url, api_key),
)
)
return clients
legacy_client = self._get_client()
if legacy_client is None:
return []
legacy_name = str(config.get("NEWZNAB_NAME", "") or "").strip() or "Newznab"
return [_NamedClient(name=legacy_name, connection_id="legacy", client=legacy_client)]
def search(
self,
book: BookMetadata,
@@ -293,8 +366,8 @@ class NewznabSource(ReleaseSource):
content_type: str = "ebook",
) -> list[Release]:
"""Search the Newznab indexer for releases matching the book."""
client = self._get_client()
if not client:
clients = self._get_clients()
if not clients:
logger.warning("Newznab not configured - skipping search")
return []
@@ -324,40 +397,60 @@ class NewznabSource(ReleaseSource):
all_results: list[dict] = []
try:
for idx, query in enumerate(queries, start=1):
_check_timeout()
if len(queries) > 1:
logger.debug("Newznab query %d/%d: '%s'", idx, len(queries), query)
for connection in clients:
try:
for idx, query in enumerate(queries, start=1):
_check_timeout()
if len(queries) > 1:
logger.debug(
"Newznab [%s] query %d/%d: '%s'",
connection.name,
idx,
len(queries),
query,
)
raw = client.search(query=query, categories=categories)
raw = connection.client.search(query=query, categories=categories)
# Auto-expand: retry without category filter if no results
if not raw and categories and auto_expand:
_check_timeout()
logger.info(
"Newznab: no results for '%s' with category filter, auto-expanding",
query,
)
raw = client.search(query=query, categories=None)
# Auto-expand: retry without category filter if no results
if not raw and categories and auto_expand:
_check_timeout()
logger.info(
"Newznab [%s]: no results for '%s' with category filter, "
"auto-expanding",
connection.name,
query,
)
raw = connection.client.search(query=query, categories=None)
for r in raw:
key = (
r.get("guid")
or r.get("downloadUrl")
or f"{r.get('indexer')}:{r.get('title')}"
)
if key in seen_keys:
continue
seen_keys.add(key)
all_results.append(r)
for raw_result in raw:
r = dict(raw_result)
# Aggregators can identify the underlying indexer. Plain feeds
# generally cannot, so use the user-configured connection name.
r["indexer"] = r.get("indexer") or connection.name
r["_newznab_connection_id"] = connection.connection_id
key = (
connection.connection_id,
r.get("guid")
or r.get("downloadUrl")
or f"{r.get('indexer')}:{r.get('title')}",
)
if key in seen_keys:
continue
seen_keys.add(key)
all_results.append(r)
except TimeoutError:
raise
except Exception:
logger.exception("Newznab search failed for %s", connection.name)
except TimeoutError as e:
logger.warning("Newznab search timed out: %s", e)
except Exception:
logger.exception("Newznab search failed")
return []
results = [_newznab_result_to_release(r, content_type, categories) for r in all_results]
if plan.indexers:
selected_indexers = set(plan.indexers)
results = [r for r in results if r.indexer in selected_indexers]
if results:
nzb_count = sum(1 for r in results if r.protocol == ReleaseProtocol.NZB)
@@ -379,5 +472,7 @@ class NewznabSource(ReleaseSource):
def is_available(self) -> bool:
if not config.get("NEWZNAB_ENABLED", False):
return False
if _parse_indexer_rows(config.get("NEWZNAB_INDEXERS", [])):
return True
url = normalize_http_url(str(config.get("NEWZNAB_URL", "") or ""))
return bool(url)
+79 -5
View File
@@ -7,6 +7,7 @@ from typing import Any, TypedDict
import requests
from shelfmark.core.config import config
from shelfmark.core.logger import setup_logger
from shelfmark.core.utils import normalize_http_url
from shelfmark.download.network import get_ssl_verify
@@ -18,6 +19,19 @@ logger = setup_logger(__name__)
_HTTP_STATUS_UNAUTHORIZED = HTTPStatus.UNAUTHORIZED
_BOOK_CATEGORY_RANGE_START = 7000
_BOOK_CATEGORY_RANGE_END = 8000
# Prowlarr's own JSON endpoints (status, indexer list) read local state and answer
# in milliseconds, so they keep a short timeout. A Torznab search is different: it
# is Prowlarr proxying a live request to the tracker, which for a Cloudflare-fronted
# indexer means waiting on FlareSolverr to solve a challenge. A cold challenge
# routinely runs past a minute, so indexer searches get their own, longer budget.
DEFAULT_INDEXER_TIMEOUT_SECONDS = 90
MIN_INDEXER_TIMEOUT_SECONDS = 5
MAX_INDEXER_TIMEOUT_SECONDS = 300
# Connecting to Prowlarr itself is a LAN hop; only the read is allowed to be slow.
_CONNECT_TIMEOUT_SECONDS = 10.0
_PROWLARR_CLIENT_ERRORS = (
requests.exceptions.RequestException,
OSError,
@@ -27,6 +41,37 @@ _PROWLARR_CLIENT_ERRORS = (
)
class ProwlarrSearchError(RuntimeError):
"""A Torznab search could not be completed.
Deliberately distinct from an empty result list. Reporting a failed search as
"this indexer has nothing" is what turns a slow FlareSolverr challenge into
"No releases found for this book" in the UI (#1249), and it also makes the
auto-expand retry fire a second request on top of the one still running.
"""
def resolve_indexer_timeout(timeout: object = None) -> int:
"""Resolve the per-indexer search timeout, falling back to config.
Out-of-range and unparsable values are clamped rather than rejected: this
feeds an HTTP timeout, and a bad setting should not take searching down.
"""
if timeout is None:
timeout = config.get("PROWLARR_INDEXER_TIMEOUT", DEFAULT_INDEXER_TIMEOUT_SECONDS)
resolved = coerce_int_like(timeout)
if resolved is None:
logger.warning(
"Invalid PROWLARR_INDEXER_TIMEOUT %r - using %ss",
timeout,
DEFAULT_INDEXER_TIMEOUT_SECONDS,
)
return DEFAULT_INDEXER_TIMEOUT_SECONDS
return max(MIN_INDEXER_TIMEOUT_SECONDS, min(MAX_INDEXER_TIMEOUT_SECONDS, resolved))
class IndexerSeedSettings(TypedDict, total=False):
ratio_limit: float
seeding_time_limit_minutes: int
@@ -77,11 +122,23 @@ def _get_field_value(fields: object, name: str) -> object | None:
class ProwlarrClient:
"""Client for interacting with the Prowlarr API."""
def __init__(self, url: str, api_key: str, timeout: int = 30) -> None:
"""Initialize the API client with base URL, key, and timeout."""
def __init__(
self, url: str, api_key: str, timeout: int = 30, indexer_timeout: int | None = None
) -> None:
"""Initialize the API client with base URL, key, and timeouts.
Args:
url: Prowlarr base URL.
api_key: Prowlarr API key.
timeout: Timeout for Prowlarr's own JSON endpoints.
indexer_timeout: Timeout for Torznab searches, which Prowlarr proxies
out to the tracker. Defaults to PROWLARR_INDEXER_TIMEOUT.
"""
self.base_url = normalize_http_url(url)
self.api_key = api_key
self.timeout = timeout
self.indexer_timeout = resolve_indexer_timeout(indexer_timeout)
self._session = requests.Session()
self._session.headers.update(
{
@@ -307,6 +364,12 @@ class ProwlarrClient:
This returns richer fields (e.g., author/booktitle, torznab tags like
FreeLeech) than the JSON /api/v1/search endpoint.
Raises:
ProwlarrSearchError: The search could not be completed. An empty list
strictly means the indexer answered with no matches, never that
the request timed out or errored.
"""
if not query:
return []
@@ -329,7 +392,7 @@ class ProwlarrClient:
response = self._session.get(
url=url,
params=params,
timeout=self.timeout,
timeout=(_CONNECT_TIMEOUT_SECONDS, self.indexer_timeout),
headers={
# Override the session default JSON accept header.
"Accept": "application/rss+xml, application/xml;q=0.9, */*;q=0.8"
@@ -347,9 +410,20 @@ class ProwlarrClient:
for r in results:
if r.get("indexerId") is None:
r["indexerId"] = int(indexer_id)
except Exception:
except requests.exceptions.Timeout as e:
logger.warning(
"Prowlarr Torznab search for indexer %s timed out after %ss. An indexer "
"behind FlareSolverr can need far longer than that on a cold Cloudflare "
"challenge - raise PROWLARR_INDEXER_TIMEOUT if this keeps happening.",
indexer_id,
self.indexer_timeout,
)
msg = f"indexer {indexer_id} did not respond within {self.indexer_timeout}s"
raise ProwlarrSearchError(msg) from e
except Exception as e:
logger.exception("Prowlarr Torznab search failed for indexer %s", indexer_id)
return []
msg = f"indexer {indexer_id} search failed: {e}"
raise ProwlarrSearchError(msg) from e
else:
return results
@@ -28,6 +28,10 @@ from shelfmark.download.clients.base_handler import (
DownloadRequest,
ExternalClientHandler,
)
from shelfmark.download.clients.torrent_utils import (
extract_file_list_from_torrent,
extract_torrent_info,
)
from shelfmark.metadata_providers import BookMetadata
from shelfmark.release_sources import register_handler
from shelfmark.release_sources.prowlarr.api import IndexerSeedSettings, ProwlarrClient
@@ -38,12 +42,14 @@ from shelfmark.release_sources.prowlarr.utils import (
coerce_int_like,
get_preferred_download_url,
get_protocol,
sanitize_download_url,
)
if TYPE_CHECKING:
from collections.abc import Callable
from shelfmark.core.models import DownloadTask
from shelfmark.download.postprocess.packs import PackFile
logger = setup_logger(__name__)
@@ -127,6 +133,24 @@ class ProwlarrHandler(ExternalClientHandler):
return settings.get(indexer_id)
def list_files(self, release_data: dict[str, Any]) -> list[PackFile] | None:
"""List a cached torrent release's files from its .torrent, without downloading.
Magnet-only and usenet releases cannot be listed ahead of time.
"""
source_id = str(release_data.get("source_id") or "")
prowlarr_result = get_release(source_id) if source_id else None
if not prowlarr_result or get_protocol(prowlarr_result) != "torrent":
return None
download_url = sanitize_download_url(str(prowlarr_result.get("downloadUrl") or "").strip())
if not download_url or download_url.startswith("magnet:"):
return None
expected_hash = str(prowlarr_result.get("infoHash") or "").strip() or None
info = extract_torrent_info(download_url, expected_hash=expected_hash)
if not info.torrent_data:
return None
return extract_file_list_from_torrent(info.torrent_data)
def _get_client(self, protocol: str) -> DownloadClient | None:
"""Compatibility shim so module-level patching still works in tests."""
return get_client(protocol)
@@ -297,6 +321,8 @@ class ProwlarrHandler(ExternalClientHandler):
search_title=title,
search_author=task.author,
)
# No language default here on purpose: this re-finds one exact release by its
# guid, and Prowlarr does not filter on plan.languages anyway.
plan = build_release_search_plan(
book,
indexers=[indexer] if indexer is not None else None,
@@ -10,12 +10,18 @@ from shelfmark.core.settings_registry import (
CheckboxField,
HeadingField,
MultiSelectField,
NumberField,
PasswordField,
SettingsField,
TextField,
register_settings,
)
from shelfmark.core.utils import normalize_http_url
from shelfmark.release_sources.prowlarr.api import (
DEFAULT_INDEXER_TIMEOUT_SECONDS,
MAX_INDEXER_TIMEOUT_SECONDS,
MIN_INDEXER_TIMEOUT_SECONDS,
)
# ==================== Dynamic Options Loaders ====================
@@ -183,6 +189,20 @@ def prowlarr_config_settings() -> list[SettingsField]:
default=[],
show_when={"field": "PROWLARR_ENABLED", "value": True},
),
NumberField(
key="PROWLARR_INDEXER_TIMEOUT",
label="Indexer Search Timeout (seconds)",
description=(
"How long to wait for a single indexer to answer a search. Indexers behind "
"FlareSolverr can need 90 seconds or more while a cold Cloudflare challenge "
"is solved; raise this if searches come back empty and the Prowlarr log "
"shows the search still running."
),
default=DEFAULT_INDEXER_TIMEOUT_SECONDS,
min_value=MIN_INDEXER_TIMEOUT_SECONDS,
max_value=MAX_INDEXER_TIMEOUT_SECONDS,
show_when={"field": "PROWLARR_ENABLED", "value": True},
),
CheckboxField(
key="PROWLARR_AUTO_EXPAND",
label="Auto-expand search on no results",
+175 -33
View File
@@ -2,6 +2,7 @@
import re
import time
from dataclasses import dataclass
from threading import Lock
from typing import TYPE_CHECKING, ClassVar, NoReturn
@@ -30,9 +31,14 @@ from shelfmark.release_sources import (
ReleaseProtocol,
ReleaseSource,
SortOption,
SourceUnavailableError,
register_source,
)
from shelfmark.release_sources.prowlarr.api import IndexerSeedSettings, ProwlarrClient
from shelfmark.release_sources.prowlarr.api import (
IndexerSeedSettings,
ProwlarrClient,
ProwlarrSearchError,
)
from shelfmark.release_sources.prowlarr.cache import cache_release
from shelfmark.release_sources.prowlarr.utils import (
build_source_id,
@@ -50,10 +56,10 @@ _PROWLARR_SOURCE_ERRORS = (AttributeError, OSError, RuntimeError, TypeError, Val
# Prowlarr indexer priority is 1-50 and lower is preferred; unknown sorts last.
_UNRANKED_INDEXER_RANK = 51
# Errors that can surface from ProwlarrClient.get_indexer_seed_settings(). The
# Errors that can surface from a ProwlarrClient call that talks to Prowlarr. The
# client raises requests exceptions (subclasses of OSError via IOError lineage
# is not guaranteed), so include RequestException explicitly.
_PROWLARR_SEED_SETTINGS_ERRORS = (*_PROWLARR_SOURCE_ERRORS, requests.exceptions.RequestException)
_PROWLARR_REQUEST_ERRORS = (*_PROWLARR_SOURCE_ERRORS, requests.exceptions.RequestException)
def _raise_timeout_error(message: str) -> NoReturn:
@@ -139,6 +145,36 @@ def _build_indexer_priority(indexers: list[dict]) -> dict[int, int]:
return priority
def _drop_unknown_indexer_ids(
selected_ids: list[int] | None, indexers: list[dict]
) -> list[int] | None:
"""Keep only selected indexer ids Prowlarr still serves.
An indexer removed or disabled in Prowlarr stays in the saved selection,
where settings can no longer show it - so it cannot be unselected, and every
search keeps querying an indexer that is gone (#1283). Dropping it here
keeps the saved selection intact for an indexer that comes back.
"""
if selected_ids is None:
return None
live_ids = {
indexer_id
for indexer in indexers
if (indexer_id := _coerce_indexer_id(indexer.get("id"))) is not None
}
kept = [indexer_id for indexer_id in selected_ids if indexer_id in live_ids]
stale = [indexer_id for indexer_id in selected_ids if indexer_id not in live_ids]
if stale:
logger.warning(
"Skipping selected Prowlarr indexers that are no longer enabled in Prowlarr: %s",
stale,
)
return kept
def _rank_for_indexer_id(indexer_id: object, priority: dict[int, int]) -> int:
"""Preference rank for an indexer id. Lower wins, unknown ranks last."""
coerced = _coerce_indexer_id(indexer_id)
@@ -232,6 +268,35 @@ ALL_BOOK_FORMATS = AUDIOBOOK_FORMATS + EBOOK_FORMATS
# Backend safeguard: cap total Prowlarr search time per request.
PROWLARR_SEARCH_TIMEOUT_SECONDS = 120.0
# The overall budget has to leave room for at least a couple of indexers to spend
# their full per-indexer timeout, otherwise raising PROWLARR_INDEXER_TIMEOUT for a
# Cloudflare-fronted tracker just moves the cutoff here. Capped short of the
# gunicorn worker timeout (300s) so the worker is never the thing that gives up.
_MAX_SEARCH_BUDGET_SECONDS = 240.0
def _search_budget_seconds(indexer_timeout: int) -> float:
"""Total time one Prowlarr search may spend, scaled to the per-indexer timeout."""
return min(
_MAX_SEARCH_BUDGET_SECONDS,
max(PROWLARR_SEARCH_TIMEOUT_SECONDS, indexer_timeout * 2.0),
)
@dataclass
class _IndexerSearchOutcome:
"""What one pass over the target indexers produced.
Separates "every indexer answered, none had this book" from "the indexers
never answered", which the caller has to tell apart before it decides to
auto-expand or to report the search as failed.
"""
results: list[dict]
attempted: int = 0
failed: int = 0
last_error: str | None = None
def _extract_format(title: str) -> str | None:
"""Extract ebook/audiobook format from release title (extension, bracketed, or standalone)."""
@@ -282,19 +347,25 @@ def _extract_mam_language(raw_title: str) -> str | None:
return None
def _extract_mam_formats(raw_title: str) -> list[str]:
"""Extract a list of formats from MyAnonamouse titles.
def _split_mam_formats(raw_title: str) -> tuple[list[str], list[str]]:
"""Split the format tokens of a MyAnonamouse title into (recognized, unrecognized).
Prowlarr's MAM parser appends a structured bracket segment like:
[ENG / EPUB MOBI PDF]
We only trust this structured segment (and do not attempt generic title
heuristics for other indexers).
Tokens after the "/" that Shelfmark does not know as a book or audiobook format
(e.g. ``[ENG / AVI]``) are returned separately so the UI can warn that the release
will download but cannot be processed, instead of showing a bare content-type icon
that looks like an ordinary result.
"""
if not raw_title:
return []
return [], []
format_set = set(ALL_BOOK_FORMATS)
first_unrecognized: list[str] | None = None
for bracket in re.findall(r"\[([^\]]+)\]", raw_title):
if "/" not in bracket:
continue
@@ -303,15 +374,26 @@ def _extract_mam_formats(raw_title: str) -> list[str]:
tokens = re.findall(r"[A-Za-z0-9]+", after_slash)
formats: list[str] = []
unrecognized: list[str] = []
for token in tokens:
fmt = token.lower()
if fmt in format_set and fmt not in formats:
formats.append(fmt)
if fmt in format_set:
if fmt not in formats:
formats.append(fmt)
elif fmt not in unrecognized:
unrecognized.append(fmt)
if formats:
return formats
return formats, unrecognized
if unrecognized and first_unrecognized is None:
first_unrecognized = unrecognized
return []
return [], first_unrecognized or []
def _extract_mam_formats(raw_title: str) -> list[str]:
"""Extract the recognized formats from a MyAnonamouse title (see _split_mam_formats)."""
return _split_mam_formats(raw_title)[0]
def _formats_display(formats: list[str]) -> str | None:
@@ -450,6 +532,7 @@ def _prowlarr_result_to_release(
format_detected: str | None = None
formats: list[str] = []
unrecognized_formats: list[str] = []
formats_display: str | None = None
language_detected: str | None = None
if enable_format_detection:
@@ -457,7 +540,7 @@ def _prowlarr_result_to_release(
if book_title:
title = book_title
formats = _extract_mam_formats(str(raw_title or ""))
formats, unrecognized_formats = _split_mam_formats(str(raw_title or ""))
format_detected = formats[0] if formats else None
formats_display = _formats_display(formats)
language_detected = _extract_mam_language(str(raw_title or ""))
@@ -519,6 +602,9 @@ def _prowlarr_result_to_release(
"info_hash": result.get("infoHash"),
"formats": formats or None,
"formats_display": formats_display,
# Format tokens the indexer declared but Shelfmark can't process (e.g. a MAM
# "[ENG / AVI]"). Lets the UI warn instead of showing a bare content icon.
"unrecognized_formats": unrecognized_formats or None,
# Raw torznab attributes for rich tooltips (enriched indexers)
"torznab_attrs": result.get("torznabAttrs"),
},
@@ -539,7 +625,7 @@ def _fetch_indexer_seed_settings(
"""Fetch per-indexer share limits, falling back to last-known-good on failure."""
try:
fetched = client.get_indexer_seed_settings(restrict_to=indexer_ids)
except _PROWLARR_SEED_SETTINGS_ERRORS:
except _PROWLARR_REQUEST_ERRORS:
with _seed_settings_lock:
fallback = dict(_last_known_seed_settings)
logger.warning(
@@ -895,8 +981,17 @@ class ProwlarrSource(ReleaseSource):
try:
auto_expand_enabled = config.get("PROWLARR_AUTO_EXPAND", False)
deadline = time.monotonic() + PROWLARR_SEARCH_TIMEOUT_SECONDS
enabled_indexers = client.get_enabled_indexers_detailed()
search_budget = _search_budget_seconds(client.indexer_timeout)
deadline = time.monotonic() + search_budget
try:
enabled_indexers = client.get_enabled_indexers_detailed(raise_on_error=True)
except _PROWLARR_REQUEST_ERRORS as e:
# Prowlarr itself is unreachable. Swallowing this leaves the search
# with no indexers to query, which the UI renders as "No releases
# found for this book" - the same lie as a swallowed timeout (#1249).
msg = f"could not reach Prowlarr: {e}"
raise SourceUnavailableError(msg) from e
indexer_ids = _drop_unknown_indexer_ids(indexer_ids, enabled_indexers)
indexer_priority = _build_indexer_priority(enabled_indexers)
# Some indexers benefit from title+author queries and extra format detection.
enriched_indexer_ids = client.get_enriched_indexer_ids(
@@ -911,18 +1006,16 @@ class ProwlarrSource(ReleaseSource):
def _check_timeout() -> None:
if time.monotonic() > deadline:
_raise_timeout_error(
f"Prowlarr search timed out after {int(PROWLARR_SEARCH_TIMEOUT_SECONDS)}s"
)
_raise_timeout_error(f"Prowlarr search timed out after {int(search_budget)}s")
def search_indexers(
query: str, cats: list[int] | None, *, enriched_query: str | None = None
) -> list[dict]:
) -> _IndexerSearchOutcome:
"""Search indexers with given categories via Torznab/Newznab."""
results: list[dict] = []
outcome = _IndexerSearchOutcome(results=[])
target_indexer_ids = self._get_search_indexer_ids(client, indexer_ids, cats)
if not target_indexer_ids:
return results
return outcome
for indexer_id in target_indexer_ids:
_check_timeout()
@@ -931,19 +1024,31 @@ class ProwlarrSource(ReleaseSource):
if indexer_id in enriched_indexer_ids_set and enriched_query
else query
)
raw = client.torznab_search(
indexer_id=indexer_id,
query=indexer_query,
categories=cats,
search_type="book",
)
outcome.attempted += 1
try:
raw = client.torznab_search(
indexer_id=indexer_id,
query=indexer_query,
categories=cats,
search_type="book",
)
except ProwlarrSearchError as e:
# One unreachable indexer must not sink the others, but it
# is not "no results" either - record it so the caller can
# report a failed search instead of an empty one.
outcome.failed += 1
outcome.last_error = str(e)
continue
if raw:
results.extend(raw)
outcome.results.extend(raw)
return results
return outcome
seen_keys: set[tuple[int | None, str]] = set()
all_results: list[dict] = []
attempted_searches = 0
failed_searches = 0
last_search_error: str | None = None
for idx, variant in enumerate(variants, start=1):
_check_timeout()
@@ -953,23 +1058,39 @@ class ProwlarrSource(ReleaseSource):
if len(variants) > 1:
logger.debug("Prowlarr query %s/%s: '%s'", idx, len(variants), query)
raw_results = search_indexers(
outcome = search_indexers(
query=query, cats=categories, enriched_query=enriched_query
)
# Auto-expand: if no results with categories and auto-expand enabled, retry without
if not raw_results and categories and auto_expand_enabled:
# Auto-expand: if no results with categories and auto-expand enabled, retry without.
# Only when every indexer actually answered: a failed search says nothing about
# whether the category filter is what hid the book, and retrying it stacks a second
# request on an indexer that is still busy solving a Cloudflare challenge (#1249).
if (
not outcome.results
and not outcome.failed
and categories
and auto_expand_enabled
):
_check_timeout()
logger.info(
"Prowlarr: no results for query '%s' with category filter, auto-expanding search",
query,
)
raw_results = search_indexers(
expanded = search_indexers(
query=query, cats=None, enriched_query=enriched_query
)
outcome.results = expanded.results
outcome.attempted += expanded.attempted
outcome.failed += expanded.failed
outcome.last_error = expanded.last_error or outcome.last_error
self.last_search_type = "expanded"
for r in raw_results:
attempted_searches += outcome.attempted
failed_searches += outcome.failed
last_search_error = outcome.last_error or last_search_error
for r in outcome.results:
key = _result_dedup_key(r)
if key is not None:
if key in seen_keys:
@@ -977,6 +1098,14 @@ class ProwlarrSource(ReleaseSource):
seen_keys.add(key)
all_results.append(r)
if failed_searches:
logger.warning(
"Prowlarr: %s of %s indexer searches failed (%s)",
failed_searches,
attempted_searches,
last_search_error,
)
if config.get("PROWLARR_COLLAPSE_DUPLICATES", True):
before_collapse = len(all_results)
all_results = _collapse_duplicate_indexer_results(all_results, indexer_priority)
@@ -1033,6 +1162,10 @@ class ProwlarrSource(ReleaseSource):
else:
logger.debug("Prowlarr: no results found")
except SourceUnavailableError:
# Already carries its own message for the caller to surface; the blanket
# handler below would turn it back into a silent empty result.
raise
except TimeoutError as e:
logger.warning("Prowlarr search timed out: %s", e)
raise
@@ -1040,6 +1173,15 @@ class ProwlarrSource(ReleaseSource):
logger.exception("Prowlarr search failed")
return []
else:
# An empty list is the UI's "No releases found for this book", so it has
# to mean the indexers answered and had nothing. When they failed instead,
# say so rather than blaming the book (#1249).
if not results and failed_searches:
msg = (
f"{failed_searches} of {attempted_searches} indexer searches failed "
f"({last_search_error})"
)
raise SourceUnavailableError(msg)
return results
def is_available(self) -> bool:
+89 -142
View File
@@ -19,7 +19,7 @@
"@types/react": "^19.2.18",
"@types/react-dom": "^19.2.4",
"@vitejs/plugin-react": "^6.0.5",
"knip": "^6.32.1",
"knip": "^6.32.2",
"oxfmt": "^0.63.0",
"oxlint": "^1.78.0",
"oxlint-tsgolint": "^7.0.2001",
@@ -130,9 +130,9 @@
}
},
"node_modules/@oxc-parser/binding-android-arm-eabi": {
"version": "0.142.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-android-arm-eabi/-/binding-android-arm-eabi-0.142.0.tgz",
"integrity": "sha512-ZiRGDutGsv1G6bL/ozy/koC0Sv39T1DqyoC4KD1DOy9ZoACm1O5UWhEK2c02Qdk+4lfLVkvFa/mQ0fm/4h1BtQ==",
"version": "0.143.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-android-arm-eabi/-/binding-android-arm-eabi-0.143.0.tgz",
"integrity": "sha512-n9uozULWflPqBtdmI8lAabLqGKNgLVNN0ZH8HfgCwpKGNtzRzauB76jTiW/3YLkcA7N1zskpi9GdVnZuu1SAvg==",
"cpu": [
"arm"
],
@@ -147,9 +147,9 @@
}
},
"node_modules/@oxc-parser/binding-android-arm64": {
"version": "0.142.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-android-arm64/-/binding-android-arm64-0.142.0.tgz",
"integrity": "sha512-WZkvGRLNQTz8lR9zP5nLjUdlroRCopBu3g9zF1p/laE6DzT1UbQo8Rdz5MWhaJUPYg/6gp+jo7HUgsyKaN1FtQ==",
"version": "0.143.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-android-arm64/-/binding-android-arm64-0.143.0.tgz",
"integrity": "sha512-9BbdjHETk6O3zH/DDid9IgBtF0GlpLabNKN231uraXpRDSfY+iiZxTP5bk1Z63GBownVdhdINFIeddmMz4MzpQ==",
"cpu": [
"arm64"
],
@@ -164,9 +164,9 @@
}
},
"node_modules/@oxc-parser/binding-darwin-arm64": {
"version": "0.142.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-darwin-arm64/-/binding-darwin-arm64-0.142.0.tgz",
"integrity": "sha512-l4khS8LQOOVYsGRVARo1gSaCT/aBSceUVXgtovWc2+drnxVuDr082WA3OCHVdVzIz5JIrP/y9CWsSKxBDNmYGg==",
"version": "0.143.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-darwin-arm64/-/binding-darwin-arm64-0.143.0.tgz",
"integrity": "sha512-gh+6ecoHUy4/sUcolBl/1qPXKBbYNxFY0Pk0ujgQvINTMSftJY7o4yb8gOkDJPeZeB8+a+u7xTe6umoP8N5HFA==",
"cpu": [
"arm64"
],
@@ -181,9 +181,9 @@
}
},
"node_modules/@oxc-parser/binding-darwin-x64": {
"version": "0.142.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-darwin-x64/-/binding-darwin-x64-0.142.0.tgz",
"integrity": "sha512-QBsNF3nqlXmcH2B1YOPqQYmCJoy4HuIjUxGbBO/k5JAJUl68ghU2psRY2zPk+RyBaWqKP/qfL4oaFgEMCdwskA==",
"version": "0.143.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-darwin-x64/-/binding-darwin-x64-0.143.0.tgz",
"integrity": "sha512-qd1hl2d+lXgHv/VQ/M9qm8TrMC5T4RqDBwtOnl+1D0QMjwcz+8AaB4JSg8STgeag0GP6a6L74XEGAsrTSJWNzQ==",
"cpu": [
"x64"
],
@@ -198,9 +198,9 @@
}
},
"node_modules/@oxc-parser/binding-freebsd-x64": {
"version": "0.142.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-freebsd-x64/-/binding-freebsd-x64-0.142.0.tgz",
"integrity": "sha512-b7Q7m4Cqc6XqNhri3R+QhU+GVy646Pn+bkdhrDdWym/Fdi0ZUa+d73H9dm5H91JtbtAQ/z1d8XKMW3oOV8a4tQ==",
"version": "0.143.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-freebsd-x64/-/binding-freebsd-x64-0.143.0.tgz",
"integrity": "sha512-M5XXcNa7aOqLPKTR41msfghKu2yQ4xWvCm11/gwU0JzOzHNk5sgW//rVEjJ+LO48+VDAMzXTSzurUVxIDKwozw==",
"cpu": [
"x64"
],
@@ -215,9 +215,9 @@
}
},
"node_modules/@oxc-parser/binding-linux-arm-gnueabihf": {
"version": "0.142.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-0.142.0.tgz",
"integrity": "sha512-3riVS5IhdH3uCZj1Y9ftDQlR0dvLsIlw/edrRqk8JhgNd5K0XSs+UBtgh50N13CAlW9/TXj6sVGXaKNBocd0Yg==",
"version": "0.143.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-0.143.0.tgz",
"integrity": "sha512-T/GXusuOkPNQhCQCSBbcU/N8j0rAypuDBl1IyFK+lyYT594XsVz80clPC/OtbSSpBGyJxj8uYEfctxVuxVYoww==",
"cpu": [
"arm"
],
@@ -232,9 +232,9 @@
}
},
"node_modules/@oxc-parser/binding-linux-arm-musleabihf": {
"version": "0.142.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-arm-musleabihf/-/binding-linux-arm-musleabihf-0.142.0.tgz",
"integrity": "sha512-NmXUOpgpTSkhl795TiXmWppTwmSJ92RC1qvD6e4XOF+slgmo3e6Ah+kEu+6AN8s7NAOEwqGmir58MgSQSWmBSA==",
"version": "0.143.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-arm-musleabihf/-/binding-linux-arm-musleabihf-0.143.0.tgz",
"integrity": "sha512-oKu4RcBlXSqo3OC62dp6YTnQaZIurNDpCX3BnAM3+bJxt7s8J2TJKMnC0UYer1qhlRaDCg6wkTaTw+2IlsZ12w==",
"cpu": [
"arm"
],
@@ -249,9 +249,9 @@
}
},
"node_modules/@oxc-parser/binding-linux-arm64-gnu": {
"version": "0.142.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-0.142.0.tgz",
"integrity": "sha512-gc0EXsKtXgerujmU2Bql3u1L1HsSQ2774R83idq/FoNMPVV/RY/1ErFsvnit7KoiP/sLvzQixeUo4Ut0ic0wmw==",
"version": "0.143.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-0.143.0.tgz",
"integrity": "sha512-WJBbD186AZmMGaSIhlktC+rPl8L3peCTXAh88Ih9uEvK0en2mPojGyCGYiL6mHtV1RPV3JyfJW5t6n5hh0lXhA==",
"cpu": [
"arm64"
],
@@ -269,9 +269,9 @@
}
},
"node_modules/@oxc-parser/binding-linux-arm64-musl": {
"version": "0.142.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-arm64-musl/-/binding-linux-arm64-musl-0.142.0.tgz",
"integrity": "sha512-F2XvmWSE0uWpie+jHKKIFgdVOe9ypGhkEZxKx5DuW215K6cbAC274yYaPkcM7EqY4Df3Weyhpcz3lsURyH2LVg==",
"version": "0.143.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-arm64-musl/-/binding-linux-arm64-musl-0.143.0.tgz",
"integrity": "sha512-t1AcYOwEzgceadT4v5e+vaCCb0AncCA3v5AyzfBAz/tMq11qzVccXKzNHtkWdjBsgvTKwRkaUF3QvT4kot8vcQ==",
"cpu": [
"arm64"
],
@@ -289,9 +289,9 @@
}
},
"node_modules/@oxc-parser/binding-linux-ppc64-gnu": {
"version": "0.142.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-0.142.0.tgz",
"integrity": "sha512-wLMbT21U/QxknQsk+VvNF0b9D2/aGWhcaQQQ+VYlE8FwD5+GoWZIPPXNzyHmkYyhm0KB3itL+TBavjMatqNnYA==",
"version": "0.143.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-0.143.0.tgz",
"integrity": "sha512-RsnO/NoD8376LMJq8JS8TwI0ieNaFRTuNe2GVJntQg6gwZNMENZsEbknHdVwjpOmxdGLGodcwaGSbAeRr5Bgjw==",
"cpu": [
"ppc64"
],
@@ -309,9 +309,9 @@
}
},
"node_modules/@oxc-parser/binding-linux-riscv64-gnu": {
"version": "0.142.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-riscv64-gnu/-/binding-linux-riscv64-gnu-0.142.0.tgz",
"integrity": "sha512-+G8F/4ckwT7FCJV4H2bt09xEzJbjNCfuL4Sp1AYNaFtFMVtgIGMuJlteT82U+K0UIZ/DzAR/LDlMFnEuajG7Kw==",
"version": "0.143.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-riscv64-gnu/-/binding-linux-riscv64-gnu-0.143.0.tgz",
"integrity": "sha512-48fSVfR9TZi5CASZFyv0VC6z6BCoeihFsX031mAD/oSH7d9PYsPgIqza7d9mjP7Z2KTEpTFyH6SIu0Ui6R1vdg==",
"cpu": [
"riscv64"
],
@@ -329,9 +329,9 @@
}
},
"node_modules/@oxc-parser/binding-linux-riscv64-musl": {
"version": "0.142.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-riscv64-musl/-/binding-linux-riscv64-musl-0.142.0.tgz",
"integrity": "sha512-hTsHtTLxMAfCo+rpF5K3qZJKW2NpPN/CHd4mYB3y7XlSdspHkd2gehDIofP64AacA9nWQw2tY3O7wR6UY8IVOA==",
"version": "0.143.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-riscv64-musl/-/binding-linux-riscv64-musl-0.143.0.tgz",
"integrity": "sha512-T8CpdD+SfE01DnIOD4HpVxu0ZJOfMJ/VhCvikKfaXAxkZ+9veyLM/D2hpi7Y2hFUyPmVQO3FNZHmYzV/WlVR4g==",
"cpu": [
"riscv64"
],
@@ -349,9 +349,9 @@
}
},
"node_modules/@oxc-parser/binding-linux-s390x-gnu": {
"version": "0.142.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-0.142.0.tgz",
"integrity": "sha512-6y7qYY3TCUDYjqswImdTGl92y+KA/80twALegQPN27kfY+bG7Ib1+L3jbmrCZQx6wrVnai9IPsEZp07I0hx7JQ==",
"version": "0.143.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-0.143.0.tgz",
"integrity": "sha512-QLdeMsCcacenPEFsfxnBUDF1y6opyz5+fmOz9bfD5Y7fiGCMupUCuB3KTPQhNwshIG1P9fPqar9MHxuBDd4bwQ==",
"cpu": [
"s390x"
],
@@ -369,9 +369,9 @@
}
},
"node_modules/@oxc-parser/binding-linux-x64-gnu": {
"version": "0.142.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-x64-gnu/-/binding-linux-x64-gnu-0.142.0.tgz",
"integrity": "sha512-i69kAWU+2LgoH5bR+zWiiu+UzAw7Oxkwv7COeJTeY19pn4e70nKQcr9Pm6cL2Z0Z54d+gl9qADlK/0yyuCPiBA==",
"version": "0.143.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-x64-gnu/-/binding-linux-x64-gnu-0.143.0.tgz",
"integrity": "sha512-659ujfqLy6k7cuH3sbzhd8b+ztSq+i6E2E9pG78Q0BmHjAExfGIdgc8cGgMdwAozDXeZFHkJ+LXYJdWsaGdgyw==",
"cpu": [
"x64"
],
@@ -389,9 +389,9 @@
}
},
"node_modules/@oxc-parser/binding-linux-x64-musl": {
"version": "0.142.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-x64-musl/-/binding-linux-x64-musl-0.142.0.tgz",
"integrity": "sha512-4SQs678MmjYVrmhAgCWD4o0vpaFszXw9xLX5p2Z9MMFcltxiLkA88wQjh80YHjPrXtpyZ2CWI5m+1yNKM0m2Pw==",
"version": "0.143.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-x64-musl/-/binding-linux-x64-musl-0.143.0.tgz",
"integrity": "sha512-/Mw/9j4TfZcnKphPrzOE6t4MMknXadcAAuVUlDRTF/ETWB5xOgQvOJV2Mh9We/bWxZdoxaGAdc+hy4GuYwQ2yQ==",
"cpu": [
"x64"
],
@@ -409,9 +409,9 @@
}
},
"node_modules/@oxc-parser/binding-openharmony-arm64": {
"version": "0.142.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-openharmony-arm64/-/binding-openharmony-arm64-0.142.0.tgz",
"integrity": "sha512-YHpx9N7Ln3a++Tc8rv+H7mrK1zyJQOAwCFg8LZ3lTs1T5afGWeZrLPhPT9HLnIwSjCyJqPWVMIrMxbjcmBr2oQ==",
"version": "0.143.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-openharmony-arm64/-/binding-openharmony-arm64-0.143.0.tgz",
"integrity": "sha512-8rIKWR2BFuifbIK/1XB9wTaSdtuJ25dlE7ZQYDnEwj/2xH2vHsxnvIjHT3ZjSVuLLwGGlSslIG/fbOJ8TV8rTw==",
"cpu": [
"arm64"
],
@@ -425,52 +425,10 @@
"node": "^20.19.0 || >=22.12.0"
}
},
"node_modules/@oxc-parser/binding-wasm32-wasi": {
"version": "0.142.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-wasm32-wasi/-/binding-wasm32-wasi-0.142.0.tgz",
"integrity": "sha512-3pLDyY3+oogW73RM5uehNgAiR/Xfb7fvO2Q1Z1gIqZ2+50XDVQmBVlRkHXZTU4gKnQHpwETNsYQVsJ3joVB2iA==",
"cpu": [
"wasm32"
],
"dev": true,
"license": "MIT",
"optional": true,
"dependencies": {
"@emnapi/core": "1.11.2",
"@emnapi/runtime": "1.11.2",
"@napi-rs/wasm-runtime": "^1.1.6"
},
"engines": {
"node": "^20.19.0 || >=22.12.0"
}
},
"node_modules/@oxc-parser/binding-wasm32-wasi/node_modules/@emnapi/core": {
"version": "1.11.2",
"resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.11.2.tgz",
"integrity": "sha512-TC8MkTuZUtcTSiFeuC0ksCh9QIJ5+F21MvZ4Wn4ORfYaFJ/0dsiudv5tVkejgwZlwQ39jL9WWDe2lz8x0WglOA==",
"dev": true,
"license": "MIT",
"optional": true,
"dependencies": {
"@emnapi/wasi-threads": "1.2.2",
"tslib": "^2.4.0"
}
},
"node_modules/@oxc-parser/binding-wasm32-wasi/node_modules/@emnapi/runtime": {
"version": "1.11.2",
"resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.11.2.tgz",
"integrity": "sha512-kyOl3X0DuTiT1h2ft8r2fYO8JYtU9a9Xis/zBSiGArNaagCOWx90N1k2wxp18czFDH+OgcWGb5ZP/XMt3dcyPA==",
"dev": true,
"license": "MIT",
"optional": true,
"dependencies": {
"tslib": "^2.4.0"
}
},
"node_modules/@oxc-parser/binding-win32-arm64-msvc": {
"version": "0.142.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-0.142.0.tgz",
"integrity": "sha512-Had/VeVY28Oyb0K+Q4FV8KCzoBycIh93oDK6pCbya9lkzdq+ikMHMgBubsdqqlybjJmQRawCQRrnBRHyQwYvcQ==",
"version": "0.143.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-0.143.0.tgz",
"integrity": "sha512-5U9kQYMfRRI6Zq7KDxgbIP0RMnKrfn3gLepRMgJuRkPSUALTiRCk9d/uyhb4lGDjUdzwK7mBkKqhLgzBPCmLpQ==",
"cpu": [
"arm64"
],
@@ -485,9 +443,9 @@
}
},
"node_modules/@oxc-parser/binding-win32-ia32-msvc": {
"version": "0.142.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-win32-ia32-msvc/-/binding-win32-ia32-msvc-0.142.0.tgz",
"integrity": "sha512-GGi3+YphVHavvgs6gum2UXoNCqzHAmPt/nXkn8ZQZstV2Q1qZD1Mn8fz/nWrDkefHQtrG/+1/XrbMxsBTo6Svw==",
"version": "0.143.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-win32-ia32-msvc/-/binding-win32-ia32-msvc-0.143.0.tgz",
"integrity": "sha512-25P7AaHk4R88Yv2XH4gToDVmh0cOu+bEURQU10CRrmvgabfRArSGAP5osmwUKeSUHj0VS50upbpbRWWW/m7mHA==",
"cpu": [
"ia32"
],
@@ -502,9 +460,9 @@
}
},
"node_modules/@oxc-parser/binding-win32-x64-msvc": {
"version": "0.142.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-win32-x64-msvc/-/binding-win32-x64-msvc-0.142.0.tgz",
"integrity": "sha512-Ny/Wv4Us1LGC/ljwNTp+Hx3r/pH15EFfeDF0p+n898gt+TtRd6C9SccHcuUhDiNTb8s5tt7jdeAMDRQZ4Vq6hg==",
"version": "0.143.0",
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-win32-x64-msvc/-/binding-win32-x64-msvc-0.143.0.tgz",
"integrity": "sha512-ORMh3JE1s6V7ySicdRK7vgaDQnn5o+UHg9ct989PlWHbel8O9ARrmWXM6kZjrBMtNucxNayQ8g69G0VfWzhANw==",
"cpu": [
"x64"
],
@@ -2979,9 +2937,9 @@
}
},
"node_modules/knip": {
"version": "6.32.1",
"resolved": "https://registry.npmjs.org/knip/-/knip-6.32.1.tgz",
"integrity": "sha512-mIiIHMTJVUgSlz0mxEgPt7wg8DmfbCp1Txqab3WpbMCJF7YHvHtC9jeAHHXfISMl72N8WzhyG71SQlaqCOGZtg==",
"version": "6.32.2",
"resolved": "https://registry.npmjs.org/knip/-/knip-6.32.2.tgz",
"integrity": "sha512-WXTXbmocrw7gqm1A1TQvFN0OgJ7hUSU6E1g6SPRIzzHFogUBhXByc7cYeOFVtJ2uODg7DP4VbESYBYnfbtBYsg==",
"dev": true,
"funding": [
{
@@ -2999,13 +2957,13 @@
"formatly": "^0.3.0",
"get-tsconfig": "4.14.1",
"jiti": "^2.7.0",
"oxc-parser": "^0.142.0",
"oxc-parser": "^0.143.0",
"oxc-resolver": "11.24.2",
"picomatch": "^4.0.5",
"smol-toml": "^1.7.1",
"strip-json-comments": "5.0.3",
"tinyglobby": "^0.2.17",
"unbash": "^4.0.4",
"unbash": "^4.0.9",
"yaml": "^2.9.0",
"zod": "^4.4.3"
},
@@ -3311,13 +3269,13 @@
"license": "MIT"
},
"node_modules/oxc-parser": {
"version": "0.142.0",
"resolved": "https://registry.npmjs.org/oxc-parser/-/oxc-parser-0.142.0.tgz",
"integrity": "sha512-kKR+jPiRJYJDexVoziIg/FVGvr1fT1FZSSJOk6tVoMKKSlsf1Cso+cgGCJkOEDWOP174vRntCPFKg+AS7InWvw==",
"version": "0.143.0",
"resolved": "https://registry.npmjs.org/oxc-parser/-/oxc-parser-0.143.0.tgz",
"integrity": "sha512-ov0NzaDCOInknS7mP1cwKdJERt3utPW8ldjtdUXQ8Ty0GEFD08wk422vCUN0d7pST6kqtV7dxoI9w1Zi0l/9TA==",
"dev": true,
"license": "MIT",
"dependencies": {
"@oxc-project/types": "^0.142.0"
"@oxc-project/types": "^0.143.0"
},
"engines": {
"node": "^20.19.0 || >=22.12.0"
@@ -3326,36 +3284,25 @@
"url": "https://github.com/sponsors/Boshen"
},
"optionalDependencies": {
"@oxc-parser/binding-android-arm-eabi": "0.142.0",
"@oxc-parser/binding-android-arm64": "0.142.0",
"@oxc-parser/binding-darwin-arm64": "0.142.0",
"@oxc-parser/binding-darwin-x64": "0.142.0",
"@oxc-parser/binding-freebsd-x64": "0.142.0",
"@oxc-parser/binding-linux-arm-gnueabihf": "0.142.0",
"@oxc-parser/binding-linux-arm-musleabihf": "0.142.0",
"@oxc-parser/binding-linux-arm64-gnu": "0.142.0",
"@oxc-parser/binding-linux-arm64-musl": "0.142.0",
"@oxc-parser/binding-linux-ppc64-gnu": "0.142.0",
"@oxc-parser/binding-linux-riscv64-gnu": "0.142.0",
"@oxc-parser/binding-linux-riscv64-musl": "0.142.0",
"@oxc-parser/binding-linux-s390x-gnu": "0.142.0",
"@oxc-parser/binding-linux-x64-gnu": "0.142.0",
"@oxc-parser/binding-linux-x64-musl": "0.142.0",
"@oxc-parser/binding-openharmony-arm64": "0.142.0",
"@oxc-parser/binding-wasm32-wasi": "0.142.0",
"@oxc-parser/binding-win32-arm64-msvc": "0.142.0",
"@oxc-parser/binding-win32-ia32-msvc": "0.142.0",
"@oxc-parser/binding-win32-x64-msvc": "0.142.0"
}
},
"node_modules/oxc-parser/node_modules/@oxc-project/types": {
"version": "0.142.0",
"resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.142.0.tgz",
"integrity": "sha512-7W+2q5AKQVU36fkaryontrHn3YDt1RyUYXatw9i5H8ocYe2sPKSFB6eS8WNPeRKiN1qAWWZUPm7gwFzJGrccqQ==",
"dev": true,
"license": "MIT",
"funding": {
"url": "https://github.com/sponsors/Boshen"
"@oxc-parser/binding-android-arm-eabi": "0.143.0",
"@oxc-parser/binding-android-arm64": "0.143.0",
"@oxc-parser/binding-darwin-arm64": "0.143.0",
"@oxc-parser/binding-darwin-x64": "0.143.0",
"@oxc-parser/binding-freebsd-x64": "0.143.0",
"@oxc-parser/binding-linux-arm-gnueabihf": "0.143.0",
"@oxc-parser/binding-linux-arm-musleabihf": "0.143.0",
"@oxc-parser/binding-linux-arm64-gnu": "0.143.0",
"@oxc-parser/binding-linux-arm64-musl": "0.143.0",
"@oxc-parser/binding-linux-ppc64-gnu": "0.143.0",
"@oxc-parser/binding-linux-riscv64-gnu": "0.143.0",
"@oxc-parser/binding-linux-riscv64-musl": "0.143.0",
"@oxc-parser/binding-linux-s390x-gnu": "0.143.0",
"@oxc-parser/binding-linux-x64-gnu": "0.143.0",
"@oxc-parser/binding-linux-x64-musl": "0.143.0",
"@oxc-parser/binding-openharmony-arm64": "0.143.0",
"@oxc-parser/binding-win32-arm64-msvc": "0.143.0",
"@oxc-parser/binding-win32-ia32-msvc": "0.143.0",
"@oxc-parser/binding-win32-x64-msvc": "0.143.0"
}
},
"node_modules/oxc-resolver": {
@@ -3874,9 +3821,9 @@
}
},
"node_modules/unbash": {
"version": "4.0.4",
"resolved": "https://registry.npmjs.org/unbash/-/unbash-4.0.4.tgz",
"integrity": "sha512-60m9IVGbavD6jholbxt0jVBXZkEB/HsMZq7Tyaghseve2/Sf0zQRAIfWsD34sde+DKP2tBxJS2wP88ZM0D1FhA==",
"version": "4.0.10",
"resolved": "https://registry.npmjs.org/unbash/-/unbash-4.0.10.tgz",
"integrity": "sha512-b7zoBQvpWp0vuN5q2vK2RRBR2SvuruQAs50DApdDveBSn3eSYd84IaHodFqQIMlvY9K2VnyBUEXgwOBuGU9GBg==",
"dev": true,
"license": "ISC",
"engines": {
+1 -1
View File
@@ -28,7 +28,7 @@
"@types/react": "^19.2.18",
"@types/react-dom": "^19.2.4",
"@vitejs/plugin-react": "^6.0.5",
"knip": "^6.32.1",
"knip": "^6.32.2",
"oxfmt": "^0.63.0",
"oxlint": "^1.78.0",
"oxlint-tsgolint": "^7.0.2001",
+10 -43
View File
@@ -58,7 +58,6 @@ import {
isApiResponseError,
updateSelfUser,
setBookTargetState,
type DownloadReleasePayload,
} from './services/api';
import type {
Book,
@@ -87,11 +86,13 @@ import { bookSupportsTargets } from './utils/bookTargetLoader';
import { buildSearchQuery } from './utils/buildSearchQuery';
import { wasDownloadQueuedAfterResponseError } from './utils/downloadRecovery';
import { getDynamicOptionGroup } from './utils/dynamicFieldOptions';
import { resolveDefaultLanguageCodes } from './utils/languageFilters';
import { getConfiguredMetadataProviderForContentType } from './utils/metadataProviders';
import { getEffectiveMetadataSort } from './utils/metadataSort';
import { isRecord } from './utils/objectHelpers';
import { policyTrace } from './utils/policyTrace';
import { buildQueryTargets, getDefaultQueryTargetKey } from './utils/queryTargets';
import { buildReleaseDownloadPayload, type ReleaseDownloadOptions } from './utils/releasePayload';
import { applyRequestNoteToPayload } from './utils/requestConfirmation';
import { bookFromRequestData } from './utils/requestFulfil';
import {
@@ -218,6 +219,7 @@ type PendingOnBehalfDownload =
release: Release;
releaseContentType: ContentType;
actingAsUser: ActingAsUserSelection;
options?: ReleaseDownloadOptions;
}
| {
type: 'combined';
@@ -1071,41 +1073,6 @@ function App() {
[],
);
const buildReleaseDownloadPayload = useCallback(
(book: Book, release: Release, releaseContentType: ContentType): DownloadReleasePayload => {
const isManual = book.provider === 'manual';
const releasePreview =
typeof release.extra?.preview === 'string' ? release.extra.preview : undefined;
const releaseAuthor =
typeof release.extra?.author === 'string' ? release.extra.author : undefined;
return {
source: release.source,
source_id: release.source_id,
title: isManual ? release.title : book.title,
author: isManual ? releaseAuthor || '' : book.author,
year: book.year,
format: release.format,
size: release.size,
size_bytes: release.size_bytes,
download_url: release.download_url,
protocol: release.protocol,
indexer: release.indexer,
seeders: release.seeders,
extra: release.extra,
preview: isManual ? releasePreview || undefined : book.preview,
content_type: releaseContentType,
series_name: book.series_name,
series_position: book.series_position,
subtitle: book.subtitle,
// From the release, never the book: book.language is the provider's
// canonical edition, which would mislabel a translated release.
language: release.language ?? undefined,
};
},
[],
);
// When downloading a book while browsing a Hardcover list the user owns,
// automatically remove it from that list (fire-and-forget).
const searchFieldLabelsRef = useRef(searchFieldLabels);
@@ -1213,12 +1180,13 @@ function App() {
release: Release,
releaseContentType: ContentType,
onBehalfOfUserId?: number,
options?: ReleaseDownloadOptions,
): Promise<void> => {
const requestStartedAtSeconds = Date.now() / 1000;
try {
trackRelease(book.id, release.source_id);
await downloadRelease(
buildReleaseDownloadPayload(book, release, releaseContentType),
buildReleaseDownloadPayload(book, release, releaseContentType, options),
onBehalfOfUserId,
);
await fetchStatus();
@@ -1300,7 +1268,6 @@ function App() {
}
},
[
buildReleaseDownloadPayload,
fetchStatus,
openRequestConfirmation,
refreshRequestPolicy,
@@ -1415,6 +1382,7 @@ function App() {
effectivePendingOnBehalfDownload.release,
effectivePendingOnBehalfDownload.releaseContentType,
onBehalfOfUserId,
effectivePendingOnBehalfDownload.options,
);
}
setPendingOnBehalfDownload(null);
@@ -1639,6 +1607,7 @@ function App() {
book: Book,
release: Release,
releaseContentType: ContentType,
options?: ReleaseDownloadOptions,
) => {
policyTrace('release.action:start', {
bookId: book.id,
@@ -1654,11 +1623,12 @@ function App() {
release,
releaseContentType,
actingAsUser: effectiveActingAsUser,
options,
});
return;
}
await executeReleaseDownload(book, release, releaseContentType);
await executeReleaseDownload(book, release, releaseContentType, undefined, options);
};
const handleReleaseRequest = useCallback(
@@ -1927,10 +1897,7 @@ function App() {
);
const supportedFormats = config?.supported_formats || DEFAULT_SUPPORTED_FORMATS;
const defaultLanguageCodes = useMemo(
() =>
config?.default_language && config.default_language.length > 0
? config.default_language
: [bookLanguages[0]?.code || 'en'],
() => resolveDefaultLanguageCodes(config?.default_language, bookLanguages),
[config?.default_language, bookLanguages],
);
@@ -0,0 +1,193 @@
import { useState } from 'react';
import type { PackBook, PackPlan, Release } from '../types';
import {
describePackPlan,
parseSeriesPositionInput,
toBookPlanPayload,
updateReviewBook,
} from '../utils/packReview';
import { ToggleSwitch } from './shared/ToggleSwitch';
interface PackReviewPanelProps {
release: Release;
plan: PackPlan;
books: PackBook[];
onChange: (books: PackBook[]) => void;
onBack: () => void;
/** `null` means "treat the whole release as one book". */
onConfirm: (books: PackBook[] | null) => Promise<void>;
isSubmitting: boolean;
}
const inputClassName =
'w-full rounded-md border border-(--border-muted) bg-(--bg) px-2 py-1 text-sm text-(--text) focus:border-emerald-500 focus:outline-none';
export const PackReviewPanel = ({
release,
plan,
books,
onChange,
onBack,
onConfirm,
isSubmitting,
}: PackReviewPanelProps) => {
const [singleBook, setSingleBook] = useState(false);
const [expandedFiles, setExpandedFiles] = useState<number | null>(null);
const [showIgnored, setShowIgnored] = useState(false);
const payloadBooks = toBookPlanPayload(books);
const canConfirm = !isSubmitting && (singleBook || payloadBooks.length > 0);
const confirmLabel = singleBook
? 'Download as one book'
: `Download ${payloadBooks.length} ${payloadBooks.length === 1 ? 'book' : 'books'}`;
return (
<div className="flex flex-col gap-4 px-5 py-4" data-testid="pack-review-panel">
<div>
<h3 className="text-base font-semibold text-(--text)">
This release contains several books
</h3>
<p className="mt-1 text-sm text-zinc-500 dark:text-zinc-400">
<span className="font-medium text-(--text)">{release.title}</span> ·{' '}
{describePackPlan(books, plan.ignored)}
</p>
<p className="mt-1 text-xs text-zinc-500 dark:text-zinc-400">
Each book below is filed separately with its own title. Fix any titles before downloading
— the author and series come from the book you searched.
</p>
</div>
<div className="flex items-center justify-between rounded-lg border border-(--border-muted) px-3 py-2">
<div>
<p className="text-sm font-medium text-(--text)">Treat as a single book</p>
<p className="text-xs text-zinc-500 dark:text-zinc-400">
Use this if the split is wrong and the files are really one audiobook.
</p>
</div>
<ToggleSwitch
checked={singleBook}
onChange={setSingleBook}
color="emerald"
ariaLabel="Treat as a single book"
disabled={isSubmitting}
/>
</div>
<div
className={`flex flex-col divide-y divide-zinc-200/60 dark:divide-zinc-800/60 ${
singleBook ? 'pointer-events-none opacity-40' : ''
}`}
>
<div className="grid grid-cols-[minmax(0,1fr)_72px_72px_80px] gap-2 pb-1 text-xs font-medium tracking-wide text-zinc-500 uppercase dark:text-zinc-400">
<span>Title</span>
<span>Series #</span>
<span>Year</span>
<span className="text-right">Files</span>
</div>
{books.map((book, index) => (
<div key={book.files[0] ?? index} className="py-2">
<div className="grid grid-cols-[minmax(0,1fr)_72px_72px_80px] items-center gap-2">
<input
type="text"
value={book.title}
onChange={(e) =>
onChange(updateReviewBook(books, index, { title: e.target.value }))
}
aria-label={`Title for book ${index + 1}`}
className={inputClassName}
disabled={isSubmitting}
/>
<input
type="text"
inputMode="decimal"
value={book.series_position ?? ''}
onChange={(e) =>
onChange(
updateReviewBook(books, index, {
series_position: parseSeriesPositionInput(e.target.value),
}),
)
}
aria-label={`Series position for book ${index + 1}`}
className={inputClassName}
disabled={isSubmitting}
/>
<input
type="text"
inputMode="numeric"
value={book.year ?? ''}
onChange={(e) => {
const parsed = parseSeriesPositionInput(e.target.value);
onChange(
updateReviewBook(books, index, {
year: parsed === null ? null : Math.trunc(parsed),
}),
);
}}
aria-label={`Year for book ${index + 1}`}
className={inputClassName}
disabled={isSubmitting}
/>
<button
type="button"
onClick={() => setExpandedFiles(expandedFiles === index ? null : index)}
className="hover-surface rounded-md px-2 py-1 text-right text-sm text-zinc-500 transition-colors dark:text-zinc-400"
aria-expanded={expandedFiles === index}
>
{book.files.length} {book.files.length === 1 ? 'file' : 'files'}
</button>
</div>
{expandedFiles === index && (
<ul className="mt-2 max-h-40 overflow-y-auto rounded-md bg-(--bg-soft) px-3 py-2 font-mono text-xs break-all text-zinc-600 dark:text-zinc-300">
{book.files.map((file) => (
<li key={file}>{file}</li>
))}
</ul>
)}
</div>
))}
</div>
{plan.ignored.length > 0 && (
<div>
<button
type="button"
onClick={() => setShowIgnored(!showIgnored)}
className="text-xs text-zinc-500 underline-offset-2 hover:underline dark:text-zinc-400"
aria-expanded={showIgnored}
>
{plan.ignored.length} {plan.ignored.length === 1 ? 'file' : 'files'} ignored (not a book
format)
</button>
{showIgnored && (
<ul className="mt-2 max-h-32 overflow-y-auto rounded-md bg-(--bg-soft) px-3 py-2 font-mono text-xs break-all text-zinc-600 dark:text-zinc-300">
{plan.ignored.map((file) => (
<li key={file}>{file}</li>
))}
</ul>
)}
</div>
)}
<div className="flex items-center justify-end gap-3 border-t border-(--border-muted) pt-4">
<button
type="button"
onClick={onBack}
disabled={isSubmitting}
className="hover-surface rounded-lg px-3 py-1.5 text-sm font-medium text-(--text) transition-colors disabled:opacity-50"
>
&larr; Back
</button>
<button
type="button"
onClick={() => void onConfirm(singleBook ? null : payloadBooks)}
disabled={!canConfirm}
className="rounded-lg bg-emerald-600 px-4 py-1.5 text-sm font-medium text-white transition-colors hover:bg-emerald-700 disabled:cursor-not-allowed disabled:opacity-50"
>
{isSubmitting ? 'Queuing…' : confirmLabel}
</button>
</div>
</div>
);
};
@@ -8,6 +8,7 @@ import {
toStringArray,
toStringValue,
} from '../utils/objectHelpers';
import { getUnrecognizedReleaseFormats } from '../utils/releaseFormats';
import { Tooltip } from './shared/Tooltip';
interface ReleaseCellProps {
@@ -424,6 +425,38 @@ export const ReleaseCell = ({
const primaryFormat = formats?.[0] || null;
const additionalFormats = formats?.slice(1) || [];
// The indexer named a format Shelfmark can't process (e.g. MAM "[ENG / AVI]").
// Downloading it would only fail post-processing, so warn instead of showing the
// bare content-type icon that makes it look like any other result.
const unrecognizedFormats = primaryFormat ? [] : getUnrecognizedReleaseFormats(release);
if (unrecognizedFormats.length > 0) {
const unsupportedLabel = unrecognizedFormats.map((fmt) => fmt.toUpperCase()).join(', ');
const unsupportedTitle = `Unsupported format (${unsupportedLabel}) - Shelfmark cannot process this release`;
if (compact) {
return (
<span
className="font-semibold text-amber-600 dark:text-amber-400"
title={unsupportedTitle}
>
{unrecognizedFormats[0].toUpperCase()}
{unrecognizedFormats.length > 1 && ` +${unrecognizedFormats.length - 1}`}
</span>
);
}
return (
<div className="flex items-center justify-start" title={unsupportedTitle}>
<span className="inline-flex items-center gap-1">
<span className="w-13 rounded-lg bg-amber-500/20 py-0.5 text-center text-[10px] font-semibold tracking-wide whitespace-nowrap text-amber-700 sm:text-[11px] dark:text-amber-400">
{unrecognizedFormats[0].toUpperCase()}
</span>
<span className="text-[10px] font-medium whitespace-nowrap text-amber-700 sm:text-[11px] dark:text-amber-400">
Unsupported
</span>
</span>
</div>
);
}
// Use blue for book, violet for audiobook when no format specified
const noFormatStyle = isAudiobook
? { bg: 'bg-violet-500/20', text: 'text-violet-600 dark:text-violet-400' }
+117 -2
View File
@@ -7,6 +7,7 @@ import { useReleaseSearchSession } from '../hooks/releaseModal/useReleaseSearchS
import { useTabIndicator } from '../hooks/ui/useTabIndicator';
import { useBodyScrollLock } from '../hooks/useBodyScrollLock';
import { useEscapeKey } from '../hooks/useEscapeKey';
import { inspectRelease } from '../services/api';
import type {
Book,
Release,
@@ -18,6 +19,8 @@ import type {
LeadingCellConfig,
ContentType,
RequestPolicyMode,
PackBook,
PackPlan,
} from '../types';
import { isMetadataBook } from '../types';
import { bookSupportsTargets } from '../utils/bookTargetLoader';
@@ -29,7 +32,9 @@ import {
buildLanguageNormalizer,
} from '../utils/languageFilters';
import { getNestedValue, toComparableText, toStringValue } from '../utils/objectHelpers';
import { toBookPlanPayload } from '../utils/packReview';
import { getReleaseFormats } from '../utils/releaseFormats';
import { buildReleaseDownloadPayload, type ReleaseDownloadOptions } from '../utils/releasePayload';
import {
getBookTitleCandidates,
getBookAuthorCandidates,
@@ -50,6 +55,7 @@ import { BookTargetDropdown } from './BookTargetDropdown';
import { Dropdown } from './Dropdown';
import { DropdownList } from './DropdownList';
import { LanguageMultiSelect } from './LanguageMultiSelect';
import { PackReviewPanel } from './PackReviewPanel';
import { ReleaseCell } from './ReleaseCell';
// Combined mode configuration for the ReleaseModal
@@ -140,7 +146,12 @@ const DEFAULT_COLUMN_CONFIG: ReleaseColumnConfig = {
interface ReleaseModalProps {
book: Book | null;
onClose: () => void;
onDownload: (book: Book, release: Release, contentType: ContentType) => Promise<void>;
onDownload: (
book: Book,
release: Release,
contentType: ContentType,
options?: ReleaseDownloadOptions,
) => Promise<void>;
onRequestRelease?: (book: Book, release: Release, contentType: ContentType) => Promise<void>;
onRequestBook?: (book: Book, contentType: ContentType) => Promise<void>;
getPolicyModeForSource?: (source: string, contentType: ContentType) => RequestPolicyMode;
@@ -762,6 +773,15 @@ const ReleaseModalSession = ({
: supportedFormats;
const [isRequestingBook, setIsRequestingBook] = useState(false);
const [selectedRelease, setSelectedRelease] = useState<Release | null>(null);
// Multi-book packs: `multiBook` is the manual header toggle (heuristic split for
// releases we can't inspect); `packReview` holds an inspected pack awaiting approval.
const [multiBook, setMultiBook] = useState(false);
const [packReview, setPackReview] = useState<{
release: Release;
plan: PackPlan;
books: PackBook[];
} | null>(null);
const [packSubmitting, setPackSubmitting] = useState(false);
const isCombinedMode = combinedMode != null;
const combinedPhase = combinedMode?.phase ?? null;
const combinedStepLabel = combinedMode?.stepLabel ?? '';
@@ -1196,7 +1216,36 @@ const ReleaseModalSession = ({
const mode = getReleaseActionMode(release);
if (mode === 'download') {
await onDownload(book, release, contentType);
// Look at the release's files before queueing so a whole-series pack can be
// reviewed and filed as separate books instead of one mangled item.
let inspected = false;
let plan: PackPlan | null = null;
let reason: string | null = null;
try {
const inspection = await inspectRelease(
buildReleaseDownloadPayload(book, release, contentType),
);
inspected = inspection.inspected;
plan = inspection.plan;
reason = inspection.reason;
} catch (error) {
console.error('Release inspection failed:', error);
}
if (inspected && plan?.is_pack) {
setPackReview({ release, plan, books: plan.books });
return;
}
// Not a pack (or couldn't be inspected): queue exactly as before. A release we
// couldn't inspect might still be an unnoticed pack, so leave a console breadcrumb
// rather than interrupting the user; the multi-book toggle forces the split.
if (!inspected && !multiBook) {
console.warn(
`Could not inspect release "${release.title}" before download${
reason ? `: ${reason}` : ''
}. If it contains several books, enable the multi-book pack toggle.`,
);
}
await onDownload(book, release, contentType, multiBook ? { multiBook: true } : {});
handleClose();
return;
}
@@ -1215,9 +1264,31 @@ const ReleaseModalSession = ({
onRequestRelease,
contentType,
handleClose,
multiBook,
],
);
const handlePackConfirm = useCallback(
async (books: PackBook[] | null): Promise<void> => {
if (!book || !packReview) {
return;
}
setPackSubmitting(true);
try {
await onDownload(
book,
packReview.release,
contentType,
books ? { multiBook: true, bookPlan: toBookPlanPayload(books) } : {},
);
handleClose();
} finally {
setPackSubmitting(false);
}
},
[book, packReview, onDownload, contentType, handleClose],
);
const titleId = `release-modal-title-${book.id}`;
const providerDisplay =
book.provider_display_name ||
@@ -1720,6 +1791,37 @@ const ReleaseModalSession = ({
</div>
<div className="flex items-center gap-3 pr-1 pl-2">
{/* Multi-book pack toggle (fallback for releases that can't be inspected) */}
{!isCombinedMode && (
<button
type="button"
onClick={() => setMultiBook((prev) => !prev)}
className={`hover-surface relative rounded-full p-2.5 text-zinc-500 transition-colors dark:text-zinc-400 ${
multiBook ? 'text-emerald-600 dark:text-emerald-400' : ''
}`}
aria-label="Multi-book pack"
aria-pressed={multiBook}
title="Multi-book pack: file each subfolder (or each file) as a separate book. Only needed when a release can't be inspected before download."
>
<svg
className="h-4 w-4"
fill="none"
stroke="currentColor"
viewBox="0 0 24 24"
strokeWidth={1.5}
>
<path
strokeLinecap="round"
strokeLinejoin="round"
d="M6.429 9.75 2.25 12l4.179 2.25m0-4.5 5.571 3 5.571-3m-11.142 0L2.25 7.5 12 2.25l9.75 5.25-4.179 2.25m0 0L21.75 12l-4.179 2.25m0 0 4.179 2.25L12 21.75 2.25 16.5l4.179-2.25m11.142 0-5.571 3-5.571-3"
/>
</svg>
{multiBook && (
<span className="absolute top-1 right-1 h-2 w-2 rounded-full bg-emerald-500" />
)}
</button>
)}
{/* Manual query button */}
<button
type="button"
@@ -2120,6 +2222,19 @@ const ReleaseModalSession = ({
{/* Release list content */}
<div className="min-h-[200px]">
{(() => {
if (packReview) {
return (
<PackReviewPanel
release={packReview.release}
plan={packReview.plan}
books={packReview.books}
onChange={(books) => setPackReview({ ...packReview, books })}
onBack={() => setPackReview(null)}
onConfirm={handlePackConfirm}
isSubmitting={packSubmitting}
/>
);
}
if (sourcesLoading) {
return <ReleaseSkeleton />;
}
+25 -13
View File
@@ -223,6 +223,14 @@ export const SearchBar = forwardRef<SearchBarHandle, SearchBarProps>(
);
const showActiveTargetLabel = queryTargets.length > 0 && activeTarget.source !== 'general';
// Manual search browses release sources directly, one media type at a time — the
// combined ("both") flow doesn't apply. Present a plain, switchable Books/Audiobooks
// choice for it, even when combined search is forced on for metadata targets.
const isManualTarget = activeTarget?.source === 'manual';
const combinedSelectionActive = combinedMode && !isManualTarget;
const combinedSelectorLocked = combinedModeLocked && !isManualTarget;
const combinedToggleAvailable = !!onCombinedModeChange && !isManualTarget;
useDismiss(isSelectorOpen, [selectorRef], () => setIsSelectorOpen(false));
useDismiss(isSelectOpen, [selectPanelRef, selectTriggerRef], () => setIsSelectOpen(false));
useDismiss(isAutocompleteOpen, [autocompletePanelRef, inputRef], () =>
@@ -356,7 +364,7 @@ export const SearchBar = forwardRef<SearchBarHandle, SearchBarProps>(
contentType,
activeTarget,
placeholder,
combinedMode,
combinedSelectionActive,
);
const effectiveInputAriaLabel = activeTarget
? `${inputAriaLabel}: ${activeTarget.label}`
@@ -537,7 +545,7 @@ export const SearchBar = forwardRef<SearchBarHandle, SearchBarProps>(
let selectorContentTypeLabel = 'audiobooks';
let selectorIcon = <AudiobookIcon />;
if (combinedMode) {
if (combinedSelectionActive) {
selectorContentTypeLabel = 'books and audiobooks';
selectorIcon = <BothIcon />;
} else if (contentType === 'ebook') {
@@ -665,7 +673,7 @@ export const SearchBar = forwardRef<SearchBarHandle, SearchBarProps>(
<div className="max-h-[min(24rem,calc(100vh-8rem))] overflow-y-auto p-3">
{showContentTypeSelector && (
<div
className={`border-b ${onCombinedModeChange ? 'pb-0' : 'pb-3'}`}
className={`border-b ${combinedToggleAvailable ? 'pb-0' : 'pb-3'}`}
style={{ borderColor: 'var(--border-muted)' }}
>
<div className="flex items-center justify-between px-1 pb-2">
@@ -712,34 +720,38 @@ export const SearchBar = forwardRef<SearchBarHandle, SearchBarProps>(
type="button"
onClick={() => handleContentTypeSelect('ebook')}
className={`flex items-center gap-2 rounded-xl border px-3 py-2 text-sm font-medium transition-colors ${
contentType === 'ebook' || combinedMode
contentType === 'ebook' || combinedSelectionActive
? 'bg-emerald-600 text-white'
: 'hover-surface'
}`}
style={
contentType === 'ebook' || combinedMode
contentType === 'ebook' || combinedSelectionActive
? { borderColor: 'rgb(16 185 129 / 0.7)' }
: { color: 'var(--text)', borderColor: 'var(--border-muted)' }
}
>
{contentType === 'ebook' || combinedMode ? <CheckIcon /> : <BookIcon />}
{contentType === 'ebook' || combinedSelectionActive ? (
<CheckIcon />
) : (
<BookIcon />
)}
<span>Books</span>
</button>
<button
type="button"
onClick={() => handleContentTypeSelect('audiobook')}
className={`flex items-center gap-2 rounded-xl border px-3 py-2 text-sm font-medium transition-colors ${
contentType === 'audiobook' || combinedMode
contentType === 'audiobook' || combinedSelectionActive
? 'bg-emerald-600 text-white'
: 'hover-surface'
}`}
style={
contentType === 'audiobook' || combinedMode
contentType === 'audiobook' || combinedSelectionActive
? { borderColor: 'rgb(16 185 129 / 0.7)' }
: { color: 'var(--text)', borderColor: 'var(--border-muted)' }
}
>
{contentType === 'audiobook' || combinedMode ? (
{contentType === 'audiobook' || combinedSelectionActive ? (
<CheckIcon />
) : (
<AudiobookIcon />
@@ -747,9 +759,9 @@ export const SearchBar = forwardRef<SearchBarHandle, SearchBarProps>(
<span>Audiobooks</span>
</button>
</div>
{onCombinedModeChange &&
{combinedToggleAvailable &&
(() => {
const lineColor = combinedMode
const lineColor = combinedSelectionActive
? 'bg-emerald-500'
: 'bg-(--border-muted) group-hover:bg-zinc-400 dark:group-hover:bg-zinc-500';
return (
@@ -787,7 +799,7 @@ export const SearchBar = forwardRef<SearchBarHandle, SearchBarProps>(
{/* Chain icon centered at bottom */}
<div
className={`relative z-10 mx-auto rounded-full p-1 transition-colors ${
combinedMode
combinedSelectionActive
? 'bg-emerald-600 text-white'
: 'bg-(--bg) text-zinc-400 group-hover:bg-zinc-200 group-hover:text-zinc-600 dark:text-zinc-500 dark:group-hover:bg-zinc-700 dark:group-hover:text-zinc-300'
}`}
@@ -800,7 +812,7 @@ export const SearchBar = forwardRef<SearchBarHandle, SearchBarProps>(
stroke="currentColor"
aria-hidden="true"
>
{combinedModeLocked ? (
{combinedSelectorLocked ? (
<path
strokeLinecap="round"
strokeLinejoin="round"
@@ -3,6 +3,7 @@ import type { Dispatch, SetStateAction } from 'react';
import { useMountEffect } from '@/hooks/useMountEffect';
import type { AppConfig, AdvancedFilterState, ContentType, SearchMode, SortOption } from '@/types';
import { buildSearchQuery } from '@/utils/buildSearchQuery';
import { resolveDefaultLanguageCodes } from '@/utils/languageFilters';
import { getEffectiveMetadataSort } from '@/utils/metadataSort';
import type { ParsedUrlSearch } from '@/utils/parseUrlSearchParams';
@@ -73,10 +74,10 @@ export const UrlSearchBootstrapMount = ({
}
const bookLanguages = config.book_languages || [];
const defaultLanguageCodes =
config.default_language && config.default_language.length > 0
? config.default_language
: [bookLanguages[0]?.code || 'en'];
const defaultLanguageCodes = resolveDefaultLanguageCodes(
config.default_language,
bookLanguages,
);
if (parsedParams.searchInput) {
setSearchInput(parsedParams.searchInput);
@@ -353,12 +353,12 @@ export const TableField = ({ field, value, onChange, disabled }: TableFieldProps
);
}
// text/path
// text/password/path
return (
<div key={col.key} className="flex min-w-0 flex-col gap-1">
{mobileLabel}
<input
type="text"
type={col.type === 'password' ? 'password' : 'text'}
value={toPrimitiveString(cellValue)}
onChange={(e) => updateCell(rowIndex, col.key, e.target.value)}
placeholder={col.placeholder}
@@ -7,7 +7,12 @@ import type {
} from '../../../types/settings';
import { HeadingField, MultiSelectField, SelectField, TextField } from '../fields';
import { FieldWrapper } from '../shared';
import { getFieldByKey, toNormalizedLowercaseTextValue, toTextValue } from './fieldHelpers';
import {
getFieldByKey,
resolveListOverride,
toNormalizedLowercaseTextValue,
toTextValue,
} from './fieldHelpers';
import type { PerUserSettings } from './types';
interface UserOverridesSectionProps {
@@ -175,16 +180,6 @@ export const UserOverridesSection = ({
label: 'Email Recipient',
description: 'Email address used for this user in Email output mode.',
};
const browserDownloadGlobalValue = Array.isArray(globalValues.DOWNLOAD_TO_BROWSER_CONTENT_TYPES)
? globalValues.DOWNLOAD_TO_BROWSER_CONTENT_TYPES.map((entry) => String(entry).trim()).filter(
(entry) => entry.length > 0,
)
: [];
const browserDownloadUserValue = Array.isArray(userSettings.DOWNLOAD_TO_BROWSER_CONTENT_TYPES)
? userSettings.DOWNLOAD_TO_BROWSER_CONTENT_TYPES.map((entry) => entry.trim()).filter(
(entry) => entry.length > 0,
)
: [];
const isOverridden = (key: DeliverySettingKey): boolean => {
if (
@@ -200,10 +195,12 @@ export const UserOverridesSection = ({
return userValue !== globalValue;
};
const isBrowserDownloadOverridden =
Object.prototype.hasOwnProperty.call(userSettings, 'DOWNLOAD_TO_BROWSER_CONTENT_TYPES') &&
userSettings.DOWNLOAD_TO_BROWSER_CONTENT_TYPES !== null &&
JSON.stringify(browserDownloadUserValue) !== JSON.stringify(browserDownloadGlobalValue);
const { value: browserDownloadContentTypes, isOverridden: isBrowserDownloadOverridden } =
resolveListOverride(
userSettings.DOWNLOAD_TO_BROWSER_CONTENT_TYPES,
globalValues.DOWNLOAD_TO_BROWSER_CONTENT_TYPES,
Object.prototype.hasOwnProperty.call(userSettings, 'DOWNLOAD_TO_BROWSER_CONTENT_TYPES'),
);
const resetKeys = (keys: DeliverySettingKey[]) => {
setUserSettings((prev) => {
@@ -232,9 +229,6 @@ export const UserOverridesSection = ({
const outputModeValue = readValue('BOOKS_OUTPUT_MODE', 'folder');
const effectiveOutputMode = normalizeMode(outputModeValue);
const browserDownloadContentTypes = isBrowserDownloadOverridden
? browserDownloadUserValue
: browserDownloadGlobalValue;
const destinationValue = readValue('DESTINATION');
const destinationAudiobookValue = readValue('DESTINATION_AUDIOBOOK');
const libraryValue = readValue('BOOKLORE_LIBRARY_ID');
@@ -1,8 +1,17 @@
import type { DeliveryPreferencesResponse } from '../../../services/api';
import type { HeadingFieldConfig, SelectFieldConfig } from '../../../types/settings';
import { HeadingField, SelectField } from '../fields';
import type {
HeadingFieldConfig,
MultiSelectFieldConfig,
SelectFieldConfig,
} from '../../../types/settings';
import { HeadingField, MultiSelectField, SelectField } from '../fields';
import { FieldWrapper } from '../shared';
import { getFieldByKey, toNormalizedLowercaseTextValue, toTextValue } from './fieldHelpers';
import {
getFieldByKey,
resolveListOverride,
toNormalizedLowercaseTextValue,
toTextValue,
} from './fieldHelpers';
import type { PerUserSettings } from './types';
interface UserSearchPreferencesSectionProps {
@@ -14,6 +23,7 @@ interface UserSearchPreferencesSectionProps {
type SearchSettingKey =
| 'SEARCH_MODE'
| 'BOOK_LANGUAGE'
| 'METADATA_PROVIDER'
| 'METADATA_PROVIDER_AUDIOBOOK'
| 'DEFAULT_RELEASE_SOURCE'
@@ -68,6 +78,15 @@ const fallbackDefaultAudiobookReleaseSourceField: SelectFieldConfig = {
options: [{ value: '', label: 'Use book release source' }],
};
const fallbackBookLanguageField: MultiSelectFieldConfig = {
type: 'MultiSelectField',
key: 'BOOK_LANGUAGE',
label: 'Default Book Languages',
description: 'Default language filter for searches.',
value: [],
options: [],
};
const searchHeading: HeadingFieldConfig = {
type: 'HeadingField',
key: 'search_preferences_heading',
@@ -120,6 +139,13 @@ export const UserSearchPreferencesSection = ({
'DEFAULT_RELEASE_SOURCE_AUDIOBOOK',
fallbackDefaultAudiobookReleaseSourceField,
);
const bookLanguageField = getFieldByKey(fields, 'BOOK_LANGUAGE', fallbackBookLanguageField);
const { value: bookLanguageValue, isOverridden: isBookLanguageOverridden } = resolveListOverride(
userSettings.BOOK_LANGUAGE,
globalValues.BOOK_LANGUAGE,
Object.prototype.hasOwnProperty.call(userSettings, 'BOOK_LANGUAGE'),
);
const isOverridden = (key: SearchSettingKey): boolean => {
if (
@@ -172,9 +198,12 @@ export const UserSearchPreferencesSection = ({
const canOverrideDefaultAudiobookReleaseSource =
isUserOverridable('DEFAULT_RELEASE_SOURCE_AUDIOBOOK') &&
preferenceKeySet.has('DEFAULT_RELEASE_SOURCE_AUDIOBOOK');
const canOverrideBookLanguage =
isUserOverridable('BOOK_LANGUAGE') && preferenceKeySet.has('BOOK_LANGUAGE');
if (
!canOverrideSearchMode &&
!canOverrideBookLanguage &&
!canOverrideMetadataProvider &&
!canOverrideAudiobookMetadataProvider &&
!canOverrideDefaultReleaseSource &&
@@ -208,6 +237,27 @@ export const UserSearchPreferencesSection = ({
</FieldWrapper>
)}
{canOverrideBookLanguage && (
<FieldWrapper
field={bookLanguageField}
resetAction={
isBookLanguageOverridden
? {
disabled: Boolean(bookLanguageField.fromEnv),
onClick: () => resetKeys(['BOOK_LANGUAGE']),
}
: undefined
}
>
<MultiSelectField
field={bookLanguageField}
value={bookLanguageValue}
onChange={(value) => setUserSettings((prev) => ({ ...prev, BOOK_LANGUAGE: value }))}
disabled={Boolean(bookLanguageField.fromEnv)}
/>
</FieldWrapper>
)}
{effectiveSearchMode === 'universal' && canOverrideMetadataProvider && (
<FieldWrapper
field={metadataProviderField}
@@ -31,6 +31,33 @@ export const toNormalizedLowercaseTextValue = (value: unknown): string => {
return toTrimmedTextValue(value).toLowerCase();
};
const toStringListValue = (value: unknown): string[] => {
if (!Array.isArray(value)) {
return [];
}
return value.map((entry) => toTrimmedTextValue(entry)).filter((entry) => entry.length > 0);
};
/**
* Resolve a list-valued per-user override against its global value.
*
* A key absent from userSettings, or set to null, is not an override. A stored list
* that matches the global one is treated as inherited, matching how
* buildUserSettingsPayload clears it on save.
*/
export const resolveListOverride = (
userValue: unknown,
globalValue: unknown,
hasUserKey: boolean,
): { value: string[]; isOverridden: boolean } => {
const globalList = toStringListValue(globalValue);
const userList = toStringListValue(userValue);
const isOverridden =
hasUserKey && userValue !== null && JSON.stringify(userList) !== JSON.stringify(globalList);
return { value: isOverridden ? userList : globalList, isOverridden };
};
export const toComparableValue = (value: unknown): string => {
if (value === null || value === undefined) {
return '';
@@ -10,6 +10,7 @@ export interface PerUserSettings {
EMAIL_RECIPIENT?: string;
DOWNLOAD_TO_BROWSER_CONTENT_TYPES?: string[];
SEARCH_MODE?: string;
BOOK_LANGUAGE?: string[];
METADATA_PROVIDER?: string;
METADATA_PROVIDER_AUDIOBOOK?: string;
DEFAULT_RELEASE_SOURCE?: string;
+14
View File
@@ -12,6 +12,8 @@ import type {
RequestSubmissionResult,
MetadataProvidersResponse,
MetadataSearchConfig,
PackBook,
InspectReleaseResponse,
} from '../types';
import type {
ActionResult,
@@ -510,6 +512,18 @@ export type DownloadReleasePayload = {
language?: string; // Release language code, for the {Language} naming variable
search_author?: string;
search_mode?: 'direct' | 'universal';
multi_book?: boolean; // Split a multi-book pack into one book per subfolder/file
book_plan?: PackBook[]; // The split the user approved before download
};
/** Inspect a release's file list before download (same body as downloadRelease). */
export const inspectRelease = async (
release: DownloadReleasePayload,
): Promise<InspectReleaseResponse> => {
return fetchJSON<InspectReleaseResponse>(`${API_BASE}/releases/inspect`, {
method: 'POST',
body: JSON.stringify(release),
});
};
export const downloadRelease = async (
@@ -7,6 +7,7 @@ import {
buildLanguageNormalizer,
getReleaseSearchLanguageParams,
releaseLanguageMatchesFilter,
resolveDefaultLanguageCodes,
} from '../utils/languageFilters';
const supportedLanguages: Language[] = [
@@ -67,3 +68,36 @@ describe('releaseLanguageMatchesFilter', () => {
expect(visibleLanguages).toHaveLength(48);
});
});
describe('resolveDefaultLanguageCodes', () => {
it('keeps an explicitly empty default as "no default filter"', () => {
// The backend stores [] to mean "do not filter"; substituting the first
// supported language here would make the UI filter where the server does not.
expect(resolveDefaultLanguageCodes([], supportedLanguages)).toEqual([]);
});
it('leaves a configured default untouched', () => {
expect(resolveDefaultLanguageCodes(['de', 'hu'], supportedLanguages)).toEqual(['de', 'hu']);
});
it('falls back to the first supported language only when nothing is configured', () => {
expect(resolveDefaultLanguageCodes(undefined, supportedLanguages)).toEqual(['en']);
expect(resolveDefaultLanguageCodes(null, [])).toEqual(['en']);
});
it('sends no language filter when an empty default is the whole selection', () => {
const defaults = resolveDefaultLanguageCodes([], supportedLanguages);
expect(
getReleaseSearchLanguageParams([LANGUAGE_OPTION_DEFAULT], supportedLanguages, defaults),
).toBe(undefined);
});
it('does not smuggle the first language into a Default+German selection', () => {
const defaults = resolveDefaultLanguageCodes([], supportedLanguages);
expect(
getReleaseSearchLanguageParams([LANGUAGE_OPTION_DEFAULT, 'de'], supportedLanguages, defaults),
).toEqual(['de']);
});
});
+53
View File
@@ -0,0 +1,53 @@
import { describe, it, expect } from 'vitest';
import type { PackBook } from '../types';
import {
describePackPlan,
parseSeriesPositionInput,
toBookPlanPayload,
updateReviewBook,
} from '../utils/packReview';
const books: PackBook[] = [
{ title: 'Leviathan Wakes', series_position: 1, year: 2011, files: ['a.m4b'] },
{ title: 'Caliban’s War', series_position: 2, year: 2012, files: ['b.m4b', 'b2.m4b'] },
];
describe('packReview.updateReviewBook', () => {
it('replaces one book without touching the others', () => {
const next = updateReviewBook(books, 1, { title: 'Caliban’s War (Unabridged)' });
expect(next[0]).toBe(books[0]);
expect(next[1]).toEqual({ ...books[1], title: 'Caliban’s War (Unabridged)' });
expect(books[1].title).toBe('Caliban’s War');
});
});
describe('packReview.parseSeriesPositionInput', () => {
it('accepts whole and fractional positions', () => {
expect(parseSeriesPositionInput('3')).toBe(3);
expect(parseSeriesPositionInput('2.5')).toBe(2.5);
});
it('treats blank or junk as no position', () => {
expect(parseSeriesPositionInput('')).toBeNull();
expect(parseSeriesPositionInput('abc')).toBeNull();
});
});
describe('packReview.toBookPlanPayload', () => {
it('trims titles, drops books without a title, and keeps file lists', () => {
const edited = updateReviewBook(books, 0, { title: ' ' });
expect(toBookPlanPayload(edited)).toEqual([
{ title: 'Caliban’s War', series_position: 2, year: 2012, files: ['b.m4b', 'b2.m4b'] },
]);
});
});
describe('packReview.describePackPlan', () => {
it('summarises books, files and ignored sidecars', () => {
expect(describePackPlan(books, ['a.txt', 'cover.jpg'])).toBe(
'2 books · 3 files · 2 files ignored',
);
expect(describePackPlan([books[0]], [])).toBe('1 book · 1 file');
});
});
+24 -1
View File
@@ -1,7 +1,7 @@
import { describe, it, expect } from 'vitest';
import type { Release } from '../types';
import { getReleaseFormats } from '../utils/releaseFormats';
import { getReleaseFormats, getUnrecognizedReleaseFormats } from '../utils/releaseFormats';
function buildRelease(overrides: Partial<Release>): Release {
return {
@@ -38,3 +38,26 @@ describe('releaseFormats.getReleaseFormats', () => {
expect(getReleaseFormats(release)).toEqual(['pdf']);
});
});
describe('releaseFormats.getUnrecognizedReleaseFormats', () => {
it('returns normalized, deduplicated unrecognized formats from extra', () => {
const release = buildRelease({
extra: { unrecognized_formats: ['AVI', ' avi ', 'WEBM'] },
});
expect(getUnrecognizedReleaseFormats(release)).toEqual(['avi', 'webm']);
});
it('accepts a single string value', () => {
const release = buildRelease({ extra: { unrecognized_formats: 'AVI' } });
expect(getUnrecognizedReleaseFormats(release)).toEqual(['avi']);
});
it('returns an empty list when nothing was flagged', () => {
expect(getUnrecognizedReleaseFormats(buildRelease({}))).toEqual([]);
expect(
getUnrecognizedReleaseFormats(buildRelease({ extra: { unrecognized_formats: null } })),
).toEqual([]);
});
});
@@ -0,0 +1,69 @@
import { describe, it, expect } from 'vitest';
import type { Book, Release } from '../types';
import { buildReleaseDownloadPayload } from '../utils/releasePayload';
const book: Book = {
id: 'hc-1',
title: 'Drive',
author: 'James S. A. Corey',
year: '2012',
preview: 'https://img/drive.jpg',
series_name: 'The Expanse',
series_position: 2.6,
subtitle: 'An Expanse Short Story',
provider: 'hardcover',
provider_id: 'hc-1',
source: 'direct_download',
};
const release: Release = {
source: 'audiobookbay',
source_id: 'abb-1',
title: 'James S. A. Corey - The Expanse Complete 2.0',
format: 'm4b',
language: 'en',
download_url: 'https://audiobookbay.lu/abss/expanse/',
};
describe('buildReleaseDownloadPayload', () => {
it('describes the searched book and the chosen release', () => {
const payload = buildReleaseDownloadPayload(book, release, 'audiobook');
expect(payload).toMatchObject({
source: 'audiobookbay',
source_id: 'abb-1',
title: 'Drive',
author: 'James S. A. Corey',
series_name: 'The Expanse',
series_position: 2.6,
language: 'en',
content_type: 'audiobook',
});
expect(payload.multi_book).toBeUndefined();
expect(payload.book_plan).toBeUndefined();
});
it('uses the release title and author for manual books', () => {
const manual: Book = { ...book, provider: 'manual', title: 'ignored' };
const withAuthor = { ...release, extra: { author: 'Release Author' } };
const payload = buildReleaseDownloadPayload(manual, withAuthor, 'audiobook');
expect(payload.title).toBe(release.title);
expect(payload.author).toBe('Release Author');
});
it('flags a manual multi-book pack', () => {
const payload = buildReleaseDownloadPayload(book, release, 'audiobook', { multiBook: true });
expect(payload.multi_book).toBe(true);
expect(payload.book_plan).toBeUndefined();
});
it('attaches the approved book plan', () => {
const plan = [{ title: 'Leviathan Wakes', series_position: 1, year: 2011, files: ['a.m4b'] }];
const payload = buildReleaseDownloadPayload(book, release, 'audiobook', {
multiBook: true,
bookPlan: plan,
});
expect(payload.multi_book).toBe(true);
expect(payload.book_plan).toEqual(plan);
});
});
+21
View File
@@ -460,6 +460,27 @@ export interface SourceSearchInfo {
}
// Response from /api/releases endpoint
/** One book split out of a multi-book pack release, files as release-relative paths. */
export interface PackBook {
title: string;
series_position: number | null;
year: number | null;
files: string[];
}
export interface PackPlan {
is_pack: boolean;
books: PackBook[];
ignored: string[];
}
export interface InspectReleaseResponse {
inspected: boolean;
reason: string | null;
files: { path: string; size: number | null }[];
plan: PackPlan | null;
}
export interface ReleasesResponse {
releases: Release[];
book: {
+7 -1
View File
@@ -146,7 +146,13 @@ export interface TableFieldColumnOption {
childOf?: string;
}
export type TableFieldColumnType = 'text' | 'select' | 'multiselect' | 'checkbox' | 'path';
export type TableFieldColumnType =
| 'text'
| 'password'
| 'select'
| 'multiselect'
| 'checkbox'
| 'path';
export interface TableFieldColumn {
key: string;
+18
View File
@@ -27,6 +27,24 @@ export const normalizeLanguageSelection = (selected: string[]): string[] => {
return unique.length ? unique : [LANGUAGE_OPTION_DEFAULT];
};
/**
* Resolve the language codes the "Default" filter option stands for.
*
* An explicitly empty list is a deliberate "no default filter" and is returned as-is;
* only a missing value falls back to the first supported language. Substituting a
* language for the empty list would make the UI filter by a language the backend
* does not apply.
*/
export const resolveDefaultLanguageCodes = (
configuredDefault: string[] | null | undefined,
supportedLanguages: Language[],
): string[] => {
if (Array.isArray(configuredDefault)) {
return configuredDefault;
}
return [supportedLanguages[0]?.code || 'en'];
};
export const getLanguageFilterValues = (
selection: string[],
supportedLanguages: Language[],
+39
View File
@@ -0,0 +1,39 @@
import type { PackBook } from '../types';
/** Return a copy of `books` with one entry patched; the input is not mutated. */
export function updateReviewBook(
books: PackBook[],
index: number,
patch: Partial<PackBook>,
): PackBook[] {
return books.map((book, i) => (i === index ? { ...book, ...patch } : book));
}
/** Parse a series-position text field: "3" → 3, "2.5" → 2.5, blank/junk → null. */
export function parseSeriesPositionInput(value: string): number | null {
const trimmed = value.trim();
if (!trimmed) return null;
const parsed = Number(trimmed);
return Number.isFinite(parsed) ? parsed : null;
}
/** The plan sent with the download: trimmed titles, untitled books dropped. */
export function toBookPlanPayload(books: PackBook[]): PackBook[] {
return books
.map((book) => ({ ...book, title: book.title.trim() }))
.filter((book) => book.title.length > 0 && book.files.length > 0);
}
function plural(count: number, noun: string): string {
return `${count} ${noun}${count === 1 ? '' : 's'}`;
}
/** "2 books · 3 files · 2 files ignored" */
export function describePackPlan(books: PackBook[], ignored: string[]): string {
const fileCount = books.reduce((sum, book) => sum + book.files.length, 0);
const parts = [plural(books.length, 'book'), plural(fileCount, 'file')];
if (ignored.length > 0) {
parts.push(`${plural(ignored.length, 'file')} ignored`);
}
return parts.join(' · ');
}
+24
View File
@@ -33,3 +33,27 @@ export function getReleaseFormats(release: Release): string[] {
return formats;
}
/**
* Format tokens the indexer declared but the backend could not map to a known
* book/audiobook format (e.g. MyAnonamouse "[ENG / AVI]"). Such a release will
* download but fail post-processing, so the UI warns instead of showing a bare
* content-type icon.
*/
export function getUnrecognizedReleaseFormats(release: Release): string[] {
const raw = release.extra?.unrecognized_formats;
const values = Array.isArray(raw) ? raw : [raw];
const formats: string[] = [];
const seen = new Set<string>();
values.forEach((value) => {
const normalized = normalizeFormatValue(value);
if (!normalized || seen.has(normalized)) {
return;
}
seen.add(normalized);
formats.push(normalized);
});
return formats;
}
+55
View File
@@ -0,0 +1,55 @@
import type { DownloadReleasePayload } from '../services/api';
import type { Book, ContentType, PackBook, Release } from '../types';
export interface ReleaseDownloadOptions {
/** Ask post-processing to split the release into one book per subfolder/file. */
multiBook?: boolean;
/** The split the user approved in the pack review panel. */
bookPlan?: PackBook[];
}
/** Build the body for /api/releases/download (and /api/releases/inspect). */
export function buildReleaseDownloadPayload(
book: Book,
release: Release,
releaseContentType: ContentType,
options: ReleaseDownloadOptions = {},
): DownloadReleasePayload {
const isManual = book.provider === 'manual';
const releasePreview =
typeof release.extra?.preview === 'string' ? release.extra.preview : undefined;
const releaseAuthor =
typeof release.extra?.author === 'string' ? release.extra.author : undefined;
const payload: DownloadReleasePayload = {
source: release.source,
source_id: release.source_id,
title: isManual ? release.title : book.title,
author: isManual ? releaseAuthor || '' : book.author,
year: book.year,
format: release.format,
size: release.size,
size_bytes: release.size_bytes,
download_url: release.download_url,
protocol: release.protocol,
indexer: release.indexer,
seeders: release.seeders,
extra: release.extra,
preview: isManual ? releasePreview || undefined : book.preview,
content_type: releaseContentType,
series_name: book.series_name,
series_position: book.series_position,
subtitle: book.subtitle,
// From the release, never the book: book.language is the provider's
// canonical edition, which would mislabel a translated release.
language: release.language ?? undefined,
};
if (options.multiBook || options.bookPlan) {
payload.multi_book = true;
}
if (options.bookPlan) {
payload.book_plan = options.bookPlan;
}
return payload;
}
+13
View File
@@ -0,0 +1,13 @@
"""AudiobookBay test fixtures."""
import pytest
from shelfmark.release_sources.audiobookbay import scraper
@pytest.fixture(autouse=True)
def _clear_detail_page_cache():
"""Detail pages are cached briefly in production; tests must not share them."""
scraper.clear_detail_page_cache()
yield
scraper.clear_detail_page_cache()
+102
View File
@@ -0,0 +1,102 @@
"""Tests for reading the torrent file list off an AudiobookBay detail page."""
from shelfmark.download.postprocess.packs import PackFile
from shelfmark.release_sources.audiobookbay import scraper
# Trimmed from a real detail page (2026-08): the file rows sit between the
# "Multifile Torrent" marker and the "Combined File Size" row.
MULTIFILE_DETAIL_HTML = """
<table>
<tr><td>Tracker:</td><td>udp://tracker.torrent.eu.org:451/announce</td></tr>
<tr><td>Creation Date:</td><td>Sun, 29 Mar 2026 21:09:39 +0200</td></tr>
<tr><td colspan='2'>This is a Multifile Torrent</td></tr>
<tr><td colspan='2'>The Expanse 9.0 - Leviathan Falls (2021).m4b 1.05 GBs</td></tr>
<tr><td colspan='2'>The Expanse 0.1 - An Expanse Novella - Drive (2012).txt 340 Bytes</td></tr>
<tr><td colspan='2'>The Expanse 0.2 - An Expanse Novella - The Churn (2014).m4b 125.72 MBs</td></tr>
<tr><td colspan='2'>The Expanse 2.0 - Caliban’s War (2012).m4b 578.97 MBs</td></tr>
<tr><td>Combined File Size:</td><td><span style='color:#00f;'>7.87</span> GBs</td></tr>
<tr><td>Info Hash:</td><td>e4a5538e26987ee58a43aa629ec2c4f2b2d46526</td></tr>
</table>
"""
SINGLE_FILE_DETAIL_HTML = """
<table>
<tr><td>Creation Date:</td><td>Sun, 29 Mar 2026 21:09:39 +0200</td></tr>
<tr><td colspan='2'>Drive.m4b 41.55 MBs</td></tr>
<tr><td>Combined File Size:</td><td><span style='color:#00f;'>41.55</span> MBs</td></tr>
<tr><td>Info Hash:</td><td>e4a5538e26987ee58a43aa629ec2c4f2b2d46526</td></tr>
</table>
"""
def test_extracts_multifile_rows_with_byte_sizes():
files = scraper.extract_file_list(MULTIFILE_DETAIL_HTML)
assert files == [
PackFile("The Expanse 9.0 - Leviathan Falls (2021).m4b", int(1.05 * 1024**3)),
PackFile("The Expanse 0.1 - An Expanse Novella - Drive (2012).txt", 340),
PackFile(
"The Expanse 0.2 - An Expanse Novella - The Churn (2014).m4b", int(125.72 * 1024**2)
),
PackFile("The Expanse 2.0 - Caliban’s War (2012).m4b", int(578.97 * 1024**2)),
]
def test_single_file_torrent_lists_the_row_before_combined_size():
assert scraper.extract_file_list(SINGLE_FILE_DETAIL_HTML) == [
PackFile("Drive.m4b", int(41.55 * 1024**2))
]
def test_page_without_file_table_returns_none():
assert scraper.extract_file_list("<html><body><p>nothing here</p></body></html>") is None
class TestHandlerListFiles:
def test_lists_files_from_detail_page(self):
from unittest.mock import patch
from shelfmark.release_sources.audiobookbay.handler import AudiobookBayHandler
with patch(
"shelfmark.release_sources.audiobookbay.handler.scraper.fetch_detail_html",
return_value=SINGLE_FILE_DETAIL_HTML,
) as fetch:
files = AudiobookBayHandler().list_files(
{"source_id": "abc", "download_url": "https://audiobookbay.lu/abss/drive/"}
)
assert files == [PackFile("Drive.m4b", int(41.55 * 1024**2))]
fetch.assert_called_once_with("https://audiobookbay.lu/abss/drive/", "audiobookbay.lu")
def test_rejects_detail_url_on_other_host(self):
from unittest.mock import patch
from shelfmark.release_sources.audiobookbay.handler import AudiobookBayHandler
with patch(
"shelfmark.release_sources.audiobookbay.handler.scraper.fetch_detail_html"
) as fetch:
files = AudiobookBayHandler().list_files(
{"source_id": "abc", "download_url": "https://evil.example/abss/drive/"}
)
assert files is None
fetch.assert_not_called()
def test_extract_magnet_link_and_file_list_share_one_page_fetch():
from unittest.mock import patch
page = MULTIFILE_DETAIL_HTML
with (
patch(
"shelfmark.release_sources.audiobookbay.scraper.downloader.html_get_page",
return_value=page,
) as get_page,
patch("shelfmark.release_sources.audiobookbay.scraper._bootstrap_abb_session"),
):
scraper.clear_detail_page_cache()
url = "https://audiobookbay.lu/abss/expanse/"
assert scraper.fetch_detail_html(url, "audiobookbay.lu") == page
magnet = scraper.extract_magnet_link(url, "audiobookbay.lu")
assert magnet is not None
assert "e4a5538e26987ee58a43aa629ec2c4f2b2d46526".upper() in magnet
assert get_page.call_count == 1
+268
View File
@@ -0,0 +1,268 @@
"""How long the internal bypasser is allowed to spend, and on what.
Issue #1276: MAX_RETRY drove *both* the outer page-load loop and the per-page method
loop, so the default of 10 meant ~40 solve attempts on one browser. That overran the
worker deadline, and the failure reached the user as `RuntimeError: TimeoutError` - a
message that says nothing about a protection challenge and sent people looking at their
reverse proxy instead.
Also covered: the undisturbed window a passive challenge gets before anything touches the
page. Anna's Archive's DDoS-Guard check has no click target and clears itself; going
straight to the click/reload methods meant the one thing that solves it was never tried.
"""
import asyncio
import pytest
@pytest.fixture
def bypass(monkeypatch):
"""internal_bypasser with sleeps and jitter removed."""
import shelfmark.bypass.internal_bypasser as internal_bypasser
async def _no_sleep(_seconds) -> None:
return None
monkeypatch.setattr(internal_bypasser.asyncio, "sleep", _no_sleep)
monkeypatch.setattr(internal_bypasser._RNG, "uniform", lambda _a, _b: 0)
return internal_bypasser
def _recording_methods(calls: list[str], count: int = 4):
def _make(name: str):
async def _method(_page) -> bool:
calls.append(name)
return False
_method.__name__ = name
return _method
return [_make(f"m{i}") for i in range(count)]
def _stub_page_state(monkeypatch, bypass, *, bypassed=False, challenge="ddos_guard"):
async def _is_bypassed(*_args, **_kwargs) -> bool:
return bypassed
async def _detect(*_args, **_kwargs) -> str:
return challenge
monkeypatch.setattr(bypass, "_is_bypassed", _is_bypassed)
monkeypatch.setattr(bypass, "_detect_challenge_type", _detect)
# --------------------------------------------------------------------------- #
# The method loop must not read MAX_RETRY
# --------------------------------------------------------------------------- #
def test_method_loop_budget_is_independent_of_max_retry(monkeypatch, bypass):
"""MAX_RETRY is the outer page-load retry; reading it here squared the budget.
Exercised against a challenge whose *type* keeps changing, because that is the case
where max_retries is what bounds the loop: the stuck-challenge guard only fires on a
run of the same type, so with a stable challenge it hid the real budget entirely.
"""
monkeypatch.setattr(type(bypass.app_config), "MAX_RETRY", 50, raising=False)
types = iter(["ddos_guard", "cloudflare"] * 100)
async def _alternating(*_args, **_kwargs) -> str:
return next(types)
calls: list[str] = []
monkeypatch.setattr(bypass, "BYPASS_METHODS", _recording_methods(calls))
monkeypatch.setattr(bypass, "_wait_for_passive_solve", _never_passes)
_stub_page_state(monkeypatch, bypass)
monkeypatch.setattr(bypass, "_detect_challenge_type", _alternating)
assert asyncio.run(bypass._bypass(object())) is False
assert len(calls) == bypass._BYPASS_METHOD_ATTEMPTS
assert len(calls) < 50, "MAX_RETRY must not reach the method loop"
def test_method_attempt_budget_is_reachable(bypass):
"""The number reported as `attempt N/X` must be a number the loop can reach.
It used to be MAX_RETRY (10) while the stuck-challenge guard capped the loop at 5,
so logs showed `4/10` and stopped, which reads like six lost attempts.
"""
assert bypass._BYPASS_METHOD_ATTEMPTS == len(bypass.BYPASS_METHODS) + 1
assert bypass._BYPASS_METHOD_ATTEMPTS >= (
max(bypass.MAX_CONSECUTIVE_SAME_CHALLENGE, len(bypass.BYPASS_METHODS) + 1)
)
# --------------------------------------------------------------------------- #
# A passive challenge gets an undisturbed window first
# --------------------------------------------------------------------------- #
async def _never_passes(*_args, **_kwargs) -> bool:
return False
def test_passive_challenge_is_given_time_before_any_method_runs(monkeypatch, bypass):
"""DDoS-Guard's JS check clears itself; nothing should click or reload first."""
calls: list[str] = []
monkeypatch.setattr(bypass, "BYPASS_METHODS", _recording_methods(calls))
_stub_page_state(monkeypatch, bypass)
async def _passes(*_args, **_kwargs) -> bool:
return True
monkeypatch.setattr(bypass, "_wait_for_passive_solve", _passes)
assert asyncio.run(bypass._bypass(object())) is True
assert calls == [], "the page must not be touched while the check can still pass"
def test_passive_wait_happens_once_not_before_every_method(monkeypatch, bypass):
"""It is a settling window, not a delay bolted onto each attempt."""
waits: list[int] = []
calls: list[str] = []
async def _count_wait(*_args, **_kwargs) -> bool:
waits.append(1)
return False
monkeypatch.setattr(bypass, "BYPASS_METHODS", _recording_methods(calls))
monkeypatch.setattr(bypass, "_wait_for_passive_solve", _count_wait)
_stub_page_state(monkeypatch, bypass)
asyncio.run(bypass._bypass(object()))
assert len(waits) == 1
assert calls == ["m0", "m1", "m2", "m3"]
def test_no_passive_wait_when_no_challenge_is_detected(monkeypatch, bypass):
"""The 'none' branch has its own settle-and-refresh handling."""
waits: list[int] = []
async def _count_wait(*_args, **_kwargs) -> bool:
waits.append(1)
return False
monkeypatch.setattr(bypass, "_wait_for_passive_solve", _count_wait)
_stub_page_state(monkeypatch, bypass, challenge="none")
class _Page:
async def reload(self, **_kwargs) -> None:
return None
asyncio.run(bypass._bypass(_Page(), max_retries=1))
assert waits == []
def test_wait_for_passive_solve_returns_as_soon_as_the_page_clears(monkeypatch, bypass):
polls = {"n": 0}
async def _is_bypassed(*_args, **_kwargs) -> bool:
polls["n"] += 1
return polls["n"] >= 3
monkeypatch.setattr(bypass, "_is_bypassed", _is_bypassed)
assert asyncio.run(bypass._wait_for_passive_solve(object())) is True
assert polls["n"] == 3
def test_wait_for_passive_solve_gives_up_at_the_window(monkeypatch, bypass):
"""It must not poll forever - the methods still need their share of the budget."""
clock = {"now": 0.0}
monkeypatch.setattr(bypass.time, "monotonic", lambda: clock["now"])
async def _tick(*_args, **_kwargs) -> bool:
clock["now"] += 1.0
return False
monkeypatch.setattr(bypass, "_is_bypassed", _tick)
assert asyncio.run(bypass._wait_for_passive_solve(object())) is False
assert clock["now"] >= bypass._PASSIVE_SOLVE_SECONDS
def test_wait_for_passive_solve_honours_cancellation(monkeypatch, bypass):
import threading
from shelfmark.bypass import BypassCancelledError
cancel = threading.Event()
cancel.set()
monkeypatch.setattr(bypass, "_is_bypassed", _never_passes)
with pytest.raises(BypassCancelledError):
asyncio.run(bypass._wait_for_passive_solve(object(), cancel))
# --------------------------------------------------------------------------- #
# The page-load loop stops while there is still time to report a real failure
# --------------------------------------------------------------------------- #
def test_page_load_loop_stops_before_the_worker_deadline(monkeypatch, bypass):
"""A stubborn challenge must produce "bypass failed", not a cancelled coroutine."""
clock = {"now": 0.0}
monkeypatch.setattr(bypass.time, "monotonic", lambda: clock["now"])
monkeypatch.delenv(bypass._BYPASS_CHILD_ENV, raising=False)
attempts = {"n": 0}
async def _create(_url):
return object()
async def _get(_url, _driver, _cancel=None) -> str:
attempts["n"] += 1
# Each pass eats a realistic slice of the budget.
clock["now"] += 120.0
return ""
async def _close(_driver) -> None:
return None
monkeypatch.setattr(bypass, "_create_cdp_browser", _create)
monkeypatch.setattr(bypass, "_get", _get)
monkeypatch.setattr(bypass, "_close_cdp_driver", _close)
class _RealWorker:
def run(self, coro, timeout=None):
return asyncio.run(coro)
monkeypatch.setattr(bypass, "_CDP_WORKER", _RealWorker())
result = bypass._run_bypass_in_current_process("https://example.com", 10)
assert result == ""
# Well short of the 10 it was asked for, and short of the deadline it had.
assert attempts["n"] < 10
budget = bypass._IN_PROCESS_BYPASS_TIMEOUT_SECONDS
assert clock["now"] < budget, "the loop must leave room to report the failure"
def test_page_load_loop_still_makes_one_attempt_on_a_spent_budget(monkeypatch, bypass):
"""The deadline check must never skip the request entirely."""
clock = {"now": 10_000.0}
monkeypatch.setattr(bypass.time, "monotonic", lambda: clock["now"])
monkeypatch.delenv(bypass._BYPASS_CHILD_ENV, raising=False)
attempts = {"n": 0}
async def _create(_url):
return object()
async def _get(_url, _driver, _cancel=None) -> str:
attempts["n"] += 1
return "<html>solved</html>"
async def _close(_driver) -> None:
return None
monkeypatch.setattr(bypass, "_create_cdp_browser", _create)
monkeypatch.setattr(bypass, "_get", _get)
monkeypatch.setattr(bypass, "_close_cdp_driver", _close)
class _RealWorker:
def run(self, coro, timeout=None):
return asyncio.run(coro)
monkeypatch.setattr(bypass, "_CDP_WORKER", _RealWorker())
assert bypass._run_bypass_in_current_process("https://example.com", 10) == "<html>solved</html>"
assert attempts["n"] == 1
+84
View File
@@ -43,6 +43,37 @@ def _store(cookies, url="https://annas-archive.gl/search"):
cs.store_extracted_cookies(url=url, cookies=cookies, user_agent="UA/1.0")
@pytest.fixture
def cookie_store_logs():
"""Collect cookie-store log messages.
The store's logger is built outside the standard hierarchy, so its records never
reach the root handler caplog installs.
"""
import logging
messages: list[str] = []
class _Capture(logging.Handler):
def emit(self, record: logging.LogRecord) -> None:
messages.append(record.getMessage())
handler = _Capture()
cs.logger.addHandler(handler)
previous = cs.logger.level
cs.logger.setLevel(logging.DEBUG)
# setup_logger builds its loggers with CustomLogger(name) rather than getLogger, so
# they are not in the manager's hierarchy - and Logger.setLevel only invalidates the
# is-enabled cache *through* the manager. Without this the logger keeps answering
# "DEBUG is off" from a cache entry made while it was at INFO.
cs.logger._cache.clear()
try:
yield messages
finally:
cs.logger.removeHandler(handler)
cs.logger.setLevel(previous)
# --------------------------------------------------------------------------- #
# Per-check cookies must not be persisted for replay
# --------------------------------------------------------------------------- #
@@ -220,3 +251,56 @@ def test_failure_only_clears_the_failing_host(monkeypatch):
assert ib.get_cf_cookies_for_domain("annas-archive.gl") == {}
assert ib.get_cf_cookies_for_domain("other-site.test") == {"__ddg1_": "other"}
# --------------------------------------------------------------------------- #
# Settling what DDoS-Guard actually treats as clearance (issue #1276)
# --------------------------------------------------------------------------- #
def test_per_check_cookies_can_be_kept_for_a_field_test(monkeypatch):
"""Which __ddg* cookies are clearance is not settled, so it has to be testable.
The store's premise - that __ddg8_/__ddg9_/__ddg10_ describe one check and must not
be replayed - is contradicted by the field reports on #1276, where every request
after a successful solve was challenged again. This env-only switch is how that gets
answered against a live host without building a branch.
"""
from shelfmark.config import env
monkeypatch.setattr(env, "DDG_REPLAY_PER_CHECK_COOKIES", True)
_store(
[
_Cookie("__ddg1_", "clearance"),
_Cookie("__ddg8_", "opaque"),
_Cookie("__ddg9_", "203.0.113.7"),
_Cookie("__ddg10_", "1786826304"),
]
)
stored = ib.get_cf_cookies_for_domain("annas-archive.gl")
assert set(stored) == {"__ddg1_", "__ddg8_", "__ddg9_", "__ddg10_"}
def test_dropping_per_check_cookies_is_the_default(monkeypatch):
"""The switch is for reproducing the question, not a behaviour change."""
from shelfmark.config import env
assert env.DDG_REPLAY_PER_CHECK_COOKIES is False
_store([_Cookie("__ddg1_", "clearance"), _Cookie("__ddg9_", "203.0.113.7")])
assert set(ib.get_cf_cookies_for_domain("annas-archive.gl")) == {"__ddg1_"}
def test_a_solve_logs_which_cookies_it_won_and_which_were_held_back(cookie_store_logs):
"""Without this, a debug log shows a solve succeed and the next request challenged,
with nothing in between to explain why."""
_store([_Cookie("__ddg1_", "clearance"), _Cookie("__ddg9_", "203.0.113.7")])
messages = cookie_store_logs
line = next((m for m in messages if "won" in m and "dropping" in m), None)
assert line is not None, messages
assert "__ddg1_" in line
assert "__ddg9_" in line
# Names only - a clearance cookie's value is a credential.
assert "clearance" not in line
assert "203.0.113.7" not in line
@@ -0,0 +1,110 @@
"""A recording that never happened must say why.
Issue #1276: the debug bundle's recording/ directory was empty, and the only trace was
three "FFmpeg already stopped" debug lines - one per bypass, each logged 20-56s after
the recorder was started, meaning ffmpeg had exited almost immediately every time. It ran
with `-loglevel 0` and no stderr capture, so nothing anywhere recorded the reason. The
screen recording is the single most useful artifact for diagnosing a bypass failure.
"""
import subprocess
import pytest
import shelfmark.bypass.internal_bypasser as ib
@pytest.fixture(autouse=True)
def _clean_display():
before = dict(ib.DISPLAY)
ib.DISPLAY["ffmpeg"] = None
ib.DISPLAY["ffmpeg_output"] = None
ib.DISPLAY["ffmpeg_error_log"] = None
yield
ib.DISPLAY.update(before)
class _Proc:
def __init__(self, returncode):
self.returncode = returncode
def poll(self):
return self.returncode
def test_ffmpeg_errors_are_captured_to_a_file_beside_the_recording(monkeypatch, tmp_path):
monkeypatch.setattr(ib, "RECORDING_DIR", tmp_path)
captured: dict[str, object] = {}
def fake_popen(cmd, **kwargs):
captured["cmd"] = cmd
captured["stderr"] = kwargs.get("stderr")
return _Proc(None)
monkeypatch.setattr(ib.subprocess, "Popen", fake_popen)
ib._start_ffmpeg_recording(display=":99")
cmd = captured["cmd"]
# Errors must not be thrown away any more.
assert "-loglevel" in cmd
assert cmd[cmd.index("-loglevel") + 1] == "error"
# stderr goes to a real file, not a pipe nothing would drain.
assert captured["stderr"] is not None
assert captured["stderr"] is not subprocess.PIPE
error_log = ib.DISPLAY["ffmpeg_error_log"]
assert error_log is not None
assert error_log.parent == tmp_path
# It sits beside the mp4, so it travels in the debug bundle.
assert error_log.name.startswith("screen_recording_")
def test_an_early_exit_is_reported_with_ffmpegs_own_reason(monkeypatch, tmp_path, caplog):
reason = "[x11grab @ 0x1] Cannot open display :99, error 1."
error_log = tmp_path / "screen_recording_x.ffmpeg.log"
error_log.write_text(reason, encoding="utf-8")
ib.DISPLAY["ffmpeg"] = _Proc(1)
ib.DISPLAY["ffmpeg_output"] = tmp_path / "screen_recording_x.mp4"
ib.DISPLAY["ffmpeg_error_log"] = error_log
messages: list[str] = []
class _Capture:
def emit(self, record):
messages.append(record.getMessage())
import logging
handler = logging.Handler()
handler.emit = _Capture().emit # type: ignore[method-assign]
ib.logger.addHandler(handler)
previous = ib.logger.level
ib.logger.setLevel(logging.DEBUG)
ib.logger._cache.clear()
try:
ib._stop_ffmpeg_recording()
finally:
ib.logger.removeHandler(handler)
ib.logger.setLevel(previous)
line = next((m for m in messages if "exited early" in m), None)
assert line is not None, messages
assert "code 1" in line
assert "Cannot open display" in line
assert ib.DISPLAY["ffmpeg"] is None
def test_summary_is_explicit_when_ffmpeg_logged_nothing(tmp_path):
empty = tmp_path / "screen_recording_y.ffmpeg.log"
empty.write_text("", encoding="utf-8")
ib.DISPLAY["ffmpeg_error_log"] = empty
assert "logged nothing" in ib._ffmpeg_error_summary()
def test_summary_survives_a_missing_log():
ib.DISPLAY["ffmpeg_error_log"] = None
assert "No FFmpeg error log" in ib._ffmpeg_error_summary()
+113
View File
@@ -0,0 +1,113 @@
"""A 429 is throttling, not a dead clearance cookie.
Issue #1276: every rejection of the cached cookies took the same exit, which cleared the
host's clearance. That is right for a 403 and for the ?check=1 redirect loop - being
challenged while presenting a cookie proves the cookie is dead - and wrong for a 429,
where the origin is rate-limiting the IP and would answer a real browser holding the very
same cookies identically.
The cost in the reported bundle: a solve completed at 13:41:23 and stored five cookies;
six seconds later a 429 threw them away, and the next query bought its own 56-second
browser solve. Reuse rate across the whole log was 0 of 2.
"""
import pytest
import shelfmark.bypass.cookie_store as cs
import shelfmark.bypass.internal_bypasser as ib
URL = "https://annas-archive.gl/search?q=dune"
HOST = "annas-archive.gl"
class _Resp:
def __init__(self, status_code, text="page"):
self.status_code = status_code
self.text = text
@pytest.fixture(autouse=True)
def _clean(monkeypatch):
monkeypatch.setattr(cs, "_cf_cookies", {})
monkeypatch.setattr(cs, "_cf_user_agents", {})
monkeypatch.setattr(cs, "_get_full_cookie_domains", set)
monkeypatch.setattr(ib, "get_proxies", lambda _url: None)
monkeypatch.setattr(ib, "get_ssl_verify", lambda _url: True)
cs._cf_cookies[HOST] = {
"__ddg1_": {"value": "clearance", "expiry": None},
"__ddg2_": {"value": "c2", "expiry": None},
}
def _cooldowns(monkeypatch):
"""Record note_rate_limited calls without arming the real per-host ladder."""
armed: list[str] = []
monkeypatch.setattr(ib.network, "note_rate_limited", lambda url: armed.append(url) or 120.0)
return armed
def test_429_keeps_the_clearance(monkeypatch):
armed = _cooldowns(monkeypatch)
monkeypatch.setattr(ib.requests, "get", lambda *a, **k: _Resp(429))
assert ib._try_with_cached_cookies(URL, HOST) is None
assert ib.get_cf_cookies_for_domain(HOST) == {"__ddg1_": "clearance", "__ddg2_": "c2"}
assert armed == [URL], "the backoff must still be armed"
def test_403_still_discards_the_clearance(monkeypatch):
"""The pre-existing behaviour for a genuine rejection must not regress."""
_cooldowns(monkeypatch)
monkeypatch.setattr(ib.requests, "get", lambda *a, **k: _Resp(403))
assert ib._try_with_cached_cookies(URL, HOST) is None
assert ib.get_cf_cookies_for_domain(HOST) == {}
def test_redirect_loop_still_discards_the_clearance(monkeypatch):
_cooldowns(monkeypatch)
def boom(*_a, **_k):
raise ib.requests.exceptions.TooManyRedirects("Exceeded 30 redirects")
monkeypatch.setattr(ib.requests, "get", boom)
assert ib._try_with_cached_cookies(URL, HOST) is None
assert ib.get_cf_cookies_for_domain(HOST) == {}
def test_a_throttled_host_is_not_handed_a_browser_solve(monkeypatch):
"""A solve cannot clear a throttle, and is itself more traffic at a host asking for
less. get_bypassed_page checks the cooldown before the queue; get() has to re-check
after it, because a request can hold for LOCKED while another collects the 429."""
monkeypatch.setattr(ib.requests, "get", lambda *a, **k: _Resp(429))
monkeypatch.setattr(ib.network, "note_rate_limited", lambda _url: 120.0)
monkeypatch.setattr(ib.network, "host_cooldown_remaining", lambda _url: 118.0)
solved: list[str] = []
monkeypatch.setattr(
ib, "_run_bypass_in_current_process", lambda url, *a, **k: solved.append(url) or "html"
)
monkeypatch.setattr(ib.env, "DOCKERMODE", False)
with pytest.raises(ib.network.RateLimitedError) as excinfo:
ib.get(URL, retry=1)
assert solved == [], "no browser should have been started"
assert "rate-limited" in str(excinfo.value)
# And the clearance survives, ready for when the cooldown clears.
assert ib.get_cf_cookies_for_domain(HOST) == {"__ddg1_": "clearance", "__ddg2_": "c2"}
def test_a_host_that_is_not_throttled_still_solves(monkeypatch):
monkeypatch.setattr(ib.requests, "get", lambda *a, **k: _Resp(403))
monkeypatch.setattr(ib.network, "host_cooldown_remaining", lambda _url: 0.0)
solved: list[str] = []
monkeypatch.setattr(
ib, "_run_bypass_in_current_process", lambda url, *a, **k: solved.append(url) or "html"
)
monkeypatch.setattr(ib.env, "DOCKERMODE", False)
assert ib.get(URL, retry=1) == "html"
assert solved == [URL]
+26
View File
@@ -66,6 +66,32 @@ def test_download_settings_booklore_destination_field_defaults_to_library():
assert option_values == {"library", "bookdrop"}
def test_download_settings_audiobook_grouping_is_opt_in():
from shelfmark.config.settings import download_settings
fields = download_settings()
organization_field = next(
field for field in fields if getattr(field, "key", None) == "FILE_ORGANIZATION_AUDIOBOOK"
)
rename_template = next(
field
for field in fields
if getattr(field, "key", None) == "template_audiobook_rename_editor"
)
assert organization_field.default == "rename"
assert [option["value"] for option in organization_field.options] == [
"none",
"rename",
"organize",
"rename_and_group",
]
assert rename_template.show_when == {
"field": "FILE_ORGANIZATION_AUDIOBOOK",
"value": ["rename", "rename_and_group"],
}
def test_download_settings_grimmory_copy_is_exposed_in_ui_metadata():
from shelfmark.config.settings import download_settings
+3 -2
View File
@@ -385,14 +385,15 @@ class TestSettingsValidation:
assert "Naming Template" in result["message"]
assert "Organize" in result["message"]
def test_downloads_audiobooks_rename_template_rejects_path_separators(self):
@pytest.mark.parametrize("organization_mode", ["rename", "rename_and_group"])
def test_downloads_audiobooks_rename_template_rejects_path_separators(self, organization_mode):
import shelfmark.config.settings # noqa: F401
from shelfmark.core.settings_registry import update_settings
result = update_settings(
"downloads",
{
"FILE_ORGANIZATION_AUDIOBOOK": "rename",
"FILE_ORGANIZATION_AUDIOBOOK": organization_mode,
"TEMPLATE_AUDIOBOOK_RENAME": "{Author}/{Title}",
},
)
+11
View File
@@ -0,0 +1,11 @@
"""Tests for general settings definitions."""
from shelfmark.config.settings import general_settings
def test_supported_formats_stay_admin_only():
"""The format lists describe the library, not a reader, so they are not overridable."""
fields = {field.key: field for field in general_settings() if hasattr(field, "key")}
assert fields["SUPPORTED_FORMATS"].user_overridable is False
assert fields["SUPPORTED_AUDIOBOOK_FORMATS"].user_overridable is False
+63
View File
@@ -1,8 +1,17 @@
"""Tests for search mode settings definitions."""
import json
import pytest
from shelfmark.config.settings import search_mode_settings
def _search_mode_field(key: str):
fields = {field.key: field for field in search_mode_settings() if hasattr(field, "key")}
return fields[key]
def test_search_mode_settings_include_release_source_links_toggle():
fields = {field.key: field for field in search_mode_settings() if hasattr(field, "key")}
@@ -11,3 +20,57 @@ def test_search_mode_settings_include_release_source_links_toggle():
assert field.label == "Show Release Source Links"
assert field.default is True
assert field.user_overridable is False
def test_book_language_is_user_overridable():
fields = {field.key: field for field in search_mode_settings() if hasattr(field, "key")}
field = fields["BOOK_LANGUAGE"]
assert field.label == "Default Book Languages"
assert field.default == ["en"]
assert field.user_overridable is True
def test_book_language_stored_under_the_general_tab_still_resolves(tmp_path):
"""BOOK_LANGUAGE moved from the General tab to Search Mode with no migration.
That is only safe because both tabs persist into the same settings.json, so an
install that stored the value while the field lived on General keeps it. If the
two tabs ever get separate files, every existing install silently falls back to
the ["en"] default instead.
"""
from shelfmark.core.settings_registry import _get_config_file_path, get_setting_value
(tmp_path / "settings.json").write_text(json.dumps({"BOOK_LANGUAGE": ["de", "fr"]}))
with pytest.MonkeyPatch.context() as monkeypatch:
monkeypatch.delenv("BOOK_LANGUAGE", raising=False)
monkeypatch.setattr("shelfmark.config.env.CONFIG_DIR", tmp_path)
assert _get_config_file_path("search_mode") == _get_config_file_path("general")
assert get_setting_value(_search_mode_field("BOOK_LANGUAGE"), "search_mode") == ["de", "fr"]
def test_book_language_uses_its_default_on_a_fresh_install(tmp_path):
from shelfmark.core.settings_registry import get_setting_value
with pytest.MonkeyPatch.context() as monkeypatch:
monkeypatch.delenv("BOOK_LANGUAGE", raising=False)
monkeypatch.setattr("shelfmark.config.env.CONFIG_DIR", tmp_path)
assert get_setting_value(_search_mode_field("BOOK_LANGUAGE"), "search_mode") == ["en"]
def test_book_language_env_var_beats_the_stored_value(tmp_path):
from shelfmark.core.settings_registry import get_setting_value, is_value_from_env
(tmp_path / "settings.json").write_text(json.dumps({"BOOK_LANGUAGE": ["de", "fr"]}))
field = _search_mode_field("BOOK_LANGUAGE")
with pytest.MonkeyPatch.context() as monkeypatch:
monkeypatch.setenv("BOOK_LANGUAGE", "es,it")
monkeypatch.setattr("shelfmark.config.env.CONFIG_DIR", tmp_path)
assert is_value_from_env(field) is True
assert get_setting_value(field, "search_mode") == ["es", "it"]
+1 -1
View File
@@ -90,7 +90,7 @@ def test_visible_self_settings_sections_field_defaults_and_options():
{
"value": "search",
"label": "Search Preferences",
"description": "Show personal search mode and provider settings.",
"description": "Show personal search mode, language, and provider settings.",
},
{
"value": "notifications",
+82
View File
@@ -508,6 +508,72 @@ class TestAdminUserUpdateEndpoint:
settings = user_db.get_user_settings(user["id"])
assert settings["DESTINATION_AUDIOBOOK"] == "/audiobooks/alice"
def test_update_user_settings_normalizes_book_languages(self, admin_client, user_db):
user = user_db.create_user(username="alice")
resp = admin_client.put(
f"/api/admin/users/{user['id']}",
json={"settings": {"BOOK_LANGUAGE": ["German", "de", " en "]}},
)
assert resp.status_code == 200
settings = user_db.get_user_settings(user["id"])
assert settings["BOOK_LANGUAGE"] == ["de", "en"]
def test_update_user_settings_rejects_unknown_book_language(self, admin_client, user_db):
user = user_db.create_user(username="alice")
resp = admin_client.put(
f"/api/admin/users/{user['id']}",
json={"settings": {"BOOK_LANGUAGE": ["de", "klingon"]}},
)
assert resp.status_code == 400
assert resp.json["error"] == "Invalid settings payload"
assert any(
"BOOK_LANGUAGE contains an unsupported language: klingon" in msg
for msg in resp.json["details"]
)
assert user_db.get_user_settings(user["id"]) == {}
def test_update_user_settings_skips_blank_book_languages(self, admin_client, user_db):
"""A trailing comma or a blank slot means "nothing there", not an unknown language."""
user = user_db.create_user(username="alice")
resp = admin_client.put(
f"/api/admin/users/{user['id']}",
json={"settings": {"BOOK_LANGUAGE": "en,"}},
)
assert resp.status_code == 200
assert user_db.get_user_settings(user["id"])["BOOK_LANGUAGE"] == ["en"]
def test_update_user_settings_normalizes_every_validated_search_key(
self, admin_client, user_db
):
"""Keys the search validator recognises keep their normalized value.
These three were validated but then stored raw, because the write-back was
gated on a hand-maintained subset of the validated keys. A padded provider
name was accepted and then persisted with its padding, so every later lookup
of it failed.
"""
user = user_db.create_user(username="alice")
resp = admin_client.put(
f"/api/admin/users/{user['id']}",
json={
"settings": {
"METADATA_PROVIDER_COMBINED": " openlibrary ",
"SHOW_COMBINED_SELECTOR": "yes",
"FORCE_COMBINED_SEARCH": "",
}
},
)
assert resp.status_code == 200
settings = user_db.get_user_settings(user["id"])
assert settings["METADATA_PROVIDER_COMBINED"] == "openlibrary"
assert settings["SHOW_COMBINED_SELECTOR"] is True
assert settings["FORCE_COMBINED_SEARCH"] is False
def test_update_user_settings_accepts_notification_overrides(self, admin_client, user_db):
user = user_db.create_user(username="alice")
@@ -1266,6 +1332,7 @@ class TestAdminSearchPreferences:
assert data["tab"] == "search_mode"
assert data["keys"] == [
"SEARCH_MODE",
"BOOK_LANGUAGE",
"SHOW_COMBINED_SELECTOR",
"FORCE_COMBINED_SEARCH",
"METADATA_PROVIDER",
@@ -1295,6 +1362,21 @@ class TestAdminSearchPreferences:
assert data["effective"]["DEFAULT_RELEASE_SOURCE_AUDIOBOOK"]["source"] == "user_override"
assert data["effective"]["DEFAULT_RELEASE_SOURCE_AUDIOBOOK"]["value"] == "audiobookbay"
def test_reports_book_language_override(self, admin_client, user_db):
user = user_db.create_user(username="alice")
user_db.set_user_settings(user["id"], {"BOOK_LANGUAGE": ["de", "en"]})
resp = admin_client.get(f"/api/admin/users/{user['id']}/search-preferences")
assert resp.status_code == 200
data = resp.json
assert data["userOverrides"]["BOOK_LANGUAGE"] == ["de", "en"]
assert data["effective"]["BOOK_LANGUAGE"] == {
"value": ["de", "en"],
"source": "user_override",
}
assert data["globalValues"]["BOOK_LANGUAGE"] == ["en"]
def test_returns_404_for_unknown_user(self, admin_client):
resp = admin_client.get("/api/admin/users/9999/search-preferences")
assert resp.status_code == 404
+3
View File
@@ -48,6 +48,7 @@ def test_config_endpoint_uses_user_scope_and_runtime_flags(main_module, client):
"DEFAULT_RELEASE_SOURCE": "prowlarr",
"DEFAULT_RELEASE_SOURCE_AUDIOBOOK": "audiobookbay",
"DOWNLOAD_TO_BROWSER_CONTENT_TYPES": ["book", "audiobook"],
"BOOK_LANGUAGE": ["de", "en"],
"AUTO_OPEN_DOWNLOADS_SIDEBAR": False,
"HARDCOVER_AUTO_REMOVE_ON_DOWNLOAD": True,
"AA_DEFAULT_SORT": "newest",
@@ -75,12 +76,14 @@ def test_config_endpoint_uses_user_scope_and_runtime_flags(main_module, client):
assert data["default_release_source"] == "prowlarr"
assert data["default_release_source_audiobook"] == "audiobookbay"
assert data["download_to_browser_content_types"] == ["book", "audiobook"]
assert data["default_language"] == ["de", "en"]
assert data["settings_enabled"] is True
assert data["metadata_default_sort"] == "relevance"
assert ("SHOW_RELEASE_SOURCE_LINKS", None) in calls
assert ("SHOW_COMBINED_SELECTOR", 42) in calls
assert ("DOWNLOAD_TO_BROWSER_CONTENT_TYPES", 42) in calls
assert ("BOOK_LANGUAGE", 42) in calls
def test_config_endpoint_falls_back_to_audiobook_metadata_provider(main_module, client):
+22
View File
@@ -143,3 +143,25 @@ def test_build_user_preferences_payload_reports_effective_sources(monkeypatch):
fields_by_key = {field["key"]: field for field in payload["fields"]}
assert fields_by_key["DESTINATION"]["fromEnv"] is False
assert fields_by_key["BOOKS_OUTPUT_MODE"]["fromEnv"] is True
def test_build_user_preferences_payload_carries_the_language_default():
"""BOOK_LANGUAGE rides along with the other search preferences on its tab."""
import shelfmark.config.settings # noqa: F401
from shelfmark.core.user_settings_overrides import build_user_preferences_payload
user_db = SimpleNamespace(get_user_settings=lambda user_id: {"BOOK_LANGUAGE": ["de", "en"]})
payload = build_user_preferences_payload(user_db, 7, "search_mode")
assert payload["tab"] == "search_mode"
assert {"SEARCH_MODE", "BOOK_LANGUAGE"}.issubset(payload["keys"])
assert payload["userOverrides"] == {"BOOK_LANGUAGE": ["de", "en"]}
assert payload["effective"]["BOOK_LANGUAGE"] == {
"value": ["de", "en"],
"source": "user_override",
}
fields_by_key = {field["key"]: field for field in payload["fields"]}
assert fields_by_key["BOOK_LANGUAGE"]["type"] == "MultiSelectField"
assert fields_by_key["BOOK_LANGUAGE"]["options"]
@@ -66,6 +66,20 @@ def test_get_file_organization_uses_current_keys(monkeypatch):
assert policy.get_file_organization(is_audiobook=True) == "none"
def test_get_file_organization_accepts_audiobook_grouping(monkeypatch):
import shelfmark.download.postprocess.policy as policy
monkeypatch.setattr(
policy.core_config.config,
"get",
lambda key, default=None: {
"FILE_ORGANIZATION_AUDIOBOOK": "rename_and_group",
}.get(key, default),
)
assert policy.get_file_organization(is_audiobook=True) == "rename_and_group"
def test_get_file_organization_ignores_pre_release_processing_mode_keys(monkeypatch):
import shelfmark.download.postprocess.policy as policy
+168 -3
View File
@@ -14,6 +14,7 @@ Two approaches to preserve torrent files for seeding:
import os
import shutil
import tempfile
import zipfile
from pathlib import Path
from threading import Event
from unittest.mock import MagicMock, patch
@@ -1003,8 +1004,13 @@ class TestTorrentSourceCleanupProtection:
Each test simulates a specific real-world content type and file structure.
"""
def _make_config_mock(self, library_path: str, hardlink: bool = True):
"""Create config mock for library/organize mode with hardlinking."""
def _make_config_mock(
self,
library_path: str,
hardlink: bool = True,
organization_mode: str = "organize",
):
"""Create a folder-output config mock for torrent transfers."""
return MagicMock(
side_effect=lambda key, default=None, **_kwargs: {
# Destination paths (what _get_final_destination uses)
@@ -1013,9 +1019,10 @@ class TestTorrentSourceCleanupProtection:
# Templates (what _get_template uses)
"TEMPLATE_ORGANIZE": "{Author}/{Title}",
"TEMPLATE_AUDIOBOOK_ORGANIZE": "{Author}/{Title}{ - PartNumber}",
"TEMPLATE_AUDIOBOOK_RENAME": "{Author} - {Title}",
# File organization mode
"FILE_ORGANIZATION": "organize",
"FILE_ORGANIZATION_AUDIOBOOK": "organize",
"FILE_ORGANIZATION_AUDIOBOOK": organization_mode,
# Hardlink toggle
"HARDLINK_TORRENTS": hardlink,
"HARDLINK_TORRENTS_AUDIOBOOK": hardlink,
@@ -1122,6 +1129,163 @@ class TestTorrentSourceCleanupProtection:
# ==================== AUDIOBOOK TESTS ====================
@pytest.mark.parametrize(
("organization_mode", "grouped"),
[("rename", False), ("rename_and_group", True)],
)
@pytest.mark.parametrize("hardlink", [True, False])
def test_torrent_audiobook_multifile_grouping_is_opt_in(
self, tmp_path, organization_mode, grouped, hardlink
):
"""Multi-file audiobooks retain their torrent folder only when opted in."""
from shelfmark.core.models import DownloadTask, SearchMode
from shelfmark.download.postprocess.router import (
post_process_download as _post_process_download,
)
torrent_dir = tmp_path / "downloads" / "Project: Hail Mary Audiobook"
torrent_dir.mkdir(parents=True)
source_files = [torrent_dir / "Part 01.mp3", torrent_dir / "Part 02.mp3"]
for index, source_file in enumerate(source_files, start=1):
source_file.write_bytes(f"audio {index}".encode())
library = tmp_path / "library"
library.mkdir()
task = DownloadTask(
task_id=f"audiobook_{organization_mode}_{hardlink}",
source="prowlarr",
title="Project Hail Mary",
author="Andy Weir",
format="mp3",
content_type="audiobook",
search_mode=SearchMode.UNIVERSAL,
original_download_path=str(torrent_dir),
)
with patch("shelfmark.core.config.config") as mock_orch:
mock_orch.get = self._make_config_mock(
str(library),
hardlink=hardlink,
organization_mode=organization_mode,
)
mock_orch.CUSTOM_SCRIPT = None
result = _post_process_download(torrent_dir, task, Event(), MagicMock())
transfer_dir = library / "Project_ Hail Mary Audiobook" if grouped else library
assert result is not None
assert Path(result).parent == transfer_dir
assert sorted(path.name for path in transfer_dir.glob("*.mp3")) == [
"Part 01.mp3",
"Part 02.mp3",
]
assert bool(list(library.glob("*.mp3"))) is not grouped
for source_file in source_files:
destination_file = transfer_dir / source_file.name
assert source_file.exists()
assert destination_file.exists()
if hardlink:
assert source_file.stat().st_ino == destination_file.stat().st_ino
else:
assert source_file.stat().st_ino != destination_file.stat().st_ino
@pytest.mark.parametrize("organization_mode", ["rename", "none", "rename_and_group"])
def test_torrent_audiobook_single_file_stays_in_destination_root(
self, tmp_path, organization_mode
):
"""Single-file audiobooks do not gain a source-folder wrapper."""
from shelfmark.core.models import DownloadTask, SearchMode
from shelfmark.download.postprocess.router import (
post_process_download as _post_process_download,
)
torrent_file = tmp_path / "downloads" / "Project Hail Mary.mp3"
torrent_file.parent.mkdir()
torrent_file.write_bytes(b"audio")
library = tmp_path / "library"
library.mkdir()
task = DownloadTask(
task_id=f"single_audiobook_{organization_mode}",
source="prowlarr",
title="Project Hail Mary",
author="Andy Weir",
format="mp3",
content_type="audiobook",
search_mode=SearchMode.UNIVERSAL,
original_download_path=str(torrent_file),
)
with patch("shelfmark.core.config.config") as mock_orch:
mock_orch.get = self._make_config_mock(
str(library), organization_mode=organization_mode
)
mock_orch.CUSTOM_SCRIPT = None
result = _post_process_download(torrent_file, task, Event(), MagicMock())
expected_name = (
"Andy Weir - Project Hail Mary.mp3"
if organization_mode in {"rename", "rename_and_group"}
else torrent_file.name
)
assert result is not None
assert Path(result) == library / expected_name
assert torrent_file.exists()
def test_torrent_audiobook_archive_groups_under_the_release_name(self, tmp_path):
"""Torrent: a single archive of chapters groups under the release name.
Simulates: a torrent whose only file is "Project Hail Mary.zip" holding
two chapters. Extraction only runs with hardlinking off, and the archive
itself must stay put for seeding.
"""
from shelfmark.core.models import DownloadTask, SearchMode
from shelfmark.download.postprocess.router import (
post_process_download as _post_process_download,
)
torrent_file = tmp_path / "downloads" / "Project Hail Mary.zip"
torrent_file.parent.mkdir(parents=True)
with zipfile.ZipFile(torrent_file, "w") as zf:
zf.writestr("Part 01.mp3", "audio 1")
zf.writestr("Part 02.mp3", "audio 2")
library = tmp_path / "library"
library.mkdir()
task = DownloadTask(
task_id="torrent_archive_audiobook",
source="prowlarr",
title="Project Hail Mary",
author="Andy Weir",
format="mp3",
content_type="audiobook",
search_mode=SearchMode.UNIVERSAL,
original_download_path=str(torrent_file),
)
with (
patch("shelfmark.core.config.config") as mock_orch,
patch("shelfmark.config.env.TMP_DIR", tmp_path / "staging"),
):
mock_orch.get = self._make_config_mock(
str(library),
hardlink=False,
organization_mode="rename_and_group",
)
mock_orch.CUSTOM_SCRIPT = None
result = _post_process_download(torrent_file, task, Event(), MagicMock())
grouped_dir = library / "Project Hail Mary"
assert result is not None
assert Path(result).parent == grouped_dir
assert sorted(path.name for path in grouped_dir.glob("*.mp3")) == [
"Part 01.mp3",
"Part 02.mp3",
]
assert not list(library.glob("*.mp3"))
# The archive stays behind for seeding, and never names the folder.
assert torrent_file.exists()
assert not (library / "Project Hail Mary.zip").exists()
def test_torrent_audiobook_multifile_hardlink(self, tmp_path):
"""Torrent: Multi-file audiobook - all source files preserved for seeding.
@@ -1184,6 +1348,7 @@ class TestTorrentSourceCleanupProtection:
# Verify library has all 12 files
library_files = list((library / "Andy Weir").glob("*.mp3"))
assert len(library_files) == 12
assert not (library / torrent_dir.name).exists()
# ==================== COMIC/CBZ TESTS ====================
+7 -1
View File
@@ -6,7 +6,13 @@ class TestReleaseSearchPlanManualQuery:
def test_manual_query_overrides_plan(self, monkeypatch):
import shelfmark.core.search_plan as sp
monkeypatch.setattr(sp.config, "BOOK_LANGUAGE", ["en", "hu"], raising=False)
monkeypatch.setattr(
sp.config,
"get",
lambda key, default=None, user_id=None: (
["en", "hu"] if key == "BOOK_LANGUAGE" else default
),
)
book = BookMetadata(
provider="hardcover",
+132
View File
@@ -748,6 +748,94 @@ def test_archive_extraction_organize_multifile_assigns_part_numbers(tmp_path):
assert files[1].name == "Archive Audio - 02.mp3"
@pytest.mark.parametrize(
("organization_mode", "grouped"),
[("rename_and_group", True), ("rename", False)],
)
def test_archive_extraction_groups_chapters_under_the_archive_stem(
tmp_path, organization_mode, grouped
):
"""An extracted multi-chapter archive groups into `Book/`, never `Book.zip/`."""
from shelfmark.download.postprocess.router import (
post_process_download as _post_process_download,
)
staging = tmp_path / "staging"
ingest = tmp_path / "ingest"
staging.mkdir()
ingest.mkdir()
archive_path = staging / "Book.zip"
with zipfile.ZipFile(archive_path, "w") as zf:
zf.writestr("Part 1.mp3", "audio1")
zf.writestr("Part 2.mp3", "audio2")
task = DownloadTask(
task_id=f"direct-archive-audio-{organization_mode}",
source="direct_download",
title="Archive Audio",
author="Tester",
format="mp3",
content_type="audiobook",
search_mode=SearchMode.DIRECT,
)
with (
patch("shelfmark.core.config.config") as mock_config,
patch("shelfmark.config.env.TMP_DIR", staging),
):
mock_config.get = _build_config(ingest, organization=organization_mode)
mock_config.CUSTOM_SCRIPT = None
_sync_config(mock_config, mock_config)
result = _post_process_download(archive_path, task, Event(), lambda *_args: None)
transfer_dir = ingest / "Book" if grouped else ingest
assert result is not None
assert Path(result).parent == transfer_dir
assert sorted(path.name for path in transfer_dir.glob("*.mp3")) == ["Part 1.mp3", "Part 2.mp3"]
assert bool(list(ingest.glob("*.mp3"))) is not grouped
# The suffix is packaging, not part of the release name.
assert not (ingest / "Book.zip").exists()
def test_usenet_audiobook_multifile_preserves_source_folder(tmp_path):
from shelfmark.download.postprocess.router import (
post_process_download as _post_process_download,
)
source_dir = tmp_path / "downloads" / "Usenet Audiobook"
source_dir.mkdir(parents=True)
for part in (1, 2):
(source_dir / f"Part {part}.mp3").write_text(f"audio{part}")
ingest = tmp_path / "ingest"
ingest.mkdir()
task = DownloadTask(
task_id="usenet-audio-grouped",
source="prowlarr",
title="Usenet Audio",
author="Tester",
format="mp3",
content_type="audiobook",
search_mode=SearchMode.UNIVERSAL,
)
with patch("shelfmark.core.config.config") as mock_config:
mock_config.get = _build_config(ingest, organization="rename_and_group")
mock_config.CUSTOM_SCRIPT = None
result = _post_process_download(source_dir, task, Event(), lambda *_args: None)
grouped_dir = ingest / "Usenet Audiobook"
assert result is not None
assert Path(result).parent == grouped_dir
assert sorted(path.name for path in grouped_dir.glob("*.mp3")) == [
"Part 1.mp3",
"Part 2.mp3",
]
def test_archive_extraction_organize_multifile_can_use_original_name(tmp_path):
from shelfmark.download.postprocess.router import (
post_process_download as _post_process_download,
@@ -1370,3 +1458,47 @@ def test_external_directory_prefers_files_over_archives_and_keeps_source(
# TMP staging should be cleaned.
assert list(staging.iterdir()) == []
def test_audiobook_multifile_mp4_chapters_are_book_files(tmp_path):
"""Per-chapter .mp4 audiobooks (as MyAnonamouse ships AAC releases) must be
recognised as book files instead of failing with "No book files found"."""
from shelfmark.download.postprocess.router import (
post_process_download as _post_process_download,
)
source_dir = tmp_path / "downloads" / "Andy Weir (2020) The Martian"
source_dir.mkdir(parents=True)
for part in (1, 2):
(source_dir / f"{part:04d} Andy Weir (2020) The Martian.mp4").write_text(f"audio{part}")
(source_dir / "cover.jpg").write_text("jpg")
ingest = tmp_path / "ingest"
ingest.mkdir()
task = DownloadTask(
task_id="mp4-audio-grouped",
source="prowlarr",
title="The Martian",
author="Andy Weir",
format="mp4",
content_type="audiobook",
search_mode=SearchMode.UNIVERSAL,
)
with patch("shelfmark.core.config.config") as mock_config:
mock_config.get = _build_config(
ingest,
organization="rename_and_group",
supported_audiobook_formats=["mp4"],
)
mock_config.CUSTOM_SCRIPT = None
result = _post_process_download(source_dir, task, Event(), lambda *_args: None)
grouped_dir = ingest / "Andy Weir (2020) The Martian"
assert result is not None
assert Path(result).parent == grouped_dir
assert sorted(path.name for path in grouped_dir.glob("*.mp4")) == [
"0001 Andy Weir (2020) The Martian.mp4",
"0002 Andy Weir (2020) The Martian.mp4",
]
+212
View File
@@ -0,0 +1,212 @@
"""Multi-book packs are filed one book at a time through the normal pipeline."""
import os
from pathlib import Path
from threading import Event
from unittest.mock import patch
from shelfmark.core.models import DownloadTask, SearchMode
from tests.core.test_processing_integration import _build_config, _sync_config
def _run(temp_path: Path, task: DownloadTask, ingest: Path, staging: Path, **config_kwargs):
from shelfmark.download.postprocess.router import post_process_download
statuses: list[tuple[str, str | None]] = []
with (
patch("shelfmark.core.config.config") as mock_config,
patch("shelfmark.config.env.TMP_DIR", staging),
):
mock_config.get = _build_config(
ingest,
organization=config_kwargs.pop("organization", "organize"),
supported_audiobook_formats=["m4b", "mp3"],
audiobook_organize_template=config_kwargs.pop(
"audiobook_organize_template", "{Author}/{Title}/{Title}{ - PartNumber}"
),
**config_kwargs,
)
mock_config.CUSTOM_SCRIPT = None
_sync_config(mock_config, mock_config)
result = post_process_download(
temp_path, task, Event(), lambda s, m=None: statuses.append((s, m))
)
return result, statuses
def _nested_pack(root: Path) -> Path:
pack = root / "Sun Eater"
for folder, name in (
("Book 1 - Empire of Silence", "empire.m4b"),
("Book 2 - Howling Dark", "howling.m4b"),
):
(pack / folder).mkdir(parents=True)
(pack / folder / name).write_text(name)
(pack / "cover.jpg").write_text("img")
return pack
def _audiobook_task(**overrides) -> DownloadTask:
fields = {
"task_id": "pack-1",
"source": "direct_download",
"title": "Drive",
"author": "James S. A. Corey",
"content_type": "audiobook",
"series_name": "The Expanse",
"series_position": 2.6,
"search_mode": SearchMode.UNIVERSAL,
}
fields.update(overrides)
return DownloadTask(**fields)
def test_approved_plan_files_each_book_with_its_own_title(tmp_path):
staging = tmp_path / "staging"
ingest = tmp_path / "ingest"
staging.mkdir()
ingest.mkdir()
pack = staging / "Expanse"
pack.mkdir()
for name in (
"The Expanse 1.0 - Leviathan Wakes (2011).m4b",
"The Expanse 2.0 - Caliban's War (2012).m4b",
):
(pack / name).write_text(name)
(pack / "The Expanse 1.0 - Leviathan Wakes (2011).txt").write_text("notes")
task = _audiobook_task(
title="Sun Eater", # the searched book; must not name the pack's books
book_plan=[
{
"title": "Leviathan Wakes (edited)",
"series_position": 1.0,
"year": 2011,
"files": ["The Expanse 1.0 - Leviathan Wakes (2011).m4b"],
},
{
"title": "Caliban's War",
"series_position": 2.0,
"year": 2012,
"files": ["The Expanse 2.0 - Caliban's War (2012).m4b"],
},
],
)
result, statuses = _run(pack, task, ingest, staging)
assert result is not None
author_dir = ingest / "James S. A. Corey"
assert sorted(p.name for p in author_dir.iterdir()) == [
"Caliban's War",
"Leviathan Wakes (edited)",
]
assert (author_dir / "Leviathan Wakes (edited)" / "Leviathan Wakes (edited).m4b").exists()
assert (author_dir / "Caliban's War" / "Caliban's War.m4b").exists()
assert statuses[-1] == ("complete", "Complete (2 books, 2 files)")
def test_multi_book_flag_splits_nested_pack_heuristically(tmp_path):
staging = tmp_path / "staging"
ingest = tmp_path / "ingest"
staging.mkdir()
ingest.mkdir()
pack = _nested_pack(staging)
result, _ = _run(pack, _audiobook_task(multi_book=True), ingest, staging)
assert result is not None
author_dir = ingest / "James S. A. Corey"
assert (author_dir / "Empire of Silence" / "Empire of Silence.m4b").exists()
assert (author_dir / "Howling Dark" / "Howling Dark.m4b").exists()
def test_pack_book_series_position_does_not_leak_from_searched_book(tmp_path):
staging = tmp_path / "staging"
ingest = tmp_path / "ingest"
staging.mkdir()
ingest.mkdir()
pack = staging / "Two"
for folder in ("Alpha", "Beta"):
(pack / folder).mkdir(parents=True)
(pack / folder / f"{folder.lower()}.m4b").write_text(folder)
result, _ = _run(
pack,
_audiobook_task(multi_book=True),
ingest,
staging,
audiobook_organize_template="{Author}/{SeriesPosition - }{Title}/{Title}",
)
assert result is not None
assert sorted(p.name for p in (ingest / "James S. A. Corey").iterdir()) == ["Alpha", "Beta"]
def test_multifile_book_inside_pack_keeps_part_numbers_per_book(tmp_path):
staging = tmp_path / "staging"
ingest = tmp_path / "ingest"
staging.mkdir()
ingest.mkdir()
pack = staging / "Pack"
(pack / "Book 1 - One").mkdir(parents=True)
(pack / "Book 2 - Two").mkdir(parents=True)
for i in (1, 2, 3):
(pack / "Book 1 - One" / f"part{i}.mp3").write_text(str(i))
(pack / "Book 2 - Two" / "two.mp3").write_text("t")
result, statuses = _run(pack, _audiobook_task(multi_book=True), ingest, staging)
assert result is not None
one = ingest / "James S. A. Corey" / "One"
assert sorted(p.name for p in one.iterdir()) == ["One - 01.mp3", "One - 02.mp3", "One - 03.mp3"]
assert (ingest / "James S. A. Corey" / "Two" / "Two.mp3").exists()
assert statuses[-1] == ("complete", "Complete (2 books, 4 files)")
def test_hardlinked_torrent_pack_leaves_source_tree_intact(tmp_path):
downloads = tmp_path / "downloads"
ingest = tmp_path / "ingest"
downloads.mkdir()
ingest.mkdir()
pack = _nested_pack(downloads)
task = _audiobook_task(source="prowlarr", multi_book=True, original_download_path=str(pack))
result, _ = _run(pack, task, ingest, tmp_path / "staging", hardlink=True)
assert result is not None
empire_src = pack / "Book 1 - Empire of Silence" / "empire.m4b"
empire_dst = ingest / "James S. A. Corey" / "Empire of Silence" / "Empire of Silence.m4b"
assert empire_src.exists()
assert empire_dst.exists()
assert os.stat(empire_src).st_ino == os.stat(empire_dst).st_ino
def test_without_pack_fields_nested_pack_is_still_one_book(tmp_path):
staging = tmp_path / "staging"
ingest = tmp_path / "ingest"
staging.mkdir()
ingest.mkdir()
pack = _nested_pack(staging)
result, _ = _run(pack, _audiobook_task(), ingest, staging)
assert result is not None
drive = ingest / "James S. A. Corey" / "Drive"
assert sorted(p.name for p in drive.iterdir()) == ["Drive - 01.m4b", "Drive - 02.m4b"]
def test_single_group_with_multi_book_flag_uses_searched_title(tmp_path):
staging = tmp_path / "staging"
ingest = tmp_path / "ingest"
staging.mkdir()
ingest.mkdir()
pack = staging / "Series" / "Book 1 - Solo"
pack.mkdir(parents=True)
(pack / "solo.m4b").write_text("s")
result, statuses = _run(pack.parent, _audiobook_task(multi_book=True), ingest, staging)
assert result is not None
assert (ingest / "James S. A. Corey" / "Drive" / "Drive.m4b").exists()
assert statuses[-1] == ("complete", "Complete")
+134
View File
@@ -0,0 +1,134 @@
"""API tests for POST /api/releases/inspect."""
from __future__ import annotations
import importlib
from unittest.mock import patch
import pytest
from shelfmark.download.postprocess.packs import PackFile
@pytest.fixture(scope="module")
def main_module():
with patch("shelfmark.download.orchestrator.start"):
import shelfmark.main as main
importlib.reload(main)
return main
@pytest.fixture
def client(main_module):
client = main_module.app.test_client()
with client.session_transaction() as sess:
sess["user_id"] = "tester"
sess["is_admin"] = False
return client
class _Handler:
def __init__(self, files):
self._files = files
def list_files(self, release_data):
if isinstance(self._files, Exception):
raise self._files
return self._files
def _inspect(client, handler, payload=None):
body = {
"source": "audiobookbay",
"source_id": "abc",
"title": "Drive",
"content_type": "audiobook",
"series_name": "The Expanse",
**(payload or {}),
}
with patch("shelfmark.core.release_inspect_routes.get_handler", return_value=handler):
return client.post("/api/releases/inspect", json=body)
def test_pack_release_returns_a_plan(client):
files = [
PackFile("The Expanse 1.0 - Leviathan Wakes (2011).m4b", 100),
PackFile("The Expanse 1.0 - Leviathan Wakes (2011).txt", 1),
PackFile("The Expanse 2.0 - Caliban's War (2012).m4b", 100),
]
resp = _inspect(client, _Handler(files))
assert resp.status_code == 200
data = resp.get_json()
assert data["inspected"] is True
assert data["reason"] is None
assert data["files"] == [
{"path": "The Expanse 1.0 - Leviathan Wakes (2011).m4b", "size": 100},
{"path": "The Expanse 1.0 - Leviathan Wakes (2011).txt", "size": 1},
{"path": "The Expanse 2.0 - Caliban's War (2012).m4b", "size": 100},
]
assert data["plan"]["is_pack"] is True
assert data["plan"]["ignored"] == ["The Expanse 1.0 - Leviathan Wakes (2011).txt"]
assert data["plan"]["books"] == [
{
"title": "Leviathan Wakes",
"series_position": 1.0,
"year": 2011,
"files": ["The Expanse 1.0 - Leviathan Wakes (2011).m4b"],
},
{
"title": "Caliban's War",
"series_position": 2.0,
"year": 2012,
"files": ["The Expanse 2.0 - Caliban's War (2012).m4b"],
},
]
def test_single_book_release_is_not_a_pack(client):
resp = _inspect(client, _Handler([PackFile("Drive.m4b", 5)]))
data = resp.get_json()
assert data["inspected"] is True
assert data["plan"]["is_pack"] is False
assert len(data["plan"]["books"]) == 1
def test_handler_without_file_list_reports_not_inspected(client):
resp = _inspect(client, _Handler(None))
assert resp.status_code == 200
data = resp.get_json()
assert data["inspected"] is False
assert data["reason"]
assert data["plan"] is None
def test_handler_failure_reports_not_inspected_without_500(client):
resp = _inspect(client, _Handler(RuntimeError("boom")))
assert resp.status_code == 200
data = resp.get_json()
assert data["inspected"] is False
assert "boom" in data["reason"]
def test_unknown_source_is_a_client_error(client):
with patch(
"shelfmark.core.release_inspect_routes.get_handler", side_effect=ValueError("no source")
):
resp = client.post(
"/api/releases/inspect", json={"source": "nope", "source_id": "x", "title": "t"}
)
assert resp.status_code == 400
def test_missing_source_id_is_a_client_error(client):
resp = client.post("/api/releases/inspect", json={"source": "audiobookbay"})
assert resp.status_code == 400
def test_requires_login(main_module):
anonymous = main_module.app.test_client()
with patch.object(main_module, "load_active_auth_mode", return_value="builtin"):
resp = anonymous.post(
"/api/releases/inspect", json={"source": "audiobookbay", "source_id": "a"}
)
assert resp.status_code == 401
+139
View File
@@ -0,0 +1,139 @@
"""GET /api/releases answers within a budget instead of outliving the caller.
Issue #1276: the endpoint is synchronous and had no deadline, while the bypass path it
reaches was allowed ~840s per URL. A protection challenge nobody could solve therefore
ran until the reverse proxy in front of Shelfmark gave up, and the user was shown
"Server unavailable (504). If using a reverse proxy, check its configuration." - which
names the wrong thing entirely.
"""
from __future__ import annotations
import importlib
from unittest.mock import patch
import pytest
from shelfmark.core import search_deadline
@pytest.fixture(scope="module")
def main_module():
with patch("shelfmark.download.orchestrator.start"):
import shelfmark.main as main
importlib.reload(main)
return main
@pytest.fixture
def client(main_module):
return main_module.app.test_client()
def _authenticate(client) -> None:
with client.session_transaction() as sess:
sess["user_id"] = "alice"
sess["is_admin"] = False
sess["db_user_id"] = 7
def _request(client, main_module, sources, search_impl):
"""Drive /api/releases with a stubbed source list and search implementation."""
class _Source:
def search(self, book, plan, *, expand_search=False, content_type="ebook"):
return search_impl(book, plan)
def get_column_config(self):
from shelfmark.release_sources import _default_column_config
return _default_column_config()
with (
patch.object(main_module, "get_auth_mode", return_value="none"),
patch("shelfmark.release_sources.list_available_sources", return_value=sources),
patch("shelfmark.release_sources.get_source", return_value=_Source()),
patch("shelfmark.release_sources.source_results_are_releases", return_value=False),
):
return client.get(
"/api/releases",
query_string={"provider": "manual", "book_id": "abc", "title": "Dune"},
)
def test_a_search_runs_under_a_budget(client, main_module):
"""The handler must put a deadline in force for whatever the sources do."""
_authenticate(client)
observed: list[float | None] = []
def _search(_book, _plan):
deadline = search_deadline.current()
observed.append(deadline.budget_seconds if deadline else None)
return []
resp = _request(client, main_module, [{"name": "direct_download", "enabled": True}], _search)
assert resp.status_code == 200
assert observed and observed[0] is not None, "no budget was in force during the search"
def test_the_budget_is_shared_across_sources(client, main_module):
"""A stuck first source must not spend the whole request on its own."""
_authenticate(client)
searched: list[str] = []
def _search(book, _plan):
searched.append(book.title)
# Whatever the first source did, it ran the clock out.
deadline = search_deadline.current()
if deadline is not None:
deadline.event.set()
return []
sources = [
{"name": "direct_download", "enabled": True},
{"name": "prowlarr", "enabled": True},
]
resp = _request(client, main_module, sources, _search)
assert len(searched) == 1, "the second source should not have been started"
assert "ran out of time" in resp.get_json()["error"]
def test_the_failure_carries_a_message_the_frontend_will_show(client, main_module):
"""The whole point of the budget.
Shelfmark answers 503 when a search comes back empty with errors, and the frontend
only substitutes its "Server unavailable ... check your reverse proxy" text when the
body carries no message of its own. So the budget has to produce a body that names
the protection challenge - and has to trip before the proxy's own timeout, where
there would be no body at all.
"""
_authenticate(client)
def _search(_book, _plan):
deadline = search_deadline.current()
if deadline is not None:
deadline.event.set()
return []
sources = [
{"name": "direct_download", "enabled": True},
{"name": "prowlarr", "enabled": True},
]
resp = _request(client, main_module, sources, _search)
message = resp.get_json()["error"]
assert "protection challenge" in message
assert "reverse proxy" not in message
# The source prefix is stripped by the handler; the sentence must survive intact.
assert message.startswith("The release search ran out of time")
def test_no_budget_leaks_out_of_the_request(client, main_module):
"""A queued download later on must not inherit a search's deadline."""
_authenticate(client)
_request(client, main_module, [{"name": "direct_download", "enabled": True}], lambda *_: [])
assert search_deadline.current() is None
@@ -114,6 +114,41 @@ def test_releases_accepts_direct_download_provider(main_module, client):
assert all(call.args == ("direct_download",) for call in mock_get_source.call_args_list)
def test_releases_falls_back_to_the_session_users_default_languages(main_module, client):
"""A request without a language filter searches in the reader's own languages."""
import shelfmark.core.search_plan as search_plan
planned_languages: list[list[str] | None] = []
class _LanguageProbeSource(_FakeDirectSource):
def search(self, book, plan, expand_search=False, content_type="ebook"):
planned_languages.append(plan.languages)
return []
def fake_get(key, default=None, user_id=None):
if key == "BOOK_LANGUAGE":
return ["de"] if user_id == 42 else ["en"]
return default
with client.session_transaction() as session:
session["user_id"] = "reader"
session["db_user_id"] = 42
with patch.object(main_module, "get_auth_mode", return_value="none"):
with patch.object(search_plan, "config", SimpleNamespace(get=fake_get)):
with patch("shelfmark.release_sources.get_source", return_value=_LanguageProbeSource()):
resp = client.get(
"/api/releases",
query_string={
"provider": "direct_download",
"book_id": "md5-abc",
},
)
assert resp.status_code == 200
assert planned_languages == [["de"]]
def test_releases_direct_provider_returns_404_when_book_missing(main_module, client):
class _MissingDirectSource:
def get_record(self, record_id, *, fetch_download_count=True):
+257
View File
@@ -0,0 +1,257 @@
"""The wall-clock budget on a release search.
Issue #1276: `/api/releases` is synchronous and nothing bounded it, while the bypass path
it can reach was allowed ~840s per URL. A search that ran into an unsolvable protection
challenge therefore outlived every reverse proxy in front of it, and the user was shown
"Server unavailable (504)" - a gateway timeout that names their proxy rather than the
challenge that actually failed.
"""
import threading
import pytest
from shelfmark.core import search_deadline
@pytest.fixture(autouse=True)
def _no_ambient_deadline():
"""Each test starts outside any budget."""
token = search_deadline._current.set(None)
yield
search_deadline._current.reset(token)
# --------------------------------------------------------------------------- #
# The budget itself
# --------------------------------------------------------------------------- #
def test_no_budget_outside_a_search():
"""A queued download must not inherit a search's budget."""
assert search_deadline.current() is None
assert search_deadline.expired() is False
assert search_deadline.cancel_event() is None
def test_budget_applies_inside_the_context_and_not_after():
with search_deadline.search_deadline(60) as deadline:
assert search_deadline.current() is deadline
assert search_deadline.expired() is False
assert search_deadline.current() is None
def test_expiry_trips_the_cancel_event():
"""The Event is the mechanism: the bypassers poll it and know nothing of deadlines."""
with search_deadline.search_deadline(0.05):
event = search_deadline.cancel_event()
assert isinstance(event, threading.Event)
assert event.wait(timeout=5) is True
assert search_deadline.expired() is True
def test_timer_is_cancelled_on_exit():
"""A finished search must not leave a timer running to fire later."""
with search_deadline.search_deadline(3600) as deadline:
pass
assert deadline._timer.finished.is_set()
def test_message_names_the_challenge_not_the_proxy():
with search_deadline.search_deadline(120):
message = search_deadline.deadline_message()
assert "120s" in message
assert "protection challenge" in message
assert "504" not in message
# --------------------------------------------------------------------------- #
# Reading the setting
# --------------------------------------------------------------------------- #
@pytest.mark.parametrize(
("configured", "expected"),
[
(600, 600.0),
("450", 450.0),
(None, search_deadline.DEFAULT_SEARCH_BUDGET_SECONDS),
("nonsense", search_deadline.DEFAULT_SEARCH_BUDGET_SECONDS),
(0, search_deadline.DEFAULT_SEARCH_BUDGET_SECONDS),
(True, search_deadline.DEFAULT_SEARCH_BUDGET_SECONDS),
(5, 30.0), # clamped up: below this nothing can finish
(99999, 1800.0), # clamped down
],
)
def test_budget_seconds_coerces_and_clamps(monkeypatch, configured, expected):
from shelfmark.core.config import config as app_config
monkeypatch.setattr(
app_config,
"get",
lambda key, default=None: configured if key == "RELEASE_SEARCH_TIMEOUT" else default,
)
assert search_deadline.budget_seconds() == expected
# --------------------------------------------------------------------------- #
# What the search path does with it
# --------------------------------------------------------------------------- #
def test_html_get_page_will_not_start_a_bypass_on_a_spent_budget(monkeypatch):
"""A minutes-long solve nobody is still waiting for is worse than a clear failure."""
import shelfmark.download.http as http
started: list[str] = []
monkeypatch.setattr(
http, "get_bypassed_page", lambda *a, **k: started.append(a[0]) or "<html/>"
)
monkeypatch.setattr(http, "_is_cf_bypass_enabled", lambda: True)
monkeypatch.setattr(http, "_bypass_grace_seconds", lambda: 1.0)
monkeypatch.setattr(http.network, "should_rotate_dns_for_url", lambda _url: True)
monkeypatch.setattr(http, "get_proxies", lambda _url: {})
monkeypatch.setattr(http.time, "sleep", lambda _s: None)
class _Selector:
current_base = "https://annas-archive.gl"
attempts_this_dns = 0
last_failure = None
def rewrite(self, url):
return url
with search_deadline.search_deadline(60) as deadline:
deadline.event.set()
result = http.html_get_page(
"https://annas-archive.gl/search",
retry=1,
selector=_Selector(),
use_bypasser=True,
success_delay=0,
)
assert result == ""
assert started == [], "no solve should have been started"
def test_search_budget_becomes_the_cancel_flag(monkeypatch):
"""This is what makes the budget bite on a solve already running."""
import shelfmark.download.http as http
seen: list[object] = []
def fake_bypass(_url, _selector=None, cancel_flag=None):
seen.append(cancel_flag)
return "<html>solved</html>"
monkeypatch.setattr(http, "get_bypassed_page", fake_bypass)
monkeypatch.setattr(http, "_is_cf_bypass_enabled", lambda: True)
monkeypatch.setattr(http, "_bypass_grace_seconds", lambda: 1.0)
monkeypatch.setattr(http.network, "should_rotate_dns_for_url", lambda _url: True)
monkeypatch.setattr(http, "get_proxies", lambda _url: {})
monkeypatch.setattr(http.time, "sleep", lambda _s: None)
class _Selector:
current_base = "https://annas-archive.gl"
attempts_this_dns = 0
last_failure = None
def rewrite(self, url):
return url
with search_deadline.search_deadline(60) as deadline:
http.html_get_page(
"https://annas-archive.gl/search",
retry=1,
selector=_Selector(),
use_bypasser=True,
success_delay=0,
)
assert seen == [deadline.event]
def test_a_callers_own_cancel_flag_is_not_replaced(monkeypatch):
"""A queued download brings its own and must keep it."""
import shelfmark.download.http as http
seen: list[object] = []
def fake_bypass(_url, _selector=None, cancel_flag=None):
seen.append(cancel_flag)
return "<html>solved</html>"
monkeypatch.setattr(http, "get_bypassed_page", fake_bypass)
monkeypatch.setattr(http, "_is_cf_bypass_enabled", lambda: True)
monkeypatch.setattr(http, "_bypass_grace_seconds", lambda: 1.0)
monkeypatch.setattr(http.network, "should_rotate_dns_for_url", lambda _url: True)
monkeypatch.setattr(http, "get_proxies", lambda _url: {})
monkeypatch.setattr(http.time, "sleep", lambda _s: None)
class _Selector:
current_base = "https://annas-archive.gl"
attempts_this_dns = 0
last_failure = None
def rewrite(self, url):
return url
own_flag = threading.Event()
with search_deadline.search_deadline(60):
http.html_get_page(
"https://annas-archive.gl/search",
retry=1,
selector=_Selector(),
cancel_flag=own_flag,
use_bypasser=True,
success_delay=0,
)
assert seen == [own_flag]
def test_expired_budget_reports_the_challenge_not_a_cancellation(monkeypatch):
"""The budget trips the same flag a user's cancel does; the messages must differ."""
import shelfmark.download.http as http
from shelfmark.bypass import BypassCancelledError
def fake_bypass(*_a, **_k):
msg = "Bypass cancelled"
raise BypassCancelledError(msg)
monkeypatch.setattr(http, "get_bypassed_page", fake_bypass)
monkeypatch.setattr(http, "_is_cf_bypass_enabled", lambda: True)
monkeypatch.setattr(http, "_bypass_grace_seconds", lambda: 1.0)
monkeypatch.setattr(http.network, "should_rotate_dns_for_url", lambda _url: True)
monkeypatch.setattr(http, "get_proxies", lambda _url: {})
monkeypatch.setattr(http.time, "sleep", lambda _s: None)
failures: list[str] = []
class _Selector:
current_base = "https://annas-archive.gl"
attempts_this_dns = 0
def rewrite(self, url):
return url
@property
def last_failure(self):
return None
@last_failure.setter
def last_failure(self, value):
if value:
failures.append(value)
with search_deadline.search_deadline(60) as deadline:
# Expire mid-solve: the flag is set, but the caller reaches the handler by way of
# BypassCancelledError, which on its own reads as "someone cancelled this".
deadline.expires_at = 0.0
http.html_get_page(
"https://annas-archive.gl/search",
retry=1,
selector=_Selector(),
use_bypasser=True,
success_delay=0,
)
assert failures, "a give-up reason should have been recorded"
assert "ran out of time" in failures[-1]
assert "cancelled" not in failures[-1]
+57 -3
View File
@@ -2,12 +2,27 @@ from shelfmark.core.search_plan import build_release_search_plan
from shelfmark.metadata_providers import BookMetadata
def _book_language_config_get(
global_languages: list[str],
user_languages: list[str] | None = None,
):
"""Stand in for config.get(), answering BOOK_LANGUAGE per user."""
def _get(key: str, default: object = None, user_id: int | None = None) -> object:
if key != "BOOK_LANGUAGE":
return default
if user_id is not None and user_languages is not None:
return user_languages
return global_languages
return _get
class TestReleaseSearchPlan:
def test_uses_default_languages_when_none(self, monkeypatch):
# config.BOOK_LANGUAGE is a Config attribute; patch the instance.
import shelfmark.core.search_plan as sp
monkeypatch.setattr(sp.config, "BOOK_LANGUAGE", ["en", "hu"], raising=False)
monkeypatch.setattr(sp.config, "get", _book_language_config_get(["en", "hu"]))
book = BookMetadata(
provider="hardcover",
@@ -46,7 +61,7 @@ class TestReleaseSearchPlan:
def test_all_language_disables_grouping(self, monkeypatch):
import shelfmark.core.search_plan as sp
monkeypatch.setattr(sp.config, "BOOK_LANGUAGE", ["en"], raising=False)
monkeypatch.setattr(sp.config, "get", _book_language_config_get(["en"]))
book = BookMetadata(
provider="hardcover",
@@ -68,3 +83,42 @@ class TestReleaseSearchPlan:
assert [(v.title, v.languages) for v in plan.grouped_title_variants] == [
("The Lightning Thief", None),
]
def test_user_default_languages_beat_the_global_default(self, monkeypatch):
import shelfmark.core.search_plan as sp
monkeypatch.setattr(
sp.config,
"get",
_book_language_config_get(["en"], user_languages=["de", "en"]),
)
book = BookMetadata(
provider="hardcover",
provider_id="123",
title="The Final Empire",
authors=["Brandon Sanderson"],
)
assert build_release_search_plan(book).languages == ["en"]
assert build_release_search_plan(book, user_id=7).languages == ["de", "en"]
def test_explicit_languages_beat_the_user_default(self, monkeypatch):
import shelfmark.core.search_plan as sp
monkeypatch.setattr(
sp.config,
"get",
_book_language_config_get(["en"], user_languages=["de", "en"]),
)
book = BookMetadata(
provider="hardcover",
provider_id="123",
title="The Final Empire",
authors=["Brandon Sanderson"],
)
plan = build_release_search_plan(book, languages=["fr"], user_id=7)
assert plan.languages == ["fr"]
@@ -0,0 +1,148 @@
"""Language filters must reach the sources as ISO codes.
Issue #1276: a debug bundle showed `BOOK_LANGUAGE=english` arriving at Anna's Archive
verbatim as `&lang=english`. AA matches that parameter against ISO codes, so it is not a
loose spelling of `lang=en` - it is a facet value AA does not have, and it empties every
search. In that bundle a successful solve of an ISBN search for Philosopher's Stone
returned zero hits, while a warm-up query carrying no language filter hit the same host
in the same minute and returned 50.
Only the per-user override was normalised (config.users_settings.validate). The global
default - which is what the env var feeds - was passed through with nothing but a
.strip(), so anyone carrying `BOOK_LANGUAGE=english` from the old docs had every search
silently filtered to nothing, with no error anywhere.
"""
import logging
import pytest
import shelfmark.core.search_plan as sp
from shelfmark.metadata_providers import BookMetadata
def _config_get(global_languages):
def _get(key: str, default: object = None, user_id: int | None = None) -> object:
return global_languages if key == "BOOK_LANGUAGE" else default
return _get
def _book() -> BookMetadata:
return BookMetadata(
provider="hardcover",
provider_id="123",
title="Harry Potter and the Philosopher's Stone",
search_title="Harry Potter and the Philosopher's Stone",
search_author="J.K. Rowling",
authors=["J.K. Rowling"],
)
@pytest.fixture
def plan_logs():
"""Collect search_plan log records.
setup_logger builds its loggers outside the standard hierarchy, so caplog's root
handler never sees them, and Logger.setLevel cannot clear their is-enabled cache.
"""
messages: list[str] = []
class _Capture(logging.Handler):
def emit(self, record: logging.LogRecord) -> None:
messages.append(record.getMessage())
handler = _Capture()
sp.logger.addHandler(handler)
previous = sp.logger.level
sp.logger.setLevel(logging.DEBUG)
sp.logger._cache.clear()
try:
yield messages
finally:
sp.logger.removeHandler(handler)
sp.logger.setLevel(previous)
# --------------------------------------------------------------------------- #
# The global default (what BOOK_LANGUAGE feeds)
# --------------------------------------------------------------------------- #
@pytest.mark.parametrize(
("configured", "expected"),
[
(["english"], ["en"]), # the spelling from the old docs - the reported bug
("english", ["en"]), # env vars arrive as a bare string
(["English"], ["en"]),
(["eng"], ["en"]), # ISO 639-2
(["en"], ["en"]), # already a code, unchanged
(["english", "german"], ["en", "de"]),
(["english", "en", "English"], ["en"]), # collapses to one code
],
)
def test_default_languages_reach_the_plan_as_iso_codes(monkeypatch, configured, expected):
monkeypatch.setattr(sp.config, "get", _config_get(configured))
plan = sp.build_release_search_plan(_book(), languages=None)
assert plan.languages == expected
def test_unrecognised_default_searches_unfiltered_rather_than_empty(monkeypatch, plan_logs):
"""Dropping the filter is the safe failure: a warned-about unfiltered search beats
telling the user a book AA is full of does not exist."""
monkeypatch.setattr(sp.config, "get", _config_get(["klingon"]))
plan = sp.build_release_search_plan(_book(), languages=None)
assert plan.languages is None
assert any("klingon" in m and "BOOK_LANGUAGE" in m for m in plan_logs), plan_logs
def test_partly_unrecognised_default_keeps_what_resolved(monkeypatch, plan_logs):
monkeypatch.setattr(sp.config, "get", _config_get(["english", "klingon"]))
plan = sp.build_release_search_plan(_book(), languages=None)
assert plan.languages == ["en"]
assert any("klingon" in m for m in plan_logs)
def test_a_clean_default_logs_no_warning(monkeypatch, plan_logs):
monkeypatch.setattr(sp.config, "get", _config_get(["en", "de"]))
sp.build_release_search_plan(_book(), languages=None)
assert not [m for m in plan_logs if "Ignoring unrecognised" in m]
# --------------------------------------------------------------------------- #
# Explicit request languages go through the same door
# --------------------------------------------------------------------------- #
def test_explicit_languages_are_normalised_too(monkeypatch):
"""The request branch had the same bare .strip(), so an API client could reproduce
the bug even with BOOK_LANGUAGE set correctly."""
monkeypatch.setattr(sp.config, "get", _config_get(["en"]))
plan = sp.build_release_search_plan(_book(), languages=["english", "German"])
assert plan.languages == ["en", "de"]
def test_all_still_means_no_language_filter(monkeypatch):
monkeypatch.setattr(sp.config, "get", _config_get(["en"]))
assert sp.build_release_search_plan(_book(), languages=["all"]).languages is None
def test_blank_entries_are_skipped(monkeypatch):
monkeypatch.setattr(sp.config, "get", _config_get(["en"]))
plan = sp.build_release_search_plan(_book(), languages=["english", "", " ", None])
assert plan.languages == ["en"]
def test_no_configured_default_leaves_the_filter_off(monkeypatch):
monkeypatch.setattr(sp.config, "get", _config_get(None))
assert sp.build_release_search_plan(_book(), languages=None).languages is None
+43
View File
@@ -85,6 +85,7 @@ def test_users_me_edit_context_includes_search_preferences_when_visible(app, use
{
"SEARCH_MODE": "universal",
"METADATA_PROVIDER": "openlibrary",
"BOOK_LANGUAGE": ["de", "en"],
},
)
client = _authed_client_for_user(app, user)
@@ -102,8 +103,13 @@ def test_users_me_edit_context_includes_search_preferences_when_visible(app, use
assert resp.json["searchPreferences"]["tab"] == "search_mode"
assert resp.json["searchPreferences"]["effective"]["SEARCH_MODE"]["value"] == "universal"
assert resp.json["searchPreferences"]["effective"]["SEARCH_MODE"]["source"] == "user_override"
assert resp.json["searchPreferences"]["effective"]["BOOK_LANGUAGE"]["value"] == ["de", "en"]
assert resp.json["searchPreferences"]["effective"]["BOOK_LANGUAGE"]["source"] == (
"user_override"
)
assert "SEARCH_MODE" in resp.json["userOverridableKeys"]
assert "METADATA_PROVIDER" in resp.json["userOverridableKeys"]
assert "BOOK_LANGUAGE" in resp.json["userOverridableKeys"]
assert resp.json["notificationPreferences"] is None
assert resp.json["userOverridableKeys"] == sorted(resp.json["userOverridableKeys"])
@@ -170,6 +176,43 @@ def test_users_me_update_accepts_visible_section_settings(app, user_db):
assert resp.json["settings"]["DESTINATION"] == "/books/alice"
def test_users_me_update_accepts_book_language_when_search_section_visible(app, user_db):
user = user_db.create_user(username="alice")
client = _authed_client_for_user(app, user)
with patch("shelfmark.core.self_user_routes.load_active_auth_mode", return_value="builtin"):
with patch(
"shelfmark.core.self_user_routes.app_config.get",
side_effect=_visible_sections_config_get(["search"]),
):
resp = client.put(
"/api/users/me",
json={"settings": {"BOOK_LANGUAGE": ["German", "en"]}},
)
assert resp.status_code == 200
assert user_db.get_user_settings(user["id"])["BOOK_LANGUAGE"] == ["de", "en"]
def test_users_me_update_rejects_book_language_when_search_section_hidden(app, user_db):
user = user_db.create_user(username="alice")
client = _authed_client_for_user(app, user)
with patch("shelfmark.core.self_user_routes.load_active_auth_mode", return_value="builtin"):
with patch(
"shelfmark.core.self_user_routes.app_config.get",
side_effect=_visible_sections_config_get(["delivery"]),
):
resp = client.put(
"/api/users/me",
json={"settings": {"BOOK_LANGUAGE": ["de"]}},
)
assert resp.status_code == 400
assert resp.json["error"] == "Some settings are admin-only"
assert user_db.get_user_settings(user["id"]) == {}
def test_users_me_update_rejects_non_object_settings_payload(app, user_db):
user = user_db.create_user(username="alice")
client = _authed_client_for_user(app, user)
+144
View File
@@ -0,0 +1,144 @@
"""A spent search budget stops the search rather than starting the next attempt.
One release search fans out: a mirror loop inside `_fetch_search_table`, then a title
variant per grouped variant, then the whole set again without the language filter. Each
of those can reach the bypasser, and `except Exception` around the variant loops was
built to keep going past a parse failure. Applied to a spent budget that meant the
variants queued up behind each other long after anyone was still waiting - which is how
a challenge failure became a gateway timeout (issue #1276).
"""
import pytest
import shelfmark.release_sources.direct_download as dd
from shelfmark.core import search_deadline
@pytest.fixture(autouse=True)
def _no_ambient_deadline():
token = search_deadline._current.set(None)
yield
search_deadline._current.reset(token)
class _Selector:
current_base = "https://annas-archive.gl"
last_failure = None
def rewrite(self, url: str) -> str:
return url
def next_mirror_or_rotate_dns(self, *, fatal: bool = False, reason: str = ""):
return None, "exhausted"
def test_fetch_search_table_gives_up_when_the_budget_is_spent(monkeypatch):
"""Every mirror shares the protection, so another mirror is another wasted solve."""
fetches: list[str] = []
monkeypatch.setattr(dd.downloader, "html_get_page", lambda url, **_k: fetches.append(url) or "")
monkeypatch.setattr(dd.network, "get_available_aa_urls", lambda: ["https://annas-archive.gl"])
with search_deadline.search_deadline(60) as deadline:
deadline.event.set()
with pytest.raises(dd.SearchUnavailableError) as excinfo:
dd._fetch_search_table("https://annas-archive.gl/search?q=dune", _Selector())
assert fetches == [], "no fetch should have been attempted"
assert "ran out of time" in str(excinfo.value)
def test_fetch_search_table_runs_normally_within_budget(monkeypatch):
page = "<html><body><main><table><tbody></tbody></table></main></body></html>"
monkeypatch.setattr(dd.downloader, "html_get_page", lambda _url, **_k: page)
monkeypatch.setattr(dd.network, "get_available_aa_urls", lambda: ["https://annas-archive.gl"])
with search_deadline.search_deadline(60):
html, table = dd._fetch_search_table("https://annas-archive.gl/search?q=dune", _Selector())
assert table is not None
assert html == page
def _plan(titles):
"""A search plan with one grouped title variant per title."""
from shelfmark.core.search_plan import ReleaseSearchPlan, ReleaseSearchVariant
variants = [ReleaseSearchVariant(t, "Frank Herbert", ["en"]) for t in titles]
return ReleaseSearchPlan(
languages=["en"],
isbn_candidates=[],
author="Frank Herbert",
title_variants=variants,
grouped_title_variants=variants,
)
def _book():
from shelfmark.metadata_providers import BookMetadata
return BookMetadata(
provider="manual",
provider_id="x",
provider_display_name="Manual",
title="Dune",
search_title="Dune",
authors=["Frank Herbert"],
)
def test_title_variants_stop_once_the_budget_is_spent(monkeypatch):
"""`except Exception` keeps this loop going past a failure; a spent budget must not."""
queries: list[str] = []
def fake_search_books(query, _filters):
queries.append(query)
# The first variant is what spends the budget.
deadline = search_deadline.current()
if deadline is not None:
deadline.event.set()
return []
monkeypatch.setattr(dd, "search_books", fake_search_books)
monkeypatch.setattr(dd, "_ensure_direct_download_available", lambda: None)
source = dd.DirectDownloadSource()
with search_deadline.search_deadline(60):
releases = source.search(_book(), _plan(["Dune", "Duna", "Dünen"]))
assert releases == []
assert len(queries) == 1, f"only the first variant should have run, got {queries}"
def test_all_title_variants_run_within_budget(monkeypatch):
queries: list[str] = []
monkeypatch.setattr(dd, "search_books", lambda q, _f: queries.append(q) or [])
monkeypatch.setattr(dd, "_ensure_direct_download_available", lambda: None)
source = dd.DirectDownloadSource()
with search_deadline.search_deadline(60):
source.search(_book(), _plan(["Dune", "Duna"]))
# Two variants with a language filter, then both again without one.
assert len(queries) == 4
def test_language_filter_retry_is_skipped_on_a_spent_budget(monkeypatch):
"""The no-language sweep doubles the work; it must not start after the deadline."""
queries: list[str] = []
def fake_search_books(query, _filters):
queries.append(query)
if len(queries) == 2:
deadline = search_deadline.current()
if deadline is not None:
deadline.event.set()
return []
monkeypatch.setattr(dd, "search_books", fake_search_books)
monkeypatch.setattr(dd, "_ensure_direct_download_available", lambda: None)
source = dd.DirectDownloadSource()
with search_deadline.search_deadline(60):
source.search(_book(), _plan(["Dune", "Duna"]))
assert len(queries) == 2, f"the retry sweep should not have started, got {queries}"
@@ -110,3 +110,22 @@ def test_unreachable_mirror_raises_search_unavailable(monkeypatch):
except dd.SearchUnavailableError:
return
raise AssertionError("expected SearchUnavailableError")
def test_recorded_failure_reason_is_surfaced_to_the_caller(monkeypatch):
"""The concrete give-up reason html_get_page stashed replaces the generic line."""
import shelfmark.release_sources.direct_download as dd
reason = "Anna's Archive returned 403 (blocked) and no bypasser is enabled."
def fake_get(url, *, selector=None, **_kwargs):
selector.last_failure = reason
return ""
monkeypatch.setattr(dd.downloader, "html_get_page", fake_get)
monkeypatch.setattr(dd.network, "get_available_aa_urls", lambda: ["a"])
selector = _Selector(["https://real.test"])
selector.last_failure = None
with pytest.raises(dd.SearchUnavailableError, match="403 .blocked."):
dd._fetch_search_table("https://real.test/search?q=dune", selector)

Some files were not shown because too many files have changed in this diff Show More