mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-11-16 07:08:11 +00:00
e575ff0fcb586bc0aa0600a2ea198d5b8c05dd5d
70
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
e575ff0fcb |
feat(activity): clearer cards, a Rejected view and Reopen for failed deliveries (#1618)
A batch of activity list changes from running it as an admin on my own
instance. They are separate commits that build on each other, so they
can be reviewed in order.
The cards read better. Every card gets a format pill (violet for audio,
orange for ebooks), the requester is named once, and finished cards say
how long a download took and how long ago it ended ("took 5m · 35m
ago"). A download that picked up work already under way shows a
Reattached pill.
Admins can tell what is automatic and what needs a person. A declined
request shows who declined it and when, anything with an `[auto]` note
reads "Declined automatically", and "Needs Review" becomes "No Match
Yet" with a line saying the search repeats on every sync.
There is a Rejected view next to History for admins, with Reopen on each
card and a way to hide the dead ones.
The part to read most carefully is the backend in `requests_service.py`
and `user_db.py`. An admin can now reopen a fulfilled request whose
download failed or was cancelled, and the reopen a failed download
triggers remembers which release failed, so the next search doesn't pick
it again.
I ran it on a scratch instance (when pulling them together to move from
my fork to upstream) as an admin with seeded requests and looked at the
pills, the durations, the wording and the Rejected view in a browser.
Reopening a fulfilled request I only exercised in the tests. The
frontend and Python suites, type check, ruff and basedpyright pass.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **New Features**
* Admins can review rejected requests in a dedicated view, hide or
restore cards, and reopen requests.
* Fulfilled requests with failed or cancelled deliveries can be
reopened; the failed release is skipped in the next automatic search.
* Activity cards now show format badges, download timing, and when a
download resumes existing work.
* **Improvements**
* Activity views clarify automatic retries and actions, and show
reviewer names for admins.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
ed0777a2e0 |
Spool large debrid downloads to disk (#1632)
Fixes #1630. Debrid downloads currently buffer entire files in RAM. The four clients now opt into a spooled temporary file: downloads up to 8 MiB stay in memory, and larger downloads roll to disk. The clients copy from the file in chunks and close it after use. Other download_url callers retain the existing BytesIO behavior. Temporary files are closed on unsuccessful attempts and cancellation; successful resumed downloads remain open for their caller. Added offline coverage for disk rollover, timeout cleanup, and resumed content. Updated the Debrid-Link test doubles for the new opt-in argument. Validation: 113 tests passed across test_http_download_url, test_debridlink_client, and test_torbox_client; Ruff check and format check passed for changed files; git diff --check passed. No live debrid services were exercised. AI assistance was used in implementing this change. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Improvements** * Large downloads are handled more efficiently, reducing memory use while preserving download and resume behavior. * **Bug Fixes** * Temporary download data is cleaned up when a download is cancelled or fails, helping prevent unnecessary resource usage. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
947ce7bfe5 |
test(e2e): serve the mock Prowlarr .torrent from Prowlarr's own origin (#1602)
The nightly client-deluge profile has failed since
|
||
|
|
e8b28d05e0 |
fix(sources): take the retry download URL from the release cache only (#1597)
Newznab and Jackett fell back to the download URL sent with the queue request when the release was no longer in the cache. Prowlarr already returns no URL in that case, so this makes the other two match it: the retry URL comes from the cache and nothing else. A request for a release that has expired from the cache now fails with "Release not found in cache" instead of being queued with whatever URL came along. The new tests cover both handlers on a cache miss and with no source_id, and they fail on main. Full suite, ruff and basedpyright pass. |
||
|
|
ac14e4dd24 |
fix(downloads): a swarm note change is not activity for the stall timer (#1572)
The torrent stall timer can be deferred for as long as the swarm keeps changing. Since #1494 the torrent handler appends a note to its progress message (`0% · 2 seeds`, and `· 0 seeds, giving up in 7m` while the no-seeders clock runs). `update_download_status` treats any changed message as activity, so a seed count that goes 2, 3, 1 between polls, or the give-up countdown ticking down every minute, refreshes `_last_activity` and the torrent never stalls. A torrent whose seed count stays constant still stalls, which is why this is easy to miss. `shelfmark/download/activity.py` already says what counts: a changed status event or a changed progress value, and a message that changes on a timer proves nothing. So `update_download_status` now compares messages through a small `activity_key()` that drops that trailing seed note. The card still gets every message and shows the live seed count, only the stall clock ignores a change in the note. A new progress percentage, a different client message or a status change still counts as activity. The handler builds the note in `_seed_note` and `activity_key` has to match it, so one test builds the note for a range of seed counts and countdowns and checks the key strips all of them. That also covers the gap #1494 left, where no test connected a handler's message to the orchestrator: the other two drive `update_download_status` with real notes and a fake clock. Without the change the note test fails and the key tests cannot even import `activity_key`. The other orchestrator test is a control that a progress or status change still counts. Full suite passes, and each new test also passes when run alone. |
||
|
|
9011319ade |
refactor: one lookup for the supported formats, public names for two shared helpers (#1576)
Six places read the Supported Book Formats and Supported Audiobook Formats settings on their own: post-processing, the post-process scan, IRC, slskd, Prowlarr and direct download. For ebooks they already agreed, since they all fall back to `DEFAULT_EBOOK_FORMATS`. For audiobooks they did not. The settings registry and slskd fall back to every audio format plus zip and rar, while post-processing and IRC fell back to m4b and mp3 only. They now share `shelfmark/core/formats.py`, with `get_supported_formats`, `get_supported_audiobook_formats` and a content-type aware `get_supported_formats_for`, and a shared `DEFAULT_AUDIOBOOK_FORMATS` in `core/utils.py` next to the ebook one. `policy.py` re-exports the same functions, so nothing that imports them from there changes. It is not in `policy.py` itself because importing the `postprocess` package pulls in the router, which imports the release sources, so a source could not import from it without a cycle. One behaviour difference, and only in the fallback: with the audiobook setting unset, or emptied, post-processing and IRC used to accept m4b and mp3 and now accept the full default list like slskd and the settings page already did. A configured list is read exactly as before. Direct download's own helper used the raw configured list and now reads the same normalized one, which is identical for any list the settings page can save. Also public now: `parse_size` in the Prowlarr source (slskd and Newznab imported it with the underscore) and `coerce_completed_path_timeout_seconds` in the client base handler (slskd imported that too). Not changed on purpose: slskd's `_coerce_int` looks like a copy of `coerce_int` but is not. It reads `"12.0"` as 12 and `True` as 0, where `coerce_int` gives 0 and 1. And the `_parse_size` in the IRC source is a different function, string to bytes. Tests: a new set checks that post-processing, IRC and slskd all fall back to the shared defaults and that the settings default is the shared one, and the scan-archive and IRC tests now patch the shared config instead of the per-module copies. Full suite passes. |
||
|
|
4f97fc9bec |
fix: keep server paths and URLs out of an error response and a debug log (#1558)
Two small leaks from going through the open CodeQL alerts (code scanning alerts 100 and 123). The local download route returned `str(e)` on an OSError, and for a missing or unreadable file that text is the file's path on the server (`[Errno 2] No such file or directory: '/audiobooks/Author/Title.epub'`). #1422 cuts a non-admin's download path down to the file name everywhere else, so this put it back. The route now answers "Could not read the file" and the detail stays in the log. `should_bypass_proxy` logged the whole request URL at debug when it could not be parsed. URLs can carry credentials, so it logs only the exception now. Each change has a test that fails without it. The full non-e2e suite, ruff and basedpyright pass. |
||
|
|
1aa9846f22 |
fix(download): trust a pinned AA mirror and repair the full e2e profile (#1518)
The destination guard from the oceanofpdf/SSRF fix (
|
||
|
|
f837506017 |
Merge commit from fork
A signed-in user picks the release fields of POST /api/releases/download, and a record id with the "oceanofpdf:" prefix reached the provider without handles() or is_enabled(). The provider then fetched source_url, POSTed to the form action found on that page and fetched the meta-refresh target, with no check on any of them. The provider now refuses to run when it is switched off, and the book page, the form action and every POST redirect hop must be on the configured mirror (same scheme, host and port). The signed file link may sit on another host but must be http(s). network.ensure_public_destination refuses a host that resolves to a loopback, private, link-local or otherwise non-global address (IPv4-mapped IPv6 judged by its IPv4 address). Hosts the operator configured as mirrors and names that do not resolve pass as before. html_get_page and download_url call it on their first URL and check each redirect hop through a response hook. The image cache uses the shared address check too. Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
|
a00f3d28bf |
fix(slskd): refuse unsafe peer paths, keep queued downloads alive, report search outages (#1512)
Fixes four defects in the Soulseek (slskd) A peer names its own files, and Shelfmark works out where slskd stored one from the last folder and the file name. `remote_directory_name` returned the whole directory when its last segment was empty. So `../../../etc//passwd.epub` or `/abs/path//book.epub` resolved outside the downloads folder, and Shelfmark imported whatever was there. For a multi-file release it moved those files into staging (`STAGE_MOVE`). - New `safe_remote_segments` refuses a name with an empty, `.` or `..` segment (which covers absolute and UNC paths), a NUL, or a drive anywhere but a Windows share root (`C:\Books\…`, which never reaches a local path). - Search (`iter_peer_files`, so `build_releases`) drops such files. - `download` refuses them before anything is queued, in both modes. The spec arrives with the queued release payload, so search filtering alone is not enough. - `local_path_for` also refuses a path that resolves outside the root (`resolve()` + `is_relative_to`), for example through a symlinked folder. `_poll`, used when isolation is off and as the fallback for slskd < 0.26, never asked for an activity grace. A file "Queued, Remotely" sent the same status and progress on every poll, so the orchestrator cancelled it after 300s and `SLSKD_QUEUE_TIMEOUT_MINUTES` never applied. It now requests, renews and releases the grace the same way isolated mode does. `search` caught every exception per query, so slskd being down or rejecting the API key showed as "no releases". When no query gets an answer it now raises `SourceUnavailableError`, as Prowlarr does, and the error reaches the user. One failed query among answered ones is still not an outage. A retry reuses the task's folder, and slskd renames a new file when one is already there. Files from a failed attempt therefore meant `_folder_complete`'s exact count could never match again. Now: - a failed or cancelled attempt removes its folder; - a new attempt removes any finished files an earlier one left (for example after a restart) before it queues the batch. `tests/slskd/conftest.py` stubbed `flask_socketio` whenever it had not been imported yet, even though it is installed. Running `pytest tests/slskd tests/` therefore broke the Flask app tests in the same worker (285 errors, also on |
||
|
|
49aeec97cd |
fix(download): stop deleting user data and losing completed downloads (#1511)
Fixes six defects. Each one breaks downloads or deletes user data, and each has a regression test that fails on |
||
|
|
6f65d6e20a |
Merge commit from fork
A release format such as /../../x was appended to the staging file name unchanged, so the direct-download and libgen handlers could write to, and on cancel unlink, a path outside TMP_DIR. Normalise the format to a plain extension at queue time and in build_filename, reject path-like source ids for the sources that use them as file names, and make both handlers resolve the staging path and refuse anything that is not a direct child of TMP_DIR. |
||
|
|
82c8c91c8b |
refactor(clients): read the stall window with a type check instead of a pyright ignore (#1498)
The only `pyright: ignore[reportArgumentType]` in `shelfmark/` is one I added in #1426, and I was being lazy when I did. I hit the same error on another PR and replaced the ignore there with a real check, so this does the same for the original. `_stall_window_seconds` reads the setting as an `object`. It now narrows to int, float or str before calling `float()`, and a bad string falls back to the default as before. A non-numeric value of another type (None, a list) falls back to the default too, which used to happen through a `TypeError`. No behaviour change. The existing range test still passes and I added `None` and a list to it. basedpyright is at 0 errors, ruff is clean and the backend suite passes. Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
|
e83bab9337 |
feat(clients): give up on a torrent with no seeders (#1494)
A torrent with no seeds won't finish. It runs on the pieces its few
peers hold and then sits still, and nothing says so until the stall
timer ends it. The card only reads "downloading". I run AudiobookBay
releases, which show no seed count in the listing, so the client is the
only place to find out.
Two new settings, both off until you turn them on:
`TORRENT_NO_SEEDERS_WAIT` (minutes, 0 turns it off) and
`TORRENT_MIN_SEEDERS` (default 1).
- A torrent that has fewer seeds than the minimum for the whole wait,
while nothing is downloading, is removed from the client with its data
and the task ends as an error.
- A torrent with no seed at all for the wait is given up on even while
it still moves.
- A torrent that met the minimum and lost seeds down to one, but is
still moving, keeps going.
- The card shows the seed count beside the progress, with a countdown
while it waits ("0 seeds, giving up in 4m").
qBittorrent only, because it reports connected and tracker seed counts.
Other clients report none and are left to the stall timer, and an
unknown count is not read as zero. A torrent that was already in the
client when the download started is never removed, and a queued torrent
isn't judged.
Tests cover the wait and countdown, the clock restarting on progress,
the tracker count, a higher minimum, off by default, a client with no
counts, a preexisting torrent, a queued torrent, and the moving cases.
Ruff and the backend suite pass.
|
||
|
|
06a67a7150 |
feat(download): a concurrency limit per source (#1495)
With one global limit, a few tasks that are only waiting hold every slot. A Soulseek download sitting in a peer's queue is the example I hit: three of those and a torrent that could start waits behind them. `MAX_CONCURRENT_DOWNLOADS_PER_SOURCE` (0 keeps the single global limit, which is the default). With it set, the coordinator starts a download only when its source is below the limit and passes over tasks of a full source without losing their place in the queue. `get_next` gets an optional filter for this, and a cancelled task is still dropped when one is given. The start-up stagger now applies only between downloads from the same source, and the pool is sized so every source can reach its limit. The second commit: qBittorrent keeps its own queue of torrents waiting for a free slot. The task polling one of those does no work, so it no longer counts against its source's limit and the next task of that source can start. The client does the pacing, as it does for its own queue. Tests cover the queue pass-over (order and priority kept, nothing eligible, cancelled tasks), the coordinator (a full source doesn't block the others, each source has its own limit, the global limit when unset) and the client-queue case. Ruff and the backend suite pass. |
||
|
|
10710abd4e |
fix(download): send every file of a multi-file download to the browser (#1489)
Download to Browser only ever sent the first file of a multi-file release (audiobook chapters, packs): the folder output returned final_paths[0] and /api/localdownload served that one path. The folder output now keeps every final path on the in-memory task (download_paths), and /api/localdownload streams them as one uncompressed ZIP named after the book when there is more than one. The archive is never staged on disk or held in memory, and ZIP64 covers large audiobooks. History still stores only the first path, so a re-download from history after a dismiss or restart serves a single file, as before. Also brings every linter back to green on all files: - pre-commit: bump the ruff hook to v0.17.1 to match uv.lock; ruff-format the Python examples in docs/dev and skills - ruff: fix the remaining issues in skills/shelfmark/download_books.py; format base_handler.py - basedpyright: Optional and attrs errors in databazeknih.py - knip: drop 31 unused exports, reuse ColumnColorHint in colorMaps - oxfmt 0.72: styles.css - direct download: fix the "Welib, mirrors" typo and the test that still expected the pre-OceanofPDF message Fixes #1467 |
||
|
|
51f32b374d |
feat(packs): let users choose which files, books and formats to download (#1480)
## Summary - **Skip files or books in a multi-book pack.** The pack review panel gets a switch per file and a "Skip this book" switch per book, using the shared `ToggleSwitch`. A partly skipped book shows e.g. "3 of 5 files" in the Files column. A fully skipped book is greyed out. - **Choose formats for a single book.** A new "Choose formats" header toggle, next to the multi-book one, opens a format picker for a release that has several book formats (EPUB, MOBI, PDF and so on). It has one switch per format with file count and size. Releases with one format, or that can't be inspected, download as before. - **Skipped files are sent as `ignored_files`.** They are stored on the task (including restart-safe retry) and removed before grouping and transfer. This matters for packs because files missing from `book_plan` would otherwise be filed by the heuristic fallback. Ignored entries match by release-relative path, then path suffix, then unique basename. - A pack that ends up with one book after skips keeps the titles you reviewed instead of falling back to the searched title. ## Screenshots <img width="1277" height="413" alt="Screenshot 2026-10-19 at 09-39-28 Shelfmark" src="https://github.com/user-attachments/assets/c005ca4b-3dac-45dd-b2ff-d02fc0f9fcc9" /> <img width="1277" height="1210" alt="Screenshot 2026-10-19 at 09-40-23 Shelfmark" src="https://github.com/user-attachments/assets/2145f3ba-e5fc-4635-9364-1790feb0c8b8" /> <img width="1277" height="748" alt="Screenshot 2026-10-19 at 09-39-03 Shelfmark" src="https://github.com/user-attachments/assets/39023fac-70e3-46d8-af89-1160ebd6e415" /> ## Test plan - [x] `pytest` pack tests (new `drop_ignored_files` cases added) - [x] `vitest`, `tsc`, `oxlint`, `oxfmt`, `ruff` - [x] Manual check in the UI with a real multi-book pack 🤖 (mostly) Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
|
0ade063db5 |
fix(download): extract RAR archives in place, not in the IO threadpool (#1459)
Fixes #1458. A RAR download fails with `Download failed: TypeError`, and the log says `child watchers are only available on the default loop`. rarfile extracts by running `unrar` as a subprocess, and `extract_archive_files` calls it through `run_blocking_io`, so it runs in gevent's IO threadpool. gevent only lets a subprocess start on the thread that owns the default loop. `run_blocking_io` already avoids the threadpool for `subprocess.run`, but a RAR gets to the subprocess through rarfile, so nothing catches it. This extracts a RAR on the calling greenlet, where the patched subprocess waits for `unrar` without blocking anything else. ZIP is pure Python and stays in the threadpool. I checked it with a real RAR4 file on a build of main with the stock image's `unrar`: from the threadpool it raises the TypeError, in place it extracts all 7 files. Two tests. One checks that a RAR doesn't go through `run_blocking_io`. The other runs under real gevent patching with an extractor that starts a process the way rarfile does. Both fail without the change, and the full suite passes. I keyed it on the `.rar` suffix to keep it small. Teaching `run_blocking_io` about rarfile's subprocess would be the more general fix, but that's a bigger change to a helper a lot of code goes through, so I left it for you to say if you want it. |
||
|
|
47707b827a |
feat(download): make the torrent stall window a setting (#1426)
#1420 gave a torrent 15 minutes to get going and 15 minutes after each step forward. That suits my swarms, but other people will want it shorter or longer, so this makes the number a setting. `TORRENT_STALL_TIMEOUT_MINUTES` sits in Download Clients with the other torrent options and takes 5 to 60. The default is 15, so nothing changes unless someone sets it. 5 is the orchestrator's own stall timeout, so choosing it asks for no extra time and gives the old behaviour. The value can arrive as text from an env var, so anything that isn't a number falls back to the default and anything out of range is clamped. The orchestrator caps one activity grace at 16 minutes, so I raised that to an hour to fit the top of the range. As far as I can tell the protection bypass is the only caller that asks for more than a few minutes, but I haven't audited every caller. If you'd rather keep the cap, I can lower the maximum to 16 minutes instead. Tests cover the default, 5 and 30, text values, out of range values and NaN, the start and movement windows following the setting, and the setting showing up in the Download Clients tab. I broke each rule on purpose to check a test catches it. The full suite fails the same tests with and without this change (the bypass and seleniumbase tests don't run in my environment). |
||
|
|
d99e9d7c4f |
fix(download): remove a cancelled torrent that never started (#1421)
Cancelling a torrent download leaves the torrent in the client. `_safe_remove_download` documents the rule: "torrents: never remove or delete client data (avoid breaking seeding)". That makes sense for a torrent that has data. A torrent that is still at 0% has nothing to seed and nothing to resume, and leaving it behind holds a download slot in qBittorrent's queue or sits there as a dead entry. I hit this on a qBittorrent that is shared with Sonarr, Radarr and Lidarr and has a download limit. Shelfmark cancelled downloads that were queued (#1420) and left them in the client. Three torrents that never fetched metadata held every active slot, and each torrent added after them waited behind them. Removing those by hand freed the slots. This removes the torrent and its files when a download is cancelled, Shelfmark added the torrent, and the client reports 0% progress. A torrent with any data is left alone as before. So is one that was already in the client when the download started, since Shelfmark didn't add it. If the removal fails it is logged and the cancel carries on. Tests cover removal at 0% for a queued and a downloading torrent, leaving one that has data, leaving one the user already had, and a failing removal. The full suite passes, and I broke each rule on purpose to check a test catches it. This changes a rule you wrote down, so I've kept it apart from the queue fix. If you'd rather have it as a setting, or only remove a torrent that is still queued or fetching metadata, I'm happy to rework it. |
||
|
|
88f73e3bc3 |
fix(download): do not cancel a queued torrent as stalled (#1420)
qBittorrent only runs a few downloads at once (`max_active_downloads` defaults to 3) and holds the rest as `queuedDL`. Shelfmark's stall timer cancels a download after five minutes without a changed status or progress. A queued torrent reports "Queued" on every poll, so it looks idle, and Shelfmark cancels a download that was only waiting for a slot. I run Shelfmark against the same qBittorrent that Sonarr, Radarr and Lidarr use, which I assume is a common setup. Their torrents and Shelfmark's share one limit, so anything that fills the slots makes Shelfmark's torrent wait. In my case three torrents that never fetched metadata held every slot. I'd expect a busy Sonarr queue to hold them the same way, but I haven't reproduced that. Once the stall timer had cancelled the waiting downloads they stayed in the client, still queued behind the same blockers. On my instance 10 AudioBookBay downloads were cancelled while qBittorrent still had them queued. This changes the torrent poll loop. When the client reports a queued torrent, it asks for an activity grace (the mechanism `html_get_page` already uses for protection bypasses) and releases it as soon as the torrent leaves the queue. The grace is renewed every ten minutes and stops at two hours, so a queue that never moves still ends. A torrent that is downloading and moving is timed as before until the new window below applies. The second commit sets the stall message before the cancel runs. The terminal hook copies the message into the history row, and it was set afterwards, so a download the timer ended was recorded with the last poll's "Queued". That left no way to tell it apart from a person cancelling. The last two commits are about slow torrents. The timer only counts a change in progress, so a torrent that needs more than five minutes to fetch metadata or find its first peer is cancelled before it has any progress to count, and one that moves a few megabytes at a time can be cancelled between bursts. I hit this with a torrent that had a single peer. A torrent now gets one 15 minute window the first time it is seen outside the queue, and each step forward in progress renews it. One that makes no progress for 15 minutes is still cancelled, so a dead torrent now takes about 15 minutes to end instead of five. That trade is the part I'm least sure about, so the number is easy to change. Tests cover the grace being requested once, released when the torrent starts, renewed, and capped, the order of the message and the cancel, and the start and movement windows. The full suite passes, and I broke each rule on purpose to check a test catches it. The two hour ceiling is a guess. If you'd rather have a setting for it, or a different number, I'm happy to change it. |
||
|
|
41a4df01c4 |
fix: hand DiamWall 513 challenges to the bypasser (#1400)
## Summary Recognize DiamWall interstitials in both the HTTP retry path and the internal browser helper. HTTP 513 challenge responses go directly to the configured bypasser, while ordinary 513 responses keep the existing error behavior. Fixes #1386 ## Testing - `uv run --frozen --extra browser pytest -q -n 0 tests/download/test_http_challenge_513.py tests/bypass/test_internal_bypasser.py`: 27 passed - Ruff lint and format checks passed |
||
|
|
b690832659 |
fix(download): stream a completed book instead of buffering it in RAM - lowering memory needs significantly (#1378)
Credit where it is due: this defect was found and measured by **@DrNgo** in [DrNgo/shelfmark-fork@ae2185c](https://github.com/DrNgo/shelfmark-fork/commit/ae2185c8a83d537e55f1dc03f745ab844b5cdcdb). He recorded a 493 MB audiobook peaking near 963 MB and OOM-killing a 1 GiB container, with the proxy access log showing a single `GET /api/localdownload` returning 502 at the exact second of the kill. The analysis is his; I am sending it because it is still open here. Serving a completed book from the live queue calls `get_book_data`, which reads the whole file into bytes so the route can wrap it in a `BytesIO` for `send_file`. That is two copies of the book, with one resident for the length of the client transfer, to hand over a file that is already sitting on disk. `get_book_path` returns the path the task already holds and `send_file` streams it. The history fallback a few lines up in the same route has always worked this way, so this makes the two paths consistent rather than introducing anything new. `get_book_data` stays for callers that genuinely want the bytes, now documented as the expensive option. ## The fetch side has the same problem, and this PR does not fix it I should note: `download_url` in `shelfmark/download/http.py` still builds the inbound file in a `BytesIO` and returns it, so a large download is fully resident while it fetches. @DrNgo's commit fixes that too, with a `tempfile.SpooledTemporaryFile(max_size=...)` so small payloads stay in memory exactly as they do now and large ones spill to disk. I left it out deliberately. It changes the return type of `download_url` from `BytesIO` to a file object, which touches several callers, and it lands in a file that has been reworked around bypass handling, waiting rooms and resume since his branch point. That deserves its own PR rather than riding along with a two-function change. I may send it later; if @DrNgo sends it first, his should win, and if you would rather have both together say so and I will hold this one. ## Verification - `tests/download/test_orchestrator_retry.py`: `get_book_path` returns the path without opening the file (the test fails the run if it does), and reports a missing file rather than handing back a dead path. - The existing `/api/localdownload` tests still pass unchanged, including the history fallback and the ownership checks. - Full suite (3222), ruff, ruff format, basedpyright, vulture green. |
||
|
|
bc03ad062e |
fix(http): keep the host of a protocol-relative download link (#1368)
`get_absolute_url()` replaced both `netloc` and `scheme` whenever either one was missing. A protocol-relative href such as `//cdn.example.org/f.epub`, scraped from a page on `https://annas-archive.org/...`, parses with a netloc and an empty scheme, so it came back pointing at the page's own host. The download then 404s and the source is skipped. Each field now falls back to the base URL only when the parsed URL does not supply it. Plain relative paths resolve exactly as before, which the control test covers. This affects the Z-Library, welib and generic download link handling in `release_sources/direct_download/annas_archive.py`. ## Verification - New `tests/download/test_http_absolute_url.py`: a protocol-relative link keeps its own host, and a plain `/path` still resolves against the base. The first fails on current main and passes here. - Full suite (3165), ruff, ruff format, basedpyright, vulture green. |
||
|
|
d4619be69a |
Feature: Show the AA search result stats (#1362)
When doing a direct search for a book, show the stats of the AA results. For example if we search for "The Great Gatsby", AA reports it has 240 hits and shows the first page of 50. Provide this stats info in the shelfmark webUI via ResultsSection.tsx and ReleaseModal.tsx. This table shows what should be displayed based on the total number of hits found: |Total |Display| |-------|-------| |1 |"Result 1 (1 Total)"| |6 |"Results 1-6 (6 Total)"| |144 |"Results 1-50 (144 Total)"| |500+ |"Results 1-50 (500+ Total)"| Currently, shelfmark also only shows us the first 50 search hits even if there were more available from AA. This could be added later if considered desirable. As usual, a picture is worth a 1000 words: <img width="1012" height="610" alt="direct-results-stat" src="https://github.com/user-attachments/assets/8c0c688f-444e-482a-a9a3-2dee5a5563b8" /> <img width="1013" height="741" alt="universal-results-info" src="https://github.com/user-attachments/assets/399bbb79-87f2-4432-a3d0-64937795f5f1" /> Coded with llama.cpp, opencode and 🤖 |
||
|
|
b7002a6eca |
feat: Add TorBox client support and settings integration (#1342)
Add **TorBox** as a torrent download client for Prowlarr releases. Users can select `TorBox` in the download client settings, configure it with the new `TORBOX_API_KEY` environment variable, and verify their credentials with the connection test button. The integration supports both magnet links and `.torrent` files. It tracks the torrent lifecycle through TorBox, downloads supported book and audiobook files from the TorBox CDN, preserves safe nested file paths, and cleans up remote and local download state. Important: Shared HTTP download logs omit full download URLs and URL-bearing exception text to avoid exposing credentials, following best practices. This applies to all clients that use the shared `download_url()` path; URLs remain available to the HTTP operations themselves. --- There is already related work in progress in #1173, which includes both torrent and direct-download support for TorBox. This PR is not intended to replace or compete with that contribution. It offers the tested torrent client functionality as a smaller, focused change that can make TorBox available to the community sooner. The direct-download integration proposed in #1173 remains valuable and could be reviewed or introduced separately. Automated tests cover configuration, connection validation, magnet and torrent-file submission, API errors, status and progress handling, file retrieval, path traversal protection, cancellation, cleanup, and sensitive URL redaction. I also validated the complete flow locally with several magnet links and `.torrent` downloads. TorBox processed the torrents and Shelfmark downloaded the resulting files as expected. AI was used to help with the implementation, with human validation. This PR and long description? Took me some good minutes at night after work, but gives me joy to open this PR to share with the community this improvement. |
||
|
|
e5dd34ae0e |
fix: unbreak main and follow up on the Blackhole handoff review (#1346)
DownloadHistoryService.record_download and updated the single production caller, but not the eleven in the test suite, leaving main red with 32 failures. Pass None, which is what the pre-#1336 behaviour recorded. For the Blackhole handoff (#1345): add_download publishes the torrent before the cancel check runs, and BlackholeClient.remove() is a no-op, so the watcher picks the file up regardless. Reporting a bare "Cancelled" hid that from the user. Name the completed handoff in the cancellation message instead, drop the _handle_cancelled_download call whose usenet branch cannot apply to a handoff-only client, and record why the orchestrator no longer verifies HandoffResult.path. Finally, make tests/direct_download a package: test_libgen_extract.py imports tests.libgen.sample_html across test directories, so without an __init__.py pytest named its modules by bare basename and a same-named module elsewhere would collide. |
||
|
|
8f608f2e64 |
fix(download): complete consumed Blackhole handoffs (#1345)
A Blackhole watcher can consume the torrent before Shelfmark checks it, leaving the task in error even though the handoff succeeded. Complete the handoff when `add_download` successfully publishes the file, and stop requiring a `HandoffResult` path to remain present. Follow-up to #1312. ## Verification - A watcher that immediately reads and removes the torrent receives the exact bytes. The task changes from ERROR before this fix to COMPLETE afterward, without running book postprocessing. - The consumed-file regression fails on current main and passes here. Resident files, write failures, cancellation, magnet rejection and normal downloads remain covered: 81 focused tests pass. - Ruff lint and formatting pass for the changed files. |
||
|
|
2bb84a17a2 |
Extract archives when zip/rar are enabled as supported formats (#1343)
The default audiobook formats include `zip` and `rar`. `scan_directory_tree` checks the supported-format list before checking for archives, so a downloaded archive lands in `book_files` and is imported as-is. The extraction branch in `collect_directory_files` is never reached. This keeps archives out of `book_files`, so they always take the archive path: extracted when extraction is allowed, imported as-is when it isn't (unchanged). Tests added in `tests/download/test_postprocess_scan_archives.py`; three of the four fail without the change. |
||
|
|
265da07d7f |
feat(download): add Blackhole torrent handoff (#1312)
## Why Blackhole users need Shelfmark to hand a torrent file to their existing downloader instead of importing the downloaded book itself. ## Change - add Blackhole as a torrent client with a configurable watched directory - prefer a fetched `.torrent` file for Blackhole while preserving magnet preference for other clients - complete the queue task after the handoff without invoking book post-processing ## Verification - `uv run pytest -q tests/prowlarr/test_blackhole_client.py tests/prowlarr/test_handler.py tests/newznab/test_handler.py tests/download/test_orchestrator_lifecycle.py` - `uv run basedpyright shelfmark/download/clients/blackhole.py shelfmark/download/clients/__init__.py shelfmark/download/clients/base_handler.py shelfmark/download/clients/settings.py shelfmark/download/orchestrator.py shelfmark/release_sources/__init__.py shelfmark/release_sources/prowlarr/utils.py shelfmark/release_sources/prowlarr/handler.py shelfmark/release_sources/newznab/handler.py tests/prowlarr/test_blackhole_client.py tests/prowlarr/test_handler.py tests/newznab/test_handler.py tests/download/test_orchestrator_lifecycle.py` Fixes #1229 |
||
|
|
c18da92569 |
fix(postprocess): attach unmatched chaptered audio files to existing book group (#1176) (#1309)
### Summary Fixes #1176 When downloading an audiobook with many chaptered tracks (e.g. 250+ `.flac` or `.mp3` files), indexer XML or release metadata often caps the file list at ~100-110 entries. When the release extracts on disk, `match_plan_to_files()` matched those first ~110 files to the planned book group, while the remaining 140+ files fell into `unmatched` and triggered fallback heuristic grouping. Because heuristic grouping parsed the folder name (`Westwell - Hot & Cold (2023)`) and stripped the series/author prefix, it generated a second book titled `Hot & Cold` containing the remaining tracks, resulting in two split book folders. ### Changes - In `match_plan_to_files()` (`shelfmark/download/postprocess/packs.py`), check `unmatched` files before falling back to heuristic multi-book splitting. - If an unmatched file is chaptered audio (`.flac`, `.mp3`, `.aac`, etc.) and shares the directory with an existing book group, or if the plan was a single-book plan, append it to that group instead of creating a secondary book. - Non-chaptered standalone books (e.g. `.m4b`, `.epub`) or files in separate subfolders continue to fall back to heuristic grouping as before. - Added unit tests in `tests/download/test_packs.py` verifying: 1. Truncated track list in single folder properly appends remaining chaptered tracks without splitting. 2. Single-book plan with multi-disc audio files (`CD1`/`CD2`) groups together cleanly. 3. Multi-book packs with unmatched chaptered tracks route each track to its respective book folder. ### Testing Ran `uv run pytest tests/download/test_packs.py` and `uv run pytest tests/core/test_processing_packs.py` (all passed cleanly). Checked type annotations with `basedpyright` (0 errors) and formatting with `ruff`. Co-authored-by: amasen02 <amasen02@users.noreply.github.com> |
||
|
|
9452ebc70d |
fix(bypass): stop handing solvers DDoS-Guard's ?check=1 probe URL (#1300)
html_get_page follows Anna's Archive redirects by hand, and DDoS-Guard's gate answers /search with a 302 to the same path plus `check=1`. The follower walks that handshake by reassigning `current_url`, so every downstream handoff - the 403 branch, the 503-challenge branch, both redirect-loop rescues - passed the *probe* URL to the bypasser rather than the page we actually wanted. A solver opens that in a fresh browser holding none of the cookies the probe exists to collect, so DDoS-Guard cannot verify it automatically and serves the manual CAPTCHA page that nothing can solve. The #1292 log is exactly that: a 403 handed off on `&check=1`, FlareSolverr answering "Challenge solved!", and a 4721-byte DDOS-GUARD captcha page coming back. - `_solvable_url()` strips the probe parameter, applied at the single choke point in `_run_bypasser` so all four handoffs are covered. Scoped to the hosts whose redirects we follow manually; a URL without the parameter is returned by identity, so nothing else is re-encoded. The same reports showed three further defects, all of which stand whatever the host was reacting to: - The external bypasser logged that the solve had not cleared the protection and then returned the challenge page as a success. That skipped the one recovery left - get_bypassed_page's retry-and-rotate loop, where the next mirror is a different DDoS-Guard host - and filed the captcha page's own __ddg cookies as that host's clearance, to be replayed on every later request. It now raises ChallengeNotSolvedError before storing anything. - "Check that the bypasser is reachable and working" was the one piece of advice guaranteed to waste the reporter's time: it was reachable, it ran a full solve, and it returned a captcha. ChallengeNotSolvedError carries the marker so the search layer can name the host as the cause instead of the bypasser. - The untabled-page fingerprint logged `attempt_url`, which html_get_page has since rotated past. The #1298 bundle reported the page against annas-archive.gl when the body had come from .pk - the triage cost #1289 added the line to remove. The search now asks for the response URL and logs that. Its give-up shape is the tuple ("", url), which is truthy, so the exhaustion check reads the body rather than the response. Regression fixtures are built from the pages in the reports. The two behavioural handoff tests were checked against the unfixed code: both fail there, reproducing the reporter's log line verbatim. Refs #1292 Refs #1298 |
||
|
|
97e289ae13 | fix: search, Prowlarr and qBittorrent follow-ups (#1276, #1283) (#1284) | ||
|
|
b25acdb2ad |
fix(packs): don't disrupt normal downloads when inspecting for packs (#1274)
Follow-ups to the multi-book pack feature (#1270), which inspects every release before download. Two behaviours leaked into the ordinary single-book flow and are corrected here: - A flat folder of chaptered audio (`01 - Chapter.mp3`, `02 - ...`) was detected as a pack, because each track name parses to a series position, so clicking download popped the review panel for one normal audiobook. Flat folders are now split one-book-per-file only with real evidence of distinct books: two or more series positions, more than one title, and no chaptered audio (only the single-file m4b/m4a containers and ebook formats qualify). Subfolder packs and flat m4b/m4a packs are unchanged. - Every release that couldn't be inspected (usenet, magnet-only, sources without a list_files hook, ABB single-file) showed an info toast on download. That is now a console.warn, so a normal download is silent again. Adds regression tests for the chaptered-mp3 cases. |
||
|
|
f441b85da2 |
feat(packs): inspect multi-book releases and file each book separately (#1270)
## Multi-book packs: inspect a release before download and file each book separately Closes #576 ### Problem One queued release is always treated as one book. When a torrent is actually a whole series (`Series/Book 1 - Title/…`, or a flat folder of `Series 1.0 - Title.m4b` files), post-processing walks the whole tree, flattens every file into one list and renames them `Title - 01…10` under the searched book's `{Author}/{Title}`. Audiobookshelf then sees a single 10-file "book" and the user has to re-file everything by hand. ### What this does Most releases expose their file list *before* anything is downloaded, so the split is decided up front and approved by the user, then the download is fire-and-forget: 1. **Inspect** – clicking a release's download button now calls `POST /api/releases/inspect` first. A new optional `DownloadHandler.list_files(release_data)` hook returns the release's files without downloading: - **AudiobookBay** reads the torrent file table off the detail page it already fetches (the page is now cached for 120 s, so inspect + download cost ABB one request). - **Prowlarr** parses `info.files` from the `.torrent` it already fetches (the existing 120 s torrent-fetch cache is reused). Magnet-only and usenet releases report "can't inspect". - Other sources default to `None`. 2. **Review** – if the plan contains more than one book, the Find Releases modal swaps the list for a review panel: one row per book with editable title / series position / year, expandable file lists, non-book sidecars (`.txt`, covers) shown as ignored, a "Treat as a single book" switch, and **Download N books**. Single-book releases queue immediately, exactly as before. 3. **File** – the approved plan travels with the task (`DownloadTask.book_plan`, retry-safe) and post-processing files each book through the existing transfer code, one book at a time (`dataclasses.replace(task, title=…, series_position=…, year=…)`), so organize/rename templates, part numbering (now scoped per book), hardlinks, torrent copy-preserve and usenet handling are unchanged. Status reads `Complete (N books, M files)`. 4. **Fallback** – when a release can't be inspected the user gets a toast, and a small "Multi-book pack" toggle in the modal header forces a heuristic split (subfolder = book, or one book per file when the file names carry series positions). Planning lives in `shelfmark/download/postprocess/packs.py` and is shared by the inspect endpoint and post-processing, so what the user approved is what gets filed. The name parser strips `Book 3 -`, `03 -`, `1.0 -`, `3.`, `[03]`, `#3`, a leading series name, labels like "An Expanse Novella -", repeated titles (`Gods of Risk 2.5 - Gods of Risk`) and a trailing `(Year)`; author and series name come from the book that was searched, and the searched book's own series position is never applied to its siblings. ### Files - `shelfmark/download/postprocess/packs.py` (new) – `PackFile/PackBook/PackPlan`, `plan_pack`, `parse_pack_book_name`, `group_files_into_books`, `match_plan_to_files` - `shelfmark/core/release_inspect_routes.py` (new) – `POST /api/releases/inspect` - `shelfmark/release_sources/__init__.py` – `DownloadHandler.list_files` hook - `shelfmark/release_sources/audiobookbay/{scraper,handler}.py` – detail-page cache, `extract_file_list`, `list_files` - `shelfmark/release_sources/prowlarr/handler.py`, `download/clients/torrent_utils.py` – `extract_file_list_from_torrent`, `list_files` - `shelfmark/core/models.py`, `download/orchestrator.py` – `multi_book` / `book_plan` fields, queue + retry serialization - `shelfmark/download/postprocess/transfer.py`, `pipeline.py`, `outputs/folder.py` – per-book transfer branch and status message - `src/frontend`: `components/PackReviewPanel.tsx` (new), `ReleaseModal.tsx`, `App.tsx`, `services/api.ts`, `types/index.ts`, `utils/releasePayload.ts` (payload builder moved out of `App.tsx`), `utils/packReview.ts` - `docs/dev/release-sources-plugin-guide.md` – documents the `list_files` hook ### Out of scope (follow-ups) - Listing files from an NZB (Shelfmark already fetches the bytes; `<file subject>` names are noisy) - Inspecting magnet links via qBittorrent's files API after a paused add - BookLore / email outputs (they ignore `book_plan`; noted in code) - The combined ebook + audiobook flow ### Testing **Automated** (`make checks`, `make python-test`, `make frontend-test` all green; the only failures on my machine are the pre-existing `tests/config/test_entrypoint_permissions.py` cases, which need bash ≥ 4 and fail identically on `main` under macOS bash 3.2): - `tests/download/test_packs.py` – name parsing (markers, series name, novella labels, repeated titles, bare numeric titles like `1984`), nested / flat / mixed / deeper-nested packs, single wrapping folder not treated as a pack, plan-to-disk matching with basename fallback - `tests/core/test_processing_packs.py` – full `post_process_download` runs on a real temp filesystem: approved plan files each book under its own `{Author}/{Title}`, heuristic split of a nested pack, searched book's series position does not leak, multi-file book inside a pack keeps `- 01/- 02` per book, hardlinked torrent pack leaves the seeding tree intact, no pack fields ⇒ behaviour unchanged, single group degrades to the searched title, status message - `tests/core/test_release_inspect_routes.py` – plan response, not-inspectable, handler errors never 500, unknown source / missing `source_id` ⇒ 400, login required - `tests/audiobookbay/test_file_list.py` – file-table scraping from real ABB markup (multi-file and single-file pages), handler host validation, one page fetch shared by magnet + file list - `tests/prowlarr/test_torrent_file_list.py` – multi-file / single-file `.torrent` parsing, handler behaviour for torrent URL vs magnet vs usenet vs cache miss - `tests/download/test_orchestrator_pack_fields.py` – queue-time parsing and retry round-trip - Frontend: `releasePayload.test.ts`, `packReview.test.ts` (vitest) **Manual, on a real deployment** (arm64 image built from this branch, run as a side container next to production with the same qBittorrent / Audiobookshelf setup, `FILE_ORGANIZATION_AUDIOBOOK=organize`, hardlinks on): - AudiobookBay "The Expanse Complete 2.0" (7.87 GB, 36 files): clicking download opened the review panel in ~1 s showing **18 books · 18 files · 18 files ignored** (the `.txt` sidecars), with series positions 0.1–9.5 and years parsed from the file names; novella labels stripped ("The Churn", "The Butcher of Anderson Station"). Editing a title in the panel works. Confirming queued one task; the magnet resolved from the cached page in ~30 ms; after the download the task reported `Complete (18 books, 18 files)`, 18 hardlinks landed as `audiobooks/James S. A. Corey/<Title>/<Title>.m4b`, the torrent kept seeding, and Audiobookshelf scanned each folder as its own book (title, author, embedded chapters). - A second pack ("Expanse [01 - 9.5]", `Title N - Title` naming) was inspected to verify the repeated-title rule and the Back button, without downloading. - Single-book releases still queue immediately with no extra UI. |
||
|
|
ddc26f01b6 |
fix(download): escalating per-host cooldown on HTTP 429 (#1263)
Anna's Archive 429-throttles the source IP after repeated automated requests. The bypasser could clear the DDoS-Guard challenge but not the 429, so each retry re-solved, re-spawned Chrome, and rotated mirrors that share the same IP - a costly loop that never converged. Add a process-global, per-host cooldown that escalates 2 -> 5 -> 10 -> 15 -> 30 minutes each time a host 429s again after its window elapsed, resetting after a long clear gap. Mirror selection skips cooling hosts and the bypasser refuses to solve one, so a throttled host fails fast instead of storming the solver. |
||
|
|
12d554a92f |
fix(download): hand a 503 carrying a challenge to the bypasser (#1238)
503 is in RETRYABLE_CODES, and the bypasser is only ever reached from the 403 branch and the AA redirect-loop rescues. Once Z-Library re-serves its DDoS-Guard interstitial with the same cookie the #1188 handshake already echoed back, the request has nothing left to try and spends every attempt on the same wall. Gate the handoff on the response body rather than the status, so a genuine overloaded-origin 503 keeps its retry path, and on allow_bypasser_fallback, so best-effort fetches still fail fast. The challenge indicators move out of internal_bypasser into shelfmark/bypass/challenge.py so http.py can use them without importing SeleniumBase, which is lazily imported precisely because it is optional. Refs #1233 |
||
|
|
651096ed7b |
fix(bypass): reuse external bypasser clearance instead of re-solving (#1223)
Direct download was unusable behind an external bypasser (FlareSolverr /
Byparr): every request paid a 403 plus a full solve, and a search that
never ran was reported to the user as "No books found".
Clearance was discarded on the external path. get_cf_cookies_for_domain
and get_cf_user_agent_for_domain returned {} / None whenever
USING_EXTERNAL_BYPASSER was set, and _fetch_via_bypasser read only
solution.response - dropping solution.cookies and solution.userAgent,
which FlareSolverr-compatible services do return. A solve therefore
cleared the one request that paid for it and nothing else, and a file
download - which the solver cannot proxy, being binary - presented no
clearance at all. Diagnosed from a v1.3.9 debug bundle: ~35s in the
bypasser per search, on every search.
- Move the cookie jar out of internal_bypasser into bypass/cookie_store.
internal_bypasser imports seleniumbase at module scope, which is the
dependency an external-bypasser deployment is entitled not to have, so
it cannot host a store the external path depends on.
- Harvest solution.cookies and solution.userAgent after a successful
solve. The existing filtering applies unchanged, so the per-check
__ddg8_/__ddg9_/__ddg10_ trio is still dropped and the external path
cannot reintroduce the ?check=1 loop fixed in
|
||
|
|
ebb833a82c |
fix(bypass): discard rejected DDoS-Guard cookies instead of replaying them (#1221)
A cookie that has been rejected was kept and presented again on every
later
request, so a single bad clearance could re-arm the challenge
indefinitely.
Cookie storage:
- Enforce expiry for every stored cookie, not just cf_clearance.
DDoS-Guard
domains have no cf_clearance, so the existing check never fired for them
and
expired cookies were replayed forever.
- Stop storing the per-check cookies __ddg8_/__ddg9_/__ddg10_ and
ddg_last_challenge. Captured live from Anna's Archive, these carry the
client
IP and the timestamp the check was issued (~40 min), versus ~1 year for
the
__ddg1_/__ddg2_/__ddgid_ clearance. Replaying an IP-bound token stops
describing the caller as soon as the egress IP changes, which is routine
behind a VPN.
Failure handling — every path that is rejected while carrying cookies
now
purges them, not just the redirect loop:
- 403 returned while presenting cookies.
- Cached-cookie attempt rejected, whether by status or by redirect loop.
- Factored the purge into _purge_clearance, guarded on a non-empty
hostname
since clear_cf_cookies("") means "every host" and would wipe clearance
for
sites that are working fine.
Also fix the search warm-up switches shipped inert in v1.3.8:
SEARCH_WARMUP_ENABLED and SEARCH_WARMUP_QUERY are not in the settings
registry, and config.get only consults the environment for keys it
knows, so
both always returned their defaults — the warm-up could not be turned
off or
retargeted. Read os.environ first.
Refs #1220. Deliberately not "Fixes": the reported failure could not be
reproduced on v1.3.8 from a stable IP (the reporter's own queries all
returned
200 on both the pre- and post-change builds), and the new purge paths
did not
fire in live testing because the failures arrive as redirect loops,
which were
already purged. These are correctness fixes with no measured effect on
that
issue. The underlying problem remains that Chrome-obtained cookies never
satisfy DDoS-Guard when replayed by requests, so every search still
re-solves.
Verified: 2542 unit tests pass; ruff, basedpyright and vulture clean;
e2e
platform baseline (10), full (6) and bypasser-external (5) all pass;
five
sequential live searches against Anna's Archive all returned 200 with
zero
"Exceeded 30 redirects".
|
||
|
|
b656f019be |
feat(download): add DoH wireformat support, mirror quarantine, and search warmup (#1218)
- Add RFC 8484 DNS wireformat codec and HTTP/2 support (httpx) for Quad9/OpenDNS DoH providers. - Quarantine dead, parked, or seized mirrors for the session on hard failure (DNS errors, connection refused, 410/451, parked pages) while preserving bypass clearance on live mirrors. - Add background startup search warmup to prime DNS, elect mirrors, and pre-solve protection challenges to eliminate cold-start search latency. - Add comprehensive test suites for DoH wireformat, mirror quarantine, parked domain detection, and search warmup. |
||
|
|
3e2a7a48d5 |
fix: clear the DDoS-Guard cookie probe on AA search (#1209)
## Summary Two failure modes on the same code path, both reported this week: Anna's Archive `/search` is gated behind a DDoS-Guard cookie probe that the manual redirect follower can never satisfy. **#1202 — the cookie is dropped on every hop.** AA URLs set `allow_redirects = False`, so `html_get_page` follows redirects by hand. The 302 to `?check=1` carries a `Set-Cookie` (`__ddg*`) that has to come back on the next request. Because cookies are passed per call and `requests` keeps no jar across manual hops, it was discarded each time and the server just re-issued the same redirect until `_MAX_REDIRECTS` raised `TooManyRedirects`. The file already had the right helper — `_new_cookies()` — but only the 503 Z-Library handshake branch called it. **#1204 — the loop never reaches the bypasser.** `TooManyRedirects` isn't in `_is_retryable_error` and carries no status code, so the 403 rescue path (`status == _HTTP_STATUS_FORBIDDEN`) never fired and all attempts repeated the identical failure — ~2.5 min, surfacing as the misleading "Network restricted or mirrors are blocked". These interact, which is why #1202's fix alone isn't enough. Requests merge as `cookies={**handshake_cookies, **cookies}`, so **stale bypasser cookies override the fresh handshake ones** — once `_cf_cookies` holds an expired `__ddg*`, the probe can never clear no matter how faithfully we echo. Hence one search per restart, exactly as #1204 describes. ## Changes 1. Harvest cookies in the same-host redirect branch, the way the 503 branch already does. `_new_cookies()` returns only *new* values, so a server re-sending an identical cookie yields an empty dict and a genuine redirect loop still terminates at `_MAX_REDIRECTS`. 2. Treat a redirect loop as a detected challenge: purge the stored cookies for that host and switch to the bypasser, instead of burning the retry budget. Gated on `allow_bypasser_fallback` and `_is_cf_bypass_enabled()`, and skipped when already bypassing, so AudiobookBay (`allow_bypasser_fallback=False`) and external-bypasser setups are unaffected. The broader point in #1204 stands — the fallback would be better gated on "challenge detected" than on specific status codes, since DDoS-Guard presents at least three faces (403 js-challenge, 429, and this redirect loop). This PR fixes the two live exits without that refactor. ## Tests Two regression tests, both failing before and passing after: - `test_html_get_page_echoes_cookies_across_same_host_redirects` — the fake server only returns results if `__ddg2_` comes back on the `?check=1` hop. - `test_html_get_page_redirect_loop_purges_cookies_and_bypasses` — asserts the stored cookies are cleared, the bypasser runs, and the loop is cut short rather than repeated per attempt. `ruff check` and `ruff format` clean. `tests/download/` passes except `test_download_url_ignores_zlib_cookie_refresh_failure`, which fails identically on unmodified `main` in my environment (no `seleniumbase` — the `browser` extra isn't installed). ## Verification Applied on a live v1.3.7 install (Debian LXC, internal CDP bypasser). Before: every search timed out through 10 retries with `TooManyRedirects`, zero results. After: ``` http.py:455 - Redirect loop detected; switching to bypasser internal_bypasser.py:756 - Bypass successful using _bypass_method_cdp_gui_click internal_bypasser.py:322 - Extracted 9 protection cookies for annas-archive.pk direct_download.py:1865 - Found 24 releases via ISBN ``` ~25 s per search, results render. Note the second search still re-solves the challenge, since the freshly stored cookies go stale immediately — the design issue #1204 raises, left for the broader fix. Fixes #1202 Fixes #1204 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012Ln3yVj3sWHG2c6T78W1we --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
|
0a5256ecbb |
fix(download): reconcile the two AA redirect-loop rescues (#1213)
#1210 and #1212 both added a DDoS-Guard `?check=1` rescue, and #1212 was branched before #1210 landed, so the merged result had two of them with identical guards. #1212's inline handoff returns before the raise that #1210's exception handler keys on, so the handler was shadowed and its stale-cookie purge — the substance of #1210 — never ran. Its regression test has been failing on main since the merge. Fold both into one path: - `_redirect_loop_handoff()` purges the host's stale clearance cookies, then bypasses, so the inline AA handoff and the exception handler cannot drift apart again. - The exception handler keeps its own reason to exist: non-AA hosts run with allow_redirects=True, so `requests` raises the loop itself and the manual AA follower never sees it. It now invokes the bypasser directly rather than setting a flag and continuing, which was a no-op at MAX_RETRY=1 for the same reason the 403 handoff was. - An unrescuable loop returns empty instead of raising TooManyRedirects into the retry path. That error is not retryable and carries no status, so `/dyn/md5/summary` (allow_bypasser_fallback=False) re-ran the full 6-redirect loop on all 10 attempts: 60 requests to AA and ~30s of backoff, measured. Every AA mirror shares the challenge, so there is nothing to rotate to. - `allow_bypasser_fallback` docs now describe what the flag actually gates; the old text predated #1198 and named the wrong callers. |
||
|
|
056ddd372a |
Send DDoS-Guard's ?check=1 redirect loop to the bypasser (#1210)
Fixes #1204. ## Problem #1198 sends a gated AA `/search` to the bypasser when the origin answers 403. DDoS-Guard has a second response: when the clearance cookies from an earlier solve go stale, it serves an endless `?check=1` redirect instead. `requests` follows that until `_raise_too_many_redirects`, and `TooManyRedirects` carries no status code, so `status == _HTTP_STATUS_FORBIDDEN` is false and the rescue never runs. All 10 retries re-send the same dead cookies, then the search fails as `Unable to reach download source. Network restricted or mirrors are blocked.` Direct-download search therefore works once per container start, and stays dead after the stored cookie ages out. v1.3.7 (`sha256:520715f3…`), internal bypasser, mirrors `.gl/.pk/.gd`: ``` 17:04:36 internal_bypasser.py:756 - Bypass successful using _bypass_method_cdp_gui_click ... 17:11:39 http.py:483 - Retry 1/10 for https://annas-archive.gl/search?...&check=1: TooManyRedirects: Too many redirects 17:12:12 http.py:493 - Giving up after 10 attempts 17:12:12 main.py:2870 - Release search failed for source direct_download: Unable to reach download source. Network restricted or mirrors are blocked. ``` The token is short-lived, which is what makes this reachable in normal use: ``` $ curl -sD - 'https://annas-archive.gl/search?...&check=1' HTTP/2 403 server: ddos-guard set-cookie: __ddg8_=…; Expires=Fri, 14-Aug-2026 15:39:38 GMT # issued 15:19:38, 20 min ``` ## Fix Handle the loop like the 403: drop the domain's stored cookies, then retry through the bypasser. The branch sits above the `status ==` ladder because `_get_status_code()` returns `None` for this exception. Cookies are purged only for the internal bypasser; with an external one `get_cf_cookies_for_domain()` already returns `{}`. Related but not changed here: `get_cf_cookies_for_domain()` enforces expiry for `cf_clearance` only, so `__ddg*` cookies are never evicted on age, which is why they go stale. This patch makes the rescue fire whatever the reason the cookies stopped working. ## Verification The regression test drives a real redirect loop through `html_get_page` (302 to `&check=1`, exception raised by the production path rather than faked) and asserts the cookies are purged and the bypasser runs once. - `pytest tests/download/test_http_bypasser_fallbacks.py`: 8 passed. `test_download_url_ignores_zlib_cookie_refresh_failure` fails in my checkout on a missing `seleniumbase`, unrelated to this change. - `ruff check`, `ruff format --check`: clean. - Running in production since 2026-08-14 on v1.3.7 with only this file replaced: six direct-download searches, five served, three books downloaded end to end, against one search per container start before. The rescue mid-download: ``` 19:12:14 http.py:449 - Redirect loop detected; switching to bypasser: https://annas-archive.gl/md5/cb8fba7abae800ddbae1adfb8d7699d9?&check=1 19:12:38 internal_bypasser.py:756 - Bypass successful using _bypass_method_cdp_gui_click 19:14:36 direct_download.py:1142 - Resolved download URL [aa-slow-nowait]: … 19:14:47 orchestrator.py:735 - download finished; starting post-processing ``` ## Separate issue this exposes DDoS-Guard does not accept a solved cookie from plain `requests` traffic, so after this patch the rescue runs for nearly every AA URL. `internal_bypasser.get()` serializes all solves on one module-wide lock and builds a fresh Chrome each time: 11-16 s uncontended, 43-52 s under concurrent load, measured on the host above. Correctness is cheap here, latency is not. Happy to open a separate PR for a warm browser session if that direction is welcome. Co-authored-by: Kukkerem <Kukkerem@users.noreply.github.com> |
||
|
|
03e219eb43 |
Let the bypasser solve bot challenges on Anna's Archive search (#1198)
Anna's Archive put a DDoS-Guard JS challenge in front of /search: the homepage still returns 200, but /search and /md5/<id> answer 403 on every mirror (.gl, .pk, .gd all confirmed). Search fetched both with allow_bypasser_fallback=False, which rotates mirrors on a 403 instead of invoking the bypasser, so it walked the whole mirror list, exhausted it, and surfaced "Unable to reach download source. Network restricted or mirrors are blocked." as a 503 on every query. Adding mirrors could not help — they sit behind the same gate — and neither could USE_CF_BYPASS, since search never reached that branch. Fetch search and the detail page with allow_bypasser_fallback=True so a 403 hands over to the bypasser, which already detects this challenge (DDOS_GUARD_INDICATORS matches the live page). Echoing the __ddg cookies back does not clear it; it needs real JS execution. The download-count fetch keeps allow_bypasser_fallback=False: it is decoration on the details modal and not worth holding the modal open for a browser solve. Fixes #1196 |
||
|
|
e320b7623d |
Fix LOG_LEVEL being ignored and Z-Library 503 cookie gate (#1188)
LOG_LEVEL never reached the app logger: env.py hardcoded the level to DEBUG or INFO, so INFO lines kept appearing under LOG_LEVEL=error. Read it from the env var and advanced settings, normalize unknown values to INFO, and expose it as a setting. entrypoint.sh now normalizes gunicorn's level too, so a typo falls back to info instead of stopping the container from booting. Z-Library gates the first hit on /md5/<hash> with a 503 whose only payload is a Set-Cookie; echoing that cookie back returns the 302 to the real page. html_get_page dropped it and re-ran the same rejected request on every retry, ending in "No download URL resolved". Retry once with the cookies the 503 issued. Fixes #1185 Fixes #1187 |
||
|
|
bb848f05bc |
fix/bypass stall watchdog (#1186)
- Fix protection bypass cancelled by stall detection at exactly 300s - make fixes - Try to fix Synology DELETE issues |
||
|
|
cc1a95f965 |
Fix protection bypass cancelled by stall detection at exactly 300s (#1184)
A download that hits Cloudflare hung on "Bypassing protection..." for five minutes and then died, regardless of which bypasser was configured. html_get_page() started a BypassHeartbeat thread to keep the download marked alive during a bypass, but the thread had no loop: it fired one status event and returned. Even with the loop restored it could not have worked, because update_download_status() dedupes identical (status, message) tuples and returns before refreshing _last_activity, and the heartbeat re-sent the byte-identical payload already emitted just above it. So _last_activity was frozen for the whole bypass, while both bypassers are allowed to run longer than STALL_TIMEOUT (external FlareSolverr ~394s at default settings, internal 420s per get() call). The watchdog always won. From a reporter's log: 403 at 07:04:33.390, cancelled at 07:09:33.987 - exactly 300.000s, and 41s before the bypasser would have finished and reported the real error, an HTTP 500 from FlareSolverr the user never saw. The regression is not one commit. |
||
|
|
816a735cde |
Add a {Language} naming template variable, and consolidate language resolution (#1142)
Fixes #1138 Fixes #1141 ## Problem Two language editions of one book resolve to the same canonical title, so they render to the same path and the second gets a `_1` collision suffix. Audiobookshelf treats a folder as exactly one library item, so the pair becomes a single book with both files as tracks and a summed runtime. Shelfmark already parses and displays the language. It just never reached the template engine. ## `{Language}` template variable A template like `{Author}/{Title}{ (Language)}/{Author} - {Title}` now yields: ``` /library/J K Rowling/Harry Potter (sv)/J K Rowling - Harry Potter.m4b /library/J K Rowling/Harry Potter/J K Rowling - Harry Potter.m4b ``` The untagged edition's path is byte-identical to today, so no existing layout shifts. Three details worth flagging: **The value is casefolded.** On a case-insensitive filesystem `(SV)` and `(sv)` would collapse back into one folder, reintroducing the exact collision being fixed. **Values meaning "we don't know" render nothing** rather than producing `Project Hail Mary (unknown)` folders. Anna's Archive reports that string literally (`direct_download.py`, `language = detected or "unknown"`). **The frontend wasn't sending the release language at all**, so the token would have stayed empty for exactly the audiobook sources in the report. Prowlarr and AudiobookBay do not put language in `extra` the way `direct_download` does, hence the payload plumbing. It reads `release.language`, never `book.language` — the latter is the provider's canonical edition and would mislabel a translation, with a regression test for that specifically. Not gated to audiobooks: Calibre-Web-Automated stages ingested files by basename and discards folder structure, so the rename (filename) template is the only lever those users have. Verified that form works: `J K Rowling - Harry Potter (sv).epub`. ## Language consolidation (#1141) Three release sources each carried their own alias map, all resolving to the same ISO 639-1 codes, alongside a bundled database that only one of them used. Adding a language meant editing three places. Aliases now live in `data/book-languages.json` beside the code and name they belong to, and `shelfmark/core/languages.py` resolves any of them — two-letter code, ISO 639-2 three-letter in either the bibliographic or terminological form, or English name. Prowlarr and AudiobookBay drop their tables. Direct Download keeps its own path-parsing heuristics, including the ambiguous short codes that collide with English words (`de`, `en`, `no`, `in`), and takes only the alias data. This also closes a coverage gap. MyAnonamouse offers 62 languages; Prowlarr mapped 37, and an unmapped code is *dropped* rather than passed through, so the other 25 carried no language at all — leaving `{Language}` empty and the collision unfixed for Latin, Farsi, Tamil, Urdu and the rest. Seven languages MAM offers had no database entry at all: Bosnian, Burmese, Estonian, Icelandic, Manx, Scottish Gaelic, Sanskrit. Also fixes the Traditional Chinese code, which used a U+2011 non-breaking hyphen. Nothing compares against the ASCII spelling today so it was latent, but it would silently defeat the first thing that did. ## Validation Verified end to end against a live Prowlarr and MyAnonamouse, not just unit tests. A real search returning both an English and a Swedish edition, through the actual `queue_release` → `DownloadTask` → naming path: ``` STEP 1 real MAM search -> 37 releases, languages: ['en', 'sv'] STEP 3 queue_release -> task.language='sv' STEP 4 build_metadata_dict -> metadata['Language']='sv' STEP 5 build_library_path -> /library/J K Rowling/Harry Potter (sv)/... two language editions resolve to DIFFERENT folders: True ``` The refactor is pinned by a snapshot of both per-source maps taken *before* they were deleted. All 131 aliases are asserted to still resolve to the same code, one parametrised test each, so a regression names the specific alias. Also verified: the filename-only template, the retry round-trip (`serialize_task_for_retry` → `_restore_task_from_retry_payload`, plus a legacy payload with no `language` key), and placeholder handling. Added a `KNOWN_TOKENS` ordering invariant test — `find_placeholder()` does a substring `.find()` in list order and nothing protected that contract, so a future token in the wrong position could silently shadow an existing one. And a lockstep guard on the frontend, since `KNOWN_TOKENS` is hand-duplicated in TypeScript. **One caveat worth stating.** Three MAM codes are confirmed by observation (`ENG`→`en`, `SWE`→`sv`, `MAL`→`ml`, the last from a real `[MAL / EPUB]` Tagore release). The remaining ~59 are derived from ISO 639-2 rather than observed, because MAM's catalogue is overwhelmingly English — enabling 27 extra languages still yielded only one non-English hit across 258 results. Mitigated rather than closed: both 639-2 variants are present for every language where they differ, and a wrong alias is an unused entry while a missing one loses the language. Happy to correct any code a maintainer knows differs. ## Test results 2056 Python tests pass (up from 1906). Frontend typecheck, lint, format and 126 unit tests pass. Pre-existing failures on my machine, unchanged by this branch and unrelated: `tests/bypass/` needs `seleniumbase`, and `tests/config/test_entrypoint_permissions.py` uses bash-4 syntax that macOS bash 3.2 rejects. --------- Co-authored-by: delize <4028612+delize@users.noreply.github.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
|
f8d3f990ca |
fix: Prowlarr expired download link refresh (#1109)
## Summary Fixes Prowlarr downloads that fail after a queued torrent result’s tracker download link expires. Prowlarr torrent results can expose a `downloadUrl` that is only a short-lived proxy to the upstream tracker. Some trackers, including MAM, include expiring credentials in that URL. Shelfmark was persisting that URL as retry data and later treating it as durable. If the in-memory Prowlarr cache was gone, or if qBittorrent tried to add a stale URL, Shelfmark could fail with a misleading torrent-hash error instead of refreshing the release. ## What changed - Stop persisting Prowlarr `downloadUrl` values as durable retry data. - Persist only source context needed to refresh the release later. - On a Prowlarr cache miss, re-query Prowlarr using the queued task context. - Accept refreshed results only when the stable identity matches the original release: - `guid == task.task_id` - or `infoUrl == task.task_id` - Cache the fresh raw Prowlarr result and build the download request from its current `downloadUrl` / `magnetUrl`. - If qBittorrent add fails with `Could not determine torrent hash from URL`, remove the stale cached Prowlarr result, refresh once, and retry with the fresh URL. - Preserve existing magnet handling: torrent results continue to use `magnetUrl` first, so the refresh path only targets Prowlarr proxy `downloadUrl` values that can expire. - Improve the user-facing failure when refresh cannot find the same release: `The indexer download link expired and the release could not be refreshed. Search again for a fresh result.` ## Why this approach The important constraint is avoiding accidental downloads of a different edition or format after the original tracker link expires. Re-running a search by title can return many plausible results, so the refresh path deliberately requires an exact stable identity match before using any new URL. This treats Prowlarr `downloadUrl` as a short-lived hint, while still allowing retries to recover when Prowlarr can find the same release again. The one-shot retry after qBittorrent add failure handles the case where Shelfmark still has a cached Prowlarr result, but that cached result contains an expired proxy URL. The refresh hook is source-specific and defaults to no-op for other external download handlers, so Newznab and other sources keep their existing retry behavior. ## Bug Fixes #1012 Context: https://github.com/calibrain/shelfmark/issues/1012#issuecomment-4917148398 ## Validation - `uv run pytest tests/prowlarr -q` - `uv run pytest tests/download/test_orchestrator_user_output_mode.py -q` - `uv run pytest tests/newznab/test_handler.py tests/audiobookbay/test_handler.py -q` - `uv run ruff check shelfmark/core/models.py shelfmark/download/orchestrator.py shelfmark/download/clients/base_handler.py shelfmark/release_sources/prowlarr/handler.py tests/prowlarr/test_handler.py tests/prowlarr/test_integration_handler.py tests/prowlarr/test_failure_scenarios.py tests/prowlarr/test_source.py tests/download/test_orchestrator_user_output_mode.py` - after resolving conflicts with latest `main`: `uv run pytest tests/prowlarr -q` - after resolving conflicts with latest `main`: `uv run ruff check shelfmark/release_sources/prowlarr/handler.py` - after resolving conflicts with latest `main`: `uv run pytest tests/prowlarr/test_handler.py tests/prowlarr/test_source.py -q` Co-authored-by: Aidan Abbott <aidanabbott@Aidans-MacBook-Pro.local> |
||
|
|
1c19326bd0 | e2e testing 2 (#1083) |