mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-11-16 04:28:16 +00:00
8fd477bf50cda3003fbb9e543d9127e44d1667ac
275
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
8fd477bf50 |
fix(releases): apply the request policy to release listing, not only to download (#1506)
The request policy is applied when a user downloads (`/api/releases/download`) or requests (`/api/requests`), but not when they list releases. A user whose mode is `request_book` or `blocked` can still search releases and list a torrent's files through the shared Prowlarr and qBittorrent credentials. Only the UI hides the list. This applies the same gate to `/api/releases`, `/api/releases/inspect` and the release-source records route, with the same codes as the download route (`policy_blocked`, `policy_requires_request`, `required_mode`). - `blocked` and `request_book` are refused. `request_release` is allowed, because that mode lists releases so the user can pick one. The UI opens the request dialog directly under `request_book` and never calls `/api/releases` there. - `/api/releases` resolves the mode per source for the content type. A refused source is dropped from the search, so a user blocked on Prowlarr no longer reaches Prowlarr even when other sources are allowed. It answers 403 only when no source is left, or when the one source asked for is refused. - The records route doesn't know the content type before it fetches, so it refuses only when every content type the source supports is refused. - Admins, auth mode `none` and an instance with requests off bypass the gate, as they do on the download route. A user whose default is `blocked` now gets a 403 toast in the release modal, where they used to get a list of "Unavailable" rows. I made no frontend change. Tests (14, new file) cover each route for blocked, request_book, request_release, an admin, and auth mode none, the per-source drop, and the records route. Ruff, basedpyright and the backend suite pass. Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
|
b05bbc8c58 |
fix(covers): key the cover cache by the image URL, not the caller-supplied id (#1505)
`/api/covers/<id>?url=...` cached the fetched image under the id in the path, and the caller chooses that id. A request with another book's id and a different URL put its image in that book's cache entry, and every user then got it. The key is now derived on the server from the image URL (a hash of it), and the id in the path stays only so links keep their shape. The SSRF checks and `fetch_and_cache` are unchanged, and `transform_cover_url` links already carry the URL, so they keep working. The URL is decoded first now, so a missing one is a 404 and bad encoding a 400, both before any fetch. I made the key the URL alone, not URL plus id. Either stops the overwrite, and the URL alone lets one image be shared when the same cover appears under two ids (the viewer redaction rewrites ids and still hits the cache). Entries cached under the old keys are not read any more, since there is no telling which were overwritten. They age out of the LRU and the cache warms once after the upgrade. Tests: a request with another id can't replace the legitimate image, in either order; the normal miss then hit; the same URL under two ids hits; no URL gives 404; bad encoding gives 400 without a fetch; and a unit test for the key. Ruff, basedpyright and the backend suite pass. |
||
|
|
71cd030484 |
Merge commit from fork
GET /api/users/me/edit-context served every user-overridable key with the admin's global value in globalValues and effective, so any user could read the global destination paths, the admin's EMAIL_RECIPIENT, the Booklore library and path ids and the global USER_NOTIFICATION_ROUTES table, whose Apprise URLs embed tokens. build_user_preferences_payload gains a for_self_service mode that replaces a global value with None unless the user set the key themselves or the field is a checkbox, number, select or multi-select switch the form needs to lay itself out (BOOKLORE_LIBRARY_ID and BOOKLORE_PATH_ID are masked despite being selects). The self-service route uses it; the admin routes keep full data. |
||
|
|
386fdb313c |
Merge commit from fork
GET /api/debug returns a bundle with application logs, process listings and environment names, and GET /api/restart exits the process. Both are registered only when DEBUG is on, but any logged-in user could call them. Add an admin_required decorator built on login_required and use it for both routes. Non-admin sessions get 403 "Admin access required"; authentication mode "none" keeps working as before. |
||
|
|
7da4669cc1 |
feat(prowlarr): hide ebook results above a maximum size (#1504)
Closes #844. A new Prowlarr setting, Maximum Ebook Size (MB). Ebook results above it are hidden from the release list. The default is 0, which shows everything as it does today. This is the idea from the issue. The filetype check suggested in the comments is a different change (#1490 has one for Prowlarr), and it doesn't catch a TV series or a software bundle that ships as a plain archive, which is what a size limit does. - A result with no reported size is always shown. - A result exactly at the limit is shown. - Audiobook searches are never limited, since they are big by nature. - A non-numeric value counts as no limit. The filter runs on the raw Prowlarr result before it is turned into a release, so a hidden result never reaches the sort or the release cache. Tests cover no limit, zero, over the limit, exactly at it, no size, audiobook searches and a bad value. Ruff, basedpyright and the backend suite pass. I added the entry to `docs/environment-variables.md` by hand because running the generator also changes unrelated parts of that file. |
||
|
|
f2269aa201 |
feat(audiobookbay): seed time and ratio settings (#1503)
Closes #1037. Two new AudiobookBay settings, Seed Time (minutes) and Seed Ratio. Both default to 0, which leaves the torrent client's own limit as it is today, so nothing changes until someone sets them. They go to the client with the magnet the same way the Prowlarr seed preferences do (`DownloadRequest.seeding_time_limit` and `ratio_limit`), so they apply with the clients that already take per-torrent limits there: qBittorrent, Transmission and Deluge. A retry that reuses the saved magnet carries them too. A value that isn't a positive number (text, negative, NaN, inf) is ignored and the client's limit stays. Tests cover reading both settings (unset, zero, each on its own, text values, bad values) and that both the fresh and the cached-magnet request carry them. Ruff, basedpyright and the backend suite pass. I haven't tried it against a live client, since the client side is the code Prowlarr already uses. I added the two entries to `docs/environment-variables.md` by hand. Running `scripts/generate_env_docs.py` here also changes unrelated parts of that file (the Soulseek table of contents link and the `SHELFMARK_API_KEY_READONLY` entry), so I left those alone. |
||
|
|
77b80eb31a |
fix(metadata): only an admin may use the provider's own account, its lists and shelves (#1501)
Refs #981. Hardcover is signed in as one account, the instance owner's, and every logged-in user shares it. A non-admin can see the owner's lists and shelves in the List filter, browse one by asking for it, and add or remove books and change read status from search results, which is what #981 describes. Now only an admin may. With authentication set to none the one user keeps everything. - `GET` targets and the batch lookup return no options for a non-admin, so the buttons don't show and the account isn't asked. - `PUT` targets answers 403. - The List filter (`hardcover_list`) has no options for a non-admin, and a request that asks for it has the field dropped before the search. Providers name these fields in `account_search_fields`, and Hardcover lists `hardcover_list`. - Public things are unchanged: author, title and series suggestions, and a pasted list URL in the search box. That pasted URL is the one gap I know of. It resolves any list the account can see, so a user who knows the slug of one of the owner's private lists could still open it. Closing it means the provider has to know who is asking, which is a bigger change than this one, so I left it. Not included from the issue: a toggle for letting users view lists and limiting which lists they see. I kept this to the admin-only part so it stays small, and I can do the toggle as a follow-up if you want it. Tests cover the routes as an admin, a user and with no authentication (targets read, batch and write, list options, a crafted list search, and public fields still passing). Ruff, basedpyright and the backend suite pass. Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
|
82c8c91c8b |
refactor(clients): read the stall window with a type check instead of a pyright ignore (#1498)
The only `pyright: ignore[reportArgumentType]` in `shelfmark/` is one I added in #1426, and I was being lazy when I did. I hit the same error on another PR and replaced the ignore there with a real check, so this does the same for the original. `_stall_window_seconds` reads the setting as an `object`. It now narrows to int, float or str before calling `float()`, and a bad string falls back to the default as before. A non-numeric value of another type (None, a list) falls back to the default too, which used to happen through a `TypeError`. No behaviour change. The existing range test still passes and I added `None` and a list to it. basedpyright is at 0 errors, ruff is clean and the backend suite passes. Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
|
e83bab9337 |
feat(clients): give up on a torrent with no seeders (#1494)
A torrent with no seeds won't finish. It runs on the pieces its few
peers hold and then sits still, and nothing says so until the stall
timer ends it. The card only reads "downloading". I run AudiobookBay
releases, which show no seed count in the listing, so the client is the
only place to find out.
Two new settings, both off until you turn them on:
`TORRENT_NO_SEEDERS_WAIT` (minutes, 0 turns it off) and
`TORRENT_MIN_SEEDERS` (default 1).
- A torrent that has fewer seeds than the minimum for the whole wait,
while nothing is downloading, is removed from the client with its data
and the task ends as an error.
- A torrent with no seed at all for the wait is given up on even while
it still moves.
- A torrent that met the minimum and lost seeds down to one, but is
still moving, keeps going.
- The card shows the seed count beside the progress, with a countdown
while it waits ("0 seeds, giving up in 4m").
qBittorrent only, because it reports connected and tracker seed counts.
Other clients report none and are left to the stall timer, and an
unknown count is not read as zero. A torrent that was already in the
client when the download started is never removed, and a queued torrent
isn't judged.
Tests cover the wait and countdown, the clock restarting on progress,
the tracker count, a higher minimum, off by default, a client with no
counts, a preexisting torrent, a queued torrent, and the moving cases.
Ruff and the backend suite pass.
|
||
|
|
9ba5816b83 |
fix(requests): an older failed attempt no longer undoes a later delivery (#1493)
A request that was retried after a failure has two tasks in the status built from download history. `sync_delivery_states_from_queue_status` walks the status buckets in `QueueStatus` order and keeps the last one it sees for a request. Error comes after complete, so a request that was delivered by its retry is set back to error and, when no retry is left, reopened as pending. The next pass can then fetch the book again. When a request has more than one task, an attempt that is still running now decides its state, else a completed one, else the failure. Ties keep the old order, so a request with a single failed task still reopens as before. Tests: an earlier failure doesn't undo a later delivery, a running retry outranks an earlier failure, and a lone failure still reopens. It happened on my instance to a request whose first try failed and whose retry succeeded. Ruff and the backend suite pass. |
||
|
|
a1ac017487 |
feat(requests): let an admin reopen a rejected request (#1491)
Closes #1445, where you said you were open to it. A rejected request was final. `update_request` won't move a terminal status and the admin routes only fulfil or reject, so an admin who changed their mind had to ask the user to request the book again, and the user kept seeing the old rejected one. `POST /api/admin/requests/<id>/reopen` moves a rejected request back to pending and clears the review fields. What the user asked for (request level, release, note) is kept. It refuses anything that isn't rejected, and refuses when that user already has a pending request for the same book. Admins get a Reopen action on rejected cards in the activity sidebar and in History. A rejected request has usually been cleared from the list by the time anyone wants it back, so reopening also clears the dismissal, the way the failed-download reopen does. Without that the pending request stays hidden. Tests cover the service (state change, what is kept, the refusals, a non-string admin note) and the route, and the card model for the new action. Ruff, tsc, oxlint, vitest and the backend suite pass. It has been running on my instance since Oct 8. |
||
|
|
88cc734db2 |
fix(activity): a reused release follows the new request, and stale rows of a delivered request are closed (#1496)
Two things that go wrong when a second request picks a release a first request already used. A task id comes from the release, so the history row is reused. **The row stays with the first request.** Recording a download kept the first request's id, notes and state. On my instance a card for one book showed up under the request for a different book, with that request's note on it. Recording a download now moves the row to the request that queued it and restarts its queue time. A retry with no request keeps the request the row had. **Old rows of a delivered request show as failures forever.** The startup sweep leaves an old request-linked row alone so its retry stays, but one whose request already has its book (delivered, or another download of it finished) is now closed as "Superseded by a completed download" instead of showing as an interrupted failure for ever. Also in the first commit: History cards used the saved cover URL as it was, an external address the page's own-origin image policy blocks, while running downloads use the cover proxy. They use it too. The second commit is about restarts. After a restart nothing is in flight, but a request in downloading, resolving or locating whose row had moved to another request was stranded, because only queued was reopened. Those states are reopened too. Tests cover the row takeover and the retry case, the cover through the proxy (and a missing cover), the superseded cases (delivered request, another row finished, an undelivered request is left alone) and the in-flight states with no row. Ruff and the backend suite pass. |
||
|
|
06a67a7150 |
feat(download): a concurrency limit per source (#1495)
With one global limit, a few tasks that are only waiting hold every slot. A Soulseek download sitting in a peer's queue is the example I hit: three of those and a torrent that could start waits behind them. `MAX_CONCURRENT_DOWNLOADS_PER_SOURCE` (0 keeps the single global limit, which is the default). With it set, the coordinator starts a download only when its source is below the limit and passes over tasks of a full source without losing their place in the queue. `get_next` gets an optional filter for this, and a cancelled task is still dropped when one is given. The start-up stagger now applies only between downloads from the same source, and the pool is sized so every source can reach its limit. The second commit: qBittorrent keeps its own queue of torrents waiting for a free slot. The task polling one of those does no work, so it no longer counts against its source's limit and the next task of that source can start. The client does the pacing, as it does for its own queue. Tests cover the queue pass-over (order and priority kept, nothing eligible, cancelled tasks), the coordinator (a full source doesn't block the others, each source has its own limit, the global limit when unset) and the client-queue case. Ruff and the backend suite pass. |
||
|
|
3b5afb1a04 |
Merge commit from fork
PUT /api/users/me set a new password from the password field alone, so a live session could replace the password and lock the owner out. A builtin user now has to send current_password with it, checked against the stored hash. A wrong or missing one is a 400, and counts toward the same lockout as a failed login (a locked account gets a 429 without being checked), so the endpoint cannot be used to guess the password. A password or current_password that is not a string is a 400. A password that was not a string raised a TypeError from len() and answered 500. The account form asks for the current password when a new one is typed. Profile and settings changes without a password are unchanged. |
||
|
|
10710abd4e |
fix(download): send every file of a multi-file download to the browser (#1489)
Download to Browser only ever sent the first file of a multi-file release (audiobook chapters, packs): the folder output returned final_paths[0] and /api/localdownload served that one path. The folder output now keeps every final path on the in-memory task (download_paths), and /api/localdownload streams them as one uncompressed ZIP named after the book when there is more than one. The archive is never staged on disk or held in memory, and ZIP64 covers large audiobooks. History still stores only the first path, so a re-download from history after a dismiss or restart serves a single file, as before. Also brings every linter back to green on all files: - pre-commit: bump the ruff hook to v0.17.1 to match uv.lock; ruff-format the Python examples in docs/dev and skills - ruff: fix the remaining issues in skills/shelfmark/download_books.py; format base_handler.py - basedpyright: Optional and attrs errors in databazeknih.py - knip: drop 31 unused exports, reuse ColumnColorHint in colorMaps - oxfmt 0.72: styles.css - direct download: fix the "Welib, mirrors" typo and the test that still expected the pre-OceanofPDF message Fixes #1467 |
||
|
|
c8d66db22e |
feat(direct_download): add OceanofPDF as a standalone search and download provider (#1485)
Closes #1149 ## What this adds OceanofPDF as a second Direct Download provider, alongside Anna's Archive. Unlike Z-Library, LibGen and Welib, which only download files Anna's Archive found, OceanofPDF searches and downloads on its own. It works with or without an AA mirror configured; with both, one search queries both and the results are merged. It plugs into the provider registry from #1337, so search, record routing and the download handler need no changes. **Search:** WordPress `/?s=` search, "title author" first, then the bare title if that finds nothing. One release per advertised format (`[PDF] [EPUB]`), with title, author and cover. Shelfmark's supported-format filter applies as usual. **Download** follows the site's three hops: 1. The book page has one POST form per format, targeting `Fetching_Resource.php`. 2. That returns an interstitial whose `<meta http-equiv="refresh">` points at a signed, expiring file URL. 3. The file URL serves the book. Pages go through `html_get_page` and its Cloudflare bypass. The form POST replays the clearance the bypasser stored, and re-solves once if challenged. Bypassers only replay GETs, so it can't be handed over directly. **Settings:** - `OCEANOFPDF_MIRROR_URLS` under Mirrors. Only the first URL is used. - An OceanofPDF toggle under Download Sources → Slow downloads. It's locked until a mirror is set and while the Cloudflare bypass is off. Its position in the list doesn't matter, and its description says so. - A saved list from before this change has no OceanofPDF entry. The settings page shows a missing option as off, so the provider treats it as off too, keeping the page and the behaviour in agreement. New installs default to on. ## Fixes found along the way - **`challenge_marker` rejected real pages behind Cloudflare** (separate commit, `fix(bypass)`). Cloudflare's bot-detection script injects `/cdn-cgi/challenge-platform/scripts/jsd/main.js` into every page it serves, after load. Bypassers return the rendered page, so a successful FlareSolverr solve of any page under 64 KB came back carrying `/cdn-cgi/challenge-platform` and was rejected as unsolved. The detector now ignores that one script path. Real challenge pages load from `/cdn-cgi/challenge-platform/h/…` and keep their other markers, so they are still caught. This also affects any other Cloudflare-fronted host, not just OceanofPDF. Happy to split it into its own PR if you prefer. - **Brotli on a plain `requests` call.** Copying `DOWNLOAD_HEADERS` into the form POST offered `br`. Cloudflare then answered in Brotli, which `requests` can't decode without the optional `brotli` package. The POST now leaves `Accept-Encoding` to `requests`. If a response still has no file link, its status, encoding, title and challenge marker are logged. ## Testing - **New tests:** 41 in `tests/direct_download/test_oceanofpdf.py` and 4 in `tests/bypass/test_cloudflare_beacon.py`. They cover search parsing, format and language filtering, form selection, the interstitial, the download flow (including a challenged POST, an HTML stub served as the file, and cancellation), the toggle, and registry wiring. Fixtures mirror the live site's markup. - **Existing tests:** `tests/direct_download`, `tests/bypass`, `tests/config`, `tests/core` and `tests/download` show no new failures against `main`. Some tests in those folders can't run on Windows; those fail identically on both. - **Lint:** `ruff check` and `ruff format --check` are clean on every changed file. - **Live:** verified end to end on a v1.4.0 Docker deployment behind FlareSolverr (external bypasser), with only an OceanofPDF mirror configured: search, then the book page, the form POST and the completed download. ## Known limits - OceanofPDF doesn't label languages, so its results always pass a language filter. - PDFs only appear when `pdf` is in the supported formats. - Each query reads the first results page only (roughly 7–10 books). - Downloaded files carry OceanofPDF's own tags/watermark, as noted in #1149. They aren't stripped. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Adam Stone <196691089+ast0ne1@users.noreply.github.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
|
51f32b374d |
feat(packs): let users choose which files, books and formats to download (#1480)
## Summary - **Skip files or books in a multi-book pack.** The pack review panel gets a switch per file and a "Skip this book" switch per book, using the shared `ToggleSwitch`. A partly skipped book shows e.g. "3 of 5 files" in the Files column. A fully skipped book is greyed out. - **Choose formats for a single book.** A new "Choose formats" header toggle, next to the multi-book one, opens a format picker for a release that has several book formats (EPUB, MOBI, PDF and so on). It has one switch per format with file count and size. Releases with one format, or that can't be inspected, download as before. - **Skipped files are sent as `ignored_files`.** They are stored on the task (including restart-safe retry) and removed before grouping and transfer. This matters for packs because files missing from `book_plan` would otherwise be filed by the heuristic fallback. Ignored entries match by release-relative path, then path suffix, then unique basename. - A pack that ends up with one book after skips keeps the titles you reviewed instead of falling back to the searched title. ## Screenshots <img width="1277" height="413" alt="Screenshot 2026-10-19 at 09-39-28 Shelfmark" src="https://github.com/user-attachments/assets/c005ca4b-3dac-45dd-b2ff-d02fc0f9fcc9" /> <img width="1277" height="1210" alt="Screenshot 2026-10-19 at 09-40-23 Shelfmark" src="https://github.com/user-attachments/assets/2145f3ba-e5fc-4635-9364-1790feb0c8b8" /> <img width="1277" height="748" alt="Screenshot 2026-10-19 at 09-39-03 Shelfmark" src="https://github.com/user-attachments/assets/39023fac-70e3-46d8-af89-1160ebd6e415" /> ## Test plan - [x] `pytest` pack tests (new `drop_ignored_files` cases added) - [x] `vitest`, `tsc`, `oxlint`, `oxfmt`, `ruff` - [x] Manual check in the UI with a real multi-book pack 🤖 (mostly) Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
|
3c5cfcb77d |
fix/redact log secrets (#1488)
- fix(logging): mask tracker API keys in log output (#1486) - fix(welib): follow slow_download links through the countdown (#1447) |
||
|
|
bb1655238a |
feat(slskd): add Soulseek (slskd) release source, building on #1335 (#1428)
## Add Soulseek (slskd) as a release source, building on #1335 This is @dskvr's work from #1335 rebased onto current `main`, with my changes on top. #1335 has been quiet since 2026-09-13 and I needed the changes below, so I carried it forward instead of starting over. Their two commits are the first two in this branch with their authorship intact, and the source plugin, handler, API client, settings tab, tests, `ReleaseProtocol.SOULSEEK`, test client compose file and frontend bits are all theirs. I haven't changed how a search turns into releases beyond what is listed here. If @dskvr would rather keep #1335 alive and take these as commits on their branch, I can do that instead and close this. ### What I changed **Downloads survive other apps using the same slskd.** The handler found its files by listing slskd's downloads, and another app on the same instance (SoulSync here) sends `DELETE /transfers/downloads/all/completed` after each of its batches. Once that ran, a finished download no longer appeared in the list and the task never saw it complete, even though the files were on disk. With the new `SLSKD_ISOLATE_DOWNLOADS` setting (on by default) each task: - enqueues one batch (`POST /transfers/downloads/batches`) into its own folder, `<SLSKD_DESTINATION_PREFIX>/<task id>` - polls each transfer by id, which slskd still answers after the list has been cleared - decides it is done from the files on disk (sizes, not names), with the `/events` log as a fallback when the transfers are gone and the files aren't visible yet - copies out of that folder, and removes it only if `SLSKD_KEEP_COMPLETED` is off and the import worked With isolation off the handler behaves as it did in #1335. If the batch endpoint isn't there (404/405) it falls back to the original per-file enqueue. **Source changes.** m4b/m4a files are their own release next to loose chapter files, so auto-download can prefer one file over a chapter set. Results from the query variants are merged per peer. The folder text is exposed as `extra["author"]` and `extra["title_raw"]`, which is what auto-download's strict match and pack detection read. `is_available()` also needs an API key now. **Smaller fixes.** Cancelling while files are still being located now removes the transfer records it created. In the settings tab, a cleared form field is no longer replaced by the saved value when testing the connection. The new settings are in `docs/environment-variables.md`. ### Testing `tests/slskd` has 192 passing and 6 live tests that skip without an instance. I wrote the new tests first and checked the handler and source rules by breaking them and confirming a test failed. Ruff is clean. On the whole suite the only failures are 4 in `tests/download` that import `seleniumbase`, which isn't installed in my environment, and they fail the same on `main`. I've been running this on my own deployment against slskd 0.26.0 behind a VPN, with SoulSync active on a second instance. A search returned a couple hundred results from about 70 peers, an epub came down and was imported, and auto-download fetched five audiobooks as single m4b files. That is a day of use, so I don't know how it behaves on other slskd versions or on Windows path mappings. --------- Co-authored-by: sandwich <dskvr@users.noreply.github.com> Co-authored-by: sandwich <299465+dskvr@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
|
b38e4e4a00 |
feat(naming): add {AuthorSort} template variable (#1484)
Closes #1178. Adds `{AuthorSort}`, the first author surname-first ("Arthur Conan Doyle" → "Doyle, Arthur Conan"), for rename and organize templates, alongside `{FirstAuthor}`. - A value already in "Last, First" form (one word before the comma) is kept as is; Jr/Sr/II/III/IV stay at the end. - Known limits, same as calibre's default: the last word is treated as the surname, so particles ("Le Guin") and surname-first names aren't detected. - Added to the settings help, env docs, the frontend variable picker/preview, and tests. |
||
|
|
b83e35ee0a |
feat(metadata): add Databáze knih metadata provider (#1479)
## What Adds [Databáze knih](https://www.databazeknih.cz), the Czech and Slovak community book catalog, as a metadata provider. It follows the Moly.hu provider (#1172): HTML scraping, no API key, disabled by default, and a settings tab with an enable checkbox and a Test Connection button. ## Why Hardcover, Open Library and Google Books cover Czech and Slovak editions poorly. Databáze knih is the de facto catalog for them, including Czech and Slovak translations of foreign works. With this provider, Universal mode works end to end for Czech and Slovak titles: the localized title and author from Databáze knih feed the release search. ## How - HTML scraping with BeautifulSoup, which is already a dependency. Book pages carry schema.org JSON-LD, so most fields come from structured data rather than CSS selectors. - **Title search** uses `/vyhledavani/knihy`, with the site's pagination (`page`) and language filter (`lang` for cs/sk/en/de). A search with a single match redirects straight to the book page, and that case is handled. - **Author search:** the site search only matches titles, so an author-only search finds the author (accent- and word-order-insensitive) and lists their books page, which is also paginated. Title + author filters the title results on the author. - **ISBN search** follows the site's redirect from `/search?q=<isbn>` to the book page. - **`get_book`** reads the full description (the JSON-LD one is truncated), ISBN-10/13, publisher, edition year, language, genres, cover (placeholder images dropped), series with id and position, and rating and rating count. It also reads the page count and original title from the more-info fragment. - The `RateLimiter` from Open Library (30 requests/min) and the standard `@cacheable` decorators. Fetch failures return `None`, so they are not cached. - `docs/environment-variables.md`: added the `DATABAZEKNIH_ENABLED` section. ## Testing - `tests/metadata/test_databazeknih_parse.py`: offline tests with shortened copies of real pages. They cover search parsing, pagination and language parameters, redirect-to-book, author listing, title + author filtering, uncached failures, book details, ISBN resolution and helpers. - `uv run pytest tests/metadata` is green (89 passed), and `ruff check` and `ruff format --check` are clean. - Verified live against databazeknih.cz: connection test, title search (pages 1 and 2), author books, title + author, `get_book`, and ISBN lookup. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a0ffb268b2 |
fix: wait for synced files before processing completed downloads (#1464)
## Summary
```text
wait for completed download path
regular file -> keep existing immediate handling
directory -> inspect files enabled for this task's content type
wait until the supported-file signature is stable across checks
```
Fixes the remote-sync race in #1463. A mapped torrent folder can appear
before its large media file finishes copying, so path existence alone
does not mean the download is ready.
## Evidence
- **Before:** In #1463, qBittorrent reported a multi-file audiobook
complete while the destination contained only `.cue` and `.jpg`;
Shelfmark immediately post-processed the folder and failed with “No book
files found in download.” Retrying after the `.m4b` sync finished
imported it.
- **After:** A regression test starts with the cue sheet and cover
present, then simulates the `.m4b` file appearing and growing. Path
waiting returns only after the supported file's size and modification
time remain stable across checks.
- Credential-free isolated `python -m pytest
tests/prowlarr/test_remote_path_mappings.py -q -n 0 -o addopts=''`: 11
passed.
- `ruff check shelfmark/download/clients/base_handler.py
tests/prowlarr/test_remote_path_mappings.py`: passed.
## Merge Danger
**Door:** Two-way
**Blast Radius:** Download handling
The wait uses the existing completed-path timeout and leaves regular
single-file downloads on their current path.
Co-authored-by: Sean Li <22557931+sean0407@users.noreply.github.com>
|
||
|
|
0ade063db5 |
fix(download): extract RAR archives in place, not in the IO threadpool (#1459)
Fixes #1458. A RAR download fails with `Download failed: TypeError`, and the log says `child watchers are only available on the default loop`. rarfile extracts by running `unrar` as a subprocess, and `extract_archive_files` calls it through `run_blocking_io`, so it runs in gevent's IO threadpool. gevent only lets a subprocess start on the thread that owns the default loop. `run_blocking_io` already avoids the threadpool for `subprocess.run`, but a RAR gets to the subprocess through rarfile, so nothing catches it. This extracts a RAR on the calling greenlet, where the patched subprocess waits for `unrar` without blocking anything else. ZIP is pure Python and stays in the threadpool. I checked it with a real RAR4 file on a build of main with the stock image's `unrar`: from the threadpool it raises the TypeError, in place it extracts all 7 files. Two tests. One checks that a RAR doesn't go through `run_blocking_io`. The other runs under real gevent patching with an extractor that starts a process the way rarfile does. Both fail without the change, and the full suite passes. I keyed it on the `.rar` suffix to keep it small. Teaching `run_blocking_io` about rarfile's subprocess would be the more general fix, but that's a bigger change to a helper a lot of code goes through, so I left it for you to say if you want it. |
||
|
|
5f3647a496 |
fix(clients): wait for qBittorrent-compatible clients to finish before post-processing (#1465)
## Summary
`QBittorrentClient.get_status()` treats a torrent as complete as soon as
`progress >= 1.0`, excluding only the `moving` state. Some
qBittorrent-compatible clients report full progress before they are
finished. rdt-client (v2.0.142) reports `progress = 1.0` once the debrid
and local downloads are done, but keeps `state = "downloading"` for a
few seconds while it finalises the torrent, and until then reports a
guessed `content_path`. It only switches to `pausedUP` with the real
path afterwards.
Shelfmark picks the torrent up inside that window, so post-processing
runs against a path that isn't there yet and fails within milliseconds
("No book files found" / `post-processing failed`). Clicking Retry then
imports it fine. Every rdt-client download hits this.
## Change
- Add `_INCOMPLETE_STATES` (all download-side states, plus `moving`) and
only mark a torrent complete when `progress >= 1.0` **and** its state is
not one of those.
- Real qBittorrent leaves these states when a torrent reaches 100%, so
its behaviour is unchanged.
## Example log (before)
```
12:04:43,215 base_handler - Remapped download path for qbittorrent (<hash>): <rdt-client download dir>\<torrent name>\ -> /downloads/<torrent name>
12:04:43,216 orchestrator - download finished; starting post-processing
12:04:43,232 orchestrator - WARNING - post-processing failed
```
## Testing
- New test
`test_get_status_full_progress_still_downloading_not_complete`: fails
before the change, passes after.
- `tests/prowlarr/`: all passing. `ruff check`, `ruff format --check`
and `basedpyright` clean on the changed files.
- Running the patched client against rdt-client: downloads now import
without Retry.
Related to #1463 / #1464, which handle the same symptom when the client
really is finished but files are still syncing to local disk. This PR
addresses a different cause: the client reports 100% progress while
still in a downloading state, so Shelfmark shouldn't treat it as
complete yet. The two changes are independent and complementary. Not
covered by #741, which fixed add-time failures. Possibly related: #607.
|
||
|
|
d3bcf0674d |
fix(direct_download): Anna's Archive search 404 on ISBN queries (#1456)
When ISBN filters are present, the search query is now the deduplicated bare ISBN strings, with the free-text query appended only if it is not already one of those tokens. Searches with no ISBN still send the quoted free-text query alone. An Anna's Archive lookup that includes an ISBN comes back as 404 Not Found, so the direct-download release search fails and the book cannot be retrieved. The failing request puts an isbn13:/isbn10: boolean expression in the q parameter. The direct-download search query tests passed. Closes #1444 Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> |
||
|
|
84d1d7196c |
fix(irc): pin the TLS context to TLS 1.2 or newer (#1455)
CodeQL on my fork flagged `ssl.create_default_context()` in the IRC client as allowing TLS 1.0 and 1.1 (`py/insecure-protocol`, rated high). In practice the shipped image is already fine. Python 3.10+ sets a TLS 1.2 floor on the default context, and on the current image (Python 3.14.7, OpenSSL 3.5.7) `minimum_version` comes back as TLS 1.2. An OpenSSL config on the host can still lower that floor, so this pins it with one line, `context.minimum_version = ssl.TLSVersion.TLSv1_2`. Added `tests/irc/test_client_tls.py`. It drops the floor to the minimum before `connect()` runs, then checks that `connect()` raised it back to 1.2. It fails without the change. Certificate and hostname verification are still off (`CERT_NONE`). I left that alone because of the comment about self-signed certs on IRC servers. Turning verification on, or making it a setting, would be the bigger fix, but that is ultimately a call for @calibrain if you want a PR for it. |
||
|
|
2b8e0da2f3 |
fix(direct-download): clarify required search mirror (#1454)
Direct Download now tells users who configure only a Z-Library mirror that they need an Anna's Archive mirror for search. Z-Library, LibGen, and Welib mirrors download results found through Anna's Archive. Fixes #1408. Validation: the Direct mode browser check, 18 focused tests, Ruff, formatting, and backend/test type checks passed. The CI-style Python suite passed 3,569 tests; `tests/core/test_hardlink.py::TestSameFilesystem::test_string_paths` was excluded because this container reports different `st_dev` values for a file and its parent directory. AI disclosure: OpenAI Codex assisted with diagnosis, implementation, and validation. |
||
|
|
756bdec19b |
fix(metadata): show the translation a search matched on Open Library and Hardcover (#1451)
Searching a translated title on Open Library or Hardcover returned the original-language work. "Der Weg der Könige" came back as "The Way of Kings" (2010), and that English title then went into the release search and the download name. Both providers keep one record per work, titled after the original, and list translations underneath it. #483 already searches releases per language with Hardcover's localized titles, which fixed the release side of #360 when the matching language is selected. This PR fixes the search result itself, so the book a user picks is the translation they searched for, and it covers Open Library, the default provider, which has no titles_by_language. - Open Library: request the matched edition (editions.*) and prefer its title, ISBNs, cover, language and publication year when its title differs from the work's. - Hardcover: use the alternative title named in the hit's highlights (alternative_titles, best index first), without the original's subtitle. A hit whose title also matched keeps the work title. - /api/releases: when the modal sends a title that differs from the book record's, it becomes search_title. Before, Hardcover's search_title from the original won, so picking "Kinder des Nebels" still searched "The Final Empire" first. - Release cache: keyed on title and language as well as the provider ID. All translations share that ID, so opening "The Way of Kings" after "Der Weg der Könige" showed the German releases until a reload. Known limitations: neither provider has descriptions per edition, so the description stays the work's (usually English). Hardcover's search hits carry no year or language for alternative titles, so those stay the work's; Open Library's do not. Google Books is unaffected, since it already returns one volume per edition. Tests: Open Library edition parsing, Hardcover alternative-title matching, the /api/releases title override, and the release cache, each written to fail first. The Python suite passes apart from the 9 test_entrypoint_permissions.py tests, which fail the same way on an unmodified main on macOS. The frontend typecheck, lint, format check and unit tests pass. Checked on a live instance with Open Library, Hardcover and Google Books. Refs #360, #483 Disclaimer: This was developed with the help of AI but reviewed and tested as well as deployed on my server by a real human. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
45548687b7 |
fix(hardcover): defensive rate limiting and 429 backoff (#1439)
Kept as small as I could so it's easy to review. It is still about 140 lines of code plus tests, in two commits so the second can be dropped on its own. Same kind of fix as #1429, so see the question there about separate PRs versus combined. This is a defensive change. I found the problem on my own deployment, which has a more automated ingestion on top of this provider: Hardcover want-to-read items turn into requests, and each pending request gets one `get_book` lookup. One pass made around 38 lookups back to back, which is more than Hardcover's burst allows, and a few came back as 429. `_execute_query` swallowed the 429 and returned `None`, so `get_book` returned `None`, and the caller read that as "no such book". Nothing in the logs said a lookup had been throttled. **What it does** - Every POST goes through one shared token bucket (burst 5, 0.8 requests per second), set under the documented Free limits. The limit is per user across all of the account's apps, so I kept well below it. - A 429 is retried up to 3 attempts in total. It waits for `Retry-After` when Hardcover sends one, otherwise 2^attempt seconds plus up to a second of jitter, and the wait holds back other threads too. A `Retry-After` over 30 seconds (a spent daily quota) is not waited out. - When the retries run out it raises `HardcoverRateLimitError`, a `RuntimeError`, instead of returning `None`. - The second commit makes `get_book` ask for that error, so a throttled or failed lookup is no longer taken for a missing book. GraphQL rejections still return `None`. **Behaviour change in the second commit.** `/api/metadata/book` already turns a `RuntimeError` into a 503, so a timeout or HTTP error there now returns 503 instead of 404. In `/api/releases` the same error goes through the operational errors handler and returns 500 with the message, where it used to return 404 "Book not found". If you'd rather keep returning `None` and only have the pacing and backoff, I can drop the second commit. **Limits.** I read them from the `hardcover-docs` repo (`api/Getting-Started.mdx`, Rate Limits), because the docs site blocks my headless browser: Free is 60 per minute, a burst of 10 and 5,000 a day, with `Retry-After` on a 429. Personal Access Tokens get double the burst. I didn't parse the `RateLimit-Policy` headers, so the bucket constants are fixed. **Testing.** The tests use a fake clock and never sleep. I broke each rule in turn (pacing, `Retry-After`, the 30 second cap, the shared hold-back, and `get_book` raising) and each break failed a test. Ruff, basedpyright, `tests/metadata` and `tests/core` pass. I haven't seen a real 429 from Hardcover with this change running, so the `Retry-After` handling comes from the docs. The one red check is the existing clock-dependent test `test_a_queued_torrent_asks_for_a_grace_once`, which fails on any runner that booted recently. It is fixed in #1426 and is not caused by this change. --------- Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
|
61cf0792bd |
fix(googlebooks): report more pages when limit is above the 40 result maximum (#1434)
Kept small on purpose so it's easy to review. Same kind of small fix as #1429 to #1432, so see the question in #1429 about separate PRs versus combined. I'm sending them separately until I hear otherwise. `search_paginated` guesses `has_more` as `len(books) >= options.limit`. Google Books returns at most 40 per page and `search` already caps `maxResults` at 40, but `/api/metadata/search` accepts a limit up to 100. With a limit over 40 a full page of 40 never reaches `limit`, so `has_more` is always False. The UI sends 40, so this only affects direct API callers. The Google provider now overrides `search_paginated` to apply the same cap before the base heuristic runs. A full page of 40 with a limit of 100 reports more results and a short page doesn't. The first test fails on current `main`. Ruff, basedpyright and `tests/metadata` pass. The one red check is the existing clock-dependent test `test_a_queued_torrent_asks_for_a_grace_once`, which fails on any runner that booted recently. It is fixed in #1426 and is not caused by this change. |
||
|
|
f0f0d46381 |
fix(queue): first terminal status wins, so a late COMPLETE cannot overwrite a cancel (#1429)
Kept small on purpose so it's easy to review. I have a few more small fixes like this from going back over recently merged PRs. Would you rather get them as separate PRs like this one, or combined into one? #1361 made `cancel_download` write under the queue lock, which stops a cancel from overwriting a download that finished first. The other direction is still open. The worker checks `cancel_flag.is_set()` and then calls `update_status(COMPLETE)` a few lines later in `_process_single_download`. If a cancel lands between those two lines the task goes CANCELLED, then the worker writes COMPLETE over it. `_apply_status_locked` allows terminal to terminal moves, so the terminal hook also fires a second time and records two events for one task. This makes the first terminal status win: `_apply_status_locked` ignores a different terminal status once the task already has one. Requeueing from a terminal status still works because QUEUED isn't terminal. I replaced the existing race test. It patched `queue.update_status`, which `cancel_download` no longer calls, so it never exercised the path it names, and its last assertion only looked at the final hook event. The new tests call the queue directly: complete then cancel is refused, cancel then complete stays cancelled with one hook event, error then complete stays error, and a terminal task can still be requeued. The two late-write tests fail on current `main`. Ruff and basedpyright are clean. `tests/core` and `tests/download` pass apart from 4 `seleniumbase` import failures in my environment that also fail on `main`. The one red check is the existing clock-dependent test `test_a_queued_torrent_asks_for_a_grace_once`, which fails on any runner that booted recently. It is fixed in #1426 and is not caused by this change. |
||
|
|
65dbf42a86 |
fix(release-search): report a ValueError from a source as that source's error (#1433)
Kept small on purpose so it's easy to review. This is the same kind of small fix as the ones in #1429 to #1432, so see the question in #1429 about separate PRs versus combined. I'm sending them separately until I hear otherwise. In `search_source_releases` the `except ValueError` comes before the branch that handles the operational errors, and `ValueError` is in that tuple. A `ValueError` from `build_release_search_plan` or `source.search()` is therefore reported as `Unknown source: <name>` instead of `<name>: <message>`. The only `ValueError` meant for that branch is the one `get_source` raises for a name that isn't registered, so that call now has its own `try` and the rest goes back to the operational errors branch. The failure test already covered `OSError` and `RuntimeError`, so I added `ValueError` to it, and that case fails on current `main`. Ruff, basedpyright and `tests/core` pass. The one red check is the existing clock-dependent test `test_a_queued_torrent_asks_for_a_grace_once`, which fails on any runner that booted recently. It is fixed in #1426 and is not caused by this change. |
||
|
|
93f58caf7a |
fix(debug): keep SHELFMARK_API_KEY_READONLY out of the debug bundle (#1432)
Kept small on purpose so it's easy to review. `genDebug.sh` filters secrets out of `environment.txt` by matching `_KEY=`, `_SECRET=`, `_PASSWORD=` and `_TOKEN=`, which only catches names that end in those words. `SHELFMARK_API_KEY_READONLY` ends in `_READONLY`, so its value lands in the file, and people attach these bundles to issues. That key only reads counters, so the impact is small, but any variable like `DB_PASSWORD_FILE` would leak the same way. The pattern now matches the secret word anywhere in the variable name. The new test runs the real filter line from the script against a made-up environment, and it fails on the old pattern. The one red check is the existing clock-dependent test `test_a_queued_torrent_asks_for_a_grace_once`, which fails on any runner that booted recently. It is fixed in #1426 and is not caused by this change. |
||
|
|
be2653bdb7 |
fix(debridlink): reuse the file list from the status check (#1431)
Kept small on purpose so it's easy to review. When a torrent is ready, `get_status` has already fetched the full record with `/seedbox/list?ids=` and `_handle_torrent_info` has checked that every file is at 100%. The retrieval thread was then started without that list, so `_process_and_download` called `_fetch_file_list`, which asks the same endpoint again. That second call runs inside the thread's generic `except`, without the flood and transient error handling the status path has, so a rate limit or a hiccup there sets the phase to error for a torrent that finished fine. This passes the files already in hand into the thread. `_process_and_download` already took an optional `files` argument. The new test starts the thread from a ready status and checks it uses the given list and never calls `_fetch_file_list`. Separately, `download_url` returns a `BytesIO`, so a whole file sits in memory. AllDebrid, TorBox and Real-Debrid do the same, so I left that for a streaming variant of the shared helper instead of fixing it for one client here. Ruff and basedpyright are clean and `tests/prowlarr` passes. The one red check is the existing clock-dependent test `test_a_queued_torrent_asks_for_a_grace_once`, which fails on any runner that booted recently. It is fixed in #1426 and is not caused by this change. |
||
|
|
e8f6505bea |
fix(library): back off after a failed index instead of retrying for every result (#1430)
Kept small on purpose so it's easy to review. `library_index._entries_for` retries `fetch_entries` on every call while the index is failing. The lookup runs once per search result, so with the Calibre path unreadable a page of 40 results makes 40 attempts and logs 40 warnings. The entrypoint runs one gevent worker and the Calibre read is synchronous sqlite, so each attempt also holds up every other request. After a failure it now answers from the stale cache (or an empty list) for 60 seconds before trying again, and a successful index clears that. The new tests cover 40 lookups against a failing library with a stale cache (one failed attempt, one warning), a failing library with no cache (tried once), and a retry once the 60 seconds have passed. The first and second fail on current `main`. I left a few other things in this module out to keep this one small: re-indexing blocks the worker for as long as a big library takes (I haven't measured how long), `test_connection` stores entries from the unsaved form path under the shared cache key, and a title with a collection word in it, like "Books of Blood", shows as "In a collection". I can send any of those separately. Ruff and basedpyright are clean and `tests/core` passes. The one red check is the existing clock-dependent test `test_a_queued_torrent_asks_for_a_grace_once`, which fails on any runner that booted recently. It is fixed in #1426 and is not caused by this change. |
||
|
|
47707b827a |
feat(download): make the torrent stall window a setting (#1426)
#1420 gave a torrent 15 minutes to get going and 15 minutes after each step forward. That suits my swarms, but other people will want it shorter or longer, so this makes the number a setting. `TORRENT_STALL_TIMEOUT_MINUTES` sits in Download Clients with the other torrent options and takes 5 to 60. The default is 15, so nothing changes unless someone sets it. 5 is the orchestrator's own stall timeout, so choosing it asks for no extra time and gives the old behaviour. The value can arrive as text from an env var, so anything that isn't a number falls back to the default and anything out of range is clamped. The orchestrator caps one activity grace at 16 minutes, so I raised that to an hour to fit the top of the range. As far as I can tell the protection bypass is the only caller that asks for more than a few minutes, but I haven't audited every caller. If you'd rather keep the cap, I can lower the maximum to 16 minutes instead. Tests cover the default, 5 and 30, text values, out of range values and NaN, the start and movement windows following the setting, and the setting showing up in the Download Clients tab. I broke each rule on purpose to check a test catches it. The full suite fails the same tests with and without this change (the bypass and seleniumbase tests don't run in my environment). |
||
|
|
9321096032 |
fix(clients): correct Debrid-Link file listing and rate-limit handling (#1425)
Follow-up to #1380. - Fetch the file list with ?ids=, the documented parameter. There is no ?id=, so a many-file torrent was not expanded and the call could fail with badArguments at the last step of every download. - Back off status checks for 60s after floodDetected, doubling to a 240s cap, instead of an hour. The orchestrator cancels a download after five minutes without a change, so the hour-long pause cancelled every active download. Only a flood on the status check starts it, and a successful check resets it. - Drop the client-side one-hour stall timer. The orchestrator stall timer always fires first. - Keep progress at 50% when file retrieval starts instead of dropping to 0% until the first file arrives. - Report queued, paused and verifying torrents as QUEUED, PAUSED and CHECKING, so a queued torrent gets the handler queue grace. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
cad18b8019 |
feat(clients): add Debrid-Link as a torrent client (#1380)
calibrain said in #1315 that he'd review and merge a Debrid-Link client if someone took a stab at it, so here it is. It's the fourth debrid service alongside AllDebrid, Real-Debrid and TorBox and it uses the same DownloadClient interface as the other three. No frontend changes, the settings UI picks it up from the backend field definitions. **I can't test this against the live API.** I don't use debrid and I don't have a Debrid-Link key. Everything here is built from their v2 API docs and cross-checked against two independent reference implementations, so the request shapes should be right, but nobody has run a real magnet through it yet. @Tatsu941 you opened the issue and mentioned they give developers free premium. Would you be able to try it? The things most worth confirming are the field names on a seedbox torrent, that `premiumLeft` is the right premium check, and that an API error comes back the way the client expects. Happy to fix whatever turns up. Two things about their API shaped how this is written. Every v2 endpoint wraps its payload in a `{success, value}` envelope and a failure can arrive with an HTTP 200, so `_request_value` decides the outcome from the body instead of the status code. A completed seedbox torrent already carries a direct `downloadUrl` on every file. The other three clients each need a separate unrestrict or link-request call per file and this one doesn't, which is why it came out shorter than them despite doing the same job. Endpoints used: | Endpoint | Used for | |---|---| | `GET account/infos` | test the key, confirm premium time remains | | `POST seedbox/add` | magnet as JSON, .torrent as multipart | | `GET seedbox/list` | poll progress | | `DELETE seedbox/{id}/remove` | clean up | Ruff and basedpyright are clean and the 36 new tests pass. The two failures in `tests/e2e/test_proxy_auth_middleware.py` are already on main and fail the same way on a pristine checkout. They assume the first proxy-authenticated user becomes an admin, which only holds while the user table has no admin in it, so they depend on how many users earlier tests created. One thing for review: the download loop uses `download_url` like the other three clients, which holds each file in memory. I kept the existing pattern rather than do something different in a new-provider PR, but say so if you'd rather it streamed here. Closes #1315 |
||
|
|
f773ac6029 |
fix(sabnzbd): strip Prowlarr API key on cross-host NZB redirects (#1424)
_fetch_nzb_content sent X-Api-Key then followed redirects with the header still attached, leaking the Prowlarr key when Prowlarr 302s to the indexer download link. Follow redirects manually with allow_redirects=False and re-evaluate headers per hop, mirroring the torrent path. |
||
|
|
755f28a6a4 |
feat(api): add a read-only API key and a /api/stats endpoint (#1419)
This is the read-only API key from #1410, where you said to go ahead. I run Shelfmark behind Homepage and wanted more on the dashboard tile than up or down, without putting an admin API key in the dashboard's config. This adds a second key, `SHELFMARK_API_KEY_READONLY`, that can read one new endpoint. `GET /api/stats` returns counts only: books added over the last 7 and 30 days by format, the queue, requests by outcome, and download failures over the last 7 days. No titles and no user names. The read-only key, the admin key or an admin session can read it. The read-only key gets a 403 on every other path and on any write, including a POST to `/api/stats` itself, and it never gets a session or a cookie. If a request carries both keys, the admin one wins, so a proxy's own `Authorization` header can't downgrade a correct `X-Api-Key`. Unset means off, like the existing key. The endpoint exists either way, but without a key only an admin session can read it. It also adds a `db_path` property on `UserDB`, so the stats module can open the database read-only. Tests cover the key scope, the 403s for the read-only key, both keys at once, who can reach the endpoint, and the counters. The full suite passes, and I broke each rule on purpose to check a test catches it. A version of this has been running on my own install behind Homepage. If you'd rather have the stats in a different shape or under a different path, I'm happy to change it. |
||
|
|
32abaff21e |
feat(naming): {Narrator} placeholder and MyAnonamouse series fallback (#1407)
## Human-Written explanation:
Following up on previous PR to use MAM ID to get nararrator and series
to show up in search, this PR will allow the series and nararrator
fields to be added to the path when saving an audiobook.
I have tested my ghcr.io image on my instance and it seemed to work
properly.
All the text below is written by Claude.
# feat(naming): {Narrator} placeholder and MyAnonamouse series fallback
Follow-up to #1390 and #1399. Related to #605 and #934 (narrator
support).
## Problem
Several narrations of the same audiobook currently land in the same
folder, because nothing in the path template tells them apart. Keeping
more than one version means renaming folders by hand after every
download.
Since #1390, MyAnonamouse results carry the release's narrator and
series in `extra`, but the naming templates can't use them: there is no
`{Narrator}` placeholder, and `{Series}` / `{SeriesPosition}` only come
from the metadata provider.
## Changes
- **`{Narrator}` placeholder** (`core/naming.py`,
`download/postprocess/transfer.py`, `core/models.py`): `DownloadTask`
gains `narrator`, read from `narrator` or `extra.narrator` when a
release is queued, and kept in the restart-safe retry payload. It's
available in audiobook templates like any other placeholder, including
prefix/suffix blocks such as `{ - Narrator}`.
- **Audiobookshelf folder style**: Audiobookshelf reads the narrator
from folder names like `Title {Narrator}`. The template `{Title}
{{Narrator}}` already matches as `{` + `{Narrator}`; the parser now also
drops the closing brace when the narrator is empty, so the folder is
`Title` rather than `Title }`.
- **No stray spaces around `/`**: an empty placeholder at the start or
end of a folder name no longer leaves a space there (e.g. `Title
/Title`). This applies to all templates.
- **Series fallback** (`release_sources/prowlarr/mam.py`, `source.py`,
`download/orchestrator.py`): MAM enrichment also stores the first
series' name and number as `extra.series_name` /
`extra.series_position`, which queueing already falls back to when the
metadata provider has no series. The provider's series still wins. A
release-level series number is only used when it names the same series
as the provider, so a provider series is never paired with the number of
a different MAM series (an omnibus, for example).
- **Settings and preview** (`config/settings.py`,
`namingTemplatePreview.ts`): the audiobook template descriptions list
`{Narrator}` and explain `{{Narrator}}`. The settings page preview
mirrors the parser changes and offers `{Narrator}` under "Insert
variable" for audiobook templates only.
- **Docs**: regenerated `environment-variables.md`. This also picked up
the Library Check settings, which weren't in the generated docs yet;
happy to drop that part to keep the diff focused.
## Example
Audiobook Path Template `{Author}/{Title} {{Narrator}}/{Title}`:
| Narrator | Result |
|----------|--------|
| Samuel Roukin | `Christopher Ruocchio/Empire of Silence {Samuel
Roukin}/Empire of Silence.m4b` |
| none | `Christopher Ruocchio/Empire of Silence/Empire of Silence.m4b`
|
## Testing
- `tests/core/test_narrator_template_variable.py` (new, 16 tests):
`{{Narrator}}` with and without a value, the `{ {Narrator}}` prefix
form, prefix/suffix blocks, sanitizing, `build_library_path`, task
metadata, retry payload round trip, MAM series name/number parsing
(including a `1-3` range having no number), and queueing (narrator from
`extra`, series fallback, no cross-series number, blank narrator).
- `namingTemplatePreview.test.ts`: the same `{{Narrator}}` cases for the
preview.
- Updated `test_generate_env_docs.py` for the new template description.
- Full `pytest -m "not integration and not e2e"` compared with an
upstream `main` worktree on the same machine: no new failures (the
remaining ones are Windows-only on both).
- `ruff check`, `ruff format --check`, `basedpyright` (0 errors) and
`vulture` on touched files; frontend `tsc --noEmit`, `oxlint`, `oxfmt
--check`, `vitest` (218 passed).
- Checked the settings page locally: the audiobook preview renders `...
{Simon Vance}.mp3` and `{Narrator}` is listed for audiobook templates
only.
No behavior change for existing templates, apart from spaces next to `/`
being trimmed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
119d3374e3 | fix(downloads): reconcile downloads a restart interrupted (#1423) | ||
|
|
95d9f1da57 |
fix(sabnzbd): prefetch NZBs served from the indexer's own domain (#1416)
Indexers often serve NZB downloads from a different host than their API (e.g. file.indexer.example for indexer.example/api). The exact-origin check added in #967 rejected those, so SABnzbd silently fell back to addurl. Trust any host in the configured indexer's registrable domain, using the Public Suffix List so shared suffixes like co.uk or duckdns.org never widen trust. Scheme and port must still match, IP literals and single-label hosts still require an exact match, and the Prowlarr API key header is unchanged. This closes this issue: https://github.com/calibrain/shelfmark/issues/1411 Disclaimer: Implemented by Claude and Co-Authored by me. Tested and verified by me alone (Why is it always this way around and not the other) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8392b43dd3 |
fix(downloads): reconcile downloads a restart interrupted (#1409)
I found this while grabbing new books and working on my fork. I redeployed the container while downloads were running, and it left a mess in my queue. It's an edge case, but a few rows were stuck until I cleared them by hand. The download queue lives in memory, so stopping the process with a download in flight leaves its history row "active" and its request "queued", with nothing working on either. The activity API shows that row as "Interrupted" when it's read, but nothing is stored and nothing looks at it again. Upstream only reopens an errored request when no manual retry is offered, and an interrupted row always offers one, so the request just sits there. This closes those rows out once at startup, before the coordinator starts, as an error with the message "Interrupted". The retry payload stays, so the manual retry still works. A request whose download was queued in the last 24 hours goes back to pending through `reopen_failed_request`. Older ones get closed out but their requests are left alone, because I didn't think a startup should reopen a backlog or fill an admin's approval queue with old requests. A request that already arrived is left alone, and running it twice changes nothing. It assumes a single worker process, which `entrypoint.sh` already sets. On my install the first start after deploying it closed 53 rows that had been orphaned for nine days. Tests are in `tests/core/test_startup_reconcile.py`, and the full suite passes locally. If the 24 hour window feels off, or you'd rather handle this another way, I'm happy to change it or drop the PR. Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
|
7888217380 |
feat(download): add Remove & Delete Files torrent completion action (#1401)
Closes #1027 ## Summary - add a **Remove & Delete Files** (`remove_and_delete`) option to the Torrent Completion Action setting - after a successful import, remove the torrent and ask the client to delete its downloaded data (`delete_files=True`), matching what the usenet "move" flow already does - clarify the setting description so it says **Remove** keeps the downloaded files (raised in the issue comments) - regenerate the `PROWLARR_TORRENT_ACTION` entry in `docs/environment-variables.md` ## Behavior The deletion runs from `post_process_cleanup`, the same place as the existing Remove and Change Category actions, so it only happens after output transfer and post-processing have succeeded. At that point every file has already been copied or hardlinked into the library, and transfer size checks have passed. A failed import never removes or deletes anything. The torrent client deletes its own data, so Shelfmark does not delete paths itself and remote path mappings are not involved. Keep, Remove, and Change Category behave as before, and Keep is still the default. If a request matched a completed torrent that was already in the client, this option deletes that torrent's data too. That is the same scope the existing Remove action already applies to. ## Validation - `make python-checks` (ruff check, ruff format, basedpyright on backend and tests, vulture): passed - `make python-test`: 3363 passed, 5 skipped, 9 failed. All 9 are in `tests/config/test_entrypoint_permissions.py` and happen because macOS `/bin/bash` 3.2 does not support `${1,,}` in `entrypoint.sh`. They do not touch this change. - new tests in `tests/prowlarr/test_handler.py`: - Remove passes `delete_files=False` and Remove & Delete Files passes `delete_files=True` - a failed import with Remove & Delete Files does not call the client - the delete case fails without the handler change - manual smoke run: `PROWLARR_TORRENT_ACTION=remove_and_delete` set through real config loading, with a stub client that deletes its folder. The torrent folder was deleted and the hardlinked library file stayed intact. - pre-commit hooks (prek) passed --------- Co-authored-by: Evan Kazakin <evan@Evans-MacBook-Pro.local> |
||
|
|
d99e9d7c4f |
fix(download): remove a cancelled torrent that never started (#1421)
Cancelling a torrent download leaves the torrent in the client. `_safe_remove_download` documents the rule: "torrents: never remove or delete client data (avoid breaking seeding)". That makes sense for a torrent that has data. A torrent that is still at 0% has nothing to seed and nothing to resume, and leaving it behind holds a download slot in qBittorrent's queue or sits there as a dead entry. I hit this on a qBittorrent that is shared with Sonarr, Radarr and Lidarr and has a download limit. Shelfmark cancelled downloads that were queued (#1420) and left them in the client. Three torrents that never fetched metadata held every active slot, and each torrent added after them waited behind them. Removing those by hand freed the slots. This removes the torrent and its files when a download is cancelled, Shelfmark added the torrent, and the client reports 0% progress. A torrent with any data is left alone as before. So is one that was already in the client when the download started, since Shelfmark didn't add it. If the removal fails it is logged and the cancel carries on. Tests cover removal at 0% for a queued and a downloading torrent, leaving one that has data, leaving one the user already had, and a failing removal. The full suite passes, and I broke each rule on purpose to check a test catches it. This changes a rule you wrote down, so I've kept it apart from the queue fix. If you'd rather have it as a setting, or only remove a torrent that is still queued or fetching metadata, I'm happy to rework it. |
||
|
|
88f73e3bc3 |
fix(download): do not cancel a queued torrent as stalled (#1420)
qBittorrent only runs a few downloads at once (`max_active_downloads` defaults to 3) and holds the rest as `queuedDL`. Shelfmark's stall timer cancels a download after five minutes without a changed status or progress. A queued torrent reports "Queued" on every poll, so it looks idle, and Shelfmark cancels a download that was only waiting for a slot. I run Shelfmark against the same qBittorrent that Sonarr, Radarr and Lidarr use, which I assume is a common setup. Their torrents and Shelfmark's share one limit, so anything that fills the slots makes Shelfmark's torrent wait. In my case three torrents that never fetched metadata held every slot. I'd expect a busy Sonarr queue to hold them the same way, but I haven't reproduced that. Once the stall timer had cancelled the waiting downloads they stayed in the client, still queued behind the same blockers. On my instance 10 AudioBookBay downloads were cancelled while qBittorrent still had them queued. This changes the torrent poll loop. When the client reports a queued torrent, it asks for an activity grace (the mechanism `html_get_page` already uses for protection bypasses) and releases it as soon as the torrent leaves the queue. The grace is renewed every ten minutes and stops at two hours, so a queue that never moves still ends. A torrent that is downloading and moving is timed as before until the new window below applies. The second commit sets the stall message before the cancel runs. The terminal hook copies the message into the history row, and it was set afterwards, so a download the timer ended was recorded with the last poll's "Queued". That left no way to tell it apart from a person cancelling. The last two commits are about slow torrents. The timer only counts a change in progress, so a torrent that needs more than five minutes to fetch metadata or find its first peer is cancelled before it has any progress to count, and one that moves a few megabytes at a time can be cancelled between bursts. I hit this with a torrent that had a single peer. A torrent now gets one 15 minute window the first time it is seen outside the queue, and each step forward in progress renews it. One that makes no progress for 15 minutes is still cancelled, so a dead torrent now takes about 15 minutes to end instead of five. That trade is the part I'm least sure about, so the number is easy to change. Tests cover the grace being requested once, released when the torrent starts, renewed, and capped, the order of the message and the cancel, and the start and movement windows. The full suite passes, and I broke each rule on purpose to check a test catches it. The two hour ceiling is a guess. If you'd rather have a setting for it, or a different number, I'm happy to change it. |
||
|
|
bac0b93566 |
fix: keep https:// for qBittorrent and hide release sources from non-admins (#1422)
qBittorrent over HTTPS (#1417) qbittorrent-api ignores the scheme in `host` and probes HTTP and HTTPS itself. When the HTTPS probe fails, for instance on a certificate Shelfmark doesn't trust, it falls back to plain HTTP against the TLS port, and a reverse proxy answers "400 The plain HTTP request was sent to HTTPS port". The download client and the Test Connection button now pass FORCE_SCHEME_FROM_HOST for https:// URLs, so the configured scheme is used as-is and a certificate problem is reported as one. http:// and bare host:port URLs keep the probe: normalization adds http:// to a bare host, and the probe follows a proxy's redirect to HTTPS. Release sources leaked to non-admins (#1418) A download queued from an approved request carries the release an admin picked, and the requester could read it from their own activity feed: the request's release_data (source_id, indexer, info URL, torrent attributes), the download's full server path, and the download id itself, which for Prowlarr is "<indexer id>:<guid>" and for a private tracker is a URL into it. That id keys every status payload, the /api/localdownload query and the cover proxy URL, so hiding release_data alone would not have closed the leak. For non-admin viewers: - request rows keep only the release fields the activity cards display - download_path is reduced to the file name, which the browser download reveals anyway - request-linked downloads are addressed by an opaque id (a keyed HMAC of the task id) in the snapshot, history, /api/status, queue order, active downloads, dismissed keys, cover URLs and the websocket status and progress events. Routes that take a download id translate it back, searching only the caller's own downloads. Downloads a user queued directly keep their real id: the user picked that release from search results that already showed it, and the release list matches its buttons to the queue by that id. Admin views are unchanged. The activity sidebar now links a fulfilled request to its download by request_id instead of release_data.source_id. Closes #1417 Closes #1418 |
||
|
|
41a4df01c4 |
fix: hand DiamWall 513 challenges to the bypasser (#1400)
## Summary Recognize DiamWall interstitials in both the HTTP retry path and the internal browser helper. HTTP 513 challenge responses go directly to the configured bypasser, while ordinary 513 responses keep the existing error behavior. Fixes #1386 ## Testing - `uv run --frozen --extra browser pytest -q -n 0 tests/download/test_http_challenge_513.py tests/bypass/test_internal_bypasser.py`: 27 passed - Ruff lint and format checks passed |
||
|
|
893f7bdb92 |
fix(mam): keep the session ID on MAM and rerun Prowlarr's exact search (#1399)
Follow-up to #1390. - The origin came from each result's infoUrl, matched by a regex that did not check the host, and the mam_id cookie had no domain. Any Prowlarr indexer returning a URL like https://evil.example?myanonamouse.net/t/1 sent the session ID to evil.example. Requests now always go to https://www.myanonamouse.net (Prowlarr's only MAM URL), with the cookie as a header and redirects off. Only results from Prowlarr's MyAnonamouse indexer are looked up, and their URLs must be on myanonamouse.net. - The lookup searched every category and read one page, so for common titles most of Prowlarr's results were missed (a "Dune" audiobook search: 56 audiobooks on the first 100 of 328 matches). It now reruns Prowlarr's exact search: the same query clean-up, the MAM main categories behind the Torznab categories searched (13/15/16 for audiobooks, 14 for e-books, all once expanded), and the MAM indexer's own search type, search-in options and languages. Further pages are read while IDs are missing, page 1 of every title first, at most 4 requests per search. - Failed requests back off for 1, 2, 4 ... up to 30 minutes. The 10th consecutive failure stops enrichment until Test MAM Session passes, the session ID changes, or Shelfmark restarts. - The detail cache prunes expired entries instead of growing for as long as Shelfmark runs. |