mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-11-15 17:18:11 +00:00
e575ff0fcb586bc0aa0600a2ea198d5b8c05dd5d
96
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 splitsec2
|
76d7df306f |
fix(requests): retrying a request after its client failed offers the same release again (#1677)
When a request's download fails in a way the queue can't resume, the request is reopened and its Retry button opens the release list. The release that just failed was drawn there as a red, disabled "Failed" button for as long as the failed task stayed in the queue status, so you couldn't pick it again even when the release was fine and the download client was the problem. A bad qBittorrent password is the easy way to hit it. Thanks @joshft91 for the report and the steps. What changes: - In the release list, a failed request download now shows Retry, the same as a failure the queue can retry already does. A direct download with nothing to retry from still shows Failed. The rule lives in one small helper used by the release modal and both button-state paths in `useDownloadTracking`. - A client that refuses the login is no longer held against the release. qbittorrent-api raises `LoginFailed` with no message for a wrong password, and only "Forbidden" or "Unauthorized" for a bad session or API key, so none of it matched the unreachable-client markers from #1655 and the release went into `failed_releases`. The qBittorrent client now says "qBittorrent authentication failed (...)" for these. - The request reopen and the auto-download retry used two copies of the same marker list. They now share one in `status_messages.py`, which also counts an explicit authentication failure as the client's fault. A bare 403 or 401 is not on it, since a dead .torrent link fails that way too and that one is the release's. Genuine release failures (no seeders, a rejected torrent, a dead .torrent link) are still remembered as #1655 intended. Tested with a scratch instance against a mock Prowlarr and a fake qBittorrent that rejects the login: before, the request's Retry opened the list with the failed release as a red X; after, it shows Retry, picking it fulfils the request again, and the request reopens without the release in `failed_releases`. New tests cover the qBittorrent messages, both reopen paths and the button state. Fixes #1674 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Failed releases are no longer skipped after download-client connection or login failures; download failures and stall cancellations remain eligible to be skipped on the next search. * Failed requests with an associated request can now show a retry option, while direct downloads without retries remain marked as failed. * qBittorrent login and authorization errors are reported more clearly. * **Documentation** * Clarified which failures cause a release to be skipped on a later search. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
  Matt Van HornandMatt Van Horn
|
0bb54ac8f8 |
fix(clients): use port 443 for HTTPS Transmission URLs that omit a port (#1673)
## Why Saving a Transmission address as HTTPS and leaving the port blank makes the connection test fail. The client opens port 9091 and reports an SSL wrong-version error, so a daemon published through a reverse proxy on the usual HTTPS port never answers. parse_transmission_url filled in 9091 whenever the URL omitted a port, and that default ignored the scheme. An https address therefore started TLS against 9091, where the peer is not speaking TLS. ## Summary An omitted port now follows the scheme: 443 for https, and 9091 for http, which still covers the empty-URL default. A port written in the URL is left as given. The parser tests cover https://transmission.myserver.org and expect https, that host, port 443, and the path /transmission/rpc. ## Testing The targeted tests for the shelfmark parse_transmission_url change pass locally. The tests passed with the shelfmark parse_transmission_url change applied; a run with it reverted gave no clear result, so no test is shown to catch this bug. Closes #1667 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * HTTPS Transmission URLs without an explicitly specified port now use port 443. HTTP URLs continue to default to port 9091, and the default RPC path is unchanged. * **Documentation** * Clarified the default ports for each URL scheme and noted that HTTPS instances on port 9091 require `:9091` in the URL. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> |
||
 Rupraj Datta
|
ed0777a2e0 |
Spool large debrid downloads to disk (#1632)
Fixes #1630. Debrid downloads currently buffer entire files in RAM. The four clients now opt into a spooled temporary file: downloads up to 8 MiB stay in memory, and larger downloads roll to disk. The clients copy from the file in chunks and close it after use. Other download_url callers retain the existing BytesIO behavior. Temporary files are closed on unsuccessful attempts and cancellation; successful resumed downloads remain open for their caller. Added offline coverage for disk rollover, timeout cleanup, and resumed content. Updated the Debrid-Link test doubles for the new opt-in argument. Validation: 113 tests passed across test_http_download_url, test_debridlink_client, and test_torbox_client; Ruff check and format check passed for changed files; git diff --check passed. No live debrid services were exercised. AI assistance was used in implementing this change. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Improvements** * Large downloads are handled more efficiently, reducing memory use while preserving download and resume behavior. * **Bug Fixes** * Temporary download data is cleaned up when a download is cancelled or fails, helping prevent unnecessary resource usage. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
splitsec2
|
cfba44b754 |
refactor: one module for taking credentials and links out of text (#1578)
Draft until #1571 merges: this is stacked on it, so the first two commits here are that PR's and only the last one is new. I will rebase once #1571 is in. Shelfmark strips credentials and links out of text in five places, written in different PRs with different patterns: the log formatter, `_safe_url` and `_redact_urls_in_text` in `torrent_utils.py`, `redact_status_message` in `viewer_redaction.py`, the `sed` rules in `genDebug.sh`, and the password handling in the settings registry. Between them there were three regexes that find a URL in text, two separate userinfo patterns and no shared list of what counts as a secret. #1571 made the gap visible, because every pattern added to the logger had to be remembered somewhere else too. This moves the text and URL part into `shelfmark/core/redaction.py`, which imports nothing from the app so the logger can depend on it: - `redact_secrets` (mask in place, for logs) moved from `core/logger.py`, patterns unchanged. The logger imports it. - `safe_url` and `redact_urls_in_text` (keep scheme, host and path, drop the query) moved from `torrent_utils.py` and are public now, so `alldebrid.py` no longer imports a private name from another module. - `strip_links` (remove links entirely) is what `redact_status_message` now calls. It and `redact_urls_in_text` share one link pattern, the one the viewer redaction already used. Two small behaviour changes come from sharing that pattern. A closing bracket or quote right after a URL is no longer swallowed into it when scrubbing an exception message. And the scheme-less path that requests prints after `url: ` now keeps only what is before its query in `redact_urls_in_text`, where before it passed through as it was (the log formatter masked the key, but the path and query stayed in the message). Also here: the two `sed` rules in `genDebug.sh` that mask userinfo now stop at `?` and `#` like the logger pattern does, with a test that fails on the old script. Not in this PR: the debug bundle zips the live log directory as it is, so log lines written before #1571 are not redacted in it. Fixing that means copying the logs to a staging folder and running them through this module, which I would do as a separate change. Tests: a new file covers `safe_url`, `redact_urls_in_text` and `strip_links` directly, and two checks that the viewer and the logger really use these. Full suite passes. |
||
|
splitsec2
|
72b094db7a |
fix(qbittorrent): map the qBittorrent 5 stopped states and forcedMetaDL (#1570)
On qBittorrent 5 a stopped torrent shows up as an unknown state. 5.0 renamed `pausedDL` and `pausedUP` to `stoppedDL` and `stoppedUP`, and `get_status` only knows the old names, so the torrent comes back as state `unknown` with the raw state name as its message. The paused handling in the download handler never applies to it, for example the grace it gives a torrent that has not started. `forcedMetaDL` is missing from the same table, so a force-started magnet fetching its metadata also shows up as unknown, although it is already in the incomplete-states set and the metadata wait. The three missing states are now mapped like their neighbours: the two stopped states as paused, `forcedMetaDL` as downloading with "Fetching metadata". The state names are the ones `qbittorrent-api` lists in its `TorrentState` enum. Tests: one parametrized test covers the old and new paused names and `forcedMetaDL` (the two old paused names are there as controls). The three new cases fail without the change. Full suite passes. |
||
|
splitsec2
|
525b094b57 |
fix(clients): hand a finished wrong-type directory on instead of polling out the timeout (#1573)
A torrent that finishes with the wrong kind of files in it holds the download for the whole completed-path timeout (60 seconds by default, up to an hour) and then fails with "Completed download has no stable supported files yet." An ebook task whose torrent holds only an `.m4b`, an audiobook task that got an `.epub`, or a `.pdf`-only release with the default format list all land here. In v1.4.0 these failed with the specific format error from post-processing, but the wait for the sync to settle (#1464) only counts files of the task's own type, so it keeps polling for files that are never going to appear. If the directory has no supported files, `_wait_for_completed_path` now looks for finished book or audio files that this task does not accept (the audiobook formats, the default ebook formats and pdf). When it finds some and they are the same on two polls in a row, it returns the path so post-processing can say what is wrong with the formats. Cover art and notes alone do not count, because that is what a directory looks like while the real files are still syncing over a remote mount, so those keep waiting as before. A wrong-type file that is still growing keeps waiting too. With no wait configured there is only one sample, and a wrong-type file is handed on straight away. The existing test that pins "no book files, no wait configured" uses a jpg, so it stays as it is. I extended the test helper to take a content type, which the wait tests never passed before. Four new cases fail without the change (a wrong-type directory for ebook, audiobook and pdf tasks), and two more check that sidecar-only and still-growing directories keep waiting. Full suite passes. |
||
|
splitsec2
|
b248bcd536 |
fix(clients): import a qBittorrent torrent only when every listed file is on disk (#1577)
Fixes #1561, with thanks to @tmussche for the write-up, which had the cause and the fix. When the files reach Shelfmark through a sync (Resilio in the report), the wait for the completed path accepts the folder once two samples a few seconds apart look the same. A sync that copies in batches can pause longer than that, so 10 of 23 files were imported and some of those were still partial copies. For qBittorrent the client can say what the finished download holds, so the wait now checks that. `DownloadClient` gets an optional `get_expected_files()` that returns `ExpectedFile(name, size)` entries, or None when the client cannot say, which is the default and keeps every other client as it was. qBittorrent answers it from `/api/v2/torrents/files`, leaving out files the torrent does not download (priority 0), zero-byte entries and BEP 47 padding files. `_wait_for_completed_path` asks once, and while any of those files is missing or smaller than its final size it keeps waiting, using the same timeout as before. When the wait runs out the error now says how far it got, for example "Completed download is not fully on disk yet (10 of 23 files)." Once everything is there the existing stability check still runs, so nothing is accepted that was accepted before. Names from the client start with the torrent's top-level folder, so each file is looked up with and without it, which also covers a torrent saved without a subfolder. A single-file torrent is checked by size too, since a file that is still syncing exists at a smaller size. Two things I would like a second look at: the comparison is on exact size, so a setup where something rewrites the files after the torrent finishes would never match and would run into the timeout; and the file list is read once per wait, not on every poll. If the client does not answer, nothing changes. I also moved the nested authenticated GET in the path derivation into a `_get_with_auth` method so both calls share it, and added a sentence about this to `docs/configuration.md`. Tests: eight for the wait (partial folder, truncated file, files arriving during the wait, complete folder, no top-level folder, single file both ways, and a client without a list), and six for the qBittorrent file list. The wait tests that depend on the new behaviour fail on current main. Full suite passes. |
||
|
splitsec2
|
9011319ade |
refactor: one lookup for the supported formats, public names for two shared helpers (#1576)
Six places read the Supported Book Formats and Supported Audiobook Formats settings on their own: post-processing, the post-process scan, IRC, slskd, Prowlarr and direct download. For ebooks they already agreed, since they all fall back to `DEFAULT_EBOOK_FORMATS`. For audiobooks they did not. The settings registry and slskd fall back to every audio format plus zip and rar, while post-processing and IRC fell back to m4b and mp3 only. They now share `shelfmark/core/formats.py`, with `get_supported_formats`, `get_supported_audiobook_formats` and a content-type aware `get_supported_formats_for`, and a shared `DEFAULT_AUDIOBOOK_FORMATS` in `core/utils.py` next to the ebook one. `policy.py` re-exports the same functions, so nothing that imports them from there changes. It is not in `policy.py` itself because importing the `postprocess` package pulls in the router, which imports the release sources, so a source could not import from it without a cycle. One behaviour difference, and only in the fallback: with the audiobook setting unset, or emptied, post-processing and IRC used to accept m4b and mp3 and now accept the full default list like slskd and the settings page already did. A configured list is read exactly as before. Direct download's own helper used the raw configured list and now reads the same normalized one, which is identical for any list the settings page can save. Also public now: `parse_size` in the Prowlarr source (slskd and Newznab imported it with the underscore) and `coerce_completed_path_timeout_seconds` in the client base handler (slskd imported that too). Not changed on purpose: slskd's `_coerce_int` looks like a copy of `coerce_int` but is not. It reads `"12.0"` as 12 and `True` as 0, where `coerce_int` gives 0 and 1. And the `_parse_size` in the IRC source is a different function, string to bytes. Tests: a new set checks that post-processing, IRC and slskd all fall back to the shared defaults and that the settings default is the shared one, and the scan-archive and IRC tests now patch the shared config instead of the per-module copies. Full suite passes. |
||
|
splitsec2
|
606e94fde2 |
fix(settings): offer the seed settings for qBittorrent only (#1574)
The two seed settings, Give Up With No Seeders and Minimum Seeders, show up for every torrent client, but only qBittorrent reports a seed count. With Transmission, Deluge or rTorrent you can set both and nothing happens. The first one says "Only qBittorrent reports seed counts" at the end of its description, the second says nothing about it. Both are now shown only when qBittorrent is the selected client, the same `show_when` the other qBittorrent fields use, and Minimum Seeders gets the same sentence. A value someone already saved for another client stays in their config, it just is not shown, and it had no effect before either. `docs/environment-variables.md` is regenerated for the changed description, which is two lines. One test checks that both fields are tied to qBittorrent and say so, and it fails without the change. |
||
|
splitsec2
|
3633dd3d97 |
fix: keep file links and API keys out of AllDebrid logs, fix Newznab debug log (#1559)
Two log fixes in the same area, both found while going through the open
code scanning alerts.
AllDebrid: the unlock path logged the file link in full. The last-resort
fallback appends `apikey=` to the link and returns it, and the "Failed
to download from ..." error carries that same link into
`logger.exception`, so the key could land in the log and the traceback.
The links are now logged through `_safe_url` (scheme, host and path
only). The GET unlock fallback also logged the exception text at debug
level, and requests puts the request URL in that message, which includes
`apikey=`. That one now logs the exception type only. This covers code
scanning alert 124.
Newznab: `logger.debug("Newznab API: GET %s params=%s", url, *params)`
star-expands the dict keys into the log arguments, so with debug logging
on it raises `TypeError: not all arguments converted during string
formatting` and the line never prints. Logging `params` as is would
print the API key, so it now logs the sorted parameter names.
Tests: the Newznab one formats the logged line and checks the key is
absent. The AllDebrid ones check the fallback and failure paths log no
query string and that the fallback still returns the link with the key.
All three fail without the change. Ruff and basedpyright are clean,
`tests/newznab` and `tests/prowlarr` pass.
|
||
|
splitsec2
|
2668705bfa |
perf: reuse connections and cache Anna's Archive download counts (#1546)
Two small changes that stop Shelfmark from opening a new HTTPS connection for requests that go to the same host over and over. Anna's Archive download counts. Every search result gets its count from `/dyn/md5/inline_info/<md5>`, and each of those was a bare `requests.get`, so a search with 50 results opened 50 TCP and TLS connections to the mirror. `search_books` also runs once per title variant, and the variants of one book mostly find the same files, so the same md5 was looked up again for every variant that found it, and again when the release list was reopened. The lookups now share one `requests.Session`, and a successful count is kept for five minutes in a bounded `CacheService` (the same class the metadata cache uses, as its own instance, so it does not depend on the metadata cache toggle). A lookup that fails or comes back without a count is not cached, so it is retried on the next search just like today. The URL, headers and 5 second timeout are unchanged. Debrid clients. Real-Debrid, AllDebrid, Debrid-Link and TorBox also used the bare `requests` helpers. The download handler polls `get_status` every two seconds for each active job, so every poll was a fresh TLS handshake with the debrid API. Each of those modules now keeps one `requests.Session` and sends its API calls through it, the way the qBittorrent, Deluge, Prowlarr and Newznab clients already do. Headers, timeouts, `verify` and error handling are still passed per call exactly as before. The existing tests that patched `<module>.requests.get/post/request` now patch `<module>._session.<verb>` instead; nothing else in them changed. How I checked it: each change has a test that fails on main and passes here. They count the requests that reach `requests.Session.request` and the sessions built while doing it, with no network. On main, three polls of each debrid client build three sessions, and two overlapping batches of AA results look `bbb` up twice. There are also tests for the count cache expiring (with a faked clock) and for a failed lookup not being cached. Full suite: 4432 passed, 48 skipped. ruff and basedpyright are clean on the changed files. I also looked at caching `/api/releases` on the server and left it out. The release modal already caches each book's results per source and content type in the browser for five minutes (longer for IRC and slskd through `cache_ttl_seconds`), and that cache survives closing and reopening the modal, so reopening it does not run a new search today. A server cache would only help after a page reload, and it would quietly undo the modal's "apply filters" and manual search refetches, which drop the browser cache on purpose, unless the frontend also learned to send a bypass. That did not seem worth it for this PR. |
||
  
|
c43da4537d |
fix(network): apply HTTP proxy to Open Library and Prowlarr requests (#1526)
Fixes #1524 Open Library metadata searches and Prowlarr API calls used `get_ssl_verify` but never `get_proxies`, so the HTTP proxy set in Settings → Network was not applied to their HTTPS requests. Both now pass the configured proxies. Relying on `requests`' `trust_env` with only `HTTP_PROXY` set leaves the same gap for `https://` destinations. Verification (Linux, `uv run --frozen pytest`): - New tests `tests/metadata/test_openlibrary_proxy.py` and `tests/prowlarr/test_api_proxy.py`: 4 passed. With the source change reverted they fail (2 failed, 2 errors). - `tests/metadata` + `tests/prowlarr`: 949 passed, 42 skipped. - The full suite in my environment showed 61 failures and 26 errors outside these areas. I did not compare them against `main`, so I'm not claiming the full suite is green here; CI will show it. - `ruff check` and `ruff format --check` on the changed files pass. Prepared with AI assistance; reviewed and tested by me before submitting. Co-authored-by: dbdefault <246079841+dbdefault@users.noreply.github.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
 Desislav LesovandCaliBrain
|
fc8a5b9230 |
fix(prowlarr): hide results whose title extension is not a supported format (#1490)
Closes #1481 ## Problem Plain (non-enriched) indexers such as Zamunda, Knaben or LimeTorrents return titles like `Иван Вазов - Под игото.doc`. They show next to the epubs. Picking one downloads the file, then post-processing discards it because `doc` is not in `SUPPORTED_FORMATS`. ## Change - `_unsupported_title_extension()` reads a trailing file extension from the release title. When it is a known file format that is not in Supported Book Formats, the result is hidden. - Kept: titles with no extension, `[EPUB]`-style tags, dotted titles like `Jane.Eyre.by.Bronte` (only known file extensions count), archives (`zip`, `rar`, `7z`), and audiobook searches. - New Prowlarr setting `PROWLARR_HIDE_UNSUPPORTED_FORMATS` ("Hide results in formats you did not enable"), default on. - One info log line per search with the hidden count; a debug line per hidden title. - `docs/environment-variables.md`: added only the new setting's rows. Running `scripts/generate_env_docs.py` on current `main` also changes unrelated lines (`SHELFMARK_API_KEY_READONLY`, the slskd anchor), so I left those alone. ## Tests - New `TestHideUnsupportedFormats` in `tests/prowlarr/test_source.py`: the extension helper (parametrised), audiobook bypass, comma-separated config, and a full `search()` with the setting on and off. - `tests/prowlarr`: 784 passed. Ruff lint/format, vulture and basedpyright on the changed files are clean. - Full suite on macOS: the same 84 bypass/entrypoint failures as on `main` (browser extra and Linux permissions), none new. I ran an earlier version of this as a local patch on v1.4.0 against a real Prowlarr + Zamunda setup: the `.doc` row is gone and epubs stay. Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
  amura11andClaude Sonnet 5.5
|
ceb23dda7c |
feat(clients): separate audiobook download directory for torrent clients (#1500)
## Summary Related to #1048 (covers the "separate book and audiobook locations" request, not per-category routing). Every download client already had a separate audiobook label/category, but the four torrent clients only had a single save directory, so books and audiobooks ended up mixed in the client's download folder. Each torrent client (qBittorrent, Transmission, Deluge, rTorrent) gets an opt-in **Separate Book and Audiobook Directories** toggle (`<CLIENT>_SEPARATE_DOWNLOAD_DIRS`, default off): - **Off (default):** unchanged. The single `<CLIENT>_DOWNLOAD_DIR` is used for everything, so existing setups behave exactly as before. - **On:** `<CLIENT>_DOWNLOAD_DIR_BOOK` and `<CLIENT>_DOWNLOAD_DIR_AUDIOBOOK` replace the shared field. A directory left empty means the client's own default; it never falls back to the shared directory or another Shelfmark setting. The content type was already passed to each client's `add_download`, so a small shared helper (`resolve_download_dir`) does the selection. Usenet clients are unchanged: their save location comes from the category configured in the client, and a separate audiobook category is already sent. ## Screenshots Default Setting <img width="817" height="271" alt="Screenshot 2026-10-20 at 14-54-26 Shelfmark" src="https://github.com/user-attachments/assets/29941f83-9034-4ea5-bcb2-8e91ce77e63a" /> Separate Locations <img width="817" height="425" alt="Screenshot 2026-10-20 at 14-54-58 Shelfmark" src="https://github.com/user-attachments/assets/772b7194-a072-4945-bac5-b6959550fd75" /> ## Not included Routing to many folders by document type (the main request in #1048) still only exists for Direct download via Content-Type Routing. Extending it to torrent and usenet sources needs a decision on what to route by. ## Testing - `pytest tests --ignore=tests/e2e` passes (3561 passed); ruff clean. The `tests/e2e/platform` suite needs a running cluster and was not run. - New unit tests for the helper (including no fallback to the shared directory) and a qBittorrent add test covering book vs audiobook directories. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
CaliBrain
|
49aeec97cd |
fix(download): stop deleting user data and losing completed downloads (#1511)
Fixes six defects. Each one breaks downloads or deletes user data, and each has a regression test that fails on |
||
|
splitsec2
|
61c7ad3c62 |
Merge commit from fork
* fix(torrent): check each redirect hop when fetching a torrent file The torrent file fetch followed redirects by hand without looking at where a hop pointed, so a download link could make the server request a loopback, link-local or LAN address. Each hop (the initial URL and every redirect Location) must now be http or https and resolve only to global addresses. The configured Prowlarr/Newznab origin is exempt, as it usually lives on the LAN. The API key still only goes to that origin. * fix(socketio): only accept websocket handshakes from the app's own origin The Socket.IO server allowed every origin. It now uses the same-origin check python-engineio applies by default (Host, or X-Forwarded-Proto and X-Forwarded-Host behind a proxy). SOCKETIO_ALLOWED_ORIGINS takes a comma separated list of extra origins for proxies that change Host without forwarding it. * fix(custom-script): do not pass a file name that starts with a dash as the first argument In relative path mode the script got the organized file's relative path as its first argument, which many scripts read as an option when the name starts with a dash. Such a name is now passed with a ./ prefix. Absolute mode is unchanged. |
||
|
splitsec2
|
7da4669cc1 |
feat(prowlarr): hide ebook results above a maximum size (#1504)
Closes #844. A new Prowlarr setting, Maximum Ebook Size (MB). Ebook results above it are hidden from the release list. The default is 0, which shows everything as it does today. This is the idea from the issue. The filetype check suggested in the comments is a different change (#1490 has one for Prowlarr), and it doesn't catch a TV series or a software bundle that ships as a plain archive, which is what a size limit does. - A result with no reported size is always shown. - A result exactly at the limit is shown. - Audiobook searches are never limited, since they are big by nature. - A non-numeric value counts as no limit. The filter runs on the raw Prowlarr result before it is turned into a release, so a hidden result never reaches the sort or the release cache. Tests cover no limit, zero, over the limit, exactly at it, no size, audiobook searches and a bad value. Ruff, basedpyright and the backend suite pass. I added the entry to `docs/environment-variables.md` by hand because running the generator also changes unrelated parts of that file. |
||
  sean0407andSean Li
|
a0ffb268b2 |
fix: wait for synced files before processing completed downloads (#1464)
## Summary
```text
wait for completed download path
regular file -> keep existing immediate handling
directory -> inspect files enabled for this task's content type
wait until the supported-file signature is stable across checks
```
Fixes the remote-sync race in #1463. A mapped torrent folder can appear
before its large media file finishes copying, so path existence alone
does not mean the download is ready.
## Evidence
- **Before:** In #1463, qBittorrent reported a multi-file audiobook
complete while the destination contained only `.cue` and `.jpg`;
Shelfmark immediately post-processed the folder and failed with “No book
files found in download.” Retrying after the `.m4b` sync finished
imported it.
- **After:** A regression test starts with the cue sheet and cover
present, then simulates the `.m4b` file appearing and growing. Path
waiting returns only after the supported file's size and modification
time remain stable across checks.
- Credential-free isolated `python -m pytest
tests/prowlarr/test_remote_path_mappings.py -q -n 0 -o addopts=''`: 11
passed.
- `ruff check shelfmark/download/clients/base_handler.py
tests/prowlarr/test_remote_path_mappings.py`: passed.
## Merge Danger
**Door:** Two-way
**Blast Radius:** Download handling
The wait uses the existing completed-path timeout and leaves regular
single-file downloads on their current path.
Co-authored-by: Sean Li <22557931+sean0407@users.noreply.github.com>
|
||
 ultraviolet150
|
5f3647a496 |
fix(clients): wait for qBittorrent-compatible clients to finish before post-processing (#1465)
## Summary
`QBittorrentClient.get_status()` treats a torrent as complete as soon as
`progress >= 1.0`, excluding only the `moving` state. Some
qBittorrent-compatible clients report full progress before they are
finished. rdt-client (v2.0.142) reports `progress = 1.0` once the debrid
and local downloads are done, but keeps `state = "downloading"` for a
few seconds while it finalises the torrent, and until then reports a
guessed `content_path`. It only switches to `pausedUP` with the real
path afterwards.
Shelfmark picks the torrent up inside that window, so post-processing
runs against a path that isn't there yet and fails within milliseconds
("No book files found" / `post-processing failed`). Clicking Retry then
imports it fine. Every rdt-client download hits this.
## Change
- Add `_INCOMPLETE_STATES` (all download-side states, plus `moving`) and
only mark a torrent complete when `progress >= 1.0` **and** its state is
not one of those.
- Real qBittorrent leaves these states when a torrent reaches 100%, so
its behaviour is unchanged.
## Example log (before)
```
12:04:43,215 base_handler - Remapped download path for qbittorrent (<hash>): <rdt-client download dir>\<torrent name>\ -> /downloads/<torrent name>
12:04:43,216 orchestrator - download finished; starting post-processing
12:04:43,232 orchestrator - WARNING - post-processing failed
```
## Testing
- New test
`test_get_status_full_progress_still_downloading_not_complete`: fails
before the change, passes after.
- `tests/prowlarr/`: all passing. `ruff check`, `ruff format --check`
and `basedpyright` clean on the changed files.
- Running the patched client against rdt-client: downloads now import
without Retry.
Related to #1463 / #1464, which handle the same symptom when the client
really is finished but files are still syncing to local disk. This PR
addresses a different cause: the client reports 100% progress while
still in a downloading state, so Shelfmark shouldn't treat it as
complete yet. The two changes are independent and complementary. Not
covered by #741, which fixed add-time failures. Possibly related: #607.
|
||
|
splitsec2
|
be2653bdb7 |
fix(debridlink): reuse the file list from the status check (#1431)
Kept small on purpose so it's easy to review. When a torrent is ready, `get_status` has already fetched the full record with `/seedbox/list?ids=` and `_handle_torrent_info` has checked that every file is at 100%. The retrieval thread was then started without that list, so `_process_and_download` called `_fetch_file_list`, which asks the same endpoint again. That second call runs inside the thread's generic `except`, without the flood and transient error handling the status path has, so a rate limit or a hiccup there sets the phase to error for a torrent that finished fine. This passes the files already in hand into the thread. `_process_and_download` already took an optional `files` argument. The new test starts the thread from a ready status and checks it uses the given list and never calls `_fetch_file_list`. Separately, `download_url` returns a `BytesIO`, so a whole file sits in memory. AllDebrid, TorBox and Real-Debrid do the same, so I left that for a streaming variant of the shared helper instead of fixing it for one client here. Ruff and basedpyright are clean and `tests/prowlarr` passes. The one red check is the existing clock-dependent test `test_a_queued_torrent_asks_for_a_grace_once`, which fails on any runner that booted recently. It is fixed in #1426 and is not caused by this change. |
||
|
CaliBrainandClaude Opus 5.5
|
9321096032 |
fix(clients): correct Debrid-Link file listing and rate-limit handling (#1425)
Follow-up to #1380. - Fetch the file list with ?ids=, the documented parameter. There is no ?id=, so a many-file torrent was not expanded and the call could fail with badArguments at the last step of every download. - Back off status checks for 60s after floodDetected, doubling to a 240s cap, instead of an hour. The orchestrator cancels a download after five minutes without a change, so the hour-long pause cancelled every active download. Only a flood on the status check starts it, and a successful check resets it. - Drop the client-side one-hour stall timer. The orchestrator stall timer always fires first. - Keep progress at 50% when file retrieval starts instead of dropping to 0% until the first file arrives. - Report queued, paused and verifying torrents as QUEUED, PAUSED and CHECKING, so a queued torrent gets the handler queue grace. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
splitsec2
|
cad18b8019 |
feat(clients): add Debrid-Link as a torrent client (#1380)
calibrain said in #1315 that he'd review and merge a Debrid-Link client if someone took a stab at it, so here it is. It's the fourth debrid service alongside AllDebrid, Real-Debrid and TorBox and it uses the same DownloadClient interface as the other three. No frontend changes, the settings UI picks it up from the backend field definitions. **I can't test this against the live API.** I don't use debrid and I don't have a Debrid-Link key. Everything here is built from their v2 API docs and cross-checked against two independent reference implementations, so the request shapes should be right, but nobody has run a real magnet through it yet. @Tatsu941 you opened the issue and mentioned they give developers free premium. Would you be able to try it? The things most worth confirming are the field names on a seedbox torrent, that `premiumLeft` is the right premium check, and that an API error comes back the way the client expects. Happy to fix whatever turns up. Two things about their API shaped how this is written. Every v2 endpoint wraps its payload in a `{success, value}` envelope and a failure can arrive with an HTTP 200, so `_request_value` decides the outcome from the body instead of the status code. A completed seedbox torrent already carries a direct `downloadUrl` on every file. The other three clients each need a separate unrestrict or link-request call per file and this one doesn't, which is why it came out shorter than them despite doing the same job. Endpoints used: | Endpoint | Used for | |---|---| | `GET account/infos` | test the key, confirm premium time remains | | `POST seedbox/add` | magnet as JSON, .torrent as multipart | | `GET seedbox/list` | poll progress | | `DELETE seedbox/{id}/remove` | clean up | Ruff and basedpyright are clean and the 36 new tests pass. The two failures in `tests/e2e/test_proxy_auth_middleware.py` are already on main and fail the same way on a pristine checkout. They assume the first proxy-authenticated user becomes an admin, which only holds while the user table has no admin in it, so they depend on how many users earlier tests created. One thing for review: the download loop uses `download_url` like the other three clients, which holds each file in memory. I kept the existing pattern rather than do something different in a new-provider PR, but say so if you'd rather it streamed here. Closes #1315 |
||
|
CaliBrain
|
f773ac6029 |
fix(sabnzbd): strip Prowlarr API key on cross-host NZB redirects (#1424)
_fetch_nzb_content sent X-Api-Key then followed redirects with the header still attached, leaking the Prowlarr key when Prowlarr 302s to the indexer download link. Follow redirects manually with allow_redirects=False and re-evaluate headers per hop, mirroring the torrent path. |
||
 HeihoffJandClaude Opus 5.5
|
95d9f1da57 |
fix(sabnzbd): prefetch NZBs served from the indexer's own domain (#1416)
Indexers often serve NZB downloads from a different host than their API (e.g. file.indexer.example for indexer.example/api). The exact-origin check added in #967 rejected those, so SABnzbd silently fell back to addurl. Trust any host in the configured indexer's registrable domain, using the Public Suffix List so shared suffixes like co.uk or duckdns.org never widen trust. Scheme and port must still match, IP literals and single-label hosts still require an exact match, and the Prowlarr API key header is unchanged. This closes this issue: https://github.com/calibrain/shelfmark/issues/1411 Disclaimer: Implemented by Claude and Co-Authored by me. Tested and verified by me alone (Why is it always this way around and not the other) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
  Evan KazakinandEvan Kazakin
|
7888217380 |
feat(download): add Remove & Delete Files torrent completion action (#1401)
Closes #1027 ## Summary - add a **Remove & Delete Files** (`remove_and_delete`) option to the Torrent Completion Action setting - after a successful import, remove the torrent and ask the client to delete its downloaded data (`delete_files=True`), matching what the usenet "move" flow already does - clarify the setting description so it says **Remove** keeps the downloaded files (raised in the issue comments) - regenerate the `PROWLARR_TORRENT_ACTION` entry in `docs/environment-variables.md` ## Behavior The deletion runs from `post_process_cleanup`, the same place as the existing Remove and Change Category actions, so it only happens after output transfer and post-processing have succeeded. At that point every file has already been copied or hardlinked into the library, and transfer size checks have passed. A failed import never removes or deletes anything. The torrent client deletes its own data, so Shelfmark does not delete paths itself and remote path mappings are not involved. Keep, Remove, and Change Category behave as before, and Keep is still the default. If a request matched a completed torrent that was already in the client, this option deletes that torrent's data too. That is the same scope the existing Remove action already applies to. ## Validation - `make python-checks` (ruff check, ruff format, basedpyright on backend and tests, vulture): passed - `make python-test`: 3363 passed, 5 skipped, 9 failed. All 9 are in `tests/config/test_entrypoint_permissions.py` and happen because macOS `/bin/bash` 3.2 does not support `${1,,}` in `entrypoint.sh`. They do not touch this change. - new tests in `tests/prowlarr/test_handler.py`: - Remove passes `delete_files=False` and Remove & Delete Files passes `delete_files=True` - a failed import with Remove & Delete Files does not call the client - the delete case fails without the handler change - manual smoke run: `PROWLARR_TORRENT_ACTION=remove_and_delete` set through real config loading, with a stub client that deletes its folder. The torrent folder was deleted and the hardlinked library file stayed intact. - pre-commit hooks (prek) passed --------- Co-authored-by: Evan Kazakin <evan@Evans-MacBook-Pro.local> |
||
|
CaliBrain
|
bac0b93566 |
fix: keep https:// for qBittorrent and hide release sources from non-admins (#1422)
qBittorrent over HTTPS (#1417) qbittorrent-api ignores the scheme in `host` and probes HTTP and HTTPS itself. When the HTTPS probe fails, for instance on a certificate Shelfmark doesn't trust, it falls back to plain HTTP against the TLS port, and a reverse proxy answers "400 The plain HTTP request was sent to HTTPS port". The download client and the Test Connection button now pass FORCE_SCHEME_FROM_HOST for https:// URLs, so the configured scheme is used as-is and a certificate problem is reported as one. http:// and bare host:port URLs keep the probe: normalization adds http:// to a bare host, and the probe follows a proxy's redirect to HTTPS. Release sources leaked to non-admins (#1418) A download queued from an approved request carries the release an admin picked, and the requester could read it from their own activity feed: the request's release_data (source_id, indexer, info URL, torrent attributes), the download's full server path, and the download id itself, which for Prowlarr is "<indexer id>:<guid>" and for a private tracker is a URL into it. That id keys every status payload, the /api/localdownload query and the cover proxy URL, so hiding release_data alone would not have closed the leak. For non-admin viewers: - request rows keep only the release fields the activity cards display - download_path is reduced to the file name, which the browser download reveals anyway - request-linked downloads are addressed by an opaque id (a keyed HMAC of the task id) in the snapshot, history, /api/status, queue order, active downloads, dismissed keys, cover URLs and the websocket status and progress events. Routes that take a download id translate it back, searching only the caller's own downloads. Downloads a user queued directly keep their real id: the user picked that release from search results that already showed it, and the release list matches its buttons to the queue by that id. Admin views are unchanged. The activity sidebar now links a fulfilled request to its download by request_id instead of release_data.source_id. Closes #1417 Closes #1418 |
||
|
CaliBrain
|
893f7bdb92 |
fix(mam): keep the session ID on MAM and rerun Prowlarr's exact search (#1399)
Follow-up to #1390. - The origin came from each result's infoUrl, matched by a regex that did not check the host, and the mam_id cookie had no domain. Any Prowlarr indexer returning a URL like https://evil.example?myanonamouse.net/t/1 sent the session ID to evil.example. Requests now always go to https://www.myanonamouse.net (Prowlarr's only MAM URL), with the cookie as a header and redirects off. Only results from Prowlarr's MyAnonamouse indexer are looked up, and their URLs must be on myanonamouse.net. - The lookup searched every category and read one page, so for common titles most of Prowlarr's results were missed (a "Dune" audiobook search: 56 audiobooks on the first 100 of 328 matches). It now reruns Prowlarr's exact search: the same query clean-up, the MAM main categories behind the Torznab categories searched (13/15/16 for audiobooks, 14 for e-books, all once expanded), and the MAM indexer's own search type, search-in options and languages. Further pages are read while IDs are missing, page 1 of every title first, at most 4 requests per search. - Failed requests back off for 1, 2, 4 ... up to 30 minutes. The 10th consecutive failure stops enrichment until Test MAM Session passes, the session ID changes, or Shelfmark restarts. - The detail cache prunes expired entries instead of growing for as long as Shelfmark runs. |
||
|
a05e002305 |
feat: narrator, series and bitrate columns for MyAnonamouse results (#1390)
# feat: narrator, series and bitrate columns for MyAnonamouse results Addresses #605 and #934 (narrator in the release list). ## Problem Both requests were closed because the narrator isn't in Torznab results, which is correct. Prowlarr's MyAnonamouse indexer reads `author_info` but drops the `narrator_info` and `series_info` MAM returns next to it, and neither Prowlarr's `ReleaseInfo` nor the Torznab output has a field for them. Shelfmark's size tooltip already looks for a `narrator` Torznab attribute, but nothing ever sends one. When a book has several narrations, choosing one means going back and forth between Shelfmark and the tracker. ## Approach Optional, opt-in enrichment using the user's own MAM session (`mam_id`), the same approach AudioBookRequest's MAM indexer uses: 1. After the Prowlarr search, releases whose info URL is `…myanonamouse.net/t/<id>` are collected. 2. Shelfmark sends the same query text to MAM's JSON search (`/tor/js/loadSearchJSONbasic.php`, normally one request, at most 3), and matches torrents back to Prowlarr's results by torrent ID. The MAM origin is taken from the result URL, so a custom Prowlarr MAM base URL is respected, and the cookie is only ever sent to `*.myanonamouse.net`. 3. Adds `extra.narrator`, `extra.series` (`The Sun Eater #1`) and `extra.bitrate` / `extra.bitrate_value`. MAM has no bitrate field, so it's parsed from the uploader's free-text tags (`64 kbps`) and some releases won't have one. Lookups are cached per torrent for an hour, respect the existing Prowlarr search deadline, go through the configured proxy (`get_proxies`), and never fail the search: a 403, timeout or bad JSON is logged and the list renders without the extra details. ## Changes - **`release_sources/prowlarr/mam.py`** (new): small MAM client (`search`, and `get_username` for the test button), `narrator_info` / `series_info` / tags parsing, cached best-effort `lookup_torrent_details()`. A 403 includes MAM's reply and a note about the IP/ASN lock. - **`release_sources/prowlarr/source.py`**: enrichment after the result loop. Series, Narrator and Bitrate columns only when a MAM ID is configured, since otherwise they would be empty for every row. A Torznab `bitrate` attribute from other indexers is also mapped to `extra.bitrate`. - **`release_sources/prowlarr/settings.py`**: "MyAnonamouse Enrichment" section with `PROWLARR_MAM_ID` (password field, env-overridable like every setting) and a **Test MAM Session** button. - **`release_sources/__init__.py`**: `ColumnSchema` gains optional `setting_key` and `content_types`. The new `apply_column_visibility()` drops gated columns and their grid tracks. Neither field is serialized. - **`main.py`**: `/api/releases` applies `apply_column_visibility()` with the request's content type and the user's effective settings. - **`config/settings.py` / `users_settings.py`**: Search Mode › "Release List Columns" with `SHOW_SERIES_COLUMN`, `SHOW_NARRATOR_COLUMN` and `SHOW_BITRATE_COLUMN`, all default on and user-overridable. Narrator and bitrate are audiobook-only, series shows for both. AudiobookBay's existing bitrate column now follows the bitrate toggle. - **Frontend**: text cells truncate with a hover title; the mobile info line wraps and skips empty text/number cells so blank optional columns don't leave orphan `·` separators; the size tooltip no longer lists Bitrate twice. - **Docs**: new `docs/myanonamouse-enrichment.md` (linked from the index), and a regenerated `environment-variables.md`. The regeneration also picked up a few pre-existing drifts from `main` (the Libgen section, `AA_DEFAULT_SORT` default, a duplicate `BOOK_LANGUAGE` row). I can drop those if you'd rather keep this diff focused. ## ⚠️ MAM sessions are IP/ASN-locked MyAnonamouse locks each session to one IP or ASN. Reusing the session Prowlarr (or a seedbox script) uses will often **403**. **A separate MAM session for Shelfmark will likely be needed** when Shelfmark reaches MAM from a different IP (another host, a VPN container, or a proxy in Shelfmark's Network settings), or when the existing session is ASN-locked to another network. The setting's description, the error message and the new doc all say so. ## Testing - `tests/prowlarr/test_mam_enrichment.py` (new, 19 tests): parsing (narrator dedupe, multiple series, missing numbers, malformed JSON, tag bitrate), lookup (stops once all IDs are found, cache, 403 and connection errors return empty, expired deadline skips the request), only MAM releases enriched, Torznab bitrate mapping, column config with and without a MAM ID for audiobook and ebook, toggles, grid-track removal, and gates not serialized. - `tests/core/test_admin_users_api.py`: the curated search-preference key list now includes the three toggles. - Full `pytest -m "not integration and not e2e"` compared with an upstream `main` worktree on the same machine: no new failures. The remaining ~115 failures on both are Windows-only (tor/entrypoint shell tests, path separators). - `ruff check` / `ruff format --check` / `basedpyright` (0 errors) / `vulture` on touched files; frontend `tsc --noEmit`, `oxlint`, `oxfmt --check`, `vitest` (201 passed). - Manually verified with a real MAM account on a Docker build of this branch: the test button, then narrator, series and bitrate on MyAnonamouse audiobook results. No behavior change unless `PROWLARR_MAM_ID` is set, apart from the bitrate toggle on AudiobookBay (default on, same as today). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
splitsec2
|
1a5b37d9d3 |
fix(deluge): send seeding ratio limit under Deluge's own keys (#1367)
Deluge's per-torrent options are `stop_at_ratio` (bool) and `stop_ratio` (float), and `torrentmanager` checks `options['stop_at_ratio'] and get_ratio() >= options['stop_ratio']`. We were putting the indexer's float into `stop_at_ratio`, which only switched stopping on and left the daemon's global ratio (default 2.0) as the one actually enforced. A `ratio_limit` of 0 turned stopping off entirely. `stop_at_ratio_enabled` is not a Deluge option at all. Deluge has no per-torrent seeding time limit, `seed_time_limit` is a global core preference, so the value is logged as unapplied instead of sent as a key the daemon drops. qBittorrent and Transmission already honour both indexer limits, so this removes a silent difference between clients. ## Verification - `tests/prowlarr/test_deluge_client.py`: the ratio arrives as `stop_ratio` with `stop_at_ratio` set, and no key Deluge does not define is sent. Both fail on current main and pass here. - Full suite (3165), ruff, ruff format, basedpyright, vulture green. |
||
 Marcelo Rodrigo
|
b7002a6eca |
feat: Add TorBox client support and settings integration (#1342)
Add **TorBox** as a torrent download client for Prowlarr releases. Users can select `TorBox` in the download client settings, configure it with the new `TORBOX_API_KEY` environment variable, and verify their credentials with the connection test button. The integration supports both magnet links and `.torrent` files. It tracks the torrent lifecycle through TorBox, downloads supported book and audiobook files from the TorBox CDN, preserves safe nested file paths, and cleans up remote and local download state. Important: Shared HTTP download logs omit full download URLs and URL-bearing exception text to avoid exposing credentials, following best practices. This applies to all clients that use the shared `download_url()` path; URLs remain available to the HTTP operations themselves. --- There is already related work in progress in #1173, which includes both torrent and direct-download support for TorBox. This PR is not intended to replace or compete with that contribution. It offers the tested torrent client functionality as a smaller, focused change that can make TorBox available to the community sooner. The direct-download integration proposed in #1173 remains valuable and could be reviewed or introduced separately. Automated tests cover configuration, connection validation, magnet and torrent-file submission, API errors, status and progress handling, file retrieval, path traversal protection, cancellation, cleanup, and sensitive URL redaction. I also validated the complete flow locally with several magnet links and `.torrent` downloads. TorBox processed the torrents and Shelfmark downloaded the resulting files as expected. AI was used to help with the implementation, with human validation. This PR and long description? Took me some good minutes at night after work, but gives me joy to open this PR to share with the community this improvement. |
||
|
CaliBrain
|
1b17fe179a |
fix(irc): rank a surname-only result as partial, not wrong (#1332) (#1334)
"David Petrie" as "D. Petrie", then ranked the answer by the full name to recover the precision the surname gave up. The two halves disagreed. author_affinity needs two agreeing tokens before it calls a name the same person, so "Petrie" - the name on the filenames a surname search exists to reach - matched one and came back AUTHOR_MISMATCH. It therefore sorted below "Unknown" and level with "Gordon Petrie", a different author who merely shares the surname. The widened query pulled those rows in and the ranker buried them. Falling short of agreement is now separated from disagreeing with it. A name whose every token fits the one asked for is an abbreviation of it and ranks AUTHOR_PARTIAL, between agreement and "no author reported"; a name carrying a token that fits nothing still ranks AUTHOR_MISMATCH. Nothing that agreed before changes tier - "Homer"/"Homer Simpson" is still a match, since the extra token must not demote a mononym that already met its one-token requirement - so Prowlarr's #1293 ordering is unchanged except that a tracker listing a bare surname stops being read as the wrong author. Measured on the issue's own case, wanted "David Petrie": before: D Petrie, Unknown, Petrie, Gordon Petrie after: D Petrie, Petrie, Unknown, Gordon Petrie Second fix, same release: a book with no title posted the surname on its own. _build_query fell back to book.search_title or book.title, which is empty on exactly the path where the plan has no title variants, so the line reaching the channel was "@search Petrie" - not a search for anything, and the kind of bare over-broad post is_available refuses unaddressed queries to avoid. It now returns "" and the existing "No search query could be built" guard takes it. Tested with make python-lint, python-format, python-dead-code, python-typecheck and python-test. |
||
 Zoltán Szabó
|
35037b35fd |
fix(irc): search by surname, and rank the answer by author (#1331) (#1332)
Fixes #1331. A search bot ANDs every term against a filename, so the given name is the term that empties the result set. Measured against irchighway's #ebooks: "Revelations David Petrie" is answered "no results", "Revelations Petrie" returns 9 matches, 6 of which parse, all filed as "D Petrie". The query now carries the title and the surname, read off the search variant so the ISBN fallback and a manual query - which set author="" on purpose - keep their current shape. Title-only, the shape #1295 settled on for Prowlarr, does not transfer: the bot caps an answer at 1000 matches, and a bare "Revelations" hits that cap with 923 parsed rows across 500 authors, so the cap itself can drop the wanted book. The surname is the token the two spellings share and it keeps the answer small. The full author then orders what comes back, reusing author_affinity from #1295, since a surname also matches a different author who shares it. It sits under server availability the way indexer priority does in #1295: a download addresses one named bot and waits 120s for it, so a match from a bot that has left the channel must not outrank a mismatch that can answer. Ranking runs on the way out rather than before the cache, because one query identity is shared by every book that produced that query. Two things found while testing: - The parser writes the literal "Unknown" when a filename has no " - " separator (parser.py:168). Ranked literally that sorts as a wrong author, so author_affinity's middle tier was unreachable here; it is now read as absent. 5 of those 923 rows are affected. - author_affinity moves to shelfmark/core/author_match.py, unchanged, so IRC does not import from the Prowlarr package. Prowlarr behaviour is untouched and its tests pass as they are. The three IRC assertions in the #1252 regression file move to the surname form. The invariant they pin - one contributor's name reaches the query, never the whole credit list - is unchanged. Tested with make python-lint, python-format, python-dead-code, python-typecheck and python-test, and end to end against irchighway with the patched source: it posts "Revelations Petrie" and returns 6 releases. |
||
 Jorge Lima
|
c45d342931 |
fix(prowlarr): skip indexers in Prowlarr failure back-off (#1324)
## What
Read `/api/v1/indexerstatus` once per search and skip indexers whose
`disabledTill` is still ahead. Skipped is neither attempted nor failed.
One client method, one counter on `_IndexerSearchOutcome`, ten tests.
## Why
Prowlarr's own search leaves out an indexer it has disabled after
repeated failures. Shelfmark queries each indexer through its Torznab
endpoint, which answers 429 instead:
```
Prowlarr Torznab error response: <error code="429" description="Indexer is disabled till 09/09/2026 15:00:34 due to recent failures." />
Prowlarr: 1 of 5 indexer searches failed (indexer 2 search failed: 429 Client Error: Too Many Requests ...)
Release search failed for source prowlarr: 1 of 5 indexer searches failed (...)
```
That counted as a failed search, so with one indexer in back-off and the
other four answering empty, `/api/releases?source=prowlarr` returned 503
for every book for the length of the back-off (one hour here).
`/api/v1/indexerstatus` on Prowlarr 2.5.2:
```json
[{"indexerId": 2, "disabledTill": "2026-09-09T15:00:34Z", "mostRecentFailure": "2026-09-09T14:00:34Z", "initialFailure": "2026-09-09T14:00:34Z"}]
```
## Behaviour
| indexers | before | after |
|---|---|---|
| 1 in back-off, 4 answer empty | 503 "1 of 5 indexer searches failed" |
"No releases found" |
| 1 in back-off, 1 answers with releases | releases | releases, one
Torznab call fewer |
| 1 in back-off, 1 times out, 3 answer empty | "1 of 5 failed" | "1 of 4
failed" |
| every indexer in back-off | 503 "5 of 5 failed" | "every indexer is
disabled by Prowlarr after recent failures (until ...)" |
| status endpoint unreachable | n/a | as before, nothing skipped |
Auto-expand no longer retries a pass in which nothing was asked.
## Tests
`uv run pytest tests/prowlarr`: 563 passed, 42 skipped. `ruff check` and
`ruff format` clean.
|
||
 Atirna
|
265da07d7f |
feat(download): add Blackhole torrent handoff (#1312)
## Why Blackhole users need Shelfmark to hand a torrent file to their existing downloader instead of importing the downloaded book itself. ## Change - add Blackhole as a torrent client with a configurable watched directory - prefer a fetched `.torrent` file for Blackhole while preserving magnet preference for other clients - complete the queue task after the handoff without invoking book post-processing ## Verification - `uv run pytest -q tests/prowlarr/test_blackhole_client.py tests/prowlarr/test_handler.py tests/newznab/test_handler.py tests/download/test_orchestrator_lifecycle.py` - `uv run basedpyright shelfmark/download/clients/blackhole.py shelfmark/download/clients/__init__.py shelfmark/download/clients/base_handler.py shelfmark/download/clients/settings.py shelfmark/download/orchestrator.py shelfmark/release_sources/__init__.py shelfmark/release_sources/prowlarr/utils.py shelfmark/release_sources/prowlarr/handler.py shelfmark/release_sources/newznab/handler.py tests/prowlarr/test_blackhole_client.py tests/prowlarr/test_handler.py tests/newznab/test_handler.py tests/download/test_orchestrator_lifecycle.py` Fixes #1229 |
||
|
Matt Van HornandMatt Van Horn
|
9f11e83e1f |
fix: keep polling queued Real-Debrid torrents (#1303)
Add `queued` to the existing set of non-terminal Real-Debrid torrent states so `_handle_torrent_info` returns an in-progress `DownloadStatus` and leaves the mutable download state eligible for subsequent polling. Keep the change within the existing status-classification path rather than introducing a new helper or changing the broader handling of unknown statuses. The native Real-Debrid client currently treats the documented `queued` torrent status as a terminal error because it is absent from `_STATUS_DOWNLOADING`. This occurs after a torrent has been added and its files selected, particularly for uncached torrents that wait before downloading. A torrent-info payload with `status: queued`, zero progress, and a filename returns a non-complete `DownloadState.DOWNLOADING` result rather than `DownloadState.ERROR`; After handling `queued`, the internal download state remains non-terminal so a later status poll can be processed instead of returning a cached error. Fixes #1268 Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> |
||
|
CaliBrain
|
cb690b45b8 |
fix(prowlarr): rank releases by author instead of querying for it (#1293) (#1295)
MyAnonamouse is the only indexer Shelfmark treats as enriched, and it
alone was sent {title} {author} while every other indexer got the title
on its own. MAM matches all search terms conjunctively, so whenever the
metadata provider spelled the author differently to the tracker -
Hardcover says Timothy Ferriss, MAM lists Tim Ferriss - the search came
back empty and the UI reported No releases found for this book, with the
release sitting on the tracker the whole time.
The enriched flag is a statement about responses: MAM returns clean
author and bookTitle attributes, which is why it earns format detection
and preferential ordering. Using that same flag to shape the request is
the actual defect, and it is why turning the flag off recovers the
search but takes format detection down with it.
So the query is title-only for every indexer now, and the author orders
the results rather than narrowing them. MAM already hands us its author
field, so agreement is judged on data we hold instead of by an AND we
cannot control. The ranking is three-way on purpose - agrees, no
metadata, disagrees - so an indexer reporting no author does not sort
below one reporting the wrong author.
A wrong verdict costs a release its position, never its visibility: a
transliteration such as Dostoevsky against Dostoyevsky sorts last
instead of vanishing. That is what makes the loose token comparison safe
to ship without a tuning knob.
Falling back to a title-only query on zero results was the alternative.
It only rescues total failure - if two of six editions happen to use the
provider's spelling, the search returns those two, no fallback fires,
and the user quietly gets a truncated list. It also spends a round trip
inside the search deadline and stacks a retry on an indexer that may
still be solving a challenge (#1249).
Manual queries skip author ranking: they are the user's own words and
should not be reordered against the metadata they were typed to
override.
|
||
|
CaliBrain
|
97e289ae13 | fix: search, Prowlarr and qBittorrent follow-ups (#1276, #1283) (#1284) | ||
|
Jorge Lima
|
c95ee72ad5 |
fix(qbittorrent): keep magnets whose metadata is still pending (#1282)
## Problem
`QBittorrentClient.add_download()` waits 20 × 0.5 s for qBittorrent to
leave `metaDL`, then raises:
```
Failed to add to qbittorrent: Torrent metadata resolution was not confirmed within the visibility grace period
(response=TorrentsAddedMetadata({'added_torrent_ids': [], 'failure_count': 0, 'pending_count': 1, 'success_count': 0}))
```
The wait exists to learn qBittorrent's primary torrent ID, which for
hybrid torrents switches from the v1 hash to the truncated v2 hash once
metadata resolves. A magnet on a thin public swarm routinely needs
longer than 10 s to find a peer that will serve metadata, and the
download is then abandoned even though the add itself succeeded. The
torrent stays in qBittorrent (`base_handler` logs "leaving in
qbittorrent") and often completes minutes later with nobody watching it.
Seen on v1.3.12 with public indexers through Prowlarr: every magnet-only
release failed this way, while `.torrent` releases from a private
indexer were fine. qBittorrent showed the same torrents at `metaDL 0%
seeds=0/0`, and they resolved on their own well after shelfmark had
given up.
## Change
Return the info hash we already have instead of raising when the grace
period expires. Reads then resolve either identity:
- `get_status()` and `get_download_path()` use `_resolve_torrent()`
instead of `_get_torrent_info()`, so a v1 hash still matches after
qBittorrent re-keys the torrent to v2. `_torrent_matches_download_id`
already compares `hash`, `infohash_v1` and `infohash_v2`.
- `remove()` and `set_category()` address the torrent by its current
primary hash through a new `_current_hash()` helper, which falls back to
the ID it was given when the torrent cannot be resolved.
- The two magic numbers become `_METADATA_WAIT_POLLS` and
`_METADATA_WAIT_INTERVAL_SECONDS`.
The happy path does not change. When metadata resolves inside the grace
period the resolved primary hash comes back as before, and
`_resolve_torrent()` tries the exact-hash lookup first, so it costs no
extra request.
## Tests
`test_add_fails_when_metadata_never_resolves` asserted the old
behaviour, so it becomes
`test_add_keeps_torrent_when_metadata_never_resolves` and asserts the
info hash is returned.
`test_get_status_resolves_hash_after_metadata_switch` is new: it reads
status by the v1 hash after qBittorrent reports the torrent under its v2
hash.
`uv run pytest tests/ --ignore=tests/e2e` gives the same 55 failures
with and without this change (they are all in `tests/bypass/` and need
Chrome, which my machine has no headless setup for), and
`tests/prowlarr/` is green at 524 passed. Ruff check and format are
clean. I have not run this branch against a live qBittorrent, so a
second pair of eyes on the `remove()` path would help.
|
||
 Lance Marks
|
f441b85da2 |
feat(packs): inspect multi-book releases and file each book separately (#1270)
## Multi-book packs: inspect a release before download and file each book separately Closes #576 ### Problem One queued release is always treated as one book. When a torrent is actually a whole series (`Series/Book 1 - Title/…`, or a flat folder of `Series 1.0 - Title.m4b` files), post-processing walks the whole tree, flattens every file into one list and renames them `Title - 01…10` under the searched book's `{Author}/{Title}`. Audiobookshelf then sees a single 10-file "book" and the user has to re-file everything by hand. ### What this does Most releases expose their file list *before* anything is downloaded, so the split is decided up front and approved by the user, then the download is fire-and-forget: 1. **Inspect** – clicking a release's download button now calls `POST /api/releases/inspect` first. A new optional `DownloadHandler.list_files(release_data)` hook returns the release's files without downloading: - **AudiobookBay** reads the torrent file table off the detail page it already fetches (the page is now cached for 120 s, so inspect + download cost ABB one request). - **Prowlarr** parses `info.files` from the `.torrent` it already fetches (the existing 120 s torrent-fetch cache is reused). Magnet-only and usenet releases report "can't inspect". - Other sources default to `None`. 2. **Review** – if the plan contains more than one book, the Find Releases modal swaps the list for a review panel: one row per book with editable title / series position / year, expandable file lists, non-book sidecars (`.txt`, covers) shown as ignored, a "Treat as a single book" switch, and **Download N books**. Single-book releases queue immediately, exactly as before. 3. **File** – the approved plan travels with the task (`DownloadTask.book_plan`, retry-safe) and post-processing files each book through the existing transfer code, one book at a time (`dataclasses.replace(task, title=…, series_position=…, year=…)`), so organize/rename templates, part numbering (now scoped per book), hardlinks, torrent copy-preserve and usenet handling are unchanged. Status reads `Complete (N books, M files)`. 4. **Fallback** – when a release can't be inspected the user gets a toast, and a small "Multi-book pack" toggle in the modal header forces a heuristic split (subfolder = book, or one book per file when the file names carry series positions). Planning lives in `shelfmark/download/postprocess/packs.py` and is shared by the inspect endpoint and post-processing, so what the user approved is what gets filed. The name parser strips `Book 3 -`, `03 -`, `1.0 -`, `3.`, `[03]`, `#3`, a leading series name, labels like "An Expanse Novella -", repeated titles (`Gods of Risk 2.5 - Gods of Risk`) and a trailing `(Year)`; author and series name come from the book that was searched, and the searched book's own series position is never applied to its siblings. ### Files - `shelfmark/download/postprocess/packs.py` (new) – `PackFile/PackBook/PackPlan`, `plan_pack`, `parse_pack_book_name`, `group_files_into_books`, `match_plan_to_files` - `shelfmark/core/release_inspect_routes.py` (new) – `POST /api/releases/inspect` - `shelfmark/release_sources/__init__.py` – `DownloadHandler.list_files` hook - `shelfmark/release_sources/audiobookbay/{scraper,handler}.py` – detail-page cache, `extract_file_list`, `list_files` - `shelfmark/release_sources/prowlarr/handler.py`, `download/clients/torrent_utils.py` – `extract_file_list_from_torrent`, `list_files` - `shelfmark/core/models.py`, `download/orchestrator.py` – `multi_book` / `book_plan` fields, queue + retry serialization - `shelfmark/download/postprocess/transfer.py`, `pipeline.py`, `outputs/folder.py` – per-book transfer branch and status message - `src/frontend`: `components/PackReviewPanel.tsx` (new), `ReleaseModal.tsx`, `App.tsx`, `services/api.ts`, `types/index.ts`, `utils/releasePayload.ts` (payload builder moved out of `App.tsx`), `utils/packReview.ts` - `docs/dev/release-sources-plugin-guide.md` – documents the `list_files` hook ### Out of scope (follow-ups) - Listing files from an NZB (Shelfmark already fetches the bytes; `<file subject>` names are noisy) - Inspecting magnet links via qBittorrent's files API after a paused add - BookLore / email outputs (they ignore `book_plan`; noted in code) - The combined ebook + audiobook flow ### Testing **Automated** (`make checks`, `make python-test`, `make frontend-test` all green; the only failures on my machine are the pre-existing `tests/config/test_entrypoint_permissions.py` cases, which need bash ≥ 4 and fail identically on `main` under macOS bash 3.2): - `tests/download/test_packs.py` – name parsing (markers, series name, novella labels, repeated titles, bare numeric titles like `1984`), nested / flat / mixed / deeper-nested packs, single wrapping folder not treated as a pack, plan-to-disk matching with basename fallback - `tests/core/test_processing_packs.py` – full `post_process_download` runs on a real temp filesystem: approved plan files each book under its own `{Author}/{Title}`, heuristic split of a nested pack, searched book's series position does not leak, multi-file book inside a pack keeps `- 01/- 02` per book, hardlinked torrent pack leaves the seeding tree intact, no pack fields ⇒ behaviour unchanged, single group degrades to the searched title, status message - `tests/core/test_release_inspect_routes.py` – plan response, not-inspectable, handler errors never 500, unknown source / missing `source_id` ⇒ 400, login required - `tests/audiobookbay/test_file_list.py` – file-table scraping from real ABB markup (multi-file and single-file pages), handler host validation, one page fetch shared by magnet + file list - `tests/prowlarr/test_torrent_file_list.py` – multi-file / single-file `.torrent` parsing, handler behaviour for torrent URL vs magnet vs usenet vs cache miss - `tests/download/test_orchestrator_pack_fields.py` – queue-time parsing and retry round-trip - Frontend: `releasePayload.test.ts`, `packReview.test.ts` (vitest) **Manual, on a real deployment** (arm64 image built from this branch, run as a side container next to production with the same qBittorrent / Audiobookshelf setup, `FILE_ORGANIZATION_AUDIOBOOK=organize`, hardlinks on): - AudiobookBay "The Expanse Complete 2.0" (7.87 GB, 36 files): clicking download opened the review panel in ~1 s showing **18 books · 18 files · 18 files ignored** (the `.txt` sidecars), with series positions 0.1–9.5 and years parsed from the file names; novella labels stripped ("The Churn", "The Butcher of Anderson Station"). Editing a title in the panel works. Confirming queued one task; the magnet resolved from the cached page in ~30 ms; after the download the task reported `Complete (18 books, 18 files)`, 18 hardlinks landed as `audiobooks/James S. A. Corey/<Title>/<Title>.m4b`, the torrent kept seeding, and Audiobookshelf scanned each folder as its own book (title, author, embedded chapters). - A second pack ("Expanse [01 - 9.5]", `Title N - Title` naming) was inspected to verify the repeated-title rule and the Back button, without downloading. - Single-book releases still queue immediately with no extra UI. |
||
  zab1996andRyan
|
02b7e9d958 |
feat(newznab): support multiple named indexers (#1271)
## Summary - add a named Newznab indexer table with per-indexer URL and API key settings - search every configured indexer and retain the originating indexer name on each result - namespace cached release IDs across connections and isolate individual indexer failures - preserve the legacy single-indexer settings as a fallback - support masked API-key cells and trusted SABnzbd prefetching for named indexers ## Validation - 121 Newznab and SABnzbd backend tests passed on Python 3.14 - Ruff passed for all changed Python files - frontend TypeScript and strict lint checks passed - all 134 frontend unit tests passed - frontend formatting check passed ## Compatibility Existing `NEWZNAB_URL` and `NEWZNAB_API_KEY` configurations continue to work whenever `NEWZNAB_INDEXERS` is empty. Co-authored-by: Ryan <zab1996@users.noreply.github.com> |
||
|
CaliBrain
|
a5595cf9f1 | Change test for fake extension that wont work (#1266) | ||
 jakesterpdxandClaude Fable 5
|
9bcf595111 |
feat(prowlarr): warn when an indexer declares a format Shelfmark can't process (#1265)
## Problem Companion to #1264, but general rather than mp4-specific. MyAnonamouse titles carry a structured `[LANG / FORMATS]` bracket that `_extract_mam_formats` parses. When every token in it is something Shelfmark doesn't know — e.g. `The Martian by Andy Weir [ENG / MP4]` — the release is rendered with **no format chip at all**, just the generic headphones/book icon with an "Audiobook" tooltip. To a user that looks like an ordinary result. It downloads fine and then fails post-processing with *"No book files found in download"*. The backend already *had* the signal (a format token it couldn't map); it just threw it away. ## Change **Backend** (`shelfmark/release_sources/prowlarr/source.py`) - `_split_mam_formats(raw_title) -> (recognized, unrecognized)` replaces the body of `_extract_mam_formats`, which is kept as a thin wrapper returning `recognized` so nothing else changes. - Releases gain `extra["unrecognized_formats"]` (list, or `None` when empty / when format detection is off). **Frontend** - `getUnrecognizedReleaseFormats(release)` in `utils/releaseFormats.ts` (normalised + deduped, same shape as `getReleaseFormats`). - `ReleaseCell` `format_content_type`: when there is **no** recognised format but the indexer named one, render an amber `MP4 Unsupported` badge (compact view: amber `MP4`) with tooltip *"Unsupported format (MP4) - Shelfmark cannot process this release"*. When a recognised format exists the existing badge is untouched, even if extra unknown tokens were present. Only the chip changes — the download button still works, so a user can still grab and hand-process the files if they want to. Happy to disable the button instead if you'd prefer. ## Tests - `tests/prowlarr/test_source.py`: `TestSplitMamFormats` (recognised / unrecognised / mixed / no bracket / wrapper compat) and `TestUnrecognizedFormatOnRelease` (lands in `extra`, empty when recognised, absent without format detection). - `src/frontend/src/tests/releaseFormats.test.ts`: 3 cases for the new helper. - `ruff check` clean; `pytest tests/prowlarr -m "not integration"` 511 passed; `tsc --noEmit`, `oxlint --deny warnings`, `vitest` all clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
CaliBrain
|
e7007865a4 |
fix(prowlarr): stop turning indexer failures into empty results and 404s (#1251)
Two independent bugs, both from an indexer that Prowlarr proxies rather
than answers for itself: the search never reported that it had failed,
and the grab never resolved what it was handed.
Search. A Torznab search is Prowlarr proxying a live request out to the
tracker, so for a Cloudflare-fronted indexer it waits on FlareSolverr.
The client gave it the 30s budget sized for Prowlarr's own JSON
endpoints, then swallowed every failure -- the timeout, the 429 Prowlarr
returns once it has disabled an indexer, a parse error -- into the same
empty list that means "this indexer has nothing". A cold challenge
routinely runs past a minute, so the UI said "No releases found for this
book" while FlareSolverr was still solving. That empty list also drove
the auto-expand retry, which fires on "no results with the category
filter". A timeout satisfies it, so Shelfmark sent a second search to an
indexer still busy with the first -- two Chromes at once, enough to take
FlareSolverr's down on a small host.
torznab_search now raises ProwlarrSearchError, and an empty list
strictly
means the indexer answered with no matches. The source records which
indexer searches failed: one dead indexer no longer sinks the others,
auto-expand runs only when every indexer genuinely answered, and zero
results with at least one failure raises SourceUnavailableError, which
the releases endpoint already turns into a 503 carrying a real message.
Prowlarr being unreachable was the same lie by another route -- the
indexer list came back empty, leaving nothing to query -- and now says
so.
Indexer searches also get their own timeout, PROWLARR_INDEXER_TIMEOUT,
defaulting to 90s and clamped to 5-300. Prowlarr's status and indexer
list keep 30s so Test Connection stays responsive, and the connect
timeout is split out at 10s so an unreachable Prowlarr fails fast rather
than hanging for the whole read budget. The overall per-request search
budget now scales to twice the indexer timeout, capped at 240s, so
raising the setting is not undone by the cap one level up while staying
under the 300s gunicorn worker timeout.
Grab. Prowlarr hands out a proxy download URL, with no magnetUrl and no
infoHash, for any indexer that only publishes torrent files. The native
Real-Debrid client built its magnet as "if not
url.startswith('magnet:') and expected_hash", so with no hash to work
from it left the URL alone and POSTed it to /torrents/addMagnet as the
magnet field. Real-Debrid answered 404 and the grab died on a raw HTTP
error. AllDebrid carried the same line and the same bug.
Both now resolve the URL first, through the extract_torrent_info path
the
torrent clients have used since #1108: pass a magnet through untouched,
follow a redirect or a response body that turns out to be a magnet,
otherwise upload the fetched .torrent, and fall back to a magnet built
from the infoHash only when the fetch failed. The file is preferred over
a synthesized urn:btih: magnet because it carries the tracker list; a
bare hash leaves the service to find the swarm on DHT alone. Fetches are
shared with the rest of the add path through the torrent fetch cache, so
resolving costs at most one request. Real-Debrid takes the file on PUT
/torrents/addTorrent with the raw bytes as the request body, AllDebrid
on
POST /magnet/upload/file as multipart files[]. A URL that resolves to
neither form now raises before any request reaches the service, so the
user reads why instead of a 404. Neither debrid client had any test
coverage; both have some now.
Fixes #1249
Fixes #1250
|
||
|
CaliBrain
|
7193036626 |
fix(rtorrent): apply the audiobook label to audiobook downloads (#1239)
add_download() picks self._audiobook_label from a content_type kwarg, but the only call site never passed one, so is_audiobook was always False and every download got RTORRENT_LABEL. category does not fill the gap: _get_category_for_task() returns None for rTorrent, which has no category concept, leaving content_type as its only audiobook signal. Pass task.content_type through from base_handler, and match it with the shared is_audiobook() helper instead of == "audiobook". normalize_content_type() treats "book (audiobook)" as an audiobook, so the exact-string check would have mislabeled that value even once it arrived. The existing rTorrent tests passed content_type straight to the client, which is why nothing caught the missing wiring; the new handler test covers the call site itself. Post-processing was never affected: destination.py reads task.content_type directly, so files already landed in DESTINATION_AUDIOBOOK correctly. Fixes #1235 |
||
 CamGoed
|
e93fbd2a9b |
Fix issue with stale v1 hash queries to qbittorrent (#1162)
## Problem
Shelfmark can lose track of hybrid v1/v2 torrents after qBittorrent
completes their metadata download.
Shelfmark initially identifies the torrent by its v1 infohash. Once
metadata resolves, qBittorrent may switch the torrent’s primary `hash`
to the truncated v2 hash, causing lookups using the original v1 hash to
return nothing.
For example:
- v1: `edf46c7f938a3c678081734d7bff8b9c652ba5e5`
- qBittorrent `hash`: `0bed5f40753b342cb143e83c2b21924cc8474731`
- full v2:
`0bed5f40753b342cb143e83c2b21924cc847473134e44d1bd300bdc58c13010f`
At that point, querying `/api/v2/torrents/info` with the original v1
hash returns no records. Querying with the new primary hash works, and
the returned record still contains the original hash in `infohash_v1`.
`find_existing()` also ignored its provided category, so audiobook
torrents fall back to the default ebook category instead. This means the
fallback method for a mismatched download ID never occurs for
audiobooks, leading to a "failed" download that is actually successful
in qBittorrent. As a result, the downloaded files are not automatically
transferred/hardlinked to the output directory.
## Fix
- Match torrents against `hash`, `infohash_v1`, and `infohash_v2`.
- Wait for magnet metadata to finish downloading before returning the
torrent ID.
- Return qBittorrent’s current primary `hash`.
- Search the provided category first, then the configured default, and
finally the full torrent list.
Logs showing the issue:
```
2026-08-03 22:42:30,262 - shelfmark.release_sources.audiobookbay.scraper - DEBUG - scraper.py:450 - Generated Magnet Link: magnet:?xt=urn:btih:EDF46C7F938A3C678081734D7BFF8B9C652BA5E5&tr=...
2026-08-03 22:42:30,376 - shelfmark.download.clients.qbittorrent - DEBUG - qbittorrent.py:521 - qBittorrent add result: TorrentsAddedMetadata({'added_torrent_ids': ['edf46c7f938a3c678081734d7bff8b9c652ba5e5'], 'failure_count': 0, 'pending_count': 0, 'success_count': 1})
2026-08-03 22:42:30,427 - shelfmark.download.clients.qbittorrent - INFO - qbittorrent.py:545 - Added torrent: edf46c7f938a3c678081734d7bff8b9c652ba5e5
2026-08-03 22:42:30,427 - shelfmark.download.clients.base_handler - INFO - base_handler.py:868 - Added to qbittorrent: edf46c7f938a3c678081734d7bff8b9c652ba5e5 for 'Pathogenesis: A History of the World in Eight Plagues'
2026-08-03 22:42:30,427 - shelfmark.download.clients.base_handler - DEBUG - base_handler.py:906 - Starting poll for edf46c7f938a3c678081734d7bff8b9c652ba5e5 (content_type=audiobook)
2026-08-03 22:42:32,590 - shelfmark.download.clients.base_handler - DEBUG - base_handler.py:958 - Download edf46c7f938a3c678081734d7bff8b9c652ba5e5 not yet visible in client (attempt 1/15)
2026-08-03 22:43:02,345 - shelfmark.download.clients.base_handler - ERROR - base_handler.py:969 - Download edf46c7f938a3c678081734d7bff8b9c652ba5e5 not found after 15 attempts
2026-08-03 22:43:02,345 - shelfmark.download.clients.base_handler - INFO - base_handler.py:426 - Skipping download client cleanup for protocol=torrent after download error (client=qbittorrent id=edf46c7f938a3c678081734d7bff8b9c652ba5e5)
```
<br>
The successful torrent:
<br>
<img width="968" height="159" alt="image"
src="https://github.com/user-attachments/assets/d63c4444-6a4d-4214-97a0-732acc338970"
/>
<br>
v1 vs v2 hash:
<br>
<img width="749" height="212" alt="image"
src="https://github.com/user-attachments/assets/d985b78a-84fe-4a26-9697-126c787e7303"
/>
I ran some python queries from the shelfmark container that show the
mismatch:
```
qBittorrent URL: http://gluetun-mam:8081
Tracked hash: edf46c7f938a3c678081734d7bff8b9c652ba5e5
=== PROPERTIES LOOKUP USING SHELFMARK HASH ===
HTTP status: 404
Not Found
=== EXACT /torrents/info HASH LOOKUP ===
Returned torrents: 0
=== FIND VISIBLE PATHOGENESIS TORRENT ===
Matching visible torrents: 1
Name: Pathogenesis: A History of the World in Eight Plagues
Primary hash: 0bed5f40753b342cb143e83c2b21924cc8474731
Category: audiobooks
State: stalledUP
Progress: 1
Properties HTTP status: 200
Infohash v1: edf46c7f938a3c678081734d7bff8b9c652ba5e5
Infohash v2: 0bed5f40753b342cb143e83c2b21924cc847473134e44d1bd300bdc58c13010f
```
|
||
|
CaliBrainandClaude Opus 5
|
21f2b6b95c |
Fix torrent post-import category follow-ups and clear the lint backlog (#1154)
rTorrent set_category and remove now uppercase the info hash, which reaches us lowercase while rTorrent's XML-RPC lookups are case sensitive. remove() had this bug before #1148, making PROWLARR_TORRENT_ACTION=remove a silent no-op for rTorrent. Transmission set_category appends the post-import label instead of replacing the whole label list. The unsupported-client path in post_process_cleanup logs at debug instead of warning, so Real-Debrid and AllDebrid users stop seeing a warning on every successful import. The Real-Debrid and AllDebrid clients now follow the conventions used by the other clients (_raise_runtime_error helpers, narrow error tuples, ClassVar, Path.open), and register_client is generic over a TypeVar bound to DownloadClient so decorated classes keep their concrete type. make fix and the lint, format and typecheck targets all pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
 Tilian B
|
ff770940ca |
Add torrent post-import category action (#1148)
## Summary - add a **Change Category** torrent completion action and conditionally show its post-import category/label setting - update kept torrents only after a successful library import - support qBittorrent categories, Transmission labels, Deluge's Label plugin, and rTorrent's `custom1` label - preserve existing Keep/Remove behavior and document the new environment setting ## Behavior Category changes happen from `post_process_cleanup`, after output transfer and post-processing complete. This keeps the existing hardlink flow unchanged. An empty post-import category is a no-op, and client API failures are logged without turning a successful library import into a failure. ## Validation - `pytest -n 0 tests/prowlarr/test_qbittorrent_client.py tests/prowlarr/test_transmission_client.py tests/prowlarr/test_deluge_client.py tests/prowlarr/test_rtorrent_client.py tests/prowlarr/test_handler.py tests/config/test_generate_env_docs.py` — 161 passed - `ruff check` on all changed Python files — passed - `basedpyright` on the changed client implementation files — passed - full `basedpyright shelfmark/download/clients` currently reports two pre-existing errors in the new debrid connection-test code at `settings.py:546` and `settings.py:563`, outside this PR's diff - multi-architecture Docker images built successfully for `linux/amd64` and `linux/arm64` |
||
 
|
816a735cde |
Add a {Language} naming template variable, and consolidate language resolution (#1142)
Fixes #1138 Fixes #1141 ## Problem Two language editions of one book resolve to the same canonical title, so they render to the same path and the second gets a `_1` collision suffix. Audiobookshelf treats a folder as exactly one library item, so the pair becomes a single book with both files as tracks and a summed runtime. Shelfmark already parses and displays the language. It just never reached the template engine. ## `{Language}` template variable A template like `{Author}/{Title}{ (Language)}/{Author} - {Title}` now yields: ``` /library/J K Rowling/Harry Potter (sv)/J K Rowling - Harry Potter.m4b /library/J K Rowling/Harry Potter/J K Rowling - Harry Potter.m4b ``` The untagged edition's path is byte-identical to today, so no existing layout shifts. Three details worth flagging: **The value is casefolded.** On a case-insensitive filesystem `(SV)` and `(sv)` would collapse back into one folder, reintroducing the exact collision being fixed. **Values meaning "we don't know" render nothing** rather than producing `Project Hail Mary (unknown)` folders. Anna's Archive reports that string literally (`direct_download.py`, `language = detected or "unknown"`). **The frontend wasn't sending the release language at all**, so the token would have stayed empty for exactly the audiobook sources in the report. Prowlarr and AudiobookBay do not put language in `extra` the way `direct_download` does, hence the payload plumbing. It reads `release.language`, never `book.language` — the latter is the provider's canonical edition and would mislabel a translation, with a regression test for that specifically. Not gated to audiobooks: Calibre-Web-Automated stages ingested files by basename and discards folder structure, so the rename (filename) template is the only lever those users have. Verified that form works: `J K Rowling - Harry Potter (sv).epub`. ## Language consolidation (#1141) Three release sources each carried their own alias map, all resolving to the same ISO 639-1 codes, alongside a bundled database that only one of them used. Adding a language meant editing three places. Aliases now live in `data/book-languages.json` beside the code and name they belong to, and `shelfmark/core/languages.py` resolves any of them — two-letter code, ISO 639-2 three-letter in either the bibliographic or terminological form, or English name. Prowlarr and AudiobookBay drop their tables. Direct Download keeps its own path-parsing heuristics, including the ambiguous short codes that collide with English words (`de`, `en`, `no`, `in`), and takes only the alias data. This also closes a coverage gap. MyAnonamouse offers 62 languages; Prowlarr mapped 37, and an unmapped code is *dropped* rather than passed through, so the other 25 carried no language at all — leaving `{Language}` empty and the collision unfixed for Latin, Farsi, Tamil, Urdu and the rest. Seven languages MAM offers had no database entry at all: Bosnian, Burmese, Estonian, Icelandic, Manx, Scottish Gaelic, Sanskrit. Also fixes the Traditional Chinese code, which used a U+2011 non-breaking hyphen. Nothing compares against the ASCII spelling today so it was latent, but it would silently defeat the first thing that did. ## Validation Verified end to end against a live Prowlarr and MyAnonamouse, not just unit tests. A real search returning both an English and a Swedish edition, through the actual `queue_release` → `DownloadTask` → naming path: ``` STEP 1 real MAM search -> 37 releases, languages: ['en', 'sv'] STEP 3 queue_release -> task.language='sv' STEP 4 build_metadata_dict -> metadata['Language']='sv' STEP 5 build_library_path -> /library/J K Rowling/Harry Potter (sv)/... two language editions resolve to DIFFERENT folders: True ``` The refactor is pinned by a snapshot of both per-source maps taken *before* they were deleted. All 131 aliases are asserted to still resolve to the same code, one parametrised test each, so a regression names the specific alias. Also verified: the filename-only template, the retry round-trip (`serialize_task_for_retry` → `_restore_task_from_retry_payload`, plus a legacy payload with no `language` key), and placeholder handling. Added a `KNOWN_TOKENS` ordering invariant test — `find_placeholder()` does a substring `.find()` in list order and nothing protected that contract, so a future token in the wrong position could silently shadow an existing one. And a lockstep guard on the frontend, since `KNOWN_TOKENS` is hand-duplicated in TypeScript. **One caveat worth stating.** Three MAM codes are confirmed by observation (`ENG`→`en`, `SWE`→`sv`, `MAL`→`ml`, the last from a real `[MAL / EPUB]` Tagore release). The remaining ~59 are derived from ISO 639-2 rather than observed, because MAM's catalogue is overwhelmingly English — enabling 27 extra languages still yielded only one non-English hit across 258 results. Mitigated rather than closed: both 639-2 variants are present for every language where they differ, and a wrong alias is an unused entry while a missing one loses the language. Happy to correct any code a maintainer knows differs. ## Test results 2056 Python tests pass (up from 1906). Frontend typecheck, lint, format and 126 unit tests pass. Pre-existing failures on my machine, unchanged by this branch and unrelated: `tests/bypass/` needs `seleniumbase`, and `tests/config/test_entrypoint_permissions.py` uses bash-4 syntax that macOS bash 3.2 rejects. --------- Co-authored-by: delize <4028612+delize@users.noreply.github.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
CaliBrain
|
a1367f431d |
Ship Prowlarr per-entry rows as opt-in, not the default (#1145)
#1140 fixed the guid-only dedup that hid results from filter-specific indexer entries, but shipped the new behaviour on by default: PROWLARR_COLLAPSE_DUPLICATES defaulted off, so every existing Prowlarr user got extra rows for any release that two indexer entries both returned, and the setting only let them opt back into what they already had. Default it on. The dedup key stays indexer-qualified, so the entries are still distinct internally; collapse then merges them back to one row, resolved by the Prowlarr priority rather than by query order as before. The visible result set matches what users had prior to #1140, and anyone who wants the per-entry rows (freeleech and the like) turns the setting off. Beyond the noisier list, the default mattered because split rows differ only by indexer name while sharing a title, size and peer count. Two of them have distinct source_ids, so the queue's duplicate guard does not fire, and the second grab's find_existing() matches the first by infohash and runs post-processing over the same download again, delivering the book twice. The search-side fallback in config.get(..., True) is flipped to agree with the field default. In production the field default governs, since the config cache is seeded from the registry and the fallback only applies to an unregistered key. The source tests monkeypatch config.get with a plain dict lookup, though, so the fallback is what they exercise: leaving it False would have kept every default-behaviour test asserting the opposite of what ships. A new test pins the two together. The deduplication tests now opt out explicitly, since they assert the split itself. test_collapse_off_by_default_keeps_both_rows becomes a pair, one for the untouched setting collapsing to a single row and one for opting out. docs/environment-variables.md is regenerated rather than hand-edited. It was already stale on main, so it also picks up RTORRENT_AUDIOBOOK_LABEL, DIRECT_DOWNLOAD_LANGUAGE_FROM_PATH, and reworded IRC_SEARCH_BOT and RTORRENT_LABEL text from earlier merges. The rest is fallout from the ruff 0.16.0 bump in #1139, which enabled a much larger default rule set and started formatting Python code blocks in Markdown: _find_existing_alias_user() uses min() instead of sorted()[0] (FURB192, currently failing Python Quality on main), and the two READMEs get their code blocks reformatted. |
||
|
Andrew Doeringanddelize
|
a4086f5e06 |
Keep Prowlarr results from distinct indexer entries separate (#1140)
Results were deduplicated on `guid` alone. When one tracker is configured in Prowlarr as several indexer entries differing only by a server-side search filter, all of them return the same guid for the same torrent, so every entry but the first was silently discarded. Freeleech and other filter-specific releases became invisible, replaced by the unfiltered entry's copy, and which copy survived depended on query ordering rather than user intent. Include the indexer id in the dedup key so the entries stay distinct. Release.source_id is qualified the same way. It keys the release cache and becomes the download task id, so rows sharing a guid would otherwise collide and a grab would route through whichever entry cached last, defeating the point of showing them separately. The handler's task matcher still accepts a bare guid or infoUrl so tasks queued before this change still resolve. Ordering now follows the priority already configured in Prowlarr (1-50, lower preferred) rather than a new setting, since users curate that ranking there and filtered entries are typically ranked ahead of their unfiltered counterparts. The enabled-indexer list is fetched once and reused for the enrichment check, so this costs no extra round trip. Releases carry extra.indexer_priority, and the sort dropdown gains an "Indexer priority" entry, ascending, alongside the existing alphabetical "Indexer" sort; SortOption grew a default_direction for that, defaulting to desc so "Peers" is unchanged. PROWLARR_COLLAPSE_DUPLICATES (default off) optionally collapses a release back to one row, resolved by the same Prowlarr priority. Left off, every entry that carried a release keeps its own row, which is what makes filtered results visible again. Identity handling is defensive about partial payloads: a result that cannot be identified is never dropped or merged, and collapse only merges on a strong identifier (guid/downloadUrl/magnetUrl/infoUrl) because merging on title alone would discard genuinely different releases that share a name. Fixes #1137 Co-authored-by: delize <4028612+delize@users.noreply.github.com> |