mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-11-15 12:58:31 +00:00
46b9e7d8a3d9ff8526c5fc4c7204c87d0dc0c324
325
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 splitsec2
|
46b9e7d8a3 |
fix(library): match a shelf title against the full title when the search title is only the series (#1587)
Fixes #1584. Thanks to @joeRob2468 for the write-up, which pinned this down to the line. Hardcover cuts a title like "Mistborn: The Final Empire" down to "Mistborn" for searching when it can't tell the series prefix from a title prefix, and search results don't carry the series. The library check then matched the shelf title "Mistborn: The Final Empire" against "Mistborn", saw "final empire" as extra words and took it for a longer, different work (the "Dune" vs "Dune Messiah" rule), so the book got no In library badge. `match_entries` now tries the search title first as before and, if that finds nothing and the full title is different, tries the full title too. The full title is the more specific of the two so it should rarely add false positives. Another book in the same series still doesn't match, there is a test for that, and the Dune and Dune Messiah test passes unchanged. The matching loop moved into a small helper to run it twice, nothing in it changed. This only touches the library check. The same short search title also feeds release searches and auto-download matching, which I left alone here because loosening or tightening those changes what gets searched and downloaded for everyone. Tests fail on current main without the change. The library tests, ruff and basedpyright are clean. |
||
|
splitsec2
|
ac14e4dd24 |
fix(downloads): a swarm note change is not activity for the stall timer (#1572)
The torrent stall timer can be deferred for as long as the swarm keeps changing. Since #1494 the torrent handler appends a note to its progress message (`0% · 2 seeds`, and `· 0 seeds, giving up in 7m` while the no-seeders clock runs). `update_download_status` treats any changed message as activity, so a seed count that goes 2, 3, 1 between polls, or the give-up countdown ticking down every minute, refreshes `_last_activity` and the torrent never stalls. A torrent whose seed count stays constant still stalls, which is why this is easy to miss. `shelfmark/download/activity.py` already says what counts: a changed status event or a changed progress value, and a message that changes on a timer proves nothing. So `update_download_status` now compares messages through a small `activity_key()` that drops that trailing seed note. The card still gets every message and shows the live seed count, only the stall clock ignores a change in the note. A new progress percentage, a different client message or a status change still counts as activity. The handler builds the note in `_seed_note` and `activity_key` has to match it, so one test builds the note for a range of seed counts and countdowns and checks the key strips all of them. That also covers the gap #1494 left, where no test connected a handler's message to the orchestrator: the other two drive `update_download_status` with real notes and a fake clock. Without the change the note test fails and the key tests cannot even import `activity_key`. The other orchestrator test is a control that a progress or status change still counts. Full suite passes, and each new test also passes when run alone. |
||
  misko903andClaude Opus 5.5
|
d50440f6c2 |
fix(webshare): clear the "Resolving" status once the download starts (#1575)
## What
A Webshare.cz download shows "Resolving Webshare link" for its entire
duration, so a large download (e.g. a 1.3 GB audiobook archive) looks
stuck.
## Why
After the link was resolved, the handler called
`status_callback("downloading", None)`. `update_download_status` treats
a `None` message as "keep the current message", so the "Resolving
Webshare link" detail stayed in the activity panel. The HTTP handler
sends `""` for exactly this reason.
## How
- Report `downloading` with an empty message, which clears the resolving
text.
- When the download response has no `Content-Length`, use the release's
size (`task.size`, parsed with `parse_size_string`) as the total for
progress. Without it, progress never moved, and stall detection would
flag a healthy long download after 5 minutes.
## Testing
- New tests in `tests/webshare/test_handler.py`: the downloading status
clears the message, and progress is computed from the release size when
there is no Content-Length.
- `uv run pytest tests/webshare` is green (34 passed), and `ruff check`
and `ruff format --check` are clean.
Follow-up to #1543.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
splitsec2
|
525b094b57 |
fix(clients): hand a finished wrong-type directory on instead of polling out the timeout (#1573)
A torrent that finishes with the wrong kind of files in it holds the download for the whole completed-path timeout (60 seconds by default, up to an hour) and then fails with "Completed download has no stable supported files yet." An ebook task whose torrent holds only an `.m4b`, an audiobook task that got an `.epub`, or a `.pdf`-only release with the default format list all land here. In v1.4.0 these failed with the specific format error from post-processing, but the wait for the sync to settle (#1464) only counts files of the task's own type, so it keeps polling for files that are never going to appear. If the directory has no supported files, `_wait_for_completed_path` now looks for finished book or audio files that this task does not accept (the audiobook formats, the default ebook formats and pdf). When it finds some and they are the same on two polls in a row, it returns the path so post-processing can say what is wrong with the formats. Cover art and notes alone do not count, because that is what a directory looks like while the real files are still syncing over a remote mount, so those keep waiting as before. A wrong-type file that is still growing keeps waiting too. With no wait configured there is only one sample, and a wrong-type file is handed on straight away. The existing test that pins "no book files, no wait configured" uses a jpg, so it stays as it is. I extended the test helper to take a content type, which the wait tests never passed before. Four new cases fail without the change (a wrong-type directory for ebook, audiobook and pdf tasks), and two more check that sidecar-only and still-growing directories keep waiting. Full suite passes. |
||
|
splitsec2
|
47f0acdd31 |
fix(logging): mask userinfo and path-segment credentials, route stray loggers through the formatter (#1571)
Logs get pasted into issues, and the formatter from #1488 only masks `?key=value` query parameters. I went through what else commonly ends up in a log line and found these unmasked: - credentials in a URL's userinfo, like a proxy's `socks5://user:pass@host:1080` or `http://admin:secret@qbittorrent:8080` - a passkey as a path segment, like `/rss/<key>` or `/passkey/<key>/get.torrent`, which private trackers use - the token in a Telegram bot, Discord webhook or Slack webhook URL, which is what a notification URL looks like when it gets logged All three are masked now, in `redact_secrets`, so they apply to every handler. The path rule only fires after a segment that names a secret (`passkey`, `apikey`, `rsskey`, `authkey`, `token`, `secret`, `rss`, `torznab`) followed by 16 or more token characters. I left out generic words like `download` on purpose, because `/api/download/<md5>/cancel` appears in our own request log and the ids there are useful when debugging. That is the tradeoff: a passkey in a `/download/<passkey>/` path still gets through. Separately, four modules created their logger with plain `logging.getLogger`, so nothing they logged went through the formatter at all: `api/websocket.py`, `download/clients/__init__.py`, and the Jackett, Newznab and Prowlarr package inits. They use `setup_logger` now like everything else. Tests: ten cases, seven for the new shapes and three that must stay readable (an md5 route id, a short `/rss/latest`, an email address), plus two that check the stray modules now carry the redacting formatter. Nine of them fail without the change. Full suite passes. Not covered here: `/api/debug` still zips the existing log files as they are on disk, so anything logged before this change rides along in a bundle. That is a separate change in `genDebug.sh`. |
||
|
splitsec2
|
ef40a38546 |
fix: report a stopped download coordinator in /api/health (#1568)
If the download coordinator thread dies, every task sits at "Queued" and nothing says why. The rest of the app keeps answering, `/api/health` stays green and the container looks fine. #823 and #1166 were both that, and the loop's `except Exception` cannot catch everything that can end a thread. `/api/health` now reports it. The response gets a `coordinator` field, `running` or `stopped`, and when it is stopped the status is `error` and the HTTP code is 503. The Docker healthcheck in the Dockerfile and the one in the readme both use `curl -f`, so the container shows as unhealthy and anything watching health (Uptime Kuma, autoheal) can act on it. It needs no login, same as before, and shows no queue contents. That 503 is the one decision here. Keeping the 200 and only adding a `degraded` entry would not trip a status-code health check, which is the whole point, so I went with 503. Easy to flip if you would rather not. The orchestrator gets a small `coordinator_running()` that asks whether the thread is alive. `start()` already knows how to restart a dead thread, so restarting it automatically would be the true fix, but I would rather the failure be visible first and decide on that separately. Tests: three for the route (ok, 503, no login needed) and three for `coordinator_running` (never started, alive, dead). They fail without the change. One existing test hits `/api/health` through an app that never starts the coordinator, so it now patches that. The readme's health section has a sentence about the 503. Full suite passes, ruff and basedpyright are clean. |
||
|
splitsec2
|
973cf472a2 |
test(image-cache): flush window test no longer depends on float rounding (#1580)
`test_cache_hit_flushes_the_index_once_the_window_has_passed` (from #1544) failed on the first attempt of two of my PRs' Python Tests runs today (#1576, #1579) and passed on rerun, with nothing in either PR touching the image cache. The test sets the clock to exactly `last_saved + ACCESS_FLUSH_SECONDS` and expects a flush. `(now + 30) - now` isn't always 30.0 in floating point. For example `105.65207077218265 + 30 - 105.65207077218265` is `29.999999999999986`, so the `>=` check is false and the flush never happens. That only shows up when the monotonic value is small, which fits a freshly booted runner (on this box the uptime is in the millions of seconds and it never fails). I haven't been able to see the runner's actual uptime in the CI logs, so that part is an inference, but pinning `_last_saved` to that value reproduces the exact failure (`assert [] == [1]`) locally. The test now puts the clock one second past the window. The production check is fine, real elapsed time doesn't have this edge, so only the test changes. |
||
|
splitsec2
|
b248bcd536 |
fix(clients): import a qBittorrent torrent only when every listed file is on disk (#1577)
Fixes #1561, with thanks to @tmussche for the write-up, which had the cause and the fix. When the files reach Shelfmark through a sync (Resilio in the report), the wait for the completed path accepts the folder once two samples a few seconds apart look the same. A sync that copies in batches can pause longer than that, so 10 of 23 files were imported and some of those were still partial copies. For qBittorrent the client can say what the finished download holds, so the wait now checks that. `DownloadClient` gets an optional `get_expected_files()` that returns `ExpectedFile(name, size)` entries, or None when the client cannot say, which is the default and keeps every other client as it was. qBittorrent answers it from `/api/v2/torrents/files`, leaving out files the torrent does not download (priority 0), zero-byte entries and BEP 47 padding files. `_wait_for_completed_path` asks once, and while any of those files is missing or smaller than its final size it keeps waiting, using the same timeout as before. When the wait runs out the error now says how far it got, for example "Completed download is not fully on disk yet (10 of 23 files)." Once everything is there the existing stability check still runs, so nothing is accepted that was accepted before. Names from the client start with the torrent's top-level folder, so each file is looked up with and without it, which also covers a torrent saved without a subfolder. A single-file torrent is checked by size too, since a file that is still syncing exists at a smaller size. Two things I would like a second look at: the comparison is on exact size, so a setup where something rewrites the files after the torrent finishes would never match and would run into the timeout; and the file list is read once per wait, not on every poll. If the client does not answer, nothing changes. I also moved the nested authenticated GET in the path derivation into a `_get_with_auth` method so both calls share it, and added a sentence about this to `docs/configuration.md`. Tests: eight for the wait (partial folder, truncated file, files arriving during the wait, complete folder, no top-level folder, single file both ways, and a client without a list), and six for the qBittorrent file list. The wait tests that depend on the new behaviour fail on current main. Full suite passes. |
||
|
splitsec2
|
9011319ade |
refactor: one lookup for the supported formats, public names for two shared helpers (#1576)
Six places read the Supported Book Formats and Supported Audiobook Formats settings on their own: post-processing, the post-process scan, IRC, slskd, Prowlarr and direct download. For ebooks they already agreed, since they all fall back to `DEFAULT_EBOOK_FORMATS`. For audiobooks they did not. The settings registry and slskd fall back to every audio format plus zip and rar, while post-processing and IRC fell back to m4b and mp3 only. They now share `shelfmark/core/formats.py`, with `get_supported_formats`, `get_supported_audiobook_formats` and a content-type aware `get_supported_formats_for`, and a shared `DEFAULT_AUDIOBOOK_FORMATS` in `core/utils.py` next to the ebook one. `policy.py` re-exports the same functions, so nothing that imports them from there changes. It is not in `policy.py` itself because importing the `postprocess` package pulls in the router, which imports the release sources, so a source could not import from it without a cycle. One behaviour difference, and only in the fallback: with the audiobook setting unset, or emptied, post-processing and IRC used to accept m4b and mp3 and now accept the full default list like slskd and the settings page already did. A configured list is read exactly as before. Direct download's own helper used the raw configured list and now reads the same normalized one, which is identical for any list the settings page can save. Also public now: `parse_size` in the Prowlarr source (slskd and Newznab imported it with the underscore) and `coerce_completed_path_timeout_seconds` in the client base handler (slskd imported that too). Not changed on purpose: slskd's `_coerce_int` looks like a copy of `coerce_int` but is not. It reads `"12.0"` as 12 and `True` as 0, where `coerce_int` gives 0 and 1. And the `_parse_size` in the IRC source is a different function, string to bytes. Tests: a new set checks that post-processing, IRC and slskd all fall back to the shared defaults and that the settings default is the shared one, and the scan-archive and IRC tests now patch the shared config instead of the per-module copies. Full suite passes. |
||
|
splitsec2
|
606e94fde2 |
fix(settings): offer the seed settings for qBittorrent only (#1574)
The two seed settings, Give Up With No Seeders and Minimum Seeders, show up for every torrent client, but only qBittorrent reports a seed count. With Transmission, Deluge or rTorrent you can set both and nothing happens. The first one says "Only qBittorrent reports seed counts" at the end of its description, the second says nothing about it. Both are now shown only when qBittorrent is the selected client, the same `show_when` the other qBittorrent fields use, and Minimum Seeders gets the same sentence. A value someone already saved for another client stays in their config, it just is not shown, and it had no effect before either. `docs/environment-variables.md` is regenerated for the changed description, which is two lines. One test checks that both fields are tied to qBittorrent and say so, and it fails without the change. |
||
|
splitsec2
|
70fe4b8c89 |
fix(hardcover): keep the scheduler alive on an unreadable interval, share one history connection per sync (#1569)
These are the follow-ups from the review notes on #1516 and #1534 that I said I would do once the series had merged. The scheduler waits for the next cycle outside the try that guards the cycle itself, so an interval setting that raised would end the thread and nothing would sync again until a restart. It now logs the error and tries again after six hours. `docs/hardcover-sync.md` says so. The history check in the sync opened a new SQLite connection for every book on the shelf. It opens one per sync now and shares it, which adds up on a slow disk or a small board. The two test nits from #1534 are in as well: a `parametrize` with a single case, and an unused argument on a helper. The thread surviving a failed wait and the shared connection each have a test that fails without the change. Ruff, basedpyright and the backend suite pass (5297 passed). |
||
|
splitsec2
|
a63d9e3e21 |
docs: regenerate the environment variable reference (#1565)
Sorry about this one. The environment variable references had also fallen well behind, and again I was a big part of that. A bunch of the gaps come from my own PRs (Hardcover sync, auto-download, the Audiobookshelf library check) where I charged ahead with the functionality and skipped the boring documentation part (did I mention that documentation is boring?). I again had Claude compare the docs to the code and generate the updates, so it would be good to have another human look this over before it merges. The doc diff is just the generator's output. The part worth a real read is the handful of new entries in `scripts/generate_env_docs.py`, since I wrote those descriptions from `entrypoint.sh` and `tor.sh` and not from a settings definition. What changed: - Regenerated `docs/environment-variables.md`. It had 287 variables and the generator now produces 311. The 21 that were missing are the Hardcover Sync tab (6), auto-download (11) and the Audiobookshelf library check (4). Nothing was removed and no existing default, type or option changed. - `SHELFMARK_API_KEY_READONLY` was only ever hand-edited into the doc, so the generator did not know it and a plain regeneration would have dropped the row. It is in the generator now. - Added `UMASK`, `TOR_CHECK_INTERVAL` and `TOR_MAX_FAILURES`. The code reads all three and no doc mentioned them. - The table of contents built its anchors by lowercasing and swapping spaces, so the Webshare.cz and Moly.hu links were already broken (GitHub drops the dots) and Soulseek (slskd) would have broken with the parentheses. They now follow GitHub's rule. Two tests cover the new entries and the anchors, and both fail without the generator change. To regenerate: `python scripts/generate_env_docs.py` on Python 3.14. A test that fails when the committed file differs from the generator output would have caught this drift, but then every PR that adds a setting has to regenerate the doc. I left that out, your call if you want it. |
||
|
splitsec2
|
d7a4b6b859 |
fix: keep exception text out of the status, config and release search responses (#1562)
Same idea as #1558, for three more places that hand `str(e)` to whoever is logged in. These are code scanning alerts 106, 142 and 143. `/api/status` and `/api/config` returned the raw exception text of an OSError, RuntimeError and so on with the 500. They now return "Could not read the download status" and "Could not load the configuration". The full error is still logged with its trace. For `/api/releases`, a source failing with an unexpected error put `source: <exception text>` into the error list that goes back to the caller. `search_source_releases` takes a new `include_error_detail` flag. The route passes True for admins (and when auth is off, where everyone is treated as an admin) and False for everyone else, who get `source: search failed`. The message of a `SourceUnavailableError` is written for the user, so it is kept either way, and the background auto-download caller keeps the default and sees the full text as before. Left alone on purpose: the `reason` in the release inspect response, since a test covers that it carries the error text and it looks intentional. Tests: two for the routes and two for the helper. All four fail without the change. Full suite passes (5295), ruff and basedpyright are clean. |
||
 
|
23b3b3a9d8 |
feat: add Jackett release source (#1521)
## Summary Adds Jackett as a separate release source (`jackett`), as requested in #1443. - New `shelfmark/release_sources/jackett/` package: API client, settings tab, source and handler. - Searches Jackett's Torznab endpoints: the `all` aggregate by default, or each selected indexer individually. Failed or timed-out searches raise `SourceUnavailableError` instead of reporting "no releases found". - Reuses Prowlarr code rather than duplicating it: the Torznab parser (now also reads `<jackettindexer>`), `_prowlarr_result_to_release` (new optional `source`/`source_id` args), the release cache, URL/protocol helpers, the timeout clamp and size-limit helper (new optional config-key args), and the handler's `list_files` / retry-restore logic. - Downloads use the existing shared torrent client settings. - `JACKETT_URL` is a trusted origin for `.torrent` fetching (it is usually on the LAN). Prowlarr's `X-Api-Key` is not sent to it. - Added a Jackett section to `docs/environment-variables.md`. ## Screenshots <img width="1117" height="936" alt="Screenshot 2026-10-21 at 13-45-34 Shelfmark" src="https://github.com/user-attachments/assets/00339999-72c7-4226-bd9a-df8b325a7e22" /> <img width="1117" height="938" alt="Screenshot 2026-10-21 at 13-50-38 Shelfmark" src="https://github.com/user-attachments/assets/3c9cd2b4-932a-4b5c-9e10-6ec22b5f987d" /> <img width="1117" height="937" alt="Screenshot 2026-10-21 at 13-50-57 Shelfmark" src="https://github.com/user-attachments/assets/9c4d0ed9-da2f-4323-b188-341b45066efd" /> ## Not included - First-run onboarding does not offer Jackett yet. - No Jackett-specific category settings; it uses the standard 7000 / 3030 categories. ## Testing - New unit tests in `tests/jackett/` (client, source, handler) using mocked responses. - Full suite passes except 11 `tests/e2e` tests that also fail on a clean tree here (cannot write `/var/log/shelfmark`). - Tested against a live Jackett instance. Closes #1443 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
splitsec2
|
3633dd3d97 |
fix: keep file links and API keys out of AllDebrid logs, fix Newznab debug log (#1559)
Two log fixes in the same area, both found while going through the open
code scanning alerts.
AllDebrid: the unlock path logged the file link in full. The last-resort
fallback appends `apikey=` to the link and returns it, and the "Failed
to download from ..." error carries that same link into
`logger.exception`, so the key could land in the log and the traceback.
The links are now logged through `_safe_url` (scheme, host and path
only). The GET unlock fallback also logged the exception text at debug
level, and requests puts the request URL in that message, which includes
`apikey=`. That one now logs the exception type only. This covers code
scanning alert 124.
Newznab: `logger.debug("Newznab API: GET %s params=%s", url, *params)`
star-expands the dict keys into the log arguments, so with debug logging
on it raises `TypeError: not all arguments converted during string
formatting` and the line never prints. Logging `params` as is would
print the API key, so it now logs the sorted parameter names.
Tests: the Newznab one formats the logged line and checks the key is
absent. The AllDebrid ones check the fallback and failure paths log no
query string and that the fallback still returns the link with the key.
All three fail without the change. Ruff and basedpyright are clean,
`tests/newznab` and `tests/prowlarr` pass.
|
||
|
splitsec2
|
4f97fc9bec |
fix: keep server paths and URLs out of an error response and a debug log (#1558)
Two small leaks from going through the open CodeQL alerts (code scanning alerts 100 and 123). The local download route returned `str(e)` on an OSError, and for a missing or unreadable file that text is the file's path on the server (`[Errno 2] No such file or directory: '/audiobooks/Author/Title.epub'`). #1422 cuts a non-admin's download path down to the file name everywhere else, so this put it back. The route now answers "Could not read the file" and the detail stays in the log. `should_bypass_proxy` logged the whole request URL at debug when it could not be parsed. URLs can carry credentials, so it logs only the exception now. Each change has a test that fails without it. The full non-e2e suite, ruff and basedpyright pass. |
||
|
splitsec2
|
86fff32297 |
feat(hardcover): owned books, simpler queries, dead sources and failed releases (#1535)
A request whose book turns out to be in the library is closed with the note `[auto] Already in the library.` instead of sitting in Needs Review. A request an admin handled by hand while a pass was running is skipped. A series label in front of a Calibre title no longer hides an owned book, and a Calibre title marked as another edition (full cast, dramatized, GraphicAudio) is not the book. When no source matches, up to three simpler forms of the title are tried on each source under the same strict match. Optionally (off by default), an unmatched audiobook request is looked up on Audible US and UK through Audiobookshelf's metadata search and the request says what the store answered. It never declines anything. A Soulseek peer whose recent transfers all failed, and an AudiobookBay post whose comments say it has no seeders, are tried after the other matches and never excluded. A failed or stalled release is kept out for AUTO_DOWNLOAD_RETRY_FAILED_RELEASE_DAYS and then tried again, and a torrent that died for one request is not offered to another. A request stuck on queued or downloading is settled from the download history. Optionally (off by default) a failed download is retried with another release within minutes. |
||
|
splitsec2
|
9d5b47d17b |
feat(hardcover): retries, error counting and a back-off for requests nothing matches (#1534)
A synced request whose download failed, was interrupted by a restart, or was cancelled by the stall timer goes back to pending after AUTO_DOWNLOAD_RETRY_DAYS (7 by default, 0 turns it off), a few per pass, oldest first. Only what never arrived is retried, the failed release is remembered, and a request is given up on after three failed retries. Downloads cancelled by a person and rejected requests are never retried. Errors count as errors: a search that failed in one source no longer reads as no match, a throttled Hardcover lookup is an error and not a skip, and a request that blows up doesn't abort the pass. Both summaries carry the error count. A request for the same Hardcover book and format is skipped when the history already has a finished or running download for it. A request that finds nothing is searched once a day after three no-match passes, then every two days, up to AUTO_DOWNLOAD_BACKOFF_MAX_DAYS (3 by default, 0 turns it off). |
||
|
splitsec2
|
4389548deb |
feat(hardcover): ebooks, audiobooks or both, and stricter matching for automatic downloads (#1533)
A shelf book can be requested as an ebook, an audiobook or both (one request per format), and each request is checked against its own library, source priority list and format guard. Ranking within a source prefers the most downloaded copy, then size, and one m4b is preferred to a set of chapter files unless it is much smaller than the other copies. Soulseek peers are ranked by free slot, queue length and speed. Automatic downloads refuse multi-book packs, sequels and other works named in the title, study guides, abridged or dramatised versions, two-in-one titles, releases in a language the requester didn't ask for, and an archive for an audiobook request. Matching folds accents on both sides, drops an alternate title for the search and the match, and reads small numbers the same in any form. AudiobookBay releases with no author in the listing title are judged on the authors credited on the detail page. |
||
|
splitsec2
|
ea69db3f8e |
feat(library): Audiobookshelf library check, and the sync fails closed when a library cannot be read (#1519)
Adds an Audiobookshelf provider for the library check, next to the Calibre one. It reads the server over its REST API (title, author, ISBN and ASIN per item), answers for audiobooks, and has its settings in General with a test button that uses the unsaved form values. Read only. The Hardcover sync and automatic downloads now fail closed when an enabled library cannot be read: the sync skips its cycle and an automatic download holds the request. A library that listed items at the last index and now lists none counts as a failed read. Badges still fail open. |
||
 ProcyonX1
|
99a6aa206b |
feat(metadata): add Deutsche Nationalbibliothek (DNB) metadata provider (#1547)
## feat(metadata): add Deutsche Nationalbibliothek (DNB) metadata provider This adds the Deutsche Nationalbibliothek (DNB) as a metadata provider. It covers German editions far better than the existing providers. No API key, disabled by default, and the same pattern as the other providers. It uses DNB's public SRU interface and parses MARC21-XML. Thats one reason why this needs more code than the Google Books or Open Library providers. The others are the search (DNB has no relevance ranking) and the language filter, see below. ### Notes - **Search box.** Search-box text goes from precise to loose until a stage has hits: the complete title, then the title as a phrase, then every word in a title or a person's name, last without German articles. - **"Only show editions in my book languages"** (optional, on by default). DNB also catalogs a lot of translations. DNB filters by the searching user's Default Book Languages, else the system default. - **No covers.** DNB's cover service is behind Anubis bot protection, so the provider returns no cover URLs. - **Proxy support** already included like Open Library since #1526 ### Changes outside the provider - DNB is in the onboarding dropdown. - Calibre's `dnb-idn` identifier is mapped, so DNB results show library ownership - In the env-var docs I added only the DNB section. Regenerating the whole file on main currently also reorders unrelated sections and drops `SHELFMARK_API_KEY_READONLY` (and the Databáze knih section sits inside the Moly.hu block). ### Ideas - **Deduplication.** DNB lists every edition. Kafka's "Die Verwandlung" has around 200 German editions of the same text. Grouping them would touch the API, the frontend and release search. - **Language-aware metadata search** for all providers. The other providers ignore Default Book Languages in metadata search. The `user_id` added here could serve them too. |
||
|
|
2007384058 |
feat(sources): add Webshare.cz as a release source (#1543)
## What Adds [Webshare.cz](https://webshare.cz), the Czech file host, as a release source for ebooks and audiobooks. It follows the Libgen source: disabled by default, with a settings tab for credentials and a **Test Login** button that reports whether the account is VIP. ## Why Webshare is one of the main places Czech and Slovak ebooks and audiobooks are shared, and none of the existing sources reach it. Together with the Databáze knih metadata provider (#1479), this lets Universal mode find Czech and Slovak titles end to end. ## How - **API client** (`webshare/api.py`): Webshare's XML API (`POST /api/<endpoint>/`). Login is the salted exchange the API documents: fetch the salt, send `SHA1(MD5_CRYPT(password, salt))` plus an MD5 digest. Python dropped the `crypt` module in 3.13, so md5-crypt is a small pure-Python port, checked against `openssl passwd -1`. The session token is cached per account and refreshed once when an authenticated call fails. File-level errors (`FILE_LINK_FATAL_*`) are not retried. XML is parsed with `defusedxml`. - **Search** (`webshare/source.py`): Webshare matches file names only, so the file name becomes the release title and its extension the format. Queries are title + author, then title only, or the manual query. Results are filtered to the configured `SUPPORTED_FORMATS` / `SUPPORTED_AUDIOBOOK_FORMATS`, plus `zip`/`rar` for audiobooks, since they are usually uploaded as archives. Password-protected files are skipped. The language is guessed from tags in the name (`CZ`, `SK`, `ENG`, ...), and the vote balance is a sortable column. API failures raise `SourceUnavailableError`. - **Download** (`webshare/handler.py`): resolves a time-limited link per attempt with `file_link`, then streams it into `TMP_DIR` with progress and cancel support. Large audiobook archives are not buffered in memory. - **Settings**: `WEBSHARE_ENABLED`, `WEBSHARE_USERNAME`, `WEBSHARE_PASSWORD` (a `PasswordField`, already covered by the `_PASSWORD` redaction in `genDebug.sh`), and `WEBSHARE_MAX_RESULTS`. The `docs/environment-variables.md` section was added. ## Testing - `tests/webshare/`: 31 offline tests. They cover md5-crypt vectors, password hashing, XML parsing, token reuse and re-login, non-retried file errors, availability gating, format filtering, query fallback, release mapping, language guessing, and the handler's streaming, small-file rejection, link errors and cancel. - `ruff check` and `ruff format --check` are clean. The rest of the unit suite fails the same way with and without this change: Windows-only tests on my machine. - Verified live against webshare.cz: search parsing and the salt endpoint's error path. - Still a draft: I haven't yet verified login and download with a real account. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
48ae3dbef4 |
feat: optional Hardcover wishlist sync with automated downloading (1 of 5) (#1516)
This replaces #1047, which @InfiniteAvenger opened and calibrain offered to re-review. It is Caleb's feature commit with his authorship kept, rebased onto current main, with small fixes, tests and a docs page on top. Everything is opt-in and off by default, so nothing changes for anyone who doesn't turn it on. What it adds is a **Settings, Hardcover Sync** tab with three parts that can be switched on separately: - **Shelf sync**: a Hardcover shelf (Want to Read by default) is pulled into requests on a schedule, or with a Sync now button. Requests carry the Hardcover metadata and cover. - **Automatic downloads**: synced requests are approved and a release is picked from a source priority list. A strict title, author and format match (plus a seeder minimum for torrents) decides, so anything without a confident match stays pending for manual review. - **Skip books you already own**: sync and automatic downloads ask whichever libraries are enabled under Library Check (Calibre today) and skip a book that is already there. On top of Caleb's commit I made the request owner a lookup instead of assuming user id 1, sent synced cover URLs through the proxy when the request is created, got ruff and basedpyright clean, and added tests and `docs/hardcover-sync.md`. His commit also had its own Audiobookshelf settings and a Test library connection button, written before the library check was split into providers. The button would have raised a TypeError against the current `test_connection`, so those settings are gone from this tab and an Audiobookshelf provider comes as a separate PR, which is the part of #1047 that is missing here (for now anyways...). I've run this daily for about a couple of weeks. Ruff, basedpyright and the backend suite pass (4426 passed). This is 1 of 5. As calibrain suggested, the rest are stacked on this one, each saying which number it is. Closes https://github.com/calibrain/shelfmark/pull/1047#issuecomment-6302071588 --------- Co-authored-by: InfiniteAvenger <calebewest02@gmail.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
splitsec2andCaliBrain
|
72b5d04939 |
perf: cut repeated work on the status and activity paths (#1545)
Two small cuts to work that repeats on every poll or request. The delivery-state sync behind /api/status and the activity snapshot listed every fulfilled request (SELECT * plus a JSON parse of book_data and release_data per row) before it looked at the queue. That history is never purged, so an idle instance paid for it on every poll. It now returns straight away when no queue bucket holds a task, since nothing could map to a request then. Results and emitted updates are unchanged. The activity snapshot gains less from this, because its status merges download history rows, so it is usually not empty. In proxy auth mode the before-request hook looked the user up by name and by session id on every API call, cover images included, each on a new SQLite connection. UserDB.get_user_cached answers repeat lookups from memory for up to five seconds, bounded at 256 entries. create_user, update_user and delete_user drop the cache as soon as they commit, and a lookup that overlaps a write is not stored, so a role change or deletion made in the app applies on the next request. Changes made outside that UserDB instance (another process, or the separate instances some helpers open) show within the five seconds. Only the middleware uses the cached path; every other get_user caller is unchanged. Each change has a test that failed before it: one counts list_requests calls for an empty queue, the others count get_user calls across repeated lookups and middleware runs, check expiry with a patched clock, and check that writes and caller mutations do not leave a stale row behind. The full suite passes (4432 passed, 48 skipped, e2e excluded), as do ruff and basedpyright on the changed files. I also looked at throttling the full status_update broadcast when only the progress message changes during torrent and usenet polling, and left it out. Those handlers poll every 2 seconds and DOWNLOAD_PROGRESS_UPDATE_INTERVAL defaults to 1 second, so a throttle on that interval would skip nothing at the default setting. A longer window would slow the speed and ETA text in the UI, and without a trailing flush it would hide the last message of a quick burst, such as "Waiting for bot response" arriving just after "Requesting file from bot", for the whole wait that follows. --------- Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
splitsec2andCaliBrain
|
237ddd53a2 |
fix(redaction): a non-admin's status message loses the links it quotes (#1541)
#1422 hid `source_id` and the download path from non-admins but left the status message alone. A failed add stores the client's exception text there, for example `401 Client Error: Unauthorized for url: <full url>`, and for a requests error that URL carries the path and query: an indexer key, a tracker passkey, a peer name. It reached the owner through `/api/status`, `/api/activity` and the `status_update` socket event. `redact_download_payload` now replaces every link in `status_message` with `(link removed)`. That covers the live queue, the socket push and the history snapshot, since all three go through it. Messages without a link are unchanged and admins still see the full text. Tests cover the payload, a whole status snapshot, and messages that must stay as they are. They fail on main and pass here. Found in a Claude pass over 1.5.0. --------- Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
splitsec2
|
a430341a93 |
perf: cut repeated work on search and cover paths (#1544)
I went through what Shelfmark does per search and per cover and measured the pieces that showed up. Four of them are cheap to fix, so this is one PR. Shelfmark runs on a single gevent worker, so any of these blocks every other request while it runs, and I'm assuming a lot of people run it on small hardware (a Pi, say). Cover cache. Every cache hit rewrote the whole `cache_index.json` just to bump the entry's last-access time. On my box that is 3.7 ms of CPU per hit at 500 entries and about 100 ms at 15,000 (2.2 MB written each time), and a results page asks for around 40 covers. The access time now stays in memory and the index is written at most every 30 seconds, or whenever something else saves it. A crash inside that window loses some access times, which only changes eviction order. Settings reload. `config.refresh()` rebuilt the cache by reading and parsing a tab's JSON file once per field, so about 250 parses per refresh, 100 ms of CPU, and the metadata search, providers and config routes each trigger it. It now reads each file once per rebuild (several tabs share `settings.json`, so the memo is keyed on the file path). Library match. When the library check is on, each search result is compared against every library entry, and the title was split into words again for every entry. The words are now computed once per result. I measured 33 ms per result at 5,000 library entries and 106 ms at 20,000, which is a second or more per page of 40 results. The matching itself is unchanged. Cover URL probe. `is_covers_cache_enabled()` creates and deletes a file in the config directory to check it is writable, and it runs for every cover, queue task and history row. The answer is remembered for 60 seconds. The probe is 0.045 ms on its own, but it is called a few hundred times per request in the worst case. Each change has a test that fails without it (call counts on the index saves, file opens during a refresh, tokenisation calls, and probe calls). Ruff, basedpyright on the changed files and the non-e2e suite pass. The timings are from a dev machine with an empty config, so the real numbers depend on cache and library size. |
||
|
splitsec2
|
2668705bfa |
perf: reuse connections and cache Anna's Archive download counts (#1546)
Two small changes that stop Shelfmark from opening a new HTTPS connection for requests that go to the same host over and over. Anna's Archive download counts. Every search result gets its count from `/dyn/md5/inline_info/<md5>`, and each of those was a bare `requests.get`, so a search with 50 results opened 50 TCP and TLS connections to the mirror. `search_books` also runs once per title variant, and the variants of one book mostly find the same files, so the same md5 was looked up again for every variant that found it, and again when the release list was reopened. The lookups now share one `requests.Session`, and a successful count is kept for five minutes in a bounded `CacheService` (the same class the metadata cache uses, as its own instance, so it does not depend on the metadata cache toggle). A lookup that fails or comes back without a count is not cached, so it is retried on the next search just like today. The URL, headers and 5 second timeout are unchanged. Debrid clients. Real-Debrid, AllDebrid, Debrid-Link and TorBox also used the bare `requests` helpers. The download handler polls `get_status` every two seconds for each active job, so every poll was a fresh TLS handshake with the debrid API. Each of those modules now keeps one `requests.Session` and sends its API calls through it, the way the qBittorrent, Deluge, Prowlarr and Newznab clients already do. Headers, timeouts, `verify` and error handling are still passed per call exactly as before. The existing tests that patched `<module>.requests.get/post/request` now patch `<module>._session.<verb>` instead; nothing else in them changed. How I checked it: each change has a test that fails on main and passes here. They count the requests that reach `requests.Session.request` and the sessions built while doing it, with no network. On main, three polls of each debrid client build three sessions, and two overlapping batches of AA results look `bbb` up twice. There are also tests for the count cache expiring (with a faked clock) and for a failed lookup not being cached. Full suite: 4432 passed, 48 skipped. ruff and basedpyright are clean on the changed files. I also looked at caching `/api/releases` on the server and left it out. The release modal already caches each book's results per source and content type in the browser for five minutes (longer for IRC and slskd through `cache_ttl_seconds`), and that cache survives closing and reopening the modal, so reopening it does not run a new search today. A server cache would only help after a page reload, and it would quietly undo the modal's "apply filters" and manual search refetches, which drop the browser cache on purpose, unless the frontend also learned to send a bypass. That did not seem worth it for this PR. |
||
|
splitsec2
|
08fdb7ea87 |
fix(startup): the reconcile guard catches the same errors the rest of main.py does (#1540)
The start-up reconcile runs at import, before the coordinator starts, inside a try that named five exception types and left out `TypeError`. `main.py` already defines `_IMPORT_OPERATIONAL_ERRORS` for this shape and the other module-level guards use it, so this switches to it. An unexpected `TypeError` from the reconcile would otherwise fail the import of `shelfmark.main`, and the worker would fail to boot on every restart. I didn't use `except Exception` because BLE001 is on and nothing in main.py does. The test reloads main with the reconcile raising `TypeError`, and with `OSError` as the control (that one already passes on main). Full non-e2e suite passes, ruff and basedpyright are clean. Found in a Claude pass over 1.5.0. |
||
|
splitsec2
|
f560b31ee4 |
fix(db): wait up to 30 s for a busy writer instead of SQLite's 5 s default (#1539)
The users, download history and activity view-state databases open their connections with SQLite's default busy timeout of 5 seconds. A start-up migration, the restart reconcile, or any statement that overlaps a backup or a second process raised "database is locked" after 5 seconds, and the reconcile finished only part of its work. One shared `DB_BUSY_TIMEOUT_SECONDS` (30) in `core/utils.py`, used by all three. The test reads SQLite's `busy_timeout` pragma back from each connection, so it doesn't need a real lock fight. I left `UserDB.initialize()` without a guard on purpose. With 30 seconds the wait covers a backup, and swallowing a schema failure there would hide a worker that can't run. Found in a Claude pass over 1.5.0. |
||
|
splitsec2
|
e27fed1d40 |
fix(queue): a late progress update no longer reopens a finished task (#1538)
#1429 made the first terminal status win, but the guard only refused terminal to terminal changes. A worker's poll that reports DOWNLOADING after a cancel still overwrote CANCELLED (the poll loop calls the status callback before it waits on the cancel flag), and the COMPLETE or ERROR that followed was then accepted as a first terminal status, so the terminal hook ran a second time. This refuses any status change away from a terminal status. `add()` and `enqueue_existing()` reset the status themselves, so retries are unaffected. The new test cancels a downloading task, sends DOWNLOADING and then COMPLETE, and expects CANCELLED and one hook call. It fails on main and passes here. Full non-e2e suite passes, ruff and basedpyright are clean. Found in a Claude pass over 1.5.0, same as my other small fixes from it. |
||
|
splitsec2
|
395c331b4a |
fix(search): give each provider its own slice of the search budget (#1537)
Fixes #1531. All Direct Download providers share the one search budget (`RELEASE_SEARCH_TIMEOUT`, 300s). Anna's Archive runs first, and when its bypass keeps failing the retries use up the whole budget. OceanofPDF then sees an expired deadline, skips its queries, and the search raises Anna's Archive's error even though OceanofPDF never got a turn. I didn't split the budget evenly, since the deadline module itself says a cold-start Anna's Archive solve can legitimately take 60 to 120s and a 150s or 37s slice would break setups that work today. Instead `search_deadline.unit_deadline(waiting=n)` opens a child deadline for one unit of work. It gets the parent's remaining time minus 30s for each unit still waiting, never less than 30s and never more than the parent has left. A single provider still gets everything, a fast provider's unused time rolls on to the next, and a stuck one stops at its own limit with the reserve intact. It's used in the provider loop in `DirectDownloadSource.search`, and in the source loop in `/api/releases` since it's the same shape and four lines. With several sources enabled the first one now gets 30s less per source behind it, which is still well above a cold solve for the usual two or three. Tests: a stuck first provider no longer starves the second (fails on main, passes here), a lone provider gets the full budget, and a child never outlives its parent. Two existing tests in `test_releases_api_budget.py` tripped the deadline in force from inside the source, which is now the child, so they use a short request budget instead. ruff, basedpyright on the changed files and the non-e2e suite are clean. OceanofPDF is inactive on my own instance (no mirror URL set), so I haven't reproduced this live. It's verified by reading the code and by the tests. I had Claude do the pass on this one. |
||
|
splitsec2
|
8e425da527 |
refactor: one default and one reader for the Supported Book Formats list (#1532)
The default for Supported Book Formats (`epub, mobi, azw3, fb2, djvu, cbz, cbr`) was written out in five places: the settings registry, `postprocess/policy.py`, `irc/parser.py`, Prowlarr's `source.py` (added in #1490) and slskd's `source.py` (added in #1428). Both of those two arrived in 1.5.0, and both are mine, so this is cleaning up after myself as much as anything. The setting was also read three different ways. Post-processing split on commas only. slskd split on commas and spaces and stripped a leading dot. IRC split on commas and kept the dot. Prowlarr split on commas and kept `""` as a format when the value was an empty string. So `.epub` or `epub mobi` worked in one source and not another. This puts `DEFAULT_EBOOK_FORMATS` and `normalize_format_list` in `core/utils.py` next to `AUDIOBOOK_FORMATS`, and the registry, post-processing, IRC, Prowlarr and slskd all use them. I took the most lenient reading of the three (comma or space split, leading dot dropped, empties ignored, default when nothing is left), so no existing config gets stricter. I put the reader in `core/utils.py` and not in `policy.py` because the release sources can't import `policy` without pulling in the whole post-processing package. A config with a sensible list reads exactly as before. The only differences are for malformed values: an emptied setting or an empty string now falls back to the default in every place, where IRC used to filter everything out and Prowlarr treated `""` as a format. The audiobook default is a separate story and I left it alone: the registry and slskd default to every processable audiobook format plus the archives, while post-processing and IRC fall back to `m4b, mp3`. They only differ when the setting is unset or emptied, but they do differ, and I'd rather ask which one you want than pick one here. Ruff, basedpyright on the changed files and the backend suite pass (4400 passed, e2e not run). New tests cover the reader and the two fallbacks. |
||
 Amalkrishna A SandCaliBrain
|
a11a63a29e |
feat(notifications): add {User}/{Email} placeholders to per-user routes (#1499)
Closes #1130. Per-user notification route URLs can now use `{Email}` and `{User}`, filled in from each user's account at send time, the same way `{User}` already works in destination paths. One default route like `mailtos://user:pass@smtp.example.com?to={Email}` now reaches every user at their own address, with no per-user setup. - Values are URL-encoded (`alice+books@example.com` keeps its `+`); placeholders are case-insensitive. - A route that needs a value the user doesn't have (e.g. no email on file) is skipped for that user; other routes still send. - The user lookup only runs when a route contains a placeholder. - The personal "Test notification" button expands the placeholders too, so testing a `{Email}` route works. - Added to the routes setting help, `docs/users-and-requests.md`, and tests. @calibrain ready for review. --------- Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
 
|
c43da4537d |
fix(network): apply HTTP proxy to Open Library and Prowlarr requests (#1526)
Fixes #1524 Open Library metadata searches and Prowlarr API calls used `get_ssl_verify` but never `get_proxies`, so the HTTP proxy set in Settings → Network was not applied to their HTTPS requests. Both now pass the configured proxies. Relying on `requests`' `trust_env` with only `HTTP_PROXY` set leaves the same gap for `https://` destinations. Verification (Linux, `uv run --frozen pytest`): - New tests `tests/metadata/test_openlibrary_proxy.py` and `tests/prowlarr/test_api_proxy.py`: 4 passed. With the source change reverted they fail (2 failed, 2 errors). - `tests/metadata` + `tests/prowlarr`: 949 passed, 42 skipped. - The full suite in my environment showed 61 failures and 26 errors outside these areas. I did not compare them against `main`, so I'm not claiming the full suite is green here; CI will show it. - `ruff check` and `ruff format --check` on the changed files pass. Prepared with AI assistance; reviewed and tested by me before submitting. Co-authored-by: dbdefault <246079841+dbdefault@users.noreply.github.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
CaliBrain
|
1aa9846f22 |
fix(download): trust a pinned AA mirror and repair the full e2e profile (#1518)
The destination guard from the oceanofpdf/SSRF fix (
|
||
 DBVamsiandCaliBrain
|
2fa0ad27f3 |
Fix [BUG] Search "Options" move out of the way when you try to click them (#1468)
Fixes #1301. ## What changed ## Summary **Root Cause (Issue #1301):** The SearchBar component uses a `useDismiss` hook that closes the main search menu on `mousedown` events occurring outside of `selectorRef` and `controlsPanelRef`. However, the AdvancedFilters panel (opened via the "Options" button) is rendered as a sibling component outside of `controlsPanelRef`. When clicking on dropdown options inside AdvancedFilters, the `mousedown` event bubbles up to the document, triggering `useDismiss` and collapsing the main menu. **Fix:** Added `onMouseDown={(e) => e.stopPropagation()}` to the `settingsForm` div in `AdvancedFilters.tsx`. This prevents the `mousedown` event from bubbling up to the document level, so the `useDismiss` hook in SearchBar never sees the event and the main menu stays open. **Test:** Created `tests/frontend/test_search_menu_dismiss.py` with 3 tests: 1. `test_advanced_filters_prevents_mousedown_bubbling` - Verifies the fix is in place 2. `test_advanced_filters_container_has_onmousedown` - Verifies the handler is on the correct element 3. `test_searchbar_uses_usedismiss_for_menu_closing` - Documents the existing dismissal behavior **Verification:** - Tests pass with the fix: ✓ - Tests fail without the fix: ✓ (2 tests fail as expected) - Existing tests still pass: ✓ (215 tests in config/) ## How it was verified - test(s) added/changed: `tests/frontend/test_search_menu_dismiss.py` - on the original code the new test fails (exit 1) - with the fix it passes (exit 0) - full test suite with the fix: exit 1, no new failures vs. baseline (exit 1) (153.7s) This was worked on in conjunction with AI tools. Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
 Desislav LesovandCaliBrain
|
fc8a5b9230 |
fix(prowlarr): hide results whose title extension is not a supported format (#1490)
Closes #1481 ## Problem Plain (non-enriched) indexers such as Zamunda, Knaben or LimeTorrents return titles like `Иван Вазов - Под игото.doc`. They show next to the epubs. Picking one downloads the file, then post-processing discards it because `doc` is not in `SUPPORTED_FORMATS`. ## Change - `_unsupported_title_extension()` reads a trailing file extension from the release title. When it is a known file format that is not in Supported Book Formats, the result is hidden. - Kept: titles with no extension, `[EPUB]`-style tags, dotted titles like `Jane.Eyre.by.Bronte` (only known file extensions count), archives (`zip`, `rar`, `7z`), and audiobook searches. - New Prowlarr setting `PROWLARR_HIDE_UNSUPPORTED_FORMATS` ("Hide results in formats you did not enable"), default on. - One info log line per search with the hidden count; a debug line per hidden title. - `docs/environment-variables.md`: added only the new setting's rows. Running `scripts/generate_env_docs.py` on current `main` also changes unrelated lines (`SHELFMARK_API_KEY_READONLY`, the slskd anchor), so I left those alone. ## Tests - New `TestHideUnsupportedFormats` in `tests/prowlarr/test_source.py`: the extension helper (parametrised), audiobook bypass, comma-separated config, and a full `search()` with the setting on and off. - `tests/prowlarr`: 784 passed. Ruff lint/format, vulture and basedpyright on the changed files are clean. - Full suite on macOS: the same 84 bypass/entrypoint failures as on `main` (browser extra and Linux permissions), none new. I ran an earlier version of this as a local patch on v1.4.0 against a real Prowlarr + Zamunda setup: the `.doc` row is gone and epubs stay. Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
CaliBrain
|
d42494c696 |
fix(tor): survive container restarts and run Tor as debian-tor (#1517)
With USING_TOR=true, any restart that reuses the container (docker
restart, a Docker daemon restart, a host reboot) put it in a restart
loop. Only recreating the container ended it.
Tor appends to /var/log/tor/notices.log, and the file survives the
restart. tor.sh waited for Tor by grepping that log for "Bootstrapped
100%", so the previous boot's line passed at once, before Tor had
connected to anything. The NAT rules then went live, Tor could not
finish bootstrapping behind them, the connectivity check could not
resolve check.torproject.org, and the next boot found the same stale
line. tor.sh now moves the previous log to notices.log.prev before it
starts Tor, so the wait and the watchdog only see this boot's output.
Tor could not finish behind the NAT rules because it ran as root. The
rules exempt only debian-tor's traffic from the transparent redirect, so
every relay connection Tor opened after they went live was redirected
into its own TransPort: the log shows dozens of TLS handshakes dying.
This also meant a Tor that lost its guard could never reconnect, on any
kernel. torrc now has `User debian-tor` and `DataDirectory
/var/lib/tor`. Tor still binds DNSPort 53 as root, then drops. It opens
its log after dropping and refuses to start if it cannot, so tor.sh
hands the fresh log to debian-tor first.
The check also failed without a word. `RESULT=$(curl -s ...)` under
`set -e` ended tor.sh with curl's exit status, so the only trace was
ExitCode=6 (could not resolve host) in `docker inspect`. A reply that
was not the expected JSON did the same through grep. The check now:
- retries twice, 10s apart, since a fresh circuit can fail its first
lookup;
- logs each failure with curl's exit status and error;
- says when it gives up and prints the last Tor log lines;
- still stops the container when traffic does not go through Tor.
- Reproduced on the v1.4.0 image (tor.sh is unchanged since then): after
one `docker restart` the container restarted 9 times in 150s, each
boot ending at "Verifying Tor connectivity...".
- With the new tor.sh mounted into the same image, the container went
through 2 restarts in a row. Each boot verified Tor and reached
gunicorn within 15-21s, and the restart policy never had to step in.
Tor runs as debian-tor and its logs belong to it.
- Restarting Tor (`supervisorctl restart tor`) once the NAT rules were
live, so it had to open new relay connections: as debian-tor it
bootstrapped again in 4s and traffic still went through Tor. As root
it had not bootstrapped after 120s, 1775 TLS handshakes had died, and
curl could not resolve check.torproject.org (exit 6).
- A SIGHUP, which the rotation monitor sends every 5 minutes, keeps
DNSPort 53 working after the drop to debian-tor.
- 9 new tests in tests/config/test_tor_script_rules.py. 8 of them fail
on the old tor.sh; the 9th checks that traffic outside Tor still stops
the container. Three need `grep -P` and skip on macOS, so I also ran
the file in the Debian image: 23 passed.
- The `make python-checks` tools pass (ruff check and format,
basedpyright on the app and tests, vulture). The non-e2e suite
passes: 4366 tests on
|
||
|
amura11andClaude Sonnet 5.5
|
ceb23dda7c |
feat(clients): separate audiobook download directory for torrent clients (#1500)
## Summary Related to #1048 (covers the "separate book and audiobook locations" request, not per-category routing). Every download client already had a separate audiobook label/category, but the four torrent clients only had a single save directory, so books and audiobooks ended up mixed in the client's download folder. Each torrent client (qBittorrent, Transmission, Deluge, rTorrent) gets an opt-in **Separate Book and Audiobook Directories** toggle (`<CLIENT>_SEPARATE_DOWNLOAD_DIRS`, default off): - **Off (default):** unchanged. The single `<CLIENT>_DOWNLOAD_DIR` is used for everything, so existing setups behave exactly as before. - **On:** `<CLIENT>_DOWNLOAD_DIR_BOOK` and `<CLIENT>_DOWNLOAD_DIR_AUDIOBOOK` replace the shared field. A directory left empty means the client's own default; it never falls back to the shared directory or another Shelfmark setting. The content type was already passed to each client's `add_download`, so a small shared helper (`resolve_download_dir`) does the selection. Usenet clients are unchanged: their save location comes from the category configured in the client, and a separate audiobook category is already sent. ## Screenshots Default Setting <img width="817" height="271" alt="Screenshot 2026-10-20 at 14-54-26 Shelfmark" src="https://github.com/user-attachments/assets/29941f83-9034-4ea5-bcb2-8e91ce77e63a" /> Separate Locations <img width="817" height="425" alt="Screenshot 2026-10-20 at 14-54-58 Shelfmark" src="https://github.com/user-attachments/assets/772b7194-a072-4945-bac5-b6959550fd75" /> ## Not included Routing to many folders by document type (the main request in #1048) still only exists for Direct download via Content-Type Routing. Extending it to torrent and usenet sources needs a decision on what to route by. ## Testing - `pytest tests --ignore=tests/e2e` passes (3561 passed); ruff clean. The `tests/e2e/platform` suite needs a running cluster and was not run. - New unit tests for the helper (including no fallback to the shared directory) and a qBittorrent add test covering book vs audiobook directories. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
splitsec2andCaliBrain
|
f837506017 |
Merge commit from fork
A signed-in user picks the release fields of POST /api/releases/download, and a record id with the "oceanofpdf:" prefix reached the provider without handles() or is_enabled(). The provider then fetched source_url, POSTed to the form action found on that page and fetched the meta-refresh target, with no check on any of them. The provider now refuses to run when it is switched off, and the book page, the form action and every POST redirect hop must be on the configured mirror (same scheme, host and port). The signed file link may sit on another host but must be http(s). network.ensure_public_destination refuses a host that resolves to a loopback, private, link-local or otherwise non-global address (IPv4-mapped IPv6 judged by its IPv4 address). Hosts the operator configured as mirrors and names that do not resolve pass as before. html_get_page and download_url call it on their first URL and check each redirect hop through a response hook. The image cache uses the shared address check too. Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
splitsec2andCaliBrain
|
cb72ce807c |
Merge commit from fork
A user could set any value for their own overridable settings through
PUT /api/users/me. DESTINATION and DESTINATION_AUDIOBOOK accepted any
string, so a download (including one an admin fulfilled for that user)
could be written to any directory the container can reach, and
get_destination created it on demand.
A user's own destination must now resolve inside the global one (with
{User} expanded on both sides). Absolute paths outside it and any ".."
segment are rejected at save time, and get_destination applies the same
check when the path is used, so a stored or hand-written value outside
the global destination is ignored with a warning and the global
destination is used instead. A relative value is taken relative to the
global destination.
Other self-service values are checked as well: select fields must use
one of their options, EMAIL_RECIPIENT must be a plain address (the same
check the orchestrator and the downloads tab use, now shared from
request_helpers), and Grimmory library and path ids must be non-negative
integers. Admin saves on a user's behalf are unchanged.
The self-service 400 response now carries a message line so the reason
reaches the user in the toast.
Co-authored-by: CaliBrain <calibrain@l4n.xyz>
|
||
|
CaliBrain
|
c269230ef3 |
fix: review regressions in metadata, notifications, activity redaction, search options and genDebug (#1510)
- **Open Library (#1451): keep the work unless the search matched a translation.** Open Library shows an English edition whenever one matches the query. So "Mistborn" returned the 2023 "Mistborn" reissue (2023) instead of "The Final Empire" (2001), and "Children of Dune" returned "Children of Dune (Dune Chronicles, Book 3)" (1987). The matched edition's title, ISBNs, cover, language and year now win only when the edition is in another language and the query doesn't name the work's own title. Checked against live Open Library: "Children of Dune" and "Mistborn" give the v1.4.0 results again, and "Der Weg der Könige" and "Kinder des Nebels" still return the German editions. - **Google Books (#1434): the ISBN cache decorator is back on `search_by_isbn`.** It had moved to the new `search_paginated`. Failed searches were cached as empty pages, and ISBN lookups lost their cache. - **Notifications: Test notification shows the plugin's failure reason again.** Apprise 2 delivers on its own worker threads, even for a single service when that service has a deadline. `_AppriseLogCapture` filtered records by thread id and dropped all of them, so the UI only said "json: delivery failed". Apprise runs those workers in a copy of the caller's context, so the capture now filters on a context variable. That still keeps concurrent dispatches apart. `AppriseAsset(async_mode=False)` would not fix this, because the sequential path also uses the worker pool when a service has a deadline. - **Activity: `activity_update` no longer sends the raw task id to non-admins.** The owner's `user_<id>` room got the raw task id. For a request-linked download that id can hold a private tracker's passkey. It now goes through `viewer_download_id`, the same mapping `/api/status` and `download_progress` use. The admins room still gets the raw id. - **SearchBar (#1476): Escape and an outside click close the homepage's inline options panel again,** as they did in v1.4.0. - **genDebug.sh: packages are listed through `importlib.metadata`,** since #1404 removed pip from the image. The script also now redacts: - `PROWLARR_MAM_ID` - `user:pass@` credentials in URLs (proxies), in both the environment and the config files - notification route URLs, which carry their service's token (Telegram, Discord and others) - `make python-checks`, `make python-test`, `make frontend-checks` and `make frontend-test` pass on top of current `main`. - New tests cover: - Open Library edition selection - Google Books caching - log capture against a real Apprise 2 delivery to a local server that answers 500, plus a concurrent dispatch on another thread - the `activity_update` payload sent to each room - genDebug redaction and the package listing Each one fails on the old code. - I checked the SearchBar fix in the running app, because the frontend has no DOM test setup: Escape closes the panel, an outside click closes it, and a click inside keeps it open. - I ran the genDebug redaction under GNU sed (Debian image) and BSD sed. The package listing ran in the pip-less image. |
||
|
CaliBrain
|
a00f3d28bf |
fix(slskd): refuse unsafe peer paths, keep queued downloads alive, report search outages (#1512)
Fixes four defects in the Soulseek (slskd) A peer names its own files, and Shelfmark works out where slskd stored one from the last folder and the file name. `remote_directory_name` returned the whole directory when its last segment was empty. So `../../../etc//passwd.epub` or `/abs/path//book.epub` resolved outside the downloads folder, and Shelfmark imported whatever was there. For a multi-file release it moved those files into staging (`STAGE_MOVE`). - New `safe_remote_segments` refuses a name with an empty, `.` or `..` segment (which covers absolute and UNC paths), a NUL, or a drive anywhere but a Windows share root (`C:\Books\…`, which never reaches a local path). - Search (`iter_peer_files`, so `build_releases`) drops such files. - `download` refuses them before anything is queued, in both modes. The spec arrives with the queued release payload, so search filtering alone is not enough. - `local_path_for` also refuses a path that resolves outside the root (`resolve()` + `is_relative_to`), for example through a symlinked folder. `_poll`, used when isolation is off and as the fallback for slskd < 0.26, never asked for an activity grace. A file "Queued, Remotely" sent the same status and progress on every poll, so the orchestrator cancelled it after 300s and `SLSKD_QUEUE_TIMEOUT_MINUTES` never applied. It now requests, renews and releases the grace the same way isolated mode does. `search` caught every exception per query, so slskd being down or rejecting the API key showed as "no releases". When no query gets an answer it now raises `SourceUnavailableError`, as Prowlarr does, and the error reaches the user. One failed query among answered ones is still not an outage. A retry reuses the task's folder, and slskd renames a new file when one is already there. Files from a failed attempt therefore meant `_folder_complete`'s exact count could never match again. Now: - a failed or cancelled attempt removes its folder; - a new attempt removes any finished files an earlier one left (for example after a restart) before it queues the batch. `tests/slskd/conftest.py` stubbed `flask_socketio` whenever it had not been imported yet, even though it is installed. Running `pytest tests/slskd tests/` therefore broke the Flask app tests in the same worker (285 errors, also on |
||
|
CaliBrain
|
49aeec97cd |
fix(download): stop deleting user data and losing completed downloads (#1511)
Fixes six defects. Each one breaks downloads or deletes user data, and each has a regression test that fails on |
||
|
splitsec2andCaliBrain
|
520b6b16d0 |
fix(activity): Clear Completed leaves out an item that is no longer finished instead of failing the batch (#1508)
Clear Completed can fail outright with "Only terminal requests can be dismissed". `dismiss-many` answers 409 and dismisses nothing when any one request or download in the batch is not terminal. The list a user clears from can be older than the server, so a request that was reopened or retried since, or a download that started again, takes the whole batch down with it. The earlier fix for pending requests (#1438) filters on the client, which only helps when the list is current. `dismiss-many` now skips such an item and carries on with the rest. The response adds `skipped_item_keys` when something was left out, and `count` is what was actually dismissed. A single `dismiss` of a non-terminal item still answers 409. I hit it with a request whose download had completed while the request itself was still pending, but any reopen between the list and the click does it. This touches the same handler as #1507, so whichever lands second needs a small rebase. Tests: a batch with a finished and a reopened request dismisses the finished one and reports the other, and a batch with a finished and a running download does the same. Ruff, basedpyright and the backend suite pass (3933 passed, e2e excluded). Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
splitsec2
|
61c7ad3c62 |
Merge commit from fork
* fix(torrent): check each redirect hop when fetching a torrent file The torrent file fetch followed redirects by hand without looking at where a hop pointed, so a download link could make the server request a loopback, link-local or LAN address. Each hop (the initial URL and every redirect Location) must now be http or https and resolve only to global addresses. The configured Prowlarr/Newznab origin is exempt, as it usually lives on the LAN. The API key still only goes to that origin. * fix(socketio): only accept websocket handshakes from the app's own origin The Socket.IO server allowed every origin. It now uses the same-origin check python-engineio applies by default (Host, or X-Forwarded-Proto and X-Forwarded-Host behind a proxy). SOCKETIO_ALLOWED_ORIGINS takes a comma separated list of extra origins for proxies that change Host without forwarding it. * fix(custom-script): do not pass a file name that starts with a dash as the first argument In relative path mode the script got the organized file's relative path as its first argument, which many scripts read as an option when the name starts with a dash. Such a name is now passed with a ./ prefix. Absolute mode is unchanged. |
||
|
splitsec2
|
8ec9db49da |
Merge commit from fork
The request line sent to the book bot comes from the request-supplied source_id and was written to the socket as-is, so a line break in it sent extra IRC commands as the operator's nick. IRCClient._send, the single writer, now raises IRCError on any control character (CTCP 0x01 excepted), the download handler rejects such a request line with a clear error before connecting, and search refuses a query that contains one. |
||
|
splitsec2
|
6f65d6e20a |
Merge commit from fork
A release format such as /../../x was appended to the staging file name unchanged, so the direct-download and libgen handlers could write to, and on cancel unlink, a path outside TMP_DIR. Normalise the format to a plain extension at queue time and in build_filename, reject path-like source ids for the sources that use them as file names, and make both handlers resolve the staging path and refuse anything that is not a direct child of TMP_DIR. |
||
|
splitsec2
|
89663e32f6 |
Merge commit from fork
A user's personal notification routes were sent to any Apprise URL, from the server. Through the test endpoint or a saved route a signed-in user could make the server POST to loopback, link-local and LAN hosts, choosing the path and, with Apprise's + and : parameters, extra headers and body fields. User routes are now limited to the hosted services that talk to a fixed vendor host (discord, slack, tgram, pover, pbul) and to ntfy and gotify, whose host must resolve only to public addresses (checked on the host urlsplit reads and on the host Apprise reads, including IPv4-mapped and numeric forms). The check runs when a route is tested, when an event is queued and again when it is sent. Admin routes are unchanged. |
||
|
splitsec2
|
d34b200414 |
fix(activity): answer another user's item like a missing one, and skip foreign history rows (#1507)
Two small fixes in `activity_routes.py`. **Another user's item answers like a missing one.** Dismissing a task or request id answered 404 when it didn't exist and 403 when it belonged to someone else, so a user could test whether an arbitrary id exists. A non-admin now gets the same 404 for both. In dismiss-many a foreign key is reported with the missing ones and the rest carry on, and the owner is still written to the log. Admins are unchanged. **A foreign row in someone's viewer state no longer gives a 500.** If a non-admin's viewer state referenced another user's download or request row, building history raised a `RuntimeError`. The row is now skipped with a warning. Not changed: history still raises when a row that viewer state points to has been deleted. That is a different case, and I didn't want to widen this. Tests: the existing forbidden test now expects the 404 and the log context, and four new ones cover a foreign download and a foreign request answering like a missing one (including dismiss-many), an admin still dismissing a foreign row, and history skipping foreign rows. Ruff, basedpyright and the backend suite pass. |