mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-11-14 16:28:24 +00:00
86fff322976699dc6c27c23fdf8f413dfd00be76
94
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 splitsec2
|
86fff32297 |
feat(hardcover): owned books, simpler queries, dead sources and failed releases (#1535)
A request whose book turns out to be in the library is closed with the note `[auto] Already in the library.` instead of sitting in Needs Review. A request an admin handled by hand while a pass was running is skipped. A series label in front of a Calibre title no longer hides an owned book, and a Calibre title marked as another edition (full cast, dramatized, GraphicAudio) is not the book. When no source matches, up to three simpler forms of the title are tried on each source under the same strict match. Optionally (off by default), an unmatched audiobook request is looked up on Audible US and UK through Audiobookshelf's metadata search and the request says what the store answered. It never declines anything. A Soulseek peer whose recent transfers all failed, and an AudiobookBay post whose comments say it has no seeders, are tried after the other matches and never excluded. A failed or stalled release is kept out for AUTO_DOWNLOAD_RETRY_FAILED_RELEASE_DAYS and then tried again, and a torrent that died for one request is not offered to another. A request stuck on queued or downloading is settled from the download history. Optionally (off by default) a failed download is retried with another release within minutes. |
||
|
splitsec2
|
9d5b47d17b |
feat(hardcover): retries, error counting and a back-off for requests nothing matches (#1534)
A synced request whose download failed, was interrupted by a restart, or was cancelled by the stall timer goes back to pending after AUTO_DOWNLOAD_RETRY_DAYS (7 by default, 0 turns it off), a few per pass, oldest first. Only what never arrived is retried, the failed release is remembered, and a request is given up on after three failed retries. Downloads cancelled by a person and rejected requests are never retried. Errors count as errors: a search that failed in one source no longer reads as no match, a throttled Hardcover lookup is an error and not a skip, and a request that blows up doesn't abort the pass. Both summaries carry the error count. A request for the same Hardcover book and format is skipped when the history already has a finished or running download for it. A request that finds nothing is searched once a day after three no-match passes, then every two days, up to AUTO_DOWNLOAD_BACKOFF_MAX_DAYS (3 by default, 0 turns it off). |
||
|
splitsec2
|
4389548deb |
feat(hardcover): ebooks, audiobooks or both, and stricter matching for automatic downloads (#1533)
A shelf book can be requested as an ebook, an audiobook or both (one request per format), and each request is checked against its own library, source priority list and format guard. Ranking within a source prefers the most downloaded copy, then size, and one m4b is preferred to a set of chapter files unless it is much smaller than the other copies. Soulseek peers are ranked by free slot, queue length and speed. Automatic downloads refuse multi-book packs, sequels and other works named in the title, study guides, abridged or dramatised versions, two-in-one titles, releases in a language the requester didn't ask for, and an archive for an audiobook request. Matching folds accents on both sides, drops an alternate title for the search and the match, and reads small numbers the same in any form. AudiobookBay releases with no author in the listing title are judged on the authors credited on the detail page. |
||
|
splitsec2
|
ea69db3f8e |
feat(library): Audiobookshelf library check, and the sync fails closed when a library cannot be read (#1519)
Adds an Audiobookshelf provider for the library check, next to the Calibre one. It reads the server over its REST API (title, author, ISBN and ASIN per item), answers for audiobooks, and has its settings in General with a test button that uses the unsaved form values. Read only. The Hardcover sync and automatic downloads now fail closed when an enabled library cannot be read: the sync skips its cycle and an automatic download holds the request. A library that listed items at the last index and now lists none counts as a failed read. Badges still fail open. |
||
 ProcyonX1
|
99a6aa206b |
feat(metadata): add Deutsche Nationalbibliothek (DNB) metadata provider (#1547)
## feat(metadata): add Deutsche Nationalbibliothek (DNB) metadata provider This adds the Deutsche Nationalbibliothek (DNB) as a metadata provider. It covers German editions far better than the existing providers. No API key, disabled by default, and the same pattern as the other providers. It uses DNB's public SRU interface and parses MARC21-XML. Thats one reason why this needs more code than the Google Books or Open Library providers. The others are the search (DNB has no relevance ranking) and the language filter, see below. ### Notes - **Search box.** Search-box text goes from precise to loose until a stage has hits: the complete title, then the title as a phrase, then every word in a title or a person's name, last without German articles. - **"Only show editions in my book languages"** (optional, on by default). DNB also catalogs a lot of translations. DNB filters by the searching user's Default Book Languages, else the system default. - **No covers.** DNB's cover service is behind Anubis bot protection, so the provider returns no cover URLs. - **Proxy support** already included like Open Library since #1526 ### Changes outside the provider - DNB is in the onboarding dropdown. - Calibre's `dnb-idn` identifier is mapped, so DNB results show library ownership - In the env-var docs I added only the DNB section. Regenerating the whole file on main currently also reorders unrelated sections and drops `SHELFMARK_API_KEY_READONLY` (and the Databáze knih section sits inside the Moly.hu block). ### Ideas - **Deduplication.** DNB lists every edition. Kafka's "Die Verwandlung" has around 200 German editions of the same text. Grouping them would touch the API, the frontend and release search. - **Language-aware metadata search** for all providers. The other providers ignore Default Book Languages in metadata search. The `user_id` added here could serve them too. |
||
  
|
2007384058 |
feat(sources): add Webshare.cz as a release source (#1543)
## What Adds [Webshare.cz](https://webshare.cz), the Czech file host, as a release source for ebooks and audiobooks. It follows the Libgen source: disabled by default, with a settings tab for credentials and a **Test Login** button that reports whether the account is VIP. ## Why Webshare is one of the main places Czech and Slovak ebooks and audiobooks are shared, and none of the existing sources reach it. Together with the Databáze knih metadata provider (#1479), this lets Universal mode find Czech and Slovak titles end to end. ## How - **API client** (`webshare/api.py`): Webshare's XML API (`POST /api/<endpoint>/`). Login is the salted exchange the API documents: fetch the salt, send `SHA1(MD5_CRYPT(password, salt))` plus an MD5 digest. Python dropped the `crypt` module in 3.13, so md5-crypt is a small pure-Python port, checked against `openssl passwd -1`. The session token is cached per account and refreshed once when an authenticated call fails. File-level errors (`FILE_LINK_FATAL_*`) are not retried. XML is parsed with `defusedxml`. - **Search** (`webshare/source.py`): Webshare matches file names only, so the file name becomes the release title and its extension the format. Queries are title + author, then title only, or the manual query. Results are filtered to the configured `SUPPORTED_FORMATS` / `SUPPORTED_AUDIOBOOK_FORMATS`, plus `zip`/`rar` for audiobooks, since they are usually uploaded as archives. Password-protected files are skipped. The language is guessed from tags in the name (`CZ`, `SK`, `ENG`, ...), and the vote balance is a sortable column. API failures raise `SourceUnavailableError`. - **Download** (`webshare/handler.py`): resolves a time-limited link per attempt with `file_link`, then streams it into `TMP_DIR` with progress and cancel support. Large audiobook archives are not buffered in memory. - **Settings**: `WEBSHARE_ENABLED`, `WEBSHARE_USERNAME`, `WEBSHARE_PASSWORD` (a `PasswordField`, already covered by the `_PASSWORD` redaction in `genDebug.sh`), and `WEBSHARE_MAX_RESULTS`. The `docs/environment-variables.md` section was added. ## Testing - `tests/webshare/`: 31 offline tests. They cover md5-crypt vectors, password hashing, XML parsing, token reuse and re-login, non-retried file errors, availability gating, format filtering, query fallback, release mapping, language guessing, and the handler's streaming, small-file rejection, link errors and cancel. - `ruff check` and `ruff format --check` are clean. The rest of the unit suite fails the same way with and without this change: Windows-only tests on my machine. - Verified live against webshare.cz: search parsing and the salt endpoint's error path. - Still a draft: I haven't yet verified login and download with a real account. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
48ae3dbef4 |
feat: optional Hardcover wishlist sync with automated downloading (1 of 5) (#1516)
This replaces #1047, which @InfiniteAvenger opened and calibrain offered to re-review. It is Caleb's feature commit with his authorship kept, rebased onto current main, with small fixes, tests and a docs page on top. Everything is opt-in and off by default, so nothing changes for anyone who doesn't turn it on. What it adds is a **Settings, Hardcover Sync** tab with three parts that can be switched on separately: - **Shelf sync**: a Hardcover shelf (Want to Read by default) is pulled into requests on a schedule, or with a Sync now button. Requests carry the Hardcover metadata and cover. - **Automatic downloads**: synced requests are approved and a release is picked from a source priority list. A strict title, author and format match (plus a seeder minimum for torrents) decides, so anything without a confident match stays pending for manual review. - **Skip books you already own**: sync and automatic downloads ask whichever libraries are enabled under Library Check (Calibre today) and skip a book that is already there. On top of Caleb's commit I made the request owner a lookup instead of assuming user id 1, sent synced cover URLs through the proxy when the request is created, got ruff and basedpyright clean, and added tests and `docs/hardcover-sync.md`. His commit also had its own Audiobookshelf settings and a Test library connection button, written before the library check was split into providers. The button would have raised a TypeError against the current `test_connection`, so those settings are gone from this tab and an Audiobookshelf provider comes as a separate PR, which is the part of #1047 that is missing here (for now anyways...). I've run this daily for about a couple of weeks. Ruff, basedpyright and the backend suite pass (4426 passed). This is 1 of 5. As calibrain suggested, the rest are stacked on this one, each saying which number it is. Closes https://github.com/calibrain/shelfmark/pull/1047#issuecomment-6302071588 --------- Co-authored-by: InfiniteAvenger <calebewest02@gmail.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
splitsec2
|
2d9491e063 |
docs(settings): say that the per-source limit replaces Max Concurrent Downloads (#1542)
With Max Concurrent Downloads Per Source above 0 the download pool is sized per source and a task is admitted when its own source is below the limit. Max Concurrent Downloads isn't read at all in that mode, so 3 per source across 3 busy sources runs 9 at once. Neither description said so. The per-source text now says it replaces the global limit and what the total can be, and the global one says it isn't used while the per-source limit is set. Text only, no behaviour change. I changed the two rows in `docs/environment-variables.md` by hand. Running `scripts/generate_env_docs.py` on main today also drops the `SHELFMARK_API_KEY_READONLY` entries and changes the slskd anchor, so the file looks hand-edited since it was last generated, and I didn't want to fold that into this. The two rows match what the generator produces for them. Found in a Claude pass over 1.5.0. |
||
 Amalkrishna A SandCaliBrain
|
a11a63a29e |
feat(notifications): add {User}/{Email} placeholders to per-user routes (#1499)
Closes #1130. Per-user notification route URLs can now use `{Email}` and `{User}`, filled in from each user's account at send time, the same way `{User}` already works in destination paths. One default route like `mailtos://user:pass@smtp.example.com?to={Email}` now reaches every user at their own address, with no per-user setup. - Values are URL-encoded (`alice+books@example.com` keeps its `+`); placeholders are case-insensitive. - A route that needs a value the user doesn't have (e.g. no email on file) is skipped for that user; other routes still send. - The user lookup only runs when a route contains a placeholder. - The personal "Test notification" button expands the placeholders too, so testing a `{Email}` route works. - Added to the routes setting help, `docs/users-and-requests.md`, and tests. @calibrain ready for review. --------- Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
CaliBrain
|
d887942630 |
fix(search): name the proxy when a gateway cuts a search off (#1522)
A reverse proxy whose read timeout is shorter than the release search budget (nginx defaults to 60s) answers a slow search with its own 504 page. describeSearchFailure found no server message there and fell back to "Unable to reach download source. Network may be restricted or mirrors blocked.", which sent users off changing mirrors (discussion #1520). - 504 and Cloudflare's 524 now say the proxy stopped waiting and that its read timeout should be at least the release search timeout. - A 502/503 with a non-JSON body keeps fetchJSON's "Server unavailable" wording instead of the mirrors line. - docs: the root-path nginx example sets proxy_read_timeout and proxy_send_timeout like the subpath ones, and a new troubleshooting section covers searches that time out behind a proxy. |
||
 Desislav LesovandCaliBrain
|
fc8a5b9230 |
fix(prowlarr): hide results whose title extension is not a supported format (#1490)
Closes #1481 ## Problem Plain (non-enriched) indexers such as Zamunda, Knaben or LimeTorrents return titles like `Иван Вазов - Под игото.doc`. They show next to the epubs. Picking one downloads the file, then post-processing discards it because `doc` is not in `SUPPORTED_FORMATS`. ## Change - `_unsupported_title_extension()` reads a trailing file extension from the release title. When it is a known file format that is not in Supported Book Formats, the result is hidden. - Kept: titles with no extension, `[EPUB]`-style tags, dotted titles like `Jane.Eyre.by.Bronte` (only known file extensions count), archives (`zip`, `rar`, `7z`), and audiobook searches. - New Prowlarr setting `PROWLARR_HIDE_UNSUPPORTED_FORMATS` ("Hide results in formats you did not enable"), default on. - One info log line per search with the hidden count; a debug line per hidden title. - `docs/environment-variables.md`: added only the new setting's rows. Running `scripts/generate_env_docs.py` on current `main` also changes unrelated lines (`SHELFMARK_API_KEY_READONLY`, the slskd anchor), so I left those alone. ## Tests - New `TestHideUnsupportedFormats` in `tests/prowlarr/test_source.py`: the extension helper (parametrised), audiobook bypass, comma-separated config, and a full `search()` with the setting on and off. - `tests/prowlarr`: 784 passed. Ruff lint/format, vulture and basedpyright on the changed files are clean. - Full suite on macOS: the same 84 bypass/entrypoint failures as on `main` (browser extra and Linux permissions), none new. I ran an earlier version of this as a local patch on v1.4.0 against a real Prowlarr + Zamunda setup: the `.doc` row is gone and epubs stay. Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
 amura11andClaude Sonnet 5.5
|
ceb23dda7c |
feat(clients): separate audiobook download directory for torrent clients (#1500)
## Summary Related to #1048 (covers the "separate book and audiobook locations" request, not per-category routing). Every download client already had a separate audiobook label/category, but the four torrent clients only had a single save directory, so books and audiobooks ended up mixed in the client's download folder. Each torrent client (qBittorrent, Transmission, Deluge, rTorrent) gets an opt-in **Separate Book and Audiobook Directories** toggle (`<CLIENT>_SEPARATE_DOWNLOAD_DIRS`, default off): - **Off (default):** unchanged. The single `<CLIENT>_DOWNLOAD_DIR` is used for everything, so existing setups behave exactly as before. - **On:** `<CLIENT>_DOWNLOAD_DIR_BOOK` and `<CLIENT>_DOWNLOAD_DIR_AUDIOBOOK` replace the shared field. A directory left empty means the client's own default; it never falls back to the shared directory or another Shelfmark setting. The content type was already passed to each client's `add_download`, so a small shared helper (`resolve_download_dir`) does the selection. Usenet clients are unchanged: their save location comes from the category configured in the client, and a separate audiobook category is already sent. ## Screenshots Default Setting <img width="817" height="271" alt="Screenshot 2026-10-20 at 14-54-26 Shelfmark" src="https://github.com/user-attachments/assets/29941f83-9034-4ea5-bcb2-8e91ce77e63a" /> Separate Locations <img width="817" height="425" alt="Screenshot 2026-10-20 at 14-54-58 Shelfmark" src="https://github.com/user-attachments/assets/772b7194-a072-4945-bac5-b6959550fd75" /> ## Not included Routing to many folders by document type (the main request in #1048) still only exists for Direct download via Content-Type Routing. Extending it to torrent and usenet sources needs a decision on what to route by. ## Testing - `pytest tests --ignore=tests/e2e` passes (3561 passed); ruff clean. The `tests/e2e/platform` suite needs a running cluster and was not run. - New unit tests for the helper (including no fallback to the shared directory) and a qBittorrent add test covering book vs audiobook directories. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
splitsec2
|
61c7ad3c62 |
Merge commit from fork
* fix(torrent): check each redirect hop when fetching a torrent file The torrent file fetch followed redirects by hand without looking at where a hop pointed, so a download link could make the server request a loopback, link-local or LAN address. Each hop (the initial URL and every redirect Location) must now be http or https and resolve only to global addresses. The configured Prowlarr/Newznab origin is exempt, as it usually lives on the LAN. The API key still only goes to that origin. * fix(socketio): only accept websocket handshakes from the app's own origin The Socket.IO server allowed every origin. It now uses the same-origin check python-engineio applies by default (Host, or X-Forwarded-Proto and X-Forwarded-Host behind a proxy). SOCKETIO_ALLOWED_ORIGINS takes a comma separated list of extra origins for proxies that change Host without forwarding it. * fix(custom-script): do not pass a file name that starts with a dash as the first argument In relative path mode the script got the organized file's relative path as its first argument, which many scripts read as an option when the name starts with a dash. Such a name is now passed with a ./ prefix. Absolute mode is unchanged. |
||
|
splitsec2
|
7da4669cc1 |
feat(prowlarr): hide ebook results above a maximum size (#1504)
Closes #844. A new Prowlarr setting, Maximum Ebook Size (MB). Ebook results above it are hidden from the release list. The default is 0, which shows everything as it does today. This is the idea from the issue. The filetype check suggested in the comments is a different change (#1490 has one for Prowlarr), and it doesn't catch a TV series or a software bundle that ships as a plain archive, which is what a size limit does. - A result with no reported size is always shown. - A result exactly at the limit is shown. - Audiobook searches are never limited, since they are big by nature. - A non-numeric value counts as no limit. The filter runs on the raw Prowlarr result before it is turned into a release, so a hidden result never reaches the sort or the release cache. Tests cover no limit, zero, over the limit, exactly at it, no size, audiobook searches and a bad value. Ruff, basedpyright and the backend suite pass. I added the entry to `docs/environment-variables.md` by hand because running the generator also changes unrelated parts of that file. |
||
|
splitsec2
|
f2269aa201 |
feat(audiobookbay): seed time and ratio settings (#1503)
Closes #1037. Two new AudiobookBay settings, Seed Time (minutes) and Seed Ratio. Both default to 0, which leaves the torrent client's own limit as it is today, so nothing changes until someone sets them. They go to the client with the magnet the same way the Prowlarr seed preferences do (`DownloadRequest.seeding_time_limit` and `ratio_limit`), so they apply with the clients that already take per-torrent limits there: qBittorrent, Transmission and Deluge. A retry that reuses the saved magnet carries them too. A value that isn't a positive number (text, negative, NaN, inf) is ignored and the client's limit stays. Tests cover reading both settings (unset, zero, each on its own, text values, bad values) and that both the fresh and the cached-magnet request carry them. Ruff, basedpyright and the backend suite pass. I haven't tried it against a live client, since the client side is the code Prowlarr already uses. I added the two entries to `docs/environment-variables.md` by hand. Running `scripts/generate_env_docs.py` here also changes unrelated parts of that file (the Soulseek table of contents link and the `SHELFMARK_API_KEY_READONLY` entry), so I left those alone. |
||
|
splitsec2
|
e83bab9337 |
feat(clients): give up on a torrent with no seeders (#1494)
A torrent with no seeds won't finish. It runs on the pieces its few
peers hold and then sits still, and nothing says so until the stall
timer ends it. The card only reads "downloading". I run AudiobookBay
releases, which show no seed count in the listing, so the client is the
only place to find out.
Two new settings, both off until you turn them on:
`TORRENT_NO_SEEDERS_WAIT` (minutes, 0 turns it off) and
`TORRENT_MIN_SEEDERS` (default 1).
- A torrent that has fewer seeds than the minimum for the whole wait,
while nothing is downloading, is removed from the client with its data
and the task ends as an error.
- A torrent with no seed at all for the wait is given up on even while
it still moves.
- A torrent that met the minimum and lost seeds down to one, but is
still moving, keeps going.
- The card shows the seed count beside the progress, with a countdown
while it waits ("0 seeds, giving up in 4m").
qBittorrent only, because it reports connected and tracker seed counts.
Other clients report none and are left to the stall timer, and an
unknown count is not read as zero. A torrent that was already in the
client when the download started is never removed, and a queued torrent
isn't judged.
Tests cover the wait and countdown, the clock restarting on progress,
the tracker count, a higher minimum, off by default, a client with no
counts, a preexisting torrent, a queued torrent, and the moving cases.
Ruff and the backend suite pass.
|
||
|
splitsec2
|
a1ac017487 |
feat(requests): let an admin reopen a rejected request (#1491)
Closes #1445, where you said you were open to it. A rejected request was final. `update_request` won't move a terminal status and the admin routes only fulfil or reject, so an admin who changed their mind had to ask the user to request the book again, and the user kept seeing the old rejected one. `POST /api/admin/requests/<id>/reopen` moves a rejected request back to pending and clears the review fields. What the user asked for (request level, release, note) is kept. It refuses anything that isn't rejected, and refuses when that user already has a pending request for the same book. Admins get a Reopen action on rejected cards in the activity sidebar and in History. A rejected request has usually been cleared from the list by the time anyone wants it back, so reopening also clears the dismissal, the way the failed-download reopen does. Without that the pending request stays hidden. Tests cover the service (state change, what is kept, the refusals, a non-string admin note) and the route, and the card model for the new action. Ruff, tsc, oxlint, vitest and the backend suite pass. It has been running on my instance since Oct 8. |
||
|
splitsec2
|
06a67a7150 |
feat(download): a concurrency limit per source (#1495)
With one global limit, a few tasks that are only waiting hold every slot. A Soulseek download sitting in a peer's queue is the example I hit: three of those and a torrent that could start waits behind them. `MAX_CONCURRENT_DOWNLOADS_PER_SOURCE` (0 keeps the single global limit, which is the default). With it set, the coordinator starts a download only when its source is below the limit and passes over tasks of a full source without losing their place in the queue. `get_next` gets an optional filter for this, and a cancelled task is still dropped when one is given. The start-up stagger now applies only between downloads from the same source, and the pool is sized so every source can reach its limit. The second commit: qBittorrent keeps its own queue of torrents waiting for a free slot. The task polling one of those does no work, so it no longer counts against its source's limit and the next task of that source can start. The client does the pacing, as it does for its own queue. Tests cover the queue pass-over (order and priority kept, nothing eligible, cancelled tasks), the coordinator (a full source doesn't block the others, each source has its own limit, the global limit when unset) and the client-queue case. Ruff and the backend suite pass. |
||
|
CaliBrain
|
10710abd4e |
fix(download): send every file of a multi-file download to the browser (#1489)
Download to Browser only ever sent the first file of a multi-file release (audiobook chapters, packs): the folder output returned final_paths[0] and /api/localdownload served that one path. The folder output now keeps every final path on the in-memory task (download_paths), and /api/localdownload streams them as one uncompressed ZIP named after the book when there is more than one. The archive is never staged on disk or held in memory, and ZIP64 covers large audiobooks. History still stores only the first path, so a re-download from history after a dismiss or restart serves a single file, as before. Also brings every linter back to green on all files: - pre-commit: bump the ruff hook to v0.17.1 to match uv.lock; ruff-format the Python examples in docs/dev and skills - ruff: fix the remaining issues in skills/shelfmark/download_books.py; format base_handler.py - basedpyright: Optional and attrs errors in databazeknih.py - knip: drop 31 unused exports, reuse ColumnColorHint in colorMaps - oxfmt 0.72: styles.css - direct download: fix the "Welib, mirrors" typo and the test that still expected the pre-OceanofPDF message Fixes #1467 |
||
 
|
c8d66db22e |
feat(direct_download): add OceanofPDF as a standalone search and download provider (#1485)
Closes #1149 ## What this adds OceanofPDF as a second Direct Download provider, alongside Anna's Archive. Unlike Z-Library, LibGen and Welib, which only download files Anna's Archive found, OceanofPDF searches and downloads on its own. It works with or without an AA mirror configured; with both, one search queries both and the results are merged. It plugs into the provider registry from #1337, so search, record routing and the download handler need no changes. **Search:** WordPress `/?s=` search, "title author" first, then the bare title if that finds nothing. One release per advertised format (`[PDF] [EPUB]`), with title, author and cover. Shelfmark's supported-format filter applies as usual. **Download** follows the site's three hops: 1. The book page has one POST form per format, targeting `Fetching_Resource.php`. 2. That returns an interstitial whose `<meta http-equiv="refresh">` points at a signed, expiring file URL. 3. The file URL serves the book. Pages go through `html_get_page` and its Cloudflare bypass. The form POST replays the clearance the bypasser stored, and re-solves once if challenged. Bypassers only replay GETs, so it can't be handed over directly. **Settings:** - `OCEANOFPDF_MIRROR_URLS` under Mirrors. Only the first URL is used. - An OceanofPDF toggle under Download Sources → Slow downloads. It's locked until a mirror is set and while the Cloudflare bypass is off. Its position in the list doesn't matter, and its description says so. - A saved list from before this change has no OceanofPDF entry. The settings page shows a missing option as off, so the provider treats it as off too, keeping the page and the behaviour in agreement. New installs default to on. ## Fixes found along the way - **`challenge_marker` rejected real pages behind Cloudflare** (separate commit, `fix(bypass)`). Cloudflare's bot-detection script injects `/cdn-cgi/challenge-platform/scripts/jsd/main.js` into every page it serves, after load. Bypassers return the rendered page, so a successful FlareSolverr solve of any page under 64 KB came back carrying `/cdn-cgi/challenge-platform` and was rejected as unsolved. The detector now ignores that one script path. Real challenge pages load from `/cdn-cgi/challenge-platform/h/…` and keep their other markers, so they are still caught. This also affects any other Cloudflare-fronted host, not just OceanofPDF. Happy to split it into its own PR if you prefer. - **Brotli on a plain `requests` call.** Copying `DOWNLOAD_HEADERS` into the form POST offered `br`. Cloudflare then answered in Brotli, which `requests` can't decode without the optional `brotli` package. The POST now leaves `Accept-Encoding` to `requests`. If a response still has no file link, its status, encoding, title and challenge marker are logged. ## Testing - **New tests:** 41 in `tests/direct_download/test_oceanofpdf.py` and 4 in `tests/bypass/test_cloudflare_beacon.py`. They cover search parsing, format and language filtering, form selection, the interstitial, the download flow (including a challenged POST, an HTML stub served as the file, and cancellation), the toggle, and registry wiring. Fixtures mirror the live site's markup. - **Existing tests:** `tests/direct_download`, `tests/bypass`, `tests/config`, `tests/core` and `tests/download` show no new failures against `main`. Some tests in those folders can't run on Windows; those fail identically on both. - **Lint:** `ruff check` and `ruff format --check` are clean on every changed file. - **Live:** verified end to end on a v1.4.0 Docker deployment behind FlareSolverr (external bypasser), with only an OceanofPDF mirror configured: search, then the book page, the form POST and the completed download. ## Known limits - OceanofPDF doesn't label languages, so its results always pass a language filter. - PDFs only appear when `pdf` is in the supported formats. - Each query reads the first results page only (roughly 7–10 books). - Downloaded files carry OceanofPDF's own tags/watermark, as noted in #1149. They aren't stripped. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Adam Stone <196691089+ast0ne1@users.noreply.github.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
  
|
bb1655238a |
feat(slskd): add Soulseek (slskd) release source, building on #1335 (#1428)
## Add Soulseek (slskd) as a release source, building on #1335 This is @dskvr's work from #1335 rebased onto current `main`, with my changes on top. #1335 has been quiet since 2026-09-13 and I needed the changes below, so I carried it forward instead of starting over. Their two commits are the first two in this branch with their authorship intact, and the source plugin, handler, API client, settings tab, tests, `ReleaseProtocol.SOULSEEK`, test client compose file and frontend bits are all theirs. I haven't changed how a search turns into releases beyond what is listed here. If @dskvr would rather keep #1335 alive and take these as commits on their branch, I can do that instead and close this. ### What I changed **Downloads survive other apps using the same slskd.** The handler found its files by listing slskd's downloads, and another app on the same instance (SoulSync here) sends `DELETE /transfers/downloads/all/completed` after each of its batches. Once that ran, a finished download no longer appeared in the list and the task never saw it complete, even though the files were on disk. With the new `SLSKD_ISOLATE_DOWNLOADS` setting (on by default) each task: - enqueues one batch (`POST /transfers/downloads/batches`) into its own folder, `<SLSKD_DESTINATION_PREFIX>/<task id>` - polls each transfer by id, which slskd still answers after the list has been cleared - decides it is done from the files on disk (sizes, not names), with the `/events` log as a fallback when the transfers are gone and the files aren't visible yet - copies out of that folder, and removes it only if `SLSKD_KEEP_COMPLETED` is off and the import worked With isolation off the handler behaves as it did in #1335. If the batch endpoint isn't there (404/405) it falls back to the original per-file enqueue. **Source changes.** m4b/m4a files are their own release next to loose chapter files, so auto-download can prefer one file over a chapter set. Results from the query variants are merged per peer. The folder text is exposed as `extra["author"]` and `extra["title_raw"]`, which is what auto-download's strict match and pack detection read. `is_available()` also needs an API key now. **Smaller fixes.** Cancelling while files are still being located now removes the transfer records it created. In the settings tab, a cleared form field is no longer replaced by the saved value when testing the connection. The new settings are in `docs/environment-variables.md`. ### Testing `tests/slskd` has 192 passing and 6 live tests that skip without an instance. I wrote the new tests first and checked the handler and source rules by breaking them and confirming a test failed. Ruff is clean. On the whole suite the only failures are 4 in `tests/download` that import `seleniumbase`, which isn't installed in my environment, and they fail the same on `main`. I've been running this on my own deployment against slskd 0.26.0 behind a VPN, with SoulSync active on a second instance. A search returned a couple hundred results from about 70 peers, an epub came down and was imported, and auto-download fetched five audiobooks as single m4b files. That is a day of use, so I don't know how it behaves on other slskd versions or on Windows path mappings. --------- Co-authored-by: sandwich <dskvr@users.noreply.github.com> Co-authored-by: sandwich <299465+dskvr@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Amalkrishna A S
|
b38e4e4a00 |
feat(naming): add {AuthorSort} template variable (#1484)
Closes #1178. Adds `{AuthorSort}`, the first author surname-first ("Arthur Conan Doyle" → "Doyle, Arthur Conan"), for rename and organize templates, alongside `{FirstAuthor}`. - A value already in "Last, First" form (one word before the comma) is kept as is; Jr/Sr/II/III/IV stay at the end. - Known limits, same as calibre's default: the last word is treated as the surname, so particles ("Le Guin") and surname-first names aren't detected. - Added to the settings help, env docs, the frontend variable picker/preview, and tests. |
||
|
misko903andClaude Opus 5.5
|
b83e35ee0a |
feat(metadata): add Databáze knih metadata provider (#1479)
## What Adds [Databáze knih](https://www.databazeknih.cz), the Czech and Slovak community book catalog, as a metadata provider. It follows the Moly.hu provider (#1172): HTML scraping, no API key, disabled by default, and a settings tab with an enable checkbox and a Test Connection button. ## Why Hardcover, Open Library and Google Books cover Czech and Slovak editions poorly. Databáze knih is the de facto catalog for them, including Czech and Slovak translations of foreign works. With this provider, Universal mode works end to end for Czech and Slovak titles: the localized title and author from Databáze knih feed the release search. ## How - HTML scraping with BeautifulSoup, which is already a dependency. Book pages carry schema.org JSON-LD, so most fields come from structured data rather than CSS selectors. - **Title search** uses `/vyhledavani/knihy`, with the site's pagination (`page`) and language filter (`lang` for cs/sk/en/de). A search with a single match redirects straight to the book page, and that case is handled. - **Author search:** the site search only matches titles, so an author-only search finds the author (accent- and word-order-insensitive) and lists their books page, which is also paginated. Title + author filters the title results on the author. - **ISBN search** follows the site's redirect from `/search?q=<isbn>` to the book page. - **`get_book`** reads the full description (the JSON-LD one is truncated), ISBN-10/13, publisher, edition year, language, genres, cover (placeholder images dropped), series with id and position, and rating and rating count. It also reads the page count and original title from the more-info fragment. - The `RateLimiter` from Open Library (30 requests/min) and the standard `@cacheable` decorators. Fetch failures return `None`, so they are not cached. - `docs/environment-variables.md`: added the `DATABAZEKNIH_ENABLED` section. ## Testing - `tests/metadata/test_databazeknih_parse.py`: offline tests with shortened copies of real pages. They cover search parsing, pagination and language parameters, redirect-to-book, author listing, title + author filtering, uncached failures, book details, ISBN resolution and helpers. - `uv run pytest tests/metadata` is green (89 passed), and `ruff check` and `ruff format --check` are clean. - Verified live against databazeknih.cz: connection test, title search (pages 1 and 2), author books, title + author, `get_book`, and ISBN lookup. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
 FarawayMagnet
|
ad62f41edc |
improve hints/docs regarding http proxy user/pass auth (#1462)
in the web UI (Settings > Network > Proxy) it's unclear that shelfmark's proxy option supports proxies that require user/pass auth. this PR adds a note to the web UI's hint to explain this. it updates the associated documentation as well. also removed the `e.g.` note, since the form field already includes an example and this was redundant. a side note: i think the verbiage "HTTPS proxy URL (leave empty to use HTTP proxy for HTTPS)" is a little confusing. i believe it's referring to the fact that a user can specify one proxy address for HTTP requests that shelfmark would make, and another proxy address for HTTPS requests that shelfmark would make. i'll leave this for the maintainer to decide whether to clarify. |
||
|
splitsec2
|
47707b827a |
feat(download): make the torrent stall window a setting (#1426)
#1420 gave a torrent 15 minutes to get going and 15 minutes after each step forward. That suits my swarms, but other people will want it shorter or longer, so this makes the number a setting. `TORRENT_STALL_TIMEOUT_MINUTES` sits in Download Clients with the other torrent options and takes 5 to 60. The default is 15, so nothing changes unless someone sets it. 5 is the orchestrator's own stall timeout, so choosing it asks for no extra time and gives the old behaviour. The value can arrive as text from an env var, so anything that isn't a number falls back to the default and anything out of range is clamped. The orchestrator caps one activity grace at 16 minutes, so I raised that to an hour to fit the top of the range. As far as I can tell the protection bypass is the only caller that asks for more than a few minutes, but I haven't audited every caller. If you'd rather keep the cap, I can lower the maximum to 16 minutes instead. Tests cover the default, 5 and 30, text values, out of range values and NaN, the start and movement windows following the setting, and the setting showing up in the Download Clients tab. I broke each rule on purpose to check a test catches it. The full suite fails the same tests with and without this change (the bypass and seleniumbase tests don't run in my environment). |
||
|
splitsec2
|
cad18b8019 |
feat(clients): add Debrid-Link as a torrent client (#1380)
calibrain said in #1315 that he'd review and merge a Debrid-Link client if someone took a stab at it, so here it is. It's the fourth debrid service alongside AllDebrid, Real-Debrid and TorBox and it uses the same DownloadClient interface as the other three. No frontend changes, the settings UI picks it up from the backend field definitions. **I can't test this against the live API.** I don't use debrid and I don't have a Debrid-Link key. Everything here is built from their v2 API docs and cross-checked against two independent reference implementations, so the request shapes should be right, but nobody has run a real magnet through it yet. @Tatsu941 you opened the issue and mentioned they give developers free premium. Would you be able to try it? The things most worth confirming are the field names on a seedbox torrent, that `premiumLeft` is the right premium check, and that an API error comes back the way the client expects. Happy to fix whatever turns up. Two things about their API shaped how this is written. Every v2 endpoint wraps its payload in a `{success, value}` envelope and a failure can arrive with an HTTP 200, so `_request_value` decides the outcome from the body instead of the status code. A completed seedbox torrent already carries a direct `downloadUrl` on every file. The other three clients each need a separate unrestrict or link-request call per file and this one doesn't, which is why it came out shorter than them despite doing the same job. Endpoints used: | Endpoint | Used for | |---|---| | `GET account/infos` | test the key, confirm premium time remains | | `POST seedbox/add` | magnet as JSON, .torrent as multipart | | `GET seedbox/list` | poll progress | | `DELETE seedbox/{id}/remove` | clean up | Ruff and basedpyright are clean and the 36 new tests pass. The two failures in `tests/e2e/test_proxy_auth_middleware.py` are already on main and fail the same way on a pristine checkout. They assume the first proxy-authenticated user becomes an admin, which only holds while the user table has no admin in it, so they depend on how many users earlier tests created. One thing for review: the download loop uses `download_url` like the other three clients, which holds each file in memory. I kept the existing pattern rather than do something different in a new-provider PR, but say so if you'd rather it streamed here. Closes #1315 |
||
|
splitsec2
|
755f28a6a4 |
feat(api): add a read-only API key and a /api/stats endpoint (#1419)
This is the read-only API key from #1410, where you said to go ahead. I run Shelfmark behind Homepage and wanted more on the dashboard tile than up or down, without putting an admin API key in the dashboard's config. This adds a second key, `SHELFMARK_API_KEY_READONLY`, that can read one new endpoint. `GET /api/stats` returns counts only: books added over the last 7 and 30 days by format, the queue, requests by outcome, and download failures over the last 7 days. No titles and no user names. The read-only key, the admin key or an admin session can read it. The read-only key gets a 403 on every other path and on any write, including a POST to `/api/stats` itself, and it never gets a session or a cookie. If a request carries both keys, the admin one wins, so a proxy's own `Authorization` header can't downgrade a correct `X-Api-Key`. Unset means off, like the existing key. The endpoint exists either way, but without a key only an admin session can read it. It also adds a `db_path` property on `UserDB`, so the stats module can open the database read-only. Tests cover the key scope, the 403s for the read-only key, both keys at once, who can reach the endpoint, and the counters. The full suite passes, and I broke each rule on purpose to check a test catches it. A version of this has been running on my own install behind Homepage. If you'd rather have the stats in a different shape or under a different path, I'm happy to change it. |
||
 adman234andClaude Opus 5.5
|
32abaff21e |
feat(naming): {Narrator} placeholder and MyAnonamouse series fallback (#1407)
## Human-Written explanation:
Following up on previous PR to use MAM ID to get nararrator and series
to show up in search, this PR will allow the series and nararrator
fields to be added to the path when saving an audiobook.
I have tested my ghcr.io image on my instance and it seemed to work
properly.
All the text below is written by Claude.
# feat(naming): {Narrator} placeholder and MyAnonamouse series fallback
Follow-up to #1390 and #1399. Related to #605 and #934 (narrator
support).
## Problem
Several narrations of the same audiobook currently land in the same
folder, because nothing in the path template tells them apart. Keeping
more than one version means renaming folders by hand after every
download.
Since #1390, MyAnonamouse results carry the release's narrator and
series in `extra`, but the naming templates can't use them: there is no
`{Narrator}` placeholder, and `{Series}` / `{SeriesPosition}` only come
from the metadata provider.
## Changes
- **`{Narrator}` placeholder** (`core/naming.py`,
`download/postprocess/transfer.py`, `core/models.py`): `DownloadTask`
gains `narrator`, read from `narrator` or `extra.narrator` when a
release is queued, and kept in the restart-safe retry payload. It's
available in audiobook templates like any other placeholder, including
prefix/suffix blocks such as `{ - Narrator}`.
- **Audiobookshelf folder style**: Audiobookshelf reads the narrator
from folder names like `Title {Narrator}`. The template `{Title}
{{Narrator}}` already matches as `{` + `{Narrator}`; the parser now also
drops the closing brace when the narrator is empty, so the folder is
`Title` rather than `Title }`.
- **No stray spaces around `/`**: an empty placeholder at the start or
end of a folder name no longer leaves a space there (e.g. `Title
/Title`). This applies to all templates.
- **Series fallback** (`release_sources/prowlarr/mam.py`, `source.py`,
`download/orchestrator.py`): MAM enrichment also stores the first
series' name and number as `extra.series_name` /
`extra.series_position`, which queueing already falls back to when the
metadata provider has no series. The provider's series still wins. A
release-level series number is only used when it names the same series
as the provider, so a provider series is never paired with the number of
a different MAM series (an omnibus, for example).
- **Settings and preview** (`config/settings.py`,
`namingTemplatePreview.ts`): the audiobook template descriptions list
`{Narrator}` and explain `{{Narrator}}`. The settings page preview
mirrors the parser changes and offers `{Narrator}` under "Insert
variable" for audiobook templates only.
- **Docs**: regenerated `environment-variables.md`. This also picked up
the Library Check settings, which weren't in the generated docs yet;
happy to drop that part to keep the diff focused.
## Example
Audiobook Path Template `{Author}/{Title} {{Narrator}}/{Title}`:
| Narrator | Result |
|----------|--------|
| Samuel Roukin | `Christopher Ruocchio/Empire of Silence {Samuel
Roukin}/Empire of Silence.m4b` |
| none | `Christopher Ruocchio/Empire of Silence/Empire of Silence.m4b`
|
## Testing
- `tests/core/test_narrator_template_variable.py` (new, 16 tests):
`{{Narrator}}` with and without a value, the `{ {Narrator}}` prefix
form, prefix/suffix blocks, sanitizing, `build_library_path`, task
metadata, retry payload round trip, MAM series name/number parsing
(including a `1-3` range having no number), and queueing (narrator from
`extra`, series fallback, no cross-series number, blank narrator).
- `namingTemplatePreview.test.ts`: the same `{{Narrator}}` cases for the
preview.
- Updated `test_generate_env_docs.py` for the new template description.
- Full `pytest -m "not integration and not e2e"` compared with an
upstream `main` worktree on the same machine: no new failures (the
remaining ones are Windows-only on both).
- `ruff check`, `ruff format --check`, `basedpyright` (0 errors) and
`vulture` on touched files; frontend `tsc --noEmit`, `oxlint`, `oxfmt
--check`, `vitest` (218 passed).
- Checked the settings page locally: the audiobook preview renders `...
{Simon Vance}.mp3` and `{Narrator}` is listed for audiobook templates
only.
No behavior change for existing templates, apart from spaces next to `/`
being trimmed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
  Evan KazakinandEvan Kazakin
|
7888217380 |
feat(download): add Remove & Delete Files torrent completion action (#1401)
Closes #1027 ## Summary - add a **Remove & Delete Files** (`remove_and_delete`) option to the Torrent Completion Action setting - after a successful import, remove the torrent and ask the client to delete its downloaded data (`delete_files=True`), matching what the usenet "move" flow already does - clarify the setting description so it says **Remove** keeps the downloaded files (raised in the issue comments) - regenerate the `PROWLARR_TORRENT_ACTION` entry in `docs/environment-variables.md` ## Behavior The deletion runs from `post_process_cleanup`, the same place as the existing Remove and Change Category actions, so it only happens after output transfer and post-processing have succeeded. At that point every file has already been copied or hardlinked into the library, and transfer size checks have passed. A failed import never removes or deletes anything. The torrent client deletes its own data, so Shelfmark does not delete paths itself and remote path mappings are not involved. Keep, Remove, and Change Category behave as before, and Keep is still the default. If a request matched a completed torrent that was already in the client, this option deletes that torrent's data too. That is the same scope the existing Remove action already applies to. ## Validation - `make python-checks` (ruff check, ruff format, basedpyright on backend and tests, vulture): passed - `make python-test`: 3363 passed, 5 skipped, 9 failed. All 9 are in `tests/config/test_entrypoint_permissions.py` and happen because macOS `/bin/bash` 3.2 does not support `${1,,}` in `entrypoint.sh`. They do not touch this change. - new tests in `tests/prowlarr/test_handler.py`: - Remove passes `delete_files=False` and Remove & Delete Files passes `delete_files=True` - a failed import with Remove & Delete Files does not call the client - the delete case fails without the handler change - manual smoke run: `PROWLARR_TORRENT_ACTION=remove_and_delete` set through real config loading, with a stub client that deletes its folder. The torrent folder was deleted and the hardlinked library file stayed intact. - pre-commit hooks (prek) passed --------- Co-authored-by: Evan Kazakin <evan@Evans-MacBook-Pro.local> |
||
|
CaliBrain
|
893f7bdb92 |
fix(mam): keep the session ID on MAM and rerun Prowlarr's exact search (#1399)
Follow-up to #1390. - The origin came from each result's infoUrl, matched by a regex that did not check the host, and the mam_id cookie had no domain. Any Prowlarr indexer returning a URL like https://evil.example?myanonamouse.net/t/1 sent the session ID to evil.example. Requests now always go to https://www.myanonamouse.net (Prowlarr's only MAM URL), with the cookie as a header and redirects off. Only results from Prowlarr's MyAnonamouse indexer are looked up, and their URLs must be on myanonamouse.net. - The lookup searched every category and read one page, so for common titles most of Prowlarr's results were missed (a "Dune" audiobook search: 56 audiobooks on the first 100 of 328 matches). It now reruns Prowlarr's exact search: the same query clean-up, the MAM main categories behind the Torznab categories searched (13/15/16 for audiobooks, 14 for e-books, all once expanded), and the MAM indexer's own search type, search-in options and languages. Further pages are read while IDs are missing, page 1 of every title first, at most 4 requests per search. - Failed requests back off for 1, 2, 4 ... up to 30 minutes. The 10th consecutive failure stops enrichment until Test MAM Session passes, the session ID changes, or Shelfmark restarts. - The detail cache prunes expired entries instead of growing for as long as Shelfmark runs. |
||
|
|
a05e002305 |
feat: narrator, series and bitrate columns for MyAnonamouse results (#1390)
# feat: narrator, series and bitrate columns for MyAnonamouse results Addresses #605 and #934 (narrator in the release list). ## Problem Both requests were closed because the narrator isn't in Torznab results, which is correct. Prowlarr's MyAnonamouse indexer reads `author_info` but drops the `narrator_info` and `series_info` MAM returns next to it, and neither Prowlarr's `ReleaseInfo` nor the Torznab output has a field for them. Shelfmark's size tooltip already looks for a `narrator` Torznab attribute, but nothing ever sends one. When a book has several narrations, choosing one means going back and forth between Shelfmark and the tracker. ## Approach Optional, opt-in enrichment using the user's own MAM session (`mam_id`), the same approach AudioBookRequest's MAM indexer uses: 1. After the Prowlarr search, releases whose info URL is `…myanonamouse.net/t/<id>` are collected. 2. Shelfmark sends the same query text to MAM's JSON search (`/tor/js/loadSearchJSONbasic.php`, normally one request, at most 3), and matches torrents back to Prowlarr's results by torrent ID. The MAM origin is taken from the result URL, so a custom Prowlarr MAM base URL is respected, and the cookie is only ever sent to `*.myanonamouse.net`. 3. Adds `extra.narrator`, `extra.series` (`The Sun Eater #1`) and `extra.bitrate` / `extra.bitrate_value`. MAM has no bitrate field, so it's parsed from the uploader's free-text tags (`64 kbps`) and some releases won't have one. Lookups are cached per torrent for an hour, respect the existing Prowlarr search deadline, go through the configured proxy (`get_proxies`), and never fail the search: a 403, timeout or bad JSON is logged and the list renders without the extra details. ## Changes - **`release_sources/prowlarr/mam.py`** (new): small MAM client (`search`, and `get_username` for the test button), `narrator_info` / `series_info` / tags parsing, cached best-effort `lookup_torrent_details()`. A 403 includes MAM's reply and a note about the IP/ASN lock. - **`release_sources/prowlarr/source.py`**: enrichment after the result loop. Series, Narrator and Bitrate columns only when a MAM ID is configured, since otherwise they would be empty for every row. A Torznab `bitrate` attribute from other indexers is also mapped to `extra.bitrate`. - **`release_sources/prowlarr/settings.py`**: "MyAnonamouse Enrichment" section with `PROWLARR_MAM_ID` (password field, env-overridable like every setting) and a **Test MAM Session** button. - **`release_sources/__init__.py`**: `ColumnSchema` gains optional `setting_key` and `content_types`. The new `apply_column_visibility()` drops gated columns and their grid tracks. Neither field is serialized. - **`main.py`**: `/api/releases` applies `apply_column_visibility()` with the request's content type and the user's effective settings. - **`config/settings.py` / `users_settings.py`**: Search Mode › "Release List Columns" with `SHOW_SERIES_COLUMN`, `SHOW_NARRATOR_COLUMN` and `SHOW_BITRATE_COLUMN`, all default on and user-overridable. Narrator and bitrate are audiobook-only, series shows for both. AudiobookBay's existing bitrate column now follows the bitrate toggle. - **Frontend**: text cells truncate with a hover title; the mobile info line wraps and skips empty text/number cells so blank optional columns don't leave orphan `·` separators; the size tooltip no longer lists Bitrate twice. - **Docs**: new `docs/myanonamouse-enrichment.md` (linked from the index), and a regenerated `environment-variables.md`. The regeneration also picked up a few pre-existing drifts from `main` (the Libgen section, `AA_DEFAULT_SORT` default, a duplicate `BOOK_LANGUAGE` row). I can drop those if you'd rather keep this diff focused. ## ⚠️ MAM sessions are IP/ASN-locked MyAnonamouse locks each session to one IP or ASN. Reusing the session Prowlarr (or a seedbox script) uses will often **403**. **A separate MAM session for Shelfmark will likely be needed** when Shelfmark reaches MAM from a different IP (another host, a VPN container, or a proxy in Shelfmark's Network settings), or when the existing session is ASN-locked to another network. The setting's description, the error message and the new doc all say so. ## Testing - `tests/prowlarr/test_mam_enrichment.py` (new, 19 tests): parsing (narrator dedupe, multiple series, missing numbers, malformed JSON, tag bitrate), lookup (stops once all IDs are found, cache, 403 and connection errors return empty, expired deadline skips the request), only MAM releases enriched, Torznab bitrate mapping, column config with and without a MAM ID for audiobook and ebook, toggles, grid-track removal, and gates not serialized. - `tests/core/test_admin_users_api.py`: the curated search-preference key list now includes the three toggles. - Full `pytest -m "not integration and not e2e"` compared with an upstream `main` worktree on the same machine: no new failures. The remaining ~115 failures on both are Windows-only (tor/entrypoint shell tests, path separators). - `ruff check` / `ruff format --check` / `basedpyright` (0 errors) / `vulture` on touched files; frontend `tsc --noEmit`, `oxlint`, `oxfmt --check`, `vitest` (201 passed). - Manually verified with a real MAM account on a Docker build of this branch: the test button, then narrator, series and bitrate on MyAnonamouse audiobook results. No behavior change unless `PROWLARR_MAM_ID` is set, apart from the bitrate toggle on AudiobookBay (default on, same as today). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
CaliBrain
|
ece6b8f341 |
fix(auth): fail closed when auth prerequisites are missing (#1387) (#1397)
Auth mode resolution fell back to "none" (anonymous full admin) whenever the configured method's prerequisites were missing: no local password admin for builtin/OIDC, no Calibre-Web database, a blank proxy header, an unrecognized AUTH_METHOD (including "OIDC" in uppercase), or any error while reading the config. Deleting or demoting the last local admin was allowed on purpose because of that fallback, which exposed OIDC instances publicly. - Only an explicit AUTH_METHOD=none disables authentication. A configured method stays active when its prerequisites are missing, so sign-in fails instead of opening up. - An unrecognized or unreadable AUTH_METHOD resolves to "unavailable", which still requires a session and accepts no login. Values are normalized, so AUTH_METHOD=OIDC works. - Restore the guard against deleting or demoting the last local password admin while builtin/OIDC is active (unless DISABLE_LOCAL_AUTH is set). - Require a local admin before enabling Local auth, as OIDC already did. - Log a recovery hint at startup when builtin/OIDC runs without a local admin, and document recovery via AUTH_METHOD=none. - Drop the "will fall back to No Authentication" UI toasts and hints. Fixes https://github.com/calibrain/shelfmark/issues/1387 |
||
|
splitsec2
|
37a77e9562 |
feat(library): mark search results already in a Calibre library (#1377)
Per discussion #1372, where you said you were fine with this specific implementation: check whether metadata.db exists, read it if so, and show a check mark saying the book is already there. Searching for a book you already own gives no hint that you own it, so the easiest way to end up with a second copy is to not remember you have the first. This reads a Calibre `metadata.db`, read only, and marks matching results with an **In library** badge in the card, list and compact views and in the details dialog. Off by default. It sits in Settings, General beside the existing Library URL, with a test button that reports how many books it indexed. No HTTP call, no token, nothing written back. Matching runs most to least confident: a shared external id, then an ISBN compared in both ISBN-10 and ISBN-13 form, then fuzzy title tokens plus the author surname. The check fails open, so an unreadable database degrades the badge and never blocks a search, and entries are cached for ten minutes with an early refresh when the file changes, so a large library costs one read rather than one per search. `text_match.py` is new and shared by the index and the provider, so title, author and ISBN matching stays consistent in one place. ## On the provider interface `library_index` talks only to a `LibraryProvider` protocol and knows nothing about Calibre. That is deliberate but it is not speculative generality, it is what let me send you the Calibre half on its own: I run an Audiobookshelf provider on the same interface in my fork, which is where the audiobook side of the badge comes from. I have left that out because it is a new service integration rather than something already in the codebase, which is the line your non-goals draw. Happy to send it separately if you ever want it, and equally happy for the answer to be no. Adding a library is a module with the `LibraryProvider` shape plus one line in `all_providers()`. ## Verification - `tests/core/test_library_index.py`: id, ISBN and fuzzy matching, per-content-type provider selection, fail-open on provider errors, stale-cache reuse, TTL and fingerprint refresh, per-provider cache isolation, and the test-connection path including unsaved form values. - `tests/core/test_text_match.py`: ISBN variants and token matching. - `src/frontend/src/tests/libraryBadge.test.ts` and the added cases in `bookTransformers.test.ts`. - Python suite (3269) and frontend suite (206) green, plus ruff, ruff format, basedpyright, vulture, tsc, oxlint, oxfmt and the production build. |
||
|
splitsec2
|
fe99d4bb5b |
feat(search): add a configurable default content type (#1371)
Closes #1018. Worth correcting the issue first: the search tab is not hardcoded. `useContentTypePreferences` has persisted the user's choice to localStorage since #564, so a browser that has picked a tab already keeps it. What is missing is the other half the issue asks for, a default for a browser that has stored nothing, and a per-user override. `DEFAULT_CONTENT_TYPE` is a user overridable select next to `BOOK_LANGUAGE`, so it follows the same path: global value in Settings, per-user value in Search Preferences, resolved with `user_id` in `/api/config`. The frontend uses it only when this browser has no stored choice, which is captured before the existing effect writes one, so nothing changes for anyone who has already picked a tab. The resolution is a pure function in `utils/contentTypePreference.ts` rather than logic inside the hook, since vitest here has no jsdom and the existing tests cover resolvers like `resolveDefaultLanguageCodes` the same way. One small move in `App.tsx`: the `config` state was declared below the hook that now reads it, so it moved above it. ## Verification - `tests/core/test_config_api.py`: the payload carries `default_content_type` and reads it with the user's id. - `tests/core/test_admin_users_api.py`: the key appears in the per-user search preferences list. - `src/frontend/src/tests/contentTypePreference.test.ts`: a stored tab wins, combined mode survives, the server default applies when nothing is stored, and an unrecognised value falls back to ebook. - Python suite (3163) and frontend suite (201) green, plus ruff, ruff format, basedpyright, vulture, tsc, oxlint, oxfmt and the production build. |
||
|
CaliBrain
|
d978896142 | fix(auth): rename the API_KEY env var to SHELFMARK_API_KEY (#1374) | ||
 Gavin McFallandClaude Fable 5.1
|
3b280009ae |
feat(auth): static API_KEY (env) accepted as Bearer or X-Api-Key, cookie or key (#1366)
Supersedes #1353, per the discussion in #1352: one `API_KEY` environment variable; when set, a request carrying it is authenticated as the first admin, and cookie sessions keep working exactly as before (cookie **or** key). Nothing else changes. No table, no UI, no settings-tab switch, no per-user keys. ## What - `API_KEY` (env). Unset → the feature is off and none of the new code runs. - `Authorization: Bearer <key>` or `X-Api-Key: <key>` on any existing `/api/*` route authenticates that request as the first admin in `users.db` (`ORDER BY id`), or as a bare admin identity (`user_id="api"`, `is_admin=True`, no local user row) if the install has no admin yet. Per request only; nothing is persisted; the admin's role is read live, so deleting or demoting that user takes effect on the next request. - Both headers are checked and either may match. That is what makes the key usable behind a reverse proxy that injects its own `Authorization` header (oauth2-proxy, Authelia, forwardAuth): send the key in `X-Api-Key`. - A credential that is **not** the key is ignored and the request continues on the normal session path, so proxy-forwarded tokens are unaffected. Without a valid session such a request gets the usual `401 {"error": "Unauthorized"}`, identical to a request with no credential, so there is nothing to probe. ## How - `shelfmark/config/env.py`: `API_KEY = os.getenv("API_KEY", "").strip()`. - `shelfmark/core/api_key.py`: `extract_api_key_candidates()` (Bearer token if the scheme is Bearer, then `X-Api-Key`) and `matches_api_key()` using `hmac.compare_digest` on bytes. - `shelfmark/core/user_db.py`: `UserDB.get_first_admin()`. - `shelfmark/main.py`: `api_key_auth_middleware` (`before_request`, registered before `proxy_auth_middleware`, which early-returns for keyed requests). Only `/api/` paths; `/api/health` and `/api/auth/*` exempt; no-op when `API_KEY` is unset or the auth mode is `none`. On a match it mirrors the proxy-auth pattern: `session.clear()` then populate `user_id` / `is_admin` / `db_user_id` for this request, `permanent = False`, `modified = False`, `g.api_key_auth = True`. An `after_request` hook guarantees no `Set-Cookie` is written for a keyed request even if a handler dirties the session. - `docs/api-access.md` (new), the `API_KEY` entry in `docs/environment-variables.md`, and a README link. ## Security - Constant-time compare; the key is never logged or echoed. - Keyed requests never mint or refresh a session cookie and ignore any cookie sent with them (a non-admin cookie plus the key yields admin for that request; the browser's own session is left untouched and usable). - The mismatch path touches neither the session nor `g`, so a stray bearer on a browser request can neither log the user out nor change how their cookie is refreshed. - Store errors during the admin lookup fail closed (`500 {"error": "Authentication error"}`), never to anonymous. - Verified against Flask's `save_session` / `should_set_cookie` ordering, and under auth modes `none`, `builtin`, `proxy`. ## Tests `tests/core/test_api_key_env.py` (36): extraction and matching; first-admin lookup; middleware behaviour on a guarded route and an admin route, with and without a user_db, `X-Api-Key`, both-headers combinations, no `Set-Cookie` when a handler dirties the session, incoming non-admin cookie ignored, browser cookie still usable after a keyed request, security headers, store error → 500, mismatch → guard's 401 / cookie path / permanent cookie untouched, unset → off, exempt paths and path probes, `none` and `proxy` modes, deleted and demoted first admin, a keyed write passing the guard. Existing auth suites unchanged. All CI gates green on the fork: https://github.com/gavinmcfall/shelfmark/pull/2 (CI-only draft). Also exercised against a running instance: 47 scripted checks including 150 concurrent requests, proxy-mode switching through the key, an unset-key restart, and a log scan for the key. ## Naming `API_KEY` as discussed. If you'd rather namespace it (`SHELFMARK_API_KEY`) to avoid clashing with other tools' env vars in shared compose files, it is a one-line change; say the word. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
splitsec2
|
acd59f7cbb |
feat(auth): provision proxy users as non-admin once an admin exists (#1356)
With `AUTH_METHOD=proxy` and no admin group configured, every user the proxy authenticates for the first time is provisioned as an admin (`is_admin = True` unless the user already exists in `users.db`). The intent to never lock an instance out makes sense, but the effect is that anyone the SSO gate lets through becomes an administrator. On an instance shared with family or a small community that is a footgun; I hit it when the first invited reader landed as an admin. This keeps the guarantee and removes the footgun: the first account is still provisioned as an admin while the instance has no admin at all, and later first-time users follow a new `PROXY_AUTH_DEFAULT_ROLE` setting (Security tab / env), default `user`. Known users keep their stored role; the `PROXY_AUTH_ADMIN_GROUP_NAME` path is unchanged and still takes precedence. I couldn't find a way with Cloudflare access to pass this along. Changes: `UserDB.has_admin()`, `_proxy_default_is_admin()` in the proxy middleware, the new `SelectField` beside the other proxy settings, the regenerated `docs/environment-variables.md` entry and a row in `docs/reverse-proxy.md`. Compatibility: the default moves from "everyone admin" to "first admin, then users". Accounts already in `users.db` are unaffected; new SSO users on an existing instance become regular users unless `PROXY_AUTH_DEFAULT_ROLE=admin` is set. If you would rather ship this purely opt-in I can flip the default to `admin`. ## Verification - `tests/core/test_auth_api.py::TestProxyProvisioningRole`: first user admin / second user not; `PROXY_AUTH_DEFAULT_ROLE=admin` restores the old behaviour; an admin from another auth source counts as "an admin exists"; a known user keeps their role whatever the default. - Full suite (3094), ruff, ruff format, basedpyright, vulture green. - Running on my own instance since 2026-09-19. |
||
 Marcelo Rodrigo
|
b7002a6eca |
feat: Add TorBox client support and settings integration (#1342)
Add **TorBox** as a torrent download client for Prowlarr releases. Users can select `TorBox` in the download client settings, configure it with the new `TORBOX_API_KEY` environment variable, and verify their credentials with the connection test button. The integration supports both magnet links and `.torrent` files. It tracks the torrent lifecycle through TorBox, downloads supported book and audiobook files from the TorBox CDN, preserves safe nested file paths, and cleans up remote and local download state. Important: Shared HTTP download logs omit full download URLs and URL-bearing exception text to avoid exposing credentials, following best practices. This applies to all clients that use the shared `download_url()` path; URLs remain available to the HTTP operations themselves. --- There is already related work in progress in #1173, which includes both torrent and direct-download support for TorBox. This PR is not intended to replace or compete with that contribution. It offers the tested torrent client functionality as a smaller, focused change that can make TorBox available to the community sooner. The direct-download integration proposed in #1173 remains valuable and could be reviewed or introduced separately. Automated tests cover configuration, connection validation, magnet and torrent-file submission, API errors, status and progress handling, file retrieval, path traversal protection, cancellation, cleanup, and sensitive URL redaction. I also validated the complete flow locally with several magnet links and `.torrent` downloads. TorBox processed the torrents and Shelfmark downloaded the resulting files as expected. AI was used to help with the implementation, with human validation. This PR and long description? Took me some good minutes at night after work, but gives me joy to open this PR to share with the community this improvement. |
||
 Vinicius Gabriel
|
c53545d9fe |
Add configurable word separator for naming templates (#1333)
Closes #1230 ## What Adds a "Word Separator" setting (Space / Dot / Underscore / Hyphen / Custom) that replaces internal whitespace in each naming-template placeholder's rendered value — e.g. `{Author}` renders "Arthur.Conan.Doyle" instead of "Arthur Conan Doyle" when Dot is selected. This follows option 2 from the issue rather than inventing new dotted-keyword template syntax (`{Author.}`), since it's a smaller surface: one setting applies uniformly across all four templates (books/audiobooks × rename/organize) instead of needing a parallel token for every existing one. ## How it works - Literal characters typed into the template itself (e.g. the `.` in `{Author}.-.{Title}`) are never touched — only whitespace *inside* a placeholder's resolved value is affected. - Default is "Space", which is a no-op: existing templates produce byte-identical output after this change (verified via the existing test suite, unmodified, still passing). ## Where - `shelfmark/core/naming.py` — `word_separator` param on `parse_naming_template` / `build_library_path`. - `shelfmark/download/postprocess/policy.py` — `get_word_separator()`, mirroring the existing `get_file_organization()` accessor. - `shelfmark/download/postprocess/transfer.py` — wires the resolved separator through the four existing template-rendering call sites. - `shelfmark/config/settings.py` — new `Word Separator` / `Custom Word Separator` fields next to the existing naming-template fields. - `src/frontend/.../namingTemplatePreview.ts` + `NamingTemplateField.tsx` — the settings UI has its own TS mirror of the Python renderer for the live preview; updated it in lockstep so the preview doesn't lie about what the separator will actually do. - Tests added on both sides (pytest + vitest). ## Testing - `uv run pytest tests/core/test_naming.py tests/core/test_destination_file_organization.py` — all pass, including new cases. - `uv run pytest` (full suite) — same pre-existing failures as on `main` before this change (browser/network-dependent bypass & e2e tests unrelated to this diff), everything else green. - `uv run ruff check` / `ruff format --check` / `basedpyright` — clean. - `npm run lint` / `format:check` / `typecheck` / `test:unit` (196 tests) — clean. |
||
|
Vinicius GabrielandClaude Sonnet 5
|
c576003319 |
feat(naming): add {FirstAuthor} template token (#1322)
Closes #930. ## What New `{FirstAuthor}` naming-template token. It renders only the first author when metadata lists several ("Author1, Author2, Author3"), so multi-author books can be filed alongside the rest of that author's work instead of getting their own "Author1, Author2, ..." folder. ``` {Author} -> Terry Pratchett, Neil Gaiman {FirstAuthor} -> Terry Pratchett ``` ## How - Added to `KNOWN_TOKENS` in `shelfmark/core/naming.py`, positioned before `author` so `{FirstAuthor}` isn't parsed as literal `First` + `{Author}`. - Derived inside `parse_naming_template` from the existing `Author` value (split on `,` / `;`), so every caller — folder transfer, rename, the settings preview — picks it up with no extra wiring. An explicit `FirstAuthor` key in the metadata still wins if one is ever passed. - `{Author}` behaviour is unchanged. - Frontend `namingTemplatePreview.ts` token list + `KNOWN_TOKENS` kept in lockstep (there's a test enforcing that), with a matching `firstAuthor` helper. - Settings field descriptions + `docs/environment-variables.md` list the new token. ## Known limitation A lone author written `Last, First` is split on the comma too and renders as `Last` — the source metadata doesn't mark which form it is. Called out in the token help text and covered by a test. `{Author}` remains available for anyone who wants the raw string. ## Checks - `make python-test` — 2963 passed - `make python-lint` / `make python-format` / `make python-typecheck` / vulture — clean - `make frontend-test` — 187 passed · `frontend-lint` / `frontend-format` / `frontend-typecheck` — clean 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
 Nicholas Velten
|
46d21cafbc |
Deep-link Search By mode via URL hash (#1311)
Closes #1228. Search By mode (General/Author/Title/Series/Manual), content type and the search query now live in the URL as a hash fragment, updated live as you search, not just parsed once on load. A shared/bookmarked link like `#search_by=manual&q=foundation` reopens in that exact mode with the query filled in. Following the direction from the issue thread: - Hash fragment instead of query string, so it stays browser side only. - Live updates via `history.replaceState`, no history spam per keystroke. - Default Search By persisted in a client side cookie as fallback when there's no hash, no user accounts needed. Tested manually against a local build (search-by switching, hash live update, deep link reload, cookie fallback when there's no hash, and hash overriding a stale cookie). |
||
 Jorge Lima
|
d7fe28595c |
fix(bypass): wait for the solved page before reading its source (#1286)
Follow-up to #1276 with a measurement from the instance I reported there. v1.3.13 solves the challenge again, but on my setup the solve was being thrown away immediately afterwards: ``` 19:26:08 Bypass successful using _bypass_method_cdp_gui_click 19:26:16 Bypass failed (attempt 1/10): TimeoutError: Time ran out while waiting for: {html} ``` `_get()` ends with `return await page.get_page_source()`, which is `find("html", timeout=1)` in SeleniumBase. One second is enough for a page that is already sitting on its content, but Anna's Archive answers a cleared check with a redirect to the real page, so the document is not there yet. The solve is discarded, the whole attempt restarts, and the extra requests are what earn the 429 that `note_rate_limited()` then parks the host for — 120 s, then 300 s. ## Change `_read_page_source()` waits for the document itself, with a `BYPASS_PAGE_SOURCE_TIMEOUT` setting (default 20 s, min 1, max 120) in Direct Download → Cloudflare Bypass, next to the existing bypasser timeouts. ## Measured on a live instance I patched the wait in the running container (`find("html", timeout=1)` → `timeout=20` in the installed seleniumbase, which is the same effect as this PR) and re-ran the same searches on the same host, k3s behind a Surfshark WireGuard exit, internal bypasser, v1.3.13: | | 1 s wait | 20 s wait | |---|---|---| | `Time ran out while waiting for: {html}` | one per solve | none | | 429 backoffs | 2 (120 s, then 300 s) | none | | Search for a book AA has | 199 s and 200 s, both errored | 61 s, 2 epub releases | A download after that took 5 s from LibGen, so the search was the whole cost. ## Tests Two tests in `tests/bypass/test_bypass_budgets.py`, the file already covering #1276: a page that needs longer than a second still yields its HTML, and `BYPASS_PAGE_SOURCE_TIMEOUT` overrides the default. `uv run pytest tests/ --ignore=tests/e2e`: 2848 passed, 47 skipped. Ruff check and format clean. The docs table is auto-generated, but running `scripts/generate_env_docs.py` here rewrote unrelated entries (Newznab, BOOK_LANGUAGE), so I added only the new entry by hand in the generator's format rather than commit that churn. One thing I could not judge from outside: whether 20 s is the right default for hosts other than AA. It only costs anything when a solve would otherwise be discarded, but I have measured it on one site. |
||
|
CaliBrain
|
97e289ae13 | fix: search, Prowlarr and qBittorrent follow-ups (#1276, #1283) (#1284) | ||
 Lance Marks
|
f441b85da2 |
feat(packs): inspect multi-book releases and file each book separately (#1270)
## Multi-book packs: inspect a release before download and file each book separately Closes #576 ### Problem One queued release is always treated as one book. When a torrent is actually a whole series (`Series/Book 1 - Title/…`, or a flat folder of `Series 1.0 - Title.m4b` files), post-processing walks the whole tree, flattens every file into one list and renames them `Title - 01…10` under the searched book's `{Author}/{Title}`. Audiobookshelf then sees a single 10-file "book" and the user has to re-file everything by hand. ### What this does Most releases expose their file list *before* anything is downloaded, so the split is decided up front and approved by the user, then the download is fire-and-forget: 1. **Inspect** – clicking a release's download button now calls `POST /api/releases/inspect` first. A new optional `DownloadHandler.list_files(release_data)` hook returns the release's files without downloading: - **AudiobookBay** reads the torrent file table off the detail page it already fetches (the page is now cached for 120 s, so inspect + download cost ABB one request). - **Prowlarr** parses `info.files` from the `.torrent` it already fetches (the existing 120 s torrent-fetch cache is reused). Magnet-only and usenet releases report "can't inspect". - Other sources default to `None`. 2. **Review** – if the plan contains more than one book, the Find Releases modal swaps the list for a review panel: one row per book with editable title / series position / year, expandable file lists, non-book sidecars (`.txt`, covers) shown as ignored, a "Treat as a single book" switch, and **Download N books**. Single-book releases queue immediately, exactly as before. 3. **File** – the approved plan travels with the task (`DownloadTask.book_plan`, retry-safe) and post-processing files each book through the existing transfer code, one book at a time (`dataclasses.replace(task, title=…, series_position=…, year=…)`), so organize/rename templates, part numbering (now scoped per book), hardlinks, torrent copy-preserve and usenet handling are unchanged. Status reads `Complete (N books, M files)`. 4. **Fallback** – when a release can't be inspected the user gets a toast, and a small "Multi-book pack" toggle in the modal header forces a heuristic split (subfolder = book, or one book per file when the file names carry series positions). Planning lives in `shelfmark/download/postprocess/packs.py` and is shared by the inspect endpoint and post-processing, so what the user approved is what gets filed. The name parser strips `Book 3 -`, `03 -`, `1.0 -`, `3.`, `[03]`, `#3`, a leading series name, labels like "An Expanse Novella -", repeated titles (`Gods of Risk 2.5 - Gods of Risk`) and a trailing `(Year)`; author and series name come from the book that was searched, and the searched book's own series position is never applied to its siblings. ### Files - `shelfmark/download/postprocess/packs.py` (new) – `PackFile/PackBook/PackPlan`, `plan_pack`, `parse_pack_book_name`, `group_files_into_books`, `match_plan_to_files` - `shelfmark/core/release_inspect_routes.py` (new) – `POST /api/releases/inspect` - `shelfmark/release_sources/__init__.py` – `DownloadHandler.list_files` hook - `shelfmark/release_sources/audiobookbay/{scraper,handler}.py` – detail-page cache, `extract_file_list`, `list_files` - `shelfmark/release_sources/prowlarr/handler.py`, `download/clients/torrent_utils.py` – `extract_file_list_from_torrent`, `list_files` - `shelfmark/core/models.py`, `download/orchestrator.py` – `multi_book` / `book_plan` fields, queue + retry serialization - `shelfmark/download/postprocess/transfer.py`, `pipeline.py`, `outputs/folder.py` – per-book transfer branch and status message - `src/frontend`: `components/PackReviewPanel.tsx` (new), `ReleaseModal.tsx`, `App.tsx`, `services/api.ts`, `types/index.ts`, `utils/releasePayload.ts` (payload builder moved out of `App.tsx`), `utils/packReview.ts` - `docs/dev/release-sources-plugin-guide.md` – documents the `list_files` hook ### Out of scope (follow-ups) - Listing files from an NZB (Shelfmark already fetches the bytes; `<file subject>` names are noisy) - Inspecting magnet links via qBittorrent's files API after a paused add - BookLore / email outputs (they ignore `book_plan`; noted in code) - The combined ebook + audiobook flow ### Testing **Automated** (`make checks`, `make python-test`, `make frontend-test` all green; the only failures on my machine are the pre-existing `tests/config/test_entrypoint_permissions.py` cases, which need bash ≥ 4 and fail identically on `main` under macOS bash 3.2): - `tests/download/test_packs.py` – name parsing (markers, series name, novella labels, repeated titles, bare numeric titles like `1984`), nested / flat / mixed / deeper-nested packs, single wrapping folder not treated as a pack, plan-to-disk matching with basename fallback - `tests/core/test_processing_packs.py` – full `post_process_download` runs on a real temp filesystem: approved plan files each book under its own `{Author}/{Title}`, heuristic split of a nested pack, searched book's series position does not leak, multi-file book inside a pack keeps `- 01/- 02` per book, hardlinked torrent pack leaves the seeding tree intact, no pack fields ⇒ behaviour unchanged, single group degrades to the searched title, status message - `tests/core/test_release_inspect_routes.py` – plan response, not-inspectable, handler errors never 500, unknown source / missing `source_id` ⇒ 400, login required - `tests/audiobookbay/test_file_list.py` – file-table scraping from real ABB markup (multi-file and single-file pages), handler host validation, one page fetch shared by magnet + file list - `tests/prowlarr/test_torrent_file_list.py` – multi-file / single-file `.torrent` parsing, handler behaviour for torrent URL vs magnet vs usenet vs cache miss - `tests/download/test_orchestrator_pack_fields.py` – queue-time parsing and retry round-trip - Frontend: `releasePayload.test.ts`, `packReview.test.ts` (vitest) **Manual, on a real deployment** (arm64 image built from this branch, run as a side container next to production with the same qBittorrent / Audiobookshelf setup, `FILE_ORGANIZATION_AUDIOBOOK=organize`, hardlinks on): - AudiobookBay "The Expanse Complete 2.0" (7.87 GB, 36 files): clicking download opened the review panel in ~1 s showing **18 books · 18 files · 18 files ignored** (the `.txt` sidecars), with series positions 0.1–9.5 and years parsed from the file names; novella labels stripped ("The Churn", "The Butcher of Anderson Station"). Editing a title in the panel works. Confirming queued one task; the magnet resolved from the cached page in ~30 ms; after the download the task reported `Complete (18 books, 18 files)`, 18 hardlinks landed as `audiobooks/James S. A. Corey/<Title>/<Title>.m4b`, the torrent kept seeding, and Audiobookshelf scanned each folder as its own book (title, author, embedded chapters). - A second pack ("Expanse [01 - 9.5]", `Title N - Title` naming) was inspected to verify the repeated-title rule and the Back button, without downloading. - Single-book releases still queue immediately with no extra UI. |
||
  zab1996andRyan
|
02b7e9d958 |
feat(newznab): support multiple named indexers (#1271)
## Summary - add a named Newznab indexer table with per-indexer URL and API key settings - search every configured indexer and retain the originating indexer name on each result - namespace cached release IDs across connections and isolate individual indexer failures - preserve the legacy single-indexer settings as a fallback - support masked API-key cells and trusted SABnzbd prefetching for named indexers ## Validation - 121 Newznab and SABnzbd backend tests passed on Python 3.14 - Ruff passed for all changed Python files - frontend TypeScript and strict lint checks passed - all 134 frontend unit tests passed - frontend formatting check passed ## Compatibility Existing `NEWZNAB_URL` and `NEWZNAB_API_KEY` configurations continue to work whenever `NEWZNAB_INDEXERS` is empty. Co-authored-by: Ryan <zab1996@users.noreply.github.com> |
||
|
463ef49ac3 |
feat(search): let each user pick their own default book languages (#1255)
## Why `BOOK_LANGUAGE` is a per-reader property, not a per-instance one. On a shared install one household member searches in German while another wants English and German — today whoever changes the setting changes it for everyone, and the only escape is re-picking languages in the filter on every single search. The per-user override machinery already carries `SEARCH_MODE`, the metadata providers and the default release sources, so the language default mostly had to opt into it. ## What changed **The field.** `BOOK_LANGUAGE` becomes `user_overridable` and moves from the **General** tab to **Search Mode**, next to the other user-overridable search defaults (per [review](https://github.com/calibrain/shelfmark/pull/1255#issuecomment-5391189094) — the first version had the Search section span two tabs, this one doesn't). Admins set it per user in the user editor, users set it in **My Account → Search Preferences**, and the Search Mode tab carries the usual "N users override this" summary. **No migration for the move.** `general` and `search_mode` both persist into `settings.json`, and a field's value is resolved through `load_config_file(tab)` for the tab it's declared on — so an install that already stores `BOOK_LANGUAGE` keeps its value. Checked against a `settings.json` written while the field still lived on General: the stored value resolves unchanged, a fresh install still gets `["en"]`, and `BOOK_LANGUAGE` in the environment still overrides both. **The two places the default is read.** - `/api/config` seeds the frontend's language filter, so it now resolves `BOOK_LANGUAGE` for the session user. - `build_release_search_plan` falls back to the default whenever a request carries no language filter — which is exactly what the filter's "Default" option sends. It takes an optional `user_id`, passed by `/api/releases` from the session and by the Prowlarr retry path from `task.user_id`, so a retry re-searches in the languages of whoever queued the download. **Validation.** Overrides go through `normalize_language()`, so `"German"`, `"ger"` and `"de"` all store as `de`, and an unknown language is rejected with a message naming it instead of being silently searched for. An empty list stays an empty list (a deliberate "no default filter"), `null` clears the override as everywhere else, and ENV still wins: with `BOOK_LANGUAGE` set in the environment the field reports `fromEnv` and overrides are ignored. **Scope.** Only the language default becomes overridable. The two format lists left behind under "Default Search Filters" stay admin-only — they describe what the library and its post-processing accept, not what a reader wants to read. There's a test pinning that. ## Verification - 2681 unit tests pass (2670 before, 11 added) - `ruff check`, `ruff format`, `basedpyright` over backend and tests, and `vulture` all clean; frontend lint, format, typecheck and 126 unit tests clean - `docs/environment-variables.md` regenerated via `scripts/generate_env_docs.py` (the `BOOK_LANGUAGE` row follows the field into the Search Mode section) - Manually against a two-user instance with builtin auth (first round, before the tab move): with user A on German and user B on English+German, `/api/config` returns each reader their own `default_language` and an unfiltered `/api/releases` plans the matching languages; an admin can set and read the same override for another user; clearing it falls back to the global value; a stray `"klingon"` is rejected; and `BOOK_LANGUAGE` in the environment overrides both users with the field marked `fromEnv` - After the tab move I re-ran the suites above plus the stored-value/fresh-install/ENV check described under "No migration for the move"; the behaviour it exercises is what the move could have broken Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
 jakesterpdxandClaude Fable 5
|
65e2e3be20 |
feat(audiobook): recognise .mp4 as an audiobook format (#1264)
## Problem Some trackers — MyAnonamouse in particular — distribute AAC audiobooks as per-chapter `.mp4` files. That's the same ISO-BMFF container as `.m4a`/`.m4b`, just with the generic extension (`ftyp isom`, audio-only). Today those releases: 1. show up in Prowlarr search results with **no format chip** — only the generic "Audiobook" icon, because no format could be inferred; 2. download successfully; then 3. fail post-processing with **"No book files found in download"**, because `.mp4` isn't in `AUDIOBOOK_FORMATS` (`shelfmark/core/utils.py`). Real example: MAM #627978, *The Martian* (Andy Weir, 2020 edition) — 142 files `0001 … 0142 Andy Weir (2020) The Martian.mp4` + `cover.jpg`, 305 MB. Every file is a valid AAC-in-MP4 chapter. Adding `mp4` to `SUPPORTED_AUDIOBOOK_FORMATS` in `settings.json` doesn't help since the hard-coded tuple is what post-processing scans against. ## Change - Add `"mp4"` to `AUDIOBOOK_FORMATS` (single source of truth — settings UI, Prowlarr parsing, IRC parser, archive extraction and post-download scan all derive from it), with a comment explaining why. - Add `".mp4"` to the two hand-maintained debrid `_BOOK_EXTENSIONS` lists (AllDebrid / Real-Debrid) so file selection matches. - Slot `mp4` into the IRC `AUDIOBOOK_FORMAT_PRIORITY` table right after `m4a` (same container family). - Update the documented default in `docs/environment-variables.md`. - New regression test `test_audiobook_multifile_mp4_chapters_are_book_files` modelled on the existing multi-file usenet test. ### Note for existing installs The legacy-default migration only widens configs that still hold the old `m4b,mp3` list, so users on the current widened default won't pick up `mp4` automatically — they'll need to tick it in Settings → Audiobook formats. New installs get it by default. Happy to extend the migration if you'd rather it be automatic. ## Testing - `ruff check` / `ruff format --check`: clean - `pytest tests/core tests/config tests/irc tests/prowlarr tests/download -m "not integration and not e2e"`: 2296 passed, new test + `test_audiobook_format_consistency.py` all green. The 10 failures in `test_entrypoint_permissions.py` / `test_orchestrator_stall.py` reproduce identically on untouched `main` on macOS and are unrelated. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
CaliBrain
|
7d56624ab6 |
fix: group multi-file audiobooks that arrive as an archive (#1254)
Follow-up to #1237. \`rename_and_group\` only grouped when the source root was a directory, so a multi-file audiobook delivered as a single archive fell through to the flat path: a \`Book.zip\` of twelve chapters landed loose in the destination root with its original chapter names — the layout #1181 is about. The \`is_dir()\` guard was there to keep \`Book.zip/\` from becoming the folder name, but skipping the file case gives up the grouping instead of naming it. A non-directory source can only produce several book files by having been extracted (\`collect_staged_files\` returns a single-element list for every other file shape), so the archive stem is the release name and the suffix is packaging: group under \`Book/\`. Also regenerates the env docs for the new option and gives it the same \"do not use with ingest folders\" caveat Rename and Organize carries, since both now create directories in the destination. Tested: reverting only the source fix makes both new tests fail and the \`rename\` control case pass, so grouping stays opt-in. Full non-e2e suite green (2653 passed). |
||
|
CaliBrain
|
e7007865a4 |
fix(prowlarr): stop turning indexer failures into empty results and 404s (#1251)
Two independent bugs, both from an indexer that Prowlarr proxies rather
than answers for itself: the search never reported that it had failed,
and the grab never resolved what it was handed.
Search. A Torznab search is Prowlarr proxying a live request out to the
tracker, so for a Cloudflare-fronted indexer it waits on FlareSolverr.
The client gave it the 30s budget sized for Prowlarr's own JSON
endpoints, then swallowed every failure -- the timeout, the 429 Prowlarr
returns once it has disabled an indexer, a parse error -- into the same
empty list that means "this indexer has nothing". A cold challenge
routinely runs past a minute, so the UI said "No releases found for this
book" while FlareSolverr was still solving. That empty list also drove
the auto-expand retry, which fires on "no results with the category
filter". A timeout satisfies it, so Shelfmark sent a second search to an
indexer still busy with the first -- two Chromes at once, enough to take
FlareSolverr's down on a small host.
torznab_search now raises ProwlarrSearchError, and an empty list
strictly
means the indexer answered with no matches. The source records which
indexer searches failed: one dead indexer no longer sinks the others,
auto-expand runs only when every indexer genuinely answered, and zero
results with at least one failure raises SourceUnavailableError, which
the releases endpoint already turns into a 503 carrying a real message.
Prowlarr being unreachable was the same lie by another route -- the
indexer list came back empty, leaving nothing to query -- and now says
so.
Indexer searches also get their own timeout, PROWLARR_INDEXER_TIMEOUT,
defaulting to 90s and clamped to 5-300. Prowlarr's status and indexer
list keep 30s so Test Connection stays responsive, and the connect
timeout is split out at 10s so an unreachable Prowlarr fails fast rather
than hanging for the whole read budget. The overall per-request search
budget now scales to twice the indexer timeout, capped at 240s, so
raising the setting is not undone by the cap one level up while staying
under the 300s gunicorn worker timeout.
Grab. Prowlarr hands out a proxy download URL, with no magnetUrl and no
infoHash, for any indexer that only publishes torrent files. The native
Real-Debrid client built its magnet as "if not
url.startswith('magnet:') and expected_hash", so with no hash to work
from it left the URL alone and POSTed it to /torrents/addMagnet as the
magnet field. Real-Debrid answered 404 and the grab died on a raw HTTP
error. AllDebrid carried the same line and the same bug.
Both now resolve the URL first, through the extract_torrent_info path
the
torrent clients have used since #1108: pass a magnet through untouched,
follow a redirect or a response body that turns out to be a magnet,
otherwise upload the fetched .torrent, and fall back to a magnet built
from the infoHash only when the fetch failed. The file is preferred over
a synthesized urn:btih: magnet because it carries the tracker list; a
bare hash leaves the service to find the swarm on DHT alone. Fetches are
shared with the rest of the add path through the torrent fetch cache, so
resolving costs at most one request. Real-Debrid takes the file on PUT
/torrents/addTorrent with the raw bytes as the request body, AllDebrid
on
POST /magnet/upload/file as multipart files[]. A URL that resolves to
neither form now raises before any request reaches the service, so the
user reads why instead of a 404. Neither debrid client had any test
coverage; both have some now.
Fixes #1249
Fixes #1250
|
||
 helgehelge123andhelgehelge123
|
7b9c416df8 |
perf(bypass): keep the helper subprocess alive between bypasses (#1222)
Every protected request spawns a fresh helper subprocess, paying interpreter start and imports before any work begins. Measured inside the container, five consecutive runs of `python -c "import shelfmark.bypass.internal_bypasser"`: ``` 3.53s 3.45s 3.55s 3.54s 3.46s ``` A single search issues several protected requests, so that is paid several times over per search. ## What changed The helper now serves one JSON request per line of stdin until the parent closes the pipe, and an idle timer (`BYPASS_BROWSER_IDLE_TIMEOUT`, default 180s) shuts it down once searching stops. Answers still travel by result file, but the file is now written to a `.part` path and renamed into place — the parent treats the file's existence as the answer, so it must never observe a half-written one. stdout and stderr stay attached to the parent's, so helper logs keep appearing in `docker logs` exactly as before. Failure handling, since a warm helper is exposed to more of it than a per-request one ever was: | Situation | Handling | | --- | --- | | Helper died between requests | Detected via `poll()`, respawned | | Pipe broken at write time (`poll()` can miss this) | One retry on a fresh process; a fresh one failing there is a real failure | | Helper exits without writing a result | `RuntimeError` naming the exit code | | Wedged past the timeout, or cancelled mid-bypass | Helper killed, then `_cleanup_orphan_processes` because a killed helper never got to close Chrome | | Idle reaper racing an arriving request | Re-checks the deadline under the lock and re-arms instead of killing a helper that just did work | The DNS config now travels with every request rather than only at spawn: a warm helper outlives changes the parent makes to its provider. ## `BYPASS_REUSE_BROWSER`, off by default This parks the CDP driver between bypasses. A driver's websockets are bound to the loop that opened them and cannot outlive their process, so the persistent helper is what makes this possible at all — and the warm path runs on `_CDP_WORKER`'s long-lived loop rather than `asyncio.run` for the same reason. The mechanism works. With it on, the browser start disappears from the second request onward: 0.7s from `Reusing warm Chrome browser` to the first bypass attempt, against roughly 16s cold. **It still ships off, because a matched-pair test shows it is a net loss against DDoS-Guard.** Each round primed with one cold bypass, waited 10s, then measured a second — identical timing in both arms, only the browser strategy differing, order balanced (fresh, warm, warm, fresh) so drift over the session cannot masquerade as an effect: | Arm | Measured request | | --- | --- | | fresh browser | 42.8s, 40.6s | | warm browser | 57.1s, 59.6s | Spread within each arm is 2.2s and 2.5s, against 16.7s between them. Reuse removes the ~15s browser start and then gives back roughly twice that in solving: a returning browser draws a harder challenge. Where the cold browser is through on the second bypass method, the warm one fails the first three and only `_bypass_method_humanlike` gets it, at ~30s for that method alone. Worth separating from a second effect I ran into while measuring: five back-to-back searches slow from ~32s to 51–98s with reuse **disabled** as well, so DDoS-Guard escalates on request rate independently of any of this. That is why the pairs above are timed identically rather than simply run in sequence. It is the larger of the two effects, but not something this project can patch around. Reuse is left available rather than dropped because Cloudflare sites may not respond the same way, and because the two concerns are independent: the helper start is pure overhead and always worth removing, the browser is not. ## Verification - 2559 unit tests pass (2542 before, 17 added in `tests/bypass/test_warm_browser.py`) - `ruff check`, `ruff format`, `basedpyright` over backend and tests, and `vulture` all clean - `docs/environment-variables.md` regenerated via `scripts/generate_env_docs.py` - Live against Anna's Archive on a warm helper: searches return their usual ~760KB and 667 results, the app's own search warm-up completes with 50 results, and the container is left with no orphan chrome/Xvfb/ffmpeg processes Happy to drop the `BYPASS_REUSE_BROWSER` half entirely if you would rather not carry a default-off path — the helper persistence stands on its own. Co-authored-by: helgehelge123 <helge.neumann@zollsoft.de> |