mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-11-15 10:58:31 +00:00
08fdb7ea873d81cc4ddb024e04f75eca55324462
298
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 splitsec2
|
08fdb7ea87 |
fix(startup): the reconcile guard catches the same errors the rest of main.py does (#1540)
The start-up reconcile runs at import, before the coordinator starts, inside a try that named five exception types and left out `TypeError`. `main.py` already defines `_IMPORT_OPERATIONAL_ERRORS` for this shape and the other module-level guards use it, so this switches to it. An unexpected `TypeError` from the reconcile would otherwise fail the import of `shelfmark.main`, and the worker would fail to boot on every restart. I didn't use `except Exception` because BLE001 is on and nothing in main.py does. The test reloads main with the reconcile raising `TypeError`, and with `OSError` as the control (that one already passes on main). Full non-e2e suite passes, ruff and basedpyright are clean. Found in a Claude pass over 1.5.0. |
||
|
splitsec2
|
f560b31ee4 |
fix(db): wait up to 30 s for a busy writer instead of SQLite's 5 s default (#1539)
The users, download history and activity view-state databases open their connections with SQLite's default busy timeout of 5 seconds. A start-up migration, the restart reconcile, or any statement that overlaps a backup or a second process raised "database is locked" after 5 seconds, and the reconcile finished only part of its work. One shared `DB_BUSY_TIMEOUT_SECONDS` (30) in `core/utils.py`, used by all three. The test reads SQLite's `busy_timeout` pragma back from each connection, so it doesn't need a real lock fight. I left `UserDB.initialize()` without a guard on purpose. With 30 seconds the wait covers a backup, and swallowing a schema failure there would hide a worker that can't run. Found in a Claude pass over 1.5.0. |
||
|
splitsec2
|
e27fed1d40 |
fix(queue): a late progress update no longer reopens a finished task (#1538)
#1429 made the first terminal status win, but the guard only refused terminal to terminal changes. A worker's poll that reports DOWNLOADING after a cancel still overwrote CANCELLED (the poll loop calls the status callback before it waits on the cancel flag), and the COMPLETE or ERROR that followed was then accepted as a first terminal status, so the terminal hook ran a second time. This refuses any status change away from a terminal status. `add()` and `enqueue_existing()` reset the status themselves, so retries are unaffected. The new test cancels a downloading task, sends DOWNLOADING and then COMPLETE, and expects CANCELLED and one hook call. It fails on main and passes here. Full non-e2e suite passes, ruff and basedpyright are clean. Found in a Claude pass over 1.5.0, same as my other small fixes from it. |
||
|
splitsec2
|
395c331b4a |
fix(search): give each provider its own slice of the search budget (#1537)
Fixes #1531. All Direct Download providers share the one search budget (`RELEASE_SEARCH_TIMEOUT`, 300s). Anna's Archive runs first, and when its bypass keeps failing the retries use up the whole budget. OceanofPDF then sees an expired deadline, skips its queries, and the search raises Anna's Archive's error even though OceanofPDF never got a turn. I didn't split the budget evenly, since the deadline module itself says a cold-start Anna's Archive solve can legitimately take 60 to 120s and a 150s or 37s slice would break setups that work today. Instead `search_deadline.unit_deadline(waiting=n)` opens a child deadline for one unit of work. It gets the parent's remaining time minus 30s for each unit still waiting, never less than 30s and never more than the parent has left. A single provider still gets everything, a fast provider's unused time rolls on to the next, and a stuck one stops at its own limit with the reserve intact. It's used in the provider loop in `DirectDownloadSource.search`, and in the source loop in `/api/releases` since it's the same shape and four lines. With several sources enabled the first one now gets 30s less per source behind it, which is still well above a cold solve for the usual two or three. Tests: a stuck first provider no longer starves the second (fails on main, passes here), a lone provider gets the full budget, and a child never outlives its parent. Two existing tests in `test_releases_api_budget.py` tripped the deadline in force from inside the source, which is now the child, so they use a short request budget instead. ruff, basedpyright on the changed files and the non-e2e suite are clean. OceanofPDF is inactive on my own instance (no mirror URL set), so I haven't reproduced this live. It's verified by reading the code and by the tests. I had Claude do the pass on this one. |
||
|
splitsec2
|
8e425da527 |
refactor: one default and one reader for the Supported Book Formats list (#1532)
The default for Supported Book Formats (`epub, mobi, azw3, fb2, djvu, cbz, cbr`) was written out in five places: the settings registry, `postprocess/policy.py`, `irc/parser.py`, Prowlarr's `source.py` (added in #1490) and slskd's `source.py` (added in #1428). Both of those two arrived in 1.5.0, and both are mine, so this is cleaning up after myself as much as anything. The setting was also read three different ways. Post-processing split on commas only. slskd split on commas and spaces and stripped a leading dot. IRC split on commas and kept the dot. Prowlarr split on commas and kept `""` as a format when the value was an empty string. So `.epub` or `epub mobi` worked in one source and not another. This puts `DEFAULT_EBOOK_FORMATS` and `normalize_format_list` in `core/utils.py` next to `AUDIOBOOK_FORMATS`, and the registry, post-processing, IRC, Prowlarr and slskd all use them. I took the most lenient reading of the three (comma or space split, leading dot dropped, empties ignored, default when nothing is left), so no existing config gets stricter. I put the reader in `core/utils.py` and not in `policy.py` because the release sources can't import `policy` without pulling in the whole post-processing package. A config with a sensible list reads exactly as before. The only differences are for malformed values: an emptied setting or an empty string now falls back to the default in every place, where IRC used to filter everything out and Prowlarr treated `""` as a format. The audiobook default is a separate story and I left it alone: the registry and slskd default to every processable audiobook format plus the archives, while post-processing and IRC fall back to `m4b, mp3`. They only differ when the setting is unset or emptied, but they do differ, and I'd rather ask which one you want than pick one here. Ruff, basedpyright on the changed files and the backend suite pass (4400 passed, e2e not run). New tests cover the reader and the two fallbacks. |
||
  Amalkrishna A SandCaliBrain
|
a11a63a29e |
feat(notifications): add {User}/{Email} placeholders to per-user routes (#1499)
Closes #1130. Per-user notification route URLs can now use `{Email}` and `{User}`, filled in from each user's account at send time, the same way `{User}` already works in destination paths. One default route like `mailtos://user:pass@smtp.example.com?to={Email}` now reaches every user at their own address, with no per-user setup. - Values are URL-encoded (`alice+books@example.com` keeps its `+`); placeholders are case-insensitive. - A route that needs a value the user doesn't have (e.g. no email on file) is skipped for that user; other routes still send. - The user lookup only runs when a route contains a placeholder. - The personal "Test notification" button expands the placeholders too, so testing a `{Email}` route works. - Added to the routes setting help, `docs/users-and-requests.md`, and tests. @calibrain ready for review. --------- Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
 
|
c43da4537d |
fix(network): apply HTTP proxy to Open Library and Prowlarr requests (#1526)
Fixes #1524 Open Library metadata searches and Prowlarr API calls used `get_ssl_verify` but never `get_proxies`, so the HTTP proxy set in Settings → Network was not applied to their HTTPS requests. Both now pass the configured proxies. Relying on `requests`' `trust_env` with only `HTTP_PROXY` set leaves the same gap for `https://` destinations. Verification (Linux, `uv run --frozen pytest`): - New tests `tests/metadata/test_openlibrary_proxy.py` and `tests/prowlarr/test_api_proxy.py`: 4 passed. With the source change reverted they fail (2 failed, 2 errors). - `tests/metadata` + `tests/prowlarr`: 949 passed, 42 skipped. - The full suite in my environment showed 61 failures and 26 errors outside these areas. I did not compare them against `main`, so I'm not claiming the full suite is green here; CI will show it. - `ruff check` and `ruff format --check` on the changed files pass. Prepared with AI assistance; reviewed and tested by me before submitting. Co-authored-by: dbdefault <246079841+dbdefault@users.noreply.github.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
CaliBrain
|
1aa9846f22 |
fix(download): trust a pinned AA mirror and repair the full e2e profile (#1518)
The destination guard from the oceanofpdf/SSRF fix (
|
||
 DBVamsiandCaliBrain
|
2fa0ad27f3 |
Fix [BUG] Search "Options" move out of the way when you try to click them (#1468)
Fixes #1301. ## What changed ## Summary **Root Cause (Issue #1301):** The SearchBar component uses a `useDismiss` hook that closes the main search menu on `mousedown` events occurring outside of `selectorRef` and `controlsPanelRef`. However, the AdvancedFilters panel (opened via the "Options" button) is rendered as a sibling component outside of `controlsPanelRef`. When clicking on dropdown options inside AdvancedFilters, the `mousedown` event bubbles up to the document, triggering `useDismiss` and collapsing the main menu. **Fix:** Added `onMouseDown={(e) => e.stopPropagation()}` to the `settingsForm` div in `AdvancedFilters.tsx`. This prevents the `mousedown` event from bubbling up to the document level, so the `useDismiss` hook in SearchBar never sees the event and the main menu stays open. **Test:** Created `tests/frontend/test_search_menu_dismiss.py` with 3 tests: 1. `test_advanced_filters_prevents_mousedown_bubbling` - Verifies the fix is in place 2. `test_advanced_filters_container_has_onmousedown` - Verifies the handler is on the correct element 3. `test_searchbar_uses_usedismiss_for_menu_closing` - Documents the existing dismissal behavior **Verification:** - Tests pass with the fix: ✓ - Tests fail without the fix: ✓ (2 tests fail as expected) - Existing tests still pass: ✓ (215 tests in config/) ## How it was verified - test(s) added/changed: `tests/frontend/test_search_menu_dismiss.py` - on the original code the new test fails (exit 1) - with the fix it passes (exit 0) - full test suite with the fix: exit 1, no new failures vs. baseline (exit 1) (153.7s) This was worked on in conjunction with AI tools. Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
 Desislav LesovandCaliBrain
|
fc8a5b9230 |
fix(prowlarr): hide results whose title extension is not a supported format (#1490)
Closes #1481 ## Problem Plain (non-enriched) indexers such as Zamunda, Knaben or LimeTorrents return titles like `Иван Вазов - Под игото.doc`. They show next to the epubs. Picking one downloads the file, then post-processing discards it because `doc` is not in `SUPPORTED_FORMATS`. ## Change - `_unsupported_title_extension()` reads a trailing file extension from the release title. When it is a known file format that is not in Supported Book Formats, the result is hidden. - Kept: titles with no extension, `[EPUB]`-style tags, dotted titles like `Jane.Eyre.by.Bronte` (only known file extensions count), archives (`zip`, `rar`, `7z`), and audiobook searches. - New Prowlarr setting `PROWLARR_HIDE_UNSUPPORTED_FORMATS` ("Hide results in formats you did not enable"), default on. - One info log line per search with the hidden count; a debug line per hidden title. - `docs/environment-variables.md`: added only the new setting's rows. Running `scripts/generate_env_docs.py` on current `main` also changes unrelated lines (`SHELFMARK_API_KEY_READONLY`, the slskd anchor), so I left those alone. ## Tests - New `TestHideUnsupportedFormats` in `tests/prowlarr/test_source.py`: the extension helper (parametrised), audiobook bypass, comma-separated config, and a full `search()` with the setting on and off. - `tests/prowlarr`: 784 passed. Ruff lint/format, vulture and basedpyright on the changed files are clean. - Full suite on macOS: the same 84 bypass/entrypoint failures as on `main` (browser extra and Linux permissions), none new. I ran an earlier version of this as a local patch on v1.4.0 against a real Prowlarr + Zamunda setup: the `.doc` row is gone and epubs stay. Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
CaliBrain
|
d42494c696 |
fix(tor): survive container restarts and run Tor as debian-tor (#1517)
With USING_TOR=true, any restart that reuses the container (docker
restart, a Docker daemon restart, a host reboot) put it in a restart
loop. Only recreating the container ended it.
Tor appends to /var/log/tor/notices.log, and the file survives the
restart. tor.sh waited for Tor by grepping that log for "Bootstrapped
100%", so the previous boot's line passed at once, before Tor had
connected to anything. The NAT rules then went live, Tor could not
finish bootstrapping behind them, the connectivity check could not
resolve check.torproject.org, and the next boot found the same stale
line. tor.sh now moves the previous log to notices.log.prev before it
starts Tor, so the wait and the watchdog only see this boot's output.
Tor could not finish behind the NAT rules because it ran as root. The
rules exempt only debian-tor's traffic from the transparent redirect, so
every relay connection Tor opened after they went live was redirected
into its own TransPort: the log shows dozens of TLS handshakes dying.
This also meant a Tor that lost its guard could never reconnect, on any
kernel. torrc now has `User debian-tor` and `DataDirectory
/var/lib/tor`. Tor still binds DNSPort 53 as root, then drops. It opens
its log after dropping and refuses to start if it cannot, so tor.sh
hands the fresh log to debian-tor first.
The check also failed without a word. `RESULT=$(curl -s ...)` under
`set -e` ended tor.sh with curl's exit status, so the only trace was
ExitCode=6 (could not resolve host) in `docker inspect`. A reply that
was not the expected JSON did the same through grep. The check now:
- retries twice, 10s apart, since a fresh circuit can fail its first
lookup;
- logs each failure with curl's exit status and error;
- says when it gives up and prints the last Tor log lines;
- still stops the container when traffic does not go through Tor.
- Reproduced on the v1.4.0 image (tor.sh is unchanged since then): after
one `docker restart` the container restarted 9 times in 150s, each
boot ending at "Verifying Tor connectivity...".
- With the new tor.sh mounted into the same image, the container went
through 2 restarts in a row. Each boot verified Tor and reached
gunicorn within 15-21s, and the restart policy never had to step in.
Tor runs as debian-tor and its logs belong to it.
- Restarting Tor (`supervisorctl restart tor`) once the NAT rules were
live, so it had to open new relay connections: as debian-tor it
bootstrapped again in 4s and traffic still went through Tor. As root
it had not bootstrapped after 120s, 1775 TLS handshakes had died, and
curl could not resolve check.torproject.org (exit 6).
- A SIGHUP, which the rotation monitor sends every 5 minutes, keeps
DNSPort 53 working after the drop to debian-tor.
- 9 new tests in tests/config/test_tor_script_rules.py. 8 of them fail
on the old tor.sh; the 9th checks that traffic outside Tor still stops
the container. Three need `grep -P` and skip on macOS, so I also ran
the file in the Debian image: 23 passed.
- The `make python-checks` tools pass (ruff check and format,
basedpyright on the app and tests, vulture). The non-e2e suite
passes: 4366 tests on
|
||
  amura11andClaude Sonnet 5.5
|
ceb23dda7c |
feat(clients): separate audiobook download directory for torrent clients (#1500)
## Summary Related to #1048 (covers the "separate book and audiobook locations" request, not per-category routing). Every download client already had a separate audiobook label/category, but the four torrent clients only had a single save directory, so books and audiobooks ended up mixed in the client's download folder. Each torrent client (qBittorrent, Transmission, Deluge, rTorrent) gets an opt-in **Separate Book and Audiobook Directories** toggle (`<CLIENT>_SEPARATE_DOWNLOAD_DIRS`, default off): - **Off (default):** unchanged. The single `<CLIENT>_DOWNLOAD_DIR` is used for everything, so existing setups behave exactly as before. - **On:** `<CLIENT>_DOWNLOAD_DIR_BOOK` and `<CLIENT>_DOWNLOAD_DIR_AUDIOBOOK` replace the shared field. A directory left empty means the client's own default; it never falls back to the shared directory or another Shelfmark setting. The content type was already passed to each client's `add_download`, so a small shared helper (`resolve_download_dir`) does the selection. Usenet clients are unchanged: their save location comes from the category configured in the client, and a separate audiobook category is already sent. ## Screenshots Default Setting <img width="817" height="271" alt="Screenshot 2026-10-20 at 14-54-26 Shelfmark" src="https://github.com/user-attachments/assets/29941f83-9034-4ea5-bcb2-8e91ce77e63a" /> Separate Locations <img width="817" height="425" alt="Screenshot 2026-10-20 at 14-54-58 Shelfmark" src="https://github.com/user-attachments/assets/772b7194-a072-4945-bac5-b6959550fd75" /> ## Not included Routing to many folders by document type (the main request in #1048) still only exists for Direct download via Content-Type Routing. Extending it to torrent and usenet sources needs a decision on what to route by. ## Testing - `pytest tests --ignore=tests/e2e` passes (3561 passed); ruff clean. The `tests/e2e/platform` suite needs a running cluster and was not run. - New unit tests for the helper (including no fallback to the shared directory) and a qBittorrent add test covering book vs audiobook directories. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
splitsec2andCaliBrain
|
f837506017 |
Merge commit from fork
A signed-in user picks the release fields of POST /api/releases/download, and a record id with the "oceanofpdf:" prefix reached the provider without handles() or is_enabled(). The provider then fetched source_url, POSTed to the form action found on that page and fetched the meta-refresh target, with no check on any of them. The provider now refuses to run when it is switched off, and the book page, the form action and every POST redirect hop must be on the configured mirror (same scheme, host and port). The signed file link may sit on another host but must be http(s). network.ensure_public_destination refuses a host that resolves to a loopback, private, link-local or otherwise non-global address (IPv4-mapped IPv6 judged by its IPv4 address). Hosts the operator configured as mirrors and names that do not resolve pass as before. html_get_page and download_url call it on their first URL and check each redirect hop through a response hook. The image cache uses the shared address check too. Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
splitsec2andCaliBrain
|
cb72ce807c |
Merge commit from fork
A user could set any value for their own overridable settings through
PUT /api/users/me. DESTINATION and DESTINATION_AUDIOBOOK accepted any
string, so a download (including one an admin fulfilled for that user)
could be written to any directory the container can reach, and
get_destination created it on demand.
A user's own destination must now resolve inside the global one (with
{User} expanded on both sides). Absolute paths outside it and any ".."
segment are rejected at save time, and get_destination applies the same
check when the path is used, so a stored or hand-written value outside
the global destination is ignored with a warning and the global
destination is used instead. A relative value is taken relative to the
global destination.
Other self-service values are checked as well: select fields must use
one of their options, EMAIL_RECIPIENT must be a plain address (the same
check the orchestrator and the downloads tab use, now shared from
request_helpers), and Grimmory library and path ids must be non-negative
integers. Admin saves on a user's behalf are unchanged.
The self-service 400 response now carries a message line so the reason
reaches the user in the toast.
Co-authored-by: CaliBrain <calibrain@l4n.xyz>
|
||
|
CaliBrain
|
c269230ef3 |
fix: review regressions in metadata, notifications, activity redaction, search options and genDebug (#1510)
- **Open Library (#1451): keep the work unless the search matched a translation.** Open Library shows an English edition whenever one matches the query. So "Mistborn" returned the 2023 "Mistborn" reissue (2023) instead of "The Final Empire" (2001), and "Children of Dune" returned "Children of Dune (Dune Chronicles, Book 3)" (1987). The matched edition's title, ISBNs, cover, language and year now win only when the edition is in another language and the query doesn't name the work's own title. Checked against live Open Library: "Children of Dune" and "Mistborn" give the v1.4.0 results again, and "Der Weg der Könige" and "Kinder des Nebels" still return the German editions. - **Google Books (#1434): the ISBN cache decorator is back on `search_by_isbn`.** It had moved to the new `search_paginated`. Failed searches were cached as empty pages, and ISBN lookups lost their cache. - **Notifications: Test notification shows the plugin's failure reason again.** Apprise 2 delivers on its own worker threads, even for a single service when that service has a deadline. `_AppriseLogCapture` filtered records by thread id and dropped all of them, so the UI only said "json: delivery failed". Apprise runs those workers in a copy of the caller's context, so the capture now filters on a context variable. That still keeps concurrent dispatches apart. `AppriseAsset(async_mode=False)` would not fix this, because the sequential path also uses the worker pool when a service has a deadline. - **Activity: `activity_update` no longer sends the raw task id to non-admins.** The owner's `user_<id>` room got the raw task id. For a request-linked download that id can hold a private tracker's passkey. It now goes through `viewer_download_id`, the same mapping `/api/status` and `download_progress` use. The admins room still gets the raw id. - **SearchBar (#1476): Escape and an outside click close the homepage's inline options panel again,** as they did in v1.4.0. - **genDebug.sh: packages are listed through `importlib.metadata`,** since #1404 removed pip from the image. The script also now redacts: - `PROWLARR_MAM_ID` - `user:pass@` credentials in URLs (proxies), in both the environment and the config files - notification route URLs, which carry their service's token (Telegram, Discord and others) - `make python-checks`, `make python-test`, `make frontend-checks` and `make frontend-test` pass on top of current `main`. - New tests cover: - Open Library edition selection - Google Books caching - log capture against a real Apprise 2 delivery to a local server that answers 500, plus a concurrent dispatch on another thread - the `activity_update` payload sent to each room - genDebug redaction and the package listing Each one fails on the old code. - I checked the SearchBar fix in the running app, because the frontend has no DOM test setup: Escape closes the panel, an outside click closes it, and a click inside keeps it open. - I ran the genDebug redaction under GNU sed (Debian image) and BSD sed. The package listing ran in the pip-less image. |
||
|
CaliBrain
|
a00f3d28bf |
fix(slskd): refuse unsafe peer paths, keep queued downloads alive, report search outages (#1512)
Fixes four defects in the Soulseek (slskd) A peer names its own files, and Shelfmark works out where slskd stored one from the last folder and the file name. `remote_directory_name` returned the whole directory when its last segment was empty. So `../../../etc//passwd.epub` or `/abs/path//book.epub` resolved outside the downloads folder, and Shelfmark imported whatever was there. For a multi-file release it moved those files into staging (`STAGE_MOVE`). - New `safe_remote_segments` refuses a name with an empty, `.` or `..` segment (which covers absolute and UNC paths), a NUL, or a drive anywhere but a Windows share root (`C:\Books\…`, which never reaches a local path). - Search (`iter_peer_files`, so `build_releases`) drops such files. - `download` refuses them before anything is queued, in both modes. The spec arrives with the queued release payload, so search filtering alone is not enough. - `local_path_for` also refuses a path that resolves outside the root (`resolve()` + `is_relative_to`), for example through a symlinked folder. `_poll`, used when isolation is off and as the fallback for slskd < 0.26, never asked for an activity grace. A file "Queued, Remotely" sent the same status and progress on every poll, so the orchestrator cancelled it after 300s and `SLSKD_QUEUE_TIMEOUT_MINUTES` never applied. It now requests, renews and releases the grace the same way isolated mode does. `search` caught every exception per query, so slskd being down or rejecting the API key showed as "no releases". When no query gets an answer it now raises `SourceUnavailableError`, as Prowlarr does, and the error reaches the user. One failed query among answered ones is still not an outage. A retry reuses the task's folder, and slskd renames a new file when one is already there. Files from a failed attempt therefore meant `_folder_complete`'s exact count could never match again. Now: - a failed or cancelled attempt removes its folder; - a new attempt removes any finished files an earlier one left (for example after a restart) before it queues the batch. `tests/slskd/conftest.py` stubbed `flask_socketio` whenever it had not been imported yet, even though it is installed. Running `pytest tests/slskd tests/` therefore broke the Flask app tests in the same worker (285 errors, also on |
||
|
CaliBrain
|
49aeec97cd |
fix(download): stop deleting user data and losing completed downloads (#1511)
Fixes six defects. Each one breaks downloads or deletes user data, and each has a regression test that fails on |
||
|
splitsec2andCaliBrain
|
520b6b16d0 |
fix(activity): Clear Completed leaves out an item that is no longer finished instead of failing the batch (#1508)
Clear Completed can fail outright with "Only terminal requests can be dismissed". `dismiss-many` answers 409 and dismisses nothing when any one request or download in the batch is not terminal. The list a user clears from can be older than the server, so a request that was reopened or retried since, or a download that started again, takes the whole batch down with it. The earlier fix for pending requests (#1438) filters on the client, which only helps when the list is current. `dismiss-many` now skips such an item and carries on with the rest. The response adds `skipped_item_keys` when something was left out, and `count` is what was actually dismissed. A single `dismiss` of a non-terminal item still answers 409. I hit it with a request whose download had completed while the request itself was still pending, but any reopen between the list and the click does it. This touches the same handler as #1507, so whichever lands second needs a small rebase. Tests: a batch with a finished and a reopened request dismisses the finished one and reports the other, and a batch with a finished and a running download does the same. Ruff, basedpyright and the backend suite pass (3933 passed, e2e excluded). Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
splitsec2
|
61c7ad3c62 |
Merge commit from fork
* fix(torrent): check each redirect hop when fetching a torrent file The torrent file fetch followed redirects by hand without looking at where a hop pointed, so a download link could make the server request a loopback, link-local or LAN address. Each hop (the initial URL and every redirect Location) must now be http or https and resolve only to global addresses. The configured Prowlarr/Newznab origin is exempt, as it usually lives on the LAN. The API key still only goes to that origin. * fix(socketio): only accept websocket handshakes from the app's own origin The Socket.IO server allowed every origin. It now uses the same-origin check python-engineio applies by default (Host, or X-Forwarded-Proto and X-Forwarded-Host behind a proxy). SOCKETIO_ALLOWED_ORIGINS takes a comma separated list of extra origins for proxies that change Host without forwarding it. * fix(custom-script): do not pass a file name that starts with a dash as the first argument In relative path mode the script got the organized file's relative path as its first argument, which many scripts read as an option when the name starts with a dash. Such a name is now passed with a ./ prefix. Absolute mode is unchanged. |
||
|
splitsec2
|
8ec9db49da |
Merge commit from fork
The request line sent to the book bot comes from the request-supplied source_id and was written to the socket as-is, so a line break in it sent extra IRC commands as the operator's nick. IRCClient._send, the single writer, now raises IRCError on any control character (CTCP 0x01 excepted), the download handler rejects such a request line with a clear error before connecting, and search refuses a query that contains one. |
||
|
splitsec2
|
6f65d6e20a |
Merge commit from fork
A release format such as /../../x was appended to the staging file name unchanged, so the direct-download and libgen handlers could write to, and on cancel unlink, a path outside TMP_DIR. Normalise the format to a plain extension at queue time and in build_filename, reject path-like source ids for the sources that use them as file names, and make both handlers resolve the staging path and refuse anything that is not a direct child of TMP_DIR. |
||
|
splitsec2
|
89663e32f6 |
Merge commit from fork
A user's personal notification routes were sent to any Apprise URL, from the server. Through the test endpoint or a saved route a signed-in user could make the server POST to loopback, link-local and LAN hosts, choosing the path and, with Apprise's + and : parameters, extra headers and body fields. User routes are now limited to the hosted services that talk to a fixed vendor host (discord, slack, tgram, pover, pbul) and to ntfy and gotify, whose host must resolve only to public addresses (checked on the host urlsplit reads and on the host Apprise reads, including IPv4-mapped and numeric forms). The check runs when a route is tested, when an event is queued and again when it is sent. Admin routes are unchanged. |
||
|
splitsec2
|
d34b200414 |
fix(activity): answer another user's item like a missing one, and skip foreign history rows (#1507)
Two small fixes in `activity_routes.py`. **Another user's item answers like a missing one.** Dismissing a task or request id answered 404 when it didn't exist and 403 when it belonged to someone else, so a user could test whether an arbitrary id exists. A non-admin now gets the same 404 for both. In dismiss-many a foreign key is reported with the missing ones and the rest carry on, and the owner is still written to the log. Admins are unchanged. **A foreign row in someone's viewer state no longer gives a 500.** If a non-admin's viewer state referenced another user's download or request row, building history raised a `RuntimeError`. The row is now skipped with a warning. Not changed: history still raises when a row that viewer state points to has been deleted. That is a different case, and I didn't want to widen this. Tests: the existing forbidden test now expects the 404 and the log context, and four new ones cover a foreign download and a foreign request answering like a missing one (including dismiss-many), an admin still dismissing a foreign row, and history skipping foreign rows. Ruff, basedpyright and the backend suite pass. |
||
|
splitsec2andCaliBrain
|
8fd477bf50 |
fix(releases): apply the request policy to release listing, not only to download (#1506)
The request policy is applied when a user downloads (`/api/releases/download`) or requests (`/api/requests`), but not when they list releases. A user whose mode is `request_book` or `blocked` can still search releases and list a torrent's files through the shared Prowlarr and qBittorrent credentials. Only the UI hides the list. This applies the same gate to `/api/releases`, `/api/releases/inspect` and the release-source records route, with the same codes as the download route (`policy_blocked`, `policy_requires_request`, `required_mode`). - `blocked` and `request_book` are refused. `request_release` is allowed, because that mode lists releases so the user can pick one. The UI opens the request dialog directly under `request_book` and never calls `/api/releases` there. - `/api/releases` resolves the mode per source for the content type. A refused source is dropped from the search, so a user blocked on Prowlarr no longer reaches Prowlarr even when other sources are allowed. It answers 403 only when no source is left, or when the one source asked for is refused. - The records route doesn't know the content type before it fetches, so it refuses only when every content type the source supports is refused. - Admins, auth mode `none` and an instance with requests off bypass the gate, as they do on the download route. A user whose default is `blocked` now gets a 403 toast in the release modal, where they used to get a list of "Unavailable" rows. I made no frontend change. Tests (14, new file) cover each route for blocked, request_book, request_release, an admin, and auth mode none, the per-source drop, and the records route. Ruff, basedpyright and the backend suite pass. Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
splitsec2
|
b05bbc8c58 |
fix(covers): key the cover cache by the image URL, not the caller-supplied id (#1505)
`/api/covers/<id>?url=...` cached the fetched image under the id in the path, and the caller chooses that id. A request with another book's id and a different URL put its image in that book's cache entry, and every user then got it. The key is now derived on the server from the image URL (a hash of it), and the id in the path stays only so links keep their shape. The SSRF checks and `fetch_and_cache` are unchanged, and `transform_cover_url` links already carry the URL, so they keep working. The URL is decoded first now, so a missing one is a 404 and bad encoding a 400, both before any fetch. I made the key the URL alone, not URL plus id. Either stops the overwrite, and the URL alone lets one image be shared when the same cover appears under two ids (the viewer redaction rewrites ids and still hits the cache). Entries cached under the old keys are not read any more, since there is no telling which were overwritten. They age out of the LRU and the cache warms once after the upgrade. Tests: a request with another id can't replace the legitimate image, in either order; the normal miss then hit; the same URL under two ids hits; no URL gives 404; bad encoding gives 400 without a fetch; and a unit test for the key. Ruff, basedpyright and the backend suite pass. |
||
|
splitsec2
|
71cd030484 |
Merge commit from fork
GET /api/users/me/edit-context served every user-overridable key with the admin's global value in globalValues and effective, so any user could read the global destination paths, the admin's EMAIL_RECIPIENT, the Booklore library and path ids and the global USER_NOTIFICATION_ROUTES table, whose Apprise URLs embed tokens. build_user_preferences_payload gains a for_self_service mode that replaces a global value with None unless the user set the key themselves or the field is a checkbox, number, select or multi-select switch the form needs to lay itself out (BOOKLORE_LIBRARY_ID and BOOKLORE_PATH_ID are masked despite being selects). The self-service route uses it; the admin routes keep full data. |
||
|
splitsec2
|
386fdb313c |
Merge commit from fork
GET /api/debug returns a bundle with application logs, process listings and environment names, and GET /api/restart exits the process. Both are registered only when DEBUG is on, but any logged-in user could call them. Add an admin_required decorator built on login_required and use it for both routes. Non-admin sessions get 403 "Admin access required"; authentication mode "none" keeps working as before. |
||
|
splitsec2
|
7da4669cc1 |
feat(prowlarr): hide ebook results above a maximum size (#1504)
Closes #844. A new Prowlarr setting, Maximum Ebook Size (MB). Ebook results above it are hidden from the release list. The default is 0, which shows everything as it does today. This is the idea from the issue. The filetype check suggested in the comments is a different change (#1490 has one for Prowlarr), and it doesn't catch a TV series or a software bundle that ships as a plain archive, which is what a size limit does. - A result with no reported size is always shown. - A result exactly at the limit is shown. - Audiobook searches are never limited, since they are big by nature. - A non-numeric value counts as no limit. The filter runs on the raw Prowlarr result before it is turned into a release, so a hidden result never reaches the sort or the release cache. Tests cover no limit, zero, over the limit, exactly at it, no size, audiobook searches and a bad value. Ruff, basedpyright and the backend suite pass. I added the entry to `docs/environment-variables.md` by hand because running the generator also changes unrelated parts of that file. |
||
|
splitsec2
|
f2269aa201 |
feat(audiobookbay): seed time and ratio settings (#1503)
Closes #1037. Two new AudiobookBay settings, Seed Time (minutes) and Seed Ratio. Both default to 0, which leaves the torrent client's own limit as it is today, so nothing changes until someone sets them. They go to the client with the magnet the same way the Prowlarr seed preferences do (`DownloadRequest.seeding_time_limit` and `ratio_limit`), so they apply with the clients that already take per-torrent limits there: qBittorrent, Transmission and Deluge. A retry that reuses the saved magnet carries them too. A value that isn't a positive number (text, negative, NaN, inf) is ignored and the client's limit stays. Tests cover reading both settings (unset, zero, each on its own, text values, bad values) and that both the fresh and the cached-magnet request carry them. Ruff, basedpyright and the backend suite pass. I haven't tried it against a live client, since the client side is the code Prowlarr already uses. I added the two entries to `docs/environment-variables.md` by hand. Running `scripts/generate_env_docs.py` here also changes unrelated parts of that file (the Soulseek table of contents link and the `SHELFMARK_API_KEY_READONLY` entry), so I left those alone. |
||
|
splitsec2andCaliBrain
|
77b80eb31a |
fix(metadata): only an admin may use the provider's own account, its lists and shelves (#1501)
Refs #981. Hardcover is signed in as one account, the instance owner's, and every logged-in user shares it. A non-admin can see the owner's lists and shelves in the List filter, browse one by asking for it, and add or remove books and change read status from search results, which is what #981 describes. Now only an admin may. With authentication set to none the one user keeps everything. - `GET` targets and the batch lookup return no options for a non-admin, so the buttons don't show and the account isn't asked. - `PUT` targets answers 403. - The List filter (`hardcover_list`) has no options for a non-admin, and a request that asks for it has the field dropped before the search. Providers name these fields in `account_search_fields`, and Hardcover lists `hardcover_list`. - Public things are unchanged: author, title and series suggestions, and a pasted list URL in the search box. That pasted URL is the one gap I know of. It resolves any list the account can see, so a user who knows the slug of one of the owner's private lists could still open it. Closing it means the provider has to know who is asking, which is a bigger change than this one, so I left it. Not included from the issue: a toggle for letting users view lists and limiting which lists they see. I kept this to the admin-only part so it stays small, and I can do the toggle as a follow-up if you want it. Tests cover the routes as an admin, a user and with no authentication (targets read, batch and write, list options, a crafted list search, and public fields still passing). Ruff, basedpyright and the backend suite pass. Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
splitsec2andCaliBrain
|
82c8c91c8b |
refactor(clients): read the stall window with a type check instead of a pyright ignore (#1498)
The only `pyright: ignore[reportArgumentType]` in `shelfmark/` is one I added in #1426, and I was being lazy when I did. I hit the same error on another PR and replaced the ignore there with a real check, so this does the same for the original. `_stall_window_seconds` reads the setting as an `object`. It now narrows to int, float or str before calling `float()`, and a bad string falls back to the default as before. A non-numeric value of another type (None, a list) falls back to the default too, which used to happen through a `TypeError`. No behaviour change. The existing range test still passes and I added `None` and a list to it. basedpyright is at 0 errors, ruff is clean and the backend suite passes. Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
splitsec2
|
e83bab9337 |
feat(clients): give up on a torrent with no seeders (#1494)
A torrent with no seeds won't finish. It runs on the pieces its few
peers hold and then sits still, and nothing says so until the stall
timer ends it. The card only reads "downloading". I run AudiobookBay
releases, which show no seed count in the listing, so the client is the
only place to find out.
Two new settings, both off until you turn them on:
`TORRENT_NO_SEEDERS_WAIT` (minutes, 0 turns it off) and
`TORRENT_MIN_SEEDERS` (default 1).
- A torrent that has fewer seeds than the minimum for the whole wait,
while nothing is downloading, is removed from the client with its data
and the task ends as an error.
- A torrent with no seed at all for the wait is given up on even while
it still moves.
- A torrent that met the minimum and lost seeds down to one, but is
still moving, keeps going.
- The card shows the seed count beside the progress, with a countdown
while it waits ("0 seeds, giving up in 4m").
qBittorrent only, because it reports connected and tracker seed counts.
Other clients report none and are left to the stall timer, and an
unknown count is not read as zero. A torrent that was already in the
client when the download started is never removed, and a queued torrent
isn't judged.
Tests cover the wait and countdown, the clock restarting on progress,
the tracker count, a higher minimum, off by default, a client with no
counts, a preexisting torrent, a queued torrent, and the moving cases.
Ruff and the backend suite pass.
|
||
|
splitsec2
|
9ba5816b83 |
fix(requests): an older failed attempt no longer undoes a later delivery (#1493)
A request that was retried after a failure has two tasks in the status built from download history. `sync_delivery_states_from_queue_status` walks the status buckets in `QueueStatus` order and keeps the last one it sees for a request. Error comes after complete, so a request that was delivered by its retry is set back to error and, when no retry is left, reopened as pending. The next pass can then fetch the book again. When a request has more than one task, an attempt that is still running now decides its state, else a completed one, else the failure. Ties keep the old order, so a request with a single failed task still reopens as before. Tests: an earlier failure doesn't undo a later delivery, a running retry outranks an earlier failure, and a lone failure still reopens. It happened on my instance to a request whose first try failed and whose retry succeeded. Ruff and the backend suite pass. |
||
|
splitsec2
|
a1ac017487 |
feat(requests): let an admin reopen a rejected request (#1491)
Closes #1445, where you said you were open to it. A rejected request was final. `update_request` won't move a terminal status and the admin routes only fulfil or reject, so an admin who changed their mind had to ask the user to request the book again, and the user kept seeing the old rejected one. `POST /api/admin/requests/<id>/reopen` moves a rejected request back to pending and clears the review fields. What the user asked for (request level, release, note) is kept. It refuses anything that isn't rejected, and refuses when that user already has a pending request for the same book. Admins get a Reopen action on rejected cards in the activity sidebar and in History. A rejected request has usually been cleared from the list by the time anyone wants it back, so reopening also clears the dismissal, the way the failed-download reopen does. Without that the pending request stays hidden. Tests cover the service (state change, what is kept, the refusals, a non-string admin note) and the route, and the card model for the new action. Ruff, tsc, oxlint, vitest and the backend suite pass. It has been running on my instance since Oct 8. |
||
|
splitsec2
|
88cc734db2 |
fix(activity): a reused release follows the new request, and stale rows of a delivered request are closed (#1496)
Two things that go wrong when a second request picks a release a first request already used. A task id comes from the release, so the history row is reused. **The row stays with the first request.** Recording a download kept the first request's id, notes and state. On my instance a card for one book showed up under the request for a different book, with that request's note on it. Recording a download now moves the row to the request that queued it and restarts its queue time. A retry with no request keeps the request the row had. **Old rows of a delivered request show as failures forever.** The startup sweep leaves an old request-linked row alone so its retry stays, but one whose request already has its book (delivered, or another download of it finished) is now closed as "Superseded by a completed download" instead of showing as an interrupted failure for ever. Also in the first commit: History cards used the saved cover URL as it was, an external address the page's own-origin image policy blocks, while running downloads use the cover proxy. They use it too. The second commit is about restarts. After a restart nothing is in flight, but a request in downloading, resolving or locating whose row had moved to another request was stranded, because only queued was reopened. Those states are reopened too. Tests cover the row takeover and the retry case, the cover through the proxy (and a missing cover), the superseded cases (delivered request, another row finished, an undelivered request is left alone) and the in-flight states with no row. Ruff and the backend suite pass. |
||
|
splitsec2
|
06a67a7150 |
feat(download): a concurrency limit per source (#1495)
With one global limit, a few tasks that are only waiting hold every slot. A Soulseek download sitting in a peer's queue is the example I hit: three of those and a torrent that could start waits behind them. `MAX_CONCURRENT_DOWNLOADS_PER_SOURCE` (0 keeps the single global limit, which is the default). With it set, the coordinator starts a download only when its source is below the limit and passes over tasks of a full source without losing their place in the queue. `get_next` gets an optional filter for this, and a cancelled task is still dropped when one is given. The start-up stagger now applies only between downloads from the same source, and the pool is sized so every source can reach its limit. The second commit: qBittorrent keeps its own queue of torrents waiting for a free slot. The task polling one of those does no work, so it no longer counts against its source's limit and the next task of that source can start. The client does the pacing, as it does for its own queue. Tests cover the queue pass-over (order and priority kept, nothing eligible, cancelled tasks), the coordinator (a full source doesn't block the others, each source has its own limit, the global limit when unset) and the client-queue case. Ruff and the backend suite pass. |
||
|
splitsec2
|
3b5afb1a04 |
Merge commit from fork
PUT /api/users/me set a new password from the password field alone, so a live session could replace the password and lock the owner out. A builtin user now has to send current_password with it, checked against the stored hash. A wrong or missing one is a 400, and counts toward the same lockout as a failed login (a locked account gets a 429 without being checked), so the endpoint cannot be used to guess the password. A password or current_password that is not a string is a 400. A password that was not a string raised a TypeError from len() and answered 500. The account form asks for the current password when a new one is typed. Profile and settings changes without a password are unchanged. |
||
|
CaliBrain
|
10710abd4e |
fix(download): send every file of a multi-file download to the browser (#1489)
Download to Browser only ever sent the first file of a multi-file release (audiobook chapters, packs): the folder output returned final_paths[0] and /api/localdownload served that one path. The folder output now keeps every final path on the in-memory task (download_paths), and /api/localdownload streams them as one uncompressed ZIP named after the book when there is more than one. The archive is never staged on disk or held in memory, and ZIP64 covers large audiobooks. History still stores only the first path, so a re-download from history after a dismiss or restart serves a single file, as before. Also brings every linter back to green on all files: - pre-commit: bump the ruff hook to v0.17.1 to match uv.lock; ruff-format the Python examples in docs/dev and skills - ruff: fix the remaining issues in skills/shelfmark/download_books.py; format base_handler.py - basedpyright: Optional and attrs errors in databazeknih.py - knip: drop 31 unused exports, reuse ColumnColorHint in colorMaps - oxfmt 0.72: styles.css - direct download: fix the "Welib, mirrors" typo and the test that still expected the pre-OceanofPDF message Fixes #1467 |
||
 
|
c8d66db22e |
feat(direct_download): add OceanofPDF as a standalone search and download provider (#1485)
Closes #1149 ## What this adds OceanofPDF as a second Direct Download provider, alongside Anna's Archive. Unlike Z-Library, LibGen and Welib, which only download files Anna's Archive found, OceanofPDF searches and downloads on its own. It works with or without an AA mirror configured; with both, one search queries both and the results are merged. It plugs into the provider registry from #1337, so search, record routing and the download handler need no changes. **Search:** WordPress `/?s=` search, "title author" first, then the bare title if that finds nothing. One release per advertised format (`[PDF] [EPUB]`), with title, author and cover. Shelfmark's supported-format filter applies as usual. **Download** follows the site's three hops: 1. The book page has one POST form per format, targeting `Fetching_Resource.php`. 2. That returns an interstitial whose `<meta http-equiv="refresh">` points at a signed, expiring file URL. 3. The file URL serves the book. Pages go through `html_get_page` and its Cloudflare bypass. The form POST replays the clearance the bypasser stored, and re-solves once if challenged. Bypassers only replay GETs, so it can't be handed over directly. **Settings:** - `OCEANOFPDF_MIRROR_URLS` under Mirrors. Only the first URL is used. - An OceanofPDF toggle under Download Sources → Slow downloads. It's locked until a mirror is set and while the Cloudflare bypass is off. Its position in the list doesn't matter, and its description says so. - A saved list from before this change has no OceanofPDF entry. The settings page shows a missing option as off, so the provider treats it as off too, keeping the page and the behaviour in agreement. New installs default to on. ## Fixes found along the way - **`challenge_marker` rejected real pages behind Cloudflare** (separate commit, `fix(bypass)`). Cloudflare's bot-detection script injects `/cdn-cgi/challenge-platform/scripts/jsd/main.js` into every page it serves, after load. Bypassers return the rendered page, so a successful FlareSolverr solve of any page under 64 KB came back carrying `/cdn-cgi/challenge-platform` and was rejected as unsolved. The detector now ignores that one script path. Real challenge pages load from `/cdn-cgi/challenge-platform/h/…` and keep their other markers, so they are still caught. This also affects any other Cloudflare-fronted host, not just OceanofPDF. Happy to split it into its own PR if you prefer. - **Brotli on a plain `requests` call.** Copying `DOWNLOAD_HEADERS` into the form POST offered `br`. Cloudflare then answered in Brotli, which `requests` can't decode without the optional `brotli` package. The POST now leaves `Accept-Encoding` to `requests`. If a response still has no file link, its status, encoding, title and challenge marker are logged. ## Testing - **New tests:** 41 in `tests/direct_download/test_oceanofpdf.py` and 4 in `tests/bypass/test_cloudflare_beacon.py`. They cover search parsing, format and language filtering, form selection, the interstitial, the download flow (including a challenged POST, an HTML stub served as the file, and cancellation), the toggle, and registry wiring. Fixtures mirror the live site's markup. - **Existing tests:** `tests/direct_download`, `tests/bypass`, `tests/config`, `tests/core` and `tests/download` show no new failures against `main`. Some tests in those folders can't run on Windows; those fail identically on both. - **Lint:** `ruff check` and `ruff format --check` are clean on every changed file. - **Live:** verified end to end on a v1.4.0 Docker deployment behind FlareSolverr (external bypasser), with only an OceanofPDF mirror configured: search, then the book page, the form POST and the completed download. ## Known limits - OceanofPDF doesn't label languages, so its results always pass a language filter. - PDFs only appear when `pdf` is in the supported formats. - Each query reads the first results page only (roughly 7–10 books). - Downloaded files carry OceanofPDF's own tags/watermark, as noted in #1149. They aren't stripped. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Adam Stone <196691089+ast0ne1@users.noreply.github.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
amura11andClaude Sonnet 5.5
|
51f32b374d |
feat(packs): let users choose which files, books and formats to download (#1480)
## Summary - **Skip files or books in a multi-book pack.** The pack review panel gets a switch per file and a "Skip this book" switch per book, using the shared `ToggleSwitch`. A partly skipped book shows e.g. "3 of 5 files" in the Files column. A fully skipped book is greyed out. - **Choose formats for a single book.** A new "Choose formats" header toggle, next to the multi-book one, opens a format picker for a release that has several book formats (EPUB, MOBI, PDF and so on). It has one switch per format with file count and size. Releases with one format, or that can't be inspected, download as before. - **Skipped files are sent as `ignored_files`.** They are stored on the task (including restart-safe retry) and removed before grouping and transfer. This matters for packs because files missing from `book_plan` would otherwise be filed by the heuristic fallback. Ignored entries match by release-relative path, then path suffix, then unique basename. - A pack that ends up with one book after skips keeps the titles you reviewed instead of falling back to the searched title. ## Screenshots <img width="1277" height="413" alt="Screenshot 2026-10-19 at 09-39-28 Shelfmark" src="https://github.com/user-attachments/assets/c005ca4b-3dac-45dd-b2ff-d02fc0f9fcc9" /> <img width="1277" height="1210" alt="Screenshot 2026-10-19 at 09-40-23 Shelfmark" src="https://github.com/user-attachments/assets/2145f3ba-e5fc-4635-9364-1790feb0c8b8" /> <img width="1277" height="748" alt="Screenshot 2026-10-19 at 09-39-03 Shelfmark" src="https://github.com/user-attachments/assets/39023fac-70e3-46d8-af89-1160ebd6e415" /> ## Test plan - [x] `pytest` pack tests (new `drop_ignored_files` cases added) - [x] `vitest`, `tsc`, `oxlint`, `oxfmt`, `ruff` - [x] Manual check in the UI with a real multi-book pack 🤖 (mostly) Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
CaliBrain
|
3c5cfcb77d |
fix/redact log secrets (#1488)
- fix(logging): mask tracker API keys in log output (#1486) - fix(welib): follow slow_download links through the countdown (#1447) |
||
  
|
bb1655238a |
feat(slskd): add Soulseek (slskd) release source, building on #1335 (#1428)
## Add Soulseek (slskd) as a release source, building on #1335 This is @dskvr's work from #1335 rebased onto current `main`, with my changes on top. #1335 has been quiet since 2026-09-13 and I needed the changes below, so I carried it forward instead of starting over. Their two commits are the first two in this branch with their authorship intact, and the source plugin, handler, API client, settings tab, tests, `ReleaseProtocol.SOULSEEK`, test client compose file and frontend bits are all theirs. I haven't changed how a search turns into releases beyond what is listed here. If @dskvr would rather keep #1335 alive and take these as commits on their branch, I can do that instead and close this. ### What I changed **Downloads survive other apps using the same slskd.** The handler found its files by listing slskd's downloads, and another app on the same instance (SoulSync here) sends `DELETE /transfers/downloads/all/completed` after each of its batches. Once that ran, a finished download no longer appeared in the list and the task never saw it complete, even though the files were on disk. With the new `SLSKD_ISOLATE_DOWNLOADS` setting (on by default) each task: - enqueues one batch (`POST /transfers/downloads/batches`) into its own folder, `<SLSKD_DESTINATION_PREFIX>/<task id>` - polls each transfer by id, which slskd still answers after the list has been cleared - decides it is done from the files on disk (sizes, not names), with the `/events` log as a fallback when the transfers are gone and the files aren't visible yet - copies out of that folder, and removes it only if `SLSKD_KEEP_COMPLETED` is off and the import worked With isolation off the handler behaves as it did in #1335. If the batch endpoint isn't there (404/405) it falls back to the original per-file enqueue. **Source changes.** m4b/m4a files are their own release next to loose chapter files, so auto-download can prefer one file over a chapter set. Results from the query variants are merged per peer. The folder text is exposed as `extra["author"]` and `extra["title_raw"]`, which is what auto-download's strict match and pack detection read. `is_available()` also needs an API key now. **Smaller fixes.** Cancelling while files are still being located now removes the transfer records it created. In the settings tab, a cleared form field is no longer replaced by the saved value when testing the connection. The new settings are in `docs/environment-variables.md`. ### Testing `tests/slskd` has 192 passing and 6 live tests that skip without an instance. I wrote the new tests first and checked the handler and source rules by breaking them and confirming a test failed. Ruff is clean. On the whole suite the only failures are 4 in `tests/download` that import `seleniumbase`, which isn't installed in my environment, and they fail the same on `main`. I've been running this on my own deployment against slskd 0.26.0 behind a VPN, with SoulSync active on a second instance. A search returned a couple hundred results from about 70 peers, an epub came down and was imported, and auto-download fetched five audiobooks as single m4b files. That is a day of use, so I don't know how it behaves on other slskd versions or on Windows path mappings. --------- Co-authored-by: sandwich <dskvr@users.noreply.github.com> Co-authored-by: sandwich <299465+dskvr@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Amalkrishna A S
|
b38e4e4a00 |
feat(naming): add {AuthorSort} template variable (#1484)
Closes #1178. Adds `{AuthorSort}`, the first author surname-first ("Arthur Conan Doyle" → "Doyle, Arthur Conan"), for rename and organize templates, alongside `{FirstAuthor}`. - A value already in "Last, First" form (one word before the comma) is kept as is; Jr/Sr/II/III/IV stay at the end. - Known limits, same as calibre's default: the last word is treated as the surname, so particles ("Le Guin") and surname-first names aren't detected. - Added to the settings help, env docs, the frontend variable picker/preview, and tests. |
||
 misko903andClaude Opus 5.5
|
b83e35ee0a |
feat(metadata): add Databáze knih metadata provider (#1479)
## What Adds [Databáze knih](https://www.databazeknih.cz), the Czech and Slovak community book catalog, as a metadata provider. It follows the Moly.hu provider (#1172): HTML scraping, no API key, disabled by default, and a settings tab with an enable checkbox and a Test Connection button. ## Why Hardcover, Open Library and Google Books cover Czech and Slovak editions poorly. Databáze knih is the de facto catalog for them, including Czech and Slovak translations of foreign works. With this provider, Universal mode works end to end for Czech and Slovak titles: the localized title and author from Databáze knih feed the release search. ## How - HTML scraping with BeautifulSoup, which is already a dependency. Book pages carry schema.org JSON-LD, so most fields come from structured data rather than CSS selectors. - **Title search** uses `/vyhledavani/knihy`, with the site's pagination (`page`) and language filter (`lang` for cs/sk/en/de). A search with a single match redirects straight to the book page, and that case is handled. - **Author search:** the site search only matches titles, so an author-only search finds the author (accent- and word-order-insensitive) and lists their books page, which is also paginated. Title + author filters the title results on the author. - **ISBN search** follows the site's redirect from `/search?q=<isbn>` to the book page. - **`get_book`** reads the full description (the JSON-LD one is truncated), ISBN-10/13, publisher, edition year, language, genres, cover (placeholder images dropped), series with id and position, and rating and rating count. It also reads the page count and original title from the more-info fragment. - The `RateLimiter` from Open Library (30 requests/min) and the standard `@cacheable` decorators. Fetch failures return `None`, so they are not cached. - `docs/environment-variables.md`: added the `DATABAZEKNIH_ENABLED` section. ## Testing - `tests/metadata/test_databazeknih_parse.py`: offline tests with shortened copies of real pages. They cover search parsing, pagination and language parameters, redirect-to-book, author listing, title + author filtering, uncached failures, book details, ISBN resolution and helpers. - `uv run pytest tests/metadata` is green (89 passed), and `ruff check` and `ruff format --check` are clean. - Verified live against databazeknih.cz: connection test, title search (pages 1 and 2), author books, title + author, `get_book`, and ISBN lookup. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
  sean0407andSean Li
|
a0ffb268b2 |
fix: wait for synced files before processing completed downloads (#1464)
## Summary
```text
wait for completed download path
regular file -> keep existing immediate handling
directory -> inspect files enabled for this task's content type
wait until the supported-file signature is stable across checks
```
Fixes the remote-sync race in #1463. A mapped torrent folder can appear
before its large media file finishes copying, so path existence alone
does not mean the download is ready.
## Evidence
- **Before:** In #1463, qBittorrent reported a multi-file audiobook
complete while the destination contained only `.cue` and `.jpg`;
Shelfmark immediately post-processed the folder and failed with “No book
files found in download.” Retrying after the `.m4b` sync finished
imported it.
- **After:** A regression test starts with the cue sheet and cover
present, then simulates the `.m4b` file appearing and growing. Path
waiting returns only after the supported file's size and modification
time remain stable across checks.
- Credential-free isolated `python -m pytest
tests/prowlarr/test_remote_path_mappings.py -q -n 0 -o addopts=''`: 11
passed.
- `ruff check shelfmark/download/clients/base_handler.py
tests/prowlarr/test_remote_path_mappings.py`: passed.
## Merge Danger
**Door:** Two-way
**Blast Radius:** Download handling
The wait uses the existing completed-path timeout and leaves regular
single-file downloads on their current path.
Co-authored-by: Sean Li <22557931+sean0407@users.noreply.github.com>
|
||
|
splitsec2
|
0ade063db5 |
fix(download): extract RAR archives in place, not in the IO threadpool (#1459)
Fixes #1458. A RAR download fails with `Download failed: TypeError`, and the log says `child watchers are only available on the default loop`. rarfile extracts by running `unrar` as a subprocess, and `extract_archive_files` calls it through `run_blocking_io`, so it runs in gevent's IO threadpool. gevent only lets a subprocess start on the thread that owns the default loop. `run_blocking_io` already avoids the threadpool for `subprocess.run`, but a RAR gets to the subprocess through rarfile, so nothing catches it. This extracts a RAR on the calling greenlet, where the patched subprocess waits for `unrar` without blocking anything else. ZIP is pure Python and stays in the threadpool. I checked it with a real RAR4 file on a build of main with the stock image's `unrar`: from the threadpool it raises the TypeError, in place it extracts all 7 files. Two tests. One checks that a RAR doesn't go through `run_blocking_io`. The other runs under real gevent patching with an extractor that starts a process the way rarfile does. Both fail without the change, and the full suite passes. I keyed it on the `.rar` suffix to keep it small. Teaching `run_blocking_io` about rarfile's subprocess would be the more general fix, but that's a bigger change to a helper a lot of code goes through, so I left it for you to say if you want it. |
||
 ultraviolet150
|
5f3647a496 |
fix(clients): wait for qBittorrent-compatible clients to finish before post-processing (#1465)
## Summary
`QBittorrentClient.get_status()` treats a torrent as complete as soon as
`progress >= 1.0`, excluding only the `moving` state. Some
qBittorrent-compatible clients report full progress before they are
finished. rdt-client (v2.0.142) reports `progress = 1.0` once the debrid
and local downloads are done, but keeps `state = "downloading"` for a
few seconds while it finalises the torrent, and until then reports a
guessed `content_path`. It only switches to `pausedUP` with the real
path afterwards.
Shelfmark picks the torrent up inside that window, so post-processing
runs against a path that isn't there yet and fails within milliseconds
("No book files found" / `post-processing failed`). Clicking Retry then
imports it fine. Every rdt-client download hits this.
## Change
- Add `_INCOMPLETE_STATES` (all download-side states, plus `moving`) and
only mark a torrent complete when `progress >= 1.0` **and** its state is
not one of those.
- Real qBittorrent leaves these states when a torrent reaches 100%, so
its behaviour is unchanged.
## Example log (before)
```
12:04:43,215 base_handler - Remapped download path for qbittorrent (<hash>): <rdt-client download dir>\<torrent name>\ -> /downloads/<torrent name>
12:04:43,216 orchestrator - download finished; starting post-processing
12:04:43,232 orchestrator - WARNING - post-processing failed
```
## Testing
- New test
`test_get_status_full_progress_still_downloading_not_complete`: fails
before the change, passes after.
- `tests/prowlarr/`: all passing. `ruff check`, `ruff format --check`
and `basedpyright` clean on the changed files.
- Running the patched client against rdt-client: downloads now import
without Retry.
Related to #1463 / #1464, which handle the same symptom when the client
really is finished but files are still syncing to local disk. This PR
addresses a different cause: the client reports 100% progress while
still in a downloading state, so Shelfmark shouldn't treat it as
complete yet. The two changes are independent and complementary. Not
covered by #741, which fixed add-time failures. Possibly related: #607.
|
||
  Matt Van HornandMatt Van Horn
|
d3bcf0674d |
fix(direct_download): Anna's Archive search 404 on ISBN queries (#1456)
When ISBN filters are present, the search query is now the deduplicated bare ISBN strings, with the free-text query appended only if it is not already one of those tokens. Searches with no ISBN still send the quoted free-text query alone. An Anna's Archive lookup that includes an ISBN comes back as 404 Not Found, so the direct-download release search fails and the book cannot be retrieved. The failing request puts an isbn13:/isbn10: boolean expression in the q parameter. The direct-download search query tests passed. Closes #1444 Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> |
||
|
splitsec2
|
84d1d7196c |
fix(irc): pin the TLS context to TLS 1.2 or newer (#1455)
CodeQL on my fork flagged `ssl.create_default_context()` in the IRC client as allowing TLS 1.0 and 1.1 (`py/insecure-protocol`, rated high). In practice the shipped image is already fine. Python 3.10+ sets a TLS 1.2 floor on the default context, and on the current image (Python 3.14.7, OpenSSL 3.5.7) `minimum_version` comes back as TLS 1.2. An OpenSSL config on the host can still lower that floor, so this pins it with one line, `context.minimum_version = ssl.TLSVersion.TLSv1_2`. Added `tests/irc/test_client_tls.py`. It drops the floor to the minimum before `connect()` runs, then checks that `connect()` raised it back to 1.2. It fails without the change. Certificate and hostname verification are still off (`CERT_NONE`). I left that alone because of the comment about self-signed certs on IRC servers. Turning verification on, or making it a setting, would be the bigger fix, but that is ultimately a call for @calibrain if you want a PR for it. |
||
 Som Samantray
|
2b8e0da2f3 |
fix(direct-download): clarify required search mirror (#1454)
Direct Download now tells users who configure only a Z-Library mirror that they need an Anna's Archive mirror for search. Z-Library, LibGen, and Welib mirrors download results found through Anna's Archive. Fixes #1408. Validation: the Direct mode browser check, 18 focused tests, Ruff, formatting, and backend/test type checks passed. The CI-style Python suite passed 3,569 tests; `tests/core/test_hardlink.py::TestSameFilesystem::test_string_paths` was excluded because this container reports different `st_dev` values for a file and its parent directory. AI disclosure: OpenAI Codex assisted with diagnosis, implementation, and validation. |