Commit Graph
279 Commits
Author SHA1 Message Date
splitsec2 8ec9db49da Merge commit from fork
The request line sent to the book bot comes from the request-supplied
source_id and was written to the socket as-is, so a line break in it sent
extra IRC commands as the operator's nick. IRCClient._send, the single
writer, now raises IRCError on any control character (CTCP 0x01 excepted),
the download handler rejects such a request line with a clear error before
connecting, and search refuses a query that contains one.
2026-10-20 23:40:33 -04:00
splitsec2 6f65d6e20a Merge commit from fork
A release format such as /../../x was appended to the staging file name
unchanged, so the direct-download and libgen handlers could write to, and on
cancel unlink, a path outside TMP_DIR. Normalise the format to a plain
extension at queue time and in build_filename, reject path-like source ids for
the sources that use them as file names, and make both handlers resolve the
staging path and refuse anything that is not a direct child of TMP_DIR.
2026-10-20 23:40:16 -04:00
splitsec2 89663e32f6 Merge commit from fork
A user's personal notification routes were sent to any Apprise URL, from the server. Through the test endpoint or a saved route a signed-in user could make the server POST to loopback, link-local and LAN hosts, choosing the path and, with Apprise's + and : parameters, extra headers and body fields.

User routes are now limited to the hosted services that talk to a fixed vendor host (discord, slack, tgram, pover, pbul) and to ntfy and gotify, whose host must resolve only to public addresses (checked on the host urlsplit reads and on the host Apprise reads, including IPv4-mapped and numeric forms). The check runs when a route is tested, when an event is queued and again when it is sent. Admin routes are unchanged.
2026-10-20 23:39:46 -04:00
splitsec2 d34b200414 fix(activity): answer another user's item like a missing one, and skip foreign history rows (#1507)
Two small fixes in `activity_routes.py`.

**Another user's item answers like a missing one.** Dismissing a task or
request id answered 404 when it didn't exist and 403 when it belonged to
someone else, so a user could test whether an arbitrary id exists. A
non-admin now gets the same 404 for both. In dismiss-many a foreign key
is reported with the missing ones and the rest carry on, and the owner
is still written to the log. Admins are unchanged.

**A foreign row in someone's viewer state no longer gives a 500.** If a
non-admin's viewer state referenced another user's download or request
row, building history raised a `RuntimeError`. The row is now skipped
with a warning.

Not changed: history still raises when a row that viewer state points to
has been deleted. That is a different case, and I didn't want to widen
this.

Tests: the existing forbidden test now expects the 404 and the log
context, and four new ones cover a foreign download and a foreign
request answering like a missing one (including dismiss-many), an admin
still dismissing a foreign row, and history skipping foreign rows. Ruff,
basedpyright and the backend suite pass.
2026-10-20 23:39:25 -04:00
splitsec2andCaliBrain 8fd477bf50 fix(releases): apply the request policy to release listing, not only to download (#1506)
The request policy is applied when a user downloads
(`/api/releases/download`) or requests (`/api/requests`), but not when
they list releases. A user whose mode is `request_book` or `blocked` can
still search releases and list a torrent's files through the shared
Prowlarr and qBittorrent credentials. Only the UI hides the list.

This applies the same gate to `/api/releases`, `/api/releases/inspect`
and the release-source records route, with the same codes as the
download route (`policy_blocked`, `policy_requires_request`,
`required_mode`).

- `blocked` and `request_book` are refused. `request_release` is
allowed, because that mode lists releases so the user can pick one. The
UI opens the request dialog directly under `request_book` and never
calls `/api/releases` there.
- `/api/releases` resolves the mode per source for the content type. A
refused source is dropped from the search, so a user blocked on Prowlarr
no longer reaches Prowlarr even when other sources are allowed. It
answers 403 only when no source is left, or when the one source asked
for is refused.
- The records route doesn't know the content type before it fetches, so
it refuses only when every content type the source supports is refused.
- Admins, auth mode `none` and an instance with requests off bypass the
gate, as they do on the download route.

A user whose default is `blocked` now gets a 403 toast in the release
modal, where they used to get a list of "Unavailable" rows. I made no
frontend change.

Tests (14, new file) cover each route for blocked, request_book,
request_release, an admin, and auth mode none, the per-source drop, and
the records route. Ruff, basedpyright and the backend suite pass.

Co-authored-by: CaliBrain <calibrain@l4n.xyz>
2026-10-20 23:38:40 -04:00
splitsec2 b05bbc8c58 fix(covers): key the cover cache by the image URL, not the caller-supplied id (#1505)
`/api/covers/<id>?url=...` cached the fetched image under the id in the
path, and the caller chooses that id. A request with another book's id
and a different URL put its image in that book's cache entry, and every
user then got it.

The key is now derived on the server from the image URL (a hash of it),
and the id in the path stays only so links keep their shape. The SSRF
checks and `fetch_and_cache` are unchanged, and `transform_cover_url`
links already carry the URL, so they keep working. The URL is decoded
first now, so a missing one is a 404 and bad encoding a 400, both before
any fetch.

I made the key the URL alone, not URL plus id. Either stops the
overwrite, and the URL alone lets one image be shared when the same
cover appears under two ids (the viewer redaction rewrites ids and still
hits the cache).

Entries cached under the old keys are not read any more, since there is
no telling which were overwritten. They age out of the LRU and the cache
warms once after the upgrade.

Tests: a request with another id can't replace the legitimate image, in
either order; the normal miss then hit; the same URL under two ids hits;
no URL gives 404; bad encoding gives 400 without a fetch; and a unit
test for the key. Ruff, basedpyright and the backend suite pass.
2026-10-20 23:35:42 -04:00
splitsec2 71cd030484 Merge commit from fork
GET /api/users/me/edit-context served every user-overridable key with the
admin's global value in globalValues and effective, so any user could read
the global destination paths, the admin's EMAIL_RECIPIENT, the Booklore
library and path ids and the global USER_NOTIFICATION_ROUTES table, whose
Apprise URLs embed tokens.

build_user_preferences_payload gains a for_self_service mode that replaces
a global value with None unless the user set the key themselves or the
field is a checkbox, number, select or multi-select switch the form needs
to lay itself out (BOOKLORE_LIBRARY_ID and BOOKLORE_PATH_ID are masked
despite being selects). The self-service route uses it; the admin routes
keep full data.
2026-10-20 23:35:02 -04:00
splitsec2 386fdb313c Merge commit from fork
GET /api/debug returns a bundle with application logs, process listings
and environment names, and GET /api/restart exits the process. Both are
registered only when DEBUG is on, but any logged-in user could call them.

Add an admin_required decorator built on login_required and use it for
both routes. Non-admin sessions get 403 "Admin access required";
authentication mode "none" keeps working as before.
2026-10-20 23:34:32 -04:00
splitsec2 7da4669cc1 feat(prowlarr): hide ebook results above a maximum size (#1504)
Closes #844.

A new Prowlarr setting, Maximum Ebook Size (MB). Ebook results above it
are hidden from the release list. The default is 0, which shows
everything as it does today.

This is the idea from the issue. The filetype check suggested in the
comments is a different change (#1490 has one for Prowlarr), and it
doesn't catch a TV series or a software bundle that ships as a plain
archive, which is what a size limit does.

- A result with no reported size is always shown.
- A result exactly at the limit is shown.
- Audiobook searches are never limited, since they are big by nature.
- A non-numeric value counts as no limit.

The filter runs on the raw Prowlarr result before it is turned into a
release, so a hidden result never reaches the sort or the release cache.

Tests cover no limit, zero, over the limit, exactly at it, no size,
audiobook searches and a bad value. Ruff, basedpyright and the backend
suite pass. I added the entry to `docs/environment-variables.md` by hand
because running the generator also changes unrelated parts of that file.
2026-10-20 23:33:43 -04:00
splitsec2 f2269aa201 feat(audiobookbay): seed time and ratio settings (#1503)
Closes #1037.

Two new AudiobookBay settings, Seed Time (minutes) and Seed Ratio. Both
default to 0, which leaves the torrent client's own limit as it is
today, so nothing changes until someone sets them.

They go to the client with the magnet the same way the Prowlarr seed
preferences do (`DownloadRequest.seeding_time_limit` and `ratio_limit`),
so they apply with the clients that already take per-torrent limits
there: qBittorrent, Transmission and Deluge. A retry that reuses the
saved magnet carries them too. A value that isn't a positive number
(text, negative, NaN, inf) is ignored and the client's limit stays.

Tests cover reading both settings (unset, zero, each on its own, text
values, bad values) and that both the fresh and the cached-magnet
request carry them. Ruff, basedpyright and the backend suite pass. I
haven't tried it against a live client, since the client side is the
code Prowlarr already uses.

I added the two entries to `docs/environment-variables.md` by hand.
Running `scripts/generate_env_docs.py` here also changes unrelated parts
of that file (the Soulseek table of contents link and the
`SHELFMARK_API_KEY_READONLY` entry), so I left those alone.
2026-10-20 23:32:54 -04:00
splitsec2andCaliBrain 77b80eb31a fix(metadata): only an admin may use the provider's own account, its lists and shelves (#1501)
Refs #981.

Hardcover is signed in as one account, the instance owner's, and every
logged-in user shares it. A non-admin can see the owner's lists and
shelves in the List filter, browse one by asking for it, and add or
remove books and change read status from search results, which is what
#981 describes.

Now only an admin may. With authentication set to none the one user
keeps everything.

- `GET` targets and the batch lookup return no options for a non-admin,
so the buttons don't show and the account isn't asked.
- `PUT` targets answers 403.
- The List filter (`hardcover_list`) has no options for a non-admin, and
a request that asks for it has the field dropped before the search.
Providers name these fields in `account_search_fields`, and Hardcover
lists `hardcover_list`.
- Public things are unchanged: author, title and series suggestions, and
a pasted list URL in the search box.

That pasted URL is the one gap I know of. It resolves any list the
account can see, so a user who knows the slug of one of the owner's
private lists could still open it. Closing it means the provider has to
know who is asking, which is a bigger change than this one, so I left
it.

Not included from the issue: a toggle for letting users view lists and
limiting which lists they see. I kept this to the admin-only part so it
stays small, and I can do the toggle as a follow-up if you want it.

Tests cover the routes as an admin, a user and with no authentication
(targets read, batch and write, list options, a crafted list search, and
public fields still passing). Ruff, basedpyright and the backend suite
pass.

Co-authored-by: CaliBrain <calibrain@l4n.xyz>
2026-10-20 23:29:05 -04:00
splitsec2andCaliBrain 82c8c91c8b refactor(clients): read the stall window with a type check instead of a pyright ignore (#1498)
The only `pyright: ignore[reportArgumentType]` in `shelfmark/` is one I
added in #1426, and I was being lazy when I did. I hit the same error on
another PR and replaced the ignore there with a real check, so this does
the same for the original.

`_stall_window_seconds` reads the setting as an `object`. It now narrows
to int, float or str before calling `float()`, and a bad string falls
back to the default as before. A non-numeric value of another type
(None, a list) falls back to the default too, which used to happen
through a `TypeError`.

No behaviour change. The existing range test still passes and I added
`None` and a list to it. basedpyright is at 0 errors, ruff is clean and
the backend suite passes.

Co-authored-by: CaliBrain <calibrain@l4n.xyz>
2026-10-20 23:28:42 -04:00
splitsec2 e83bab9337 feat(clients): give up on a torrent with no seeders (#1494)
A torrent with no seeds won't finish. It runs on the pieces its few
peers hold and then sits still, and nothing says so until the stall
timer ends it. The card only reads "downloading". I run AudiobookBay
releases, which show no seed count in the listing, so the client is the
only place to find out.

Two new settings, both off until you turn them on:
`TORRENT_NO_SEEDERS_WAIT` (minutes, 0 turns it off) and
`TORRENT_MIN_SEEDERS` (default 1).

- A torrent that has fewer seeds than the minimum for the whole wait,
while nothing is downloading, is removed from the client with its data
and the task ends as an error.
- A torrent with no seed at all for the wait is given up on even while
it still moves.
- A torrent that met the minimum and lost seeds down to one, but is
still moving, keeps going.
- The card shows the seed count beside the progress, with a countdown
while it waits ("0 seeds, giving up in 4m").

qBittorrent only, because it reports connected and tracker seed counts.
Other clients report none and are left to the stall timer, and an
unknown count is not read as zero. A torrent that was already in the
client when the download started is never removed, and a queued torrent
isn't judged.

Tests cover the wait and countdown, the clock restarting on progress,
the tracker count, a higher minimum, off by default, a client with no
counts, a preexisting torrent, a queued torrent, and the moving cases.
Ruff and the backend suite pass.
2026-10-20 23:26:13 -04:00
splitsec2 9ba5816b83 fix(requests): an older failed attempt no longer undoes a later delivery (#1493)
A request that was retried after a failure has two tasks in the status
built from download history. `sync_delivery_states_from_queue_status`
walks the status buckets in `QueueStatus` order and keeps the last one
it sees for a request. Error comes after complete, so a request that was
delivered by its retry is set back to error and, when no retry is left,
reopened as pending. The next pass can then fetch the book again.

When a request has more than one task, an attempt that is still running
now decides its state, else a completed one, else the failure. Ties keep
the old order, so a request with a single failed task still reopens as
before.

Tests: an earlier failure doesn't undo a later delivery, a running retry
outranks an earlier failure, and a lone failure still reopens. It
happened on my instance to a request whose first try failed and whose
retry succeeded. Ruff and the backend suite pass.
2026-10-20 23:21:16 -04:00
splitsec2 a1ac017487 feat(requests): let an admin reopen a rejected request (#1491)
Closes #1445, where you said you were open to it.

A rejected request was final. `update_request` won't move a terminal
status and the admin routes only fulfil or reject, so an admin who
changed their mind had to ask the user to request the book again, and
the user kept seeing the old rejected one.

`POST /api/admin/requests/<id>/reopen` moves a rejected request back to
pending and clears the review fields. What the user asked for (request
level, release, note) is kept. It refuses anything that isn't rejected,
and refuses when that user already has a pending request for the same
book.

Admins get a Reopen action on rejected cards in the activity sidebar and
in History. A rejected request has usually been cleared from the list by
the time anyone wants it back, so reopening also clears the dismissal,
the way the failed-download reopen does. Without that the pending
request stays hidden.

Tests cover the service (state change, what is kept, the refusals, a
non-string admin note) and the route, and the card model for the new
action. Ruff, tsc, oxlint, vitest and the backend suite pass. It has
been running on my instance since Oct 8.
2026-10-20 23:20:26 -04:00
splitsec2 88cc734db2 fix(activity): a reused release follows the new request, and stale rows of a delivered request are closed (#1496)
Two things that go wrong when a second request picks a release a first
request already used. A task id comes from the release, so the history
row is reused.

**The row stays with the first request.** Recording a download kept the
first request's id, notes and state. On my instance a card for one book
showed up under the request for a different book, with that request's
note on it. Recording a download now moves the row to the request that
queued it and restarts its queue time. A retry with no request keeps the
request the row had.

**Old rows of a delivered request show as failures forever.** The
startup sweep leaves an old request-linked row alone so its retry stays,
but one whose request already has its book (delivered, or another
download of it finished) is now closed as "Superseded by a completed
download" instead of showing as an interrupted failure for ever.

Also in the first commit: History cards used the saved cover URL as it
was, an external address the page's own-origin image policy blocks,
while running downloads use the cover proxy. They use it too.

The second commit is about restarts. After a restart nothing is in
flight, but a request in downloading, resolving or locating whose row
had moved to another request was stranded, because only queued was
reopened. Those states are reopened too.

Tests cover the row takeover and the retry case, the cover through the
proxy (and a missing cover), the superseded cases (delivered request,
another row finished, an undelivered request is left alone) and the
in-flight states with no row. Ruff and the backend suite pass.
2026-10-20 23:20:04 -04:00
splitsec2 06a67a7150 feat(download): a concurrency limit per source (#1495)
With one global limit, a few tasks that are only waiting hold every
slot. A Soulseek download sitting in a peer's queue is the example I
hit: three of those and a torrent that could start waits behind them.

`MAX_CONCURRENT_DOWNLOADS_PER_SOURCE` (0 keeps the single global limit,
which is the default). With it set, the coordinator starts a download
only when its source is below the limit and passes over tasks of a full
source without losing their place in the queue. `get_next` gets an
optional filter for this, and a cancelled task is still dropped when one
is given. The start-up stagger now applies only between downloads from
the same source, and the pool is sized so every source can reach its
limit.

The second commit: qBittorrent keeps its own queue of torrents waiting
for a free slot. The task polling one of those does no work, so it no
longer counts against its source's limit and the next task of that
source can start. The client does the pacing, as it does for its own
queue.

Tests cover the queue pass-over (order and priority kept, nothing
eligible, cancelled tasks), the coordinator (a full source doesn't block
the others, each source has its own limit, the global limit when unset)
and the client-queue case. Ruff and the backend suite pass.
2026-10-20 23:19:33 -04:00
splitsec2 3b5afb1a04 Merge commit from fork
PUT /api/users/me set a new password from the password field alone, so a live session could replace the password and lock the owner out. A builtin user now has to send current_password with it, checked against the stored hash. A wrong or missing one is a 400, and counts toward the same lockout as a failed login (a locked account gets a 429 without being checked), so the endpoint cannot be used to guess the password.

A password or current_password that is not a string is a 400. A password that was not a string raised a TypeError from len() and answered 500.

The account form asks for the current password when a new one is typed. Profile and settings changes without a password are unchanged.
2026-10-20 23:18:13 -04:00
CaliBrain 10710abd4e fix(download): send every file of a multi-file download to the browser (#1489)
Download to Browser only ever sent the first file of a multi-file
release
(audiobook chapters, packs): the folder output returned final_paths[0]
and
/api/localdownload served that one path.

The folder output now keeps every final path on the in-memory task
(download_paths), and /api/localdownload streams them as one
uncompressed
ZIP named after the book when there is more than one. The archive is
never
staged on disk or held in memory, and ZIP64 covers large audiobooks.
History still stores only the first path, so a re-download from history
after a dismiss or restart serves a single file, as before.

Also brings every linter back to green on all files:
- pre-commit: bump the ruff hook to v0.17.1 to match uv.lock;
ruff-format
  the Python examples in docs/dev and skills
- ruff: fix the remaining issues in skills/shelfmark/download_books.py;
  format base_handler.py
- basedpyright: Optional and attrs errors in databazeknih.py
- knip: drop 31 unused exports, reuse ColumnColorHint in colorMaps
- oxfmt 0.72: styles.css
- direct download: fix the "Welib, mirrors" typo and the test that still
  expected the pre-OceanofPDF message

Fixes #1467
2026-10-20 02:33:53 -04:00
c8d66db22e feat(direct_download): add OceanofPDF as a standalone search and download provider (#1485)
Closes #1149

## What this adds

OceanofPDF as a second Direct Download provider, alongside Anna's
Archive. Unlike Z-Library, LibGen and Welib, which only download files
Anna's Archive found, OceanofPDF searches and downloads on its own. It
works with or without an AA mirror configured; with both, one search
queries both and the results are merged.

It plugs into the provider registry from #1337, so search, record
routing and the download handler need no changes.

**Search:** WordPress `/?s=` search, "title author" first, then the bare
title if that finds nothing. One release per advertised format (`[PDF]
[EPUB]`), with title, author and cover. Shelfmark's supported-format
filter applies as usual.

**Download** follows the site's three hops:

1. The book page has one POST form per format, targeting
`Fetching_Resource.php`.
2. That returns an interstitial whose `<meta http-equiv="refresh">`
points at a signed, expiring file URL.
3. The file URL serves the book.

Pages go through `html_get_page` and its Cloudflare bypass. The form
POST replays the clearance the bypasser stored, and re-solves once if
challenged. Bypassers only replay GETs, so it can't be handed over
directly.

**Settings:**
- `OCEANOFPDF_MIRROR_URLS` under Mirrors. Only the first URL is used.
- An OceanofPDF toggle under Download Sources → Slow downloads. It's
locked until a mirror is set and while the Cloudflare bypass is off. Its
position in the list doesn't matter, and its description says so.
- A saved list from before this change has no OceanofPDF entry. The
settings page shows a missing option as off, so the provider treats it
as off too, keeping the page and the behaviour in agreement. New
installs default to on.

## Fixes found along the way

- **`challenge_marker` rejected real pages behind Cloudflare** (separate
commit, `fix(bypass)`). Cloudflare's bot-detection script injects
`/cdn-cgi/challenge-platform/scripts/jsd/main.js` into every page it
serves, after load. Bypassers return the rendered page, so a successful
FlareSolverr solve of any page under 64 KB came back carrying
`/cdn-cgi/challenge-platform` and was rejected as unsolved. The detector
now ignores that one script path. Real challenge pages load from
`/cdn-cgi/challenge-platform/h/…` and keep their other markers, so they
are still caught. This also affects any other Cloudflare-fronted host,
not just OceanofPDF. Happy to split it into its own PR if you prefer.
- **Brotli on a plain `requests` call.** Copying `DOWNLOAD_HEADERS` into
the form POST offered `br`. Cloudflare then answered in Brotli, which
`requests` can't decode without the optional `brotli` package. The POST
now leaves `Accept-Encoding` to `requests`. If a response still has no
file link, its status, encoding, title and challenge marker are logged.

## Testing

- **New tests:** 41 in `tests/direct_download/test_oceanofpdf.py` and 4
in `tests/bypass/test_cloudflare_beacon.py`. They cover search parsing,
format and language filtering, form selection, the interstitial, the
download flow (including a challenged POST, an HTML stub served as the
file, and cancellation), the toggle, and registry wiring. Fixtures
mirror the live site's markup.
- **Existing tests:** `tests/direct_download`, `tests/bypass`,
`tests/config`, `tests/core` and `tests/download` show no new failures
against `main`. Some tests in those folders can't run on Windows; those
fail identically on both.
- **Lint:** `ruff check` and `ruff format --check` are clean on every
changed file.
- **Live:** verified end to end on a v1.4.0 Docker deployment behind
FlareSolverr (external bypasser), with only an OceanofPDF mirror
configured: search, then the book page, the form POST and the completed
download.

## Known limits

- OceanofPDF doesn't label languages, so its results always pass a
language filter.
- PDFs only appear when `pdf` is in the supported formats.
- Each query reads the first results page only (roughly 7–10 books).
- Downloaded files carry OceanofPDF's own tags/watermark, as noted in
#1149. They aren't stripped.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Adam Stone <196691089+ast0ne1@users.noreply.github.com>
Co-authored-by: CaliBrain <calibrain@l4n.xyz>
2026-10-20 02:07:23 -04:00
amura11andClaude Sonnet 5.5 51f32b374d feat(packs): let users choose which files, books and formats to download (#1480)
## Summary
- **Skip files or books in a multi-book pack.** The pack review panel
gets a switch per file and a "Skip this book" switch per book, using the
shared `ToggleSwitch`. A partly skipped book shows e.g. "3 of 5 files"
in the Files column. A fully skipped book is greyed out.
- **Choose formats for a single book.** A new "Choose formats" header
toggle, next to the multi-book one, opens a format picker for a release
that has several book formats (EPUB, MOBI, PDF and so on). It has one
switch per format with file count and size. Releases with one format, or
that can't be inspected, download as before.
- **Skipped files are sent as `ignored_files`.** They are stored on the
task (including restart-safe retry) and removed before grouping and
transfer. This matters for packs because files missing from `book_plan`
would otherwise be filed by the heuristic fallback. Ignored entries
match by release-relative path, then path suffix, then unique basename.
- A pack that ends up with one book after skips keeps the titles you
reviewed instead of falling back to the searched title.

## Screenshots
<img width="1277" height="413" alt="Screenshot 2026-10-19 at 09-39-28
Shelfmark"
src="https://github.com/user-attachments/assets/c005ca4b-3dac-45dd-b2ff-d02fc0f9fcc9"
/>
<img width="1277" height="1210" alt="Screenshot 2026-10-19 at 09-40-23
Shelfmark"
src="https://github.com/user-attachments/assets/2145f3ba-e5fc-4635-9364-1790feb0c8b8"
/>
<img width="1277" height="748" alt="Screenshot 2026-10-19 at 09-39-03
Shelfmark"
src="https://github.com/user-attachments/assets/39023fac-70e3-46d8-af89-1160ebd6e415"
/>


## Test plan
- [x] `pytest` pack tests (new `drop_ignored_files` cases added)
- [x] `vitest`, `tsc`, `oxlint`, `oxfmt`, `ruff`
- [x] Manual check in the UI with a real multi-book pack

🤖 (mostly) Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-20 02:03:05 -04:00
CaliBrain 3c5cfcb77d fix/redact log secrets (#1488)
- fix(logging): mask tracker API keys in log output (#1486)
- fix(welib): follow slow_download links through the countdown (#1447)
2026-10-20 01:55:49 -04:00
bb1655238a feat(slskd): add Soulseek (slskd) release source, building on #1335 (#1428)
## Add Soulseek (slskd) as a release source, building on #1335

This is @dskvr's work from #1335 rebased onto current `main`, with my
changes on top. #1335 has been quiet since 2026-09-13 and I needed the
changes below, so I carried it forward instead of starting over. Their
two commits are the first two in this branch with their authorship
intact, and the source plugin, handler, API client, settings tab, tests,
`ReleaseProtocol.SOULSEEK`, test client compose file and frontend bits
are all theirs. I haven't changed how a search turns into releases
beyond what is listed here.

If @dskvr would rather keep #1335 alive and take these as commits on
their branch, I can do that instead and close this.

### What I changed

**Downloads survive other apps using the same slskd.** The handler found
its files by listing slskd's downloads, and another app on the same
instance (SoulSync here) sends `DELETE
/transfers/downloads/all/completed` after each of its batches. Once that
ran, a finished download no longer appeared in the list and the task
never saw it complete, even though the files were on disk. With the new
`SLSKD_ISOLATE_DOWNLOADS` setting (on by default) each task:

- enqueues one batch (`POST /transfers/downloads/batches`) into its own
folder, `<SLSKD_DESTINATION_PREFIX>/<task id>`
- polls each transfer by id, which slskd still answers after the list
has been cleared
- decides it is done from the files on disk (sizes, not names), with the
`/events` log as a fallback when the transfers are gone and the files
aren't visible yet
- copies out of that folder, and removes it only if
`SLSKD_KEEP_COMPLETED` is off and the import worked

With isolation off the handler behaves as it did in #1335. If the batch
endpoint isn't there (404/405) it falls back to the original per-file
enqueue.

**Source changes.** m4b/m4a files are their own release next to loose
chapter files, so auto-download can prefer one file over a chapter set.
Results from the query variants are merged per peer. The folder text is
exposed as `extra["author"]` and `extra["title_raw"]`, which is what
auto-download's strict match and pack detection read. `is_available()`
also needs an API key now.

**Smaller fixes.** Cancelling while files are still being located now
removes the transfer records it created. In the settings tab, a cleared
form field is no longer replaced by the saved value when testing the
connection. The new settings are in `docs/environment-variables.md`.

### Testing

`tests/slskd` has 192 passing and 6 live tests that skip without an
instance. I wrote the new tests first and checked the handler and source
rules by breaking them and confirming a test failed. Ruff is clean. On
the whole suite the only failures are 4 in `tests/download` that import
`seleniumbase`, which isn't installed in my environment, and they fail
the same on `main`.

I've been running this on my own deployment against slskd 0.26.0 behind
a VPN, with SoulSync active on a second instance. A search returned a
couple hundred results from about 70 peers, an epub came down and was
imported, and auto-download fetched five audiobooks as single m4b files.
That is a day of use, so I don't know how it behaves on other slskd
versions or on Windows path mappings.

---------

Co-authored-by: sandwich <dskvr@users.noreply.github.com>
Co-authored-by: sandwich <299465+dskvr@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-10-20 01:33:12 -04:00
Amalkrishna A S b38e4e4a00 feat(naming): add {AuthorSort} template variable (#1484)
Closes #1178.

Adds `{AuthorSort}`, the first author surname-first ("Arthur Conan
Doyle" → "Doyle, Arthur Conan"), for rename and organize templates,
alongside `{FirstAuthor}`.

- A value already in "Last, First" form (one word before the comma) is
kept as is; Jr/Sr/II/III/IV stay at the end.
- Known limits, same as calibre's default: the last word is treated as
the surname, so particles ("Le Guin") and surname-first names aren't
detected.
- Added to the settings help, env docs, the frontend variable
picker/preview, and tests.
2026-10-20 01:22:05 -04:00
misko903andClaude Opus 5.5 b83e35ee0a feat(metadata): add Databáze knih metadata provider (#1479)
## What

Adds [Databáze knih](https://www.databazeknih.cz), the Czech and Slovak
community book catalog, as a metadata provider. It follows the Moly.hu
provider (#1172): HTML scraping, no API key, disabled by default, and a
settings tab with an enable checkbox and a Test Connection button.

## Why

Hardcover, Open Library and Google Books cover Czech and Slovak editions
poorly. Databáze knih is the de facto catalog for them, including Czech
and Slovak translations of foreign works. With this provider, Universal
mode works end to end for Czech and Slovak titles: the localized title
and author from Databáze knih feed the release search.

## How

- HTML scraping with BeautifulSoup, which is already a dependency. Book
pages carry schema.org JSON-LD, so most fields come from structured data
rather than CSS selectors.
- **Title search** uses `/vyhledavani/knihy`, with the site's pagination
(`page`) and language filter (`lang` for cs/sk/en/de). A search with a
single match redirects straight to the book page, and that case is
handled.
- **Author search:** the site search only matches titles, so an
author-only search finds the author (accent- and word-order-insensitive)
and lists their books page, which is also paginated. Title + author
filters the title results on the author.
- **ISBN search** follows the site's redirect from `/search?q=<isbn>` to
the book page.
- **`get_book`** reads the full description (the JSON-LD one is
truncated), ISBN-10/13, publisher, edition year, language, genres, cover
(placeholder images dropped), series with id and position, and rating
and rating count. It also reads the page count and original title from
the more-info fragment.
- The `RateLimiter` from Open Library (30 requests/min) and the standard
`@cacheable` decorators. Fetch failures return `None`, so they are not
cached.
- `docs/environment-variables.md`: added the `DATABAZEKNIH_ENABLED`
section.

## Testing

- `tests/metadata/test_databazeknih_parse.py`: offline tests with
shortened copies of real pages. They cover search parsing, pagination
and language parameters, redirect-to-book, author listing, title +
author filtering, uncached failures, book details, ISBN resolution and
helpers.
- `uv run pytest tests/metadata` is green (89 passed), and `ruff check`
and `ruff format --check` are clean.
- Verified live against databazeknih.cz: connection test, title search
(pages 1 and 2), author books, title + author, `get_book`, and ISBN
lookup.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 01:20:26 -04:00
sean0407andSean Li a0ffb268b2 fix: wait for synced files before processing completed downloads (#1464)
## Summary

```text
wait for completed download path
  regular file -> keep existing immediate handling
  directory -> inspect files enabled for this task's content type
              wait until the supported-file signature is stable across checks
```

Fixes the remote-sync race in #1463. A mapped torrent folder can appear
before its large media file finishes copying, so path existence alone
does not mean the download is ready.

## Evidence

- **Before:** In #1463, qBittorrent reported a multi-file audiobook
complete while the destination contained only `.cue` and `.jpg`;
Shelfmark immediately post-processed the folder and failed with “No book
files found in download.” Retrying after the `.m4b` sync finished
imported it.
- **After:** A regression test starts with the cue sheet and cover
present, then simulates the `.m4b` file appearing and growing. Path
waiting returns only after the supported file's size and modification
time remain stable across checks.
- Credential-free isolated `python -m pytest
tests/prowlarr/test_remote_path_mappings.py -q -n 0 -o addopts=''`: 11
passed.
- `ruff check shelfmark/download/clients/base_handler.py
tests/prowlarr/test_remote_path_mappings.py`: passed.

## Merge Danger

**Door:** Two-way

**Blast Radius:** Download handling

The wait uses the existing completed-path timeout and leaves regular
single-file downloads on their current path.

Co-authored-by: Sean Li <22557931+sean0407@users.noreply.github.com>
2026-10-20 01:12:49 -04:00
splitsec2 0ade063db5 fix(download): extract RAR archives in place, not in the IO threadpool (#1459)
Fixes #1458.

A RAR download fails with `Download failed: TypeError`, and the log says
`child watchers are only available on the default loop`. rarfile
extracts by running `unrar` as a subprocess, and `extract_archive_files`
calls it through `run_blocking_io`, so it runs in gevent's IO
threadpool. gevent only lets a subprocess start on the thread that owns
the default loop. `run_blocking_io` already avoids the threadpool for
`subprocess.run`, but a RAR gets to the subprocess through rarfile, so
nothing catches it.

This extracts a RAR on the calling greenlet, where the patched
subprocess waits for `unrar` without blocking anything else. ZIP is pure
Python and stays in the threadpool. I checked it with a real RAR4 file
on a build of main with the stock image's `unrar`: from the threadpool
it raises the TypeError, in place it extracts all 7 files.

Two tests. One checks that a RAR doesn't go through `run_blocking_io`.
The other runs under real gevent patching with an extractor that starts
a process the way rarfile does. Both fail without the change, and the
full suite passes.

I keyed it on the `.rar` suffix to keep it small. Teaching
`run_blocking_io` about rarfile's subprocess would be the more general
fix, but that's a bigger change to a helper a lot of code goes through,
so I left it for you to say if you want it.
2026-10-20 01:12:10 -04:00
ultraviolet150 5f3647a496 fix(clients): wait for qBittorrent-compatible clients to finish before post-processing (#1465)
## Summary

`QBittorrentClient.get_status()` treats a torrent as complete as soon as
`progress >= 1.0`, excluding only the `moving` state. Some
qBittorrent-compatible clients report full progress before they are
finished. rdt-client (v2.0.142) reports `progress = 1.0` once the debrid
and local downloads are done, but keeps `state = "downloading"` for a
few seconds while it finalises the torrent, and until then reports a
guessed `content_path`. It only switches to `pausedUP` with the real
path afterwards.

Shelfmark picks the torrent up inside that window, so post-processing
runs against a path that isn't there yet and fails within milliseconds
("No book files found" / `post-processing failed`). Clicking Retry then
imports it fine. Every rdt-client download hits this.

## Change

- Add `_INCOMPLETE_STATES` (all download-side states, plus `moving`) and
only mark a torrent complete when `progress >= 1.0` **and** its state is
not one of those.
- Real qBittorrent leaves these states when a torrent reaches 100%, so
its behaviour is unchanged.

## Example log (before)

```
12:04:43,215 base_handler - Remapped download path for qbittorrent (<hash>): <rdt-client download dir>\<torrent name>\ -> /downloads/<torrent name>
12:04:43,216 orchestrator - download finished; starting post-processing
12:04:43,232 orchestrator - WARNING - post-processing failed
```

## Testing

- New test
`test_get_status_full_progress_still_downloading_not_complete`: fails
before the change, passes after.
- `tests/prowlarr/`: all passing. `ruff check`, `ruff format --check`
and `basedpyright` clean on the changed files.
- Running the patched client against rdt-client: downloads now import
without Retry.

Related to #1463 / #1464, which handle the same symptom when the client
really is finished but files are still syncing to local disk. This PR
addresses a different cause: the client reports 100% progress while
still in a downloading state, so Shelfmark shouldn't treat it as
complete yet. The two changes are independent and complementary. Not
covered by #741, which fixed add-time failures. Possibly related: #607.
2026-10-20 01:11:33 -04:00
Matt Van HornandMatt Van Horn d3bcf0674d fix(direct_download): Anna's Archive search 404 on ISBN queries (#1456)
When ISBN filters are present, the search query is now the deduplicated
bare ISBN strings, with the free-text query appended only if it is not
already one of those tokens. Searches with no ISBN still send the quoted
free-text query alone. An Anna's Archive lookup that includes an ISBN
comes back as 404 Not Found, so the direct-download release search fails
and the book cannot be retrieved. The failing request puts an
isbn13:/isbn10: boolean expression in the q parameter.

The direct-download search query tests passed.

Closes #1444

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-10-20 01:10:03 -04:00
splitsec2 84d1d7196c fix(irc): pin the TLS context to TLS 1.2 or newer (#1455)
CodeQL on my fork flagged `ssl.create_default_context()` in the IRC
client as allowing TLS 1.0 and 1.1 (`py/insecure-protocol`, rated high).

In practice the shipped image is already fine. Python 3.10+ sets a TLS
1.2 floor on the default context, and on the current image (Python
3.14.7, OpenSSL 3.5.7) `minimum_version` comes back as TLS 1.2. An
OpenSSL config on the host can still lower that floor, so this pins it
with one line, `context.minimum_version = ssl.TLSVersion.TLSv1_2`.

Added `tests/irc/test_client_tls.py`. It drops the floor to the minimum
before `connect()` runs, then checks that `connect()` raised it back to
1.2. It fails without the change.

Certificate and hostname verification are still off (`CERT_NONE`). I
left that alone because of the comment about self-signed certs on IRC
servers. Turning verification on, or making it a setting, would be the
bigger fix, but that is ultimately a call for @calibrain if you want a
PR for it.
2026-10-20 01:09:26 -04:00
Som Samantray 2b8e0da2f3 fix(direct-download): clarify required search mirror (#1454)
Direct Download now tells users who configure only a Z-Library mirror
that they need an Anna's Archive mirror for search. Z-Library, LibGen,
and Welib mirrors download results found through Anna's Archive.

Fixes #1408. Validation: the Direct mode browser check, 18 focused
tests, Ruff, formatting, and backend/test type checks passed. The
CI-style Python suite passed 3,569 tests;
`tests/core/test_hardlink.py::TestSameFilesystem::test_string_paths` was
excluded because this container reports different `st_dev` values for a
file and its parent directory. AI disclosure: OpenAI Codex assisted with
diagnosis, implementation, and validation.
2026-10-20 01:09:09 -04:00
HeihoffJandClaude Opus 5.5 756bdec19b fix(metadata): show the translation a search matched on Open Library and Hardcover (#1451)
Searching a translated title on Open Library or Hardcover returned the
original-language work. "Der Weg der Könige" came back as "The Way of
Kings" (2010), and that English title then went into the release search
and the download name. Both providers keep one record per work, titled
after the original, and list translations underneath it.

#483 already searches releases per language with Hardcover's localized
titles, which fixed the release side of #360 when the matching language
is selected. This PR fixes the search result itself, so the book a user
picks is the translation they searched for, and it covers Open Library,
the default provider, which has no titles_by_language.

- Open Library: request the matched edition (editions.*) and prefer its
title, ISBNs, cover, language and publication year when its title
differs from the work's.
- Hardcover: use the alternative title named in the hit's highlights
(alternative_titles, best index first), without the original's subtitle.
A hit whose title also matched keeps the work title.
- /api/releases: when the modal sends a title that differs from the book
record's, it becomes search_title. Before, Hardcover's search_title from
the original won, so picking "Kinder des Nebels" still searched "The
Final Empire" first.
- Release cache: keyed on title and language as well as the provider ID.
All translations share that ID, so opening "The Way of Kings" after "Der
Weg der Könige" showed the German releases until a reload.

Known limitations: neither provider has descriptions per edition, so the
description stays the work's (usually English). Hardcover's search hits
carry no year or language for alternative titles, so those stay the
work's; Open Library's do not. Google Books is unaffected, since it
already returns one volume per edition.

Tests: Open Library edition parsing, Hardcover alternative-title
matching, the /api/releases title override, and the release cache, each
written to fail first. The Python suite passes apart from the 9
test_entrypoint_permissions.py tests, which fail the same way on an
unmodified main on macOS. The frontend typecheck, lint, format check and
unit tests pass. Checked on a live instance with Open Library, Hardcover
and Google Books.

Refs #360, #483

Disclaimer:
This was developed with the help of AI but reviewed and tested as well
as deployed on my server by a real human.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 01:08:45 -04:00
splitsec2andCaliBrain 45548687b7 fix(hardcover): defensive rate limiting and 429 backoff (#1439)
Kept as small as I could so it's easy to review. It is still about 140
lines of code plus tests, in two commits so the second can be dropped on
its own. Same kind of fix as #1429, so see the question there about
separate PRs versus combined.

This is a defensive change. I found the problem on my own deployment,
which has a more automated ingestion on top of this provider: Hardcover
want-to-read items turn into requests, and each pending request gets one
`get_book` lookup. One pass made around 38 lookups back to back, which
is more than Hardcover's burst allows, and a few came back as 429.
`_execute_query` swallowed the 429 and returned `None`, so `get_book`
returned `None`, and the caller read that as "no such book". Nothing in
the logs said a lookup had been throttled.

**What it does**
- Every POST goes through one shared token bucket (burst 5, 0.8 requests
per second), set under the documented Free limits. The limit is per user
across all of the account's apps, so I kept well below it.
- A 429 is retried up to 3 attempts in total. It waits for `Retry-After`
when Hardcover sends one, otherwise 2^attempt seconds plus up to a
second of jitter, and the wait holds back other threads too. A
`Retry-After` over 30 seconds (a spent daily quota) is not waited out.
- When the retries run out it raises `HardcoverRateLimitError`, a
`RuntimeError`, instead of returning `None`.
- The second commit makes `get_book` ask for that error, so a throttled
or failed lookup is no longer taken for a missing book. GraphQL
rejections still return `None`.

**Behaviour change in the second commit.** `/api/metadata/book` already
turns a `RuntimeError` into a 503, so a timeout or HTTP error there now
returns 503 instead of 404. In `/api/releases` the same error goes
through the operational errors handler and returns 500 with the message,
where it used to return 404 "Book not found". If you'd rather keep
returning `None` and only have the pacing and backoff, I can drop the
second commit.

**Limits.** I read them from the `hardcover-docs` repo
(`api/Getting-Started.mdx`, Rate Limits), because the docs site blocks
my headless browser: Free is 60 per minute, a burst of 10 and 5,000 a
day, with `Retry-After` on a 429. Personal Access Tokens get double the
burst. I didn't parse the `RateLimit-Policy` headers, so the bucket
constants are fixed.

**Testing.** The tests use a fake clock and never sleep. I broke each
rule in turn (pacing, `Retry-After`, the 30 second cap, the shared
hold-back, and `get_book` raising) and each break failed a test. Ruff,
basedpyright, `tests/metadata` and `tests/core` pass. I haven't seen a
real 429 from Hardcover with this change running, so the `Retry-After`
handling comes from the docs.

The one red check is the existing clock-dependent test
`test_a_queued_torrent_asks_for_a_grace_once`, which fails on any runner
that booted recently. It is fixed in #1426 and is not caused by this
change.

---------

Co-authored-by: CaliBrain <calibrain@l4n.xyz>
2026-10-20 01:06:56 -04:00
splitsec2 61cf0792bd fix(googlebooks): report more pages when limit is above the 40 result maximum (#1434)
Kept small on purpose so it's easy to review. Same kind of small fix as
#1429 to #1432, so see the question in #1429 about separate PRs versus
combined. I'm sending them separately until I hear otherwise.

`search_paginated` guesses `has_more` as `len(books) >= options.limit`.
Google Books returns at most 40 per page and `search` already caps
`maxResults` at 40, but `/api/metadata/search` accepts a limit up to
100. With a limit over 40 a full page of 40 never reaches `limit`, so
`has_more` is always False. The UI sends 40, so this only affects direct
API callers.

The Google provider now overrides `search_paginated` to apply the same
cap before the base heuristic runs. A full page of 40 with a limit of
100 reports more results and a short page doesn't. The first test fails
on current `main`. Ruff, basedpyright and `tests/metadata` pass.

The one red check is the existing clock-dependent test
`test_a_queued_torrent_asks_for_a_grace_once`, which fails on any runner
that booted recently. It is fixed in #1426 and is not caused by this
change.
2026-10-20 01:01:38 -04:00
splitsec2 f0f0d46381 fix(queue): first terminal status wins, so a late COMPLETE cannot overwrite a cancel (#1429)
Kept small on purpose so it's easy to review. I have a few more small
fixes like this from going back over recently merged PRs. Would you
rather get them as separate PRs like this one, or combined into one?

#1361 made `cancel_download` write under the queue lock, which stops a
cancel from overwriting a download that finished first. The other
direction is still open. The worker checks `cancel_flag.is_set()` and
then calls `update_status(COMPLETE)` a few lines later in
`_process_single_download`. If a cancel lands between those two lines
the task goes CANCELLED, then the worker writes COMPLETE over it.
`_apply_status_locked` allows terminal to terminal moves, so the
terminal hook also fires a second time and records two events for one
task.

This makes the first terminal status win: `_apply_status_locked` ignores
a different terminal status once the task already has one. Requeueing
from a terminal status still works because QUEUED isn't terminal.

I replaced the existing race test. It patched `queue.update_status`,
which `cancel_download` no longer calls, so it never exercised the path
it names, and its last assertion only looked at the final hook event.
The new tests call the queue directly: complete then cancel is refused,
cancel then complete stays cancelled with one hook event, error then
complete stays error, and a terminal task can still be requeued. The two
late-write tests fail on current `main`.

Ruff and basedpyright are clean. `tests/core` and `tests/download` pass
apart from 4 `seleniumbase` import failures in my environment that also
fail on `main`.

The one red check is the existing clock-dependent test
`test_a_queued_torrent_asks_for_a_grace_once`, which fails on any runner
that booted recently. It is fixed in #1426 and is not caused by this
change.
2026-10-20 01:00:21 -04:00
splitsec2 65dbf42a86 fix(release-search): report a ValueError from a source as that source's error (#1433)
Kept small on purpose so it's easy to review. This is the same kind of
small fix as the ones in #1429 to #1432, so see the question in #1429
about separate PRs versus combined. I'm sending them separately until I
hear otherwise.

In `search_source_releases` the `except ValueError` comes before the
branch that handles the operational errors, and `ValueError` is in that
tuple. A `ValueError` from `build_release_search_plan` or
`source.search()` is therefore reported as `Unknown source: <name>`
instead of `<name>: <message>`. The only `ValueError` meant for that
branch is the one `get_source` raises for a name that isn't registered,
so that call now has its own `try` and the rest goes back to the
operational errors branch.

The failure test already covered `OSError` and `RuntimeError`, so I
added `ValueError` to it, and that case fails on current `main`. Ruff,
basedpyright and `tests/core` pass.

The one red check is the existing clock-dependent test
`test_a_queued_torrent_asks_for_a_grace_once`, which fails on any runner
that booted recently. It is fixed in #1426 and is not caused by this
change.
2026-10-20 00:59:54 -04:00
splitsec2 93f58caf7a fix(debug): keep SHELFMARK_API_KEY_READONLY out of the debug bundle (#1432)
Kept small on purpose so it's easy to review.

`genDebug.sh` filters secrets out of `environment.txt` by matching
`_KEY=`, `_SECRET=`, `_PASSWORD=` and `_TOKEN=`, which only catches
names that end in those words. `SHELFMARK_API_KEY_READONLY` ends in
`_READONLY`, so its value lands in the file, and people attach these
bundles to issues. That key only reads counters, so the impact is small,
but any variable like `DB_PASSWORD_FILE` would leak the same way.

The pattern now matches the secret word anywhere in the variable name.
The new test runs the real filter line from the script against a made-up
environment, and it fails on the old pattern.

The one red check is the existing clock-dependent test
`test_a_queued_torrent_asks_for_a_grace_once`, which fails on any runner
that booted recently. It is fixed in #1426 and is not caused by this
change.
2026-10-20 00:59:10 -04:00
splitsec2 be2653bdb7 fix(debridlink): reuse the file list from the status check (#1431)
Kept small on purpose so it's easy to review.

When a torrent is ready, `get_status` has already fetched the full
record with `/seedbox/list?ids=` and `_handle_torrent_info` has checked
that every file is at 100%. The retrieval thread was then started
without that list, so `_process_and_download` called `_fetch_file_list`,
which asks the same endpoint again. That second call runs inside the
thread's generic `except`, without the flood and transient error
handling the status path has, so a rate limit or a hiccup there sets the
phase to error for a torrent that finished fine.

This passes the files already in hand into the thread.
`_process_and_download` already took an optional `files` argument. The
new test starts the thread from a ready status and checks it uses the
given list and never calls `_fetch_file_list`.

Separately, `download_url` returns a `BytesIO`, so a whole file sits in
memory. AllDebrid, TorBox and Real-Debrid do the same, so I left that
for a streaming variant of the shared helper instead of fixing it for
one client here.

Ruff and basedpyright are clean and `tests/prowlarr` passes.

The one red check is the existing clock-dependent test
`test_a_queued_torrent_asks_for_a_grace_once`, which fails on any runner
that booted recently. It is fixed in #1426 and is not caused by this
change.
2026-10-20 00:59:00 -04:00
splitsec2 e8f6505bea fix(library): back off after a failed index instead of retrying for every result (#1430)
Kept small on purpose so it's easy to review.

`library_index._entries_for` retries `fetch_entries` on every call while
the index is failing. The lookup runs once per search result, so with
the Calibre path unreadable a page of 40 results makes 40 attempts and
logs 40 warnings. The entrypoint runs one gevent worker and the Calibre
read is synchronous sqlite, so each attempt also holds up every other
request.

After a failure it now answers from the stale cache (or an empty list)
for 60 seconds before trying again, and a successful index clears that.
The new tests cover 40 lookups against a failing library with a stale
cache (one failed attempt, one warning), a failing library with no cache
(tried once), and a retry once the 60 seconds have passed. The first and
second fail on current `main`.

I left a few other things in this module out to keep this one small:
re-indexing blocks the worker for as long as a big library takes (I
haven't measured how long), `test_connection` stores entries from the
unsaved form path under the shared cache key, and a title with a
collection word in it, like "Books of Blood", shows as "In a
collection". I can send any of those separately.

Ruff and basedpyright are clean and `tests/core` passes.

The one red check is the existing clock-dependent test
`test_a_queued_torrent_asks_for_a_grace_once`, which fails on any runner
that booted recently. It is fixed in #1426 and is not caused by this
change.
2026-10-20 00:58:49 -04:00
splitsec2 47707b827a feat(download): make the torrent stall window a setting (#1426)
#1420 gave a torrent 15 minutes to get going and 15 minutes after each
step forward. That suits my swarms, but other people will want it
shorter or longer, so this makes the number a setting.

`TORRENT_STALL_TIMEOUT_MINUTES` sits in Download Clients with the other
torrent options and takes 5 to 60. The default is 15, so nothing changes
unless someone sets it. 5 is the orchestrator's own stall timeout, so
choosing it asks for no extra time and gives the old behaviour. The
value can arrive as text from an env var, so anything that isn't a
number falls back to the default and anything out of range is clamped.

The orchestrator caps one activity grace at 16 minutes, so I raised that
to an hour to fit the top of the range. As far as I can tell the
protection bypass is the only caller that asks for more than a few
minutes, but I haven't audited every caller. If you'd rather keep the
cap, I can lower the maximum to 16 minutes instead.

Tests cover the default, 5 and 30, text values, out of range values and
NaN, the start and movement windows following the setting, and the
setting showing up in the Download Clients tab. I broke each rule on
purpose to check a test catches it. The full suite fails the same tests
with and without this change (the bypass and seleniumbase tests don't
run in my environment).
2026-10-20 00:58:24 -04:00
CaliBrainandClaude Opus 5.5 9321096032 fix(clients): correct Debrid-Link file listing and rate-limit handling (#1425)
Follow-up to #1380.

- Fetch the file list with ?ids=, the documented parameter. There is no
  ?id=, so a many-file torrent was not expanded and the call could fail
  with badArguments at the last step of every download.
- Back off status checks for 60s after floodDetected, doubling to a 240s
cap, instead of an hour. The orchestrator cancels a download after five
minutes without a change, so the hour-long pause cancelled every active
  download. Only a flood on the status check starts it, and a successful
  check resets it.
- Drop the client-side one-hour stall timer. The orchestrator stall
timer
  always fires first.
- Keep progress at 50% when file retrieval starts instead of dropping to
  0% until the first file arrives.
- Report queued, paused and verifying torrents as QUEUED, PAUSED and
  CHECKING, so a queued torrent gets the handler queue grace.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 08:08:32 -04:00
splitsec2 cad18b8019 feat(clients): add Debrid-Link as a torrent client (#1380)
calibrain said in #1315 that he'd review and merge a Debrid-Link client
if someone took a stab at it, so here it is. It's the fourth debrid
service alongside AllDebrid, Real-Debrid and TorBox and it uses the same
DownloadClient interface as the other three. No frontend changes, the
settings UI picks it up from the backend field definitions.

**I can't test this against the live API.** I don't use debrid and I
don't have a Debrid-Link key. Everything here is built from their v2 API
docs and cross-checked against two independent reference
implementations, so the request shapes should be right, but nobody has
run a real magnet through it yet.

@Tatsu941 you opened the issue and mentioned they give developers free
premium. Would you be able to try it? The things most worth confirming
are the field names on a seedbox torrent, that `premiumLeft` is the
right premium check, and that an API error comes back the way the client
expects. Happy to fix whatever turns up.

Two things about their API shaped how this is written.

Every v2 endpoint wraps its payload in a `{success, value}` envelope and
a failure can arrive with an HTTP 200, so `_request_value` decides the
outcome from the body instead of the status code.

A completed seedbox torrent already carries a direct `downloadUrl` on
every file. The other three clients each need a separate unrestrict or
link-request call per file and this one doesn't, which is why it came
out shorter than them despite doing the same job.

Endpoints used:

| Endpoint | Used for |
|---|---|
| `GET account/infos` | test the key, confirm premium time remains |
| `POST seedbox/add` | magnet as JSON, .torrent as multipart |
| `GET seedbox/list` | poll progress |
| `DELETE seedbox/{id}/remove` | clean up |

Ruff and basedpyright are clean and the 36 new tests pass. The two
failures in `tests/e2e/test_proxy_auth_middleware.py` are already on
main and fail the same way on a pristine checkout. They assume the first
proxy-authenticated user becomes an admin, which only holds while the
user table has no admin in it, so they depend on how many users earlier
tests created.

One thing for review: the download loop uses `download_url` like the
other three clients, which holds each file in memory. I kept the
existing pattern rather than do something different in a new-provider
PR, but say so if you'd rather it streamed here.

Closes #1315
2026-10-03 07:55:36 -04:00
CaliBrain f773ac6029 fix(sabnzbd): strip Prowlarr API key on cross-host NZB redirects (#1424)
_fetch_nzb_content sent X-Api-Key
then followed redirects with the header still attached, leaking the
Prowlarr key when Prowlarr 302s to the indexer download link. Follow
redirects manually with allow_redirects=False and re-evaluate headers
 per hop, mirroring the torrent path.
2026-10-02 22:46:16 -04:00
splitsec2 755f28a6a4 feat(api): add a read-only API key and a /api/stats endpoint (#1419)
This is the read-only API key from #1410, where you said to go ahead.

I run Shelfmark behind Homepage and wanted more on the dashboard tile
than up or down, without putting an admin API key in the dashboard's
config. This adds a second key, `SHELFMARK_API_KEY_READONLY`, that can
read one new endpoint.

`GET /api/stats` returns counts only: books added over the last 7 and 30
days by format, the queue, requests by outcome, and download failures
over the last 7 days. No titles and no user names. The read-only key,
the admin key or an admin session can read it.

The read-only key gets a 403 on every other path and on any write,
including a POST to `/api/stats` itself, and it never gets a session or
a cookie. If a request carries both keys, the admin one wins, so a
proxy's own `Authorization` header can't downgrade a correct
`X-Api-Key`. Unset means off, like the existing key. The endpoint exists
either way, but without a key only an admin session can read it.

It also adds a `db_path` property on `UserDB`, so the stats module can
open the database read-only.

Tests cover the key scope, the 403s for the read-only key, both keys at
once, who can reach the endpoint, and the counters. The full suite
passes, and I broke each rule on purpose to check a test catches it. A
version of this has been running on my own install behind Homepage.

If you'd rather have the stats in a different shape or under a different
path, I'm happy to change it.
2026-10-02 22:45:48 -04:00
adman234andClaude Opus 5.5 32abaff21e feat(naming): {Narrator} placeholder and MyAnonamouse series fallback (#1407)
## Human-Written explanation:

Following up on previous PR to use MAM ID to get nararrator and series
to show up in search, this PR will allow the series and nararrator
fields to be added to the path when saving an audiobook.

I have tested my ghcr.io image on my instance and it seemed to work
properly.

All the text below is written by Claude.

# feat(naming): {Narrator} placeholder and MyAnonamouse series fallback

Follow-up to #1390 and #1399. Related to #605 and #934 (narrator
support).

## Problem

Several narrations of the same audiobook currently land in the same
folder, because nothing in the path template tells them apart. Keeping
more than one version means renaming folders by hand after every
download.

Since #1390, MyAnonamouse results carry the release's narrator and
series in `extra`, but the naming templates can't use them: there is no
`{Narrator}` placeholder, and `{Series}` / `{SeriesPosition}` only come
from the metadata provider.

## Changes

- **`{Narrator}` placeholder** (`core/naming.py`,
`download/postprocess/transfer.py`, `core/models.py`): `DownloadTask`
gains `narrator`, read from `narrator` or `extra.narrator` when a
release is queued, and kept in the restart-safe retry payload. It's
available in audiobook templates like any other placeholder, including
prefix/suffix blocks such as `{ - Narrator}`.
- **Audiobookshelf folder style**: Audiobookshelf reads the narrator
from folder names like `Title {Narrator}`. The template `{Title}
{{Narrator}}` already matches as `{` + `{Narrator}`; the parser now also
drops the closing brace when the narrator is empty, so the folder is
`Title` rather than `Title }`.
- **No stray spaces around `/`**: an empty placeholder at the start or
end of a folder name no longer leaves a space there (e.g. `Title
/Title`). This applies to all templates.
- **Series fallback** (`release_sources/prowlarr/mam.py`, `source.py`,
`download/orchestrator.py`): MAM enrichment also stores the first
series' name and number as `extra.series_name` /
`extra.series_position`, which queueing already falls back to when the
metadata provider has no series. The provider's series still wins. A
release-level series number is only used when it names the same series
as the provider, so a provider series is never paired with the number of
a different MAM series (an omnibus, for example).
- **Settings and preview** (`config/settings.py`,
`namingTemplatePreview.ts`): the audiobook template descriptions list
`{Narrator}` and explain `{{Narrator}}`. The settings page preview
mirrors the parser changes and offers `{Narrator}` under "Insert
variable" for audiobook templates only.
- **Docs**: regenerated `environment-variables.md`. This also picked up
the Library Check settings, which weren't in the generated docs yet;
happy to drop that part to keep the diff focused.

## Example

Audiobook Path Template `{Author}/{Title} {{Narrator}}/{Title}`:

| Narrator | Result |
|----------|--------|
| Samuel Roukin | `Christopher Ruocchio/Empire of Silence {Samuel
Roukin}/Empire of Silence.m4b` |
| none | `Christopher Ruocchio/Empire of Silence/Empire of Silence.m4b`
|

## Testing

- `tests/core/test_narrator_template_variable.py` (new, 16 tests):
`{{Narrator}}` with and without a value, the `{ {Narrator}}` prefix
form, prefix/suffix blocks, sanitizing, `build_library_path`, task
metadata, retry payload round trip, MAM series name/number parsing
(including a `1-3` range having no number), and queueing (narrator from
`extra`, series fallback, no cross-series number, blank narrator).
- `namingTemplatePreview.test.ts`: the same `{{Narrator}}` cases for the
preview.
- Updated `test_generate_env_docs.py` for the new template description.
- Full `pytest -m "not integration and not e2e"` compared with an
upstream `main` worktree on the same machine: no new failures (the
remaining ones are Windows-only on both).
- `ruff check`, `ruff format --check`, `basedpyright` (0 errors) and
`vulture` on touched files; frontend `tsc --noEmit`, `oxlint`, `oxfmt
--check`, `vitest` (218 passed).
- Checked the settings page locally: the audiobook preview renders `...
{Simon Vance}.mp3` and `{Narrator}` is listed for audiobook templates
only.

No behavior change for existing templates, apart from spaces next to `/`
being trimmed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 22:45:10 -04:00
CaliBrain 119d3374e3 fix(downloads): reconcile downloads a restart interrupted (#1423) 2026-10-02 22:35:09 -04:00
HeihoffJandClaude Opus 5.5 95d9f1da57 fix(sabnzbd): prefetch NZBs served from the indexer's own domain (#1416)
Indexers often serve NZB downloads from a different host than their API
(e.g. file.indexer.example for indexer.example/api). The exact-origin
check added in #967 rejected those, so SABnzbd silently fell back to
addurl.

Trust any host in the configured indexer's registrable domain, using the
Public Suffix List so shared suffixes like co.uk or duckdns.org never
widen trust. Scheme and port must still match, IP literals and
single-label hosts still require an exact match, and the Prowlarr API
key header is unchanged.

This closes this issue:
https://github.com/calibrain/shelfmark/issues/1411 

Disclaimer: Implemented by Claude and Co-Authored by me. Tested and
verified by me alone (Why is it always this way around and not the
other)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 22:25:36 -04:00
splitsec2andCaliBrain 8392b43dd3 fix(downloads): reconcile downloads a restart interrupted (#1409)
I found this while grabbing new books and working on my fork. I
redeployed the container while downloads were running, and it left a
mess in my queue. It's an edge case, but a few rows were stuck until I
cleared them by hand.

The download queue lives in memory, so stopping the process with a
download in flight leaves its history row "active" and its request
"queued", with nothing working on either. The activity API shows that
row as "Interrupted" when it's read, but nothing is stored and nothing
looks at it again. Upstream only reopens an errored request when no
manual retry is offered, and an interrupted row always offers one, so
the request just sits there.

This closes those rows out once at startup, before the coordinator
starts, as an error with the message "Interrupted". The retry payload
stays, so the manual retry still works. A request whose download was
queued in the last 24 hours goes back to pending through
`reopen_failed_request`. Older ones get closed out but their requests
are left alone, because I didn't think a startup should reopen a backlog
or fill an admin's approval queue with old requests. A request that
already arrived is left alone, and running it twice changes nothing.

It assumes a single worker process, which `entrypoint.sh` already sets.

On my install the first start after deploying it closed 53 rows that had
been orphaned for nine days. Tests are in
`tests/core/test_startup_reconcile.py`, and the full suite passes
locally.

If the 24 hour window feels off, or you'd rather handle this another
way, I'm happy to change it or drop the PR.

Co-authored-by: CaliBrain <calibrain@l4n.xyz>
2026-10-02 22:23:28 -04:00
Evan KazakinandEvan Kazakin 7888217380 feat(download): add Remove & Delete Files torrent completion action (#1401)
Closes #1027

## Summary

- add a **Remove & Delete Files** (`remove_and_delete`) option to the
Torrent Completion Action setting
- after a successful import, remove the torrent and ask the client to
delete its downloaded data (`delete_files=True`), matching what the
usenet "move" flow already does
- clarify the setting description so it says **Remove** keeps the
downloaded files (raised in the issue comments)
- regenerate the `PROWLARR_TORRENT_ACTION` entry in
`docs/environment-variables.md`

## Behavior

The deletion runs from `post_process_cleanup`, the same place as the
existing Remove and Change Category actions, so it only happens after
output transfer and post-processing have succeeded. At that point every
file has already been copied or hardlinked into the library, and
transfer size checks have passed. A failed import never removes or
deletes anything. The torrent client deletes its own data, so Shelfmark
does not delete paths itself and remote path mappings are not involved.
Keep, Remove, and Change Category behave as before, and Keep is still
the default.

If a request matched a completed torrent that was already in the client,
this option deletes that torrent's data too. That is the same scope the
existing Remove action already applies to.

## Validation

- `make python-checks` (ruff check, ruff format, basedpyright on backend
and tests, vulture): passed
- `make python-test`: 3363 passed, 5 skipped, 9 failed. All 9 are in
`tests/config/test_entrypoint_permissions.py` and happen because macOS
`/bin/bash` 3.2 does not support `${1,,}` in `entrypoint.sh`. They do
not touch this change.
- new tests in `tests/prowlarr/test_handler.py`:
- Remove passes `delete_files=False` and Remove & Delete Files passes
`delete_files=True`
  - a failed import with Remove & Delete Files does not call the client
  - the delete case fails without the handler change
- manual smoke run: `PROWLARR_TORRENT_ACTION=remove_and_delete` set
through real config loading, with a stub client that deletes its folder.
The torrent folder was deleted and the hardlinked library file stayed
intact.
- pre-commit hooks (prek) passed

---------

Co-authored-by: Evan Kazakin <evan@Evans-MacBook-Pro.local>
2026-10-02 22:15:17 -04:00
splitsec2 d99e9d7c4f fix(download): remove a cancelled torrent that never started (#1421)
Cancelling a torrent download leaves the torrent in the client.
`_safe_remove_download` documents the rule: "torrents: never remove or
delete client data (avoid breaking seeding)". That makes sense for a
torrent that has data. A torrent that is still at 0% has nothing to seed
and nothing to resume, and leaving it behind holds a download slot in
qBittorrent's queue or sits there as a dead entry.

I hit this on a qBittorrent that is shared with Sonarr, Radarr and
Lidarr and has a download limit. Shelfmark cancelled downloads that were
queued (#1420) and left them in the client. Three torrents that never
fetched metadata held every active slot, and each torrent added after
them waited behind them. Removing those by hand freed the slots.

This removes the torrent and its files when a download is cancelled,
Shelfmark added the torrent, and the client reports 0% progress. A
torrent with any data is left alone as before. So is one that was
already in the client when the download started, since Shelfmark didn't
add it. If the removal fails it is logged and the cancel carries on.

Tests cover removal at 0% for a queued and a downloading torrent,
leaving one that has data, leaving one the user already had, and a
failing removal. The full suite passes, and I broke each rule on purpose
to check a test catches it.

This changes a rule you wrote down, so I've kept it apart from the queue
fix. If you'd rather have it as a setting, or only remove a torrent that
is still queued or fetching metadata, I'm happy to rework it.
2026-10-02 22:12:51 -04:00