A failed or cancelled card showed the log line, things like `Peer
transfer failed: Errored` or a 500 with a URL in it, so a red card meant
going to the logs to find out whether anything was coming next. They now
say what happened and what happens next.
For a download that belongs to a request, the card says another copy
will be tried in about N minutes when the quick retry from #1535 is on,
and points at Retry or reopening the request when it isn't. A manual
download is never promised an automatic retry. The Soulseek cases each
get their own sentence: the peer dropped the transfer, the peer could
not be reached, slskd refused to queue it, and the peer never answered.
The "tried twice" wording leans on the retry from #1590.
The saved message is left alone because auto-download reads it to tell a
failed release from an interrupted one. Only what the activity list
shows is reworded, on a copy, and anything the new code doesn't
recognise comes back as it was.
It matches on how the message starts, so it is only as current as the
wording the rest of the code uses. A message that gets reworded
elsewhere falls back to the original line instead of breaking, which
seemed fine for something that only changes the display, but it is the
part I would look at first.
It has been running on my instance since mid-October. Ten new tests
cover each message and the next step, and the activity route tests still
pass. Ruff and basedpyright are clean on the changed files.
Last of the planned moves is the download recording (row 9 in Discussion
#1536). The queue hooks that write a download to the history when it is
queued, finalize it on a terminal status, notify, push the activity
update and reopen a failed request move from `main.py` to
`core/download_recording.py`, registered with
`register_download_recording(...)`.
None of it was tested directly, so the first commit adds 38 tests: the
queued record for direct and request linked downloads, the terminal
snapshot for complete, error and cancelled, ignoring non terminal
statuses, who gets notified, reopening a failed request, and what
happens when a service is missing or a call fails. They pass on current
`main`. The second commit is the move.
`_resolve_status_scope` and the request update emitter stay in `main.py`
because other code uses them, so the emitter is passed in, and the user
database and history service are looked up per call as before. The hooks
are registered at the same spot. Some existing test lines changed with
the move and no assertion did: the `notify_admin`, `notify_user` and
`reopen_failed_request` patches point at the new module, the direct
calls to `_record_download_queued` go through the queue's hook, and two
tests import `QueueStatus` and `viewer_download_id` themselves now that
`main.py` no longer imports them.
The URL map is identical, all 77 rules, and the full suite, ruff and
basedpyright are clean.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **Improvements**
* Download history and activity updates reflect queued downloads and
their completion statuses; activity updates are emitted after successful
finalization.
* Admins and eligible download owners receive notifications when
downloads reach a terminal status.
* Requests linked to downloads that fail and cannot be retried are
reopened with a failure reason.
* Activity updates use an opaque download identifier rather than
exposing the underlying task identifier.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Next from the route plan in Discussion #1536 are the queue routes:
status, cancel, retry, priority, reorder, queue order, active downloads,
the local file download and the cover proxy. They move from `main.py` to
`core/queue_routes.py`, registered with `register_queue_routes(app,
login_required, ...)`, and `main.py` is about 600 lines shorter.
The first commit adds 33 tests for what the guardrail tests did not
cover, mostly the local download in each shape, the admin paths, the
validation errors and the failure responses. They pass on current
`main`. The second commit is the move.
`resolve_status_scope` and the request update emitter stay in `main.py`
because the release routes and the request code still use them, so they
come in as parameters, with the user database and the history service as
callables evaluated per request. Two test lines changed: the patches of
`send_file` and of the delivery state sync now point at `queue_routes`,
since that is where they get looked up. Log lines from these routes now
name `shelfmark.core.queue_routes`.
The URL map is identical, all 77 rules. The full suite, ruff and
basedpyright are clean, and I ran the app from the branch and hit
status, queue order, active downloads, the local download and the cover
route, which answer as before. The exception text a few of them return
in a 500 is unchanged, that is for the follow-up on exception text.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **Improvements**
* Queue status, downloads, and queue controls respect account ownership,
with administrative access retained.
* More specific errors clarify when queue items or files are missing or
unavailable.
* Queue actions support cancellation, retrying, prioritization, and
reordering.
* Cover requests can use cached images when caching is enabled.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Two small leaks from going through the open CodeQL alerts (code scanning
alerts 100 and 123).
The local download route returned `str(e)` on an OSError, and for a
missing or unreadable file that text is the file's path on the server
(`[Errno 2] No such file or directory:
'/audiobooks/Author/Title.epub'`). #1422 cuts a non-admin's download
path down to the file name everywhere else, so this put it back. The
route now answers "Could not read the file" and the detail stays in the
log.
`should_bypass_proxy` logged the whole request URL at debug when it
could not be parsed. URLs can carry credentials, so it logs only the
exception now.
Each change has a test that fails without it. The full non-e2e suite,
ruff and basedpyright pass.
Clear Completed can fail outright with "Only terminal requests can be
dismissed". `dismiss-many` answers 409 and dismisses nothing when any
one request or download in the batch is not terminal. The list a user
clears from can be older than the server, so a request that was reopened
or retried since, or a download that started again, takes the whole
batch down with it. The earlier fix for pending requests (#1438) filters
on the client, which only helps when the list is current.
`dismiss-many` now skips such an item and carries on with the rest. The
response adds `skipped_item_keys` when something was left out, and
`count` is what was actually dismissed. A single `dismiss` of a
non-terminal item still answers 409.
I hit it with a request whose download had completed while the request
itself was still pending, but any reopen between the list and the click
does it.
This touches the same handler as #1507, so whichever lands second needs
a small rebase.
Tests: a batch with a finished and a reopened request dismisses the
finished one and reports the other, and a batch with a finished and a
running download does the same. Ruff, basedpyright and the backend suite
pass (3933 passed, e2e excluded).
Co-authored-by: CaliBrain <calibrain@l4n.xyz>
Two small fixes in `activity_routes.py`.
**Another user's item answers like a missing one.** Dismissing a task or
request id answered 404 when it didn't exist and 403 when it belonged to
someone else, so a user could test whether an arbitrary id exists. A
non-admin now gets the same 404 for both. In dismiss-many a foreign key
is reported with the missing ones and the rest carry on, and the owner
is still written to the log. Admins are unchanged.
**A foreign row in someone's viewer state no longer gives a 500.** If a
non-admin's viewer state referenced another user's download or request
row, building history raised a `RuntimeError`. The row is now skipped
with a warning.
Not changed: history still raises when a row that viewer state points to
has been deleted. That is a different case, and I didn't want to widen
this.
Tests: the existing forbidden test now expects the 404 and the log
context, and four new ones cover a foreign download and a foreign
request answering like a missing one (including dismiss-many), an admin
still dismissing a foreign row, and history skipping foreign rows. Ruff,
basedpyright and the backend suite pass.
Download to Browser only ever sent the first file of a multi-file
release
(audiobook chapters, packs): the folder output returned final_paths[0]
and
/api/localdownload served that one path.
The folder output now keeps every final path on the in-memory task
(download_paths), and /api/localdownload streams them as one
uncompressed
ZIP named after the book when there is more than one. The archive is
never
staged on disk or held in memory, and ZIP64 covers large audiobooks.
History still stores only the first path, so a re-download from history
after a dismiss or restart serves a single file, as before.
Also brings every linter back to green on all files:
- pre-commit: bump the ruff hook to v0.17.1 to match uv.lock;
ruff-format
the Python examples in docs/dev and skills
- ruff: fix the remaining issues in skills/shelfmark/download_books.py;
format base_handler.py
- basedpyright: Optional and attrs errors in databazeknih.py
- knip: drop 31 unused exports, reuse ColumnColorHint in colorMaps
- oxfmt 0.72: styles.css
- direct download: fix the "Welib, mirrors" typo and the test that still
expected the pre-OceanofPDF message
Fixes#1467
qBittorrent over HTTPS (#1417)
qbittorrent-api ignores the scheme in `host` and probes HTTP and HTTPS
itself. When the HTTPS probe fails, for instance on a certificate
Shelfmark doesn't trust, it falls back to plain HTTP against the TLS
port, and a reverse proxy answers "400 The plain HTTP request was sent
to HTTPS port". The download client and the Test Connection button now
pass FORCE_SCHEME_FROM_HOST for https:// URLs, so the configured scheme
is used as-is and a certificate problem is reported as one. http:// and
bare host:port URLs keep the probe: normalization adds http:// to a bare
host, and the probe follows a proxy's redirect to HTTPS.
Release sources leaked to non-admins (#1418)
A download queued from an approved request carries the release an admin
picked, and the requester could read it from their own activity feed:
the request's release_data (source_id, indexer, info URL, torrent
attributes), the download's full server path, and the download id
itself, which for Prowlarr is "<indexer id>:<guid>" and for a private
tracker is a URL into it. That id keys every status payload, the
/api/localdownload query and the cover proxy URL, so hiding
release_data alone would not have closed the leak.
For non-admin viewers:
- request rows keep only the release fields the activity cards display
- download_path is reduced to the file name, which the browser download
reveals anyway
- request-linked downloads are addressed by an opaque id (a keyed HMAC
of the task id) in the snapshot, history, /api/status, queue order,
active downloads, dismissed keys, cover URLs and the websocket status
and progress events. Routes that take a download id translate it
back, searching only the caller's own downloads.
Downloads a user queued directly keep their real id: the user picked
that release from search results that already showed it, and the
release list matches its buttons to the queue by that id. Admin views
are unchanged. The activity sidebar now links a fulfilled request to
its download by request_id instead of release_data.source_id.
Closes#1417Closes#1418
`/api/localdownload` resolves the file through the live queue and
returns it before checking who owns the task; the owner check only runs
on the download-history fallback, once the task has aged out of the
queue. Task ids are source ids, so two users who searched the same book
can end up with the same id.
This applies the same rule on the queue path, reusing the
`_task_owned_by_actor` helper the cancel/retry/priority routes already
use, so both paths answer a non-owner with the same 404. Admin behaviour
is unchanged.
The 404 matches what this endpoint's history path already returns for a
non-owner rather than the 403 `download_not_owned` the
cancel/retry/priority routes use, happy to switch it if you prefer
consistency with the siblings instead.
## Verification
- `tests/core/test_activity_routes_api.py`: the owner still receives
their queued file; a different user receives 404. The new case fails on
current main and passes here; the existing history-fallback test is
unchanged.
- Full suite (3148), ruff, ruff format, basedpyright, vulture green.
DownloadHistoryService.record_download and updated the single production
caller, but not the eleven in the test suite, leaving main red with 32
failures. Pass None, which is what the pre-#1336 behaviour recorded.
For the Blackhole handoff (#1345): add_download publishes the torrent
before the cancel check runs, and BlackholeClient.remove() is a no-op,
so
the watcher picks the file up regardless. Reporting a bare "Cancelled"
hid
that from the user. Name the completed handoff in the cancellation
message
instead, drop the _handle_cancelled_download call whose usenet branch
cannot apply to a handoff-only client, and record why the orchestrator
no
longer verifies HandoffResult.path.
Finally, make tests/direct_download a package: test_libgen_extract.py
imports tests.libgen.sample_html across test directories, so without an
__init__.py pytest named its modules by bare basename and a same-named
module elsewhere would collide.
- Clean up a few uses of config options that may miss the env variable
if this is set
- Add enhanced retry availability utilising the DB to persist download
errors / retries across restarts, request failures, and pass Prowlarr
detail through the download task to maintain retry data.
- Strip back entrypoint permissions for less intensive chown operations.
Fixes#796
- Updated frontend CSS to Tailwind v4
- Reverted socket IO origin restriction
- Fixed search queries not persisting after auth redirect
- Move advanced search options to left UI selector
- Unlock IRC source to be used for audiobook content_type
- Tweaked security settings env var syncing to be prioritised
- Fix AA "all languages" query generation
- Added language-free AA query as second fallback in case of no results
- Testing moving SeleniumBase scratch files to /tmp via symlink
- Added enhanced logging for activity dismissals and other events
- Removed iFrame restrictions
- Added canonical per-user visibility of requests and downloads via new
activity view table. Users get fully independent activity and history
views, while admins still see all.
- Replaces janky frontend + backend combination
Two-phase download history: downloads are now recorded in the DB at
queue time (not just at terminal time), eliminating the need to
reconstruct metadata in the terminal hook and removing the
`_is_graduated_request_download()` request-scan mess
- Much simpler handling of downloads in the activity sidebar, and
improved storage, persistence and UI behavior.
- Replace `ActivityService` with direct storage on
`DownloadHistoryService` and `download_requests` and removes the
activity_log/activity_dismissals tables
- Simplify no-auth mode by removing the fake user row pattern, handled
internally
- Add local download fallback so history entries can still serve files
after tasks leave the queue
- Downloads, requests and history are now entirely persistent between
updates / restarts, and correctly tied to each user.
- Add certificate validation setting
- Fix some OIDC providers not linking emails to local users
- Reintroduce sort by peers option for prowlarr results
- Fix "All languages" search query reverting to default language
- Fix download/request dismissal with multiple admin users
- Fix download / request behavior on details modal
- Refactored activity backend for full user-level management, using the
db file
- Revamped the activity sidebar UX and categorisation
- Added download history and user filtering
- Added User Preferences modal, giving limited configuration for
non-admins - replaces the "restrict settings" config option.
- Many many bug fixes
- Many many new tests
splitsec2
Alex