Commit Graph
4 Commits
Author SHA1 Message Date
CaliBrain 49aeec97cd fix(download): stop deleting user data and losing completed downloads (#1511)
Fixes six defects. Each one breaks downloads or deletes user data, and
each has a regression test that fails on 10710ab and on 77b80eb (current
main).

- **Completed-path wait (#1464)**: `_wait_for_completed_path` only
returned a directory after two identical signatures of supported
formats.
- A directory holding only archives (a scene `book.rar` + `.nfo` +
`.sfv`) never got a signature and failed with "Completed download has no
stable supported files yet". Archives (`.zip`/`.rar`, via `is_archive`)
now count toward the signature, so post-processing extracts them again,
as in v1.4.0.
- With `DOWNLOAD_CLIENT_COMPLETED_PATH_TIMEOUT=0`, `max_attempts` is 1,
so every directory failed. One sample with book files or archives is now
accepted when no retry is possible.
- **Remove & Delete Files (#1401)**: `post_process_cleanup` called
`remove(id, delete_files=True)` on torrents matched by `find_existing`
that Shelfmark never added, deleting the user's seeding data.
`_cleanup_refs` now records `added_by_shelfmark`. A pre-existing torrent
(complete, or in progress and joined) is removed but keeps its files.
- **Cancelling an unstarted torrent (#1421)**:
`_remove_unstarted_torrent` deleted with files whenever `progress == 0`.
That included `DownloadStatus.error()`, which always reports 0, and
checking states that are re-verifying data already on disk. The torrent
must now be QUEUED, PAUSED or DOWNLOADING. rTorrent's
`remove(delete_files=True)` always returns False, so a new
`DownloadClient.can_delete_files` flag (False on rTorrent) makes it
remove without files instead.
- **Queue grace overwritten**: when a QUEUED torrent's progress rose on
the same poll, the 900 s queue grace was immediately replaced by a
shorter stall window, because the orchestrator keeps one deadline. The
stall window is now requested only when it is longer than what is left
of the queue grace.
- **`drop_ignored_files` (#1480)**: the suffix rule `w.endswith("/" +
rel)` dropped a kept root `Book.epub` when `Extras/Book.epub` was
skipped. Each skipped entry now drops at most one file: the exact path,
else the file sharing the most trailing components, else a unique
basename, else an AudiobookBay glued `"<folder> <file>"` name (the
fallback `match_plan_to_files` uses, matched on folder plus file so
chaptered `Book 2 01.mp3` picks the right folder). Ties drop nothing.
- **Email and BookLore outputs** ignored `task.ignored_files` and sent
every prepared file. Both now apply `drop_ignored_files` once after
`prepare_output_files`, and report "No book files found" if the user
skipped everything. It is applied in the outputs rather than in
`prepare` because the folder output already filters in
`transfer_book_files`, and filtering twice could match a kept file on
the second pass.

- `make python-checks`: ruff, ruff format, basedpyright (backend +
tests), vulture all pass.
- `uv run pytest tests/ -m "not integration and not e2e"`: 4054 passed
on the rebased branch.
- New tests run against unfixed 77b80eb and 10710ab: all 18 new
regression cases fail, and the 11 existing post-process-cleanup tests
also fail there because they now build the 4-tuple cleanup ref.
2026-10-21 01:56:19 -04:00
splitsec2andCaliBrain 82c8c91c8b refactor(clients): read the stall window with a type check instead of a pyright ignore (#1498)
The only `pyright: ignore[reportArgumentType]` in `shelfmark/` is one I
added in #1426, and I was being lazy when I did. I hit the same error on
another PR and replaced the ignore there with a real check, so this does
the same for the original.

`_stall_window_seconds` reads the setting as an `object`. It now narrows
to int, float or str before calling `float()`, and a bad string falls
back to the default as before. A non-numeric value of another type
(None, a list) falls back to the default too, which used to happen
through a `TypeError`.

No behaviour change. The existing range test still passes and I added
`None` and a list to it. basedpyright is at 0 errors, ruff is clean and
the backend suite passes.

Co-authored-by: CaliBrain <calibrain@l4n.xyz>
2026-10-20 23:28:42 -04:00
splitsec2 47707b827a feat(download): make the torrent stall window a setting (#1426)
#1420 gave a torrent 15 minutes to get going and 15 minutes after each
step forward. That suits my swarms, but other people will want it
shorter or longer, so this makes the number a setting.

`TORRENT_STALL_TIMEOUT_MINUTES` sits in Download Clients with the other
torrent options and takes 5 to 60. The default is 15, so nothing changes
unless someone sets it. 5 is the orchestrator's own stall timeout, so
choosing it asks for no extra time and gives the old behaviour. The
value can arrive as text from an env var, so anything that isn't a
number falls back to the default and anything out of range is clamped.

The orchestrator caps one activity grace at 16 minutes, so I raised that
to an hour to fit the top of the range. As far as I can tell the
protection bypass is the only caller that asks for more than a few
minutes, but I haven't audited every caller. If you'd rather keep the
cap, I can lower the maximum to 16 minutes instead.

Tests cover the default, 5 and 30, text values, out of range values and
NaN, the start and movement windows following the setting, and the
setting showing up in the Download Clients tab. I broke each rule on
purpose to check a test catches it. The full suite fails the same tests
with and without this change (the bypass and seleniumbase tests don't
run in my environment).
2026-10-20 00:58:24 -04:00
splitsec2 88f73e3bc3 fix(download): do not cancel a queued torrent as stalled (#1420)
qBittorrent only runs a few downloads at once (`max_active_downloads`
defaults to 3) and holds the rest as `queuedDL`. Shelfmark's stall timer
cancels a download after five minutes without a changed status or
progress. A queued torrent reports "Queued" on every poll, so it looks
idle, and Shelfmark cancels a download that was only waiting for a slot.

I run Shelfmark against the same qBittorrent that Sonarr, Radarr and
Lidarr use, which I assume is a common setup. Their torrents and
Shelfmark's share one limit, so anything that fills the slots makes
Shelfmark's torrent wait. In my case three torrents that never fetched
metadata held every slot. I'd expect a busy Sonarr queue to hold them
the same way, but I haven't reproduced that. Once the stall timer had
cancelled the waiting downloads they stayed in the client, still queued
behind the same blockers. On my instance 10 AudioBookBay downloads were
cancelled while qBittorrent still had them queued.

This changes the torrent poll loop. When the client reports a queued
torrent, it asks for an activity grace (the mechanism `html_get_page`
already uses for protection bypasses) and releases it as soon as the
torrent leaves the queue. The grace is renewed every ten minutes and
stops at two hours, so a queue that never moves still ends. A torrent
that is downloading and moving is timed as before until the new window
below applies.

The second commit sets the stall message before the cancel runs. The
terminal hook copies the message into the history row, and it was set
afterwards, so a download the timer ended was recorded with the last
poll's "Queued". That left no way to tell it apart from a person
cancelling.

The last two commits are about slow torrents. The timer only counts a
change in progress, so a torrent that needs more than five minutes to
fetch metadata or find its first peer is cancelled before it has any
progress to count, and one that moves a few megabytes at a time can be
cancelled between bursts. I hit this with a torrent that had a single
peer. A torrent now gets one 15 minute window the first time it is seen
outside the queue, and each step forward in progress renews it. One that
makes no progress for 15 minutes is still cancelled, so a dead torrent
now takes about 15 minutes to end instead of five. That trade is the
part I'm least sure about, so the number is easy to change.

Tests cover the grace being requested once, released when the torrent
starts, renewed, and capped, the order of the message and the cancel,
and the start and movement windows. The full suite passes, and I broke
each rule on purpose to check a test catches it.

The two hour ceiling is a guess. If you'd rather have a setting for it,
or a different number, I'm happy to change it.
2026-10-02 22:12:25 -04:00