A torrent that finishes with the wrong kind of files in it holds the
download for the whole completed-path timeout (60 seconds by default, up
to an hour) and then fails with "Completed download has no stable
supported files yet." An ebook task whose torrent holds only an `.m4b`,
an audiobook task that got an `.epub`, or a `.pdf`-only release with the
default format list all land here. In v1.4.0 these failed with the
specific format error from post-processing, but the wait for the sync to
settle (#1464) only counts files of the task's own type, so it keeps
polling for files that are never going to appear.
If the directory has no supported files, `_wait_for_completed_path` now
looks for finished book or audio files that this task does not accept
(the audiobook formats, the default ebook formats and pdf). When it
finds some and they are the same on two polls in a row, it returns the
path so post-processing can say what is wrong with the formats. Cover
art and notes alone do not count, because that is what a directory looks
like while the real files are still syncing over a remote mount, so
those keep waiting as before. A wrong-type file that is still growing
keeps waiting too. With no wait configured there is only one sample, and
a wrong-type file is handed on straight away.
The existing test that pins "no book files, no wait configured" uses a
jpg, so it stays as it is. I extended the test helper to take a content
type, which the wait tests never passed before. Four new cases fail
without the change (a wrong-type directory for ebook, audiobook and pdf
tasks), and two more check that sidecar-only and still-growing
directories keep waiting. Full suite passes.
Fixes#1561, with thanks to @tmussche for the write-up, which had the
cause and the fix. When the files reach Shelfmark through a sync
(Resilio in the report), the wait for the completed path accepts the
folder once two samples a few seconds apart look the same. A sync that
copies in batches can pause longer than that, so 10 of 23 files were
imported and some of those were still partial copies.
For qBittorrent the client can say what the finished download holds, so
the wait now checks that. `DownloadClient` gets an optional
`get_expected_files()` that returns `ExpectedFile(name, size)` entries,
or None when the client cannot say, which is the default and keeps every
other client as it was. qBittorrent answers it from
`/api/v2/torrents/files`, leaving out files the torrent does not
download (priority 0), zero-byte entries and BEP 47 padding files.
`_wait_for_completed_path` asks once, and while any of those files is
missing or smaller than its final size it keeps waiting, using the same
timeout as before. When the wait runs out the error now says how far it
got, for example "Completed download is not fully on disk yet (10 of 23
files)." Once everything is there the existing stability check still
runs, so nothing is accepted that was accepted before.
Names from the client start with the torrent's top-level folder, so each
file is looked up with and without it, which also covers a torrent saved
without a subfolder. A single-file torrent is checked by size too, since
a file that is still syncing exists at a smaller size.
Two things I would like a second look at: the comparison is on exact
size, so a setup where something rewrites the files after the torrent
finishes would never match and would run into the timeout; and the file
list is read once per wait, not on every poll. If the client does not
answer, nothing changes.
I also moved the nested authenticated GET in the path derivation into a
`_get_with_auth` method so both calls share it, and added a sentence
about this to `docs/configuration.md`. Tests: eight for the wait
(partial folder, truncated file, files arriving during the wait,
complete folder, no top-level folder, single file both ways, and a
client without a list), and six for the qBittorrent file list. The wait
tests that depend on the new behaviour fail on current main. Full suite
passes.
Fixes six defects. Each one breaks downloads or deletes user data, and
each has a regression test that fails on 10710ab and on 77b80eb (current
main).
- **Completed-path wait (#1464)**: `_wait_for_completed_path` only
returned a directory after two identical signatures of supported
formats.
- A directory holding only archives (a scene `book.rar` + `.nfo` +
`.sfv`) never got a signature and failed with "Completed download has no
stable supported files yet". Archives (`.zip`/`.rar`, via `is_archive`)
now count toward the signature, so post-processing extracts them again,
as in v1.4.0.
- With `DOWNLOAD_CLIENT_COMPLETED_PATH_TIMEOUT=0`, `max_attempts` is 1,
so every directory failed. One sample with book files or archives is now
accepted when no retry is possible.
- **Remove & Delete Files (#1401)**: `post_process_cleanup` called
`remove(id, delete_files=True)` on torrents matched by `find_existing`
that Shelfmark never added, deleting the user's seeding data.
`_cleanup_refs` now records `added_by_shelfmark`. A pre-existing torrent
(complete, or in progress and joined) is removed but keeps its files.
- **Cancelling an unstarted torrent (#1421)**:
`_remove_unstarted_torrent` deleted with files whenever `progress == 0`.
That included `DownloadStatus.error()`, which always reports 0, and
checking states that are re-verifying data already on disk. The torrent
must now be QUEUED, PAUSED or DOWNLOADING. rTorrent's
`remove(delete_files=True)` always returns False, so a new
`DownloadClient.can_delete_files` flag (False on rTorrent) makes it
remove without files instead.
- **Queue grace overwritten**: when a QUEUED torrent's progress rose on
the same poll, the 900 s queue grace was immediately replaced by a
shorter stall window, because the orchestrator keeps one deadline. The
stall window is now requested only when it is longer than what is left
of the queue grace.
- **`drop_ignored_files` (#1480)**: the suffix rule `w.endswith("/" +
rel)` dropped a kept root `Book.epub` when `Extras/Book.epub` was
skipped. Each skipped entry now drops at most one file: the exact path,
else the file sharing the most trailing components, else a unique
basename, else an AudiobookBay glued `"<folder> <file>"` name (the
fallback `match_plan_to_files` uses, matched on folder plus file so
chaptered `Book 2 01.mp3` picks the right folder). Ties drop nothing.
- **Email and BookLore outputs** ignored `task.ignored_files` and sent
every prepared file. Both now apply `drop_ignored_files` once after
`prepare_output_files`, and report "No book files found" if the user
skipped everything. It is applied in the outputs rather than in
`prepare` because the folder output already filters in
`transfer_book_files`, and filtering twice could match a kept file on
the second pass.
- `make python-checks`: ruff, ruff format, basedpyright (backend +
tests), vulture all pass.
- `uv run pytest tests/ -m "not integration and not e2e"`: 4054 passed
on the rebased branch.
- New tests run against unfixed 77b80eb and 10710ab: all 18 new
regression cases fail, and the 11 existing post-process-cleanup tests
also fail there because they now build the 4-tuple cleanup ref.
## Summary
```text
wait for completed download path
regular file -> keep existing immediate handling
directory -> inspect files enabled for this task's content type
wait until the supported-file signature is stable across checks
```
Fixes the remote-sync race in #1463. A mapped torrent folder can appear
before its large media file finishes copying, so path existence alone
does not mean the download is ready.
## Evidence
- **Before:** In #1463, qBittorrent reported a multi-file audiobook
complete while the destination contained only `.cue` and `.jpg`;
Shelfmark immediately post-processed the folder and failed with “No book
files found in download.” Retrying after the `.m4b` sync finished
imported it.
- **After:** A regression test starts with the cue sheet and cover
present, then simulates the `.m4b` file appearing and growing. Path
waiting returns only after the supported file's size and modification
time remain stable across checks.
- Credential-free isolated `python -m pytest
tests/prowlarr/test_remote_path_mappings.py -q -n 0 -o addopts=''`: 11
passed.
- `ruff check shelfmark/download/clients/base_handler.py
tests/prowlarr/test_remote_path_mappings.py`: passed.
## Merge Danger
**Door:** Two-way
**Blast Radius:** Download handling
The wait uses the existing completed-path timeout and leaves regular
single-file downloads on their current path.
Co-authored-by: Sean Li <22557931+sean0407@users.noreply.github.com>
- Add configurable completed-path wait for external clients : - Add a
configurable Advanced setting, DOWNLOAD_CLIENT_COMPLETED_PATH_TIMEOUT,
for how long Shelfmark waits after a torrent or usenet client reports
completion before treating the completed path as missing. - Keep the
default at the existing 60-second grace period, with a maximum of 3600
seconds
- Add e2e testing
- Add e2e testing
Should fix#861
- Migrate download client handling from /prowlarr to /download. Moves
all torrent/usenet handling to app-level and gives ABB this
functionality.
- ABB Scraper now uses shared HTTP infrastructure instead of raw
requests, adding retry and proxy support
- Added author, age and bitrate info to ABB search results
- Added "best match" sorting option for releases
- Added size and bitrate sorting options for ABB
- Removed bundled default ABB hostname, must be configured by the user
- Added URL normalisation for ABB hostname
- Rearranged settings UI, moved download clients to its own section.
- More tests