Fixes#1561, with thanks to @tmussche for the write-up, which had the
cause and the fix. When the files reach Shelfmark through a sync
(Resilio in the report), the wait for the completed path accepts the
folder once two samples a few seconds apart look the same. A sync that
copies in batches can pause longer than that, so 10 of 23 files were
imported and some of those were still partial copies.
For qBittorrent the client can say what the finished download holds, so
the wait now checks that. `DownloadClient` gets an optional
`get_expected_files()` that returns `ExpectedFile(name, size)` entries,
or None when the client cannot say, which is the default and keeps every
other client as it was. qBittorrent answers it from
`/api/v2/torrents/files`, leaving out files the torrent does not
download (priority 0), zero-byte entries and BEP 47 padding files.
`_wait_for_completed_path` asks once, and while any of those files is
missing or smaller than its final size it keeps waiting, using the same
timeout as before. When the wait runs out the error now says how far it
got, for example "Completed download is not fully on disk yet (10 of 23
files)." Once everything is there the existing stability check still
runs, so nothing is accepted that was accepted before.
Names from the client start with the torrent's top-level folder, so each
file is looked up with and without it, which also covers a torrent saved
without a subfolder. A single-file torrent is checked by size too, since
a file that is still syncing exists at a smaller size.
Two things I would like a second look at: the comparison is on exact
size, so a setup where something rewrites the files after the torrent
finishes would never match and would run into the timeout; and the file
list is read once per wait, not on every poll. If the client does not
answer, nothing changes.
I also moved the nested authenticated GET in the path derivation into a
`_get_with_auth` method so both calls share it, and added a sentence
about this to `docs/configuration.md`. Tests: eight for the wait
(partial folder, truncated file, files arriving during the wait,
complete folder, no top-level folder, single file both ways, and a
client without a list), and six for the qBittorrent file list. The wait
tests that depend on the new behaviour fail on current main. Full suite
passes.
These are the leftovers from the docs check, the fixes that did not
belong in the other three.
Claude did the comparison and wrote the changes here too, so I would
still like a human look before it merges. The one new piece of prose
worth a read is `troubleshooting.md`. It was a TODO page that two other
pages link to, so I filled it with only what I could confirm in the
code: where the logs go, `LOG_LEVEL`, the `DEBUG=true` bundle from the
bug report template, and reporting an issue. Happy to drop it back to a
stub if you would rather write that one yourself.
- `configuration.md`, `custom-scripts.md`, `users-and-requests.md`:
Booklore is Grimmory in the UI now. The stored mode value is still
`booklore`, and the custom script doc says so. The hardlink heading now
matches the "Hardlink Book Torrents" and "Hardlink Audiobook Torrents"
checkboxes.
- `custom-scripts.md`: the email output runs scripts too
(`phase=post_email`, `output.mode=email`, details under
`output.details.email`). It was missing from the phase list.
- `users-and-requests.md`: a delivery that fails with no retry available
goes back to pending on its own, and the admin reopen action is for
rejected requests. The page said an admin reopens a failed delivery.
Also noted that rules cannot use Request Book.
- `hardcover-sync.md`: added the missing
`AUTO_DOWNLOAD_AUDIOBOOK_MIN_SIZE_PERCENT` row and a sentence in the
ranking paragraph.
- `api-access.md`: an admin session also reads `/api/stats`, and what
the 403 says.
- `installation.md`: points at the Lite, Tor and WireGuard variants.
Docs only, no code touched.
- Add configurable completed-path wait for external clients : - Add a
configurable Advanced setting, DOWNLOAD_CLIENT_COMPLETED_PATH_TIMEOUT,
for how long Shelfmark waits after a torrent or usenet client reports
completion before treating the completed path as missing. - Keep the
default at the existing 60-second grace period, with a maximum of 3600
seconds
- Add e2e testing
- Add e2e testing
Should fix#861
- Updated mirror selection
- Removed built-in mirror options, users must provide their own
configurations
- Set Universal search to default, added ability to disable direct
source
- Updated documentation
- Updated makefile
- Adds a non-root startup path at user 1000:1000 - skips privilege
escalation and ownership checks. Works e.g. for kubernetes deployments
(user 1000:1000 and runAsNonRoot enabled).
- Remove startup check/chown commands for user-owned folders. Checks can
be done with a "Test destination" button in settings which performs a
test write. Users are responsible for fixing their own permissions.
- Update docs
- Add new booklore API file formats
- Renamed cookie for better login persistence with reverse proxy
- Updated fs.py to try hardlink before atomic move from tmp dir
- Fix transmission URL parsing
- Fix scenario where file processing of huge files starves the
healthcheck
- Large enhancements to custom scripting, including passing JSON
download info, more consistent activation across output types,
decoupling from staging behavior, and added full documentation.
- Added new book output option **upload to Booklore**, available in
download settings
- Got annoyed at my messy processing code while implementing Booklore so
refactored the whole thing
- Full black box file processing testing with randomised configuration
- Deluge: Connect via WebUI auth for simplified setup
- Added env vars documentation, auto generated via script, and unlocked
most settings to be used as env vars
splitsec2
CaliBrain
Alex
Alex