- Added new book output option **upload to Booklore**, available in
download settings
- Got annoyed at my messy processing code while implementing Booklore so
refactored the whole thing
- Full black box file processing testing with randomised configuration
- Deluge: Connect via WebUI auth for simplified setup
- Added env vars documentation, auto generated via script, and unlocked
most settings to be used as env vars
This adds in rtorrent for
https://github.com/calibrain/shelfmark/issues/420
The one weird thing I noticed is that the download path needs to be the
same for both (that's not how I typically set it up)
But it definitely adds to rtorrent and gives progress.
**rTorrent client integration:**
* Added a new `RTorrentClient` class in
`shelfmark/release_sources/prowlarr/clients/rtorrent.py` that implements
the download client interface for rTorrent using XML-RPC, supporting
adding, removing, and querying torrent status.
* Registered the rTorrent client in the client registry in
`shelfmark/release_sources/prowlarr/clients/__init__.py`.
**Settings and configuration:**
* Extended the Prowlarr client settings UI and backend
(`shelfmark/release_sources/prowlarr/settings.py`) to add rTorrent as a
selectable client, provide rTorrent-specific configuration fields (URL,
username, password, label, download directory), and implement a
connection test action.
[[1]](diffhunk://#diff-052272b85804cb61162870f262cc7544ef321596ff3ebf08117a6c25afaa3ec5R390)
[[2]](diffhunk://#diff-052272b85804cb61162870f262cc7544ef321596ff3ebf08117a6c25afaa3ec5R539-R582)
[[3]](diffhunk://#diff-052272b85804cb61162870f262cc7544ef321596ff3ebf08117a6c25afaa3ec5R198-R225)
**Test environment and scripts:**
* Updated `docker-compose.test-clients.yml` to add an rTorrent service
for local testing, including configuration, ports, and documentation
updates.
[[1]](diffhunk://#diff-a9fe4200dec6a29947e21c338305d04c8b64a7bddd9b0e519f4ab5382c478ba6R17-L19)
[[2]](diffhunk://#diff-a9fe4200dec6a29947e21c338305d04c8b64a7bddd9b0e519f4ab5382c478ba6R39)
[[3]](diffhunk://#diff-a9fe4200dec6a29947e21c338305d04c8b64a7bddd9b0e519f4ab5382c478ba6R66)
[[4]](diffhunk://#diff-a9fe4200dec6a29947e21c338305d04c8b64a7bddd9b0e519f4ab5382c478ba6R164-R181)
* Enhanced `scripts/test_clients.py` to include rTorrent in the test
suite, with logic for connecting, adding, and removing torrents via
XML-RPC.
[[1]](diffhunk://#diff-c7146552cddc9665e380aec1473363fd8592535ab85f06443478464da8f5a99eR26)
[[2]](diffhunk://#diff-c7146552cddc9665e380aec1473363fd8592535ab85f06443478464da8f5a99eR57)
[[3]](diffhunk://#diff-c7146552cddc9665e380aec1473363fd8592535ab85f06443478464da8f5a99eR88-R90)
[[4]](diffhunk://#diff-c7146552cddc9665e380aec1473363fd8592535ab85f06443478464da8f5a99eR395-R468)
[[5]](diffhunk://#diff-c7146552cddc9665e380aec1473363fd8592535ab85f06443478464da8f5a99eR492)
Closes https://github.com/calibrain/shelfmark/issues/420
- Changes the auth settings to support more than two auth types
- Added a proxy auth type with settings for user and optionally group
headers
- Added a global middleware `proxy_auth_middleware` to handle proxy auth
(it does nothing if any other auth mode is set)
- Added support for proxy auth to `get_auth_mode`, `login_required`,
`api_login/out`, and `api_auth_check`
- Added a backend check to make protect the API for settings when admin
is required
---------
Co-authored-by: Joshua Tag Howard <git@jthoward.dev>
Co-authored-by: Alex <alex.bilbie1@gmail.com>
- Added more robust config directory initialisation and file creation
- Fixed category fallback not triggering correctly for one content type
if another is cached
- fix(fs): handle NFS permission errors with robust fallback - Catch
PermissionError/OSError(EPERM) in atomic_move and atomic_copy -
Implement layered fallback: shutil.copyfile (content only) -> system
mv/cp - Add _perform_nfs_fallback and _system_op helpers to reduce
duplication - Set fallback logging to DEBUG to reduce spam on NFS mounts
- make os mv and cp commands non interactive (-f)
Fix for #423
- Catch PermissionError/OSError(EPERM) in atomic_move and atomic_copy
- Implement layered fallback: shutil.copyfile (content only) -> system
mv/cp
- Add _perform_nfs_fallback and _system_op helpers to reduce duplication
- Set fallback logging to DEBUG to reduce spam on NFS mounts
- Restructured the file processing settings to make more coherent
- Added hide settings UI for CWA non-users
- Added sort options for ReleaseModal listview entries
- Added separate Audiobook category selection for download clients -
- Added alternative file processing mode. Save files directly into a
library folder and set up file names / directories based on user
preference.
- Uses template based naming and directory creation. E.g. {Author} /
{Series} {Title} {Part} etc. Works for saving correctly to libraries
such as Audiobookshelf.
- Use torrent hardlinking directly into library directories.
- Added a `content_type` field to switch metadata providers, prowlarr
search category, and file formats on the frontend.
- Switch between Book / Audiobook in the header dropdown.
- Only Prowlarr declares itself as a supported audiobook source.
Internally switches to category 3030 for searches.
- Updated torrent client handling to accept and process magnet links
- Prowlarr: Added automatic fallback to search without category filter
when indexers return no results with book category (7000), improving
compatibility with indexers that don't support category filtering
- Prowlarr: Hide language filter in UI since Prowlarr has unreliable
language metadata
- Bypass: Refactored internal bypasser with code cleanup, extracted
helper functions, and added health check capability
- Bypass: Added fingerprint module for screen size handling
- qBittorrent: Fixed connection test to use web API version instead of
app version
- Frontend: Added supported_filters config to control which filters
display per source
- Auth: Improved CWA database path validation (now uses Path object
properly)
- Updated Selenium to 4.45.6. Includes various crash and memory leak
fixes, plus new bypasser methods
- Bypasser now uses CDP captcha solving as priority - Faster, more
efficient, no PyAutoGUI needed. Fallback to existing methods.
- Better detection and cleanup of old Selenium instances to save memory.
- Added Hardcover graphQL API header detection
- Added AA download counts in details modal
- More robust switching of internal/external bypasser, fixed settings UI
toggle behavior.
Fixed an issue where hitting save with an empty username would fail
silently. This led users to think their changes were saved when they
actually weren't. Now it automatically defaults to "admin" if you set a
password but leave the username blank, making the save process reliable.
Cheers,
Your swiss librarian ;)
Reordered the auth priority because the old logic was misleading. It
would automatically default to "builtin" mode if credentials existed,
completely ignoring the CWA database even if you wanted to use it. You
wouldn't even notice it was happening until you realized the DB
integration wasn't actually active. This fix ensures explicit CWA auth
takes priority so you don't have to wipe your settings just to switch
methods.
Cheers,
Your swiss librarian ;)
## Headline features
### Prowlarr plugin - search trackers and download usenet/torrent books
- Search any usenet/torrent tracker via Prowlarr, returns books within
Universal search
- Configure download clients in the app settings (Qbittorrent, Deluge,
Transmission, NZBget, SABnzbd)
- Unified download and file handling within the app, same as AA.
### IRC plugin
- Search IRCHighway #ebooks channel for books and download right in the
app.
- No setup needed
- Credit to OpenBooks for the broad idea and inspiration for best
practices for ebook-specific search and download.
### Google Books Metadata Provider
- Create a Google Cloud API key and use Google Books as a metadata
provider
- Not the best source (Hardcover is still recommended), but another
option and further redundancy for universal search
### Book series support
- New "Series" search field in Hardcover provider
- "Series order" sort option - lists books in reading order
- "View Series" button in book details modal to search the full series
- Series info display (e.g., "3 of 12 in The Wheel of Time")
## Others:
- Better format filtering, helpful errors when formats rejected (e.g.,
"Found 3 ebooks but format not supported (.pdf). Enable in Settings >
Formats."
- Directory processing - Handles multi-file torrent/usenet downloads
properly
- Expand search toggle - Skip ISBN search to find more editions
- Filtered authors - Uses primary authors only (excludes
translators/narrators) for better search results
- Language multi-select - Filter releases by multiple languages
Docker / Build / Testing
- pip cache mounts - Faster Docker builds via BuildKit cache
- npm cache mounts - Faster frontend builds
- APT cleanup - Smaller final image size
- Added make restart command for quick restarts without rebuild
- New pytest-based test framework with proper configuration
(pyproject.toml)
- Unit tests for all download clients (qBittorrent, Transmission,
Deluge, NZBGet, SABnzbd)
- Bencode parsing tests
- Cache tests
- Integration tests for Prowlarr handler
- E2E test framework
- Further pass on settings UI, rearranging and adding further options
- Full RAR/ZIP support, including automatic unzipping and moving valid
file formats to ingest folder
- SOCK5 proxy support
- Full pass on the orchestrator to handle RAR/ZIP and category-specific
ingest dirs regardless of release source.
- Enhanced debug output to include new config JSON files
- Further ReleaseModal refinement
Also:
Adjusted Welib/Zlib/Libgen URLs to be dynamically generated via hash.
Fixed Zlib downloads and user agent flow. AA URLS are now fetched lazily
if another source is prioritised.
## Changelog
### 🌐 Network Resilience
- **Auto DNS rotation**: New `CUSTOM_DNS=auto` mode (now default) starts
with system DNS and automatically rotates through Cloudflare, Google,
Quad9, and OpenDNS when failures are detected. DNS results are cached to
improve performance.
- **Mirror failover**: Anna's Archive requests automatically fail over
between mirrors (.org, .se, .li) when one is unreachable
- **Round-robin source distribution**: Concurrent downloads are
distributed across different AA partner servers to avoid rate limiting
### 📥 Download Reliability
- **Much more reliable downloads**: Improved parsing of Anna's Archive
pages, smarter source prioritization, and better retry logic with
exponential backoff
- **Download resume support**: Interrupted downloads can now resume from
where they left off (if the server supports Range requests)
- **Cookie sharing**: Cloudflare bypass cookies are extracted and shared
with subsequent requests, often avoiding the need for re-bypass entirely
- **Stall detection**: Downloads with no progress for 5 minutes are
automatically cancelled and retried
- **Staggered concurrent downloads**: Small delays between starting
concurrent downloads to avoid hitting rate limits
- **Source failure tracking**: After multiple failures from the same
source type (e.g., Libgen), that source is temporarily skipped
- **Lazy welib loading**: Welib sources are fetched as a fallback only
when primary sources fail (unless `PRIORITIZE_WELIB` is enabled)
### 🛡️ Cloudflare & Protection Bypass
- **DDOS-Guard support**: Internal bypasser now detects and handles
DDOS-Guard challenges with dedicated bypass strategies
- **Cancellation support**: Bypass operations can now be cancelled
mid-operation when user cancels a download
- **Smart warmup**: Chrome driver is pre-warmed when first client
connects (controlled by `BYPASS_WARMUP_ON_CONNECT` env var) and shuts
down after periods of inactivity
### 🔌 External Bypasser (FlareSolverr)
- **Improved resilience**: Retry with exponential backoff, mirror/DNS
rotation on failure, and proper timeout handling
- **Cancellation support**: External bypasser operations respect
cancellation flags
### 🖥️ Web UI Improvements
- **Simplified download status**: Removed intermediate states
(bypassing, verifying, ingesting) — now just shows Queued → Resolving →
Downloading → Complete
- **Status messages**: Downloads show detailed status like "Trying
Anna's Archive (Server 3)" or "Server busy, trying next...", or live
waitlist countdowns.
- **Improved download sidebar**:
- Downloads sorted by add time (newest first)
- X button moved to top-right corner for better UX
- Wave animation on in-progress items
- Error messages shown directly on failed items
- X button on completed/errored items clears them from the list
### ⚙️ Configuration Changes
- **`CUSTOM_DNS=auto`** is now the default (previously empty/system DNS)
- **`DOWNLOAD_PROGRESS_UPDATE_INTERVAL`** default changed from 5s to 1s
for smoother progress
- **`BYPASS_WARMUP_ON_CONNECT`** (default: true) — warm up Chrome when
first client connects
### 🐛 Bug Fixes
- **Download cancellation actually works**: Fixed issue where cancelling
downloads didn't properly stop in-progress operations
- **WELIB prioritization**: Fixed `PRIORITIZE_WELIB` not being respected
- **File exists handling**: Downloads to same filename now get `_1`,
`_2` suffix instead of overwriting
- **Empty search results**: "No books found" now returns empty list
instead of throwing exception
- **Search unavailable error**: Network/mirror failures during search
now return proper 503 error to client
Hey, made the tweaks we discussed, plus a couple related fixes :)
- Removed APP_ENV entirely. All dev-specific functionality is enabled
via `DEBUG: true` env var
- Set secure cookie handling to false by default, added to the readme to
enable if exclusively using HTTPS connection
- Fixed healthcheck potentially not working with auth enabled
- Removed APP_ENV from docker compose files and made sure app.db lines
are included in all versions.
APP_ENV in people's existing composes should get ignored entirely and
will be put on the default env, so no issues when updating.
Alex
Patrick Veverka
CaliBrain
bischoffjeremy
Ronnoceel