CaliBrain cc1a95f965 Fix protection bypass cancelled by stall detection at exactly 300s (#1184)
A download that hits Cloudflare hung on "Bypassing protection..." for
five
minutes and then died, regardless of which bypasser was configured.

html_get_page() started a BypassHeartbeat thread to keep the download
marked
alive during a bypass, but the thread had no loop: it fired one status
event
and returned. Even with the loop restored it could not have worked,
because
update_download_status() dedupes identical (status, message) tuples and
returns before refreshing _last_activity, and the heartbeat re-sent the
byte-identical payload already emitted just above it.

So _last_activity was frozen for the whole bypass, while both bypassers
are
allowed to run longer than STALL_TIMEOUT (external FlareSolverr ~394s at
default settings, internal 420s per get() call). The watchdog always
won.
From a reporter's log: 403 at 07:04:33.390, cancelled at 07:09:33.987 -
exactly 300.000s, and 41s before the bypasser would have finished and
reported the real error, an HTTP 500 from FlareSolverr the user never
saw.

The regression is not one commit. 1f093de (#536) added the heartbeat and
the
dedup together and refreshed activity before the dedup return, so it
worked.
ff094be (#832) moved the refresh below that return while tightening
stall
detection for #823. 3a3a3ce (#845) then deleted the heartbeat's while
loop
to silence a B023 lint, removing the last evidence of intent.

The dedup itself is correct and stays: a keep-alive that ticks on a
timer
proves nothing about whether an operation is progressing, so letting it
refresh the stall clock would make a wedged download immortal. Split the
two
concerns instead.

Add shelfmark/download/activity.py. A long single-shot operation
declares its
own upper bound once, over a sentinel status carried on the existing
status_callback channel - so no new parameter has to be threaded through
every
handler, post-processor and output module. The orchestrator intercepts
the
sentinel in its per-task closure and records an absolute deadline in
_activity_grace, which stall detection honours alongside STALL_TIMEOUT.
The
grace never extends itself and is clamped to
_MAX_ACTIVITY_GRACE_SECONDS, so
an operation that overruns its own declared budget is still cancelled.

Each bypasser now reports max_duration_seconds() derived from its own
retry
and timeout settings, and http.py asks whichever is active, plus 30s of
slack
so the bypasser's own deadline expires first and the user sees its real
failure. On that path html_get_page() also emits
status_callback("error", ...)
rather than silently returning an empty page.

Three further fixes on the same code path:

- Extract the watchdog into _find_stalled_tasks() and
_cancel_stalled_task().
It was the only place holding _progress_lock across a call into
book_queue,
whose terminal-status hooks reach a sqlite write that gevent does not
patch,
blocking the hub and every download worker. It now holds the lock for
dict
  reads only.
- Bound _CDP_WORKER.run(), which waited with timeout=None while holding
the
module-wide LOCKED, so a single wedged in-process CDP session blocked
every
  subsequent bypass forever on non-Docker installs.
- Broaden the coordinator loop's except clause back to Exception, with
  escalating backoff. 8d98e12 (#868) narrowed it to a six-type tuple to
silence BLE001, which let gevent's LoopExit and similar kill the only
thread
driving the download queue - undoing #832's fix for #823 and resurfacing
it
  as #1166. GreenletExit and gevent.Timeout still propagate.

Fixes #1001
Refs #1166, #823
2026-08-11 02:25:28 -04:00
2026-04-12 12:39:15 +01:00
2026-04-13 11:40:41 +01:00
2026-04-12 12:39:15 +01:00
2024-12-16 22:48:26 +00:00
2026-06-23 15:38:46 -04:00

📚 Shelfmark: Book Search & Request Tool

Shelfmark

Note

This project is in a stable state as of May 2026 but is not under active maintenance.

Shelfmark is a self-hosted web interface for searching and requesting books and audiobooks across multiple sources. Bring your own sources, metadata providers, and download clients to build a single hub for your digital library. Supports multiple users with a built-in request system, so you can share your instance with others and let them browse and request books on their own.

Works great alongside the following library tools, with support for automatic imports:

Features

  • One-Stop Interface - A clean, modern UI to search, browse, and download from multiple configured sources in one place
  • Multiple Sources - Configurable web, torrent, usenet, and IRC source support
  • Audiobook Support - Full audiobook search and download with dedicated processing
  • Flexible Search - Search metadata providers (Hardcover, Open Library, Google Books) for rich book and audiobook discovery, or query configured sources directly
  • Multi-User & Requests - Share your instance with others, let users browse and request books, and manage approvals with configurable notifications
  • Authentication - Built-in login, OIDC single sign-on, proxy auth, and Calibre-Web database support
  • Real-Time Progress - Unified download queue with live status updates across all sources
  • Network Flexibility - Configurable proxy support, DNS settings, and optional Cloudflare handling for protected sources

🖼️ Screenshots

Home screen Home screen

Search results Search results

Multi-source downloads Multi-source downloads

Download queue Download queue

🚀 Quick Start

Prerequisites

  • Docker & Docker Compose
  • At least 2 GB of RAM available to the container when using the standard image — see Memory Requirements

Installation

  1. Download the docker-compose file:

    curl -O https://raw.githubusercontent.com/calibrain/shelfmark/main/compose/docker-compose.yml
    
  2. Start the service:

    docker compose up -d
    
  3. Open http://localhost:8084

Open the web interface, then configure the sources and settings you want to use.

Volume Setup

volumes:
  - /your/config/path:/config # Config, database, and artwork cache directory
  - /your/download/path:/books # Downloaded books
  - /client/path:/client/path # Optional: For Torrent/Usenet downloads, match your client directory exactly.

Tip

: Point the download volume to your CWA or Grimmory ingest folder for automatic import.

Note

: CIFS shares require nobrl mount option to avoid database lock errors.

Non-root container mode

  • Start the container as 1000:1000 with Docker user: "1000:1000" or docker run --user 1000:1000.
  • For Kubernetes, set runAsUser: 1000, runAsGroup: 1000, and runAsNonRoot: true together.
  • PUID/PGID keep the default root startup flow.
  • Mounted paths must already be writable by 1000:1000.
  • USING_TOR=true requires root startup.

⚙️ Configuration

Search Modes

Direct

  • Queries configured sources directly

Universal (recommended)

  • Search via metadata providers (Hardcover, Open Library, Google Books) for richer results
  • Aggregates releases from multiple configured sources
  • Full audiobook support

Environment Variables

Environment variables work for initial setup and Docker deployments. They serve as defaults that can be overridden in the web interface.

Variable Description Default
FLASK_PORT Web interface port 8084
INGEST_DIR Book download directory /books
TZ Container timezone UTC
PUID / PGID Runtime user/group for the default root-startup flow (also supports legacy UID/GID) 1000 / 1000
SEARCH_MODE direct or universal universal
USING_TOR Enable Tor routing (requires root startup) false
USING_WIREGUARD Enable WireGuard VPN egress with kill-switch (requires root startup) false
WIREGUARD_CONFIG Path to the mounted wg-quick config /config/wg0.conf
WIREGUARD_INTERFACE WireGuard interface name wg0
LAN_NETWORK Comma-separated CIDRs kept off the tunnel so the WebUI / internal clients stay reachable 127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
WIREGUARD_ENFORCE_DNS Pin the resolver (via WIREGUARD_DNS, else the config's DNS =) so DNS can't silently fall back to an off-tunnel path. Designed for a trusted LAN resolver kept reachable via LAN_NETWORK (query leaves over the LAN; download still egresses via the tunnel) — it does not force queries through the tunnel. Docker's embedded resolver (127.0.0.11) is preserved when present so container-name resolution keeps working; pin its upstream via the container's dns: list. Fails closed if no resolver is available or /etc/resolv.conf is not writable. true
WIREGUARD_DNS Explicit resolver(s) to pin (comma/space separated). Use when the VPN's pushed DNS filters domains you need; point at a resolver reachable via the tunnel or an allowed LAN resolver. (unset; uses config DNS =)
WIREGUARD_DISABLE_IPV6 Strip IPv6 from the tunnel config (many container kernels lack the ip6tables raw table wg-quick needs) and remove IPv6 as a leak surface. true
WIREGUARD_ALLOW_IPV6_LEAK Escape hatch: continue even when an IPv6 kill-switch can't be installed AND IPv6 can't be disabled. Only set if the container has no IPv6 connectivity. false
WIREGUARD_ALLOW_WEBUI_OFFTUNNEL Opt-in off-tunnel WebUI reachability. Default (false) keeps the kill-switch strictly fail-closed: the only off-tunnel egress is loopback, the tunnel device and the LAN allowlist. Set true only if a non-LAN client (e.g. a public reverse proxy on another segment) must reach the WebUI; it permits app-server replies (--sport FLASK_PORT, conntrack REPLY) off-tunnel — server replies only, never client-initiated egress. LAN clients never need it (covered by LAN_NETWORK). false
WIREGUARD_STALE_AFTER Seconds since the last handshake before the healthcheck bounces the tunnel. 180

See the full Environment Variables Reference for all available options.

Some of the additional options available in Settings:

  • Prowlarr - Configure indexers and download clients to download books and audiobooks
  • Additional audiobook sources - Configure additional sources for audiobook discovery
  • IRC - Add details for IRC book sources and download directly from the UI
  • Library Link - Add a link to your Calibre-Web or Grimmory instance in the UI header
  • File processing - Customiseable download paths, file renaming and directory creation with template-based renaming
  • Network Settings - Custom proxy support (SOCKS5 + HTTP/S) and configurable DNS
  • Format & Language - Filter downloads by preferred formats, languages and sorting order
  • Metadata Providers - Configure API keys for Hardcover, Open Library, etc.

🐳 Docker Variants

Standard

docker compose up -d

The full-featured image with all network capabilities included.

Memory Requirements

The standard image ships a real Chromium browser, which it launches to solve Cloudflare challenges for Direct Download. Chromium needs room to run:

  • 2 GB of RAM available to the container is a safe minimum; 1 GB or less is where problems usually start
  • Only relevant if you use Direct Download. Prowlarr, IRC and audiobook sources don't start the browser

When the container is starved of memory, Chromium fails to start and every Direct Download fails with unrelated-looking errors — repeated 403 detected; switching to bypasser followed by No download URL found, and downloads that never complete. If you're seeing that, check the container's memory limit and the host's free memory before suspecting your ISP or DNS.

If you can't spare the memory, use the Lite image with an external resolver (e.g. FlareSolverr) running elsewhere.

Tor Routing

Optional Tor support for network privacy:

curl -O https://raw.githubusercontent.com/calibrain/shelfmark/main/compose/docker-compose.tor.yml
docker compose -f docker-compose.tor.yml up -d

Notes:

  • Requires root startup
  • Requires NET_ADMIN and NET_RAW capabilities
  • Timezone is auto-detected from Tor exit node
  • Custom DNS/proxy settings are ignored when Tor is active

WireGuard VPN Routing

Optional WireGuard support to route all external egress through a VPN tunnel with a fail-closed kill-switch:

curl -O https://raw.githubusercontent.com/calibrain/shelfmark/main/compose/docker-compose.wireguard.yml
# place your wg-quick config where the compose mounts /config, as wg0.conf
docker compose -f docker-compose.wireguard.yml up -d

Notes:

  • Requires root startup
  • Requires NET_ADMIN and NET_RAW capabilities
  • Mount a standard wg-quick config at WIREGUARD_CONFIG (default /config/wg0.conf)
  • All non-LAN egress is forced through the tunnel; if the tunnel drops, external traffic fails closed while LAN ranges (WebUI, Prowlarr, qBittorrent) stay reachable
  • IPv4 and IPv6 both fail closed. On kernels without a usable ip6tables, disable IPv6 for the container (sysctls: net.ipv6.conf.all.disable_ipv6=1, as in the compose example) or the container refuses to start rather than risk an IPv6 leak
  • A supervised healthcheck bounces the tunnel if the handshake goes stale, and refreshes the endpoint allow rules so a roaming/rotated peer endpoint can reconnect
  • Mutually exclusive with USING_TOR
  • DNS trust: WIREGUARD_DNS must be a resolver you trust on a trusted network segment. When it is a LAN resolver (kept reachable off-tunnel by LAN_NETWORK), the query to that resolver leaves as plaintext UDP/53 on the LAN — the resolver is responsible for encrypting upstream. Two resolver paths exist: (1) when Docker's embedded resolver (127.0.0.11) is present it is preserved so container names (Prowlarr, qBittorrent) resolve — you MUST pin its upstream to a trusted resolver via the container's compose dns: list, since WIREGUARD_DNS cannot repoint the embedded resolver from inside the container; (2) otherwise WIREGUARD_DNS/the config DNS = line is written to /etc/resolv.conf. Setting WIREGUARD_ENFORCE_DNS=false is a foot-gun: with no embedded resolver present the container then uses its inherited resolver, which forwards to the Docker daemon's upstream off-tunnel, leaking your DNS. Leave enforcement on unless you have pinned the resolver another way.

Lite

A lighter image without the built-in browser automation. Ideal for:

  • External services - Already running FlareSolverr or similar for other applications
  • Alternative sources - Using Prowlarr, IRC, or other configured sources
  • Audiobooks - Using Shelfmark primarily for audiobooks
  • Constrained hosts - No bundled browser, so it runs comfortably below the standard image's memory requirements
curl -O https://raw.githubusercontent.com/calibrain/shelfmark/main/compose/docker-compose.lite.yml
docker compose -f docker-compose.lite.yml up -d

If you need browser-based access with the Lite image, configure an external resolver in Settings.

🔐 Authentication

Authentication is optional but recommended for shared or exposed instances. Multiple authentication methods are available in Settings:

1. Single Username/Password

2. Proxy (Forward) Authentication

Proxy auth trusts headers set by your reverse proxy (e.g. X-Auth-User). Ensure Shelfmark is not directly exposed, and configure your proxy to strip/overwrite these headers for all inbound requests.

3. OIDC (OpenID Connect)

Integrate with your identity provider (Authelia, Authentik, Keycloak, etc.) for single sign-on. Supports PKCE flow, auto-discovery, group-based admin mapping, and auto-provisioning of new users.

4. Calibre-Web Database

If you're running Calibre-Web, you can reuse its user database by mounting it:

volumes:
  - /path/to/calibre-web/app.db:/auth/app.db:ro

Multi-User Support

With any authentication method enabled, Shelfmark supports multi-user management with admin/user roles. Users can have per-user settings for download destinations, email recipients, and notification preferences. Non-admin users only see their own downloads and can submit book requests for admin review. Admins can configure request policies per source to control whether users can download directly, must submit a request, or are blocked entirely.

Project Scope

Shelfmark is a manual search and download tool, the entry point to your book library, not a library manager. It finds books, downloads them, and sends them to a configured destination. That's the full scope.

Shelfmark intentionally does not:

  • Track or manage your library - it doesn't know or care what you already own
  • Integrate with library software - what happens after delivery is up to your library tool
  • Monitor authors, series, or new releases - there is no background automation
  • Queue future downloads - if a book isn't available now, Shelfmark won't watch for it

These are non-goals, not missing features.

Contributing

Shelfmark's core feature set is complete. Development focuses on stability, bug fixes, quality-of-life improvements, and refining the search experience. Contributions in these areas are welcome, please file issues or submit pull requests on GitHub.

Feature requests that fall outside the project scope (library integration, automation, collection management) will be closed. If you're unsure whether something fits, open a discussion first.

Health Monitoring

The application exposes a health endpoint at /api/health (no authentication required). Add a health check to your compose:

healthcheck:
  test: ["CMD", "curl", "-sf", "http://localhost:8084/api/health"]
  interval: 30s
  timeout: 30s
  retries: 3

Logging

Logs are available via:

  • docker logs <container-name>
  • /var/log/shelfmark/ inside the container (when ENABLE_LOGGING=true)

Log level is configurable via Settings or LOG_LEVEL environment variable.

Development

# Quality checks
make checks              # Run ALL static analysis (frontend + Python)
make python-checks       # Run Ruff, BasedPyright, and Vulture
make install-python-dev  # Sync Python runtime + dev tools with uv

# Frontend development
make install     # Install dependencies
make dev         # Start Vite dev server (localhost:5173)
make build       # Production build
make frontend-typecheck  # TypeScript checks

# Backend (Docker)
make up          # Start backend via docker-compose.dev.yml
make down        # Stop services
make refresh     # Rebuild and restart
make restart     # Restart container

The frontend dev server proxies to the backend on port 8084.

License

MIT License - see LICENSE for details.

⚠️ Disclaimer

Shelfmark is a search interface that displays results from external metadata providers and sources. It does not host, store, or distribute any content. The developers are not responsible for how the tool is used or what is accessed through it.

Users are solely responsible for:

  • Ensuring they have the legal right to download any material they access
  • Complying with copyright laws and intellectual property rights in their jurisdiction
  • Understanding and accepting the terms of any sources they configure

Use of this tool is entirely at your own risk.

Support

For issues or questions, please file an issue on GitHub.

S
Description
No description provided
Readme MIT
76 MiB
v1.3.15
Latest
2026-09-02 21:12:52 +01:00
Languages
Python 76%
TypeScript 21.5%
Shell 1.8%
CSS 0.3%
Dockerfile 0.2%
Other 0.2%