ffmpeg is in the default image for one thing, the Debug mode screen
recording (`_start_ffmpeg_recording`). Nothing else calls it,
SeleniumBase included, and it brings 93 packages with it. On real builds
that's 406 Trivy findings (149 High) and 78 MB of the gzipped amd64
image. This takes it out of the image and installs it when Debug mode
needs it.
It came out of the base image discussion on #1616 and helps on any base,
though on Wolfi the install line would need `apk`.
## What changed
- **Dockerfile:** ffmpeg is gone from the full image's install.
- **Guard:** `_start_ffmpeg_recording` checks `shutil.which("ffmpeg")`
first. If it's missing it logs a warning and returns, and the bypass
carries on. Without this, Debug on with no ffmpeg fails every bypass
with `FileNotFoundError`. There's a new test for it.
- **Entrypoint:** when Debug is on, the internal bypasser is in use and
ffmpeg isn't already there, it installs ffmpeg in the background so the
app doesn't wait for it. The download and the install each run under a
120 s `timeout` with `DEBIAN_FRONTEND=noninteractive`. Downloading first
means a hung mirror can only be killed before dpkg starts, and the next
start runs `dpkg --configure -a` in case a restart cut dpkg off. It logs
when it starts, succeeds, times out or fails.
- **Docs:** a short paragraph in `docs/troubleshooting.md` covering the
install and the non-root limit.
## Debug has two sources
Debug can come from the `DEBUG` env var or from the Debug Mode setting
in the UI, which is saved to `/config/plugins/advanced.json`. The
existing block in `entrypoint.sh` only reads the env var, so an install
keyed on `$DEBUG` would miss anyone who turned it on in the UI.
The new check asks the app instead (`shelfmark.config.env.DEBUG`), which
reads both. The UI setting is `requires_restart`, so the restart it
already asks for is what brings it through the entrypoint.
## Non-root mode
A container started as `1000:1000` can't run `apt-get`, so the
entrypoint logs that it can't install ffmpeg, the guard warns on each
bypass, and there's no recording. Everything else works. The workaround
for a debugging session is to start as root with `PUID`/`PGID`.
## Startup cost
None that I could measure. With Debug on, `/api/health` answers in the
same time as main (3-4 s with Debug only in config, 9-10 s from env),
and ffmpeg was ready 6-8 s later on the runners.
The trade-off is that the first bypass after startup may be unrecorded
while the install runs. It logs the warning and passes. I saw both
outcomes on a bypass started right after health.
From reading the code: the install lives in the container, so recreating
the container installs it again. It also runs after `tor.sh` and
`wireguard.sh`, so apt goes through whichever egress is active.
## Testing
Real Docker builds on GitHub Actions on my fork, amd64 and arm64, with
the same result on both for every case below.
| Case | Result |
|---|---|
| Debug off | No install, `/api/health` in 3-4 s, same as main |
| Debug only in config | Health in 3-4 s, ffmpeg ready 6-8 s later, real
mp4 after a bypass |
| Debug from env | Health in 9-10 s (same as main), real mp4 after a
bypass |
| Started as root, app dropped to uid 1000 | Smoke and bypass pass |
| Started with `--user 1000`, Debug on | Logs both messages, bypass
passes, no mp4 |
| apt blocked (closed proxy) | Logs `Could not install ffmpeg`, bypass
passes, no mp4 |
| apt's proxy on a dropped address, so no TCP reply at all | Health in
9-10 s, a bypass during the install passes with the warning and no mp4.
apt gives up by itself in under a minute. Nothing left running, `dpkg
--audit` clean |
| DNS server on a dropped address | Health in 9-10 s, `Timed out
installing ffmpeg after 120s`. Nothing left running, `dpkg --audit`
clean |
| `--network none`, Debug on | No hang, health 200 from inside the
container |
| Container restarted while dpkg was running | Next start finishes the
install, `dpkg --audit` clean, real mp4 |
| Fresh named volume, Debug only in config | Recording made, uid 1000
can write |
| Existing uid-1000 volume, Debug only in config | Same, nothing
root-owned afterwards |
| `find /config /var/log/shelfmark -user root` | 0 files, same as main |
| `find /app -user root` | 245, same count as main |
| `tests/e2e/platform/run-e2e.sh` with `dns-blocked` and `baseline` |
Pass |
The bypass checks ran against example.com. A real Cloudflare challenge
was served by a demo page in the same runs, and no build solved it,
including main, so that part doesn't say anything about this change.
Locally: 5494 Python tests pass, and `ruff`, `basedpyright`, `vulture`
and `shellcheck` are clean.
The full results are in [the base image
report](https://github.com/splitsec2/shelfmark/blob/7afcecfb76727724a71561cbc078f091569b58dc/docs/research/2026-11-01-base-image-report.md).
LOG_LEVEL never reached the app logger: env.py hardcoded the level to
DEBUG or INFO, so INFO lines kept appearing under LOG_LEVEL=error. Read
it from the env var and advanced settings, normalize unknown values to
INFO, and expose it as a setting. entrypoint.sh now normalizes
gunicorn's level too, so a typo falls back to info instead of stopping
the container from booting.
Z-Library gates the first hit on /md5/<hash> with a 503 whose only
payload is a Set-Cookie; echoing that cookie back returns the 302 to the
real page. html_get_page dropped it and re-ran the same rejected request
on every retry, ending in "No download URL resolved". Retry once with
the
cookies the 503 issued.
Fixes#1185Fixes#1187
- Fixed internal bypasser startup with newer Chromium/SeleniumBase by
isolating the browser helper from Gunicorn/gevent, serialising helper
failures cleanly, and cleaning up orphan processes after a failure
- Stopped using /app as runtime home state, now moved to /home/shelfmark
or /tmp/shelfmark/home as fallback.
- Added tests
- Adds a non-root startup path at user 1000:1000 - skips privilege
escalation and ownership checks. Works e.g. for kubernetes deployments
(user 1000:1000 and runAsNonRoot enabled).
- Remove startup check/chown commands for user-owned folders. Checks can
be done with a "Test destination" button in settings which performs a
test write. Users are responsible for fixing their own permissions.
- Update docs
- Adds `uv`, `ruff`, `pyright`, `vulture` and `pytest-xdist`
- Move project, lockfile, docker build etc to uv
- Align python tooling on 3.14
- Huge bulk of ruff linter fixes applied. Still in progress but all the
core types are now enforced
- Update CI and test helpers
- Clean up a few uses of config options that may miss the env variable
if this is set
- Add enhanced retry availability utilising the DB to persist download
errors / retries across restarts, request failures, and pass Prowlarr
detail through the download task to maintain retry data.
- Strip back entrypoint permissions for less intensive chown operations.
Fixes#796
- Added a path for rootless permissions in the entrypoint script
- Routed prowlarr searches through torznab for seedtime info
- Added additional request flow for download permissions
- Updated frontend CSS to Tailwind v4
- Reverted socket IO origin restriction
- Fixed search queries not persisting after auth redirect
- Move advanced search options to left UI selector
- Unlock IRC source to be used for audiobook content_type
- Tweaked security settings env var syncing to be prioritised
- Fix AA "all languages" query generation
- Added language-free AA query as second fallback in case of no results
- Testing moving SeleniumBase scratch files to /tmp via symlink
- Added enhanced logging for activity dismissals and other events
- Removed iFrame restrictions
- Refactored user and request code to avoid any database conflicts
- Fix threading behavior with custom script execution
- Harden the no_auth activity user filtering
- Add a hint to add local admin if none is created
- Added secret key to persist login states across updates / restarts
- Added notification support via Apprise dependency
- Notifications can be configured globally or per user, with full
customization of events and notification type.
- Added expanded ActivityCard for increased detail of each request, file
info, and managing the attached file.
- Enhanced tests
Hey, made the tweaks we discussed, plus a couple related fixes :)
- Removed APP_ENV entirely. All dev-specific functionality is enabled
via `DEBUG: true` env var
- Set secure cookie handling to false by default, added to the readme to
enable if exclusively using HTTPS connection
- Fixed healthcheck potentially not working with auth enabled
- Removed APP_ENV from docker compose files and made sure app.db lines
are included in all versions.
APP_ENV in people's existing composes should get ignored entirely and
will be put on the default env, so no issues when updating.
This PR was coauthored by alexhb1 and davidemarcoli. It builds on the FE
rework created by alex, but adds a myriad of additional tweaks and
optimizations to make the frontend feel modern, fast, and responsive.
The summary of the changes is as follows:
### Architecture Changes
React/TypeScript Migration: Refactored frontend from template/JS
structure to React/TypeScript application for better maintainability and
scalability
WebSocket Integration: Implemented real-time updates for download status
and progress with automatic fallback to polling
Gevent Worker: Configured production WebSocket support
### UI/UX Improvements
<img width="1502" height="890" alt="Screenshot 2025-11-10 at 10 02
59 AM"
src="https://github.com/user-attachments/assets/86bf8649-623f-413c-b8e5-656e687e55a8"
/>
Downloads Sidebar: Replaced bottom downloads section with sidebar
interface for better organization
<img width="201" height="450" alt="Screenshot 2025-11-10 at 10 07 52 AM"
src="https://github.com/user-attachments/assets/92b98e7c-c3bc-4b7e-80f1-252c3a760e33"
/>
Status Badges: Color-coded download status indicators instead of plain
text
Pinned Header: Fixed header position for consistent navigation
Enhanced Book Cards: Improved layout and hover states with info modal
button
<img width="1474" height="899" alt="Screenshot 2025-11-10 at 10 08
18 AM"
src="https://github.com/user-attachments/assets/9216d8a3-f662-434d-80e6-2a69b96abc31"
/>
Download Progress: Circular progress indicator on download buttons
Toast Notifications: Added user feedback for actions
Spinner Feedback: Loading indicators on search and download buttons
Animations: Smooth transitions and fluid progress updates
### Mobile & Responsive Design
Mobile-friendly Layouts: Optimized book cards and search interface for
mobile
<img width="225" height="450" alt="Screenshot 2025-11-10 at 10 05 49 AM"
src="https://github.com/user-attachments/assets/c8236c1c-5837-4309-9577-46db7292a54b"
/>
Keyboard Handling: Improved mobile keyboard behavior with proper input
types
PWA Improvements: Enhanced progressive web app functionality
Responsive Search: Better search box width and positioning across
devices
### Developer Experience
Development Mode: Separate frontend dev server that works with existing
backend container
Makefile: Added build automation and development commands
Documentation: Updated README with frontend architecture details
### Bug Fixes
Fixed "Clear completed" functionality
Fixed dark mode toggle text
Fixed sticky header behavior
Fixed mobile search box positioning
Removed active downloads requirement for initial state view
### Additional Features
ESC Key: Close downloads sidebar with ESC key
Calibre-Web Button: Direct link to Calibre-Web instance
<img width="282" height="83" alt="Screenshot 2025-11-11 at 9 38 05 AM"
src="https://github.com/user-attachments/assets/273075be-9743-4e13-9e48-5bf498f6c067"
/>
Granular Status Tracking: More detailed download progress information
obtained via websockets
---------
Co-authored-by: Alex <alex.bilbie1@gmail.com>
Co-authored-by: Zack Yancey <yanceyz@proton.me>
Co-authored-by: davidemarcoli <davide@marcoli.ch>
Adding support for an external CloudFlare bypasser service and
introducing a new Docker image build with a dedicated target.
Key Changes
- Added `cloudflare_bypasser_external.py` for external bypasser
integration.
- Updated Docker Compose files to support the new service.
- Introduced a new Docker target for building a separate image for the
external bypasser.
- Refactored relevant modules to utilize the external bypasser when
configured.
- Documentation and configuration updates to reflect new options and
Docker targets.
Impact
- Users can now choose between internal and external CloudFlare
bypassing.
- New Docker image and target streamline deployment of the external
bypasser.
- Improved modularity and maintainability.
- No breaking changes for existing workflows.
Testing
- Manual and E2E tests performed for both bypasser modes.
- Docker Compose setups and new image build verified for development and
production.
Notes
Please review the new configuration options and Docker targets. Update
your environment and deployment scripts as needed. Feedback and
suggestions are welcome!
I tried running the docker-compose.yml referenced in this comment
https://github.com/calibrain/calibre-web-automated-book-downloader/issues/120#issuecomment-2816539268
from issue #120 and was able to reproduce the same 'Server is not
responding' error. I found the issue would happen whenever the captcha
bypass took more than 30 seconds due to tor's slower connection. Adding
-t 300 would let gunicorn wait up to 5 minutes now.
Fix permission issues that occur if user forgets to create the
`cwa-book-downloader` folder before creating the docker container since
docker makes the directory with root ownership.
`sudo -E` was replacing `appuser`'s `$HOME` environment variable of
`/app` with `/root` causing permission issues when selenium tried to
write to `~/.cache/selenium`. This fixes [issue
#130](https://github.com/calibrain/calibre-web-automated-book-downloader/issues/130)
by allowing Driver from seleniumbase to successfully return instead of
timing out on line 172 in `cloudflare_bypasser.py`.
Refactor: Improve Docker build, add Tor support, use SeleniumBase
- Overhauled Dockerfile:
- Switched to python:3.10-slim base.
- Implemented multi-stage builds (base, standard, tor).
- Consolidated RUN layers for efficiency.
- Added locale/timezone setup.
- Added Tor setup and iptables configuration in dedicated stage/script.
- Replaced DrissionPage with SeleniumBase for Cloudflare bypassing.
- Added Tor support via `docker-compose.tor.yml` and `tor.sh` script.
- Updated `docker-compose.yml` to build locally and changed default port
to 8083.
- Added `docker-compose.dev.yml` and `docker-compose.tor.dev.yml`.
- Updated `entrypoint.sh` for timezone and sudo usage.
- Added/Updated environment variables (`TZ`, `USING_TOR`, etc.).
- Improved `.dockerignore`.
- Updated `readme.md` to reflect port changes, build process, document
`TZ`, and add details about the new Tor variant.
The Calibre dependency was due to the script testing for validity of the
downloaded file, as often they would be corrupted from aa. But CWA is
already doing that, so we are just having redundant code here.
For the cloudflarebypasser, I basically run my own version now, instead
of depending on an external library, this way we have better control for
debugging and on the docker image.
Fixes#18, #33, #27, #48, #65, #78, #86, #88, #89
---------
Co-authored-by: mik593 <91991279+mik593@users.noreply.github.com>
splitsec2
CaliBrain
Alex SchittkoandCaliBrain
Alex
Alex
Federico Della Rovere
CCarpo
Jacob Wolfe
DanielandDaniel Egberts
