Adding support for an external CloudFlare bypasser service and
introducing a new Docker image build with a dedicated target.
Key Changes
- Added `cloudflare_bypasser_external.py` for external bypasser
integration.
- Updated Docker Compose files to support the new service.
- Introduced a new Docker target for building a separate image for the
external bypasser.
- Refactored relevant modules to utilize the external bypasser when
configured.
- Documentation and configuration updates to reflect new options and
Docker targets.
Impact
- Users can now choose between internal and external CloudFlare
bypassing.
- New Docker image and target streamline deployment of the external
bypasser.
- Improved modularity and maintainability.
- No breaking changes for existing workflows.
Testing
- Manual and E2E tests performed for both bypasser modes.
- Docker Compose setups and new image build verified for development and
production.
Notes
Please review the new configuration options and Docker targets. Update
your environment and deployment scripts as needed. Feedback and
suggestions are welcome!
I tried running the docker-compose.yml referenced in this comment
https://github.com/calibrain/calibre-web-automated-book-downloader/issues/120#issuecomment-2816539268
from issue #120 and was able to reproduce the same 'Server is not
responding' error. I found the issue would happen whenever the captcha
bypass took more than 30 seconds due to tor's slower connection. Adding
-t 300 would let gunicorn wait up to 5 minutes now.
Fix permission issues that occur if user forgets to create the
`cwa-book-downloader` folder before creating the docker container since
docker makes the directory with root ownership.
`sudo -E` was replacing `appuser`'s `$HOME` environment variable of
`/app` with `/root` causing permission issues when selenium tried to
write to `~/.cache/selenium`. This fixes [issue
#130](https://github.com/calibrain/calibre-web-automated-book-downloader/issues/130)
by allowing Driver from seleniumbase to successfully return instead of
timing out on line 172 in `cloudflare_bypasser.py`.
Refactor: Improve Docker build, add Tor support, use SeleniumBase
- Overhauled Dockerfile:
- Switched to python:3.10-slim base.
- Implemented multi-stage builds (base, standard, tor).
- Consolidated RUN layers for efficiency.
- Added locale/timezone setup.
- Added Tor setup and iptables configuration in dedicated stage/script.
- Replaced DrissionPage with SeleniumBase for Cloudflare bypassing.
- Added Tor support via `docker-compose.tor.yml` and `tor.sh` script.
- Updated `docker-compose.yml` to build locally and changed default port
to 8083.
- Added `docker-compose.dev.yml` and `docker-compose.tor.dev.yml`.
- Updated `entrypoint.sh` for timezone and sudo usage.
- Added/Updated environment variables (`TZ`, `USING_TOR`, etc.).
- Improved `.dockerignore`.
- Updated `readme.md` to reflect port changes, build process, document
`TZ`, and add details about the new Tor variant.
The Calibre dependency was due to the script testing for validity of the
downloaded file, as often they would be corrupted from aa. But CWA is
already doing that, so we are just having redundant code here.
For the cloudflarebypasser, I basically run my own version now, instead
of depending on an external library, this way we have better control for
debugging and on the docker image.
Fixes#18, #33, #27, #48, #65, #78, #86, #88, #89
---------
Co-authored-by: mik593 <91991279+mik593@users.noreply.github.com>
This PR add the option to set UID/GID during runtime, making sure the
application is running as a non-root user.
Besides this, it also helps with making sure UID/GID match between
containers.
CaliBrain
Federico Della Rovere
CCarpo
Jacob Wolfe
DanielandDaniel Egberts
Vincent De Borger