# πŸ“š Calibre-Web-Automated-Book-Downloader Calibre-Web Automated Book Downloader An intuitive web interface for searching and requesting book downloads, designed to work seamlessly with [Calibre-Web-Automated](https://github.com/crocodilestick/Calibre-Web-Automated). This project streamlines the process of downloading books and preparing them for integration into your Calibre library. ## ✨ Features - 🌐 User-friendly web interface for book search and download - πŸ”„ Automated download to your specified ingest folder - πŸ”Œ Seamless integration with Calibre-Web-Automated - πŸ“– Support for multiple book formats (epub, mobi, azw3, fb2, djvu, cbz, cbr) - πŸ›‘οΈ Cloudflare bypass capability for reliable downloads - 🐳 Docker-based deployment for quick setup ## πŸ–ΌοΈ Screenshots ![Homescreen](README_images/homescreen.png 'Homescreen') ![Search results](README_images/search-results.png 'Search results') ![Active downloads](README_images/downloads.png 'Active downloads') ## πŸš€ Quick Start ### Prerequisites - Docker - Docker Compose - A running instance of [Calibre-Web-Automated](https://github.com/crocodilestick/Calibre-Web-Automated) (recommended) ### Installation Steps 1. Get the docker-compose.yml: ```bash curl -O https://raw.githubusercontent.com/calibrain/calibre-web-automated-book-downloader/refs/heads/main/docker-compose.yml ``` 2. Start the service: ```bash docker compose up -d ``` 3. Access the web interface at `http://localhost:8084` ## βš™οΈ Configuration ### Environment Variables #### Application Settings | Variable | Description | Default Value | | ----------------- | ----------------------- | ------------------ | | `FLASK_PORT` | Web interface port | `8084` | | `FLASK_HOST` | Web interface binding | `0.0.0.0` | | `DEBUG` | Debug mode toggle | `false` | | `INGEST_DIR` | Book download directory | `/cwa-book-ingest` | | `TZ` | Container timezone | `UTC` | | `UID` | Runtime user ID | `1000` | | `GID` | Runtime group ID | `100` | | `CWA_DB_PATH` | Calibre-Web's database | None | | `ENABLE_LOGGING` | Enable log file | `true` | | `LOG_LEVEL` | Log level to use | `info` | | `SESSION_COOKIE_SECURE` | Secure cookie enforcement - Use for HTTPS connections only | `false` | | `CALIBRE_WEB_URL` | Custom WebUI library link | None | | `BYPASS_WARMUP_ON_CONNECT` | Warm up Cloudflare bypasser when first client connects | `true` | If you wish to enable authentication, you must set `CWA_DB_PATH` to point to Calibre-Web's `app.db`, in order to match the username and password. Set `CALIBRE_WEB_URL` to your Calibre-Web / Booklore base URL. A β€˜Go to library’ button will appear in the Web UI for quick access while downloading, and it also provides library access when CWA-BD is installed as a mobile PWA. If logging is enabled, log folder default location is `/var/log/cwa-book-downloader` Available log levels: `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`. Higher levels show fewer messages. Note that if using TOR, the TZ will be calculated automatically based on IP. #### Download Settings | Variable | Description | Default Value | | ---------------------- | --------------------------------------------------------- | --------------------------------- | | `MAX_RETRY` | Maximum retry attempts | `3` | | `DEFAULT_SLEEP` | Retry delay (seconds) | `5` | | `MAIN_LOOP_SLEEP_TIME` | Processing loop delay (seconds) | `5` | | `SUPPORTED_FORMATS` | Supported book formats | `epub,mobi,azw3,fb2,djvu,cbz,cbr` | | `BOOK_LANGUAGE` | Preferred language for books | `en` | | `AA_DONATOR_KEY` | Optional Donator key for Anna's Archive fast download API | `` | | `USE_BOOK_TITLE` | Use book title as filename instead of ID | `false` | | `PRIORITIZE_WELIB` | When downloading, download from WELIB first instead of AA | `false` | | `ALLOW_USE_WELIB` | Allow usage of welib for downloading books if found there | `true` | If you change `BOOK_LANGUAGE`, you can add multiple comma separated languages, such as `en,fr,ru` etc. Use the following environment variables to set specific folders in which to download different content types (Book, Magazine, Comic, etc.): | Variable | Description | Default Value | |---------------------------------|--------------------------------|---------------| | `INGEST_DIR_BOOK_FICTION` | Book (fiction) folder name | `` | | `INGEST_DIR_BOOK_NON_FICTION` | Book (non-fiction) folder name | `` | | `INGEST_DIR_BOOK_UNKNOWN` | Book (unknown) folder name | `` | | `INGEST_DIR_MAGAZINE` | Magazine folder name | `` | | `INGEST_DIR_COMIC_BOOK` | Comic book folder name | `` | | `INGEST_DIR_AUDIOBOOK` | Audiobook folder name | `` | | `INGEST_DIR_STANDARDS_DOCUMENT` | Standards document folder name | `` | | `INGEST_DIR_MUSICAL_SCORE` | Musical score folder name | `` | If no specific path is set for a content type the default is `INGEST_DIR`. Remember to map the specified paths to where your instance of Calibre-Web-Automated (CWA) will find them, e.g.: ``` volumes: - /tmp/data/calibre-web/comicbook-ingest:/cwa-comicbook-ingest ``` if `INGEST_DIR_COMIC_BOOK=/cwa-comicbook-ingest` and your CWA is configured to use `/tmp/data/calibre-web/comicbook-ingest` for comic books. #### AA | Variable | Description | Default Value | | ---------------------- | --------------------------------------------------------- | --------------------------------- | | `AA_BASE_URL` | Base URL of Annas-Archive (could be changed for a proxy) | `https://annas-archive.org` | | `USE_CF_BYPASS` | Disable CF bypass and use alternative links instead | `true` | If you are a donator on AA, you can use your Key in `AA_DONATOR_KEY` to speed up downloads and bypass the wait times. If disabling the cloudflare bypass, you will be using alternative download hosts, such as libgen or z-lib, but they usually have a delay before getting the more recent books and their collection is not as big as aa's. But this setting should work for the majority of books. #### Network Settings | Variable | Description | Default Value | | ---------------------- | ------------------------------- | ----------------------- | | `AA_ADDITIONAL_URLS` | Proxy URLs for AA (, separated) | `` | | `HTTP_PROXY` | HTTP proxy URL | `` | | `HTTPS_PROXY` | HTTPS proxy URL | `` | | `CUSTOM_DNS` | DNS configuration | `auto` | | `USE_DOH` | Use DNS over HTTPS | `false` | **Proxy Configuration** For proxy configuration, you can specify URLs in the following format: ```bash # Basic proxy HTTP_PROXY=http://proxy.example.com:8080 HTTPS_PROXY=http://proxy.example.com:8080 # Proxy with authentication HTTP_PROXY=http://username:password@proxy.example.com:8080 HTTPS_PROXY=http://username:password@proxy.example.com:8080 ``` **DNS Configuration** The `CUSTOM_DNS` setting controls how DNS resolution works. By default, it is set to `auto` which provides automatic failover for reliable connectivity. **Auto Mode (Default)** When `CUSTOM_DNS=auto`, the application starts with your system's default DNS. If DNS resolution fails, it automatically rotates through alternative providers using DNS over HTTPS (DoH): 1. System DNS (initial) 2. Cloudflare (1.1.1.1) 3. Google (8.8.8.8) 4. Quad9 (9.9.9.9) 5. OpenDNS (208.67.222.222) This automatic rotation helps bypass ISP-level blocks and DNS issues without any manual configuration. **Manual DNS Configuration** If you prefer to use a specific DNS configuration, you can override the auto behavior: 1. **Preset DNS Providers**: Use one of these predefined options: - `google` - Google DNS (8.8.8.8, 8.8.4.4) - `quad9` - Quad9 DNS (9.9.9.9, 149.112.112.112) - `cloudflare` - Cloudflare DNS (1.1.1.1, 1.0.0.1) - `opendns` - OpenDNS (208.67.222.222, 208.67.220.220) 2. **Custom DNS Servers**: A comma-separated list of DNS server IP addresses - Example: `127.0.0.53,127.0.1.53` (useful for PiHole) - Supports both IPv4 and IPv6 addresses When using preset providers, you can optionally enable DNS over HTTPS with `USE_DOH=true`: ```bash CUSTOM_DNS=cloudflare USE_DOH=true ``` Note: When using custom IP addresses, the `USE_DOH` flag is ignored since DoH requires a known provider endpoint. #### Custom configuration | Variable | Description | Default Value | | ---------------------- | ----------------------------------------------------------- | ----------------------- | | `CUSTOM_SCRIPT` | Path to an executable script that tuns after each download | `` | If `CUSTOM_SCRIPT` is set, it will be executed after each successful download but before the file is moved to the ingest directory. This allows for custom processing like format conversion or validation. The script is called with the full path of the downloaded file as its argument. Important notes: - The script must preserve the original filename for proper processing - The file can be modified or even deleted if needed - The file will be moved to `/cwa-book-ingest` after the script execution (if not deleted) You can specify these configuration in this format : ``` environment: - CUSTOM_SCRIPT=/scripts/process-book.sh volumes: - local/scripts/custom_script.sh:/scripts/process-book.sh ``` ### Volume Configuration ```yaml volumes: - /your/local/path:/cwa-book-ingest - /cwa/config/path/app.db:/auth/app.db:ro ``` **Note** - If your library volume is on a cifs share, you will get a "database locked" error until you add **nobrl** to your mount line in your fstab file. e.g. //192.168.1.1/Books /media/books cifs credentials=.smbcredentials,uid=1000,gid=1000,iocharset=utf8,**nobrl** - See https://github.com/crocodilestick/Calibre-Web-Automated/issues/64#issuecomment-2712769777 Mount should align with your Calibre-Web-Automated ingest folder. ## Variants: ### πŸ§… Tor Variant This application also offers a variant that routes all its traffic through the Tor network. This can be useful for enhanced privacy or bypassing network restrictions. To use the Tor variant: 1. Get the Tor-specific docker-compose file: ```bash curl -O https://raw.githubusercontent.com/calibrain/calibre-web-automated-book-downloader/refs/heads/main/docker-compose.tor.yml ``` 2. Start the service using this file: ```bash docker compose -f docker-compose.tor.yml up -d ``` **Important Considerations for Tor:** * **Capabilities:** This variant requires the `NET_ADMIN` and `NET_RAW` Docker capabilities to configure `iptables` for transparent Tor proxying. * **Timezone:** When running in Tor mode, the container will attempt to determine the timezone based on the Tor exit node's IP address and set it automatically. This will override the `TZ` environment variable if it is set. * **Network Settings:** Custom DNS, DoH, and HTTP(S) proxy settings (`CUSTOM_DNS`, `USE_DOH`, `HTTP_PROXY`, `HTTPS_PROXY`) are ignored when using the Tor variant, as all traffic goes through Tor. ### External Cloudflare resolver variant This variant allows the application to use an external service to bypass Cloudflare protection, instead of relying on the built-in bypasser. This is useful if you already have a dedicated Cloudflare resolver (such as [FlareSolverr](https://github.com/FlareSolverr/FlareSolverr) or compatible services like [ByParr](https://github.com/ThePhaseless/Byparr)) running elsewhere. #### How it works: - When enabled, all requests that require Cloudflare bypass are sent to your external resolver service. - The application communicates with the resolver using its API. #### Configuration | Variable | Description | Default Value | | ---------------------- | ----------------------------------------------------------- | ----------------------- | | `EXT_BYPASSER_URL` | The full URL of your external resolver (required) | | | `EXT_BYPASSER_PATH` | API path for the resolver (usually `/v1`) | `/v1` | | `EXT_BYPASSER_TIMEOUT` | Timeout for page loading (in milliseconds) | `60000` | #### Important This feature follows the same configuration of the built-in Cloudflare bypasser, so you should turn on the `USE_CF_BYPASS` configuration to enable it. #### To use the External Cloudflare resolver variant: 1. Get the extbp-specific docker-compose file: ```bash curl -O https://raw.githubusercontent.com/calibrain/calibre-web-automated-book-downloader/refs/heads/main/docker-compose.extbp.yml ``` 2. Start the service using this file: ```bash docker compose -f docker-compose.extbp.yml up -d ``` #### Compatibility: This feature is designed to work with any resolver that implements the `FlareSolverr` API schema, including `ByParr` and similar projects. #### Internal vs External Bypasser The **internal bypasser** (default) is custom-designed for this application's specific needs. It handles session management, cookie persistence, and retry logic optimized for book downloading workflows. For most users, this provides the most reliable experience out of the box. The **external bypasser** is better suited if you: - Already run FlareSolverr/ByParr for other services and want to consolidate - Need to share bypass infrastructure across multiple applications - Want to offload browser automation to a dedicated, more powerful container If you're unsure which to use, start with the default internal bypasser. ## πŸ—οΈ Architecture The application consists of a Flask backend with a React-based frontend: ### Backend - **Flask Application**: Python-based backend (`app.py`, `backend.py`) providing REST API and WebSocket support - **Download Manager**: Handles book search, download requests, and queue management (`downloader.py`, `book_manager.py`) - **Network Layer**: Cloudflare bypass and proxy support (`cloudflare_bypasser.py`, `network.py`) ### Frontend - **React + TypeScript**: Modern web interface built with Vite (`src/frontend`) - **Real-time Updates**: WebSocket integration for live download status - **Responsive UI**: TailwindCSS-based design for mobile and desktop For frontend development, use the provided Makefile: ```bash make install # Install dependencies make dev # Start development server make build # Build for production ``` If you run the docker compose file, the frontend will be built and served automatically. But if you run the frontend dev server it will supercede the docker compose frontend. ## πŸ₯ Health Monitoring Built-in health checks monitor: - Web interface availability - Download service status - Cloudflare bypass service connection Checks run every 30 seconds with a 30-second timeout and 3 retries. You can enable by adding this to your compose : ``` HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \ CMD curl -s http://localhost:8084/api/status || exit 1 ``` ## πŸ“ Logging Logs are available in: - Container: `/var/logs/cwa-book-downloader.log` - Docker logs: Access via `docker logs` ## 🀝 Contributing Contributions are welcome! Feel free to submit a Pull Request. ## πŸ“„ License This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. ## ⚠️ Important Disclaimers ### Copyright Notice While this tool can access various sources including those that might contain copyrighted material (e.g., Anna's Archive), it is designed for legitimate use only. Users are responsible for: - Ensuring they have the right to download requested materials - Respecting copyright laws and intellectual property rights - Using the tool in compliance with their local regulations ### Duplicate Downloads Warning Please note that the current version: - Does not check for existing files in the download directory - Does not verify if books already exist in your Calibre database - Exercise caution when requesting multiple books to avoid duplicates ## πŸ’¬ Support For issues or questions, please file an issue on the GitHub repository.