25 Commits
Author SHA1 Message Date
Alex 4472fbe8cf Download overhaul - DNS fallback, bypasser enhancements, revamped error handling, better frontend UX (#336)
## Changelog

### 🌐 Network Resilience

- **Auto DNS rotation**: New `CUSTOM_DNS=auto` mode (now default) starts
with system DNS and automatically rotates through Cloudflare, Google,
Quad9, and OpenDNS when failures are detected. DNS results are cached to
improve performance.
- **Mirror failover**: Anna's Archive requests automatically fail over
between mirrors (.org, .se, .li) when one is unreachable
- **Round-robin source distribution**: Concurrent downloads are
distributed across different AA partner servers to avoid rate limiting

### 📥 Download Reliability

- **Much more reliable downloads**: Improved parsing of Anna's Archive
pages, smarter source prioritization, and better retry logic with
exponential backoff
- **Download resume support**: Interrupted downloads can now resume from
where they left off (if the server supports Range requests)
- **Cookie sharing**: Cloudflare bypass cookies are extracted and shared
with subsequent requests, often avoiding the need for re-bypass entirely
- **Stall detection**: Downloads with no progress for 5 minutes are
automatically cancelled and retried
- **Staggered concurrent downloads**: Small delays between starting
concurrent downloads to avoid hitting rate limits
- **Source failure tracking**: After multiple failures from the same
source type (e.g., Libgen), that source is temporarily skipped
- **Lazy welib loading**: Welib sources are fetched as a fallback only
when primary sources fail (unless `PRIORITIZE_WELIB` is enabled)

### 🛡️ Cloudflare & Protection Bypass

- **DDOS-Guard support**: Internal bypasser now detects and handles
DDOS-Guard challenges with dedicated bypass strategies
- **Cancellation support**: Bypass operations can now be cancelled
mid-operation when user cancels a download
- **Smart warmup**: Chrome driver is pre-warmed when first client
connects (controlled by `BYPASS_WARMUP_ON_CONNECT` env var) and shuts
down after periods of inactivity

### 🔌 External Bypasser (FlareSolverr)

- **Improved resilience**: Retry with exponential backoff, mirror/DNS
rotation on failure, and proper timeout handling
- **Cancellation support**: External bypasser operations respect
cancellation flags

### 🖥️ Web UI Improvements

- **Simplified download status**: Removed intermediate states
(bypassing, verifying, ingesting) — now just shows Queued → Resolving →
Downloading → Complete
- **Status messages**: Downloads show detailed status like "Trying
Anna's Archive (Server 3)" or "Server busy, trying next...", or live
waitlist countdowns.
- **Improved download sidebar**:
  - Downloads sorted by add time (newest first)
  - X button moved to top-right corner for better UX
  - Wave animation on in-progress items
  - Error messages shown directly on failed items
  - X button on completed/errored items clears them from the list

### ⚙️ Configuration Changes

- **`CUSTOM_DNS=auto`** is now the default (previously empty/system DNS)
- **`DOWNLOAD_PROGRESS_UPDATE_INTERVAL`** default changed from 5s to 1s
for smoother progress
- **`BYPASS_WARMUP_ON_CONNECT`** (default: true) — warm up Chrome when
first client connects

### 🐛 Bug Fixes

- **Download cancellation actually works**: Fixed issue where cancelling
downloads didn't properly stop in-progress operations
- **WELIB prioritization**: Fixed `PRIORITIZE_WELIB` not being respected
- **File exists handling**: Downloads to same filename now get `_1`,
`_2` suffix instead of overwriting
- **Empty search results**: "No books found" now returns empty list
instead of throwing exception
- **Search unavailable error**: Network/mirror failures during search
now return proper 503 error to client
2025-12-14 21:18:05 -05:00
2b831dcfa5 [FEATURE] Separate download folders #122 (#297)
Re: Issue #122
Fetches content type from search results - displays it on thumbnails in
results grid;
Fetches content type from book id detail page (dfaults to "Other") and
uses it to construct the `final_path`.

---------

Co-authored-by: Patricia Ritter <pritter@events.com>
Co-authored-by: CaliBrain <calibrain@l4n.xyz>
2025-11-23 17:11:43 -05:00
CaliBrain 6492bd6a3c Fix rare case where special character might break parsing (#325)
Actual fix for #322
2025-11-23 00:05:15 -05:00
742da1c43a WebUI - Frontend Refactor (#302)
This PR was coauthored by alexhb1 and davidemarcoli. It builds on the FE
rework created by alex, but adds a myriad of additional tweaks and
optimizations to make the frontend feel modern, fast, and responsive.
The summary of the changes is as follows:

### Architecture Changes
React/TypeScript Migration: Refactored frontend from template/JS
structure to React/TypeScript application for better maintainability and
scalability
WebSocket Integration: Implemented real-time updates for download status
and progress with automatic fallback to polling
Gevent Worker: Configured production WebSocket support

### UI/UX Improvements
<img width="1502" height="890" alt="Screenshot 2025-11-10 at 10 02
59 AM"
src="https://github.com/user-attachments/assets/86bf8649-623f-413c-b8e5-656e687e55a8"
/>

Downloads Sidebar: Replaced bottom downloads section with sidebar
interface for better organization
<img width="201" height="450" alt="Screenshot 2025-11-10 at 10 07 52 AM"
src="https://github.com/user-attachments/assets/92b98e7c-c3bc-4b7e-80f1-252c3a760e33"
/>

Status Badges: Color-coded download status indicators instead of plain
text
Pinned Header: Fixed header position for consistent navigation
Enhanced Book Cards: Improved layout and hover states with info modal
button
<img width="1474" height="899" alt="Screenshot 2025-11-10 at 10 08
18 AM"
src="https://github.com/user-attachments/assets/9216d8a3-f662-434d-80e6-2a69b96abc31"
/>

Download Progress: Circular progress indicator on download buttons
Toast Notifications: Added user feedback for actions
Spinner Feedback: Loading indicators on search and download buttons
Animations: Smooth transitions and fluid progress updates

### Mobile & Responsive Design
Mobile-friendly Layouts: Optimized book cards and search interface for
mobile
<img width="225" height="450" alt="Screenshot 2025-11-10 at 10 05 49 AM"
src="https://github.com/user-attachments/assets/c8236c1c-5837-4309-9577-46db7292a54b"
/>

Keyboard Handling: Improved mobile keyboard behavior with proper input
types
PWA Improvements: Enhanced progressive web app functionality
Responsive Search: Better search box width and positioning across
devices

### Developer Experience
Development Mode: Separate frontend dev server that works with existing
backend container
Makefile: Added build automation and development commands
Documentation: Updated README with frontend architecture details

### Bug Fixes
Fixed "Clear completed" functionality
Fixed dark mode toggle text
Fixed sticky header behavior
Fixed mobile search box positioning
Removed active downloads requirement for initial state view

### Additional Features
ESC Key: Close downloads sidebar with ESC key
Calibre-Web Button: Direct link to Calibre-Web instance
<img width="282" height="83" alt="Screenshot 2025-11-11 at 9 38 05 AM"
src="https://github.com/user-attachments/assets/273075be-9743-4e13-9e48-5bf498f6c067"
/>
Granular Status Tracking: More detailed download progress information
obtained via websockets

---------

Co-authored-by: Alex <alex.bilbie1@gmail.com>
Co-authored-by: Zack Yancey <yanceyz@proton.me>
Co-authored-by: davidemarcoli <davide@marcoli.ch>
2025-11-14 15:48:44 -05:00
CaliBrain cd16f09f2e Fix local download (#282) 2025-09-16 11:19:09 -04:00
CaliBrain e4d3a372c8 Add retry logic for failed file copy (#261) 2025-08-29 22:32:04 -04:00
CaliBrain c8f21b8f8d Fix progression in download (#248) 2025-08-25 23:59:05 -04:00
Stephon ParkerandCaliBrain 5e04b6bfb8 Add Download Queue Management & Concurrent Downloads (#231)
## Summary

This PR completely overhauls the download system to support concurrent
downloads, cancellation, and advanced queue
management. No more waiting for stuck downloads to block the entire
queue!

  ## New Features

  ###  **Concurrent Downloads**
  - **3 simultaneous downloads** by default (configurable)
- ThreadPoolExecutor-based architecture replaces single-threaded
bottleneck
  - Downloads no longer block each other

  ###  **Download Cancellation**
  - **Cancel button** for active/queued downloads
  - Clean cancellation with partial file cleanup
  - Thread-safe cancellation flags

  ### **Queue Priority & Reordering**
  - **Priority-based queue** (lower number = higher priority)
  - **Editable priority inputs** in UI for queued items
  - **Bulk reordering** API support

  ## UI Enhancements

- **Enhanced status table** with Priority, Progress, and Actions columns
  - **Progress bars** for active downloads
  - **Cancel buttons** for downloads
- **Queue management controls**: Refresh, Clear Completed, Active
Counter
  - **Real-time updates** every 60 seconds

## Implementation

  ### New Architecture
  - `queue.PriorityQueue` replaces simple `set()` for proper ordering
  - `concurrent.futures.ThreadPoolExecutor` handles worker threads
  - `threading.Event` flags enable clean cancellation
  - Thread-safe operations with proper locking

  ### New API Endpoints
  DELETE /api/download/{id}/cancel     # Cancel downloads
PUT /api/queue/{id}/priority # Set priorityPOST /api/queue/reorder #
Bulk reorder
  GET /api/downloads/active          # Active tracking
  DELETE /api/queue/clear            # Clear completed

  ### New Environment Variables
  ```bash
  MAX_CONCURRENT_DOWNLOADS=3          # Worker thread count
  DOWNLOAD_PROGRESS_UPDATE_INTERVAL=5  # Progress frequency
  ```

### Fully backward compatible - existing functionality unchanged

### Fixes

  - Resolves: Downloads blocking when one gets stuck
  - Resolves: No way to cancel problematic downloads
  - Resolves: No queue management or prioritization
  
  #183

---------

Co-authored-by: CaliBrain <calibrain@l4n.xyz>
2025-08-25 13:28:35 -04:00
CaliBrain e1d94f9cb5 Update download handling and URL selection logic (#203)
Changed intermediate download file naming in backend.py to use
.crdownload extension. Refactored URL selection logic in book_manager.py
to adjust order and inclusion based on Cloudflare bypass and donator
key. Added extra Chromium arguments in cloudflare_bypasser.py to ignore
certificate and SSL errors for improved bypass reliability.
2025-07-14 16:05:09 -04:00
CaliBrain 690c2b699c Fix download for other files than epub (#174)
Fix #173
2025-05-06 08:44:43 -04:00
CaliBrain 457351fa0b Better File check (#165)
- Better file permission initialization
- Fix chown bug
2025-04-27 13:49:07 -04:00
CaliBrain 49287ac86e Revert "Better file permission initialization" (#164)
Reverts calibrain/calibre-web-automated-book-downloader#162
2025-04-27 12:43:07 -04:00
CaliBrain 4ca8298b7e Better file permission initialization (#162) 2025-04-27 01:27:48 -04:00
CaliBrain 0ba7031a83 Fix download + add test + fix book move (#140) 2025-04-16 11:18:03 -04:00
CaliBrain e0e7470ac6 Add Backup method to move book after download (#119)
Fix #118
2025-04-04 18:10:08 -04:00
CaliBrain 93a3481750 Fix errors and improve logging (#102)
Fix #100
Fix #101
2025-03-22 17:35:12 -04:00
CaliBrain c2e7acd473 Use a tmp name while downloading files (#99)
Should help with #98
2025-03-20 00:56:54 -04:00
CaliBrain 392a8b5262 feat: add USE_BOOK_TITLE option for customizing download filenames (#95)
Implements #94
2025-03-16 14:42:19 -04:00
CaliBrain 909f4b403c Multiple bug fixes (#93) 2025-03-16 03:33:59 -04:00
CaliBrain d5b2e24423 feat: Add CUSTOM_SCRIPT documentation and implementation details (#90) 2025-03-16 02:40:09 -04:00
CaliBrainandmik593 3a92c5de78 CF BYPASS (#24)
The Calibre dependency was due to the script testing for validity of the
downloaded file, as often they would be corrupted from aa. But CWA is
already doing that, so we are just having redundant code here.

For the cloudflarebypasser, I basically run my own version now, instead
of depending on an external library, this way we have better control for
debugging and on the docker image.

Fixes #18, #33, #27, #48, #65, #78, #86, #88, #89

---------

Co-authored-by: mik593 <91991279+mik593@users.noreply.github.com>
2025-03-16 02:25:15 -04:00
mik593 7f56c6b832 feat: support search filters (#55)
Still a WIP but an initial attempt of search filters. #7 

Currently supporting: 

- ISBN
- Author
- Title
- Language
- Sort
- Content


first attempt at some sort of UI for it 👇

![image](https://github.com/user-attachments/assets/7237ae96-7e97-4db0-98da-06c3171aca3b)

smaller/mobile

![image](https://github.com/user-attachments/assets/ba20de0e-deb1-4aa1-9c5f-27cf473670e6)
2025-01-22 16:51:06 -05:00
CaliBrain b8a7247538 [FIX] Refactor code to fix typing (#23)
This also fixes some bug with the CF bypasser and the file checker
2024-12-22 22:11:35 -05:00
CaliBrain 7597054649 Fix #20 : Network error when using alternative download streams (#22) 2024-12-22 19:25:03 -05:00
CaliBrain 18b146af64 Innitial Release 2024-12-16 22:48:26 +00:00