starred/shelfmark
1
0
Fork 0
mirror of https://github.com/calibrain/shelfmark.git synced 2026-10-05 22:05:50 +01:00
Code Issues Packages Projects Releases 10 Wiki Activity

Add Download Queue Management & Concurrent Downloads (#231)

Browse Source 
## Summary

This PR completely overhauls the download system to support concurrent
downloads, cancellation, and advanced queue
management. No more waiting for stuck downloads to block the entire
queue!

  ## New Features

  ###  **Concurrent Downloads**
  - **3 simultaneous downloads** by default (configurable)
- ThreadPoolExecutor-based architecture replaces single-threaded
bottleneck
  - Downloads no longer block each other

  ###  **Download Cancellation**
  - **Cancel button** for active/queued downloads
  - Clean cancellation with partial file cleanup
  - Thread-safe cancellation flags

  ### **Queue Priority & Reordering**
  - **Priority-based queue** (lower number = higher priority)
  - **Editable priority inputs** in UI for queued items
  - **Bulk reordering** API support

  ## UI Enhancements

- **Enhanced status table** with Priority, Progress, and Actions columns
  - **Progress bars** for active downloads
  - **Cancel buttons** for downloads
- **Queue management controls**: Refresh, Clear Completed, Active
Counter
  - **Real-time updates** every 60 seconds

## Implementation

  ### New Architecture
  - `queue.PriorityQueue` replaces simple `set()` for proper ordering
  - `concurrent.futures.ThreadPoolExecutor` handles worker threads
  - `threading.Event` flags enable clean cancellation
  - Thread-safe operations with proper locking

  ### New API Endpoints
  DELETE /api/download/{id}/cancel     # Cancel downloads
PUT /api/queue/{id}/priority # Set priorityPOST /api/queue/reorder #
Bulk reorder
  GET /api/downloads/active          # Active tracking
  DELETE /api/queue/clear            # Clear completed

  ### New Environment Variables
  ```bash
  MAX_CONCURRENT_DOWNLOADS=3          # Worker thread count
  DOWNLOAD_PROGRESS_UPDATE_INTERVAL=5  # Progress frequency
  ```

### Fully backward compatible - existing functionality unchanged

### Fixes

  - Resolves: Downloads blocking when one gets stuck
  - Resolves: No way to cancel problematic downloads
  - Resolves: No queue management or prioritization
  
  #183

---------

Co-authored-by: CaliBrain <calibrain@l4n.xyz>
This commit is contained in:
Stephon Parker
2025-08-25 13:28:35 -04:00
committed by GitHub
co-authored by CaliBrain
parent 09bd5ae9f0
commit 5e04b6bfb8
8 changed files with 988 additions and 100 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docker-compose.yml
+8 -2
View File
@@ -1,6 +1,9 @@
services:
calibre-web-automated-book-downloader:
image: ghcr.io/calibrain/calibre-web-automated-book-downloader:latest
# Uncomment to build the image from the Dockerfile for local testing changes.
# Remember to comment out the image line above.
#build: .
container_name: calibre-web-automated-book-downloader
environment:
FLASK_PORT: 8084
@@ -11,7 +14,10 @@ services:
APP_ENV: prod
UID: 1000
GID: 100
CWA_DB_PATH: /auth/app.db
# CWA_DB_PATH: /auth/app.db # Comment out to disable authentication
# Queue management settings
MAX_CONCURRENT_DOWNLOADS: 3
DOWNLOAD_PROGRESS_UPDATE_INTERVAL: 5
ports:
- 8084:8084
restart: unless-stopped
@@ -20,5 +26,5 @@ services:
# the same as whatever you gave in "calibre-web-automated"
- /tmp/data/calibre-web/ingest:/cwa-book-ingest
# This is the location of CWA's app.db, which contains authentication
# details
# details. Comment out to disable authentication
#- /cwa/config/path/app.db:/auth/app.db:ro