mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-09-24 20:20:20 +01:00
Lance Marks
f441b85da2
## Multi-book packs: inspect a release before download and file each book separately Closes #576 ### Problem One queued release is always treated as one book. When a torrent is actually a whole series (`Series/Book 1 - Title/…`, or a flat folder of `Series 1.0 - Title.m4b` files), post-processing walks the whole tree, flattens every file into one list and renames them `Title - 01…10` under the searched book's `{Author}/{Title}`. Audiobookshelf then sees a single 10-file "book" and the user has to re-file everything by hand. ### What this does Most releases expose their file list *before* anything is downloaded, so the split is decided up front and approved by the user, then the download is fire-and-forget: 1. **Inspect** – clicking a release's download button now calls `POST /api/releases/inspect` first. A new optional `DownloadHandler.list_files(release_data)` hook returns the release's files without downloading: - **AudiobookBay** reads the torrent file table off the detail page it already fetches (the page is now cached for 120 s, so inspect + download cost ABB one request). - **Prowlarr** parses `info.files` from the `.torrent` it already fetches (the existing 120 s torrent-fetch cache is reused). Magnet-only and usenet releases report "can't inspect". - Other sources default to `None`. 2. **Review** – if the plan contains more than one book, the Find Releases modal swaps the list for a review panel: one row per book with editable title / series position / year, expandable file lists, non-book sidecars (`.txt`, covers) shown as ignored, a "Treat as a single book" switch, and **Download N books**. Single-book releases queue immediately, exactly as before. 3. **File** – the approved plan travels with the task (`DownloadTask.book_plan`, retry-safe) and post-processing files each book through the existing transfer code, one book at a time (`dataclasses.replace(task, title=…, series_position=…, year=…)`), so organize/rename templates, part numbering (now scoped per book), hardlinks, torrent copy-preserve and usenet handling are unchanged. Status reads `Complete (N books, M files)`. 4. **Fallback** – when a release can't be inspected the user gets a toast, and a small "Multi-book pack" toggle in the modal header forces a heuristic split (subfolder = book, or one book per file when the file names carry series positions). Planning lives in `shelfmark/download/postprocess/packs.py` and is shared by the inspect endpoint and post-processing, so what the user approved is what gets filed. The name parser strips `Book 3 -`, `03 -`, `1.0 -`, `3.`, `[03]`, `#3`, a leading series name, labels like "An Expanse Novella -", repeated titles (`Gods of Risk 2.5 - Gods of Risk`) and a trailing `(Year)`; author and series name come from the book that was searched, and the searched book's own series position is never applied to its siblings. ### Files - `shelfmark/download/postprocess/packs.py` (new) – `PackFile/PackBook/PackPlan`, `plan_pack`, `parse_pack_book_name`, `group_files_into_books`, `match_plan_to_files` - `shelfmark/core/release_inspect_routes.py` (new) – `POST /api/releases/inspect` - `shelfmark/release_sources/__init__.py` – `DownloadHandler.list_files` hook - `shelfmark/release_sources/audiobookbay/{scraper,handler}.py` – detail-page cache, `extract_file_list`, `list_files` - `shelfmark/release_sources/prowlarr/handler.py`, `download/clients/torrent_utils.py` – `extract_file_list_from_torrent`, `list_files` - `shelfmark/core/models.py`, `download/orchestrator.py` – `multi_book` / `book_plan` fields, queue + retry serialization - `shelfmark/download/postprocess/transfer.py`, `pipeline.py`, `outputs/folder.py` – per-book transfer branch and status message - `src/frontend`: `components/PackReviewPanel.tsx` (new), `ReleaseModal.tsx`, `App.tsx`, `services/api.ts`, `types/index.ts`, `utils/releasePayload.ts` (payload builder moved out of `App.tsx`), `utils/packReview.ts` - `docs/dev/release-sources-plugin-guide.md` – documents the `list_files` hook ### Out of scope (follow-ups) - Listing files from an NZB (Shelfmark already fetches the bytes; `<file subject>` names are noisy) - Inspecting magnet links via qBittorrent's files API after a paused add - BookLore / email outputs (they ignore `book_plan`; noted in code) - The combined ebook + audiobook flow ### Testing **Automated** (`make checks`, `make python-test`, `make frontend-test` all green; the only failures on my machine are the pre-existing `tests/config/test_entrypoint_permissions.py` cases, which need bash ≥ 4 and fail identically on `main` under macOS bash 3.2): - `tests/download/test_packs.py` – name parsing (markers, series name, novella labels, repeated titles, bare numeric titles like `1984`), nested / flat / mixed / deeper-nested packs, single wrapping folder not treated as a pack, plan-to-disk matching with basename fallback - `tests/core/test_processing_packs.py` – full `post_process_download` runs on a real temp filesystem: approved plan files each book under its own `{Author}/{Title}`, heuristic split of a nested pack, searched book's series position does not leak, multi-file book inside a pack keeps `- 01/- 02` per book, hardlinked torrent pack leaves the seeding tree intact, no pack fields ⇒ behaviour unchanged, single group degrades to the searched title, status message - `tests/core/test_release_inspect_routes.py` – plan response, not-inspectable, handler errors never 500, unknown source / missing `source_id` ⇒ 400, login required - `tests/audiobookbay/test_file_list.py` – file-table scraping from real ABB markup (multi-file and single-file pages), handler host validation, one page fetch shared by magnet + file list - `tests/prowlarr/test_torrent_file_list.py` – multi-file / single-file `.torrent` parsing, handler behaviour for torrent URL vs magnet vs usenet vs cache miss - `tests/download/test_orchestrator_pack_fields.py` – queue-time parsing and retry round-trip - Frontend: `releasePayload.test.ts`, `packReview.test.ts` (vitest) **Manual, on a real deployment** (arm64 image built from this branch, run as a side container next to production with the same qBittorrent / Audiobookshelf setup, `FILE_ORGANIZATION_AUDIOBOOK=organize`, hardlinks on): - AudiobookBay "The Expanse Complete 2.0" (7.87 GB, 36 files): clicking download opened the review panel in ~1 s showing **18 books · 18 files · 18 files ignored** (the `.txt` sidecars), with series positions 0.1–9.5 and years parsed from the file names; novella labels stripped ("The Churn", "The Butcher of Anderson Station"). Editing a title in the panel works. Confirming queued one task; the magnet resolved from the cached page in ~30 ms; after the download the task reported `Complete (18 books, 18 files)`, 18 hardlinks landed as `audiobooks/James S. A. Corey/<Title>/<Title>.m4b`, the torrent kept seeding, and Audiobookshelf scanned each folder as its own book (title, author, embedded chapters). - A second pack ("Expanse [01 - 9.5]", `Title N - Title` naming) was inspected to verify the repeated-title rule and the Back button, without downloading. - Single-book releases still queue immediately with no extra UI.
244 lines
4.5 KiB
Plaintext
244 lines
4.5 KiB
Plaintext
# Created by https://www.toptal.com/developers/gitignore/api/macos,visualstudiocode,python
|
|
# Edit at https://www.toptal.com/developers/gitignore?templates=macos,visualstudiocode,python
|
|
|
|
### macOS ###
|
|
# General
|
|
.DS_Store
|
|
.AppleDouble
|
|
.LSOverride
|
|
|
|
# Icon must end with two \r
|
|
Icon
|
|
|
|
|
|
# Thumbnails
|
|
._*
|
|
|
|
# Files that might appear in the root of a volume
|
|
.DocumentRevisions-V100
|
|
.fseventsd
|
|
.Spotlight-V100
|
|
.TemporaryItems
|
|
.Trashes
|
|
.VolumeIcon.icns
|
|
.com.apple.timemachine.donotpresent
|
|
|
|
# Directories potentially created on remote AFP share
|
|
.AppleDB
|
|
.AppleDesktop
|
|
Network Trash Folder
|
|
Temporary Items
|
|
.apdisk
|
|
|
|
### macOS Patch ###
|
|
# iCloud generated files
|
|
*.icloud
|
|
|
|
### Python ###
|
|
# Byte-compiled / optimized / DLL files
|
|
__pycache__/
|
|
*.py[cod]
|
|
*$py.class
|
|
|
|
# C extensions
|
|
*.so
|
|
|
|
# Distribution / packaging
|
|
.Python
|
|
build/
|
|
develop-eggs/
|
|
dist/
|
|
downloads/
|
|
eggs/
|
|
.eggs/
|
|
lib/
|
|
lib64/
|
|
parts/
|
|
sdist/
|
|
var/
|
|
wheels/
|
|
share/python-wheels/
|
|
*.egg-info/
|
|
.installed.cfg
|
|
*.egg
|
|
MANIFEST
|
|
|
|
# PyInstaller
|
|
# Usually these files are written by a python script from a template
|
|
# before PyInstaller builds the exe, so as to inject date/other infos into it.
|
|
*.manifest
|
|
*.spec
|
|
|
|
# Installer logs
|
|
pip-log.txt
|
|
pip-delete-this-directory.txt
|
|
|
|
# Unit test / coverage reports
|
|
src/frontend/coverage/
|
|
htmlcov/
|
|
.tox/
|
|
.nox/
|
|
.coverage
|
|
.coverage.*
|
|
.cache
|
|
nosetests.xml
|
|
coverage.xml
|
|
*.cover
|
|
*.py,cover
|
|
.hypothesis/
|
|
.pytest_cache/
|
|
cover/
|
|
|
|
# Translations
|
|
*.mo
|
|
*.pot
|
|
|
|
# Django stuff:
|
|
*.log
|
|
local_settings.py
|
|
db.sqlite3
|
|
db.sqlite3-journal
|
|
|
|
# Flask stuff:
|
|
instance/
|
|
.webassets-cache
|
|
|
|
# Scrapy stuff:
|
|
.scrapy
|
|
|
|
# Sphinx documentation
|
|
docs/_build/
|
|
|
|
# PyBuilder
|
|
.pybuilder/
|
|
target/
|
|
|
|
# Jupyter Notebook
|
|
.ipynb_checkpoints
|
|
|
|
# IPython
|
|
profile_default/
|
|
ipython_config.py
|
|
|
|
# pyenv
|
|
# For a library or package, you might want to ignore these files since the code is
|
|
# intended to run in multiple environments; otherwise, check them in:
|
|
# .python-version
|
|
|
|
# pipenv
|
|
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
|
|
# However, in case of collaboration, if having platform-specific dependencies or dependencies
|
|
# having no cross-platform support, pipenv may install dependencies that don't work, or not
|
|
# install all needed dependencies.
|
|
#Pipfile.lock
|
|
|
|
# poetry
|
|
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
|
|
# This is especially recommended for binary packages to ensure reproducibility, and is more
|
|
# commonly ignored for libraries.
|
|
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
|
|
#poetry.lock
|
|
|
|
# pdm
|
|
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
|
|
#pdm.lock
|
|
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
|
|
# in version control.
|
|
# https://pdm.fming.dev/#use-with-ide
|
|
.pdm.toml
|
|
|
|
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
|
|
__pypackages__/
|
|
|
|
# Celery stuff
|
|
celerybeat-schedule
|
|
celerybeat.pid
|
|
|
|
# SageMath parsed files
|
|
*.sage.py
|
|
|
|
# Environments
|
|
.env
|
|
.venv
|
|
env/
|
|
venv/
|
|
ENV/
|
|
env.bak/
|
|
venv.bak/
|
|
|
|
# ...but the e2e platform test profiles live in an env/ dir and must be tracked
|
|
!tests/e2e/platform/env/
|
|
!tests/e2e/platform/env/*.env
|
|
|
|
# Spyder project settings
|
|
.spyderproject
|
|
.spyproject
|
|
|
|
# Rope project settings
|
|
.ropeproject
|
|
|
|
# mkdocs documentation
|
|
/site
|
|
|
|
# mypy
|
|
.mypy_cache/
|
|
.dmypy.json
|
|
dmypy.json
|
|
|
|
# Pyre type checker
|
|
.pyre/
|
|
|
|
# pytype static type analyzer
|
|
.pytype/
|
|
|
|
# Cython debug symbols
|
|
cython_debug/
|
|
|
|
# PyCharm
|
|
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
|
|
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
|
|
# and can be added to the global gitignore or merged into this file. For a more nuclear
|
|
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
|
|
#.idea/
|
|
|
|
### Python Patch ###
|
|
# Poetry local configuration file - https://python-poetry.org/docs/configuration/#local-configuration
|
|
poetry.toml
|
|
|
|
# ruff
|
|
.ruff_cache/
|
|
|
|
# LSP config files
|
|
pyrightconfig.json
|
|
|
|
### VisualStudioCode ###
|
|
.vscode/*
|
|
!.vscode/settings.json
|
|
!.vscode/tasks.json
|
|
!.vscode/launch.json
|
|
!.vscode/extensions.json
|
|
!.vscode/*.code-snippets
|
|
|
|
# Local History for Visual Studio Code
|
|
.history/
|
|
|
|
# Built Visual Studio Code Extensions
|
|
*.vsix
|
|
|
|
### VisualStudioCode Patch ###
|
|
# Ignore all local history of files
|
|
.history
|
|
.ionide
|
|
|
|
# End of https://www.toptal.com/developers/gitignore/api/macos,visualstudiocode,python
|
|
/downloaded_files
|
|
/.local/
|
|
*.local.*
|
|
AGENTS.md
|
|
.claude/
|
|
CLAUDE.md
|
|
.nvmrc
|
|
.playwright-mcp/
|
|
frontend-dist/
|
|
node_modules/
|