mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-09-24 20:40:27 +01:00
## Multi-book packs: inspect a release before download and file each book separately Closes #576 ### Problem One queued release is always treated as one book. When a torrent is actually a whole series (`Series/Book 1 - Title/…`, or a flat folder of `Series 1.0 - Title.m4b` files), post-processing walks the whole tree, flattens every file into one list and renames them `Title - 01…10` under the searched book's `{Author}/{Title}`. Audiobookshelf then sees a single 10-file "book" and the user has to re-file everything by hand. ### What this does Most releases expose their file list *before* anything is downloaded, so the split is decided up front and approved by the user, then the download is fire-and-forget: 1. **Inspect** – clicking a release's download button now calls `POST /api/releases/inspect` first. A new optional `DownloadHandler.list_files(release_data)` hook returns the release's files without downloading: - **AudiobookBay** reads the torrent file table off the detail page it already fetches (the page is now cached for 120 s, so inspect + download cost ABB one request). - **Prowlarr** parses `info.files` from the `.torrent` it already fetches (the existing 120 s torrent-fetch cache is reused). Magnet-only and usenet releases report "can't inspect". - Other sources default to `None`. 2. **Review** – if the plan contains more than one book, the Find Releases modal swaps the list for a review panel: one row per book with editable title / series position / year, expandable file lists, non-book sidecars (`.txt`, covers) shown as ignored, a "Treat as a single book" switch, and **Download N books**. Single-book releases queue immediately, exactly as before. 3. **File** – the approved plan travels with the task (`DownloadTask.book_plan`, retry-safe) and post-processing files each book through the existing transfer code, one book at a time (`dataclasses.replace(task, title=…, series_position=…, year=…)`), so organize/rename templates, part numbering (now scoped per book), hardlinks, torrent copy-preserve and usenet handling are unchanged. Status reads `Complete (N books, M files)`. 4. **Fallback** – when a release can't be inspected the user gets a toast, and a small "Multi-book pack" toggle in the modal header forces a heuristic split (subfolder = book, or one book per file when the file names carry series positions). Planning lives in `shelfmark/download/postprocess/packs.py` and is shared by the inspect endpoint and post-processing, so what the user approved is what gets filed. The name parser strips `Book 3 -`, `03 -`, `1.0 -`, `3.`, `[03]`, `#3`, a leading series name, labels like "An Expanse Novella -", repeated titles (`Gods of Risk 2.5 - Gods of Risk`) and a trailing `(Year)`; author and series name come from the book that was searched, and the searched book's own series position is never applied to its siblings. ### Files - `shelfmark/download/postprocess/packs.py` (new) – `PackFile/PackBook/PackPlan`, `plan_pack`, `parse_pack_book_name`, `group_files_into_books`, `match_plan_to_files` - `shelfmark/core/release_inspect_routes.py` (new) – `POST /api/releases/inspect` - `shelfmark/release_sources/__init__.py` – `DownloadHandler.list_files` hook - `shelfmark/release_sources/audiobookbay/{scraper,handler}.py` – detail-page cache, `extract_file_list`, `list_files` - `shelfmark/release_sources/prowlarr/handler.py`, `download/clients/torrent_utils.py` – `extract_file_list_from_torrent`, `list_files` - `shelfmark/core/models.py`, `download/orchestrator.py` – `multi_book` / `book_plan` fields, queue + retry serialization - `shelfmark/download/postprocess/transfer.py`, `pipeline.py`, `outputs/folder.py` – per-book transfer branch and status message - `src/frontend`: `components/PackReviewPanel.tsx` (new), `ReleaseModal.tsx`, `App.tsx`, `services/api.ts`, `types/index.ts`, `utils/releasePayload.ts` (payload builder moved out of `App.tsx`), `utils/packReview.ts` - `docs/dev/release-sources-plugin-guide.md` – documents the `list_files` hook ### Out of scope (follow-ups) - Listing files from an NZB (Shelfmark already fetches the bytes; `<file subject>` names are noisy) - Inspecting magnet links via qBittorrent's files API after a paused add - BookLore / email outputs (they ignore `book_plan`; noted in code) - The combined ebook + audiobook flow ### Testing **Automated** (`make checks`, `make python-test`, `make frontend-test` all green; the only failures on my machine are the pre-existing `tests/config/test_entrypoint_permissions.py` cases, which need bash ≥ 4 and fail identically on `main` under macOS bash 3.2): - `tests/download/test_packs.py` – name parsing (markers, series name, novella labels, repeated titles, bare numeric titles like `1984`), nested / flat / mixed / deeper-nested packs, single wrapping folder not treated as a pack, plan-to-disk matching with basename fallback - `tests/core/test_processing_packs.py` – full `post_process_download` runs on a real temp filesystem: approved plan files each book under its own `{Author}/{Title}`, heuristic split of a nested pack, searched book's series position does not leak, multi-file book inside a pack keeps `- 01/- 02` per book, hardlinked torrent pack leaves the seeding tree intact, no pack fields ⇒ behaviour unchanged, single group degrades to the searched title, status message - `tests/core/test_release_inspect_routes.py` – plan response, not-inspectable, handler errors never 500, unknown source / missing `source_id` ⇒ 400, login required - `tests/audiobookbay/test_file_list.py` – file-table scraping from real ABB markup (multi-file and single-file pages), handler host validation, one page fetch shared by magnet + file list - `tests/prowlarr/test_torrent_file_list.py` – multi-file / single-file `.torrent` parsing, handler behaviour for torrent URL vs magnet vs usenet vs cache miss - `tests/download/test_orchestrator_pack_fields.py` – queue-time parsing and retry round-trip - Frontend: `releasePayload.test.ts`, `packReview.test.ts` (vitest) **Manual, on a real deployment** (arm64 image built from this branch, run as a side container next to production with the same qBittorrent / Audiobookshelf setup, `FILE_ORGANIZATION_AUDIOBOOK=organize`, hardlinks on): - AudiobookBay "The Expanse Complete 2.0" (7.87 GB, 36 files): clicking download opened the review panel in ~1 s showing **18 books · 18 files · 18 files ignored** (the `.txt` sidecars), with series positions 0.1–9.5 and years parsed from the file names; novella labels stripped ("The Churn", "The Butcher of Anderson Station"). Editing a title in the panel works. Confirming queued one task; the magnet resolved from the cached page in ~30 ms; after the download the task reported `Complete (18 books, 18 files)`, 18 hardlinks landed as `audiobooks/James S. A. Corey/<Title>/<Title>.m4b`, the torrent kept seeding, and Audiobookshelf scanned each folder as its own book (title, author, embedded chapters). - A second pack ("Expanse [01 - 9.5]", `Title N - Title` naming) was inspected to verify the repeated-title rule and the Back button, without downloading. - Single-book releases still queue immediately with no extra UI.
213 lines
7.3 KiB
Python
213 lines
7.3 KiB
Python
"""Multi-book packs are filed one book at a time through the normal pipeline."""
|
|
|
|
import os
|
|
from pathlib import Path
|
|
from threading import Event
|
|
from unittest.mock import patch
|
|
|
|
from shelfmark.core.models import DownloadTask, SearchMode
|
|
from tests.core.test_processing_integration import _build_config, _sync_config
|
|
|
|
|
|
def _run(temp_path: Path, task: DownloadTask, ingest: Path, staging: Path, **config_kwargs):
|
|
from shelfmark.download.postprocess.router import post_process_download
|
|
|
|
statuses: list[tuple[str, str | None]] = []
|
|
with (
|
|
patch("shelfmark.core.config.config") as mock_config,
|
|
patch("shelfmark.config.env.TMP_DIR", staging),
|
|
):
|
|
mock_config.get = _build_config(
|
|
ingest,
|
|
organization=config_kwargs.pop("organization", "organize"),
|
|
supported_audiobook_formats=["m4b", "mp3"],
|
|
audiobook_organize_template=config_kwargs.pop(
|
|
"audiobook_organize_template", "{Author}/{Title}/{Title}{ - PartNumber}"
|
|
),
|
|
**config_kwargs,
|
|
)
|
|
mock_config.CUSTOM_SCRIPT = None
|
|
_sync_config(mock_config, mock_config)
|
|
result = post_process_download(
|
|
temp_path, task, Event(), lambda s, m=None: statuses.append((s, m))
|
|
)
|
|
return result, statuses
|
|
|
|
|
|
def _nested_pack(root: Path) -> Path:
|
|
pack = root / "Sun Eater"
|
|
for folder, name in (
|
|
("Book 1 - Empire of Silence", "empire.m4b"),
|
|
("Book 2 - Howling Dark", "howling.m4b"),
|
|
):
|
|
(pack / folder).mkdir(parents=True)
|
|
(pack / folder / name).write_text(name)
|
|
(pack / "cover.jpg").write_text("img")
|
|
return pack
|
|
|
|
|
|
def _audiobook_task(**overrides) -> DownloadTask:
|
|
fields = {
|
|
"task_id": "pack-1",
|
|
"source": "direct_download",
|
|
"title": "Drive",
|
|
"author": "James S. A. Corey",
|
|
"content_type": "audiobook",
|
|
"series_name": "The Expanse",
|
|
"series_position": 2.6,
|
|
"search_mode": SearchMode.UNIVERSAL,
|
|
}
|
|
fields.update(overrides)
|
|
return DownloadTask(**fields)
|
|
|
|
|
|
def test_approved_plan_files_each_book_with_its_own_title(tmp_path):
|
|
staging = tmp_path / "staging"
|
|
ingest = tmp_path / "ingest"
|
|
staging.mkdir()
|
|
ingest.mkdir()
|
|
pack = staging / "Expanse"
|
|
pack.mkdir()
|
|
for name in (
|
|
"The Expanse 1.0 - Leviathan Wakes (2011).m4b",
|
|
"The Expanse 2.0 - Caliban's War (2012).m4b",
|
|
):
|
|
(pack / name).write_text(name)
|
|
(pack / "The Expanse 1.0 - Leviathan Wakes (2011).txt").write_text("notes")
|
|
|
|
task = _audiobook_task(
|
|
title="Sun Eater", # the searched book; must not name the pack's books
|
|
book_plan=[
|
|
{
|
|
"title": "Leviathan Wakes (edited)",
|
|
"series_position": 1.0,
|
|
"year": 2011,
|
|
"files": ["The Expanse 1.0 - Leviathan Wakes (2011).m4b"],
|
|
},
|
|
{
|
|
"title": "Caliban's War",
|
|
"series_position": 2.0,
|
|
"year": 2012,
|
|
"files": ["The Expanse 2.0 - Caliban's War (2012).m4b"],
|
|
},
|
|
],
|
|
)
|
|
|
|
result, statuses = _run(pack, task, ingest, staging)
|
|
|
|
assert result is not None
|
|
author_dir = ingest / "James S. A. Corey"
|
|
assert sorted(p.name for p in author_dir.iterdir()) == [
|
|
"Caliban's War",
|
|
"Leviathan Wakes (edited)",
|
|
]
|
|
assert (author_dir / "Leviathan Wakes (edited)" / "Leviathan Wakes (edited).m4b").exists()
|
|
assert (author_dir / "Caliban's War" / "Caliban's War.m4b").exists()
|
|
assert statuses[-1] == ("complete", "Complete (2 books, 2 files)")
|
|
|
|
|
|
def test_multi_book_flag_splits_nested_pack_heuristically(tmp_path):
|
|
staging = tmp_path / "staging"
|
|
ingest = tmp_path / "ingest"
|
|
staging.mkdir()
|
|
ingest.mkdir()
|
|
pack = _nested_pack(staging)
|
|
|
|
result, _ = _run(pack, _audiobook_task(multi_book=True), ingest, staging)
|
|
|
|
assert result is not None
|
|
author_dir = ingest / "James S. A. Corey"
|
|
assert (author_dir / "Empire of Silence" / "Empire of Silence.m4b").exists()
|
|
assert (author_dir / "Howling Dark" / "Howling Dark.m4b").exists()
|
|
|
|
|
|
def test_pack_book_series_position_does_not_leak_from_searched_book(tmp_path):
|
|
staging = tmp_path / "staging"
|
|
ingest = tmp_path / "ingest"
|
|
staging.mkdir()
|
|
ingest.mkdir()
|
|
pack = staging / "Two"
|
|
for folder in ("Alpha", "Beta"):
|
|
(pack / folder).mkdir(parents=True)
|
|
(pack / folder / f"{folder.lower()}.m4b").write_text(folder)
|
|
|
|
result, _ = _run(
|
|
pack,
|
|
_audiobook_task(multi_book=True),
|
|
ingest,
|
|
staging,
|
|
audiobook_organize_template="{Author}/{SeriesPosition - }{Title}/{Title}",
|
|
)
|
|
|
|
assert result is not None
|
|
assert sorted(p.name for p in (ingest / "James S. A. Corey").iterdir()) == ["Alpha", "Beta"]
|
|
|
|
|
|
def test_multifile_book_inside_pack_keeps_part_numbers_per_book(tmp_path):
|
|
staging = tmp_path / "staging"
|
|
ingest = tmp_path / "ingest"
|
|
staging.mkdir()
|
|
ingest.mkdir()
|
|
pack = staging / "Pack"
|
|
(pack / "Book 1 - One").mkdir(parents=True)
|
|
(pack / "Book 2 - Two").mkdir(parents=True)
|
|
for i in (1, 2, 3):
|
|
(pack / "Book 1 - One" / f"part{i}.mp3").write_text(str(i))
|
|
(pack / "Book 2 - Two" / "two.mp3").write_text("t")
|
|
|
|
result, statuses = _run(pack, _audiobook_task(multi_book=True), ingest, staging)
|
|
|
|
assert result is not None
|
|
one = ingest / "James S. A. Corey" / "One"
|
|
assert sorted(p.name for p in one.iterdir()) == ["One - 01.mp3", "One - 02.mp3", "One - 03.mp3"]
|
|
assert (ingest / "James S. A. Corey" / "Two" / "Two.mp3").exists()
|
|
assert statuses[-1] == ("complete", "Complete (2 books, 4 files)")
|
|
|
|
|
|
def test_hardlinked_torrent_pack_leaves_source_tree_intact(tmp_path):
|
|
downloads = tmp_path / "downloads"
|
|
ingest = tmp_path / "ingest"
|
|
downloads.mkdir()
|
|
ingest.mkdir()
|
|
pack = _nested_pack(downloads)
|
|
task = _audiobook_task(source="prowlarr", multi_book=True, original_download_path=str(pack))
|
|
|
|
result, _ = _run(pack, task, ingest, tmp_path / "staging", hardlink=True)
|
|
|
|
assert result is not None
|
|
empire_src = pack / "Book 1 - Empire of Silence" / "empire.m4b"
|
|
empire_dst = ingest / "James S. A. Corey" / "Empire of Silence" / "Empire of Silence.m4b"
|
|
assert empire_src.exists()
|
|
assert empire_dst.exists()
|
|
assert os.stat(empire_src).st_ino == os.stat(empire_dst).st_ino
|
|
|
|
|
|
def test_without_pack_fields_nested_pack_is_still_one_book(tmp_path):
|
|
staging = tmp_path / "staging"
|
|
ingest = tmp_path / "ingest"
|
|
staging.mkdir()
|
|
ingest.mkdir()
|
|
pack = _nested_pack(staging)
|
|
|
|
result, _ = _run(pack, _audiobook_task(), ingest, staging)
|
|
|
|
assert result is not None
|
|
drive = ingest / "James S. A. Corey" / "Drive"
|
|
assert sorted(p.name for p in drive.iterdir()) == ["Drive - 01.m4b", "Drive - 02.m4b"]
|
|
|
|
|
|
def test_single_group_with_multi_book_flag_uses_searched_title(tmp_path):
|
|
staging = tmp_path / "staging"
|
|
ingest = tmp_path / "ingest"
|
|
staging.mkdir()
|
|
ingest.mkdir()
|
|
pack = staging / "Series" / "Book 1 - Solo"
|
|
pack.mkdir(parents=True)
|
|
(pack / "solo.m4b").write_text("s")
|
|
|
|
result, statuses = _run(pack.parent, _audiobook_task(multi_book=True), ingest, staging)
|
|
|
|
assert result is not None
|
|
assert (ingest / "James S. A. Corey" / "Drive" / "Drive.m4b").exists()
|
|
assert statuses[-1] == ("complete", "Complete")
|