Files
shelfmark/tests/core/test_processing_packs.py
T
Lance Marks f441b85da2 feat(packs): inspect multi-book releases and file each book separately (#1270)
## Multi-book packs: inspect a release before download and file each
book separately

Closes #576

### Problem

One queued release is always treated as one book. When a torrent is
actually a whole series
(`Series/Book 1 - Title/…`, or a flat folder of `Series 1.0 - Title.m4b`
files), post-processing
walks the whole tree, flattens every file into one list and renames them
`Title - 01…10` under the
searched book's `{Author}/{Title}`. Audiobookshelf then sees a single
10-file "book" and the user
has to re-file everything by hand.

### What this does

Most releases expose their file list *before* anything is downloaded, so
the split is decided up
front and approved by the user, then the download is fire-and-forget:

1. **Inspect** – clicking a release's download button now calls `POST
/api/releases/inspect`
first. A new optional `DownloadHandler.list_files(release_data)` hook
returns the release's
   files without downloading:
- **AudiobookBay** reads the torrent file table off the detail page it
already fetches (the
page is now cached for 120 s, so inspect + download cost ABB one
request).
- **Prowlarr** parses `info.files` from the `.torrent` it already
fetches (the existing 120 s
torrent-fetch cache is reused). Magnet-only and usenet releases report
"can't inspect".
   - Other sources default to `None`.
2. **Review** – if the plan contains more than one book, the Find
Releases modal swaps the list
for a review panel: one row per book with editable title / series
position / year, expandable
file lists, non-book sidecars (`.txt`, covers) shown as ignored, a
"Treat as a single book"
switch, and **Download N books**. Single-book releases queue
immediately, exactly as before.
3. **File** – the approved plan travels with the task
(`DownloadTask.book_plan`, retry-safe) and
post-processing files each book through the existing transfer code, one
book at a time
(`dataclasses.replace(task, title=…, series_position=…, year=…)`), so
organize/rename
templates, part numbering (now scoped per book), hardlinks, torrent
copy-preserve and usenet
   handling are unchanged. Status reads `Complete (N books, M files)`.
4. **Fallback** – when a release can't be inspected the user gets a
toast, and a small
"Multi-book pack" toggle in the modal header forces a heuristic split
(subfolder = book, or
   one book per file when the file names carry series positions).

Planning lives in `shelfmark/download/postprocess/packs.py` and is
shared by the inspect endpoint
and post-processing, so what the user approved is what gets filed. The
name parser strips
`Book 3 -`, `03 -`, `1.0 -`, `3.`, `[03]`, `#3`, a leading series name,
labels like
"An Expanse Novella -", repeated titles (`Gods of Risk 2.5 - Gods of
Risk`) and a trailing
`(Year)`; author and series name come from the book that was searched,
and the searched book's
own series position is never applied to its siblings.

### Files

- `shelfmark/download/postprocess/packs.py` (new) –
`PackFile/PackBook/PackPlan`, `plan_pack`,
`parse_pack_book_name`, `group_files_into_books`, `match_plan_to_files`
- `shelfmark/core/release_inspect_routes.py` (new) – `POST
/api/releases/inspect`
- `shelfmark/release_sources/__init__.py` – `DownloadHandler.list_files`
hook
- `shelfmark/release_sources/audiobookbay/{scraper,handler}.py` –
detail-page cache,
  `extract_file_list`, `list_files`
- `shelfmark/release_sources/prowlarr/handler.py`,
`download/clients/torrent_utils.py` –
  `extract_file_list_from_torrent`, `list_files`
- `shelfmark/core/models.py`, `download/orchestrator.py` – `multi_book`
/ `book_plan` fields,
  queue + retry serialization
- `shelfmark/download/postprocess/transfer.py`, `pipeline.py`,
`outputs/folder.py` – per-book
  transfer branch and status message
- `src/frontend`: `components/PackReviewPanel.tsx` (new),
`ReleaseModal.tsx`, `App.tsx`,
`services/api.ts`, `types/index.ts`, `utils/releasePayload.ts` (payload
builder moved out of
  `App.tsx`), `utils/packReview.ts`
- `docs/dev/release-sources-plugin-guide.md` – documents the
`list_files` hook

### Out of scope (follow-ups)

- Listing files from an NZB (Shelfmark already fetches the bytes; `<file
subject>` names are noisy)
- Inspecting magnet links via qBittorrent's files API after a paused add
- BookLore / email outputs (they ignore `book_plan`; noted in code)
- The combined ebook + audiobook flow

### Testing

**Automated** (`make checks`, `make python-test`, `make frontend-test`
all green; the only
failures on my machine are the pre-existing
`tests/config/test_entrypoint_permissions.py` cases,
which need bash ≥ 4 and fail identically on `main` under macOS bash
3.2):

- `tests/download/test_packs.py` – name parsing (markers, series name,
novella labels, repeated
titles, bare numeric titles like `1984`), nested / flat / mixed /
deeper-nested packs, single
wrapping folder not treated as a pack, plan-to-disk matching with
basename fallback
- `tests/core/test_processing_packs.py` – full `post_process_download`
runs on a real temp
filesystem: approved plan files each book under its own
`{Author}/{Title}`, heuristic split
of a nested pack, searched book's series position does not leak,
multi-file book inside a pack
keeps `- 01/- 02` per book, hardlinked torrent pack leaves the seeding
tree intact, no pack
fields ⇒ behaviour unchanged, single group degrades to the searched
title, status message
- `tests/core/test_release_inspect_routes.py` – plan response,
not-inspectable, handler errors
  never 500, unknown source / missing `source_id` ⇒ 400, login required
- `tests/audiobookbay/test_file_list.py` – file-table scraping from real
ABB markup (multi-file
and single-file pages), handler host validation, one page fetch shared
by magnet + file list
- `tests/prowlarr/test_torrent_file_list.py` – multi-file / single-file
`.torrent` parsing,
  handler behaviour for torrent URL vs magnet vs usenet vs cache miss
- `tests/download/test_orchestrator_pack_fields.py` – queue-time parsing
and retry round-trip
- Frontend: `releasePayload.test.ts`, `packReview.test.ts` (vitest)

**Manual, on a real deployment** (arm64 image built from this branch,
run as a side container
next to production with the same qBittorrent / Audiobookshelf setup,
`FILE_ORGANIZATION_AUDIOBOOK=organize`,
hardlinks on):

- AudiobookBay "The Expanse Complete 2.0" (7.87 GB, 36 files): clicking
download opened the review
panel in ~1 s showing **18 books · 18 files · 18 files ignored** (the
`.txt` sidecars), with
series positions 0.1–9.5 and years parsed from the file names; novella
labels stripped
("The Churn", "The Butcher of Anderson Station"). Editing a title in the
panel works.
Confirming queued one task; the magnet resolved from the cached page in
~30 ms; after the
download the task reported `Complete (18 books, 18 files)`, 18 hardlinks
landed as
`audiobooks/James S. A. Corey/<Title>/<Title>.m4b`, the torrent kept
seeding, and
Audiobookshelf scanned each folder as its own book (title, author,
embedded chapters).
- A second pack ("Expanse [01 - 9.5]", `Title N - Title` naming) was
inspected to verify the
  repeated-title rule and the Back button, without downloading.
- Single-book releases still queue immediately with no extra UI.
2026-08-27 00:40:08 -04:00

213 lines
7.3 KiB
Python

"""Multi-book packs are filed one book at a time through the normal pipeline."""
import os
from pathlib import Path
from threading import Event
from unittest.mock import patch
from shelfmark.core.models import DownloadTask, SearchMode
from tests.core.test_processing_integration import _build_config, _sync_config
def _run(temp_path: Path, task: DownloadTask, ingest: Path, staging: Path, **config_kwargs):
from shelfmark.download.postprocess.router import post_process_download
statuses: list[tuple[str, str | None]] = []
with (
patch("shelfmark.core.config.config") as mock_config,
patch("shelfmark.config.env.TMP_DIR", staging),
):
mock_config.get = _build_config(
ingest,
organization=config_kwargs.pop("organization", "organize"),
supported_audiobook_formats=["m4b", "mp3"],
audiobook_organize_template=config_kwargs.pop(
"audiobook_organize_template", "{Author}/{Title}/{Title}{ - PartNumber}"
),
**config_kwargs,
)
mock_config.CUSTOM_SCRIPT = None
_sync_config(mock_config, mock_config)
result = post_process_download(
temp_path, task, Event(), lambda s, m=None: statuses.append((s, m))
)
return result, statuses
def _nested_pack(root: Path) -> Path:
pack = root / "Sun Eater"
for folder, name in (
("Book 1 - Empire of Silence", "empire.m4b"),
("Book 2 - Howling Dark", "howling.m4b"),
):
(pack / folder).mkdir(parents=True)
(pack / folder / name).write_text(name)
(pack / "cover.jpg").write_text("img")
return pack
def _audiobook_task(**overrides) -> DownloadTask:
fields = {
"task_id": "pack-1",
"source": "direct_download",
"title": "Drive",
"author": "James S. A. Corey",
"content_type": "audiobook",
"series_name": "The Expanse",
"series_position": 2.6,
"search_mode": SearchMode.UNIVERSAL,
}
fields.update(overrides)
return DownloadTask(**fields)
def test_approved_plan_files_each_book_with_its_own_title(tmp_path):
staging = tmp_path / "staging"
ingest = tmp_path / "ingest"
staging.mkdir()
ingest.mkdir()
pack = staging / "Expanse"
pack.mkdir()
for name in (
"The Expanse 1.0 - Leviathan Wakes (2011).m4b",
"The Expanse 2.0 - Caliban's War (2012).m4b",
):
(pack / name).write_text(name)
(pack / "The Expanse 1.0 - Leviathan Wakes (2011).txt").write_text("notes")
task = _audiobook_task(
title="Sun Eater", # the searched book; must not name the pack's books
book_plan=[
{
"title": "Leviathan Wakes (edited)",
"series_position": 1.0,
"year": 2011,
"files": ["The Expanse 1.0 - Leviathan Wakes (2011).m4b"],
},
{
"title": "Caliban's War",
"series_position": 2.0,
"year": 2012,
"files": ["The Expanse 2.0 - Caliban's War (2012).m4b"],
},
],
)
result, statuses = _run(pack, task, ingest, staging)
assert result is not None
author_dir = ingest / "James S. A. Corey"
assert sorted(p.name for p in author_dir.iterdir()) == [
"Caliban's War",
"Leviathan Wakes (edited)",
]
assert (author_dir / "Leviathan Wakes (edited)" / "Leviathan Wakes (edited).m4b").exists()
assert (author_dir / "Caliban's War" / "Caliban's War.m4b").exists()
assert statuses[-1] == ("complete", "Complete (2 books, 2 files)")
def test_multi_book_flag_splits_nested_pack_heuristically(tmp_path):
staging = tmp_path / "staging"
ingest = tmp_path / "ingest"
staging.mkdir()
ingest.mkdir()
pack = _nested_pack(staging)
result, _ = _run(pack, _audiobook_task(multi_book=True), ingest, staging)
assert result is not None
author_dir = ingest / "James S. A. Corey"
assert (author_dir / "Empire of Silence" / "Empire of Silence.m4b").exists()
assert (author_dir / "Howling Dark" / "Howling Dark.m4b").exists()
def test_pack_book_series_position_does_not_leak_from_searched_book(tmp_path):
staging = tmp_path / "staging"
ingest = tmp_path / "ingest"
staging.mkdir()
ingest.mkdir()
pack = staging / "Two"
for folder in ("Alpha", "Beta"):
(pack / folder).mkdir(parents=True)
(pack / folder / f"{folder.lower()}.m4b").write_text(folder)
result, _ = _run(
pack,
_audiobook_task(multi_book=True),
ingest,
staging,
audiobook_organize_template="{Author}/{SeriesPosition - }{Title}/{Title}",
)
assert result is not None
assert sorted(p.name for p in (ingest / "James S. A. Corey").iterdir()) == ["Alpha", "Beta"]
def test_multifile_book_inside_pack_keeps_part_numbers_per_book(tmp_path):
staging = tmp_path / "staging"
ingest = tmp_path / "ingest"
staging.mkdir()
ingest.mkdir()
pack = staging / "Pack"
(pack / "Book 1 - One").mkdir(parents=True)
(pack / "Book 2 - Two").mkdir(parents=True)
for i in (1, 2, 3):
(pack / "Book 1 - One" / f"part{i}.mp3").write_text(str(i))
(pack / "Book 2 - Two" / "two.mp3").write_text("t")
result, statuses = _run(pack, _audiobook_task(multi_book=True), ingest, staging)
assert result is not None
one = ingest / "James S. A. Corey" / "One"
assert sorted(p.name for p in one.iterdir()) == ["One - 01.mp3", "One - 02.mp3", "One - 03.mp3"]
assert (ingest / "James S. A. Corey" / "Two" / "Two.mp3").exists()
assert statuses[-1] == ("complete", "Complete (2 books, 4 files)")
def test_hardlinked_torrent_pack_leaves_source_tree_intact(tmp_path):
downloads = tmp_path / "downloads"
ingest = tmp_path / "ingest"
downloads.mkdir()
ingest.mkdir()
pack = _nested_pack(downloads)
task = _audiobook_task(source="prowlarr", multi_book=True, original_download_path=str(pack))
result, _ = _run(pack, task, ingest, tmp_path / "staging", hardlink=True)
assert result is not None
empire_src = pack / "Book 1 - Empire of Silence" / "empire.m4b"
empire_dst = ingest / "James S. A. Corey" / "Empire of Silence" / "Empire of Silence.m4b"
assert empire_src.exists()
assert empire_dst.exists()
assert os.stat(empire_src).st_ino == os.stat(empire_dst).st_ino
def test_without_pack_fields_nested_pack_is_still_one_book(tmp_path):
staging = tmp_path / "staging"
ingest = tmp_path / "ingest"
staging.mkdir()
ingest.mkdir()
pack = _nested_pack(staging)
result, _ = _run(pack, _audiobook_task(), ingest, staging)
assert result is not None
drive = ingest / "James S. A. Corey" / "Drive"
assert sorted(p.name for p in drive.iterdir()) == ["Drive - 01.m4b", "Drive - 02.m4b"]
def test_single_group_with_multi_book_flag_uses_searched_title(tmp_path):
staging = tmp_path / "staging"
ingest = tmp_path / "ingest"
staging.mkdir()
ingest.mkdir()
pack = staging / "Series" / "Book 1 - Solo"
pack.mkdir(parents=True)
(pack / "solo.m4b").write_text("s")
result, statuses = _run(pack.parent, _audiobook_task(multi_book=True), ingest, staging)
assert result is not None
assert (ingest / "James S. A. Corey" / "Drive" / "Drive.m4b").exists()
assert statuses[-1] == ("complete", "Complete")