Files
shelfmark/tests/download/test_packs.py
T
Lance Marks f441b85da2 feat(packs): inspect multi-book releases and file each book separately (#1270)
## Multi-book packs: inspect a release before download and file each
book separately

Closes #576

### Problem

One queued release is always treated as one book. When a torrent is
actually a whole series
(`Series/Book 1 - Title/…`, or a flat folder of `Series 1.0 - Title.m4b`
files), post-processing
walks the whole tree, flattens every file into one list and renames them
`Title - 01…10` under the
searched book's `{Author}/{Title}`. Audiobookshelf then sees a single
10-file "book" and the user
has to re-file everything by hand.

### What this does

Most releases expose their file list *before* anything is downloaded, so
the split is decided up
front and approved by the user, then the download is fire-and-forget:

1. **Inspect** – clicking a release's download button now calls `POST
/api/releases/inspect`
first. A new optional `DownloadHandler.list_files(release_data)` hook
returns the release's
   files without downloading:
- **AudiobookBay** reads the torrent file table off the detail page it
already fetches (the
page is now cached for 120 s, so inspect + download cost ABB one
request).
- **Prowlarr** parses `info.files` from the `.torrent` it already
fetches (the existing 120 s
torrent-fetch cache is reused). Magnet-only and usenet releases report
"can't inspect".
   - Other sources default to `None`.
2. **Review** – if the plan contains more than one book, the Find
Releases modal swaps the list
for a review panel: one row per book with editable title / series
position / year, expandable
file lists, non-book sidecars (`.txt`, covers) shown as ignored, a
"Treat as a single book"
switch, and **Download N books**. Single-book releases queue
immediately, exactly as before.
3. **File** – the approved plan travels with the task
(`DownloadTask.book_plan`, retry-safe) and
post-processing files each book through the existing transfer code, one
book at a time
(`dataclasses.replace(task, title=…, series_position=…, year=…)`), so
organize/rename
templates, part numbering (now scoped per book), hardlinks, torrent
copy-preserve and usenet
   handling are unchanged. Status reads `Complete (N books, M files)`.
4. **Fallback** – when a release can't be inspected the user gets a
toast, and a small
"Multi-book pack" toggle in the modal header forces a heuristic split
(subfolder = book, or
   one book per file when the file names carry series positions).

Planning lives in `shelfmark/download/postprocess/packs.py` and is
shared by the inspect endpoint
and post-processing, so what the user approved is what gets filed. The
name parser strips
`Book 3 -`, `03 -`, `1.0 -`, `3.`, `[03]`, `#3`, a leading series name,
labels like
"An Expanse Novella -", repeated titles (`Gods of Risk 2.5 - Gods of
Risk`) and a trailing
`(Year)`; author and series name come from the book that was searched,
and the searched book's
own series position is never applied to its siblings.

### Files

- `shelfmark/download/postprocess/packs.py` (new) –
`PackFile/PackBook/PackPlan`, `plan_pack`,
`parse_pack_book_name`, `group_files_into_books`, `match_plan_to_files`
- `shelfmark/core/release_inspect_routes.py` (new) – `POST
/api/releases/inspect`
- `shelfmark/release_sources/__init__.py` – `DownloadHandler.list_files`
hook
- `shelfmark/release_sources/audiobookbay/{scraper,handler}.py` –
detail-page cache,
  `extract_file_list`, `list_files`
- `shelfmark/release_sources/prowlarr/handler.py`,
`download/clients/torrent_utils.py` –
  `extract_file_list_from_torrent`, `list_files`
- `shelfmark/core/models.py`, `download/orchestrator.py` – `multi_book`
/ `book_plan` fields,
  queue + retry serialization
- `shelfmark/download/postprocess/transfer.py`, `pipeline.py`,
`outputs/folder.py` – per-book
  transfer branch and status message
- `src/frontend`: `components/PackReviewPanel.tsx` (new),
`ReleaseModal.tsx`, `App.tsx`,
`services/api.ts`, `types/index.ts`, `utils/releasePayload.ts` (payload
builder moved out of
  `App.tsx`), `utils/packReview.ts`
- `docs/dev/release-sources-plugin-guide.md` – documents the
`list_files` hook

### Out of scope (follow-ups)

- Listing files from an NZB (Shelfmark already fetches the bytes; `<file
subject>` names are noisy)
- Inspecting magnet links via qBittorrent's files API after a paused add
- BookLore / email outputs (they ignore `book_plan`; noted in code)
- The combined ebook + audiobook flow

### Testing

**Automated** (`make checks`, `make python-test`, `make frontend-test`
all green; the only
failures on my machine are the pre-existing
`tests/config/test_entrypoint_permissions.py` cases,
which need bash ≥ 4 and fail identically on `main` under macOS bash
3.2):

- `tests/download/test_packs.py` – name parsing (markers, series name,
novella labels, repeated
titles, bare numeric titles like `1984`), nested / flat / mixed /
deeper-nested packs, single
wrapping folder not treated as a pack, plan-to-disk matching with
basename fallback
- `tests/core/test_processing_packs.py` – full `post_process_download`
runs on a real temp
filesystem: approved plan files each book under its own
`{Author}/{Title}`, heuristic split
of a nested pack, searched book's series position does not leak,
multi-file book inside a pack
keeps `- 01/- 02` per book, hardlinked torrent pack leaves the seeding
tree intact, no pack
fields ⇒ behaviour unchanged, single group degrades to the searched
title, status message
- `tests/core/test_release_inspect_routes.py` – plan response,
not-inspectable, handler errors
  never 500, unknown source / missing `source_id` ⇒ 400, login required
- `tests/audiobookbay/test_file_list.py` – file-table scraping from real
ABB markup (multi-file
and single-file pages), handler host validation, one page fetch shared
by magnet + file list
- `tests/prowlarr/test_torrent_file_list.py` – multi-file / single-file
`.torrent` parsing,
  handler behaviour for torrent URL vs magnet vs usenet vs cache miss
- `tests/download/test_orchestrator_pack_fields.py` – queue-time parsing
and retry round-trip
- Frontend: `releasePayload.test.ts`, `packReview.test.ts` (vitest)

**Manual, on a real deployment** (arm64 image built from this branch,
run as a side container
next to production with the same qBittorrent / Audiobookshelf setup,
`FILE_ORGANIZATION_AUDIOBOOK=organize`,
hardlinks on):

- AudiobookBay "The Expanse Complete 2.0" (7.87 GB, 36 files): clicking
download opened the review
panel in ~1 s showing **18 books · 18 files · 18 files ignored** (the
`.txt` sidecars), with
series positions 0.1–9.5 and years parsed from the file names; novella
labels stripped
("The Churn", "The Butcher of Anderson Station"). Editing a title in the
panel works.
Confirming queued one task; the magnet resolved from the cached page in
~30 ms; after the
download the task reported `Complete (18 books, 18 files)`, 18 hardlinks
landed as
`audiobooks/James S. A. Corey/<Title>/<Title>.m4b`, the torrent kept
seeding, and
Audiobookshelf scanned each folder as its own book (title, author,
embedded chapters).
- A second pack ("Expanse [01 - 9.5]", `Title N - Title` naming) was
inspected to verify the
  repeated-title rule and the Back button, without downloading.
- Single-book releases still queue immediately with no extra UI.
2026-08-27 00:40:08 -04:00

283 lines
12 KiB
Python

"""Tests for multi-book pack planning (shelfmark.download.postprocess.packs)."""
from pathlib import Path
import pytest
from shelfmark.download.postprocess.packs import (
PackBook,
PackFile,
group_files_into_books,
match_plan_to_files,
parse_pack_book_name,
plan_pack,
)
AUDIO = {"m4b", "mp3"}
class TestParsePackBookName:
@pytest.mark.parametrize(
("name", "expected"),
[
("Book 3 - Howling Dark", ("Howling Dark", 3.0, None)),
("Book 03: Howling Dark", ("Howling Dark", 3.0, None)),
("03 - Empire of Silence", ("Empire of Silence", 3.0, None)),
("2.5 - Interlude", ("Interlude", 2.5, None)),
("[03] Empire of Silence", ("Empire of Silence", 3.0, None)),
("#3 Empire of Silence", ("Empire of Silence", 3.0, None)),
("3. Empire of Silence", ("Empire of Silence", 3.0, None)),
("Empire of Silence", ("Empire of Silence", None, None)),
("Empire of Silence (2018)", ("Empire of Silence", None, 2018)),
],
)
def test_strips_series_markers(self, name, expected):
assert parse_pack_book_name(name, series_name=None) == expected
def test_strips_leading_series_name_and_trailing_year(self):
assert parse_pack_book_name(
"The Expanse 1.0 - Leviathan Wakes (2011)", series_name="The Expanse"
) == ("Leviathan Wakes", 1.0, 2011)
def test_series_name_match_is_case_insensitive(self):
assert parse_pack_book_name(
"the expanse 2.5 - Gods of Risk", series_name="The Expanse"
) == (
"Gods of Risk",
2.5,
None,
)
@pytest.mark.parametrize(
"name",
[
"The Expanse 0.2 - An Expanse Novella - The Churn (2014)",
"The Expanse 0.2 - The Expanse Novella - The Churn (2014)",
"The Expanse 0.2 - An Expanse Short Story - The Churn (2014)",
],
)
def test_strips_series_novella_label(self, name):
assert parse_pack_book_name(name, series_name="The Expanse") == ("The Churn", 0.2, 2014)
@pytest.mark.parametrize(
("name", "expected"),
[
("Uncrowned Cradle, Book 7", ("Uncrowned", 7.0, None)),
("Reaper Cradle, Volume 10", ("Reaper", 10.0, None)),
("Soulsmith Cradle, Book 2", ("Soulsmith", 2.0, None)),
("Wintersteel - Cradle Book 8", ("Wintersteel", 8.0, None)),
("Wintersteel (Cradle, Book 8)", ("Wintersteel", 8.0, None)),
# A trailing bare number is a chapter/part, never a series position.
("Unsouled - 02", ("Unsouled - 02", None, None)),
],
)
def test_trailing_series_marker(self, name, expected):
assert parse_pack_book_name(name, series_name="Cradle") == expected
def test_trailing_series_name_only_stripped_with_a_marker(self):
# "Stories from Cradle" is the title; nothing marks a position, so keep it.
assert parse_pack_book_name("Threshold: Stories from Cradle", series_name="Cradle") == (
"Threshold: Stories from Cradle",
None,
None,
)
@pytest.mark.parametrize(
("name", "expected"),
[
# AudiobookBay renders "<folder> <file>" as one flat string.
(
"Will Wight - Unsouled Cradle, Book 1 Will Wight - Unsouled Cradle, Book 1",
("Unsouled", 1.0, None),
),
(
"Will Wight - Skysworn Cradle, Book 4 Skysworn Cradle, Book 4",
("Skysworn", 4.0, None),
),
("Will Wight - Bloodline Cradle, Book 9", ("Bloodline", 9.0, None)),
],
)
def test_strips_author_prefix_and_glued_folder_name(self, name, expected):
assert (
parse_pack_book_name(name, series_name="Cradle", author_name="Will Wight") == expected
)
@pytest.mark.parametrize(
("name", "expected"),
[
("Gods of Risk 2.5 - Gods of Risk", ("Gods of Risk", 2.5, None)),
("Cibola Burn 4 - Cibola Burn (2014)", ("Cibola Burn", 4.0, 2014)),
("cibola burn 4 - Cibola Burn", ("Cibola Burn", 4.0, None)),
# Different text on each side is a real "Series N - Title" name, not a repeat.
("Sun Eater 2 - Howling Dark", ("Sun Eater 2 - Howling Dark", None, None)),
],
)
def test_collapses_title_repeated_around_the_position(self, name, expected):
assert parse_pack_book_name(name, series_name=None) == expected
def test_bare_numeric_title_is_left_alone(self):
assert parse_pack_book_name("1984", series_name=None) == ("1984", None, None)
def test_marker_that_would_leave_nothing_is_left_alone(self):
assert parse_pack_book_name("Book 3", series_name=None) == ("Book 3", None, None)
class TestPlanPack:
def test_nested_subfolders_become_separate_books(self):
files = [
PackFile("Sun Eater/Book 1 - Empire of Silence/Empire of Silence.m4b", 10),
PackFile("Sun Eater/Book 2 - Howling Dark/Howling Dark.m4b", 20),
PackFile("Sun Eater/cover.jpg", 1),
]
plan = plan_pack(files, supported_extensions=AUDIO, series_name=None)
assert plan.is_pack
assert [b.title for b in plan.books] == ["Empire of Silence", "Howling Dark"]
assert [b.series_position for b in plan.books] == [1.0, 2.0]
assert plan.books[0].files == ["Sun Eater/Book 1 - Empire of Silence/Empire of Silence.m4b"]
assert plan.ignored == ["Sun Eater/cover.jpg"]
def test_flat_pack_becomes_one_book_per_file_and_ignores_sidecars(self):
files = [
PackFile("The Expanse 1.0 - Leviathan Wakes (2011).m4b", 100),
PackFile("The Expanse 1.0 - Leviathan Wakes (2011).txt", 1),
PackFile("The Expanse 2.0 - Caliban's War (2012).m4b", 100),
]
plan = plan_pack(files, supported_extensions=AUDIO, series_name="The Expanse")
assert plan.is_pack
assert [(b.title, b.series_position, b.year) for b in plan.books] == [
("Leviathan Wakes", 1.0, 2011),
("Caliban's War", 2.0, 2012),
]
assert plan.ignored == ["The Expanse 1.0 - Leviathan Wakes (2011).txt"]
def test_audiobookbay_flat_list_with_trailing_markers_is_a_pack(self):
# ABB's file table has no folder separators: "<folder> <file> <size>".
files = [
PackFile(
"Will Wight - Unsouled Cradle, Book 1 Will Wight - Unsouled Cradle, Book 1.sfv", 1
),
PackFile(
"Will Wight - Unsouled Cradle, Book 1 Will Wight - Unsouled Cradle, Book 1.m4a", 9
),
PackFile("Will Wight - Skysworn Cradle, Book 4 Skysworn Cradle, Book 4.m4b", 9),
PackFile("Uncrowned Cradle, Book 7.m4b", 9),
PackFile(
"Will Wight - Reaper Cradle, Volume 10 Will Wight - Reaper Cradle, Volume 10.m4b", 9
),
]
plan = plan_pack(
files,
supported_extensions={"m4a", "m4b"},
series_name="Cradle",
author_name="Will Wight",
)
assert plan.is_pack
assert [(b.title, b.series_position) for b in plan.books] == [
("Unsouled", 1.0),
("Skysworn", 4.0),
("Uncrowned", 7.0),
("Reaper", 10.0),
]
def test_deeper_nesting_collapses_onto_book_folder(self):
files = [
PackFile("Book 1/CD1/01.mp3"),
PackFile("Book 1/CD2/01.mp3"),
PackFile("Book 2/01.mp3"),
]
plan = plan_pack(files, supported_extensions=AUDIO, series_name=None)
assert [b.files for b in plan.books] == [
["Book 1/CD1/01.mp3", "Book 1/CD2/01.mp3"],
["Book 2/01.mp3"],
]
def test_single_wrapping_folder_is_not_a_book_boundary(self):
# A torrent named "Series" containing one multi-part book is a single book.
files = [PackFile("Series/Book 1/01.mp3"), PackFile("Series/Book 1/02.mp3")]
plan = plan_pack(files, supported_extensions=AUDIO, series_name=None)
assert not plan.is_pack
assert len(plan.books) == 1
def test_root_files_and_subfolders_coexist(self):
files = [PackFile("Novella.m4b"), PackFile("Book 1/a.m4b"), PackFile("Book 1/b.m4b")]
plan = plan_pack(files, supported_extensions=AUDIO, series_name=None)
assert [b.files for b in plan.books] == [["Novella.m4b"], ["Book 1/a.m4b", "Book 1/b.m4b"]]
def test_single_file_is_not_a_pack(self):
plan = plan_pack([PackFile("Book.m4b")], supported_extensions=AUDIO, series_name=None)
assert not plan.is_pack
assert plan.books[0].title == "Book"
def test_empty_input(self):
plan = plan_pack([], supported_extensions=AUDIO, series_name=None)
assert plan.books == []
assert not plan.is_pack
class TestGroupFilesIntoBooks:
def test_groups_on_disk_files_by_top_level_folder(self, tmp_path: Path):
a = tmp_path / "Book 1 - A" / "a.m4b"
b = tmp_path / "Book 2 - B" / "b.m4b"
for f in (a, b):
f.parent.mkdir(parents=True)
f.write_bytes(b"x")
groups = group_files_into_books([a, b], series_name=None)
assert [(g.title, g.series_position, g.files) for g in groups] == [
("A", 1.0, [a]),
("B", 2.0, [b]),
]
class TestMatchPlanToFiles:
def test_matches_by_relative_path_then_basename(self, tmp_path: Path):
root = tmp_path / "staging" / "Sun Eater"
a = root / "Book 1 - A" / "a.m4b"
b = root / "Book 2 - B" / "b.m4b"
for f in (a, b):
f.parent.mkdir(parents=True)
f.write_bytes(b"x")
plan = [
PackBook(title="Alpha", series_position=1.0, year=2001, files=["Book 1 - A/a.m4b"]),
PackBook(title="Beta", series_position=2.0, year=None, files=["b.m4b"]),
]
groups = match_plan_to_files(plan, [a, b])
assert [(g.title, g.series_position, g.year, g.files) for g in groups] == [
("Alpha", 1.0, 2001, [a]),
("Beta", 2.0, None, [b]),
]
def test_matches_glued_plan_path_by_basename_suffix(self, tmp_path: Path):
# The plan came from ABB's "<folder> <file>" strings; on disk the file sits in a folder.
root = tmp_path / "Cradle - Will Wight Books 1-10"
a = root / "Will Wight - Skysworn Cradle, Book 4" / "Skysworn Cradle, Book 4.m4b"
b = root / "Uncrowned Cradle, Book 7.m4b"
for f in (a, b):
f.parent.mkdir(parents=True, exist_ok=True)
f.write_bytes(b"x")
plan = [
PackBook(
title="Skysworn",
series_position=4.0,
year=None,
files=["Will Wight - Skysworn Cradle, Book 4 Skysworn Cradle, Book 4.m4b"],
),
PackBook(
title="Uncrowned",
series_position=7.0,
year=None,
files=["Uncrowned Cradle, Book 7.m4b"],
),
]
groups = match_plan_to_files(plan, [a, b])
assert [(g.title, g.files) for g in groups] == [("Skysworn", [a]), ("Uncrowned", [b])]
def test_unmatched_files_fall_back_to_heuristic_groups(self, tmp_path: Path):
a = tmp_path / "Book 1 - A" / "a.m4b"
c = tmp_path / "Book 3 - C" / "c.m4b"
for f in (a, c):
f.parent.mkdir(parents=True)
f.write_bytes(b"x")
plan = [PackBook(title="Alpha", series_position=1.0, year=None, files=["Book 1 - A/a.m4b"])]
groups = match_plan_to_files(plan, [a, c])
assert [(g.title, g.files) for g in groups] == [("Alpha", [a]), ("C", [c])]