mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-09-24 13:40:21 +01:00
Lance Marks
f441b85da2
## Multi-book packs: inspect a release before download and file each book separately Closes #576 ### Problem One queued release is always treated as one book. When a torrent is actually a whole series (`Series/Book 1 - Title/…`, or a flat folder of `Series 1.0 - Title.m4b` files), post-processing walks the whole tree, flattens every file into one list and renames them `Title - 01…10` under the searched book's `{Author}/{Title}`. Audiobookshelf then sees a single 10-file "book" and the user has to re-file everything by hand. ### What this does Most releases expose their file list *before* anything is downloaded, so the split is decided up front and approved by the user, then the download is fire-and-forget: 1. **Inspect** – clicking a release's download button now calls `POST /api/releases/inspect` first. A new optional `DownloadHandler.list_files(release_data)` hook returns the release's files without downloading: - **AudiobookBay** reads the torrent file table off the detail page it already fetches (the page is now cached for 120 s, so inspect + download cost ABB one request). - **Prowlarr** parses `info.files` from the `.torrent` it already fetches (the existing 120 s torrent-fetch cache is reused). Magnet-only and usenet releases report "can't inspect". - Other sources default to `None`. 2. **Review** – if the plan contains more than one book, the Find Releases modal swaps the list for a review panel: one row per book with editable title / series position / year, expandable file lists, non-book sidecars (`.txt`, covers) shown as ignored, a "Treat as a single book" switch, and **Download N books**. Single-book releases queue immediately, exactly as before. 3. **File** – the approved plan travels with the task (`DownloadTask.book_plan`, retry-safe) and post-processing files each book through the existing transfer code, one book at a time (`dataclasses.replace(task, title=…, series_position=…, year=…)`), so organize/rename templates, part numbering (now scoped per book), hardlinks, torrent copy-preserve and usenet handling are unchanged. Status reads `Complete (N books, M files)`. 4. **Fallback** – when a release can't be inspected the user gets a toast, and a small "Multi-book pack" toggle in the modal header forces a heuristic split (subfolder = book, or one book per file when the file names carry series positions). Planning lives in `shelfmark/download/postprocess/packs.py` and is shared by the inspect endpoint and post-processing, so what the user approved is what gets filed. The name parser strips `Book 3 -`, `03 -`, `1.0 -`, `3.`, `[03]`, `#3`, a leading series name, labels like "An Expanse Novella -", repeated titles (`Gods of Risk 2.5 - Gods of Risk`) and a trailing `(Year)`; author and series name come from the book that was searched, and the searched book's own series position is never applied to its siblings. ### Files - `shelfmark/download/postprocess/packs.py` (new) – `PackFile/PackBook/PackPlan`, `plan_pack`, `parse_pack_book_name`, `group_files_into_books`, `match_plan_to_files` - `shelfmark/core/release_inspect_routes.py` (new) – `POST /api/releases/inspect` - `shelfmark/release_sources/__init__.py` – `DownloadHandler.list_files` hook - `shelfmark/release_sources/audiobookbay/{scraper,handler}.py` – detail-page cache, `extract_file_list`, `list_files` - `shelfmark/release_sources/prowlarr/handler.py`, `download/clients/torrent_utils.py` – `extract_file_list_from_torrent`, `list_files` - `shelfmark/core/models.py`, `download/orchestrator.py` – `multi_book` / `book_plan` fields, queue + retry serialization - `shelfmark/download/postprocess/transfer.py`, `pipeline.py`, `outputs/folder.py` – per-book transfer branch and status message - `src/frontend`: `components/PackReviewPanel.tsx` (new), `ReleaseModal.tsx`, `App.tsx`, `services/api.ts`, `types/index.ts`, `utils/releasePayload.ts` (payload builder moved out of `App.tsx`), `utils/packReview.ts` - `docs/dev/release-sources-plugin-guide.md` – documents the `list_files` hook ### Out of scope (follow-ups) - Listing files from an NZB (Shelfmark already fetches the bytes; `<file subject>` names are noisy) - Inspecting magnet links via qBittorrent's files API after a paused add - BookLore / email outputs (they ignore `book_plan`; noted in code) - The combined ebook + audiobook flow ### Testing **Automated** (`make checks`, `make python-test`, `make frontend-test` all green; the only failures on my machine are the pre-existing `tests/config/test_entrypoint_permissions.py` cases, which need bash ≥ 4 and fail identically on `main` under macOS bash 3.2): - `tests/download/test_packs.py` – name parsing (markers, series name, novella labels, repeated titles, bare numeric titles like `1984`), nested / flat / mixed / deeper-nested packs, single wrapping folder not treated as a pack, plan-to-disk matching with basename fallback - `tests/core/test_processing_packs.py` – full `post_process_download` runs on a real temp filesystem: approved plan files each book under its own `{Author}/{Title}`, heuristic split of a nested pack, searched book's series position does not leak, multi-file book inside a pack keeps `- 01/- 02` per book, hardlinked torrent pack leaves the seeding tree intact, no pack fields ⇒ behaviour unchanged, single group degrades to the searched title, status message - `tests/core/test_release_inspect_routes.py` – plan response, not-inspectable, handler errors never 500, unknown source / missing `source_id` ⇒ 400, login required - `tests/audiobookbay/test_file_list.py` – file-table scraping from real ABB markup (multi-file and single-file pages), handler host validation, one page fetch shared by magnet + file list - `tests/prowlarr/test_torrent_file_list.py` – multi-file / single-file `.torrent` parsing, handler behaviour for torrent URL vs magnet vs usenet vs cache miss - `tests/download/test_orchestrator_pack_fields.py` – queue-time parsing and retry round-trip - Frontend: `releasePayload.test.ts`, `packReview.test.ts` (vitest) **Manual, on a real deployment** (arm64 image built from this branch, run as a side container next to production with the same qBittorrent / Audiobookshelf setup, `FILE_ORGANIZATION_AUDIOBOOK=organize`, hardlinks on): - AudiobookBay "The Expanse Complete 2.0" (7.87 GB, 36 files): clicking download opened the review panel in ~1 s showing **18 books · 18 files · 18 files ignored** (the `.txt` sidecars), with series positions 0.1–9.5 and years parsed from the file names; novella labels stripped ("The Churn", "The Butcher of Anderson Station"). Editing a title in the panel works. Confirming queued one task; the magnet resolved from the cached page in ~30 ms; after the download the task reported `Complete (18 books, 18 files)`, 18 hardlinks landed as `audiobooks/James S. A. Corey/<Title>/<Title>.m4b`, the torrent kept seeding, and Audiobookshelf scanned each folder as its own book (title, author, embedded chapters). - A second pack ("Expanse [01 - 9.5]", `Title N - Title` naming) was inspected to verify the repeated-title rule and the Back button, without downloading. - Single-book releases still queue immediately with no extra UI.
103 lines
4.3 KiB
Python
103 lines
4.3 KiB
Python
"""Tests for reading the torrent file list off an AudiobookBay detail page."""
|
||
|
||
from shelfmark.download.postprocess.packs import PackFile
|
||
from shelfmark.release_sources.audiobookbay import scraper
|
||
|
||
# Trimmed from a real detail page (2026-08): the file rows sit between the
|
||
# "Multifile Torrent" marker and the "Combined File Size" row.
|
||
MULTIFILE_DETAIL_HTML = """
|
||
<table>
|
||
<tr><td>Tracker:</td><td>udp://tracker.torrent.eu.org:451/announce</td></tr>
|
||
<tr><td>Creation Date:</td><td>Sun, 29 Mar 2026 21:09:39 +0200</td></tr>
|
||
<tr><td colspan='2'>This is a Multifile Torrent</td></tr>
|
||
<tr><td colspan='2'>The Expanse 9.0 - Leviathan Falls (2021).m4b 1.05 GBs</td></tr>
|
||
<tr><td colspan='2'>The Expanse 0.1 - An Expanse Novella - Drive (2012).txt 340 Bytes</td></tr>
|
||
<tr><td colspan='2'>The Expanse 0.2 - An Expanse Novella - The Churn (2014).m4b 125.72 MBs</td></tr>
|
||
<tr><td colspan='2'>The Expanse 2.0 - Caliban’s War (2012).m4b 578.97 MBs</td></tr>
|
||
<tr><td>Combined File Size:</td><td><span style='color:#00f;'>7.87</span> GBs</td></tr>
|
||
<tr><td>Info Hash:</td><td>e4a5538e26987ee58a43aa629ec2c4f2b2d46526</td></tr>
|
||
</table>
|
||
"""
|
||
|
||
SINGLE_FILE_DETAIL_HTML = """
|
||
<table>
|
||
<tr><td>Creation Date:</td><td>Sun, 29 Mar 2026 21:09:39 +0200</td></tr>
|
||
<tr><td colspan='2'>Drive.m4b 41.55 MBs</td></tr>
|
||
<tr><td>Combined File Size:</td><td><span style='color:#00f;'>41.55</span> MBs</td></tr>
|
||
<tr><td>Info Hash:</td><td>e4a5538e26987ee58a43aa629ec2c4f2b2d46526</td></tr>
|
||
</table>
|
||
"""
|
||
|
||
|
||
def test_extracts_multifile_rows_with_byte_sizes():
|
||
files = scraper.extract_file_list(MULTIFILE_DETAIL_HTML)
|
||
assert files == [
|
||
PackFile("The Expanse 9.0 - Leviathan Falls (2021).m4b", int(1.05 * 1024**3)),
|
||
PackFile("The Expanse 0.1 - An Expanse Novella - Drive (2012).txt", 340),
|
||
PackFile(
|
||
"The Expanse 0.2 - An Expanse Novella - The Churn (2014).m4b", int(125.72 * 1024**2)
|
||
),
|
||
PackFile("The Expanse 2.0 - Caliban’s War (2012).m4b", int(578.97 * 1024**2)),
|
||
]
|
||
|
||
|
||
def test_single_file_torrent_lists_the_row_before_combined_size():
|
||
assert scraper.extract_file_list(SINGLE_FILE_DETAIL_HTML) == [
|
||
PackFile("Drive.m4b", int(41.55 * 1024**2))
|
||
]
|
||
|
||
|
||
def test_page_without_file_table_returns_none():
|
||
assert scraper.extract_file_list("<html><body><p>nothing here</p></body></html>") is None
|
||
|
||
|
||
class TestHandlerListFiles:
|
||
def test_lists_files_from_detail_page(self):
|
||
from unittest.mock import patch
|
||
|
||
from shelfmark.release_sources.audiobookbay.handler import AudiobookBayHandler
|
||
|
||
with patch(
|
||
"shelfmark.release_sources.audiobookbay.handler.scraper.fetch_detail_html",
|
||
return_value=SINGLE_FILE_DETAIL_HTML,
|
||
) as fetch:
|
||
files = AudiobookBayHandler().list_files(
|
||
{"source_id": "abc", "download_url": "https://audiobookbay.lu/abss/drive/"}
|
||
)
|
||
assert files == [PackFile("Drive.m4b", int(41.55 * 1024**2))]
|
||
fetch.assert_called_once_with("https://audiobookbay.lu/abss/drive/", "audiobookbay.lu")
|
||
|
||
def test_rejects_detail_url_on_other_host(self):
|
||
from unittest.mock import patch
|
||
|
||
from shelfmark.release_sources.audiobookbay.handler import AudiobookBayHandler
|
||
|
||
with patch(
|
||
"shelfmark.release_sources.audiobookbay.handler.scraper.fetch_detail_html"
|
||
) as fetch:
|
||
files = AudiobookBayHandler().list_files(
|
||
{"source_id": "abc", "download_url": "https://evil.example/abss/drive/"}
|
||
)
|
||
assert files is None
|
||
fetch.assert_not_called()
|
||
|
||
|
||
def test_extract_magnet_link_and_file_list_share_one_page_fetch():
|
||
from unittest.mock import patch
|
||
|
||
page = MULTIFILE_DETAIL_HTML
|
||
with (
|
||
patch(
|
||
"shelfmark.release_sources.audiobookbay.scraper.downloader.html_get_page",
|
||
return_value=page,
|
||
) as get_page,
|
||
patch("shelfmark.release_sources.audiobookbay.scraper._bootstrap_abb_session"),
|
||
):
|
||
scraper.clear_detail_page_cache()
|
||
url = "https://audiobookbay.lu/abss/expanse/"
|
||
assert scraper.fetch_detail_html(url, "audiobookbay.lu") == page
|
||
magnet = scraper.extract_magnet_link(url, "audiobookbay.lu")
|
||
assert magnet is not None
|
||
assert "e4a5538e26987ee58a43aa629ec2c4f2b2d46526".upper() in magnet
|
||
assert get_page.call_count == 1
|