From f441b85da2619b63ed4387fc9f4cd382fff342e3 Mon Sep 17 00:00:00 2001 From: Lance Marks <89031362+lancemarks@users.noreply.github.com> Date: Thu, 27 Aug 2026 00:40:08 -0400 Subject: [PATCH] feat(packs): inspect multi-book releases and file each book separately (#1270) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Multi-book packs: inspect a release before download and file each book separately Closes #576 ### Problem One queued release is always treated as one book. When a torrent is actually a whole series (`Series/Book 1 - Title/…`, or a flat folder of `Series 1.0 - Title.m4b` files), post-processing walks the whole tree, flattens every file into one list and renames them `Title - 01…10` under the searched book's `{Author}/{Title}`. Audiobookshelf then sees a single 10-file "book" and the user has to re-file everything by hand. ### What this does Most releases expose their file list *before* anything is downloaded, so the split is decided up front and approved by the user, then the download is fire-and-forget: 1. **Inspect** – clicking a release's download button now calls `POST /api/releases/inspect` first. A new optional `DownloadHandler.list_files(release_data)` hook returns the release's files without downloading: - **AudiobookBay** reads the torrent file table off the detail page it already fetches (the page is now cached for 120 s, so inspect + download cost ABB one request). - **Prowlarr** parses `info.files` from the `.torrent` it already fetches (the existing 120 s torrent-fetch cache is reused). Magnet-only and usenet releases report "can't inspect". - Other sources default to `None`. 2. **Review** – if the plan contains more than one book, the Find Releases modal swaps the list for a review panel: one row per book with editable title / series position / year, expandable file lists, non-book sidecars (`.txt`, covers) shown as ignored, a "Treat as a single book" switch, and **Download N books**. Single-book releases queue immediately, exactly as before. 3. **File** – the approved plan travels with the task (`DownloadTask.book_plan`, retry-safe) and post-processing files each book through the existing transfer code, one book at a time (`dataclasses.replace(task, title=…, series_position=…, year=…)`), so organize/rename templates, part numbering (now scoped per book), hardlinks, torrent copy-preserve and usenet handling are unchanged. Status reads `Complete (N books, M files)`. 4. **Fallback** – when a release can't be inspected the user gets a toast, and a small "Multi-book pack" toggle in the modal header forces a heuristic split (subfolder = book, or one book per file when the file names carry series positions). Planning lives in `shelfmark/download/postprocess/packs.py` and is shared by the inspect endpoint and post-processing, so what the user approved is what gets filed. The name parser strips `Book 3 -`, `03 -`, `1.0 -`, `3.`, `[03]`, `#3`, a leading series name, labels like "An Expanse Novella -", repeated titles (`Gods of Risk 2.5 - Gods of Risk`) and a trailing `(Year)`; author and series name come from the book that was searched, and the searched book's own series position is never applied to its siblings. ### Files - `shelfmark/download/postprocess/packs.py` (new) – `PackFile/PackBook/PackPlan`, `plan_pack`, `parse_pack_book_name`, `group_files_into_books`, `match_plan_to_files` - `shelfmark/core/release_inspect_routes.py` (new) – `POST /api/releases/inspect` - `shelfmark/release_sources/__init__.py` – `DownloadHandler.list_files` hook - `shelfmark/release_sources/audiobookbay/{scraper,handler}.py` – detail-page cache, `extract_file_list`, `list_files` - `shelfmark/release_sources/prowlarr/handler.py`, `download/clients/torrent_utils.py` – `extract_file_list_from_torrent`, `list_files` - `shelfmark/core/models.py`, `download/orchestrator.py` – `multi_book` / `book_plan` fields, queue + retry serialization - `shelfmark/download/postprocess/transfer.py`, `pipeline.py`, `outputs/folder.py` – per-book transfer branch and status message - `src/frontend`: `components/PackReviewPanel.tsx` (new), `ReleaseModal.tsx`, `App.tsx`, `services/api.ts`, `types/index.ts`, `utils/releasePayload.ts` (payload builder moved out of `App.tsx`), `utils/packReview.ts` - `docs/dev/release-sources-plugin-guide.md` – documents the `list_files` hook ### Out of scope (follow-ups) - Listing files from an NZB (Shelfmark already fetches the bytes; `` names are noisy) - Inspecting magnet links via qBittorrent's files API after a paused add - BookLore / email outputs (they ignore `book_plan`; noted in code) - The combined ebook + audiobook flow ### Testing **Automated** (`make checks`, `make python-test`, `make frontend-test` all green; the only failures on my machine are the pre-existing `tests/config/test_entrypoint_permissions.py` cases, which need bash ≥ 4 and fail identically on `main` under macOS bash 3.2): - `tests/download/test_packs.py` – name parsing (markers, series name, novella labels, repeated titles, bare numeric titles like `1984`), nested / flat / mixed / deeper-nested packs, single wrapping folder not treated as a pack, plan-to-disk matching with basename fallback - `tests/core/test_processing_packs.py` – full `post_process_download` runs on a real temp filesystem: approved plan files each book under its own `{Author}/{Title}`, heuristic split of a nested pack, searched book's series position does not leak, multi-file book inside a pack keeps `- 01/- 02` per book, hardlinked torrent pack leaves the seeding tree intact, no pack fields ⇒ behaviour unchanged, single group degrades to the searched title, status message - `tests/core/test_release_inspect_routes.py` – plan response, not-inspectable, handler errors never 500, unknown source / missing `source_id` ⇒ 400, login required - `tests/audiobookbay/test_file_list.py` – file-table scraping from real ABB markup (multi-file and single-file pages), handler host validation, one page fetch shared by magnet + file list - `tests/prowlarr/test_torrent_file_list.py` – multi-file / single-file `.torrent` parsing, handler behaviour for torrent URL vs magnet vs usenet vs cache miss - `tests/download/test_orchestrator_pack_fields.py` – queue-time parsing and retry round-trip - Frontend: `releasePayload.test.ts`, `packReview.test.ts` (vitest) **Manual, on a real deployment** (arm64 image built from this branch, run as a side container next to production with the same qBittorrent / Audiobookshelf setup, `FILE_ORGANIZATION_AUDIOBOOK=organize`, hardlinks on): - AudiobookBay "The Expanse Complete 2.0" (7.87 GB, 36 files): clicking download opened the review panel in ~1 s showing **18 books · 18 files · 18 files ignored** (the `.txt` sidecars), with series positions 0.1–9.5 and years parsed from the file names; novella labels stripped ("The Churn", "The Butcher of Anderson Station"). Editing a title in the panel works. Confirming queued one task; the magnet resolved from the cached page in ~30 ms; after the download the task reported `Complete (18 books, 18 files)`, 18 hardlinks landed as `audiobooks/James S. A. Corey//<Title>.m4b`, the torrent kept seeding, and Audiobookshelf scanned each folder as its own book (title, author, embedded chapters). - A second pack ("Expanse [01 - 9.5]", `Title N - Title` naming) was inspected to verify the repeated-title rule and the Back button, without downloading. - Single-book releases still queue immediately with no extra UI. --- .gitignore | 1 + docs/dev/release-sources-plugin-guide.md | 21 + shelfmark/core/models.py | 6 + shelfmark/core/release_inspect_routes.py | 102 +++++ shelfmark/download/clients/torrent_utils.py | 51 +++ shelfmark/download/orchestrator.py | 35 ++ shelfmark/download/outputs/folder.py | 11 +- shelfmark/download/postprocess/packs.py | 406 ++++++++++++++++++ shelfmark/download/postprocess/pipeline.py | 2 + shelfmark/download/postprocess/transfer.py | 110 +++++ shelfmark/main.py | 4 + shelfmark/release_sources/__init__.py | 9 + .../release_sources/audiobookbay/handler.py | 16 +- .../release_sources/audiobookbay/scraper.py | 136 ++++-- shelfmark/release_sources/prowlarr/handler.py | 24 ++ src/frontend/src/App.tsx | 47 +- .../src/components/PackReviewPanel.tsx | 193 +++++++++ src/frontend/src/components/ReleaseModal.tsx | 114 ++++- src/frontend/src/services/api.ts | 14 + src/frontend/src/tests/packReview.test.ts | 53 +++ src/frontend/src/tests/releasePayload.test.ts | 69 +++ src/frontend/src/types/index.ts | 21 + src/frontend/src/utils/packReview.ts | 39 ++ src/frontend/src/utils/releasePayload.ts | 55 +++ tests/audiobookbay/conftest.py | 13 + tests/audiobookbay/test_file_list.py | 102 +++++ tests/core/test_processing_packs.py | 212 +++++++++ tests/core/test_release_inspect_routes.py | 134 ++++++ .../download/test_orchestrator_pack_fields.py | 62 +++ tests/download/test_packs.py | 282 ++++++++++++ tests/prowlarr/test_torrent_file_list.py | 98 +++++ 31 files changed, 2370 insertions(+), 72 deletions(-) create mode 100644 shelfmark/core/release_inspect_routes.py create mode 100644 shelfmark/download/postprocess/packs.py create mode 100644 src/frontend/src/components/PackReviewPanel.tsx create mode 100644 src/frontend/src/tests/packReview.test.ts create mode 100644 src/frontend/src/tests/releasePayload.test.ts create mode 100644 src/frontend/src/utils/packReview.ts create mode 100644 src/frontend/src/utils/releasePayload.ts create mode 100644 tests/audiobookbay/conftest.py create mode 100644 tests/audiobookbay/test_file_list.py create mode 100644 tests/core/test_processing_packs.py create mode 100644 tests/core/test_release_inspect_routes.py create mode 100644 tests/download/test_orchestrator_pack_fields.py create mode 100644 tests/download/test_packs.py create mode 100644 tests/prowlarr/test_torrent_file_list.py diff --git a/.gitignore b/.gitignore index 6c60a2fe..892e797c 100644 --- a/.gitignore +++ b/.gitignore @@ -236,6 +236,7 @@ pyrightconfig.json *.local.* AGENTS.md .claude/ +CLAUDE.md .nvmrc .playwright-mcp/ frontend-dist/ diff --git a/docs/dev/release-sources-plugin-guide.md b/docs/dev/release-sources-plugin-guide.md index 10039aa3..37709201 100644 --- a/docs/dev/release-sources-plugin-guide.md +++ b/docs/dev/release-sources-plugin-guide.md @@ -276,6 +276,27 @@ class DownloadHandler(ABC): pass ``` +### Optional: Listing Files Before Download + +Some releases bundle several books (a whole-series torrent). Shelfmark inspects a +release before queueing it so the user can review how it will be split into books. +Override `list_files` when your source can enumerate a release's files without +downloading it; the default returns `None`, which the UI reports as "can't inspect": + +```python +from shelfmark.download.postprocess.packs import PackFile + +def list_files(self, release_data: dict[str, Any]) -> list[PackFile] | None: + """Return the release's files (release-relative paths + sizes), or None.""" + torrent_bytes = ... # e.g. fetch the .torrent, or scrape the indexer's detail page + return extract_file_list_from_torrent(torrent_bytes) # from download.clients.torrent_utils +``` + +`release_data` is the same payload the frontend sends to `/api/releases/download` +(`source_id`, `download_url`, `content_type`, `series_name`, ...). Built-in examples: +Prowlarr parses the `.torrent` it already fetches (magnet-only releases return +`None`), and AudiobookBay reads the file table off its detail page. + ### Download Method Parameters | Parameter | Type | Description | diff --git a/shelfmark/core/models.py b/shelfmark/core/models.py index d27ebd64..20fd3374 100644 --- a/shelfmark/core/models.py +++ b/shelfmark/core/models.py @@ -136,6 +136,12 @@ class DownloadTask: default_factory=dict ) # Per-output parameters (e.g. email recipient) + # Multi-book packs: one release holding several books. `book_plan` is the split the + # user approved before download (list of {title, series_position, year, files}); + # `multi_book` asks post-processing to split heuristically when no plan exists. + multi_book: bool = False + book_plan: list[dict[str, Any]] | None = None + # User association (multi-user support) user_id: int | None = None # DB user ID who queued this download username: str | None = None # Username for {User} template variable diff --git a/shelfmark/core/release_inspect_routes.py b/shelfmark/core/release_inspect_routes.py new file mode 100644 index 00000000..f9381628 --- /dev/null +++ b/shelfmark/core/release_inspect_routes.py @@ -0,0 +1,102 @@ +"""Pre-download release inspection: list a release's files and plan a multi-book split.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Any + +from flask import jsonify, request + +from shelfmark.core.logger import setup_logger +from shelfmark.core.utils import is_audiobook +from shelfmark.download.postprocess.packs import PackFile, PackPlan, plan_pack +from shelfmark.download.postprocess.policy import ( + get_supported_audiobook_formats, + get_supported_formats, +) +from shelfmark.release_sources import get_handler + +if TYPE_CHECKING: + from collections.abc import Callable + + from flask import Flask, Response + +logger = setup_logger(__name__) + +_INSPECT_ERRORS = (OSError, RuntimeError, ValueError, TypeError, KeyError, AttributeError) +NOT_INSPECTABLE_REASON = "This source cannot list the release's files before downloading" + + +def _serialize_plan(plan: PackPlan) -> dict[str, Any]: + return { + "is_pack": plan.is_pack, + "ignored": plan.ignored, + "books": [ + { + "title": book.title, + "series_position": book.series_position, + "year": book.year, + "files": book.files, + } + for book in plan.books + ], + } + + +def inspect_release(data: dict[str, Any]) -> dict[str, Any]: + """Build the inspect response for a release payload (same shape as a download).""" + source = str(data["source"]) + handler = get_handler(source) + try: + files: list[PackFile] | None = handler.list_files(data) + except _INSPECT_ERRORS as exc: + logger.warning( + "Could not list files for %s release %s: %s", source, data.get("source_id"), exc + ) + return {"inspected": False, "reason": str(exc), "files": [], "plan": None} + + if files is None: + return {"inspected": False, "reason": NOT_INSPECTABLE_REASON, "files": [], "plan": None} + + content_type = data.get("content_type") + supported = ( + get_supported_audiobook_formats() + if is_audiobook(content_type if isinstance(content_type, str) else None) + else get_supported_formats() + ) + series_name = data.get("series_name") + author_name = data.get("author") + plan = plan_pack( + files, + supported_extensions=set(supported), + series_name=series_name if isinstance(series_name, str) else None, + author_name=author_name if isinstance(author_name, str) else None, + ) + return { + "inspected": True, + "reason": None, + "files": [{"path": f.path, "size": f.size} for f in files], + "plan": _serialize_plan(plan), + } + + +def register_release_inspect_routes( + app: Flask, + login_required: Callable[..., Any], +) -> None: + """Register POST /api/releases/inspect.""" + + @app.route("/api/releases/inspect", methods=["POST"]) + @login_required + def api_inspect_release() -> Response | tuple[Response, int]: + data = request.get_json(silent=True) + if not isinstance(data, dict): + return jsonify({"error": "No data provided"}), 400 + if not data.get("source_id"): + return jsonify({"error": "source_id is required"}), 400 + if not data.get("source"): + return jsonify({"error": "source is required"}), 400 + try: + get_handler(str(data["source"])) + except ValueError as exc: + return jsonify({"error": str(exc)}), 400 + return jsonify(inspect_release(data)) diff --git a/shelfmark/download/clients/torrent_utils.py b/shelfmark/download/clients/torrent_utils.py index fd810540..526d1130 100644 --- a/shelfmark/download/clients/torrent_utils.py +++ b/shelfmark/download/clients/torrent_utils.py @@ -17,6 +17,7 @@ from shelfmark.core.config import config from shelfmark.core.logger import setup_logger from shelfmark.core.utils import normalize_http_url from shelfmark.download.network import get_ssl_verify +from shelfmark.download.postprocess.packs import PackFile logger = setup_logger(__name__) @@ -433,6 +434,56 @@ def extract_info_hash_from_torrent(torrent_data: bytes) -> str | None: return None +def _decode_torrent_text(value: object) -> str | None: + if isinstance(value, bytes): + return value.decode("utf-8", errors="replace") + if isinstance(value, str): + return value + return None + + +def extract_file_list_from_torrent(torrent_data: bytes) -> list[PackFile] | None: + """List the files a .torrent describes, release-relative, without downloading it. + + Multi-file torrents nest every path under the torrent name (which becomes the + client's save folder); single-file torrents are just the named file. + """ + try: + decoded, _ = bencode_decode(torrent_data) + except _TORRENT_PARSE_ERRORS as e: + logger.debug("Failed to parse torrent file list: %s", e) + return None + if not isinstance(decoded, dict): + return None + info = decoded.get(b"info") + if not isinstance(info, dict): + return None + + name = _decode_torrent_text(info.get(b"name")) or "" + raw_files = info.get(b"files") + if not isinstance(raw_files, list): + length = info.get(b"length") + if not name: + return None + return [PackFile(name, length if isinstance(length, int) else None)] + + files: list[PackFile] = [] + for entry in raw_files: + if not isinstance(entry, dict): + continue + raw_path = entry.get(b"path") + if not isinstance(raw_path, list): + continue + segments = [seg for seg in (_decode_torrent_text(part) for part in raw_path) if seg] + if not segments: + continue + if name: + segments.insert(0, name) + length = entry.get(b"length") + files.append(PackFile("/".join(segments), length if isinstance(length, int) else None)) + return files + + def extract_hash_from_magnet(magnet_url: str) -> str | None: """Extract info_hash from a magnet URL.""" if not magnet_url.startswith("magnet:"): diff --git a/shelfmark/download/orchestrator.py b/shelfmark/download/orchestrator.py index bd4b38f2..cfc03b94 100644 --- a/shelfmark/download/orchestrator.py +++ b/shelfmark/download/orchestrator.py @@ -265,6 +265,8 @@ def queue_release( series_position = release_data.get("series_position") or extra.get("series_position") subtitle = release_data.get("subtitle") or extra.get("subtitle") language = release_data.get("language") or extra.get("language") + multi_book = bool(release_data.get("multi_book") or extra.get("multi_book")) + book_plan = _normalize_book_plan(release_data.get("book_plan") or extra.get("book_plan")) books_output_mode = ( str(config.get("BOOKS_OUTPUT_MODE", "folder", user_id=user_id) or "folder") @@ -300,6 +302,8 @@ def queue_release( series_position=series_position, subtitle=subtitle, language=language, + multi_book=multi_book or book_plan is not None, + book_plan=book_plan, search_mode=search_mode, output_mode=output_mode, output_args=output_args, @@ -408,6 +412,33 @@ def can_retry_download_task( return _has_staged_retry_source(task) +def _normalize_book_plan(value: object) -> list[dict[str, Any]] | None: + """Keep only well-formed pack books: a title plus a non-empty list of file paths.""" + if not isinstance(value, list): + return None + books: list[dict[str, Any]] = [] + for entry in value: + if not isinstance(entry, dict): + continue + title = normalize_optional_text(entry.get("title")) + raw_files = entry.get("files") + if title is None or not isinstance(raw_files, list): + continue + files = [f for f in raw_files if isinstance(f, str) and f.strip()] + if not files: + continue + year = entry.get("year") + books.append( + { + "title": title, + "series_position": _optional_number(entry.get("series_position")), + "year": year if isinstance(year, int) and not isinstance(year, bool) else None, + "files": files, + } + ) + return books or None + + def serialize_task_for_retry(task: DownloadTask) -> dict[str, Any]: """Serialize the task state needed for restart-safe retries.""" raw_search_mode = getattr(task, "search_mode", None) @@ -437,6 +468,8 @@ def serialize_task_for_retry(task: DownloadTask) -> dict[str, Any]: "subtitle": getattr(task, "subtitle", None), "language": getattr(task, "language", None), "search_mode": search_mode, + "multi_book": bool(getattr(task, "multi_book", False)), + "book_plan": _normalize_book_plan(getattr(task, "book_plan", None)), "output_mode": getattr(task, "output_mode", None), "output_args": dict(raw_output_args) if isinstance(raw_output_args, dict) else {}, "user_id": getattr(task, "user_id", None), @@ -495,6 +528,8 @@ def _restore_task_from_retry_payload(payload: object) -> DownloadTask | None: subtitle=normalize_optional_text(payload.get("subtitle")), language=normalize_optional_text(payload.get("language")), search_mode=search_mode, + multi_book=bool(payload.get("multi_book", False)), + book_plan=_normalize_book_plan(payload.get("book_plan")), output_mode=normalize_optional_text(payload.get("output_mode")), output_args=dict(output_args) if isinstance(output_args, dict) else {}, user_id=normalize_positive_int(payload.get("user_id")), diff --git a/shelfmark/download/outputs/folder.py b/shelfmark/download/outputs/folder.py index 7e3278f2..3f549d35 100644 --- a/shelfmark/download/outputs/folder.py +++ b/shelfmark/download/outputs/folder.py @@ -105,6 +105,7 @@ def process_folder_output( maybe_run_custom_script, prepare_output_files, record_step, + resolve_book_groups, transfer_book_files, ) @@ -260,7 +261,15 @@ def process_folder_output( prepared.cleanup_paths, ) - message = "Complete" if len(final_paths) == 1 else f"Complete ({len(final_paths)} files)" + pack_groups = resolve_book_groups( + task, prepared.files, organization_mode=plan.organization_mode + ) + if pack_groups is not None: + message = f"Complete ({len(pack_groups)} books, {len(final_paths)} files)" + elif len(final_paths) == 1: + message = "Complete" + else: + message = f"Complete ({len(final_paths)} files)" status_callback("complete", message) return str(final_paths[0]) diff --git a/shelfmark/download/postprocess/packs.py b/shelfmark/download/postprocess/packs.py new file mode 100644 index 00000000..ee13dc25 --- /dev/null +++ b/shelfmark/download/postprocess/packs.py @@ -0,0 +1,406 @@ +"""Multi-book ("pack") release planning. + +A pack is one release that contains several books: a whole-series torrent with one +subfolder per book, or a flat folder of `Series 1.0 - Title.m4b` files. The same +planning rules serve pre-download inspection (the file list comes from the release +source) and post-processing (the file list comes from disk), so what the user +approved in the modal is what gets filed. +""" + +from __future__ import annotations + +import os +import re +from dataclasses import dataclass +from pathlib import Path, PurePosixPath + +_YEAR_SUFFIX_RE = re.compile(r"\s*\(\s*(?P<year>\d{4})\s*\)\s*$") +_SERIES_MARKER_RE = re.compile( + r""" + ^\s* + (?: + \[\s*\#?(?P<bracket>\d+(?:\.\d+)?)\s*\] # [03] / [#3] + | \#(?P<hash>\d+(?:\.\d+)?) # #3 + | book\.?\s*(?P<book>\d+(?:\.\d+)?) # Book 3 / Book. 03 + | (?P<plain>\d+(?:\.\d+)?)(?=[\s\-:.]) # 03 - / 1.0 - / 3. + ) + \s*(?:[-:.]\s*)? + """, + re.IGNORECASE | re.VERBOSE, +) +_SEPARATOR_CHARS = " \t-_:." +# "Gods of Risk 2.5 - Gods of Risk": the title repeated on both sides of the position. +_REPEATED_TITLE_RE = re.compile( + r"^(?P<left>.+?)\s+(?P<position>\d+(?:\.\d+)?)\s*[-:\u2013]\s*(?P<right>.+)$" +) +_SERIES_LABEL_WORDS = r"(?:novella|novellas|short\s+story|short|story|novel)" +# "Uncrowned Cradle, Book 7" / "Reaper Cradle, Volume 10" / "Wintersteel (Cradle, Book 8)": +# an explicit word marks the position at the END of the name. A bare trailing number +# is deliberately not matched — "Title - 02" is a chapter, not a series position. +_TRAILING_MARKER_RE = re.compile( + r""" + [\s,\-:\u2013(]* + (?:book|volume|vol\.?)\s*\#?(?P<position>\d+(?:\.\d+)?) + \s*\)?\s*$ + """, + re.IGNORECASE | re.VERBOSE, +) +# AudiobookBay renders a file inside a folder as "<folder> <file>" with no separator, +# so a pack row reads "Author - Title Series, Book 1 Title Series, Book 1". +_GLUED_FOLDER_RE = re.compile( + r"^(?P<prefix>.+?\s[-\u2013]\s)?(?P<core>.+?)\s+(?P=core)$", re.IGNORECASE +) + + +@dataclass(frozen=True) +class PackFile: + """One file inside a release, path relative to the release root.""" + + path: str + size: int | None = None + + +@dataclass(frozen=True) +class PackBook: + """One book split out of a pack, files as release-relative paths.""" + + title: str + series_position: float | None + year: int | None + files: list[str] + + +@dataclass(frozen=True) +class PackPlan: + books: list[PackBook] + ignored: list[str] + + @property + def is_pack(self) -> bool: + return len(self.books) > 1 + + +@dataclass(frozen=True) +class BookGroup: + """One book's on-disk files, ready for transfer.""" + + title: str + series_position: float | None + year: int | None + files: list[Path] + + +def _strip_series_name(name: str, series_name: str | None) -> str: + if not series_name: + return name + prefix = series_name.strip() + if not prefix or not name.lower().startswith(prefix.lower()): + return name + remainder = name[len(prefix) :] + if remainder and remainder[0].isalnum(): + return name + return remainder.lstrip(_SEPARATOR_CHARS) + + +def _strip_series_label(work: str, series_name: str | None) -> str: + """Drop a leading "An <Series> Novella - " style label that some packs prepend.""" + if not series_name: + return work + # "The Expanse" is labelled "An Expanse Novella", so match without the article. + core = re.sub(r"^(?:the|an?)\s+", "", series_name.strip(), flags=re.IGNORECASE) + if not core: + return work + pattern = re.compile( + rf"^(?:an?\s+|the\s+)?{re.escape(core)}\s+{_SERIES_LABEL_WORDS}\s*[-:\u2013]\s*", + re.IGNORECASE, + ) + return pattern.sub("", work, count=1) + + +def _collapse_glued_folder(name: str) -> str: + match = _GLUED_FOLDER_RE.match(name) + if not match: + return name + prefix = match.group("prefix") or "" + core = match.group("core") + # "Author - X X" → "Author - X" (the folder carried the author, the file did not). + return (prefix + core).strip() + + +def _strip_author_name(name: str, author_name: str | None) -> str: + """Drop a leading "Author - " (packs are often filed as `Author - Title`).""" + if not author_name: + return name + prefix = author_name.strip() + if not prefix or not name.lower().startswith(prefix.lower()): + return name + remainder = name[len(prefix) :] + stripped = remainder.lstrip(_SEPARATOR_CHARS + "\u2013") + if stripped == remainder: # no separator after the author: part of the title + return name + return stripped + + +def _strip_trailing_series_name(work: str, series_name: str | None) -> str: + """Drop a trailing series name left behind by a trailing position marker.""" + if not series_name: + return work + suffix = series_name.strip() + if not suffix or not work.lower().endswith(suffix.lower()): + return work + remainder = work[: -len(suffix)] + stripped = remainder.rstrip(_SEPARATOR_CHARS + ",(\u2013") + if not stripped or stripped == remainder: + return work + return stripped + + +def parse_pack_book_name( + name: str, *, series_name: str | None, author_name: str | None = None +) -> tuple[str, float | None, int | None]: + """Split a book folder/file-stem name into (title, series position, year). + + Strips a leading series name, a leading position marker (`Book 3 - `, `03 - `, + `1.0 - `, `3. `, `[03] `, `#3 `) and a trailing `(YYYY)`. Also understands a + trailing marker (`Title Series, Book 3`, `Title (Series, Volume 3)`), a leading + `Author - `, and AudiobookBay's glued `<folder> <file>` names. Returns the name + unchanged with no position/year when nothing would be left of the title. + """ + work = _collapse_glued_folder(name.strip()) + work = _strip_author_name(work, author_name) + work = _strip_series_name(work, series_name) + + year: int | None = None + year_match = _YEAR_SUFFIX_RE.search(work) + if year_match: + year = int(year_match.group("year")) + work = work[: year_match.start()] + + position: float | None = None + repeated = _REPEATED_TITLE_RE.match(work.strip()) + if ( + repeated + and repeated.group("left").strip().lower() == repeated.group("right").strip().lower() + ): + return repeated.group("right").strip(), float(repeated.group("position")), year + + marker = _SERIES_MARKER_RE.match(work) + if marker: + raw = ( + marker.group("bracket") + or marker.group("hash") + or marker.group("book") + or marker.group("plain") + ) + position = float(raw) + work = work[marker.end() :] + else: + trailing = _TRAILING_MARKER_RE.search(work) + if trailing and trailing.start() > 0: + position = float(trailing.group("position")) + work = _strip_trailing_series_name(work[: trailing.start()], series_name) + + work = _strip_series_label(work, series_name) + title = work.strip().strip(_SEPARATOR_CHARS).strip() + if not title: + return name, None, None + return title, position, year + + +def _book_from_name( + name: str, files: list[str], series_name: str | None, author_name: str | None = None +) -> PackBook: + title, position, year = parse_pack_book_name( + name, series_name=series_name, author_name=author_name + ) + return PackBook(title=title, series_position=position, year=year, files=files) + + +def _common_root_parts(paths: list[PurePosixPath]) -> tuple[str, ...]: + parents = [p.parent.parts for p in paths] + common: list[str] = [] + for parts in zip(*parents, strict=False): + if len(set(parts)) != 1: + break + common.append(parts[0]) + return tuple(common) + + +def plan_pack( + files: list[PackFile], + *, + supported_extensions: set[str], + series_name: str | None, + author_name: str | None = None, + root_depth: int | None = None, +) -> PackPlan: + """Group a release's file list into books. + + Files in a subfolder (relative to the common root) group by that subfolder. Files + directly in the root split one-book-per-file only when at least two of them carry + a series position in their names; otherwise they are one book (a chaptered + audiobook, e.g. `01.mp3`, `02.mp3`). `root_depth` fixes how many leading path + components form the root instead of deriving it from the files' common parent. + """ + supported = {ext.lower().lstrip(".") for ext in supported_extensions} + book_files: list[PurePosixPath] = [] + ignored: list[str] = [] + for pack_file in files: + rel = PurePosixPath(pack_file.path.replace("\\", "/").lstrip("./")) + if rel.suffix.lower().lstrip(".") in supported: + book_files.append(rel) + else: + ignored.append(pack_file.path) + + if not book_files: + return PackPlan(books=[], ignored=ignored) + + root_parts = ( + _common_root_parts(book_files) if root_depth is None else book_files[0].parts[:root_depth] + ) + depth = len(root_parts) + + root_files: list[PurePosixPath] = [] + folders: dict[str, list[str]] = {} + for rel in book_files: + remainder = rel.parts[depth:] + if len(remainder) > 1: + folders.setdefault(remainder[0], []).append(str(rel)) + else: + root_files.append(rel) + + books: list[PackBook] = [] + if root_files: + parsed = [ + parse_pack_book_name(f.stem, series_name=series_name, author_name=author_name) + for f in root_files + ] + positions = {p[1] for p in parsed if p[1] is not None} + if len(positions) >= 2: + books.extend( + PackBook(title=title, series_position=position, year=year, files=[str(f)]) + for f, (title, position, year) in zip(root_files, parsed, strict=True) + ) + elif len(root_files) == 1: + books.append( + _book_from_name(root_files[0].stem, [str(root_files[0])], series_name, author_name) + ) + else: + group_name = root_parts[-1] if root_parts else "" + books.append( + _book_from_name(group_name, [str(f) for f in root_files], series_name, author_name) + ) + + books.extend( + _book_from_name(folder, paths, series_name, author_name) + for folder, paths in folders.items() + ) + return PackPlan(books=books, ignored=ignored) + + +def _relative_paths( + book_files: list[Path], root: Path | None = None +) -> tuple[Path, dict[Path, str]]: + if root is None: + root = Path(os.path.commonpath([str(f.parent) for f in book_files])) + return root, {f: f.relative_to(root).as_posix() for f in book_files} + + +def group_files_into_books( + book_files: list[Path], + *, + series_name: str | None, + author_name: str | None = None, + root: Path | None = None, +) -> list[BookGroup]: + """Heuristically split on-disk files into books (see `plan_pack`). + + `root` pins the release root when grouping a subset of a larger file set. + """ + if not book_files: + return [] + _root, rel_by_path = _relative_paths(book_files, root) + path_by_rel = {rel: path for path, rel in rel_by_path.items()} + extensions = {f.suffix.lower().lstrip(".") for f in book_files} + plan = plan_pack( + [PackFile(rel) for rel in rel_by_path.values()], + supported_extensions=extensions, + series_name=series_name, + author_name=author_name, + root_depth=None if root is None else 0, + ) + return [ + BookGroup( + title=book.title, + series_position=book.series_position, + year=book.year, + files=[path_by_rel[rel] for rel in book.files], + ) + for book in plan.books + ] + + +def match_plan_to_files( + plan: list[PackBook], + book_files: list[Path], + *, + series_name: str | None = None, + author_name: str | None = None, +) -> list[BookGroup]: + """Apply an approved plan to on-disk files. + + Files match by release-relative path first, then by basename (archive extraction + and client save paths can shift the root), then by the on-disk basename being a + suffix of the planned name (sources that glue folder and file names together). + Book files the plan does not mention fall back to heuristic grouping so nothing + is silently dropped. + """ + if not book_files: + return [] + root, rel_by_path = _relative_paths(book_files) + by_rel = {rel: path for path, rel in rel_by_path.items()} + by_name: dict[str, list[Path]] = {} + for path in book_files: + by_name.setdefault(path.name, []).append(path) + + claimed: set[Path] = set() + groups: list[BookGroup] = [] + for book in plan: + matched: list[Path] = [] + for wanted in book.files: + wanted_rel = wanted.replace("\\", "/").lstrip("./") + candidate = by_rel.get(wanted_rel) + if candidate is None: + candidates = [ + p for p in by_name.get(PurePosixPath(wanted_rel).name, []) if p not in claimed + ] + candidate = candidates[0] if candidates else None + if candidate is None: + wanted_name = PurePosixPath(wanted_rel).name.lower() + candidates = [ + p + for p in book_files + if p not in claimed and wanted_name.endswith(p.name.lower()) + ] + candidate = candidates[0] if len(candidates) == 1 else None + if candidate is not None and candidate not in claimed: + claimed.add(candidate) + matched.append(candidate) + if matched: + groups.append( + BookGroup( + title=book.title, + series_position=book.series_position, + year=book.year, + files=matched, + ) + ) + + unmatched = [p for p in book_files if p not in claimed] + if unmatched: + groups.extend( + group_files_into_books( + unmatched, series_name=series_name, author_name=author_name, root=root + ) + ) + return groups diff --git a/shelfmark/download/postprocess/pipeline.py b/shelfmark/download/postprocess/pipeline.py index f015350f..c9034c56 100644 --- a/shelfmark/download/postprocess/pipeline.py +++ b/shelfmark/download/postprocess/pipeline.py @@ -40,6 +40,7 @@ from .transfer import ( build_metadata_dict, is_torrent_source, process_directory, + resolve_book_groups, resolve_hardlink_source, should_hardlink, transfer_book_files, @@ -80,6 +81,7 @@ __all__ = [ "process_directory", "record_step", "resolve_custom_script_target", + "resolve_book_groups", "resolve_hardlink_source", "run_custom_script", "safe_cleanup_path", diff --git a/shelfmark/download/postprocess/transfer.py b/shelfmark/download/postprocess/transfer.py index 94118ca7..efdc0540 100644 --- a/shelfmark/download/postprocess/transfer.py +++ b/shelfmark/download/postprocess/transfer.py @@ -2,6 +2,7 @@ from __future__ import annotations +import dataclasses import os from pathlib import Path from typing import TYPE_CHECKING @@ -26,6 +27,7 @@ from shelfmark.download.fs import ( ) from shelfmark.download.postprocess.policy import get_file_organization, get_template +from .packs import BookGroup, PackBook, group_files_into_books, match_plan_to_files from .scan import collect_directory_files, scan_directory_tree from .types import TransferPlan from .workspace import safe_cleanup_path @@ -196,6 +198,19 @@ def transfer_book_files( is_audiobook = check_audiobook(task.content_type) organization_mode = organization_mode or get_file_organization(is_audiobook=is_audiobook) + + groups = resolve_book_groups(task, book_files, organization_mode=organization_mode) + if groups is not None: + return _transfer_book_groups( + groups, + destination, + task, + use_hardlink=use_hardlink, + is_torrent=is_torrent, + preserve_source=preserve_source, + organization_mode=organization_mode, + ) + max_attempts = _max_attempts_for_batch(len(book_files)) final_paths: list[Path] = [] @@ -299,6 +314,101 @@ def transfer_book_files( return final_paths, None, op_counts +def resolve_book_groups( + task: DownloadTask, + book_files: list[Path], + *, + organization_mode: str, +) -> list[BookGroup] | None: + """Split a multi-book pack into per-book groups, or None to file as one book. + + An approved `book_plan` wins; a bare `multi_book` flag falls back to heuristic + grouping. Organization `none` keeps files as-is, and a split that yields a single + group is not a pack at all. + """ + if organization_mode == "none" or not (task.book_plan or task.multi_book): + return None + if task.book_plan: + plan = [ + PackBook( + title=str(entry.get("title") or ""), + series_position=entry.get("series_position"), + year=entry.get("year"), + files=list(entry.get("files") or []), + ) + for entry in task.book_plan + if isinstance(entry, dict) + ] + groups = match_plan_to_files( + plan, book_files, series_name=task.series_name, author_name=task.author + ) + else: + groups = group_files_into_books( + book_files, series_name=task.series_name, author_name=task.author + ) + return groups if len(groups) > 1 else None + + +def _transfer_book_groups( + groups: list[BookGroup], + destination: Path, + task: DownloadTask, + *, + use_hardlink: bool, + is_torrent: bool, + preserve_source: bool, + organization_mode: str, +) -> tuple[list[Path], str | None, dict[str, int]]: + """Transfer each book of a pack through the normal single-book path. + + Each book gets an isolated task copy (the single-file path mutates `task.format`) + carrying its own title, position and year; the searched book's position must not + leak onto its siblings, while author and series name apply to all of them. + """ + all_paths: list[Path] = [] + totals: dict[str, int] = {"hardlink": 0, "copy": 0, "move": 0} + errors: list[str] = [] + + for group in groups: + book_task = dataclasses.replace( + task, + title=group.title or task.title, + year=str(group.year) if group.year is not None else None, + subtitle=None, + series_position=group.series_position, + multi_book=False, + book_plan=None, + ) + paths, error, op_counts = transfer_book_files( + group.files, + destination, + book_task, + use_hardlink=use_hardlink, + is_torrent=is_torrent, + preserve_source=preserve_source, + organization_mode=organization_mode, + source_root=group.files[0].parent, + ) + for op, count in op_counts.items(): + totals[op] = totals.get(op, 0) + count + if error: + errors.append(f"{group.title}: {error}") + logger.warning("Task %s: pack book %r failed: %s", task.task_id, group.title, error) + continue + all_paths.extend(paths) + + if not all_paths: + return [], "; ".join(errors) or "No book files found", totals + if errors: + logger.warning( + "Task %s: pack filed with %d failed book(s): %s", + task.task_id, + len(errors), + "; ".join(errors), + ) + return all_paths, None, totals + + def process_directory( directory: Path, ingest_dir: Path, diff --git a/shelfmark/main.py b/shelfmark/main.py index 68903b34..5f60008b 100644 --- a/shelfmark/main.py +++ b/shelfmark/main.py @@ -62,6 +62,7 @@ from shelfmark.core.notifications import ( notify_user, ) from shelfmark.core.prefix_middleware import PrefixMiddleware +from shelfmark.core.release_inspect_routes import register_release_inspect_routes from shelfmark.core.request_helpers import ( coerce_bool, emit_ws_event, @@ -1024,6 +1025,9 @@ def _serialize_release(release: Release) -> dict: return result +register_release_inspect_routes(app, login_required) + + @app.route("/api/releases/download", methods=["POST"]) @login_required def api_download_release() -> Response | tuple[Response, int]: diff --git a/shelfmark/release_sources/__init__.py b/shelfmark/release_sources/__init__.py index b0c751d4..deb69e40 100644 --- a/shelfmark/release_sources/__init__.py +++ b/shelfmark/release_sources/__init__.py @@ -13,6 +13,7 @@ if TYPE_CHECKING: from shelfmark.core.models import DownloadTask from shelfmark.core.search_plan import ReleaseSearchPlan + from shelfmark.download.postprocess.packs import PackFile from shelfmark.metadata_providers import BookMetadata @@ -400,6 +401,14 @@ class DownloadHandler(ABC): """Return private queue-time fields needed for restart-safe retry.""" return {} + def list_files(self, release_data: dict[str, Any]) -> list[PackFile] | None: + """Return the release's file list without downloading it. + + Lets the UI review a multi-book pack before queueing. Return None when the + source cannot know the files ahead of time (magnet links, usenet, ...). + """ + return None + @abstractmethod def cancel(self, task_id: str) -> bool: """Cancel an in-progress download.""" diff --git a/shelfmark/release_sources/audiobookbay/handler.py b/shelfmark/release_sources/audiobookbay/handler.py index 723fbfcc..f080ec9f 100644 --- a/shelfmark/release_sources/audiobookbay/handler.py +++ b/shelfmark/release_sources/audiobookbay/handler.py @@ -1,6 +1,6 @@ """AudiobookBay download handler - resolves magnet links and uses shared client lifecycle.""" -from typing import TYPE_CHECKING +from typing import TYPE_CHECKING, Any from urllib.parse import urlparse from shelfmark.core.config import config @@ -22,6 +22,7 @@ if TYPE_CHECKING: from collections.abc import Callable from shelfmark.core.models import DownloadTask + from shelfmark.download.postprocess.packs import PackFile logger = setup_logger(__name__) DEFAULT_ABB_HOSTNAME = "audiobookbay.lu" @@ -68,6 +69,19 @@ class AudiobookBayHandler(ExternalClientHandler): return task_id return None + def list_files(self, release_data: dict[str, Any]) -> list[PackFile] | None: + """Read the torrent's file list off the detail page, without downloading.""" + raw_url = release_data.get("download_url") or release_data.get("source_url") + detail_url = raw_url.strip() if isinstance(raw_url, str) else "" + hostname = _resolve_allowed_detail_hostname() + if not detail_url or not _detail_url_matches_host(detail_url, hostname): + logger.debug("Cannot list files for AudiobookBay release without a valid detail URL") + return None + detail_html = scraper.fetch_detail_html(detail_url, hostname) + if not detail_html: + return None + return scraper.extract_file_list(detail_html) + def _get_client(self, protocol: str) -> DownloadClient | None: """Compatibility shim so module-level patching still works in tests.""" return get_client(protocol) diff --git a/shelfmark/release_sources/audiobookbay/scraper.py b/shelfmark/release_sources/audiobookbay/scraper.py index 217fe7d3..7fd65b14 100644 --- a/shelfmark/release_sources/audiobookbay/scraper.py +++ b/shelfmark/release_sources/audiobookbay/scraper.py @@ -2,6 +2,7 @@ import re import time +from threading import Lock from urllib.parse import quote, quote_plus import requests @@ -10,6 +11,7 @@ from bs4 import BeautifulSoup from shelfmark.core.config import config from shelfmark.core.logger import setup_logger from shelfmark.download import http as downloader +from shelfmark.download.postprocess.packs import PackFile from shelfmark.release_sources.audiobookbay.utils import normalize_search_punctuation logger = setup_logger(__name__) @@ -32,6 +34,13 @@ FIRST_PAGE_SESSION_REFRESH_ATTEMPTS = 2 # Legacy search parameter used by older ABB flows LEGACY_CATEGORY_QUERY = "undefined%2Cundefined" +# Detail pages are fetched once and shared by inspection (file list) and download +# (magnet link) so a "review then download" round trip costs ABB a single request. +DETAIL_PAGE_CACHE_TTL_SECONDS = 120.0 +DETAIL_PAGE_CACHE_MAX_ENTRIES = 8 +_detail_page_cache: dict[str, tuple[float, str]] = {} +_detail_page_cache_lock = Lock() + # Precompiled patterns used while parsing result cards LANGUAGE_PATTERN = re.compile(r"Language:\s*([A-Za-z]+)") POSTED_PATTERN = re.compile(r"Posted:\s*(\d+\s+[A-Za-z]+\s+\d{4})") @@ -39,6 +48,11 @@ FORMAT_PATTERN = re.compile(r"Format:\s*([A-Za-z0-9]+)") BITRATE_PATTERN = re.compile(r"Bitrate:\s*([\d]+\s*[A-Za-z/]+)") SIZE_PATTERN = re.compile(r"File Size:\s*([\d.]+)\s*([A-Za-z]+)") INFO_HASH_LABEL_PATTERN = re.compile(r"Info Hash", re.IGNORECASE) +FILE_ROW_SIZE_PATTERN = re.compile( + r"^(?P<name>.+?)\s+(?P<size>\d+(?:\.\d+)?)\s*(?P<unit>Bytes?|KBs?|MBs?|GBs?|TBs?)$", + re.IGNORECASE, +) +_FILE_SIZE_MULTIPLIERS = {"b": 1, "k": 1024, "m": 1024**2, "g": 1024**3, "t": 1024**4} def _coerce_non_negative_float(value: object, default: float) -> float: @@ -348,6 +362,98 @@ def search_audiobookbay( return results +def _get_cached_detail_page(details_url: str) -> str | None: + with _detail_page_cache_lock: + entry = _detail_page_cache.get(details_url) + if entry is None: + return None + fetched_at, html = entry + if time.monotonic() - fetched_at > DETAIL_PAGE_CACHE_TTL_SECONDS: + del _detail_page_cache[details_url] + return None + return html + + +def _store_cached_detail_page(details_url: str, html: str) -> None: + with _detail_page_cache_lock: + _detail_page_cache[details_url] = (time.monotonic(), html) + while len(_detail_page_cache) > DETAIL_PAGE_CACHE_MAX_ENTRIES: + oldest = min(_detail_page_cache, key=lambda key: _detail_page_cache[key][0]) + del _detail_page_cache[oldest] + + +def clear_detail_page_cache() -> None: + """Drop cached detail pages (used by tests).""" + with _detail_page_cache_lock: + _detail_page_cache.clear() + + +def _fetch_detail_page_once(details_url: str, hostname: str) -> str: + session = requests.Session() + _bootstrap_abb_session(hostname, session, DETAIL_PAGE_RETRY_ATTEMPTS) + return _coerce_markup_to_html( + downloader.html_get_page( + details_url, + retry=DETAIL_PAGE_RETRY_ATTEMPTS, + use_bypasser=False, + allow_bypasser_fallback=False, + success_delay=0, + session=session, + ) + ) + + +def fetch_detail_html(details_url: str, hostname: str = "audiobookbay.lu") -> str: + """Fetch a detail page (one retry with a fresh session), cached briefly per URL.""" + cached = _get_cached_detail_page(details_url) + if cached is not None: + logger.debug("Reusing recently fetched detail page: %s", details_url) + return cached + detail_html = _fetch_detail_page_once(details_url, hostname) + if not detail_html: + detail_html = _fetch_detail_page_once(details_url, hostname) + if detail_html: + _store_cached_detail_page(details_url, detail_html) + return detail_html + + +def _parse_file_row(text: str) -> PackFile | None: + match = FILE_ROW_SIZE_PATTERN.match(text.strip()) + if not match: + return None + multiplier = _FILE_SIZE_MULTIPLIERS[match.group("unit")[0].lower()] + return PackFile(match.group("name"), int(float(match.group("size")) * multiplier)) + + +def extract_file_list(detail_html: str) -> list[PackFile] | None: + """Read the torrent file rows off a detail page. + + ABB renders the torrent's file table as single-cell rows between the + "This is a Multifile Torrent" marker (absent for single-file torrents) and the + "Combined File Size" row. Returns None when the page has no such table. + """ + soup = BeautifulSoup(detail_html, "html.parser") + rows: list[PackFile] = [] + for row in soup.find_all("tr"): + cells = row.find_all("td") + if not cells: + continue + label = cells[0].get_text(" ", strip=True) + if label.lower().startswith("combined file size"): + return rows or None + if len(cells) != 1: + rows = [] # a two-column metadata row means we're not in the file table yet + continue + text = cells[0].get_text(" ", strip=True) + if "multifile torrent" in text.lower(): + rows = [] + continue + parsed = _parse_file_row(text) + if parsed is not None: + rows.append(parsed) + return None + + def extract_magnet_link(details_url: str, hostname: str = "audiobookbay.lu") -> str | None: """Extract info hash and trackers from book detail page, then construct magnet link. @@ -360,35 +466,7 @@ def extract_magnet_link(details_url: str, hostname: str = "audiobookbay.lu") -> """ try: - session = requests.Session() - _bootstrap_abb_session(hostname, session, DETAIL_PAGE_RETRY_ATTEMPTS) - - # Fetch detail page - detail_html = _coerce_markup_to_html( - downloader.html_get_page( - details_url, - retry=DETAIL_PAGE_RETRY_ATTEMPTS, - use_bypasser=False, - allow_bypasser_fallback=False, - success_delay=0, - session=session, - ) - ) - - if not detail_html: - session = requests.Session() - _bootstrap_abb_session(hostname, session, DETAIL_PAGE_RETRY_ATTEMPTS) - detail_html = _coerce_markup_to_html( - downloader.html_get_page( - details_url, - retry=DETAIL_PAGE_RETRY_ATTEMPTS, - use_bypasser=False, - allow_bypasser_fallback=False, - success_delay=0, - session=session, - ) - ) - + detail_html = fetch_detail_html(details_url, hostname) if not detail_html: logger.warning("Failed to fetch details page") return None diff --git a/shelfmark/release_sources/prowlarr/handler.py b/shelfmark/release_sources/prowlarr/handler.py index cf432ad7..78ca74c5 100644 --- a/shelfmark/release_sources/prowlarr/handler.py +++ b/shelfmark/release_sources/prowlarr/handler.py @@ -28,6 +28,10 @@ from shelfmark.download.clients.base_handler import ( DownloadRequest, ExternalClientHandler, ) +from shelfmark.download.clients.torrent_utils import ( + extract_file_list_from_torrent, + extract_torrent_info, +) from shelfmark.metadata_providers import BookMetadata from shelfmark.release_sources import register_handler from shelfmark.release_sources.prowlarr.api import IndexerSeedSettings, ProwlarrClient @@ -38,12 +42,14 @@ from shelfmark.release_sources.prowlarr.utils import ( coerce_int_like, get_preferred_download_url, get_protocol, + sanitize_download_url, ) if TYPE_CHECKING: from collections.abc import Callable from shelfmark.core.models import DownloadTask + from shelfmark.download.postprocess.packs import PackFile logger = setup_logger(__name__) @@ -127,6 +133,24 @@ class ProwlarrHandler(ExternalClientHandler): return settings.get(indexer_id) + def list_files(self, release_data: dict[str, Any]) -> list[PackFile] | None: + """List a cached torrent release's files from its .torrent, without downloading. + + Magnet-only and usenet releases cannot be listed ahead of time. + """ + source_id = str(release_data.get("source_id") or "") + prowlarr_result = get_release(source_id) if source_id else None + if not prowlarr_result or get_protocol(prowlarr_result) != "torrent": + return None + download_url = sanitize_download_url(str(prowlarr_result.get("downloadUrl") or "").strip()) + if not download_url or download_url.startswith("magnet:"): + return None + expected_hash = str(prowlarr_result.get("infoHash") or "").strip() or None + info = extract_torrent_info(download_url, expected_hash=expected_hash) + if not info.torrent_data: + return None + return extract_file_list_from_torrent(info.torrent_data) + def _get_client(self, protocol: str) -> DownloadClient | None: """Compatibility shim so module-level patching still works in tests.""" return get_client(protocol) diff --git a/src/frontend/src/App.tsx b/src/frontend/src/App.tsx index 65a572b7..bad19116 100644 --- a/src/frontend/src/App.tsx +++ b/src/frontend/src/App.tsx @@ -58,7 +58,6 @@ import { isApiResponseError, updateSelfUser, setBookTargetState, - type DownloadReleasePayload, } from './services/api'; import type { Book, @@ -93,6 +92,7 @@ import { getEffectiveMetadataSort } from './utils/metadataSort'; import { isRecord } from './utils/objectHelpers'; import { policyTrace } from './utils/policyTrace'; import { buildQueryTargets, getDefaultQueryTargetKey } from './utils/queryTargets'; +import { buildReleaseDownloadPayload, type ReleaseDownloadOptions } from './utils/releasePayload'; import { applyRequestNoteToPayload } from './utils/requestConfirmation'; import { bookFromRequestData } from './utils/requestFulfil'; import { @@ -219,6 +219,7 @@ type PendingOnBehalfDownload = release: Release; releaseContentType: ContentType; actingAsUser: ActingAsUserSelection; + options?: ReleaseDownloadOptions; } | { type: 'combined'; @@ -1072,41 +1073,6 @@ function App() { [], ); - const buildReleaseDownloadPayload = useCallback( - (book: Book, release: Release, releaseContentType: ContentType): DownloadReleasePayload => { - const isManual = book.provider === 'manual'; - const releasePreview = - typeof release.extra?.preview === 'string' ? release.extra.preview : undefined; - const releaseAuthor = - typeof release.extra?.author === 'string' ? release.extra.author : undefined; - - return { - source: release.source, - source_id: release.source_id, - title: isManual ? release.title : book.title, - author: isManual ? releaseAuthor || '' : book.author, - year: book.year, - format: release.format, - size: release.size, - size_bytes: release.size_bytes, - download_url: release.download_url, - protocol: release.protocol, - indexer: release.indexer, - seeders: release.seeders, - extra: release.extra, - preview: isManual ? releasePreview || undefined : book.preview, - content_type: releaseContentType, - series_name: book.series_name, - series_position: book.series_position, - subtitle: book.subtitle, - // From the release, never the book: book.language is the provider's - // canonical edition, which would mislabel a translated release. - language: release.language ?? undefined, - }; - }, - [], - ); - // When downloading a book while browsing a Hardcover list the user owns, // automatically remove it from that list (fire-and-forget). const searchFieldLabelsRef = useRef(searchFieldLabels); @@ -1214,12 +1180,13 @@ function App() { release: Release, releaseContentType: ContentType, onBehalfOfUserId?: number, + options?: ReleaseDownloadOptions, ): Promise<void> => { const requestStartedAtSeconds = Date.now() / 1000; try { trackRelease(book.id, release.source_id); await downloadRelease( - buildReleaseDownloadPayload(book, release, releaseContentType), + buildReleaseDownloadPayload(book, release, releaseContentType, options), onBehalfOfUserId, ); await fetchStatus(); @@ -1301,7 +1268,6 @@ function App() { } }, [ - buildReleaseDownloadPayload, fetchStatus, openRequestConfirmation, refreshRequestPolicy, @@ -1416,6 +1382,7 @@ function App() { effectivePendingOnBehalfDownload.release, effectivePendingOnBehalfDownload.releaseContentType, onBehalfOfUserId, + effectivePendingOnBehalfDownload.options, ); } setPendingOnBehalfDownload(null); @@ -1640,6 +1607,7 @@ function App() { book: Book, release: Release, releaseContentType: ContentType, + options?: ReleaseDownloadOptions, ) => { policyTrace('release.action:start', { bookId: book.id, @@ -1655,11 +1623,12 @@ function App() { release, releaseContentType, actingAsUser: effectiveActingAsUser, + options, }); return; } - await executeReleaseDownload(book, release, releaseContentType); + await executeReleaseDownload(book, release, releaseContentType, undefined, options); }; const handleReleaseRequest = useCallback( diff --git a/src/frontend/src/components/PackReviewPanel.tsx b/src/frontend/src/components/PackReviewPanel.tsx new file mode 100644 index 00000000..007ccf1a --- /dev/null +++ b/src/frontend/src/components/PackReviewPanel.tsx @@ -0,0 +1,193 @@ +import { useState } from 'react'; + +import type { PackBook, PackPlan, Release } from '../types'; +import { + describePackPlan, + parseSeriesPositionInput, + toBookPlanPayload, + updateReviewBook, +} from '../utils/packReview'; +import { ToggleSwitch } from './shared/ToggleSwitch'; + +interface PackReviewPanelProps { + release: Release; + plan: PackPlan; + books: PackBook[]; + onChange: (books: PackBook[]) => void; + onBack: () => void; + /** `null` means "treat the whole release as one book". */ + onConfirm: (books: PackBook[] | null) => Promise<void>; + isSubmitting: boolean; +} + +const inputClassName = + 'w-full rounded-md border border-(--border-muted) bg-(--bg) px-2 py-1 text-sm text-(--text) focus:border-emerald-500 focus:outline-none'; + +export const PackReviewPanel = ({ + release, + plan, + books, + onChange, + onBack, + onConfirm, + isSubmitting, +}: PackReviewPanelProps) => { + const [singleBook, setSingleBook] = useState(false); + const [expandedFiles, setExpandedFiles] = useState<number | null>(null); + const [showIgnored, setShowIgnored] = useState(false); + + const payloadBooks = toBookPlanPayload(books); + const canConfirm = !isSubmitting && (singleBook || payloadBooks.length > 0); + const confirmLabel = singleBook + ? 'Download as one book' + : `Download ${payloadBooks.length} ${payloadBooks.length === 1 ? 'book' : 'books'}`; + + return ( + <div className="flex flex-col gap-4 px-5 py-4" data-testid="pack-review-panel"> + <div> + <h3 className="text-base font-semibold text-(--text)"> + This release contains several books + </h3> + <p className="mt-1 text-sm text-zinc-500 dark:text-zinc-400"> + <span className="font-medium text-(--text)">{release.title}</span> ·{' '} + {describePackPlan(books, plan.ignored)} + </p> + <p className="mt-1 text-xs text-zinc-500 dark:text-zinc-400"> + Each book below is filed separately with its own title. Fix any titles before downloading + — the author and series come from the book you searched. + </p> + </div> + + <div className="flex items-center justify-between rounded-lg border border-(--border-muted) px-3 py-2"> + <div> + <p className="text-sm font-medium text-(--text)">Treat as a single book</p> + <p className="text-xs text-zinc-500 dark:text-zinc-400"> + Use this if the split is wrong and the files are really one audiobook. + </p> + </div> + <ToggleSwitch + checked={singleBook} + onChange={setSingleBook} + color="emerald" + ariaLabel="Treat as a single book" + disabled={isSubmitting} + /> + </div> + + <div + className={`flex flex-col divide-y divide-zinc-200/60 dark:divide-zinc-800/60 ${ + singleBook ? 'pointer-events-none opacity-40' : '' + }`} + > + <div className="grid grid-cols-[minmax(0,1fr)_72px_72px_80px] gap-2 pb-1 text-xs font-medium tracking-wide text-zinc-500 uppercase dark:text-zinc-400"> + <span>Title</span> + <span>Series #</span> + <span>Year</span> + <span className="text-right">Files</span> + </div> + {books.map((book, index) => ( + <div key={book.files[0] ?? index} className="py-2"> + <div className="grid grid-cols-[minmax(0,1fr)_72px_72px_80px] items-center gap-2"> + <input + type="text" + value={book.title} + onChange={(e) => + onChange(updateReviewBook(books, index, { title: e.target.value })) + } + aria-label={`Title for book ${index + 1}`} + className={inputClassName} + disabled={isSubmitting} + /> + <input + type="text" + inputMode="decimal" + value={book.series_position ?? ''} + onChange={(e) => + onChange( + updateReviewBook(books, index, { + series_position: parseSeriesPositionInput(e.target.value), + }), + ) + } + aria-label={`Series position for book ${index + 1}`} + className={inputClassName} + disabled={isSubmitting} + /> + <input + type="text" + inputMode="numeric" + value={book.year ?? ''} + onChange={(e) => { + const parsed = parseSeriesPositionInput(e.target.value); + onChange( + updateReviewBook(books, index, { + year: parsed === null ? null : Math.trunc(parsed), + }), + ); + }} + aria-label={`Year for book ${index + 1}`} + className={inputClassName} + disabled={isSubmitting} + /> + <button + type="button" + onClick={() => setExpandedFiles(expandedFiles === index ? null : index)} + className="hover-surface rounded-md px-2 py-1 text-right text-sm text-zinc-500 transition-colors dark:text-zinc-400" + aria-expanded={expandedFiles === index} + > + {book.files.length} {book.files.length === 1 ? 'file' : 'files'} + </button> + </div> + {expandedFiles === index && ( + <ul className="mt-2 max-h-40 overflow-y-auto rounded-md bg-(--bg-soft) px-3 py-2 font-mono text-xs break-all text-zinc-600 dark:text-zinc-300"> + {book.files.map((file) => ( + <li key={file}>{file}</li> + ))} + </ul> + )} + </div> + ))} + </div> + + {plan.ignored.length > 0 && ( + <div> + <button + type="button" + onClick={() => setShowIgnored(!showIgnored)} + className="text-xs text-zinc-500 underline-offset-2 hover:underline dark:text-zinc-400" + aria-expanded={showIgnored} + > + {plan.ignored.length} {plan.ignored.length === 1 ? 'file' : 'files'} ignored (not a book + format) + </button> + {showIgnored && ( + <ul className="mt-2 max-h-32 overflow-y-auto rounded-md bg-(--bg-soft) px-3 py-2 font-mono text-xs break-all text-zinc-600 dark:text-zinc-300"> + {plan.ignored.map((file) => ( + <li key={file}>{file}</li> + ))} + </ul> + )} + </div> + )} + + <div className="flex items-center justify-end gap-3 border-t border-(--border-muted) pt-4"> + <button + type="button" + onClick={onBack} + disabled={isSubmitting} + className="hover-surface rounded-lg px-3 py-1.5 text-sm font-medium text-(--text) transition-colors disabled:opacity-50" + > + ← Back + </button> + <button + type="button" + onClick={() => void onConfirm(singleBook ? null : payloadBooks)} + disabled={!canConfirm} + className="rounded-lg bg-emerald-600 px-4 py-1.5 text-sm font-medium text-white transition-colors hover:bg-emerald-700 disabled:cursor-not-allowed disabled:opacity-50" + > + {isSubmitting ? 'Queuing…' : confirmLabel} + </button> + </div> + </div> + ); +}; diff --git a/src/frontend/src/components/ReleaseModal.tsx b/src/frontend/src/components/ReleaseModal.tsx index 0359316f..adca13fc 100644 --- a/src/frontend/src/components/ReleaseModal.tsx +++ b/src/frontend/src/components/ReleaseModal.tsx @@ -7,6 +7,7 @@ import { useReleaseSearchSession } from '../hooks/releaseModal/useReleaseSearchS import { useTabIndicator } from '../hooks/ui/useTabIndicator'; import { useBodyScrollLock } from '../hooks/useBodyScrollLock'; import { useEscapeKey } from '../hooks/useEscapeKey'; +import { inspectRelease } from '../services/api'; import type { Book, Release, @@ -18,6 +19,8 @@ import type { LeadingCellConfig, ContentType, RequestPolicyMode, + PackBook, + PackPlan, } from '../types'; import { isMetadataBook } from '../types'; import { bookSupportsTargets } from '../utils/bookTargetLoader'; @@ -29,7 +32,9 @@ import { buildLanguageNormalizer, } from '../utils/languageFilters'; import { getNestedValue, toComparableText, toStringValue } from '../utils/objectHelpers'; +import { toBookPlanPayload } from '../utils/packReview'; import { getReleaseFormats } from '../utils/releaseFormats'; +import { buildReleaseDownloadPayload, type ReleaseDownloadOptions } from '../utils/releasePayload'; import { getBookTitleCandidates, getBookAuthorCandidates, @@ -50,6 +55,7 @@ import { BookTargetDropdown } from './BookTargetDropdown'; import { Dropdown } from './Dropdown'; import { DropdownList } from './DropdownList'; import { LanguageMultiSelect } from './LanguageMultiSelect'; +import { PackReviewPanel } from './PackReviewPanel'; import { ReleaseCell } from './ReleaseCell'; // Combined mode configuration for the ReleaseModal @@ -140,7 +146,12 @@ const DEFAULT_COLUMN_CONFIG: ReleaseColumnConfig = { interface ReleaseModalProps { book: Book | null; onClose: () => void; - onDownload: (book: Book, release: Release, contentType: ContentType) => Promise<void>; + onDownload: ( + book: Book, + release: Release, + contentType: ContentType, + options?: ReleaseDownloadOptions, + ) => Promise<void>; onRequestRelease?: (book: Book, release: Release, contentType: ContentType) => Promise<void>; onRequestBook?: (book: Book, contentType: ContentType) => Promise<void>; getPolicyModeForSource?: (source: string, contentType: ContentType) => RequestPolicyMode; @@ -762,6 +773,15 @@ const ReleaseModalSession = ({ : supportedFormats; const [isRequestingBook, setIsRequestingBook] = useState(false); const [selectedRelease, setSelectedRelease] = useState<Release | null>(null); + // Multi-book packs: `multiBook` is the manual header toggle (heuristic split for + // releases we can't inspect); `packReview` holds an inspected pack awaiting approval. + const [multiBook, setMultiBook] = useState(false); + const [packReview, setPackReview] = useState<{ + release: Release; + plan: PackPlan; + books: PackBook[]; + } | null>(null); + const [packSubmitting, setPackSubmitting] = useState(false); const isCombinedMode = combinedMode != null; const combinedPhase = combinedMode?.phase ?? null; const combinedStepLabel = combinedMode?.stepLabel ?? ''; @@ -1196,7 +1216,30 @@ const ReleaseModalSession = ({ const mode = getReleaseActionMode(release); if (mode === 'download') { - await onDownload(book, release, contentType); + // Look at the release's files before queueing so a whole-series pack can be + // reviewed and filed as separate books instead of one mangled item. + let inspected = false; + let plan: PackPlan | null = null; + try { + const inspection = await inspectRelease( + buildReleaseDownloadPayload(book, release, contentType), + ); + inspected = inspection.inspected; + plan = inspection.plan; + } catch (error) { + console.error('Release inspection failed:', error); + } + if (inspected && plan?.is_pack) { + setPackReview({ release, plan, books: plan.books }); + return; + } + if (!inspected && !multiBook) { + onShowToast?.( + "Couldn't check this release's files before download. If it contains several books, turn on the multi-book pack toggle first.", + 'info', + ); + } + await onDownload(book, release, contentType, multiBook ? { multiBook: true } : {}); handleClose(); return; } @@ -1215,9 +1258,32 @@ const ReleaseModalSession = ({ onRequestRelease, contentType, handleClose, + multiBook, + onShowToast, ], ); + const handlePackConfirm = useCallback( + async (books: PackBook[] | null): Promise<void> => { + if (!book || !packReview) { + return; + } + setPackSubmitting(true); + try { + await onDownload( + book, + packReview.release, + contentType, + books ? { multiBook: true, bookPlan: toBookPlanPayload(books) } : {}, + ); + handleClose(); + } finally { + setPackSubmitting(false); + } + }, + [book, packReview, onDownload, contentType, handleClose], + ); + const titleId = `release-modal-title-${book.id}`; const providerDisplay = book.provider_display_name || @@ -1720,6 +1786,37 @@ const ReleaseModalSession = ({ </div> <div className="flex items-center gap-3 pr-1 pl-2"> + {/* Multi-book pack toggle (fallback for releases that can't be inspected) */} + {!isCombinedMode && ( + <button + type="button" + onClick={() => setMultiBook((prev) => !prev)} + className={`hover-surface relative rounded-full p-2.5 text-zinc-500 transition-colors dark:text-zinc-400 ${ + multiBook ? 'text-emerald-600 dark:text-emerald-400' : '' + }`} + aria-label="Multi-book pack" + aria-pressed={multiBook} + title="Multi-book pack: file each subfolder (or each file) as a separate book. Only needed when a release can't be inspected before download." + > + <svg + className="h-4 w-4" + fill="none" + stroke="currentColor" + viewBox="0 0 24 24" + strokeWidth={1.5} + > + <path + strokeLinecap="round" + strokeLinejoin="round" + d="M6.429 9.75 2.25 12l4.179 2.25m0-4.5 5.571 3 5.571-3m-11.142 0L2.25 7.5 12 2.25l9.75 5.25-4.179 2.25m0 0L21.75 12l-4.179 2.25m0 0 4.179 2.25L12 21.75 2.25 16.5l4.179-2.25m11.142 0-5.571 3-5.571-3" + /> + </svg> + {multiBook && ( + <span className="absolute top-1 right-1 h-2 w-2 rounded-full bg-emerald-500" /> + )} + </button> + )} + {/* Manual query button */} <button type="button" @@ -2120,6 +2217,19 @@ const ReleaseModalSession = ({ {/* Release list content */} <div className="min-h-[200px]"> {(() => { + if (packReview) { + return ( + <PackReviewPanel + release={packReview.release} + plan={packReview.plan} + books={packReview.books} + onChange={(books) => setPackReview({ ...packReview, books })} + onBack={() => setPackReview(null)} + onConfirm={handlePackConfirm} + isSubmitting={packSubmitting} + /> + ); + } if (sourcesLoading) { return <ReleaseSkeleton />; } diff --git a/src/frontend/src/services/api.ts b/src/frontend/src/services/api.ts index 61a26ed2..2537da8a 100644 --- a/src/frontend/src/services/api.ts +++ b/src/frontend/src/services/api.ts @@ -12,6 +12,8 @@ import type { RequestSubmissionResult, MetadataProvidersResponse, MetadataSearchConfig, + PackBook, + InspectReleaseResponse, } from '../types'; import type { ActionResult, @@ -510,6 +512,18 @@ export type DownloadReleasePayload = { language?: string; // Release language code, for the {Language} naming variable search_author?: string; search_mode?: 'direct' | 'universal'; + multi_book?: boolean; // Split a multi-book pack into one book per subfolder/file + book_plan?: PackBook[]; // The split the user approved before download +}; + +/** Inspect a release's file list before download (same body as downloadRelease). */ +export const inspectRelease = async ( + release: DownloadReleasePayload, +): Promise<InspectReleaseResponse> => { + return fetchJSON<InspectReleaseResponse>(`${API_BASE}/releases/inspect`, { + method: 'POST', + body: JSON.stringify(release), + }); }; export const downloadRelease = async ( diff --git a/src/frontend/src/tests/packReview.test.ts b/src/frontend/src/tests/packReview.test.ts new file mode 100644 index 00000000..c4d6fad5 --- /dev/null +++ b/src/frontend/src/tests/packReview.test.ts @@ -0,0 +1,53 @@ +import { describe, it, expect } from 'vitest'; + +import type { PackBook } from '../types'; +import { + describePackPlan, + parseSeriesPositionInput, + toBookPlanPayload, + updateReviewBook, +} from '../utils/packReview'; + +const books: PackBook[] = [ + { title: 'Leviathan Wakes', series_position: 1, year: 2011, files: ['a.m4b'] }, + { title: 'Caliban’s War', series_position: 2, year: 2012, files: ['b.m4b', 'b2.m4b'] }, +]; + +describe('packReview.updateReviewBook', () => { + it('replaces one book without touching the others', () => { + const next = updateReviewBook(books, 1, { title: 'Caliban’s War (Unabridged)' }); + expect(next[0]).toBe(books[0]); + expect(next[1]).toEqual({ ...books[1], title: 'Caliban’s War (Unabridged)' }); + expect(books[1].title).toBe('Caliban’s War'); + }); +}); + +describe('packReview.parseSeriesPositionInput', () => { + it('accepts whole and fractional positions', () => { + expect(parseSeriesPositionInput('3')).toBe(3); + expect(parseSeriesPositionInput('2.5')).toBe(2.5); + }); + + it('treats blank or junk as no position', () => { + expect(parseSeriesPositionInput('')).toBeNull(); + expect(parseSeriesPositionInput('abc')).toBeNull(); + }); +}); + +describe('packReview.toBookPlanPayload', () => { + it('trims titles, drops books without a title, and keeps file lists', () => { + const edited = updateReviewBook(books, 0, { title: ' ' }); + expect(toBookPlanPayload(edited)).toEqual([ + { title: 'Caliban’s War', series_position: 2, year: 2012, files: ['b.m4b', 'b2.m4b'] }, + ]); + }); +}); + +describe('packReview.describePackPlan', () => { + it('summarises books, files and ignored sidecars', () => { + expect(describePackPlan(books, ['a.txt', 'cover.jpg'])).toBe( + '2 books · 3 files · 2 files ignored', + ); + expect(describePackPlan([books[0]], [])).toBe('1 book · 1 file'); + }); +}); diff --git a/src/frontend/src/tests/releasePayload.test.ts b/src/frontend/src/tests/releasePayload.test.ts new file mode 100644 index 00000000..f767cba2 --- /dev/null +++ b/src/frontend/src/tests/releasePayload.test.ts @@ -0,0 +1,69 @@ +import { describe, it, expect } from 'vitest'; + +import type { Book, Release } from '../types'; +import { buildReleaseDownloadPayload } from '../utils/releasePayload'; + +const book: Book = { + id: 'hc-1', + title: 'Drive', + author: 'James S. A. Corey', + year: '2012', + preview: 'https://img/drive.jpg', + series_name: 'The Expanse', + series_position: 2.6, + subtitle: 'An Expanse Short Story', + provider: 'hardcover', + provider_id: 'hc-1', + source: 'direct_download', +}; + +const release: Release = { + source: 'audiobookbay', + source_id: 'abb-1', + title: 'James S. A. Corey - The Expanse Complete 2.0', + format: 'm4b', + language: 'en', + download_url: 'https://audiobookbay.lu/abss/expanse/', +}; + +describe('buildReleaseDownloadPayload', () => { + it('describes the searched book and the chosen release', () => { + const payload = buildReleaseDownloadPayload(book, release, 'audiobook'); + expect(payload).toMatchObject({ + source: 'audiobookbay', + source_id: 'abb-1', + title: 'Drive', + author: 'James S. A. Corey', + series_name: 'The Expanse', + series_position: 2.6, + language: 'en', + content_type: 'audiobook', + }); + expect(payload.multi_book).toBeUndefined(); + expect(payload.book_plan).toBeUndefined(); + }); + + it('uses the release title and author for manual books', () => { + const manual: Book = { ...book, provider: 'manual', title: 'ignored' }; + const withAuthor = { ...release, extra: { author: 'Release Author' } }; + const payload = buildReleaseDownloadPayload(manual, withAuthor, 'audiobook'); + expect(payload.title).toBe(release.title); + expect(payload.author).toBe('Release Author'); + }); + + it('flags a manual multi-book pack', () => { + const payload = buildReleaseDownloadPayload(book, release, 'audiobook', { multiBook: true }); + expect(payload.multi_book).toBe(true); + expect(payload.book_plan).toBeUndefined(); + }); + + it('attaches the approved book plan', () => { + const plan = [{ title: 'Leviathan Wakes', series_position: 1, year: 2011, files: ['a.m4b'] }]; + const payload = buildReleaseDownloadPayload(book, release, 'audiobook', { + multiBook: true, + bookPlan: plan, + }); + expect(payload.multi_book).toBe(true); + expect(payload.book_plan).toEqual(plan); + }); +}); diff --git a/src/frontend/src/types/index.ts b/src/frontend/src/types/index.ts index 3c35b84e..6195afa1 100644 --- a/src/frontend/src/types/index.ts +++ b/src/frontend/src/types/index.ts @@ -460,6 +460,27 @@ export interface SourceSearchInfo { } // Response from /api/releases endpoint +/** One book split out of a multi-book pack release, files as release-relative paths. */ +export interface PackBook { + title: string; + series_position: number | null; + year: number | null; + files: string[]; +} + +export interface PackPlan { + is_pack: boolean; + books: PackBook[]; + ignored: string[]; +} + +export interface InspectReleaseResponse { + inspected: boolean; + reason: string | null; + files: { path: string; size: number | null }[]; + plan: PackPlan | null; +} + export interface ReleasesResponse { releases: Release[]; book: { diff --git a/src/frontend/src/utils/packReview.ts b/src/frontend/src/utils/packReview.ts new file mode 100644 index 00000000..cbe53edd --- /dev/null +++ b/src/frontend/src/utils/packReview.ts @@ -0,0 +1,39 @@ +import type { PackBook } from '../types'; + +/** Return a copy of `books` with one entry patched; the input is not mutated. */ +export function updateReviewBook( + books: PackBook[], + index: number, + patch: Partial<PackBook>, +): PackBook[] { + return books.map((book, i) => (i === index ? { ...book, ...patch } : book)); +} + +/** Parse a series-position text field: "3" → 3, "2.5" → 2.5, blank/junk → null. */ +export function parseSeriesPositionInput(value: string): number | null { + const trimmed = value.trim(); + if (!trimmed) return null; + const parsed = Number(trimmed); + return Number.isFinite(parsed) ? parsed : null; +} + +/** The plan sent with the download: trimmed titles, untitled books dropped. */ +export function toBookPlanPayload(books: PackBook[]): PackBook[] { + return books + .map((book) => ({ ...book, title: book.title.trim() })) + .filter((book) => book.title.length > 0 && book.files.length > 0); +} + +function plural(count: number, noun: string): string { + return `${count} ${noun}${count === 1 ? '' : 's'}`; +} + +/** "2 books · 3 files · 2 files ignored" */ +export function describePackPlan(books: PackBook[], ignored: string[]): string { + const fileCount = books.reduce((sum, book) => sum + book.files.length, 0); + const parts = [plural(books.length, 'book'), plural(fileCount, 'file')]; + if (ignored.length > 0) { + parts.push(`${plural(ignored.length, 'file')} ignored`); + } + return parts.join(' · '); +} diff --git a/src/frontend/src/utils/releasePayload.ts b/src/frontend/src/utils/releasePayload.ts new file mode 100644 index 00000000..9d4874b9 --- /dev/null +++ b/src/frontend/src/utils/releasePayload.ts @@ -0,0 +1,55 @@ +import type { DownloadReleasePayload } from '../services/api'; +import type { Book, ContentType, PackBook, Release } from '../types'; + +export interface ReleaseDownloadOptions { + /** Ask post-processing to split the release into one book per subfolder/file. */ + multiBook?: boolean; + /** The split the user approved in the pack review panel. */ + bookPlan?: PackBook[]; +} + +/** Build the body for /api/releases/download (and /api/releases/inspect). */ +export function buildReleaseDownloadPayload( + book: Book, + release: Release, + releaseContentType: ContentType, + options: ReleaseDownloadOptions = {}, +): DownloadReleasePayload { + const isManual = book.provider === 'manual'; + const releasePreview = + typeof release.extra?.preview === 'string' ? release.extra.preview : undefined; + const releaseAuthor = + typeof release.extra?.author === 'string' ? release.extra.author : undefined; + + const payload: DownloadReleasePayload = { + source: release.source, + source_id: release.source_id, + title: isManual ? release.title : book.title, + author: isManual ? releaseAuthor || '' : book.author, + year: book.year, + format: release.format, + size: release.size, + size_bytes: release.size_bytes, + download_url: release.download_url, + protocol: release.protocol, + indexer: release.indexer, + seeders: release.seeders, + extra: release.extra, + preview: isManual ? releasePreview || undefined : book.preview, + content_type: releaseContentType, + series_name: book.series_name, + series_position: book.series_position, + subtitle: book.subtitle, + // From the release, never the book: book.language is the provider's + // canonical edition, which would mislabel a translated release. + language: release.language ?? undefined, + }; + + if (options.multiBook || options.bookPlan) { + payload.multi_book = true; + } + if (options.bookPlan) { + payload.book_plan = options.bookPlan; + } + return payload; +} diff --git a/tests/audiobookbay/conftest.py b/tests/audiobookbay/conftest.py new file mode 100644 index 00000000..615257ee --- /dev/null +++ b/tests/audiobookbay/conftest.py @@ -0,0 +1,13 @@ +"""AudiobookBay test fixtures.""" + +import pytest + +from shelfmark.release_sources.audiobookbay import scraper + + +@pytest.fixture(autouse=True) +def _clear_detail_page_cache(): + """Detail pages are cached briefly in production; tests must not share them.""" + scraper.clear_detail_page_cache() + yield + scraper.clear_detail_page_cache() diff --git a/tests/audiobookbay/test_file_list.py b/tests/audiobookbay/test_file_list.py new file mode 100644 index 00000000..f25290e6 --- /dev/null +++ b/tests/audiobookbay/test_file_list.py @@ -0,0 +1,102 @@ +"""Tests for reading the torrent file list off an AudiobookBay detail page.""" + +from shelfmark.download.postprocess.packs import PackFile +from shelfmark.release_sources.audiobookbay import scraper + +# Trimmed from a real detail page (2026-08): the file rows sit between the +# "Multifile Torrent" marker and the "Combined File Size" row. +MULTIFILE_DETAIL_HTML = """ +<table> +<tr><td>Tracker:</td><td>udp://tracker.torrent.eu.org:451/announce</td></tr> +<tr><td>Creation Date:</td><td>Sun, 29 Mar 2026 21:09:39 +0200</td></tr> +<tr><td colspan='2'>This is a Multifile Torrent</td></tr> +<tr><td colspan='2'>The Expanse 9.0 - Leviathan Falls (2021).m4b 1.05 GBs</td></tr> +<tr><td colspan='2'>The Expanse 0.1 - An Expanse Novella - Drive (2012).txt 340 Bytes</td></tr> +<tr><td colspan='2'>The Expanse 0.2 - An Expanse Novella - The Churn (2014).m4b 125.72 MBs</td></tr> +<tr><td colspan='2'>The Expanse 2.0 - Caliban’s War (2012).m4b 578.97 MBs</td></tr> +<tr><td>Combined File Size:</td><td><span style='color:#00f;'>7.87</span> GBs</td></tr> +<tr><td>Info Hash:</td><td>e4a5538e26987ee58a43aa629ec2c4f2b2d46526</td></tr> +</table> +""" + +SINGLE_FILE_DETAIL_HTML = """ +<table> +<tr><td>Creation Date:</td><td>Sun, 29 Mar 2026 21:09:39 +0200</td></tr> +<tr><td colspan='2'>Drive.m4b 41.55 MBs</td></tr> +<tr><td>Combined File Size:</td><td><span style='color:#00f;'>41.55</span> MBs</td></tr> +<tr><td>Info Hash:</td><td>e4a5538e26987ee58a43aa629ec2c4f2b2d46526</td></tr> +</table> +""" + + +def test_extracts_multifile_rows_with_byte_sizes(): + files = scraper.extract_file_list(MULTIFILE_DETAIL_HTML) + assert files == [ + PackFile("The Expanse 9.0 - Leviathan Falls (2021).m4b", int(1.05 * 1024**3)), + PackFile("The Expanse 0.1 - An Expanse Novella - Drive (2012).txt", 340), + PackFile( + "The Expanse 0.2 - An Expanse Novella - The Churn (2014).m4b", int(125.72 * 1024**2) + ), + PackFile("The Expanse 2.0 - Caliban’s War (2012).m4b", int(578.97 * 1024**2)), + ] + + +def test_single_file_torrent_lists_the_row_before_combined_size(): + assert scraper.extract_file_list(SINGLE_FILE_DETAIL_HTML) == [ + PackFile("Drive.m4b", int(41.55 * 1024**2)) + ] + + +def test_page_without_file_table_returns_none(): + assert scraper.extract_file_list("<html><body><p>nothing here</p></body></html>") is None + + +class TestHandlerListFiles: + def test_lists_files_from_detail_page(self): + from unittest.mock import patch + + from shelfmark.release_sources.audiobookbay.handler import AudiobookBayHandler + + with patch( + "shelfmark.release_sources.audiobookbay.handler.scraper.fetch_detail_html", + return_value=SINGLE_FILE_DETAIL_HTML, + ) as fetch: + files = AudiobookBayHandler().list_files( + {"source_id": "abc", "download_url": "https://audiobookbay.lu/abss/drive/"} + ) + assert files == [PackFile("Drive.m4b", int(41.55 * 1024**2))] + fetch.assert_called_once_with("https://audiobookbay.lu/abss/drive/", "audiobookbay.lu") + + def test_rejects_detail_url_on_other_host(self): + from unittest.mock import patch + + from shelfmark.release_sources.audiobookbay.handler import AudiobookBayHandler + + with patch( + "shelfmark.release_sources.audiobookbay.handler.scraper.fetch_detail_html" + ) as fetch: + files = AudiobookBayHandler().list_files( + {"source_id": "abc", "download_url": "https://evil.example/abss/drive/"} + ) + assert files is None + fetch.assert_not_called() + + +def test_extract_magnet_link_and_file_list_share_one_page_fetch(): + from unittest.mock import patch + + page = MULTIFILE_DETAIL_HTML + with ( + patch( + "shelfmark.release_sources.audiobookbay.scraper.downloader.html_get_page", + return_value=page, + ) as get_page, + patch("shelfmark.release_sources.audiobookbay.scraper._bootstrap_abb_session"), + ): + scraper.clear_detail_page_cache() + url = "https://audiobookbay.lu/abss/expanse/" + assert scraper.fetch_detail_html(url, "audiobookbay.lu") == page + magnet = scraper.extract_magnet_link(url, "audiobookbay.lu") + assert magnet is not None + assert "e4a5538e26987ee58a43aa629ec2c4f2b2d46526".upper() in magnet + assert get_page.call_count == 1 diff --git a/tests/core/test_processing_packs.py b/tests/core/test_processing_packs.py new file mode 100644 index 00000000..1710afd6 --- /dev/null +++ b/tests/core/test_processing_packs.py @@ -0,0 +1,212 @@ +"""Multi-book packs are filed one book at a time through the normal pipeline.""" + +import os +from pathlib import Path +from threading import Event +from unittest.mock import patch + +from shelfmark.core.models import DownloadTask, SearchMode +from tests.core.test_processing_integration import _build_config, _sync_config + + +def _run(temp_path: Path, task: DownloadTask, ingest: Path, staging: Path, **config_kwargs): + from shelfmark.download.postprocess.router import post_process_download + + statuses: list[tuple[str, str | None]] = [] + with ( + patch("shelfmark.core.config.config") as mock_config, + patch("shelfmark.config.env.TMP_DIR", staging), + ): + mock_config.get = _build_config( + ingest, + organization=config_kwargs.pop("organization", "organize"), + supported_audiobook_formats=["m4b", "mp3"], + audiobook_organize_template=config_kwargs.pop( + "audiobook_organize_template", "{Author}/{Title}/{Title}{ - PartNumber}" + ), + **config_kwargs, + ) + mock_config.CUSTOM_SCRIPT = None + _sync_config(mock_config, mock_config) + result = post_process_download( + temp_path, task, Event(), lambda s, m=None: statuses.append((s, m)) + ) + return result, statuses + + +def _nested_pack(root: Path) -> Path: + pack = root / "Sun Eater" + for folder, name in ( + ("Book 1 - Empire of Silence", "empire.m4b"), + ("Book 2 - Howling Dark", "howling.m4b"), + ): + (pack / folder).mkdir(parents=True) + (pack / folder / name).write_text(name) + (pack / "cover.jpg").write_text("img") + return pack + + +def _audiobook_task(**overrides) -> DownloadTask: + fields = { + "task_id": "pack-1", + "source": "direct_download", + "title": "Drive", + "author": "James S. A. Corey", + "content_type": "audiobook", + "series_name": "The Expanse", + "series_position": 2.6, + "search_mode": SearchMode.UNIVERSAL, + } + fields.update(overrides) + return DownloadTask(**fields) + + +def test_approved_plan_files_each_book_with_its_own_title(tmp_path): + staging = tmp_path / "staging" + ingest = tmp_path / "ingest" + staging.mkdir() + ingest.mkdir() + pack = staging / "Expanse" + pack.mkdir() + for name in ( + "The Expanse 1.0 - Leviathan Wakes (2011).m4b", + "The Expanse 2.0 - Caliban's War (2012).m4b", + ): + (pack / name).write_text(name) + (pack / "The Expanse 1.0 - Leviathan Wakes (2011).txt").write_text("notes") + + task = _audiobook_task( + title="Sun Eater", # the searched book; must not name the pack's books + book_plan=[ + { + "title": "Leviathan Wakes (edited)", + "series_position": 1.0, + "year": 2011, + "files": ["The Expanse 1.0 - Leviathan Wakes (2011).m4b"], + }, + { + "title": "Caliban's War", + "series_position": 2.0, + "year": 2012, + "files": ["The Expanse 2.0 - Caliban's War (2012).m4b"], + }, + ], + ) + + result, statuses = _run(pack, task, ingest, staging) + + assert result is not None + author_dir = ingest / "James S. A. Corey" + assert sorted(p.name for p in author_dir.iterdir()) == [ + "Caliban's War", + "Leviathan Wakes (edited)", + ] + assert (author_dir / "Leviathan Wakes (edited)" / "Leviathan Wakes (edited).m4b").exists() + assert (author_dir / "Caliban's War" / "Caliban's War.m4b").exists() + assert statuses[-1] == ("complete", "Complete (2 books, 2 files)") + + +def test_multi_book_flag_splits_nested_pack_heuristically(tmp_path): + staging = tmp_path / "staging" + ingest = tmp_path / "ingest" + staging.mkdir() + ingest.mkdir() + pack = _nested_pack(staging) + + result, _ = _run(pack, _audiobook_task(multi_book=True), ingest, staging) + + assert result is not None + author_dir = ingest / "James S. A. Corey" + assert (author_dir / "Empire of Silence" / "Empire of Silence.m4b").exists() + assert (author_dir / "Howling Dark" / "Howling Dark.m4b").exists() + + +def test_pack_book_series_position_does_not_leak_from_searched_book(tmp_path): + staging = tmp_path / "staging" + ingest = tmp_path / "ingest" + staging.mkdir() + ingest.mkdir() + pack = staging / "Two" + for folder in ("Alpha", "Beta"): + (pack / folder).mkdir(parents=True) + (pack / folder / f"{folder.lower()}.m4b").write_text(folder) + + result, _ = _run( + pack, + _audiobook_task(multi_book=True), + ingest, + staging, + audiobook_organize_template="{Author}/{SeriesPosition - }{Title}/{Title}", + ) + + assert result is not None + assert sorted(p.name for p in (ingest / "James S. A. Corey").iterdir()) == ["Alpha", "Beta"] + + +def test_multifile_book_inside_pack_keeps_part_numbers_per_book(tmp_path): + staging = tmp_path / "staging" + ingest = tmp_path / "ingest" + staging.mkdir() + ingest.mkdir() + pack = staging / "Pack" + (pack / "Book 1 - One").mkdir(parents=True) + (pack / "Book 2 - Two").mkdir(parents=True) + for i in (1, 2, 3): + (pack / "Book 1 - One" / f"part{i}.mp3").write_text(str(i)) + (pack / "Book 2 - Two" / "two.mp3").write_text("t") + + result, statuses = _run(pack, _audiobook_task(multi_book=True), ingest, staging) + + assert result is not None + one = ingest / "James S. A. Corey" / "One" + assert sorted(p.name for p in one.iterdir()) == ["One - 01.mp3", "One - 02.mp3", "One - 03.mp3"] + assert (ingest / "James S. A. Corey" / "Two" / "Two.mp3").exists() + assert statuses[-1] == ("complete", "Complete (2 books, 4 files)") + + +def test_hardlinked_torrent_pack_leaves_source_tree_intact(tmp_path): + downloads = tmp_path / "downloads" + ingest = tmp_path / "ingest" + downloads.mkdir() + ingest.mkdir() + pack = _nested_pack(downloads) + task = _audiobook_task(source="prowlarr", multi_book=True, original_download_path=str(pack)) + + result, _ = _run(pack, task, ingest, tmp_path / "staging", hardlink=True) + + assert result is not None + empire_src = pack / "Book 1 - Empire of Silence" / "empire.m4b" + empire_dst = ingest / "James S. A. Corey" / "Empire of Silence" / "Empire of Silence.m4b" + assert empire_src.exists() + assert empire_dst.exists() + assert os.stat(empire_src).st_ino == os.stat(empire_dst).st_ino + + +def test_without_pack_fields_nested_pack_is_still_one_book(tmp_path): + staging = tmp_path / "staging" + ingest = tmp_path / "ingest" + staging.mkdir() + ingest.mkdir() + pack = _nested_pack(staging) + + result, _ = _run(pack, _audiobook_task(), ingest, staging) + + assert result is not None + drive = ingest / "James S. A. Corey" / "Drive" + assert sorted(p.name for p in drive.iterdir()) == ["Drive - 01.m4b", "Drive - 02.m4b"] + + +def test_single_group_with_multi_book_flag_uses_searched_title(tmp_path): + staging = tmp_path / "staging" + ingest = tmp_path / "ingest" + staging.mkdir() + ingest.mkdir() + pack = staging / "Series" / "Book 1 - Solo" + pack.mkdir(parents=True) + (pack / "solo.m4b").write_text("s") + + result, statuses = _run(pack.parent, _audiobook_task(multi_book=True), ingest, staging) + + assert result is not None + assert (ingest / "James S. A. Corey" / "Drive" / "Drive.m4b").exists() + assert statuses[-1] == ("complete", "Complete") diff --git a/tests/core/test_release_inspect_routes.py b/tests/core/test_release_inspect_routes.py new file mode 100644 index 00000000..b8de7777 --- /dev/null +++ b/tests/core/test_release_inspect_routes.py @@ -0,0 +1,134 @@ +"""API tests for POST /api/releases/inspect.""" + +from __future__ import annotations + +import importlib +from unittest.mock import patch + +import pytest + +from shelfmark.download.postprocess.packs import PackFile + + +@pytest.fixture(scope="module") +def main_module(): + with patch("shelfmark.download.orchestrator.start"): + import shelfmark.main as main + + importlib.reload(main) + return main + + +@pytest.fixture +def client(main_module): + client = main_module.app.test_client() + with client.session_transaction() as sess: + sess["user_id"] = "tester" + sess["is_admin"] = False + return client + + +class _Handler: + def __init__(self, files): + self._files = files + + def list_files(self, release_data): + if isinstance(self._files, Exception): + raise self._files + return self._files + + +def _inspect(client, handler, payload=None): + body = { + "source": "audiobookbay", + "source_id": "abc", + "title": "Drive", + "content_type": "audiobook", + "series_name": "The Expanse", + **(payload or {}), + } + with patch("shelfmark.core.release_inspect_routes.get_handler", return_value=handler): + return client.post("/api/releases/inspect", json=body) + + +def test_pack_release_returns_a_plan(client): + files = [ + PackFile("The Expanse 1.0 - Leviathan Wakes (2011).m4b", 100), + PackFile("The Expanse 1.0 - Leviathan Wakes (2011).txt", 1), + PackFile("The Expanse 2.0 - Caliban's War (2012).m4b", 100), + ] + resp = _inspect(client, _Handler(files)) + assert resp.status_code == 200 + data = resp.get_json() + assert data["inspected"] is True + assert data["reason"] is None + assert data["files"] == [ + {"path": "The Expanse 1.0 - Leviathan Wakes (2011).m4b", "size": 100}, + {"path": "The Expanse 1.0 - Leviathan Wakes (2011).txt", "size": 1}, + {"path": "The Expanse 2.0 - Caliban's War (2012).m4b", "size": 100}, + ] + assert data["plan"]["is_pack"] is True + assert data["plan"]["ignored"] == ["The Expanse 1.0 - Leviathan Wakes (2011).txt"] + assert data["plan"]["books"] == [ + { + "title": "Leviathan Wakes", + "series_position": 1.0, + "year": 2011, + "files": ["The Expanse 1.0 - Leviathan Wakes (2011).m4b"], + }, + { + "title": "Caliban's War", + "series_position": 2.0, + "year": 2012, + "files": ["The Expanse 2.0 - Caliban's War (2012).m4b"], + }, + ] + + +def test_single_book_release_is_not_a_pack(client): + resp = _inspect(client, _Handler([PackFile("Drive.m4b", 5)])) + data = resp.get_json() + assert data["inspected"] is True + assert data["plan"]["is_pack"] is False + assert len(data["plan"]["books"]) == 1 + + +def test_handler_without_file_list_reports_not_inspected(client): + resp = _inspect(client, _Handler(None)) + assert resp.status_code == 200 + data = resp.get_json() + assert data["inspected"] is False + assert data["reason"] + assert data["plan"] is None + + +def test_handler_failure_reports_not_inspected_without_500(client): + resp = _inspect(client, _Handler(RuntimeError("boom"))) + assert resp.status_code == 200 + data = resp.get_json() + assert data["inspected"] is False + assert "boom" in data["reason"] + + +def test_unknown_source_is_a_client_error(client): + with patch( + "shelfmark.core.release_inspect_routes.get_handler", side_effect=ValueError("no source") + ): + resp = client.post( + "/api/releases/inspect", json={"source": "nope", "source_id": "x", "title": "t"} + ) + assert resp.status_code == 400 + + +def test_missing_source_id_is_a_client_error(client): + resp = client.post("/api/releases/inspect", json={"source": "audiobookbay"}) + assert resp.status_code == 400 + + +def test_requires_login(main_module): + anonymous = main_module.app.test_client() + with patch.object(main_module, "load_active_auth_mode", return_value="builtin"): + resp = anonymous.post( + "/api/releases/inspect", json={"source": "audiobookbay", "source_id": "a"} + ) + assert resp.status_code == 401 diff --git a/tests/download/test_orchestrator_pack_fields.py b/tests/download/test_orchestrator_pack_fields.py new file mode 100644 index 00000000..61e19f9e --- /dev/null +++ b/tests/download/test_orchestrator_pack_fields.py @@ -0,0 +1,62 @@ +"""Pack fields (multi_book / book_plan) survive queueing and restart-safe retry.""" + +from shelfmark.core.models import DownloadTask +from shelfmark.download import orchestrator + +PLAN = [ + {"title": "Leviathan Wakes", "series_position": 1.0, "year": 2011, "files": ["a.m4b"]}, + {"title": "Caliban's War", "series_position": 2.0, "year": 2012, "files": ["b.m4b"]}, +] + + +def test_task_defaults_to_single_book(): + task = DownloadTask(task_id="t", source="prowlarr", title="T") + assert task.multi_book is False + assert task.book_plan is None + + +def test_retry_payload_round_trips_pack_fields(): + task = DownloadTask(task_id="t", source="prowlarr", title="T", multi_book=True, book_plan=PLAN) + payload = orchestrator.serialize_task_for_retry(task) + restored = orchestrator._restore_task_from_retry_payload(payload) + assert restored is not None + assert restored.multi_book is True + assert restored.book_plan == PLAN + + +def test_retry_payload_drops_malformed_plan(): + payload = orchestrator.serialize_task_for_retry( + DownloadTask(task_id="t", source="prowlarr", title="T") + ) + payload["book_plan"] = "not a list" + restored = orchestrator._restore_task_from_retry_payload(payload) + assert restored is not None + assert restored.book_plan is None + + +def test_queue_release_reads_pack_fields(monkeypatch): + captured: dict[str, DownloadTask] = {} + + def fake_add(task: DownloadTask) -> bool: + captured["task"] = task + return True + + monkeypatch.setattr(orchestrator.config, "get", lambda _key, default=None, **_kw: default) + monkeypatch.setattr(orchestrator, "_source_unavailable_message", lambda _source: None) + monkeypatch.setattr(orchestrator.book_queue, "add", fake_add) + monkeypatch.setattr(orchestrator, "ws_manager", None) + + ok, error = orchestrator.queue_release( + { + "source": "direct_download", + "source_id": "abc", + "title": "The Expanse", + "content_type": "audiobook", + "multi_book": True, + "book_plan": PLAN, + }, + 0, + ) + assert ok, error + assert captured["task"].multi_book is True + assert captured["task"].book_plan == PLAN diff --git a/tests/download/test_packs.py b/tests/download/test_packs.py new file mode 100644 index 00000000..9e8eaaed --- /dev/null +++ b/tests/download/test_packs.py @@ -0,0 +1,282 @@ +"""Tests for multi-book pack planning (shelfmark.download.postprocess.packs).""" + +from pathlib import Path + +import pytest + +from shelfmark.download.postprocess.packs import ( + PackBook, + PackFile, + group_files_into_books, + match_plan_to_files, + parse_pack_book_name, + plan_pack, +) + +AUDIO = {"m4b", "mp3"} + + +class TestParsePackBookName: + @pytest.mark.parametrize( + ("name", "expected"), + [ + ("Book 3 - Howling Dark", ("Howling Dark", 3.0, None)), + ("Book 03: Howling Dark", ("Howling Dark", 3.0, None)), + ("03 - Empire of Silence", ("Empire of Silence", 3.0, None)), + ("2.5 - Interlude", ("Interlude", 2.5, None)), + ("[03] Empire of Silence", ("Empire of Silence", 3.0, None)), + ("#3 Empire of Silence", ("Empire of Silence", 3.0, None)), + ("3. Empire of Silence", ("Empire of Silence", 3.0, None)), + ("Empire of Silence", ("Empire of Silence", None, None)), + ("Empire of Silence (2018)", ("Empire of Silence", None, 2018)), + ], + ) + def test_strips_series_markers(self, name, expected): + assert parse_pack_book_name(name, series_name=None) == expected + + def test_strips_leading_series_name_and_trailing_year(self): + assert parse_pack_book_name( + "The Expanse 1.0 - Leviathan Wakes (2011)", series_name="The Expanse" + ) == ("Leviathan Wakes", 1.0, 2011) + + def test_series_name_match_is_case_insensitive(self): + assert parse_pack_book_name( + "the expanse 2.5 - Gods of Risk", series_name="The Expanse" + ) == ( + "Gods of Risk", + 2.5, + None, + ) + + @pytest.mark.parametrize( + "name", + [ + "The Expanse 0.2 - An Expanse Novella - The Churn (2014)", + "The Expanse 0.2 - The Expanse Novella - The Churn (2014)", + "The Expanse 0.2 - An Expanse Short Story - The Churn (2014)", + ], + ) + def test_strips_series_novella_label(self, name): + assert parse_pack_book_name(name, series_name="The Expanse") == ("The Churn", 0.2, 2014) + + @pytest.mark.parametrize( + ("name", "expected"), + [ + ("Uncrowned Cradle, Book 7", ("Uncrowned", 7.0, None)), + ("Reaper Cradle, Volume 10", ("Reaper", 10.0, None)), + ("Soulsmith Cradle, Book 2", ("Soulsmith", 2.0, None)), + ("Wintersteel - Cradle Book 8", ("Wintersteel", 8.0, None)), + ("Wintersteel (Cradle, Book 8)", ("Wintersteel", 8.0, None)), + # A trailing bare number is a chapter/part, never a series position. + ("Unsouled - 02", ("Unsouled - 02", None, None)), + ], + ) + def test_trailing_series_marker(self, name, expected): + assert parse_pack_book_name(name, series_name="Cradle") == expected + + def test_trailing_series_name_only_stripped_with_a_marker(self): + # "Stories from Cradle" is the title; nothing marks a position, so keep it. + assert parse_pack_book_name("Threshold: Stories from Cradle", series_name="Cradle") == ( + "Threshold: Stories from Cradle", + None, + None, + ) + + @pytest.mark.parametrize( + ("name", "expected"), + [ + # AudiobookBay renders "<folder> <file>" as one flat string. + ( + "Will Wight - Unsouled Cradle, Book 1 Will Wight - Unsouled Cradle, Book 1", + ("Unsouled", 1.0, None), + ), + ( + "Will Wight - Skysworn Cradle, Book 4 Skysworn Cradle, Book 4", + ("Skysworn", 4.0, None), + ), + ("Will Wight - Bloodline Cradle, Book 9", ("Bloodline", 9.0, None)), + ], + ) + def test_strips_author_prefix_and_glued_folder_name(self, name, expected): + assert ( + parse_pack_book_name(name, series_name="Cradle", author_name="Will Wight") == expected + ) + + @pytest.mark.parametrize( + ("name", "expected"), + [ + ("Gods of Risk 2.5 - Gods of Risk", ("Gods of Risk", 2.5, None)), + ("Cibola Burn 4 - Cibola Burn (2014)", ("Cibola Burn", 4.0, 2014)), + ("cibola burn 4 - Cibola Burn", ("Cibola Burn", 4.0, None)), + # Different text on each side is a real "Series N - Title" name, not a repeat. + ("Sun Eater 2 - Howling Dark", ("Sun Eater 2 - Howling Dark", None, None)), + ], + ) + def test_collapses_title_repeated_around_the_position(self, name, expected): + assert parse_pack_book_name(name, series_name=None) == expected + + def test_bare_numeric_title_is_left_alone(self): + assert parse_pack_book_name("1984", series_name=None) == ("1984", None, None) + + def test_marker_that_would_leave_nothing_is_left_alone(self): + assert parse_pack_book_name("Book 3", series_name=None) == ("Book 3", None, None) + + +class TestPlanPack: + def test_nested_subfolders_become_separate_books(self): + files = [ + PackFile("Sun Eater/Book 1 - Empire of Silence/Empire of Silence.m4b", 10), + PackFile("Sun Eater/Book 2 - Howling Dark/Howling Dark.m4b", 20), + PackFile("Sun Eater/cover.jpg", 1), + ] + plan = plan_pack(files, supported_extensions=AUDIO, series_name=None) + assert plan.is_pack + assert [b.title for b in plan.books] == ["Empire of Silence", "Howling Dark"] + assert [b.series_position for b in plan.books] == [1.0, 2.0] + assert plan.books[0].files == ["Sun Eater/Book 1 - Empire of Silence/Empire of Silence.m4b"] + assert plan.ignored == ["Sun Eater/cover.jpg"] + + def test_flat_pack_becomes_one_book_per_file_and_ignores_sidecars(self): + files = [ + PackFile("The Expanse 1.0 - Leviathan Wakes (2011).m4b", 100), + PackFile("The Expanse 1.0 - Leviathan Wakes (2011).txt", 1), + PackFile("The Expanse 2.0 - Caliban's War (2012).m4b", 100), + ] + plan = plan_pack(files, supported_extensions=AUDIO, series_name="The Expanse") + assert plan.is_pack + assert [(b.title, b.series_position, b.year) for b in plan.books] == [ + ("Leviathan Wakes", 1.0, 2011), + ("Caliban's War", 2.0, 2012), + ] + assert plan.ignored == ["The Expanse 1.0 - Leviathan Wakes (2011).txt"] + + def test_audiobookbay_flat_list_with_trailing_markers_is_a_pack(self): + # ABB's file table has no folder separators: "<folder> <file> <size>". + files = [ + PackFile( + "Will Wight - Unsouled Cradle, Book 1 Will Wight - Unsouled Cradle, Book 1.sfv", 1 + ), + PackFile( + "Will Wight - Unsouled Cradle, Book 1 Will Wight - Unsouled Cradle, Book 1.m4a", 9 + ), + PackFile("Will Wight - Skysworn Cradle, Book 4 Skysworn Cradle, Book 4.m4b", 9), + PackFile("Uncrowned Cradle, Book 7.m4b", 9), + PackFile( + "Will Wight - Reaper Cradle, Volume 10 Will Wight - Reaper Cradle, Volume 10.m4b", 9 + ), + ] + plan = plan_pack( + files, + supported_extensions={"m4a", "m4b"}, + series_name="Cradle", + author_name="Will Wight", + ) + assert plan.is_pack + assert [(b.title, b.series_position) for b in plan.books] == [ + ("Unsouled", 1.0), + ("Skysworn", 4.0), + ("Uncrowned", 7.0), + ("Reaper", 10.0), + ] + + def test_deeper_nesting_collapses_onto_book_folder(self): + files = [ + PackFile("Book 1/CD1/01.mp3"), + PackFile("Book 1/CD2/01.mp3"), + PackFile("Book 2/01.mp3"), + ] + plan = plan_pack(files, supported_extensions=AUDIO, series_name=None) + assert [b.files for b in plan.books] == [ + ["Book 1/CD1/01.mp3", "Book 1/CD2/01.mp3"], + ["Book 2/01.mp3"], + ] + + def test_single_wrapping_folder_is_not_a_book_boundary(self): + # A torrent named "Series" containing one multi-part book is a single book. + files = [PackFile("Series/Book 1/01.mp3"), PackFile("Series/Book 1/02.mp3")] + plan = plan_pack(files, supported_extensions=AUDIO, series_name=None) + assert not plan.is_pack + assert len(plan.books) == 1 + + def test_root_files_and_subfolders_coexist(self): + files = [PackFile("Novella.m4b"), PackFile("Book 1/a.m4b"), PackFile("Book 1/b.m4b")] + plan = plan_pack(files, supported_extensions=AUDIO, series_name=None) + assert [b.files for b in plan.books] == [["Novella.m4b"], ["Book 1/a.m4b", "Book 1/b.m4b"]] + + def test_single_file_is_not_a_pack(self): + plan = plan_pack([PackFile("Book.m4b")], supported_extensions=AUDIO, series_name=None) + assert not plan.is_pack + assert plan.books[0].title == "Book" + + def test_empty_input(self): + plan = plan_pack([], supported_extensions=AUDIO, series_name=None) + assert plan.books == [] + assert not plan.is_pack + + +class TestGroupFilesIntoBooks: + def test_groups_on_disk_files_by_top_level_folder(self, tmp_path: Path): + a = tmp_path / "Book 1 - A" / "a.m4b" + b = tmp_path / "Book 2 - B" / "b.m4b" + for f in (a, b): + f.parent.mkdir(parents=True) + f.write_bytes(b"x") + groups = group_files_into_books([a, b], series_name=None) + assert [(g.title, g.series_position, g.files) for g in groups] == [ + ("A", 1.0, [a]), + ("B", 2.0, [b]), + ] + + +class TestMatchPlanToFiles: + def test_matches_by_relative_path_then_basename(self, tmp_path: Path): + root = tmp_path / "staging" / "Sun Eater" + a = root / "Book 1 - A" / "a.m4b" + b = root / "Book 2 - B" / "b.m4b" + for f in (a, b): + f.parent.mkdir(parents=True) + f.write_bytes(b"x") + plan = [ + PackBook(title="Alpha", series_position=1.0, year=2001, files=["Book 1 - A/a.m4b"]), + PackBook(title="Beta", series_position=2.0, year=None, files=["b.m4b"]), + ] + groups = match_plan_to_files(plan, [a, b]) + assert [(g.title, g.series_position, g.year, g.files) for g in groups] == [ + ("Alpha", 1.0, 2001, [a]), + ("Beta", 2.0, None, [b]), + ] + + def test_matches_glued_plan_path_by_basename_suffix(self, tmp_path: Path): + # The plan came from ABB's "<folder> <file>" strings; on disk the file sits in a folder. + root = tmp_path / "Cradle - Will Wight Books 1-10" + a = root / "Will Wight - Skysworn Cradle, Book 4" / "Skysworn Cradle, Book 4.m4b" + b = root / "Uncrowned Cradle, Book 7.m4b" + for f in (a, b): + f.parent.mkdir(parents=True, exist_ok=True) + f.write_bytes(b"x") + plan = [ + PackBook( + title="Skysworn", + series_position=4.0, + year=None, + files=["Will Wight - Skysworn Cradle, Book 4 Skysworn Cradle, Book 4.m4b"], + ), + PackBook( + title="Uncrowned", + series_position=7.0, + year=None, + files=["Uncrowned Cradle, Book 7.m4b"], + ), + ] + groups = match_plan_to_files(plan, [a, b]) + assert [(g.title, g.files) for g in groups] == [("Skysworn", [a]), ("Uncrowned", [b])] + + def test_unmatched_files_fall_back_to_heuristic_groups(self, tmp_path: Path): + a = tmp_path / "Book 1 - A" / "a.m4b" + c = tmp_path / "Book 3 - C" / "c.m4b" + for f in (a, c): + f.parent.mkdir(parents=True) + f.write_bytes(b"x") + plan = [PackBook(title="Alpha", series_position=1.0, year=None, files=["Book 1 - A/a.m4b"])] + groups = match_plan_to_files(plan, [a, c]) + assert [(g.title, g.files) for g in groups] == [("Alpha", [a]), ("C", [c])] diff --git a/tests/prowlarr/test_torrent_file_list.py b/tests/prowlarr/test_torrent_file_list.py new file mode 100644 index 00000000..322f3f9f --- /dev/null +++ b/tests/prowlarr/test_torrent_file_list.py @@ -0,0 +1,98 @@ +"""Tests for listing the files inside a .torrent without downloading it.""" + +from shelfmark.download.clients.torrent_utils import ( + bencode_encode, + extract_file_list_from_torrent, +) +from shelfmark.download.postprocess.packs import PackFile + + +def _torrent(info: dict) -> bytes: + return bencode_encode({b"announce": b"http://t/announce", b"info": info}) + + +def test_multi_file_torrent_lists_release_relative_paths(): + data = _torrent( + { + b"name": b"Sun Eater", + b"piece length": 16384, + b"pieces": b"x" * 20, + b"files": [ + {b"length": 10, b"path": [b"Book 1 - Empire of Silence", b"empire.m4b"]}, + {b"length": 20, b"path": [b"Book 2 - Howling Dark", b"howling.m4b"]}, + {b"length": 1, b"path": [b"cover.jpg"]}, + ], + } + ) + assert extract_file_list_from_torrent(data) == [ + PackFile("Sun Eater/Book 1 - Empire of Silence/empire.m4b", 10), + PackFile("Sun Eater/Book 2 - Howling Dark/howling.m4b", 20), + PackFile("Sun Eater/cover.jpg", 1), + ] + + +def test_single_file_torrent_lists_its_one_file(): + data = _torrent( + {b"name": b"Book.m4b", b"length": 42, b"piece length": 16384, b"pieces": b"x" * 20} + ) + assert extract_file_list_from_torrent(data) == [PackFile("Book.m4b", 42)] + + +def test_unparseable_data_returns_none(): + assert extract_file_list_from_torrent(b"not a torrent") is None + + +class TestProwlarrHandlerListFiles: + def _handler(self): + from shelfmark.release_sources.prowlarr.handler import ProwlarrHandler + + return ProwlarrHandler() + + def test_lists_files_from_torrent_url(self): + from unittest.mock import patch + + from shelfmark.download.clients.torrent_utils import TorrentInfo + + data = _torrent( + {b"name": b"Book.m4b", b"length": 42, b"piece length": 16384, b"pieces": b"x" * 20} + ) + release = { + "protocol": "torrent", + "downloadUrl": "http://prowlarr/dl.torrent", + "magnetUrl": "magnet:?xt=urn:btih:abc", + "infoHash": "abc", + } + with ( + patch("shelfmark.release_sources.prowlarr.handler.get_release", return_value=release), + patch( + "shelfmark.release_sources.prowlarr.handler.extract_torrent_info", + return_value=TorrentInfo(info_hash="abc", torrent_data=data, is_magnet=False), + ) as extract, + ): + files = self._handler().list_files({"source_id": "rel-1"}) + assert files == [PackFile("Book.m4b", 42)] + extract.assert_called_once_with("http://prowlarr/dl.torrent", expected_hash="abc") + + def test_magnet_only_release_cannot_be_listed(self): + from unittest.mock import patch + + release = {"protocol": "torrent", "magnetUrl": "magnet:?xt=urn:btih:abc"} + with ( + patch("shelfmark.release_sources.prowlarr.handler.get_release", return_value=release), + patch("shelfmark.release_sources.prowlarr.handler.extract_torrent_info") as extract, + ): + assert self._handler().list_files({"source_id": "rel-1"}) is None + extract.assert_not_called() + + def test_usenet_release_cannot_be_listed(self): + from unittest.mock import patch + + release = {"protocol": "usenet", "downloadUrl": "http://prowlarr/dl.nzb"} + with patch("shelfmark.release_sources.prowlarr.handler.get_release", return_value=release): + assert self._handler().list_files({"source_id": "rel-1"}) is None + + def test_unknown_release_cannot_be_listed(self): + from unittest.mock import patch + + with patch("shelfmark.release_sources.prowlarr.handler.get_release", return_value=None): + assert self._handler().list_files({"source_id": "missing"}) is None