feat(packs): inspect multi-book releases and file each book separately (#1270)

## Multi-book packs: inspect a release before download and file each
book separately

Closes #576

### Problem

One queued release is always treated as one book. When a torrent is
actually a whole series
(`Series/Book 1 - Title/…`, or a flat folder of `Series 1.0 - Title.m4b`
files), post-processing
walks the whole tree, flattens every file into one list and renames them
`Title - 01…10` under the
searched book's `{Author}/{Title}`. Audiobookshelf then sees a single
10-file "book" and the user
has to re-file everything by hand.

### What this does

Most releases expose their file list *before* anything is downloaded, so
the split is decided up
front and approved by the user, then the download is fire-and-forget:

1. **Inspect** – clicking a release's download button now calls `POST
/api/releases/inspect`
first. A new optional `DownloadHandler.list_files(release_data)` hook
returns the release's
   files without downloading:
- **AudiobookBay** reads the torrent file table off the detail page it
already fetches (the
page is now cached for 120 s, so inspect + download cost ABB one
request).
- **Prowlarr** parses `info.files` from the `.torrent` it already
fetches (the existing 120 s
torrent-fetch cache is reused). Magnet-only and usenet releases report
"can't inspect".
   - Other sources default to `None`.
2. **Review** – if the plan contains more than one book, the Find
Releases modal swaps the list
for a review panel: one row per book with editable title / series
position / year, expandable
file lists, non-book sidecars (`.txt`, covers) shown as ignored, a
"Treat as a single book"
switch, and **Download N books**. Single-book releases queue
immediately, exactly as before.
3. **File** – the approved plan travels with the task
(`DownloadTask.book_plan`, retry-safe) and
post-processing files each book through the existing transfer code, one
book at a time
(`dataclasses.replace(task, title=…, series_position=…, year=…)`), so
organize/rename
templates, part numbering (now scoped per book), hardlinks, torrent
copy-preserve and usenet
   handling are unchanged. Status reads `Complete (N books, M files)`.
4. **Fallback** – when a release can't be inspected the user gets a
toast, and a small
"Multi-book pack" toggle in the modal header forces a heuristic split
(subfolder = book, or
   one book per file when the file names carry series positions).

Planning lives in `shelfmark/download/postprocess/packs.py` and is
shared by the inspect endpoint
and post-processing, so what the user approved is what gets filed. The
name parser strips
`Book 3 -`, `03 -`, `1.0 -`, `3.`, `[03]`, `#3`, a leading series name,
labels like
"An Expanse Novella -", repeated titles (`Gods of Risk 2.5 - Gods of
Risk`) and a trailing
`(Year)`; author and series name come from the book that was searched,
and the searched book's
own series position is never applied to its siblings.

### Files

- `shelfmark/download/postprocess/packs.py` (new) –
`PackFile/PackBook/PackPlan`, `plan_pack`,
`parse_pack_book_name`, `group_files_into_books`, `match_plan_to_files`
- `shelfmark/core/release_inspect_routes.py` (new) – `POST
/api/releases/inspect`
- `shelfmark/release_sources/__init__.py` – `DownloadHandler.list_files`
hook
- `shelfmark/release_sources/audiobookbay/{scraper,handler}.py` –
detail-page cache,
  `extract_file_list`, `list_files`
- `shelfmark/release_sources/prowlarr/handler.py`,
`download/clients/torrent_utils.py` –
  `extract_file_list_from_torrent`, `list_files`
- `shelfmark/core/models.py`, `download/orchestrator.py` – `multi_book`
/ `book_plan` fields,
  queue + retry serialization
- `shelfmark/download/postprocess/transfer.py`, `pipeline.py`,
`outputs/folder.py` – per-book
  transfer branch and status message
- `src/frontend`: `components/PackReviewPanel.tsx` (new),
`ReleaseModal.tsx`, `App.tsx`,
`services/api.ts`, `types/index.ts`, `utils/releasePayload.ts` (payload
builder moved out of
  `App.tsx`), `utils/packReview.ts`
- `docs/dev/release-sources-plugin-guide.md` – documents the
`list_files` hook

### Out of scope (follow-ups)

- Listing files from an NZB (Shelfmark already fetches the bytes; `<file
subject>` names are noisy)
- Inspecting magnet links via qBittorrent's files API after a paused add
- BookLore / email outputs (they ignore `book_plan`; noted in code)
- The combined ebook + audiobook flow

### Testing

**Automated** (`make checks`, `make python-test`, `make frontend-test`
all green; the only
failures on my machine are the pre-existing
`tests/config/test_entrypoint_permissions.py` cases,
which need bash ≥ 4 and fail identically on `main` under macOS bash
3.2):

- `tests/download/test_packs.py` – name parsing (markers, series name,
novella labels, repeated
titles, bare numeric titles like `1984`), nested / flat / mixed /
deeper-nested packs, single
wrapping folder not treated as a pack, plan-to-disk matching with
basename fallback
- `tests/core/test_processing_packs.py` – full `post_process_download`
runs on a real temp
filesystem: approved plan files each book under its own
`{Author}/{Title}`, heuristic split
of a nested pack, searched book's series position does not leak,
multi-file book inside a pack
keeps `- 01/- 02` per book, hardlinked torrent pack leaves the seeding
tree intact, no pack
fields ⇒ behaviour unchanged, single group degrades to the searched
title, status message
- `tests/core/test_release_inspect_routes.py` – plan response,
not-inspectable, handler errors
  never 500, unknown source / missing `source_id` ⇒ 400, login required
- `tests/audiobookbay/test_file_list.py` – file-table scraping from real
ABB markup (multi-file
and single-file pages), handler host validation, one page fetch shared
by magnet + file list
- `tests/prowlarr/test_torrent_file_list.py` – multi-file / single-file
`.torrent` parsing,
  handler behaviour for torrent URL vs magnet vs usenet vs cache miss
- `tests/download/test_orchestrator_pack_fields.py` – queue-time parsing
and retry round-trip
- Frontend: `releasePayload.test.ts`, `packReview.test.ts` (vitest)

**Manual, on a real deployment** (arm64 image built from this branch,
run as a side container
next to production with the same qBittorrent / Audiobookshelf setup,
`FILE_ORGANIZATION_AUDIOBOOK=organize`,
hardlinks on):

- AudiobookBay "The Expanse Complete 2.0" (7.87 GB, 36 files): clicking
download opened the review
panel in ~1 s showing **18 books · 18 files · 18 files ignored** (the
`.txt` sidecars), with
series positions 0.1–9.5 and years parsed from the file names; novella
labels stripped
("The Churn", "The Butcher of Anderson Station"). Editing a title in the
panel works.
Confirming queued one task; the magnet resolved from the cached page in
~30 ms; after the
download the task reported `Complete (18 books, 18 files)`, 18 hardlinks
landed as
`audiobooks/James S. A. Corey/<Title>/<Title>.m4b`, the torrent kept
seeding, and
Audiobookshelf scanned each folder as its own book (title, author,
embedded chapters).
- A second pack ("Expanse [01 - 9.5]", `Title N - Title` naming) was
inspected to verify the
  repeated-title rule and the Back button, without downloading.
- Single-book releases still queue immediately with no extra UI.
This commit is contained in:
Lance Marks
2026-08-27 00:40:08 -04:00
committed by GitHub
parent 02b7e9d958
commit f441b85da2
31 changed files with 2370 additions and 72 deletions
+1
View File
@@ -236,6 +236,7 @@ pyrightconfig.json
*.local.* *.local.*
AGENTS.md AGENTS.md
.claude/ .claude/
CLAUDE.md
.nvmrc .nvmrc
.playwright-mcp/ .playwright-mcp/
frontend-dist/ frontend-dist/
+21
View File
@@ -276,6 +276,27 @@ class DownloadHandler(ABC):
pass pass
``` ```
### Optional: Listing Files Before Download
Some releases bundle several books (a whole-series torrent). Shelfmark inspects a
release before queueing it so the user can review how it will be split into books.
Override `list_files` when your source can enumerate a release's files without
downloading it; the default returns `None`, which the UI reports as "can't inspect":
```python
from shelfmark.download.postprocess.packs import PackFile
def list_files(self, release_data: dict[str, Any]) -> list[PackFile] | None:
"""Return the release's files (release-relative paths + sizes), or None."""
torrent_bytes = ... # e.g. fetch the .torrent, or scrape the indexer's detail page
return extract_file_list_from_torrent(torrent_bytes) # from download.clients.torrent_utils
```
`release_data` is the same payload the frontend sends to `/api/releases/download`
(`source_id`, `download_url`, `content_type`, `series_name`, ...). Built-in examples:
Prowlarr parses the `.torrent` it already fetches (magnet-only releases return
`None`), and AudiobookBay reads the file table off its detail page.
### Download Method Parameters ### Download Method Parameters
| Parameter | Type | Description | | Parameter | Type | Description |
+6
View File
@@ -136,6 +136,12 @@ class DownloadTask:
default_factory=dict default_factory=dict
) # Per-output parameters (e.g. email recipient) ) # Per-output parameters (e.g. email recipient)
# Multi-book packs: one release holding several books. `book_plan` is the split the
# user approved before download (list of {title, series_position, year, files});
# `multi_book` asks post-processing to split heuristically when no plan exists.
multi_book: bool = False
book_plan: list[dict[str, Any]] | None = None
# User association (multi-user support) # User association (multi-user support)
user_id: int | None = None # DB user ID who queued this download user_id: int | None = None # DB user ID who queued this download
username: str | None = None # Username for {User} template variable username: str | None = None # Username for {User} template variable
+102
View File
@@ -0,0 +1,102 @@
"""Pre-download release inspection: list a release's files and plan a multi-book split."""
from __future__ import annotations
from typing import TYPE_CHECKING, Any
from flask import jsonify, request
from shelfmark.core.logger import setup_logger
from shelfmark.core.utils import is_audiobook
from shelfmark.download.postprocess.packs import PackFile, PackPlan, plan_pack
from shelfmark.download.postprocess.policy import (
get_supported_audiobook_formats,
get_supported_formats,
)
from shelfmark.release_sources import get_handler
if TYPE_CHECKING:
from collections.abc import Callable
from flask import Flask, Response
logger = setup_logger(__name__)
_INSPECT_ERRORS = (OSError, RuntimeError, ValueError, TypeError, KeyError, AttributeError)
NOT_INSPECTABLE_REASON = "This source cannot list the release's files before downloading"
def _serialize_plan(plan: PackPlan) -> dict[str, Any]:
return {
"is_pack": plan.is_pack,
"ignored": plan.ignored,
"books": [
{
"title": book.title,
"series_position": book.series_position,
"year": book.year,
"files": book.files,
}
for book in plan.books
],
}
def inspect_release(data: dict[str, Any]) -> dict[str, Any]:
"""Build the inspect response for a release payload (same shape as a download)."""
source = str(data["source"])
handler = get_handler(source)
try:
files: list[PackFile] | None = handler.list_files(data)
except _INSPECT_ERRORS as exc:
logger.warning(
"Could not list files for %s release %s: %s", source, data.get("source_id"), exc
)
return {"inspected": False, "reason": str(exc), "files": [], "plan": None}
if files is None:
return {"inspected": False, "reason": NOT_INSPECTABLE_REASON, "files": [], "plan": None}
content_type = data.get("content_type")
supported = (
get_supported_audiobook_formats()
if is_audiobook(content_type if isinstance(content_type, str) else None)
else get_supported_formats()
)
series_name = data.get("series_name")
author_name = data.get("author")
plan = plan_pack(
files,
supported_extensions=set(supported),
series_name=series_name if isinstance(series_name, str) else None,
author_name=author_name if isinstance(author_name, str) else None,
)
return {
"inspected": True,
"reason": None,
"files": [{"path": f.path, "size": f.size} for f in files],
"plan": _serialize_plan(plan),
}
def register_release_inspect_routes(
app: Flask,
login_required: Callable[..., Any],
) -> None:
"""Register POST /api/releases/inspect."""
@app.route("/api/releases/inspect", methods=["POST"])
@login_required
def api_inspect_release() -> Response | tuple[Response, int]:
data = request.get_json(silent=True)
if not isinstance(data, dict):
return jsonify({"error": "No data provided"}), 400
if not data.get("source_id"):
return jsonify({"error": "source_id is required"}), 400
if not data.get("source"):
return jsonify({"error": "source is required"}), 400
try:
get_handler(str(data["source"]))
except ValueError as exc:
return jsonify({"error": str(exc)}), 400
return jsonify(inspect_release(data))
@@ -17,6 +17,7 @@ from shelfmark.core.config import config
from shelfmark.core.logger import setup_logger from shelfmark.core.logger import setup_logger
from shelfmark.core.utils import normalize_http_url from shelfmark.core.utils import normalize_http_url
from shelfmark.download.network import get_ssl_verify from shelfmark.download.network import get_ssl_verify
from shelfmark.download.postprocess.packs import PackFile
logger = setup_logger(__name__) logger = setup_logger(__name__)
@@ -433,6 +434,56 @@ def extract_info_hash_from_torrent(torrent_data: bytes) -> str | None:
return None return None
def _decode_torrent_text(value: object) -> str | None:
if isinstance(value, bytes):
return value.decode("utf-8", errors="replace")
if isinstance(value, str):
return value
return None
def extract_file_list_from_torrent(torrent_data: bytes) -> list[PackFile] | None:
"""List the files a .torrent describes, release-relative, without downloading it.
Multi-file torrents nest every path under the torrent name (which becomes the
client's save folder); single-file torrents are just the named file.
"""
try:
decoded, _ = bencode_decode(torrent_data)
except _TORRENT_PARSE_ERRORS as e:
logger.debug("Failed to parse torrent file list: %s", e)
return None
if not isinstance(decoded, dict):
return None
info = decoded.get(b"info")
if not isinstance(info, dict):
return None
name = _decode_torrent_text(info.get(b"name")) or ""
raw_files = info.get(b"files")
if not isinstance(raw_files, list):
length = info.get(b"length")
if not name:
return None
return [PackFile(name, length if isinstance(length, int) else None)]
files: list[PackFile] = []
for entry in raw_files:
if not isinstance(entry, dict):
continue
raw_path = entry.get(b"path")
if not isinstance(raw_path, list):
continue
segments = [seg for seg in (_decode_torrent_text(part) for part in raw_path) if seg]
if not segments:
continue
if name:
segments.insert(0, name)
length = entry.get(b"length")
files.append(PackFile("/".join(segments), length if isinstance(length, int) else None))
return files
def extract_hash_from_magnet(magnet_url: str) -> str | None: def extract_hash_from_magnet(magnet_url: str) -> str | None:
"""Extract info_hash from a magnet URL.""" """Extract info_hash from a magnet URL."""
if not magnet_url.startswith("magnet:"): if not magnet_url.startswith("magnet:"):
+35
View File
@@ -265,6 +265,8 @@ def queue_release(
series_position = release_data.get("series_position") or extra.get("series_position") series_position = release_data.get("series_position") or extra.get("series_position")
subtitle = release_data.get("subtitle") or extra.get("subtitle") subtitle = release_data.get("subtitle") or extra.get("subtitle")
language = release_data.get("language") or extra.get("language") language = release_data.get("language") or extra.get("language")
multi_book = bool(release_data.get("multi_book") or extra.get("multi_book"))
book_plan = _normalize_book_plan(release_data.get("book_plan") or extra.get("book_plan"))
books_output_mode = ( books_output_mode = (
str(config.get("BOOKS_OUTPUT_MODE", "folder", user_id=user_id) or "folder") str(config.get("BOOKS_OUTPUT_MODE", "folder", user_id=user_id) or "folder")
@@ -300,6 +302,8 @@ def queue_release(
series_position=series_position, series_position=series_position,
subtitle=subtitle, subtitle=subtitle,
language=language, language=language,
multi_book=multi_book or book_plan is not None,
book_plan=book_plan,
search_mode=search_mode, search_mode=search_mode,
output_mode=output_mode, output_mode=output_mode,
output_args=output_args, output_args=output_args,
@@ -408,6 +412,33 @@ def can_retry_download_task(
return _has_staged_retry_source(task) return _has_staged_retry_source(task)
def _normalize_book_plan(value: object) -> list[dict[str, Any]] | None:
"""Keep only well-formed pack books: a title plus a non-empty list of file paths."""
if not isinstance(value, list):
return None
books: list[dict[str, Any]] = []
for entry in value:
if not isinstance(entry, dict):
continue
title = normalize_optional_text(entry.get("title"))
raw_files = entry.get("files")
if title is None or not isinstance(raw_files, list):
continue
files = [f for f in raw_files if isinstance(f, str) and f.strip()]
if not files:
continue
year = entry.get("year")
books.append(
{
"title": title,
"series_position": _optional_number(entry.get("series_position")),
"year": year if isinstance(year, int) and not isinstance(year, bool) else None,
"files": files,
}
)
return books or None
def serialize_task_for_retry(task: DownloadTask) -> dict[str, Any]: def serialize_task_for_retry(task: DownloadTask) -> dict[str, Any]:
"""Serialize the task state needed for restart-safe retries.""" """Serialize the task state needed for restart-safe retries."""
raw_search_mode = getattr(task, "search_mode", None) raw_search_mode = getattr(task, "search_mode", None)
@@ -437,6 +468,8 @@ def serialize_task_for_retry(task: DownloadTask) -> dict[str, Any]:
"subtitle": getattr(task, "subtitle", None), "subtitle": getattr(task, "subtitle", None),
"language": getattr(task, "language", None), "language": getattr(task, "language", None),
"search_mode": search_mode, "search_mode": search_mode,
"multi_book": bool(getattr(task, "multi_book", False)),
"book_plan": _normalize_book_plan(getattr(task, "book_plan", None)),
"output_mode": getattr(task, "output_mode", None), "output_mode": getattr(task, "output_mode", None),
"output_args": dict(raw_output_args) if isinstance(raw_output_args, dict) else {}, "output_args": dict(raw_output_args) if isinstance(raw_output_args, dict) else {},
"user_id": getattr(task, "user_id", None), "user_id": getattr(task, "user_id", None),
@@ -495,6 +528,8 @@ def _restore_task_from_retry_payload(payload: object) -> DownloadTask | None:
subtitle=normalize_optional_text(payload.get("subtitle")), subtitle=normalize_optional_text(payload.get("subtitle")),
language=normalize_optional_text(payload.get("language")), language=normalize_optional_text(payload.get("language")),
search_mode=search_mode, search_mode=search_mode,
multi_book=bool(payload.get("multi_book", False)),
book_plan=_normalize_book_plan(payload.get("book_plan")),
output_mode=normalize_optional_text(payload.get("output_mode")), output_mode=normalize_optional_text(payload.get("output_mode")),
output_args=dict(output_args) if isinstance(output_args, dict) else {}, output_args=dict(output_args) if isinstance(output_args, dict) else {},
user_id=normalize_positive_int(payload.get("user_id")), user_id=normalize_positive_int(payload.get("user_id")),
+10 -1
View File
@@ -105,6 +105,7 @@ def process_folder_output(
maybe_run_custom_script, maybe_run_custom_script,
prepare_output_files, prepare_output_files,
record_step, record_step,
resolve_book_groups,
transfer_book_files, transfer_book_files,
) )
@@ -260,7 +261,15 @@ def process_folder_output(
prepared.cleanup_paths, prepared.cleanup_paths,
) )
message = "Complete" if len(final_paths) == 1 else f"Complete ({len(final_paths)} files)" pack_groups = resolve_book_groups(
task, prepared.files, organization_mode=plan.organization_mode
)
if pack_groups is not None:
message = f"Complete ({len(pack_groups)} books, {len(final_paths)} files)"
elif len(final_paths) == 1:
message = "Complete"
else:
message = f"Complete ({len(final_paths)} files)"
status_callback("complete", message) status_callback("complete", message)
return str(final_paths[0]) return str(final_paths[0])
+406
View File
@@ -0,0 +1,406 @@
"""Multi-book ("pack") release planning.
A pack is one release that contains several books: a whole-series torrent with one
subfolder per book, or a flat folder of `Series 1.0 - Title.m4b` files. The same
planning rules serve pre-download inspection (the file list comes from the release
source) and post-processing (the file list comes from disk), so what the user
approved in the modal is what gets filed.
"""
from __future__ import annotations
import os
import re
from dataclasses import dataclass
from pathlib import Path, PurePosixPath
_YEAR_SUFFIX_RE = re.compile(r"\s*\(\s*(?P<year>\d{4})\s*\)\s*$")
_SERIES_MARKER_RE = re.compile(
r"""
^\s*
(?:
\[\s*\#?(?P<bracket>\d+(?:\.\d+)?)\s*\] # [03] / [#3]
| \#(?P<hash>\d+(?:\.\d+)?) # #3
| book\.?\s*(?P<book>\d+(?:\.\d+)?) # Book 3 / Book. 03
| (?P<plain>\d+(?:\.\d+)?)(?=[\s\-:.]) # 03 - / 1.0 - / 3.
)
\s*(?:[-:.]\s*)?
""",
re.IGNORECASE | re.VERBOSE,
)
_SEPARATOR_CHARS = " \t-_:."
# "Gods of Risk 2.5 - Gods of Risk": the title repeated on both sides of the position.
_REPEATED_TITLE_RE = re.compile(
r"^(?P<left>.+?)\s+(?P<position>\d+(?:\.\d+)?)\s*[-:\u2013]\s*(?P<right>.+)$"
)
_SERIES_LABEL_WORDS = r"(?:novella|novellas|short\s+story|short|story|novel)"
# "Uncrowned Cradle, Book 7" / "Reaper Cradle, Volume 10" / "Wintersteel (Cradle, Book 8)":
# an explicit word marks the position at the END of the name. A bare trailing number
# is deliberately not matched — "Title - 02" is a chapter, not a series position.
_TRAILING_MARKER_RE = re.compile(
r"""
[\s,\-:\u2013(]*
(?:book|volume|vol\.?)\s*\#?(?P<position>\d+(?:\.\d+)?)
\s*\)?\s*$
""",
re.IGNORECASE | re.VERBOSE,
)
# AudiobookBay renders a file inside a folder as "<folder> <file>" with no separator,
# so a pack row reads "Author - Title Series, Book 1 Title Series, Book 1".
_GLUED_FOLDER_RE = re.compile(
r"^(?P<prefix>.+?\s[-\u2013]\s)?(?P<core>.+?)\s+(?P=core)$", re.IGNORECASE
)
@dataclass(frozen=True)
class PackFile:
"""One file inside a release, path relative to the release root."""
path: str
size: int | None = None
@dataclass(frozen=True)
class PackBook:
"""One book split out of a pack, files as release-relative paths."""
title: str
series_position: float | None
year: int | None
files: list[str]
@dataclass(frozen=True)
class PackPlan:
books: list[PackBook]
ignored: list[str]
@property
def is_pack(self) -> bool:
return len(self.books) > 1
@dataclass(frozen=True)
class BookGroup:
"""One book's on-disk files, ready for transfer."""
title: str
series_position: float | None
year: int | None
files: list[Path]
def _strip_series_name(name: str, series_name: str | None) -> str:
if not series_name:
return name
prefix = series_name.strip()
if not prefix or not name.lower().startswith(prefix.lower()):
return name
remainder = name[len(prefix) :]
if remainder and remainder[0].isalnum():
return name
return remainder.lstrip(_SEPARATOR_CHARS)
def _strip_series_label(work: str, series_name: str | None) -> str:
"""Drop a leading "An <Series> Novella - " style label that some packs prepend."""
if not series_name:
return work
# "The Expanse" is labelled "An Expanse Novella", so match without the article.
core = re.sub(r"^(?:the|an?)\s+", "", series_name.strip(), flags=re.IGNORECASE)
if not core:
return work
pattern = re.compile(
rf"^(?:an?\s+|the\s+)?{re.escape(core)}\s+{_SERIES_LABEL_WORDS}\s*[-:\u2013]\s*",
re.IGNORECASE,
)
return pattern.sub("", work, count=1)
def _collapse_glued_folder(name: str) -> str:
match = _GLUED_FOLDER_RE.match(name)
if not match:
return name
prefix = match.group("prefix") or ""
core = match.group("core")
# "Author - X X" → "Author - X" (the folder carried the author, the file did not).
return (prefix + core).strip()
def _strip_author_name(name: str, author_name: str | None) -> str:
"""Drop a leading "Author - " (packs are often filed as `Author - Title`)."""
if not author_name:
return name
prefix = author_name.strip()
if not prefix or not name.lower().startswith(prefix.lower()):
return name
remainder = name[len(prefix) :]
stripped = remainder.lstrip(_SEPARATOR_CHARS + "\u2013")
if stripped == remainder: # no separator after the author: part of the title
return name
return stripped
def _strip_trailing_series_name(work: str, series_name: str | None) -> str:
"""Drop a trailing series name left behind by a trailing position marker."""
if not series_name:
return work
suffix = series_name.strip()
if not suffix or not work.lower().endswith(suffix.lower()):
return work
remainder = work[: -len(suffix)]
stripped = remainder.rstrip(_SEPARATOR_CHARS + ",(\u2013")
if not stripped or stripped == remainder:
return work
return stripped
def parse_pack_book_name(
name: str, *, series_name: str | None, author_name: str | None = None
) -> tuple[str, float | None, int | None]:
"""Split a book folder/file-stem name into (title, series position, year).
Strips a leading series name, a leading position marker (`Book 3 - `, `03 - `,
`1.0 - `, `3. `, `[03] `, `#3 `) and a trailing `(YYYY)`. Also understands a
trailing marker (`Title Series, Book 3`, `Title (Series, Volume 3)`), a leading
`Author - `, and AudiobookBay's glued `<folder> <file>` names. Returns the name
unchanged with no position/year when nothing would be left of the title.
"""
work = _collapse_glued_folder(name.strip())
work = _strip_author_name(work, author_name)
work = _strip_series_name(work, series_name)
year: int | None = None
year_match = _YEAR_SUFFIX_RE.search(work)
if year_match:
year = int(year_match.group("year"))
work = work[: year_match.start()]
position: float | None = None
repeated = _REPEATED_TITLE_RE.match(work.strip())
if (
repeated
and repeated.group("left").strip().lower() == repeated.group("right").strip().lower()
):
return repeated.group("right").strip(), float(repeated.group("position")), year
marker = _SERIES_MARKER_RE.match(work)
if marker:
raw = (
marker.group("bracket")
or marker.group("hash")
or marker.group("book")
or marker.group("plain")
)
position = float(raw)
work = work[marker.end() :]
else:
trailing = _TRAILING_MARKER_RE.search(work)
if trailing and trailing.start() > 0:
position = float(trailing.group("position"))
work = _strip_trailing_series_name(work[: trailing.start()], series_name)
work = _strip_series_label(work, series_name)
title = work.strip().strip(_SEPARATOR_CHARS).strip()
if not title:
return name, None, None
return title, position, year
def _book_from_name(
name: str, files: list[str], series_name: str | None, author_name: str | None = None
) -> PackBook:
title, position, year = parse_pack_book_name(
name, series_name=series_name, author_name=author_name
)
return PackBook(title=title, series_position=position, year=year, files=files)
def _common_root_parts(paths: list[PurePosixPath]) -> tuple[str, ...]:
parents = [p.parent.parts for p in paths]
common: list[str] = []
for parts in zip(*parents, strict=False):
if len(set(parts)) != 1:
break
common.append(parts[0])
return tuple(common)
def plan_pack(
files: list[PackFile],
*,
supported_extensions: set[str],
series_name: str | None,
author_name: str | None = None,
root_depth: int | None = None,
) -> PackPlan:
"""Group a release's file list into books.
Files in a subfolder (relative to the common root) group by that subfolder. Files
directly in the root split one-book-per-file only when at least two of them carry
a series position in their names; otherwise they are one book (a chaptered
audiobook, e.g. `01.mp3`, `02.mp3`). `root_depth` fixes how many leading path
components form the root instead of deriving it from the files' common parent.
"""
supported = {ext.lower().lstrip(".") for ext in supported_extensions}
book_files: list[PurePosixPath] = []
ignored: list[str] = []
for pack_file in files:
rel = PurePosixPath(pack_file.path.replace("\\", "/").lstrip("./"))
if rel.suffix.lower().lstrip(".") in supported:
book_files.append(rel)
else:
ignored.append(pack_file.path)
if not book_files:
return PackPlan(books=[], ignored=ignored)
root_parts = (
_common_root_parts(book_files) if root_depth is None else book_files[0].parts[:root_depth]
)
depth = len(root_parts)
root_files: list[PurePosixPath] = []
folders: dict[str, list[str]] = {}
for rel in book_files:
remainder = rel.parts[depth:]
if len(remainder) > 1:
folders.setdefault(remainder[0], []).append(str(rel))
else:
root_files.append(rel)
books: list[PackBook] = []
if root_files:
parsed = [
parse_pack_book_name(f.stem, series_name=series_name, author_name=author_name)
for f in root_files
]
positions = {p[1] for p in parsed if p[1] is not None}
if len(positions) >= 2:
books.extend(
PackBook(title=title, series_position=position, year=year, files=[str(f)])
for f, (title, position, year) in zip(root_files, parsed, strict=True)
)
elif len(root_files) == 1:
books.append(
_book_from_name(root_files[0].stem, [str(root_files[0])], series_name, author_name)
)
else:
group_name = root_parts[-1] if root_parts else ""
books.append(
_book_from_name(group_name, [str(f) for f in root_files], series_name, author_name)
)
books.extend(
_book_from_name(folder, paths, series_name, author_name)
for folder, paths in folders.items()
)
return PackPlan(books=books, ignored=ignored)
def _relative_paths(
book_files: list[Path], root: Path | None = None
) -> tuple[Path, dict[Path, str]]:
if root is None:
root = Path(os.path.commonpath([str(f.parent) for f in book_files]))
return root, {f: f.relative_to(root).as_posix() for f in book_files}
def group_files_into_books(
book_files: list[Path],
*,
series_name: str | None,
author_name: str | None = None,
root: Path | None = None,
) -> list[BookGroup]:
"""Heuristically split on-disk files into books (see `plan_pack`).
`root` pins the release root when grouping a subset of a larger file set.
"""
if not book_files:
return []
_root, rel_by_path = _relative_paths(book_files, root)
path_by_rel = {rel: path for path, rel in rel_by_path.items()}
extensions = {f.suffix.lower().lstrip(".") for f in book_files}
plan = plan_pack(
[PackFile(rel) for rel in rel_by_path.values()],
supported_extensions=extensions,
series_name=series_name,
author_name=author_name,
root_depth=None if root is None else 0,
)
return [
BookGroup(
title=book.title,
series_position=book.series_position,
year=book.year,
files=[path_by_rel[rel] for rel in book.files],
)
for book in plan.books
]
def match_plan_to_files(
plan: list[PackBook],
book_files: list[Path],
*,
series_name: str | None = None,
author_name: str | None = None,
) -> list[BookGroup]:
"""Apply an approved plan to on-disk files.
Files match by release-relative path first, then by basename (archive extraction
and client save paths can shift the root), then by the on-disk basename being a
suffix of the planned name (sources that glue folder and file names together).
Book files the plan does not mention fall back to heuristic grouping so nothing
is silently dropped.
"""
if not book_files:
return []
root, rel_by_path = _relative_paths(book_files)
by_rel = {rel: path for path, rel in rel_by_path.items()}
by_name: dict[str, list[Path]] = {}
for path in book_files:
by_name.setdefault(path.name, []).append(path)
claimed: set[Path] = set()
groups: list[BookGroup] = []
for book in plan:
matched: list[Path] = []
for wanted in book.files:
wanted_rel = wanted.replace("\\", "/").lstrip("./")
candidate = by_rel.get(wanted_rel)
if candidate is None:
candidates = [
p for p in by_name.get(PurePosixPath(wanted_rel).name, []) if p not in claimed
]
candidate = candidates[0] if candidates else None
if candidate is None:
wanted_name = PurePosixPath(wanted_rel).name.lower()
candidates = [
p
for p in book_files
if p not in claimed and wanted_name.endswith(p.name.lower())
]
candidate = candidates[0] if len(candidates) == 1 else None
if candidate is not None and candidate not in claimed:
claimed.add(candidate)
matched.append(candidate)
if matched:
groups.append(
BookGroup(
title=book.title,
series_position=book.series_position,
year=book.year,
files=matched,
)
)
unmatched = [p for p in book_files if p not in claimed]
if unmatched:
groups.extend(
group_files_into_books(
unmatched, series_name=series_name, author_name=author_name, root=root
)
)
return groups
@@ -40,6 +40,7 @@ from .transfer import (
build_metadata_dict, build_metadata_dict,
is_torrent_source, is_torrent_source,
process_directory, process_directory,
resolve_book_groups,
resolve_hardlink_source, resolve_hardlink_source,
should_hardlink, should_hardlink,
transfer_book_files, transfer_book_files,
@@ -80,6 +81,7 @@ __all__ = [
"process_directory", "process_directory",
"record_step", "record_step",
"resolve_custom_script_target", "resolve_custom_script_target",
"resolve_book_groups",
"resolve_hardlink_source", "resolve_hardlink_source",
"run_custom_script", "run_custom_script",
"safe_cleanup_path", "safe_cleanup_path",
+110
View File
@@ -2,6 +2,7 @@
from __future__ import annotations from __future__ import annotations
import dataclasses
import os import os
from pathlib import Path from pathlib import Path
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
@@ -26,6 +27,7 @@ from shelfmark.download.fs import (
) )
from shelfmark.download.postprocess.policy import get_file_organization, get_template from shelfmark.download.postprocess.policy import get_file_organization, get_template
from .packs import BookGroup, PackBook, group_files_into_books, match_plan_to_files
from .scan import collect_directory_files, scan_directory_tree from .scan import collect_directory_files, scan_directory_tree
from .types import TransferPlan from .types import TransferPlan
from .workspace import safe_cleanup_path from .workspace import safe_cleanup_path
@@ -196,6 +198,19 @@ def transfer_book_files(
is_audiobook = check_audiobook(task.content_type) is_audiobook = check_audiobook(task.content_type)
organization_mode = organization_mode or get_file_organization(is_audiobook=is_audiobook) organization_mode = organization_mode or get_file_organization(is_audiobook=is_audiobook)
groups = resolve_book_groups(task, book_files, organization_mode=organization_mode)
if groups is not None:
return _transfer_book_groups(
groups,
destination,
task,
use_hardlink=use_hardlink,
is_torrent=is_torrent,
preserve_source=preserve_source,
organization_mode=organization_mode,
)
max_attempts = _max_attempts_for_batch(len(book_files)) max_attempts = _max_attempts_for_batch(len(book_files))
final_paths: list[Path] = [] final_paths: list[Path] = []
@@ -299,6 +314,101 @@ def transfer_book_files(
return final_paths, None, op_counts return final_paths, None, op_counts
def resolve_book_groups(
task: DownloadTask,
book_files: list[Path],
*,
organization_mode: str,
) -> list[BookGroup] | None:
"""Split a multi-book pack into per-book groups, or None to file as one book.
An approved `book_plan` wins; a bare `multi_book` flag falls back to heuristic
grouping. Organization `none` keeps files as-is, and a split that yields a single
group is not a pack at all.
"""
if organization_mode == "none" or not (task.book_plan or task.multi_book):
return None
if task.book_plan:
plan = [
PackBook(
title=str(entry.get("title") or ""),
series_position=entry.get("series_position"),
year=entry.get("year"),
files=list(entry.get("files") or []),
)
for entry in task.book_plan
if isinstance(entry, dict)
]
groups = match_plan_to_files(
plan, book_files, series_name=task.series_name, author_name=task.author
)
else:
groups = group_files_into_books(
book_files, series_name=task.series_name, author_name=task.author
)
return groups if len(groups) > 1 else None
def _transfer_book_groups(
groups: list[BookGroup],
destination: Path,
task: DownloadTask,
*,
use_hardlink: bool,
is_torrent: bool,
preserve_source: bool,
organization_mode: str,
) -> tuple[list[Path], str | None, dict[str, int]]:
"""Transfer each book of a pack through the normal single-book path.
Each book gets an isolated task copy (the single-file path mutates `task.format`)
carrying its own title, position and year; the searched book's position must not
leak onto its siblings, while author and series name apply to all of them.
"""
all_paths: list[Path] = []
totals: dict[str, int] = {"hardlink": 0, "copy": 0, "move": 0}
errors: list[str] = []
for group in groups:
book_task = dataclasses.replace(
task,
title=group.title or task.title,
year=str(group.year) if group.year is not None else None,
subtitle=None,
series_position=group.series_position,
multi_book=False,
book_plan=None,
)
paths, error, op_counts = transfer_book_files(
group.files,
destination,
book_task,
use_hardlink=use_hardlink,
is_torrent=is_torrent,
preserve_source=preserve_source,
organization_mode=organization_mode,
source_root=group.files[0].parent,
)
for op, count in op_counts.items():
totals[op] = totals.get(op, 0) + count
if error:
errors.append(f"{group.title}: {error}")
logger.warning("Task %s: pack book %r failed: %s", task.task_id, group.title, error)
continue
all_paths.extend(paths)
if not all_paths:
return [], "; ".join(errors) or "No book files found", totals
if errors:
logger.warning(
"Task %s: pack filed with %d failed book(s): %s",
task.task_id,
len(errors),
"; ".join(errors),
)
return all_paths, None, totals
def process_directory( def process_directory(
directory: Path, directory: Path,
ingest_dir: Path, ingest_dir: Path,
+4
View File
@@ -62,6 +62,7 @@ from shelfmark.core.notifications import (
notify_user, notify_user,
) )
from shelfmark.core.prefix_middleware import PrefixMiddleware from shelfmark.core.prefix_middleware import PrefixMiddleware
from shelfmark.core.release_inspect_routes import register_release_inspect_routes
from shelfmark.core.request_helpers import ( from shelfmark.core.request_helpers import (
coerce_bool, coerce_bool,
emit_ws_event, emit_ws_event,
@@ -1024,6 +1025,9 @@ def _serialize_release(release: Release) -> dict:
return result return result
register_release_inspect_routes(app, login_required)
@app.route("/api/releases/download", methods=["POST"]) @app.route("/api/releases/download", methods=["POST"])
@login_required @login_required
def api_download_release() -> Response | tuple[Response, int]: def api_download_release() -> Response | tuple[Response, int]:
+9
View File
@@ -13,6 +13,7 @@ if TYPE_CHECKING:
from shelfmark.core.models import DownloadTask from shelfmark.core.models import DownloadTask
from shelfmark.core.search_plan import ReleaseSearchPlan from shelfmark.core.search_plan import ReleaseSearchPlan
from shelfmark.download.postprocess.packs import PackFile
from shelfmark.metadata_providers import BookMetadata from shelfmark.metadata_providers import BookMetadata
@@ -400,6 +401,14 @@ class DownloadHandler(ABC):
"""Return private queue-time fields needed for restart-safe retry.""" """Return private queue-time fields needed for restart-safe retry."""
return {} return {}
def list_files(self, release_data: dict[str, Any]) -> list[PackFile] | None:
"""Return the release's file list without downloading it.
Lets the UI review a multi-book pack before queueing. Return None when the
source cannot know the files ahead of time (magnet links, usenet, ...).
"""
return None
@abstractmethod @abstractmethod
def cancel(self, task_id: str) -> bool: def cancel(self, task_id: str) -> bool:
"""Cancel an in-progress download.""" """Cancel an in-progress download."""
@@ -1,6 +1,6 @@
"""AudiobookBay download handler - resolves magnet links and uses shared client lifecycle.""" """AudiobookBay download handler - resolves magnet links and uses shared client lifecycle."""
from typing import TYPE_CHECKING from typing import TYPE_CHECKING, Any
from urllib.parse import urlparse from urllib.parse import urlparse
from shelfmark.core.config import config from shelfmark.core.config import config
@@ -22,6 +22,7 @@ if TYPE_CHECKING:
from collections.abc import Callable from collections.abc import Callable
from shelfmark.core.models import DownloadTask from shelfmark.core.models import DownloadTask
from shelfmark.download.postprocess.packs import PackFile
logger = setup_logger(__name__) logger = setup_logger(__name__)
DEFAULT_ABB_HOSTNAME = "audiobookbay.lu" DEFAULT_ABB_HOSTNAME = "audiobookbay.lu"
@@ -68,6 +69,19 @@ class AudiobookBayHandler(ExternalClientHandler):
return task_id return task_id
return None return None
def list_files(self, release_data: dict[str, Any]) -> list[PackFile] | None:
"""Read the torrent's file list off the detail page, without downloading."""
raw_url = release_data.get("download_url") or release_data.get("source_url")
detail_url = raw_url.strip() if isinstance(raw_url, str) else ""
hostname = _resolve_allowed_detail_hostname()
if not detail_url or not _detail_url_matches_host(detail_url, hostname):
logger.debug("Cannot list files for AudiobookBay release without a valid detail URL")
return None
detail_html = scraper.fetch_detail_html(detail_url, hostname)
if not detail_html:
return None
return scraper.extract_file_list(detail_html)
def _get_client(self, protocol: str) -> DownloadClient | None: def _get_client(self, protocol: str) -> DownloadClient | None:
"""Compatibility shim so module-level patching still works in tests.""" """Compatibility shim so module-level patching still works in tests."""
return get_client(protocol) return get_client(protocol)
+107 -29
View File
@@ -2,6 +2,7 @@
import re import re
import time import time
from threading import Lock
from urllib.parse import quote, quote_plus from urllib.parse import quote, quote_plus
import requests import requests
@@ -10,6 +11,7 @@ from bs4 import BeautifulSoup
from shelfmark.core.config import config from shelfmark.core.config import config
from shelfmark.core.logger import setup_logger from shelfmark.core.logger import setup_logger
from shelfmark.download import http as downloader from shelfmark.download import http as downloader
from shelfmark.download.postprocess.packs import PackFile
from shelfmark.release_sources.audiobookbay.utils import normalize_search_punctuation from shelfmark.release_sources.audiobookbay.utils import normalize_search_punctuation
logger = setup_logger(__name__) logger = setup_logger(__name__)
@@ -32,6 +34,13 @@ FIRST_PAGE_SESSION_REFRESH_ATTEMPTS = 2
# Legacy search parameter used by older ABB flows # Legacy search parameter used by older ABB flows
LEGACY_CATEGORY_QUERY = "undefined%2Cundefined" LEGACY_CATEGORY_QUERY = "undefined%2Cundefined"
# Detail pages are fetched once and shared by inspection (file list) and download
# (magnet link) so a "review then download" round trip costs ABB a single request.
DETAIL_PAGE_CACHE_TTL_SECONDS = 120.0
DETAIL_PAGE_CACHE_MAX_ENTRIES = 8
_detail_page_cache: dict[str, tuple[float, str]] = {}
_detail_page_cache_lock = Lock()
# Precompiled patterns used while parsing result cards # Precompiled patterns used while parsing result cards
LANGUAGE_PATTERN = re.compile(r"Language:\s*([A-Za-z]+)") LANGUAGE_PATTERN = re.compile(r"Language:\s*([A-Za-z]+)")
POSTED_PATTERN = re.compile(r"Posted:\s*(\d+\s+[A-Za-z]+\s+\d{4})") POSTED_PATTERN = re.compile(r"Posted:\s*(\d+\s+[A-Za-z]+\s+\d{4})")
@@ -39,6 +48,11 @@ FORMAT_PATTERN = re.compile(r"Format:\s*([A-Za-z0-9]+)")
BITRATE_PATTERN = re.compile(r"Bitrate:\s*([\d]+\s*[A-Za-z/]+)") BITRATE_PATTERN = re.compile(r"Bitrate:\s*([\d]+\s*[A-Za-z/]+)")
SIZE_PATTERN = re.compile(r"File Size:\s*([\d.]+)\s*([A-Za-z]+)") SIZE_PATTERN = re.compile(r"File Size:\s*([\d.]+)\s*([A-Za-z]+)")
INFO_HASH_LABEL_PATTERN = re.compile(r"Info Hash", re.IGNORECASE) INFO_HASH_LABEL_PATTERN = re.compile(r"Info Hash", re.IGNORECASE)
FILE_ROW_SIZE_PATTERN = re.compile(
r"^(?P<name>.+?)\s+(?P<size>\d+(?:\.\d+)?)\s*(?P<unit>Bytes?|KBs?|MBs?|GBs?|TBs?)$",
re.IGNORECASE,
)
_FILE_SIZE_MULTIPLIERS = {"b": 1, "k": 1024, "m": 1024**2, "g": 1024**3, "t": 1024**4}
def _coerce_non_negative_float(value: object, default: float) -> float: def _coerce_non_negative_float(value: object, default: float) -> float:
@@ -348,6 +362,98 @@ def search_audiobookbay(
return results return results
def _get_cached_detail_page(details_url: str) -> str | None:
with _detail_page_cache_lock:
entry = _detail_page_cache.get(details_url)
if entry is None:
return None
fetched_at, html = entry
if time.monotonic() - fetched_at > DETAIL_PAGE_CACHE_TTL_SECONDS:
del _detail_page_cache[details_url]
return None
return html
def _store_cached_detail_page(details_url: str, html: str) -> None:
with _detail_page_cache_lock:
_detail_page_cache[details_url] = (time.monotonic(), html)
while len(_detail_page_cache) > DETAIL_PAGE_CACHE_MAX_ENTRIES:
oldest = min(_detail_page_cache, key=lambda key: _detail_page_cache[key][0])
del _detail_page_cache[oldest]
def clear_detail_page_cache() -> None:
"""Drop cached detail pages (used by tests)."""
with _detail_page_cache_lock:
_detail_page_cache.clear()
def _fetch_detail_page_once(details_url: str, hostname: str) -> str:
session = requests.Session()
_bootstrap_abb_session(hostname, session, DETAIL_PAGE_RETRY_ATTEMPTS)
return _coerce_markup_to_html(
downloader.html_get_page(
details_url,
retry=DETAIL_PAGE_RETRY_ATTEMPTS,
use_bypasser=False,
allow_bypasser_fallback=False,
success_delay=0,
session=session,
)
)
def fetch_detail_html(details_url: str, hostname: str = "audiobookbay.lu") -> str:
"""Fetch a detail page (one retry with a fresh session), cached briefly per URL."""
cached = _get_cached_detail_page(details_url)
if cached is not None:
logger.debug("Reusing recently fetched detail page: %s", details_url)
return cached
detail_html = _fetch_detail_page_once(details_url, hostname)
if not detail_html:
detail_html = _fetch_detail_page_once(details_url, hostname)
if detail_html:
_store_cached_detail_page(details_url, detail_html)
return detail_html
def _parse_file_row(text: str) -> PackFile | None:
match = FILE_ROW_SIZE_PATTERN.match(text.strip())
if not match:
return None
multiplier = _FILE_SIZE_MULTIPLIERS[match.group("unit")[0].lower()]
return PackFile(match.group("name"), int(float(match.group("size")) * multiplier))
def extract_file_list(detail_html: str) -> list[PackFile] | None:
"""Read the torrent file rows off a detail page.
ABB renders the torrent's file table as single-cell rows between the
"This is a Multifile Torrent" marker (absent for single-file torrents) and the
"Combined File Size" row. Returns None when the page has no such table.
"""
soup = BeautifulSoup(detail_html, "html.parser")
rows: list[PackFile] = []
for row in soup.find_all("tr"):
cells = row.find_all("td")
if not cells:
continue
label = cells[0].get_text(" ", strip=True)
if label.lower().startswith("combined file size"):
return rows or None
if len(cells) != 1:
rows = [] # a two-column metadata row means we're not in the file table yet
continue
text = cells[0].get_text(" ", strip=True)
if "multifile torrent" in text.lower():
rows = []
continue
parsed = _parse_file_row(text)
if parsed is not None:
rows.append(parsed)
return None
def extract_magnet_link(details_url: str, hostname: str = "audiobookbay.lu") -> str | None: def extract_magnet_link(details_url: str, hostname: str = "audiobookbay.lu") -> str | None:
"""Extract info hash and trackers from book detail page, then construct magnet link. """Extract info hash and trackers from book detail page, then construct magnet link.
@@ -360,35 +466,7 @@ def extract_magnet_link(details_url: str, hostname: str = "audiobookbay.lu") ->
""" """
try: try:
session = requests.Session() detail_html = fetch_detail_html(details_url, hostname)
_bootstrap_abb_session(hostname, session, DETAIL_PAGE_RETRY_ATTEMPTS)
# Fetch detail page
detail_html = _coerce_markup_to_html(
downloader.html_get_page(
details_url,
retry=DETAIL_PAGE_RETRY_ATTEMPTS,
use_bypasser=False,
allow_bypasser_fallback=False,
success_delay=0,
session=session,
)
)
if not detail_html:
session = requests.Session()
_bootstrap_abb_session(hostname, session, DETAIL_PAGE_RETRY_ATTEMPTS)
detail_html = _coerce_markup_to_html(
downloader.html_get_page(
details_url,
retry=DETAIL_PAGE_RETRY_ATTEMPTS,
use_bypasser=False,
allow_bypasser_fallback=False,
success_delay=0,
session=session,
)
)
if not detail_html: if not detail_html:
logger.warning("Failed to fetch details page") logger.warning("Failed to fetch details page")
return None return None
@@ -28,6 +28,10 @@ from shelfmark.download.clients.base_handler import (
DownloadRequest, DownloadRequest,
ExternalClientHandler, ExternalClientHandler,
) )
from shelfmark.download.clients.torrent_utils import (
extract_file_list_from_torrent,
extract_torrent_info,
)
from shelfmark.metadata_providers import BookMetadata from shelfmark.metadata_providers import BookMetadata
from shelfmark.release_sources import register_handler from shelfmark.release_sources import register_handler
from shelfmark.release_sources.prowlarr.api import IndexerSeedSettings, ProwlarrClient from shelfmark.release_sources.prowlarr.api import IndexerSeedSettings, ProwlarrClient
@@ -38,12 +42,14 @@ from shelfmark.release_sources.prowlarr.utils import (
coerce_int_like, coerce_int_like,
get_preferred_download_url, get_preferred_download_url,
get_protocol, get_protocol,
sanitize_download_url,
) )
if TYPE_CHECKING: if TYPE_CHECKING:
from collections.abc import Callable from collections.abc import Callable
from shelfmark.core.models import DownloadTask from shelfmark.core.models import DownloadTask
from shelfmark.download.postprocess.packs import PackFile
logger = setup_logger(__name__) logger = setup_logger(__name__)
@@ -127,6 +133,24 @@ class ProwlarrHandler(ExternalClientHandler):
return settings.get(indexer_id) return settings.get(indexer_id)
def list_files(self, release_data: dict[str, Any]) -> list[PackFile] | None:
"""List a cached torrent release's files from its .torrent, without downloading.
Magnet-only and usenet releases cannot be listed ahead of time.
"""
source_id = str(release_data.get("source_id") or "")
prowlarr_result = get_release(source_id) if source_id else None
if not prowlarr_result or get_protocol(prowlarr_result) != "torrent":
return None
download_url = sanitize_download_url(str(prowlarr_result.get("downloadUrl") or "").strip())
if not download_url or download_url.startswith("magnet:"):
return None
expected_hash = str(prowlarr_result.get("infoHash") or "").strip() or None
info = extract_torrent_info(download_url, expected_hash=expected_hash)
if not info.torrent_data:
return None
return extract_file_list_from_torrent(info.torrent_data)
def _get_client(self, protocol: str) -> DownloadClient | None: def _get_client(self, protocol: str) -> DownloadClient | None:
"""Compatibility shim so module-level patching still works in tests.""" """Compatibility shim so module-level patching still works in tests."""
return get_client(protocol) return get_client(protocol)
+8 -39
View File
@@ -58,7 +58,6 @@ import {
isApiResponseError, isApiResponseError,
updateSelfUser, updateSelfUser,
setBookTargetState, setBookTargetState,
type DownloadReleasePayload,
} from './services/api'; } from './services/api';
import type { import type {
Book, Book,
@@ -93,6 +92,7 @@ import { getEffectiveMetadataSort } from './utils/metadataSort';
import { isRecord } from './utils/objectHelpers'; import { isRecord } from './utils/objectHelpers';
import { policyTrace } from './utils/policyTrace'; import { policyTrace } from './utils/policyTrace';
import { buildQueryTargets, getDefaultQueryTargetKey } from './utils/queryTargets'; import { buildQueryTargets, getDefaultQueryTargetKey } from './utils/queryTargets';
import { buildReleaseDownloadPayload, type ReleaseDownloadOptions } from './utils/releasePayload';
import { applyRequestNoteToPayload } from './utils/requestConfirmation'; import { applyRequestNoteToPayload } from './utils/requestConfirmation';
import { bookFromRequestData } from './utils/requestFulfil'; import { bookFromRequestData } from './utils/requestFulfil';
import { import {
@@ -219,6 +219,7 @@ type PendingOnBehalfDownload =
release: Release; release: Release;
releaseContentType: ContentType; releaseContentType: ContentType;
actingAsUser: ActingAsUserSelection; actingAsUser: ActingAsUserSelection;
options?: ReleaseDownloadOptions;
} }
| { | {
type: 'combined'; type: 'combined';
@@ -1072,41 +1073,6 @@ function App() {
[], [],
); );
const buildReleaseDownloadPayload = useCallback(
(book: Book, release: Release, releaseContentType: ContentType): DownloadReleasePayload => {
const isManual = book.provider === 'manual';
const releasePreview =
typeof release.extra?.preview === 'string' ? release.extra.preview : undefined;
const releaseAuthor =
typeof release.extra?.author === 'string' ? release.extra.author : undefined;
return {
source: release.source,
source_id: release.source_id,
title: isManual ? release.title : book.title,
author: isManual ? releaseAuthor || '' : book.author,
year: book.year,
format: release.format,
size: release.size,
size_bytes: release.size_bytes,
download_url: release.download_url,
protocol: release.protocol,
indexer: release.indexer,
seeders: release.seeders,
extra: release.extra,
preview: isManual ? releasePreview || undefined : book.preview,
content_type: releaseContentType,
series_name: book.series_name,
series_position: book.series_position,
subtitle: book.subtitle,
// From the release, never the book: book.language is the provider's
// canonical edition, which would mislabel a translated release.
language: release.language ?? undefined,
};
},
[],
);
// When downloading a book while browsing a Hardcover list the user owns, // When downloading a book while browsing a Hardcover list the user owns,
// automatically remove it from that list (fire-and-forget). // automatically remove it from that list (fire-and-forget).
const searchFieldLabelsRef = useRef(searchFieldLabels); const searchFieldLabelsRef = useRef(searchFieldLabels);
@@ -1214,12 +1180,13 @@ function App() {
release: Release, release: Release,
releaseContentType: ContentType, releaseContentType: ContentType,
onBehalfOfUserId?: number, onBehalfOfUserId?: number,
options?: ReleaseDownloadOptions,
): Promise<void> => { ): Promise<void> => {
const requestStartedAtSeconds = Date.now() / 1000; const requestStartedAtSeconds = Date.now() / 1000;
try { try {
trackRelease(book.id, release.source_id); trackRelease(book.id, release.source_id);
await downloadRelease( await downloadRelease(
buildReleaseDownloadPayload(book, release, releaseContentType), buildReleaseDownloadPayload(book, release, releaseContentType, options),
onBehalfOfUserId, onBehalfOfUserId,
); );
await fetchStatus(); await fetchStatus();
@@ -1301,7 +1268,6 @@ function App() {
} }
}, },
[ [
buildReleaseDownloadPayload,
fetchStatus, fetchStatus,
openRequestConfirmation, openRequestConfirmation,
refreshRequestPolicy, refreshRequestPolicy,
@@ -1416,6 +1382,7 @@ function App() {
effectivePendingOnBehalfDownload.release, effectivePendingOnBehalfDownload.release,
effectivePendingOnBehalfDownload.releaseContentType, effectivePendingOnBehalfDownload.releaseContentType,
onBehalfOfUserId, onBehalfOfUserId,
effectivePendingOnBehalfDownload.options,
); );
} }
setPendingOnBehalfDownload(null); setPendingOnBehalfDownload(null);
@@ -1640,6 +1607,7 @@ function App() {
book: Book, book: Book,
release: Release, release: Release,
releaseContentType: ContentType, releaseContentType: ContentType,
options?: ReleaseDownloadOptions,
) => { ) => {
policyTrace('release.action:start', { policyTrace('release.action:start', {
bookId: book.id, bookId: book.id,
@@ -1655,11 +1623,12 @@ function App() {
release, release,
releaseContentType, releaseContentType,
actingAsUser: effectiveActingAsUser, actingAsUser: effectiveActingAsUser,
options,
}); });
return; return;
} }
await executeReleaseDownload(book, release, releaseContentType); await executeReleaseDownload(book, release, releaseContentType, undefined, options);
}; };
const handleReleaseRequest = useCallback( const handleReleaseRequest = useCallback(
@@ -0,0 +1,193 @@
import { useState } from 'react';
import type { PackBook, PackPlan, Release } from '../types';
import {
describePackPlan,
parseSeriesPositionInput,
toBookPlanPayload,
updateReviewBook,
} from '../utils/packReview';
import { ToggleSwitch } from './shared/ToggleSwitch';
interface PackReviewPanelProps {
release: Release;
plan: PackPlan;
books: PackBook[];
onChange: (books: PackBook[]) => void;
onBack: () => void;
/** `null` means "treat the whole release as one book". */
onConfirm: (books: PackBook[] | null) => Promise<void>;
isSubmitting: boolean;
}
const inputClassName =
'w-full rounded-md border border-(--border-muted) bg-(--bg) px-2 py-1 text-sm text-(--text) focus:border-emerald-500 focus:outline-none';
export const PackReviewPanel = ({
release,
plan,
books,
onChange,
onBack,
onConfirm,
isSubmitting,
}: PackReviewPanelProps) => {
const [singleBook, setSingleBook] = useState(false);
const [expandedFiles, setExpandedFiles] = useState<number | null>(null);
const [showIgnored, setShowIgnored] = useState(false);
const payloadBooks = toBookPlanPayload(books);
const canConfirm = !isSubmitting && (singleBook || payloadBooks.length > 0);
const confirmLabel = singleBook
? 'Download as one book'
: `Download ${payloadBooks.length} ${payloadBooks.length === 1 ? 'book' : 'books'}`;
return (
<div className="flex flex-col gap-4 px-5 py-4" data-testid="pack-review-panel">
<div>
<h3 className="text-base font-semibold text-(--text)">
This release contains several books
</h3>
<p className="mt-1 text-sm text-zinc-500 dark:text-zinc-400">
<span className="font-medium text-(--text)">{release.title}</span> ·{' '}
{describePackPlan(books, plan.ignored)}
</p>
<p className="mt-1 text-xs text-zinc-500 dark:text-zinc-400">
Each book below is filed separately with its own title. Fix any titles before downloading
the author and series come from the book you searched.
</p>
</div>
<div className="flex items-center justify-between rounded-lg border border-(--border-muted) px-3 py-2">
<div>
<p className="text-sm font-medium text-(--text)">Treat as a single book</p>
<p className="text-xs text-zinc-500 dark:text-zinc-400">
Use this if the split is wrong and the files are really one audiobook.
</p>
</div>
<ToggleSwitch
checked={singleBook}
onChange={setSingleBook}
color="emerald"
ariaLabel="Treat as a single book"
disabled={isSubmitting}
/>
</div>
<div
className={`flex flex-col divide-y divide-zinc-200/60 dark:divide-zinc-800/60 ${
singleBook ? 'pointer-events-none opacity-40' : ''
}`}
>
<div className="grid grid-cols-[minmax(0,1fr)_72px_72px_80px] gap-2 pb-1 text-xs font-medium tracking-wide text-zinc-500 uppercase dark:text-zinc-400">
<span>Title</span>
<span>Series #</span>
<span>Year</span>
<span className="text-right">Files</span>
</div>
{books.map((book, index) => (
<div key={book.files[0] ?? index} className="py-2">
<div className="grid grid-cols-[minmax(0,1fr)_72px_72px_80px] items-center gap-2">
<input
type="text"
value={book.title}
onChange={(e) =>
onChange(updateReviewBook(books, index, { title: e.target.value }))
}
aria-label={`Title for book ${index + 1}`}
className={inputClassName}
disabled={isSubmitting}
/>
<input
type="text"
inputMode="decimal"
value={book.series_position ?? ''}
onChange={(e) =>
onChange(
updateReviewBook(books, index, {
series_position: parseSeriesPositionInput(e.target.value),
}),
)
}
aria-label={`Series position for book ${index + 1}`}
className={inputClassName}
disabled={isSubmitting}
/>
<input
type="text"
inputMode="numeric"
value={book.year ?? ''}
onChange={(e) => {
const parsed = parseSeriesPositionInput(e.target.value);
onChange(
updateReviewBook(books, index, {
year: parsed === null ? null : Math.trunc(parsed),
}),
);
}}
aria-label={`Year for book ${index + 1}`}
className={inputClassName}
disabled={isSubmitting}
/>
<button
type="button"
onClick={() => setExpandedFiles(expandedFiles === index ? null : index)}
className="hover-surface rounded-md px-2 py-1 text-right text-sm text-zinc-500 transition-colors dark:text-zinc-400"
aria-expanded={expandedFiles === index}
>
{book.files.length} {book.files.length === 1 ? 'file' : 'files'}
</button>
</div>
{expandedFiles === index && (
<ul className="mt-2 max-h-40 overflow-y-auto rounded-md bg-(--bg-soft) px-3 py-2 font-mono text-xs break-all text-zinc-600 dark:text-zinc-300">
{book.files.map((file) => (
<li key={file}>{file}</li>
))}
</ul>
)}
</div>
))}
</div>
{plan.ignored.length > 0 && (
<div>
<button
type="button"
onClick={() => setShowIgnored(!showIgnored)}
className="text-xs text-zinc-500 underline-offset-2 hover:underline dark:text-zinc-400"
aria-expanded={showIgnored}
>
{plan.ignored.length} {plan.ignored.length === 1 ? 'file' : 'files'} ignored (not a book
format)
</button>
{showIgnored && (
<ul className="mt-2 max-h-32 overflow-y-auto rounded-md bg-(--bg-soft) px-3 py-2 font-mono text-xs break-all text-zinc-600 dark:text-zinc-300">
{plan.ignored.map((file) => (
<li key={file}>{file}</li>
))}
</ul>
)}
</div>
)}
<div className="flex items-center justify-end gap-3 border-t border-(--border-muted) pt-4">
<button
type="button"
onClick={onBack}
disabled={isSubmitting}
className="hover-surface rounded-lg px-3 py-1.5 text-sm font-medium text-(--text) transition-colors disabled:opacity-50"
>
&larr; Back
</button>
<button
type="button"
onClick={() => void onConfirm(singleBook ? null : payloadBooks)}
disabled={!canConfirm}
className="rounded-lg bg-emerald-600 px-4 py-1.5 text-sm font-medium text-white transition-colors hover:bg-emerald-700 disabled:cursor-not-allowed disabled:opacity-50"
>
{isSubmitting ? 'Queuing…' : confirmLabel}
</button>
</div>
</div>
);
};
+112 -2
View File
@@ -7,6 +7,7 @@ import { useReleaseSearchSession } from '../hooks/releaseModal/useReleaseSearchS
import { useTabIndicator } from '../hooks/ui/useTabIndicator'; import { useTabIndicator } from '../hooks/ui/useTabIndicator';
import { useBodyScrollLock } from '../hooks/useBodyScrollLock'; import { useBodyScrollLock } from '../hooks/useBodyScrollLock';
import { useEscapeKey } from '../hooks/useEscapeKey'; import { useEscapeKey } from '../hooks/useEscapeKey';
import { inspectRelease } from '../services/api';
import type { import type {
Book, Book,
Release, Release,
@@ -18,6 +19,8 @@ import type {
LeadingCellConfig, LeadingCellConfig,
ContentType, ContentType,
RequestPolicyMode, RequestPolicyMode,
PackBook,
PackPlan,
} from '../types'; } from '../types';
import { isMetadataBook } from '../types'; import { isMetadataBook } from '../types';
import { bookSupportsTargets } from '../utils/bookTargetLoader'; import { bookSupportsTargets } from '../utils/bookTargetLoader';
@@ -29,7 +32,9 @@ import {
buildLanguageNormalizer, buildLanguageNormalizer,
} from '../utils/languageFilters'; } from '../utils/languageFilters';
import { getNestedValue, toComparableText, toStringValue } from '../utils/objectHelpers'; import { getNestedValue, toComparableText, toStringValue } from '../utils/objectHelpers';
import { toBookPlanPayload } from '../utils/packReview';
import { getReleaseFormats } from '../utils/releaseFormats'; import { getReleaseFormats } from '../utils/releaseFormats';
import { buildReleaseDownloadPayload, type ReleaseDownloadOptions } from '../utils/releasePayload';
import { import {
getBookTitleCandidates, getBookTitleCandidates,
getBookAuthorCandidates, getBookAuthorCandidates,
@@ -50,6 +55,7 @@ import { BookTargetDropdown } from './BookTargetDropdown';
import { Dropdown } from './Dropdown'; import { Dropdown } from './Dropdown';
import { DropdownList } from './DropdownList'; import { DropdownList } from './DropdownList';
import { LanguageMultiSelect } from './LanguageMultiSelect'; import { LanguageMultiSelect } from './LanguageMultiSelect';
import { PackReviewPanel } from './PackReviewPanel';
import { ReleaseCell } from './ReleaseCell'; import { ReleaseCell } from './ReleaseCell';
// Combined mode configuration for the ReleaseModal // Combined mode configuration for the ReleaseModal
@@ -140,7 +146,12 @@ const DEFAULT_COLUMN_CONFIG: ReleaseColumnConfig = {
interface ReleaseModalProps { interface ReleaseModalProps {
book: Book | null; book: Book | null;
onClose: () => void; onClose: () => void;
onDownload: (book: Book, release: Release, contentType: ContentType) => Promise<void>; onDownload: (
book: Book,
release: Release,
contentType: ContentType,
options?: ReleaseDownloadOptions,
) => Promise<void>;
onRequestRelease?: (book: Book, release: Release, contentType: ContentType) => Promise<void>; onRequestRelease?: (book: Book, release: Release, contentType: ContentType) => Promise<void>;
onRequestBook?: (book: Book, contentType: ContentType) => Promise<void>; onRequestBook?: (book: Book, contentType: ContentType) => Promise<void>;
getPolicyModeForSource?: (source: string, contentType: ContentType) => RequestPolicyMode; getPolicyModeForSource?: (source: string, contentType: ContentType) => RequestPolicyMode;
@@ -762,6 +773,15 @@ const ReleaseModalSession = ({
: supportedFormats; : supportedFormats;
const [isRequestingBook, setIsRequestingBook] = useState(false); const [isRequestingBook, setIsRequestingBook] = useState(false);
const [selectedRelease, setSelectedRelease] = useState<Release | null>(null); const [selectedRelease, setSelectedRelease] = useState<Release | null>(null);
// Multi-book packs: `multiBook` is the manual header toggle (heuristic split for
// releases we can't inspect); `packReview` holds an inspected pack awaiting approval.
const [multiBook, setMultiBook] = useState(false);
const [packReview, setPackReview] = useState<{
release: Release;
plan: PackPlan;
books: PackBook[];
} | null>(null);
const [packSubmitting, setPackSubmitting] = useState(false);
const isCombinedMode = combinedMode != null; const isCombinedMode = combinedMode != null;
const combinedPhase = combinedMode?.phase ?? null; const combinedPhase = combinedMode?.phase ?? null;
const combinedStepLabel = combinedMode?.stepLabel ?? ''; const combinedStepLabel = combinedMode?.stepLabel ?? '';
@@ -1196,7 +1216,30 @@ const ReleaseModalSession = ({
const mode = getReleaseActionMode(release); const mode = getReleaseActionMode(release);
if (mode === 'download') { if (mode === 'download') {
await onDownload(book, release, contentType); // Look at the release's files before queueing so a whole-series pack can be
// reviewed and filed as separate books instead of one mangled item.
let inspected = false;
let plan: PackPlan | null = null;
try {
const inspection = await inspectRelease(
buildReleaseDownloadPayload(book, release, contentType),
);
inspected = inspection.inspected;
plan = inspection.plan;
} catch (error) {
console.error('Release inspection failed:', error);
}
if (inspected && plan?.is_pack) {
setPackReview({ release, plan, books: plan.books });
return;
}
if (!inspected && !multiBook) {
onShowToast?.(
"Couldn't check this release's files before download. If it contains several books, turn on the multi-book pack toggle first.",
'info',
);
}
await onDownload(book, release, contentType, multiBook ? { multiBook: true } : {});
handleClose(); handleClose();
return; return;
} }
@@ -1215,9 +1258,32 @@ const ReleaseModalSession = ({
onRequestRelease, onRequestRelease,
contentType, contentType,
handleClose, handleClose,
multiBook,
onShowToast,
], ],
); );
const handlePackConfirm = useCallback(
async (books: PackBook[] | null): Promise<void> => {
if (!book || !packReview) {
return;
}
setPackSubmitting(true);
try {
await onDownload(
book,
packReview.release,
contentType,
books ? { multiBook: true, bookPlan: toBookPlanPayload(books) } : {},
);
handleClose();
} finally {
setPackSubmitting(false);
}
},
[book, packReview, onDownload, contentType, handleClose],
);
const titleId = `release-modal-title-${book.id}`; const titleId = `release-modal-title-${book.id}`;
const providerDisplay = const providerDisplay =
book.provider_display_name || book.provider_display_name ||
@@ -1720,6 +1786,37 @@ const ReleaseModalSession = ({
</div> </div>
<div className="flex items-center gap-3 pr-1 pl-2"> <div className="flex items-center gap-3 pr-1 pl-2">
{/* Multi-book pack toggle (fallback for releases that can't be inspected) */}
{!isCombinedMode && (
<button
type="button"
onClick={() => setMultiBook((prev) => !prev)}
className={`hover-surface relative rounded-full p-2.5 text-zinc-500 transition-colors dark:text-zinc-400 ${
multiBook ? 'text-emerald-600 dark:text-emerald-400' : ''
}`}
aria-label="Multi-book pack"
aria-pressed={multiBook}
title="Multi-book pack: file each subfolder (or each file) as a separate book. Only needed when a release can't be inspected before download."
>
<svg
className="h-4 w-4"
fill="none"
stroke="currentColor"
viewBox="0 0 24 24"
strokeWidth={1.5}
>
<path
strokeLinecap="round"
strokeLinejoin="round"
d="M6.429 9.75 2.25 12l4.179 2.25m0-4.5 5.571 3 5.571-3m-11.142 0L2.25 7.5 12 2.25l9.75 5.25-4.179 2.25m0 0L21.75 12l-4.179 2.25m0 0 4.179 2.25L12 21.75 2.25 16.5l4.179-2.25m11.142 0-5.571 3-5.571-3"
/>
</svg>
{multiBook && (
<span className="absolute top-1 right-1 h-2 w-2 rounded-full bg-emerald-500" />
)}
</button>
)}
{/* Manual query button */} {/* Manual query button */}
<button <button
type="button" type="button"
@@ -2120,6 +2217,19 @@ const ReleaseModalSession = ({
{/* Release list content */} {/* Release list content */}
<div className="min-h-[200px]"> <div className="min-h-[200px]">
{(() => { {(() => {
if (packReview) {
return (
<PackReviewPanel
release={packReview.release}
plan={packReview.plan}
books={packReview.books}
onChange={(books) => setPackReview({ ...packReview, books })}
onBack={() => setPackReview(null)}
onConfirm={handlePackConfirm}
isSubmitting={packSubmitting}
/>
);
}
if (sourcesLoading) { if (sourcesLoading) {
return <ReleaseSkeleton />; return <ReleaseSkeleton />;
} }
+14
View File
@@ -12,6 +12,8 @@ import type {
RequestSubmissionResult, RequestSubmissionResult,
MetadataProvidersResponse, MetadataProvidersResponse,
MetadataSearchConfig, MetadataSearchConfig,
PackBook,
InspectReleaseResponse,
} from '../types'; } from '../types';
import type { import type {
ActionResult, ActionResult,
@@ -510,6 +512,18 @@ export type DownloadReleasePayload = {
language?: string; // Release language code, for the {Language} naming variable language?: string; // Release language code, for the {Language} naming variable
search_author?: string; search_author?: string;
search_mode?: 'direct' | 'universal'; search_mode?: 'direct' | 'universal';
multi_book?: boolean; // Split a multi-book pack into one book per subfolder/file
book_plan?: PackBook[]; // The split the user approved before download
};
/** Inspect a release's file list before download (same body as downloadRelease). */
export const inspectRelease = async (
release: DownloadReleasePayload,
): Promise<InspectReleaseResponse> => {
return fetchJSON<InspectReleaseResponse>(`${API_BASE}/releases/inspect`, {
method: 'POST',
body: JSON.stringify(release),
});
}; };
export const downloadRelease = async ( export const downloadRelease = async (
+53
View File
@@ -0,0 +1,53 @@
import { describe, it, expect } from 'vitest';
import type { PackBook } from '../types';
import {
describePackPlan,
parseSeriesPositionInput,
toBookPlanPayload,
updateReviewBook,
} from '../utils/packReview';
const books: PackBook[] = [
{ title: 'Leviathan Wakes', series_position: 1, year: 2011, files: ['a.m4b'] },
{ title: 'Calibans War', series_position: 2, year: 2012, files: ['b.m4b', 'b2.m4b'] },
];
describe('packReview.updateReviewBook', () => {
it('replaces one book without touching the others', () => {
const next = updateReviewBook(books, 1, { title: 'Calibans War (Unabridged)' });
expect(next[0]).toBe(books[0]);
expect(next[1]).toEqual({ ...books[1], title: 'Calibans War (Unabridged)' });
expect(books[1].title).toBe('Calibans War');
});
});
describe('packReview.parseSeriesPositionInput', () => {
it('accepts whole and fractional positions', () => {
expect(parseSeriesPositionInput('3')).toBe(3);
expect(parseSeriesPositionInput('2.5')).toBe(2.5);
});
it('treats blank or junk as no position', () => {
expect(parseSeriesPositionInput('')).toBeNull();
expect(parseSeriesPositionInput('abc')).toBeNull();
});
});
describe('packReview.toBookPlanPayload', () => {
it('trims titles, drops books without a title, and keeps file lists', () => {
const edited = updateReviewBook(books, 0, { title: ' ' });
expect(toBookPlanPayload(edited)).toEqual([
{ title: 'Calibans War', series_position: 2, year: 2012, files: ['b.m4b', 'b2.m4b'] },
]);
});
});
describe('packReview.describePackPlan', () => {
it('summarises books, files and ignored sidecars', () => {
expect(describePackPlan(books, ['a.txt', 'cover.jpg'])).toBe(
'2 books · 3 files · 2 files ignored',
);
expect(describePackPlan([books[0]], [])).toBe('1 book · 1 file');
});
});
@@ -0,0 +1,69 @@
import { describe, it, expect } from 'vitest';
import type { Book, Release } from '../types';
import { buildReleaseDownloadPayload } from '../utils/releasePayload';
const book: Book = {
id: 'hc-1',
title: 'Drive',
author: 'James S. A. Corey',
year: '2012',
preview: 'https://img/drive.jpg',
series_name: 'The Expanse',
series_position: 2.6,
subtitle: 'An Expanse Short Story',
provider: 'hardcover',
provider_id: 'hc-1',
source: 'direct_download',
};
const release: Release = {
source: 'audiobookbay',
source_id: 'abb-1',
title: 'James S. A. Corey - The Expanse Complete 2.0',
format: 'm4b',
language: 'en',
download_url: 'https://audiobookbay.lu/abss/expanse/',
};
describe('buildReleaseDownloadPayload', () => {
it('describes the searched book and the chosen release', () => {
const payload = buildReleaseDownloadPayload(book, release, 'audiobook');
expect(payload).toMatchObject({
source: 'audiobookbay',
source_id: 'abb-1',
title: 'Drive',
author: 'James S. A. Corey',
series_name: 'The Expanse',
series_position: 2.6,
language: 'en',
content_type: 'audiobook',
});
expect(payload.multi_book).toBeUndefined();
expect(payload.book_plan).toBeUndefined();
});
it('uses the release title and author for manual books', () => {
const manual: Book = { ...book, provider: 'manual', title: 'ignored' };
const withAuthor = { ...release, extra: { author: 'Release Author' } };
const payload = buildReleaseDownloadPayload(manual, withAuthor, 'audiobook');
expect(payload.title).toBe(release.title);
expect(payload.author).toBe('Release Author');
});
it('flags a manual multi-book pack', () => {
const payload = buildReleaseDownloadPayload(book, release, 'audiobook', { multiBook: true });
expect(payload.multi_book).toBe(true);
expect(payload.book_plan).toBeUndefined();
});
it('attaches the approved book plan', () => {
const plan = [{ title: 'Leviathan Wakes', series_position: 1, year: 2011, files: ['a.m4b'] }];
const payload = buildReleaseDownloadPayload(book, release, 'audiobook', {
multiBook: true,
bookPlan: plan,
});
expect(payload.multi_book).toBe(true);
expect(payload.book_plan).toEqual(plan);
});
});
+21
View File
@@ -460,6 +460,27 @@ export interface SourceSearchInfo {
} }
// Response from /api/releases endpoint // Response from /api/releases endpoint
/** One book split out of a multi-book pack release, files as release-relative paths. */
export interface PackBook {
title: string;
series_position: number | null;
year: number | null;
files: string[];
}
export interface PackPlan {
is_pack: boolean;
books: PackBook[];
ignored: string[];
}
export interface InspectReleaseResponse {
inspected: boolean;
reason: string | null;
files: { path: string; size: number | null }[];
plan: PackPlan | null;
}
export interface ReleasesResponse { export interface ReleasesResponse {
releases: Release[]; releases: Release[];
book: { book: {
+39
View File
@@ -0,0 +1,39 @@
import type { PackBook } from '../types';
/** Return a copy of `books` with one entry patched; the input is not mutated. */
export function updateReviewBook(
books: PackBook[],
index: number,
patch: Partial<PackBook>,
): PackBook[] {
return books.map((book, i) => (i === index ? { ...book, ...patch } : book));
}
/** Parse a series-position text field: "3" → 3, "2.5" → 2.5, blank/junk → null. */
export function parseSeriesPositionInput(value: string): number | null {
const trimmed = value.trim();
if (!trimmed) return null;
const parsed = Number(trimmed);
return Number.isFinite(parsed) ? parsed : null;
}
/** The plan sent with the download: trimmed titles, untitled books dropped. */
export function toBookPlanPayload(books: PackBook[]): PackBook[] {
return books
.map((book) => ({ ...book, title: book.title.trim() }))
.filter((book) => book.title.length > 0 && book.files.length > 0);
}
function plural(count: number, noun: string): string {
return `${count} ${noun}${count === 1 ? '' : 's'}`;
}
/** "2 books · 3 files · 2 files ignored" */
export function describePackPlan(books: PackBook[], ignored: string[]): string {
const fileCount = books.reduce((sum, book) => sum + book.files.length, 0);
const parts = [plural(books.length, 'book'), plural(fileCount, 'file')];
if (ignored.length > 0) {
parts.push(`${plural(ignored.length, 'file')} ignored`);
}
return parts.join(' · ');
}
+55
View File
@@ -0,0 +1,55 @@
import type { DownloadReleasePayload } from '../services/api';
import type { Book, ContentType, PackBook, Release } from '../types';
export interface ReleaseDownloadOptions {
/** Ask post-processing to split the release into one book per subfolder/file. */
multiBook?: boolean;
/** The split the user approved in the pack review panel. */
bookPlan?: PackBook[];
}
/** Build the body for /api/releases/download (and /api/releases/inspect). */
export function buildReleaseDownloadPayload(
book: Book,
release: Release,
releaseContentType: ContentType,
options: ReleaseDownloadOptions = {},
): DownloadReleasePayload {
const isManual = book.provider === 'manual';
const releasePreview =
typeof release.extra?.preview === 'string' ? release.extra.preview : undefined;
const releaseAuthor =
typeof release.extra?.author === 'string' ? release.extra.author : undefined;
const payload: DownloadReleasePayload = {
source: release.source,
source_id: release.source_id,
title: isManual ? release.title : book.title,
author: isManual ? releaseAuthor || '' : book.author,
year: book.year,
format: release.format,
size: release.size,
size_bytes: release.size_bytes,
download_url: release.download_url,
protocol: release.protocol,
indexer: release.indexer,
seeders: release.seeders,
extra: release.extra,
preview: isManual ? releasePreview || undefined : book.preview,
content_type: releaseContentType,
series_name: book.series_name,
series_position: book.series_position,
subtitle: book.subtitle,
// From the release, never the book: book.language is the provider's
// canonical edition, which would mislabel a translated release.
language: release.language ?? undefined,
};
if (options.multiBook || options.bookPlan) {
payload.multi_book = true;
}
if (options.bookPlan) {
payload.book_plan = options.bookPlan;
}
return payload;
}
+13
View File
@@ -0,0 +1,13 @@
"""AudiobookBay test fixtures."""
import pytest
from shelfmark.release_sources.audiobookbay import scraper
@pytest.fixture(autouse=True)
def _clear_detail_page_cache():
"""Detail pages are cached briefly in production; tests must not share them."""
scraper.clear_detail_page_cache()
yield
scraper.clear_detail_page_cache()
+102
View File
@@ -0,0 +1,102 @@
"""Tests for reading the torrent file list off an AudiobookBay detail page."""
from shelfmark.download.postprocess.packs import PackFile
from shelfmark.release_sources.audiobookbay import scraper
# Trimmed from a real detail page (2026-08): the file rows sit between the
# "Multifile Torrent" marker and the "Combined File Size" row.
MULTIFILE_DETAIL_HTML = """
<table>
<tr><td>Tracker:</td><td>udp://tracker.torrent.eu.org:451/announce</td></tr>
<tr><td>Creation Date:</td><td>Sun, 29 Mar 2026 21:09:39 +0200</td></tr>
<tr><td colspan='2'>This is a Multifile Torrent</td></tr>
<tr><td colspan='2'>The Expanse 9.0 - Leviathan Falls (2021).m4b 1.05 GBs</td></tr>
<tr><td colspan='2'>The Expanse 0.1 - An Expanse Novella - Drive (2012).txt 340 Bytes</td></tr>
<tr><td colspan='2'>The Expanse 0.2 - An Expanse Novella - The Churn (2014).m4b 125.72 MBs</td></tr>
<tr><td colspan='2'>The Expanse 2.0 - Calibans War (2012).m4b 578.97 MBs</td></tr>
<tr><td>Combined File Size:</td><td><span style='color:#00f;'>7.87</span> GBs</td></tr>
<tr><td>Info Hash:</td><td>e4a5538e26987ee58a43aa629ec2c4f2b2d46526</td></tr>
</table>
"""
SINGLE_FILE_DETAIL_HTML = """
<table>
<tr><td>Creation Date:</td><td>Sun, 29 Mar 2026 21:09:39 +0200</td></tr>
<tr><td colspan='2'>Drive.m4b 41.55 MBs</td></tr>
<tr><td>Combined File Size:</td><td><span style='color:#00f;'>41.55</span> MBs</td></tr>
<tr><td>Info Hash:</td><td>e4a5538e26987ee58a43aa629ec2c4f2b2d46526</td></tr>
</table>
"""
def test_extracts_multifile_rows_with_byte_sizes():
files = scraper.extract_file_list(MULTIFILE_DETAIL_HTML)
assert files == [
PackFile("The Expanse 9.0 - Leviathan Falls (2021).m4b", int(1.05 * 1024**3)),
PackFile("The Expanse 0.1 - An Expanse Novella - Drive (2012).txt", 340),
PackFile(
"The Expanse 0.2 - An Expanse Novella - The Churn (2014).m4b", int(125.72 * 1024**2)
),
PackFile("The Expanse 2.0 - Calibans War (2012).m4b", int(578.97 * 1024**2)),
]
def test_single_file_torrent_lists_the_row_before_combined_size():
assert scraper.extract_file_list(SINGLE_FILE_DETAIL_HTML) == [
PackFile("Drive.m4b", int(41.55 * 1024**2))
]
def test_page_without_file_table_returns_none():
assert scraper.extract_file_list("<html><body><p>nothing here</p></body></html>") is None
class TestHandlerListFiles:
def test_lists_files_from_detail_page(self):
from unittest.mock import patch
from shelfmark.release_sources.audiobookbay.handler import AudiobookBayHandler
with patch(
"shelfmark.release_sources.audiobookbay.handler.scraper.fetch_detail_html",
return_value=SINGLE_FILE_DETAIL_HTML,
) as fetch:
files = AudiobookBayHandler().list_files(
{"source_id": "abc", "download_url": "https://audiobookbay.lu/abss/drive/"}
)
assert files == [PackFile("Drive.m4b", int(41.55 * 1024**2))]
fetch.assert_called_once_with("https://audiobookbay.lu/abss/drive/", "audiobookbay.lu")
def test_rejects_detail_url_on_other_host(self):
from unittest.mock import patch
from shelfmark.release_sources.audiobookbay.handler import AudiobookBayHandler
with patch(
"shelfmark.release_sources.audiobookbay.handler.scraper.fetch_detail_html"
) as fetch:
files = AudiobookBayHandler().list_files(
{"source_id": "abc", "download_url": "https://evil.example/abss/drive/"}
)
assert files is None
fetch.assert_not_called()
def test_extract_magnet_link_and_file_list_share_one_page_fetch():
from unittest.mock import patch
page = MULTIFILE_DETAIL_HTML
with (
patch(
"shelfmark.release_sources.audiobookbay.scraper.downloader.html_get_page",
return_value=page,
) as get_page,
patch("shelfmark.release_sources.audiobookbay.scraper._bootstrap_abb_session"),
):
scraper.clear_detail_page_cache()
url = "https://audiobookbay.lu/abss/expanse/"
assert scraper.fetch_detail_html(url, "audiobookbay.lu") == page
magnet = scraper.extract_magnet_link(url, "audiobookbay.lu")
assert magnet is not None
assert "e4a5538e26987ee58a43aa629ec2c4f2b2d46526".upper() in magnet
assert get_page.call_count == 1
+212
View File
@@ -0,0 +1,212 @@
"""Multi-book packs are filed one book at a time through the normal pipeline."""
import os
from pathlib import Path
from threading import Event
from unittest.mock import patch
from shelfmark.core.models import DownloadTask, SearchMode
from tests.core.test_processing_integration import _build_config, _sync_config
def _run(temp_path: Path, task: DownloadTask, ingest: Path, staging: Path, **config_kwargs):
from shelfmark.download.postprocess.router import post_process_download
statuses: list[tuple[str, str | None]] = []
with (
patch("shelfmark.core.config.config") as mock_config,
patch("shelfmark.config.env.TMP_DIR", staging),
):
mock_config.get = _build_config(
ingest,
organization=config_kwargs.pop("organization", "organize"),
supported_audiobook_formats=["m4b", "mp3"],
audiobook_organize_template=config_kwargs.pop(
"audiobook_organize_template", "{Author}/{Title}/{Title}{ - PartNumber}"
),
**config_kwargs,
)
mock_config.CUSTOM_SCRIPT = None
_sync_config(mock_config, mock_config)
result = post_process_download(
temp_path, task, Event(), lambda s, m=None: statuses.append((s, m))
)
return result, statuses
def _nested_pack(root: Path) -> Path:
pack = root / "Sun Eater"
for folder, name in (
("Book 1 - Empire of Silence", "empire.m4b"),
("Book 2 - Howling Dark", "howling.m4b"),
):
(pack / folder).mkdir(parents=True)
(pack / folder / name).write_text(name)
(pack / "cover.jpg").write_text("img")
return pack
def _audiobook_task(**overrides) -> DownloadTask:
fields = {
"task_id": "pack-1",
"source": "direct_download",
"title": "Drive",
"author": "James S. A. Corey",
"content_type": "audiobook",
"series_name": "The Expanse",
"series_position": 2.6,
"search_mode": SearchMode.UNIVERSAL,
}
fields.update(overrides)
return DownloadTask(**fields)
def test_approved_plan_files_each_book_with_its_own_title(tmp_path):
staging = tmp_path / "staging"
ingest = tmp_path / "ingest"
staging.mkdir()
ingest.mkdir()
pack = staging / "Expanse"
pack.mkdir()
for name in (
"The Expanse 1.0 - Leviathan Wakes (2011).m4b",
"The Expanse 2.0 - Caliban's War (2012).m4b",
):
(pack / name).write_text(name)
(pack / "The Expanse 1.0 - Leviathan Wakes (2011).txt").write_text("notes")
task = _audiobook_task(
title="Sun Eater", # the searched book; must not name the pack's books
book_plan=[
{
"title": "Leviathan Wakes (edited)",
"series_position": 1.0,
"year": 2011,
"files": ["The Expanse 1.0 - Leviathan Wakes (2011).m4b"],
},
{
"title": "Caliban's War",
"series_position": 2.0,
"year": 2012,
"files": ["The Expanse 2.0 - Caliban's War (2012).m4b"],
},
],
)
result, statuses = _run(pack, task, ingest, staging)
assert result is not None
author_dir = ingest / "James S. A. Corey"
assert sorted(p.name for p in author_dir.iterdir()) == [
"Caliban's War",
"Leviathan Wakes (edited)",
]
assert (author_dir / "Leviathan Wakes (edited)" / "Leviathan Wakes (edited).m4b").exists()
assert (author_dir / "Caliban's War" / "Caliban's War.m4b").exists()
assert statuses[-1] == ("complete", "Complete (2 books, 2 files)")
def test_multi_book_flag_splits_nested_pack_heuristically(tmp_path):
staging = tmp_path / "staging"
ingest = tmp_path / "ingest"
staging.mkdir()
ingest.mkdir()
pack = _nested_pack(staging)
result, _ = _run(pack, _audiobook_task(multi_book=True), ingest, staging)
assert result is not None
author_dir = ingest / "James S. A. Corey"
assert (author_dir / "Empire of Silence" / "Empire of Silence.m4b").exists()
assert (author_dir / "Howling Dark" / "Howling Dark.m4b").exists()
def test_pack_book_series_position_does_not_leak_from_searched_book(tmp_path):
staging = tmp_path / "staging"
ingest = tmp_path / "ingest"
staging.mkdir()
ingest.mkdir()
pack = staging / "Two"
for folder in ("Alpha", "Beta"):
(pack / folder).mkdir(parents=True)
(pack / folder / f"{folder.lower()}.m4b").write_text(folder)
result, _ = _run(
pack,
_audiobook_task(multi_book=True),
ingest,
staging,
audiobook_organize_template="{Author}/{SeriesPosition - }{Title}/{Title}",
)
assert result is not None
assert sorted(p.name for p in (ingest / "James S. A. Corey").iterdir()) == ["Alpha", "Beta"]
def test_multifile_book_inside_pack_keeps_part_numbers_per_book(tmp_path):
staging = tmp_path / "staging"
ingest = tmp_path / "ingest"
staging.mkdir()
ingest.mkdir()
pack = staging / "Pack"
(pack / "Book 1 - One").mkdir(parents=True)
(pack / "Book 2 - Two").mkdir(parents=True)
for i in (1, 2, 3):
(pack / "Book 1 - One" / f"part{i}.mp3").write_text(str(i))
(pack / "Book 2 - Two" / "two.mp3").write_text("t")
result, statuses = _run(pack, _audiobook_task(multi_book=True), ingest, staging)
assert result is not None
one = ingest / "James S. A. Corey" / "One"
assert sorted(p.name for p in one.iterdir()) == ["One - 01.mp3", "One - 02.mp3", "One - 03.mp3"]
assert (ingest / "James S. A. Corey" / "Two" / "Two.mp3").exists()
assert statuses[-1] == ("complete", "Complete (2 books, 4 files)")
def test_hardlinked_torrent_pack_leaves_source_tree_intact(tmp_path):
downloads = tmp_path / "downloads"
ingest = tmp_path / "ingest"
downloads.mkdir()
ingest.mkdir()
pack = _nested_pack(downloads)
task = _audiobook_task(source="prowlarr", multi_book=True, original_download_path=str(pack))
result, _ = _run(pack, task, ingest, tmp_path / "staging", hardlink=True)
assert result is not None
empire_src = pack / "Book 1 - Empire of Silence" / "empire.m4b"
empire_dst = ingest / "James S. A. Corey" / "Empire of Silence" / "Empire of Silence.m4b"
assert empire_src.exists()
assert empire_dst.exists()
assert os.stat(empire_src).st_ino == os.stat(empire_dst).st_ino
def test_without_pack_fields_nested_pack_is_still_one_book(tmp_path):
staging = tmp_path / "staging"
ingest = tmp_path / "ingest"
staging.mkdir()
ingest.mkdir()
pack = _nested_pack(staging)
result, _ = _run(pack, _audiobook_task(), ingest, staging)
assert result is not None
drive = ingest / "James S. A. Corey" / "Drive"
assert sorted(p.name for p in drive.iterdir()) == ["Drive - 01.m4b", "Drive - 02.m4b"]
def test_single_group_with_multi_book_flag_uses_searched_title(tmp_path):
staging = tmp_path / "staging"
ingest = tmp_path / "ingest"
staging.mkdir()
ingest.mkdir()
pack = staging / "Series" / "Book 1 - Solo"
pack.mkdir(parents=True)
(pack / "solo.m4b").write_text("s")
result, statuses = _run(pack.parent, _audiobook_task(multi_book=True), ingest, staging)
assert result is not None
assert (ingest / "James S. A. Corey" / "Drive" / "Drive.m4b").exists()
assert statuses[-1] == ("complete", "Complete")
+134
View File
@@ -0,0 +1,134 @@
"""API tests for POST /api/releases/inspect."""
from __future__ import annotations
import importlib
from unittest.mock import patch
import pytest
from shelfmark.download.postprocess.packs import PackFile
@pytest.fixture(scope="module")
def main_module():
with patch("shelfmark.download.orchestrator.start"):
import shelfmark.main as main
importlib.reload(main)
return main
@pytest.fixture
def client(main_module):
client = main_module.app.test_client()
with client.session_transaction() as sess:
sess["user_id"] = "tester"
sess["is_admin"] = False
return client
class _Handler:
def __init__(self, files):
self._files = files
def list_files(self, release_data):
if isinstance(self._files, Exception):
raise self._files
return self._files
def _inspect(client, handler, payload=None):
body = {
"source": "audiobookbay",
"source_id": "abc",
"title": "Drive",
"content_type": "audiobook",
"series_name": "The Expanse",
**(payload or {}),
}
with patch("shelfmark.core.release_inspect_routes.get_handler", return_value=handler):
return client.post("/api/releases/inspect", json=body)
def test_pack_release_returns_a_plan(client):
files = [
PackFile("The Expanse 1.0 - Leviathan Wakes (2011).m4b", 100),
PackFile("The Expanse 1.0 - Leviathan Wakes (2011).txt", 1),
PackFile("The Expanse 2.0 - Caliban's War (2012).m4b", 100),
]
resp = _inspect(client, _Handler(files))
assert resp.status_code == 200
data = resp.get_json()
assert data["inspected"] is True
assert data["reason"] is None
assert data["files"] == [
{"path": "The Expanse 1.0 - Leviathan Wakes (2011).m4b", "size": 100},
{"path": "The Expanse 1.0 - Leviathan Wakes (2011).txt", "size": 1},
{"path": "The Expanse 2.0 - Caliban's War (2012).m4b", "size": 100},
]
assert data["plan"]["is_pack"] is True
assert data["plan"]["ignored"] == ["The Expanse 1.0 - Leviathan Wakes (2011).txt"]
assert data["plan"]["books"] == [
{
"title": "Leviathan Wakes",
"series_position": 1.0,
"year": 2011,
"files": ["The Expanse 1.0 - Leviathan Wakes (2011).m4b"],
},
{
"title": "Caliban's War",
"series_position": 2.0,
"year": 2012,
"files": ["The Expanse 2.0 - Caliban's War (2012).m4b"],
},
]
def test_single_book_release_is_not_a_pack(client):
resp = _inspect(client, _Handler([PackFile("Drive.m4b", 5)]))
data = resp.get_json()
assert data["inspected"] is True
assert data["plan"]["is_pack"] is False
assert len(data["plan"]["books"]) == 1
def test_handler_without_file_list_reports_not_inspected(client):
resp = _inspect(client, _Handler(None))
assert resp.status_code == 200
data = resp.get_json()
assert data["inspected"] is False
assert data["reason"]
assert data["plan"] is None
def test_handler_failure_reports_not_inspected_without_500(client):
resp = _inspect(client, _Handler(RuntimeError("boom")))
assert resp.status_code == 200
data = resp.get_json()
assert data["inspected"] is False
assert "boom" in data["reason"]
def test_unknown_source_is_a_client_error(client):
with patch(
"shelfmark.core.release_inspect_routes.get_handler", side_effect=ValueError("no source")
):
resp = client.post(
"/api/releases/inspect", json={"source": "nope", "source_id": "x", "title": "t"}
)
assert resp.status_code == 400
def test_missing_source_id_is_a_client_error(client):
resp = client.post("/api/releases/inspect", json={"source": "audiobookbay"})
assert resp.status_code == 400
def test_requires_login(main_module):
anonymous = main_module.app.test_client()
with patch.object(main_module, "load_active_auth_mode", return_value="builtin"):
resp = anonymous.post(
"/api/releases/inspect", json={"source": "audiobookbay", "source_id": "a"}
)
assert resp.status_code == 401
@@ -0,0 +1,62 @@
"""Pack fields (multi_book / book_plan) survive queueing and restart-safe retry."""
from shelfmark.core.models import DownloadTask
from shelfmark.download import orchestrator
PLAN = [
{"title": "Leviathan Wakes", "series_position": 1.0, "year": 2011, "files": ["a.m4b"]},
{"title": "Caliban's War", "series_position": 2.0, "year": 2012, "files": ["b.m4b"]},
]
def test_task_defaults_to_single_book():
task = DownloadTask(task_id="t", source="prowlarr", title="T")
assert task.multi_book is False
assert task.book_plan is None
def test_retry_payload_round_trips_pack_fields():
task = DownloadTask(task_id="t", source="prowlarr", title="T", multi_book=True, book_plan=PLAN)
payload = orchestrator.serialize_task_for_retry(task)
restored = orchestrator._restore_task_from_retry_payload(payload)
assert restored is not None
assert restored.multi_book is True
assert restored.book_plan == PLAN
def test_retry_payload_drops_malformed_plan():
payload = orchestrator.serialize_task_for_retry(
DownloadTask(task_id="t", source="prowlarr", title="T")
)
payload["book_plan"] = "not a list"
restored = orchestrator._restore_task_from_retry_payload(payload)
assert restored is not None
assert restored.book_plan is None
def test_queue_release_reads_pack_fields(monkeypatch):
captured: dict[str, DownloadTask] = {}
def fake_add(task: DownloadTask) -> bool:
captured["task"] = task
return True
monkeypatch.setattr(orchestrator.config, "get", lambda _key, default=None, **_kw: default)
monkeypatch.setattr(orchestrator, "_source_unavailable_message", lambda _source: None)
monkeypatch.setattr(orchestrator.book_queue, "add", fake_add)
monkeypatch.setattr(orchestrator, "ws_manager", None)
ok, error = orchestrator.queue_release(
{
"source": "direct_download",
"source_id": "abc",
"title": "The Expanse",
"content_type": "audiobook",
"multi_book": True,
"book_plan": PLAN,
},
0,
)
assert ok, error
assert captured["task"].multi_book is True
assert captured["task"].book_plan == PLAN
+282
View File
@@ -0,0 +1,282 @@
"""Tests for multi-book pack planning (shelfmark.download.postprocess.packs)."""
from pathlib import Path
import pytest
from shelfmark.download.postprocess.packs import (
PackBook,
PackFile,
group_files_into_books,
match_plan_to_files,
parse_pack_book_name,
plan_pack,
)
AUDIO = {"m4b", "mp3"}
class TestParsePackBookName:
@pytest.mark.parametrize(
("name", "expected"),
[
("Book 3 - Howling Dark", ("Howling Dark", 3.0, None)),
("Book 03: Howling Dark", ("Howling Dark", 3.0, None)),
("03 - Empire of Silence", ("Empire of Silence", 3.0, None)),
("2.5 - Interlude", ("Interlude", 2.5, None)),
("[03] Empire of Silence", ("Empire of Silence", 3.0, None)),
("#3 Empire of Silence", ("Empire of Silence", 3.0, None)),
("3. Empire of Silence", ("Empire of Silence", 3.0, None)),
("Empire of Silence", ("Empire of Silence", None, None)),
("Empire of Silence (2018)", ("Empire of Silence", None, 2018)),
],
)
def test_strips_series_markers(self, name, expected):
assert parse_pack_book_name(name, series_name=None) == expected
def test_strips_leading_series_name_and_trailing_year(self):
assert parse_pack_book_name(
"The Expanse 1.0 - Leviathan Wakes (2011)", series_name="The Expanse"
) == ("Leviathan Wakes", 1.0, 2011)
def test_series_name_match_is_case_insensitive(self):
assert parse_pack_book_name(
"the expanse 2.5 - Gods of Risk", series_name="The Expanse"
) == (
"Gods of Risk",
2.5,
None,
)
@pytest.mark.parametrize(
"name",
[
"The Expanse 0.2 - An Expanse Novella - The Churn (2014)",
"The Expanse 0.2 - The Expanse Novella - The Churn (2014)",
"The Expanse 0.2 - An Expanse Short Story - The Churn (2014)",
],
)
def test_strips_series_novella_label(self, name):
assert parse_pack_book_name(name, series_name="The Expanse") == ("The Churn", 0.2, 2014)
@pytest.mark.parametrize(
("name", "expected"),
[
("Uncrowned Cradle, Book 7", ("Uncrowned", 7.0, None)),
("Reaper Cradle, Volume 10", ("Reaper", 10.0, None)),
("Soulsmith Cradle, Book 2", ("Soulsmith", 2.0, None)),
("Wintersteel - Cradle Book 8", ("Wintersteel", 8.0, None)),
("Wintersteel (Cradle, Book 8)", ("Wintersteel", 8.0, None)),
# A trailing bare number is a chapter/part, never a series position.
("Unsouled - 02", ("Unsouled - 02", None, None)),
],
)
def test_trailing_series_marker(self, name, expected):
assert parse_pack_book_name(name, series_name="Cradle") == expected
def test_trailing_series_name_only_stripped_with_a_marker(self):
# "Stories from Cradle" is the title; nothing marks a position, so keep it.
assert parse_pack_book_name("Threshold: Stories from Cradle", series_name="Cradle") == (
"Threshold: Stories from Cradle",
None,
None,
)
@pytest.mark.parametrize(
("name", "expected"),
[
# AudiobookBay renders "<folder> <file>" as one flat string.
(
"Will Wight - Unsouled Cradle, Book 1 Will Wight - Unsouled Cradle, Book 1",
("Unsouled", 1.0, None),
),
(
"Will Wight - Skysworn Cradle, Book 4 Skysworn Cradle, Book 4",
("Skysworn", 4.0, None),
),
("Will Wight - Bloodline Cradle, Book 9", ("Bloodline", 9.0, None)),
],
)
def test_strips_author_prefix_and_glued_folder_name(self, name, expected):
assert (
parse_pack_book_name(name, series_name="Cradle", author_name="Will Wight") == expected
)
@pytest.mark.parametrize(
("name", "expected"),
[
("Gods of Risk 2.5 - Gods of Risk", ("Gods of Risk", 2.5, None)),
("Cibola Burn 4 - Cibola Burn (2014)", ("Cibola Burn", 4.0, 2014)),
("cibola burn 4 - Cibola Burn", ("Cibola Burn", 4.0, None)),
# Different text on each side is a real "Series N - Title" name, not a repeat.
("Sun Eater 2 - Howling Dark", ("Sun Eater 2 - Howling Dark", None, None)),
],
)
def test_collapses_title_repeated_around_the_position(self, name, expected):
assert parse_pack_book_name(name, series_name=None) == expected
def test_bare_numeric_title_is_left_alone(self):
assert parse_pack_book_name("1984", series_name=None) == ("1984", None, None)
def test_marker_that_would_leave_nothing_is_left_alone(self):
assert parse_pack_book_name("Book 3", series_name=None) == ("Book 3", None, None)
class TestPlanPack:
def test_nested_subfolders_become_separate_books(self):
files = [
PackFile("Sun Eater/Book 1 - Empire of Silence/Empire of Silence.m4b", 10),
PackFile("Sun Eater/Book 2 - Howling Dark/Howling Dark.m4b", 20),
PackFile("Sun Eater/cover.jpg", 1),
]
plan = plan_pack(files, supported_extensions=AUDIO, series_name=None)
assert plan.is_pack
assert [b.title for b in plan.books] == ["Empire of Silence", "Howling Dark"]
assert [b.series_position for b in plan.books] == [1.0, 2.0]
assert plan.books[0].files == ["Sun Eater/Book 1 - Empire of Silence/Empire of Silence.m4b"]
assert plan.ignored == ["Sun Eater/cover.jpg"]
def test_flat_pack_becomes_one_book_per_file_and_ignores_sidecars(self):
files = [
PackFile("The Expanse 1.0 - Leviathan Wakes (2011).m4b", 100),
PackFile("The Expanse 1.0 - Leviathan Wakes (2011).txt", 1),
PackFile("The Expanse 2.0 - Caliban's War (2012).m4b", 100),
]
plan = plan_pack(files, supported_extensions=AUDIO, series_name="The Expanse")
assert plan.is_pack
assert [(b.title, b.series_position, b.year) for b in plan.books] == [
("Leviathan Wakes", 1.0, 2011),
("Caliban's War", 2.0, 2012),
]
assert plan.ignored == ["The Expanse 1.0 - Leviathan Wakes (2011).txt"]
def test_audiobookbay_flat_list_with_trailing_markers_is_a_pack(self):
# ABB's file table has no folder separators: "<folder> <file> <size>".
files = [
PackFile(
"Will Wight - Unsouled Cradle, Book 1 Will Wight - Unsouled Cradle, Book 1.sfv", 1
),
PackFile(
"Will Wight - Unsouled Cradle, Book 1 Will Wight - Unsouled Cradle, Book 1.m4a", 9
),
PackFile("Will Wight - Skysworn Cradle, Book 4 Skysworn Cradle, Book 4.m4b", 9),
PackFile("Uncrowned Cradle, Book 7.m4b", 9),
PackFile(
"Will Wight - Reaper Cradle, Volume 10 Will Wight - Reaper Cradle, Volume 10.m4b", 9
),
]
plan = plan_pack(
files,
supported_extensions={"m4a", "m4b"},
series_name="Cradle",
author_name="Will Wight",
)
assert plan.is_pack
assert [(b.title, b.series_position) for b in plan.books] == [
("Unsouled", 1.0),
("Skysworn", 4.0),
("Uncrowned", 7.0),
("Reaper", 10.0),
]
def test_deeper_nesting_collapses_onto_book_folder(self):
files = [
PackFile("Book 1/CD1/01.mp3"),
PackFile("Book 1/CD2/01.mp3"),
PackFile("Book 2/01.mp3"),
]
plan = plan_pack(files, supported_extensions=AUDIO, series_name=None)
assert [b.files for b in plan.books] == [
["Book 1/CD1/01.mp3", "Book 1/CD2/01.mp3"],
["Book 2/01.mp3"],
]
def test_single_wrapping_folder_is_not_a_book_boundary(self):
# A torrent named "Series" containing one multi-part book is a single book.
files = [PackFile("Series/Book 1/01.mp3"), PackFile("Series/Book 1/02.mp3")]
plan = plan_pack(files, supported_extensions=AUDIO, series_name=None)
assert not plan.is_pack
assert len(plan.books) == 1
def test_root_files_and_subfolders_coexist(self):
files = [PackFile("Novella.m4b"), PackFile("Book 1/a.m4b"), PackFile("Book 1/b.m4b")]
plan = plan_pack(files, supported_extensions=AUDIO, series_name=None)
assert [b.files for b in plan.books] == [["Novella.m4b"], ["Book 1/a.m4b", "Book 1/b.m4b"]]
def test_single_file_is_not_a_pack(self):
plan = plan_pack([PackFile("Book.m4b")], supported_extensions=AUDIO, series_name=None)
assert not plan.is_pack
assert plan.books[0].title == "Book"
def test_empty_input(self):
plan = plan_pack([], supported_extensions=AUDIO, series_name=None)
assert plan.books == []
assert not plan.is_pack
class TestGroupFilesIntoBooks:
def test_groups_on_disk_files_by_top_level_folder(self, tmp_path: Path):
a = tmp_path / "Book 1 - A" / "a.m4b"
b = tmp_path / "Book 2 - B" / "b.m4b"
for f in (a, b):
f.parent.mkdir(parents=True)
f.write_bytes(b"x")
groups = group_files_into_books([a, b], series_name=None)
assert [(g.title, g.series_position, g.files) for g in groups] == [
("A", 1.0, [a]),
("B", 2.0, [b]),
]
class TestMatchPlanToFiles:
def test_matches_by_relative_path_then_basename(self, tmp_path: Path):
root = tmp_path / "staging" / "Sun Eater"
a = root / "Book 1 - A" / "a.m4b"
b = root / "Book 2 - B" / "b.m4b"
for f in (a, b):
f.parent.mkdir(parents=True)
f.write_bytes(b"x")
plan = [
PackBook(title="Alpha", series_position=1.0, year=2001, files=["Book 1 - A/a.m4b"]),
PackBook(title="Beta", series_position=2.0, year=None, files=["b.m4b"]),
]
groups = match_plan_to_files(plan, [a, b])
assert [(g.title, g.series_position, g.year, g.files) for g in groups] == [
("Alpha", 1.0, 2001, [a]),
("Beta", 2.0, None, [b]),
]
def test_matches_glued_plan_path_by_basename_suffix(self, tmp_path: Path):
# The plan came from ABB's "<folder> <file>" strings; on disk the file sits in a folder.
root = tmp_path / "Cradle - Will Wight Books 1-10"
a = root / "Will Wight - Skysworn Cradle, Book 4" / "Skysworn Cradle, Book 4.m4b"
b = root / "Uncrowned Cradle, Book 7.m4b"
for f in (a, b):
f.parent.mkdir(parents=True, exist_ok=True)
f.write_bytes(b"x")
plan = [
PackBook(
title="Skysworn",
series_position=4.0,
year=None,
files=["Will Wight - Skysworn Cradle, Book 4 Skysworn Cradle, Book 4.m4b"],
),
PackBook(
title="Uncrowned",
series_position=7.0,
year=None,
files=["Uncrowned Cradle, Book 7.m4b"],
),
]
groups = match_plan_to_files(plan, [a, b])
assert [(g.title, g.files) for g in groups] == [("Skysworn", [a]), ("Uncrowned", [b])]
def test_unmatched_files_fall_back_to_heuristic_groups(self, tmp_path: Path):
a = tmp_path / "Book 1 - A" / "a.m4b"
c = tmp_path / "Book 3 - C" / "c.m4b"
for f in (a, c):
f.parent.mkdir(parents=True)
f.write_bytes(b"x")
plan = [PackBook(title="Alpha", series_position=1.0, year=None, files=["Book 1 - A/a.m4b"])]
groups = match_plan_to_files(plan, [a, c])
assert [(g.title, g.files) for g in groups] == [("Alpha", [a]), ("C", [c])]
+98
View File
@@ -0,0 +1,98 @@
"""Tests for listing the files inside a .torrent without downloading it."""
from shelfmark.download.clients.torrent_utils import (
bencode_encode,
extract_file_list_from_torrent,
)
from shelfmark.download.postprocess.packs import PackFile
def _torrent(info: dict) -> bytes:
return bencode_encode({b"announce": b"http://t/announce", b"info": info})
def test_multi_file_torrent_lists_release_relative_paths():
data = _torrent(
{
b"name": b"Sun Eater",
b"piece length": 16384,
b"pieces": b"x" * 20,
b"files": [
{b"length": 10, b"path": [b"Book 1 - Empire of Silence", b"empire.m4b"]},
{b"length": 20, b"path": [b"Book 2 - Howling Dark", b"howling.m4b"]},
{b"length": 1, b"path": [b"cover.jpg"]},
],
}
)
assert extract_file_list_from_torrent(data) == [
PackFile("Sun Eater/Book 1 - Empire of Silence/empire.m4b", 10),
PackFile("Sun Eater/Book 2 - Howling Dark/howling.m4b", 20),
PackFile("Sun Eater/cover.jpg", 1),
]
def test_single_file_torrent_lists_its_one_file():
data = _torrent(
{b"name": b"Book.m4b", b"length": 42, b"piece length": 16384, b"pieces": b"x" * 20}
)
assert extract_file_list_from_torrent(data) == [PackFile("Book.m4b", 42)]
def test_unparseable_data_returns_none():
assert extract_file_list_from_torrent(b"not a torrent") is None
class TestProwlarrHandlerListFiles:
def _handler(self):
from shelfmark.release_sources.prowlarr.handler import ProwlarrHandler
return ProwlarrHandler()
def test_lists_files_from_torrent_url(self):
from unittest.mock import patch
from shelfmark.download.clients.torrent_utils import TorrentInfo
data = _torrent(
{b"name": b"Book.m4b", b"length": 42, b"piece length": 16384, b"pieces": b"x" * 20}
)
release = {
"protocol": "torrent",
"downloadUrl": "http://prowlarr/dl.torrent",
"magnetUrl": "magnet:?xt=urn:btih:abc",
"infoHash": "abc",
}
with (
patch("shelfmark.release_sources.prowlarr.handler.get_release", return_value=release),
patch(
"shelfmark.release_sources.prowlarr.handler.extract_torrent_info",
return_value=TorrentInfo(info_hash="abc", torrent_data=data, is_magnet=False),
) as extract,
):
files = self._handler().list_files({"source_id": "rel-1"})
assert files == [PackFile("Book.m4b", 42)]
extract.assert_called_once_with("http://prowlarr/dl.torrent", expected_hash="abc")
def test_magnet_only_release_cannot_be_listed(self):
from unittest.mock import patch
release = {"protocol": "torrent", "magnetUrl": "magnet:?xt=urn:btih:abc"}
with (
patch("shelfmark.release_sources.prowlarr.handler.get_release", return_value=release),
patch("shelfmark.release_sources.prowlarr.handler.extract_torrent_info") as extract,
):
assert self._handler().list_files({"source_id": "rel-1"}) is None
extract.assert_not_called()
def test_usenet_release_cannot_be_listed(self):
from unittest.mock import patch
release = {"protocol": "usenet", "downloadUrl": "http://prowlarr/dl.nzb"}
with patch("shelfmark.release_sources.prowlarr.handler.get_release", return_value=release):
assert self._handler().list_files({"source_id": "rel-1"}) is None
def test_unknown_release_cannot_be_listed(self):
from unittest.mock import patch
with patch("shelfmark.release_sources.prowlarr.handler.get_release", return_value=None):
assert self._handler().list_files({"source_id": "missing"}) is None