shelfmark

562 Commits 4 Branches 49 Tags

3 Commits

Include renames

Author	SHA1	Message	Date
Lance Marks	f441b85da2	feat(packs): inspect multi-book releases and file each book separately (#1270 ) ## Multi-book packs: inspect a release before download and file each book separately Closes #576 ### Problem One queued release is always treated as one book. When a torrent is actually a whole series (`Series/Book 1 - Title/…`, or a flat folder of `Series 1.0 - Title.m4b` files), post-processing walks the whole tree, flattens every file into one list and renames them `Title - 01…10` under the searched book's `{Author}/{Title}`. Audiobookshelf then sees a single 10-file "book" and the user has to re-file everything by hand. ### What this does Most releases expose their file list before anything is downloaded, so the split is decided up front and approved by the user, then the download is fire-and-forget: 1. Inspect – clicking a release's download button now calls `POST /api/releases/inspect` first. A new optional `DownloadHandler.list_files(release_data)` hook returns the release's files without downloading: - AudiobookBay reads the torrent file table off the detail page it already fetches (the page is now cached for 120 s, so inspect + download cost ABB one request). - Prowlarr parses `info.files` from the `.torrent` it already fetches (the existing 120 s torrent-fetch cache is reused). Magnet-only and usenet releases report "can't inspect". - Other sources default to `None`. 2. Review – if the plan contains more than one book, the Find Releases modal swaps the list for a review panel: one row per book with editable title / series position / year, expandable file lists, non-book sidecars (`.txt`, covers) shown as ignored, a "Treat as a single book" switch, and Download N books. Single-book releases queue immediately, exactly as before. 3. File – the approved plan travels with the task (`DownloadTask.book_plan`, retry-safe) and post-processing files each book through the existing transfer code, one book at a time (`dataclasses.replace(task, title=…, series_position=…, year=…)`), so organize/rename templates, part numbering (now scoped per book), hardlinks, torrent copy-preserve and usenet handling are unchanged. Status reads `Complete (N books, M files)`. 4. Fallback – when a release can't be inspected the user gets a toast, and a small "Multi-book pack" toggle in the modal header forces a heuristic split (subfolder = book, or one book per file when the file names carry series positions). Planning lives in `shelfmark/download/postprocess/packs.py` and is shared by the inspect endpoint and post-processing, so what the user approved is what gets filed. The name parser strips `Book 3 -`, `03 -`, `1.0 -`, `3.`, `[03]`, `#3`, a leading series name, labels like "An Expanse Novella -", repeated titles (`Gods of Risk 2.5 - Gods of Risk`) and a trailing `(Year)`; author and series name come from the book that was searched, and the searched book's own series position is never applied to its siblings. ### Files - `shelfmark/download/postprocess/packs.py` (new) – `PackFile/PackBook/PackPlan`, `plan_pack`, `parse_pack_book_name`, `group_files_into_books`, `match_plan_to_files` - `shelfmark/core/release_inspect_routes.py` (new) – `POST /api/releases/inspect` - `shelfmark/release_sources/__init__.py` – `DownloadHandler.list_files` hook - `shelfmark/release_sources/audiobookbay/{scraper,handler}.py` – detail-page cache, `extract_file_list`, `list_files` - `shelfmark/release_sources/prowlarr/handler.py`, `download/clients/torrent_utils.py` – `extract_file_list_from_torrent`, `list_files` - `shelfmark/core/models.py`, `download/orchestrator.py` – `multi_book` / `book_plan` fields, queue + retry serialization - `shelfmark/download/postprocess/transfer.py`, `pipeline.py`, `outputs/folder.py` – per-book transfer branch and status message - `src/frontend`: `components/PackReviewPanel.tsx` (new), `ReleaseModal.tsx`, `App.tsx`, `services/api.ts`, `types/index.ts`, `utils/releasePayload.ts` (payload builder moved out of `App.tsx`), `utils/packReview.ts` - `docs/dev/release-sources-plugin-guide.md` – documents the `list_files` hook ### Out of scope (follow-ups) - Listing files from an NZB (Shelfmark already fetches the bytes; `<file subject>` names are noisy) - Inspecting magnet links via qBittorrent's files API after a paused add - BookLore / email outputs (they ignore `book_plan`; noted in code) - The combined ebook + audiobook flow ### Testing Automated (`make checks`, `make python-test`, `make frontend-test` all green; the only failures on my machine are the pre-existing `tests/config/test_entrypoint_permissions.py` cases, which need bash ≥ 4 and fail identically on `main` under macOS bash 3.2): - `tests/download/test_packs.py` – name parsing (markers, series name, novella labels, repeated titles, bare numeric titles like `1984`), nested / flat / mixed / deeper-nested packs, single wrapping folder not treated as a pack, plan-to-disk matching with basename fallback - `tests/core/test_processing_packs.py` – full `post_process_download` runs on a real temp filesystem: approved plan files each book under its own `{Author}/{Title}`, heuristic split of a nested pack, searched book's series position does not leak, multi-file book inside a pack keeps `- 01/- 02` per book, hardlinked torrent pack leaves the seeding tree intact, no pack fields ⇒ behaviour unchanged, single group degrades to the searched title, status message - `tests/core/test_release_inspect_routes.py` – plan response, not-inspectable, handler errors never 500, unknown source / missing `source_id` ⇒ 400, login required - `tests/audiobookbay/test_file_list.py` – file-table scraping from real ABB markup (multi-file and single-file pages), handler host validation, one page fetch shared by magnet + file list - `tests/prowlarr/test_torrent_file_list.py` – multi-file / single-file `.torrent` parsing, handler behaviour for torrent URL vs magnet vs usenet vs cache miss - `tests/download/test_orchestrator_pack_fields.py` – queue-time parsing and retry round-trip - Frontend: `releasePayload.test.ts`, `packReview.test.ts` (vitest) Manual, on a real deployment (arm64 image built from this branch, run as a side container next to production with the same qBittorrent / Audiobookshelf setup, `FILE_ORGANIZATION_AUDIOBOOK=organize`, hardlinks on): - AudiobookBay "The Expanse Complete 2.0" (7.87 GB, 36 files): clicking download opened the review panel in ~1 s showing 18 books · 18 files · 18 files ignored (the `.txt` sidecars), with series positions 0.1–9.5 and years parsed from the file names; novella labels stripped ("The Churn", "The Butcher of Anderson Station"). Editing a title in the panel works. Confirming queued one task; the magnet resolved from the cached page in ~30 ms; after the download the task reported `Complete (18 books, 18 files)`, 18 hardlinks landed as `audiobooks/James S. A. Corey/<Title>/<Title>.m4b`, the torrent kept seeding, and Audiobookshelf scanned each folder as its own book (title, author, embedded chapters). - A second pack ("Expanse [01 - 9.5]", `Title N - Title` naming) was inspected to verify the repeated-title rule and the Back button, without downloading. - Single-book releases still queue immediately with no extra UI.	2026-08-27 00:40:08 -04:00
Alex	68608b6162	Feature: Multi-user request system (#615 ) - Adds a comprehensive multi-user request system to the existing download flow - Request configuration is policy based. Configure global settings for content type, or narrow down policy for specific sources (E.g. allow direct downloads, set prowlarr to request only, block IRC completely, etc). - Global policy configuration and per-user overrides for tailored configs - Replaced downloads sidebar with ActivitySidebar, combining active downloads with requests. Admin management of user requests is done here, and admins have view of downloads from all users. Sidebar can now be pinned. - Request either a standard book or a specific release. Release-requests are used if you permit one source differently than the other. On book-level requests, admins pick the specific file to be attached to the fulfilled request. - Users can request books with a note This is WIP so some features are still not complete (notifications, more automatic release selection, among others).	2026-02-14 11:08:20 +00:00
Alex	fd74021594	File processing refactor and Booklore upload support (#474 ) - Added new book output option upload to Booklore, available in download settings - Got annoyed at my messy processing code while implementing Booklore so refactored the whole thing - Full black box file processing testing with randomised configuration - Deluge: Connect via WebUI auth for simplified setup - Added env vars documentation, auto generated via script, and unlocked most settings to be used as env vars	2026-01-16 14:45:00 +00:00

Author

SHA1

Message

Date

Lance Marks

f441b85da2

feat(packs): inspect multi-book releases and file each book separately (#1270 )

## Multi-book packs: inspect a release before download and file each
book separately

Closes #576

### Problem

One queued release is always treated as one book. When a torrent is
actually a whole series
(`Series/Book 1 - Title/…`, or a flat folder of `Series 1.0 - Title.m4b`
files), post-processing
walks the whole tree, flattens every file into one list and renames them
`Title - 01…10` under the
searched book's `{Author}/{Title}`. Audiobookshelf then sees a single
10-file "book" and the user
has to re-file everything by hand.

### What this does

Most releases expose their file list *before* anything is downloaded, so
the split is decided up
front and approved by the user, then the download is fire-and-forget:

1. **Inspect** – clicking a release's download button now calls `POST
/api/releases/inspect`
first. A new optional `DownloadHandler.list_files(release_data)` hook
returns the release's
   files without downloading:
- **AudiobookBay** reads the torrent file table off the detail page it
already fetches (the
page is now cached for 120 s, so inspect + download cost ABB one
request).
- **Prowlarr** parses `info.files` from the `.torrent` it already
fetches (the existing 120 s
torrent-fetch cache is reused). Magnet-only and usenet releases report
"can't inspect".
   - Other sources default to `None`.
2. **Review** – if the plan contains more than one book, the Find
Releases modal swaps the list
for a review panel: one row per book with editable title / series
position / year, expandable
file lists, non-book sidecars (`.txt`, covers) shown as ignored, a
"Treat as a single book"
switch, and **Download N books**. Single-book releases queue
immediately, exactly as before.
3. **File** – the approved plan travels with the task
(`DownloadTask.book_plan`, retry-safe) and
post-processing files each book through the existing transfer code, one
book at a time
(`dataclasses.replace(task, title=…, series_position=…, year=…)`), so
organize/rename
templates, part numbering (now scoped per book), hardlinks, torrent
copy-preserve and usenet
   handling are unchanged. Status reads `Complete (N books, M files)`.
4. **Fallback** – when a release can't be inspected the user gets a
toast, and a small
"Multi-book pack" toggle in the modal header forces a heuristic split
(subfolder = book, or
   one book per file when the file names carry series positions).

Planning lives in `shelfmark/download/postprocess/packs.py` and is
shared by the inspect endpoint
and post-processing, so what the user approved is what gets filed. The
name parser strips
`Book 3 -`, `03 -`, `1.0 -`, `3.`, `[03]`, `#3`, a leading series name,
labels like
"An Expanse Novella -", repeated titles (`Gods of Risk 2.5 - Gods of
Risk`) and a trailing
`(Year)`; author and series name come from the book that was searched,
and the searched book's
own series position is never applied to its siblings.

### Files

- `shelfmark/download/postprocess/packs.py` (new) –
`PackFile/PackBook/PackPlan`, `plan_pack`,
`parse_pack_book_name`, `group_files_into_books`, `match_plan_to_files`
- `shelfmark/core/release_inspect_routes.py` (new) – `POST
/api/releases/inspect`
- `shelfmark/release_sources/__init__.py` – `DownloadHandler.list_files`
hook
- `shelfmark/release_sources/audiobookbay/{scraper,handler}.py` –
detail-page cache,
  `extract_file_list`, `list_files`
- `shelfmark/release_sources/prowlarr/handler.py`,
`download/clients/torrent_utils.py` –
  `extract_file_list_from_torrent`, `list_files`
- `shelfmark/core/models.py`, `download/orchestrator.py` – `multi_book`
/ `book_plan` fields,
  queue + retry serialization
- `shelfmark/download/postprocess/transfer.py`, `pipeline.py`,
`outputs/folder.py` – per-book
  transfer branch and status message
- `src/frontend`: `components/PackReviewPanel.tsx` (new),
`ReleaseModal.tsx`, `App.tsx`,
`services/api.ts`, `types/index.ts`, `utils/releasePayload.ts` (payload
builder moved out of
  `App.tsx`), `utils/packReview.ts`
- `docs/dev/release-sources-plugin-guide.md` – documents the
`list_files` hook

### Out of scope (follow-ups)

- Listing files from an NZB (Shelfmark already fetches the bytes; `<file
subject>` names are noisy)
- Inspecting magnet links via qBittorrent's files API after a paused add
- BookLore / email outputs (they ignore `book_plan`; noted in code)
- The combined ebook + audiobook flow

### Testing

**Automated** (`make checks`, `make python-test`, `make frontend-test`
all green; the only
failures on my machine are the pre-existing
`tests/config/test_entrypoint_permissions.py` cases,
which need bash ≥ 4 and fail identically on `main` under macOS bash
3.2):

- `tests/download/test_packs.py` – name parsing (markers, series name,
novella labels, repeated
titles, bare numeric titles like `1984`), nested / flat / mixed /
deeper-nested packs, single
wrapping folder not treated as a pack, plan-to-disk matching with
basename fallback
- `tests/core/test_processing_packs.py` – full `post_process_download`
runs on a real temp
filesystem: approved plan files each book under its own
`{Author}/{Title}`, heuristic split
of a nested pack, searched book's series position does not leak,
multi-file book inside a pack
keeps `- 01/- 02` per book, hardlinked torrent pack leaves the seeding
tree intact, no pack
fields ⇒ behaviour unchanged, single group degrades to the searched
title, status message
- `tests/core/test_release_inspect_routes.py` – plan response,
not-inspectable, handler errors
  never 500, unknown source / missing `source_id` ⇒ 400, login required
- `tests/audiobookbay/test_file_list.py` – file-table scraping from real
ABB markup (multi-file
and single-file pages), handler host validation, one page fetch shared
by magnet + file list
- `tests/prowlarr/test_torrent_file_list.py` – multi-file / single-file
`.torrent` parsing,
  handler behaviour for torrent URL vs magnet vs usenet vs cache miss
- `tests/download/test_orchestrator_pack_fields.py` – queue-time parsing
and retry round-trip
- Frontend: `releasePayload.test.ts`, `packReview.test.ts` (vitest)

**Manual, on a real deployment** (arm64 image built from this branch,
run as a side container
next to production with the same qBittorrent / Audiobookshelf setup,
`FILE_ORGANIZATION_AUDIOBOOK=organize`,
hardlinks on):

- AudiobookBay "The Expanse Complete 2.0" (7.87 GB, 36 files): clicking
download opened the review
panel in ~1 s showing **18 books · 18 files · 18 files ignored** (the
`.txt` sidecars), with
series positions 0.1–9.5 and years parsed from the file names; novella
labels stripped
("The Churn", "The Butcher of Anderson Station"). Editing a title in the
panel works.
Confirming queued one task; the magnet resolved from the cached page in
~30 ms; after the
download the task reported `Complete (18 books, 18 files)`, 18 hardlinks
landed as
`audiobooks/James S. A. Corey/<Title>/<Title>.m4b`, the torrent kept
seeding, and
Audiobookshelf scanned each folder as its own book (title, author,
embedded chapters).
- A second pack ("Expanse [01 - 9.5]", `Title N - Title` naming) was
inspected to verify the
  repeated-title rule and the Back button, without downloading.
- Single-book releases still queue immediately with no extra UI.

2026-08-27 00:40:08 -04:00

Alex

68608b6162

Feature: Multi-user request system (#615 )

- Adds a comprehensive multi-user request system to the existing
download flow
- Request configuration is policy based. Configure global settings for
content type, or narrow down policy for specific sources (E.g. allow
direct downloads, set prowlarr to request only, block IRC completely,
etc).
- Global policy configuration and per-user overrides for tailored
configs
- Replaced downloads sidebar with ActivitySidebar, combining active
downloads with requests. Admin management of user requests is done here,
and admins have view of downloads from all users. Sidebar can now be
pinned.
- Request either a standard book or a specific release. Release-requests
are used if you permit one source differently than the other. On
book-level requests, admins pick the specific file to be attached to the
fulfilled request.
- Users can request books with a note

This is WIP so some features are still not complete (notifications, more
automatic release selection, among others).

2026-02-14 11:08:20 +00:00

Alex

fd74021594

File processing refactor and Booklore upload support (#474 )

- Added new book output option **upload to Booklore**, available in
download settings
- Got annoyed at my messy processing code while implementing Booklore so
refactored the whole thing
- Full black box file processing testing with randomised configuration
- Deluge: Connect via WebUI auth for simplified setup
- Added env vars documentation, auto generated via script, and unlocked
most settings to be used as env vars

2026-01-16 14:45:00 +00:00