starred/shelfmark
1
0 Fork 0
mirror of https://github.com/calibrain/shelfmark.git synced 2026-11-14 06:28:25 +00:00
Code Issues Packages Projects Releases 10 Wiki Activity
781 Commits 5 Branches 52 Tags
v1.6.2
Commit Graph
6 Commits
Author SHA1 Message Date
splitsec2 0a7e114681 docs(dev): new routes go in a register_<group>_routes module (#1622) 
Nothing in the developer docs says where a new route goes, so the next
contributor adds it to `main.py` again. This writes down the convention
from Discussion #1536: a new route group gets its own
`register_<group>_routes` module under `shelfmark/core/`, and `main.py`
calls it once.

It is one paragraph on a new page, `docs/dev/routes.md`, with a line in
the developer index. It also says that a few older routes are still in
`main.py` and are being moved out one group at a time. The route moves
in that discussion will follow what this says, so this goes first.
 2026-10-31 03:52:01 -06:00
splitsec2 e273e7b0ed docs(dev): bring the plugin guides in line with the current code (#1566) 
This is the biggest of the docs batch, and this time I don't think it
was my PRs that caused it. The plugin guides fell behind as the plugin
API changed over several releases, and nobody updated them along the
way.

Same arrangement as the others: Claude compared the guides to the code
and wrote the changes, so another human look before this merges would be
good. Someone who knows the plugin API well would catch things I would
not. These are the parts I would read first: the search plan table, the
`SourceUnavailableError` advice in the error handling section, and the
`locating` status row.

What was wrong and is now fixed, all checked against the code:

Release sources guide
- `search` takes `(book, plan, *, expand_search, content_type)`. Every
sample had the old one-argument signature, so a source written from the
guide raised a TypeError on its first search. There is now a short table
of the search plan fields.
- A source is registered by adding its module to
`_BUILTIN_SOURCE_MODULES`. The guide said to add an import at the bottom
of `__init__.py`.
- `get_column_config` is an instance method, not a classmethod, and
`ColumnRenderType.SEEDERS` does not exist (the real members are listed).
- `download` can return a `HandoffResult`. `list_available_sources`
lists every source with an `enabled` flag, it does not filter.
- `group="sources"` is gone, since only `direct_download` and
`metadata_providers` exist.
- Field lists for `Release`, `DownloadTask`, `BookMetadata`,
`ColumnSchema` and `ReleaseColumnConfig` are brought up to date, with
the protocol enum and the `locating` status.
- The optional hooks that were not mentioned (`post_process_cleanup`,
`build_retry_resolution_fields`, `list_authors`, `seed_complaints`,
`get_record`, `get_destination_override`, `search_results_are_releases`)
are documented.

Plugin settings guide
- The examples imported from `shelfmark.metadata_providers.base` and
`shelfmark.release_sources.base`, which do not exist. The metadata
example also skipped the abstract `search_by_isbn` and the kwargs
factory.
- Value priority now starts with the environment variable, and
`search_mode` is listed next to `general` as using `settings.json`. The
`metadata_providers` group order is 12, not 50.
- Documented `TagListField`, `OrderableListField`, `TableField`,
`env_var`, `env_supported`, `user_overridable`, `universal_only`,
`filter_by_field`, `register_on_save` and the `current_values` argument
for action callbacks.

Also in here: `docs/dev/index.md` was a TODO stub and is now a short
index linked from `docs/index.md`, a dead link in the metadata providers
README is fixed, and the README table gains Databáze knih and Moly.hu.

To check the samples were not just plausible, I ran the quick start and
the full BookNet example against the real registry. Both register,
instantiate and return their column config. The old BookNet example
failed on import with a NameError, because the Test Connection callback
was defined below the settings function that references it. The
decorator calls that function straight away, so both settings snippets
now define the callback first.
 2026-10-23 22:02:18 -06:00
CaliBrain 10710abd4e fix(download): send every file of a multi-file download to the browser (#1489) 
Download to Browser only ever sent the first file of a multi-file
release
(audiobook chapters, packs): the folder output returned final_paths[0]
and
/api/localdownload served that one path.

The folder output now keeps every final path on the in-memory task
(download_paths), and /api/localdownload streams them as one
uncompressed
ZIP named after the book when there is more than one. The archive is
never
staged on disk or held in memory, and ZIP64 covers large audiobooks.
History still stores only the first path, so a re-download from history
after a dismiss or restart serves a single file, as before.

Also brings every linter back to green on all files:
- pre-commit: bump the ruff hook to v0.17.1 to match uv.lock;
ruff-format
  the Python examples in docs/dev and skills
- ruff: fix the remaining issues in skills/shelfmark/download_books.py;
  format base_handler.py
- basedpyright: Optional and attrs errors in databazeknih.py
- knip: drop 31 unused exports, reuse ColumnColorHint in colorMaps
- oxfmt 0.72: styles.css
- direct download: fix the "Welib, mirrors" typo and the test that still
  expected the pre-OceanofPDF message

Fixes #1467
 2026-10-20 02:33:53 -04:00
Lance Marks f441b85da2 feat(packs): inspect multi-book releases and file each book separately (#1270) 
## Multi-book packs: inspect a release before download and file each
book separately

Closes #576

### Problem

One queued release is always treated as one book. When a torrent is
actually a whole series
(`Series/Book 1 - Title/…`, or a flat folder of `Series 1.0 - Title.m4b`
files), post-processing
walks the whole tree, flattens every file into one list and renames them
`Title - 01…10` under the
searched book's `{Author}/{Title}`. Audiobookshelf then sees a single
10-file "book" and the user
has to re-file everything by hand.

### What this does

Most releases expose their file list *before* anything is downloaded, so
the split is decided up
front and approved by the user, then the download is fire-and-forget:

1. **Inspect** – clicking a release's download button now calls `POST
/api/releases/inspect`
first. A new optional `DownloadHandler.list_files(release_data)` hook
returns the release's
   files without downloading:
- **AudiobookBay** reads the torrent file table off the detail page it
already fetches (the
page is now cached for 120 s, so inspect + download cost ABB one
request).
- **Prowlarr** parses `info.files` from the `.torrent` it already
fetches (the existing 120 s
torrent-fetch cache is reused). Magnet-only and usenet releases report
"can't inspect".
   - Other sources default to `None`.
2. **Review** – if the plan contains more than one book, the Find
Releases modal swaps the list
for a review panel: one row per book with editable title / series
position / year, expandable
file lists, non-book sidecars (`.txt`, covers) shown as ignored, a
"Treat as a single book"
switch, and **Download N books**. Single-book releases queue
immediately, exactly as before.
3. **File** – the approved plan travels with the task
(`DownloadTask.book_plan`, retry-safe) and
post-processing files each book through the existing transfer code, one
book at a time
(`dataclasses.replace(task, title=…, series_position=…, year=…)`), so
organize/rename
templates, part numbering (now scoped per book), hardlinks, torrent
copy-preserve and usenet
   handling are unchanged. Status reads `Complete (N books, M files)`.
4. **Fallback** – when a release can't be inspected the user gets a
toast, and a small
"Multi-book pack" toggle in the modal header forces a heuristic split
(subfolder = book, or
   one book per file when the file names carry series positions).

Planning lives in `shelfmark/download/postprocess/packs.py` and is
shared by the inspect endpoint
and post-processing, so what the user approved is what gets filed. The
name parser strips
`Book 3 -`, `03 -`, `1.0 -`, `3.`, `[03]`, `#3`, a leading series name,
labels like
"An Expanse Novella -", repeated titles (`Gods of Risk 2.5 - Gods of
Risk`) and a trailing
`(Year)`; author and series name come from the book that was searched,
and the searched book's
own series position is never applied to its siblings.

### Files

- `shelfmark/download/postprocess/packs.py` (new) –
`PackFile/PackBook/PackPlan`, `plan_pack`,
`parse_pack_book_name`, `group_files_into_books`, `match_plan_to_files`
- `shelfmark/core/release_inspect_routes.py` (new) – `POST
/api/releases/inspect`
- `shelfmark/release_sources/__init__.py` – `DownloadHandler.list_files`
hook
- `shelfmark/release_sources/audiobookbay/{scraper,handler}.py` –
detail-page cache,
  `extract_file_list`, `list_files`
- `shelfmark/release_sources/prowlarr/handler.py`,
`download/clients/torrent_utils.py` –
  `extract_file_list_from_torrent`, `list_files`
- `shelfmark/core/models.py`, `download/orchestrator.py` – `multi_book`
/ `book_plan` fields,
  queue + retry serialization
- `shelfmark/download/postprocess/transfer.py`, `pipeline.py`,
`outputs/folder.py` – per-book
  transfer branch and status message
- `src/frontend`: `components/PackReviewPanel.tsx` (new),
`ReleaseModal.tsx`, `App.tsx`,
`services/api.ts`, `types/index.ts`, `utils/releasePayload.ts` (payload
builder moved out of
  `App.tsx`), `utils/packReview.ts`
- `docs/dev/release-sources-plugin-guide.md` – documents the
`list_files` hook

### Out of scope (follow-ups)

- Listing files from an NZB (Shelfmark already fetches the bytes; `<file
subject>` names are noisy)
- Inspecting magnet links via qBittorrent's files API after a paused add
- BookLore / email outputs (they ignore `book_plan`; noted in code)
- The combined ebook + audiobook flow

### Testing

**Automated** (`make checks`, `make python-test`, `make frontend-test`
all green; the only
failures on my machine are the pre-existing
`tests/config/test_entrypoint_permissions.py` cases,
which need bash ≥ 4 and fail identically on `main` under macOS bash
3.2):

- `tests/download/test_packs.py` – name parsing (markers, series name,
novella labels, repeated
titles, bare numeric titles like `1984`), nested / flat / mixed /
deeper-nested packs, single
wrapping folder not treated as a pack, plan-to-disk matching with
basename fallback
- `tests/core/test_processing_packs.py` – full `post_process_download`
runs on a real temp
filesystem: approved plan files each book under its own
`{Author}/{Title}`, heuristic split
of a nested pack, searched book's series position does not leak,
multi-file book inside a pack
keeps `- 01/- 02` per book, hardlinked torrent pack leaves the seeding
tree intact, no pack
fields ⇒ behaviour unchanged, single group degrades to the searched
title, status message
- `tests/core/test_release_inspect_routes.py` – plan response,
not-inspectable, handler errors
  never 500, unknown source / missing `source_id` ⇒ 400, login required
- `tests/audiobookbay/test_file_list.py` – file-table scraping from real
ABB markup (multi-file
and single-file pages), handler host validation, one page fetch shared
by magnet + file list
- `tests/prowlarr/test_torrent_file_list.py` – multi-file / single-file
`.torrent` parsing,
  handler behaviour for torrent URL vs magnet vs usenet vs cache miss
- `tests/download/test_orchestrator_pack_fields.py` – queue-time parsing
and retry round-trip
- Frontend: `releasePayload.test.ts`, `packReview.test.ts` (vitest)

**Manual, on a real deployment** (arm64 image built from this branch,
run as a side container
next to production with the same qBittorrent / Audiobookshelf setup,
`FILE_ORGANIZATION_AUDIOBOOK=organize`,
hardlinks on):

- AudiobookBay "The Expanse Complete 2.0" (7.87 GB, 36 files): clicking
download opened the review
panel in ~1 s showing **18 books · 18 files · 18 files ignored** (the
`.txt` sidecars), with
series positions 0.1–9.5 and years parsed from the file names; novella
labels stripped
("The Churn", "The Butcher of Anderson Station"). Editing a title in the
panel works.
Confirming queued one task; the magnet resolved from the cached page in
~30 ms; after the
download the task reported `Complete (18 books, 18 files)`, 18 hardlinks
landed as
`audiobooks/James S. A. Corey/<Title>/<Title>.m4b`, the torrent kept
seeding, and
Audiobookshelf scanned each folder as its own book (title, author,
embedded chapters).
- A second pack ("Expanse [01 - 9.5]", `Title N - Title` naming) was
inspected to verify the
  repeated-title rule and the Back button, without downloading.
- Single-book releases still queue immediately with no extra UI.
 2026-08-27 00:40:08 -04:00
Alex 68608b6162 Feature: Multi-user request system (#615) 
- Adds a comprehensive multi-user request system to the existing
download flow
- Request configuration is policy based. Configure global settings for
content type, or narrow down policy for specific sources (E.g. allow
direct downloads, set prowlarr to request only, block IRC completely,
etc).
- Global policy configuration and per-user overrides for tailored
configs
- Replaced downloads sidebar with ActivitySidebar, combining active
downloads with requests. Admin management of user requests is done here,
and admins have view of downloads from all users. Sidebar can now be
pinned.
- Request either a standard book or a specific release. Release-requests
are used if you permit one source differently than the other. On
book-level requests, admins pick the specific file to be attached to the
fulfilled request.
- Users can request books with a note

This is WIP so some features are still not complete (notifications, more
automatic release selection, among others).
 2026-02-14 11:08:20 +00:00
Alex fd74021594 File processing refactor and Booklore upload support (#474) 
- Added new book output option **upload to Booklore**, available in
download settings
- Got annoyed at my messy processing code while implementing Booklore so
refactored the whole thing
- Full black box file processing testing with randomised configuration
- Deluge: Connect via WebUI auth for simplified setup
- Added env vars documentation, auto generated via script, and unlocked
most settings to be used as env vars
 2026-01-16 14:45:00 +00:00