mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-11-16 03:58:14 +00:00
756bdec19bd52edd7f527cb7a64b362a9da8de13
112
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
756bdec19b |
fix(metadata): show the translation a search matched on Open Library and Hardcover (#1451)
Searching a translated title on Open Library or Hardcover returned the original-language work. "Der Weg der Könige" came back as "The Way of Kings" (2010), and that English title then went into the release search and the download name. Both providers keep one record per work, titled after the original, and list translations underneath it. #483 already searches releases per language with Hardcover's localized titles, which fixed the release side of #360 when the matching language is selected. This PR fixes the search result itself, so the book a user picks is the translation they searched for, and it covers Open Library, the default provider, which has no titles_by_language. - Open Library: request the matched edition (editions.*) and prefer its title, ISBNs, cover, language and publication year when its title differs from the work's. - Hardcover: use the alternative title named in the hit's highlights (alternative_titles, best index first), without the original's subtitle. A hit whose title also matched keeps the work title. - /api/releases: when the modal sends a title that differs from the book record's, it becomes search_title. Before, Hardcover's search_title from the original won, so picking "Kinder des Nebels" still searched "The Final Empire" first. - Release cache: keyed on title and language as well as the provider ID. All translations share that ID, so opening "The Way of Kings" after "Der Weg der Könige" showed the German releases until a reload. Known limitations: neither provider has descriptions per edition, so the description stays the work's (usually English). Hardcover's search hits carry no year or language for alternative titles, so those stay the work's; Open Library's do not. Google Books is unaffected, since it already returns one volume per edition. Tests: Open Library edition parsing, Hardcover alternative-title matching, the /api/releases title override, and the release cache, each written to fail first. The Python suite passes apart from the 9 test_entrypoint_permissions.py tests, which fail the same way on an unmodified main on macOS. The frontend typecheck, lint, format check and unit tests pass. Checked on a live instance with Open Library, Hardcover and Google Books. Refs #360, #483 Disclaimer: This was developed with the help of AI but reviewed and tested as well as deployed on my server by a real human. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f0f0d46381 |
fix(queue): first terminal status wins, so a late COMPLETE cannot overwrite a cancel (#1429)
Kept small on purpose so it's easy to review. I have a few more small fixes like this from going back over recently merged PRs. Would you rather get them as separate PRs like this one, or combined into one? #1361 made `cancel_download` write under the queue lock, which stops a cancel from overwriting a download that finished first. The other direction is still open. The worker checks `cancel_flag.is_set()` and then calls `update_status(COMPLETE)` a few lines later in `_process_single_download`. If a cancel lands between those two lines the task goes CANCELLED, then the worker writes COMPLETE over it. `_apply_status_locked` allows terminal to terminal moves, so the terminal hook also fires a second time and records two events for one task. This makes the first terminal status win: `_apply_status_locked` ignores a different terminal status once the task already has one. Requeueing from a terminal status still works because QUEUED isn't terminal. I replaced the existing race test. It patched `queue.update_status`, which `cancel_download` no longer calls, so it never exercised the path it names, and its last assertion only looked at the final hook event. The new tests call the queue directly: complete then cancel is refused, cancel then complete stays cancelled with one hook event, error then complete stays error, and a terminal task can still be requeued. The two late-write tests fail on current `main`. Ruff and basedpyright are clean. `tests/core` and `tests/download` pass apart from 4 `seleniumbase` import failures in my environment that also fail on `main`. The one red check is the existing clock-dependent test `test_a_queued_torrent_asks_for_a_grace_once`, which fails on any runner that booted recently. It is fixed in #1426 and is not caused by this change. |
||
|
|
65dbf42a86 |
fix(release-search): report a ValueError from a source as that source's error (#1433)
Kept small on purpose so it's easy to review. This is the same kind of small fix as the ones in #1429 to #1432, so see the question in #1429 about separate PRs versus combined. I'm sending them separately until I hear otherwise. In `search_source_releases` the `except ValueError` comes before the branch that handles the operational errors, and `ValueError` is in that tuple. A `ValueError` from `build_release_search_plan` or `source.search()` is therefore reported as `Unknown source: <name>` instead of `<name>: <message>`. The only `ValueError` meant for that branch is the one `get_source` raises for a name that isn't registered, so that call now has its own `try` and the rest goes back to the operational errors branch. The failure test already covered `OSError` and `RuntimeError`, so I added `ValueError` to it, and that case fails on current `main`. Ruff, basedpyright and `tests/core` pass. The one red check is the existing clock-dependent test `test_a_queued_torrent_asks_for_a_grace_once`, which fails on any runner that booted recently. It is fixed in #1426 and is not caused by this change. |
||
|
|
93f58caf7a |
fix(debug): keep SHELFMARK_API_KEY_READONLY out of the debug bundle (#1432)
Kept small on purpose so it's easy to review. `genDebug.sh` filters secrets out of `environment.txt` by matching `_KEY=`, `_SECRET=`, `_PASSWORD=` and `_TOKEN=`, which only catches names that end in those words. `SHELFMARK_API_KEY_READONLY` ends in `_READONLY`, so its value lands in the file, and people attach these bundles to issues. That key only reads counters, so the impact is small, but any variable like `DB_PASSWORD_FILE` would leak the same way. The pattern now matches the secret word anywhere in the variable name. The new test runs the real filter line from the script against a made-up environment, and it fails on the old pattern. The one red check is the existing clock-dependent test `test_a_queued_torrent_asks_for_a_grace_once`, which fails on any runner that booted recently. It is fixed in #1426 and is not caused by this change. |
||
|
|
e8f6505bea |
fix(library): back off after a failed index instead of retrying for every result (#1430)
Kept small on purpose so it's easy to review. `library_index._entries_for` retries `fetch_entries` on every call while the index is failing. The lookup runs once per search result, so with the Calibre path unreadable a page of 40 results makes 40 attempts and logs 40 warnings. The entrypoint runs one gevent worker and the Calibre read is synchronous sqlite, so each attempt also holds up every other request. After a failure it now answers from the stale cache (or an empty list) for 60 seconds before trying again, and a successful index clears that. The new tests cover 40 lookups against a failing library with a stale cache (one failed attempt, one warning), a failing library with no cache (tried once), and a retry once the 60 seconds have passed. The first and second fail on current `main`. I left a few other things in this module out to keep this one small: re-indexing blocks the worker for as long as a big library takes (I haven't measured how long), `test_connection` stores entries from the unsaved form path under the shared cache key, and a title with a collection word in it, like "Books of Blood", shows as "In a collection". I can send any of those separately. Ruff and basedpyright are clean and `tests/core` passes. The one red check is the existing clock-dependent test `test_a_queued_torrent_asks_for_a_grace_once`, which fails on any runner that booted recently. It is fixed in #1426 and is not caused by this change. |
||
|
|
755f28a6a4 |
feat(api): add a read-only API key and a /api/stats endpoint (#1419)
This is the read-only API key from #1410, where you said to go ahead. I run Shelfmark behind Homepage and wanted more on the dashboard tile than up or down, without putting an admin API key in the dashboard's config. This adds a second key, `SHELFMARK_API_KEY_READONLY`, that can read one new endpoint. `GET /api/stats` returns counts only: books added over the last 7 and 30 days by format, the queue, requests by outcome, and download failures over the last 7 days. No titles and no user names. The read-only key, the admin key or an admin session can read it. The read-only key gets a 403 on every other path and on any write, including a POST to `/api/stats` itself, and it never gets a session or a cookie. If a request carries both keys, the admin one wins, so a proxy's own `Authorization` header can't downgrade a correct `X-Api-Key`. Unset means off, like the existing key. The endpoint exists either way, but without a key only an admin session can read it. It also adds a `db_path` property on `UserDB`, so the stats module can open the database read-only. Tests cover the key scope, the 403s for the read-only key, both keys at once, who can reach the endpoint, and the counters. The full suite passes, and I broke each rule on purpose to check a test catches it. A version of this has been running on my own install behind Homepage. If you'd rather have the stats in a different shape or under a different path, I'm happy to change it. |
||
|
|
32abaff21e |
feat(naming): {Narrator} placeholder and MyAnonamouse series fallback (#1407)
## Human-Written explanation:
Following up on previous PR to use MAM ID to get nararrator and series
to show up in search, this PR will allow the series and nararrator
fields to be added to the path when saving an audiobook.
I have tested my ghcr.io image on my instance and it seemed to work
properly.
All the text below is written by Claude.
# feat(naming): {Narrator} placeholder and MyAnonamouse series fallback
Follow-up to #1390 and #1399. Related to #605 and #934 (narrator
support).
## Problem
Several narrations of the same audiobook currently land in the same
folder, because nothing in the path template tells them apart. Keeping
more than one version means renaming folders by hand after every
download.
Since #1390, MyAnonamouse results carry the release's narrator and
series in `extra`, but the naming templates can't use them: there is no
`{Narrator}` placeholder, and `{Series}` / `{SeriesPosition}` only come
from the metadata provider.
## Changes
- **`{Narrator}` placeholder** (`core/naming.py`,
`download/postprocess/transfer.py`, `core/models.py`): `DownloadTask`
gains `narrator`, read from `narrator` or `extra.narrator` when a
release is queued, and kept in the restart-safe retry payload. It's
available in audiobook templates like any other placeholder, including
prefix/suffix blocks such as `{ - Narrator}`.
- **Audiobookshelf folder style**: Audiobookshelf reads the narrator
from folder names like `Title {Narrator}`. The template `{Title}
{{Narrator}}` already matches as `{` + `{Narrator}`; the parser now also
drops the closing brace when the narrator is empty, so the folder is
`Title` rather than `Title }`.
- **No stray spaces around `/`**: an empty placeholder at the start or
end of a folder name no longer leaves a space there (e.g. `Title
/Title`). This applies to all templates.
- **Series fallback** (`release_sources/prowlarr/mam.py`, `source.py`,
`download/orchestrator.py`): MAM enrichment also stores the first
series' name and number as `extra.series_name` /
`extra.series_position`, which queueing already falls back to when the
metadata provider has no series. The provider's series still wins. A
release-level series number is only used when it names the same series
as the provider, so a provider series is never paired with the number of
a different MAM series (an omnibus, for example).
- **Settings and preview** (`config/settings.py`,
`namingTemplatePreview.ts`): the audiobook template descriptions list
`{Narrator}` and explain `{{Narrator}}`. The settings page preview
mirrors the parser changes and offers `{Narrator}` under "Insert
variable" for audiobook templates only.
- **Docs**: regenerated `environment-variables.md`. This also picked up
the Library Check settings, which weren't in the generated docs yet;
happy to drop that part to keep the diff focused.
## Example
Audiobook Path Template `{Author}/{Title} {{Narrator}}/{Title}`:
| Narrator | Result |
|----------|--------|
| Samuel Roukin | `Christopher Ruocchio/Empire of Silence {Samuel
Roukin}/Empire of Silence.m4b` |
| none | `Christopher Ruocchio/Empire of Silence/Empire of Silence.m4b`
|
## Testing
- `tests/core/test_narrator_template_variable.py` (new, 16 tests):
`{{Narrator}}` with and without a value, the `{ {Narrator}}` prefix
form, prefix/suffix blocks, sanitizing, `build_library_path`, task
metadata, retry payload round trip, MAM series name/number parsing
(including a `1-3` range having no number), and queueing (narrator from
`extra`, series fallback, no cross-series number, blank narrator).
- `namingTemplatePreview.test.ts`: the same `{{Narrator}}` cases for the
preview.
- Updated `test_generate_env_docs.py` for the new template description.
- Full `pytest -m "not integration and not e2e"` compared with an
upstream `main` worktree on the same machine: no new failures (the
remaining ones are Windows-only on both).
- `ruff check`, `ruff format --check`, `basedpyright` (0 errors) and
`vulture` on touched files; frontend `tsc --noEmit`, `oxlint`, `oxfmt
--check`, `vitest` (218 passed).
- Checked the settings page locally: the audiobook preview renders `...
{Simon Vance}.mp3` and `{Narrator}` is listed for audiobook templates
only.
No behavior change for existing templates, apart from spaces next to `/`
being trimmed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
119d3374e3 | fix(downloads): reconcile downloads a restart interrupted (#1423) | ||
|
|
8392b43dd3 |
fix(downloads): reconcile downloads a restart interrupted (#1409)
I found this while grabbing new books and working on my fork. I redeployed the container while downloads were running, and it left a mess in my queue. It's an edge case, but a few rows were stuck until I cleared them by hand. The download queue lives in memory, so stopping the process with a download in flight leaves its history row "active" and its request "queued", with nothing working on either. The activity API shows that row as "Interrupted" when it's read, but nothing is stored and nothing looks at it again. Upstream only reopens an errored request when no manual retry is offered, and an interrupted row always offers one, so the request just sits there. This closes those rows out once at startup, before the coordinator starts, as an error with the message "Interrupted". The retry payload stays, so the manual retry still works. A request whose download was queued in the last 24 hours goes back to pending through `reopen_failed_request`. Older ones get closed out but their requests are left alone, because I didn't think a startup should reopen a backlog or fill an admin's approval queue with old requests. A request that already arrived is left alone, and running it twice changes nothing. It assumes a single worker process, which `entrypoint.sh` already sets. On my install the first start after deploying it closed 53 rows that had been orphaned for nine days. Tests are in `tests/core/test_startup_reconcile.py`, and the full suite passes locally. If the 24 hour window feels off, or you'd rather handle this another way, I'm happy to change it or drop the PR. Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
|
bac0b93566 |
fix: keep https:// for qBittorrent and hide release sources from non-admins (#1422)
qBittorrent over HTTPS (#1417) qbittorrent-api ignores the scheme in `host` and probes HTTP and HTTPS itself. When the HTTPS probe fails, for instance on a certificate Shelfmark doesn't trust, it falls back to plain HTTP against the TLS port, and a reverse proxy answers "400 The plain HTTP request was sent to HTTPS port". The download client and the Test Connection button now pass FORCE_SCHEME_FROM_HOST for https:// URLs, so the configured scheme is used as-is and a certificate problem is reported as one. http:// and bare host:port URLs keep the probe: normalization adds http:// to a bare host, and the probe follows a proxy's redirect to HTTPS. Release sources leaked to non-admins (#1418) A download queued from an approved request carries the release an admin picked, and the requester could read it from their own activity feed: the request's release_data (source_id, indexer, info URL, torrent attributes), the download's full server path, and the download id itself, which for Prowlarr is "<indexer id>:<guid>" and for a private tracker is a URL into it. That id keys every status payload, the /api/localdownload query and the cover proxy URL, so hiding release_data alone would not have closed the leak. For non-admin viewers: - request rows keep only the release fields the activity cards display - download_path is reduced to the file name, which the browser download reveals anyway - request-linked downloads are addressed by an opaque id (a keyed HMAC of the task id) in the snapshot, history, /api/status, queue order, active downloads, dismissed keys, cover URLs and the websocket status and progress events. Routes that take a download id translate it back, searching only the caller's own downloads. Downloads a user queued directly keep their real id: the user picked that release from search results that already showed it, and the release list matches its buttons to the queue by that id. Admin views are unchanged. The activity sidebar now links a fulfilled request to its download by request_id instead of release_data.source_id. Closes #1417 Closes #1418 |
||
|
|
a05e002305 |
feat: narrator, series and bitrate columns for MyAnonamouse results (#1390)
# feat: narrator, series and bitrate columns for MyAnonamouse results Addresses #605 and #934 (narrator in the release list). ## Problem Both requests were closed because the narrator isn't in Torznab results, which is correct. Prowlarr's MyAnonamouse indexer reads `author_info` but drops the `narrator_info` and `series_info` MAM returns next to it, and neither Prowlarr's `ReleaseInfo` nor the Torznab output has a field for them. Shelfmark's size tooltip already looks for a `narrator` Torznab attribute, but nothing ever sends one. When a book has several narrations, choosing one means going back and forth between Shelfmark and the tracker. ## Approach Optional, opt-in enrichment using the user's own MAM session (`mam_id`), the same approach AudioBookRequest's MAM indexer uses: 1. After the Prowlarr search, releases whose info URL is `…myanonamouse.net/t/<id>` are collected. 2. Shelfmark sends the same query text to MAM's JSON search (`/tor/js/loadSearchJSONbasic.php`, normally one request, at most 3), and matches torrents back to Prowlarr's results by torrent ID. The MAM origin is taken from the result URL, so a custom Prowlarr MAM base URL is respected, and the cookie is only ever sent to `*.myanonamouse.net`. 3. Adds `extra.narrator`, `extra.series` (`The Sun Eater #1`) and `extra.bitrate` / `extra.bitrate_value`. MAM has no bitrate field, so it's parsed from the uploader's free-text tags (`64 kbps`) and some releases won't have one. Lookups are cached per torrent for an hour, respect the existing Prowlarr search deadline, go through the configured proxy (`get_proxies`), and never fail the search: a 403, timeout or bad JSON is logged and the list renders without the extra details. ## Changes - **`release_sources/prowlarr/mam.py`** (new): small MAM client (`search`, and `get_username` for the test button), `narrator_info` / `series_info` / tags parsing, cached best-effort `lookup_torrent_details()`. A 403 includes MAM's reply and a note about the IP/ASN lock. - **`release_sources/prowlarr/source.py`**: enrichment after the result loop. Series, Narrator and Bitrate columns only when a MAM ID is configured, since otherwise they would be empty for every row. A Torznab `bitrate` attribute from other indexers is also mapped to `extra.bitrate`. - **`release_sources/prowlarr/settings.py`**: "MyAnonamouse Enrichment" section with `PROWLARR_MAM_ID` (password field, env-overridable like every setting) and a **Test MAM Session** button. - **`release_sources/__init__.py`**: `ColumnSchema` gains optional `setting_key` and `content_types`. The new `apply_column_visibility()` drops gated columns and their grid tracks. Neither field is serialized. - **`main.py`**: `/api/releases` applies `apply_column_visibility()` with the request's content type and the user's effective settings. - **`config/settings.py` / `users_settings.py`**: Search Mode › "Release List Columns" with `SHOW_SERIES_COLUMN`, `SHOW_NARRATOR_COLUMN` and `SHOW_BITRATE_COLUMN`, all default on and user-overridable. Narrator and bitrate are audiobook-only, series shows for both. AudiobookBay's existing bitrate column now follows the bitrate toggle. - **Frontend**: text cells truncate with a hover title; the mobile info line wraps and skips empty text/number cells so blank optional columns don't leave orphan `·` separators; the size tooltip no longer lists Bitrate twice. - **Docs**: new `docs/myanonamouse-enrichment.md` (linked from the index), and a regenerated `environment-variables.md`. The regeneration also picked up a few pre-existing drifts from `main` (the Libgen section, `AA_DEFAULT_SORT` default, a duplicate `BOOK_LANGUAGE` row). I can drop those if you'd rather keep this diff focused. ## ⚠️ MAM sessions are IP/ASN-locked MyAnonamouse locks each session to one IP or ASN. Reusing the session Prowlarr (or a seedbox script) uses will often **403**. **A separate MAM session for Shelfmark will likely be needed** when Shelfmark reaches MAM from a different IP (another host, a VPN container, or a proxy in Shelfmark's Network settings), or when the existing session is ASN-locked to another network. The setting's description, the error message and the new doc all say so. ## Testing - `tests/prowlarr/test_mam_enrichment.py` (new, 19 tests): parsing (narrator dedupe, multiple series, missing numbers, malformed JSON, tag bitrate), lookup (stops once all IDs are found, cache, 403 and connection errors return empty, expired deadline skips the request), only MAM releases enriched, Torznab bitrate mapping, column config with and without a MAM ID for audiobook and ebook, toggles, grid-track removal, and gates not serialized. - `tests/core/test_admin_users_api.py`: the curated search-preference key list now includes the three toggles. - Full `pytest -m "not integration and not e2e"` compared with an upstream `main` worktree on the same machine: no new failures. The remaining ~115 failures on both are Windows-only (tor/entrypoint shell tests, path separators). - `ruff check` / `ruff format --check` / `basedpyright` (0 errors) / `vulture` on touched files; frontend `tsc --noEmit`, `oxlint`, `oxfmt --check`, `vitest` (201 passed). - Manually verified with a real MAM account on a Docker build of this branch: the test button, then narrator, series and bitrate on MyAnonamouse audiobook results. No behavior change unless `PROWLARR_MAM_ID` is set, apart from the bitrate toggle on AudiobookBay (default on, same as today). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
|
ece6b8f341 |
fix(auth): fail closed when auth prerequisites are missing (#1387) (#1397)
Auth mode resolution fell back to "none" (anonymous full admin) whenever the configured method's prerequisites were missing: no local password admin for builtin/OIDC, no Calibre-Web database, a blank proxy header, an unrecognized AUTH_METHOD (including "OIDC" in uppercase), or any error while reading the config. Deleting or demoting the last local admin was allowed on purpose because of that fallback, which exposed OIDC instances publicly. - Only an explicit AUTH_METHOD=none disables authentication. A configured method stays active when its prerequisites are missing, so sign-in fails instead of opening up. - An unrecognized or unreadable AUTH_METHOD resolves to "unavailable", which still requires a session and accepts no login. Values are normalized, so AUTH_METHOD=OIDC works. - Restore the guard against deleting or demoting the last local password admin while builtin/OIDC is active (unless DISABLE_LOCAL_AUTH is set). - Require a local admin before enabling Local auth, as OIDC already did. - Log a recovery hint at startup when builtin/OIDC runs without a local admin, and document recovery via AUTH_METHOD=none. - Drop the "will fall back to No Authentication" UI toasts and hints. Fixes https://github.com/calibrain/shelfmark/issues/1387 |
||
|
|
37a77e9562 |
feat(library): mark search results already in a Calibre library (#1377)
Per discussion #1372, where you said you were fine with this specific implementation: check whether metadata.db exists, read it if so, and show a check mark saying the book is already there. Searching for a book you already own gives no hint that you own it, so the easiest way to end up with a second copy is to not remember you have the first. This reads a Calibre `metadata.db`, read only, and marks matching results with an **In library** badge in the card, list and compact views and in the details dialog. Off by default. It sits in Settings, General beside the existing Library URL, with a test button that reports how many books it indexed. No HTTP call, no token, nothing written back. Matching runs most to least confident: a shared external id, then an ISBN compared in both ISBN-10 and ISBN-13 form, then fuzzy title tokens plus the author surname. The check fails open, so an unreadable database degrades the badge and never blocks a search, and entries are cached for ten minutes with an early refresh when the file changes, so a large library costs one read rather than one per search. `text_match.py` is new and shared by the index and the provider, so title, author and ISBN matching stays consistent in one place. ## On the provider interface `library_index` talks only to a `LibraryProvider` protocol and knows nothing about Calibre. That is deliberate but it is not speculative generality, it is what let me send you the Calibre half on its own: I run an Audiobookshelf provider on the same interface in my fork, which is where the audiobook side of the badge comes from. I have left that out because it is a new service integration rather than something already in the codebase, which is the line your non-goals draw. Happy to send it separately if you ever want it, and equally happy for the answer to be no. Adding a library is a module with the `LibraryProvider` shape plus one line in `all_providers()`. ## Verification - `tests/core/test_library_index.py`: id, ISBN and fuzzy matching, per-content-type provider selection, fail-open on provider errors, stale-cache reuse, TTL and fingerprint refresh, per-provider cache isolation, and the test-connection path including unsaved form values. - `tests/core/test_text_match.py`: ISBN variants and token matching. - `src/frontend/src/tests/libraryBadge.test.ts` and the added cases in `bookTransformers.test.ts`. - Python suite (3269) and frontend suite (206) green, plus ruff, ruff format, basedpyright, vulture, tsc, oxlint, oxfmt and the production build. |
||
|
|
fe99d4bb5b |
feat(search): add a configurable default content type (#1371)
Closes #1018. Worth correcting the issue first: the search tab is not hardcoded. `useContentTypePreferences` has persisted the user's choice to localStorage since #564, so a browser that has picked a tab already keeps it. What is missing is the other half the issue asks for, a default for a browser that has stored nothing, and a per-user override. `DEFAULT_CONTENT_TYPE` is a user overridable select next to `BOOK_LANGUAGE`, so it follows the same path: global value in Settings, per-user value in Search Preferences, resolved with `user_id` in `/api/config`. The frontend uses it only when this browser has no stored choice, which is captured before the existing effect writes one, so nothing changes for anyone who has already picked a tab. The resolution is a pure function in `utils/contentTypePreference.ts` rather than logic inside the hook, since vitest here has no jsdom and the existing tests cover resolvers like `resolveDefaultLanguageCodes` the same way. One small move in `App.tsx`: the `config` state was declared below the hook that now reads it, so it moved above it. ## Verification - `tests/core/test_config_api.py`: the payload carries `default_content_type` and reads it with the user's id. - `tests/core/test_admin_users_api.py`: the key appears in the per-user search preferences list. - `src/frontend/src/tests/contentTypePreference.test.ts`: a stored tab wins, combined mode survives, the server default applies when nothing is stored, and an unrecognised value falls back to ebook. - Python suite (3163) and frontend suite (201) green, plus ruff, ruff format, basedpyright, vulture, tsc, oxlint, oxfmt and the production build. |
||
|
|
d978896142 | fix(auth): rename the API_KEY env var to SHELFMARK_API_KEY (#1374) | ||
|
|
3b280009ae |
feat(auth): static API_KEY (env) accepted as Bearer or X-Api-Key, cookie or key (#1366)
Supersedes #1353, per the discussion in #1352: one `API_KEY` environment variable; when set, a request carrying it is authenticated as the first admin, and cookie sessions keep working exactly as before (cookie **or** key). Nothing else changes. No table, no UI, no settings-tab switch, no per-user keys. ## What - `API_KEY` (env). Unset → the feature is off and none of the new code runs. - `Authorization: Bearer <key>` or `X-Api-Key: <key>` on any existing `/api/*` route authenticates that request as the first admin in `users.db` (`ORDER BY id`), or as a bare admin identity (`user_id="api"`, `is_admin=True`, no local user row) if the install has no admin yet. Per request only; nothing is persisted; the admin's role is read live, so deleting or demoting that user takes effect on the next request. - Both headers are checked and either may match. That is what makes the key usable behind a reverse proxy that injects its own `Authorization` header (oauth2-proxy, Authelia, forwardAuth): send the key in `X-Api-Key`. - A credential that is **not** the key is ignored and the request continues on the normal session path, so proxy-forwarded tokens are unaffected. Without a valid session such a request gets the usual `401 {"error": "Unauthorized"}`, identical to a request with no credential, so there is nothing to probe. ## How - `shelfmark/config/env.py`: `API_KEY = os.getenv("API_KEY", "").strip()`. - `shelfmark/core/api_key.py`: `extract_api_key_candidates()` (Bearer token if the scheme is Bearer, then `X-Api-Key`) and `matches_api_key()` using `hmac.compare_digest` on bytes. - `shelfmark/core/user_db.py`: `UserDB.get_first_admin()`. - `shelfmark/main.py`: `api_key_auth_middleware` (`before_request`, registered before `proxy_auth_middleware`, which early-returns for keyed requests). Only `/api/` paths; `/api/health` and `/api/auth/*` exempt; no-op when `API_KEY` is unset or the auth mode is `none`. On a match it mirrors the proxy-auth pattern: `session.clear()` then populate `user_id` / `is_admin` / `db_user_id` for this request, `permanent = False`, `modified = False`, `g.api_key_auth = True`. An `after_request` hook guarantees no `Set-Cookie` is written for a keyed request even if a handler dirties the session. - `docs/api-access.md` (new), the `API_KEY` entry in `docs/environment-variables.md`, and a README link. ## Security - Constant-time compare; the key is never logged or echoed. - Keyed requests never mint or refresh a session cookie and ignore any cookie sent with them (a non-admin cookie plus the key yields admin for that request; the browser's own session is left untouched and usable). - The mismatch path touches neither the session nor `g`, so a stray bearer on a browser request can neither log the user out nor change how their cookie is refreshed. - Store errors during the admin lookup fail closed (`500 {"error": "Authentication error"}`), never to anonymous. - Verified against Flask's `save_session` / `should_set_cookie` ordering, and under auth modes `none`, `builtin`, `proxy`. ## Tests `tests/core/test_api_key_env.py` (36): extraction and matching; first-admin lookup; middleware behaviour on a guarded route and an admin route, with and without a user_db, `X-Api-Key`, both-headers combinations, no `Set-Cookie` when a handler dirties the session, incoming non-admin cookie ignored, browser cookie still usable after a keyed request, security headers, store error → 500, mismatch → guard's 401 / cookie path / permanent cookie untouched, unset → off, exempt paths and path probes, `none` and `proxy` modes, deleted and demoted first admin, a keyed write passing the guard. Existing auth suites unchanged. All CI gates green on the fork: https://github.com/gavinmcfall/shelfmark/pull/2 (CI-only draft). Also exercised against a running instance: 47 scripted checks including 150 concurrent requests, proxy-mode switching through the key, an unset-key restart, and a log scan for the key. ## Naming `API_KEY` as discussed. If you'd rather namespace it (`SHELFMARK_API_KEY`) to avoid clashing with other tools' env vars in shared compose files, it is a one-line change; say the word. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
|
ab3aa9a8b0 |
fix(requests): reject non-object items in the batch endpoint (#1369)
`POST /api/requests/batch` checks that `requests` is a non-empty list and then hands each element to the shared preparation helper, which calls `.get()` on it. A bare string, number or null in the list raises `AttributeError` and the caller gets a 500, while `POST /api/requests` answers 400 with a message for the same mistake. This validates the element type beside the existing list check. One bad item rejects the whole batch rather than being reported per item, which matches the endpoint's current contract: every other failure path already aborts the batch with a single error body, as `test_batch_create_requests_is_atomic` asserts. Responses for valid payloads are unchanged. ## Verification - `tests/core/test_request_routes_api.py`: a case per bad shape (int, str, null, list), plus a mixed valid and invalid batch that asserts nothing was created. All fail on current main with a 500 and pass here. - Full suite (3168), ruff, ruff format, basedpyright, vulture green. |
||
|
|
c1315a2b23 |
fix(download): check task ownership before serving queued files (#1357)
`/api/localdownload` resolves the file through the live queue and returns it before checking who owns the task; the owner check only runs on the download-history fallback, once the task has aged out of the queue. Task ids are source ids, so two users who searched the same book can end up with the same id. This applies the same rule on the queue path, reusing the `_task_owned_by_actor` helper the cancel/retry/priority routes already use, so both paths answer a non-owner with the same 404. Admin behaviour is unchanged. The 404 matches what this endpoint's history path already returns for a non-owner rather than the 403 `download_not_owned` the cancel/retry/priority routes use, happy to switch it if you prefer consistency with the siblings instead. ## Verification - `tests/core/test_activity_routes_api.py`: the owner still receives their queued file; a different user receives 404. The new case fails on current main and passes here; the existing history-fallback test is unchanged. - Full suite (3148), ruff, ruff format, basedpyright, vulture green. |
||
|
|
7934924678 |
fix(queue): don't stamp CANCELLED over a finished download (#1361)
`cancel_download` reads the task status under the queue lock, releases it, and only then writes CANCELLED through `update_status`. A download that finishes in that window has its COMPLETE overwritten. The queue and the UI show the task as cancelled while the file is already on disk, and the terminal hook fires for both statuses. The check and the write now happen in a single lock hold. Because the lock is non-reentrant and the terminal hook has to run after it is released (the stall canceller depends on that), the lock-held part of `update_status` moved into a small private helper that both paths share; `update_status` is a thin wrapper over it. A cancel arriving once the task is already terminal still returns `False`. ## Verification - `tests/core/test_queue.py::test_cancel_does_not_overwrite_a_download_that_finished_first`: a worker thread completes the download while the cancel is in flight, with the handover driven by events rather than sleeps. The task stays complete. Fails on main, passes here. - Full suite (3147), plus `tests/download/` and `tests/core/test_download_api_guardrails.py`, ruff, ruff format, basedpyright, vulture green. |
||
|
|
a6204a318e |
fix(oidc): reject backslash paths in the return_to sanitizer (#1359)
The OIDC `return_to` sanitizer rejects values starting with `//` and then relies on `urlsplit` to catch anything carrying a netloc. A value such as `/\host` has no netloc, so it is stored in the session and used as the post-login redirect target and browsers resolve the backslash as a path separator, which lands the user outside the app after a successful login. `_normalize_return_to` now also rejects values whose path contains a backslash. That matches the frontend sanitizer in `authRedirect.ts`, which parses with `URL` and already discards those forms, so the two ends agree again. The check covers the path only, so query and fragment backslashes still round-trip, and it also catches the script-root case where `/app/\host` strips to `/\host`. ## Verification - New cases in `tests/core/test_oidc_routes.py` cover the rejected forms, including under a script root, and confirm `/`, `/settings` and `/search?q=x#frag` are unaffected. They fail on current main and pass here. - Full suite (3155), ruff, ruff format, basedpyright, vulture green. |
||
|
|
545480c557 |
fix(download): default is_admin to False in the request policy guard (#1358)
`_resolve_policy_mode_for_current_user` reads `session.get("is_admin",
True)`, so a session carrying `user_id` but no `is_admin` key skips the
request policy entirely, while every other admin check in the codebase
defaults the key to `False`.
This uses the same default here. Every authenticated login path
(builtin, CWA, proxy, OIDC) writes `is_admin` into the session, and
`AUTH_METHOD=none` is already short-circuited one line earlier, so
sessions from those flows behave exactly as before.
## Verification
- `tests/core/test_request_routes_api.py::TestDownloadPolicyGuards`: a
session without `is_admin` now gets `policy_requires_request` and
nothing is queued. Fails on current main, passes here.
- Full suite (3147), ruff, ruff format, basedpyright, vulture green.
|
||
|
|
127dd82615 |
fix(users): apply user updates only after the payload validates (#1360)
`PUT /api/users/me` and `PUT /api/admin/users/<id>` write the new password hash, and then the profile fields, before the rest of the payload is checked. When the request is rejected further down as an invalid role, an admin-only setting, an invalid settings value, the route answers 400 with those writes already committed, so the caller sees an error while the password has in fact changed. Both routes now validate the whole payload before touching the database, and the password hash is folded into the same `update_user` call as the other fields so the field write is a single transaction. Error messages, status codes and the order they are reported in are unchanged. ## Verification - New tests in `tests/core/test_self_user_routes.py` and `tests/core/test_admin_users_api.py` assert that a rejected update leaves the password, profile fields and role as they were, plus a positive case that a valid payload still applies all three. They fail on current main and pass here. - Full suite (3150), ruff, ruff format, basedpyright, vulture green. |
||
|
|
acd59f7cbb |
feat(auth): provision proxy users as non-admin once an admin exists (#1356)
With `AUTH_METHOD=proxy` and no admin group configured, every user the proxy authenticates for the first time is provisioned as an admin (`is_admin = True` unless the user already exists in `users.db`). The intent to never lock an instance out makes sense, but the effect is that anyone the SSO gate lets through becomes an administrator. On an instance shared with family or a small community that is a footgun; I hit it when the first invited reader landed as an admin. This keeps the guarantee and removes the footgun: the first account is still provisioned as an admin while the instance has no admin at all, and later first-time users follow a new `PROXY_AUTH_DEFAULT_ROLE` setting (Security tab / env), default `user`. Known users keep their stored role; the `PROXY_AUTH_ADMIN_GROUP_NAME` path is unchanged and still takes precedence. I couldn't find a way with Cloudflare access to pass this along. Changes: `UserDB.has_admin()`, `_proxy_default_is_admin()` in the proxy middleware, the new `SelectField` beside the other proxy settings, the regenerated `docs/environment-variables.md` entry and a row in `docs/reverse-proxy.md`. Compatibility: the default moves from "everyone admin" to "first admin, then users". Accounts already in `users.db` are unaffected; new SSO users on an existing instance become regular users unless `PROXY_AUTH_DEFAULT_ROLE=admin` is set. If you would rather ship this purely opt-in I can flip the default to `admin`. ## Verification - `tests/core/test_auth_api.py::TestProxyProvisioningRole`: first user admin / second user not; `PROXY_AUTH_DEFAULT_ROLE=admin` restores the old behaviour; an admin from another auth source counts as "an admin exists"; a known user keeps their role whatever the default. - Full suite (3094), ruff, ruff format, basedpyright, vulture green. - Running on my own instance since 2026-09-19. |
||
|
|
44f4e13cce |
refactor: extract the per-source release search out of /api/releases (#1355)
The `/api/releases` route carries an inner `_search_source_releases` helper that builds the search plan for one source, logs the planned query type, runs the search and turns `SourceUnavailableError`/operational errors into an error message instead of raising. Anything outside the route that wants to search one source with exactly those semantics has to go through Flask today. This moves that helper into `shelfmark/core/release_search.py` as `search_source_releases()` and has the route delegate to it. Behaviour is unchanged: same plan construction (including the caller's `user_id`, so per-user default languages still apply), same logging, same error-to-message handling. It is the refactor half of #1047 by @InfiniteAvenger, split out on its own as you asked for other PRs (#1318). Their authorship is preserved on the commit; I rebased it onto current `main` and added tests. ## Verification - `tests/core/test_release_search.py`: unknown source → `"Unknown source: …"`, `SourceUnavailableError` and operational errors → `"<source>: <error>"`, success path forwards `expand_search` / `content_type` and returns the source instance, the plan receives languages / manual query / indexers / `user_id`. (These tests are type-annotated; happy to strip the annotations if you prefer the suite's bare style.) - Full suite, ruff, ruff format, basedpyright, vulture green; the existing `/api/releases` route tests are unchanged and pass. Co-authored-by: InfiniteAvenger <calebewest02@gmail.com> |
||
|
|
e5dd34ae0e |
fix: unbreak main and follow up on the Blackhole handoff review (#1346)
DownloadHistoryService.record_download and updated the single production caller, but not the eleven in the test suite, leaving main red with 32 failures. Pass None, which is what the pre-#1336 behaviour recorded. For the Blackhole handoff (#1345): add_download publishes the torrent before the cancel check runs, and BlackholeClient.remove() is a no-op, so the watcher picks the file up regardless. Reporting a bare "Cancelled" hid that from the user. Name the completed handoff in the cancellation message instead, drop the _handle_cancelled_download call whose usenet branch cannot apply to a handoff-only client, and record why the orchestrator no longer verifies HandoffResult.path. Finally, make tests/direct_download a package: test_libgen_extract.py imports tests.libgen.sample_html across test directories, so without an __init__.py pytest named its modules by bare basename and a same-named module elsewhere would collide. |
||
|
|
c53545d9fe |
Add configurable word separator for naming templates (#1333)
Closes #1230 ## What Adds a "Word Separator" setting (Space / Dot / Underscore / Hyphen / Custom) that replaces internal whitespace in each naming-template placeholder's rendered value — e.g. `{Author}` renders "Arthur.Conan.Doyle" instead of "Arthur Conan Doyle" when Dot is selected. This follows option 2 from the issue rather than inventing new dotted-keyword template syntax (`{Author.}`), since it's a smaller surface: one setting applies uniformly across all four templates (books/audiobooks × rename/organize) instead of needing a parallel token for every existing one. ## How it works - Literal characters typed into the template itself (e.g. the `.` in `{Author}.-.{Title}`) are never touched — only whitespace *inside* a placeholder's resolved value is affected. - Default is "Space", which is a no-op: existing templates produce byte-identical output after this change (verified via the existing test suite, unmodified, still passing). ## Where - `shelfmark/core/naming.py` — `word_separator` param on `parse_naming_template` / `build_library_path`. - `shelfmark/download/postprocess/policy.py` — `get_word_separator()`, mirroring the existing `get_file_organization()` accessor. - `shelfmark/download/postprocess/transfer.py` — wires the resolved separator through the four existing template-rendering call sites. - `shelfmark/config/settings.py` — new `Word Separator` / `Custom Word Separator` fields next to the existing naming-template fields. - `src/frontend/.../namingTemplatePreview.ts` + `NamingTemplateField.tsx` — the settings UI has its own TS mirror of the Python renderer for the live preview; updated it in lockstep so the preview doesn't lie about what the separator will actually do. - Tests added on both sides (pytest + vitest). ## Testing - `uv run pytest tests/core/test_naming.py tests/core/test_destination_file_organization.py` — all pass, including new cases. - `uv run pytest` (full suite) — same pre-existing failures as on `main` before this change (browser/network-dependent bypass & e2e tests unrelated to this diff), everything else green. - `uv run ruff check` / `ruff format --check` / `basedpyright` — clean. - `npm run lint` / `format:check` / `typecheck` / `test:unit` (196 tests) — clean. |
||
|
|
35037b35fd |
fix(irc): search by surname, and rank the answer by author (#1331) (#1332)
Fixes #1331. A search bot ANDs every term against a filename, so the given name is the term that empties the result set. Measured against irchighway's #ebooks: "Revelations David Petrie" is answered "no results", "Revelations Petrie" returns 9 matches, 6 of which parse, all filed as "D Petrie". The query now carries the title and the surname, read off the search variant so the ISBN fallback and a manual query - which set author="" on purpose - keep their current shape. Title-only, the shape #1295 settled on for Prowlarr, does not transfer: the bot caps an answer at 1000 matches, and a bare "Revelations" hits that cap with 923 parsed rows across 500 authors, so the cap itself can drop the wanted book. The surname is the token the two spellings share and it keeps the answer small. The full author then orders what comes back, reusing author_affinity from #1295, since a surname also matches a different author who shares it. It sits under server availability the way indexer priority does in #1295: a download addresses one named bot and waits 120s for it, so a match from a bot that has left the channel must not outrank a mismatch that can answer. Ranking runs on the way out rather than before the cache, because one query identity is shared by every book that produced that query. Two things found while testing: - The parser writes the literal "Unknown" when a filename has no " - " separator (parser.py:168). Ranked literally that sorts as a wrong author, so author_affinity's middle tier was unreachable here; it is now read as absent. 5 of those 923 rows are affected. - author_affinity moves to shelfmark/core/author_match.py, unchanged, so IRC does not import from the Prowlarr package. Prowlarr behaviour is untouched and its tests pass as they are. The three IRC assertions in the #1252 regression file move to the surname form. The invariant they pin - one contributor's name reaches the query, never the whole credit list - is unchanged. Tested with make python-lint, python-format, python-dead-code, python-typecheck and python-test, and end to end against irchighway with the patched source: it posts "Revelations Petrie" and returns 6 releases. |
||
|
|
1e3fd48b8b |
fix: share rotating log file handlers (#1316)
This patch shares (for each log file) the `RotatingFileHandler` for logging across all modules, reducing the number of open file descriptors from ~78 to 1. I had originally assumed this issue was a resource leak, but it seems to just be a large fixed number of file descriptors. So this change mostly just (1) shrinks the number of open file descriptors to a reasonable level and (2) prevents two modules in the same process competing to write to a log file. |
||
|
|
c576003319 |
feat(naming): add {FirstAuthor} template token (#1322)
Closes #930. ## What New `{FirstAuthor}` naming-template token. It renders only the first author when metadata lists several ("Author1, Author2, Author3"), so multi-author books can be filed alongside the rest of that author's work instead of getting their own "Author1, Author2, ..." folder. ``` {Author} -> Terry Pratchett, Neil Gaiman {FirstAuthor} -> Terry Pratchett ``` ## How - Added to `KNOWN_TOKENS` in `shelfmark/core/naming.py`, positioned before `author` so `{FirstAuthor}` isn't parsed as literal `First` + `{Author}`. - Derived inside `parse_naming_template` from the existing `Author` value (split on `,` / `;`), so every caller — folder transfer, rename, the settings preview — picks it up with no extra wiring. An explicit `FirstAuthor` key in the metadata still wins if one is ever passed. - `{Author}` behaviour is unchanged. - Frontend `namingTemplatePreview.ts` token list + `KNOWN_TOKENS` kept in lockstep (there's a test enforcing that), with a matching `firstAuthor` helper. - Settings field descriptions + `docs/environment-variables.md` list the new token. ## Known limitation A lone author written `Last, First` is split on the comma too and renders as `Last` — the source metadata doesn't mark which form it is. Called out in the token help text and covered by a test. `{Author}` remains available for anyone who wants the raw string. ## Checks - `make python-test` — 2963 passed - `make python-lint` / `make python-format` / `make python-typecheck` / vulture — clean - `make frontend-test` — 187 passed · `frontend-lint` / `frontend-format` / `frontend-typecheck` — clean 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
3d7ea40088 |
fix(search): reach the server's deadline, query one author (#1285, #1252) (#1287)
Two independent reasons a working search reported failure to the user. 1. The client gave up before the server did (#1285) `/api/releases` bounds one release search with RELEASE_SEARCH_TIMEOUT (default 300s) and answers a spent budget with a sentence naming the real cause - the machinery added for #1276. The frontend then aborted the direct_download search at a hard-coded 180s, so it always won the race: the user saw "Request timed out. Check your network connection or proxy configuration." instead, and raising RELEASE_SEARCH_TIMEOUT changed nothing they could observe, the 180s being baked into the hashed bundle inside the image. - /api/config reports the effective (clamped) budget, and the client derives its abort from it plus a margin, so the server always answers first. - Direct-mode search shows what the server actually said. Every non-auth failure was relabelled "Unable to reach download source. Network may be restricted or mirrors blocked.", which discarded the explanation and blamed the user's network. ApiResponseError now carries `serverMessage`, set only when the server explained itself, so the status-line placeholder still falls back. Two latency fixes for the cost that made the timeout reachable at all: - Fetch each distinct AA search URL once per search. The language-filter retry re-runs every title variant, and with DIRECT_DOWNLOAD_LANGUAGE_FROM_PATH on both passes build a byte-identical URL - behind DDoS-Guard each repeat is a fresh browser solve. - Drop the solve-only bypass method. `_bypass_method_cdp_gui_click` opens with exactly that call and returns the moment it works, so the entry ahead of it could only repeat the half that had already failed, plus the backoff before the method that does work started. Reported at 0/19 successes and ~5.5s of each ~26s solve against DDoS-Guard. 2. The query carried every contributor, not one author (#1252) `_pick_search_author` returned `book.search_author` verbatim while the authors[] fallback beside it deliberately narrowed to the first name before a comma. Both fields routinely arrive holding every contributor joined with ", ": the frontend builds `book.author` as `authors.join(', ')` for display (bookTransformers.ts) and the release modal sends that display string straight back as the `author` parameter, and `browse_record_to_book_metadata` and the manual-search branch both split the joined text into `authors` while still passing the unsplit string as `search_author`, so the split was never used. A book whose metadata lists translators was therefore searched for as Blindness Jose Saramago, Giovanni Pontiero, <persian translator> which matches nothing on Anna's Archive. The bypass succeeds, the search comes back empty, and the user is told the book has no releases. Narrowed in one place, `search_plan.first_author`, so the two branches cannot drift apart again, and applied to the IRC source, which built its query with the same verbatim preference. Hardcover is unaffected: it already sets `search_author` from `_simplify_author_for_search(authors[0])`, which resolves "Last, First" itself and never yields a multi-author string. |
||
|
|
69ff0d6a78 |
fix: trim a credit list in search_author to the first name (#1290)
Fixes #1252 for the case in the second report. `_pick_search_author` returns `search_author` untouched but trims `authors[0]` to its first comma-separated name. So the same credit list searches differently depending on which field carries it: ``` via authors[0] -> "Blindness Jose Saramago" via search_author -> "Blindness Jose Saramago, Giovanni Pontiero, Zohreh Eftekhari" ``` Anna's Archive answers the second one with nothing. That is the query in @theDoz12's log, and it explains the shape of the report: the bypass succeeds, the search runs, and the UI still says no releases. Nothing in the download path is broken, the query simply cannot match. Measured against live AA on 1.3.14, same book, same source, only the field carrying the author changed: | query | releases | | --- | --- | | `Blindness Jose Saramago, Giovanni Pontiero, Zohreh Eftekhari` | 0 | | `Blindness Jose Saramago` | 49 | | `Blindness` | 50 | With the patch the second form is produced from either field, and the same search returns 49. Three regression tests added, including one that asserts both fields yield the same query. On `tests/core/test_search_plan.py` the run goes from 5 failures to 3; the 3 that remain are the language tests, which fail identically with and without this change on my machine. Worth saying what this does not cover: the first report in that issue ends with `Found 2 releases via ISBN` and still shows nothing, so that one is a different fault further along. I could not reproduce it here. |
||
|
|
97e289ae13 | fix: search, Prowlarr and qBittorrent follow-ups (#1276, #1283) (#1284) | ||
|
|
f441b85da2 |
feat(packs): inspect multi-book releases and file each book separately (#1270)
## Multi-book packs: inspect a release before download and file each book separately Closes #576 ### Problem One queued release is always treated as one book. When a torrent is actually a whole series (`Series/Book 1 - Title/…`, or a flat folder of `Series 1.0 - Title.m4b` files), post-processing walks the whole tree, flattens every file into one list and renames them `Title - 01…10` under the searched book's `{Author}/{Title}`. Audiobookshelf then sees a single 10-file "book" and the user has to re-file everything by hand. ### What this does Most releases expose their file list *before* anything is downloaded, so the split is decided up front and approved by the user, then the download is fire-and-forget: 1. **Inspect** – clicking a release's download button now calls `POST /api/releases/inspect` first. A new optional `DownloadHandler.list_files(release_data)` hook returns the release's files without downloading: - **AudiobookBay** reads the torrent file table off the detail page it already fetches (the page is now cached for 120 s, so inspect + download cost ABB one request). - **Prowlarr** parses `info.files` from the `.torrent` it already fetches (the existing 120 s torrent-fetch cache is reused). Magnet-only and usenet releases report "can't inspect". - Other sources default to `None`. 2. **Review** – if the plan contains more than one book, the Find Releases modal swaps the list for a review panel: one row per book with editable title / series position / year, expandable file lists, non-book sidecars (`.txt`, covers) shown as ignored, a "Treat as a single book" switch, and **Download N books**. Single-book releases queue immediately, exactly as before. 3. **File** – the approved plan travels with the task (`DownloadTask.book_plan`, retry-safe) and post-processing files each book through the existing transfer code, one book at a time (`dataclasses.replace(task, title=…, series_position=…, year=…)`), so organize/rename templates, part numbering (now scoped per book), hardlinks, torrent copy-preserve and usenet handling are unchanged. Status reads `Complete (N books, M files)`. 4. **Fallback** – when a release can't be inspected the user gets a toast, and a small "Multi-book pack" toggle in the modal header forces a heuristic split (subfolder = book, or one book per file when the file names carry series positions). Planning lives in `shelfmark/download/postprocess/packs.py` and is shared by the inspect endpoint and post-processing, so what the user approved is what gets filed. The name parser strips `Book 3 -`, `03 -`, `1.0 -`, `3.`, `[03]`, `#3`, a leading series name, labels like "An Expanse Novella -", repeated titles (`Gods of Risk 2.5 - Gods of Risk`) and a trailing `(Year)`; author and series name come from the book that was searched, and the searched book's own series position is never applied to its siblings. ### Files - `shelfmark/download/postprocess/packs.py` (new) – `PackFile/PackBook/PackPlan`, `plan_pack`, `parse_pack_book_name`, `group_files_into_books`, `match_plan_to_files` - `shelfmark/core/release_inspect_routes.py` (new) – `POST /api/releases/inspect` - `shelfmark/release_sources/__init__.py` – `DownloadHandler.list_files` hook - `shelfmark/release_sources/audiobookbay/{scraper,handler}.py` – detail-page cache, `extract_file_list`, `list_files` - `shelfmark/release_sources/prowlarr/handler.py`, `download/clients/torrent_utils.py` – `extract_file_list_from_torrent`, `list_files` - `shelfmark/core/models.py`, `download/orchestrator.py` – `multi_book` / `book_plan` fields, queue + retry serialization - `shelfmark/download/postprocess/transfer.py`, `pipeline.py`, `outputs/folder.py` – per-book transfer branch and status message - `src/frontend`: `components/PackReviewPanel.tsx` (new), `ReleaseModal.tsx`, `App.tsx`, `services/api.ts`, `types/index.ts`, `utils/releasePayload.ts` (payload builder moved out of `App.tsx`), `utils/packReview.ts` - `docs/dev/release-sources-plugin-guide.md` – documents the `list_files` hook ### Out of scope (follow-ups) - Listing files from an NZB (Shelfmark already fetches the bytes; `<file subject>` names are noisy) - Inspecting magnet links via qBittorrent's files API after a paused add - BookLore / email outputs (they ignore `book_plan`; noted in code) - The combined ebook + audiobook flow ### Testing **Automated** (`make checks`, `make python-test`, `make frontend-test` all green; the only failures on my machine are the pre-existing `tests/config/test_entrypoint_permissions.py` cases, which need bash ≥ 4 and fail identically on `main` under macOS bash 3.2): - `tests/download/test_packs.py` – name parsing (markers, series name, novella labels, repeated titles, bare numeric titles like `1984`), nested / flat / mixed / deeper-nested packs, single wrapping folder not treated as a pack, plan-to-disk matching with basename fallback - `tests/core/test_processing_packs.py` – full `post_process_download` runs on a real temp filesystem: approved plan files each book under its own `{Author}/{Title}`, heuristic split of a nested pack, searched book's series position does not leak, multi-file book inside a pack keeps `- 01/- 02` per book, hardlinked torrent pack leaves the seeding tree intact, no pack fields ⇒ behaviour unchanged, single group degrades to the searched title, status message - `tests/core/test_release_inspect_routes.py` – plan response, not-inspectable, handler errors never 500, unknown source / missing `source_id` ⇒ 400, login required - `tests/audiobookbay/test_file_list.py` – file-table scraping from real ABB markup (multi-file and single-file pages), handler host validation, one page fetch shared by magnet + file list - `tests/prowlarr/test_torrent_file_list.py` – multi-file / single-file `.torrent` parsing, handler behaviour for torrent URL vs magnet vs usenet vs cache miss - `tests/download/test_orchestrator_pack_fields.py` – queue-time parsing and retry round-trip - Frontend: `releasePayload.test.ts`, `packReview.test.ts` (vitest) **Manual, on a real deployment** (arm64 image built from this branch, run as a side container next to production with the same qBittorrent / Audiobookshelf setup, `FILE_ORGANIZATION_AUDIOBOOK=organize`, hardlinks on): - AudiobookBay "The Expanse Complete 2.0" (7.87 GB, 36 files): clicking download opened the review panel in ~1 s showing **18 books · 18 files · 18 files ignored** (the `.txt` sidecars), with series positions 0.1–9.5 and years parsed from the file names; novella labels stripped ("The Churn", "The Butcher of Anderson Station"). Editing a title in the panel works. Confirming queued one task; the magnet resolved from the cached page in ~30 ms; after the download the task reported `Complete (18 books, 18 files)`, 18 hardlinks landed as `audiobooks/James S. A. Corey/<Title>/<Title>.m4b`, the torrent kept seeding, and Audiobookshelf scanned each folder as its own book (title, author, embedded chapters). - A second pack ("Expanse [01 - 9.5]", `Title N - Title` naming) was inspected to verify the repeated-title rule and the Back button, without downloading. - Single-book releases still queue immediately with no extra UI. |
||
|
|
ff06a1a581 |
fix(search): follow-ups to per-user book languages (#1267)
Review follow-ups to #1255, all in the code that PR touched. Drop the dead user_id from the Prowlarr retry path. ProwlarrSource.search never reads plan.languages, and _refresh_release builds a synthetic book with no titles_by_language, so the title variants came out identical with and without it. It also should not language-filter: it re-finds one exact release by its guid. Pin the tab move in tests. BOOK_LANGUAGE moved from the General tab to Search Mode with no migration, which only works because both tabs persist into the same settings.json. Nothing asserted that, so splitting the files later would silently reset every install to ["en"]. Covers the stored value, a fresh install, and ENV precedence. Stop the UI inventing a default language. An empty BOOK_LANGUAGE is a deliberate "no default filter" that the backend preserves, but the two frontend call sites replaced it with the first supported language, so the filter said English where the server filtered nothing. resolveDefaultLanguageCodes now falls back only when the value is absent. Keep the normalized value for every validated search key. validate_user_settings gated the write-back on a hand-maintained subset of the keys the search validator recognises, so METADATA_PROVIDER_COMBINED, SHOW_COMBINED_SELECTOR and FORCE_COMBINED_SEARCH were validated and then stored raw -- a padded provider name was accepted and persisted with its padding. Reuse the validator's own key set instead. Skip blank language entries rather than rejecting them, so "" and "en," mean the same as [] and ["en"] instead of erroring on an unnamed language. Extract resolveListOverride for the list-override detection that was copy-pasted between the two user-settings sections, and mention languages in the Search Preferences section description. |
||
|
|
463ef49ac3 |
feat(search): let each user pick their own default book languages (#1255)
## Why `BOOK_LANGUAGE` is a per-reader property, not a per-instance one. On a shared install one household member searches in German while another wants English and German — today whoever changes the setting changes it for everyone, and the only escape is re-picking languages in the filter on every single search. The per-user override machinery already carries `SEARCH_MODE`, the metadata providers and the default release sources, so the language default mostly had to opt into it. ## What changed **The field.** `BOOK_LANGUAGE` becomes `user_overridable` and moves from the **General** tab to **Search Mode**, next to the other user-overridable search defaults (per [review](https://github.com/calibrain/shelfmark/pull/1255#issuecomment-5391189094) — the first version had the Search section span two tabs, this one doesn't). Admins set it per user in the user editor, users set it in **My Account → Search Preferences**, and the Search Mode tab carries the usual "N users override this" summary. **No migration for the move.** `general` and `search_mode` both persist into `settings.json`, and a field's value is resolved through `load_config_file(tab)` for the tab it's declared on — so an install that already stores `BOOK_LANGUAGE` keeps its value. Checked against a `settings.json` written while the field still lived on General: the stored value resolves unchanged, a fresh install still gets `["en"]`, and `BOOK_LANGUAGE` in the environment still overrides both. **The two places the default is read.** - `/api/config` seeds the frontend's language filter, so it now resolves `BOOK_LANGUAGE` for the session user. - `build_release_search_plan` falls back to the default whenever a request carries no language filter — which is exactly what the filter's "Default" option sends. It takes an optional `user_id`, passed by `/api/releases` from the session and by the Prowlarr retry path from `task.user_id`, so a retry re-searches in the languages of whoever queued the download. **Validation.** Overrides go through `normalize_language()`, so `"German"`, `"ger"` and `"de"` all store as `de`, and an unknown language is rejected with a message naming it instead of being silently searched for. An empty list stays an empty list (a deliberate "no default filter"), `null` clears the override as everywhere else, and ENV still wins: with `BOOK_LANGUAGE` set in the environment the field reports `fromEnv` and overrides are ignored. **Scope.** Only the language default becomes overridable. The two format lists left behind under "Default Search Filters" stay admin-only — they describe what the library and its post-processing accept, not what a reader wants to read. There's a test pinning that. ## Verification - 2681 unit tests pass (2670 before, 11 added) - `ruff check`, `ruff format`, `basedpyright` over backend and tests, and `vulture` all clean; frontend lint, format, typecheck and 126 unit tests clean - `docs/environment-variables.md` regenerated via `scripts/generate_env_docs.py` (the `BOOK_LANGUAGE` row follows the field into the Search Mode section) - Manually against a two-user instance with builtin auth (first round, before the tab move): with user A on German and user B on English+German, `/api/config` returns each reader their own `default_language` and an unfiltered `/api/releases` plans the matching languages; an admin can set and read the same override for another user; clearing it falls back to the global value; a stray `"klingon"` is rejected; and `BOOK_LANGUAGE` in the environment overrides both users with the field marked `fromEnv` - After the tab move I re-ran the suites above plus the stored-value/fresh-install/ENV check described under "No migration for the move"; the behaviour it exercises is what the move could have broken Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
|
65e2e3be20 |
feat(audiobook): recognise .mp4 as an audiobook format (#1264)
## Problem Some trackers — MyAnonamouse in particular — distribute AAC audiobooks as per-chapter `.mp4` files. That's the same ISO-BMFF container as `.m4a`/`.m4b`, just with the generic extension (`ftyp isom`, audio-only). Today those releases: 1. show up in Prowlarr search results with **no format chip** — only the generic "Audiobook" icon, because no format could be inferred; 2. download successfully; then 3. fail post-processing with **"No book files found in download"**, because `.mp4` isn't in `AUDIOBOOK_FORMATS` (`shelfmark/core/utils.py`). Real example: MAM #627978, *The Martian* (Andy Weir, 2020 edition) — 142 files `0001 … 0142 Andy Weir (2020) The Martian.mp4` + `cover.jpg`, 305 MB. Every file is a valid AAC-in-MP4 chapter. Adding `mp4` to `SUPPORTED_AUDIOBOOK_FORMATS` in `settings.json` doesn't help since the hard-coded tuple is what post-processing scans against. ## Change - Add `"mp4"` to `AUDIOBOOK_FORMATS` (single source of truth — settings UI, Prowlarr parsing, IRC parser, archive extraction and post-download scan all derive from it), with a comment explaining why. - Add `".mp4"` to the two hand-maintained debrid `_BOOK_EXTENSIONS` lists (AllDebrid / Real-Debrid) so file selection matches. - Slot `mp4` into the IRC `AUDIOBOOK_FORMAT_PRIORITY` table right after `m4a` (same container family). - Update the documented default in `docs/environment-variables.md`. - New regression test `test_audiobook_multifile_mp4_chapters_are_book_files` modelled on the existing multi-file usenet test. ### Note for existing installs The legacy-default migration only widens configs that still hold the old `m4b,mp3` list, so users on the current widened default won't pick up `mp4` automatically — they'll need to tick it in Settings → Audiobook formats. New installs get it by default. Happy to extend the migration if you'd rather it be automatic. ## Testing - `ruff check` / `ruff format --check`: clean - `pytest tests/core tests/config tests/irc tests/prowlarr tests/download -m "not integration and not e2e"`: 2296 passed, new test + `test_audiobook_format_consistency.py` all green. The 10 failures in `test_entrypoint_permissions.py` / `test_orchestrator_stall.py` reproduce identically on untouched `main` on macOS and are unrelated. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
7d56624ab6 |
fix: group multi-file audiobooks that arrive as an archive (#1254)
Follow-up to #1237. \`rename_and_group\` only grouped when the source root was a directory, so a multi-file audiobook delivered as a single archive fell through to the flat path: a \`Book.zip\` of twelve chapters landed loose in the destination root with its original chapter names — the layout #1181 is about. The \`is_dir()\` guard was there to keep \`Book.zip/\` from becoming the folder name, but skipping the file case gives up the grouping instead of naming it. A non-directory source can only produce several book files by having been extracted (\`collect_staged_files\` returns a single-element list for every other file shape), so the archive stem is the release name and the suffix is packaging: group under \`Book/\`. Also regenerates the env docs for the new option and gives it the same \"do not use with ingest folders\" caveat Rename and Organize carries, since both now create directories in the destination. Tested: reverting only the source fix makes both new tests fail and the \`rename\` control case pass, so grouping stays opt-in. Full non-e2e suite green (2653 passed). |
||
|
|
f4421ff189 |
fix: preserve multi-file audiobook folders (#1237)
Pass the effective source root from `process_folder_output` into `transfer_book_files`, and have the transfer layer select a sanitized child directory named after that source root when an audiobook has multiple files and its organization mode is `none` or `rename`. Create that grouping directory before applying the existing hardlink/copy/move logic so operation accounting, torrent seeding preservation, collision handling, cleanup, and custom-script final paths continue to use the established production path. Completed multi-file audiobook torrents arrive as a directory whose chapter filenames may not identify the book, but folder output currently sends every discovered chapter directly to the configured destination in `none` and `rename` modes. This flattens chapters from unrelated books together and causes directory-oriented consumers such as Audiobookshelf to interpret individual chapters as separate books. A multi-file audiobook torrent in the default `rename` mode copies or hardlinks all supported chapter files beneath `<destination>/<original torrent directory>/` with their original chapter filenames, and places no chapters directly in the destination root; A multi-file audiobook in `none` mode receives the same source-folder grouping without renaming its chapter files. Fixes #1181 --------- Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
|
58a5b5ed27 |
fix: sync renamed CWA usernames safely (#1203)
## Summary - sync an existing CWA-backed user's username when CWA renames it - keep username collisions safe by assigning a stable `__cwa` alias instead of overwriting a local account - allow username updates through `UserDB` and cover rename/collision/repeat-sync behavior Fixes #1197. ## Testing - `uv run ruff check shelfmark tests` - `uv run ruff format --check shelfmark tests` - `uv run vulture shelfmark` - `uv run pytest tests/core/test_cwa_user_sync.py tests/core/test_user_db.py tests/core/test_admin_users_api.py tests/core/test_auth_api.py -k "cwa or update_user"` (36 passed) - `uv run pytest tests/ -x --tb=short -m "not integration and not e2e" --ignore=tests/config/test_entrypoint_permissions.py -q` (2445 passed, 5 skipped) The entrypoint permission tests were excluded locally because macOS ships Bash 3.2, which does not support the `${1,,}` expansion used by `entrypoint.sh`; the same failure reproduces on an unchanged checkout. `make python-typecheck` also currently reports the existing `settings.py:147` callback return-type mismatch on the unchanged base. Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
|
d0e008adde |
Stop dropping audiobook releases that are not m4b or mp3 (#1199)
An IRC audiobook search returned nothing while OpenBooks, reading the same @search answer from the same channel, listed results. Three separate defects were discarding them. The audiobook format list was maintained by hand in four places and had drifted. The settings UI offered only m4b/mp3/m4a/zip/rar, and that list is the only one a user's config can be built from, so flac, opus, ogg, aac, wav and wma were unreachable everywhere — even though the IRC parser recognized them, the IRC sorter ranked them (dead code that could never fire), archive extraction knew them and Prowlarr searched for them. A FLAC audiobook was invisible in search and, if it arrived anyway, rejected after download as "format not supported". AUDIOBOOK_FORMATS and ARCHIVE_FORMATS now live once in core.utils and every layer derives from them, which also restored the missing .opus in the post-download scan's trackable extensions. Widening the default alone would not have reached anyone already affected: initialize_default_configs() writes field defaults only when a tab has no config file yet, so an existing install keeps its persisted m4b/mp3 list forever. migrate_audiobook_formats rewrites a list that still matches the old default exactly and leaves every other value alone — re-enabling formats someone had deliberately turned off would be worse than leaving them narrow. The IRC parser filtered by file extension alone. Multi-file audiobooks ship as a .rar or .zip of MP3s, which matched neither SUPPORTED_FORMATS nor SUPPORTED_AUDIOBOOK_FORMATS, so they fell out of the ebook bucket and the audiobook bucket both. Results are now classified before the format filter is applied: an audio extension means audiobook, an ebook extension means ebook, and for a container — where the extension says nothing about the contents — the release name decides. An ebook archive stays out of audiobook results. RESULT_LINE_REGEX matched \w+ after any dot, so a line carrying no file extension parsed as format "5mb" out of "::INFO:: 620.5MB", taking the title and the size down with it and guaranteeing every downstream filter dropped it. Any decimal size did this. The extension is now matched against the known formats, so such a line falls through to the simple pattern and comes back as "unknown", which the rest of the parser already handles. ALL_RECOGNIZED_FORMATS became an ordered tuple in the process: it was a set, so which extension won for a line naming two of them depended on set iteration order and could vary between restarts. Refs #1129 |
||
|
|
e320b7623d |
Fix LOG_LEVEL being ignored and Z-Library 503 cookie gate (#1188)
LOG_LEVEL never reached the app logger: env.py hardcoded the level to DEBUG or INFO, so INFO lines kept appearing under LOG_LEVEL=error. Read it from the env var and advanced settings, normalize unknown values to INFO, and expose it as a setting. entrypoint.sh now normalizes gunicorn's level too, so a typo falls back to info instead of stopping the container from booting. Z-Library gates the first hit on /md5/<hash> with a 503 whose only payload is a Set-Cookie; echoing that cookie back returns the 302 to the real page. html_get_page dropped it and re-ran the same rejected request on every retry, ending in "No download URL resolved". Retry once with the cookies the 503 issued. Fixes #1185 Fixes #1187 |
||
|
|
bb848f05bc |
fix/bypass stall watchdog (#1186)
- Fix protection bypass cancelled by stall detection at exactly 300s - make fixes - Try to fix Synology DELETE issues |
||
|
|
3c51b7cfaa |
Fix OIDC redirects on custom ports (#1180)
Fixes #1175 Preserves the forwarded host and port when Shelfmark builds OIDC callback URLs. The reverse-proxy examples now retain custom ports as well. Tests: - `uv run pytest -n 0 -q tests/core/test_proxy_headers.py tests/core/test_oidc_routes.py` - `uv run ruff check shelfmark/main.py tests/core/test_proxy_headers.py` - `uv run ruff format --check shelfmark/main.py tests/core/test_proxy_headers.py` |
||
|
|
816a735cde |
Add a {Language} naming template variable, and consolidate language resolution (#1142)
Fixes #1138 Fixes #1141 ## Problem Two language editions of one book resolve to the same canonical title, so they render to the same path and the second gets a `_1` collision suffix. Audiobookshelf treats a folder as exactly one library item, so the pair becomes a single book with both files as tracks and a summed runtime. Shelfmark already parses and displays the language. It just never reached the template engine. ## `{Language}` template variable A template like `{Author}/{Title}{ (Language)}/{Author} - {Title}` now yields: ``` /library/J K Rowling/Harry Potter (sv)/J K Rowling - Harry Potter.m4b /library/J K Rowling/Harry Potter/J K Rowling - Harry Potter.m4b ``` The untagged edition's path is byte-identical to today, so no existing layout shifts. Three details worth flagging: **The value is casefolded.** On a case-insensitive filesystem `(SV)` and `(sv)` would collapse back into one folder, reintroducing the exact collision being fixed. **Values meaning "we don't know" render nothing** rather than producing `Project Hail Mary (unknown)` folders. Anna's Archive reports that string literally (`direct_download.py`, `language = detected or "unknown"`). **The frontend wasn't sending the release language at all**, so the token would have stayed empty for exactly the audiobook sources in the report. Prowlarr and AudiobookBay do not put language in `extra` the way `direct_download` does, hence the payload plumbing. It reads `release.language`, never `book.language` — the latter is the provider's canonical edition and would mislabel a translation, with a regression test for that specifically. Not gated to audiobooks: Calibre-Web-Automated stages ingested files by basename and discards folder structure, so the rename (filename) template is the only lever those users have. Verified that form works: `J K Rowling - Harry Potter (sv).epub`. ## Language consolidation (#1141) Three release sources each carried their own alias map, all resolving to the same ISO 639-1 codes, alongside a bundled database that only one of them used. Adding a language meant editing three places. Aliases now live in `data/book-languages.json` beside the code and name they belong to, and `shelfmark/core/languages.py` resolves any of them — two-letter code, ISO 639-2 three-letter in either the bibliographic or terminological form, or English name. Prowlarr and AudiobookBay drop their tables. Direct Download keeps its own path-parsing heuristics, including the ambiguous short codes that collide with English words (`de`, `en`, `no`, `in`), and takes only the alias data. This also closes a coverage gap. MyAnonamouse offers 62 languages; Prowlarr mapped 37, and an unmapped code is *dropped* rather than passed through, so the other 25 carried no language at all — leaving `{Language}` empty and the collision unfixed for Latin, Farsi, Tamil, Urdu and the rest. Seven languages MAM offers had no database entry at all: Bosnian, Burmese, Estonian, Icelandic, Manx, Scottish Gaelic, Sanskrit. Also fixes the Traditional Chinese code, which used a U+2011 non-breaking hyphen. Nothing compares against the ASCII spelling today so it was latent, but it would silently defeat the first thing that did. ## Validation Verified end to end against a live Prowlarr and MyAnonamouse, not just unit tests. A real search returning both an English and a Swedish edition, through the actual `queue_release` → `DownloadTask` → naming path: ``` STEP 1 real MAM search -> 37 releases, languages: ['en', 'sv'] STEP 3 queue_release -> task.language='sv' STEP 4 build_metadata_dict -> metadata['Language']='sv' STEP 5 build_library_path -> /library/J K Rowling/Harry Potter (sv)/... two language editions resolve to DIFFERENT folders: True ``` The refactor is pinned by a snapshot of both per-source maps taken *before* they were deleted. All 131 aliases are asserted to still resolve to the same code, one parametrised test each, so a regression names the specific alias. Also verified: the filename-only template, the retry round-trip (`serialize_task_for_retry` → `_restore_task_from_retry_payload`, plus a legacy payload with no `language` key), and placeholder handling. Added a `KNOWN_TOKENS` ordering invariant test — `find_placeholder()` does a substring `.find()` in list order and nothing protected that contract, so a future token in the wrong position could silently shadow an existing one. And a lockstep guard on the frontend, since `KNOWN_TOKENS` is hand-duplicated in TypeScript. **One caveat worth stating.** Three MAM codes are confirmed by observation (`ENG`→`en`, `SWE`→`sv`, `MAL`→`ml`, the last from a real `[MAL / EPUB]` Tagore release). The remaining ~59 are derived from ISO 639-2 rather than observed, because MAM's catalogue is overwhelmingly English — enabling 27 extra languages still yielded only one non-English hit across 258 results. Mitigated rather than closed: both 639-2 variants are present for every language where they differ, and a wrong alias is an unused entry while a missing one loses the language. Happy to correct any code a maintainer knows differs. ## Test results 2056 Python tests pass (up from 1906). Frontend typecheck, lint, format and 126 unit tests pass. Pre-existing failures on my machine, unchanged by this branch and unrelated: `tests/bypass/` needs `seleniumbase`, and `tests/config/test_entrypoint_permissions.py` uses bash-4 syntax that macOS bash 3.2 rejects. --------- Co-authored-by: delize <4028612+delize@users.noreply.github.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
|
81a057c8a0 | Try and fix OIDC (#1127) | ||
|
|
91b1064689 |
fix(naming): respect subfolder order when assigning audiobook part numbers (#1123)
## Summary Audiobooks whose files are split across nested subfolders now keep their on-disk folder order when "Rename and Organize" assigns part numbers. Previously `natural_sort_key` keyed on `Path(path).name`, so identically named files from different folders (`001.mp3` in `00_Introduction/` vs `06_Side 6/`) collapsed together and the part sequence scrambled. ## Background zazizou reported in #1007 that the total file count was right but `06_Side 6/001.mp3` came out as part 1 instead of `00_Introduction/001_About.mp3`. `scan.py` flattens every nested file into one list and `assign_part_numbers` sorts it with `natural_sort_key`, which discarded the parent-folder prefix. The fix pads the existing `PAD_NUMBERS_PATTERN` numbers over the full relative path (`str(path).lower()`) rather than the basename, so nested audiobooks sort by folder first and then by filename. Flat directories are unaffected because their files share a prefix, and `natural_sort_key` is only called from `assign_part_numbers`, so nothing else changes. I updated the test that pinned the old basename-only ordering and added nested-subfolder and duplicate-basename cases; `pytest tests/core/test_naming.py tests/core/test_part_number_extraction.py` passes (102 tests). Closes #1007 Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> |
||
|
|
aba1a68dda |
Backport bug fixes from NemesisHubris/litfinder (addresses #999, #956, #1010, #1021, #1025, #1040) (#1066)
## Backport bug fixes from `NemesisHubris/litfinder` Forwards a curated set of bug fixes from [NemesisHubris/litfinder](https://github.com/NemesisHubris/litfinder) — a community fork of this project — that address open issues here. All commits preserve original authorship via `git cherry-pick`; this PR is a backport rather than original work. Each fix has been reviewed locally, lint/format-cleaned to match this repo's existing ruff config, and verified with the test suite. Rebrand strings, license switches, and features have been deliberately excluded. ### Upstream issues addressed - **#999** — Mirror URLs with query params no longer break search requests (strip query string/fragment in `normalize_http_url`) - **#956** — Apprise notifications now respect the configured proxy (proxy env vars injected before dispatch) - **#1025** — rTorrent: separate `RTORRENT_AUDIOBOOK_LABEL` setting, falls back to book label if unset - **#1010** — Stop button in Activity no longer makes the panel disappear (snapshot refresh on cancel) - **#1021** — Anna's Archive slow-download countdown now caps retries instead of looping forever - **#1040** — Empty destination directory cleaned up when write probe fails - **PR #1031** — Language detection from Anna's Archive distant path when listing metadata is missing ### Additional fixes (no open issue but clear bugs) - **fix: Python 2 `except` syntax across 27 files** — `except X, Y:` is a SyntaxError in Python 3 and prevents affected modules from importing at runtime. Mechanical sweep to `except (X, Y):`. - **fix(abb): info hash validation with magnet fallback** — adds SHA-1/SHA-256 hex validation on extracted info hashes; falls back to scanning the full page for a magnet link (e.g. posted in comments) when the table value is malformed. Also extends the exact-phrase fallback to manual queries and defaults the ABB listing language to `en` when missing, preventing valid results from being hidden by the language filter. Includes a small test-fixture fix (`test(abb): use valid hex info hashes in scraper test fixtures`) since the existing fixtures used non-hex placeholders that the new validation correctly rejects. - **fix: Anna's Archive title parser** — handles nested edition spans and filters `lgli` catalog descriptor entries (e.g. "Book/Online Audio") that were polluting search results. ### Deliberately not included - LitFinder rebranding (UI strings, Apprise app ID, logo). The `fix: three upstream bugs` commit (#999/#956/#1025) was cherry-picked with Apprise app-id, description, and logo-URL strings reverted from "LitFinder" back to "Shelfmark"; noted in the commit body. - Features from the LitFinder fork (multi-variant title search, multi-book flat-folder grouping, fuzzy text matching, "Leave in Place" output handler, admin display name, custom-source plugin system). These are larger behavior changes that each warrant their own focused review — happy to send any of them separately if of interest. - LitFinder-specific test environment and CI infrastructure. ### Verification - Backend: **1879 passed**, 96 skipped (1 preexisting failure on `seleniumbase`-dependent test in local venv; runs fine in the standard Docker image with the `browser` extra) - Lint, format, dead-code: all clean against this repo's existing ruff/vulture config - One follow-up cleanup commit (`style: ruff lint and format fixes for ported commits`) brings the cherry-picked code into compliance with this repo's ruff settings — no behavior changes there ### Etiquette / credit Per-commit authorship preserved by cherry-pick. The only edits to the original commits are: - `fix: three upstream bugs` — Apprise rebrand strings reverted to "Shelfmark" (noted in commit body, original author retained as `Co-Authored-By` via cherry-pick) - One follow-up `style:` commit for ruff config alignment Big thanks to [@NemesisHubris](https://github.com/NemesisHubris) for the original work in LitFinder; this PR exists to make sure these fixes reach Shelfmark's wider user base. Happy to revise scope, split into smaller PRs, or split off the Py2 cleanup separately if that's preferable. --------- Co-authored-by: NemesisHubris <155838970+NemesisHubris@users.noreply.github.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz> |
||
|
|
0633cfde87 |
Add content_type=combined URL parameter and FORCE_COMBINED_SEARCH user setting (#1058)
Adds support for `content_type=combined` in URL search parameters, letting users force combined-mode searches via a bookmarkable link rather than relying on the last-used preference from localStorage. The override is applied only in Universal mode and only when combined mode is actually available (universal enabled, `show_combined_selector` on, neither content type blocked by policy). Otherwise, it's silently ignored, consistent with how `content_type=ebook`/`audiobook` already behave outside Universal. Existing `content_type=ebook`/`audiobook` URLs now also force combined mode off, so the URL is authoritative regardless of prior preference. Also adds a per-user `FORCE_COMBINED_SEARCH` setting that locks combined mode on whenever it's available. URL `content_type=ebook`/`audiobook` overrides are also superseded by force-combined for the same reason: the search bar wouldn't let users switch back, so honoring the URL param would leave them in a state they couldn't escape from. |
||
|
|
0e120abfaf | Validate request policy release source (#975) | ||
|
|
f6357ead41 | Contain remote path mappings (#974) |