## Problem
Some trackers — MyAnonamouse in particular — distribute AAC audiobooks
as per-chapter `.mp4` files. That's the same ISO-BMFF container as
`.m4a`/`.m4b`, just with the generic extension (`ftyp isom`,
audio-only).
Today those releases:
1. show up in Prowlarr search results with **no format chip** — only the
generic "Audiobook" icon, because no format could be inferred;
2. download successfully; then
3. fail post-processing with **"No book files found in download"**,
because `.mp4` isn't in `AUDIOBOOK_FORMATS` (`shelfmark/core/utils.py`).
Real example: MAM #627978, *The Martian* (Andy Weir, 2020 edition) — 142
files `0001 … 0142 Andy Weir (2020) The Martian.mp4` + `cover.jpg`, 305
MB. Every file is a valid AAC-in-MP4 chapter.
Adding `mp4` to `SUPPORTED_AUDIOBOOK_FORMATS` in `settings.json` doesn't
help since the hard-coded tuple is what post-processing scans against.
## Change
- Add `"mp4"` to `AUDIOBOOK_FORMATS` (single source of truth — settings
UI, Prowlarr parsing, IRC parser, archive extraction and post-download
scan all derive from it), with a comment explaining why.
- Add `".mp4"` to the two hand-maintained debrid `_BOOK_EXTENSIONS`
lists (AllDebrid / Real-Debrid) so file selection matches.
- Slot `mp4` into the IRC `AUDIOBOOK_FORMAT_PRIORITY` table right after
`m4a` (same container family).
- Update the documented default in `docs/environment-variables.md`.
- New regression test
`test_audiobook_multifile_mp4_chapters_are_book_files` modelled on the
existing multi-file usenet test.
### Note for existing installs
The legacy-default migration only widens configs that still hold the old
`m4b,mp3` list, so users on the current widened default won't pick up
`mp4` automatically — they'll need to tick it in Settings → Audiobook
formats. New installs get it by default. Happy to extend the migration
if you'd rather it be automatic.
## Testing
- `ruff check` / `ruff format --check`: clean
- `pytest tests/core tests/config tests/irc tests/prowlarr
tests/download -m "not integration and not e2e"`: 2296 passed, new test
+ `test_audiobook_format_consistency.py` all green. The 10 failures in
`test_entrypoint_permissions.py` / `test_orchestrator_stall.py`
reproduce identically on untouched `main` on macOS and are unrelated.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Follow-up to #1237. \`rename_and_group\` only grouped when the source
root was a directory, so a multi-file audiobook delivered as a single
archive fell through to the flat path: a \`Book.zip\` of twelve chapters
landed loose in the destination root with its original chapter names —
the layout #1181 is about.
The \`is_dir()\` guard was there to keep \`Book.zip/\` from becoming the
folder name, but skipping the file case gives up the grouping instead of
naming it. A non-directory source can only produce several book files by
having been extracted (\`collect_staged_files\` returns a single-element
list for every other file shape), so the archive stem is the release
name and the suffix is packaging: group under \`Book/\`.
Also regenerates the env docs for the new option and gives it the same
\"do not use with ingest folders\" caveat Rename and Organize carries,
since both now create directories in the destination.
Tested: reverting only the source fix makes both new tests fail and the
\`rename\` control case pass, so grouping stays opt-in. Full non-e2e
suite green (2653 passed).
Pass the effective source root from `process_folder_output` into
`transfer_book_files`, and have the transfer layer select a sanitized
child directory named after that source root when an audiobook has
multiple files and its organization mode is `none` or `rename`. Create
that grouping directory before applying the existing hardlink/copy/move
logic so operation accounting, torrent seeding preservation, collision
handling, cleanup, and custom-script final paths continue to use the
established production path. Completed multi-file audiobook torrents
arrive as a directory whose chapter filenames may not identify the book,
but folder output currently sends every discovered chapter directly to
the configured destination in `none` and `rename` modes. This flattens
chapters from unrelated books together and causes directory-oriented
consumers such as Audiobookshelf to interpret individual chapters as
separate books.
A multi-file audiobook torrent in the default `rename` mode copies or
hardlinks all supported chapter files beneath `<destination>/<original
torrent directory>/` with their original chapter filenames, and places
no chapters directly in the destination root; A multi-file audiobook in
`none` mode receives the same source-folder grouping without renaming
its chapter files.
Fixes#1181
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: CaliBrain <calibrain@l4n.xyz>
## Summary
- sync an existing CWA-backed user's username when CWA renames it
- keep username collisions safe by assigning a stable `__cwa` alias
instead of overwriting a local account
- allow username updates through `UserDB` and cover
rename/collision/repeat-sync behavior
Fixes#1197.
## Testing
- `uv run ruff check shelfmark tests`
- `uv run ruff format --check shelfmark tests`
- `uv run vulture shelfmark`
- `uv run pytest tests/core/test_cwa_user_sync.py
tests/core/test_user_db.py tests/core/test_admin_users_api.py
tests/core/test_auth_api.py -k "cwa or update_user"` (36 passed)
- `uv run pytest tests/ -x --tb=short -m "not integration and not e2e"
--ignore=tests/config/test_entrypoint_permissions.py -q` (2445 passed, 5
skipped)
The entrypoint permission tests were excluded locally because macOS
ships Bash 3.2, which does not support the `${1,,}` expansion used by
`entrypoint.sh`; the same failure reproduces on an unchanged checkout.
`make python-typecheck` also currently reports the existing
`settings.py:147` callback return-type mismatch on the unchanged base.
Co-authored-by: CaliBrain <calibrain@l4n.xyz>
An IRC audiobook search returned nothing while OpenBooks, reading the
same @search answer from the same channel, listed results. Three
separate defects were discarding them.
The audiobook format list was maintained by hand in four places and had
drifted. The settings UI offered only m4b/mp3/m4a/zip/rar, and that list
is the only one a user's config can be built from, so flac, opus, ogg,
aac, wav and wma were unreachable everywhere — even though the IRC
parser recognized them, the IRC sorter ranked them (dead code that could
never fire), archive extraction knew them and Prowlarr searched for
them. A FLAC audiobook was invisible in search and, if it arrived
anyway, rejected after download as "format not supported".
AUDIOBOOK_FORMATS and ARCHIVE_FORMATS now live once in core.utils and
every layer derives from them, which also restored the missing .opus in
the post-download scan's trackable extensions.
Widening the default alone would not have reached anyone already
affected: initialize_default_configs() writes field defaults only when a
tab has no config file yet, so an existing install keeps its persisted
m4b/mp3 list forever. migrate_audiobook_formats rewrites a list that
still matches the old default exactly and leaves every other value
alone — re-enabling formats someone had deliberately turned off would be
worse than leaving them narrow.
The IRC parser filtered by file extension alone. Multi-file audiobooks
ship as a .rar or .zip of MP3s, which matched neither SUPPORTED_FORMATS
nor SUPPORTED_AUDIOBOOK_FORMATS, so they fell out of the ebook bucket
and the audiobook bucket both. Results are now classified before the
format filter is applied: an audio extension means audiobook, an ebook
extension means ebook, and for a container — where the extension says
nothing about the contents — the release name decides. An ebook archive
stays out of audiobook results.
RESULT_LINE_REGEX matched \w+ after any dot, so a line carrying no file
extension parsed as format "5mb" out of "::INFO:: 620.5MB", taking the
title and the size down with it and guaranteeing every downstream filter
dropped it. Any decimal size did this. The extension is now matched
against the known formats, so such a line falls through to the simple
pattern and comes back as "unknown", which the rest of the parser
already handles. ALL_RECOGNIZED_FORMATS became an ordered tuple in the
process: it was a set, so which extension won for a line naming two of
them depended on set iteration order and could vary between restarts.
Refs #1129
LOG_LEVEL never reached the app logger: env.py hardcoded the level to
DEBUG or INFO, so INFO lines kept appearing under LOG_LEVEL=error. Read
it from the env var and advanced settings, normalize unknown values to
INFO, and expose it as a setting. entrypoint.sh now normalizes
gunicorn's level too, so a typo falls back to info instead of stopping
the container from booting.
Z-Library gates the first hit on /md5/<hash> with a 503 whose only
payload is a Set-Cookie; echoing that cookie back returns the 302 to the
real page. html_get_page dropped it and re-ran the same rejected request
on every retry, ending in "No download URL resolved". Retry once with
the
cookies the 503 issued.
Fixes#1185Fixes#1187
Fixes#1175
Preserves the forwarded host and port when Shelfmark builds OIDC
callback URLs. The reverse-proxy examples now retain custom ports as
well.
Tests:
- `uv run pytest -n 0 -q tests/core/test_proxy_headers.py
tests/core/test_oidc_routes.py`
- `uv run ruff check shelfmark/main.py tests/core/test_proxy_headers.py`
- `uv run ruff format --check shelfmark/main.py
tests/core/test_proxy_headers.py`
Fixes#1138Fixes#1141
## Problem
Two language editions of one book resolve to the same canonical title,
so they render to the same path and the second gets a `_1` collision
suffix. Audiobookshelf treats a folder as exactly one library item, so
the pair becomes a single book with both files as tracks and a summed
runtime.
Shelfmark already parses and displays the language. It just never
reached the template engine.
## `{Language}` template variable
A template like `{Author}/{Title}{ (Language)}/{Author} - {Title}` now
yields:
```
/library/J K Rowling/Harry Potter (sv)/J K Rowling - Harry Potter.m4b
/library/J K Rowling/Harry Potter/J K Rowling - Harry Potter.m4b
```
The untagged edition's path is byte-identical to today, so no existing
layout shifts.
Three details worth flagging:
**The value is casefolded.** On a case-insensitive filesystem `(SV)` and
`(sv)` would collapse back into one folder, reintroducing the exact
collision being fixed.
**Values meaning "we don't know" render nothing** rather than producing
`Project Hail Mary (unknown)` folders. Anna's Archive reports that
string literally (`direct_download.py`, `language = detected or
"unknown"`).
**The frontend wasn't sending the release language at all**, so the
token would have stayed empty for exactly the audiobook sources in the
report. Prowlarr and AudiobookBay do not put language in `extra` the way
`direct_download` does, hence the payload plumbing. It reads
`release.language`, never `book.language` — the latter is the provider's
canonical edition and would mislabel a translation, with a regression
test for that specifically.
Not gated to audiobooks: Calibre-Web-Automated stages ingested files by
basename and discards folder structure, so the rename (filename)
template is the only lever those users have. Verified that form works:
`J K Rowling - Harry Potter (sv).epub`.
## Language consolidation (#1141)
Three release sources each carried their own alias map, all resolving to
the same ISO 639-1 codes, alongside a bundled database that only one of
them used. Adding a language meant editing three places.
Aliases now live in `data/book-languages.json` beside the code and name
they belong to, and `shelfmark/core/languages.py` resolves any of them —
two-letter code, ISO 639-2 three-letter in either the bibliographic or
terminological form, or English name. Prowlarr and AudiobookBay drop
their tables. Direct Download keeps its own path-parsing heuristics,
including the ambiguous short codes that collide with English words
(`de`, `en`, `no`, `in`), and takes only the alias data.
This also closes a coverage gap. MyAnonamouse offers 62 languages;
Prowlarr mapped 37, and an unmapped code is *dropped* rather than passed
through, so the other 25 carried no language at all — leaving
`{Language}` empty and the collision unfixed for Latin, Farsi, Tamil,
Urdu and the rest. Seven languages MAM offers had no database entry at
all: Bosnian, Burmese, Estonian, Icelandic, Manx, Scottish Gaelic,
Sanskrit.
Also fixes the Traditional Chinese code, which used a U+2011
non-breaking hyphen. Nothing compares against the ASCII spelling today
so it was latent, but it would silently defeat the first thing that did.
## Validation
Verified end to end against a live Prowlarr and MyAnonamouse, not just
unit tests. A real search returning both an English and a Swedish
edition, through the actual `queue_release` → `DownloadTask` → naming
path:
```
STEP 1 real MAM search -> 37 releases, languages: ['en', 'sv']
STEP 3 queue_release -> task.language='sv'
STEP 4 build_metadata_dict -> metadata['Language']='sv'
STEP 5 build_library_path -> /library/J K Rowling/Harry Potter (sv)/...
two language editions resolve to DIFFERENT folders: True
```
The refactor is pinned by a snapshot of both per-source maps taken
*before* they were deleted. All 131 aliases are asserted to still
resolve to the same code, one parametrised test each, so a regression
names the specific alias.
Also verified: the filename-only template, the retry round-trip
(`serialize_task_for_retry` → `_restore_task_from_retry_payload`, plus a
legacy payload with no `language` key), and placeholder handling.
Added a `KNOWN_TOKENS` ordering invariant test — `find_placeholder()`
does a substring `.find()` in list order and nothing protected that
contract, so a future token in the wrong position could silently shadow
an existing one. And a lockstep guard on the frontend, since
`KNOWN_TOKENS` is hand-duplicated in TypeScript.
**One caveat worth stating.** Three MAM codes are confirmed by
observation (`ENG`→`en`, `SWE`→`sv`, `MAL`→`ml`, the last from a real
`[MAL / EPUB]` Tagore release). The remaining ~59 are derived from ISO
639-2 rather than observed, because MAM's catalogue is overwhelmingly
English — enabling 27 extra languages still yielded only one non-English
hit across 258 results. Mitigated rather than closed: both 639-2
variants are present for every language where they differ, and a wrong
alias is an unused entry while a missing one loses the language. Happy
to correct any code a maintainer knows differs.
## Test results
2056 Python tests pass (up from 1906). Frontend typecheck, lint, format
and 126 unit tests pass.
Pre-existing failures on my machine, unchanged by this branch and
unrelated: `tests/bypass/` needs `seleniumbase`, and
`tests/config/test_entrypoint_permissions.py` uses bash-4 syntax that
macOS bash 3.2 rejects.
---------
Co-authored-by: delize <4028612+delize@users.noreply.github.com>
Co-authored-by: CaliBrain <calibrain@l4n.xyz>
## Summary
Audiobooks whose files are split across nested subfolders now keep their
on-disk folder order when "Rename and Organize" assigns part numbers.
Previously `natural_sort_key` keyed on `Path(path).name`, so identically
named files from different folders (`001.mp3` in `00_Introduction/` vs
`06_Side 6/`) collapsed together and the part sequence scrambled.
## Background
zazizou reported in #1007 that the total file count was right but
`06_Side 6/001.mp3` came out as part 1 instead of
`00_Introduction/001_About.mp3`. `scan.py` flattens every nested file
into one list and `assign_part_numbers` sorts it with
`natural_sort_key`, which discarded the parent-folder prefix. The fix
pads the existing `PAD_NUMBERS_PATTERN` numbers over the full relative
path (`str(path).lower()`) rather than the basename, so nested
audiobooks sort by folder first and then by filename. Flat directories
are unaffected because their files share a prefix, and
`natural_sort_key` is only called from `assign_part_numbers`, so nothing
else changes. I updated the test that pinned the old basename-only
ordering and added nested-subfolder and duplicate-basename cases;
`pytest tests/core/test_naming.py
tests/core/test_part_number_extraction.py` passes (102 tests).
Closes#1007
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
## Backport bug fixes from `NemesisHubris/litfinder`
Forwards a curated set of bug fixes from
[NemesisHubris/litfinder](https://github.com/NemesisHubris/litfinder) —
a community fork of this project — that address open issues here. All
commits preserve original authorship via `git cherry-pick`; this PR is a
backport rather than original work. Each fix has been reviewed locally,
lint/format-cleaned to match this repo's existing ruff config, and
verified with the test suite. Rebrand strings, license switches, and
features have been deliberately excluded.
### Upstream issues addressed
- **#999** — Mirror URLs with query params no longer break search
requests (strip query string/fragment in `normalize_http_url`)
- **#956** — Apprise notifications now respect the configured proxy
(proxy env vars injected before dispatch)
- **#1025** — rTorrent: separate `RTORRENT_AUDIOBOOK_LABEL` setting,
falls back to book label if unset
- **#1010** — Stop button in Activity no longer makes the panel
disappear (snapshot refresh on cancel)
- **#1021** — Anna's Archive slow-download countdown now caps retries
instead of looping forever
- **#1040** — Empty destination directory cleaned up when write probe
fails
- **PR #1031** — Language detection from Anna's Archive distant path
when listing metadata is missing
### Additional fixes (no open issue but clear bugs)
- **fix: Python 2 `except` syntax across 27 files** — `except X, Y:` is
a SyntaxError in Python 3 and prevents affected modules from importing
at runtime. Mechanical sweep to `except (X, Y):`.
- **fix(abb): info hash validation with magnet fallback** — adds
SHA-1/SHA-256 hex validation on extracted info hashes; falls back to
scanning the full page for a magnet link (e.g. posted in comments) when
the table value is malformed. Also extends the exact-phrase fallback to
manual queries and defaults the ABB listing language to `en` when
missing, preventing valid results from being hidden by the language
filter. Includes a small test-fixture fix (`test(abb): use valid hex
info hashes in scraper test fixtures`) since the existing fixtures used
non-hex placeholders that the new validation correctly rejects.
- **fix: Anna's Archive title parser** — handles nested edition spans
and filters `lgli` catalog descriptor entries (e.g. "Book/Online Audio")
that were polluting search results.
### Deliberately not included
- LitFinder rebranding (UI strings, Apprise app ID, logo). The `fix:
three upstream bugs` commit (#999/#956/#1025) was cherry-picked with
Apprise app-id, description, and logo-URL strings reverted from
"LitFinder" back to "Shelfmark"; noted in the commit body.
- Features from the LitFinder fork (multi-variant title search,
multi-book flat-folder grouping, fuzzy text matching, "Leave in Place"
output handler, admin display name, custom-source plugin system). These
are larger behavior changes that each warrant their own focused review —
happy to send any of them separately if of interest.
- LitFinder-specific test environment and CI infrastructure.
### Verification
- Backend: **1879 passed**, 96 skipped (1 preexisting failure on
`seleniumbase`-dependent test in local venv; runs fine in the standard
Docker image with the `browser` extra)
- Lint, format, dead-code: all clean against this repo's existing
ruff/vulture config
- One follow-up cleanup commit (`style: ruff lint and format fixes for
ported commits`) brings the cherry-picked code into compliance with this
repo's ruff settings — no behavior changes there
### Etiquette / credit
Per-commit authorship preserved by cherry-pick. The only edits to the
original commits are:
- `fix: three upstream bugs` — Apprise rebrand strings reverted to
"Shelfmark" (noted in commit body, original author retained as
`Co-Authored-By` via cherry-pick)
- One follow-up `style:` commit for ruff config alignment
Big thanks to [@NemesisHubris](https://github.com/NemesisHubris) for the
original work in LitFinder; this PR exists to make sure these fixes
reach Shelfmark's wider user base. Happy to revise scope, split into
smaller PRs, or split off the Py2 cleanup separately if that's
preferable.
---------
Co-authored-by: NemesisHubris <155838970+NemesisHubris@users.noreply.github.com>
Co-authored-by: CaliBrain <calibrain@l4n.xyz>
Adds support for `content_type=combined` in URL search parameters,
letting users force combined-mode searches via a bookmarkable link
rather than relying on the last-used preference from localStorage.
The override is applied only in Universal mode and only when combined
mode is actually available (universal enabled, `show_combined_selector`
on, neither content type blocked by policy). Otherwise, it's silently
ignored, consistent with how `content_type=ebook`/`audiobook` already
behave outside Universal.
Existing `content_type=ebook`/`audiobook` URLs now also force combined
mode off, so the URL is authoritative regardless of prior preference.
Also adds a per-user `FORCE_COMBINED_SEARCH` setting that locks combined
mode on whenever it's available.
URL `content_type=ebook`/`audiobook` overrides are also superseded by
force-combined for the same reason: the search bar wouldn't let users
switch back, so honoring the URL param would leave them in a state they
couldn't escape from.
- Updated mirror selection
- Removed built-in mirror options, users must provide their own
configurations
- Set Universal search to default, added ability to disable direct
source
- Updated documentation
- Updated makefile
- Adds `uv`, `ruff`, `pyright`, `vulture` and `pytest-xdist`
- Move project, lockfile, docker build etc to uv
- Align python tooling on 3.14
- Huge bulk of ruff linter fixes applied. Still in progress but all the
core types are now enforced
- Update CI and test helpers
- Clean up a few uses of config options that may miss the env variable
if this is set
- Add enhanced retry availability utilising the DB to persist download
errors / retries across restarts, request failures, and pass Prowlarr
detail through the download task to maintain retry data.
- Strip back entrypoint permissions for less intensive chown operations.
Fixes#796
- Added a path for rootless permissions in the entrypoint script
- Routed prowlarr searches through torznab for seedtime info
- Added additional request flow for download permissions
- Adds a combined search option in the search bar selector
- Choose both a book and audiobook file in a two-step release modal, and
download both simultaneously from a single search result.
- Works for requests. Request both a book+audiobook at once, or works
seamlessly with request policies that differ between book + audiobook
(E.g. automatically download the ebook portion, while the audiobook gets
sent as a request)
- Hidden for users who have book or audiobooks blocked.
Closes#611
- Updated frontend CSS to Tailwind v4
- Reverted socket IO origin restriction
- Fixed search queries not persisting after auth redirect
- Move advanced search options to left UI selector
- Unlock IRC source to be used for audiobook content_type
- Tweaked security settings env var syncing to be prioritised
- Fix AA "all languages" query generation
- Added language-free AA query as second fallback in case of no results
- Testing moving SeleniumBase scratch files to /tmp via symlink
- Added enhanced logging for activity dismissals and other events
- Removed iFrame restrictions
- Added canonical per-user visibility of requests and downloads via new
activity view table. Users get fully independent activity and history
views, while admins still see all.
- Replaces janky frontend + backend combination
Two-phase download history: downloads are now recorded in the DB at
queue time (not just at terminal time), eliminating the need to
reconstruct metadata in the terminal hook and removing the
`_is_graduated_request_download()` request-scan mess
- Much simpler handling of downloads in the activity sidebar, and
improved storage, persistence and UI behavior.
- Replace `ActivityService` with direct storage on
`DownloadHistoryService` and `download_requests` and removes the
activity_log/activity_dismissals tables
- Simplify no-auth mode by removing the fake user row pattern, handled
internally
- Add local download fallback so history entries can still serve files
after tasks leave the queue
- Downloads, requests and history are now entirely persistent between
updates / restarts, and correctly tied to each user.
- Refactored user and request code to avoid any database conflicts
- Fix threading behavior with custom script execution
- Harden the no_auth activity user filtering
- Add a hint to add local admin if none is created
- Added secret key to persist login states across updates / restarts
- Added the manual retry option for failed downloads
- Added the ability to retry failed post-processing using existing
downloaded file
- Added admin-visible "Download as" selector, admin chooses a user to
download on-behalf of - inherits their output preferences.
- Added search mode and default metadata provider / release source
options to User Preferences and My Account settings.
- Added sort by format option in release results
- Added {OriginalName} renaming field option, to retain the exact
downloaded filename
- Frontend dependency updates - fixes rollup vulnerability from this
week
Closes#662#656#649#562
jakesterpdxandClaude Fable 5
CaliBrain
FlozeandCaliBrain
Guflly
Alex