The Google Books search builds `maxResults` as `min(limit, 40)` because
the API caps a page at 40 volumes, but advances `startIndex` by the full
`limit`. The pages then stop tiling. With `limit=50`, page 1 covers
items 0 to 39 and page 2 starts at 50, so items 40 to 49 are never
returned and every later page drops another 10.
This computes the page size once and uses it for both `maxResults` and
`startIndex`. The shipped frontend asks for 40 and is unaffected.
`/api/metadata/search` clamps `limit` to 100, so an API caller passing
41 to 100 was hitting it.
One thing I left alone. The provider uses the base `search_paginated`
heuristic, `has_more = len(books) >= options.limit`, which still reports
`has_more: false` for a limit above 40 since Google can never return
that many. That was already the behaviour before this change, and fixing
it means either touching the shared heuristic or adding a provider
override, so I kept this patch to the stride. Happy to follow up if you
want it.
## Verification
- `tests/metadata/test_googlebooks_parse.py`: pages 1 and 2 at
`limit=50` must tile exactly, plus a guard that `limit=25` still strides
by 25. The first fails on current main and passes here.
- Full suite (3165), ruff, ruff format, basedpyright, vulture green.
Hardcover replaced its ~500 char JWTs with short opaque personal access
tokens ("hc_pat_..."), and the connection test rejected anything under
100 chars before a request ever left Shelfmark, so every newly created
key failed with "API key seems too short".
The length floor now applies only to keys without the hc_pat_ prefix; a
prefixed key goes straight to Hardcover, which is the authority on
whether it is valid. Also strip a pasted "bearer " prefix regardless of
casing -- Hardcover's docs tell users to paste the token into an
"authorization" header, so the prefix rides along on the copy, and the
old case-sensitive removeprefix() sent it through as part of the token.
The API key field now names the expected shape.
Note that Hardcover's PAT path currently answers every hc_pat_ token
with a 500, a fabricated one included, while non-PAT tokens still get a
clean 401. So a new key cannot connect yet regardless of this change --
that failure is server-side and not something this code can reach.
Refs #1240
Advanced title search, advanced title+author search, and the title
typeahead returned zero results every time, and the sort fallback added
in #1183 blamed the sort value for it.
Hardcover turns the `fields` search parameter into Typesense's
`query_by`
but keeps `num_typos` and `query_by_weights` as fixed-length presets per
query_type. For query_type=Book the preset expects exactly five fields,
so a shorter list is not searched loosely - the whole search is rejected
with a null results body. Confirmed against the live API: 1, 2, 3, 4 and
6 fields are all rejected, only 5 works, and weights must match
one-for-one when sent. Every Book-type list we sent was the wrong length
- the title typeahead and advanced title search sent 2, title+author
sent 3.
- Send BOOK_SEARCH_FIELDS (the full five) for every narrowed Book search
and express the intent through weights instead. Weights only bias
ranking - a field weighted 0 still matches - so a title search now
ranks titles first rather than restricting to them. That is the
closest behaviour Hardcover still allows, and there is no client-side
filter to restore the old precision.
- Pin the field and weight counts in tests, since the failure mode is a
silent zero results rather than an error.
The sort fallback from #1183 also misread these rejections:
- Select the `error` field on every search and log Hardcover's own
explanation. The reason is only ever in that sibling field, so a
rejection surfaced as "returned no result body" with nothing to act
on. Reading it is what made the field-count rule findable.
- Drop `sort` entirely on the retry instead of sending an empty string.
An empty sort is a value like any other and can be rejected too.
- Arm the 900s sticky window only after the sortless retry succeeds. It
was armed before the retry and never rolled back, so one rejected
typeahead disabled sorting process-wide for 15 minutes whatever the
actual cause.
Verified against the live Hardcover API: advanced title search 0 -> 84
results, title+author 0 -> 139, title typeahead 0 -> 84 with the exact
title top. 2566 unit tests pass; ruff, basedpyright and vulture clean.
Refs #1183. The sort_by regression #1183 was written for is gone from
Hardcover's side - every sort value it rejected, including the one in
the report, is accepted again today. Two plain-search rejections in that
report (fields=None) remain unexplained: they could not be reproduced
under any per_page, page depth, sort value or query shape, and are most
likely transient upstream. They now self-report the reason if they
recur.
Hardcover forwards the `sort` argument to Typesense's `sort_by` and
rejects
the entire search if it dislikes the value -- an unknown field, a bare
field
name with no direction, or more than three sort keys. A rejected search
is
not a GraphQL error: it comes back as HTTP 200, no `errors` key, and a
null
`results` body.
_extract_typesense_hits() reads that null as `hits=[], found=0`, so a
failed
search was indistinguishable from one that matched nothing. Users saw
zero
results with a healthy container and no log line explaining why.
Add _execute_search_query(), used by the three sort-bearing call sites
(book
search, field typeahead, series resolution):
- Detect the rejection via the null `results` body. A search that
genuinely
matched nothing still returns a results object with `found: 0`, so empty
result sets are not mistaken for failures.
- Retry once with an empty sort, which Hardcover always accepts, so
searches
return results instead of nothing.
- Keep that fallback sticky for 15 minutes so every subsequent search
does
not pay for a request known to fail, and let it expire so sort order
comes
back on its own if the index is fixed upstream.
- Log rejections that no sort can explain, and retries that also fail,
at
ERROR instead of discarding them.
While the fallback is active, results fall back to Typesense's default
ordering regardless of the selected sort. Degraded ordering beats no
results,
and it is now logged rather than silent.
SORT_MAPPING itself is unchanged: all five of its values were verified
against
the live API and return results. The `sort: "relevance"` reported in
#1179 was
the raw SortOrder value sent by v1.3.5; the mapping already fixed that.
What
remained unfixed, and is fixed here, is that the failure was invisible.
Fixes#1179
First of all, I don't know if you even want to merge a scraper-based
metadata provider. I made this just for my use-case. If you'd rather
not, I completely understand it.
An alternative would be adopting [Audiobookshelf's Metadata Provider
API](https://audiobookshelf.org/docs/documentation/community/community-providers)
which I contributed to it for exactly the reason to not have scrapers.
## What
Adds [Moly.hu](https://moly.hu) — the Hungarian community book catalog —
as a metadata provider, following the existing provider plugin
architecture (`@register_provider` + settings tab with enable checkbox
and Test Connection button, disabled by default).
## Why
None of the current providers cover Hungarian editions well: Hardcover
and Open Library rarely index them, and Google Books coverage is spotty.
Moly.hu is the de-facto catalog for Hungarian books (local editions
*and* Hungarian translations of foreign works). With this provider,
Universal mode works end-to-end for Hungarian titles: moly search →
localized title/author feed the release search → indexers that carry
Hungarian content can actually match. Related pain points: #595 (books
missing from metadata providers), #1035 (interest in niche sources).
## How
- HTML scraping with BeautifulSoup (already a dependency), no API key
needed
- Scraping approach (search URL, page structure, language-tag mapping)
adapted from the long-lived Calibre `Moly_hu` plugin (GPL v3, credited
in the module docstring), with fallback selector chains inherited from
it
- Sliding-window rate limit (30 req/min) to stay polite to a small
community site
- Standard `@cacheable` decorators; fetch failures return `None` so they
are not cached (same behavior as the Google Books provider)
- Search results carry cover thumbnails, rating and series info as
display fields; `get_book` parses title (zero-width chars stripped,
nested series link excluded), authors, ISBN-13/10, publisher, publish
year, description (spoiler-warning prefix stripped), tags/genres, cover,
and language (from moly's language tags, defaulting to `hu`)
- ISBN search resolves through moly's site search
## Testing
- `tests/metadata/test_moly_parse.py`: offline tests with fixture HTML
mirroring live moly.hu markup — search parsing/dedup, pagination guard,
failure-not-cached behavior, book-page parsing, ISBN resolution, ISBN
validation helper
- `uv run pytest tests/metadata` green (45 passed), `ruff check` / `ruff
format` clean
- Verified live against moly.hu (search, get_book, ISBN lookup) and
running in Docker alongside Hardcover
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
- Adds `uv`, `ruff`, `pyright`, `vulture` and `pytest-xdist`
- Move project, lockfile, docker build etc to uv
- Align python tooling on 3.14
- Huge bulk of ruff linter fixes applied. Still in progress but all the
core types are now enforced
- Update CI and test helpers
Closes#515
## Summary
This adds a new `audible` metadata provider backed by the Audimeta API.
The provider supports:
- Audible/Audimeta metadata lookup without authentication
- region selection (`us`, `ca`, `uk`, `au`, `fr`, `de`, `jp`, `it`,
`in`, `es`, `br`)
- ASIN book lookup
- ISBN lookup with fallback search
- series suggestions and series-order browsing
- richer audiobook metadata such as narrators, runtime, rating,
subtitle, cover, publisher, and series info
- configurable Audimeta base URL, timeout, cache usage, default sort,
and unreleased filtering
## Notes
A few Audimeta-specific integration details were needed:
- send a meaningful `User-Agent`, otherwise Audimeta rejects requests
with `403`
- send the `cache` parameter in the format Audimeta expects
- use `keywords` for general search instead of `query`, which gave
poor/irrelevant results for title-style
searches
## Validation
Tested locally with:
- `python -m py_compile shelfmark/metadata_providers/audible.py`
- `python -m pytest tests/metadata/test_audible.py -v`
- `python -m pytest
tests/metadata/test_metadata_provider_capabilities.py -v`
Also verified manually in a Podman test container:
- searching for `Discount Dan` returns Audible title `B0DXLXRNGG`
- book details and series metadata load correctly
## Scope
This PR intentionally keeps the change localized to the provider layer
and docs:
- new Audible provider
- provider registration
- provider docs
- generated environment variable docs
- Added full Hardcover reading status types into the list selector
- Split reading status entries from dedicated lists
- Added option to disable the automatic removal of books when downloaded
from a Hardcover list
- Fixed browser download not firing when the completed state was
triggered in specific cases
- Adds full interaction with Hardcover lists, including adding and
removing from lists + want to read status
- List selection exposed in search results, details modal and release
modal
- Added automatic list dropdown when selecting "list" search
- Added auto-removal of books from a list when downloading from that
specific list page
- Changed search selector to hover-activated
- Restructured search field options into the left-hand selector.
Includes dynamic options for each provider.
- Moved Hardcover list and manual search mode into the left hand
selector
- Added search mode and metadata provider into the search options area
- Added new Hardcover series API query and live series suggestions
- Added live Hardcover author and title suggestions
- Added heuristic-based author and title query creation, stripping out
unnecessary elements that could limit searches
- Improved language support when using Hardcover. Searches will now be
conducted on a per-language basis using localized book titles.
- Added manual search override option in the release modal.
splitsec2
CaliBrain
FlyinPancakeandCopilot Autofix powered by AI
Alex
cadric
Alex