Commit Graph
31 Commits
Author SHA1 Message Date
splitsec2 2668705bfa perf: reuse connections and cache Anna's Archive download counts (#1546)
Two small changes that stop Shelfmark from opening a new HTTPS
connection for requests that go to the same host over and over.

Anna's Archive download counts. Every search result gets its count from
`/dyn/md5/inline_info/<md5>`, and each of those was a bare
`requests.get`, so a search with 50 results opened 50 TCP and TLS
connections to the mirror. `search_books` also runs once per title
variant, and the variants of one book mostly find the same files, so the
same md5 was looked up again for every variant that found it, and again
when the release list was reopened. The lookups now share one
`requests.Session`, and a successful count is kept for five minutes in a
bounded `CacheService` (the same class the metadata cache uses, as its
own instance, so it does not depend on the metadata cache toggle). A
lookup that fails or comes back without a count is not cached, so it is
retried on the next search just like today. The URL, headers and 5
second timeout are unchanged.

Debrid clients. Real-Debrid, AllDebrid, Debrid-Link and TorBox also used
the bare `requests` helpers. The download handler polls `get_status`
every two seconds for each active job, so every poll was a fresh TLS
handshake with the debrid API. Each of those modules now keeps one
`requests.Session` and sends its API calls through it, the way the
qBittorrent, Deluge, Prowlarr and Newznab clients already do. Headers,
timeouts, `verify` and error handling are still passed per call exactly
as before. The existing tests that patched
`<module>.requests.get/post/request` now patch
`<module>._session.<verb>` instead; nothing else in them changed.

How I checked it: each change has a test that fails on main and passes
here. They count the requests that reach `requests.Session.request` and
the sessions built while doing it, with no network. On main, three polls
of each debrid client build three sessions, and two overlapping batches
of AA results look `bbb` up twice. There are also tests for the count
cache expiring (with a faked clock) and for a failed lookup not being
cached. Full suite: 4432 passed, 48 skipped. ruff and basedpyright are
clean on the changed files.

I also looked at caching `/api/releases` on the server and left it out.
The release modal already caches each book's results per source and
content type in the browser for five minutes (longer for IRC and slskd
through `cache_ttl_seconds`), and that cache survives closing and
reopening the modal, so reopening it does not run a new search today. A
server cache would only help after a page reload, and it would quietly
undo the modal's "apply filters" and manual search refetches, which drop
the browser cache on purpose, unless the frontend also learned to send a
bypass. That did not seem worth it for this PR.
2026-10-22 20:24:40 -04:00
splitsec2 395c331b4a fix(search): give each provider its own slice of the search budget (#1537)
Fixes #1531.

All Direct Download providers share the one search budget
(`RELEASE_SEARCH_TIMEOUT`, 300s). Anna's Archive runs first, and when
its bypass keeps failing the retries use up the whole budget. OceanofPDF
then sees an expired deadline, skips its queries, and the search raises
Anna's Archive's error even though OceanofPDF never got a turn.

I didn't split the budget evenly, since the deadline module itself says
a cold-start Anna's Archive solve can legitimately take 60 to 120s and a
150s or 37s slice would break setups that work today. Instead
`search_deadline.unit_deadline(waiting=n)` opens a child deadline for
one unit of work. It gets the parent's remaining time minus 30s for each
unit still waiting, never less than 30s and never more than the parent
has left. A single provider still gets everything, a fast provider's
unused time rolls on to the next, and a stuck one stops at its own limit
with the reserve intact.

It's used in the provider loop in `DirectDownloadSource.search`, and in
the source loop in `/api/releases` since it's the same shape and four
lines. With several sources enabled the first one now gets 30s less per
source behind it, which is still well above a cold solve for the usual
two or three.

Tests: a stuck first provider no longer starves the second (fails on
main, passes here), a lone provider gets the full budget, and a child
never outlives its parent. Two existing tests in
`test_releases_api_budget.py` tripped the deadline in force from inside
the source, which is now the child, so they use a short request budget
instead. ruff, basedpyright on the changed files and the non-e2e suite
are clean.

OceanofPDF is inactive on my own instance (no mirror URL set), so I
haven't reproduced this live. It's verified by reading the code and by
the tests. I had Claude do the pass on this one.
2026-10-22 20:17:44 -04:00
splitsec2andCaliBrain f837506017 Merge commit from fork
A signed-in user picks the release fields of POST /api/releases/download, and a record
id with the "oceanofpdf:" prefix reached the provider without handles() or is_enabled().
The provider then fetched source_url, POSTed to the form action found on that page and
fetched the meta-refresh target, with no check on any of them.

The provider now refuses to run when it is switched off, and the book page, the form
action and every POST redirect hop must be on the configured mirror (same scheme, host
and port). The signed file link may sit on another host but must be http(s).

network.ensure_public_destination refuses a host that resolves to a loopback, private,
link-local or otherwise non-global address (IPv4-mapped IPv6 judged by its IPv4 address).
Hosts the operator configured as mirrors and names that do not resolve pass as before.
html_get_page and download_url call it on their first URL and check each redirect hop
through a response hook. The image cache uses the shared address check too.

Co-authored-by: CaliBrain <calibrain@l4n.xyz>
2026-10-21 02:06:31 -04:00
splitsec2 6f65d6e20a Merge commit from fork
A release format such as /../../x was appended to the staging file name
unchanged, so the direct-download and libgen handlers could write to, and on
cancel unlink, a path outside TMP_DIR. Normalise the format to a plain
extension at queue time and in build_filename, reject path-like source ids for
the sources that use them as file names, and make both handlers resolve the
staging path and refuse anything that is not a direct child of TMP_DIR.
2026-10-20 23:40:16 -04:00
CaliBrain 10710abd4e fix(download): send every file of a multi-file download to the browser (#1489)
Download to Browser only ever sent the first file of a multi-file
release
(audiobook chapters, packs): the folder output returned final_paths[0]
and
/api/localdownload served that one path.

The folder output now keeps every final path on the in-memory task
(download_paths), and /api/localdownload streams them as one
uncompressed
ZIP named after the book when there is more than one. The archive is
never
staged on disk or held in memory, and ZIP64 covers large audiobooks.
History still stores only the first path, so a re-download from history
after a dismiss or restart serves a single file, as before.

Also brings every linter back to green on all files:
- pre-commit: bump the ruff hook to v0.17.1 to match uv.lock;
ruff-format
  the Python examples in docs/dev and skills
- ruff: fix the remaining issues in skills/shelfmark/download_books.py;
  format base_handler.py
- basedpyright: Optional and attrs errors in databazeknih.py
- knip: drop 31 unused exports, reuse ColumnColorHint in colorMaps
- oxfmt 0.72: styles.css
- direct download: fix the "Welib, mirrors" typo and the test that still
  expected the pre-OceanofPDF message

Fixes #1467
2026-10-20 02:33:53 -04:00
c8d66db22e feat(direct_download): add OceanofPDF as a standalone search and download provider (#1485)
Closes #1149

## What this adds

OceanofPDF as a second Direct Download provider, alongside Anna's
Archive. Unlike Z-Library, LibGen and Welib, which only download files
Anna's Archive found, OceanofPDF searches and downloads on its own. It
works with or without an AA mirror configured; with both, one search
queries both and the results are merged.

It plugs into the provider registry from #1337, so search, record
routing and the download handler need no changes.

**Search:** WordPress `/?s=` search, "title author" first, then the bare
title if that finds nothing. One release per advertised format (`[PDF]
[EPUB]`), with title, author and cover. Shelfmark's supported-format
filter applies as usual.

**Download** follows the site's three hops:

1. The book page has one POST form per format, targeting
`Fetching_Resource.php`.
2. That returns an interstitial whose `<meta http-equiv="refresh">`
points at a signed, expiring file URL.
3. The file URL serves the book.

Pages go through `html_get_page` and its Cloudflare bypass. The form
POST replays the clearance the bypasser stored, and re-solves once if
challenged. Bypassers only replay GETs, so it can't be handed over
directly.

**Settings:**
- `OCEANOFPDF_MIRROR_URLS` under Mirrors. Only the first URL is used.
- An OceanofPDF toggle under Download Sources → Slow downloads. It's
locked until a mirror is set and while the Cloudflare bypass is off. Its
position in the list doesn't matter, and its description says so.
- A saved list from before this change has no OceanofPDF entry. The
settings page shows a missing option as off, so the provider treats it
as off too, keeping the page and the behaviour in agreement. New
installs default to on.

## Fixes found along the way

- **`challenge_marker` rejected real pages behind Cloudflare** (separate
commit, `fix(bypass)`). Cloudflare's bot-detection script injects
`/cdn-cgi/challenge-platform/scripts/jsd/main.js` into every page it
serves, after load. Bypassers return the rendered page, so a successful
FlareSolverr solve of any page under 64 KB came back carrying
`/cdn-cgi/challenge-platform` and was rejected as unsolved. The detector
now ignores that one script path. Real challenge pages load from
`/cdn-cgi/challenge-platform/h/…` and keep their other markers, so they
are still caught. This also affects any other Cloudflare-fronted host,
not just OceanofPDF. Happy to split it into its own PR if you prefer.
- **Brotli on a plain `requests` call.** Copying `DOWNLOAD_HEADERS` into
the form POST offered `br`. Cloudflare then answered in Brotli, which
`requests` can't decode without the optional `brotli` package. The POST
now leaves `Accept-Encoding` to `requests`. If a response still has no
file link, its status, encoding, title and challenge marker are logged.

## Testing

- **New tests:** 41 in `tests/direct_download/test_oceanofpdf.py` and 4
in `tests/bypass/test_cloudflare_beacon.py`. They cover search parsing,
format and language filtering, form selection, the interstitial, the
download flow (including a challenged POST, an HTML stub served as the
file, and cancellation), the toggle, and registry wiring. Fixtures
mirror the live site's markup.
- **Existing tests:** `tests/direct_download`, `tests/bypass`,
`tests/config`, `tests/core` and `tests/download` show no new failures
against `main`. Some tests in those folders can't run on Windows; those
fail identically on both.
- **Lint:** `ruff check` and `ruff format --check` are clean on every
changed file.
- **Live:** verified end to end on a v1.4.0 Docker deployment behind
FlareSolverr (external bypasser), with only an OceanofPDF mirror
configured: search, then the book page, the form POST and the completed
download.

## Known limits

- OceanofPDF doesn't label languages, so its results always pass a
language filter.
- PDFs only appear when `pdf` is in the supported formats.
- Each query reads the first results page only (roughly 7–10 books).
- Downloaded files carry OceanofPDF's own tags/watermark, as noted in
#1149. They aren't stripped.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Adam Stone <196691089+ast0ne1@users.noreply.github.com>
Co-authored-by: CaliBrain <calibrain@l4n.xyz>
2026-10-20 02:07:23 -04:00
CaliBrain 3c5cfcb77d fix/redact log secrets (#1488)
- fix(logging): mask tracker API keys in log output (#1486)
- fix(welib): follow slow_download links through the countdown (#1447)
2026-10-20 01:55:49 -04:00
Matt Van HornandMatt Van Horn d3bcf0674d fix(direct_download): Anna's Archive search 404 on ISBN queries (#1456)
When ISBN filters are present, the search query is now the deduplicated
bare ISBN strings, with the free-text query appended only if it is not
already one of those tokens. Searches with no ISBN still send the quoted
free-text query alone. An Anna's Archive lookup that includes an ISBN
comes back as 404 Not Found, so the direct-download release search fails
and the book cannot be retrieved. The failing request puts an
isbn13:/isbn10: boolean expression in the q parameter.

The direct-download search query tests passed.

Closes #1444

Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-10-20 01:10:03 -04:00
Som Samantray 2b8e0da2f3 fix(direct-download): clarify required search mirror (#1454)
Direct Download now tells users who configure only a Z-Library mirror
that they need an Anna's Archive mirror for search. Z-Library, LibGen,
and Welib mirrors download results found through Anna's Archive.

Fixes #1408. Validation: the Direct mode browser check, 18 focused
tests, Ruff, formatting, and backend/test type checks passed. The
CI-style Python suite passed 3,569 tests;
`tests/core/test_hardlink.py::TestSameFilesystem::test_string_paths` was
excluded because this container reports different `st_dev` values for a
file and its parent directory. AI disclosure: OpenAI Codex assisted with
diagnosis, implementation, and validation.
2026-10-20 01:09:09 -04:00
CaliBrain e1c3f057ab fix: bypass recordings, welib wrong-md5 links, footer build sha (#1364) (#1373)
Debug screen recordings never started. Every bypass logged "Capturearea
1540x1050 at position 0.0 outside the screen size 1440x1880".We ask
ffmpeg for the fingerprint screen size plus margin, the size wealso pass
SeleniumBase as xvfb_metrics. SeleniumBase builds thatdisplay with
use_xauth=True, the image ships no xauth binary, so itfalls back to a
fixed 1440x1880 Xvfb and the requested size neverexists. Drop
-video_size so x11grab records the whole screen, whateversize it turned
out to be.

welib could hand back a link for a different book. Welib
answers/md5/<md5> with a search for that md5; when it does not have the
file,the resolver took the first "Download" on the results page
(md5a2c1dc0c... resolved to auto_download/9c8cf85d...). On an
/md5/<md5>page a GET/Download link is now only taken when its href names
thatmd5; otherwise the source is reported as not having the file.
Alsoremoves _get_download_urls_from_welib and _is_source_enabled:
themd5-template branch in _get_urls_for_source always handles welib
first,so that resolver could never run.

The footer showed the build date instead of the commit. CI
stampsBUILD_VERSION as <yyyy-mm-dd>-<sha> (pr-<sha> for PR images) and
thefooter kept its first seven characters, so dev images read
"Shelfmarkmain (2026-09)". Take the trailing commit sha instead: "main
(1a5b37d)".The full BUILD_VERSION stays in the hover title
2026-09-21 00:15:41 -04:00
Paul Rimmer d4619be69a Feature: Show the AA search result stats (#1362)
When doing a direct search for a book, show the stats of the AA results.
For example if we search for "The Great Gatsby", AA reports it has 240
hits and shows the first page of 50. Provide this stats info in the
shelfmark webUI via ResultsSection.tsx and ReleaseModal.tsx. This table
shows what should be displayed based on the total number of hits found:

|Total	|Display|
|-------|-------|
|1	|"Result 1 (1 Total)"|
|6	|"Results 1-6 (6 Total)"|
|144	|"Results 1-50 (144 Total)"|
|500+	|"Results 1-50 (500+ Total)"|

Currently, shelfmark also only shows us the first 50 search hits even if
there were more available from AA. This could be added later if
considered desirable.

As usual, a picture is worth a 1000 words:

<img width="1012" height="610" alt="direct-results-stat"
src="https://github.com/user-attachments/assets/8c0c688f-444e-482a-a9a3-2dee5a5563b8"
/>

<img width="1013" height="741" alt="universal-results-info"
src="https://github.com/user-attachments/assets/399bbb79-87f2-4432-a3d0-64937795f5f1"
/>

Coded with llama.cpp, opencode and 🤖
2026-09-20 13:03:32 -04:00
CaliBrain e5dd34ae0e fix: unbreak main and follow up on the Blackhole handoff review (#1346)
DownloadHistoryService.record_download and updated the single production
caller, but not the eleven in the test suite, leaving main red with 32
failures. Pass None, which is what the pre-#1336 behaviour recorded.

For the Blackhole handoff (#1345): add_download publishes the torrent
before the cancel check runs, and BlackholeClient.remove() is a no-op,
so
the watcher picks the file up regardless. Reporting a bare "Cancelled"
hid
that from the user. Name the completed handoff in the cancellation
message
instead, drop the _handle_cancelled_download call whose usenet branch
cannot apply to a handoff-only client, and record why the orchestrator
no
longer verifies HandoffResult.path.

Finally, make tests/direct_download a package: test_libgen_extract.py
imports tests.libgen.sample_html across test directories, so without an
__init__.py pytest named its modules by bare basename and a same-named
module elsewhere would collide.
2026-09-17 16:27:53 -04:00
Alex Guerrieri c6b70a6844 fix(sources): send a Referer when fetching libgen ads.php pages (#1340)
## What

libgen.li's `ads.php?md5=` now returns an **empty `200`** to any request
without a `Referer` — an anti-hotlinking check the mirrors added
recently. Both libgen paths fetch it without one, so the page comes back
blank and the download silently fails while **search keeps working**
(which is exactly why it looks like rate-limiting or mirror drift rather
than a bug).

Same one-line cause, two call sites: the Libgen search source
(`libgen/scraper.py:fetch_page`) and the AA-md5 → libgen fallback
(`direct_download/annas_archive.py:_extract_libgen_download_url`). Fix:
send a same-origin `Referer: <scheme>://<host>/` on the `ads.php` fetch
in both.

## Worth a look in review

- **The referer goes on the *resolution* fetch, not the download.**
`download_url(..., referer=...)` was already correct — the blank page
happens one step earlier, at the `ads.php` GET.
- Reproduced against live mirrors: `ads.php` returns `Content-Length: 0`
bare, the full page with a `Referer`, and resolvable files download
valid bytes again.

Regression tests in `tests/libgen/` and `tests/direct_download/` assert
the header on both paths. Lint/format/typecheck clean.

Follow-up to #1326.
2026-09-17 15:47:45 -04:00
CaliBrain 35b89b0d78 fix(sources): restore Direct Download search errors and language matches (#1339)
Fixes two regressions from the provider-driven refactor (#1337). First,
the composite search caught RuntimeError, TypeError, ValueError and
request errors from each provider and returned an empty list, so a
failed search looked like one with no hits. It now raises the first
provider failure when no provider returned releases. Second, the shared
parser re-matched every row's language locally, dropping rows Anna's
Archive had already matched with &lang= (free-text cells like 'English,
French' or 'unknown'). parse_search_items gains a filter_languages
option, which AA turns off, so AA's own language-from-path filter is
again the only local one.
2026-09-14 01:30:05 -04:00
Tom a5cd9f0bfb refactor: make direct download provider-driven (#1337)
This is the refactor for the download handler
2026-09-14 01:16:53 -04:00
CaliBrain 9452ebc70d fix(bypass): stop handing solvers DDoS-Guard's ?check=1 probe URL (#1300)
html_get_page follows Anna's Archive redirects by hand, and DDoS-Guard's
gate
answers /search with a 302 to the same path plus `check=1`. The follower
walks
that handshake by reassigning `current_url`, so every downstream handoff
- the
403 branch, the 503-challenge branch, both redirect-loop rescues -
passed the
*probe* URL to the bypasser rather than the page we actually wanted.

A solver opens that in a fresh browser holding none of the cookies the
probe
exists to collect, so DDoS-Guard cannot verify it automatically and
serves the
manual CAPTCHA page that nothing can solve. The #1292 log is exactly
that: a 403
handed off on `&check=1`, FlareSolverr answering "Challenge solved!",
and a
4721-byte DDOS-GUARD captcha page coming back.

- `_solvable_url()` strips the probe parameter, applied at the single
choke point
in `_run_bypasser` so all four handoffs are covered. Scoped to the hosts
whose
redirects we follow manually; a URL without the parameter is returned by
  identity, so nothing else is re-encoded.

The same reports showed three further defects, all of which stand
whatever the
host was reacting to:

- The external bypasser logged that the solve had not cleared the
protection and
then returned the challenge page as a success. That skipped the one
recovery
left - get_bypassed_page's retry-and-rotate loop, where the next mirror
is a
different DDoS-Guard host - and filed the captcha page's own __ddg
cookies as
that host's clearance, to be replayed on every later request. It now
raises
  ChallengeNotSolvedError before storing anything.

- "Check that the bypasser is reachable and working" was the one piece
of advice
guaranteed to waste the reporter's time: it was reachable, it ran a full
solve,
and it returned a captcha. ChallengeNotSolvedError carries the marker so
the
  search layer can name the host as the cause instead of the bypasser.

- The untabled-page fingerprint logged `attempt_url`, which
html_get_page has
since rotated past. The #1298 bundle reported the page against
annas-archive.gl
when the body had come from .pk - the triage cost #1289 added the line
to
remove. The search now asks for the response URL and logs that. Its
give-up
shape is the tuple ("", url), which is truthy, so the exhaustion check
reads
  the body rather than the response.

Regression fixtures are built from the pages in the reports. The two
behavioural
handoff tests were checked against the unfixed code: both fail there,
reproducing
the reporter's log line verbatim.

Refs #1292
Refs #1298
2026-09-02 16:12:52 -04:00
CaliBrain 3d7ea40088 fix(search): reach the server's deadline, query one author (#1285, #1252) (#1287)
Two independent reasons a working search reported failure to the user.

1. The client gave up before the server did (#1285)

`/api/releases` bounds one release search with RELEASE_SEARCH_TIMEOUT
(default
300s) and answers a spent budget with a sentence naming the real cause -
the
machinery added for #1276. The frontend then aborted the direct_download
search
at a hard-coded 180s, so it always won the race: the user saw "Request
timed
out. Check your network connection or proxy configuration." instead, and
raising RELEASE_SEARCH_TIMEOUT changed nothing they could observe, the
180s
being baked into the hashed bundle inside the image.

- /api/config reports the effective (clamped) budget, and the client
derives its
  abort from it plus a margin, so the server always answers first.
- Direct-mode search shows what the server actually said. Every non-auth
failure
was relabelled "Unable to reach download source. Network may be
restricted or
mirrors blocked.", which discarded the explanation and blamed the user's
network. ApiResponseError now carries `serverMessage`, set only when the
server
  explained itself, so the status-line placeholder still falls back.

Two latency fixes for the cost that made the timeout reachable at all:

- Fetch each distinct AA search URL once per search. The language-filter
retry
re-runs every title variant, and with DIRECT_DOWNLOAD_LANGUAGE_FROM_PATH
on
both passes build a byte-identical URL - behind DDoS-Guard each repeat
is a
  fresh browser solve.
- Drop the solve-only bypass method. `_bypass_method_cdp_gui_click`
opens with
exactly that call and returns the moment it works, so the entry ahead of
it
could only repeat the half that had already failed, plus the backoff
before
the method that does work started. Reported at 0/19 successes and ~5.5s
of
  each ~26s solve against DDoS-Guard.

2. The query carried every contributor, not one author (#1252)

`_pick_search_author` returned `book.search_author` verbatim while the
authors[]
fallback beside it deliberately narrowed to the first name before a
comma. Both
fields routinely arrive holding every contributor joined with ", ": the
frontend
builds `book.author` as `authors.join(', ')` for display
(bookTransformers.ts)
and the release modal sends that display string straight back as the
`author`
parameter, and `browse_record_to_book_metadata` and the manual-search
branch
both split the joined text into `authors` while still passing the
unsplit string
as `search_author`, so the split was never used.

A book whose metadata lists translators was therefore searched for as

    Blindness Jose Saramago, Giovanni Pontiero, <persian translator>

which matches nothing on Anna's Archive. The bypass succeeds, the search
comes
back empty, and the user is told the book has no releases.

Narrowed in one place, `search_plan.first_author`, so the two branches
cannot
drift apart again, and applied to the IRC source, which built its query
with the
same verbatim preference. Hardcover is unaffected: it already sets
`search_author` from `_simplify_author_for_search(authors[0])`, which
resolves
"Last, First" itself and never yields a multi-author string.
2026-09-01 12:38:57 -04:00
CaliBrain 633004ecf0 fix(search): stop reading real Anna's Archive pages as unsolved challenges (#1294)
`_looks_like_challenge_page` substring-matched "ddos-guard"/"cloudflare"
over
the whole document. DDoS-Guard-fronted sites carry those strings on
their own
pages - Anna's Archive ships a `DDOS-GUARD` comment in the inline JS it
serves
on every page - so every real AA response that was not a results table
was
reported as an unsolved protection challenge, sending users off to fix a
bypasser that had just succeeded.

Measured against live pages: a served AA page (HTTP 200) is 182,685
bytes and
matched the old detector; the real interstitial is 902 bytes.

- `_looks_like_challenge_page` now delegates to the shared
`challenge_marker()`,
whose 64 KB cap is what separates a few-KB interstitial from the page
behind
it. `download/http.py` already used it; this module carried an unguarded
  private copy.
- `_looks_like_aa_page` is checked ahead of the challenge branch. A
genuine
interstitial carries no AA markers, so nothing actually blocked leaks
through.

Also adds the diagnostics whose absence made #1289 guesswork: the debug
bundle
carries no response bodies, so "unsolved protection challenge" and
FlareSolverr's
"Challenge solved!" were indistinguishable after the fact.

- `_log_untabled_search_page()` fingerprints the one ambiguous shape at
INFO -
size, size-cap verdict, AA markers, challenge marker - with a bounded
700-char
  head at DEBUG. Best-effort: it swallows its own errors.
- The external bypasser records what it actually returned, and warns
when it
  reports success while handing back a challenge page.

Regression tests use fixtures built from the live pages rather than
invented
ones; the previous fixtures were two-line synthetic pages with no
"ddos-guard"
substring, which is why nothing caught this.

Closes #1289
Closes #1292
2026-09-01 11:19:01 -04:00
CaliBrain 97e289ae13 fix: search, Prowlarr and qBittorrent follow-ups (#1276, #1283) (#1284) 2026-08-30 03:09:13 -04:00
CaliBrain 95e34670f7 fix/group archive extracted audiobooks (#1261)
- fix: group multi-file audiobooks that arrive as an archive
- Surface the concrete reason when a direct-download fetch fails
2026-08-24 01:25:00 -04:00
CaliBrain 651096ed7b fix(bypass): reuse external bypasser clearance instead of re-solving (#1223)
Direct download was unusable behind an external bypasser (FlareSolverr /
Byparr): every request paid a 403 plus a full solve, and a search that
never ran was reported to the user as "No books found".

Clearance was discarded on the external path. get_cf_cookies_for_domain
and get_cf_user_agent_for_domain returned {} / None whenever
USING_EXTERNAL_BYPASSER was set, and _fetch_via_bypasser read only
solution.response - dropping solution.cookies and solution.userAgent,
which FlareSolverr-compatible services do return. A solve therefore
cleared the one request that paid for it and nothing else, and a file
download - which the solver cannot proxy, being binary - presented no
clearance at all. Diagnosed from a v1.3.9 debug bundle: ~35s in the
bypasser per search, on every search.

- Move the cookie jar out of internal_bypasser into bypass/cookie_store.
  internal_bypasser imports seleniumbase at module scope, which is the
  dependency an external-bypasser deployment is entitled not to have, so
  it cannot host a store the external path depends on.
- Harvest solution.cookies and solution.userAgent after a successful
  solve. The existing filtering applies unchanged, so the per-check
  __ddg8_/__ddg9_/__ddg10_ trio is still dropped and the external path
  cannot reintroduce the ?check=1 loop fixed in ebb833a. The UA matters
  as much as the cookies: Cloudflare ties cf_clearance to the UA that
  solved the challenge.
- Read cookie fields from either shape - CDP objects or JSON mappings.
  Both use the same field names, expires included.
- Point http.py's getters and _purge_clearance at the shared store, so
  either bypasser fills and drains the same jar.
- Give the Docker helper-subprocess handoff explicit export_store /
  import_store rather than reaching into module globals.

An unsolved challenge was also indistinguishable from an empty result.
_looks_like_aa_page() counted the challenge markers as "recognisably
AA", so _fetch_search_table handed a DDoS-Guard interstitial back as a
legitimate no-table response and the user was told their query found
nothing when the search never ran. Split challenge detection out and
raise SearchUnavailableError with the reason instead. The mirror is
still not quarantined - every mirror shares the same protection, so it
is not the mirror's fault.

Verified: 2531 unit tests pass; ruff, basedpyright and vulture clean;
e2e bypasser-external profile passes (5). Its mock FlareSolverr already
returned cookies and userAgent from /v1 - the contract was there,
shelfmark was not reading it.

Refs #1220. Deliberately not "Fixes": this removes the re-solve and
makes a failed solve legible, but if Byparr genuinely cannot clear AA's
current DDoS-Guard, the reporter now gets that as an error rather than a
silent "no books found". The download path may swallow interstitials the
same way; not audited here.
2026-08-16 12:08:45 -04:00
CaliBrain b656f019be feat(download): add DoH wireformat support, mirror quarantine, and search warmup (#1218)
- Add RFC 8484 DNS wireformat codec and HTTP/2 support (httpx) for
Quad9/OpenDNS DoH providers.
- Quarantine dead, parked, or seized mirrors for the session on hard
failure (DNS errors, connection refused, 410/451, parked pages) while
preserving bypass clearance on live mirrors.
- Add background startup search warmup to prime DNS, elect mirrors, and
pre-solve protection challenges to eliminate cold-start search latency.
- Add comprehensive test suites for DoH wireformat, mirror quarantine,
parked domain detection, and search warmup.
2026-08-15 14:17:44 -04:00
aba1a68dda Backport bug fixes from NemesisHubris/litfinder (addresses #999, #956, #1010, #1021, #1025, #1040) (#1066)
## Backport bug fixes from `NemesisHubris/litfinder`

Forwards a curated set of bug fixes from
[NemesisHubris/litfinder](https://github.com/NemesisHubris/litfinder) —
a community fork of this project — that address open issues here. All
commits preserve original authorship via `git cherry-pick`; this PR is a
backport rather than original work. Each fix has been reviewed locally,
lint/format-cleaned to match this repo's existing ruff config, and
verified with the test suite. Rebrand strings, license switches, and
features have been deliberately excluded.

### Upstream issues addressed

- **#999** — Mirror URLs with query params no longer break search
requests (strip query string/fragment in `normalize_http_url`)
- **#956** — Apprise notifications now respect the configured proxy
(proxy env vars injected before dispatch)
- **#1025** — rTorrent: separate `RTORRENT_AUDIOBOOK_LABEL` setting,
falls back to book label if unset
- **#1010** — Stop button in Activity no longer makes the panel
disappear (snapshot refresh on cancel)
- **#1021** — Anna's Archive slow-download countdown now caps retries
instead of looping forever
- **#1040** — Empty destination directory cleaned up when write probe
fails
- **PR #1031** — Language detection from Anna's Archive distant path
when listing metadata is missing

### Additional fixes (no open issue but clear bugs)

- **fix: Python 2 `except` syntax across 27 files** — `except X, Y:` is
a SyntaxError in Python 3 and prevents affected modules from importing
at runtime. Mechanical sweep to `except (X, Y):`.
- **fix(abb): info hash validation with magnet fallback** — adds
SHA-1/SHA-256 hex validation on extracted info hashes; falls back to
scanning the full page for a magnet link (e.g. posted in comments) when
the table value is malformed. Also extends the exact-phrase fallback to
manual queries and defaults the ABB listing language to `en` when
missing, preventing valid results from being hidden by the language
filter. Includes a small test-fixture fix (`test(abb): use valid hex
info hashes in scraper test fixtures`) since the existing fixtures used
non-hex placeholders that the new validation correctly rejects.
- **fix: Anna's Archive title parser** — handles nested edition spans
and filters `lgli` catalog descriptor entries (e.g. "Book/Online Audio")
that were polluting search results.

### Deliberately not included

- LitFinder rebranding (UI strings, Apprise app ID, logo). The `fix:
three upstream bugs` commit (#999/#956/#1025) was cherry-picked with
Apprise app-id, description, and logo-URL strings reverted from
"LitFinder" back to "Shelfmark"; noted in the commit body.
- Features from the LitFinder fork (multi-variant title search,
multi-book flat-folder grouping, fuzzy text matching, "Leave in Place"
output handler, admin display name, custom-source plugin system). These
are larger behavior changes that each warrant their own focused review —
happy to send any of them separately if of interest.
- LitFinder-specific test environment and CI infrastructure.

### Verification

- Backend: **1879 passed**, 96 skipped (1 preexisting failure on
`seleniumbase`-dependent test in local venv; runs fine in the standard
Docker image with the `browser` extra)
- Lint, format, dead-code: all clean against this repo's existing
ruff/vulture config
- One follow-up cleanup commit (`style: ruff lint and format fixes for
ported commits`) brings the cherry-picked code into compliance with this
repo's ruff settings — no behavior changes there

### Etiquette / credit

Per-commit authorship preserved by cherry-pick. The only edits to the
original commits are:
- `fix: three upstream bugs` — Apprise rebrand strings reverted to
"Shelfmark" (noted in commit body, original author retained as
`Co-Authored-By` via cherry-pick)
- One follow-up `style:` commit for ruff config alignment

Big thanks to [@NemesisHubris](https://github.com/NemesisHubris) for the
original work in LitFinder; this PR exists to make sure these fixes
reach Shelfmark's wider user base. Happy to revise scope, split into
smaller PRs, or split off the Py2 cleanup separately if that's
preferable.

---------

Co-authored-by: NemesisHubris <155838970+NemesisHubris@users.noreply.github.com>
Co-authored-by: CaliBrain <calibrain@l4n.xyz>
2026-06-15 00:28:43 -04:00
Alex b28ad55d46 Relax URL hardening (#984) 2026-05-12 08:47:53 +01:00
Alex d1fd93f180 Harden Welib URL validation (#979) 2026-05-10 15:20:43 +01:00
Alex b038867d8d Validate Welib fallback URLs (#977) 2026-05-10 10:28:52 +01:00
Alex e35b4c47a7 Direct source refactor (#895)
- Updated mirror selection
- Removed built-in mirror options, users must provide their own
configurations
- Set Universal search to default, added ability to disable direct
source
- Updated documentation
- Updated makefile
2026-04-15 18:50:13 +01:00
Alex d7b9f2e67f Backend test hardening + quality enforcement (#872)
- Reworked many tests
- Enforcing lint + type checking for test suite
- Fixed various issues surfaced by the new tests
- CI tweaks
2026-04-12 12:01:52 +01:00
Alex 8bb188c903 Refactor direct source to use universal API (#711) 2026-03-06 12:59:37 +00:00
Alex 3be99effe4 Base url additions and bug fixes (#519)
- Base URL option in settings for reverse proxy setups
- Fix NZB downloads not deleting on completion
- Fix handling for audiobook files over 100+ parts
- Fix prowlarr search timeout 
- Fix prowlarr categorisation for expanded searches
2026-01-23 13:03:02 +00:00
Alex f7375d56e2 Heuristic searches, full language support, manual search override (#483)
- Added heuristic-based author and title query creation, stripping out
unnecessary elements that could limit searches
- Improved language support when using Hardcover. Searches will now be
conducted on a per-language basis using localized book titles.
- Added manual search override option in the release modal.
2026-01-17 18:56:10 +00:00