html_get_page follows Anna's Archive redirects by hand, and DDoS-Guard's
gate
answers /search with a 302 to the same path plus `check=1`. The follower
walks
that handshake by reassigning `current_url`, so every downstream handoff
- the
403 branch, the 503-challenge branch, both redirect-loop rescues -
passed the
*probe* URL to the bypasser rather than the page we actually wanted.
A solver opens that in a fresh browser holding none of the cookies the
probe
exists to collect, so DDoS-Guard cannot verify it automatically and
serves the
manual CAPTCHA page that nothing can solve. The #1292 log is exactly
that: a 403
handed off on `&check=1`, FlareSolverr answering "Challenge solved!",
and a
4721-byte DDOS-GUARD captcha page coming back.
- `_solvable_url()` strips the probe parameter, applied at the single
choke point
in `_run_bypasser` so all four handoffs are covered. Scoped to the hosts
whose
redirects we follow manually; a URL without the parameter is returned by
identity, so nothing else is re-encoded.
The same reports showed three further defects, all of which stand
whatever the
host was reacting to:
- The external bypasser logged that the solve had not cleared the
protection and
then returned the challenge page as a success. That skipped the one
recovery
left - get_bypassed_page's retry-and-rotate loop, where the next mirror
is a
different DDoS-Guard host - and filed the captcha page's own __ddg
cookies as
that host's clearance, to be replayed on every later request. It now
raises
ChallengeNotSolvedError before storing anything.
- "Check that the bypasser is reachable and working" was the one piece
of advice
guaranteed to waste the reporter's time: it was reachable, it ran a full
solve,
and it returned a captcha. ChallengeNotSolvedError carries the marker so
the
search layer can name the host as the cause instead of the bypasser.
- The untabled-page fingerprint logged `attempt_url`, which
html_get_page has
since rotated past. The #1298 bundle reported the page against
annas-archive.gl
when the body had come from .pk - the triage cost #1289 added the line
to
remove. The search now asks for the response URL and logs that. Its
give-up
shape is the tuple ("", url), which is truthy, so the exhaustion check
reads
the body rather than the response.
Regression fixtures are built from the pages in the reports. The two
behavioural
handoff tests were checked against the unfixed code: both fail there,
reproducing
the reporter's log line verbatim.
Refs #1292
Refs #1298
Two independent reasons a working search reported failure to the user.
1. The client gave up before the server did (#1285)
`/api/releases` bounds one release search with RELEASE_SEARCH_TIMEOUT
(default
300s) and answers a spent budget with a sentence naming the real cause -
the
machinery added for #1276. The frontend then aborted the direct_download
search
at a hard-coded 180s, so it always won the race: the user saw "Request
timed
out. Check your network connection or proxy configuration." instead, and
raising RELEASE_SEARCH_TIMEOUT changed nothing they could observe, the
180s
being baked into the hashed bundle inside the image.
- /api/config reports the effective (clamped) budget, and the client
derives its
abort from it plus a margin, so the server always answers first.
- Direct-mode search shows what the server actually said. Every non-auth
failure
was relabelled "Unable to reach download source. Network may be
restricted or
mirrors blocked.", which discarded the explanation and blamed the user's
network. ApiResponseError now carries `serverMessage`, set only when the
server
explained itself, so the status-line placeholder still falls back.
Two latency fixes for the cost that made the timeout reachable at all:
- Fetch each distinct AA search URL once per search. The language-filter
retry
re-runs every title variant, and with DIRECT_DOWNLOAD_LANGUAGE_FROM_PATH
on
both passes build a byte-identical URL - behind DDoS-Guard each repeat
is a
fresh browser solve.
- Drop the solve-only bypass method. `_bypass_method_cdp_gui_click`
opens with
exactly that call and returns the moment it works, so the entry ahead of
it
could only repeat the half that had already failed, plus the backoff
before
the method that does work started. Reported at 0/19 successes and ~5.5s
of
each ~26s solve against DDoS-Guard.
2. The query carried every contributor, not one author (#1252)
`_pick_search_author` returned `book.search_author` verbatim while the
authors[]
fallback beside it deliberately narrowed to the first name before a
comma. Both
fields routinely arrive holding every contributor joined with ", ": the
frontend
builds `book.author` as `authors.join(', ')` for display
(bookTransformers.ts)
and the release modal sends that display string straight back as the
`author`
parameter, and `browse_record_to_book_metadata` and the manual-search
branch
both split the joined text into `authors` while still passing the
unsplit string
as `search_author`, so the split was never used.
A book whose metadata lists translators was therefore searched for as
Blindness Jose Saramago, Giovanni Pontiero, <persian translator>
which matches nothing on Anna's Archive. The bypass succeeds, the search
comes
back empty, and the user is told the book has no releases.
Narrowed in one place, `search_plan.first_author`, so the two branches
cannot
drift apart again, and applied to the IRC source, which built its query
with the
same verbatim preference. Hardcover is unaffected: it already sets
`search_author` from `_simplify_author_for_search(authors[0])`, which
resolves
"Last, First" itself and never yields a multi-author string.
Follow-up to #1276 with a measurement from the instance I reported
there. v1.3.13 solves the challenge again, but on my setup the solve was
being thrown away immediately afterwards:
```
19:26:08 Bypass successful using _bypass_method_cdp_gui_click
19:26:16 Bypass failed (attempt 1/10): TimeoutError: Time ran out while waiting for: {html}
```
`_get()` ends with `return await page.get_page_source()`, which is
`find("html", timeout=1)` in SeleniumBase. One second is enough for a
page that is already sitting on its content, but Anna's Archive answers
a cleared check with a redirect to the real page, so the document is not
there yet. The solve is discarded, the whole attempt restarts, and the
extra requests are what earn the 429 that `note_rate_limited()` then
parks the host for — 120 s, then 300 s.
## Change
`_read_page_source()` waits for the document itself, with a
`BYPASS_PAGE_SOURCE_TIMEOUT` setting (default 20 s, min 1, max 120) in
Direct Download → Cloudflare Bypass, next to the existing bypasser
timeouts.
## Measured on a live instance
I patched the wait in the running container (`find("html", timeout=1)` →
`timeout=20` in the installed seleniumbase, which is the same effect as
this PR) and re-ran the same searches on the same host, k3s behind a
Surfshark WireGuard exit, internal bypasser, v1.3.13:
| | 1 s wait | 20 s wait |
|---|---|---|
| `Time ran out while waiting for: {html}` | one per solve | none |
| 429 backoffs | 2 (120 s, then 300 s) | none |
| Search for a book AA has | 199 s and 200 s, both errored | 61 s, 2
epub releases |
A download after that took 5 s from LibGen, so the search was the whole
cost.
## Tests
Two tests in `tests/bypass/test_bypass_budgets.py`, the file already
covering #1276: a page that needs longer than a second still yields its
HTML, and `BYPASS_PAGE_SOURCE_TIMEOUT` overrides the default.
`uv run pytest tests/ --ignore=tests/e2e`: 2848 passed, 47 skipped. Ruff
check and format clean. The docs table is auto-generated, but running
`scripts/generate_env_docs.py` here rewrote unrelated entries (Newznab,
BOOK_LANGUAGE), so I added only the new entry by hand in the generator's
format rather than commit that churn.
One thing I could not judge from outside: whether 20 s is the right
default for hosts other than AA. It only costs anything when a solve
would otherwise be discarded, but I have measured it on one site.
assumptions the code around it still made were written for a helper that
was killed after every request.
A bypass that hits the child's deadline is cancelled from the calling
thread, which returns the moment the cancellation is scheduled - so the
helper went on to serve the next request while the abandoned one was
still closing its browser, on the same loop, sharing the DISPLAY globals
and one process group. The deadline now lives inside the loop, where
asyncio.wait_for() waits for the unwind before it raises, with the
calling thread keeping a bounded backstop in case the cleanup wedges
too. Both budgets are set so the child still answers before the parent
gives up on it.
The helper's cookie store survived the request as well, and the whole of
it is exported back to the parent on every answer - so clearance the
parent had purged for one host came back the next time some other host
was solved, the dead-cookie resurrection _redirect_loop_handoff purges
to avoid. The child starts each request from an empty store again; the
parent already runs the cached-cookie check against a superset of it.
DNS config is compared against what the helper is actually resolving
through rather than skipped whenever the parent reports "auto", so a
user flipping CUSTOM_DNS back to auto - which applies live - reaches a
warm helper instead of leaving it on an abandoned DoH resolver.
The 15s exit grace is now asked only of a helper that can still read its
stdin. One dropped mid-bypass never returns to that read, so the grace
could only end in the kill - while a user cancelling a download, and
every bypass queued behind them on LOCKED, waited it out.
Result files are cleaned on the timeout and cancellation paths too,
staging file included, rather than only when the answer was read.
Every protected request spawns a fresh helper subprocess, paying
interpreter start and imports before any work begins. Measured inside
the container, five consecutive runs of `python -c "import
shelfmark.bypass.internal_bypasser"`:
```
3.53s 3.45s 3.55s 3.54s 3.46s
```
A single search issues several protected requests, so that is paid
several times over per search.
## What changed
The helper now serves one JSON request per line of stdin until the
parent closes the pipe, and an idle timer
(`BYPASS_BROWSER_IDLE_TIMEOUT`, default 180s) shuts it down once
searching stops.
Answers still travel by result file, but the file is now written to a
`.part` path and renamed into place — the parent treats the file's
existence as the answer, so it must never observe a half-written one.
stdout and stderr stay attached to the parent's, so helper logs keep
appearing in `docker logs` exactly as before.
Failure handling, since a warm helper is exposed to more of it than a
per-request one ever was:
| Situation | Handling |
| --- | --- |
| Helper died between requests | Detected via `poll()`, respawned |
| Pipe broken at write time (`poll()` can miss this) | One retry on a
fresh process; a fresh one failing there is a real failure |
| Helper exits without writing a result | `RuntimeError` naming the exit
code |
| Wedged past the timeout, or cancelled mid-bypass | Helper killed, then
`_cleanup_orphan_processes` because a killed helper never got to close
Chrome |
| Idle reaper racing an arriving request | Re-checks the deadline under
the lock and re-arms instead of killing a helper that just did work |
The DNS config now travels with every request rather than only at spawn:
a warm helper outlives changes the parent makes to its provider.
## `BYPASS_REUSE_BROWSER`, off by default
This parks the CDP driver between bypasses. A driver's websockets are
bound to the loop that opened them and cannot outlive their process, so
the persistent helper is what makes this possible at all — and the warm
path runs on `_CDP_WORKER`'s long-lived loop rather than `asyncio.run`
for the same reason.
The mechanism works. With it on, the browser start disappears from the
second request onward: 0.7s from `Reusing warm Chrome browser` to the
first bypass attempt, against roughly 16s cold.
**It still ships off, because a matched-pair test shows it is a net loss
against DDoS-Guard.** Each round primed with one cold bypass, waited
10s, then measured a second — identical timing in both arms, only the
browser strategy differing, order balanced (fresh, warm, warm, fresh) so
drift over the session cannot masquerade as an effect:
| Arm | Measured request |
| --- | --- |
| fresh browser | 42.8s, 40.6s |
| warm browser | 57.1s, 59.6s |
Spread within each arm is 2.2s and 2.5s, against 16.7s between them.
Reuse removes the ~15s browser start and then gives back roughly twice
that in solving: a returning browser draws a harder challenge. Where the
cold browser is through on the second bypass method, the warm one fails
the first three and only `_bypass_method_humanlike` gets it, at ~30s for
that method alone.
Worth separating from a second effect I ran into while measuring: five
back-to-back searches slow from ~32s to 51–98s with reuse **disabled**
as well, so DDoS-Guard escalates on request rate independently of any of
this. That is why the pairs above are timed identically rather than
simply run in sequence. It is the larger of the two effects, but not
something this project can patch around.
Reuse is left available rather than dropped because Cloudflare sites may
not respond the same way, and because the two concerns are independent:
the helper start is pure overhead and always worth removing, the browser
is not.
## Verification
- 2559 unit tests pass (2542 before, 17 added in
`tests/bypass/test_warm_browser.py`)
- `ruff check`, `ruff format`, `basedpyright` over backend and tests,
and `vulture` all clean
- `docs/environment-variables.md` regenerated via
`scripts/generate_env_docs.py`
- Live against Anna's Archive on a warm helper: searches return their
usual ~760KB and 667 results, the app's own search warm-up completes
with 50 results, and the container is left with no orphan
chrome/Xvfb/ffmpeg processes
Happy to drop the `BYPASS_REUSE_BROWSER` half entirely if you would
rather not carry a default-off path — the helper persistence stands on
its own.
Co-authored-by: helgehelge123 <helge.neumann@zollsoft.de>
The orphan sweep ran a container-wide 'pkill -9 -f
chrome|chromium|Xvfb|ffmpeg', so it also matched browsers another bypass
was still driving. Scope it by process group: kill only our own group
and groups whose leader has died. Spawn the helper with
start_new_session so its browser tree is identifiable, tear the whole
group down after every run (a timed-out helper used to leak its Chrome
and Xvfb), and have an orphaned helper take its browser down with it.
Fixes#1231.
Direct download was unusable behind an external bypasser (FlareSolverr /
Byparr): every request paid a 403 plus a full solve, and a search that
never ran was reported to the user as "No books found".
Clearance was discarded on the external path. get_cf_cookies_for_domain
and get_cf_user_agent_for_domain returned {} / None whenever
USING_EXTERNAL_BYPASSER was set, and _fetch_via_bypasser read only
solution.response - dropping solution.cookies and solution.userAgent,
which FlareSolverr-compatible services do return. A solve therefore
cleared the one request that paid for it and nothing else, and a file
download - which the solver cannot proxy, being binary - presented no
clearance at all. Diagnosed from a v1.3.9 debug bundle: ~35s in the
bypasser per search, on every search.
- Move the cookie jar out of internal_bypasser into bypass/cookie_store.
internal_bypasser imports seleniumbase at module scope, which is the
dependency an external-bypasser deployment is entitled not to have, so
it cannot host a store the external path depends on.
- Harvest solution.cookies and solution.userAgent after a successful
solve. The existing filtering applies unchanged, so the per-check
__ddg8_/__ddg9_/__ddg10_ trio is still dropped and the external path
cannot reintroduce the ?check=1 loop fixed in ebb833a. The UA matters
as much as the cookies: Cloudflare ties cf_clearance to the UA that
solved the challenge.
- Read cookie fields from either shape - CDP objects or JSON mappings.
Both use the same field names, expires included.
- Point http.py's getters and _purge_clearance at the shared store, so
either bypasser fills and drains the same jar.
- Give the Docker helper-subprocess handoff explicit export_store /
import_store rather than reaching into module globals.
An unsolved challenge was also indistinguishable from an empty result.
_looks_like_aa_page() counted the challenge markers as "recognisably
AA", so _fetch_search_table handed a DDoS-Guard interstitial back as a
legitimate no-table response and the user was told their query found
nothing when the search never ran. Split challenge detection out and
raise SearchUnavailableError with the reason instead. The mirror is
still not quarantined - every mirror shares the same protection, so it
is not the mirror's fault.
Verified: 2531 unit tests pass; ruff, basedpyright and vulture clean;
e2e bypasser-external profile passes (5). Its mock FlareSolverr already
returned cookies and userAgent from /v1 - the contract was there,
shelfmark was not reading it.
Refs #1220. Deliberately not "Fixes": this removes the re-solve and
makes a failed solve legible, but if Byparr genuinely cannot clear AA's
current DDoS-Guard, the reporter now gets that as an error rather than a
silent "no books found". The download path may swallow interstitials the
same way; not audited here.
A cookie that has been rejected was kept and presented again on every
later
request, so a single bad clearance could re-arm the challenge
indefinitely.
Cookie storage:
- Enforce expiry for every stored cookie, not just cf_clearance.
DDoS-Guard
domains have no cf_clearance, so the existing check never fired for them
and
expired cookies were replayed forever.
- Stop storing the per-check cookies __ddg8_/__ddg9_/__ddg10_ and
ddg_last_challenge. Captured live from Anna's Archive, these carry the
client
IP and the timestamp the check was issued (~40 min), versus ~1 year for
the
__ddg1_/__ddg2_/__ddgid_ clearance. Replaying an IP-bound token stops
describing the caller as soon as the egress IP changes, which is routine
behind a VPN.
Failure handling — every path that is rejected while carrying cookies
now
purges them, not just the redirect loop:
- 403 returned while presenting cookies.
- Cached-cookie attempt rejected, whether by status or by redirect loop.
- Factored the purge into _purge_clearance, guarded on a non-empty
hostname
since clear_cf_cookies("") means "every host" and would wipe clearance
for
sites that are working fine.
Also fix the search warm-up switches shipped inert in v1.3.8:
SEARCH_WARMUP_ENABLED and SEARCH_WARMUP_QUERY are not in the settings
registry, and config.get only consults the environment for keys it
knows, so
both always returned their defaults — the warm-up could not be turned
off or
retargeted. Read os.environ first.
Refs #1220. Deliberately not "Fixes": the reported failure could not be
reproduced on v1.3.8 from a stable IP (the reporter's own queries all
returned
200 on both the pre- and post-change builds), and the new purge paths
did not
fire in live testing because the failures arrive as redirect loops,
which were
already purged. These are correctness fixes with no measured effect on
that
issue. The underlying problem remains that Chrome-obtained cookies never
satisfy DDoS-Guard when replayed by requests, so every search still
re-solves.
Verified: 2542 unit tests pass; ruff, basedpyright and vulture clean;
e2e
platform baseline (10), full (6) and bypasser-external (5) all pass;
five
sequential live searches against Anna's Archive all returned 200 with
zero
"Exceeded 30 redirects".
A download that hits Cloudflare hung on "Bypassing protection..." for
five
minutes and then died, regardless of which bypasser was configured.
html_get_page() started a BypassHeartbeat thread to keep the download
marked
alive during a bypass, but the thread had no loop: it fired one status
event
and returned. Even with the loop restored it could not have worked,
because
update_download_status() dedupes identical (status, message) tuples and
returns before refreshing _last_activity, and the heartbeat re-sent the
byte-identical payload already emitted just above it.
So _last_activity was frozen for the whole bypass, while both bypassers
are
allowed to run longer than STALL_TIMEOUT (external FlareSolverr ~394s at
default settings, internal 420s per get() call). The watchdog always
won.
From a reporter's log: 403 at 07:04:33.390, cancelled at 07:09:33.987 -
exactly 300.000s, and 41s before the bypasser would have finished and
reported the real error, an HTTP 500 from FlareSolverr the user never
saw.
The regression is not one commit. 1f093de (#536) added the heartbeat and
the
dedup together and refreshed activity before the dedup return, so it
worked.
ff094be (#832) moved the refresh below that return while tightening
stall
detection for #823. 3a3a3ce (#845) then deleted the heartbeat's while
loop
to silence a B023 lint, removing the last evidence of intent.
The dedup itself is correct and stays: a keep-alive that ticks on a
timer
proves nothing about whether an operation is progressing, so letting it
refresh the stall clock would make a wedged download immortal. Split the
two
concerns instead.
Add shelfmark/download/activity.py. A long single-shot operation
declares its
own upper bound once, over a sentinel status carried on the existing
status_callback channel - so no new parameter has to be threaded through
every
handler, post-processor and output module. The orchestrator intercepts
the
sentinel in its per-task closure and records an absolute deadline in
_activity_grace, which stall detection honours alongside STALL_TIMEOUT.
The
grace never extends itself and is clamped to
_MAX_ACTIVITY_GRACE_SECONDS, so
an operation that overruns its own declared budget is still cancelled.
Each bypasser now reports max_duration_seconds() derived from its own
retry
and timeout settings, and http.py asks whichever is active, plus 30s of
slack
so the bypasser's own deadline expires first and the user sees its real
failure. On that path html_get_page() also emits
status_callback("error", ...)
rather than silently returning an empty page.
Three further fixes on the same code path:
- Extract the watchdog into _find_stalled_tasks() and
_cancel_stalled_task().
It was the only place holding _progress_lock across a call into
book_queue,
whose terminal-status hooks reach a sqlite write that gevent does not
patch,
blocking the hub and every download worker. It now holds the lock for
dict
reads only.
- Bound _CDP_WORKER.run(), which waited with timeout=None while holding
the
module-wide LOCKED, so a single wedged in-process CDP session blocked
every
subsequent bypass forever on non-Docker installs.
- Broaden the coordinator loop's except clause back to Exception, with
escalating backoff. 8d98e12 (#868) narrowed it to a six-type tuple to
silence BLE001, which let gevent's LoopExit and similar kill the only
thread
driving the download queue - undoing #832's fix for #823 and resurfacing
it
as #1166. GreenletExit and gevent.Timeout still propagate.
Fixes#1001
Refs #1166, #823
- Fixed internal bypasser startup with newer Chromium/SeleniumBase by
isolating the browser helper from Gunicorn/gevent, serialising helper
failures cleanly, and cleaning up orphan processes after a failure
- Stopped using /app as runtime home state, now moved to /home/shelfmark
or /tmp/shelfmark/home as fallback.
- Added tests
- Updated mirror selection
- Removed built-in mirror options, users must provide their own
configurations
- Set Universal search to default, added ability to disable direct
source
- Updated documentation
- Updated makefile
- Added notification support via Apprise dependency
- Notifications can be configured globally or per user, with full
customization of events and notification type.
- Added expanded ActivityCard for increased detail of each request, file
info, and managing the attached file.
- Enhanced tests
CaliBrain
Jorge Lima
helgehelge123andhelgehelge123
Alex
Alex