Commit Graph
20 Commits
Author SHA1 Message Date
Austin Brogle 99e0cfde3d fix: prevent Anna's Archive download countdown resets by preserving browser sessions (#1325)
## Observed bug

Anna's Archive slow-download pages can return a JavaScript countdown
before a download link is available. The internal browser returns that
waiting-room HTML and closes its incognito session. The downloader then
sleeps and fetches the URL again, which can create a new queue session
and **restart the countdown instead of reaching the download link**.

## Fix

- **Preserve the queue session:** keep the original browser tab open
while the site's own countdown and automatic navigation finish. HTTP 200
and cached-cookie waiting-room responses enter the same flow.
- **Return a consistent page:** capture HTML and readiness together in
one browser evaluation so navigation cannot pair a new page's status
with stale protection-page HTML. Share cache validation and
page-readiness rules across their callers.
- **Keep waiting cancellable and bounded:** poll cancellation while a
slow browser read remains pending, rather than repeatedly cancelling and
reissuing it. Apply a **300-second waiting-room limit** within the
existing browser watchdog.
- **Report queue timeouts accurately:** preserve the timeout across the
helper-process boundary and stop the solve without restarting the
browser or rotating mirrors.

Waiting-room detection is limited to Anna's Archive `/slow_download/`
pages containing an actual `.js-partner-countdown` element. The site
controls the countdown and refresh. External-bypasser behavior and
file-transfer timeouts are unchanged; the PR adds no deployment
configuration or dependencies.

## Validation

Validated at `c81e0a2`:

| Check | Result |
| --- | --- |
| Full Linux unit suite | **2,978 passed** on Python 3.14 in a non-root
environment with entrypoint test stubs enabled |
| Focused regression coverage | **30 passed**, covering countdown
completion, zero timers, navigation, both cookie-cache paths,
cancellation, stuck queues, slow reads, and timeout propagation |
| Navigation-race regression | Fails against the previous PR
implementation and passes with the fix |
| Python static checks | Ruff lint/format, BasedPyright for backend and
tests, and Vulture passed |
| Real Chromium fixture | Queue cookie persisted through 1.5-second DOM
reads and one automatic refresh; the CDP connection survived multiple
polling intervals |
| Live source check | Observed **19 → 14 → 9 → 4 → download link** while
retaining the browser session; the patched browser path also completed
the waiting room |

Full unit-suite command:

```sh
pytest tests/ -n 2 --tb=short -m "not integration and not e2e"
```

The live check validates waiting-room completion and link resolution.
Remote file-host availability remains a separate concern. The unit suite
emitted two existing Authlib deprecation warnings.
2026-09-11 00:51:48 -04:00
Jorge Lima 97d1bb0df4 fix(bypass): keep Anna's Archive's aa_ddg_check so clearance replays (#1305)
## What

Add `aa_ddg_check` to the cookie-store allowlist. One name, one test.

## Why

Every replay of stored clearance ends in the `?check=1` redirect loop,
so each search pays a fresh browser solve. On this instance (v1.3.15,
WireGuard egress, 0 VPN restarts across the traces) not one replay was
accepted in three days of DEBUG logs.

The `__ddg*` cookies are stored and replayed correctly. Anna's Archive
also sets a cookie of its own, `aa_ddg_check`, and its `?check=1` hop
only answers with the page when that cookie is present too. The
allowlist keeps `cf_*` and `__ddg*` names, so this one was never stored.

## Measured, same egress IP, cookies taken from one solve

| replayed | plain `requests` | `curl_cffi`, Chrome TLS fingerprint |
|---|---|---|
| filtered `__ddg*` only (current behaviour) | 302 → 302 → 302 … loop |
302 → 302 → 302 … loop |
| filtered + `__ddg8_/9_/10_` | loop | loop |
| filtered + `aa_ddg_check` | **302 → 200, real search page** | 302 →
200 |
| `aa_ddg_check` alone | 302 → 403 | — |

So the TLS fingerprint is not the problem, the per-check trio is not the
answer, and the cookie needs the `__ddg*` clearance next to it. Cookie
attributes as issued: domain `.annas-archive.gl`, path `/`, expiry 90
days. It is not bound to the query, and it is accepted with a stock
Python User-Agent.

## Through the real fetch path

Same process, `html_get_page`, the name allowlisted, three different
queries:

```
1st: solve expected         25.8s  bypass_calls=1  title='frankenstein shelley - search - an'  md5=True
2nd: other query             9.5s  bypass_calls=0  title='pride and prejudice austen - searc'  md5=True
3rd: third query             4.8s  bypass_calls=0  title='dracula stoker - search - anna's a'  md5=True
```

## Notes

- `tests/bypass/test_ddg_cookie_reuse.py` gains
`test_aa_check_cookie_is_stored`; its docstring table gains the row. The
bypass tests need seleniumbase to import and do not run on my macOS
host, so this leans on CI. `ruff check` and `ruff format --check` pass.
The logic was checked directly against `cookie_store` with the settings
registry stubbed.
- `__ddgmark_` carries a 24 h expiry, so the store's clearance is good
for about a day before the next solve, which is what a browser would see
too.
- Follow-up to #1286. Same instance, same method: DEBUG trace, then a
probe script inside the container.
2026-09-03 23:20:48 -04:00
CaliBrain 9452ebc70d fix(bypass): stop handing solvers DDoS-Guard's ?check=1 probe URL (#1300)
html_get_page follows Anna's Archive redirects by hand, and DDoS-Guard's
gate
answers /search with a 302 to the same path plus `check=1`. The follower
walks
that handshake by reassigning `current_url`, so every downstream handoff
- the
403 branch, the 503-challenge branch, both redirect-loop rescues -
passed the
*probe* URL to the bypasser rather than the page we actually wanted.

A solver opens that in a fresh browser holding none of the cookies the
probe
exists to collect, so DDoS-Guard cannot verify it automatically and
serves the
manual CAPTCHA page that nothing can solve. The #1292 log is exactly
that: a 403
handed off on `&check=1`, FlareSolverr answering "Challenge solved!",
and a
4721-byte DDOS-GUARD captcha page coming back.

- `_solvable_url()` strips the probe parameter, applied at the single
choke point
in `_run_bypasser` so all four handoffs are covered. Scoped to the hosts
whose
redirects we follow manually; a URL without the parameter is returned by
  identity, so nothing else is re-encoded.

The same reports showed three further defects, all of which stand
whatever the
host was reacting to:

- The external bypasser logged that the solve had not cleared the
protection and
then returned the challenge page as a success. That skipped the one
recovery
left - get_bypassed_page's retry-and-rotate loop, where the next mirror
is a
different DDoS-Guard host - and filed the captcha page's own __ddg
cookies as
that host's clearance, to be replayed on every later request. It now
raises
  ChallengeNotSolvedError before storing anything.

- "Check that the bypasser is reachable and working" was the one piece
of advice
guaranteed to waste the reporter's time: it was reachable, it ran a full
solve,
and it returned a captcha. ChallengeNotSolvedError carries the marker so
the
  search layer can name the host as the cause instead of the bypasser.

- The untabled-page fingerprint logged `attempt_url`, which
html_get_page has
since rotated past. The #1298 bundle reported the page against
annas-archive.gl
when the body had come from .pk - the triage cost #1289 added the line
to
remove. The search now asks for the response URL and logs that. Its
give-up
shape is the tuple ("", url), which is truthy, so the exhaustion check
reads
  the body rather than the response.

Regression fixtures are built from the pages in the reports. The two
behavioural
handoff tests were checked against the unfixed code: both fail there,
reproducing
the reporter's log line verbatim.

Refs #1292
Refs #1298
2026-09-02 16:12:52 -04:00
CaliBrain 3d7ea40088 fix(search): reach the server's deadline, query one author (#1285, #1252) (#1287)
Two independent reasons a working search reported failure to the user.

1. The client gave up before the server did (#1285)

`/api/releases` bounds one release search with RELEASE_SEARCH_TIMEOUT
(default
300s) and answers a spent budget with a sentence naming the real cause -
the
machinery added for #1276. The frontend then aborted the direct_download
search
at a hard-coded 180s, so it always won the race: the user saw "Request
timed
out. Check your network connection or proxy configuration." instead, and
raising RELEASE_SEARCH_TIMEOUT changed nothing they could observe, the
180s
being baked into the hashed bundle inside the image.

- /api/config reports the effective (clamped) budget, and the client
derives its
  abort from it plus a margin, so the server always answers first.
- Direct-mode search shows what the server actually said. Every non-auth
failure
was relabelled "Unable to reach download source. Network may be
restricted or
mirrors blocked.", which discarded the explanation and blamed the user's
network. ApiResponseError now carries `serverMessage`, set only when the
server
  explained itself, so the status-line placeholder still falls back.

Two latency fixes for the cost that made the timeout reachable at all:

- Fetch each distinct AA search URL once per search. The language-filter
retry
re-runs every title variant, and with DIRECT_DOWNLOAD_LANGUAGE_FROM_PATH
on
both passes build a byte-identical URL - behind DDoS-Guard each repeat
is a
  fresh browser solve.
- Drop the solve-only bypass method. `_bypass_method_cdp_gui_click`
opens with
exactly that call and returns the moment it works, so the entry ahead of
it
could only repeat the half that had already failed, plus the backoff
before
the method that does work started. Reported at 0/19 successes and ~5.5s
of
  each ~26s solve against DDoS-Guard.

2. The query carried every contributor, not one author (#1252)

`_pick_search_author` returned `book.search_author` verbatim while the
authors[]
fallback beside it deliberately narrowed to the first name before a
comma. Both
fields routinely arrive holding every contributor joined with ", ": the
frontend
builds `book.author` as `authors.join(', ')` for display
(bookTransformers.ts)
and the release modal sends that display string straight back as the
`author`
parameter, and `browse_record_to_book_metadata` and the manual-search
branch
both split the joined text into `authors` while still passing the
unsplit string
as `search_author`, so the split was never used.

A book whose metadata lists translators was therefore searched for as

    Blindness Jose Saramago, Giovanni Pontiero, <persian translator>

which matches nothing on Anna's Archive. The bypass succeeds, the search
comes
back empty, and the user is told the book has no releases.

Narrowed in one place, `search_plan.first_author`, so the two branches
cannot
drift apart again, and applied to the IRC source, which built its query
with the
same verbatim preference. Hardcover is unaffected: it already sets
`search_author` from `_simplify_author_for_search(authors[0])`, which
resolves
"Last, First" itself and never yields a multi-author string.
2026-09-01 12:38:57 -04:00
Jorge Lima d7fe28595c fix(bypass): wait for the solved page before reading its source (#1286)
Follow-up to #1276 with a measurement from the instance I reported
there. v1.3.13 solves the challenge again, but on my setup the solve was
being thrown away immediately afterwards:

```
19:26:08 Bypass successful using _bypass_method_cdp_gui_click
19:26:16 Bypass failed (attempt 1/10): TimeoutError: Time ran out while waiting for: {html}
```

`_get()` ends with `return await page.get_page_source()`, which is
`find("html", timeout=1)` in SeleniumBase. One second is enough for a
page that is already sitting on its content, but Anna's Archive answers
a cleared check with a redirect to the real page, so the document is not
there yet. The solve is discarded, the whole attempt restarts, and the
extra requests are what earn the 429 that `note_rate_limited()` then
parks the host for — 120 s, then 300 s.

## Change

`_read_page_source()` waits for the document itself, with a
`BYPASS_PAGE_SOURCE_TIMEOUT` setting (default 20 s, min 1, max 120) in
Direct Download → Cloudflare Bypass, next to the existing bypasser
timeouts.

## Measured on a live instance

I patched the wait in the running container (`find("html", timeout=1)` →
`timeout=20` in the installed seleniumbase, which is the same effect as
this PR) and re-ran the same searches on the same host, k3s behind a
Surfshark WireGuard exit, internal bypasser, v1.3.13:

| | 1 s wait | 20 s wait |
|---|---|---|
| `Time ran out while waiting for: {html}` | one per solve | none |
| 429 backoffs | 2 (120 s, then 300 s) | none |
| Search for a book AA has | 199 s and 200 s, both errored | 61 s, 2
epub releases |

A download after that took 5 s from LibGen, so the search was the whole
cost.

## Tests

Two tests in `tests/bypass/test_bypass_budgets.py`, the file already
covering #1276: a page that needs longer than a second still yields its
HTML, and `BYPASS_PAGE_SOURCE_TIMEOUT` overrides the default.

`uv run pytest tests/ --ignore=tests/e2e`: 2848 passed, 47 skipped. Ruff
check and format clean. The docs table is auto-generated, but running
`scripts/generate_env_docs.py` here rewrote unrelated entries (Newznab,
BOOK_LANGUAGE), so I added only the new entry by hand in the generator's
format rather than commit that churn.

One thing I could not judge from outside: whether 20 s is the right
default for hosts other than AA. It only costs anything when a solve
would otherwise be discarded, but I have measured it on one site.
2026-08-30 19:17:15 -04:00
CaliBrain 97e289ae13 fix: search, Prowlarr and qBittorrent follow-ups (#1276, #1283) (#1284) 2026-08-30 03:09:13 -04:00
CaliBrain 5247ec6124 fix(bypass): close the gaps a helper that outlives its request opened (#1244)
assumptions the code around it still made were written for a helper that
was killed after every request.

A bypass that hits the child's deadline is cancelled from the calling
thread, which returns the moment the cancellation is scheduled - so the
helper went on to serve the next request while the abandoned one was
still closing its browser, on the same loop, sharing the DISPLAY globals
and one process group. The deadline now lives inside the loop, where
asyncio.wait_for() waits for the unwind before it raises, with the
calling thread keeping a bounded backstop in case the cleanup wedges
too. Both budgets are set so the child still answers before the parent
gives up on it.

The helper's cookie store survived the request as well, and the whole of
it is exported back to the parent on every answer - so clearance the
parent had purged for one host came back the next time some other host
was solved, the dead-cookie resurrection _redirect_loop_handoff purges
to avoid. The child starts each request from an empty store again; the
parent already runs the cached-cookie check against a superset of it.

DNS config is compared against what the helper is actually resolving
through rather than skipped whenever the parent reports "auto", so a
user flipping CUSTOM_DNS back to auto - which applies live - reaches a
warm helper instead of leaving it on an abandoned DoH resolver.

The 15s exit grace is now asked only of a helper that can still read its
stdin. One dropped mid-bypass never returns to that read, so the grace
could only end in the kill - while a user cancelling a download, and
every bypass queued behind them on LOCKED, waited it out.

Result files are cleaned on the timeout and cancellation paths too,
staging file included, rather than only when the answer was read.
2026-08-20 19:29:33 -04:00
helgehelge123andhelgehelge123 7b9c416df8 perf(bypass): keep the helper subprocess alive between bypasses (#1222)
Every protected request spawns a fresh helper subprocess, paying
interpreter start and imports before any work begins. Measured inside
the container, five consecutive runs of `python -c "import
shelfmark.bypass.internal_bypasser"`:

```
3.53s  3.45s  3.55s  3.54s  3.46s
```

A single search issues several protected requests, so that is paid
several times over per search.

## What changed

The helper now serves one JSON request per line of stdin until the
parent closes the pipe, and an idle timer
(`BYPASS_BROWSER_IDLE_TIMEOUT`, default 180s) shuts it down once
searching stops.

Answers still travel by result file, but the file is now written to a
`.part` path and renamed into place — the parent treats the file's
existence as the answer, so it must never observe a half-written one.
stdout and stderr stay attached to the parent's, so helper logs keep
appearing in `docker logs` exactly as before.

Failure handling, since a warm helper is exposed to more of it than a
per-request one ever was:

| Situation | Handling |
| --- | --- |
| Helper died between requests | Detected via `poll()`, respawned |
| Pipe broken at write time (`poll()` can miss this) | One retry on a
fresh process; a fresh one failing there is a real failure |
| Helper exits without writing a result | `RuntimeError` naming the exit
code |
| Wedged past the timeout, or cancelled mid-bypass | Helper killed, then
`_cleanup_orphan_processes` because a killed helper never got to close
Chrome |
| Idle reaper racing an arriving request | Re-checks the deadline under
the lock and re-arms instead of killing a helper that just did work |

The DNS config now travels with every request rather than only at spawn:
a warm helper outlives changes the parent makes to its provider.

## `BYPASS_REUSE_BROWSER`, off by default

This parks the CDP driver between bypasses. A driver's websockets are
bound to the loop that opened them and cannot outlive their process, so
the persistent helper is what makes this possible at all — and the warm
path runs on `_CDP_WORKER`'s long-lived loop rather than `asyncio.run`
for the same reason.

The mechanism works. With it on, the browser start disappears from the
second request onward: 0.7s from `Reusing warm Chrome browser` to the
first bypass attempt, against roughly 16s cold.

**It still ships off, because a matched-pair test shows it is a net loss
against DDoS-Guard.** Each round primed with one cold bypass, waited
10s, then measured a second — identical timing in both arms, only the
browser strategy differing, order balanced (fresh, warm, warm, fresh) so
drift over the session cannot masquerade as an effect:

| Arm | Measured request |
| --- | --- |
| fresh browser | 42.8s, 40.6s |
| warm browser | 57.1s, 59.6s |

Spread within each arm is 2.2s and 2.5s, against 16.7s between them.
Reuse removes the ~15s browser start and then gives back roughly twice
that in solving: a returning browser draws a harder challenge. Where the
cold browser is through on the second bypass method, the warm one fails
the first three and only `_bypass_method_humanlike` gets it, at ~30s for
that method alone.

Worth separating from a second effect I ran into while measuring: five
back-to-back searches slow from ~32s to 51–98s with reuse **disabled**
as well, so DDoS-Guard escalates on request rate independently of any of
this. That is why the pairs above are timed identically rather than
simply run in sequence. It is the larger of the two effects, but not
something this project can patch around.

Reuse is left available rather than dropped because Cloudflare sites may
not respond the same way, and because the two concerns are independent:
the helper start is pure overhead and always worth removing, the browser
is not.

## Verification

- 2559 unit tests pass (2542 before, 17 added in
`tests/bypass/test_warm_browser.py`)
- `ruff check`, `ruff format`, `basedpyright` over backend and tests,
and `vulture` all clean
- `docs/environment-variables.md` regenerated via
`scripts/generate_env_docs.py`
- Live against Anna's Archive on a warm helper: searches return their
usual ~760KB and 667 results, the app's own search warm-up completes
with 50 results, and the container is left with no orphan
chrome/Xvfb/ffmpeg processes

Happy to drop the `BYPASS_REUSE_BROWSER` half entirely if you would
rather not carry a default-off path — the helper persistence stands on
its own.

Co-authored-by: helgehelge123 <helge.neumann@zollsoft.de>
2026-08-20 19:00:06 -04:00
CaliBrain fae6140c6a fix(bypass): scope browser cleanup to the calling session (#1232)
The orphan sweep ran a container-wide 'pkill -9 -f
chrome|chromium|Xvfb|ffmpeg', so it also matched browsers another bypass
was still driving. Scope it by process group: kill only our own group
and groups whose leader has died. Spawn the helper with
start_new_session so its browser tree is identifiable, tear the whole
group down after every run (a timed-out helper used to leak its Chrome
and Xvfb), and have an orphaned helper take its browser down with it.
Fixes #1231.
2026-08-18 23:24:43 -04:00
CaliBrain 651096ed7b fix(bypass): reuse external bypasser clearance instead of re-solving (#1223)
Direct download was unusable behind an external bypasser (FlareSolverr /
Byparr): every request paid a 403 plus a full solve, and a search that
never ran was reported to the user as "No books found".

Clearance was discarded on the external path. get_cf_cookies_for_domain
and get_cf_user_agent_for_domain returned {} / None whenever
USING_EXTERNAL_BYPASSER was set, and _fetch_via_bypasser read only
solution.response - dropping solution.cookies and solution.userAgent,
which FlareSolverr-compatible services do return. A solve therefore
cleared the one request that paid for it and nothing else, and a file
download - which the solver cannot proxy, being binary - presented no
clearance at all. Diagnosed from a v1.3.9 debug bundle: ~35s in the
bypasser per search, on every search.

- Move the cookie jar out of internal_bypasser into bypass/cookie_store.
  internal_bypasser imports seleniumbase at module scope, which is the
  dependency an external-bypasser deployment is entitled not to have, so
  it cannot host a store the external path depends on.
- Harvest solution.cookies and solution.userAgent after a successful
  solve. The existing filtering applies unchanged, so the per-check
  __ddg8_/__ddg9_/__ddg10_ trio is still dropped and the external path
  cannot reintroduce the ?check=1 loop fixed in ebb833a. The UA matters
  as much as the cookies: Cloudflare ties cf_clearance to the UA that
  solved the challenge.
- Read cookie fields from either shape - CDP objects or JSON mappings.
  Both use the same field names, expires included.
- Point http.py's getters and _purge_clearance at the shared store, so
  either bypasser fills and drains the same jar.
- Give the Docker helper-subprocess handoff explicit export_store /
  import_store rather than reaching into module globals.

An unsolved challenge was also indistinguishable from an empty result.
_looks_like_aa_page() counted the challenge markers as "recognisably
AA", so _fetch_search_table handed a DDoS-Guard interstitial back as a
legitimate no-table response and the user was told their query found
nothing when the search never ran. Split challenge detection out and
raise SearchUnavailableError with the reason instead. The mirror is
still not quarantined - every mirror shares the same protection, so it
is not the mirror's fault.

Verified: 2531 unit tests pass; ruff, basedpyright and vulture clean;
e2e bypasser-external profile passes (5). Its mock FlareSolverr already
returned cookies and userAgent from /v1 - the contract was there,
shelfmark was not reading it.

Refs #1220. Deliberately not "Fixes": this removes the re-solve and
makes a failed solve legible, but if Byparr genuinely cannot clear AA's
current DDoS-Guard, the reporter now gets that as an error rather than a
silent "no books found". The download path may swallow interstitials the
same way; not audited here.
2026-08-16 12:08:45 -04:00
CaliBrain ebb833a82c fix(bypass): discard rejected DDoS-Guard cookies instead of replaying them (#1221)
A cookie that has been rejected was kept and presented again on every
later
request, so a single bad clearance could re-arm the challenge
indefinitely.

Cookie storage:
- Enforce expiry for every stored cookie, not just cf_clearance.
DDoS-Guard
domains have no cf_clearance, so the existing check never fired for them
and
  expired cookies were replayed forever.
- Stop storing the per-check cookies __ddg8_/__ddg9_/__ddg10_ and
ddg_last_challenge. Captured live from Anna's Archive, these carry the
client
IP and the timestamp the check was issued (~40 min), versus ~1 year for
the
  __ddg1_/__ddg2_/__ddgid_ clearance. Replaying an IP-bound token stops
describing the caller as soon as the egress IP changes, which is routine
  behind a VPN.

Failure handling — every path that is rejected while carrying cookies
now
purges them, not just the redirect loop:
- 403 returned while presenting cookies.
- Cached-cookie attempt rejected, whether by status or by redirect loop.
- Factored the purge into _purge_clearance, guarded on a non-empty
hostname
since clear_cf_cookies("") means "every host" and would wipe clearance
for
  sites that are working fine.

Also fix the search warm-up switches shipped inert in v1.3.8:
SEARCH_WARMUP_ENABLED and SEARCH_WARMUP_QUERY are not in the settings
registry, and config.get only consults the environment for keys it
knows, so
both always returned their defaults — the warm-up could not be turned
off or
retargeted. Read os.environ first.

Refs #1220. Deliberately not "Fixes": the reported failure could not be
reproduced on v1.3.8 from a stable IP (the reporter's own queries all
returned
200 on both the pre- and post-change builds), and the new purge paths
did not
fire in live testing because the failures arrive as redirect loops,
which were
already purged. These are correctness fixes with no measured effect on
that
issue. The underlying problem remains that Chrome-obtained cookies never
satisfy DDoS-Guard when replayed by requests, so every search still
re-solves.

Verified: 2542 unit tests pass; ruff, basedpyright and vulture clean;
e2e
platform baseline (10), full (6) and bypasser-external (5) all pass;
five
sequential live searches against Anna's Archive all returned 200 with
zero
"Exceeded 30 redirects".
2026-08-15 17:08:11 -04:00
CaliBrain cc1a95f965 Fix protection bypass cancelled by stall detection at exactly 300s (#1184)
A download that hits Cloudflare hung on "Bypassing protection..." for
five
minutes and then died, regardless of which bypasser was configured.

html_get_page() started a BypassHeartbeat thread to keep the download
marked
alive during a bypass, but the thread had no loop: it fired one status
event
and returned. Even with the loop restored it could not have worked,
because
update_download_status() dedupes identical (status, message) tuples and
returns before refreshing _last_activity, and the heartbeat re-sent the
byte-identical payload already emitted just above it.

So _last_activity was frozen for the whole bypass, while both bypassers
are
allowed to run longer than STALL_TIMEOUT (external FlareSolverr ~394s at
default settings, internal 420s per get() call). The watchdog always
won.
From a reporter's log: 403 at 07:04:33.390, cancelled at 07:09:33.987 -
exactly 300.000s, and 41s before the bypasser would have finished and
reported the real error, an HTTP 500 from FlareSolverr the user never
saw.

The regression is not one commit. 1f093de (#536) added the heartbeat and
the
dedup together and refreshed activity before the dedup return, so it
worked.
ff094be (#832) moved the refresh below that return while tightening
stall
detection for #823. 3a3a3ce (#845) then deleted the heartbeat's while
loop
to silence a B023 lint, removing the last evidence of intent.

The dedup itself is correct and stays: a keep-alive that ticks on a
timer
proves nothing about whether an operation is progressing, so letting it
refresh the stall clock would make a wedged download immortal. Split the
two
concerns instead.

Add shelfmark/download/activity.py. A long single-shot operation
declares its
own upper bound once, over a sentinel status carried on the existing
status_callback channel - so no new parameter has to be threaded through
every
handler, post-processor and output module. The orchestrator intercepts
the
sentinel in its per-task closure and records an absolute deadline in
_activity_grace, which stall detection honours alongside STALL_TIMEOUT.
The
grace never extends itself and is clamped to
_MAX_ACTIVITY_GRACE_SECONDS, so
an operation that overruns its own declared budget is still cancelled.

Each bypasser now reports max_duration_seconds() derived from its own
retry
and timeout settings, and http.py asks whichever is active, plus 30s of
slack
so the bypasser's own deadline expires first and the user sees its real
failure. On that path html_get_page() also emits
status_callback("error", ...)
rather than silently returning an empty page.

Three further fixes on the same code path:

- Extract the watchdog into _find_stalled_tasks() and
_cancel_stalled_task().
It was the only place holding _progress_lock across a call into
book_queue,
whose terminal-status hooks reach a sqlite write that gevent does not
patch,
blocking the hub and every download worker. It now holds the lock for
dict
  reads only.
- Bound _CDP_WORKER.run(), which waited with timeout=None while holding
the
module-wide LOCKED, so a single wedged in-process CDP session blocked
every
  subsequent bypass forever on non-Docker installs.
- Broaden the coordinator loop's except clause back to Exception, with
  escalating backoff. 8d98e12 (#868) narrowed it to a six-type tuple to
silence BLE001, which let gevent's LoopExit and similar kill the only
thread
driving the download queue - undoing #832's fix for #823 and resurfacing
it
  as #1166. GreenletExit and gevent.Timeout still propagate.

Fixes #1001
Refs #1166, #823
2026-08-11 02:25:28 -04:00
CaliBrain 61b3eaf351 Fix internal bypasser using system DNS in subprocess, breaking downloads behind ISP DNS blocks (#1028) (#1080) 2026-06-21 00:42:03 -04:00
Alex eba04115ae Fix internal bypasser and /home directory (#919)
- Fixed internal bypasser startup with newer Chromium/SeleniumBase by
isolating the browser helper from Gunicorn/gevent, serialising helper
failures cleanly, and cleaning up orphan processes after a failure
- Stopped using /app as runtime home state, now moved to /home/shelfmark
or /tmp/shelfmark/home as fallback.
- Added tests
2026-04-26 15:01:54 +01:00
Alex e35b4c47a7 Direct source refactor (#895)
- Updated mirror selection
- Removed built-in mirror options, users must provide their own
configurations
- Set Universal search to default, added ability to disable direct
source
- Updated documentation
- Updated makefile
2026-04-15 18:50:13 +01:00
Alex d7b9f2e67f Backend test hardening + quality enforcement (#872)
- Reworked many tests
- Enforcing lint + type checking for test suite
- Fixed various issues surfaced by the new tests
- CI tweaks
2026-04-12 12:01:52 +01:00
Alex 8d98e122ec Linter followup (#868)
Expanded Ruff rules and completed fixes
2026-04-11 16:18:52 +01:00
Alex 1931eb96a5 Feature: Notification support + Enhanced request management (#618)
- Added notification support via Apprise dependency
- Notifications can be configured globally or per user, with full
customization of events and notification type.
- Added expanded ActivityCard for increased detail of each request, file
info, and managing the attached file.
- Enhanced tests
2026-02-15 17:59:53 +00:00
Alex b10458a48b Patch: Migrate bypasser to pure CDP + Misc fixes (#575)
Bypasser:
- Refactored internal bypasser logic to use SeleniumBase Pure CDP mode,
removed chromedriver dependencies and UC code.
- Added dedicated threading for internal bypasser functions, fixes any
potential asyncio CPU spike behavior
- Fixed WebGL issue with Chromium 144. Reverted 1.0.3 hotfix and updated
to latest Chromium

Misc: 
- Added M4A color mapping
- Fix frontend language filtering with multi-language releases
- Added "days" age for usenet/torrent releases
- Improved entrypoint chown efficiency
- Added `ONBOARDING` env variable, default true
2026-02-02 20:32:19 +00:00
Alex e5ccabe1ef Patch: Various additions (#564)
- Added rich Prowlarr search results for whitelisted indexers
- Added torznab query for whitelisted indexers
- Added flags for all Prowlarr indexers
- Added completed external download retry mechanism and "locating" state
- Added client side preference storage of Book/Audiobook search
preference
- Fixed reverse proxy base URL in edge cases
- Added gevent locking for I/O operations, keeps healthcheck alive on
intensive processing operations
- Added M4A supported audiobook option
- Improved file transfer counting and logging with hardlink fallback
warnings
- Fixed proxy auth header for REMOTE_USER scenario
- Dependency tweak for internal bypasser
2026-01-31 12:53:11 +00:00