Commit Graph
2 Commits
Author SHA1 Message Date
CaliBrain 5247ec6124 fix(bypass): close the gaps a helper that outlives its request opened (#1244)
assumptions the code around it still made were written for a helper that
was killed after every request.

A bypass that hits the child's deadline is cancelled from the calling
thread, which returns the moment the cancellation is scheduled - so the
helper went on to serve the next request while the abandoned one was
still closing its browser, on the same loop, sharing the DISPLAY globals
and one process group. The deadline now lives inside the loop, where
asyncio.wait_for() waits for the unwind before it raises, with the
calling thread keeping a bounded backstop in case the cleanup wedges
too. Both budgets are set so the child still answers before the parent
gives up on it.

The helper's cookie store survived the request as well, and the whole of
it is exported back to the parent on every answer - so clearance the
parent had purged for one host came back the next time some other host
was solved, the dead-cookie resurrection _redirect_loop_handoff purges
to avoid. The child starts each request from an empty store again; the
parent already runs the cached-cookie check against a superset of it.

DNS config is compared against what the helper is actually resolving
through rather than skipped whenever the parent reports "auto", so a
user flipping CUSTOM_DNS back to auto - which applies live - reaches a
warm helper instead of leaving it on an abandoned DoH resolver.

The 15s exit grace is now asked only of a helper that can still read its
stdin. One dropped mid-bypass never returns to that read, so the grace
could only end in the kill - while a user cancelling a download, and
every bypass queued behind them on LOCKED, waited it out.

Result files are cleaned on the timeout and cancellation paths too,
staging file included, rather than only when the answer was read.
2026-08-20 19:29:33 -04:00
helgehelge123andhelgehelge123 7b9c416df8 perf(bypass): keep the helper subprocess alive between bypasses (#1222)
Every protected request spawns a fresh helper subprocess, paying
interpreter start and imports before any work begins. Measured inside
the container, five consecutive runs of `python -c "import
shelfmark.bypass.internal_bypasser"`:

```
3.53s  3.45s  3.55s  3.54s  3.46s
```

A single search issues several protected requests, so that is paid
several times over per search.

## What changed

The helper now serves one JSON request per line of stdin until the
parent closes the pipe, and an idle timer
(`BYPASS_BROWSER_IDLE_TIMEOUT`, default 180s) shuts it down once
searching stops.

Answers still travel by result file, but the file is now written to a
`.part` path and renamed into place — the parent treats the file's
existence as the answer, so it must never observe a half-written one.
stdout and stderr stay attached to the parent's, so helper logs keep
appearing in `docker logs` exactly as before.

Failure handling, since a warm helper is exposed to more of it than a
per-request one ever was:

| Situation | Handling |
| --- | --- |
| Helper died between requests | Detected via `poll()`, respawned |
| Pipe broken at write time (`poll()` can miss this) | One retry on a
fresh process; a fresh one failing there is a real failure |
| Helper exits without writing a result | `RuntimeError` naming the exit
code |
| Wedged past the timeout, or cancelled mid-bypass | Helper killed, then
`_cleanup_orphan_processes` because a killed helper never got to close
Chrome |
| Idle reaper racing an arriving request | Re-checks the deadline under
the lock and re-arms instead of killing a helper that just did work |

The DNS config now travels with every request rather than only at spawn:
a warm helper outlives changes the parent makes to its provider.

## `BYPASS_REUSE_BROWSER`, off by default

This parks the CDP driver between bypasses. A driver's websockets are
bound to the loop that opened them and cannot outlive their process, so
the persistent helper is what makes this possible at all — and the warm
path runs on `_CDP_WORKER`'s long-lived loop rather than `asyncio.run`
for the same reason.

The mechanism works. With it on, the browser start disappears from the
second request onward: 0.7s from `Reusing warm Chrome browser` to the
first bypass attempt, against roughly 16s cold.

**It still ships off, because a matched-pair test shows it is a net loss
against DDoS-Guard.** Each round primed with one cold bypass, waited
10s, then measured a second — identical timing in both arms, only the
browser strategy differing, order balanced (fresh, warm, warm, fresh) so
drift over the session cannot masquerade as an effect:

| Arm | Measured request |
| --- | --- |
| fresh browser | 42.8s, 40.6s |
| warm browser | 57.1s, 59.6s |

Spread within each arm is 2.2s and 2.5s, against 16.7s between them.
Reuse removes the ~15s browser start and then gives back roughly twice
that in solving: a returning browser draws a harder challenge. Where the
cold browser is through on the second bypass method, the warm one fails
the first three and only `_bypass_method_humanlike` gets it, at ~30s for
that method alone.

Worth separating from a second effect I ran into while measuring: five
back-to-back searches slow from ~32s to 51–98s with reuse **disabled**
as well, so DDoS-Guard escalates on request rate independently of any of
this. That is why the pairs above are timed identically rather than
simply run in sequence. It is the larger of the two effects, but not
something this project can patch around.

Reuse is left available rather than dropped because Cloudflare sites may
not respond the same way, and because the two concerns are independent:
the helper start is pure overhead and always worth removing, the browser
is not.

## Verification

- 2559 unit tests pass (2542 before, 17 added in
`tests/bypass/test_warm_browser.py`)
- `ruff check`, `ruff format`, `basedpyright` over backend and tests,
and `vulture` all clean
- `docs/environment-variables.md` regenerated via
`scripts/generate_env_docs.py`
- Live against Anna's Archive on a warm helper: searches return their
usual ~760KB and 667 results, the app's own search warm-up completes
with 50 results, and the container is left with no orphan
chrome/Xvfb/ffmpeg processes

Happy to drop the `BYPASS_REUSE_BROWSER` half entirely if you would
rather not carry a default-off path — the helper persistence stands on
its own.

Co-authored-by: helgehelge123 <helge.neumann@zollsoft.de>
2026-08-20 19:00:06 -04:00