invisible-core 49.46+ looks up the egress IP and checks GitHub for a newer
GeoIP database on every browser launch, so a network that blocks those hosts
delayed every request by the lookup timeouts. Memoize both lookups (success for
the process lifetime, failures retried after a backoff, proxied egress lookups
uncached), warm them up in the FastAPI lifespan before the port opens, and gate
get_browser() on the same warm-up. Drop the unused --init mode.
* chore(deps): lock file maintenance
* fix: fall back to the host timezone when the egress lookup raises
invisible-core 49.46 always looks up the egress IP and raises
GeoTimezoneError when an "auto" lookup fails, instead of returning an
empty zone. Catch it so a failed lookup uses the host timezone again
rather than failing the request, update the explicit-settings test to
the new always-lookup behaviour, and correct the README.
---------
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: ThePhaseless <kukubaorch@gmail.com>
The retry in read_settled_html covered page.content() racing a
navigation. #418 only showed that race on the early read taken right
after domcontentloaded, and this branch already removes that read. The
late read now matches the Cloudflare path, which reads after the idle
wait and has never retried.
When the retry fires at the end of the budget, remaining_ms() is 1 ms.
The second idle wait then times out at once, and the second read lands
in the same 60-150 ms window that failed. So the retry adds a helper and
threads timer into content.py for a case it does not cover.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fallback zone used after a failed egress lookup came from hand-rolled
TZ and /etc/localtime parsing. That parsing passed values such as
TZ=UTC0, TZ=CET-1CEST,M3.5.0,M10.5.0/3 or TZ=:/usr/share/zoneinfo/... to
the browser unchanged, and Playwright rejects them as an invalid
timezone ID. tzlocal.get_localzone_name() handles those forms and returns
only names it could load, or None. A None result still falls back to UTC.
tzlocal already came in through trafilatura (htmldate -> dateparser).
It is now a direct dependency because Byparr imports it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every request builds a new InvisiblePlaywright, and its __aenter__ calls
prepare_session_geo(): with the default timezone="" and no proxy that
discovers the egress IP through api.ipify.org, icanhazip.com and
checkip.amazonaws.com, then calls ensure_geoip_mmdb(), which HEADs GitHub
for a newer GeoIP release, three times per session. On a network where
those hosts are slow or blocked every request paid for it before the
browser even started. Measured in the app image: ipify black-holed adds
~5s per request, all three echo hosts black-holed ~12.5s, GitHub
black-holed ~75s. Setting BROWSER_LOCALE alone did not help, because
the timezone was still "auto".
Without a proxy the egress is the host's own, so the answer does not
change between requests. Resolve it on the first request, off the event
loop and under a lock so concurrent first requests share one lookup,
then hand the zone and the locale's primary tag to InvisiblePlaywright
explicitly; an explicit zone and tag make prepare_session_geo touch no
network. The resulting browser is the same as with "auto": passing the
primary tag back reproduces the decided language list for every Firefox
build the locale table can pick, and without a proxy the egress IP,
coordinates and srflx are not used by InvisiblePlaywright anyway.
When the lookup fails upstream falls back to the host timezone by
passing "", but passing "" again would repeat the lookup on every
request, so the host zone is named explicitly (TZ, else the
/etc/localtime link, else UTC) and the lookup is retried after ten
minutes.
Sessions behind a proxy (PROXY_SERVER or X-Proxy-Server) still resolve
per request through the proxy: each proxy has its own egress, and
upstream deliberately fails the launch when it cannot be resolved.
BROWSER_TIMEZONE mirrors BROWSER_LOCALE and wins also behind a proxy;
with both set the lookup is skipped entirely.
Fixes#416
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
invisible_playwright drives Firefox through a vendored copy of Playwright
(invisible_playwright._pw), and its Error/TimeoutError are not subclasses
of playwright.async_api's classes or of the builtin TimeoutError. Every
except and suppress in Byparr named the playwright.async_api classes, so
in production none of them matched:
- a Turnstile widget read that outlasted its 1s timeout escaped the
suppress() in the solve loop and turned the whole request into a 500
(the Locator.bounding_box traceback in #419), even with budget left
- an unreachable host was a 500 instead of a 502
- a goto/domcontentloaded timeout was a 500 instead of a 408
- a page that never reaches networkidle was a 500 instead of returning
the loaded page, and /load returned empty content for it
Nothing in Byparr runs the real playwright package at runtime any more;
it is only used for type hints. So the handlers now import the classes
invisible_playwright re-exports from its public async_api, and nothing
else. That module exports them since 0.15.0, hence the raised floor.
The unit-test fakes raised the playwright.async_api classes too, which
is why the suite stayed green. They now raise the vendored classes, and
a new test covers a widget read timing out mid-solve.
Refs #419
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When no Cloudflare interstitial was detected, _navigate_and_solve()
snapshotted page.content() right after domcontentloaded and
build_response_content() returned that snapshot, even though
_wait_for_networkidle() ran afterwards. A site with its own JS guard
(com-x.life: 302 to a proof-of-work page that sets cookies and calls
location.replace) finishes its redirect during that wait, so Byparr
returned the guard page while solution.url already named the target.
The snapshot came in with the localized-detection change (#388) only to
reuse HTML that had already been read; before it the body was always
read after the idle wait. Drop it, along with the challenge_detected and
page_html plumbing, and read the settled page on both branches.
The early read was also a liability of its own: a guard redirecting
within a few hundred milliseconds of domcontentloaded makes that read
fail with "Failed to find execution context", which escaped read_item
as a 500. The late read can still race a navigation that lands right at
the budget's edge, so if it fails, wait for network idle again and read
once more. The error is caught as invisible_playwright's Error, which is
what the browser actually raises.
Fixes#418. Thanks to c4uran for the analysis.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
trafilatura 2.3 (bumped in #421) separates an <h1> from the following
paragraph with a newline; 2.2 glued them together ("Example TitleThis is
..."). test_networkidle_timeout_still_extracts_content pinned the old
glued string, so the Docker test stage has failed on every main run since
#421 merged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Firefox's HTTPS-First mode upgrades an http:// navigation, sees the
challenge's 403 and falls back to http as a fresh navigation. That
aborts page.goto() with "interrupted by another navigation", so any
http:// URL to a challenged site failed before the solver could run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A host that will not resolve arrived as a plain PlaywrightError, which
nothing caught, so Byparr answered 500 and read as its own defect. It is
the upstream that failed, and the failure is usually transient, so 502
also lets clients retry where a 4xx would tell them not to. The timeout
branch stays ahead of it, since PlaywrightTimeoutError subclasses Error.
scroll_into_view_if_needed waits for actionability, and the Cloudflare
widget animates without pause, so the call timed out at every ancestor
depth and aborted the measurement before the box was ever read. The
container is already in the viewport; drop the scroll and read the box.
Three tests checked solution.status against HTTPStatus.OK, which the
endpoint now hardcodes, so the check could never fail. Drop it everywhere
except the test that exists to pin that behaviour.
That left the self-clearing case asserting only that nothing raised, which
the click test already covers. Give it a measurable widget and assert we
never press it: a challenge that clears on its own is the one case where
touching the checkbox would be wrong, and nothing tested it. Verified by
moving the click ahead of the exit check, which the assertion now catches.
Drop the fake page.evaluate return as well, since /v1 stopped calling it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
FlareSolverr hardcodes the solution status ("todo: fix, selenium not
provides this info"), so clients built against it never see anything else.
Byparr handed back the real navigation code on the branch without a
challenge and 200 on the branch with one, which is neither honest nor
compatible. Always report 200.
Move the challenge handling into src/challenge.py and the response bodies
into src/content.py, leaving endpoints.py with the routes and navigation.
That also confines the import of playwright_captcha's private detection
module to a single file, so a patch release can only break one import
instead of the app and the test module at once.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nothing waited for the destination once the interstitial let go: page_html
stayed unset and the networkidle wait only ran on the branch that never saw
a challenge, so a challenge clearing on its own returned whatever had loaded
by then. Wait for it on both branches.
An exhausted budget produced timeout=0, which Playwright reads as no timeout
at all, turning every remaining wait unbounded exactly when it should fail
fast. Floor it instead.
Scroll the widget into view before measuring it, since bounding_box reports
viewport coordinates and an off-screen widget was clicked at a point that hit
nothing, and reject containers taller than a checkbox row so a full-page
wrapper cannot pass for one - both reported success while clicking blank
space.
Drop max_attempts, which nothing reads now that the solver is gone, and
refresh AGENTS.md, which still described camoufox and a solver in the
dependency.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
nowsecure.nl carries no interstitial, so the turnstile detector only widened
the entry condition without ever being able to satisfy the exit one, leaving
a turnstile-only page reported as solved on the first poll. Detect on the
interstitial alone.
Restore the user-agent guard that went missing: an absent header made
Solution reject None and turned into an unhandled 500, where it used to
degrade to an empty string. Fall back to "" directly rather than reaching
for evaluate(), which CDP refuses.
Bound the widget measurement, which inherited Playwright's 30s default at
each of four depths and so could run far past the request budget, throttle
the probe when no click lands so the scan no longer repeats every tick, and
release the mouse button through a finally so a failed press cannot leave it
held down.
Cover the click path: the fixture pinned bounding_box to None, so nothing
exercised the code this branch exists to add.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nothing calls ClickSolver any more now that the checkbox is clicked through
the mouse, so take it out of the browser dependency instead of leaving it
constructed but unused.
Assert on the returned body as well: a 200 alone passed even when the
interstitial itself was handed back, which is the defect this branch fixes.
Judge on _cf_chl_opt, since Cloudflare keeps serving cdn-cgi/challenge-platform
as a beacon on pages that are already cleared.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measuring the widget with page.evaluate() made Cloudflare reissue the
challenge every few seconds, so the interstitial never cleared. Locate the
container with locators instead and click it through the mouse, which leaves
its closed shadow root untouched.
Keep the solver lazy as well: ClickSolver.prepare() patches attachShadow,
which is what escalated a self-clearing challenge into a checkbox in the
first place.
A click can land while the widget is still self-verifying, so retry on a
cooldown for as long as the request budget lasts rather than stopping after
the first one, and confirm the marker is gone twice before reporting success
since it drops out between challenge rounds.
The bypass tests now pass on their own, so drop the xfail marks.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
detect_cloudflare_challenge(page, 'interstitial') matches the challenge on
the page and stops matching once it clears, measured on ext.to (True then
False), and on yggtorrent, nowsecure.nl and google.com, none of which carry
a /cdn-cgi/challenge-platform/ script when cleared. So the custom marker set
was unnecessary.
The turnstile variant is not usable for this: nowsecure.nl embeds turnstile
scripts on its normal page, so it reports a challenge even when cleared.
Also adds PlaywrightTimeoutError to the retryable set. It is a different
class from the builtin TimeoutError -- playwright's derives from its own
Error -- so a Playwright timeout inside the solver escaped the loop and
returned 408 without retrying.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TDMac4vGGcBhoUB5V6bvFK
solve_captcha decides it failed by waiting for networkidle, which returns
while Cloudflare is still verifying, so it reported failure on challenges
that had already passed and retried until the whole max_timeout burned into
a 408. Cap each solve attempt and let the challenge markup decide instead.
CHALLENGE_MARKERS has to match the orchestrator specifically: the bare
/cdn-cgi/challenge-platform/ path also matches the jsd beacon Cloudflare
serves on cleared pages, so detect_cloudflare_challenge never reports
success on its own.
Also switches the solver back to FrameworkType.PLAYWRIGHT, since PATCHRIGHT
injects the shadow-root unlock over CDP and Firefox has no CDP session, and
turns off COOP/COEP so the widget's iframe appears in page.frames at all.
The four sites Cloudflare refuses on this browser are xfail rather than
skip, so a regression still shows and a pass records as xpass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TDMac4vGGcBhoUB5V6bvFK
Removes every explanatory comment added by this branch, inlines the browser
prefs rather than holding them in a module constant, folds _cloudflare_frame
into its only caller, and cuts the added docstrings to one line each.
No behaviour change: 13 unit tests pass, and removing the checked-box guard
still fails its test.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TDMac4vGGcBhoUB5V6bvFK
Cuts ~100 lines of commentary that restated the diff or recorded dead
investigation, and merges _press_point back into _press_checkbox now that
the checked guard is one condition rather than the extra return that
tripped the too-many-returns lint.
Corrects the COOP/COEP note, which claimed the pair changed no outcome.
Without those prefs the widget's iframe never appears in page.frames at
all: measured on ext.to, eight presses land with them and none without.
No behaviour change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TDMac4vGGcBhoUB5V6bvFK
The reason said Cloudflare refuses the checkbox click from datacenter IPs
and that the visitor's address was being judged rather than our code. That
is false. Measured from one datacenter IP within a single hour, byparr
v2.1.0 clears ext.to in 18s, speed.cd/login in 20s and extratorrent.st in
19s, each returning cf_clearance, while no configuration of the current
stack clears any of them.
Two candidate causes were measured and eliminated rather than assumed. The
JS-visible fingerprint is not it: camoufox is the less coherent of the two
browsers -- no WebGL at all, oscpu leaking Linux beneath a Windows UA -- and
passes regardless. The TLS handshake is not it either: re-enabling cipher
0xC009 reproduces camoufox's JA4 byte for byte
(t13d1717h2_5b57614c22b0_3cbfd9057e0d) and the challenge is still refused.
Also records that devtools.jsonview.enabled is load-bearing for #394, and
that the COOP/COEP pair changed no outcome on any site or network measured.
No behaviour change; comments and the xfail reason only.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TDMac4vGGcBhoUB5V6bvFK
playwright-captcha's ClickSolver clicks the challenge's input element
directly. That input sits under a styled overlay, so Playwright reports a
successful click while `checked` never flips -- which is why the
interactive challenge has never been solved here. The solver also judged
its own click by waiting for networkidle, which returned 9ms later while
Cloudflare was still verifying, so it reported failure on challenges that
were about to pass.
Replace it with a poll loop that watches for the challenge markup to go
away and presses the widget's visible pixels whenever an unchecked box is
on offer. A box that is already checked is left alone: pressing over the
top of Cloudflare's verification restarts it, and ext.to and speed.cd sat
on "performing security verification" for a full 300s budget while being
pressed a dozen times.
Measured on a residential connection, driving the real /v1 handler:
nowsecure.nl passes in 3s, extratorrent.st in 116s and 1337x.to in 198s,
all three returning cf_clearance. extratorrent.st had never cleared
before, on any network or solver. ext.to and speed.cd still refuse -- the
press registers and the widget re-serves a fresh unchecked box -- so they
stay in the xfail list.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TDMac4vGGcBhoUB5V6bvFK
Follow-up to the earlier CI fix, after A/B-ing every change against main and
against this branch's original commit.
What measurably changed, and what did not:
- The solver's retry loop was unbounded (max_attempts = sys.maxsize). On a
challenge it cannot clear it retried ~1300 times per request and the caller
waited out the entire max_timeout for a 408 it was always going to get.
_solve_challenge now clicks, waits for the challenge markup to actually
disappear, and gives up when the budget does.
- That wait exists because the solver's own verdict is worthless here: it
judges its click with wait_for_load_state("networkidle"), which returned 9ms
after the click while Cloudflare was still showing "verifying you are
human", and then reported failure.
- The "is it still up?" check cannot use detect_cloudflare_challenge alone.
That matches any script under /cdn-cgi/challenge-platform/, and Cloudflare
serves its jsd bot-scoring beacon from the same path on cleared pages. Nor
can it use the widget iframe: a cleared nowsecure.nl carries two of those
with no challenge present. CHALLENGE_MARKERS matches the challenge
orchestrator script and the interstitial's own markup.
- test_tls_handshake_looks_like_firefox pins what this branch is actually for.
Measured through /v1 on the same host: main offers 52 cipher suites, this
branch 16, and real Firefox offers 16. route.fetch() was re-issuing
navigations through Playwright's HTTP client, and that is a fingerprint no
header spoofing hides. Unlike a Cloudflare verdict the count is
deterministic, so it is the one assertion here that cannot flake.
- Disabling COOP/COEP does let the solver reach and click the checkbox for the
first time (Cloudflare advances to "verifying you are human"), but it changed
no outcome across eight sites, and real Firefox ships those policies on.
Recorded in a comment rather than shipped.
test_bypass keeps a hard assertion against targets that clear from any network.
The four Cloudflare guards hardest move to xfail rather than skip: they still
run and still report, but Cloudflare's opinion of the runner's IP cannot turn
the build red.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The bypass tests were failing on CI with 408s after 78 minutes. Neither the
runner's speed nor this branch's TLS change was responsible.
On the sites that fail, Cloudflare serves its interactive checkbox challenge.
playwright-captcha locates the widget iframe inside the shadow root and then
calls ElementHandle.content_frame(), which this Firefox build refuses:
Protocol error (Page.describeNode): Permission denied to access property
"docShell" on cross-origin object
Its fallback -- matching page.frames by URL -- cannot help either, because the
challenge frame exposes an empty URL to the parent. Every attempt therefore
ends in CaptchaDetectionError: Cloudflare iframes not found.
MAX_ATTEMPTS was sys.maxsize, so that repeated until the request budget ran
out: 432 docShell errors and 1326 retry iterations in a single request on the
runner, and with max_timeout raised to 360 and --retries 3, a 1h18m job.
Three changes:
- max_attempts defaults to 5. An unreachable widget stays unreachable, so the
retries were not buying anything; the caller now hears about it in seconds.
- _solve_challenge translates the solver's own give-up exceptions into the 408
read_item already reports for timeouts. Without this, bounding max_attempts
would have turned the hang into an unhandled 500.
- The solver framework goes back to PLAYWRIGHT. PATCHRIGHT skips the
unlockShadowRoot init script and injects over CDP instead, which Firefox has
no session for ("CDP session is only available in Chromium"). Cloudflare
builds its widget in a closed shadow root, so on this branch the challenge
iframe was invisible even to page.locator: 1 -> 0 against the same sites on
the same runner.
test_bypass drops the max_timeout=360 override and skips again on 408.
Whether Cloudflare shows the interactive challenge depends on the visitor, so
the runner's luck should not decide whether a regression of ours is reported.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
page.evaluate runs eval() in the page's main world, which fails with 'call to eval() blocked by CSP' under any CSP that disallows unsafe-eval - HTTP headers (already stripped), meta tags (not strippable), or internal viewer documents (#394).
The navigation request already carries the UA the site actually saw, so take user_agent from page_request.request.headers and keep evaluate only as a best-effort fallback whose failure can no longer 500 the request.
Firefox renders application/json documents in a built-in viewer whose own
CSP (<script-src resource:>) blocks Playwright's eval-based page.evaluate,
crashing /v1 with a 500 on JSON APIs (closes#394). Setting
devtools.jsonview.enabled=false renders JSON as plain text, which also
returns the raw JSON body instead of the viewer's syntax-highlighted HTML.
fake_dep's AsyncMock page made page.locator() return an un-awaited
coroutine, so detect_cloudflare_challenge swallowed an AttributeError
and reported a challenge. The networkidle-timeout test silently ran the
solver branch and never exercised its intended path, plus emitted a
'coroutine ... was never awaited' RuntimeWarning in CI.
Make page.locator() sync-returning (as in real Playwright) with an
awaitable count() that finds no elements, and assert the solver is never
invoked.
Run trafilatura server-side on the rendered DOM (page.content()), so
JS-rendered pages stay fully visible to the extractor; fall back to
innerText when trafilatura cannot score any main content.
- Move OWUI_API_KEY into pydantic settings (src/consts.py); drop the
Dockerfile ENV entry so the key is only ever set at runtime
- Enforce auth before the browser is launched via dependency ordering
- Compare bearer tokens in constant time (hmac.compare_digest)
- Keep extracting when networkidle times out, matching /v1 behavior
- Type page as Page, drop redundant comments and docstrings
Add /load endpoint for Open WebUI's WEB_LOADER_ENGINE=external integration.
Uses document.body.innerText for content extraction.
Configure in Open WebUI:
WEB_LOADER_ENGINE=external
EXTERNAL_WEB_LOADER_URL=http://byparr:8191/load
EXTERNAL_WEB_LOADER_API_KEY=<OWUI_API_KEY env var>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
A page whose network never goes idle (background analytics, websockets)
used to fail the whole request with a 408 once the networkidle wait
expired. Since the DOM is fully usable after domcontentloaded, treat a
networkidle timeout as non-fatal and return the loaded page instead.
Fatal timeouts during initial load or challenge solving still return 408.
Adds unit coverage for both paths using a fake page that fails
configured load-state waits.
Add a maxTimeout alias to LinkRequest.max_timeout for FlareSolverr
drop-in compatibility. Values of 1000 or more are treated as
milliseconds and normalized to seconds; smaller values keep the
native seconds semantics. Closes#382.
- Catch both builtins.TimeoutError and playwright TimeoutError as 408
- Check challenge title before networkidle to avoid timeout on Cloudflare interstitial
- Add blockMedia and returnOnlyCookies request options
- Return raw PDF bytes as base64 with contentType application/pdf
- Skip tests on 408 timeouts; add PDF handling test
ThePhaseless
renovate[bot]andThePhaseless
Alex Thomson
marchingphoenixandClaude Opus 4.5