/cache holds the browser baked into the image. A tmpfs over it hides
that browser, so Byparr downloads it again on every start (the cause
behind #439). Also move shm_size and tmpfs in compose.yaml out of the
build section, where they never applied to the running container.
Without -s, tini warns and cannot adopt orphaned browser processes when it is
not PID 1, as in a pod with shareProcessNamespace. As PID 1 the flag changes
nothing.
* fix(deps): pin dependencies
* fix: stop passing log_level to FastAPI
FastAPI never had a log_level parameter; it went into **extra and was
ignored until 0.162 removed **extra, where it raises TypeError at import.
uvicorn.run already receives the log level.
---------
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: ThePhaseless <kukubaorch@gmail.com>
invisible-core 49.46+ looks up the egress IP and checks GitHub for a newer
GeoIP database on every browser launch, so a network that blocks those hosts
delayed every request by the lookup timeouts. Memoize both lookups (success for
the process lifetime, failures retried after a backoff, proxied egress lookups
uncached), warm them up in the FastAPI lifespan before the port opens, and gate
get_browser() on the same warm-up. Drop the unused --init mode.
* chore(deps): lock file maintenance
* fix: fall back to the host timezone when the egress lookup raises
invisible-core 49.46 always looks up the egress IP and raises
GeoTimezoneError when an "auto" lookup fails, instead of returning an
empty zone. Catch it so a failed lookup uses the host timezone again
rather than failing the request, update the explicit-settings test to
the new always-lookup behaviour, and correct the README.
---------
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: ThePhaseless <kukubaorch@gmail.com>
The retry in read_settled_html covered page.content() racing a
navigation. #418 only showed that race on the early read taken right
after domcontentloaded, and this branch already removes that read. The
late read now matches the Cloudflare path, which reads after the idle
wait and has never retried.
When the retry fires at the end of the budget, remaining_ms() is 1 ms.
The second idle wait then times out at once, and the second read lands
in the same 60-150 ms window that failed. So the retry adds a helper and
threads timer into content.py for a case it does not cover.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fallback zone used after a failed egress lookup came from hand-rolled
TZ and /etc/localtime parsing. That parsing passed values such as
TZ=UTC0, TZ=CET-1CEST,M3.5.0,M10.5.0/3 or TZ=:/usr/share/zoneinfo/... to
the browser unchanged, and Playwright rejects them as an invalid
timezone ID. tzlocal.get_localzone_name() handles those forms and returns
only names it could load, or None. A None result still falls back to UTC.
tzlocal already came in through trafilatura (htmldate -> dateparser).
It is now a direct dependency because Byparr imports it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every request builds a new InvisiblePlaywright, and its __aenter__ calls
prepare_session_geo(): with the default timezone="" and no proxy that
discovers the egress IP through api.ipify.org, icanhazip.com and
checkip.amazonaws.com, then calls ensure_geoip_mmdb(), which HEADs GitHub
for a newer GeoIP release, three times per session. On a network where
those hosts are slow or blocked every request paid for it before the
browser even started. Measured in the app image: ipify black-holed adds
~5s per request, all three echo hosts black-holed ~12.5s, GitHub
black-holed ~75s. Setting BROWSER_LOCALE alone did not help, because
the timezone was still "auto".
Without a proxy the egress is the host's own, so the answer does not
change between requests. Resolve it on the first request, off the event
loop and under a lock so concurrent first requests share one lookup,
then hand the zone and the locale's primary tag to InvisiblePlaywright
explicitly; an explicit zone and tag make prepare_session_geo touch no
network. The resulting browser is the same as with "auto": passing the
primary tag back reproduces the decided language list for every Firefox
build the locale table can pick, and without a proxy the egress IP,
coordinates and srflx are not used by InvisiblePlaywright anyway.
When the lookup fails upstream falls back to the host timezone by
passing "", but passing "" again would repeat the lookup on every
request, so the host zone is named explicitly (TZ, else the
/etc/localtime link, else UTC) and the lookup is retried after ten
minutes.
Sessions behind a proxy (PROXY_SERVER or X-Proxy-Server) still resolve
per request through the proxy: each proxy has its own egress, and
upstream deliberately fails the launch when it cannot be resolved.
BROWSER_TIMEZONE mirrors BROWSER_LOCALE and wins also behind a proxy;
with both set the lookup is skipped entirely.
Fixes#416
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
invisible_playwright drives Firefox through a vendored copy of Playwright
(invisible_playwright._pw), and its Error/TimeoutError are not subclasses
of playwright.async_api's classes or of the builtin TimeoutError. Every
except and suppress in Byparr named the playwright.async_api classes, so
in production none of them matched:
- a Turnstile widget read that outlasted its 1s timeout escaped the
suppress() in the solve loop and turned the whole request into a 500
(the Locator.bounding_box traceback in #419), even with budget left
- an unreachable host was a 500 instead of a 502
- a goto/domcontentloaded timeout was a 500 instead of a 408
- a page that never reaches networkidle was a 500 instead of returning
the loaded page, and /load returned empty content for it
Nothing in Byparr runs the real playwright package at runtime any more;
it is only used for type hints. So the handlers now import the classes
invisible_playwright re-exports from its public async_api, and nothing
else. That module exports them since 0.15.0, hence the raised floor.
The unit-test fakes raised the playwright.async_api classes too, which
is why the suite stayed green. They now raise the vendored classes, and
a new test covers a widget read timing out mid-solve.
Refs #419
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When no Cloudflare interstitial was detected, _navigate_and_solve()
snapshotted page.content() right after domcontentloaded and
build_response_content() returned that snapshot, even though
_wait_for_networkidle() ran afterwards. A site with its own JS guard
(com-x.life: 302 to a proof-of-work page that sets cookies and calls
location.replace) finishes its redirect during that wait, so Byparr
returned the guard page while solution.url already named the target.
The snapshot came in with the localized-detection change (#388) only to
reuse HTML that had already been read; before it the body was always
read after the idle wait. Drop it, along with the challenge_detected and
page_html plumbing, and read the settled page on both branches.
The early read was also a liability of its own: a guard redirecting
within a few hundred milliseconds of domcontentloaded makes that read
fail with "Failed to find execution context", which escaped read_item
as a 500. The late read can still race a navigation that lands right at
the budget's edge, so if it fails, wait for network idle again and read
once more. The error is caught as invisible_playwright's Error, which is
what the browser actually raises.
Fixes#418. Thanks to c4uran for the analysis.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
trafilatura 2.3 (bumped in #421) separates an <h1> from the following
paragraph with a newline; 2.2 glued them together ("Example TitleThis is
..."). test_networkidle_timeout_still_extracts_content pinned the old
glued string, so the Docker test stage has failed on every main run since
#421 merged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Firefox's HTTPS-First mode upgrades an http:// navigation, sees the
challenge's 403 and falls back to http as a fresh navigation. That
aborts page.goto() with "interrupted by another navigation", so any
http:// URL to a challenged site failed before the solver could run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
logger is currently hooking into uvicorn.error https://github.com/ThePhaseless/Byparr/blob/main/src/utils.py#L27 but uvicorn doesn't know that and in run() sets the log level back to INFO.
Repro: none of Byparr's (invisible_playwright is a different logger so it does work) logger.debug calls ever actually show, even when LOG_LEVEL is set to debug
Separately, logger should probably be Dispatcharr's own instance instead of using uvicorn.error.
context.cookies() with no argument returns every cookie in the browser
context, so a page that loads adverts hands the client cookies for dozens
of unrelated domains. One malformed third-party value is enough to make a
client discard the whole jar, cf_clearance with it, and the solve looks
like it failed.
Scope the call to the requested URL, plus the final page URL when a
redirect moved it. That matches what Selenium gave FlareSolverr, which is
the shape its clients are written against.
renovate[bot]
ThePhaseless
zkhcohen
DatGuy1
Nicholas Velten