invisible_playwright drives Firefox through a vendored copy of Playwright
(invisible_playwright._pw), and its Error/TimeoutError are not subclasses
of playwright.async_api's classes or of the builtin TimeoutError. Every
except and suppress in Byparr named the playwright.async_api classes, so
in production none of them matched:
- a Turnstile widget read that outlasted its 1s timeout escaped the
suppress() in the solve loop and turned the whole request into a 500
(the Locator.bounding_box traceback in #419), even with budget left
- an unreachable host was a 500 instead of a 502
- a goto/domcontentloaded timeout was a 500 instead of a 408
- a page that never reaches networkidle was a 500 instead of returning
the loaded page, and /load returned empty content for it
Nothing in Byparr runs the real playwright package at runtime any more;
it is only used for type hints. So the handlers now import the classes
invisible_playwright re-exports from its public async_api, and nothing
else. That module exports them since 0.15.0, hence the raised floor.
The unit-test fakes raised the playwright.async_api classes too, which
is why the suite stayed green. They now raise the vendored classes, and
a new test covers a widget read timing out mid-solve.
Refs #419
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
trafilatura 2.3 (bumped in #421) separates an <h1> from the following
paragraph with a newline; 2.2 glued them together ("Example TitleThis is
..."). test_networkidle_timeout_still_extracts_content pinned the old
glued string, so the Docker test stage has failed on every main run since
#421 merged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing calls ClickSolver any more now that the checkbox is clicked through
the mouse, so take it out of the browser dependency instead of leaving it
constructed but unused.
Assert on the returned body as well: a 200 alone passed even when the
interstitial itself was handed back, which is the defect this branch fixes.
Judge on _cf_chl_opt, since Cloudflare keeps serving cdn-cgi/challenge-platform
as a beacon on pages that are already cleared.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Run trafilatura server-side on the rendered DOM (page.content()), so
JS-rendered pages stay fully visible to the extractor; fall back to
innerText when trafilatura cannot score any main content.
- Move OWUI_API_KEY into pydantic settings (src/consts.py); drop the
Dockerfile ENV entry so the key is only ever set at runtime
- Enforce auth before the browser is launched via dependency ordering
- Compare bearer tokens in constant time (hmac.compare_digest)
- Keep extracting when networkidle times out, matching /v1 behavior
- Type page as Page, drop redundant comments and docstrings
Add /load endpoint for Open WebUI's WEB_LOADER_ENGINE=external integration.
Uses document.body.innerText for content extraction.
Configure in Open WebUI:
WEB_LOADER_ENGINE=external
EXTERNAL_WEB_LOADER_URL=http://byparr:8191/load
EXTERNAL_WEB_LOADER_API_KEY=<OWUI_API_KEY env var>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
ThePhaselessandClaude Opus 5.5
Alex Thomson
marchingphoenixandClaude Opus 4.5