page.evaluate runs eval() in the page's main world, which fails with 'call to eval() blocked by CSP' under any CSP that disallows unsafe-eval - HTTP headers (already stripped), meta tags (not strippable), or internal viewer documents (#394).
The navigation request already carries the UA the site actually saw, so take user_agent from page_request.request.headers and keep evaluate only as a best-effort fallback whose failure can no longer 500 the request.
Firefox renders application/json documents in a built-in viewer whose own
CSP (<script-src resource:>) blocks Playwright's eval-based page.evaluate,
crashing /v1 with a 500 on JSON APIs (closes#394). Setting
devtools.jsonview.enabled=false renders JSON as plain text, which also
returns the raw JSON body instead of the viewer's syntax-highlighted HTML.
fake_dep's AsyncMock page made page.locator() return an un-awaited
coroutine, so detect_cloudflare_challenge swallowed an AttributeError
and reported a challenge. The networkidle-timeout test silently ran the
solver branch and never exercised its intended path, plus emitted a
'coroutine ... was never awaited' RuntimeWarning in CI.
Make page.locator() sync-returning (as in real Playwright) with an
awaitable count() that finds no elements, and assert the solver is never
invoked.
Run trafilatura server-side on the rendered DOM (page.content()), so
JS-rendered pages stay fully visible to the extractor; fall back to
innerText when trafilatura cannot score any main content.
- Move OWUI_API_KEY into pydantic settings (src/consts.py); drop the
Dockerfile ENV entry so the key is only ever set at runtime
- Enforce auth before the browser is launched via dependency ordering
- Compare bearer tokens in constant time (hmac.compare_digest)
- Keep extracting when networkidle times out, matching /v1 behavior
- Type page as Page, drop redundant comments and docstrings
Add /load endpoint for Open WebUI's WEB_LOADER_ENGINE=external integration.
Uses document.body.innerText for content extraction.
Configure in Open WebUI:
WEB_LOADER_ENGINE=external
EXTERNAL_WEB_LOADER_URL=http://byparr:8191/load
EXTERNAL_WEB_LOADER_API_KEY=<OWUI_API_KEY env var>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
A page whose network never goes idle (background analytics, websockets)
used to fail the whole request with a 408 once the networkidle wait
expired. Since the DOM is fully usable after domcontentloaded, treat a
networkidle timeout as non-fatal and return the loaded page instead.
Fatal timeouts during initial load or challenge solving still return 408.
Adds unit coverage for both paths using a fake page that fails
configured load-state waits.
Add a maxTimeout alias to LinkRequest.max_timeout for FlareSolverr
drop-in compatibility. Values of 1000 or more are treated as
milliseconds and normalized to seconds; smaller values keep the
native seconds semantics. Closes#382.
- Catch both builtins.TimeoutError and playwright TimeoutError as 408
- Check challenge title before networkidle to avoid timeout on Cloudflare interstitial
- Add blockMedia and returnOnlyCookies request options
- Return raw PDF bytes as base64 with contentType application/pdf
- Skip tests on 408 timeouts; add PDF handling test
ThePhaseless
marchingphoenixandClaude Opus 4.5
Alvaro Santos Andres