page.goto() returns None only for about:blank or a same-URL-different-hash
navigation, so page_request is always present for a real request and its
headers always carry the UA. The evaluate call was therefore unreachable
as a fallback and, once moved before navigation, silently became the
primary source instead.
Request headers are also the correct source: consumers replay them with
the clearance cookies, so the UA the server saw is the one to report.
This restores the ordering d3a828e established.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
page.evaluate runs eval() in the page's main world, which fails with 'call to eval() blocked by CSP' under any CSP that disallows unsafe-eval - HTTP headers (already stripped), meta tags (not strippable), or internal viewer documents (#394).
The navigation request already carries the UA the site actually saw, so take user_agent from page_request.request.headers and keep evaluate only as a best-effort fallback whose failure can no longer 500 the request.
Firefox renders application/json documents in a built-in viewer whose own
CSP (<script-src resource:>) blocks Playwright's eval-based page.evaluate,
crashing /v1 with a 500 on JSON APIs (closes#394). Setting
devtools.jsonview.enabled=false renders JSON as plain text, which also
returns the raw JSON body instead of the viewer's syntax-highlighted HTML.
Resolved conflicts in src/consts.py and src/endpoints.py:
- consts.py: take theirs (CHALLENGE_TITLES removed, browser_locale added,
CaptchaType import no longer needed — detection is now library-based)
- endpoints.py: merge both refactors — keep theirs' detect_cloudflare_challenge
+ page_html capture, reapply my helper extraction (setup_routes,
_navigate_and_solve, _solve_challenge, _wait_for_networkidle,
build_response_content, _fetch_pdf_content) on top
fake_dep's AsyncMock page made page.locator() return an un-awaited
coroutine, so detect_cloudflare_challenge swallowed an AttributeError
and reported a challenge. The networkidle-timeout test silently ran the
solver branch and never exercised its intended path, plus emitted a
'coroutine ... was never awaited' RuntimeWarning in CI.
Make page.locator() sync-returning (as in real Playwright) with an
awaitable count() that finds no elements, and assert the solver is never
invoked.
HOME=/tmp put the uv-managed Python at /tmp/.local/share/uv, so a
tmpfs mount on /tmp (e.g. compose tmpfs: /tmp) wiped the interpreter at
container start, leaving the /app/.venv/bin/python symlink dangling and
startup failing with 'exec /app/.venv/bin/python failed: No such file
or directory' (#389).
Move HOME to /home/byparr and apply the OpenShift permission pattern
(owner uid 1000, group 0, group=user) so both the default user and
arbitrary-UID runtimes (docker run --user, OpenShift) can write to it.
Apply the same pattern to /cache, where invisible_playwright keeps
runtime browser/profile data and which arbitrary UIDs previously could
not write.
Fixes#389
Cloudflare localizes its interstitial page title per visitor language
(e.g. Polish "Cierpliwości..." served by 1337x.to), so the hard-coded
["Just a moment..."] title check missed every non-English visitor:
Byparr returned the raw challenge page (HTTP 403, no cf_clearance
cookie, no "Challenge detected" log) and Prowlarr reported "Unable to
access 1337x.to, blocked by CloudFlare Protection." (issue #385, still
open on 3.0.1 after the compression fix).
Replace the title-based gate with the playwright-captcha library's own
language-independent DOM detection (detect_cloudflare_challenge), which
matches Cloudflare's challenge scripts directly:
- interstitial: script[src*="/cdn-cgi/challenge-platform/"]
- turnstile: input[name="cf-turnstile-response"],
script[src*="challenges.cloudflare.com/turnstile/v0"]
Both selectors match the live 1337x "Cierpliwości..." interstitial.
The navigation/detect/solve flow lives in _navigate_and_solve(); the
timeout-to-408 translation is inlined at the call site in read_item.
The now-unused title map is removed from src/consts.py.
Verified live (built image): "Challenge detected" now fires on 1337x
(0 -> 1 in logs) where the title check never fired; example.com negative
control returns 200 with no challenge path entered. End-to-end clearing
still depends on the requester's public IP (README caveat).
Root cause of remaining cache misses: the base stage declared
ARG VERSION, and the build job passed VERSION=${{ github.sha }}.
Since VERSION changes every commit, every base/app layer cache key
changed with it — so layers rebuilt every run regardless of scope.
Additionally the test job passed no build-args while the build job
passed GITHUB_BUILD=true + VERSION, so test's cached base/app layers
had different keys from build's — cross-job reuse never hit either.
Fix:
- Dockerfile: move ARG VERSION / ENV VERSION from base to the final
runtime stage (FROM app). VERSION is only read at runtime by
src.consts via Pydantic settings; base/app layers don't use it.
base/app now cache without per-commit VERSION variation.
- workflow: pass --build-arg GITHUB_BUILD=true in the test step so
test and build share identical base/app cache keys (cross-job reuse).
VERSION is intentionally NOT passed to the test job: the test stage
(FROM app AS test) doesn't read VERSION, and omitting it keeps the
base/app cache keys identical between test and build.
push: branches: ["*"] matched feature branches, so every push to a
branch with an open PR fired both a 'push' and a 'pull_request' event.
Their concurrency groups differ (refs/heads/<branch> vs refs/pull/<n>/merge),
so cancel-in-progress could not dedup them — the full multi-arch build
ran twice on each push, doubling CI minutes.
Scope push to branches: ["main"]; pull_request remains the validator for
feature branches. Tag pushes (v*.*.*), schedule, and workflow_dispatch
are under separate filters and are unaffected.
- test job: scope x64 -> amd64 to match build matrix amd64 leg
- build job: scope ${{ matrix.platform }} -> ${{ steps.vars.outputs.SURFIX }}
(yields amd64/arm64), avoiding the gha backend's / path-separator
bug that mangled scope=linux/arm64 and broke arm64 cache reuse
test (amd64) and build-amd64 now share scope=amd64 so build reuses
the app/base layers the test job cached earlier in the same run.
build-arm64 gets a working scope=arm64 that persists across runs.
metadata-action emits tags newline-separated, so FIRST_TAG=${TAGS%% *}
kept the entire multi-line value and expanded to 4 args on tag releases,
making `imagetools inspect` fail before the manifest could be signed.
Split on the first line instead.
route.fulfill(response=...) re-serves the raw bytes fetched by
route.fetch(), so compressed (gzip/brotli/zstd) documents arrive
at the browser still compressed while the forwarded headers claim
otherwise - page.content() then returns garbled binary, breaking
indexers like uindex.org and 1337x.to (issue #385).
Fetch with accept-encoding: identity so the re-served body is plain
text, and drop content-encoding/content-length alongside the CSP
headers since they are stale after the rewrite.
Run trafilatura server-side on the rendered DOM (page.content()), so
JS-rendered pages stay fully visible to the extractor; fall back to
innerText when trafilatura cannot score any main content.
- Move OWUI_API_KEY into pydantic settings (src/consts.py); drop the
Dockerfile ENV entry so the key is only ever set at runtime
- Enforce auth before the browser is launched via dependency ordering
- Compare bearer tokens in constant time (hmac.compare_digest)
- Keep extracting when networkidle times out, matching /v1 behavior
- Type page as Page, drop redundant comments and docstrings
Add /load endpoint for Open WebUI's WEB_LOADER_ENGINE=external integration.
Uses document.body.innerText for content extraction.
Configure in Open WebUI:
WEB_LOADER_ENGINE=external
EXTERNAL_WEB_LOADER_URL=http://byparr:8191/load
EXTERNAL_WEB_LOADER_API_KEY=<OWUI_API_KEY env var>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
ThePhaseless
Alex Thomson
renovate[bot]
marchingphoenixandClaude Opus 4.5