Resolved conflicts in src/consts.py and src/endpoints.py:
- consts.py: take theirs (CHALLENGE_TITLES removed, browser_locale added,
CaptchaType import no longer needed — detection is now library-based)
- endpoints.py: merge both refactors — keep theirs' detect_cloudflare_challenge
+ page_html capture, reapply my helper extraction (setup_routes,
_navigate_and_solve, _solve_challenge, _wait_for_networkidle,
build_response_content, _fetch_pdf_content) on top
Cloudflare localizes its interstitial page title per visitor language
(e.g. Polish "Cierpliwości..." served by 1337x.to), so the hard-coded
["Just a moment..."] title check missed every non-English visitor:
Byparr returned the raw challenge page (HTTP 403, no cf_clearance
cookie, no "Challenge detected" log) and Prowlarr reported "Unable to
access 1337x.to, blocked by CloudFlare Protection." (issue #385, still
open on 3.0.1 after the compression fix).
Replace the title-based gate with the playwright-captcha library's own
language-independent DOM detection (detect_cloudflare_challenge), which
matches Cloudflare's challenge scripts directly:
- interstitial: script[src*="/cdn-cgi/challenge-platform/"]
- turnstile: input[name="cf-turnstile-response"],
script[src*="challenges.cloudflare.com/turnstile/v0"]
Both selectors match the live 1337x "Cierpliwości..." interstitial.
The navigation/detect/solve flow lives in _navigate_and_solve(); the
timeout-to-408 translation is inlined at the call site in read_item.
The now-unused title map is removed from src/consts.py.
Verified live (built image): "Challenge detected" now fires on 1337x
(0 -> 1 in logs) where the title check never fired; example.com negative
control returns 200 with no challenge path entered. End-to-end clearing
still depends on the requester's public IP (README caveat).
route.fulfill(response=...) re-serves the raw bytes fetched by
route.fetch(), so compressed (gzip/brotli/zstd) documents arrive
at the browser still compressed while the forwarded headers claim
otherwise - page.content() then returns garbled binary, breaking
indexers like uindex.org and 1337x.to (issue #385).
Fetch with accept-encoding: identity so the re-served body is plain
text, and drop content-encoding/content-length alongside the CSP
headers since they are stale after the rewrite.
A page whose network never goes idle (background analytics, websockets)
used to fail the whole request with a 408 once the networkidle wait
expired. Since the DOM is fully usable after domcontentloaded, treat a
networkidle timeout as non-fatal and return the loaded page instead.
Fatal timeouts during initial load or challenge solving still return 408.
Adds unit coverage for both paths using a fake page that fails
configured load-state waits.
The Firefox engine evaluates JS via eval(), which pages whose CSP
lacks 'unsafe-eval' block - every page.evaluate() then fails with
"call to eval() blocked by CSP". yggtorrent's search URL redirects to
a page with such a CSP, crashing the user-agent read and 500ing /v1.
Rewrite document responses without CSP headers via route.fetch +
fulfill. Juggler only routes the first request of a redirect chain,
so follow redirects inside the fetch and record the final URL
ourselves instead of relying on page.url.
- Catch both builtins.TimeoutError and playwright TimeoutError as 408
- Check challenge title before networkidle to avoid timeout on Cloudflare interstitial
- Add blockMedia and returnOnlyCookies request options
- Return raw PDF bytes as base64 with contentType application/pdf
- Skip tests on 408 timeouts; add PDF handling test