fix: bound the challenge solver and pin the TLS handshake

Follow-up to the earlier CI fix, after A/B-ing every change against main and
against this branch's original commit.

What measurably changed, and what did not:

- The solver's retry loop was unbounded (max_attempts = sys.maxsize). On a
  challenge it cannot clear it retried ~1300 times per request and the caller
  waited out the entire max_timeout for a 408 it was always going to get.
  _solve_challenge now clicks, waits for the challenge markup to actually
  disappear, and gives up when the budget does.

- That wait exists because the solver's own verdict is worthless here: it
  judges its click with wait_for_load_state("networkidle"), which returned 9ms
  after the click while Cloudflare was still showing "verifying you are
  human", and then reported failure.

- The "is it still up?" check cannot use detect_cloudflare_challenge alone.
  That matches any script under /cdn-cgi/challenge-platform/, and Cloudflare
  serves its jsd bot-scoring beacon from the same path on cleared pages. Nor
  can it use the widget iframe: a cleared nowsecure.nl carries two of those
  with no challenge present. CHALLENGE_MARKERS matches the challenge
  orchestrator script and the interstitial's own markup.

- test_tls_handshake_looks_like_firefox pins what this branch is actually for.
  Measured through /v1 on the same host: main offers 52 cipher suites, this
  branch 16, and real Firefox offers 16. route.fetch() was re-issuing
  navigations through Playwright's HTTP client, and that is a fingerprint no
  header spoofing hides. Unlike a Cloudflare verdict the count is
  deterministic, so it is the one assertion here that cannot flake.

- Disabling COOP/COEP does let the solver reach and click the checkbox for the
  first time (Cloudflare advances to "verifying you are human"), but it changed
  no outcome across eight sites, and real Firefox ships those policies on.
  Recorded in a comment rather than shipped.

test_bypass keeps a hard assertion against targets that clear from any network.
The four Cloudflare guards hardest move to xfail rather than skip: they still
run and still report, but Cloudflare's opinion of the runner's IP cannot turn
the build red.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

src/endpoints.py

@@ -1,12 +1,13 @@
 import base64
 import time
 import warnings
-from asyncio import wait_for
+from asyncio import sleep, wait_for
 from http import HTTPStatus
 from typing import Annotated
 
 from fastapi import APIRouter, Depends, HTTPException
 from fastapi.responses import RedirectResponse
+from playwright.async_api import Page
 from playwright.async_api import TimeoutError as PlaywrightTimeoutError
 from playwright_captcha import CaptchaType
 from playwright_captcha.solvers.click.cloudflare.utils.detection import (
@@ -32,6 +33,24 @@ router = APIRouter()
 
 BrowserDep = Annotated[BrowserDepClass, Depends(get_browser)]
 
+# Markup only an unsolved challenge has. Two near misses to avoid:
+#
+#   script[src*="/cdn-cgi/challenge-platform/"] on its own also matches the jsd
+#   bot-scoring beacon Cloudflare serves from that path on ordinary pages, so it
+#   has to be narrowed to the challenge orchestrator (chl_page).
+#
+#   iframe[src*="challenges.cloudflare.com"] looks like the widget but outlives
+#   it: a cleared nowsecure.nl carries two of them with no challenge in sight.
+CHALLENGE_MARKERS = (
+    'script[src*="/cdn-cgi/challenge-platform/"][src*="chl_page"], '
+    "#challenge-error-text, #challenge-running, #challenge-stage"
+)
+
+# How long to let Cloudflare verify a click before trying again, and how often
+# to look. Verification took 5-15s in testing.
+CHALLENGE_SETTLE_SECONDS = 20.0
+CHALLENGE_POLL_SECONDS = 1.0
+
 
 @router.get("/", include_in_schema=False)
 def read_root():
@@ -146,28 +165,78 @@ async def _solve_challenge(dep: BrowserDep, timer: TimeoutTimer) -> None:
     """
     Attempt to solve a detected Cloudflare interstitial challenge.
 
-    Raises TimeoutError when the challenge outlives the attempt, so the caller
-    reports the same 408 whether the solver ran out of time or ran out of
-    attempts. The latter is what a challenge Byparr cannot clear from this
-    network looks like: the widget lives in an iframe whose content_frame() the
-    Firefox build refuses to hand over ("Permission denied to access property
-    docShell on cross-origin object"), so the solver never reaches the checkbox.
+    The solver's own verdict is not trusted. It judges its click by waiting for
+    networkidle, which returns the moment the network happens to be quiet --
+    measured at 9ms after the click -- and then reports "challenge still
+    present". Cloudflare needs seconds: it replaces the checkbox with
+    "verifying you are human" before the page turns over. So try, then wait for
+    the challenge itself to go away, and only give up once the budget is gone.
+
+    That bounds what used to be unbounded. With max_attempts at sys.maxsize the
+    solver retried an unreachable widget ~1300 times per request and the caller
+    waited out the whole max_timeout for a 408 it was always going to get.
     """
     logger.info("Challenge detected, attempting to solve...")
+    while timer.remaining() > 0:
+        try:
+            await wait_for(
+                dep.solver.solve_captcha(  # pyright: ignore[reportUnknownMemberType,reportUnknownArgumentType]
+                    captcha_container=dep.page,
+                    captcha_type=CaptchaType.CLOUDFLARE_INTERSTITIAL,
+                    wait_checkbox_attempts=1,
+                    wait_checkbox_delay=0.5,
+                ),
+                timeout=timer.remaining(),
+            )
+        except (CaptchaDetectionError, CaptchaSolvingError) as e:
+            # Both mean "not solved yet", not "cannot be solved": the widget is
+            # mid-verification and momentarily absent, or the click has landed
+            # and the verdict was taken too early.
+            logger.debug(f"Solver attempt inconclusive: {e}")
+
+        if await _challenge_cleared(dep, timer):
+            logger.info("Challenge cleared")
+            return
+
+    message = "Challenge still present when the request budget ran out"
+    raise TimeoutError(message)
+
+
+async def _challenge_cleared(dep: BrowserDep, timer: TimeoutTimer) -> bool:
+    """
+    Wait out Cloudflare's post-click verification, up to CHALLENGE_SETTLE_SECONDS.
+
+    Polls rather than waiting for a load event: Cloudflare swaps the widget for
+    a spinner in place and only navigates once it is satisfied, so there is no
+    single event to await.
+    """
+    deadline = min(CHALLENGE_SETTLE_SECONDS, timer.remaining())
+    waited = 0.0
+    while waited < deadline:
+        if not await _challenge_visible(dep.page):
+            return True
+        await sleep(CHALLENGE_POLL_SECONDS)
+        waited += CHALLENGE_POLL_SECONDS
+    return not await _challenge_visible(dep.page)
+
+
+async def _challenge_visible(page: Page) -> bool:
+    """
+    Report whether an unsolved challenge is still on the page.
+
+    Cloudflare serves two different scripts from /cdn-cgi/challenge-platform/:
+    the challenge orchestrator on an interstitial, and the jsd bot-scoring
+    beacon on ordinary pages once a visitor is cleared. detect_cloudflare_
+    challenge() matches both, so on its own it never reports success. Match the
+    orchestrator and the widget instead.
+    """
     try:
-        await wait_for(
-            dep.solver.solve_captcha(  # pyright: ignore[reportUnknownMemberType,reportUnknownArgumentType]
-                captcha_container=dep.page,
-                captcha_type=CaptchaType.CLOUDFLARE_INTERSTITIAL,
-                wait_checkbox_attempts=1,
-                wait_checkbox_delay=0.5,
-            ),
-            timeout=timer.remaining(),
-        )
-    except (CaptchaDetectionError, CaptchaSolvingError) as e:
-        logger.warning(f"Solver gave up on the challenge: {e}")
-        raise TimeoutError(str(e)) from e
-    logger.debug("Challenge solved successfully.")
+        return await page.locator(CHALLENGE_MARKERS).count() > 0
+    except Exception:
+        # A navigation tore down the execution context mid-check, which only
+        # happens once Cloudflare has moved us on.
+        logger.debug("Challenge lookup interrupted by a navigation")
+        return False
 
 
 async def _wait_for_networkidle(dep: BrowserDep, timer: TimeoutTimer) -> None:

src/utils.py

@@ -35,6 +35,24 @@ if len(logger.handlers) == 0:
     logger.addHandler(logging.StreamHandler())
 
 
+# Cloudflare embeds its challenge widget in an iframe carrying
+# allow="cross-origin-isolated". Firefox honours that by moving the iframe into
+# a cross-origin-isolated content process, where Juggler sees a frame with no
+# docShell and no URL, so content_frame() raises "Permission denied to access
+# property docShell on cross-origin object" and the solver never reaches the
+# checkbox.
+#
+# Setting browser.tabs.remote.useCrossOrigin{Opener,Embedder}Policy=false (as
+# upstream Playwright's Firefox does) undoes that, and the solver then clicks
+# the checkbox successfully -- but Cloudflare rejects the click anyway from
+# datacenter ranges, so it changed no outcome across eight sites measured.
+# Left off: real Firefox ships these policies on, and deviating from that is a
+# fingerprint signal not worth paying for an unproven gain.
+BROWSER_PREFS = {
+    "devtools.jsonview.enabled": False,
+}
+
+
 class TimeoutTimer(BaseModel):
     duration: int  # in seconds
     start_time: float = Field(default_factory=time.perf_counter)
@@ -96,7 +114,7 @@ async def get_browser(
         proxy=proxy_config,
         humanize=True,
         locale=BROWSER_LOCALE or "auto",
-        extra_prefs={"devtools.jsonview.enabled": False},
+        extra_prefs=BROWSER_PREFS,
     ) as browser_raw:
         # InvisiblePlaywright yields a Browser instance
         browser = cast("Browser", browser_raw)

tests/main_test.py

@@ -1,4 +1,6 @@
 import base64
+import json
+import re
 from http import HTTPStatus
 from json import JSONDecodeError
 from unittest.mock import AsyncMock, MagicMock
@@ -7,33 +9,52 @@ import httpx2
 import pytest
 from fastapi import HTTPException
 from playwright.async_api import TimeoutError as PlaywrightTimeoutError
-from playwright_captcha.utils.exceptions import CaptchaDetectionError
+from playwright_captcha.utils.exceptions import (
+    CaptchaDetectionError,
+    CaptchaSolvingError,
+)
 from starlette.testclient import TestClient
 
 from main import app
-from src.endpoints import read_item
+from src.endpoints import CHALLENGE_MARKERS, read_item
 from src.models import LinkRequest
 from src.utils import BrowserDepClass
 
 client = TestClient(app)
 
+# Real Firefox advertises 16 cipher suites; Playwright's HTTP client advertised
+# 52. A small margin absorbs Firefox version drift without letting 52 through.
+FIREFOX_CIPHER_SUITE_CEILING = 20
+
+# Sites Byparr clears from any network, datacenter ranges included. These carry
+# the hard assertion: if the bypass breaks, one of these goes red.
 test_websites = [
+    # Purpose-built Cloudflare challenge target. Serves a real interstitial and
+    # hands back a cf_clearance cookie once it is passed, so a pass here means
+    # the challenge was solved rather than never presented.
+    "https://nowsecure.nl/",
+    'https://www.yggtorrent.top/engine/search?do=search&order=desc&sort=publish_date&name="UNESCAPED"+"DOUBLEQUOTES"&category=2145',
+]
+
+# Cloudflare hands these its interactive checkbox challenge and then refuses the
+# click from datacenter ranges: the widget goes to "verifying you are human" and
+# comes back as a fresh unchecked box, indefinitely. Measured over four fresh
+# navigations and nine clicks, and reproduced from two unrelated hosting
+# providers on two architectures -- it is the visitor's IP being judged, not our
+# code. They still run rather than being skipped, so a real regression is
+# visible in the report and a pass is recorded as xpass, but the runner's luck
+# with Cloudflare cannot turn the build red.
+datacenter_hostile_websites = [
     "https://ext.to/",
     # "https://www.ygg.re/",
     "https://extratorrent.st/",
     "https://speed.cd/login",
-    'https://www.yggtorrent.top/engine/search?do=search&order=desc&sort=publish_date&name="UNESCAPED"+"DOUBLEQUOTES"&category=2145',
     "https://1337x.to/home/",
 ]
 
 
-@pytest.mark.parametrize("website", test_websites)
-def test_bypass(website: str):
-    """
-    Tests if the service can bypass cloudflare/DDOS-GUARD on given websites.
-
-    This test is skipped if the website is not reachable or does not have cloudflare/DDOS-GUARD.
-    """
+def _bypass(website: str) -> None:
+    """Ask Byparr for the page and require a clean answer."""
     test_request = httpx2.get(
         website,
     )
@@ -54,18 +75,25 @@ def test_bypass(website: str):
         json=LinkRequest.model_construct(url=website, cmd="request.get").model_dump(),
     )
 
-    if response.status_code == HTTPStatus.REQUEST_TIMEOUT:
-        # Cloudflare serves its interactive checkbox challenge to some networks
-        # and not others, and the widget lives in an iframe whose content_frame()
-        # this Firefox build will not hand over, so the solver cannot reach the
-        # checkbox to click it. Whether a given run sees that challenge is
-        # Cloudflare's call, not ours -- fail the run for our own regressions,
-        # not for the reputation of whatever IP the runner drew.
-        pytest.skip(f"Skipping {website} - challenge not solvable from this network")
-
     assert response.status_code == HTTPStatus.OK
 
 
+@pytest.mark.parametrize("website", test_websites)
+def test_bypass(website: str):
+    """Tests if the service can bypass cloudflare/DDOS-GUARD on given websites."""
+    _bypass(website)
+
+
+@pytest.mark.xfail(
+    reason="Cloudflare refuses the checkbox click from datacenter IPs",
+    strict=False,
+)
+@pytest.mark.parametrize("website", datacenter_hostile_websites)
+def test_bypass_datacenter_hostile(website: str):
+    """Same check against sites Cloudflare guards hardest, outcome permitting."""
+    _bypass(website)
+
+
 def test_json_api():
     """JSON APIs must return 200, not crash on the UA evaluate.
 
@@ -94,6 +122,38 @@ def test_json_api():
     assert '"ip"' in solution["response"]
 
 
+def test_tls_handshake_looks_like_firefox():
+    """
+    The handshake must be Firefox's, not the HTTP client's (#398).
+
+    route.fetch() re-issued navigations through Playwright's own HTTP stack, so
+    the ClientHello advertised 52 cipher suites where Firefox offers 16 -- a
+    fingerprint no amount of header spoofing hides. Unlike a Cloudflare verdict
+    this is deterministic, so it pins the regression that motivated this branch.
+    """
+    url = "https://www.howsmyssl.com/a/check"
+    if httpx2.get(url).status_code >= HTTPStatus.INTERNAL_SERVER_ERROR:
+        pytest.skip("Skipping TLS check - howsmyssl is down")
+
+    response = client.post(
+        "/v1",
+        json=LinkRequest.model_construct(url=url, cmd="request.get").model_dump(),
+    )
+    assert response.status_code == HTTPStatus.OK
+
+    body = response.json()["solution"]["response"]
+    report = json.loads(
+        re.sub(r"<[^>]+>", "", re.search(r"\{.*\}", body, re.DOTALL).group(0))
+    )
+    suites = len(report["given_cipher_suites"])
+
+    # Firefox offers 16; Playwright's client offered 52. Anything in between
+    # means the navigation is no longer going through the browser.
+    assert suites <= FIREFOX_CIPHER_SUITE_CEILING, (
+        f"{suites} cipher suites offered - the handshake is not Firefox's"
+    )
+
+
 def test_health_check():
     """
     Tests the health check endpoint.
@@ -147,12 +207,14 @@ def fake_dep(
     *,
     fail_states: set[str] | None = None,
     challenged: bool = False,
+    marker_counts: list[int] | None = None,
 ) -> BrowserDepClass:
     """
     Build a browser dependency triple backed by mocks.
 
-    `challenged` makes every selector match, which is how the detector reports
-    a Cloudflare challenge on the page.
+    `challenged` makes the detector report a Cloudflare challenge.
+    `marker_counts` drives the "is it still up?" check that runs after each
+    solve attempt: one entry per look, the last one repeating forever.
     """
     page = AsyncMock()
     page.url = "https://example.test/login"
@@ -164,9 +226,22 @@ def fake_dep(
     page.title.return_value = "Login"
     page.evaluate.return_value = "UnitTestBrowser/1.0"
     page.content.return_value = "<html><title>Login</title></html>"
-    locator = MagicMock()
-    locator.count = AsyncMock(return_value=1 if challenged else 0)
-    page.locator = MagicMock(return_value=locator)
+
+    remaining = list(marker_counts or [])
+
+    def count_for(selector: str) -> int:
+        """Answer the marker check from the script, everything else from `challenged`."""
+        if selector != CHALLENGE_MARKERS or not remaining:
+            return 1 if challenged else 0
+        return remaining.pop(0) if len(remaining) > 1 else remaining[0]
+
+    def locator(selector: str) -> MagicMock:
+        handle = MagicMock()
+        handle.count = AsyncMock(return_value=None)
+        handle.count.side_effect = lambda: count_for(selector)
+        return handle
+
+    page.locator = MagicMock(side_effect=locator)
 
     def wait_for_load_state(state: str, **_kwargs: object) -> None:
         """Fail the wait when asked for a configured state."""
@@ -209,22 +284,40 @@ async def test_domcontentloaded_timeout_returns_408():
 
 
 @pytest.mark.asyncio
-async def test_unreachable_challenge_widget_returns_408():
+async def test_challenge_that_clears_after_the_click_succeeds():
     """
-    A solver that runs out of attempts is a timeout, not a 500.
+    The solver's own "challenge still present" verdict must not end the request.
 
-    The solver raises CaptchaDetectionError once it has exhausted max_attempts
-    without reaching the challenge iframe. read_item only caught timeouts, so
-    that surfaced as an unhandled 500 the moment max_attempts stopped being
-    effectively infinite.
+    It judges its click by waiting for networkidle, which returns as soon as the
+    network happens to be quiet -- 9ms after the click, in practice -- while
+    Cloudflare is still showing "verifying you are human". Byparr has to wait
+    for the challenge markup itself to go away.
     """
-    dep = fake_dep(challenged=True)
+    dep = fake_dep(challenged=True, marker_counts=[1, 0])
+    dep.solver.solve_captcha.side_effect = CaptchaSolvingError(
+        "challenge still present or expected content not detected"
+    )
+
+    response = await read_item(
+        LinkRequest(url="https://example.test/login", max_timeout=5), dep
+    )
+
+    assert response.status == "ok"
+    assert response.solution.status == HTTPStatus.OK
+
+
+@pytest.mark.asyncio
+async def test_challenge_that_never_clears_returns_408():
+    """A challenge still up when the budget runs out is a timeout, not a 500."""
+    dep = fake_dep(challenged=True, marker_counts=[1])
     dep.solver.solve_captcha.side_effect = CaptchaDetectionError(
         "Cloudflare iframes not found"
     )
 
     with pytest.raises(HTTPException) as exc:
-        await read_item(LinkRequest(url="https://example.test/login"), dep)
+        await read_item(
+            LinkRequest(url="https://example.test/login", max_timeout=2), dep
+        )
 
     assert exc.value.status_code == HTTPStatus.REQUEST_TIMEOUT