Files
shelfmark/tests/download
Zoltán SzabóandKukkerem 056ddd372a Send DDoS-Guard's ?check=1 redirect loop to the bypasser (#1210)
Fixes #1204.

## Problem

#1198 sends a gated AA `/search` to the bypasser when the origin answers
403.
DDoS-Guard has a second response: when the clearance cookies from an
earlier
solve go stale, it serves an endless `?check=1` redirect instead.

`requests` follows that until `_raise_too_many_redirects`, and
`TooManyRedirects` carries no status code, so `status ==
_HTTP_STATUS_FORBIDDEN`
is false and the rescue never runs. All 10 retries re-send the same dead
cookies, then the search fails as `Unable to reach download source.
Network
restricted or mirrors are blocked.`

Direct-download search therefore works once per container start, and
stays dead
after the stored cookie ages out.

v1.3.7 (`sha256:520715f3…`), internal bypasser, mirrors `.gl/.pk/.gd`:

```
17:04:36 internal_bypasser.py:756 - Bypass successful using _bypass_method_cdp_gui_click
...
17:11:39 http.py:483 - Retry 1/10 for https://annas-archive.gl/search?...&check=1:
    TooManyRedirects: Too many redirects
17:12:12 http.py:493 - Giving up after 10 attempts
17:12:12 main.py:2870 - Release search failed for source direct_download:
    Unable to reach download source. Network restricted or mirrors are blocked.
```

The token is short-lived, which is what makes this reachable in normal
use:

```
$ curl -sD - 'https://annas-archive.gl/search?...&check=1'
HTTP/2 403
server: ddos-guard
set-cookie: __ddg8_=…; Expires=Fri, 14-Aug-2026 15:39:38 GMT   # issued 15:19:38, 20 min
```

## Fix

Handle the loop like the 403: drop the domain's stored cookies, then
retry
through the bypasser. The branch sits above the `status ==` ladder
because
`_get_status_code()` returns `None` for this exception.

Cookies are purged only for the internal bypasser; with an external one
`get_cf_cookies_for_domain()` already returns `{}`.

Related but not changed here: `get_cf_cookies_for_domain()` enforces
expiry for
`cf_clearance` only, so `__ddg*` cookies are never evicted on age, which
is why
they go stale. This patch makes the rescue fire whatever the reason the
cookies
stopped working.

## Verification

The regression test drives a real redirect loop through `html_get_page`
(302 to `&check=1`, exception raised by the production path rather than
faked)
and asserts the cookies are purged and the bypasser runs once.

- `pytest tests/download/test_http_bypasser_fallbacks.py`: 8 passed.
`test_download_url_ignores_zlib_cookie_refresh_failure` fails in my
checkout
  on a missing `seleniumbase`, unrelated to this change.
- `ruff check`, `ruff format --check`: clean.
- Running in production since 2026-08-14 on v1.3.7 with only this file
replaced:
six direct-download searches, five served, three books downloaded end to
end,
against one search per container start before. The rescue mid-download:

```
19:12:14 http.py:449 - Redirect loop detected; switching to bypasser:
    https://annas-archive.gl/md5/cb8fba7abae800ddbae1adfb8d7699d9?&check=1
19:12:38 internal_bypasser.py:756 - Bypass successful using _bypass_method_cdp_gui_click
19:14:36 direct_download.py:1142 - Resolved download URL [aa-slow-nowait]: …
19:14:47 orchestrator.py:735 - download finished; starting post-processing
```

## Separate issue this exposes

DDoS-Guard does not accept a solved cookie from plain `requests`
traffic, so
after this patch the rescue runs for nearly every AA URL.
`internal_bypasser.get()`
serializes all solves on one module-wide lock and builds a fresh Chrome
each
time: 11-16 s uncontended, 43-52 s under concurrent load, measured on
the host
above. Correctness is cheap here, latency is not. Happy to open a
separate PR
for a warm browser session if that direction is welcome.

Co-authored-by: Kukkerem <Kukkerem@users.noreply.github.com>
2026-08-15 10:59:41 -04:00
..