mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-11-17 14:10:34 +00:00
Fixes #1204. ## Problem #1198 sends a gated AA `/search` to the bypasser when the origin answers 403. DDoS-Guard has a second response: when the clearance cookies from an earlier solve go stale, it serves an endless `?check=1` redirect instead. `requests` follows that until `_raise_too_many_redirects`, and `TooManyRedirects` carries no status code, so `status == _HTTP_STATUS_FORBIDDEN` is false and the rescue never runs. All 10 retries re-send the same dead cookies, then the search fails as `Unable to reach download source. Network restricted or mirrors are blocked.` Direct-download search therefore works once per container start, and stays dead after the stored cookie ages out. v1.3.7 (`sha256:520715f3…`), internal bypasser, mirrors `.gl/.pk/.gd`: ``` 17:04:36 internal_bypasser.py:756 - Bypass successful using _bypass_method_cdp_gui_click ... 17:11:39 http.py:483 - Retry 1/10 for https://annas-archive.gl/search?...&check=1: TooManyRedirects: Too many redirects 17:12:12 http.py:493 - Giving up after 10 attempts 17:12:12 main.py:2870 - Release search failed for source direct_download: Unable to reach download source. Network restricted or mirrors are blocked. ``` The token is short-lived, which is what makes this reachable in normal use: ``` $ curl -sD - 'https://annas-archive.gl/search?...&check=1' HTTP/2 403 server: ddos-guard set-cookie: __ddg8_=…; Expires=Fri, 14-Aug-2026 15:39:38 GMT # issued 15:19:38, 20 min ``` ## Fix Handle the loop like the 403: drop the domain's stored cookies, then retry through the bypasser. The branch sits above the `status ==` ladder because `_get_status_code()` returns `None` for this exception. Cookies are purged only for the internal bypasser; with an external one `get_cf_cookies_for_domain()` already returns `{}`. Related but not changed here: `get_cf_cookies_for_domain()` enforces expiry for `cf_clearance` only, so `__ddg*` cookies are never evicted on age, which is why they go stale. This patch makes the rescue fire whatever the reason the cookies stopped working. ## Verification The regression test drives a real redirect loop through `html_get_page` (302 to `&check=1`, exception raised by the production path rather than faked) and asserts the cookies are purged and the bypasser runs once. - `pytest tests/download/test_http_bypasser_fallbacks.py`: 8 passed. `test_download_url_ignores_zlib_cookie_refresh_failure` fails in my checkout on a missing `seleniumbase`, unrelated to this change. - `ruff check`, `ruff format --check`: clean. - Running in production since 2026-08-14 on v1.3.7 with only this file replaced: six direct-download searches, five served, three books downloaded end to end, against one search per container start before. The rescue mid-download: ``` 19:12:14 http.py:449 - Redirect loop detected; switching to bypasser: https://annas-archive.gl/md5/cb8fba7abae800ddbae1adfb8d7699d9?&check=1 19:12:38 internal_bypasser.py:756 - Bypass successful using _bypass_method_cdp_gui_click 19:14:36 direct_download.py:1142 - Resolved download URL [aa-slow-nowait]: … 19:14:47 orchestrator.py:735 - download finished; starting post-processing ``` ## Separate issue this exposes DDoS-Guard does not accept a solved cookie from plain `requests` traffic, so after this patch the rescue runs for nearly every AA URL. `internal_bypasser.get()` serializes all solves on one module-wide lock and builds a fresh Chrome each time: 11-16 s uncontended, 43-52 s under concurrent load, measured on the host above. Correctness is cheap here, latency is not. Happy to open a separate PR for a warm browser session if that direction is welcome. Co-authored-by: Kukkerem <Kukkerem@users.noreply.github.com>