mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-09-24 20:20:20 +01:00
3e2a7a48d5
## Summary Two failure modes on the same code path, both reported this week: Anna's Archive `/search` is gated behind a DDoS-Guard cookie probe that the manual redirect follower can never satisfy. **#1202 — the cookie is dropped on every hop.** AA URLs set `allow_redirects = False`, so `html_get_page` follows redirects by hand. The 302 to `?check=1` carries a `Set-Cookie` (`__ddg*`) that has to come back on the next request. Because cookies are passed per call and `requests` keeps no jar across manual hops, it was discarded each time and the server just re-issued the same redirect until `_MAX_REDIRECTS` raised `TooManyRedirects`. The file already had the right helper — `_new_cookies()` — but only the 503 Z-Library handshake branch called it. **#1204 — the loop never reaches the bypasser.** `TooManyRedirects` isn't in `_is_retryable_error` and carries no status code, so the 403 rescue path (`status == _HTTP_STATUS_FORBIDDEN`) never fired and all attempts repeated the identical failure — ~2.5 min, surfacing as the misleading "Network restricted or mirrors are blocked". These interact, which is why #1202's fix alone isn't enough. Requests merge as `cookies={**handshake_cookies, **cookies}`, so **stale bypasser cookies override the fresh handshake ones** — once `_cf_cookies` holds an expired `__ddg*`, the probe can never clear no matter how faithfully we echo. Hence one search per restart, exactly as #1204 describes. ## Changes 1. Harvest cookies in the same-host redirect branch, the way the 503 branch already does. `_new_cookies()` returns only *new* values, so a server re-sending an identical cookie yields an empty dict and a genuine redirect loop still terminates at `_MAX_REDIRECTS`. 2. Treat a redirect loop as a detected challenge: purge the stored cookies for that host and switch to the bypasser, instead of burning the retry budget. Gated on `allow_bypasser_fallback` and `_is_cf_bypass_enabled()`, and skipped when already bypassing, so AudiobookBay (`allow_bypasser_fallback=False`) and external-bypasser setups are unaffected. The broader point in #1204 stands — the fallback would be better gated on "challenge detected" than on specific status codes, since DDoS-Guard presents at least three faces (403 js-challenge, 429, and this redirect loop). This PR fixes the two live exits without that refactor. ## Tests Two regression tests, both failing before and passing after: - `test_html_get_page_echoes_cookies_across_same_host_redirects` — the fake server only returns results if `__ddg2_` comes back on the `?check=1` hop. - `test_html_get_page_redirect_loop_purges_cookies_and_bypasses` — asserts the stored cookies are cleared, the bypasser runs, and the loop is cut short rather than repeated per attempt. `ruff check` and `ruff format` clean. `tests/download/` passes except `test_download_url_ignores_zlib_cookie_refresh_failure`, which fails identically on unmodified `main` in my environment (no `seleniumbase` — the `browser` extra isn't installed). ## Verification Applied on a live v1.3.7 install (Debian LXC, internal CDP bypasser). Before: every search timed out through 10 retries with `TooManyRedirects`, zero results. After: ``` http.py:455 - Redirect loop detected; switching to bypasser internal_bypasser.py:756 - Bypass successful using _bypass_method_cdp_gui_click internal_bypasser.py:322 - Extracted 9 protection cookies for annas-archive.pk direct_download.py:1865 - Found 24 releases via ISBN ``` ~25 s per search, results render. Note the second search still re-solves the challenge, since the freshly stored cookies go stale immediately — the design issue #1204 raises, left for the broader fix. Fixes #1202 Fixes #1204 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_012Ln3yVj3sWHG2c6T78W1we --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz>