mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-09-24 13:40:21 +01:00
fix(bypass): keep Anna's Archive's aa_ddg_check so clearance replays (#1305)
## What Add `aa_ddg_check` to the cookie-store allowlist. One name, one test. ## Why Every replay of stored clearance ends in the `?check=1` redirect loop, so each search pays a fresh browser solve. On this instance (v1.3.15, WireGuard egress, 0 VPN restarts across the traces) not one replay was accepted in three days of DEBUG logs. The `__ddg*` cookies are stored and replayed correctly. Anna's Archive also sets a cookie of its own, `aa_ddg_check`, and its `?check=1` hop only answers with the page when that cookie is present too. The allowlist keeps `cf_*` and `__ddg*` names, so this one was never stored. ## Measured, same egress IP, cookies taken from one solve | replayed | plain `requests` | `curl_cffi`, Chrome TLS fingerprint | |---|---|---| | filtered `__ddg*` only (current behaviour) | 302 → 302 → 302 … loop | 302 → 302 → 302 … loop | | filtered + `__ddg8_/9_/10_` | loop | loop | | filtered + `aa_ddg_check` | **302 → 200, real search page** | 302 → 200 | | `aa_ddg_check` alone | 302 → 403 | — | So the TLS fingerprint is not the problem, the per-check trio is not the answer, and the cookie needs the `__ddg*` clearance next to it. Cookie attributes as issued: domain `.annas-archive.gl`, path `/`, expiry 90 days. It is not bound to the query, and it is accepted with a stock Python User-Agent. ## Through the real fetch path Same process, `html_get_page`, the name allowlisted, three different queries: ``` 1st: solve expected 25.8s bypass_calls=1 title='frankenstein shelley - search - an' md5=True 2nd: other query 9.5s bypass_calls=0 title='pride and prejudice austen - searc' md5=True 3rd: third query 4.8s bypass_calls=0 title='dracula stoker - search - anna's a' md5=True ``` ## Notes - `tests/bypass/test_ddg_cookie_reuse.py` gains `test_aa_check_cookie_is_stored`; its docstring table gains the row. The bypass tests need seleniumbase to import and do not run on my macOS host, so this leans on CI. `ruff check` and `ruff format --check` pass. The logic was checked directly against `cookie_store` with the settings registry stubbed. - `__ddgmark_` carries a 24 h expiry, so the store's clearance is good for about a day before the next solve, which is what a browser would see too. - Follow-up to #1286. Same instance, same method: DEBUG trace, then a probe script inside the container.
This commit is contained in:
@@ -38,6 +38,10 @@ DDG_COOKIE_NAMES = {
|
||||
"ddg_last_challenge",
|
||||
}
|
||||
|
||||
# Anna's Archive's own pass for its ?check=1 hop. Without it the hop 302s back
|
||||
# forever, however good the __ddg* clearance is.
|
||||
AA_COOKIE_NAMES = {"aa_ddg_check"}
|
||||
|
||||
# DDoS-Guard cookies that describe *one* check rather than granting clearance, and so
|
||||
# must never be replayed on a later request. Observed live on Anna's Archive:
|
||||
#
|
||||
@@ -88,7 +92,8 @@ def _should_extract_cookie(name: str, *, extract_all: bool) -> bool:
|
||||
return True
|
||||
is_cf = name in CF_COOKIE_NAMES or name.startswith("cf_")
|
||||
is_ddg = name in DDG_COOKIE_NAMES or name.startswith("__ddg")
|
||||
return is_cf or is_ddg
|
||||
is_aa = name in AA_COOKIE_NAMES
|
||||
return is_cf or is_ddg or is_aa
|
||||
|
||||
|
||||
def _cookie_field(cookie: Any, name: str) -> Any:
|
||||
|
||||
@@ -1,11 +1,12 @@
|
||||
"""DDoS-Guard cookie reuse between requests.
|
||||
|
||||
Anna's Archive issues nine cookies after a solve, and they are not equivalent:
|
||||
Anna's Archive issues ten cookies after a solve, and they are not equivalent:
|
||||
|
||||
__ddg1_/__ddg2_/__ddgid_ ~1 year clearance
|
||||
__ddgmark_ ~1 day
|
||||
__ddg5_ session
|
||||
__ddg8_/__ddg9_/__ddg10_ ~40 min one check: token, CLIENT IP, TIMESTAMP
|
||||
aa_ddg_check ~90 days Anna's Archive's own pass for its ?check=1 hop
|
||||
|
||||
Replaying the last three is what produces the ?check=1 redirect loop. They describe a
|
||||
single check, so once the timestamp ages out - or the egress IP changes, routine
|
||||
@@ -97,6 +98,15 @@ def test_per_check_cookies_are_not_stored():
|
||||
assert ephemeral not in stored
|
||||
|
||||
|
||||
def test_aa_check_cookie_is_stored():
|
||||
"""Without aa_ddg_check the ?check=1 hop loops, whatever __ddg* is replayed."""
|
||||
_store([_Cookie("__ddg1_", "a"), _Cookie("__ddg5_", "b"), _Cookie("aa_ddg_check", "ok")])
|
||||
|
||||
stored = ib.get_cf_cookies_for_domain("annas-archive.gl")
|
||||
|
||||
assert stored == {"__ddg1_": "a", "__ddg5_": "b", "aa_ddg_check": "ok"}
|
||||
|
||||
|
||||
def test_clearance_cookies_survive():
|
||||
_store([_Cookie("__ddg1_", "a"), _Cookie("__ddg2_", "b"), _Cookie("__ddgid_", "c")])
|
||||
|
||||
|
||||
Reference in New Issue
Block a user