shelfmark

571 Commits 5 Branches 48 Tags

2 Commits

Author	SHA1	Message	Date
Zoltán Szabó	35037b35fd	fix(irc): search by surname, and rank the answer by author (#1331 ) (#1332 ) Fixes #1331. A search bot ANDs every term against a filename, so the given name is the term that empties the result set. Measured against irchighway's #ebooks: "Revelations David Petrie" is answered "no results", "Revelations Petrie" returns 9 matches, 6 of which parse, all filed as "D Petrie". The query now carries the title and the surname, read off the search variant so the ISBN fallback and a manual query - which set author="" on purpose - keep their current shape. Title-only, the shape #1295 settled on for Prowlarr, does not transfer: the bot caps an answer at 1000 matches, and a bare "Revelations" hits that cap with 923 parsed rows across 500 authors, so the cap itself can drop the wanted book. The surname is the token the two spellings share and it keeps the answer small. The full author then orders what comes back, reusing author_affinity from #1295, since a surname also matches a different author who shares it. It sits under server availability the way indexer priority does in #1295: a download addresses one named bot and waits 120s for it, so a match from a bot that has left the channel must not outrank a mismatch that can answer. Ranking runs on the way out rather than before the cache, because one query identity is shared by every book that produced that query. Two things found while testing: - The parser writes the literal "Unknown" when a filename has no " - " separator (parser.py:168). Ranked literally that sorts as a wrong author, so author_affinity's middle tier was unreachable here; it is now read as absent. 5 of those 923 rows are affected. - author_affinity moves to shelfmark/core/author_match.py, unchanged, so IRC does not import from the Prowlarr package. Prowlarr behaviour is untouched and its tests pass as they are. The three IRC assertions in the #1252 regression file move to the surname form. The invariant they pin - one contributor's name reaches the query, never the whole credit list - is unchanged. Tested with make python-lint, python-format, python-dead-code, python-typecheck and python-test, and end to end against irchighway with the patched source: it posts "Revelations Petrie" and returns 6 releases.	2026-09-11 22:14:37 -04:00
Zoltán SzabóandKukkerem	056ddd372a	Send DDoS-Guard's ?check=1 redirect loop to the bypasser (#1210 ) Fixes #1204. ## Problem #1198 sends a gated AA `/search` to the bypasser when the origin answers 403. DDoS-Guard has a second response: when the clearance cookies from an earlier solve go stale, it serves an endless `?check=1` redirect instead. `requests` follows that until `_raise_too_many_redirects`, and `TooManyRedirects` carries no status code, so `status == _HTTP_STATUS_FORBIDDEN` is false and the rescue never runs. All 10 retries re-send the same dead cookies, then the search fails as `Unable to reach download source. Network restricted or mirrors are blocked.` Direct-download search therefore works once per container start, and stays dead after the stored cookie ages out. v1.3.7 (`sha256:520715f3…`), internal bypasser, mirrors `.gl/.pk/.gd`: ``` 17:04:36 internal_bypasser.py:756 - Bypass successful using _bypass_method_cdp_gui_click ... 17:11:39 http.py:483 - Retry 1/10 for https://annas-archive.gl/search?...&check=1: TooManyRedirects: Too many redirects 17:12:12 http.py:493 - Giving up after 10 attempts 17:12:12 main.py:2870 - Release search failed for source direct_download: Unable to reach download source. Network restricted or mirrors are blocked. ``` The token is short-lived, which is what makes this reachable in normal use: ``` $ curl -sD - 'https://annas-archive.gl/search?...&check=1' HTTP/2 403 server: ddos-guard set-cookie: __ddg8_=…; Expires=Fri, 14-Aug-2026 15:39:38 GMT # issued 15:19:38, 20 min ``` ## Fix Handle the loop like the 403: drop the domain's stored cookies, then retry through the bypasser. The branch sits above the `status ==` ladder because `_get_status_code()` returns `None` for this exception. Cookies are purged only for the internal bypasser; with an external one `get_cf_cookies_for_domain()` already returns `{}`. Related but not changed here: `get_cf_cookies_for_domain()` enforces expiry for `cf_clearance` only, so `__ddg*` cookies are never evicted on age, which is why they go stale. This patch makes the rescue fire whatever the reason the cookies stopped working. ## Verification The regression test drives a real redirect loop through `html_get_page` (302 to `&check=1`, exception raised by the production path rather than faked) and asserts the cookies are purged and the bypasser runs once. - `pytest tests/download/test_http_bypasser_fallbacks.py`: 8 passed. `test_download_url_ignores_zlib_cookie_refresh_failure` fails in my checkout on a missing `seleniumbase`, unrelated to this change. - `ruff check`, `ruff format --check`: clean. - Running in production since 2026-08-14 on v1.3.7 with only this file replaced: six direct-download searches, five served, three books downloaded end to end, against one search per container start before. The rescue mid-download: ``` 19:12:14 http.py:449 - Redirect loop detected; switching to bypasser: https://annas-archive.gl/md5/cb8fba7abae800ddbae1adfb8d7699d9?&check=1 19:12:38 internal_bypasser.py:756 - Bypass successful using _bypass_method_cdp_gui_click 19:14:36 direct_download.py:1142 - Resolved download URL [aa-slow-nowait]: … 19:14:47 orchestrator.py:735 - download finished; starting post-processing ``` ## Separate issue this exposes DDoS-Guard does not accept a solved cookie from plain `requests` traffic, so after this patch the rescue runs for nearly every AA URL. `internal_bypasser.get()` serializes all solves on one module-wide lock and builds a fresh Chrome each time: 11-16 s uncontended, 43-52 s under concurrent load, measured on the host above. Correctness is cheap here, latency is not. Happy to open a separate PR for a warm browser session if that direction is welcome. Co-authored-by: Kukkerem <Kukkerem@users.noreply.github.com>	2026-08-15 10:59:41 -04:00

Author

SHA1

Message

Date

Zoltán Szabó

35037b35fd

fix(irc): search by surname, and rank the answer by author (#1331 ) (#1332 )

Fixes #1331.

A search bot ANDs every term against a filename, so the given name is
the term
that empties the result set. Measured against irchighway's #ebooks:
"Revelations
David Petrie" is answered "no results", "Revelations Petrie" returns 9
matches,
6 of which parse, all filed as "D Petrie".

The query now carries the title and the surname, read off the search
variant so
the ISBN fallback and a manual query - which set author="" on purpose -
keep
their current shape.

Title-only, the shape #1295 settled on for Prowlarr, does not transfer:
the bot
caps an answer at 1000 matches, and a bare "Revelations" hits that cap
with 923
parsed rows across 500 authors, so the cap itself can drop the wanted
book. The
surname is the token the two spellings share and it keeps the answer
small.

The full author then orders what comes back, reusing author_affinity
from #1295,
since a surname also matches a different author who shares it. It sits
under
server availability the way indexer priority does in #1295: a download
addresses
one named bot and waits 120s for it, so a match from a bot that has left
the
channel must not outrank a mismatch that can answer. Ranking runs on the
way out
rather than before the cache, because one query identity is shared by
every book
that produced that query.

Two things found while testing:

- The parser writes the literal "Unknown" when a filename has no " - "
separator
  (parser.py:168). Ranked literally that sorts as a wrong author, so
author_affinity's middle tier was unreachable here; it is now read as
absent.
  5 of those 923 rows are affected.
- author_affinity moves to shelfmark/core/author_match.py, unchanged, so
IRC
does not import from the Prowlarr package. Prowlarr behaviour is
untouched and
  its tests pass as they are.

The three IRC assertions in the #1252 regression file move to the
surname form.
The invariant they pin - one contributor's name reaches the query, never
the
whole credit list - is unchanged.

Tested with make python-lint, python-format, python-dead-code,
python-typecheck
and python-test, and end to end against irchighway with the patched
source: it
posts "Revelations Petrie" and returns 6 releases.

2026-09-11 22:14:37 -04:00

Zoltán SzabóandKukkerem

056ddd372a

Send DDoS-Guard's ?check=1 redirect loop to the bypasser (#1210 )

Fixes #1204.

## Problem

#1198 sends a gated AA `/search` to the bypasser when the origin answers
403.
DDoS-Guard has a second response: when the clearance cookies from an
earlier
solve go stale, it serves an endless `?check=1` redirect instead.

`requests` follows that until `_raise_too_many_redirects`, and
`TooManyRedirects` carries no status code, so `status ==
_HTTP_STATUS_FORBIDDEN`
is false and the rescue never runs. All 10 retries re-send the same dead
cookies, then the search fails as `Unable to reach download source.
Network
restricted or mirrors are blocked.`

Direct-download search therefore works once per container start, and
stays dead
after the stored cookie ages out.

v1.3.7 (`sha256:520715f3…`), internal bypasser, mirrors `.gl/.pk/.gd`:

```
17:04:36 internal_bypasser.py:756 - Bypass successful using _bypass_method_cdp_gui_click
...
17:11:39 http.py:483 - Retry 1/10 for https://annas-archive.gl/search?...&check=1:
    TooManyRedirects: Too many redirects
17:12:12 http.py:493 - Giving up after 10 attempts
17:12:12 main.py:2870 - Release search failed for source direct_download:
    Unable to reach download source. Network restricted or mirrors are blocked.
```

The token is short-lived, which is what makes this reachable in normal
use:

```
$ curl -sD - 'https://annas-archive.gl/search?...&check=1'
HTTP/2 403
server: ddos-guard
set-cookie: __ddg8_=…; Expires=Fri, 14-Aug-2026 15:39:38 GMT   # issued 15:19:38, 20 min
```

## Fix

Handle the loop like the 403: drop the domain's stored cookies, then
retry
through the bypasser. The branch sits above the `status ==` ladder
because
`_get_status_code()` returns `None` for this exception.

Cookies are purged only for the internal bypasser; with an external one
`get_cf_cookies_for_domain()` already returns `{}`.

Related but not changed here: `get_cf_cookies_for_domain()` enforces
expiry for
`cf_clearance` only, so `__ddg*` cookies are never evicted on age, which
is why
they go stale. This patch makes the rescue fire whatever the reason the
cookies
stopped working.

## Verification

The regression test drives a real redirect loop through `html_get_page`
(302 to `&check=1`, exception raised by the production path rather than
faked)
and asserts the cookies are purged and the bypasser runs once.

- `pytest tests/download/test_http_bypasser_fallbacks.py`: 8 passed.
`test_download_url_ignores_zlib_cookie_refresh_failure` fails in my
checkout
  on a missing `seleniumbase`, unrelated to this change.
- `ruff check`, `ruff format --check`: clean.
- Running in production since 2026-08-14 on v1.3.7 with only this file
replaced:
six direct-download searches, five served, three books downloaded end to
end,
against one search per container start before. The rescue mid-download:

```
19:12:14 http.py:449 - Redirect loop detected; switching to bypasser:
    https://annas-archive.gl/md5/cb8fba7abae800ddbae1adfb8d7699d9?&check=1
19:12:38 internal_bypasser.py:756 - Bypass successful using _bypass_method_cdp_gui_click
19:14:36 direct_download.py:1142 - Resolved download URL [aa-slow-nowait]: …
19:14:47 orchestrator.py:735 - download finished; starting post-processing
```

## Separate issue this exposes

DDoS-Guard does not accept a solved cookie from plain `requests`
traffic, so
after this patch the rescue runs for nearly every AA URL.
`internal_bypasser.get()`
serializes all solves on one module-wide lock and builds a fresh Chrome
each
time: 11-16 s uncontended, 43-52 s under concurrent load, measured on
the host
above. Correctness is cheap here, latency is not. Happy to open a
separate PR
for a warm browser session if that direction is welcome.

Co-authored-by: Kukkerem <Kukkerem@users.noreply.github.com>

2026-08-15 10:59:41 -04:00