mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-11-18 04:30:31 +00:00
After a DDoS-Guard check clears itself, an Anna's Archive search can come back as "No results found" for a book AA has. On 1.7.0: ``` Bypass successful: ddos_guard challenge cleared itself Search page has no results table: https://annas-archive.gl/search?... (bytes=122986, title='... - Search - Anna's Archive', aa_markers=['/md5/', '/dyn/'], challenge_marker=None, over_challenge_size_cap=True) No results table found for query: ... ``` The same URL in a browser is about 745 KB, and the first `<table>` sits near byte 270,000. `_read_page_source` returns the `html` element as soon as it exists, which is as soon as the first bytes are parsed. AA streams the head and filters before the results, and the passive-solve check is satisfied by 50 characters of body text, so the read can land mid-document. Here it was one bypassed search in eight. This waits, inside the same `BYPASS_PAGE_SOURCE_TIMEOUT` budget, for `document.readyState` to leave `loading`, and reads state and HTML in one `evaluate` (same reasoning as the atomic snapshot in `_wait_for_aa_download_page`). If the budget runs out or the reads keep failing, it returns the element's HTML as before and logs a warning. Tests are in `tests/bypass/test_bypass_budgets.py`, next to the existing page-source ones. With the source change reverted the three new tests fail (two get the first chunk instead of the whole page). `make python-checks` is clean and `tests/bypass` passes. I also reproduced it with a real Chromium against a local server that serves a challenge page, then the page in two chunks 6 s apart: unpatched, 7,244 of 7,311 bytes and no table; patched, the whole page. I can share the script. One consequence of the old behaviour: the half-read page raises instead of returning an empty result, so the retry without the language filter never runs. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Page content is captured after the document finishes loading, improving the completeness of results. * If loading takes too long or a read fails, the current page content is used as a fallback so processing can continue. * If the page changes while content is being read, the fallback uses a fresh page reference to retrieve the available content. * Page-source reads continue to respect the configured timeout. <!-- end of auto-generated comment: release notes by coderabbit.ai -->