Files
shelfmark/shelfmark
David Lacroix 9b89857923 fix(bypass): read the solved page once it has finished loading (#1721)
After a DDoS-Guard check clears itself, an Anna's Archive search can
come back as "No results found" for a book AA has. On 1.7.0:

```
Bypass successful: ddos_guard challenge cleared itself
Search page has no results table: https://annas-archive.gl/search?... (bytes=122986, title='... - Search - Anna's Archive', aa_markers=['/md5/', '/dyn/'], challenge_marker=None, over_challenge_size_cap=True)
No results table found for query: ...
```

The same URL in a browser is about 745 KB, and the first `<table>` sits
near byte 270,000. `_read_page_source` returns the `html` element as
soon as it exists, which is as soon as the first bytes are parsed. AA
streams the head and filters before the results, and the passive-solve
check is satisfied by 50 characters of body text, so the read can land
mid-document. Here it was one bypassed search in eight.

This waits, inside the same `BYPASS_PAGE_SOURCE_TIMEOUT` budget, for
`document.readyState` to leave `loading`, and reads state and HTML in
one `evaluate` (same reasoning as the atomic snapshot in
`_wait_for_aa_download_page`). If the budget runs out or the reads keep
failing, it returns the element's HTML as before and logs a warning.

Tests are in `tests/bypass/test_bypass_budgets.py`, next to the existing
page-source ones. With the source change reverted the three new tests
fail (two get the first chunk instead of the whole page). `make
python-checks` is clean and `tests/bypass` passes.

I also reproduced it with a real Chromium against a local server that
serves a challenge page, then the page in two chunks 6 s apart:
unpatched, 7,244 of 7,311 bytes and no table; patched, the whole page. I
can share the script.

One consequence of the old behaviour: the half-read page raises instead
of returning an empty result, so the retry without the language filter
never runs.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Page content is captured after the document finishes loading,
improving the completeness of results.
* If loading takes too long or a read fails, the current page content is
used as a fallback so processing can continue.
* If the page changes while content is being read, the fallback uses a
fresh page reference to retrieve the available content.
  * Page-source reads continue to respect the configured timeout.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-15 18:28:47 -05:00
..
2026-04-12 07:35:36 +01:00