mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-11-14 08:38:25 +00:00
c8d66db22e
Closes #1149 ## What this adds OceanofPDF as a second Direct Download provider, alongside Anna's Archive. Unlike Z-Library, LibGen and Welib, which only download files Anna's Archive found, OceanofPDF searches and downloads on its own. It works with or without an AA mirror configured; with both, one search queries both and the results are merged. It plugs into the provider registry from #1337, so search, record routing and the download handler need no changes. **Search:** WordPress `/?s=` search, "title author" first, then the bare title if that finds nothing. One release per advertised format (`[PDF] [EPUB]`), with title, author and cover. Shelfmark's supported-format filter applies as usual. **Download** follows the site's three hops: 1. The book page has one POST form per format, targeting `Fetching_Resource.php`. 2. That returns an interstitial whose `<meta http-equiv="refresh">` points at a signed, expiring file URL. 3. The file URL serves the book. Pages go through `html_get_page` and its Cloudflare bypass. The form POST replays the clearance the bypasser stored, and re-solves once if challenged. Bypassers only replay GETs, so it can't be handed over directly. **Settings:** - `OCEANOFPDF_MIRROR_URLS` under Mirrors. Only the first URL is used. - An OceanofPDF toggle under Download Sources → Slow downloads. It's locked until a mirror is set and while the Cloudflare bypass is off. Its position in the list doesn't matter, and its description says so. - A saved list from before this change has no OceanofPDF entry. The settings page shows a missing option as off, so the provider treats it as off too, keeping the page and the behaviour in agreement. New installs default to on. ## Fixes found along the way - **`challenge_marker` rejected real pages behind Cloudflare** (separate commit, `fix(bypass)`). Cloudflare's bot-detection script injects `/cdn-cgi/challenge-platform/scripts/jsd/main.js` into every page it serves, after load. Bypassers return the rendered page, so a successful FlareSolverr solve of any page under 64 KB came back carrying `/cdn-cgi/challenge-platform` and was rejected as unsolved. The detector now ignores that one script path. Real challenge pages load from `/cdn-cgi/challenge-platform/h/…` and keep their other markers, so they are still caught. This also affects any other Cloudflare-fronted host, not just OceanofPDF. Happy to split it into its own PR if you prefer. - **Brotli on a plain `requests` call.** Copying `DOWNLOAD_HEADERS` into the form POST offered `br`. Cloudflare then answered in Brotli, which `requests` can't decode without the optional `brotli` package. The POST now leaves `Accept-Encoding` to `requests`. If a response still has no file link, its status, encoding, title and challenge marker are logged. ## Testing - **New tests:** 41 in `tests/direct_download/test_oceanofpdf.py` and 4 in `tests/bypass/test_cloudflare_beacon.py`. They cover search parsing, format and language filtering, form selection, the interstitial, the download flow (including a challenged POST, an HTML stub served as the file, and cancellation), the toggle, and registry wiring. Fixtures mirror the live site's markup. - **Existing tests:** `tests/direct_download`, `tests/bypass`, `tests/config`, `tests/core` and `tests/download` show no new failures against `main`. Some tests in those folders can't run on Windows; those fail identically on both. - **Lint:** `ruff check` and `ruff format --check` are clean on every changed file. - **Live:** verified end to end on a v1.4.0 Docker deployment behind FlareSolverr (external bypasser), with only an OceanofPDF mirror configured: search, then the book page, the form POST and the completed download. ## Known limits - OceanofPDF doesn't label languages, so its results always pass a language filter. - PDFs only appear when `pdf` is in the supported formats. - Each query reads the first results page only (roughly 7–10 books). - Downloaded files carry OceanofPDF's own tags/watermark, as noted in #1149. They aren't stripped. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Adam Stone <196691089+ast0ne1@users.noreply.github.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz>