Files
shelfmark/shelfmark/core/mirrors.py
T
c8d66db22e feat(direct_download): add OceanofPDF as a standalone search and download provider (#1485)
Closes #1149

## What this adds

OceanofPDF as a second Direct Download provider, alongside Anna's
Archive. Unlike Z-Library, LibGen and Welib, which only download files
Anna's Archive found, OceanofPDF searches and downloads on its own. It
works with or without an AA mirror configured; with both, one search
queries both and the results are merged.

It plugs into the provider registry from #1337, so search, record
routing and the download handler need no changes.

**Search:** WordPress `/?s=` search, "title author" first, then the bare
title if that finds nothing. One release per advertised format (`[PDF]
[EPUB]`), with title, author and cover. Shelfmark's supported-format
filter applies as usual.

**Download** follows the site's three hops:

1. The book page has one POST form per format, targeting
`Fetching_Resource.php`.
2. That returns an interstitial whose `<meta http-equiv="refresh">`
points at a signed, expiring file URL.
3. The file URL serves the book.

Pages go through `html_get_page` and its Cloudflare bypass. The form
POST replays the clearance the bypasser stored, and re-solves once if
challenged. Bypassers only replay GETs, so it can't be handed over
directly.

**Settings:**
- `OCEANOFPDF_MIRROR_URLS` under Mirrors. Only the first URL is used.
- An OceanofPDF toggle under Download Sources → Slow downloads. It's
locked until a mirror is set and while the Cloudflare bypass is off. Its
position in the list doesn't matter, and its description says so.
- A saved list from before this change has no OceanofPDF entry. The
settings page shows a missing option as off, so the provider treats it
as off too, keeping the page and the behaviour in agreement. New
installs default to on.

## Fixes found along the way

- **`challenge_marker` rejected real pages behind Cloudflare** (separate
commit, `fix(bypass)`). Cloudflare's bot-detection script injects
`/cdn-cgi/challenge-platform/scripts/jsd/main.js` into every page it
serves, after load. Bypassers return the rendered page, so a successful
FlareSolverr solve of any page under 64 KB came back carrying
`/cdn-cgi/challenge-platform` and was rejected as unsolved. The detector
now ignores that one script path. Real challenge pages load from
`/cdn-cgi/challenge-platform/h/…` and keep their other markers, so they
are still caught. This also affects any other Cloudflare-fronted host,
not just OceanofPDF. Happy to split it into its own PR if you prefer.
- **Brotli on a plain `requests` call.** Copying `DOWNLOAD_HEADERS` into
the form POST offered `br`. Cloudflare then answered in Brotli, which
`requests` can't decode without the optional `brotli` package. The POST
now leaves `Accept-Encoding` to `requests`. If a response still has no
file link, its status, encoding, title and challenge marker are logged.

## Testing

- **New tests:** 41 in `tests/direct_download/test_oceanofpdf.py` and 4
in `tests/bypass/test_cloudflare_beacon.py`. They cover search parsing,
format and language filtering, form selection, the interstitial, the
download flow (including a challenged POST, an HTML stub served as the
file, and cancellation), the toggle, and registry wiring. Fixtures
mirror the live site's markup.
- **Existing tests:** `tests/direct_download`, `tests/bypass`,
`tests/config`, `tests/core` and `tests/download` show no new failures
against `main`. Some tests in those folders can't run on Windows; those
fail identically on both.
- **Lint:** `ruff check` and `ruff format --check` are clean on every
changed file.
- **Live:** verified end to end on a v1.4.0 Docker deployment behind
FlareSolverr (external bypasser), with only an OceanofPDF mirror
configured: search, then the book page, the form POST and the completed
download.

## Known limits

- OceanofPDF doesn't label languages, so its results always pass a
language filter.
- PDFs only appear when `pdf` is in the supported formats.
- Each query reads the first results page only (roughly 7–10 books).
- Downloaded files carry OceanofPDF's own tags/watermark, as noted in
#1149. They aren't stripped.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Adam Stone <196691089+ast0ne1@users.noreply.github.com>
Co-authored-by: CaliBrain <calibrain@l4n.xyz>
2026-10-20 02:07:23 -04:00

283 lines
8.3 KiB
Python

"""Centralized mirror configuration for direct-download sources."""
from __future__ import annotations
from typing import TYPE_CHECKING
from shelfmark.core.utils import normalize_http_url
if TYPE_CHECKING:
from shelfmark.core.config import Config
_config_module = None
def _get_config() -> Config:
"""Lazy import of config module to avoid circular imports."""
global _config_module
if _config_module is None:
from shelfmark.core.config import config
_config_module = config
return _config_module
# Mirror URLs are intentionally user-supplied only.
DEFAULT_AA_MIRRORS: list[str] = []
DEFAULT_LIBGEN_MIRRORS: list[str] = []
DEFAULT_ZLIB_MIRRORS: list[str] = []
DEFAULT_WELIB_MIRRORS: list[str] = []
_DOWNLOAD_SOURCE_MIRROR_LABELS = {
"aa-fast": "Anna's Archive",
"aa-slow": "Anna's Archive",
"aa-slow-nowait": "Anna's Archive",
"aa-slow-wait": "Anna's Archive",
"libgen": "LibGen",
"zlib": "Z-Library",
"welib": "Welib",
"oceanofpdf": "OceanofPDF",
}
def _normalize_mirror_url(url: str) -> str:
return normalize_http_url(url, default_scheme="https")
def _string_config_value(value: object) -> str:
"""Normalize mirror-related config values to strings."""
return value if isinstance(value, str) else str(value or "")
def _normalize_configured_urls(value: object) -> list[str]:
"""Normalize list or comma-separated mirror config into unique URLs."""
if isinstance(value, list):
parts = value
elif isinstance(value, str) and value.strip():
parts = value.split(",")
else:
return []
normalized_urls: list[str] = []
for raw_url in parts:
normalized = _normalize_mirror_url(str(raw_url))
if normalized and normalized not in normalized_urls:
normalized_urls.append(normalized)
return normalized_urls
def _get_primary_mirror_url(key: str) -> str | None:
"""Return a configured primary mirror URL, if present."""
config = _get_config()
primary = _normalize_mirror_url(_string_config_value(config.get(key, "")))
return primary or None
def _build_primary_and_additional_mirrors(primary_key: str, additional_key: str) -> list[str]:
"""Build an ordered mirror list from primary + additional config values."""
config = _get_config()
mirrors: list[str] = []
primary = _get_primary_mirror_url(primary_key)
if primary:
mirrors.append(primary)
for url in _normalize_configured_urls(config.get(additional_key, "")):
if url not in mirrors:
mirrors.append(url)
return mirrors
def get_aa_mirrors() -> list[str]:
"""Get Anna's Archive mirrors.
Returns:
Ordered list of user-configured AA mirror URLs.
Notes:
- The list is used to populate the AA mirror dropdown in Settings.
- When AA_BASE_URL is set to 'auto', mirrors are tried in the order listed.
"""
config = _get_config()
configured_list = _normalize_configured_urls(config.get("AA_MIRROR_URLS", None))
if configured_list:
return configured_list
return _normalize_configured_urls(config.get("AA_ADDITIONAL_URLS", ""))
def has_aa_mirror_configuration() -> bool:
"""Return True when direct-download search has at least one AA base URL to use."""
if get_aa_mirrors():
return True
configured_base_url = normalize_http_url(
_string_config_value(_get_config().get("AA_BASE_URL", "auto")),
default_scheme="https",
allow_special=("auto",),
)
return bool(configured_base_url and configured_base_url != "auto")
def get_libgen_mirrors() -> list[str]:
"""Get user-configured LibGen mirrors.
Returns:
List of LibGen mirror URLs.
"""
config = _get_config()
configured_list = _normalize_configured_urls(config.get("LIBGEN_MIRROR_URLS", None))
if configured_list:
return configured_list
return _normalize_configured_urls(config.get("LIBGEN_ADDITIONAL_URLS", ""))
def has_libgen_mirror_configuration() -> bool:
"""Return True when at least one LibGen mirror URL is configured."""
return bool(get_libgen_mirrors())
def get_zlib_mirrors() -> list[str]:
"""Get user-configured Z-Library mirrors, with primary first.
Returns:
List of Z-Library mirror URLs, primary first.
"""
config = _get_config()
configured_list = _normalize_configured_urls(config.get("ZLIB_MIRROR_URLS", None))
if configured_list:
return configured_list
return _build_primary_and_additional_mirrors("ZLIB_PRIMARY_URL", "ZLIB_ADDITIONAL_URLS")
def has_zlib_mirror_configuration() -> bool:
"""Return True when at least one Z-Library mirror URL is configured."""
return bool(get_zlib_mirrors())
def get_zlib_primary_url() -> str | None:
"""Get the primary Z-Library mirror URL.
Returns:
Primary Z-Library mirror URL, if configured.
"""
mirrors = get_zlib_mirrors()
return mirrors[0] if mirrors else None
def get_zlib_url_template() -> str | None:
"""Get Z-Library URL template using configured primary mirror.
Returns:
URL template with {md5} placeholder, if configured.
"""
primary = get_zlib_primary_url()
return f"{primary}/md5/{{md5}}" if primary else None
def get_welib_mirrors() -> list[str]:
"""Get user-configured Welib mirrors, with primary first.
Returns:
List of Welib mirror URLs, primary first.
"""
config = _get_config()
configured_list = _normalize_configured_urls(config.get("WELIB_MIRROR_URLS", None))
if configured_list:
return configured_list
return _build_primary_and_additional_mirrors("WELIB_PRIMARY_URL", "WELIB_ADDITIONAL_URLS")
def has_welib_mirror_configuration() -> bool:
"""Return True when at least one Welib mirror URL is configured."""
return bool(get_welib_mirrors())
def get_oceanofpdf_mirrors() -> list[str]:
"""Get user-configured OceanofPDF mirrors, in the order they were entered."""
return _normalize_configured_urls(_get_config().get("OCEANOFPDF_MIRROR_URLS", None))
def get_oceanofpdf_base_url() -> str | None:
"""Get the OceanofPDF mirror in use. Only the first configured one is used."""
mirrors = get_oceanofpdf_mirrors()
return mirrors[0] if mirrors else None
def has_oceanofpdf_mirror_configuration() -> bool:
"""Return True when at least one OceanofPDF mirror URL is configured."""
return bool(get_oceanofpdf_mirrors())
def has_download_source_mirror_configuration(source_id: str) -> bool:
"""Return True when the requested direct-download source has mirror config."""
if source_id in {"aa-fast", "aa-slow", "aa-slow-nowait", "aa-slow-wait"}:
return has_aa_mirror_configuration()
if source_id == "libgen":
return has_libgen_mirror_configuration()
if source_id == "zlib":
return has_zlib_mirror_configuration()
if source_id == "welib":
return has_welib_mirror_configuration()
if source_id == "oceanofpdf":
return has_oceanofpdf_mirror_configuration()
return False
def get_download_source_missing_mirror_reason(source_id: str) -> str | None:
"""Return a user-facing reason when a direct-download source has no mirror config."""
if has_download_source_mirror_configuration(source_id):
return None
label = _DOWNLOAD_SOURCE_MIRROR_LABELS.get(source_id)
if not label:
return None
return f"Add at least one {label} mirror in Mirrors"
def get_welib_primary_url() -> str | None:
"""Get the primary Welib mirror URL.
Returns:
Primary Welib mirror URL, if configured.
"""
mirrors = get_welib_mirrors()
return mirrors[0] if mirrors else None
def get_welib_url_template() -> str | None:
"""Get Welib URL template using configured primary mirror.
Returns:
URL template with {md5} placeholder, if configured.
"""
primary = get_welib_primary_url()
return f"{primary}/md5/{{md5}}" if primary else None
def get_zlib_cookie_domains() -> set:
"""Get set of Z-Library domains that need full cookie handling.
Used by internal_bypasser for CF bypass cookie management.
Returns:
Set of domain strings (without protocol).
"""
domains = set()
for url in get_zlib_mirrors():
domain = url.replace("https://", "").replace("http://", "").split("/")[0]
domains.add(domain)
return domains