mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-11-17 18:00:43 +00:00
Closes #1149 ## What this adds OceanofPDF as a second Direct Download provider, alongside Anna's Archive. Unlike Z-Library, LibGen and Welib, which only download files Anna's Archive found, OceanofPDF searches and downloads on its own. It works with or without an AA mirror configured; with both, one search queries both and the results are merged. It plugs into the provider registry from #1337, so search, record routing and the download handler need no changes. **Search:** WordPress `/?s=` search, "title author" first, then the bare title if that finds nothing. One release per advertised format (`[PDF] [EPUB]`), with title, author and cover. Shelfmark's supported-format filter applies as usual. **Download** follows the site's three hops: 1. The book page has one POST form per format, targeting `Fetching_Resource.php`. 2. That returns an interstitial whose `<meta http-equiv="refresh">` points at a signed, expiring file URL. 3. The file URL serves the book. Pages go through `html_get_page` and its Cloudflare bypass. The form POST replays the clearance the bypasser stored, and re-solves once if challenged. Bypassers only replay GETs, so it can't be handed over directly. **Settings:** - `OCEANOFPDF_MIRROR_URLS` under Mirrors. Only the first URL is used. - An OceanofPDF toggle under Download Sources → Slow downloads. It's locked until a mirror is set and while the Cloudflare bypass is off. Its position in the list doesn't matter, and its description says so. - A saved list from before this change has no OceanofPDF entry. The settings page shows a missing option as off, so the provider treats it as off too, keeping the page and the behaviour in agreement. New installs default to on. ## Fixes found along the way - **`challenge_marker` rejected real pages behind Cloudflare** (separate commit, `fix(bypass)`). Cloudflare's bot-detection script injects `/cdn-cgi/challenge-platform/scripts/jsd/main.js` into every page it serves, after load. Bypassers return the rendered page, so a successful FlareSolverr solve of any page under 64 KB came back carrying `/cdn-cgi/challenge-platform` and was rejected as unsolved. The detector now ignores that one script path. Real challenge pages load from `/cdn-cgi/challenge-platform/h/…` and keep their other markers, so they are still caught. This also affects any other Cloudflare-fronted host, not just OceanofPDF. Happy to split it into its own PR if you prefer. - **Brotli on a plain `requests` call.** Copying `DOWNLOAD_HEADERS` into the form POST offered `br`. Cloudflare then answered in Brotli, which `requests` can't decode without the optional `brotli` package. The POST now leaves `Accept-Encoding` to `requests`. If a response still has no file link, its status, encoding, title and challenge marker are logged. ## Testing - **New tests:** 41 in `tests/direct_download/test_oceanofpdf.py` and 4 in `tests/bypass/test_cloudflare_beacon.py`. They cover search parsing, format and language filtering, form selection, the interstitial, the download flow (including a challenged POST, an HTML stub served as the file, and cancellation), the toggle, and registry wiring. Fixtures mirror the live site's markup. - **Existing tests:** `tests/direct_download`, `tests/bypass`, `tests/config`, `tests/core` and `tests/download` show no new failures against `main`. Some tests in those folders can't run on Windows; those fail identically on both. - **Lint:** `ruff check` and `ruff format --check` are clean on every changed file. - **Live:** verified end to end on a v1.4.0 Docker deployment behind FlareSolverr (external bypasser), with only an OceanofPDF mirror configured: search, then the book page, the form POST and the completed download. ## Known limits - OceanofPDF doesn't label languages, so its results always pass a language filter. - PDFs only appear when `pdf` is in the supported formats. - Each query reads the first results page only (roughly 7–10 books). - Downloaded files carry OceanofPDF's own tags/watermark, as noted in #1149. They aren't stripped. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Adam Stone <196691089+ast0ne1@users.noreply.github.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz>
283 lines
8.3 KiB
Python
283 lines
8.3 KiB
Python
"""Centralized mirror configuration for direct-download sources."""
|
|
|
|
from __future__ import annotations
|
|
|
|
from typing import TYPE_CHECKING
|
|
|
|
from shelfmark.core.utils import normalize_http_url
|
|
|
|
if TYPE_CHECKING:
|
|
from shelfmark.core.config import Config
|
|
|
|
_config_module = None
|
|
|
|
|
|
def _get_config() -> Config:
|
|
"""Lazy import of config module to avoid circular imports."""
|
|
global _config_module
|
|
if _config_module is None:
|
|
from shelfmark.core.config import config
|
|
|
|
_config_module = config
|
|
return _config_module
|
|
|
|
|
|
# Mirror URLs are intentionally user-supplied only.
|
|
DEFAULT_AA_MIRRORS: list[str] = []
|
|
DEFAULT_LIBGEN_MIRRORS: list[str] = []
|
|
DEFAULT_ZLIB_MIRRORS: list[str] = []
|
|
DEFAULT_WELIB_MIRRORS: list[str] = []
|
|
|
|
_DOWNLOAD_SOURCE_MIRROR_LABELS = {
|
|
"aa-fast": "Anna's Archive",
|
|
"aa-slow": "Anna's Archive",
|
|
"aa-slow-nowait": "Anna's Archive",
|
|
"aa-slow-wait": "Anna's Archive",
|
|
"libgen": "LibGen",
|
|
"zlib": "Z-Library",
|
|
"welib": "Welib",
|
|
"oceanofpdf": "OceanofPDF",
|
|
}
|
|
|
|
|
|
def _normalize_mirror_url(url: str) -> str:
|
|
return normalize_http_url(url, default_scheme="https")
|
|
|
|
|
|
def _string_config_value(value: object) -> str:
|
|
"""Normalize mirror-related config values to strings."""
|
|
return value if isinstance(value, str) else str(value or "")
|
|
|
|
|
|
def _normalize_configured_urls(value: object) -> list[str]:
|
|
"""Normalize list or comma-separated mirror config into unique URLs."""
|
|
if isinstance(value, list):
|
|
parts = value
|
|
elif isinstance(value, str) and value.strip():
|
|
parts = value.split(",")
|
|
else:
|
|
return []
|
|
|
|
normalized_urls: list[str] = []
|
|
for raw_url in parts:
|
|
normalized = _normalize_mirror_url(str(raw_url))
|
|
if normalized and normalized not in normalized_urls:
|
|
normalized_urls.append(normalized)
|
|
return normalized_urls
|
|
|
|
|
|
def _get_primary_mirror_url(key: str) -> str | None:
|
|
"""Return a configured primary mirror URL, if present."""
|
|
config = _get_config()
|
|
primary = _normalize_mirror_url(_string_config_value(config.get(key, "")))
|
|
return primary or None
|
|
|
|
|
|
def _build_primary_and_additional_mirrors(primary_key: str, additional_key: str) -> list[str]:
|
|
"""Build an ordered mirror list from primary + additional config values."""
|
|
config = _get_config()
|
|
mirrors: list[str] = []
|
|
|
|
primary = _get_primary_mirror_url(primary_key)
|
|
if primary:
|
|
mirrors.append(primary)
|
|
|
|
for url in _normalize_configured_urls(config.get(additional_key, "")):
|
|
if url not in mirrors:
|
|
mirrors.append(url)
|
|
|
|
return mirrors
|
|
|
|
|
|
def get_aa_mirrors() -> list[str]:
|
|
"""Get Anna's Archive mirrors.
|
|
|
|
Returns:
|
|
Ordered list of user-configured AA mirror URLs.
|
|
|
|
Notes:
|
|
- The list is used to populate the AA mirror dropdown in Settings.
|
|
- When AA_BASE_URL is set to 'auto', mirrors are tried in the order listed.
|
|
|
|
"""
|
|
config = _get_config()
|
|
configured_list = _normalize_configured_urls(config.get("AA_MIRROR_URLS", None))
|
|
if configured_list:
|
|
return configured_list
|
|
return _normalize_configured_urls(config.get("AA_ADDITIONAL_URLS", ""))
|
|
|
|
|
|
def has_aa_mirror_configuration() -> bool:
|
|
"""Return True when direct-download search has at least one AA base URL to use."""
|
|
if get_aa_mirrors():
|
|
return True
|
|
|
|
configured_base_url = normalize_http_url(
|
|
_string_config_value(_get_config().get("AA_BASE_URL", "auto")),
|
|
default_scheme="https",
|
|
allow_special=("auto",),
|
|
)
|
|
return bool(configured_base_url and configured_base_url != "auto")
|
|
|
|
|
|
def get_libgen_mirrors() -> list[str]:
|
|
"""Get user-configured LibGen mirrors.
|
|
|
|
Returns:
|
|
List of LibGen mirror URLs.
|
|
|
|
"""
|
|
config = _get_config()
|
|
configured_list = _normalize_configured_urls(config.get("LIBGEN_MIRROR_URLS", None))
|
|
if configured_list:
|
|
return configured_list
|
|
return _normalize_configured_urls(config.get("LIBGEN_ADDITIONAL_URLS", ""))
|
|
|
|
|
|
def has_libgen_mirror_configuration() -> bool:
|
|
"""Return True when at least one LibGen mirror URL is configured."""
|
|
return bool(get_libgen_mirrors())
|
|
|
|
|
|
def get_zlib_mirrors() -> list[str]:
|
|
"""Get user-configured Z-Library mirrors, with primary first.
|
|
|
|
Returns:
|
|
List of Z-Library mirror URLs, primary first.
|
|
|
|
"""
|
|
config = _get_config()
|
|
configured_list = _normalize_configured_urls(config.get("ZLIB_MIRROR_URLS", None))
|
|
if configured_list:
|
|
return configured_list
|
|
return _build_primary_and_additional_mirrors("ZLIB_PRIMARY_URL", "ZLIB_ADDITIONAL_URLS")
|
|
|
|
|
|
def has_zlib_mirror_configuration() -> bool:
|
|
"""Return True when at least one Z-Library mirror URL is configured."""
|
|
return bool(get_zlib_mirrors())
|
|
|
|
|
|
def get_zlib_primary_url() -> str | None:
|
|
"""Get the primary Z-Library mirror URL.
|
|
|
|
Returns:
|
|
Primary Z-Library mirror URL, if configured.
|
|
|
|
"""
|
|
mirrors = get_zlib_mirrors()
|
|
return mirrors[0] if mirrors else None
|
|
|
|
|
|
def get_zlib_url_template() -> str | None:
|
|
"""Get Z-Library URL template using configured primary mirror.
|
|
|
|
Returns:
|
|
URL template with {md5} placeholder, if configured.
|
|
|
|
"""
|
|
primary = get_zlib_primary_url()
|
|
return f"{primary}/md5/{{md5}}" if primary else None
|
|
|
|
|
|
def get_welib_mirrors() -> list[str]:
|
|
"""Get user-configured Welib mirrors, with primary first.
|
|
|
|
Returns:
|
|
List of Welib mirror URLs, primary first.
|
|
|
|
"""
|
|
config = _get_config()
|
|
configured_list = _normalize_configured_urls(config.get("WELIB_MIRROR_URLS", None))
|
|
if configured_list:
|
|
return configured_list
|
|
return _build_primary_and_additional_mirrors("WELIB_PRIMARY_URL", "WELIB_ADDITIONAL_URLS")
|
|
|
|
|
|
def has_welib_mirror_configuration() -> bool:
|
|
"""Return True when at least one Welib mirror URL is configured."""
|
|
return bool(get_welib_mirrors())
|
|
|
|
|
|
def get_oceanofpdf_mirrors() -> list[str]:
|
|
"""Get user-configured OceanofPDF mirrors, in the order they were entered."""
|
|
return _normalize_configured_urls(_get_config().get("OCEANOFPDF_MIRROR_URLS", None))
|
|
|
|
|
|
def get_oceanofpdf_base_url() -> str | None:
|
|
"""Get the OceanofPDF mirror in use. Only the first configured one is used."""
|
|
mirrors = get_oceanofpdf_mirrors()
|
|
return mirrors[0] if mirrors else None
|
|
|
|
|
|
def has_oceanofpdf_mirror_configuration() -> bool:
|
|
"""Return True when at least one OceanofPDF mirror URL is configured."""
|
|
return bool(get_oceanofpdf_mirrors())
|
|
|
|
|
|
def has_download_source_mirror_configuration(source_id: str) -> bool:
|
|
"""Return True when the requested direct-download source has mirror config."""
|
|
if source_id in {"aa-fast", "aa-slow", "aa-slow-nowait", "aa-slow-wait"}:
|
|
return has_aa_mirror_configuration()
|
|
if source_id == "libgen":
|
|
return has_libgen_mirror_configuration()
|
|
if source_id == "zlib":
|
|
return has_zlib_mirror_configuration()
|
|
if source_id == "welib":
|
|
return has_welib_mirror_configuration()
|
|
if source_id == "oceanofpdf":
|
|
return has_oceanofpdf_mirror_configuration()
|
|
return False
|
|
|
|
|
|
def get_download_source_missing_mirror_reason(source_id: str) -> str | None:
|
|
"""Return a user-facing reason when a direct-download source has no mirror config."""
|
|
if has_download_source_mirror_configuration(source_id):
|
|
return None
|
|
|
|
label = _DOWNLOAD_SOURCE_MIRROR_LABELS.get(source_id)
|
|
if not label:
|
|
return None
|
|
|
|
return f"Add at least one {label} mirror in Mirrors"
|
|
|
|
|
|
def get_welib_primary_url() -> str | None:
|
|
"""Get the primary Welib mirror URL.
|
|
|
|
Returns:
|
|
Primary Welib mirror URL, if configured.
|
|
|
|
"""
|
|
mirrors = get_welib_mirrors()
|
|
return mirrors[0] if mirrors else None
|
|
|
|
|
|
def get_welib_url_template() -> str | None:
|
|
"""Get Welib URL template using configured primary mirror.
|
|
|
|
Returns:
|
|
URL template with {md5} placeholder, if configured.
|
|
|
|
"""
|
|
primary = get_welib_primary_url()
|
|
return f"{primary}/md5/{{md5}}" if primary else None
|
|
|
|
|
|
def get_zlib_cookie_domains() -> set:
|
|
"""Get set of Z-Library domains that need full cookie handling.
|
|
|
|
Used by internal_bypasser for CF bypass cookie management.
|
|
|
|
Returns:
|
|
Set of domain strings (without protocol).
|
|
|
|
"""
|
|
domains = set()
|
|
|
|
for url in get_zlib_mirrors():
|
|
domain = url.replace("https://", "").replace("http://", "").split("/")[0]
|
|
domains.add(domain)
|
|
|
|
return domains
|