mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-11-15 11:28:33 +00:00
feat(direct_download): add OceanofPDF as a standalone search and download provider (#1485)
Closes #1149 ## What this adds OceanofPDF as a second Direct Download provider, alongside Anna's Archive. Unlike Z-Library, LibGen and Welib, which only download files Anna's Archive found, OceanofPDF searches and downloads on its own. It works with or without an AA mirror configured; with both, one search queries both and the results are merged. It plugs into the provider registry from #1337, so search, record routing and the download handler need no changes. **Search:** WordPress `/?s=` search, "title author" first, then the bare title if that finds nothing. One release per advertised format (`[PDF] [EPUB]`), with title, author and cover. Shelfmark's supported-format filter applies as usual. **Download** follows the site's three hops: 1. The book page has one POST form per format, targeting `Fetching_Resource.php`. 2. That returns an interstitial whose `<meta http-equiv="refresh">` points at a signed, expiring file URL. 3. The file URL serves the book. Pages go through `html_get_page` and its Cloudflare bypass. The form POST replays the clearance the bypasser stored, and re-solves once if challenged. Bypassers only replay GETs, so it can't be handed over directly. **Settings:** - `OCEANOFPDF_MIRROR_URLS` under Mirrors. Only the first URL is used. - An OceanofPDF toggle under Download Sources → Slow downloads. It's locked until a mirror is set and while the Cloudflare bypass is off. Its position in the list doesn't matter, and its description says so. - A saved list from before this change has no OceanofPDF entry. The settings page shows a missing option as off, so the provider treats it as off too, keeping the page and the behaviour in agreement. New installs default to on. ## Fixes found along the way - **`challenge_marker` rejected real pages behind Cloudflare** (separate commit, `fix(bypass)`). Cloudflare's bot-detection script injects `/cdn-cgi/challenge-platform/scripts/jsd/main.js` into every page it serves, after load. Bypassers return the rendered page, so a successful FlareSolverr solve of any page under 64 KB came back carrying `/cdn-cgi/challenge-platform` and was rejected as unsolved. The detector now ignores that one script path. Real challenge pages load from `/cdn-cgi/challenge-platform/h/…` and keep their other markers, so they are still caught. This also affects any other Cloudflare-fronted host, not just OceanofPDF. Happy to split it into its own PR if you prefer. - **Brotli on a plain `requests` call.** Copying `DOWNLOAD_HEADERS` into the form POST offered `br`. Cloudflare then answered in Brotli, which `requests` can't decode without the optional `brotli` package. The POST now leaves `Accept-Encoding` to `requests`. If a response still has no file link, its status, encoding, title and challenge marker are logged. ## Testing - **New tests:** 41 in `tests/direct_download/test_oceanofpdf.py` and 4 in `tests/bypass/test_cloudflare_beacon.py`. They cover search parsing, format and language filtering, form selection, the interstitial, the download flow (including a challenged POST, an HTML stub served as the file, and cancellation), the toggle, and registry wiring. Fixtures mirror the live site's markup. - **Existing tests:** `tests/direct_download`, `tests/bypass`, `tests/config`, `tests/core` and `tests/download` show no new failures against `main`. Some tests in those folders can't run on Windows; those fail identically on both. - **Lint:** `ruff check` and `ruff format --check` are clean on every changed file. - **Live:** verified end to end on a v1.4.0 Docker deployment behind FlareSolverr (external bypasser), with only an OceanofPDF mirror configured: search, then the book page, the form POST and the completed download. ## Known limits - OceanofPDF doesn't label languages, so its results always pass a language filter. - PDFs only appear when `pdf` is in the supported formats. - Each query reads the first results page only (roughly 7–10 books). - Downloaded files carry OceanofPDF's own tags/watermark, as noted in #1149. They aren't stripped. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Adam Stone <196691089+ast0ne1@users.noreply.github.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz>
This commit is contained in:
10 files changed
+1174
-5
No files matched your search
@@ -2750,6 +2750,7 @@ How long the bypass helper process may sit unused before it is shut down. Higher
|
||||
| `LIBGEN_MIRROR_URLS` | Mirrors are tried in the order you add them until one works. | string (comma-separated) | _empty list_ |
|
||||
| `ZLIB_MIRROR_URLS` | Only the first mirror in the list is used. | string (comma-separated) | _empty list_ |
|
||||
| `WELIB_MIRROR_URLS` | Only the first mirror in the list is used. | string (comma-separated) | _empty list_ |
|
||||
| `OCEANOFPDF_MIRROR_URLS` | Searches and downloads directly, without Anna's Archive. Only the first mirror in the list is used. The site is behind Cloudflare, so the bypass must be enabled. | string (comma-separated) | _empty list_ |
|
||||
|
||||
<details>
|
||||
<summary>Detailed descriptions</summary>
|
||||
@@ -2800,4 +2801,13 @@ Only the first mirror in the list is used.
|
||||
- **Type:** string (comma-separated)
|
||||
- **Default:** _empty list_
|
||||
|
||||
#### `OCEANOFPDF_MIRROR_URLS`
|
||||
|
||||
**OceanofPDF**
|
||||
|
||||
Searches and downloads directly, without Anna's Archive. Only the first mirror in the list is used. The site is behind Cloudflare, so the bypass must be enabled.
|
||||
|
||||
- **Type:** string (comma-separated)
|
||||
- **Default:** _empty list_
|
||||
|
||||
</details>
|
||||
@@ -35,6 +35,13 @@ _RAW_HTML_MARKERS = (
|
||||
"/.well-known/ddos-guard/",
|
||||
)
|
||||
|
||||
# Cloudflare's bot-detection beacon. Its script injects an iframe loading this path into
|
||||
# every page a Cloudflare site serves, so a browser-rendered page - what FlareSolverr and
|
||||
# the internal bypasser hand back - carries it after a *successful* solve. Real challenge
|
||||
# pages load from /cdn-cgi/challenge-platform/h/... and keep their other markers, so
|
||||
# discarding this one path before the scan does not hide them.
|
||||
_CLOUDFLARE_BEACON_PATH = "/cdn-cgi/challenge-platform/scripts/jsd/"
|
||||
|
||||
# An interstitial is a few KB of markup. Past that it is a real page that happens to
|
||||
# mention a marker - a protected site links its own DDoS-Guard endpoints on every page.
|
||||
MAX_CHALLENGE_HTML_CHARS = 64 * 1024
|
||||
@@ -49,7 +56,7 @@ def challenge_marker(html: str) -> str | None:
|
||||
"""
|
||||
if not html or len(html) > MAX_CHALLENGE_HTML_CHARS:
|
||||
return None
|
||||
lowered = html.lower()
|
||||
lowered = html.lower().replace(_CLOUDFLARE_BEACON_PATH, "")
|
||||
for marker in (
|
||||
*_RAW_HTML_MARKERS,
|
||||
*DDOS_GUARD_INDICATORS,
|
||||
|
||||
@@ -1592,6 +1592,13 @@ def _get_slow_source_options() -> list[dict[str, str | bool | None]]:
|
||||
"isLocked": _get_reason("zlib") is not None,
|
||||
"disabledReason": _get_reason("zlib"),
|
||||
},
|
||||
{
|
||||
"id": "oceanofpdf",
|
||||
"label": "OceanofPDF",
|
||||
"description": "Searches and downloads on its own, so its position does not matter",
|
||||
"isLocked": _get_reason("oceanofpdf") is not None,
|
||||
"disabledReason": _get_reason("oceanofpdf"),
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
@@ -1604,6 +1611,7 @@ def _get_slow_source_defaults() -> list[dict[str, str | bool]]:
|
||||
{"id": "aa-slow-wait", "enabled": True},
|
||||
{"id": "welib", "enabled": _LEGACY_ALLOW_USE_WELIB},
|
||||
{"id": "zlib", "enabled": True},
|
||||
{"id": "oceanofpdf", "enabled": True},
|
||||
]
|
||||
|
||||
|
||||
@@ -1833,6 +1841,7 @@ def _on_save_mirrors(values: dict[str, Any]) -> dict[str, Any]:
|
||||
"LIBGEN_MIRROR_URLS",
|
||||
"ZLIB_MIRROR_URLS",
|
||||
"WELIB_MIRROR_URLS",
|
||||
"OCEANOFPDF_MIRROR_URLS",
|
||||
}
|
||||
|
||||
for key in mirror_list_keys:
|
||||
@@ -1918,6 +1927,17 @@ def mirror_settings() -> list[SettingsField]:
|
||||
description="Only the first mirror in the list is used.",
|
||||
placeholder="https://your-welib-mirror.example",
|
||||
),
|
||||
# === OCEANOFPDF ===
|
||||
TagListField(
|
||||
key="OCEANOFPDF_MIRROR_URLS",
|
||||
label="OceanofPDF",
|
||||
description=(
|
||||
"Searches and downloads directly, without Anna's Archive. Only the first "
|
||||
"mirror in the list is used. The site is behind Cloudflare, so the bypass "
|
||||
"must be enabled."
|
||||
),
|
||||
placeholder="https://oceanofpdf.com",
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
|
||||
@@ -36,6 +36,7 @@ _DOWNLOAD_SOURCE_MIRROR_LABELS = {
|
||||
"libgen": "LibGen",
|
||||
"zlib": "Z-Library",
|
||||
"welib": "Welib",
|
||||
"oceanofpdf": "OceanofPDF",
|
||||
}
|
||||
|
||||
|
||||
@@ -198,6 +199,22 @@ def has_welib_mirror_configuration() -> bool:
|
||||
return bool(get_welib_mirrors())
|
||||
|
||||
|
||||
def get_oceanofpdf_mirrors() -> list[str]:
|
||||
"""Get user-configured OceanofPDF mirrors, in the order they were entered."""
|
||||
return _normalize_configured_urls(_get_config().get("OCEANOFPDF_MIRROR_URLS", None))
|
||||
|
||||
|
||||
def get_oceanofpdf_base_url() -> str | None:
|
||||
"""Get the OceanofPDF mirror in use. Only the first configured one is used."""
|
||||
mirrors = get_oceanofpdf_mirrors()
|
||||
return mirrors[0] if mirrors else None
|
||||
|
||||
|
||||
def has_oceanofpdf_mirror_configuration() -> bool:
|
||||
"""Return True when at least one OceanofPDF mirror URL is configured."""
|
||||
return bool(get_oceanofpdf_mirrors())
|
||||
|
||||
|
||||
def has_download_source_mirror_configuration(source_id: str) -> bool:
|
||||
"""Return True when the requested direct-download source has mirror config."""
|
||||
if source_id in {"aa-fast", "aa-slow", "aa-slow-nowait", "aa-slow-wait"}:
|
||||
@@ -208,6 +225,8 @@ def has_download_source_mirror_configuration(source_id: str) -> bool:
|
||||
return has_zlib_mirror_configuration()
|
||||
if source_id == "welib":
|
||||
return has_welib_mirror_configuration()
|
||||
if source_id == "oceanofpdf":
|
||||
return has_oceanofpdf_mirror_configuration()
|
||||
return False
|
||||
|
||||
|
||||
|
||||
@@ -897,6 +897,7 @@ def _get_env_value_for_field(field: FieldBase) -> tuple[bool, object | None]:
|
||||
"LIBGEN_MIRROR_URLS",
|
||||
"ZLIB_MIRROR_URLS",
|
||||
"WELIB_MIRROR_URLS",
|
||||
"OCEANOFPDF_MIRROR_URLS",
|
||||
} and isinstance(parsed, list):
|
||||
parsed = _normalize_mirror_env_urls(parsed)
|
||||
if field.key == "LOG_LEVEL" and isinstance(parsed, str):
|
||||
|
||||
@@ -421,9 +421,10 @@ def _get_source_priority() -> list[SourcePriorityEntry]:
|
||||
):
|
||||
source["enabled"] = False
|
||||
|
||||
# OceanofPDF shares the list for its toggle but is its own provider, not an MD5 mirror.
|
||||
slow_sources = _parse_source_priority_entries(
|
||||
config.get("SOURCE_PRIORITY"),
|
||||
excluded_ids={"aa-fast", "libgen"},
|
||||
excluded_ids={"aa-fast", "libgen", "oceanofpdf"},
|
||||
)
|
||||
for source in slow_sources:
|
||||
if not mirrors.has_download_source_mirror_configuration(source["id"]):
|
||||
|
||||
@@ -0,0 +1,480 @@
|
||||
"""OceanofPDF search and download.
|
||||
|
||||
Unlike the Z-Library, LibGen and Welib sources, which only download files Anna's Archive
|
||||
found, OceanofPDF searches and downloads on its own. The whole site sits behind Cloudflare,
|
||||
so every page goes through `html_get_page` and its bypasser handoff.
|
||||
|
||||
The download is three hops:
|
||||
|
||||
1. The book page carries one POST form per format, targeting ``Fetching_Resource.php``
|
||||
with hidden ``id`` (a file server) and ``filename`` fields.
|
||||
2. That endpoint answers with an interstitial page whose ``<meta http-equiv="refresh">``
|
||||
points at a signed, expiring file URL on a separate file host.
|
||||
3. The file URL serves the book itself.
|
||||
"""
|
||||
|
||||
import re
|
||||
from typing import TYPE_CHECKING
|
||||
from urllib.parse import quote_plus, urljoin, urlparse
|
||||
|
||||
import requests
|
||||
from bs4 import BeautifulSoup, Tag
|
||||
|
||||
from shelfmark.bypass.challenge import challenge_marker
|
||||
from shelfmark.core import search_deadline
|
||||
from shelfmark.core.config import config
|
||||
from shelfmark.core.logger import setup_logger
|
||||
from shelfmark.core.models import SearchFilters
|
||||
from shelfmark.download import http as downloader
|
||||
from shelfmark.download import network
|
||||
from shelfmark.release_sources.direct_download.common import (
|
||||
MIN_VALID_FILE_SIZE,
|
||||
DirectDownloadUnavailableError,
|
||||
ParsedSearchResult,
|
||||
get_attr,
|
||||
html_response_text,
|
||||
normalize_size,
|
||||
parse_search_page,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
from threading import Event
|
||||
|
||||
from shelfmark.core.search_plan import ReleaseSearchPlan
|
||||
from shelfmark.metadata_providers import BookMetadata
|
||||
from shelfmark.release_sources import BrowseRecord
|
||||
|
||||
logger = setup_logger(__name__)
|
||||
|
||||
PROVIDER_ID = "oceanofpdf"
|
||||
|
||||
_FORMAT_TAG_PATTERN = re.compile(r"\[([A-Za-z0-9]{2,5})\]")
|
||||
# Result summaries read "[PDF] [EPUB] Dune Download by Frank Herbert. Download Dune by...".
|
||||
# Anchored on the following "Download" because author names carry their own full stops
|
||||
# ("Anissa M. Bouziane").
|
||||
_AUTHOR_PATTERN = re.compile(r"\bDownload by (.+?)\.\s+Download\b")
|
||||
_META_REFRESH_PATTERN = re.compile(r"^\s*(\d+)?\s*;?\s*url\s*=\s*['\"]?([^'\"]+)", re.IGNORECASE)
|
||||
_FILE_SIZE_PATTERN = re.compile(
|
||||
r"\b([A-Z0-9]{2,5}) File Size:\s*([\d.,]+\s*[KMGT]B)", re.IGNORECASE
|
||||
)
|
||||
_FETCH_FORM_MARKER = "fetching_resource"
|
||||
# Every page of the site's Genesis theme has this skip link; a challenge page does not.
|
||||
_SITE_PAGE_MARKER = "genesis-content"
|
||||
_HTML_SIGNATURES = (b"<!doctype", b"<html")
|
||||
# The interstitial asks for a few seconds before redirecting. Honour it, but never let a
|
||||
# changed page park a download for minutes.
|
||||
_MAX_REDIRECT_DELAY_SECONDS = 10
|
||||
_POST_TIMEOUT = (10, 30)
|
||||
_HTTP_FORBIDDEN = 403
|
||||
_HTTP_SERVICE_UNAVAILABLE = 503
|
||||
|
||||
|
||||
def _base_url() -> str | None:
|
||||
from shelfmark.core import mirrors
|
||||
|
||||
return mirrors.get_oceanofpdf_base_url()
|
||||
|
||||
|
||||
def _is_switched_on() -> bool:
|
||||
"""Whether OceanofPDF's toggle in Download Sources -> Slow downloads is on.
|
||||
|
||||
A saved list with no OceanofPDF entry was saved before this provider existed. The
|
||||
settings page shows a missing option as off, so it counts as off here too - otherwise
|
||||
the page would show it off while searches still used it. With nothing saved at all,
|
||||
the field's default applies, and that has it on.
|
||||
"""
|
||||
entries = config.get("SOURCE_PRIORITY")
|
||||
if not isinstance(entries, list):
|
||||
return True
|
||||
for entry in entries:
|
||||
if isinstance(entry, dict) and entry.get("id") == PROVIDER_ID:
|
||||
return bool(entry.get("enabled", True))
|
||||
return False
|
||||
|
||||
|
||||
def _search_url(base_url: str, query: str) -> str:
|
||||
return f"{base_url}/?s={quote_plus(query)}"
|
||||
|
||||
|
||||
def _image_src(article: Tag) -> str | None:
|
||||
"""Return the cover URL, skipping the lazy-load SVG placeholder in ``src``."""
|
||||
for img in article.find_all("img"):
|
||||
if not isinstance(img, Tag):
|
||||
continue
|
||||
for attr in ("data-src", "src"):
|
||||
value = get_attr(img, attr)
|
||||
if value and not value.startswith("data:"):
|
||||
return value
|
||||
return None
|
||||
|
||||
|
||||
def _extract_search_result(article: Tag) -> ParsedSearchResult | None:
|
||||
"""Extract one OceanofPDF search result for the shared parser."""
|
||||
link = article.select_one("h2.entry-title a[href]")
|
||||
if not isinstance(link, Tag):
|
||||
return None
|
||||
book_url = get_attr(link, "href")
|
||||
title = link.get_text(" ", strip=True)
|
||||
if not book_url or not title:
|
||||
return None
|
||||
|
||||
label = get_attr(article, "aria-label") or ""
|
||||
formats = tuple(dict.fromkeys(tag.lower() for tag in _FORMAT_TAG_PATTERN.findall(label)))
|
||||
|
||||
summary = article.select_one(".entry-content, .entry-summary")
|
||||
summary_text = summary.get_text(" ", strip=True) if isinstance(summary, Tag) else ""
|
||||
author_match = _AUTHOR_PATTERN.search(summary_text)
|
||||
|
||||
return ParsedSearchResult(
|
||||
key=book_url,
|
||||
title=title,
|
||||
formats=formats,
|
||||
author=author_match.group(1).strip() if author_match else None,
|
||||
preview=_image_src(article),
|
||||
content="book (unknown)",
|
||||
source_url=book_url,
|
||||
)
|
||||
|
||||
|
||||
def parse_search_results(html: str, filters: SearchFilters | None) -> list[BrowseRecord]:
|
||||
"""Parse an OceanofPDF search page into one record per result and format."""
|
||||
return parse_search_page(
|
||||
html,
|
||||
filters,
|
||||
provider_id=PROVIDER_ID,
|
||||
item_selector="article",
|
||||
extract_item=_extract_search_result,
|
||||
)
|
||||
|
||||
|
||||
def _fetch_forms(soup: BeautifulSoup) -> list[Tag]:
|
||||
return [
|
||||
form
|
||||
for form in soup.find_all("form")
|
||||
if isinstance(form, Tag) and _FETCH_FORM_MARKER in (get_attr(form, "action") or "").lower()
|
||||
]
|
||||
|
||||
|
||||
def _form_fields(form: Tag) -> dict[str, str]:
|
||||
fields: dict[str, str] = {}
|
||||
for field in form.find_all("input"):
|
||||
if not isinstance(field, Tag):
|
||||
continue
|
||||
name = get_attr(field, "name")
|
||||
if name:
|
||||
fields[name] = get_attr(field, "value") or ""
|
||||
return fields
|
||||
|
||||
|
||||
def pick_download_form(
|
||||
soup: BeautifulSoup, book_format: str | None
|
||||
) -> tuple[str, dict[str, str]] | None:
|
||||
"""Return the form action and fields that fetch ``book_format``.
|
||||
|
||||
Without a requested format the first form wins, which is the page's own order.
|
||||
"""
|
||||
candidates = [
|
||||
(get_attr(form, "action") or "", _form_fields(form)) for form in _fetch_forms(soup)
|
||||
]
|
||||
if not candidates:
|
||||
return None
|
||||
if not book_format:
|
||||
return candidates[0]
|
||||
suffix = f".{book_format.lower()}"
|
||||
for action, fields in candidates:
|
||||
if fields.get("filename", "").lower().endswith(suffix):
|
||||
return action, fields
|
||||
return None
|
||||
|
||||
|
||||
def file_size_from_page(soup: BeautifulSoup, book_format: str | None) -> str | None:
|
||||
"""Read the "EPUB File Size: 952 KB" line for the requested format, if present."""
|
||||
if not book_format:
|
||||
return None
|
||||
for label, size in _FILE_SIZE_PATTERN.findall(soup.get_text(" ", strip=True)):
|
||||
if label.lower() == book_format.lower():
|
||||
return normalize_size(size)
|
||||
return None
|
||||
|
||||
|
||||
def parse_file_redirect(html: str, page_url: str) -> tuple[str, int] | None:
|
||||
"""Return the signed file URL and redirect delay from the interstitial page."""
|
||||
soup = BeautifulSoup(html, "html.parser")
|
||||
for meta in soup.find_all("meta"):
|
||||
if not isinstance(meta, Tag):
|
||||
continue
|
||||
if (get_attr(meta, "http-equiv") or "").lower() != "refresh":
|
||||
continue
|
||||
match = _META_REFRESH_PATTERN.match(get_attr(meta, "content") or "")
|
||||
if match:
|
||||
return urljoin(page_url, match.group(2).strip()), int(match.group(1) or 0)
|
||||
|
||||
# Fallback for a page that drops the meta refresh for a plain link.
|
||||
for anchor in soup.find_all("a", href=True):
|
||||
href = get_attr(anchor, "href") or ""
|
||||
if urlparse(href).path.lower().endswith((".epub", ".pdf")):
|
||||
return urljoin(page_url, href), 0
|
||||
return None
|
||||
|
||||
|
||||
def _post_fetch_form(action_url: str, fields: dict[str, str], referer: str) -> requests.Response:
|
||||
"""POST the download form, presenting any clearance the bypasser won for this host."""
|
||||
hostname = urlparse(action_url).hostname or ""
|
||||
# Not DOWNLOAD_HEADERS wholesale: its Accept-Encoding offers "br", Cloudflare answers
|
||||
# in Brotli when offered, and requests cannot decode it without the brotli package -
|
||||
# the page came back as compressed bytes with no link to find. Leaving Accept-Encoding
|
||||
# to requests offers only what it can decode.
|
||||
headers = {
|
||||
"User-Agent": downloader.get_cf_user_agent_for_domain(hostname)
|
||||
or downloader.DOWNLOAD_HEADERS["User-Agent"],
|
||||
"Accept-Language": downloader.DOWNLOAD_HEADERS["Accept-Language"],
|
||||
"Referer": referer,
|
||||
"Origin": f"{urlparse(referer).scheme}://{urlparse(referer).netloc}",
|
||||
}
|
||||
return requests.post(
|
||||
action_url,
|
||||
data=fields,
|
||||
headers=headers,
|
||||
cookies=downloader.get_cf_cookies_for_domain(hostname),
|
||||
timeout=_POST_TIMEOUT,
|
||||
proxies=network.get_proxies(action_url),
|
||||
verify=network.get_ssl_verify(action_url),
|
||||
)
|
||||
|
||||
|
||||
def _log_unresolved_fetch_page(response: requests.Response) -> None:
|
||||
"""Record what the download form returned when no file link could be read off it.
|
||||
|
||||
The response body is not in the debug bundle, so without this a changed page, a
|
||||
challenge and an undecoded body all look the same: "did not return a file link".
|
||||
Diagnostics must never fail a download, so this swallows its own errors.
|
||||
"""
|
||||
try:
|
||||
text = response.text or ""
|
||||
title_match = re.search(
|
||||
r"<title[^>]*>(.*?)</title>", text[:8000], re.IGNORECASE | re.DOTALL
|
||||
)
|
||||
logger.warning(
|
||||
"OceanofPDF download page had no file link: status=%s url=%s bytes=%d "
|
||||
"content-type=%r content-encoding=%r title=%r meta_refresh=%s challenge_marker=%r",
|
||||
response.status_code,
|
||||
response.url,
|
||||
len(text),
|
||||
response.headers.get("Content-Type"),
|
||||
response.headers.get("Content-Encoding"),
|
||||
" ".join(title_match.group(1).split())[:120] if title_match else None,
|
||||
"http-equiv" in text.lower() and "refresh" in text.lower(),
|
||||
challenge_marker(text),
|
||||
)
|
||||
except Exception:
|
||||
logger.debug("Could not describe the OceanofPDF download page", exc_info=True)
|
||||
|
||||
|
||||
def _is_challenge(response: requests.Response) -> bool:
|
||||
return response.status_code in (
|
||||
_HTTP_FORBIDDEN,
|
||||
_HTTP_SERVICE_UNAVAILABLE,
|
||||
) and bool(challenge_marker(response.text))
|
||||
|
||||
|
||||
class OceanofPDFProvider:
|
||||
"""OceanofPDF provider: searches the site and downloads from its file host."""
|
||||
|
||||
id = PROVIDER_ID
|
||||
display_name = "OceanofPDF"
|
||||
|
||||
def is_enabled(self) -> bool:
|
||||
# Every page is behind Cloudflare, so without the bypass a search can only fail.
|
||||
return (
|
||||
_base_url() is not None
|
||||
and bool(config.get("USE_CF_BYPASS", True))
|
||||
and _is_switched_on()
|
||||
)
|
||||
|
||||
def handles(self, url: str) -> bool:
|
||||
base_url = _base_url()
|
||||
if not base_url:
|
||||
return False
|
||||
hostname = (urlparse(url).hostname or "").lower().rstrip(".")
|
||||
return bool(hostname) and hostname == (urlparse(base_url).hostname or "").lower()
|
||||
|
||||
# --- Search ---
|
||||
|
||||
def _fetch_search_page(self, base_url: str, query: str) -> str:
|
||||
url = _search_url(base_url, query)
|
||||
selector = network.AAMirrorSelector()
|
||||
response = downloader.html_get_page(url, selector=selector, allow_bypasser_fallback=True)
|
||||
html = html_response_text(response)
|
||||
if not html:
|
||||
detail = getattr(selector, "last_failure", None) or "The site could not be reached."
|
||||
msg = f"Unable to reach OceanofPDF. {detail}"
|
||||
raise DirectDownloadUnavailableError(msg)
|
||||
# Checked against the theme's own markup, not just the marker: a protected site's
|
||||
# real pages can carry the protection's markers too (see #1289/#1292 for AA).
|
||||
if challenge_marker(html) and _SITE_PAGE_MARKER not in html:
|
||||
msg = (
|
||||
"OceanofPDF answered with a Cloudflare challenge that was not cleared. "
|
||||
"Check that the bypass is enabled and working."
|
||||
)
|
||||
raise DirectDownloadUnavailableError(msg)
|
||||
return html
|
||||
|
||||
def _queries(self, plan: ReleaseSearchPlan) -> list[str]:
|
||||
"""Title + author first, then the bare title as a looser fallback.
|
||||
|
||||
WordPress search ANDs every word across the post, and each post names its author,
|
||||
so adding the author narrows "Dune" from forty pages to the right book.
|
||||
"""
|
||||
if plan.manual_query:
|
||||
return [plan.manual_query]
|
||||
queries: list[str] = []
|
||||
for variant in plan.grouped_title_variants:
|
||||
for query in (f"{variant.title} {plan.author}".strip(), variant.title.strip()):
|
||||
if query and query not in queries:
|
||||
queries.append(query)
|
||||
return queries
|
||||
|
||||
def search(
|
||||
self,
|
||||
book: BookMetadata,
|
||||
plan: ReleaseSearchPlan,
|
||||
*,
|
||||
expand_search: bool = False,
|
||||
content_type: str = "ebook",
|
||||
) -> list[BrowseRecord]:
|
||||
"""Search OceanofPDF, stopping at the first query that finds anything."""
|
||||
del book, expand_search, content_type
|
||||
base_url = _base_url()
|
||||
if not base_url:
|
||||
return []
|
||||
|
||||
if plan.source_filters is not None:
|
||||
filters = plan.source_filters
|
||||
queries = [plan.manual_query] if plan.manual_query else []
|
||||
else:
|
||||
filters = SearchFilters(lang=plan.languages)
|
||||
queries = self._queries(plan)
|
||||
|
||||
for query in queries:
|
||||
if search_deadline.expired():
|
||||
logger.info("Release search budget spent; skipping remaining OceanofPDF queries")
|
||||
break
|
||||
logger.debug("Searching OceanofPDF: %s", query)
|
||||
records = parse_search_results(self._fetch_search_page(base_url, query), filters)
|
||||
if records:
|
||||
logger.info("Found %s OceanofPDF releases for %r", len(records), query)
|
||||
return records
|
||||
return []
|
||||
|
||||
# --- Download ---
|
||||
|
||||
def _resolve_file_url(
|
||||
self,
|
||||
action_url: str,
|
||||
fields: dict[str, str],
|
||||
book_url: str,
|
||||
cancel_flag: Event | None,
|
||||
) -> tuple[str, int] | None:
|
||||
"""POST the download form and read the file URL off the interstitial."""
|
||||
response = _post_fetch_form(action_url, fields, book_url)
|
||||
if _is_challenge(response):
|
||||
# The clearance expired between the book page and this POST. Bypassers only
|
||||
# replay GETs, so re-solve on the book page and post again with fresh cookies.
|
||||
logger.info("OceanofPDF download form was challenged; refreshing clearance")
|
||||
downloader.html_get_page(book_url, cancel_flag=cancel_flag, use_bypasser=True)
|
||||
response = _post_fetch_form(action_url, fields, book_url)
|
||||
response.raise_for_status()
|
||||
resolved = parse_file_redirect(response.text, response.url or action_url)
|
||||
if resolved is None:
|
||||
_log_unresolved_fetch_page(response)
|
||||
return resolved
|
||||
|
||||
def download(
|
||||
self,
|
||||
book_info: BrowseRecord,
|
||||
book_path: Path,
|
||||
progress_callback: Callable[[float], None] | None,
|
||||
cancel_flag: Event | None,
|
||||
status_callback: Callable[[str, str | None], None] | None,
|
||||
) -> str | None:
|
||||
"""Download the requested format of an OceanofPDF book. Returns the file URL."""
|
||||
book_url = book_info.source_url
|
||||
if not book_url:
|
||||
msg = f"OceanofPDF record {book_info.id!r} has no book page URL"
|
||||
raise RuntimeError(msg)
|
||||
|
||||
def status(message: str) -> None:
|
||||
if status_callback:
|
||||
status_callback("resolving", message)
|
||||
|
||||
try:
|
||||
status("Fetching OceanofPDF book page")
|
||||
page = html_response_text(
|
||||
downloader.html_get_page(
|
||||
book_url,
|
||||
cancel_flag=cancel_flag,
|
||||
status_callback=status_callback,
|
||||
allow_bypasser_fallback=True,
|
||||
)
|
||||
)
|
||||
if not page:
|
||||
logger.warning("OceanofPDF book page could not be fetched: %s", book_url)
|
||||
return None
|
||||
|
||||
soup = BeautifulSoup(page, "html.parser")
|
||||
form = pick_download_form(soup, book_info.format)
|
||||
if form is None:
|
||||
logger.warning(
|
||||
"No OceanofPDF download form for format %r on %s", book_info.format, book_url
|
||||
)
|
||||
return None
|
||||
action, fields = form
|
||||
action_url = urljoin(book_url, action)
|
||||
|
||||
if cancel_flag and cancel_flag.is_set():
|
||||
return None
|
||||
status("Requesting OceanofPDF download link")
|
||||
resolved = self._resolve_file_url(action_url, fields, book_url, cancel_flag)
|
||||
if resolved is None:
|
||||
logger.warning("OceanofPDF did not return a file link for %s", book_url)
|
||||
return None
|
||||
file_url, delay = resolved
|
||||
|
||||
wait = min(delay, _MAX_REDIRECT_DELAY_SECONDS)
|
||||
if wait > 0:
|
||||
status(f"OceanofPDF - Waiting {wait}s")
|
||||
if cancel_flag and cancel_flag.wait(timeout=wait):
|
||||
return None
|
||||
|
||||
logger.info("Resolved OceanofPDF file URL for %s", book_info.title)
|
||||
data = downloader.download_url(
|
||||
file_url,
|
||||
file_size_from_page(soup, book_info.format) or book_info.size or "",
|
||||
progress_callback,
|
||||
cancel_flag,
|
||||
None,
|
||||
status_callback,
|
||||
referer=action_url,
|
||||
)
|
||||
if not data:
|
||||
logger.warning("No data received from OceanofPDF for %s", book_info.title)
|
||||
return None
|
||||
|
||||
size = data.getbuffer().nbytes
|
||||
data.seek(0)
|
||||
head = data.read(64).lstrip().lower()
|
||||
if size < MIN_VALID_FILE_SIZE or head.startswith(_HTML_SIGNATURES):
|
||||
logger.warning(
|
||||
"OceanofPDF returned an HTML page or a stub (%s bytes), not a book", size
|
||||
)
|
||||
return None
|
||||
|
||||
with book_path.open("wb") as f:
|
||||
f.write(data.getbuffer())
|
||||
except (requests.exceptions.RequestException, OSError, ValueError) as e:
|
||||
logger.warning("OceanofPDF download failed for %s: %s", book_url, e)
|
||||
return None
|
||||
else:
|
||||
return file_url
|
||||
@@ -5,6 +5,7 @@ from typing import TYPE_CHECKING
|
||||
|
||||
from shelfmark.core.config import config
|
||||
from shelfmark.release_sources.direct_download.annas_archive import AnnasArchiveProvider
|
||||
from shelfmark.release_sources.direct_download.oceanofpdf import OceanofPDFProvider
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Sequence
|
||||
@@ -13,7 +14,7 @@ if TYPE_CHECKING:
|
||||
from shelfmark.release_sources.direct_download.common import DirectDownloadProvider
|
||||
|
||||
|
||||
PROVIDER_TYPES = (AnnasArchiveProvider,)
|
||||
PROVIDER_TYPES = (AnnasArchiveProvider, OceanofPDFProvider)
|
||||
_AA_MD5_PATTERN = re.compile(r"^[0-9a-f]{32}$", re.IGNORECASE)
|
||||
|
||||
|
||||
@@ -38,8 +39,8 @@ def get_unavailable_reason(
|
||||
return "Direct Download is disabled. Enable the source in Settings."
|
||||
if not enabled_providers(providers):
|
||||
return (
|
||||
"Direct Download is not configured. Add at least one Anna's Archive mirror URL in "
|
||||
"Settings. Z-Library, LibGen, and Welib mirrors only download results found through "
|
||||
"Direct Download is not configured. Add at least one Anna's Archive or OceanofPDF mirror URL in "
|
||||
"Settings. Z-Library, LibGen, Welib, mirrors only download results found through "
|
||||
"Anna's Archive."
|
||||
)
|
||||
return None
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
"""A rendered page carrying Cloudflare's bot-detection beacon is not a challenge.
|
||||
|
||||
Cloudflare injects an iframe loading /cdn-cgi/challenge-platform/scripts/jsd/main.js into
|
||||
every page a protected site serves. The raw HTML has no such reference; the browser adds
|
||||
it after load. FlareSolverr returns the rendered DOM, so every successful OceanofPDF solve
|
||||
came back carrying "/cdn-cgi/challenge-platform" and was rejected as an unsolved
|
||||
challenge - 26 KB results pages, retried five times each, then reported as unreachable.
|
||||
"""
|
||||
|
||||
from shelfmark.bypass.challenge import challenge_marker
|
||||
|
||||
# Shape of a rendered OceanofPDF search page: the site's own markup plus the beacon.
|
||||
RENDERED_PAGE_WITH_BEACON = (
|
||||
"<html><head><title>You searched for dune - OceanofPDF</title></head><body>"
|
||||
'<a href="#genesis-content">Skip to main content</a>'
|
||||
'<main id="genesis-content"><article aria-label="[PDF] [EPUB] Dune Download">'
|
||||
'<h2 class="entry-title"><a href="/authors/frank-herbert/dune/">Dune</a></h2>'
|
||||
"</article></main>"
|
||||
"<script>(function(){var a=document.createElement('script');"
|
||||
"a.src='/cdn-cgi/challenge-platform/scripts/jsd/main.js';"
|
||||
"document.getElementsByTagName('head')[0].appendChild(a);})();</script>"
|
||||
'<iframe height="1" width="1" style="visibility:hidden"></iframe>'
|
||||
"</body></html>"
|
||||
)
|
||||
|
||||
# Cloudflare's managed challenge, trimmed to its markers. It can carry the beacon too.
|
||||
MANAGED_CHALLENGE = (
|
||||
"<html><head><title>Just a moment...</title></head><body>"
|
||||
"<script>window._cf_chl_opt={cType:'managed'};</script>"
|
||||
'<script src="/cdn-cgi/challenge-platform/h/b/orchestrate/chl_page/v1"></script>'
|
||||
'<script src="/cdn-cgi/challenge-platform/scripts/jsd/main.js"></script>'
|
||||
"</body></html>"
|
||||
)
|
||||
|
||||
|
||||
def test_beacon_on_a_rendered_page_is_not_a_challenge():
|
||||
assert challenge_marker(RENDERED_PAGE_WITH_BEACON) is None
|
||||
|
||||
|
||||
def test_beacon_path_in_any_case_is_ignored():
|
||||
assert challenge_marker(RENDERED_PAGE_WITH_BEACON.replace("scripts/jsd", "Scripts/JSD")) is None
|
||||
|
||||
|
||||
def test_managed_challenge_is_still_caught_with_the_beacon_present():
|
||||
assert challenge_marker(MANAGED_CHALLENGE) is not None
|
||||
|
||||
|
||||
def test_challenge_script_path_alone_is_still_caught():
|
||||
page = '<html><script src="/cdn-cgi/challenge-platform/h/g/orchestrate/x"></script></html>'
|
||||
|
||||
assert challenge_marker(page) == "/cdn-cgi/challenge-platform"
|
||||
@@ -0,0 +1,579 @@
|
||||
"""OceanofPDF provider: search parsing, download resolution and registry wiring.
|
||||
|
||||
The fixtures mirror the live site's markup (Genesis WordPress theme) as of 2026-10:
|
||||
search results are <article> posts, the book page carries one POST form per format, and
|
||||
the form's response is an interstitial with a meta refresh to a signed file URL.
|
||||
"""
|
||||
|
||||
from io import BytesIO
|
||||
from threading import Event
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
from shelfmark.core import mirrors
|
||||
from shelfmark.core.models import SearchFilters
|
||||
from shelfmark.release_sources import BrowseRecord
|
||||
from shelfmark.release_sources.direct_download import oceanofpdf as op
|
||||
from shelfmark.release_sources.direct_download import registry
|
||||
from shelfmark.release_sources.direct_download.common import DirectDownloadUnavailableError
|
||||
|
||||
BASE = "https://ocean.example"
|
||||
DUNE_URL = f"{BASE}/authors/frank-herbert/pdf-epub-dune-download-85663993149/"
|
||||
SONG_URL = f"{BASE}/authors/anissa-m-bouziane/pdf-epub-dune-song-download/"
|
||||
ART_URL = f"{BASE}/authors/alejandro-jodorowsky/pdf-dune-art-book-download/"
|
||||
FETCH_URL = f"{BASE}/Fetching_Resource.php"
|
||||
FILE_URL = "https://files.example/OceanofPDF.com/Dune_1_-_Frank_Herbert.epub?md5=abc&expires=1"
|
||||
|
||||
|
||||
def _article(url: str, title: str, label: str, summary: str, cover: str = "") -> str:
|
||||
return f"""
|
||||
<article class="post type-post entry" aria-label="{label}">
|
||||
<header class="entry-header">
|
||||
<a class="entry-image-link" href="{url}"><img src="data:image/svg+xml,placeholder"
|
||||
data-src="{cover}"></a>
|
||||
<h2 class="entry-title"><a class="entry-title-link" href="{url}">{title}</a></h2>
|
||||
<p class="entry-meta"><time class="entry-time">September 27, 2019</time></p>
|
||||
</header>
|
||||
<div class="entry-content"><p>{summary} [Read more…]</p></div>
|
||||
</article>"""
|
||||
|
||||
|
||||
def _page(body: str) -> str:
|
||||
return f"""<html><head><title>OceanofPDF</title></head><body>
|
||||
<a class="genesis-skip-link" href="#genesis-content">Skip to main content</a>
|
||||
<main id="genesis-content">{body}</main></body></html>"""
|
||||
|
||||
|
||||
SEARCH_PAGE = _page(
|
||||
_article(
|
||||
DUNE_URL,
|
||||
"Dune",
|
||||
"[PDF] [EPUB] Dune Download",
|
||||
"[PDF] [EPUB] Dune Download by Frank Herbert. Download Dune by Frank Herbert in PDF "
|
||||
"EPUB format complete free.",
|
||||
cover="https://media.example/dune.jpg",
|
||||
)
|
||||
+ _article(
|
||||
SONG_URL,
|
||||
"Dune Song",
|
||||
"[PDF] [EPUB] Dune Song Download",
|
||||
"[PDF] [EPUB] Dune Song Download by Anissa M. Bouziane. Download Dune Song by Anissa "
|
||||
"M. Bouziane in PDF EPUB format complete free.",
|
||||
)
|
||||
+ _article(
|
||||
ART_URL,
|
||||
"Dune Art Book",
|
||||
"[PDF] Dune Art Book Download",
|
||||
"[PDF] Dune Art Book Download by Alejandro Jodorowsky. Download Dune Art Book in PDF "
|
||||
"format complete free.",
|
||||
)
|
||||
)
|
||||
|
||||
EMPTY_SEARCH_PAGE = _page("<h1>Nothing Found</h1>")
|
||||
|
||||
BOOK_PAGE = _page(
|
||||
"""
|
||||
<div class="entry-content">
|
||||
<ul><li>Full Book Name: Dune</li><li>Author Name: Frank Herbert</li>
|
||||
<li>PDF File Size: 2.1 MB</li><li>EPUB File Size: 952 KB</li></ul>
|
||||
<form action="https://ocean.example/" method="get"><input name="s" type="search"></form>
|
||||
<form action="https://ocean.example/Fetching_Resource.php" method="post" target="_blank">
|
||||
<input type="hidden" name="id" value="srv3">
|
||||
<input type="hidden" name="filename" value="Dune_1_-_Frank_Herbert.pdf">
|
||||
</form>
|
||||
<form action="https://ocean.example/Fetching_Resource.php" method="post" target="_blank">
|
||||
<input type="hidden" name="id" value="srv3">
|
||||
<input type="hidden" name="filename" value="Dune_1_-_Frank_Herbert.epub">
|
||||
</form>
|
||||
</div>"""
|
||||
)
|
||||
|
||||
INTERSTITIAL_PAGE = _page(
|
||||
f'<meta http-equiv="refresh" content="0; url={FILE_URL}">'
|
||||
"<p>Don't worry! Your download will still start.</p>"
|
||||
)
|
||||
|
||||
CHALLENGE_PAGE = (
|
||||
"<html><head><title>Just a moment...</title></head><body>"
|
||||
'<div id="challenge-form"></div><script src="/cdn-cgi/challenge-platform/x.js"></script>'
|
||||
"</body></html>"
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def configured(monkeypatch):
|
||||
"""Configure an OceanofPDF mirror and keep every request off the network.
|
||||
|
||||
Returns the settings the provider reads, so a test can flip the toggle or the bypass.
|
||||
"""
|
||||
settings = {
|
||||
"USE_CF_BYPASS": True,
|
||||
"SOURCE_PRIORITY": [{"id": "zlib", "enabled": True}, {"id": "oceanofpdf", "enabled": True}],
|
||||
}
|
||||
monkeypatch.setattr(
|
||||
op, "config", SimpleNamespace(get=lambda key, default=None: settings.get(key, default))
|
||||
)
|
||||
monkeypatch.setattr(mirrors, "_get_config", lambda: {"OCEANOFPDF_MIRROR_URLS": [BASE]})
|
||||
monkeypatch.setattr(op.network, "AAMirrorSelector", lambda: SimpleNamespace(last_failure=None))
|
||||
monkeypatch.setattr(op.network, "get_proxies", lambda _url: {})
|
||||
monkeypatch.setattr(op.network, "get_ssl_verify", lambda _url: True)
|
||||
monkeypatch.setattr(op.downloader, "get_cf_cookies_for_domain", lambda _host: {})
|
||||
monkeypatch.setattr(op.downloader, "get_cf_user_agent_for_domain", lambda _host: None)
|
||||
return settings
|
||||
|
||||
|
||||
def _plan(**overrides):
|
||||
values = {
|
||||
"manual_query": None,
|
||||
"source_filters": None,
|
||||
"languages": None,
|
||||
"author": "Frank Herbert",
|
||||
"grouped_title_variants": [SimpleNamespace(title="Dune")],
|
||||
}
|
||||
values.update(overrides)
|
||||
return SimpleNamespace(**values)
|
||||
|
||||
|
||||
def _serve_pages(monkeypatch, pages: dict[str, str]) -> list[str]:
|
||||
"""Answer html_get_page from ``pages`` by URL, recording each URL requested."""
|
||||
requested: list[str] = []
|
||||
|
||||
def fake_html_get_page(url, *_args, **_kwargs):
|
||||
requested.append(url)
|
||||
return pages.get(url, "")
|
||||
|
||||
monkeypatch.setattr(op.downloader, "html_get_page", fake_html_get_page)
|
||||
return requested
|
||||
|
||||
|
||||
# --- Search parsing ---
|
||||
|
||||
|
||||
def test_search_results_split_into_one_record_per_format():
|
||||
records = op.parse_search_results(SEARCH_PAGE, SearchFilters(format=["pdf", "epub"]))
|
||||
|
||||
assert [(r.title, r.format) for r in records] == [
|
||||
("Dune", "pdf"),
|
||||
("Dune", "epub"),
|
||||
("Dune Song", "pdf"),
|
||||
("Dune Song", "epub"),
|
||||
("Dune Art Book", "pdf"),
|
||||
]
|
||||
dune_pdf, dune_epub = records[0], records[1]
|
||||
assert dune_pdf.id.startswith("oceanofpdf:")
|
||||
assert dune_pdf.id != dune_epub.id
|
||||
assert dune_pdf.source_url == DUNE_URL
|
||||
assert dune_pdf.author == "Frank Herbert"
|
||||
assert dune_pdf.preview == "https://media.example/dune.jpg"
|
||||
assert dune_pdf.source == "direct_download"
|
||||
|
||||
|
||||
def test_record_ids_are_stable_across_searches():
|
||||
first = op.parse_search_results(SEARCH_PAGE, SearchFilters())
|
||||
second = op.parse_search_results(SEARCH_PAGE, SearchFilters())
|
||||
|
||||
assert [r.id for r in first] == [r.id for r in second]
|
||||
|
||||
|
||||
def test_author_with_initials_is_not_cut_at_the_first_full_stop():
|
||||
records = op.parse_search_results(SEARCH_PAGE, SearchFilters())
|
||||
|
||||
assert {r.author for r in records if r.title == "Dune Song"} == {"Anissa M. Bouziane"}
|
||||
|
||||
|
||||
def test_unfiltered_search_keeps_only_the_configured_supported_formats(monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
"shelfmark.release_sources.direct_download.common.get_supported_formats",
|
||||
lambda: ["epub", "mobi"],
|
||||
)
|
||||
|
||||
records = op.parse_search_results(SEARCH_PAGE, SearchFilters())
|
||||
|
||||
assert {r.format for r in records} == {"epub"}
|
||||
|
||||
|
||||
def test_format_filter_drops_other_formats():
|
||||
records = op.parse_search_results(SEARCH_PAGE, SearchFilters(format=["epub"]))
|
||||
|
||||
assert {r.format for r in records} == {"epub"}
|
||||
assert "Dune Art Book" not in {r.title for r in records}
|
||||
|
||||
|
||||
def test_results_without_a_language_survive_a_language_filter():
|
||||
records = op.parse_search_results(SEARCH_PAGE, SearchFilters(lang=["en"]))
|
||||
|
||||
assert records
|
||||
|
||||
|
||||
def test_empty_search_page_parses_to_no_records():
|
||||
assert op.parse_search_results(EMPTY_SEARCH_PAGE, SearchFilters()) == []
|
||||
|
||||
|
||||
# --- Book page and interstitial parsing ---
|
||||
|
||||
|
||||
def test_download_form_is_picked_by_format():
|
||||
soup = BeautifulSoup(BOOK_PAGE, "html.parser")
|
||||
|
||||
action, fields = op.pick_download_form(soup, "epub")
|
||||
|
||||
assert action == FETCH_URL
|
||||
assert fields == {"id": "srv3", "filename": "Dune_1_-_Frank_Herbert.epub"}
|
||||
|
||||
|
||||
def test_download_form_ignores_the_search_form_and_defaults_to_the_first():
|
||||
soup = BeautifulSoup(BOOK_PAGE, "html.parser")
|
||||
|
||||
_action, fields = op.pick_download_form(soup, None)
|
||||
|
||||
assert fields["filename"].endswith(".pdf")
|
||||
|
||||
|
||||
def test_download_form_is_none_for_a_format_the_page_lacks():
|
||||
soup = BeautifulSoup(BOOK_PAGE, "html.parser")
|
||||
|
||||
assert op.pick_download_form(soup, "mobi") is None
|
||||
|
||||
|
||||
def test_file_size_is_read_for_the_requested_format():
|
||||
soup = BeautifulSoup(BOOK_PAGE, "html.parser")
|
||||
|
||||
assert op.file_size_from_page(soup, "epub") == "952 KB"
|
||||
assert op.file_size_from_page(soup, "pdf") == "2.1 MB"
|
||||
assert op.file_size_from_page(soup, "mobi") is None
|
||||
|
||||
|
||||
def test_meta_refresh_gives_the_file_url_and_delay():
|
||||
html = f'<meta http-equiv="Refresh" content="5;URL=\'{FILE_URL}\'">'
|
||||
|
||||
assert op.parse_file_redirect(html, FETCH_URL) == (FILE_URL, 5)
|
||||
|
||||
|
||||
def test_relative_refresh_target_is_resolved_against_the_page():
|
||||
html = '<meta http-equiv="refresh" content="3; url=/files/book.epub">'
|
||||
|
||||
assert op.parse_file_redirect(html, FETCH_URL) == (f"{BASE}/files/book.epub", 3)
|
||||
|
||||
|
||||
def test_plain_file_link_is_used_when_there_is_no_meta_refresh():
|
||||
html = '<a href="https://files.example/x/book.epub?md5=1">Download</a>'
|
||||
|
||||
assert op.parse_file_redirect(html, FETCH_URL) == ("https://files.example/x/book.epub?md5=1", 0)
|
||||
|
||||
|
||||
def test_interstitial_without_a_file_link_resolves_to_none():
|
||||
assert op.parse_file_redirect("<p>Please donate</p>", FETCH_URL) is None
|
||||
|
||||
|
||||
# --- Provider search ---
|
||||
|
||||
|
||||
def test_search_queries_title_and_author_first(configured, monkeypatch):
|
||||
requested = _serve_pages(monkeypatch, {f"{BASE}/?s=Dune+Frank+Herbert": SEARCH_PAGE})
|
||||
|
||||
records = op.OceanofPDFProvider().search(SimpleNamespace(), _plan())
|
||||
|
||||
assert requested == [f"{BASE}/?s=Dune+Frank+Herbert"]
|
||||
assert records[0].title == "Dune"
|
||||
|
||||
|
||||
def test_search_falls_back_to_the_bare_title(configured, monkeypatch):
|
||||
requested = _serve_pages(
|
||||
monkeypatch,
|
||||
{f"{BASE}/?s=Dune+Frank+Herbert": EMPTY_SEARCH_PAGE, f"{BASE}/?s=Dune": SEARCH_PAGE},
|
||||
)
|
||||
|
||||
records = op.OceanofPDFProvider().search(SimpleNamespace(), _plan())
|
||||
|
||||
assert requested == [f"{BASE}/?s=Dune+Frank+Herbert", f"{BASE}/?s=Dune"]
|
||||
assert records
|
||||
|
||||
|
||||
def test_manual_search_sends_the_query_as_typed(configured, monkeypatch):
|
||||
requested = _serve_pages(monkeypatch, {f"{BASE}/?s=dune+song": SEARCH_PAGE})
|
||||
|
||||
op.OceanofPDFProvider().search(
|
||||
SimpleNamespace(), _plan(manual_query="dune song", source_filters=SearchFilters())
|
||||
)
|
||||
|
||||
assert requested == [f"{BASE}/?s=dune+song"]
|
||||
|
||||
|
||||
def test_unreachable_site_is_reported_not_treated_as_no_results(configured, monkeypatch):
|
||||
_serve_pages(monkeypatch, {})
|
||||
|
||||
with pytest.raises(DirectDownloadUnavailableError, match="Unable to reach OceanofPDF"):
|
||||
op.OceanofPDFProvider().search(SimpleNamespace(), _plan())
|
||||
|
||||
|
||||
def test_uncleared_challenge_is_reported(configured, monkeypatch):
|
||||
_serve_pages(monkeypatch, {f"{BASE}/?s=Dune+Frank+Herbert": CHALLENGE_PAGE})
|
||||
|
||||
with pytest.raises(DirectDownloadUnavailableError, match="Cloudflare challenge"):
|
||||
op.OceanofPDFProvider().search(SimpleNamespace(), _plan())
|
||||
|
||||
|
||||
def test_real_page_carrying_challenge_markers_is_not_misread(configured, monkeypatch):
|
||||
page = SEARCH_PAGE.replace("</body>", '<script src="/cdn-cgi/challenge-platform/x.js">')
|
||||
_serve_pages(monkeypatch, {f"{BASE}/?s=Dune+Frank+Herbert": page})
|
||||
|
||||
assert op.OceanofPDFProvider().search(SimpleNamespace(), _plan())
|
||||
|
||||
|
||||
def test_search_without_a_mirror_returns_nothing(monkeypatch):
|
||||
monkeypatch.setattr(mirrors, "_get_config", lambda: {})
|
||||
|
||||
assert op.OceanofPDFProvider().search(SimpleNamespace(), _plan()) == []
|
||||
|
||||
|
||||
# --- Provider download ---
|
||||
|
||||
|
||||
class _FakeResponse:
|
||||
def __init__(self, text: str, status_code: int = 200, url: str = FETCH_URL):
|
||||
self.text = text
|
||||
self.status_code = status_code
|
||||
self.url = url
|
||||
|
||||
def raise_for_status(self):
|
||||
if self.status_code >= 400:
|
||||
raise op.requests.HTTPError(f"{self.status_code} error")
|
||||
|
||||
|
||||
def _record(book_format: str = "epub") -> BrowseRecord:
|
||||
return BrowseRecord(
|
||||
id="oceanofpdf:abc",
|
||||
title="Dune",
|
||||
source="direct_download",
|
||||
format=book_format,
|
||||
source_url=DUNE_URL,
|
||||
)
|
||||
|
||||
|
||||
def _wire_download(monkeypatch, *, post_responses, payload: bytes):
|
||||
"""Serve the book page, queue POST responses and return the file as ``payload``."""
|
||||
_serve_pages(monkeypatch, {DUNE_URL: BOOK_PAGE})
|
||||
posts: list[dict] = []
|
||||
downloads: list[dict] = []
|
||||
queued = list(post_responses)
|
||||
|
||||
def fake_post(url, *, data, headers, **_kwargs):
|
||||
posts.append({"url": url, "data": data, "headers": headers})
|
||||
return queued.pop(0)
|
||||
|
||||
def fake_download_url(link, size, *_args, referer=None, **_kwargs):
|
||||
downloads.append({"link": link, "size": size, "referer": referer})
|
||||
return BytesIO(payload)
|
||||
|
||||
monkeypatch.setattr(op.requests, "post", fake_post)
|
||||
monkeypatch.setattr(op.downloader, "download_url", fake_download_url)
|
||||
return posts, downloads
|
||||
|
||||
|
||||
def test_download_posts_the_form_and_saves_the_file(configured, monkeypatch, tmp_path):
|
||||
posts, downloads = _wire_download(
|
||||
monkeypatch, post_responses=[_FakeResponse(INTERSTITIAL_PAGE)], payload=b"PK" * 10_000
|
||||
)
|
||||
book_path = tmp_path / "dune.epub"
|
||||
|
||||
result = op.OceanofPDFProvider().download(_record(), book_path, None, Event(), None)
|
||||
|
||||
assert result == FILE_URL
|
||||
assert book_path.read_bytes() == b"PK" * 10_000
|
||||
assert posts[0]["url"] == FETCH_URL
|
||||
assert posts[0]["data"] == {"id": "srv3", "filename": "Dune_1_-_Frank_Herbert.epub"}
|
||||
assert posts[0]["headers"]["Referer"] == DUNE_URL
|
||||
assert downloads == [{"link": FILE_URL, "size": "952 KB", "referer": FETCH_URL}]
|
||||
|
||||
|
||||
def test_download_form_post_never_offers_brotli(configured, monkeypatch, tmp_path):
|
||||
# requests cannot decode Brotli without the optional package, and Cloudflare answers
|
||||
# in Brotli whenever it is offered - the interstitial came back unreadable.
|
||||
posts, _downloads = _wire_download(
|
||||
monkeypatch, post_responses=[_FakeResponse(INTERSTITIAL_PAGE)], payload=b"PK" * 10_000
|
||||
)
|
||||
|
||||
op.OceanofPDFProvider().download(_record(), tmp_path / "d.epub", None, Event(), None)
|
||||
|
||||
assert "br" not in posts[0]["headers"].get("Accept-Encoding", "")
|
||||
|
||||
|
||||
def test_page_without_a_file_link_is_described_in_the_log(configured, monkeypatch, tmp_path):
|
||||
page = _FakeResponse("<html><title>Fetching Resource</title><p>Please donate</p></html>")
|
||||
page.headers = {"Content-Type": "text/html", "Content-Encoding": "br"}
|
||||
_wire_download(monkeypatch, post_responses=[page], payload=b"")
|
||||
warnings: list[str] = []
|
||||
monkeypatch.setattr(op.logger, "warning", lambda msg, *args: warnings.append(msg % args))
|
||||
|
||||
result = op.OceanofPDFProvider().download(_record(), tmp_path / "d.epub", None, Event(), None)
|
||||
|
||||
assert result is None
|
||||
described = [message for message in warnings if "had no file link" in message]
|
||||
assert described
|
||||
assert "content-encoding='br'" in described[0]
|
||||
assert "title='Fetching Resource'" in described[0]
|
||||
|
||||
|
||||
def test_download_rejects_an_html_page_served_as_the_file(configured, monkeypatch, tmp_path):
|
||||
_wire_download(
|
||||
monkeypatch,
|
||||
post_responses=[_FakeResponse(INTERSTITIAL_PAGE)],
|
||||
payload=b"<!DOCTYPE html>" + b" " * 20_000,
|
||||
)
|
||||
book_path = tmp_path / "dune.epub"
|
||||
|
||||
assert op.OceanofPDFProvider().download(_record(), book_path, None, Event(), None) is None
|
||||
assert not book_path.exists()
|
||||
|
||||
|
||||
def test_challenged_form_post_refreshes_clearance_and_retries(configured, monkeypatch, tmp_path):
|
||||
posts, _downloads = _wire_download(
|
||||
monkeypatch,
|
||||
post_responses=[_FakeResponse(CHALLENGE_PAGE, 403), _FakeResponse(INTERSTITIAL_PAGE)],
|
||||
payload=b"PK" * 10_000,
|
||||
)
|
||||
bypassed: list[str] = []
|
||||
page_fetch = op.downloader.html_get_page
|
||||
|
||||
def fake_html_get_page(url, *args, use_bypasser=False, **kwargs):
|
||||
if use_bypasser:
|
||||
bypassed.append(url)
|
||||
return page_fetch(url, *args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(op.downloader, "html_get_page", fake_html_get_page)
|
||||
|
||||
result = op.OceanofPDFProvider().download(_record(), tmp_path / "d.epub", None, Event(), None)
|
||||
|
||||
assert result == FILE_URL
|
||||
assert bypassed == [DUNE_URL]
|
||||
assert len(posts) == 2
|
||||
|
||||
|
||||
def test_missing_format_on_the_book_page_fails_cleanly(configured, monkeypatch, tmp_path):
|
||||
posts, _downloads = _wire_download(monkeypatch, post_responses=[], payload=b"")
|
||||
|
||||
result = op.OceanofPDFProvider().download(
|
||||
_record("mobi"), tmp_path / "d.mobi", None, Event(), None
|
||||
)
|
||||
|
||||
assert result is None
|
||||
assert posts == []
|
||||
|
||||
|
||||
def test_cancelled_download_never_posts(configured, monkeypatch, tmp_path):
|
||||
posts, _downloads = _wire_download(monkeypatch, post_responses=[], payload=b"")
|
||||
cancelled = Event()
|
||||
cancelled.set()
|
||||
|
||||
result = op.OceanofPDFProvider().download(_record(), tmp_path / "d.epub", None, cancelled, None)
|
||||
|
||||
assert result is None
|
||||
assert posts == []
|
||||
|
||||
|
||||
# --- Registry wiring ---
|
||||
|
||||
|
||||
def test_oceanofpdf_records_route_to_the_oceanofpdf_provider(configured):
|
||||
provider = registry.provider_for_record(_record())
|
||||
|
||||
assert provider is not None
|
||||
assert provider.id == "oceanofpdf"
|
||||
|
||||
|
||||
def test_provider_is_enabled_only_with_a_mirror(configured, monkeypatch):
|
||||
provider = op.OceanofPDFProvider()
|
||||
assert provider.is_enabled()
|
||||
|
||||
monkeypatch.setattr(mirrors, "_get_config", lambda: {})
|
||||
assert not provider.is_enabled()
|
||||
|
||||
|
||||
def test_toggle_off_disables_the_provider(configured):
|
||||
configured["SOURCE_PRIORITY"] = [{"id": "oceanofpdf", "enabled": False}]
|
||||
|
||||
assert not op.OceanofPDFProvider().is_enabled()
|
||||
|
||||
|
||||
def test_saved_list_from_before_oceanofpdf_counts_as_off(configured):
|
||||
# The settings page shows an option missing from the saved list as off; agree with it.
|
||||
configured["SOURCE_PRIORITY"] = [{"id": "zlib", "enabled": True}]
|
||||
|
||||
assert not op.OceanofPDFProvider().is_enabled()
|
||||
|
||||
|
||||
def test_nothing_saved_falls_back_to_on(configured):
|
||||
configured["SOURCE_PRIORITY"] = None
|
||||
|
||||
assert op.OceanofPDFProvider().is_enabled()
|
||||
|
||||
|
||||
def test_provider_is_off_without_the_cloudflare_bypass(configured):
|
||||
configured["USE_CF_BYPASS"] = False
|
||||
|
||||
assert not op.OceanofPDFProvider().is_enabled()
|
||||
|
||||
|
||||
def test_toggle_is_offered_in_slow_sources_and_on_by_default(configured):
|
||||
from shelfmark.config import settings as settings_module
|
||||
|
||||
options = {o["id"]: o for o in settings_module._get_slow_source_options()}
|
||||
defaults = {d["id"]: d for d in settings_module._get_slow_source_defaults()}
|
||||
|
||||
assert options["oceanofpdf"]["label"] == "OceanofPDF"
|
||||
assert options["oceanofpdf"]["isLocked"] is False
|
||||
assert defaults["oceanofpdf"]["enabled"] is True
|
||||
|
||||
|
||||
def test_toggle_is_locked_until_a_mirror_is_added(configured, monkeypatch):
|
||||
from shelfmark.config import settings as settings_module
|
||||
|
||||
monkeypatch.setattr(mirrors, "_get_config", lambda: {})
|
||||
option = next(o for o in settings_module._get_slow_source_options() if o["id"] == "oceanofpdf")
|
||||
|
||||
assert option["isLocked"] is True
|
||||
assert option["disabledReason"] == "Add at least one OceanofPDF mirror in Mirrors"
|
||||
|
||||
|
||||
def test_annas_archive_cascade_ignores_the_oceanofpdf_entry(monkeypatch):
|
||||
from shelfmark.release_sources.direct_download import annas_archive as aa
|
||||
|
||||
priority = [{"id": "zlib", "enabled": True}, {"id": "oceanofpdf", "enabled": True}]
|
||||
monkeypatch.setattr(
|
||||
aa.config,
|
||||
"get",
|
||||
lambda key, default=None: priority if key == "SOURCE_PRIORITY" else default,
|
||||
)
|
||||
|
||||
assert "oceanofpdf" not in [entry["id"] for entry in aa._get_source_priority()]
|
||||
|
||||
|
||||
def test_provider_handles_only_its_configured_host(configured):
|
||||
provider = op.OceanofPDFProvider()
|
||||
|
||||
assert provider.handles(DUNE_URL)
|
||||
assert not provider.handles("https://annas-archive.example/md5/abc")
|
||||
|
||||
|
||||
def test_direct_download_is_available_with_only_oceanofpdf(configured, monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
registry.config,
|
||||
"get",
|
||||
lambda key, default=None: True if key == "DIRECT_DOWNLOAD_ENABLED" else default,
|
||||
)
|
||||
|
||||
assert registry.get_unavailable_reason() is None
|
||||
assert [p.id for p in registry.enabled_providers()] == ["oceanofpdf"]
|
||||
|
||||
|
||||
def test_mirror_list_is_normalised_and_first_entry_wins(monkeypatch):
|
||||
monkeypatch.setattr(
|
||||
mirrors,
|
||||
"_get_config",
|
||||
lambda: {"OCEANOFPDF_MIRROR_URLS": ["ocean.example/", "https://backup.example"]},
|
||||
)
|
||||
|
||||
assert mirrors.get_oceanofpdf_mirrors() == ["https://ocean.example", "https://backup.example"]
|
||||
assert mirrors.get_oceanofpdf_base_url() == "https://ocean.example"
|
||||
Reference in new issue
Block a user