mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-11-14 23:08:26 +00:00
CaliBrain
1aa9846f22
The destination guard from the oceanofpdf/SSRF fix (f837506) trusts only
hosts in
the mirror lists and refuses any other host that resolves to a private
address.
The full e2e profile has failed on every PR since then: its AA mirror
was mock-aa,
but the slow-download links pointed at cf.mock.test (172.30.0.11). The
guard refused
the slow-download page before the internal Chrome bypasser ever started,
so the
download ended with "All download sources failed". Adding cf.mock.test
as a second
AA mirror would not help: AAMirrorSelector.rewrite would then move the
gated link
onto mock-aa, and the test would pass without Chrome solving anything.
The profile
now uses the gate as its only AA mirror. The cloudflare mock takes
CF_GATED_PATHS,
and the full profile challenges only /slow_download/, so search, detail
and the
final file pass through while the download still needs Chrome. The
unused
SLOW_DOWNLOAD_BASE indirection is gone.
The same allowlist missed a supported setup: Primary Mirror
(AA_BASE_URL) may pin a
URL that is not in the mirror list, and network uses it as the active AA
base. With
it on a LAN address, every AA request was refused. The guard now trusts
the pinned
URL too (mirrors.get_pinned_aa_base_url), and
has_aa_mirror_configuration reuses
the same helper.
download_url also ran the guard a second time outside its try block,
which repeated
the DNS lookup and could raise instead of returning None. The duplicate
call is gone.
47 lines
2.0 KiB
Bash
47 lines
2.0 KiB
Bash
# Profile: full (the "everything real" heavy profile — nightly/manual only)
|
|
#
|
|
# Test book: Moby-Dick (Herman Melville, public domain).
|
|
#
|
|
# What it exercises end to end, with NO mock bypasser:
|
|
# 1. REAL Chrome internal bypasser: the AA mirror is the Cloudflare gate, which
|
|
# passes search/detail through but challenges the *slow-download* page. So a
|
|
# real DOWNLOAD forces the in-image headless Chrome (seleniumbase CDP) to solve
|
|
# the JS challenge and harvest cf_clearance — the literal "spin a chrome
|
|
# browser" path. Success = Moby-Dick lands in /books, which is only possible
|
|
# if Chrome solved the gate.
|
|
# 2. DoH enabled (USE_DOH) so the DNS-over-HTTPS code path runs at boot.
|
|
# 3. REAL download client: a mock Prowlarr returns a webseed .torrent; a real
|
|
# qBittorrent downloads the payload over HTTP from mock-aa and shelfmark
|
|
# moves the completed file into /books.
|
|
#
|
|
# Heavy (Chrome + qBittorrent), so it is excluded from the PR matrix.
|
|
COMPOSE_PROFILES=full
|
|
|
|
# The CF gate is the only AA mirror: shelfmark refuses unconfigured hosts that
|
|
# resolve to private addresses, and a second AA mirror would let the mirror selector
|
|
# rewrite the gated link onto the ungated one. Search/detail and the final file pass
|
|
# straight through; the slow-download page is challenged, so the internal Chrome
|
|
# bypasser must solve it (search uses no bypasser, by design).
|
|
SM_AA_URL=http://cf.mock.test
|
|
SM_CF_GATED_PATHS=/slow_download/
|
|
SM_AA_FILE_BASE=http://cf.mock.test
|
|
SM_USE_CF_BYPASS=true
|
|
SM_USING_EXTERNAL_BYPASSER=false
|
|
|
|
# DNS over HTTPS on.
|
|
SM_USE_DOH=true
|
|
|
|
# Prowlarr indexer + real qBittorrent client.
|
|
SM_PROWLARR_ENABLED=true
|
|
SM_PROWLARR_URL=http://mock-prowlarr
|
|
SM_PROWLARR_API_KEY=e2e-test-key
|
|
SM_PROWLARR_TORRENT_CLIENT=qbittorrent
|
|
SM_QBITTORRENT_URL=http://qbittorrent:8080
|
|
SM_QBITTORRENT_USERNAME=admin
|
|
SM_QBITTORRENT_PASSWORD=adminadmin
|
|
SM_QBITTORRENT_CATEGORY=books
|
|
|
|
# The real client download needs more headroom than a direct HTTP fetch.
|
|
E2E_DOWNLOAD_TIMEOUT=300
|
|
E2E_PROFILE=full
|