mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-10-06 08:44:38 +01:00
Alex Guerrieri
c6b70a6844
## What libgen.li's `ads.php?md5=` now returns an **empty `200`** to any request without a `Referer` — an anti-hotlinking check the mirrors added recently. Both libgen paths fetch it without one, so the page comes back blank and the download silently fails while **search keeps working** (which is exactly why it looks like rate-limiting or mirror drift rather than a bug). Same one-line cause, two call sites: the Libgen search source (`libgen/scraper.py:fetch_page`) and the AA-md5 → libgen fallback (`direct_download/annas_archive.py:_extract_libgen_download_url`). Fix: send a same-origin `Referer: <scheme>://<host>/` on the `ads.php` fetch in both. ## Worth a look in review - **The referer goes on the *resolution* fetch, not the download.** `download_url(..., referer=...)` was already correct — the blank page happens one step earlier, at the `ads.php` GET. - Reproduced against live mirrors: `ads.php` returns `Content-Length: 0` bare, the full page with a `Referer`, and resolvable files download valid bytes again. Regression tests in `tests/libgen/` and `tests/direct_download/` assert the header on both paths. Lint/format/typecheck clean. Follow-up to #1326.
34 lines
1.3 KiB
Python
34 lines
1.3 KiB
Python
"""Tests for the direct-download Libgen ads.php resolution (AA-md5 -> libgen fallback)."""
|
|
|
|
from unittest.mock import patch
|
|
|
|
from shelfmark.release_sources.direct_download import annas_archive
|
|
from tests.libgen import sample_html as html
|
|
|
|
|
|
def test_extract_libgen_download_url_sends_same_origin_referer():
|
|
# libgen.li's ads.php returns an empty 200 without a Referer; the resolver must send a
|
|
# same-origin one or it never finds the get.php link and the download silently fails.
|
|
captured = {}
|
|
|
|
class FakeResponse:
|
|
status_code = 200
|
|
text = html.ADS_HTML
|
|
url = "https://libgen.li/ads.php?md5=" + html.MD5_A
|
|
|
|
def fake_get(link, **kwargs):
|
|
captured["headers"] = kwargs["headers"]
|
|
return FakeResponse()
|
|
|
|
with (
|
|
patch.object(annas_archive.requests, "get", side_effect=fake_get),
|
|
patch.object(annas_archive.network, "get_proxies", return_value=None),
|
|
patch.object(annas_archive.network, "get_ssl_verify", return_value=True),
|
|
):
|
|
url = annas_archive._extract_libgen_download_url(
|
|
f"https://libgen.li/ads.php?md5={html.MD5_A}"
|
|
)
|
|
|
|
assert captured["headers"]["Referer"] == "https://libgen.li/"
|
|
assert url == f"https://libgen.li/get.php?md5={html.MD5_A}&key={html.GET_KEY}"
|