fix(http): keep the host of a protocol-relative download link (#1368)

`get_absolute_url()` replaced both `netloc` and `scheme` whenever either
one was missing. A protocol-relative href such as
`//cdn.example.org/f.epub`, scraped from a page on
`https://annas-archive.org/...`, parses with a netloc and an empty
scheme, so it came back pointing at the page's own host. The download
then 404s and the source is skipped.

Each field now falls back to the base URL only when the parsed URL does
not supply it. Plain relative paths resolve exactly as before, which the
control test covers.

This affects the Z-Library, welib and generic download link handling in
`release_sources/direct_download/annas_archive.py`.

## Verification

- New `tests/download/test_http_absolute_url.py`: a protocol-relative
link keeps its own host, and a plain `/path` still resolves against the
base. The first fails on current main and passes here.
- Full suite (3165), ruff, ruff format, basedpyright, vulture green.
This commit is contained in:
splitsec2
2026-09-20 23:04:22 -04:00
committed by GitHub
parent ab3aa9a8b0
commit bc03ad062e
2 changed files with 20 additions and 1 deletions
+3 -1
View File
@@ -1104,5 +1104,7 @@ def get_absolute_url(base_url: str, url: str) -> str:
parsed = urlparse(url)
base = urlparse(base_url)
if not parsed.netloc or not parsed.scheme:
parsed = parsed._replace(netloc=base.netloc, scheme=base.scheme)
parsed = parsed._replace(
netloc=parsed.netloc or base.netloc, scheme=parsed.scheme or base.scheme
)
return parsed.geturl()
+17
View File
@@ -0,0 +1,17 @@
"""Tests for resolving a scraped link against the page it came from."""
def test_get_absolute_url_keeps_a_protocol_relative_host():
import shelfmark.download.http as http
result = http.get_absolute_url("https://annas-archive.org/md5/abc", "//cdn.example.org/f.epub")
assert result == "https://cdn.example.org/f.epub"
def test_get_absolute_url_resolves_a_relative_path_against_the_base():
import shelfmark.download.http as http
result = http.get_absolute_url("https://annas-archive.org/md5/abc", "/slow_download/abc/0/1")
assert result == "https://annas-archive.org/slow_download/abc/0/1"