starred/shelfmark
1
0
Fork 0
mirror of https://github.com/calibrain/shelfmark.git synced 2026-10-05 10:11:12 +01:00
Code Issues Packages Projects Releases 10 Wiki Activity
Files
ade4878cfd66a2536a0eb2f018d8fcdfebcf0e71
shelfmark/tests/audiobookbay/test_scraper.py
T
spindriftNemesisHubrisCaliBrain
 aba1a68dda Backport bug fixes from NemesisHubris/litfinder (addresses #999, #956, #1010, #1021, #1025, #1040) (#1066) 
## Backport bug fixes from `NemesisHubris/litfinder`

Forwards a curated set of bug fixes from
[NemesisHubris/litfinder](https://github.com/NemesisHubris/litfinder) —
a community fork of this project — that address open issues here. All
commits preserve original authorship via `git cherry-pick`; this PR is a
backport rather than original work. Each fix has been reviewed locally,
lint/format-cleaned to match this repo's existing ruff config, and
verified with the test suite. Rebrand strings, license switches, and
features have been deliberately excluded.

### Upstream issues addressed

- **#999** — Mirror URLs with query params no longer break search
requests (strip query string/fragment in `normalize_http_url`)
- **#956** — Apprise notifications now respect the configured proxy
(proxy env vars injected before dispatch)
- **#1025** — rTorrent: separate `RTORRENT_AUDIOBOOK_LABEL` setting,
falls back to book label if unset
- **#1010** — Stop button in Activity no longer makes the panel
disappear (snapshot refresh on cancel)
- **#1021** — Anna's Archive slow-download countdown now caps retries
instead of looping forever
- **#1040** — Empty destination directory cleaned up when write probe
fails
- **PR #1031** — Language detection from Anna's Archive distant path
when listing metadata is missing

### Additional fixes (no open issue but clear bugs)

- **fix: Python 2 `except` syntax across 27 files** — `except X, Y:` is
a SyntaxError in Python 3 and prevents affected modules from importing
at runtime. Mechanical sweep to `except (X, Y):`.
- **fix(abb): info hash validation with magnet fallback** — adds
SHA-1/SHA-256 hex validation on extracted info hashes; falls back to
scanning the full page for a magnet link (e.g. posted in comments) when
the table value is malformed. Also extends the exact-phrase fallback to
manual queries and defaults the ABB listing language to `en` when
missing, preventing valid results from being hidden by the language
filter. Includes a small test-fixture fix (`test(abb): use valid hex
info hashes in scraper test fixtures`) since the existing fixtures used
non-hex placeholders that the new validation correctly rejects.
- **fix: Anna's Archive title parser** — handles nested edition spans
and filters `lgli` catalog descriptor entries (e.g. "Book/Online Audio")
that were polluting search results.

### Deliberately not included

- LitFinder rebranding (UI strings, Apprise app ID, logo). The `fix:
three upstream bugs` commit (#999/#956/#1025) was cherry-picked with
Apprise app-id, description, and logo-URL strings reverted from
"LitFinder" back to "Shelfmark"; noted in the commit body.
- Features from the LitFinder fork (multi-variant title search,
multi-book flat-folder grouping, fuzzy text matching, "Leave in Place"
output handler, admin display name, custom-source plugin system). These
are larger behavior changes that each warrant their own focused review —
happy to send any of them separately if of interest.
- LitFinder-specific test environment and CI infrastructure.

### Verification

- Backend: **1879 passed**, 96 skipped (1 preexisting failure on
`seleniumbase`-dependent test in local venv; runs fine in the standard
Docker image with the `browser` extra)
- Lint, format, dead-code: all clean against this repo's existing
ruff/vulture config
- One follow-up cleanup commit (`style: ruff lint and format fixes for
ported commits`) brings the cherry-picked code into compliance with this
repo's ruff settings — no behavior changes there

### Etiquette / credit

Per-commit authorship preserved by cherry-pick. The only edits to the
original commits are:
- `fix: three upstream bugs` — Apprise rebrand strings reverted to
"Shelfmark" (noted in commit body, original author retained as
`Co-Authored-By` via cherry-pick)
- One follow-up `style:` commit for ruff config alignment

Big thanks to [@NemesisHubris](https://github.com/NemesisHubris) for the
original work in LitFinder; this PR exists to make sure these fixes
reach Shelfmark's wider user base. Happy to revise scope, split into
smaller PRs, or split off the Py2 cleanup separately if that's
preferable.

---------

Co-authored-by: NemesisHubris <155838970+NemesisHubris@users.noreply.github.com>
Co-authored-by: CaliBrain <calibrain@l4n.xyz>
2026-06-15 00:28:43 -04:00

460 lines
19 KiB
Python
Raw Blame History

"""
Tests for AudiobookBay scraper functions.
"""
from unittest.mock import patch
from shelfmark.release_sources.audiobookbay import scraper
# Mock HTML based on real ABB structure
SAMPLE_SEARCH_HTML = """
<html>
<body>
<div class="post">
<div class="postTitle"><h2><a href="/abss/test-book-title-by-author/" rel="bookmark">Test Book Title - Test Author</a></h2></div>
<div class="postInfo">Category: Genre&nbsp; <br>Language: English<span style="margin-left:100px;">Keywords: Test Keywords&nbsp;</span><br></div>
<div class="postContent">
<div class="center">
<p class="center">Shared by:<a href="/member/users/index?&mode=userinfo&username=testuser">testuser</a></p>
<p class="center"><a href="https://audiobookbay.lu/abss/test-book-title-by-author/"><img src="https://example.com/cover.jpg" alt="Test Cover" width="250"></a></p>
</div>
<p style="text-align:center;">Posted: 01 Jan 2024<br>Format: <span style="color:#a00;">M4B</span> / Bitrate: <span style="color:#a00;">128 Kbps</span><br>File Size: <span style="color:#00f;">500.00</span> MBs</p>
</div>
<div class="postMeta">
<span class="postLink"><a href="https://audiobookbay.lu/abss/test-book-title-by-author/">Audiobook Details</a></span>
<span class="postComments"><a href="/dload-now?ll=test" rel="nofollow">Direct Download</a></span>
</div>
</div>
<div class="post">
<div class="postTitle"><h2><a href="/abss/another-test-book/" rel="bookmark">Another Test Book - Another Author</a></h2></div>
<div class="postInfo">Category: Fiction&nbsp; <br>Language: Spanish<span style="margin-left:100px;">Keywords: Test&nbsp;</span><br></div>
<div class="postContent">
<div class="center">
<p class="center">Shared by:<a href="/member/users/index?&mode=userinfo&username=user2">user2</a></p>
<p class="center"><a href="https://audiobookbay.lu/abss/another-test-book/"><img src="https://example.com/cover2.jpg" alt="Cover 2" width="250"></a></p>
</div>
<p style="text-align:center;">Posted: 15 Nov 2023<br>Format: <span style="color:#a00;">MP3</span> / Bitrate: <span style="color:#a00;">256 Kbps</span><br>File Size: <span style="color:#00f;">1.01</span> GBs</p>
</div>
<div class="postMeta">
<span class="postLink"><a href="https://audiobookbay.lu/abss/another-test-book/">Audiobook Details</a></span>
<span class="postComments"><a href="/dload-now?ll=test2" rel="nofollow">Direct Download</a></span>
</div>
</div>
</body>
</html>
"""
EMPTY_SEARCH_HTML = """
<html>
<body>
</body>
</html>
"""
# Mock HTML for detail page with info hash and trackers
SAMPLE_DETAIL_HTML = """
<html>
<body>
<table>
<tr>
<td>Info Hash</td>
<td>ABC123DEF456789012345678901234567890ABCD</td>
</tr>
<tr>
<td>Tracker 1</td>
<td>udp://tracker.openbittorrent.com:80</td>
</tr>
<tr>
<td>Tracker 2</td>
<td>http://tracker.example.com:8080</td>
</tr>
<tr>
<td>Other Info</td>
<td>Some other data</td>
</tr>
</table>
</body>
</html>
"""
DETAIL_HTML_NO_TRACKERS = """
<html>
<body>
<table>
<tr>
<td>Info Hash</td>
<td>ABC123DEF456789012345678901234567890ABCD</td>
</tr>
</table>
</body>
</html>
"""
class TestSearchAudiobookbay:
"""Tests for the search_audiobookbay function."""
@patch("shelfmark.release_sources.audiobookbay.scraper.downloader.html_get_page")
@patch("shelfmark.release_sources.audiobookbay.scraper.config.get")
def test_search_audiobookbay_success(self, mock_config_get, mock_html_get):
"""Test successful search with results."""
mock_config_get.return_value = 1.0 # rate_limit_delay
mock_html_get.return_value = (
SAMPLE_SEARCH_HTML,
"https://audiobookbay.lu/page/1/?s=test+query&cat=undefined%2Cundefined",
)
results = scraper.search_audiobookbay("test query", max_pages=1, hostname="audiobookbay.lu")
assert len(results) == 2
assert results[0]["title"] == "Test Book Title - Test Author"
assert results[0]["link"] == "https://audiobookbay.lu/abss/test-book-title-by-author/"
assert results[0]["language"] == "English"
assert results[0]["format"] == "M4B"
assert results[0]["bitrate"] == "128 Kbps"
assert results[0]["size"] == "500.00 MB"
assert results[0]["posted_date"] == "01 Jan 2024"
assert results[0]["cover"] == "https://example.com/cover.jpg"
assert results[1]["title"] == "Another Test Book - Another Author"
assert results[1]["language"] == "Spanish"
assert results[1]["format"] == "MP3"
assert results[1]["size"] == "1.01 GB"
@patch("shelfmark.release_sources.audiobookbay.scraper.downloader.html_get_page")
@patch("shelfmark.release_sources.audiobookbay.scraper.config.get")
def test_search_audiobookbay_pagination(self, mock_config_get, mock_html_get):
"""Test pagination through multiple pages."""
mock_config_get.return_value = 0.0 # No delay for faster tests
mock_html_get.side_effect = [
(EMPTY_SEARCH_HTML, "https://audiobookbay.lu/"), # Session bootstrap
(
SAMPLE_SEARCH_HTML,
"https://audiobookbay.lu/page/1/?s=test&cat=undefined%2Cundefined",
),
(EMPTY_SEARCH_HTML, "https://audiobookbay.lu/page/2/?s=test&cat=undefined%2Cundefined"),
]
results = scraper.search_audiobookbay("test", max_pages=2, hostname="audiobookbay.lu")
assert len(results) == 2 # Only from first page
assert mock_html_get.call_count == 3
@patch("shelfmark.release_sources.audiobookbay.scraper.downloader.html_get_page")
@patch("shelfmark.release_sources.audiobookbay.scraper.config.get")
def test_search_audiobookbay_page_one_uses_root_search_endpoint(
self, mock_config_get, mock_html_get
):
"""Test page 1 search uses ABB root endpoint instead of /page/1/."""
mock_config_get.return_value = 0.0
mock_html_get.return_value = (
SAMPLE_SEARCH_HTML,
"https://audiobookbay.lu/?s=test",
)
results = scraper.search_audiobookbay("test", max_pages=1, hostname="audiobookbay.lu")
assert len(results) == 2
requested_url = mock_html_get.call_args.args[0]
assert requested_url.startswith("https://audiobookbay.lu/?s=test")
assert "/page/1/" not in requested_url
assert "cat=undefined%2Cundefined" in requested_url
@patch("shelfmark.release_sources.audiobookbay.scraper.downloader.html_get_page")
@patch("shelfmark.release_sources.audiobookbay.scraper.config.get")
def test_search_audiobookbay_bootstraps_and_reuses_session(
self, mock_config_get, mock_html_get
):
"""Test ABB search initializes and reuses a request session for cookie continuity."""
mock_config_get.return_value = 0.0
mock_html_get.side_effect = [
("", "https://audiobookbay.lu/"), # Bootstrap attempt
(SAMPLE_SEARCH_HTML, "https://audiobookbay.lu/?s=test&cat=undefined%2Cundefined"),
]
results = scraper.search_audiobookbay("test", max_pages=1, hostname="audiobookbay.lu")
assert len(results) == 2
assert mock_html_get.call_count == 2
bootstrap_call = mock_html_get.call_args_list[0]
search_call = mock_html_get.call_args_list[1]
assert bootstrap_call.args[0] == "https://audiobookbay.lu/"
assert search_call.args[0].startswith("https://audiobookbay.lu/?s=test")
assert bootstrap_call.kwargs["session"] is not None
assert search_call.kwargs["session"] is bootstrap_call.kwargs["session"]
@patch("shelfmark.release_sources.audiobookbay.scraper.downloader.html_get_page")
@patch("shelfmark.release_sources.audiobookbay.scraper.config.get")
def test_search_audiobookbay_empty(self, mock_config_get, mock_html_get):
"""Test search with no results."""
mock_config_get.return_value = 1.0
mock_html_get.return_value = (
EMPTY_SEARCH_HTML,
"https://audiobookbay.lu/page/1/?s=test&cat=undefined%2Cundefined",
)
results = scraper.search_audiobookbay("test", max_pages=1, hostname="audiobookbay.lu")
assert len(results) == 0
@patch("shelfmark.release_sources.audiobookbay.scraper.downloader.html_get_page")
@patch("shelfmark.release_sources.audiobookbay.scraper.config.get")
def test_search_audiobookbay_error_non_200(self, mock_config_get, mock_html_get):
"""Test error handling for non-200 status code."""
mock_config_get.return_value = 1.0
mock_html_get.return_value = (
"",
"https://audiobookbay.lu/page/1/?s=test&cat=undefined%2Cundefined",
)
results = scraper.search_audiobookbay("test", max_pages=1, hostname="audiobookbay.lu")
assert len(results) == 0
@patch("shelfmark.release_sources.audiobookbay.scraper.downloader.html_get_page")
@patch("shelfmark.release_sources.audiobookbay.scraper.config.get")
def test_search_audiobookbay_redirect_to_homepage(self, mock_config_get, mock_html_get):
"""Test handling redirect to homepage (blocked/invalid search)."""
mock_config_get.return_value = 1.0
mock_html_get.return_value = (
EMPTY_SEARCH_HTML,
"https://audiobookbay.lu", # Redirected to homepage
)
results = scraper.search_audiobookbay("test", max_pages=1, hostname="audiobookbay.lu")
assert len(results) == 0
@patch("shelfmark.release_sources.audiobookbay.scraper.downloader.html_get_page")
@patch("shelfmark.release_sources.audiobookbay.scraper.config.get")
def test_search_audiobookbay_request_exception(self, mock_config_get, mock_html_get):
"""Test handling request exceptions."""
mock_config_get.return_value = 1.0
mock_html_get.return_value = (
"",
"https://audiobookbay.lu/page/1/?s=test&cat=undefined%2Cundefined",
)
results = scraper.search_audiobookbay("test", max_pages=1, hostname="audiobookbay.lu")
assert len(results) == 0
@patch("shelfmark.release_sources.audiobookbay.scraper.downloader.html_get_page")
@patch("shelfmark.release_sources.audiobookbay.scraper.config.get")
def test_search_audiobookbay_relative_link(self, mock_config_get, mock_html_get):
"""Test handling relative links in results."""
mock_config_get.return_value = 1.0
html_with_relative_link = """
<div class="post">
<div class="postTitle"><h2><a href="/abss/relative-link/">Test Book</a></h2></div>
<div class="postInfo">Language: English</div>
<div class="postContent">
<p style="text-align:center;">Posted: 01 Jan 2024<br>Format: M4B<br>File Size: 100 MBs</p>
</div>
</div>
"""
mock_html_get.return_value = (
html_with_relative_link,
"https://audiobookbay.lu/page/1/?s=test&cat=undefined%2Cundefined",
)
results = scraper.search_audiobookbay("test", max_pages=1, hostname="audiobookbay.lu")
assert len(results) == 1
assert results[0]["link"] == "https://audiobookbay.lu/abss/relative-link/"
@patch("shelfmark.release_sources.audiobookbay.scraper.downloader.html_get_page")
@patch("shelfmark.release_sources.audiobookbay.scraper.config.get")
def test_search_audiobookbay_protocol_relative_links(self, mock_config_get, mock_html_get):
"""Test protocol-relative links are normalized without duplicating hostname."""
mock_config_get.return_value = 0.0
html_with_protocol_relative_links = """
<div class="post">
<div class="postTitle"><h2><a href="//audiobookbay.lu/abss/protocol-relative/">Protocol Relative Book</a></h2></div>
<div class="postInfo">Language: English</div>
<div class="postContent">
<div class="center">
<img src="//audiobookbay.lu/wp-content/uploads/cover.jpg" alt="Cover" />
</div>
<p style="text-align:center;">Posted: 01 Jan 2024<br>Format: M4B<br>File Size: 100 MBs</p>
</div>
</div>
"""
mock_html_get.return_value = (
html_with_protocol_relative_links,
"https://audiobookbay.lu/?s=test",
)
results = scraper.search_audiobookbay("test", max_pages=1, hostname="audiobookbay.lu")
assert len(results) == 1
assert results[0]["link"] == "https://audiobookbay.lu/abss/protocol-relative/"
assert results[0]["cover"] == "https://audiobookbay.lu/wp-content/uploads/cover.jpg"
@patch("shelfmark.release_sources.audiobookbay.scraper.downloader.html_get_page")
@patch("shelfmark.release_sources.audiobookbay.scraper.config.get")
def test_search_audiobookbay_exact_phrase_query(self, mock_config_get, mock_html_get):
"""Test exact phrase wrapping and encoding in search URL."""
mock_config_get.return_value = 0.0
mock_html_get.return_value = (
SAMPLE_SEARCH_HTML,
"https://audiobookbay.lu/page/1/?s=%22test+query%22",
)
results = scraper.search_audiobookbay(
"test query",
max_pages=1,
hostname="audiobookbay.lu",
exact_phrase=True,
)
assert len(results) == 2
requested_url = mock_html_get.call_args.args[0]
assert "s=%22test+query%22" in requested_url
assert "cat=undefined%2Cundefined" in requested_url
@patch("shelfmark.release_sources.audiobookbay.scraper.downloader.html_get_page")
@patch("shelfmark.release_sources.audiobookbay.scraper.config.get")
def test_search_audiobookbay_always_uses_legacy_category_query(
self, mock_config_get, mock_html_get
):
"""Test ABB search always includes legacy category query and does not fallback."""
mock_config_get.return_value = 0.0
mock_html_get.return_value = (
"",
"https://audiobookbay.lu/?s=test&cat=undefined%2Cundefined",
)
results = scraper.search_audiobookbay("test", max_pages=1, hostname="audiobookbay.lu")
assert len(results) == 0
assert mock_html_get.call_count >= 2
search_urls = [
call.args[0] for call in mock_html_get.call_args_list if "?s=test" in call.args[0]
]
assert search_urls
assert all(
url == "https://audiobookbay.lu/?s=test&cat=undefined%2Cundefined"
for url in search_urls
)
class TestExtractMagnetLink:
"""Tests for the extract_magnet_link function."""
@patch("shelfmark.release_sources.audiobookbay.scraper.downloader.html_get_page")
def test_extract_magnet_link_success(self, mock_html_get):
"""Test successful magnet link extraction."""
mock_html_get.side_effect = [
("", "https://audiobookbay.lu/"), # Bootstrap attempt
SAMPLE_DETAIL_HTML,
]
magnet_link = scraper.extract_magnet_link(
"https://audiobookbay.lu/abss/test-book/", hostname="audiobookbay.lu"
)
assert magnet_link is not None
assert magnet_link.startswith("magnet:?xt=urn:btih:")
assert "ABC123DEF456789012345678901234567890ABCD" in magnet_link
assert "udp%3A//tracker.openbittorrent.com%3A80" in magnet_link
assert "http%3A//tracker.example.com%3A8080" in magnet_link
assert mock_html_get.call_count == 2
@patch("shelfmark.release_sources.audiobookbay.scraper.downloader.html_get_page")
def test_extract_magnet_link_reuses_bootstrap_session(self, mock_html_get):
"""Test detail page fetch reuses the bootstrap session for ABB cookies."""
mock_html_get.side_effect = [
("", "https://audiobookbay.lu/"),
SAMPLE_DETAIL_HTML,
]
scraper.extract_magnet_link(
"https://audiobookbay.lu/abss/test-book/", hostname="audiobookbay.lu"
)
assert mock_html_get.call_count == 2
bootstrap_call = mock_html_get.call_args_list[0]
detail_call = mock_html_get.call_args_list[1]
assert bootstrap_call.args[0] == "https://audiobookbay.lu/"
assert detail_call.args[0] == "https://audiobookbay.lu/abss/test-book/"
assert detail_call.kwargs["session"] is bootstrap_call.kwargs["session"]
@patch("shelfmark.release_sources.audiobookbay.scraper.downloader.html_get_page")
def test_extract_magnet_link_fallback(self, mock_html_get):
"""Test fallback to default trackers when none found."""
mock_html_get.return_value = DETAIL_HTML_NO_TRACKERS
magnet_link = scraper.extract_magnet_link(
"https://audiobookbay.lu/abss/test-book/", hostname="audiobookbay.lu"
)
assert magnet_link is not None
assert magnet_link.startswith("magnet:?xt=urn:btih:")
assert "ABC123DEF456789012345678901234567890ABCD" in magnet_link
# Should contain default trackers
assert "udp%3A//tracker.openbittorrent.com%3A80" in magnet_link
@patch("shelfmark.release_sources.audiobookbay.scraper.downloader.html_get_page")
def test_extract_magnet_link_no_info_hash(self, mock_html_get):
"""Test handling missing info hash."""
mock_html_get.return_value = "<html><body></body></html>"
magnet_link = scraper.extract_magnet_link(
"https://audiobookbay.lu/abss/test-book/", hostname="audiobookbay.lu"
)
assert magnet_link is None
@patch("shelfmark.release_sources.audiobookbay.scraper.downloader.html_get_page")
def test_extract_magnet_link_non_200(self, mock_html_get):
"""Test handling non-200 status code."""
mock_html_get.return_value = ""
magnet_link = scraper.extract_magnet_link(
"https://audiobookbay.lu/abss/test-book/", hostname="audiobookbay.lu"
)
assert magnet_link is None
@patch("shelfmark.release_sources.audiobookbay.scraper.downloader.html_get_page")
def test_extract_magnet_link_request_exception(self, mock_html_get):
"""Test handling request exceptions."""
mock_html_get.return_value = ""
magnet_link = scraper.extract_magnet_link(
"https://audiobookbay.lu/abss/test-book/", hostname="audiobookbay.lu"
)
assert magnet_link is None
@patch("shelfmark.release_sources.audiobookbay.scraper.downloader.html_get_page")
def test_extract_magnet_link_cleans_info_hash(self, mock_html_get):
"""Test that info hash whitespace is cleaned."""
html_with_whitespace = """
<html>
<body>
<table>
<tr>
<td>Info Hash</td>
<td>ABC 123 DEF 456 789 012 345 678 901 234 567 890 ABC D</td>
</tr>
</table>
</body>
</html>
"""
mock_html_get.return_value = html_with_whitespace
magnet_link = scraper.extract_magnet_link(
"https://audiobookbay.lu/abss/test-book/", hostname="audiobookbay.lu"
)
assert magnet_link is not None
# Info hash should be cleaned (no spaces, uppercase)
assert "ABC123DEF456" in magnet_link
Reference in New Issue View Git Blame Copy Permalink