mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-09-24 11:40:46 +01:00
816a735cde
Fixes #1138 Fixes #1141 ## Problem Two language editions of one book resolve to the same canonical title, so they render to the same path and the second gets a `_1` collision suffix. Audiobookshelf treats a folder as exactly one library item, so the pair becomes a single book with both files as tracks and a summed runtime. Shelfmark already parses and displays the language. It just never reached the template engine. ## `{Language}` template variable A template like `{Author}/{Title}{ (Language)}/{Author} - {Title}` now yields: ``` /library/J K Rowling/Harry Potter (sv)/J K Rowling - Harry Potter.m4b /library/J K Rowling/Harry Potter/J K Rowling - Harry Potter.m4b ``` The untagged edition's path is byte-identical to today, so no existing layout shifts. Three details worth flagging: **The value is casefolded.** On a case-insensitive filesystem `(SV)` and `(sv)` would collapse back into one folder, reintroducing the exact collision being fixed. **Values meaning "we don't know" render nothing** rather than producing `Project Hail Mary (unknown)` folders. Anna's Archive reports that string literally (`direct_download.py`, `language = detected or "unknown"`). **The frontend wasn't sending the release language at all**, so the token would have stayed empty for exactly the audiobook sources in the report. Prowlarr and AudiobookBay do not put language in `extra` the way `direct_download` does, hence the payload plumbing. It reads `release.language`, never `book.language` — the latter is the provider's canonical edition and would mislabel a translation, with a regression test for that specifically. Not gated to audiobooks: Calibre-Web-Automated stages ingested files by basename and discards folder structure, so the rename (filename) template is the only lever those users have. Verified that form works: `J K Rowling - Harry Potter (sv).epub`. ## Language consolidation (#1141) Three release sources each carried their own alias map, all resolving to the same ISO 639-1 codes, alongside a bundled database that only one of them used. Adding a language meant editing three places. Aliases now live in `data/book-languages.json` beside the code and name they belong to, and `shelfmark/core/languages.py` resolves any of them — two-letter code, ISO 639-2 three-letter in either the bibliographic or terminological form, or English name. Prowlarr and AudiobookBay drop their tables. Direct Download keeps its own path-parsing heuristics, including the ambiguous short codes that collide with English words (`de`, `en`, `no`, `in`), and takes only the alias data. This also closes a coverage gap. MyAnonamouse offers 62 languages; Prowlarr mapped 37, and an unmapped code is *dropped* rather than passed through, so the other 25 carried no language at all — leaving `{Language}` empty and the collision unfixed for Latin, Farsi, Tamil, Urdu and the rest. Seven languages MAM offers had no database entry at all: Bosnian, Burmese, Estonian, Icelandic, Manx, Scottish Gaelic, Sanskrit. Also fixes the Traditional Chinese code, which used a U+2011 non-breaking hyphen. Nothing compares against the ASCII spelling today so it was latent, but it would silently defeat the first thing that did. ## Validation Verified end to end against a live Prowlarr and MyAnonamouse, not just unit tests. A real search returning both an English and a Swedish edition, through the actual `queue_release` → `DownloadTask` → naming path: ``` STEP 1 real MAM search -> 37 releases, languages: ['en', 'sv'] STEP 3 queue_release -> task.language='sv' STEP 4 build_metadata_dict -> metadata['Language']='sv' STEP 5 build_library_path -> /library/J K Rowling/Harry Potter (sv)/... two language editions resolve to DIFFERENT folders: True ``` The refactor is pinned by a snapshot of both per-source maps taken *before* they were deleted. All 131 aliases are asserted to still resolve to the same code, one parametrised test each, so a regression names the specific alias. Also verified: the filename-only template, the retry round-trip (`serialize_task_for_retry` → `_restore_task_from_retry_payload`, plus a legacy payload with no `language` key), and placeholder handling. Added a `KNOWN_TOKENS` ordering invariant test — `find_placeholder()` does a substring `.find()` in list order and nothing protected that contract, so a future token in the wrong position could silently shadow an existing one. And a lockstep guard on the frontend, since `KNOWN_TOKENS` is hand-duplicated in TypeScript. **One caveat worth stating.** Three MAM codes are confirmed by observation (`ENG`→`en`, `SWE`→`sv`, `MAL`→`ml`, the last from a real `[MAL / EPUB]` Tagore release). The remaining ~59 are derived from ISO 639-2 rather than observed, because MAM's catalogue is overwhelmingly English — enabling 27 extra languages still yielded only one non-English hit across 258 results. Mitigated rather than closed: both 639-2 variants are present for every language where they differ, and a wrong alias is an unused entry while a missing one loses the language. Happy to correct any code a maintainer knows differs. ## Test results 2056 Python tests pass (up from 1906). Frontend typecheck, lint, format and 126 unit tests pass. Pre-existing failures on my machine, unchanged by this branch and unrelated: `tests/bypass/` needs `seleniumbase`, and `tests/config/test_entrypoint_permissions.py` uses bash-4 syntax that macOS bash 3.2 rejects. --------- Co-authored-by: delize <4028612+delize@users.noreply.github.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz>
445 lines
14 KiB
Python
445 lines
14 KiB
Python
from threading import Event
|
|
from unittest.mock import MagicMock
|
|
|
|
import pytest
|
|
|
|
from shelfmark.core.models import DownloadTask, SearchMode
|
|
|
|
|
|
class _AvailableSource:
|
|
display_name = "Test Source"
|
|
|
|
def is_available(self):
|
|
return True
|
|
|
|
|
|
class _UnavailableSource:
|
|
display_name = "Direct Download"
|
|
|
|
def is_available(self):
|
|
return False
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def source_available_by_default(monkeypatch):
|
|
import shelfmark.download.orchestrator as orchestrator
|
|
|
|
monkeypatch.setattr(orchestrator, "get_source", lambda _source: _AvailableSource())
|
|
|
|
|
|
def enable_prowlarr_seed_preferences(monkeypatch, orchestrator):
|
|
monkeypatch.setattr(
|
|
orchestrator.config,
|
|
"get",
|
|
lambda key, default=None, user_id=None: (
|
|
True if key == "PROWLARR_USE_SEED_PREFERENCES" else default
|
|
),
|
|
)
|
|
|
|
|
|
def test_queue_release_uses_user_specific_books_output_mode(monkeypatch):
|
|
import shelfmark.download.orchestrator as orchestrator
|
|
|
|
captured: dict[str, object] = {}
|
|
config_calls: list[tuple[str, object]] = []
|
|
|
|
def fake_config_get(key, default=None, user_id=None):
|
|
config_calls.append((key, user_id))
|
|
if key == "BOOKS_OUTPUT_MODE":
|
|
return "email" if user_id == 42 else "folder"
|
|
if key == "EMAIL_RECIPIENT":
|
|
return "alice@example.com" if user_id == 42 else ""
|
|
return default
|
|
|
|
def fake_add(task):
|
|
captured["task"] = task
|
|
return True
|
|
|
|
monkeypatch.setattr(orchestrator.config, "get", fake_config_get)
|
|
monkeypatch.setattr(orchestrator.book_queue, "add", fake_add)
|
|
monkeypatch.setattr(orchestrator, "ws_manager", None)
|
|
|
|
release_data = {
|
|
"source": "direct_download",
|
|
"source_id": "release-1",
|
|
"title": "Release Title",
|
|
"content_type": "book (fiction)",
|
|
"format": "epub",
|
|
"size": "1 MB",
|
|
"download_url": "https://audiobookbay.lu/abss/release-title/",
|
|
}
|
|
|
|
success, error = orchestrator.queue_release(release_data, user_id=42, username="alice")
|
|
|
|
assert success is True
|
|
assert error is None
|
|
task = captured["task"]
|
|
assert task.output_mode == "email"
|
|
assert task.output_args == {"to": "alice@example.com"}
|
|
assert task.source_url == "https://audiobookbay.lu/abss/release-title/"
|
|
assert task.search_mode == SearchMode.UNIVERSAL
|
|
assert ("BOOKS_OUTPUT_MODE", 42) in config_calls
|
|
|
|
|
|
def test_queue_release_preserves_direct_search_mode_from_payload(monkeypatch):
|
|
import shelfmark.download.orchestrator as orchestrator
|
|
|
|
captured: dict[str, object] = {}
|
|
|
|
def fake_add(task):
|
|
captured["task"] = task
|
|
return True
|
|
|
|
monkeypatch.setattr(orchestrator.book_queue, "add", fake_add)
|
|
monkeypatch.setattr(orchestrator, "ws_manager", None)
|
|
|
|
success, error = orchestrator.queue_release(
|
|
{
|
|
"source": "direct_download",
|
|
"source_id": "release-direct",
|
|
"title": "Direct Title",
|
|
"content_type": "ebook",
|
|
"search_mode": "direct",
|
|
},
|
|
user_id=42,
|
|
username="alice",
|
|
)
|
|
|
|
assert success is True
|
|
assert error is None
|
|
assert captured["task"].search_mode == SearchMode.DIRECT
|
|
|
|
|
|
def test_queue_release_rejects_unavailable_source(monkeypatch):
|
|
import shelfmark.download.orchestrator as orchestrator
|
|
|
|
monkeypatch.setattr(orchestrator, "get_source", lambda _source: _UnavailableSource())
|
|
monkeypatch.setattr(orchestrator.book_queue, "add", MagicMock())
|
|
|
|
success, error = orchestrator.queue_release(
|
|
{
|
|
"source": "direct_download",
|
|
"source_id": "release-disabled-direct",
|
|
"title": "Disabled Direct Release",
|
|
"content_type": "ebook",
|
|
},
|
|
user_id=42,
|
|
username="alice",
|
|
)
|
|
|
|
assert success is False
|
|
assert error == "Direct Download is unavailable. Enable and configure the source in Settings."
|
|
orchestrator.book_queue.add.assert_not_called()
|
|
|
|
|
|
def test_queue_release_email_mode_without_recipient_is_queued(monkeypatch):
|
|
import shelfmark.download.orchestrator as orchestrator
|
|
|
|
captured: dict[str, object] = {}
|
|
|
|
def fake_config_get(key, default=None, user_id=None):
|
|
if key == "BOOKS_OUTPUT_MODE":
|
|
return "email" if user_id == 42 else "folder"
|
|
if key == "EMAIL_RECIPIENT":
|
|
return ""
|
|
return default
|
|
|
|
def fake_add(task):
|
|
captured["task"] = task
|
|
return True
|
|
|
|
monkeypatch.setattr(orchestrator.config, "get", fake_config_get)
|
|
monkeypatch.setattr(orchestrator.book_queue, "add", fake_add)
|
|
monkeypatch.setattr(orchestrator, "ws_manager", None)
|
|
|
|
release_data = {
|
|
"source": "direct_download",
|
|
"source_id": "release-1",
|
|
"title": "Release Title",
|
|
"content_type": "book (fiction)",
|
|
"format": "epub",
|
|
"size": "1 MB",
|
|
}
|
|
|
|
success, error = orchestrator.queue_release(release_data, user_id=42, username="alice")
|
|
|
|
assert success is True
|
|
assert error is None
|
|
task = captured["task"]
|
|
assert task.output_mode == "email"
|
|
assert task.output_args == {}
|
|
|
|
|
|
def test_download_task_rejects_unavailable_source_before_handler(monkeypatch):
|
|
import shelfmark.download.orchestrator as orchestrator
|
|
|
|
task = DownloadTask(
|
|
task_id="disabled-task",
|
|
source="direct_download",
|
|
title="Disabled Direct Release",
|
|
)
|
|
status_messages: list[tuple[str, str]] = []
|
|
|
|
monkeypatch.setattr(orchestrator, "get_source", lambda _source: _UnavailableSource())
|
|
monkeypatch.setattr(orchestrator, "get_handler", MagicMock())
|
|
monkeypatch.setattr(orchestrator.book_queue, "get_task", lambda _task_id: task)
|
|
monkeypatch.setattr(
|
|
orchestrator.book_queue,
|
|
"update_status_message",
|
|
lambda task_id, message: status_messages.append((task_id, message)),
|
|
)
|
|
|
|
result = orchestrator._download_task("disabled-task", Event())
|
|
|
|
assert result is None
|
|
assert task.last_error_type == "SourceUnavailable"
|
|
assert task.last_error_message == (
|
|
"Direct Download is unavailable. Enable and configure the source in Settings."
|
|
)
|
|
assert status_messages == [
|
|
(
|
|
"disabled-task",
|
|
"Direct Download is unavailable. Enable and configure the source in Settings.",
|
|
)
|
|
]
|
|
orchestrator.get_handler.assert_not_called()
|
|
|
|
|
|
def test_queue_release_persists_prowlarr_retry_context_without_download_url(monkeypatch):
|
|
import shelfmark.download.orchestrator as orchestrator
|
|
|
|
captured: dict[str, object] = {}
|
|
|
|
def fake_add(task):
|
|
captured["task"] = task
|
|
return True
|
|
|
|
monkeypatch.setattr(orchestrator.book_queue, "add", fake_add)
|
|
monkeypatch.setattr(orchestrator, "ws_manager", None)
|
|
enable_prowlarr_seed_preferences(monkeypatch, orchestrator)
|
|
|
|
success, error = orchestrator.queue_release(
|
|
{
|
|
"source": "prowlarr",
|
|
"source_id": "prowlarr-release-1",
|
|
"title": "Queued Prowlarr Release",
|
|
"download_url": "magnet:?xt=urn:btih:abc123",
|
|
"protocol": "torrent",
|
|
"indexer": "MyIndexer",
|
|
"extra": {
|
|
"indexer_id": 12,
|
|
"configured_ratio_limit": 1.25,
|
|
"configured_seed_time_minutes": 90,
|
|
"info_hash": "ABC123",
|
|
},
|
|
},
|
|
user_id=42,
|
|
username="alice",
|
|
)
|
|
|
|
assert success is True
|
|
assert error is None
|
|
task = captured["task"]
|
|
assert task.retry_download_url is None
|
|
assert task.retry_download_protocol is None
|
|
assert task.retry_source_context == {
|
|
"source_id": "prowlarr-release-1",
|
|
"indexer": "MyIndexer",
|
|
"indexer_id": 12,
|
|
}
|
|
assert task.retry_release_name == "Queued Prowlarr Release"
|
|
assert task.retry_expected_hash == "ABC123"
|
|
assert task.retry_ratio_limit == 1.25
|
|
assert task.retry_seeding_time_limit_minutes == 90
|
|
assert task.can_retry_without_staged_source is True
|
|
|
|
payload = orchestrator.serialize_task_for_retry(task)
|
|
assert payload["retry_download_url"] is None
|
|
assert payload["retry_source_context"] == task.retry_source_context
|
|
|
|
|
|
def test_queue_release_prefers_configured_seed_time_minutes_for_retry(monkeypatch):
|
|
import shelfmark.download.orchestrator as orchestrator
|
|
|
|
captured: dict[str, object] = {}
|
|
|
|
def fake_add(task):
|
|
captured["task"] = task
|
|
return True
|
|
|
|
monkeypatch.setattr(orchestrator.book_queue, "add", fake_add)
|
|
monkeypatch.setattr(orchestrator, "ws_manager", None)
|
|
enable_prowlarr_seed_preferences(monkeypatch, orchestrator)
|
|
|
|
success, error = orchestrator.queue_release(
|
|
{
|
|
"source": "prowlarr",
|
|
"source_id": "prowlarr-release-configured-seed-time",
|
|
"title": "Queued Prowlarr Release",
|
|
"download_url": "magnet:?xt=urn:btih:abc123",
|
|
"protocol": "torrent",
|
|
"extra": {
|
|
"configured_ratio_limit": 2,
|
|
"configured_seed_time_minutes": 7200,
|
|
"minimum_ratio": 1,
|
|
"minimum_seed_time": 259200,
|
|
},
|
|
},
|
|
user_id=42,
|
|
username="alice",
|
|
)
|
|
|
|
assert success is True
|
|
assert error is None
|
|
task = captured["task"]
|
|
assert task.retry_ratio_limit == 2.0
|
|
assert task.retry_seeding_time_limit_minutes == 7200
|
|
|
|
|
|
def test_queue_release_ignores_configured_seed_time_when_disabled_for_retry(monkeypatch):
|
|
import shelfmark.download.orchestrator as orchestrator
|
|
|
|
captured: dict[str, object] = {}
|
|
|
|
def fake_add(task):
|
|
captured["task"] = task
|
|
return True
|
|
|
|
monkeypatch.setattr(orchestrator.book_queue, "add", fake_add)
|
|
monkeypatch.setattr(orchestrator, "ws_manager", None)
|
|
|
|
success, error = orchestrator.queue_release(
|
|
{
|
|
"source": "prowlarr",
|
|
"source_id": "prowlarr-release-configured-seed-time-disabled",
|
|
"title": "Queued Prowlarr Release",
|
|
"download_url": "magnet:?xt=urn:btih:abc123",
|
|
"protocol": "torrent",
|
|
"extra": {
|
|
"configured_ratio_limit": 2,
|
|
"configured_seed_time_minutes": 7200,
|
|
},
|
|
},
|
|
user_id=42,
|
|
username="alice",
|
|
)
|
|
|
|
assert success is True
|
|
assert error is None
|
|
task = captured["task"]
|
|
assert task.retry_ratio_limit is None
|
|
assert task.retry_seeding_time_limit_minutes is None
|
|
|
|
|
|
def test_queue_release_ignores_torznab_minimum_seed_criteria_for_retry(monkeypatch):
|
|
import shelfmark.download.orchestrator as orchestrator
|
|
|
|
captured: dict[str, object] = {}
|
|
|
|
def fake_add(task):
|
|
captured["task"] = task
|
|
return True
|
|
|
|
monkeypatch.setattr(orchestrator.book_queue, "add", fake_add)
|
|
monkeypatch.setattr(orchestrator, "ws_manager", None)
|
|
|
|
success, error = orchestrator.queue_release(
|
|
{
|
|
"source": "prowlarr",
|
|
"source_id": "prowlarr-release-minimum-only",
|
|
"title": "Queued Prowlarr Release",
|
|
"download_url": "magnet:?xt=urn:btih:abc123",
|
|
"protocol": "torrent",
|
|
"extra": {
|
|
"minimum_ratio": 1,
|
|
"minimum_seed_time": 259200,
|
|
},
|
|
},
|
|
user_id=42,
|
|
username="alice",
|
|
)
|
|
|
|
assert success is True
|
|
assert error is None
|
|
task = captured["task"]
|
|
assert task.retry_ratio_limit is None
|
|
assert task.retry_seeding_time_limit_minutes is None
|
|
|
|
|
|
def test_queue_release_returns_error_for_operational_queue_failure(monkeypatch):
|
|
import shelfmark.download.orchestrator as orchestrator
|
|
|
|
monkeypatch.setattr(
|
|
orchestrator.book_queue,
|
|
"add",
|
|
MagicMock(side_effect=RuntimeError("queue offline")),
|
|
)
|
|
monkeypatch.setattr(orchestrator, "ws_manager", None)
|
|
|
|
success, error = orchestrator.queue_release(
|
|
{
|
|
"source": "direct_download",
|
|
"source_id": "release-broken-1",
|
|
"title": "Broken Queue",
|
|
"content_type": "ebook",
|
|
}
|
|
)
|
|
|
|
assert success is False
|
|
assert error == "Error queueing release: queue offline"
|
|
|
|
|
|
def _queue_and_capture(monkeypatch, release_data):
|
|
import shelfmark.download.orchestrator as orchestrator
|
|
|
|
captured: dict[str, object] = {}
|
|
|
|
monkeypatch.setattr(orchestrator.config, "get", lambda key, default=None, user_id=None: default)
|
|
monkeypatch.setattr(
|
|
orchestrator.book_queue, "add", lambda task: captured.setdefault("task", task) or True
|
|
)
|
|
monkeypatch.setattr(orchestrator, "ws_manager", None)
|
|
|
|
success, error = orchestrator.queue_release(release_data, user_id=1, username="alice")
|
|
|
|
assert success is True, error
|
|
return captured["task"]
|
|
|
|
|
|
def test_queue_release_carries_top_level_language(monkeypatch):
|
|
task = _queue_and_capture(
|
|
monkeypatch,
|
|
{
|
|
"source": "prowlarr",
|
|
"source_id": "release-sv",
|
|
"title": "Project Hail Mary",
|
|
"language": "sv",
|
|
},
|
|
)
|
|
|
|
assert task.language == "sv"
|
|
|
|
|
|
def test_queue_release_falls_back_to_language_in_extra(monkeypatch):
|
|
# direct_download sets language inside extra as well as top level.
|
|
task = _queue_and_capture(
|
|
monkeypatch,
|
|
{
|
|
"source": "direct_download",
|
|
"source_id": "release-de",
|
|
"title": "Project Hail Mary",
|
|
"extra": {"language": "de"},
|
|
},
|
|
)
|
|
|
|
assert task.language == "de"
|
|
|
|
|
|
def test_queue_release_without_language_leaves_it_unset(monkeypatch):
|
|
task = _queue_and_capture(
|
|
monkeypatch,
|
|
{"source": "prowlarr", "source_id": "release-none", "title": "Project Hail Mary"},
|
|
)
|
|
|
|
assert task.language is None
|