From c269230ef325ed77a3161be966467ae512fdb41a Mon Sep 17 00:00:00 2001 From: CaliBrain Date: Wed, 21 Oct 2026 01:58:41 -0400 Subject: [PATCH] fix: review regressions in metadata, notifications, activity redaction, search options and genDebug (#1510) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - **Open Library (#1451): keep the work unless the search matched a translation.** Open Library shows an English edition whenever one matches the query. So "Mistborn" returned the 2023 "Mistborn" reissue (2023) instead of "The Final Empire" (2001), and "Children of Dune" returned "Children of Dune (Dune Chronicles, Book 3)" (1987). The matched edition's title, ISBNs, cover, language and year now win only when the edition is in another language and the query doesn't name the work's own title. Checked against live Open Library: "Children of Dune" and "Mistborn" give the v1.4.0 results again, and "Der Weg der Könige" and "Kinder des Nebels" still return the German editions. - **Google Books (#1434): the ISBN cache decorator is back on `search_by_isbn`.** It had moved to the new `search_paginated`. Failed searches were cached as empty pages, and ISBN lookups lost their cache. - **Notifications: Test notification shows the plugin's failure reason again.** Apprise 2 delivers on its own worker threads, even for a single service when that service has a deadline. `_AppriseLogCapture` filtered records by thread id and dropped all of them, so the UI only said "json: delivery failed". Apprise runs those workers in a copy of the caller's context, so the capture now filters on a context variable. That still keeps concurrent dispatches apart. `AppriseAsset(async_mode=False)` would not fix this, because the sequential path also uses the worker pool when a service has a deadline. - **Activity: `activity_update` no longer sends the raw task id to non-admins.** The owner's `user_` room got the raw task id. For a request-linked download that id can hold a private tracker's passkey. It now goes through `viewer_download_id`, the same mapping `/api/status` and `download_progress` use. The admins room still gets the raw id. - **SearchBar (#1476): Escape and an outside click close the homepage's inline options panel again,** as they did in v1.4.0. - **genDebug.sh: packages are listed through `importlib.metadata`,** since #1404 removed pip from the image. The script also now redacts: - `PROWLARR_MAM_ID` - `user:pass@` credentials in URLs (proxies), in both the environment and the config files - notification route URLs, which carry their service's token (Telegram, Discord and others) - `make python-checks`, `make python-test`, `make frontend-checks` and `make frontend-test` pass on top of current `main`. - New tests cover: - Open Library edition selection - Google Books caching - log capture against a real Apprise 2 delivery to a local server that answers 500, plus a concurrent dispatch on another thread - the `activity_update` payload sent to each room - genDebug redaction and the package listing Each one fails on the old code. - I checked the SearchBar fix in the running app, because the frontend has no DOM test setup: Escape closes the panel, an outside click closes it, and a click inside keeps it open. - I ran the genDebug redaction under GNU sed (Debian image) and BSD sed. The package listing ran in the pip-less image. --- genDebug.sh | 30 ++- shelfmark/core/notifications.py | 19 +- shelfmark/main.py | 8 + shelfmark/metadata_providers/googlebooks.py | 8 +- shelfmark/metadata_providers/openlibrary.py | 102 ++++++---- src/frontend/src/components/SearchBar.tsx | 6 +- .../core/test_activity_terminal_snapshots.py | 90 +++++++++ tests/core/test_gen_debug_env_filter.py | 86 ++++++++- tests/core/test_notifications.py | 75 ++++++++ tests/metadata/test_googlebooks_parse.py | 30 +++ tests/metadata/test_openlibrary_parse.py | 180 +++++++++++++++++- 11 files changed, 580 insertions(+), 54 deletions(-) diff --git a/genDebug.sh b/genDebug.sh index d4f6f372..cf2c47dd 100755 --- a/genDebug.sh +++ b/genDebug.sh @@ -91,8 +91,14 @@ echo | openssl s_client -connect cloudflare.com:443 2>&1 | grep -E "Verify retur echo "" >> "$LOG_DIR/network_info.txt" # Add installed packages +# The image ships without pip, so ask the app's interpreter directly echo "=== Installed Python Packages ===" > "$LOG_DIR/packages.txt" -pip list >> "$LOG_DIR/packages.txt" 2>&1 || echo "pip not found" >> "$LOG_DIR/packages.txt" +python - >> "$LOG_DIR/packages.txt" 2>&1 <<'PY' || echo "Could not list Python packages" >> "$LOG_DIR/packages.txt" +import importlib.metadata as metadata + +for dist in sorted(metadata.distributions(), key=lambda dist: (dist.name or "").lower()): + print(f"{dist.name}=={dist.version}") +PY echo "" >> "$LOG_DIR/packages.txt" # Check Permissions @@ -133,8 +139,19 @@ else echo "Not running in Docker container" >> "$LOG_DIR/container_info.txt" fi -# Add environment variables (redacting sensitive info) -env | grep -v -E "^[^=]*(_KEY|_SECRET|_PASSWORD|_TOKEN)[^=]*=" | sort > "$LOG_DIR/environment.txt" +# Add environment variables (redacting sensitive info and the credentials in URLs +# such as a proxy's http://user:pass@host:port) +env | grep -v -E "^[^=]*(_KEY|_SECRET|_PASSWORD|_TOKEN|_MAM_ID)[^=]*=" | sed -E 's#(://)[^/@[:space:]]+@#\1[REDACTED]@#g' | sort > "$LOG_DIR/environment.txt" + +# Redact sensitive fields (API keys, passwords, tokens, the MAM session id), the +# credentials in URLs, and notification URLs, which carry their service's token +redact_config_json() { + sed -E \ + -e 's/("(AA_DONATOR_KEY|HARDCOVER_API_KEY|[^"]*_KEY|[^"]*_SECRET|[^"]*_PASSWORD|[^"]*_TOKEN|[^"]*_MAM_ID)"[[:space:]]*:[[:space:]]*")[^"]+"/\1[REDACTED]"/g' \ + -e 's#(://)[^/@[:space:]"]+@#\1[REDACTED]@#g' \ + -e 's#("url"[[:space:]]*:[[:space:]]*"([[:alnum:]+.-]+://)?)[^"]+"#\1[REDACTED]"#g' \ + "$1" +} # Add configuration files (redacting sensitive values) CONFIG_DIR=${CONFIG_DIR:-"/config"} @@ -143,9 +160,7 @@ if [ -d "$CONFIG_DIR" ]; then # Copy and redact main settings file if [ -f "$CONFIG_DIR/settings.json" ]; then - # Redact sensitive fields (API keys, passwords, tokens) - sed -E 's/("(AA_DONATOR_KEY|HARDCOVER_API_KEY|[^"]*_KEY|[^"]*_SECRET|[^"]*_PASSWORD|[^"]*_TOKEN)"[[:space:]]*:[[:space:]]*")[^"]+"/\1[REDACTED]"/g' \ - "$CONFIG_DIR/settings.json" > "$LOG_DIR/config/settings.json" 2>/dev/null + redact_config_json "$CONFIG_DIR/settings.json" > "$LOG_DIR/config/settings.json" 2>/dev/null fi # Copy and redact plugin config files @@ -154,8 +169,7 @@ if [ -d "$CONFIG_DIR" ]; then for config_file in "$CONFIG_DIR/plugins"/*.json; do if [ -f "$config_file" ]; then filename=$(basename "$config_file") - sed -E 's/("(AA_DONATOR_KEY|HARDCOVER_API_KEY|[^"]*_KEY|[^"]*_SECRET|[^"]*_PASSWORD|[^"]*_TOKEN)"[[:space:]]*:[[:space:]]*")[^"]+"/\1[REDACTED]"/g' \ - "$config_file" > "$LOG_DIR/config/plugins/$filename" 2>/dev/null + redact_config_json "$config_file" > "$LOG_DIR/config/plugins/$filename" 2>/dev/null fi done fi diff --git a/shelfmark/core/notifications.py b/shelfmark/core/notifications.py index aa038958..f7f7a6f8 100644 --- a/shelfmark/core/notifications.py +++ b/shelfmark/core/notifications.py @@ -8,6 +8,7 @@ import socket import threading from concurrent.futures import ThreadPoolExecutor from contextlib import contextmanager, suppress +from contextvars import ContextVar from dataclasses import dataclass from enum import StrEnum from typing import TYPE_CHECKING, Any, Protocol, TypeGuard @@ -135,14 +136,22 @@ def _extract_url_schemes(urls: Iterable[str]) -> list[str]: return schemes +# The capture active in the current context. Apprise 2 delivers on its own worker +# threads but runs them in a copy of the caller's context, so a context variable +# (unlike the thread id) still tells this dispatch's records apart from those of +# a dispatch running concurrently on another thread. +_active_apprise_capture: ContextVar[_AppriseLogCapture | None] = ContextVar( + "active_apprise_capture", default=None +) + + class _AppriseLogCapture(logging.Handler): - def __init__(self, *, thread_id: int) -> None: + def __init__(self) -> None: super().__init__(level=logging.INFO) self.records: list[tuple[int, str, str, str]] = [] - self._thread_id = thread_id def emit(self, record: logging.LogRecord) -> None: - if record.thread != self._thread_id: + if _active_apprise_capture.get() is not self: return message = record.getMessage() @@ -164,7 +173,8 @@ def _capture_apprise_logs( ) -> Iterator[list[tuple[int, str, str, str]]]: apprise_logger = logging.getLogger(_APPRISE_LOGGER_NAME) previous_level = apprise_logger.level - handler = _AppriseLogCapture(thread_id=threading.get_ident()) + handler = _AppriseLogCapture() + token = _active_apprise_capture.set(handler) apprise_logger.addHandler(handler) if previous_level == logging.NOTSET or previous_level > min_level: @@ -175,6 +185,7 @@ def _capture_apprise_logs( finally: apprise_logger.removeHandler(handler) apprise_logger.setLevel(previous_level) + _active_apprise_capture.reset(token) def _log_apprise_records(records: Iterable[tuple[int, str, str, str]]) -> None: diff --git a/shelfmark/main.py b/shelfmark/main.py index edf78e89..ec99d5af 100644 --- a/shelfmark/main.py +++ b/shelfmark/main.py @@ -1622,6 +1622,14 @@ def _emit_activity_update_for_task(*, payload: dict[str, Any], task: Any) -> Non ) if owner_user_id is None: return + # The owner knows a request-linked download by its opaque id, as /api/status + # shows it, since the task id can carry a private tracker's passkey (#1418). + task_id = payload.get("task_id") + if isinstance(task_id, str): + payload = { + **payload, + "task_id": viewer_download_id(task_id, getattr(task, "request_id", None)), + } emit_ws_event( ws_manager, event_name="activity_update", diff --git a/shelfmark/metadata_providers/googlebooks.py b/shelfmark/metadata_providers/googlebooks.py index 1cf1bd34..d7d24652 100644 --- a/shelfmark/metadata_providers/googlebooks.py +++ b/shelfmark/metadata_providers/googlebooks.py @@ -214,15 +214,15 @@ class GoogleBooksProvider(MetadataProvider): logger.exception("Google Books get_book error") return None + def search_paginated(self, options: MetadataSearchOptions) -> SearchResult: + """Google returns at most 40 per page, so a full page is 40 whatever limit asked for.""" + return super().search_paginated(replace(options, limit=min(options.limit, 40))) + @cacheable( ttl_key="METADATA_CACHE_BOOK_TTL", ttl_default=600, key_prefix="googlebooks:isbn", ) - def search_paginated(self, options: MetadataSearchOptions) -> SearchResult: - """Google returns at most 40 per page, so a full page is 40 whatever limit asked for.""" - return super().search_paginated(replace(options, limit=min(options.limit, 40))) - def search_by_isbn(self, isbn: str) -> BookMetadata | None: """Search for a book by ISBN-10 or ISBN-13.""" # Clean ISBN (remove hyphens and spaces) diff --git a/shelfmark/metadata_providers/openlibrary.py b/shelfmark/metadata_providers/openlibrary.py index 6ee2925e..f626739d 100644 --- a/shelfmark/metadata_providers/openlibrary.py +++ b/shelfmark/metadata_providers/openlibrary.py @@ -3,6 +3,7 @@ import re import threading import time +import unicodedata from collections import deque from http import HTTPStatus from typing import Any, ClassVar @@ -10,6 +11,7 @@ from typing import Any, ClassVar import requests from shelfmark.core.cache import cacheable +from shelfmark.core.languages import normalize_language from shelfmark.core.logger import setup_logger from shelfmark.core.settings_registry import ( ActionButton, @@ -19,6 +21,7 @@ from shelfmark.core.settings_registry import ( SettingsField, register_settings, ) +from shelfmark.core.text_match import STOPWORDS from shelfmark.download.network import get_ssl_verify from shelfmark.metadata_providers import ( BookMetadata, @@ -103,6 +106,41 @@ SORT_MAPPING: dict[str, str | None] = { } +def _title_words(text: str) -> set[str]: + """Return the significant words of a title, folded for comparison.""" + decomposed = unicodedata.normalize("NFKD", text.casefold()) + folded = "".join(ch for ch in decomposed if not unicodedata.combining(ch)) + return {word for word in re.findall(r"\w+", folded) if len(word) >= 2} - STOPWORDS + + +def _is_translation_match(work_title: str, edition: dict, query: str) -> bool: + """Return whether a search's matched edition is a translation of the work. + + Open Library shows the English edition of a work whenever one satisfies the + query, so an English edition says nothing about which edition the user meant: + searching "Mistborn" picks the 2023 "Mistborn" reissue of "The Final Empire", + and "Children of Dune" may pick "Children of Dune (Dune Chronicles, Book 3)". + An edition in another language comes back when only that language's edition + matched, which is what searching a translated title looks like. A `lang` + preference also brings one back for a search that names the work itself + ("Dune-Zyklus" for "Children of Dune"), so such a search keeps the work. + """ + edition_title = edition.get("title") + if not edition_title or _title_words(edition_title) == _title_words(work_title): + return False + + edition_languages = {normalize_language(code) for code in edition.get("language") or []} + if not edition_languages - {None} or "en" in edition_languages: + return False + + query_words = _title_words(query) + work_words = _title_words(work_title) + names_work = bool(query_words and work_words) and ( + work_words <= query_words or query_words <= work_words + ) + return not names_work + + @register_provider("openlibrary") class OpenLibraryProvider(MetadataProvider): """Open Library metadata provider using REST API.""" @@ -209,7 +247,7 @@ class OpenLibraryProvider(MetadataProvider): data = response.json() for doc in data.get("docs", []): - book = self._parse_search_doc(doc) + book = self._parse_search_doc(doc, title_value or options.query) if book: books.append(book) @@ -331,8 +369,12 @@ class OpenLibraryProvider(MetadataProvider): logger.exception("Open Library ISBN search parsing error") return None - def _parse_search_doc(self, doc: dict) -> BookMetadata | None: - """Parse a search document into BookMetadata.""" + def _parse_search_doc(self, doc: dict, query: str = "") -> BookMetadata | None: + """Parse a search document into BookMetadata. + + ``query`` is what the user searched for (the title field when given), used + to tell a search for a translated title apart from one for the work itself. + """ try: # Extract work ID from key key = doc.get("key", "") @@ -372,44 +414,38 @@ class OpenLibraryProvider(MetadataProvider): # matched the search query (e.g. querying "Der Weg der Könige" matches # the German edition of "The Way of Kings", but the work's title field # still says "The Way of Kings"). The nested `editions` block, requested - # above, is populated with the specific edition whose own fields - # satisfied the query, so prefer its title/isbn/cover/language when one - # is present and differs from the work - otherwise both the result shown - # to the user and the release search query (which uses `search_title` or - # else falls back to `title`, see search_plan._pick_search_title) end up - # in the wrong language even though the user searched for a specific - # translation by name. + # above, holds the edition Open Library picked to show for the query. + # When that edition is a translation, prefer its title/isbn/cover/ + # language - otherwise both the result shown to the user and the + # release search query (which uses `search_title` or else falls back to + # `title`, see search_plan._pick_search_title) end up in the wrong + # language even though the user searched for a specific translation by + # name. Any other edition keeps the work's own title and first year. title = doc["title"] search_title = None publish_year = doc.get("first_publish_year") matched_edition = next(iter(doc.get("editions", {}).get("docs", [])), None) - if matched_edition and matched_edition.get("title"): - edition_title = matched_edition["title"] - if edition_title != title: - search_title = edition_title - title = edition_title + if matched_edition and _is_translation_match(title, matched_edition, query): + search_title = matched_edition["title"] + title = search_title - edition_isbns = matched_edition.get("isbn", []) - edition_isbn_10 = next( - (i for i in edition_isbns if len(i) == ISBN_10_LENGTH), None - ) - edition_isbn_13 = next( - (i for i in edition_isbns if len(i) == ISBN_13_LENGTH), None - ) - isbn_10 = edition_isbn_10 or isbn_10 - isbn_13 = edition_isbn_13 or isbn_13 + edition_isbns = matched_edition.get("isbn", []) + edition_isbn_10 = next((i for i in edition_isbns if len(i) == ISBN_10_LENGTH), None) + edition_isbn_13 = next((i for i in edition_isbns if len(i) == ISBN_13_LENGTH), None) + isbn_10 = edition_isbn_10 or isbn_10 + isbn_13 = edition_isbn_13 or isbn_13 - cover_id = matched_edition.get("cover_i") or cover_id + cover_id = matched_edition.get("cover_i") or cover_id - edition_languages = matched_edition.get("language") or [] - if edition_languages: - language = edition_languages[0] + edition_languages = matched_edition.get("language") or [] + if edition_languages: + language = edition_languages[0] - # A translation is published later than the work it translates, - # e.g. 2015 for "Der Weg der Könige" against 2010 for the work. - edition_years = matched_edition.get("publish_year") or [] - if edition_years: - publish_year = min(edition_years) + # A translation is published later than the work it translates, + # e.g. 2015 for "Der Weg der Könige" against 2010 for the work. + edition_years = matched_edition.get("publish_year") or [] + if edition_years: + publish_year = min(edition_years) cover_url = f"{COVERS_BASE_URL}/b/id/{cover_id}-L.jpg" if cover_id else None diff --git a/src/frontend/src/components/SearchBar.tsx b/src/frontend/src/components/SearchBar.tsx index 32b9e088..c4de3ef0 100644 --- a/src/frontend/src/components/SearchBar.tsx +++ b/src/frontend/src/components/SearchBar.tsx @@ -241,6 +241,7 @@ export const SearchBar = forwardRef( const selectPanelRef = useRef(null); const autocompletePanelRef = useRef(null); const optionsPanelId = useId(); + const optionsPanelRef = useRef(null); const isPopover = optionsLayout === 'popover'; const layoutClasses = OPTIONS_LAYOUT_CLASSES[optionsLayout]; const [isOptionsOpen, setIsOptionsOpen] = useState(false); @@ -270,7 +271,8 @@ export const SearchBar = forwardRef( const combinedSelectorLocked = combinedModeLocked && !isManualTarget; const combinedToggleAvailable = !!onCombinedModeChange && !isManualTarget; - useDismiss(isOptionsOpen && isPopover, [selectorRef], closeOptions); + // The inline panel renders below the bar, outside selectorRef. + useDismiss(isOptionsOpen, [selectorRef, optionsPanelRef], closeOptions); useDismiss(isSelectOpen, [selectPanelRef, selectTriggerRef], () => setIsSelectOpen(false)); useDismiss(isAutocompleteOpen, [autocompletePanelRef, inputRef], () => setIsAutocompleteOpen(false), @@ -628,7 +630,7 @@ export const SearchBar = forwardRef( ) : null; const optionsPanel = ( -
+
{showContentTypeSelector && (
diff --git a/tests/core/test_activity_terminal_snapshots.py b/tests/core/test_activity_terminal_snapshots.py index a4922c1c..f60bce98 100644 --- a/tests/core/test_activity_terminal_snapshots.py +++ b/tests/core/test_activity_terminal_snapshots.py @@ -11,6 +11,7 @@ import pytest from shelfmark.core.models import DownloadTask, QueueStatus from shelfmark.core.notifications import NotificationEvent +from shelfmark.core.viewer_redaction import public_download_id @pytest.fixture(scope="module") @@ -39,6 +40,32 @@ def _read_download_history_row(main_module, task_id: str): conn.close() +def _create_release_request(main_module, user: dict, task_id: str) -> dict: + return main_module.user_db.create_request( + user_id=user["id"], + content_type="ebook", + request_level="release", + policy_mode="request_release", + book_data={ + "title": "Requested Activity", + "author": "Activity Author", + "provider": "openlibrary", + "provider_id": "activity-req", + }, + release_data={"source": "prowlarr", "source_id": task_id, "title": "Requested.epub"}, + status="fulfilled", + delivery_state="queued", + ) + + +def _activity_update_payloads(mock_emit) -> dict[str, dict]: + return { + call.kwargs["to"]: call.args[1] + for call in mock_emit.call_args_list + if call.args[0] == "activity_update" + } + + class TestTerminalSnapshotCapture: def test_complete_transition_records_direct_snapshot(self, main_module): user = _create_user(main_module, prefix="snap-direct") @@ -193,6 +220,35 @@ class TestTerminalSnapshotCapture: finally: main_module.backend.book_queue.cancel_download(task_id) + def test_activity_update_hides_request_linked_task_id_from_owner(self, main_module): + user = _create_user(main_module, prefix="snap-activity-redact") + task_id = f"7:https://tracker.example/download/1?passkey={uuid.uuid4().hex[:8]}" + request_row = _create_release_request(main_module, user, task_id) + task = DownloadTask( + task_id=task_id, + source="prowlarr", + title="Requested Activity", + user_id=user["id"], + username=user["username"], + request_id=request_row["id"], + ) + assert main_module.backend.book_queue.add(task) is True + + try: + with patch.object(main_module.ws_manager, "is_enabled", return_value=True): + with patch.object(main_module.ws_manager.socketio, "emit") as mock_emit: + main_module.backend.book_queue.update_status(task_id, QueueStatus.COMPLETE) + + payloads = _activity_update_payloads(mock_emit) + assert payloads["admins"]["task_id"] == task_id + assert payloads[f"user_{user['id']}"] == { + "kind": "download_terminal", + "task_id": public_download_id(task_id), + "status": "complete", + } + finally: + main_module.backend.book_queue.cancel_download(task_id) + def test_error_transition_triggers_download_failed_notification(self, main_module): user = _create_user(main_module, prefix="snap-notify-error") task_id = f"notify-error-{uuid.uuid4().hex[:8]}" @@ -420,6 +476,40 @@ class TestTerminalSnapshotCapture: to=f"user_{user['id']}", ) + def test_queue_hook_reset_hides_request_linked_task_id_from_owner(self, main_module): + user = _create_user(main_module, prefix="snap-reset-redact") + task_id = f"7:https://tracker.example/download/2?passkey={uuid.uuid4().hex[:8]}" + request_row = _create_release_request(main_module, user, task_id) + main_module.activity_view_state_service.dismiss( + viewer_scope=f"user:{user['id']}", + item_type="download", + item_key=f"download:{task_id}", + ) + + task = SimpleNamespace( + user_id=user["id"], + username=user["username"], + request_id=request_row["id"], + source="prowlarr", + title="Requested Activity", + author="Activity Author", + format="epub", + size="1 MB", + preview=None, + content_type="ebook", + ) + + with patch.object(main_module.ws_manager, "is_enabled", return_value=True): + with patch.object(main_module.ws_manager.socketio, "emit") as mock_emit: + main_module._record_download_queued(task_id, task) + + payloads = _activity_update_payloads(mock_emit) + assert payloads["admins"] == {"kind": "activity_reset", "task_id": task_id} + assert payloads[f"user_{user['id']}"] == { + "kind": "activity_reset", + "task_id": public_download_id(task_id), + } + def test_cancelled_transition_does_not_trigger_notification(self, main_module): user = _create_user(main_module, prefix="snap-notify-cancel") task_id = f"notify-cancel-{uuid.uuid4().hex[:8]}" diff --git a/tests/core/test_gen_debug_env_filter.py b/tests/core/test_gen_debug_env_filter.py index cdc72109..0ef31bd6 100644 --- a/tests/core/test_gen_debug_env_filter.py +++ b/tests/core/test_gen_debug_env_filter.py @@ -4,14 +4,16 @@ People attach the debug bundle to public issues, so the filter is tested by runn real pipeline line from the script against a controlled environment. """ +import json import re import subprocess +import sys from pathlib import Path SCRIPT = Path(__file__).resolve().parents[2] / "genDebug.sh" -def _filtered_env(env: dict[str, str]) -> set[str]: +def _filtered_env_lines(env: dict[str, str]) -> list[str]: line = next( line for line in SCRIPT.read_text().splitlines() if line.startswith("env | grep -v") ) @@ -23,7 +25,25 @@ def _filtered_env(env: dict[str, str]) -> set[str]: text=True, check=True, ).stdout - return {entry.split("=", 1)[0] for entry in out.splitlines()} + return out.splitlines() + + +def _filtered_env(env: dict[str, str]) -> set[str]: + return {entry.split("=", 1)[0] for entry in _filtered_env_lines(env)} + + +def _redacted_config(tmp_path: Path, settings: dict[str, object]) -> dict[str, object]: + match = re.search(r"^redact_config_json\(\) \{\n.*?^\}\n", SCRIPT.read_text(), re.M | re.S) + assert match is not None + config_file = tmp_path / "settings.json" + config_file.write_text(json.dumps(settings, indent=2)) + out = subprocess.run( + ["bash", "-c", f'{match.group(0)}redact_config_json "$1"', "bash", str(config_file)], + capture_output=True, + text=True, + check=True, + ).stdout + return json.loads(out) def test_secret_variables_are_left_out() -> None: @@ -37,6 +57,7 @@ def test_secret_variables_are_left_out() -> None: "SOME_SECRET": "e", "DB_PASSWORD_FILE": "f", "SERVICE_TOKEN_VALUE": "g", + "PROWLARR_MAM_ID": "h", } ) @@ -49,6 +70,7 @@ def test_secret_variables_are_left_out() -> None: "SOME_SECRET", "DB_PASSWORD_FILE", "SERVICE_TOKEN_VALUE", + "PROWLARR_MAM_ID", } ) @@ -59,3 +81,63 @@ def test_ordinary_variables_are_kept() -> None: ) assert {"TZ", "PUID", "FLASK_PORT"} <= names + + +def test_proxy_credentials_are_redacted() -> None: + lines = _filtered_env_lines( + { + "PATH": "/usr/bin:/bin", + "HTTP_PROXY": "http://alice:s3cret@proxy.local:3128", + "HTTPS_PROXY": "socks5://bob@10.0.0.1:1080", + "NO_PROXY": "localhost", + } + ) + + assert "HTTP_PROXY=http://[REDACTED]@proxy.local:3128" in lines + assert "HTTPS_PROXY=socks5://[REDACTED]@10.0.0.1:1080" in lines + assert "NO_PROXY=localhost" in lines + + +def test_config_secrets_are_redacted(tmp_path: Path) -> None: + redacted = _redacted_config( + tmp_path, + { + "PROWLARR_API_KEY": "key", + "PROWLARR_MAM_ID": "mam-session", + "QBITTORRENT_PASSWORD": "password", + "HTTP_PROXY": "http://alice:s3cret@proxy.local:3128", + "PROWLARR_URL": "http://prowlarr:9696", + "ADMIN_NOTIFICATION_ROUTES": [ + {"event": "all", "url": "tgram://123456:bot-token/987"}, + {"event": "download_failed", "url": "discord://webhook-id/webhook-token"}, + ], + "BOOK_LANGUAGE": ["en"], + }, + ) + + assert redacted == { + "PROWLARR_API_KEY": "[REDACTED]", + "PROWLARR_MAM_ID": "[REDACTED]", + "QBITTORRENT_PASSWORD": "[REDACTED]", + "HTTP_PROXY": "http://[REDACTED]@proxy.local:3128", + "PROWLARR_URL": "http://prowlarr:9696", + "ADMIN_NOTIFICATION_ROUTES": [ + {"event": "all", "url": "tgram://[REDACTED]"}, + {"event": "download_failed", "url": "discord://[REDACTED]"}, + ], + "BOOK_LANGUAGE": ["en"], + } + + +def test_package_listing_does_not_need_pip() -> None: + """The image ships without pip, so packages are listed through importlib.metadata.""" + match = re.search(r"<<'PY'.*?\n(.*?)^PY$", SCRIPT.read_text(), re.M | re.S) + assert match is not None + out = subprocess.run( + [sys.executable, "-I", "-c", match.group(1)], + capture_output=True, + text=True, + check=True, + ).stdout + + assert any(line.startswith("pytest==") for line in out.splitlines()) diff --git a/tests/core/test_notifications.py b/tests/core/test_notifications.py index 95ededee..d07ea655 100644 --- a/tests/core/test_notifications.py +++ b/tests/core/test_notifications.py @@ -1,6 +1,11 @@ """Tests for core notification rendering and dispatch helpers.""" +import contextvars import logging +import threading +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + +import apprise from shelfmark.core import notifications as notifications_module @@ -366,6 +371,76 @@ def test_dispatch_to_apprise_logs_captured_apprise_warning_messages(monkeypatch) ) +class _ServerErrorHandler(BaseHTTPRequestHandler): + def do_POST(self): + self.send_response(500) + self.end_headers() + + def log_message(self, format, *args): # noqa: A002 + return + + +def test_dispatch_to_apprise_reports_failure_logged_on_apprise_worker_thread(): + """Apprise 2 delivers on its own worker threads, which log the plugin's reason.""" + server = ThreadingHTTPServer(("127.0.0.1", 0), _ServerErrorHandler) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + try: + result = notifications_module._dispatch_to_apprise( + [f"json://127.0.0.1:{server.server_address[1]}/hook"], + title="Test", + body="Body", + notify_type=apprise.NotifyType.INFO, + ) + finally: + server.shutdown() + server.server_close() + thread.join(timeout=5) + + assert result["success"] is False + assert len(result["details"]) == 1 + assert result["details"][0].startswith("json: ") + assert "error=500" in result["details"][0] + + +def test_dispatch_to_apprise_ignores_apprise_records_of_other_dispatches(monkeypatch): + fake_apprise = _FakeAppriseModule() + monkeypatch.setattr(notifications_module, "apprise", fake_apprise) + + def _log_warning(message): + logging.getLogger("apprise.plugins.json").warning(message) + + def _notify(**kwargs): + _ = kwargs + # A dispatch running concurrently on another thread, in its own context. + other = threading.Thread( + target=_log_warning, + args=("Another dispatch failed",), + context=contextvars.Context(), + ) + other.start() + other.join() + # Apprise's worker thread, running in a copy of this dispatch's context. + worker = threading.Thread( + target=contextvars.copy_context().run, + args=(_log_warning, "This dispatch failed"), + ) + worker.start() + worker.join() + return False + + monkeypatch.setattr(fake_apprise.client, "notify", _notify) + + result = notifications_module._dispatch_to_apprise( + ["json://example.invalid/hook"], + title="Test", + body="Body", + notify_type=_FakeNotifyType.INFO, + ) + + assert result["details"] == ["json: apprise.plugins.json: This dispatch failed"] + + def test_dispatch_to_apprise_logs_add_exception_at_debug_with_trace(monkeypatch): fake_apprise = _FakeAppriseModule() fake_apprise.client.instantiate_exceptions_by_url = { diff --git a/tests/metadata/test_googlebooks_parse.py b/tests/metadata/test_googlebooks_parse.py index b45e4b61..ee719dbc 100644 --- a/tests/metadata/test_googlebooks_parse.py +++ b/tests/metadata/test_googlebooks_parse.py @@ -63,6 +63,36 @@ def test_googlebooks_search_does_not_cache_request_failures(): assert [book.title for book in result] == ["Recovered Book"] +def test_googlebooks_paginated_search_does_not_cache_request_failures(): + get_metadata_cache().clear() + provider = GoogleBooksProvider(api_key="test-key") + session = _FlakyGoogleBooksSession() + provider.session = session + options = MetadataSearchOptions(query="Recovered Book") + + assert provider.search_paginated(options).books == [] + + result = provider.search_paginated(options) + + assert session.calls == 2 + assert [book.title for book in result.books] == ["Recovered Book"] + + +def test_googlebooks_isbn_lookup_is_cached(): + get_metadata_cache().clear() + provider = GoogleBooksProvider(api_key="test-key") + session = _FlakyGoogleBooksSession() + session.calls = 1 # skip the simulated timeout + provider.session = session + + first = provider.search_by_isbn("978-0-00-000000-2") + second = provider.search_by_isbn("978-0-00-000000-2") + + assert first is not None + assert second == first + assert session.calls == 2 + + def test_googlebooks_pages_tile_when_limit_exceeds_api_maximum(): get_metadata_cache().clear() provider = GoogleBooksProvider(api_key="test-key") diff --git a/tests/metadata/test_openlibrary_parse.py b/tests/metadata/test_openlibrary_parse.py index 177397dd..6ac0468b 100644 --- a/tests/metadata/test_openlibrary_parse.py +++ b/tests/metadata/test_openlibrary_parse.py @@ -61,7 +61,13 @@ class TestOpenLibraryParseSearchDoc: "author_name": ["Brandon Sanderson"], "first_publish_year": 2010, "editions": { - "docs": [{"key": "/books/OL32771812M", "title": "Der Weg der Könige"}] + "docs": [ + { + "key": "/books/OL32771812M", + "title": "Der Weg der Könige", + "language": ["ger"], + } + ] }, } ) @@ -116,3 +122,175 @@ class TestOpenLibraryParseSearchDoc: assert result is not None assert result.search_title is None assert result.isbn_13 == "9780765326355" + + def test_keeps_work_for_an_english_edition_with_another_title(self): + """Open Library shows an English edition whenever one matches the query. + + Searching "Mistborn" picks the 2023 "Mistborn" reissue of "The Final + Empire", which is the same book under a different English title, so the + result keeps the work's title, ISBNs and first publication year. + """ + provider = OpenLibraryProvider() + + result = provider._parse_search_doc( + { + "key": "/works/OL5738148W", + "title": "The Final Empire", + "author_name": ["Brandon Sanderson"], + "isbn": ["9780765311788"], + "cover_i": 14658160, + "language": ["fre", "ger", "eng"], + "first_publish_year": 2001, + "editions": { + "docs": [ + { + "key": "/books/OL62594701M", + "title": "Mistborn", + "language": ["eng"], + "isbn": ["9781250318541"], + "cover_i": 1, + "publish_year": [2023], + } + ] + }, + }, + "Mistborn", + ) + + assert result is not None + assert result.title == "The Final Empire" + assert result.search_title is None + assert result.publish_year == 2001 + assert result.isbn_13 == "9780765311788" + assert result.cover_url == "https://covers.openlibrary.org/b/id/14658160-L.jpg" + + def test_keeps_work_for_a_subtitled_reprint(self): + provider = OpenLibraryProvider() + + result = provider._parse_search_doc( + { + "key": "/works/OL893516W", + "title": "Children of Dune", + "author_name": ["Frank Herbert"], + "first_publish_year": 1976, + "editions": { + "docs": [ + { + "key": "/books/OL1M", + "title": "Children of Dune (Dune Chronicles, Book 3)", + "language": ["eng"], + "publish_year": [1987], + } + ] + }, + }, + "Children of Dune", + ) + + assert result is not None + assert result.title == "Children of Dune" + assert result.search_title is None + assert result.publish_year == 1976 + + def test_keeps_work_when_the_query_names_the_work(self): + """A `lang` preference shows a translation even for the work's own title.""" + provider = OpenLibraryProvider() + + result = provider._parse_search_doc( + { + "key": "/works/OL893516W", + "title": "Children of Dune", + "author_name": ["Frank Herbert"], + "first_publish_year": 1976, + "editions": { + "docs": [ + { + "key": "/books/OL2M", + "title": "Dune-Zyklus", + "language": ["ger"], + "publish_year": [1985], + } + ] + }, + }, + "children of dune", + ) + + assert result is not None + assert result.title == "Children of Dune" + assert result.search_title is None + assert result.publish_year == 1976 + + def test_keeps_work_for_an_edition_of_unknown_language(self): + provider = OpenLibraryProvider() + + result = provider._parse_search_doc( + { + "key": "/works/OL5738148W", + "title": "The Final Empire", + "author_name": ["Brandon Sanderson"], + "first_publish_year": 2001, + "editions": { + "docs": [{"key": "/books/OL3M", "title": "Mistborn", "publish_year": [2023]}] + }, + }, + "Mistborn", + ) + + assert result is not None + assert result.title == "The Final Empire" + assert result.publish_year == 2001 + + def test_ignores_edition_differing_only_in_case_and_accents(self): + provider = OpenLibraryProvider() + + result = provider._parse_search_doc( + { + "key": "/works/OL1W", + "title": "L’étranger", + "author_name": ["Albert Camus"], + "first_publish_year": 1942, + "editions": { + "docs": [ + { + "key": "/books/OL4M", + "title": "L'Etranger", + "language": ["fre"], + "publish_year": [1995], + } + ] + }, + } + ) + + assert result is not None + assert result.title == "L’étranger" + assert result.publish_year == 1942 + + def test_prefers_translation_matched_by_query(self): + provider = OpenLibraryProvider() + + result = provider._parse_search_doc( + { + "key": "/works/OL5738148W", + "title": "The Final Empire", + "author_name": ["Brandon Sanderson"], + "first_publish_year": 2001, + "editions": { + "docs": [ + { + "key": "/books/OL5M", + "title": "Kinder des Nebels", + "language": ["ger"], + "publish_year": [2018], + } + ] + }, + }, + "Kinder des Nebels", + ) + + assert result is not None + assert result.title == "Kinder des Nebels" + assert result.search_title == "Kinder des Nebels" + assert result.publish_year == 2018