mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-11-18 00:10:35 +00:00
fix: review regressions in metadata, notifications, activity redaction, search options and genDebug (#1510)
- **Open Library (#1451): keep the work unless the search matched a translation.** Open Library shows an English edition whenever one matches the query. So "Mistborn" returned the 2023 "Mistborn" reissue (2023) instead of "The Final Empire" (2001), and "Children of Dune" returned "Children of Dune (Dune Chronicles, Book 3)" (1987). The matched edition's title, ISBNs, cover, language and year now win only when the edition is in another language and the query doesn't name the work's own title. Checked against live Open Library: "Children of Dune" and "Mistborn" give the v1.4.0 results again, and "Der Weg der Könige" and "Kinder des Nebels" still return the German editions. - **Google Books (#1434): the ISBN cache decorator is back on `search_by_isbn`.** It had moved to the new `search_paginated`. Failed searches were cached as empty pages, and ISBN lookups lost their cache. - **Notifications: Test notification shows the plugin's failure reason again.** Apprise 2 delivers on its own worker threads, even for a single service when that service has a deadline. `_AppriseLogCapture` filtered records by thread id and dropped all of them, so the UI only said "json: delivery failed". Apprise runs those workers in a copy of the caller's context, so the capture now filters on a context variable. That still keeps concurrent dispatches apart. `AppriseAsset(async_mode=False)` would not fix this, because the sequential path also uses the worker pool when a service has a deadline. - **Activity: `activity_update` no longer sends the raw task id to non-admins.** The owner's `user_<id>` room got the raw task id. For a request-linked download that id can hold a private tracker's passkey. It now goes through `viewer_download_id`, the same mapping `/api/status` and `download_progress` use. The admins room still gets the raw id. - **SearchBar (#1476): Escape and an outside click close the homepage's inline options panel again,** as they did in v1.4.0. - **genDebug.sh: packages are listed through `importlib.metadata`,** since #1404 removed pip from the image. The script also now redacts: - `PROWLARR_MAM_ID` - `user:pass@` credentials in URLs (proxies), in both the environment and the config files - notification route URLs, which carry their service's token (Telegram, Discord and others) - `make python-checks`, `make python-test`, `make frontend-checks` and `make frontend-test` pass on top of current `main`. - New tests cover: - Open Library edition selection - Google Books caching - log capture against a real Apprise 2 delivery to a local server that answers 500, plus a concurrent dispatch on another thread - the `activity_update` payload sent to each room - genDebug redaction and the package listing Each one fails on the old code. - I checked the SearchBar fix in the running app, because the frontend has no DOM test setup: Escape closes the panel, an outside click closes it, and a click inside keeps it open. - I ran the genDebug redaction under GNU sed (Debian image) and BSD sed. The package listing ran in the pip-less image.
This commit is contained in:
1 parent
a00f3d28bf
commit
c269230ef3
11 files changed
+580
-54
No files matched your search
+22
-8
@@ -91,8 +91,14 @@ echo | openssl s_client -connect cloudflare.com:443 2>&1 | grep -E "Verify retur
|
||||
echo "" >> "$LOG_DIR/network_info.txt"
|
||||
|
||||
# Add installed packages
|
||||
# The image ships without pip, so ask the app's interpreter directly
|
||||
echo "=== Installed Python Packages ===" > "$LOG_DIR/packages.txt"
|
||||
pip list >> "$LOG_DIR/packages.txt" 2>&1 || echo "pip not found" >> "$LOG_DIR/packages.txt"
|
||||
python - >> "$LOG_DIR/packages.txt" 2>&1 <<'PY' || echo "Could not list Python packages" >> "$LOG_DIR/packages.txt"
|
||||
import importlib.metadata as metadata
|
||||
|
||||
for dist in sorted(metadata.distributions(), key=lambda dist: (dist.name or "").lower()):
|
||||
print(f"{dist.name}=={dist.version}")
|
||||
PY
|
||||
echo "" >> "$LOG_DIR/packages.txt"
|
||||
|
||||
# Check Permissions
|
||||
@@ -133,8 +139,19 @@ else
|
||||
echo "Not running in Docker container" >> "$LOG_DIR/container_info.txt"
|
||||
fi
|
||||
|
||||
# Add environment variables (redacting sensitive info)
|
||||
env | grep -v -E "^[^=]*(_KEY|_SECRET|_PASSWORD|_TOKEN)[^=]*=" | sort > "$LOG_DIR/environment.txt"
|
||||
# Add environment variables (redacting sensitive info and the credentials in URLs
|
||||
# such as a proxy's http://user:pass@host:port)
|
||||
env | grep -v -E "^[^=]*(_KEY|_SECRET|_PASSWORD|_TOKEN|_MAM_ID)[^=]*=" | sed -E 's#(://)[^/@[:space:]]+@#\1[REDACTED]@#g' | sort > "$LOG_DIR/environment.txt"
|
||||
|
||||
# Redact sensitive fields (API keys, passwords, tokens, the MAM session id), the
|
||||
# credentials in URLs, and notification URLs, which carry their service's token
|
||||
redact_config_json() {
|
||||
sed -E \
|
||||
-e 's/("(AA_DONATOR_KEY|HARDCOVER_API_KEY|[^"]*_KEY|[^"]*_SECRET|[^"]*_PASSWORD|[^"]*_TOKEN|[^"]*_MAM_ID)"[[:space:]]*:[[:space:]]*")[^"]+"/\1[REDACTED]"/g' \
|
||||
-e 's#(://)[^/@[:space:]"]+@#\1[REDACTED]@#g' \
|
||||
-e 's#("url"[[:space:]]*:[[:space:]]*"([[:alnum:]+.-]+://)?)[^"]+"#\1[REDACTED]"#g' \
|
||||
"$1"
|
||||
}
|
||||
|
||||
# Add configuration files (redacting sensitive values)
|
||||
CONFIG_DIR=${CONFIG_DIR:-"/config"}
|
||||
@@ -143,9 +160,7 @@ if [ -d "$CONFIG_DIR" ]; then
|
||||
|
||||
# Copy and redact main settings file
|
||||
if [ -f "$CONFIG_DIR/settings.json" ]; then
|
||||
# Redact sensitive fields (API keys, passwords, tokens)
|
||||
sed -E 's/("(AA_DONATOR_KEY|HARDCOVER_API_KEY|[^"]*_KEY|[^"]*_SECRET|[^"]*_PASSWORD|[^"]*_TOKEN)"[[:space:]]*:[[:space:]]*")[^"]+"/\1[REDACTED]"/g' \
|
||||
"$CONFIG_DIR/settings.json" > "$LOG_DIR/config/settings.json" 2>/dev/null
|
||||
redact_config_json "$CONFIG_DIR/settings.json" > "$LOG_DIR/config/settings.json" 2>/dev/null
|
||||
fi
|
||||
|
||||
# Copy and redact plugin config files
|
||||
@@ -154,8 +169,7 @@ if [ -d "$CONFIG_DIR" ]; then
|
||||
for config_file in "$CONFIG_DIR/plugins"/*.json; do
|
||||
if [ -f "$config_file" ]; then
|
||||
filename=$(basename "$config_file")
|
||||
sed -E 's/("(AA_DONATOR_KEY|HARDCOVER_API_KEY|[^"]*_KEY|[^"]*_SECRET|[^"]*_PASSWORD|[^"]*_TOKEN)"[[:space:]]*:[[:space:]]*")[^"]+"/\1[REDACTED]"/g' \
|
||||
"$config_file" > "$LOG_DIR/config/plugins/$filename" 2>/dev/null
|
||||
redact_config_json "$config_file" > "$LOG_DIR/config/plugins/$filename" 2>/dev/null
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
@@ -8,6 +8,7 @@ import socket
|
||||
import threading
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from contextlib import contextmanager, suppress
|
||||
from contextvars import ContextVar
|
||||
from dataclasses import dataclass
|
||||
from enum import StrEnum
|
||||
from typing import TYPE_CHECKING, Any, Protocol, TypeGuard
|
||||
@@ -135,14 +136,22 @@ def _extract_url_schemes(urls: Iterable[str]) -> list[str]:
|
||||
return schemes
|
||||
|
||||
|
||||
# The capture active in the current context. Apprise 2 delivers on its own worker
|
||||
# threads but runs them in a copy of the caller's context, so a context variable
|
||||
# (unlike the thread id) still tells this dispatch's records apart from those of
|
||||
# a dispatch running concurrently on another thread.
|
||||
_active_apprise_capture: ContextVar[_AppriseLogCapture | None] = ContextVar(
|
||||
"active_apprise_capture", default=None
|
||||
)
|
||||
|
||||
|
||||
class _AppriseLogCapture(logging.Handler):
|
||||
def __init__(self, *, thread_id: int) -> None:
|
||||
def __init__(self) -> None:
|
||||
super().__init__(level=logging.INFO)
|
||||
self.records: list[tuple[int, str, str, str]] = []
|
||||
self._thread_id = thread_id
|
||||
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
if record.thread != self._thread_id:
|
||||
if _active_apprise_capture.get() is not self:
|
||||
return
|
||||
|
||||
message = record.getMessage()
|
||||
@@ -164,7 +173,8 @@ def _capture_apprise_logs(
|
||||
) -> Iterator[list[tuple[int, str, str, str]]]:
|
||||
apprise_logger = logging.getLogger(_APPRISE_LOGGER_NAME)
|
||||
previous_level = apprise_logger.level
|
||||
handler = _AppriseLogCapture(thread_id=threading.get_ident())
|
||||
handler = _AppriseLogCapture()
|
||||
token = _active_apprise_capture.set(handler)
|
||||
apprise_logger.addHandler(handler)
|
||||
|
||||
if previous_level == logging.NOTSET or previous_level > min_level:
|
||||
@@ -175,6 +185,7 @@ def _capture_apprise_logs(
|
||||
finally:
|
||||
apprise_logger.removeHandler(handler)
|
||||
apprise_logger.setLevel(previous_level)
|
||||
_active_apprise_capture.reset(token)
|
||||
|
||||
|
||||
def _log_apprise_records(records: Iterable[tuple[int, str, str, str]]) -> None:
|
||||
|
||||
@@ -1622,6 +1622,14 @@ def _emit_activity_update_for_task(*, payload: dict[str, Any], task: Any) -> Non
|
||||
)
|
||||
if owner_user_id is None:
|
||||
return
|
||||
# The owner knows a request-linked download by its opaque id, as /api/status
|
||||
# shows it, since the task id can carry a private tracker's passkey (#1418).
|
||||
task_id = payload.get("task_id")
|
||||
if isinstance(task_id, str):
|
||||
payload = {
|
||||
**payload,
|
||||
"task_id": viewer_download_id(task_id, getattr(task, "request_id", None)),
|
||||
}
|
||||
emit_ws_event(
|
||||
ws_manager,
|
||||
event_name="activity_update",
|
||||
|
||||
@@ -214,15 +214,15 @@ class GoogleBooksProvider(MetadataProvider):
|
||||
logger.exception("Google Books get_book error")
|
||||
return None
|
||||
|
||||
def search_paginated(self, options: MetadataSearchOptions) -> SearchResult:
|
||||
"""Google returns at most 40 per page, so a full page is 40 whatever limit asked for."""
|
||||
return super().search_paginated(replace(options, limit=min(options.limit, 40)))
|
||||
|
||||
@cacheable(
|
||||
ttl_key="METADATA_CACHE_BOOK_TTL",
|
||||
ttl_default=600,
|
||||
key_prefix="googlebooks:isbn",
|
||||
)
|
||||
def search_paginated(self, options: MetadataSearchOptions) -> SearchResult:
|
||||
"""Google returns at most 40 per page, so a full page is 40 whatever limit asked for."""
|
||||
return super().search_paginated(replace(options, limit=min(options.limit, 40)))
|
||||
|
||||
def search_by_isbn(self, isbn: str) -> BookMetadata | None:
|
||||
"""Search for a book by ISBN-10 or ISBN-13."""
|
||||
# Clean ISBN (remove hyphens and spaces)
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
import re
|
||||
import threading
|
||||
import time
|
||||
import unicodedata
|
||||
from collections import deque
|
||||
from http import HTTPStatus
|
||||
from typing import Any, ClassVar
|
||||
@@ -10,6 +11,7 @@ from typing import Any, ClassVar
|
||||
import requests
|
||||
|
||||
from shelfmark.core.cache import cacheable
|
||||
from shelfmark.core.languages import normalize_language
|
||||
from shelfmark.core.logger import setup_logger
|
||||
from shelfmark.core.settings_registry import (
|
||||
ActionButton,
|
||||
@@ -19,6 +21,7 @@ from shelfmark.core.settings_registry import (
|
||||
SettingsField,
|
||||
register_settings,
|
||||
)
|
||||
from shelfmark.core.text_match import STOPWORDS
|
||||
from shelfmark.download.network import get_ssl_verify
|
||||
from shelfmark.metadata_providers import (
|
||||
BookMetadata,
|
||||
@@ -103,6 +106,41 @@ SORT_MAPPING: dict[str, str | None] = {
|
||||
}
|
||||
|
||||
|
||||
def _title_words(text: str) -> set[str]:
|
||||
"""Return the significant words of a title, folded for comparison."""
|
||||
decomposed = unicodedata.normalize("NFKD", text.casefold())
|
||||
folded = "".join(ch for ch in decomposed if not unicodedata.combining(ch))
|
||||
return {word for word in re.findall(r"\w+", folded) if len(word) >= 2} - STOPWORDS
|
||||
|
||||
|
||||
def _is_translation_match(work_title: str, edition: dict, query: str) -> bool:
|
||||
"""Return whether a search's matched edition is a translation of the work.
|
||||
|
||||
Open Library shows the English edition of a work whenever one satisfies the
|
||||
query, so an English edition says nothing about which edition the user meant:
|
||||
searching "Mistborn" picks the 2023 "Mistborn" reissue of "The Final Empire",
|
||||
and "Children of Dune" may pick "Children of Dune (Dune Chronicles, Book 3)".
|
||||
An edition in another language comes back when only that language's edition
|
||||
matched, which is what searching a translated title looks like. A `lang`
|
||||
preference also brings one back for a search that names the work itself
|
||||
("Dune-Zyklus" for "Children of Dune"), so such a search keeps the work.
|
||||
"""
|
||||
edition_title = edition.get("title")
|
||||
if not edition_title or _title_words(edition_title) == _title_words(work_title):
|
||||
return False
|
||||
|
||||
edition_languages = {normalize_language(code) for code in edition.get("language") or []}
|
||||
if not edition_languages - {None} or "en" in edition_languages:
|
||||
return False
|
||||
|
||||
query_words = _title_words(query)
|
||||
work_words = _title_words(work_title)
|
||||
names_work = bool(query_words and work_words) and (
|
||||
work_words <= query_words or query_words <= work_words
|
||||
)
|
||||
return not names_work
|
||||
|
||||
|
||||
@register_provider("openlibrary")
|
||||
class OpenLibraryProvider(MetadataProvider):
|
||||
"""Open Library metadata provider using REST API."""
|
||||
@@ -209,7 +247,7 @@ class OpenLibraryProvider(MetadataProvider):
|
||||
data = response.json()
|
||||
|
||||
for doc in data.get("docs", []):
|
||||
book = self._parse_search_doc(doc)
|
||||
book = self._parse_search_doc(doc, title_value or options.query)
|
||||
if book:
|
||||
books.append(book)
|
||||
|
||||
@@ -331,8 +369,12 @@ class OpenLibraryProvider(MetadataProvider):
|
||||
logger.exception("Open Library ISBN search parsing error")
|
||||
return None
|
||||
|
||||
def _parse_search_doc(self, doc: dict) -> BookMetadata | None:
|
||||
"""Parse a search document into BookMetadata."""
|
||||
def _parse_search_doc(self, doc: dict, query: str = "") -> BookMetadata | None:
|
||||
"""Parse a search document into BookMetadata.
|
||||
|
||||
``query`` is what the user searched for (the title field when given), used
|
||||
to tell a search for a translated title apart from one for the work itself.
|
||||
"""
|
||||
try:
|
||||
# Extract work ID from key
|
||||
key = doc.get("key", "")
|
||||
@@ -372,44 +414,38 @@ class OpenLibraryProvider(MetadataProvider):
|
||||
# matched the search query (e.g. querying "Der Weg der Könige" matches
|
||||
# the German edition of "The Way of Kings", but the work's title field
|
||||
# still says "The Way of Kings"). The nested `editions` block, requested
|
||||
# above, is populated with the specific edition whose own fields
|
||||
# satisfied the query, so prefer its title/isbn/cover/language when one
|
||||
# is present and differs from the work - otherwise both the result shown
|
||||
# to the user and the release search query (which uses `search_title` or
|
||||
# else falls back to `title`, see search_plan._pick_search_title) end up
|
||||
# in the wrong language even though the user searched for a specific
|
||||
# translation by name.
|
||||
# above, holds the edition Open Library picked to show for the query.
|
||||
# When that edition is a translation, prefer its title/isbn/cover/
|
||||
# language - otherwise both the result shown to the user and the
|
||||
# release search query (which uses `search_title` or else falls back to
|
||||
# `title`, see search_plan._pick_search_title) end up in the wrong
|
||||
# language even though the user searched for a specific translation by
|
||||
# name. Any other edition keeps the work's own title and first year.
|
||||
title = doc["title"]
|
||||
search_title = None
|
||||
publish_year = doc.get("first_publish_year")
|
||||
matched_edition = next(iter(doc.get("editions", {}).get("docs", [])), None)
|
||||
if matched_edition and matched_edition.get("title"):
|
||||
edition_title = matched_edition["title"]
|
||||
if edition_title != title:
|
||||
search_title = edition_title
|
||||
title = edition_title
|
||||
if matched_edition and _is_translation_match(title, matched_edition, query):
|
||||
search_title = matched_edition["title"]
|
||||
title = search_title
|
||||
|
||||
edition_isbns = matched_edition.get("isbn", [])
|
||||
edition_isbn_10 = next(
|
||||
(i for i in edition_isbns if len(i) == ISBN_10_LENGTH), None
|
||||
)
|
||||
edition_isbn_13 = next(
|
||||
(i for i in edition_isbns if len(i) == ISBN_13_LENGTH), None
|
||||
)
|
||||
isbn_10 = edition_isbn_10 or isbn_10
|
||||
isbn_13 = edition_isbn_13 or isbn_13
|
||||
edition_isbns = matched_edition.get("isbn", [])
|
||||
edition_isbn_10 = next((i for i in edition_isbns if len(i) == ISBN_10_LENGTH), None)
|
||||
edition_isbn_13 = next((i for i in edition_isbns if len(i) == ISBN_13_LENGTH), None)
|
||||
isbn_10 = edition_isbn_10 or isbn_10
|
||||
isbn_13 = edition_isbn_13 or isbn_13
|
||||
|
||||
cover_id = matched_edition.get("cover_i") or cover_id
|
||||
cover_id = matched_edition.get("cover_i") or cover_id
|
||||
|
||||
edition_languages = matched_edition.get("language") or []
|
||||
if edition_languages:
|
||||
language = edition_languages[0]
|
||||
edition_languages = matched_edition.get("language") or []
|
||||
if edition_languages:
|
||||
language = edition_languages[0]
|
||||
|
||||
# A translation is published later than the work it translates,
|
||||
# e.g. 2015 for "Der Weg der Könige" against 2010 for the work.
|
||||
edition_years = matched_edition.get("publish_year") or []
|
||||
if edition_years:
|
||||
publish_year = min(edition_years)
|
||||
# A translation is published later than the work it translates,
|
||||
# e.g. 2015 for "Der Weg der Könige" against 2010 for the work.
|
||||
edition_years = matched_edition.get("publish_year") or []
|
||||
if edition_years:
|
||||
publish_year = min(edition_years)
|
||||
|
||||
cover_url = f"{COVERS_BASE_URL}/b/id/{cover_id}-L.jpg" if cover_id else None
|
||||
|
||||
|
||||
@@ -241,6 +241,7 @@ export const SearchBar = forwardRef<SearchBarHandle, SearchBarProps>(
|
||||
const selectPanelRef = useRef<HTMLDivElement>(null);
|
||||
const autocompletePanelRef = useRef<HTMLDivElement>(null);
|
||||
const optionsPanelId = useId();
|
||||
const optionsPanelRef = useRef<HTMLDivElement>(null);
|
||||
const isPopover = optionsLayout === 'popover';
|
||||
const layoutClasses = OPTIONS_LAYOUT_CLASSES[optionsLayout];
|
||||
const [isOptionsOpen, setIsOptionsOpen] = useState(false);
|
||||
@@ -270,7 +271,8 @@ export const SearchBar = forwardRef<SearchBarHandle, SearchBarProps>(
|
||||
const combinedSelectorLocked = combinedModeLocked && !isManualTarget;
|
||||
const combinedToggleAvailable = !!onCombinedModeChange && !isManualTarget;
|
||||
|
||||
useDismiss(isOptionsOpen && isPopover, [selectorRef], closeOptions);
|
||||
// The inline panel renders below the bar, outside selectorRef.
|
||||
useDismiss(isOptionsOpen, [selectorRef, optionsPanelRef], closeOptions);
|
||||
useDismiss(isSelectOpen, [selectPanelRef, selectTriggerRef], () => setIsSelectOpen(false));
|
||||
useDismiss(isAutocompleteOpen, [autocompletePanelRef, inputRef], () =>
|
||||
setIsAutocompleteOpen(false),
|
||||
@@ -628,7 +630,7 @@ export const SearchBar = forwardRef<SearchBarHandle, SearchBarProps>(
|
||||
) : null;
|
||||
|
||||
const optionsPanel = (
|
||||
<div id={optionsPanelId} className={layoutClasses.panel}>
|
||||
<div id={optionsPanelId} ref={optionsPanelRef} className={layoutClasses.panel}>
|
||||
{showContentTypeSelector && (
|
||||
<div className={layoutClasses.group}>
|
||||
<OptionsGroupHeader label="Content" action={optionsButton} />
|
||||
|
||||
@@ -11,6 +11,7 @@ import pytest
|
||||
|
||||
from shelfmark.core.models import DownloadTask, QueueStatus
|
||||
from shelfmark.core.notifications import NotificationEvent
|
||||
from shelfmark.core.viewer_redaction import public_download_id
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
@@ -39,6 +40,32 @@ def _read_download_history_row(main_module, task_id: str):
|
||||
conn.close()
|
||||
|
||||
|
||||
def _create_release_request(main_module, user: dict, task_id: str) -> dict:
|
||||
return main_module.user_db.create_request(
|
||||
user_id=user["id"],
|
||||
content_type="ebook",
|
||||
request_level="release",
|
||||
policy_mode="request_release",
|
||||
book_data={
|
||||
"title": "Requested Activity",
|
||||
"author": "Activity Author",
|
||||
"provider": "openlibrary",
|
||||
"provider_id": "activity-req",
|
||||
},
|
||||
release_data={"source": "prowlarr", "source_id": task_id, "title": "Requested.epub"},
|
||||
status="fulfilled",
|
||||
delivery_state="queued",
|
||||
)
|
||||
|
||||
|
||||
def _activity_update_payloads(mock_emit) -> dict[str, dict]:
|
||||
return {
|
||||
call.kwargs["to"]: call.args[1]
|
||||
for call in mock_emit.call_args_list
|
||||
if call.args[0] == "activity_update"
|
||||
}
|
||||
|
||||
|
||||
class TestTerminalSnapshotCapture:
|
||||
def test_complete_transition_records_direct_snapshot(self, main_module):
|
||||
user = _create_user(main_module, prefix="snap-direct")
|
||||
@@ -193,6 +220,35 @@ class TestTerminalSnapshotCapture:
|
||||
finally:
|
||||
main_module.backend.book_queue.cancel_download(task_id)
|
||||
|
||||
def test_activity_update_hides_request_linked_task_id_from_owner(self, main_module):
|
||||
user = _create_user(main_module, prefix="snap-activity-redact")
|
||||
task_id = f"7:https://tracker.example/download/1?passkey={uuid.uuid4().hex[:8]}"
|
||||
request_row = _create_release_request(main_module, user, task_id)
|
||||
task = DownloadTask(
|
||||
task_id=task_id,
|
||||
source="prowlarr",
|
||||
title="Requested Activity",
|
||||
user_id=user["id"],
|
||||
username=user["username"],
|
||||
request_id=request_row["id"],
|
||||
)
|
||||
assert main_module.backend.book_queue.add(task) is True
|
||||
|
||||
try:
|
||||
with patch.object(main_module.ws_manager, "is_enabled", return_value=True):
|
||||
with patch.object(main_module.ws_manager.socketio, "emit") as mock_emit:
|
||||
main_module.backend.book_queue.update_status(task_id, QueueStatus.COMPLETE)
|
||||
|
||||
payloads = _activity_update_payloads(mock_emit)
|
||||
assert payloads["admins"]["task_id"] == task_id
|
||||
assert payloads[f"user_{user['id']}"] == {
|
||||
"kind": "download_terminal",
|
||||
"task_id": public_download_id(task_id),
|
||||
"status": "complete",
|
||||
}
|
||||
finally:
|
||||
main_module.backend.book_queue.cancel_download(task_id)
|
||||
|
||||
def test_error_transition_triggers_download_failed_notification(self, main_module):
|
||||
user = _create_user(main_module, prefix="snap-notify-error")
|
||||
task_id = f"notify-error-{uuid.uuid4().hex[:8]}"
|
||||
@@ -420,6 +476,40 @@ class TestTerminalSnapshotCapture:
|
||||
to=f"user_{user['id']}",
|
||||
)
|
||||
|
||||
def test_queue_hook_reset_hides_request_linked_task_id_from_owner(self, main_module):
|
||||
user = _create_user(main_module, prefix="snap-reset-redact")
|
||||
task_id = f"7:https://tracker.example/download/2?passkey={uuid.uuid4().hex[:8]}"
|
||||
request_row = _create_release_request(main_module, user, task_id)
|
||||
main_module.activity_view_state_service.dismiss(
|
||||
viewer_scope=f"user:{user['id']}",
|
||||
item_type="download",
|
||||
item_key=f"download:{task_id}",
|
||||
)
|
||||
|
||||
task = SimpleNamespace(
|
||||
user_id=user["id"],
|
||||
username=user["username"],
|
||||
request_id=request_row["id"],
|
||||
source="prowlarr",
|
||||
title="Requested Activity",
|
||||
author="Activity Author",
|
||||
format="epub",
|
||||
size="1 MB",
|
||||
preview=None,
|
||||
content_type="ebook",
|
||||
)
|
||||
|
||||
with patch.object(main_module.ws_manager, "is_enabled", return_value=True):
|
||||
with patch.object(main_module.ws_manager.socketio, "emit") as mock_emit:
|
||||
main_module._record_download_queued(task_id, task)
|
||||
|
||||
payloads = _activity_update_payloads(mock_emit)
|
||||
assert payloads["admins"] == {"kind": "activity_reset", "task_id": task_id}
|
||||
assert payloads[f"user_{user['id']}"] == {
|
||||
"kind": "activity_reset",
|
||||
"task_id": public_download_id(task_id),
|
||||
}
|
||||
|
||||
def test_cancelled_transition_does_not_trigger_notification(self, main_module):
|
||||
user = _create_user(main_module, prefix="snap-notify-cancel")
|
||||
task_id = f"notify-cancel-{uuid.uuid4().hex[:8]}"
|
||||
|
||||
@@ -4,14 +4,16 @@ People attach the debug bundle to public issues, so the filter is tested by runn
|
||||
real pipeline line from the script against a controlled environment.
|
||||
"""
|
||||
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
SCRIPT = Path(__file__).resolve().parents[2] / "genDebug.sh"
|
||||
|
||||
|
||||
def _filtered_env(env: dict[str, str]) -> set[str]:
|
||||
def _filtered_env_lines(env: dict[str, str]) -> list[str]:
|
||||
line = next(
|
||||
line for line in SCRIPT.read_text().splitlines() if line.startswith("env | grep -v")
|
||||
)
|
||||
@@ -23,7 +25,25 @@ def _filtered_env(env: dict[str, str]) -> set[str]:
|
||||
text=True,
|
||||
check=True,
|
||||
).stdout
|
||||
return {entry.split("=", 1)[0] for entry in out.splitlines()}
|
||||
return out.splitlines()
|
||||
|
||||
|
||||
def _filtered_env(env: dict[str, str]) -> set[str]:
|
||||
return {entry.split("=", 1)[0] for entry in _filtered_env_lines(env)}
|
||||
|
||||
|
||||
def _redacted_config(tmp_path: Path, settings: dict[str, object]) -> dict[str, object]:
|
||||
match = re.search(r"^redact_config_json\(\) \{\n.*?^\}\n", SCRIPT.read_text(), re.M | re.S)
|
||||
assert match is not None
|
||||
config_file = tmp_path / "settings.json"
|
||||
config_file.write_text(json.dumps(settings, indent=2))
|
||||
out = subprocess.run(
|
||||
["bash", "-c", f'{match.group(0)}redact_config_json "$1"', "bash", str(config_file)],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
).stdout
|
||||
return json.loads(out)
|
||||
|
||||
|
||||
def test_secret_variables_are_left_out() -> None:
|
||||
@@ -37,6 +57,7 @@ def test_secret_variables_are_left_out() -> None:
|
||||
"SOME_SECRET": "e",
|
||||
"DB_PASSWORD_FILE": "f",
|
||||
"SERVICE_TOKEN_VALUE": "g",
|
||||
"PROWLARR_MAM_ID": "h",
|
||||
}
|
||||
)
|
||||
|
||||
@@ -49,6 +70,7 @@ def test_secret_variables_are_left_out() -> None:
|
||||
"SOME_SECRET",
|
||||
"DB_PASSWORD_FILE",
|
||||
"SERVICE_TOKEN_VALUE",
|
||||
"PROWLARR_MAM_ID",
|
||||
}
|
||||
)
|
||||
|
||||
@@ -59,3 +81,63 @@ def test_ordinary_variables_are_kept() -> None:
|
||||
)
|
||||
|
||||
assert {"TZ", "PUID", "FLASK_PORT"} <= names
|
||||
|
||||
|
||||
def test_proxy_credentials_are_redacted() -> None:
|
||||
lines = _filtered_env_lines(
|
||||
{
|
||||
"PATH": "/usr/bin:/bin",
|
||||
"HTTP_PROXY": "http://alice:s3cret@proxy.local:3128",
|
||||
"HTTPS_PROXY": "socks5://bob@10.0.0.1:1080",
|
||||
"NO_PROXY": "localhost",
|
||||
}
|
||||
)
|
||||
|
||||
assert "HTTP_PROXY=http://[REDACTED]@proxy.local:3128" in lines
|
||||
assert "HTTPS_PROXY=socks5://[REDACTED]@10.0.0.1:1080" in lines
|
||||
assert "NO_PROXY=localhost" in lines
|
||||
|
||||
|
||||
def test_config_secrets_are_redacted(tmp_path: Path) -> None:
|
||||
redacted = _redacted_config(
|
||||
tmp_path,
|
||||
{
|
||||
"PROWLARR_API_KEY": "key",
|
||||
"PROWLARR_MAM_ID": "mam-session",
|
||||
"QBITTORRENT_PASSWORD": "password",
|
||||
"HTTP_PROXY": "http://alice:s3cret@proxy.local:3128",
|
||||
"PROWLARR_URL": "http://prowlarr:9696",
|
||||
"ADMIN_NOTIFICATION_ROUTES": [
|
||||
{"event": "all", "url": "tgram://123456:bot-token/987"},
|
||||
{"event": "download_failed", "url": "discord://webhook-id/webhook-token"},
|
||||
],
|
||||
"BOOK_LANGUAGE": ["en"],
|
||||
},
|
||||
)
|
||||
|
||||
assert redacted == {
|
||||
"PROWLARR_API_KEY": "[REDACTED]",
|
||||
"PROWLARR_MAM_ID": "[REDACTED]",
|
||||
"QBITTORRENT_PASSWORD": "[REDACTED]",
|
||||
"HTTP_PROXY": "http://[REDACTED]@proxy.local:3128",
|
||||
"PROWLARR_URL": "http://prowlarr:9696",
|
||||
"ADMIN_NOTIFICATION_ROUTES": [
|
||||
{"event": "all", "url": "tgram://[REDACTED]"},
|
||||
{"event": "download_failed", "url": "discord://[REDACTED]"},
|
||||
],
|
||||
"BOOK_LANGUAGE": ["en"],
|
||||
}
|
||||
|
||||
|
||||
def test_package_listing_does_not_need_pip() -> None:
|
||||
"""The image ships without pip, so packages are listed through importlib.metadata."""
|
||||
match = re.search(r"<<'PY'.*?\n(.*?)^PY$", SCRIPT.read_text(), re.M | re.S)
|
||||
assert match is not None
|
||||
out = subprocess.run(
|
||||
[sys.executable, "-I", "-c", match.group(1)],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
).stdout
|
||||
|
||||
assert any(line.startswith("pytest==") for line in out.splitlines())
|
||||
@@ -1,6 +1,11 @@
|
||||
"""Tests for core notification rendering and dispatch helpers."""
|
||||
|
||||
import contextvars
|
||||
import logging
|
||||
import threading
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
||||
import apprise
|
||||
|
||||
from shelfmark.core import notifications as notifications_module
|
||||
|
||||
@@ -366,6 +371,76 @@ def test_dispatch_to_apprise_logs_captured_apprise_warning_messages(monkeypatch)
|
||||
)
|
||||
|
||||
|
||||
class _ServerErrorHandler(BaseHTTPRequestHandler):
|
||||
def do_POST(self):
|
||||
self.send_response(500)
|
||||
self.end_headers()
|
||||
|
||||
def log_message(self, format, *args): # noqa: A002
|
||||
return
|
||||
|
||||
|
||||
def test_dispatch_to_apprise_reports_failure_logged_on_apprise_worker_thread():
|
||||
"""Apprise 2 delivers on its own worker threads, which log the plugin's reason."""
|
||||
server = ThreadingHTTPServer(("127.0.0.1", 0), _ServerErrorHandler)
|
||||
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
try:
|
||||
result = notifications_module._dispatch_to_apprise(
|
||||
[f"json://127.0.0.1:{server.server_address[1]}/hook"],
|
||||
title="Test",
|
||||
body="Body",
|
||||
notify_type=apprise.NotifyType.INFO,
|
||||
)
|
||||
finally:
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
thread.join(timeout=5)
|
||||
|
||||
assert result["success"] is False
|
||||
assert len(result["details"]) == 1
|
||||
assert result["details"][0].startswith("json: ")
|
||||
assert "error=500" in result["details"][0]
|
||||
|
||||
|
||||
def test_dispatch_to_apprise_ignores_apprise_records_of_other_dispatches(monkeypatch):
|
||||
fake_apprise = _FakeAppriseModule()
|
||||
monkeypatch.setattr(notifications_module, "apprise", fake_apprise)
|
||||
|
||||
def _log_warning(message):
|
||||
logging.getLogger("apprise.plugins.json").warning(message)
|
||||
|
||||
def _notify(**kwargs):
|
||||
_ = kwargs
|
||||
# A dispatch running concurrently on another thread, in its own context.
|
||||
other = threading.Thread(
|
||||
target=_log_warning,
|
||||
args=("Another dispatch failed",),
|
||||
context=contextvars.Context(),
|
||||
)
|
||||
other.start()
|
||||
other.join()
|
||||
# Apprise's worker thread, running in a copy of this dispatch's context.
|
||||
worker = threading.Thread(
|
||||
target=contextvars.copy_context().run,
|
||||
args=(_log_warning, "This dispatch failed"),
|
||||
)
|
||||
worker.start()
|
||||
worker.join()
|
||||
return False
|
||||
|
||||
monkeypatch.setattr(fake_apprise.client, "notify", _notify)
|
||||
|
||||
result = notifications_module._dispatch_to_apprise(
|
||||
["json://example.invalid/hook"],
|
||||
title="Test",
|
||||
body="Body",
|
||||
notify_type=_FakeNotifyType.INFO,
|
||||
)
|
||||
|
||||
assert result["details"] == ["json: apprise.plugins.json: This dispatch failed"]
|
||||
|
||||
|
||||
def test_dispatch_to_apprise_logs_add_exception_at_debug_with_trace(monkeypatch):
|
||||
fake_apprise = _FakeAppriseModule()
|
||||
fake_apprise.client.instantiate_exceptions_by_url = {
|
||||
|
||||
@@ -63,6 +63,36 @@ def test_googlebooks_search_does_not_cache_request_failures():
|
||||
assert [book.title for book in result] == ["Recovered Book"]
|
||||
|
||||
|
||||
def test_googlebooks_paginated_search_does_not_cache_request_failures():
|
||||
get_metadata_cache().clear()
|
||||
provider = GoogleBooksProvider(api_key="test-key")
|
||||
session = _FlakyGoogleBooksSession()
|
||||
provider.session = session
|
||||
options = MetadataSearchOptions(query="Recovered Book")
|
||||
|
||||
assert provider.search_paginated(options).books == []
|
||||
|
||||
result = provider.search_paginated(options)
|
||||
|
||||
assert session.calls == 2
|
||||
assert [book.title for book in result.books] == ["Recovered Book"]
|
||||
|
||||
|
||||
def test_googlebooks_isbn_lookup_is_cached():
|
||||
get_metadata_cache().clear()
|
||||
provider = GoogleBooksProvider(api_key="test-key")
|
||||
session = _FlakyGoogleBooksSession()
|
||||
session.calls = 1 # skip the simulated timeout
|
||||
provider.session = session
|
||||
|
||||
first = provider.search_by_isbn("978-0-00-000000-2")
|
||||
second = provider.search_by_isbn("978-0-00-000000-2")
|
||||
|
||||
assert first is not None
|
||||
assert second == first
|
||||
assert session.calls == 2
|
||||
|
||||
|
||||
def test_googlebooks_pages_tile_when_limit_exceeds_api_maximum():
|
||||
get_metadata_cache().clear()
|
||||
provider = GoogleBooksProvider(api_key="test-key")
|
||||
|
||||
@@ -61,7 +61,13 @@ class TestOpenLibraryParseSearchDoc:
|
||||
"author_name": ["Brandon Sanderson"],
|
||||
"first_publish_year": 2010,
|
||||
"editions": {
|
||||
"docs": [{"key": "/books/OL32771812M", "title": "Der Weg der Könige"}]
|
||||
"docs": [
|
||||
{
|
||||
"key": "/books/OL32771812M",
|
||||
"title": "Der Weg der Könige",
|
||||
"language": ["ger"],
|
||||
}
|
||||
]
|
||||
},
|
||||
}
|
||||
)
|
||||
@@ -116,3 +122,175 @@ class TestOpenLibraryParseSearchDoc:
|
||||
assert result is not None
|
||||
assert result.search_title is None
|
||||
assert result.isbn_13 == "9780765326355"
|
||||
|
||||
def test_keeps_work_for_an_english_edition_with_another_title(self):
|
||||
"""Open Library shows an English edition whenever one matches the query.
|
||||
|
||||
Searching "Mistborn" picks the 2023 "Mistborn" reissue of "The Final
|
||||
Empire", which is the same book under a different English title, so the
|
||||
result keeps the work's title, ISBNs and first publication year.
|
||||
"""
|
||||
provider = OpenLibraryProvider()
|
||||
|
||||
result = provider._parse_search_doc(
|
||||
{
|
||||
"key": "/works/OL5738148W",
|
||||
"title": "The Final Empire",
|
||||
"author_name": ["Brandon Sanderson"],
|
||||
"isbn": ["9780765311788"],
|
||||
"cover_i": 14658160,
|
||||
"language": ["fre", "ger", "eng"],
|
||||
"first_publish_year": 2001,
|
||||
"editions": {
|
||||
"docs": [
|
||||
{
|
||||
"key": "/books/OL62594701M",
|
||||
"title": "Mistborn",
|
||||
"language": ["eng"],
|
||||
"isbn": ["9781250318541"],
|
||||
"cover_i": 1,
|
||||
"publish_year": [2023],
|
||||
}
|
||||
]
|
||||
},
|
||||
},
|
||||
"Mistborn",
|
||||
)
|
||||
|
||||
assert result is not None
|
||||
assert result.title == "The Final Empire"
|
||||
assert result.search_title is None
|
||||
assert result.publish_year == 2001
|
||||
assert result.isbn_13 == "9780765311788"
|
||||
assert result.cover_url == "https://covers.openlibrary.org/b/id/14658160-L.jpg"
|
||||
|
||||
def test_keeps_work_for_a_subtitled_reprint(self):
|
||||
provider = OpenLibraryProvider()
|
||||
|
||||
result = provider._parse_search_doc(
|
||||
{
|
||||
"key": "/works/OL893516W",
|
||||
"title": "Children of Dune",
|
||||
"author_name": ["Frank Herbert"],
|
||||
"first_publish_year": 1976,
|
||||
"editions": {
|
||||
"docs": [
|
||||
{
|
||||
"key": "/books/OL1M",
|
||||
"title": "Children of Dune (Dune Chronicles, Book 3)",
|
||||
"language": ["eng"],
|
||||
"publish_year": [1987],
|
||||
}
|
||||
]
|
||||
},
|
||||
},
|
||||
"Children of Dune",
|
||||
)
|
||||
|
||||
assert result is not None
|
||||
assert result.title == "Children of Dune"
|
||||
assert result.search_title is None
|
||||
assert result.publish_year == 1976
|
||||
|
||||
def test_keeps_work_when_the_query_names_the_work(self):
|
||||
"""A `lang` preference shows a translation even for the work's own title."""
|
||||
provider = OpenLibraryProvider()
|
||||
|
||||
result = provider._parse_search_doc(
|
||||
{
|
||||
"key": "/works/OL893516W",
|
||||
"title": "Children of Dune",
|
||||
"author_name": ["Frank Herbert"],
|
||||
"first_publish_year": 1976,
|
||||
"editions": {
|
||||
"docs": [
|
||||
{
|
||||
"key": "/books/OL2M",
|
||||
"title": "Dune-Zyklus",
|
||||
"language": ["ger"],
|
||||
"publish_year": [1985],
|
||||
}
|
||||
]
|
||||
},
|
||||
},
|
||||
"children of dune",
|
||||
)
|
||||
|
||||
assert result is not None
|
||||
assert result.title == "Children of Dune"
|
||||
assert result.search_title is None
|
||||
assert result.publish_year == 1976
|
||||
|
||||
def test_keeps_work_for_an_edition_of_unknown_language(self):
|
||||
provider = OpenLibraryProvider()
|
||||
|
||||
result = provider._parse_search_doc(
|
||||
{
|
||||
"key": "/works/OL5738148W",
|
||||
"title": "The Final Empire",
|
||||
"author_name": ["Brandon Sanderson"],
|
||||
"first_publish_year": 2001,
|
||||
"editions": {
|
||||
"docs": [{"key": "/books/OL3M", "title": "Mistborn", "publish_year": [2023]}]
|
||||
},
|
||||
},
|
||||
"Mistborn",
|
||||
)
|
||||
|
||||
assert result is not None
|
||||
assert result.title == "The Final Empire"
|
||||
assert result.publish_year == 2001
|
||||
|
||||
def test_ignores_edition_differing_only_in_case_and_accents(self):
|
||||
provider = OpenLibraryProvider()
|
||||
|
||||
result = provider._parse_search_doc(
|
||||
{
|
||||
"key": "/works/OL1W",
|
||||
"title": "L’étranger",
|
||||
"author_name": ["Albert Camus"],
|
||||
"first_publish_year": 1942,
|
||||
"editions": {
|
||||
"docs": [
|
||||
{
|
||||
"key": "/books/OL4M",
|
||||
"title": "L'Etranger",
|
||||
"language": ["fre"],
|
||||
"publish_year": [1995],
|
||||
}
|
||||
]
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
assert result is not None
|
||||
assert result.title == "L’étranger"
|
||||
assert result.publish_year == 1942
|
||||
|
||||
def test_prefers_translation_matched_by_query(self):
|
||||
provider = OpenLibraryProvider()
|
||||
|
||||
result = provider._parse_search_doc(
|
||||
{
|
||||
"key": "/works/OL5738148W",
|
||||
"title": "The Final Empire",
|
||||
"author_name": ["Brandon Sanderson"],
|
||||
"first_publish_year": 2001,
|
||||
"editions": {
|
||||
"docs": [
|
||||
{
|
||||
"key": "/books/OL5M",
|
||||
"title": "Kinder des Nebels",
|
||||
"language": ["ger"],
|
||||
"publish_year": [2018],
|
||||
}
|
||||
]
|
||||
},
|
||||
},
|
||||
"Kinder des Nebels",
|
||||
)
|
||||
|
||||
assert result is not None
|
||||
assert result.title == "Kinder des Nebels"
|
||||
assert result.search_title == "Kinder des Nebels"
|
||||
assert result.publish_year == 2018
|
||||
Reference in new issue
Block a user