starred/shelfmark
1
0
Fork 0
mirror of https://github.com/calibrain/shelfmark.git synced 2026-10-04 22:05:45 +01:00
Code Issues Packages Projects Releases 10 Wiki Activity

Patch: Script improvements + bug fixes (#591)

Browse Source 
- Add new booklore API file formats
- Renamed cookie for better login persistence with reverse proxy
- Updated fs.py to try hardlink before atomic move from tmp dir
- Fix transmission URL parsing 
- Fix scenario where file processing of huge files starves the
healthcheck
- Large enhancements to custom scripting, including passing JSON
download info, more consistent activation across output types,
decoupling from staging behavior, and added full documentation.
This commit is contained in:
Alex
2026-02-06 13:51:23 +00:00
committed by GitHub
parent f84fb082ad
commit a560089ce3
21 changed files with 1155 additions and 216 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/configuration.md
+124 -2
View File
@@ -1,3 +1,125 @@
# Configuration
# Directory and Volume Setup
TODO
This guide explains how to configure directories and Docker volumes for Shelfmark. It focuses on the difference between the destination folder and your download client paths, and how to make those paths line up inside containers.
## Conceptual Overview
```
DIRECT DOWNLOADS
Shelfmark downloads directly -> destination
TORRENT / USENET
Prowlarr -> Download client saves to <client path>
-> Shelfmark reads from <client path>
-> Shelfmark processes to destination
```
Key point: For torrent and usenet downloads, Shelfmark must see the same file path that your download client reports. The container path must match in both containers.
## Direct Download Setup
Direct downloads do not use an external download client. A simple two-folder setup is enough.
Required volumes:
| Container path | Purpose | Notes |
| --- | --- | --- |
| `/config` | Settings, database, cover cache | Configurable via `CONFIG_DIR` |
| `/books` | Destination folder for completed files | Configurable via `INGEST_DIR` and Settings -> Downloads -> Destination |
Example `docker-compose`:
```yaml
services:
shelfmark:
image: ghcr.io/calibrain/shelfmark:latest
volumes:
- /path/to/config:/config
- /path/to/books:/books
```
Notes:
- Point `/books` to your library ingest folder (Calibre-Web, Booklore, Audiobookshelf, etc) for automatic import.
- If you set Books Output Mode to Booklore (API), books are uploaded via API instead of written to `/books`. Audiobooks still use a destination folder.
- Ensure `PUID`/`PGID` (or legacy `UID`/`GID`) match the owner of the host directories to avoid permission errors.
## Torrent / Usenet Setup
For torrents and usenet, your download client reports a path (for example `/data/torrents/books/MyBook.epub`). Shelfmark must be able to read that exact path inside its own container.
Required volumes:
| Container path | Purpose | Notes |
| --- | --- | --- |
| `/config` | Settings, database, cover cache | Configurable via `CONFIG_DIR` |
| `/books` | Destination folder for processed files | Configurable via `INGEST_DIR` |
| `<client path>` | Download client path | Must match the download client container path exactly |
Side-by-side example with qBittorrent:
```yaml
services:
shelfmark:
volumes:
- /path/to/config:/config
- /path/to/books:/books
- /path/to/downloads:/data/torrents # Must match client
qbittorrent:
volumes:
- /path/to/downloads:/data/torrents # Same container path
```
Host paths can be anything. The container path (for example `/data/torrents`) must be identical in both containers.
### Remote Path Mappings
If paths cannot match (different machines or a fixed setup), use Remote Path Mappings.
Where to configure:
- Settings -> Advanced -> Remote Path Mappings
Example:
- Client reports `/data/torrents/books/...`
- Shelfmark can see the same files at `/downloads/books/...`
- Add a mapping from Remote Path `/data/torrents` to Local Path `/downloads`
## File Processing Options
### Transfer Method (Torrent / Usenet Only)
Available methods:
- Copy (default). Works everywhere.
- Hardlink. Preserves seeding without duplicating files.
Hardlink requirements and behavior:
- Source and destination must be on the same filesystem.
- If hardlinking is enabled but not possible, Shelfmark falls back to copying.
- Archive extraction is disabled while hardlinking is enabled.
- Do not use hardlinking if your destination is a library ingest folder.
### File Organization
Shelfmark supports three organization modes for the destination:
- None. Keep original filenames from the source.
- Rename Only. Rename files using a template.
- Rename and Organize. Create folders and rename using templates. Do not use with ingest folders.
Configure templates in Settings -> Downloads. Template syntax details are documented separately.
## Common Mistakes
- "Download failed - file not found": Path mismatch between Shelfmark and the download client. Ensure container paths match or use Remote Path Mappings.
- "Permission denied": `PUID`/`PGID` do not match the host directories. Ensure Shelfmark can read the client path and write to the destination.
- "Hardlinks not working" or "Files being copied instead": Source and destination are on different filesystems. Move the destination or accept copy fallback.
- "Downloads work but library does not see them": Destination does not point to the library ingest folder. Check Settings -> Downloads -> Destination.
- CIFS/SMB shares: Use the `nobrl` mount option to avoid database lock errors. Example: `//server/share /mnt/share cifs nobrl,... 0 0`
## Related Documentation
- Environment Variables Reference: `docs/environment-variables.md`
- Custom Scripts: `docs/custom-scripts.md`
- Installation: `docs/installation.md`
- Troubleshooting: `docs/troubleshooting.md`
docs/custom-scripts.md
+184
View File
@@ -0,0 +1,184 @@
# Custom Scripts
Shelfmark can run an executable you provide after a download task completes successfully. The script runs after the selected output has finished (for example: transfer to the folder destination, or upload to Booklore).
## Quick Start (Recommended)
1. Put your script on the machine that runs Shelfmark.
1. Make it executable.
1. Set it in Shelfmark (Settings -> Advanced -> Custom Script Path).
Example:
```bash
chmod +x /path/to/your/scripts/post_process.sh
```
### Docker Users
If you run Shelfmark in Docker, the script must exist inside the container. The easiest way is to mount a folder of scripts, then point Shelfmark at the container path in the UI.
```yaml
services:
shelfmark:
image: ghcr.io/calibrain/shelfmark:latest
volumes:
- /path/to/your/scripts:/scripts:ro
```
Then set:
- Settings -> Advanced -> Custom Script Path: `/scripts/post_process.sh`
<details>
<summary>Docker Compose: Configure Via Environment Variables (Optional)</summary>
```yaml
services:
shelfmark:
environment:
- CUSTOM_SCRIPT=/scripts/post_process.sh
- CUSTOM_SCRIPT_PATH_MODE=absolute
- CUSTOM_SCRIPT_JSON_PAYLOAD=true
```
</details>
## Script Behaviour
When enabled, Shelfmark runs your script once per successful task:
```bash
<custom_script_path> "<target_path>"
```
- `$1` is always set to the target path.
- If **Custom Script JSON Payload** is enabled, Shelfmark writes a JSON document to stdin (UTF-8).
- If JSON payload is disabled, stdin is empty (EOF).
- Timeout: 300 seconds (5 minutes)
- Exit code: `0` = success; anything else = the task is marked as **Error**
- Concurrency: downloads can run in parallel, so your script may be invoked concurrently for different tasks.
- Runtime: the script runs inside the Shelfmark container (if you use Docker) under the same user as Shelfmark.
## The Target Path (`$1`)
Shelfmark chooses a "best single path" for the task:
- If the output produced exactly one local file: that file path.
- If the output produced multiple local files: a directory path (the common parent directory of those files).
What the target path refers to depends on the output mode:
- Folder output (`output.mode=folder`, `phase=post_transfer`): the final imported file or folder inside your destination.
- Booklore output (`output.mode=booklore`, `phase=post_upload`): the local file or folder that was uploaded (the destination is remote).
By default, `$1` is an absolute path inside the Shelfmark container (or on your host, if you are not using Docker).
## JSON Payload (stdin)
Configure in: Settings -> Advanced -> Custom Script JSON Payload
When enabled, Shelfmark sends a versioned JSON payload to your script via stdin (and still passes `$1`). This is the recommended way to write robust scripts, especially for multi-file imports (audiobooks) and output-specific context (like Booklore).
- The JSON payload always includes absolute paths in `paths.*`, even if you set Custom Script Path Mode to `relative` for `$1`.
- `output.mode` tells you which output ran.
- `output.details` is output-specific. For Booklore output, `output.details.booklore` includes connection details such as `base_url`, `library_id`, and `path_id`.
- `phase` indicates when the script is running. Current values: `post_transfer` (folder output), `post_upload` (Booklore output).
- `transfer` is only included for outputs that do a local transfer (for example the folder output).
If JSON payload is disabled, stdin is empty (EOF). Don't `cat` stdin unless you've enabled the payload.
Example payload shape:
```json
{
"version": 1,
"phase": "post_transfer",
"task": {
"task_id": "abc123",
"source": "direct",
"title": "Foundation",
"author": "Isaac Asimov"
},
"output": {
"mode": "folder",
"organization_mode": "organize"
},
"paths": {
"destination": "/data/library/books",
"target": "/data/library/books/Isaac Asimov/Foundation/Foundation.epub",
"final_paths": [
"/data/library/books/Isaac Asimov/Foundation/Foundation.epub"
]
},
"transfer": {
"op_counts": {"copy": 1, "move": 0, "hardlink": 0},
"use_hardlink": false,
"is_torrent": false,
"preserve_source": false
}
}
```
Example (bash + jq) (JSON payload must be enabled):
```bash
payload="$(cat)"
mode="$(echo "$payload" | jq -r '.output.mode')"
title="$(echo "$payload" | jq -r '.task.title')"
final_paths="$(echo "$payload" | jq -r '.paths.final_paths[]')"
echo "mode=$mode title=$title" >&2
echo "$final_paths" >&2
```
Example (Python) (works whether JSON payload is enabled or not):
```python
#!/usr/bin/env python3
import json
import sys
target = sys.argv[1]
raw = sys.stdin.read()
payload = json.loads(raw) if raw.strip() else None
print(f"target={target}", file=sys.stderr)
if payload:
print(f"mode={payload['output']['mode']} phase={payload['phase']}", file=sys.stderr)
```
<details>
<summary>Advanced Options</summary>
### Absolute vs Relative Target Paths
Configure in: Settings -> Advanced -> Custom Script Path Mode
This setting controls what gets passed as `$1`:
- `absolute` (default): pass an absolute path.
- `relative`: pass a path relative to the output's "destination root", and run the script with `$PWD` set to that root.
For folder output, the destination root is your configured destination folder. For Booklore output, it's the local upload folder.
Example (folder destination is `/data/library/books`, and the imported file ended up in `Isaac Asimov/Foundation/Foundation.epub`):
```bash
# Absolute mode:
$PWD is unchanged
$1 = /data/library/books/Isaac Asimov/Foundation/Foundation.epub
# Relative mode:
$PWD = /data/library/books
$1 = Isaac Asimov/Foundation/Foundation.epub
```
Note: if the target is the destination folder itself, `relative` mode may pass `.`.
</details>
## Notes And Caveats
- **Hardlinks and torrents:** if you use hardlinking to keep seeding, avoid scripts that modify file contents, since hardlinked files share data with the seeding copy.
- **Booklore output mode:** scripts run after upload. `$1` will point at the local uploaded file (or staging folder).
docs/environment-variables.md
+14
View File
@@ -578,6 +578,7 @@ Comma-separated hosts to bypass proxy (e.g., localhost,127.0.0.1,10.*,*.local)
| `DOWNLOAD_PROGRESS_UPDATE_INTERVAL` | How often download progress is broadcast to the UI. | number | `1` |
| `CUSTOM_SCRIPT` | Path to a script to run after each successful download. Must be executable. | string | _none_ |
| `CUSTOM_SCRIPT_PATH_MODE` | Pass the path to the custom script as an absolute path or relative to the destination folder. | string (choice) | `absolute` |
| `CUSTOM_SCRIPT_JSON_PAYLOAD` | Send a JSON payload to the custom script via stdin. | boolean | `false` |
| `COVERS_CACHE_ENABLED` | Cache book covers on the server for faster loading. | boolean | `true` |
| `COVERS_CACHE_TTL` | How long to keep cached covers. Set to 0 to keep forever (recommended for static artwork). | number | `0` |
| `COVERS_CACHE_MAX_SIZE_MB` | Maximum disk space for cached covers. Oldest images are removed when limit is reached. | number | `500` |
@@ -636,6 +637,8 @@ How often download progress is broadcast to the UI.
Path to a script to run after each successful download. Must be executable.
See `docs/custom-scripts.md` for the user guide, including how the target path argument (`$1`) and optional JSON payload work.
- **Type:** string
- **Default:** _none_
@@ -649,6 +652,17 @@ Pass the path to the custom script as an absolute path or relative to the destin
- **Default:** `absolute`
- **Options:** `absolute` (Absolute), `relative` (Relative)
#### `CUSTOM_SCRIPT_JSON_PAYLOAD`
**Custom Script JSON Payload**
Send a JSON payload to the custom script via stdin (in addition to the target path argument).
See `docs/custom-scripts.md` for an example payload and usage patterns.
- **Type:** boolean
- **Default:** `false`
#### `COVERS_CACHE_ENABLED`
**Enable Cover Cache**
shelfmark/config/env.py
+1
View File
@@ -113,6 +113,7 @@ FLASK_PORT = int(os.getenv("FLASK_PORT", "8084"))
# =============================================================================
SESSION_COOKIE_SECURE_ENV = os.getenv("SESSION_COOKIE_SECURE", "false")
SESSION_COOKIE_NAME = "shelfmark_session"
CWA_DB_PATH = _resolve_cwa_db_path()
shelfmark/config/settings.py
+6
View File
@@ -1294,6 +1294,12 @@ def advanced_settings():
],
default="absolute",
),
CheckboxField(
key="CUSTOM_SCRIPT_JSON_PAYLOAD",
label="Custom Script JSON Payload",
description="Send a JSON payload to the script via stdin. Useful for multi-file imports (audiobooks) or richer metadata without relying on path parsing.",
default=False,
),
HeadingField(
key="remote_path_mappings_heading",
title="Remote Path Mappings",
shelfmark/download/fs.py
+136 -58
View File
@@ -8,6 +8,7 @@ import errno
import os
import shutil
import subprocess
import tempfile
import time
from pathlib import Path
from typing import Any, Callable, Optional, TypeVar
@@ -190,11 +191,72 @@ def _claim_destination(path: Path) -> bool:
return True
def _hardlink_not_supported(error: OSError) -> bool:
err = error.errno
return err in {
errno.EXDEV,
errno.EMLINK,
errno.EPERM,
errno.EACCES,
getattr(errno, "ENOTSUP", errno.EPERM),
getattr(errno, "EOPNOTSUPP", errno.EPERM),
errno.EINVAL,
}
def _create_temp_path(dest_path: Path) -> Path:
fd, temp_path = tempfile.mkstemp(
prefix=f".{dest_path.name}.",
suffix=".tmp",
dir=str(dest_path.parent),
)
os.close(fd)
return Path(temp_path)
def _publish_temp_file(temp_path: Path, dest_path: Path) -> bool:
"""Publish a temp file to its final path without overwriting existing files.
Returns True on success, False if the destination already exists.
"""
try:
os.link(str(temp_path), str(dest_path))
temp_path.unlink(missing_ok=True)
return True
except FileExistsError:
return False
except OSError as e:
if _is_permission_error(e):
log_transfer_permission_context(
"publish_hardlink",
source=temp_path,
dest=dest_path,
error=e,
)
if _hardlink_not_supported(e):
logger.debug(
"Hardlink publish unsupported; falling back to claim+replace: %s -> %s (%s)",
temp_path,
dest_path,
e,
)
claimed = _claim_destination(dest_path)
if not claimed:
return False
try:
os.replace(str(temp_path), str(dest_path))
except Exception:
dest_path.unlink(missing_ok=True)
raise
return True
raise
def atomic_move(source_path: Path, dest_path: Path, max_attempts: int = 100) -> Path:
"""Move a file with collision detection.
Uses os.rename() for same-filesystem moves (atomic, triggers inotify events),
falls back to exclusive create + shutil.move for cross-filesystem moves.
falls back to copy-then-publish for cross-filesystem moves.
Note: We use os.rename() instead of hardlink+unlink because os.rename()
triggers proper inotify IN_MOVED_TO events that file watchers (like Calibre's
@@ -242,23 +304,21 @@ def atomic_move(source_path: Path, dest_path: Path, max_attempts: int = 100) ->
try_path.unlink(missing_ok=True)
continue
except OSError as e:
# Cross-filesystem - fall back to exclusive create + verified copy + delete.
# Cross-filesystem - copy to temp and publish atomically.
if e.errno != errno.EXDEV:
if claimed:
try_path.unlink(missing_ok=True)
raise
expected_size = source_path.stat().st_size
if claimed:
try_path.unlink(missing_ok=True)
claimed = False
temp_path: Optional[Path] = None
try:
if not claimed:
# Claim destination path atomically.
fd = os.open(str(try_path), os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o666)
os.close(fd)
# Copy to a temp file first, then replace to avoid partial files.
temp_path = try_path.parent / f".{try_path.name}.tmp"
try:
temp_path = _create_temp_path(try_path)
try:
run_blocking_io(shutil.copy2, str(source_path), str(temp_path))
except (PermissionError, OSError) as copy_error:
@@ -273,21 +333,33 @@ def atomic_move(source_path: Path, dest_path: Path, max_attempts: int = 100) ->
else:
raise
temp_path.replace(try_path)
_verify_transfer_size(try_path, expected_size, "move")
_verify_transfer_size(temp_path, expected_size, "move")
published = _publish_temp_file(temp_path, try_path)
if not published:
temp_path.unlink(missing_ok=True)
continue
try:
_verify_transfer_size(try_path, expected_size, "move")
except Exception:
try_path.unlink(missing_ok=True)
raise
source_path.unlink()
if attempt > 0:
logger.info(f"File collision resolved: {try_path.name}")
return try_path
except FileExistsError:
if temp_path:
temp_path.unlink(missing_ok=True)
continue
except Exception:
try_path.unlink(missing_ok=True)
temp_path.unlink(missing_ok=True)
if temp_path:
temp_path.unlink(missing_ok=True)
raise
except FileExistsError:
continue
except (PermissionError, OSError) as e:
if _is_permission_error(e):
log_transfer_permission_context(
@@ -371,8 +443,8 @@ def atomic_hardlink(source_path: Path, dest_path: Path, max_attempts: int = 100)
def atomic_copy(source_path: Path, dest_path: Path, max_attempts: int = 100) -> Path:
"""Copy a file with atomic collision detection.
Uses exclusive create to claim destination, then copies via temp file
to avoid partial files on failure.
Uses a temp file in the destination directory and publishes it atomically,
avoiding partial files on failure.
Args:
source_path: Source file to copy
@@ -388,57 +460,63 @@ def atomic_copy(source_path: Path, dest_path: Path, max_attempts: int = 100) ->
base = dest_path.stem
ext = dest_path.suffix
parent = dest_path.parent
expected_size = source_path.stat().st_size
for attempt in range(max_attempts):
try_path = dest_path if attempt == 0 else parent / f"{base}_{attempt}{ext}"
if try_path.exists():
continue
temp_path: Optional[Path] = None
try:
# Atomically claim the destination by creating an exclusive file
fd = os.open(str(try_path), os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o666)
os.close(fd)
# Copy to temp file first, then replace to avoid partial files
temp_path = try_path.parent / f".{try_path.name}.tmp"
temp_path = _create_temp_path(try_path)
try:
try:
run_blocking_io(shutil.copy2, str(source_path), str(temp_path))
except (PermissionError, OSError) as e:
# Handle NFS permission errors immediately here
if _is_permission_error(e):
log_transfer_permission_context(
"atomic_copy",
source=source_path,
dest=temp_path,
error=e,
)
logger.debug(
"Permission error during copy, falling back to copyfile (%s -> %s): %s",
run_blocking_io(shutil.copy2, str(source_path), str(temp_path))
except (PermissionError, OSError) as e:
# Handle NFS permission errors immediately here
if _is_permission_error(e):
log_transfer_permission_context(
"atomic_copy",
source=source_path,
dest=temp_path,
error=e,
)
logger.debug(
"Permission error during copy, falling back to copyfile (%s -> %s): %s",
source_path,
temp_path,
e,
)
try:
_perform_nfs_fallback(source_path, temp_path, is_move=False)
except Exception as fallback_error:
logger.error(
"NFS fallback also failed (%s -> %s): %s",
source_path,
temp_path,
e,
fallback_error,
)
try:
_perform_nfs_fallback(source_path, temp_path, is_move=False)
except Exception as fallback_error:
logger.error(
"NFS fallback also failed (%s -> %s): %s",
source_path,
temp_path,
fallback_error,
)
raise e from fallback_error
else:
raise
temp_path.replace(try_path)
_verify_transfer_size(try_path, source_path.stat().st_size, "copy")
if attempt > 0:
logger.info(f"File collision resolved: {try_path.name}")
return try_path
raise e from fallback_error
else:
raise
_verify_transfer_size(temp_path, expected_size, "copy")
published = _publish_temp_file(temp_path, try_path)
if not published:
temp_path.unlink(missing_ok=True)
continue
try:
_verify_transfer_size(try_path, expected_size, "copy")
except Exception:
try_path.unlink(missing_ok=True)
temp_path.unlink(missing_ok=True)
raise
except FileExistsError:
continue
if attempt > 0:
logger.info(f"File collision resolved: {try_path.name}")
return try_path
except Exception:
if temp_path:
temp_path.unlink(missing_ok=True)
raise
raise RuntimeError(f"Could not copy file after {max_attempts} attempts: {dest_path}")
shelfmark/download/outputs/booklore.py
+31 -1
View File
@@ -1,5 +1,6 @@
from __future__ import annotations
import os
from dataclasses import dataclass
from pathlib import Path
from threading import Event
@@ -17,7 +18,7 @@ from shelfmark.download.staging import STAGE_MOVE, STAGE_NONE, build_staging_dir
logger = setup_logger(__name__)
BOOKLORE_OUTPUT_MODE = "booklore"
BOOKLORE_SUPPORTED_EXTENSIONS = {".cb7", ".cbr", ".cbz", ".epub", ".fb2", ".pdf"}
BOOKLORE_SUPPORTED_EXTENSIONS = {".azw", ".azw3", ".cb7", ".cbr", ".cbz", ".epub", ".fb2", ".mobi", ".pdf"}
BOOKLORE_SUPPORTED_FORMATS_LABEL = ", ".join(
ext.lstrip(".").upper() for ext in sorted(BOOKLORE_SUPPORTED_EXTENSIONS)
)
@@ -197,9 +198,11 @@ def _post_process_booklore(
status_callback,
) -> Optional[str]:
from shelfmark.download.postprocess.pipeline import (
CustomScriptContext,
OutputPlan,
cleanup_output_staging,
is_managed_workspace_path,
maybe_run_custom_script,
prepare_output_files,
)
@@ -265,6 +268,33 @@ def _post_process_booklore(
logger.info("Task %s: uploaded %d file(s) to Booklore", task.task_id, len(prepared.files))
destination: Optional[Path]
if len(prepared.files) == 1:
destination = prepared.files[0].parent
else:
try:
destination = Path(os.path.commonpath([str(p.parent) for p in prepared.files]))
except ValueError:
destination = prepared.files[0].parent if prepared.files else None
script_context = CustomScriptContext(
task=task,
phase="post_upload",
output_mode=BOOKLORE_OUTPUT_MODE,
destination=destination,
final_paths=prepared.files,
output_details={
"booklore": {
"base_url": booklore_config.base_url,
"library_id": booklore_config.library_id,
"path_id": booklore_config.path_id,
"refresh_after_upload": bool(booklore_config.refresh_after_upload),
}
},
)
if not maybe_run_custom_script(script_context, status_callback=status_callback):
return None
message = "Uploaded to Booklore"
if len(prepared.files) > 1:
message = f"Uploaded to Booklore ({len(prepared.files)} files)"
shelfmark/download/outputs/folder.py
+26 -98
View File
@@ -1,7 +1,6 @@
from __future__ import annotations
import os
import subprocess
from dataclasses import dataclass
from pathlib import Path
from threading import Event
@@ -11,7 +10,6 @@ import shelfmark.core.config as core_config
from shelfmark.core.logger import setup_logger
from shelfmark.core.models import DownloadTask
from shelfmark.core.utils import is_audiobook as check_audiobook
from shelfmark.download.archive import is_archive
from shelfmark.download.outputs import register_output
from shelfmark.download.staging import StageAction, STAGE_NONE
@@ -20,19 +18,6 @@ logger = setup_logger(__name__)
FOLDER_OUTPUT_MODE = "folder"
def _resolve_custom_script_target(target_path: Path, destination: Path, path_mode: str) -> Path:
mode = (path_mode or "absolute").strip().lower()
if mode != "relative":
return target_path
try:
return target_path.relative_to(destination)
except ValueError:
if target_path.is_absolute():
return Path(target_path.name)
return target_path
def _format_op_counts(op_counts: dict[str, int]) -> str:
parts = [f"{op}={count}" for op, count in op_counts.items() if count]
return ", ".join(parts) if parts else "none"
@@ -108,12 +93,14 @@ def process_folder_output(
) -> Optional[str]:
"""Post-process download to the configured folder destination."""
from shelfmark.download.postprocess.pipeline import (
CustomScriptContext,
CustomScriptTransferSummary,
cleanup_output_staging,
is_torrent_source,
log_plan_steps,
prepare_output_files,
maybe_run_custom_script,
record_step,
safe_cleanup_path,
transfer_book_files,
)
@@ -146,73 +133,9 @@ def process_folder_output(
step_name = f"stage_{prepared.output_plan.stage_action}"
record_step(steps, step_name, source=str(temp_file), dest=str(prepared.output_plan.staging_dir))
def run_custom_script(script_path: str, target_path: Path, phase: str) -> bool:
path_mode = core_config.config.get("CUSTOM_SCRIPT_PATH_MODE", "absolute")
script_target = _resolve_custom_script_target(target_path, plan.destination, path_mode)
env = {
**os.environ,
"SHELFMARK_CUSTOM_SCRIPT_TARGET": str(target_path),
"SHELFMARK_CUSTOM_SCRIPT_RELATIVE": str(_resolve_custom_script_target(target_path, plan.destination, "relative")),
"SHELFMARK_CUSTOM_SCRIPT_DESTINATION": str(plan.destination),
"SHELFMARK_CUSTOM_SCRIPT_MODE": str(path_mode),
"SHELFMARK_CUSTOM_SCRIPT_PHASE": phase,
}
record_step(
steps,
"custom_script",
script=str(script_path),
target=str(script_target),
target_abs=str(target_path),
mode=str(path_mode),
phase=phase,
)
log_plan_steps(task.task_id, steps)
logger.info(
"Task %s: running custom script %s on %s (%s)",
task.task_id,
script_path,
script_target,
phase,
)
try:
result = subprocess.run(
[script_path, str(script_target)],
check=True,
timeout=300, # 5 minute timeout
capture_output=True,
text=True,
env=env,
)
if result.stdout:
logger.debug("Task %s: custom script stdout: %s", task.task_id, result.stdout.strip())
return True
except FileNotFoundError:
logger.error("Task %s: custom script not found: %s", task.task_id, script_path)
status_callback("error", f"Custom script not found: {script_path}")
return False
except PermissionError:
logger.error("Task %s: custom script not executable: %s", task.task_id, script_path)
status_callback("error", f"Custom script not executable: {script_path}")
return False
except subprocess.TimeoutExpired:
logger.error("Task %s: custom script timed out after 300s: %s", task.task_id, script_path)
status_callback("error", "Custom script timed out")
return False
except subprocess.CalledProcessError as e:
stderr = e.stderr.strip() if e.stderr else "No error output"
logger.error(
"Task %s: custom script failed (exit code %s): %s",
task.task_id,
e.returncode,
stderr,
)
status_callback("error", f"Custom script failed: {stderr[:100]}")
return False
# Custom script is run post-transfer (see below).
# If we staged a copy into TMP_DIR (e.g. for custom script), transfer from the staged
# path and disable hardlinking for this transfer.
# If we staged into TMP_DIR, transfer from the staged path and disable hardlinking.
use_hardlink = plan.use_hardlink and prepared.output_plan.stage_action == STAGE_NONE
source_path = plan.hardlink_source if use_hardlink and plan.hardlink_source else prepared.working_path
is_torrent = is_torrent_source(source_path, task)
@@ -290,24 +213,29 @@ def process_folder_output(
len(final_paths),
)
# Run custom script once per successful task, after transfer.
if core_config.config.CUSTOM_SCRIPT:
if len(final_paths) == 1:
target_path = final_paths[0]
else:
try:
target_path = Path(os.path.commonpath([str(p.parent) for p in final_paths]))
except ValueError:
target_path = plan.destination
script_context = CustomScriptContext(
task=task,
phase="post_transfer",
output_mode=plan.output_mode,
organization_mode=plan.organization_mode,
destination=plan.destination,
final_paths=final_paths,
transfer=CustomScriptTransferSummary(
op_counts=op_counts,
use_hardlink=use_hardlink,
is_torrent=is_torrent,
preserve_source=preserve_source,
),
)
if not run_custom_script(core_config.config.CUSTOM_SCRIPT, target_path, phase="post_transfer"):
cleanup_output_staging(
prepared.output_plan,
prepared.working_path,
task,
prepared.cleanup_paths,
)
return None
if not maybe_run_custom_script(script_context, status_callback=status_callback, steps=steps):
cleanup_output_staging(
prepared.output_plan,
prepared.working_path,
task,
prepared.cleanup_paths,
)
return None
cleanup_output_staging(
prepared.output_plan,
shelfmark/download/postprocess/custom_script.py
+296
View File
@@ -0,0 +1,296 @@
from __future__ import annotations
import json
import os
import subprocess
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any, Optional
import shelfmark.core.config as core_config
from shelfmark.core.logger import setup_logger
from shelfmark.core.models import DownloadTask
from shelfmark.download.fs import run_blocking_io
from .steps import log_plan_steps, record_step
from .types import PlanStep
logger = setup_logger(__name__)
DEFAULT_CUSTOM_SCRIPT_TIMEOUT_SECONDS = 300 # 5 minutes
def resolve_custom_script_target(target_path: Path, destination: Path, path_mode: str) -> Path:
"""Resolve the path that should be passed as the custom script argument.
In absolute mode, we pass the full target path.
In relative mode, we pass a path relative to the destination folder. If the
target is not within the destination, fall back to just the filename to
avoid leaking unrelated absolute paths.
"""
mode = (path_mode or "absolute").strip().lower()
if mode != "relative":
return target_path
try:
return target_path.relative_to(destination)
except ValueError:
if target_path.is_absolute():
return Path(target_path.name)
return target_path
@dataclass(frozen=True)
class CustomScriptExecution:
script_path: str
target_arg: Path
target_abs: Path
destination: Path
mode: str
phase: str
payload_json: Optional[str] = None
@dataclass(frozen=True)
class CustomScriptTransferSummary:
op_counts: dict[str, int]
use_hardlink: bool
is_torrent: bool
preserve_source: bool
@dataclass(frozen=True)
class CustomScriptContext:
task: DownloadTask
phase: str
output_mode: str
destination: Optional[Path] = None
final_paths: list[Path] = field(default_factory=list)
target_path: Optional[Path] = None
organization_mode: Optional[str] = None
transfer: Optional[CustomScriptTransferSummary] = None
output_details: dict[str, Any] = field(default_factory=dict)
def prepare_custom_script_execution(
script_path: str,
*,
target_path: Path,
destination: Path,
path_mode: str,
phase: str,
payload: Optional[dict[str, Any]] = None,
) -> CustomScriptExecution:
mode = (path_mode or "absolute").strip().lower()
if mode != "relative":
mode = "absolute"
target_arg = resolve_custom_script_target(target_path, destination, mode)
return CustomScriptExecution(
script_path=str(script_path),
target_arg=target_arg,
target_abs=target_path,
destination=destination,
mode=mode,
phase=phase,
payload_json=json.dumps(payload, indent=2, sort_keys=True) + "\n" if payload else None,
)
def run_custom_script(
execution: CustomScriptExecution,
*,
task_id: str,
status_callback,
timeout_seconds: int = DEFAULT_CUSTOM_SCRIPT_TIMEOUT_SECONDS,
) -> bool:
cwd: Optional[str] = None
if execution.mode == "relative":
# Make relative paths unambiguous by running the script from the destination folder.
cwd = str(execution.destination)
logger.info(
"Task %s: running custom script %s on %s (%s)",
task_id,
execution.script_path,
execution.target_arg,
execution.phase,
)
try:
# If we are not sending a JSON payload, close stdin so scripts that try
# to read it won't block indefinitely.
stdin = None if execution.payload_json is not None else subprocess.DEVNULL
result = run_blocking_io(
subprocess.run,
[execution.script_path, str(execution.target_arg)],
check=True,
timeout=timeout_seconds,
capture_output=True,
text=True,
cwd=cwd,
stdin=stdin,
input=execution.payload_json,
)
if result.stdout:
logger.debug("Task %s: custom script stdout: %s", task_id, result.stdout.strip())
return True
except FileNotFoundError:
logger.error("Task %s: custom script not found: %s", task_id, execution.script_path)
status_callback("error", f"Custom script not found: {execution.script_path}")
return False
except PermissionError:
logger.error("Task %s: custom script not executable: %s", task_id, execution.script_path)
status_callback("error", f"Custom script not executable: {execution.script_path}")
return False
except subprocess.TimeoutExpired:
logger.error(
"Task %s: custom script timed out after %ss: %s",
task_id,
timeout_seconds,
execution.script_path,
)
status_callback("error", "Custom script timed out")
return False
except subprocess.CalledProcessError as exc:
stderr = exc.stderr.strip() if exc.stderr else "No error output"
logger.error(
"Task %s: custom script failed (exit code %s): %s",
task_id,
exc.returncode,
stderr,
)
status_callback("error", f"Custom script failed: {stderr[:100]}")
return False
def _choose_custom_script_target(
*,
explicit_target: Optional[Path],
destination: Optional[Path],
final_paths: list[Path],
) -> Optional[Path]:
if explicit_target is not None:
return explicit_target
if len(final_paths) == 1:
return final_paths[0]
if len(final_paths) > 1:
try:
return Path(os.path.commonpath([str(p.parent) for p in final_paths]))
except ValueError:
return destination or final_paths[0].parent
return destination
def _build_custom_script_payload(context: CustomScriptContext, *, target_path: Path) -> dict[str, Any]:
payload: dict[str, Any] = {
"version": 1,
"phase": context.phase,
"task": {
"task_id": context.task.task_id,
"source": context.task.source,
"search_mode": context.task.search_mode.value if context.task.search_mode else None,
"title": context.task.title,
"author": context.task.author,
"year": context.task.year,
"format": context.task.format,
"content_type": context.task.content_type,
"series_name": context.task.series_name,
"series_position": context.task.series_position,
"subtitle": context.task.subtitle,
"original_download_path": context.task.original_download_path,
},
"output": {
"mode": context.output_mode,
"organization_mode": context.organization_mode,
},
"paths": {
"destination": str(context.destination) if context.destination else None,
"target": str(target_path),
"final_paths": [str(p) for p in context.final_paths],
},
}
if context.output_details:
payload["output"]["details"] = context.output_details
if context.transfer:
payload["transfer"] = {
"op_counts": context.transfer.op_counts,
"use_hardlink": context.transfer.use_hardlink,
"is_torrent": context.transfer.is_torrent,
"preserve_source": context.transfer.preserve_source,
}
return payload
def maybe_run_custom_script(
context: CustomScriptContext,
*,
status_callback,
steps: Optional[list[PlanStep]] = None,
) -> bool:
"""Run the custom script hook (if configured).
The output handler provides a `CustomScriptContext` describing what it did.
This function is responsible for choosing the script target, building the
optional JSON payload, and executing the script.
"""
script_path = getattr(core_config.config, "CUSTOM_SCRIPT", None)
if not isinstance(script_path, str) or not script_path.strip():
return True
target_path = _choose_custom_script_target(
explicit_target=context.target_path,
destination=context.destination,
final_paths=context.final_paths,
)
if not target_path:
logger.warning(
"Task %s: custom script configured but no target could be determined; skipping",
context.task.task_id,
)
return True
path_mode = core_config.config.get("CUSTOM_SCRIPT_PATH_MODE", "absolute")
payload: Optional[dict[str, Any]] = None
if core_config.config.get("CUSTOM_SCRIPT_JSON_PAYLOAD", False):
payload = _build_custom_script_payload(context, target_path=target_path)
# If no destination is available for this output, fall back to the target's
# parent directory so the script can still run consistently.
execution_destination = context.destination or target_path.parent
execution = prepare_custom_script_execution(
script_path,
target_path=target_path,
destination=execution_destination,
path_mode=path_mode,
phase=context.phase,
payload=payload,
)
if steps is not None:
payload_bytes = len(execution.payload_json.encode("utf-8")) if execution.payload_json else 0
record_step(
steps,
"custom_script",
script=str(execution.script_path),
target=str(execution.target_arg),
target_abs=str(execution.target_abs),
mode=str(execution.mode),
phase=str(execution.phase),
payload_stdin=bool(execution.payload_json),
payload_bytes=payload_bytes,
)
log_plan_steps(context.task.task_id, steps)
return run_custom_script(execution, task_id=context.task.task_id, status_callback=status_callback)
shelfmark/download/postprocess/pipeline.py
+16
View File
@@ -17,6 +17,15 @@ implementation stay modular.
from __future__ import annotations
from .custom_script import (
CustomScriptExecution,
CustomScriptContext,
CustomScriptTransferSummary,
maybe_run_custom_script,
prepare_custom_script_execution,
resolve_custom_script_target,
run_custom_script,
)
from .destination import get_final_destination, validate_destination
from .prepare import build_output_plan, prepare_output_files
from .scan import (
@@ -50,6 +59,9 @@ __all__ = [
"PlanStep",
"PreparedFiles",
"TransferPlan",
"CustomScriptExecution",
"CustomScriptContext",
"CustomScriptTransferSummary",
"build_metadata_dict",
"build_output_plan",
"cleanup_output_staging",
@@ -62,10 +74,13 @@ __all__ = [
"is_torrent_source",
"is_within_tmp_dir",
"log_plan_steps",
"maybe_run_custom_script",
"prepare_output_files",
"prepare_custom_script_execution",
"process_directory",
"record_step",
"resolve_hardlink_source",
"resolve_custom_script_target",
"safe_cleanup_path",
"scan_directory_tree",
"should_hardlink",
@@ -73,4 +88,5 @@ __all__ = [
"transfer_directory_to_library",
"transfer_file_to_library",
"validate_destination",
"run_custom_script",
]
shelfmark/download/postprocess/prepare.py
+1 -6
View File
@@ -3,10 +3,8 @@ from __future__ import annotations
from pathlib import Path
from typing import Optional
import shelfmark.core.config as core_config
from shelfmark.core.logger import setup_logger
from shelfmark.core.models import DownloadTask
from shelfmark.download.archive import is_archive
from shelfmark.download.staging import STAGE_COPY, STAGE_NONE, get_staging_dir, stage_path
from .scan import collect_staged_files
@@ -27,14 +25,11 @@ def build_output_plan(
"""Build an output plan that describes staging behavior for file-based outputs."""
transfer_plan = resolve_hardlink_source(temp_file, task, destination, status_callback)
runs_custom_script = bool(core_config.config.CUSTOM_SCRIPT) and temp_file.is_file() and not is_archive(temp_file)
stage_action = STAGE_COPY if runs_custom_script and not is_managed_workspace_path(temp_file) else STAGE_NONE
staging_dir = get_staging_dir()
return OutputPlan(
mode=output_mode,
stage_action=stage_action,
stage_action=STAGE_NONE,
staging_dir=staging_dir,
allow_archive_extraction=transfer_plan.allow_archive_extraction,
transfer_plan=transfer_plan,
shelfmark/download/postprocess/scan.py
+45 -18
View File
@@ -8,6 +8,7 @@ from shelfmark.core.logger import setup_logger
from shelfmark.core.models import DownloadTask
from shelfmark.core.utils import is_audiobook as check_audiobook
from shelfmark.download.archive import ArchiveExtractionError, extract_archive, is_archive
from shelfmark.download.fs import run_blocking_io
from shelfmark.download.permissions_debug import log_path_permission_context
from shelfmark.download.postprocess.policy import (
get_supported_audiobook_formats,
@@ -55,7 +56,12 @@ def extract_archive_files(
content_type = task.content_type
try:
extracted_files, warnings, rejected_files = extract_archive(archive_path, output_dir, content_type)
extracted_files, warnings, rejected_files = run_blocking_io(
extract_archive,
archive_path,
output_dir,
content_type,
)
except ArchiveExtractionError as exc:
logger.warning(
"Task %s: archive extraction failed for %s: %s",
@@ -111,10 +117,6 @@ def scan_directory_tree(
logger.warning(f"Cannot access download folder: {directory} ({exc})")
return [], [], [], f"Cannot access download folder: {directory} ({exc})"
book_files: List[Path] = []
rejected_files: List[Path] = []
archive_files: List[Path] = []
supported_formats = get_supported_formats(content_type)
supported_exts = {f".{fmt}" for fmt in supported_formats}
@@ -146,18 +148,35 @@ def scan_directory_tree(
else:
logger.debug(f"Error scanning directory tree: {error}")
for root, _, files in os.walk(directory, onerror=onerror):
for filename in files:
file_path = Path(root) / filename
suffix = file_path.suffix.lower()
def _walk_tree() -> Tuple[List[Path], List[Path], List[Path]]:
book_files: List[Path] = []
rejected_files: List[Path] = []
archive_files: List[Path] = []
if suffix in supported_exts:
book_files.append(file_path)
elif suffix in trackable_exts:
rejected_files.append(file_path)
for root, _, files in os.walk(directory, onerror=onerror):
for filename in files:
file_path = Path(root) / filename
suffix = file_path.suffix.lower()
if is_archive(file_path):
archive_files.append(file_path)
if suffix in supported_exts:
book_files.append(file_path)
elif suffix in trackable_exts:
rejected_files.append(file_path)
if is_archive(file_path):
archive_files.append(file_path)
return book_files, rejected_files, archive_files
try:
book_files, rejected_files, archive_files = run_blocking_io(_walk_tree)
except PermissionError as exc:
log_path_permission_context("scan_directory_walk", directory)
logger.warning(f"Permission denied scanning directory: {directory} ({exc})")
return [], [], [], f"Permission denied accessing download folder: {directory}"
except (FileNotFoundError, NotADirectoryError, OSError) as exc:
logger.warning(f"Cannot access download folder: {directory} ({exc})")
return [], [], [], f"Cannot access download folder: {directory} ({exc})"
return book_files, rejected_files, archive_files, None
@@ -194,12 +213,15 @@ def collect_directory_files(
if archive_files:
if not allow_archive_extraction:
logger.warning(
"Task %s: archive extraction disabled (torrent hardlinking enabled) for %s",
# When extraction is disabled (typically due to torrent hardlinking),
# treat archives as the final importable "files" rather than failing.
logger.info(
"Task %s: archive extraction disabled; importing %d archive(s) as-is from %s",
task.task_id,
len(archive_files),
directory,
)
return [], rejected_files, [], "Archive extraction is disabled when torrent hardlinking is enabled"
return archive_files, rejected_files, [], None
if status_callback:
status_callback("resolving", "Extracting archives")
@@ -293,6 +315,11 @@ def collect_staged_files(
return extracted_files, rejected_files, cleanup_paths, error
if is_archive(working_path) and not allow_archive_extraction:
# When extraction is disabled (typically due to torrent hardlinking),
# import the archive as-is rather than treating it as an unsupported file.
return [working_path], [], [], None
# Single-file download result (non-archive).
# Ensure we respect the user's supported format settings.
suffix = working_path.suffix.lower()
shelfmark/main.py
+3 -1
View File
@@ -235,7 +235,7 @@ werkzeug_logger.addFilter(LogNoiseFilter())
# Set up authentication defaults
# The secret key will reset every time we restart, which will
# require users to authenticate again
from shelfmark.config.env import SESSION_COOKIE_SECURE_ENV, string_to_bool
from shelfmark.config.env import SESSION_COOKIE_NAME, SESSION_COOKIE_SECURE_ENV, string_to_bool
SESSION_COOKIE_SECURE = string_to_bool(SESSION_COOKIE_SECURE_ENV)
@@ -244,10 +244,12 @@ app.config.update(
SESSION_COOKIE_HTTPONLY = True,
SESSION_COOKIE_SAMESITE = 'Lax',
SESSION_COOKIE_SECURE = SESSION_COOKIE_SECURE,
SESSION_COOKIE_NAME = SESSION_COOKIE_NAME,
PERMANENT_SESSION_LIFETIME = 604800 # 7 days in seconds
)
logger.info(f"Session cookie secure setting: {SESSION_COOKIE_SECURE} (from env: {SESSION_COOKIE_SECURE_ENV})")
logger.info(f"Session cookie name: {SESSION_COOKIE_NAME}")
@app.before_request
def proxy_auth_middleware():
shelfmark/release_sources/prowlarr/clients/torrent_utils.py
+6 -3
View File
@@ -134,9 +134,12 @@ def extract_torrent_info(
return TorrentInfo(info_hash=expected_hash, torrent_data=None, is_magnet=False)
def parse_transmission_url(url: str) -> Tuple[str, int, str]:
"""Parse Transmission URL into (host, port, path)."""
def parse_transmission_url(url: str) -> Tuple[str, str, int, str]:
"""Parse Transmission URL into (protocol, host, port, path)."""
parsed = urlparse(url)
protocol = (parsed.scheme or "http").lower()
if protocol not in ("http", "https"):
protocol = "http"
host = parsed.hostname or "localhost"
port = parsed.port or 9091
path = parsed.path or "/transmission/rpc"
@@ -145,7 +148,7 @@ def parse_transmission_url(url: str) -> Tuple[str, int, str]:
if not path.endswith("/rpc"):
path = path.rstrip("/") + "/transmission/rpc"
return host, port, path
return protocol, host, port, path
def bencode_decode(data: bytes) -> tuple:
shelfmark/release_sources/prowlarr/clients/transmission.py
+23 -8
View File
@@ -46,15 +46,30 @@ class TransmissionClient(DownloadClient):
password = config.get("TRANSMISSION_PASSWORD", "")
# Parse URL to extract host, port, and path
host, port, path = parse_transmission_url(url)
protocol, host, port, path = parse_transmission_url(url)
self._client = Client(
host=host,
port=port,
path=path,
username=username if username else None,
password=password if password else None,
)
client_kwargs = {
"host": host,
"port": port,
"path": path,
"username": username if username else None,
"password": password if password else None,
"protocol": protocol,
}
try:
self._client = Client(**client_kwargs)
except TypeError as e:
# Older transmission-rpc versions may not accept protocol as a kwarg.
if "protocol" not in str(e):
raise
client_kwargs.pop("protocol", None)
self._client = Client(**client_kwargs)
# Some versions expose protocol as an attribute rather than kwarg.
if protocol == "https" and hasattr(self._client, "protocol"):
try:
setattr(self._client, "protocol", protocol)
except Exception:
pass
self._category = config.get("TRANSMISSION_CATEGORY", "books")
@staticmethod
shelfmark/release_sources/prowlarr/settings.py
+22 -9
View File
@@ -160,15 +160,28 @@ def _test_transmission_connection(current_values: Optional[Dict[str, Any]] = Non
from transmission_rpc import Client
# Parse URL to extract host, port, and path
host, port, path = parse_transmission_url(url)
protocol, host, port, path = parse_transmission_url(url)
client = Client(
host=host,
port=port,
path=path,
username=username if username else None,
password=password if password else None,
)
client_kwargs = {
"host": host,
"port": port,
"path": path,
"username": username if username else None,
"password": password if password else None,
"protocol": protocol,
}
try:
client = Client(**client_kwargs)
except TypeError as e:
if "protocol" not in str(e):
raise
client_kwargs.pop("protocol", None)
client = Client(**client_kwargs)
if protocol == "https" and hasattr(client, "protocol"):
try:
setattr(client, "protocol", protocol)
except Exception:
pass
session = client.get_session()
version = session.version
return {"success": True, "message": f"Connected to Transmission {version}"}
@@ -547,7 +560,7 @@ def prowlarr_clients_settings():
TextField(
key="TRANSMISSION_URL",
label="Transmission URL",
description="URL of your Transmission instance",
description="URL of your Transmission instance (use https:// for TLS)",
placeholder="http://transmission:9091",
show_when={"field": "PROWLARR_TORRENT_CLIENT", "value": "transmission"},
),
tests/core/test_download_processing.py
+101
View File
@@ -7,6 +7,7 @@ Covers:
- Custom script execution
"""
import json
import os
import pytest
import shutil
@@ -712,6 +713,106 @@ class TestCustomScriptExecution:
result_path = Path(result)
assert call_args[0][0] == ["/path/to/script.sh", str(result_path)]
def test_runs_custom_script_with_json_payload_on_stdin(self, temp_dirs, sample_direct_task):
"""Sends a JSON payload to the custom script via stdin when enabled."""
from shelfmark.download.postprocess.router import post_process_download as _post_process_download
temp_file = temp_dirs["staging"] / "book.epub"
temp_file.write_bytes(b"content")
status_cb = MagicMock()
cancel_flag = Event()
with patch('shelfmark.core.config.config') as mock_config, \
patch('shelfmark.config.env.TMP_DIR', temp_dirs["staging"]), \
patch('subprocess.run') as mock_run:
mock_config.USE_BOOK_TITLE = False
mock_config.CUSTOM_SCRIPT = "/path/to/script.sh"
_sync_core_config(mock_config, mock_config)
mock_config.get = _mock_destination_config(
temp_dirs["ingest"],
{"CUSTOM_SCRIPT_JSON_PAYLOAD": True},
)
_sync_core_config(mock_config, mock_config)
mock_run.return_value = MagicMock(stdout="", returncode=0)
result = _post_process_download(
temp_file=temp_file,
task=sample_direct_task,
cancel_flag=cancel_flag,
status_callback=status_cb,
)
assert result is not None
result_path = Path(result)
payload_json = mock_run.call_args.kwargs.get("input")
assert payload_json
payload = json.loads(payload_json)
assert payload["version"] == 1
assert payload["phase"] == "post_transfer"
assert payload["task"]["task_id"] == sample_direct_task.task_id
assert payload["paths"]["destination"] == str(temp_dirs["ingest"])
assert payload["paths"]["target"] == str(result_path)
assert payload["paths"]["final_paths"] == [str(result_path)]
def test_runs_custom_script_for_booklore_output_with_json_payload(self, temp_dirs, sample_direct_task):
"""Runs the custom script hook after a successful Booklore upload."""
from shelfmark.download.postprocess.router import post_process_download as _post_process_download
temp_file = temp_dirs["staging"] / "book.epub"
temp_file.write_bytes(b"content")
sample_direct_task.task_id = "direct-booklore"
status_cb = MagicMock()
cancel_flag = Event()
with patch('shelfmark.core.config.config') as mock_config, \
patch('shelfmark.config.env.TMP_DIR', temp_dirs["staging"]), \
patch('shelfmark.download.outputs.booklore.booklore_login', return_value="token"), \
patch('shelfmark.download.outputs.booklore.booklore_upload_file'), \
patch('shelfmark.download.outputs.booklore.booklore_refresh_library'), \
patch('subprocess.run') as mock_run:
mock_config.USE_BOOK_TITLE = False
mock_config.CUSTOM_SCRIPT = "/path/to/script.sh"
_sync_core_config(mock_config, mock_config)
mock_config.get = MagicMock(side_effect=lambda key, default=None: {
"BOOKS_OUTPUT_MODE": "booklore",
"BOOKLORE_HOST": "http://booklore:6060",
"BOOKLORE_USERNAME": "user",
"BOOKLORE_PASSWORD": "pass",
"BOOKLORE_LIBRARY_ID": 1,
"BOOKLORE_PATH_ID": 2,
"CUSTOM_SCRIPT_JSON_PAYLOAD": True,
}.get(key, default))
_sync_core_config(mock_config, mock_config)
mock_run.return_value = MagicMock(stdout="", returncode=0)
result = _post_process_download(
temp_file=temp_file,
task=sample_direct_task,
cancel_flag=cancel_flag,
status_callback=status_cb,
)
assert result == "booklore://direct-booklore"
payload_json = mock_run.call_args.kwargs.get("input")
assert payload_json
payload = json.loads(payload_json)
assert payload["version"] == 1
assert payload["phase"] == "post_upload"
assert payload["output"]["mode"] == "booklore"
assert payload["output"]["details"]["booklore"]["library_id"] == 1
assert payload["output"]["details"]["booklore"]["path_id"] == 2
assert payload["paths"]["target"].endswith("/book.epub")
def test_runs_custom_script_with_relative_path_mode(self, temp_dirs, sample_direct_task):
"""Runs custom script with a destination-relative path when configured."""
from shelfmark.download.postprocess.router import post_process_download as _post_process_download
tests/core/test_processing_integration.py
+3 -3
View File
@@ -492,7 +492,7 @@ def test_booklore_mode_rejects_unsupported_files(tmp_path):
staging = tmp_path / "staging"
staging.mkdir()
temp_file = staging / "book.mobi"
temp_file = staging / "book.djvu"
temp_file.write_text("content")
task = DownloadTask(
@@ -500,7 +500,7 @@ def test_booklore_mode_rejects_unsupported_files(tmp_path):
source="direct_download",
title="Unsupported Book",
author="Tester",
format="mobi",
format="djvu",
search_mode=SearchMode.DIRECT,
)
@@ -761,7 +761,7 @@ def test_postprocess_torrent_blackbox_matrix(
def test_custom_script_external_source_stages_copy_and_preserves_source(tmp_path):
"""External (usenet-like) files should be staged into TMP before a custom script runs."""
"""Custom script should run against the final imported file; external source must be preserved."""
from shelfmark.download.postprocess.router import post_process_download as _post_process_download
tests/download/test_postprocess_scan_blocking_io.py
+69
View File
@@ -0,0 +1,69 @@
from __future__ import annotations
from pathlib import Path
from shelfmark.core.models import DownloadTask
from shelfmark.download.postprocess import scan as scan_mod
def test_scan_directory_tree_runs_walk_via_run_blocking_io(tmp_path, monkeypatch) -> None:
(tmp_path / "book.epub").write_text("x", encoding="utf-8")
inside_run_blocking = False
walk_called = False
original_walk = scan_mod.os.walk
def walk_wrapper(*args, **kwargs):
nonlocal walk_called
walk_called = True
assert inside_run_blocking, "os.walk should run within run_blocking_io"
return original_walk(*args, **kwargs)
def run_blocking_io_stub(func, *args, **kwargs):
nonlocal inside_run_blocking
inside_run_blocking = True
try:
return func(*args, **kwargs)
finally:
inside_run_blocking = False
monkeypatch.setattr(scan_mod.os, "walk", walk_wrapper)
monkeypatch.setattr(scan_mod, "run_blocking_io", run_blocking_io_stub)
scan_mod.scan_directory_tree(tmp_path, content_type=None)
assert walk_called, "Expected scan_directory_tree to call os.walk"
def test_extract_archive_files_runs_extract_via_run_blocking_io(tmp_path, monkeypatch) -> None:
inside_run_blocking = False
extract_called = False
def extract_archive_stub(*args, **kwargs):
nonlocal extract_called
extract_called = True
assert inside_run_blocking, "extract_archive should run within run_blocking_io"
return [], [], []
def run_blocking_io_stub(func, *args, **kwargs):
nonlocal inside_run_blocking
inside_run_blocking = True
try:
return func(*args, **kwargs)
finally:
inside_run_blocking = False
monkeypatch.setattr(scan_mod, "extract_archive", extract_archive_stub)
monkeypatch.setattr(scan_mod, "run_blocking_io", run_blocking_io_stub)
task = DownloadTask(task_id="t", source="prowlarr", title="Test", content_type="book")
scan_mod.extract_archive_files(
archive_path=Path("/tmp/fake.zip"),
output_dir=tmp_path,
task=task,
cleanup_archive=False,
)
assert extract_called, "Expected extract_archive_files to call extract_archive"
tests/prowlarr/test_torrent_utils.py
+18 -9
View File
@@ -27,63 +27,72 @@ class TestParseTransmissionUrl:
def test_parse_simple_url(self):
"""Test parsing a simple URL with host and port."""
host, port, path = parse_transmission_url("http://localhost:9091")
protocol, host, port, path = parse_transmission_url("http://localhost:9091")
assert protocol == "http"
assert host == "localhost"
assert port == 9091
assert path == "/transmission/rpc"
def test_parse_url_with_custom_port(self):
"""Test parsing URL with custom port."""
host, port, path = parse_transmission_url("http://myserver:8080")
protocol, host, port, path = parse_transmission_url("http://myserver:8080")
assert protocol == "http"
assert host == "myserver"
assert port == 8080
assert path == "/transmission/rpc"
def test_parse_url_with_path(self):
"""Test parsing URL with existing path."""
host, port, path = parse_transmission_url("http://localhost:9091/transmission/rpc")
protocol, host, port, path = parse_transmission_url("http://localhost:9091/transmission/rpc")
assert protocol == "http"
assert host == "localhost"
assert port == 9091
assert path == "/transmission/rpc"
def test_parse_url_with_partial_path(self):
"""Test parsing URL with partial path appends /rpc."""
host, port, path = parse_transmission_url("http://localhost:9091/custom")
protocol, host, port, path = parse_transmission_url("http://localhost:9091/custom")
assert protocol == "http"
assert host == "localhost"
assert port == 9091
assert path == "/custom/transmission/rpc"
def test_parse_url_with_trailing_slash(self):
"""Test parsing URL with trailing slash."""
host, port, path = parse_transmission_url("http://localhost:9091/")
protocol, host, port, path = parse_transmission_url("http://localhost:9091/")
assert protocol == "http"
assert host == "localhost"
assert port == 9091
assert path == "/transmission/rpc"
def test_parse_url_without_port(self):
"""Test parsing URL without port uses default 9091."""
host, port, path = parse_transmission_url("http://transmission")
protocol, host, port, path = parse_transmission_url("http://transmission")
assert protocol == "http"
assert host == "transmission"
assert port == 9091
assert path == "/transmission/rpc"
def test_parse_https_url(self):
"""Test parsing HTTPS URL."""
host, port, path = parse_transmission_url("https://secure.transmission.local:9091")
protocol, host, port, path = parse_transmission_url("https://secure.transmission.local:9091")
assert protocol == "https"
assert host == "secure.transmission.local"
assert port == 9091
assert path == "/transmission/rpc"
def test_parse_url_with_ip_address(self):
"""Test parsing URL with IP address."""
host, port, path = parse_transmission_url("http://192.168.1.100:9091")
protocol, host, port, path = parse_transmission_url("http://192.168.1.100:9091")
assert protocol == "http"
assert host == "192.168.1.100"
assert port == 9091
assert path == "/transmission/rpc"
def test_parse_empty_url_uses_defaults(self):
"""Test parsing empty URL uses localhost defaults."""
host, port, path = parse_transmission_url("")
protocol, host, port, path = parse_transmission_url("")
assert protocol == "http"
assert host == "localhost"
assert port == 9091
assert path == "/transmission/rpc"
tests/prowlarr/test_transmission_client.py
+30
View File
@@ -124,6 +124,36 @@ class TestTransmissionClientIsConfigured:
class TestTransmissionClientTestConnection:
"""Tests for TransmissionClient.test_connection()."""
def test_init_passes_https_protocol(self, monkeypatch):
"""Test HTTPS URL causes protocol=https to be passed to transmission-rpc Client."""
config_values = {
"TRANSMISSION_URL": "https://localhost:9091",
"TRANSMISSION_USERNAME": "admin",
"TRANSMISSION_PASSWORD": "password",
"TRANSMISSION_CATEGORY": "test",
}
monkeypatch.setattr(
"shelfmark.release_sources.prowlarr.clients.transmission.config.get",
make_config_getter(config_values),
)
mock_client_instance = MagicMock()
mock_client_instance.get_session.return_value = MockSession(version="4.0.5")
mock_transmission_rpc = create_mock_transmission_rpc_module()
mock_transmission_rpc.Client.return_value = mock_client_instance
with patch.dict("sys.modules", {"transmission_rpc": mock_transmission_rpc}):
if "shelfmark.release_sources.prowlarr.clients.transmission" in sys.modules:
del sys.modules["shelfmark.release_sources.prowlarr.clients.transmission"]
from shelfmark.release_sources.prowlarr.clients.transmission import (
TransmissionClient,
)
TransmissionClient()
assert mock_transmission_rpc.Client.call_args.kwargs.get("protocol") == "https"
def test_test_connection_success(self, monkeypatch):
"""Test successful connection."""
config_values = {