Files
shelfmark/tests
helgehelge123andhelgehelge123 7b9c416df8 perf(bypass): keep the helper subprocess alive between bypasses (#1222)
Every protected request spawns a fresh helper subprocess, paying
interpreter start and imports before any work begins. Measured inside
the container, five consecutive runs of `python -c "import
shelfmark.bypass.internal_bypasser"`:

```
3.53s  3.45s  3.55s  3.54s  3.46s
```

A single search issues several protected requests, so that is paid
several times over per search.

## What changed

The helper now serves one JSON request per line of stdin until the
parent closes the pipe, and an idle timer
(`BYPASS_BROWSER_IDLE_TIMEOUT`, default 180s) shuts it down once
searching stops.

Answers still travel by result file, but the file is now written to a
`.part` path and renamed into place — the parent treats the file's
existence as the answer, so it must never observe a half-written one.
stdout and stderr stay attached to the parent's, so helper logs keep
appearing in `docker logs` exactly as before.

Failure handling, since a warm helper is exposed to more of it than a
per-request one ever was:

| Situation | Handling |
| --- | --- |
| Helper died between requests | Detected via `poll()`, respawned |
| Pipe broken at write time (`poll()` can miss this) | One retry on a
fresh process; a fresh one failing there is a real failure |
| Helper exits without writing a result | `RuntimeError` naming the exit
code |
| Wedged past the timeout, or cancelled mid-bypass | Helper killed, then
`_cleanup_orphan_processes` because a killed helper never got to close
Chrome |
| Idle reaper racing an arriving request | Re-checks the deadline under
the lock and re-arms instead of killing a helper that just did work |

The DNS config now travels with every request rather than only at spawn:
a warm helper outlives changes the parent makes to its provider.

## `BYPASS_REUSE_BROWSER`, off by default

This parks the CDP driver between bypasses. A driver's websockets are
bound to the loop that opened them and cannot outlive their process, so
the persistent helper is what makes this possible at all — and the warm
path runs on `_CDP_WORKER`'s long-lived loop rather than `asyncio.run`
for the same reason.

The mechanism works. With it on, the browser start disappears from the
second request onward: 0.7s from `Reusing warm Chrome browser` to the
first bypass attempt, against roughly 16s cold.

**It still ships off, because a matched-pair test shows it is a net loss
against DDoS-Guard.** Each round primed with one cold bypass, waited
10s, then measured a second — identical timing in both arms, only the
browser strategy differing, order balanced (fresh, warm, warm, fresh) so
drift over the session cannot masquerade as an effect:

| Arm | Measured request |
| --- | --- |
| fresh browser | 42.8s, 40.6s |
| warm browser | 57.1s, 59.6s |

Spread within each arm is 2.2s and 2.5s, against 16.7s between them.
Reuse removes the ~15s browser start and then gives back roughly twice
that in solving: a returning browser draws a harder challenge. Where the
cold browser is through on the second bypass method, the warm one fails
the first three and only `_bypass_method_humanlike` gets it, at ~30s for
that method alone.

Worth separating from a second effect I ran into while measuring: five
back-to-back searches slow from ~32s to 51–98s with reuse **disabled**
as well, so DDoS-Guard escalates on request rate independently of any of
this. That is why the pairs above are timed identically rather than
simply run in sequence. It is the larger of the two effects, but not
something this project can patch around.

Reuse is left available rather than dropped because Cloudflare sites may
not respond the same way, and because the two concerns are independent:
the helper start is pure overhead and always worth removing, the browser
is not.

## Verification

- 2559 unit tests pass (2542 before, 17 added in
`tests/bypass/test_warm_browser.py`)
- `ruff check`, `ruff format`, `basedpyright` over backend and tests,
and `vulture` all clean
- `docs/environment-variables.md` regenerated via
`scripts/generate_env_docs.py`
- Live against Anna's Archive on a warm helper: searches return their
usual ~760KB and 667 results, the app's own search warm-up completes
with 50 results, and the container is left with no orphan
chrome/Xvfb/ffmpeg processes

Happy to drop the `BYPASS_REUSE_BROWSER` half entirely if you would
rather not carry a default-off path — the helper persistence stands on
its own.

Co-authored-by: helgehelge123 <helge.neumann@zollsoft.de>
2026-08-20 19:00:06 -04:00
..
2026-06-23 16:58:24 -04:00

Test Suite

This directory contains the test suite for Shelfmark. Tests are organized by scope and component.

Quick Start

# Sync the local Python environment once
make install-python-dev

# Run all unit tests locally (fast, no external dependencies)
uv run pytest tests/ -v -m "not integration and not e2e"

# Run all Python static analysis (lint, format, typecheck, dead code)
make python-checks

# Run E2E API tests against a running app stack
uv run pytest tests/e2e/ -v -m e2e

# Run everything except integration tests locally
uv run pytest tests/ -v -m "not integration"

Test Structure

tests/
├── config/                 # Settings & configuration tests
│   ├── test_docker_volumes.py          # Docker volume mapping
│   ├── test_environment.py             # Environment variable handling
│   ├── test_mirror_settings_live_apply.py  # Mirror settings live reload
│   ├── test_mirror_settings_options.py # Mirror settings options
│   ├── test_security.py               # Security settings
│   └── test_oidc_settings.py          # OIDC settings fields & show_when conditions
│
├── core/                   # Core application logic tests
│   ├── test_admin_users_api.py         # Admin user CRUD API endpoints
│   ├── test_booklore_multiuser.py      # BookLore per-user override merging
│   ├── test_builtin_multiuser.py       # Builtin auth multi-user migration
│   ├── test_download_processing.py     # Download file processing
│   ├── test_hardlink.py               # Hardlink/copy operations
│   ├── test_library_processing.py      # Library file processing
│   ├── test_manual_query.py           # Manual search query handling
│   ├── test_mirrors_config.py         # Mirror configuration
│   ├── test_naming.py                 # File naming templates
│   ├── test_oidc_auth.py             # OIDC auth helpers (group claims, user provisioning)
│   ├── test_oidc_integration.py       # OIDC integration into auth system (logic mirror)
│   ├── test_oidc_routes.py           # OIDC Flask route handlers
│   ├── test_part_number_extraction.py  # Part number extraction
│   ├── test_per_user_downloads.py     # Per-user download queue filtering
│   ├── test_permission_handling.py    # File permission handling
│   ├── test_processing_integration.py  # Processing integration
│   ├── test_search_plan.py           # Search plan logic
│   ├── test_user_db.py               # UserDB CRUD operations
│   └── test_user_template_variable.py  # {User} template variable in naming
│
├── e2e/                    # End-to-end API tests
│   ├── conftest.py         # Fixtures (APIClient, DownloadTracker)
│   ├── test_api.py         # Core API endpoint tests
│   ├── test_download_flow.py   # Full download journey tests
│   └── test_prowlarr_flow.py   # Prowlarr-specific tests
│
├── prowlarr/               # Prowlarr plugin tests
│   ├── conftest.py         # Shared fixtures
│   ├── test_clients.py     # DownloadClient base, registry, DownloadStatus
│   ├── test_qbittorrent_client.py  # qBittorrent client unit tests
│   ├── test_transmission_client.py # Transmission client unit tests
│   ├── test_nzbget_client.py       # NZBGet client unit tests
│   ├── test_sabnzbd_client.py      # SABnzbd client unit tests
│   ├── test_handler.py     # ProwlarrHandler unit tests
│   ├── test_torrent_utils.py   # Bencode, hash extraction, URL parsing
│   ├── test_bencode.py     # Bencode encoding/decoding
│   ├── test_source.py      # Release source (size parsing, format detection)
│   ├── test_cache.py       # Release cache
│   ├── test_integration_clients.py  # Integration tests (require Docker stack)
│   └── test_integration_handler.py  # Handler integration tests
│
└── README.md               # This file

Test Types

Unit Tests

Fast tests that mock external dependencies. Run these frequently during development.

uv run pytest tests/prowlarr/ -v -m "not integration"

What they test:

  • Download client logic (status mapping, URL handling, error cases)
  • Bencode encoding/decoding for torrent files
  • Hash extraction from magnet links and .torrent files
  • Protocol detection (torrent vs usenet)
  • Release cache operations
  • Handler download flow logic
  • User database (CRUD, settings, OIDC subject linking)
  • OIDC authentication (group claims, user provisioning, route handlers)
  • Admin user management API (create, update, delete, password, per-user settings)
  • Multi-user download queue filtering and per-user overrides
  • Settings configuration (OIDC fields, show_when conditions)

E2E Tests

Test the full application through its HTTP API. Require the app to be running.

uv run pytest tests/e2e/ -v -m e2e

What they test:

  • Health check endpoint
  • Configuration endpoint
  • Metadata provider search (Hardcover, etc.)
  • Release source listing
  • Download queue operations (add, cancel, reorder, clear)
  • Settings API
  • Prowlarr integration

Integration Tests

Test against real services (qBittorrent, Transmission, etc.). Require the full Docker test stack.

# Start the test stack first
docker compose -f docker-compose.test-clients.yml up -d

# Run integration tests
docker compose -f docker-compose.test-clients.yml exec shelfmark uv run pytest tests/prowlarr/ -v -m integration

What they test:

  • Real connections to download clients
  • Adding/removing actual torrents
  • Status polling from real clients

Test Markers

Marker Description When to Skip
integration Requires running services (qBittorrent, etc.) Default skip with -m "not integration"
e2e End-to-end API tests When app isn't running
slow Tests that take longer (network calls, polling) Quick feedback with -m "not slow"

Common Commands

# Run specific test file
uv run pytest tests/prowlarr/test_clients.py -v

# Run specific test class
uv run pytest tests/e2e/test_api.py::TestHealthEndpoint -v

# Run specific test
uv run pytest tests/e2e/test_api.py::TestHealthEndpoint::test_health_returns_ok -v

# Run with short traceback (cleaner output)
uv run pytest tests/ -v --tb=short -m "not integration"

# Run and stop on first failure
uv run pytest tests/ -v -x -m "not integration"

# Run with coverage (if pytest-cov installed)
uv run pytest tests/ --cov=shelfmark -m "not integration"

Writing New Tests

Unit Test Example

from unittest.mock import MagicMock, patch


class TestMyFeature:
    def test_something(self, monkeypatch):
        # Mock config values
        monkeypatch.setattr(
            "shelfmark.module.config.get",
            lambda key, default="": {"KEY": "value"}.get(key, default),
        )

        # Test your code
        result = my_function()
        assert result == expected

E2E Test Example

import pytest
from .conftest import APIClient, DownloadTracker


@pytest.mark.e2e
class TestMyEndpoint:
    def test_endpoint_works(self, protected_api_client: APIClient):
        resp = protected_api_client.get("/api/my-endpoint")
        assert resp.status_code == 200

    def test_with_cleanup(
        self,
        protected_api_client: APIClient,
        download_tracker: DownloadTracker,
    ):
        # Track IDs for automatic cleanup after test
        download_tracker.track("some-id")
        # ... test code ...

Test Fixtures

E2E Fixtures (tests/e2e/conftest.py)

Fixture Scope Description
api_client function Fresh HTTP client for general E2E calls
protected_api_client function Authenticated client for protected-route E2Es
download_tracker function Tracks downloads for cleanup
server_config session Cached server configuration

Prowlarr Fixtures (tests/prowlarr/conftest.py)

Fixture Scope Description
transmission_client module Real Transmission client (integration)
qbittorrent_client module Real qBittorrent client (integration)
deluge_client module Real Deluge client (integration)
nzbget_client module Real NZBGet client (integration)
sabnzbd_client module Real SABnzbd client (integration)

Expected Skips

Some tests skip when external services aren't available. This is normal:

  • "No metadata providers available" - Metadata provider not responding
  • "Prowlarr not configured" - Prowlarr settings not set up
  • "No releases found" - No indexers configured in Prowlarr
  • "Legacy search source unavailable" - Direct download source offline
  • "Transmission/qBittorrent not available" - Docker test stack not running

Troubleshooting

Tests can't connect to app

# Check the app/container is running
docker ps

# Check app logs
docker logs <your-shelfmark-container>

Import errors

# Sync the local Python environment first
uv sync --locked --extra browser

# Then run tests from the repo root
uv run pytest ...

Integration tests failing

# Make sure test stack is running
docker compose -f docker-compose.test-clients.yml up -d

# Check client containers
docker ps | grep -E "qbittorrent|transmission|deluge|nzbget|sabnzbd"

Stale test data

Restart the container to reset the in-memory queue between test runs:

docker restart <your-shelfmark-container>