mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-09-24 13:50:22 +01:00
Fix LOG_LEVEL being ignored and Z-Library 503 cookie gate (#1188)
LOG_LEVEL never reached the app logger: env.py hardcoded the level to DEBUG or INFO, so INFO lines kept appearing under LOG_LEVEL=error. Read it from the env var and advanced settings, normalize unknown values to INFO, and expose it as a setting. entrypoint.sh now normalizes gunicorn's level too, so a typo falls back to info instead of stopping the container from booting. Z-Library gates the first hit on /md5/<hash> with a 503 whose only payload is a Set-Cookie; echoing that cookie back returns the 302 to the real page. html_get_page dropped it and re-ran the same rejected request on every retry, ending in "No download URL resolved". Retry once with the cookies the 503 issued. Fixes #1185 Fixes #1187
This commit is contained in:
@@ -23,6 +23,7 @@ This document lists all configuration options that can be set via environment va
|
|||||||
- [Hardcover](#metadata-providers-hardcover)
|
- [Hardcover](#metadata-providers-hardcover)
|
||||||
- [Open Library](#metadata-providers-open-library)
|
- [Open Library](#metadata-providers-open-library)
|
||||||
- [Google Books](#metadata-providers-google-books)
|
- [Google Books](#metadata-providers-google-books)
|
||||||
|
- [Moly.hu](#metadata-providers-moly.hu)
|
||||||
- [Direct Download](#direct-download)
|
- [Direct Download](#direct-download)
|
||||||
- [Download Sources](#direct-download-download-sources)
|
- [Download Sources](#direct-download-download-sources)
|
||||||
- [Cloudflare Bypass](#direct-download-cloudflare-bypass)
|
- [Cloudflare Bypass](#direct-download-cloudflare-bypass)
|
||||||
@@ -1046,6 +1047,7 @@ Comma-separated hosts to bypass proxy (e.g., localhost,127.0.0.1,10.*,*.local)
|
|||||||
|----------|-------------|------|---------|
|
|----------|-------------|------|---------|
|
||||||
| `URL_BASE` | Optional URL path prefix. Use a path like /shelfmark (no hostname). Leave blank for root. | string | _none_ |
|
| `URL_BASE` | Optional URL path prefix. Use a path like /shelfmark (no hostname). Leave blank for root. | string | _none_ |
|
||||||
| `DEBUG` | Enable verbose logging to console and file. Not recommended for normal use. | boolean | `false` |
|
| `DEBUG` | Enable verbose logging to console and file. Not recommended for normal use. | boolean | `false` |
|
||||||
|
| `LOG_LEVEL` | Lowest severity written to the console and log file. Ignored while Debug Mode is on, which forces Debug. | string (choice) | `INFO` |
|
||||||
| `MAIN_LOOP_SLEEP_TIME` | How often the download queue is checked for new items. | number | `5` |
|
| `MAIN_LOOP_SLEEP_TIME` | How often the download queue is checked for new items. | number | `5` |
|
||||||
| `DOWNLOAD_PROGRESS_UPDATE_INTERVAL` | How often download progress is broadcast to the UI. | number | `1` |
|
| `DOWNLOAD_PROGRESS_UPDATE_INTERVAL` | How often download progress is broadcast to the UI. | number | `1` |
|
||||||
| `CUSTOM_SCRIPT` | Path to a script to run after each successful download. Must be executable. | string | _none_ |
|
| `CUSTOM_SCRIPT` | Path to a script to run after each successful download. Must be executable. | string | _none_ |
|
||||||
@@ -1082,6 +1084,17 @@ Enable verbose logging to console and file. Not recommended for normal use.
|
|||||||
- **Default:** `false`
|
- **Default:** `false`
|
||||||
- **Requires restart:** Yes
|
- **Requires restart:** Yes
|
||||||
|
|
||||||
|
#### `LOG_LEVEL`
|
||||||
|
|
||||||
|
**Log Level**
|
||||||
|
|
||||||
|
Lowest severity written to the console and log file. Ignored while Debug Mode is on, which forces Debug.
|
||||||
|
|
||||||
|
- **Type:** string (choice)
|
||||||
|
- **Default:** `INFO`
|
||||||
|
- **Requires restart:** Yes
|
||||||
|
- **Options:** `DEBUG` (Debug), `INFO` (Info), `WARNING` (Warning), `ERROR` (Error), `CRITICAL` (Critical)
|
||||||
|
|
||||||
#### `MAIN_LOOP_SLEEP_TIME`
|
#### `MAIN_LOOP_SLEEP_TIME`
|
||||||
|
|
||||||
**Queue Check Interval (seconds)**
|
**Queue Check Interval (seconds)**
|
||||||
@@ -1508,6 +1521,8 @@ How long to keep cached search results before they expire.
|
|||||||
| Variable | Description | Type | Default |
|
| Variable | Description | Type | Default |
|
||||||
|----------|-------------|------|---------|
|
|----------|-------------|------|---------|
|
||||||
| `PROWLARR_TORRENT_CLIENT` | Choose which torrent client to use | string (choice) | _empty string_ |
|
| `PROWLARR_TORRENT_CLIENT` | Choose which torrent client to use | string (choice) | _empty string_ |
|
||||||
|
| `ALLDEBRID_API_KEY` | AllDebrid API Key (apiv4) from your AllDebrid account settings | string (secret) | _none_ |
|
||||||
|
| `REALDEBRID_API_KEY` | Real-Debrid API Key (Secret Token) from your Real-Debrid account settings | string (secret) | _none_ |
|
||||||
| `QBITTORRENT_URL` | Web UI URL of your qBittorrent instance | string | _none_ |
|
| `QBITTORRENT_URL` | Web UI URL of your qBittorrent instance | string | _none_ |
|
||||||
| `QBITTORRENT_USERNAME` | qBittorrent Web UI username | string | _none_ |
|
| `QBITTORRENT_USERNAME` | qBittorrent Web UI username | string | _none_ |
|
||||||
| `QBITTORRENT_PASSWORD` | qBittorrent Web UI password | string (secret) | _none_ |
|
| `QBITTORRENT_PASSWORD` | qBittorrent Web UI password | string (secret) | _none_ |
|
||||||
@@ -1559,7 +1574,25 @@ Choose which torrent client to use
|
|||||||
|
|
||||||
- **Type:** string (choice)
|
- **Type:** string (choice)
|
||||||
- **Default:** _empty string_
|
- **Default:** _empty string_
|
||||||
- **Options:** `""` (None), `qbittorrent` (qBittorrent), `transmission` (Transmission), `deluge` (Deluge), `rtorrent` (rTorrent)
|
- **Options:** `""` (None), `alldebrid` (AllDebrid), `qbittorrent` (qBittorrent), `realdebrid` (Real-Debrid), `transmission` (Transmission), `deluge` (Deluge), `rtorrent` (rTorrent)
|
||||||
|
|
||||||
|
#### `ALLDEBRID_API_KEY`
|
||||||
|
|
||||||
|
**API Key**
|
||||||
|
|
||||||
|
AllDebrid API Key (apiv4) from your AllDebrid account settings
|
||||||
|
|
||||||
|
- **Type:** string (secret)
|
||||||
|
- **Default:** _none_
|
||||||
|
|
||||||
|
#### `REALDEBRID_API_KEY`
|
||||||
|
|
||||||
|
**API Key**
|
||||||
|
|
||||||
|
Real-Debrid API Key (Secret Token) from your Real-Debrid account settings
|
||||||
|
|
||||||
|
- **Type:** string (secret)
|
||||||
|
- **Default:** _none_
|
||||||
|
|
||||||
#### `QBITTORRENT_URL`
|
#### `QBITTORRENT_URL`
|
||||||
|
|
||||||
@@ -2064,6 +2097,26 @@ Default sort order for Google Books search results.
|
|||||||
|
|
||||||
</details>
|
</details>
|
||||||
|
|
||||||
|
### Metadata Providers: Moly.hu
|
||||||
|
|
||||||
|
| Variable | Description | Type | Default |
|
||||||
|
|----------|-------------|------|---------|
|
||||||
|
| `MOLY_ENABLED` | Enable Moly.hu as a metadata provider for book searches | boolean | `false` |
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary>Detailed descriptions</summary>
|
||||||
|
|
||||||
|
#### `MOLY_ENABLED`
|
||||||
|
|
||||||
|
**Enable Moly.hu**
|
||||||
|
|
||||||
|
Enable Moly.hu as a metadata provider for book searches
|
||||||
|
|
||||||
|
- **Type:** boolean
|
||||||
|
- **Default:** `false`
|
||||||
|
|
||||||
|
</details>
|
||||||
|
|
||||||
## Direct Download
|
## Direct Download
|
||||||
|
|
||||||
### Direct Download: Download Sources
|
### Direct Download: Download Sources
|
||||||
|
|||||||
+12
-1
@@ -484,7 +484,18 @@ fi
|
|||||||
# Always run Gunicorn (even when DEBUG=true) to ensure Socket.IO WebSocket
|
# Always run Gunicorn (even when DEBUG=true) to ensure Socket.IO WebSocket
|
||||||
# upgrades work reliably on customer machines.
|
# upgrades work reliably on customer machines.
|
||||||
# Map app LOG_LEVEL (often DEBUG/INFO/...) to gunicorn's --log-level (lowercase).
|
# Map app LOG_LEVEL (often DEBUG/INFO/...) to gunicorn's --log-level (lowercase).
|
||||||
gunicorn_loglevel=$([ "$DEBUG" = "true" ] && echo debug || echo "${LOG_LEVEL:-info}" | tr '[:upper:]' '[:lower:]')
|
# Gunicorn rejects anything outside its own list, so normalize and fall back to
|
||||||
|
# info rather than letting a typo stop the container from booting.
|
||||||
|
if [ "$DEBUG" = "true" ]; then
|
||||||
|
gunicorn_loglevel=debug
|
||||||
|
else
|
||||||
|
gunicorn_loglevel=$(echo "${LOG_LEVEL:-info}" | tr '[:upper:]' '[:lower:]')
|
||||||
|
[ "$gunicorn_loglevel" = "warn" ] && gunicorn_loglevel=warning
|
||||||
|
case "$gunicorn_loglevel" in
|
||||||
|
debug|info|warning|error|critical) ;;
|
||||||
|
*) gunicorn_loglevel=info ;;
|
||||||
|
esac
|
||||||
|
fi
|
||||||
command="${GUNICORN_BIN} --log-level ${gunicorn_loglevel} --access-logfile - --error-logfile - --worker-class geventwebsocket.gunicorn.workers.GeventWebSocketWorker --workers 1 -t 300 -b ${FLASK_HOST:-0.0.0.0}:${FLASK_PORT:-8084} shelfmark.main:app"
|
command="${GUNICORN_BIN} --log-level ${gunicorn_loglevel} --access-logfile - --error-logfile - --worker-class geventwebsocket.gunicorn.workers.GeventWebSocketWorker --workers 1 -t 300 -b ${FLASK_HOST:-0.0.0.0}:${FLASK_PORT:-8084} shelfmark.main:app"
|
||||||
|
|
||||||
# If DEBUG and not using an external bypass
|
# If DEBUG and not using an external bypass
|
||||||
|
|||||||
@@ -260,7 +260,10 @@ Logs are available via:
|
|||||||
- `docker logs <container-name>`
|
- `docker logs <container-name>`
|
||||||
- `/var/log/shelfmark/` inside the container (when `ENABLE_LOGGING=true`)
|
- `/var/log/shelfmark/` inside the container (when `ENABLE_LOGGING=true`)
|
||||||
|
|
||||||
Log level is configurable via Settings or `LOG_LEVEL` environment variable.
|
Log level is configurable under Settings → Advanced or via the `LOG_LEVEL` environment
|
||||||
|
variable (`DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`; case-insensitive, defaults to
|
||||||
|
`INFO`). The environment variable wins over the setting, and `DEBUG=true` forces `DEBUG`
|
||||||
|
regardless of either. Changes take effect on restart.
|
||||||
|
|
||||||
## Development
|
## Development
|
||||||
|
|
||||||
|
|||||||
+57
-13
@@ -6,34 +6,78 @@ import shutil
|
|||||||
import tempfile
|
import tempfile
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
|
LOG_LEVELS = ("DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL")
|
||||||
|
|
||||||
|
|
||||||
def string_to_bool(s: str) -> bool:
|
def string_to_bool(s: str) -> bool:
|
||||||
"""Convert string to boolean."""
|
"""Convert string to boolean."""
|
||||||
return s.lower() in ["true", "yes", "1", "y"]
|
return s.lower() in ["true", "yes", "1", "y"]
|
||||||
|
|
||||||
|
|
||||||
|
def _read_advanced_config(key: str) -> object | None:
|
||||||
|
"""Read a key from the advanced settings file (import-time safe)."""
|
||||||
|
config_dir = Path(os.getenv("CONFIG_DIR", "/config"))
|
||||||
|
config_file = config_dir / "plugins" / "advanced.json"
|
||||||
|
|
||||||
|
if config_file.exists():
|
||||||
|
try:
|
||||||
|
with config_file.open() as f:
|
||||||
|
config = json.load(f)
|
||||||
|
if key in config:
|
||||||
|
return config[key]
|
||||||
|
except json.JSONDecodeError, OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
def _read_debug_from_config() -> bool:
|
def _read_debug_from_config() -> bool:
|
||||||
"""Read DEBUG from env var or config file (import-time safe)."""
|
"""Read DEBUG from env var or config file (import-time safe)."""
|
||||||
env_debug = os.environ.get("DEBUG")
|
env_debug = os.environ.get("DEBUG")
|
||||||
if env_debug is not None:
|
if env_debug is not None:
|
||||||
return string_to_bool(env_debug)
|
return string_to_bool(env_debug)
|
||||||
|
|
||||||
# Try to read from config file
|
value = _read_advanced_config("DEBUG")
|
||||||
config_dir = Path(os.getenv("CONFIG_DIR", "/config"))
|
if value is not None:
|
||||||
config_file = config_dir / "plugins" / "advanced.json"
|
return bool(value)
|
||||||
|
|
||||||
if config_file.exists():
|
|
||||||
try:
|
|
||||||
with config_file.open() as f:
|
|
||||||
config = json.load(f)
|
|
||||||
if "DEBUG" in config:
|
|
||||||
return bool(config["DEBUG"])
|
|
||||||
except json.JSONDecodeError, OSError:
|
|
||||||
pass
|
|
||||||
|
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_log_level(raw: str | None) -> str:
|
||||||
|
"""Normalize a log level name, falling back to INFO when unrecognized."""
|
||||||
|
if raw is None:
|
||||||
|
return "INFO"
|
||||||
|
|
||||||
|
normalized = raw.strip().upper()
|
||||||
|
# "WARN" is a logging alias, but gunicorn only accepts "warning".
|
||||||
|
if normalized == "WARN":
|
||||||
|
normalized = "WARNING"
|
||||||
|
|
||||||
|
if normalized not in LOG_LEVELS:
|
||||||
|
return "INFO"
|
||||||
|
|
||||||
|
return normalized
|
||||||
|
|
||||||
|
|
||||||
|
def _read_log_level_from_config(debug: bool) -> str:
|
||||||
|
"""Resolve the app log level from DEBUG, env var, or config file.
|
||||||
|
|
||||||
|
DEBUG wins when enabled, mirroring how entrypoint.sh picks gunicorn's level.
|
||||||
|
Otherwise LOG_LEVEL is read from the env var, then the settings file, and
|
||||||
|
falls back to INFO when unset or unrecognized.
|
||||||
|
"""
|
||||||
|
if debug:
|
||||||
|
return "DEBUG"
|
||||||
|
|
||||||
|
raw = os.environ.get("LOG_LEVEL")
|
||||||
|
if raw is None:
|
||||||
|
value = _read_advanced_config("LOG_LEVEL")
|
||||||
|
raw = value if isinstance(value, str) else None
|
||||||
|
|
||||||
|
return normalize_log_level(raw)
|
||||||
|
|
||||||
|
|
||||||
def _is_sqlite_file(path: Path) -> bool:
|
def _is_sqlite_file(path: Path) -> bool:
|
||||||
"""Check if a file is a valid SQLite database by reading magic bytes."""
|
"""Check if a file is a valid SQLite database by reading magic bytes."""
|
||||||
try:
|
try:
|
||||||
@@ -101,7 +145,7 @@ INGEST_DIR = Path(os.getenv("INGEST_DIR", "/books"))
|
|||||||
# =============================================================================
|
# =============================================================================
|
||||||
|
|
||||||
DEBUG = _read_debug_from_config()
|
DEBUG = _read_debug_from_config()
|
||||||
LOG_LEVEL = "DEBUG" if DEBUG else "INFO"
|
LOG_LEVEL = _read_log_level_from_config(DEBUG)
|
||||||
ENABLE_LOGGING = string_to_bool(os.getenv("ENABLE_LOGGING", "true"))
|
ENABLE_LOGGING = string_to_bool(os.getenv("ENABLE_LOGGING", "true"))
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1761,6 +1761,23 @@ def advanced_settings() -> list[SettingsField]:
|
|||||||
default=False,
|
default=False,
|
||||||
requires_restart=True,
|
requires_restart=True,
|
||||||
),
|
),
|
||||||
|
SelectField(
|
||||||
|
key="LOG_LEVEL",
|
||||||
|
label="Log Level",
|
||||||
|
description=(
|
||||||
|
"Lowest severity written to the console and log file. "
|
||||||
|
"Ignored while Debug Mode is on, which forces Debug."
|
||||||
|
),
|
||||||
|
options=[
|
||||||
|
{"value": "DEBUG", "label": "Debug", "description": "Everything, very noisy."},
|
||||||
|
{"value": "INFO", "label": "Info", "description": "Normal activity (default)."},
|
||||||
|
{"value": "WARNING", "label": "Warning", "description": "Warnings and problems."},
|
||||||
|
{"value": "ERROR", "label": "Error", "description": "Failures only."},
|
||||||
|
{"value": "CRITICAL", "label": "Critical", "description": "Fatal errors only."},
|
||||||
|
],
|
||||||
|
default="INFO",
|
||||||
|
requires_restart=True,
|
||||||
|
),
|
||||||
NumberField(
|
NumberField(
|
||||||
key="MAIN_LOOP_SLEEP_TIME",
|
key="MAIN_LOOP_SLEEP_TIME",
|
||||||
label="Queue Check Interval (seconds)",
|
label="Queue Check Interval (seconds)",
|
||||||
|
|||||||
@@ -9,6 +9,7 @@ from typing import TYPE_CHECKING, Any
|
|||||||
|
|
||||||
from werkzeug.utils import secure_filename
|
from werkzeug.utils import secure_filename
|
||||||
|
|
||||||
|
from shelfmark.config.env import normalize_log_level
|
||||||
from shelfmark.core.logger import setup_logger
|
from shelfmark.core.logger import setup_logger
|
||||||
from shelfmark.core.request_helpers import coerce_bool, normalize_optional_text
|
from shelfmark.core.request_helpers import coerce_bool, normalize_optional_text
|
||||||
|
|
||||||
@@ -898,6 +899,9 @@ def _get_env_value_for_field(field: FieldBase) -> tuple[bool, object | None]:
|
|||||||
"WELIB_MIRROR_URLS",
|
"WELIB_MIRROR_URLS",
|
||||||
} and isinstance(parsed, list):
|
} and isinstance(parsed, list):
|
||||||
parsed = _normalize_mirror_env_urls(parsed)
|
parsed = _normalize_mirror_env_urls(parsed)
|
||||||
|
if field.key == "LOG_LEVEL" and isinstance(parsed, str):
|
||||||
|
# LOG_LEVEL is commonly set lowercase; the field options are uppercase.
|
||||||
|
parsed = normalize_log_level(parsed)
|
||||||
return True, parsed
|
return True, parsed
|
||||||
|
|
||||||
if field.key == "AA_MIRROR_URLS":
|
if field.key == "AA_MIRROR_URLS":
|
||||||
|
|||||||
@@ -27,9 +27,14 @@ logger = setup_logger(__name__)
|
|||||||
_RNG = random.SystemRandom()
|
_RNG = random.SystemRandom()
|
||||||
|
|
||||||
_MAX_REDIRECTS = 5
|
_MAX_REDIRECTS = 5
|
||||||
|
# Z-Library answers the first hit with a 503 whose only real payload is a Set-Cookie; echoing
|
||||||
|
# that cookie back returns the 302 to the real page. Two attempts cover the handshake without
|
||||||
|
# letting a server that keeps re-issuing cookies hold us in the loop.
|
||||||
|
_MAX_COOKIE_HANDSHAKE_RETRIES = 2
|
||||||
_HTTP_STATUS_FORBIDDEN = HTTPStatus.FORBIDDEN
|
_HTTP_STATUS_FORBIDDEN = HTTPStatus.FORBIDDEN
|
||||||
_HTTP_STATUS_NOT_FOUND = HTTPStatus.NOT_FOUND
|
_HTTP_STATUS_NOT_FOUND = HTTPStatus.NOT_FOUND
|
||||||
_HTTP_STATUS_RATE_LIMITED = HTTPStatus.TOO_MANY_REQUESTS
|
_HTTP_STATUS_RATE_LIMITED = HTTPStatus.TOO_MANY_REQUESTS
|
||||||
|
_HTTP_STATUS_SERVICE_UNAVAILABLE = HTTPStatus.SERVICE_UNAVAILABLE
|
||||||
_HTTP_STATUS_OK = HTTPStatus.OK
|
_HTTP_STATUS_OK = HTTPStatus.OK
|
||||||
_HTTP_STATUS_RANGE_NOT_SATISFIABLE = HTTPStatus.REQUESTED_RANGE_NOT_SATISFIABLE
|
_HTTP_STATUS_RANGE_NOT_SATISFIABLE = HTTPStatus.REQUESTED_RANGE_NOT_SATISFIABLE
|
||||||
_HTTP_STATUS_PARTIAL_CONTENT = HTTPStatus.PARTIAL_CONTENT
|
_HTTP_STATUS_PARTIAL_CONTENT = HTTPStatus.PARTIAL_CONTENT
|
||||||
@@ -58,6 +63,18 @@ def _raise_too_many_redirects(message: str) -> NoReturn:
|
|||||||
raise requests.exceptions.TooManyRedirects(message)
|
raise requests.exceptions.TooManyRedirects(message)
|
||||||
|
|
||||||
|
|
||||||
|
def _new_cookies(response: requests.Response, already_sent: dict[str, str]) -> dict[str, str]:
|
||||||
|
"""Cookies a response set that we were not already echoing back.
|
||||||
|
|
||||||
|
Returning only the *new* ones is what makes the retry terminate: a server that keeps
|
||||||
|
re-issuing the same cookie yields nothing here, so we stop instead of spinning.
|
||||||
|
"""
|
||||||
|
jar = getattr(response, "cookies", None)
|
||||||
|
if not jar:
|
||||||
|
return {}
|
||||||
|
return {name: value for name, value in jar.items() if already_sent.get(name) != value}
|
||||||
|
|
||||||
|
|
||||||
def _get_internal_bypasser() -> ModuleType:
|
def _get_internal_bypasser() -> ModuleType:
|
||||||
"""Lazy import of internal bypasser module."""
|
"""Lazy import of internal bypasser module."""
|
||||||
global _internal_bypasser
|
global _internal_bypasser
|
||||||
@@ -278,6 +295,9 @@ def html_get_page(
|
|||||||
original_url = url
|
original_url = url
|
||||||
current_url = selector.rewrite(original_url)
|
current_url = selector.rewrite(original_url)
|
||||||
use_bypasser_now = use_bypasser
|
use_bypasser_now = use_bypasser
|
||||||
|
# Survives across attempts so a cookie won once is still presented on later retries.
|
||||||
|
handshake_cookies: dict[str, str] = {}
|
||||||
|
handshake_retries = 0
|
||||||
|
|
||||||
for attempt in range(1, retry_limit + 1):
|
for attempt in range(1, retry_limit + 1):
|
||||||
# Check for cancellation before each attempt
|
# Check for cancellation before each attempt
|
||||||
@@ -332,12 +352,31 @@ def html_get_page(
|
|||||||
current_url,
|
current_url,
|
||||||
proxies=get_proxies(current_url),
|
proxies=get_proxies(current_url),
|
||||||
timeout=REQUEST_TIMEOUT,
|
timeout=REQUEST_TIMEOUT,
|
||||||
cookies=cookies,
|
# Bypasser-derived cookies win: they came from a real solved challenge.
|
||||||
|
cookies={**handshake_cookies, **cookies},
|
||||||
headers=headers,
|
headers=headers,
|
||||||
allow_redirects=allow_redirects,
|
allow_redirects=allow_redirects,
|
||||||
verify=get_ssl_verify(current_url),
|
verify=get_ssl_verify(current_url),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# Z-Library gates the first hit with a 503 that carries nothing but a
|
||||||
|
# Set-Cookie; echoing it back yields the 302 to the real page. Without this
|
||||||
|
# the cookie is dropped and every retry re-runs the same rejected request.
|
||||||
|
if (
|
||||||
|
response.status_code == _HTTP_STATUS_SERVICE_UNAVAILABLE
|
||||||
|
and handshake_retries < _MAX_COOKIE_HANDSHAKE_RETRIES
|
||||||
|
):
|
||||||
|
issued = _new_cookies(response, handshake_cookies)
|
||||||
|
if issued:
|
||||||
|
handshake_cookies.update(issued)
|
||||||
|
handshake_retries += 1
|
||||||
|
logger.debug(
|
||||||
|
"503 set %s cookie(s); retrying with them: %s",
|
||||||
|
len(issued),
|
||||||
|
current_url,
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
|
||||||
if is_aa_url and response.is_redirect:
|
if is_aa_url and response.is_redirect:
|
||||||
location = response.headers.get("Location", "")
|
location = response.headers.get("Location", "")
|
||||||
if not location:
|
if not location:
|
||||||
@@ -366,6 +405,7 @@ def html_get_page(
|
|||||||
current_url = new_url
|
current_url = new_url
|
||||||
# Reset per-request state for the new host.
|
# Reset per-request state for the new host.
|
||||||
headers = {"User-Agent": DOWNLOAD_HEADERS["User-Agent"]}
|
headers = {"User-Agent": DOWNLOAD_HEADERS["User-Agent"]}
|
||||||
|
handshake_cookies.clear()
|
||||||
is_aa_url = network.should_rotate_dns_for_url(current_url)
|
is_aa_url = network.should_rotate_dns_for_url(current_url)
|
||||||
allow_redirects = not is_aa_url
|
allow_redirects = not is_aa_url
|
||||||
redirects_followed = 0
|
redirects_followed = 0
|
||||||
@@ -435,6 +475,7 @@ def html_get_page(
|
|||||||
new_url = _try_rotation(original_url, current_url, selector)
|
new_url = _try_rotation(original_url, current_url, selector)
|
||||||
if new_url:
|
if new_url:
|
||||||
current_url = new_url
|
current_url = new_url
|
||||||
|
handshake_cookies.clear()
|
||||||
continue
|
continue
|
||||||
|
|
||||||
# Retry with backoff
|
# Retry with backoff
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ Run with: uv run pytest tests/config/test_environment.py -v
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
import importlib
|
import importlib
|
||||||
|
import json
|
||||||
import os
|
import os
|
||||||
import tempfile
|
import tempfile
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
@@ -15,6 +16,15 @@ from unittest.mock import patch
|
|||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
|
|
||||||
|
def _restore_env(monkeypatch, name: str, value: str | None) -> None:
|
||||||
|
"""Restore an env var to its pre-test value."""
|
||||||
|
if value is None:
|
||||||
|
monkeypatch.delenv(name, raising=False)
|
||||||
|
else:
|
||||||
|
monkeypatch.setenv(name, value)
|
||||||
|
|
||||||
|
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
# Directory Setup Tests
|
# Directory Setup Tests
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
@@ -436,6 +446,7 @@ class TestDebugConfiguration:
|
|||||||
original_debug = os.environ.get("DEBUG")
|
original_debug = os.environ.get("DEBUG")
|
||||||
|
|
||||||
try:
|
try:
|
||||||
|
monkeypatch.delenv("LOG_LEVEL", raising=False)
|
||||||
monkeypatch.setenv("DEBUG", "true")
|
monkeypatch.setenv("DEBUG", "true")
|
||||||
importlib.reload(env_module)
|
importlib.reload(env_module)
|
||||||
assert env_module.DEBUG is True
|
assert env_module.DEBUG is True
|
||||||
@@ -452,6 +463,76 @@ class TestDebugConfiguration:
|
|||||||
monkeypatch.setenv("DEBUG", original_debug)
|
monkeypatch.setenv("DEBUG", original_debug)
|
||||||
importlib.reload(env_module)
|
importlib.reload(env_module)
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("raw", "expected"),
|
||||||
|
[
|
||||||
|
("error", "ERROR"),
|
||||||
|
("ERROR", "ERROR"),
|
||||||
|
(" Warning ", "WARNING"),
|
||||||
|
("warn", "WARNING"),
|
||||||
|
("critical", "CRITICAL"),
|
||||||
|
("nonsense", "INFO"),
|
||||||
|
("", "INFO"),
|
||||||
|
(None, "INFO"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_normalize_log_level(self, raw, expected):
|
||||||
|
"""Log level names are case-insensitive and fall back to INFO."""
|
||||||
|
from shelfmark.config.env import normalize_log_level
|
||||||
|
|
||||||
|
assert normalize_log_level(raw) == expected
|
||||||
|
|
||||||
|
def test_log_level_from_env_var(self, monkeypatch):
|
||||||
|
"""LOG_LEVEL env var should set the app log level when DEBUG is off."""
|
||||||
|
import shelfmark.config.env as env_module
|
||||||
|
|
||||||
|
original_debug = os.environ.get("DEBUG")
|
||||||
|
original_level = os.environ.get("LOG_LEVEL")
|
||||||
|
|
||||||
|
try:
|
||||||
|
monkeypatch.setenv("DEBUG", "false")
|
||||||
|
monkeypatch.setenv("LOG_LEVEL", "error")
|
||||||
|
importlib.reload(env_module)
|
||||||
|
assert env_module.LOG_LEVEL == "ERROR"
|
||||||
|
|
||||||
|
# DEBUG wins over LOG_LEVEL, matching entrypoint.sh.
|
||||||
|
monkeypatch.setenv("DEBUG", "true")
|
||||||
|
importlib.reload(env_module)
|
||||||
|
assert env_module.LOG_LEVEL == "DEBUG"
|
||||||
|
finally:
|
||||||
|
_restore_env(monkeypatch, "DEBUG", original_debug)
|
||||||
|
_restore_env(monkeypatch, "LOG_LEVEL", original_level)
|
||||||
|
importlib.reload(env_module)
|
||||||
|
|
||||||
|
def test_log_level_from_config_file(self, monkeypatch, tmp_path):
|
||||||
|
"""LOG_LEVEL should fall back to the advanced settings file."""
|
||||||
|
import shelfmark.config.env as env_module
|
||||||
|
|
||||||
|
original_debug = os.environ.get("DEBUG")
|
||||||
|
original_level = os.environ.get("LOG_LEVEL")
|
||||||
|
original_config_dir = os.environ.get("CONFIG_DIR")
|
||||||
|
|
||||||
|
advanced = tmp_path / "plugins" / "advanced.json"
|
||||||
|
advanced.parent.mkdir(parents=True)
|
||||||
|
advanced.write_text(json.dumps({"LOG_LEVEL": "WARNING"}))
|
||||||
|
|
||||||
|
try:
|
||||||
|
monkeypatch.setenv("DEBUG", "false")
|
||||||
|
monkeypatch.delenv("LOG_LEVEL", raising=False)
|
||||||
|
monkeypatch.setenv("CONFIG_DIR", str(tmp_path))
|
||||||
|
importlib.reload(env_module)
|
||||||
|
assert env_module.LOG_LEVEL == "WARNING"
|
||||||
|
|
||||||
|
# Env var takes precedence over the stored setting.
|
||||||
|
monkeypatch.setenv("LOG_LEVEL", "critical")
|
||||||
|
importlib.reload(env_module)
|
||||||
|
assert env_module.LOG_LEVEL == "CRITICAL"
|
||||||
|
finally:
|
||||||
|
_restore_env(monkeypatch, "DEBUG", original_debug)
|
||||||
|
_restore_env(monkeypatch, "LOG_LEVEL", original_level)
|
||||||
|
_restore_env(monkeypatch, "CONFIG_DIR", original_config_dir)
|
||||||
|
importlib.reload(env_module)
|
||||||
|
|
||||||
|
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
# Proxy and Network Configuration Tests
|
# Proxy and Network Configuration Tests
|
||||||
|
|||||||
@@ -28,6 +28,9 @@ _BOOTSTRAP_ENV_ACCESS_ALLOWLIST = {
|
|||||||
}
|
}
|
||||||
_BOOTSTRAP_ENV_ACCESS_KEY_ALLOWLIST = {
|
_BOOTSTRAP_ENV_ACCESS_KEY_ALLOWLIST = {
|
||||||
(Path("shelfmark/config/settings.py"), "USING_TOR"),
|
(Path("shelfmark/config/settings.py"), "USING_TOR"),
|
||||||
|
# Loggers are configured while settings_registry itself is still importing,
|
||||||
|
# so the level has to come from the bootstrap env module.
|
||||||
|
(Path("shelfmark/core/logger.py"), "LOG_LEVEL"),
|
||||||
}
|
}
|
||||||
_RAW_CONFIG_READ_ALLOWLIST = {
|
_RAW_CONFIG_READ_ALLOWLIST = {
|
||||||
Path("shelfmark/config/notifications_settings.py"),
|
Path("shelfmark/config/notifications_settings.py"),
|
||||||
@@ -205,7 +208,10 @@ class ConfigAccessVisitor(ast.NodeVisitor):
|
|||||||
return
|
return
|
||||||
for alias in node.names:
|
for alias in node.names:
|
||||||
imported_name = alias.name
|
imported_name = alias.name
|
||||||
if imported_name in self.registered_keys:
|
if (
|
||||||
|
imported_name in self.registered_keys
|
||||||
|
and imported_name not in self._bootstrap_env_key_allowlist
|
||||||
|
):
|
||||||
self._record_violation(
|
self._record_violation(
|
||||||
node,
|
node,
|
||||||
"direct env-module import",
|
"direct env-module import",
|
||||||
|
|||||||
@@ -0,0 +1,103 @@
|
|||||||
|
"""Tests for the Z-Library 503 cookie handshake."""
|
||||||
|
|
||||||
|
import requests
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeResponse:
|
||||||
|
"""Minimal stand-in for requests.Response covering what html_get_page touches."""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
status_code: int,
|
||||||
|
*,
|
||||||
|
url: str = "https://z-lib.fm/md5/abc",
|
||||||
|
text: str = "",
|
||||||
|
cookies: dict[str, str] | None = None,
|
||||||
|
) -> None:
|
||||||
|
self.status_code = status_code
|
||||||
|
self.url = url
|
||||||
|
self.text = text
|
||||||
|
self.cookies = cookies or {}
|
||||||
|
self.headers = {"Content-Type": "text/html;charset=utf-8"}
|
||||||
|
self.is_redirect = False
|
||||||
|
|
||||||
|
def raise_for_status(self) -> None:
|
||||||
|
if self.status_code >= 400:
|
||||||
|
error = requests.exceptions.HTTPError(f"{self.status_code} Error")
|
||||||
|
error.response = self
|
||||||
|
raise error
|
||||||
|
|
||||||
|
|
||||||
|
def _neutralize_network(monkeypatch, http):
|
||||||
|
monkeypatch.setattr(http, "_apply_cf_bypass", lambda _url, _headers: {})
|
||||||
|
monkeypatch.setattr(http, "get_proxies", lambda _url: {})
|
||||||
|
monkeypatch.setattr(http, "get_ssl_verify", lambda _url: True)
|
||||||
|
monkeypatch.setattr(http.network, "should_rotate_dns_for_url", lambda _url: False)
|
||||||
|
monkeypatch.setattr(http.time, "sleep", lambda _seconds: None)
|
||||||
|
|
||||||
|
|
||||||
|
def test_html_get_page_echoes_503_cookie(monkeypatch):
|
||||||
|
"""A 503 that only sets a cookie is cleared by sending that cookie back."""
|
||||||
|
import shelfmark.download.http as http
|
||||||
|
|
||||||
|
_neutralize_network(monkeypatch, http)
|
||||||
|
|
||||||
|
sent_cookies: list[dict[str, str]] = []
|
||||||
|
|
||||||
|
def fake_get(_url: str, **kwargs):
|
||||||
|
sent_cookies.append(dict(kwargs["cookies"]))
|
||||||
|
if len(sent_cookies) == 1:
|
||||||
|
return _FakeResponse(503, cookies={"zlib_sid": "s3cr3t"})
|
||||||
|
return _FakeResponse(200, text="<html>real page</html>")
|
||||||
|
|
||||||
|
monkeypatch.setattr(http.requests, "get", fake_get)
|
||||||
|
|
||||||
|
html = http.html_get_page("https://z-lib.fm/md5/abc", retry=3, success_delay=0)
|
||||||
|
|
||||||
|
assert html == "<html>real page</html>"
|
||||||
|
# The first hit carries nothing; the retry echoes back exactly what the 503 issued.
|
||||||
|
assert sent_cookies == [{}, {"zlib_sid": "s3cr3t"}]
|
||||||
|
|
||||||
|
|
||||||
|
def test_html_get_page_stops_echoing_when_cookie_is_reissued(monkeypatch):
|
||||||
|
"""A server repeating the same cookie must not spin the request loop forever."""
|
||||||
|
import shelfmark.download.http as http
|
||||||
|
|
||||||
|
_neutralize_network(monkeypatch, http)
|
||||||
|
monkeypatch.setattr(http, "_is_cf_bypass_enabled", lambda: False)
|
||||||
|
|
||||||
|
sent_cookies: list[dict[str, str]] = []
|
||||||
|
|
||||||
|
def fake_get(_url: str, **kwargs):
|
||||||
|
sent_cookies.append(dict(kwargs["cookies"]))
|
||||||
|
return _FakeResponse(503, cookies={"zlib_sid": "same"})
|
||||||
|
|
||||||
|
monkeypatch.setattr(http.requests, "get", fake_get)
|
||||||
|
|
||||||
|
html = http.html_get_page("https://z-lib.fm/md5/abc", retry=1, success_delay=0)
|
||||||
|
|
||||||
|
assert html == ""
|
||||||
|
# One initial hit plus one echo; the reissued identical cookie yields no third request.
|
||||||
|
assert sent_cookies == [{}, {"zlib_sid": "same"}]
|
||||||
|
|
||||||
|
|
||||||
|
def test_html_get_page_leaves_cookieless_503_on_the_retry_path(monkeypatch):
|
||||||
|
"""A plain overloaded-server 503 keeps its existing retry behaviour."""
|
||||||
|
import shelfmark.download.http as http
|
||||||
|
|
||||||
|
_neutralize_network(monkeypatch, http)
|
||||||
|
monkeypatch.setattr(http, "_is_cf_bypass_enabled", lambda: False)
|
||||||
|
|
||||||
|
attempts: list[dict[str, str]] = []
|
||||||
|
|
||||||
|
def fake_get(_url: str, **kwargs):
|
||||||
|
attempts.append(dict(kwargs["cookies"]))
|
||||||
|
return _FakeResponse(503)
|
||||||
|
|
||||||
|
monkeypatch.setattr(http.requests, "get", fake_get)
|
||||||
|
|
||||||
|
html = http.html_get_page("https://z-lib.fm/md5/abc", retry=2, success_delay=0)
|
||||||
|
|
||||||
|
assert html == ""
|
||||||
|
# Two ordinary attempts, no extra in-place retry and no cookies invented.
|
||||||
|
assert attempts == [{}, {}]
|
||||||
Reference in New Issue
Block a user