mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-09-24 19:40:26 +01:00
A download that hits Cloudflare hung on "Bypassing protection..." for five minutes and then died, regardless of which bypasser was configured. html_get_page() started a BypassHeartbeat thread to keep the download marked alive during a bypass, but the thread had no loop: it fired one status event and returned. Even with the loop restored it could not have worked, because update_download_status() dedupes identical (status, message) tuples and returns before refreshing _last_activity, and the heartbeat re-sent the byte-identical payload already emitted just above it. So _last_activity was frozen for the whole bypass, while both bypassers are allowed to run longer than STALL_TIMEOUT (external FlareSolverr ~394s at default settings, internal 420s per get() call). The watchdog always won. From a reporter's log: 403 at 07:04:33.390, cancelled at 07:09:33.987 - exactly 300.000s, and 41s before the bypasser would have finished and reported the real error, an HTTP 500 from FlareSolverr the user never saw. The regression is not one commit.1f093de(#536) added the heartbeat and the dedup together and refreshed activity before the dedup return, so it worked.ff094be(#832) moved the refresh below that return while tightening stall detection for #823.3a3a3ce(#845) then deleted the heartbeat's while loop to silence a B023 lint, removing the last evidence of intent. The dedup itself is correct and stays: a keep-alive that ticks on a timer proves nothing about whether an operation is progressing, so letting it refresh the stall clock would make a wedged download immortal. Split the two concerns instead. Add shelfmark/download/activity.py. A long single-shot operation declares its own upper bound once, over a sentinel status carried on the existing status_callback channel - so no new parameter has to be threaded through every handler, post-processor and output module. The orchestrator intercepts the sentinel in its per-task closure and records an absolute deadline in _activity_grace, which stall detection honours alongside STALL_TIMEOUT. The grace never extends itself and is clamped to _MAX_ACTIVITY_GRACE_SECONDS, so an operation that overruns its own declared budget is still cancelled. Each bypasser now reports max_duration_seconds() derived from its own retry and timeout settings, and http.py asks whichever is active, plus 30s of slack so the bypasser's own deadline expires first and the user sees its real failure. On that path html_get_page() also emits status_callback("error", ...) rather than silently returning an empty page. Three further fixes on the same code path: - Extract the watchdog into _find_stalled_tasks() and _cancel_stalled_task(). It was the only place holding _progress_lock across a call into book_queue, whose terminal-status hooks reach a sqlite write that gevent does not patch, blocking the hub and every download worker. It now holds the lock for dict reads only. - Bound _CDP_WORKER.run(), which waited with timeout=None while holding the module-wide LOCKED, so a single wedged in-process CDP session blocked every subsequent bypass forever on non-Docker installs. - Broaden the coordinator loop's except clause back to Exception, with escalating backoff.8d98e12(#868) narrowed it to a six-type tuple to silence BLE001, which let gevent's LoopExit and similar kill the only thread driving the download queue - undoing #832's fix for #823 and resurfacing it as #1166. GreenletExit and gevent.Timeout still propagate. Fixes #1001 Refs #1166, #823
242 lines
9.5 KiB
Docker
242 lines
9.5 KiB
Docker
ARG TARGETPLATFORM
|
|
ARG TARGETARCH
|
|
ARG BUILDPLATFORM
|
|
ARG BUILDARCH
|
|
|
|
# Frontend build stage.
|
|
FROM --platform=$BUILDPLATFORM node:24-alpine@sha256:d32cdf619f63fe0471182d08996dd516c6275bb5fd31ae06e55a570bd9e1ad43 AS frontend-builder
|
|
|
|
# Helpful debug output to see what platforms BuildKit thinks it's using
|
|
RUN echo "BUILDPLATFORM=$BUILDPLATFORM BUILDARCH=$BUILDARCH TARGETPLATFORM=$TARGETPLATFORM TARGETARCH=$TARGETARCH"
|
|
|
|
WORKDIR /frontend
|
|
|
|
# Copy frontend package files
|
|
COPY src/frontend/package*.json ./
|
|
|
|
# Install dependencies (cache mount for faster rebuilds)
|
|
RUN --mount=type=cache,target=/root/.npm \
|
|
npm ci
|
|
|
|
# Copy frontend source
|
|
COPY src/frontend/ ./
|
|
|
|
# Build the frontend
|
|
RUN npm run build
|
|
|
|
# Use python-slim as the base image
|
|
FROM python:3.14.6-slim@sha256:cea0e6040540fb2b965b6e7fb5ffa00871e632eef63719f0ea54bca189ce14a6 AS base
|
|
|
|
COPY --from=ghcr.io/astral-sh/uv:0.11.3@sha256:90bbb3c16635e9627f49eec6539f956d70746c409209041800a0280b93152823 /uv /uvx /bin/
|
|
|
|
# Add build argument for version
|
|
ARG BUILD_VERSION
|
|
ENV BUILD_VERSION=${BUILD_VERSION}
|
|
ARG RELEASE_VERSION
|
|
ENV RELEASE_VERSION=${RELEASE_VERSION}
|
|
|
|
# Set shell to bash with pipefail option
|
|
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
|
|
|
|
# Consistent environment variables grouped together
|
|
ENV DEBIAN_FRONTEND=noninteractive \
|
|
DOCKERMODE=true \
|
|
UV_LINK_MODE=copy \
|
|
PYTHONUNBUFFERED=1 \
|
|
PYTHONDONTWRITEBYTECODE=1 \
|
|
PYTHONIOENCODING=UTF-8 \
|
|
NAME=Shelfmark \
|
|
PATH=/app/.venv/bin:$PATH \
|
|
PYTHONPATH=/app \
|
|
# PUID/PGID will be handled by entrypoint script, but TZ/Locale are still needed
|
|
LANG=en_US.UTF-8 \
|
|
LANGUAGE=en_US:en \
|
|
LC_ALL=en_US.UTF-8
|
|
|
|
# Set ARG for build-time expansion (FLASK_PORT), ENV for runtime access
|
|
ENV FLASK_PORT=8084
|
|
|
|
# Configure locale, timezone, and perform initial cleanup in a single layer
|
|
RUN apt-get update && \
|
|
apt-get install -y --no-install-recommends \
|
|
# For building C-extensions (cffi, gevent, etc.)
|
|
gcc \
|
|
g++ \
|
|
libffi-dev \
|
|
python3-dev \
|
|
# For locale
|
|
locales tzdata \
|
|
# For healthcheck
|
|
curl \
|
|
# For entrypoint
|
|
dumb-init \
|
|
# For debug
|
|
zip iputils-ping \
|
|
# For user switching
|
|
gosu \
|
|
# --- Tor support (activated via USING_TOR=true) ---
|
|
tor \
|
|
supervisor \
|
|
iptables \
|
|
# --- WireGuard support (activated via USING_WIREGUARD=true) ---
|
|
wireguard-tools \
|
|
iproute2 \
|
|
procps \
|
|
ca-certificates && \
|
|
# Configure iptables alternatives for tor.sh compatibility
|
|
update-alternatives --set iptables /usr/sbin/iptables-legacy && \
|
|
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy && \
|
|
# Cleanup APT cache *after* all installs in this layer
|
|
apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false && \
|
|
apt-get clean && \
|
|
rm -rf /var/lib/apt/lists/* && \
|
|
# Default to UTC timezone but will be overridden by the entrypoint script
|
|
ln -snf /usr/share/zoneinfo/UTC /etc/localtime && echo UTC > /etc/timezone && \
|
|
# Configure locale
|
|
sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && \
|
|
locale-gen en_US.UTF-8 && \
|
|
echo "LC_ALL=en_US.UTF-8" >> /etc/environment && \
|
|
echo "LANG=en_US.UTF-8" > /etc/locale.conf
|
|
|
|
# Create a fixed runtime user/group so hardened Docker/Kubernetes deployments
|
|
# can start the container directly as a non-root user with a passwd entry.
|
|
RUN groupadd -g 1000 shelfmark && \
|
|
useradd -u 1000 -g shelfmark -d /home/shelfmark -s /usr/sbin/nologin shelfmark && \
|
|
mkdir -p /home/shelfmark && \
|
|
chown 1000:1000 /home/shelfmark
|
|
|
|
# Set working directory
|
|
WORKDIR /app
|
|
|
|
# Install core Python dependencies first for better layer caching
|
|
COPY pyproject.toml uv.lock ./
|
|
RUN --mount=type=cache,target=/root/.cache/uv \
|
|
uv sync --locked --no-default-groups
|
|
|
|
# Runtime dependencies are installed into /app/.venv during the build. Remove the
|
|
# base image's system pip so stale installer CVEs do not ship in the final image.
|
|
RUN rm -rf \
|
|
/usr/local/bin/pip \
|
|
/usr/local/bin/pip3 \
|
|
/usr/local/bin/pip3.* \
|
|
/usr/local/lib/python*/site-packages/pip \
|
|
/usr/local/lib/python*/site-packages/pip-*.dist-info
|
|
|
|
# Copy application code *after* dependencies are installed
|
|
COPY . .
|
|
|
|
# Copy built frontend from frontend-builder stage
|
|
COPY --from=frontend-builder /frontend/dist /app/frontend-dist
|
|
|
|
# Final setup: create image-owned runtime paths for the fixed non-root user.
|
|
# Root/PUID mode still re-homes ownership at startup when needed.
|
|
RUN mkdir -p \
|
|
/config \
|
|
/books \
|
|
/var/log/shelfmark \
|
|
/tmp/shelfmark/seleniumbase/downloaded_files \
|
|
/tmp/shelfmark/seleniumbase/archived_files && \
|
|
rm -rf /app/downloaded_files /app/archived_files && \
|
|
ln -s /tmp/shelfmark/seleniumbase/downloaded_files /app/downloaded_files && \
|
|
ln -s /tmp/shelfmark/seleniumbase/archived_files /app/archived_files && \
|
|
chown -R 1000:1000 /config /books /home/shelfmark /tmp/shelfmark /var/log/shelfmark && \
|
|
chmod -R a+rX /app && \
|
|
chmod +x /app/entrypoint.sh /app/tor.sh /app/wireguard.sh /app/genDebug.sh
|
|
|
|
# Expose the application port
|
|
EXPOSE ${FLASK_PORT}
|
|
|
|
# Add healthcheck for container status
|
|
# Uses /api/health which doesn't require authentication
|
|
HEALTHCHECK --interval=60s --timeout=60s --start-period=60s --retries=3 \
|
|
CMD curl -s http://localhost:${FLASK_PORT}/api/health > /dev/null || exit 1
|
|
|
|
# Use dumb-init as the entrypoint to handle signals properly
|
|
ENTRYPOINT ["/usr/bin/dumb-init", "--"]
|
|
|
|
|
|
FROM base AS shelfmark
|
|
|
|
# --- Chromium (PINNED to 149.0.7827.196) ---
|
|
# Debian's chromium 150.0.7871.46-1~deb13u1 security update (trixie-security,
|
|
# 2026-07-05) no longer opens the DevTools remote-debugging TCP port at all
|
|
# (no listener, no DevToolsActivePort file, even with a custom --user-data-dir;
|
|
# the RemoteDebuggingAllowed policy does not restore it). The SeleniumBase
|
|
# Pure-CDP driver connects through that port (/json/version), so with 150 every
|
|
# internal bypass dies with "Pure CDP browser startup failed" and all
|
|
# CF-gated downloads fail. Install the last working version from
|
|
# snapshot.debian.org until the bypasser can talk to Chromium >= 150 (e.g.
|
|
# pipe-based DevTools / UC mode) or seleniumbase ships a fix.
|
|
# Chrome 144+ requires --enable-unsafe-swiftshader for WebGL in Docker.
|
|
# This flag is set in internal_bypasser.py _get_browser_args()
|
|
ARG CHROMIUM_VERSION=149.0.7827.196-1~deb13u1
|
|
ARG CHROMIUM_SNAPSHOT=20260704T000000Z
|
|
|
|
RUN echo "deb [check-valid-until=no] https://snapshot.debian.org/archive/debian-security/${CHROMIUM_SNAPSHOT}/ trixie-security main" \
|
|
> /etc/apt/sources.list.d/chromium-pin-snapshot.list && \
|
|
apt-get update -o Acquire::Retries=5 && \
|
|
apt-get install -y --no-install-recommends -o Acquire::Retries=5 \
|
|
# For dumb display
|
|
xvfb \
|
|
# For screen recording
|
|
ffmpeg \
|
|
chromium=${CHROMIUM_VERSION} \
|
|
chromium-common=${CHROMIUM_VERSION} \
|
|
# For tkinter (pyautogui)
|
|
python3-tk \
|
|
# For RAR extraction
|
|
unrar-free && \
|
|
# Keep apt from "upgrading" chromium past the pin inside derived images
|
|
printf 'Package: chromium chromium-common\nPin: version %s\nPin-Priority: 1001\n' "${CHROMIUM_VERSION}" \
|
|
> /etc/apt/preferences.d/chromium-pin && \
|
|
rm /etc/apt/sources.list.d/chromium-pin-snapshot.list && \
|
|
# Create symlink so rarfile library can find unrar
|
|
ln -sf /usr/bin/unrar-free /usr/bin/unrar && \
|
|
# Cleanup APT cache
|
|
apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false && \
|
|
apt-get clean && \
|
|
rm -rf /var/lib/apt/lists/*
|
|
|
|
# Install the browser automation stack used by the full image
|
|
RUN --mount=type=cache,target=/root/.cache/uv \
|
|
uv sync --locked --no-default-groups --extra browser
|
|
|
|
# Deterministically resolve the Xlib namespace collision.
|
|
# pyautogui/mouseinfo pull the stale `python3-xlib` (0.15, 2014), while the
|
|
# `--extra browser` set pulls `python-xlib` (0.33). Both packages install into
|
|
# the same top-level `Xlib/` namespace, so whichever lands last wins. When the
|
|
# 2014 build wins, `Xlib.X` is missing `FamilyServerInterpreted`, which the
|
|
# SeleniumBase Pure-CDP driver requires at browser startup -> every bypass fails
|
|
# with "module 'Xlib.X' has no attribute 'FamilyServerInterpreted'" and no
|
|
# Cloudflare/DDoS-Guard protected download can complete. Drop the stale package
|
|
# and force python-xlib 0.33 to own the namespace. pyautogui runs fine against
|
|
# 0.33 (superset API).
|
|
RUN --mount=type=cache,target=/root/.cache/uv \
|
|
uv pip uninstall --python /app/.venv/bin/python python3-xlib && \
|
|
uv pip install --python /app/.venv/bin/python --reinstall python-xlib==0.33 && \
|
|
/app/.venv/bin/python -c "import Xlib.X; assert hasattr(Xlib.X, 'FamilyServerInterpreted'), 'Xlib.X.FamilyServerInterpreted missing after fix'; print('Xlib namespace OK:', Xlib.__version__)"
|
|
|
|
# uv is only needed while building the image.
|
|
RUN rm -f /usr/bin/uv /usr/bin/uvx
|
|
|
|
# Keep SeleniumBase's bundled driver cache writable for the fixed non-root user.
|
|
RUN SELENIUMBASE_DRIVERS_DIR=$(/app/.venv/bin/python -c "import pathlib, seleniumbase; print(pathlib.Path(seleniumbase.__file__).resolve().parent / 'drivers')") && \
|
|
chown -R 1000:1000 "${SELENIUMBASE_DRIVERS_DIR}" && \
|
|
chmod -R u+rwX,go+rX "${SELENIUMBASE_DRIVERS_DIR}" && \
|
|
if [ -f "${SELENIUMBASE_DRIVERS_DIR}/uc_driver" ]; then chmod +x "${SELENIUMBASE_DRIVERS_DIR}/uc_driver"; fi
|
|
|
|
# Grant read/execute permissions to others
|
|
RUN chmod -R o+rx /usr/bin/chromium
|
|
|
|
# Default command to run the application entrypoint script
|
|
CMD ["/app/entrypoint.sh"]
|
|
|
|
FROM base AS shelfmark-lite
|
|
|
|
ENV USING_EXTERNAL_BYPASSER=true
|
|
|
|
# uv is only needed while building the image.
|
|
RUN rm -f /usr/bin/uv /usr/bin/uvx
|
|
|
|
CMD ["/app/entrypoint.sh"]
|