Files
shelfmark/Dockerfile
T
splitsec2 c6685feba5 fix(docker): drop ffmpeg from the default image; install on demand in Debug mode (#1631)
ffmpeg is in the default image for one thing, the Debug mode screen
recording (`_start_ffmpeg_recording`). Nothing else calls it,
SeleniumBase included, and it brings 93 packages with it. On real builds
that's 406 Trivy findings (149 High) and 78 MB of the gzipped amd64
image. This takes it out of the image and installs it when Debug mode
needs it.

It came out of the base image discussion on #1616 and helps on any base,
though on Wolfi the install line would need `apk`.

## What changed

- **Dockerfile:** ffmpeg is gone from the full image's install.
- **Guard:** `_start_ffmpeg_recording` checks `shutil.which("ffmpeg")`
first. If it's missing it logs a warning and returns, and the bypass
carries on. Without this, Debug on with no ffmpeg fails every bypass
with `FileNotFoundError`. There's a new test for it.
- **Entrypoint:** when Debug is on, the internal bypasser is in use and
ffmpeg isn't already there, it installs ffmpeg in the background so the
app doesn't wait for it. The download and the install each run under a
120 s `timeout` with `DEBIAN_FRONTEND=noninteractive`. Downloading first
means a hung mirror can only be killed before dpkg starts, and the next
start runs `dpkg --configure -a` in case a restart cut dpkg off. It logs
when it starts, succeeds, times out or fails.
- **Docs:** a short paragraph in `docs/troubleshooting.md` covering the
install and the non-root limit.

## Debug has two sources

Debug can come from the `DEBUG` env var or from the Debug Mode setting
in the UI, which is saved to `/config/plugins/advanced.json`. The
existing block in `entrypoint.sh` only reads the env var, so an install
keyed on `$DEBUG` would miss anyone who turned it on in the UI.

The new check asks the app instead (`shelfmark.config.env.DEBUG`), which
reads both. The UI setting is `requires_restart`, so the restart it
already asks for is what brings it through the entrypoint.

## Non-root mode

A container started as `1000:1000` can't run `apt-get`, so the
entrypoint logs that it can't install ffmpeg, the guard warns on each
bypass, and there's no recording. Everything else works. The workaround
for a debugging session is to start as root with `PUID`/`PGID`.

## Startup cost

None that I could measure. With Debug on, `/api/health` answers in the
same time as main (3-4 s with Debug only in config, 9-10 s from env),
and ffmpeg was ready 6-8 s later on the runners.

The trade-off is that the first bypass after startup may be unrecorded
while the install runs. It logs the warning and passes. I saw both
outcomes on a bypass started right after health.

From reading the code: the install lives in the container, so recreating
the container installs it again. It also runs after `tor.sh` and
`wireguard.sh`, so apt goes through whichever egress is active.

## Testing

Real Docker builds on GitHub Actions on my fork, amd64 and arm64, with
the same result on both for every case below.

| Case | Result |
|---|---|
| Debug off | No install, `/api/health` in 3-4 s, same as main |
| Debug only in config | Health in 3-4 s, ffmpeg ready 6-8 s later, real
mp4 after a bypass |
| Debug from env | Health in 9-10 s (same as main), real mp4 after a
bypass |
| Started as root, app dropped to uid 1000 | Smoke and bypass pass |
| Started with `--user 1000`, Debug on | Logs both messages, bypass
passes, no mp4 |
| apt blocked (closed proxy) | Logs `Could not install ffmpeg`, bypass
passes, no mp4 |
| apt's proxy on a dropped address, so no TCP reply at all | Health in
9-10 s, a bypass during the install passes with the warning and no mp4.
apt gives up by itself in under a minute. Nothing left running, `dpkg
--audit` clean |
| DNS server on a dropped address | Health in 9-10 s, `Timed out
installing ffmpeg after 120s`. Nothing left running, `dpkg --audit`
clean |
| `--network none`, Debug on | No hang, health 200 from inside the
container |
| Container restarted while dpkg was running | Next start finishes the
install, `dpkg --audit` clean, real mp4 |
| Fresh named volume, Debug only in config | Recording made, uid 1000
can write |
| Existing uid-1000 volume, Debug only in config | Same, nothing
root-owned afterwards |
| `find /config /var/log/shelfmark -user root` | 0 files, same as main |
| `find /app -user root` | 245, same count as main |
| `tests/e2e/platform/run-e2e.sh` with `dns-blocked` and `baseline` |
Pass |

The bypass checks ran against example.com. A real Cloudflare challenge
was served by a demo page in the same runs, and no build solved it,
including main, so that part doesn't say anything about this change.

Locally: 5494 Python tests pass, and `ruff`, `basedpyright`, `vulture`
and `shellcheck` are clean.

The full results are in [the base image
report](https://github.com/splitsec2/shelfmark/blob/7afcecfb76727724a71561cbc078f091569b58dc/docs/research/2026-11-01-base-image-report.md).
2026-11-01 18:09:11 -07:00

299 lines
13 KiB
Docker

ARG TARGETPLATFORM
ARG TARGETARCH
ARG BUILDPLATFORM
ARG BUILDARCH
# Frontend build stage.
FROM --platform=$BUILDPLATFORM node:24-alpine@sha256:ebfe2f90462722a7a4de65e91990e97fe0d401c70e0e762c5b53302f905ec1c1 AS frontend-builder
# Helpful debug output to see what platforms BuildKit thinks it's using
RUN echo "BUILDPLATFORM=$BUILDPLATFORM BUILDARCH=$BUILDARCH TARGETPLATFORM=$TARGETPLATFORM TARGETARCH=$TARGETARCH"
WORKDIR /frontend
# Copy frontend package files
COPY src/frontend/package*.json ./
# Install dependencies (cache mount for faster rebuilds)
RUN --mount=type=cache,target=/root/.npm \
npm ci
# Copy frontend source
COPY src/frontend/ ./
# Build the frontend
RUN npm run build
# uv is a build-time tool only, so it is mounted into the RUNs that need it rather
# than copied into the image. A COPY here would land ~24 MB in a `base` layer that
# every published image inherits, and a later `rm` cannot take it back out again --
# a RUN adds a layer, it does not rewrite the one underneath.
FROM ghcr.io/astral-sh/uv:0.13.6@sha256:af22a52f5eafef3d78e551599d405c02df61919d18028f0cea8eda93bc9986d0 AS uv
# Use python-slim as the base image
FROM python:3.14.9-slim@sha256:35caba83af2fe6da9a5d922f9b0156f21d1246a48278c58a08bd253303ba42e8 AS base
# Set shell to bash with pipefail option
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
# Consistent environment variables grouped together
ENV DEBIAN_FRONTEND=noninteractive \
DOCKERMODE=true \
UV_LINK_MODE=copy \
PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
PYTHONIOENCODING=UTF-8 \
NAME=Shelfmark \
PATH=/app/.venv/bin:$PATH \
PYTHONPATH=/app \
# PUID/PGID will be handled by entrypoint script, but TZ/Locale are still needed
LANG=en_US.UTF-8 \
LANGUAGE=en_US:en \
LC_ALL=en_US.UTF-8
# Set ARG for build-time expansion (FLASK_PORT), ENV for runtime access
ENV FLASK_PORT=8084
# Configure locale, timezone, and perform initial cleanup in a single layer
RUN apt-get update && \
apt-get install -y --no-install-recommends \
# No C toolchain: every compiled dependency (cffi, gevent, greenlet,
# zope-interface) ships a cp314 manylinux wheel for amd64 and arm64, and
# the sdist-only ones are pure Python. The toolchain was 290MB of the
# runtime image, and python3-dev pulled Debian's libpython3.13 into a 3.14
# image. If a dependency ever needs compiling, build it in a separate stage.
# For locale
locales tzdata \
# For healthcheck
curl \
# For entrypoint
dumb-init \
# For debug
zip iputils-ping \
# For user switching
gosu \
# --- Tor support (activated via USING_TOR=true) ---
tor \
supervisor \
iptables \
# --- WireGuard support (activated via USING_WIREGUARD=true) ---
wireguard-tools \
iproute2 \
procps \
ca-certificates && \
# Configure iptables alternatives for tor.sh compatibility
update-alternatives --set iptables /usr/sbin/iptables-legacy && \
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy && \
# Cleanup APT cache *after* all installs in this layer
apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* && \
# Default to UTC timezone but will be overridden by the entrypoint script
ln -snf /usr/share/zoneinfo/UTC /etc/localtime && echo UTC > /etc/timezone && \
# Configure locale
sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && \
locale-gen en_US.UTF-8 && \
echo "LC_ALL=en_US.UTF-8" >> /etc/environment && \
echo "LANG=en_US.UTF-8" > /etc/locale.conf
# Create a fixed runtime user/group so hardened Docker/Kubernetes deployments
# can start the container directly as a non-root user with a passwd entry.
RUN groupadd -g 1000 shelfmark && \
useradd -u 1000 -g shelfmark -d /home/shelfmark -s /usr/sbin/nologin shelfmark && \
mkdir -p /home/shelfmark && \
chown 1000:1000 /home/shelfmark
# Set working directory
WORKDIR /app
# Install core Python dependencies first for better layer caching
COPY pyproject.toml uv.lock ./
RUN --mount=type=cache,target=/root/.cache/uv \
--mount=from=uv,source=/uv,target=/usr/local/bin/uv \
uv sync --locked --no-default-groups
# Runtime dependencies are installed into /app/.venv during the build. Remove the
# base image's system pip so stale installer CVEs do not ship in the final image.
RUN rm -rf \
/usr/local/bin/pip \
/usr/local/bin/pip3 \
/usr/local/bin/pip3.* \
/usr/local/lib/python*/site-packages/pip \
/usr/local/lib/python*/site-packages/pip-*.dist-info
# The application code is deliberately NOT copied here. `base` is shared by the
# final stages, so a COPY of the source at this point invalidates every layer
# built on top of it -- the Chromium install, the browser dependency sync and
# the SeleniumBase driver download -- on any source change. Each stage copies
# the source as its last step instead, so a code-only rebuild rewrites one small
# layer and every expensive layer is reused. `[tool.uv] package = false` is what
# makes this safe: no `uv sync` needs the project source.
# Expose the application port
EXPOSE ${FLASK_PORT}
# Add healthcheck for container status
# Uses /api/health which doesn't require authentication.
# curl needs -f so an HTTP error status fails the probe instead of passing it:
# plain `curl -s` exits 0 on a 500, which reported a broken app as healthy.
# timeout stays well under interval so a hung probe cannot occupy a whole cycle.
# --start-interval matches the daemon default (5s), made explicit so startup
# probing does not depend on that default staying put.
HEALTHCHECK --interval=30s --timeout=10s --start-period=90s --start-interval=5s --retries=3 \
CMD curl -fsS http://localhost:${FLASK_PORT}/api/health > /dev/null || exit 1
# Use dumb-init as the entrypoint to handle signals properly
ENTRYPOINT ["/usr/bin/dumb-init", "--"]
FROM base AS shelfmark
# --- Chromium (PINNED to 156.0.8078.73) ---
ARG CHROMIUM_VERSION=156.0.8078.73-1~deb13u1
ARG CHROMIUM_SNAPSHOT=20261029T020000Z
RUN echo "deb [check-valid-until=no] https://snapshot.debian.org/archive/debian-security/${CHROMIUM_SNAPSHOT}/ trixie-security main" \
> /etc/apt/sources.list.d/chromium-pin-snapshot.list && \
apt-get update -o Acquire::Retries=5 && \
apt-get install -y --no-install-recommends -o Acquire::Retries=5 \
# For dumb display
xvfb \
chromium=${CHROMIUM_VERSION} \
chromium-common=${CHROMIUM_VERSION} \
# For tkinter (pyautogui)
python3-tk \
# For RAR extraction
unrar-free && \
# Keep apt from "upgrading" chromium past the pin inside derived images
printf 'Package: chromium chromium-common\nPin: version %s\nPin-Priority: 1001\n' "${CHROMIUM_VERSION}" \
> /etc/apt/preferences.d/chromium-pin && \
rm /etc/apt/sources.list.d/chromium-pin-snapshot.list && \
# Create symlink so rarfile library can find unrar
ln -sf /usr/bin/unrar-free /usr/bin/unrar && \
# Cleanup APT cache
apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
# Install the browser automation stack used by the full image
RUN --mount=type=cache,target=/root/.cache/uv \
--mount=from=uv,source=/uv,target=/usr/local/bin/uv \
uv sync --locked --no-default-groups --extra browser && \
# Selenium ships prebuilt selenium-manager binaries (about 22MB) for every
# platform. They only run when selenium has to locate a webdriver, and the
# bypasser drives the browser through SeleniumBase's Pure-CDP driver, which
# needs none. Removed in this layer so the image does not carry them.
find /app/.venv/lib -name 'selenium-manager*' -delete
# Deterministically resolve the Xlib namespace collision.
# pyautogui/mouseinfo pull the stale `python3-xlib` (0.15, 2014), while the
# `--extra browser` set pulls `python-xlib` (0.33). Both packages install into
# the same top-level `Xlib/` namespace, so whichever lands last wins. When the
# 2014 build wins, `Xlib.X` is missing `FamilyServerInterpreted`, which the
# SeleniumBase Pure-CDP driver requires at browser startup -> every bypass fails
# with "module 'Xlib.X' has no attribute 'FamilyServerInterpreted'" and no
# Cloudflare/DDoS-Guard protected download can complete. Drop the stale package
# and force python-xlib 0.33 to own the namespace. pyautogui runs fine against
# 0.33 (superset API).
RUN --mount=type=cache,target=/root/.cache/uv \
--mount=from=uv,source=/uv,target=/usr/local/bin/uv \
uv pip uninstall --python /app/.venv/bin/python python3-xlib && \
uv pip install --python /app/.venv/bin/python --reinstall python-xlib==0.33 && \
/app/.venv/bin/python -c "import Xlib.X; assert hasattr(Xlib.X, 'FamilyServerInterpreted'), 'Xlib.X.FamilyServerInterpreted missing after fix'; print('Xlib namespace OK:', Xlib.__version__)"
# The venv's own pip goes too. uv seeded a copy into /app/.venv and nothing
# installs at runtime. setuptools deliberately STAYS: several deps still import
# pkg_resources on the hot path.
RUN rm -rf /app/.venv/bin/pip /app/.venv/bin/pip3 /app/.venv/bin/pip3.* \
/app/.venv/lib/python*/site-packages/pip \
/app/.venv/lib/python*/site-packages/pip-*.dist-info
# Keep SeleniumBase's bundled driver cache writable for the fixed non-root user.
RUN SELENIUMBASE_DRIVERS_DIR=$(/app/.venv/bin/python -c "import pathlib, seleniumbase; print(pathlib.Path(seleniumbase.__file__).resolve().parent / 'drivers')") && \
chown -R 1000:1000 "${SELENIUMBASE_DRIVERS_DIR}" && \
chmod -R u+rwX,go+rX "${SELENIUMBASE_DRIVERS_DIR}" && \
if [ -f "${SELENIUMBASE_DRIVERS_DIR}/uc_driver" ]; then chmod +x "${SELENIUMBASE_DRIVERS_DIR}/uc_driver"; fi
# Grant read/execute permissions to others
RUN chmod -R o+rx /usr/bin/chromium
# --- Application code: last, so every expensive layer above stays cached ---
COPY . .
COPY --from=frontend-builder /frontend/dist /app/frontend-dist
# The frontend SOURCE cannot be excluded via .dockerignore — that file applies
# to every stage and frontend-builder needs src/. Only the built dist (copied
# above) is served at runtime, so drop the source here.
RUN rm -rf /app/src
# Image-owned runtime paths for the fixed non-root user. Root/PUID mode still
# re-homes ownership at startup when needed.
RUN mkdir -p \
/config \
/books \
/var/log/shelfmark \
/tmp/shelfmark/seleniumbase/downloaded_files \
/tmp/shelfmark/seleniumbase/archived_files && \
rm -rf /app/downloaded_files /app/archived_files && \
ln -s /tmp/shelfmark/seleniumbase/downloaded_files /app/downloaded_files && \
ln -s /tmp/shelfmark/seleniumbase/archived_files /app/archived_files && \
chown -R 1000:1000 /config /books /home/shelfmark /tmp/shelfmark /var/log/shelfmark && \
chmod -R a+rX /app && \
chmod +x /app/entrypoint.sh /app/tor.sh /app/wireguard.sh /app/genDebug.sh
# Default command to run the application entrypoint script
# Version stamp last. These carry the commit sha, so they change on every
# build and everything below them rebuilds. Kept here, the dependency and
# browser layers above stay valid and a pull only fetches what changed.
ARG BUILD_VERSION
ENV BUILD_VERSION=${BUILD_VERSION}
ARG RELEASE_VERSION
ENV RELEASE_VERSION=${RELEASE_VERSION}
CMD ["/app/entrypoint.sh"]
FROM base AS shelfmark-lite
ENV USING_EXTERNAL_BYPASSER=true
# --- Application code: last, so every expensive layer above stays cached ---
COPY . .
COPY --from=frontend-builder /frontend/dist /app/frontend-dist
# The frontend SOURCE cannot be excluded via .dockerignore — that file applies
# to every stage and frontend-builder needs src/. Only the built dist (copied
# above) is served at runtime, so drop the source here.
RUN rm -rf /app/src
# Image-owned runtime paths for the fixed non-root user. Root/PUID mode still
# re-homes ownership at startup when needed.
RUN mkdir -p \
/config \
/books \
/var/log/shelfmark \
/tmp/shelfmark/seleniumbase/downloaded_files \
/tmp/shelfmark/seleniumbase/archived_files && \
rm -rf /app/downloaded_files /app/archived_files && \
ln -s /tmp/shelfmark/seleniumbase/downloaded_files /app/downloaded_files && \
ln -s /tmp/shelfmark/seleniumbase/archived_files /app/archived_files && \
chown -R 1000:1000 /config /books /home/shelfmark /tmp/shelfmark /var/log/shelfmark && \
chmod -R a+rX /app && \
chmod +x /app/entrypoint.sh /app/tor.sh /app/wireguard.sh /app/genDebug.sh
# Version stamp last. These carry the commit sha, so they change on every
# build and everything below them rebuilds. Kept here, the dependency and
# browser layers above stay valid and a pull only fetches what changed.
ARG BUILD_VERSION
ENV BUILD_VERSION=${BUILD_VERSION}
ARG RELEASE_VERSION
ENV RELEASE_VERSION=${RELEASE_VERSION}
CMD ["/app/entrypoint.sh"]