mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-11-10 12:28:27 +00:00
splitsec2
c6685feba5
ffmpeg is in the default image for one thing, the Debug mode screen recording (`_start_ffmpeg_recording`). Nothing else calls it, SeleniumBase included, and it brings 93 packages with it. On real builds that's 406 Trivy findings (149 High) and 78 MB of the gzipped amd64 image. This takes it out of the image and installs it when Debug mode needs it. It came out of the base image discussion on #1616 and helps on any base, though on Wolfi the install line would need `apk`. ## What changed - **Dockerfile:** ffmpeg is gone from the full image's install. - **Guard:** `_start_ffmpeg_recording` checks `shutil.which("ffmpeg")` first. If it's missing it logs a warning and returns, and the bypass carries on. Without this, Debug on with no ffmpeg fails every bypass with `FileNotFoundError`. There's a new test for it. - **Entrypoint:** when Debug is on, the internal bypasser is in use and ffmpeg isn't already there, it installs ffmpeg in the background so the app doesn't wait for it. The download and the install each run under a 120 s `timeout` with `DEBIAN_FRONTEND=noninteractive`. Downloading first means a hung mirror can only be killed before dpkg starts, and the next start runs `dpkg --configure -a` in case a restart cut dpkg off. It logs when it starts, succeeds, times out or fails. - **Docs:** a short paragraph in `docs/troubleshooting.md` covering the install and the non-root limit. ## Debug has two sources Debug can come from the `DEBUG` env var or from the Debug Mode setting in the UI, which is saved to `/config/plugins/advanced.json`. The existing block in `entrypoint.sh` only reads the env var, so an install keyed on `$DEBUG` would miss anyone who turned it on in the UI. The new check asks the app instead (`shelfmark.config.env.DEBUG`), which reads both. The UI setting is `requires_restart`, so the restart it already asks for is what brings it through the entrypoint. ## Non-root mode A container started as `1000:1000` can't run `apt-get`, so the entrypoint logs that it can't install ffmpeg, the guard warns on each bypass, and there's no recording. Everything else works. The workaround for a debugging session is to start as root with `PUID`/`PGID`. ## Startup cost None that I could measure. With Debug on, `/api/health` answers in the same time as main (3-4 s with Debug only in config, 9-10 s from env), and ffmpeg was ready 6-8 s later on the runners. The trade-off is that the first bypass after startup may be unrecorded while the install runs. It logs the warning and passes. I saw both outcomes on a bypass started right after health. From reading the code: the install lives in the container, so recreating the container installs it again. It also runs after `tor.sh` and `wireguard.sh`, so apt goes through whichever egress is active. ## Testing Real Docker builds on GitHub Actions on my fork, amd64 and arm64, with the same result on both for every case below. | Case | Result | |---|---| | Debug off | No install, `/api/health` in 3-4 s, same as main | | Debug only in config | Health in 3-4 s, ffmpeg ready 6-8 s later, real mp4 after a bypass | | Debug from env | Health in 9-10 s (same as main), real mp4 after a bypass | | Started as root, app dropped to uid 1000 | Smoke and bypass pass | | Started with `--user 1000`, Debug on | Logs both messages, bypass passes, no mp4 | | apt blocked (closed proxy) | Logs `Could not install ffmpeg`, bypass passes, no mp4 | | apt's proxy on a dropped address, so no TCP reply at all | Health in 9-10 s, a bypass during the install passes with the warning and no mp4. apt gives up by itself in under a minute. Nothing left running, `dpkg --audit` clean | | DNS server on a dropped address | Health in 9-10 s, `Timed out installing ffmpeg after 120s`. Nothing left running, `dpkg --audit` clean | | `--network none`, Debug on | No hang, health 200 from inside the container | | Container restarted while dpkg was running | Next start finishes the install, `dpkg --audit` clean, real mp4 | | Fresh named volume, Debug only in config | Recording made, uid 1000 can write | | Existing uid-1000 volume, Debug only in config | Same, nothing root-owned afterwards | | `find /config /var/log/shelfmark -user root` | 0 files, same as main | | `find /app -user root` | 245, same count as main | | `tests/e2e/platform/run-e2e.sh` with `dns-blocked` and `baseline` | Pass | The bypass checks ran against example.com. A real Cloudflare challenge was served by a demo page in the same runs, and no build solved it, including main, so that part doesn't say anything about this change. Locally: 5494 Python tests pass, and `ruff`, `basedpyright`, `vulture` and `shellcheck` are clean. The full results are in [the base image report](https://github.com/splitsec2/shelfmark/blob/7afcecfb76727724a71561cbc078f091569b58dc/docs/research/2026-11-01-base-image-report.md).
92 lines
3.1 KiB
Python
92 lines
3.1 KiB
Python
"""The Debug mode ffmpeg install: what it runs, what it logs, and what it can't block."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
import shutil
|
|
import subprocess
|
|
from pathlib import Path
|
|
|
|
ENTRYPOINT_PATH = Path(__file__).resolve().parents[2] / "entrypoint.sh"
|
|
BASH_PATH = shutil.which("bash") or "/bin/bash"
|
|
|
|
|
|
def _stub(bin_dir: Path, name: str, body: str) -> None:
|
|
path = bin_dir / name
|
|
path.write_text(f"#!/bin/sh\n{body}\n")
|
|
path.chmod(0o755)
|
|
|
|
|
|
def _install_ffmpeg_source() -> str:
|
|
text = ENTRYPOINT_PATH.read_text()
|
|
start = text.index("install_ffmpeg() {")
|
|
return text[start : text.index("\n}\n", start) + 3]
|
|
|
|
|
|
def _run(
|
|
tmp_path: Path, *, apt_body: str, ffmpeg_works: bool = True
|
|
) -> tuple[subprocess.CompletedProcess[str], list[str]]:
|
|
bin_dir = tmp_path / "bin"
|
|
bin_dir.mkdir()
|
|
calls = tmp_path / "apt-calls.txt"
|
|
_stub(
|
|
bin_dir,
|
|
"apt-get",
|
|
f'echo "frontend=$DEBIAN_FRONTEND $*" >> "{calls}"\n{apt_body}',
|
|
)
|
|
_stub(bin_dir, "dpkg", "exit 0")
|
|
_stub(bin_dir, "ffmpeg", "exit 0" if ffmpeg_works else "exit 1")
|
|
|
|
env = os.environ.copy()
|
|
env.update({"PATH": f"{bin_dir}:{env.get('PATH', '')}", "FFMPEG_INSTALL_TIMEOUT": "1"})
|
|
result = subprocess.run(
|
|
[BASH_PATH, "-c", f"{_install_ffmpeg_source()}\ninstall_ffmpeg"],
|
|
capture_output=True,
|
|
env=env,
|
|
text=True,
|
|
check=False,
|
|
timeout=30,
|
|
)
|
|
return result, calls.read_text().splitlines() if calls.exists() else []
|
|
|
|
|
|
def test_install_downloads_first_then_installs_without_the_network(tmp_path):
|
|
result, calls = _run(tmp_path, apt_body="exit 0")
|
|
|
|
assert "ffmpeg installed" in result.stdout
|
|
assert [call.split()[1] for call in calls] == ["update", "install", "install"]
|
|
assert "--download-only" in calls[1]
|
|
assert "--no-download" in calls[2]
|
|
assert all(call.startswith("frontend=noninteractive ") for call in calls)
|
|
|
|
|
|
def test_install_gives_up_on_a_mirror_that_never_answers(tmp_path):
|
|
result, calls = _run(tmp_path, apt_body="exec sleep 60")
|
|
|
|
assert "Timed out installing ffmpeg after 1s" in result.stdout
|
|
assert "ffmpeg installed" not in result.stdout
|
|
# Killed during the download, so dpkg's install step never ran.
|
|
assert not any("--no-download" in call for call in calls)
|
|
|
|
|
|
def test_install_reports_a_failure(tmp_path):
|
|
result, calls = _run(tmp_path, apt_body="exit 100")
|
|
|
|
assert "Could not install ffmpeg (exit 100)" in result.stdout
|
|
assert not any("--no-download" in call for call in calls)
|
|
|
|
|
|
def test_install_does_not_claim_success_when_ffmpeg_does_not_run(tmp_path):
|
|
result, _ = _run(tmp_path, apt_body="exit 0", ffmpeg_works=False)
|
|
|
|
assert "ffmpeg does not run" in result.stdout
|
|
assert "ffmpeg installed" not in result.stdout
|
|
|
|
|
|
def test_install_starts_after_the_log_pipe_is_released():
|
|
"""stop_file_logging waits for the log pipe to close, so the job must not hold it."""
|
|
text = ENTRYPOINT_PATH.read_text()
|
|
|
|
assert text.rindex("stop_file_logging\n") < text.index('if [ "$INSTALL_FFMPEG" = "true" ]')
|
|
assert text.index('if [ "$INSTALL_FFMPEG" = "true" ]') < text.rindex("exec_as_target_user env")
|