mirror of
https://github.com/calibrain/shelfmark.git
synced 2026-09-24 19:40:26 +01:00
## Backport bug fixes from `NemesisHubris/litfinder` Forwards a curated set of bug fixes from [NemesisHubris/litfinder](https://github.com/NemesisHubris/litfinder) — a community fork of this project — that address open issues here. All commits preserve original authorship via `git cherry-pick`; this PR is a backport rather than original work. Each fix has been reviewed locally, lint/format-cleaned to match this repo's existing ruff config, and verified with the test suite. Rebrand strings, license switches, and features have been deliberately excluded. ### Upstream issues addressed - **#999** — Mirror URLs with query params no longer break search requests (strip query string/fragment in `normalize_http_url`) - **#956** — Apprise notifications now respect the configured proxy (proxy env vars injected before dispatch) - **#1025** — rTorrent: separate `RTORRENT_AUDIOBOOK_LABEL` setting, falls back to book label if unset - **#1010** — Stop button in Activity no longer makes the panel disappear (snapshot refresh on cancel) - **#1021** — Anna's Archive slow-download countdown now caps retries instead of looping forever - **#1040** — Empty destination directory cleaned up when write probe fails - **PR #1031** — Language detection from Anna's Archive distant path when listing metadata is missing ### Additional fixes (no open issue but clear bugs) - **fix: Python 2 `except` syntax across 27 files** — `except X, Y:` is a SyntaxError in Python 3 and prevents affected modules from importing at runtime. Mechanical sweep to `except (X, Y):`. - **fix(abb): info hash validation with magnet fallback** — adds SHA-1/SHA-256 hex validation on extracted info hashes; falls back to scanning the full page for a magnet link (e.g. posted in comments) when the table value is malformed. Also extends the exact-phrase fallback to manual queries and defaults the ABB listing language to `en` when missing, preventing valid results from being hidden by the language filter. Includes a small test-fixture fix (`test(abb): use valid hex info hashes in scraper test fixtures`) since the existing fixtures used non-hex placeholders that the new validation correctly rejects. - **fix: Anna's Archive title parser** — handles nested edition spans and filters `lgli` catalog descriptor entries (e.g. "Book/Online Audio") that were polluting search results. ### Deliberately not included - LitFinder rebranding (UI strings, Apprise app ID, logo). The `fix: three upstream bugs` commit (#999/#956/#1025) was cherry-picked with Apprise app-id, description, and logo-URL strings reverted from "LitFinder" back to "Shelfmark"; noted in the commit body. - Features from the LitFinder fork (multi-variant title search, multi-book flat-folder grouping, fuzzy text matching, "Leave in Place" output handler, admin display name, custom-source plugin system). These are larger behavior changes that each warrant their own focused review — happy to send any of them separately if of interest. - LitFinder-specific test environment and CI infrastructure. ### Verification - Backend: **1879 passed**, 96 skipped (1 preexisting failure on `seleniumbase`-dependent test in local venv; runs fine in the standard Docker image with the `browser` extra) - Lint, format, dead-code: all clean against this repo's existing ruff/vulture config - One follow-up cleanup commit (`style: ruff lint and format fixes for ported commits`) brings the cherry-picked code into compliance with this repo's ruff settings — no behavior changes there ### Etiquette / credit Per-commit authorship preserved by cherry-pick. The only edits to the original commits are: - `fix: three upstream bugs` — Apprise rebrand strings reverted to "Shelfmark" (noted in commit body, original author retained as `Co-Authored-By` via cherry-pick) - One follow-up `style:` commit for ruff config alignment Big thanks to [@NemesisHubris](https://github.com/NemesisHubris) for the original work in LitFinder; this PR exists to make sure these fixes reach Shelfmark's wider user base. Happy to revise scope, split into smaller PRs, or split off the Py2 cleanup separately if that's preferable. --------- Co-authored-by: NemesisHubris <155838970+NemesisHubris@users.noreply.github.com> Co-authored-by: CaliBrain <calibrain@l4n.xyz>
345 lines
9.7 KiB
Bash
345 lines
9.7 KiB
Bash
#!/bin/bash
|
|
|
|
is_truthy() {
|
|
case "${1,,}" in
|
|
true|yes|1|y) return 0 ;;
|
|
*) return 1 ;;
|
|
esac
|
|
}
|
|
|
|
ENABLE_LOGGING_VALUE="${ENABLE_LOGGING:-true}"
|
|
|
|
LOG_DIR=${LOG_ROOT:-/var/log/}/shelfmark
|
|
LOG_FILE="${LOG_DIR}/shelfmark_tor.log"
|
|
|
|
if is_truthy "$ENABLE_LOGGING_VALUE"; then
|
|
mkdir -p "$LOG_DIR"
|
|
|
|
exec 3>&1 4>&2
|
|
exec > >(tee -a "$LOG_FILE") 2>&1
|
|
fi
|
|
echo "Starting tor script"
|
|
if is_truthy "$ENABLE_LOGGING_VALUE"; then
|
|
echo "Log file: $LOG_FILE"
|
|
else
|
|
echo "File logging disabled (ENABLE_LOGGING=$ENABLE_LOGGING_VALUE)"
|
|
fi
|
|
|
|
set +x
|
|
set -e
|
|
|
|
# Check if EXT_BYPASSER_URL is defined
|
|
if [ -n "$EXT_BYPASSER_URL" ]; then
|
|
echo "Extracting hostname and ip from bypasser into /etc/hosts"
|
|
|
|
# Extract hostname
|
|
hostname=$(echo "$EXT_BYPASSER_URL" | cut -d'/' -f3 | cut -d':' -f1)
|
|
|
|
# Resolve to IP (using current DNS before switching to TOR)
|
|
ip=$(getent hosts "$hostname" 2>/dev/null | awk '{print $1}')
|
|
|
|
# If getent fails, try dig
|
|
if [ -z "$ip" ]; then
|
|
ip=$(dig +short "$hostname" 2>/dev/null | head -n1)
|
|
fi
|
|
|
|
# Only proceed if we got an IP and hostname is not already an IP
|
|
if [ -n "$ip" ] && [ "$ip" != "$hostname" ]; then
|
|
# Add to /etc/hosts (remove existing entry first to avoid duplicates)
|
|
sudo sed -i "/[[:space:]]$hostname$/d" /etc/hosts
|
|
echo "$ip $hostname" | sudo tee -a /etc/hosts > /dev/null
|
|
echo "Added to /etc/hosts: $ip $hostname"
|
|
else
|
|
echo "Skipping: $hostname is already an IP or could not be resolved"
|
|
fi
|
|
else
|
|
echo "EXT_BYPASSER_URL not defined, skipping /etc/hosts update"
|
|
fi
|
|
|
|
echo "[*] Running tor script..."
|
|
|
|
echo "Build version: $BUILD_VERSION"
|
|
echo "Release version: $RELEASE_VERSION"
|
|
|
|
echo "[*] Installing Tor and dependencies..."
|
|
echo "[*] Writing Tor transparent proxy config..."
|
|
|
|
cat <<EOF > /etc/tor/torrc
|
|
VirtualAddrNetworkIPv4 10.192.0.0/10
|
|
AutomapHostsOnResolve 1
|
|
TransPort 9040
|
|
DNSPort 53
|
|
Log notice file /var/log/tor/notices.log
|
|
|
|
# Circuit management to prevent stale circuits after inactivity
|
|
MaxCircuitDirtiness 600
|
|
NewCircuitPeriod 30
|
|
CircuitBuildTimeout 60
|
|
LearnCircuitBuildTimeout 0
|
|
|
|
# Keep circuits alive
|
|
KeepalivePeriod 60
|
|
CircuitStreamTimeout 60
|
|
|
|
# Prevent connection timeouts
|
|
SocksTimeout 120
|
|
EOF
|
|
|
|
echo "[*] Setting up DNS..."
|
|
cat <<EOF > /etc/resolv.conf
|
|
nameserver 127.0.0.1
|
|
EOF
|
|
|
|
echo "[*] Starting Tor..."
|
|
echo "[*] Configuring Supervisor..."
|
|
mkdir -p /var/log/supervisor
|
|
cat <<EOF > /etc/supervisor/supervisord.conf
|
|
[supervisord]
|
|
nodaemon=false
|
|
logfile=/var/log/supervisor/supervisord.log
|
|
pidfile=/var/run/supervisord.pid
|
|
user=root
|
|
|
|
[unix_http_server]
|
|
file=/var/run/supervisor.sock ; (the path to the socket file)
|
|
|
|
[rpcinterface:supervisor]
|
|
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
|
|
|
|
[supervisorctl]
|
|
serverurl=unix:///var/run/supervisor.sock ; use a unix:// URL for a unix socket
|
|
|
|
[program:tor]
|
|
command=/usr/bin/tor -f /etc/tor/torrc
|
|
autostart=true
|
|
autorestart=true
|
|
startretries=100
|
|
stdout_logfile=/var/log/supervisor/tor.log
|
|
stderr_logfile=/var/log/supervisor/tor.err.log
|
|
|
|
[program:tor-healthcheck]
|
|
command=/app/tor_healthcheck.sh
|
|
autostart=true
|
|
autorestart=true
|
|
stdout_logfile=/var/log/supervisor/healthcheck.log
|
|
stderr_logfile=/var/log/supervisor/healthcheck.err.log
|
|
EOF
|
|
|
|
# Create healthcheck script
|
|
cat <<'HC' > /app/tor_healthcheck.sh
|
|
#!/bin/bash
|
|
|
|
# Function to dynamically wait for Tor bootstrap
|
|
wait_for_tor() {
|
|
echo "$(date): Waiting for Tor to finish bootstrapping..."
|
|
|
|
> /var/log/tor/notices.log 2>/dev/null || true
|
|
|
|
sleep 10
|
|
|
|
TIMEOUT=300
|
|
ELAPSED=0
|
|
while [ $ELAPSED -lt $TIMEOUT ]; do
|
|
if grep -q "Bootstrapped 100%" /var/log/tor/notices.log 2>/dev/null; then
|
|
echo "$(date): Tor bootstrap complete."
|
|
return 0
|
|
fi
|
|
sleep 5
|
|
ELAPSED=$((ELAPSED + 5))
|
|
# Show progress
|
|
CURRENT=$(tail -n 1 /var/log/tor/notices.log 2>/dev/null | grep -oP 'Bootstrapped \d+%' || echo "waiting...")
|
|
echo "$(date): Bootstrap progress: $CURRENT ($ELAPSED/${TIMEOUT}s)"
|
|
done
|
|
|
|
echo "$(date): WARNING - Tor bootstrap timed out after ${TIMEOUT}s"
|
|
return 1
|
|
}
|
|
|
|
tor_is_healthy() {
|
|
supervisorctl status tor | grep -q "RUNNING" &&
|
|
grep -q "Bootstrapped 100%" /var/log/tor/notices.log 2>/dev/null
|
|
}
|
|
|
|
FAIL_COUNT=0
|
|
while true; do
|
|
if tor_is_healthy; then
|
|
FAIL_COUNT=0
|
|
else
|
|
FAIL_COUNT=$((FAIL_COUNT+1))
|
|
echo "$(date): Healthcheck failed (Count: $FAIL_COUNT)"
|
|
fi
|
|
|
|
# If failed 3 times in a row, restart Tor
|
|
if [ "$FAIL_COUNT" -ge 3 ]; then
|
|
echo "$(date): restart trigger - Restarting Tor..."
|
|
supervisorctl restart tor
|
|
FAIL_COUNT=0
|
|
|
|
# Wait for it to come back using the dynamic check
|
|
wait_for_tor
|
|
fi
|
|
|
|
sleep 30
|
|
done
|
|
HC
|
|
chmod +x /app/tor_healthcheck.sh
|
|
|
|
echo "[*] Starting Tor via Supervisor..."
|
|
/usr/bin/supervisord -c /etc/supervisor/supervisord.conf
|
|
|
|
# Wait a bit to ensure Tor has bootstrapped
|
|
echo "[*] Waiting for Tor to finish bootstrapping... (up to 5 minutes)"
|
|
BOOTSTRAP_TIMEOUT=300
|
|
BOOTSTRAP_START=$(date +%s)
|
|
while true; do
|
|
if grep -q "Bootstrapped 100%" /var/log/tor/notices.log 2>/dev/null; then
|
|
echo ""
|
|
echo "[✓] Tor bootstrap complete."
|
|
break
|
|
fi
|
|
|
|
ELAPSED=$(($(date +%s) - BOOTSTRAP_START))
|
|
if [ $ELAPSED -ge $BOOTSTRAP_TIMEOUT ]; then
|
|
echo ""
|
|
echo "[✗] Tor bootstrap timed out after ${BOOTSTRAP_TIMEOUT}s"
|
|
exit 1
|
|
fi
|
|
|
|
CURRENT_LOG=$(tail -n 1 /var/log/tor/notices.log 2>/dev/null || true)
|
|
printf "\r\033[K[%ds] %s" "$ELAPSED" "$CURRENT_LOG"
|
|
sleep 1
|
|
done
|
|
echo "[✓] Tor is ready."
|
|
|
|
|
|
echo "[*] Setting up iptables rules..."
|
|
|
|
iptables -F
|
|
iptables -t nat -F
|
|
TOR_UID=$(id -u debian-tor)
|
|
|
|
# Allow loopback
|
|
iptables -t nat -A OUTPUT -o lo -j RETURN
|
|
|
|
# Allow Tor itself to reach the network
|
|
iptables -t nat -A OUTPUT -m owner --uid-owner "$TOR_UID" -j RETURN
|
|
|
|
# For UDP DNS queries
|
|
iptables -t nat -A OUTPUT -p udp --dport 53 ! -d 127.0.0.1 -j DNAT --to-destination 127.0.0.1:53
|
|
|
|
# For TCP DNS queries (some DNS queries may use TCP)
|
|
iptables -t nat -A OUTPUT -p tcp --dport 53 ! -d 127.0.0.1 -j DNAT --to-destination 127.0.0.1:53
|
|
|
|
# Bypass Tor for local/private networks
|
|
iptables -t nat -A OUTPUT -d 127.0.0.0/8 -j RETURN
|
|
iptables -t nat -A OUTPUT -d 10.0.0.0/8 -j RETURN
|
|
iptables -t nat -A OUTPUT -d 172.16.0.0/12 -j RETURN
|
|
iptables -t nat -A OUTPUT -d 192.168.0.0/16 -j RETURN
|
|
|
|
# Redirect all TCP to Tor's TransPort
|
|
iptables -t nat -A OUTPUT -p tcp --syn -j REDIRECT --to-ports 9040
|
|
|
|
echo "[✓] Transparent Tor routing enabled."
|
|
|
|
sleep 5
|
|
# Check if outgoing IP is using Tor
|
|
echo "[*] Verifying Tor connectivity..."
|
|
RESULT=$(curl -s https://check.torproject.org/api/ip)
|
|
echo "RESULT: $RESULT"
|
|
IS_TOR=$(echo "$RESULT" | grep -oP '"IsTor":\s*\K(true|false)')
|
|
IP=$(echo "$RESULT" | grep -oP '"IP":\s*"\K[^"]+')
|
|
if [[ "$IS_TOR" == "true" ]]; then
|
|
echo "[✓] Success! Traffic is routed through Tor. Current IP: $IP"
|
|
else
|
|
echo "[✗] Warning: Traffic is NOT using Tor. Current IP: $IP"
|
|
exit 1
|
|
fi
|
|
|
|
# Set correct timezone
|
|
# First check what is the timezone based on the IP
|
|
# Then set the timezone
|
|
|
|
# Get timezone from IP
|
|
sleep 1
|
|
TIMEZONE=$(curl -s https://ipapi.co/timezone) || \
|
|
TIMEZONE=$(curl -s http://ip-api.com/line?fields=timezone) || \
|
|
TIMEZONE=$(curl -s http://worldtimeapi.org/api/ip | grep -oP '"timezone":"\K[^"]+') || \
|
|
TIMEZONE=$(curl -s https://ip2tz.isthe.link/v2 | grep -oP '"timezone": *"\K[^"]+') || \
|
|
true
|
|
|
|
# If TIMEZONE is not set, use the default timezone
|
|
echo "[*] Current Timezone : $(date +%Z). IP Timezone: $TIMEZONE"
|
|
|
|
# Set timezone in Docker-compatible way
|
|
if [ -f "/usr/share/zoneinfo/$TIMEZONE" ]; then
|
|
# Remove existing symlink if it exists
|
|
rm -f /etc/localtime
|
|
# Create new symlink
|
|
ln -sf /usr/share/zoneinfo/$TIMEZONE /etc/localtime
|
|
# Set timezone file
|
|
echo "$TIMEZONE" > /etc/timezone
|
|
# Set TZ environment variable
|
|
export TZ=$TIMEZONE
|
|
# Verify the change
|
|
echo "[✓] Timezone set to $TIMEZONE"
|
|
echo "[*] Current time: $(date)"
|
|
echo "[*] Timezone verification: $(date +%Z)"
|
|
else
|
|
echo "[!] Warning: Timezone file not found: $TIMEZONE"
|
|
echo "[*] Available timezones:"
|
|
ls -la /usr/share/zoneinfo/
|
|
echo "[*] Falling back to container's default timezone: $TZ"
|
|
fi
|
|
|
|
# Start a background circuit rotation process
|
|
echo "[*] Starting Tor circuit rotation monitor..."
|
|
rotation_monitor() {
|
|
rotation_count=0
|
|
|
|
# Wait for initial stability
|
|
sleep 120
|
|
|
|
while true; do
|
|
rotation_count=$((rotation_count + 1))
|
|
echo "[*] Circuit rotation #$rotation_count at $(date)"
|
|
|
|
# Test DNS resolution through Tor
|
|
dns_ok=true
|
|
if ! timeout 10 nslookup google.com 127.0.0.1 > /dev/null 2>&1; then
|
|
echo "[!] $(date): DNS resolution slow/failing, rotating circuits..."
|
|
pkill -HUP tor || true
|
|
sleep 10
|
|
dns_ok=false
|
|
fi
|
|
|
|
# Proactively rotate circuits every 5 minutes to keep them fresh
|
|
# Skip if we already rotated for DNS failure this cycle
|
|
if $dns_ok; then
|
|
echo "[*] $(date): Proactive circuit rotation..."
|
|
pkill -HUP tor || true
|
|
fi
|
|
|
|
# Verify Tor is still responsive after rotation
|
|
sleep 5
|
|
if timeout 10 curl -s --max-time 8 https://check.torproject.org/api/ip > /dev/null 2>&1; then
|
|
echo "[✓] $(date): Circuit rotation successful, Tor responsive"
|
|
else
|
|
echo "[!] $(date): Tor unresponsive after rotation - supervisor healthcheck will handle recovery"
|
|
fi
|
|
|
|
sleep 300
|
|
done
|
|
}
|
|
|
|
if is_truthy "$ENABLE_LOGGING_VALUE"; then
|
|
rotation_monitor >> "$LOG_FILE" 2>&1 &
|
|
else
|
|
rotation_monitor &
|
|
fi
|
|
|
|
ROTATION_PID=$!
|
|
echo "[✓] Tor circuit rotation monitor started in background (PID: $ROTATION_PID)"
|
|
|
|
# Run the entrypoint script
|
|
echo "[*] End of tor script"
|