mirror of
https://github.com/ThePhaseless/Byparr.git
synced 2026-11-10 11:08:20 +00:00
fix: fetch GeoIP data once at start instead of on every browser launch
invisible-core 49.46+ looks up the egress IP and checks GitHub for a newer GeoIP database on every browser launch, so a network that blocks those hosts delayed every request by the lookup timeouts. Memoize both lookups (success for the process lifetime, failures retried after a backoff, proxied egress lookups uncached), warm them up in the FastAPI lifespan before the port opens, and gate get_browser() on the same warm-up. Drop the unused --init mode.
This commit is contained in:
1 parent
3e8c09ea39
commit
406ad428c4
6 files changed
+414
-30
No files matched your search
@@ -18,6 +18,7 @@
|
||||
- Models use Pydantic v2 with camelCase aliasing for responses (see src/models.py).
|
||||
- LinkResponse.invalid() is the standard error response shape; keep fields consistent with FlareSolverr style.
|
||||
- get_camoufox() constructs AsyncCamoufox with addons and optional proxy config from env vars.
|
||||
- src/utils.py memoizes invisible-core's egress-IP and GeoIP lookups by replacing the private module attributes invisible_core._geo.discover_egress_ip and invisible_core.download.ensure_geoip_mmdb with RememberedLookup wrappers; proxied egress lookups pass straight through. warm_up() runs them once at start from the FastAPI lifespan, and get_browser() awaits it first so paths without the lifespan (TestClient without `with`) are gated too.
|
||||
|
||||
## Config and environment
|
||||
|
||||
@@ -27,7 +28,6 @@
|
||||
## Developer workflows
|
||||
|
||||
- Local run: uv sync && uv run main.py
|
||||
- Init mode: uv run main.py --init (pre-warms health check via browser setup)
|
||||
- Tests: uv sync --group test && uv run pytest --retries 5
|
||||
- Docker troubleshooting: docker build --target test .
|
||||
|
||||
|
||||
@@ -30,9 +30,15 @@ Valid tags are maintained in the [IANA Language Subtag Registry](https://www.ian
|
||||
|
||||
Set `BROWSER_TIMEZONE` to an [IANA timezone name](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) like `Europe/Berlin` or `Asia/Shanghai` to fix the browser's timezone. An explicit value is used as is, also behind a proxy.
|
||||
|
||||
When either `BROWSER_TIMEZONE` or `BROWSER_LOCALE` is unset, Byparr looks up its public IP through `api.ipify.org`, `icanhazip.com` or `checkip.amazonaws.com` and maps it to a timezone and language with a GeoIP database downloaded from GitHub. Without a proxy this happens once, on the first request, and the result is reused until Byparr restarts (restart it if its public IP moves to another country). If that lookup fails, the host timezone (`TZ`, otherwise the system's, otherwise UTC) and `en-US` are used, and the lookup is retried after 10 minutes. With a proxy (`PROXY_SERVER` or the `X-Proxy-Server` header) it runs for every request, through the proxy.
|
||||
When Byparr starts, it downloads the GeoIP database from GitHub before it accepts requests. The image does not include the database, so every new container downloads it once. Without `PROXY_SERVER`, Byparr also looks up its public IP through `api.ipify.org`, `icanhazip.com` or `checkip.amazonaws.com` at the same time. When `BROWSER_TIMEZONE` or `BROWSER_LOCALE` is unset, the database maps that IP to a timezone and language. Byparr keeps both results until it restarts, so restart it if its public IP moves to another country or to get a newer database.
|
||||
|
||||
If those hosts are slow or blocked on your network, set both `BROWSER_TIMEZONE` and `BROWSER_LOCALE`. The public IP is still looked up, by Byparr and again each time the browser starts, because the browser uses it for WebRTC and geolocation, but the values you set no longer depend on it: a failed lookup keeps them. A blocked host can still delay each browser start by its connection timeout.
|
||||
This usually adds a few seconds to start-up, and the port stays closed until it is done. If GitHub is unreachable it can take about a minute, and a stalled download a few minutes. A download that keeps going but crawls is capped by `INVISIBLE_DOWNLOAD_DEADLINE`, in seconds per attempt (default 1800, up to 3 attempts). Because an open port means Byparr is ready, a TCP readiness probe on `PORT` (or an HTTP probe on `/docs`) keeps traffic away from a container that is still starting.
|
||||
|
||||
If the lookup or the download fails, Byparr starts anyway and logs a warning. Browsers then get the host timezone (`TZ`, otherwise the system's, otherwise UTC) and `en-US`. Byparr tries the public IP again on the first request at least 10 minutes after the failure. The database is retried sooner, on the first request at least 30 seconds after; each failure in a row doubles that wait, up to 10 minutes.
|
||||
|
||||
With a proxy (`PROXY_SERVER` or the `X-Proxy-Server` header), Byparr looks up the public IP through the proxy for every request, and all requests share the one database. If the database download failed, proxied requests fail until a retry succeeds, unless `BROWSER_TIMEZONE` is set.
|
||||
|
||||
If those hosts are slow or blocked on your network, set both `BROWSER_TIMEZONE` and `BROWSER_LOCALE`. Byparr still looks up the public IP once at start, because the browser uses it for WebRTC and geolocation, but a failed lookup keeps the values you set.
|
||||
|
||||
## Proxy Recommendation
|
||||
|
||||
|
||||
@@ -1,25 +1,33 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import sys
|
||||
from collections.abc import AsyncGenerator
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
import uvicorn
|
||||
from fastapi import FastAPI
|
||||
from fastapi.middleware.gzip import GZipMiddleware
|
||||
|
||||
from src.consts import HOST, LOG_LEVEL, PORT, VERSION
|
||||
from src.endpoints import health_check, router
|
||||
from src.endpoints import router
|
||||
from src.middlewares import LogRequest
|
||||
from src.owui import router as owui_router
|
||||
from src.utils import get_browser, logger
|
||||
from src.utils import logger, warm_up
|
||||
|
||||
LOG_LEVEL_NAME = logging.getLevelName(LOG_LEVEL).lower()
|
||||
|
||||
logger.info("Using version %s", VERSION)
|
||||
logger.info("Log level set to %s", LOG_LEVEL_NAME)
|
||||
|
||||
app = FastAPI(debug=LOG_LEVEL == logging.DEBUG, log_level=LOG_LEVEL)
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(_: FastAPI) -> AsyncGenerator[None]:
|
||||
"""Fetch the GeoIP database before uvicorn opens the port."""
|
||||
await warm_up()
|
||||
yield
|
||||
|
||||
|
||||
app = FastAPI(debug=LOG_LEVEL == logging.DEBUG, log_level=LOG_LEVEL, lifespan=lifespan)
|
||||
app.add_middleware(GZipMiddleware)
|
||||
app.add_middleware(LogRequest)
|
||||
|
||||
@@ -27,19 +35,5 @@ app.include_router(router=router)
|
||||
app.include_router(router=owui_router)
|
||||
|
||||
|
||||
async def init():
|
||||
"""Initialize the application."""
|
||||
async for browser in get_browser():
|
||||
await health_check(browser)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# Check for --init flag to run the app in development mode
|
||||
if "--init" in sys.argv:
|
||||
logger.info("Running initialization script...")
|
||||
loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(loop)
|
||||
loop.run_until_complete(init())
|
||||
logger.info("Initialization complete.")
|
||||
else:
|
||||
uvicorn.run(app, host=HOST, port=PORT, log_level=LOG_LEVEL_NAME)
|
||||
uvicorn.run(app, host=HOST, port=PORT, log_level=LOG_LEVEL_NAME)
|
||||
+109
-1
@@ -1,12 +1,15 @@
|
||||
import asyncio
|
||||
import logging
|
||||
import math
|
||||
import threading
|
||||
import time
|
||||
from collections.abc import AsyncGenerator
|
||||
from collections.abc import AsyncGenerator, Awaitable, Callable
|
||||
from typing import Annotated, NamedTuple, cast
|
||||
|
||||
from fastapi import Header
|
||||
from invisible_core import GeoTimezoneError, SessionGeo, prepare_session_geo
|
||||
from invisible_core import _geo as invisible_geo # pyright: ignore[reportPrivateUsage]
|
||||
from invisible_core import download as invisible_download
|
||||
from invisible_playwright.async_api import InvisiblePlaywright
|
||||
from playwright.async_api import Browser, BrowserContext, Page
|
||||
from pydantic import BaseModel, Field
|
||||
@@ -58,6 +61,70 @@ class BrowserDepClass(NamedTuple):
|
||||
|
||||
|
||||
GEO_RETRY_SECONDS = 600
|
||||
# Short at first: proxied launches with an "auto" timezone fail outright while the
|
||||
# database is missing. Doubles per failure up to GEO_RETRY_SECONDS, because every
|
||||
# launch waits on the lock while a retry runs.
|
||||
GEOIP_RETRY_SECONDS = 30
|
||||
|
||||
|
||||
class RememberedLookup[T]:
|
||||
"""
|
||||
Remember a direct (unproxied) invisible-core lookup for the process.
|
||||
|
||||
invisible-core asks for the egress IP and checks GitHub for the GeoIP
|
||||
database on every browser launch. A success is kept until restart. A
|
||||
failure is kept for ``retry_seconds``, doubled per consecutive failure up
|
||||
to GEO_RETRY_SECONDS; a proxied call always passes through, since each
|
||||
proxy has its own egress.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self, lookup: Callable[..., T], retry_seconds: float = GEO_RETRY_SECONDS
|
||||
) -> None:
|
||||
"""Wrap ``lookup``, kept public so tests can patch it."""
|
||||
self.lookup = lookup
|
||||
self.retry_seconds = retry_seconds
|
||||
self._lock = threading.Lock()
|
||||
self._value: T | None = None
|
||||
self._error: Exception | None = None
|
||||
self._expires_at = 0.0
|
||||
self._failures = 0
|
||||
|
||||
def __call__(self, proxy: dict[str, str] | None = None, **kwargs: object) -> T:
|
||||
"""Call ``lookup`` through the proxy, or return the remembered result."""
|
||||
if proxy:
|
||||
return self.lookup(proxy, **kwargs)
|
||||
with self._lock:
|
||||
if time.monotonic() >= self._expires_at:
|
||||
try:
|
||||
self._value, self._error = self.lookup(**kwargs), None
|
||||
self._expires_at, self._failures = math.inf, 0
|
||||
except Exception as exc:
|
||||
self._value, self._error = None, exc
|
||||
self._expires_at = time.monotonic() + min(
|
||||
self.retry_seconds * 2**self._failures, GEO_RETRY_SECONDS
|
||||
)
|
||||
# Capped so a week-long outage cannot overflow a float window.
|
||||
self._failures = min(self._failures + 1, 16)
|
||||
if self._error is not None:
|
||||
raise self._error.with_traceback(None)
|
||||
return cast("T", self._value)
|
||||
|
||||
def clear(self) -> None:
|
||||
"""Forget the remembered result."""
|
||||
with self._lock:
|
||||
self._value, self._error, self._expires_at = None, None, 0.0
|
||||
self._failures = 0
|
||||
|
||||
|
||||
# shortcut: private invisible-core seams (no public parameter in 55.50); tests/utils_test.py guards them, switch to an upstream option if one appears.
|
||||
discover_egress_ip = RememberedLookup(invisible_geo.discover_egress_ip)
|
||||
ensure_geoip_mmdb = RememberedLookup(
|
||||
invisible_download.ensure_geoip_mmdb, # pyright: ignore[reportPrivateImportUsage]
|
||||
GEOIP_RETRY_SECONDS,
|
||||
)
|
||||
invisible_geo.discover_egress_ip = discover_egress_ip
|
||||
invisible_download.ensure_geoip_mmdb = ensure_geoip_mmdb # pyright: ignore[reportPrivateImportUsage]
|
||||
|
||||
|
||||
class BrowserGeo(NamedTuple):
|
||||
@@ -105,6 +172,45 @@ async def get_browser_geo() -> BrowserGeo:
|
||||
return _geo
|
||||
|
||||
|
||||
_warm_lock = asyncio.Lock()
|
||||
_warmed = False
|
||||
|
||||
|
||||
def _warm_geoip_database() -> None:
|
||||
try:
|
||||
ensure_geoip_mmdb()
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"Could not get the GeoIP database: %s. Browsers start without it; "
|
||||
"the next request after %d seconds retries.",
|
||||
exc,
|
||||
ensure_geoip_mmdb.retry_seconds,
|
||||
)
|
||||
|
||||
|
||||
async def warm_up() -> None:
|
||||
"""
|
||||
Fetch the GeoIP database and, without a proxy, the egress geo once.
|
||||
|
||||
Every browser launch waits for this, so a fresh container downloads the
|
||||
database once instead of once per concurrent first request. A failure is
|
||||
kept by RememberedLookup and retried after its retry_seconds.
|
||||
"""
|
||||
global _warmed # noqa: PLW0603
|
||||
async with _warm_lock:
|
||||
if _warmed:
|
||||
return
|
||||
logger.info(
|
||||
"Fetching the GeoIP database and egress IP before starting browsers"
|
||||
)
|
||||
lookups: list[Awaitable[object]] = [asyncio.to_thread(_warm_geoip_database)]
|
||||
# With PROXY_SERVER every session is proxied, so the direct IP is never used.
|
||||
if not PROXY_SERVER:
|
||||
lookups.append(get_browser_geo())
|
||||
await asyncio.gather(*lookups)
|
||||
_warmed = True
|
||||
|
||||
|
||||
async def get_browser(
|
||||
x_proxy_server: Annotated[
|
||||
str | None,
|
||||
@@ -127,6 +233,8 @@ async def get_browser(
|
||||
] = None,
|
||||
) -> AsyncGenerator[BrowserDepClass]:
|
||||
"""Get InvisiblePlaywright browser instance."""
|
||||
# The lifespan already ran it under uvicorn; this covers callers without one.
|
||||
await warm_up()
|
||||
header_server = x_proxy_server
|
||||
header_username = x_proxy_username
|
||||
header_password = x_proxy_password
|
||||
|
||||
+11
-1
@@ -1,7 +1,7 @@
|
||||
import base64
|
||||
from http import HTTPStatus
|
||||
from json import JSONDecodeError
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
import httpx2
|
||||
import pytest
|
||||
@@ -93,6 +93,16 @@ def test_json_api():
|
||||
assert '"ip"' in solution["response"]
|
||||
|
||||
|
||||
def test_lifespan_warms_up_before_serving():
|
||||
"""Uvicorn opens the port only after the GeoIP warm-up has finished."""
|
||||
with patch("main.warm_up", new=AsyncMock()) as warm, TestClient(app) as local:
|
||||
warm.assert_awaited_once()
|
||||
assert (
|
||||
local.get("/", follow_redirects=False).status_code
|
||||
== HTTPStatus.MOVED_PERMANENTLY
|
||||
)
|
||||
|
||||
|
||||
def test_health_check():
|
||||
"""
|
||||
Tests the health check endpoint.
|
||||
|
||||
+271
-5
@@ -1,9 +1,16 @@
|
||||
# pyright: reportPrivateUsage=false, reportPrivateImportUsage=false
|
||||
import asyncio
|
||||
import logging
|
||||
import math
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
import time
|
||||
from collections.abc import Callable, Iterator
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from pathlib import Path
|
||||
from unittest.mock import DEFAULT, AsyncMock, MagicMock, patch
|
||||
|
||||
import pytest
|
||||
from invisible_core import GeoTimezoneError, SessionGeo, SessionLocale
|
||||
from invisible_playwright.async_api import prepare_session_geo
|
||||
|
||||
from src import utils
|
||||
|
||||
@@ -18,10 +25,17 @@ FAILED = SessionGeo(
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def fresh_geo_cache(monkeypatch: pytest.MonkeyPatch):
|
||||
"""Every test starts with nothing resolved."""
|
||||
def fresh_geo_cache(monkeypatch: pytest.MonkeyPatch) -> Iterator[None]:
|
||||
"""Every test starts with nothing resolved and the warm-up already done."""
|
||||
monkeypatch.setattr(utils, "_geo", None)
|
||||
monkeypatch.setattr(utils, "_geo_lock", asyncio.Lock())
|
||||
monkeypatch.setattr(utils, "_warm_lock", asyncio.Lock())
|
||||
monkeypatch.setattr(utils, "_warmed", True)
|
||||
utils.discover_egress_ip.clear()
|
||||
utils.ensure_geoip_mmdb.clear()
|
||||
yield
|
||||
utils.discover_egress_ip.clear()
|
||||
utils.ensure_geoip_mmdb.clear()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@@ -88,8 +102,10 @@ async def test_explicit_timezone_and_locale_survive_a_failed_lookup():
|
||||
with (
|
||||
patch("src.utils.BROWSER_TIMEZONE", "Asia/Shanghai"),
|
||||
patch("src.utils.BROWSER_LOCALE", "zh-CN"),
|
||||
patch("invisible_core._geo.discover_egress_ip", side_effect=OSError("blocked")),
|
||||
patch("invisible_core.download.ensure_geoip_mmdb") as geoip,
|
||||
patch.object(
|
||||
utils.discover_egress_ip, "lookup", side_effect=OSError("blocked")
|
||||
),
|
||||
patch.object(utils.ensure_geoip_mmdb, "lookup") as geoip,
|
||||
):
|
||||
geo = await utils.get_browser_geo()
|
||||
|
||||
@@ -131,3 +147,253 @@ async def test_proxy_sessions_resolve_their_own_egress(
|
||||
kwargs = launcher.call_args.kwargs
|
||||
assert {"timezone": kwargs["timezone"], "locale": kwargs["locale"]} == expected
|
||||
assert lookup.called is (proxy_header is None)
|
||||
|
||||
|
||||
EGRESS_IP = "203.0.113.7"
|
||||
|
||||
|
||||
def test_launches_reuse_one_egress_and_geoip_lookup():
|
||||
"""Seam guard: the launcher's geo step goes through the remembered lookups."""
|
||||
with (
|
||||
patch.object(utils.discover_egress_ip, "lookup", return_value=EGRESS_IP) as ip,
|
||||
patch.object(
|
||||
utils.ensure_geoip_mmdb, "lookup", return_value=Path("geo.mmdb")
|
||||
) as geoip,
|
||||
patch("invisible_core._geo.ip_to_coordinates", return_value=(52.5, 13.4)),
|
||||
):
|
||||
results = [
|
||||
prepare_session_geo("Europe/Berlin", None, "de-DE") for _ in range(3)
|
||||
]
|
||||
|
||||
ip.assert_called_once_with()
|
||||
geoip.assert_called_once_with()
|
||||
assert {geo.egress_ip for geo in results} == {EGRESS_IP}
|
||||
|
||||
|
||||
def test_failed_lookup_is_skipped_until_the_window_ends(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
):
|
||||
"""A failure is kept for GEO_RETRY_SECONDS instead of being repaid per launch."""
|
||||
with (
|
||||
patch.object(
|
||||
utils.discover_egress_ip, "lookup", side_effect=GeoTimezoneError("blocked")
|
||||
) as ip,
|
||||
patch.object(utils.ensure_geoip_mmdb, "lookup"),
|
||||
):
|
||||
for _ in range(2):
|
||||
assert prepare_session_geo("UTC", None, "en-US").egress_ip is None
|
||||
assert ip.call_count == 1
|
||||
|
||||
monkeypatch.setattr(utils.discover_egress_ip, "retry_seconds", 0)
|
||||
utils.discover_egress_ip.clear()
|
||||
for _ in range(2):
|
||||
prepare_session_geo("UTC", None, "en-US")
|
||||
assert ip.call_count == 3 # noqa: PLR2004
|
||||
|
||||
|
||||
def test_failed_geoip_download_is_retried_after_a_short_window():
|
||||
"""A boot blip is retried within seconds; a lasting outage backs off to 10 minutes."""
|
||||
now = time.monotonic()
|
||||
|
||||
def fail_at(offset: float) -> None:
|
||||
with (
|
||||
patch("src.utils.time.monotonic", return_value=now + offset),
|
||||
pytest.raises(RuntimeError),
|
||||
):
|
||||
utils.ensure_geoip_mmdb()
|
||||
|
||||
with patch.object(
|
||||
utils.ensure_geoip_mmdb, "lookup", side_effect=RuntimeError("offline")
|
||||
) as geoip:
|
||||
fail_at(0)
|
||||
fail_at(utils.GEOIP_RETRY_SECONDS - 1)
|
||||
assert geoip.call_count == 1
|
||||
|
||||
# Each failure doubles the next window until it reaches GEO_RETRY_SECONDS.
|
||||
offset, window, calls = 0.0, utils.GEOIP_RETRY_SECONDS, 1
|
||||
while window < utils.GEO_RETRY_SECONDS:
|
||||
offset += window
|
||||
fail_at(offset)
|
||||
calls += 1
|
||||
window *= 2
|
||||
fail_at(offset + min(window, utils.GEO_RETRY_SECONDS) - 1)
|
||||
assert geoip.call_count == calls
|
||||
offset += utils.GEO_RETRY_SECONDS
|
||||
fail_at(offset)
|
||||
fail_at(offset + utils.GEO_RETRY_SECONDS - 1)
|
||||
assert geoip.call_count == calls + 1
|
||||
|
||||
utils.ensure_geoip_mmdb.clear()
|
||||
fail_at(offset)
|
||||
fail_at(offset + utils.GEOIP_RETRY_SECONDS)
|
||||
assert geoip.call_count == calls + 3
|
||||
assert utils.GEOIP_RETRY_SECONDS < utils.GEO_RETRY_SECONDS
|
||||
|
||||
|
||||
def test_long_outage_keeps_raising_the_lookup_error():
|
||||
"""Thousands of failures with a float window still back off and raise the real error."""
|
||||
lookup = utils.RememberedLookup(MagicMock(side_effect=RuntimeError("offline")), 0.5)
|
||||
for step in range(1100):
|
||||
with (
|
||||
patch("src.utils.time.monotonic", return_value=step * 1e6),
|
||||
pytest.raises(RuntimeError, match="offline"),
|
||||
):
|
||||
lookup()
|
||||
assert lookup.lookup.call_count == 1100 # noqa: PLR2004
|
||||
|
||||
|
||||
def test_proxied_egress_lookups_are_not_cached():
|
||||
"""Each proxy has its own egress, so proxied lookups always pass through."""
|
||||
proxy = {"server": "socks5://proxy.test:1080"}
|
||||
with patch.object(utils.discover_egress_ip, "lookup", return_value=EGRESS_IP) as ip:
|
||||
for _ in range(2):
|
||||
assert utils.discover_egress_ip(proxy, timeout=5) == EGRESS_IP
|
||||
assert ip.call_count == 2 # noqa: PLR2004
|
||||
ip.assert_called_with(proxy, timeout=5)
|
||||
|
||||
with patch.object(
|
||||
utils.discover_egress_ip, "lookup", side_effect=GeoTimezoneError("down")
|
||||
) as ip:
|
||||
for _ in range(2):
|
||||
with pytest.raises(GeoTimezoneError):
|
||||
utils.discover_egress_ip(proxy, timeout=5)
|
||||
assert ip.call_count == 2 # noqa: PLR2004
|
||||
|
||||
|
||||
def test_concurrent_first_lookups_share_one_request():
|
||||
"""Threads that ask at once wait for the one lookup already running."""
|
||||
|
||||
def slow_lookup(**_: object) -> str:
|
||||
time.sleep(0.05)
|
||||
return EGRESS_IP
|
||||
|
||||
with (
|
||||
patch.object(utils.discover_egress_ip, "lookup", side_effect=slow_lookup) as ip,
|
||||
ThreadPoolExecutor(4) as pool,
|
||||
):
|
||||
results = list(pool.map(utils.discover_egress_ip, [None] * 4))
|
||||
|
||||
assert results == [EGRESS_IP] * 4
|
||||
assert ip.call_count == 1
|
||||
|
||||
|
||||
async def drain_browser(**kwargs: str | None) -> None:
|
||||
"""Run get_browser() to the end, as FastAPI does for one request."""
|
||||
async for _ in utils.get_browser(**kwargs):
|
||||
pass
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_warm_up_runs_once_before_any_browser(monkeypatch: pytest.MonkeyPatch):
|
||||
"""Concurrent first launches wait for one warm-up instead of each fetching."""
|
||||
monkeypatch.setattr(utils, "_warmed", False)
|
||||
|
||||
def slow(value: object) -> Callable[..., object]:
|
||||
def lookup(**_: object) -> object:
|
||||
time.sleep(0.05)
|
||||
return value
|
||||
|
||||
return lookup
|
||||
|
||||
launcher = fake_invisible_playwright()
|
||||
seen: list[tuple[bool, int, int]] = []
|
||||
with (
|
||||
patch("src.utils.PROXY_SERVER", None),
|
||||
patch("src.utils.BROWSER_LOCALE", "de-DE"),
|
||||
patch("src.utils.InvisiblePlaywright", launcher),
|
||||
patch.object(
|
||||
utils.discover_egress_ip, "lookup", side_effect=slow(EGRESS_IP)
|
||||
) as ip,
|
||||
patch.object(
|
||||
utils.ensure_geoip_mmdb, "lookup", side_effect=slow(Path("geo.mmdb"))
|
||||
) as geoip,
|
||||
patch("invisible_core._geo.ip_to_coordinates", return_value=(52.5, 13.4)),
|
||||
patch("invisible_core._geo.ip_to_timezone", return_value="Europe/Berlin"),
|
||||
):
|
||||
|
||||
def record(**_: object) -> object:
|
||||
seen.append((utils._warmed, ip.call_count, geoip.call_count)) # noqa: SLF001
|
||||
return DEFAULT
|
||||
|
||||
launcher.side_effect = record
|
||||
await asyncio.gather(*(drain_browser() for _ in range(3)))
|
||||
|
||||
ip.assert_called_once_with()
|
||||
geoip.assert_called_once_with()
|
||||
assert seen == [(True, 1, 1)] * 3
|
||||
assert launcher.call_args.kwargs["timezone"] == "Europe/Berlin"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_failed_warm_up_still_serves_with_the_fallback(
|
||||
monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
|
||||
):
|
||||
"""A failed warm-up logs one warning per fact and browsers use the fallback."""
|
||||
monkeypatch.setattr(utils, "_warmed", False)
|
||||
caplog.set_level(logging.WARNING, logger="uvicorn.error")
|
||||
launcher = fake_invisible_playwright()
|
||||
with (
|
||||
patch("src.utils.PROXY_SERVER", None),
|
||||
patch("src.utils.get_localzone_name", return_value=None),
|
||||
patch("src.utils.InvisiblePlaywright", launcher),
|
||||
patch.object(
|
||||
utils.discover_egress_ip, "lookup", side_effect=GeoTimezoneError("blocked")
|
||||
) as ip,
|
||||
patch.object(
|
||||
utils.ensure_geoip_mmdb,
|
||||
"lookup",
|
||||
side_effect=RuntimeError("geoip mmdb unavailable"),
|
||||
) as geoip,
|
||||
):
|
||||
await utils.warm_up()
|
||||
assert utils._warmed is True # noqa: SLF001
|
||||
|
||||
messages = [
|
||||
r.getMessage() for r in caplog.records if r.levelno >= logging.WARNING
|
||||
]
|
||||
assert len([m for m in messages if "GeoIP" in m]) == 1
|
||||
assert len([m for m in messages if "timezone" in m]) == 1
|
||||
|
||||
await drain_browser()
|
||||
|
||||
kwargs = launcher.call_args.kwargs
|
||||
assert (kwargs["timezone"], kwargs["locale"]) == ("UTC", "en-US")
|
||||
assert (ip.call_count, geoip.call_count) == (1, 1)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_warm_up_skips_the_direct_egress_behind_a_proxy(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
):
|
||||
"""With PROXY_SERVER set every session is proxied, so the direct IP is unused."""
|
||||
monkeypatch.setattr(utils, "_warmed", False)
|
||||
with (
|
||||
patch("src.utils.PROXY_SERVER", "socks5://proxy.test:1080"),
|
||||
patch.object(utils.discover_egress_ip, "lookup") as ip,
|
||||
patch.object(
|
||||
utils.ensure_geoip_mmdb, "lookup", return_value=Path("geo.mmdb")
|
||||
) as geoip,
|
||||
):
|
||||
await utils.warm_up()
|
||||
|
||||
geoip.assert_called_once_with()
|
||||
ip.assert_not_called()
|
||||
assert utils._geo is None # noqa: SLF001
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_warm_up_is_a_no_op_once_done(monkeypatch: pytest.MonkeyPatch):
|
||||
"""The _warmed flag, not the remembered lookup, short-circuits a second call."""
|
||||
monkeypatch.setattr(utils, "_warmed", False)
|
||||
with (
|
||||
patch("src.utils.PROXY_SERVER", None),
|
||||
patch("src.utils.prepare_session_geo", return_value=BERLIN),
|
||||
patch.object(
|
||||
utils.ensure_geoip_mmdb, "lookup", return_value=Path("geo.mmdb")
|
||||
) as geoip,
|
||||
):
|
||||
await utils.warm_up()
|
||||
utils.ensure_geoip_mmdb.clear()
|
||||
await utils.warm_up()
|
||||
|
||||
assert geoip.call_count == 1
|
||||
Reference in new issue
Block a user