Commit Graph
38 Commits
Author SHA1 Message Date
ThePhaseless d3a828e814 fix(v1): source User-Agent from request headers; evaluate only as fallback
page.evaluate runs eval() in the page's main world, which fails with 'call to eval() blocked by CSP' under any CSP that disallows unsafe-eval - HTTP headers (already stripped), meta tags (not strippable), or internal viewer documents (#394).

The navigation request already carries the UA the site actually saw, so take user_agent from page_request.request.headers and keep evaluate only as a best-effort fallback whose failure can no longer 500 the request.
2026-08-11 11:14:10 +02:00
ThePhaseless 46a3c68eb0 fix: disable Firefox JSON viewer so evaluate works on JSON APIs
Firefox renders application/json documents in a built-in viewer whose own
CSP (<script-src resource:>) blocks Playwright's eval-based page.evaluate,
crashing /v1 with a 500 on JSON APIs (closes #394). Setting
devtools.jsonview.enabled=false renders JSON as plain text, which also
returns the raw JSON body instead of the viewer's syntax-highlighted HTML.
2026-08-11 11:08:04 +02:00
ThePhaseless 23374ce58e chore: migrate tests from httpx to httpx2
Starlette's TestClient deprecates httpx; httpx2 is the maintained
successor (Pydantic stewardship) with a drop-in API.
2026-08-08 01:42:10 +02:00
ThePhaseless 490e2fad97 refactor: keep networkidle timeout handling inline, mock-based tests 2026-08-08 00:37:09 +02:00
ThePhaseless 7b904a5ffd feat: continue after networkidle timeout once domcontentloaded completes
A page whose network never goes idle (background analytics, websockets)
used to fail the whole request with a 408 once the networkidle wait
expired. Since the DOM is fully usable after domcontentloaded, treat a
networkidle timeout as non-fatal and return the loaded page instead.
Fatal timeouts during initial load or challenge solving still return 408.

Adds unit coverage for both paths using a fake page that fails
configured load-state waits.
2026-08-08 00:27:54 +02:00
ThePhaseless 0b4d1a91ce feat: accept FlareSolverr maxTimeout in milliseconds
Add a maxTimeout alias to LinkRequest.max_timeout for FlareSolverr
drop-in compatibility. Values of 1000 or more are treated as
milliseconds and normalized to seconds; smaller values keep the
native seconds semantics. Closes #382.
2026-08-07 23:51:44 +02:00
ThePhaseless 80a608a629 feat: add blockMedia, returnOnlyCookies, PDF handling and fix timeout/networkidle
- Catch both builtins.TimeoutError and playwright TimeoutError as 408

- Check challenge title before networkidle to avoid timeout on Cloudflare interstitial

- Add blockMedia and returnOnlyCookies request options

- Return raw PDF bytes as base64 with contentType application/pdf

- Skip tests on 408 timeouts; add PDF handling test
2026-07-04 17:34:54 +02:00
ThePhaseless f98e823991 Migrate to camoufox (#235)
* use camoufox

* build custom branches

* organize compose

* ignore missing stubs

* add init method

* create ADDON PATH

* check if solving is required

* type checking and remove logger

* uv sync

* add timeout

* add descriptiuon

* cancel previous runs

* wildcard minor

* fix proxy

* syntax fix

* [skip ci] fix loop warning

* fix proxy format

* copy over docker ignore

* remove testing line

* comment out port expose

* remove idope

* use strict typechecking

* refactor

* adjust compose

* run init only before test

* try test without init

* add curl

* handle page response

* remove turnsile test

* fix dockerignore symlink

* use newer debian

* bump pytest

* add docker in docker in devcontaienr

* remove unused values

* handle timeouts

* fix edgecase

* use new envs for proxy

* remove init command

* remove testing line

* remove setuptools

* fix linters

* reimport

* readd setuptools

* better float handling
2025-09-05 23:08:35 +02:00
Alvaro Santos Andres 39f758859d fix: disabled HEADLESS mode by default and updated dependencies 2025-07-25 13:51:19 +02:00
ThePhaseless 9fc9af3ba9 add unescaped chars test link 2025-05-18 01:14:32 +02:00
ThePhaseless 1e6bd9e656 do not use proxy on tests 2025-02-27 19:10:47 +00:00
ThePhaseless 8e53210014 fix tests not being skipped properly 2025-02-20 20:37:02 +00:00
ThePhaseless ecbb2ab182 separate files, add gzip, save screenshots on exception, add PROXY support 2025-02-17 23:08:14 +00:00
ThePhaseless 4bfb7845dd remove btmet due to dns issue 2025-02-06 17:50:44 +00:00
ThePhaseless cbe4cf57a5 use all urls always 2024-11-30 12:43:05 +00:00
ThePhaseless 2ae289ef0c use renovate instead of dependabot 2024-11-28 12:33:04 +00:00
ThePhaseless c366ce7db6 add healthcheck 2024-11-25 10:19:09 +00:00
ThePhaseless 77fd2a47f7 remaster 2024-11-24 22:00:58 +00:00
ThePhaseless b01ee50444 skip tests on github arm 2024-11-21 22:59:27 +00:00
ThePhaseless e41c831c73 separated github dependent websites 2024-10-27 18:46:19 +00:00
ThePhaseless c409ca22f9 remove untestable websites from github 2024-10-27 18:43:36 +00:00
ThePhaseless 8e5b7061e8 test if cloudflare blocks after 2 requests 2024-10-27 18:40:21 +00:00
ThePhaseless aabff0a65f do not skip challenged 2024-10-27 18:32:06 +00:00
ThePhaseless 0535c5d9bd add logger even if passing 2024-10-21 13:46:26 +00:00
ThePhaseless d26ddf3bcf add healthcheck 2024-10-21 13:34:07 +00:00
ThePhaseless 361e11cda2 skip if test if client cant reach 2024-10-21 13:19:31 +00:00
ThePhaseless 4e3876e0b7 skip test if rate limited 2024-10-19 21:47:16 +00:00
ThePhaseless 2dfc5b2201 refactor and test in dockerfile 2024-10-19 20:35:11 +00:00
ThePhaseless 8b0caf6adc dont test (fck it) 2024-10-18 16:33:27 +00:00
ThePhaseless dbc91bc982 reduce max time to 30 sec 2024-10-18 16:03:17 +00:00
ThePhaseless 464a618f29 print current html on error 2024-10-18 15:53:36 +00:00
ThePhaseless 2e819c9adf skip if rate limited 2024-10-18 15:53:36 +00:00
ThePhaseless 29a5432684 increase timeout in tests 2024-10-18 15:53:36 +00:00
ThePhaseless ee9e8ca98f disable invalid websites 2024-09-13 19:22:37 +00:00
ThePhaseless e25415012a quick check 2024-09-13 19:13:37 +00:00
ThePhaseless 22fba55373 adjustments 2024-09-13 18:48:31 +00:00
ThePhaseless d8d92ec3de use fastapi to test 2024-09-13 18:28:36 +00:00
ThePhaseless 79fb20082f add timed tests 2024-09-13 18:04:37 +00:00