Commit Graph
29 Commits
Author SHA1 Message Date
ThePhaseless 89dcf5e16c Merge branch 'main' into chore/ruff-lint-cleanup
Resolved conflicts in src/consts.py and src/endpoints.py:
- consts.py: take theirs (CHALLENGE_TITLES removed, browser_locale added,
  CaptchaType import no longer needed — detection is now library-based)
- endpoints.py: merge both refactors — keep theirs' detect_cloudflare_challenge
  + page_html capture, reapply my helper extraction (setup_routes,
  _navigate_and_solve, _solve_challenge, _wait_for_networkidle,
  build_response_content, _fetch_pdf_content) on top
2026-08-11 00:20:55 +02:00
ThePhaseless 3c45ef9691 chore: fix ruff lint findings and refactor read_item
- Fix I001: sort imports in src/consts.py
- Fix PLC0415: move `import base64` to top of tests/main_test.py
- Fix UP037: remove quotes from LinkResponse return annotation
- Fix D213: correct multi-line docstring summary placement
- Remove unused `# noqa: BLE001` in src/owui.py
- Refactor read_item into helpers: setup_routes, load_page_and_solve,
  build_response_content, _fetch_pdf_content — resolves C901 and PLR0915
- Add CPY001, BLE001 to ruff ignore list
2026-08-11 00:15:39 +02:00
ThePhaseless 8ef4c62249 fix: detect Cloudflare challenges regardless of language (#385)
Cloudflare localizes its interstitial page title per visitor language
(e.g. Polish "Cierpliwości..." served by 1337x.to), so the hard-coded
["Just a moment..."] title check missed every non-English visitor:
Byparr returned the raw challenge page (HTTP 403, no cf_clearance
cookie, no "Challenge detected" log) and Prowlarr reported "Unable to
access 1337x.to, blocked by CloudFlare Protection." (issue #385, still
open on 3.0.1 after the compression fix).

Replace the title-based gate with the playwright-captcha library's own
language-independent DOM detection (detect_cloudflare_challenge), which
matches Cloudflare's challenge scripts directly:
  - interstitial:  script[src*="/cdn-cgi/challenge-platform/"]
  - turnstile:     input[name="cf-turnstile-response"],
                   script[src*="challenges.cloudflare.com/turnstile/v0"]
Both selectors match the live 1337x "Cierpliwości..." interstitial.

The navigation/detect/solve flow lives in _navigate_and_solve(); the
timeout-to-408 translation is inlined at the call site in read_item.
The now-unused title map is removed from src/consts.py.

Verified live (built image): "Challenge detected" now fires on 1337x
(0 -> 1 in logs) where the title check never fired; example.com negative
control returns 200 with no challenge path entered. End-to-end clearing
still depends on the requester's public IP (README caveat).
2026-08-10 12:05:51 +02:00
ThePhaseless 0c44ce1a4d fix: request uncompressed bodies in CSP-strip route
route.fulfill(response=...) re-serves the raw bytes fetched by
route.fetch(), so compressed (gzip/brotli/zstd) documents arrive
at the browser still compressed while the forwarded headers claim
otherwise - page.content() then returns garbled binary, breaking
indexers like uindex.org and 1337x.to (issue #385).

Fetch with accept-encoding: identity so the re-served body is plain
text, and drop content-encoding/content-length alongside the CSP
headers since they are stale after the rewrite.
2026-08-09 19:07:05 +02:00
ThePhaseless 5c4d0393b4 chore: drop redundant comment on networkidle best-effort wait 2026-08-08 00:50:10 +02:00
Jakub Orchowski 0dc4643d03 Merge branch 'main' into handle-networkidle-timeouts 2026-08-08 00:45:16 +02:00
ThePhaseless 490e2fad97 refactor: keep networkidle timeout handling inline, mock-based tests 2026-08-08 00:37:09 +02:00
ThePhaseless e2fd2e6d42 refactor: simplify CSP stripping handler
Drop error handling and conditional branches that duplicated the
pass-through path; rely on the goto timeout as before.
2026-08-08 00:31:25 +02:00
ThePhaseless 7b904a5ffd feat: continue after networkidle timeout once domcontentloaded completes
A page whose network never goes idle (background analytics, websockets)
used to fail the whole request with a 408 once the networkidle wait
expired. Since the DOM is fully usable after domcontentloaded, treat a
networkidle timeout as non-fatal and return the loaded page instead.
Fatal timeouts during initial load or challenge solving still return 408.

Adds unit coverage for both paths using a fake page that fails
configured load-state waits.
2026-08-08 00:27:54 +02:00
ThePhaseless 6d447a0d67 fix: strip CSP headers from page responses so evaluate works
The Firefox engine evaluates JS via eval(), which pages whose CSP
lacks 'unsafe-eval' block - every page.evaluate() then fails with
"call to eval() blocked by CSP". yggtorrent's search URL redirects to
a page with such a CSP, crashing the user-agent read and 500ing /v1.

Rewrite document responses without CSP headers via route.fetch +
fulfill. Juggler only routes the first request of a redirect chain,
so follow redirects inside the fetch and record the final URL
ourselves instead of relying on page.url.
2026-08-08 00:25:46 +02:00
ThePhaseless 80a608a629 feat: add blockMedia, returnOnlyCookies, PDF handling and fix timeout/networkidle
- Catch both builtins.TimeoutError and playwright TimeoutError as 408

- Check challenge title before networkidle to avoid timeout on Cloudflare interstitial

- Add blockMedia and returnOnlyCookies request options

- Return raw PDF bytes as base64 with contentType application/pdf

- Skip tests on 408 timeouts; add PDF handling test
2026-07-04 17:34:54 +02:00
ThePhaseless 849e33dff3 use timer to timeout functions 2025-09-11 20:26:11 +00:00
ThePhaseless c66d4e7084 return 200 if intersite challenge 2025-09-11 16:54:50 +00:00
ThePhaseless 9ffc2bb5b3 fix exception 2025-09-10 20:47:59 +00:00
ThePhaseless 258f0bdbad add faster turnstile check 2025-09-10 22:45:40 +02:00
ThePhaseless f98e823991 Migrate to camoufox (#235)
* use camoufox

* build custom branches

* organize compose

* ignore missing stubs

* add init method

* create ADDON PATH

* check if solving is required

* type checking and remove logger

* uv sync

* add timeout

* add descriptiuon

* cancel previous runs

* wildcard minor

* fix proxy

* syntax fix

* [skip ci] fix loop warning

* fix proxy format

* copy over docker ignore

* remove testing line

* comment out port expose

* remove idope

* use strict typechecking

* refactor

* adjust compose

* run init only before test

* try test without init

* add curl

* handle page response

* remove turnsile test

* fix dockerignore symlink

* use newer debian

* bump pytest

* add docker in docker in devcontaienr

* remove unused values

* handle timeouts

* fix edgecase

* use new envs for proxy

* remove init command

* remove testing line

* remove setuptools

* fix linters

* reimport

* readd setuptools

* better float handling
2025-09-05 23:08:35 +02:00
ThePhaseless 4ab01abbf0 ignore regex syntax warning 2025-08-16 22:49:24 +00:00
ThePhaseless 858bba598f bump deps 2025-08-16 22:41:26 +00:00
Alvaro Santos Andres b153d4ae34 fix: simplify code to fix PR suggestions 2025-08-11 06:57:42 +00:00
Alvaro Santos Andres 707a9c7a83 fix: use of CDP Mode 2025-07-25 16:31:28 +00:00
Alvaro Santos Andres 6b0da3cc5e fix: add more logging to debug errors 2025-07-25 14:38:22 +00:00
ThePhaseless cdb3223a87 fix typo 2025-05-17 22:45:55 +00:00
ThePhaseless fe797ba2ac add userAgent and version to healthcheck 2025-04-17 11:27:17 +00:00
ThePhaseless 08cfd95d6a make all pyfiles as not executable 2025-04-12 20:47:37 +00:00
ThePhaseless c75c79a2ad rename and add missing fields 2025-04-08 13:13:01 +00:00
Maxime Gagnon dd74f80c18 restore flaresolverr compatibility 2025-03-30 16:31:05 -04:00
ThePhaseless 7e2c895a11 faster title check 2025-02-26 11:03:46 +01:00
ThePhaseless 43cd9297cb fix #86 2025-02-20 20:36:48 +00:00
ThePhaseless ecbb2ab182 separate files, add gzip, save screenshots on exception, add PROXY support 2025-02-17 23:08:14 +00:00