Compare commits

...
63 Commits
Author SHA1 Message Date
Alex 4881adc19f Fix tor error retry (#941) 2026-05-01 19:30:32 +01:00
Alex ee54033d23 Fix entrypoint handling on Unraid (#939) 2026-05-01 18:41:43 +01:00
Alex 3554d01c81 Change path default for audiobooks + description fixes (#933) 2026-04-30 18:20:05 +01:00
Alex 9dd445f2af Revert image size streaming (#932) 2026-04-30 17:07:43 +01:00
Alex 4e41b1a8ec fix: hardcover author search (#929)
- Re-adds author search suggestions for Hardcover
- Correctly routes author queries to correct ID or best match
author-filtered book query.
2026-04-29 19:10:37 +01:00
Alex c7619488cd fix: re-add cwa db immutable (#928)
Fixes #926
2026-04-29 18:34:28 +01:00
Alex eba04115ae Fix internal bypasser and /home directory (#919)
- Fixed internal bypasser startup with newer Chromium/SeleniumBase by
isolating the browser helper from Gunicorn/gevent, serialising helper
failures cleanly, and cleaning up orphan processes after a failure
- Stopped using /app as runtime home state, now moved to /home/shelfmark
or /tmp/shelfmark/home as fallback.
- Added tests
2026-04-26 15:01:54 +01:00
dependabot[bot] 26bba25777 Bump the python-deps group across 1 directory with 6 updates (#900)
Bumps the python-deps group with 5 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [authlib](https://github.com/authlib/authlib) | `1.6.9` | `1.7.0` |
| [seleniumbase](https://github.com/seleniumbase/SeleniumBase) |
`4.47.9` | `4.48.2` |
| [basedpyright](https://github.com/detachhead/basedpyright) | `1.39.0`
| `1.39.3` |
| [prek](https://github.com/j178/prek) | `0.3.8` | `0.3.10` |
| [ruff](https://github.com/astral-sh/ruff) | `0.15.10` | `0.15.11` |


Updates `authlib` from 1.6.9 to 1.7.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/authlib/authlib/releases">authlib's
releases</a>.</em></p>
<blockquote>
<h2>v1.7.0</h2>
<h2>What's Changed</h2>
<ul>
<li>Authorization and token endpoints request empty scope parameter
management by <a
href="https://github.com/azmeuk"><code>@​azmeuk</code></a> in <a
href="https://redirect.github.com/authlib/authlib/pull/847">authlib/authlib#847</a></li>
<li>Support from Python 3.10 to 3.14 by <a
href="https://github.com/azmeuk"><code>@​azmeuk</code></a> in <a
href="https://redirect.github.com/authlib/authlib/pull/850">authlib/authlib#850</a></li>
<li>Allow composition of AuthorizationServerMetadata by <a
href="https://github.com/azmeuk"><code>@​azmeuk</code></a> in <a
href="https://redirect.github.com/authlib/authlib/pull/853">authlib/authlib#853</a></li>
<li>Make require_oauth parenthesis optional by <a
href="https://github.com/azmeuk"><code>@​azmeuk</code></a> in <a
href="https://redirect.github.com/authlib/authlib/pull/855">authlib/authlib#855</a></li>
<li>Fix <code>expires_at</code> behavior when its value is 0 by <a
href="https://github.com/azmeuk"><code>@​azmeuk</code></a> in <a
href="https://redirect.github.com/authlib/authlib/pull/854">authlib/authlib#854</a></li>
<li>Migration to joserfc by <a
href="https://github.com/lepture"><code>@​lepture</code></a> in <a
href="https://redirect.github.com/authlib/authlib/pull/852">authlib/authlib#852</a></li>
<li>RP-initiated logout by <a
href="https://github.com/frohrlich"><code>@​frohrlich</code></a> in <a
href="https://redirect.github.com/authlib/authlib/pull/849">authlib/authlib#849</a></li>
<li>Fix <code>get_jwt_config</code> by <a
href="https://github.com/lepture"><code>@​lepture</code></a> in <a
href="https://redirect.github.com/authlib/authlib/pull/858">authlib/authlib#858</a></li>
<li>chore(ci): Update PyPy version from 3.10 to 3.11 by <a
href="https://github.com/cclauss"><code>@​cclauss</code></a> in <a
href="https://redirect.github.com/authlib/authlib/pull/863">authlib/authlib#863</a></li>
<li>fix: remove &quot;none&quot; from default authlib.jose.jwt
algorithms by <a
href="https://github.com/lepture"><code>@​lepture</code></a> in <a
href="https://redirect.github.com/authlib/authlib/pull/860">authlib/authlib#860</a></li>
<li>fix: normalize resolve_client_public_key method by <a
href="https://github.com/lepture"><code>@​lepture</code></a> in <a
href="https://redirect.github.com/authlib/authlib/pull/861">authlib/authlib#861</a></li>
<li>Implement rfc9700 PKCE downgrade countermeasure by <a
href="https://github.com/azmeuk"><code>@​azmeuk</code></a> in <a
href="https://redirect.github.com/authlib/authlib/pull/864">authlib/authlib#864</a></li>
<li>Use correct syntax for tox.requires in tox.ini by <a
href="https://github.com/alex-ball"><code>@​alex-ball</code></a> in <a
href="https://redirect.github.com/authlib/authlib/pull/868">authlib/authlib#868</a></li>
<li>Set client session User-Agent when fetching server metadata and JWKs
by <a href="https://github.com/alex-ball"><code>@​alex-ball</code></a>
in <a
href="https://redirect.github.com/authlib/authlib/pull/867">authlib/authlib#867</a></li>
<li>fix: use the real application object for Flask by <a
href="https://github.com/nblock"><code>@​nblock</code></a> in <a
href="https://redirect.github.com/authlib/authlib/pull/869">authlib/authlib#869</a></li>
<li>Accept the issuer URL as a valid audience by <a
href="https://github.com/azmeuk"><code>@​azmeuk</code></a> in <a
href="https://redirect.github.com/authlib/authlib/pull/865">authlib/authlib#865</a></li>
<li>Don't nest InvalidTokenError extra attribute by <a
href="https://github.com/azmeuk"><code>@​azmeuk</code></a> in <a
href="https://redirect.github.com/authlib/authlib/pull/872">authlib/authlib#872</a></li>
<li>Documentation overhaul by <a
href="https://github.com/azmeuk"><code>@​azmeuk</code></a> in <a
href="https://redirect.github.com/authlib/authlib/pull/875">authlib/authlib#875</a></li>
<li>Update README.md docs.authlib.org/en/latest =&gt;
docs.authlib.org/en/stable by <a
href="https://github.com/guillett"><code>@​guillett</code></a> in <a
href="https://redirect.github.com/authlib/authlib/pull/876">authlib/authlib#876</a></li>
<li>Merge release/1.6 branch by <a
href="https://github.com/lepture"><code>@​lepture</code></a> in <a
href="https://redirect.github.com/authlib/authlib/pull/877">authlib/authlib#877</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/frohrlich"><code>@​frohrlich</code></a>
made their first contribution in <a
href="https://redirect.github.com/authlib/authlib/pull/849">authlib/authlib#849</a></li>
<li><a href="https://github.com/cclauss"><code>@​cclauss</code></a> made
their first contribution in <a
href="https://redirect.github.com/authlib/authlib/pull/863">authlib/authlib#863</a></li>
<li><a href="https://github.com/alex-ball"><code>@​alex-ball</code></a>
made their first contribution in <a
href="https://redirect.github.com/authlib/authlib/pull/868">authlib/authlib#868</a></li>
<li><a href="https://github.com/nblock"><code>@​nblock</code></a> made
their first contribution in <a
href="https://redirect.github.com/authlib/authlib/pull/869">authlib/authlib#869</a></li>
<li><a href="https://github.com/guillett"><code>@​guillett</code></a>
made their first contribution in <a
href="https://redirect.github.com/authlib/authlib/pull/876">authlib/authlib#876</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/authlib/authlib/compare/v1.6.10...v1.7.0">https://github.com/authlib/authlib/compare/v1.6.10...v1.7.0</a></p>
<h2>v1.6.11</h2>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/authlib/authlib/compare/v1.6.10...v1.6.11">https://github.com/authlib/authlib/compare/v1.6.10...v1.6.11</a></p>
<ul>
<li>Fix CSRF issue with starlette client</li>
</ul>
<h2>v1.6.10</h2>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/authlib/authlib/compare/v1.6.9...v1.6.10">https://github.com/authlib/authlib/compare/v1.6.9...v1.6.10</a></p>
<ul>
<li>Fix redirecting to unvalidated <code>redirect_uri</code> on
<code>UnsupportedResponseTypeError</code>.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/authlib/authlib/commit/5d2e603ec5f10bd2c4bf20e2495c076370d65b74"><code>5d2e603</code></a>
chore: release 1.7.0</li>
<li><a
href="https://github.com/authlib/authlib/commit/767f08bb80ad6635beb7b54ad98ee9494b84bc26"><code>767f08b</code></a>
fix: CSRF issue with starlette client</li>
<li><a
href="https://github.com/authlib/authlib/commit/e9aaef33933717e1bb5cf77a2cc1ceeb4c948620"><code>e9aaef3</code></a>
Merge pull request <a
href="https://redirect.github.com/authlib/authlib/issues/877">#877</a>
from authlib/merge/1.6</li>
<li><a
href="https://github.com/authlib/authlib/commit/3c8ec9a8895ce3f7c588a9c3bf8e48cdf0c5b2c3"><code>3c8ec9a</code></a>
Merge branch 'main' into merge/1.6</li>
<li><a
href="https://github.com/authlib/authlib/commit/ef09aebbba4439dedb22bd15777d1b3458b6f0ab"><code>ef09aeb</code></a>
chore: release 1.6.10</li>
<li><a
href="https://github.com/authlib/authlib/commit/3be08468201a7766a93012ce149ea12822cab096"><code>3be0846</code></a>
fix: redirecting to unvalidated redirect_uri on
UnsupportedResponseTypeError</li>
<li><a
href="https://github.com/authlib/authlib/commit/4cf6f9792c56a5d598d1525af6384380cdccc4ec"><code>4cf6f97</code></a>
Merge pull request <a
href="https://redirect.github.com/authlib/authlib/issues/876">#876</a>
from guillett/patch-1</li>
<li><a
href="https://github.com/authlib/authlib/commit/23f67b440ca4e2283139b6a57b3c979dbb3c4a50"><code>23f67b4</code></a>
Update README.md docs.authlib.org/en/latest =&gt;
docs.authlib.org/en/stable</li>
<li><a
href="https://github.com/authlib/authlib/commit/10401635d06f59aa282d3367b38dba73574a9127"><code>1040163</code></a>
chore: prek autoupdate</li>
<li><a
href="https://github.com/authlib/authlib/commit/491209fb393c829cb38aebecd614f612191286b2"><code>491209f</code></a>
Merge pull request <a
href="https://redirect.github.com/authlib/authlib/issues/875">#875</a>
from azmeuk/doc</li>
<li>Additional commits viewable in <a
href="https://github.com/authlib/authlib/compare/v1.6.9...v1.7.0">compare
view</a></li>
</ul>
</details>
<br />

Updates `seleniumbase` from 4.47.9 to 4.48.2
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/seleniumbase/SeleniumBase/releases">seleniumbase's
releases</a>.</em></p>
<blockquote>
<h2>Fix unzip in powershell</h2>
<ul>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/2bc7763221c6a0726010caf49476a2f5e87413d7">Fix
unzip in powershell for paths that contain spaces</a></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/b0518dc2bbd3bf28f659a310cec12ef6e07cb4f9">Refresh
Python dependencies</a></li>
</ul>
<h2>What's Changed</h2>
<ul>
<li>Fix unzip in powershell by <a
href="https://github.com/mdmintz"><code>@​mdmintz</code></a> in <a
href="https://redirect.github.com/seleniumbase/SeleniumBase/pull/4330">seleniumbase/SeleniumBase#4330</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/seleniumbase/SeleniumBase/compare/v4.48.1...v4.48.2">https://github.com/seleniumbase/SeleniumBase/compare/v4.48.1...v4.48.2</a></p>
<h2>4.48.1 - Browser config updates</h2>
<h2>Browser config updates</h2>
<ul>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/bfb94e2fbdceea6e34f9f56297ee44eac9045c25">Update
default config for Chrome and Edge</a></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/77ea8fcb8c7bc355aa3b9e1a225b5545a90e4431">Refresh
Python dependencies</a></li>
</ul>
<h2>What's Changed</h2>
<ul>
<li>Browser config updates by <a
href="https://github.com/mdmintz"><code>@​mdmintz</code></a> in <a
href="https://redirect.github.com/seleniumbase/SeleniumBase/pull/4323">seleniumbase/SeleniumBase#4323</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/seleniumbase/SeleniumBase/compare/v4.48.0...v4.48.1">https://github.com/seleniumbase/SeleniumBase/compare/v4.48.0...v4.48.1</a></p>
<h2>4.48.0 - Selenium and CDP updates</h2>
<h2>Selenium and CDP updates</h2>
<ul>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/8123a0e674f507629264b178afa4ba9585776056">Update
CDP Mode</a>
--&gt; <code>solve_captcha()</code> now supports &quot;Friendly
Captcha&quot;
--&gt; <code>nest-asyncio</code> is patched from within to avoid
deprecation warnings on Python 3.14</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/657c7231e62ab19deef514ac002995211b7ab814">Refresh
Python dependencies</a>
--&gt; Includes a <code>selenium</code> upgrade</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/9cc818b48e0cea0718421eff2dbb6ca8f924aed2">Add
an example that bypasses &quot;Friendly Captcha&quot;</a></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/afef6b86e848cdd699f03fe2fa4078c9893ff10d">Add
Stealthy Playwright examples for BrowserScan</a></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/184f079229055b845c514596ef08e284995e9823">Warn
when await is missing from async method calls</a></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/7abc5e4f0f746bf979730a514fc2854ddb5f10fd">Support
&quot;Friendly Captcha&quot; solving in async mode</a></li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/7d0f56963af0baf69b09c1975b6b0c95b2e6855f">Add
async example of solving &quot;Friendly Captcha&quot;</a></li>
</ul>
<h2>What's Changed</h2>
<ul>
<li>Selenium and CDP updates by <a
href="https://github.com/mdmintz"><code>@​mdmintz</code></a> in <a
href="https://redirect.github.com/seleniumbase/SeleniumBase/pull/4319">seleniumbase/SeleniumBase#4319</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/seleniumbase/SeleniumBase/compare/v4.47.9...v4.48.0">https://github.com/seleniumbase/SeleniumBase/compare/v4.47.9...v4.48.0</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/a9605aea13b4498a8b60065a23201dae80413316"><code>a9605ae</code></a>
Merge pull request <a
href="https://redirect.github.com/seleniumbase/SeleniumBase/issues/4330">#4330</a>
from seleniumbase/fix-unzip-in-powershell</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/3a38776ea74d463b4f605753962137401d1ce847"><code>3a38776</code></a>
Version 4.48.2</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/b0518dc2bbd3bf28f659a310cec12ef6e07cb4f9"><code>b0518dc</code></a>
Refresh Python dependencies</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/2bc7763221c6a0726010caf49476a2f5e87413d7"><code>2bc7763</code></a>
Fix unzip in powershell for paths that contain spaces</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/a0aceea5080a14342fdf90919383f407d314f618"><code>a0aceea</code></a>
Merge pull request <a
href="https://redirect.github.com/seleniumbase/SeleniumBase/issues/4323">#4323</a>
from seleniumbase/browser-config-updates</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/9dc951b431e0faab1c586e70adf9ba92a26ad721"><code>9dc951b</code></a>
Version 4.48.1</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/77ea8fcb8c7bc355aa3b9e1a225b5545a90e4431"><code>77ea8fc</code></a>
Refresh Python dependencies</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/bfb94e2fbdceea6e34f9f56297ee44eac9045c25"><code>bfb94e2</code></a>
Update default config for Chrome and Edge</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/a381c354bb1cbef92466d75cadfccd7b2f8a6cb1"><code>a381c35</code></a>
Update documentation in migration examples</li>
<li><a
href="https://github.com/seleniumbase/SeleniumBase/commit/377c3d218e5e688ddbb04aae9ebd71685e6725bf"><code>377c3d2</code></a>
Merge pull request <a
href="https://redirect.github.com/seleniumbase/SeleniumBase/issues/4319">#4319</a>
from seleniumbase/selenium-and-cdp-updates</li>
<li>Additional commits viewable in <a
href="https://github.com/seleniumbase/SeleniumBase/compare/v4.47.9...v4.48.2">compare
view</a></li>
</ul>
</details>
<br />

Updates `basedpyright` from 1.39.0 to 1.39.3
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/DetachHead/basedpyright/commit/5f6672701c5b6a739563894256418845239be2a2"><code>5f66727</code></a>
1.39.3</li>
<li><a
href="https://github.com/DetachHead/basedpyright/commit/d8741dfc88be16e11c66cceff78e6e2634ddf49b"><code>d8741df</code></a>
hopefully fix docs deployment job</li>
<li><a
href="https://github.com/DetachHead/basedpyright/commit/ceb200c192bb53629b4c62fdc082ec27b108f316"><code>ceb200c</code></a>
1.39.2</li>
<li><a
href="https://github.com/DetachHead/basedpyright/commit/eb7a92cacd2697acb4e36e2f3147333588c88fa7"><code>eb7a92c</code></a>
try to fix browser-basedpyright being published with nothing in it</li>
<li><a
href="https://github.com/DetachHead/basedpyright/commit/dec5306e9aabbce3f9b1b9ec283981d92dd43d80"><code>dec5306</code></a>
update package-lock.json</li>
<li><a
href="https://github.com/DetachHead/basedpyright/commit/6db43de5dda601e3cd83998c669c4ecbbfde26f6"><code>6db43de</code></a>
1.39.1</li>
<li><a
href="https://github.com/DetachHead/basedpyright/commit/335e48619fa223fe3be0b6560d746a10934509c8"><code>335e486</code></a>
fix failing tests from merge</li>
<li><a
href="https://github.com/DetachHead/basedpyright/commit/bbe71b686e84ebbff7699d35dab3376d411bb99d"><code>bbe71b6</code></a>
delete newly added github action from upstream that we don't use</li>
<li><a
href="https://github.com/DetachHead/basedpyright/commit/8f2261891e8ceb278963e84658ebaf054b88e559"><code>8f22618</code></a>
update prettierignore</li>
<li><a
href="https://github.com/DetachHead/basedpyright/commit/ff1f5a4af835b14927545c8890b303f48553acac"><code>ff1f5a4</code></a>
fix compile errors from merge</li>
<li>Additional commits viewable in <a
href="https://github.com/detachhead/basedpyright/compare/v1.39.0...v1.39.3">compare
view</a></li>
</ul>
</details>
<br />

Updates `prek` from 0.3.8 to 0.3.10
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/j178/prek/releases">prek's
releases</a>.</em></p>
<blockquote>
<h2>0.3.10</h2>
<h2>Release Notes</h2>
<p>Released on 2026-04-21.</p>
<h3>Enhancements</h3>
<ul>
<li>Disallow rev for non-remote repos in schema (<a
href="https://redirect.github.com/j178/prek/pull/1964">#1964</a>)</li>
<li>Hide up-to-date output in non-verbose mode (<a
href="https://redirect.github.com/j178/prek/pull/1942">#1942</a>)</li>
<li>Improve merge conflict marker detection (<a
href="https://redirect.github.com/j178/prek/pull/1937">#1937</a>)</li>
<li>Keep finished hooks visible (<a
href="https://redirect.github.com/j178/prek/pull/1967">#1967</a>)</li>
<li>Preserve frozen comment spacing in auto-update (<a
href="https://redirect.github.com/j178/prek/pull/1945">#1945</a>)</li>
<li>Reimplement <code>@j178/prek</code> npm package (<a
href="https://redirect.github.com/j178/prek/pull/1973">#1973</a>)</li>
</ul>
<h3>Bug fixes</h3>
<ul>
<li>Prefer stable Rust toolchains (<a
href="https://redirect.github.com/j178/prek/pull/1974">#1974</a>)</li>
</ul>
<h3>Documentation</h3>
<ul>
<li>Add <code>SKILL.md</code> for prek (<a
href="https://redirect.github.com/j178/prek/pull/1950">#1950</a>)</li>
<li>Document <code>gh skill install j178/prek prek</code> to install
prek skill for agents (<a
href="https://redirect.github.com/j178/prek/pull/1951">#1951</a>)</li>
<li>Improve compatibility and migration docs (<a
href="https://redirect.github.com/j178/prek/pull/1940">#1940</a>)</li>
</ul>
<h3>Other changes</h3>
<ul>
<li>Sync latest identify tags (<a
href="https://redirect.github.com/j178/prek/pull/1947">#1947</a>)</li>
</ul>
<h3>Contributors</h3>
<ul>
<li><a
href="https://github.com/github-actions"><code>@​github-actions</code></a></li>
<li><a
href="https://github.com/renovate"><code>@​renovate</code></a></li>
<li><a href="https://github.com/j178"><code>@​j178</code></a></li>
</ul>
<h2>Install prek 0.3.10</h2>
<h3>Install prebuilt binaries via shell script</h3>
<pre lang="sh"><code>curl --proto '=https' --tlsv1.2 -LsSf
https://github.com/j178/prek/releases/download/v0.3.10/prek-installer.sh
| sh
</code></pre>
<h3>Install prebuilt binaries via powershell script</h3>
<pre lang="sh"><code>powershell -ExecutionPolicy Bypass -c &quot;irm
https://github.com/j178/prek/releases/download/v0.3.10/prek-installer.ps1
| iex&quot;
</code></pre>
<h3>Install prebuilt binaries via Homebrew</h3>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/j178/prek/blob/master/CHANGELOG.md">prek's
changelog</a>.</em></p>
<blockquote>
<h2>0.3.10</h2>
<p>Released on 2026-04-21.</p>
<h3>Enhancements</h3>
<ul>
<li>Disallow rev for non-remote repos in schema (<a
href="https://redirect.github.com/j178/prek/pull/1964">#1964</a>)</li>
<li>Hide up-to-date output in non-verbose mode (<a
href="https://redirect.github.com/j178/prek/pull/1942">#1942</a>)</li>
<li>Improve merge conflict marker detection (<a
href="https://redirect.github.com/j178/prek/pull/1937">#1937</a>)</li>
<li>Keep finished hooks visible (<a
href="https://redirect.github.com/j178/prek/pull/1967">#1967</a>)</li>
<li>Preserve frozen comment spacing in auto-update (<a
href="https://redirect.github.com/j178/prek/pull/1945">#1945</a>)</li>
<li>Reimplement <code>@j178/prek</code> npm package (<a
href="https://redirect.github.com/j178/prek/pull/1973">#1973</a>)</li>
</ul>
<h3>Bug fixes</h3>
<ul>
<li>Prefer stable Rust toolchains (<a
href="https://redirect.github.com/j178/prek/pull/1974">#1974</a>)</li>
</ul>
<h3>Documentation</h3>
<ul>
<li>Add <code>SKILL.md</code> for prek (<a
href="https://redirect.github.com/j178/prek/pull/1950">#1950</a>)</li>
<li>Document <code>gh skill install j178/prek prek</code> to install
prek skill for agents (<a
href="https://redirect.github.com/j178/prek/pull/1951">#1951</a>)</li>
<li>Improve compatibility and migration docs (<a
href="https://redirect.github.com/j178/prek/pull/1940">#1940</a>)</li>
</ul>
<h3>Other changes</h3>
<ul>
<li>Sync latest identify tags (<a
href="https://redirect.github.com/j178/prek/pull/1947">#1947</a>)</li>
</ul>
<h3>Contributors</h3>
<ul>
<li><a
href="https://github.com/github-actions"><code>@​github-actions</code></a></li>
<li><a
href="https://github.com/renovate"><code>@​renovate</code></a></li>
<li><a href="https://github.com/j178"><code>@​j178</code></a></li>
</ul>
<h2>0.3.9</h2>
<p>Released on 2026-04-13.</p>
<h3>Highlight</h3>
<p><code>prek auto-update</code> is now stricter about pinned revisions
and more useful in CI.
It now keeps <code>rev</code> and <code># frozen:</code> comments in
sync, can detect
<a href="https://docs.zizmor.sh/audits/#impostor-commit">impostor
commits</a> when validating pinned SHAs,
and lets you use <code>prek auto-update --check</code> to fail on both
available updates and frozen-ref
mismatches without rewriting the config.</p>
<p>Examples:</p>
<pre lang="console"><code>$ prek auto-update
# updates revs and repairs stale `# frozen:` comments
&lt;/tr&gt;&lt;/table&gt; 
</code></pre>
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/j178/prek/commit/1981c51f599ed9db4a428eae13e4a8baf0cb0427"><code>1981c51</code></a>
Bump version to 0.3.10</li>
<li><a
href="https://github.com/j178/prek/commit/ba745f6f407d43e70d2155c7929004a7701ca7e8"><code>ba745f6</code></a>
Prefer stable Rust toolchains (<a
href="https://redirect.github.com/j178/prek/issues/1974">#1974</a>)</li>
<li><a
href="https://github.com/j178/prek/commit/b6c591dbbc4d9917e62b4b6c1dd65419cffeb464"><code>b6c591d</code></a>
Reimplement <code>@j178/prek</code> npm package (<a
href="https://redirect.github.com/j178/prek/issues/1973">#1973</a>)</li>
<li><a
href="https://github.com/j178/prek/commit/fba1c85b1f2ce244e4b491ccaa0edffbdee01348"><code>fba1c85</code></a>
Keep finished hooks visible (<a
href="https://redirect.github.com/j178/prek/issues/1967">#1967</a>)</li>
<li><a
href="https://github.com/j178/prek/commit/bce57a3415d688a9b9d893018348570b381023e3"><code>bce57a3</code></a>
Disallow rev for non-remote repos in schema (<a
href="https://redirect.github.com/j178/prek/issues/1964">#1964</a>)</li>
<li><a
href="https://github.com/j178/prek/commit/afac10cf3ae355f6e2948c62ec25c54536bea40d"><code>afac10c</code></a>
Split <code>auto_update.rs</code> into modules (<a
href="https://redirect.github.com/j178/prek/issues/1962">#1962</a>)</li>
<li><a
href="https://github.com/j178/prek/commit/c3b9f115dabff92fde388c5edb091a10c00fb18f"><code>c3b9f11</code></a>
Update GitHub Actions (major) (<a
href="https://redirect.github.com/j178/prek/issues/1961">#1961</a>)</li>
<li><a
href="https://github.com/j178/prek/commit/6ca0443ecca17e8bc6df972740f445dd9795b4bd"><code>6ca0443</code></a>
Update Rust crate annotate-snippets to v0.12.15 (<a
href="https://redirect.github.com/j178/prek/issues/1955">#1955</a>)</li>
<li><a
href="https://github.com/j178/prek/commit/776277e104a751634711821c10f871a95b5bcd70"><code>776277e</code></a>
Update Rust crate tokio to v1.51.1 (<a
href="https://redirect.github.com/j178/prek/issues/1957">#1957</a>)</li>
<li><a
href="https://github.com/j178/prek/commit/c2b445a722db49f6b0dfeee763c19944bc77e757"><code>c2b445a</code></a>
Update Rust crate toml_edit to v0.25.11 (<a
href="https://redirect.github.com/j178/prek/issues/1958">#1958</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/j178/prek/compare/v0.3.8...v0.3.10">compare
view</a></li>
</ul>
</details>
<br />

Updates `pytest` from 9.0.2 to 9.0.3
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/pytest-dev/pytest/releases">pytest's
releases</a>.</em></p>
<blockquote>
<h2>9.0.3</h2>
<h1>pytest 9.0.3 (2026-04-07)</h1>
<h2>Bug fixes</h2>
<ul>
<li>
<p><a
href="https://redirect.github.com/pytest-dev/pytest/issues/12444">#12444</a>:
Fixed <code>pytest.approx</code> which now correctly takes into account
<code>~collections.abc.Mapping</code> keys order to compare them.</p>
</li>
<li>
<p><a
href="https://redirect.github.com/pytest-dev/pytest/issues/13634">#13634</a>:
Blocking a <code>conftest.py</code> file using the <code>-p no:</code>
option is now explicitly disallowed.</p>
<p>Previously this resulted in an internal assertion failure during
plugin loading.</p>
<p>Pytest now raises a clear <code>UsageError</code> explaining that
conftest files are not plugins and cannot be disabled via
<code>-p</code>.</p>
</li>
<li>
<p><a
href="https://redirect.github.com/pytest-dev/pytest/issues/13734">#13734</a>:
Fixed crash when a test raises an exceptiongroup with
<code>__tracebackhide__ = True</code>.</p>
</li>
<li>
<p><a
href="https://redirect.github.com/pytest-dev/pytest/issues/14195">#14195</a>:
Fixed an issue where non-string messages passed to <!-- raw HTML omitted
-->unittest.TestCase.subTest()<!-- raw HTML omitted --> were not
printed.</p>
</li>
<li>
<p><a
href="https://redirect.github.com/pytest-dev/pytest/issues/14343">#14343</a>:
Fixed use of insecure temporary directory (CVE-2025-71176).</p>
</li>
</ul>
<h2>Improved documentation</h2>
<ul>
<li><a
href="https://redirect.github.com/pytest-dev/pytest/issues/13388">#13388</a>:
Clarified documentation for <code>-p</code> vs
<code>PYTEST_PLUGINS</code> plugin loading and fixed an incorrect
<code>-p</code> example.</li>
<li><a
href="https://redirect.github.com/pytest-dev/pytest/issues/13731">#13731</a>:
Clarified that capture fixtures (e.g. <code>capsys</code> and
<code>capfd</code>) take precedence over the <code>-s</code> /
<code>--capture=no</code> command-line options in <code>Accessing
captured output from a test function
&lt;accessing-captured-output&gt;</code>.</li>
<li><a
href="https://redirect.github.com/pytest-dev/pytest/issues/14088">#14088</a>:
Clarified that the default <code>pytest_collection</code> hook sets
<code>session.items</code> before it calls
<code>pytest_collection_finish</code>, not after.</li>
<li><a
href="https://redirect.github.com/pytest-dev/pytest/issues/14255">#14255</a>:
TOML integer log levels must be quoted: Updating reference
documentation.</li>
</ul>
<h2>Contributor-facing changes</h2>
<ul>
<li>
<p><a
href="https://redirect.github.com/pytest-dev/pytest/issues/12689">#12689</a>:
The test reports are now published to Codecov from GitHub Actions.
The test statistics is visible <a
href="https://app.codecov.io/gh/pytest-dev/pytest/tests">on the web
interface</a>.</p>
<p>-- by <code>aleguy02</code></p>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/pytest-dev/pytest/commit/a7d58d7a21b78581e636bbbdea13c66ad1657c1e"><code>a7d58d7</code></a>
Prepare release version 9.0.3</li>
<li><a
href="https://github.com/pytest-dev/pytest/commit/089d98199c253d8f89a040243bc4f2aa6cd5ab22"><code>089d981</code></a>
Merge pull request <a
href="https://redirect.github.com/pytest-dev/pytest/issues/14366">#14366</a>
from bluetech/revert-14193-backport</li>
<li><a
href="https://github.com/pytest-dev/pytest/commit/8127eaf4ab7f6b2fdd0dc1b38343ec97aeef05ac"><code>8127eaf</code></a>
Revert &quot;Fix: assertrepr_compare respects dict insertion order (<a
href="https://redirect.github.com/pytest-dev/pytest/issues/14050">#14050</a>)
(<a
href="https://redirect.github.com/pytest-dev/pytest/issues/14193">#14193</a>)&quot;</li>
<li><a
href="https://github.com/pytest-dev/pytest/commit/99a7e6029e7a6e8d53e5df114b1346e035370241"><code>99a7e60</code></a>
Merge pull request <a
href="https://redirect.github.com/pytest-dev/pytest/issues/14363">#14363</a>
from pytest-dev/patchback/backports/9.0.x/95d8423bd...</li>
<li><a
href="https://github.com/pytest-dev/pytest/commit/ddee02a578da30dd43aedc39c1c1f1aaadfcee95"><code>ddee02a</code></a>
Merge pull request <a
href="https://redirect.github.com/pytest-dev/pytest/issues/14343">#14343</a>
from bluetech/cve-2025-71176-simple</li>
<li><a
href="https://github.com/pytest-dev/pytest/commit/74eac6916fee34726cb194f16c516e96fbd29619"><code>74eac69</code></a>
doc: Update training info (<a
href="https://redirect.github.com/pytest-dev/pytest/issues/14298">#14298</a>)
(<a
href="https://redirect.github.com/pytest-dev/pytest/issues/14301">#14301</a>)</li>
<li><a
href="https://github.com/pytest-dev/pytest/commit/f92dee777cfdb77d1c43633d02766ddf1f07c869"><code>f92dee7</code></a>
Merge pull request <a
href="https://redirect.github.com/pytest-dev/pytest/issues/14267">#14267</a>
from pytest-dev/patchback/backports/9.0.x/d6fa26c62...</li>
<li><a
href="https://github.com/pytest-dev/pytest/commit/7ee58acc8777c31ac6cf388d01addf5a414a7439"><code>7ee58ac</code></a>
Merge pull request <a
href="https://redirect.github.com/pytest-dev/pytest/issues/12378">#12378</a>
from Pierre-Sassoulas/fix-implicit-str-concat-and-d...</li>
<li><a
href="https://github.com/pytest-dev/pytest/commit/37da870d37e3a2f5177cae075c7b9ae279432bf8"><code>37da870</code></a>
Merge pull request <a
href="https://redirect.github.com/pytest-dev/pytest/issues/14259">#14259</a>
from mitre88/patch-4 (<a
href="https://redirect.github.com/pytest-dev/pytest/issues/14268">#14268</a>)</li>
<li><a
href="https://github.com/pytest-dev/pytest/commit/c34bfa3b7acb65b594707c714f1d8461b0304eed"><code>c34bfa3</code></a>
Add explanation for string context diffs (<a
href="https://redirect.github.com/pytest-dev/pytest/issues/14257">#14257</a>)
(<a
href="https://redirect.github.com/pytest-dev/pytest/issues/14266">#14266</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/pytest-dev/pytest/compare/9.0.2...9.0.3">compare
view</a></li>
</ul>
</details>
<br />

Updates `ruff` from 0.15.10 to 0.15.11
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/astral-sh/ruff/releases">ruff's
releases</a>.</em></p>
<blockquote>
<h2>0.15.11</h2>
<h2>Release Notes</h2>
<p>Released on 2026-04-16.</p>
<h3>Preview features</h3>
<ul>
<li>[<code>ruff</code>] Ignore <code>RUF029</code> when function is
decorated with <code>asynccontextmanager</code> (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24642">#24642</a>)</li>
<li>[<code>airflow</code>] Implement
<code>airflow-xcom-pull-in-template-string</code> (<code>AIR201</code>)
(<a
href="https://redirect.github.com/astral-sh/ruff/pull/23583">#23583</a>)</li>
<li>[<code>flake8-bandit</code>] Fix <code>S103</code> false positives
and negatives in mask analysis (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24424">#24424</a>)</li>
</ul>
<h3>Bug fixes</h3>
<ul>
<li>[<code>flake8-async</code>] Omit overridden methods for
<code>ASYNC109</code> (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24648">#24648</a>)</li>
</ul>
<h3>Documentation</h3>
<ul>
<li>[<code>flake8-async</code>] Add override mention to
<code>ASYNC109</code> docs (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24666">#24666</a>)</li>
<li>Update Neovim config examples to use <code>vim.lsp.config</code> (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24577">#24577</a>)</li>
</ul>
<h3>Contributors</h3>
<ul>
<li><a
href="https://github.com/augustelalande"><code>@​augustelalande</code></a></li>
<li><a
href="https://github.com/anishgirianish"><code>@​anishgirianish</code></a></li>
<li><a
href="https://github.com/benberryallwood"><code>@​benberryallwood</code></a></li>
<li><a
href="https://github.com/charliermarsh"><code>@​charliermarsh</code></a></li>
<li><a href="https://github.com/Dev-iL"><code>@​Dev-iL</code></a></li>
</ul>
<h2>Install ruff 0.15.11</h2>
<h3>Install prebuilt binaries via shell script</h3>
<pre lang="sh"><code>curl --proto '=https' --tlsv1.2 -LsSf
https://releases.astral.sh/github/ruff/releases/download/0.15.11/ruff-installer.sh
| sh
</code></pre>
<h3>Install prebuilt binaries via powershell script</h3>
<pre lang="sh"><code>powershell -ExecutionPolicy Bypass -c &quot;irm
https://releases.astral.sh/github/ruff/releases/download/0.15.11/ruff-installer.ps1
| iex&quot;
</code></pre>
<h2>Download ruff 0.15.11</h2>
<table>
<thead>
<tr>
<th>File</th>
<th>Platform</th>
<th>Checksum</th>
</tr>
</thead>
<tbody>
<tr>
<td><a
href="https://releases.astral.sh/github/ruff/releases/download/0.15.11/ruff-aarch64-apple-darwin.tar.gz">ruff-aarch64-apple-darwin.tar.gz</a></td>
<td>Apple Silicon macOS</td>
<td><a
href="https://releases.astral.sh/github/ruff/releases/download/0.15.11/ruff-aarch64-apple-darwin.tar.gz.sha256">checksum</a></td>
</tr>
<tr>
<td><a
href="https://releases.astral.sh/github/ruff/releases/download/0.15.11/ruff-x86_64-apple-darwin.tar.gz">ruff-x86_64-apple-darwin.tar.gz</a></td>
<td>Intel macOS</td>
<td><a
href="https://releases.astral.sh/github/ruff/releases/download/0.15.11/ruff-x86_64-apple-darwin.tar.gz.sha256">checksum</a></td>
</tr>
<tr>
<td><a
href="https://releases.astral.sh/github/ruff/releases/download/0.15.11/ruff-aarch64-pc-windows-msvc.zip">ruff-aarch64-pc-windows-msvc.zip</a></td>
<td>ARM64 Windows</td>
<td><a
href="https://releases.astral.sh/github/ruff/releases/download/0.15.11/ruff-aarch64-pc-windows-msvc.zip.sha256">checksum</a></td>
</tr>
<tr>
<td><a
href="https://releases.astral.sh/github/ruff/releases/download/0.15.11/ruff-i686-pc-windows-msvc.zip">ruff-i686-pc-windows-msvc.zip</a></td>
<td>x86 Windows</td>
<td><a
href="https://releases.astral.sh/github/ruff/releases/download/0.15.11/ruff-i686-pc-windows-msvc.zip.sha256">checksum</a></td>
</tr>
</tbody>
</table>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md">ruff's
changelog</a>.</em></p>
<blockquote>
<h2>0.15.11</h2>
<p>Released on 2026-04-16.</p>
<h3>Preview features</h3>
<ul>
<li>[<code>ruff</code>] Ignore <code>RUF029</code> when function is
decorated with <code>asynccontextmanager</code> (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24642">#24642</a>)</li>
<li>[<code>airflow</code>] Implement
<code>airflow-xcom-pull-in-template-string</code> (<code>AIR201</code>)
(<a
href="https://redirect.github.com/astral-sh/ruff/pull/23583">#23583</a>)</li>
<li>[<code>flake8-bandit</code>] Fix <code>S103</code> false positives
and negatives in mask analysis (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24424">#24424</a>)</li>
</ul>
<h3>Bug fixes</h3>
<ul>
<li>[<code>flake8-async</code>] Omit overridden methods for
<code>ASYNC109</code> (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24648">#24648</a>)</li>
</ul>
<h3>Documentation</h3>
<ul>
<li>[<code>flake8-async</code>] Add override mention to
<code>ASYNC109</code> docs (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24666">#24666</a>)</li>
<li>Update Neovim config examples to use <code>vim.lsp.config</code> (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24577">#24577</a>)</li>
</ul>
<h3>Contributors</h3>
<ul>
<li><a
href="https://github.com/augustelalande"><code>@​augustelalande</code></a></li>
<li><a
href="https://github.com/anishgirianish"><code>@​anishgirianish</code></a></li>
<li><a
href="https://github.com/benberryallwood"><code>@​benberryallwood</code></a></li>
<li><a
href="https://github.com/charliermarsh"><code>@​charliermarsh</code></a></li>
<li><a href="https://github.com/Dev-iL"><code>@​Dev-iL</code></a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/astral-sh/ruff/commit/53554b1cfe837f2eb992a81794480699478f1116"><code>53554b1</code></a>
Bump 0.15.11 (<a
href="https://redirect.github.com/astral-sh/ruff/issues/24678">#24678</a>)</li>
<li><a
href="https://github.com/astral-sh/ruff/commit/08c56c83cffbb1025cbf5bdede6c6d8be591cf47"><code>08c56c8</code></a>
Factor out the <code>mdtest</code> crate (<a
href="https://redirect.github.com/astral-sh/ruff/issues/24616">#24616</a>)</li>
<li><a
href="https://github.com/astral-sh/ruff/commit/725fbb736d2a999971449b61190b914abd26102a"><code>725fbb7</code></a>
[ty] Use partially qualified names when reporting diagnostics regarding
bad c...</li>
<li><a
href="https://github.com/astral-sh/ruff/commit/ddd6a30ff5fa27694dc1c50d0749885a1519d0a7"><code>ddd6a30</code></a>
[ty] Do not suggest argument completion when at value of keyword
argument (<a
href="https://redirect.github.com/astral-sh/ruff/issues/2">#2</a>...</li>
<li><a
href="https://github.com/astral-sh/ruff/commit/9282e61d482a36da08d66bb8271afeef50b3bc45"><code>9282e61</code></a>
Disallow <a
href="https://github.com/disjoint"><code>@​disjoint</code></a>_base on
TypedDicts and Protocols (<a
href="https://redirect.github.com/astral-sh/ruff/issues/24671">#24671</a>)</li>
<li><a
href="https://github.com/astral-sh/ruff/commit/e9986d8e3008eefe2e387312c4dc8b9c60f6f362"><code>e9986d8</code></a>
[ty] Reject using properties with <code>Never</code> setters or deleters
(<a
href="https://redirect.github.com/astral-sh/ruff/issues/24510">#24510</a>)</li>
<li><a
href="https://github.com/astral-sh/ruff/commit/9cf212ff82f7b66b4a275ad6a9b1564aee1fa4a8"><code>9cf212f</code></a>
[ty] Normalize property setter and deleter wrappers (<a
href="https://redirect.github.com/astral-sh/ruff/issues/24509">#24509</a>)</li>
<li><a
href="https://github.com/astral-sh/ruff/commit/12a1589de4d7120cf99441ee4c14871bdc20968d"><code>12a1589</code></a>
Add override mention to ASYNC109 docs (<a
href="https://redirect.github.com/astral-sh/ruff/issues/24666">#24666</a>)</li>
<li><a
href="https://github.com/astral-sh/ruff/commit/dccb03d010f4442ed60624f8d2ba932706abaabb"><code>dccb03d</code></a>
[ty] Avoid panicking on overloaded <code>Callable</code> type context
(<a
href="https://redirect.github.com/astral-sh/ruff/issues/24661">#24661</a>)</li>
<li><a
href="https://github.com/astral-sh/ruff/commit/61f9a0a5763fb068cd2f26c0ee9d63a277fb26c2"><code>61f9a0a</code></a>
[ty] Sync vendored typeshed stubs (<a
href="https://redirect.github.com/astral-sh/ruff/issues/24646">#24646</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/astral-sh/ruff/compare/0.15.10...0.15.11">compare
view</a></li>
</ul>
</details>
<br />

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-25 13:12:35 +01:00
dependabot[bot] 795887ae45 Bump the npm-deps group in /src/frontend with 8 updates (#914)
Bumps the npm-deps group in /src/frontend with 8 updates:

| Package | From | To |
| --- | --- | --- |
|
[@tailwindcss/vite](https://github.com/tailwindlabs/tailwindcss/tree/HEAD/packages/@tailwindcss-vite)
| `4.2.2` | `4.2.4` |
|
[react-router-dom](https://github.com/remix-run/react-router/tree/HEAD/packages/react-router-dom)
| `7.14.1` | `7.14.2` |
| [knip](https://github.com/webpro-nl/knip/tree/HEAD/packages/knip) |
`6.4.1` | `6.6.2` |
| [oxfmt](https://github.com/oxc-project/oxc/tree/HEAD/npm/oxfmt) |
`0.45.0` | `0.46.0` |
| [oxlint](https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint) |
`1.60.0` | `1.61.0` |
|
[tailwindcss](https://github.com/tailwindlabs/tailwindcss/tree/HEAD/packages/tailwindcss)
| `4.2.2` | `4.2.4` |
| [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) |
`8.0.8` | `8.0.10` |
|
[vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest)
| `4.1.4` | `4.1.5` |

Updates `@tailwindcss/vite` from 4.2.2 to 4.2.4
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/tailwindlabs/tailwindcss/releases"><code>@​tailwindcss/vite</code>'s
releases</a>.</em></p>
<blockquote>
<h2>v4.2.4</h2>
<h3>Fixed</h3>
<ul>
<li>Ensure imports in <code>@import</code> and <code>@plugin</code>
still resolve correctly when using Vite aliases in
<code>@tailwindcss/vite</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19947">#19947</a>)</li>
</ul>
<h2>v4.2.3</h2>
<h3>Fixed</h3>
<ul>
<li>Canonicalization: improve canonicalizations for
<code>tracking-*</code> utilities by preferring non-negative utilities
(e.g. <code>-tracking-tighter</code> → <code>tracking-wider</code>) (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19827">#19827</a>)</li>
<li>Fix crash due to invalid characters in candidate (exceeding valid
unicode code point range) (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19829">#19829</a>)</li>
<li>Ensure query params in imports are considered unique resources when
using <code>@tailwindcss/webpack</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19723">#19723</a>)</li>
<li>Canonicalization: collapse arbitrary values into shorthand utilities
(e.g. <code>px-[1.2rem] py-[1.2rem]</code> → <code>p-[1.2rem]</code>)
(<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19837">#19837</a>)</li>
<li>Canonicalization: collapse <code>border-{t,b}-*</code> into
<code>border-y-*</code>, <code>border-{l,r}-*</code> into
<code>border-x-*</code>, and <code>border-{t,r,b,l}-*</code> into
<code>border-*</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19842">#19842</a>)</li>
<li>Canonicalization: collapse <code>scroll-m{t,b}-*</code> into
<code>scroll-my-*</code>, <code>scroll-m{l,r}-*</code> into
<code>scroll-mx-*</code>, and <code>scroll-m{t,r,b,l}-*</code> into
<code>scroll-m-*</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19842">#19842</a>)</li>
<li>Canonicalization: collapse <code>scroll-p{t,b}-*</code> into
<code>scroll-py-*</code>, <code>scroll-p{l,r}-*</code> into
<code>scroll-px-*</code>, and <code>scroll-p{t,r,b,l}-*</code> into
<code>scroll-p-*</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19842">#19842</a>)</li>
<li>Canonicalization: collapse <code>overflow-{x,y}-*</code> into
<code>overflow-*</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19842">#19842</a>)</li>
<li>Canonicalization: collapse <code>overscroll-{x,y}-*</code> into
<code>overscroll-*</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19842">#19842</a>)</li>
<li>Read from <code>--placeholder-color</code> instead of
<code>--background-color</code> for <code>placeholder-*</code> utilities
(<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19843">#19843</a>)</li>
<li>Upgrade: ensure files are not emptied out when killing the upgrade
process while it's running (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19846">#19846</a>)</li>
<li>Upgrade: use <code>config.content</code> when migrating from
Tailwind CSS v3 to Tailwind CSS v4 (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19846">#19846</a>)</li>
<li>Upgrade: never migrate files that are ignored by git (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19846">#19846</a>)</li>
<li>Add <code>.env</code> and <code>.env.*</code> to default ignored
content files (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19846">#19846</a>)</li>
<li>Canonicalization: migrate <code>overflow-ellipsis</code> into
<code>text-ellipsis</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19849">#19849</a>)</li>
<li>Canonicalization: migrate <code>start-full</code> →
<code>inset-s-full</code>, <code>start-auto</code> →
<code>inset-s-auto</code>, <code>start-px</code> →
<code>inset-s-px</code>, and <code>start-&lt;number&gt;</code> →
<code>inset-s-&lt;number&gt;</code> as well as negative versions (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19849">#19849</a>)</li>
<li>Canonicalization: migrate <code>end-full</code> →
<code>inset-e-full</code>, <code>end-auto</code> →
<code>inset-e-auto</code>, <code>end-px</code> →
<code>inset-e-px</code>, and <code>end-&lt;number&gt;</code> →
<code>inset-e-&lt;number&gt;</code> as well as negative versions (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19849">#19849</a>)</li>
<li>Canonicalization: move the <code>-</code> sign inside the arbitrary
value <code>-left-[9rem]</code> → <code>left-[-9rem]</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19858">#19858</a>)</li>
<li>Canonicalization: move the <code>-</code> sign outside the arbitrary
value <code>ml-[calc(-1*var(--width))]</code> →
<code>-ml-(--width)</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19858">#19858</a>)</li>
<li>Improve performance when scanning JSONL / NDJSON files (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19862">#19862</a>)</li>
<li>Support <code>NODE_PATH</code> environment variable in standalone
CLI (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19617">#19617</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/tailwindlabs/tailwindcss/blob/main/CHANGELOG.md"><code>@​tailwindcss/vite</code>'s
changelog</a>.</em></p>
<blockquote>
<h2>[4.2.4] - 2026-04-21</h2>
<h3>Fixed</h3>
<ul>
<li>Ensure imports in <code>@import</code> and <code>@plugin</code>
still resolve correctly when using Vite aliases in
<code>@tailwindcss/vite</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19947">#19947</a>)</li>
</ul>
<h2>[4.2.3] - 2026-04-20</h2>
<h3>Fixed</h3>
<ul>
<li>Canonicalization: improve canonicalizations for
<code>tracking-*</code> utilities by preferring non-negative utilities
(e.g. <code>-tracking-tighter</code> → <code>tracking-wider</code>) (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19827">#19827</a>)</li>
<li>Fix crash due to invalid characters in candidate (exceeding valid
unicode code point range) (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19829">#19829</a>)</li>
<li>Ensure query params in imports are considered unique resources when
using <code>@tailwindcss/webpack</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19723">#19723</a>)</li>
<li>Canonicalization: collapse arbitrary values into shorthand utilities
(e.g. <code>px-[1.2rem] py-[1.2rem]</code> → <code>p-[1.2rem]</code>)
(<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19837">#19837</a>)</li>
<li>Canonicalization: collapse <code>border-{t,b}-*</code> into
<code>border-y-*</code>, <code>border-{l,r}-*</code> into
<code>border-x-*</code>, and <code>border-{t,r,b,l}-*</code> into
<code>border-*</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19842">#19842</a>)</li>
<li>Canonicalization: collapse <code>scroll-m{t,b}-*</code> into
<code>scroll-my-*</code>, <code>scroll-m{l,r}-*</code> into
<code>scroll-mx-*</code>, and <code>scroll-m{t,r,b,l}-*</code> into
<code>scroll-m-*</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19842">#19842</a>)</li>
<li>Canonicalization: collapse <code>scroll-p{t,b}-*</code> into
<code>scroll-py-*</code>, <code>scroll-p{l,r}-*</code> into
<code>scroll-px-*</code>, and <code>scroll-p{t,r,b,l}-*</code> into
<code>scroll-p-*</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19842">#19842</a>)</li>
<li>Canonicalization: collapse <code>overflow-{x,y}-*</code> into
<code>overflow-*</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19842">#19842</a>)</li>
<li>Canonicalization: collapse <code>overscroll-{x,y}-*</code> into
<code>overscroll-*</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19842">#19842</a>)</li>
<li>Read from <code>--placeholder-color</code> instead of
<code>--background-color</code> for <code>placeholder-*</code> utilities
(<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19843">#19843</a>)</li>
<li>Upgrade: ensure files are not emptied out when killing the upgrade
process while it's running (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19846">#19846</a>)</li>
<li>Upgrade: use <code>config.content</code> when migrating from
Tailwind CSS v3 to Tailwind CSS v4 (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19846">#19846</a>)</li>
<li>Upgrade: never migrate files that are ignored by git (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19846">#19846</a>)</li>
<li>Add <code>.env</code> and <code>.env.*</code> to default ignored
content files (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19846">#19846</a>)</li>
<li>Canonicalization: migrate <code>overflow-ellipsis</code> into
<code>text-ellipsis</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19849">#19849</a>)</li>
<li>Canonicalization: migrate <code>start-full</code> →
<code>inset-s-full</code>, <code>start-auto</code> →
<code>inset-s-auto</code>, <code>start-px</code> →
<code>inset-s-px</code>, and <code>start-&lt;number&gt;</code> →
<code>inset-s-&lt;number&gt;</code> as well as negative versions (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19849">#19849</a>)</li>
<li>Canonicalization: migrate <code>end-full</code> →
<code>inset-e-full</code>, <code>end-auto</code> →
<code>inset-e-auto</code>, <code>end-px</code> →
<code>inset-e-px</code>, and <code>end-&lt;number&gt;</code> →
<code>inset-e-&lt;number&gt;</code> as well as negative versions (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19849">#19849</a>)</li>
<li>Canonicalization: move the <code>-</code> sign inside the arbitrary
value <code>-left-[9rem]</code> → <code>left-[-9rem]</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19858">#19858</a>)</li>
<li>Canonicalization: move the <code>-</code> sign outside the arbitrary
value <code>ml-[calc(-1*var(--width))]</code> →
<code>-ml-(--width)</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19858">#19858</a>)</li>
<li>Improve performance when scanning JSONL / NDJSON files (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19862">#19862</a>)</li>
<li>Support <code>NODE_PATH</code> environment variable in standalone
CLI (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19617">#19617</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/tailwindlabs/tailwindcss/commit/69ad7cc5ec091125842aba783c5bfce536e9c234"><code>69ad7cc</code></a>
4.2.4 (<a
href="https://github.com/tailwindlabs/tailwindcss/tree/HEAD/packages/@tailwindcss-vite/issues/19948">#19948</a>)</li>
<li><a
href="https://github.com/tailwindlabs/tailwindcss/commit/685c19e266a6b75842747acab6fe7076e82df62d"><code>685c19e</code></a>
Fix issue around resolving paths in <code>@tailwindcss/vite</code> (<a
href="https://github.com/tailwindlabs/tailwindcss/tree/HEAD/packages/@tailwindcss-vite/issues/19947">#19947</a>)</li>
<li><a
href="https://github.com/tailwindlabs/tailwindcss/commit/2e3fa490a535a8341fd2853d75098f0d4dbcc825"><code>2e3fa49</code></a>
4.2.3 (<a
href="https://github.com/tailwindlabs/tailwindcss/tree/HEAD/packages/@tailwindcss-vite/issues/19944">#19944</a>)</li>
<li><a
href="https://github.com/tailwindlabs/tailwindcss/commit/5cb1efdf416a6d37012d31cc1306c41cb35bbdc1"><code>5cb1efd</code></a>
fix(vite): resolve tsconfig paths in CSS and JS resolvers (<a
href="https://github.com/tailwindlabs/tailwindcss/tree/HEAD/packages/@tailwindcss-vite/issues/19803">#19803</a>)</li>
<li>See full diff in <a
href="https://github.com/tailwindlabs/tailwindcss/commits/v4.2.4/packages/@tailwindcss-vite">compare
view</a></li>
</ul>
</details>
<details>
<summary>Maintainer changes</summary>
<p>This version was pushed to npm by <a
href="https://www.npmjs.com/~GitHub%20Actions">GitHub Actions</a>, a new
releaser for <code>@​tailwindcss/vite</code> since your current
version.</p>
</details>
<br />

Updates `react-router-dom` from 7.14.1 to 7.14.2
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/remix-run/react-router/blob/main/packages/react-router-dom/CHANGELOG.md">react-router-dom's
changelog</a>.</em></p>
<blockquote>
<h2>v7.14.2</h2>
<h3>Patch Changes</h3>
<ul>
<li>Updated dependencies:
<ul>
<li><a
href="https://github.com/remix-run/react-router/releases/tag/react-router@7.14.2"><code>react-router@7.14.2</code></a></li>
</ul>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/remix-run/react-router/commit/cf1d25003aa1217dc21c16e95d483601940ae9af"><code>cf1d250</code></a>
Release v7.14.2 (<a
href="https://github.com/remix-run/react-router/tree/HEAD/packages/react-router-dom/issues/14993">#14993</a>)</li>
<li>See full diff in <a
href="https://github.com/remix-run/react-router/commits/react-router-dom@7.14.2/packages/react-router-dom">compare
view</a></li>
</ul>
</details>
<br />

Updates `knip` from 6.4.1 to 6.6.2
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/webpro-nl/knip/releases">knip's
releases</a>.</em></p>
<blockquote>
<h2>Release 6.6.2</h2>
<ul>
<li>Don't track typeof value refs at top of exported type alias (resolve
<a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1697">#1697</a>)
(1a9904832878f5e8e4b633bdb08fb327baa17ed6)</li>
<li>Treat Props as used when Astro.props is referenced (resolve <a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1629">#1629</a>)
(46cb33871b236249cf9b09679988d530a61a545f)</li>
<li>Update dependencies (dcf53157452b996f970d91d3149c4b7bb4f45edd)</li>
</ul>
<h2>Release 6.6.1</h2>
<ul>
<li>Fix website papercuts (<a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1696">#1696</a>)
(c6d9c436d4c0b85f8aae34bab1197e3c2b5eb139) - thanks <a
href="https://github.com/skoeva"><code>@​skoeva</code></a>!</li>
<li>fix(rslib): resolve entry points from rslib config (<a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1695">#1695</a>)
(af83d68d9856b3666af5111338f25dc2b4897cc7) - thanks <a
href="https://github.com/rpereira-anchor"><code>@​rpereira-anchor</code></a>!</li>
<li>Update react-email plugin for react-email@6 (<a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1694">#1694</a>)
(200db55064811eba4a168598615417d36d77e3aa) - thanks <a
href="https://github.com/krystofspiller"><code>@​krystofspiller</code></a>!</li>
<li>Introduce Manifest wrapper with scriptNames + getMajor helpers
(6fded2f00f4e9cf8328231f9e30280d62c85e95d)</li>
</ul>
<h2>Release 6.6.0</h2>
<ul>
<li>picocolors → styleText
(e7abd6bb1a42c8997ae79555e55b7d9d19f56a48)</li>
<li><code>@​nodelib/fs</code>.walk → fdir
(6f781b84f9776d211c6fd1b0d9b280457701051a)</li>
<li>Improve performance, but only slightly
(c2c0323efd1a327936c4cdd65d335c913eb0e0ef)</li>
<li>Fix package-entry hint for bare specifier in main (resolve <a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1693">#1693</a>)
(2e8cf477dada77b93c54a3717d4c21f3107f5f23)</li>
<li>Extract vite.resolve.alias from Astro config AST (resolve <a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1692">#1692</a>)
(fab587198659cfc095e97699cf3a751034aa2160)</li>
<li>Add plugin-contributed source-map rules
(e11c2b1c02c7bf390f791e675fcdc3c7ec57a007)</li>
<li>Add <code>@sveltejs/package</code> plugin (resolve <a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1690">#1690</a>)
(8b2d402c9fff1a18e13512bfc08b5c3f9af891df)</li>
<li>Windows ❤️ (91964e428b44750ed0d03e3acf1560ba3ec1c202)</li>
</ul>
<h2>Release 6.5.0</h2>
<ul>
<li>Drop project-extension-redundant config hint (resolve <a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1683">#1683</a>)
(f86092949d6dbd041fd621876da674fd3eac7534)</li>
<li>Add instructions to .agents/PLUGINS.md
(e2943ed8fb6d2c0ab4cf12ff04d10bd5ab9fd5f7)</li>
<li>Add a plugin for react-email (<a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1684">#1684</a>)
(d8ae4d3ccb810b9a9211fd43e9d1e7d7b704fcaf) - thanks <a
href="https://github.com/xaqrox"><code>@​xaqrox</code></a>!</li>
<li>Replace <code>fast-glob</code> with <code>tinyglobby</code> (<a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1462">#1462</a>)
(9f6b4c8aa6857fea40e66d008c905c948af10939) - thanks <a
href="https://github.com/gameroman"><code>@​gameroman</code></a>!</li>
<li>Added plugin for Serverless framework (<a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1478">#1478</a>)
(f885f1ded52427d984c67e7172e3141eb4e5ee29) - thanks <a
href="https://github.com/BenCrinion-IW"><code>@​BenCrinion-IW</code></a>!</li>
<li>Add args to Prettier plugin to resolve --config CLI flag (<a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1685">#1685</a>)
(f4658c84e714afd8a2233b72be6169da1ac3723e) - thanks <a
href="https://github.com/xaqrox"><code>@​xaqrox</code></a>!</li>
<li>fix(util): tolerate JSONC and array extends in findRootDirsBase (<a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1681">#1681</a>)
(f7e5464a6f70e9024a341fdb923766b5ff53a831) - thanks <a
href="https://github.com/Hoffs"><code>@​Hoffs</code></a>!</li>
<li>Format (15bd7e7a56b470096cfee1690263d5d13ddb8fb6)</li>
<li>Inherit outDir/rootDir from tsconfig project references (resolve <a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1680">#1680</a>)
(b89b4f716f4c117b0106e9a212e9e5c46aa85035)</li>
<li>tsc → tsgo (a6e09ca1b65936b3790a6c3628a4646f030d18b8)</li>
<li>Update dependencies (4cb05c96a2ae7c790d29ee76ff13288c5bbb97a2)</li>
<li>Housekeep (28c56cb8bf68031b1c8e9bee75b18ee7274ec981)</li>
<li>Test test test (7eb4ab3a8a6635dcaf756bb2b4a88e1298615994)</li>
<li>Add pino plugin with <code>transportCall</code> visitor (resolve <a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1480">#1480</a>)
(53a033e4ddc5036c6a4a0e55c0abc42c5c64e4f5)</li>
<li>Add signal to projects using knip
(dbedd665c1f8d735030600d3f68ef1825d9a2668)</li>
<li>Tune logos (3148f4d0485875370634b9b53c1b3aa7f6eafcc3)</li>
<li>Tweak <code>knip-run</code> tool response
(42940381a947c46996ae4055e8789f6ea39cfca4)</li>
<li>Add <code>workspace</code> option to <code>knip-run</code> tool
(64c4aaea89e3abb41d8695ebffc5538878520b21)</li>
<li>Remove old lingering <code>experimentalTags</code>
(e503d108e5535800ac6467f4d92c7dd6a9e90037)</li>
<li>Fix compiler type (resolve <a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1689">#1689</a>)
(e7a69adb5e584eb6e5af9b4007820afcbcf27a08)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/webpro-nl/knip/commit/2527ddddc25355407e298485435e7baf42998c94"><code>2527ddd</code></a>
Release knip@6.6.2</li>
<li><a
href="https://github.com/webpro-nl/knip/commit/dcf53157452b996f970d91d3149c4b7bb4f45edd"><code>dcf5315</code></a>
Update dependencies</li>
<li><a
href="https://github.com/webpro-nl/knip/commit/46cb33871b236249cf9b09679988d530a61a545f"><code>46cb338</code></a>
Treat Props as used when Astro.props is referenced (resolve <a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1629">#1629</a>)</li>
<li><a
href="https://github.com/webpro-nl/knip/commit/1a9904832878f5e8e4b633bdb08fb327baa17ed6"><code>1a99048</code></a>
Don't track typeof value refs at top of exported type alias (resolve <a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1697">#1697</a>)</li>
<li><a
href="https://github.com/webpro-nl/knip/commit/2b1b2d4ae4ca67ab42a5324deee463b61d3989f4"><code>2b1b2d4</code></a>
Release knip@6.6.1</li>
<li><a
href="https://github.com/webpro-nl/knip/commit/6fded2f00f4e9cf8328231f9e30280d62c85e95d"><code>6fded2f</code></a>
Introduce Manifest wrapper with scriptNames + getMajor helpers</li>
<li><a
href="https://github.com/webpro-nl/knip/commit/200db55064811eba4a168598615417d36d77e3aa"><code>200db55</code></a>
Update react-email plugin for react-email@6 (<a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1694">#1694</a>)</li>
<li><a
href="https://github.com/webpro-nl/knip/commit/af83d68d9856b3666af5111338f25dc2b4897cc7"><code>af83d68</code></a>
fix(rslib): resolve entry points from rslib config (<a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1695">#1695</a>)</li>
<li><a
href="https://github.com/webpro-nl/knip/commit/38a17d22c1bdc080fe38227dfbde04b902865e78"><code>38a17d2</code></a>
Release knip@6.6.0</li>
<li><a
href="https://github.com/webpro-nl/knip/commit/91964e428b44750ed0d03e3acf1560ba3ec1c202"><code>91964e4</code></a>
Windows ❤️</li>
<li>Additional commits viewable in <a
href="https://github.com/webpro-nl/knip/commits/knip@6.6.2/packages/knip">compare
view</a></li>
</ul>
</details>
<br />

Updates `oxfmt` from 0.45.0 to 0.46.0
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/oxc-project/oxc/blob/main/npm/oxfmt/CHANGELOG.md">oxfmt's
changelog</a>.</em></p>
<blockquote>
<h1>Changelog</h1>
<p>All notable changes to this package will be documented in this
file.</p>
<p>The format is based on <a
href="https://keepachangelog.com/en/1.0.0">Keep a Changelog</a>.</p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/oxc-project/oxc/commit/f5bd1dd35f220976fb04b815b2a1d0dfbea82ccf"><code>f5bd1dd</code></a>
release(apps): oxlint v1.61.0 &amp;&amp; oxfmt v0.46.0 (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxfmt/issues/21583">#21583</a>)</li>
<li>See full diff in <a
href="https://github.com/oxc-project/oxc/commits/oxfmt_v0.46.0/npm/oxfmt">compare
view</a></li>
</ul>
</details>
<br />

Updates `oxlint` from 1.60.0 to 1.61.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/oxc-project/oxc/releases">oxlint's
releases</a>.</em></p>
<blockquote>
<h2>oxlint v1.27.0 &amp;&amp; oxfmt v0.12.0</h2>
<h1>Oxlint v1.27.0</h1>
<h3>🚀 Features</h3>
<ul>
<li>222a8f0 linter/plugins: Implement
<code>SourceCode#isSpaceBetween</code> (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15498">#15498</a>)
(overlookmotel)</li>
<li>2f9735d linter/plugins: Implement
<code>context.languageOptions</code> (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15486">#15486</a>)
(overlookmotel)</li>
<li>bc731ff linter/plugins: Stub out all <code>Context</code> APIs (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15479">#15479</a>)
(overlookmotel)</li>
<li>5822cb4 linter/plugins: Add <code>extend</code> method to
<code>FILE_CONTEXT</code> (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15477">#15477</a>)
(overlookmotel)</li>
<li>7b1e6f3 apps: Add pure rust binaries and release to github (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15469">#15469</a>)
(Boshen)</li>
<li>2a89b43 linter: Introduce debug assertions after fixes to assert
validity (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15389">#15389</a>)
(camc314)</li>
<li>ad3c45a editor: Add <code>oxc.path.node</code> option (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15040">#15040</a>)
(Sysix)</li>
</ul>
<h3>🐛 Bug Fixes</h3>
<ul>
<li>6f3cd77 linter/no-var: Incorrect warning for blocks (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15504">#15504</a>)
(Hamir Mahal)</li>
<li>6957fb9 linter/plugins: Do not allow access to
<code>Context#id</code> in <code>createOnce</code> (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15489">#15489</a>)
(overlookmotel)</li>
<li>7409630 linter/plugins: Allow access to <code>cwd</code> in
<code>createOnce</code> in ESLint interop mode (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15488">#15488</a>)
(overlookmotel)</li>
<li>732205e parser: Reject <code>using</code> / <code>await using</code>
in a switch <code>case</code> / <code>default</code> clause (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15225">#15225</a>)
(sapphi-red)</li>
<li>a17ca32 linter/plugins: Replace <code>Context</code> class (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15448">#15448</a>)
(overlookmotel)</li>
<li>ecf2f7b language_server: Fail gracefully when tsgolint executable
not found (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15436">#15436</a>)
(camc314)</li>
<li>3c8d3a7 lang-server: Improve logging in failure case for tsgolint
(<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15299">#15299</a>)
(camc314)</li>
<li>ef71410 linter: Use jsx if source type is JS in fix debug assertion
(<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15434">#15434</a>)
(camc314)</li>
<li>e32bbf6 linter/no-var: Handle TypeScript declare keyword in fixer
(<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15426">#15426</a>)
(camc314)</li>
<li>6565dbe linter/switch-case-braces: Skip comments when searching for
<code>:</code> token (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15425">#15425</a>)
(camc314)</li>
<li>85bd19a linter/prefer-class-fields: Insert value after type
annotation in fixer (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15423">#15423</a>)
(camc314)</li>
<li>fde753e linter/plugins: Block access to
<code>context.settings</code> in <code>createOnce</code> (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15394">#15394</a>)
(overlookmotel)</li>
<li>ddd9f9f linter/forward-ref-uses-ref: Dont suggest removing wrapper
in invalid positions (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15388">#15388</a>)
(camc314)</li>
<li>dac2a9c linter/no-template-curly-in-string: Remove fixer (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15387">#15387</a>)
(camc314)</li>
<li>989b8e3 linter/no-var: Only fix to <code>const</code> if the var has
an initializer (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15385">#15385</a>)
(camc314)</li>
<li>cc403f5 linter/plugins: Return empty object for unimplemented
parserServices (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15364">#15364</a>)
(magic-akari)</li>
</ul>
<h3>⚡ Performance</h3>
<ul>
<li>25d577e language_server: Start tools in parallel (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15500">#15500</a>)
(Sysix)</li>
<li>3c57291 linter/plugins: Optimize loops (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15449">#15449</a>)
(overlookmotel)</li>
<li>3166233 linter/plugins: Remove <code>Arc</code>s (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15431">#15431</a>)
(overlookmotel)</li>
<li>9de1322 linter/plugins: Lazily deserialize settings JSON (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15395">#15395</a>)
(overlookmotel)</li>
<li>3049ec2 linter/plugins: Optimize <code>deepFreezeSettings</code> (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15392">#15392</a>)
(overlookmotel)</li>
<li>444ebfd linter/plugins: Use single object for
<code>parserServices</code> (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15378">#15378</a>)
(overlookmotel)</li>
</ul>
<h3>📚 Documentation</h3>
<ul>
<li>97d2104 linter: Update comment in lint.rs about default value for
tsconfig path (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15530">#15530</a>)
(Connor Shea)</li>
<li>2c6bd9e linter: Always refer as &quot;ES2015&quot; instead of
&quot;ES6&quot; (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15411">#15411</a>)
(sapphi-red)</li>
<li>a0c5203 linter/import/named: Update &quot;ES7&quot; comment in
examples (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15410">#15410</a>)
(sapphi-red)</li>
<li>3dc24b5 linter,minifier: Always refer as &quot;ES Modules&quot;
instead of &quot;ES6 Modules&quot; (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15409">#15409</a>)
(sapphi-red)</li>
<li>2ad77fb linter/no-this-before-super: Correct &quot;Why is this
bad?&quot; section (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15408">#15408</a>)
(sapphi-red)</li>
<li>57f0ce1 linter: Add backquotes where appropriate (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15407">#15407</a>)
(sapphi-red)</li>
</ul>
<h1>Oxfmt v0.12.0</h1>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/oxc-project/oxc/blob/main/npm/oxlint/CHANGELOG.md">oxlint's
changelog</a>.</em></p>
<blockquote>
<h2>[1.61.0] - 2026-04-20</h2>
<h3>🚀 Features</h3>
<ul>
<li>38d8090 linter/jest: Implemented jest <code>version</code> settings
in config file. (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/21522">#21522</a>)
(Said Atrahouch)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/oxc-project/oxc/commit/f5bd1dd35f220976fb04b815b2a1d0dfbea82ccf"><code>f5bd1dd</code></a>
release(apps): oxlint v1.61.0 &amp;&amp; oxfmt v0.46.0 (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/21583">#21583</a>)</li>
<li><a
href="https://github.com/oxc-project/oxc/commit/38d809096e4fa1e6a46cf326a08845348b8b7d27"><code>38d8090</code></a>
feat(linter/jest): Implemented jest <code>version</code> settings in
config file. (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/21522">#21522</a>)</li>
<li>See full diff in <a
href="https://github.com/oxc-project/oxc/commits/oxlint_v1.61.0/npm/oxlint">compare
view</a></li>
</ul>
</details>
<br />

Updates `tailwindcss` from 4.2.2 to 4.2.4
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/tailwindlabs/tailwindcss/releases">tailwindcss's
releases</a>.</em></p>
<blockquote>
<h2>v4.2.4</h2>
<h3>Fixed</h3>
<ul>
<li>Ensure imports in <code>@import</code> and <code>@plugin</code>
still resolve correctly when using Vite aliases in
<code>@tailwindcss/vite</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19947">#19947</a>)</li>
</ul>
<h2>v4.2.3</h2>
<h3>Fixed</h3>
<ul>
<li>Canonicalization: improve canonicalizations for
<code>tracking-*</code> utilities by preferring non-negative utilities
(e.g. <code>-tracking-tighter</code> → <code>tracking-wider</code>) (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19827">#19827</a>)</li>
<li>Fix crash due to invalid characters in candidate (exceeding valid
unicode code point range) (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19829">#19829</a>)</li>
<li>Ensure query params in imports are considered unique resources when
using <code>@tailwindcss/webpack</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19723">#19723</a>)</li>
<li>Canonicalization: collapse arbitrary values into shorthand utilities
(e.g. <code>px-[1.2rem] py-[1.2rem]</code> → <code>p-[1.2rem]</code>)
(<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19837">#19837</a>)</li>
<li>Canonicalization: collapse <code>border-{t,b}-*</code> into
<code>border-y-*</code>, <code>border-{l,r}-*</code> into
<code>border-x-*</code>, and <code>border-{t,r,b,l}-*</code> into
<code>border-*</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19842">#19842</a>)</li>
<li>Canonicalization: collapse <code>scroll-m{t,b}-*</code> into
<code>scroll-my-*</code>, <code>scroll-m{l,r}-*</code> into
<code>scroll-mx-*</code>, and <code>scroll-m{t,r,b,l}-*</code> into
<code>scroll-m-*</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19842">#19842</a>)</li>
<li>Canonicalization: collapse <code>scroll-p{t,b}-*</code> into
<code>scroll-py-*</code>, <code>scroll-p{l,r}-*</code> into
<code>scroll-px-*</code>, and <code>scroll-p{t,r,b,l}-*</code> into
<code>scroll-p-*</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19842">#19842</a>)</li>
<li>Canonicalization: collapse <code>overflow-{x,y}-*</code> into
<code>overflow-*</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19842">#19842</a>)</li>
<li>Canonicalization: collapse <code>overscroll-{x,y}-*</code> into
<code>overscroll-*</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19842">#19842</a>)</li>
<li>Read from <code>--placeholder-color</code> instead of
<code>--background-color</code> for <code>placeholder-*</code> utilities
(<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19843">#19843</a>)</li>
<li>Upgrade: ensure files are not emptied out when killing the upgrade
process while it's running (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19846">#19846</a>)</li>
<li>Upgrade: use <code>config.content</code> when migrating from
Tailwind CSS v3 to Tailwind CSS v4 (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19846">#19846</a>)</li>
<li>Upgrade: never migrate files that are ignored by git (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19846">#19846</a>)</li>
<li>Add <code>.env</code> and <code>.env.*</code> to default ignored
content files (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19846">#19846</a>)</li>
<li>Canonicalization: migrate <code>overflow-ellipsis</code> into
<code>text-ellipsis</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19849">#19849</a>)</li>
<li>Canonicalization: migrate <code>start-full</code> →
<code>inset-s-full</code>, <code>start-auto</code> →
<code>inset-s-auto</code>, <code>start-px</code> →
<code>inset-s-px</code>, and <code>start-&lt;number&gt;</code> →
<code>inset-s-&lt;number&gt;</code> as well as negative versions (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19849">#19849</a>)</li>
<li>Canonicalization: migrate <code>end-full</code> →
<code>inset-e-full</code>, <code>end-auto</code> →
<code>inset-e-auto</code>, <code>end-px</code> →
<code>inset-e-px</code>, and <code>end-&lt;number&gt;</code> →
<code>inset-e-&lt;number&gt;</code> as well as negative versions (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19849">#19849</a>)</li>
<li>Canonicalization: move the <code>-</code> sign inside the arbitrary
value <code>-left-[9rem]</code> → <code>left-[-9rem]</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19858">#19858</a>)</li>
<li>Canonicalization: move the <code>-</code> sign outside the arbitrary
value <code>ml-[calc(-1*var(--width))]</code> →
<code>-ml-(--width)</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19858">#19858</a>)</li>
<li>Improve performance when scanning JSONL / NDJSON files (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19862">#19862</a>)</li>
<li>Support <code>NODE_PATH</code> environment variable in standalone
CLI (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19617">#19617</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/tailwindlabs/tailwindcss/blob/main/CHANGELOG.md">tailwindcss's
changelog</a>.</em></p>
<blockquote>
<h2>[4.2.4] - 2026-04-21</h2>
<h3>Fixed</h3>
<ul>
<li>Ensure imports in <code>@import</code> and <code>@plugin</code>
still resolve correctly when using Vite aliases in
<code>@tailwindcss/vite</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19947">#19947</a>)</li>
</ul>
<h2>[4.2.3] - 2026-04-20</h2>
<h3>Fixed</h3>
<ul>
<li>Canonicalization: improve canonicalizations for
<code>tracking-*</code> utilities by preferring non-negative utilities
(e.g. <code>-tracking-tighter</code> → <code>tracking-wider</code>) (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19827">#19827</a>)</li>
<li>Fix crash due to invalid characters in candidate (exceeding valid
unicode code point range) (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19829">#19829</a>)</li>
<li>Ensure query params in imports are considered unique resources when
using <code>@tailwindcss/webpack</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19723">#19723</a>)</li>
<li>Canonicalization: collapse arbitrary values into shorthand utilities
(e.g. <code>px-[1.2rem] py-[1.2rem]</code> → <code>p-[1.2rem]</code>)
(<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19837">#19837</a>)</li>
<li>Canonicalization: collapse <code>border-{t,b}-*</code> into
<code>border-y-*</code>, <code>border-{l,r}-*</code> into
<code>border-x-*</code>, and <code>border-{t,r,b,l}-*</code> into
<code>border-*</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19842">#19842</a>)</li>
<li>Canonicalization: collapse <code>scroll-m{t,b}-*</code> into
<code>scroll-my-*</code>, <code>scroll-m{l,r}-*</code> into
<code>scroll-mx-*</code>, and <code>scroll-m{t,r,b,l}-*</code> into
<code>scroll-m-*</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19842">#19842</a>)</li>
<li>Canonicalization: collapse <code>scroll-p{t,b}-*</code> into
<code>scroll-py-*</code>, <code>scroll-p{l,r}-*</code> into
<code>scroll-px-*</code>, and <code>scroll-p{t,r,b,l}-*</code> into
<code>scroll-p-*</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19842">#19842</a>)</li>
<li>Canonicalization: collapse <code>overflow-{x,y}-*</code> into
<code>overflow-*</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19842">#19842</a>)</li>
<li>Canonicalization: collapse <code>overscroll-{x,y}-*</code> into
<code>overscroll-*</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19842">#19842</a>)</li>
<li>Read from <code>--placeholder-color</code> instead of
<code>--background-color</code> for <code>placeholder-*</code> utilities
(<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19843">#19843</a>)</li>
<li>Upgrade: ensure files are not emptied out when killing the upgrade
process while it's running (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19846">#19846</a>)</li>
<li>Upgrade: use <code>config.content</code> when migrating from
Tailwind CSS v3 to Tailwind CSS v4 (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19846">#19846</a>)</li>
<li>Upgrade: never migrate files that are ignored by git (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19846">#19846</a>)</li>
<li>Add <code>.env</code> and <code>.env.*</code> to default ignored
content files (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19846">#19846</a>)</li>
<li>Canonicalization: migrate <code>overflow-ellipsis</code> into
<code>text-ellipsis</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19849">#19849</a>)</li>
<li>Canonicalization: migrate <code>start-full</code> →
<code>inset-s-full</code>, <code>start-auto</code> →
<code>inset-s-auto</code>, <code>start-px</code> →
<code>inset-s-px</code>, and <code>start-&lt;number&gt;</code> →
<code>inset-s-&lt;number&gt;</code> as well as negative versions (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19849">#19849</a>)</li>
<li>Canonicalization: migrate <code>end-full</code> →
<code>inset-e-full</code>, <code>end-auto</code> →
<code>inset-e-auto</code>, <code>end-px</code> →
<code>inset-e-px</code>, and <code>end-&lt;number&gt;</code> →
<code>inset-e-&lt;number&gt;</code> as well as negative versions (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19849">#19849</a>)</li>
<li>Canonicalization: move the <code>-</code> sign inside the arbitrary
value <code>-left-[9rem]</code> → <code>left-[-9rem]</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19858">#19858</a>)</li>
<li>Canonicalization: move the <code>-</code> sign outside the arbitrary
value <code>ml-[calc(-1*var(--width))]</code> →
<code>-ml-(--width)</code> (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19858">#19858</a>)</li>
<li>Improve performance when scanning JSONL / NDJSON files (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19862">#19862</a>)</li>
<li>Support <code>NODE_PATH</code> environment variable in standalone
CLI (<a
href="https://redirect.github.com/tailwindlabs/tailwindcss/pull/19617">#19617</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/tailwindlabs/tailwindcss/commit/69ad7cc5ec091125842aba783c5bfce536e9c234"><code>69ad7cc</code></a>
4.2.4 (<a
href="https://github.com/tailwindlabs/tailwindcss/tree/HEAD/packages/tailwindcss/issues/19948">#19948</a>)</li>
<li><a
href="https://github.com/tailwindlabs/tailwindcss/commit/2e3fa490a535a8341fd2853d75098f0d4dbcc825"><code>2e3fa49</code></a>
4.2.3 (<a
href="https://github.com/tailwindlabs/tailwindcss/tree/HEAD/packages/tailwindcss/issues/19944">#19944</a>)</li>
<li><a
href="https://github.com/tailwindlabs/tailwindcss/commit/df6209ab8b957e56e6e00042a3e4080c14a072ea"><code>df6209a</code></a>
Canonicalize negative arbitrary values (<a
href="https://github.com/tailwindlabs/tailwindcss/tree/HEAD/packages/tailwindcss/issues/19858">#19858</a>)</li>
<li><a
href="https://github.com/tailwindlabs/tailwindcss/commit/52fd421cc941aad650db769f04b77d08c56951df"><code>52fd421</code></a>
Small refactor of canonicalization tests (<a
href="https://github.com/tailwindlabs/tailwindcss/tree/HEAD/packages/tailwindcss/issues/19851">#19851</a>)</li>
<li><a
href="https://github.com/tailwindlabs/tailwindcss/commit/c385fd36bcce707a02c229ebeb66ea88e12eb041"><code>c385fd3</code></a>
use <code>test.each</code> instead of manual loop</li>
<li><a
href="https://github.com/tailwindlabs/tailwindcss/commit/0d6e038889f49a809a4077ef7badc7d43184cd42"><code>0d6e038</code></a>
fix index in test name</li>
<li><a
href="https://github.com/tailwindlabs/tailwindcss/commit/88a2d22c2fdb65c5f8c3b2f68db9713f8bf2e656"><code>88a2d22</code></a>
Add more canonicalization rules for deprecated utilities (<a
href="https://github.com/tailwindlabs/tailwindcss/tree/HEAD/packages/tailwindcss/issues/19849">#19849</a>)</li>
<li><a
href="https://github.com/tailwindlabs/tailwindcss/commit/2c1ef9eb2564f2c83df970eb9f68c19d340c02e5"><code>2c1ef9e</code></a>
Use <code>--placeholder-color</code> instead of
<code>--background-color</code> for <code>placeholder-*</code>...</li>
<li><a
href="https://github.com/tailwindlabs/tailwindcss/commit/28d526859dfd1efe204c8f037c3fe472d9b5d08e"><code>28d5268</code></a>
Collapse more utilities by expanding their declarations (<a
href="https://github.com/tailwindlabs/tailwindcss/tree/HEAD/packages/tailwindcss/issues/19842">#19842</a>)</li>
<li><a
href="https://github.com/tailwindlabs/tailwindcss/commit/b55d96002ce4516c5158ad3565229267b948c1ac"><code>b55d960</code></a>
fix(canonicalize): collapse arbitrary values into shorthand utilities
(<a
href="https://github.com/tailwindlabs/tailwindcss/tree/HEAD/packages/tailwindcss/issues/19837">#19837</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/tailwindlabs/tailwindcss/commits/v4.2.4/packages/tailwindcss">compare
view</a></li>
</ul>
</details>
<br />

Updates `vite` from 8.0.8 to 8.0.10
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/vitejs/vite/releases">vite's
releases</a>.</em></p>
<blockquote>
<h2>v8.0.10</h2>
<p>Please refer to <a
href="https://github.com/vitejs/vite/blob/v8.0.10/packages/vite/CHANGELOG.md">CHANGELOG.md</a>
for details.</p>
<h2>v8.0.9</h2>
<p>Please refer to <a
href="https://github.com/vitejs/vite/blob/v8.0.9/packages/vite/CHANGELOG.md">CHANGELOG.md</a>
for details.</p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/vitejs/vite/blob/main/packages/vite/CHANGELOG.md">vite's
changelog</a>.</em></p>
<blockquote>
<h2><!-- raw HTML omitted --><a
href="https://github.com/vitejs/vite/compare/v8.0.9...v8.0.10">8.0.10</a>
(2026-04-23)<!-- raw HTML omitted --></h2>
<h3>Features</h3>
<ul>
<li>update rolldown to 1.0.0-rc.17 (<a
href="https://redirect.github.com/vitejs/vite/issues/22299">#22299</a>)
(<a
href="https://github.com/vitejs/vite/commit/a4d06d9015167d30fe8ac63d1ce2edc146cdca31">a4d06d9</a>)</li>
</ul>
<h3>Bug Fixes</h3>
<ul>
<li><code>hmrClient.logger.debug</code> and
<code>hmrClient.logger.error</code> looked different from other HMR logs
(<a
href="https://redirect.github.com/vitejs/vite/issues/22147">#22147</a>)
(<a
href="https://github.com/vitejs/vite/commit/a4d828f2d5ed85440bc0774eab342e6f9a5e5f62">a4d828f</a>)</li>
<li><strong>css:</strong> show filename in CSS minification warnings for
<code>.css?inline</code> (<a
href="https://redirect.github.com/vitejs/vite/issues/22292">#22292</a>)
(<a
href="https://github.com/vitejs/vite/commit/83f0a785a2ae48d6761fb69f4b0523a24ae9342c">83f0a78</a>)</li>
<li><strong>optimizer:</strong> allow user transform.target to override
default in optimizeDeps (<a
href="https://redirect.github.com/vitejs/vite/issues/22273">#22273</a>)
(<a
href="https://github.com/vitejs/vite/commit/5c7cec69b637544ab16009d8758df7dbbf7f2674">5c7cec6</a>)</li>
<li>remove format sniffing module resolution from JS resolver (<a
href="https://redirect.github.com/vitejs/vite/issues/22297">#22297</a>)
(<a
href="https://github.com/vitejs/vite/commit/b8a21cc821c1434ac9d2b85ec53005df9edc306b">b8a21cc</a>)</li>
</ul>
<h3>Code Refactoring</h3>
<ul>
<li>enable some typecheck rules (<a
href="https://redirect.github.com/vitejs/vite/issues/22278">#22278</a>)
(<a
href="https://github.com/vitejs/vite/commit/943751801f70057ae94f9092e349c8f3fd9ccdf2">9437518</a>)</li>
<li>typecheck client directory (<a
href="https://redirect.github.com/vitejs/vite/issues/22284">#22284</a>)
(<a
href="https://github.com/vitejs/vite/commit/40a0847276502b33a3942b3cfab04b20218f3543">40a0847</a>)</li>
</ul>
<h2><!-- raw HTML omitted --><a
href="https://github.com/vitejs/vite/compare/v8.0.8...v8.0.9">8.0.9</a>
(2026-04-20)<!-- raw HTML omitted --></h2>
<h3>Features</h3>
<ul>
<li>update rolldown to 1.0.0-rc.16 (<a
href="https://redirect.github.com/vitejs/vite/issues/22248">#22248</a>)
(<a
href="https://github.com/vitejs/vite/commit/2947edd57ceb64a0b4dc43269743e8e44e68c09b">2947edd</a>)</li>
</ul>
<h3>Bug Fixes</h3>
<ul>
<li>allow binding when strictPort is set but wildcard port is in use (<a
href="https://redirect.github.com/vitejs/vite/issues/22150">#22150</a>)
(<a
href="https://github.com/vitejs/vite/commit/dfc8aa5057dd8ec2b1223980d1e2eeb946ac3384">dfc8aa5</a>)</li>
<li><strong>build:</strong> emptyOutDir should happen for watch rebuilds
(<a
href="https://redirect.github.com/vitejs/vite/issues/22207">#22207</a>)
(<a
href="https://github.com/vitejs/vite/commit/ee522672bb374c7ff95a347f14732491121b1cd6">ee52267</a>)</li>
<li><strong>bundled-dev:</strong> reject requests to HMR patch files in
non potentially trustworthy origins (<a
href="https://redirect.github.com/vitejs/vite/issues/22269">#22269</a>)
(<a
href="https://github.com/vitejs/vite/commit/868f1411a6f474baa4417f2d6524692dd452f760">868f141</a>)</li>
<li><strong>css:</strong> use unique key for cssEntriesMap to prevent
same-basename collision (<a
href="https://redirect.github.com/vitejs/vite/issues/22039">#22039</a>)
(<a
href="https://github.com/vitejs/vite/commit/374bb5d597fcd0485e929565c698d8ed219136f8">374bb5d</a>)</li>
<li><strong>deps:</strong> update all non-major dependencies (<a
href="https://redirect.github.com/vitejs/vite/issues/22219">#22219</a>)
(<a
href="https://github.com/vitejs/vite/commit/4cd0d6760edd5fb0841abe86538de3c225e880a1">4cd0d67</a>)</li>
<li><strong>deps:</strong> update all non-major dependencies (<a
href="https://redirect.github.com/vitejs/vite/issues/22268">#22268</a>)
(<a
href="https://github.com/vitejs/vite/commit/c28e9c12a849f80e6fdc93f42283ad2863ab9dbc">c28e9c1</a>)</li>
<li>detect Deno workspace root (fix <a
href="https://redirect.github.com/vitejs/vite/issues/22237">#22237</a>)
(<a
href="https://redirect.github.com/vitejs/vite/issues/22238">#22238</a>)
(<a
href="https://github.com/vitejs/vite/commit/1b793c0e1726467fffd06ffad9bc81c61a840188">1b793c0</a>)</li>
<li><strong>dev:</strong> handle errors in <code>watchChange</code> hook
(<a
href="https://redirect.github.com/vitejs/vite/issues/22188">#22188</a>)
(<a
href="https://github.com/vitejs/vite/commit/fc08bdab9bba871b03689f2f6997c3a4ba4351da">fc08bda</a>)</li>
<li><strong>optimizer:</strong> handle more chars that will be sanitized
(<a
href="https://redirect.github.com/vitejs/vite/issues/22208">#22208</a>)
(<a
href="https://github.com/vitejs/vite/commit/3f24533ac4845ed22547279d1721bd82a35345e3">3f24533</a>)</li>
<li>skip fallback sourcemap generation for <code>?raw</code> imports (<a
href="https://redirect.github.com/vitejs/vite/issues/22148">#22148</a>)
(<a
href="https://github.com/vitejs/vite/commit/3ec9cdaac7936ca32d0956c4cb1eb6e172945996">3ec9cda</a>)</li>
</ul>
<h3>Documentation</h3>
<ul>
<li>align the descriptions in READMEs (<a
href="https://redirect.github.com/vitejs/vite/issues/22231">#22231</a>)
(<a
href="https://github.com/vitejs/vite/commit/44c42b97639bb6ad777e66d752b2829cccb9a27a">44c42b9</a>)</li>
<li>fix reuses wording in dev environment comment (<a
href="https://redirect.github.com/vitejs/vite/issues/22173">#22173</a>)
(<a
href="https://github.com/vitejs/vite/commit/9163412fdfec7fb1656529713326a5b5c5e986ea">9163412</a>)</li>
<li>fix wording in sass error comment (<a
href="https://redirect.github.com/vitejs/vite/issues/22214">#22214</a>)
(<a
href="https://github.com/vitejs/vite/commit/bc5c6a7a498845dff20dc410c395355b79a4b753">bc5c6a7</a>)</li>
<li>update build CLI defaults (<a
href="https://redirect.github.com/vitejs/vite/issues/22261">#22261</a>)
(<a
href="https://github.com/vitejs/vite/commit/605bb97994678a1bb70a8de9a85c29d5f5d48c5a">605bb97</a>)</li>
</ul>
<h3>Miscellaneous Chores</h3>
<ul>
<li><strong>deps:</strong> update dependency dotenv-expand to v13 (<a
href="https://redirect.github.com/vitejs/vite/issues/22271">#22271</a>)
(<a
href="https://github.com/vitejs/vite/commit/0a3887da18812cacb254c616e4dd35631e776fda">0a3887d</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/vitejs/vite/commit/32c29780404c353f5a7c5ba4d06fc5e676741714"><code>32c2978</code></a>
release: v8.0.10</li>
<li><a
href="https://github.com/vitejs/vite/commit/a4d06d9015167d30fe8ac63d1ce2edc146cdca31"><code>a4d06d9</code></a>
feat: update rolldown to 1.0.0-rc.17 (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/22299">#22299</a>)</li>
<li><a
href="https://github.com/vitejs/vite/commit/a4d828f2d5ed85440bc0774eab342e6f9a5e5f62"><code>a4d828f</code></a>
fix: <code>hmrClient.logger.debug</code> and
<code>hmrClient.logger.error</code> looked different f...</li>
<li><a
href="https://github.com/vitejs/vite/commit/83f0a785a2ae48d6761fb69f4b0523a24ae9342c"><code>83f0a78</code></a>
fix(css): show filename in CSS minification warnings for
<code>.css?inline</code> (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/22292">#22292</a>)</li>
<li><a
href="https://github.com/vitejs/vite/commit/b8a21cc821c1434ac9d2b85ec53005df9edc306b"><code>b8a21cc</code></a>
fix: remove format sniffing module resolution from JS resolver (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/22297">#22297</a>)</li>
<li><a
href="https://github.com/vitejs/vite/commit/40a0847276502b33a3942b3cfab04b20218f3543"><code>40a0847</code></a>
refactor: typecheck client directory (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/22284">#22284</a>)</li>
<li><a
href="https://github.com/vitejs/vite/commit/5c7cec69b637544ab16009d8758df7dbbf7f2674"><code>5c7cec6</code></a>
fix(optimizer): allow user transform.target to override default in
optimizeDe...</li>
<li><a
href="https://github.com/vitejs/vite/commit/943751801f70057ae94f9092e349c8f3fd9ccdf2"><code>9437518</code></a>
refactor: enable some typecheck rules (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/22278">#22278</a>)</li>
<li><a
href="https://github.com/vitejs/vite/commit/ce729f5fa1a5adca373b2adcb0e1b18099164a14"><code>ce729f5</code></a>
release: v8.0.9</li>
<li><a
href="https://github.com/vitejs/vite/commit/605bb97994678a1bb70a8de9a85c29d5f5d48c5a"><code>605bb97</code></a>
docs: update build CLI defaults (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/22261">#22261</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/vitejs/vite/commits/v8.0.10/packages/vite">compare
view</a></li>
</ul>
</details>
<br />

Updates `vitest` from 4.1.4 to 4.1.5
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/vitest-dev/vitest/releases">vitest's
releases</a>.</em></p>
<blockquote>
<h2>v4.1.5</h2>
<h3>   🚀 Experimental Features</h3>
<ul>
<li><strong>coverage</strong>: Istanbul to support
<code>instrumenter</code> option  -  by <a
href="https://github.com/BartWaardenburg"><code>@​BartWaardenburg</code></a>
and <a
href="https://github.com/AriPerkkio"><code>@​AriPerkkio</code></a> in <a
href="https://redirect.github.com/vitest-dev/vitest/issues/10119">vitest-dev/vitest#10119</a>
<a href="https://github.com/vitest-dev/vitest/commit/0e0ff41c7"><!-- raw
HTML omitted -->(0e0ff)<!-- raw HTML omitted --></a></li>
</ul>
<h3>   🐞 Bug Fixes</h3>
<ul>
<li>--project negation excludes browser instances  -  by <a
href="https://github.com/felamaslen"><code>@​felamaslen</code></a> in <a
href="https://redirect.github.com/vitest-dev/vitest/issues/10131">vitest-dev/vitest#10131</a>
<a href="https://github.com/vitest-dev/vitest/commit/9423dc084"><!-- raw
HTML omitted -->(9423d)<!-- raw HTML omitted --></a></li>
<li>Project color label on html reporter  -  by <a
href="https://github.com/hi-ogawa"><code>@​hi-ogawa</code></a> in <a
href="https://redirect.github.com/vitest-dev/vitest/issues/10142">vitest-dev/vitest#10142</a>
<a href="https://github.com/vitest-dev/vitest/commit/596f73986"><!-- raw
HTML omitted -->(596f7)<!-- raw HTML omitted --></a></li>
<li>Fix <code>vi.defineHelper</code> called as object method  -  by <a
href="https://github.com/hi-ogawa"><code>@​hi-ogawa</code></a> in <a
href="https://redirect.github.com/vitest-dev/vitest/issues/10163">vitest-dev/vitest#10163</a>
<a href="https://github.com/vitest-dev/vitest/commit/122c25b5b"><!-- raw
HTML omitted -->(122c2)<!-- raw HTML omitted --></a></li>
<li>Alias <code>agent</code> reporter to <code>minimal</code>  -  by <a
href="https://github.com/sheremet-va"><code>@​sheremet-va</code></a> in
<a
href="https://redirect.github.com/vitest-dev/vitest/issues/10157">vitest-dev/vitest#10157</a>
<a href="https://github.com/vitest-dev/vitest/commit/663b99fe3"><!-- raw
HTML omitted -->(663b9)<!-- raw HTML omitted --></a></li>
<li>Respect diff config options in soft assertions  -  by <a
href="https://github.com/Copilot"><code>@​Copilot</code></a>,
<strong>sheremet-va</strong> and <a
href="https://github.com/sheremet-va"><code>@​sheremet-va</code></a> in
<a
href="https://redirect.github.com/vitest-dev/vitest/issues/8696">vitest-dev/vitest#8696</a>
<a href="https://github.com/vitest-dev/vitest/commit/9787dedad"><!-- raw
HTML omitted -->(9787d)<!-- raw HTML omitted --></a></li>
<li>Respect diff config options in soft assertions &quot;  -  by <a
href="https://github.com/sheremet-va"><code>@​sheremet-va</code></a> in
<a
href="https://redirect.github.com/vitest-dev/vitest/issues/8696">vitest-dev/vitest#8696</a>
<a href="https://github.com/vitest-dev/vitest/commit/7dc6d54fd"><!-- raw
HTML omitted -->(7dc6d)<!-- raw HTML omitted --></a></li>
<li><strong>ast-collect</strong>: Recognize _<em>vi_import</em> prefix
in static test discovery  -  by <a
href="https://github.com/Yejneshwar"><code>@​Yejneshwar</code></a> in <a
href="https://redirect.github.com/vitest-dev/vitest/issues/10129">vitest-dev/vitest#10129</a>
<a href="https://github.com/vitest-dev/vitest/commit/325463ab2"><!-- raw
HTML omitted -->(32546)<!-- raw HTML omitted --></a></li>
<li><strong>coverage</strong>: Descriptive error message when reports
directory is removed during test run  -  by <a
href="https://github.com/DaveT1991"><code>@​DaveT1991</code></a> and <a
href="https://github.com/AriPerkkio"><code>@​AriPerkkio</code></a> in <a
href="https://redirect.github.com/vitest-dev/vitest/issues/10117">vitest-dev/vitest#10117</a>
<a href="https://github.com/vitest-dev/vitest/commit/1413382e1"><!-- raw
HTML omitted -->(14133)<!-- raw HTML omitted --></a></li>
<li><strong>snapshot</strong>: Increase default snapshot max output
length  -  by <a
href="https://github.com/hi-ogawa"><code>@​hi-ogawa</code></a> and
<strong>Codex</strong> in <a
href="https://redirect.github.com/vitest-dev/vitest/issues/10150">vitest-dev/vitest#10150</a>
<a href="https://github.com/vitest-dev/vitest/commit/21e66ff63"><!-- raw
HTML omitted -->(21e66)<!-- raw HTML omitted --></a></li>
<li><strong>ui</strong>: Fix jsx/tsx syntax highlight  -  by <a
href="https://github.com/hi-ogawa"><code>@​hi-ogawa</code></a> in <a
href="https://redirect.github.com/vitest-dev/vitest/issues/10152">vitest-dev/vitest#10152</a>
<a href="https://github.com/vitest-dev/vitest/commit/f1b1f6c7b"><!-- raw
HTML omitted -->(f1b1f)<!-- raw HTML omitted --></a></li>
<li><strong>web-worker</strong>: Support MessagePort objects referenced
inside postMessage data  -  by <a
href="https://github.com/whitphx"><code>@​whitphx</code></a> and
<strong>Claude Opus 4.6 (1M context)</strong> in <a
href="https://redirect.github.com/vitest-dev/vitest/issues/9927">vitest-dev/vitest#9927</a>
and <a
href="https://redirect.github.com/vitest-dev/vitest/issues/10124">vitest-dev/vitest#10124</a>
<a href="https://github.com/vitest-dev/vitest/commit/7ad7d39af"><!-- raw
HTML omitted -->(7ad7d)<!-- raw HTML omitted --></a></li>
<li><strong>api</strong>: Make test-specification options writable  - 
by <a
href="https://github.com/sheremet-va"><code>@​sheremet-va</code></a> in
<a
href="https://redirect.github.com/vitest-dev/vitest/issues/10154">vitest-dev/vitest#10154</a>
<a href="https://github.com/vitest-dev/vitest/commit/6abd557b7"><!-- raw
HTML omitted -->(6abd5)<!-- raw HTML omitted --></a></li>
</ul>
<h5>    <a
href="https://github.com/vitest-dev/vitest/compare/v4.1.4...v4.1.5">View
changes on GitHub</a></h5>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/vitest-dev/vitest/commit/e399846850fedf10b8228cbe46a419628998acd9"><code>e399846</code></a>
chore: release v4.1.5</li>
<li><a
href="https://github.com/vitest-dev/vitest/commit/7dc6d54fd9dda0fe6fee2fb6451d0611a9ecb6e7"><code>7dc6d54</code></a>
Revert &quot;fix: respect diff config options in soft assertions (<a
href="https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest/issues/8696">#8696</a>)&quot;</li>
<li><a
href="https://github.com/vitest-dev/vitest/commit/9787dedade9896a6d3eeed7739177d6c583a68a7"><code>9787ded</code></a>
fix: respect diff config options in soft assertions (<a
href="https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest/issues/8696">#8696</a>)</li>
<li><a
href="https://github.com/vitest-dev/vitest/commit/325463ab292c45c3ef27aa21ec7da380c307052c"><code>325463a</code></a>
fix(ast-collect): recognize _<em>vi_import</em> prefix in static test
discovery (<a
href="https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest/issues/10">#10</a>...</li>
<li><a
href="https://github.com/vitest-dev/vitest/commit/0e0ff41c7e86d6e2bf581f074dc216805d10d371"><code>0e0ff41</code></a>
feat(coverage): istanbul to support <code>instrumenter</code> option (<a
href="https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest/issues/10119">#10119</a>)</li>
<li><a
href="https://github.com/vitest-dev/vitest/commit/663b99fe3e6a60fc8a7ccd2d9941d1cbe929b606"><code>663b99f</code></a>
fix: alias <code>agent</code> reporter to <code>minimal</code> (<a
href="https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest/issues/10157">#10157</a>)</li>
<li><a
href="https://github.com/vitest-dev/vitest/commit/122c25b5b157ffd31b376561b16ab983aa23e7bc"><code>122c25b</code></a>
fix: fix <code>vi.defineHelper</code> called as object method (<a
href="https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest/issues/10163">#10163</a>)</li>
<li><a
href="https://github.com/vitest-dev/vitest/commit/6abd557b7219156893dd13a1dbe86501d5542d2e"><code>6abd557</code></a>
feat(api): make test-specification options writable (<a
href="https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest/issues/10154">#10154</a>)</li>
<li><a
href="https://github.com/vitest-dev/vitest/commit/596f73986abe2161a9a06f0ca03df68e82690b21"><code>596f739</code></a>
fix: project color label on html reporter (<a
href="https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest/issues/10142">#10142</a>)</li>
<li><a
href="https://github.com/vitest-dev/vitest/commit/9423dc0841e97b6dcac8a73cdb8e656b3d6ba909"><code>9423dc0</code></a>
fix: --project negation excludes browser instances (<a
href="https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest/issues/10131">#10131</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/vitest-dev/vitest/commits/v4.1.5/packages/vitest">compare
view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-25 13:12:25 +01:00
dependabot[bot] 74a466139d Bump actions/setup-node from 6.3.0 to 6.4.0 in the gh-actions group (#913)
Bumps the gh-actions group with 1 update:
[actions/setup-node](https://github.com/actions/setup-node).

Updates `actions/setup-node` from 6.3.0 to 6.4.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/actions/setup-node/releases">actions/setup-node's
releases</a>.</em></p>
<blockquote>
<h2>v6.4.0</h2>
<h2>What's Changed</h2>
<h3>Dependency updates:</h3>
<ul>
<li>Upgrade <a
href="https://github.com/actions"><code>@​actions</code></a>
dependencies by <a
href="https://github.com/Copilot"><code>@​Copilot</code></a> in <a
href="https://redirect.github.com/actions/setup-node/pull/1525">actions/setup-node#1525</a></li>
<li>Update Node.js versions in versions.yml and bump package to v6.4.0
by <a
href="https://github.com/priya-kinthali"><code>@​priya-kinthali</code></a>
in <a
href="https://redirect.github.com/actions/setup-node/pull/1533">actions/setup-node#1533</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/Copilot"><code>@​Copilot</code></a> made
their first contribution in <a
href="https://redirect.github.com/actions/setup-node/pull/1525">actions/setup-node#1525</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/setup-node/compare/v6...v6.4.0">https://github.com/actions/setup-node/compare/v6...v6.4.0</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/actions/setup-node/commit/48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e"><code>48b55a0</code></a>
Update Node.js versions in versions.yml and bump package to v6.4.0 (<a
href="https://redirect.github.com/actions/setup-node/issues/1533">#1533</a>)</li>
<li><a
href="https://github.com/actions/setup-node/commit/ab72c7e7eba0eaa11f8cab0f5679243900c2cac9"><code>ab72c7e</code></a>
Upgrade <a href="https://github.com/actions"><code>@​actions</code></a>
dependencies (<a
href="https://redirect.github.com/actions/setup-node/issues/1525">#1525</a>)</li>
<li>See full diff in <a
href="https://github.com/actions/setup-node/compare/53b83947a5a98c8d113130e565377fae1a50d02f...48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=actions/setup-node&package-manager=github_actions&previous-version=6.3.0&new-version=6.4.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-25 13:12:13 +01:00
Humaid KhanandHumaid Arif Khan d6a7588d22 Remove immutable to allow wal checkpoints (#916)
Fixes https://github.com/calibrain/shelfmark/issues/915

## Description

Remove immutable=1 to allow wal checkpoints to fire and get latest auth
info.

Note db is still in readonly mode, thanks to mode=ro

## Test case

1. Run CWA/CW
2. Change Admin password from admin/admin123 to anything else
3. Run shelfmark with CWA db auth
4. Shelfmark should be able to login after password change

Co-authored-by: Humaid Arif Khan <humaid.k@tlt.local>
2026-04-25 13:08:27 +01:00
dependabot[bot]andAlex 700935ec17 Bump the npm-deps group in /src/frontend with 6 updates (#899)
Bumps the npm-deps group in /src/frontend with 6 updates:

| Package | From | To |
| --- | --- | --- |
|
[react-router-dom](https://github.com/remix-run/react-router/tree/HEAD/packages/react-router-dom)
| `7.14.0` | `7.14.1` |
| [knip](https://github.com/webpro-nl/knip/tree/HEAD/packages/knip) |
`6.4.0` | `6.4.1` |
| [oxfmt](https://github.com/oxc-project/oxc/tree/HEAD/npm/oxfmt) |
`0.44.0` | `0.45.0` |
| [oxlint](https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint) |
`1.59.0` | `1.60.0` |
| [oxlint-tsgolint](https://github.com/oxc-project/tsgolint) | `0.20.0`
| `0.21.1` |
| [typescript](https://github.com/microsoft/TypeScript) | `6.0.2` |
`6.0.3` |

Updates `react-router-dom` from 7.14.0 to 7.14.1
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/remix-run/react-router/blob/main/packages/react-router-dom/CHANGELOG.md">react-router-dom's
changelog</a>.</em></p>
<blockquote>
<h2>v7.14.1</h2>
<h3>Patch Changes</h3>
<ul>
<li>Updated dependencies:
<ul>
<li><a
href="https://github.com/remix-run/react-router/releases/tag/react-router@7.14.1"><code>react-router@7.14.1</code></a></li>
</ul>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/remix-run/react-router/commit/197674ba9fc1b72b452e17894e5e783bdab7a087"><code>197674b</code></a>
Release 7.14.1 (<a
href="https://github.com/remix-run/react-router/tree/HEAD/packages/react-router-dom/issues/14973">#14973</a>)</li>
<li><a
href="https://github.com/remix-run/react-router/commit/a87774f997d8ba497c97562840f0766250c3e4ce"><code>a87774f</code></a>
Add new release process (<a
href="https://github.com/remix-run/react-router/tree/HEAD/packages/react-router-dom/issues/14916">#14916</a>)</li>
<li>See full diff in <a
href="https://github.com/remix-run/react-router/commits/react-router-dom@7.14.1/packages/react-router-dom">compare
view</a></li>
</ul>
</details>
<br />

Updates `knip` from 6.4.0 to 6.4.1
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/webpro-nl/knip/releases">knip's
releases</a>.</em></p>
<blockquote>
<h2>Release 6.4.1</h2>
<ul>
<li>license (2d3d8d86ea51f18224c3558a38c28df00113f683)</li>
<li>Handle file path arguments in Bun plugin (resolve <a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1678">#1678</a>)
(cdbe298efcb226ae4baec6567a0cbb6fdf001ee8)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/webpro-nl/knip/commit/ea70d72232b13dca6338105f9fd4ea211595dfba"><code>ea70d72</code></a>
Release knip@6.4.1</li>
<li><a
href="https://github.com/webpro-nl/knip/commit/cdbe298efcb226ae4baec6567a0cbb6fdf001ee8"><code>cdbe298</code></a>
Handle file path arguments in Bun plugin (resolve <a
href="https://github.com/webpro-nl/knip/tree/HEAD/packages/knip/issues/1678">#1678</a>)</li>
<li>See full diff in <a
href="https://github.com/webpro-nl/knip/commits/knip@6.4.1/packages/knip">compare
view</a></li>
</ul>
</details>
<br />

Updates `oxfmt` from 0.44.0 to 0.45.0
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/oxc-project/oxc/blob/main/npm/oxfmt/CHANGELOG.md">oxfmt's
changelog</a>.</em></p>
<blockquote>
<h2>[0.45.0] - 2026-04-13</h2>
<h3>🐛 Bug Fixes</h3>
<ul>
<li>50c389b oxfmt: Support <code>.editorconfig</code>
<code>quote_type</code> (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxfmt/issues/20989">#20989</a>)
(leaysgur)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/oxc-project/oxc/commit/e16848eb4c29e51e2e3e2bbda36c76ede42ab974"><code>e16848e</code></a>
release(apps): oxlint v1.60.0 &amp;&amp; oxfmt v0.45.0 (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxfmt/issues/21375">#21375</a>)</li>
<li><a
href="https://github.com/oxc-project/oxc/commit/50c389b64ffa65c6a889f725f53c748d7e3bc507"><code>50c389b</code></a>
fix(oxfmt): Support <code>.editorconfig</code> <code>quote_type</code>
(<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxfmt/issues/20989">#20989</a>)</li>
<li>See full diff in <a
href="https://github.com/oxc-project/oxc/commits/oxfmt_v0.45.0/npm/oxfmt">compare
view</a></li>
</ul>
</details>
<br />

Updates `oxlint` from 1.59.0 to 1.60.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/oxc-project/oxc/releases">oxlint's
releases</a>.</em></p>
<blockquote>
<h2>oxlint v1.27.0 &amp;&amp; oxfmt v0.12.0</h2>
<h1>Oxlint v1.27.0</h1>
<h3>🚀 Features</h3>
<ul>
<li>222a8f0 linter/plugins: Implement
<code>SourceCode#isSpaceBetween</code> (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15498">#15498</a>)
(overlookmotel)</li>
<li>2f9735d linter/plugins: Implement
<code>context.languageOptions</code> (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15486">#15486</a>)
(overlookmotel)</li>
<li>bc731ff linter/plugins: Stub out all <code>Context</code> APIs (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15479">#15479</a>)
(overlookmotel)</li>
<li>5822cb4 linter/plugins: Add <code>extend</code> method to
<code>FILE_CONTEXT</code> (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15477">#15477</a>)
(overlookmotel)</li>
<li>7b1e6f3 apps: Add pure rust binaries and release to github (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15469">#15469</a>)
(Boshen)</li>
<li>2a89b43 linter: Introduce debug assertions after fixes to assert
validity (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15389">#15389</a>)
(camc314)</li>
<li>ad3c45a editor: Add <code>oxc.path.node</code> option (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15040">#15040</a>)
(Sysix)</li>
</ul>
<h3>🐛 Bug Fixes</h3>
<ul>
<li>6f3cd77 linter/no-var: Incorrect warning for blocks (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15504">#15504</a>)
(Hamir Mahal)</li>
<li>6957fb9 linter/plugins: Do not allow access to
<code>Context#id</code> in <code>createOnce</code> (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15489">#15489</a>)
(overlookmotel)</li>
<li>7409630 linter/plugins: Allow access to <code>cwd</code> in
<code>createOnce</code> in ESLint interop mode (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15488">#15488</a>)
(overlookmotel)</li>
<li>732205e parser: Reject <code>using</code> / <code>await using</code>
in a switch <code>case</code> / <code>default</code> clause (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15225">#15225</a>)
(sapphi-red)</li>
<li>a17ca32 linter/plugins: Replace <code>Context</code> class (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15448">#15448</a>)
(overlookmotel)</li>
<li>ecf2f7b language_server: Fail gracefully when tsgolint executable
not found (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15436">#15436</a>)
(camc314)</li>
<li>3c8d3a7 lang-server: Improve logging in failure case for tsgolint
(<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15299">#15299</a>)
(camc314)</li>
<li>ef71410 linter: Use jsx if source type is JS in fix debug assertion
(<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15434">#15434</a>)
(camc314)</li>
<li>e32bbf6 linter/no-var: Handle TypeScript declare keyword in fixer
(<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15426">#15426</a>)
(camc314)</li>
<li>6565dbe linter/switch-case-braces: Skip comments when searching for
<code>:</code> token (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15425">#15425</a>)
(camc314)</li>
<li>85bd19a linter/prefer-class-fields: Insert value after type
annotation in fixer (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15423">#15423</a>)
(camc314)</li>
<li>fde753e linter/plugins: Block access to
<code>context.settings</code> in <code>createOnce</code> (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15394">#15394</a>)
(overlookmotel)</li>
<li>ddd9f9f linter/forward-ref-uses-ref: Dont suggest removing wrapper
in invalid positions (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15388">#15388</a>)
(camc314)</li>
<li>dac2a9c linter/no-template-curly-in-string: Remove fixer (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15387">#15387</a>)
(camc314)</li>
<li>989b8e3 linter/no-var: Only fix to <code>const</code> if the var has
an initializer (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15385">#15385</a>)
(camc314)</li>
<li>cc403f5 linter/plugins: Return empty object for unimplemented
parserServices (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15364">#15364</a>)
(magic-akari)</li>
</ul>
<h3>⚡ Performance</h3>
<ul>
<li>25d577e language_server: Start tools in parallel (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15500">#15500</a>)
(Sysix)</li>
<li>3c57291 linter/plugins: Optimize loops (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15449">#15449</a>)
(overlookmotel)</li>
<li>3166233 linter/plugins: Remove <code>Arc</code>s (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15431">#15431</a>)
(overlookmotel)</li>
<li>9de1322 linter/plugins: Lazily deserialize settings JSON (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15395">#15395</a>)
(overlookmotel)</li>
<li>3049ec2 linter/plugins: Optimize <code>deepFreezeSettings</code> (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15392">#15392</a>)
(overlookmotel)</li>
<li>444ebfd linter/plugins: Use single object for
<code>parserServices</code> (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15378">#15378</a>)
(overlookmotel)</li>
</ul>
<h3>📚 Documentation</h3>
<ul>
<li>97d2104 linter: Update comment in lint.rs about default value for
tsconfig path (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15530">#15530</a>)
(Connor Shea)</li>
<li>2c6bd9e linter: Always refer as &quot;ES2015&quot; instead of
&quot;ES6&quot; (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15411">#15411</a>)
(sapphi-red)</li>
<li>a0c5203 linter/import/named: Update &quot;ES7&quot; comment in
examples (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15410">#15410</a>)
(sapphi-red)</li>
<li>3dc24b5 linter,minifier: Always refer as &quot;ES Modules&quot;
instead of &quot;ES6 Modules&quot; (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15409">#15409</a>)
(sapphi-red)</li>
<li>2ad77fb linter/no-this-before-super: Correct &quot;Why is this
bad?&quot; section (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15408">#15408</a>)
(sapphi-red)</li>
<li>57f0ce1 linter: Add backquotes where appropriate (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/15407">#15407</a>)
(sapphi-red)</li>
</ul>
<h1>Oxfmt v0.12.0</h1>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/oxc-project/oxc/blob/main/npm/oxlint/CHANGELOG.md">oxlint's
changelog</a>.</em></p>
<blockquote>
<h2>[1.60.0] - 2026-04-13</h2>
<h3>📚 Documentation</h3>
<ul>
<li>cfd8a4f linter: Don't rely on old eslint doc for available globals
(<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/21334">#21334</a>)
(Nicolas Le Cam)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/oxc-project/oxc/commit/e16848eb4c29e51e2e3e2bbda36c76ede42ab974"><code>e16848e</code></a>
release(apps): oxlint v1.60.0 &amp;&amp; oxfmt v0.45.0 (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/21375">#21375</a>)</li>
<li><a
href="https://github.com/oxc-project/oxc/commit/cfd8a4f7a46bb056e58f6d531caca2ec487948ba"><code>cfd8a4f</code></a>
docs(linter): don't rely on old eslint doc for available globals (<a
href="https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint/issues/21334">#21334</a>)</li>
<li>See full diff in <a
href="https://github.com/oxc-project/oxc/commits/oxlint_v1.60.0/npm/oxlint">compare
view</a></li>
</ul>
</details>
<br />

Updates `oxlint-tsgolint` from 0.20.0 to 0.21.1
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/oxc-project/tsgolint/releases">oxlint-tsgolint's
releases</a>.</em></p>
<blockquote>
<h2>v0.21.1</h2>
<h2>What's Changed</h2>
<ul>
<li>fix(no-unnecessary-condition): handle null overlap in narrowed
generic intersections by <a
href="https://github.com/camc314"><code>@​camc314</code></a> in <a
href="https://redirect.github.com/oxc-project/tsgolint/pull/891">oxc-project/tsgolint#891</a></li>
<li>revert(no-unnecessary-type-arguments): drop inference reporting by
<a href="https://github.com/camc314"><code>@​camc314</code></a> in <a
href="https://redirect.github.com/oxc-project/tsgolint/pull/892">oxc-project/tsgolint#892</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/oxc-project/tsgolint/compare/v0.21.0...v0.21.1">https://github.com/oxc-project/tsgolint/compare/v0.21.0...v0.21.1</a></p>
<h2>v0.21.0</h2>
<h2>What's Changed</h2>
<ul>
<li>chore: migrate gen-json-schemas to TS by <a
href="https://github.com/camc314"><code>@​camc314</code></a> in <a
href="https://redirect.github.com/oxc-project/tsgolint/pull/874">oxc-project/tsgolint#874</a></li>
<li>chore: update typescript-go submodule by <a
href="https://github.com/camc314"><code>@​camc314</code></a> in <a
href="https://redirect.github.com/oxc-project/tsgolint/pull/879">oxc-project/tsgolint#879</a></li>
<li>chore(deps): update github-actions by <a
href="https://github.com/renovate"><code>@​renovate</code></a>[bot] in
<a
href="https://redirect.github.com/oxc-project/tsgolint/pull/883">oxc-project/tsgolint#883</a></li>
<li>chore(deps): update gomod by <a
href="https://github.com/renovate"><code>@​renovate</code></a>[bot] in
<a
href="https://redirect.github.com/oxc-project/tsgolint/pull/884">oxc-project/tsgolint#884</a></li>
<li>chore(deps): update npm packages by <a
href="https://github.com/renovate"><code>@​renovate</code></a>[bot] in
<a
href="https://redirect.github.com/oxc-project/tsgolint/pull/885">oxc-project/tsgolint#885</a></li>
<li>feat: improve <code>consistent-type-exports</code> diagnostics
quality by <a
href="https://github.com/camchenry"><code>@​camchenry</code></a> in <a
href="https://redirect.github.com/oxc-project/tsgolint/pull/880">oxc-project/tsgolint#880</a></li>
<li>chore(deps): update softprops/action-gh-release action to v3 by <a
href="https://github.com/renovate"><code>@​renovate</code></a>[bot] in
<a
href="https://redirect.github.com/oxc-project/tsgolint/pull/886">oxc-project/tsgolint#886</a></li>
<li>feat: enrich the <code>no-array-delete</code> diagnostic by <a
href="https://github.com/camchenry"><code>@​camchenry</code></a> in <a
href="https://redirect.github.com/oxc-project/tsgolint/pull/881">oxc-project/tsgolint#881</a></li>
<li>feat: enrich <code>no-duplicate-type-constituents</code> diagnostic
by <a href="https://github.com/camchenry"><code>@​camchenry</code></a>
in <a
href="https://redirect.github.com/oxc-project/tsgolint/pull/882">oxc-project/tsgolint#882</a></li>
<li>fix(no-meaningless-void-operator): align with typescript-eslint
union handling by <a
href="https://github.com/camc314"><code>@​camc314</code></a> in <a
href="https://redirect.github.com/oxc-project/tsgolint/pull/887">oxc-project/tsgolint#887</a></li>
<li>chore(deps): update crate-ci/typos action to v1.45.1 by <a
href="https://github.com/renovate"><code>@​renovate</code></a>[bot] in
<a
href="https://redirect.github.com/oxc-project/tsgolint/pull/888">oxc-project/tsgolint#888</a></li>
<li>fix(no-deprecated): avoid false positive on array destructuring
bindings by <a
href="https://github.com/camc314"><code>@​camc314</code></a> in <a
href="https://redirect.github.com/oxc-project/tsgolint/pull/890">oxc-project/tsgolint#890</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/oxc-project/tsgolint/compare/v0.20.0...v0.21.0">https://github.com/oxc-project/tsgolint/compare/v0.20.0...v0.21.0</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/oxc-project/tsgolint/commit/9cb95a8635bd9650759c9bb59b9d87b1d76b2a36"><code>9cb95a8</code></a>
revert(no-unnecessary-type-arguments): drop inference reporting (<a
href="https://redirect.github.com/oxc-project/tsgolint/issues/892">#892</a>)</li>
<li><a
href="https://github.com/oxc-project/tsgolint/commit/f1049e402095722698f7364717b0ad03bf21edb1"><code>f1049e4</code></a>
fix(no-unnecessary-condition): handle null overlap in narrowed generic
inters...</li>
<li><a
href="https://github.com/oxc-project/tsgolint/commit/f9adc305c454101984c5f0886f0ce6a71e780d15"><code>f9adc30</code></a>
fix(no-deprecated): avoid false positive on array destructuring bindings
(<a
href="https://redirect.github.com/oxc-project/tsgolint/issues/890">#890</a>)</li>
<li><a
href="https://github.com/oxc-project/tsgolint/commit/260701f61d543d0d3fc31bb5760407d07e272ae8"><code>260701f</code></a>
chore(deps): update crate-ci/typos action to v1.45.1 (<a
href="https://redirect.github.com/oxc-project/tsgolint/issues/888">#888</a>)</li>
<li><a
href="https://github.com/oxc-project/tsgolint/commit/6dcc81a8ef17d2204e23ca4584a2c70248b30097"><code>6dcc81a</code></a>
fix(no-meaningless-void-operator): align with typescript-eslint union
handlin...</li>
<li><a
href="https://github.com/oxc-project/tsgolint/commit/97315a7a2555edfc398ac2ef1c83d6d89ee95cc7"><code>97315a7</code></a>
feat: enrich <code>no-duplicate-type-constituents</code> diagnostic (<a
href="https://redirect.github.com/oxc-project/tsgolint/issues/882">#882</a>)</li>
<li><a
href="https://github.com/oxc-project/tsgolint/commit/2b8447cde4907a7d131e85747c9289cde3a66cc2"><code>2b8447c</code></a>
feat: enrich the <code>no-array-delete</code> diagnostic (<a
href="https://redirect.github.com/oxc-project/tsgolint/issues/881">#881</a>)</li>
<li><a
href="https://github.com/oxc-project/tsgolint/commit/ae6a5493416f9c348a246d9686d73a075c5b797f"><code>ae6a549</code></a>
chore(deps): update softprops/action-gh-release action to v3 (<a
href="https://redirect.github.com/oxc-project/tsgolint/issues/886">#886</a>)</li>
<li><a
href="https://github.com/oxc-project/tsgolint/commit/1fe50bb1c6b381de333ca20fbd8beb12ec5b1d21"><code>1fe50bb</code></a>
feat: improve <code>consistent-type-exports</code> diagnostics quality
(<a
href="https://redirect.github.com/oxc-project/tsgolint/issues/880">#880</a>)</li>
<li><a
href="https://github.com/oxc-project/tsgolint/commit/834bbb81049885777cbaf51670344e7df32862de"><code>834bbb8</code></a>
chore(deps): update npm packages (<a
href="https://redirect.github.com/oxc-project/tsgolint/issues/885">#885</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/oxc-project/tsgolint/compare/v0.20.0...v0.21.1">compare
view</a></li>
</ul>
</details>
<br />

Updates `typescript` from 6.0.2 to 6.0.3
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/microsoft/TypeScript/releases">typescript's
releases</a>.</em></p>
<blockquote>
<h2>TypeScript 6.0.3</h2>
<!-- raw HTML omitted -->
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/microsoft/TypeScript/commit/050880ce59e30b356b686bd3144efe24f875ebc8"><code>050880c</code></a>
Bump version to 6.0.3 and LKG</li>
<li><a
href="https://github.com/microsoft/TypeScript/commit/eeae9dd0f17aa494658e4ec079dc002e02dd625e"><code>eeae9dd</code></a>
🤖 Pick PR <a
href="https://redirect.github.com/microsoft/TypeScript/issues/63401">#63401</a>
(Also check package name validity in...) into release-6.0 (#...</li>
<li><a
href="https://github.com/microsoft/TypeScript/commit/ad1c695fada682764bb510dd680e8f175ae54094"><code>ad1c695</code></a>
🤖 Pick PR <a
href="https://redirect.github.com/microsoft/TypeScript/issues/63368">#63368</a>
(Harden ATA package name filtering) into release-6.0 (<a
href="https://redirect.github.com/microsoft/TypeScript/issues/63372">#63372</a>)</li>
<li><a
href="https://github.com/microsoft/TypeScript/commit/0725fb4664a1d5ec94040b6d94db77dc1cc354e4"><code>0725fb4</code></a>
🤖 Pick PR <a
href="https://redirect.github.com/microsoft/TypeScript/issues/63310">#63310</a>
(Mark class property initializers as...) into release-6.0 (#...</li>
<li>See full diff in <a
href="https://github.com/microsoft/TypeScript/compare/v6.0.2...v6.0.3">compare
view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions


</details>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Alex <25013571+alexhb1@users.noreply.github.com>
2026-04-23 21:03:46 +01:00
dependabot[bot] f7f920683d Bump the gh-actions group with 2 updates (#898)
Bumps the gh-actions group with 2 updates:
[astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) and
[github/codeql-action](https://github.com/github/codeql-action).

Updates `astral-sh/setup-uv` from 8.0.0 to 8.1.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/astral-sh/setup-uv/releases">astral-sh/setup-uv's
releases</a>.</em></p>
<blockquote>
<h2>v8.1.0 🌈 New input <code>no-project</code></h2>
<h2>Changes</h2>
<p>This add the a new boolean input <code>no-project</code>.
It only makes sense to use in combination with
<code>activate-environment: true</code> and will append <code>--no
project</code> to the <code>uv venv</code> call. This is for example
useful <a
href="https://redirect.github.com/astral-sh/setup-uv/issues/854">if you
have a pyproject.toml file with parts unparseable by uv</a></p>
<h2>🚀 Enhancements</h2>
<ul>
<li>Add input no-project in combination with activate-environment <a
href="https://github.com/eifinger"><code>@​eifinger</code></a> (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/856">#856</a>)</li>
</ul>
<h2>🧰 Maintenance</h2>
<ul>
<li>fix: grant contents:write to validate-release job <a
href="https://github.com/eifinger"><code>@​eifinger</code></a> (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/860">#860</a>)</li>
<li>Add a release-gate step to the release workflow <a
href="https://github.com/zanieb"><code>@​zanieb</code></a> (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/859">#859</a>)</li>
<li>Draft commitish releases <a
href="https://github.com/eifinger"><code>@​eifinger</code></a> (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/858">#858</a>)</li>
<li>Add action-types.yml to instructions <a
href="https://github.com/eifinger"><code>@​eifinger</code></a> (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/857">#857</a>)</li>
<li>chore: update known checksums for 0.11.7 @<a
href="https://github.com/apps/github-actions">github-actions[bot]</a>
(<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/853">#853</a>)</li>
<li>Refactor version resolving <a
href="https://github.com/eifinger"><code>@​eifinger</code></a> (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/852">#852</a>)</li>
<li>chore: update known checksums for 0.11.6 @<a
href="https://github.com/apps/github-actions">github-actions[bot]</a>
(<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/850">#850</a>)</li>
<li>chore: update known checksums for 0.11.5 @<a
href="https://github.com/apps/github-actions">github-actions[bot]</a>
(<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/845">#845</a>)</li>
<li>chore: update known checksums for 0.11.4 @<a
href="https://github.com/apps/github-actions">github-actions[bot]</a>
(<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/843">#843</a>)</li>
<li>Add a release workflow <a
href="https://github.com/zanieb"><code>@​zanieb</code></a> (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/839">#839</a>)</li>
<li>chore: update known checksums for 0.11.3 @<a
href="https://github.com/apps/github-actions">github-actions[bot]</a>
(<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/836">#836</a>)</li>
</ul>
<h2>📚 Documentation</h2>
<ul>
<li>Update ignore-nothing-to-cache documentation <a
href="https://github.com/eifinger"><code>@​eifinger</code></a> (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/833">#833</a>)</li>
<li>Pin setup-uv docs to v8 <a
href="https://github.com/eifinger"><code>@​eifinger</code></a> (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/829">#829</a>)</li>
</ul>
<h2>⬆️ Dependency updates</h2>
<ul>
<li>chore(deps): bump release-drafter/release-drafter from 7.1.1 to
7.2.0 @<a href="https://github.com/apps/dependabot">dependabot[bot]</a>
(<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/855">#855</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/08807647e7069bb48b6ef5acd8ec9567f424441b"><code>0880764</code></a>
fix: grant contents:write to validate-release job (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/860">#860</a>)</li>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/717d6aba0f15312f509f5c4999e34d71ecbab8a9"><code>717d6ab</code></a>
Add a release-gate step to the release workflow (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/859">#859</a>)</li>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/5a911eb3a3983b5e650f2dad95c1ce698ca94378"><code>5a911eb</code></a>
Draft commitish releases (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/858">#858</a>)</li>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/080c31e04cd7155b0ca676d08c7bc260a4476a23"><code>080c31e</code></a>
Add action-types.yml to instructions (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/857">#857</a>)</li>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/b3e97d2ba1a1eed7e9d1f8456dd06c3b725bc3a6"><code>b3e97d2</code></a>
Add input no-project in combination with activate-environment (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/856">#856</a>)</li>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/7dd591db9557f680290587fcc578372813b9ff64"><code>7dd591d</code></a>
chore(deps): bump release-drafter/release-drafter from 7.1.1 to 7.2.0
(<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/855">#855</a>)</li>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/1541b7762698877904805605192ecd63d0e4787a"><code>1541b77</code></a>
chore: update known checksums for 0.11.7 (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/853">#853</a>)</li>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/cdfb2ee6dde255817c739680168ad81e184c4bfb"><code>cdfb2ee</code></a>
Refactor version resolving (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/852">#852</a>)</li>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/cb84d12dc6a0d495b82fcae14fa4559b90698660"><code>cb84d12</code></a>
chore: update known checksums for 0.11.6 (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/850">#850</a>)</li>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/1912cc65f2e839707d7a16f2372f30b57d35fd80"><code>1912cc6</code></a>
chore: update known checksums for 0.11.5 (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/845">#845</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/astral-sh/setup-uv/compare/cec208311dfd045dd5311c1add060b2062131d57...08807647e7069bb48b6ef5acd8ec9567f424441b">compare
view</a></li>
</ul>
</details>
<br />

Updates `github/codeql-action` from 4.35.1 to 4.35.2
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/github/codeql-action/releases">github/codeql-action's
releases</a>.</em></p>
<blockquote>
<h2>v4.35.2</h2>
<ul>
<li>The undocumented TRAP cache cleanup feature that could be enabled
using the <code>CODEQL_ACTION_CLEANUP_TRAP_CACHES</code> environment
variable is deprecated and will be removed in May 2026. If you are
affected by this, we recommend disabling TRAP caching by passing the
<code>trap-caching: false</code> input to the <code>init</code> Action.
<a
href="https://redirect.github.com/github/codeql-action/pull/3795">#3795</a></li>
<li>The Git version 2.36.0 requirement for improved incremental analysis
now only applies to repositories that contain submodules. <a
href="https://redirect.github.com/github/codeql-action/pull/3789">#3789</a></li>
<li>Python analysis on GHES no longer extracts the standard library,
relying instead on models of the standard library. This should result in
significantly faster extraction and analysis times, while the effect on
alerts should be minimal. <a
href="https://redirect.github.com/github/codeql-action/pull/3794">#3794</a></li>
<li>Fixed a bug in the validation of OIDC configurations for private
registries that was added in CodeQL Action 4.33.0 / 3.33.0. <a
href="https://redirect.github.com/github/codeql-action/pull/3807">#3807</a></li>
<li>Update default CodeQL bundle version to <a
href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.25.2">2.25.2</a>.
<a
href="https://redirect.github.com/github/codeql-action/pull/3823">#3823</a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/github/codeql-action/blob/main/CHANGELOG.md">github/codeql-action's
changelog</a>.</em></p>
<blockquote>
<h1>CodeQL Action Changelog</h1>
<p>See the <a
href="https://github.com/github/codeql-action/releases">releases
page</a> for the relevant changes to the CodeQL CLI and language
packs.</p>
<h2>[UNRELEASED]</h2>
<p>No user facing changes.</p>
<h2>4.35.2 - 15 Apr 2026</h2>
<ul>
<li>The undocumented TRAP cache cleanup feature that could be enabled
using the <code>CODEQL_ACTION_CLEANUP_TRAP_CACHES</code> environment
variable is deprecated and will be removed in May 2026. If you are
affected by this, we recommend disabling TRAP caching by passing the
<code>trap-caching: false</code> input to the <code>init</code> Action.
<a
href="https://redirect.github.com/github/codeql-action/pull/3795">#3795</a></li>
<li>The Git version 2.36.0 requirement for improved incremental analysis
now only applies to repositories that contain submodules. <a
href="https://redirect.github.com/github/codeql-action/pull/3789">#3789</a></li>
<li>Python analysis on GHES no longer extracts the standard library,
relying instead on models of the standard library. This should result in
significantly faster extraction and analysis times, while the effect on
alerts should be minimal. <a
href="https://redirect.github.com/github/codeql-action/pull/3794">#3794</a></li>
<li>Fixed a bug in the validation of OIDC configurations for private
registries that was added in CodeQL Action 4.33.0 / 3.33.0. <a
href="https://redirect.github.com/github/codeql-action/pull/3807">#3807</a></li>
<li>Update default CodeQL bundle version to <a
href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.25.2">2.25.2</a>.
<a
href="https://redirect.github.com/github/codeql-action/pull/3823">#3823</a></li>
</ul>
<h2>4.35.1 - 27 Mar 2026</h2>
<ul>
<li>Fix incorrect minimum required Git version for <a
href="https://redirect.github.com/github/roadmap/issues/1158">improved
incremental analysis</a>: it should have been 2.36.0, not 2.11.0. <a
href="https://redirect.github.com/github/codeql-action/pull/3781">#3781</a></li>
</ul>
<h2>4.35.0 - 27 Mar 2026</h2>
<ul>
<li>Reduced the minimum Git version required for <a
href="https://redirect.github.com/github/roadmap/issues/1158">improved
incremental analysis</a> from 2.38.0 to 2.11.0. <a
href="https://redirect.github.com/github/codeql-action/pull/3767">#3767</a></li>
<li>Update default CodeQL bundle version to <a
href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.25.1">2.25.1</a>.
<a
href="https://redirect.github.com/github/codeql-action/pull/3773">#3773</a></li>
</ul>
<h2>4.34.1 - 20 Mar 2026</h2>
<ul>
<li>Downgrade default CodeQL bundle version to <a
href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.24.3">2.24.3</a>
due to issues with a small percentage of Actions and JavaScript
analyses. <a
href="https://redirect.github.com/github/codeql-action/pull/3762">#3762</a></li>
</ul>
<h2>4.34.0 - 20 Mar 2026</h2>
<ul>
<li>Added an experimental change which disables TRAP caching when <a
href="https://redirect.github.com/github/roadmap/issues/1158">improved
incremental analysis</a> is enabled, since improved incremental analysis
supersedes TRAP caching. This will improve performance and reduce
Actions cache usage. We expect to roll this change out to everyone in
March. <a
href="https://redirect.github.com/github/codeql-action/pull/3569">#3569</a></li>
<li>We are rolling out improved incremental analysis to C/C++ analyses
that use build mode <code>none</code>. We expect this rollout to be
complete by the end of April 2026. <a
href="https://redirect.github.com/github/codeql-action/pull/3584">#3584</a></li>
<li>Update default CodeQL bundle version to <a
href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.25.0">2.25.0</a>.
<a
href="https://redirect.github.com/github/codeql-action/pull/3585">#3585</a></li>
</ul>
<h2>4.33.0 - 16 Mar 2026</h2>
<ul>
<li>
<p>Upcoming change: Starting April 2026, the CodeQL Action will skip
collecting file coverage information on pull requests to improve
analysis performance. File coverage information will still be computed
on non-PR analyses. Pull request analyses will log a warning about this
upcoming change. <a
href="https://redirect.github.com/github/codeql-action/pull/3562">#3562</a></p>
<p>To opt out of this change:</p>
<ul>
<li><strong>Repositories owned by an organization:</strong> Create a
custom repository property with the name
<code>github-codeql-file-coverage-on-prs</code> and the type
&quot;True/false&quot;, then set this property to <code>true</code> in
the repository's settings. For more information, see <a
href="https://docs.github.com/en/organizations/managing-organization-settings/managing-custom-properties-for-repositories-in-your-organization">Managing
custom properties for repositories in your organization</a>.
Alternatively, if you are using an advanced setup workflow, you can set
the <code>CODEQL_ACTION_FILE_COVERAGE_ON_PRS</code> environment variable
to <code>true</code> in your workflow.</li>
<li><strong>User-owned repositories using default setup:</strong> Switch
to an advanced setup workflow and set the
<code>CODEQL_ACTION_FILE_COVERAGE_ON_PRS</code> environment variable to
<code>true</code> in your workflow.</li>
<li><strong>User-owned repositories using advanced setup:</strong> Set
the <code>CODEQL_ACTION_FILE_COVERAGE_ON_PRS</code> environment variable
to <code>true</code> in your workflow.</li>
</ul>
</li>
<li>
<p>Fixed <a
href="https://redirect.github.com/github/codeql-action/issues/3555">a
bug</a> which caused the CodeQL Action to fail loading repository
properties if a &quot;Multi select&quot; repository property was
configured for the repository. <a
href="https://redirect.github.com/github/codeql-action/pull/3557">#3557</a></p>
</li>
<li>
<p>The CodeQL Action now loads <a
href="https://docs.github.com/en/organizations/managing-organization-settings/managing-custom-properties-for-repositories-in-your-organization">custom
repository properties</a> on GitHub Enterprise Server, enabling the
customization of features such as
<code>github-codeql-disable-overlay</code> that was previously only
available on GitHub.com. <a
href="https://redirect.github.com/github/codeql-action/pull/3559">#3559</a></p>
</li>
<li>
<p>Once <a
href="https://docs.github.com/en/code-security/how-tos/secure-at-scale/configure-organization-security/manage-usage-and-access/giving-org-access-private-registries">private
package registries</a> can be configured with OIDC-based authentication
for organizations, the CodeQL Action will now be able to accept such
configurations. <a
href="https://redirect.github.com/github/codeql-action/pull/3563">#3563</a></p>
</li>
<li>
<p>Fixed the retry mechanism for database uploads. Previously this would
fail with the error &quot;Response body object should not be disturbed
or locked&quot;. <a
href="https://redirect.github.com/github/codeql-action/pull/3564">#3564</a></p>
</li>
<li>
<p>A warning is now emitted if the CodeQL Action detects a repository
property whose name suggests that it relates to the CodeQL Action, but
which is not one of the properties recognised by the current version of
the CodeQL Action. <a
href="https://redirect.github.com/github/codeql-action/pull/3570">#3570</a></p>
</li>
</ul>
<h2>4.32.6 - 05 Mar 2026</h2>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/github/codeql-action/commit/95e58e9a2cdfd71adc6e0353d5c52f41a045d225"><code>95e58e9</code></a>
Merge pull request <a
href="https://redirect.github.com/github/codeql-action/issues/3824">#3824</a>
from github/update-v4.35.2-d2e135a73</li>
<li><a
href="https://github.com/github/codeql-action/commit/6f31bfe060e817d81e938dbec767969d20031e25"><code>6f31bfe</code></a>
Update changelog for v4.35.2</li>
<li><a
href="https://github.com/github/codeql-action/commit/d2e135a73a39154e3a231aeb49163c4661c5b8b1"><code>d2e135a</code></a>
Merge pull request <a
href="https://redirect.github.com/github/codeql-action/issues/3823">#3823</a>
from github/update-bundle/codeql-bundle-v2.25.2</li>
<li><a
href="https://github.com/github/codeql-action/commit/60abb65df09fcf213c398e064c8a80db1f15cdaf"><code>60abb65</code></a>
Add changelog note</li>
<li><a
href="https://github.com/github/codeql-action/commit/5a0a562209255e956ad8aafcee303294e64eefa2"><code>5a0a562</code></a>
Update default bundle to codeql-bundle-v2.25.2</li>
<li><a
href="https://github.com/github/codeql-action/commit/65216971a11ded447a6b76263d5a144519e5eee1"><code>6521697</code></a>
Merge pull request <a
href="https://redirect.github.com/github/codeql-action/issues/3820">#3820</a>
from github/dependabot/github_actions/dot-github/wor...</li>
<li><a
href="https://github.com/github/codeql-action/commit/3c45af2dd258e1623af1898da5c86545b514e028"><code>3c45af2</code></a>
Merge pull request <a
href="https://redirect.github.com/github/codeql-action/issues/3821">#3821</a>
from github/dependabot/npm_and_yarn/npm-minor-345b93...</li>
<li><a
href="https://github.com/github/codeql-action/commit/f1c339364c12f922998186ed897e45e3b4ae8874"><code>f1c3393</code></a>
Rebuild</li>
<li><a
href="https://github.com/github/codeql-action/commit/1024fc496c87e944a93e98d8cf2c09e2c7602a30"><code>1024fc4</code></a>
Rebuild</li>
<li><a
href="https://github.com/github/codeql-action/commit/9dd4cfed96030ccdfe1af4daf7a7964322704fed"><code>9dd4cfe</code></a>
Bump the npm-minor group across 1 directory with 6 updates</li>
<li>Additional commits viewable in <a
href="https://github.com/github/codeql-action/compare/c10b8064de6f491fea524254123dbe5e09572f13...95e58e9a2cdfd71adc6e0353d5c52f41a045d225">compare
view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-23 21:02:44 +01:00
Alex 819d794039 UI enhancement pass: Image streaming, dropdown UI fixes (#909)
- Added max image sizes and compression for frontend cached images
- Fixed various UI issues with dropdown placement and animation
2026-04-23 20:57:15 +01:00
Alex e0980a84d9 Fix temp filename max size (#912)
Fixes #891
2026-04-23 20:57:00 +01:00
Alex b8fdb2c841 Backend auth config check (#911)
Enforces auth config required fields, providing a warning if fields are
left empty upon save

Fixes #907
2026-04-23 20:56:45 +01:00
Alex 8bba0e28d7 Improved combined search behavior (#910)
- Added functionality to continue with a combined search if one option
is unfulfilled
- Added option to deselect currently selected release

Fixes #890
2026-04-23 20:56:32 +01:00
Alex 791ba184ee PrimaryTitle field and path template enhancements (#908)
- Added `PrimaryTitle` field for renaming templates
- Improved UI for creating a template, including live preview and custom
field picker

Fixes #905
2026-04-23 20:56:10 +01:00
Alex e35b4c47a7 Direct source refactor (#895)
- Updated mirror selection
- Removed built-in mirror options, users must provide their own
configurations
- Set Universal search to default, added ability to disable direct
source
- Updated documentation
- Updated makefile
2026-04-15 18:50:13 +01:00
blades 18a3f0bf44 Newznab capability (#867)
I've added a plugin using the same architecture as the prowlarr plugin
to enable Newznab as a source.
I've tested locally with nzbhydra2 and it all seems to work as intended.
I've added some unit tests for this feature, and found that a couple of
other unit tests weren't behaving so fixed those up while I was at it. I
also ran all of the linters in the makefile against it and fixed those
up, too, so hopefully this should be as clean and as compatible as it
can be.
2026-04-14 09:43:03 +01:00
Alex 0f7bcf8fd9 Fix lockfile (#888) 2026-04-13 11:40:41 +01:00
Alex 98f891916d Remove slash cleanup for OIDC discovery URL (#887)
Fixes #869
2026-04-13 11:21:44 +01:00
Alex a7694eb8ad Frontend tooling + Linter pass + React rule enforcement (#886)
- Added `oxlint`, `oxfmt`, `vitest`, `knip`
- Configured oxlint rules including react best practices (e.g. for
effect usage)
- Full linting pass on frontend code
- Full react rules pass on effect usage
- Full reformatting using oxfmt
- Full dead code cleanup using knip
2026-04-13 10:56:09 +01:00
Alex 87d5f127d6 Add prek + pytest-cov (#873) 2026-04-12 12:39:15 +01:00
Alex d7b9f2e67f Backend test hardening + quality enforcement (#872)
- Reworked many tests
- Enforcing lint + type checking for test suite
- Fixed various issues surfaced by the new tests
- CI tweaks
2026-04-12 12:01:52 +01:00
Alex 41c4aa1d72 Updated permissions model and non-root support (#871)
- Adds a non-root startup path at user 1000:1000 - skips privilege
escalation and ownership checks. Works e.g. for kubernetes deployments
(user 1000:1000 and runAsNonRoot enabled).
- Remove startup check/chown commands for user-owned folders. Checks can
be done with a "Test destination" button in settings which performs a
test write. Users are responsible for fixing their own permissions.
- Update docs
2026-04-12 08:22:32 +01:00
Alex 704da62202 Typecheck enforcement (#870)
All BasedPyright standard rules enforced and corrected
2026-04-12 07:35:36 +01:00
Alex 8d98e122ec Linter followup (#868)
Expanded Ruff rules and completed fixes
2026-04-11 16:18:52 +01:00
Alex 28eef75de0 Update path handling errors (#864) 2026-04-11 08:29:21 +01:00
Alex b3b8f34a13 Fix JSON script blocking behavior + tests (#862)
Fixes #859
2026-04-10 20:46:07 +01:00
Alex 8e78fea947 Add BasedPyright + Makefile commands (#858) 2026-04-10 17:12:54 +01:00
Alex 7bc6a9f8c6 Switch dev builds to nightly schedule (#856)
- `dev` images generated on a nightly schedule if new commits exist
- lite container smoke test added to PR CI checks
2026-04-10 15:12:23 +01:00
Alex 962e0ec68b Regenerate frontend lockfile (#855) 2026-04-10 14:33:47 +01:00
Alex 3d68b5eb2f Bump Vite to 8.0.8 + Typescript to 6.0.2 (#854) 2026-04-10 14:26:14 +01:00
dependabot[bot] ba4090aee2 Bump ruff from 0.15.9 to 0.15.10 (#848)
Bumps [ruff](https://github.com/astral-sh/ruff) from 0.15.9 to 0.15.10.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/astral-sh/ruff/releases">ruff's
releases</a>.</em></p>
<blockquote>
<h2>0.15.10</h2>
<h2>Release Notes</h2>
<p>Released on 2026-04-09.</p>
<h3>Preview features</h3>
<ul>
<li>[<code>flake8-logging</code>] Allow closures in except handlers
(<code>LOG004</code>) (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24464">#24464</a>)</li>
<li>[<code>flake8-self</code>] Make <code>SLF</code> diagnostics robust
to non-self-named variables (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24281">#24281</a>)</li>
<li>[<code>flake8-simplify</code>] Make the fix for
<code>collapsible-if</code> safe in <code>preview</code>
(<code>SIM102</code>) (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24371">#24371</a>)</li>
</ul>
<h3>Bug fixes</h3>
<ul>
<li>Avoid emitting multi-line f-string elements before Python 3.12 (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24377">#24377</a>)</li>
<li>Avoid syntax error from <code>E502</code> fixes in f-strings and
t-strings (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24410">#24410</a>)</li>
<li>Strip form feeds from indent passed to <code>dedent_to</code> (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24381">#24381</a>)</li>
<li>[<code>pyupgrade</code>] Fix panic caused by handling of octals
(<code>UP012</code>) (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24390">#24390</a>)</li>
<li>Reject multi-line f-string elements before Python 3.12 (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24355">#24355</a>)</li>
</ul>
<h3>Rule changes</h3>
<ul>
<li>[<code>ruff</code>] Treat f-string interpolation as potential side
effect (<code>RUF019</code>) (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24426">#24426</a>)</li>
</ul>
<h3>Server</h3>
<ul>
<li>Add support for custom file extensions (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24463">#24463</a>)</li>
</ul>
<h3>Documentation</h3>
<ul>
<li>Document adding fixes in CONTRIBUTING.md (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24393">#24393</a>)</li>
<li>Fix JSON typo in settings example (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24517">#24517</a>)</li>
</ul>
<h3>Contributors</h3>
<ul>
<li><a
href="https://github.com/charliermarsh"><code>@​charliermarsh</code></a></li>
<li><a href="https://github.com/dylwil3"><code>@​dylwil3</code></a></li>
<li><a
href="https://github.com/silverstein"><code>@​silverstein</code></a></li>
<li><a
href="https://github.com/anishgirianish"><code>@​anishgirianish</code></a></li>
<li><a
href="https://github.com/shizukushq"><code>@​shizukushq</code></a></li>
<li><a href="https://github.com/zanieb"><code>@​zanieb</code></a></li>
<li><a
href="https://github.com/AlexWaygood"><code>@​AlexWaygood</code></a></li>
</ul>
<h2>Install ruff 0.15.10</h2>
<h3>Install prebuilt binaries via shell script</h3>
<pre lang="sh"><code>curl --proto '=https' --tlsv1.2 -LsSf
https://releases.astral.sh/github/ruff/releases/download/0.15.10/ruff-installer.sh
| sh
</code></pre>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md">ruff's
changelog</a>.</em></p>
<blockquote>
<h2>0.15.10</h2>
<p>Released on 2026-04-09.</p>
<h3>Preview features</h3>
<ul>
<li>[<code>flake8-logging</code>] Allow closures in except handlers
(<code>LOG004</code>) (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24464">#24464</a>)</li>
<li>[<code>flake8-self</code>] Make <code>SLF</code> diagnostics robust
to non-self-named variables (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24281">#24281</a>)</li>
<li>[<code>flake8-simplify</code>] Make the fix for
<code>collapsible-if</code> safe in <code>preview</code>
(<code>SIM102</code>) (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24371">#24371</a>)</li>
</ul>
<h3>Bug fixes</h3>
<ul>
<li>Avoid emitting multi-line f-string elements before Python 3.12 (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24377">#24377</a>)</li>
<li>Avoid syntax error from <code>E502</code> fixes in f-strings and
t-strings (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24410">#24410</a>)</li>
<li>Strip form feeds from indent passed to <code>dedent_to</code> (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24381">#24381</a>)</li>
<li>[<code>pyupgrade</code>] Fix panic caused by handling of octals
(<code>UP012</code>) (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24390">#24390</a>)</li>
<li>Reject multi-line f-string elements before Python 3.12 (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24355">#24355</a>)</li>
</ul>
<h3>Rule changes</h3>
<ul>
<li>[<code>ruff</code>] Treat f-string interpolation as potential side
effect (<code>RUF019</code>) (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24426">#24426</a>)</li>
</ul>
<h3>Server</h3>
<ul>
<li>Add support for custom file extensions (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24463">#24463</a>)</li>
</ul>
<h3>Documentation</h3>
<ul>
<li>Document adding fixes in CONTRIBUTING.md (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24393">#24393</a>)</li>
<li>Fix JSON typo in settings example (<a
href="https://redirect.github.com/astral-sh/ruff/pull/24517">#24517</a>)</li>
</ul>
<h3>Contributors</h3>
<ul>
<li><a
href="https://github.com/charliermarsh"><code>@​charliermarsh</code></a></li>
<li><a href="https://github.com/dylwil3"><code>@​dylwil3</code></a></li>
<li><a
href="https://github.com/silverstein"><code>@​silverstein</code></a></li>
<li><a
href="https://github.com/anishgirianish"><code>@​anishgirianish</code></a></li>
<li><a
href="https://github.com/shizukushq"><code>@​shizukushq</code></a></li>
<li><a href="https://github.com/zanieb"><code>@​zanieb</code></a></li>
<li><a
href="https://github.com/AlexWaygood"><code>@​AlexWaygood</code></a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/astral-sh/ruff/commit/252f76102a618bff6537b6c53c316ca3837f4abf"><code>252f761</code></a>
Bump 0.15.10 (<a
href="https://redirect.github.com/astral-sh/ruff/issues/24519">#24519</a>)</li>
<li><a
href="https://github.com/astral-sh/ruff/commit/37a1ec8bb8e30955787b0cdf6e97f7f2254dba7f"><code>37a1ec8</code></a>
[ty] Fix assignability of intersections with bounded typevars (<a
href="https://redirect.github.com/astral-sh/ruff/issues/24502">#24502</a>)</li>
<li><a
href="https://github.com/astral-sh/ruff/commit/f518cc9ca0c830773dd49c3964eb5e49d52c8aed"><code>f518cc9</code></a>
[ty] Allow partially stringified <code>type[…]</code> annotations (<a
href="https://redirect.github.com/astral-sh/ruff/issues/24518">#24518</a>)</li>
<li><a
href="https://github.com/astral-sh/ruff/commit/16c4090d0a711b9c0523b932014f3daf140f35bc"><code>16c4090</code></a>
docs: fix JSON typo in settings example (<a
href="https://redirect.github.com/astral-sh/ruff/issues/24517">#24517</a>)</li>
<li><a
href="https://github.com/astral-sh/ruff/commit/99d97bd72f1934ac2af93e52468c10ef1c7a1a4e"><code>99d97bd</code></a>
[ty] Tighten up a few edge cases in <code>Concatenate</code>
type-expression parsing (<a
href="https://redirect.github.com/astral-sh/ruff/issues/2">#2</a>...</li>
<li><a
href="https://github.com/astral-sh/ruff/commit/2714e345bdd64a5baae3844c0d25db7b0b9fe330"><code>2714e34</code></a>
[ty] Enable <code>pull-diagnostics</code> by default in E2E tests (<a
href="https://redirect.github.com/astral-sh/ruff/issues/24516">#24516</a>)</li>
<li><a
href="https://github.com/astral-sh/ruff/commit/d8bc700722ab1b7272a4d724839da7c569b349d4"><code>d8bc700</code></a>
LSP: Add support for custom extensions (<a
href="https://redirect.github.com/astral-sh/ruff/issues/24463">#24463</a>)</li>
<li><a
href="https://github.com/astral-sh/ruff/commit/a45f96d65dbd4f958b07accd718f8d2af48cb956"><code>a45f96d</code></a>
[ty] stop special-casing str constructor (<a
href="https://redirect.github.com/astral-sh/ruff/issues/24514">#24514</a>)</li>
<li><a
href="https://github.com/astral-sh/ruff/commit/87a0f01cfd016e0297ef05ab638cde006bf8d947"><code>87a0f01</code></a>
[ruff] Treat f-string interpolation as potential side effect in RUF019
(<a
href="https://redirect.github.com/astral-sh/ruff/issues/24426">#24426</a>)</li>
<li><a
href="https://github.com/astral-sh/ruff/commit/e9ba8489b8d1f1fd5fd66887a74d5f2f58f733d4"><code>e9ba848</code></a>
[ty] Fix excess subscript argument inference for non-generic types (<a
href="https://redirect.github.com/astral-sh/ruff/issues/24354">#24354</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/astral-sh/ruff/compare/0.15.9...0.15.10">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=ruff&package-manager=uv&previous-version=0.15.9&new-version=0.15.10)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-10 13:40:28 +01:00
dependabot[bot] 03ec7d1c06 Bump the gh-actions group across 1 directory with 2 updates (#851)
Bumps the gh-actions group with 2 updates in the / directory:
[docker/build-push-action](https://github.com/docker/build-push-action)
and [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv).

Updates `docker/build-push-action` from 7.0.0 to 7.1.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/docker/build-push-action/releases">docker/build-push-action's
releases</a>.</em></p>
<blockquote>
<h2>v7.1.0</h2>
<ul>
<li>Git context <a
href="https://docs.docker.com/build/concepts/context/#url-queries">query
format</a> support by <a
href="https://github.com/crazy-max"><code>@​crazy-max</code></a> in <a
href="https://redirect.github.com/docker/build-push-action/pull/1505">docker/build-push-action#1505</a></li>
<li>Bump <code>@​docker/actions-toolkit</code> from 0.79.0 to 0.87.0 by
<a href="https://github.com/crazy-max"><code>@​crazy-max</code></a> in
<a
href="https://redirect.github.com/docker/build-push-action/pull/1505">docker/build-push-action#1505</a></li>
<li>Bump brace-expansion from 1.1.12 to 1.1.13 in <a
href="https://redirect.github.com/docker/build-push-action/pull/1500">docker/build-push-action#1500</a></li>
<li>Bump fast-xml-parser from 5.4.2 to 5.5.7 in <a
href="https://redirect.github.com/docker/build-push-action/pull/1489">docker/build-push-action#1489</a></li>
<li>Bump flatted from 3.3.3 to 3.4.2 in <a
href="https://redirect.github.com/docker/build-push-action/pull/1491">docker/build-push-action#1491</a></li>
<li>Bump glob from 10.3.12 to 10.5.0 in <a
href="https://redirect.github.com/docker/build-push-action/pull/1490">docker/build-push-action#1490</a></li>
<li>Bump handlebars from 4.7.8 to 4.7.9 in <a
href="https://redirect.github.com/docker/build-push-action/pull/1497">docker/build-push-action#1497</a></li>
<li>Bump lodash from 4.17.23 to 4.18.1 in <a
href="https://redirect.github.com/docker/build-push-action/pull/1510">docker/build-push-action#1510</a></li>
<li>Bump picomatch from 4.0.3 to 4.0.4 in <a
href="https://redirect.github.com/docker/build-push-action/pull/1496">docker/build-push-action#1496</a></li>
<li>Bump undici from 6.23.0 to 6.24.1 in <a
href="https://redirect.github.com/docker/build-push-action/pull/1486">docker/build-push-action#1486</a></li>
<li>Bump vite from 7.3.1 to 7.3.2 in <a
href="https://redirect.github.com/docker/build-push-action/pull/1509">docker/build-push-action#1509</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/docker/build-push-action/compare/v7.0.0...v7.1.0">https://github.com/docker/build-push-action/compare/v7.0.0...v7.1.0</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/docker/build-push-action/commit/bcafcacb16a39f128d818304e6c9c0c18556b85f"><code>bcafcac</code></a>
Merge pull request <a
href="https://redirect.github.com/docker/build-push-action/issues/1509">#1509</a>
from docker/dependabot/npm_and_yarn/vite-7.3.2</li>
<li><a
href="https://github.com/docker/build-push-action/commit/18e62f1158d9c45a4a84a58a6828d21f8ed3644b"><code>18e62f1</code></a>
Merge pull request <a
href="https://redirect.github.com/docker/build-push-action/issues/1510">#1510</a>
from docker/dependabot/npm_and_yarn/lodash-4.18.1</li>
<li><a
href="https://github.com/docker/build-push-action/commit/46580d2c9d43b0888270cb6fa90956e483de56fc"><code>46580d2</code></a>
chore: update generated content</li>
<li><a
href="https://github.com/docker/build-push-action/commit/3f80b252ca2331f6ec3e890f4346b5506ee1dc81"><code>3f80b25</code></a>
chore(deps): Bump lodash from 4.17.23 to 4.18.1</li>
<li><a
href="https://github.com/docker/build-push-action/commit/efeec9557c40a646afe433e39a1e94ca689103f0"><code>efeec95</code></a>
Merge pull request <a
href="https://redirect.github.com/docker/build-push-action/issues/1505">#1505</a>
from crazy-max/refactor-git-context</li>
<li><a
href="https://github.com/docker/build-push-action/commit/ddf04b08eb12882258ed936fea4a2806754ff349"><code>ddf04b0</code></a>
Merge pull request <a
href="https://redirect.github.com/docker/build-push-action/issues/1511">#1511</a>
from docker/dependabot/github_actions/crazy-max-dot-...</li>
<li><a
href="https://github.com/docker/build-push-action/commit/db08d97a08e4a0d15f85d1c4e64dfd5f88cbe1a9"><code>db08d97</code></a>
chore(deps): Bump the crazy-max-dot-github group with 2 updates</li>
<li><a
href="https://github.com/docker/build-push-action/commit/ef1fb9688fc3626d0fd5e462f502cbbdc6456feb"><code>ef1fb96</code></a>
Merge pull request <a
href="https://redirect.github.com/docker/build-push-action/issues/1508">#1508</a>
from docker/dependabot/github_actions/docker/login-a...</li>
<li><a
href="https://github.com/docker/build-push-action/commit/2d8f2a1a378a5c302dcd7b2b4326cefa24180bb1"><code>2d8f2a1</code></a>
chore: update generated content</li>
<li><a
href="https://github.com/docker/build-push-action/commit/919ac7bd7d1aa8cb13fe4de76545abea8d8b5ed2"><code>919ac7b</code></a>
fix test since secrets are not written to temp path anymore</li>
<li>Additional commits viewable in <a
href="https://github.com/docker/build-push-action/compare/d08e5c354a6adb9ed34480a06d141179aa583294...bcafcacb16a39f128d818304e6c9c0c18556b85f">compare
view</a></li>
</ul>
</details>
<br />

Updates `astral-sh/setup-uv` from 7.3.1 to 8.0.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/astral-sh/setup-uv/releases">astral-sh/setup-uv's
releases</a>.</em></p>
<blockquote>
<h2>v8.0.0 🌈 Immutable releases and secure tags</h2>
<h1>This is the first immutable release of <code>setup-uv</code> 🥳</h1>
<p>All future releases are also immutable, if you want to know more
about what this means checkout <a
href="https://docs.github.com/en/code-security/concepts/supply-chain-security/immutable-releases">the
docs</a>.</p>
<p>This release also has two breaking changes</p>
<h2>New format for <code>manifest-file</code></h2>
<p>The previously deprecated way of defining a custom version manifest
to control which <code>uv</code> versions are available and where to
download them from got removed. The functionality is still there but you
have to use the <a
href="https://github.com/astral-sh/setup-uv/blob/main/docs/customization.md#format">new
format</a>.</p>
<h2>No more major and minor tags</h2>
<p>To increase <strong>security</strong> even more we will <strong>stop
publishing minor tags</strong>. You won't be able to use
<code>@v8</code> or <code>@v8.0</code> any longer. We do this because
pinning to major releases opens up users to supply chain attacks like
what happened to <a
href="https://unit42.paloaltonetworks.com/github-actions-supply-chain-attack/">tj-actions</a>.</p>
<blockquote>
<p>[!TIP]
Use the immutable tag as a version
<code>astral-sh/setup-uv@v8.0.0</code>
Or even better the githash
<code>astral-sh/setup-uv@cec208311dfd045dd5311c1add060b2062131d57</code></p>
</blockquote>
<h2>🚨 Breaking changes</h2>
<ul>
<li>Remove update-major-minor-tags workflow <a
href="https://github.com/eifinger"><code>@​eifinger</code></a> (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/826">#826</a>)</li>
<li>Remove deprecrated custom manifest <a
href="https://github.com/eifinger"><code>@​eifinger</code></a> (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/813">#813</a>)</li>
</ul>
<h2>🧰 Maintenance</h2>
<ul>
<li>Shortcircuit latest version from manifest <a
href="https://github.com/eifinger"><code>@​eifinger</code></a> (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/828">#828</a>)</li>
<li>Simplify inputs.ts <a
href="https://github.com/eifinger"><code>@​eifinger</code></a> (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/827">#827</a>)</li>
<li>Bump release-drafter to v7.1.1 <a
href="https://github.com/eifinger"><code>@​eifinger</code></a> (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/825">#825</a>)</li>
<li>Refactor inputs <a
href="https://github.com/eifinger"><code>@​eifinger</code></a> (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/823">#823</a>)</li>
<li>Replace inline compile args with tsconfig <a
href="https://github.com/eifinger"><code>@​eifinger</code></a> (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/824">#824</a>)</li>
<li>chore: update known checksums for 0.11.2 @<a
href="https://github.com/apps/github-actions">github-actions[bot]</a>
(<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/821">#821</a>)</li>
<li>chore: update known checksums for 0.11.1 @<a
href="https://github.com/apps/github-actions">github-actions[bot]</a>
(<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/817">#817</a>)</li>
<li>chore: update known checksums for 0.11.0 @<a
href="https://github.com/apps/github-actions">github-actions[bot]</a>
(<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/815">#815</a>)</li>
<li>Fix latest-version workflow check <a
href="https://github.com/eifinger"><code>@​eifinger</code></a> (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/812">#812</a>)</li>
<li>chore: update known checksums for 0.10.11/0.10.12 @<a
href="https://github.com/apps/github-actions">github-actions[bot]</a>
(<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/811">#811</a>)</li>
</ul>
<h2>v7.6.0 🌈 Fetch uv from Astral's mirror by default</h2>
<h2>Changes</h2>
<p>We now default to download uv from <code>releases.astral.sh</code>.
This means by default we don't hit the GitHub API at all and shouldn't
see any rate limits and timeouts any more.</p>
<h2>🚀 Enhancements</h2>
<ul>
<li>Fetch uv from Astral's mirror by default <a
href="https://github.com/zsol"><code>@​zsol</code></a> (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/809">#809</a>)</li>
</ul>
<h2>🧰 Maintenance</h2>
<ul>
<li>Switch to ESM for source and test, use CommonJS for dist <a
href="https://github.com/eifinger"><code>@​eifinger</code></a> (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/806">#806</a>)</li>
<li>chore: update known checksums for 0.10.10 @<a
href="https://github.com/apps/github-actions">github-actions[bot]</a>
(<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/804">#804</a>)</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/cec208311dfd045dd5311c1add060b2062131d57"><code>cec2083</code></a>
Shortcircuit latest version from manifest (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/828">#828</a>)</li>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/4dd8ab45206a76f8c1dfe399fa88df10a7264f27"><code>4dd8ab4</code></a>
Simplify inputs.ts (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/827">#827</a>)</li>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/7fdbe7cf0c8ef50cfd0878eed7b5180abc6b53c7"><code>7fdbe7c</code></a>
Remove update-major-minor-tags workflow (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/826">#826</a>)</li>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/485abd05e5c74a247f0a309e333d2433ab9a353a"><code>485abd0</code></a>
Bump release-drafter to v7.1.1 (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/825">#825</a>)</li>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/f82eb19c06057c455674b2602e0139fd906f1428"><code>f82eb19</code></a>
Refactor inputs (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/823">#823</a>)</li>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/868d1f74d9d862d7b40219546bfe35299c6dd452"><code>868d1f7</code></a>
Replace inline compile args with tsconfig (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/824">#824</a>)</li>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/447e6d02b15d65b3247cce2d6019f11957285d11"><code>447e6d0</code></a>
chore: update known checksums for 0.11.2 (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/821">#821</a>)</li>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/5c62c5926145985eec91f09e2e0a75f40daed929"><code>5c62c59</code></a>
chore: update known checksums for 0.11.1 (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/817">#817</a>)</li>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/e1a7373adb857afd2a70b971e8ebdacc64ed27d0"><code>e1a7373</code></a>
chore: update known checksums for 0.11.0 (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/815">#815</a>)</li>
<li><a
href="https://github.com/astral-sh/setup-uv/commit/89709315bb3bd4bf0f4b1db4b710e99009087ab5"><code>8970931</code></a>
Remove deprecrated custom manifest (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/813">#813</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/astral-sh/setup-uv/compare/5a095e7a2014a4212f075830d4f7277575a9d098...cec208311dfd045dd5311c1add060b2062131d57">compare
view</a></li>
</ul>
</details>
<br />

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-10 13:40:18 +01:00
dependabot[bot] 71ee56b7b7 Bump the npm-deps group across 1 directory with 4 updates (#852)
Bumps the npm-deps group with 4 updates in the /src/frontend directory:
[react](https://github.com/facebook/react/tree/HEAD/packages/react),
[react-dom](https://github.com/facebook/react/tree/HEAD/packages/react-dom),
[@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node)
and [postcss](https://github.com/postcss/postcss).

Updates `react` from 19.2.4 to 19.2.5
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/facebook/react/releases">react's
releases</a>.</em></p>
<blockquote>
<h2>19.2.5 (April 8th, 2026)</h2>
<h2>React Server Components</h2>
<ul>
<li>Add more cycle protections (<a
href="https://redirect.github.com/facebook/react/pull/36236">#36236</a>
by <a href="https://github.com/eps1lon"><code>@​eps1lon</code></a> and
<a
href="https://github.com/unstubbable"><code>@​unstubbable</code></a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/facebook/react/commit/23f4f9f30da9e9af2108c18bb197bae75ab584ea"><code>23f4f9f</code></a>
19.2.5</li>
<li>See full diff in <a
href="https://github.com/facebook/react/commits/v19.2.5/packages/react">compare
view</a></li>
</ul>
</details>
<br />

Updates `react-dom` from 19.2.4 to 19.2.5
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/facebook/react/releases">react-dom's
releases</a>.</em></p>
<blockquote>
<h2>19.2.5 (April 8th, 2026)</h2>
<h2>React Server Components</h2>
<ul>
<li>Add more cycle protections (<a
href="https://redirect.github.com/facebook/react/pull/36236">#36236</a>
by <a href="https://github.com/eps1lon"><code>@​eps1lon</code></a> and
<a
href="https://github.com/unstubbable"><code>@​unstubbable</code></a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/facebook/react/commit/23f4f9f30da9e9af2108c18bb197bae75ab584ea"><code>23f4f9f</code></a>
19.2.5</li>
<li>See full diff in <a
href="https://github.com/facebook/react/commits/v19.2.5/packages/react-dom">compare
view</a></li>
</ul>
</details>
<br />

Updates `@types/node` from 25.5.0 to 25.6.0
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a
href="https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node">compare
view</a></li>
</ul>
</details>
<br />

Updates `postcss` from 8.5.8 to 8.5.9
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/postcss/postcss/releases">postcss's
releases</a>.</em></p>
<blockquote>
<h2>8.5.9</h2>
<ul>
<li>Speed up source map encoding paring in case of the error.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/postcss/postcss/blob/main/CHANGELOG.md">postcss's
changelog</a>.</em></p>
<blockquote>
<h2>8.5.9</h2>
<ul>
<li>Speed up source map encoding paring in case of the error.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/postcss/postcss/commit/fe88ac29c06b7b218be32994cdc6ca1525bdf2c9"><code>fe88ac2</code></a>
Release 8.5.9 version</li>
<li><a
href="https://github.com/postcss/postcss/commit/c551632496b87ab3f1965bfda5dc386b6c71963e"><code>c551632</code></a>
Avoid RegExp when we can use simple JS</li>
<li><a
href="https://github.com/postcss/postcss/commit/89a6b744060eb8dee743351c785a9fbe37d4525a"><code>89a6b74</code></a>
Move SECURITY.txt for docs folder to keep GitHub page cleaner</li>
<li><a
href="https://github.com/postcss/postcss/commit/6ceb8a46af9f9de821faee98f861bdf84617347b"><code>6ceb8a4</code></a>
Create SECURITY.md</li>
<li><a
href="https://github.com/postcss/postcss/commit/02ccae6bc359ee386f84d8fbc36eb423acbdbc3f"><code>02ccae6</code></a>
Another way to fix CI with .ts ext in tests on old Node.js</li>
<li><a
href="https://github.com/postcss/postcss/commit/2c36658c7cd4ffee4d42fafd0a501b9c13b63a52"><code>2c36658</code></a>
Another way to fix CI with TS on old Node.js</li>
<li><a
href="https://github.com/postcss/postcss/commit/b90600361f33a3834fd708831819cb1106590908"><code>b906003</code></a>
Another way to fix CI with old Node.js</li>
<li><a
href="https://github.com/postcss/postcss/commit/04d32cdc69ac598fcd33d3d748de12c4c32e83fb"><code>04d32cd</code></a>
Fix another issue with Node.js 10 on CI</li>
<li><a
href="https://github.com/postcss/postcss/commit/df86cdfff016205260901394432574ecd2a2fa65"><code>df86cdf</code></a>
Try to fix Node.js 10 on CI</li>
<li><a
href="https://github.com/postcss/postcss/commit/82bec0dd8e194f653743f606e6a2b7e27cde9088"><code>82bec0d</code></a>
Move to oxfmt</li>
<li>Additional commits viewable in <a
href="https://github.com/postcss/postcss/compare/8.5.8...8.5.9">compare
view</a></li>
</ul>
</details>
<br />

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-10 13:40:08 +01:00
Alex e3d5bd91fc Bump node to 24 LTS (#853) 2026-04-10 13:29:48 +01:00
Alex af38540991 Group dependabot PRs (#849) 2026-04-10 13:07:59 +01:00
Alex 3a3a3ce449 Add new python tooling + apply ruff linter cleanup (#845)
- Adds `uv`, `ruff`, `pyright`, `vulture` and `pytest-xdist`
- Move project, lockfile, docker build etc to uv
- Align python tooling on 3.14
- Huge bulk of ruff linter fixes applied. Still in progress but all the
core types are now enforced
- Update CI and test helpers
2026-04-10 13:03:25 +01:00
Alex ff094bed56 Fix: Orchestrator timeout and exception handling (#832)
Fixes #823
2026-04-03 09:38:58 +01:00
Alex c1143f808a Batch dependency updates (#831)
- github/codeql-action 3.32.6 → 4.35.1
- seleniumbase 4.47.3 → 4.47.9
- react-router-dom 7.13.1 → 7.14.0
- docker/login-action 3.7.0 → 4.1.0
- tailwindcss 4.2.1 → 4.2.2
- @tailwindcss/vite 4.2.1 → 4.2.2
- @tailwindcss/postcss 4.2.1 → 4.2.2
- actions/setup-python 5.6.0 → 6.2.0
- docker/setup-buildx-action 3.12.0 → 4.0.0
2026-04-03 09:36:31 +01:00
Alex 9bfcf828ea Fixes: Env variable config usage, retry availability, Entrypoint permissions (#817)
- Clean up a few uses of config options that may miss the env variable
if this is set
- Add enhanced retry availability utilising the DB to persist download
errors / retries across restarts, request failures, and pass Prowlarr
detail through the download task to maintain retry data.
- Strip back entrypoint permissions for less intensive chown operations.

Fixes #796
2026-03-29 16:39:40 +01:00
Alex 678c54cba2 Fixes: Entrypoint, seedtime, request policy flow (#805)
- Added a path for rootless permissions in the entrypoint script
- Routed prowlarr searches through torznab for seedtime info
- Added additional request flow for download permissions
2026-03-25 18:34:42 +00:00
Alex 019d36b27e Rename Booklore to Grimmory 2 (#792) 2026-03-21 15:48:06 +00:00
Alex 698eb07e71 Rename Booklore to Grimmory (#791) 2026-03-21 15:38:38 +00:00
Alex 8f949a73d5 Remove audible provider (#778) 2026-03-18 18:29:13 +00:00
Alex 2c6f46fc88 Combined mode follow-up (#777) 2026-03-18 18:27:39 +00:00
Alex 3f90c3805f Use title+author query for whitelisted indexers (#774) 2026-03-16 19:52:43 +00:00
Alex cb093f61c6 Feature: Combined book+audiobook downloads (#773)
- Adds a combined search option in the search bar selector
- Choose both a book and audiobook file in a two-step release modal, and
download both simultaneously from a single search result.
- Works for requests. Request both a book+audiobook at once, or works
seamlessly with request policies that differ between book + audiobook
(E.g. automatically download the ebook portion, while the audiobook gets
sent as a request)
- Hidden for users who have book or audiobooks blocked. 

Closes #611
2026-03-16 18:34:46 +00:00
Alex b464d62672 Fix entrypoint gosu test write (#772)
- Let gosu run the full test write
- Delete stale compose

Fixes #771 - gosu pipe broken on Proxmox LXC installs
2026-03-15 18:04:41 +00:00
Alex f3f26488b1 Fix TS error (#770) 2026-03-15 10:27:03 +00:00
Alex fec9d31c8a Frontend improvements for Audible provider (#769)
- Added square artwork support
- Added dedicated length and narrator icons
- Added audiobook info to release and details modals
- Moved search options button to accommodate larger Audible search
fields
2026-03-15 10:17:37 +00:00
cadric 3295be82a7 Add Audible metadata provider via Audimeta (#762)
Closes #515

  ## Summary

This adds a new `audible` metadata provider backed by the Audimeta API.

  The provider supports:
  - Audible/Audimeta metadata lookup without authentication
- region selection (`us`, `ca`, `uk`, `au`, `fr`, `de`, `jp`, `it`,
`in`, `es`, `br`)
  - ASIN book lookup
  - ISBN lookup with fallback search
  - series suggestions and series-order browsing
- richer audiobook metadata such as narrators, runtime, rating,
subtitle, cover, publisher, and series info
- configurable Audimeta base URL, timeout, cache usage, default sort,
and unreleased filtering

  ## Notes

  A few Audimeta-specific integration details were needed:
- send a meaningful `User-Agent`, otherwise Audimeta rejects requests
with `403`
  - send the `cache` parameter in the format Audimeta expects
- use `keywords` for general search instead of `query`, which gave
poor/irrelevant results for title-style
  searches

  ## Validation

  Tested locally with:
  - `python -m py_compile shelfmark/metadata_providers/audible.py`
  - `python -m pytest tests/metadata/test_audible.py -v`
- `python -m pytest
tests/metadata/test_metadata_provider_capabilities.py -v`

  Also verified manually in a Podman test container:
  - searching for `Discount Dan` returns Audible title `B0DXLXRNGG`
  - book details and series metadata load correctly

  ## Scope

This PR intentionally keeps the change localized to the provider layer
and docs:
  - new Audible provider
  - provider registration
  - provider docs
  - generated environment variable docs
2026-03-15 10:09:24 +00:00
Alex fff0fd07a1 Fix stale activity dismiss handling (#768)
Fixes #764
2026-03-15 10:08:45 +00:00
Alexanddependabot[bot] 3f1a14843b Update to React 19 (#766)
Supersedes the React dependabot patches
- `react`
- `react-dom`
- `@types/react`
- `@types/react-dom`

Left out Vite / tooling updates

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-13 18:46:56 +00:00
Alexanddependabot[bot] 21a11b06b9 Dependency updates roll-up (#765)
Supersedes the dependabot PRs after testing: 
- `actions/attest-build-provenance`
- `docker/build-push-action`
- `actions/checkout`
- `seleniumbase`
- `docker/metadata-action`
- `actions/setup-node`
- `python:3.14-slim`
- `@types/node`
- `postcss`
- `react-router-dom`

Left out for now: 
- `node:25-alpine`
- `vite@8`
- `@vitejs/plugin-react@6`

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-13 18:41:02 +00:00
Alex 0d856a3ef5 CodeQL fixes (#763)
- Block SSRF in image cover proxy (validate URL scheme and reject
private IPs)
- Sanitize settings tab name to prevent path traversal
2026-03-13 17:47:34 +00:00
Alex ebf4312174 Repo spring cleaning (#746)
- Add CI workflow (pytest + frontend typecheck/tests) on PRs
- Add CodeQL static analysis for Python and JS/TS
- Add Dependabot for pip, npm, Docker, and GitHub Actions
- Tighten workflow permissions
2026-03-13 17:09:49 +00:00
Alex 685c35d552 Hardcover list separation + Browser download fix (#745)
- Added full Hardcover reading status types into the list selector
- Split reading status entries from dedicated lists
- Added option to disable the automatic removal of books when downloaded
from a Hardcover list
- Fixed browser download not firing when the completed state was
triggered in specific cases
2026-03-13 14:33:10 +00:00
Alex 3d72f9e258 Various requested small features (#741)
- Added torrent removal option
- Pass Prowlarr seedtimes to download clients (excluding rTorrent)
- Split default release source option by content type
- Split download to browser option by content type
- Add "hide links" option
2026-03-12 17:36:07 +00:00
Alex 7f79da11e6 Activity routes logging (#736)
- Added specific logging for activity routes errors
- Fixed scrollbars
2026-03-11 19:59:12 +00:00
Alex c59ea46540 Frontend update + Misc fixes (#735)
- Updated frontend CSS to Tailwind v4
- Reverted socket IO origin restriction
- Fixed search queries not persisting after auth redirect
- Move advanced search options to left UI selector
- Unlock IRC source to be used for audiobook content_type
- Tweaked security settings env var syncing to be prioritised
- Fix AA "all languages" query generation
- Added language-free AA query as second fallback in case of no results
- Testing moving SeleniumBase scratch files to /tmp via symlink
- Added enhanced logging for activity dismissals and other events
- Removed iFrame restrictions
2026-03-11 18:16:34 +00:00
477 changed files with 52457 additions and 27938 deletions
+1
View File
@@ -1,6 +1,7 @@
.git
.github
.vscode
.local
.mypy_cache
README_images
.gitignore
+4 -4
View File
@@ -11,9 +11,9 @@ assignees: ''
## Steps To Reproduce
<!-- Steps to reproduce the behavior -->
1.
2.
3.
1.
2.
3.
## Expected Behavior
<!-- A clear and concise description of what you expected to happen -->
@@ -34,4 +34,4 @@ When running in debug mode, a DEBUG button will appear in the interface. Please
<!-- If applicable, please provide your full docker-compose (redacted from any secrets) -->
## Additional Context
<!-- Add any other context about the problem here -->
<!-- Add any other context about the problem here -->
+49
View File
@@ -0,0 +1,49 @@
version: 2
updates:
# Python dependencies
- package-ecosystem: "uv"
directory: "/"
schedule:
interval: "weekly"
open-pull-requests-limit: 10
groups:
python-deps:
patterns: ["*"]
update-types: ["minor", "patch"]
# Frontend npm dependencies
- package-ecosystem: "npm"
directory: "/src/frontend"
schedule:
interval: "weekly"
open-pull-requests-limit: 10
groups:
npm-deps:
patterns: ["*"]
update-types: ["minor", "patch"]
# Dockerfile base images
- package-ecosystem: "docker"
directory: "/"
schedule:
interval: "weekly"
open-pull-requests-limit: 5
groups:
docker-images:
patterns: ["*"]
ignore:
# Node.js: block major-version bumps so dependabot never proposes
# moving from one LTS line to a non-LTS "Current" release (e.g. 24 -> 25).
# Node LTS is even-numbered only; major bumps should be deliberate.
- dependency-name: "node"
update-types: ["version-update:semver-major"]
# GitHub Actions
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
open-pull-requests-limit: 5
groups:
gh-actions:
patterns: ["*"]
@@ -1,16 +1,54 @@
name: Create and publish Docker images
on:
push:
branches:
- 'main'
tags:
- 'v*'
schedule:
# Nightly at 03:17 UTC — only builds if there are new commits on main
# since the last successful run (see check-changes job).
- cron: '17 3 * * *'
workflow_dispatch:
permissions: read-all
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository_owner }}/shelfmark
jobs:
check-changes:
runs-on: ubuntu-latest
outputs:
should_build: ${{ steps.check.outputs.should_build }}
steps:
- name: Check for new commits since last successful build
id: check
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
EVENT_NAME: ${{ github.event_name }}
CURRENT_SHA: ${{ github.sha }}
REPO: ${{ github.repository }}
run: |
# Always build on tag pushes and manual dispatch.
if [[ "$EVENT_NAME" != "schedule" ]]; then
echo "Event is $EVENT_NAME — building unconditionally."
echo "should_build=true" >> "$GITHUB_OUTPUT"
exit 0
fi
# Scheduled run: only build if HEAD differs from the last successful build on main.
LAST_SHA=$(gh api "/repos/${REPO}/actions/workflows/build-and-publish-docker-image.yml/runs?branch=main&status=success&per_page=1" --jq '.workflow_runs[0].head_sha' 2>/dev/null || true)
echo "Last successful build SHA: ${LAST_SHA:-<none>}"
echo "Current HEAD SHA: ${CURRENT_SHA}"
if [[ -z "$LAST_SHA" || "$LAST_SHA" != "$CURRENT_SHA" ]]; then
echo "New commits detected — building."
echo "should_build=true" >> "$GITHUB_OUTPUT"
else
echo "No new commits since last successful build — skipping."
echo "should_build=false" >> "$GITHUB_OUTPUT"
fi
build-and-push-images:
needs: check-changes
if: needs.check-changes.outputs.should_build == 'true'
runs-on: ubuntu-latest
permissions:
contents: read
@@ -27,20 +65,20 @@ jobs:
- name: Get current date
id: date
run: echo "date=$(date +'%Y-%m-%d')" >> $GITHUB_OUTPUT
- name: Checkout repository
uses: actions/checkout@v4
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Log in to the Container registry
uses: docker/login-action@v3
uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract metadata for ${{ matrix.target }} image
id: meta
uses: docker/metadata-action@v5
uses: docker/metadata-action@030e881283bb7a6894de51c315a6bfe6a94e05cf # v6.0.0
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}${{ matrix.image_name_suffix }}
tags: |
@@ -50,13 +88,13 @@ jobs:
type=semver,pattern={{major}}.{{minor}}
type=sha
type=ref,event=tag
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
- name: Build and push ${{ matrix.target }} Docker image
id: push
uses: docker/build-push-action@v5
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0
with:
platforms: linux/amd64,linux/arm64
context: .
@@ -67,10 +105,10 @@ jobs:
RELEASE_VERSION=${{ github.ref_name }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
- name: Generate artifact attestation for ${{ matrix.target }} image
if: github.event_name != 'pull_request'
uses: actions/attest-build-provenance@v2
uses: actions/attest-build-provenance@a2bbfa25375fe432b6a289bc6b6cd05ecd0c4c32 # v4.1.0
with:
subject-name: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}${{ matrix.image_name_suffix }}
subject-digest: ${{ steps.push.outputs.digest }}
@@ -89,14 +127,14 @@ jobs:
LEGACY_NAME: calibre-web-automated-book-downloader
steps:
- name: Log in to registry
uses: docker/login-action@v3
uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
- name: Create legacy aliases
run: |
+158
View File
@@ -0,0 +1,158 @@
name: CI
on:
pull_request:
branches: [main]
permissions:
contents: read
jobs:
python-quality:
name: Python Quality
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Install uv and Python
uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
with:
version: "0.11.3"
python-version: "3.14"
enable-cache: true
- name: Sync dependencies
run: make install-python-dev
- name: Lint
run: make python-lint
- name: Check formatting
run: make python-format
- name: Check dead code
run: make python-dead-code
python-typechecks:
name: Python Typechecks
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Install uv and Python
uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
with:
version: "0.11.3"
python-version: "3.14"
enable-cache: true
- name: Sync dependencies
run: make install-python-dev
- name: Typecheck
run: make python-typecheck
python-tests:
name: Python Tests
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Install uv and Python
uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
with:
version: "0.11.3"
python-version: "3.14"
enable-cache: true
- name: Sync dependencies
run: make install-python-dev
- name: Run tests
run: make python-test
docker-build-check:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
- name: Build shelfmark-lite image
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0
with:
context: .
target: shelfmark-lite
platforms: linux/amd64
push: false
build-args: |
BUILD_VERSION=pr-${{ github.sha }}
RELEASE_VERSION=pr-${{ github.event.pull_request.number }}
frontend-quality:
name: Frontend Quality
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Set up Node
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: 24
cache: "npm"
cache-dependency-path: src/frontend/package-lock.json
- name: Install dependencies
run: make install-ci
- name: Lint
run: make frontend-lint
- name: Check formatting
run: make frontend-format
frontend-typechecks:
name: Frontend Typechecks
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Set up Node
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: 24
cache: "npm"
cache-dependency-path: src/frontend/package-lock.json
- name: Install dependencies
run: make install-ci
- name: Typecheck
run: make frontend-typecheck
frontend-tests:
name: Frontend Tests
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Set up Node
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: 24
cache: "npm"
cache-dependency-path: src/frontend/package-lock.json
- name: Install dependencies
run: make install-ci
- name: Unit tests
run: make frontend-test
+38
View File
@@ -0,0 +1,38 @@
name: CodeQL
on:
push:
branches: [main]
pull_request:
branches: [main]
schedule:
- cron: "0 6 * * 1" # Weekly on Monday at 6am UTC
permissions:
contents: read
jobs:
analyze:
runs-on: ubuntu-latest
permissions:
security-events: write
strategy:
fail-fast: false
matrix:
language: [python, javascript-typescript]
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Initialize CodeQL
uses: github/codeql-action/init@95e58e9a2cdfd71adc6e0353d5c52f41a045d225 # v3
with:
languages: ${{ matrix.language }}
- name: Autobuild
uses: github/codeql-action/autobuild@95e58e9a2cdfd71adc6e0353d5c52f41a045d225 # v3
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@95e58e9a2cdfd71adc6e0353d5c52f41a045d225 # v3
with:
category: "/language:${{ matrix.language }}"
+2
View File
@@ -74,6 +74,7 @@ pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
src/frontend/coverage/
htmlcov/
.tox/
.nox/
@@ -231,6 +232,7 @@ pyrightconfig.json
*.local.*
AGENTS.md
.claude/
.nvmrc
.playwright-mcp/
frontend-dist/
node_modules/
+23
View File
@@ -0,0 +1,23 @@
repos:
- repo: builtin
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-added-large-files
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.15.10
hooks:
- id: ruff-check
args: [--fix]
- id: ruff-format
- repo: local
hooks:
- id: oxfmt
name: oxfmt
entry: npx --prefix src/frontend oxfmt --config src/frontend/.oxfmtrc.json
language: system
types_or: [javascript, jsx, ts, tsx, css, json]
files: ^src/frontend/
exclude: package-lock\.json
+1 -1
View File
@@ -61,4 +61,4 @@
]
}
]
}
}
+39 -19
View File
@@ -4,7 +4,7 @@ ARG BUILDPLATFORM
ARG BUILDARCH
# Frontend build stage.
FROM --platform=$BUILDPLATFORM node:20-alpine AS frontend-builder
FROM --platform=$BUILDPLATFORM node:24-alpine AS frontend-builder
# Helpful debug output to see what platforms BuildKit thinks it's using
RUN echo "BUILDPLATFORM=$BUILDPLATFORM BUILDARCH=$BUILDARCH TARGETPLATFORM=$TARGETPLATFORM TARGETARCH=$TARGETARCH"
@@ -25,7 +25,9 @@ COPY src/frontend/ ./
RUN npm run build
# Use python-slim as the base image
FROM python:3.10-slim AS base
FROM python:3.14-slim AS base
COPY --from=ghcr.io/astral-sh/uv:0.11.3 /uv /uvx /bin/
# Add build argument for version
ARG BUILD_VERSION
@@ -39,13 +41,12 @@ SHELL ["/bin/bash", "-o", "pipefail", "-c"]
# Consistent environment variables grouped together
ENV DEBIAN_FRONTEND=noninteractive \
DOCKERMODE=true \
UV_LINK_MODE=copy \
PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
PYTHONIOENCODING=UTF-8 \
PIP_NO_CACHE_DIR=1 \
PIP_DISABLE_PIP_VERSION_CHECK=1 \
PIP_DEFAULT_TIMEOUT=100 \
NAME=Shelfmark \
PATH=/app/.venv/bin:$PATH \
PYTHONPATH=/app \
# PUID/PGID will be handled by entrypoint script, but TZ/Locale are still needed
LANG=en_US.UTF-8 \
@@ -56,7 +57,6 @@ ENV DEBIAN_FRONTEND=noninteractive \
ENV FLASK_PORT=8084
# Configure locale, timezone, and perform initial cleanup in a single layer
# User/group creation is removed
RUN apt-get update && \
apt-get install -y --no-install-recommends \
# For locale
@@ -88,15 +88,20 @@ RUN apt-get update && \
echo "LC_ALL=en_US.UTF-8" >> /etc/environment && \
echo "LANG=en_US.UTF-8" > /etc/locale.conf
# Create a fixed runtime user/group so hardened Docker/Kubernetes deployments
# can start the container directly as a non-root user with a passwd entry.
RUN groupadd -g 1000 shelfmark && \
useradd -u 1000 -g shelfmark -d /home/shelfmark -s /usr/sbin/nologin shelfmark && \
mkdir -p /home/shelfmark && \
chown 1000:1000 /home/shelfmark
# Set working directory
WORKDIR /app
# Install Python dependencies using pip
# Copying requirements files separately leverages build cache
# Cache mount persists pip cache between builds for faster installs
COPY requirements-base.txt requirements-shelfmark.txt ./
RUN --mount=type=cache,target=/root/.cache/pip \
pip install -r requirements-base.txt
# Install core Python dependencies first for better layer caching
COPY pyproject.toml uv.lock ./
RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --locked --no-default-groups
# Copy application code *after* dependencies are installed
COPY . .
@@ -104,10 +109,19 @@ COPY . .
# Copy built frontend from frontend-builder stage
COPY --from=frontend-builder /frontend/dist /app/frontend-dist
# Final setup: permissions and directories in one layer
# Only creating directories and setting executable bits.
# Ownership will be handled by the entrypoint script.
RUN mkdir -p /var/log/shelfmark /books && \
# Final setup: create image-owned runtime paths for the fixed non-root user.
# Root/PUID mode still re-homes ownership at startup when needed.
RUN mkdir -p \
/config \
/books \
/var/log/shelfmark \
/tmp/shelfmark/seleniumbase/downloaded_files \
/tmp/shelfmark/seleniumbase/archived_files && \
rm -rf /app/downloaded_files /app/archived_files && \
ln -s /tmp/shelfmark/seleniumbase/downloaded_files /app/downloaded_files && \
ln -s /tmp/shelfmark/seleniumbase/archived_files /app/archived_files && \
chown -R 1000:1000 /config /books /home/shelfmark /tmp/shelfmark /var/log/shelfmark && \
chmod -R a+rX /app && \
chmod +x /app/entrypoint.sh /app/tor.sh /app/genDebug.sh
# Expose the application port
@@ -146,9 +160,15 @@ RUN apt-get update && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
# Install additional dependencies (requirements file already copied in base stage)
RUN --mount=type=cache,target=/root/.cache/pip \
pip install -r requirements-shelfmark.txt
# Install the browser automation stack used by the full image
RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --locked --no-default-groups --extra browser
# Keep SeleniumBase's bundled driver cache writable for the fixed non-root user.
RUN SELENIUMBASE_DRIVERS_DIR=$(/app/.venv/bin/python -c "import pathlib, seleniumbase; print(pathlib.Path(seleniumbase.__file__).resolve().parent / 'drivers')") && \
chown -R 1000:1000 "${SELENIUMBASE_DRIVERS_DIR}" && \
chmod -R u+rwX,go+rX "${SELENIUMBASE_DRIVERS_DIR}" && \
if [ -f "${SELENIUMBASE_DRIVERS_DIR}/uc_driver" ]; then chmod +x "${SELENIUMBASE_DRIVERS_DIR}/uc_driver"; fi
# Grant read/execute permissions to others
RUN chmod -R o+rx /usr/bin/chromium
+102 -3
View File
@@ -1,4 +1,4 @@
.PHONY: help install dev build preview typecheck frontend-test clean up down docker-build refresh restart
.PHONY: help install install-ci install-python-dev dev build preview frontend-typecheck frontend-lint frontend-format frontend-format-fix frontend-checks frontend-test clean up down docker-build refresh restart build-serve python-lint python-lint-fix python-format python-format-fix python-typecheck python-dead-code python-checks python-test python-test-cov checks fix
# Frontend directory
FRONTEND_DIR := src/frontend
@@ -10,13 +10,34 @@ COMPOSE_FILE := docker-compose.dev.yml
help:
@echo "Available targets:"
@echo ""
@echo "Quality:"
@echo " checks - Run ALL static analysis checks (frontend + Python)"
@echo " fix - Auto-fix lint + format issues (frontend + Python)"
@echo ""
@echo "Frontend:"
@echo " install - Install frontend dependencies"
@echo " dev - Start development server"
@echo " build - Build frontend for production"
@echo " build-serve - Build and serve via Flask (test prod build without Docker)"
@echo " preview - Preview production build"
@echo " typecheck - Run TypeScript type checking"
@echo " frontend-typecheck - Run TypeScript type checking"
@echo " frontend-lint - Run Oxlint against frontend code"
@echo " frontend-format - Check frontend formatting with Oxfmt"
@echo " frontend-format-fix - Format frontend code with Oxfmt"
@echo " frontend-checks - Run all frontend static analysis checks"
@echo " frontend-test - Run frontend unit tests"
@echo ""
@echo "Python:"
@echo " install-python-dev - Sync Python runtime + dev tooling with uv"
@echo " python-lint - Run Ruff against Python code (backend + tests)"
@echo " python-lint-fix - Run Ruff with safe auto-fixes"
@echo " python-format - Check Python formatting with Ruff"
@echo " python-format-fix - Format Python code with Ruff"
@echo " python-typecheck - Run BasedPyright against backend + tests"
@echo " python-dead-code - Run Vulture against backend code"
@echo " python-checks - Run all Python static analysis checks"
@echo " python-test - Run unit tests"
@echo " python-test-cov - Run unit tests with coverage report"
@echo " clean - Remove node_modules and build artifacts"
@echo ""
@echo "Backend (Docker):"
@@ -31,6 +52,17 @@ install:
@echo "Installing frontend dependencies..."
cd $(FRONTEND_DIR) && npm install
install-ci:
@echo "Installing frontend dependencies (CI, lockfile-strict)..."
cd $(FRONTEND_DIR) && npm ci
# Install Python development dependencies
install-python-dev:
@echo "Syncing Python runtime and dev tooling with uv..."
uv sync --locked --extra browser
@echo "Installing prek git hooks..."
uv run prek install
# Start development server
dev:
@echo "Starting development server..."
@@ -41,21 +73,88 @@ build:
@echo "Building frontend for production..."
cd $(FRONTEND_DIR) && npm run build
# Build frontend and sync to frontend-dist for the running container to serve
build-serve: build
@echo "Syncing build to frontend-dist..."
@mkdir -p frontend-dist
rsync -a --delete $(FRONTEND_DIR)/dist/ frontend-dist/
@echo "Done. Hit the Flask backend (port 8084) to test the production build."
# Preview production build
preview:
@echo "Previewing production build..."
cd $(FRONTEND_DIR) && npm run preview
# Type checking
typecheck:
frontend-typecheck:
@echo "Running TypeScript type checking..."
cd $(FRONTEND_DIR) && npm run typecheck
# Python linting (backend + tests)
python-lint:
@echo "Running Ruff..."
uv run ruff check shelfmark tests
python-lint-fix:
@echo "Running Ruff with safe auto-fixes..."
uv run ruff check shelfmark tests --fix
python-format:
@echo "Checking Python formatting with Ruff..."
uv run ruff format --check shelfmark tests
python-format-fix:
@echo "Formatting Python code with Ruff..."
uv run ruff format shelfmark tests
python-typecheck:
@echo "Running BasedPyright..."
uv run basedpyright
@echo "Running BasedPyright against tests..."
uv run basedpyright tests --skipunannotated
python-dead-code:
@echo "Running Vulture..."
uv run vulture shelfmark
python-checks: python-lint python-format python-typecheck python-dead-code
python-test:
@echo "Running tests..."
uv run pytest tests/ -x --tb=short -m "not integration and not e2e"
python-test-cov:
@echo "Running tests with coverage..."
uv run pytest tests/ -x --tb=short -m "not integration and not e2e" --cov --cov-report=term-missing
# Frontend linting
frontend-lint:
@echo "Running Oxlint..."
cd $(FRONTEND_DIR) && npm run lint
# Frontend formatting
frontend-format:
@echo "Checking frontend formatting with Oxfmt..."
cd $(FRONTEND_DIR) && npm run format:check
frontend-format-fix:
@echo "Formatting frontend code with Oxfmt..."
cd $(FRONTEND_DIR) && npm run format
# All frontend static analysis
frontend-checks: frontend-lint frontend-format frontend-typecheck
# Run frontend unit tests
frontend-test:
@echo "Running frontend unit tests..."
cd $(FRONTEND_DIR) && npm run test:unit
# All static analysis checks (frontend + Python)
checks: frontend-checks python-checks
# Auto-fix lint + format issues (frontend + Python)
fix: python-lint-fix python-format-fix frontend-format-fix
# Clean build artifacts and dependencies
clean:
@echo "Cleaning build artifacts and dependencies..."
+1 -1
View File
@@ -1,4 +1,4 @@
# Routes all traffic through Tor - requires NET_ADMIN capability
# Routes all traffic through Tor - requires root startup
services:
shelfmark-tor:
image: ghcr.io/calibrain/shelfmark:latest
+1 -1
View File
@@ -69,4 +69,4 @@
{ "language": "Uyghur", "code": "ug" },
{ "language": "Armenian", "code": "hy" },
{ "language": "Shan", "code": "shn" }
]
]
-39
View File
@@ -1,39 +0,0 @@
# Bypass testing - switch between dev build and v1.0.1
# Usage:
# Test dev build: docker compose -f docker-compose.bypass-test.yml up shelfmark-dev
# Test v1.0.1: docker compose -f docker-compose.bypass-test.yml up shelfmark-stable
# Pull latest dev: docker compose -f docker-compose.bypass-test.yml build shelfmark-dev
# Pull v1.0.1: docker compose -f docker-compose.bypass-test.yml pull shelfmark-stable
services:
# Dev image from registry
shelfmark-dev:
image: ghcr.io/calibrain/shelfmark:dev
container_name: shelfmark-bypass-dev
environment:
PUID: 1000
PGID: 1000
DEBUG: true
ports:
- 8084:8084
volumes:
- ./.local/bypass-test/config-dev:/config
- ./.local/bypass-test/books:/books
- ./.local/bypass-test/log-dev:/var/log/shelfmark
- ./.local/bypass-test/tmp:/tmp/shelfmark
# Stable v1.0.1 for comparison
shelfmark-stable:
image: ghcr.io/calibrain/shelfmark:1.0.1
container_name: shelfmark-bypass-stable
environment:
PUID: 1000
PGID: 1000
DEBUG: true
ports:
- 8085:8084
volumes:
- ./.local/bypass-test/config-stable:/config
- ./.local/bypass-test/books:/books
- ./.local/bypass-test/log-stable:/var/log/shelfmark
- ./.local/bypass-test/tmp:/tmp/shelfmark
+1
View File
@@ -20,5 +20,6 @@ services:
- ./.local/log:/var/log/shelfmark
- ./.local/tmp:/tmp/shelfmark
- ./shelfmark:/app/shelfmark:ro
- ./frontend-dist:/app/frontend-dist:ro
# Required for torrent / usenet - path must match your download client's volume exactly
# - /path/to/downloads:/path/to/downloads
+19
View File
@@ -13,6 +13,7 @@
# - Transmission: http://localhost:9091 (admin / admin)
# - Deluge: http://localhost:8112 (password: deluge)
# - NZBGet: http://localhost:6789 (nzbget / tegbzn6789)
# - NZBHydra: http://localhost:5076 (no auth by default)
# - SABnzbd: http://localhost:8085 (complete setup wizard for API key)
# - rTorrent: http://localhost:8000 (admin / admin - if auth enabled)
#
@@ -35,6 +36,7 @@ services:
# - Transmission: http://transmission:9091
# - Deluge Web UI: http://deluge:8112
# - NZBGet: http://nzbget:6789
# - NZBHydra: http://nzbhydra:5076
# - SABnzbd: http://sabnzbd:8080
# - rTorrent: http://rtorrent:80 (XMLRPC via HTTP) or rtorrent (port 5000 for SCGI)
ports:
@@ -54,11 +56,13 @@ services:
# Mount tests for running pytest in container
- ./tests:/app/tests:ro
- ./pyproject.toml:/app/pyproject.toml:ro
- ./uv.lock:/app/uv.lock:ro
# Mount client configs for integration tests to read credentials
- ./.local/test-clients/qbittorrent/config:/qbittorrent-config:ro
- ./.local/test-clients/sabnzbd/config:/sabnzbd-config:ro
depends_on:
- nzbget
- nzbhydra
- sabnzbd
- qbittorrent
- transmission
@@ -179,3 +183,18 @@ services:
- "50000:50000" # Incoming connections
- "6881:6881/udp"
restart: unless-stopped
nzbhydra:
image: lscr.io/linuxserver/nzbhydra2:latest
container_name: nzbhydra
environment:
- PUID=1000
- PGID=1000
- TZ=Europe/London
volumes:
- ./.local/test-clients/nzbhydra/config:/config
- ./.local/test-clients/downloads:/downloads
ports:
- 5076:5076
restart: unless-stopped
+9 -3
View File
@@ -18,9 +18,9 @@ Prowlarr -> Download client saves to <client path>
Key point: For torrent and usenet downloads, Shelfmark must see the same file path that your download client reports. The container path must match in both containers.
## Direct Download Setup
## Direct Download Volume Setup
Direct downloads do not use an external download client. A simple two-folder setup is enough.
If you plan to use Direct Download, it does not use an external download client. A simple two-folder setup is enough.
Required volumes:
@@ -43,7 +43,12 @@ services:
Notes:
- Point `/books` to your library ingest folder (Calibre-Web, Booklore, Audiobookshelf, etc) for automatic import.
- If you set Books Output Mode to Booklore (API), books are uploaded via API instead of written to `/books`. Audiobooks still use a destination folder.
- Ensure `PUID`/`PGID` (or legacy `UID`/`GID`) match the owner of the host directories to avoid permission errors.
- Ensure `PUID`/`PGID` (or legacy `UID`/`GID`) match the owner of the host directories.
- For non-root mode, start the container as `1000:1000`.
- On Kubernetes, set `runAsUser: 1000`, `runAsGroup: 1000`, and `runAsNonRoot: true` together.
- `PUID`/`PGID` keep the default root startup flow.
- In non-root mode, mounted paths must already be writable by `1000:1000`.
- `USING_TOR=true` requires root startup.
## Torrent / Usenet Setup
@@ -113,6 +118,7 @@ Configure templates in Settings -> Downloads. Template syntax details are docume
- "Download failed - file not found": Path mismatch between Shelfmark and the download client. Ensure container paths match or use Remote Path Mappings.
- "Permission denied": `PUID`/`PGID` do not match the host directories. Ensure Shelfmark can read the client path and write to the destination.
- "Permission denied" in non-root Docker/Kubernetes mode: ensure the mounted path is writable by UID/GID `1000:1000`, or switch back to root startup with `PUID`/`PGID`.
- "Hardlinks not working" or "Files being copied instead": Source and destination are on different filesystems. Move the destination or accept copy fallback.
- "Downloads work but library does not see them": Destination does not point to the library ingest folder. Check Settings -> Downloads -> Destination.
- CIFS/SMB shares: Use the `nobrl` mount option to avoid database lock errors. Example: `//server/share /mnt/share cifs nobrl,... 0 0`
+17 -6
View File
@@ -121,15 +121,26 @@ Example payload shape:
}
```
Example (bash + jq) (JSON payload must be enabled):
Example (bash + python3) (JSON payload must be enabled):
```bash
payload="$(cat)"
mode="$(echo "$payload" | jq -r '.output.mode')"
title="$(echo "$payload" | jq -r '.task.title')"
final_paths="$(echo "$payload" | jq -r '.paths.final_paths[]')"
echo "mode=$mode title=$title" >&2
echo "$final_paths" >&2
target="$1"
PAYLOAD="$payload" TARGET="$target" python3 - <<'PY'
import json
import os
import sys
payload = json.loads(os.environ["PAYLOAD"])
print(f"target={os.environ['TARGET']}", file=sys.stderr)
print(
f"mode={payload['output']['mode']} title={payload['task']['title']}",
file=sys.stderr,
)
for path in payload["paths"]["final_paths"]:
print(path, file=sys.stderr)
PY
```
Example (Python) (works whether JSON payload is enabled or not):
+207 -104
View File
@@ -14,6 +14,7 @@ This document lists all configuration options that can be set via environment va
- [Network](#network)
- [Advanced](#advanced)
- [Prowlarr](#prowlarr)
- [Newznab](#newznab)
- [AudiobookBay](#audiobookbay)
- [IRC](#irc)
- [Download Clients](#download-clients)
@@ -124,7 +125,8 @@ Show the onboarding wizard on first run. Set to false to skip (useful for epheme
| Variable | Description | Type | Default |
|----------|-------------|------|---------|
| `CALIBRE_WEB_URL` | Adds a navigation button to your book library (Calibre-Web Automated, Booklore, etc). | string | _none_ |
| `SEARCH_PAGE_TITLE` | Title shown above the main search box on the homepage. | string | `Shelfmark` |
| `CALIBRE_WEB_URL` | Adds a navigation button to your book library (Calibre-Web Automated, Grimmory, etc). | string | _none_ |
| `AUDIOBOOK_LIBRARY_URL` | Adds a separate navigation button for your audiobook library (Audiobookshelf, Plex, etc). When both URLs are set, icons are shown instead of text. | string | _none_ |
| `SUPPORTED_FORMATS` | Book formats to include in search results. ZIP/RAR archives are extracted automatically and book files are used if found. | string (comma-separated) | `epub,mobi,azw3,fb2,djvu,cbz,cbr` |
| `SUPPORTED_AUDIOBOOK_FORMATS` | Audiobook formats to include in search results. ZIP/RAR archives are extracted automatically and audiobook files are used if found. | string (comma-separated) | `m4b,mp3` |
@@ -133,11 +135,20 @@ Show the onboarding wizard on first run. Set to false to skip (useful for epheme
<details>
<summary>Detailed descriptions</summary>
#### `SEARCH_PAGE_TITLE`
**Search Page Title**
Title shown above the main search box on the homepage.
- **Type:** string
- **Default:** `Shelfmark`
#### `CALIBRE_WEB_URL`
**Library URL**
Adds a navigation button to your book library (Calibre-Web Automated, Booklore, etc).
Adds a navigation button to your book library (Calibre-Web Automated, Grimmory, etc).
- **Type:** string
- **Default:** _none_
@@ -184,11 +195,15 @@ Default language filter for searches.
| Variable | Description | Type | Default |
|----------|-------------|------|---------|
| `SEARCH_MODE` | How you want to search for and download books. | string (choice) | `direct` |
| `SEARCH_MODE` | How you want to search for and download books. | string (choice) | `universal` |
| `AA_DEFAULT_SORT` | Default sort order for search results. | string (choice) | `relevance` |
| `SHOW_RELEASE_SOURCE_LINKS` | Show clickable release-source links in release and details modals. Metadata provider links stay enabled. | boolean | `true` |
| `SHOW_COMBINED_SELECTOR` | Show the option to search for and download both a book and audiobook together. | boolean | `true` |
| `METADATA_PROVIDER` | Choose which metadata provider to use for book searches. | string (choice) | `openlibrary` |
| `METADATA_PROVIDER_AUDIOBOOK` | Metadata provider for audiobook searches. Uses the book provider if not set. | string (choice) | _empty string_ |
| `DEFAULT_RELEASE_SOURCE` | The release source tab to open by default in the release modal. | string (choice) | `direct_download` |
| `METADATA_PROVIDER_COMBINED` | Metadata provider for combined mode searches. Uses the book provider if not set. | string (choice) | _empty string_ |
| `DEFAULT_RELEASE_SOURCE` | The release source tab to open by default in the release modal for books. Leave unset to use the first available source. | string (choice) | _empty string_ |
| `DEFAULT_RELEASE_SOURCE_AUDIOBOOK` | The release source tab to open by default in the release modal for audiobooks. Uses the book release source if not set. | string (choice) | _empty string_ |
<details>
<summary>Detailed descriptions</summary>
@@ -200,7 +215,7 @@ Default language filter for searches.
How you want to search for and download books.
- **Type:** string (choice)
- **Default:** `direct`
- **Default:** `universal`
- **Options:** `direct` (Direct), `universal` (Universal)
#### `AA_DEFAULT_SORT`
@@ -213,6 +228,24 @@ Default sort order for search results.
- **Default:** `relevance`
- **Options:** `relevance` (Most relevant), `newest` (Newest (publication year)), `oldest` (Oldest (publication year)), `largest` (Largest (filesize)), `smallest` (Smallest (filesize)), `newest_added` (Newest (open sourced)), `oldest_added` (Oldest (open sourced))
#### `SHOW_RELEASE_SOURCE_LINKS`
**Show Release Source Links**
Show clickable release-source links in release and details modals. Metadata provider links stay enabled.
- **Type:** boolean
- **Default:** `true`
#### `SHOW_COMBINED_SELECTOR`
**Show Combined Download Selector**
Show the option to search for and download both a book and audiobook together.
- **Type:** boolean
- **Default:** `true`
#### `METADATA_PROVIDER`
**Book Metadata Provider**
@@ -233,15 +266,35 @@ Metadata provider for audiobook searches. Uses the book provider if not set.
- **Default:** _empty string_
- **Options:** `""` (Use book provider), `""` (No providers enabled)
#### `DEFAULT_RELEASE_SOURCE`
#### `METADATA_PROVIDER_COMBINED`
**Default Release Source**
**Combined Mode Metadata Provider**
The release source tab to open by default in the release modal.
Metadata provider for combined mode searches. Uses the book provider if not set.
- **Type:** string (choice)
- **Default:** `direct_download`
- **Options:** `direct_download` (Direct Download), `prowlarr` (Prowlarr), `audiobookbay` (AudiobookBay)
- **Default:** _empty string_
- **Options:** `""` (Use book provider), `""` (No providers enabled)
#### `DEFAULT_RELEASE_SOURCE`
**Default Book Release Source**
The release source tab to open by default in the release modal for books. Leave unset to use the first available source.
- **Type:** string (choice)
- **Default:** _empty string_
- **Options:** `""` (Use first available source)
#### `DEFAULT_RELEASE_SOURCE_AUDIOBOOK`
**Default Audiobook Release Source**
The release source tab to open by default in the release modal for audiobooks. Uses the book release source if not set.
- **Type:** string (choice)
- **Default:** _empty string_
- **Options:** `""` (Use book release source)
</details>
@@ -252,15 +305,15 @@ The release source tab to open by default in the release modal.
| `BOOKS_OUTPUT_MODE` | Choose where completed book files are sent. | string (choice) | `folder` |
| `INGEST_DIR` | Directory where downloaded files are saved. Use {User} for per-user folders (e.g. /books/{User}). | string | `/books` |
| `FILE_ORGANIZATION` | Choose how downloaded book files are named and organized. | string (choice) | `rename` |
| `TEMPLATE_RENAME` | Variables: {Author}, {Title}, {Year}, {User}, {OriginalName} (source filename without extension). Universal adds: {Series}, {SeriesPosition}, {Subtitle}. Use arbitrary prefix/suffix: {Vol. SeriesPosition - } outputs 'Vol. 2 - ' when set, nothing when empty. Rename templates are filename-only (no '/' or '\'); use Organize for folders. Applies to single-file downloads. | string | `{Author} - {Title} ({Year})` |
| `TEMPLATE_ORGANIZE` | Use / to create folders. Variables: {Author}, {Title}, {Year}, {User}, {OriginalName} (source filename without extension). Universal adds: {Series}, {SeriesPosition}, {Subtitle}. Use arbitrary prefix/suffix: {Vol. SeriesPosition - } outputs 'Vol. 2 - ' when set, nothing when empty. | string | `{Author}/{Title} ({Year})` |
| `TEMPLATE_RENAME` | Variables: {Author}, {Title}, {Year}, {User}, {OriginalName} (source filename without extension). Universal adds: {Series}, {SeriesPosition}, {Subtitle}, {PrimaryTitle}. Use arbitrary prefix/suffix: {Vol. SeriesPosition - } outputs 'Vol. 2 - ' when set, nothing when empty. Rename templates are filename-only (no '/' or '\'); use Organize for folders. Applies to single-file downloads. | string | `{Author} - {Title} ({Year})` |
| `TEMPLATE_ORGANIZE` | Use / to create folders. Variables: {Author}, {Title}, {Year}, {User}, {OriginalName} (source filename without extension). Universal adds: {Series}, {SeriesPosition}, {Subtitle}, {PrimaryTitle}. Use arbitrary prefix/suffix: {Vol. SeriesPosition - } outputs 'Vol. 2 - ' when set, nothing when empty. | string | `{Author}/{Title} ({Year})` |
| `HARDLINK_TORRENTS` | Create hardlinks instead of copying. Preserves seeding but archives won't be extracted. Don't use if destination is a library ingest folder. | boolean | `false` |
| `BOOKLORE_HOST` | Base URL of your Booklore instance | string | _none_ |
| `BOOKLORE_USERNAME` | Booklore account username | string | _none_ |
| `BOOKLORE_PASSWORD` | Booklore account password | string (secret) | _none_ |
| `BOOKLORE_HOST` | Base URL of your Grimmory instance | string | _none_ |
| `BOOKLORE_USERNAME` | Grimmory account username | string | _none_ |
| `BOOKLORE_PASSWORD` | Grimmory account password | string (secret) | _none_ |
| `BOOKLORE_DESTINATION` | Choose whether uploads go directly to a specific library path or to Bookdrop for review. | string (choice) | `library` |
| `BOOKLORE_LIBRARY_ID` | Booklore library to upload into. | string (choice) | _none_ |
| `BOOKLORE_PATH_ID` | Booklore library path for uploads. | string (choice) | _none_ |
| `BOOKLORE_LIBRARY_ID` | Grimmory library to upload into. | string (choice) | _none_ |
| `BOOKLORE_PATH_ID` | Grimmory library path for uploads. | string (choice) | _none_ |
| `EMAIL_RECIPIENT` | Optional fallback email address when no per-user email recipient override is configured. | string | _none_ |
| `EMAIL_ATTACHMENT_SIZE_LIMIT_MB` | Maximum total attachment size per email. Email encoding adds overhead; keep this below your provider's limit. | number | `25` |
| `EMAIL_SMTP_HOST` | SMTP server hostname or IP (e.g., smtp.gmail.com). | string | _none_ |
@@ -269,16 +322,16 @@ The release source tab to open by default in the release modal.
| `EMAIL_SMTP_USERNAME` | SMTP username (leave empty for no authentication). | string | _none_ |
| `EMAIL_SMTP_PASSWORD` | SMTP password (required if Username is set). | string (secret) | _none_ |
| `EMAIL_FROM` | From address used for the email. You can include a display name (e.g., Shelfmark <mail@example.com>). Leave blank to default to the SMTP username (when it is an email address). | string | _none_ |
| `EMAIL_SUBJECT_TEMPLATE` | Email subject. Variables: {Author}, {Title}, {Year}, {Series}, {SeriesPosition}, {Subtitle}, {Format}. | string | `{Title}` |
| `EMAIL_SUBJECT_TEMPLATE` | Email subject. Variables: {Author}, {Title}, {PrimaryTitle}, {Year}, {Series}, {SeriesPosition}, {Subtitle}, {Format}. | string | `{Title}` |
| `EMAIL_SMTP_TIMEOUT_SECONDS` | How long to wait for SMTP operations before failing. | number | `60` |
| `EMAIL_ALLOW_UNVERIFIED_TLS` | Disable TLS certificate verification (not recommended). | boolean | `false` |
| `DESTINATION_AUDIOBOOK` | Directory where downloaded audiobook files are saved. Leave empty to use the Books destination. | string | _none_ |
| `FILE_ORGANIZATION_AUDIOBOOK` | Choose how downloaded audiobook files are named and organized. | string (choice) | `rename` |
| `TEMPLATE_AUDIOBOOK_RENAME` | Variables: {Author}, {Title}, {Year}, {User}, {OriginalName} (source filename without extension), {Series}, {SeriesPosition}, {Subtitle}, {PartNumber}. Use arbitrary prefix/suffix: {Vol. SeriesPosition - } outputs 'Vol. 2 - ' when set, nothing when empty. Rename templates are filename-only (no '/' or '\'); use Organize for folders. Applies to single-file downloads. | string | `{Author} - {Title}` |
| `TEMPLATE_AUDIOBOOK_ORGANIZE` | Use / to create folders. Variables: {Author}, {Title}, {Year}, {User}, {OriginalName} (source filename without extension), {Series}, {SeriesPosition}, {Subtitle}, {PartNumber}. Use arbitrary prefix/suffix: {Vol. SeriesPosition - } outputs 'Vol. 2 - ' when set, nothing when empty. | string | `{Author}/{Title}` |
| `TEMPLATE_AUDIOBOOK_RENAME` | Variables: {Author}, {Title}, {Year}, {User}, {OriginalName} (source filename without extension), {Series}, {SeriesPosition}, {Subtitle}, {PrimaryTitle}, {PartNumber}. Use arbitrary prefix/suffix: {Vol. SeriesPosition - } outputs 'Vol. 2 - ' when set, nothing when empty. Rename templates are filename-only (no '/' or '\'); use Organize for folders. Applies to single-file downloads. | string | `{Author} - {Title}` |
| `TEMPLATE_AUDIOBOOK_ORGANIZE` | Use / to create folders. Variables: {Author}, {Title}, {Year}, {User}, {OriginalName} (source filename without extension), {Series}, {SeriesPosition}, {Subtitle}, {PrimaryTitle}, {PartNumber}. Use arbitrary prefix/suffix: {Vol. SeriesPosition - } outputs 'Vol. 2 - ' when set, nothing when empty. | string | `{Author}/{Title}/{Title}` |
| `HARDLINK_TORRENTS_AUDIOBOOK` | Create hardlinks instead of copying. Preserves seeding but archives won't be extracted. Don't use if destination is a library ingest folder. | boolean | `true` |
| `AUTO_OPEN_DOWNLOADS_SIDEBAR` | Automatically open the downloads sidebar when a new download is queued. | boolean | `false` |
| `DOWNLOAD_TO_BROWSER` | Automatically download completed files to your browser. | boolean | `false` |
| `DOWNLOAD_TO_BROWSER_CONTENT_TYPES` | Automatically download completed files to your browser for the selected content types. | string (comma-separated) | _empty list_ |
| `MAX_CONCURRENT_DOWNLOADS` | Maximum number of simultaneous downloads. | number | `3` |
| `STATUS_TIMEOUT` | How long to keep completed/failed downloads in the queue display. | number | `3600` |
@@ -293,7 +346,7 @@ Choose where completed book files are sent.
- **Type:** string (choice)
- **Default:** `folder`
- **Options:** `folder` (Folder), `email` (Email (SMTP)), `booklore` (Booklore (API))
- **Options:** `folder` (Folder), `email` (Email (SMTP)), `booklore` (Grimmory (API))
#### `INGEST_DIR`
@@ -309,7 +362,7 @@ Directory where downloaded files are saved. Use {User} for per-user folders (e.g
**File Organization**
Choose how downloaded book files are named and organized.
Choose how downloaded book files are named and organized.
- **Type:** string (choice)
- **Default:** `rename`
@@ -319,7 +372,7 @@ Choose how downloaded book files are named and organized.
**Naming Template**
Variables: {Author}, {Title}, {Year}, {User}, {OriginalName} (source filename without extension). Universal adds: {Series}, {SeriesPosition}, {Subtitle}. Use arbitrary prefix/suffix: {Vol. SeriesPosition - } outputs 'Vol. 2 - ' when set, nothing when empty. Rename templates are filename-only (no '/' or '\'); use Organize for folders. Applies to single-file downloads.
Variables: {Author}, {Title}, {Year}, {User}, {OriginalName} (source filename without extension). Universal adds: {Series}, {SeriesPosition}, {Subtitle}, {PrimaryTitle}. Use arbitrary prefix/suffix: {Vol. SeriesPosition - } outputs 'Vol. 2 - ' when set, nothing when empty. Rename templates are filename-only (no '/' or '\'); use Organize for folders. Applies to single-file downloads.
- **Type:** string
- **Default:** `{Author} - {Title} ({Year})`
@@ -328,7 +381,7 @@ Variables: {Author}, {Title}, {Year}, {User}, {OriginalName} (source filename wi
**Path Template**
Use / to create folders. Variables: {Author}, {Title}, {Year}, {User}, {OriginalName} (source filename without extension). Universal adds: {Series}, {SeriesPosition}, {Subtitle}. Use arbitrary prefix/suffix: {Vol. SeriesPosition - } outputs 'Vol. 2 - ' when set, nothing when empty.
Use / to create folders. Variables: {Author}, {Title}, {Year}, {User}, {OriginalName} (source filename without extension). Universal adds: {Series}, {SeriesPosition}, {Subtitle}, {PrimaryTitle}. Use arbitrary prefix/suffix: {Vol. SeriesPosition - } outputs 'Vol. 2 - ' when set, nothing when empty.
- **Type:** string
- **Default:** `{Author}/{Title} ({Year})`
@@ -344,9 +397,9 @@ Create hardlinks instead of copying. Preserves seeding but archives won't be ext
#### `BOOKLORE_HOST`
**Booklore URL**
**Grimmory URL**
Base URL of your Booklore instance
Base URL of your Grimmory instance
- **Type:** string
- **Default:** _none_
@@ -356,7 +409,7 @@ Base URL of your Booklore instance
**Username**
Booklore account username
Grimmory account username
- **Type:** string
- **Default:** _none_
@@ -366,7 +419,7 @@ Booklore account username
**Password**
Booklore account password
Grimmory account password
- **Type:** string (secret)
- **Default:** _none_
@@ -386,7 +439,7 @@ Choose whether uploads go directly to a specific library path or to Bookdrop for
**Library**
Booklore library to upload into.
Grimmory library to upload into.
- **Type:** string (choice)
- **Default:** _none_
@@ -396,7 +449,7 @@ Booklore library to upload into.
**Path**
Booklore library path for uploads.
Grimmory library path for uploads.
- **Type:** string (choice)
- **Default:** _none_
@@ -482,7 +535,7 @@ From address used for the email. You can include a display name (e.g., Shelfmark
**Subject Template**
Email subject. Variables: {Author}, {Title}, {Year}, {Series}, {SeriesPosition}, {Subtitle}, {Format}.
Email subject. Variables: {Author}, {Title}, {PrimaryTitle}, {Year}, {Series}, {SeriesPosition}, {Subtitle}, {Format}.
- **Type:** string
- **Default:** `{Title}`
@@ -529,7 +582,7 @@ Choose how downloaded audiobook files are named and organized.
**Naming Template**
Variables: {Author}, {Title}, {Year}, {User}, {OriginalName} (source filename without extension), {Series}, {SeriesPosition}, {Subtitle}, {PartNumber}. Use arbitrary prefix/suffix: {Vol. SeriesPosition - } outputs 'Vol. 2 - ' when set, nothing when empty. Rename templates are filename-only (no '/' or '\'); use Organize for folders. Applies to single-file downloads.
Variables: {Author}, {Title}, {Year}, {User}, {OriginalName} (source filename without extension), {Series}, {SeriesPosition}, {Subtitle}, {PrimaryTitle}, {PartNumber}. Use arbitrary prefix/suffix: {Vol. SeriesPosition - } outputs 'Vol. 2 - ' when set, nothing when empty. Rename templates are filename-only (no '/' or '\'); use Organize for folders. Applies to single-file downloads.
- **Type:** string
- **Default:** `{Author} - {Title}`
@@ -538,10 +591,10 @@ Variables: {Author}, {Title}, {Year}, {User}, {OriginalName} (source filename wi
**Path Template**
Use / to create folders. Variables: {Author}, {Title}, {Year}, {User}, {OriginalName} (source filename without extension), {Series}, {SeriesPosition}, {Subtitle}, {PartNumber}. Use arbitrary prefix/suffix: {Vol. SeriesPosition - } outputs 'Vol. 2 - ' when set, nothing when empty.
Use / to create folders. Variables: {Author}, {Title}, {Year}, {User}, {OriginalName} (source filename without extension), {Series}, {SeriesPosition}, {Subtitle}, {PrimaryTitle}, {PartNumber}. Use arbitrary prefix/suffix: {Vol. SeriesPosition - } outputs 'Vol. 2 - ' when set, nothing when empty.
- **Type:** string
- **Default:** `{Author}/{Title}`
- **Default:** `{Author}/{Title}/{Title}`
#### `HARDLINK_TORRENTS_AUDIOBOOK`
@@ -561,14 +614,14 @@ Automatically open the downloads sidebar when a new download is queued.
- **Type:** boolean
- **Default:** `false`
#### `DOWNLOAD_TO_BROWSER`
#### `DOWNLOAD_TO_BROWSER_CONTENT_TYPES`
**Download to Browser**
Automatically download completed files to your browser.
Automatically download completed files to your browser for the selected content types.
- **Type:** boolean
- **Default:** `false`
- **Type:** string (comma-separated)
- **Default:** _empty list_
#### `MAX_CONCURRENT_DOWNLOADS`
@@ -597,7 +650,7 @@ How long to keep completed/failed downloads in the queue display.
| Variable | Description | Type | Default |
|----------|-------------|------|---------|
| `AUTH_METHOD` | Select the authentication method for accessing Shelfmark. | string (choice) | `none` |
| `AUTH_METHOD` | Select the authentication method for accessing Shelfmark. Restart container after changing Calibre-Web passwords. | string (choice) | `none` |
| `PROXY_AUTH_USER_HEADER` | The HTTP header your proxy uses to pass the authenticated username. | string | `X-Auth-User` |
| `PROXY_AUTH_LOGOUT_URL` | The URL to redirect users to for logging out. Leave empty to disable logout functionality. | string | _empty string_ |
| `PROXY_AUTH_ADMIN_GROUP_HEADER` | Optional: header your proxy uses to pass user groups/roles. | string | `X-Auth-Groups` |
@@ -605,7 +658,7 @@ How long to keep completed/failed downloads in the queue display.
| `OIDC_DISCOVERY_URL` | OpenID Connect discovery endpoint URL. Usually ends with /.well-known/openid-configuration. | string | _none_ |
| `OIDC_CLIENT_ID` | OAuth2 client ID from your identity provider. | string | _none_ |
| `OIDC_CLIENT_SECRET` | OAuth2 client secret from your identity provider. | string (secret) | _none_ |
| `OIDC_SCOPES` | OAuth2 scopes to request from the identity provider. Managed automatically: includes essential scopes and the group claim when using admin group authorization. | string | `openid,email,profile` |
| `OIDC_SCOPES` | OAuth2 scopes to request from the identity provider. Managed automatically: includes essential scopes and the group claim when using admin group authorization. | string (comma-separated) | `openid,email,profile` |
| `OIDC_GROUP_CLAIM` | The name of the claim in the ID token that contains user groups. | string | `groups` |
| `OIDC_ADMIN_GROUP` | Users in this group will be given admin access (if enabled below). Leave empty to use database roles only. | string | _empty string_ |
| `OIDC_USE_ADMIN_GROUP` | When enabled, users in the Admin Group are granted admin access. When disabled, admin access is determined solely by database roles. | boolean | `true` |
@@ -619,7 +672,7 @@ How long to keep completed/failed downloads in the queue display.
**Authentication Method**
Select the authentication method for accessing Shelfmark.
Select the authentication method for accessing Shelfmark. Restart container after changing Calibre-Web passwords.
- **Type:** string (choice)
- **Default:** `none`
@@ -697,7 +750,7 @@ OAuth2 client secret from your identity provider.
OAuth2 scopes to request from the identity provider. Managed automatically: includes essential scopes and the group claim when using admin group authorization.
- **Type:** string
- **Type:** string (comma-separated)
- **Default:** `openid,email,profile`
#### `OIDC_GROUP_CLAIM`
@@ -755,7 +808,7 @@ Custom label for the OIDC sign-in button on the login page.
| `CUSTOM_DNS` | DNS provider for domain resolution. 'Auto' rotates through providers on failure. | string (choice) | `auto` |
| `CUSTOM_DNS_MANUAL` | Comma-separated list of DNS server IP addresses (e.g., 8.8.8.8, 1.1.1.1). | string | _none_ |
| `USE_DOH` | Use encrypted DNS queries for improved reliability and privacy. | boolean | `true` |
| `USING_TOR` | Route all traffic through Tor for enhanced privacy. | boolean | `false` |
| `USING_TOR` | Route all traffic through Tor for enhanced privacy. Requires root startup. | boolean | `false` |
| `PROXY_MODE` | Choose proxy type. SOCKS5 handles all traffic through a single proxy. | string (choice) | `none` |
| `HTTP_PROXY` | HTTP proxy URL (e.g., http://proxy:8080) | string | _none_ |
| `HTTPS_PROXY` | HTTPS proxy URL (leave empty to use HTTP proxy for HTTPS) | string | _none_ |
@@ -807,7 +860,7 @@ Use encrypted DNS queries for improved reliability and privacy.
**Tor Routing**
Route all traffic through Tor for enhanced privacy.
Route all traffic through Tor for enhanced privacy. Requires root startup.
- **Type:** boolean
- **Default:** `false`
@@ -1073,6 +1126,57 @@ Automatically retry search without category filtering if no results are found
</details>
## Newznab
| Variable | Description | Type | Default |
|----------|-------------|------|---------|
| `NEWZNAB_ENABLED` | Enable searching for books via a Newznab-compatible indexer | boolean | `false` |
| `NEWZNAB_URL` | Base URL of your Newznab indexer or aggregator | string | _none_ |
| `NEWZNAB_API_KEY` | Your Newznab API key (leave blank if not required) | string (secret) | _none_ |
| `NEWZNAB_AUTO_EXPAND` | Automatically retry search without category filtering if no results are found | boolean | `false` |
<details>
<summary>Detailed descriptions</summary>
#### `NEWZNAB_ENABLED`
**Enable Newznab source**
Enable searching for books via a Newznab-compatible indexer
- **Type:** boolean
- **Default:** `false`
#### `NEWZNAB_URL`
**Newznab URL**
Base URL of your Newznab indexer or aggregator
- **Type:** string
- **Default:** _none_
- **Required:** Yes
#### `NEWZNAB_API_KEY`
**API Key**
Your Newznab API key (leave blank if not required)
- **Type:** string (secret)
- **Default:** _none_
#### `NEWZNAB_AUTO_EXPAND`
**Auto-expand search on no results**
Automatically retry search without category filtering if no results are found
- **Type:** boolean
- **Default:** `false`
</details>
## AudiobookBay
| Variable | Description | Type | Default |
@@ -1231,7 +1335,7 @@ How long to keep cached search results before they expire.
| `QBITTORRENT_CATEGORY` | Category to assign to book downloads in qBittorrent | string | `books` |
| `QBITTORRENT_CATEGORY_AUDIOBOOK` | Category for audiobook downloads. Leave empty to use the book category. | string | _empty string_ |
| `QBITTORRENT_DOWNLOAD_DIR` | Server-side directory where torrents are downloaded (optional, uses qBittorrent default if not specified) | string | _none_ |
| `QBITTORRENT_TAG` | Tag(s) to assign to qBittorrent downloads. Leave empty for no tags. | string | _empty list_ |
| `QBITTORRENT_TAG` | Tag(s) to assign to qBittorrent downloads. Leave empty for no tags. | string (comma-separated) | _empty list_ |
| `TRANSMISSION_URL` | URL of your Transmission instance (use https:// for TLS) | string | _none_ |
| `TRANSMISSION_USERNAME` | Transmission RPC username (if authentication enabled) | string | _none_ |
| `TRANSMISSION_PASSWORD` | Transmission RPC password | string (secret) | _none_ |
@@ -1249,6 +1353,7 @@ How long to keep cached search results before they expire.
| `RTORRENT_PASSWORD` | HTTP Basic auth password | string (secret) | _none_ |
| `RTORRENT_LABEL` | Label to assign to book downloads in rTorrent | string | `cwabd` |
| `RTORRENT_DOWNLOAD_DIR` | Server-side directory where torrents are downloaded (optional, uses rTorrent default if not specified) | string | _none_ |
| `PROWLARR_TORRENT_ACTION` | Remove deletes the torrent from your client immediately after import (stops seeding, files are kept); Keep leaves it in the client to continue seeding | string (choice) | `keep` |
| `PROWLARR_USENET_CLIENT` | Choose which usenet client to use | string (choice) | _empty string_ |
| `NZBGET_URL` | URL of your NZBGet instance | string | _none_ |
| `NZBGET_USERNAME` | NZBGet control username | string | `nzbget` |
@@ -1334,7 +1439,7 @@ Server-side directory where torrents are downloaded (optional, uses qBittorrent
Tag(s) to assign to qBittorrent downloads. Leave empty for no tags.
- **Type:** string
- **Type:** string (comma-separated)
- **Default:** _empty list_
#### `TRANSMISSION_URL`
@@ -1490,6 +1595,16 @@ Server-side directory where torrents are downloaded (optional, uses rTorrent def
- **Type:** string
- **Default:** _none_
#### `PROWLARR_TORRENT_ACTION`
**Torrent Completion Action**
Remove deletes the torrent from your client immediately after import (stops seeding, files are kept); Keep leaves it in the client to continue seeding
- **Type:** string (choice)
- **Default:** `keep`
- **Options:** `keep` (Keep), `remove` (Remove)
#### `PROWLARR_USENET_CLIENT`
**Usenet Client**
@@ -1604,6 +1719,7 @@ Move deletes the job from your usenet client after import; Copy keeps it in the
| `HARDCOVER_DEFAULT_SORT` | Default sort order for Hardcover search results. | string (choice) | `relevance` |
| `HARDCOVER_EXCLUDE_COMPILATIONS` | Filter out compilations, anthologies, and omnibus editions from search results | boolean | `false` |
| `HARDCOVER_EXCLUDE_UNRELEASED` | Filter out books with a release year in the future | boolean | `false` |
| `HARDCOVER_AUTO_REMOVE_ON_DOWNLOAD` | Automatically remove a book from the active Hardcover list when you download it | boolean | `true` |
<details>
<summary>Detailed descriptions</summary>
@@ -1655,6 +1771,15 @@ Filter out books with a release year in the future
- **Type:** boolean
- **Default:** `false`
#### `HARDCOVER_AUTO_REMOVE_ON_DOWNLOAD`
**Auto-Remove from List on Download**
Automatically remove a book from the active Hardcover list when you download it
- **Type:** boolean
- **Default:** `true`
</details>
### Metadata Providers: Open Library
@@ -1736,6 +1861,7 @@ Default sort order for Google Books search results.
| Variable | Description | Type | Default |
|----------|-------------|------|---------|
| `DIRECT_DOWNLOAD_ENABLED` | Show Direct Download in release-source lists and allow Direct mode searches. Add your own mirror URLs in the Mirrors tab before using it. | boolean | `false` |
| `AA_DONATOR_KEY` | Enables fast download access on AA. Get this from your donator account page. | string (secret) | _none_ |
| `FAST_SOURCES_DISPLAY` | Always tried first, no waiting or bypass required. | JSON array | _see UI for defaults_ |
| `SOURCE_PRIORITY` | Fallback sources, may have waiting. Requires bypasser. Drag to reorder. | JSON array | _see UI for defaults_ |
@@ -1754,6 +1880,15 @@ Default sort order for Google Books search results.
<details>
<summary>Detailed descriptions</summary>
#### `DIRECT_DOWNLOAD_ENABLED`
**Enable Direct Download Source**
Show Direct Download in release-source lists and allow Direct mode searches. Add your own mirror URLs in the Mirrors tab before using it.
- **Type:** boolean
- **Default:** `false`
#### `AA_DONATOR_KEY`
**Account Donator Key**
@@ -1938,14 +2073,11 @@ Timeout for external bypasser requests in milliseconds.
| Variable | Description | Type | Default |
|----------|-------------|------|---------|
| `AA_BASE_URL` | Select 'Auto' to try mirrors from your list on startup and fall back on failures. Choosing a specific mirror locks Shelfmark to that mirror (no fallback). | string (choice) | `auto` |
| `AA_MIRROR_URLS` | Editable list of AA mirrors. Used to populate the Primary Mirror dropdown and the order used when Auto is selected. Type a URL and press Enter to add. Order matters for auto-rotation | string | `https://annas-archive.gl,https://annas-archive.pk,https://annas-archive.vg,https://annas-archive.gd` |
| `AA_ADDITIONAL_URLS` | Deprecated. Use Mirrors instead. This is kept for backwards compatibility with existing installs and environment variables. | string | _none_ |
| `LIBGEN_ADDITIONAL_URLS` | Comma-separated list of custom LibGen mirrors to add to the defaults. | string | _none_ |
| `ZLIB_PRIMARY_URL` | Z-Library mirror to use for downloads. | string (choice) | `https://z-lib.fm` |
| `ZLIB_ADDITIONAL_URLS` | Comma-separated list of custom Z-Library mirror URLs. | string | _none_ |
| `WELIB_PRIMARY_URL` | Welib mirror to use for downloads. | string (choice) | `https://welib.org` |
| `WELIB_ADDITIONAL_URLS` | Comma-separated list of custom Welib mirror URLs. | string | _none_ |
| `AA_BASE_URL` | Select Auto to try mirrors from your list on startup and fail over on errors. Choosing a specific mirror pins Shelfmark to that URL. | string (choice) | `auto` |
| `AA_MIRROR_URLS` | List the Anna's Archive mirror URLs you want Shelfmark to use. Type a URL and press Enter to add it. Order matters when Auto is selected. | string (comma-separated) | _empty list_ |
| `LIBGEN_MIRROR_URLS` | Mirrors are tried in the order you add them until one works. | string (comma-separated) | _empty list_ |
| `ZLIB_MIRROR_URLS` | Only the first mirror in the list is used. | string (comma-separated) | _empty list_ |
| `WELIB_MIRROR_URLS` | Only the first mirror in the list is used. | string (comma-separated) | _empty list_ |
<details>
<summary>Detailed descriptions</summary>
@@ -1954,75 +2086,46 @@ Timeout for external bypasser requests in milliseconds.
**Primary Mirror**
Select 'Auto' to try mirrors from your list on startup and fall back on failures. Choosing a specific mirror locks Shelfmark to that mirror (no fallback).
Select Auto to try mirrors from your list on startup and fail over on errors. Choosing a specific mirror pins Shelfmark to that URL.
- **Type:** string (choice)
- **Default:** `auto`
- **Options:** `auto` (Auto (Recommended)), `https://annas-archive.gl` (annas-archive.gl), `https://annas-archive.pk` (annas-archive.pk), `https://annas-archive.vg` (annas-archive.vg), `https://annas-archive.gd` (annas-archive.gd)
- **Options:** `auto` (Auto (Recommended))
#### `AA_MIRROR_URLS`
**Mirrors**
Editable list of AA mirrors. Used to populate the Primary Mirror dropdown and the order used when Auto is selected. Type a URL and press Enter to add. Order matters for auto-rotation
List the Anna's Archive mirror URLs you want Shelfmark to use. Type a URL and press Enter to add it. Order matters when Auto is selected.
- **Type:** string
- **Default:** `https://annas-archive.gl,https://annas-archive.pk,https://annas-archive.vg,https://annas-archive.gd`
- **Type:** string (comma-separated)
- **Default:** _empty list_
#### `AA_ADDITIONAL_URLS`
#### `LIBGEN_MIRROR_URLS`
**Additional Mirrors (Legacy)**
**LibGen**
Deprecated. Use Mirrors instead. This is kept for backwards compatibility with existing installs and environment variables.
Mirrors are tried in the order you add them until one works.
- **Type:** string
- **Default:** _none_
- **Type:** string (comma-separated)
- **Default:** _empty list_
#### `LIBGEN_ADDITIONAL_URLS`
#### `ZLIB_MIRROR_URLS`
**Additional Mirrors**
**Z-Library**
Comma-separated list of custom LibGen mirrors to add to the defaults.
Only the first mirror in the list is used.
- **Type:** string
- **Default:** _none_
- **Type:** string (comma-separated)
- **Default:** _empty list_
#### `ZLIB_PRIMARY_URL`
#### `WELIB_MIRROR_URLS`
**Primary Mirror**
**Welib**
Z-Library mirror to use for downloads.
Only the first mirror in the list is used.
- **Type:** string (choice)
- **Default:** `https://z-lib.fm`
- **Options:** `https://z-lib.fm` (z-lib.fm), `https://z-lib.gs` (z-lib.gs), `https://z-lib.id` (z-lib.id), `https://z-library.sk` (z-library.sk), `https://zlibrary-global.se` (zlibrary-global.se)
#### `ZLIB_ADDITIONAL_URLS`
**Additional Mirrors**
Comma-separated list of custom Z-Library mirror URLs.
- **Type:** string
- **Default:** _none_
#### `WELIB_PRIMARY_URL`
**Primary Mirror**
Welib mirror to use for downloads.
- **Type:** string (choice)
- **Default:** `https://welib.org`
- **Options:** `https://welib.org` (welib.org)
#### `WELIB_ADDITIONAL_URLS`
**Additional Mirrors**
Comma-separated list of custom Welib mirror URLs.
- **Type:** string
- **Default:** _none_
- **Type:** string (comma-separated)
- **Default:** _empty list_
</details>
+21 -1
View File
@@ -1,3 +1,23 @@
# Shelfmark Documentation
TODO
Shelfmark is a self-hosted interface for searching, requesting, and delivering books and audiobooks through the sources and services you choose to configure.
Use the guides below to set up the app, connect your library tools, and understand the main configuration areas.
## Getting Started
- [Installation](installation.md)
- [Directory and Volume Setup](configuration.md)
- [Environment Variables](environment-variables.md)
## Core Guides
- [Users & Requests](users-and-requests.md)
- [Reverse Proxy](reverse-proxy.md)
- [OIDC](oidc.md)
- [URL Search Parameters](url-search-parameters.md)
- [Custom Scripts](custom-scripts.md)
## Help
- [Troubleshooting](troubleshooting.md)
+31 -1
View File
@@ -1,3 +1,33 @@
# Installation
TODO
Shelfmark is typically deployed with Docker Compose.
## Quick Start
1. Download the compose file from the repository:
```bash
curl -O https://raw.githubusercontent.com/calibrain/shelfmark/main/compose/docker-compose.yml
```
2. Start the service:
```bash
docker compose up -d
```
3. Open `http://localhost:8084`
4. Configure the sources, metadata providers, and delivery settings you want to use
## Next Steps
- For volume and path setup, see [Directory and Volume Setup](configuration.md)
- For environment-based setup, see [Environment Variables](environment-variables.md)
- For authentication and user management, see [Users & Requests](users-and-requests.md) and [OIDC](oidc.md)
## Notes
- Universal search is the default mode for new installs
- Direct Download is optional and must be enabled and configured before it can be used
- Torrent and usenet setups require matching download paths between Shelfmark and your download client
+24 -3
View File
@@ -6,6 +6,15 @@ Shelfmark can run behind a reverse proxy at the root path (recommended) or under
If you can serve Shelfmark at the root path (`https://shelfmark.example.com/`), leave `URL_BASE` empty. This is the simplest option and avoids extra subpath configuration.
Define this once in your Nginx `http` block so websocket upgrades are only sent when the client actually requests them:
```nginx
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
```
```nginx
server {
listen 443 ssl;
@@ -19,7 +28,7 @@ server {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Connection $connection_upgrade;
}
}
```
@@ -53,7 +62,7 @@ location /shelfmark/ {
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Connection $connection_upgrade;
proxy_read_timeout 86400;
proxy_send_timeout 86400;
proxy_buffering off;
@@ -133,7 +142,7 @@ location /shelfmark/ {
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Connection $connection_upgrade;
proxy_read_timeout 86400;
proxy_send_timeout 86400;
proxy_buffering off;
@@ -142,6 +151,18 @@ location /shelfmark/ {
---
## Troubleshooting false network errors
If login, settings saves, or downloads appear to fail in the browser but the action still completes on the server, check your proxy headers first.
- Do not force `Connection: upgrade` on every request. That can break normal `POST` and `PUT` responses while the backend still processes them.
- If your proxy UI does not support conditional websocket headers, remove the forced websocket headers entirely and let Shelfmark fall back to polling.
- Keep the standard forwarded headers: `Host`, `X-Forwarded-For`, `X-Forwarded-Proto`, and `X-Forwarded-Host` when using a subpath or OIDC.
This is especially relevant for Nginx Proxy Manager or custom advanced config snippets that add websocket headers globally.
---
## Health checks
Health checks work at `/shelfmark/api/health` when using a subpath configuration.
+2 -2
View File
@@ -65,9 +65,9 @@ Some parameters support multiple values by repeating the parameter:
## Search Mode Behavior
### Direct Download Mode (default)
### Direct Mode
All parameters are used to filter results from the direct download source.
When Search Mode is set to Direct, all parameters are used to filter results from the configured direct source.
`content_type` is ignored in Direct mode.
### Universal Mode
+330 -136
View File
@@ -1,5 +1,7 @@
#!/bin/bash
set -e
is_truthy() {
case "${1,,}" in
true|yes|1|y) return 0 ;;
@@ -11,6 +13,16 @@ ENABLE_LOGGING_VALUE="${ENABLE_LOGGING:-true}"
LOG_PIPE_DIR=""
LOG_PIPE=""
TEE_PID=""
FILE_LOGGING_ENABLED="false"
CURRENT_UID=$(id -u)
CURRENT_GID=$(id -g)
RUN_AS_NON_ROOT="false"
RUNTIME_TMP_DIR="${TMP_DIR:-/tmp/shelfmark}"
DEFAULT_RUNTIME_HOME="${RUNTIME_TMP_DIR}/home"
if [ "$CURRENT_UID" != "0" ]; then
RUN_AS_NON_ROOT="true"
fi
start_file_logging() {
local logfile="$1"
@@ -43,30 +55,68 @@ stop_file_logging() {
if is_truthy "$ENABLE_LOGGING_VALUE"; then
LOG_DIR=${LOG_ROOT:-/var/log/}/shelfmark
mkdir -p "$LOG_DIR"
LOG_FILE="${LOG_DIR}/shelfmark_entrypoint.log"
# Keep the previous entrypoint log instead of deleting all history on boot.
[ -f "${LOG_FILE}.prev" ] && rm -f "${LOG_FILE}.prev"
[ -f "$LOG_FILE" ] && mv "$LOG_FILE" "${LOG_FILE}.prev"
if mkdir -p "$LOG_DIR" 2>/dev/null; then
LOG_FILE="${LOG_DIR}/shelfmark_entrypoint.log"
# Keep the previous entrypoint log instead of deleting all history on boot.
rotation_ok="true"
if [ -f "${LOG_FILE}.prev" ] && ! rm -f "${LOG_FILE}.prev"; then
echo "Warning: could not remove previous entrypoint log ${LOG_FILE}.prev, continuing without file logging" >&2
rotation_ok="false"
fi
if [ "$rotation_ok" = "true" ] && [ -f "$LOG_FILE" ] && ! mv "$LOG_FILE" "${LOG_FILE}.prev"; then
echo "Warning: could not rotate entrypoint log $LOG_FILE, continuing without file logging" >&2
rotation_ok="false"
fi
if [ "$rotation_ok" = "true" ]; then
FILE_LOGGING_ENABLED="true"
else
ENABLE_LOGGING_VALUE="false"
export ENABLE_LOGGING="false"
fi
else
echo "Warning: could not create log directory $LOG_DIR, continuing without file logging" >&2
ENABLE_LOGGING_VALUE="false"
export ENABLE_LOGGING="false"
fi
fi
(
if [ "$USING_TOR" = "true" ]; then
./tor.sh
if [ "$USING_TOR" = "true" ]; then
if [ "$RUN_AS_NON_ROOT" = "true" ]; then
echo "USING_TOR=true requires the container to start as root." >&2
echo "Non-root mode skips the privileged filesystem and network setup Tor depends on." >&2
exit 1
fi
)
./tor.sh
fi
if is_truthy "$ENABLE_LOGGING_VALUE"; then
if [ "$FILE_LOGGING_ENABLED" = "true" ]; then
start_file_logging "$LOG_FILE"
fi
echo "Starting entrypoint script"
if is_truthy "$ENABLE_LOGGING_VALUE"; then
if [ "$FILE_LOGGING_ENABLED" = "true" ]; then
echo "Log file: $LOG_FILE"
else
echo "File logging disabled (ENABLE_LOGGING=$ENABLE_LOGGING_VALUE)"
fi
set -e
PYTHON_BIN="/app/.venv/bin/python"
if [ ! -x "$PYTHON_BIN" ]; then
PYTHON_BIN="python3"
fi
# Defensive: some orchestrators (e.g. Unraid Dockhand templates) inject a default
# PATH that drops the venv bin directory baked in by the Dockerfile. Prepend it
# so subprocesses launched without an absolute path still resolve correctly.
case ":${PATH}:" in
*":/app/.venv/bin:"*) ;;
*) export PATH="/app/.venv/bin:${PATH}" ;;
esac
GUNICORN_BIN="/app/.venv/bin/gunicorn"
if [ ! -x "$GUNICORN_BIN" ]; then
GUNICORN_BIN="gunicorn"
fi
# Print build version
echo "Build version: $BUILD_VERSION"
@@ -74,57 +124,104 @@ echo "Release version: $RELEASE_VERSION"
# Configure timezone
if [ "$TZ" ]; then
echo "Setting timezone to $TZ"
ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
if [ "$RUN_AS_NON_ROOT" = "true" ]; then
echo "TZ is set to $TZ (non-root mode leaves /etc/localtime unchanged)"
else
echo "Setting timezone to $TZ"
ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
fi
fi
# Determine user ID with proper precedence:
# 1. PUID (LinuxServer.io standard - recommended)
# 2. UID (legacy, for backward compatibility with existing installs)
# 3. Default to 1000
#
# Note: $UID is a bash builtin that's always set. We use `printenv` to detect
# if UID was explicitly set as an environment variable (e.g., via docker-compose).
if [ -n "$PUID" ]; then
RUN_UID="$PUID"
echo "Using PUID=$RUN_UID"
elif printenv UID >/dev/null 2>&1; then
RUN_UID="$(printenv UID)"
echo "Using UID=$RUN_UID (legacy - consider migrating to PUID)"
if [ "$RUN_AS_NON_ROOT" = "true" ]; then
RUN_UID="$CURRENT_UID"
RUN_GID="$CURRENT_GID"
USERNAME=$(getent passwd "$RUN_UID" 2>/dev/null | cut -d: -f1 || true)
if [ -z "$USERNAME" ]; then
USERNAME="$RUN_UID"
echo "No passwd entry found for UID $RUN_UID; using numeric identity"
fi
TARGET_USER_SPEC="${RUN_UID}:${RUN_GID}"
else
RUN_UID=1000
echo "Using default UID=$RUN_UID"
# Determine user ID with proper precedence:
# 1. PUID (LinuxServer.io standard - recommended)
# 2. UID (legacy, for backward compatibility with existing installs)
# 3. Default to 1000
#
# Note: $UID is a bash builtin that's always set. We use `printenv` to detect
# if UID was explicitly set as an environment variable (e.g., via docker-compose).
if [ -n "$PUID" ]; then
RUN_UID="$PUID"
echo "Using PUID=$RUN_UID"
elif printenv UID >/dev/null 2>&1; then
RUN_UID="$(printenv UID)"
echo "Using UID=$RUN_UID (legacy - consider migrating to PUID)"
else
RUN_UID=1000
echo "Using default UID=$RUN_UID"
fi
# Determine group ID with proper precedence:
# 1. PGID (LinuxServer.io standard - recommended)
# 2. GID (legacy, for backward compatibility with existing installs)
# 3. Default to 1000
if [ -n "$PGID" ]; then
RUN_GID="$PGID"
echo "Using PGID=$RUN_GID"
elif [ -n "$GID" ]; then
RUN_GID="$GID"
echo "Using GID=$RUN_GID (legacy - consider migrating to PGID)"
else
RUN_GID=1000
echo "Using default GID=$RUN_GID"
fi
if ! getent group "$RUN_GID" >/dev/null; then
echo "Adding group $RUN_GID with name appuser"
groupadd -g "$RUN_GID" appuser
fi
# Create user if it doesn't exist for this UID yet.
if ! getent passwd "$RUN_UID" >/dev/null; then
echo "Adding user $RUN_UID with name appuser"
useradd -u "$RUN_UID" -g "$RUN_GID" -d "$DEFAULT_RUNTIME_HOME" -s /sbin/nologin appuser
fi
# Get username for the UID (whether we just created it or it existed)
USERNAME=$(getent passwd "$RUN_UID" | cut -d: -f1)
if [ -z "$USERNAME" ]; then
USERNAME="$RUN_UID"
fi
TARGET_USER_SPEC="${RUN_UID}:${RUN_GID}"
fi
# Determine group ID with proper precedence:
# 1. PGID (LinuxServer.io standard - recommended)
# 2. GID (legacy, for backward compatibility with existing installs)
# 3. Default to 1000
if [ -n "$PGID" ]; then
RUN_GID="$PGID"
echo "Using PGID=$RUN_GID"
elif [ -n "$GID" ]; then
RUN_GID="$GID"
echo "Using GID=$RUN_GID (legacy - consider migrating to PGID)"
else
RUN_GID=1000
echo "Using default GID=$RUN_GID"
fi
# Avoid unnecessary gosu hops when we're already running as the target user.
# Some nested LXC setups spin on root-to-root gosu invocations.
needs_user_switch() {
local current_uid
local current_gid
if ! getent group "$RUN_GID" >/dev/null; then
echo "Adding group $RUN_GID with name appuser"
groupadd -g "$RUN_GID" appuser
fi
current_uid=$(id -u)
current_gid=$(id -g)
# Create user if it doesn't exist
if ! id -u "$RUN_UID" >/dev/null 2>&1; then
echo "Adding user $RUN_UID with name appuser"
useradd -u "$RUN_UID" -g "$RUN_GID" -d /app -s /sbin/nologin appuser
fi
[ "$current_uid" != "$RUN_UID" ] || [ "$current_gid" != "$RUN_GID" ]
}
# Get username for the UID (whether we just created it or it existed)
USERNAME=$(getent passwd "$RUN_UID" | cut -d: -f1)
echo "Username for UID $RUN_UID is $USERNAME"
run_as_target_user() {
if needs_user_switch; then
gosu "$TARGET_USER_SPEC" "$@"
return $?
fi
"$@"
}
exec_as_target_user() {
if needs_user_switch; then
exec gosu "$TARGET_USER_SPEC" "$@"
fi
exec "$@"
}
test_write() {
local folder=$1
@@ -138,9 +235,7 @@ test_write() {
return 1
fi
if ! (
echo 0123456789_TEST | gosu "$USERNAME" env HOME=/app tee "$test_file" > /dev/null
); then
if ! run_as_target_user sh -c 'echo 0123456789_TEST > "$1"' _ "$test_file"; then
echo "Failed to write test file in $folder as $USERNAME"
return 1
fi
@@ -159,34 +254,43 @@ test_write() {
}
make_writable() {
folder=$1
did_full_chown=0
local folder="$1"
local mode="${2:-tree}"
local did_full_chown=0
local is_writable
set +e
test_write $folder
test_write "$folder"
is_writable=$?
set -e
if [ $is_writable -eq 0 ]; then
echo "Folder $folder is writable, no need to change ownership"
else
echo "Folder $folder is not writable, changing ownership"
change_ownership $folder
chmod -R g+r,g+w $folder || echo "Failed to change group permissions for ${folder}, continuing..."
if [ "$mode" = "root" ]; then
echo "Folder $folder is not writable, fixing top-level ownership and permissions"
mkdir -p "$folder"
chown "${RUN_UID}:${RUN_GID}" "$folder" || echo "Failed to change ownership for ${folder}, continuing..."
chmod u+rwx "$folder" || echo "Failed to change owner permissions for ${folder}, continuing..."
else
echo "Folder $folder is not writable, changing ownership"
change_ownership "$folder"
chmod -R g+r,g+w "$folder" || echo "Failed to change group permissions for ${folder}, continuing..."
fi
did_full_chown=1
fi
# Fix any misowned subdirectories/files (e.g., from previous runs as root)
if [ "$did_full_chown" -eq 0 ] && [ -d "$folder" ]; then
if [ "$mode" = "tree" ] && [ "$did_full_chown" -eq 0 ] && [ -d "$folder" ]; then
echo "Checking for misowned files/directories in $folder"
# Stay on the same filesystem to avoid traversing mounted subpaths
# (for example read-only bind mounts under /app in dev setups).
find "$folder" -xdev -mindepth 1 \( ! -user "$RUN_UID" -o ! -group "$RUN_GID" \) \
-exec chown "$RUN_UID:$RUN_GID" {} + 2>/dev/null || true
fi
test_write $folder || echo "Failed to test write to ${folder}, continuing..."
test_write "$folder" || echo "Failed to test write to ${folder}, continuing..."
}
fix_misowned() {
folder=$1
mkdir -p $folder
local folder="$1"
mkdir -p "$folder"
echo "Checking for misowned files/directories in $folder"
# Stay on the same filesystem to avoid traversing mounted subpaths
# (for example read-only bind mounts under /app in dev setups).
@@ -196,81 +300,154 @@ fix_misowned() {
# Ensure proper ownership of application directories
change_ownership() {
folder=$1
mkdir -p $folder
local folder="$1"
mkdir -p "$folder"
echo "Changing ownership of $folder to $USERNAME:$RUN_GID"
chown -R "${RUN_UID}:${RUN_GID}" "${folder}" || echo "Failed to change ownership for ${folder}, continuing..."
}
fix_misowned /app
fix_misowned /var/log/shelfmark
fix_misowned /tmp/shelfmark
require_writable_dir() {
local folder="$1"
local label="${2:-Directory}"
# SeleniumBase (internal bypasser) writes a patched chromedriver binary (uc_driver)
# into its own drivers directory. Some NAS/docker setups can apply restrictive ACLs
# to extracted image layers that block non-root writes; ensure the runtime UID owns it.
if [ "${USING_EXTERNAL_BYPASSER}" != "true" ]; then
set +e
SELENIUMBASE_DRIVERS_DIR=$(python3 -c "import pathlib, seleniumbase; print(pathlib.Path(seleniumbase.__file__).resolve().parent / 'drivers')" 2>/dev/null)
set -e
if ! mkdir -p "$folder"; then
echo "Failed to create ${label} directory: $folder"
exit 1
fi
if [ -n "$SELENIUMBASE_DRIVERS_DIR" ] && [ -d "$SELENIUMBASE_DRIVERS_DIR" ]; then
change_ownership "$SELENIUMBASE_DRIVERS_DIR"
if ! test_write "$folder"; then
echo "${label} directory is not writable in non-root mode: $folder"
echo "Prepare ownership outside the container (for example with a pre-owned volume or Kubernetes fsGroup)."
exit 1
fi
}
# If the driver already exists, ensure it's executable for the runtime user.
if [ -f "${SELENIUMBASE_DRIVERS_DIR}/uc_driver" ]; then
chmod +x "${SELENIUMBASE_DRIVERS_DIR}/uc_driver" || echo "Failed to chmod uc_driver, continuing..."
resolve_runtime_home() {
local runtime_home
runtime_home=$(getent passwd "$RUN_UID" 2>/dev/null | cut -d: -f6 || true)
case "$runtime_home" in
""|/|/app|/nonexistent)
runtime_home="$DEFAULT_RUNTIME_HOME"
;;
esac
printf '%s\n' "$runtime_home"
}
ensure_tree_writable() {
local folder="$1"
make_writable "$folder"
if [ -d "$folder" ]; then
chmod -R u+rwX,g+rwX "$folder" || echo "Failed to relax permissions for ${folder}, continuing..."
fi
}
ensure_symlinked_dir() {
local link_path="$1"
local target_path="$2"
ensure_tree_writable "$target_path"
if [ -L "$link_path" ]; then
local current_target
current_target=$(readlink "$link_path" 2>/dev/null || echo "")
if [ "$current_target" = "$target_path" ]; then
echo "$link_path already points to $target_path"
return 0
fi
echo "Replacing symlink $link_path -> $current_target with $target_path"
rm -f "$link_path" || echo "Failed to replace symlink ${link_path}, continuing..."
elif [ -d "$link_path" ]; then
echo "Moving existing scratch files from $link_path to $target_path"
find "$link_path" -xdev -mindepth 1 -maxdepth 1 -exec mv -t "$target_path" {} + 2>/dev/null || true
ensure_tree_writable "$target_path"
if ! rmdir "$link_path" 2>/dev/null; then
echo "Could not replace $link_path with symlink, leaving existing directory in place"
ensure_tree_writable "$link_path"
return 0
fi
elif [ -e "$link_path" ]; then
echo "$link_path exists and is not a directory, leaving it in place"
return 0
fi
if [ ! -e "$link_path" ]; then
ln -s "$target_path" "$link_path" || echo "Failed to create symlink ${link_path}, continuing..."
fi
}
if [ "$RUN_AS_NON_ROOT" = "true" ]; then
require_writable_dir /tmp/shelfmark "Temporary"
if [ "${USING_EXTERNAL_BYPASSER}" != "true" ]; then
require_writable_dir /tmp/shelfmark/seleniumbase/downloaded_files "SeleniumBase downloads"
require_writable_dir /tmp/shelfmark/seleniumbase/archived_files "SeleniumBase archive"
fi
require_writable_dir "${CONFIG_DIR:-/config}" "Config"
else
fix_misowned /var/log/shelfmark
fix_misowned /tmp/shelfmark
# Keep SeleniumBase on its default /app-based paths, but redirect the scratch
# directories into /tmp so bypasser startup doesn't depend on image-layer writes.
if [ "${USING_EXTERNAL_BYPASSER}" != "true" ]; then
ensure_symlinked_dir /app/downloaded_files /tmp/shelfmark/seleniumbase/downloaded_files
ensure_symlinked_dir /app/archived_files /tmp/shelfmark/seleniumbase/archived_files
# Keep SeleniumBase's bundled drivers directory writable as well for
# compatibility with legacy UC code paths that still probe bundled assets.
set +e
SELENIUMBASE_DRIVERS_DIR=$("$PYTHON_BIN" -c "import pathlib, seleniumbase; print(pathlib.Path(seleniumbase.__file__).resolve().parent / 'drivers')" 2>/dev/null)
set -e
if [ -n "$SELENIUMBASE_DRIVERS_DIR" ] && [ -d "$SELENIUMBASE_DRIVERS_DIR" ]; then
change_ownership "$SELENIUMBASE_DRIVERS_DIR"
# If the legacy driver already exists, ensure it's executable for the runtime user.
if [ -f "${SELENIUMBASE_DRIVERS_DIR}/uc_driver" ]; then
chmod +x "${SELENIUMBASE_DRIVERS_DIR}/uc_driver" || echo "Failed to chmod uc_driver, continuing..."
fi
fi
fi
fi
# Test write to all folders
make_writable ${CONFIG_DIR:-/config}
make_writable ${INGEST_DIR:-/books}
# Config is Shelfmark-owned state, so it keeps the thorough repair path.
make_writable "${CONFIG_DIR:-/config}" tree
# Fix permissions on directories configured in settings
echo "Checking for additional configured directories..."
if [ -f /app/scripts/fix_permissions.py ]; then
configured_dirs=$(python3 /app/scripts/fix_permissions.py 2>/dev/null || echo "")
if [ -n "$configured_dirs" ]; then
echo "$configured_dirs" | while read -r dir; do
if [ -n "$dir" ] && [ -d "$dir" ]; then
echo "Checking configured directory: $dir"
make_writable "$dir"
fi
done
fi
fi
# Fallback to root if config dir is still not writable (common on NAS/Unraid after upgrade from v0.4.0)
CONFIG_PATH=${CONFIG_DIR:-/config}
set +e
test_write "$CONFIG_PATH" >/dev/null 2>&1
config_ok=$?
set -e
# Fallback to root if config dir is still not writable (common on NAS/Unraid after upgrade from v0.4.0)
CONFIG_PATH=${CONFIG_DIR:-/config}
set +e
test_write "$CONFIG_PATH" >/dev/null 2>&1
config_ok=$?
set -e
if [ $config_ok -ne 0 ] && [ "$RUN_UID" != "0" ]; then
config_owner=$(stat -c '%u' "$CONFIG_PATH" 2>/dev/null || echo "unknown")
if [ "$config_owner" = "0" ]; then
echo ""
echo "========================================================"
echo "WARNING: Permission issue detected!"
echo ""
echo "Config directory is owned by root but PUID=$RUN_UID."
echo "This typically happens after upgrading from v0.4.0 where"
echo "PUID/PGID settings were not respected."
echo ""
echo "Falling back to running as root to prevent data loss."
echo ""
echo "To fix this permanently, run on your HOST machine:"
echo " chown -R $RUN_UID:$RUN_GID /path/to/config"
echo ""
echo "Then restart the container."
echo "========================================================"
echo ""
RUN_UID=0
RUN_GID=0
USERNAME=root
if [ $config_ok -ne 0 ] && [ "$RUN_UID" != "0" ]; then
config_owner=$(stat -c '%u' "$CONFIG_PATH" 2>/dev/null || echo "unknown")
if [ "$config_owner" = "0" ]; then
echo ""
echo "========================================================"
echo "WARNING: Permission issue detected!"
echo ""
echo "Config directory is owned by root but PUID=$RUN_UID."
echo "This typically happens after upgrading from v0.4.0 where"
echo "PUID/PGID settings were not respected."
echo ""
echo "Falling back to running as root to prevent data loss."
echo ""
echo "To fix this permanently, run on your HOST machine:"
echo " chown -R $RUN_UID:$RUN_GID /path/to/config"
echo ""
echo "Then restart the container."
echo "========================================================"
echo ""
RUN_UID=0
RUN_GID=0
USERNAME=root
TARGET_USER_SPEC="0:0"
fi
fi
fi
@@ -278,7 +455,7 @@ fi
# upgrades work reliably on customer machines.
# Map app LOG_LEVEL (often DEBUG/INFO/...) to gunicorn's --log-level (lowercase).
gunicorn_loglevel=$([ "$DEBUG" = "true" ] && echo debug || echo "${LOG_LEVEL:-info}" | tr '[:upper:]' '[:lower:]')
command="gunicorn --log-level ${gunicorn_loglevel} --access-logfile - --error-logfile - --worker-class geventwebsocket.gunicorn.workers.GeventWebSocketWorker --workers 1 -t 300 -b ${FLASK_HOST:-0.0.0.0}:${FLASK_PORT:-8084} shelfmark.main:app"
command="${GUNICORN_BIN} --log-level ${gunicorn_loglevel} --access-logfile - --error-logfile - --worker-class geventwebsocket.gunicorn.workers.GeventWebSocketWorker --workers 1 -t 300 -b ${FLASK_HOST:-0.0.0.0}:${FLASK_PORT:-8084} shelfmark.main:app"
# If DEBUG and not using an external bypass
if [ "$DEBUG" = "true" ] && [ "$USING_EXTERNAL_BYPASSER" != "true" ]; then
@@ -286,7 +463,7 @@ if [ "$DEBUG" = "true" ] && [ "$USING_EXTERNAL_BYPASSER" != "true" ]; then
set -x
echo "vvvvvvvvvvvv DEBUG MODE vvvvvvvvvvvv"
echo "Starting Xvfb for debugging"
python3 -c "from pyvirtualdisplay import Display; Display(visible=False, size=(1440,1880)).start()"
"$PYTHON_BIN" -c "from pyvirtualdisplay import Display; Display(visible=False, size=(1440,1880)).start()"
id
free -h
uname -a
@@ -306,7 +483,7 @@ if [ "$DEBUG" = "true" ] && [ "$USING_EXTERNAL_BYPASSER" != "true" ]; then
--enable-logging --v=1 --log-level=0 \
--log-file=/tmp/chrome_entrypoint_test.log \
--crash-dumps-dir=/tmp/chrome_crash_dumps \
< /dev/null
< /dev/null
EXIT_CODE=$?
echo "Chrome exit code: $EXIT_CODE"
ls -lh /tmp/chrome_entrypoint_test.log
@@ -346,7 +523,24 @@ else
exit 1
fi
echo "Running command: '$command' as '$USERNAME' (debug=$is_debug)"
RUNTIME_HOME=$(resolve_runtime_home)
if [ "$RUN_AS_NON_ROOT" = "true" ]; then
require_writable_dir "$RUNTIME_HOME" "Home"
else
mkdir -p "$RUNTIME_HOME"
make_writable "$RUNTIME_HOME" tree
fi
if [ "$RUN_AS_NON_ROOT" = "true" ]; then
echo "Startup mode: non-root"
elif [ "$RUN_UID" = "0" ] && [ "$RUN_GID" = "0" ]; then
echo "Startup mode: root"
else
echo "Startup mode: root bootstrap with privilege drop"
fi
echo "Runtime identity: $USERNAME (${RUN_UID}:${RUN_GID})"
echo "Running command: '$command' as '$USERNAME' (debug=${DEBUG:-false})"
# Set umask for file permissions (default: 0022 = files 644, dirs 755)
UMASK_VALUE=${UMASK:-0022}
@@ -354,4 +548,4 @@ echo "Setting umask to $UMASK_VALUE"
umask $UMASK_VALUE
stop_file_logging
exec gosu "$USERNAME" env HOME=/app $command
exec_as_target_user env HOME="$RUNTIME_HOME" $command
-1
View File
@@ -199,4 +199,3 @@ else
echo "Failed to create debug archive"
exit 1
fi
+152 -6
View File
@@ -2,7 +2,47 @@
name = "shelfmark"
version = "0.1.0"
description = "Shelfmark - Book Downloader"
requires-python = ">=3.10"
requires-python = ">=3.14"
dependencies = [
"flask",
"flask-cors",
"flask-socketio",
"python-socketio",
"requests[socks]",
"defusedxml",
"beautifulsoup4",
"tqdm",
"dnspython",
"gunicorn",
"gevent",
"gevent-websocket",
"psutil",
"emoji",
"rarfile",
"qbittorrent-api",
"transmission-rpc",
"authlib>=1.7.0,<1.8",
"apprise>=1.9.0",
]
[project.optional-dependencies]
browser = [
"pyvirtualdisplay",
"pyautogui",
"seleniumbase==4.48.2",
"python-xlib",
]
[dependency-groups]
dev = [
"basedpyright>=1.39.3",
"prek",
"pytest",
"pytest-cov",
"pytest-xdist>=3.8.0",
"ruff==0.15.11",
"vulture>=2.14",
]
[tool.pytest.ini_options]
testpaths = ["tests"]
@@ -12,6 +52,8 @@ python_functions = ["test_*"]
addopts = [
"-v",
"--tb=short",
"-n",
"auto",
]
markers = [
"integration: marks tests that require running services (deselect with '-m \"not integration\"')",
@@ -19,8 +61,112 @@ markers = [
"e2e: marks end-to-end tests that require the full application stack",
]
[tool.mypy]
python_version = "3.10"
warn_return_any = true
warn_unused_ignores = true
ignore_missing_imports = true
[tool.ruff]
line-length = 100
extend-exclude = [".local"]
[tool.ruff.lint]
select = [
"F", "I", "UP", "B", "C4", "SIM", "PTH", "RET", "PIE", "FURB", "PERF", "TRY",
"A", "DTZ", "N",
"BLE001",
"ANN001", "ANN002", "ANN003", "ANN201", "ANN202", "ANN204",
"E402",
"ERA001",
"E731",
"S101",
"S110",
"S105", "S108",
"S311", "S324",
"S607", "S608",
"G003", "G004",
"PGH003",
"PLC0414",
"PLR1714",
"PLW1510",
"PLW2901",
"PLW0108",
"PT028",
"PYI034",
"Q000",
"RUF005", "RUF012", "RUF013", "RUF059", "RUF100",
"TC001", "TC002", "TC003",
]
ignore = ["D", "EM", "FBT", "PLR2004", "UP035", "TRY003", "E501", "TD002", "S104", "S603"]
[tool.ruff.lint.per-file-ignores]
"scripts/**/*.py" = [
"BLE001",
"S",
"TRY",
]
"tests/**/*.py" = [
"ANN",
"BLE001",
"B010",
"B017",
"B028",
"DTZ",
"E402",
"E731",
"ERA001",
"FURB",
"G003",
"G004",
"PERF",
"PIE",
"PLC0414",
"PLW0108",
"PLW1510",
"PLW2901",
"PTH",
"PT028",
"PYI034",
"Q000",
"RET",
"RUF012",
"S",
"SIM",
"TC001",
"TC002",
"TC003",
"TRY",
"UP028",
]
[tool.basedpyright]
include = ["shelfmark"]
exclude = [".local", "tests", "**/__pycache__", "**/node_modules"]
pythonVersion = "3.14"
typeCheckingMode = "standard"
[tool.vulture]
paths = ["shelfmark"]
exclude = [".local", "tests"]
ignore_decorators = [
"@app.route",
"@app.before_request",
"@app.after_request",
"@app.errorhandler",
"@socketio.on",
"@register_provider",
"@register_provider_kwargs",
"@register_settings",
"@register_source",
"@register_handler",
"@register_client",
"@register_output",
]
min_confidence = 90
sort_by_size = true
[tool.coverage.run]
source = ["shelfmark"]
branch = true
[tool.coverage.report]
show_missing = true
skip_empty = true
[tool.uv]
package = false
+45 -40
View File
@@ -1,30 +1,26 @@
# 📚 Shelfmark: Book Downloader
Formerly *Calibre Web Automated Book Downloader (CWABD)*
# 📚 Shelfmark: Book Search & Request Tool
<img src="src/frontend/public/logo.png" alt="Shelfmark" width="200">
Shelfmark is a self-hosted web interface for searching and downloading books and audiobooks from multiple sources. Works out of the box with popular web sources, no configuration required. Add metadata providers, additional release sources, and download clients to build a single hub for your digital library. Supports multiple users with a built-in request system, so you can share your instance with others and let them browse and request books on their own.
Shelfmark is a self-hosted web interface for searching and requesting books and audiobooks across multiple sources. Bring your own sources, metadata providers, and download clients to build a single hub for your digital library. Supports multiple users with a built-in request system, so you can share your instance with others and let them browse and request books on their own.
**Fully standalone** - no external dependencies required. Works great alongside the following library tools, with support for automatic imports:
Works great alongside the following library tools, with support for automatic imports:
- [Calibre](https://calibre-ebook.com/)
- [Calibre-Web](https://github.com/janeczku/calibre-web)
- [Calibre-Web-Automated](https://github.com/crocodilestick/Calibre-Web-Automated)
- [Booklore](https://github.com/booklore-app/booklore)
- [Grimmory](https://github.com/grimmory-tools/grimmory)
- [Audiobookshelf](https://github.com/advplyr/audiobookshelf)
## ✨ Features
- **One-Stop Interface** - A clean, modern UI to search, browse, and download from multiple sources in one place
- **Multiple Sources** - Popular archive websites, Torrent, Usenet, and IRC download support
- **One-Stop Interface** - A clean, modern UI to search, browse, and download from multiple configured sources in one place
- **Multiple Sources** - Configurable web, torrent, usenet, and IRC source support
- **Audiobook Support** - Full audiobook search and download with dedicated processing
- **Two Search Modes**:
- **Direct** - Search popular web sources
- **Universal** - Search metadata providers (Hardcover, Open Library) for richer book and audiobook discovery, with multi-source downloads
- **Flexible Search** - Search metadata providers (Hardcover, Open Library, Google Books) for rich book and audiobook discovery, or query configured sources directly
- **Multi-User & Requests** - Share your instance with others, let users browse and request books, and manage approvals with configurable notifications
- **Authentication** - Built-in login, OIDC single sign-on, proxy auth, and Calibre-Web database support
- **Real-Time Progress** - Unified download queue with live status updates across all sources
- **Cloudflare Bypass** - Built-in bypasser for reliable access to protected sources
- **Network Flexibility** - Configurable proxy support, DNS settings, and optional Cloudflare handling for protected sources
## 🖼️ Screenshots
@@ -60,7 +56,7 @@ Shelfmark is a self-hosted web interface for searching and downloading books and
3. Open `http://localhost:8084`
That's it! Configure settings through the web interface as needed.
Open the web interface, then configure the sources and settings you want to use.
### Volume Setup
@@ -68,27 +64,32 @@ That's it! Configure settings through the web interface as needed.
volumes:
- /your/config/path:/config # Config, database, and artwork cache directory
- /your/download/path:/books # Downloaded books
- /client/path:/client/path # Optional: For Torrent/Usenet downloads, match your client directory exactly.
- /client/path:/client/path # Optional: For Torrent/Usenet downloads, match your client directory exactly.
```
> **Tip**: Point the download volume to your CWA or Booklore ingest folder for automatic import.
> **Tip**: Point the download volume to your CWA or Grimmory ingest folder for automatic import.
> **Note**: CIFS shares require `nobrl` mount option to avoid database lock errors.
### Non-root container mode
- Start the container as `1000:1000` with Docker `user: "1000:1000"` or `docker run --user 1000:1000`.
- For Kubernetes, set `runAsUser: 1000`, `runAsGroup: 1000`, and `runAsNonRoot: true` together.
- `PUID`/`PGID` keep the default root startup flow.
- Mounted paths must already be writable by `1000:1000`.
- `USING_TOR=true` requires root startup.
## ⚙️ Configuration
### Search Modes
**Direct** (default)
- Works out of the box, no setup required
- Searches a huge library of books directly
- Returns downloadable releases immediately
**Direct**
- Queries configured sources directly
**Universal**
- Cleaner search results via metadata providers (Hardcover is recommended)
**Universal** (recommended)
- Search via metadata providers (Hardcover, Open Library, Google Books) for richer results
- Aggregates releases from multiple configured sources
- Full Audiobook support
- Requires manual setup (API keys, additional sources)
- Full audiobook support
### Environment Variables
@@ -99,20 +100,19 @@ Environment variables work for initial setup and Docker deployments. They serve
| `FLASK_PORT` | Web interface port | `8084` |
| `INGEST_DIR` | Book download directory | `/books` |
| `TZ` | Container timezone | `UTC` |
| `PUID` / `PGID` | Runtime user/group ID (also supports legacy `UID`/`GID`) | `1000` / `1000` |
| `SEARCH_MODE` | `direct` or `universal` | `direct` |
| `USING_TOR` | Enable Tor routing (requires `NET_ADMIN` capability) | `false` |
| `PUID` / `PGID` | Runtime user/group for the default root-startup flow (also supports legacy `UID`/`GID`) | `1000` / `1000` |
| `SEARCH_MODE` | `direct` or `universal` | `universal` |
| `USING_TOR` | Enable Tor routing (requires root startup) | `false` |
See the full [Environment Variables Reference](docs/environment-variables.md) for all available options.
Some of the additional options available in Settings:
- **Fast Download Key** - Use your paid account to skip Cloudflare challenges entirely and use faster, direct downloads
- **Prowlarr** - Configure indexers and download clients to download books and audiobooks
- **AudiobookBay** - Web scraping source for audiobook torrents (audiobooks only)
- **Additional audiobook sources** - Configure additional sources for audiobook discovery
- **IRC** - Add details for IRC book sources and download directly from the UI
- **Library Link** - Add a link to your Calibre-Web or Booklore instance in the UI header
- **Library Link** - Add a link to your Calibre-Web or Grimmory instance in the UI header
- **File processing** - Customiseable download paths, file renaming and directory creation with template-based renaming
- **Network Resilience** - Auto DNS rotation and mirror fallback when sources are unreachable. Custom proxy support (SOCK5 + HTTP/S), Tor routing.
- **Network Settings** - Custom proxy support (SOCKS5 + HTTP/S) and configurable DNS
- **Format & Language** - Filter downloads by preferred formats, languages and sorting order
- **Metadata Providers** - Configure API keys for Hardcover, Open Library, etc.
@@ -123,34 +123,34 @@ Some of the additional options available in Settings:
docker compose up -d
```
The full-featured image with built-in Cloudflare bypass.
The full-featured image with all network capabilities included.
#### Enable Tor Routing
Routes all traffic through Tor for enhanced privacy:
#### Tor Routing
Optional Tor support for network privacy:
```bash
curl -O https://raw.githubusercontent.com/calibrain/shelfmark/main/compose/docker-compose.tor.yml
docker compose -f docker-compose.tor.yml up -d
```
**Notes:**
- Requires root startup
- Requires `NET_ADMIN` and `NET_RAW` capabilities
- Timezone is auto-detected from Tor exit node
- Custom DNS/proxy settings are ignored when Tor is active
### Lite
A smaller image without the built-in Cloudflare bypasser. Ideal for:
A lighter image without the built-in browser automation. Ideal for:
- **External bypassers** - Already running FlareSolverr or ByParr for other services
- **Fast downloads** - Using fast download sources
- **Alternative sources only** - Exclusively using Prowlarr, AudiobookBay, IRC, or other sources
- **Audiobooks** - Using Shelfmark exclusively for audiobooks
- **External services** - Already running FlareSolverr or similar for other applications
- **Alternative sources** - Using Prowlarr, IRC, or other configured sources
- **Audiobooks** - Using Shelfmark primarily for audiobooks
```bash
curl -O https://raw.githubusercontent.com/calibrain/shelfmark/main/compose/docker-compose.lite.yml
docker compose -f docker-compose.lite.yml up -d
```
If you need Cloudflare bypass with the Lite image, configure an external resolver (FlareSolverr/ByParr) in Settings under the Cloudflare tab.
If you need browser-based access with the Lite image, configure an external resolver in Settings.
## 🔐 Authentication
@@ -221,11 +221,16 @@ Log level is configurable via Settings or `LOG_LEVEL` environment variable.
## Development
```bash
# Quality checks
make checks # Run ALL static analysis (frontend + Python)
make python-checks # Run Ruff, BasedPyright, and Vulture
make install-python-dev # Sync Python runtime + dev tools with uv
# Frontend development
make install # Install dependencies
make dev # Start Vite dev server (localhost:5173)
make build # Production build
make typecheck # TypeScript checks
make frontend-typecheck # TypeScript checks
# Backend (Docker)
make up # Start backend via docker-compose.dev.yml
-19
View File
@@ -1,19 +0,0 @@
flask
flask-cors
flask-socketio
python-socketio
requests[socks]
defusedxml
beautifulsoup4
tqdm
dnspython
gunicorn
gevent
gevent-websocket
psutil
emoji
rarfile
qbittorrent-api
transmission-rpc
authlib>=1.6.6,<1.7
apprise>=1.9.0
-4
View File
@@ -1,4 +0,0 @@
pyvirtualdisplay
pyautogui
seleniumbase==4.45.10
python-xlib
+246
View File
@@ -0,0 +1,246 @@
#!/usr/bin/env bash
set -euo pipefail
LATEST_IMAGE="${LATEST_IMAGE:-ghcr.io/calibrain/shelfmark:latest}"
LEGACY_IMAGE="${LEGACY_IMAGE:-ghcr.io/calibrain/shelfmark:v1.0.2}"
WAIT_SECONDS="${WAIT_SECONDS:-5}"
STARTUP_TIMEOUT_SECONDS="${STARTUP_TIMEOUT_SECONDS:-120}"
require_cmd() {
command -v "$1" >/dev/null 2>&1 || {
echo "Missing required command: $1" >&2
exit 1
}
}
cleanup() {
local name="$1"
docker rm -f "$name" >/dev/null 2>&1 || true
}
wait_for_startup() {
local name="$1"
local elapsed=0
while [ "$elapsed" -lt "$STARTUP_TIMEOUT_SECONDS" ]; do
if ! docker inspect "$name" >/dev/null 2>&1; then
echo "Container $name no longer exists" >&2
return 1
fi
if [ "$(docker inspect -f '{{.State.Status}}' "$name" 2>/dev/null)" != "running" ]; then
echo "Container $name exited before startup completed" >&2
docker logs --tail 120 "$name" 2>&1 || true
return 1
fi
if docker exec "$name" sh -lc "getent passwd 1000 >/dev/null 2>&1 && ps -eo comm,args | awk '\$1 == \"gunicorn\" && index(\$0, \"shelfmark.main:app\") { found=1 } END { exit(found ? 0 : 1) }'" >/dev/null 2>&1; then
return 0
fi
sleep 1
elapsed=$((elapsed + 1))
done
echo "Timed out waiting for $name to finish startup" >&2
docker logs --tail 120 "$name" 2>&1 || true
return 1
}
start_container() {
local name="$1"
local image="$2"
local pre_entrypoint_script="${3:-}"
cleanup "$name"
if [ -n "$pre_entrypoint_script" ]; then
docker run -d \
--name "$name" \
--entrypoint sh \
-e PUID=1000 \
-e PGID=1000 \
-e TZ=UTC \
"$image" \
-lc "$pre_entrypoint_script
exec /app/entrypoint.sh" >/dev/null
else
docker run -d \
--name "$name" \
-e PUID=1000 \
-e PGID=1000 \
-e TZ=UTC \
"$image" >/dev/null
sleep "$WAIT_SECONDS"
fi
wait_for_startup "$name"
}
run_probe() {
local name="$1"
local mode="${2:-default}"
docker exec -u 1000:1000 -e PROBE_MODE="$mode" "$name" sh -lc 'python3 - <<'"'"'PY'"'"'
import asyncio
import os
import shelfmark.bypass.internal_bypasser as ib
async def run_probe():
driver = None
probe_mode = os.environ.get("PROBE_MODE", "default")
if probe_mode == "proxy_auth" and hasattr(ib, "_get_proxy_string"):
ib._get_proxy_string = lambda _url: "user:pass@127.0.0.1:8888"
if hasattr(ib, "_create_cdp_browser"):
try:
driver = await ib._create_cdp_browser("https://example.com")
profile = getattr(getattr(driver, "config", None), "user_data_dir", "")
print(f"PROBE=OK mode={probe_mode} fn=_create_cdp_browser profile={profile}")
except Exception as e:
print(f"PROBE=ERR mode={probe_mode} fn=_create_cdp_browser type={type(e).__name__} msg={e}")
finally:
if driver and hasattr(ib, "_close_cdp_driver"):
await ib._close_cdp_driver(driver)
return
if hasattr(ib, "_create_driver"):
try:
driver = await ib._create_driver()
print(f"PROBE=OK mode={probe_mode} fn=_create_driver driver_type={type(driver).__name__}")
except Exception as e:
print(f"PROBE=ERR mode={probe_mode} fn=_create_driver type={type(e).__name__} msg={e}")
finally:
if driver and hasattr(ib, "_quit_driver"):
await ib._quit_driver(driver)
return
print(f"PROBE=ERR mode={probe_mode} fn=unknown type=RuntimeError msg=no supported startup function found")
asyncio.run(run_probe())
PY'
}
show_logs() {
local name="$1"
docker logs --tail 80 "$name" 2>&1 | tail -n 20
}
scenario_latest_baseline() {
local name="sb-lab-latest-baseline"
echo
echo "== latest baseline =="
start_container "$name" "$LATEST_IMAGE"
run_probe "$name"
cleanup "$name"
}
scenario_latest_drivers_readonly() {
local name="sb-lab-latest-drivers"
echo
echo "== latest drivers readonly =="
start_container "$name" "$LATEST_IMAGE" '
chown -R root:root /usr/local/lib/python3.10/site-packages/seleniumbase/drivers &&
chmod -R a-w /usr/local/lib/python3.10/site-packages/seleniumbase/drivers &&
ls -ld /usr/local/lib/python3.10/site-packages/seleniumbase/drivers
'
run_probe "$name"
cleanup "$name"
}
scenario_latest_proxy_auth_baseline() {
local name="sb-lab-latest-proxy-baseline"
echo
echo "== latest proxy auth baseline =="
start_container "$name" "$LATEST_IMAGE"
run_probe "$name" "proxy_auth"
cleanup "$name"
}
scenario_latest_downloads_readonly() {
local name="sb-lab-latest-downloads"
echo
echo "== latest downloaded_files readonly =="
start_container "$name" "$LATEST_IMAGE" '
mkdir -p /app/downloaded_files &&
touch /app/downloaded_files/pipfinding.lock /app/downloaded_files/proxy_dir.lock &&
chown -R root:root /app/downloaded_files &&
chmod -R a-w /app/downloaded_files &&
find /app/downloaded_files -maxdepth 2 -printf "%M %u:%g %p\n"
'
run_probe "$name"
show_logs "$name"
cleanup "$name"
}
scenario_latest_proxy_auth_downloads_readonly() {
local name="sb-lab-latest-proxy-downloads"
echo
echo "== latest proxy auth with readonly downloaded_files =="
start_container "$name" "$LATEST_IMAGE" '
mkdir -p /app/downloaded_files &&
touch /app/downloaded_files/pipfinding.lock /app/downloaded_files/proxy_dir.lock &&
chown 1000:1000 /app/downloaded_files/pipfinding.lock /app/downloaded_files/proxy_dir.lock &&
chmod 0666 /app/downloaded_files/pipfinding.lock /app/downloaded_files/proxy_dir.lock &&
chown root:root /app/downloaded_files &&
chmod 0555 /app/downloaded_files &&
ls -ld /app/downloaded_files &&
ls -la /app/downloaded_files
'
run_probe "$name" "proxy_auth"
show_logs "$name"
cleanup "$name"
}
scenario_latest_bind_mount_readonly() {
local name="sb-lab-latest-bind-ro"
local bind_dir
bind_dir="$(mktemp -d /tmp/sb-lab-bind.XXXXXX)"
echo
echo "== latest readonly bind mount for downloaded_files =="
chmod 0555 "$bind_dir"
cleanup "$name"
docker run -d \
--name "$name" \
-e PUID=1000 \
-e PGID=1000 \
-e TZ=UTC \
--mount "type=bind,src=${bind_dir},target=/app/downloaded_files,readonly" \
"$LATEST_IMAGE" >/dev/null
wait_for_startup "$name"
run_probe "$name"
show_logs "$name"
cleanup "$name"
rm -rf "$bind_dir"
}
scenario_legacy_drivers_readonly() {
local name="sb-lab-legacy-drivers"
echo
echo "== legacy drivers readonly =="
start_container "$name" "$LEGACY_IMAGE" '
chown -R root:root /usr/local/lib/python3.10/site-packages/seleniumbase/drivers &&
chmod -R a-w /usr/local/lib/python3.10/site-packages/seleniumbase/drivers &&
ls -ld /usr/local/lib/python3.10/site-packages/seleniumbase/drivers
'
run_probe "$name"
show_logs "$name"
cleanup "$name"
}
main() {
require_cmd docker
scenario_latest_baseline
scenario_latest_drivers_readonly
scenario_latest_proxy_auth_baseline
scenario_latest_downloads_readonly
scenario_latest_proxy_auth_downloads_readonly
scenario_latest_bind_mount_readonly
scenario_legacy_drivers_readonly
}
main "$@"
-92
View File
@@ -1,92 +0,0 @@
#!/usr/bin/env python3
"""Fix permissions on all configured directories.
This script is called by the entrypoint to ensure all user-configured
directories have correct ownership. It reads directory paths from:
- CONFIG_DIR environment variable
- Config files in CONFIG_DIR/plugins/
Outputs directory paths that need permission fixing (one per line).
The entrypoint handles the actual chown operations.
"""
import json
import os
import sys
from pathlib import Path
def get_directories_from_config() -> set[str]:
"""Extract all directory paths from config files."""
directories = set()
config_dir = Path(os.getenv("CONFIG_DIR", "/config"))
plugins_dir = config_dir / "plugins"
if not plugins_dir.exists():
return directories
# Keys that contain directory paths
directory_keys = {
# Main destinations
"DESTINATION",
"DESTINATION_AUDIOBOOK",
# Content type routing directories
"AA_CONTENT_TYPE_DIR_FICTION",
"AA_CONTENT_TYPE_DIR_NON_FICTION",
"AA_CONTENT_TYPE_DIR_UNKNOWN",
"AA_CONTENT_TYPE_DIR_MAGAZINE",
"AA_CONTENT_TYPE_DIR_COMIC",
"AA_CONTENT_TYPE_DIR_STANDARDS",
"AA_CONTENT_TYPE_DIR_MUSICAL_SCORE",
"AA_CONTENT_TYPE_DIR_OTHER",
# Legacy keys (in case of old configs)
"INGEST_DIR",
"INGEST_DIR_AUDIOBOOK",
"INGEST_DIR_BOOK_FICTION",
"INGEST_DIR_BOOK_NON_FICTION",
"INGEST_DIR_BOOK_UNKNOWN",
"INGEST_DIR_MAGAZINE",
"INGEST_DIR_COMIC_BOOK",
"INGEST_DIR_STANDARDS_DOCUMENT",
"INGEST_DIR_MUSICAL_SCORE",
"INGEST_DIR_OTHER",
"LIBRARY_PATH",
"LIBRARY_PATH_AUDIOBOOK",
}
# Read all JSON config files
for config_file in plugins_dir.glob("*.json"):
try:
with open(config_file, "r") as f:
config = json.load(f)
for key in directory_keys:
if key in config:
value = config[key]
if value and isinstance(value, str) and value.startswith("/"):
directories.add(value)
except (json.JSONDecodeError, OSError):
continue
return directories
def main():
"""Output all configured directories that exist."""
directories = get_directories_from_config()
# Filter to directories that actually exist
existing = []
for dir_path in directories:
path = Path(dir_path)
if path.exists() and path.is_dir():
existing.append(dir_path)
# Output one directory per line
for dir_path in sorted(existing):
print(dir_path)
if __name__ == "__main__":
main()
+39 -48
View File
@@ -17,16 +17,15 @@ The generated documentation includes:
import argparse
import sys
from datetime import datetime
from pathlib import Path
from typing import Any, Dict, List, Optional
from typing import Any
# Add project root to path
project_root = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(project_root))
def get_field_type_name(field) -> str:
def get_field_type_name(field: Any) -> str:
"""Get a human-readable type name for a field."""
from shelfmark.core.settings_registry import (
CheckboxField,
@@ -35,54 +34,55 @@ def get_field_type_name(field) -> str:
OrderableListField,
PasswordField,
SelectField,
TagListField,
TextField,
)
if isinstance(field, CheckboxField):
return "boolean"
elif isinstance(field, NumberField):
if isinstance(field, NumberField):
return "number"
elif isinstance(field, SelectField):
if isinstance(field, SelectField):
return "string (choice)"
elif isinstance(field, MultiSelectField):
if isinstance(field, MultiSelectField):
return "string (comma-separated)"
elif isinstance(field, OrderableListField):
if isinstance(field, TagListField):
return "string (comma-separated)"
if isinstance(field, OrderableListField):
return "JSON array"
elif isinstance(field, PasswordField):
if isinstance(field, PasswordField):
return "string (secret)"
elif isinstance(field, TextField):
return "string"
else:
if isinstance(field, TextField):
return "string"
return "string"
def format_default_value(field) -> str:
def format_default_value(field: Any) -> str:
"""Format the default value for display."""
default = field.default
if default is None:
return "_none_"
elif isinstance(default, bool):
if isinstance(default, bool):
return f"`{str(default).lower()}`"
elif isinstance(default, (int, float)):
if isinstance(default, (int, float)):
return f"`{default}`"
elif isinstance(default, str):
if isinstance(default, str):
if default == "":
return "_empty string_"
return f"`{default}`"
elif isinstance(default, list):
if isinstance(default, list):
if not default:
return "_empty list_"
# For simple lists, show comma-separated values
if all(isinstance(item, str) for item in default):
return f"`{','.join(default)}`"
# For complex lists (e.g., OrderableListField defaults), summarize
return f"_see UI for defaults_"
else:
return f"`{default}`"
return "_see UI for defaults_"
return f"`{default}`"
def get_select_options(field) -> Optional[List[str]]:
def get_select_options(field: Any) -> list[str] | None:
"""Get the available options for a SelectField.
Returns options formatted as 'value (label)' or just 'value' if they match,
@@ -119,7 +119,7 @@ def get_select_options(field) -> Optional[List[str]]:
return result
def _generate_bootstrap_env_docs() -> List[str]:
def _generate_bootstrap_env_docs() -> list[str]:
"""Generate documentation for bootstrap environment variables from env.py."""
# These are environment variables defined in env.py that are used before
# the settings registry is available
@@ -195,8 +195,10 @@ def _generate_bootstrap_env_docs() -> List[str]:
"|----------|-------------|------|---------|",
]
for var in bootstrap_vars:
lines.append(f"| `{var['name']}` | {var['description']} | {var['type']} | `{var['default']}` |")
lines.extend(
f"| `{var['name']}` | {var['description']} | {var['type']} | `{var['default']}` |"
for var in bootstrap_vars
)
lines.append("")
lines.append("<details>")
@@ -221,17 +223,14 @@ def _generate_bootstrap_env_docs() -> List[str]:
def generate_env_docs() -> str:
"""Generate markdown documentation for all environment variables."""
# Import settings modules to ensure all settings are registered
import shelfmark.config.settings # noqa: F401
import shelfmark.config.security # noqa: F401
import shelfmark.release_sources.irc.settings # noqa: F401
import shelfmark.config.security
import shelfmark.config.settings
import shelfmark.metadata_providers.googlebooks
import shelfmark.metadata_providers.hardcover
import shelfmark.metadata_providers.openlibrary
import shelfmark.release_sources.irc.settings
import shelfmark.release_sources.prowlarr.settings # noqa: F401
import shelfmark.metadata_providers.hardcover # noqa: F401
import shelfmark.metadata_providers.openlibrary # noqa: F401
import shelfmark.metadata_providers.googlebooks # noqa: F401
from shelfmark.core.settings_registry import (
ActionButton,
HeadingField,
get_all_groups,
get_all_settings_tabs,
)
@@ -240,7 +239,7 @@ def generate_env_docs() -> str:
groups = {g.name: g for g in get_all_groups()}
# Organize tabs by group
grouped_tabs: Dict[Optional[str], List] = {None: []}
grouped_tabs: dict[str | None, list] = {None: []}
for group_name in groups:
grouped_tabs[group_name] = []
@@ -309,33 +308,24 @@ def generate_env_docs() -> str:
return "\n".join(lines)
def _generate_tab_docs(tab, group_prefix: Optional[str] = None) -> List[str]:
def _generate_tab_docs(tab: Any, group_prefix: str | None = None) -> list[str]:
"""Generate documentation for a single settings tab."""
from shelfmark.core.settings_registry import ActionButton, CustomComponentField, HeadingField
from shelfmark.core.settings_registry import iter_value_fields
lines = []
# Section header
if group_prefix:
lines.append(f"### {group_prefix}: {tab.display_name}")
anchor_id = f"{group_prefix}-{tab.display_name}".lower().replace(" ", "-")
else:
lines.append(f"## {tab.display_name}")
lines.append("")
# Collect env-supported fields
env_fields = []
for field in tab.fields:
# Skip non-value fields
if isinstance(field, (ActionButton, CustomComponentField, HeadingField)):
continue
# Skip fields that don't support ENV vars
if not getattr(field, "env_supported", True):
continue
env_fields.append(field)
env_fields = [
field for field in iter_value_fields(tab) if getattr(field, "env_supported", True)
]
if not env_fields:
lines.append("_No environment variables for this section._")
@@ -391,6 +381,7 @@ def _generate_tab_docs(tab, group_prefix: Optional[str] = None) -> List[str]:
# Show constraints for NumberField
from shelfmark.core.settings_registry import NumberField
if isinstance(field, NumberField):
constraints = []
if field.min_value is not None:
@@ -408,7 +399,7 @@ def _generate_tab_docs(tab, group_prefix: Optional[str] = None) -> List[str]:
return lines
def main():
def main() -> None:
parser = argparse.ArgumentParser(
description="Generate markdown documentation for environment variables"
)
+38 -27
View File
@@ -9,7 +9,7 @@ Usage:
2. Wait for containers to initialize (first run takes ~30s)
3. Run this script to verify clients are accessible:
python scripts/test_clients.py
uv run python scripts/test_clients.py
4. Access cwabd at http://localhost:8084
- Go to Settings > Prowlarr > Download Clients
@@ -26,7 +26,7 @@ Web UIs:
- rTorrent: http://localhost:8000 (web ui http://localhost:8089 via ruTorrent)
Prerequisites (for running this script locally):
pip install requests transmission-rpc qbittorrent-api
uv sync --locked
First-Time Setup:
qBittorrent:
@@ -51,7 +51,8 @@ First-Time Setup:
import sys
import time
from xmlrpc import client
from pathlib import Path
from typing import Any
# Test configuration - matches docker-compose.test-clients.yml
CONFIG = {
@@ -89,7 +90,7 @@ CONFIG = {
TEST_MAGNET = "magnet:?xt=urn:btih:3b245504cf5f11bbdbe1201cea6a6bf45aee1bc0&dn=ubuntu-22.04.3-live-server-amd64.iso"
def test_nzbget():
def test_nzbget() -> bool:
"""Test NZBGet connection."""
import requests
@@ -138,7 +139,7 @@ def test_nzbget():
return False
def test_sabnzbd():
def test_sabnzbd() -> bool:
"""Test SABnzbd connection."""
import requests
@@ -152,10 +153,9 @@ def test_sabnzbd():
# Try to get API key from config if not set
if not api_key:
try:
import os
ini_path = ".local/test-clients/sabnzbd/config/sabnzbd.ini"
if os.path.exists(ini_path):
with open(ini_path) as f:
ini_path = Path(".local/test-clients/sabnzbd/config/sabnzbd.ini")
if ini_path.exists():
with ini_path.open() as f:
for line in f:
if line.startswith("api_key"):
api_key = line.split("=")[1].strip()
@@ -204,7 +204,7 @@ def test_sabnzbd():
return False
def test_qbittorrent():
def test_qbittorrent() -> bool:
"""Test qBittorrent connection."""
print("\n" + "=" * 50)
print("Testing qBittorrent")
@@ -219,6 +219,7 @@ def test_qbittorrent():
# Parse URL for host/port
from urllib.parse import urlparse
parsed = urlparse(url)
client = qbittorrentapi.Client(
@@ -260,7 +261,7 @@ def test_qbittorrent():
except ImportError:
print(" ERROR: qbittorrent-api not installed")
print(" Run: pip install qbittorrent-api")
print(" Run: uv sync --locked")
return False
except Exception as e:
print(f" ERROR: {e}")
@@ -271,16 +272,17 @@ def test_qbittorrent():
return False
def test_transmission():
def test_transmission() -> bool:
"""Test Transmission connection."""
print("\n" + "=" * 50)
print("Testing Transmission")
print("=" * 50)
try:
from transmission_rpc import Client
from urllib.parse import urlparse
from transmission_rpc import Client
url = CONFIG["transmission"]["url"]
parsed = urlparse(url)
@@ -317,14 +319,14 @@ def test_transmission():
except ImportError:
print(" ERROR: transmission-rpc not installed")
print(" Run: pip install transmission-rpc")
print(" Run: uv sync --locked")
return False
except Exception as e:
print(f" ERROR: {e}")
return False
def test_deluge():
def test_deluge() -> bool:
"""Test Deluge Web UI (JSON-RPC) connection."""
import requests
@@ -336,7 +338,7 @@ def test_deluge():
password = CONFIG["deluge"]["password"]
rpc_url = f"{base_url}/json"
def rpc_call(session: requests.Session, rpc_id: int, method: str, *params):
def rpc_call(session: requests.Session, rpc_id: int, method: str, *params: Any) -> Any:
payload = {"id": rpc_id, "method": method, "params": list(params)}
resp = session.post(rpc_url, json=payload, timeout=10)
resp.raise_for_status()
@@ -366,7 +368,11 @@ def test_deluge():
host_id = hosts[0][0]
for entry in hosts:
if isinstance(entry, list) and len(entry) >= 2 and entry[1] in {"127.0.0.1", "localhost"}:
if (
isinstance(entry, list)
and len(entry) >= 2
and entry[1] in {"127.0.0.1", "localhost"}
):
host_id = entry[0]
break
@@ -386,13 +392,18 @@ def test_deluge():
# Test adding a torrent (then remove it)
print(" Testing add/remove torrent...")
torrent_id = rpc_call(session, 8, "core.add_torrent_magnet", TEST_MAGNET, {"add_paused": True})
torrent_id = rpc_call(
session, 8, "core.add_torrent_magnet", TEST_MAGNET, {"add_paused": True}
)
if torrent_id:
torrent_id = str(torrent_id)
print(f" Added test torrent: {torrent_id[:20]}...")
status = rpc_call(session, 9, "core.get_torrent_status", torrent_id, ["state", "progress"]) or {}
status = (
rpc_call(session, 9, "core.get_torrent_status", torrent_id, ["state", "progress"])
or {}
)
state = status.get("state", "unknown") if isinstance(status, dict) else "unknown"
progress = status.get("progress", 0) if isinstance(status, dict) else 0
print(f" Status: {state} ({progress:.1f}%)")
@@ -418,7 +429,8 @@ def test_deluge():
print(" Check Deluge Web UI password (default: deluge)")
return False
def test_rtorrent():
def test_rtorrent() -> bool:
"""Test rTorrent connection."""
print("\n" + "=" * 50)
print("Testing rTorrent")
@@ -457,19 +469,18 @@ def test_rtorrent():
# rtorrent is weird in that it doesn't return the torrent ID/hash on add
client.load.start("", TEST_MAGNET, ";".join(commands))
# but we know that it is 3b245504cf5f11bbdbe1201cea6a6bf45aee1bc0 from the magnet link
torrent_id = "3B245504CF5F11BBDBE1201CEA6A6BF45AEE1BC0" # rtorrent uses uppercase hashes
torrent_id = "3B245504CF5F11BBDBE1201CEA6A6BF45AEE1BC0" # rtorrent uses uppercase hashes
print(f" Added test torrent: {torrent_id}")
torrents = client.download_list()
print(f" Active torrents: {len(torrents)}")
print(f" Active torrents: {len(torrents)}")
torrent_list = client.d.multicall.filtered(
"",
"default",
f"equal={{d.hash=,cat={torrent_id}}}"
"d.hash=",
f"equal={{d.hash=,cat={torrent_id}}}d.hash=",
"d.state=",
"d.completed_bytes=",
"d.size_bytes=",
@@ -483,7 +494,7 @@ def test_rtorrent():
if not torrent:
print(" ERROR: Could not find added torrent in list")
return False
# let's test the base path call
details = client.d.multicall.filtered(
"",
@@ -511,7 +522,7 @@ def test_rtorrent():
return False
def main():
def main() -> int:
print("Download Client Test Suite")
print("=" * 50)
print("Make sure containers are running:")
+17 -2
View File
@@ -1,8 +1,23 @@
"""Package entry point for `python -m shelfmark`."""
from shelfmark.main import app, socketio
from shelfmark.config.env import FLASK_HOST, FLASK_PORT
from shelfmark.core.config import config
from shelfmark.main import app, socketio
def _resolve_debug_flag(value: object) -> bool:
"""Normalize DEBUG config values for Flask-SocketIO startup."""
if isinstance(value, bool):
return value
if isinstance(value, str):
return value.strip().lower() in {"1", "true", "yes", "on"}
return bool(value)
if __name__ == "__main__":
socketio.run(app, host=FLASK_HOST, port=FLASK_PORT, debug=config.get("DEBUG", False))
socketio.run(
app,
host=FLASK_HOST,
port=FLASK_PORT,
debug=_resolve_debug_flag(config.get("DEBUG", False)),
)
+95 -128
View File
@@ -1,124 +1,81 @@
"""WebSocket manager for real-time status updates."""
from __future__ import annotations
import logging
import threading
from typing import Optional, Dict, Any, Callable, List
from typing import TYPE_CHECKING, Any
from flask_socketio import SocketIO, join_room, leave_room
if TYPE_CHECKING:
from collections.abc import Callable
from flask import Flask
logger = logging.getLogger(__name__)
class WebSocketManager:
"""Manages WebSocket connections and broadcasts."""
def __init__(self):
self.socketio: Optional[SocketIO] = None
def __init__(self) -> None:
"""Initialize in-memory connection and room tracking."""
self.socketio: SocketIO | None = None
self._enabled = False
self._connection_count = 0
self._connection_lock = threading.Lock()
self._on_first_connect_callbacks: List[Callable[[], None]] = []
self._on_all_disconnect_callbacks: List[Callable[[], None]] = []
self._needs_rewarm = False # Flag to trigger warmup callbacks on next connect
self._user_rooms: Dict[str, int] = {} # room_name -> ref count
self._sid_rooms: Dict[str, str] = {} # sid -> room_name
self._user_rooms: dict[str, int] = {} # room_name -> ref count
self._sid_rooms: dict[str, str] = {} # sid -> room_name
self._rooms_lock = threading.Lock()
self._queue_status_fn: Optional[Callable] = None # Reference to queue_status()
self._queue_status_fn: Callable | None = None # Reference to queue_status()
def init_app(self, app, socketio: SocketIO):
def init_app(self, app: Flask, socketio: SocketIO) -> None:
"""Initialize the WebSocket manager with Flask-SocketIO instance."""
self.socketio = socketio
self._enabled = True
logger.info("WebSocket manager initialized")
def register_on_first_connect(self, callback: Callable[[], None]):
"""Register a callback for when the first client connects."""
self._on_first_connect_callbacks.append(callback)
logger.debug(f"Registered on_first_connect callback: {callback.__name__}")
def register_on_all_disconnect(self, callback: Callable[[], None]):
"""Register a callback for when all clients disconnect."""
self._on_all_disconnect_callbacks.append(callback)
logger.debug(f"Registered on_all_disconnect callback: {callback.__name__}")
def request_warmup_on_next_connect(self):
"""Request warmup callbacks on the next client connect (e.g., after idle shutdown)."""
with self._connection_lock:
self._needs_rewarm = True
logger.debug("Warmup requested for next client connect")
def client_connected(self):
def client_connected(self) -> None:
"""Track a new client connection. Call this from the connect event handler."""
with self._connection_lock:
was_zero = self._connection_count == 0
needs_rewarm = self._needs_rewarm
self._connection_count += 1
current_count = self._connection_count
# Clear rewarm flag if we're going to trigger warmup
if was_zero or needs_rewarm:
self._needs_rewarm = False
logger.debug(f"Client connected. Active connections: {current_count}")
logger.debug("Client connected. Active connections: %s", current_count)
# Trigger warmup callbacks if this is the first connection OR if rewarm was requested
# (rewarm is requested when bypasser shuts down due to idle while clients are connected)
if was_zero or needs_rewarm:
reason = "First client connected" if was_zero else "Rewarm requested after idle shutdown"
logger.info(f"{reason}, triggering warmup callbacks...")
for callback in self._on_first_connect_callbacks:
try:
# Run callbacks in a separate thread to not block the connection
thread = threading.Thread(target=callback, daemon=True)
thread.start()
except Exception as e:
logger.error(f"Error in on_first_connect callback {callback.__name__}: {e}")
def client_disconnected(self):
def client_disconnected(self) -> None:
"""Track a client disconnection. Call this from the disconnect event handler."""
with self._connection_lock:
self._connection_count = max(0, self._connection_count - 1)
current_count = self._connection_count
is_now_zero = current_count == 0
logger.debug(f"Client disconnected. Active connections: {current_count}")
# If all clients have disconnected, trigger cleanup callbacks
if is_now_zero:
logger.info("All clients disconnected, triggering disconnect callbacks...")
for callback in self._on_all_disconnect_callbacks:
try:
callback()
except Exception as e:
logger.error(f"Error in on_all_disconnect callback {callback.__name__}: {e}")
def get_connection_count(self) -> int:
"""Get the current number of active WebSocket connections."""
with self._connection_lock:
return self._connection_count
def has_active_connections(self) -> bool:
"""Check if there are any active WebSocket connections."""
return self.get_connection_count() > 0
logger.debug("Client disconnected. Active connections: %s", current_count)
def is_enabled(self) -> bool:
"""Check if WebSocket is enabled and ready."""
return self._enabled and self.socketio is not None
def set_queue_status_fn(self, fn: Callable):
def _get_socketio(self) -> SocketIO | None:
if not self._enabled:
return None
return self.socketio
def set_queue_status_fn(self, fn: Callable) -> None:
"""Set the queue_status function reference for per-room filtering."""
self._queue_status_fn = fn
def _increment_user_room_locked(self, room: str):
def _increment_user_room_locked(self, room: str) -> None:
self._user_rooms[room] = self._user_rooms.get(room, 0) + 1
def _decrement_user_room_locked(self, room: str):
def _decrement_user_room_locked(self, room: str) -> None:
count = self._user_rooms.get(room, 1) - 1
if count <= 0:
self._user_rooms.pop(room, None)
else:
self._user_rooms[room] = count
def _set_sid_room_locked(self, sid: str, room: Optional[str]):
def _set_sid_room_locked(self, sid: str, room: str | None) -> None:
current_room = self._sid_rooms.get(sid)
if current_room == room:
return
@@ -135,9 +92,14 @@ class WebSocketManager:
if room.startswith("user_"):
self._increment_user_room_locked(room)
def sync_user_room(self, sid: str, is_admin: bool, db_user_id: Optional[int] = None):
def sync_user_room(
self,
sid: str,
is_admin: bool,
db_user_id: int | None = None,
) -> None:
"""Ensure a SID is in exactly one room matching the current session scope."""
room: Optional[str] = None
room: str | None = None
if is_admin:
room = "admins"
elif db_user_id is not None:
@@ -146,24 +108,36 @@ class WebSocketManager:
with self._rooms_lock:
self._set_sid_room_locked(sid, room)
def join_user_room(self, sid: str, is_admin: bool, db_user_id: Optional[int] = None):
def join_user_room(
self,
sid: str,
is_admin: bool,
db_user_id: int | None = None,
) -> None:
"""Join the appropriate room based on user role."""
self.sync_user_room(sid, is_admin, db_user_id)
self.sync_user_room(sid, is_admin=is_admin, db_user_id=db_user_id)
def leave_user_room(self, sid: str, is_admin: bool = False, db_user_id: Optional[int] = None):
def leave_user_room(
self,
sid: str,
*,
is_admin: bool = False,
db_user_id: int | None = None,
) -> None:
"""Leave whichever room the SID currently belongs to."""
del is_admin, db_user_id # Backward-compatible signature; routing is SID-based.
with self._rooms_lock:
self._set_sid_room_locked(sid, None)
def broadcast_status_update(self, status_data: Dict[str, Any]):
def broadcast_status_update(self, status_data: dict[str, Any]) -> None:
"""Broadcast status update to all connected clients, filtered by user room."""
if not self.is_enabled():
socketio = self._get_socketio()
if socketio is None:
return
try:
# Admins (and no-auth users) get full status
self.socketio.emit('status_update', status_data, to="admins")
socketio.emit("status_update", status_data, to="admins")
# Each user room gets filtered status
with self._rooms_lock:
@@ -171,56 +145,48 @@ class WebSocketManager:
if active_rooms and self._queue_status_fn:
for room in active_rooms:
try:
# Extract user_id from room name "user_123"
uid = int(room.split("_", 1)[1])
filtered = self._queue_status_fn(user_id=uid)
self.socketio.emit('status_update', filtered, to=room)
except Exception as e:
logger.error(f"Failed to send status update for room {room}: {e}")
self._broadcast_status_update_to_room(room)
logger.debug("Broadcasted status update to all rooms")
except Exception as e:
logger.error(f"Error broadcasting status update: {e}")
except Exception:
logger.exception("Error broadcasting status update")
def broadcast_download_progress(self, book_id: str, progress: float, status: str, user_id: Optional[int] = None):
"""Broadcast download progress update for a specific book."""
if not self.is_enabled():
def _broadcast_status_update_to_room(self, room: str) -> None:
"""Broadcast status update to one user room."""
socketio = self._get_socketio()
if socketio is None:
return
try:
data = {
'book_id': book_id,
'progress': progress,
'status': status
}
# Extract user_id from room name "user_123"
uid = int(room.split("_", 1)[1])
filtered = self._queue_status_fn(user_id=uid) if self._queue_status_fn else None
if filtered is not None:
socketio.emit("status_update", filtered, to=room)
except Exception:
logger.exception("Failed to send status update for room %s", room)
def broadcast_download_progress(
self, book_id: str, progress: float, status: str, user_id: int | None = None
) -> None:
"""Broadcast download progress update for a specific book."""
socketio = self._get_socketio()
if socketio is None:
return
try:
data = {"book_id": book_id, "progress": progress, "status": status}
# Admins always see all progress
self.socketio.emit('download_progress', data, to="admins")
socketio.emit("download_progress", data, to="admins")
# If task belongs to a specific user, send to their room too
if user_id is not None:
room = f"user_{user_id}"
with self._rooms_lock:
if room in self._user_rooms:
self.socketio.emit('download_progress', data, to=room)
logger.debug(f"Broadcasted progress for book {book_id}: {progress}%")
except Exception as e:
logger.error(f"Error broadcasting download progress: {e}")
def broadcast_notification(self, message: str, notification_type: str = 'info'):
"""Broadcast a notification message to all clients."""
if not self.is_enabled():
return
try:
data = {
'message': message,
'type': notification_type
}
# When calling socketio.emit() outside event handlers, it broadcasts by default
self.socketio.emit('notification', data)
logger.debug(f"Broadcasted notification: {message}")
except Exception as e:
logger.error(f"Error broadcasting notification: {e}")
socketio.emit("download_progress", data, to=room)
logger.debug("Broadcasted progress for book %s: %s%%", book_id, progress)
except Exception:
logger.exception("Error broadcasting download progress")
def broadcast_search_status(
self,
@@ -228,23 +194,24 @@ class WebSocketManager:
provider: str,
book_id: str,
message: str,
phase: str = 'searching'
):
phase: str = "searching",
) -> None:
"""Broadcast search status update for a release source search."""
if not self.is_enabled():
socketio = self._get_socketio()
if socketio is None:
return
try:
data = {
'source': source,
'provider': provider,
'book_id': book_id,
'message': message,
'phase': phase,
"source": source,
"provider": provider,
"book_id": book_id,
"message": message,
"phase": phase,
}
self.socketio.emit('search_status', data)
except Exception as e:
logger.error(f"Error broadcasting search status: {e}")
socketio.emit("search_status", data)
except Exception:
logger.exception("Error broadcasting search status")
# Global WebSocket manager instance
+1 -1
View File
@@ -1,5 +1,5 @@
"""Cloudflare bypass utilities."""
class BypassCancelledException(Exception):
class BypassCancelledError(Exception):
"""Raised when a bypass operation is cancelled."""
+75 -32
View File
@@ -2,21 +2,23 @@
import random
import time
from threading import Event
from typing import TYPE_CHECKING, Optional
from typing import TYPE_CHECKING
import requests
from shelfmark.bypass import BypassCancelledException
from shelfmark.bypass import BypassCancelledError
from shelfmark.core.config import config
from shelfmark.core.logger import setup_logger
from shelfmark.core.utils import normalize_http_url
from shelfmark.download.network import get_ssl_verify
if TYPE_CHECKING:
from threading import Event
from shelfmark.download import network
logger = setup_logger(__name__)
_RNG = random.SystemRandom()
# Timeout constants (seconds)
CONNECT_TIMEOUT = 10
@@ -29,15 +31,36 @@ BACKOFF_BASE = 1.0
BACKOFF_CAP = 10.0
def _fetch_via_bypasser(target_url: str) -> Optional[str]:
def _coerce_config_str(value: object, default: str) -> str:
"""Return a string config value or a safe default."""
if isinstance(value, str):
return value
return default
def _coerce_timeout_ms(value: object, default: int) -> int:
"""Return a positive timeout in milliseconds or the default."""
if isinstance(value, bool):
return default
if isinstance(value, int) and value > 0:
return value
return default
def _fetch_via_bypasser(target_url: str) -> str | None:
"""Make a single request to the external bypasser service. Returns HTML or None."""
raw_bypasser_url = config.get("EXT_BYPASSER_URL", "http://flaresolverr:8191")
bypasser_path = config.get("EXT_BYPASSER_PATH", "/v1")
bypasser_timeout = config.get("EXT_BYPASSER_TIMEOUT", 60000)
raw_bypasser_url = _coerce_config_str(
config.get("EXT_BYPASSER_URL", "http://flaresolverr:8191"),
"http://flaresolverr:8191",
)
bypasser_path = _coerce_config_str(config.get("EXT_BYPASSER_PATH", "/v1"), "/v1")
bypasser_timeout = _coerce_timeout_ms(config.get("EXT_BYPASSER_TIMEOUT", 60000), 60000)
bypasser_url = normalize_http_url(raw_bypasser_url)
if not bypasser_url or not bypasser_path:
logger.error("External bypasser not configured. Check EXT_BYPASSER_URL and EXT_BYPASSER_PATH.")
logger.error(
"External bypasser not configured. Check EXT_BYPASSER_URL and EXT_BYPASSER_PATH."
)
return None
read_timeout = min((bypasser_timeout / 1000) + READ_TIMEOUT_BUFFER, MAX_READ_TIMEOUT)
@@ -46,48 +69,63 @@ def _fetch_via_bypasser(target_url: str) -> Optional[str]:
response = requests.post(
f"{bypasser_url}{bypasser_path}",
headers={"Content-Type": "application/json"},
json={"cmd": "request.get", "url": target_url, "maxTimeout": bypasser_timeout},
json={
"cmd": "request.get",
"url": target_url,
"maxTimeout": bypasser_timeout,
},
timeout=(CONNECT_TIMEOUT, read_timeout),
verify=get_ssl_verify(bypasser_url),
)
response.raise_for_status()
result = response.json()
status = result.get('status', 'unknown')
message = result.get('message', '')
logger.debug(f"External bypasser response for '{target_url}': {status} - {message}")
status = result.get("status", "unknown")
message = result.get("message", "")
logger.debug("External bypasser response for '%s': %s - %s", target_url, status, message)
if status != 'ok':
logger.warning(f"External bypasser failed for '{target_url}': {status} - {message}")
if status != "ok":
logger.warning(
"External bypasser failed for '%s': %s - %s",
target_url,
status,
message,
)
return None
solution = result.get('solution')
html = solution.get('response', '') if solution else ''
solution = result.get("solution")
html = solution.get("response", "") if solution else ""
if not html:
logger.warning(f"External bypasser returned empty response for '{target_url}'")
logger.warning("External bypasser returned empty response for '%s'", target_url)
return None
return html
except requests.exceptions.Timeout:
logger.warning(f"External bypasser timed out for '{target_url}' (connect: {CONNECT_TIMEOUT}s, read: {read_timeout:.0f}s)")
logger.warning(
"External bypasser timed out for '%s' (connect: %ss, read: %.0fs)",
target_url,
CONNECT_TIMEOUT,
read_timeout,
)
except requests.exceptions.RequestException as e:
logger.warning(f"External bypasser request failed for '{target_url}': {e}")
logger.warning("External bypasser request failed for '%s': %s", target_url, e)
except (KeyError, TypeError, ValueError) as e:
logger.warning(f"External bypasser returned malformed response for '{target_url}': {e}")
logger.warning("External bypasser returned malformed response for '%s': %s", target_url, e)
else:
return html
return None
def _check_cancelled(cancel_flag: Optional[Event], context: str) -> None:
def _check_cancelled(cancel_flag: Event | None, context: str) -> None:
"""Check if operation was cancelled and raise exception if so."""
if cancel_flag and cancel_flag.is_set():
logger.info(f"External bypasser cancelled {context}")
raise BypassCancelledException("Bypass cancelled")
logger.info("External bypasser cancelled %s", context)
msg = "Bypass cancelled"
raise BypassCancelledError(msg)
def _sleep_with_cancellation(seconds: float, cancel_flag: Optional[Event]) -> None:
def _sleep_with_cancellation(seconds: float, cancel_flag: Event | None) -> None:
"""Sleep for the specified duration, checking for cancellation each second."""
for _ in range(int(seconds)):
_check_cancelled(cancel_flag, "during backoff")
@@ -99,9 +137,9 @@ def _sleep_with_cancellation(seconds: float, cancel_flag: Optional[Event]) -> No
def get_bypassed_page(
url: str,
selector: Optional["network.AAMirrorSelector"] = None,
cancel_flag: Optional[Event] = None
) -> Optional[str]:
selector: network.AAMirrorSelector | None = None,
cancel_flag: Event | None = None,
) -> str | None:
"""Fetch HTML via external bypasser with retries and mirror rotation."""
from shelfmark.download import network as network_module
@@ -118,13 +156,18 @@ def get_bypassed_page(
if attempt == MAX_RETRY:
break
delay = min(BACKOFF_CAP, BACKOFF_BASE * (2 ** (attempt - 1))) + random.random()
logger.info(f"External bypasser attempt {attempt}/{MAX_RETRY} failed, retrying in {delay:.1f}s")
delay = min(BACKOFF_CAP, BACKOFF_BASE * (2 ** (attempt - 1))) + _RNG.random()
logger.info(
"External bypasser attempt %s/%s failed, retrying in %.1fs",
attempt,
MAX_RETRY,
delay,
)
_sleep_with_cancellation(delay, cancel_flag)
new_base, action = sel.next_mirror_or_rotate_dns()
if action in ("mirror", "dns") and new_base:
logger.info(f"Rotated {action} for retry")
logger.info("Rotated %s for retry", action)
return None
+29 -16
View File
@@ -1,52 +1,65 @@
"""Browser fingerprint profile management for bypass stealth."""
import random
from typing import Optional
from shelfmark.core.logger import setup_logger
logger = setup_logger(__name__)
COMMON_RESOLUTIONS = [
(1920, 1080, 0.35),
(1366, 768, 0.18),
(1536, 864, 0.10),
(1440, 900, 0.08),
(1280, 720, 0.07),
(1600, 900, 0.06),
(1280, 800, 0.05),
(2560, 1440, 0.04),
(1680, 1050, 0.04),
(1920, 1200, 0.03),
(1920, 1080, 0.35),
(1366, 768, 0.18),
(1536, 864, 0.10),
(1440, 900, 0.08),
(1280, 720, 0.07),
(1600, 900, 0.06),
(1280, 800, 0.05),
(2560, 1440, 0.04),
(1680, 1050, 0.04),
(1920, 1200, 0.03),
]
# Current screen size (module-level singleton)
_current_screen_size: Optional[tuple[int, int]] = None
_current_screen_size: tuple[int, int] | None = None
_RNG = random.SystemRandom()
def get_screen_size() -> tuple[int, int]:
"""Return the current synthetic screen size, generating one if needed."""
global _current_screen_size
if _current_screen_size is None:
_current_screen_size = _generate_screen_size()
logger.debug(f"Generated initial screen size: {_current_screen_size[0]}x{_current_screen_size[1]}")
logger.debug(
"Generated initial screen size: %sx%s",
_current_screen_size[0],
_current_screen_size[1],
)
return _current_screen_size
def rotate_screen_size() -> tuple[int, int]:
"""Rotate to a new synthetic screen size and return it."""
global _current_screen_size
old_size = _current_screen_size
_current_screen_size = _generate_screen_size()
width, height = _current_screen_size
if old_size:
logger.info(f"Rotated screen size: {old_size[0]}x{old_size[1]} -> {width}x{height}")
logger.info(
"Rotated screen size: %sx%s -> %sx%s",
old_size[0],
old_size[1],
width,
height,
)
else:
logger.info(f"Generated screen size: {width}x{height}")
logger.info("Generated screen size: %sx%s", width, height)
return _current_screen_size
def clear_screen_size() -> None:
"""Clear the cached synthetic screen size."""
global _current_screen_size
_current_screen_size = None
@@ -54,4 +67,4 @@ def clear_screen_size() -> None:
def _generate_screen_size() -> tuple[int, int]:
resolutions = [(w, h) for w, h, _ in COMMON_RESOLUTIONS]
weights = [weight for _, _, weight in COMMON_RESOLUTIONS]
return random.choices(resolutions, weights=weights)[0]
return _RNG.choices(resolutions, weights=weights)[0]
File diff suppressed because it is too large Load Diff
+26 -20
View File
@@ -1,3 +1,5 @@
"""Helpers for Booklore settings validation, option loading, and connection tests."""
from __future__ import annotations
from typing import Any
@@ -121,7 +123,7 @@ def get_booklore_library_options() -> list[dict[str, Any]]:
base_url = str(config.get("BOOKLORE_HOST", "") or "").strip().rstrip("/")
username = str(config.get("BOOKLORE_USERNAME", "") or "").strip()
password = config.get("BOOKLORE_PASSWORD", "") or ""
password = str(config.get("BOOKLORE_PASSWORD", "") or "")
if not base_url or not username or not password:
return []
@@ -130,12 +132,13 @@ def get_booklore_library_options() -> list[dict[str, Any]]:
try:
library_options, _ = _get_booklore_cached_options(base_url, username, password)
return library_options
except Exception as exc:
logger.error(f"Failed to fetch Booklore libraries: {exc}")
except Exception:
logger.exception("Failed to fetch Booklore libraries")
if _BOOKLORE_OPTIONS_CACHE.get("key") == cache_key:
return _BOOKLORE_OPTIONS_CACHE.get("library_options", [])
return []
else:
return library_options
def get_booklore_path_options() -> list[dict[str, Any]]:
@@ -145,7 +148,7 @@ def get_booklore_path_options() -> list[dict[str, Any]]:
base_url = str(config.get("BOOKLORE_HOST", "") or "").strip().rstrip("/")
username = str(config.get("BOOKLORE_USERNAME", "") or "").strip()
password = config.get("BOOKLORE_PASSWORD", "") or ""
password = str(config.get("BOOKLORE_PASSWORD", "") or "")
if not base_url or not username or not password:
return []
@@ -154,19 +157,22 @@ def get_booklore_path_options() -> list[dict[str, Any]]:
try:
_, path_options = _get_booklore_cached_options(base_url, username, password)
return path_options
except Exception as exc:
logger.error(f"Failed to fetch Booklore paths: {exc}")
except Exception:
logger.exception("Failed to fetch Booklore paths")
if _BOOKLORE_OPTIONS_CACHE.get("key") == cache_key:
return _BOOKLORE_OPTIONS_CACHE.get("path_options", [])
return []
else:
return path_options
def test_booklore_connection(current_values: dict[str, Any] | None = None) -> dict[str, Any]:
def check_booklore_connection(
current_values: dict[str, Any] | None = None,
) -> dict[str, Any]:
"""Test the Booklore connection using current form values."""
current_values = current_values or {}
def _get_value(key: str, default: Any = None) -> Any:
def _get_value(key: str, default: object = None) -> object:
value = current_values.get(key)
if value not in (None, ""):
return value
@@ -176,22 +182,22 @@ def test_booklore_connection(current_values: dict[str, Any] | None = None) -> di
base_url = str(_get_value("BOOKLORE_HOST", "") or "").strip().rstrip("/")
username = str(_get_value("BOOKLORE_USERNAME", "") or "").strip()
password = _get_value("BOOKLORE_PASSWORD", "") or ""
password = str(_get_value("BOOKLORE_PASSWORD", "") or "")
if not base_url:
return {"success": False, "message": "Booklore URL is required"}
return {"success": False, "message": "Grimmory URL is required"}
if not username:
return {"success": False, "message": "Booklore username is required"}
return {"success": False, "message": "Grimmory username is required"}
if not password:
return {"success": False, "message": "Booklore password is required"}
return {"success": False, "message": "Grimmory password is required"}
try:
library_options, _ = _get_booklore_select_options(base_url, username, password)
message = "Connected to Booklore"
if library_options:
message = f"Connected to Booklore ({len(library_options)} libraries)"
return {"success": True, "message": message}
except BookloreError as exc:
return {"success": False, "message": str(exc)}
else:
message = "Connected to Grimmory"
if library_options:
message = f"Connected to Grimmory ({len(library_options)} libraries)"
return {"success": True, "message": message}
@@ -0,0 +1,110 @@
from __future__ import annotations
import re
from pathlib import Path
from typing import Any
_USER_PLACEHOLDER_PATTERN = re.compile(r"\{user\}", re.IGNORECASE)
def _get_download_setting_value(
current_values: dict[str, Any] | None,
key: str,
*,
default: object = None,
) -> object:
"""Read a downloads setting from unsaved form values first, then persisted config."""
from shelfmark.core.config import config
current_values = current_values or {}
if key in current_values:
return current_values[key]
if default is None:
return config.get(key)
return config.get(key, default)
def _resolve_destination_test_path(
configured_path: str,
) -> tuple[Path, str | None]:
"""Resolve a safe path to validate for destination test actions."""
stripped_path = configured_path.strip()
if not _USER_PLACEHOLDER_PATTERN.search(stripped_path):
return Path(stripped_path), None
base_prefix = _USER_PLACEHOLDER_PATTERN.split(stripped_path, maxsplit=1)[0].rstrip("/")
if not base_prefix and not stripped_path.startswith("/"):
return Path(stripped_path), None
base_path = base_prefix or "/"
return Path(base_path), (
f" (tested base path {base_path} from configured template {stripped_path})"
)
def _test_folder_destination(
*,
current_values: dict[str, Any] | None = None,
is_audiobook: bool,
) -> dict[str, Any]:
"""Validate a folder destination using current form values."""
from shelfmark.download.postprocess.destination import validate_destination
destination_value = _get_download_setting_value(
current_values,
"DESTINATION",
default="/books",
)
destination = str(destination_value or "").strip()
label = "Books destination"
message_suffix = ""
if is_audiobook:
audiobook_value = _get_download_setting_value(
current_values,
"DESTINATION_AUDIOBOOK",
default="",
)
audiobook_destination = str(audiobook_value or "").strip()
if audiobook_destination:
destination = audiobook_destination
label = "Audiobook destination"
else:
label = "Audiobook destination"
message_suffix = " (using the Books destination)"
if not destination:
return {"success": False, "message": f"{label} is required"}
test_path, path_message = _resolve_destination_test_path(destination)
if path_message:
message_suffix += path_message
errors: list[str] = []
def _status_callback(status: str, message: str | None) -> None:
if status == "error" and message:
errors.append(message)
if not validate_destination(test_path, _status_callback):
message = errors[-1] if errors else f"Cannot access destination: {test_path}"
if message_suffix:
message = f"{message}{message_suffix}"
return {"success": False, "message": message}
return {
"success": True,
"message": f"{label} is writable: {test_path}{message_suffix}",
}
def check_books_destination(current_values: dict[str, Any] | None = None) -> dict[str, Any]:
"""Validate the configured books destination."""
return _test_folder_destination(current_values=current_values, is_audiobook=False)
def check_audiobook_destination(current_values: dict[str, Any] | None = None) -> dict[str, Any]:
"""Validate the configured audiobook destination."""
return _test_folder_destination(current_values=current_values, is_audiobook=True)
+16 -8
View File
@@ -1,17 +1,25 @@
"""Helpers for email settings validation and SMTP connection tests."""
from __future__ import annotations
import smtplib
from typing import Any
from shelfmark.core.config import config
from shelfmark.download.outputs.email import EmailOutputError, build_email_smtp_config, test_smtp_connection
from shelfmark.download.outputs.email import (
EmailOutputError,
build_email_smtp_config,
test_smtp_connection,
)
def test_email_connection(current_values: dict[str, Any] | None = None) -> dict[str, Any]:
def check_email_connection(
current_values: dict[str, Any] | None = None,
) -> dict[str, Any]:
"""Test SMTP connectivity using current form values (including unsaved changes)."""
current_values = current_values or {}
def _get_value(key: str, default: Any = None) -> Any:
def _get_value(key: str, default: object = None) -> object:
value = current_values.get(key)
if value not in (None, ""):
return value
@@ -28,15 +36,15 @@ def test_email_connection(current_values: dict[str, Any] | None = None) -> dict[
"EMAIL_FROM": _get_value("EMAIL_FROM", ""),
"EMAIL_SUBJECT_TEMPLATE": _get_value("EMAIL_SUBJECT_TEMPLATE", "{Title}"),
"EMAIL_SMTP_TIMEOUT_SECONDS": _get_value("EMAIL_SMTP_TIMEOUT_SECONDS", 60),
"EMAIL_ALLOW_UNVERIFIED_TLS": _get_value("EMAIL_ALLOW_UNVERIFIED_TLS", False),
"EMAIL_ALLOW_UNVERIFIED_TLS": _get_value("EMAIL_ALLOW_UNVERIFIED_TLS", default=False),
}
try:
smtp_config = build_email_smtp_config(settings)
test_smtp_connection(smtp_config)
return {"success": True, "message": "Connected to SMTP server"}
except EmailOutputError as exc:
return {"success": False, "message": str(exc)}
except Exception as exc:
except (OSError, smtplib.SMTPException) as exc:
return {"success": False, "message": f"SMTP test failed: {exc}"}
else:
return {"success": True, "message": "Connected to SMTP server"}
+14 -9
View File
@@ -3,6 +3,7 @@
import json
import os
import shutil
import tempfile
from pathlib import Path
@@ -23,11 +24,11 @@ def _read_debug_from_config() -> bool:
if config_file.exists():
try:
with open(config_file, "r") as f:
with config_file.open() as f:
config = json.load(f)
if "DEBUG" in config:
return bool(config["DEBUG"])
except (json.JSONDecodeError, OSError):
except json.JSONDecodeError, OSError:
pass
return False
@@ -36,10 +37,10 @@ def _read_debug_from_config() -> bool:
def _is_sqlite_file(path: Path) -> bool:
"""Check if a file is a valid SQLite database by reading magic bytes."""
try:
with open(path, "rb") as f:
with path.open("rb") as f:
header = f.read(16)
return header[:16] == b"SQLite format 3\x00"
except (OSError, PermissionError):
except OSError, PermissionError:
return False
@@ -67,16 +68,20 @@ def _is_config_dir_writable() -> bool:
test_file = CONFIG_DIR / ".write_test"
test_file.touch()
test_file.unlink()
return True
except (OSError, PermissionError):
except OSError, PermissionError:
return False
else:
return True
def is_covers_cache_enabled() -> bool:
"""Check if cover caching is enabled (requires setting + writable config dir)."""
from shelfmark.core.config import config
setting_enabled = config.get("COVERS_CACHE_ENABLED", True)
return setting_enabled and _is_config_dir_writable()
if isinstance(setting_enabled, str):
return string_to_bool(setting_enabled) and _is_config_dir_writable()
return bool(setting_enabled) and _is_config_dir_writable()
# =============================================================================
@@ -87,7 +92,7 @@ CONFIG_DIR = Path(os.getenv("CONFIG_DIR", "/config"))
LOG_ROOT = Path(os.getenv("LOG_ROOT", "/var/log/"))
LOG_DIR = LOG_ROOT / "shelfmark"
LOG_FILE = LOG_DIR / "shelfmark.log"
TMP_DIR = Path(os.getenv("TMP_DIR", "/tmp/shelfmark"))
TMP_DIR = Path(os.getenv("TMP_DIR", (Path(tempfile.gettempdir()) / "shelfmark").as_posix()))
INGEST_DIR = Path(os.getenv("INGEST_DIR", "/books"))
@@ -151,7 +156,7 @@ ONBOARDING = string_to_bool(os.getenv("ONBOARDING", "true"))
# Debug: skip specific download sources for testing fallback chains
# Comma-separated values: aa-fast, aa-slow-nowait, aa-slow-wait, libgen, zlib, welib
_DEBUG_SKIP_SOURCES_RAW = os.getenv("DEBUG_SKIP_SOURCES", "").strip().lower()
DEBUG_SKIP_SOURCES = set(s.strip() for s in _DEBUG_SKIP_SOURCES_RAW.split(",") if s.strip())
DEBUG_SKIP_SOURCES = {s.strip() for s in _DEBUG_SKIP_SOURCES_RAW.split(",") if s.strip()}
# =============================================================================
+48 -34
View File
@@ -1,8 +1,14 @@
"""Configuration migration helpers."""
import json
from typing import Any, Callable
from __future__ import annotations
import json
from pathlib import Path
from typing import TYPE_CHECKING, Any, Protocol
if TYPE_CHECKING:
from collections.abc import Callable
from os import PathLike
_DEPRECATED_SETTINGS_RESTRICTION_KEYS = (
"PROXY_AUTH_RESTRICT_SETTINGS_TO_ADMIN",
@@ -11,7 +17,17 @@ _DEPRECATED_SETTINGS_RESTRICTION_KEYS = (
)
def _as_bool(value: Any) -> bool:
class MigrationLogger(Protocol):
"""Logger surface used by config migration helpers."""
def info(self, msg: str, *args: object) -> object: ...
def debug(self, msg: str, *args: object) -> object: ...
def exception(self, msg: str, *args: object) -> object: ...
def _as_bool(value: object) -> bool:
if isinstance(value, bool):
return value
if isinstance(value, str):
@@ -23,10 +39,7 @@ def _pick_legacy_settings_restriction(config: dict[str, Any]) -> bool | None:
"""Pick the best legacy admin-restriction value to migrate."""
auth_method = str(config.get("AUTH_METHOD", "")).strip().lower()
if (
auth_method == "proxy"
and "PROXY_AUTH_RESTRICT_SETTINGS_TO_ADMIN" in config
):
if auth_method == "proxy" and "PROXY_AUTH_RESTRICT_SETTINGS_TO_ADMIN" in config:
return _as_bool(config.get("PROXY_AUTH_RESTRICT_SETTINGS_TO_ADMIN"))
if auth_method == "cwa" and "CWA_RESTRICT_SETTINGS_TO_ADMIN" in config:
@@ -50,9 +63,9 @@ def migrate_security_settings(
load_users_config: Callable[[], dict[str, Any]],
save_users_config: Callable[[dict[str, Any]], None],
ensure_config_dir: Callable[[], None],
get_config_path: Callable[[], Any],
get_config_path: Callable[[], str | PathLike[str]],
sync_builtin_admin_user: Callable[[str, str], None],
logger: Any,
logger: MigrationLogger,
) -> None:
"""Migrate legacy security keys and sync builtin admin credentials."""
try:
@@ -67,13 +80,12 @@ def migrate_security_settings(
if old_value:
config["AUTH_METHOD"] = "cwa"
logger.info("Migrated USE_CWA_AUTH=True to AUTH_METHOD='cwa'")
elif config.get("BUILTIN_USERNAME") and config.get("BUILTIN_PASSWORD_HASH"):
config["AUTH_METHOD"] = "builtin"
logger.info("Migrated USE_CWA_AUTH=False to AUTH_METHOD='builtin'")
else:
if config.get("BUILTIN_USERNAME") and config.get("BUILTIN_PASSWORD_HASH"):
config["AUTH_METHOD"] = "builtin"
logger.info("Migrated USE_CWA_AUTH=False to AUTH_METHOD='builtin'")
else:
config["AUTH_METHOD"] = "none"
logger.info("Migrated USE_CWA_AUTH=False to AUTH_METHOD='none'")
config["AUTH_METHOD"] = "none"
logger.info("Migrated USE_CWA_AUTH=False to AUTH_METHOD='none'")
migrated_security = True
else:
logger.info("Removed deprecated USE_CWA_AUTH setting (AUTH_METHOD already exists)")
@@ -82,14 +94,17 @@ def migrate_security_settings(
# Backfill AUTH_METHOD for configs that have builtin credentials but
# were never migrated from USE_CWA_AUTH (e.g. dev builds that predated
# the AUTH_METHOD field).
if "AUTH_METHOD" not in config:
if config.get("BUILTIN_USERNAME") and config.get("BUILTIN_PASSWORD_HASH"):
config["AUTH_METHOD"] = "builtin"
migrated_security = True
logger.info(
"Backfilled AUTH_METHOD='builtin' from legacy "
"BUILTIN_USERNAME/BUILTIN_PASSWORD_HASH credentials"
)
if (
"AUTH_METHOD" not in config
and config.get("BUILTIN_USERNAME")
and config.get("BUILTIN_PASSWORD_HASH")
):
config["AUTH_METHOD"] = "builtin"
migrated_security = True
logger.info(
"Backfilled AUTH_METHOD='builtin' from legacy "
"BUILTIN_USERNAME/BUILTIN_PASSWORD_HASH credentials"
)
if "RESTRICT_SETTINGS_TO_ADMIN" not in users_config:
legacy_restrict = _pick_legacy_settings_restriction(config)
@@ -97,31 +112,30 @@ def migrate_security_settings(
save_users_config({"RESTRICT_SETTINGS_TO_ADMIN": legacy_restrict})
migrated_users = True
logger.info(
"Migrated legacy settings-admin restriction to users.RESTRICT_SETTINGS_TO_ADMIN="
f"{legacy_restrict}"
"Migrated legacy settings-admin restriction to users.RESTRICT_SETTINGS_TO_ADMIN=%s",
legacy_restrict,
)
for deprecated_key in _DEPRECATED_SETTINGS_RESTRICTION_KEYS:
if deprecated_key in config:
config.pop(deprecated_key, None)
migrated_security = True
logger.info(f"Removed deprecated security setting: {deprecated_key}")
logger.info("Removed deprecated security setting: %s", deprecated_key)
try:
sync_builtin_admin_user(
config.get("BUILTIN_USERNAME", ""),
config.get("BUILTIN_PASSWORD_HASH", ""),
)
except Exception as exc:
logger.error(
"Failed to sync builtin credentials to users database during migration: "
f"{exc}"
except Exception:
logger.exception(
"Failed to sync builtin credentials to users database during migration"
)
if migrated_security:
ensure_config_dir()
config_path = get_config_path()
with open(config_path, "w") as f:
config_path = Path(get_config_path())
with config_path.open("w") as f:
json.dump(config, f, indent=2)
logger.info("Security settings migration completed successfully")
elif migrated_users:
@@ -131,5 +145,5 @@ def migrate_security_settings(
except FileNotFoundError:
logger.debug("No existing security config file found - nothing to migrate")
except Exception as exc:
logger.error(f"Failed to migrate security settings: {exc}")
except Exception:
logger.exception("Failed to migrate security settings")
+8 -4
View File
@@ -6,10 +6,12 @@ import re
from typing import Any
from urllib.parse import urlsplit
from shelfmark.core.config import config as app_config
from shelfmark.core.notifications import NotificationEvent, send_test_notification
from shelfmark.core.settings_registry import (
ActionButton,
HeadingField,
SettingsField,
TableField,
load_config_file,
register_on_save,
@@ -123,7 +125,7 @@ def _count_invalid_route_urls(routes: list[dict[str, Any]]) -> int:
def _ensure_default_route_row(routes: list[dict[str, Any]]) -> list[dict[str, Any]]:
return routes if routes else [dict(row) for row in _DEFAULT_ROUTE_ROWS]
return routes or [dict(row) for row in _DEFAULT_ROUTE_ROWS]
def _extract_unique_route_urls(routes: list[dict[str, Any]]) -> list[str]:
@@ -141,6 +143,7 @@ def _extract_unique_route_urls(routes: list[dict[str, Any]]) -> list[str]:
def build_notification_test_result(routes_input: Any, *, scope_label: str) -> dict[str, Any]:
"""Validate routes and return a test-notification result payload."""
invalid_event_count = _count_invalid_route_events(routes_input)
if invalid_event_count:
return {
@@ -245,8 +248,9 @@ def _on_save_notifications(values: dict[str, Any]) -> dict[str, Any]:
def _test_admin_notification_action(current_values: dict[str, Any]) -> dict[str, Any]:
persisted = load_config_file("notifications")
effective: dict[str, Any] = dict(persisted)
effective: dict[str, Any] = {
"ADMIN_NOTIFICATION_ROUTES": app_config.get("ADMIN_NOTIFICATION_ROUTES", []),
}
if isinstance(current_values, dict):
effective.update(current_values)
@@ -258,7 +262,7 @@ register_on_save("notifications", _on_save_notifications)
@register_settings("notifications", "Notifications", icon="bell", order=7)
def notifications_settings():
def notifications_settings() -> list[SettingsField]:
"""Global notifications settings."""
return [
HeadingField(
+46 -30
View File
@@ -1,27 +1,32 @@
"""Authentication settings registration."""
from typing import Any, Dict, Callable
from typing import TYPE_CHECKING, Any
from shelfmark.config.migrations import migrate_security_settings
from shelfmark.config.security_handlers import (
check_oidc_connection,
on_save_security,
test_oidc_connection,
)
from shelfmark.core.config import config as app_config
from shelfmark.core.logger import setup_logger
from shelfmark.core.settings_registry import (
register_settings,
register_on_save,
load_config_file,
TextField,
SelectField,
PasswordField,
CheckboxField,
ActionButton,
TagListField,
CheckboxField,
CustomComponentField,
PasswordField,
SelectField,
SettingsField,
TagListField,
TextField,
load_config_file,
register_on_save,
register_settings,
)
from shelfmark.core.user_db import sync_builtin_admin_user
if TYPE_CHECKING:
from collections.abc import Callable
logger = setup_logger(__name__)
@@ -35,15 +40,18 @@ def _auth_field(factory: Callable[..., Any], auth_method: str, **kwargs: Any) ->
def _migrate_security_settings() -> None:
from shelfmark.core.settings_registry import (
_get_config_file_path,
_ensure_config_dir,
_get_config_file_path,
save_config_file,
)
def _save_users_config(values: dict[str, Any]) -> None:
save_config_file("users", values)
migrate_security_settings(
load_security_config=lambda: load_config_file("security"),
load_users_config=lambda: load_config_file("users"),
save_users_config=lambda values: save_config_file("users", values),
save_users_config=_save_users_config,
ensure_config_dir=lambda: _ensure_config_dir("security"),
get_config_path=lambda: _get_config_file_path("security"),
sync_builtin_admin_user=sync_builtin_admin_user,
@@ -51,21 +59,22 @@ def _migrate_security_settings() -> None:
)
def _on_save_security(values: Dict[str, Any]) -> Dict[str, Any]:
def _on_save_security(values: dict[str, Any]) -> dict[str, Any]:
return on_save_security(values)
def _test_oidc_connection(current_values: Dict[str, Any] = None) -> Dict[str, Any]:
return test_oidc_connection(
load_security_config=lambda: load_config_file("security"),
def _test_oidc_connection(current_values: dict[str, Any] | None = None) -> dict[str, Any]:
return check_oidc_connection(
load_security_config=lambda: {
"OIDC_DISCOVERY_URL": app_config.get("OIDC_DISCOVERY_URL", ""),
},
current_values=current_values or {},
logger=logger,
)
@register_settings("security", "Security", icon="shield", order=5)
def security_settings():
def security_settings() -> list[SettingsField]:
"""Security and authentication settings."""
from shelfmark.config.env import CWA_DB_PATH
@@ -83,7 +92,10 @@ def security_settings():
SelectField(
key="AUTH_METHOD",
label="Authentication Method",
description="Select the authentication method for accessing Shelfmark.",
description=(
"Select the authentication method for accessing Shelfmark. "
"Restart container after changing Calibre-Web passwords."
),
options=auth_method_options,
default="none",
),
@@ -102,18 +114,22 @@ def security_settings():
label="A local admin account is required before OIDC can be enabled.",
show_when=_auth_condition("oidc"),
),
*([] if cwa_db_available else [
CustomComponentField(
key="cwa_db_missing",
component="oidc_admin_hint",
label=(
"Calibre-Web database not detected. Mount your app.db to "
"/auth/app.db to enable this method. Authentication will fall "
"back to none until the database is available."
*(
[]
if cwa_db_available
else [
CustomComponentField(
key="cwa_db_missing",
component="oidc_admin_hint",
label=(
"Calibre-Web database not detected. Mount your app.db to "
"/auth/app.db to enable this method. Authentication will fall "
"back to none until the database is available."
),
show_when=_auth_condition("cwa"),
),
show_when=_auth_condition("cwa"),
),
]),
]
),
ActionButton(
key="open_users_tab",
label="Go to Users",
+64 -11
View File
@@ -1,21 +1,55 @@
"""Operational handlers for security settings (save/actions)."""
import os
from typing import Any, Callable
from pathlib import Path
from typing import TYPE_CHECKING, Any
from shelfmark.core.utils import normalize_http_url
from shelfmark.core.user_db import UserDB
from shelfmark.core.utils import normalize_http_url
from shelfmark.download.network import get_ssl_verify
if TYPE_CHECKING:
from collections.abc import Callable
_OIDC_LOCKOUT_MESSAGE = "A local admin account with a password is required before enabling OIDC. Use the 'Go to Users' button above to create one. This ensures you can still sign in if your identity provider is unavailable."
_OIDC_REQUIRED_FIELDS = (
("OIDC_DISCOVERY_URL", "Discovery URL"),
("OIDC_CLIENT_ID", "Client ID"),
("OIDC_CLIENT_SECRET", "Client Secret"),
)
def _has_local_password_admin() -> bool:
root = os.environ.get("CONFIG_DIR", "/config")
user_db = UserDB(os.path.join(root, "users.db"))
user_db = UserDB(str(Path(root) / "users.db"))
user_db.initialize()
return any(user.get("password_hash") and user.get("role") == "admin" for user in user_db.list_users())
return any(
user.get("password_hash") and user.get("role") == "admin" for user in user_db.list_users()
)
def _load_effective_security_values(values: dict[str, Any]) -> dict[str, Any]:
"""Merge the current save payload onto the persisted security config."""
from shelfmark.core.settings_registry import load_config_file
effective_values = load_config_file("security")
effective_values.update(values)
return effective_values
def _get_missing_oidc_required_fields(effective_values: dict[str, Any]) -> list[str]:
"""Return missing required OIDC field labels from the effective config."""
missing_fields: list[str] = []
for key, label in _OIDC_REQUIRED_FIELDS:
value = effective_values.get(key)
if value is None:
missing_fields.append(label)
continue
if isinstance(value, str) and not value.strip():
missing_fields.append(label)
return missing_fields
def on_save_security(
@@ -29,6 +63,7 @@ def on_save_security(
normalized_values["OIDC_DISCOVERY_URL"] = normalize_http_url(
str(discovery_url),
default_scheme="https",
strip_trailing_slash=False,
)
proxy_logout_url = normalized_values.get("PROXY_AUTH_LOGOUT_URL")
@@ -39,13 +74,26 @@ def on_save_security(
strip_trailing_slash=False,
)
if normalized_values.get("AUTH_METHOD") == "oidc" and not _has_local_password_admin():
return {"error": True, "message": _OIDC_LOCKOUT_MESSAGE, "values": normalized_values}
effective_values = _load_effective_security_values(normalized_values)
auth_method = str(effective_values.get("AUTH_METHOD", "") or "").strip().lower()
if auth_method == "oidc":
if not _has_local_password_admin():
return {"error": True, "message": _OIDC_LOCKOUT_MESSAGE, "values": normalized_values}
missing_fields = _get_missing_oidc_required_fields(effective_values)
if missing_fields:
missing_fields_text = ", ".join(missing_fields)
return {
"error": True,
"message": f"OIDC configuration is incomplete: missing {missing_fields_text}.",
"values": normalized_values,
}
return {"error": False, "values": normalized_values}
def test_oidc_connection(
def check_oidc_connection(
*,
load_security_config: Callable[[], dict[str, Any]],
current_values: dict[str, Any] | None = None,
@@ -56,7 +104,9 @@ def test_oidc_connection(
try:
# Prefer the current (unsaved) form value over the saved config
discovery_url = (current_values or {}).get("OIDC_DISCOVERY_URL") or load_security_config().get("OIDC_DISCOVERY_URL", "")
discovery_url = (current_values or {}).get(
"OIDC_DISCOVERY_URL"
) or load_security_config().get("OIDC_DISCOVERY_URL", "")
if not discovery_url:
return {"success": False, "message": "Discovery URL is not configured."}
@@ -67,9 +117,12 @@ def test_oidc_connection(
required_fields = ["issuer", "authorization_endpoint", "token_endpoint"]
missing_fields = [field for field in required_fields if field not in document]
if missing_fields:
return {"success": False, "message": f"Discovery document missing fields: {', '.join(missing_fields)}"}
return {
"success": False,
"message": f"Discovery document missing fields: {', '.join(missing_fields)}",
}
return {"success": True, "message": f"Connected to {document['issuer']}"}
except Exception as exc:
logger.error(f"OIDC connection test failed: {exc}")
return {"success": False, "message": f"Connection failed: {str(exc)}"}
logger.exception("OIDC connection test failed")
return {"success": False, "message": f"Connection failed: {exc!s}"}
File diff suppressed because it is too large Load Diff
+77 -56
View File
@@ -7,6 +7,11 @@ that talks to /api/admin/users endpoints.
from typing import Any
from shelfmark.core.request_policy import (
get_source_content_type_capabilities,
parse_policy_mode,
validate_policy_rules,
)
from shelfmark.core.settings_registry import (
CheckboxField,
CustomComponentField,
@@ -14,16 +19,11 @@ from shelfmark.core.settings_registry import (
MultiSelectField,
NumberField,
SelectField,
SettingsField,
TableField,
register_on_save,
register_settings,
)
from shelfmark.core.request_policy import (
get_source_content_type_capabilities,
parse_policy_mode,
validate_policy_rules,
)
_REQUEST_DEFAULT_MODE_OPTIONS = [
{
@@ -72,10 +72,16 @@ _SELF_SETTINGS_SECTION_OPTIONS = [
_SELF_SETTINGS_SECTION_VALUES = {option["value"] for option in _SELF_SETTINGS_SECTION_OPTIONS}
_SELF_SETTINGS_SECTION_DEFAULTS = [option["value"] for option in _SELF_SETTINGS_SECTION_OPTIONS]
_SEARCH_MODE_VALUES = {"direct", "universal"}
_SEARCH_PREFERENCE_PROVIDER_KEYS = {"METADATA_PROVIDER", "METADATA_PROVIDER_AUDIOBOOK"}
_SEARCH_PREFERENCE_PROVIDER_KEYS = {
"METADATA_PROVIDER",
"METADATA_PROVIDER_AUDIOBOOK",
"METADATA_PROVIDER_COMBINED",
}
_SEARCH_PREFERENCE_VALIDATABLE_KEYS = {
"SEARCH_MODE",
"DEFAULT_RELEASE_SOURCE",
"DEFAULT_RELEASE_SOURCE_AUDIOBOOK",
"SHOW_COMBINED_SELECTOR",
*_SEARCH_PREFERENCE_PROVIDER_KEYS,
}
@@ -102,35 +108,45 @@ _USERS_HEADING_DESCRIPTION_BY_AUTH_MODE = {
}
def _get_request_source_options():
def _get_request_source_options() -> list[dict[str, str]]:
"""Build request-policy source options from registered release sources."""
from shelfmark.release_sources import list_available_sources
options = []
return [
{
"value": source["name"],
"label": source["display_name"],
}
for source in list_available_sources()
]
def _get_valid_release_source_names_for_content_type(content_type: str) -> set[str]:
"""Return registered release source names that support the requested content type."""
from shelfmark.release_sources import list_available_sources
valid_sources: set[str] = set()
for source in list_available_sources():
options.append(
{
"value": source["name"],
"label": source["display_name"],
}
)
return options
supported_types = source.get("supported_content_types", ["ebook", "audiobook"])
if content_type in supported_types:
valid_sources.add(source["name"])
return valid_sources
def _get_request_policy_rule_columns():
def _get_request_policy_rule_columns() -> list[dict[str, object]]:
source_capabilities = get_source_content_type_capabilities()
content_type_options = []
for source_name, supported_types in source_capabilities.items():
normalized_types = [t for t in ("ebook", "audiobook") if t in supported_types]
for content_type in normalized_types:
content_type_options.append(
{
"value": content_type,
"label": "Ebook" if content_type == "ebook" else "Audiobook",
"childOf": source_name,
}
)
content_type_options.extend(
{
"value": content_type,
"label": "Ebook" if content_type == "ebook" else "Audiobook",
"childOf": source_name,
}
for content_type in normalized_types
)
return [
{
@@ -189,23 +205,28 @@ def validate_search_preference_value(key: str, value: Any) -> tuple[Any, str | N
)
return normalized_value, None
if key == "DEFAULT_RELEASE_SOURCE":
if key in {"DEFAULT_RELEASE_SOURCE", "DEFAULT_RELEASE_SOURCE_AUDIOBOOK"}:
if normalized_value == "":
return "", None
from shelfmark.release_sources import list_available_sources
valid_sources = {source["name"] for source in list_available_sources()}
valid_sources = _get_valid_release_source_names_for_content_type(
"audiobook" if key == "DEFAULT_RELEASE_SOURCE_AUDIOBOOK" else "ebook"
)
if normalized_value not in valid_sources:
return (
value,
"DEFAULT_RELEASE_SOURCE must be a valid release source name or empty",
f"{key} must be a valid release source name or empty",
)
return normalized_value, None
if key == "SHOW_COMBINED_SELECTOR":
if isinstance(value, bool):
return value, None
return bool(value), None
return value, None
def _on_save_users(values):
def _on_save_users(values: dict[str, object]) -> dict[str, object]:
"""Validate users/request-policy settings before persistence."""
if "VISIBLE_SELF_SETTINGS_SECTIONS" in values:
raw_sections = values["VISIBLE_SELF_SETTINGS_SECTIONS"]
@@ -214,7 +235,9 @@ def _on_save_users(values):
elif isinstance(raw_sections, str):
candidate_sections = [s.strip() for s in raw_sections.split(",") if s.strip()]
elif isinstance(raw_sections, (list, tuple, set)):
candidate_sections = [str(section).strip() for section in raw_sections if str(section).strip()]
candidate_sections = [
str(section).strip() for section in raw_sections if str(section).strip()
]
else:
return {
"error": True,
@@ -239,21 +262,25 @@ def _on_save_users(values):
values["VISIBLE_SELF_SETTINGS_SECTIONS"] = normalized_sections
if "REQUEST_POLICY_DEFAULT_EBOOK" in values:
if parse_policy_mode(values["REQUEST_POLICY_DEFAULT_EBOOK"]) is None:
return {
"error": True,
"message": "REQUEST_POLICY_DEFAULT_EBOOK must be a valid policy mode",
"values": values,
}
if (
"REQUEST_POLICY_DEFAULT_EBOOK" in values
and parse_policy_mode(values["REQUEST_POLICY_DEFAULT_EBOOK"]) is None
):
return {
"error": True,
"message": "REQUEST_POLICY_DEFAULT_EBOOK must be a valid policy mode",
"values": values,
}
if "REQUEST_POLICY_DEFAULT_AUDIOBOOK" in values:
if parse_policy_mode(values["REQUEST_POLICY_DEFAULT_AUDIOBOOK"]) is None:
return {
"error": True,
"message": "REQUEST_POLICY_DEFAULT_AUDIOBOOK must be a valid policy mode",
"values": values,
}
if (
"REQUEST_POLICY_DEFAULT_AUDIOBOOK" in values
and parse_policy_mode(values["REQUEST_POLICY_DEFAULT_AUDIOBOOK"]) is None
):
return {
"error": True,
"message": "REQUEST_POLICY_DEFAULT_AUDIOBOOK must be a valid policy mode",
"values": values,
}
if "REQUEST_POLICY_RULES" in values:
normalized_rules, errors = validate_policy_rules(values["REQUEST_POLICY_RULES"])
@@ -284,7 +311,7 @@ register_on_save("users", _on_save_users)
@register_settings("users", "Users & Requests", icon="users", order=6)
def users_settings():
def users_settings() -> list[SettingsField]:
"""User management tab - rendered as a custom component on the frontend."""
return [
HeadingField(
@@ -311,9 +338,7 @@ def users_settings():
HeadingField(
key="requests_heading",
title="Requests",
description=(
"Choose what users can download directly and what needs approval first."
),
description=("Choose what users can download directly and what needs approval first."),
),
CheckboxField(
key="REQUESTS_ENABLED",
@@ -337,9 +362,7 @@ def users_settings():
SelectField(
key="REQUEST_POLICY_DEFAULT_EBOOK",
label="Default Ebook Mode",
description=(
"Sets the baseline for all ebook sources."
),
description=("Sets the baseline for all ebook sources."),
options=_REQUEST_DEFAULT_MODE_OPTIONS,
default="download",
user_overridable=True,
@@ -347,9 +370,7 @@ def users_settings():
SelectField(
key="REQUEST_POLICY_DEFAULT_AUDIOBOOK",
label="Default Audiobook Mode",
description=(
"Sets the baseline for all audiobook sources."
),
description=("Sets the baseline for all audiobook sources."),
options=_REQUEST_DEFAULT_MODE_OPTIONS,
default="download",
user_overridable=True,
+11 -2
View File
@@ -1,5 +1,14 @@
"""Core module - shared models, queue, and utilities."""
from shelfmark.core.models import QueueItem, SearchFilters, QueueStatus
from shelfmark.core.queue import BookQueue, book_queue
from shelfmark.core.logger import setup_logger
from shelfmark.core.models import QueueItem, QueueStatus, SearchFilters
from shelfmark.core.queue import BookQueue, book_queue
__all__ = [
"BookQueue",
"QueueItem",
"QueueStatus",
"SearchFilters",
"book_queue",
"setup_logger",
]
File diff suppressed because it is too large Load Diff
+44 -27
View File
@@ -8,7 +8,6 @@ from typing import Any
from shelfmark.core.request_helpers import now_utc_iso
VALID_ACTIVITY_ITEM_TYPES = frozenset({"download", "request"})
ADMIN_VIEWER_SCOPE = "admin:shared"
NOAUTH_VIEWER_SCOPE = "noauth:shared"
@@ -16,59 +15,69 @@ USER_VIEWER_SCOPE_PREFIX = "user:"
def user_viewer_scope(user_id: int) -> str:
"""Build the persisted viewer scope string for a specific user."""
if not isinstance(user_id, int) or user_id < 1:
raise ValueError("user_id must be a positive integer")
msg = "user_id must be a positive integer"
raise ValueError(msg)
return f"{USER_VIEWER_SCOPE_PREFIX}{user_id}"
def normalize_viewer_scope(viewer_scope: Any) -> str:
def normalize_viewer_scope(viewer_scope: object) -> str:
"""Validate and normalize a persisted viewer scope string."""
if not isinstance(viewer_scope, str) or not viewer_scope.strip():
raise ValueError("viewer_scope must be a non-empty string")
msg = "viewer_scope must be a non-empty string"
raise ValueError(msg)
normalized = viewer_scope.strip()
if normalized in {ADMIN_VIEWER_SCOPE, NOAUTH_VIEWER_SCOPE}:
return normalized
if not normalized.startswith(USER_VIEWER_SCOPE_PREFIX):
raise ValueError(
"viewer_scope must be one of: admin:shared, noauth:shared, or user:<id>"
)
msg = "viewer_scope must be one of: admin:shared, noauth:shared, or user:<id>"
raise ValueError(msg)
raw_user_id = normalized[len(USER_VIEWER_SCOPE_PREFIX):].strip()
raw_user_id = normalized[len(USER_VIEWER_SCOPE_PREFIX) :].strip()
try:
parsed_user_id = int(raw_user_id)
except (TypeError, ValueError) as exc:
raise ValueError("viewer_scope user id must be a positive integer") from exc
msg = "viewer_scope user id must be a positive integer"
raise ValueError(msg) from exc
return user_viewer_scope(parsed_user_id)
def _normalize_item_type(item_type: Any) -> str:
def _normalize_item_type(item_type: object) -> str:
if not isinstance(item_type, str) or not item_type.strip():
raise ValueError("item_type must be a non-empty string")
msg = "item_type must be a non-empty string"
raise ValueError(msg)
normalized = item_type.strip().lower()
if normalized not in VALID_ACTIVITY_ITEM_TYPES:
raise ValueError("item_type must be one of: download, request")
msg = "item_type must be one of: download, request"
raise ValueError(msg)
return normalized
def _normalize_item_key(item_key: Any, *, item_type: str) -> str:
def _normalize_item_key(item_key: object, *, item_type: str) -> str:
if not isinstance(item_key, str) or not item_key.strip():
raise ValueError("item_key must be a non-empty string")
msg = "item_key must be a non-empty string"
raise ValueError(msg)
normalized = item_key.strip()
expected_prefix = f"{item_type}:"
if not normalized.startswith(expected_prefix):
raise ValueError(f"item_key must be in the format {expected_prefix}<id>")
msg_0 = f"item_key must be in the format {expected_prefix}<id>"
raise ValueError(msg_0)
if not normalized.split(":", 1)[1].strip():
raise ValueError(f"item_key must be in the format {expected_prefix}<id>")
msg_0 = f"item_key must be in the format {expected_prefix}<id>"
raise ValueError(msg_0)
return normalized
class ActivityViewStateService:
"""Service for per-viewer activity dismissal and history visibility."""
def __init__(self, db_path: str):
def __init__(self, db_path: str) -> None:
"""Initialize the service with the SQLite state database path."""
self._db_path = db_path
self._lock = threading.Lock()
@@ -84,6 +93,7 @@ class ActivityViewStateService:
viewer_scope: str,
limit: int | None = None,
) -> list[dict[str, Any]]:
"""Return dismissed rows for a viewer, including cleared history entries."""
normalized_scope = normalize_viewer_scope(viewer_scope)
normalized_limit = None if limit is None else max(1, int(limit))
query = """
@@ -112,6 +122,7 @@ class ActivityViewStateService:
limit: int = 50,
offset: int = 0,
) -> list[dict[str, Any]]:
"""Return active dismissal history rows for a viewer."""
normalized_scope = normalize_viewer_scope(viewer_scope)
normalized_limit = max(1, min(int(limit), 5000))
normalized_offset = max(0, int(offset))
@@ -141,6 +152,7 @@ class ActivityViewStateService:
item_type: str,
item_key: str,
) -> int:
"""Mark a single activity item as dismissed for a viewer."""
normalized_scope = normalize_viewer_scope(viewer_scope)
normalized_type = _normalize_item_type(item_type)
normalized_key = _normalize_item_key(item_key, item_type=normalized_type)
@@ -177,6 +189,7 @@ class ActivityViewStateService:
viewer_scope: str,
items: list[dict[str, str]],
) -> int:
"""Mark multiple activity items as dismissed for a viewer."""
normalized_scope = normalize_viewer_scope(viewer_scope)
if not items:
return 0
@@ -215,7 +228,12 @@ class ActivityViewStateService:
dismissed_at = excluded.dismissed_at,
cleared_at = NULL
""",
(normalized_scope, normalized_type, normalized_key, dismissed_at),
(
normalized_scope,
normalized_type,
normalized_key,
dismissed_at,
),
)
rowcount = int(cursor.rowcount) if cursor.rowcount is not None else 0
total += max(rowcount, 0)
@@ -225,6 +243,7 @@ class ActivityViewStateService:
conn.close()
def clear_history(self, *, viewer_scope: str) -> int:
"""Mark all dismissed items as cleared for a viewer."""
normalized_scope = normalize_viewer_scope(viewer_scope)
cleared_at = now_utc_iso()
@@ -248,6 +267,7 @@ class ActivityViewStateService:
conn.close()
def clear_item_for_all_viewers(self, *, item_type: str, item_key: str) -> int:
"""Delete a dismissed item record for every viewer."""
normalized_type = _normalize_item_type(item_type)
normalized_key = _normalize_item_key(item_key, item_type=normalized_type)
@@ -268,6 +288,7 @@ class ActivityViewStateService:
conn.close()
def delete_viewer_scope(self, *, viewer_scope: str) -> int:
"""Delete all activity-view state rows for a viewer scope."""
normalized_scope = normalize_viewer_scope(viewer_scope)
with self._lock:
@@ -284,24 +305,20 @@ class ActivityViewStateService:
conn.close()
def delete_items(self, *, item_type: str, item_keys: list[str]) -> int:
"""Delete multiple dismissed item records for a given item type."""
normalized_type = _normalize_item_type(item_type)
normalized_keys = [
_normalize_item_key(item_key, item_type=normalized_type)
for item_key in item_keys
_normalize_item_key(item_key, item_type=normalized_type) for item_key in item_keys
]
if not normalized_keys:
return 0
placeholders = ",".join("?" for _ in normalized_keys)
with self._lock:
conn = self._connect()
try:
cursor = conn.execute(
f"""
DELETE FROM activity_view_state
WHERE item_type = ? AND item_key IN ({placeholders})
""",
(normalized_type, *normalized_keys),
cursor = conn.executemany(
"DELETE FROM activity_view_state WHERE item_type = ? AND item_key = ?",
[(normalized_type, normalized_key) for normalized_key in normalized_keys],
)
conn.commit()
rowcount = int(cursor.rowcount) if cursor.rowcount is not None else 0
+161 -107
View File
@@ -4,12 +4,14 @@ Registers /api/admin/users CRUD endpoints for managing users.
All endpoints require admin session.
"""
from functools import wraps
from __future__ import annotations
import os
import sqlite3
from typing import Any
from functools import wraps
from typing import TYPE_CHECKING, Any, ParamSpec
from flask import Flask, g, jsonify, request, session
from flask import Flask, Response, g, jsonify, request, session
from werkzeug.security import generate_password_hash
from shelfmark.config.booklore_settings import (
@@ -30,12 +32,29 @@ from shelfmark.core.auth_modes import (
load_active_auth_mode,
normalize_auth_source,
)
from shelfmark.core.config import config as app_config
from shelfmark.core.cwa_user_sync import sync_cwa_users_from_rows
from shelfmark.core.logger import setup_logger
from shelfmark.core.settings_registry import load_config_file
from shelfmark.core.user_db import UserDB
if TYPE_CHECKING:
from collections.abc import Callable
from flask.typing import ResponseReturnValue
from shelfmark.core.user_db import UserDB
P = ParamSpec("P")
logger = setup_logger(__name__)
MIN_PASSWORD_LENGTH = 4
_CONFIG_REFRESH_ERRORS = (ImportError, OSError, RuntimeError, TypeError, ValueError)
__all__ = [
"get_booklore_library_options",
"get_booklore_path_options",
"register_admin_routes",
"validate_user_settings",
]
def _get_user_edit_capabilities(
@@ -47,10 +66,12 @@ def _get_user_edit_capabilities(
user.get("auth_source"),
user.get("oidc_subject"),
)
if security_config is None and auth_source == AUTH_SOURCE_OIDC:
security_config = load_config_file("security")
oidc_use_admin_group = bool((security_config or {}).get("OIDC_USE_ADMIN_GROUP", True))
oidc_use_admin_group = bool(
(security_config or {}).get(
"OIDC_USE_ADMIN_GROUP",
app_config.get("OIDC_USE_ADMIN_GROUP", True),
)
)
role_managed_by_oidc_group = auth_source == AUTH_SOURCE_OIDC and oidc_use_admin_group
can_edit_role = auth_source == AUTH_SOURCE_BUILTIN or (
auth_source == AUTH_SOURCE_OIDC and not role_managed_by_oidc_group
@@ -72,16 +93,18 @@ def _sanitize_user(user: dict) -> dict:
return sanitized
def _oidc_role_management_message(security_config: dict[str, Any]) -> str:
admin_group = security_config.get("OIDC_ADMIN_GROUP", "")
def _oidc_role_management_message(security_config: dict[str, Any] | None = None) -> str:
admin_group = (security_config or {}).get(
"OIDC_ADMIN_GROUP",
app_config.get("OIDC_ADMIN_GROUP", ""),
)
if admin_group:
return (
"Admin roles for OIDC users are managed by the "
f"'{admin_group}' group in your identity provider"
)
return (
"Disable 'Use Admin Group for Authorization' in security settings "
"to manage roles manually"
"Disable 'Use Admin Group for Authorization' in security settings to manage roles manually"
)
@@ -104,12 +127,11 @@ def _serialize_user(
return payload
def _sync_all_cwa_users(user_db: UserDB) -> dict[str, int]:
"""Sync all users from the Calibre-Web database into users.db."""
if not CWA_DB_PATH or not CWA_DB_PATH.exists():
raise FileNotFoundError("Calibre-Web database is not available")
msg = "Calibre-Web database is not available"
raise FileNotFoundError(msg)
db_path = os.fspath(CWA_DB_PATH)
db_uri = f"file:{db_path}?mode=ro&immutable=1"
@@ -127,15 +149,18 @@ def _sync_all_cwa_users(user_db: UserDB) -> dict[str, int]:
def register_admin_routes(app: Flask, user_db: UserDB) -> None:
"""Register admin user management routes on the Flask app."""
def _require_admin(f):
"""Decorator to require admin session for admin routes.
def _require_admin(
f: Callable[P, ResponseReturnValue],
) -> Callable[P, ResponseReturnValue]:
"""Require an admin session for admin routes.
In no-auth mode, everyone has access (is_admin defaults True).
In auth-required modes, requires an authenticated session with admin role.
Caches the resolved auth_mode in ``g.auth_mode`` for the request.
"""
@wraps(f)
def decorated(*args, **kwargs):
def decorated(*args: P.args, **kwargs: P.kwargs) -> ResponseReturnValue:
auth_mode = load_active_auth_mode(CWA_DB_PATH, user_db=user_db)
g.auth_mode = auth_mode
if auth_mode != "none":
@@ -144,23 +169,20 @@ def register_admin_routes(app: Flask, user_db: UserDB) -> None:
if not session.get("is_admin", False):
return jsonify({"error": "Admin access required"}), 403
return f(*args, **kwargs)
return decorated
@app.route("/api/admin/users", methods=["GET"])
@_require_admin
def admin_list_users():
def admin_list_users() -> Response | tuple[Response, int]:
"""List all users."""
users = user_db.list_users()
auth_mode = g.auth_mode
security_config = load_config_file("security")
return jsonify([
_serialize_user(u, auth_mode, security_config=security_config)
for u in users
])
return jsonify([_serialize_user(u, auth_mode) for u in users])
@app.route("/api/admin/users", methods=["POST"])
@_require_admin
def admin_create_user():
def admin_create_user() -> Response | tuple[Response, int]:
"""Create a new user with password authentication."""
data = request.get_json() or {}
auth_mode = g.auth_mode
@@ -172,18 +194,22 @@ def register_admin_routes(app: Flask, user_db: UserDB) -> None:
role = data.get("role", "user")
if auth_mode in {AUTH_SOURCE_PROXY, AUTH_SOURCE_CWA}:
return jsonify({
"error": "Local user creation is disabled in this authentication mode",
"message": (
"Users are provisioned by your external authentication source. "
"Switch to builtin or OIDC mode to create local users."
),
}), 400
return jsonify(
{
"error": "Local user creation is disabled in this authentication mode",
"message": (
"Users are provisioned by your external authentication source. "
"Switch to builtin or OIDC mode to create local users."
),
}
), 400
if not username:
return jsonify({"error": "Username is required"}), 400
if not password or len(password) < 4:
return jsonify({"error": "Password must be at least 4 characters"}), 400
if not password or len(password) < MIN_PASSWORD_LENGTH:
return jsonify(
{"error": f"Password must be at least {MIN_PASSWORD_LENGTH} characters"}
), 400
if role not in ("admin", "user"):
return jsonify({"error": "Role must be 'admin' or 'user'"}), 400
@@ -209,21 +235,22 @@ def register_admin_routes(app: Flask, user_db: UserDB) -> None:
except ValueError:
return jsonify({"error": "Username already exists"}), 409
logger.info(
"Shelfmark user created "
f"(source=manual_admin_create, created_by={session.get('user_id', 'unknown')}, "
f"username={username}, role={role}, auth_source={AUTH_SOURCE_BUILTIN})"
"Shelfmark user created (source=manual_admin_create, created_by=%s, username=%s, role=%s, auth_source=%s)",
session.get("user_id", "unknown"),
username,
role,
AUTH_SOURCE_BUILTIN,
)
return jsonify(
_serialize_user(
user,
g.auth_mode,
security_config=load_config_file("security"),
)
), 201
@app.route("/api/admin/users/<int:user_id>", methods=["GET"])
@_require_admin
def admin_get_user(user_id):
def admin_get_user(user_id: int) -> Response | tuple[Response, int]:
"""Get a user by ID with their settings."""
user = user_db.get_user(user_id=user_id)
if not user:
@@ -232,37 +259,39 @@ def register_admin_routes(app: Flask, user_db: UserDB) -> None:
result = _serialize_user(
user,
g.auth_mode,
security_config=load_config_file("security"),
)
result["settings"] = user_db.get_user_settings(user_id)
return jsonify(result)
@app.route("/api/admin/users/<int:user_id>", methods=["PUT"])
@_require_admin
def admin_update_user(user_id):
def admin_update_user(user_id: int) -> Response | tuple[Response, int]:
"""Update user fields and/or settings."""
user = user_db.get_user(user_id=user_id)
if not user:
return jsonify({"error": "User not found"}), 404
data = request.get_json() or {}
security_config = load_config_file("security")
auth_source = normalize_auth_source(
user.get("auth_source"),
user.get("oidc_subject"),
)
capabilities = _get_user_edit_capabilities(user, security_config=security_config)
capabilities = _get_user_edit_capabilities(user)
# Handle optional password update
password = data.get("password", "")
if password:
if not capabilities["canSetPassword"]:
return jsonify({
"error": f"Cannot set password for {auth_source.upper()} users",
"message": "Password authentication is only available for local users.",
}), 400
if len(password) < 4:
return jsonify({"error": "Password must be at least 4 characters"}), 400
return jsonify(
{
"error": f"Cannot set password for {auth_source.upper()} users",
"message": "Password authentication is only available for local users.",
}
), 400
if len(password) < MIN_PASSWORD_LENGTH:
return jsonify(
{"error": f"Password must be at least {MIN_PASSWORD_LENGTH} characters"}
), 400
user_db.update_user(user_id, password_hash=generate_password_hash(password))
# Update user fields
@@ -276,40 +305,49 @@ def register_admin_routes(app: Flask, user_db: UserDB) -> None:
role_changed = "role" in user_fields and user_fields["role"] != user.get("role")
email_changed = "email" in user_fields and user_fields["email"] != user.get("email")
display_name_changed = (
"display_name" in user_fields
and user_fields["display_name"] != user.get("display_name")
)
display_name_changed = "display_name" in user_fields and user_fields[
"display_name"
] != user.get("display_name")
if role_changed and not capabilities["canEditRole"]:
if auth_source == AUTH_SOURCE_OIDC:
return jsonify({
"error": "Cannot change role for OIDC user when group-based authorization is enabled",
"message": _oidc_role_management_message(security_config),
}), 400
return jsonify(
{
"error": "Cannot change role for OIDC user when group-based authorization is enabled",
"message": _oidc_role_management_message(),
}
), 400
return jsonify({
"error": f"Cannot change role for {auth_source.upper()} users",
"message": "Role is managed by the external authentication source.",
}), 400
return jsonify(
{
"error": f"Cannot change role for {auth_source.upper()} users",
"message": "Role is managed by the external authentication source.",
}
), 400
if email_changed and not capabilities["canEditEmail"]:
if auth_source == AUTH_SOURCE_CWA:
return jsonify({
"error": "Cannot change email for CWA users",
"message": "Email is synced from Calibre-Web.",
}), 400
return jsonify(
{
"error": "Cannot change email for CWA users",
"message": "Email is synced from Calibre-Web.",
}
), 400
return jsonify({
"error": "Cannot change email for OIDC users",
"message": "Email is managed by your identity provider.",
}), 400
return jsonify(
{
"error": "Cannot change email for OIDC users",
"message": "Email is managed by your identity provider.",
}
), 400
if display_name_changed and not capabilities["canEditDisplayName"]:
return jsonify({
"error": "Cannot change display name for OIDC users",
"message": "Display name is managed by your identity provider.",
}), 400
return jsonify(
{
"error": "Cannot change display name for OIDC users",
"message": "Display name is managed by your identity provider.",
}
), 400
# Allow demoting the last admin account.
# Auth mode resolution automatically falls back to "none" when no
@@ -330,50 +368,62 @@ def register_admin_routes(app: Flask, user_db: UserDB) -> None:
validated_settings, validation_errors = validate_user_settings(data["settings"])
if validation_errors:
return jsonify({
"error": "Invalid settings payload",
"details": validation_errors,
}), 400
return jsonify(
{
"error": "Invalid settings payload",
"details": validation_errors,
}
), 400
user_db.set_user_settings(user_id, validated_settings)
# Ensure runtime reads see updated per-user overrides immediately.
try:
from shelfmark.core.config import config as app_config
app_config.refresh(force=True)
except Exception:
pass
except _CONFIG_REFRESH_ERRORS as exc:
logger.warning(
"Updated settings for user %s but failed to refresh runtime config: %s",
user_id,
exc,
)
updated = user_db.get_user(user_id=user_id)
if not isinstance(updated, dict):
return jsonify({"error": "User not found"}), 404
result = _serialize_user(
updated,
g.auth_mode,
security_config=security_config,
)
result["settings"] = user_db.get_user_settings(user_id)
logger.info(f"Admin updated user {user_id}")
logger.info("Admin updated user %s", user_id)
return jsonify(result)
@app.route("/api/admin/users/sync-cwa", methods=["POST"])
@_require_admin
def admin_sync_cwa_users():
def admin_sync_cwa_users() -> Response | tuple[Response, int]:
"""Manually sync users from Calibre-Web into users.db."""
if g.auth_mode != AUTH_SOURCE_CWA:
return jsonify({
"error": "CWA sync is only available when CWA authentication is enabled",
}), 400
return jsonify(
{
"error": "CWA sync is only available when CWA authentication is enabled",
}
), 400
try:
summary = _sync_all_cwa_users(user_db)
except FileNotFoundError:
return jsonify({
"error": "Calibre-Web database is not available",
"message": "Verify app.db is mounted and readable at /auth/app.db.",
}), 503
except Exception as exc:
logger.error(f"Failed to sync CWA users: {exc}")
return jsonify({
"error": "Failed to sync users from Calibre-Web",
}), 500
return jsonify(
{
"error": "Calibre-Web database is not available",
"message": "Verify app.db is mounted and readable at /auth/app.db.",
}
), 503
except Exception:
logger.exception("Failed to sync CWA users")
return jsonify(
{
"error": "Failed to sync users from Calibre-Web",
}
), 500
message = (
f"Synced {summary['total']} CWA users "
@@ -381,17 +431,19 @@ def register_admin_routes(app: Flask, user_db: UserDB) -> None:
f"{summary.get('deleted', 0)} deleted)."
)
logger.info(message)
return jsonify({
"success": True,
"message": message,
**summary,
})
return jsonify(
{
"success": True,
"message": message,
**summary,
}
)
register_admin_settings_routes(app, user_db, _require_admin)
@app.route("/api/admin/users/<int:user_id>", methods=["DELETE"])
@_require_admin
def admin_delete_user(user_id):
def admin_delete_user(user_id: int) -> Response | tuple[Response, int]:
"""Delete a user."""
# Prevent self-deletion
if session.get("db_user_id") == user_id:
@@ -406,15 +458,17 @@ def register_admin_routes(app: Flask, user_db: UserDB) -> None:
user.get("oidc_subject"),
)
if auth_source == AUTH_SOURCE_CWA and auth_source == g.auth_mode:
return jsonify({
"error": f"Cannot delete active {auth_source.upper()} users",
"message": f"{auth_source.upper()} users are automatically re-provisioned on login.",
}), 400
return jsonify(
{
"error": f"Cannot delete active {auth_source.upper()} users",
"message": f"{auth_source.upper()} users are automatically re-provisioned on login.",
}
), 400
# Allow deleting the last local admin account.
# Auth mode resolution automatically falls back to "none" when no
# local password admin remains.
user_db.delete_user(user_id)
logger.info(f"Admin deleted user {user_id}: {user['username']}")
logger.info("Admin deleted user %s: %s", user_id, user["username"])
return jsonify({"success": True})
+90 -38
View File
@@ -1,6 +1,8 @@
"""Admin settings-introspection routes and settings validation helpers."""
from typing import Any, Callable
from __future__ import annotations
from typing import TYPE_CHECKING, Any
from flask import Flask, jsonify, request
@@ -10,17 +12,31 @@ from shelfmark.config.notifications_settings import (
normalize_notification_routes,
)
from shelfmark.config.users_settings import validate_search_preference_value
from shelfmark.core.config import config as app_config
from shelfmark.core.request_policy import parse_policy_mode, validate_policy_rules
from shelfmark.core.settings_registry import load_config_file
from shelfmark.core.user_settings_overrides import (
build_user_preferences_payload as _build_user_preferences_payload,
)
from shelfmark.core.user_settings_overrides import (
get_ordered_user_overridable_fields as _get_ordered_user_overridable_fields,
)
from shelfmark.core.user_settings_overrides import (
get_settings_registry as _get_settings_registry,
)
from shelfmark.core.user_db import UserDB
from shelfmark.core.request_policy import parse_policy_mode, validate_policy_rules
if TYPE_CHECKING:
from collections.abc import Callable
from flask.typing import ResponseReturnValue
from shelfmark.core.user_db import UserDB
def validate_user_settings(settings: dict[str, Any]) -> tuple[dict[str, Any], list[str]]:
def validate_user_settings(
settings: dict[str, Any],
) -> tuple[dict[str, Any], list[str]]:
"""Validate and normalize per-user settings overrides."""
settings_registry = _get_settings_registry()
field_map = settings_registry.get_settings_field_map()
overridable_map = settings_registry.get_user_overridable_fields()
@@ -38,10 +54,12 @@ def validate_user_settings(settings: dict[str, Any]) -> tuple[dict[str, Any], li
valid[key] = None
continue
if key in {"REQUEST_POLICY_DEFAULT_EBOOK", "REQUEST_POLICY_DEFAULT_AUDIOBOOK"}:
if parse_policy_mode(value) is None:
errors.append(f"Invalid policy mode for {key}: {value}")
continue
if (
key in {"REQUEST_POLICY_DEFAULT_EBOOK", "REQUEST_POLICY_DEFAULT_AUDIOBOOK"}
and parse_policy_mode(value) is None
):
errors.append(f"Invalid policy mode for {key}: {value}")
continue
if key == "REQUEST_POLICY_RULES":
normalized_rules, rule_errors = validate_policy_rules(value)
@@ -60,16 +78,16 @@ def validate_user_settings(settings: dict[str, Any]) -> tuple[dict[str, Any], li
)
if invalid_count:
errors.append(
(
f"Invalid value for {key}: found {invalid_count} invalid URL(s). "
"Use URL values with a valid scheme, e.g. discord://... or ntfys://..."
)
f"Invalid value for {key}: found {invalid_count} invalid URL(s). "
"Use URL values with a valid scheme, e.g. discord://... or ntfys://..."
)
continue
valid[key] = normalized_routes
continue
normalized_search_value, search_validation_error = validate_search_preference_value(key, value)
normalized_search_value, search_validation_error = validate_search_preference_value(
key, value
)
if search_validation_error:
errors.append(search_validation_error)
continue
@@ -78,10 +96,37 @@ def validate_user_settings(settings: dict[str, Any]) -> tuple[dict[str, Any], li
"METADATA_PROVIDER",
"METADATA_PROVIDER_AUDIOBOOK",
"DEFAULT_RELEASE_SOURCE",
"DEFAULT_RELEASE_SOURCE_AUDIOBOOK",
}:
valid[key] = normalized_search_value
continue
if key == "DOWNLOAD_TO_BROWSER_CONTENT_TYPES":
if not isinstance(value, list):
errors.append(f"Invalid value for {key}: must be a list")
continue
candidate_values = [
str(entry).strip().lower() for entry in value if str(entry).strip()
]
normalized_values: list[str] = []
has_invalid_value = False
for entry in candidate_values:
if entry not in {"book", "audiobook"}:
errors.append(
f"Invalid value for {key}: unsupported content type '{entry}'"
)
has_invalid_value = True
continue
if entry not in normalized_values:
normalized_values.append(entry)
if has_invalid_value:
continue
valid[key] = normalized_values
continue
valid[key] = value
return valid, errors
@@ -90,8 +135,9 @@ def validate_user_settings(settings: dict[str, Any]) -> tuple[dict[str, Any], li
def build_user_notification_test_response(
*,
user_id: int,
payload: Any,
payload: object,
) -> tuple[dict[str, Any], int]:
"""Build a notification test response using effective per-user routes."""
from shelfmark.core.config import config as app_config
routes_input = app_config.get("USER_NOTIFICATION_ROUTES", [], user_id=user_id)
@@ -109,36 +155,40 @@ def build_user_notification_test_response(
def register_admin_settings_routes(
app: Flask,
user_db: UserDB,
require_admin: Callable[[Callable[..., Any]], Callable[..., Any]],
require_admin: Callable[
[Callable[..., ResponseReturnValue]], Callable[..., ResponseReturnValue]
],
) -> None:
"""Register admin endpoints for user-specific settings and defaults."""
@app.route("/api/admin/download-defaults", methods=["GET"])
@require_admin
def admin_download_defaults():
config = load_config_file("downloads")
def admin_download_defaults() -> ResponseReturnValue:
defaults = {
key: ("" if (value := config.get(key, field.default)) is None else value)
key: ("" if (value := app_config.get(key, field.default)) is None else value)
for key, field in _get_ordered_user_overridable_fields("downloads")
}
security_config = load_config_file("security")
defaults["OIDC_ADMIN_GROUP"] = security_config.get("OIDC_ADMIN_GROUP", "")
defaults["OIDC_USE_ADMIN_GROUP"] = security_config.get("OIDC_USE_ADMIN_GROUP", True)
defaults["OIDC_AUTO_PROVISION"] = security_config.get("OIDC_AUTO_PROVISION", True)
defaults["OIDC_ADMIN_GROUP"] = app_config.get("OIDC_ADMIN_GROUP", "")
defaults["OIDC_USE_ADMIN_GROUP"] = app_config.get("OIDC_USE_ADMIN_GROUP", True)
defaults["OIDC_AUTO_PROVISION"] = app_config.get("OIDC_AUTO_PROVISION", True)
return jsonify(defaults)
@app.route("/api/admin/booklore-options", methods=["GET"])
@require_admin
def admin_booklore_options():
def admin_booklore_options() -> ResponseReturnValue:
from shelfmark.core import admin_routes
return jsonify({
"libraries": admin_routes.get_booklore_library_options(),
"paths": admin_routes.get_booklore_path_options(),
})
return jsonify(
{
"libraries": admin_routes.get_booklore_library_options(),
"paths": admin_routes.get_booklore_path_options(),
}
)
@app.route("/api/admin/users/<int:user_id>/delivery-preferences", methods=["GET"])
@require_admin
def admin_get_delivery_preferences(user_id):
def admin_get_delivery_preferences(user_id: int) -> ResponseReturnValue:
user = user_db.get_user(user_id=user_id)
if not user:
return jsonify({"error": "User not found"}), 404
@@ -152,7 +202,7 @@ def register_admin_settings_routes(
@app.route("/api/admin/users/<int:user_id>/search-preferences", methods=["GET"])
@require_admin
def admin_get_search_preferences(user_id):
def admin_get_search_preferences(user_id: int) -> ResponseReturnValue:
user = user_db.get_user(user_id=user_id)
if not user:
return jsonify({"error": "User not found"}), 404
@@ -166,7 +216,7 @@ def register_admin_settings_routes(
@app.route("/api/admin/users/<int:user_id>/notification-preferences", methods=["GET"])
@require_admin
def admin_get_notification_preferences(user_id):
def admin_get_notification_preferences(user_id: int) -> ResponseReturnValue:
user = user_db.get_user(user_id=user_id)
if not user:
return jsonify({"error": "User not found"}), 404
@@ -180,7 +230,7 @@ def register_admin_settings_routes(
@app.route("/api/admin/users/<int:user_id>/notification-preferences/test", methods=["POST"])
@require_admin
def admin_test_notification_preferences(user_id):
def admin_test_notification_preferences(user_id: int) -> ResponseReturnValue:
user = user_db.get_user(user_id=user_id)
if not user:
return jsonify({"error": "User not found"}), 404
@@ -194,7 +244,7 @@ def register_admin_settings_routes(
@app.route("/api/admin/settings/overrides-summary", methods=["GET"])
@require_admin
def admin_settings_overrides_summary():
def admin_settings_overrides_summary() -> ResponseReturnValue:
settings_registry = _get_settings_registry()
tab_name = (request.args.get("tab") or "downloads").strip()
@@ -213,11 +263,13 @@ def register_admin_settings_routes(
if key not in user_settings or user_settings[key] is None:
continue
entry = keys_payload.setdefault(key, {"count": 0, "users": []})
entry["users"].append({
"userId": user_record["id"],
"username": user_record["username"],
"value": user_settings[key],
})
entry["users"].append(
{
"userId": user_record["id"],
"username": user_record["username"],
"value": user_settings[key],
}
)
for summary in keys_payload.values():
summary["count"] = len(summary["users"])
@@ -226,7 +278,7 @@ def register_admin_settings_routes(
@app.route("/api/admin/users/<int:user_id>/effective-settings", methods=["GET"])
@require_admin
def admin_get_effective_settings(user_id):
def admin_get_effective_settings(user_id: int) -> ResponseReturnValue:
user = user_db.get_user(user_id=user_id)
if not user:
return jsonify({"error": "User not found"}), 404
+39 -14
View File
@@ -1,7 +1,14 @@
"""Authentication mode, auth-source normalization, and admin access policy helpers."""
from __future__ import annotations
import os
from typing import Any, Mapping
import sqlite3
from pathlib import Path
from typing import TYPE_CHECKING, Any, Protocol, TypeGuard
if TYPE_CHECKING:
from collections.abc import Mapping
AUTH_SOURCE_BUILTIN = "builtin"
AUTH_SOURCE_OIDC = "oidc"
@@ -17,7 +24,18 @@ AUTH_SOURCE_SET = frozenset(AUTH_SOURCES)
_ALWAYS_ADMIN_SETTINGS_TABS = frozenset({"security", "users"})
def has_local_password_admin(user_db: Any | None = None) -> bool:
class _UserDBWithAdminPassword(Protocol):
"""Minimal user DB surface needed for local-admin checks."""
def has_admin_with_password(self) -> bool: ...
def _has_admin_password_api(candidate: object) -> TypeGuard[_UserDBWithAdminPassword]:
"""Return True when *candidate* exposes the admin-password lookup we need."""
return callable(getattr(candidate, "has_admin_with_password", None))
def has_local_password_admin(user_db: object | None = None) -> bool:
"""Return True when at least one local admin with a password exists."""
try:
db = user_db
@@ -25,17 +43,19 @@ def has_local_password_admin(user_db: Any | None = None) -> bool:
from shelfmark.core.user_db import UserDB
config_root = os.environ.get("CONFIG_DIR", "/config")
db = UserDB(os.path.join(config_root, "users.db"))
db = UserDB(str(Path(config_root) / "users.db"))
db.initialize()
if not _has_admin_password_api(db):
return False
return db.has_admin_with_password()
except Exception:
except AttributeError, ImportError, OSError, RuntimeError, TypeError, ValueError, sqlite3.Error:
return False
def normalize_auth_source(
source: Any,
oidc_subject: Any = None,
source: object,
oidc_subject: object = None,
) -> str:
"""Resolve a stable auth source value from persisted fields."""
normalized = str(source or "").strip().lower()
@@ -48,7 +68,7 @@ def normalize_auth_source(
def determine_auth_mode(
security_config: Mapping[str, Any],
cwa_db_path: Any | None,
cwa_db_path: object | None,
*,
has_local_admin: bool = True,
) -> str:
@@ -76,21 +96,26 @@ def determine_auth_mode(
def load_active_auth_mode(
cwa_db_path: Any | None,
cwa_db_path: object | None,
*,
user_db: Any | None = None,
user_db: object | None = None,
) -> str:
"""Resolve active auth mode using current security config and runtime prerequisites."""
try:
from shelfmark.core.settings_registry import load_config_file
from shelfmark.core.config import config as app_config
security_config = load_config_file("security")
security_config = {
"AUTH_METHOD": app_config.get("AUTH_METHOD", "none"),
"PROXY_AUTH_USER_HEADER": app_config.get("PROXY_AUTH_USER_HEADER", ""),
"OIDC_DISCOVERY_URL": app_config.get("OIDC_DISCOVERY_URL", ""),
"OIDC_CLIENT_ID": app_config.get("OIDC_CLIENT_ID", ""),
}
return determine_auth_mode(
security_config,
cwa_db_path,
has_local_admin=has_local_password_admin(user_db),
)
except Exception:
except ImportError, OSError, RuntimeError, TypeError, ValueError, sqlite3.Error:
return "none"
@@ -104,7 +129,7 @@ def is_user_active_for_auth_mode(user: Mapping[str, Any], auth_mode: str) -> boo
def is_settings_or_onboarding_path(path: str) -> bool:
"""Return True when request path targets protected admin settings routes."""
return path.startswith("/api/settings") or path.startswith("/api/onboarding")
return path.startswith(("/api/settings", "/api/onboarding"))
def get_settings_tab_from_path(path: str) -> str | None:
@@ -112,7 +137,7 @@ def get_settings_tab_from_path(path: str) -> str | None:
if not path.startswith("/api/settings/"):
return None
suffix = path[len("/api/settings/"):]
suffix = path[len("/api/settings/") :]
if not suffix:
return None
+47 -39
View File
@@ -4,32 +4,37 @@ import threading
import time
from dataclasses import dataclass
from functools import wraps
from typing import Any, Callable, Dict, Optional, TypeVar
from typing import TYPE_CHECKING, ParamSpec, TypeVar, cast
from shelfmark.core.logger import setup_logger
if TYPE_CHECKING:
from collections.abc import Callable
logger = setup_logger(__name__)
T = TypeVar("T")
P = ParamSpec("P")
R = TypeVar("R")
@dataclass
class CacheEntry:
"""A cached value with expiration time."""
value: Any
value: object
expires_at: float
class CacheService:
"""Thread-safe in-memory cache with TTL support."""
def __init__(self, max_size: int = 1000):
def __init__(self, max_size: int = 1000) -> None:
"""Initialize cache with max_size entries before eviction."""
self._cache: Dict[str, CacheEntry] = {}
self._cache: dict[str, CacheEntry] = {}
self._lock = threading.Lock()
self._max_size = max_size
def get(self, key: str) -> Optional[Any]:
def get(self, key: str) -> object | None:
"""Get cached value if not expired."""
with self._lock:
entry = self._cache.get(key)
@@ -42,17 +47,14 @@ class CacheService:
return entry.value
def set(self, key: str, value: Any, ttl: int) -> None:
def set(self, key: str, value: object, ttl: int) -> None:
"""Cache value with TTL in seconds."""
with self._lock:
# Evict oldest entries if at capacity
if len(self._cache) >= self._max_size:
self._evict_oldest()
self._cache[key] = CacheEntry(
value=value,
expires_at=time.time() + ttl
)
self._cache[key] = CacheEntry(value=value, expires_at=time.time() + ttl)
def invalidate(self, key: str) -> bool:
"""Remove specific cache entry. Returns True if found."""
@@ -79,10 +81,7 @@ class CacheService:
"""Remove all expired entries. Returns count removed."""
with self._lock:
now = time.time()
expired_keys = [
key for key, entry in self._cache.items()
if entry.expires_at < now
]
expired_keys = [key for key, entry in self._cache.items() if entry.expires_at < now]
for key in expired_keys:
del self._cache[key]
return len(expired_keys)
@@ -94,21 +93,15 @@ class CacheService:
# Remove ~10% of entries, oldest first
entries_to_remove = max(1, len(self._cache) // 10)
sorted_entries = sorted(
self._cache.items(),
key=lambda x: x[1].expires_at
)
sorted_entries = sorted(self._cache.items(), key=lambda x: x[1].expires_at)
for key, _ in sorted_entries[:entries_to_remove]:
del self._cache[key]
def stats(self) -> Dict[str, int]:
def stats(self) -> dict[str, int]:
"""Get cache statistics (size, max_size)."""
with self._lock:
return {
"size": len(self._cache),
"max_size": self._max_size
}
return {"size": len(self._cache), "max_size": self._max_size}
# Global cache instance for metadata providers
@@ -120,23 +113,38 @@ def get_metadata_cache() -> CacheService:
return _metadata_cache
def cache_key(*args, **kwargs) -> str:
def cache_key(*args: object, **kwargs: object) -> str:
"""Generate cache key from arguments."""
parts = [str(arg) for arg in args]
parts.extend(f"{k}={v}" for k, v in sorted(kwargs.items()))
return ":".join(parts)
def _coerce_ttl_seconds(value: object, *, default: int) -> int:
"""Normalize cache TTL values read from config or decorator arguments."""
if isinstance(value, bool):
return default
if isinstance(value, int):
return value if value > 0 else default
if isinstance(value, str):
stripped = value.strip()
if stripped.isdigit():
parsed = int(stripped)
return parsed if parsed > 0 else default
return default
def cacheable(
ttl: Optional[int] = None,
ttl_key: Optional[str] = None,
ttl: int | None = None,
ttl_key: str | None = None,
ttl_default: int = 300,
key_prefix: str = ""
):
"""Decorator for caching function results. Use ttl (static) or ttl_key (from config)."""
def decorator(func: Callable[..., T]) -> Callable[..., T]:
key_prefix: str = "",
) -> Callable[[Callable[P, R]], Callable[P, R]]:
"""Cache function results with a static or config-backed TTL."""
def decorator(func: Callable[P, R]) -> Callable[P, R]:
@wraps(func)
def wrapper(*args, **kwargs) -> T:
def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
# Check if metadata caching is enabled
from shelfmark.core.config import config
@@ -148,7 +156,10 @@ def cacheable(
if ttl is not None:
effective_ttl = ttl
elif ttl_key:
effective_ttl = config.get(ttl_key, ttl_default)
effective_ttl = _coerce_ttl_seconds(
config.get(ttl_key, ttl_default),
default=ttl_default,
)
else:
effective_ttl = ttl_default
@@ -156,16 +167,12 @@ def cacheable(
# Skip 'self' argument if present (first arg of method)
cache_args = args[1:] if args and hasattr(args[0], func.__name__) else args
key = cache_key(
key_prefix or func.__name__,
*cache_args,
**kwargs
)
key = cache_key(key_prefix or func.__name__, *cache_args, **kwargs)
# Check cache
cached = _metadata_cache.get(key)
if cached is not None:
return cached
return cast("R", cached)
# Execute function and cache result
result = func(*args, **kwargs)
@@ -177,4 +184,5 @@ def cacheable(
return result
return wrapper
return decorator
+68 -56
View File
@@ -3,67 +3,85 @@
import os
import sqlite3
import time
from importlib import import_module
from pathlib import Path
from threading import Lock
from typing import Any, Dict, Optional
from typing import TYPE_CHECKING, Any, Self
if TYPE_CHECKING:
from types import ModuleType
from shelfmark.core.user_db import UserDB
# Import lazily to avoid circular imports
_registry_module = None
_env_module = None
_user_db_module = None
_SETTINGS_REFRESH_COOLDOWN_SECONDS = 0.05
def _get_registry():
def _get_registry() -> ModuleType:
"""Lazy import of settings registry to avoid circular imports."""
global _registry_module
if _registry_module is None:
from shelfmark.core import settings_registry
_registry_module = settings_registry
return _registry_module
def _get_env():
def _get_env() -> ModuleType:
"""Lazy import of env module for fallback values."""
global _env_module
if _env_module is None:
from shelfmark.config import env
_env_module = env
return _env_module
def _get_user_db_module():
def _get_user_db_module() -> type[UserDB]:
"""Lazy import of user DB module to avoid optional dependency loops."""
global _user_db_module
if _user_db_module is None:
from shelfmark.core.user_db import UserDB
_user_db_module = UserDB
return _user_db_module
class Config:
"""
Dynamic configuration singleton that provides live settings access.
"""Dynamic configuration singleton that provides live settings access.
Settings are resolved with priority: ENV var > config file > default.
Values are cached for performance and can be refreshed when settings change.
"""
_instance: Optional['Config'] = None
_instance: Self | None = None
_lock = Lock()
def __new__(cls) -> 'Config':
def __new__(cls) -> Self:
"""Return the shared configuration singleton instance."""
if cls._instance is None:
with cls._lock:
if cls._instance is None:
cls._instance = super().__new__(cls)
cls._instance._initialized = False
return cls._instance
instance = cls._instance
if instance is None:
msg = "Config singleton failed to initialize"
raise RuntimeError(msg)
return instance
def __init__(self):
def __init__(self) -> None:
"""Initialize caches and backing stores for the singleton."""
if self._initialized:
return
self._cache: Dict[str, Any] = {}
self._field_map: Dict[str, tuple] = {} # key -> (field, tab_name)
self._cache: dict[str, Any] = {}
self._field_map: dict[str, tuple] = {} # key -> (field, tab_name)
self._cache_lock = Lock()
self._user_settings_cache: Dict[int, Dict[str, Any]] = {}
self._user_settings_cache: dict[int, dict[str, Any]] = {}
self._user_settings_cache_lock = Lock()
self._user_db = None
self._user_db_load_attempted = False
@@ -85,10 +103,12 @@ class Config:
# Ensure all settings modules are imported before loading
# This handles cases where config is accessed before settings are registered
try:
import shelfmark.config.settings # noqa: F401 - main app settings
import shelfmark.config.notifications_settings # noqa: F401 - notifications settings
import shelfmark.release_sources # noqa: F401 - plugin settings
import shelfmark.metadata_providers # noqa: F401 - plugin settings
import_module("shelfmark.config.notifications_settings")
import_module("shelfmark.config.security")
import_module("shelfmark.config.settings")
import_module("shelfmark.config.users_settings")
import_module("shelfmark.metadata_providers")
import_module("shelfmark.release_sources")
except ImportError:
pass
@@ -96,7 +116,7 @@ class Config:
# On first load, sync ENV values to config files
# This ensures ENV values persist even if ENV vars are later removed
if not hasattr(self, '_env_synced'):
if not hasattr(self, "_env_synced"):
registry.sync_env_to_config()
self._env_synced = True
@@ -104,24 +124,14 @@ class Config:
self._field_map.clear()
self._cache.clear()
for tab in registry.get_all_settings_tabs():
for field in tab.fields:
# Skip action buttons and headings - they don't have values
if isinstance(field, (registry.ActionButton, registry.HeadingField)):
continue
key = field.key
self._field_map[key] = (field, tab.name)
# Load current value
value = registry.get_setting_value(field, tab.name)
self._cache[key] = value
for key, (field, tab_name) in registry.get_settings_field_map().items():
self._field_map[key] = (field, tab_name)
self._cache[key] = registry.get_setting_value(field, tab_name)
self._loaded = True
def refresh(self, force: bool = False) -> None:
"""
Refresh all cached settings from config files.
def refresh(self, *, force: bool = False) -> None:
"""Refresh all cached settings from config files.
Call this after settings are updated via the UI to ensure
the config singleton reflects the new values.
@@ -132,7 +142,7 @@ class Config:
(e.g. after a settings write).
"""
now = time.monotonic()
if not force and (now - self._last_refresh_time) < 0.05:
if not force and (now - self._last_refresh_time) < _SETTINGS_REFRESH_COOLDOWN_SECONDS:
return
with self._cache_lock:
@@ -144,7 +154,7 @@ class Config:
self._user_db_load_attempted = False
self._last_refresh_time = time.monotonic()
def _get_user_db(self):
def _get_user_db(self) -> UserDB | None:
"""Get or initialize a UserDB handle if available."""
if self._user_db is not None:
return self._user_db
@@ -154,16 +164,17 @@ class Config:
self._user_db_load_attempted = True
try:
user_db_cls = _get_user_db_module()
db_path = os.path.join(os.environ.get("CONFIG_DIR", "/config"), "users.db")
db_path = str(Path(os.environ.get("CONFIG_DIR", "/config")) / "users.db")
user_db = user_db_cls(db_path)
user_db.initialize()
self._user_db = user_db
return self._user_db
except Exception:
except ImportError, OSError, sqlite3.Error:
# Multi-user support is optional; fall back to global config when unavailable.
return None
else:
self._user_db = user_db
return self._user_db
def _get_user_settings(self, user_id: int) -> Dict[str, Any]:
def _get_user_settings(self, user_id: int) -> dict[str, Any]:
"""Get cached per-user settings from user DB."""
with self._user_settings_cache_lock:
if user_id in self._user_settings_cache:
@@ -175,7 +186,7 @@ class Config:
try:
settings = user_db.get_user_settings(user_id)
except (sqlite3.OperationalError, OSError, ValueError, TypeError):
except sqlite3.OperationalError, OSError, ValueError, TypeError:
return {}
if not isinstance(settings, dict):
@@ -185,14 +196,13 @@ class Config:
self._user_settings_cache[user_id] = settings
return settings
def _get_user_override(self, user_id: int, key: str) -> Any:
def _get_user_override(self, user_id: int, key: str) -> object:
"""Get a user override for a specific key."""
user_settings = self._get_user_settings(user_id)
return user_settings.get(key)
def get(self, key: str, default: Any = None, user_id: Optional[int] = None) -> Any:
"""
Get a setting value by key.
def get(self, key: str, default: object = None, user_id: int | None = None) -> object:
"""Get a setting value by key.
Args:
key: The setting key (e.g., 'MAX_RETRY')
@@ -201,6 +211,7 @@ class Config:
Returns:
The setting value, or default if not found
"""
self._ensure_loaded()
@@ -220,15 +231,15 @@ class Config:
return self._cache.get(key, default)
def __getattr__(self, name: str) -> Any:
"""
Allow attribute-style access to settings.
def __getattr__(self, name: str) -> object:
"""Allow attribute-style access to settings.
Example: config.MAX_RETRY instead of config.get('MAX_RETRY')
"""
# Avoid recursion for internal attributes
if name.startswith('_'):
raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
if name.startswith("_"):
msg = f"'{type(self).__name__}' object has no attribute '{name}'"
raise AttributeError(msg)
self._ensure_loaded()
@@ -241,17 +252,18 @@ class Config:
if hasattr(env, name):
return getattr(env, name)
raise AttributeError(f"Setting '{name}' not found in config or env")
msg = f"Setting '{name}' not found in config or env"
raise AttributeError(msg)
def is_from_env(self, key: str) -> bool:
"""
Check if a setting's value comes from an environment variable.
"""Check if a setting's value comes from an environment variable.
Args:
key: The setting key
Returns:
True if the value is set via ENV var, False otherwise
"""
self._ensure_loaded()
@@ -262,12 +274,12 @@ class Config:
registry = _get_registry()
return registry.is_value_from_env(field)
def get_all(self) -> Dict[str, Any]:
"""
Get all cached settings as a dictionary.
def get_all(self) -> dict[str, Any]:
"""Get all cached settings as a dictionary.
Returns:
Dict of all setting keys to their current values
"""
self._ensure_loaded()
return dict(self._cache)
+16 -8
View File
@@ -2,16 +2,20 @@
from __future__ import annotations
from typing import Any, Iterable
from typing import TYPE_CHECKING, Any
from shelfmark.core.auth_modes import AUTH_SOURCE_CWA, normalize_auth_source
from shelfmark.core.external_user_linking import upsert_external_user
from shelfmark.core.user_db import UserDB
if TYPE_CHECKING:
from collections.abc import Iterable
from shelfmark.core.user_db import UserDB
_CWA_ALIAS_SUFFIX = "__cwa"
def _normalize_email(value: Any) -> str | None:
def _normalize_email(value: object) -> str | None:
if value is None:
return None
email = str(value).strip()
@@ -40,7 +44,8 @@ def upsert_cwa_user(
context=context,
)
if user is None:
raise RuntimeError("Unexpected CWA user sync result: no user returned")
msg = "Unexpected CWA user sync result: no user returned"
raise RuntimeError(msg)
return user, action
@@ -73,10 +78,13 @@ def sync_cwa_users_from_rows(
deleted = 0
for existing_user in user_db.list_users():
if normalize_auth_source(
existing_user.get("auth_source"),
existing_user.get("oidc_subject"),
) != AUTH_SOURCE_CWA:
if (
normalize_auth_source(
existing_user.get("auth_source"),
existing_user.get("oidc_subject"),
)
!= AUTH_SOURCE_CWA
):
continue
existing_id = int(existing_user.get("id") or 0)
+157 -30
View File
@@ -2,15 +2,20 @@
from __future__ import annotations
import os
import json
import sqlite3
import threading
from datetime import datetime, timezone
from typing import Any
from datetime import UTC, datetime
from pathlib import Path
from typing import Any, SupportsIndex, SupportsInt, TypeGuard
from shelfmark.core.logger import setup_logger
from shelfmark.core.models import TERMINAL_QUEUE_STATUSES
from shelfmark.core.request_helpers import normalize_optional_positive_int, normalize_optional_text, now_utc_iso
from shelfmark.core.request_helpers import (
normalize_optional_positive_int,
normalize_optional_text,
now_utc_iso,
)
logger = setup_logger(__name__)
@@ -20,40 +25,64 @@ ACTIVE_DOWNLOAD_STATUS = "active"
VALID_ORIGINS = frozenset({"direct", "requested"})
def _normalize_task_id(task_id: Any) -> str:
def _is_convertible_to_int(
value: object,
) -> TypeGuard[str | bytes | bytearray | SupportsInt | SupportsIndex]:
"""Return True when *value* can be safely passed to ``int``."""
return (
isinstance(value, (str, bytes, bytearray))
or hasattr(value, "__int__")
or hasattr(value, "__index__")
)
def _coerce_int_value(value: object) -> int:
"""Normalize int-like values and raise TypeError for unsupported inputs."""
if isinstance(value, bool) or not _is_convertible_to_int(value):
msg = "limit must be an integer"
raise TypeError(msg)
return int(value)
def _normalize_task_id(task_id: object) -> str:
normalized = normalize_optional_text(task_id)
if normalized is None:
raise ValueError("task_id must be a non-empty string")
msg = "task_id must be a non-empty string"
raise ValueError(msg)
return normalized
def _normalize_origin(origin: Any) -> str:
def _normalize_origin(origin: object) -> str:
normalized = normalize_optional_text(origin)
if normalized is None:
return "direct"
lowered = normalized.lower()
if lowered not in VALID_ORIGINS:
raise ValueError("origin must be one of: direct, requested")
msg = "origin must be one of: direct, requested"
raise ValueError(msg)
return lowered
def _normalize_final_status(final_status: Any) -> str:
def _normalize_final_status(final_status: object) -> str:
normalized = normalize_optional_text(final_status)
if normalized is None:
raise ValueError("final_status must be a non-empty string")
msg = "final_status must be a non-empty string"
raise ValueError(msg)
lowered = normalized.lower()
if lowered not in VALID_TERMINAL_STATUSES:
raise ValueError("final_status must be one of: complete, error, cancelled")
msg = "final_status must be one of: complete, error, cancelled"
raise ValueError(msg)
return lowered
def _normalize_limit(value: Any, *, default: int, minimum: int, maximum: int) -> int:
def _normalize_limit(value: object, *, default: int, minimum: int, maximum: int) -> int:
if value is None:
return default
try:
parsed = int(value)
parsed = _coerce_int_value(value)
except (TypeError, ValueError) as exc:
raise ValueError("limit must be an integer") from exc
msg = "limit must be an integer"
raise ValueError(msg) from exc
if parsed < minimum:
return minimum
if parsed > maximum:
@@ -64,7 +93,8 @@ def _normalize_limit(value: Any, *, default: int, minimum: int, maximum: int) ->
class DownloadHistoryService:
"""Service for persisted canonical download activity rows."""
def __init__(self, db_path: str):
def __init__(self, db_path: str) -> None:
"""Initialize the service with the SQLite history database path."""
self._db_path = db_path
self._lock = threading.Lock()
@@ -74,23 +104,99 @@ class DownloadHistoryService:
conn.execute("PRAGMA foreign_keys = ON")
return conn
@staticmethod
def _row_to_dict(row: sqlite3.Row | None) -> dict[str, Any] | None:
return dict(row) if row is not None else None
@classmethod
def _normalize_row_dict(cls, row: dict[str, Any] | None) -> dict[str, Any] | None:
if row is None:
return None
normalized = dict(row)
normalized["retry_payload"] = cls._deserialize_retry_payload(
normalized.get("retry_payload")
)
return normalized
@classmethod
def _row_to_dict(cls, row: sqlite3.Row | None) -> dict[str, Any] | None:
return cls._normalize_row_dict(dict(row) if row is not None else None)
@staticmethod
def _to_item_key(task_id: str) -> str:
return f"download:{task_id}"
@staticmethod
def _resolve_existing_download_path(value: Any) -> str | None:
def _resolve_existing_download_path(value: object) -> str | None:
normalized = normalize_optional_text(value)
if normalized is None:
return None
return normalized if os.path.exists(normalized) else None
return normalized if Path(normalized).exists() else None
@staticmethod
def _serialize_retry_payload(payload: object) -> str | None:
if payload is None:
return None
try:
return json.dumps(payload, ensure_ascii=False, separators=(",", ":"))
except (TypeError, ValueError) as exc:
msg = "retry_payload must be JSON-serializable"
raise ValueError(msg) from exc
@staticmethod
def _deserialize_retry_payload(value: object) -> dict[str, Any] | None:
if isinstance(value, dict):
return dict(value)
normalized = normalize_optional_text(value)
if normalized is None:
return None
try:
parsed = json.loads(normalized)
except json.JSONDecodeError:
return None
return parsed if isinstance(parsed, dict) else None
@staticmethod
def _has_staged_retry_source(retry_payload: dict[str, Any]) -> bool:
staged_path = retry_payload.get("staged_path")
normalized_staged_path = normalize_optional_text(staged_path)
if normalized_staged_path is None:
return False
return Path(normalized_staged_path).exists()
@staticmethod
def _can_retry_without_staged_source(retry_payload: dict[str, Any]) -> bool:
return bool(retry_payload.get("can_retry_without_staged_source", True))
@staticmethod
def is_retry_available(row: dict[str, Any]) -> bool:
"""Return whether a persisted download row can be retried."""
final_status = (
str(row.get("retry_final_status") or row.get("final_status") or "").strip().lower()
)
retry_payload = DownloadHistoryService._deserialize_retry_payload(row.get("retry_payload"))
if retry_payload is None:
return False
has_staged_retry_source = DownloadHistoryService._has_staged_retry_source(retry_payload)
can_retry_without_staged_source = DownloadHistoryService._can_retry_without_staged_source(
retry_payload
)
request_id = normalize_optional_positive_int(row.get("request_id"), "request_id")
if request_id is None:
if final_status in {ACTIVE_DOWNLOAD_STATUS, "cancelled"}:
return can_retry_without_staged_source
if final_status == "error":
return has_staged_retry_source or can_retry_without_staged_source
return False
if final_status in {ACTIVE_DOWNLOAD_STATUS, "cancelled"}:
return can_retry_without_staged_source
if final_status != "error":
return False
return has_staged_retry_source
@staticmethod
def to_download_payload(row: dict[str, Any]) -> dict[str, Any]:
"""Build the sidebar/history download payload for a persisted row."""
return {
"id": row.get("task_id"),
"title": row.get("title"),
@@ -102,15 +208,18 @@ class DownloadHistoryService:
"source": row.get("source"),
"source_display_name": row.get("source_display_name"),
"status_message": row.get("status_message"),
"download_path": DownloadHistoryService._resolve_existing_download_path(row.get("download_path")),
"download_path": DownloadHistoryService._resolve_existing_download_path(
row.get("download_path")
),
"added_time": DownloadHistoryService._iso_to_epoch(row.get("queued_at")),
"user_id": row.get("user_id"),
"username": row.get("username"),
"request_id": row.get("request_id"),
"retry_available": DownloadHistoryService.is_retry_available(row),
}
@staticmethod
def _iso_to_epoch(value: Any) -> float | None:
def _iso_to_epoch(value: object) -> float | None:
if not isinstance(value, str) or not value.strip():
return None
normalized = value.strip().replace("Z", "+00:00")
@@ -119,11 +228,12 @@ class DownloadHistoryService:
except ValueError:
return None
if parsed.tzinfo is None:
parsed = parsed.replace(tzinfo=timezone.utc)
parsed = parsed.replace(tzinfo=UTC)
return parsed.timestamp()
@classmethod
def to_history_row(cls, row: dict[str, Any], *, dismissed_at: str) -> dict[str, Any]:
"""Build the activity-history payload for a persisted download row."""
task_id = str(row.get("task_id") or "").strip()
item_key = cls._to_item_key(task_id)
download_payload = cls.to_download_payload(row)
@@ -158,11 +268,12 @@ class DownloadHistoryService:
source_display_name: str | None,
title: str,
author: str | None,
format: str | None,
file_format: str | None,
size: str | None,
preview: str | None,
content_type: str | None,
origin: str,
retry_payload: dict[str, Any] | None = None,
) -> None:
"""Record a download at queue time with final_status='active'.
@@ -175,11 +286,14 @@ class DownloadHistoryService:
normalized_request_id = normalize_optional_positive_int(request_id, "request_id")
normalized_source = normalize_optional_text(source)
if normalized_source is None:
raise ValueError("source must be a non-empty string")
msg = "source must be a non-empty string"
raise ValueError(msg)
normalized_title = normalize_optional_text(title)
if normalized_title is None:
raise ValueError("title must be a non-empty string")
msg = "title must be a non-empty string"
raise ValueError(msg)
normalized_origin = _normalize_origin(origin)
normalized_retry_payload = self._serialize_retry_payload(retry_payload)
recorded_at = now_utc_iso()
with self._lock:
@@ -192,14 +306,15 @@ class DownloadHistoryService:
source, source_display_name,
title, author, format, size, preview, content_type,
origin, final_status,
status_message, download_path,
status_message, download_path, retry_payload,
queued_at, terminal_at
)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'active', NULL, NULL, ?, ?)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'active', NULL, NULL, ?, ?, ?)
ON CONFLICT(task_id) DO UPDATE SET
final_status = 'active',
status_message = NULL,
download_path = NULL,
retry_payload = excluded.retry_payload,
terminal_at = ?
""",
(
@@ -211,11 +326,12 @@ class DownloadHistoryService:
normalize_optional_text(source_display_name),
normalized_title,
normalize_optional_text(author),
normalize_optional_text(format),
normalize_optional_text(file_format),
normalize_optional_text(size),
normalize_optional_text(preview),
normalize_optional_text(content_type),
normalized_origin,
normalized_retry_payload,
recorded_at,
recorded_at,
recorded_at,
@@ -232,12 +348,14 @@ class DownloadHistoryService:
final_status: str,
status_message: str | None = None,
download_path: str | None = None,
retry_payload: dict[str, Any] | None = None,
) -> None:
"""Update an existing download row to its terminal state."""
normalized_task_id = _normalize_task_id(task_id)
normalized_final_status = _normalize_final_status(final_status)
normalized_status_message = normalize_optional_text(status_message)
normalized_download_path = normalize_optional_text(download_path)
normalized_retry_payload = self._serialize_retry_payload(retry_payload)
effective_terminal_at = now_utc_iso()
with self._lock:
@@ -249,6 +367,7 @@ class DownloadHistoryService:
SET final_status = ?,
status_message = ?,
download_path = ?,
retry_payload = COALESCE(?, retry_payload),
terminal_at = ?
WHERE task_id = ? AND final_status = 'active'
""",
@@ -256,6 +375,7 @@ class DownloadHistoryService:
normalized_final_status,
normalized_status_message,
normalized_download_path,
normalized_retry_payload,
effective_terminal_at,
normalized_task_id,
),
@@ -271,6 +391,7 @@ class DownloadHistoryService:
conn.close()
def get_by_task_id(self, task_id: str) -> dict[str, Any] | None:
"""Return a persisted download row for the given task id."""
normalized_task_id = _normalize_task_id(task_id)
conn = self._connect()
try:
@@ -288,6 +409,7 @@ class DownloadHistoryService:
user_id: int | None,
limit: int = 200,
) -> list[dict[str, Any]]:
"""Return recent persisted download rows, optionally scoped to one user."""
normalized_user_id = normalize_optional_positive_int(user_id, "user_id")
normalized_limit = _normalize_limit(limit, default=200, minimum=1, maximum=1000)
query = "SELECT * FROM download_history"
@@ -301,6 +423,11 @@ class DownloadHistoryService:
conn = self._connect()
try:
rows = conn.execute(query, params).fetchall()
return [dict(row) for row in rows]
result: list[dict[str, Any]] = []
for row in rows:
normalized = self._normalize_row_dict(dict(row))
if normalized is not None:
result.append(normalized)
return result
finally:
conn.close()
+54 -29
View File
@@ -3,11 +3,13 @@
from __future__ import annotations
import re
from typing import Any, Literal
from typing import TYPE_CHECKING, Any, Literal
from shelfmark.core.auth_modes import normalize_auth_source
from shelfmark.core.logger import setup_logger
from shelfmark.core.user_db import UserDB
if TYPE_CHECKING:
from shelfmark.core.user_db import UserDB
UNSET = object()
@@ -21,18 +23,18 @@ MatchReason = Literal[
logger = setup_logger(__name__)
def _normalize_username(value: Any) -> str:
def _normalize_username(value: object) -> str:
return str(value or "").strip()
def _normalize_email(value: Any) -> str | None:
def _normalize_email(value: object) -> str | None:
if value is None:
return None
email = str(value).strip()
return email or None
def _normalize_display_name(value: Any) -> str | None:
def _normalize_display_name(value: object) -> str | None:
if value is None:
return None
name = str(value).strip()
@@ -43,11 +45,13 @@ def _email_key(value: str | None) -> str:
return (value or "").strip().lower()
def _normalize_role(value: Any) -> str:
def _normalize_role(value: object) -> str:
return "admin" if str(value or "").strip().lower() == "admin" else "user"
def _get_by_subject(user_db: UserDB, subject_field: str | None, subject: str | None) -> dict[str, Any] | None:
def _get_by_subject(
user_db: UserDB, subject_field: str | None, subject: str | None
) -> dict[str, Any] | None:
if not subject_field or not subject:
return None
if subject_field == "oidc_subject":
@@ -56,6 +60,7 @@ def _get_by_subject(user_db: UserDB, subject_field: str | None, subject: str | N
def find_unique_user_by_email(user_db: UserDB, email: str | None) -> dict[str, Any] | None:
"""Return the unique local user matching an email address, if any."""
key = _email_key(_normalize_email(email))
if not key:
return None
@@ -83,10 +88,14 @@ def find_external_user_match(
return by_subject, "subject_match"
by_username = user_db.get_user(username=normalized_username)
if by_username and normalize_auth_source(
by_username.get("auth_source"),
by_username.get("oidc_subject"),
) == auth_source:
if (
by_username
and normalize_auth_source(
by_username.get("auth_source"),
by_username.get("oidc_subject"),
)
== auth_source
):
return by_username, "existing_source_username_match"
if allow_email_link:
@@ -133,7 +142,8 @@ def _find_existing_alias_user(
) -> dict[str, Any] | None:
pattern = re.compile(rf"^{re.escape(alias_base)}(?:_\d+)?$")
candidates = [
user for user in user_db.list_users()
user
for user in user_db.list_users()
if pattern.match(str(user.get("username") or ""))
and normalize_auth_source(user.get("auth_source"), user.get("oidc_subject")) == auth_source
]
@@ -158,7 +168,11 @@ def _resolve_create_username(
return None, existing, "username_collision_takeover"
if strategy == "suffix":
return _next_suffix_username(user_db, requested_username), None, "username_collision_suffix"
return (
_next_suffix_username(user_db, requested_username),
None,
"username_collision_suffix",
)
alias_base = f"{requested_username}{alias_suffix}"
alias_existing = _find_existing_alias_user(
@@ -197,7 +211,8 @@ def upsert_external_user(
"""
normalized_username = _normalize_username(username)
if not normalized_username:
raise ValueError("External username is required")
msg = "External username is required"
raise ValueError(msg)
normalized_email = _normalize_email(email) if email is not UNSET else None
normalized_display_name = (
@@ -227,18 +242,22 @@ def upsert_external_user(
user_db.update_user(matched["id"], **updates)
mapped = user_db.get_user(user_id=matched["id"]) or matched
logger.info(
"External user mapped to existing Shelfmark user "
f"(source={auth_source}, context={context or 'unspecified'}, reason={match_reason}, "
f"external_username={normalized_username}, shelfmark_user_id={mapped['id']}, "
f"shelfmark_username={mapped['username']})"
"External user mapped to existing Shelfmark user (source=%s, context=%s, reason=%s, external_username=%s, shelfmark_user_id=%s, shelfmark_username=%s)",
auth_source,
context or "unspecified",
match_reason,
normalized_username,
mapped["id"],
mapped["username"],
)
return mapped, "updated"
if not allow_create:
logger.info(
"External user could not be mapped and creation is disabled "
f"(source={auth_source}, context={context or 'unspecified'}, "
f"external_username={normalized_username})"
"External user could not be mapped and creation is disabled (source=%s, context=%s, external_username=%s)",
auth_source,
context or "unspecified",
normalized_username,
)
return None, "not_found"
@@ -254,10 +273,13 @@ def upsert_external_user(
user_db.update_user(takeover_target["id"], **updates)
mapped = user_db.get_user(user_id=takeover_target["id"]) or takeover_target
logger.info(
"External user mapped to existing Shelfmark user "
f"(source={auth_source}, context={context or 'unspecified'}, reason={create_reason}, "
f"external_username={normalized_username}, shelfmark_user_id={mapped['id']}, "
f"shelfmark_username={mapped['username']})"
"External user mapped to existing Shelfmark user (source=%s, context=%s, reason=%s, external_username=%s, shelfmark_user_id=%s, shelfmark_username=%s)",
auth_source,
context or "unspecified",
create_reason,
normalized_username,
mapped["id"],
mapped["username"],
)
return mapped, "updated"
@@ -275,9 +297,12 @@ def upsert_external_user(
created = user_db.create_user(**create_kwargs)
logger.info(
"External user created Shelfmark user "
f"(source={auth_source}, context={context or 'unspecified'}, reason={create_reason}, "
f"external_username={normalized_username}, shelfmark_user_id={created['id']}, "
f"shelfmark_username={created['username']})"
"External user created Shelfmark user (source=%s, context=%s, reason=%s, external_username=%s, shelfmark_user_id=%s, shelfmark_username=%s)",
auth_source,
context or "unspecified",
create_reason,
normalized_username,
created["id"],
created["username"],
)
return created, "created"
+162 -112
View File
@@ -1,34 +1,40 @@
"""Disk-based image cache with LRU eviction."""
import ipaddress
import json
import os
import socket
import threading
import time
from http import HTTPStatus
from io import BytesIO
from pathlib import Path
from typing import Any, Dict, Optional, Tuple
from typing import TYPE_CHECKING, Any
from urllib.parse import urlparse
import requests
from shelfmark.core.logger import setup_logger
from shelfmark.core.request_helpers import coerce_int
from shelfmark.download.network import get_ssl_verify
if TYPE_CHECKING:
from pathlib import Path
logger = setup_logger(__name__)
# Image type detection via magic bytes
IMAGE_SIGNATURES = {
b'\xff\xd8\xff': ('image/jpeg', 'jpg'),
b'\x89PNG\r\n\x1a\n': ('image/png', 'png'),
b'GIF87a': ('image/gif', 'gif'),
b'GIF89a': ('image/gif', 'gif'),
b'RIFF': ('image/webp', 'webp'), # WebP starts with RIFF
b"\xff\xd8\xff": ("image/jpeg", "jpg"),
b"\x89PNG\r\n\x1a\n": ("image/png", "png"),
b"GIF87a": ("image/gif", "gif"),
b"GIF89a": ("image/gif", "gif"),
b"RIFF": ("image/webp", "webp"), # WebP starts with RIFF
}
# HTTP headers for image fetching
FETCH_HEADERS = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/129.0.0.0 Safari/537.36',
'Accept': 'image/webp,image/apng,image/*,*/*;q=0.8',
'Accept-Language': 'en-US,en;q=0.5',
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/129.0.0.0 Safari/537.36",
"Accept": "image/webp,image/apng,image/*,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.5",
}
# Maximum image size to fetch (5 MB)
@@ -41,8 +47,11 @@ NEGATIVE_CACHE_TTL = 3600
# Short enough to retry soon, long enough to prevent spam during one page view
TRANSIENT_CACHE_TTL = 60
_MIN_WEBP_HEADER_LENGTH = 12
HTTP_NOT_FOUND = HTTPStatus.NOT_FOUND
def _detect_image_type(data: bytes) -> Optional[Tuple[str, str]]:
def _detect_image_type(data: bytes) -> tuple[str, str] | None:
"""Detect image type from magic bytes.
Args:
@@ -50,14 +59,15 @@ def _detect_image_type(data: bytes) -> Optional[Tuple[str, str]]:
Returns:
Tuple of (content_type, extension) or None if not recognized
"""
for signature, (content_type, ext) in IMAGE_SIGNATURES.items():
if data.startswith(signature):
return content_type, ext
# Special case for WebP - check for WEBP after RIFF
if data.startswith(b'RIFF') and len(data) > 12 and data[8:12] == b'WEBP':
return 'image/webp', 'webp'
if data.startswith(b"RIFF") and len(data) > _MIN_WEBP_HEADER_LENGTH and data[8:12] == b"WEBP":
return "image/webp", "webp"
return None
@@ -65,20 +75,21 @@ def _detect_image_type(data: bytes) -> Optional[Tuple[str, str]]:
class ImageCacheService:
"""Persistent image cache with LRU eviction and TTL support."""
def __init__(self, cache_dir: Path, max_size_mb: int = 500, ttl_seconds: int = 0):
def __init__(self, cache_dir: Path, max_size_mb: int = 500, ttl_seconds: int = 0) -> None:
"""Initialize the image cache.
Args:
cache_dir: Directory to store cached images
max_size_mb: Maximum cache size in megabytes
ttl_seconds: Time-to-live in seconds (0 = forever)
"""
self.cache_dir = cache_dir
self.max_size_bytes = max_size_mb * 1024 * 1024
self.ttl_seconds = ttl_seconds
self.index_path = cache_dir / "cache_index.json"
self._lock = threading.RLock()
self._index: Dict[str, Dict[str, Any]] = {}
self._index: dict[str, dict[str, Any]] = {}
# Stats tracking
self._hits = 0
@@ -98,9 +109,9 @@ class ImageCacheService:
return
try:
with open(self.index_path, 'r') as f:
with self.index_path.open() as f:
self._index = json.load(f)
except (json.JSONDecodeError, IOError):
except OSError, json.JSONDecodeError:
self._index = {}
def _sync_index_with_files(self) -> None:
@@ -110,12 +121,12 @@ class ImageCacheService:
- Removes entries for files that no longer exist (non-negative only)
- Preserves negative cache entries (they have no files)
"""
image_extensions = {'.jpg', '.jpeg', '.png', '.gif', '.webp'}
image_extensions = {".jpg", ".jpeg", ".png", ".gif", ".webp"}
added_count = 0
removed_count = 0
# Build set of files that exist on disk
existing_files: Dict[str, Path] = {}
existing_files: dict[str, Path] = {}
for file_path in self.cache_dir.iterdir():
if not file_path.is_file():
continue
@@ -128,31 +139,31 @@ class ImageCacheService:
if cache_id in self._index:
continue
ext = file_path.suffix.lstrip('.')
ext = file_path.suffix.lstrip(".")
stat = file_path.stat()
# Detect content type
try:
with open(file_path, 'rb') as f:
with file_path.open("rb") as f:
header = f.read(16)
detected = _detect_image_type(header)
content_type = detected[0] if detected else f'image/{ext}'
except IOError:
content_type = f'image/{ext}'
content_type = detected[0] if detected else f"image/{ext}"
except OSError:
content_type = f"image/{ext}"
self._index[cache_id] = {
'ext': ext,
'content_type': content_type,
'size': stat.st_size,
'cached_at': stat.st_mtime,
'accessed_at': stat.st_mtime,
"ext": ext,
"content_type": content_type,
"size": stat.st_size,
"cached_at": stat.st_mtime,
"accessed_at": stat.st_mtime,
}
added_count += 1
# Remove index entries for missing files (skip negative cache entries)
stale_entries = []
for cache_id, entry in self._index.items():
if entry.get('negative', False):
if entry.get("negative", False):
continue # Negative entries don't have files
if cache_id not in existing_files:
stale_entries.append(cache_id)
@@ -168,39 +179,39 @@ class ImageCacheService:
"""Save cache index to disk."""
try:
# Write to temp file first, then rename for atomicity
temp_path = self.index_path.with_suffix('.tmp')
with open(temp_path, 'w') as f:
temp_path = self.index_path.with_suffix(".tmp")
with temp_path.open("w") as f:
json.dump(self._index, f)
temp_path.rename(self.index_path)
except IOError:
except OSError:
pass
def _get_image_path(self, cache_id: str, ext: str) -> Path:
"""Get the file path for a cached image."""
return self.cache_dir / f"{cache_id}.{ext}"
def _is_expired(self, entry: Dict[str, Any]) -> bool:
def _is_expired(self, entry: dict[str, Any]) -> bool:
"""Check if a cache entry is expired."""
if self.ttl_seconds == 0:
return False
return (time.time() - entry.get('cached_at', 0)) > self.ttl_seconds
return (time.time() - entry.get("cached_at", 0)) > self.ttl_seconds
def _is_negative_expired(self, entry: Dict[str, Any]) -> bool:
def _is_negative_expired(self, entry: dict[str, Any]) -> bool:
"""Check if a negative cache entry is expired.
Transient failures (timeouts) expire after TRANSIENT_CACHE_TTL (60s).
Permanent failures (404s) expire after NEGATIVE_CACHE_TTL (1 hour).
"""
if not entry.get('negative', False):
if not entry.get("negative", False):
return False
cached_at = entry.get('cached_at', 0)
ttl = TRANSIENT_CACHE_TTL if entry.get('transient', False) else NEGATIVE_CACHE_TTL
cached_at = entry.get("cached_at", 0)
ttl = TRANSIENT_CACHE_TTL if entry.get("transient", False) else NEGATIVE_CACHE_TTL
return (time.time() - cached_at) > ttl
def _calculate_total_size(self) -> int:
"""Calculate total size of cached images."""
return sum(entry.get('size', 0) for entry in self._index.values())
return sum(entry.get("size", 0) for entry in self._index.values())
def _evict_if_needed(self, required_space: int = 0) -> None:
"""Evict old entries if cache is over size limit.
@@ -214,10 +225,7 @@ class ImageCacheService:
return
# Sort entries by accessed_at (oldest first)
sorted_entries = sorted(
self._index.items(),
key=lambda x: x[1].get('accessed_at', 0)
)
sorted_entries = sorted(self._index.items(), key=lambda x: x[1].get("accessed_at", 0))
evicted_count = 0
for cache_id, entry in sorted_entries:
@@ -225,23 +233,23 @@ class ImageCacheService:
break
# Delete the image file
ext = entry.get('ext', 'jpg')
ext = entry.get("ext", "jpg")
image_path = self._get_image_path(cache_id, ext)
try:
if image_path.exists():
image_path.unlink()
except IOError:
except OSError:
pass
# Update tracking
current_size -= entry.get('size', 0)
current_size -= entry.get("size", 0)
del self._index[cache_id]
evicted_count += 1
if evicted_count > 0:
self._save_index()
def get(self, cache_id: str) -> Optional[Tuple[bytes, str]]:
def get(self, cache_id: str) -> tuple[bytes, str] | None:
"""Get a cached image.
Args:
@@ -249,6 +257,7 @@ class ImageCacheService:
Returns:
Tuple of (image_data, content_type) or None if not cached/expired
"""
with self._lock:
entry = self._index.get(cache_id)
@@ -262,7 +271,7 @@ class ImageCacheService:
return None
# Check for negative cache (failed fetch)
if entry.get('negative', False):
if entry.get("negative", False):
if self._is_negative_expired(entry):
# Negative cache expired, allow retry
del self._index[cache_id]
@@ -275,12 +284,12 @@ class ImageCacheService:
# Check for expired entry
if self._is_expired(entry):
# Remove expired entry
ext = entry.get('ext', 'jpg')
ext = entry.get("ext", "jpg")
image_path = self._get_image_path(cache_id, ext)
try:
if image_path.exists():
image_path.unlink()
except IOError:
except OSError:
pass
del self._index[cache_id]
self._save_index()
@@ -288,9 +297,10 @@ class ImageCacheService:
return None
# Try to read the cached image
ext = entry.get('ext', 'jpg')
content_type = entry.get('content_type', 'image/jpeg')
ext = entry.get("ext", "jpg")
content_type = entry.get("content_type", "image/jpeg")
image_path = self._get_image_path(cache_id, ext)
result: tuple[bytes, str] | None = None
try:
if not image_path.exists():
@@ -300,19 +310,20 @@ class ImageCacheService:
self._misses += 1
return None
with open(image_path, 'rb') as f:
with image_path.open("rb") as f:
data = f.read()
# Update accessed time
entry['accessed_at'] = time.time()
entry["accessed_at"] = time.time()
self._save_index()
result = data, content_type
self._hits += 1
return data, content_type
except IOError:
except OSError:
self._misses += 1
return None
else:
self._hits += 1
return result
def put(self, cache_id: str, data: bytes, content_type: str) -> bool:
"""Store an image in the cache.
@@ -324,24 +335,24 @@ class ImageCacheService:
Returns:
True if stored successfully
"""
with self._lock:
# Detect image type for extension
detected = _detect_image_type(data)
if detected:
content_type, ext = detected
# Fall back to content-type header
elif "jpeg" in content_type or "jpg" in content_type:
ext = "jpg"
elif "png" in content_type:
ext = "png"
elif "gif" in content_type:
ext = "gif"
elif "webp" in content_type:
ext = "webp"
else:
# Fall back to content-type header
if 'jpeg' in content_type or 'jpg' in content_type:
ext = 'jpg'
elif 'png' in content_type:
ext = 'png'
elif 'gif' in content_type:
ext = 'gif'
elif 'webp' in content_type:
ext = 'webp'
else:
ext = 'jpg' # Default
ext = "jpg" # Default
image_size = len(data)
@@ -351,36 +362,37 @@ class ImageCacheService:
# Write image to disk
image_path = self._get_image_path(cache_id, ext)
try:
with open(image_path, 'wb') as f:
with image_path.open("wb") as f:
f.write(data)
except IOError:
except OSError:
return False
# Update index
now = time.time()
self._index[cache_id] = {
'ext': ext,
'content_type': content_type,
'size': image_size,
'cached_at': now,
'accessed_at': now,
'negative': False,
"ext": ext,
"content_type": content_type,
"size": image_size,
"cached_at": now,
"accessed_at": now,
"negative": False,
}
self._save_index()
return True
def put_negative(self, cache_id: str, transient: bool = False) -> None:
def put_negative(self, cache_id: str, *, transient: bool = False) -> None:
"""Store a negative cache entry (failed fetch).
Args:
cache_id: Cache key
transient: If True, uses shorter TTL (for timeouts/connection errors)
"""
with self._lock:
self._index[cache_id] = {
'negative': True,
'transient': transient,
'cached_at': time.time(),
"negative": True,
"transient": transient,
"cached_at": time.time(),
}
self._save_index()
@@ -392,6 +404,7 @@ class ImageCacheService:
Returns:
True if entry existed and was deleted
"""
with self._lock:
entry = self._index.get(cache_id)
@@ -399,13 +412,13 @@ class ImageCacheService:
return False
# Delete file if it exists
if not entry.get('negative', False):
ext = entry.get('ext', 'jpg')
if not entry.get("negative", False):
ext = entry.get("ext", "jpg")
image_path = self._get_image_path(cache_id, ext)
try:
if image_path.exists():
image_path.unlink()
except IOError:
except OSError:
pass
del self._index[cache_id]
@@ -417,19 +430,20 @@ class ImageCacheService:
Returns:
Number of entries cleared
"""
with self._lock:
count = len(self._index)
# Delete all image files
for cache_id, entry in self._index.items():
if not entry.get('negative', False):
ext = entry.get('ext', 'jpg')
if not entry.get("negative", False):
ext = entry.get("ext", "jpg")
image_path = self._get_image_path(cache_id, ext)
try:
if image_path.exists():
image_path.unlink()
except IOError:
except OSError:
pass
# Clear index
@@ -442,31 +456,57 @@ class ImageCacheService:
return count
def stats(self) -> Dict[str, Any]:
def stats(self) -> dict[str, Any]:
"""Get cache statistics.
Returns:
Dict with size, count, hit rate, etc.
"""
with self._lock:
total_size = self._calculate_total_size()
entry_count = len(self._index)
negative_count = sum(1 for e in self._index.values() if e.get('negative', False))
negative_count = sum(1 for e in self._index.values() if e.get("negative", False))
total_requests = self._hits + self._misses
hit_rate = (self._hits / total_requests * 100) if total_requests > 0 else 0
return {
'entry_count': entry_count,
'negative_count': negative_count,
'total_size_bytes': total_size,
'total_size_mb': round(total_size / (1024 * 1024), 2),
'max_size_mb': self.max_size_bytes / (1024 * 1024),
'hits': self._hits,
'misses': self._misses,
'hit_rate': round(hit_rate, 1),
"entry_count": entry_count,
"negative_count": negative_count,
"total_size_bytes": total_size,
"total_size_mb": round(total_size / (1024 * 1024), 2),
"max_size_mb": self.max_size_bytes / (1024 * 1024),
"hits": self._hits,
"misses": self._misses,
"hit_rate": round(hit_rate, 1),
}
def fetch_and_cache(self, cache_id: str, url: str) -> Optional[Tuple[bytes, str]]:
@staticmethod
def _is_safe_url(url: str) -> bool:
"""Check that a URL is safe to fetch (no SSRF to internal resources)."""
try:
parsed = urlparse(url)
hostname = parsed.hostname
except ValueError:
return False
if parsed.scheme not in ("http", "https"):
return False
if not hostname:
return False
try:
resolved = socket.getaddrinfo(hostname, None)
for _, _, _, _, sockaddr in resolved:
ip = ipaddress.ip_address(sockaddr[0])
if ip.is_private or ip.is_loopback or ip.is_link_local or ip.is_reserved:
return False
except socket.gaierror, ValueError:
return False
return True
def fetch_and_cache(self, cache_id: str, url: str) -> tuple[bytes, str] | None:
"""Fetch an image from URL and cache it.
Args:
@@ -475,8 +515,13 @@ class ImageCacheService:
Returns:
Tuple of (image_data, content_type) or None on failure
"""
cached_data: tuple[bytes, str] | None = None
try:
if not self._is_safe_url(url):
logger.warning("Blocked request to disallowed URL: %s", url)
return None
response = requests.get(
url,
@@ -488,8 +533,8 @@ class ImageCacheService:
response.raise_for_status()
# Validate content type
content_type = response.headers.get('content-type', '')
if not content_type.startswith('image/'):
content_type = response.headers.get("content-type", "")
if not content_type.startswith("image/"):
self.put_negative(cache_id)
return None
@@ -513,9 +558,7 @@ class ImageCacheService:
detected = _detect_image_type(image_data)
if detected:
content_type = detected[0]
return image_data, content_type
return None
cached_data = image_data, content_type
except requests.exceptions.Timeout:
self.put_negative(cache_id, transient=True)
@@ -524,15 +567,17 @@ class ImageCacheService:
self.put_negative(cache_id, transient=True)
return None
except requests.exceptions.HTTPError as e:
is_404 = e.response is not None and e.response.status_code == 404
is_404 = e.response is not None and e.response.status_code == HTTP_NOT_FOUND
self.put_negative(cache_id, transient=not is_404)
return None
except Exception:
except requests.exceptions.RequestException:
return None
else:
return cached_data
# Singleton instance (initialized lazily when config is available)
_instance: Optional[ImageCacheService] = None
_instance: ImageCacheService | None = None
_instance_lock = threading.Lock()
@@ -546,12 +591,12 @@ def get_image_cache() -> ImageCacheService:
if _instance is None:
with _instance_lock:
if _instance is None:
from shelfmark.core.config import config
from shelfmark.config.env import CONFIG_DIR
from shelfmark.core.config import config
cache_dir = CONFIG_DIR / "covers"
max_size_mb = config.get("COVERS_CACHE_MAX_SIZE_MB", 500)
ttl_days = config.get("COVERS_CACHE_TTL", 0)
max_size_mb = coerce_int(config.get("COVERS_CACHE_MAX_SIZE_MB", 500), 500)
ttl_days = coerce_int(config.get("COVERS_CACHE_TTL", 0), 0)
ttl_seconds = ttl_days * 86400 if ttl_days > 0 else 0
_instance = ImageCacheService(
@@ -559,7 +604,12 @@ def get_image_cache() -> ImageCacheService:
max_size_mb=max_size_mb,
ttl_seconds=ttl_seconds,
)
logger.debug(f"Initialized image cache: {cache_dir} (max {max_size_mb}MB, TTL {ttl_days} days)")
logger.debug(
"Initialized image cache: %s (max %sMB, TTL %s days)",
cache_dir,
max_size_mb,
ttl_days,
)
return _instance
+91 -39
View File
@@ -2,62 +2,86 @@
import logging
import sys
from pathlib import Path
from collections.abc import Mapping
from logging.handlers import RotatingFileHandler
from typing import Any
from typing import TYPE_CHECKING
from shelfmark.config.env import LOG_FILE, ENABLE_LOGGING, LOG_LEVEL
from shelfmark.config.env import ENABLE_LOGGING, LOG_FILE, LOG_LEVEL
if TYPE_CHECKING:
from pathlib import Path
class CustomLogger(logging.Logger):
"""Custom logger class with additional error_trace method."""
def error_trace(self, msg: Any, *args: Any, **kwargs: Any) -> None:
def error_trace(self, msg: object, *args: object, **kwargs: object) -> None:
"""Log an error message with full stack trace."""
self.log_resource_usage()
kwargs.pop('exc_info', None)
self.error(msg, *args, exc_info=True, **kwargs)
stack_info, stacklevel, extra = _extract_log_kwargs(kwargs)
self.error(
msg,
*args,
exc_info=True,
stack_info=stack_info,
stacklevel=stacklevel,
extra=extra,
)
def warning_trace(self, msg: Any, *args: Any, **kwargs: Any) -> None:
"""Log a warning message with full stack trace."""
self.log_resource_usage()
kwargs.pop('exc_info', None)
self.warning(msg, *args, exc_info=True, **kwargs)
def info_trace(self, msg: Any, *args: Any, **kwargs: Any) -> None:
"""Log an info message (stack trace only if exception active)."""
kwargs.pop('exc_info', None)
# Only include exc_info if there's actually an exception
has_exception = sys.exc_info()[0] is not None
self.info(msg, *args, exc_info=has_exception, **kwargs)
def debug_trace(self, msg: Any, *args: Any, **kwargs: Any) -> None:
def debug_trace(self, msg: object, *args: object, **kwargs: object) -> None:
"""Log a debug message (stack trace only if exception active)."""
kwargs.pop('exc_info', None)
stack_info, stacklevel, extra = _extract_log_kwargs(kwargs)
# Only include exc_info if there's actually an exception
has_exception = sys.exc_info()[0] is not None
self.debug(msg, *args, exc_info=has_exception, **kwargs)
self.debug(
msg,
*args,
exc_info=has_exception,
stack_info=stack_info,
stacklevel=stacklevel,
extra=extra,
)
def log_resource_usage(self):
# Best-effort only; this should never raise during exception logging.
def log_resource_usage(self) -> None:
"""Log best-effort CPU and memory usage for the current container."""
try:
import psutil
except ImportError:
return
# Best-effort only; this should never raise during exception logging.
try:
def _get_process_rss_mb(proc: object) -> float | None:
try:
proc_info = getattr(proc, "info", None)
if not isinstance(proc_info, Mapping):
return None
mem = proc_info.get("memory_info")
rss = getattr(mem, "rss", None)
if isinstance(rss, int | float):
return rss / (1024 * 1024)
except (
psutil.NoSuchProcess,
psutil.AccessDenied,
KeyError,
AttributeError,
):
return None
return None
# Sum RSS of all processes for actual app memory (container-friendly),
# but fall back gracefully on platforms that restrict process enumeration.
app_memory_mb = 0.0
try:
for proc in psutil.process_iter(['memory_info']):
try:
mem = proc.info.get('memory_info')
if mem:
app_memory_mb += mem.rss / (1024 * 1024)
except (psutil.NoSuchProcess, psutil.AccessDenied, KeyError, AttributeError):
continue
except (PermissionError, psutil.AccessDenied, OSError):
for proc in psutil.process_iter(["memory_info"]):
proc_rss_mb = _get_process_rss_mb(proc)
if proc_rss_mb is not None:
app_memory_mb += proc_rss_mb
except PermissionError, psutil.AccessDenied, OSError:
try:
app_memory_mb = psutil.Process().memory_info().rss / (1024 * 1024)
except Exception:
except AttributeError, OSError, psutil.Error:
app_memory_mb = 0.0
memory = psutil.virtual_memory()
@@ -68,11 +92,36 @@ class CustomLogger(logging.Logger):
f"Container Memory: App={app_memory_mb:.2f} MB, System={system_used_mb:.2f} MB, "
f"Available={available_mb:.2f} MB, CPU: {cpu_percent:.2f}%"
)
except Exception:
except AttributeError, OSError, psutil.Error:
# Avoid breaking the original log call if psutil is missing or restricted.
return
def _extract_log_kwargs(
kwargs: Mapping[str, object],
) -> tuple[bool, int, Mapping[str, object] | None]:
stack_info = kwargs.get("stack_info")
normalized_stack_info = stack_info if isinstance(stack_info, bool) else False
stacklevel = kwargs.get("stacklevel")
normalized_stacklevel = stacklevel if isinstance(stacklevel, int) else 1
extra = kwargs.get("extra")
normalized_extra = _normalize_log_extra(extra)
return normalized_stack_info, normalized_stacklevel, normalized_extra
def _normalize_log_extra(value: object) -> Mapping[str, object] | None:
if not isinstance(value, Mapping):
return None
if all(isinstance(key, str) for key in value):
return value
return None
def setup_logger(name: str, log_file: Path = LOG_FILE) -> CustomLogger:
"""Set up and configure a logger instance.
@@ -82,6 +131,7 @@ def setup_logger(name: str, log_file: Path = LOG_FILE) -> CustomLogger:
Returns:
CustomLogger: Configured logger instance with error_trace method
"""
# Register our custom logger class
logging.setLoggerClass(CustomLogger)
@@ -92,19 +142,21 @@ def setup_logger(name: str, log_file: Path = LOG_FILE) -> CustomLogger:
logger.setLevel(log_level)
formatter = logging.Formatter(
'%(asctime)s - %(name)s - %(levelname)s - %(filename)s:%(lineno)d - %(message)s'
"%(asctime)s - %(name)s - %(levelname)s - %(filename)s:%(lineno)d - %(message)s"
)
# Console handler for Docker output
console_handler = logging.StreamHandler(sys.stdout)
console_handler.setFormatter(formatter)
console_handler.setLevel(log_level)
console_handler.addFilter(lambda record: record.levelno < logging.ERROR) # Only allow logs below ERROR to stdout
console_handler.addFilter(
lambda record: record.levelno < logging.ERROR
) # Only allow logs below ERROR to stdout
logger.addHandler(console_handler)
# Error handler for stderr
error_handler = logging.StreamHandler(sys.stderr)
error_handler.setLevel(logging.ERROR) # Error and above go to stderr
error_handler.setLevel(logging.ERROR) # Error and above go to stderr
error_handler.setFormatter(formatter)
logger.addHandler(error_handler)
@@ -117,11 +169,11 @@ def setup_logger(name: str, log_file: Path = LOG_FILE) -> CustomLogger:
file_handler = RotatingFileHandler(
log_file,
maxBytes=10485760, # 10MB
backupCount=5
backupCount=5,
)
file_handler.setFormatter(formatter)
logger.addHandler(file_handler)
except Exception as e:
except (OSError, TypeError, ValueError) as e:
logger.error_trace(f"Failed to create log file: {e}", exc_info=True)
return logger
+175 -168
View File
@@ -1,256 +1,263 @@
"""Centralized mirror configuration for all download sources."""
"""Centralized mirror configuration for direct-download sources."""
from typing import List
from __future__ import annotations
from typing import TYPE_CHECKING
from shelfmark.core.utils import normalize_http_url
# Lazy import to avoid circular imports
if TYPE_CHECKING:
from shelfmark.core.config import Config
_config_module = None
def _get_config():
def _get_config() -> Config:
"""Lazy import of config module to avoid circular imports."""
global _config_module
if _config_module is None:
from shelfmark.core.config import config
_config_module = config
return _config_module
# Default mirror lists (hardcoded fallbacks)
DEFAULT_AA_MIRRORS = [
"https://annas-archive.gl",
"https://annas-archive.pk",
"https://annas-archive.vg",
"https://annas-archive.gd",
]
# Mirror URLs are intentionally user-supplied only.
DEFAULT_AA_MIRRORS: list[str] = []
DEFAULT_LIBGEN_MIRRORS: list[str] = []
DEFAULT_ZLIB_MIRRORS: list[str] = []
DEFAULT_WELIB_MIRRORS: list[str] = []
DEFAULT_LIBGEN_MIRRORS = [
"https://libgen.gl",
"https://libgen.li",
"https://libgen.bz",
"https://libgen.la",
"https://libgen.vg",
]
DEFAULT_ZLIB_MIRRORS = [
"https://z-lib.fm",
"https://z-lib.gs",
"https://z-lib.id",
"https://z-library.sk",
"https://zlibrary-global.se",
]
DEFAULT_WELIB_MIRRORS = [
"https://welib.org",
]
_DOWNLOAD_SOURCE_MIRROR_LABELS = {
"aa-fast": "Anna's Archive",
"aa-slow": "Anna's Archive",
"aa-slow-nowait": "Anna's Archive",
"aa-slow-wait": "Anna's Archive",
"libgen": "LibGen",
"zlib": "Z-Library",
"welib": "Welib",
}
def _normalize_mirror_url(url: str) -> str:
return normalize_http_url(url, default_scheme="https")
def get_aa_mirrors() -> List[str]:
"""
Get Anna's Archive mirrors.
def _string_config_value(value: object) -> str:
"""Normalize mirror-related config values to strings."""
return value if isinstance(value, str) else str(value or "")
Returns:
Ordered list of AA mirror URLs.
If AA_MIRROR_URLS is configured, it is treated as the full list.
Otherwise, defaults are used and AA_ADDITIONAL_URLS (legacy) is appended.
def _normalize_configured_urls(value: object) -> list[str]:
"""Normalize list or comma-separated mirror config into unique URLs."""
if isinstance(value, list):
parts = value
elif isinstance(value, str) and value.strip():
parts = value.split(",")
else:
return []
Notes:
- The list is used to populate the AA mirror dropdown in Settings.
- When AA_BASE_URL is set to 'auto', mirrors are tried in the order listed.
"""
normalized_urls: list[str] = []
for raw_url in parts:
normalized = _normalize_mirror_url(str(raw_url))
if normalized and normalized not in normalized_urls:
normalized_urls.append(normalized)
return normalized_urls
def _get_primary_mirror_url(key: str) -> str | None:
"""Return a configured primary mirror URL, if present."""
config = _get_config()
primary = _normalize_mirror_url(_string_config_value(config.get(key, "")))
return primary or None
def _build_primary_and_additional_mirrors(primary_key: str, additional_key: str) -> list[str]:
"""Build an ordered mirror list from primary + additional config values."""
config = _get_config()
mirrors: list[str] = []
configured_list = config.get("AA_MIRROR_URLS", None)
if isinstance(configured_list, list):
for url in configured_list:
normalized = _normalize_mirror_url(str(url))
if normalized and normalized not in mirrors:
mirrors.append(normalized)
elif isinstance(configured_list, str) and configured_list.strip():
# Allow comma-separated env/manual configs.
for url in configured_list.split(","):
normalized = _normalize_mirror_url(url)
if normalized and normalized not in mirrors:
mirrors.append(normalized)
primary = _get_primary_mirror_url(primary_key)
if primary:
mirrors.append(primary)
if not mirrors:
mirrors = [_normalize_mirror_url(url) for url in DEFAULT_AA_MIRRORS]
mirrors = [url for url in mirrors if url]
# Backwards-compatible append-only behavior for legacy configs/env.
additional = config.get("AA_ADDITIONAL_URLS", "")
if additional:
for url in additional.split(","):
normalized = _normalize_mirror_url(url)
if normalized and normalized not in mirrors:
mirrors.append(normalized)
for url in _normalize_configured_urls(config.get(additional_key, "")):
if url not in mirrors:
mirrors.append(url)
return mirrors
def get_libgen_mirrors() -> List[str]:
"""
Get LibGen mirrors: defaults + any additional from config.
def get_aa_mirrors() -> list[str]:
"""Get Anna's Archive mirrors.
Returns:
List of LibGen mirror URLs (defaults first, then custom additions).
Ordered list of user-configured AA mirror URLs.
Notes:
- The list is used to populate the AA mirror dropdown in Settings.
- When AA_BASE_URL is set to 'auto', mirrors are tried in the order listed.
"""
mirrors = [_normalize_mirror_url(url) for url in DEFAULT_LIBGEN_MIRRORS]
mirrors = [url for url in mirrors if url]
config = _get_config()
additional = config.get("LIBGEN_ADDITIONAL_URLS", "")
if additional:
for url in additional.split(","):
normalized = _normalize_mirror_url(url)
if normalized and normalized not in mirrors:
mirrors.append(normalized)
return mirrors
configured_list = _normalize_configured_urls(config.get("AA_MIRROR_URLS", None))
if configured_list:
return configured_list
return _normalize_configured_urls(config.get("AA_ADDITIONAL_URLS", ""))
def get_zlib_mirrors() -> List[str]:
def has_aa_mirror_configuration() -> bool:
"""Return True when direct-download search has at least one AA base URL to use."""
if get_aa_mirrors():
return True
configured_base_url = normalize_http_url(
_string_config_value(_get_config().get("AA_BASE_URL", "auto")),
default_scheme="https",
allow_special=("auto",),
)
return bool(configured_base_url and configured_base_url != "auto")
def get_libgen_mirrors() -> list[str]:
"""Get user-configured LibGen mirrors.
Returns:
List of LibGen mirror URLs.
"""
Get Z-Library mirrors, with primary first.
config = _get_config()
configured_list = _normalize_configured_urls(config.get("LIBGEN_MIRROR_URLS", None))
if configured_list:
return configured_list
return _normalize_configured_urls(config.get("LIBGEN_ADDITIONAL_URLS", ""))
def has_libgen_mirror_configuration() -> bool:
"""Return True when at least one LibGen mirror URL is configured."""
return bool(get_libgen_mirrors())
def get_zlib_mirrors() -> list[str]:
"""Get user-configured Z-Library mirrors, with primary first.
Returns:
List of Z-Library mirror URLs, primary first.
"""
config = _get_config()
primary = _normalize_mirror_url(config.get("ZLIB_PRIMARY_URL", DEFAULT_ZLIB_MIRRORS[0]))
if not primary:
primary = _normalize_mirror_url(DEFAULT_ZLIB_MIRRORS[0])
mirrors = [primary]
# Add other defaults (excluding primary)
for url in DEFAULT_ZLIB_MIRRORS:
normalized = _normalize_mirror_url(url)
if normalized and normalized != primary:
mirrors.append(normalized)
# Add custom mirrors
additional = config.get("ZLIB_ADDITIONAL_URLS", "")
if additional:
for url in additional.split(","):
normalized = _normalize_mirror_url(url)
if normalized and normalized not in mirrors:
mirrors.append(normalized)
return mirrors
configured_list = _normalize_configured_urls(config.get("ZLIB_MIRROR_URLS", None))
if configured_list:
return configured_list
return _build_primary_and_additional_mirrors("ZLIB_PRIMARY_URL", "ZLIB_ADDITIONAL_URLS")
def get_zlib_primary_url() -> str:
"""
Get the primary Z-Library mirror URL.
def has_zlib_mirror_configuration() -> bool:
"""Return True when at least one Z-Library mirror URL is configured."""
return bool(get_zlib_mirrors())
def get_zlib_primary_url() -> str | None:
"""Get the primary Z-Library mirror URL.
Returns:
Primary Z-Library mirror URL.
Primary Z-Library mirror URL, if configured.
"""
config = _get_config()
primary = _normalize_mirror_url(config.get("ZLIB_PRIMARY_URL", DEFAULT_ZLIB_MIRRORS[0]))
return primary or _normalize_mirror_url(DEFAULT_ZLIB_MIRRORS[0])
mirrors = get_zlib_mirrors()
return mirrors[0] if mirrors else None
def get_zlib_url_template() -> str:
"""
Get Z-Library URL template using configured primary mirror.
def get_zlib_url_template() -> str | None:
"""Get Z-Library URL template using configured primary mirror.
Returns:
URL template with {md5} placeholder.
URL template with {md5} placeholder, if configured.
"""
primary = get_zlib_primary_url()
return f"{primary}/md5/{{md5}}"
return f"{primary}/md5/{{md5}}" if primary else None
def get_welib_mirrors() -> List[str]:
"""
Get Welib mirrors, with primary first.
def get_welib_mirrors() -> list[str]:
"""Get user-configured Welib mirrors, with primary first.
Returns:
List of Welib mirror URLs, primary first.
"""
config = _get_config()
primary = _normalize_mirror_url(config.get("WELIB_PRIMARY_URL", DEFAULT_WELIB_MIRRORS[0]))
if not primary:
primary = _normalize_mirror_url(DEFAULT_WELIB_MIRRORS[0])
mirrors = [primary]
# Add other defaults (excluding primary)
for url in DEFAULT_WELIB_MIRRORS:
normalized = _normalize_mirror_url(url)
if normalized and normalized != primary:
mirrors.append(normalized)
# Add custom mirrors
additional = config.get("WELIB_ADDITIONAL_URLS", "")
if additional:
for url in additional.split(","):
normalized = _normalize_mirror_url(url)
if normalized and normalized not in mirrors:
mirrors.append(normalized)
return mirrors
configured_list = _normalize_configured_urls(config.get("WELIB_MIRROR_URLS", None))
if configured_list:
return configured_list
return _build_primary_and_additional_mirrors("WELIB_PRIMARY_URL", "WELIB_ADDITIONAL_URLS")
def get_welib_primary_url() -> str:
"""
Get the primary Welib mirror URL.
def has_welib_mirror_configuration() -> bool:
"""Return True when at least one Welib mirror URL is configured."""
return bool(get_welib_mirrors())
def has_download_source_mirror_configuration(source_id: str) -> bool:
"""Return True when the requested direct-download source has mirror config."""
if source_id in {"aa-fast", "aa-slow", "aa-slow-nowait", "aa-slow-wait"}:
return has_aa_mirror_configuration()
if source_id == "libgen":
return has_libgen_mirror_configuration()
if source_id == "zlib":
return has_zlib_mirror_configuration()
if source_id == "welib":
return has_welib_mirror_configuration()
return False
def get_download_source_missing_mirror_reason(source_id: str) -> str | None:
"""Return a user-facing reason when a direct-download source has no mirror config."""
if has_download_source_mirror_configuration(source_id):
return None
label = _DOWNLOAD_SOURCE_MIRROR_LABELS.get(source_id)
if not label:
return None
return f"Add at least one {label} mirror in Mirrors"
def get_welib_primary_url() -> str | None:
"""Get the primary Welib mirror URL.
Returns:
Primary Welib mirror URL.
Primary Welib mirror URL, if configured.
"""
config = _get_config()
primary = _normalize_mirror_url(config.get("WELIB_PRIMARY_URL", DEFAULT_WELIB_MIRRORS[0]))
return primary or _normalize_mirror_url(DEFAULT_WELIB_MIRRORS[0])
mirrors = get_welib_mirrors()
return mirrors[0] if mirrors else None
def get_welib_url_template() -> str:
"""
Get Welib URL template using configured primary mirror.
def get_welib_url_template() -> str | None:
"""Get Welib URL template using configured primary mirror.
Returns:
URL template with {md5} placeholder.
URL template with {md5} placeholder, if configured.
"""
primary = get_welib_primary_url()
return f"{primary}/md5/{{md5}}"
return f"{primary}/md5/{{md5}}" if primary else None
def get_zlib_cookie_domains() -> set:
"""
Get set of Z-Library domains that need full cookie handling.
"""Get set of Z-Library domains that need full cookie handling.
Used by internal_bypasser for CF bypass cookie management.
Returns:
Set of domain strings (without protocol).
"""
domains = set()
# Add all default domains
for url in DEFAULT_ZLIB_MIRRORS:
normalized = _normalize_mirror_url(url)
if normalized:
domain = normalized.replace("https://", "").replace("http://", "").split("/")[0]
domains.add(domain)
# Add custom domains
config = _get_config()
additional = config.get("ZLIB_ADDITIONAL_URLS", "")
if additional:
for url in additional.split(","):
normalized = _normalize_mirror_url(url)
if normalized:
domain = normalized.replace("https://", "").replace("http://", "").split("/")[0]
domains.add(domain)
for url in get_zlib_mirrors():
domain = url.replace("https://", "").replace("http://", "").split("/")[0]
domains.add(domain)
return domains
+81 -50
View File
@@ -1,19 +1,20 @@
"""Data structures and models used across the application."""
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any, Dict, List, Optional
from enum import Enum
import re
import time
from dataclasses import dataclass, field
from enum import StrEnum
from pathlib import Path
from typing import Any
def build_filename(
title: str,
author: Optional[str] = None,
year: Optional[str] = None,
fmt: Optional[str] = None,
author: str | None = None,
year: str | None = None,
fmt: str | None = None,
) -> str:
"""Build a filesystem-safe filename from book metadata."""
parts = []
if author:
parts.append(author)
@@ -23,7 +24,7 @@ def build_filename(
parts.append(f" ({year})")
filename = "".join(parts)
filename = re.sub(r'[\\/:*?"<>|]', '_', filename.strip())[:245]
filename = re.sub(r'[\\/:*?"<>|]', "_", filename.strip())[:245]
if fmt:
filename = f"{filename}.{fmt}"
@@ -31,8 +32,9 @@ def build_filename(
return filename
class QueueStatus(str, Enum):
class QueueStatus(StrEnum):
"""Enum for possible book queue statuses."""
QUEUED = "queued"
RESOLVING = "resolving"
LOCATING = "locating"
@@ -42,16 +44,27 @@ class QueueStatus(str, Enum):
CANCELLED = "cancelled"
TERMINAL_QUEUE_STATUSES: frozenset[QueueStatus] = frozenset({
QueueStatus.COMPLETE, QueueStatus.ERROR, QueueStatus.CANCELLED,
})
TERMINAL_QUEUE_STATUSES: frozenset[QueueStatus] = frozenset(
{
QueueStatus.COMPLETE,
QueueStatus.ERROR,
QueueStatus.CANCELLED,
}
)
ACTIVE_QUEUE_STATUSES: frozenset[QueueStatus] = frozenset({
QueueStatus.QUEUED, QueueStatus.RESOLVING, QueueStatus.LOCATING, QueueStatus.DOWNLOADING,
})
ACTIVE_QUEUE_STATUSES: frozenset[QueueStatus] = frozenset(
{
QueueStatus.QUEUED,
QueueStatus.RESOLVING,
QueueStatus.LOCATING,
QueueStatus.DOWNLOADING,
}
)
class SearchMode(str, Enum):
class SearchMode(StrEnum):
"""Search modes supported by the Shelfmark UI and API."""
DIRECT = "direct"
UNIVERSAL = "universal"
@@ -59,11 +72,12 @@ class SearchMode(str, Enum):
@dataclass
class QueueItem:
"""Queue item with priority and metadata."""
book_id: str
priority: int
added_time: float
def __lt__(self, other):
def __lt__(self, other: QueueItem) -> bool:
"""Compare items for priority queue (lower priority number = higher precedence)."""
if self.priority != other.priority:
return self.priority < other.priority
@@ -72,53 +86,69 @@ class QueueItem:
@dataclass
class DownloadTask:
task_id: str # Unique ID (e.g., AA MD5 hash, Prowlarr GUID)
source: str # Handler name ("direct_download", "prowlarr")
title: str # Display title for queue sidebar
"""Mutable download task state tracked throughout the pipeline."""
task_id: str # Unique ID (e.g., AA MD5 hash, Prowlarr GUID)
source: str # Handler name ("direct_download", "prowlarr")
title: str # Display title for queue sidebar
# Display info for queue sidebar
author: Optional[str] = None
year: Optional[str] = None
format: Optional[str] = None
size: Optional[str] = None
preview: Optional[str] = None
content_type: Optional[str] = None # "book (fiction)", "audiobook", "magazine", etc.
source_url: Optional[str] = None # Original release URL used by source-specific handlers
author: str | None = None
year: str | None = None
format: str | None = None
size: str | None = None
preview: str | None = None
content_type: str | None = None # "book (fiction)", "audiobook", "magazine", etc.
source_url: str | None = None # Original release URL used by source-specific handlers
retry_download_url: str | None = None # Resolved download URL for restart-safe retries
retry_download_protocol: str | None = (
None # Protocol for retry_download_url (e.g. torrent, usenet)
)
retry_release_name: str | None = None # Display name to send back to external download clients
retry_expected_hash: str | None = None # Optional torrent hash used to match client downloads
retry_ratio_limit: float | None = None # Optional post-download seeding ratio
retry_seeding_time_limit_minutes: int | None = None # Optional post-download seeding time limit
can_retry_without_staged_source: bool = (
True # Whether the source can restart without a preserved staged file
)
# Series info (for library naming templates)
series_name: Optional[str] = None
series_position: Optional[float] = None # Float for novellas (e.g., 1.5)
subtitle: Optional[str] = None # Book subtitle for naming templates
series_name: str | None = None
series_position: float | None = None # Float for novellas (e.g., 1.5)
subtitle: str | None = None # Book subtitle for naming templates
# Hardlinking support
original_download_path: Optional[str] = None # Path in download client (for hardlinking)
original_download_path: str | None = None # Path in download client (for hardlinking)
# Search mode - determines post-download processing behavior
# See SearchMode enum for behavioral differences
search_mode: Optional[SearchMode] = None
search_mode: SearchMode | None = None
# Output selection for post-processing.
# This is captured at queue time so in-flight tasks are not affected if the user changes settings later.
output_mode: Optional[str] = None # e.g. "folder", "booklore", "email"
output_args: Dict[str, Any] = field(default_factory=dict) # Per-output parameters (e.g. email recipient)
output_mode: str | None = None
output_args: dict[str, Any] = field(
default_factory=dict
) # Per-output parameters (e.g. email recipient)
# User association (multi-user support)
user_id: Optional[int] = None # DB user ID who queued this download
username: Optional[str] = None # Username for {User} template variable
request_id: Optional[int] = None # Origin request ID when queued from request fulfilment
user_id: int | None = None # DB user ID who queued this download
username: str | None = None # Username for {User} template variable
request_id: int | None = None # Origin request ID when queued from request fulfilment
# Runtime state
priority: int = 0
added_time: float = field(default_factory=time.time)
progress: float = 0.0
status: QueueStatus = QueueStatus.QUEUED
status_message: Optional[str] = None
download_path: Optional[str] = None
last_error_message: Optional[str] = None
last_error_type: Optional[str] = None
staged_path: Optional[str] = None
status_message: str | None = None
download_path: str | None = None
last_error_message: str | None = None
last_error_type: str | None = None
staged_path: str | None = None
def __lt__(self, other):
def __lt__(self, other: DownloadTask) -> bool:
"""Compare tasks for priority queue (lower priority number = higher precedence)."""
if self.priority != other.priority:
return self.priority < other.priority
@@ -134,10 +164,11 @@ class DownloadTask:
@dataclass
class SearchFilters:
"""Filters for book search queries."""
isbn: Optional[List[str]] = None
author: Optional[List[str]] = None
title: Optional[List[str]] = None
lang: Optional[List[str]] = None
sort: Optional[str] = None
content: Optional[List[str]] = None
format: Optional[List[str]] = None
isbn: list[str] | None = None
author: list[str] | None = None
title: list[str] | None = None
lang: list[str] | None = None
sort: str | None = None
content: list[str] | None = None
format: list[str] | None = None
+85 -62
View File
@@ -1,48 +1,51 @@
"""Template-based naming for library organization."""
import os
import re
from pathlib import Path
from typing import Dict, Optional, Union, Mapping
from typing import TYPE_CHECKING
from shelfmark.core.logger import setup_logger
if TYPE_CHECKING:
from collections.abc import Mapping
logger = setup_logger(__name__)
# Known variable tokens, sorted longest-first to avoid partial matches
# e.g., "SeriesPosition" must match before "Series"
KNOWN_TOKENS = [
'seriesposition',
'originalname',
'partnumber',
'subtitle',
'author',
'series',
'title',
'year',
'user',
"seriesposition",
"primarytitle",
"originalname",
"partnumber",
"subtitle",
"author",
"series",
"title",
"year",
"user",
]
# Match any {...} block for template parsing
BRACE_PATTERN = re.compile(r'\{([^}]+)\}')
BRACE_PATTERN = re.compile(r"\{([^}]+)\}")
# Characters that are invalid in filenames on various filesystems
INVALID_CHARS = re.compile(r'[\\/:*?"<>|]')
def _sanitize(name: Optional[str], max_length: int = 245) -> str:
def _sanitize(name: str | None, max_length: int = 245) -> str:
"""Sanitize a string for filesystem use."""
if not name:
return ""
sanitized = INVALID_CHARS.sub('_', name)
sanitized = re.sub(r'^[\s.]+|[\s.]+$', '', sanitized) # Strip whitespace and dots
sanitized = re.sub(r'_+', '_', sanitized) # Collapse underscores
sanitized = INVALID_CHARS.sub("_", name)
sanitized = re.sub(r"^[\s.]+|[\s.]+$", "", sanitized) # Strip whitespace and dots
sanitized = re.sub(r"_+", "_", sanitized) # Collapse underscores
return sanitized[:max_length]
def sanitize_filename(name: Optional[str], max_length: int = 245) -> str:
def sanitize_filename(name: str | None, max_length: int = 245) -> str:
"""Sanitize a string for use as a filename or path component."""
return _sanitize(name, max_length)
@@ -51,7 +54,8 @@ def sanitize_filename(name: Optional[str], max_length: int = 245) -> str:
sanitize_path_component = sanitize_filename
def format_series_position(position: Optional[Union[str, int, float]]) -> str:
def format_series_position(position: str | float | None) -> str:
"""Format a series position for naming templates."""
if position is None:
return ""
@@ -62,11 +66,30 @@ def format_series_position(position: Optional[Union[str, int, float]]) -> str:
return str(position)
def derive_primary_title(title: str | None, subtitle: str | None) -> str:
"""Return the title without an explicit subtitle suffix when possible."""
title_value = " ".join(str(title or "").split()).strip()
if not title_value:
return ""
subtitle_value = " ".join(str(subtitle or "").split()).strip()
if not subtitle_value:
return title_value
pattern = rf"^(?P<primary>.+?)(?:\s*:\s*|\s+-\s+){re.escape(subtitle_value)}$"
match = re.match(pattern, title_value, flags=re.IGNORECASE)
if not match:
return title_value
primary = match.group("primary").strip()
return primary or title_value
# Pads numbers to 9 digits for natural sorting (e.g., "Part 2" -> "Part 000000002")
PAD_NUMBERS_PATTERN = re.compile(r'\d+')
PAD_NUMBERS_PATTERN = re.compile(r"\d+")
def natural_sort_key(path: Union[str, Path]) -> str:
def natural_sort_key(path: str | Path) -> str:
"""Generate a sort key with padded numbers for natural sorting."""
filename = Path(path).name.lower()
return PAD_NUMBERS_PATTERN.sub(lambda m: m.group().zfill(9), filename)
@@ -89,40 +112,41 @@ def assign_part_numbers(
def parse_naming_template(
template: str,
metadata: Mapping[str, Optional[Union[str, int, float]]],
metadata: Mapping[str, str | int | float | None],
*,
allow_path_separators: bool = True,
) -> str:
"""Render a naming template with Shelfmark metadata placeholders."""
if not template:
return ""
# Normalize metadata keys to lowercase for case-insensitive matching
normalized = {k.lower(): v for k, v in metadata.items()}
def find_token(content: str) -> tuple[Optional[str], int]:
def find_placeholder(content: str) -> tuple[str | None, int]:
content_lower = content.lower()
for token in KNOWN_TOKENS:
idx = content_lower.find(token)
for placeholder_name in KNOWN_TOKENS:
idx = content_lower.find(placeholder_name)
if idx != -1:
return token, idx
return placeholder_name, idx
return None, -1
def token_value(token: str) -> str:
value = normalized.get(token)
if token == 'seriesposition':
def placeholder_value(placeholder_name: str) -> str:
value = normalized.get(placeholder_name)
if placeholder_name == "seriesposition":
value = format_series_position(value)
if value is None:
return ""
return str(value).strip()
def render_block(content: str) -> Optional[str]:
token, idx = find_token(content)
if token is None:
def render_block(content: str) -> str | None:
placeholder_name, idx = find_placeholder(content)
if placeholder_name is None:
return None
prefix = content[:idx]
suffix = content[idx + len(token):]
value = token_value(token)
suffix = content[idx + len(placeholder_name) :]
value = placeholder_value(placeholder_name)
if not value:
return ""
@@ -140,7 +164,7 @@ def parse_naming_template(
parts: list[str] = []
cursor = 0
for idx, match in enumerate(matches):
parts.append(template[cursor:match.start()])
parts.append(template[cursor : match.start()])
content = match.group(1)
rendered = render_block(content)
@@ -151,17 +175,16 @@ def parse_naming_template(
include_literal = False
if idx + 1 < len(matches) and match.end() == matches[idx + 1].start():
next_content = matches[idx + 1].group(1)
next_token, _next_idx = find_token(next_content)
if next_token is not None:
next_placeholder_name, _next_idx = find_placeholder(next_content)
if next_placeholder_name is not None:
conditional_literal = True
include_literal = bool(token_value(next_token))
include_literal = bool(placeholder_value(next_placeholder_name))
if include_literal:
parts.append(content)
elif not conditional_literal:
elif not conditional_literal and re.search(r"\s", content):
# Preserve blocks that look like literal text, but treat bare unknown
# placeholders as missing variables.
if re.search(r"\s", content):
parts.append(match.group(0))
parts.append(match.group(0))
cursor = match.end()
@@ -169,41 +192,40 @@ def parse_naming_template(
result = "".join(parts)
# Clean up any double slashes that might result from empty tokens
result = re.sub(r'/+', '/', result)
result = re.sub(r"/+", "/", result)
# Remove leading/trailing slashes
result = result.strip('/')
result = result.strip("/")
# Clean up any orphaned separators (e.g., " - " at start/end, or " - - ")
result = re.sub(r'^[\s\-_.]+', '', result)
result = re.sub(r'[\s\-_.]+$', '', result)
result = re.sub(r'(\s*-\s*){2,}', ' - ', result)
result = re.sub(r"^[\s\-_.]+", "", result)
result = re.sub(r"[\s\-_.]+$", "", result)
result = re.sub(r"(\s*-\s*){2,}", " - ", result)
# Clean up empty parentheses/brackets
result = re.sub(r'\(\s*\)', '', result)
result = re.sub(r'\[\s*\]', '', result)
result = re.sub(r"\(\s*\)", "", result)
result = re.sub(r"\[\s*\]", "", result)
# Final trim of any trailing separators left after cleanup
result = re.sub(r'[\s\-_.]+$', '', result)
return result
return re.sub(r"[\s\-_.]+$", "", result)
def build_library_path(
base_path: str,
template: str,
metadata: Mapping[str, Optional[Union[str, int, float]]],
extension: Optional[str] = None,
metadata: Mapping[str, str | int | float | None],
extension: str | None = None,
) -> Path:
"""Build a final library path from a template and metadata."""
relative = parse_naming_template(template, metadata, allow_path_separators=True)
if not relative:
# Fallback to title if template produces empty result
title = metadata.get('Title') or metadata.get('title') or 'Unknown'
title = metadata.get("Title") or metadata.get("title") or "Unknown"
relative = sanitize_filename(str(title))
# Remove any path traversal attempts
relative = relative.replace('..', '')
relative = relative.replace("..", "")
base = Path(base_path).resolve()
full_path = (base / relative).resolve()
@@ -211,11 +233,12 @@ def build_library_path(
# Verify the path is within the base directory
try:
full_path.relative_to(base)
except ValueError:
raise ValueError(f"Path traversal detected: template would escape library directory")
except ValueError as exc:
msg = "Path traversal detected: template would escape library directory"
raise ValueError(msg) from exc
if extension:
ext = extension.lstrip('.')
ext = extension.lstrip(".")
# Don't use with_suffix() - it replaces everything after the first dot
# e.g., "2.5 - Title" would become "2.epub" instead of "2.5 - Title.epub"
full_path = Path(f"{full_path}.{ext}")
@@ -223,27 +246,27 @@ def build_library_path(
return full_path
def same_filesystem(path1: Union[str, Path], path2: Union[str, Path]) -> bool:
def same_filesystem(path1: str | Path, path2: str | Path) -> bool:
"""Check if two paths are on the same filesystem."""
path1 = Path(path1)
path2 = Path(path2)
def get_device(p: Path) -> Optional[int]:
def get_device(p: Path) -> int | None:
try:
while not p.exists():
p = p.parent
if p == p.parent:
break
return os.stat(p).st_dev
return p.stat().st_dev
except (OSError, PermissionError) as e:
logger.debug(f"Cannot stat {p}: {e}")
logger.debug("Cannot stat %s: %s", p, e)
return None
dev1 = get_device(path1)
dev2 = get_device(path2)
if dev1 is None or dev2 is None:
logger.warning(f"Cannot determine filesystem for hardlink check, falling back to copy")
logger.warning("Cannot determine filesystem for hardlink check, falling back to copy")
return False
return dev1 == dev2
+90 -49
View File
@@ -5,19 +5,23 @@ from __future__ import annotations
import logging
import threading
from concurrent.futures import ThreadPoolExecutor
from contextlib import contextmanager
from contextlib import contextmanager, suppress
from dataclasses import dataclass
from enum import Enum
from typing import Any, Iterable, Iterator
from enum import StrEnum
from typing import TYPE_CHECKING, Any, Protocol, TypeGuard
from urllib.parse import urlsplit
try:
import apprise
except Exception: # pragma: no cover - exercised in tests via monkeypatch
except ImportError: # pragma: no cover - exercised in tests via monkeypatch
apprise = None # type: ignore[assignment]
from shelfmark.core.config import config as app_config
from shelfmark.core.logger import setup_logger
from shelfmark.core.request_helpers import normalize_positive_int
if TYPE_CHECKING:
from collections.abc import Iterable, Iterator
logger = setup_logger(__name__)
@@ -30,9 +34,36 @@ _APPRISE_LOGO_URL = (
"https://raw.githubusercontent.com/calibrain/shelfmark/main/src/frontend/public/logo.png"
)
_APPRISE_LOGGER_NAME = "apprise"
_APPRISE_DISPATCH_ERRORS = (RuntimeError, TypeError, ValueError)
class NotificationEvent(str, Enum):
class _ApprisePluginWithUrl(Protocol):
app_id: object
def url(self, *, privacy: bool = False) -> str:
_ = privacy
return ""
class _AppriseClient(Protocol):
asset: object
def add(self, plugin: object) -> object: ...
def notify(self, *, title: str, body: str, notify_type: object) -> object: ...
def _is_apprise_client(candidate: object) -> TypeGuard[_AppriseClient]:
return callable(getattr(candidate, "add", None)) and callable(
getattr(candidate, "notify", None)
)
def _has_plugin_url(candidate: object) -> TypeGuard[_ApprisePluginWithUrl]:
return callable(getattr(candidate, "url", None))
class NotificationEvent(StrEnum):
"""Global notification event identifiers."""
REQUEST_CREATED = "request_created"
@@ -57,7 +88,7 @@ class NotificationContext:
error_message: str | None = None
def _normalize_urls(value: Any) -> list[str]:
def _normalize_urls(value: object) -> list[str]:
if value is None:
return []
@@ -103,7 +134,7 @@ def _extract_url_schemes(urls: Iterable[str]) -> list[str]:
class _AppriseLogCapture(logging.Handler):
def __init__(self, *, thread_id: int):
def __init__(self, *, thread_id: int) -> None:
super().__init__(level=logging.INFO)
self.records: list[tuple[int, str, str, str]] = []
self._thread_id = thread_id
@@ -126,7 +157,9 @@ class _AppriseLogCapture(logging.Handler):
@contextmanager
def _capture_apprise_logs(*, min_level: int = logging.INFO) -> Iterator[list[tuple[int, str, str, str]]]:
def _capture_apprise_logs(
*, min_level: int = logging.INFO
) -> Iterator[list[tuple[int, str, str, str]]]:
apprise_logger = logging.getLogger(_APPRISE_LOGGER_NAME)
previous_level = apprise_logger.level
handler = _AppriseLogCapture(thread_id=threading.get_ident())
@@ -170,7 +203,7 @@ def _log_apprise_exception_debug(*, action: str, scheme: str, exc: Exception) ->
type(exc).__name__,
scheme,
exc,
exc_info=True,
exc_info=(type(exc), exc, exc.__traceback__),
)
@@ -197,7 +230,7 @@ def _build_apprise_warning_detail(
return None
def _normalize_routes(value: Any) -> list[dict[str, str]]:
def _normalize_routes(value: object) -> list[dict[str, str]]:
if not isinstance(value, list):
return []
@@ -248,14 +281,8 @@ def _resolve_admin_routes() -> list[dict[str, str]]:
return _normalize_routes(app_config.get("ADMIN_NOTIFICATION_ROUTES", []))
def _normalize_user_id(value: Any) -> int | None:
try:
user_id = int(value)
except (TypeError, ValueError):
return None
if user_id < 1:
return None
return user_id
def _normalize_user_id(value: object) -> int | None:
return normalize_positive_int(value)
def _resolve_user_routes(user_id: int | None) -> list[dict[str, str]]:
@@ -291,7 +318,7 @@ def _resolve_route_urls_for_event(
return selected
def _resolve_notify_type(event: NotificationEvent) -> Any:
def _resolve_notify_type(event: NotificationEvent) -> object:
if apprise is None:
fallback = {
NotificationEvent.REQUEST_CREATED: "info",
@@ -312,7 +339,7 @@ def _resolve_notify_type(event: NotificationEvent) -> Any:
return mapping[event]
def _clean_text(value: Any, fallback: str) -> str:
def _clean_text(value: object, fallback: str) -> str:
text = str(value or "").strip()
return text or fallback
@@ -330,7 +357,10 @@ def _render_message(context: NotificationContext) -> tuple[str, str]:
if event == NotificationEvent.REQUEST_REJECTED:
note = _clean_text(context.admin_note, "")
note_line = f"\nNote: {note}" if note else ""
return "Request Rejected", f'Request for "{title}" by {author} was rejected.{note_line}'
return (
"Request Rejected",
f'Request for "{title}" by {author} was rejected.{note_line}',
)
if event == NotificationEvent.DOWNLOAD_COMPLETE:
return "Download Complete", f'"{title}" by {author} downloaded successfully.'
@@ -339,7 +369,7 @@ def _render_message(context: NotificationContext) -> tuple[str, str]:
return "Download Failed", f'Failed to download "{title}" by {author}.{error_line}'
def _plugin_label(plugin: Any, fallback_scheme: str) -> str:
def _plugin_label(plugin: object, fallback_scheme: str) -> str:
"""Build a human-readable label from a validated Apprise plugin.
Combines the URL scheme with the plugin's service name (app_id) and
@@ -351,10 +381,9 @@ def _plugin_label(plugin: Any, fallback_scheme: str) -> str:
app_id = getattr(plugin, "app_id", None)
if app_id and str(app_id) != fallback_scheme:
privacy_url: str | None = None
try:
privacy_url = plugin.url(privacy=True)
except Exception:
pass
if _has_plugin_url(plugin):
with suppress(Exception):
privacy_url = plugin.url(privacy=True)
suffix = str(app_id)
if privacy_url:
@@ -369,7 +398,7 @@ def _dispatch_to_apprise(
*,
title: str,
body: str,
notify_type: Any,
notify_type: object,
) -> dict[str, Any]:
normalized_urls = _normalize_urls(list(urls))
url_schemes = _extract_url_schemes(normalized_urls)
@@ -395,7 +424,7 @@ def _dispatch_to_apprise(
with _capture_apprise_logs(min_level=logging.INFO) as apprise_records:
try:
plugin = apprise.Apprise.instantiate(url, asset=getattr(apobj, "asset", None))
except Exception as exc:
except _APPRISE_DISPATCH_ERRORS as exc:
logger.warning(
"Failed to register notification route URL for scheme '%s': %s",
scheme,
@@ -429,7 +458,7 @@ def _dispatch_to_apprise(
try:
delivered = bool(apobj.notify(title=title, body=body, notify_type=notify_type))
except Exception as exc:
except _APPRISE_DISPATCH_ERRORS as exc:
_log_apprise_records(apprise_records)
failed_delivery_urls += 1
logger.warning(
@@ -443,9 +472,7 @@ def _dispatch_to_apprise(
if warning_detail:
failure_details.append(warning_detail)
else:
failure_details.append(
f"{scheme}: notify raised {type(exc).__name__}: {exc}"
)
failure_details.append(f"{scheme}: notify raised {type(exc).__name__}: {exc}")
continue
_log_apprise_records(apprise_records)
@@ -502,7 +529,7 @@ def _dispatch_to_apprise(
return result
def _create_apprise_client() -> Any:
def _create_apprise_client() -> _AppriseClient | None:
if apprise is None:
return None
@@ -512,7 +539,8 @@ def _create_apprise_client() -> Any:
apprise_asset_cls = getattr(apprise, "AppriseAsset", None)
if apprise_asset_cls is None:
return apprise_cls()
client = apprise_cls()
return client if _is_apprise_client(client) else None
try:
asset = apprise_asset_cls(
@@ -522,20 +550,25 @@ def _create_apprise_client() -> Any:
)
except TypeError:
# Support older Apprise versions that do not expose image_url_logo.
asset = apprise_asset_cls(
app_id=_APPRISE_APP_ID,
app_desc=_APPRISE_APP_DESC,
)
except Exception:
return apprise_cls()
try:
asset = apprise_asset_cls(
app_id=_APPRISE_APP_ID,
app_desc=_APPRISE_APP_DESC,
)
except TypeError:
client = apprise_cls()
return client if _is_apprise_client(client) else None
try:
return apprise_cls(asset=asset)
except Exception:
return apprise_cls()
client = apprise_cls(asset=asset)
except TypeError:
client = apprise_cls()
return client if _is_apprise_client(client) else None
def _send_admin_event(event: NotificationEvent, context: NotificationContext, urls: list[str]) -> dict[str, Any]:
def _send_admin_event(
event: NotificationEvent, context: NotificationContext, urls: list[str]
) -> dict[str, Any]:
title, body = _render_message(context)
notify_type = _resolve_notify_type(event)
return _dispatch_to_apprise(urls, title=title, body=body, notify_type=notify_type)
@@ -550,11 +583,13 @@ def notify_admin(event: NotificationEvent, context: NotificationContext) -> None
try:
_executor.submit(_dispatch_admin_async, event, context, urls)
except Exception as exc:
except RuntimeError as exc:
logger.warning("Failed to queue admin notification '%s': %s", event.value, exc)
def notify_user(user_id: int | None, event: NotificationEvent, context: NotificationContext) -> None:
def notify_user(
user_id: int | None, event: NotificationEvent, context: NotificationContext
) -> None:
"""Send a per-user notification for an event if subscribed."""
normalized_user_id = _normalize_user_id(user_id)
if normalized_user_id is None:
@@ -567,7 +602,7 @@ def notify_user(user_id: int | None, event: NotificationEvent, context: Notifica
try:
_executor.submit(_dispatch_user_async, normalized_user_id, event, context, urls)
except Exception as exc:
except RuntimeError as exc:
logger.warning(
"Failed to queue user notification '%s' for user_id=%s: %s",
event.value,
@@ -576,10 +611,16 @@ def notify_user(user_id: int | None, event: NotificationEvent, context: Notifica
)
def _dispatch_admin_async(event: NotificationEvent, context: NotificationContext, urls: list[str]) -> None:
def _dispatch_admin_async(
event: NotificationEvent, context: NotificationContext, urls: list[str]
) -> None:
result = _send_admin_event(event, context, urls)
if not result.get("success", False):
logger.warning("Admin notification failed for event '%s': %s", event.value, result.get("message"))
logger.warning(
"Admin notification failed for event '%s': %s",
event.value,
result.get("message"),
)
def _dispatch_user_async(
+11 -7
View File
@@ -4,12 +4,15 @@ Handles group claim parsing, user info extraction, and user provisioning.
Flask route handlers are registered separately in main.py.
"""
from typing import Any, Dict, List, Optional
from typing import TYPE_CHECKING, Any
from shelfmark.core.external_user_linking import upsert_external_user
from shelfmark.core.user_db import UserDB
def parse_group_claims(id_token: Dict[str, Any], group_claim: str) -> List[str]:
if TYPE_CHECKING:
from shelfmark.core.user_db import UserDB
def parse_group_claims(id_token: dict[str, Any], group_claim: str) -> list[str]:
"""Extract group list from an ID token claim.
Supports list, comma-separated string, or pipe-separated string.
@@ -26,7 +29,7 @@ def parse_group_claims(id_token: Dict[str, Any], group_claim: str) -> List[str]:
return []
def extract_user_info(id_token: Dict[str, Any]) -> Dict[str, Any]:
def extract_user_info(id_token: dict[str, Any]) -> dict[str, Any]:
"""Extract user info from OIDC ID token claims.
Returns a dict with keys: oidc_subject, username, email, display_name.
@@ -47,11 +50,12 @@ def extract_user_info(id_token: Dict[str, Any]) -> Dict[str, Any]:
def provision_oidc_user(
db: UserDB,
user_info: Dict[str, Any],
is_admin: Optional[bool] = None,
user_info: dict[str, Any],
*,
is_admin: bool | None = None,
allow_email_link: bool = False,
allow_create: bool = True,
) -> Optional[Dict[str, Any]]:
) -> dict[str, Any] | None:
"""Create or update a user from OIDC claims.
Matching and collision handling use the shared external user linker:
+151 -48
View File
@@ -4,39 +4,64 @@ Registers /api/auth/oidc/login and /api/auth/oidc/callback endpoints.
Business logic remains in oidc_auth.py.
"""
from typing import Any
from urllib.parse import quote
from __future__ import annotations
from authlib.jose.errors import InvalidClaimError
from collections.abc import Mapping
from typing import TYPE_CHECKING, Any, Protocol, TypeGuard
from urllib.parse import urlencode, urlsplit, urlunsplit
from authlib.integrations.base_client.errors import OAuthError
from authlib.integrations.flask_client import OAuth
from authlib.jose.errors import InvalidClaimError
from flask import Flask, jsonify, redirect, request, session
from shelfmark.core.config import config as app_config
from shelfmark.core.logger import setup_logger
from shelfmark.core.oidc_auth import (
extract_user_info,
parse_group_claims,
provision_oidc_user,
)
from shelfmark.core.settings_registry import load_config_file
from shelfmark.core.user_db import UserDB
from shelfmark.download.network import get_ssl_verify
if TYPE_CHECKING:
from flask.typing import ResponseReturnValue
from shelfmark.core.user_db import UserDB
logger = setup_logger(__name__)
oauth = OAuth()
_RETURN_TO_SESSION_KEY = "oidc_return_to"
_OIDC_CLIENT_ERRORS = (OAuthError, OSError, RuntimeError, TypeError, ValueError)
def _normalize_claims(raw_claims: Any) -> dict[str, Any]:
class _ClaimsMappingLike(Protocol):
"""Protocol for Authlib claims payloads that expose a to_dict method."""
def to_dict(self) -> Mapping[object, object]: ...
def _has_claims_to_dict(candidate: object) -> TypeGuard[_ClaimsMappingLike]:
"""Return True when a claims object exposes a callable to_dict method."""
return callable(getattr(candidate, "to_dict", None))
def _normalize_claim_mapping(raw_claims: Mapping[object, object]) -> dict[str, Any]:
"""Return only string-keyed claims for downstream OIDC helpers."""
return {key: value for key, value in raw_claims.items() if isinstance(key, str)}
def _normalize_claims(raw_claims: object) -> dict[str, Any]:
"""Return a plain dict for claims from Authlib token/userinfo payloads."""
if raw_claims is None:
return {}
if isinstance(raw_claims, dict):
return raw_claims
if hasattr(raw_claims, "to_dict"):
return raw_claims.to_dict() # type: ignore[no-any-return]
try:
return dict(raw_claims)
except Exception:
return {}
if isinstance(raw_claims, Mapping):
return _normalize_claim_mapping(raw_claims)
if _has_claims_to_dict(raw_claims):
converted_claims = raw_claims.to_dict()
if isinstance(converted_claims, Mapping):
return _normalize_claim_mapping(converted_claims)
return {}
def _has_username_or_email(claims: dict[str, Any]) -> bool:
@@ -52,36 +77,97 @@ def _login_error_url(message: str) -> str:
"""Build a login URL (with script_root) that includes an OIDC error message."""
script_root = request.script_root.rstrip("/")
login_url = f"{script_root}/login" if script_root else "/login"
return f"{login_url}?oidc_error={quote(message)}"
params = {"oidc_error": message}
return_to = _get_pending_return_to()
if return_to and return_to != "/":
params["return_to"] = return_to
return f"{login_url}?{urlencode(params)}"
def _normalize_return_to(raw_return_to: object) -> str | None:
"""Return a safe app-relative post-login target."""
if not isinstance(raw_return_to, str):
return None
value = raw_return_to.strip()
if not value or not value.startswith("/") or value.startswith("//"):
return None
parsed = urlsplit(value)
if parsed.scheme or parsed.netloc:
return None
script_root = request.script_root.rstrip("/")
path = parsed.path or "/"
if script_root:
if path == script_root:
path = "/"
elif path.startswith(f"{script_root}/"):
path = path[len(script_root) :] or "/"
if path in {"/login", "/api"} or path.startswith(("/login/", "/api/")):
return None
return urlunsplit(("", "", path, parsed.query, parsed.fragment))
def _get_pending_return_to(*, clear: bool = False) -> str | None:
"""Read the pending post-login target from the session."""
raw_return_to = (
session.pop(_RETURN_TO_SESSION_KEY, None) if clear else session.get(_RETURN_TO_SESSION_KEY)
)
normalized = _normalize_return_to(raw_return_to)
if normalized is None and not clear:
session.pop(_RETURN_TO_SESSION_KEY, None)
return normalized
def _post_login_redirect_target(return_to: str | None) -> str:
"""Build the final redirect target, honoring script_root when present."""
normalized = _normalize_return_to(return_to) or "/"
script_root = request.script_root.rstrip("/")
if not script_root:
return normalized
if normalized == "/":
return f"{script_root}/"
return f"{script_root}{normalized}"
def _get_oidc_client() -> tuple[Any, dict[str, Any]]:
"""Register and return an OIDC client from the current security config."""
config = load_config_file("security")
discovery_url = config.get("OIDC_DISCOVERY_URL", "")
client_id = config.get("OIDC_CLIENT_ID", "")
discovery_url = str(app_config.get("OIDC_DISCOVERY_URL", "") or "")
client_id = str(app_config.get("OIDC_CLIENT_ID", "") or "")
if not discovery_url or not client_id:
raise ValueError("OIDC not configured")
msg = "OIDC not configured"
raise ValueError(msg)
configured_scopes = config.get("OIDC_SCOPES", ["openid", "email", "profile"])
configured_scopes = app_config.get("OIDC_SCOPES", ["openid", "email", "profile"])
if isinstance(configured_scopes, list):
scope_values = [str(scope).strip() for scope in configured_scopes if str(scope).strip()]
elif isinstance(configured_scopes, str):
delimiter = "," if "," in configured_scopes else " "
scope_values = [scope.strip() for scope in configured_scopes.split(delimiter) if scope.strip()]
scope_values = [
scope.strip() for scope in configured_scopes.split(delimiter) if scope.strip()
]
else:
scope_values = []
scopes = list(dict.fromkeys(["openid"] + scope_values))
scopes = list(dict.fromkeys(["openid", *scope_values]))
admin_group = config.get("OIDC_ADMIN_GROUP", "")
group_claim = config.get("OIDC_GROUP_CLAIM", "groups")
use_admin_group = config.get("OIDC_USE_ADMIN_GROUP", True)
admin_group_value = app_config.get("OIDC_ADMIN_GROUP", "")
admin_group = admin_group_value.strip() if isinstance(admin_group_value, str) else ""
group_claim_value = app_config.get("OIDC_GROUP_CLAIM", "groups")
group_claim = (
group_claim_value.strip()
if isinstance(group_claim_value, str) and group_claim_value.strip()
else "groups"
)
use_admin_group = app_config.get("OIDC_USE_ADMIN_GROUP", True)
if admin_group and use_admin_group and group_claim and group_claim not in scopes:
scopes.append(group_claim)
def _ssl_compliance_fix(session, **kwargs):
def _ssl_compliance_fix(session: Any, **kwargs: Any) -> Any:
"""Set session.verify based on the Certificate Validation setting."""
session.verify = get_ssl_verify(discovery_url)
return session
@@ -90,7 +176,7 @@ def _get_oidc_client() -> tuple[Any, dict[str, Any]]:
oauth.register(
name="shelfmark_idp",
client_id=client_id,
client_secret=config.get("OIDC_CLIENT_SECRET", ""),
client_secret=app_config.get("OIDC_CLIENT_SECRET", ""),
server_metadata_url=discovery_url,
client_kwargs={
"scope": " ".join(scopes),
@@ -102,9 +188,16 @@ def _get_oidc_client() -> tuple[Any, dict[str, Any]]:
client = oauth.create_client("shelfmark_idp")
if client is None:
raise RuntimeError("OIDC client initialization failed")
msg = "OIDC client initialization failed"
raise RuntimeError(msg)
return client, config
return client, {
"OIDC_DISCOVERY_URL": discovery_url,
"OIDC_GROUP_CLAIM": group_claim,
"OIDC_ADMIN_GROUP": admin_group,
"OIDC_AUTO_PROVISION": app_config.get("OIDC_AUTO_PROVISION", True),
"OIDC_USE_ADMIN_GROUP": use_admin_group,
}
def register_oidc_routes(app: Flask, user_db: UserDB) -> None:
@@ -112,25 +205,30 @@ def register_oidc_routes(app: Flask, user_db: UserDB) -> None:
oauth.init_app(app)
@app.route("/api/auth/oidc/login", methods=["GET"])
def oidc_login():
def oidc_login() -> ResponseReturnValue:
"""Initiate OIDC login flow and redirect to the provider."""
try:
client, _ = _get_oidc_client()
return_to = _normalize_return_to(request.args.get("return_to"))
if return_to and return_to != "/":
session[_RETURN_TO_SESSION_KEY] = return_to
else:
session.pop(_RETURN_TO_SESSION_KEY, None)
redirect_uri = request.url_root.rstrip("/") + "/api/auth/oidc/callback"
return client.authorize_redirect(redirect_uri)
except ValueError:
return jsonify({"error": "OIDC not configured"}), 500
except Exception as e:
logger.error(f"OIDC login error: {e}")
except Exception:
logger.exception("OIDC login error")
return jsonify({"error": "OIDC login failed"}), 500
@app.route("/api/auth/oidc/callback", methods=["GET"])
def oidc_callback():
def oidc_callback() -> ResponseReturnValue:
"""Handle OIDC callback from identity provider."""
try:
error = request.args.get("error")
if error:
logger.warning(f"OIDC callback error from IdP: {error}")
logger.warning("OIDC callback error from IdP: %s", error)
return redirect(_login_error_url("Authentication failed"))
client, config = _get_oidc_client()
@@ -144,13 +242,15 @@ def register_oidc_routes(app: Flask, user_db: UserDB) -> None:
metadata = client.load_server_metadata()
if isinstance(metadata, dict):
provider_issuer = str(metadata.get("issuer", ""))
except Exception as metadata_error:
logger.debug(f"OIDC metadata lookup failed during claim diagnostics: {metadata_error}")
except _OIDC_CLIENT_ERRORS as metadata_error:
logger.debug(
"OIDC metadata lookup failed during claim diagnostics: %s",
metadata_error,
)
logger.error(
"OIDC callback claim validation failed: claim=%s error=%s discovery_url=%s provider_issuer=%s",
logger.exception(
"OIDC callback claim validation failed: claim=%s discovery_url=%s provider_issuer=%s",
claim_name,
e,
discovery_url or "<unset>",
provider_issuer or "<unknown>",
)
@@ -161,7 +261,9 @@ def register_oidc_routes(app: Flask, user_db: UserDB) -> None:
)
return redirect(_login_error_url(msg))
return redirect(_login_error_url(f"OIDC token claim validation failed: {claim_name}"))
return redirect(
_login_error_url(f"OIDC token claim validation failed: {claim_name}")
)
claims = _normalize_claims(token.get("userinfo"))
# If userinfo is missing or claims are too sparse, request it explicitly.
@@ -171,8 +273,8 @@ def register_oidc_routes(app: Flask, user_db: UserDB) -> None:
fetched_claims = _normalize_claims(client.userinfo(token=token))
except TypeError:
fetched_claims = _normalize_claims(client.userinfo())
except Exception as e:
logger.error(f"Failed to fetch OIDC userinfo: {e}")
except Exception:
logger.exception("Failed to fetch OIDC userinfo")
if fetched_claims:
claims = {**claims, **fetched_claims}
@@ -203,7 +305,8 @@ def register_oidc_routes(app: Flask, user_db: UserDB) -> None:
)
if user is None:
logger.warning(
f"OIDC login rejected: auto-provision disabled for {user_info['username']}"
"OIDC login rejected: auto-provision disabled for %s",
user_info["username"],
)
return redirect(_login_error_url("Account not found. Contact your administrator."))
@@ -212,12 +315,12 @@ def register_oidc_routes(app: Flask, user_db: UserDB) -> None:
session["db_user_id"] = user["id"]
session.permanent = True
logger.info(f"OIDC login successful: {user['username']} (admin={is_admin})")
return redirect(request.script_root or "/")
logger.info("OIDC login successful: %s (admin=%s)", user["username"], is_admin)
return redirect(_post_login_redirect_target(_get_pending_return_to(clear=True)))
except ValueError as e:
logger.error(f"OIDC callback error: {e}")
logger.exception("OIDC callback error")
return redirect(_login_error_url(str(e)))
except Exception as e:
logger.error(f"OIDC callback error: {e}")
except Exception:
logger.exception("OIDC callback error")
return redirect(_login_error_url("Authentication failed"))
+455 -147
View File
@@ -1,5 +1,4 @@
"""
Onboarding wizard configuration.
"""Onboarding wizard configuration.
Defines the steps and fields for the first-run onboarding experience.
Reuses field definitions from the settings registry where possible.
@@ -8,27 +7,32 @@ Reuses field definitions from the settings registry where possible.
import json
from dataclasses import replace
from pathlib import Path
from typing import Any, Dict, List, Optional
from typing import Any
from shelfmark.core.logger import setup_logger
from shelfmark.core.settings_registry import (
HeadingField,
MultiSelectField,
SettingsField,
get_settings_tab,
serialize_field,
save_config_file,
get_setting_value,
get_settings_field_map,
get_settings_tab,
save_config_file,
serialize_field,
)
logger = setup_logger(__name__)
ONBOARDING_STORAGE_KEY = "onboarding_complete"
ONBOARDING_RELEASE_SOURCES_KEY = "ONBOARDING_RELEASE_SOURCES"
_ONBOARDING_VIRTUAL_KEYS = {ONBOARDING_RELEASE_SOURCES_KEY}
def _get_config_dir() -> Path:
"""Get the config directory path."""
from shelfmark.config.env import CONFIG_DIR
return Path(CONFIG_DIR)
@@ -45,11 +49,11 @@ def is_onboarding_complete() -> bool:
return False
try:
with open(config_file, 'r') as f:
with config_file.open() as f:
config = json.load(f)
return config.get(ONBOARDING_STORAGE_KEY, False)
except (json.JSONDecodeError, OSError) as e:
logger.warning(f"Could not read onboarding status from settings.json: {e}")
logger.warning("Could not read onboarding status from settings.json: %s", e)
return False
@@ -57,14 +61,13 @@ def mark_onboarding_complete() -> bool:
"""Mark onboarding as complete."""
try:
return save_config_file("general", {ONBOARDING_STORAGE_KEY: True})
except Exception as e:
logger.error(f"Failed to mark onboarding complete: {e}")
except Exception:
logger.exception("Failed to mark onboarding complete")
return False
def _get_field_from_tab(tab_name: str, field_key: str) -> Optional[SettingsField]:
"""
Extract a specific field from a registered settings tab.
def _get_field_from_tab(tab_name: str, field_key: str) -> SettingsField | None:
"""Extract a specific field from a registered settings tab.
Args:
tab_name: Name of the settings tab (e.g., 'search_mode', 'hardcover')
@@ -72,37 +75,145 @@ def _get_field_from_tab(tab_name: str, field_key: str) -> Optional[SettingsField
Returns:
The field if found, None otherwise
"""
tab = get_settings_tab(tab_name)
if not tab:
logger.warning(f"Settings tab not found: {tab_name}")
logger.warning("Settings tab not found: %s", tab_name)
return None
for field in tab.fields:
if hasattr(field, 'key') and field.key == field_key:
if hasattr(field, "key") and field.key == field_key:
return field
logger.warning(f"Field {field_key} not found in tab {tab_name}")
logger.warning("Field %s not found in tab %s", field_key, tab_name)
return None
def _clone_field_with_overrides(field: SettingsField, **overrides) -> SettingsField:
"""
Clone a field with optional attribute overrides.
def _get_field_tab_name(field: SettingsField, fallback_tab_name: str) -> str:
"""Return the owning settings tab for a value field."""
field_key = getattr(field, "key", None)
if not field_key:
return fallback_tab_name
field_map = get_settings_field_map()
field_entry = field_map.get(field_key)
if field_entry is None:
return fallback_tab_name
return field_entry[1]
def _clone_field_with_overrides(field: SettingsField, **overrides: object) -> SettingsField:
"""Clone a field with optional attribute overrides.
Useful for customizing labels, descriptions, or defaults for onboarding context.
"""
return replace(field, **overrides)
def _get_fields_from_tab(
tab_name: str,
field_keys: list[str],
*,
strip_show_when_keys: set[str] | None = None,
) -> list[SettingsField]:
"""Return the requested fields from a settings tab in the supplied order."""
fields: list[SettingsField] = []
for field_key in field_keys:
field = _get_field_from_tab(tab_name, field_key)
if field:
show_when = getattr(field, "show_when", None)
stripped_show_when = _strip_show_when_keys(show_when, strip_show_when_keys or set())
if stripped_show_when != show_when:
field = replace(field, show_when=stripped_show_when)
fields.append(field)
return fields
def _strip_show_when_keys(
show_when: dict[str, Any] | list[dict[str, Any]] | None,
field_keys: set[str],
) -> dict[str, Any] | list[dict[str, Any]] | None:
"""Remove conditions tied to fields that onboarding handles implicitly."""
if not show_when or not field_keys:
return show_when
if isinstance(show_when, list):
remaining = [
condition for condition in show_when if condition.get("field") not in field_keys
]
return remaining or None
if show_when.get("field") in field_keys:
return None
return show_when
def _is_release_source_selected(values: dict[str, Any], source_name: str) -> bool:
"""Return True when a release source has been chosen during onboarding."""
raw_sources = values.get(ONBOARDING_RELEASE_SOURCES_KEY, [])
if not isinstance(raw_sources, list):
return False
return source_name in raw_sources
def _evaluate_show_when_condition(condition: dict[str, Any], values: dict[str, Any]) -> bool:
"""Evaluate one onboarding show_when condition against submitted values."""
current_value = values.get(condition["field"])
expected_value = condition.get("value")
if condition.get("notEmpty"):
if isinstance(current_value, list):
return len(current_value) > 0
return current_value not in (None, "")
if isinstance(current_value, list):
if isinstance(expected_value, list):
return all(item in current_value for item in expected_value)
return expected_value in current_value
if isinstance(expected_value, list):
return current_value in expected_value
return current_value == expected_value
def _is_step_visible(step_config: dict[str, Any], values: dict[str, Any]) -> bool:
"""Return True when a step should be included for the provided values."""
show_when = step_config.get("show_when")
if not show_when:
return True
return all(_evaluate_show_when_condition(condition, values) for condition in show_when)
def _is_field_visible(field: SettingsField, values: dict[str, Any]) -> bool:
"""Return True when a field should be included in the onboarding save."""
if getattr(field, "hidden_in_ui", False):
return False
if getattr(field, "universal_only", False) and values.get("SEARCH_MODE") != "universal":
return False
show_when = getattr(field, "show_when", None)
if not show_when:
return True
if isinstance(show_when, list):
return all(_evaluate_show_when_condition(condition, values) for condition in show_when)
return _evaluate_show_when_condition(show_when, values)
# =============================================================================
# Step Definitions
# =============================================================================
def get_search_mode_fields() -> List[SettingsField]:
def get_search_mode_fields() -> list[SettingsField]:
"""Step 1: Choose search mode - uses actual SEARCH_MODE field from settings."""
fields: List[SettingsField] = [
fields: list[SettingsField] = [
HeadingField(
key="welcome_heading",
title="Welcome to Shelfmark",
@@ -114,17 +225,19 @@ def get_search_mode_fields() -> List[SettingsField]:
search_mode_field = _get_field_from_tab("search_mode", "SEARCH_MODE")
if search_mode_field:
# Clone with onboarding-specific description
fields.append(_clone_field_with_overrides(
search_mode_field,
description="Choose how you want to find books.",
))
fields.append(
_clone_field_with_overrides(
search_mode_field,
description="Choose how you want to find books.",
)
)
return fields
def get_metadata_provider_fields() -> List[SettingsField]:
def get_metadata_provider_fields() -> list[SettingsField]:
"""Step 2: Choose metadata provider - uses actual METADATA_PROVIDER field."""
fields: List[SettingsField] = [
fields: list[SettingsField] = [
HeadingField(
key="metadata_heading",
title="Metadata Provider",
@@ -155,18 +268,20 @@ def get_metadata_provider_fields() -> List[SettingsField]:
]
# Clone with onboarding-specific options and default
fields.append(_clone_field_with_overrides(
provider_field,
default="hardcover",
options=onboarding_options,
))
fields.append(
_clone_field_with_overrides(
provider_field,
default="hardcover",
options=onboarding_options,
)
)
return fields
def get_hardcover_setup_fields() -> List[SettingsField]:
def get_hardcover_setup_fields() -> list[SettingsField]:
"""Step 3a: Configure Hardcover - uses actual API key and test connection fields."""
fields: List[SettingsField] = [
fields: list[SettingsField] = [
HeadingField(
key="hardcover_setup_heading",
title="Hardcover Setup",
@@ -189,9 +304,9 @@ def get_hardcover_setup_fields() -> List[SettingsField]:
return fields
def get_googlebooks_setup_fields() -> List[SettingsField]:
def get_googlebooks_setup_fields() -> list[SettingsField]:
"""Step 3b: Configure Google Books - uses actual API key and test connection fields."""
fields: List[SettingsField] = [
fields: list[SettingsField] = [
HeadingField(
key="googlebooks_setup_heading",
title="Google Books Setup",
@@ -214,129 +329,270 @@ def get_googlebooks_setup_fields() -> List[SettingsField]:
return fields
def get_prowlarr_fields() -> List[SettingsField]:
"""Step 4: Configure Prowlarr connection - uses actual Prowlarr fields."""
fields: List[SettingsField] = [
def get_release_source_selection_fields() -> list[SettingsField]:
"""Choose which release sources to configure during onboarding."""
fields: list[SettingsField] = [
HeadingField(
key="prowlarr_heading",
title="Prowlarr Integration (Optional)",
description="Connect to Prowlarr to search your indexers for torrents and NZBs. Skip this step if you only want to use Direct Download.",
key="release_sources_heading",
title="Release Sources",
description=(
"Choose the release sources you want to configure now. You can always add or "
"change sources later in Settings."
),
),
MultiSelectField(
key=ONBOARDING_RELEASE_SOURCES_KEY,
label="Sources to Set Up",
description="Select one or more release sources to configure now.",
default=[],
variant="dropdown",
env_supported=False,
options=[
{
"value": "direct_download",
"label": "Direct Download",
"description": "Configure your own Anna's Archive mirror URLs for direct ebook downloads.",
},
{
"value": "prowlarr",
"label": "Prowlarr",
"description": "Search your torrent and Usenet indexers through Prowlarr.",
},
{
"value": "audiobookbay",
"label": "AudiobookBay",
"description": "Search AudiobookBay directly for audiobook releases.",
},
{
"value": "irc",
"label": "IRC",
"description": "Connect to IRC for ebook and audiobook release searches.",
},
],
),
]
# Get actual Prowlarr connection fields
prowlarr_fields = ["PROWLARR_ENABLED", "PROWLARR_URL", "PROWLARR_API_KEY", "test_prowlarr"]
for field_key in prowlarr_fields:
field = _get_field_from_tab("prowlarr_config", field_key)
if field:
fields.append(field)
return fields
def get_prowlarr_indexers_fields() -> List[SettingsField]:
"""Step 5: Select Prowlarr indexers to search."""
fields: List[SettingsField] = [
def get_direct_download_setup_fields() -> list[SettingsField]:
"""Render trimmed direct-download essentials for onboarding."""
fields: list[SettingsField] = [
HeadingField(
key="prowlarr_indexers_heading",
title="Select Indexers",
description="Choose which indexers to search for books. Leave empty to search all available indexers.",
),
key="direct_download_setup_onboarding_heading",
title="Direct Download Setup",
description=(
"Add at least one Anna's Archive mirror URL to enable Direct Download. If you "
"have an Anna's Archive donator key, you can add it here too. You can configure "
"alternative mirrors later in Settings."
),
)
]
# Get the indexers multi-select field
indexers_field = _get_field_from_tab("prowlarr_config", "PROWLARR_INDEXERS")
if indexers_field:
fields.append(indexers_field)
fields.extend(_get_fields_from_tab("download_sources", ["AA_DONATOR_KEY"]))
fields.extend(_get_fields_from_tab("mirrors", ["AA_MIRROR_URLS"]))
return fields
# =============================================================================
# Step Configuration
# =============================================================================
ONBOARDING_STEPS = [
{
"id": "search_mode",
"title": "Search Mode",
"tab": "search_mode",
"get_fields": get_search_mode_fields,
},
{
"id": "metadata_provider",
"title": "Metadata Provider",
"tab": "search_mode",
"get_fields": get_metadata_provider_fields,
"show_when": [{"field": "SEARCH_MODE", "value": "universal"}],
},
{
"id": "hardcover_setup",
"title": "Hardcover Setup",
"tab": "hardcover",
"get_fields": get_hardcover_setup_fields,
# Must be universal mode AND hardcover selected
"show_when": [
{"field": "SEARCH_MODE", "value": "universal"},
{"field": "METADATA_PROVIDER", "value": "hardcover"},
def get_direct_download_bypass_fields() -> list[SettingsField]:
"""Render only the core Cloudflare bypass fields for onboarding."""
return _get_fields_from_tab(
"cloudflare_bypass",
[
"USE_CF_BYPASS",
"USING_EXTERNAL_BYPASSER",
"EXT_BYPASSER_URL",
"EXT_BYPASSER_PATH",
],
},
{
"id": "googlebooks_setup",
"title": "Google Books Setup",
"tab": "googlebooks",
"get_fields": get_googlebooks_setup_fields,
# Must be universal mode AND googlebooks selected
"show_when": [
{"field": "SEARCH_MODE", "value": "universal"},
{"field": "METADATA_PROVIDER", "value": "googlebooks"},
],
},
{
"id": "prowlarr",
"title": "Prowlarr",
"tab": "prowlarr_config",
"get_fields": get_prowlarr_fields,
"show_when": [{"field": "SEARCH_MODE", "value": "universal"}],
"optional": True,
},
{
"id": "prowlarr_indexers",
"title": "Indexers",
"tab": "prowlarr_config",
"get_fields": get_prowlarr_indexers_fields,
# Only show when Prowlarr is enabled
"show_when": [
{"field": "SEARCH_MODE", "value": "universal"},
{"field": "PROWLARR_ENABLED", "value": True},
],
"optional": True,
},
]
)
def get_onboarding_config() -> Dict[str, Any]:
"""
Get the full onboarding configuration including steps and current values.
"""
def get_prowlarr_fields() -> list[SettingsField]:
"""Render trimmed Prowlarr setup fields for onboarding."""
return _get_fields_from_tab(
"prowlarr_config",
[
"prowlarr_heading",
"PROWLARR_URL",
"PROWLARR_API_KEY",
"test_prowlarr",
"PROWLARR_INDEXERS",
],
strip_show_when_keys={"PROWLARR_ENABLED"},
)
def get_audiobookbay_fields() -> list[SettingsField]:
"""Render trimmed AudiobookBay setup fields for onboarding."""
return [
HeadingField(
key="audiobookbay_onboarding_heading",
title="AudiobookBay",
description="Add the AudiobookBay domain you want Shelfmark to search.",
),
*_get_fields_from_tab(
"audiobookbay_config",
["ABB_HOSTNAME"],
strip_show_when_keys={"ABB_ENABLED"},
),
]
def get_irc_fields() -> list[SettingsField]:
"""Render trimmed IRC setup fields for onboarding."""
return _get_fields_from_tab(
"irc",
[
"heading",
"IRC_SERVER",
"IRC_PORT",
"IRC_USE_TLS",
"IRC_CHANNEL",
"IRC_NICK",
"IRC_SEARCH_BOT",
],
)
def get_onboarding_steps() -> list[dict[str, Any]]:
"""Return the full onboarding step configuration."""
return [
{
"id": "search_mode",
"title": "Search Mode",
"tab": "search_mode",
"get_fields": get_search_mode_fields,
},
{
"id": "metadata_provider",
"title": "Metadata Provider",
"tab": "search_mode",
"get_fields": get_metadata_provider_fields,
"show_when": [{"field": "SEARCH_MODE", "value": "universal"}],
},
{
"id": "hardcover_setup",
"title": "Hardcover Setup",
"tab": "hardcover",
"get_fields": get_hardcover_setup_fields,
"show_when": [
{"field": "SEARCH_MODE", "value": "universal"},
{"field": "METADATA_PROVIDER", "value": "hardcover"},
],
},
{
"id": "googlebooks_setup",
"title": "Google Books Setup",
"tab": "googlebooks",
"get_fields": get_googlebooks_setup_fields,
"show_when": [
{"field": "SEARCH_MODE", "value": "universal"},
{"field": "METADATA_PROVIDER", "value": "googlebooks"},
],
},
{
"id": "release_sources",
"title": "Release Sources",
"tab": "search_mode",
"get_fields": get_release_source_selection_fields,
"show_when": [{"field": "SEARCH_MODE", "value": "universal"}],
"optional": True,
},
{
"id": "direct_download_setup_direct_mode",
"title": "Direct Download Setup",
"tab": "download_sources",
"get_fields": get_direct_download_setup_fields,
"show_when": [{"field": "SEARCH_MODE", "value": "direct"}],
},
{
"id": "direct_download_cloudflare_bypass_direct_mode",
"title": "Cloudflare Bypass",
"tab": "cloudflare_bypass",
"get_fields": get_direct_download_bypass_fields,
"show_when": [{"field": "SEARCH_MODE", "value": "direct"}],
},
{
"id": "direct_download_setup",
"title": "Direct Download Setup",
"tab": "download_sources",
"get_fields": get_direct_download_setup_fields,
"show_when": [
{"field": "SEARCH_MODE", "value": "universal"},
{"field": ONBOARDING_RELEASE_SOURCES_KEY, "value": "direct_download"},
],
"optional": True,
},
{
"id": "direct_download_cloudflare_bypass",
"title": "Cloudflare Bypass",
"tab": "cloudflare_bypass",
"get_fields": get_direct_download_bypass_fields,
"show_when": [
{"field": "SEARCH_MODE", "value": "universal"},
{"field": ONBOARDING_RELEASE_SOURCES_KEY, "value": "direct_download"},
],
"optional": True,
},
{
"id": "prowlarr",
"title": "Prowlarr",
"tab": "prowlarr_config",
"get_fields": get_prowlarr_fields,
"show_when": [
{"field": "SEARCH_MODE", "value": "universal"},
{"field": ONBOARDING_RELEASE_SOURCES_KEY, "value": "prowlarr"},
],
"optional": True,
},
{
"id": "audiobookbay",
"title": "AudiobookBay",
"tab": "audiobookbay_config",
"get_fields": get_audiobookbay_fields,
"show_when": [
{"field": "SEARCH_MODE", "value": "universal"},
{"field": ONBOARDING_RELEASE_SOURCES_KEY, "value": "audiobookbay"},
],
"optional": True,
},
{
"id": "irc",
"title": "IRC",
"tab": "irc",
"get_fields": get_irc_fields,
"show_when": [
{"field": "SEARCH_MODE", "value": "universal"},
{"field": ONBOARDING_RELEASE_SOURCES_KEY, "value": "irc"},
],
"optional": True,
},
]
def get_onboarding_config() -> dict[str, Any]:
"""Get the full onboarding configuration including steps and current values."""
steps = []
all_values = {}
for step_config in ONBOARDING_STEPS:
for step_config in get_onboarding_steps():
fields = step_config["get_fields"]()
tab_name = step_config["tab"]
# Serialize fields with current values
serialized_fields = []
for field in fields:
serialized = serialize_field(field, tab_name, include_value=True)
field_tab_name = _get_field_tab_name(field, tab_name)
serialized = serialize_field(field, field_tab_name, include_value=True)
serialized_fields.append(serialized)
# Collect values (skip HeadingFields)
if hasattr(field, 'key') and field.key and not isinstance(field, HeadingField):
value = get_setting_value(field, tab_name)
all_values[field.key] = value if value is not None else getattr(field, 'default', '')
if hasattr(field, "env_supported") and getattr(field, "key", None):
if field.key in _ONBOARDING_VIRTUAL_KEYS:
value = getattr(field, "default", "")
else:
value = get_setting_value(field, field_tab_name)
all_values[field.key] = (
value if value is not None else getattr(field, "default", "")
)
step = {
"id": step_config["id"],
@@ -359,22 +615,24 @@ def get_onboarding_config() -> Dict[str, Any]:
}
def save_onboarding_settings(values: Dict[str, Any]) -> Dict[str, Any]:
"""
Save onboarding settings and mark as complete.
def save_onboarding_settings(values: dict[str, Any]) -> dict[str, Any]:
"""Save onboarding settings and mark as complete.
Args:
values: Dict of field key -> value
Returns:
Dict with success status and message
"""
try:
# Group values by their target tab
tab_values: Dict[str, Dict[str, Any]] = {}
tab_values: dict[str, dict[str, Any]] = {}
for step_config in get_onboarding_steps():
if not _is_step_visible(step_config, values):
continue
for step_config in ONBOARDING_STEPS:
tab_name = step_config["tab"]
fields = step_config["get_fields"]()
for field in fields:
@@ -382,7 +640,12 @@ def save_onboarding_settings(values: Dict[str, Any]) -> Dict[str, Any]:
continue
key = field.key
if key in _ONBOARDING_VIRTUAL_KEYS:
continue
if not _is_field_visible(field, values):
continue
if key in values:
tab_name = _get_field_tab_name(field, step_config["tab"])
if tab_name not in tab_values:
tab_values[tab_name] = {}
tab_values[tab_name][key] = values[key]
@@ -391,10 +654,9 @@ def save_onboarding_settings(values: Dict[str, Any]) -> Dict[str, Any]:
for tab_name, tab_data in tab_values.items():
if tab_data:
save_config_file(tab_name, tab_data)
logger.info(f"Saved onboarding settings to {tab_name}: {list(tab_data.keys())}")
logger.info("Saved onboarding settings to %s: %s", tab_name, list(tab_data.keys()))
# Enable the selected metadata provider
search_mode = values.get("SEARCH_MODE", "direct")
search_mode = values.get("SEARCH_MODE", "universal")
if search_mode == "universal":
provider = values.get("METADATA_PROVIDER", "hardcover")
if provider:
@@ -416,7 +678,52 @@ def save_onboarding_settings(values: Dict[str, Any]) -> Dict[str, Any]:
provider_config["GOOGLEBOOKS_API_KEY"] = values["GOOGLEBOOKS_API_KEY"]
save_config_file(provider, provider_config)
logger.info(f"Enabled metadata provider: {provider} with keys: {list(provider_config.keys())}")
logger.info(
"Enabled metadata provider: %s with keys: %s",
provider,
list(provider_config.keys()),
)
selected_release_sources = values.get(ONBOARDING_RELEASE_SOURCES_KEY, [])
if not isinstance(selected_release_sources, list):
selected_release_sources = []
source_updates: dict[str, dict[str, Any]] = {}
if search_mode == "direct":
source_updates.setdefault("download_sources", {})["DIRECT_DOWNLOAD_ENABLED"] = True
else:
if _is_release_source_selected(values, "direct_download"):
source_updates.setdefault("download_sources", {})["DIRECT_DOWNLOAD_ENABLED"] = True
if _is_release_source_selected(values, "prowlarr"):
source_updates.setdefault("prowlarr_config", {})["PROWLARR_ENABLED"] = True
if _is_release_source_selected(values, "audiobookbay"):
source_updates.setdefault("audiobookbay_config", {})["ABB_ENABLED"] = True
if not values.get("DEFAULT_RELEASE_SOURCE"):
for source_name in selected_release_sources:
if source_name in {"direct_download", "prowlarr", "irc"}:
source_updates.setdefault("search_mode", {})["DEFAULT_RELEASE_SOURCE"] = (
source_name
)
break
if not values.get("DEFAULT_RELEASE_SOURCE_AUDIOBOOK"):
for source_name in selected_release_sources:
if source_name in {"prowlarr", "audiobookbay", "irc"}:
source_updates.setdefault("search_mode", {})[
"DEFAULT_RELEASE_SOURCE_AUDIOBOOK"
] = source_name
break
for tab_name, tab_data in source_updates.items():
if tab_data:
save_config_file(tab_name, tab_data)
logger.info(
"Enabled onboarding release source settings for %s: %s",
tab_name,
list(tab_data.keys()),
)
# Mark onboarding as complete
mark_onboarding_complete()
@@ -424,12 +731,13 @@ def save_onboarding_settings(values: Dict[str, Any]) -> Dict[str, Any]:
# Refresh config
try:
from shelfmark.core.config import config
config.refresh()
except ImportError as e:
logger.debug(f"Could not refresh config after onboarding: {e}")
return {"success": True, "message": "Onboarding complete!"}
logger.debug("Could not refresh config after onboarding: %s", e)
except Exception as e:
logger.error(f"Failed to save onboarding settings: {e}")
logger.exception("Failed to save onboarding settings")
return {"success": False, "message": str(e)}
else:
return {"success": True, "message": "Onboarding complete!"}
+25 -11
View File
@@ -11,11 +11,18 @@ from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Iterable, Optional
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from collections.abc import Iterable
_WINDOWS_DRIVE_PREFIX_LENGTH = 2
@dataclass(frozen=True)
class RemotePathMapping:
"""Mapping from a remote path prefix to a local path prefix."""
host: str
remote_path: str
local_path: str
@@ -36,14 +43,15 @@ def _normalize_prefix(path: str) -> str:
def _is_windows_path(path: str) -> bool:
"""Check if a path looks like a Windows path (has a drive letter like C:/)."""
return len(path) >= 2 and path[1] == ":" and path[0].isalpha()
return len(path) >= _WINDOWS_DRIVE_PREFIX_LENGTH and path[1] == ":" and path[0].isalpha()
def _normalize_host(host: str) -> str:
return str(host or "").strip().lower()
def parse_remote_path_mappings(value: Any) -> list[RemotePathMapping]:
def parse_remote_path_mappings(value: object) -> list[RemotePathMapping]:
"""Parse configured remote-path mapping rows into normalized mappings."""
if not value or not isinstance(value, list):
return []
@@ -60,7 +68,9 @@ def parse_remote_path_mappings(value: Any) -> list[RemotePathMapping]:
if not host or not remote_path or not local_path:
continue
mappings.append(RemotePathMapping(host=host, remote_path=remote_path, local_path=local_path))
mappings.append(
RemotePathMapping(host=host, remote_path=remote_path, local_path=local_path)
)
mappings.sort(key=lambda m: len(m.remote_path), reverse=True)
return mappings
@@ -72,6 +82,7 @@ def remap_remote_to_local_with_match(
host: str,
remote_path: str | Path,
) -> tuple[Path, bool]:
"""Remap a remote path and report whether a configured mapping matched."""
host_normalized = _normalize_host(host)
remote_normalized = _normalize_prefix(str(remote_path))
@@ -96,16 +107,17 @@ def remap_remote_to_local_with_match(
prefix_lower = remote_prefix.lower()
matches = remote_lower == prefix_lower or remote_lower.startswith(prefix_lower + "/")
else:
matches = remote_normalized == remote_prefix or remote_normalized.startswith(remote_prefix + "/")
matches = remote_normalized == remote_prefix or remote_normalized.startswith(
remote_prefix + "/"
)
if matches:
# Use the length of the original prefix to extract remainder
# This preserves the original case in folder names
remainder = remote_normalized[len(remote_prefix):]
remainder = remote_normalized[len(remote_prefix) :]
local_prefix = _normalize_prefix(mapping.local_path)
if remainder.startswith("/"):
remainder = remainder[1:]
remainder = remainder.removeprefix("/")
remapped = Path(local_prefix) / remainder if remainder else Path(local_prefix)
return remapped, True
@@ -113,7 +125,10 @@ def remap_remote_to_local_with_match(
return Path(remote_normalized), False
def remap_remote_to_local(*, mappings: Iterable[RemotePathMapping], host: str, remote_path: str | Path) -> Path:
def remap_remote_to_local(
*, mappings: Iterable[RemotePathMapping], host: str, remote_path: str | Path
) -> Path:
"""Remap a remote path to a local path using the configured mappings."""
remapped, _ = remap_remote_to_local_with_match(
mappings=mappings,
host=host,
@@ -122,13 +137,12 @@ def remap_remote_to_local(*, mappings: Iterable[RemotePathMapping], host: str, r
return remapped
def get_client_host_identifier(client: Any) -> Optional[str]:
def get_client_host_identifier(client: object) -> str | None:
"""Return a stable identifier used by the mapping UI.
Sonarr uses the download client's configured host. Shelfmark currently uses
the download client 'name' (e.g. qbittorrent, sabnzbd).
"""
name = getattr(client, "name", None)
if isinstance(name, str) and name.strip():
return name.strip().lower()
+16 -5
View File
@@ -2,19 +2,30 @@
from __future__ import annotations
from typing import Iterable, Optional
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from collections.abc import Callable, Iterable
class PrefixMiddleware:
"""Strip a configured URL prefix from PATH_INFO before routing."""
def __init__(self, app, prefix: str, bypass_paths: Optional[Iterable[str]] = None) -> None:
def __init__(
self,
app: Callable[[dict[str, object], Callable[..., object]], object],
prefix: str,
bypass_paths: Iterable[str] | None = None,
) -> None:
"""Initialize the middleware with a prefix and optional bypass paths."""
self.app = app
self.prefix = prefix.rstrip("/")
self.bypass_paths = set(bypass_paths or [])
def __call__(self, environ, start_response):
path = environ.get("PATH_INFO", "") or ""
def __call__(self, environ: dict[str, object], start_response: Callable[..., object]) -> object:
"""Rewrite prefixed requests before handing them to the wrapped app."""
raw_path = environ.get("PATH_INFO", "")
path = raw_path if isinstance(raw_path, str) else str(raw_path or "")
if path in self.bypass_paths:
return self.app(environ, start_response)
@@ -24,7 +35,7 @@ class PrefixMiddleware:
if path == self.prefix or path.startswith(self.prefix + "/"):
environ["SCRIPT_NAME"] = self.prefix
environ["PATH_INFO"] = path[len(self.prefix):] or "/"
environ["PATH_INFO"] = path[len(self.prefix) :] or "/"
return self.app(environ, start_response)
start_response("404 Not Found", [("Content-Type", "text/plain")])
+117 -86
View File
@@ -2,22 +2,46 @@
import queue
import time
from datetime import datetime, timedelta
from datetime import UTC, datetime, timedelta
from pathlib import Path
from threading import Lock, Event
from typing import Dict, List, Optional, Tuple, Any, Callable
from threading import Event, Lock
from typing import TYPE_CHECKING, Any
from shelfmark.core.config import config as app_config
from shelfmark.core.logger import setup_logger
from shelfmark.core.models import QueueStatus, QueueItem, DownloadTask, TERMINAL_QUEUE_STATUSES
from shelfmark.core.models import (
TERMINAL_QUEUE_STATUSES,
DownloadTask,
QueueItem,
QueueStatus,
)
if TYPE_CHECKING:
from collections.abc import Callable
logger = setup_logger(__name__)
_QUEUE_HOOK_ERRORS = (OSError, RuntimeError, TypeError, ValueError)
def _coerce_status_timeout_seconds(value: object, *, default: int) -> int:
"""Normalize STATUS_TIMEOUT into a usable positive integer."""
if isinstance(value, bool):
return default
if isinstance(value, int):
return value if value > 0 else default
if isinstance(value, str):
stripped = value.strip()
if stripped.isdigit():
parsed = int(stripped)
return parsed if parsed > 0 else default
return default
class BookQueue:
"""Thread-safe download queue manager with priority support and cancellation."""
def __init__(self) -> None:
"""Initialize queue state, locks, and lifecycle hooks."""
self._queue: queue.PriorityQueue[QueueItem] = queue.PriorityQueue()
self._lock = Lock()
self._status: dict[str, QueueStatus] = {}
@@ -25,24 +49,30 @@ class BookQueue:
self._status_timestamps: dict[str, datetime] = {} # Track when each status was last updated
self._cancel_flags: dict[str, Event] = {} # Cancellation flags for active downloads
self._active_downloads: dict[str, bool] = {} # Track currently downloading tasks
self._terminal_status_hook: Optional[
Callable[[str, QueueStatus, DownloadTask], None]
] = None
self._queue_hook: Optional[Callable[[str, DownloadTask], None]] = None
self._terminal_status_hook: Callable[[str, QueueStatus, DownloadTask], None] | None = None
self._queue_hook: Callable[[str, DownloadTask], None] | None = None
@property
def _status_timeout(self) -> timedelta:
"""Get status timeout from config (allows live updates)."""
return timedelta(seconds=app_config.get("STATUS_TIMEOUT", 3600))
return timedelta(
seconds=_coerce_status_timeout_seconds(
app_config.get("STATUS_TIMEOUT", 3600),
default=3600,
)
)
def add(self, task: DownloadTask) -> bool:
"""Add a download task to the queue. Returns False if already exists."""
hook: Optional[Callable[[str, DownloadTask], None]] = None
hook: Callable[[str, DownloadTask], None] | None = None
with self._lock:
task_id = task.task_id
# Don't add if already exists and not in error/cancelled state
if task_id in self._status and self._status[task_id] not in [QueueStatus.ERROR, QueueStatus.CANCELLED]:
if task_id in self._status and self._status[task_id] not in [
QueueStatus.ERROR,
QueueStatus.CANCELLED,
]:
return False
# Ensure added_time is set
@@ -58,11 +88,11 @@ class BookQueue:
if hook is not None:
try:
hook(task_id, task)
except Exception as exc:
except _QUEUE_HOOK_ERRORS as exc:
logger.warning("Queue hook failed while adding task %s: %s", task_id, exc)
return True
def get_next(self) -> Optional[Tuple[str, Event]]:
def get_next(self) -> tuple[str, Event] | None:
"""Get next task ID from queue with cancellation flag."""
# Use iterative approach to avoid stack overflow if many items are cancelled
while True:
@@ -79,29 +109,29 @@ class BookQueue:
cancel_flag = Event()
self._cancel_flags[task_id] = cancel_flag
self._active_downloads[task_id] = True
return task_id, cancel_flag
except queue.Empty:
return None
else:
return task_id, cancel_flag
def get_task(self, task_id: str) -> Optional[DownloadTask]:
def get_task(self, task_id: str) -> DownloadTask | None:
"""Get a task by its ID."""
with self._lock:
return self._task_data.get(task_id)
def get_task_status(self, task_id: str) -> Optional[QueueStatus]:
def get_task_status(self, task_id: str) -> QueueStatus | None:
"""Get queue status for a task id."""
with self._lock:
return self._status.get(task_id)
def _update_status(self, book_id: str, status: QueueStatus) -> None:
"""Internal method to update status and timestamp."""
"""Update the status and timestamp for a task."""
self._status[book_id] = status
self._status_timestamps[book_id] = datetime.now()
self._status_timestamps[book_id] = datetime.now(UTC)
def set_terminal_status_hook(
self,
hook: Optional[Callable[[str, QueueStatus, DownloadTask], None]],
hook: Callable[[str, QueueStatus, DownloadTask], None] | None,
) -> None:
"""Register a callback invoked when a task first enters a terminal status."""
with self._lock:
@@ -109,7 +139,7 @@ class BookQueue:
def set_queue_hook(
self,
hook: Optional[Callable[[str, DownloadTask], None]],
hook: Callable[[str, DownloadTask], None] | None,
) -> None:
"""Register a callback invoked when a task is added to the queue."""
with self._lock:
@@ -117,8 +147,8 @@ class BookQueue:
def update_status(self, book_id: str, status: QueueStatus) -> None:
"""Update status of a book in the queue."""
hook: Optional[Callable[[str, QueueStatus, DownloadTask], None]] = None
hook_task: Optional[DownloadTask] = None
hook: Callable[[str, QueueStatus, DownloadTask], None] | None = None
hook_task: DownloadTask | None = None
with self._lock:
previous_status = self._status.get(book_id)
self._update_status(book_id, status)
@@ -159,16 +189,19 @@ class BookQueue:
if task_id in self._task_data:
self._task_data[task_id].status_message = message
def get_status(self, user_id: Optional[int] = None) -> Dict[QueueStatus, Dict[str, DownloadTask]]:
def get_status(self, user_id: int | None = None) -> dict[QueueStatus, dict[str, DownloadTask]]:
"""Get current queue status grouped by status.
Args:
user_id: If provided, only return tasks belonging to this user.
If None, return all.
"""
self.refresh()
with self._lock:
result: Dict[QueueStatus, Dict[str, DownloadTask]] = {status: {} for status in QueueStatus}
result: dict[QueueStatus, dict[str, DownloadTask]] = {
status: {} for status in QueueStatus
}
for task_id, status in self._status.items():
if task_id in self._task_data:
task = self._task_data[task_id]
@@ -177,7 +210,7 @@ class BookQueue:
result[status][task_id] = task
return result
def get_queue_order(self) -> List[Dict[str, Any]]:
def get_queue_order(self) -> list[dict[str, Any]]:
"""Get current queue order for display."""
with self._lock:
queue_items = []
@@ -185,39 +218,42 @@ class BookQueue:
# Get items from priority queue without removing them
temp_items = []
while not self._queue.empty():
try:
item = self._queue.get_nowait()
temp_items.append(item)
task_id = item.book_id # QueueItem uses book_id as the ID field
if task_id in self._task_data:
task = self._task_data[task_id]
queue_items.append({
'id': task_id,
'title': task.title,
'author': task.author,
'priority': item.priority,
'added_time': item.added_time,
'status': self._status.get(task_id, QueueStatus.QUEUED)
})
except queue.Empty:
break
item = self._queue.get_nowait()
temp_items.append(item)
task_id = item.book_id # QueueItem uses book_id as the ID field
if task_id in self._task_data:
task = self._task_data[task_id]
queue_items.append(
{
"id": task_id,
"title": task.title,
"author": task.author,
"priority": item.priority,
"added_time": item.added_time,
"status": self._status.get(task_id, QueueStatus.QUEUED),
}
)
# Put items back in queue
for item in temp_items:
self._queue.put(item)
return sorted(queue_items, key=lambda x: (x['priority'], x['added_time']))
return sorted(queue_items, key=lambda x: (x["priority"], x["added_time"]))
def cancel_download(self, task_id: str) -> bool:
"""Cancel an active or queued download."""
with self._lock:
current_status = self._status.get(task_id)
if current_status in [QueueStatus.RESOLVING, QueueStatus.LOCATING, QueueStatus.DOWNLOADING]:
if current_status in [
QueueStatus.RESOLVING,
QueueStatus.LOCATING,
QueueStatus.DOWNLOADING,
]:
# Signal active download to stop
if task_id in self._cancel_flags:
self._cancel_flags[task_id].set()
elif current_status not in [QueueStatus.QUEUED]:
elif current_status != QueueStatus.QUEUED:
# Not in a cancellable state
return False
@@ -235,20 +271,17 @@ class BookQueue:
found = False
while not self._queue.empty():
try:
item = self._queue.get_nowait()
if item.book_id == task_id: # QueueItem uses book_id as the ID field
# Create new item with updated priority
new_item = QueueItem(task_id, new_priority, item.added_time)
temp_items.append(new_item)
found = True
# Update task data priority
if task_id in self._task_data:
self._task_data[task_id].priority = new_priority
else:
temp_items.append(item)
except queue.Empty:
break
item = self._queue.get_nowait()
if item.book_id == task_id: # QueueItem uses book_id as the ID field
# Create new item with updated priority
new_item = QueueItem(task_id, new_priority, item.added_time)
temp_items.append(new_item)
found = True
# Update task data priority
if task_id in self._task_data:
self._task_data[task_id].priority = new_priority
else:
temp_items.append(item)
# Put all items back
for item in temp_items:
@@ -256,13 +289,13 @@ class BookQueue:
return found
def enqueue_existing(self, task_id: str, *, priority: Optional[int] = None) -> bool:
def enqueue_existing(self, task_id: str, *, priority: int | None = None) -> bool:
"""Requeue an existing task regardless of current status.
This is used for retries where task metadata should be preserved.
"""
hook: Optional[Callable[[str, DownloadTask], None]] = None
hook_task: Optional[DownloadTask] = None
hook: Callable[[str, DownloadTask], None] | None = None
hook_task: DownloadTask | None = None
with self._lock:
task = self._task_data.get(task_id)
if task is None:
@@ -278,10 +311,7 @@ class BookQueue:
# De-duplicate queue entries for this task id.
temp_items: list[QueueItem] = []
while not self._queue.empty():
try:
item = self._queue.get_nowait()
except queue.Empty:
break
item = self._queue.get_nowait()
if item.book_id != task_id:
temp_items.append(item)
@@ -297,29 +327,26 @@ class BookQueue:
if hook is not None and hook_task is not None:
try:
hook(task_id, hook_task)
except Exception as exc:
except _QUEUE_HOOK_ERRORS as exc:
logger.warning("Queue hook failed while requeueing task %s: %s", task_id, exc)
return True
def reorder_queue(self, task_priorities: Dict[str, int]) -> bool:
def reorder_queue(self, task_priorities: dict[str, int]) -> bool:
"""Bulk reorder queue by mapping task_id to new priority."""
with self._lock:
# Extract all items from queue
all_items = []
while not self._queue.empty():
try:
item = self._queue.get_nowait()
task_id = item.book_id # QueueItem uses book_id as the ID field
# Update priority if specified
if task_id in task_priorities:
new_priority = task_priorities[task_id]
item = QueueItem(task_id, new_priority, item.added_time)
# Update task data priority
if task_id in self._task_data:
self._task_data[task_id].priority = new_priority
all_items.append(item)
except queue.Empty:
break
item = self._queue.get_nowait()
task_id = item.book_id # QueueItem uses book_id as the ID field
# Update priority if specified
if task_id in task_priorities:
new_priority = task_priorities[task_id]
item = QueueItem(task_id, new_priority, item.added_time)
# Update task data priority
if task_id in self._task_data:
self._task_data[task_id].priority = new_priority
all_items.append(item)
# Put all items back with updated priorities
for item in all_items:
@@ -327,7 +354,7 @@ class BookQueue:
return True
def get_active_downloads(self) -> List[str]:
def get_active_downloads(self) -> list[str]:
"""Get list of currently active download task IDs."""
with self._lock:
return list(self._active_downloads.keys())
@@ -343,7 +370,7 @@ class BookQueue:
"""Remove any tasks that are done downloading or have stale status."""
terminal_statuses = TERMINAL_QUEUE_STATUSES
with self._lock:
current_time = datetime.now()
current_time = datetime.now(UTC)
to_remove = []
for task_id, status in self._status.items():
@@ -357,9 +384,12 @@ class BookQueue:
# Check for stale status entries
last_update = self._status_timestamps.get(task_id)
if last_update and (current_time - last_update) > self._status_timeout:
if status in terminal_statuses:
to_remove.append(task_id)
if (
last_update
and (current_time - last_update) > self._status_timeout
and status in terminal_statuses
):
to_remove.append(task_id)
# Remove stale entries
for task_id in to_remove:
@@ -367,5 +397,6 @@ class BookQueue:
self._status_timestamps.pop(task_id, None)
self._task_data.pop(task_id, None)
# Global instance of BookQueue
book_queue = BookQueue()
+79 -24
View File
@@ -2,22 +2,55 @@
from __future__ import annotations
from datetime import datetime, timezone
from typing import Any
from datetime import UTC, datetime
from typing import Any, Protocol, SupportsIndex, SupportsInt, TypeGuard
from shelfmark.core.config import config as app_config
from shelfmark.core.logger import setup_logger
from shelfmark.core.settings_registry import load_config_file
_logger = setup_logger(__name__)
type _ConvertibleToInt = str | bytes | bytearray | SupportsInt | SupportsIndex
class _MappingWithGet(Protocol):
"""Minimal mapping protocol for session-like objects."""
def get(self, key: str, default: object = None, /) -> object: ...
class _UserDBLike(Protocol):
"""Minimal user DB protocol for username population helpers."""
def get_user(self, *, user_id: int) -> dict[str, Any] | None: ...
def _is_mapping_with_get(candidate: object) -> TypeGuard[_MappingWithGet]:
"""Return True when *candidate* exposes a mapping-style get method."""
return callable(getattr(candidate, "get", None))
def _is_user_db_like(candidate: object) -> TypeGuard[_UserDBLike]:
"""Return True when *candidate* exposes the user lookup API we need."""
return callable(getattr(candidate, "get_user", None))
def _is_convertible_to_int(value: object) -> TypeGuard[_ConvertibleToInt]:
"""Return True when *value* can be passed to ``int`` safely."""
return (
isinstance(value, (str, bytes, bytearray))
or hasattr(value, "__int__")
or hasattr(value, "__index__")
)
def now_utc_iso() -> str:
"""Return the current UTC time as a seconds-precision ISO 8601 string."""
return datetime.now(timezone.utc).isoformat(timespec="seconds")
return datetime.now(UTC).isoformat(timespec="seconds")
def emit_ws_event(
ws_manager: Any,
ws_manager: object,
*,
event_name: str,
payload: dict[str, Any],
@@ -32,16 +65,23 @@ def emit_ws_event(
if socketio is None or not callable(is_enabled) or not is_enabled():
return
socketio.emit(event_name, payload, to=room)
except Exception as exc:
_logger.warning("Failed to emit WebSocket event '%s' to room '%s': %s", event_name, room, exc)
except (AttributeError, RuntimeError, TypeError, ValueError) as exc:
_logger.warning(
"Failed to emit WebSocket event '%s' to room '%s': %s",
event_name,
room,
exc,
)
def load_users_request_policy_settings() -> dict[str, Any]:
"""Load global request-policy settings from the users config file."""
return load_config_file("users")
from shelfmark.core.request_policy import REQUEST_POLICY_KEYS
return {key: app_config.get(key) for key in REQUEST_POLICY_KEYS}
def coerce_bool(value: Any, default: bool = False) -> bool:
def coerce_bool(value: object, *, default: bool = False) -> bool:
"""Coerce arbitrary values into booleans with string-friendly semantics."""
if isinstance(value, bool):
return value
@@ -56,24 +96,26 @@ def coerce_bool(value: Any, default: bool = False) -> bool:
return bool(value)
def get_session_db_user_id(session_obj: Any) -> int | None:
def get_session_db_user_id(session_obj: object) -> int | None:
"""Extract and coerce `db_user_id` from a Flask session to ``int | None``."""
raw = session_obj.get("db_user_id") if session_obj is not None else None
raw = session_obj.get("db_user_id") if _is_mapping_with_get(session_obj) else None
try:
return int(raw) if raw is not None else None
except (TypeError, ValueError):
return int(raw) if raw is not None and _is_convertible_to_int(raw) else None
except TypeError, ValueError:
return None
def coerce_int(value: Any, default: int) -> int:
def coerce_int(value: object, default: int) -> int:
"""Best-effort integer coercion with fallback to default."""
if not _is_convertible_to_int(value):
return default
try:
return int(value)
except (TypeError, ValueError):
except TypeError, ValueError:
return default
def normalize_optional_text(value: Any) -> str | None:
def normalize_optional_text(value: object) -> str | None:
"""Return a trimmed string or None for empty/non-string input."""
if not isinstance(value, str):
return None
@@ -81,16 +123,18 @@ def normalize_optional_text(value: Any) -> str | None:
return normalized or None
def normalize_positive_int(value: Any) -> int | None:
def normalize_positive_int(value: object) -> int | None:
"""Parse *value* as a positive integer, returning ``None`` on failure."""
if not _is_convertible_to_int(value):
return None
try:
parsed = int(value)
except (TypeError, ValueError):
except TypeError, ValueError:
return None
return parsed if parsed > 0 else None
def normalize_optional_positive_int(value: Any, field_name: str = "value") -> int | None:
def normalize_optional_positive_int(value: object, field_name: str = "value") -> int | None:
"""Parse *value* as a positive integer or ``None``.
Raises ``ValueError`` when *value* is present but not a valid
@@ -98,27 +142,38 @@ def normalize_optional_positive_int(value: Any, field_name: str = "value") -> in
"""
if value is None:
return None
if not _is_convertible_to_int(value):
msg = f"{field_name} must be a positive integer when provided"
raise ValueError(msg)
try:
parsed = int(value)
except (TypeError, ValueError) as exc:
raise ValueError(f"{field_name} must be a positive integer when provided") from exc
msg = f"{field_name} must be a positive integer when provided"
raise ValueError(msg) from exc
if parsed < 1:
raise ValueError(f"{field_name} must be a positive integer when provided")
msg = f"{field_name} must be a positive integer when provided"
raise ValueError(msg)
return parsed
def populate_request_usernames(rows: list[dict[str, Any]], user_db: Any) -> None:
def populate_request_usernames(rows: list[dict[str, Any]], user_db: object) -> None:
"""Add 'username' to each request row by looking up user_id."""
if not _is_user_db_like(user_db):
return
cache: dict[int, str] = {}
for row in rows:
requester_id = row["user_id"]
requester_id = normalize_positive_int(row.get("user_id"))
if requester_id is None:
row["username"] = ""
continue
if requester_id not in cache:
requester = user_db.get_user(user_id=requester_id)
cache[requester_id] = requester.get("username", "") if requester else ""
row["username"] = cache[requester_id]
def extract_release_source_id(release_data: Any) -> str | None:
def extract_release_source_id(release_data: object) -> str | None:
"""Extract and normalize release_data.source_id."""
if not isinstance(release_data, dict):
return None
+26 -17
View File
@@ -6,11 +6,12 @@ routes/services and tested independently.
from __future__ import annotations
from enum import Enum
from typing import Any, Iterable, Mapping, Sequence
from collections.abc import Iterable, Mapping, Sequence
from enum import StrEnum
from typing import Any
class PolicyMode(str, Enum):
class PolicyMode(StrEnum):
"""Allowed request-policy modes.
Ordered from most to least permissive. The content-type default acts as a
@@ -33,7 +34,9 @@ _MODE_PERMISSIVENESS: dict[PolicyMode, int] = {
}
# Modes allowed in REQUEST_POLICY_RULES matrix rows.
MATRIX_ALLOWED_MODES = frozenset({PolicyMode.DOWNLOAD, PolicyMode.REQUEST_RELEASE, PolicyMode.BLOCKED})
MATRIX_ALLOWED_MODES = frozenset(
{PolicyMode.DOWNLOAD, PolicyMode.REQUEST_RELEASE, PolicyMode.BLOCKED}
)
def cap_mode(mode: PolicyMode, ceiling: PolicyMode) -> PolicyMode:
@@ -48,6 +51,7 @@ def _source_results_are_releases(source: Any) -> bool:
if normalized_source in {"", "*"}:
return False
from shelfmark.release_sources import source_results_are_releases
return source_results_are_releases(normalized_source)
@@ -105,7 +109,9 @@ def merge_request_policy_settings(
(source, content_type): (source, content_type, mode)
for source, content_type, mode in global_rules
}
for source, content_type, mode in _iter_rules(user_filtered.get("REQUEST_POLICY_RULES", [])):
for source, content_type, mode in _iter_rules(
user_filtered.get("REQUEST_POLICY_RULES", [])
):
merged_rules[(source, content_type)] = (source, content_type, mode)
merged["REQUEST_POLICY_RULES"] = [
{"source": source, "content_type": content_type, "mode": mode.value}
@@ -181,7 +187,7 @@ def get_source_content_type_capabilities() -> dict[str, set[str]]:
"""Return source -> supported content type map from registered sources."""
try:
from shelfmark.release_sources import list_available_sources
except Exception:
except ImportError:
return {}
capabilities: dict[str, set[str]] = {}
@@ -219,9 +225,15 @@ def validate_policy_rules(
- known source names
- source/content-type compatibility from source declarations
"""
capabilities = source_capabilities if source_capabilities is not None else get_source_content_type_capabilities()
capabilities = (
source_capabilities
if source_capabilities is not None
else get_source_content_type_capabilities()
)
normalized_capabilities = {
normalize_source(source): {normalize_content_type(content_type) for content_type in content_types}
normalize_source(source): {
normalize_content_type(content_type) for content_type in content_types
}
for source, content_types in capabilities.items()
}
@@ -248,26 +260,24 @@ def validate_policy_rules(
if source is None:
errors.append(f"{row_label}: source is required")
continue
if (
raw_content_type is None
or (isinstance(raw_content_type, str) and not raw_content_type.strip())
if raw_content_type is None or (
isinstance(raw_content_type, str) and not raw_content_type.strip()
):
errors.append(f"{row_label}: content_type is required")
continue
if content_type is None:
errors.append(f"{row_label}: invalid content_type '{rule.get('content_type')}'")
continue
if (
raw_mode is None
or (isinstance(raw_mode, str) and not raw_mode.strip())
):
if raw_mode is None or (isinstance(raw_mode, str) and not raw_mode.strip()):
errors.append(f"{row_label}: mode is required")
continue
if mode is None:
errors.append(f"{row_label}: invalid mode '{rule.get('mode')}'")
continue
if mode not in MATRIX_ALLOWED_MODES:
errors.append(f"{row_label}: mode '{mode.value}' is not allowed in matrix rules (use content-type defaults instead)")
errors.append(
f"{row_label}: mode '{mode.value}' is not allowed in matrix rules (use content-type defaults instead)"
)
continue
if source != "*" and source not in normalized_capabilities:
@@ -340,7 +350,6 @@ def resolve_policy_mode(
- sources whose browse results are already concrete releases normalize
request_book to request_release.
"""
effective = merge_request_policy_settings(global_settings, user_settings)
normalized_source = normalize_source(source)
normalized_content_type = normalize_content_type(content_type)
+456 -191
View File
@@ -2,29 +2,11 @@
from __future__ import annotations
from typing import Any, Callable
from typing import TYPE_CHECKING, Any
from flask import Flask, jsonify, request, session
from flask import Flask, Response, jsonify, request, session
from shelfmark.core.logger import setup_logger
from shelfmark.core.request_policy import (
PolicyMode,
REQUEST_POLICY_DEFAULT_FALLBACK_MODE,
get_source_content_type_capabilities,
merge_request_policy_settings,
normalize_content_type,
normalize_source,
parse_policy_mode,
resolve_policy_mode,
)
from shelfmark.core.request_validation import RequestStatus
from shelfmark.core.requests_service import (
RequestServiceError,
cancel_request,
create_request,
fulfil_request,
reject_request,
)
from shelfmark.core.notifications import (
NotificationContext,
NotificationEvent,
@@ -40,9 +22,35 @@ from shelfmark.core.request_helpers import (
normalize_positive_int,
populate_request_usernames,
)
from shelfmark.core.user_db import UserDB
from shelfmark.core.request_policy import (
REQUEST_POLICY_DEFAULT_FALLBACK_MODE,
PolicyMode,
get_source_content_type_capabilities,
merge_request_policy_settings,
normalize_content_type,
normalize_source,
parse_policy_mode,
resolve_policy_mode,
)
from shelfmark.core.request_validation import RequestStatus
from shelfmark.core.requests_service import (
RequestServiceError,
cancel_request,
create_request,
create_requests,
fulfil_request,
reject_request,
)
if TYPE_CHECKING:
from collections.abc import Callable
from flask.typing import ResponseReturnValue
from shelfmark.core.user_db import UserDB
logger = setup_logger(__name__)
_NOTIFICATION_TRIGGER_ERRORS = (RuntimeError, TypeError, ValueError)
def _error_response(
@@ -51,7 +59,7 @@ def _error_response(
*,
code: str | None = None,
required_mode: str | None = None,
):
) -> tuple[Response, int]:
payload: dict[str, Any] = {"error": message}
if code is not None:
payload["code"] = code
@@ -60,7 +68,9 @@ def _error_response(
return jsonify(payload), status_code
def _require_request_endpoints_available(resolve_auth_mode: Callable[[], str]):
def _require_request_endpoints_available(
resolve_auth_mode: Callable[[], str],
) -> tuple[Response, int] | None:
auth_mode = resolve_auth_mode()
if auth_mode == "none":
return _error_response(
@@ -73,7 +83,8 @@ def _require_request_endpoints_available(resolve_auth_mode: Callable[[], str]):
return None
def _require_db_user_id() -> tuple[int | None, Any | None]:
def _require_db_user_id() -> tuple[int | None, ResponseReturnValue | None]:
"""Return the logged-in DB user id or a ready-made error response."""
raw_user_id = session.get("db_user_id")
if raw_user_id is None:
return None, _error_response(
@@ -81,26 +92,26 @@ def _require_db_user_id() -> tuple[int | None, Any | None]:
403,
code="user_identity_unavailable",
)
try:
return int(raw_user_id), None
except (TypeError, ValueError):
normalized_user_id = normalize_positive_int(raw_user_id)
if normalized_user_id is None:
return None, _error_response(
"User identity is unavailable for request workflow",
403,
code="user_identity_unavailable",
)
return normalized_user_id, None
def _require_admin_user_id() -> tuple[int | None, Any | None]:
def _require_admin_user_id() -> tuple[int | None, ResponseReturnValue | None]:
if not session.get("is_admin", False):
return None, (jsonify({"error": "Admin access required"}), 403)
raw_admin_id = session.get("db_user_id")
if raw_admin_id is None:
return None, (jsonify({"error": "Admin user identity unavailable"}), 403)
try:
return int(raw_admin_id), None
except (TypeError, ValueError):
normalized_admin_user_id = normalize_positive_int(raw_admin_id)
if normalized_admin_user_id is None:
return None, (jsonify({"error": "Admin user identity unavailable"}), 403)
return normalized_admin_user_id, None
def _resolve_effective_policy(
@@ -111,11 +122,11 @@ def _resolve_effective_policy(
global_settings = load_users_request_policy_settings()
user_settings = user_db.get_user_settings(db_user_id) if db_user_id is not None else {}
effective = merge_request_policy_settings(global_settings, user_settings)
requests_enabled = coerce_bool(effective.get("REQUESTS_ENABLED"), False)
requests_enabled = coerce_bool(effective.get("REQUESTS_ENABLED"), default=False)
return global_settings, user_settings, effective, requests_enabled
def _resolve_title_from_book_data(book_data: Any) -> str:
def _resolve_title_from_book_data(book_data: object) -> str:
if isinstance(book_data, dict):
title = normalize_optional_text(book_data.get("title"))
if title is not None:
@@ -123,7 +134,7 @@ def _resolve_title_from_book_data(book_data: Any) -> str:
return "Unknown title"
def _normalize_optional_source_id(value: Any) -> str | None:
def _normalize_optional_source_id(value: object) -> str | None:
"""Normalize source identifiers while allowing integer provider ids."""
if isinstance(value, bool) or value is None:
return None
@@ -139,9 +150,9 @@ def _build_release_result_data_from_book_data(
content_type: str,
) -> dict[str, Any]:
"""Build release-level payload fields for sources whose browse results are releases."""
source_id = _normalize_optional_source_id(book_data.get("provider_id")) or _normalize_optional_source_id(
book_data.get("id")
)
source_id = _normalize_optional_source_id(
book_data.get("provider_id")
) or _normalize_optional_source_id(book_data.get("id"))
payload: dict[str, Any] = {
"source": source,
"source_id": source_id,
@@ -163,15 +174,16 @@ def _source_results_are_releases(source: str) -> bool:
if normalized_source in {"", "*"}:
return False
from shelfmark.release_sources import source_results_are_releases
return source_results_are_releases(normalized_source)
def _normalize_release_result_request_payload(
*,
source: str,
request_level: Any,
book_data: Any,
release_data: Any,
request_level: object,
book_data: object,
release_data: object,
content_type: str,
) -> tuple[Any, Any]:
"""Concrete-release browse results are always handled as release-level requests."""
@@ -193,13 +205,15 @@ def _normalize_release_result_request_payload(
if normalized_release_data.get("content_type") is None:
normalized_release_data["content_type"] = content_type
normalized_source_id = _normalize_optional_source_id(normalized_release_data.get("source_id"))
normalized_source_id = _normalize_optional_source_id(
normalized_release_data.get("source_id")
)
if normalized_source_id is not None:
normalized_release_data["source_id"] = normalized_source_id
elif isinstance(book_data, dict):
fallback_source_id = _normalize_optional_source_id(book_data.get("provider_id")) or _normalize_optional_source_id(
book_data.get("id")
)
fallback_source_id = _normalize_optional_source_id(
book_data.get("provider_id")
) or _normalize_optional_source_id(book_data.get("id"))
if fallback_source_id is not None:
normalized_release_data["source_id"] = fallback_source_id
@@ -231,7 +245,163 @@ def _format_requester_label(user_db: UserDB, request_row: dict[str, Any]) -> str
return _format_user_label(None, user_id)
def _resolve_request_source_and_format(request_row: dict[str, Any]) -> tuple[str, str | None]:
def _resolve_request_user_context(
user_db: UserDB,
*,
actor_user_id: int,
actor_username: str | None,
on_behalf_of_user_id: object,
) -> tuple[int, str | None, str]:
if on_behalf_of_user_id in (None, ""):
actor_label = _format_user_label(actor_username, actor_user_id)
return actor_user_id, actor_username, actor_label
if not session.get("is_admin", False):
msg = "Admin required"
raise RequestServiceError(msg, status_code=403)
target_user_id = normalize_positive_int(on_behalf_of_user_id)
if target_user_id is None:
msg = "Invalid on_behalf_of_user_id"
raise RequestServiceError(msg, status_code=400)
target_user = user_db.get_user(user_id=target_user_id)
if not target_user:
msg = "User not found"
raise RequestServiceError(msg, status_code=404)
target_username = normalize_optional_text(target_user.get("username"))
actor_label = _format_user_label(actor_username, actor_user_id)
target_label = _format_user_label(target_username, target_user_id)
return target_user_id, target_username, f"{actor_label} on behalf of {target_label}"
def _prepare_request_create_arguments(
user_db: UserDB,
data: dict[str, Any],
) -> dict[str, Any]:
db_user_id, db_gate = _require_db_user_id()
if db_gate is not None or db_user_id is None:
msg = "User identity is unavailable for request workflow"
raise RequestServiceError(
msg,
status_code=403,
code="user_identity_unavailable",
)
actor_username = normalize_optional_text(session.get("user_id"))
target_user_id, _, actor_label = _resolve_request_user_context(
user_db,
actor_user_id=db_user_id,
actor_username=actor_username,
on_behalf_of_user_id=data.get("on_behalf_of_user_id"),
)
context = data.get("context") or {}
if not isinstance(context, dict):
msg = "context must be an object"
raise RequestServiceError(msg, status_code=400)
source = normalize_source(context.get("source"))
release_data = data.get("release_data")
request_level = context.get("request_level")
if request_level is None:
request_level = "book" if release_data is None else "release"
book_data = data.get("book_data")
if not isinstance(book_data, dict):
msg = "book_data must be an object"
raise RequestServiceError(msg, status_code=400)
request_title = _resolve_title_from_book_data(book_data)
content_type = normalize_content_type(
context.get("content_type") or data.get("content_type") or book_data.get("content_type")
)
request_level, release_data = _normalize_release_result_request_payload(
source=source,
request_level=request_level,
book_data=book_data,
release_data=release_data,
content_type=content_type,
)
global_settings, user_settings, effective, requests_enabled = _resolve_effective_policy(
user_db,
db_user_id=target_user_id,
)
if not requests_enabled:
msg = "Request workflow is disabled by policy"
raise RequestServiceError(
msg,
status_code=403,
code="requests_unavailable",
)
max_pending = coerce_int(
effective.get("MAX_PENDING_REQUESTS_PER_USER"),
default=20,
)
max_pending = max(max_pending, 1)
max_pending = min(max_pending, 1000)
allow_notes = coerce_bool(effective.get("REQUESTS_ALLOW_NOTES"), default=True)
note_value = data.get("note") if allow_notes else None
resolved_mode = resolve_policy_mode(
source=source,
content_type=content_type,
global_settings=global_settings,
user_settings=user_settings,
)
logger.debug(
"request create policy actor=%s target_user_id=%s source=%s content_type=%s request_level=%s resolved_mode=%s",
actor_label,
target_user_id,
source,
content_type,
request_level,
resolved_mode.value,
)
if resolved_mode == PolicyMode.BLOCKED:
msg = "Requesting is blocked by policy"
raise RequestServiceError(
msg,
status_code=403,
code="policy_blocked",
required_mode=PolicyMode.BLOCKED.value,
)
requested_level = str(request_level).strip().lower() if isinstance(request_level, str) else ""
if resolved_mode == PolicyMode.REQUEST_BOOK and requested_level != "book":
msg = "Policy requires book-level requests"
raise RequestServiceError(
msg,
status_code=403,
code="policy_requires_request",
required_mode=PolicyMode.REQUEST_BOOK.value,
)
return {
"create_args": {
"user_id": target_user_id,
"source_hint": source,
"content_type": content_type,
"request_level": request_level,
"policy_mode": resolved_mode.value,
"book_data": book_data,
"release_data": release_data,
"note": note_value,
"max_pending_per_user": max_pending,
},
"actor_label": actor_label,
"request_title": request_title,
"resolved_mode": resolved_mode,
}
def _resolve_request_source_and_format(
request_row: dict[str, Any],
) -> tuple[str, str | None]:
release_data = request_row.get("release_data")
if isinstance(release_data, dict):
source = normalize_source(release_data.get("source") or request_row.get("source_hint"))
@@ -244,6 +414,69 @@ def _resolve_request_source_and_format(request_row: dict[str, Any]) -> tuple[str
return normalize_source(request_row.get("source_hint")), None
def _build_queued_download_result(
*,
create_args: dict[str, Any],
request_title: str,
) -> dict[str, Any]:
release_data = create_args.get("release_data")
source = create_args.get("source_hint")
source_id: str | None = None
if isinstance(release_data, dict):
source = release_data.get("source") or source
source_id = _normalize_optional_source_id(release_data.get("source_id"))
return {
"kind": "download",
"status": "queued",
"priority": 0,
"title": request_title,
"source": normalize_source(source),
"source_id": source_id,
"content_type": create_args.get("content_type"),
}
def _queue_prepared_download_submission(
user_db: UserDB,
*,
queue_release: Callable[..., tuple[bool, str | None]],
create_args: dict[str, Any],
request_title: str,
) -> dict[str, Any]:
release_data = create_args.get("release_data")
if not isinstance(release_data, dict):
msg = "Download policy requires a concrete release"
raise RequestServiceError(
msg,
status_code=400,
code="policy_requires_download",
required_mode=PolicyMode.DOWNLOAD.value,
)
requester = user_db.get_user(user_id=create_args["user_id"])
if requester is None:
msg = "Requesting user not found"
raise RequestServiceError(msg, status_code=404)
success, error = queue_release(
dict(release_data),
0,
user_id=create_args["user_id"],
username=requester.get("username"),
)
if not success:
raise RequestServiceError(
error or "Failed to queue release",
status_code=409,
code="queue_failed",
)
return _build_queued_download_result(
create_args=create_args,
request_title=request_title,
)
def _notify_admin_for_request_event(
@@ -274,7 +507,7 @@ def _notify_admin_for_request_event(
owner_user_id = normalize_positive_int(request_row.get("user_id"))
try:
notify_admin(event, context)
except Exception as exc:
except _NOTIFICATION_TRIGGER_ERRORS as exc:
logger.warning(
"Failed to trigger admin notification for request event '%s': %s",
event.value,
@@ -284,7 +517,7 @@ def _notify_admin_for_request_event(
return
try:
notify_user(owner_user_id, event, context)
except Exception as exc:
except _NOTIFICATION_TRIGGER_ERRORS as exc:
logger.warning(
"Failed to trigger user notification for request event '%s' (user_id=%s): %s",
event.value,
@@ -299,12 +532,12 @@ def register_request_routes(
*,
resolve_auth_mode: Callable[[], str],
queue_release: Callable[..., tuple[bool, str | None]],
ws_manager: Any | None = None,
ws_manager: object | None = None,
) -> None:
"""Register request policy and request lifecycle routes."""
@app.route("/api/request-policy", methods=["GET"])
def api_request_policy():
def api_request_policy() -> ResponseReturnValue:
auth_gate = _require_request_endpoints_available(resolve_auth_mode)
if auth_gate is not None:
return auth_gate
@@ -316,12 +549,7 @@ def register_request_routes(
if db_gate is not None:
return db_gate
else:
raw_id = session.get("db_user_id")
if raw_id is not None:
try:
db_user_id = int(raw_id)
except (TypeError, ValueError):
db_user_id = None
db_user_id = normalize_positive_int(session.get("db_user_id"))
global_settings, user_settings, effective, requests_enabled = _resolve_effective_policy(
user_db,
@@ -333,6 +561,7 @@ def register_request_routes(
source_capabilities = get_source_content_type_capabilities()
from shelfmark.release_sources import source_results_are_releases
source_modes = []
for source_name in sorted(source_capabilities):
supported_types = sorted(
@@ -380,135 +609,38 @@ def register_request_routes(
)
@app.route("/api/requests", methods=["POST"])
def api_create_request():
def api_create_request() -> ResponseReturnValue:
auth_gate = _require_request_endpoints_available(resolve_auth_mode)
if auth_gate is not None:
return auth_gate
db_user_id, db_gate = _require_db_user_id()
if db_gate is not None or db_user_id is None:
return db_gate
actor_username = normalize_optional_text(session.get("user_id"))
actor_label = _format_user_label(actor_username, db_user_id)
data = request.get_json(silent=True)
if not isinstance(data, dict):
return jsonify({"error": "No data provided"}), 400
context = data.get("context") or {}
if not isinstance(context, dict):
return jsonify({"error": "context must be an object"}), 400
source = normalize_source(context.get("source"))
release_data = data.get("release_data")
request_level = context.get("request_level")
if request_level is None:
request_level = "book" if release_data is None else "release"
book_data = data.get("book_data")
if not isinstance(book_data, dict):
return jsonify({"error": "book_data must be an object"}), 400
request_title = _resolve_title_from_book_data(book_data)
content_type = normalize_content_type(
context.get("content_type")
or data.get("content_type")
or book_data.get("content_type")
)
request_level, release_data = _normalize_release_result_request_payload(
source=source,
request_level=request_level,
book_data=book_data,
release_data=release_data,
content_type=content_type,
)
global_settings, user_settings, effective, requests_enabled = _resolve_effective_policy(
user_db,
db_user_id=db_user_id,
)
if not requests_enabled:
logger.debug(
"Request not created for '%s' by %s: requests are disabled",
request_title,
actor_label,
)
return _error_response(
"Request workflow is disabled by policy",
403,
code="requests_unavailable",
)
max_pending = coerce_int(
effective.get("MAX_PENDING_REQUESTS_PER_USER"),
default=20,
)
if max_pending < 1:
max_pending = 1
if max_pending > 1000:
max_pending = 1000
allow_notes = coerce_bool(effective.get("REQUESTS_ALLOW_NOTES"), default=True)
note_value = data.get("note") if allow_notes else None
resolved_mode = resolve_policy_mode(
source=source,
content_type=content_type,
global_settings=global_settings,
user_settings=user_settings,
)
logger.debug(
"request create policy user=%s db_user_id=%s source=%s content_type=%s request_level=%s resolved_mode=%s",
session.get("user_id"),
db_user_id,
source,
content_type,
request_level,
resolved_mode.value,
)
if resolved_mode == PolicyMode.BLOCKED:
logger.debug(
"Request blocked by policy for '%s' by %s",
request_title,
actor_label,
)
return _error_response(
"Requesting is blocked by policy",
403,
code="policy_blocked",
required_mode=PolicyMode.BLOCKED.value,
)
if resolved_mode == PolicyMode.REQUEST_BOOK:
requested_level = str(request_level).strip().lower() if isinstance(request_level, str) else ""
if requested_level != "book":
logger.debug(
"Request not created for '%s' by %s: policy requires book-level requests",
request_title,
actor_label,
)
return _error_response(
"Policy requires book-level requests",
403,
code="policy_requires_request",
required_mode=PolicyMode.REQUEST_BOOK.value,
)
try:
created = create_request(
user_db,
user_id=db_user_id,
source_hint=source,
content_type=content_type,
request_level=request_level,
policy_mode=resolved_mode.value,
book_data=book_data,
release_data=release_data,
note=note_value,
max_pending_per_user=max_pending,
)
prepared = _prepare_request_create_arguments(user_db, data)
if prepared["resolved_mode"] == PolicyMode.DOWNLOAD:
queued = _queue_prepared_download_submission(
user_db,
queue_release=queue_release,
create_args=prepared["create_args"],
request_title=prepared["request_title"],
)
logger.info(
"Policy download queued for '%s' by %s",
prepared["request_title"],
prepared["actor_label"],
)
return jsonify(queued), 200
created = create_request(user_db, **prepared["create_args"])
except RequestServiceError as exc:
return _error_response(str(exc), exc.status_code, code=exc.code)
return _error_response(
str(exc),
exc.status_code,
code=exc.code,
required_mode=exc.required_mode,
)
event_payload = {
"request_id": created["id"],
@@ -519,7 +651,7 @@ def register_request_routes(
"Request created #%s for '%s' by %s",
created["id"],
event_payload["title"],
actor_label,
prepared["actor_label"],
)
emit_ws_event(
ws_manager,
@@ -531,7 +663,7 @@ def register_request_routes(
ws_manager,
event_name="request_update",
payload=event_payload,
room=f"user_{db_user_id}",
room=f"user_{created['user_id']}",
)
_notify_admin_for_request_event(
@@ -542,15 +674,135 @@ def register_request_routes(
return jsonify(created), 201
@app.route("/api/requests/batch", methods=["POST"])
def api_create_requests_batch() -> ResponseReturnValue:
auth_gate = _require_request_endpoints_available(resolve_auth_mode)
if auth_gate is not None:
return auth_gate
data = request.get_json(silent=True)
if not isinstance(data, dict):
return jsonify({"error": "No data provided"}), 400
raw_requests = data.get("requests")
if not isinstance(raw_requests, list) or len(raw_requests) == 0:
return jsonify({"error": "requests must contain at least one request"}), 400
try:
prepared_requests = [
_prepare_request_create_arguments(user_db, raw_request)
for raw_request in raw_requests
]
except RequestServiceError as exc:
return _error_response(
str(exc),
exc.status_code,
code=exc.code,
required_mode=exc.required_mode,
)
request_prepared_items: list[tuple[int, dict[str, Any]]] = []
download_prepared_items: list[tuple[int, dict[str, Any]]] = []
for index, prepared in enumerate(prepared_requests):
if prepared["resolved_mode"] == PolicyMode.DOWNLOAD:
download_prepared_items.append((index, prepared))
continue
request_prepared_items.append((index, prepared))
created_rows: list[dict[str, Any]] = []
if request_prepared_items:
try:
created_rows = create_requests(
user_db,
requests=[prepared["create_args"] for _, prepared in request_prepared_items],
)
except RequestServiceError as exc:
return _error_response(
str(exc),
exc.status_code,
code=exc.code,
required_mode=exc.required_mode,
)
results_by_index: dict[int, dict[str, Any]] = {}
for (index, prepared), created in zip(
request_prepared_items,
created_rows,
strict=True,
):
event_payload = {
"request_id": created["id"],
"status": created["status"],
"title": _resolve_request_title(created),
}
logger.info(
"Request created #%s for '%s' by %s",
created["id"],
event_payload["title"],
prepared["actor_label"],
)
emit_ws_event(
ws_manager,
event_name="new_request",
payload=event_payload,
room="admins",
)
emit_ws_event(
ws_manager,
event_name="request_update",
payload=event_payload,
room=f"user_{created['user_id']}",
)
_notify_admin_for_request_event(
user_db,
event=NotificationEvent.REQUEST_CREATED,
request_row=created,
)
results_by_index[index] = created
for index, prepared in download_prepared_items:
try:
results_by_index[index] = _queue_prepared_download_submission(
user_db,
queue_release=queue_release,
create_args=prepared["create_args"],
request_title=prepared["request_title"],
)
except RequestServiceError as exc:
return _error_response(
str(exc),
exc.status_code,
code=exc.code,
required_mode=exc.required_mode,
)
logger.info(
"Policy download queued for '%s' by %s",
prepared["request_title"],
prepared["actor_label"],
)
ordered_results = [results_by_index[index] for index in range(len(prepared_requests))]
status_code = 201 if request_prepared_items else 200
return jsonify(ordered_results), status_code
@app.route("/api/requests", methods=["GET"])
def api_list_requests():
def api_list_requests() -> ResponseReturnValue:
auth_gate = _require_request_endpoints_available(resolve_auth_mode)
if auth_gate is not None:
return auth_gate
db_user_id, db_gate = _require_db_user_id()
if db_gate is not None or db_user_id is None:
if db_gate is not None:
return db_gate
if db_user_id is None:
return _error_response(
"User identity is unavailable for request workflow",
403,
code="user_identity_unavailable",
)
status = request.args.get("status")
limit = request.args.get("limit", type=int)
@@ -568,14 +820,20 @@ def register_request_routes(
return jsonify(rows)
@app.route("/api/requests/<int:request_id>", methods=["DELETE"])
def api_cancel_request(request_id: int):
def api_cancel_request(request_id: int) -> ResponseReturnValue:
auth_gate = _require_request_endpoints_available(resolve_auth_mode)
if auth_gate is not None:
return auth_gate
db_user_id, db_gate = _require_db_user_id()
if db_gate is not None or db_user_id is None:
if db_gate is not None:
return db_gate
if db_user_id is None:
return _error_response(
"User identity is unavailable for request workflow",
403,
code="user_identity_unavailable",
)
try:
updated = cancel_request(
@@ -591,7 +849,9 @@ def register_request_routes(
"status": updated["status"],
"title": _resolve_request_title(updated),
}
actor_label = _format_user_label(normalize_optional_text(session.get("user_id")), db_user_id)
actor_label = _format_user_label(
normalize_optional_text(session.get("user_id")), db_user_id
)
logger.info(
"Request cancelled #%s for '%s' by %s",
updated["id"],
@@ -614,7 +874,7 @@ def register_request_routes(
return jsonify(updated)
@app.route("/api/admin/requests", methods=["GET"])
def api_admin_list_requests():
def api_admin_list_requests() -> ResponseReturnValue:
auth_gate = _require_request_endpoints_available(resolve_auth_mode)
if auth_gate is not None:
return auth_gate
@@ -635,17 +895,14 @@ def register_request_routes(
return jsonify(rows)
@app.route("/api/admin/requests/count", methods=["GET"])
def api_admin_request_counts():
def api_admin_request_counts() -> ResponseReturnValue:
auth_gate = _require_request_endpoints_available(resolve_auth_mode)
if auth_gate is not None:
return auth_gate
if not session.get("is_admin", False):
return jsonify({"error": "Admin access required"}), 403
by_status = {
status: len(user_db.list_requests(status=status))
for status in RequestStatus
}
by_status = {status: len(user_db.list_requests(status=status)) for status in RequestStatus}
return jsonify(
{
"pending": by_status[RequestStatus.PENDING],
@@ -655,7 +912,7 @@ def register_request_routes(
)
@app.route("/api/admin/requests/<int:request_id>/fulfil", methods=["POST"])
def api_admin_fulfil_request(request_id: int):
def api_admin_fulfil_request(request_id: int) -> ResponseReturnValue:
auth_gate = _require_request_endpoints_available(resolve_auth_mode)
if auth_gate is not None:
return auth_gate
@@ -663,6 +920,8 @@ def register_request_routes(
admin_user_id, admin_gate = _require_admin_user_id()
if admin_gate is not None:
return admin_gate
if admin_user_id is None:
return jsonify({"error": "Admin user identity unavailable"}), 403
data = request.get_json(silent=True) or {}
if not isinstance(data, dict):
@@ -686,7 +945,9 @@ def register_request_routes(
"status": updated["status"],
"title": _resolve_request_title(updated),
}
admin_label = _format_user_label(normalize_optional_text(session.get("user_id")), admin_user_id)
admin_label = _format_user_label(
normalize_optional_text(session.get("user_id")), admin_user_id
)
requester_label = _format_requester_label(user_db, updated)
logger.info(
"Request fulfilled #%s for '%s' by %s (requested by %s)",
@@ -717,7 +978,7 @@ def register_request_routes(
return jsonify(updated)
@app.route("/api/admin/requests/<int:request_id>/reject", methods=["POST"])
def api_admin_reject_request(request_id: int):
def api_admin_reject_request(request_id: int) -> ResponseReturnValue:
auth_gate = _require_request_endpoints_available(resolve_auth_mode)
if auth_gate is not None:
return auth_gate
@@ -725,6 +986,8 @@ def register_request_routes(
admin_user_id, admin_gate = _require_admin_user_id()
if admin_gate is not None:
return admin_gate
if admin_user_id is None:
return jsonify({"error": "Admin user identity unavailable"}), 403
data = request.get_json(silent=True) or {}
if not isinstance(data, dict):
@@ -745,7 +1008,9 @@ def register_request_routes(
"status": updated["status"],
"title": _resolve_request_title(updated),
}
admin_label = _format_user_label(normalize_optional_text(session.get("user_id")), admin_user_id)
admin_label = _format_user_label(
normalize_optional_text(session.get("user_id")), admin_user_id
)
requester_label = _format_requester_label(user_db, updated)
logger.info(
"Request rejected #%s for '%s' by %s (requested by %s)",
+36 -22
View File
@@ -2,15 +2,15 @@
from __future__ import annotations
from enum import Enum
from typing import Any
from enum import StrEnum
from shelfmark.core.models import QueueStatus
from shelfmark.core.request_policy import parse_policy_mode
class RequestStatus(str, Enum):
class RequestStatus(StrEnum):
"""Enum for request lifecycle statuses."""
PENDING = "pending"
FULFILLED = "fulfilled"
REJECTED = "rejected"
@@ -20,65 +20,79 @@ class RequestStatus(str, Enum):
DELIVERY_STATE_NONE = "none"
VALID_REQUEST_STATUSES = frozenset(RequestStatus)
TERMINAL_REQUEST_STATUSES = frozenset({
RequestStatus.FULFILLED, RequestStatus.REJECTED, RequestStatus.CANCELLED,
})
TERMINAL_REQUEST_STATUSES = frozenset(
{
RequestStatus.FULFILLED,
RequestStatus.REJECTED,
RequestStatus.CANCELLED,
}
)
VALID_REQUEST_LEVELS = frozenset({"book", "release"})
VALID_DELIVERY_STATES = frozenset({DELIVERY_STATE_NONE} | set(QueueStatus))
def normalize_request_status(status: Any) -> str:
def normalize_request_status(status: object) -> str:
"""Validate and normalize request status values."""
if not isinstance(status, str):
raise ValueError(f"Invalid request status: {status}")
msg = f"Invalid request status: {status}"
raise TypeError(msg)
normalized = status.strip().lower()
if normalized not in VALID_REQUEST_STATUSES:
raise ValueError(f"Invalid request status: {status}")
msg = f"Invalid request status: {status}"
raise ValueError(msg)
return normalized
def normalize_policy_mode(mode: Any) -> str:
def normalize_policy_mode(mode: object) -> str:
"""Validate and normalize policy mode values."""
parsed = parse_policy_mode(mode)
if parsed is None:
raise ValueError(f"Invalid policy_mode: {mode}")
msg = f"Invalid policy_mode: {mode}"
raise ValueError(msg)
return parsed.value
def normalize_request_level(request_level: Any) -> str:
def normalize_request_level(request_level: object) -> str:
"""Validate and normalize request level values."""
if not isinstance(request_level, str):
raise ValueError(f"Invalid request_level: {request_level}")
msg = f"Invalid request_level: {request_level}"
raise TypeError(msg)
normalized = request_level.strip().lower()
if normalized not in VALID_REQUEST_LEVELS:
raise ValueError(f"Invalid request_level: {request_level}")
msg = f"Invalid request_level: {request_level}"
raise ValueError(msg)
return normalized
def normalize_delivery_state(state: Any) -> str:
def normalize_delivery_state(state: object) -> str:
"""Validate and normalize delivery-state values."""
if not isinstance(state, str):
raise ValueError(f"Invalid delivery_state: {state}")
msg = f"Invalid delivery_state: {state}"
raise TypeError(msg)
normalized = state.strip().lower()
if normalized not in VALID_DELIVERY_STATES:
raise ValueError(f"Invalid delivery_state: {state}")
msg = f"Invalid delivery_state: {state}"
raise ValueError(msg)
return normalized
def validate_request_level_payload(request_level: Any, release_data: Any) -> str:
def validate_request_level_payload(request_level: object, release_data: object) -> str:
"""Validate request_level and release_data shape coupling."""
normalized_level = normalize_request_level(request_level)
if normalized_level == "release" and release_data is None:
raise ValueError("request_level=release requires non-null release_data")
msg = "request_level=release requires non-null release_data"
raise ValueError(msg)
if normalized_level == "book" and release_data is not None:
raise ValueError("request_level=book requires null release_data")
msg = "request_level=book requires null release_data"
raise ValueError(msg)
return normalized_level
def validate_status_transition(current_status: Any, new_status: Any) -> tuple[str, str]:
def validate_status_transition(current_status: object, new_status: object) -> tuple[str, str]:
"""Validate request status transitions and terminal immutability."""
current = normalize_request_status(current_status)
new = normalize_request_status(new_status)
if current in TERMINAL_REQUEST_STATUSES and new != current:
raise ValueError("Terminal request statuses are immutable")
msg = "Terminal request statuses are immutable"
raise ValueError(msg)
return current, new
+240 -80
View File
@@ -2,29 +2,30 @@
from __future__ import annotations
from datetime import datetime, timezone
import json
from typing import Any, Callable, TYPE_CHECKING
from datetime import UTC, datetime
from typing import TYPE_CHECKING, Any
from shelfmark.core.request_policy import normalize_content_type
from shelfmark.core.models import QueueStatus
from shelfmark.core.request_helpers import (
extract_release_source_id,
normalize_positive_int,
)
from shelfmark.core.request_policy import normalize_content_type
from shelfmark.core.request_validation import (
DELIVERY_STATE_NONE,
RequestStatus,
normalize_policy_mode,
normalize_request_level,
normalize_request_status,
validate_request_level_payload,
validate_status_transition,
)
from shelfmark.core.request_helpers import extract_release_source_id, normalize_positive_int
MAX_REQUEST_NOTE_LENGTH = 1000
MAX_REQUEST_JSON_BLOB_BYTES = 10 * 1024
if TYPE_CHECKING:
from collections.abc import Callable
from shelfmark.core.user_db import UserDB
@@ -37,67 +38,78 @@ class RequestServiceError(ValueError):
*,
status_code: int = 400,
code: str | None = None,
):
required_mode: str | None = None,
) -> None:
"""Initialize the error with HTTP metadata for API callers."""
super().__init__(message)
self.status_code = status_code
self.code = code
self.required_mode = required_mode
def _normalize_match_text(value: Any) -> str:
def _normalize_match_text(value: object) -> str:
if not isinstance(value, str):
return ""
return value.strip().lower()
def normalize_note(note: Any) -> str | None:
def normalize_note(note: object) -> str | None:
"""Validate request notes and normalize empty strings to None."""
if note is None:
return None
if not isinstance(note, str):
raise RequestServiceError("note must be a string", status_code=400)
msg = "note must be a string"
raise RequestServiceError(msg, status_code=400)
normalized = note.strip()
if len(normalized) > MAX_REQUEST_NOTE_LENGTH:
msg_0 = f"note must be <= {MAX_REQUEST_NOTE_LENGTH} characters"
raise RequestServiceError(
f"note must be <= {MAX_REQUEST_NOTE_LENGTH} characters",
msg_0,
status_code=400,
)
return normalized or None
def _validate_book_data(book_data: Any) -> dict[str, Any]:
def _validate_book_data(book_data: object) -> dict[str, Any]:
if not isinstance(book_data, dict):
raise RequestServiceError("book_data must be an object", status_code=400)
msg = "book_data must be an object"
raise RequestServiceError(msg, status_code=400)
required_fields = ("title", "author", "provider", "provider_id")
missing = [field for field in required_fields if not _normalize_match_text(book_data.get(field))]
missing = [
field for field in required_fields if not _normalize_match_text(book_data.get(field))
]
if missing:
msg_0 = f"book_data missing required field(s): {', '.join(missing)}"
raise RequestServiceError(
f"book_data missing required field(s): {', '.join(missing)}",
msg_0,
status_code=400,
)
return dict(book_data)
def _validate_json_blob_size(field: str, payload: Any) -> None:
def _validate_json_blob_size(field: str, payload: object) -> None:
if payload is None:
return
try:
serialized = json.dumps(payload, separators=(",", ":"), ensure_ascii=False)
except (TypeError, ValueError) as exc:
raise RequestServiceError(f"{field} must be JSON-serializable", status_code=400) from exc
msg = f"{field} must be JSON-serializable"
raise RequestServiceError(msg, status_code=400) from exc
payload_size = len(serialized.encode("utf-8"))
if payload_size > MAX_REQUEST_JSON_BLOB_BYTES:
msg = f"{field} must be <= {MAX_REQUEST_JSON_BLOB_BYTES} bytes"
raise RequestServiceError(
f"{field} must be <= {MAX_REQUEST_JSON_BLOB_BYTES} bytes",
msg,
status_code=400,
code="request_payload_too_large",
)
def _find_duplicate_pending_request(
user_db: "UserDB",
user_db: UserDB,
*,
user_id: int,
title: str,
@@ -121,19 +133,59 @@ def _find_duplicate_pending_request(
def _now_timestamp() -> str:
return datetime.now(timezone.utc).isoformat(timespec="seconds")
return datetime.now(UTC).isoformat(timespec="seconds")
def _normalize_admin_note(admin_note: Any) -> str | None:
def _normalize_admin_note(admin_note: object) -> str | None:
if admin_note is None:
return None
if not isinstance(admin_note, str):
raise RequestServiceError("admin_note must be a string", status_code=400)
msg = "admin_note must be a string"
raise RequestServiceError(msg, status_code=400)
return admin_note.strip() or None
def _prepare_request_create(
*,
user_id: int,
source_hint: str | None,
content_type: object,
request_level: object,
policy_mode: object,
book_data: object,
release_data: object = None,
note: object = None,
) -> dict[str, Any]:
validated_book_data = _validate_book_data(book_data)
normalized_note = normalize_note(note)
normalized_content_type = normalize_content_type(
content_type or validated_book_data.get("content_type")
)
validated_book_data["content_type"] = normalized_content_type
try:
normalized_request_level = validate_request_level_payload(request_level, release_data)
normalized_policy_mode = normalize_policy_mode(policy_mode)
except (ValueError, TypeError) as exc:
raise RequestServiceError(str(exc), status_code=400) from exc
_validate_json_blob_size("book_data", validated_book_data)
_validate_json_blob_size("release_data", release_data)
return {
"user_id": user_id,
"source_hint": source_hint,
"content_type": normalized_content_type,
"request_level": normalized_request_level,
"policy_mode": normalized_policy_mode,
"book_data": validated_book_data,
"release_data": release_data,
"note": normalized_note,
}
def sync_delivery_states_from_queue_status(
user_db: "UserDB",
user_db: UserDB,
*,
queue_status: dict[str, dict[str, Any]],
user_id: int | None = None,
@@ -157,6 +209,7 @@ def sync_delivery_states_from_queue_status(
unique_request_ids_by_source.pop(source_id, None)
request_delivery_states: dict[int, str] = {}
request_delivery_payloads: dict[int, dict[str, Any]] = {}
for status_key in QueueStatus:
status_bucket = queue_status.get(status_key)
if not isinstance(status_bucket, dict):
@@ -170,22 +223,42 @@ def sync_delivery_states_from_queue_status(
if request_id is None:
continue
request_delivery_states[request_id] = status_key
if isinstance(task_payload, dict):
request_delivery_payloads[request_id] = dict(task_payload)
if not request_delivery_states:
return []
updated: list[dict[str, Any]] = []
for row in fulfilled_rows:
delivery_state = request_delivery_states.get(int(row["id"]))
request_id = int(row["id"])
delivery_state = request_delivery_states.get(request_id)
if delivery_state is None:
continue
task_payload = request_delivery_payloads.get(request_id) or {}
retry_available = task_payload.get("retry_available")
if delivery_state == QueueStatus.ERROR and retry_available is False:
raw_status_message = task_payload.get("status_message")
failure_reason = (
raw_status_message.strip()
if isinstance(raw_status_message, str) and raw_status_message.strip()
else "Download failed"
)
reopened = user_db.reopen_failed_request(
request_id,
failure_reason=failure_reason,
)
if reopened is not None:
updated.append(reopened)
continue
if row.get("delivery_state", DELIVERY_STATE_NONE) == delivery_state:
continue
updated.append(
user_db.update_request(
row["id"],
request_id,
delivery_state=delivery_state,
delivery_updated_at=_now_timestamp(),
)
@@ -195,40 +268,36 @@ def sync_delivery_states_from_queue_status(
def create_request(
user_db: "UserDB",
user_db: UserDB,
*,
user_id: int,
source_hint: str | None,
content_type: Any,
request_level: Any,
policy_mode: Any,
book_data: Any,
release_data: Any = None,
note: Any = None,
content_type: object,
request_level: object,
policy_mode: object,
book_data: object,
release_data: object = None,
note: object = None,
max_pending_per_user: int | None = None,
) -> dict[str, Any]:
"""Create a pending request after service-level validation."""
validated_book_data = _validate_book_data(book_data)
normalized_note = normalize_note(note)
normalized_content_type = normalize_content_type(
content_type or validated_book_data.get("content_type")
prepared_request = _prepare_request_create(
user_id=user_id,
source_hint=source_hint,
content_type=content_type,
request_level=request_level,
policy_mode=policy_mode,
book_data=book_data,
release_data=release_data,
note=note,
)
validated_book_data["content_type"] = normalized_content_type
try:
normalized_request_level = validate_request_level_payload(request_level, release_data)
normalized_policy_mode = normalize_policy_mode(policy_mode)
except ValueError as exc:
raise RequestServiceError(str(exc), status_code=400) from exc
_validate_json_blob_size("book_data", validated_book_data)
_validate_json_blob_size("release_data", release_data)
if max_pending_per_user is not None:
pending_count = user_db.count_user_pending_requests(user_id)
if pending_count >= max_pending_per_user:
msg = "Maximum pending requests reached for this user"
raise RequestServiceError(
"Maximum pending requests reached for this user",
msg,
status_code=409,
code="max_pending_reached",
)
@@ -236,34 +305,109 @@ def create_request(
duplicate = _find_duplicate_pending_request(
user_db,
user_id=user_id,
title=_normalize_match_text(validated_book_data.get("title")),
author=_normalize_match_text(validated_book_data.get("author")),
content_type=normalized_content_type,
title=_normalize_match_text(prepared_request["book_data"].get("title")),
author=_normalize_match_text(prepared_request["book_data"].get("author")),
content_type=prepared_request["content_type"],
)
if duplicate is not None:
msg = "Duplicate pending request exists for this title/author/content_type"
raise RequestServiceError(
"Duplicate pending request exists for this title/author/content_type",
msg,
status_code=409,
code="duplicate_pending_request",
)
try:
return user_db.create_request(
return user_db.create_request(**prepared_request)
except (ValueError, TypeError) as exc:
raise RequestServiceError(str(exc), status_code=400) from exc
def create_requests(
user_db: UserDB,
*,
requests: list[dict[str, Any]],
) -> list[dict[str, Any]]:
"""Create multiple pending requests atomically after validation."""
if not isinstance(requests, list) or len(requests) == 0:
msg = "requests must contain at least one request"
raise RequestServiceError(msg, status_code=400)
prepared_requests: list[dict[str, Any]] = []
pending_counts_by_user: dict[int, int] = {}
seen_request_keys: set[tuple[int, str, str, str]] = set()
for request in requests:
if not isinstance(request, dict):
msg = "requests must contain objects"
raise RequestServiceError(msg, status_code=400)
user_id = int(request["user_id"])
prepared_request = _prepare_request_create(
user_id=user_id,
source_hint=source_hint,
content_type=normalized_content_type,
request_level=normalized_request_level,
policy_mode=normalized_policy_mode,
book_data=validated_book_data,
release_data=release_data,
note=normalized_note,
source_hint=request.get("source_hint"),
content_type=request.get("content_type"),
request_level=request.get("request_level"),
policy_mode=request.get("policy_mode"),
book_data=request.get("book_data"),
release_data=request.get("release_data"),
note=request.get("note"),
)
request_key = (
user_id,
_normalize_match_text(prepared_request["book_data"].get("title")),
_normalize_match_text(prepared_request["book_data"].get("author")),
prepared_request["content_type"],
)
if request_key in seen_request_keys:
msg = "Duplicate pending request exists for this title/author/content_type"
raise RequestServiceError(
msg,
status_code=409,
code="duplicate_pending_request",
)
seen_request_keys.add(request_key)
max_pending_per_user = request.get("max_pending_per_user")
if max_pending_per_user is not None:
existing_pending = pending_counts_by_user.get(user_id)
if existing_pending is None:
existing_pending = user_db.count_user_pending_requests(user_id)
if existing_pending >= max_pending_per_user:
msg = "Maximum pending requests reached for this user"
raise RequestServiceError(
msg,
status_code=409,
code="max_pending_reached",
)
pending_counts_by_user[user_id] = existing_pending + 1
duplicate = _find_duplicate_pending_request(
user_db,
user_id=user_id,
title=request_key[1],
author=request_key[2],
content_type=request_key[3],
)
if duplicate is not None:
msg = "Duplicate pending request exists for this title/author/content_type"
raise RequestServiceError(
msg,
status_code=409,
code="duplicate_pending_request",
)
prepared_requests.append(prepared_request)
try:
return user_db.create_requests(prepared_requests)
except ValueError as exc:
raise RequestServiceError(str(exc), status_code=400) from exc
def ensure_request_access(
user_db: "UserDB",
user_db: UserDB,
*,
request_id: int,
actor_user_id: int | None,
@@ -272,26 +416,28 @@ def ensure_request_access(
"""Get request by ID and enforce ownership for non-admin actors."""
request_row = user_db.get_request(request_id)
if request_row is None:
raise RequestServiceError("Request not found", status_code=404)
msg = "Request not found"
raise RequestServiceError(msg, status_code=404)
if not is_admin:
if actor_user_id is None or request_row["user_id"] != actor_user_id:
raise RequestServiceError("Forbidden", status_code=403)
if not is_admin and (actor_user_id is None or request_row["user_id"] != actor_user_id):
msg = "Forbidden"
raise RequestServiceError(msg, status_code=403)
return request_row
def _require_pending(request_row: dict[str, Any]) -> None:
if request_row["status"] != RequestStatus.PENDING:
msg = "Request is already in a terminal state"
raise RequestServiceError(
"Request is already in a terminal state",
msg,
status_code=409,
code="stale_transition",
)
def cancel_request(
user_db: "UserDB",
user_db: UserDB,
*,
request_id: int,
actor_user_id: int,
@@ -311,16 +457,18 @@ def cancel_request(
expected_current_status=RequestStatus.PENDING,
status=RequestStatus.CANCELLED,
)
except TypeError as exc:
raise RequestServiceError(str(exc), status_code=400) from exc
except ValueError as exc:
raise RequestServiceError(str(exc), status_code=409, code="stale_transition") from exc
def reject_request(
user_db: "UserDB",
user_db: UserDB,
*,
request_id: int,
admin_user_id: int,
admin_note: Any = None,
admin_note: object = None,
) -> dict[str, Any]:
"""Reject a pending request as admin."""
request_row = ensure_request_access(
@@ -342,19 +490,21 @@ def reject_request(
reviewed_by=admin_user_id,
reviewed_at=_now_timestamp(),
)
except TypeError as exc:
raise RequestServiceError(str(exc), status_code=400) from exc
except ValueError as exc:
raise RequestServiceError(str(exc), status_code=409, code="stale_transition") from exc
def fulfil_request(
user_db: "UserDB",
user_db: UserDB,
*,
request_id: int,
admin_user_id: int,
queue_release: Callable[..., tuple[bool, str | None]],
release_data: Any = None,
admin_note: Any = None,
manual_approval: Any = False,
release_data: object = None,
admin_note: object = None,
manual_approval: object = False,
) -> dict[str, Any]:
"""Fulfil a pending request and queue the release under requesting-user identity."""
request_row = ensure_request_access(
@@ -368,11 +518,15 @@ def fulfil_request(
normalized_admin_note = _normalize_admin_note(admin_note)
if not isinstance(manual_approval, bool):
raise RequestServiceError("manual_approval must be a boolean", status_code=400)
msg = "manual_approval must be a boolean"
raise RequestServiceError(msg, status_code=400)
selected_release_data = release_data if release_data is not None else request_row.get("release_data")
selected_release_data = (
release_data if release_data is not None else request_row.get("release_data")
)
if selected_release_data is not None and not isinstance(selected_release_data, dict):
raise RequestServiceError("release_data must be an object", status_code=400)
msg = "release_data must be an object"
raise RequestServiceError(msg, status_code=400)
if selected_release_data is None and manual_approval:
try:
@@ -388,12 +542,15 @@ def fulfil_request(
reviewed_by=admin_user_id,
reviewed_at=_now_timestamp(),
)
except TypeError as exc:
raise RequestServiceError(str(exc), status_code=400) from exc
except ValueError as exc:
raise RequestServiceError(str(exc), status_code=409, code="stale_transition") from exc
if selected_release_data is None:
msg = "release_data is required to fulfil requests"
raise RequestServiceError(
"release_data is required to fulfil requests",
msg,
status_code=400,
)
@@ -401,7 +558,8 @@ def fulfil_request(
requester = user_db.get_user(user_id=request_row["user_id"])
if requester is None:
raise RequestServiceError("Requesting user not found", status_code=404)
msg = "Requesting user not found"
raise RequestServiceError(msg, status_code=404)
original_release_data = request_row.get("release_data")
try:
@@ -417,6 +575,8 @@ def fulfil_request(
reviewed_by=admin_user_id,
reviewed_at=_now_timestamp(),
)
except TypeError as exc:
raise RequestServiceError(str(exc), status_code=400) from exc
except ValueError as exc:
raise RequestServiceError(str(exc), status_code=409, code="stale_transition") from exc
@@ -453,7 +613,7 @@ def fulfil_request(
def reopen_failed_request(
user_db: "UserDB",
user_db: UserDB,
*,
request_id: int,
failure_reason: str | None = None,
+38 -27
View File
@@ -1,18 +1,23 @@
"""Helpers for building release search plans from metadata and user input."""
from __future__ import annotations
from collections.abc import Iterable
from dataclasses import dataclass
from typing import List, Optional
MANUAL_QUERY_MAX_LEN = 256
from typing import TYPE_CHECKING
from shelfmark.core.config import config
from shelfmark.core.models import SearchFilters
from shelfmark.metadata_providers import (
BookMetadata,
group_languages_by_localized_title,
build_localized_search_titles,
group_languages_by_localized_title,
)
if TYPE_CHECKING:
from shelfmark.core.models import SearchFilters
MANUAL_QUERY_MAX_LEN = 256
@dataclass(frozen=True)
class ReleaseSearchVariant:
@@ -20,10 +25,11 @@ class ReleaseSearchVariant:
title: str
author: str
languages: Optional[List[str]] = None
languages: list[str] | None = None
@property
def query(self) -> str:
"""Return the combined title-and-author query for this variant."""
return " ".join(part for part in [self.title, self.author] if part).strip()
@@ -31,28 +37,33 @@ class ReleaseSearchVariant:
class ReleaseSearchPlan:
"""Pre-computed search inputs shared across release sources."""
languages: Optional[List[str]]
isbn_candidates: List[str]
languages: list[str] | None
isbn_candidates: list[str]
author: str
title_variants: List[ReleaseSearchVariant]
grouped_title_variants: List[ReleaseSearchVariant]
manual_query: Optional[str] = None
indexers: Optional[List[str]] = None # Indexer names for Prowlarr (overrides settings)
source_filters: Optional[SearchFilters] = None
title_variants: list[ReleaseSearchVariant]
grouped_title_variants: list[ReleaseSearchVariant]
manual_query: str | None = None
indexers: list[str] | None = None # Indexer names for Prowlarr (overrides settings)
source_filters: SearchFilters | None = None
@property
def primary_query(self) -> str:
"""Return the first expanded title query, if one exists."""
return self.title_variants[0].query if self.title_variants else ""
def _normalize_languages(languages: Optional[List[str]]) -> Optional[List[str]]:
def _normalize_languages(languages: list[str] | None) -> list[str] | None:
if not languages:
default = config.BOOK_LANGUAGE
if not default:
default = getattr(config, "BOOK_LANGUAGE", None)
if isinstance(default, str):
default_values: list[object] = [default]
elif isinstance(default, Iterable) and not isinstance(default, (bytes, bytearray, dict)):
default_values = list(default)
else:
return None
return [str(lang).strip() for lang in default if str(lang).strip()]
return [str(lang).strip() for lang in default_values if str(lang).strip()]
normalized: List[str] = []
normalized: list[str] = []
for lang in languages:
if not lang:
continue
@@ -87,11 +98,12 @@ def _pick_search_title(book: BookMetadata) -> str:
def build_release_search_plan(
book: BookMetadata,
languages: Optional[List[str]] = None,
manual_query: Optional[str] = None,
indexers: Optional[List[str]] = None,
source_filters: Optional[SearchFilters] = None,
languages: list[str] | None = None,
manual_query: str | None = None,
indexers: list[str] | None = None,
source_filters: SearchFilters | None = None,
) -> ReleaseSearchPlan:
"""Build normalized search variants shared across release sources."""
resolved_languages = _normalize_languages(languages)
resolved_manual_query = None
@@ -115,7 +127,7 @@ def build_release_search_plan(
source_filters=source_filters,
)
isbn_candidates: List[str] = []
isbn_candidates: list[str] = []
if book.isbn_13:
isbn_candidates.append(book.isbn_13)
if book.isbn_10 and book.isbn_10 not in isbn_candidates:
@@ -135,7 +147,7 @@ def build_release_search_plan(
titles_by_language=titles_by_language,
)
grouped_variants: List[ReleaseSearchVariant] = [
grouped_variants: list[ReleaseSearchVariant] = [
ReleaseSearchVariant(title=title, author=author, languages=langs)
for title, langs in grouped
if title
@@ -148,7 +160,7 @@ def build_release_search_plan(
excluded_languages={"en", "eng", "english"},
)
title_variants: List[ReleaseSearchVariant] = [
title_variants: list[ReleaseSearchVariant] = [
ReleaseSearchVariant(title=title, author=author, languages=None)
for title in expanded_titles
if title
@@ -157,8 +169,7 @@ def build_release_search_plan(
# If no titles could be built, fall back to ISBN queries.
if not title_variants and isbn_candidates:
title_variants = [
ReleaseSearchVariant(title=isbn, author="", languages=None)
for isbn in isbn_candidates
ReleaseSearchVariant(title=isbn, author="", languages=None) for isbn in isbn_candidates
]
return ReleaseSearchPlan(
+119 -55
View File
@@ -1,9 +1,10 @@
"""Self-service user account routes."""
import sqlite3
from functools import wraps
from typing import Any, Callable, Mapping
from typing import TYPE_CHECKING, Any
from flask import Flask, g, jsonify, request, session
from flask import Flask, Response, g, jsonify, request, session
from werkzeug.security import generate_password_hash
from shelfmark.config.env import CWA_DB_PATH
@@ -20,13 +21,19 @@ from shelfmark.core.auth_modes import (
load_active_auth_mode,
normalize_auth_source,
)
from shelfmark.core.config import config as app_config
from shelfmark.core.logger import setup_logger
from shelfmark.core.settings_registry import load_config_file
from shelfmark.core.user_settings_overrides import (
build_user_preferences_payload as _build_user_preferences_payload,
)
from shelfmark.core.user_settings_overrides import (
get_ordered_user_overridable_fields as _get_ordered_user_overridable_fields,
)
from shelfmark.core.user_db import UserDB
if TYPE_CHECKING:
from collections.abc import Callable, Mapping
from shelfmark.core.user_db import UserDB
logger = setup_logger(__name__)
@@ -41,13 +48,19 @@ _VALID_SELF_SETTINGS_SECTIONS = (
_SELF_SETTINGS_SECTION_NOTIFICATIONS,
)
_DEFAULT_VISIBLE_SELF_SETTINGS_SECTIONS = list(_VALID_SELF_SETTINGS_SECTIONS)
_USER_PREFERENCES_FALLBACK_ERRORS = (ImportError, OSError, RuntimeError, TypeError, sqlite3.Error)
_CONFIG_REFRESH_ERRORS = (ImportError, OSError, RuntimeError, TypeError, ValueError)
def _get_current_user(user_db: UserDB) -> tuple[int | None, dict[str, Any] | None, tuple[Any, int] | None]:
def _get_current_user(
user_db: UserDB,
) -> tuple[int | None, dict[str, Any] | None, tuple[Response, int] | None]:
raw_user_id = session.get("db_user_id")
if raw_user_id is None:
return None, None, (jsonify({"error": "Invalid user context"}), 400)
try:
user_id = int(raw_user_id)
except (TypeError, ValueError):
except TypeError, ValueError:
return None, None, (jsonify({"error": "Invalid user context"}), 400)
user = user_db.get_user(user_id=user_id)
@@ -83,7 +96,37 @@ def _serialize_self_user(user: Mapping[str, Any], auth_mode: str) -> dict[str, A
return payload
def _normalize_visible_self_settings_sections(raw_sections: Any) -> list[str]:
def _build_optional_user_preferences(
user_db: UserDB,
*,
user_id: int,
tab_name: str,
missing_tab_error: str,
preference_label: str,
) -> tuple[dict[str, Any] | None, tuple[Response, int] | None]:
try:
return _build_user_preferences_payload(user_db, user_id, tab_name), None
except ValueError as exc:
if str(exc) == missing_tab_error:
return None, (jsonify({"error": missing_tab_error}), 500)
logger.warning(
"Failed to build user %s preferences for user_id=%s: %s",
preference_label,
user_id,
exc,
)
return None, None
except _USER_PREFERENCES_FALLBACK_ERRORS as exc:
logger.warning(
"Failed to build user %s preferences for user_id=%s: %s",
preference_label,
user_id,
exc,
)
return None, None
def _normalize_visible_self_settings_sections(raw_sections: object) -> list[str]:
"""Normalize users.VISIBLE_SELF_SETTINGS_SECTIONS to a safe ordered list."""
if raw_sections is None:
return list(_DEFAULT_VISIBLE_SELF_SETTINGS_SECTIONS)
@@ -91,7 +134,9 @@ def _normalize_visible_self_settings_sections(raw_sections: Any) -> list[str]:
if isinstance(raw_sections, str):
candidate_sections = [s.strip() for s in raw_sections.split(",") if s.strip()]
elif isinstance(raw_sections, (list, tuple, set)):
candidate_sections = [str(section).strip() for section in raw_sections if str(section).strip()]
candidate_sections = [
str(section).strip() for section in raw_sections if str(section).strip()
]
else:
return list(_DEFAULT_VISIBLE_SELF_SETTINGS_SECTIONS)
@@ -108,8 +153,10 @@ def _normalize_visible_self_settings_sections(raw_sections: Any) -> list[str]:
def _get_visible_self_settings_sections() -> list[str]:
users_config = load_config_file("users")
raw_sections = users_config.get(_VISIBLE_SELF_SETTINGS_SECTIONS_KEY)
raw_sections = app_config.get(
_VISIBLE_SELF_SETTINGS_SECTIONS_KEY,
list(_DEFAULT_VISIBLE_SELF_SETTINGS_SECTIONS),
)
return _normalize_visible_self_settings_sections(raw_sections)
@@ -118,14 +165,10 @@ def _get_allowed_self_settings_keys(visible_sections: list[str]) -> set[str]:
visible_sections_set = set(visible_sections)
if _SELF_SETTINGS_SECTION_DELIVERY in visible_sections_set:
allowed_keys |= {
key for key, _field in _get_ordered_user_overridable_fields("downloads")
}
allowed_keys |= {key for key, _field in _get_ordered_user_overridable_fields("downloads")}
if _SELF_SETTINGS_SECTION_SEARCH in visible_sections_set:
allowed_keys |= {
key for key, _field in _get_ordered_user_overridable_fields("search_mode")
}
allowed_keys |= {key for key, _field in _get_ordered_user_overridable_fields("search_mode")}
if _SELF_SETTINGS_SECTION_NOTIFICATIONS in visible_sections_set:
allowed_keys |= {
@@ -138,28 +181,36 @@ def _get_allowed_self_settings_keys(visible_sections: list[str]) -> set[str]:
def register_self_user_routes(app: Flask, user_db: UserDB) -> None:
"""Register self-service user endpoints."""
def _require_authenticated_user(f: Callable[..., Any]) -> Callable[..., Any]:
"""Decorator requiring an authenticated session linked to a local user row.
def _require_authenticated_user(
f: Callable[..., Response | tuple[Response, int]],
) -> Callable[..., Response | tuple[Response, int]]:
"""Require an authenticated session linked to a local user row.
Caches the resolved auth_mode in ``g.auth_mode`` for the request.
"""
@wraps(f)
def decorated(*args, **kwargs):
def decorated(*args: object, **kwargs: object) -> Response | tuple[Response, int]:
auth_mode = load_active_auth_mode(CWA_DB_PATH, user_db=user_db)
g.auth_mode = auth_mode
if auth_mode != "none" and "user_id" not in session:
return jsonify({"error": "Authentication required"}), 401
if "db_user_id" not in session:
return jsonify({"error": "Authenticated session is missing local user context"}), 403
return jsonify(
{"error": "Authenticated session is missing local user context"}
), 403
return f(*args, **kwargs)
return decorated
@app.route("/api/users/me/edit-context", methods=["GET"])
@_require_authenticated_user
def users_me_edit_context():
def users_me_edit_context() -> Response | tuple[Response, int]:
user_id, user, user_error = _get_current_user(user_db)
if user_error:
return user_error
if user_id is None or user is None:
return jsonify({"error": "User not found"}), 404
serialized_user = _serialize_self_user(user, g.auth_mode)
serialized_user["settings"] = user_db.get_user_settings(user_id)
@@ -167,33 +218,39 @@ def register_self_user_routes(app: Flask, user_db: UserDB) -> None:
delivery_preferences = None
if _SELF_SETTINGS_SECTION_DELIVERY in visible_self_settings_sections:
try:
delivery_preferences = _build_user_preferences_payload(user_db, user_id, "downloads")
except ValueError:
return jsonify({"error": "Downloads settings tab not found"}), 500
except Exception as exc:
logger.warning(f"Failed to build user delivery preferences for user_id={user_id}: {exc}")
delivery_preferences = None
delivery_preferences, error_response = _build_optional_user_preferences(
user_db,
user_id=user_id,
tab_name="downloads",
missing_tab_error="Downloads settings tab not found",
preference_label="delivery",
)
if error_response:
return error_response
search_preferences = None
if _SELF_SETTINGS_SECTION_SEARCH in visible_self_settings_sections:
try:
search_preferences = _build_user_preferences_payload(user_db, user_id, "search_mode")
except ValueError:
return jsonify({"error": "Search mode settings tab not found"}), 500
except Exception as exc:
logger.warning(f"Failed to build user search preferences for user_id={user_id}: {exc}")
search_preferences = None
search_preferences, error_response = _build_optional_user_preferences(
user_db,
user_id=user_id,
tab_name="search_mode",
missing_tab_error="Search mode settings tab not found",
preference_label="search",
)
if error_response:
return error_response
notification_preferences = None
if _SELF_SETTINGS_SECTION_NOTIFICATIONS in visible_self_settings_sections:
try:
notification_preferences = _build_user_preferences_payload(user_db, user_id, "notifications")
except ValueError:
return jsonify({"error": "Notifications settings tab not found"}), 500
except Exception as exc:
logger.warning(f"Failed to build user notification preferences for user_id={user_id}: {exc}")
notification_preferences = None
notification_preferences, error_response = _build_optional_user_preferences(
user_db,
user_id=user_id,
tab_name="notifications",
missing_tab_error="Notifications settings tab not found",
preference_label="notification",
)
if error_response:
return error_response
user_overridable_keys = sorted(
set(delivery_preferences.get("keys", []) if delivery_preferences else [])
@@ -214,7 +271,7 @@ def register_self_user_routes(app: Flask, user_db: UserDB) -> None:
@app.route("/api/users/me/notification-preferences/test", methods=["POST"])
@_require_authenticated_user
def users_me_test_notification_preferences():
def users_me_test_notification_preferences() -> Response | tuple[Response, int]:
user_id, _user, user_error = _get_current_user(user_db)
if user_error:
return user_error
@@ -230,10 +287,12 @@ def register_self_user_routes(app: Flask, user_db: UserDB) -> None:
@app.route("/api/users/me", methods=["PUT"])
@_require_authenticated_user
def users_me_update():
def users_me_update() -> Response | tuple[Response, int]:
user_id, user, user_error = _get_current_user(user_db)
if user_error:
return user_error
if user_id is None or user is None:
return jsonify({"error": "User not found"}), 404
data = request.get_json() or {}
if not isinstance(data, dict):
@@ -252,7 +311,9 @@ def register_self_user_routes(app: Flask, user_db: UserDB) -> None:
}
), 400
if len(password) < MIN_PASSWORD_LENGTH:
return jsonify({"error": f"Password must be at least {MIN_PASSWORD_LENGTH} characters"}), 400
return jsonify(
{"error": f"Password must be at least {MIN_PASSWORD_LENGTH} characters"}
), 400
user_db.update_user(user_id, password_hash=generate_password_hash(password))
user_fields: dict[str, Any] = {}
@@ -271,10 +332,9 @@ def register_self_user_routes(app: Flask, user_db: UserDB) -> None:
)
email_changed = "email" in user_fields and user_fields["email"] != user.get("email")
display_name_changed = (
"display_name" in user_fields
and user_fields["display_name"] != user.get("display_name")
)
display_name_changed = "display_name" in user_fields and user_fields[
"display_name"
] != user.get("display_name")
if email_changed and not capabilities["canEditEmail"]:
if auth_source == AUTH_SOURCE_CWA:
@@ -312,7 +372,9 @@ def register_self_user_routes(app: Flask, user_db: UserDB) -> None:
return jsonify({"error": "Settings must be an object"}), 400
visible_self_settings_sections = _get_visible_self_settings_sections()
allowed_user_settings_keys = _get_allowed_self_settings_keys(visible_self_settings_sections)
allowed_user_settings_keys = _get_allowed_self_settings_keys(
visible_self_settings_sections
)
disallowed_keys = sorted(
key for key in settings_payload if key not in allowed_user_settings_keys
)
@@ -337,11 +399,13 @@ def register_self_user_routes(app: Flask, user_db: UserDB) -> None:
user_db.set_user_settings(user_id, validated_settings)
try:
from shelfmark.core.config import config as app_config
app_config.refresh(force=True)
except Exception:
pass
except _CONFIG_REFRESH_ERRORS as exc:
logger.warning(
"Updated settings for user %s but failed to refresh runtime config: %s",
user_id,
exc,
)
updated = user_db.get_user(user_id=user_id)
if not updated:
@@ -349,5 +413,5 @@ def register_self_user_routes(app: Flask, user_db: UserDB) -> None:
result = _serialize_self_user(updated, g.auth_mode)
result["settings"] = user_db.get_user_settings(user_id)
logger.info(f"User {user_id} updated their own account")
logger.info("User %s updated their own account", user_id)
return jsonify(result)
File diff suppressed because it is too large Load Diff
+283 -171
View File
@@ -4,12 +4,12 @@ import json
import os
import sqlite3
import threading
from typing import Any, Dict, List, Optional
from pathlib import Path
from typing import Any, ClassVar
from shelfmark.core.auth_modes import AUTH_SOURCE_BUILTIN, AUTH_SOURCE_SET
from shelfmark.core.activity_view_state_service import user_viewer_scope
from shelfmark.core.auth_modes import AUTH_SOURCE_BUILTIN, AUTH_SOURCE_SET
from shelfmark.core.logger import setup_logger
from shelfmark.core.request_helpers import normalize_optional_positive_int
from shelfmark.core.models import QueueStatus
from shelfmark.core.request_validation import (
DELIVERY_STATE_NONE,
@@ -85,6 +85,7 @@ CREATE TABLE IF NOT EXISTS download_history (
final_status TEXT NOT NULL,
status_message TEXT,
download_path TEXT,
retry_payload TEXT,
queued_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
terminal_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);
@@ -115,16 +116,24 @@ WHERE dismissed_at IS NOT NULL;
"""
def get_users_db_path(config_dir: Optional[str] = None) -> str:
def _require_loaded_user(user: dict[str, Any] | None) -> dict[str, Any]:
"""Return a loaded user row or raise when the DB insert result is inconsistent."""
if user is None:
msg = "Failed to load newly created user"
raise RuntimeError(msg)
return user
def get_users_db_path(config_dir: str | None = None) -> str:
"""Return the configured users database path."""
root = config_dir or os.environ.get("CONFIG_DIR", "/config")
return os.path.join(root, "users.db")
return str(Path(root) / "users.db")
def sync_builtin_admin_user(
username: str,
password_hash: str,
db_path: Optional[str] = None,
db_path: str | None = None,
) -> None:
"""Ensure a local admin user exists for configured builtin credentials."""
normalized_username = (username or "").strip()
@@ -137,7 +146,9 @@ def sync_builtin_admin_user(
existing = user_db.get_user(username=normalized_username)
if existing:
existing_auth_source = str(existing.get("auth_source") or AUTH_SOURCE_BUILTIN).strip().lower()
existing_auth_source = (
str(existing.get("auth_source") or AUTH_SOURCE_BUILTIN).strip().lower()
)
if existing_auth_source != AUTH_SOURCE_BUILTIN:
logger.warning(
"Skipped builtin admin sync for username '%s' because it belongs to auth_source='%s'",
@@ -154,7 +165,7 @@ def sync_builtin_admin_user(
updates["auth_source"] = AUTH_SOURCE_BUILTIN
if updates:
user_db.update_user(existing["id"], **updates)
logger.info(f"Updated local admin user '{normalized_username}' from builtin settings")
logger.info("Updated local admin user '%s' from builtin settings", normalized_username)
return
user_db.create_user(
@@ -163,15 +174,16 @@ def sync_builtin_admin_user(
auth_source=AUTH_SOURCE_BUILTIN,
role="admin",
)
logger.info(f"Created local admin user '{normalized_username}' from builtin settings")
logger.info("Created local admin user '%s' from builtin settings", normalized_username)
class UserDB:
"""Thread-safe SQLite user database."""
_VALID_AUTH_SOURCES = set(AUTH_SOURCE_SET)
_VALID_AUTH_SOURCES: ClassVar[frozenset[str]] = frozenset(AUTH_SOURCE_SET)
def __init__(self, db_path: str):
def __init__(self, db_path: str) -> None:
"""Initialize the user database wrapper for the given SQLite path."""
self._db_path = db_path
self._lock = threading.Lock()
@@ -190,6 +202,7 @@ class UserDB:
self._migrate_auth_source_column(conn)
self._migrate_request_delivery_columns(conn)
self._migrate_download_history_queued_at(conn)
self._migrate_download_history_retry_payload(conn)
conn.commit()
# WAL mode must be changed outside an open transaction.
conn.execute("PRAGMA journal_mode=WAL")
@@ -202,14 +215,10 @@ class UserDB:
column_names = {str(col["name"]) for col in columns}
if "auth_source" not in column_names:
conn.execute(
"ALTER TABLE users ADD COLUMN auth_source TEXT NOT NULL DEFAULT 'builtin'"
)
conn.execute("ALTER TABLE users ADD COLUMN auth_source TEXT NOT NULL DEFAULT 'builtin'")
# Backfill OIDC-origin users created before auth_source existed.
conn.execute(
"UPDATE users SET auth_source = 'oidc' WHERE oidc_subject IS NOT NULL"
)
conn.execute("UPDATE users SET auth_source = 'oidc' WHERE oidc_subject IS NOT NULL")
# Defensive cleanup for any legacy null/blank values.
conn.execute(
"UPDATE users SET auth_source = 'builtin' WHERE auth_source IS NULL OR auth_source = ''"
@@ -254,19 +263,27 @@ class UserDB:
"UPDATE download_history SET queued_at = CURRENT_TIMESTAMP WHERE queued_at IS NULL"
)
def _migrate_download_history_retry_payload(self, conn: sqlite3.Connection) -> None:
"""Ensure download_history.retry_payload exists for restart-safe retries."""
columns = conn.execute("PRAGMA table_info(download_history)").fetchall()
column_names = {str(col["name"]) for col in columns}
if "retry_payload" not in column_names:
conn.execute("ALTER TABLE download_history ADD COLUMN retry_payload TEXT")
def create_user(
self,
username: str,
email: Optional[str] = None,
display_name: Optional[str] = None,
password_hash: Optional[str] = None,
oidc_subject: Optional[str] = None,
email: str | None = None,
display_name: str | None = None,
password_hash: str | None = None,
oidc_subject: str | None = None,
auth_source: str = "builtin",
role: str = "user",
) -> Dict[str, Any]:
) -> dict[str, Any]:
"""Create a new user. Raises ValueError if username or oidc_subject already exists."""
if auth_source not in self._VALID_AUTH_SOURCES:
raise ValueError(f"Invalid auth_source: {auth_source}")
msg = f"Invalid auth_source: {auth_source}"
raise ValueError(msg)
with self._lock:
conn = self._connect()
try:
@@ -287,27 +304,30 @@ class UserDB:
)
conn.commit()
user_id = cursor.lastrowid
return self._get_user_by_id(conn, user_id)
if not isinstance(user_id, int):
msg = "Failed to create user"
raise TypeError(msg)
created_user = self._get_user_by_id(conn, user_id)
return _require_loaded_user(created_user)
except sqlite3.IntegrityError as e:
raise ValueError(f"User already exists: {e}")
msg = f"User already exists: {e}"
raise ValueError(msg) from e
finally:
conn.close()
def get_user(
self,
user_id: Optional[int] = None,
username: Optional[str] = None,
oidc_subject: Optional[str] = None,
) -> Optional[Dict[str, Any]]:
user_id: int | None = None,
username: str | None = None,
oidc_subject: str | None = None,
) -> dict[str, Any] | None:
"""Get a user by id, username, or oidc_subject. Returns None if not found."""
conn = self._connect()
try:
if user_id is not None:
return self._get_user_by_id(conn, user_id)
elif username is not None:
row = conn.execute(
"SELECT * FROM users WHERE username = ?", (username,)
).fetchone()
if username is not None:
row = conn.execute("SELECT * FROM users WHERE username = ?", (username,)).fetchone()
elif oidc_subject is not None:
row = conn.execute(
"SELECT * FROM users WHERE oidc_subject = ?", (oidc_subject,)
@@ -318,37 +338,49 @@ class UserDB:
finally:
conn.close()
def _get_user_by_id(self, conn: sqlite3.Connection, user_id: int) -> Optional[Dict[str, Any]]:
def _get_user_by_id(self, conn: sqlite3.Connection, user_id: int) -> dict[str, Any] | None:
row = conn.execute("SELECT * FROM users WHERE id = ?", (user_id,)).fetchone()
return dict(row) if row else None
_ALLOWED_UPDATE_COLUMNS = {
"email",
"display_name",
"password_hash",
"oidc_subject",
"auth_source",
"role",
_ALLOWED_UPDATE_COLUMNS: ClassVar[frozenset[str]] = frozenset(
{
"email",
"display_name",
"password_hash",
"oidc_subject",
"auth_source",
"role",
}
)
_USER_UPDATE_STATEMENTS: ClassVar[dict[str, str]] = {
"email": "UPDATE users SET email = ? WHERE id = ?",
"display_name": "UPDATE users SET display_name = ? WHERE id = ?",
"password_hash": "UPDATE users SET password_hash = ? WHERE id = ?",
"oidc_subject": "UPDATE users SET oidc_subject = ? WHERE id = ?",
"auth_source": "UPDATE users SET auth_source = ? WHERE id = ?",
"role": "UPDATE users SET role = ? WHERE id = ?",
}
def update_user(self, user_id: int, **kwargs) -> None:
def update_user(self, user_id: int, **kwargs: object) -> None:
"""Update user fields. Raises ValueError if user not found or invalid column."""
if not kwargs:
return
for k in kwargs:
if k not in self._ALLOWED_UPDATE_COLUMNS:
raise ValueError(f"Invalid column: {k}")
msg = f"Invalid column: {k}"
raise ValueError(msg)
if "auth_source" in kwargs and kwargs["auth_source"] not in self._VALID_AUTH_SOURCES:
raise ValueError(f"Invalid auth_source: {kwargs['auth_source']}")
msg = f"Invalid auth_source: {kwargs['auth_source']}"
raise ValueError(msg)
with self._lock:
conn = self._connect()
try:
# Verify user exists
if not self._get_user_by_id(conn, user_id):
raise ValueError(f"User {user_id} not found")
sets = ", ".join(f"{k} = ?" for k in kwargs)
values = list(kwargs.values()) + [user_id]
conn.execute(f"UPDATE users SET {sets} WHERE id = ?", values)
msg = f"User {user_id} not found"
raise ValueError(msg)
for column, value in kwargs.items():
conn.execute(self._USER_UPDATE_STATEMENTS[column], (value, user_id))
conn.commit()
finally:
conn.close()
@@ -364,26 +396,24 @@ class UserDB:
).fetchall()
request_item_keys = [f"request:{row['id']}" for row in request_rows]
if request_item_keys:
placeholders = ",".join("?" for _ in request_item_keys)
conn.execute(
f"""
DELETE FROM activity_view_state
WHERE item_type = 'request'
AND item_key IN ({placeholders})
""",
request_item_keys,
conn.executemany(
"DELETE FROM activity_view_state WHERE item_type = 'request' AND item_key = ?",
[(item_key,) for item_key in request_item_keys],
)
conn.execute(
"DELETE FROM activity_view_state WHERE viewer_scope = ?",
(user_viewer_scope(user_id),),
)
conn.execute("UPDATE download_requests SET reviewed_by = NULL WHERE reviewed_by = ?", (user_id,))
conn.execute(
"UPDATE download_requests SET reviewed_by = NULL WHERE reviewed_by = ?",
(user_id,),
)
conn.execute("DELETE FROM users WHERE id = ?", (user_id,))
conn.commit()
finally:
conn.close()
def list_users(self) -> List[Dict[str, Any]]:
def list_users(self) -> list[dict[str, Any]]:
"""List all users."""
conn = self._connect()
try:
@@ -405,7 +435,7 @@ class UserDB:
finally:
conn.close()
def get_user_settings(self, user_id: int) -> Dict[str, Any]:
def get_user_settings(self, user_id: int) -> dict[str, Any]:
"""Get per-user settings. Returns empty dict if none set."""
conn = self._connect()
try:
@@ -418,7 +448,7 @@ class UserDB:
finally:
conn.close()
def set_user_settings(self, user_id: int, settings: Dict[str, Any]) -> None:
def set_user_settings(self, user_id: int, settings: dict[str, Any]) -> None:
"""Merge settings into user's existing settings."""
with self._lock:
conn = self._connect()
@@ -445,16 +475,17 @@ class UserDB:
conn.close()
@staticmethod
def _serialize_json(value: Any, field: str) -> Optional[str]:
def _serialize_json(value: Any, field: str) -> str | None:
if value is None:
return None
try:
return json.dumps(value)
except TypeError as exc:
raise ValueError(f"{field} must be JSON-serializable") from exc
msg = f"{field} must be JSON-serializable"
raise ValueError(msg) from exc
@staticmethod
def _parse_request_row(row: Optional[sqlite3.Row]) -> Optional[Dict[str, Any]]:
def _parse_request_row(row: sqlite3.Row | None) -> dict[str, Any] | None:
if row is None:
return None
@@ -466,10 +497,77 @@ class UserDB:
continue
try:
payload[key] = json.loads(raw_value)
except (ValueError, TypeError):
except ValueError, TypeError:
payload[key] = None
return payload
def _insert_request(
self,
conn: sqlite3.Connection,
*,
user_id: int,
content_type: str,
request_level: str,
policy_mode: str,
book_data: dict[str, Any],
release_data: dict[str, Any] | None = None,
status: str = RequestStatus.PENDING,
source_hint: str | None = None,
note: str | None = None,
admin_note: str | None = None,
reviewed_by: int | None = None,
reviewed_at: str | None = None,
delivery_state: str = DELIVERY_STATE_NONE,
delivery_updated_at: str | None = None,
) -> dict[str, Any]:
cursor = conn.execute(
"""
INSERT INTO download_requests (
user_id,
status,
delivery_state,
source_hint,
content_type,
request_level,
policy_mode,
book_data,
release_data,
note,
admin_note,
reviewed_by,
reviewed_at,
delivery_updated_at
)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
(
user_id,
status,
delivery_state,
source_hint,
content_type,
request_level,
policy_mode,
self._serialize_json(book_data, "book_data"),
self._serialize_json(release_data, "release_data"),
note,
admin_note,
reviewed_by,
reviewed_at,
delivery_updated_at,
),
)
request_id = cursor.lastrowid
row = conn.execute(
"SELECT * FROM download_requests WHERE id = ?",
(request_id,),
).fetchone()
parsed = self._parse_request_row(row)
if parsed is None:
msg = f"Request {request_id} not found after creation"
raise ValueError(msg)
return parsed
def create_request(
self,
*,
@@ -477,24 +575,27 @@ class UserDB:
content_type: str,
request_level: str,
policy_mode: str,
book_data: Dict[str, Any],
release_data: Optional[Dict[str, Any]] = None,
book_data: dict[str, Any],
release_data: dict[str, Any] | None = None,
status: str = RequestStatus.PENDING,
source_hint: Optional[str] = None,
note: Optional[str] = None,
admin_note: Optional[str] = None,
reviewed_by: Optional[int] = None,
reviewed_at: Optional[str] = None,
source_hint: str | None = None,
note: str | None = None,
admin_note: str | None = None,
reviewed_by: int | None = None,
reviewed_at: str | None = None,
delivery_state: str = DELIVERY_STATE_NONE,
delivery_updated_at: Optional[str] = None,
) -> Dict[str, Any]:
delivery_updated_at: str | None = None,
) -> dict[str, Any]:
"""Create a download request row and return the created record."""
if not isinstance(book_data, dict):
raise ValueError("book_data must be an object")
msg = "book_data must be an object"
raise TypeError(msg)
if release_data is not None and not isinstance(release_data, dict):
raise ValueError("release_data must be an object when provided")
msg = "release_data must be an object when provided"
raise TypeError(msg)
if not content_type:
raise ValueError("content_type is required")
msg = "content_type is required"
raise ValueError(msg)
normalized_status = normalize_request_status(status)
normalized_delivery_state = normalize_delivery_state(delivery_state)
@@ -504,57 +605,40 @@ class UserDB:
with self._lock:
conn = self._connect()
try:
cursor = conn.execute(
"""
INSERT INTO download_requests (
user_id,
status,
delivery_state,
source_hint,
content_type,
request_level,
policy_mode,
book_data,
release_data,
note,
admin_note,
reviewed_by,
reviewed_at,
delivery_updated_at
)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
(
user_id,
normalized_status,
normalized_delivery_state,
source_hint,
content_type,
normalized_request_level,
normalized_policy_mode,
self._serialize_json(book_data, "book_data"),
self._serialize_json(release_data, "release_data"),
note,
admin_note,
reviewed_by,
reviewed_at,
delivery_updated_at,
),
created = self._insert_request(
conn,
user_id=user_id,
content_type=content_type,
request_level=normalized_request_level,
policy_mode=normalized_policy_mode,
book_data=book_data,
release_data=release_data,
status=normalized_status,
source_hint=source_hint,
note=note,
admin_note=admin_note,
reviewed_by=reviewed_by,
reviewed_at=reviewed_at,
delivery_state=normalized_delivery_state,
delivery_updated_at=delivery_updated_at,
)
conn.commit()
request_id = cursor.lastrowid
row = conn.execute(
"SELECT * FROM download_requests WHERE id = ?",
(request_id,),
).fetchone()
parsed = self._parse_request_row(row)
if parsed is None:
raise ValueError(f"Request {request_id} not found after creation")
return parsed
return created
finally:
conn.close()
def get_request(self, request_id: int) -> Optional[Dict[str, Any]]:
def create_requests(self, requests: list[dict[str, Any]]) -> list[dict[str, Any]]:
"""Create multiple request rows atomically and return them in input order."""
with self._lock:
conn = self._connect()
try:
created = [self._insert_request(conn, **request) for request in requests]
conn.commit()
return created
finally:
conn.close()
def get_request(self, request_id: int) -> dict[str, Any] | None:
"""Get a request row by ID."""
conn = self._connect()
try:
@@ -569,14 +653,14 @@ class UserDB:
def list_requests(
self,
*,
user_id: Optional[int] = None,
status: Optional[str] = None,
limit: Optional[int] = None,
user_id: int | None = None,
status: str | None = None,
limit: int | None = None,
offset: int = 0,
) -> List[Dict[str, Any]]:
) -> list[dict[str, Any]]:
"""List requests with optional user/status filters."""
where_clauses: List[str] = []
params: List[Any] = []
where_clauses: list[str] = []
params: list[Any] = []
if user_id is not None:
where_clauses.append("user_id = ?")
@@ -604,7 +688,7 @@ class UserDB:
conn = self._connect()
try:
rows = conn.execute(query, params).fetchall()
results: List[Dict[str, Any]] = []
results: list[dict[str, Any]] = []
for row in rows:
parsed = self._parse_request_row(row)
if parsed is not None:
@@ -613,43 +697,64 @@ class UserDB:
finally:
conn.close()
_ALLOWED_REQUEST_UPDATE_COLUMNS = {
"status",
"source_hint",
"content_type",
"request_level",
"policy_mode",
"book_data",
"release_data",
"note",
"admin_note",
"reviewed_by",
"reviewed_at",
"delivery_state",
"delivery_updated_at",
"last_failure_reason",
_ALLOWED_REQUEST_UPDATE_COLUMNS: ClassVar[frozenset[str]] = frozenset(
{
"status",
"source_hint",
"content_type",
"request_level",
"policy_mode",
"book_data",
"release_data",
"note",
"admin_note",
"reviewed_by",
"reviewed_at",
"delivery_state",
"delivery_updated_at",
"last_failure_reason",
}
)
_REQUEST_UPDATE_STATEMENTS: ClassVar[dict[str, str]] = {
"status": "UPDATE download_requests SET status = ? WHERE id = ?",
"source_hint": "UPDATE download_requests SET source_hint = ? WHERE id = ?",
"content_type": "UPDATE download_requests SET content_type = ? WHERE id = ?",
"request_level": "UPDATE download_requests SET request_level = ? WHERE id = ?",
"policy_mode": "UPDATE download_requests SET policy_mode = ? WHERE id = ?",
"book_data": "UPDATE download_requests SET book_data = ? WHERE id = ?",
"release_data": "UPDATE download_requests SET release_data = ? WHERE id = ?",
"note": "UPDATE download_requests SET note = ? WHERE id = ?",
"admin_note": "UPDATE download_requests SET admin_note = ? WHERE id = ?",
"reviewed_by": "UPDATE download_requests SET reviewed_by = ? WHERE id = ?",
"reviewed_at": "UPDATE download_requests SET reviewed_at = ? WHERE id = ?",
"delivery_state": "UPDATE download_requests SET delivery_state = ? WHERE id = ?",
"delivery_updated_at": "UPDATE download_requests SET delivery_updated_at = ? WHERE id = ?",
"last_failure_reason": "UPDATE download_requests SET last_failure_reason = ? WHERE id = ?",
}
def update_request(
self,
request_id: int,
expected_current_status: Optional[str] = None,
**kwargs,
) -> Dict[str, Any]:
expected_current_status: str | None = None,
**kwargs: object,
) -> dict[str, Any]:
"""Update request fields and return the updated record."""
if not kwargs:
request = self.get_request(request_id)
if request is None:
raise ValueError(f"Request {request_id} not found")
msg = f"Request {request_id} not found"
raise ValueError(msg)
if expected_current_status is not None:
normalized_expected_status = normalize_request_status(expected_current_status)
if request["status"] != normalized_expected_status:
raise ValueError("Request state changed before update")
msg = "Request state changed before update"
raise ValueError(msg)
return request
for key in kwargs:
if key not in self._ALLOWED_REQUEST_UPDATE_COLUMNS:
raise ValueError(f"Invalid request column: {key}")
msg = f"Invalid request column: {key}"
raise ValueError(msg)
with self._lock:
conn = self._connect()
@@ -660,12 +765,14 @@ class UserDB:
).fetchone()
current = self._parse_request_row(row)
if current is None:
raise ValueError(f"Request {request_id} not found")
msg = f"Request {request_id} not found"
raise ValueError(msg)
if expected_current_status is not None:
normalized_expected_status = normalize_request_status(expected_current_status)
if current["status"] != normalized_expected_status:
raise ValueError("Request state changed before update")
msg = "Request state changed before update"
raise ValueError(msg)
updates = dict(kwargs)
@@ -685,33 +792,35 @@ class UserDB:
if "delivery_updated_at" in updates:
delivery_updated_at = updates["delivery_updated_at"]
if delivery_updated_at is not None and not isinstance(delivery_updated_at, str):
raise ValueError("delivery_updated_at must be a string when provided")
msg = "delivery_updated_at must be a string when provided"
raise TypeError(msg)
if "content_type" in updates and not updates["content_type"]:
raise ValueError("content_type is required")
msg = "content_type is required"
raise ValueError(msg)
if "request_level" in updates:
updates["request_level"] = normalize_request_level(updates["request_level"])
if "book_data" in updates:
if not isinstance(updates["book_data"], dict):
raise ValueError("book_data must be an object")
msg = "book_data must be an object"
raise TypeError(msg)
updates["book_data"] = self._serialize_json(updates["book_data"], "book_data")
if "release_data" in updates:
if updates["release_data"] is not None and not isinstance(updates["release_data"], dict):
raise ValueError("release_data must be an object when provided")
if updates["release_data"] is not None and not isinstance(
updates["release_data"], dict
):
msg = "release_data must be an object when provided"
raise TypeError(msg)
updates["release_data"] = self._serialize_json(
updates["release_data"],
"release_data",
)
set_clause = ", ".join(f"{column} = ?" for column in updates)
values = list(updates.values()) + [request_id]
conn.execute(
f"UPDATE download_requests SET {set_clause} WHERE id = ?",
values,
)
for column, value in updates.items():
conn.execute(self._REQUEST_UPDATE_STATEMENTS[column], (value, request_id))
conn.commit()
updated_row = conn.execute(
@@ -720,7 +829,8 @@ class UserDB:
).fetchone()
parsed = self._parse_request_row(updated_row)
if parsed is None:
raise ValueError(f"Request {request_id} not found after update")
msg = f"Request {request_id} not found after update"
raise ValueError(msg)
return parsed
finally:
conn.close()
@@ -729,8 +839,8 @@ class UserDB:
self,
request_id: int,
*,
failure_reason: Optional[str] = None,
) -> Optional[Dict[str, Any]]:
failure_reason: str | None = None,
) -> dict[str, Any] | None:
"""Reopen a failed fulfilled request so admins can re-approve it."""
normalized_failure_reason = None
if isinstance(failure_reason, str):
@@ -789,9 +899,9 @@ class UserDB:
self,
request_id: int,
*,
release_data: Optional[Dict[str, Any]],
last_failure_reason: Optional[str] = None,
) -> Dict[str, Any]:
release_data: dict[str, Any] | None,
last_failure_reason: str | None = None,
) -> dict[str, Any]:
"""Restore a request to pending after fulfilment claimed it but queueing failed."""
with self._lock:
conn = self._connect()
@@ -802,7 +912,8 @@ class UserDB:
).fetchone()
current = self._parse_request_row(row)
if current is None:
raise ValueError(f"Request {request_id} not found")
msg = f"Request {request_id} not found"
raise ValueError(msg)
conn.execute(
"""
@@ -830,7 +941,8 @@ class UserDB:
conn.commit()
parsed = self._parse_request_row(updated_row)
if parsed is None:
raise ValueError(f"Request {request_id} not found after rollback")
msg = f"Request {request_id} not found after rollback"
raise ValueError(msg)
return parsed
finally:
conn.close()
+20 -9
View File
@@ -1,23 +1,30 @@
"""Shared helpers for user-overridable settings metadata and payloads."""
from typing import Any
from importlib import import_module
from typing import TYPE_CHECKING, Any
from shelfmark.core.settings_registry import load_config_file
from shelfmark.core.user_db import UserDB
if TYPE_CHECKING:
from types import ModuleType
from shelfmark.core.user_db import UserDB
def get_settings_registry():
def get_settings_registry() -> ModuleType:
"""Load settings modules and return the shared settings registry module."""
# Ensure settings modules are loaded before reading registry metadata.
import shelfmark.config.settings # noqa: F401
import shelfmark.config.security # noqa: F401
import shelfmark.config.notifications_settings # noqa: F401
import shelfmark.config.users_settings # noqa: F401
import_module("shelfmark.config.notifications_settings")
import_module("shelfmark.config.security")
import_module("shelfmark.config.settings")
import_module("shelfmark.config.users_settings")
from shelfmark.core import settings_registry
return settings_registry
def get_ordered_user_overridable_fields(tab_name: str) -> list[tuple[str, Any]]:
"""Return user-overridable fields for a tab in UI display order."""
settings_registry = get_settings_registry()
tab = settings_registry.get_settings_tab(tab_name)
if not tab:
@@ -27,13 +34,15 @@ def get_ordered_user_overridable_fields(tab_name: str) -> list[tuple[str, Any]]:
def build_user_preferences_payload(user_db: UserDB, user_id: int, tab_name: str) -> dict[str, Any]:
"""Build the effective user-preferences payload for a settings tab."""
from shelfmark.core.config import config as app_config
settings_registry = get_settings_registry()
ordered_fields = get_ordered_user_overridable_fields(tab_name)
if not ordered_fields:
tab_label = tab_name.capitalize()
raise ValueError(f"{tab_label} settings tab not found")
msg = f"{tab_label} settings tab not found"
raise ValueError(msg)
tab_config = load_config_file(tab_name)
user_settings = user_db.get_user_settings(user_id)
@@ -45,7 +54,9 @@ def build_user_preferences_payload(user_db: UserDB, user_id: int, tab_name: str)
for key, field in ordered_fields:
serialized = settings_registry.serialize_field(field, tab_name, include_value=False)
serialized["fromEnv"] = bool(field.env_supported and settings_registry.is_value_from_env(field))
serialized["fromEnv"] = bool(
field.env_supported and settings_registry.is_value_from_env(field)
)
fields_payload.append(serialized)
global_values[key] = app_config.get(key, field.default)
+64 -33
View File
@@ -4,15 +4,20 @@ import base64
import importlib
import os
import re
from threading import Lock
from types import ModuleType
import sqlite3
from pathlib import Path
from typing import Optional
from threading import Lock
from typing import TYPE_CHECKING
from urllib.parse import urlparse
from shelfmark.core.request_helpers import normalize_optional_text
if TYPE_CHECKING:
from types import ModuleType
def normalize_http_url(
url: Optional[str],
url: str | None,
*,
default_scheme: str = "http",
strip_trailing_slash: bool = True,
@@ -26,7 +31,7 @@ def normalize_http_url(
if not normalized:
return ""
if (normalized.startswith("\"") and normalized.endswith("\"")) or (
if (normalized.startswith('"') and normalized.endswith('"')) or (
normalized.startswith("'") and normalized.endswith("'")
):
normalized = normalized[1:-1].strip()
@@ -34,11 +39,7 @@ def normalize_http_url(
return ""
if allow_special:
special_map = {
value.lower(): value
for value in allow_special
if isinstance(value, str)
}
special_map = {value.lower(): value for value in allow_special if isinstance(value, str)}
special_match = special_map.get(normalized.lower())
if special_match is not None:
return special_match
@@ -59,6 +60,7 @@ def normalize_http_url(
_xmlrpc_patch_lock = Lock()
_xmlrpc_patch_applied = False
_XMLRPC_PATCH_ERRORS = (ImportError, AttributeError, OSError, RuntimeError)
def get_hardened_xmlrpc_client() -> ModuleType:
@@ -72,14 +74,14 @@ def get_hardened_xmlrpc_client() -> ModuleType:
monkey_patch()
_xmlrpc_patch_applied = True
except Exception:
except _XMLRPC_PATCH_ERRORS:
# Keep runtime behavior unchanged if defusedxml is unavailable.
_xmlrpc_patch_applied = False
return importlib.import_module("xmlrpc.client")
def normalize_base_path(value: Optional[str]) -> str:
def normalize_base_path(value: str | None) -> str:
"""Normalize a URL base path for reverse proxy subpath deployments."""
if not isinstance(value, str):
return ""
@@ -101,7 +103,7 @@ def normalize_base_path(value: Optional[str]) -> str:
return path.rstrip("/")
def is_audiobook(content_type: Optional[str]) -> bool:
def is_audiobook(content_type: str | None) -> bool:
"""Check if content type indicates an audiobook."""
return bool(content_type and "audiobook" in content_type.lower())
@@ -156,8 +158,8 @@ def _sanitize_user_for_path(username: str) -> str:
def _resolve_destination_username(
user_id: Optional[int] = None,
username: Optional[str] = None,
user_id: int | None = None,
username: str | None = None,
) -> str:
explicit = str(username or "").strip()
if explicit:
@@ -169,20 +171,20 @@ def _resolve_destination_username(
try:
from shelfmark.core.user_db import UserDB
user_db = UserDB(os.path.join(os.environ.get("CONFIG_DIR", "/config"), "users.db"))
user_db = UserDB(str(Path(os.environ.get("CONFIG_DIR", "/config")) / "users.db"))
user_db.initialize()
user = user_db.get_user(user_id=user_id)
if not user:
return ""
return str(user.get("username") or "").strip()
except Exception:
except ImportError, OSError, sqlite3.Error:
return ""
def _expand_user_destination_placeholder(
path_value: str,
user_id: Optional[int] = None,
username: Optional[str] = None,
user_id: int | None = None,
username: str | None = None,
) -> str:
"""Expand `{User}` placeholders in destination paths."""
if not isinstance(path_value, str):
@@ -198,9 +200,10 @@ def _expand_user_destination_placeholder(
def get_destination(
*,
is_audiobook: bool = False,
user_id: Optional[int] = None,
username: Optional[str] = None,
user_id: int | None = None,
username: str | None = None,
) -> Path:
"""Get base destination directory. Audiobooks fall back to main destination."""
from shelfmark.core.config import config
@@ -219,7 +222,9 @@ def get_destination(
# Main destination (also fallback for audiobooks)
# Check new setting first, then legacy INGEST_DIR
destination = config.get("DESTINATION", "", user_id=user_id) or config.get("INGEST_DIR", "/books")
destination = config.get("DESTINATION", "", user_id=user_id) or config.get(
"INGEST_DIR", "/books"
)
return Path(
_expand_user_destination_placeholder(
str(destination),
@@ -229,12 +234,14 @@ def get_destination(
)
def get_aa_content_type_dir(content_type: Optional[str] = None) -> Optional[Path]:
def get_aa_content_type_dir(content_type: str | None = None) -> Path | None:
"""Get override directory for AA content-type routing if configured."""
from shelfmark.core.config import config
# Check if content-type routing is enabled (new or legacy setting)
if not config.get("AA_CONTENT_TYPE_ROUTING", False) and not config.get("USE_CONTENT_TYPE_DIRECTORIES", False):
if not config.get("AA_CONTENT_TYPE_ROUTING", False) and not config.get(
"USE_CONTENT_TYPE_DIRECTORIES", False
):
return None
if not content_type:
@@ -246,19 +253,23 @@ def get_aa_content_type_dir(content_type: Optional[str] = None) -> Optional[Path
for mapping in (_AA_CONTENT_TYPE_TO_CONFIG_KEY, _LEGACY_CONTENT_TYPE_TO_CONFIG_KEY):
config_key = mapping.get(content_type_lower)
if config_key:
custom_dir = config.get(config_key, "")
if custom_dir:
return Path(custom_dir)
custom_dir = _coerce_config_path(config.get(config_key, ""))
if custom_dir is not None:
return custom_dir
return None
def get_ingest_dir(content_type: Optional[str] = None) -> Path:
"""DEPRECATED: Use get_destination() and get_aa_content_type_dir() instead."""
def get_ingest_dir(content_type: str | None = None) -> Path:
"""Return the legacy ingest directory for a content type."""
from shelfmark.core.config import config
# Check new DESTINATION setting first, then legacy INGEST_DIR
default_ingest_dir = Path(config.get("DESTINATION", "") or config.get("INGEST_DIR", "/books"))
default_ingest_dir = _coerce_config_path(config.get("DESTINATION", "")) or _coerce_config_path(
config.get("INGEST_DIR", "/books")
)
if default_ingest_dir is None:
default_ingest_dir = Path("/books")
if not content_type:
return default_ingest_dir
@@ -271,17 +282,18 @@ def get_ingest_dir(content_type: Optional[str] = None) -> Path:
return default_ingest_dir
def transform_cover_url(cover_url: Optional[str], cache_id: str) -> Optional[str]:
def transform_cover_url(cover_url: str | None, cache_id: str) -> str | None:
"""Transform external cover URL to local proxy URL when caching is enabled."""
if not cover_url:
return cover_url
# Skip if already a local URL (starts with /)
if cover_url.startswith('/'):
if cover_url.startswith("/"):
return cover_url
# Check if cover caching is enabled
from shelfmark.config.env import is_covers_cache_enabled
if not is_covers_cache_enabled():
return cover_url
@@ -289,7 +301,26 @@ def transform_cover_url(cover_url: Optional[str], cache_id: str) -> Optional[str
# Encode the original URL and create a proxy URL
encoded_url = base64.urlsafe_b64encode(cover_url.encode()).decode()
base_path = normalize_base_path(app_config.get("URL_BASE", ""))
base_path = normalize_base_path(normalize_optional_text(app_config.get("URL_BASE", "")))
if base_path:
return f"{base_path}/api/covers/{cache_id}?url={encoded_url}"
return f"/api/covers/{cache_id}?url={encoded_url}"
def _coerce_config_path(value: object) -> Path | None:
if isinstance(value, os.PathLike):
path_value = os.fspath(value)
if isinstance(path_value, str):
normalized = path_value.strip()
if normalized:
return Path(normalized)
return None
if not isinstance(value, str):
return None
normalized = value.strip()
if not normalized:
return None
return Path(normalized)
+85 -52
View File
@@ -1,21 +1,43 @@
"""Archive extraction utilities for downloaded book archives."""
import os
import shutil
import zipfile
from pathlib import Path
from typing import List, Optional, Tuple
from typing import TYPE_CHECKING
from shelfmark.core.logger import setup_logger
from shelfmark.core.utils import is_audiobook as check_audiobook
from shelfmark.download.fs import atomic_write
from shelfmark.download.postprocess.policy import (
get_supported_audiobook_formats,
get_supported_formats,
)
from shelfmark.core.utils import is_audiobook as check_audiobook
from shelfmark.download.fs import atomic_write
logger = setup_logger(__name__)
if TYPE_CHECKING:
import rarfile
ArchiveType = zipfile.ZipFile | rarfile.RarFile
else:
ArchiveType = zipfile.ZipFile
def _delete_file_with_logging(file_path: Path, file_type_label: str, *, rejected: bool) -> None:
"""Delete a file and log the outcome."""
try:
file_path.unlink()
if rejected:
logger.debug("Deleted rejected %s file: %s", file_type_label, file_path.name)
else:
logger.debug("Deleted non-%s file: %s", file_type_label, file_path.name)
except OSError as e:
if rejected:
logger.warning(
"Failed to delete rejected %s file %s: %s", file_type_label, file_path, e
)
else:
logger.warning("Failed to delete non-%s file %s: %s", file_type_label, file_path, e)
# Check for rarfile availability at module load
try:
@@ -30,20 +52,14 @@ except ImportError:
class ArchiveExtractionError(Exception):
"""Raised when archive extraction fails."""
pass
class PasswordProtectedError(ArchiveExtractionError):
"""Raised when archive requires a password."""
pass
class CorruptedArchiveError(ArchiveExtractionError):
"""Raised when archive is corrupted."""
pass
def is_archive(file_path: Path) -> bool:
"""Check if file is a supported archive format."""
@@ -51,7 +67,7 @@ def is_archive(file_path: Path) -> bool:
return suffix in ("zip", "rar")
def _is_supported_file(file_path: Path, content_type: Optional[str] = None) -> bool:
def _is_supported_file(file_path: Path, content_type: str | None = None) -> bool:
"""Check if file matches user's supported formats setting based on content type."""
ext = file_path.suffix.lower().lstrip(".")
if check_audiobook(content_type):
@@ -62,16 +78,30 @@ def _is_supported_file(file_path: Path, content_type: Optional[str] = None) -> b
# All known ebook extensions (superset of what user might enable)
ALL_EBOOK_EXTENSIONS = {'.pdf', '.epub', '.mobi', '.azw', '.azw3', '.fb2', '.djvu', '.cbz', '.cbr', '.doc', '.docx', '.rtf', '.txt'}
ALL_EBOOK_EXTENSIONS = {
".pdf",
".epub",
".mobi",
".azw",
".azw3",
".fb2",
".djvu",
".cbz",
".cbr",
".doc",
".docx",
".rtf",
".txt",
}
# All known audio extensions (superset of what user might enable for audiobooks)
ALL_AUDIO_EXTENSIONS = {'.m4b', '.mp3', '.m4a', '.aac', '.flac', '.ogg', '.wma', '.wav', '.opus'}
ALL_AUDIO_EXTENSIONS = {".m4b", ".mp3", ".m4a", ".aac", ".flac", ".ogg", ".wma", ".wav", ".opus"}
def _filter_files(
extracted_files: List[Path],
content_type: Optional[str] = None,
) -> Tuple[List[Path], List[Path], List[Path]]:
extracted_files: list[Path],
content_type: str | None = None,
) -> tuple[list[Path], list[Path], list[Path]]:
"""Filter files by content type. Returns (matched, rejected_format, other)."""
is_audiobook = check_audiobook(content_type)
known_extensions = ALL_AUDIO_EXTENSIONS if is_audiobook else ALL_EBOOK_EXTENSIONS
@@ -94,8 +124,8 @@ def _filter_files(
def extract_archive(
archive_path: Path,
output_dir: Path,
content_type: Optional[str] = None,
) -> Tuple[List[Path], List[str], List[Path]]:
content_type: str | None = None,
) -> tuple[list[Path], list[str], list[Path]]:
"""Extract archive and filter by content type. Returns (matched, warnings, rejected)."""
suffix = archive_path.suffix.lower().lstrip(".")
@@ -104,7 +134,8 @@ def extract_archive(
elif suffix == "rar":
extracted_files, warnings = _extract_rar(archive_path, output_dir)
else:
raise ArchiveExtractionError(f"Unsupported archive format: {suffix}")
msg = f"Unsupported archive format: {suffix}"
raise ArchiveExtractionError(msg)
is_audiobook = check_audiobook(content_type)
file_type_label = "audiobook" if is_audiobook else "book"
@@ -114,23 +145,17 @@ def extract_archive(
# Delete rejected files (valid formats but not enabled by user)
for rejected_file in rejected_files:
try:
rejected_file.unlink()
logger.debug(f"Deleted rejected {file_type_label} file: {rejected_file.name}")
except OSError as e:
logger.warning(f"Failed to delete rejected {file_type_label} file {rejected_file}: {e}")
_delete_file_with_logging(rejected_file, file_type_label, rejected=True)
if rejected_files:
rejected_exts = sorted(set(f.suffix.lower() for f in rejected_files))
warnings.append(f"Skipped {len(rejected_files)} {file_type_label}(s) with unsupported format: {', '.join(rejected_exts)}")
rejected_exts = sorted({f.suffix.lower() for f in rejected_files})
warnings.append(
f"Skipped {len(rejected_files)} {file_type_label}(s) with unsupported format: {', '.join(rejected_exts)}"
)
# Delete other files (images, html, etc)
for other_file in other_files:
try:
other_file.unlink()
logger.debug(f"Deleted non-{file_type_label} file: {other_file.name}")
except OSError as e:
logger.warning(f"Failed to delete non-{file_type_label} file {other_file}: {e}")
_delete_file_with_logging(other_file, file_type_label, rejected=False)
if other_files:
warnings.append(f"Skipped {len(other_files)} non-{file_type_label} file(s)")
@@ -141,7 +166,7 @@ def extract_archive(
def extract_archive_raw(
archive_path: Path,
output_dir: Path,
) -> Tuple[List[Path], List[str]]:
) -> tuple[list[Path], list[str]]:
"""Extract archive without filtering (returns all extracted files)."""
suffix = archive_path.suffix.lower().lstrip(".")
@@ -150,10 +175,11 @@ def extract_archive_raw(
if suffix == "rar":
return _extract_rar(archive_path, output_dir)
raise ArchiveExtractionError(f"Unsupported archive format: {suffix}")
msg = f"Unsupported archive format: {suffix}"
raise ArchiveExtractionError(msg)
def _extract_files_from_archive(archive, output_dir: Path) -> List[Path]:
def _extract_files_from_archive(archive: ArchiveType, output_dir: Path) -> list[Path]:
"""Extract files from ZipFile or RarFile to output_dir with security checks."""
extracted_files = []
@@ -169,7 +195,7 @@ def _extract_files_from_archive(archive, output_dir: Path) -> List[Path]:
# Security: reject filenames with null bytes or path separators
# Check both / and \ since archives may be created on different OSes
if "\x00" in filename or "/" in filename or "\\" in filename:
logger.warning(f"Skipping suspicious filename in archive: {info.filename!r}")
logger.warning("Skipping suspicious filename in archive: %r", info.filename)
continue
# Extract to output_dir with flat structure
@@ -179,50 +205,56 @@ def _extract_files_from_archive(archive, output_dir: Path) -> List[Path]:
try:
target_path.resolve().relative_to(output_dir.resolve())
except ValueError:
logger.warning(f"Path traversal attempt blocked: {info.filename!r}")
logger.warning("Path traversal attempt blocked: %r", info.filename)
continue
with archive.open(info) as src:
data = src.read()
final_path = atomic_write(target_path, data)
extracted_files.append(final_path)
logger.debug(f"Extracted: {filename}")
logger.debug("Extracted: %s", filename)
return extracted_files
def _extract_zip(archive_path: Path, output_dir: Path) -> Tuple[List[Path], List[str]]:
def _extract_zip(archive_path: Path, output_dir: Path) -> tuple[list[Path], list[str]]:
"""Extract files from a ZIP archive."""
try:
with zipfile.ZipFile(archive_path, "r") as zf:
# Check for password protection
for info in zf.infolist():
if info.flag_bits & 0x1: # Encrypted flag
raise PasswordProtectedError("ZIP archive is password protected")
msg = "ZIP archive is password protected"
raise PasswordProtectedError(msg)
# Test archive integrity
bad_file = zf.testzip()
if bad_file:
raise CorruptedArchiveError(f"Corrupted file in archive: {bad_file}")
msg = f"Corrupted file in archive: {bad_file}"
raise CorruptedArchiveError(msg)
return _extract_files_from_archive(zf, output_dir), []
except zipfile.BadZipFile as e:
raise CorruptedArchiveError(f"Invalid or corrupted ZIP: {e}")
msg = f"Invalid or corrupted ZIP: {e}"
raise CorruptedArchiveError(msg) from e
except PermissionError as e:
raise ArchiveExtractionError(f"Permission denied: {e}")
msg = f"Permission denied: {e}"
raise ArchiveExtractionError(msg) from e
def _extract_rar(archive_path: Path, output_dir: Path) -> Tuple[List[Path], List[str]]:
def _extract_rar(archive_path: Path, output_dir: Path) -> tuple[list[Path], list[str]]:
"""Extract files from a RAR archive."""
if not RAR_AVAILABLE:
raise ArchiveExtractionError("RAR extraction not available - rarfile library not installed")
msg = "RAR extraction not available - rarfile library not installed"
raise ArchiveExtractionError(msg)
try:
with rarfile.RarFile(archive_path, "r") as rf:
# Check for password protection
if rf.needs_password():
raise PasswordProtectedError("RAR archive is password protected")
msg = "RAR archive is password protected"
raise PasswordProtectedError(msg)
# Test archive integrity
rf.testrar()
@@ -230,10 +262,11 @@ def _extract_rar(archive_path: Path, output_dir: Path) -> Tuple[List[Path], List
return _extract_files_from_archive(rf, output_dir), []
except rarfile.BadRarFile as e:
raise CorruptedArchiveError(f"Invalid or corrupted RAR: {e}")
except rarfile.RarCannotExec:
raise ArchiveExtractionError("unrar binary not found - install unrar package")
msg = f"Invalid or corrupted RAR: {e}"
raise CorruptedArchiveError(msg) from e
except rarfile.RarCannotExec as e:
msg = "unrar binary not found - install unrar package"
raise ArchiveExtractionError(msg) from e
except PermissionError as e:
raise ArchiveExtractionError(f"Permission denied: {e}")
msg = f"Permission denied: {e}"
raise ArchiveExtractionError(msg) from e
+125 -91
View File
@@ -1,5 +1,4 @@
"""
Shared download client infrastructure for external release sources.
"""Shared download client infrastructure for external release sources.
This module provides:
- DownloadState: Enum of valid download states
@@ -18,14 +17,19 @@ from abc import ABC, abstractmethod
from dataclasses import dataclass
from enum import Enum
from functools import wraps
from typing import Callable, Dict, List, Optional, Tuple, Type, TypeVar, Union, cast, Any
from importlib import import_module
from pathlib import Path
from typing import TYPE_CHECKING, TypeVar, cast
import requests
if TYPE_CHECKING:
from collections.abc import Callable
_logger = logging.getLogger(__name__)
# Type variable for generic return type
T = TypeVar('T')
T = TypeVar("T")
# Exceptions that should trigger a retry
RETRYABLE_EXCEPTIONS = (
@@ -33,6 +37,10 @@ RETRYABLE_EXCEPTIONS = (
requests.exceptions.Timeout,
requests.exceptions.HTTPError,
)
_MIN_RETRYABLE_STATUS = 500
_MIN_PROGRESS_PERCENT = 0
_MAX_PROGRESS_PERCENT = 100
_RNG = random.SystemRandom()
def with_retry(
@@ -41,8 +49,7 @@ def with_retry(
max_delay: float = 10.0,
jitter: float = 0.5,
) -> Callable[[Callable[..., T]], Callable[..., T]]:
"""
Decorator for retrying API calls with exponential backoff.
"""Retry API calls with exponential backoff.
Args:
max_attempts: Maximum number of attempts (default 3)
@@ -58,10 +65,12 @@ def with_retry(
Does NOT retry on:
- HTTP 4xx client errors (bad request, auth failures)
- Other exceptions (programming errors)
"""
def decorator(func: Callable[..., T]) -> Callable[..., T]:
@wraps(func)
def wrapper(*args, **kwargs) -> T:
def wrapper(*args: object, **kwargs: object) -> T:
last_exception = None
for attempt in range(1, max_attempts + 1):
@@ -69,7 +78,7 @@ def with_retry(
return func(*args, **kwargs)
except requests.exceptions.HTTPError as e:
# Only retry on server errors (5xx), not client errors (4xx)
if e.response is not None and e.response.status_code < 500:
if e.response is not None and e.response.status_code < _MIN_RETRYABLE_STATUS:
raise
last_exception = e
except RETRYABLE_EXCEPTIONS as e:
@@ -79,19 +88,25 @@ def with_retry(
# Calculate delay with exponential backoff
delay = min(base_delay * (2 ** (attempt - 1)), max_delay)
# Add jitter to prevent thundering herd
delay += random.uniform(0, delay * jitter)
delay += _RNG.uniform(0, delay * jitter)
_logger.debug(
f"Retry {attempt}/{max_attempts} for {func.__name__} "
f"after {delay:.1f}s (error: {last_exception})"
"Retry %s/%s for %s after %.1fs (error: %s)",
attempt,
max_attempts,
func.__name__,
delay,
last_exception,
)
time.sleep(delay)
# All retries exhausted
if last_exception is None:
raise RuntimeError("Retry failed without exception")
raise cast(Exception, last_exception)
msg = "Retry failed without exception"
raise RuntimeError(msg)
raise cast("Exception", last_exception)
return wrapper
return decorator
@@ -114,15 +129,15 @@ class DownloadStatus:
"""Status of an external download (immutable)."""
progress: float # 0-100
state: Union[DownloadState, str] # Prefer DownloadState enum; strings auto-normalized
message: Optional[str] # Status message
state: DownloadState | str # Prefer DownloadState enum; strings auto-normalized
message: str | None # Status message
complete: bool # True when download finished
file_path: Optional[str] # Path in client's download dir (when complete)
download_speed: Optional[int] = None # Bytes per second
eta: Optional[int] = None # Seconds remaining
file_path: str | None # Path in client's download dir (when complete)
download_speed: int | None = None # Bytes per second
eta: int | None = None # Seconds remaining
@classmethod
def error(cls, message: str) -> "DownloadStatus":
def error(cls, message: str) -> DownloadStatus:
"""Create an error status."""
return cls(
progress=0,
@@ -132,21 +147,25 @@ class DownloadStatus:
file_path=None,
)
def __post_init__(self):
def __post_init__(self) -> None:
"""Validate and normalize state."""
# Normalize string states to enum
if isinstance(self.state, str):
try:
normalized_state = DownloadState(self.state)
object.__setattr__(self, 'state', normalized_state)
object.__setattr__(self, "state", normalized_state)
except ValueError:
# Unknown state string - keep as-is for backwards compatibility
_logger.warning(f"Unknown download state '{self.state}', keeping as string")
_logger.warning(
_logger.warning("Unknown download state '%s', keeping as string", self.state)
)
# Validate progress is in range
if not 0 <= self.progress <= 100:
_logger.debug(f"Progress {self.progress} out of range, clamping to [0, 100]")
object.__setattr__(self, 'progress', max(0, min(100, self.progress)))
if not _MIN_PROGRESS_PERCENT <= self.progress <= _MAX_PROGRESS_PERCENT:
_logger.debug(
_logger.debug("Progress %s out of range, clamping to [0, 100]", self.progress)
)
object.__setattr__(self, "progress", max(0, min(100, self.progress)))
@property
def state_value(self) -> str:
@@ -157,8 +176,7 @@ class DownloadStatus:
class DownloadClient(ABC):
"""
Base class for external download clients.
"""Base class for external download clients.
Subclasses implement protocol-specific download management:
- Torrent clients: qBittorrent, Transmission, Deluge
@@ -174,8 +192,7 @@ class DownloadClient(ABC):
name: str
def _log_error(self, method: str, e: Exception, level: str = "error") -> str:
"""
Log a client error with consistent formatting.
"""Log a client error with consistent formatting.
Args:
method: Name of the method that failed (e.g., "get_status")
@@ -184,6 +201,7 @@ class DownloadClient(ABC):
Returns:
Formatted error message string (for use in DownloadStatus.error())
"""
error_type = type(e).__name__
msg = f"{self.name} {method} failed ({error_type}): {e}"
@@ -198,15 +216,15 @@ class DownloadClient(ABC):
return f"{error_type}: {e}"
def _build_path(self, *components: str) -> Optional[str]:
"""
Safely build a file path from components.
def _build_path(self, *components: str) -> str | None:
"""Safely build a file path from components.
Args:
*components: Path components to join (e.g., save_path, name)
Returns:
Normalized path string, or None if any component is empty/None.
"""
# Filter out empty/None components
valid = [c for c in components if c]
@@ -214,9 +232,9 @@ class DownloadClient(ABC):
return None
# Join and normalize
return os.path.normpath(os.path.join(*valid))
return os.path.normpath(str(Path(valid[0]).joinpath(*valid[1:])))
def __init_subclass__(cls, **kwargs):
def __init_subclass__(cls, **kwargs: object) -> None:
"""Validate that subclasses define required class attributes."""
super().__init_subclass__(**kwargs)
@@ -225,48 +243,46 @@ class DownloadClient(ABC):
return
# Validate protocol attribute
if not hasattr(cls, 'protocol') or not cls.protocol:
raise TypeError(f"{cls.__name__} must define 'protocol' class attribute")
if cls.protocol not in ('torrent', 'usenet'):
raise TypeError(
f"{cls.__name__}.protocol must be 'torrent' or 'usenet', got '{cls.protocol}'"
)
if not hasattr(cls, "protocol") or not cls.protocol:
msg = f"{cls.__name__} must define 'protocol' class attribute"
raise TypeError(msg)
if cls.protocol not in ("torrent", "usenet"):
msg = f"{cls.__name__}.protocol must be 'torrent' or 'usenet', got '{cls.protocol}'"
raise TypeError(msg)
# Validate name attribute
if not hasattr(cls, 'name') or not cls.name:
raise TypeError(f"{cls.__name__} must define 'name' class attribute")
if not hasattr(cls, "name") or not cls.name:
msg = f"{cls.__name__} must define 'name' class attribute"
raise TypeError(msg)
@staticmethod
@abstractmethod
def is_configured() -> bool:
"""
Check if this client is configured.
"""Check if this client is configured.
Returns:
True if required settings (URL, etc.) are present.
"""
pass
@abstractmethod
def test_connection(self) -> Tuple[bool, str]:
"""
Test connectivity to the client.
def test_connection(self) -> tuple[bool, str]:
"""Test connectivity to the client.
Returns:
Tuple of (success, message).
"""
pass
@abstractmethod
def add_download(
self,
url: str,
name: str,
category: Optional[str] = None,
expected_hash: Optional[str] = None,
**kwargs: Any,
category: str | None = None,
expected_hash: str | None = None,
**kwargs: object,
) -> str:
"""Add a download to the client.
Args:
@@ -274,32 +290,31 @@ class DownloadClient(ABC):
name: Display name for the download
category: Category/label for organization (None = client default)
expected_hash: Optional info_hash hint (torrents only)
**kwargs: Client-specific options passed through to the implementation.
Returns:
Client-specific download ID (hash for torrents, ID for NZBGet).
Raises:
Exception: If adding fails.
"""
pass
@abstractmethod
def get_status(self, download_id: str) -> DownloadStatus:
"""
Get status of a download.
"""Get status of a download.
Args:
download_id: The ID returned by add_download()
Returns:
Current download status.
"""
pass
@abstractmethod
def remove(self, download_id: str, delete_files: bool = False) -> bool:
"""
Remove a download from the client.
def remove(self, download_id: str, *, delete_files: bool = False) -> bool:
"""Remove a download from the client.
Args:
download_id: The ID returned by add_download()
@@ -307,27 +322,25 @@ class DownloadClient(ABC):
Returns:
True if removal succeeded.
"""
pass
@abstractmethod
def get_download_path(self, download_id: str) -> Optional[str]:
"""
Get the path where files were downloaded.
def get_download_path(self, download_id: str) -> str | None:
"""Get the path where files were downloaded.
Args:
download_id: The ID returned by add_download()
Returns:
File or directory path, or None if not available.
"""
pass
def find_existing(
self, url: str, category: Optional[str] = None
) -> Optional[Tuple[str, DownloadStatus]]:
"""
Check if a download for this URL already exists in the client.
self, url: str, category: str | None = None
) -> tuple[str, DownloadStatus] | None:
"""Check if a download for this URL already exists in the client.
This is useful for detecting already-completed downloads so we can
skip re-downloading and just copy the existing file.
@@ -339,17 +352,39 @@ class DownloadClient(ABC):
Returns:
Tuple of (download_id, status) if found, None if not found.
Default implementation returns None.
"""
return None
# Client registry: protocol -> list of client classes
_CLIENTS: Dict[str, List[Type[DownloadClient]]] = {}
_CLIENTS: dict[str, list[type[DownloadClient]]] = {}
_BUILTIN_CLIENT_MODULES = (
"shelfmark.download.clients.deluge",
"shelfmark.download.clients.nzbget",
"shelfmark.download.clients.qbittorrent",
"shelfmark.download.clients.rtorrent",
"shelfmark.download.clients.sabnzbd",
"shelfmark.download.clients.transmission",
)
_builtin_client_state = {"loaded": False}
def register_client(protocol: str):
"""
Decorator to register a download client for a protocol.
def _ensure_builtin_clients_registered() -> None:
"""Import built-in client modules once to populate the registry."""
if _builtin_client_state["loaded"]:
return
for module_name in _BUILTIN_CLIENT_MODULES:
import_module(module_name)
_builtin_client_state["loaded"] = True
def register_client(
protocol: str,
) -> Callable[[type[DownloadClient]], type[DownloadClient]]:
"""Register a download client for a protocol.
Multiple clients can be registered for the same protocol.
The `is_configured()` method determines which one is active.
@@ -361,9 +396,10 @@ def register_client(protocol: str):
@register_client("torrent")
class QBittorrentClient(DownloadClient):
...
"""
def decorator(cls: Type[DownloadClient]) -> Type[DownloadClient]:
def decorator(cls: type[DownloadClient]) -> type[DownloadClient]:
if protocol not in _CLIENTS:
_CLIENTS[protocol] = []
_CLIENTS[protocol].append(cls)
@@ -372,9 +408,8 @@ def register_client(protocol: str):
return decorator
def get_client(protocol: str) -> Optional[DownloadClient]:
"""
Get a configured client instance for the given protocol.
def get_client(protocol: str) -> DownloadClient | None:
"""Get a configured client instance for the given protocol.
Iterates through all registered clients for the protocol and
returns the first one that is configured.
@@ -384,7 +419,10 @@ def get_client(protocol: str) -> Optional[DownloadClient]:
Returns:
Configured client instance, or None if not available/configured.
"""
_ensure_builtin_clients_registered()
if protocol not in _CLIENTS:
return None
@@ -395,13 +433,15 @@ def get_client(protocol: str) -> Optional[DownloadClient]:
return None
def list_configured_clients() -> List[str]:
"""
List protocols that have configured clients.
def list_configured_clients() -> list[str]:
"""List protocols that have configured clients.
Returns:
List of protocol names (e.g., ["torrent", "usenet"]).
"""
_ensure_builtin_clients_registered()
result = []
for protocol, client_classes in _CLIENTS.items():
for cls in client_classes:
@@ -411,21 +451,15 @@ def list_configured_clients() -> List[str]:
return result
def get_all_clients() -> Dict[str, List[Type[DownloadClient]]]:
"""
Get all registered client classes.
def get_all_clients() -> dict[str, list[type[DownloadClient]]]:
"""Get all registered client classes.
Returns:
Dict of protocol -> list of client classes.
"""
_ensure_builtin_clients_registered()
return dict(_CLIENTS)
# Import client implementations to trigger registration
# These imports are at the bottom to avoid circular imports
from shelfmark.download.clients import qbittorrent # noqa: F401, E402
from shelfmark.download.clients import nzbget # noqa: F401, E402
from shelfmark.download.clients import sabnzbd # noqa: F401, E402
from shelfmark.download.clients import transmission # noqa: F401, E402
from shelfmark.download.clients import deluge # noqa: F401, E402
from shelfmark.download.clients import rtorrent # noqa: F401, E402
_ensure_builtin_clients_registered()
+45
View File
@@ -0,0 +1,45 @@
"""Shared coercion helpers for download client config and option values."""
from shelfmark.core.utils import normalize_http_url
def config_text(value: object, default: str = "") -> str:
"""Coerce config values to strings without losing explicit empty defaults."""
if value is None:
return default
if isinstance(value, str):
return value
return str(value)
def normalize_http_config_url(value: object, *, require_string: bool = False) -> str:
"""Normalize HTTP(S) config URLs with optional strict string-only input handling."""
if require_string and not isinstance(value, str):
return ""
return normalize_http_url(config_text(value))
def coerce_optional_int(value: object) -> int | None:
"""Convert optional numeric inputs to ints."""
if value is None:
return None
if isinstance(value, int):
return value
if isinstance(value, float):
return int(value)
if isinstance(value, str):
return int(value)
msg = f"Expected int-compatible value, got {type(value).__name__}"
raise TypeError(msg)
def coerce_optional_float(value: object) -> float | None:
"""Convert optional numeric inputs to floats."""
if value is None:
return None
if isinstance(value, (int, float)):
return float(value)
if isinstance(value, str):
return float(value)
msg = f"Expected float-compatible value, got {type(value).__name__}"
raise TypeError(msg)
+285 -140
View File
@@ -1,30 +1,57 @@
"""Shared download handler for external torrent/usenet clients."""
from __future__ import annotations
import errno
import shutil
import time
from abc import ABC, abstractmethod
from dataclasses import dataclass
from pathlib import Path
from threading import Event
from typing import Callable, Optional
from typing import TYPE_CHECKING, Protocol, TypeGuard
from shelfmark.core.config import config
from shelfmark.core.logger import setup_logger
from shelfmark.core.models import DownloadTask
from shelfmark.core.request_helpers import normalize_optional_text
from shelfmark.core.utils import is_audiobook
from shelfmark.download.clients import (
DownloadClient,
DownloadState,
DownloadStatus,
get_client,
list_configured_clients,
)
from shelfmark.download.fs import run_blocking_io
from shelfmark.download.permissions_debug import log_path_permission_context
from shelfmark.release_sources import DownloadHandler
if TYPE_CHECKING:
from collections.abc import Callable
from threading import Event
from shelfmark.core.models import DownloadTask
logger = setup_logger(__name__)
_CLIENT_CLEANUP_ERRORS = (AttributeError, KeyError, OSError, RuntimeError, TypeError, ValueError)
class _SabnzbdLikeClient(Protocol):
name: str
def remove(
self, download_id: str, *, delete_files: bool = False, archive: bool = True
) -> bool: ...
def _is_sabnzbd_like_client(candidate: DownloadClient) -> TypeGuard[_SabnzbdLikeClient]:
return getattr(candidate, "name", "") == "sabnzbd"
# How often to poll the download client for status (seconds)
POLL_INTERVAL = 2
WINDOWS_DRIVE_PREFIX_LENGTH = 2
SECONDS_PER_MINUTE = 60
SECONDS_PER_HOUR = 3600
# How long to wait for completed files to appear (seconds)
COMPLETED_PATH_RETRY_INTERVAL = 5
COMPLETED_PATH_MAX_ATTEMPTS = 12 # 12 attempts * 5s = 60s grace period
@@ -37,21 +64,23 @@ class DownloadRequest:
url: str
protocol: str
release_name: str
expected_hash: Optional[str]
expected_hash: str | None
seeding_time_limit: int | None = None # minutes
ratio_limit: float | None = None
def _diagnose_path_issue(path: str) -> str:
"""
Analyze a path and return diagnostic hints for common issues.
"""Analyze a path and return diagnostic hints for common issues.
Args:
path: The path that failed to be accessed
Returns:
A hint string to help users diagnose the issue.
"""
# Detect Windows-style paths (won't work in Linux containers)
if len(path) >= 2 and path[1] == ':':
if len(path) >= WINDOWS_DRIVE_PREFIX_LENGTH and path[1] == ":":
return (
f"Path '{path}' appears to be a Windows path. "
f"Shelfmark runs in Linux and cannot access Windows paths directly. "
@@ -74,10 +103,34 @@ def _diagnose_path_issue(path: str) -> str:
)
def _format_probe_error(error: OSError | None) -> str:
"""Render an OSError for inclusion in log/status messages."""
if error is None:
return "none"
code = errno.errorcode.get(error.errno, str(error.errno)) if error.errno else "?"
return f"{code}: {error.strerror or error}"
def _probe_completed_path(path: Path) -> tuple[bool, OSError | None]:
"""Probe a completed download path and preserve the underlying stat error.
`Path.exists()` silently converts every `OSError` to `False`, which hides
whether a failure is ENOENT (not yet written), EACCES (permission denied),
ESTALE (NFS stale handle), or something else. Callers need the real errno
to decide whether the condition is retryable and to surface diagnostics.
"""
try:
run_blocking_io(path.stat)
except OSError as error:
return False, error
return True, None
class ExternalClientHandler(DownloadHandler, ABC):
"""Shared lifecycle handler for sources that hand off to torrent/usenet clients."""
def __init__(self):
def __init__(self) -> None:
"""Initialize cleanup tracking for client-managed downloads."""
# Track downloads that may need client-side cleanup after Shelfmark completes import.
# task_id -> (client, download_id, protocol)
self._cleanup_refs: dict[str, tuple[DownloadClient, str, str]] = {}
@@ -86,15 +139,15 @@ class ExternalClientHandler(DownloadHandler, ABC):
def _resolve_download(
self,
task: DownloadTask,
status_callback: Callable[[str, Optional[str]], None],
) -> Optional[DownloadRequest]:
status_callback: Callable[[str, str | None], None],
) -> DownloadRequest | None:
"""Resolve source-specific task metadata into a client download request."""
def _on_download_complete(self, task: DownloadTask) -> None:
"""Hook called after successful completion; override for source cleanup."""
"""Run post-completion source cleanup hooks."""
return
def _get_client(self, protocol: str) -> Optional[DownloadClient]:
def _get_client(self, protocol: str) -> DownloadClient | None:
"""Resolve the active client for a protocol."""
return get_client(protocol)
@@ -103,7 +156,7 @@ class ExternalClientHandler(DownloadHandler, ABC):
return list_configured_clients()
def _poll_interval(self) -> float:
"""Polling interval for status checks (seconds)."""
"""Return the polling interval for status checks."""
return POLL_INTERVAL
def _completed_path_retry_interval(self) -> float:
@@ -114,7 +167,7 @@ class ExternalClientHandler(DownloadHandler, ABC):
"""Maximum attempts when waiting for completed files."""
return COMPLETED_PATH_MAX_ATTEMPTS
def _get_category_for_task(self, client: DownloadClient, task: DownloadTask) -> Optional[str]:
def _get_category_for_task(self, client: DownloadClient, task: DownloadTask) -> str | None:
"""Get audiobook category if configured and applicable, else None for default."""
if not is_audiobook(task.content_type):
return None
@@ -128,9 +181,19 @@ class ExternalClientHandler(DownloadHandler, ABC):
"sabnzbd": "SABNZBD_CATEGORY_AUDIOBOOK",
}
audiobook_key = audiobook_keys.get(client.name)
return config.get(audiobook_key, "") or None if audiobook_key else None
if audiobook_key is None:
return None
configured_category = config.get(audiobook_key, "")
normalized_category = normalize_optional_text(configured_category)
if normalized_category is not None:
return normalized_category
if configured_category is None:
return None
fallback_category = str(configured_category).strip()
return fallback_category or None
def post_process_cleanup(self, task: DownloadTask, success: bool) -> None:
def post_process_cleanup(self, task: DownloadTask, *, success: bool) -> None:
"""Clean up external-client state after post-processing finishes."""
if not success:
self._cleanup_refs.pop(task.task_id, None)
return
@@ -140,20 +203,34 @@ class ExternalClientHandler(DownloadHandler, ABC):
return
client, download_id, protocol = client_ref
if protocol != "usenet":
return
# "Move" means copy into ingest then let the usenet client delete its own files.
if config.get("PROWLARR_USENET_ACTION", "move") != "move":
return
if protocol == "usenet":
# "Move" means copy into ingest then let the usenet client delete its own files.
if config.get("PROWLARR_USENET_ACTION", "move") != "move":
return
try:
self._delete_local_download_data(client, download_id)
self._remove_usenet_download(client, download_id, delete_files=True, archive=True)
except _CLIENT_CLEANUP_ERRORS as e:
logger.warning(
"Failed to cleanup usenet download %s in %s: %s",
download_id,
getattr(client, "name", "client"),
e,
)
try:
self._delete_local_download_data(client, download_id)
self._remove_usenet_download(client, download_id, delete_files=True, archive=True)
except Exception as e:
logger.warning(
f"Failed to cleanup usenet download {download_id} in {getattr(client, 'name', 'client')}: {e}"
)
elif protocol == "torrent":
if config.get("PROWLARR_TORRENT_ACTION", "keep") != "remove":
return
try:
client.remove(download_id, delete_files=False)
except _CLIENT_CLEANUP_ERRORS as e:
logger.warning(
"Failed to remove torrent %s from %s: %s",
download_id,
getattr(client, "name", "client"),
e,
)
def _remove_usenet_download(
self,
@@ -164,7 +241,7 @@ class ExternalClientHandler(DownloadHandler, ABC):
archive: bool = True,
) -> None:
"""Remove a usenet download with SABnzbd-specific archive handling."""
if getattr(client, "name", "") == "sabnzbd":
if _is_sabnzbd_like_client(client):
client.remove(download_id, delete_files=delete_files, archive=archive)
else:
client.remove(download_id, delete_files=delete_files)
@@ -173,12 +250,14 @@ class ExternalClientHandler(DownloadHandler, ABC):
"""Best-effort local deletion of client download data."""
try:
raw_path = client.get_download_path(download_id)
except Exception as e:
logger.debug(f"Failed to resolve download path for {client.name} {download_id}: {e}")
except _CLIENT_CLEANUP_ERRORS as e:
logger.debug(
"Failed to resolve download path for %s %s: %s", client.name, download_id, e
)
return
if not raw_path:
logger.debug(f"No download path available for {client.name} {download_id}")
logger.debug("No download path available for %s %s", client.name, download_id)
return
from shelfmark.core.path_mappings import (
@@ -200,11 +279,16 @@ class ExternalClientHandler(DownloadHandler, ABC):
delete_path = remapped if matched_mapping else source_path_obj
if str(delete_path) in ("", "/"):
logger.warning(f"Refusing to delete unsafe path for {client.name} {download_id}: {delete_path}")
logger.warning(
"Refusing to delete unsafe path for %s %s: %s",
client.name,
download_id,
delete_path,
)
return
if not run_blocking_io(delete_path.exists):
logger.debug(f"Local download path does not exist for cleanup: {delete_path}")
logger.debug("Local download path does not exist for cleanup: %s", delete_path)
return
try:
@@ -212,18 +296,27 @@ class ExternalClientHandler(DownloadHandler, ABC):
run_blocking_io(shutil.rmtree, delete_path)
else:
run_blocking_io(delete_path.unlink)
logger.info(f"Deleted local download data for {client.name} {download_id}: {delete_path}")
except Exception as e:
logger.warning(f"Failed to delete local download data for {client.name} {download_id}: {e}")
logger.info(
"Deleted local download data for %s %s: %s", client.name, download_id, delete_path
)
except _CLIENT_CLEANUP_ERRORS as e:
logger.warning(
"Failed to delete local download data for %s %s: %s", client.name, download_id, e
)
def _safe_remove_download(self, client, download_id: str, protocol: str, reason: str) -> None:
def _safe_remove_download(
self,
client: DownloadClient,
download_id: str,
protocol: str,
reason: str,
) -> None:
"""Best-effort removal of a failed/cancelled download from the client.
Safety policy:
- torrents: never remove or delete client data (avoid breaking seeding)
- usenet: keep legacy behavior (delete client files on removal)
"""
if protocol != "usenet":
logger.info(
"Skipping download client cleanup for protocol=%s after %s (client=%s id=%s)",
@@ -238,9 +331,13 @@ class ExternalClientHandler(DownloadHandler, ABC):
# Permanent delete for failed usenet downloads (SABnzbd archive=0).
self._delete_local_download_data(client, download_id)
self._remove_usenet_download(client, download_id, delete_files=True, archive=False)
except Exception as e:
except _CLIENT_CLEANUP_ERRORS as e:
logger.warning(
f"Failed to remove download {download_id} from {client.name} after {reason}: {e}"
"Failed to remove download %s from %s after %s: %s",
download_id,
client.name,
reason,
e,
)
def _handle_cancelled_download(
@@ -248,20 +345,26 @@ class ExternalClientHandler(DownloadHandler, ABC):
client: DownloadClient,
download_id: str,
protocol: str,
status_callback: Callable[[str, Optional[str]], None],
status_callback: Callable[[str, str | None], None],
) -> None:
if protocol == "usenet":
logger.info(f"Download cancelled, removing from {client.name}: {download_id}")
logger.info("Download cancelled, removing from %s: %s", client.name, download_id)
try:
self._delete_local_download_data(client, download_id)
self._remove_usenet_download(client, download_id, delete_files=True, archive=True)
except Exception as e:
except _CLIENT_CLEANUP_ERRORS as e:
logger.warning(
f"Failed to remove download {download_id} from {client.name} after cancellation: {e}"
"Failed to remove download %s from %s after cancellation: %s",
download_id,
client.name,
e,
)
else:
logger.info(
f"Download cancelled for protocol={protocol}; leaving in {client.name}: {download_id}"
"Download cancelled for protocol=%s; leaving in %s: %s",
protocol,
client.name,
download_id,
)
status_callback("cancelled", "Cancelled")
@@ -271,7 +374,7 @@ class ExternalClientHandler(DownloadHandler, ABC):
download_id: str,
*,
log_details: bool,
) -> tuple[Optional[Path], Optional[str]]:
) -> tuple[Path | None, str | None]:
"""Resolve and validate the completed download path once."""
try:
raw_path = client.get_download_path(download_id)
@@ -281,12 +384,12 @@ class ExternalClientHandler(DownloadHandler, ABC):
f"Check volume mappings and category settings."
)
if log_details:
logger.error(
f"Failed to resolve download path for {client.name} {download_id}: {e}"
logger.exception(
"Failed to resolve download path for %s %s", client.name, download_id
)
else:
logger.debug(
f"Failed to resolve download path for {client.name} {download_id}: {e}"
"Failed to resolve download path for %s %s: %s", client.name, download_id, e
)
return None, message
@@ -296,9 +399,13 @@ class ExternalClientHandler(DownloadHandler, ABC):
f"Check volume mappings and category settings."
)
if log_details:
logger.error(f"Download client returned empty path for {client.name} {download_id}")
logger.error(
"Download client returned empty path for %s %s", client.name, download_id
)
else:
logger.debug(f"Download client returned empty path for {client.name} {download_id}")
logger.debug(
"Download client returned empty path for %s %s", client.name, download_id
)
return None, message
from shelfmark.core.path_mappings import (
@@ -327,19 +434,21 @@ class ExternalClientHandler(DownloadHandler, ABC):
remote_path=source_path_obj,
)
if log_details:
remapped_exists = run_blocking_io(remapped.exists)
logger.debug(
"Remap result: %s -> %s (exists=%s, changed=%s, matched=%s)",
source_path_obj,
remapped,
remapped_exists,
remapped != source_path_obj,
matched_mapping,
)
if matched_mapping:
if run_blocking_io(remapped.exists):
remapped_exists, remapped_error = _probe_completed_path(remapped)
if log_details:
logger.debug(
"Remap result: %s -> %s (exists=%s, probe_error=%s, changed=%s, matched=%s)",
source_path_obj,
remapped,
remapped_exists,
_format_probe_error(remapped_error),
remapped != source_path_obj,
matched_mapping,
)
if remapped_exists:
logger.info(
"Remapped download path for %s (%s): %s -> %s",
client.name,
@@ -353,69 +462,87 @@ class ExternalClientHandler(DownloadHandler, ABC):
f"Remapped path '{remapped}' does not exist. "
f"Check your Docker volume mounts match the Local Path in Settings > Advanced > Remote Path Mappings."
)
failure_log = "Download path does not exist after remapping: %s -> %s (probe_error=%s). Client: %s, ID: %s."
failure_args = (
raw_path,
remapped,
_format_probe_error(remapped_error),
client.name,
download_id,
)
if log_details:
logger.error(
f"Download path does not exist after remapping: {raw_path} -> {remapped}. "
f"Client: {client.name}, ID: {download_id}."
)
log_path_permission_context("completed_download_remap", remapped)
logger.error(failure_log, *failure_args)
else:
logger.debug(
f"Download path does not exist after remapping: {raw_path} -> {remapped}. "
f"Client: {client.name}, ID: {download_id}."
)
logger.debug(failure_log, *failure_args)
return None, message
if mappings:
if run_blocking_io(source_path_obj.exists):
source_exists, source_error = _probe_completed_path(source_path_obj)
if log_details:
logger.debug(
"Remap result: %s -> %s (exists=%s, probe_error=%s, changed=%s, matched=%s)",
source_path_obj,
remapped,
source_exists,
_format_probe_error(source_error),
remapped != source_path_obj,
matched_mapping,
)
if source_exists:
if mappings:
logger.info(
"No remote path mapping matched for %s (%s); using client path: %s",
client.name,
download_id,
source_path_obj,
)
return source_path_obj, None
return source_path_obj, None
hint = _diagnose_path_issue(raw_path)
hint = _diagnose_path_issue(raw_path)
if mappings:
message = f"{hint} No remote path mapping matched for client '{client.name}'."
if log_details:
logger.error(
f"Download path does not exist and no remote path mapping matched for {client.name} "
f"({download_id}): {raw_path}. {hint}"
)
else:
logger.debug(
f"Download path does not exist and no remote path mapping matched for {client.name} "
f"({download_id}): {raw_path}. {hint}"
)
return None, message
if not run_blocking_io(source_path_obj.exists):
hint = _diagnose_path_issue(raw_path)
failure_label = "completed_download_original"
failure_log = "Download path does not exist and no remote path mapping matched for %s (%s): %s (probe_error=%s). %s"
failure_args = (
client.name,
download_id,
raw_path,
_format_probe_error(source_error),
hint,
)
else:
message = hint
if log_details:
logger.error(
f"Download path does not exist: {raw_path}. "
f"Client: {client.name}, ID: {download_id}. {hint}"
)
else:
logger.debug(
f"Download path does not exist: {raw_path}. "
f"Client: {client.name}, ID: {download_id}. {hint}"
)
return None, message
failure_label = "completed_download_direct"
failure_log = (
"Download path does not exist: %s (probe_error=%s). Client: %s, ID: %s. %s"
)
failure_args = (
raw_path,
_format_probe_error(source_error),
client.name,
download_id,
hint,
)
return source_path_obj, None
if log_details:
log_path_permission_context(failure_label, source_path_obj)
logger.error(failure_log, *failure_args)
else:
logger.debug(failure_log, *failure_args)
return None, message
def _wait_for_completed_path(
self,
client: DownloadClient,
download_id: str,
*,
cancel_flag: Optional[Event],
status_callback: Callable[[str, Optional[str]], None],
) -> tuple[Optional[Path], Optional[str]]:
cancel_flag: Event | None,
status_callback: Callable[[str, str | None], None],
) -> tuple[Path | None, str | None]:
"""Wait briefly for completed files to appear on disk."""
last_error: Optional[str] = None
last_error: str | None = None
max_attempts = self._completed_path_max_attempts()
retry_interval = self._completed_path_retry_interval()
@@ -452,7 +579,7 @@ class ExternalClientHandler(DownloadHandler, ABC):
return None, last_error
def _build_progress_message(self, status) -> str:
def _build_progress_message(self, status: DownloadStatus) -> str:
"""Build a progress message from download status."""
msg = f"{status.progress:.0f}%"
@@ -461,12 +588,15 @@ class ExternalClientHandler(DownloadHandler, ABC):
msg += f" ({speed_mb:.1f} MB/s)"
if status.eta and status.eta > 0:
if status.eta < 60:
if status.eta < SECONDS_PER_MINUTE:
msg += f" - {status.eta}s left"
elif status.eta < 3600:
msg += f" - {status.eta // 60}m left"
elif status.eta < SECONDS_PER_HOUR:
msg += f" - {status.eta // SECONDS_PER_MINUTE}m left"
else:
msg += f" - {status.eta // 3600}h {(status.eta % 3600) // 60}m left"
msg += (
f" - {status.eta // SECONDS_PER_HOUR}h "
f"{(status.eta % SECONDS_PER_HOUR) // SECONDS_PER_MINUTE}m left"
)
return msg
@@ -475,8 +605,8 @@ class ExternalClientHandler(DownloadHandler, ABC):
task: DownloadTask,
cancel_flag: Event,
progress_callback: Callable[[float], None],
status_callback: Callable[[str, Optional[str]], None],
) -> Optional[str]:
status_callback: Callable[[str, str | None], None],
) -> str | None:
"""Execute download via configured torrent/usenet client. Returns file path or None."""
try:
if cancel_flag.is_set():
@@ -506,7 +636,7 @@ class ExternalClientHandler(DownloadHandler, ABC):
if existing:
download_id, existing_status = existing
logger.info(f"Found existing download in {client.name}: {download_id}")
logger.info("Found existing download in %s: %s", client.name, download_id)
# If already complete, skip straight to file handling
if existing_status.complete:
@@ -553,13 +683,17 @@ class ExternalClientHandler(DownloadHandler, ABC):
name=request.release_name,
category=category,
expected_hash=request.expected_hash,
seeding_time_limit=request.seeding_time_limit,
ratio_limit=request.ratio_limit,
)
except Exception as e:
logger.error(f"Failed to add to {client.name}: {e}")
logger.exception("Failed to add to %s", client.name)
status_callback("error", f"Failed to add to {client.name}: {e}")
return None
logger.info(f"Added to {client.name}: {download_id} for '{request.release_name}'")
logger.info(
"Added to %s: %s for '%s'", client.name, download_id, request.release_name
)
# Poll for progress
return self._poll_and_complete(
@@ -573,7 +707,7 @@ class ExternalClientHandler(DownloadHandler, ABC):
)
except Exception as e:
logger.error(f"External client download error: {e}")
logger.exception("External client download error")
status_callback("error", str(e))
return None
@@ -585,8 +719,8 @@ class ExternalClientHandler(DownloadHandler, ABC):
task: DownloadTask,
cancel_flag: Event,
progress_callback: Callable[[float], None],
status_callback: Callable[[str, Optional[str]], None],
) -> Optional[str]:
status_callback: Callable[[str, str | None], None],
) -> str | None:
"""Poll the download client for progress and handle completion."""
poll_interval = self._poll_interval()
# Track consecutive "not found" errors - torrents may take time to appear in client
@@ -594,7 +728,8 @@ class ExternalClientHandler(DownloadHandler, ABC):
max_not_found_retries = 15 # 15 retries * poll interval ~= 30s grace period
try:
logger.debug(f"Starting poll for {download_id} (content_type={task.content_type})")
result: str | None = None
logger.debug("Starting poll for %s (content_type=%s)", download_id, task.content_type)
while not cancel_flag.is_set():
status = client.get_status(download_id)
progress_callback(status.progress)
@@ -602,12 +737,18 @@ class ExternalClientHandler(DownloadHandler, ABC):
# Check for completion
if status.complete:
if status.state == DownloadState.ERROR:
logger.error(f"Download {download_id} completed with error: {status.message}")
logger.error(
"Download %s completed with error: %s", download_id, status.message
)
status_callback("error", status.message or "Download failed")
self._safe_remove_download(client, download_id, protocol, "completion error")
self._safe_remove_download(
client, download_id, protocol, "completion error"
)
return None
# Download complete - break to handle file
logger.debug(f"Download {download_id} complete, file_path={status.file_path}")
logger.debug(
"Download %s complete, file_path=%s", download_id, status.file_path
)
break
# Check for error state
@@ -641,8 +782,10 @@ class ExternalClientHandler(DownloadHandler, ABC):
not_found_count += 1
if not_found_count < max_not_found_retries:
logger.debug(
f"Download {download_id} not yet visible in client "
f"(attempt {not_found_count}/{max_not_found_retries})"
"Download %s not yet visible in client (attempt %s/%s)",
download_id,
not_found_count,
max_not_found_retries,
)
status_callback("resolving", "Waiting for download client...")
if cancel_flag.wait(timeout=poll_interval):
@@ -650,11 +793,13 @@ class ExternalClientHandler(DownloadHandler, ABC):
continue
logger.error(
f"Download {download_id} not found after {max_not_found_retries} attempts"
"Download %s not found after %s attempts",
download_id,
max_not_found_retries,
)
else:
# Fail fast on actionable errors (auth, connectivity, API issues)
logger.error(f"Download {download_id} error state: {status.message}")
logger.error("Download %s error state: %s", download_id, status.message)
status_callback("error", status.message or "Download failed")
self._safe_remove_download(client, download_id, protocol, "download error")
@@ -705,26 +850,26 @@ class ExternalClientHandler(DownloadHandler, ABC):
status_callback=status_callback,
)
# Clean up on success
if result:
self._on_download_complete(task)
self._cleanup_refs[task.task_id] = (client, download_id, protocol)
return result
except Exception as e:
logger.error(f"Error during download polling: {e}")
logger.exception("Error during download polling")
status_callback("error", str(e))
self._safe_remove_download(client, download_id, protocol, "polling exception")
return None
# Clean up on success
if result:
self._on_download_complete(task)
self._cleanup_refs[task.task_id] = (client, download_id, protocol)
return result
def _handle_completed_file(
self,
source_path: Path,
protocol: str,
task: DownloadTask,
status_callback: Callable[[str, Optional[str]], None],
) -> Optional[str]:
status_callback: Callable[[str, str | None], None],
) -> str | None:
"""Handle a completed download and return its path.
For external download clients (torrents/usenet), staging large payloads into TMP_DIR
@@ -739,15 +884,15 @@ class ExternalClientHandler(DownloadHandler, ABC):
if protocol == "torrent":
task.original_download_path = str(source_path)
logger.debug(f"Download complete, returning original path: {source_path}")
logger.debug("Download complete, returning original path: %s", source_path)
return str(source_path)
except Exception as e:
logger.error(f"Failed to finalize completed download at {source_path}: {e}")
logger.exception("Failed to finalize completed download at %s", source_path)
status_callback("error", f"Failed to finalize completed download: {e}")
return None
def cancel(self, task_id: str) -> bool:
"""Default cancellation (primary cancellation happens via cancel_flag)."""
logger.debug(f"Cancel requested for external client task: {task_id}")
logger.debug("Cancel requested for external client task: %s", task_id)
return True
+145 -75
View File
@@ -12,13 +12,13 @@ Requirements:
"""
import base64
from typing import Any, Optional, Tuple
from contextlib import suppress
from typing import Any, NoReturn
from urllib.parse import urlparse
import requests
from shelfmark.core.config import config
from shelfmark.download.network import get_ssl_verify
from shelfmark.core.logger import setup_logger
from shelfmark.core.utils import normalize_http_url
from shelfmark.download.clients import (
@@ -26,25 +26,54 @@ from shelfmark.download.clients import (
DownloadStatus,
register_client,
)
from shelfmark.download.clients._coercion import (
coerce_optional_float,
coerce_optional_int,
config_text,
)
from shelfmark.download.clients.torrent_utils import (
extract_torrent_info,
)
from shelfmark.download.network import get_ssl_verify
logger = setup_logger(__name__)
MIN_DAEMON_HOST_ENTRY_LENGTH = 2
MIN_DAEMON_HOST_STATUS_ENTRY_LENGTH = 4
DOWNLOAD_COMPLETE_PROGRESS = 100
ONE_WEEK_IN_SECONDS = 604800
class DelugeRpcError(RuntimeError):
def __init__(self, message: str, code: int | None = None):
"""Raised when Deluge returns a JSON-RPC error response."""
def __init__(self, message: str, code: int | None = None) -> None:
"""Initialize the RPC error with an optional Deluge error code."""
super().__init__(message)
self.code = code
def _get_error_message(error: Any) -> Tuple[str, int | None]:
_DELUGE_CLIENT_ERRORS = (
AttributeError,
DelugeRpcError,
OSError,
requests.exceptions.RequestException,
RuntimeError,
TypeError,
ValueError,
)
def _get_error_message(error: object) -> tuple[str, int | None]:
if isinstance(error, dict):
return str(error.get("message") or error), error.get("code")
return str(error), None
def _raise_runtime_error(message: str) -> NoReturn:
raise RuntimeError(message)
@register_client("torrent")
class DelugeClient(DownloadClient):
"""Deluge download client using Deluge Web UI JSON-RPC."""
@@ -52,15 +81,18 @@ class DelugeClient(DownloadClient):
protocol = "torrent"
name = "deluge"
def __init__(self):
raw_host = str(config.get("DELUGE_HOST", "localhost") or "")
raw_port = str(config.get("DELUGE_PORT", "8112") or "8112")
password = str(config.get("DELUGE_PASSWORD", "") or "")
def __init__(self) -> None:
"""Initialize the client from the configured Deluge connection settings."""
raw_host = config_text(config.get("DELUGE_HOST", "localhost"))
raw_port = config_text(config.get("DELUGE_PORT", "8112"), "8112")
password = config_text(config.get("DELUGE_PASSWORD", ""))
if not raw_host:
raise ValueError("DELUGE_HOST is required")
msg = "DELUGE_HOST is required"
raise ValueError(msg)
if not password:
raise ValueError("DELUGE_PASSWORD is required")
msg = "DELUGE_PASSWORD is required"
raise ValueError(msg)
scheme = "http"
base_path = ""
@@ -69,7 +101,8 @@ class DelugeClient(DownloadClient):
# (useful when Deluge is behind a reverse proxy path).
raw_host = normalize_http_url(raw_host, strip_trailing_slash=False) if raw_host else ""
if not raw_host:
raise ValueError("DELUGE_HOST is invalid")
msg = "DELUGE_HOST is invalid"
raise ValueError(msg)
host = raw_host
port = int(raw_port)
@@ -81,13 +114,12 @@ class DelugeClient(DownloadClient):
if parsed.port is not None:
port = parsed.port
base_path = (parsed.path or "").rstrip("/")
else:
# Allow "host:port" in DELUGE_HOST for convenience.
if ":" in raw_host and raw_host.count(":") == 1:
host_part, port_part = raw_host.split(":", 1)
if host_part and port_part.isdigit():
host = host_part
port = int(port_part)
# Allow "host:port" in DELUGE_HOST for convenience.
elif ":" in raw_host and raw_host.count(":") == 1:
host_part, port_part = raw_host.split(":", 1)
if host_part and port_part.isdigit():
host = host_part
port = int(port_part)
self._rpc_url = f"{scheme}://{host}:{port}{base_path}/json"
self._password = password
@@ -97,21 +129,26 @@ class DelugeClient(DownloadClient):
self._connected = False
self._rpc_id = 0
self._category = str(config.get("DELUGE_CATEGORY", "books") or "books")
self._download_dir = str(config.get("DELUGE_DOWNLOAD_DIR", "") or "")
self._category = config_text(config.get("DELUGE_CATEGORY", "books"), "books")
self._download_dir = config_text(config.get("DELUGE_DOWNLOAD_DIR", ""))
def _next_rpc_id(self) -> int:
self._rpc_id += 1
return self._rpc_id
def _rpc_call(self, method: str, *params: Any, timeout: int = 15) -> Any:
def _rpc_call(self, method: str, *params: object, timeout: int = 15) -> Any:
payload = {
"id": self._next_rpc_id(),
"method": method,
"params": list(params),
}
response = self._session.post(self._rpc_url, json=payload, timeout=timeout, verify=get_ssl_verify(self._rpc_url))
response = self._session.post(
self._rpc_url,
json=payload,
timeout=timeout,
verify=get_ssl_verify(self._rpc_url),
)
response.raise_for_status()
data = response.json()
@@ -124,20 +161,28 @@ class DelugeClient(DownloadClient):
def _login(self) -> None:
result = self._rpc_call("auth.login", self._password)
if result is not True:
raise DelugeRpcError("Deluge Web UI authentication failed")
msg = "Deluge Web UI authentication failed"
raise DelugeRpcError(msg)
self._authenticated = True
def _select_daemon_host_id(self, hosts: list) -> str:
# Hosts returned by web.get_hosts look like:
# [[host_id, host, port, status], ...]
# Deluge returns entries containing host id, host, port, and status.
preferred_hosts = {"127.0.0.1", "localhost"}
for entry in hosts:
if isinstance(entry, list) and len(entry) >= 2 and entry[1] in preferred_hosts:
if (
isinstance(entry, list)
and len(entry) >= MIN_DAEMON_HOST_ENTRY_LENGTH
and entry[1] in preferred_hosts
):
return str(entry[0])
for entry in hosts:
if isinstance(entry, list) and len(entry) >= 4 and str(entry[3]).lower() == "online":
if (
isinstance(entry, list)
and len(entry) >= MIN_DAEMON_HOST_STATUS_ENTRY_LENGTH
and str(entry[3]).lower() == "online"
):
return str(entry[0])
return str(hosts[0][0])
@@ -155,31 +200,30 @@ class DelugeClient(DownloadClient):
hosts = self._rpc_call("web.get_hosts") or []
if not hosts:
raise DelugeRpcError(
msg = (
"Deluge Web UI isn't connected to Deluge core (no hosts configured). "
"Add/connect a daemon in Deluge Web UI → Connection Manager."
)
raise DelugeRpcError(msg)
host_id = self._select_daemon_host_id(hosts)
self._rpc_call("web.connect", host_id)
if self._rpc_call("web.connected") is not True:
raise DelugeRpcError(
msg = (
"Deluge Web UI couldn't connect to Deluge core. "
"Check daemon status in Deluge Web UI → Connection Manager."
)
raise DelugeRpcError(msg)
self._connected = True
def _get_daemon_version(self) -> Any:
def _get_daemon_version(self) -> object:
"""Fetch daemon version, preferring daemon.get_version when available."""
try:
with suppress(*_DELUGE_CLIENT_ERRORS):
methods = self._rpc_call("system.listMethods")
if isinstance(methods, list) and "daemon.get_version" in methods:
return self._rpc_call("daemon.get_version")
except Exception:
# Fall back to daemon.info to preserve existing behavior.
pass
return self._rpc_call("daemon.info")
@@ -190,40 +234,42 @@ class DelugeClient(DownloadClient):
try:
# label.add will error if the plugin is unavailable or the label exists.
try:
with suppress(*_DELUGE_CLIENT_ERRORS):
self._rpc_call("label.add", label)
except Exception:
pass
self._rpc_call("label.set_torrent", torrent_id, label)
except Exception as e:
logger.debug(f"Could not set Deluge label '{label}' for {torrent_id}: {e}")
except _DELUGE_CLIENT_ERRORS as e:
logger.debug("Could not set Deluge label '%s' for %s: %s", label, torrent_id, e)
@staticmethod
def is_configured() -> bool:
client = config.get("PROWLARR_TORRENT_CLIENT", "")
host = config.get("DELUGE_HOST", "")
password = config.get("DELUGE_PASSWORD", "")
"""Return whether Deluge is the active configured torrent client."""
client = config_text(config.get("PROWLARR_TORRENT_CLIENT", ""))
host = config_text(config.get("DELUGE_HOST", ""))
password = config_text(config.get("DELUGE_PASSWORD", ""))
return client == "deluge" and bool(host) and bool(password)
def test_connection(self) -> Tuple[bool, str]:
def test_connection(self) -> tuple[bool, str]:
"""Test connectivity and authentication against the Deluge server."""
try:
self._ensure_connected()
version = self._get_daemon_version()
return True, f"Connected to Deluge {version}"
except Exception as e:
except _DELUGE_CLIENT_ERRORS as e:
self._authenticated = False
self._connected = False
return False, f"Connection failed: {str(e)}"
return False, f"Connection failed: {e!s}"
else:
return True, f"Connected to Deluge {version}"
def add_download(
self,
url: str,
name: str,
category: Optional[str] = None,
expected_hash: Optional[str] = None,
**kwargs,
category: str | None = None,
expected_hash: str | None = None,
**kwargs: object,
) -> str:
"""Add a torrent to Deluge and return the torrent id."""
try:
self._ensure_connected()
@@ -231,19 +277,28 @@ class DelugeClient(DownloadClient):
torrent_info = extract_torrent_info(url, expected_hash=expected_hash)
if not torrent_info.is_magnet and not torrent_info.torrent_data:
raise Exception("Failed to fetch torrent file")
_raise_runtime_error("Failed to fetch torrent file")
options: dict[str, Any] = {}
if self._download_dir:
options["download_location"] = self._download_dir
# Per-torrent seeding limits from indexer
seeding_time_limit = coerce_optional_int(kwargs.get("seeding_time_limit"))
if seeding_time_limit is not None:
options["seed_time_limit"] = seeding_time_limit
ratio_limit = coerce_optional_float(kwargs.get("ratio_limit"))
if ratio_limit is not None:
options["stop_at_ratio"] = ratio_limit
options["stop_at_ratio_enabled"] = True
if torrent_info.is_magnet:
magnet_url = torrent_info.magnet_url or url
torrent_id = self._rpc_call("core.add_torrent_magnet", magnet_url, options)
else:
torrent_data = torrent_info.torrent_data
if torrent_data is None:
raise Exception("Failed to fetch torrent file")
_raise_runtime_error("Failed to fetch torrent file")
torrent_data_bytes: bytes = torrent_data
filedump = base64.b64encode(torrent_data_bytes).decode("ascii")
@@ -255,28 +310,37 @@ class DelugeClient(DownloadClient):
)
if not torrent_id:
raise Exception("Deluge returned no torrent ID")
_raise_runtime_error("Deluge returned no torrent ID")
torrent_id = str(torrent_id).lower()
self._try_set_label(torrent_id, category_value)
logger.info(f"Added torrent to Deluge: {torrent_id}")
return torrent_id
logger.info("Added torrent to Deluge: %s", torrent_id)
except Exception as e:
except _DELUGE_CLIENT_ERRORS:
self._authenticated = False
self._connected = False
logger.error(f"Deluge add failed: {e}")
logger.exception("Deluge add failed")
raise
else:
return torrent_id
def get_status(self, download_id: str) -> DownloadStatus:
"""Return the current Deluge status for a torrent."""
try:
self._ensure_connected()
status = self._rpc_call(
"core.get_torrent_status",
download_id,
["state", "progress", "download_payload_rate", "eta", "save_path", "name"],
[
"state",
"progress",
"download_payload_rate",
"eta",
"save_path",
"name",
],
)
if not status:
@@ -299,7 +363,7 @@ class DelugeClient(DownloadClient):
progress = float(status.get("progress", 0))
# Don't mark complete while files are being moved
complete = progress >= 100 and deluge_state != "Moving"
complete = progress >= DOWNLOAD_COMPLETE_PROGRESS and deluge_state != "Moving"
if complete:
message = "Complete"
@@ -308,10 +372,10 @@ class DelugeClient(DownloadClient):
if eta is not None:
try:
eta = int(eta)
except Exception:
except TypeError, ValueError:
eta = None
if eta is not None and (eta < 0 or eta > 604800):
if eta is not None and (eta < 0 or eta > ONE_WEEK_IN_SECONDS):
eta = None
file_path = None
@@ -332,27 +396,31 @@ class DelugeClient(DownloadClient):
eta=eta,
)
except Exception as e:
except _DELUGE_CLIENT_ERRORS as e:
return DownloadStatus.error(self._log_error("get_status", e))
def remove(self, download_id: str, delete_files: bool = False) -> bool:
def remove(self, download_id: str, *, delete_files: bool = False) -> bool:
"""Remove a torrent from Deluge, optionally deleting its files."""
try:
self._ensure_connected()
result = self._rpc_call("core.remove_torrent", download_id, delete_files)
if result:
logger.info(
f"Removed torrent from Deluge: {download_id}"
+ (" (with files)" if delete_files else "")
"Removed torrent from Deluge: %s%s",
download_id,
" (with files)" if delete_files else "",
)
return True
return False
except Exception as e:
except _DELUGE_CLIENT_ERRORS as e:
self._log_error("remove", e)
return False
else:
return False
def get_download_path(self, download_id: str) -> Optional[str]:
def get_download_path(self, download_id: str) -> str | None:
"""Return the resolved download path for a Deluge torrent."""
try:
self._ensure_connected()
@@ -367,15 +435,17 @@ class DelugeClient(DownloadClient):
str(status.get("save_path", "")),
str(status.get("name", "")),
)
return None
except Exception as e:
except _DELUGE_CLIENT_ERRORS as e:
self._log_error("get_download_path", e, level="debug")
return None
else:
return None
def find_existing(
self, url: str, category: Optional[str] = None
) -> Optional[Tuple[str, DownloadStatus]]:
self, url: str, category: str | None = None
) -> tuple[str, DownloadStatus] | None:
"""Find an existing Deluge torrent matching a release URL."""
try:
self._ensure_connected()
@@ -393,10 +463,10 @@ class DelugeClient(DownloadClient):
full_status = self.get_status(torrent_info.info_hash)
return (torrent_info.info_hash, full_status)
return None
except Exception as e:
except _DELUGE_CLIENT_ERRORS as e:
self._authenticated = False
self._connected = False
logger.debug(f"Error checking for existing torrent: {e}")
logger.debug("Error checking for existing torrent: %s", e)
return None
else:
return None
+102 -82
View File
@@ -1,26 +1,30 @@
"""
NZBGet download client for Prowlarr integration.
"""NZBGet download client for Prowlarr integration.
Uses NZBGet's JSON-RPC API directly via requests (no external dependency).
"""
import json
from typing import Any, Optional, Tuple
from typing import Any, NoReturn
import requests
from shelfmark.core.config import config
from shelfmark.core.logger import setup_logger
from shelfmark.core.utils import normalize_http_url
from shelfmark.download.network import get_ssl_verify
from shelfmark.download.clients import (
DownloadClient,
DownloadStatus,
register_client,
with_retry,
)
from shelfmark.download.clients._coercion import config_text, normalize_http_config_url
from shelfmark.download.network import get_ssl_verify
logger = setup_logger(__name__)
_NZBGET_CLIENT_ERRORS = (AttributeError, OSError, RuntimeError, TypeError, ValueError)
def _raise_runtime_error(message: str) -> NoReturn:
raise RuntimeError(message)
@register_client("usenet")
@@ -30,30 +34,44 @@ class NZBGetClient(DownloadClient):
protocol = "usenet"
name = "nzbget"
def __init__(self):
def __init__(self) -> None:
"""Initialize NZBGet client with settings from config."""
raw_url = config.get("NZBGET_URL", "")
raw_url = config_text(config.get("NZBGET_URL", ""))
if not raw_url:
raise ValueError("NZBGET_URL is required")
msg = "NZBGET_URL is required"
raise ValueError(msg)
self.url = normalize_http_url(raw_url)
self.url = normalize_http_config_url(raw_url)
if not self.url:
raise ValueError("NZBGET_URL is invalid")
self.username = config.get("NZBGET_USERNAME", "nzbget")
self.password = config.get("NZBGET_PASSWORD", "")
self._category = config.get("NZBGET_CATEGORY", "Books")
msg = "NZBGET_URL is invalid"
raise ValueError(msg)
self.username = config_text(config.get("NZBGET_USERNAME", "nzbget"), "nzbget")
self.password = config_text(config.get("NZBGET_PASSWORD", ""))
self._category = config_text(config.get("NZBGET_CATEGORY", "Books"), "Books")
@staticmethod
def is_configured() -> bool:
"""Check if NZBGet is configured and selected as the usenet client."""
client = config.get("PROWLARR_USENET_CLIENT", "")
url = normalize_http_url(config.get("NZBGET_URL", ""))
client = config_text(config.get("PROWLARR_USENET_CLIENT", ""))
url = normalize_http_config_url(config.get("NZBGET_URL", ""))
return client == "nzbget" and bool(url)
def _try_remove_command(
self, command: str, nzb_id: int, download_id: str
) -> tuple[bool, Exception | None]:
"""Try one NZBGet delete command and return any error."""
try:
result = self._rpc_call("editqueue", [command, 0, "", nzb_id])
if result:
logger.info("Removed NZB from NZBGet (%s): %s", command, download_id)
return True, None
except _NZBGET_CLIENT_ERRORS as e:
return False, e
return False, None
@with_retry()
def _rpc_call(self, method: str, params: Optional[list] = None) -> Any:
"""
Make a JSON-RPC call to NZBGet.
def _rpc_call(self, method: str, params: list[object] | None = None) -> Any:
"""Make a JSON-RPC call to NZBGet.
Args:
method: RPC method name
@@ -64,15 +82,19 @@ class NZBGetClient(DownloadClient):
Raises:
Exception: If RPC call fails after retries.
"""
rpc_url = f"{self.url}/jsonrpc"
payload = json.dumps({
"jsonrpc": "2.0",
"id": 1,
"method": method,
"params": params or [],
}, separators=(',', ':'))
payload = json.dumps(
{
"jsonrpc": "2.0",
"id": 1,
"method": method,
"params": params or [],
},
separators=(",", ":"),
)
response = requests.post(
rpc_url,
@@ -85,34 +107,34 @@ class NZBGetClient(DownloadClient):
response.raise_for_status()
result = response.json()
if "error" in result and result["error"]:
raise Exception(result["error"].get("message", "RPC error"))
if result.get("error"):
raise RuntimeError(result["error"].get("message", "RPC error"))
return result.get("result")
def test_connection(self) -> Tuple[bool, str]:
def test_connection(self) -> tuple[bool, str]:
"""Test connection to NZBGet."""
try:
status = self._rpc_call("status")
version = status.get("Version", "unknown")
return True, f"Connected to NZBGet {version}"
except requests.exceptions.ConnectionError:
return False, "Could not connect to NZBGet"
except requests.exceptions.Timeout:
return False, "Connection timed out"
except Exception as e:
return False, f"Connection failed: {str(e)}"
except _NZBGET_CLIENT_ERRORS as e:
return False, f"Connection failed: {e!s}"
else:
return True, f"Connected to NZBGet {version}"
def add_download(
self,
url: str,
name: str,
category: Optional[str] = None,
expected_hash: Optional[str] = None,
**kwargs,
category: str | None = None,
expected_hash: str | None = None,
**kwargs: object,
) -> str:
"""
Add NZB by URL.
"""Add NZB by URL.
Fetches the NZB content from the URL (e.g., Prowlarr proxy) and sends
it base64-encoded to NZBGet, since NZBGet may not handle redirects well.
@@ -122,27 +144,29 @@ class NZBGetClient(DownloadClient):
name: Display name for the download
category: Category for organization (uses configured default if not specified)
expected_hash: Optional info_hash hint (unused)
**kwargs: Client-specific options passed through to the implementation.
Returns:
NZBGet download ID (NZBID).
Raises:
Exception: If adding fails.
"""
import base64
# Use configured category if not explicitly provided
category = category or self._category
resolved_category = category or self._category
try:
# Fetch NZB content from the URL (handles Prowlarr proxy redirects)
logger.debug(f"Fetching NZB from: {url}")
logger.debug("Fetching NZB from: %s", url)
response = requests.get(url, timeout=30, verify=get_ssl_verify(url))
response.raise_for_status()
nzb_content = base64.b64encode(response.content).decode('ascii')
nzb_content = base64.b64encode(response.content).decode("ascii")
# Ensure filename has .nzb extension
nzb_filename = name if name.endswith('.nzb') else f"{name}.nzb"
nzb_filename = name if name.endswith(".nzb") else f"{name}.nzb"
# NZBGet append method parameters (all 10 required):
# NZBFilename, Content, Category, Priority, AddToTop, AddPaused,
@@ -152,7 +176,7 @@ class NZBGetClient(DownloadClient):
[
nzb_filename, # NZBFilename
nzb_content, # Content (base64-encoded NZB)
category, # Category
resolved_category, # Category
0, # Priority (0 = normal)
False, # AddToTop
False, # AddPaused
@@ -163,27 +187,34 @@ class NZBGetClient(DownloadClient):
],
)
if nzb_id and nzb_id > 0:
logger.info(f"Added NZB to NZBGet: {nzb_id}")
if isinstance(nzb_id, int) and nzb_id > 0:
logger.info("Added NZB to NZBGet: %s", nzb_id)
return str(nzb_id)
raise Exception("NZBGet returned invalid ID")
if isinstance(nzb_id, str):
stripped_nzb_id = nzb_id.strip()
if stripped_nzb_id.isdigit() and int(stripped_nzb_id) > 0:
logger.info("Added NZB to NZBGet: %s", stripped_nzb_id)
return stripped_nzb_id
_raise_runtime_error("NZBGet returned invalid ID")
except requests.RequestException as e:
logger.error(f"Failed to fetch NZB from URL: {e}")
raise Exception(f"Failed to fetch NZB: {e}")
except Exception as e:
logger.error(f"NZBGet add failed: {e}")
logger.exception("Failed to fetch NZB from URL")
msg = f"Failed to fetch NZB: {e}"
raise RuntimeError(msg) from e
except _NZBGET_CLIENT_ERRORS:
logger.exception("NZBGet add failed")
raise
def get_status(self, download_id: str) -> DownloadStatus:
"""
Get NZB status by ID.
"""Get NZB status by ID.
Args:
download_id: NZBGet NZBID
Returns:
Current download status.
"""
try:
nzb_id = int(download_id)
@@ -195,18 +226,12 @@ class NZBGetClient(DownloadClient):
if group.get("NZBID") == nzb_id:
# Calculate progress
# NZBGet uses Hi/Lo for 64-bit values on 32-bit systems
file_size = (group.get("FileSizeHi", 0) << 32) + group.get(
"FileSizeLo", 0
)
file_size = (group.get("FileSizeHi", 0) << 32) + group.get("FileSizeLo", 0)
remaining = (group.get("RemainingSizeHi", 0) << 32) + group.get(
"RemainingSizeLo", 0
)
progress = (
((file_size - remaining) / file_size * 100)
if file_size > 0
else 0
)
progress = ((file_size - remaining) / file_size * 100) if file_size > 0 else 0
status = group.get("Status", "")
# Map NZBGet status to our states
@@ -229,9 +254,7 @@ class NZBGetClient(DownloadClient):
file_path=None,
download_speed=group.get("DownloadRate"),
eta=(
group.get("RemainingSec")
if group.get("RemainingSec", 0) > 0
else None
group.get("RemainingSec") if group.get("RemainingSec", 0) > 0 else None
),
)
@@ -254,7 +277,6 @@ class NZBGetClient(DownloadClient):
else:
file_path = None
if "SUCCESS" in status:
return DownloadStatus(
progress=100,
@@ -263,21 +285,20 @@ class NZBGetClient(DownloadClient):
complete=True,
file_path=file_path,
)
else:
return DownloadStatus(
progress=100,
state="error",
message=f"Download failed: {status}",
complete=True,
file_path=file_path,
)
return DownloadStatus(
progress=100,
state="error",
message=f"Download failed: {status}",
complete=True,
file_path=file_path,
)
# Not found in queue or history
return DownloadStatus.error("Download not found")
except Exception as e:
except _NZBGET_CLIENT_ERRORS as e:
return DownloadStatus.error(self._log_error("get_status", e))
def remove(self, download_id: str, delete_files: bool = False) -> bool:
def remove(self, download_id: str, *, delete_files: bool = False) -> bool:
"""Remove a download from NZBGet.
NZBGet can remove items from either the active queue (Group* commands) or from
@@ -289,6 +310,7 @@ class NZBGetClient(DownloadClient):
Returns:
True if successful.
"""
try:
nzb_id = int(download_id)
@@ -303,29 +325,27 @@ class NZBGetClient(DownloadClient):
else:
commands = ["GroupDelete", "HistoryDelete"]
last_error: Optional[Exception] = None
last_error: Exception | None = None
for command in commands:
try:
result = self._rpc_call("editqueue", [command, 0, "", nzb_id])
if result:
logger.info(f"Removed NZB from NZBGet ({command}): {download_id}")
return True
except Exception as e:
last_error = e
success, error = self._try_remove_command(command, nzb_id, download_id)
if success:
return True
if error is not None:
last_error = error
if last_error is not None:
self._log_error("remove", last_error)
return False
def get_download_path(self, download_id: str) -> Optional[str]:
"""
Get the path where NZB files are located.
def get_download_path(self, download_id: str) -> str | None:
"""Get the path where NZB files are located.
Args:
download_id: NZBGet NZBID
Returns:
Destination directory, or None.
"""
status = self.get_status(download_id)
return status.file_path
+244 -129
View File
@@ -1,37 +1,95 @@
"""qBittorrent download client for Prowlarr integration."""
from __future__ import annotations
import time
from http import HTTPStatus
from pathlib import Path
from types import SimpleNamespace
from typing import Optional, Tuple
from typing import NoReturn, TypedDict
import requests
from shelfmark.core.config import config
from shelfmark.core.logger import setup_logger
from shelfmark.core.utils import normalize_http_url
from shelfmark.download.network import get_ssl_verify
from shelfmark.download.clients import (
DownloadClient,
DownloadStatus,
register_client,
)
from shelfmark.download.clients._coercion import (
coerce_optional_float,
coerce_optional_int,
config_text,
normalize_http_config_url,
)
from shelfmark.download.clients.torrent_utils import (
extract_torrent_info,
)
from shelfmark.download.network import get_ssl_verify
try:
import qbittorrentapi as _qbittorrentapi
except ImportError:
_ImportedQBittorrentApiError = RuntimeError
_ImportedQBittorrentLoginFailed = RuntimeError
else:
_ImportedQBittorrentApiError = getattr(_qbittorrentapi, "APIError", RuntimeError)
_ImportedQBittorrentLoginFailed = getattr(_qbittorrentapi, "LoginFailed", RuntimeError)
logger = setup_logger(__name__)
_HASH_LENGTH_40 = 40
_HASH_LENGTH_ED2K = 32
_HTTP_STATUS_FORBIDDEN = HTTPStatus.FORBIDDEN
_HTTP_STATUS_NOT_FOUND = HTTPStatus.NOT_FOUND
_ONE_WEEK_IN_SECONDS = 604800
class _QBittorrentAddKwargs(TypedDict, total=False):
rename: str
category: str
save_path: str
tags: str
seeding_time_limit: int
ratio_limit: float
def _resolve_qbittorrent_exception_type(candidate: object) -> type[Exception]:
if isinstance(candidate, type) and issubclass(candidate, Exception):
return candidate
return RuntimeError
_QBittorrentApiError = _resolve_qbittorrent_exception_type(_ImportedQBittorrentApiError)
_QBittorrentLoginFailed = _resolve_qbittorrent_exception_type(_ImportedQBittorrentLoginFailed)
_QBITTORRENT_CLIENT_ERRORS = (
_QBittorrentLoginFailed,
_QBittorrentApiError,
AttributeError,
OSError,
RuntimeError,
TypeError,
ValueError,
)
def _hashes_match(hash1: str, hash2: str) -> bool:
"""Compare hashes, handling Amarr's 40-char zero-padded hashes vs 32-char ed2k hashes."""
h1, h2 = hash1.lower(), hash2.lower()
if h1 == h2:
return True
if len(h1) == 40 and len(h2) == 32 and h1.endswith("00000000"):
return h1[:32] == h2
if len(h2) == 40 and len(h1) == 32 and h2.endswith("00000000"):
return h2[:32] == h1
if len(h1) == _HASH_LENGTH_40 and len(h2) == _HASH_LENGTH_ED2K and h1.endswith("00000000"):
return h1[:_HASH_LENGTH_ED2K] == h2
if len(h2) == _HASH_LENGTH_40 and len(h1) == _HASH_LENGTH_ED2K and h2.endswith("00000000"):
return h2[:_HASH_LENGTH_ED2K] == h1
return False
def _raise_runtime_error(message: str) -> NoReturn:
raise RuntimeError(message)
def _normalize_tags(raw_tags: object) -> list[str]:
"""Normalize tag input to a clean, de-duplicated list of strings."""
if raw_tags is None:
@@ -61,11 +119,28 @@ def _normalize_tags(raw_tags: object) -> list[str]:
return tags
def _normalize_add_result(raw_result: object) -> str:
"""Normalize qBittorrent add responses to a comparable string."""
if raw_result is None:
return ""
if isinstance(raw_result, bytes):
return raw_result.decode("utf-8", errors="replace").strip()
return str(raw_result).strip()
def _is_explicit_add_failure(raw_result: object) -> bool:
"""Detect add responses that clearly indicate failure."""
normalized = _normalize_add_result(raw_result).rstrip(".").lower()
return normalized in {"fail", "fails", "error", "errors"}
@register_client("torrent")
class QBittorrentClient(DownloadClient):
"""qBittorrent download client."""
def _is_torrent_loaded(self, torrent_hash: str) -> tuple[bool, Optional[str]]:
def _is_torrent_loaded(self, torrent_hash: str) -> tuple[bool, str | None]:
"""Check whether qBittorrent has registered a torrent yet.
Uses `/api/v2/torrents/properties?hash=<hash>`.
@@ -75,9 +150,8 @@ class QBittorrentClient(DownloadClient):
Notes:
A false result with no error means "not loaded yet".
"""
import requests
"""
url = f"{self._base_url}/api/v2/torrents/properties"
params = {"hash": torrent_hash}
@@ -86,23 +160,24 @@ class QBittorrentClient(DownloadClient):
response = self._client._session.get(url, params=params, timeout=10)
# Re-authenticate and retry once on 403
if response.status_code == 403:
logger.debug("qBittorrent returned 403 for properties; re-authenticating and retrying")
if response.status_code == _HTTP_STATUS_FORBIDDEN:
logger.debug(
"qBittorrent returned 403 for properties; re-authenticating and retrying"
)
self._client.auth_log_in()
response = self._client._session.get(url, params=params, timeout=10)
if response.status_code == 403:
if response.status_code == _HTTP_STATUS_FORBIDDEN:
return False, "qBittorrent authentication failed (HTTP 403)"
# qBittorrent returns 404/409-ish responses depending on version when missing.
if response.status_code == 404:
if response.status_code == _HTTP_STATUS_NOT_FOUND:
return False, None
response.raise_for_status()
return True, None
except requests.exceptions.HTTPError as e:
status = getattr(getattr(e, "response", None), "status_code", None)
if status == 404:
if status == _HTTP_STATUS_NOT_FOUND:
return False, None
if status:
return False, f"qBittorrent API request failed (HTTP {status})"
@@ -111,42 +186,54 @@ class QBittorrentClient(DownloadClient):
return False, f"Cannot connect to qBittorrent at {self._base_url}"
except requests.exceptions.Timeout:
return False, f"qBittorrent request timed out at {self._base_url}"
except Exception as e:
except requests.exceptions.InvalidSchema:
return (
False,
"qBittorrent URL is invalid (missing http:// or https://). "
f"Configured: {self._base_url}",
)
except _QBITTORRENT_CLIENT_ERRORS as e:
return False, f"qBittorrent API error: {type(e).__name__}: {e}"
else:
return True, None
protocol = "torrent"
name = "qbittorrent"
def __init__(self):
def __init__(self) -> None:
"""Initialize qBittorrent client with settings from config."""
# Lazy import to avoid dependency issues if not using torrents
from qbittorrentapi import Client
raw_url = config.get("QBITTORRENT_URL", "")
if not raw_url:
raise ValueError("QBITTORRENT_URL is required")
msg = "QBITTORRENT_URL is required"
raise ValueError(msg)
# We use `_base_url` for direct HTTP calls, so it must be a fully-qualified URL.
self._base_url = normalize_http_url(raw_url)
self._base_url = normalize_http_config_url(raw_url, require_string=True)
if not self._base_url:
raise ValueError("QBITTORRENT_URL is invalid")
msg = "QBITTORRENT_URL is invalid"
raise ValueError(msg)
username = config_text(config.get("QBITTORRENT_USERNAME", ""))
password = config_text(config.get("QBITTORRENT_PASSWORD", ""))
# qbittorrent-api accepts either a full URL or host:port; prefer the normalized URL
# for consistency.
self._client = Client(
host=self._base_url,
username=config.get("QBITTORRENT_USERNAME", ""),
password=config.get("QBITTORRENT_PASSWORD", ""),
username=username,
password=password,
VERIFY_WEBUI_CERTIFICATE=get_ssl_verify(self._base_url),
)
self._category = config.get("QBITTORRENT_CATEGORY", "books")
self._download_dir = config.get("QBITTORRENT_DOWNLOAD_DIR", "")
self._category = config_text(config.get("QBITTORRENT_CATEGORY", "books"))
self._download_dir = config_text(config.get("QBITTORRENT_DOWNLOAD_DIR", ""))
self._tags = _normalize_tags(config.get("QBITTORRENT_TAG", []))
def _get_torrents_info(
self, torrent_hash: Optional[str] = None
) -> tuple[list[SimpleNamespace], Optional[str]]:
self, torrent_hash: str | None = None
) -> tuple[list[SimpleNamespace], str | None]:
"""Get torrent info using GET.
Behaviors:
@@ -157,9 +244,8 @@ class QBittorrentClient(DownloadClient):
Returns:
(torrents, error_message)
"""
import requests
"""
url = f"{self._base_url}/api/v2/torrents/info"
def do_request(params: dict[str, str]) -> requests.Response:
@@ -171,13 +257,13 @@ class QBittorrentClient(DownloadClient):
response: requests.Response,
*,
request_params: dict[str, str],
) -> tuple[list[SimpleNamespace], Optional[str]]:
if response.status_code == 403:
) -> tuple[list[SimpleNamespace], str | None]:
if response.status_code == _HTTP_STATUS_FORBIDDEN:
logger.debug("qBittorrent returned 403; re-authenticating and retrying")
self._client.auth_log_in()
response = self._client._session.get(url, params=request_params, timeout=10)
if response.status_code == 403:
if response.status_code == _HTTP_STATUS_FORBIDDEN:
logger.warning("qBittorrent authentication failed (HTTP 403)")
return [], "qBittorrent authentication failed (HTTP 403)"
@@ -219,48 +305,49 @@ class QBittorrentClient(DownloadClient):
return all_torrents, None
return torrents, None
except requests.exceptions.HTTPError as e:
status = getattr(getattr(e, "response", None), "status_code", None)
if status:
logger.warning(f"qBittorrent API error (HTTP {status}): {e}")
logger.warning("qBittorrent API error (HTTP %s): %s", status, e)
return [], f"qBittorrent API request failed (HTTP {status})"
logger.warning(f"qBittorrent API error: {e}")
logger.warning("qBittorrent API error: %s", e)
return [], "qBittorrent API request failed"
except requests.exceptions.ConnectionError:
logger.warning(f"Cannot connect to qBittorrent at {self._base_url}")
logger.warning("Cannot connect to qBittorrent at %s", self._base_url)
return [], f"Cannot connect to qBittorrent at {self._base_url}"
except requests.exceptions.Timeout:
logger.warning(f"qBittorrent request timed out at {self._base_url}")
logger.warning("qBittorrent request timed out at %s", self._base_url)
return [], f"qBittorrent request timed out at {self._base_url}"
except Exception as e:
logger.debug(f"Failed to get torrents info: {e}")
# requests raises InvalidSchema when the base URL doesn't include http(s)
if type(e).__name__ == "InvalidSchema":
return (
[],
"qBittorrent URL is invalid (missing http:// or https://). "
f"Configured: {self._base_url}",
)
except requests.exceptions.InvalidSchema:
logger.debug("Failed to get torrents info: invalid qBittorrent URL: %s", self._base_url)
return (
[],
"qBittorrent URL is invalid (missing http:// or https://). "
f"Configured: {self._base_url}",
)
except _QBITTORRENT_CLIENT_ERRORS as e:
logger.debug("Failed to get torrents info: %s", e)
return [], f"qBittorrent API error: {type(e).__name__}: {e}"
else:
return torrents, None
@staticmethod
def is_configured() -> bool:
"""Check if qBittorrent is configured and selected as the torrent client."""
client = config.get("PROWLARR_TORRENT_CLIENT", "")
url = normalize_http_url(config.get("QBITTORRENT_URL", ""))
client = config_text(config.get("PROWLARR_TORRENT_CLIENT", ""))
url = normalize_http_config_url(config.get("QBITTORRENT_URL", ""), require_string=True)
return client == "qbittorrent" and bool(url)
def test_connection(self) -> Tuple[bool, str]:
def test_connection(self) -> tuple[bool, str]:
"""Test connection to qBittorrent."""
try:
self._client.auth_log_in()
api_version = self._client.app.web_api_version
except _QBITTORRENT_CLIENT_ERRORS as e:
return False, f"Connection failed: {e!s}"
else:
return True, f"Connected to qBittorrent (API v{api_version})"
except Exception as e:
return False, f"Connection failed: {str(e)}"
def add_download(
self,
@@ -268,50 +355,67 @@ class QBittorrentClient(DownloadClient):
name: str,
category: str | None = None,
expected_hash: str | None = None,
**kwargs,
**kwargs: object,
) -> str:
"""
Add torrent by URL (magnet or .torrent).
"""Add torrent by URL (magnet or .torrent).
Args:
url: Magnet link or .torrent URL
name: Display name for the torrent
category: Category for organization (uses configured default if not specified)
expected_hash: Optional info_hash hint (from Prowlarr)
**kwargs: Client-specific options passed through to the implementation.
Returns:
Torrent hash (info_hash).
Raises:
Exception: If adding fails.
"""
try:
# Use configured category if not explicitly provided
category = category or self._category
tags = self._tags
seeding_time_limit: int | None = None
ratio_limit: float | None = None
# Ensure category exists (may already exist, which is fine)
try:
self._client.torrents_create_category(name=category)
except Exception as e:
# Conflict409Error means category exists - that's expected
# Log other errors but continue since download may still work
if "Conflict" not in type(e).__name__ and "409" not in str(e):
logger.debug(f"Could not create category '{category}': {type(e).__name__}: {e}")
if category:
try:
self._client.torrents_create_category(name=category)
except _QBITTORRENT_CLIENT_ERRORS as e:
# Conflict409Error means category exists - that's expected
# Log other errors but continue since download may still work
if "Conflict" not in type(e).__name__ and "409" not in str(e):
logger.debug(
"Could not create category '%s': %s: %s",
category,
type(e).__name__,
e,
)
torrent_info = extract_torrent_info(url, expected_hash=expected_hash)
expected_hash = torrent_info.info_hash
torrent_data = torrent_info.torrent_data
# Add the torrent - use file content if we have it, otherwise URL
add_kwargs = {
"category": category,
"rename": name,
}
# Per-torrent seeding limits from indexer
seeding_time_limit_value = kwargs.get("seeding_time_limit")
seeding_time_limit = coerce_optional_int(seeding_time_limit_value)
ratio_limit_value = kwargs.get("ratio_limit")
ratio_limit = coerce_optional_float(ratio_limit_value)
add_kwargs: _QBittorrentAddKwargs = {"rename": name}
if category:
add_kwargs["category"] = category
if self._download_dir:
add_kwargs["save_path"] = self._download_dir
if tags:
add_kwargs["tags"] = ",".join(tags)
if seeding_time_limit is not None:
add_kwargs["seeding_time_limit"] = seeding_time_limit
if ratio_limit is not None:
add_kwargs["ratio_limit"] = ratio_limit
if torrent_data:
result = self._client.torrents_add(
@@ -326,42 +430,46 @@ class QBittorrentClient(DownloadClient):
**add_kwargs,
)
logger.debug(f"qBittorrent add result: {result}")
result_text = _normalize_add_result(result)
logger.debug("qBittorrent add result: %s", result_text)
if result == "Ok.":
if not expected_hash:
raise Exception("Could not determine torrent hash from URL")
if not expected_hash:
_raise_runtime_error("Could not determine torrent hash from URL")
# Wait for torrent to appear in client.
# Use `/torrents/properties?hash=` rather than relying on `torrents/info`
# listing being immediately consistent.
for _ in range(10):
loaded, error = self._is_torrent_loaded(expected_hash)
if error:
logger.debug(f"qBittorrent add_download: {error}")
if loaded:
logger.info(f"Added torrent: {expected_hash}")
return expected_hash.lower()
time.sleep(0.5)
if _is_explicit_add_failure(result):
_raise_runtime_error(f"Failed to add torrent: {result_text}")
# Client said Ok, trust it
logger.warning(f"Torrent not yet visible, returning expected hash")
return expected_hash
# Some qBittorrent-compatible clients return HTTP 200 with an empty body
# instead of qBittorrent's literal "Ok." response. Prefer verifying that
# the torrent becomes visible over trusting the response body alone.
for _ in range(10):
loaded, error = self._is_torrent_loaded(expected_hash)
if error:
logger.debug("qBittorrent add_download: %s", error)
if loaded:
logger.info("Added torrent: %s", expected_hash)
return expected_hash.lower()
time.sleep(0.5)
raise Exception(f"Failed to add torrent: {result}")
except Exception as e:
logger.error(f"qBittorrent add failed: {e}")
logger.warning(
"Torrent add was not confirmed within the visibility grace period (response=%s), returning expected hash",
result_text,
)
except _QBITTORRENT_CLIENT_ERRORS:
logger.exception("qBittorrent add failed")
raise
else:
return expected_hash
def get_status(self, download_id: str) -> DownloadStatus:
"""
Get torrent status by hash.
"""Get torrent status by hash.
Args:
download_id: Torrent info_hash
Returns:
Current download status.
"""
try:
torrents, error = self._get_torrents_info(download_id)
@@ -373,7 +481,7 @@ class QBittorrentClient(DownloadClient):
t
for t in torrents
if isinstance(getattr(t, "hash", None), str)
and _hashes_match(getattr(t, "hash"), download_id)
and _hashes_match(t.hash, download_id)
),
None,
)
@@ -382,7 +490,10 @@ class QBittorrentClient(DownloadClient):
# Map qBittorrent states to our states and user-friendly messages
state_info = {
"downloading": ("downloading", None), # None = use default progress message
"downloading": (
"downloading",
None,
), # None = use default progress message
"stalledDL": ("downloading", "Stalled"),
"metaDL": ("downloading", "Fetching metadata"),
"forcedDL": ("downloading", None),
@@ -417,7 +528,11 @@ class QBittorrentClient(DownloadClient):
message = "Complete"
torrent_eta = getattr(torrent, "eta", 0)
eta = torrent_eta if isinstance(torrent_eta, int) and 0 < torrent_eta < 604800 else None
eta = (
torrent_eta
if isinstance(torrent_eta, int) and 0 < torrent_eta < _ONE_WEEK_IN_SECONDS
else None
)
# Get file path for completed downloads
file_path = None
@@ -436,12 +551,11 @@ class QBittorrentClient(DownloadClient):
download_speed=torrent_speed,
eta=eta,
)
except Exception as e:
except _QBITTORRENT_CLIENT_ERRORS as e:
return DownloadStatus.error(self._log_error("get_status", e))
def remove(self, download_id: str, delete_files: bool = False) -> bool:
"""
Remove a torrent from qBittorrent.
def remove(self, download_id: str, *, delete_files: bool = False) -> bool:
"""Remove a torrent from qBittorrent.
Args:
download_id: Torrent info_hash
@@ -449,21 +563,22 @@ class QBittorrentClient(DownloadClient):
Returns:
True if successful.
"""
try:
self._client.torrents_delete(
torrent_hashes=download_id, delete_files=delete_files
)
self._client.torrents_delete(torrent_hashes=download_id, delete_files=delete_files)
logger.info(
f"Removed torrent from qBittorrent: {download_id}"
+ (" (with files)" if delete_files else "")
"Removed torrent from qBittorrent: %s%s",
download_id,
" (with files)" if delete_files else "",
)
return True
except Exception as e:
except _QBITTORRENT_CLIENT_ERRORS as e:
self._log_error("remove", e)
return False
else:
return True
def get_download_path(self, download_id: str) -> Optional[str]:
def get_download_path(self, download_id: str) -> str | None:
"""Get the path where torrent files are located.
Prefer `content_path` when available.
@@ -474,12 +589,10 @@ class QBittorrentClient(DownloadClient):
- `/api/v2/torrents/files?hash=<hash>` for the first file name
- join `save_path` with the torrent's top-level directory
"""
import os
try:
torrents, error = self._get_torrents_info(download_id)
if error:
logger.debug(f"qBittorrent get_download_path: {error}")
logger.debug("qBittorrent get_download_path: %s", error)
return None
torrent = next(
@@ -487,7 +600,7 @@ class QBittorrentClient(DownloadClient):
t
for t in torrents
if isinstance(getattr(t, "hash", None), str)
and _hashes_match(getattr(t, "hash"), download_id)
and _hashes_match(t.hash, download_id)
),
None,
)
@@ -495,11 +608,11 @@ class QBittorrentClient(DownloadClient):
return None
return self._resolve_completed_download_path(torrent)
except Exception as e:
except _QBITTORRENT_CLIENT_ERRORS as e:
self._log_error("get_download_path", e, level="debug")
return None
def _resolve_completed_download_path(self, torrent: SimpleNamespace) -> Optional[str]:
def _resolve_completed_download_path(self, torrent: SimpleNamespace) -> str | None:
"""Resolve the completed path for a torrent.
Centralizes the logic shared by `get_status()` and `get_download_path()`:
@@ -507,7 +620,6 @@ class QBittorrentClient(DownloadClient):
- otherwise derive via properties+files
- finally fall back to `save_path + name`
"""
# Prefer content_path, but treat content_path == save_path as invalid.
content_path = getattr(torrent, "content_path", "")
save_path = getattr(torrent, "save_path", "")
@@ -526,19 +638,18 @@ class QBittorrentClient(DownloadClient):
getattr(torrent, "name", ""),
)
def _derive_download_path_from_files(self, download_id: str) -> Optional[str]:
def _derive_download_path_from_files(self, download_id: str) -> str | None:
"""Derive completed download path using `/torrents/properties` + `/torrents/files`.
This mirrors how common automation apps derive the path when
`content_path` isn't provided.
"""
import os
import requests
def get_with_auth(url: str, params: dict[str, str]) -> requests.Response:
self._client.auth_log_in()
resp = self._client._session.get(url, params=params, timeout=10)
if resp.status_code == 403:
if resp.status_code == _HTTP_STATUS_FORBIDDEN:
logger.debug("qBittorrent returned 403; re-authenticating and retrying")
self._client.auth_log_in()
resp = self._client._session.get(url, params=params, timeout=10)
@@ -549,7 +660,7 @@ class QBittorrentClient(DownloadClient):
files_url = f"{self._base_url}/api/v2/torrents/files"
props_resp = get_with_auth(properties_url, {"hash": download_id})
if props_resp.status_code == 404:
if props_resp.status_code == _HTTP_STATUS_NOT_FOUND:
return None
props_resp.raise_for_status()
props = props_resp.json() if isinstance(props_resp.json(), dict) else {}
@@ -559,7 +670,7 @@ class QBittorrentClient(DownloadClient):
return None
files_resp = get_with_auth(files_url, {"hash": download_id})
if files_resp.status_code == 404:
if files_resp.status_code == _HTTP_STATUS_NOT_FOUND:
return None
files_resp.raise_for_status()
files = files_resp.json() if isinstance(files_resp.json(), list) else []
@@ -576,14 +687,18 @@ class QBittorrentClient(DownloadClient):
if not top_level:
return None
return os.path.normpath(os.path.join(save_path, top_level))
except Exception as e:
logger.debug(f"qBittorrent could not derive path from files: {type(e).__name__}: {e}")
return os.path.normpath(str(Path(save_path) / top_level))
except _QBITTORRENT_CLIENT_ERRORS as e:
logger.debug(
"qBittorrent could not derive path from files: %s: %s",
type(e).__name__,
e,
)
return None
def find_existing(
self, url: str, category: Optional[str] = None
) -> Optional[Tuple[str, DownloadStatus]]:
self, url: str, category: str | None = None
) -> tuple[str, DownloadStatus] | None:
"""Check if a torrent for this URL already exists in qBittorrent."""
try:
torrent_info = extract_torrent_info(url)
@@ -592,7 +707,7 @@ class QBittorrentClient(DownloadClient):
torrents, error = self._get_torrents_info(torrent_info.info_hash)
if error:
logger.debug(f"qBittorrent find_existing: {error}")
logger.debug("qBittorrent find_existing: %s", error)
return None
torrent = next(
@@ -600,15 +715,15 @@ class QBittorrentClient(DownloadClient):
t
for t in torrents
if isinstance(getattr(t, "hash", None), str)
and _hashes_match(getattr(t, "hash"), torrent_info.info_hash)
and _hashes_match(t.hash, torrent_info.info_hash)
),
None,
)
if torrent and isinstance(getattr(torrent, "hash", None), str):
torrent_hash = getattr(torrent, "hash")
torrent_hash = torrent.hash
return (torrent_hash.lower(), self.get_status(torrent_hash.lower()))
except _QBITTORRENT_CLIENT_ERRORS as e:
logger.debug("Error checking for existing torrent: %s", e)
return None
except Exception as e:
logger.debug(f"Error checking for existing torrent: {e}")
else:
return None
+150 -85
View File
@@ -1,30 +1,73 @@
"""
rTorrent download client for Prowlarr integration.
"""rTorrent download client for Prowlarr integration.
Uses xmlrpc to communicate with rTorrent's RPC interface.
"""
import ssl
from typing import Any, Optional, Tuple
import xmlrpc.client as stdlib_xmlrpc_client
from typing import Any, NoReturn, Protocol, cast
from urllib.parse import urlparse
from shelfmark.core.config import config
from shelfmark.core.logger import setup_logger
from shelfmark.core.utils import normalize_http_url, get_hardened_xmlrpc_client
from shelfmark.download.network import get_ssl_verify
from shelfmark.core.utils import get_hardened_xmlrpc_client
from shelfmark.download.clients import (
DownloadClient,
DownloadStatus,
register_client,
)
from shelfmark.download.clients._coercion import config_text, normalize_http_config_url
from shelfmark.download.clients.torrent_utils import (
extract_torrent_info,
)
from shelfmark.download.network import get_ssl_verify
logger = setup_logger(__name__)
def _create_rtorrent_server_proxy(url: str) -> Any:
_ETA_MAX_SECONDS = 604800
_RTORRENT_CLIENT_ERRORS = (
AttributeError,
OSError,
RuntimeError,
TypeError,
ValueError,
stdlib_xmlrpc_client.Error,
)
class _RTorrentSystemProtocol(Protocol):
def client_version(self) -> object: ...
class _RTorrentLoadProtocol(Protocol):
def raw_start(self, target: str, torrent_data: bytes, commands: str) -> object: ...
def start(self, target: str, url: str, commands: str) -> object: ...
class _RTorrentDownloadProtocol(Protocol):
def multicall2(self, *args: object) -> list[list[Any]]: ...
def delete_tied(self, download_id: str) -> object: ...
def erase(self, download_id: str) -> object: ...
def stop(self, download_id: str) -> object: ...
class _RTorrentDirectoryProtocol(Protocol):
def default(self) -> str: ...
class _RTorrentRpcProtocol(Protocol):
system: _RTorrentSystemProtocol
load: _RTorrentLoadProtocol
d: _RTorrentDownloadProtocol
directory: _RTorrentDirectoryProtocol
def _create_rtorrent_server_proxy(url: str) -> _RTorrentRpcProtocol:
"""Create an XML-RPC ServerProxy honoring certificate validation mode."""
xmlrpc_client = get_hardened_xmlrpc_client()
@@ -34,9 +77,13 @@ def _create_rtorrent_server_proxy(url: str) -> Any:
ssl_context.check_hostname = False
ssl_context.verify_mode = ssl.CERT_NONE
transport = xmlrpc_client.SafeTransport(context=ssl_context)
return xmlrpc_client.ServerProxy(url, transport=transport)
return cast(_RTorrentRpcProtocol, xmlrpc_client.ServerProxy(url, transport=transport))
return xmlrpc_client.ServerProxy(url)
return cast(_RTorrentRpcProtocol, xmlrpc_client.ServerProxy(url))
def _raise_runtime_error(message: str) -> NoReturn:
raise RuntimeError(message)
@register_client("torrent")
@@ -46,66 +93,68 @@ class RTorrentClient(DownloadClient):
protocol = "torrent"
name = "rtorrent"
def __init__(self):
def __init__(self) -> None:
"""Initialize rTorrent client with settings from config."""
raw_url = config.get("RTORRENT_URL", "")
raw_url = config_text(config.get("RTORRENT_URL", ""))
if not raw_url:
raise ValueError("RTORRENT_URL is required")
msg = "RTORRENT_URL is required"
raise ValueError(msg)
self._base_url = normalize_http_url(raw_url)
self._base_url = normalize_http_config_url(raw_url)
if not self._base_url:
raise ValueError("RTORRENT_URL is invalid")
msg = "RTORRENT_URL is invalid"
raise ValueError(msg)
username = config.get("RTORRENT_USERNAME", "")
password = config.get("RTORRENT_PASSWORD", "")
username = config_text(config.get("RTORRENT_USERNAME", ""))
password = config_text(config.get("RTORRENT_PASSWORD", ""))
if username and password:
parsed = urlparse(self._base_url)
self._base_url = (
f"{parsed.scheme}://{username}:{password}@{parsed.netloc}{parsed.path}"
)
self._base_url = f"{parsed.scheme}://{username}:{password}@{parsed.netloc}{parsed.path}"
self._rpc = _create_rtorrent_server_proxy(self._base_url)
self._download_dir = config.get("RTORRENT_DOWNLOAD_DIR", "")
self._label = config.get("RTORRENT_LABEL", "")
self._download_dir = config_text(config.get("RTORRENT_DOWNLOAD_DIR", ""))
self._label = config_text(config.get("RTORRENT_LABEL", ""))
@staticmethod
def is_configured() -> bool:
"""Check if rTorrent is configured and selected as the torrent client."""
client = config.get("PROWLARR_TORRENT_CLIENT", "")
url = normalize_http_url(config.get("RTORRENT_URL", ""))
client = config_text(config.get("PROWLARR_TORRENT_CLIENT", ""))
url = normalize_http_config_url(config.get("RTORRENT_URL", ""))
return client == "rtorrent" and bool(url)
def test_connection(self) -> Tuple[bool, str]:
def test_connection(self) -> tuple[bool, str]:
"""Test connection to rTorrent."""
try:
version = self._rpc.system.client_version()
except _RTORRENT_CLIENT_ERRORS as e:
return False, f"Connection failed: {e!s}"
else:
return True, f"Connected to rTorrent {version}"
except Exception as e:
return False, f"Connection failed: {str(e)}"
def add_download(
self,
url: str,
name: str,
category: Optional[str] = None,
expected_hash: Optional[str] = None,
**kwargs,
category: str | None = None,
expected_hash: str | None = None,
**kwargs: object,
) -> str:
"""
Add torrent by URL (magnet or .torrent).
"""Add torrent by URL (magnet or .torrent).
Args:
url: Magnet link or .torrent URL
name: Display name for the torrent
category: Category for organization (uses configured label if not specified)
expected_hash: Optional info_hash hint (from Prowlarr)
**kwargs: Client-specific options passed through to the implementation.
Returns:
Torrent hash (info_hash).
Raises:
Exception: If adding fails.
"""
try:
torrent_info = extract_torrent_info(url, expected_hash=expected_hash)
@@ -114,44 +163,53 @@ class RTorrentClient(DownloadClient):
label = category or self._label
if label:
logger.debug(f"Setting rTorrent label: {label}")
logger.debug("Setting rTorrent label: %s", label)
commands.append(f"d.custom1.set={label}")
download_dir = self._download_dir or self._get_download_dir()
if download_dir:
logger.debug(f"Setting rTorrent download directory: {download_dir}")
logger.debug("Setting rTorrent download directory: %s", download_dir)
commands.append(f"d.directory.set={download_dir}")
if torrent_info.torrent_data:
logger.debug(f"Adding torrent data directly to rTorrent for: {name} with commands: {commands} with data size: {len(torrent_info.torrent_data)}")
self._rpc.load.raw_start(
"", torrent_info.torrent_data, ";".join(commands)
logger.debug(
"Adding torrent data directly to rTorrent for: %s with commands: %s with data size: %s",
name,
commands,
len(torrent_info.torrent_data),
)
self._rpc.load.raw_start("", torrent_info.torrent_data, ";".join(commands))
else:
logger.debug(f"Adding torrent URL to rTorrent for: {name} with commands: {commands} with URL: {url}")
logger.debug(
"Adding torrent URL to rTorrent for: %s with commands: %s with URL: %s",
name,
commands,
url,
)
add_url = torrent_info.magnet_url or url
self._rpc.load.start("", add_url, ";".join(commands))
torrent_hash = torrent_info.info_hash or expected_hash
if not torrent_hash:
raise Exception("Could not determine torrent hash from URL")
_raise_runtime_error("Could not determine torrent hash from URL")
logger.debug(f"Added torrent to rTorrent: {torrent_hash}")
logger.debug("Added torrent to rTorrent: %s", torrent_hash)
except _RTORRENT_CLIENT_ERRORS:
logger.exception("rTorrent add failed")
raise
else:
return torrent_hash
except Exception as e:
logger.error(f"rTorrent add failed: {e}")
raise
def get_status(self, download_id: str) -> DownloadStatus:
"""
Get torrent status by hash.
"""Get torrent status by hash.
Args:
download_id: Torrent info_hash
Returns:
Current download status.
"""
try:
# rtorrent is somehow case sensitive and requires uppercase hashes for look
@@ -169,39 +227,40 @@ class RTorrentClient(DownloadClient):
"d.complete=",
)
torrent_list = [t for t in all_torrents if t and t[0] == download_id]
logger.debug(f"Fetched torrent status from rTorrent for: {download_id} - {torrent_list}")
logger.debug(
"Fetched torrent status from rTorrent for: %s - %s",
download_id,
torrent_list,
)
if not torrent_list:
logger.warning(f"Torrent not found in rTorrent: {download_id}")
logger.warning("Torrent not found in rTorrent: %s", download_id)
return DownloadStatus.error("Torrent not found")
torrent = torrent_list[0]
if not torrent:
logger.warning(f"Torrent data is empty for: {download_id}")
logger.warning("Torrent data is empty for: %s", download_id)
return DownloadStatus.error("Torrent not found")
logger.debug(f"Torrent data for {download_id}: {torrent}")
logger.debug("Torrent data for %s: %s", download_id, torrent)
(
torrent_hash,
_torrent_hash,
state,
bytes_downloaded,
bytes_total,
down_rate,
up_rate,
custom_category,
_up_rate,
_custom_category,
complete,
) = torrent
try:
state = int(state)
except Exception:
except TypeError, ValueError:
state = 0
complete = bool(complete)
if bytes_total > 0:
progress = (bytes_downloaded / bytes_total) * 100
else:
progress = 0
progress = (bytes_downloaded / bytes_total) * 100 if bytes_total > 0 else 0
bytes_left = max(0, bytes_total - bytes_downloaded)
@@ -221,7 +280,7 @@ class RTorrentClient(DownloadClient):
eta = None
if down_rate > 0 and bytes_left > 0:
eta_seconds = bytes_left / down_rate
if eta_seconds < 604800:
if eta_seconds < _ETA_MAX_SECONDS:
eta = int(eta_seconds)
file_path = None
@@ -238,14 +297,13 @@ class RTorrentClient(DownloadClient):
eta=eta,
)
except Exception as e:
except _RTORRENT_CLIENT_ERRORS as e:
error_type = type(e).__name__
logger.error(f"rTorrent get_status failed ({error_type}): {e}")
logger.exception("rTorrent get_status failed (%s)", error_type)
return DownloadStatus.error(f"{error_type}: {e}")
def remove(self, download_id: str, delete_files: bool = False) -> bool:
"""
Remove a torrent from rTorrent.
def remove(self, download_id: str, *, delete_files: bool = False) -> bool:
"""Remove a torrent from rTorrent.
Args:
download_id: Torrent info_hash
@@ -253,6 +311,7 @@ class RTorrentClient(DownloadClient):
Returns:
True if successful.
"""
try:
if delete_files:
@@ -263,35 +322,37 @@ class RTorrentClient(DownloadClient):
self._rpc.d.erase(download_id)
logger.info(
f"Removed torrent from rTorrent: {download_id}"
+ (" (with files)" if delete_files else "")
"Removed torrent from rTorrent: %s%s",
download_id,
" (with files)" if delete_files else "",
)
return True
except Exception as e:
except _RTORRENT_CLIENT_ERRORS as e:
error_type = type(e).__name__
logger.error(f"rTorrent remove failed ({error_type}): {e}")
logger.exception("rTorrent remove failed (%s)", error_type)
return False
else:
return True
def get_download_path(self, download_id: str) -> Optional[str]:
"""
Get the path where torrent files are located.
def get_download_path(self, download_id: str) -> str | None:
"""Get the path where torrent files are located.
Args:
download_id: Torrent info_hash
Returns:
Content path (file or directory), or None.
"""
try:
return self._get_torrent_path(download_id)
except Exception as e:
except _RTORRENT_CLIENT_ERRORS as e:
error_type = type(e).__name__
logger.debug(f"rTorrent get_download_path failed ({error_type}): {e}")
logger.debug("rTorrent get_download_path failed (%s): %s", error_type, e)
return None
def find_existing(
self, url: str, category: Optional[str] = None
) -> Optional[Tuple[str, DownloadStatus]]:
self, url: str, category: str | None = None
) -> tuple[str, DownloadStatus] | None:
"""Check if a torrent for this URL already exists in rTorrent."""
try:
torrent_info = extract_torrent_info(url)
@@ -302,23 +363,26 @@ class RTorrentClient(DownloadClient):
status = self.get_status(torrent_info.info_hash)
if status.state != DownloadStatus.error("").state:
return (torrent_info.info_hash, status)
except Exception:
pass
except _RTORRENT_CLIENT_ERRORS as exc:
logger.debug(
"Could not fetch existing rTorrent status for %s: %s",
torrent_info.info_hash,
exc,
)
except _RTORRENT_CLIENT_ERRORS as e:
logger.debug("Error checking for existing torrent: %s", e)
return None
except Exception as e:
logger.debug(f"Error checking for existing torrent: {e}")
else:
return None
def _get_download_dir(self) -> str:
"""Get the download directory from rTorrent config."""
try:
download_dir = self._rpc.directory.default()
return download_dir
except Exception:
return self._rpc.directory.default()
except _RTORRENT_CLIENT_ERRORS:
return "/downloads"
def _get_torrent_path(self, download_id: str) -> Optional[str]:
def _get_torrent_path(self, download_id: str) -> str | None:
"""Get the file path of a torrent by hash.
Uses `d.base_path` for the item output path. In the xmlrpc interface
@@ -337,6 +401,7 @@ class RTorrentClient(DownloadClient):
if not details:
return None
path = details[0][0]
return path if path else None
except Exception:
except _RTORRENT_CLIENT_ERRORS:
return None
else:
return str(path) if path else None
+135 -102
View File
@@ -1,49 +1,59 @@
"""
SABnzbd download client for Prowlarr integration.
"""SABnzbd download client for Prowlarr integration.
Uses SABnzbd's REST API directly via requests (no external dependency).
"""
from typing import Any, Optional, Tuple
from typing import Any
from urllib.parse import urlparse
import requests
from shelfmark.core.config import config
from shelfmark.core.logger import setup_logger
from shelfmark.core.utils import normalize_http_url
from shelfmark.download.network import get_ssl_verify
from shelfmark.download.clients import (
DownloadClient,
DownloadStatus,
register_client,
with_retry,
)
from shelfmark.download.clients._coercion import config_text, normalize_http_config_url
from shelfmark.download.network import get_ssl_verify
logger = setup_logger(__name__)
_ETA_PART_COUNT = 3
_SPEED_PARTS_MIN = 2
_SABNZBD_CLIENT_ERRORS = (
requests.exceptions.RequestException,
AttributeError,
RuntimeError,
TypeError,
ValueError,
)
_SabnzbdRequestParam = str | int | float | bool
def _parse_eta(eta_str: str) -> Optional[int]:
def _parse_eta(eta_str: str) -> int | None:
"""Parse SABnzbd ETA string (format: 'H:MM:SS') to seconds."""
if not eta_str or eta_str == "0:00:00":
return None
try:
parts = eta_str.split(":")
if len(parts) == 3:
if len(parts) == _ETA_PART_COUNT:
return int(parts[0]) * 3600 + int(parts[1]) * 60 + int(parts[2])
except (ValueError, IndexError):
except ValueError, IndexError:
pass
return None
def _parse_speed(slot: dict) -> Optional[int]:
def _parse_speed(slot: dict) -> int | None:
"""Parse download speed from SABnzbd slot data, returning bytes/sec."""
# Prefer kbpersec field (more reliable numeric value)
kbpersec_str = slot.get("kbpersec", "")
if kbpersec_str:
try:
return int(float(kbpersec_str) * 1024)
except (ValueError, TypeError):
except ValueError, TypeError:
pass
# Fall back to human-readable speed field
@@ -53,7 +63,7 @@ def _parse_speed(slot: dict) -> Optional[int]:
try:
speed_parts = speed_str.split()
if len(speed_parts) < 2:
if len(speed_parts) < _SPEED_PARTS_MIN:
return None
speed_val = float(speed_parts[0])
unit = speed_parts[1].upper()
@@ -62,7 +72,7 @@ def _parse_speed(slot: dict) -> Optional[int]:
if prefix in unit:
return int(speed_val * mult)
return int(speed_val)
except (ValueError, IndexError):
except ValueError, IndexError:
return None
@@ -100,34 +110,36 @@ class SABnzbdClient(DownloadClient):
protocol = "usenet"
name = "sabnzbd"
def __init__(self):
def __init__(self) -> None:
"""Initialize SABnzbd client with settings from config."""
raw_url = config.get("SABNZBD_URL", "")
raw_url = config_text(config.get("SABNZBD_URL", ""))
if not raw_url:
raise ValueError("SABNZBD_URL is required")
msg = "SABNZBD_URL is required"
raise ValueError(msg)
api_key = config.get("SABNZBD_API_KEY", "")
api_key = config_text(config.get("SABNZBD_API_KEY", ""))
if not api_key:
raise ValueError("SABNZBD_API_KEY is required")
msg = "SABNZBD_API_KEY is required"
raise ValueError(msg)
self.url = normalize_http_url(raw_url)
self.url = normalize_http_config_url(raw_url)
if not self.url:
raise ValueError("SABNZBD_URL is invalid")
msg = "SABNZBD_URL is invalid"
raise ValueError(msg)
self.api_key = api_key
self._category = config.get("SABNZBD_CATEGORY", "books")
self._category = config_text(config.get("SABNZBD_CATEGORY", "books"))
@staticmethod
def is_configured() -> bool:
"""Check if SABnzbd is configured and selected as the usenet client."""
client = config.get("PROWLARR_USENET_CLIENT", "")
url = normalize_http_url(config.get("SABNZBD_URL", ""))
api_key = config.get("SABNZBD_API_KEY", "")
client = config_text(config.get("PROWLARR_USENET_CLIENT", ""))
url = normalize_http_config_url(config.get("SABNZBD_URL", ""))
api_key = config_text(config.get("SABNZBD_API_KEY", ""))
return client == "sabnzbd" and bool(url) and bool(api_key)
@with_retry()
def _api_call(self, mode: str, params: Optional[dict] = None) -> Any:
"""
Make an API call to SABnzbd.
def _api_call(self, mode: str, params: dict[str, _SabnzbdRequestParam] | None = None) -> Any:
"""Make an API call to SABnzbd.
Args:
mode: API mode (e.g., "version", "addurl", "queue", "history")
@@ -138,10 +150,11 @@ class SABnzbdClient(DownloadClient):
Raises:
Exception: If API call fails after retries.
"""
api_url = f"{self.url}/api"
request_params = {
request_params: dict[str, _SabnzbdRequestParam] = {
"apikey": self.api_key,
"mode": mode,
"output": "json",
@@ -149,7 +162,9 @@ class SABnzbdClient(DownloadClient):
if params:
request_params.update(params)
response = requests.get(api_url, params=request_params, timeout=30, verify=get_ssl_verify(api_url))
response = requests.get(
api_url, params=request_params, timeout=30, verify=get_ssl_verify(api_url)
)
response.raise_for_status()
result = response.json()
@@ -157,19 +172,22 @@ class SABnzbdClient(DownloadClient):
# Check for error in response
if isinstance(result, dict) and result.get("status") is False:
error = result.get("error", "Unknown error")
raise Exception(f"SABnzbd error: {error}")
msg = f"SABnzbd error: {error}"
raise RuntimeError(msg)
return result
def _api_post_file(self, nzb_content: bytes, filename: str, nzb_name: str, category: str) -> Any:
"""
Upload an NZB file to SABnzbd using addfile.
def _api_post_file(
self, nzb_content: bytes, filename: str, nzb_name: str, category: str
) -> Any:
"""Upload an NZB file to SABnzbd using addfile.
Returns:
JSON response from SABnzbd.
"""
api_url = f"{self.url}/api"
request_params = {
request_params: dict[str, _SabnzbdRequestParam] = {
"apikey": self.api_key,
"mode": "addfile",
"output": "json",
@@ -178,13 +196,20 @@ class SABnzbdClient(DownloadClient):
}
files = {"name": (filename, nzb_content, "application/x-nzb")}
response = requests.post(api_url, params=request_params, files=files, timeout=30, verify=get_ssl_verify(api_url))
response = requests.post(
api_url,
params=request_params,
files=files,
timeout=30,
verify=get_ssl_verify(api_url),
)
response.raise_for_status()
result = response.json()
if isinstance(result, dict) and result.get("status") is False:
error = result.get("error", "Unknown error")
raise Exception(f"SABnzbd error: {error}")
msg = f"SABnzbd error: {error}"
raise RuntimeError(msg)
return result
@@ -196,12 +221,12 @@ class SABnzbdClient(DownloadClient):
return response.content
def _get_prowlarr_headers(self, url: str) -> dict:
# TODO: Move this source-specific Prowlarr auth handling into a source hook.
# TODO(shelfmark): Move this source-specific Prowlarr auth handling into a source hook.
api_key = str(config.get("PROWLARR_API_KEY", "") or "").strip()
if not api_key:
return {}
prowlarr_url = normalize_http_url(config.get("PROWLARR_URL", ""))
prowlarr_url = normalize_http_config_url(config.get("PROWLARR_URL", ""))
if not prowlarr_url:
return {}
@@ -242,9 +267,10 @@ class SABnzbdClient(DownloadClient):
return f"{base_name}.nzb"
@staticmethod
def _extract_nzo_id(result: Any) -> str:
def _extract_nzo_id(result: object) -> str:
if not isinstance(result, dict):
raise Exception("SABnzbd returned invalid response")
msg = "SABnzbd returned invalid response"
raise TypeError(msg)
nzo_ids = result.get("nzo_ids") or result.get("nzo_id")
if isinstance(nzo_ids, list) and nzo_ids:
@@ -254,57 +280,61 @@ class SABnzbdClient(DownloadClient):
if isinstance(nzo_ids, int):
return str(nzo_ids)
raise Exception("SABnzbd returned no nzo_id")
msg = "SABnzbd returned no nzo_id"
raise RuntimeError(msg)
def test_connection(self) -> Tuple[bool, str]:
def test_connection(self) -> tuple[bool, str]:
"""Test connection to SABnzbd."""
try:
result = self._api_call("version")
version = result.get("version", "unknown")
return True, f"Connected to SABnzbd {version}"
except requests.exceptions.ConnectionError:
return False, "Could not connect to SABnzbd"
except requests.exceptions.Timeout:
return False, "Connection timed out"
except Exception as e:
return False, f"Connection failed: {str(e)}"
except _SABNZBD_CLIENT_ERRORS as e:
return False, f"Connection failed: {e!s}"
else:
return True, f"Connected to SABnzbd {version}"
def add_download(
self,
url: str,
name: str,
category: Optional[str] = None,
expected_hash: Optional[str] = None,
**kwargs,
category: str | None = None,
expected_hash: str | None = None,
**kwargs: object,
) -> str:
"""
Add NZB by URL.
"""Add NZB by URL.
Args:
url: NZB URL (can be Prowlarr proxy URL)
name: Display name for the download
category: Category for organization (uses configured default if not specified)
expected_hash: Optional info_hash hint (unused)
**kwargs: Client-specific options passed through to the implementation.
Returns:
SABnzbd nzo_id.
Raises:
Exception: If adding fails.
"""
# Use configured category if not explicitly provided
category = category or self._category
resolved_category = category or self._category
try:
logger.debug(f"Adding NZB to SABnzbd: {name}")
logger.debug("Adding NZB to SABnzbd: %s", name)
nzb_filename = self._build_nzb_filename(name, url)
nzb_content = self._fetch_nzb_content(url)
result = self._api_post_file(nzb_content, nzb_filename, name, category)
result = self._api_post_file(nzb_content, nzb_filename, name, resolved_category)
nzo_id = self._extract_nzo_id(result)
logger.info(f"Added NZB to SABnzbd: {nzo_id}")
logger.info("Added NZB to SABnzbd: %s", nzo_id)
except _SABNZBD_CLIENT_ERRORS as e:
logger.warning("SABnzbd addfile failed, falling back to addurl: %s", e)
else:
return nzo_id
except Exception as e:
logger.warning(f"SABnzbd addfile failed, falling back to addurl: {e}")
try:
result = self._api_call(
@@ -312,25 +342,26 @@ class SABnzbdClient(DownloadClient):
{
"name": url,
"nzbname": name,
"cat": category,
"cat": resolved_category,
},
)
nzo_id = self._extract_nzo_id(result)
logger.info(f"Added NZB to SABnzbd via addurl: {nzo_id}")
return nzo_id
except Exception as e:
logger.error(f"SABnzbd add failed: {e}")
logger.info("Added NZB to SABnzbd via addurl: %s", nzo_id)
except _SABNZBD_CLIENT_ERRORS:
logger.exception("SABnzbd add failed")
raise
else:
return nzo_id
def get_status(self, download_id: str) -> DownloadStatus:
"""
Get NZB status by nzo_id.
"""Get NZB status by nzo_id.
Args:
download_id: SABnzbd nzo_id
Returns:
Current download status.
"""
try:
# Check active queue first
@@ -383,7 +414,12 @@ class SABnzbdClient(DownloadClient):
storage = slot.get("storage", "")
if storage is None:
storage = ""
logger.debug(f"SABnzbd history: {download_id} status={status_text} storage='{storage}'")
logger.debug(
"SABnzbd history: %s status=%s storage='%s'",
download_id,
status_text,
storage,
)
if status_text == "COMPLETED":
title = slot.get("name") or slot.get("nzb_name") or ""
@@ -396,7 +432,7 @@ class SABnzbdClient(DownloadClient):
complete=True,
file_path=resolved_storage,
)
elif status_text == "FAILED":
if status_text == "FAILED":
fail_message = slot.get("fail_message", "Download failed")
title = slot.get("name") or slot.get("nzb_name") or ""
resolved_storage = self._resolve_completed_storage_path(storage, title)
@@ -407,27 +443,25 @@ class SABnzbdClient(DownloadClient):
complete=True,
file_path=resolved_storage,
)
else:
# Post-processing states: Queued, QuickCheck, Verifying,
# Repairing, Fetching, Extracting, Moving, Running
# Keep polling - not yet complete
return DownloadStatus(
progress=100,
state="processing",
message=status_text.title(),
complete=False,
file_path=None,
)
# Post-processing states: Queued, QuickCheck, Verifying,
# Repairing, Fetching, Extracting, Moving, Running
# Keep polling - not yet complete
return DownloadStatus(
progress=100,
state="processing",
message=status_text.title(),
complete=False,
file_path=None,
)
# Not found
logger.warning(f"SABnzbd: download {download_id} not found in queue or history")
logger.warning("SABnzbd: download %s not found in queue or history", download_id)
return DownloadStatus.error("Download not found")
except Exception as e:
except _SABNZBD_CLIENT_ERRORS as e:
return DownloadStatus.error(self._log_error("get_status", e))
def remove(self, download_id: str, delete_files: bool = False, archive: bool = True) -> bool:
"""
Remove a download from SABnzbd.
def remove(self, download_id: str, *, delete_files: bool = False, archive: bool = True) -> bool:
"""Remove a download from SABnzbd.
Args:
download_id: SABnzbd nzo_id
@@ -436,6 +470,7 @@ class SABnzbdClient(DownloadClient):
Returns:
True if successful.
"""
# First try to remove from queue. If it isn't there (common for completed jobs),
# fall back to history removal instead of failing fast on a SABnzbd error response.
@@ -450,10 +485,10 @@ class SABnzbdClient(DownloadClient):
)
if result.get("status"):
logger.info(f"Removed NZB from SABnzbd queue: {download_id}")
logger.info("Removed NZB from SABnzbd queue: %s", download_id)
return True
except Exception as e:
logger.debug(f"SABnzbd queue delete skipped for {download_id}: {e}")
except _SABNZBD_CLIENT_ERRORS as e:
logger.debug("SABnzbd queue delete skipped for %s: %s", download_id, e)
# If not in queue (or queue delete failed), try to remove from history.
try:
@@ -469,32 +504,31 @@ class SABnzbdClient(DownloadClient):
if result.get("status"):
action = "archived" if archive else "removed"
logger.info(f"NZB {action} from SABnzbd history: {download_id}")
logger.info("NZB %s from SABnzbd history: %s", action, download_id)
return True
except Exception as e:
except _SABNZBD_CLIENT_ERRORS as e:
self._log_error("remove", e)
return False
return False
def get_download_path(self, download_id: str) -> Optional[str]:
"""
Get the path where NZB files are located.
def get_download_path(self, download_id: str) -> str | None:
"""Get the path where NZB files are located.
Args:
download_id: SABnzbd nzo_id
Returns:
Storage directory, or None.
"""
status = self.get_status(download_id)
return status.file_path
def find_existing(
self, url: str, category: Optional[str] = None
) -> Optional[Tuple[str, DownloadStatus]]:
"""
Check if an NZB for this URL already exists in SABnzbd.
self, url: str, category: str | None = None
) -> tuple[str, DownloadStatus] | None:
"""Check if an NZB for this URL already exists in SABnzbd.
Note: Unlike torrents which have a unique info_hash, usenet NZBs don't have
a universal unique identifier. SABnzbd generates an nzo_id when adding,
@@ -507,23 +541,22 @@ class SABnzbdClient(DownloadClient):
Returns:
Tuple of (nzo_id, status) if found, None if not found.
"""
try:
# Extract NZB name from URL (last path component without extension)
from urllib.parse import unquote, urlparse
parsed = urlparse(url)
path = unquote(parsed.path)
# Get filename from path
if "/" in path:
filename = path.rsplit("/", 1)[-1]
else:
filename = path
filename = path.rsplit("/", 1)[-1] if "/" in path else path
# Remove common NZB extensions
for ext in [".nzb", ".nzb.gz"]:
if filename.lower().endswith(ext):
filename = filename[:-len(ext)]
filename = filename[: -len(ext)]
break
if not filename:
@@ -543,7 +576,7 @@ class SABnzbdClient(DownloadClient):
nzo_id = slot.get("nzo_id")
if nzo_id:
status = self.get_status(nzo_id)
logger.debug(f"Found existing NZB in SABnzbd queue: {nzo_id}")
logger.debug("Found existing NZB in SABnzbd queue: %s", nzo_id)
return (nzo_id, status)
# Search history (SABnzbd uses "category" field in history)
@@ -557,11 +590,11 @@ class SABnzbdClient(DownloadClient):
nzo_id = slot.get("nzo_id")
if nzo_id:
status = self.get_status(nzo_id)
logger.debug(f"Found existing NZB in SABnzbd history: {nzo_id}")
logger.debug("Found existing NZB in SABnzbd history: %s", nzo_id)
return (nzo_id, status)
except _SABNZBD_CLIENT_ERRORS as e:
logger.debug("Error checking for existing NZB: %s", e)
return None
except Exception as e:
logger.debug(f"Error checking for existing NZB: {e}")
else:
return None
+244 -86
View File
@@ -1,25 +1,124 @@
"""Shared download client settings registration."""
from contextlib import contextmanager
from typing import Any, Dict, Optional
from __future__ import annotations
import importlib
from contextlib import contextmanager, suppress
from typing import TYPE_CHECKING, Any, NoReturn, Protocol, TypeGuard
from shelfmark.core.settings_registry import (
register_settings,
HeadingField,
TextField,
PasswordField,
ActionButton,
HeadingField,
PasswordField,
SelectField,
SettingsField,
TagListField,
TextField,
register_settings,
)
from shelfmark.core.utils import normalize_http_url, get_hardened_xmlrpc_client
from shelfmark.core.utils import get_hardened_xmlrpc_client, normalize_http_url
from shelfmark.download.network import get_ssl_verify
try:
import qbittorrentapi as _qbittorrentapi
except ImportError:
_ImportedQBittorrentApiError = RuntimeError
_ImportedQBittorrentLoginFailed = RuntimeError
else:
_ImportedQBittorrentApiError = getattr(_qbittorrentapi, "APIError", RuntimeError)
_ImportedQBittorrentLoginFailed = getattr(_qbittorrentapi, "LoginFailed", RuntimeError)
try:
from transmission_rpc import TransmissionError as _ImportedTransmissionError
except ImportError:
_ImportedTransmissionError = RuntimeError
if TYPE_CHECKING:
from collections.abc import Callable, Iterator
# ==================== Test Connection Callbacks ====================
_DELUGE_HOST_ENTRY_MIN_LENGTH = 2
class _SessionWithVerify(Protocol):
verify: bool
class _RequestsModuleWithSession(Protocol):
Session: Callable[..., _SessionWithVerify]
class _TransmissionClientWithProtocol(Protocol):
protocol: str
def _resolve_exception_type(candidate: object) -> type[Exception]:
if isinstance(candidate, type) and issubclass(candidate, Exception):
return candidate
return RuntimeError
_QBittorrentApiError = _resolve_exception_type(_ImportedQBittorrentApiError)
_QBittorrentLoginFailed = _resolve_exception_type(_ImportedQBittorrentLoginFailed)
_TransmissionError = _resolve_exception_type(_ImportedTransmissionError)
_QBITTORRENT_SETTINGS_ERRORS = (
_QBittorrentLoginFailed,
_QBittorrentApiError,
AttributeError,
OSError,
RuntimeError,
TypeError,
ValueError,
)
_TRANSMISSION_SETTINGS_ERRORS = (
_TransmissionError,
AttributeError,
OSError,
RuntimeError,
TypeError,
ValueError,
)
def _raise_runtime_error(message: str) -> NoReturn:
raise RuntimeError(message)
def _is_requests_module_with_session(candidate: object) -> TypeGuard[_RequestsModuleWithSession]:
return callable(getattr(candidate, "Session", None))
def _has_protocol_attr(candidate: object) -> TypeGuard[_TransmissionClientWithProtocol]:
return hasattr(candidate, "protocol")
def _set_transmission_protocol_if_supported(client: object, protocol: str) -> None:
if protocol != "https" or not _has_protocol_attr(client):
return
with suppress(AttributeError, OSError, RuntimeError, TypeError, ValueError):
client.protocol = protocol
def _resolve_string_setting(
current_values: dict[str, Any],
config_get: Callable[[str, str], object],
key: str,
*,
default: str = "",
) -> str:
current_value = current_values.get(key)
if isinstance(current_value, str) and current_value:
return current_value
config_value = config_get(key, default)
if isinstance(config_value, str) and config_value:
return config_value
return default
@contextmanager
def _transmission_session_verify_override(url: str):
def _transmission_session_verify_override(url: str) -> Iterator[None]:
"""Ensure transmission-rpc constructor uses the configured TLS verify mode."""
verify = get_ssl_verify(url)
if verify:
@@ -27,34 +126,39 @@ def _transmission_session_verify_override(url: str):
return
try:
import transmission_rpc.client as transmission_rpc_client
except Exception:
transmission_rpc_client = importlib.import_module("transmission_rpc.client")
except ImportError:
yield
return
original_session_factory = transmission_rpc_client.requests.Session
requests_module = getattr(transmission_rpc_client, "requests", None)
if not _is_requests_module_with_session(requests_module):
yield
return
original_session_factory = requests_module.Session
def _session_factory(*args: Any, **kwargs: Any) -> Any:
session = original_session_factory(*args, **kwargs)
session.verify = False
return session
transmission_rpc_client.requests.Session = _session_factory
requests_module.Session = _session_factory
try:
yield
finally:
transmission_rpc_client.requests.Session = original_session_factory
requests_module.Session = original_session_factory
def _test_qbittorrent_connection(current_values: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:
def _test_qbittorrent_connection(current_values: dict[str, Any] | None = None) -> dict[str, Any]:
"""Test the qBittorrent connection using current form values."""
from shelfmark.core.config import config
current_values = current_values or {}
raw_url = current_values.get("QBITTORRENT_URL") or config.get("QBITTORRENT_URL", "")
username = current_values.get("QBITTORRENT_USERNAME") or config.get("QBITTORRENT_USERNAME", "")
password = current_values.get("QBITTORRENT_PASSWORD") or config.get("QBITTORRENT_PASSWORD", "")
raw_url = _resolve_string_setting(current_values, config.get, "QBITTORRENT_URL")
username = _resolve_string_setting(current_values, config.get, "QBITTORRENT_USERNAME")
password = _resolve_string_setting(current_values, config.get, "QBITTORRENT_PASSWORD")
if not raw_url:
return {"success": False, "message": "qBittorrent URL is required"}
@@ -66,17 +170,23 @@ def _test_qbittorrent_connection(current_values: Optional[Dict[str, Any]] = None
if not url:
return {"success": False, "message": "qBittorrent URL is invalid"}
client = Client(host=url, username=username, password=password, VERIFY_WEBUI_CERTIFICATE=get_ssl_verify(url))
client = Client(
host=url,
username=username,
password=password,
VERIFY_WEBUI_CERTIFICATE=get_ssl_verify(url),
)
client.auth_log_in()
api_version = client.app.web_api_version
return {"success": True, "message": f"Connected to qBittorrent (API v{api_version})"}
except ImportError:
return {"success": False, "message": "qbittorrent-api package not installed"}
except Exception as e:
return {"success": False, "message": f"Connection failed: {str(e)}"}
except _QBITTORRENT_SETTINGS_ERRORS as e:
return {"success": False, "message": f"Connection failed: {e!s}"}
else:
return {"success": True, "message": f"Connected to qBittorrent (API v{api_version})"}
def _test_transmission_connection(current_values: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:
def _test_transmission_connection(current_values: dict[str, Any] | None = None) -> dict[str, Any]:
"""Test the Transmission connection using current form values."""
from shelfmark.core.config import config
from shelfmark.download.clients.torrent_utils import (
@@ -85,9 +195,9 @@ def _test_transmission_connection(current_values: Optional[Dict[str, Any]] = Non
current_values = current_values or {}
raw_url = current_values.get("TRANSMISSION_URL") or config.get("TRANSMISSION_URL", "")
username = current_values.get("TRANSMISSION_USERNAME") or config.get("TRANSMISSION_USERNAME", "")
password = current_values.get("TRANSMISSION_PASSWORD") or config.get("TRANSMISSION_PASSWORD", "")
raw_url = _resolve_string_setting(current_values, config.get, "TRANSMISSION_URL")
username = _resolve_string_setting(current_values, config.get, "TRANSMISSION_USERNAME")
password = _resolve_string_setting(current_values, config.get, "TRANSMISSION_PASSWORD")
if not raw_url:
return {"success": False, "message": "Transmission URL is required"}
@@ -106,8 +216,8 @@ def _test_transmission_connection(current_values: Optional[Dict[str, Any]] = Non
"host": host,
"port": port,
"path": path,
"username": username if username else None,
"password": password if password else None,
"username": username or None,
"password": password or None,
"protocol": protocol,
}
try:
@@ -119,11 +229,7 @@ def _test_transmission_connection(current_values: Optional[Dict[str, Any]] = Non
client_kwargs.pop("protocol", None)
with _transmission_session_verify_override(url):
client = Client(**client_kwargs)
if protocol == "https" and hasattr(client, "protocol"):
try:
setattr(client, "protocol", protocol)
except Exception:
pass
_set_transmission_protocol_if_supported(client, protocol)
# Keep session verify aligned for subsequent calls beyond constructor bootstrap.
http_session = getattr(client, "_http_session", None)
@@ -132,25 +238,29 @@ def _test_transmission_connection(current_values: Optional[Dict[str, Any]] = Non
session = client.get_session()
version = session.version
return {"success": True, "message": f"Connected to Transmission {version}"}
except ImportError:
return {"success": False, "message": "transmission-rpc package not installed"}
except Exception as e:
return {"success": False, "message": f"Connection failed: {str(e)}"}
except _TRANSMISSION_SETTINGS_ERRORS as e:
return {"success": False, "message": f"Connection failed: {e!s}"}
else:
return {"success": True, "message": f"Connected to Transmission {version}"}
def _test_deluge_connection(current_values: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:
def _test_deluge_connection(current_values: dict[str, Any] | None = None) -> dict[str, Any]:
"""Test Deluge Web UI JSON-RPC connection using current form values."""
from urllib.parse import urlparse
import requests
from shelfmark.core.config import config
current_values = current_values or {}
raw_host = current_values.get("DELUGE_HOST") or config.get("DELUGE_HOST", "localhost")
raw_port = current_values.get("DELUGE_PORT") or config.get("DELUGE_PORT", "8112")
password = current_values.get("DELUGE_PASSWORD") or config.get("DELUGE_PASSWORD", "")
raw_host = _resolve_string_setting(
current_values, config.get, "DELUGE_HOST", default="localhost"
)
raw_port = _resolve_string_setting(current_values, config.get, "DELUGE_PORT", default="8112")
password = _resolve_string_setting(current_values, config.get, "DELUGE_PASSWORD")
if not raw_host:
return {"success": False, "message": "Deluge host is required"}
@@ -177,13 +287,12 @@ def _test_deluge_connection(current_values: Optional[Dict[str, Any]] = None) ->
if parsed.port is not None:
port = parsed.port
base_path = (parsed.path or "").rstrip("/")
else:
# Allow "host:port" in DELUGE_HOST for convenience.
if ":" in raw_host and raw_host.count(":") == 1:
host_part, port_part = raw_host.split(":", 1)
if host_part and port_part.isdigit():
host = host_part
port = int(port_part)
# Allow "host:port" in DELUGE_HOST for convenience.
elif ":" in raw_host and raw_host.count(":") == 1:
host_part, port_part = raw_host.split(":", 1)
if host_part and port_part.isdigit():
host = host_part
port = int(port_part)
rpc_url = f"{scheme}://{host}:{port}{base_path}/json"
@@ -195,8 +304,8 @@ def _test_deluge_connection(current_values: Optional[Dict[str, Any]] = None) ->
if data.get("error"):
error = data["error"]
if isinstance(error, dict):
raise Exception(error.get("message") or str(error))
raise Exception(str(error))
raise RuntimeError(error.get("message") or str(error))
raise RuntimeError(str(error))
return data.get("result")
def get_daemon_version(session: requests.Session, rpc_id: int) -> Any:
@@ -204,7 +313,7 @@ def _test_deluge_connection(current_values: Optional[Dict[str, Any]] = None) ->
methods = rpc_call(session, rpc_id, "system.listMethods")
if isinstance(methods, list) and "daemon.get_version" in methods:
return rpc_call(session, rpc_id + 1, "daemon.get_version")
except Exception:
except requests.exceptions.RequestException, RuntimeError, ValueError, TypeError:
# Fall back to daemon.info to preserve existing behavior.
pass
@@ -226,7 +335,11 @@ def _test_deluge_connection(current_values: Optional[Dict[str, Any]] = None) ->
host_id = hosts[0][0]
for entry in hosts:
if isinstance(entry, list) and len(entry) >= 2 and entry[1] in {"127.0.0.1", "localhost"}:
if (
isinstance(entry, list)
and len(entry) >= _DELUGE_HOST_ENTRY_MIN_LENGTH
and entry[1] in {"127.0.0.1", "localhost"}
):
host_id = entry[0]
break
@@ -239,27 +352,36 @@ def _test_deluge_connection(current_values: Optional[Dict[str, Any]] = None) ->
}
version = get_daemon_version(session, 6)
return {"success": True, "message": f"Connected to Deluge {version}"}
except requests.exceptions.ConnectionError:
return {"success": False, "message": "Could not connect to Deluge Web UI"}
except requests.exceptions.Timeout:
return {"success": False, "message": "Connection timed out"}
except Exception as e:
return {"success": False, "message": f"Connection failed: {str(e)}"}
except (
requests.exceptions.RequestException,
RuntimeError,
ValueError,
TypeError,
KeyError,
IndexError,
AttributeError,
) as e:
return {"success": False, "message": f"Connection failed: {e!s}"}
else:
return {"success": True, "message": f"Connected to Deluge {version}"}
def _test_rtorrent_connection(current_values: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:
def _test_rtorrent_connection(current_values: dict[str, Any] | None = None) -> dict[str, Any]:
"""Test the rTorrent connection using current form values."""
from shelfmark.core.config import config
import ssl
from urllib.parse import urlparse
from shelfmark.core.config import config
current_values = current_values or {}
raw_url = current_values.get("RTORRENT_URL") or config.get("RTORRENT_URL", "")
username = current_values.get("RTORRENT_USERNAME") or config.get("RTORRENT_USERNAME", "")
password = current_values.get("RTORRENT_PASSWORD") or config.get("RTORRENT_PASSWORD", "")
raw_url = _resolve_string_setting(current_values, config.get, "RTORRENT_URL")
username = _resolve_string_setting(current_values, config.get, "RTORRENT_USERNAME")
password = _resolve_string_setting(current_values, config.get, "RTORRENT_PASSWORD")
if not raw_url:
return {"success": False, "message": "rTorrent URL is required"}
@@ -270,7 +392,10 @@ def _test_rtorrent_connection(current_values: Optional[Dict[str, Any]] = None) -
try:
xmlrpc_client = get_hardened_xmlrpc_client()
except (RuntimeError, OSError, ValueError, TypeError) as e:
return {"success": False, "message": f"Connection failed: {e!s}"}
try:
# Add HTTP auth to URL if credentials provided
if username and password:
parsed = urlparse(url)
@@ -290,21 +415,29 @@ def _test_rtorrent_connection(current_values: Optional[Dict[str, Any]] = None) -
rpc = xmlrpc_client.ServerProxy(rpc_url)
version = rpc.system.client_version()
except (xmlrpc_client.Error, RuntimeError, OSError, ValueError, TypeError) as e:
return {"success": False, "message": f"Connection failed: {e!s}"}
else:
return {"success": True, "message": f"Connected to rTorrent {version}"}
except Exception as e:
return {"success": False, "message": f"Connection failed: {str(e)}"}
def _test_nzbget_connection(current_values: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:
def _test_nzbget_connection(current_values: dict[str, Any] | None = None) -> dict[str, Any]:
"""Test the NZBGet connection using current form values."""
import requests
from shelfmark.core.config import config
current_values = current_values or {}
raw_url = current_values.get("NZBGET_URL") or config.get("NZBGET_URL", "")
username = current_values.get("NZBGET_USERNAME") or config.get("NZBGET_USERNAME", "nzbget")
password = current_values.get("NZBGET_PASSWORD") or config.get("NZBGET_PASSWORD", "")
raw_url = _resolve_string_setting(current_values, config.get, "NZBGET_URL")
username = _resolve_string_setting(
current_values,
config.get,
"NZBGET_USERNAME",
default="nzbget",
)
password = _resolve_string_setting(current_values, config.get, "NZBGET_PASSWORD")
if not raw_url:
return {"success": False, "message": "NZBGet URL is required"}
@@ -316,30 +449,44 @@ def _test_nzbget_connection(current_values: Optional[Dict[str, Any]] = None) ->
try:
rpc_url = f"{url.rstrip('/')}/jsonrpc"
payload = {"jsonrpc": "2.0", "method": "status", "params": [], "id": 1}
response = requests.post(rpc_url, json=payload, auth=(username, password), timeout=30, verify=get_ssl_verify(rpc_url))
response = requests.post(
rpc_url,
json=payload,
auth=(username, password),
timeout=30,
verify=get_ssl_verify(rpc_url),
)
response.raise_for_status()
result = response.json()
if "error" in result and result["error"]:
raise Exception(result["error"].get("message", "RPC error"))
if result.get("error"):
_raise_runtime_error(result["error"].get("message", "RPC error"))
version = result.get("result", {}).get("Version", "unknown")
return {"success": True, "message": f"Connected to NZBGet {version}"}
except requests.exceptions.ConnectionError:
return {"success": False, "message": "Could not connect to NZBGet"}
except requests.exceptions.Timeout:
return {"success": False, "message": "Connection timed out"}
except Exception as e:
return {"success": False, "message": f"Connection failed: {str(e)}"}
except (
requests.exceptions.RequestException,
RuntimeError,
ValueError,
AttributeError,
TypeError,
) as e:
return {"success": False, "message": f"Connection failed: {e!s}"}
else:
return {"success": True, "message": f"Connected to NZBGet {version}"}
def _test_sabnzbd_connection(current_values: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:
def _test_sabnzbd_connection(current_values: dict[str, Any] | None = None) -> dict[str, Any]:
"""Test the SABnzbd connection using current form values."""
import requests
from shelfmark.core.config import config
current_values = current_values or {}
raw_url = current_values.get("SABNZBD_URL") or config.get("SABNZBD_URL", "")
api_key = current_values.get("SABNZBD_API_KEY") or config.get("SABNZBD_API_KEY", "")
raw_url = _resolve_string_setting(current_values, config.get, "SABNZBD_URL")
api_key = _resolve_string_setting(current_values, config.get, "SABNZBD_API_KEY")
if not raw_url:
return {"success": False, "message": "SABnzbd URL is required"}
@@ -357,24 +504,32 @@ def _test_sabnzbd_connection(current_values: Optional[Dict[str, Any]] = None) ->
response.raise_for_status()
result = response.json()
version = result.get("version", "unknown")
return {"success": True, "message": f"Connected to SABnzbd {version}"}
except requests.exceptions.ConnectionError:
return {"success": False, "message": "Could not connect to SABnzbd"}
except requests.exceptions.Timeout:
return {"success": False, "message": "Connection timed out"}
except Exception as e:
return {"success": False, "message": f"Connection failed: {str(e)}"}
except (
requests.exceptions.RequestException,
RuntimeError,
ValueError,
AttributeError,
TypeError,
) as e:
return {"success": False, "message": f"Connection failed: {e!s}"}
else:
return {"success": True, "message": f"Connected to SABnzbd {version}"}
# ==================== Download Clients Tab ====================
@register_settings(
name="prowlarr_clients",
display_name="Download Clients",
icon="cog",
order=110,
)
def prowlarr_clients_settings():
def prowlarr_clients_settings() -> list[SettingsField]:
"""Download client settings shared by external release sources."""
return [
# --- Torrent Client Selection ---
@@ -396,7 +551,6 @@ def prowlarr_clients_settings():
],
default="",
),
# --- qBittorrent Settings ---
TextField(
key="QBITTORRENT_URL",
@@ -458,7 +612,6 @@ def prowlarr_clients_settings():
normalize_urls=False,
show_when={"field": "PROWLARR_TORRENT_CLIENT", "value": "qbittorrent"},
),
# --- Transmission Settings ---
TextField(
key="TRANSMISSION_URL",
@@ -510,7 +663,6 @@ def prowlarr_clients_settings():
placeholder="/downloads",
show_when={"field": "PROWLARR_TORRENT_CLIENT", "value": "transmission"},
),
# --- Deluge Settings ---
TextField(
key="DELUGE_HOST",
@@ -565,7 +717,6 @@ def prowlarr_clients_settings():
placeholder="/downloads",
show_when={"field": "PROWLARR_TORRENT_CLIENT", "value": "deluge"},
),
# --- rTorrent Settings ---
TextField(
key="RTORRENT_URL",
@@ -610,8 +761,17 @@ def prowlarr_clients_settings():
show_when={"field": "PROWLARR_TORRENT_CLIENT", "value": "rtorrent"},
),
# Note: Torrent client download path must be mounted identically in both containers.
# Torrents are always copied (not moved) to preserve seeding capability.
SelectField(
key="PROWLARR_TORRENT_ACTION",
label="Torrent Completion Action",
description="Remove deletes the torrent from your client immediately after import (stops seeding, files are kept); Keep leaves it in the client to continue seeding",
options=[
{"value": "keep", "label": "Keep"},
{"value": "remove", "label": "Remove"},
],
default="keep",
show_when={"field": "PROWLARR_TORRENT_CLIENT", "notEmpty": True},
),
# --- Usenet Client Selection ---
HeadingField(
key="usenet_heading",
@@ -629,7 +789,6 @@ def prowlarr_clients_settings():
],
default="",
),
# --- NZBGet Settings ---
TextField(
key="NZBGET_URL",
@@ -676,7 +835,6 @@ def prowlarr_clients_settings():
default="",
show_when={"field": "PROWLARR_USENET_CLIENT", "value": "nzbget"},
),
# --- SABnzbd Settings ---
TextField(
key="SABNZBD_URL",
+129 -84
View File
@@ -1,10 +1,12 @@
"""Shared utilities for torrent clients."""
from __future__ import annotations
import base64
import hashlib
import re
from binascii import Error as BinasciiError
from dataclasses import dataclass
from typing import Optional, Tuple
from urllib.parse import parse_qs, urljoin, urlparse
import requests
@@ -15,24 +17,42 @@ from shelfmark.download.network import get_ssl_verify
logger = setup_logger(__name__)
_MAGNET_RESPONSE_MAX_BYTES = 2000
_BASE32_BTMH_TAG_BYTES = 34
_BTIH_INFO_BYTE_HEX = 0x20
_BTIH_PREFIX_BYTE = 0x12
_BTIH_DIGEST_LENGTH = 32
_BTIH_HASH_LENGTH_40 = 40
_BTIH_HASH_LENGTH_32 = 32
_TORRENT_FETCH_ERRORS = (
requests.exceptions.RequestException,
OSError,
RuntimeError,
TypeError,
ValueError,
)
_TORRENT_PARSE_ERRORS = (IndexError, KeyError, TypeError, ValueError)
type BencodeValue = dict[str | bytes, BencodeValue] | list[BencodeValue] | int | bytes | str
@dataclass
class TorrentInfo:
"""Parsed information from a torrent URL."""
info_hash: Optional[str]
info_hash: str | None
"""Lowercase hex info_hash (32 or 40 chars), or None if extraction failed."""
torrent_data: Optional[bytes]
torrent_data: bytes | None
"""Raw .torrent file content, only populated for .torrent URLs."""
is_magnet: bool
"""True if the URL was a magnet link."""
magnet_url: Optional[str] = None
magnet_url: str | None = None
"""The actual magnet URL, if available."""
def with_info_hash(self, info_hash: Optional[str]) -> "TorrentInfo":
def with_info_hash(self, info_hash: str | None) -> TorrentInfo:
"""Return a copy with the info_hash replaced when provided."""
if info_hash:
return TorrentInfo(
@@ -46,8 +66,9 @@ class TorrentInfo:
def extract_torrent_info(
url: str,
*,
fetch_torrent: bool = True,
expected_hash: Optional[str] = None,
expected_hash: str | None = None,
) -> TorrentInfo:
"""Extract info_hash from magnet link or .torrent URL.
@@ -58,6 +79,7 @@ def extract_torrent_info(
Redirects to magnet links are handled explicitly so we can extract a
hash from the magnet when available.
"""
is_magnet = url.startswith("magnet:")
@@ -73,7 +95,7 @@ def extract_torrent_info(
return TorrentInfo(info_hash=expected_hash, torrent_data=None, is_magnet=False)
headers: dict[str, str] = {"Accept": "application/x-bittorrent"}
# TODO: Move this source-specific Prowlarr auth handling into a source hook.
# TODO(shelfmark): Move this source-specific Prowlarr auth handling into a source hook.
api_key = str(config.get("PROWLARR_API_KEY", "") or "").strip()
if api_key:
headers["X-Api-Key"] = api_key
@@ -85,11 +107,17 @@ def extract_torrent_info(
return urljoin(current, location)
try:
logger.debug(f"Fetching torrent file from: {url[:80]}...")
logger.debug("Fetching torrent file from: %s...", url[:80])
# Use allow_redirects=False to handle magnet link redirects manually
# Some indexers redirect download URLs to magnet links
resp = requests.get(url, timeout=30, allow_redirects=False, headers=headers, verify=get_ssl_verify(url))
resp = requests.get(
url,
timeout=30,
allow_redirects=False,
headers=headers,
verify=get_ssl_verify(url),
)
# Check if this is a redirect to a magnet link
if resp.status_code in (301, 302, 303, 307, 308):
@@ -100,43 +128,51 @@ def extract_torrent_info(
if not info_hash and expected_hash:
info_hash = expected_hash
return TorrentInfo(
info_hash=info_hash, torrent_data=None, is_magnet=True, magnet_url=redirect_url
info_hash=info_hash,
torrent_data=None,
is_magnet=True,
magnet_url=redirect_url,
)
# Not a magnet redirect, follow it manually
logger.debug(f"Following redirect to: {redirect_url[:80]}...")
resp = requests.get(redirect_url, timeout=30, headers=headers, verify=get_ssl_verify(redirect_url))
logger.debug("Following redirect to: %s...", redirect_url[:80])
resp = requests.get(
redirect_url,
timeout=30,
headers=headers,
verify=get_ssl_verify(redirect_url),
)
resp.raise_for_status()
torrent_data = resp.content
# Check if response is actually a magnet link (text response)
# Some indexers return magnet links as plain text instead of redirecting
if len(torrent_data) < 2000: # Magnet links are typically short
try:
text_content = torrent_data.decode("utf-8", errors="ignore").strip()
if text_content.startswith("magnet:"):
logger.debug("Download URL returned magnet link as response body")
info_hash = extract_hash_from_magnet(text_content)
if not info_hash and expected_hash:
info_hash = expected_hash
return TorrentInfo(
info_hash=info_hash, torrent_data=None, is_magnet=True, magnet_url=text_content
)
except Exception:
pass # Not text, continue with torrent parsing
if len(torrent_data) < _MAGNET_RESPONSE_MAX_BYTES: # Magnet links are typically short
text_content = torrent_data.decode("utf-8", errors="ignore").strip()
if text_content.startswith("magnet:"):
logger.debug("Download URL returned magnet link as response body")
info_hash = extract_hash_from_magnet(text_content)
if not info_hash and expected_hash:
info_hash = expected_hash
return TorrentInfo(
info_hash=info_hash,
torrent_data=None,
is_magnet=True,
magnet_url=text_content,
)
info_hash = extract_info_hash_from_torrent(torrent_data) or expected_hash
if info_hash:
logger.debug(f"Extracted hash from torrent file: {info_hash}")
logger.debug("Extracted hash from torrent file: %s", info_hash)
else:
logger.warning("Could not extract hash from torrent file")
return TorrentInfo(info_hash=info_hash, torrent_data=torrent_data, is_magnet=False)
except Exception as e:
logger.debug(f"Could not fetch torrent file: {e}")
except _TORRENT_FETCH_ERRORS as e:
logger.debug("Could not fetch torrent file: %s", e)
return TorrentInfo(info_hash=expected_hash, torrent_data=None, is_magnet=False)
def parse_transmission_url(url: str) -> Tuple[str, str, int, str]:
def parse_transmission_url(url: str) -> tuple[str, str, int, str]:
"""Parse Transmission URL into (protocol, host, port, path)."""
parsed = urlparse(url)
protocol = (parsed.scheme or "http").lower()
@@ -155,89 +191,90 @@ def parse_transmission_url(url: str) -> Tuple[str, str, int, str]:
def bencode_decode(data: bytes) -> tuple:
"""Decode bencoded data. Returns (value, remaining_bytes)."""
if data[0:1] == b'd':
if data[0:1] == b"d":
# Dictionary
result = {}
data = data[1:]
while data[0:1] != b'e':
while data[0:1] != b"e":
key, data = bencode_decode(data)
value, data = bencode_decode(data)
result[key] = value
return result, data[1:]
elif data[0:1] == b'l':
if data[0:1] == b"l":
# List
result = []
data = data[1:]
while data[0:1] != b'e':
while data[0:1] != b"e":
value, data = bencode_decode(data)
result.append(value)
return result, data[1:]
elif data[0:1] == b'i':
if data[0:1] == b"i":
# Integer
end = data.index(b'e')
return int(data[1:end]), data[end + 1:]
elif data[0:1].isdigit():
end = data.index(b"e")
return int(data[1:end]), data[end + 1 :]
if data[0:1].isdigit():
# Byte string
colon = data.index(b':')
colon = data.index(b":")
length = int(data[:colon])
start = colon + 1
return data[start:start + length], data[start + length:]
else:
first_byte = data[0:1]
raise ValueError(
f"Invalid bencode data: expected 'd', 'l', 'i', or digit, "
f"got {first_byte!r}. First 20 bytes: {data[:20]!r}"
)
return data[start : start + length], data[start + length :]
first_byte = data[0:1]
msg = (
f"Invalid bencode data: expected 'd', 'l', 'i', or digit, "
f"got {first_byte!r}. First 20 bytes: {data[:20]!r}"
)
raise ValueError(msg)
def bencode_encode(data) -> bytes:
def bencode_encode(data: BencodeValue) -> bytes:
"""Encode data to bencode format."""
if isinstance(data, dict):
# Keys must be sorted (bencode spec requirement)
result = b'd'
result = b"d"
for key in sorted(data.keys()):
result += bencode_encode(key)
result += bencode_encode(data[key])
result += b'e'
result += b"e"
return result
elif isinstance(data, list):
result = b'l'
if isinstance(data, list):
result = b"l"
for item in data:
result += bencode_encode(item)
result += b'e'
result += b"e"
return result
elif isinstance(data, int):
return f'i{data}e'.encode()
elif isinstance(data, bytes):
return f'{len(data)}:'.encode() + data
elif isinstance(data, str):
encoded = data.encode('utf-8')
return f'{len(encoded)}:'.encode() + encoded
else:
raise ValueError(
f"Cannot bencode type {type(data).__name__}: "
f"expected dict, list, int, bytes, or str. Value: {data!r}"
)
if isinstance(data, int):
return f"i{data}e".encode()
if isinstance(data, bytes):
return f"{len(data)}:".encode() + data
if isinstance(data, str):
encoded = data.encode("utf-8")
return f"{len(encoded)}:".encode() + encoded
msg = (
f"Cannot bencode type {type(data).__name__}: "
f"expected dict, list, int, bytes, or str. Value: {data!r}"
)
raise ValueError(msg)
def extract_info_hash_from_torrent(torrent_data: bytes) -> Optional[str]:
def extract_info_hash_from_torrent(torrent_data: bytes) -> str | None:
"""Extract info_hash from .torrent file data."""
try:
decoded, _ = bencode_decode(torrent_data)
if b'info' not in decoded:
if b"info" not in decoded:
return None
info_bencoded = bencode_encode(decoded[b'info'])
info_dict = decoded[b'info']
if isinstance(info_dict, dict) and b'pieces' in info_dict:
return hashlib.sha1(info_bencoded).hexdigest().lower()
info_bencoded = bencode_encode(decoded[b"info"])
info_dict = decoded[b"info"]
if isinstance(info_dict, dict) and b"pieces" in info_dict:
# BitTorrent v1 info hashes are defined as SHA-1.
return hashlib.sha1(info_bencoded).hexdigest().lower() # noqa: S324
return hashlib.sha256(info_bencoded).hexdigest().lower()
except Exception as e:
logger.debug(f"Failed to parse torrent file: {e}")
except _TORRENT_PARSE_ERRORS as e:
logger.debug("Failed to parse torrent file: %s", e)
return None
def extract_hash_from_magnet(magnet_url: str) -> Optional[str]:
def extract_hash_from_magnet(magnet_url: str) -> str | None:
"""Extract info_hash from a magnet URL."""
if not magnet_url.startswith("magnet:"):
return None
@@ -245,12 +282,12 @@ def extract_hash_from_magnet(magnet_url: str) -> Optional[str]:
parsed = urlparse(magnet_url)
params = parse_qs(parsed.query)
def extract_btmh(value: str) -> Optional[str]:
def extract_btmh(value: str) -> str | None:
raw_value = value.strip()
if not raw_value:
return None
data: Optional[bytes] = None
data: bytes | None = None
if re.fullmatch(r"[a-fA-F0-9]+", raw_value):
if len(raw_value) % 2 != 0:
return None
@@ -262,18 +299,22 @@ def extract_hash_from_magnet(magnet_url: str) -> Optional[str]:
padded = raw_value.upper() + "=" * (-len(raw_value) % 8)
try:
data = base64.b32decode(padded, casefold=True)
except Exception:
except BinasciiError, ValueError:
return None
if not data:
return None
if len(data) >= 34 and data[0] == 0x12 and data[1] == 0x20:
digest = data[2:34]
if len(digest) == 32:
if (
len(data) >= _BASE32_BTMH_TAG_BYTES
and data[0] == _BTIH_PREFIX_BYTE
and data[1] == _BTIH_INFO_BYTE_HEX
):
digest = data[2:_BASE32_BTMH_TAG_BYTES]
if len(digest) == _BTIH_DIGEST_LENGTH:
return digest.hex().lower()
if len(data) == 32:
if len(data) == _BTIH_HASH_LENGTH_32:
return data.hex().lower()
return None
@@ -287,22 +328,26 @@ def extract_hash_from_magnet(magnet_url: str) -> Optional[str]:
hash_value = match.group(1)
# 40-char hex or 32-char hex (ED2K) - return as-is
if len(hash_value) == 40 or re.match(r'^[a-fA-F0-9]{32}$', hash_value):
if len(hash_value) == _BTIH_HASH_LENGTH_40 or re.match(
r"^[a-fA-F0-9]{32}$", hash_value
):
return hash_value.lower()
# 32-char base32 - decode to hex
if re.match(r'^[A-Z2-7]{32}$', hash_value.upper()):
if re.match(r"^[A-Z2-7]{32}$", hash_value.upper()):
try:
return base64.b32decode(hash_value.upper()).hex().lower()
except Exception:
pass
except BinasciiError, ValueError:
logger.debug(
"Could not decode base32 BTIH hash from magnet URI: %s", hash_value
)
# Fallback: return as-is
return hash_value.lower()
for xt in xt_values:
if xt.startswith("urn:btmh:"):
btmh_value = xt[len("urn:btmh:"):]
btmh_value = xt[len("urn:btmh:") :]
btmh_hash = extract_btmh(btmh_value)
if btmh_hash:
return btmh_hash
+171 -81
View File
@@ -1,29 +1,93 @@
"""
Transmission download client for Prowlarr integration.
"""Transmission download client for Prowlarr integration.
Uses the transmission-rpc library to communicate with Transmission's RPC API.
"""
from contextlib import contextmanager
from typing import Any, Iterator, Optional, Tuple
from __future__ import annotations
import importlib
from contextlib import contextmanager, suppress
from typing import TYPE_CHECKING, Protocol, TypeGuard
from shelfmark.core.config import config
from shelfmark.core.logger import setup_logger
from shelfmark.core.utils import normalize_http_url
from shelfmark.download.network import get_ssl_verify
from shelfmark.download.clients import (
DownloadClient,
DownloadStatus,
register_client,
)
from shelfmark.download.clients._coercion import (
coerce_optional_float,
coerce_optional_int,
config_text,
normalize_http_config_url,
)
from shelfmark.download.clients.torrent_utils import (
extract_torrent_info,
parse_transmission_url,
)
from shelfmark.download.network import get_ssl_verify
try:
from transmission_rpc import TransmissionError as _ImportedTransmissionError
except ImportError:
_ImportedTransmissionError = RuntimeError
if TYPE_CHECKING:
from collections.abc import Iterator
logger = setup_logger(__name__)
_SEEDING_PROGRESS_PERCENT = 100
_ETA_MAX_SECONDS = 604800
_TransmissionError = (
_ImportedTransmissionError
if isinstance(_ImportedTransmissionError, type)
and issubclass(_ImportedTransmissionError, Exception)
else RuntimeError
)
_TRANSMISSION_CLIENT_ERRORS = (
_TransmissionError,
AttributeError,
OSError,
RuntimeError,
TypeError,
ValueError,
)
class _TransmissionSessionProtocol(Protocol):
verify: bool
class _TransmissionSessionFactory(Protocol):
def __call__(self, *args: object, **kwargs: object) -> _TransmissionSessionProtocol: ...
class _TransmissionRequestsNamespace(Protocol):
Session: _TransmissionSessionFactory
class _TransmissionProtocolAttribute(Protocol):
protocol: str
def _is_requests_namespace_with_session(
candidate: object,
) -> TypeGuard[_TransmissionRequestsNamespace]:
return hasattr(candidate, "Session") and callable(getattr(candidate, "Session", None))
def _has_protocol_attr(candidate: object) -> TypeGuard[_TransmissionProtocolAttribute]:
return hasattr(candidate, "protocol")
def _set_transmission_protocol_if_supported(client: object, protocol: str) -> None:
if protocol != "https" or not _has_protocol_attr(client):
return
with suppress(AttributeError, OSError, RuntimeError, TypeError, ValueError):
client.protocol = protocol
@contextmanager
def _transmission_session_verify_override(url: str) -> Iterator[None]:
@@ -38,34 +102,39 @@ def _transmission_session_verify_override(url: str) -> Iterator[None]:
return
try:
import transmission_rpc.client as transmission_rpc_client
except Exception:
transmission_rpc_client = importlib.import_module("transmission_rpc.client")
requests_namespace = getattr(transmission_rpc_client, "requests", None)
except ImportError:
# If internals differ, gracefully fall back to default behavior.
yield
return
original_session_factory = transmission_rpc_client.requests.Session
if not _is_requests_namespace_with_session(requests_namespace):
yield
return
def _session_factory(*args: Any, **kwargs: Any) -> Any:
original_session_factory = requests_namespace.Session
def _session_factory(*args: object, **kwargs: object) -> _TransmissionSessionProtocol:
session = original_session_factory(*args, **kwargs)
session.verify = False
return session
transmission_rpc_client.requests.Session = _session_factory
requests_namespace.Session = _session_factory
try:
yield
finally:
transmission_rpc_client.requests.Session = original_session_factory
requests_namespace.Session = original_session_factory
def _apply_transmission_ssl_verify(client: Any, url: str) -> None:
def _apply_transmission_ssl_verify(client: object, url: str) -> None:
"""Apply global certificate validation policy to transmission-rpc client."""
session = getattr(client, "_http_session", None)
if session is None:
return
try:
session.verify = get_ssl_verify(url)
except Exception as e:
except (AttributeError, OSError, TypeError, ValueError) as e:
logger.debug("Unable to apply Transmission TLS verify setting: %s", e)
@@ -76,20 +145,22 @@ class TransmissionClient(DownloadClient):
protocol = "torrent"
name = "transmission"
def __init__(self):
def __init__(self) -> None:
"""Initialize Transmission client with settings from config."""
from transmission_rpc import Client
raw_url = config.get("TRANSMISSION_URL", "")
raw_url = config_text(config.get("TRANSMISSION_URL", ""))
if not raw_url:
raise ValueError("TRANSMISSION_URL is required")
msg = "TRANSMISSION_URL is required"
raise ValueError(msg)
url = normalize_http_url(raw_url)
url = normalize_http_config_url(raw_url)
if not url:
raise ValueError("TRANSMISSION_URL is invalid")
msg = "TRANSMISSION_URL is invalid"
raise ValueError(msg)
username = config.get("TRANSMISSION_USERNAME", "")
password = config.get("TRANSMISSION_PASSWORD", "")
username = config_text(config.get("TRANSMISSION_USERNAME", ""))
password = config_text(config.get("TRANSMISSION_PASSWORD", ""))
# Parse URL to extract host, port, and path
protocol, host, port, path = parse_transmission_url(url)
@@ -98,8 +169,8 @@ class TransmissionClient(DownloadClient):
"host": host,
"port": port,
"path": path,
"username": username if username else None,
"password": password if password else None,
"username": username or None,
"password": password or None,
"protocol": protocol,
}
try:
@@ -113,56 +184,54 @@ class TransmissionClient(DownloadClient):
with _transmission_session_verify_override(url):
self._client = Client(**client_kwargs)
# Some versions expose protocol as an attribute rather than kwarg.
if protocol == "https" and hasattr(self._client, "protocol"):
try:
setattr(self._client, "protocol", protocol)
except Exception:
pass
_set_transmission_protocol_if_supported(self._client, protocol)
_apply_transmission_ssl_verify(self._client, url)
self._category = config.get("TRANSMISSION_CATEGORY", "books")
self._download_dir = config.get("TRANSMISSION_DOWNLOAD_DIR", "")
self._category = config_text(config.get("TRANSMISSION_CATEGORY", "books"))
self._download_dir = config_text(config.get("TRANSMISSION_DOWNLOAD_DIR", ""))
@staticmethod
def is_configured() -> bool:
"""Check if Transmission is configured and selected as the torrent client."""
client = config.get("PROWLARR_TORRENT_CLIENT", "")
url = normalize_http_url(config.get("TRANSMISSION_URL", ""))
client = config_text(config.get("PROWLARR_TORRENT_CLIENT", ""))
url = normalize_http_config_url(config.get("TRANSMISSION_URL", ""))
return client == "transmission" and bool(url)
def test_connection(self) -> Tuple[bool, str]:
def test_connection(self) -> tuple[bool, str]:
"""Test connection to Transmission."""
try:
session = self._client.get_session()
version = session.version
except _TRANSMISSION_CLIENT_ERRORS as e:
return False, f"Connection failed: {e!s}"
else:
return True, f"Connected to Transmission {version}"
except Exception as e:
return False, f"Connection failed: {str(e)}"
def add_download(
self,
url: str,
name: str,
category: Optional[str] = None,
expected_hash: Optional[str] = None,
**kwargs,
category: str | None = None,
expected_hash: str | None = None,
**kwargs: object,
) -> str:
"""
Add torrent by URL (magnet or .torrent).
"""Add torrent by URL (magnet or .torrent).
Args:
url: Magnet link or .torrent URL
name: Display name for the torrent
category: Category for organization (uses configured default if not specified)
expected_hash: Optional info_hash hint (from Prowlarr)
**kwargs: Client-specific options passed through to the implementation.
Returns:
Torrent hash (info_hash).
Raises:
Exception: If adding fails.
"""
try:
resolved_category = category or self._category or ""
resolved_category = category or self._category
torrent_info = extract_torrent_info(url, expected_hash=expected_hash)
add_kwargs = {}
@@ -186,23 +255,39 @@ class TransmissionClient(DownloadClient):
)
torrent_hash = torrent.hashString.lower()
logger.info(f"Added torrent to Transmission: {torrent_hash}")
logger.info("Added torrent to Transmission: %s", torrent_hash)
# Apply per-torrent seeding limits from indexer
seed_kwargs = {}
seeding_time_limit = coerce_optional_int(kwargs.get("seeding_time_limit"))
if seeding_time_limit is not None:
seed_kwargs["seed_idle_limit"] = seeding_time_limit
seed_kwargs["seed_idle_mode"] = 1 # per-torrent
ratio_limit = coerce_optional_float(kwargs.get("ratio_limit"))
if ratio_limit is not None:
seed_kwargs["seed_ratio_limit"] = ratio_limit
seed_kwargs["seed_ratio_mode"] = 1 # per-torrent
if seed_kwargs:
try:
self._client.change_torrent(ids=torrent_hash, **seed_kwargs)
except _TRANSMISSION_CLIENT_ERRORS as e:
logger.warning("Failed to set seeding limits for %s: %s", torrent_hash, e)
except _TRANSMISSION_CLIENT_ERRORS:
logger.exception("Transmission add failed")
raise
else:
return torrent_hash
except Exception as e:
logger.error(f"Transmission add failed: {e}")
raise
def get_status(self, download_id: str) -> DownloadStatus:
"""
Get torrent status by hash.
"""Get torrent status by hash.
Args:
download_id: Torrent info_hash
Returns:
Current download status.
"""
try:
torrent = self._client.get_torrent(download_id)
@@ -216,7 +301,9 @@ class TransmissionClient(DownloadClient):
# 5: seed pending
# 6: seeding
# torrent.status is an enum with .value as string
status_value = torrent.status.value if hasattr(torrent.status, 'value') else str(torrent.status)
status_value = (
torrent.status.value if hasattr(torrent.status, "value") else str(torrent.status)
)
status_map = {
"stopped": ("paused", "Paused"),
"check pending": ("checking", "Waiting to check"),
@@ -230,30 +317,30 @@ class TransmissionClient(DownloadClient):
state, message = status_map.get(status_value, ("downloading", "Downloading"))
progress = torrent.percent_done * 100
# Only mark complete when seeding - seed pending means files still being moved
complete = progress >= 100 and status_value == "seeding"
complete = progress >= _SEEDING_PROGRESS_PERCENT and status_value == "seeding"
if complete:
message = "Complete"
# Get ETA if available and reasonable (less than 1 week)
eta = None
if hasattr(torrent, 'eta') and torrent.eta:
if hasattr(torrent, "eta") and torrent.eta:
eta_seconds = torrent.eta.total_seconds()
if 0 < eta_seconds < 604800:
if 0 < eta_seconds < _ETA_MAX_SECONDS:
eta = int(eta_seconds)
# Get download speed
download_speed = torrent.rate_download if hasattr(torrent, 'rate_download') else None
download_speed = torrent.rate_download if hasattr(torrent, "rate_download") else None
# Get file path for completed downloads
file_path = None
if complete:
# Output path is downloadDir + torrent name (with ':' replaced)
torrent_name = getattr(torrent, 'name', '')
torrent_name = getattr(torrent, "name", "")
if isinstance(torrent_name, str):
torrent_name = torrent_name.replace(':', '_')
torrent_name = torrent_name.replace(":", "_")
file_path = self._build_path(
getattr(torrent, 'download_dir', ''),
getattr(torrent, "download_dir", ""),
torrent_name,
)
@@ -269,12 +356,11 @@ class TransmissionClient(DownloadClient):
except KeyError:
return DownloadStatus.error("Torrent not found")
except Exception as e:
except _TRANSMISSION_CLIENT_ERRORS as e:
return DownloadStatus.error(self._log_error("get_status", e))
def remove(self, download_id: str, delete_files: bool = False) -> bool:
"""
Remove a torrent from Transmission.
def remove(self, download_id: str, *, delete_files: bool = False) -> bool:
"""Remove a torrent from Transmission.
Args:
download_id: Torrent info_hash
@@ -282,6 +368,7 @@ class TransmissionClient(DownloadClient):
Returns:
True if successful.
"""
try:
self._client.remove_torrent(
@@ -289,40 +376,42 @@ class TransmissionClient(DownloadClient):
delete_data=delete_files,
)
logger.info(
f"Removed torrent from Transmission: {download_id}"
+ (" (with files)" if delete_files else "")
"Removed torrent from Transmission: %s%s",
download_id,
" (with files)" if delete_files else "",
)
return True
except Exception as e:
except _TRANSMISSION_CLIENT_ERRORS as e:
self._log_error("remove", e)
return False
else:
return True
def get_download_path(self, download_id: str) -> Optional[str]:
"""
Get the path where torrent files are located.
def get_download_path(self, download_id: str) -> str | None:
"""Get the path where torrent files are located.
Args:
download_id: Torrent info_hash
Returns:
Content path (file or directory), or None.
"""
try:
torrent = self._client.get_torrent(download_id)
torrent_name = getattr(torrent, 'name', '')
if isinstance(torrent_name, str):
torrent_name = torrent_name.replace(':', '_')
return self._build_path(
getattr(torrent, 'download_dir', ''),
torrent_name,
)
except Exception as e:
torrent = self._client.get_torrent(download_id)
torrent_name = getattr(torrent, "name", "")
if isinstance(torrent_name, str):
torrent_name = torrent_name.replace(":", "_")
return self._build_path(
getattr(torrent, "download_dir", ""),
torrent_name,
)
except _TRANSMISSION_CLIENT_ERRORS as e:
self._log_error("get_download_path", e, level="debug")
return None
def find_existing(
self, url: str, category: Optional[str] = None
) -> Optional[Tuple[str, DownloadStatus]]:
self, url: str, category: str | None = None
) -> tuple[str, DownloadStatus] | None:
"""Check if a torrent for this URL already exists in Transmission."""
try:
torrent_info = extract_torrent_info(url)
@@ -332,9 +421,10 @@ class TransmissionClient(DownloadClient):
try:
self._client.get_torrent(torrent_info.info_hash)
status = self.get_status(torrent_info.info_hash)
return (torrent_info.info_hash, status)
except KeyError:
return None
except Exception as e:
logger.debug(f"Error checking for existing torrent: {e}")
else:
return (torrent_info.info_hash, status)
except _TRANSMISSION_CLIENT_ERRORS as e:
logger.debug("Error checking for existing torrent: %s", e)
return None
+142 -60
View File
@@ -11,44 +11,60 @@ import subprocess
import tempfile
import time
from pathlib import Path
from typing import Any, Callable, Optional, TypeVar, cast
from typing import TYPE_CHECKING, Any, TypeVar, cast
from shelfmark.core.logger import setup_logger
from shelfmark.download.permissions_debug import log_transfer_permission_context
if TYPE_CHECKING:
from collections.abc import Callable
from gevent.threadpool import ThreadPool
logger = setup_logger(__name__)
try:
from gevent import monkey as _gevent_monkey
from gevent.threadpool import ThreadPool as _GeventThreadPool
except Exception:
except ImportError:
_gevent_monkey = None
_GeventThreadPool = None
T = TypeVar("T")
_IO_THREADPOOL: Optional["_GeventThreadPool"] = None
_IO_THREADPOOL: ThreadPool | None = None
def _use_gevent_threadpool() -> bool:
return bool(
_gevent_monkey
and _GeventThreadPool
and _gevent_monkey.is_module_patched("threading")
_gevent_monkey and _GeventThreadPool and _gevent_monkey.is_module_patched("threading")
)
def _get_io_threadpool() -> "_GeventThreadPool":
def _get_io_threadpool() -> ThreadPool:
global _IO_THREADPOOL
if _IO_THREADPOOL is None:
pool_size = max(2, min(8, os.cpu_count() or 2))
_IO_THREADPOOL = _GeventThreadPool(pool_size)
threadpool_cls = _GeventThreadPool
if threadpool_cls is None:
msg = "gevent threadpool is unavailable"
raise RuntimeError(msg)
_IO_THREADPOOL = threadpool_cls(pool_size)
return _IO_THREADPOOL
def _call_and_capture(func: Callable[..., T], args: tuple[Any, ...], kwargs: dict[str, Any]) -> tuple[bool, T | Exception]:
def _call_and_capture[T](
func: Callable[..., T], args: tuple[Any, ...], kwargs: dict[str, Any]
) -> tuple[bool, T | Exception]:
try:
return True, func(*args, **kwargs)
except Exception as exc:
except (
AttributeError,
OSError,
RuntimeError,
TypeError,
ValueError,
subprocess.SubprocessError,
) as exc:
return False, exc
@@ -60,13 +76,10 @@ def _must_avoid_gevent_threadpool(func: Callable[..., Any]) -> bool:
# gevent.subprocess requires child watchers on the default event loop.
# Executing patched subprocess functions in a worker thread can raise:
# "TypeError: child watchers are only available on the default loop".
if _gevent_monkey.is_object_patched("subprocess", "run") and func is subprocess.run:
return True
return False
return _gevent_monkey.is_object_patched("subprocess", "run") and func is subprocess.run
def run_blocking_io(func: Callable[..., T], *args: Any, **kwargs: Any) -> T:
def run_blocking_io[T](func: Callable[..., T], *args: Any, **kwargs: Any) -> T:
"""Run blocking I/O in a native thread when under gevent.
gevent's threadpool will eagerly log exceptions raised inside worker threads,
@@ -80,14 +93,16 @@ def run_blocking_io(func: Callable[..., T], *args: Any, **kwargs: Any) -> T:
if _use_gevent_threadpool():
ok, result = _get_io_threadpool().apply(_call_and_capture, (func, args, kwargs))
if ok:
return cast(T, result)
exc = cast(Exception, result)
return cast("T", result)
exc = cast("Exception", result)
raise exc
return func(*args, **kwargs)
_VERIFY_IO_WAIT_SECONDS = 3.0
_PUBLISH_VERIFY_RETRY_SECONDS = 0.25
_TEMPFILE_PREFIX = ".shelfmark."
_TEMPFILE_SUFFIX = ".tmp"
def _verify_transfer_size(
@@ -106,17 +121,60 @@ def _verify_transfer_size(
return
logger.debug(
f"File {action} size mismatch, waiting for filesystem sync: {dest} "
f"({actual_size} != {expected_size})"
"File %s size mismatch, waiting for filesystem sync: %s (%s != %s)",
action,
dest,
actual_size,
expected_size,
)
time.sleep(_VERIFY_IO_WAIT_SECONDS)
actual_size = run_blocking_io(dest.stat).st_size
if actual_size != expected_size:
raise IOError(
msg = (
f"File {action} incomplete, data loss may have occurred. "
f"'{dest}' was {actual_size} bytes instead of expected {expected_size}."
)
raise OSError(msg)
def _is_stale_handle_error(error: Exception) -> bool:
return isinstance(error, OSError) and error.errno == getattr(errno, "ESTALE", 116)
def _verify_published_file(
dest: Path,
expected_size: int,
action: str,
) -> None:
"""Best-effort verify after publishing a temp file into place.
The temp file was already verified before publish. Some NFS mounts can report
a transient stale handle immediately after `os.replace()` makes the final path
visible, so retry once and then trust the successful publish instead of
turning the handoff into a false failure.
"""
try:
_verify_transfer_size(dest, expected_size, action)
except OSError as error:
if not _is_stale_handle_error(error):
raise
else:
return
time.sleep(_PUBLISH_VERIFY_RETRY_SECONDS)
try:
_verify_transfer_size(dest, expected_size, action)
except OSError as retry_error:
if not _is_stale_handle_error(retry_error):
raise
logger.warning(
"Skipping post-publish verification for %s after stale handle on %s: %s",
action,
dest,
retry_error,
)
def atomic_write(dest_path: Path, data: bytes, max_attempts: int = 100) -> Path:
@@ -135,6 +193,7 @@ def atomic_write(dest_path: Path, data: bytes, max_attempts: int = 100) -> Path:
Raises:
RuntimeError: If no unique path found after max_attempts
"""
base = dest_path.stem
ext = dest_path.suffix
@@ -155,12 +214,14 @@ def atomic_write(dest_path: Path, data: bytes, max_attempts: int = 100) -> Path:
finally:
run_blocking_io(os.close, fd)
if attempt > 0:
logger.info(f"File collision resolved: {try_path.name}")
return try_path
logger.info("File collision resolved: %s", try_path.name)
except FileExistsError:
continue
else:
return try_path
raise RuntimeError(f"Could not write file after {max_attempts} attempts: {dest_path}")
msg = f"Could not write file after {max_attempts} attempts: {dest_path}"
raise RuntimeError(msg)
def _is_permission_error(e: Exception) -> bool:
@@ -180,7 +241,7 @@ def _system_op(op: str, source: Path, dest: Path) -> None:
)
def _perform_nfs_fallback(source: Path, dest: Path, is_move: bool) -> None:
def _perform_nfs_fallback(source: Path, dest: Path, *, is_move: bool) -> None:
"""Handle NFS/SMB permission errors by falling back to copyfile -> system op."""
expected_size = run_blocking_io(source.stat).st_size
@@ -191,15 +252,16 @@ def _perform_nfs_fallback(source: Path, dest: Path, is_move: bool) -> None:
if is_move:
run_blocking_io(source.unlink)
return
except Exception as copy_error:
# Clean up failed copy attempt if it exists
run_blocking_io(dest.unlink, missing_ok=True)
if _is_permission_error(copy_error):
log_transfer_permission_context("nfs_fallback_copyfile", source=source, dest=dest, error=copy_error)
logger.error("Fallback copyfile failed (%s -> %s): %s", source, dest, copy_error)
log_transfer_permission_context(
"nfs_fallback_copyfile", source=source, dest=dest, error=copy_error
)
logger.exception("Fallback copyfile failed (%s -> %s)", source, dest)
# Fallback 2: system command
op = "mv" if is_move else "cp"
@@ -211,10 +273,14 @@ def _perform_nfs_fallback(source: Path, dest: Path, is_move: bool) -> None:
if is_move:
run_blocking_io(source.unlink, missing_ok=True)
except subprocess.CalledProcessError as sys_error:
log_transfer_permission_context("nfs_fallback_system", source=source, dest=dest, error=sys_error)
logger.error("System %s failed (%s -> %s): %s", op, source, dest, sys_error.stderr)
log_transfer_permission_context(
"nfs_fallback_system", source=source, dest=dest, error=sys_error
)
logger.exception("System %s failed (%s -> %s): %s", op, source, dest, sys_error.stderr)
run_blocking_io(dest.unlink, missing_ok=True)
raise
else:
return
def _is_enoent_error(error: Exception) -> bool:
@@ -224,7 +290,7 @@ def _is_enoent_error(error: Exception) -> bool:
def _can_use_partial_copy_after_enoent(
temp_path: Optional[Path],
temp_path: Path | None,
expected_size: int,
action: str,
) -> bool:
@@ -234,9 +300,10 @@ def _can_use_partial_copy_after_enoent(
try:
_verify_transfer_size(temp_path, expected_size, action)
return True
except Exception:
except OSError:
return False
else:
return True
def _claim_destination(path: Path) -> bool:
@@ -274,10 +341,16 @@ def _hardlink_not_supported(error: OSError) -> bool:
def _create_temp_path(dest_path: Path) -> Path:
"""Create a destination-adjacent temp file without inheriting the full basename.
Reusing the entire destination filename in the temp prefix can push otherwise
valid long names over the filesystem component limit once `tempfile` adds its
random suffix.
"""
fd, temp_path = run_blocking_io(
tempfile.mkstemp,
prefix=f".{dest_path.name}.",
suffix=".tmp",
prefix=_TEMPFILE_PREFIX,
suffix=_TEMPFILE_SUFFIX,
dir=str(dest_path.parent),
)
run_blocking_io(os.close, fd)
@@ -306,7 +379,6 @@ def _publish_temp_file(temp_path: Path, dest_path: Path) -> bool:
run_blocking_io(os.close, fd)
except OSError:
pass
return True
except Exception as e:
if _is_permission_error(e):
log_transfer_permission_context(
@@ -317,6 +389,8 @@ def _publish_temp_file(temp_path: Path, dest_path: Path) -> bool:
)
run_blocking_io(dest_path.unlink, missing_ok=True)
raise
else:
return True
def atomic_move(source_path: Path, dest_path: Path, max_attempts: int = 100) -> Path:
@@ -339,6 +413,7 @@ def atomic_move(source_path: Path, dest_path: Path, max_attempts: int = 100) ->
Raises:
RuntimeError: If no unique path found after max_attempts
"""
base = dest_path.stem
ext = dest_path.suffix
@@ -363,8 +438,7 @@ def atomic_move(source_path: Path, dest_path: Path, max_attempts: int = 100) ->
else:
run_blocking_io(os.rename, str(source_path), str(try_path))
if attempt > 0:
logger.info(f"File collision resolved: {try_path.name}")
return try_path
logger.info("File collision resolved: %s", try_path.name)
except FileExistsError:
# Race condition: file created between exists() check and rename()
if claimed:
@@ -382,7 +456,7 @@ def atomic_move(source_path: Path, dest_path: Path, max_attempts: int = 100) ->
run_blocking_io(try_path.unlink, missing_ok=True)
claimed = False
temp_path: Optional[Path] = None
temp_path: Path | None = None
try:
try:
temp_path = _create_temp_path(try_path)
@@ -417,7 +491,7 @@ def atomic_move(source_path: Path, dest_path: Path, max_attempts: int = 100) ->
continue
try:
_verify_transfer_size(try_path, expected_size, "move")
_verify_published_file(try_path, expected_size, "move")
except Exception:
run_blocking_io(try_path.unlink, missing_ok=True)
raise
@@ -425,9 +499,7 @@ def atomic_move(source_path: Path, dest_path: Path, max_attempts: int = 100) ->
run_blocking_io(source_path.unlink)
if attempt > 0:
logger.info(f"File collision resolved: {try_path.name}")
return try_path
logger.info("File collision resolved: %s", try_path.name)
except FileExistsError:
if temp_path:
run_blocking_io(temp_path.unlink, missing_ok=True)
@@ -436,6 +508,8 @@ def atomic_move(source_path: Path, dest_path: Path, max_attempts: int = 100) ->
if temp_path:
run_blocking_io(temp_path.unlink, missing_ok=True)
raise
else:
return try_path
except (PermissionError, OSError) as e:
if _is_permission_error(e):
@@ -454,19 +528,22 @@ def atomic_move(source_path: Path, dest_path: Path, max_attempts: int = 100) ->
try:
_perform_nfs_fallback(source_path, try_path, is_move=True)
if attempt > 0:
logger.info(f"File collision resolved (fallback): {try_path.name}")
return try_path
logger.info("File collision resolved (fallback): %s", try_path.name)
except Exception as fallback_error:
logger.error(
"NFS fallback also failed (%s -> %s): %s",
logger.exception(
"NFS fallback also failed (%s -> %s)",
source_path,
try_path,
fallback_error,
)
raise e from fallback_error
else:
return try_path
raise
else:
return try_path
raise RuntimeError(f"Could not move file after {max_attempts} attempts: {dest_path}")
msg = f"Could not move file after {max_attempts} attempts: {dest_path}"
raise RuntimeError(msg)
def atomic_hardlink(source_path: Path, dest_path: Path, max_attempts: int = 100) -> Path:
@@ -482,6 +559,7 @@ def atomic_hardlink(source_path: Path, dest_path: Path, max_attempts: int = 100)
Raises:
RuntimeError: If no unique path found after max_attempts
"""
base = dest_path.stem
ext = dest_path.suffix
@@ -492,8 +570,7 @@ def atomic_hardlink(source_path: Path, dest_path: Path, max_attempts: int = 100)
try:
run_blocking_io(os.link, str(source_path), str(try_path))
if attempt > 0:
logger.info(f"File collision resolved: {try_path.name}")
return try_path
logger.info("File collision resolved: %s", try_path.name)
except FileExistsError:
continue
except OSError as e:
@@ -514,8 +591,11 @@ def atomic_hardlink(source_path: Path, dest_path: Path, max_attempts: int = 100)
)
return atomic_copy(source_path, dest_path, max_attempts=max_attempts)
raise
else:
return try_path
raise RuntimeError(f"Could not create hardlink after {max_attempts} attempts: {dest_path}")
msg = f"Could not create hardlink after {max_attempts} attempts: {dest_path}"
raise RuntimeError(msg)
def atomic_copy(source_path: Path, dest_path: Path, max_attempts: int = 100) -> Path:
@@ -534,6 +614,7 @@ def atomic_copy(source_path: Path, dest_path: Path, max_attempts: int = 100) ->
Raises:
RuntimeError: If no unique path found after max_attempts
"""
base = dest_path.stem
ext = dest_path.suffix
@@ -544,7 +625,7 @@ def atomic_copy(source_path: Path, dest_path: Path, max_attempts: int = 100) ->
try_path = dest_path if attempt == 0 else parent / f"{base}_{attempt}{ext}"
if run_blocking_io(try_path.exists):
continue
temp_path: Optional[Path] = None
temp_path: Path | None = None
try:
temp_path = _create_temp_path(try_path)
try:
@@ -567,11 +648,10 @@ def atomic_copy(source_path: Path, dest_path: Path, max_attempts: int = 100) ->
try:
_perform_nfs_fallback(source_path, temp_path, is_move=False)
except Exception as fallback_error:
logger.error(
"NFS fallback also failed (%s -> %s): %s",
logger.exception(
"NFS fallback also failed (%s -> %s)",
source_path,
temp_path,
fallback_error,
)
raise e from fallback_error
elif _is_enoent_error(e) and _can_use_partial_copy_after_enoent(
@@ -594,17 +674,19 @@ def atomic_copy(source_path: Path, dest_path: Path, max_attempts: int = 100) ->
continue
try:
_verify_transfer_size(try_path, expected_size, "copy")
_verify_published_file(try_path, expected_size, "copy")
except Exception:
run_blocking_io(try_path.unlink, missing_ok=True)
raise
if attempt > 0:
logger.info(f"File collision resolved: {try_path.name}")
return try_path
logger.info("File collision resolved: %s", try_path.name)
except Exception:
if temp_path:
run_blocking_io(temp_path.unlink, missing_ok=True)
raise
else:
return try_path
raise RuntimeError(f"Could not copy file after {max_attempts} attempts: {dest_path}")
msg = f"Could not copy file after {max_attempts} attempts: {dest_path}"
raise RuntimeError(msg)
+267 -121
View File
@@ -2,86 +2,126 @@
import random
import time
from http import HTTPStatus
from io import BytesIO
from threading import Event, Thread
from typing import Callable, Optional
from urllib.parse import urlparse, urljoin
from typing import TYPE_CHECKING, NoReturn
from urllib.parse import urljoin, urlparse
import requests
from tqdm import tqdm
from shelfmark.download import network
from shelfmark.download.network import get_proxies, get_ssl_verify
from shelfmark.bypass import BypassCancelledError
from shelfmark.core.config import config as app_config
from shelfmark.core.logger import setup_logger
from shelfmark.core.request_helpers import coerce_bool, normalize_positive_int
from shelfmark.download import network
from shelfmark.download.network import get_proxies, get_ssl_verify
if TYPE_CHECKING:
from collections.abc import Callable
from types import ModuleType
logger = setup_logger(__name__)
_RNG = random.SystemRandom()
_MAX_REDIRECTS = 5
_HTTP_STATUS_FORBIDDEN = HTTPStatus.FORBIDDEN
_HTTP_STATUS_NOT_FOUND = HTTPStatus.NOT_FOUND
_HTTP_STATUS_RATE_LIMITED = HTTPStatus.TOO_MANY_REQUESTS
_HTTP_STATUS_OK = HTTPStatus.OK
_HTTP_STATUS_RANGE_NOT_SATISFIABLE = HTTPStatus.REQUESTED_RANGE_NOT_SATISFIABLE
_HTTP_STATUS_PARTIAL_CONTENT = HTTPStatus.PARTIAL_CONTENT
_HTTP_STATUS_NON_RETRYABLE = (_HTTP_STATUS_FORBIDDEN, _HTTP_STATUS_NOT_FOUND)
_STATUS_CALLBACK_ERRORS = (AttributeError, KeyError, OSError, RuntimeError, TypeError, ValueError)
_BYPASSER_ERRORS = (
AttributeError,
BypassCancelledError,
KeyError,
OSError,
RuntimeError,
TypeError,
ValueError,
requests.exceptions.RequestException,
)
# Bypasser modules are imported lazily to support dynamic selection based on config
_internal_bypasser = None
_external_bypasser = None
def _get_internal_bypasser():
def _raise_too_many_redirects(message: str) -> NoReturn:
raise requests.exceptions.TooManyRedirects(message)
def _get_internal_bypasser() -> ModuleType:
"""Lazy import of internal bypasser module."""
global _internal_bypasser
if _internal_bypasser is None:
try:
from shelfmark.bypass import internal_bypasser
_internal_bypasser = internal_bypasser
except ImportError as e:
raise RuntimeError(
msg = (
f"Failed to import internal bypasser: {e}. "
"Check that all dependencies are installed. "
"You may need to disable CF bypass or use the external bypasser."
) from e
)
raise RuntimeError(msg) from e
return _internal_bypasser
def _get_external_bypasser():
def _get_external_bypasser() -> ModuleType:
"""Lazy import of external bypasser module."""
global _external_bypasser
if _external_bypasser is None:
try:
from shelfmark.bypass import external_bypasser
_external_bypasser = external_bypasser
except ImportError as e:
raise RuntimeError(
msg = (
f"Failed to import external bypasser: {e}. "
"Check that the external bypasser is properly configured."
) from e
)
raise RuntimeError(msg) from e
return _external_bypasser
def _is_using_external_bypasser() -> bool:
"""Check if external bypasser is configured (reads from config, not just env)."""
return app_config.get("USING_EXTERNAL_BYPASSER", False)
return coerce_bool(app_config.get("USING_EXTERNAL_BYPASSER", False))
def _is_cf_bypass_enabled() -> bool:
"""Check if Cloudflare bypass is enabled."""
return app_config.get("USE_CF_BYPASS", True)
return coerce_bool(app_config.get("USE_CF_BYPASS", True))
def get_bypassed_page(url, selector=None, cancel_flag=None):
"""Wrapper that delegates to the appropriate bypasser based on config."""
def get_bypassed_page(
url: str,
selector: network.AAMirrorSelector | None = None,
cancel_flag: Event | None = None,
) -> str | None:
"""Fetch a bypassed page using the active bypasser implementation."""
if _is_using_external_bypasser():
return _get_external_bypasser().get_bypassed_page(url, selector, cancel_flag)
return _get_internal_bypasser().get_bypassed_page(url, selector, cancel_flag)
def get_cf_cookies_for_domain(domain):
def get_cf_cookies_for_domain(domain: str) -> dict[str, str]:
"""Get CF cookies - only available with internal bypasser."""
if _is_using_external_bypasser():
logger.debug(f"External bypasser in use, CF cookies not available for {domain}")
logger.debug("External bypasser in use, CF cookies not available for %s", domain)
return {}
return _get_internal_bypasser().get_cf_cookies_for_domain(domain)
def get_cf_user_agent_for_domain(domain):
def get_cf_user_agent_for_domain(domain: str) -> str | None:
"""Get CF user agent - only available with internal bypasser."""
if _is_using_external_bypasser():
logger.debug(f"External bypasser in use, CF user agent not available for {domain}")
logger.debug("External bypasser in use, CF user agent not available for %s", domain)
return None
return _get_internal_bypasser().get_cf_user_agent_for_domain(domain)
@@ -100,7 +140,7 @@ def _apply_cf_bypass(url: str, headers: dict) -> dict:
cookies = get_cf_cookies_for_domain(hostname)
stored_ua = get_cf_user_agent_for_domain(hostname)
if stored_ua:
headers['User-Agent'] = stored_ua
headers["User-Agent"] = stored_ua
return cookies
@@ -110,19 +150,22 @@ MAX_DOWNLOAD_RETRIES = 2
MAX_RESUME_ATTEMPTS = 3
RETRYABLE_CODES = (429, 500, 502, 503, 504)
CONNECTION_ERRORS = (requests.exceptions.ConnectionError, requests.exceptions.Timeout,
requests.exceptions.SSLError, requests.exceptions.ChunkedEncodingError)
CONNECTION_ERRORS = (
requests.exceptions.ConnectionError,
requests.exceptions.Timeout,
requests.exceptions.SSLError,
requests.exceptions.ChunkedEncodingError,
)
DOWNLOAD_HEADERS = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Accept-Language': 'en-US,en;q=0.5',
'Accept-Encoding': 'gzip, deflate, br',
'Connection': 'keep-alive',
'Upgrade-Insecure-Requests': '1',
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36",
"Accept-Language": "en-US,en;q=0.5",
"Accept-Encoding": "gzip, deflate, br",
"Connection": "keep-alive",
"Upgrade-Insecure-Requests": "1",
}
def parse_size_string(size: str) -> Optional[float]:
def parse_size_string(size: str) -> float | None:
"""Parse a human-readable size string (e.g., '10.5 MB') into bytes."""
if not size:
return None
@@ -133,20 +176,22 @@ def parse_size_string(size: str) -> Optional[float]:
if normalized.endswith(suffix):
return float(normalized[:-2]) * mult
return float(normalized)
except (ValueError, IndexError):
except ValueError, IndexError:
return None
def _backoff_delay(attempt: int, base: float = 0.25, cap: float = 3.0) -> float:
"""Exponential backoff with jitter."""
return min(cap, base * (2 ** (attempt - 1))) + random.random() * base
return min(cap, base * (2 ** (attempt - 1))) + _RNG.random() * base
def _get_status_code(e: Exception) -> Optional[int]:
def _get_status_code(e: Exception) -> int | None:
"""Extract HTTP status code from an exception, or None if not applicable."""
if isinstance(e, requests.exceptions.HTTPError) and e.response is not None:
return e.response.status_code
return None
def _is_retryable_error(e: Exception) -> bool:
"""Check if error is retryable (connection error or retryable HTTP status)."""
if isinstance(e, CONNECTION_ERRORS):
@@ -155,90 +200,109 @@ def _is_retryable_error(e: Exception) -> bool:
return status is not None and status in RETRYABLE_CODES
def _try_rotation(original_url: str, current_url: str, selector: network.AAMirrorSelector) -> Optional[str]:
def _try_rotation(
original_url: str, current_url: str, selector: network.AAMirrorSelector
) -> str | None:
"""Try mirror/DNS rotation. Returns new URL or None."""
if current_url.startswith(network.get_aa_base_url()):
aa_base_url = network.get_aa_base_url()
if aa_base_url and current_url.startswith(aa_base_url):
new_base, action = selector.next_mirror_or_rotate_dns()
if action in ("mirror", "dns") and new_base:
new_url = selector.rewrite(original_url)
logger.info(f"[{action}] switching to: {new_url}")
logger.info("[%s] switching to: %s", action, new_url)
return new_url
elif network.should_rotate_dns_for_url(current_url) and network.rotate_dns_provider():
logger.info(f"[dns-rotate] retrying: {original_url}")
logger.info("[dns-rotate] retrying: %s", original_url)
return original_url
return None
def html_get_page(
url: str,
retry: Optional[int] = None,
retry: int | None = None,
selector: network.AAMirrorSelector | None = None,
cancel_flag: Event | None = None,
status_callback: Callable[[str, str | None], None] | None = None,
*,
use_bypasser: bool = False,
selector: Optional[network.AAMirrorSelector] = None,
cancel_flag: Optional[Event] = None,
status_callback: Optional[Callable[[str, Optional[str]], None]] = None,
allow_bypasser_fallback: bool = True,
include_response_url: bool = False,
success_delay: float = 1.0,
session: Optional[requests.Session] = None,
session: requests.Session | None = None,
) -> str | tuple[str, str]:
"""Fetch HTML content from a URL with retry mechanism.
Args:
url: URL to fetch.
retry: Maximum number of attempts before giving up.
selector: Mirror selector used for AA mirror and DNS rotation.
cancel_flag: Optional event used to abort retries early.
status_callback: Optional callback for UI status updates.
allow_bypasser_fallback: If False, 403 errors will trigger mirror rotation
instead of switching to the bypasser. Use for search operations.
use_bypasser: Whether to start with the bypasser instead of direct HTTP.
include_response_url: If True, return `(html, final_url)` to expose the
resolved response URL after redirects.
success_delay: Optional delay (seconds) after successful fetch.
session: Optional requests session to reuse across attempts.
"""
def _result(html: str, response_url: str) -> str | tuple[str, str]:
if include_response_url:
return html, response_url
return html
retry = retry if retry is not None else app_config.MAX_RETRY
configured_retry = normalize_positive_int(app_config.MAX_RETRY)
retry_limit = (
retry if retry is not None else (configured_retry if configured_retry is not None else 1)
)
selector = selector or network.AAMirrorSelector()
original_url = url
current_url = selector.rewrite(original_url)
use_bypasser_now = use_bypasser
for attempt in range(1, retry + 1):
for attempt in range(1, retry_limit + 1):
# Check for cancellation before each attempt
if cancel_flag and cancel_flag.is_set():
logger.info(f"html_get_page cancelled before attempt {attempt}")
logger.info("html_get_page cancelled before attempt %s", attempt)
return _result("", current_url)
cookies: dict[str, str] = {}
try:
if use_bypasser_now and _is_cf_bypass_enabled():
logger.debug(f"GET (bypasser): {current_url}")
if status_callback:
status_callback("resolving", "Bypassing protection...")
heartbeat_stop = Event()
heartbeat_thread: Optional[Thread] = None
heartbeat_thread: Thread | None = None
if status_callback:
def _heartbeat() -> None:
# Keep the download "alive" during long bypass operations so the orchestrator
# doesn't flag it as stalled.
while not heartbeat_stop.wait(timeout=30):
if cancel_flag and cancel_flag.is_set():
return
try:
status_callback("resolving", "Bypassing protection...")
except Exception:
return
heartbeat_thread = Thread(target=_heartbeat, daemon=True, name="BypassHeartbeat")
if cancel_flag and cancel_flag.is_set():
return
try:
status_callback("resolving", "Bypassing protection...")
except _STATUS_CALLBACK_ERRORS:
return
heartbeat_thread = Thread(
target=_heartbeat, daemon=True, name="BypassHeartbeat"
)
heartbeat_thread.start()
try:
result = get_bypassed_page(current_url, selector, cancel_flag)
return _result(result or "", current_url)
except Exception as e:
logger.warning(f"Bypasser error: {type(e).__name__}: {e}")
except _BYPASSER_ERRORS as e:
logger.warning("Bypasser error: %s: %s", type(e).__name__, e)
return _result("", current_url)
finally:
heartbeat_stop.set()
if heartbeat_thread:
heartbeat_thread.join(timeout=1)
logger.debug(f"GET: {current_url}")
logger.debug("GET: %s", current_url)
# Use a browser-like UA by default (AA can behave differently for python-requests UA).
headers = {"User-Agent": DOWNLOAD_HEADERS["User-Agent"]}
@@ -267,7 +331,9 @@ def html_get_page(
if is_aa_url and response.is_redirect:
location = response.headers.get("Location", "")
if not location:
raise requests.exceptions.TooManyRedirects(f"Redirect with no Location header: {current_url}")
_raise_too_many_redirects(
f"Redirect with no Location header: {current_url}"
)
redirect_url = urljoin(current_url, location)
current_host = urlparse(current_url).hostname or ""
@@ -305,8 +371,8 @@ def html_get_page(
# Same-host redirect (relative or absolute) - follow manually.
redirects_followed += 1
if redirects_followed > 5:
raise requests.exceptions.TooManyRedirects(f"Too many redirects for {current_url}")
if redirects_followed > _MAX_REDIRECTS:
_raise_too_many_redirects(f"Too many redirects for {current_url}")
current_url = redirect_url
continue
@@ -319,14 +385,14 @@ def html_get_page(
status = _get_status_code(e)
# 403 = Cloudflare/DDoS-Guard protection
if status == 403:
if status == _HTTP_STATUS_FORBIDDEN:
# If bypasser fallback is disabled, try mirrors instead
if not allow_bypasser_fallback:
new_url = _try_rotation(original_url, current_url, selector)
if new_url:
current_url = new_url
continue
logger.warning(f"403 error, mirrors exhausted: {current_url}")
logger.warning("403 error, mirrors exhausted: %s", current_url)
return _result("", current_url)
if _is_cf_bypass_enabled() and not use_bypasser_now:
@@ -336,19 +402,22 @@ def html_get_page(
fresh_cookies = get_cf_cookies_for_domain(parsed.hostname or "")
if fresh_cookies and not cookies:
# Cookies are now available - retry with cookies before using bypasser
logger.debug(f"403 but cookies now available - retrying with cookies: {current_url}")
logger.debug(
"403 but cookies now available - retrying with cookies: %s",
current_url,
)
continue
logger.info(f"403 detected; switching to bypasser: {current_url}")
logger.info("403 detected; switching to bypasser: %s", current_url)
if status_callback:
status_callback("resolving", "Bypassing protection...")
use_bypasser_now = True
continue
logger.warning(f"403 error, giving up: {current_url}")
logger.warning("403 error, giving up: %s", current_url)
return _result("", current_url)
# 404 = Not found
if status == 404:
logger.warning(f"404 error: {current_url}")
if status == _HTTP_STATUS_NOT_FOUND:
logger.warning("404 error: %s", current_url)
return _result("", current_url)
# Try mirror/DNS rotation on retryable errors
@@ -359,11 +428,18 @@ def html_get_page(
continue
# Retry with backoff
if attempt < retry:
logger.warning(f"Retry {attempt}/{retry} for {current_url}: {type(e).__name__}: {e}")
if attempt < retry_limit:
logger.warning(
"Retry %s/%s for %s: %s: %s",
attempt,
retry_limit,
current_url,
type(e).__name__,
e,
)
time.sleep(_backoff_delay(attempt))
else:
logger.error(f"Giving up after {retry} attempts: {current_url}")
logger.exception("Giving up after %s attempts: %s", retry_limit, current_url)
return _result("", current_url)
@@ -371,12 +447,12 @@ def html_get_page(
def download_url(
link: str,
size: str = "",
progress_callback: Optional[Callable[[float], None]] = None,
cancel_flag: Optional[Event] = None,
_selector: Optional[network.AAMirrorSelector] = None,
status_callback: Optional[Callable[[str, Optional[str]], None]] = None,
referer: Optional[str] = None,
) -> Optional[BytesIO]:
progress_callback: Callable[[float], None] | None = None,
cancel_flag: Event | None = None,
_selector: network.AAMirrorSelector | None = None,
status_callback: Callable[[str, str | None], None] | None = None,
referer: str | None = None,
) -> BytesIO | None:
"""Download content from URL with automatic retry and resume support."""
selector = _selector or network.AAMirrorSelector()
current_url = selector.rewrite(link)
@@ -384,7 +460,7 @@ def download_url(
# Build headers with optional referer
headers = DOWNLOAD_HEADERS.copy()
if referer:
headers['Referer'] = referer
headers["Referer"] = referer
total_size = parse_size_string(size) or 0
attempt = 0
@@ -399,19 +475,35 @@ def download_url(
try:
if attempt > 0 and status_callback:
status_callback("resolving", f"Connecting (Attempt {attempt + 1}/{MAX_DOWNLOAD_RETRIES})")
status_callback(
"resolving",
f"Connecting (Attempt {attempt + 1}/{MAX_DOWNLOAD_RETRIES})",
)
logger.info(f"Downloading: {current_url} (attempt {attempt + 1}/{MAX_DOWNLOAD_RETRIES})")
logger.info(
"Downloading: %s (attempt %s/%s)",
current_url,
attempt + 1,
MAX_DOWNLOAD_RETRIES,
)
# Try with CF cookies/UA if available
cookies = _apply_cf_bypass(current_url, headers)
response = requests.get(current_url, stream=True, proxies=get_proxies(current_url), timeout=REQUEST_TIMEOUT, cookies=cookies, headers=headers, verify=get_ssl_verify(current_url))
response = requests.get(
current_url,
stream=True,
proxies=get_proxies(current_url),
timeout=REQUEST_TIMEOUT,
cookies=cookies,
headers=headers,
verify=get_ssl_verify(current_url),
)
response.raise_for_status()
if status_callback:
status_callback("downloading", "")
total_size = total_size or float(response.headers.get('content-length', 0))
pbar = tqdm(total=total_size, unit='B', unit_scale=True, desc='Downloading')
total_size = total_size or float(response.headers.get("content-length", 0))
pbar = tqdm(total=total_size, unit="B", unit_scale=True, desc="Downloading")
for chunk in response.iter_content(chunk_size=8192):
if chunk:
@@ -426,55 +518,69 @@ def download_url(
pbar.close()
# Validate - check we didn't get HTML instead of file
if total_size > 0 and bytes_downloaded < total_size * 0.9:
if response.headers.get('content-type', '').startswith('text/html'):
logger.warning(f"Received HTML instead of file: {current_url}")
return None
if (
total_size > 0
and bytes_downloaded < total_size * 0.9
and response.headers.get("content-type", "").startswith("text/html")
):
logger.warning("Received HTML instead of file: %s", current_url)
return None
logger.debug(f"Download completed: {bytes_downloaded} bytes")
return buffer
logger.debug("Download completed: %s bytes", bytes_downloaded)
except requests.exceptions.RequestException as e:
status = _get_status_code(e)
retryable = _is_retryable_error(e)
# Z-Library 403 - try refreshing cookies via bypasser once before giving up
if status == 403 and _is_cf_bypass_enabled() and not zlib_cookie_refresh_attempted:
if (
status == _HTTP_STATUS_FORBIDDEN
and _is_cf_bypass_enabled()
and not zlib_cookie_refresh_attempted
):
parsed = urlparse(current_url)
if parsed.hostname and 'z-lib' in parsed.hostname and referer:
if _is_configured_zlib_host(parsed.hostname) and referer:
zlib_cookie_refresh_attempted = True
logger.info(f"Z-Library 403 - refreshing cookies via referer: {referer}")
logger.info("Z-Library 403 - refreshing cookies via referer: %s", referer)
try:
get_bypassed_page(referer, selector, cancel_flag)
time.sleep(0.5)
# Retry with fresh cookies (don't increment attempt)
continue
except Exception as cookie_err:
logger.warning(f"Z-Library cookie refresh failed: {cookie_err}")
except _BYPASSER_ERRORS as cookie_err:
logger.warning("Z-Library cookie refresh failed: %s", cookie_err)
# Non-retryable errors
if status in (403, 404):
logger.warning(f"Download failed ({status}): {current_url}")
if status in _HTTP_STATUS_NON_RETRYABLE:
logger.warning("Download failed (%s): %s", status, current_url)
return None
# Rate limited - skip to next source immediately
# (waiting doesn't help with concurrent downloads hitting the same server)
if status == 429:
logger.info(f"Rate limited (429) - trying next source")
if status == _HTTP_STATUS_RATE_LIMITED:
logger.info("Rate limited (429) - trying next source")
if status_callback:
status_callback("resolving", "Server busy, trying next")
return None
# Timeout - don't retry, server likely overloaded
if isinstance(e, requests.exceptions.Timeout):
logger.warning(f"Timeout: {current_url} - skipping to next source")
logger.warning("Timeout: %s - skipping to next source", current_url)
if status_callback:
status_callback("resolving", "Server timed out, trying next")
return None
# Try to resume if we got some data
if bytes_downloaded > 0 and retryable:
resumed = _try_resume(current_url, buffer, bytes_downloaded, total_size, progress_callback, cancel_flag, headers)
resumed = _try_resume(
current_url,
buffer,
bytes_downloaded,
total_size,
progress_callback,
cancel_flag,
headers,
)
if resumed:
return resumed
@@ -486,49 +592,88 @@ def download_url(
attempt += 1
continue
logger.warning(f"Download error: {type(e).__name__}: {e}")
logger.warning("Download error: %s: %s", type(e).__name__, e)
if attempt < MAX_DOWNLOAD_RETRIES - 1:
time.sleep(_backoff_delay(attempt + 1))
attempt += 1
else:
return buffer
logger.error(f"Download failed after {MAX_DOWNLOAD_RETRIES} attempts: {link}")
logger.error("Download failed after %s attempts: %s", MAX_DOWNLOAD_RETRIES, link)
return None
def _is_configured_zlib_host(hostname: str | None) -> bool:
"""Return True when a hostname matches a configured Z-Library mirror."""
if not hostname:
return False
from shelfmark.core.mirrors import get_zlib_cookie_domains
hostname = hostname.lower()
base_domain = ".".join(hostname.split(".")[-2:]) if "." in hostname else hostname
for domain in get_zlib_cookie_domains():
candidate = str(domain).lower()
if hostname == candidate or hostname.endswith(f".{candidate}") or base_domain == candidate:
return True
return False
def _try_resume(
url: str,
buffer: BytesIO,
start_byte: int,
total_size: float,
progress_callback: Optional[Callable[[float], None]],
cancel_flag: Optional[Event],
base_headers: Optional[dict] = None,
) -> Optional[BytesIO]:
progress_callback: Callable[[float], None] | None,
cancel_flag: Event | None,
base_headers: dict | None = None,
) -> BytesIO | None:
"""Try to resume an interrupted download."""
for attempt in range(MAX_RESUME_ATTEMPTS):
logger.info(f"Resuming from {start_byte} bytes (attempt {attempt + 1}/{MAX_RESUME_ATTEMPTS})")
logger.info(
"Resuming from %s bytes (attempt %s/%s)",
start_byte,
attempt + 1,
MAX_RESUME_ATTEMPTS,
)
time.sleep(_backoff_delay(attempt + 1, base=0.5, cap=5.0))
try:
# Try with CF cookies/UA if available
resume_headers = {**(base_headers or DOWNLOAD_HEADERS), 'Range': f'bytes={start_byte}-'}
resume_headers = {
**(base_headers or DOWNLOAD_HEADERS),
"Range": f"bytes={start_byte}-",
}
cookies = _apply_cf_bypass(url, resume_headers)
response = requests.get(
url, stream=True, proxies=get_proxies(url), timeout=REQUEST_TIMEOUT,
headers=resume_headers, cookies=cookies, verify=get_ssl_verify(url)
url,
stream=True,
proxies=get_proxies(url),
timeout=REQUEST_TIMEOUT,
headers=resume_headers,
cookies=cookies,
verify=get_ssl_verify(url),
)
# Check resume support
if response.status_code == 200: # Server doesn't support resume
if response.status_code == _HTTP_STATUS_OK: # Server doesn't support resume
logger.info("Server doesn't support resume")
return None
if response.status_code == 416: # Range not satisfiable
if response.status_code == _HTTP_STATUS_RANGE_NOT_SATISFIABLE: # Range not satisfiable
logger.warning("Range not satisfiable")
return None
if response.status_code != 206:
if response.status_code != _HTTP_STATUS_PARTIAL_CONTENT:
response.raise_for_status()
pbar = tqdm(total=total_size, initial=start_byte, unit='B', unit_scale=True, desc='Resuming')
pbar = tqdm(
total=total_size,
initial=start_byte,
unit="B",
unit_scale=True,
desc="Resuming",
)
for chunk in response.iter_content(chunk_size=8192):
if chunk:
buffer.write(chunk)
@@ -540,14 +685,15 @@ def _try_resume(
pbar.close()
return None
pbar.close()
logger.info(f"Resume completed: {start_byte} bytes")
return buffer
logger.info("Resume completed: %s bytes", start_byte)
except requests.exceptions.RequestException as e:
logger.debug(f"Resume attempt {attempt + 1} failed: {e}")
logger.warning(f"Resume failed after {MAX_RESUME_ATTEMPTS} attempts")
logger.debug("Resume attempt %s failed: %s", attempt + 1, e)
else:
return buffer
logger.warning("Resume failed after %s attempts", MAX_RESUME_ATTEMPTS)
return None
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+34 -11
View File
@@ -1,18 +1,38 @@
"""Output registry and shared types for post-download delivery handlers."""
from __future__ import annotations
from collections.abc import Callable
from dataclasses import dataclass
from pathlib import Path
from threading import Event
from typing import Callable, Optional
from typing import TYPE_CHECKING, Protocol
from shelfmark.core.models import DownloadTask
if TYPE_CHECKING:
from pathlib import Path
from threading import Event
StatusCallback = Callable[[str, Optional[str]], None]
OutputHandler = Callable[[Path, DownloadTask, Event, StatusCallback, bool], Optional[str]]
from shelfmark.core.models import DownloadTask
StatusCallback = Callable[[str, str | None], None]
class OutputHandler(Protocol):
"""Callable contract for post-download output handlers."""
def __call__(
self,
temp_file: Path,
task: DownloadTask,
cancel_flag: Event,
status_callback: StatusCallback,
*,
preserve_source_on_failure: bool = False,
) -> str | None: ...
@dataclass(frozen=True)
class OutputRegistration:
"""Registered output handler with support checks and priority metadata."""
mode: str
supports_task: Callable[[DownloadTask], bool]
handler: OutputHandler
@@ -28,6 +48,8 @@ def register_output(
supports_task: Callable[[DownloadTask], bool],
priority: int = 0,
) -> Callable[[OutputHandler], OutputHandler]:
"""Register an output handler for a named delivery mode."""
def decorator(handler: OutputHandler) -> OutputHandler:
_OUTPUT_REGISTRY.append(
OutputRegistration(
@@ -44,13 +66,14 @@ def register_output(
def load_output_handlers() -> None:
"""Load built-in output handlers exactly once."""
global _OUTPUTS_LOADED
if _OUTPUTS_LOADED:
return
from . import booklore # noqa: F401
from . import email # noqa: F401
from . import folder # noqa: F401
from . import booklore as booklore
from . import email as email
from . import folder as folder
_OUTPUTS_LOADED = True
@@ -65,7 +88,6 @@ def _derive_output_mode(task: DownloadTask) -> str:
Prefer the mode captured at queue time. Fall back to current config for
legacy tasks that do not have `output_mode` populated.
"""
mode = _normalize_output_mode(getattr(task, "output_mode", None))
if mode:
return mode
@@ -80,7 +102,8 @@ def _derive_output_mode(task: DownloadTask) -> str:
return _normalize_output_mode(config.get("BOOKS_OUTPUT_MODE", "folder")) or "folder"
def resolve_output_handler(task: DownloadTask) -> Optional[OutputRegistration]:
def resolve_output_handler(task: DownloadTask) -> OutputRegistration | None:
"""Resolve the best output handler for a download task."""
load_output_handlers()
desired_mode = _derive_output_mode(task)

Some files were not shown because too many files have changed in this diff Show More