I went through what Shelfmark does per search and per cover and measured
the pieces that showed up. Four of them are cheap to fix, so this is one
PR. Shelfmark runs on a single gevent worker, so any of these blocks
every other request while it runs, and I'm assuming a lot of people run
it on small hardware (a Pi, say).
Cover cache. Every cache hit rewrote the whole `cache_index.json` just
to bump the entry's last-access time. On my box that is 3.7 ms of CPU
per hit at 500 entries and about 100 ms at 15,000 (2.2 MB written each
time), and a results page asks for around 40 covers. The access time now
stays in memory and the index is written at most every 30 seconds, or
whenever something else saves it. A crash inside that window loses some
access times, which only changes eviction order.
Settings reload. `config.refresh()` rebuilt the cache by reading and
parsing a tab's JSON file once per field, so about 250 parses per
refresh, 100 ms of CPU, and the metadata search, providers and config
routes each trigger it. It now reads each file once per rebuild (several
tabs share `settings.json`, so the memo is keyed on the file path).
Library match. When the library check is on, each search result is
compared against every library entry, and the title was split into words
again for every entry. The words are now computed once per result. I
measured 33 ms per result at 5,000 library entries and 106 ms at 20,000,
which is a second or more per page of 40 results. The matching itself is
unchanged.
Cover URL probe. `is_covers_cache_enabled()` creates and deletes a file
in the config directory to check it is writable, and it runs for every
cover, queue task and history row. The answer is remembered for 60
seconds. The probe is 0.045 ms on its own, but it is called a few
hundred times per request in the worst case.
Each change has a test that fails without it (call counts on the index
saves, file opens during a refresh, tokenisation calls, and probe
calls). Ruff, basedpyright on the changed files and the non-e2e suite
pass. The timings are from a dev machine with an empty config, so the
real numbers depend on cache and library size.
A signed-in user picks the release fields of POST /api/releases/download, and a record
id with the "oceanofpdf:" prefix reached the provider without handles() or is_enabled().
The provider then fetched source_url, POSTed to the form action found on that page and
fetched the meta-refresh target, with no check on any of them.
The provider now refuses to run when it is switched off, and the book page, the form
action and every POST redirect hop must be on the configured mirror (same scheme, host
and port). The signed file link may sit on another host but must be http(s).
network.ensure_public_destination refuses a host that resolves to a loopback, private,
link-local or otherwise non-global address (IPv4-mapped IPv6 judged by its IPv4 address).
Hosts the operator configured as mirrors and names that do not resolve pass as before.
html_get_page and download_url call it on their first URL and check each redirect hop
through a response hook. The image cache uses the shared address check too.
Co-authored-by: CaliBrain <calibrain@l4n.xyz>
`/api/covers/<id>?url=...` cached the fetched image under the id in the
path, and the caller chooses that id. A request with another book's id
and a different URL put its image in that book's cache entry, and every
user then got it.
The key is now derived on the server from the image URL (a hash of it),
and the id in the path stays only so links keep their shape. The SSRF
checks and `fetch_and_cache` are unchanged, and `transform_cover_url`
links already carry the URL, so they keep working. The URL is decoded
first now, so a missing one is a 404 and bad encoding a 400, both before
any fetch.
I made the key the URL alone, not URL plus id. Either stops the
overwrite, and the URL alone lets one image be shared when the same
cover appears under two ids (the viewer redaction rewrites ids and still
hits the cache).
Entries cached under the old keys are not read any more, since there is
no telling which were overwritten. They age out of the LRU and the cache
warms once after the upgrade.
Tests: a request with another id can't replace the legitimate image, in
either order; the normal miss then hit; the same URL under two ids hits;
no URL gives 404; bad encoding gives 400 without a fetch; and a unit
test for the key. Ruff, basedpyright and the backend suite pass.
splitsec2
Alex