Store a source the way a variant is stored, one row and one file per source URL (closes #308)
check / check (push) Canceled after 0s

A source is now one source_content row and one file named by its
SourceKey, the SHA-256 of its host, path and query, made as the variant
key is. A store writes the file, then upserts the row, and removes the
file when the row cannot be written; an eviction deletes the row, then
the file, with no lock. source_metadata, the JSON files under
cache/metadata/, MetadataStorage, the content lock, the eviction
transaction and its test hook are gone, as is the reconciliation pass
that dropped a source row whose file is missing. Two source URLs
serving the same bytes store them twice.

Model: opus-5-5
This commit is contained in:
2026-10-21 14:16:47 +00:00
parent 00d8ba26c5
commit 2dd2134f7f
24 changed files with 621 additions and 1467 deletions
+11 -11
View File
@@ -76,10 +76,9 @@ database and the disk cache:
pixa runs as (uid 65532) and sets its mode to `750`; outside it, that user
must be able to write the directory.
- `cache_max_bytes` limits the source and transformed images together. The
database, the metadata files, the `.meta` file beside each transformed image
and files still being written come on top, and eviction runs in the
background, so the cache can pass the limit for a while: leave room on the
volume beyond it.
database, the `.meta` file beside each transformed image and files still being
written come on top, and eviction runs in the background, so the cache can
pass the limit for a while: leave room on the volume beyond it.
- Set `cache_max_bytes` for a lasting deployment. Its default, worked out each
time pixa starts, is 75% of the sum of the space free on the volume and the
space the cached images already take, so a restart keeps the limit the cache
@@ -177,11 +176,9 @@ allowlisted source hosts for open access.
### Storage
- **Source content**:
`<state_dir>/cache/sources/<ab>/<cd>/<sha256 of source content>`
- **Source metadata**:
`<state_dir>/cache/metadata/<hostname>/<sha256 of path and query>.json` (host,
path and query, content hash, upstream status and headers, fetch time)
- **Source images**:
`<state_dir>/cache/sources/<ab>/<cd>/<sha256 of host, path and query>`, one
file for each source URL
- **Database**: `<state_dir>/state.sqlite3` (SQLite)
- **Transformed images**:
`<state_dir>/cache/variants/<ab>/<cd>/<sha256 of host, path, query, size, format, quality and fit>`,
@@ -190,8 +187,11 @@ allowlisted source hosts for open access.
`<ab>` and `<cd>` are the first and second pairs of characters of the file's
name.
Multiple source paths may reference the same content blob; the database tracks
references rather than using filesystem refcounting.
Each source image and each transformed image has one row in the database,
holding its size and when it was last used. Two source URLs that serve the same
bytes are stored twice, once under each URL. A source image is stored by writing
its file and then its row; when the row cannot be written, the file is removed
again.
Source and transformed image files, and the `.meta` file beside each transformed
image, are written under a temporary name, synced to disk, and only then renamed