Store a source the way a variant is stored, one row and one file per source URL (closes #308)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
A source is now one source_content row and one file named by its SourceKey, the SHA-256 of its host, path and query, made as the variant key is. A store writes the file, then upserts the row, and removes the file when the row cannot be written; an eviction deletes the row, then the file, with no lock. source_metadata, the JSON files under cache/metadata/, MetadataStorage, the content lock, the eviction transaction and its test hook are gone, as is the reconciliation pass that dropped a source row whose file is missing. Two source URLs serving the same bytes store them twice. Model: opus-5-5
This commit was merged in pull request #324.
This commit is contained in:
@@ -76,10 +76,9 @@ database and the disk cache:
|
||||
pixa runs as (uid 65532) and sets its mode to `750`; outside it, that user
|
||||
must be able to write the directory.
|
||||
- `cache_max_bytes` limits the source and transformed images together. The
|
||||
database, the metadata files, the `.meta` file beside each transformed image
|
||||
and files still being written come on top, and eviction runs in the
|
||||
background, so the cache can pass the limit for a while: leave room on the
|
||||
volume beyond it.
|
||||
database, the `.meta` file beside each transformed image and files still being
|
||||
written come on top, and eviction runs in the background, so the cache can
|
||||
pass the limit for a while: leave room on the volume beyond it.
|
||||
- Set `cache_max_bytes` for a lasting deployment. Its default, worked out each
|
||||
time pixa starts, is 75% of the sum of the space free on the volume and the
|
||||
space the cached images already take, so a restart keeps the limit the cache
|
||||
@@ -177,11 +176,9 @@ allowlisted source hosts for open access.
|
||||
|
||||
### Storage
|
||||
|
||||
- **Source content**:
|
||||
`<state_dir>/cache/sources/<ab>/<cd>/<sha256 of source content>`
|
||||
- **Source metadata**:
|
||||
`<state_dir>/cache/metadata/<hostname>/<sha256 of path and query>.json` (host,
|
||||
path and query, content hash, upstream status and headers, fetch time)
|
||||
- **Source images**:
|
||||
`<state_dir>/cache/sources/<ab>/<cd>/<sha256 of host, path and query>`, one
|
||||
file for each source URL
|
||||
- **Database**: `<state_dir>/state.sqlite3` (SQLite)
|
||||
- **Transformed images**:
|
||||
`<state_dir>/cache/variants/<ab>/<cd>/<sha256 of host, path, query, size, format, quality and fit>`,
|
||||
@@ -190,8 +187,11 @@ allowlisted source hosts for open access.
|
||||
`<ab>` and `<cd>` are the first and second pairs of characters of the file's
|
||||
name.
|
||||
|
||||
Multiple source paths may reference the same content blob; the database tracks
|
||||
references rather than using filesystem refcounting.
|
||||
Each source image and each transformed image has one row in the database,
|
||||
holding its size and when it was last used. Two source URLs that serve the same
|
||||
bytes are stored twice, once under each URL. A source image is stored by writing
|
||||
its file and then its row; when the row cannot be written, the file is removed
|
||||
again.
|
||||
|
||||
Source and transformed image files, and the `.meta` file beside each transformed
|
||||
image, are written under a temporary name, synced to disk, and only then renamed
|
||||
|
||||
Reference in New Issue
Block a user