Keep cached sources and variants in one FileStore type (closes #318) #335

Merged
clawbot merged 1 commits from issue-318-one-file-store into next 2026-10-26 20:24:34 +01:00
Collaborator

Implements #318, item 12 of #168.

The cache now keeps its source images in one imgcache.FileStore and its variants in another. A FileStore is a directory of files named by a hex key, two levels of subdirectories deep, as before. ContentStorage and VariantStorage are gone.

  • FileStore has Write, Open (the file and its size), Size and Delete. Keys are plain strings; the cache converts its source and variant keys at each call.
  • Write is the only caller of writeFileSynced, the write through a synced temp file.
  • Cache.Variant reads the content type from the variant file's first bytes itself. outputContentType and errNotAnOutputFormat moved to cache.go. A hit still opens one file and makes no database call.
  • storage.go is filestore.go. Its tests are TestFileStore_*, plus a new one for Size.
  • internal/imgcache/testdata/red.avif is deleted, since no test used it.

Not in the diff: the directory layout, the write and the reconciliation pass are unchanged. The README already describes them, so it is unchanged.

Disclosures:

  • Judgement call: reconcileVariantRows called VariantStorage.Exists in production. FileStore has no Exists, so it calls Size and keeps a row only when that succeeds, as Exists did.
  • Judgement call: the old Store returned the size written. The moved write test checks the size Open reports instead.
  • The nolint:gosec comments on the two old os.Open calls are gone; gosec raises nothing on the one in FileStore.Open.
  • Unverified: make bench needs libvips on the host, which this host lacks. The benchmark compiles with the tests and serves hits through the same Cache.Variant.

Model: opus-5-5

Implements https://git.eeqj.de/sneak/pixa/issues/318, item 12 of https://git.eeqj.de/sneak/pixa/issues/168. The cache now keeps its source images in one `imgcache.FileStore` and its variants in another. A `FileStore` is a directory of files named by a hex key, two levels of subdirectories deep, as before. `ContentStorage` and `VariantStorage` are gone. - `FileStore` has `Write`, `Open` (the file and its size), `Size` and `Delete`. Keys are plain strings; the cache converts its source and variant keys at each call. - `Write` is the only caller of `writeFileSynced`, the write through a synced temp file. - `Cache.Variant` reads the content type from the variant file's first bytes itself. `outputContentType` and `errNotAnOutputFormat` moved to `cache.go`. A hit still opens one file and makes no database call. - `storage.go` is `filestore.go`. Its tests are `TestFileStore_*`, plus a new one for `Size`. - `internal/imgcache/testdata/red.avif` is deleted, since no test used it. Not in the diff: the directory layout, the write and the reconciliation pass are unchanged. The README already describes them, so it is unchanged. Disclosures: - Judgement call: `reconcileVariantRows` called `VariantStorage.Exists` in production. `FileStore` has no `Exists`, so it calls `Size` and keeps a row only when that succeeds, as `Exists` did. - Judgement call: the old `Store` returned the size written. The moved write test checks the size `Open` reports instead. - The `nolint:gosec` comments on the two old `os.Open` calls are gone; gosec raises nothing on the one in `FileStore.Open`. - Unverified: `make bench` needs libvips on the host, which this host lacks. The benchmark compiles with the tests and serves hits through the same `Cache.Variant`. Model: opus-5-5
clawbot added the needs-review label 2026-10-26 20:13:32 +01:00
clawbot self-assigned this 2026-10-26 20:13:32 +01:00
clawbot added 1 commit 2026-10-26 20:13:32 +01:00
imgcache.FileStore is a directory of files named by a hex key, at
<ab>/<cd>/<key>. It writes a file under a temp name, syncs it and renames
it into place. It opens, reports the size of and deletes the file for a
key. The cache keeps its sources in one FileStore and its variants in
another, so ContentStorage and VariantStorage are gone. FileStore has no
Exists: the reconciliation pass checks a variant's file with Size, and the
tests look for the file on disk. Cache.Variant reads the content type from
the file's first bytes itself. storage.go is now filestore.go, and the
unused testdata/red.avif is gone.

Model: opus-5-5
Author
Collaborator

PASS: one imgcache.FileStore now holds the only copy of the synced temp-file write and the two-level layout, the cache keeps its sources in one and its variants in another, and a cache hit still opens one file and makes no database call.

  • Judgement call: the reconciliation pass's Size call keeps a row in exactly the cases the old Exists did, so it is the same check.
  • Judgement call: the moved write test now checks the size read back from disk instead of a return value that no longer exists, and the sizes the cache records are still checked by TestUsageBytesAccountsSourceAndVariantBytes, so no check is weaker.
  • Unverified: make bench was not run, as it needs libvips on the host.

Model: opus-5-5

PASS: one `imgcache.FileStore` now holds the only copy of the synced temp-file write and the two-level layout, the cache keeps its sources in one and its variants in another, and a cache hit still opens one file and makes no database call. - Judgement call: the reconciliation pass's `Size` call keeps a row in exactly the cases the old `Exists` did, so it is the same check. - Judgement call: the moved write test now checks the size read back from disk instead of a return value that no longer exists, and the sizes the cache records are still checked by `TestUsageBytesAccountsSourceAndVariantBytes`, so no check is weaker. - Unverified: `make bench` was not run, as it needs libvips on the host. Model: opus-5-5
clawbot merged commit ac850823f4 into next 2026-10-26 20:24:34 +01:00
clawbot deleted branch issue-318-one-file-store 2026-10-26 20:24:34 +01:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/pixa#335