Item 2 of 15 of #168: "Store sources the way variants are stored: one row and one file per key, with no sharing between URLs". Starts when #307 is squash-merged to next.
Today (next at 989da78)
A variant is one variant_content row plus one file named by its key. A source is two rows (source_metadata for the URL, source_content for the bytes, joined on content_hash), a file named by the SHA-256 of its bytes, and a JSON file under cache/metadata/ that nothing reads (MetadataStorage.Load has no caller). Sharing one file between URLs with identical bytes is what forces contentLock (internal/imgcache/contentlock.go) to serialize StoreSource against evictSourceBlob, the transaction in evictSourceBlob that deletes every referencing row before the unlink, sourceReferences, evictSourceBlobTestHook, and the source half of reconcileAccounting.
Recommendation
A source is one row and one file named by a hash of its host, path and query, made the way the variant key is made. Storing is "write the file, upsert the row"; evicting is "delete the row, delete the file"; a row whose file is missing is a miss, as LookupSource already treats it. Two URLs serving identical bytes then store them twice.
This reverses the sentence in README.md "Multiple source paths may reference the same content blob; the database tracks references rather than using filesystem refcounting" and the shared-content design of #51.
Names, recorded here as the choice: the one table keeps the name source_content, beside variant_content; the key is a SourceKey (the proposal in #168) in a column source_key.
Definition of done
One table, source_content, with one row per source URL: its key, host, path, query, content type, size, fetch time and last access time. source_metadata and the content_hash column are gone. 001_schema.sql is edited in place, with no new migration file.
A source's file is named by its key, a SHA-256 of host, path and query made the way the variant key is made, under cache/sources/ with the same two-level fan-out.
Storing a source writes the file, then upserts the row; when the row cannot be written the store fails and the file is removed.
Evicting a source deletes its row, then its file, with no lock and no transaction around them.
contentLock and contentlock.go, the transaction in evictSourceBlob, sourceReferences, evictSourceBlobTestHook, the source half of reconcileAccounting, the JSON files under cache/metadata/ and MetadataStorage are gone.
At startup, source files that have no row and stale temp files are removed. The existing check that removes a cached file that is empty or not the size its row records stays.
New tests: two URLs with identical bytes give two rows and two files; evicting one leaves the other served; a failed row write leaves no file behind.
README.md's Storage section describes this layout, and the sentence about several source paths sharing one content blob is gone.
Approval and tests
sneak asked for each main recommendation of #168 to be its own issue, built before 1.0, one at a time in the order listed (#168 (comment)). That instruction is his approval of this item, including its reversal of the README.md sentence and the shared-content design of #51, and the deletion of the tests of the code it removes. Existing tests change only as this item needs: they follow code that moves or is renamed, and the tests of code this item removes go with it; no other assertion is weakened or dropped.
Model: opus-5-5
Item 2 of 15 of https://git.eeqj.de/sneak/pixa/issues/168: "Store sources the way variants are stored: one row and one file per key, with no sharing between URLs". Starts when https://git.eeqj.de/sneak/pixa/issues/307 is squash-merged to `next`.
## Today (`next` at `989da78`)
A variant is one `variant_content` row plus one file named by its key. A source is two rows (`source_metadata` for the URL, `source_content` for the bytes, joined on `content_hash`), a file named by the SHA-256 of its bytes, and a JSON file under `cache/metadata/` that nothing reads (`MetadataStorage.Load` has no caller). Sharing one file between URLs with identical bytes is what forces `contentLock` (`internal/imgcache/contentlock.go`) to serialize `StoreSource` against `evictSourceBlob`, the transaction in `evictSourceBlob` that deletes every referencing row before the unlink, `sourceReferences`, `evictSourceBlobTestHook`, and the source half of `reconcileAccounting`.
## Recommendation
A source is one row and one file named by a hash of its host, path and query, made the way the variant key is made. Storing is "write the file, upsert the row"; evicting is "delete the row, delete the file"; a row whose file is missing is a miss, as `LookupSource` already treats it. Two URLs serving identical bytes then store them twice.
This reverses the sentence in `README.md` "Multiple source paths may reference the same content blob; the database tracks references rather than using filesystem refcounting" and the shared-content design of https://git.eeqj.de/sneak/pixa/issues/51.
Names, recorded here as the choice: the one table keeps the name `source_content`, beside `variant_content`; the key is a `SourceKey` (the proposal in https://git.eeqj.de/sneak/pixa/issues/168) in a column `source_key`.
## Definition of done
- One table, `source_content`, with one row per source URL: its key, host, path, query, content type, size, fetch time and last access time. `source_metadata` and the `content_hash` column are gone. `001_schema.sql` is edited in place, with no new migration file.
- A source's file is named by its key, a SHA-256 of host, path and query made the way the variant key is made, under `cache/sources/` with the same two-level fan-out.
- Storing a source writes the file, then upserts the row; when the row cannot be written the store fails and the file is removed.
- Evicting a source deletes its row, then its file, with no lock and no transaction around them.
- `contentLock` and `contentlock.go`, the transaction in `evictSourceBlob`, `sourceReferences`, `evictSourceBlobTestHook`, the source half of `reconcileAccounting`, the JSON files under `cache/metadata/` and `MetadataStorage` are gone.
- At startup, source files that have no row and stale temp files are removed. The existing check that removes a cached file that is empty or not the size its row records stays.
- New tests: two URLs with identical bytes give two rows and two files; evicting one leaves the other served; a failed row write leaves no file behind.
- `README.md`'s Storage section describes this layout, and the sentence about several source paths sharing one content blob is gone.
## Approval and tests
sneak asked for each main recommendation of https://git.eeqj.de/sneak/pixa/issues/168 to be its own issue, built before 1.0, one at a time in the order listed (https://git.eeqj.de/sneak/pixa/issues/168#issuecomment-153513). That instruction is his approval of this item, including its reversal of the `README.md` sentence and the shared-content design of https://git.eeqj.de/sneak/pixa/issues/51, and the deletion of the tests of the code it removes. Existing tests change only as this item needs: they follow code that moves or is renamed, and the tests of code this item removes go with it; no other assertion is weakened or dropped.
Model: opus-5-5
clawbot
added this to the 1.0.0 milestone 2026-10-21 12:49:02 +02:00
clawbot
self-assigned this 2026-10-21 12:49:03 +02:00
Built in #324: a source is one source_content row and one file named by its SourceKey; source_metadata, the JSON files under cache/metadata/ and the content lock are gone. One point is open there, on whether the pass that drops a source row whose file is missing goes too: #324 (comment)
Model: opus-5-5
Built in https://git.eeqj.de/sneak/pixa/pulls/324: a source is one `source_content` row and one file named by its `SourceKey`; `source_metadata`, the JSON files under `cache/metadata/` and the content lock are gone. One point is open there, on whether the pass that drops a source row whose file is missing goes too: https://git.eeqj.de/sneak/pixa/pulls/324#issuecomment-153806
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Item 2 of 15 of #168: "Store sources the way variants are stored: one row and one file per key, with no sharing between URLs". Starts when #307 is squash-merged to
next.Today (
nextat989da78)A variant is one
variant_contentrow plus one file named by its key. A source is two rows (source_metadatafor the URL,source_contentfor the bytes, joined oncontent_hash), a file named by the SHA-256 of its bytes, and a JSON file undercache/metadata/that nothing reads (MetadataStorage.Loadhas no caller). Sharing one file between URLs with identical bytes is what forcescontentLock(internal/imgcache/contentlock.go) to serializeStoreSourceagainstevictSourceBlob, the transaction inevictSourceBlobthat deletes every referencing row before the unlink,sourceReferences,evictSourceBlobTestHook, and the source half ofreconcileAccounting.Recommendation
A source is one row and one file named by a hash of its host, path and query, made the way the variant key is made. Storing is "write the file, upsert the row"; evicting is "delete the row, delete the file"; a row whose file is missing is a miss, as
LookupSourcealready treats it. Two URLs serving identical bytes then store them twice.This reverses the sentence in
README.md"Multiple source paths may reference the same content blob; the database tracks references rather than using filesystem refcounting" and the shared-content design of #51.Names, recorded here as the choice: the one table keeps the name
source_content, besidevariant_content; the key is aSourceKey(the proposal in #168) in a columnsource_key.Definition of done
source_content, with one row per source URL: its key, host, path, query, content type, size, fetch time and last access time.source_metadataand thecontent_hashcolumn are gone.001_schema.sqlis edited in place, with no new migration file.cache/sources/with the same two-level fan-out.contentLockandcontentlock.go, the transaction inevictSourceBlob,sourceReferences,evictSourceBlobTestHook, the source half ofreconcileAccounting, the JSON files undercache/metadata/andMetadataStorageare gone.README.md's Storage section describes this layout, and the sentence about several source paths sharing one content blob is gone.Approval and tests
sneak asked for each main recommendation of #168 to be its own issue, built before 1.0, one at a time in the order listed (#168 (comment)). That instruction is his approval of this item, including its reversal of the
README.mdsentence and the shared-content design of #51, and the deletion of the tests of the code it removes. Existing tests change only as this item needs: they follow code that moves or is renamed, and the tests of code this item removes go with it; no other assertion is weakened or dropped.Model: opus-5-5
Built in #324: a source is one
source_contentrow and one file named by itsSourceKey;source_metadata, the JSON files undercache/metadata/and the content lock are gone. One point is open there, on whether the pass that drops a source row whose file is missing goes too: #324 (comment)Model: opus-5-5