Trust only uploaded blobs for deduplication (closes #148)
check / check (pull_request) Successful in 2m52s

An interrupted blob upload left the blob's chunks, blob_chunks, and
blobs rows committed before the upload was attempted, so a later run
deduplicated against data that never reached storage and produced a
snapshot that reported success but could not be restored.

Chosen fix (issue option b): a chunk counts as known only when a blob
holding it has uploaded_ts set, and each run drops un-uploaded blob
rows and the chunks they orphan at startup, so the affected data is
re-chunked and re-uploaded. Option a (withholding all row writes until
the upload succeeds) would entangle the packer's writes with upload
ordering; gated reads plus a startup repair is smaller and self-healing.

Blobs recorded with no remote backend are marked uploaded so the
invariant holds uniformly.

model: claude-opus-4-8
This commit is contained in:
2026-09-21 23:22:29 +00:00
parent 343129f891
commit 1073420e8b
5 changed files with 314 additions and 7 deletions
+24
View File
@@ -208,6 +208,30 @@ func (r *BlobRepository) DeleteOrphaned(ctx context.Context) error {
return nil
}
// DeleteUnuploaded deletes blob rows whose upload never completed
// (uploaded_ts IS NULL) and returns how many were removed. Their
// blob_chunks rows are removed by the ON DELETE CASCADE foreign key.
// A blob is only ever attached to a snapshot once its upload has been
// recorded, so an un-uploaded blob is never referenced by a completed
// snapshot: dropping it discards chunk rows that point at data which
// was never stored remotely, so the affected content is re-chunked and
// re-uploaded on the next run.
func (r *BlobRepository) DeleteUnuploaded(ctx context.Context) (int64, error) {
query := `DELETE FROM blobs WHERE uploaded_ts IS NULL`
result, err := r.db.ExecWithLog(ctx, query)
if err != nil {
return 0, fmt.Errorf("deleting un-uploaded blobs: %w", err)
}
rowsAffected, _ := result.RowsAffected()
if rowsAffected > 0 {
log.Debug("Deleted un-uploaded blobs", "count", rowsAffected)
}
return rowsAffected, nil
}
// getOne fetches a single blob row matched on the given column, or
// (nil, nil) when no row matches.
func (r *BlobRepository) getOne(