Trust only uploaded blobs for deduplication (closes #148)
check / check (pull_request) Successful in 2m43s

An interrupted blob upload left the blob's chunks, blob_chunks, and
blobs rows committed before the upload was attempted, so a later run
deduplicated against data that never reached storage and produced a
snapshot that reported success but could not be restored.

Fix (issue option b): a chunk counts as known only when a blob holding
it has uploaded_ts set, and each run drops un-uploaded blob rows and the
chunks they orphan at startup, so the affected data is re-chunked and
re-uploaded. A blob recorded with no remote backend is marked uploaded
so the invariant holds uniformly.

Option b also names the interrupted metadata-export case (blobs
uploaded, manifest never written); that repair is split to a follow-up
issue.

Model: opus-4-8
This commit is contained in:
2026-09-22 07:35:01 +00:00
parent 994e5de613
commit 36d3791ed2
5 changed files with 314 additions and 7 deletions
+24
View File
@@ -208,6 +208,30 @@ func (r *BlobRepository) DeleteOrphaned(ctx context.Context) error {
return nil
}
// DeleteUnuploaded deletes blob rows whose upload never completed
// (uploaded_ts IS NULL) and returns how many were removed. Their
// blob_chunks rows are removed by the ON DELETE CASCADE foreign key.
// A blob is only ever attached to a snapshot once its upload has been
// recorded, so an un-uploaded blob is never referenced by a completed
// snapshot: dropping it discards chunk rows that point at data which
// was never stored remotely, so the affected content is re-chunked and
// re-uploaded on the next run.
func (r *BlobRepository) DeleteUnuploaded(ctx context.Context) (int64, error) {
query := `DELETE FROM blobs WHERE uploaded_ts IS NULL`
result, err := r.db.ExecWithLog(ctx, query)
if err != nil {
return 0, fmt.Errorf("deleting un-uploaded blobs: %w", err)
}
rowsAffected, _ := result.RowsAffected()
if rowsAffected > 0 {
log.Debug("Deleted un-uploaded blobs", "count", rowsAffected)
}
return rowsAffected, nil
}
// getOne fetches a single blob row matched on the given column, or
// (nil, nil) when no row matches.
func (r *BlobRepository) getOne(