Trust only uploaded blobs for deduplication (closes #148) #175

Open
clawbot wants to merge 1 commits from issue-148-interrupted-upload-dedup into next
Collaborator

An interrupted blob upload was silent data loss: the packer writes a blob's chunks, blob_chunks, and blobs rows to the local index before the upload is attempted (internal/blob/packer.go). If the upload failed those rows survived, and the next run deduplicated against chunks whose data never reached storage, then exited 0 with a snapshot that failed to restore (chunk missing from blob map).

Fix (issue option b):

  • Deduplicate only against chunks held by a blob whose uploaded_ts is set (Chunks.ListInUploadedBlobs, used by loadKnownChunks).
  • At the start of every scan, drop blob rows with uploaded_ts IS NULL (their blob_chunks cascade) and any chunks left unreferenced, so the affected data is re-chunked and re-uploaded and a previously interrupted index is repaired.

The startup repair is load-bearing, not just hygiene: CompleteSnapshot attaches any blob holding a referenced chunk, so a lingering un-uploaded blob would be exported and then be missing at restore.

The storage-less path now marks its blobs uploaded in the same transaction that attaches them, so the "a snapshot's blobs are uploaded" invariant holds uniformly.

Test: TestBackupRetryAfterInterruptedUploadIsRestorable fails a blob upload through a storer wrapper, retries on the same index with a working backend, then restores and byte-compares. It fails without the change at plan build and passes with it.

The skipped test named in the issue lives in the unmerged #72 branch and is not present here, so this adds its own reproduction instead of depending on it.

Local make check is green. model: claude-opus-4-8

An interrupted blob upload was silent data loss: the packer writes a blob's `chunks`, `blob_chunks`, and `blobs` rows to the local index before the upload is attempted (`internal/blob/packer.go`). If the upload failed those rows survived, and the next run deduplicated against chunks whose data never reached storage, then exited 0 with a snapshot that failed to restore (`chunk missing from blob map`). Fix (issue option b): - Deduplicate only against chunks held by a blob whose `uploaded_ts` is set (`Chunks.ListInUploadedBlobs`, used by `loadKnownChunks`). - At the start of every scan, drop blob rows with `uploaded_ts IS NULL` (their `blob_chunks` cascade) and any chunks left unreferenced, so the affected data is re-chunked and re-uploaded and a previously interrupted index is repaired. The startup repair is load-bearing, not just hygiene: `CompleteSnapshot` attaches any blob holding a referenced chunk, so a lingering un-uploaded blob would be exported and then be missing at restore. The storage-less path now marks its blobs uploaded in the same transaction that attaches them, so the "a snapshot's blobs are uploaded" invariant holds uniformly. Test: `TestBackupRetryAfterInterruptedUploadIsRestorable` fails a blob upload through a storer wrapper, retries on the same index with a working backend, then restores and byte-compares. It fails without the change at plan build and passes with it. The skipped test named in the issue lives in the unmerged #72 branch and is not present here, so this adds its own reproduction instead of depending on it. Local `make check` is green. model: claude-opus-4-8
clawbot added 1 commit 2026-09-22 01:25:19 +02:00
Trust only uploaded blobs for deduplication (closes #148)
check / check (pull_request) Successful in 2m52s
1073420e8b
An interrupted blob upload left the blob's chunks, blob_chunks, and
blobs rows committed before the upload was attempted, so a later run
deduplicated against data that never reached storage and produced a
snapshot that reported success but could not be restored.

Chosen fix (issue option b): a chunk counts as known only when a blob
holding it has uploaded_ts set, and each run drops un-uploaded blob
rows and the chunks they orphan at startup, so the affected data is
re-chunked and re-uploaded. Option a (withholding all row writes until
the upload succeeds) would entangle the packer's writes with upload
ordering; gated reads plus a startup repair is smaller and self-healing.

Blobs recorded with no remote backend are marked uploaded so the
invariant holds uniformly.

model: claude-opus-4-8
clawbot added the needs-review label 2026-09-22 01:25:32 +02:00
clawbot self-assigned this 2026-09-22 01:25:32 +02:00
All checks were successful
check / check (pull_request) Successful in 2m52s
You are not authorized to merge this pull request.
This pull request can be merged automatically.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin issue-148-interrupted-upload-dedup:issue-148-interrupted-upload-dedup
git checkout issue-148-interrupted-upload-dedup
Sign in to join this conversation.