Record the stored size of a blob a backup finds already stored #341

Open
clawbot wants to merge 1 commits from issue-300-reused-blob-size into next
Collaborator

Fixes #300.

What was wrong: when a finished blob was already at the destination, snapshot create skipped the upload, but the blob row in the local index kept the size of the local copy, and the manifest and the snapshot's database took it from there. If compression_level or the recipients changed since the stored object was written, the two sizes can differ, and then shallow and deep snapshot verify failed on a good snapshot and the listed sizes were wrong.

What changed: marking a blob uploaded (UpdateUploaded) now also writes its compressed_size, taken from the object in storage: the size Stat reports when the upload is skipped, the size just written otherwise. uploadBlobIfNeeded returns that size. docs/DATAMODEL.md says what compressed_size holds after upload.

TestVerifyPassesWhenBackupReusesStoredBlob backs up a file, backs it up again from an empty local index with an extra recipient, and runs shallow and deep verify on the second snapshot.

  • Judgement call: the test adds a recipient instead of changing the compression level, because an extra age recipient always makes the local copy larger, while two zstd levels need not compress the test data differently.
  • Not changed: the "already exists ... Skipping upload" line and its log record still print the local copy's size.

Model: opus-5-5

Fixes https://git.eeqj.de/sneak/vaultik/issues/300. **What was wrong:** when a finished blob was already at the destination, `snapshot create` skipped the upload, but the blob row in the local index kept the size of the local copy, and the manifest and the snapshot's database took it from there. If `compression_level` or the recipients changed since the stored object was written, the two sizes can differ, and then shallow and deep `snapshot verify` failed on a good snapshot and the listed sizes were wrong. **What changed:** marking a blob uploaded (`UpdateUploaded`) now also writes its `compressed_size`, taken from the object in storage: the size `Stat` reports when the upload is skipped, the size just written otherwise. `uploadBlobIfNeeded` returns that size. `docs/DATAMODEL.md` says what `compressed_size` holds after upload. `TestVerifyPassesWhenBackupReusesStoredBlob` backs up a file, backs it up again from an empty local index with an extra recipient, and runs shallow and deep verify on the second snapshot. - Judgement call: the test adds a recipient instead of changing the compression level, because an extra age recipient always makes the local copy larger, while two zstd levels need not compress the test data differently. - Not changed: the "already exists ... Skipping upload" line and its log record still print the local copy's size. Model: opus-5-5
clawbot added the needs-review label 2026-10-13 06:10:37 +02:00
clawbot self-assigned this 2026-10-13 06:10:37 +02:00
Author
Collaborator
  • docs/DATAMODEL.md (the compressed_size line) and the new TODO.md entry say the stored object's size differs from the local copy's whenever the compression level or the recipients changed. It only can differ: replacing a recipient with another of the same kind gives the same size, and two compression levels can compress the same data to the same size, as the PR body itself says. Acceptable: "can differ" in both places, as the UpdateUploaded comment already says.
  • The branch conflicts with current next in TODO.md only (two Completed Steps entries at the same spot). Acceptable: rebased onto next with both entries kept, this PR's on top.

Unverified: make check on the branch rebased onto current next, and the test suite with the fix reverted; the shared build lock stayed taken for four hours. That the new test fails without the fix was checked by reading the code only.

Model: opus-5-5

- `docs/DATAMODEL.md` (the `compressed_size` line) and the new `TODO.md` entry say the stored object's size differs from the local copy's whenever the compression level or the recipients changed. It only can differ: replacing a recipient with another of the same kind gives the same size, and two compression levels can compress the same data to the same size, as the PR body itself says. Acceptable: "can differ" in both places, as the `UpdateUploaded` comment already says. - The branch conflicts with current `next` in `TODO.md` only (two Completed Steps entries at the same spot). Acceptable: rebased onto `next` with both entries kept, this PR's on top. Unverified: `make check` on the branch rebased onto current `next`, and the test suite with the fix reverted; the shared build lock stayed taken for four hours. That the new test fails without the fix was checked by reading the code only. Model: opus-5-5
clawbot added needs-rework and removed needs-review labels 2026-10-13 10:59:47 +02:00
clawbot force-pushed issue-300-reused-blob-size from e8ed3d0c27 to bfebf3d0c0 2026-10-13 11:01:54 +02:00 Compare
clawbot added 1 commit 2026-10-13 12:02:21 +02:00
When a finished blob already existed at the destination, the upload was
skipped but the blob row, and so the manifest, kept the size of the
local copy. That copy can differ from the stored object when
compression_level or the recipients changed since it was written, and
then shallow and deep verify failed on a good snapshot. Marking a blob
uploaded now records the size of the object in storage: for an existing
blob, the size Stat reports.

Judgement call: the test adds a recipient rather than changing the
compression level, because an extra age recipient always makes the
local copy larger, while two zstd levels need not compress the test
data differently.

Model: opus-5-5
clawbot force-pushed issue-300-reused-blob-size from bfebf3d0c0 to c5bcd49b03 2026-10-13 12:02:21 +02:00 Compare
Some checks are pending
check / check (push) Waiting to run
You are not authorized to merge this pull request.
This pull request can be merged automatically.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin issue-300-reused-blob-size:issue-300-reused-blob-size
git checkout issue-300-reused-blob-size
Sign in to join this conversation.