Count each file, byte and upload once in backup statistics (closes #225)
check / check (push) Successful in 14m1s

The scanner added a changed file's bytes again for each new chunk and
counted a file as unchanged for each chunk already stored. It now
counts files and bytes once, in the scan phase, and counts its own
uploads, so a --cron run, which has no progress reporter, records
them. The blob count no longer adds earlier paths' blobs again.

The snapshots row now stores the size of all files in total_size and
the referenced blobs' sizes in blob_size, blob_uncompressed_size and
compression_ratio, as docs/DATAMODEL.md says; snapshot_blobs is filled
before the stats for that. DATAMODEL.md now says chunk_count and
blob_count count what the run added.

Removed the uncalled UpdateSnapshotStats, which stored bytes uploaded
in blob_size.

Model: opus-5-5
This commit is contained in:
2026-10-06 20:45:09 +00:00
parent 5d685f03ce
commit f0c30c9fba
10 changed files with 373 additions and 133 deletions
+2 -2
View File
@@ -99,8 +99,8 @@ type Snapshot struct {
StartedAt time.Time
CompletedAt *time.Time // nil if still in progress
FileCount int64
ChunkCount int64
BlobCount int64
ChunkCount int64 // Chunks this snapshot stored that were not stored before
BlobCount int64 // Blobs this snapshot created
TotalSize int64 // Total size of all referenced files
// BlobSize is the total size of all referenced blobs (compressed and
+4 -3
View File
@@ -127,6 +127,7 @@ func (r *SnapshotRepository) UpdateExtendedStats(
snapshotID string,
blobUncompressedSize int64,
compressionLevel int,
uploadBytes int64,
uploadDurationMs int64,
) error {
compressionRatio, err := r.extendedCompressionRatio(
@@ -141,7 +142,7 @@ func (r *SnapshotRepository) UpdateExtendedStats(
SET blob_uncompressed_size = ?,
compression_ratio = ?,
compression_level = ?,
upload_bytes = blob_size,
upload_bytes = ?,
upload_duration_ms = ?
WHERE id = ?
`
@@ -149,11 +150,11 @@ func (r *SnapshotRepository) UpdateExtendedStats(
if tx != nil {
_, err = tx.ExecContext(ctx, query,
blobUncompressedSize, compressionRatio, compressionLevel,
uploadDurationMs, snapshotID)
uploadBytes, uploadDurationMs, snapshotID)
} else {
_, err = r.db.ExecWithLog(ctx, query,
blobUncompressedSize, compressionRatio, compressionLevel,
uploadDurationMs, snapshotID)
uploadBytes, uploadDurationMs, snapshotID)
}
if err != nil {