Count each file, byte and upload once in backup statistics (closes #225)
check / check (push) Successful in 14m1s
check / check (push) Successful in 14m1s
The scanner added a changed file's bytes again for each new chunk and counted a file as unchanged for each chunk already stored. It now counts files and bytes once, in the scan phase, and counts its own uploads, so a --cron run, which has no progress reporter, records them. The blob count no longer adds earlier paths' blobs again. The snapshots row now stores the size of all files in total_size and the referenced blobs' sizes in blob_size, blob_uncompressed_size and compression_ratio, as docs/DATAMODEL.md says; snapshot_blobs is filled before the stats for that. DATAMODEL.md now says chunk_count and blob_count count what the run added. Removed the uncalled UpdateSnapshotStats, which stored bytes uploaded in blob_size. Model: opus-5-5
This commit is contained in:
@@ -99,8 +99,8 @@ type Snapshot struct {
|
||||
StartedAt time.Time
|
||||
CompletedAt *time.Time // nil if still in progress
|
||||
FileCount int64
|
||||
ChunkCount int64
|
||||
BlobCount int64
|
||||
ChunkCount int64 // Chunks this snapshot stored that were not stored before
|
||||
BlobCount int64 // Blobs this snapshot created
|
||||
TotalSize int64 // Total size of all referenced files
|
||||
|
||||
// BlobSize is the total size of all referenced blobs (compressed and
|
||||
|
||||
@@ -127,6 +127,7 @@ func (r *SnapshotRepository) UpdateExtendedStats(
|
||||
snapshotID string,
|
||||
blobUncompressedSize int64,
|
||||
compressionLevel int,
|
||||
uploadBytes int64,
|
||||
uploadDurationMs int64,
|
||||
) error {
|
||||
compressionRatio, err := r.extendedCompressionRatio(
|
||||
@@ -141,7 +142,7 @@ func (r *SnapshotRepository) UpdateExtendedStats(
|
||||
SET blob_uncompressed_size = ?,
|
||||
compression_ratio = ?,
|
||||
compression_level = ?,
|
||||
upload_bytes = blob_size,
|
||||
upload_bytes = ?,
|
||||
upload_duration_ms = ?
|
||||
WHERE id = ?
|
||||
`
|
||||
@@ -149,11 +150,11 @@ func (r *SnapshotRepository) UpdateExtendedStats(
|
||||
if tx != nil {
|
||||
_, err = tx.ExecContext(ctx, query,
|
||||
blobUncompressedSize, compressionRatio, compressionLevel,
|
||||
uploadDurationMs, snapshotID)
|
||||
uploadBytes, uploadDurationMs, snapshotID)
|
||||
} else {
|
||||
_, err = r.db.ExecWithLog(ctx, query,
|
||||
blobUncompressedSize, compressionRatio, compressionLevel,
|
||||
uploadDurationMs, snapshotID)
|
||||
uploadBytes, uploadDurationMs, snapshotID)
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
|
||||
Reference in New Issue
Block a user