5 Commits
Author SHA1 Message Date
sneak 0b6d69fbd6 Stop restore --verify re-reporting files that failed to restore (closes #323)
check / check (push) Waiting to run
With --skip-errors, a file that failed to restore was verified anyway,
failed again because it was never written, and was added to the same
failure list as the restore failure. restore --verify then listed it
twice and called every entry a checksum mismatch.

Verification now skips the files that failed to restore and keeps its
own failures in a separate list, so failed restores and failed
verifications are reported apart, each path once. The restore still
exits non-zero. The tests cover a restore with skipped files where
nothing fails verification, and one where another restored file also
fails it.

Judgement call: the second test has its own trailing-byte file-system
wrapper; the one in restore_perms_test.go is in the internal test package.

Model: opus-5-5
2026-10-13 15:19:58 +00:00
clawbot e48bafccef Restore the other files when a chunk is in no blob under --skip-errors (closes #326)
check / check (push) Waiting to run
Restore planning stopped at the first file needing a chunk that no blob
holds, so --skip-errors never saw that file and nothing was restored.
The planner now leaves such a file out of the plan and returns it with
the reason, and each one goes through handleRestoreFileError like any
other failed file: skipped and counted under --skip-errors, fatal
without it. The fatal error now suggests --skip-errors, as other restore
failures do.

The skip-errors tests now share one backup helper; the new case removes
a blob from the snapshot metadata and exports it again.

Model: opus-5-5
2026-10-13 16:53:05 +02:00
clawbot bffd4578fc Record each snapshot's new chunk size when its backup runs (closes #311)
check / check (push) Waiting to run
snapshot list worked out NEW CHUNK SIZE from the snapshot's file rows. A
file's row is shared by every snapshot that lists it and rewritten when
the file changes, so a snapshot whose change was a modified file showed
0 B, and older snapshots' figures moved after later backups.

The scanner now totals the size of the chunks it stores that were not
stored before (the chunks chunk_count counts), the backup writes that
total to a new snapshots column, new_chunk_size, and snapshot list reads
it.

Judgement call: the figure is fixed when the backup runs, so removing an
earlier snapshot no longer raises a later snapshot's figure.

Model: opus-5-5
2026-10-13 16:44:14 +02:00
clawbot f4ab1ba5ae Exit as interrupted when Ctrl-C lands during a local index write (closes #301)
check / check (push) Waiting to run
Once the run's context is cancelled, database/sql rolls an open
transaction back itself, so the rollback in WithTx returned
sql.ErrTxDone. WithTx treated any rollback error as fatal and called
os.Exit(1): an interrupted backup printed "FATAL: failed to rollback
transaction", exited 1 instead of 130 and skipped its deferred cleanup.

WithTx now ignores sql.ErrTxDone from the rollback and returns the
body's error alone. A rollback that really fails is returned together
with the body's error instead of ending the process; on a panic it is
dropped and the panic carries on. The cancel test checks that the
returned error carries no sql.ErrTxDone; the other tests make the
rollback really fail by running ROLLBACK inside the body.

Model: opus-5-5
2026-10-13 16:28:59 +02:00
clawbot 005927a7fc Print snapshot list and purge sizes like every other command (closes #312)
check / check (push) Waiting to run
`snapshot list`, the `snapshot purge` listing and the `bytes` value in
terminal log lines divided by 1024 while labelling the result KB or MB,
so 6,329,198 bytes read 6.0 MB there and 6.3 MB in every other command.
All three now use humanize.Bytes, as the rest of the program does, and
the 1024-based helper in internal/vaultik is removed.

The size shared by the list tests is now 5,000,000 bytes, which the two
ways of counting print differently.

Judgement call: the log handler's copy of the formatter is changed too,
since the issue asks that every command print a byte count the same way.

Model: opus-5-5
2026-10-13 16:22:03 +02:00
19 changed files with 653 additions and 128 deletions
+7 -5
View File
@@ -293,8 +293,8 @@ needed.
**`snapshot list`**: Show every snapshot known to this host — the union
of the local index and the backup destination store — with timestamps
and three sizes per snapshot (compressed remote size; total
uncompressed chunk size; size of chunks newly referenced by that
snapshot).
uncompressed chunk size; size of the chunks that snapshot stored that
were not stored before, recorded when its backup ran).
Listing the destination store does **not** require the age secret key,
so it works in vaultik's intended configuration, where the backed-up
@@ -307,8 +307,8 @@ only the hash, so hostname and snapshot name exist solely in the local
index and in the encrypted per-snapshot database. Snapshots found only
on the destination store are therefore listed as
`<remote only:<abbreviated remote key>>` and show `<remote only>` in
the uncompressed and "new chunk" columns, which can only be computed
from the local index. Their timestamp and compressed size are real,
the uncompressed and "new chunk" columns, which come only from the
local index. Their timestamp and compressed size are real,
read from the manifest.
Snapshots in the local index with no counterpart on the destination
@@ -375,7 +375,9 @@ on the destination in one go, use `vaultik remote nuke --force`.
* Optional path arguments to restore specific files/directories (default: all)
* Preserves file permissions, timestamps, ownership (ownership requires root),
symlinks, and empty directories
* `--verify`: After restoring, verify every file's chunk hashes match
* `--verify`: After restoring, verify every restored file's chunk hashes
match. A file skipped under `--skip-errors` is reported as failed to
restore and is not verified.
**`prune`**: Tidy up everything that isn't needed. Runs three passes:
(1) reconcile the local index against the destination store — any
+1
View File
@@ -119,6 +119,7 @@ Tracks backup snapshots.
- `completed_at` (INTEGER) - Completion timestamp (NULL if in progress)
- `file_count` (INTEGER) - Number of files in snapshot
- `chunk_count` (INTEGER) - Number of chunks this snapshot stored that were not stored before
- `new_chunk_size` (INTEGER) - Total size of those chunks before compression; `snapshot list` shows it as NEW CHUNK SIZE
- `blob_count` (INTEGER) - Number of blobs this snapshot created
- `total_size` (INTEGER) - Total size of all files
- `blob_size` (INTEGER) - Total compressed size of all referenced blobs
+12 -9
View File
@@ -3,6 +3,7 @@ package database
import (
"context"
"database/sql"
"errors"
"fmt"
)
@@ -60,22 +61,24 @@ func (r *Repositories) WithTx(ctx context.Context, fn TxFunc) error {
defer func() {
if p := recover(); p != nil {
rollbackErr := tx.Rollback()
if rollbackErr != nil {
Fatalf("failed to rollback transaction: %v", rollbackErr)
}
// The panic is the failure to report, so a rollback error
// is dropped.
_ = tx.Rollback()
panic(p)
} else if err != nil {
rollbackErr := tx.Rollback()
if rollbackErr != nil {
Fatalf("failed to rollback transaction: %v", rollbackErr)
}
}
}()
err = fn(ctx, tx)
if err != nil {
// Once ctx is cancelled, database/sql rolls the transaction
// back itself and Rollback returns sql.ErrTxDone.
rollbackErr := tx.Rollback()
if rollbackErr != nil && !errors.Is(rollbackErr, sql.ErrTxDone) {
return fmt.Errorf("%w; rolling back transaction: %w",
err, rollbackErr)
}
return err
}
+120
View File
@@ -4,6 +4,7 @@ import (
"context"
"database/sql"
"errors"
"strings"
"testing"
"time"
@@ -235,6 +236,125 @@ func TestRepositoriesTransactionRollback(t *testing.T) {
}
}
// When the context is cancelled during the body, database/sql rolls the
// transaction back itself, so WithTx's own rollback finds it already
// done. WithTx must return the body's error alone, without that rollback.
func TestRepositoriesTransactionContextCancelled(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
repos := database.NewRepositories(db)
err := repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
file := &database.File{
Path: "/test/cancelled_file.txt",
MTime: time.Now().Truncate(time.Second),
Size: 1024,
Mode: 0644,
UID: 1000,
GID: 1000,
}
err := repos.Files.Create(ctx, tx, file)
if err != nil {
return err
}
cancel()
// database/sql rolls back from a goroutine of its own; wait
// until it has, so the test does not depend on that race.
for {
_, err = tx.ExecContext(context.WithoutCancel(ctx), "SELECT 1")
if errors.Is(err, sql.ErrTxDone) {
return ctx.Err()
}
time.Sleep(time.Millisecond)
}
})
if !errors.Is(err, context.Canceled) {
t.Fatalf("expected context.Canceled, got: %v", err)
}
if errors.Is(err, sql.ErrTxDone) {
t.Fatalf("the already-done rollback was reported: %v", err)
}
file, err := repos.Files.GetByPath(
context.Background(), "/test/cancelled_file.txt",
)
if err != nil {
t.Fatalf("error checking for file: %v", err)
}
if file != nil {
t.Error("file should not exist after the cancelled transaction")
}
}
// A body that ends the transaction itself with ROLLBACK makes WithTx's
// own rollback fail. WithTx must return that failure together with the
// body's error.
func TestRepositoriesTransactionRollbackFails(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repos := database.NewRepositories(db)
err := repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
_, err := tx.ExecContext(ctx, "ROLLBACK")
if err != nil {
return err
}
return errIntentionalRollback
})
if !errors.Is(err, errIntentionalRollback) {
t.Fatalf("expected the body's error, got: %v", err)
}
if !strings.Contains(err.Error(), "rolling back transaction") {
t.Errorf("expected the failed rollback in the error, got: %v", err)
}
}
// When the body ends the transaction itself with ROLLBACK and then
// panics, WithTx's own rollback fails; the panic must still reach the
// caller.
func TestRepositoriesTransactionPanicRollbackFails(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repos := database.NewRepositories(db)
defer func() {
if recover() == nil {
t.Error("expected the body's panic to reach the caller")
}
}()
_ = repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
_, err := tx.ExecContext(ctx, "ROLLBACK")
if err != nil {
return err
}
panic(errIntentionalRollback)
})
}
func TestRepositoriesReadTransaction(t *testing.T) {
t.Parallel()
+1
View File
@@ -86,6 +86,7 @@ CREATE TABLE IF NOT EXISTS snapshots (
completed_at INTEGER,
file_count INTEGER NOT NULL DEFAULT 0,
chunk_count INTEGER NOT NULL DEFAULT 0,
new_chunk_size INTEGER NOT NULL DEFAULT 0,
blob_count INTEGER NOT NULL DEFAULT 0,
total_size INTEGER NOT NULL DEFAULT 0,
blob_size INTEGER NOT NULL DEFAULT 0,
+14 -33
View File
@@ -129,6 +129,7 @@ func (r *SnapshotRepository) UpdateExtendedStats(
compressionLevel int,
uploadBytes int64,
uploadDurationMs int64,
newChunkSize int64,
) error {
compressionRatio, err := r.extendedCompressionRatio(
ctx, tx, snapshotID, blobUncompressedSize,
@@ -143,18 +144,19 @@ func (r *SnapshotRepository) UpdateExtendedStats(
compression_ratio = ?,
compression_level = ?,
upload_bytes = ?,
upload_duration_ms = ?
upload_duration_ms = ?,
new_chunk_size = ?
WHERE id = ?
`
if tx != nil {
_, err = tx.ExecContext(ctx, query,
blobUncompressedSize, compressionRatio, compressionLevel,
uploadBytes, uploadDurationMs, snapshotID)
uploadBytes, uploadDurationMs, newChunkSize, snapshotID)
} else {
_, err = r.db.ExecWithLog(ctx, query,
blobUncompressedSize, compressionRatio, compressionLevel,
uploadBytes, uploadDurationMs, snapshotID)
uploadBytes, uploadDurationMs, newChunkSize, snapshotID)
}
if err != nil {
@@ -594,45 +596,24 @@ func (r *SnapshotRepository) GetSnapshotUncompressedChunkSize(
return totalSize, nil
}
// GetSnapshotNewChunkSize returns the sum of plaintext sizes of chunks that are
// referenced by this snapshot but not by any earlier completed snapshot known to
// the local database. The result is the marginal uncompressed data this snapshot
// added to the dedup pool — i.e., the delta from prior snapshots.
// GetSnapshotNewChunkSize returns the total plaintext size of the chunks the
// snapshot's backup stored that were not stored before, as the backup
// recorded it. It is not derived from the snapshot's files: a file's row is
// shared by every snapshot that lists it and is rewritten when the file
// changes, so it shows only the file's latest chunks.
func (r *SnapshotRepository) GetSnapshotNewChunkSize(
ctx context.Context, snapshotID string,
) (int64, error) {
query := `
WITH this_snap_chunks AS (
SELECT DISTINCT fc.chunk_hash
FROM snapshot_files sf
JOIN file_chunks fc ON sf.file_id = fc.file_id
WHERE sf.snapshot_id = ?
),
prior_chunks AS (
SELECT DISTINCT fc.chunk_hash
FROM snapshots s
JOIN snapshot_files sf ON sf.snapshot_id = s.id
JOIN file_chunks fc ON fc.file_id = sf.file_id
WHERE s.completed_at IS NOT NULL
AND s.id != ?
AND s.started_at < (SELECT started_at FROM snapshots WHERE id = ?)
)
SELECT COALESCE(SUM(c.size), 0)
FROM chunks c
JOIN this_snap_chunks t ON c.chunk_hash = t.chunk_hash
WHERE c.chunk_hash NOT IN (SELECT chunk_hash FROM prior_chunks)
`
query := `SELECT new_chunk_size FROM snapshots WHERE id = ?`
var totalSize int64
var newChunkSize int64
err := r.db.conn.QueryRowContext(
ctx, query, snapshotID, snapshotID, snapshotID,
).Scan(&totalSize)
err := r.db.conn.QueryRowContext(ctx, query, snapshotID).Scan(&newChunkSize)
if err != nil {
return 0, fmt.Errorf("querying new chunk size: %w", err)
}
return totalSize, nil
return newChunkSize, nil
}
// GetIncompleteSnapshots returns all snapshots that haven't been completed
+7 -10
View File
@@ -10,6 +10,8 @@ import (
"sync"
"time"
"unicode"
"github.com/dustin/go-humanize"
)
// groupSeparator joins an open group path to an attribute key. This
@@ -304,18 +306,13 @@ func formatDuration(d time.Duration) string {
}
}
// formatBytes formats bytes in a human-readable way
// formatBytes formats a byte count with humanize.Bytes, as every command
// prints one. humanize.Bytes takes no negative count, so one is printed
// as it is.
func formatBytes(b int64) string {
const unit = 1024
if b < unit {
if b < 0 {
return fmt.Sprintf("%d B", b)
}
div, exp := int64(unit), 0
for n := b / unit; n >= unit; n /= unit {
div *= unit
exp++
}
return fmt.Sprintf("%.1f %cB", float64(b)/float64(div), "KMGTPE"[exp])
return humanize.Bytes(uint64(b))
}
+5 -4
View File
@@ -200,7 +200,8 @@ func TestTTYHandlerWithGroupQualifiesKeys(t *testing.T) {
func TestTTYHandlerByteFormattingSurvivesGrouping(t *testing.T) {
t.Parallel()
const oneAndAHalfKiB = 1536
// Printed as "1.0 MB", in powers of 1000 like every other byte count.
const oneMegabyte = 1_000_000
for name, testCase := range map[string]struct {
derive func(*slog.Logger) *slog.Logger
@@ -223,13 +224,13 @@ func TestTTYHandlerByteFormattingSurvivesGrouping(t *testing.T) {
var buf bytes.Buffer
logger := slog.New(log.NewTTYHandler(&buf, debugHandlerOptions()))
testCase.derive(logger).Info("uploaded", "bytes", oneAndAHalfKiB)
testCase.derive(logger).Info("uploaded", "bytes", oneMegabyte)
line := ansiEscape.ReplaceAllString(buf.String(), "")
assert.Contains(t, line, testCase.key+"=1.5 KB",
assert.Contains(t, line, testCase.key+"=1.0 MB",
"a byte count must be human-readable however it is qualified")
assert.NotContains(t, line, strconv.Itoa(oneAndAHalfKiB),
assert.NotContains(t, line, strconv.Itoa(oneMegabyte),
"the raw number must not survive the formatting")
})
}
+8 -4
View File
@@ -135,7 +135,9 @@ type ScannerConfig struct {
// are counted per file: BytesScanned is the size of the new and changed
// files, BytesSkipped that of the unchanged ones. FilesFailed counts the
// new and changed files that phase 2 could not store; FilesScanned
// includes them and BytesScanned does not.
// includes them and BytesScanned does not. ChunksCreated counts the
// chunks that were not already stored, and NewChunkSize is their total
// size.
type ScanResult struct {
FilesScanned int
FilesSkipped int
@@ -145,6 +147,7 @@ type ScanResult struct {
BytesSkipped int64
BytesDeleted int64
ChunksCreated int
NewChunkSize int64
BlobsCreated int
BlobsUploaded int
BytesUploaded int64
@@ -1866,11 +1869,12 @@ func (s *Scanner) processFileStreaming(
return nil
}
// updateChunkStats counts a chunk that was not already stored. The scan
// result's file counts, BytesScanned and BytesSkipped are not touched
// here: the scan phase counts each file once.
// updateChunkStats counts a chunk that was not already stored, and its
// size. The scan result's file counts, BytesScanned and BytesSkipped are
// not touched here: the scan phase counts each file once.
func (s *Scanner) updateChunkStats(chunkSize int64, result *ScanResult) {
result.ChunksCreated++
result.NewChunkSize += chunkSize
if s.progress != nil {
s.progress.GetStats().ChunksCreated.Add(1)
+2
View File
@@ -199,6 +199,7 @@ func (sm *SnapshotManager) UpdateSnapshotStatsExtended(
stats.CompressionLevel,
stats.BytesUploaded,
stats.UploadDurationMs,
stats.NewChunkSize,
)
})
}
@@ -906,6 +907,7 @@ type ExtendedBackupStats struct {
BlobUncompressedSize int64 // Total uncompressed size of all referenced blobs
CompressionLevel int // Compression level used for this snapshot
UploadDurationMs int64 // Total milliseconds spent uploading to S3
NewChunkSize int64 // Total size of the chunks counted in ChunksCreated
}
// deleteOtherSnapshots deletes all snapshots except the current one
-16
View File
@@ -89,22 +89,6 @@ type SnapshotInfo struct {
RemotePresent *bool `json:"remote_present"`
}
// formatBytes formats bytes in a human-readable format
func formatBytes(bytes int64) string {
const unit = 1024
if bytes < unit {
return fmt.Sprintf("%d B", bytes)
}
div, exp := int64(unit), 0
for n := bytes / unit; n >= unit; n /= unit {
div *= unit
exp++
}
return fmt.Sprintf("%.1f %cB", float64(bytes)/float64(div), "KMGTPE"[exp])
}
// parseSnapshotTimestamp extracts the timestamp from a snapshot ID
// Format: hostname_snapshotname_2026-01-12T14:41:15Z
func parseSnapshotTimestamp(snapshotID string) (time.Time, error) {
+133
View File
@@ -0,0 +1,133 @@
package vaultik_test
import (
"bytes"
"context"
"maps"
"path/filepath"
"testing"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/chunker"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// TestNewChunkSizeOfChangedFileStaysFixed covers
// https://git.eeqj.de/sneak/vaultik/issues/311. A backup whose only change
// is a modified file lists the size of the chunks that change added as its
// NEW CHUNK SIZE, and a later backup that changes the file again leaves
// every earlier snapshot's figure as it was. The file's row in the local
// index is shared by all three snapshots and rewritten by each backup.
func TestNewChunkSizeOfChangedFileStaysFixed(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
fs := afero.NewOsFs()
tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "src")
dbPath := filepath.Join(tempDir, "index.sqlite")
changedPath := filepath.Join(dataDir, "changed.bin")
ctx := context.Background()
files := writeFaultSourceTree(t, fs, dataDir)
cfg := changedFileConfig(dataDir, dbPath)
appendRandomBytes(t, fs, files, changedPath, appendedFileSize)
store, err := storage.NewFileStorer(filepath.Join(tempDir, "remote"))
require.NoError(t, err)
db, err := database.New(ctx, dbPath)
require.NoError(t, err)
t.Cleanup(func() { _ = db.Close() })
repos := database.NewRepositories(db)
v := newBackupVaultik(ctx, cfg, store, repos, db, fs)
require.NoError(t, backUp(v, "first"))
firstFiles := maps.Clone(files)
firstID := localSnapshotID(ctx, t, repos, "first")
appendRandomBytes(t, fs, files, changedPath, faultChunkSize)
require.NoError(t, backUp(v, "second"))
secondID := localSnapshotID(ctx, t, repos, "second")
secondAdded := sizeOfNewChunks(t, files, firstFiles)
require.Positive(t, secondAdded)
assert.Equal(t, secondAdded, listedNewChunkSizes(t, v)[secondID],
"the second snapshot must count the chunks the change added")
appendRandomBytes(t, fs, files, changedPath, faultChunkSize)
require.NoError(t, backUp(v, "third"))
sizes := listedNewChunkSizes(t, v)
assert.Equal(t, sizeOfNewChunks(t, firstFiles, nil), sizes[firstID],
"a later backup must not change the first snapshot's figure")
assert.Equal(t, secondAdded, sizes[secondID],
"a later backup must not change the second snapshot's figure")
}
// sizeOfNewChunks returns the total size of the distinct chunks of the
// files in after that no file in before has, cut the way the backup cuts
// them.
func sizeOfNewChunks(t *testing.T, after, before map[string][]byte) int64 {
t.Helper()
known := make(map[string]bool)
for _, chunk := range chunksOf(t, before) {
known[chunk.Hash] = true
}
var total int64
for _, chunk := range chunksOf(t, after) {
if !known[chunk.Hash] {
known[chunk.Hash] = true
total += chunk.Size
}
}
return total
}
// chunksOf returns the chunks of every file in files.
func chunksOf(t *testing.T, files map[string][]byte) []chunker.Chunk {
t.Helper()
c := chunker.NewChunker(faultChunkSize)
var all []chunker.Chunk
for _, content := range files {
chunks, err := c.ChunkReader(bytes.NewReader(content))
require.NoError(t, err)
all = append(all, chunks...)
}
return all
}
// listedNewChunkSizes runs `snapshot list --json` and returns each listed
// snapshot's new_chunk_size by snapshot ID.
func listedNewChunkSizes(t *testing.T, v *vaultik.Vaultik) map[string]int64 {
t.Helper()
var out bytes.Buffer
v.Stdout = &out
require.NoError(t, v.ListSnapshots(true))
sizes := make(map[string]int64)
for _, row := range decodeListJSON(t, out.String()) {
sizes[row.ID] = row.NewChunkSize
}
return sizes
}
+37 -11
View File
@@ -103,11 +103,12 @@ type RestoreResult struct {
BlobsDownloaded int
BytesDownloaded int64
Duration time.Duration
FilesFailed int
FailedFiles []string // Paths of files that failed to restore
// Verification results (only populated if Verify option is set)
FilesVerified int
BytesVerified int64
FilesFailed int
FailedFiles []string // Paths of files that failed verification
VerifyFailedFiles []string // Paths of restored files that failed verification
}
// Restore restores files from a snapshot to the target directory
@@ -309,11 +310,22 @@ func (v *Vaultik) restoreAllFiles(
return nil, err
}
plan, err := newRestorePlan(v.ctx, repos, files, chunkToBlobMap, blobIDToHash)
plan, unplanned, err := newRestorePlan(
v.ctx, repos, files, chunkToBlobMap, blobIDToHash)
if err != nil {
return nil, fmt.Errorf("building restore plan: %w", err)
}
// Ranging over files rather than the map reports them in a stable order.
for _, file := range files {
if planErr, ok := unplanned[file.ID]; ok {
err = v.handleRestoreFileError(plan, opts, result, file, file.ID, planErr)
if err != nil {
return nil, err
}
}
}
filesByID, totalBytesExpected := indexRestoreFiles(files)
v.UI.Beginf("Restoring %s files (%s) to %s.",
@@ -647,20 +659,34 @@ func (v *Vaultik) handleRestoreVerification(
opts *RestoreOptions,
result *RestoreResult,
) error {
err := v.verifyRestoredFiles(v.ctx, repos, files, opts.TargetDir, result)
// A file that failed to restore was not written and is already
// reported as such, so it is not verified.
failedToRestore := make(map[string]bool, len(result.FailedFiles))
for _, path := range result.FailedFiles {
failedToRestore[path] = true
}
restored := make([]*database.File, 0, len(files))
for _, file := range files {
if !failedToRestore[file.Path.String()] {
restored = append(restored, file)
}
}
err := v.verifyRestoredFiles(v.ctx, repos, restored, opts.TargetDir, result)
if err != nil {
return fmt.Errorf("verification failed: %w", err)
}
if result.FilesFailed > 0 {
if len(result.VerifyFailedFiles) > 0 {
v.UI.Errorf("Verification failed: %s files did not match expected checksums.",
v.UI.Count(result.FilesFailed))
v.UI.Count(len(result.VerifyFailedFiles)))
for _, path := range result.FailedFiles {
for _, path := range result.VerifyFailedFiles {
v.UI.Detailf("%s", v.UI.Path(path))
}
return fmt.Errorf("%d %w", result.FilesFailed, errFilesFailedVerify)
return fmt.Errorf("%d %w", len(result.VerifyFailedFiles), errFilesFailedVerify)
}
v.UI.Completef("Verified %s files (%s).",
@@ -1420,8 +1446,8 @@ func (v *Vaultik) verifyRestoredFiles(
if err != nil {
log.Error("File verification failed", "path", file.Path, "error", err)
result.FilesFailed++
result.FailedFiles = append(result.FailedFiles, file.Path.String())
result.VerifyFailedFiles = append(
result.VerifyFailedFiles, file.Path.String())
}
bytesProcessed += file.Size
@@ -1437,7 +1463,7 @@ func (v *Vaultik) verifyRestoredFiles(
log.Info("Verification complete",
"files_verified", result.FilesVerified,
"bytes_verified", ubytes(result.BytesVerified),
"files_failed", result.FilesFailed,
"files_failed", len(result.VerifyFailedFiles),
)
return nil
@@ -61,8 +61,8 @@ func craftedFile(
}
// TestRestoreShortChunkHashInNoBlob proves a file whose short chunk hash
// has no blob_chunks row fails restore planning and the chunk write with
// an error.
// has no blob_chunks row is left out of the restore plan with an error,
// and fails the chunk write with an error.
func TestRestoreShortChunkHashInNoBlob(t *testing.T) {
t.Parallel()
@@ -77,9 +77,11 @@ func TestRestoreShortChunkHashInNoBlob(t *testing.T) {
chunkToBlobMap, err := v.buildChunkToBlobMap(ctx, repos)
require.NoError(t, err)
_, err = newRestorePlan(ctx, repos, []*database.File{file},
plan, unplanned, err := newRestorePlan(ctx, repos, []*database.File{file},
chunkToBlobMap, map[string]string{})
require.ErrorIs(t, err, errPlanChunkMissing)
require.NoError(t, err)
require.ErrorIs(t, unplanned[file.ID], errPlanChunkMissing)
require.False(t, plan.hasPending())
fileChunks, err := repos.FileChunks.GetByFileID(ctx, file.ID)
require.NoError(t, err)
+17 -8
View File
@@ -41,20 +41,25 @@ type restorePlan struct {
// newRestorePlan builds the file→blob index for the given files. Files
// whose chunks reference no blobs (symlinks, directories) start in the
// ready queue immediately.
// ready queue immediately. A file with a chunk that no blob holds cannot
// be restored: it is left out of the plan and returned, by ID, with the
// reason.
func newRestorePlan(
ctx context.Context,
repos *database.Repositories,
files []*database.File,
chunkToBlobMap map[string]*database.BlobChunk,
blobIDToHash map[string]string,
) (*restorePlan, error) {
) (*restorePlan, map[types.FileID]error, error) {
p := &restorePlan{
fileBlobs: make(map[types.FileID]map[string]struct{}, len(files)),
blobFiles: make(map[string]map[types.FileID]struct{}),
ready: make([]types.FileID, 0, len(files)),
cached: make(map[string]struct{}),
}
unplanned := make(map[types.FileID]error)
nextFile:
for _, f := range files {
if f.IsSymlink() || f.Mode&uint32(os.ModeDir) != 0 {
// No chunks to fetch — restore can run immediately.
@@ -66,7 +71,7 @@ func newRestorePlan(
fileChunks, err := repos.FileChunks.GetByFileID(ctx, f.ID)
if err != nil {
return nil, fmt.Errorf("planning %s: %w", f.Path, err)
return nil, nil, fmt.Errorf("planning %s: %w", f.Path, err)
}
blobs := make(map[string]struct{})
@@ -74,14 +79,18 @@ func newRestorePlan(
for _, fc := range fileChunks {
bc, ok := chunkToBlobMap[fc.ChunkHash.String()]
if !ok {
return nil, fmt.Errorf("planning %s: %w: %s",
f.Path, errPlanChunkMissing, shortHash(fc.ChunkHash.String()))
unplanned[f.ID] = fmt.Errorf("%w: %s",
errPlanChunkMissing, shortHash(fc.ChunkHash.String()))
continue nextFile
}
hash, ok := blobIDToHash[bc.BlobID.String()]
if !ok {
return nil, fmt.Errorf("planning %s: %w: %s",
f.Path, errPlanBlobIDMissing, bc.BlobID)
unplanned[f.ID] = fmt.Errorf("%w: %s",
errPlanBlobIDMissing, bc.BlobID)
continue nextFile
}
blobs[hash] = struct{}{}
@@ -103,7 +112,7 @@ func newRestorePlan(
}
}
return p, nil
return p, unplanned, nil
}
// markBlobCached records that the named blob is now resident in the
+229 -16
View File
@@ -4,7 +4,9 @@ import (
"bytes"
"context"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"github.com/spf13/afero"
@@ -28,7 +30,8 @@ const (
)
// missingBlobBackup is a snapshot of single-chunk files spread over two
// blobs, with one of those blobs deleted from the store.
// blobs, with one of those blobs lost: deleted from the store, or missing
// from the snapshot metadata.
type missingBlobBackup struct {
fs afero.Fs
cfg *config.Config
@@ -37,20 +40,39 @@ type missingBlobBackup struct {
restoreDir string
// files holds the original content by source path.
files map[string][]byte
// lost holds the source paths whose chunk is in the deleted blob.
// lost holds the source paths whose chunk is in the lost blob.
lost map[string]bool
}
// TestRestoreSkipErrorsSkipsFilesOfMissingBlob restores with SkipErrors
// after one blob of a two-blob snapshot was deleted. Every file stored in
// that blob must be reported as failed and left absent, every other file
// must be restored intact, and Restore must still return an error.
// after one blob of a two-blob snapshot was deleted from the store.
func TestRestoreSkipErrorsSkipsFilesOfMissingBlob(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
backup := backupThenDeleteOneBlob(ctx, t)
assertRestoreSkipsLostFiles(ctx, t, backupThenDeleteOneBlob(ctx, t))
}
// TestRestoreSkipErrorsSkipsFilesWithChunkInNoBlob restores with
// SkipErrors a snapshot whose metadata lists chunks that no blob holds, as
// earlier backup defects left behind.
func TestRestoreSkipErrorsSkipsFilesWithChunkInNoBlob(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
assertRestoreSkipsLostFiles(ctx, t, backupThenDropOneBlobFromMetadata(ctx, t))
}
// assertRestoreSkipsLostFiles restores backup with SkipErrors. Every file
// stored in the lost blob must be reported as failed and left absent,
// every other file must be restored intact, and Restore must still return
// an error.
func assertRestoreSkipsLostFiles(
ctx context.Context, t *testing.T, backup *missingBlobBackup,
) {
t.Helper()
var out bytes.Buffer
@@ -72,18 +94,142 @@ func TestRestoreSkipErrorsSkipsFilesOfMissingBlob(t *testing.T) {
if backup.lost[path] {
assert.Containsf(t, out.String(), path,
"%s needs the deleted blob and must be reported", path)
"%s needs the lost blob and must be reported", path)
assert.NoFileExists(t, restored)
continue
}
got, err := afero.ReadFile(backup.fs, restored)
require.NoErrorf(t, err, "%s does not need the deleted blob", path)
require.NoErrorf(t, err, "%s does not need the lost blob", path)
assert.Equalf(t, content, got, "%s restored with wrong content", path)
}
}
// TestRestoreSkipErrorsVerifyChecksOnlyRestoredFiles restores with
// SkipErrors and Verify after one blob of a two-blob snapshot was
// deleted. Verification must check only the files that were written, so
// a file of the deleted blob is reported as failed to restore and not
// also as failing verification, and Restore must still return an error.
func TestRestoreSkipErrorsVerifyChecksOnlyRestoredFiles(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
backup := backupThenDeleteOneBlob(ctx, t)
var out bytes.Buffer
v := newReaderVaultik(ctx, backup.cfg, backup.storer, nil, backup.fs)
v.UI = ui.NewWithColor(&out, false)
err := v.Restore(&vaultik.RestoreOptions{
SnapshotID: backup.snapshotID,
TargetDir: backup.restoreDir,
Verify: true,
SkipErrors: true,
})
require.Error(t, err, "restore must fail when files were skipped")
assert.Contains(t, err.Error(),
fmt.Sprintf("%d file(s) failed to restore", len(backup.lost)))
written := len(backup.files) - len(backup.lost)
assert.Contains(t, out.String(), fmt.Sprintf("Verifying %d files", written))
assert.Contains(t, out.String(), fmt.Sprintf("Verified %d files", written))
assert.NotContains(t, out.String(), "Verification failed")
}
// TestRestoreSkipErrorsVerifyReportsFailuresApart restores with
// SkipErrors and Verify after one blob of a two-blob snapshot was
// deleted, while one file that does restore is written one byte too long.
// The restore warning must list only the files of the deleted blob, the
// verification summary only the long file, each path once, and Restore
// must return an error.
func TestRestoreSkipErrorsVerifyReportsFailuresApart(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
backup := backupThenDeleteOneBlob(ctx, t)
var long string
for path := range backup.files {
if !backup.lost[path] {
long = path
break
}
}
var out bytes.Buffer
fs := trailingByteFs{Fs: backup.fs, suffix: long}
v := newReaderVaultik(ctx, backup.cfg, backup.storer, nil, fs)
v.UI = ui.NewWithColor(&out, false)
err := v.Restore(&vaultik.RestoreOptions{
SnapshotID: backup.snapshotID,
TargetDir: backup.restoreDir,
Verify: true,
SkipErrors: true,
})
require.Error(t, err, "restore must fail when files failed")
_, afterWarning, found := strings.Cut(out.String(),
fmt.Sprintf("%d file(s) failed to restore:", len(backup.lost)))
require.Truef(t, found, "no restore warning in:\n%s", out.String())
warning, summary, found := strings.Cut(afterWarning,
"Verification failed: 1 files did not match")
require.Truef(t, found, "no verification summary in:\n%s", out.String())
for path := range backup.lost {
assert.Equalf(t, 1, strings.Count(warning, path),
"%s must be listed once as failed to restore", path)
assert.NotContainsf(t, summary, path,
"%s was not restored and must not fail verification", path)
}
assert.NotContainsf(t, warning, long,
"%s was restored and must not be listed as failed to restore", long)
assert.Equalf(t, 1, strings.Count(summary, long),
"%s must be listed once as failing verification", long)
}
// trailingByteFs appends one byte, on close, to the file it opens whose
// path ends in suffix, so that file is restored one byte too long and
// fails verification.
type trailingByteFs struct {
afero.Fs
suffix string
}
//nolint:ireturn // afero.Fs.OpenFile is defined to return the interface
func (fs trailingByteFs) OpenFile(
name string, flag int, perm os.FileMode,
) (afero.File, error) {
f, err := fs.Fs.OpenFile(name, flag, perm)
if err != nil || !strings.HasSuffix(name, fs.suffix) {
return f, err
}
return trailingByteFile{File: f}, nil
}
type trailingByteFile struct {
afero.File
}
func (f trailingByteFile) Close() error {
_, _ = f.Write([]byte{'x'})
return f.File.Close()
}
// TestRestoreMissingBlobAbortsWithoutSkipErrors checks that a deleted blob
// still ends the restore with an error when SkipErrors is not set.
func TestRestoreMissingBlobAbortsWithoutSkipErrors(t *testing.T) {
@@ -103,6 +249,30 @@ func TestRestoreMissingBlobAbortsWithoutSkipErrors(t *testing.T) {
require.ErrorIs(t, err, storage.ErrNotFound)
}
// TestRestoreChunkInNoBlobAbortsWithoutSkipErrors checks that a chunk held
// by no blob still ends the restore, before any file is written, when
// SkipErrors is not set.
func TestRestoreChunkInNoBlobAbortsWithoutSkipErrors(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
backup := backupThenDropOneBlobFromMetadata(ctx, t)
v := newReaderVaultik(ctx, backup.cfg, backup.storer, nil, backup.fs)
err := v.Restore(&vaultik.RestoreOptions{
SnapshotID: backup.snapshotID,
TargetDir: backup.restoreDir,
})
require.ErrorContains(t, err, "chunk missing from blob map")
for path := range backup.files {
assert.NoFileExists(t, filepath.Join(backup.restoreDir, path))
}
}
// backupThenDeleteOneBlob backs up missingBlobFileCount single-chunk
// files, then deletes from the store the blob holding the first of them.
func backupThenDeleteOneBlob(
@@ -110,6 +280,54 @@ func backupThenDeleteOneBlob(
) *missingBlobBackup {
t.Helper()
backup, db, lostBlob := backupSingleChunkFiles(ctx, t)
require.NoError(t, db.Close())
require.NoError(t, backup.storer.Delete(ctx, fmt.Sprintf(
"blobs/%s/%s/%s", lostBlob[:2], lostBlob[2:4], lostBlob)))
return backup
}
// backupThenDropOneBlobFromMetadata backs up missingBlobFileCount
// single-chunk files, then removes the blob holding the first of them from
// the snapshot and exports the snapshot metadata again. The exported
// metadata then lists chunks that no blob holds.
func backupThenDropOneBlobFromMetadata(
ctx context.Context, t *testing.T,
) *missingBlobBackup {
t.Helper()
backup, db, lostBlob := backupSingleChunkFiles(ctx, t)
// The uploads row refers to the blob row, so it would keep the export
// from deleting it.
for _, query := range []string{
"DELETE FROM snapshot_blobs WHERE blob_hash = ?",
"DELETE FROM uploads WHERE blob_hash = ?",
} {
_, err := db.Conn().ExecContext(ctx, query, lostBlob)
require.NoError(t, err)
}
sm := newFaultSnapshotManager(
backup.fs, backup.storer, backup.cfg, database.NewRepositories(db))
require.NoError(t,
sm.ExportSnapshotMetadata(ctx, db.Path(), backup.snapshotID))
require.NoError(t, db.Close())
return backup
}
// backupSingleChunkFiles backs up missingBlobFileCount single-chunk files.
// It returns the backup, with lost naming the files stored in the same
// blob as the first of them, the still open local index, and the hash of
// that blob.
func backupSingleChunkFiles(
ctx context.Context, t *testing.T,
) (*missingBlobBackup, *database.DB, string) {
t.Helper()
fs := afero.NewOsFs()
tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "src")
@@ -143,16 +361,11 @@ func backupThenDeleteOneBlob(
blobOfFile[path] = blobHashOfSingleChunkFile(ctx, t, repos, path)
}
require.NoError(t, db.Close())
deleted := blobOfFile[filepath.Join(dataDir, "file-00.bin")]
require.NoError(t, storer.Delete(ctx, fmt.Sprintf(
"blobs/%s/%s/%s", deleted[:2], deleted[2:4], deleted)))
lostBlob := blobOfFile[filepath.Join(dataDir, "file-00.bin")]
lost := make(map[string]bool)
for path, hash := range blobOfFile {
if hash == deleted {
if hash == lostBlob {
lost[path] = true
}
}
@@ -168,7 +381,7 @@ func backupThenDeleteOneBlob(
restoreDir: filepath.Join(tempDir, "restored"),
files: files,
lost: lost,
}
}, db, lostBlob
}
// blobHashOfSingleChunkFile returns the hash of the blob holding the one
+4 -1
View File
@@ -209,6 +209,7 @@ type snapshotStats struct {
totalFiles int
totalBytes int64
totalChunks int
newChunkSize int64
totalBlobs int
totalBytesSkipped int64
totalFilesSkipped int
@@ -341,6 +342,7 @@ func (v *Vaultik) scanAllDirectories(
stats.totalFiles += result.FilesScanned
stats.totalBytes += result.BytesScanned
stats.totalChunks += result.ChunksCreated
stats.newChunkSize += result.NewChunkSize
stats.totalBlobs += result.BlobsCreated
stats.totalFilesSkipped += result.FilesSkipped
stats.totalFilesFailed += result.FilesFailed
@@ -403,6 +405,7 @@ func (v *Vaultik) finalizeSnapshotMetadata(
BlobUncompressedSize: stats.blobUncompressedSize,
CompressionLevel: v.Config.CompressionLevel,
UploadDurationMs: stats.uploadDuration.Milliseconds(),
NewChunkSize: stats.newChunkSize,
}
err = v.SnapshotManager.UpdateSnapshotStatsExtended(v.ctx, snapshotID, extStats)
@@ -633,7 +636,7 @@ func (v *Vaultik) confirmAndExecutePurge(
v.stdoutf(" %s (%s, %s)\n",
snap.ID,
snap.Timestamp.Format("2006-01-02 15:04:05"),
formatBytes(snap.CompressedSize))
ubytes(snap.CompressedSize))
}
}
+3 -3
View File
@@ -553,8 +553,8 @@ func (v *Vaultik) printSnapshotTable(snapshots []SnapshotInfo) error {
if snap.LocallyTracked {
id = snap.ID.String()
uncompressed = formatBytes(snap.UncompressedSize)
newChunks = formatBytes(snap.NewChunkSize)
uncompressed = ubytes(snap.UncompressedSize)
newChunks = ubytes(snap.NewChunkSize)
} else {
id = formatRemoteOnlyID(snap.RemoteKey)
uncompressed = remoteOnlyCell
@@ -564,7 +564,7 @@ func (v *Vaultik) printSnapshotTable(snapshots []SnapshotInfo) error {
_, err = fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n",
id,
snap.Timestamp.Format("2006-01-02 15:04:05"),
formatBytes(snap.CompressedSize),
ubytes(snap.CompressedSize),
uncompressed,
newChunks)
if err != nil {
+45 -2
View File
@@ -212,8 +212,8 @@ const (
listConfiguredName = "home"
listLocalID = "testhost_home_2026-03-01T10:00:00Z"
listRemoteID = "otherhost_media_2026-03-02T11:22:33Z"
// fiveMegabytes formats as "5.0 MB" through formatBytes.
fiveMegabytes = 5 * 1024 * 1024
// fiveMegabytes is printed as "5.0 MB" by every command.
fiveMegabytes = 5_000_000
)
// TestListSnapshots_RemoteWithoutSecretKey is the regression guard for
@@ -306,6 +306,48 @@ func TestListSnapshots_RemoteOnlyRowRendering(t *testing.T) {
"are not in the local index")
}
// TestSnapshotListAndPurgePrintSizesLikeRemoteInfo checks that the
// `snapshot list` table and the purge listing print a byte count the way
// `remote info` does: in powers of 1000, as the MB label says.
func TestSnapshotListAndPurgePrintSizesLikeRemoteInfo(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
const wantSize = "5.0 MB" // fiveMegabytes
env := newListEnv(t)
// A remote-only snapshot: `snapshot list` and `remote info` both take
// its size from the manifest.
remoteKey := env.addRemote(t, listRemoteID,
time.Date(2026, 3, 2, 11, 22, 33, 0, time.UTC))
// A locally tracked snapshot of the same size for the purge to list.
// The purge prints the snapshot's blob size, the last argument here.
start := time.Date(2026, 3, 1, 10, 0, 0, 0, time.UTC)
env.addLocal(t, listLocalID, start)
env.addRemote(t, listLocalID, start)
require.NoError(t, env.v.Repositories.Snapshots.UpdateCounts(
context.Background(), nil, listLocalID, 0, 0, 0, 0, fiveMegabytes))
require.NoError(t, env.v.RemoteInfo(false))
assert.Contains(t, findTableRow(t, env.stdout.String(), remoteKey[:16]),
wantSize, "remote info")
env.stdout.Reset()
require.NoError(t, env.v.ListSnapshots(false))
assert.Contains(t, findTableRow(t, env.stdout.String(),
"<remote only:"+remoteKey[:12]+">"), wantSize, "snapshot list")
// Without --force and with nothing on stdin the purge lists what it
// would delete, reads no confirmation, and deletes nothing.
env.stdout.Reset()
require.NoError(t, env.v.PurgeSnapshotsWithOptions(
&vaultik.SnapshotPurgeOptions{OlderThan: "1d"}))
assert.Contains(t, findTableRow(t, env.stdout.String(), listLocalID),
wantSize, "snapshot purge")
}
// findTableRow returns the single output line containing needle.
func findTableRow(t *testing.T, out, needle string) string {
t.Helper()
@@ -445,6 +487,7 @@ type listJSONRow struct {
RemoteKey string `json:"remote_key"`
Timestamp string `json:"timestamp"`
CompressedSize int64 `json:"compressed_size"`
NewChunkSize int64 `json:"new_chunk_size"`
LocallyTracked bool `json:"locally_tracked"`
RemotePresent *bool `json:"remote_present"`
}