Compare commits

..
2 Commits
Author SHA1 Message Date
sneak 54211d84f6 Keep command output to the README's stdout and stderr rules (closes #224)
check / check (push) Successful in 12m58s
The startup banner moves from stdout to stderr, so a `completion`
script, a `config get` value and the hidden `__complete` command print
only their own output. `--quiet`, `--cron` and `--json` still suppress
it.

A failing `remote info`, `prune` or `snapshot remove` under `--json`
now reports its error on stderr. Their reporters returned early under
`--json`, so the failure reached neither stream.

`snapshot verify --quiet` writes no report. A failure is still
returned and printed on stderr, with the same exit status.

Judgement call: the banner's stream, posted on the issue for the owner.

Model: opus-5-5
2026-10-06 20:52:09 +00:00
clawbot 5d685f03ce List the files under a path by path, not by string prefix (closes #223)
check / check (push) Successful in 12m16s
FileRepository.ListByPrefix matched with SQL LIKE: a plain string
prefix that ignores ASCII case and treats _ and % as wildcards.
Restoring /home/u/doc also restored doc2, DOC and doc.txt.bak, and a
backup counted the files of a longer sibling path as deleted. It is
now ListUnderPath, which returns the file at the path and every file
whose path starts with the path plus a slash, compared exactly. A
trailing slash is ignored, so "/" still lists every file.

Three tests relied on string-prefix matching and now name a full path
or call ListAll.

ListIDsWithChunksNotInUploadedBlobs keeps its LIKE: it only adds file
IDs the scan never looks up.

Model: opus-5-5
2026-10-06 22:12:16 +02:00
13 changed files with 143 additions and 64 deletions
+11
View File
@@ -31,6 +31,17 @@ the tag exists and is exercised; what is left is merging `next` to
`snapshot verify --quiet` printed its whole report; it now prints `snapshot verify --quiet` printed its whole report; it now prints
none, and a failure still reaches stderr with the same exit status. none, and a failure still reaches stderr with the same exit status.
- 2026-10-06: Made a restore path argument select only that path and
what is beneath it
([issue #223](https://git.eeqj.de/sneak/vaultik/issues/223)). The
lookup matched with SQL `LIKE`, so `/home/u/doc` also restored
`doc2`, `DOC` and `doc.txt.bak`, and a `_` or `%` in the path acted
as a wildcard. A backup used the same lookup to load the known files
of each configured path, so files of a longer sibling path were
counted as deleted. `FileRepository.ListUnderPath`, which replaces
`ListByPrefix`, returns the file at the path and every file whose
path starts with the path plus `/`, compared case-sensitively.
- 2026-10-06: Made restore return an error instead of panicking on a - 2026-10-06: Made restore return an error instead of panicking on a
malformed snapshot database malformed snapshot database
([issue #231](https://git.eeqj.de/sneak/vaultik/issues/231)). A chunk ([issue #231](https://git.eeqj.de/sneak/vaultik/issues/231)). A chunk
+14 -40
View File
@@ -41,18 +41,9 @@ const (
someSnapshotID = "host_2026-01-01T00:00:00Z" someSnapshotID = "host_2026-01-01T00:00:00Z"
) )
// placeholderJSONDocument stands in for whatever document a --json
// command writes to stdout. `snapshot list --json` with no snapshots
// prints exactly this; the other --json commands print an object rather
// than an array, but this test is not about their shape. It is about
// what is on stdout *before* them, which is the same for all of them
// because Entry prints the banner before cobra has parsed anything and
// therefore before it can know which command is running.
const placeholderJSONDocument = "[]\n"
// jsonArgumentVectors are the argument vectors of every --json // jsonArgumentVectors are the argument vectors of every --json
// invocation the CLI accepts, with the program name stripped exactly as // invocation the CLI accepts, with the program name stripped exactly as
// Entry strips it. Each one must leave stdout untouched by the banner. // Entry strips it. Each one must suppress the banner.
// //
//nolint:gochecknoglobals // read-only test fixture shared by two tests //nolint:gochecknoglobals // read-only test fixture shared by two tests
var jsonArgumentVectors = map[string][]string{ var jsonArgumentVectors = map[string][]string{
@@ -74,39 +65,23 @@ var jsonArgumentVectors = map[string][]string{
}, },
} }
// TestJSONInvocationStdoutIsExactlyOneDocument is the CLI-layer // TestJSONInvocationSuppressesBanner checks that every --json
// regression guard for issue #106: `vaultik snapshot list --json | jq` // invocation suppresses the startup banner, as the README says --json
// must work with no other flags. // does along with --quiet and --cron. The scan is over the raw argument
// // vector, so each position and spelling of --json is listed.
// internal/vaultik's TestListSnapshots_JSONStdoutIsOnlyTheDocument func TestJSONInvocationSuppressesBanner(t *testing.T) {
// guards the same contract one layer down, but it calls the library
// function directly and so cannot see Entry, which is where the
// contamination was: the startup banner was written to stdout before
// cobra parses anything, and the suppression scan did not know about
// --json. The two banner lines and the blank line landed ahead of the
// document and `jq` refused the result.
//
// The document is a constant here because this test is about the
// argument vectors, one per --json command; the one that runs a real
// command end to end is TestEntryJSONStdoutIsExactlyOneDocument below.
func TestJSONInvocationStdoutIsExactlyOneDocument(t *testing.T) {
t.Parallel() t.Parallel()
for name, argv := range jsonArgumentVectors { for name, argv := range jsonArgumentVectors {
t.Run(name, func(t *testing.T) { t.Run(name, func(t *testing.T) {
t.Parallel() t.Parallel()
var stdout bytes.Buffer var banner bytes.Buffer
emitStartupBanner(argv, &stdout) emitStartupBanner(argv, &banner)
require.Empty(t, stdout.String(), assert.Empty(t, banner.String(),
"nothing may reach stdout ahead of a --json document") "--json suppresses the banner")
_, err := stdout.WriteString(placeholderJSONDocument)
require.NoError(t, err)
requireExactlyOneJSONDocument(t, stdout.String())
}) })
} }
} }
@@ -127,11 +102,11 @@ func TestBannerStillPrintedWithoutSuppressingFlag(t *testing.T) {
t.Run(name, func(t *testing.T) { t.Run(name, func(t *testing.T) {
t.Parallel() t.Parallel()
var stdout bytes.Buffer var banner bytes.Buffer
emitStartupBanner(argv, &stdout) emitStartupBanner(argv, &banner)
assert.Contains(t, stdout.String(), "starting up at", assert.Contains(t, banner.String(), "starting up at",
"the banner belongs on invocations that did not opt out") "the banner belongs on invocations that did not opt out")
}) })
} }
@@ -247,8 +222,7 @@ func TestEntryJSONStdoutIsExactlyOneDocument(t *testing.T) {
// captureProcessStdout redirects the process's own stdout to a pipe for // captureProcessStdout redirects the process's own stdout to a pipe for
// the duration of fn and returns what was written to it. The redirection // the duration of fn and returns what was written to it. The redirection
// has to be at the file-descriptor level rather than through an injected // has to be at the file-descriptor level rather than through an injected
// writer, because Entry and the commands it runs each reach os.Stdout // writer, because the commands Entry runs reach os.Stdout directly.
// directly.
// //
// Not parallel-safe: os.Stdout is process-global. // Not parallel-safe: os.Stdout is process-global.
func captureProcessStdout(t *testing.T, fn func()) string { func captureProcessStdout(t *testing.T, fn func()) string {
+11 -6
View File
@@ -228,19 +228,24 @@ func (r *FileRepository) DeleteByID(
return nil return nil
} }
// ListByPrefix returns all files whose path starts with prefix, ordered by // ListUnderPath returns the file at path and every file beneath it,
// path. // ordered by path. Paths are compared case-sensitively, and a trailing
func (r *FileRepository) ListByPrefix( // slash on path is ignored, so "/" lists every file.
ctx context.Context, prefix string, func (r *FileRepository) ListUnderPath(
ctx context.Context, path string,
) ([]*File, error) { ) ([]*File, error) {
path = strings.TrimRight(path, "/")
dirPrefix := path + "/"
// LIKE would ignore ASCII case and treat _ and % in path as wildcards.
query := ` query := `
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
FROM files FROM files
WHERE path LIKE ? || '%' WHERE path = ? OR substr(path, 1, length(?)) = ?
ORDER BY path ORDER BY path
` `
rows, err := r.db.conn.QueryContext(ctx, query, prefix) rows, err := r.db.conn.QueryContext(ctx, query, path, dirPrefix, dirPrefix)
if err != nil { if err != nil {
return nil, fmt.Errorf("querying files: %w", err) return nil, fmt.Errorf("querying files: %w", err)
} }
+78
View File
@@ -5,10 +5,12 @@ import (
"database/sql" "database/sql"
"errors" "errors"
"os" "os"
"slices"
"testing" "testing"
"time" "time"
"sneak.berlin/go/vaultik/internal/database" "sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/types"
) )
// errTestRollback is the sentinel returned from transaction bodies to // errTestRollback is the sentinel returned from transaction bodies to
@@ -134,6 +136,82 @@ func TestFileRepositoryListDelete(t *testing.T) {
} }
} }
func TestFileRepositoryListUnderPath(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repo := database.NewFileRepository(db)
const (
docDir = "/home/u/doc"
docFile = "/home/u/doc/a.txt"
)
// In path order, so the root case can expect all of them as listed.
paths := []string{
"/home/u/50%/x.txt",
"/home/u/50percent/y.txt",
"/home/u/DOC/c.txt",
"/home/u/a_b/x.txt",
"/home/u/axb/y.txt",
docDir,
"/home/u/doc.txt.bak",
docFile,
"/home/u/doc/sub/b.txt",
"/home/u/doc2/b.txt",
}
for _, path := range paths {
err := repo.Create(ctx, nil, &database.File{
Path: types.FilePath(path),
MTime: time.Now().Truncate(time.Second),
Mode: 0644,
})
if err != nil {
t.Fatalf("failed to create %s: %v", path, err)
}
}
docTree := []string{docDir, docFile, "/home/u/doc/sub/b.txt"}
tests := []struct {
name string
path string
want []string
}{
{"directory", docDir, docTree},
{"directory with trailing slash", docDir + "/", docTree},
{"directory differing only in case", "/home/u/DOC",
[]string{"/home/u/DOC/c.txt"}},
{"file", docFile, []string{docFile}},
{"underscore is literal", "/home/u/a_b",
[]string{"/home/u/a_b/x.txt"}},
{"percent is literal", "/home/u/50%",
[]string{"/home/u/50%/x.txt"}},
{"root", "/", paths},
}
for _, tt := range tests {
files, err := repo.ListUnderPath(ctx, tt.path)
if err != nil {
t.Fatalf("%s: failed to list files: %v", tt.name, err)
}
got := make([]string, 0, len(files))
for _, f := range files {
got = append(got, f.Path.String())
}
if !slices.Equal(got, tt.want) {
t.Errorf("%s: listing %q got %q, want %q",
tt.name, tt.path, got, tt.want)
}
}
}
func TestFileRepositorySymlink(t *testing.T) { func TestFileRepositorySymlink(t *testing.T) {
t.Parallel() t.Parallel()
@@ -824,7 +824,7 @@ func TestTransactionIsolation(t *testing.T) {
} }
// Verify the file was not created (transaction rolled back) // Verify the file was not created (transaction rolled back)
files, err := repos.Files.ListByPrefix(ctx, "/tx-test") files, err := repos.Files.ListUnderPath(ctx, "/tx-test.txt")
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -916,7 +916,7 @@ func TestConcurrentOrphanedCleanup(t *testing.T) {
} }
// Verify correct files were deleted // Verify correct files were deleted
files, err := repos.Files.ListByPrefix(ctx, "/concurrent-") files, err := repos.Files.ListAll(ctx)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
+1 -1
View File
@@ -147,7 +147,7 @@ func TestOrphanedFileCleanupDebug(t *testing.T) {
t.Logf("Files count after cleanup: %d", count) t.Logf("Files count after cleanup: %d", count)
// List remaining files // List remaining files
files, err := repos.Files.ListByPrefix(ctx, "/") files, err := repos.Files.ListUnderPath(ctx, "/")
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -442,12 +442,12 @@ func TestLargeDatasets(t *testing.T) {
createLargeDatasetFiles(t, repos, snapshot.ID.String(), fileCount) createLargeDatasetFiles(t, repos, snapshot.ID.String(), fileCount)
}) })
// Test ListByPrefix performance // Test ListUnderPath performance
//nolint:paralleltest // phases share one database and are order-dependent //nolint:paralleltest // phases share one database and are order-dependent
t.Run("list by prefix performance", func(t *testing.T) { t.Run("list under path performance", func(t *testing.T) {
start := time.Now() start := time.Now()
files, err := repos.Files.ListByPrefix(ctx, "/large/") files, err := repos.Files.ListUnderPath(ctx, "/large/")
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -472,7 +472,7 @@ func TestLargeDatasets(t *testing.T) {
t.Logf("Cleaned up orphaned files in %v", time.Since(start)) t.Logf("Cleaned up orphaned files in %v", time.Since(start))
// Verify correct number remain // Verify correct number remain
files, err := repos.Files.ListByPrefix(ctx, "/large/") files, err := repos.Files.ListUnderPath(ctx, "/large/")
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
+1 -1
View File
@@ -71,7 +71,7 @@ func verifyBackupFiles(
) { ) {
t.Helper() t.Helper()
files, err := repos.Files.ListByPrefix(ctx, "") files, err := repos.Files.ListAll(ctx)
if err != nil { if err != nil {
t.Fatalf("Failed to list files: %v", err) t.Fatalf("Failed to list files: %v", err)
} }
+6 -4
View File
@@ -456,14 +456,16 @@ func (s *Scanner) finalizeScanResult(ctx context.Context, result *ScanResult) {
result.EndTime = time.Now().UTC() result.EndTime = time.Now().UTC()
} }
// loadKnownFiles loads all known files from the database into a map for fast lookup // loadKnownFiles loads the known files at and beneath path from the
// This avoids per-file database queries during the scan phase // database into a map for fast lookup. Every loaded file the scan does
// not find is counted as deleted. This avoids per-file database queries
// during the scan phase.
func (s *Scanner) loadKnownFiles( func (s *Scanner) loadKnownFiles(
ctx context.Context, path string, ctx context.Context, path string,
) (map[string]*database.File, error) { ) (map[string]*database.File, error) {
files, err := s.repos.Files.ListByPrefix(ctx, path) files, err := s.repos.Files.ListUnderPath(ctx, path)
if err != nil { if err != nil {
return nil, fmt.Errorf("listing files by prefix: %w", err) return nil, fmt.Errorf("listing files under %s: %w", path, err)
} }
result := make(map[string]*database.File, len(files)) result := make(map[string]*database.File, len(files))
+1 -1
View File
@@ -71,7 +71,7 @@ func verifySimpleScanDatabase(
t.Helper() t.Helper()
// Verify files in database - includes regular files and directories // Verify files in database - includes regular files and directories
files, err := repos.Files.ListByPrefix(ctx, "/source") files, err := repos.Files.ListUnderPath(ctx, "/source")
if err != nil { if err != nil {
t.Fatalf("failed to list files: %v", err) t.Fatalf("failed to list files: %v", err)
} }
+1 -1
View File
@@ -249,7 +249,7 @@ func verifyEndToEndBackupState(
assert.Positive(t, blobUploads, "Should upload at least one blob") assert.Positive(t, blobUploads, "Should upload at least one blob")
// Verify files in database // Verify files in database
files, err := repos.Files.ListByPrefix(ctx, "/home/user") files, err := repos.Files.ListUnderPath(ctx, "/home/user")
require.NoError(t, err) require.NoError(t, err)
// Count only regular files (not directories) // Count only regular files (not directories)
regularFiles := 0 regularFiles := 0
+2 -3
View File
@@ -830,10 +830,9 @@ func (v *Vaultik) getFilesToRestore(
// Normalize the filter path // Normalize the filter path
filter = filepath.Clean(filter) filter = filepath.Clean(filter)
// Get files with this prefix files, err := repos.Files.ListUnderPath(ctx, filter)
files, err := repos.Files.ListByPrefix(ctx, filter)
if err != nil { if err != nil {
return nil, fmt.Errorf("listing files with prefix %s: %w", filter, err) return nil, fmt.Errorf("listing files under %s: %w", filter, err)
} }
for _, file := range files { for _, file := range files {
+11 -1
View File
@@ -78,10 +78,20 @@ func TestVerify_QuietSuppressesReport(t *testing.T) {
require.Equal(t, "ok", decodeVerifyResult(t, stdout.Bytes()).Status, require.Equal(t, "ok", decodeVerifyResult(t, stdout.Bytes()).Status,
"the --json document must still emit under --quiet") "the --json document must still emit under --quiet")
// A snapshot without its encrypted database fails shallow verify. // A snapshot without its encrypted database fails shallow verify. A
// failed report also lists each missing blob and each blob of the
// wrong size, so remove one blob and grow another.
require.NoError(t, os.Remove(filepath.Join(storeDir, "metadata", require.NoError(t, os.Remove(filepath.Join(storeDir, "metadata",
snapshot.RemoteSnapshotKey(snapshotID), "db.zst.age"))) snapshot.RemoteSnapshotKey(snapshotID), "db.zst.age")))
blobFiles, err := filepath.Glob(
filepath.Join(storeDir, "blobs", "*", "*", "*"))
require.NoError(t, err)
require.GreaterOrEqual(t, len(blobFiles), 2,
"the snapshot must span two blobs, one to remove and one to grow")
require.NoError(t, os.Remove(blobFiles[0]))
growOneBlob(t, fs, filepath.Join(storeDir, "blobs"))
stdout.Reset() stdout.Reset()
require.Error(t, newQuietVerifier().VerifySnapshotWithOptions( require.Error(t, newQuietVerifier().VerifySnapshotWithOptions(