diff --git a/README.md b/README.md index 06a531b..9879211 100644 --- a/README.md +++ b/README.md @@ -2,9 +2,9 @@ `vaultik` is an incremental backup tool written in Go. It encrypts data using an `age` public key and uploads each encrypted blob directly to a remote -S3-compatible object store. It requires no private keys, secrets, or credentials -(other than those required to PUT to encrypted object storage, such as S3 API -keys) stored on the backed-up system. +S3-compatible object store. The backed-up system never needs the private key. A +backup needs only the public key and the credentials to write to the object +store, such as S3 API keys. ## quickstart @@ -171,6 +171,7 @@ vaultik [--config ] config get vaultik [--config ] config set vaultik [--config ] snapshot create [snapshot-names...] [--cron] [--prune] [--keep-newer-than ] vaultik [--config ] snapshot list [--json] # alias: ls +vaultik [--config ] snapshot files [path] [--json] vaultik [--config ] snapshot verify [--deep] [--json] vaultik [--config ] snapshot purge [--keep-latest | --older-than ] [--snapshot ...] [--force] vaultik [--config ] snapshot remove [--dry-run] [--force] [--local-only] [--json] # alias: rm @@ -223,12 +224,24 @@ the backup can delete it; the new snapshot then cannot be restored. `remote nuke` deletes everything regardless. To not depend on the grace period, give each host or user its own prefix (or directory) on the destination. -Read-only commands — `info`, `snapshot list`, `snapshot verify`, and -`remote info` — do not take the lock and are never blocked, so they run even -while a backup is in progress. `snapshot restore` does not take the lock either: -it writes only to the target directory you name, not the local index or the -destination store. `config`, `database delete`, `completion`, and `version` do -not take the lock. +These read-only commands do not take the lock, so they are never blocked, even +while a backup is in progress: + +- `info` +- `snapshot list` +- `snapshot files` +- `snapshot verify` +- `remote info` + +`snapshot restore` does not take the lock either: it writes only to the target +directory you name, not the local index or the destination store. + +These commands do not take the lock either: + +- `config` +- `database delete` +- `completion` +- `version` ### stdout and stderr @@ -261,9 +274,9 @@ work as written. ### environment variables - `VAULTIK_AGE_SECRET_KEY`: Age private key for decryption (required for - `snapshot restore` and `snapshot verify --deep`). May hold the whole - `age-keygen` file — comments and every identity in it are accepted. Set it - from the file, e.g. + `snapshot restore`, `snapshot files` and `snapshot verify --deep`). May hold + the whole `age-keygen` file — comments and every identity in it are accepted. + Set it from the file, e.g. `export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"`, so the key is not typed into your shell history. - `VAULTIK_CONFIG`: Path to config file (overridden by `--config`) @@ -408,6 +421,26 @@ point behind. truncated listing all go to stderr through the logger, so stdout stays a single parseable document. +**`snapshot files`**: List the files in a snapshot without restoring it. Each +row of the table shows mode, size in bytes, modification time and path, sorted +by path. + +- Requires `VAULTIK_AGE_SECRET_KEY`: the file list is only in the snapshot's + encrypted database, which the command downloads and decrypts. +- Accepts the same identifiers as `snapshot restore`. +- An optional path limits the list to that path and what is under it. A path + that matches nothing in the snapshot is an error. +- `--json`: Output an array of objects with `path`, `size`, `mode` and `mtime`. +- In the table, a path that holds a newline or another non-printable character + is shown in double quotes, with that character escaped, as in `"a\nb"`. +- The mode is in Go's file-mode notation, in the table and in `--json`. Letters + before the nine permission characters show the file type and any special bit, + as in `dtrwxrwxrwx` for a sticky directory. These letters differ from `ls -l`: + - `L`: a symbolic link, which `ls -l` shows as `l`. + - `u`: setuid, which `ls -l` shows as `s` in place of the owner's `x`. + - `g`: setgid, which `ls -l` shows as `s` in place of the group's `x`. + - `t`: sticky, which `ls -l` shows as `t` in place of the others' `x`. + **`snapshot verify`**: Verify snapshot integrity. - Default (shallow): checks that every blob the manifest lists is present in @@ -722,7 +755,7 @@ annotated example also lives in | Field | Default | Description | | -------------------- | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `age_recipients` | (required by `snapshot create`) | Age public keys for encryption. Other commands run without one, so a machine that only restores can leave it empty | -| `age_secret_key` | (unset) | Age private key for decryption (`snapshot restore`, `snapshot verify --deep`). Setting it in the config file places the private key on the backed-up host, defeating the public-key-only design (see "why" above). Prefer the `VAULTIK_AGE_SECRET_KEY` environment variable, supplied only on the machine you restore from. | +| `age_secret_key` | (unset) | Age private key for decryption (`snapshot restore`, `snapshot files`, `snapshot verify --deep`). Setting it in the config file places the private key on the backed-up host, defeating the public-key-only design (see "why" above). Prefer the `VAULTIK_AGE_SECRET_KEY` environment variable, supplied only on the machine you restore from. | | `snapshots` | (required) | Named snapshot definitions with paths and excludes | | `storage_url` | | Storage backend URL (`s3://`, `file://`, `rclone://`) | | `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) | @@ -839,15 +872,27 @@ environment variable is unset (https://no-color.org/). `database delete`. - **The data a command exists to produce** is written plain, with no marker and no color, because a marker would corrupt a table or a parsed document. This - covers the `version`, `info`, `remote info` and `snapshot verify` reports, the - `snapshot list` table, `config get` values, and every `--json` document. - `--quiet` silences the human reports and tables (`version`, `info`, - `remote info`, `snapshot verify`, `snapshot list`) but never the - machine-consumed `config get` value or the `--json` documents, which a script - depends on. The confirmation prompts of `database delete`, `prune`, - `snapshot purge` and `snapshot remove`, each with what it asks to delete, are - also written this way and always shown: they are an interactive exchange the - operator must see. + covers the report or table of each of these commands: + - `version` + - `info` + - `remote info` + - `snapshot verify` + - `snapshot list` + - `snapshot files` + + `config get` values and every `--json` document are written plain too. + `--quiet` silences the reports and tables above, but never a `config get` + value or a `--json` document, which a script depends on. + + These commands write their confirmation prompt plain too, with what it asks + to delete: + - `database delete` + - `prune` + - `snapshot purge` + - `snapshot remove` + + `--quiet` never silences these prompts: each is an interactive exchange the + operator must see. `internal/ui` writes to stdout; it is the output the user asked for. The exceptions are the startup banner and the error a failed command ends with, diff --git a/docs/REPOSTRUCTURE.md b/docs/REPOSTRUCTURE.md index 81f34c9..856d63f 100644 --- a/docs/REPOSTRUCTURE.md +++ b/docs/REPOSTRUCTURE.md @@ -265,15 +265,23 @@ so an operator can weigh them rather than discover them. blob holds hundreds of chunks, so individual chunk lengths are not visible in the blob's size; lowering the limit toward the chunk size begins to expose them. -4. **Decrypted data on local disk.** Several commands stage plaintext under - `$TMPDIR`: `snapshot restore` writes decrypted blobs under - `vaultik-blobcache-*/` (no size cap) and the decrypted metadata database at - `vaultik-restore-*/snapshot.db`; `verify --deep` writes that database at - `vaultik-verify-*/snapshot.db`; `snapshot create` keeps a plaintext copy of - the index at `vaultik-snapshot-*/snapshot.db`. These files are created `0600` - and removed on success, but a `kill -9` or a power loss leaves them behind — - delete any leftover `vaultik-*` directory under `$TMPDIR` by hand. `$TMPDIR` - should be trusted to the same degree as the restore target. +4. **Decrypted data on local disk.** These commands stage plaintext under + `$TMPDIR`: + - `snapshot restore`: decrypted blobs under `vaultik-blobcache-*/`, with no + size cap, and the decrypted metadata database at + `vaultik-restore-*/snapshot.db`. + - `snapshot files`: the decrypted metadata database at + `vaultik-restore-*/snapshot.db`. + - `verify --deep`: the decrypted metadata database at + `vaultik-verify-*/snapshot.db`. + - `snapshot create`: a plaintext copy of the index at + `vaultik-snapshot-*/snapshot.db`. + + These files are created `0600` and removed on success, but a `kill -9` or a + power loss leaves them behind — delete any leftover `vaultik-*` directory + under `$TMPDIR` by hand. `$TMPDIR` should be trusted to the same degree as + the restore target. + 5. **Store permissions differ per command.** The backed-up host needs only PUT to run `snapshot create`: it writes blobs and metadata and neither reads nor deletes them. Other commands need more — `snapshot verify`, diff --git a/internal/cli/app.go b/internal/cli/app.go index 35f8e2d..e264bc8 100644 --- a/internal/cli/app.go +++ b/internal/cli/app.go @@ -38,10 +38,11 @@ const ( // prune, remote nuke) write the local index or the remote store. They // hold the PID lock so that at most one runs at a time. mutating lockMode = iota - // readOnly commands (info, snapshot list, snapshot verify, remote info, - // snapshot restore) do not write the local index or the remote store, - // so they run without the lock and are never blocked by a running - // mutator. restore writes only to the target directory it is given. + // readOnly commands (info, snapshot list, snapshot files, snapshot + // verify, remote info, snapshot restore) do not write the local index + // or the remote store, so they run without the lock and are never + // blocked by a running mutator. restore writes only to the target + // directory it is given. readOnly ) @@ -310,7 +311,7 @@ func RunOperation( } // runVaultikApp runs the standard single-operation command lifecycle -// shared by the snapshot list/purge/remove and remote nuke subcommands: +// shared by the snapshot list/files/purge/remove and remote nuke subcommands: // resolve the config, then run op against the Vaultik instance through // RunOperation, reporting a failure prefixed with failMsg on stderr. mode // says whether the command takes the PID lock. jsonOutput marks a command diff --git a/internal/cli/entry_snapshot_files_test.go b/internal/cli/entry_snapshot_files_test.go new file mode 100644 index 0000000..c534f3d --- /dev/null +++ b/internal/cli/entry_snapshot_files_test.go @@ -0,0 +1,47 @@ +package cli //nolint:testpackage // shares runEntry and the backup config helpers + +import ( + "encoding/json" + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "sneak.berlin/go/vaultik/internal/vaultik" +) + +// cmdFiles is the word of `snapshot files`, used to build argument +// vectors below. +const cmdFiles = "files" + +// TestEntrySnapshotFilesListsOnlyThePath backs up two files and runs +// `snapshot files` with the snapshot ID and the path of one of them, +// which must be the only file listed. +// +//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals +func TestEntrySnapshotFilesListsOnlyThePath(t *testing.T) { + configPath, _ := writeBackupConfigWithPrivateKey(t) + + // writeBackupConfig puts file.txt in the source directory beside the + // config file. + sourceDir := filepath.Join(filepath.Dir(configPath), "source") + listed := filepath.Join(sourceDir, "file.txt") + + require.NoError(t, os.WriteFile( + filepath.Join(sourceDir, "other.txt"), []byte("other"), 0o600)) + + code, _, stderr := runEntry(t, + flagConfig, configPath, cmdSnapshot, cmdCreate) + require.Equal(t, 0, code, stderr) + + code, stdout, stderr := runEntry(t, flagConfig, configPath, cmdSnapshot, + cmdFiles, onlySnapshotID(t, configPath), listed, flagJSON) + require.Equal(t, 0, code, stderr) + + var entries []vaultik.ListedFile + + require.NoError(t, json.Unmarshal([]byte(stdout), &entries)) + require.Len(t, entries, 1, "only the path given may be listed") + assert.Equal(t, listed, entries[0].Path) +} diff --git a/internal/cli/entry_test.go b/internal/cli/entry_test.go index 60877ed..a2f103d 100644 --- a/internal/cli/entry_test.go +++ b/internal/cli/entry_test.go @@ -48,7 +48,7 @@ func TestCLIEntry(t *testing.T) { } else { // Check snapshot subcommands expectedSubCommands := []string{ - "create", "list", "purge", "verify", "remove", "restore", + "create", "list", "files", "purge", "verify", "remove", "restore", } for _, expected := range expectedSubCommands { found := false diff --git a/internal/cli/snapshot.go b/internal/cli/snapshot.go index 27056fc..5fb5278 100644 --- a/internal/cli/snapshot.go +++ b/internal/cli/snapshot.go @@ -9,6 +9,10 @@ import ( "sneak.berlin/go/vaultik/internal/vaultik" ) +// filesMaxArgs is the maximum positional argument count of +// `snapshot files [path]`. +const filesMaxArgs = 2 + var ( errSnapshotIDRequired = errors.New("snapshot ID required") errWrongArgCount = errors.New("wrong argument count") @@ -52,6 +56,7 @@ func NewSnapshotCommand() *cobra.Command { // Add subcommands cmd.AddCommand(newSnapshotCreateCommand()) cmd.AddCommand(newSnapshotListCommand()) + cmd.AddCommand(newSnapshotFilesCommand()) cmd.AddCommand(newSnapshotPurgeCommand()) cmd.AddCommand(newSnapshotVerifyCommand()) cmd.AddCommand(newSnapshotRemoveCommand()) @@ -172,6 +177,44 @@ func newSnapshotListCommand() *cobra.Command { return cmd } +// newSnapshotFilesCommand creates the 'snapshot files' subcommand +func newSnapshotFilesCommand() *cobra.Command { + var jsonOutput bool + + cmd := &cobra.Command{ + Use: "files [path]", + Short: "List the files in a snapshot", + Long: `Lists the path, size, mode and modification time of every file in a +snapshot, without restoring it. With a path, lists only that path and +what is under it; a path that matches nothing is an error. + +The snapshot may be named by its ID or, on a host with no local index, +by the remote key that 'snapshot list' prints for a remote-only snapshot +(an unambiguous leading part is enough). + +The file list is only in the snapshot's encrypted database, so this +downloads and decrypts it, and requires the age private key in the +VAULTIK_AGE_SECRET_KEY environment variable, as 'snapshot restore' does.`, + Args: cobra.RangeArgs(1, filesMaxArgs), + RunE: func(cmd *cobra.Command, args []string) error { + path := "" + if len(args) == filesMaxArgs { + path = args[1] + } + + return runVaultikApp(cmd, readOnly, jsonOutput, + "Failed to list snapshot files", + func(v *vaultik.Vaultik) error { + return v.ListSnapshotFiles(args[0], path, jsonOutput) + }) + }, + } + + cmd.Flags().BoolVar(&jsonOutput, "json", false, "Output in JSON format") + + return cmd +} + // newSnapshotPurgeCommand creates the 'snapshot purge' subcommand func newSnapshotPurgeCommand() *cobra.Command { opts := &vaultik.SnapshotPurgeOptions{} diff --git a/internal/database/database.go b/internal/database/database.go index 5e680c5..8f0ee05 100644 --- a/internal/database/database.go +++ b/internal/database/database.go @@ -219,7 +219,8 @@ func retryOpen(ctx context.Context, path string) (*DB, error) { // errUntrustedSnapshotSchema is returned when a downloaded snapshot // database carries schema objects the real schema never defines, or is -// missing a table the restore and deep-verify queries read. +// missing a table the restore, deep-verify and `snapshot files` queries +// read. var errUntrustedSnapshotSchema = errors.New( "downloaded snapshot database has an untrusted schema") @@ -241,12 +242,12 @@ func snapshotReadOnlyDSN(path string) string { // OpenReadOnly opens an already-materialized SQLite file for read-only // querying of a snapshot database downloaded from the store, used by -// restore and deep verify. Unlike New it never applies schema migrations -// and never writes: the connection is opened read-only with query_only -// and trusted_schema=OFF. It refuses any file whose schema carries a -// trigger, view or virtual table, or lacks an expected table, so a forged -// file cannot redefine what the restore queries return. The caller owns -// the file and must remove it. +// restore, deep verify and `snapshot files`. Unlike New it never applies +// schema migrations and never writes: the connection is opened read-only +// with query_only and trusted_schema=OFF. It refuses any file whose schema +// carries a trigger, view or virtual table, or lacks an expected table, so +// a forged file cannot redefine what the restore queries return. The +// caller owns the file and must remove it. func OpenReadOnly(ctx context.Context, path string) (*DB, error) { conn, err := sql.Open("sqlite", snapshotReadOnlyDSN(path)) if err != nil { @@ -276,9 +277,9 @@ func OpenReadOnly(ctx context.Context, path string) (*DB, error) { // the plain table set the real schema defines. Any trigger, view or // virtual table, or a missing expected table, fails the open. func verifySnapshotSchema(ctx context.Context, conn *sql.DB) error { - // expectedSnapshotTables are the tables the restore and deep-verify - // queries read. A downloaded database missing any of them is not a - // genuine snapshot database and is refused. + // expectedSnapshotTables are the tables the restore, deep-verify and + // `snapshot files` queries read. A downloaded database missing any of + // them is not a genuine snapshot database and is refused. expectedSnapshotTables := []string{ "blob_chunks", "blobs", diff --git a/internal/database/snapshots.go b/internal/database/snapshots.go index 7ada98c..61b97a0 100644 --- a/internal/database/snapshots.go +++ b/internal/database/snapshots.go @@ -238,10 +238,11 @@ func (r *SnapshotRepository) GetByID( // GetOnlySnapshot returns the sole snapshot in an exported per-snapshot // metadata database. The backup path writes each snapshot's database with -// exactly one snapshot row (see cleanSnapshotDB), so restore and deep -// verify expect exactly one. Zero rows return ErrNoSnapshotInDatabase and -// more than one returns ErrMultipleSnapshotsInDatabase; callers treat -// either as a failed identity check on the downloaded database. +// exactly one snapshot row (see cleanSnapshotDB), so restore, deep verify +// and `snapshot files` expect exactly one. Zero rows return +// ErrNoSnapshotInDatabase and more than one returns +// ErrMultipleSnapshotsInDatabase; callers treat either as a failed +// identity check on the downloaded database. func (r *SnapshotRepository) GetOnlySnapshot(ctx context.Context) (*Snapshot, error) { query := ` SELECT id, hostname, vaultik_version, vaultik_git_revision, diff --git a/internal/ui/ui.go b/internal/ui/ui.go index 7a08d53..0584777 100644 --- a/internal/ui/ui.go +++ b/internal/ui/ui.go @@ -17,6 +17,9 @@ // Count, Percent) return ANSI-colored strings the caller composes into // the message body. When color is disabled (non-TTY output or NO_COLOR // set) all formatters return plain text. +// +// QuoteUnprintable is the same escaping without colour, for plain tables +// that print paths from the snapshot database. package ui import ( @@ -227,17 +230,17 @@ func (w *Writer) Hex(s string) string { short = s[:hexAbbrevLen] + "..." } - return w.paint(ansiCyan, sanitize(short)) + return w.paint(ansiCyan, QuoteUnprintable(short)) } // Snapshot colorizes a snapshot ID (full, no abbreviation). func (w *Writer) Snapshot(id string) string { - return w.paint(ansiCyan+ansiBold, sanitize(id)) + return w.paint(ansiCyan+ansiBold, QuoteUnprintable(id)) } // Path colorizes a filesystem path. func (w *Writer) Path(p string) string { - return w.paint(ansiBlue, sanitize(p)) + return w.paint(ansiBlue, QuoteUnprintable(p)) } // Size colorizes a byte count using humanize.Bytes. @@ -312,14 +315,15 @@ func (w *Writer) paint(color, s string) string { return color + s + ansiReset } -// sanitize returns s unchanged when every rune in it is printable, and a -// double-quoted, backslash-escaped form (\n, \x1b, …) otherwise. The +// QuoteUnprintable returns s unchanged when every rune in it is printable, +// and a double-quoted, backslash-escaped form (\n, \x1b, …) otherwise. The // string value formatters escape their argument through this before // painting: identifiers, paths and symlink targets they render come from // the snapshot database, which is not trusted, and escaping must happen // before colour is applied — the painted result already contains the // escape codes the raw text would otherwise be indistinguishable from. -func sanitize(s string) string { +// Plain tables call it directly. +func QuoteUnprintable(s string) string { for _, r := range s { if !unicode.IsPrint(r) { return strconv.Quote(s) diff --git a/internal/vaultik/blob_fetch.go b/internal/vaultik/blob_fetch.go index 4a2ccc1..8e7ffdb 100644 --- a/internal/vaultik/blob_fetch.go +++ b/internal/vaultik/blob_fetch.go @@ -160,7 +160,8 @@ func (v *Vaultik) FetchBlob( // Not every backend's reader stops on cancel by itself (the file // backend's does not), and a read stalled on a slow or hung source would // otherwise outlast the shutdown timeout, so the process would exit -// before restore or deep verify removed its decrypted temporary files. +// before restore, deep verify or `snapshot files` removed its decrypted +// temporary files. type closeOnCancelReader struct { io.ReadCloser diff --git a/internal/vaultik/restore.go b/internal/vaultik/restore.go index 5f41d96..8855aeb 100644 --- a/internal/vaultik/restore.go +++ b/internal/vaultik/restore.go @@ -31,7 +31,7 @@ var ( errPathsNotInSnapshot = errors.New("requested path(s) not in the snapshot") errFilesFailedVerify = errors.New("files failed verification") errDecryptionKeyRequired = errors.New( - "decryption key required for restore\n\n" + + "decryption key required\n\n" + "Set the VAULTIK_AGE_SECRET_KEY environment variable to your " + "age private key file:\n" + " export VAULTIK_AGE_SECRET_KEY=\"$(cat vaultik_backup_private_key.txt)\"") @@ -153,18 +153,7 @@ func (v *Vaultik) Restore(opts *RestoreOptions) error { return fmt.Errorf("downloading snapshot database: %w", err) } - defer func() { - err := tempDB.Close() - if err != nil { - log.Debug("Failed to close temp database", "error", err) - } - // Remove the whole private directory, so the decrypted database - // and any SQLite side files it produced are gone on every path. - err = v.Fs.RemoveAll(tempDir) - if err != nil { - log.Debug("Failed to remove temp database directory", "error", err) - } - }() + defer v.closeSnapshotDB(tempDB, tempDir) repos := database.NewRepositories(tempDB) @@ -325,9 +314,10 @@ func (v *Vaultik) reportUnmatchedPaths(unmatched []string) error { // identity it contains. The value may be a single key line or a whole // age-keygen file with several identities; all of them are returned so // blobgen (via age.Decrypt) can read a blob encrypted to any of their -// recipients. This is the first step of both restore and deep verify, so -// a missing or unparseable key fails before anything is downloaded. The -// error names the configuration source but never the key value. +// recipients. This is the first step of restore, deep verify and +// `snapshot files`, so a missing or unparseable key fails before anything +// is downloaded. The error names the configuration source but never the +// key value. func (v *Vaultik) restoreIdentities() ([]age.Identity, error) { if v.Config.AgeSecretKey == "" { return nil, errDecryptionKeyRequired @@ -880,18 +870,34 @@ func (v *Vaultik) downloadSnapshotDB( return db, tempDir, nil } +// closeSnapshotDB closes a database returned by downloadSnapshotDB and +// removes its private directory, so the decrypted database and any SQLite +// side files it produced are gone. +func (v *Vaultik) closeSnapshotDB(db *database.DB, tempDir string) { + err := db.Close() + if err != nil { + log.Debug("Failed to close temp database", "error", err) + } + + err = v.Fs.RemoveAll(tempDir) + if err != nil { + log.Debug("Failed to remove temp database directory", "error", err) + } +} + // verifySnapshotDBIdentity confirms the decrypted metadata database really // is the snapshot named by remoteKey. age decryption proves the database // is readable, not that the object served at // metadata//db.zst.age is the snapshot that was requested: an // attacker who swaps in another valid db.zst.age (which needs no key -// material) would otherwise redirect restore and deep verify to a -// different snapshot's contents. The exported per-snapshot database holds -// exactly one snapshot row, and a snapshot's remote key is derived from -// that row's ID, so the database is the requested one exactly when its -// sole snapshot hashes back to remoteKey. Comparing the requested -// identifier directly would not do: it may be a remote-key prefix a -// recovery host uses in place of a human snapshot ID it cannot know. +// material) would otherwise redirect restore, deep verify and +// `snapshot files` to a different snapshot's contents. The exported +// per-snapshot database holds exactly one snapshot row, and a snapshot's +// remote key is derived from that row's ID, so the database is the +// requested one exactly when its sole snapshot hashes back to remoteKey. +// Comparing the requested identifier directly would not do: it may be a +// remote-key prefix a recovery host uses in place of a human snapshot ID +// it cannot know. func (v *Vaultik) verifySnapshotDBIdentity( db *database.DB, requested, remoteKey string, ) error { diff --git a/internal/vaultik/snapshot_files.go b/internal/vaultik/snapshot_files.go new file mode 100644 index 0000000..ec4dc46 --- /dev/null +++ b/internal/vaultik/snapshot_files.go @@ -0,0 +1,115 @@ +package vaultik + +import ( + "encoding/json" + "errors" + "fmt" + "os" + "path/filepath" + "text/tabwriter" + "time" + + "sneak.berlin/go/vaultik/internal/database" + "sneak.berlin/go/vaultik/internal/ui" +) + +// ListedFile is one entry printed by `snapshot files`. Mode is the Go +// file mode string, which shows the entry type and the permission bits +// (for example -rw-r--r-- or drwxr-xr-x). +type ListedFile struct { + Path string `json:"path"` + Size int64 `json:"size"` + Mode string `json:"mode"` + MTime time.Time `json:"mtime"` +} + +// ListSnapshotFiles prints the path, size, mode and modification time of +// every file the snapshot holds at or under path, or of every file when +// path is empty. The file list is only in the snapshot's encrypted +// database, so this downloads and decrypts it as restore does, and needs +// the private key. +func (v *Vaultik) ListSnapshotFiles(snapshotID, path string, jsonOutput bool) error { + identities, err := v.restoreIdentities() + if err != nil { + return err + } + + remoteKey, err := v.resolveSnapshotRemoteKey(snapshotID) + if err != nil { + return err + } + + db, tempDir, err := v.downloadSnapshotDB(snapshotID, remoteKey, identities) + if errors.Is(err, errWrongPrivateKey) { + return err + } + + if err != nil { + return fmt.Errorf("downloading snapshot database: %w", err) + } + + defer v.closeSnapshotDB(db, tempDir) + + repos := database.NewRepositories(db) + + var files []*database.File + + if path == "" { + files, err = repos.Files.ListAll(v.ctx) + } else { + files, err = repos.Files.ListUnderPath(v.ctx, filepath.Clean(path)) + } + + if err != nil { + return fmt.Errorf("listing files: %w", err) + } + + if path != "" && len(files) == 0 { + return fmt.Errorf("%w: %s", errPathsNotInSnapshot, path) + } + + entries := make([]ListedFile, 0, len(files)) + + for _, file := range files { + entries = append(entries, ListedFile{ + Path: file.Path.String(), + Size: file.Size, + Mode: os.FileMode(file.Mode).String(), + MTime: file.MTime, + }) + } + + if jsonOutput { + encoder := json.NewEncoder(v.Stdout) + encoder.SetIndent("", " ") + + return encoder.Encode(entries) + } + + if v.UI.Quiet() { + return nil + } + + return v.printListedFileTable(entries) +} + +// printListedFileTable writes the entries as a plain table to stdout. +func (v *Vaultik) printListedFileTable(entries []ListedFile) error { + w := tabwriter.NewWriter(v.Stdout, 0, 0, tabPadding, ' ', 0) + + _, err := fmt.Fprintln(w, "MODE\tSIZE\tMODIFIED\tPATH") + if err != nil { + return err + } + + for _, entry := range entries { + _, err = fmt.Fprintf(w, "%s\t%d\t%s\t%s\n", + entry.Mode, entry.Size, + entry.MTime.Format(time.RFC3339), ui.QuoteUnprintable(entry.Path)) + if err != nil { + return err + } + } + + return w.Flush() +} diff --git a/internal/vaultik/snapshot_files_test.go b/internal/vaultik/snapshot_files_test.go new file mode 100644 index 0000000..e23eafb --- /dev/null +++ b/internal/vaultik/snapshot_files_test.go @@ -0,0 +1,197 @@ +package vaultik_test + +import ( + "bytes" + "context" + "encoding/json" + "path/filepath" + "strconv" + "strings" + "testing" + + "github.com/spf13/afero" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "sneak.berlin/go/vaultik/internal/database" + "sneak.berlin/go/vaultik/internal/log" + "sneak.berlin/go/vaultik/internal/storage" + "sneak.berlin/go/vaultik/internal/vaultik" +) + +// These tests cover https://git.eeqj.de/sneak/vaultik/issues/435: +// `snapshot files` lists what a snapshot holds without restoring it. + +// unprintableFileName is a file name holding a newline and an escape +// sequence, which the table must not print raw. +const unprintableFileName = "odd\nname\x1b[31m" + +// backUpForFileListing backs up the fault source tree, plus a file named +// unprintableFileName at its top, and returns a reader whose stdout is +// out, the snapshot ID, and the source directory. +func backUpForFileListing( + t *testing.T, out *bytes.Buffer, +) (*vaultik.Vaultik, string, string) { + t.Helper() + + fs := afero.NewOsFs() + tempDir := t.TempDir() + dataDir := filepath.Join(tempDir, "src") + dbPath := filepath.Join(tempDir, "index.sqlite") + + ctx := context.Background() + cfg := faultTestConfig() + + writeFaultSourceTree(t, fs, dataDir) + require.NoError(t, afero.WriteFile(fs, + filepath.Join(dataDir, unprintableFileName), []byte("odd"), 0o644)) + + store, err := storage.NewFileStorer(t.TempDir()) + require.NoError(t, err) + + db, err := database.New(ctx, dbPath) + require.NoError(t, err) + + id := fullFaultBackup(ctx, t, fs, store, cfg, + database.NewRepositories(db), dataDir, dbPath, "files") + require.NoError(t, db.Close()) + + v := newReaderVaultik(ctx, cfg, store, nil, fs) + v.Stdout = out + + return v, id, dataDir +} + +func TestSnapshotFilesJSONListsEveryFileWithItsMetadata(t *testing.T) { + log.Initialize(log.Config{}) + t.Parallel() + + var out bytes.Buffer + + v, id, dataDir := backUpForFileListing(t, &out) + + require.NoError(t, v.ListSnapshotFiles(id, "", true)) + + var entries []vaultik.ListedFile + + require.NoError(t, json.Unmarshal(out.Bytes(), &entries)) + + byPath := make(map[string]vaultik.ListedFile, len(entries)) + for _, entry := range entries { + byPath[entry.Path] = entry + } + + for _, rel := range []string{"small.txt", "a.bin", "sub", "sub/b.bin", "sub/c.bin"} { + path := filepath.Join(dataDir, rel) + + entry, ok := byPath[path] + require.True(t, ok, "%s must be listed", path) + + info, err := afero.NewOsFs().Stat(path) + require.NoError(t, err) + + assert.Equal(t, info.Mode().String(), entry.Mode, path) + assert.True(t, info.ModTime().Equal(entry.MTime), + "%s: mtime %v, listed %v", path, info.ModTime(), entry.MTime) + + if !info.IsDir() { + assert.Equal(t, info.Size(), entry.Size, path) + } + } +} + +func TestSnapshotFilesListsOnlyWhatIsUnderThePath(t *testing.T) { + log.Initialize(log.Config{}) + t.Parallel() + + var out bytes.Buffer + + v, id, dataDir := backUpForFileListing(t, &out) + + require.NoError(t, v.ListSnapshotFiles(id, filepath.Join(dataDir, "sub"), false)) + + text := out.String() + assert.Contains(t, text, filepath.Join(dataDir, "sub", "b.bin")) + assert.Contains(t, text, filepath.Join(dataDir, "sub", "c.bin")) + assert.NotContains(t, text, filepath.Join(dataDir, "small.txt")) + assert.True(t, strings.HasPrefix(text, "MODE"), "the table must have a header") +} + +// TestSnapshotFilesTableEscapesAnUnprintablePath checks that the table +// prints a path holding a newline and an escape sequence quoted and +// escaped on one row: the newline must not start a forged row, and the +// escape sequence must not reach the terminal. +func TestSnapshotFilesTableEscapesAnUnprintablePath(t *testing.T) { + log.Initialize(log.Config{}) + t.Parallel() + + var out bytes.Buffer + + v, id, dataDir := backUpForFileListing(t, &out) + + require.NoError(t, v.ListSnapshotFiles(id, "", false)) + + text := out.String() + assert.Contains(t, text, + strconv.Quote(filepath.Join(dataDir, unprintableFileName))) + assert.NotContains(t, text, "\x1b", "no escape byte may reach the output") + + for line := range strings.SplitSeq(text, "\n") { + assert.False(t, strings.HasPrefix(line, "name"), + "the newline in the name must not start a row: %q", line) + } +} + +// TestSnapshotFilesQuietSuppressesTableNotJSON is the --quiet contract +// for `snapshot files`: the table is silenced, but the --json document a +// script depends on is still written. +func TestSnapshotFilesQuietSuppressesTableNotJSON(t *testing.T) { + log.Initialize(log.Config{}) + t.Parallel() + + var out bytes.Buffer + + v, id, _ := backUpForFileListing(t, &out) + v.UI.SetQuiet(true) + + require.NoError(t, v.ListSnapshotFiles(id, "", false)) + require.Empty(t, out.String(), "the table must be silenced under --quiet") + + require.NoError(t, v.ListSnapshotFiles(id, "", true)) + + var entries []vaultik.ListedFile + + require.NoError(t, json.Unmarshal(out.Bytes(), &entries)) + assert.NotEmpty(t, entries, + "the --json document must still be written under --quiet") +} + +func TestSnapshotFilesFailsOnAPathThatMatchesNothing(t *testing.T) { + log.Initialize(log.Config{}) + t.Parallel() + + var out bytes.Buffer + + v, id, dataDir := backUpForFileListing(t, &out) + missing := filepath.Join(dataDir, "typo.txt") + + err := v.ListSnapshotFiles(id, missing, false) + + require.Error(t, err) + assert.Contains(t, err.Error(), missing, "the error must name the path") + assert.Empty(t, out.String(), "nothing must be printed") +} + +func TestSnapshotFilesRequiresThePrivateKey(t *testing.T) { + log.Initialize(log.Config{}) + t.Parallel() + + var out bytes.Buffer + + v, id, _ := backUpForFileListing(t, &out) + v.Config.AgeSecretKey = "" + + err := v.ListSnapshotFiles(id, "", false) + + require.Error(t, err) + assert.Contains(t, err.Error(), "VAULTIK_AGE_SECRET_KEY") +} diff --git a/internal/vaultik/snapshot_list.go b/internal/vaultik/snapshot_list.go index 7862e2e..0f2ca48 100644 --- a/internal/vaultik/snapshot_list.go +++ b/internal/vaultik/snapshot_list.go @@ -39,7 +39,8 @@ const maxRemoteOnlyRows = 1000 // manifest reads need throttling. const remoteManifestFetchConcurrency = 8 -// tabPadding is the tabwriter cell padding for the snapshot table. +// tabPadding is the tabwriter cell padding for the `snapshot list` and +// `snapshot files` tables. const tabPadding = 3 // ListSnapshots prints the table of snapshots known to this host: the