Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
68f4ccf5b9 | ||
|
|
aab6a87f8c |
@@ -104,7 +104,12 @@ Version: 2025-06-08
|
|||||||
|
|
||||||
13. Pre-1.0: NEVER write database migrations. There are no live databases
|
13. Pre-1.0: NEVER write database migrations. There are no live databases
|
||||||
anywhere — every user's local index can be rebuilt from a fresh full
|
anywhere — every user's local index can be rebuilt from a fresh full
|
||||||
backup. When the schema changes, just change `schema.sql` (and any code
|
backup. To change the schema, edit `internal/database/schema/001.sql`
|
||||||
that touches the affected tables). The local index is disposable until
|
(and any code that touches the affected tables) directly; do not add new
|
||||||
1.0 ships and is tagged.
|
numbered schema files. Those numbered files and the `schema_migrations`
|
||||||
|
table they populate only bootstrap a fresh database — they are not an
|
||||||
|
upgrade path. The local index is disposable until 1.0 ships and is
|
||||||
|
tagged; once 1.0 is tagged that clause expires and the question of
|
||||||
|
upgrading existing indexes returns. See [`docs/DATAMODEL.md`](docs/DATAMODEL.md)
|
||||||
|
for the full explanation.
|
||||||
|
|
||||||
|
|||||||
@@ -84,6 +84,57 @@ VAULTIK_AGE_SECRET_KEY='AGE-SECRET-KEY-...' vaultik snapshot restore <snapshot-i
|
|||||||
# 0 3 * * * vaultik snapshot create --cron --prune --keep-newer-than 4w
|
# 0 3 * * * vaultik snapshot create --cron --prune --keep-newer-than 4w
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## restoring on another machine
|
||||||
|
|
||||||
|
Restoring on a host that never ran the backup — a replacement machine
|
||||||
|
after the original is gone — is the case vaultik is built for. That host
|
||||||
|
needs only three things: the `vaultik` binary, the age **private** key,
|
||||||
|
and the storage credentials for the destination. It does **not** need the
|
||||||
|
local index, the original config file, or the original hostname.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# install
|
||||||
|
go install sneak.berlin/go/vaultik/cmd/vaultik@latest
|
||||||
|
|
||||||
|
# create a config and point it at the ORIGINAL backup destination
|
||||||
|
vaultik config init
|
||||||
|
vaultik config set storage_url "s3://bucket/prefix?endpoint=https://s3.example.com"
|
||||||
|
vaultik config set s3.access_key_id "..."
|
||||||
|
vaultik config set s3.secret_access_key "..."
|
||||||
|
|
||||||
|
# see what is on the destination store
|
||||||
|
vaultik snapshot list
|
||||||
|
```
|
||||||
|
|
||||||
|
`snapshot list` reads the destination store without the private key. A
|
||||||
|
snapshot that is not in this host's (empty) local index is shown as
|
||||||
|
remote-only: its row is identified by `<remote only:...>` rather than by
|
||||||
|
a `hostname_name_timestamp` name, because the name lives only in the
|
||||||
|
local index and the encrypted database and cannot be recovered from the
|
||||||
|
store. Its timestamp and compressed size are real. (See the `snapshot
|
||||||
|
list` description under [command details](#command-details) for the full
|
||||||
|
explanation.)
|
||||||
|
|
||||||
|
Use that remote key — the hex printed inside `<remote only:...>`, or the
|
||||||
|
full `remote_key` from `snapshot list --json` — to restore and verify:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# restore everything to /tmp/restored, then check every restored file's
|
||||||
|
# chunk hashes
|
||||||
|
VAULTIK_AGE_SECRET_KEY='AGE-SECRET-KEY-...' \
|
||||||
|
vaultik snapshot restore --verify <remote-key> /tmp/restored
|
||||||
|
|
||||||
|
# optionally, deep-verify the snapshot against the store (downloads and
|
||||||
|
# cryptographically checks every blob)
|
||||||
|
VAULTIK_AGE_SECRET_KEY='AGE-SECRET-KEY-...' \
|
||||||
|
vaultik snapshot verify --deep <remote-key>
|
||||||
|
```
|
||||||
|
|
||||||
|
`age_recipients` (the public key) is not needed to restore — only the
|
||||||
|
private key in `VAULTIK_AGE_SECRET_KEY`. Both the abbreviated key printed
|
||||||
|
in the table and the full 64-character key from `--json` are accepted; a
|
||||||
|
leading part of the key is enough as long as it is unambiguous.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## cli
|
## cli
|
||||||
@@ -245,6 +296,8 @@ local index alone, and still exits zero.
|
|||||||
* Default (shallow): checks that all blobs referenced in the manifest exist in storage
|
* Default (shallow): checks that all blobs referenced in the manifest exist in storage
|
||||||
* `--deep`: Downloads and decrypts each blob, verifies chunk hashes against the
|
* `--deep`: Downloads and decrypts each blob, verifies chunk hashes against the
|
||||||
encrypted metadata database
|
encrypted metadata database
|
||||||
|
* Accepts the same identifiers as `snapshot restore`: a snapshot ID, or a
|
||||||
|
remote-only snapshot's remote key (or an unambiguous leading part of it)
|
||||||
* `--json`: Output results as JSON
|
* `--json`: Output results as JSON
|
||||||
|
|
||||||
**`snapshot purge`**: Remove old snapshots based on criteria. Retention is
|
**`snapshot purge`**: Remove old snapshots based on criteria. Retention is
|
||||||
@@ -275,6 +328,10 @@ on the destination in one go, use `vaultik remote nuke --force`.
|
|||||||
|
|
||||||
**`snapshot restore`**: Restore files from a backup snapshot.
|
**`snapshot restore`**: Restore files from a backup snapshot.
|
||||||
* Requires `VAULTIK_AGE_SECRET_KEY` environment variable
|
* Requires `VAULTIK_AGE_SECRET_KEY` environment variable
|
||||||
|
* Accepts a snapshot ID, or — for a snapshot only on the destination
|
||||||
|
store — its remote key (or an unambiguous leading part of it) as shown
|
||||||
|
by `snapshot list`. See
|
||||||
|
[restoring on another machine](#restoring-on-another-machine).
|
||||||
* Optional path arguments to restore specific files/directories (default: all)
|
* Optional path arguments to restore specific files/directories (default: all)
|
||||||
* Preserves file permissions, timestamps, ownership (ownership requires root),
|
* Preserves file permissions, timestamps, ownership (ownership requires root),
|
||||||
symlinks, and empty directories
|
symlinks, and empty directories
|
||||||
@@ -457,9 +514,13 @@ Key fields:
|
|||||||
sequentially. Restore speed is bound by single-stream throughput.
|
sequentially. Restore speed is bound by single-stream throughput.
|
||||||
* **Device nodes, named pipes, and sockets are silently skipped.** Only
|
* **Device nodes, named pipes, and sockets are silently skipped.** Only
|
||||||
regular files, directories, and symlinks are backed up.
|
regular files, directories, and symlinks are backed up.
|
||||||
* **No database migrations.** If the local SQLite schema changes between
|
* **No upgrade path between versions.** There is no supported way to carry
|
||||||
versions, delete the local database (`vaultik database delete`) and run
|
an existing local index across a schema change; if the local SQLite
|
||||||
a full backup. Remote storage is unaffected.
|
schema changes between versions, delete the local database (`vaultik
|
||||||
|
database delete`) and run a full backup. Remote storage is unaffected.
|
||||||
|
(The binary does embed numbered schema files and a `schema_migrations`
|
||||||
|
table to bootstrap a fresh database — see [`docs/DATAMODEL.md`](docs/DATAMODEL.md)
|
||||||
|
— but that is not an upgrade path.)
|
||||||
* **Files that change during backup may be inconsistent.** There is no
|
* **Files that change during backup may be inconsistent.** There is no
|
||||||
filesystem snapshot or freeze. If a file is modified between the scan
|
filesystem snapshot or freeze. If a file is modified between the scan
|
||||||
and chunk phases, the backed-up copy may reflect a partial write.
|
and chunk phases, the backed-up copy may reflect a partial write.
|
||||||
@@ -525,14 +586,12 @@ priority.
|
|||||||
|
|
||||||
### infrastructure
|
### infrastructure
|
||||||
|
|
||||||
* **Cross-machine restore documentation.** The "restore from
|
* **Cross-version schema upgrades.** There is no upgrade path between
|
||||||
another host" workflow works but isn't documented as a
|
released versions — pre-1.0 schema changes are handled by `vaultik
|
||||||
first-class operation in this README. Worth a dedicated section
|
database delete` plus a full re-scan (see
|
||||||
once it's settled.
|
[`docs/DATAMODEL.md`](docs/DATAMODEL.md)). Post-1.0 we'll need a
|
||||||
* **Schema migrations.** Currently nonexistent — pre-1.0 schema
|
migration story to keep existing index databases usable across
|
||||||
changes are handled by `vaultik database delete` plus a full
|
upgrades.
|
||||||
re-scan. Post-1.0 we'll need a migration story to keep existing
|
|
||||||
index databases usable across upgrades.
|
|
||||||
* **Storage backend coverage tests.** S3, file://, and rclone://
|
* **Storage backend coverage tests.** S3, file://, and rclone://
|
||||||
all share the Storer interface but the rclone path is the least
|
all share the Storer interface but the rclone path is the least
|
||||||
exercised in CI.
|
exercised in CI.
|
||||||
|
|||||||
+24
-5
@@ -5,11 +5,30 @@
|
|||||||
Vaultik uses a local SQLite database to track file metadata, chunk mappings, and blob associations during the backup process. This database serves as an index for incremental backups and enables efficient deduplication.
|
Vaultik uses a local SQLite database to track file metadata, chunk mappings, and blob associations during the backup process. This database serves as an index for incremental backups and enables efficient deduplication.
|
||||||
|
|
||||||
**Important Notes:**
|
**Important Notes:**
|
||||||
- **No Migration Support (pre-1.0)**: Vaultik does not support database schema
|
|
||||||
migrations. The local index is treated as disposable — if the schema changes,
|
This section is the authoritative explanation of the schema/migration story;
|
||||||
delete the local SQLite database (`vaultik database delete`) and run a full
|
other documents (the README and `AGENTS.md`) link here.
|
||||||
backup. The remote storage is unaffected; the new index will re-deduplicate
|
|
||||||
against existing remote blobs.
|
- **No upgrade path between versions (pre-1.0)**: Vaultik has no supported way to
|
||||||
|
carry an existing local index across a schema change. The index is disposable
|
||||||
|
— if the on-disk schema changes between versions, delete the local SQLite
|
||||||
|
database (`vaultik database delete`) and run a full backup. Remote storage is
|
||||||
|
unaffected; the new index re-deduplicates against existing remote blobs. This
|
||||||
|
is the standing project policy, and it is separate from the schema bootstrap
|
||||||
|
described next.
|
||||||
|
- **Schema bootstrap**: a fresh database is populated from numbered SQL files
|
||||||
|
embedded in the binary under `internal/database/schema/`. `000.sql` creates the
|
||||||
|
`schema_migrations` table; `001.sql` creates the application tables. On opening
|
||||||
|
a database the code applies each numbered file that has not yet run and records
|
||||||
|
its version in `schema_migrations`. This bootstraps a new database; it does not
|
||||||
|
upgrade an existing one between released versions.
|
||||||
|
- **Changing the schema (pre-1.0)**: edit `internal/database/schema/001.sql` (and
|
||||||
|
the code that touches the affected tables) directly. Do not add new numbered
|
||||||
|
files — there is no installed base to migrate.
|
||||||
|
- **Disposability expires at 1.0**: the index is treated as disposable only until
|
||||||
|
1.0 ships and is tagged. Once 1.0 is tagged that clause expires and the
|
||||||
|
question of upgrading existing indexes returns. It is deliberately left open
|
||||||
|
here.
|
||||||
- **Version Compatibility**: In rare cases, you may need to use the same version
|
- **Version Compatibility**: In rare cases, you may need to use the same version
|
||||||
of Vaultik to restore a backup as was used to create it. This ensures
|
of Vaultik to restore a backup as was used to create it. This ensures
|
||||||
compatibility with the metadata format stored in S3.
|
compatibility with the metadata format stored in S3.
|
||||||
|
|||||||
@@ -221,7 +221,10 @@ func newSnapshotVerifyCommand() *cobra.Command {
|
|||||||
cmd := &cobra.Command{
|
cmd := &cobra.Command{
|
||||||
Use: "verify <snapshot-id>",
|
Use: "verify <snapshot-id>",
|
||||||
Short: "Verify snapshot integrity",
|
Short: "Verify snapshot integrity",
|
||||||
Long: "Verifies that all blobs referenced in a snapshot exist",
|
Long: "Verifies that all blobs referenced in a snapshot exist.\n\n" +
|
||||||
|
"The snapshot may be named by its ID or, on a host with no local\n" +
|
||||||
|
"index, by the remote key that 'snapshot list' prints for a\n" +
|
||||||
|
"remote-only snapshot (an unambiguous leading part is enough).",
|
||||||
Args: requireSnapshotIDArg,
|
Args: requireSnapshotIDArg,
|
||||||
RunE: func(cmd *cobra.Command, args []string) error {
|
RunE: func(cmd *cobra.Command, args []string) error {
|
||||||
snapshotID := args[0]
|
snapshotID := args[0]
|
||||||
|
|||||||
@@ -48,6 +48,10 @@ target directory.
|
|||||||
If no paths are specified, all files are restored.
|
If no paths are specified, all files are restored.
|
||||||
If paths are specified, only matching files/directories are restored.
|
If paths are specified, only matching files/directories are restored.
|
||||||
|
|
||||||
|
The snapshot may be named by its ID or, when restoring on a host with no
|
||||||
|
local index, by the remote key that 'snapshot list' prints for a
|
||||||
|
remote-only snapshot (an unambiguous leading part is enough).
|
||||||
|
|
||||||
Requires the VAULTIK_AGE_SECRET_KEY environment variable to be set with
|
Requires the VAULTIK_AGE_SECRET_KEY environment variable to be set with
|
||||||
the age private key.
|
the age private key.
|
||||||
|
|
||||||
|
|||||||
@@ -1,198 +0,0 @@
|
|||||||
package storage_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bytes"
|
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"io"
|
|
||||||
"reflect"
|
|
||||||
"sort"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"sneak.berlin/go/vaultik/internal/storage"
|
|
||||||
)
|
|
||||||
|
|
||||||
// runStorerConformance is the shared Storer contract. Every backend that
|
|
||||||
// can run in-process is expected to pass it: TestFileStorer runs it against
|
|
||||||
// file://, TestS3Storer against s3://. A new backend inherits this coverage
|
|
||||||
// by passing its own constructor, so the contract is defined once.
|
|
||||||
//
|
|
||||||
// It exercises the public Storer interface: round-trip, stat, list with
|
|
||||||
// prefix filtering, overwrite, delete, delete-of-missing, and not-found on
|
|
||||||
// Get and Stat. Each section takes its own fresh backend instance, so the
|
|
||||||
// order of sections never matters and no section sees another's objects.
|
|
||||||
func runStorerConformance(t *testing.T, newStorer func(*testing.T) storage.Storer) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
conformanceRoundTrip(t, newStorer(t))
|
|
||||||
conformanceOverwrite(t, newStorer(t))
|
|
||||||
conformanceList(t, newStorer(t))
|
|
||||||
conformanceDelete(t, newStorer(t))
|
|
||||||
conformanceNotFound(t, newStorer(t))
|
|
||||||
}
|
|
||||||
|
|
||||||
// conformanceRoundTrip stores a nested key, then reads it back and stats it.
|
|
||||||
func conformanceRoundTrip(t *testing.T, s storage.Storer) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
ctx := context.Background()
|
|
||||||
key := "blobs/aa/bb/object.bin"
|
|
||||||
want := []byte("round-trip payload")
|
|
||||||
|
|
||||||
err := s.Put(ctx, key, bytes.NewReader(want))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Put: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
got := getBytes(t, s, key)
|
|
||||||
if !bytes.Equal(got, want) {
|
|
||||||
t.Errorf("Get returned %q, want %q", got, want)
|
|
||||||
}
|
|
||||||
|
|
||||||
info, err := s.Stat(ctx, key)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Stat: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if info.Key != key {
|
|
||||||
t.Errorf("Stat key = %q, want %q", info.Key, key)
|
|
||||||
}
|
|
||||||
|
|
||||||
if info.Size != int64(len(want)) {
|
|
||||||
t.Errorf("Stat size = %d, want %d", info.Size, len(want))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// conformanceOverwrite checks that a second Put replaces the first.
|
|
||||||
func conformanceOverwrite(t *testing.T, s storage.Storer) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
ctx := context.Background()
|
|
||||||
key := "meta/snapshot.json"
|
|
||||||
|
|
||||||
err := s.Put(ctx, key, bytes.NewReader([]byte("first")))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("first Put: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
want := []byte("second and longer payload")
|
|
||||||
|
|
||||||
err = s.Put(ctx, key, bytes.NewReader(want))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("second Put: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
got := getBytes(t, s, key)
|
|
||||||
if !bytes.Equal(got, want) {
|
|
||||||
t.Errorf("after overwrite Get returned %q, want %q", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// conformanceList checks prefix filtering and the empty result for a
|
|
||||||
// prefix that matches nothing.
|
|
||||||
func conformanceList(t *testing.T, s storage.Storer) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
ctx := context.Background()
|
|
||||||
keys := []string{"blobs/aa/one", "blobs/bb/two", "meta/three"}
|
|
||||||
|
|
||||||
for _, k := range keys {
|
|
||||||
err := s.Put(ctx, k, bytes.NewReader([]byte("data")))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Put %q: %v", k, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if got := listSorted(t, s, ""); !reflect.DeepEqual(got, keys) {
|
|
||||||
t.Errorf("List(\"\") = %v, want %v", got, keys)
|
|
||||||
}
|
|
||||||
|
|
||||||
wantBlobs := []string{"blobs/aa/one", "blobs/bb/two"}
|
|
||||||
if got := listSorted(t, s, "blobs/"); !reflect.DeepEqual(got, wantBlobs) {
|
|
||||||
t.Errorf("List(\"blobs/\") = %v, want %v", got, wantBlobs)
|
|
||||||
}
|
|
||||||
|
|
||||||
if got := listSorted(t, s, "absent/"); len(got) != 0 {
|
|
||||||
t.Errorf("List(\"absent/\") = %v, want empty", got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// conformanceDelete checks that Delete removes an object and that deleting
|
|
||||||
// a missing key is not an error.
|
|
||||||
func conformanceDelete(t *testing.T, s storage.Storer) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
ctx := context.Background()
|
|
||||||
key := "blobs/cc/gone.bin"
|
|
||||||
|
|
||||||
err := s.Put(ctx, key, bytes.NewReader([]byte("temporary")))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Put: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = s.Delete(ctx, key)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Delete: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = s.Get(ctx, key)
|
|
||||||
if !errors.Is(err, storage.ErrNotFound) {
|
|
||||||
t.Errorf("Get after Delete error = %v, want ErrNotFound", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = s.Delete(ctx, key)
|
|
||||||
if err != nil {
|
|
||||||
t.Errorf("Delete of missing key = %v, want nil", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// conformanceNotFound checks Get and Stat on an absent key.
|
|
||||||
func conformanceNotFound(t *testing.T, s storage.Storer) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
ctx := context.Background()
|
|
||||||
key := "never/written"
|
|
||||||
|
|
||||||
_, err := s.Get(ctx, key)
|
|
||||||
if !errors.Is(err, storage.ErrNotFound) {
|
|
||||||
t.Errorf("Get error = %v, want ErrNotFound", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = s.Stat(ctx, key)
|
|
||||||
if !errors.Is(err, storage.ErrNotFound) {
|
|
||||||
t.Errorf("Stat error = %v, want ErrNotFound", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// getBytes reads a key fully and closes the reader.
|
|
||||||
func getBytes(t *testing.T, s storage.Storer, key string) []byte {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
rc, err := s.Get(context.Background(), key)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Get %q: %v", key, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = rc.Close() }()
|
|
||||||
|
|
||||||
data, err := io.ReadAll(rc)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("read %q: %v", key, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return data
|
|
||||||
}
|
|
||||||
|
|
||||||
// listSorted returns the keys under a prefix in a stable order.
|
|
||||||
func listSorted(t *testing.T, s storage.Storer, prefix string) []string {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
keys, err := s.List(context.Background(), prefix)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("List %q: %v", prefix, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
sort.Strings(keys)
|
|
||||||
|
|
||||||
return keys
|
|
||||||
}
|
|
||||||
@@ -1,27 +0,0 @@
|
|||||||
package storage_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"sneak.berlin/go/vaultik/internal/storage"
|
|
||||||
)
|
|
||||||
|
|
||||||
// newFileStorer builds a file:// backend rooted at a fresh temp directory.
|
|
||||||
//
|
|
||||||
//nolint:ireturn // conformance runs against the Storer interface by design
|
|
||||||
func newFileStorer(t *testing.T) storage.Storer {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
s, err := storage.NewFileStorer(t.TempDir())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("NewFileStorer: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return s
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestFileStorer runs the shared Storer contract against the file:// backend.
|
|
||||||
func TestFileStorer(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
runStorerConformance(t, newFileStorer)
|
|
||||||
}
|
|
||||||
@@ -1,58 +0,0 @@
|
|||||||
package storage_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"sneak.berlin/go/vaultik/internal/storage"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The rclone backend is a thin adapter over the rclone library: it turns a
|
|
||||||
// (remote, path) pair into rclone's "remote:path" string, hands it to
|
|
||||||
// rclone, and maps rclone's own results back to the Storer interface. What
|
|
||||||
// can be tested in-process, without a configured remote or network, is that
|
|
||||||
// adapter layer — how the arguments are shaped and how construction errors
|
|
||||||
// are reported. The data-plane operations (Put/Get/List/Delete) are rclone's
|
|
||||||
// own, exercised against a real provider (drive, s3-via-rclone, ...), which
|
|
||||||
// needs a configured remote with credentials and network access and so is
|
|
||||||
// out of reach of a unit test. The shared Storer conformance suite therefore
|
|
||||||
// runs against the in-process file and s3 backends; the rclone backend
|
|
||||||
// inherits that contract once a remote is configured.
|
|
||||||
//
|
|
||||||
// These tests use rclone's ":local:" on-the-fly backend, which addresses the
|
|
||||||
// local filesystem directly without any configured remote, so construction
|
|
||||||
// runs entirely in-process.
|
|
||||||
|
|
||||||
// TestNewRcloneStorerConstruction checks that a valid remote constructs a
|
|
||||||
// backend and that Info() reports the shaped "remote:path" location.
|
|
||||||
//
|
|
||||||
//nolint:paralleltest // NewRcloneStorer installs the process-global rclone config
|
|
||||||
func TestNewRcloneStorerConstruction(t *testing.T) {
|
|
||||||
dir := t.TempDir()
|
|
||||||
|
|
||||||
s, err := storage.NewRcloneStorer(context.Background(), ":local", dir)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("NewRcloneStorer: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Info().Location is the "remote:path" string the adapter builds from
|
|
||||||
// its two arguments, so asserting it confirms the argument shaping.
|
|
||||||
want := ":local:" + dir
|
|
||||||
if got := s.Info().Location; got != want {
|
|
||||||
t.Errorf("Info().Location = %q, want %q", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestNewRcloneStorerUnknownRemote checks that a remote that is not in the
|
|
||||||
// rclone config fails construction with the ErrRemoteNotFound sentinel,
|
|
||||||
// rather than silently returning a backend pointed nowhere.
|
|
||||||
//
|
|
||||||
//nolint:paralleltest // NewRcloneStorer installs the process-global rclone config
|
|
||||||
func TestNewRcloneStorerUnknownRemote(t *testing.T) {
|
|
||||||
_, err := storage.NewRcloneStorer(
|
|
||||||
context.Background(), "vaultik-no-such-remote", "path")
|
|
||||||
if !errors.Is(err, storage.ErrRemoteNotFound) {
|
|
||||||
t.Errorf("NewRcloneStorer error = %v, want ErrRemoteNotFound", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
+14
-36
@@ -13,23 +13,18 @@ import (
|
|||||||
"sneak.berlin/go/vaultik/internal/storage"
|
"sneak.berlin/go/vaultik/internal/storage"
|
||||||
)
|
)
|
||||||
|
|
||||||
// s3TestBucket is the bucket created for each in-process S3 server.
|
// TestS3StorerMissingKeyMapsToErrNotFound verifies that the s3 backend reports
|
||||||
const s3TestBucket = "test-bucket"
|
// a missing object as storage.ErrNotFound, matching the file and rclone
|
||||||
|
// backends and the Storer contract. Without the mapping, Get and Stat leak the
|
||||||
// newS3Storer builds an s3:// backend backed by a fresh in-process
|
// raw SDK error and errors.Is(err, storage.ErrNotFound) is false.
|
||||||
// S3 server. It reuses the same in-memory S3 harness (gofakes3 + s3mem
|
|
||||||
// over httptest) that internal/s3 and the not-found regression test use,
|
|
||||||
// so no new mock or dependency is introduced. Each call gets its own
|
|
||||||
// server, bucket, and client, so the conformance suite's per-section
|
|
||||||
// instances stay isolated.
|
|
||||||
//
|
//
|
||||||
//nolint:ireturn // conformance runs against the Storer interface by design
|
//nolint:paralleltest // shares an in-process S3 server via t.Cleanup
|
||||||
func newS3Storer(t *testing.T) storage.Storer {
|
func TestS3StorerMissingKeyMapsToErrNotFound(t *testing.T) {
|
||||||
t.Helper()
|
const bucket = "test-bucket"
|
||||||
|
|
||||||
backend := s3mem.New()
|
backend := s3mem.New()
|
||||||
|
|
||||||
err := backend.CreateBucket(s3TestBucket)
|
err := backend.CreateBucket(bucket)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("create bucket: %v", err)
|
t.Fatalf("create bucket: %v", err)
|
||||||
}
|
}
|
||||||
@@ -37,9 +32,11 @@ func newS3Storer(t *testing.T) storage.Storer {
|
|||||||
srv := httptest.NewServer(gofakes3.New(backend).Server())
|
srv := httptest.NewServer(gofakes3.New(backend).Server())
|
||||||
t.Cleanup(srv.Close)
|
t.Cleanup(srv.Close)
|
||||||
|
|
||||||
client, err := s3.NewClient(context.Background(), s3.Config{
|
ctx := context.Background()
|
||||||
|
|
||||||
|
client, err := s3.NewClient(ctx, s3.Config{
|
||||||
Endpoint: srv.URL,
|
Endpoint: srv.URL,
|
||||||
Bucket: s3TestBucket,
|
Bucket: bucket,
|
||||||
AccessKeyID: "test",
|
AccessKeyID: "test",
|
||||||
SecretAccessKey: "test",
|
SecretAccessKey: "test",
|
||||||
Region: "us-east-1",
|
Region: "us-east-1",
|
||||||
@@ -48,28 +45,9 @@ func newS3Storer(t *testing.T) storage.Storer {
|
|||||||
t.Fatalf("new client: %v", err)
|
t.Fatalf("new client: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
return storage.NewS3Storer(client)
|
storer := storage.NewS3Storer(client)
|
||||||
}
|
|
||||||
|
|
||||||
// TestS3Storer runs the shared Storer contract against the s3:// backend,
|
_, err = storer.Get(ctx, "does-not-exist")
|
||||||
// so it is held to the same round-trip, list, delete, and not-found
|
|
||||||
// behaviour as the file:// backend.
|
|
||||||
func TestS3Storer(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
runStorerConformance(t, newS3Storer)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestS3StorerMissingKeyMapsToErrNotFound pins the specific contract that a
|
|
||||||
// missing object surfaces as storage.ErrNotFound rather than the raw AWS SDK
|
|
||||||
// error. Without the mapping, errors.Is(err, storage.ErrNotFound) is false on
|
|
||||||
// s3 and callers would branch differently per backend.
|
|
||||||
func TestS3StorerMissingKeyMapsToErrNotFound(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
storer := newS3Storer(t)
|
|
||||||
ctx := context.Background()
|
|
||||||
|
|
||||||
_, err := storer.Get(ctx, "does-not-exist")
|
|
||||||
if !errors.Is(err, storage.ErrNotFound) {
|
if !errors.Is(err, storage.ErrNotFound) {
|
||||||
t.Errorf("Get on missing key: got %v, want ErrNotFound", err)
|
t.Errorf("Get on missing key: got %v, want ErrNotFound", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,110 +0,0 @@
|
|||||||
package storage_test
|
|
||||||
|
|
||||||
import (
|
|
||||||
"errors"
|
|
||||||
"reflect"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"sneak.berlin/go/vaultik/internal/storage"
|
|
||||||
)
|
|
||||||
|
|
||||||
// TestParseStorageURLValid checks that each supported scheme parses into
|
|
||||||
// the expected fields, since those fields decide which backend is built.
|
|
||||||
func TestParseStorageURLValid(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const bucket = "mybucket"
|
|
||||||
|
|
||||||
cases := []struct {
|
|
||||||
name string
|
|
||||||
raw string
|
|
||||||
want *storage.URL
|
|
||||||
}{
|
|
||||||
{
|
|
||||||
name: "file absolute path",
|
|
||||||
raw: "file:///var/backups/vaultik",
|
|
||||||
want: &storage.URL{Scheme: "file", Prefix: "/var/backups/vaultik"},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "s3 bucket and prefix, ssl defaults on",
|
|
||||||
raw: "s3://mybucket/backups/host",
|
|
||||||
want: &storage.URL{
|
|
||||||
Scheme: "s3", Bucket: bucket,
|
|
||||||
Prefix: "backups/host", UseSSL: true,
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "s3 bucket only",
|
|
||||||
raw: "s3://mybucket",
|
|
||||||
want: &storage.URL{Scheme: "s3", Bucket: bucket, UseSSL: true},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "s3 with endpoint, region, ssl off",
|
|
||||||
raw: "s3://mybucket?endpoint=minio.example.com®ion=us-west-2&ssl=false",
|
|
||||||
want: &storage.URL{
|
|
||||||
Scheme: "s3", Bucket: bucket,
|
|
||||||
Endpoint: "minio.example.com", Region: "us-west-2", UseSSL: false,
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "rclone remote and path",
|
|
||||||
raw: "rclone://gdrive/backups/host",
|
|
||||||
want: &storage.URL{
|
|
||||||
Scheme: "rclone", RcloneRemote: "gdrive", Prefix: "backups/host",
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
name: "rclone remote only",
|
|
||||||
raw: "rclone://gdrive",
|
|
||||||
want: &storage.URL{Scheme: "rclone", RcloneRemote: "gdrive"},
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tc := range cases {
|
|
||||||
t.Run(tc.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
got, err := storage.ParseStorageURL(tc.raw)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("ParseStorageURL(%q) returned error: %v", tc.raw, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if !reflect.DeepEqual(got, tc.want) {
|
|
||||||
t.Errorf("ParseStorageURL(%q) = %+v, want %+v", tc.raw, got, tc.want)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestParseStorageURLErrors checks that empty, missing, and unknown-scheme
|
|
||||||
// inputs fail with the documented sentinel errors instead of parsing to a
|
|
||||||
// wrong destination.
|
|
||||||
func TestParseStorageURLErrors(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
cases := []struct {
|
|
||||||
name string
|
|
||||||
raw string
|
|
||||||
wantErr error
|
|
||||||
}{
|
|
||||||
{"empty url", "", storage.ErrEmptyStorageURL},
|
|
||||||
{"file empty path", "file://", storage.ErrEmptyFilePath},
|
|
||||||
{"s3 missing bucket", "s3://", storage.ErrMissingBucket},
|
|
||||||
{"s3 missing bucket with path", "s3:///justprefix", storage.ErrMissingBucket},
|
|
||||||
{"rclone missing remote", "rclone://", storage.ErrMissingRemote},
|
|
||||||
{"unknown scheme", "gs://bucket/x", storage.ErrUnsupportedScheme},
|
|
||||||
{"no scheme", "/local/path", storage.ErrUnsupportedScheme},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, tc := range cases {
|
|
||||||
t.Run(tc.name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
_, err := storage.ParseStorageURL(tc.raw)
|
|
||||||
if !errors.Is(err, tc.wantErr) {
|
|
||||||
t.Errorf("ParseStorageURL(%q) error = %v, want %v",
|
|
||||||
tc.raw, err, tc.wantErr)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -18,7 +18,6 @@ import (
|
|||||||
"sneak.berlin/go/vaultik/internal/blobgen"
|
"sneak.berlin/go/vaultik/internal/blobgen"
|
||||||
"sneak.berlin/go/vaultik/internal/database"
|
"sneak.berlin/go/vaultik/internal/database"
|
||||||
"sneak.berlin/go/vaultik/internal/log"
|
"sneak.berlin/go/vaultik/internal/log"
|
||||||
"sneak.berlin/go/vaultik/internal/snapshot"
|
|
||||||
"sneak.berlin/go/vaultik/internal/types"
|
"sneak.berlin/go/vaultik/internal/types"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -577,14 +576,20 @@ func (v *Vaultik) handleRestoreVerification(
|
|||||||
}
|
}
|
||||||
|
|
||||||
// downloadSnapshotDB downloads and decrypts the snapshot metadata
|
// downloadSnapshotDB downloads and decrypts the snapshot metadata
|
||||||
// database. The snapshotID is the human ID; we hash it to the remote
|
// database. The identifier is resolved to the snapshot's remote key: a
|
||||||
// key for the storage path.
|
// human ID is hashed, and a remote key (or its abbreviation, as printed
|
||||||
|
// for a remote-only snapshot) is used as-is, so a host with no local
|
||||||
|
// index can restore the snapshots it can only see on the store.
|
||||||
func (v *Vaultik) downloadSnapshotDB(
|
func (v *Vaultik) downloadSnapshotDB(
|
||||||
snapshotID string, identity age.Identity,
|
snapshotID string, identity age.Identity,
|
||||||
) (*database.DB, error) {
|
) (*database.DB, error) {
|
||||||
|
remoteKey, err := v.resolveSnapshotRemoteKey(snapshotID)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
// Download encrypted database from storage
|
// Download encrypted database from storage
|
||||||
dbKey := fmt.Sprintf("metadata/%s/db.zst.age",
|
dbKey := fmt.Sprintf("metadata/%s/db.zst.age", remoteKey)
|
||||||
snapshot.RemoteSnapshotKey(snapshotID))
|
|
||||||
|
|
||||||
reader, err := v.Storage.Get(v.ctx, dbKey)
|
reader, err := v.Storage.Get(v.ctx, dbKey)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -0,0 +1,167 @@
|
|||||||
|
package vaultik_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"io"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/spf13/afero"
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"sneak.berlin/go/vaultik/internal/config"
|
||||||
|
"sneak.berlin/go/vaultik/internal/database"
|
||||||
|
"sneak.berlin/go/vaultik/internal/log"
|
||||||
|
"sneak.berlin/go/vaultik/internal/snapshot"
|
||||||
|
"sneak.berlin/go/vaultik/internal/storage"
|
||||||
|
"sneak.berlin/go/vaultik/internal/ui"
|
||||||
|
"sneak.berlin/go/vaultik/internal/vaultik"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestRestoreOnAnotherMachine proves the disaster-recovery path: a host
|
||||||
|
// that has only the vaultik binary, the age secret key, and the storage
|
||||||
|
// credentials — no local index, a different hostname, and no
|
||||||
|
// age_recipients configured — can list, restore, and verify a snapshot
|
||||||
|
// straight from the destination store.
|
||||||
|
//
|
||||||
|
// The backup half writes a snapshot with one index and hostname. The
|
||||||
|
// restore half throws that index away entirely: a fresh, empty index and
|
||||||
|
// a config that shares nothing with the original but the storage location
|
||||||
|
// and the secret key. If restore or verify needed the original local
|
||||||
|
// index — or the human snapshot ID that only that index holds — this test
|
||||||
|
// could not run, because the recovery host can know neither.
|
||||||
|
func TestRestoreOnAnotherMachine(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
fs := afero.NewOsFs()
|
||||||
|
tempDir := t.TempDir()
|
||||||
|
|
||||||
|
dataDir := filepath.Join(tempDir, "source")
|
||||||
|
storeDir := filepath.Join(tempDir, "remote")
|
||||||
|
restoreDir := filepath.Join(tempDir, "restored")
|
||||||
|
dbPath := filepath.Join(tempDir, "index.sqlite")
|
||||||
|
|
||||||
|
chunkSize := int64(64 * 1024)
|
||||||
|
maxBlobSize := int64(512 * 1024)
|
||||||
|
|
||||||
|
sourceFiles := writeRecoverySourceTree(t, fs, dataDir, chunkSize)
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
// Backup host: one index, hostname test-host, age_recipients set.
|
||||||
|
// runFileStorageBackup closes the index before returning, so nothing
|
||||||
|
// below can lean on it.
|
||||||
|
_, storer, originalID := runFileStorageBackup(
|
||||||
|
ctx, t, fs, dataDir, storeDir, dbPath, chunkSize, maxBlobSize)
|
||||||
|
|
||||||
|
// Recovery host: a fresh empty index, a different hostname, and no
|
||||||
|
// age_recipients — only the secret key and the same storage location.
|
||||||
|
recovery, stdout := newRecoveryHost(ctx, t, fs, storer)
|
||||||
|
|
||||||
|
// The recovery index really is empty. This is the assertion that makes
|
||||||
|
// the test a guard against restore quietly depending on the original
|
||||||
|
// index: if it did, an empty index would make restore fail.
|
||||||
|
localSnaps, err := recovery.Repositories.Snapshots.ListRecent(ctx, 100)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Empty(t, localSnaps, "recovery host must start with no local index")
|
||||||
|
|
||||||
|
// List: the snapshot shows up as remote-only, identified by its remote
|
||||||
|
// key, with no recoverable human ID.
|
||||||
|
require.NoError(t, recovery.ListSnapshots(true))
|
||||||
|
|
||||||
|
rows := decodeListJSON(t, stdout.String())
|
||||||
|
require.Len(t, rows, 1)
|
||||||
|
|
||||||
|
remote := rows[0]
|
||||||
|
assert.False(t, remote.LocallyTracked, "snapshot must be remote-only here")
|
||||||
|
assert.Empty(t, remote.ID, "the human ID is unknown to the recovery host")
|
||||||
|
require.Len(t, remote.RemoteKey, 64)
|
||||||
|
assert.Equal(t, snapshot.RemoteSnapshotKey(originalID), remote.RemoteKey,
|
||||||
|
"the listed key is the hashed snapshot ID")
|
||||||
|
|
||||||
|
// Restore driven by the abbreviated identifier the table prints (the
|
||||||
|
// first 12 hex of the remote key), then deep-verify from the store
|
||||||
|
// keyed by the full remote key. Both are what a recovery host can know.
|
||||||
|
require.NoError(t, recovery.Restore(&vaultik.RestoreOptions{
|
||||||
|
SnapshotID: remote.RemoteKey[:12],
|
||||||
|
TargetDir: restoreDir,
|
||||||
|
Verify: true,
|
||||||
|
}))
|
||||||
|
require.NoError(t, recovery.RunDeepVerify(
|
||||||
|
remote.RemoteKey, &vaultik.VerifyOptions{Deep: true}))
|
||||||
|
|
||||||
|
assertRestoredTreeMatches(t, fs, restoreDir, sourceFiles)
|
||||||
|
}
|
||||||
|
|
||||||
|
// writeRecoverySourceTree writes a small source tree spanning several
|
||||||
|
// chunks (so restore reassembles real multi-chunk files) and returns the
|
||||||
|
// content keyed by absolute path.
|
||||||
|
func writeRecoverySourceTree(
|
||||||
|
t *testing.T, fs afero.Fs, dataDir string, chunkSize int64,
|
||||||
|
) map[string][]byte {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
sourceFiles := map[string][]byte{
|
||||||
|
filepath.Join(dataDir, "notes.txt"): []byte("recover me"),
|
||||||
|
filepath.Join(dataDir, "sub", "big.bin"): bytesPattern("big-", int(chunkSize*3)),
|
||||||
|
filepath.Join(dataDir, "sub", "small.bin"): bytesPattern("small-", 128),
|
||||||
|
}
|
||||||
|
|
||||||
|
for path, content := range sourceFiles {
|
||||||
|
require.NoError(t, fs.MkdirAll(filepath.Dir(path), 0o755))
|
||||||
|
require.NoError(t, afero.WriteFile(fs, path, content, 0o644))
|
||||||
|
}
|
||||||
|
|
||||||
|
return sourceFiles
|
||||||
|
}
|
||||||
|
|
||||||
|
// newRecoveryHost builds the Vaultik a replacement machine would run: an
|
||||||
|
// empty in-memory index, a hostname different from the backup host, no
|
||||||
|
// age_recipients, and only the secret key plus the shared storer. It
|
||||||
|
// returns the instance and the buffer its stdout is wired to.
|
||||||
|
func newRecoveryHost(
|
||||||
|
ctx context.Context, t *testing.T, fs afero.Fs, storer storage.Storer,
|
||||||
|
) (*vaultik.Vaultik, *bytes.Buffer) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
recoveryDB, err := database.New(ctx, ":memory:")
|
||||||
|
require.NoError(t, err)
|
||||||
|
t.Cleanup(func() { _ = recoveryDB.Close() })
|
||||||
|
|
||||||
|
stdout := &bytes.Buffer{}
|
||||||
|
|
||||||
|
recovery := &vaultik.Vaultik{
|
||||||
|
Config: &config.Config{
|
||||||
|
AgeSecretKey: testAgeSecretKey,
|
||||||
|
Hostname: "recovery-host",
|
||||||
|
},
|
||||||
|
Storage: storer,
|
||||||
|
Fs: fs,
|
||||||
|
Repositories: database.NewRepositories(recoveryDB),
|
||||||
|
DB: recoveryDB,
|
||||||
|
Stdout: stdout,
|
||||||
|
Stderr: io.Discard,
|
||||||
|
UI: ui.NewWithColor(io.Discard, false),
|
||||||
|
}
|
||||||
|
recovery.SetContext(ctx)
|
||||||
|
|
||||||
|
return recovery, stdout
|
||||||
|
}
|
||||||
|
|
||||||
|
// assertRestoredTreeMatches byte-compares every restored file against its
|
||||||
|
// source content.
|
||||||
|
func assertRestoredTreeMatches(
|
||||||
|
t *testing.T, fs afero.Fs, restoreDir string, sourceFiles map[string][]byte,
|
||||||
|
) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
for origPath, expected := range sourceFiles {
|
||||||
|
restored := filepath.Join(restoreDir, origPath)
|
||||||
|
got, err := afero.ReadFile(fs, restored)
|
||||||
|
require.NoErrorf(t, err, "restored file missing: %s", restored)
|
||||||
|
require.Truef(t, bytes.Equal(got, expected),
|
||||||
|
"byte mismatch for %s", origPath)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -670,9 +670,11 @@ func (v *Vaultik) VerifySnapshotWithOptions(
|
|||||||
|
|
||||||
v.printVerifyHeader(snapshotID, opts)
|
v.printVerifyHeader(snapshotID, opts)
|
||||||
|
|
||||||
// Download and parse manifest. The caller supplies a human
|
// Resolve the identifier to the snapshot's remote key and download the
|
||||||
// snapshot ID; we hash it to address remote storage.
|
// manifest. A human ID is hashed; a remote key (or its abbreviation,
|
||||||
manifest, err := v.downloadManifestByKey(snapshot.RemoteSnapshotKey(snapshotID))
|
// as printed for a remote-only snapshot) is used as-is, so a host with
|
||||||
|
// no local index can verify a snapshot it can only see on the store.
|
||||||
|
manifest, err := v.resolveAndDownloadManifest(snapshotID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if opts.JSON {
|
if opts.JSON {
|
||||||
result.Status = verifyStatusFailed
|
result.Status = verifyStatusFailed
|
||||||
|
|||||||
@@ -0,0 +1,101 @@
|
|||||||
|
package vaultik
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"sneak.berlin/go/vaultik/internal/snapshot"
|
||||||
|
)
|
||||||
|
|
||||||
|
// remoteKeyHexLen is the length of a full remote snapshot key: a SHA256
|
||||||
|
// digest rendered as lowercase hex.
|
||||||
|
const remoteKeyHexLen = 64
|
||||||
|
|
||||||
|
// Sentinel errors for resolving a snapshot identifier against the store.
|
||||||
|
var (
|
||||||
|
errSnapshotKeyNotFound = errors.New(
|
||||||
|
"no snapshot on the destination store matches this identifier")
|
||||||
|
errSnapshotKeyAmbiguous = errors.New(
|
||||||
|
"identifier matches more than one snapshot on the destination store")
|
||||||
|
)
|
||||||
|
|
||||||
|
// resolveSnapshotRemoteKey turns a snapshot identifier supplied on the
|
||||||
|
// command line into the remote key that names the snapshot's metadata
|
||||||
|
// directory on the destination store. Every remote path a restore or
|
||||||
|
// verify reads is built from that key.
|
||||||
|
//
|
||||||
|
// Two forms are accepted, matching the two things a host can know:
|
||||||
|
//
|
||||||
|
// - A human snapshot ID (hostname_name_timestamp), which a host holding
|
||||||
|
// the local index has. It is hashed to its remote key; the store is
|
||||||
|
// not consulted.
|
||||||
|
// - A remote key, or the leading part of one, which is all a host with
|
||||||
|
// no local index can know — it is exactly what `snapshot list` prints
|
||||||
|
// for a remote-only snapshot (see formatRemoteOnlyID). It is resolved
|
||||||
|
// against the destination store's metadata listing; an identifier that
|
||||||
|
// matches no snapshot, or more than one, is an error.
|
||||||
|
//
|
||||||
|
// The two are told apart by shape: a remote key is lowercase hex, and a
|
||||||
|
// human snapshot ID never is (it carries a hostname, underscores, and an
|
||||||
|
// RFC3339 timestamp).
|
||||||
|
func (v *Vaultik) resolveSnapshotRemoteKey(identifier string) (string, error) {
|
||||||
|
if !isRemoteKeyOrPrefix(identifier) {
|
||||||
|
return snapshot.RemoteSnapshotKey(identifier), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
keys, err := v.listAllRemoteSnapshotKeys()
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf(
|
||||||
|
"listing destination store to resolve %q: %w", identifier, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var matches []string
|
||||||
|
|
||||||
|
for _, key := range keys {
|
||||||
|
if strings.HasPrefix(key, identifier) {
|
||||||
|
matches = append(matches, key)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
switch len(matches) {
|
||||||
|
case 1:
|
||||||
|
return matches[0], nil
|
||||||
|
case 0:
|
||||||
|
return "", fmt.Errorf("%w: %s", errSnapshotKeyNotFound, identifier)
|
||||||
|
default:
|
||||||
|
return "", fmt.Errorf("%w: %s (%d matches)",
|
||||||
|
errSnapshotKeyAmbiguous, identifier, len(matches))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// resolveAndDownloadManifest resolves a snapshot identifier to its remote
|
||||||
|
// key (see resolveSnapshotRemoteKey) and downloads that snapshot's
|
||||||
|
// manifest.
|
||||||
|
func (v *Vaultik) resolveAndDownloadManifest(
|
||||||
|
identifier string,
|
||||||
|
) (*snapshot.Manifest, error) {
|
||||||
|
remoteKey, err := v.resolveSnapshotRemoteKey(identifier)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
return v.downloadManifestByKey(remoteKey)
|
||||||
|
}
|
||||||
|
|
||||||
|
// isRemoteKeyOrPrefix reports whether s is a full remote key or the
|
||||||
|
// leading part of one: 1 to 64 lowercase hex characters. A human snapshot
|
||||||
|
// ID is never all hex, so this shape test is enough to tell the two apart.
|
||||||
|
func isRemoteKeyOrPrefix(s string) bool {
|
||||||
|
if s == "" || len(s) > remoteKeyHexLen {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, r := range s {
|
||||||
|
if (r < '0' || r > '9') && (r < 'a' || r > 'f') {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return true
|
||||||
|
}
|
||||||
@@ -138,8 +138,15 @@ func (v *Vaultik) RunDeepVerify(snapshotID string, opts *VerifyOptions) error {
|
|||||||
func (v *Vaultik) loadVerificationData(
|
func (v *Vaultik) loadVerificationData(
|
||||||
snapshotID string, opts *VerifyOptions, result *VerifyResult,
|
snapshotID string, opts *VerifyOptions, result *VerifyResult,
|
||||||
) (*snapshot.Manifest, *tempDB, []snapshot.BlobInfo, error) {
|
) (*snapshot.Manifest, *tempDB, []snapshot.BlobInfo, error) {
|
||||||
// All remote paths use the hashed key derived from the human ID.
|
// Resolve the identifier to the snapshot's remote key. A human ID is
|
||||||
remoteKey := snapshot.RemoteSnapshotKey(snapshotID)
|
// hashed; a remote key (or its abbreviation, as printed for a
|
||||||
|
// remote-only snapshot) is used as-is, so a host with no local index
|
||||||
|
// can verify a snapshot it can only see on the store.
|
||||||
|
remoteKey, err := v.resolveSnapshotRemoteKey(snapshotID)
|
||||||
|
if err != nil {
|
||||||
|
return nil, nil, nil, v.deepVerifyFailure(result, opts,
|
||||||
|
fmt.Sprintf("resolving snapshot identifier: %v", err), err)
|
||||||
|
}
|
||||||
|
|
||||||
// Download manifest. downloadManifestByKey is the single reader for
|
// Download manifest. downloadManifestByKey is the single reader for
|
||||||
// remote manifests; see its doc comment.
|
// remote manifests; see its doc comment.
|
||||||
@@ -186,7 +193,7 @@ func (v *Vaultik) loadVerificationData(
|
|||||||
fmt.Errorf("failed to decrypt database: %w", err))
|
fmt.Errorf("failed to decrypt database: %w", err))
|
||||||
}
|
}
|
||||||
|
|
||||||
dbBlobs, err := v.getBlobsFromDatabase(snapshotID, tdb.DB)
|
dbBlobs, err := v.getBlobsFromDatabase(tdb.DB)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
_ = tdb.Close()
|
_ = tdb.Close()
|
||||||
|
|
||||||
@@ -501,19 +508,21 @@ func (v *Vaultik) verifyBlobFinalIntegrity(
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// getBlobsFromDatabase gets all blobs for the snapshot from the database
|
// getBlobsFromDatabase gets all blobs for the snapshot from the database.
|
||||||
func (v *Vaultik) getBlobsFromDatabase(
|
//
|
||||||
snapshotID string, db *sql.DB,
|
// The exported per-snapshot database holds exactly one snapshot's data
|
||||||
) ([]snapshot.BlobInfo, error) {
|
// (see cleanSnapshotDB), so every row in snapshot_blobs belongs to it.
|
||||||
|
// We select them directly rather than filtering by the human snapshot ID,
|
||||||
|
// which a host restoring from the store alone does not have.
|
||||||
|
func (v *Vaultik) getBlobsFromDatabase(db *sql.DB) ([]snapshot.BlobInfo, error) {
|
||||||
query := `
|
query := `
|
||||||
SELECT b.blob_hash, b.compressed_size
|
SELECT b.blob_hash, b.compressed_size
|
||||||
FROM snapshot_blobs sb
|
FROM snapshot_blobs sb
|
||||||
JOIN blobs b ON sb.blob_hash = b.blob_hash
|
JOIN blobs b ON sb.blob_hash = b.blob_hash
|
||||||
WHERE sb.snapshot_id = ?
|
|
||||||
ORDER BY b.blob_hash
|
ORDER BY b.blob_hash
|
||||||
`
|
`
|
||||||
|
|
||||||
rows, err := db.QueryContext(v.ctx, query, snapshotID)
|
rows, err := db.QueryContext(v.ctx, query)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("failed to query snapshot blobs: %w", err)
|
return nil, fmt.Errorf("failed to query snapshot blobs: %w", err)
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user