sneak/vaultik
1
0
Fork 0
Code Issues 8 Pull Requests 3 Actions 52 Packages Projects Releases Wiki Activity

Compare commits

base: sneak/vaultik:3edc1889e3fbecc55494db2a791df5d3d5b26e90
Branches Tags
sneak/vaultik:main sneak/vaultik:fix-unreadable-symlink-skip-errors sneak/vaultik:issue-266-atomic-rclone-uploads sneak/vaultik:next
..
compare: sneak/vaultik:next
Branches Tags
sneak/vaultik:fix-unreadable-symlink-skip-errors sneak/vaultik:issue-266-atomic-rclone-uploads sneak/vaultik:next sneak/vaultik:main
20 Commits
3edc1889e3 .. next
Author SHA1 Message Date
clawbot b5389a62b5 Exit 130 and say so when a command is interrupted (closes #267)
check / check (push) Waiting to run
Details 
Ctrl-C or SIGTERM during snapshot create, restore or verify exited 0
with no error line (1, also silent, under snapshot verify --json), so
an unfinished --cron backup looked like a success. RunOperation now
records whether op returned while the Vaultik context was still live;
a run where it had not by the time RunWithApp returned is interrupted,
whatever op returned. It used to look for context.Canceled in op's
error, which verify --json does not return. Entry prints "interrupted
before the command finished" on stderr for it and returns 130, under
--cron and --json too.

SIGTERM also gives 130, as the issue asks, not 143.
The test does not cover an op still running when the 30s shutdown
timeout ends.

Model: opus-5-5
 2026-10-07 21:59:27 +02:00
clawbot d87202fb70 Correct doc and help sentences that are false about the code (closes #233)
check / check (push) Waiting to run
Details 
A blob is written whole to a temporary file and uploaded once finished,
not streamed to storage. The README, ARCHITECTURE.md and
config.example.yml now say a backup needs free temporary space for each
blob (twice that for an rclone destination that cannot stream uploads)
and for the metadata export's copies of the local index, in $TMPDIR, or
partly in /var/tmp when TMPDIR is unset. Also corrected: the snapshot ID
format, what restore reads and how incomplete snapshots are removed in
docs/DATAMODEL.md, what source_path holds, the index_path default, the
config search order in the snapshot create help, what snapshot remove
cleans up in the prune help, how the release installs Go, and the
script/release and script/fmt-check comments.

Model: opus-5-5
 2026-10-07 17:59:27 +02:00
clawbot e161343eac Speed up the vaultik and database package tests (closes #235)
check / check (push) Waiting to run
Details 
In the Dockerfile test phase, internal/vaultik spent 23.6s of 31.9s in
36 tests run one at a time. 24 of them were serial only because they
call log.Initialize; they now call it before t.Parallel(), so the
logger is replaced before any parallel test runs. The 12 still serial
change the umask, TMPDIR, os.Stderr or time.Local.

TestLargeDatasets took 8.4s of the 9.8s internal/database run by
committing each of its 1,500 inserts on its own; it now makes them in
one transaction. TestDedupOnlySnapshotRestores gives its second backup
its own snapshot name instead of sleeping 1.1s for a new snapshot ID.

Model: opus-5-5
 2026-10-07 17:12:07 +02:00
clawbot d53202eb86 Fix two misleading messages (closes #240)
check / check (push) Waiting to run
Details 
The warning for a config file that others can read always said the
file contained S3 credentials, so a file:// config with none got a
false claim. When s3.access_key_id or s3.secret_access_key is set it
now says the file may contain them, because Load sees the values only
after smartconfig has replaced any ${...} reference, so a set
credential need not be in the file. Otherwise it says the file is
readable by others.

snapshot purge wrapped the listing error, which already starts with
"listing remote snapshots:", in that prefix a second time.
syncWithRemote now returns it unwrapped, as CleanupLocalSnapshots
does.

Model: opus-5-5
 2026-10-07 15:29:07 +02:00
clawbot 8b22ae8d42 Pass s3.part_size to the multipart uploader (closes #232)
check / check (push) Waiting to run
Details 
s3.part_size was loaded and defaulted but never reached the S3 client,
whose uploader used a fixed 10MiB part. The client now takes the part
size from the config, for storage_url and for the s3.* fields, and
config load rejects a value below 5MiB or above 5GiB, an explicit 0
included. A blob too large for S3's limit of 10,000 parts at that size
is uploaded in larger parts, since the uploader cannot learn the size of
the reader it is given. The docs gave the default as 5MB, which the
config file reads as 5,000,000 bytes, below the minimum; they now say
5MiB.

Judgement call: the 5GiB maximum is enforced along with the 5MiB
minimum the issue names.

Model: opus-5-5
 2026-10-07 14:29:10 +02:00
clawbot 3fc8a8f2f4 Read the snapshot name using the stored hostname (closes #230)
check / check (push) Waiting to run
Details 
A snapshot ID is hostname_name_timestamp, and purge took the name to be
everything between the first and the last underscore. With a hostname
such as my_host the name home came out as host_home, so
`snapshot purge --keep-latest --snapshot home` found nothing to delete
and `snapshot create --prune` purged nothing without a message. The name
is now read by removing the hostname stored with the snapshot, in the
short form the ID uses, so both may contain underscores. This was chosen
over rejecting underscores in `hostname` when the config loads, which
would also stop restores on such a host.

The purge consistency test stored a hostname that did not match its
snapshot IDs; it now matches, as it always does in production.

Model: opus-5-5
 2026-10-07 12:12:08 +02:00
clawbot 7696f83258 Leave remote info orphan figures unknown when a manifest is unreadable (closes #228)
check / check (push) Waiting to run
Details 
When a manifest could not be read, remote info skipped it, counted that
snapshot's blobs as orphaned and advised running prune. The orphan
figures are now unknown in that case, with no prune advice; --json gives
them as null and lists the unreadable remote keys in
unreadable_manifests. Only a listed manifest.json.zst is read, so a
directory without one, as an interrupted backup leaves, keeps the
figures known.

Names under metadata/ were used unchecked and printed raw. A name that
is not a remote key is now skipped with a warning. A manifest under it
is not read either, so it also leaves the figures unknown; --json counts
such manifests in skipped_manifest_count.

Model: opus-5-5
 2026-10-07 10:59:26 +02:00
clawbot 5d1118d143 Quote a string setting that YAML would read as a number (closes #229)
check / check (push) Waiting to run
Details 
config set wrote every value as an unquoted YAML scalar, and config.Load
reads the file through untyped YAML, so an access key 00112233 loaded as
38043 and a hostname 007 as 7.

config set now looks the key up in config.Config by the fields' yaml
tags. A string setting is tagged !!str, which the encoder quotes
wherever YAML would read a number or a boolean. Other settings stay
unquoted, so compression_level 9 is still a number. A value that is not
valid UTF-8 stays untagged and is written as !!binary, which loads back
unchanged.

Judgement call: the type comes from reflection over config.Config.

Model: opus-5-5
 2026-10-07 09:29:19 +02:00
clawbot b57ce2277d Store and compare file mtimes to the nanosecond (closes #226)
check / check (push) Waiting to run
Details 
The files table held mtime in whole seconds and the scanner compared
whole seconds. A file rewritten with its size unchanged and a new mtime
in the same second as the indexed one was treated as unchanged, and
every later snapshot restored the old content. A new mtime_nsec column
now holds the nanoseconds within the second that mtime holds, and the
scanner compares the full mtime.

A local index created before this change lacks the column and is
rebuilt with `vaultik database delete` and a full backup. A snapshot
made before it cannot be restored by this version.

Model: opus-5-5
 2026-10-07 07:12:12 +02:00
clawbot 8496404d8b Make the process-wide lock atomic with flock (closes #227)
check / check (push) Waiting to run
Details 
Acquire read vaultik.pid, checked whether that PID was alive, then
wrote its own, so two writers started together could both pass the
check and both run. The lock is now an flock on vaultik.pid, held
while the file stays open; the kernel drops it when the process exits,
so the stale-PID check is gone.

Release empties the file instead of deleting it. Deleting it would let
a process that opened the old file a moment earlier lock it while
another creates and locks a new one.

The new concurrent test fails against the old code only when the race
is hit, not on every run; against the fix it cannot admit two callers.

Model: opus-5-5
 2026-10-07 06:12:10 +02:00
clawbot cdc60c4dfa Write only the document to stdout from snapshot remove --json (closes #251)
check / check (push) Waiting to run
Details 
When the destination store cannot be reached, `snapshot remove` still
removes the snapshot from the local index and warns. The warning went
through the UI, which writes to stdout, so under `--json` it landed
ahead of the document and `| jq` failed on a command that exited 0.
Under `--json` the UI warning is now skipped; the logger's warning on
stderr carries the follow-up, with the snapshot ID as a field.

The follow-up, in the warning, the README and the command's help, said
`vaultik prune` would finish the cleanup, but `prune` never removes
snapshot metadata. They now say to run `vaultik snapshot remove` for the
snapshot again once the destination store is reachable.

Model: opus-5-5
 2026-10-07 04:46:13 +02:00
clawbot 49eed7a3e5 Count each file, byte and upload once in backup statistics (closes #225)
check / check (push) Waiting to run
Details 
The scanner added a changed file's bytes again for each new chunk and
counted a file as unchanged for each chunk already stored. It now
counts files and bytes once, in the scan phase, and counts its own
uploads, so a --cron run, which has no progress reporter, records
them. The blob count no longer adds earlier paths' blobs again.

The snapshots row now stores the size of all files in total_size and
the referenced blobs' sizes in blob_size, blob_uncompressed_size and
compression_ratio, as docs/DATAMODEL.md says. Those sizes come from
one query, and a failed query fails the snapshot. DATAMODEL.md now
says chunk_count and blob_count count what the run added.

Removed UpdateSnapshotStats and GetCountBySnapshot, which nothing
calls any more.

Model: opus-5-5
 2026-10-07 02:59:27 +02:00
clawbot 85d4ef118d Keep command output to the README's stdout and stderr rules (closes #224)
check / check (push) Successful in 16m49s
Details 
The startup banner moves from stdout to stderr, so a `completion`
script, a `config get` value and the hidden `__complete` command print
only their own output. `--quiet`, `--cron` and `--json` still suppress
it.

A failing `remote info`, `prune` or `snapshot remove` under `--json`
now reports its error on stderr. Their reporters returned early under
`--json`, so the failure reached neither stream.

`snapshot verify --quiet` writes no report. A failure is still
returned and printed on stderr, with the same exit status.

Judgement call: the banner's stream, posted on the issue for the owner.

Model: opus-5-5
 2026-10-07 01:12:15 +02:00
clawbot b06f992152 Start the progress reporter once per snapshot, not once per path (closes #253)
check / check (push) Successful in 19m54s
Details 
A backup without --cron of a snapshot with two or more paths panicked
with "close of closed channel". Scan runs once per path, and it started
the progress reporter and deferred its Stop each time; Stop closes the
reporter's signal channel, so the second path's Stop panicked.
scanAllDirectories now starts the reporter before the first path and
stops it after the last, and Scan no longer starts or stops it. A new
test backs up a two-path snapshot with the reporter on and restores
both paths.

Model: opus-5-5
 2026-10-07 00:12:10 +02:00
clawbot 5d685f03ce List the files under a path by path, not by string prefix (closes #223)
check / check (push) Successful in 12m16s
Details 
FileRepository.ListByPrefix matched with SQL LIKE: a plain string
prefix that ignores ASCII case and treats _ and % as wildcards.
Restoring /home/u/doc also restored doc2, DOC and doc.txt.bak, and a
backup counted the files of a longer sibling path as deleted. It is
now ListUnderPath, which returns the file at the path and every file
whose path starts with the path plus a slash, compared exactly. A
trailing slash is ignored, so "/" still lists every file.

Three tests relied on string-prefix matching and now name a full path
or call ListAll.

ListIDsWithChunksNotInUploadedBlobs keeps its LIKE: it only adds file
IDs the scan never looks up.

Model: opus-5-5
 2026-10-06 22:12:16 +02:00
clawbot f59086c0e5 Return an error, not a panic, on a malformed snapshot database (closes #231)
check / check (push) Successful in 13m39s
Details 
Restore cut chunk hashes from the snapshot database to 16 characters
for its error messages, so a shorter hash panicked. Those messages now
use shortHash. Under --verify, a file_chunks row with no chunks row was
dereferenced, and the chunk size from the database was allocated in
one piece, so a negative or huge size panicked. A missing row is now an
error, a negative size is rejected, and each chunk is hashed by
streaming it from the restored file.

A restored file shorter than its chunks now fails verify as a short
read instead of an unexpected EOF.

Model: opus-5-5
 2026-10-06 21:16:28 +02:00
clawbot d276d891da Join the S3 prefix to every key with one slash (closes #222)
check / check (push) Successful in 10m59s
Details 
The S3 client built each key as prefix + key, and the URL parser keeps
the prefix as written, so s3://bucket/p stored p + "blobs/..." with no
slash while s3://bucket/p/ stored p/blobs/.... A recovery host that wrote
the URL the other way found no snapshots.

NewClient now strips trailing slashes from the prefix and adds one back
when anything is left, giving the README layout for both URL forms; an
empty prefix stays at the bucket root. The s3.prefix config setting
goes through the same client and gets the same join.

A new test writes through each URL shape against an in-process S3
server, checks the key in the bucket, and lists through both List and
ListStream, which every snapshot listing uses.

Model: opus-5-5
 2026-10-06 19:46:09 +02:00
clawbot 5aa5ba5544 Apply restored owners, modes and times in an order that keeps them (closes #219)
check / check (push) Successful in 11m53s
Details 
A directory got its stored mode and mtime before its contents were
written, so a read-only directory came back without its files and a
non-empty one carried the time of the restore. Directories are now
created owner-only (0700) and get their stored owner, mode and mtime
after the restore loop, each before its parent, skipping any whose
place a symlink has since taken. A file's mode is now applied after its
chown, which on Linux clears setuid and setgid. A symlink gets its
stored owner (as root) and mtime on the link itself, through
golang.org/x/sys/unix, now a direct dependency.

An interrupted restore leaves its directories at 0700.

Model: opus-5-5
 2026-10-06 18:29:09 +02:00
clawbot 315b6483b8 Mark github.com/spf13/pflag as a direct dependency in go.mod (closes #246)
check / check (push) Successful in 12m34s
Details 
internal/cli/snapshot_restore_test.go imports github.com/spf13/pflag
directly, but go.mod still marked it // indirect. script/precommit runs
go mod tidy and fails when that changes go.mod, so the pre-commit hook
stopped every commit. This is the go mod tidy output: pflag moves to the
direct require block, and go.sum does not change. script/cibuild does
not run the tidy, which is why the gate stayed green.

Model: opus-5-5
 2026-10-06 16:59:26 +02:00
clawbot 14fc4c9893 Load a config that has no age recipient (closes #221)
check / check (push) Successful in 13m47s
Details 
The README's steps for restoring on another machine failed at the first
command: `config init` wrote a placeholder recipient, and `config.Load`
rejects any recipient that does not parse. `config init` now writes an
empty `age_recipients` list, `config.Load` accepts an empty list, and
`snapshot create` refuses to start without a recipient. A malformed
recipient is still rejected at load.

The recovery-host test now builds its config with `config init` and
`config set` and reads it through `config.Load`, so it imports
`internal/cli`. On a fresh file, `config set age_recipients.0` writes the
list in flow style (`[age1...]`).

Model: opus-5-5
 2026-10-06 15:46:13 +02:00
69 changed files with 3682 additions and 734 deletions
Expand all files Collapse all files
ARCHITECTURE.md
+9 -5
View File
@@ -54,10 +54,12 @@ The database tracks five primary entities and their relationships:
#### File (`database.File`) #### File (`database.File`)
Represents a file, directory, or symlink in the backup system. Stores metadata needed for restoration: Represents a file, directory, or symlink in the backup system. Stores metadata needed for restoration:
- Path, source_path (for restore path stripping), mtime - Path, mtime
- Size, mode, ownership (uid, gid) - Size, mode, ownership (uid, gid)
- Symlink target (if applicable) - Symlink target (if applicable)
It also stores `source_path`, the source directory the scan found it under, made absolute and with symlinks resolved. Restore does not read it.
#### Chunk (`database.Chunk`) #### Chunk (`database.Chunk`)
A content-addressed unit of data. Files are split into variable-size chunks using the FastCDC algorithm: A content-addressed unit of data. Files are split into variable-size chunks using the FastCDC algorithm:
- `ChunkHash`: SHA256 hash of chunk content (primary key) - `ChunkHash`: SHA256 hash of chunk content (primary key)
@@ -82,9 +84,11 @@ The final storage unit uploaded to S3. Contains many compressed and encrypted ch
Blob creation process: Blob creation process:
1. Chunks are accumulated (up to MaxBlobSize, typically 10GB) 1. Chunks are accumulated (up to MaxBlobSize, typically 10GB)
2. As each chunk is added, its uncompressed bytes are fed to a running SHA-256 2. As each chunk is added, its uncompressed bytes are fed to a running SHA-256
3. Concurrently, the same bytes are compressed with zstd, then encrypted with age (recipients configured in config), and streamed to storage 3. Concurrently, the same bytes are compressed with zstd, then encrypted with age (recipients configured in config), and written to a temporary file
4. On finalize, the blob's name is the double SHA-256 of the uncompressed contents — `hex(SHA256(SHA256(...)))` — not a hash of the compressed, encrypted bytes 4. On finalize, the blob's name is the double SHA-256 of the uncompressed contents — `hex(SHA256(SHA256(...)))` — not a hash of the compressed, encrypted bytes
5. Uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}` 5. The finished file is uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}` and then deleted
A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp`.
#### BlobChunk (`database.BlobChunk`) #### BlobChunk (`database.BlobChunk`)
Maps chunks to their position within blobs: Maps chunks to their position within blobs:
@@ -335,10 +339,10 @@ CreateSnapshot(opts)
│ │ │ │
│ └─► Accumulate statistics │ └─► Accumulate statistics
│ │
├─► SnapshotManager.UpdateSnapshotStatsExtended()
│
├─► SnapshotManager.PopulateSnapshotBlobs() // record referenced blobs ├─► SnapshotManager.PopulateSnapshotBlobs() // record referenced blobs
│ │
├─► SnapshotManager.UpdateSnapshotStatsExtended()
│
├─► SnapshotManager.ExportSnapshotMetadata() ├─► SnapshotManager.ExportSnapshotMetadata()
│ │ │ │
│ ├─► Copy database to temp file │ ├─► Copy database to temp file
README.md
+40 -28
View File
@@ -200,8 +200,10 @@ local index or the destination store. `config`, `database delete`,
### stdout and stderr ### stdout and stderr
Log output — everything from `--verbose` and `--debug`, and every Log output — everything from `--verbose` and `--debug`, and every
warning and error the logger emits — goes to **stderr**. stdout carries warning and error the logger emits — goes to **stderr**, and so does the
the output you asked for: tables, and the documents produced by `--json`. startup banner. stdout carries the output you asked for: tables, the
documents produced by `--json`, `config get` values, and completion
scripts.
This means `vaultik snapshot list --verbose > out.txt` captures the This means `vaultik snapshot list --verbose > out.txt` captures the
listing and leaves the diagnostics on your terminal. To capture both, listing and leaves the diagnostics on your terminal. To capture both,
@@ -350,8 +352,9 @@ may hold snapshots this host doesn't know about), which is what
prune` invocation to run as a follow-up. Local row cleanup (files, prune` invocation to run as a follow-up. Local row cleanup (files,
chunks, blobs the snapshot was the last referrer for) runs chunks, blobs the snapshot was the last referrer for) runs
automatically. If the destination store is unreachable, the local-DB automatically. If the destination store is unreachable, the local-DB
removal still completes and a warning is emitted; rerun `vaultik prune` removal still completes and a warning is emitted; run `vaultik snapshot
once the store is reachable to finish remote cleanup. To wipe everything remove <snapshot-id>` again once the store is reachable to remove the
snapshot's metadata from it (`vaultik prune` does not). To wipe everything
on the destination in one go, use `vaultik remote nuke --force`. on the destination in one go, use `vaultik remote nuke --force`.
* `--local-only`: Skip remote cleanup; only touch the local index * `--local-only`: Skip remote cleanup; only touch the local index
* `--dry-run`: Show what would be deleted without deleting * `--dry-run`: Show what would be deleted without deleting
@@ -387,7 +390,13 @@ recipients, and local database statistics.
**`remote info`**: Show storage backend type and location plus detailed **`remote info`**: Show storage backend type and location plus detailed
remote storage inventory: per-snapshot metadata sizes, blob counts, and remote storage inventory: per-snapshot metadata sizes, blob counts, and
orphaned blob detection. orphaned blob detection. A name under `metadata/` that is not a remote
key is skipped with a warning and is not printed. If a listed
`manifest.json.zst` cannot be read, or sits under a skipped name, the
orphaned blob figures are reported as unknown; `--json` gives them as
`null`, lists the remote key of each unreadable manifest in
`unreadable_manifests` and counts the manifests under skipped names in
`skipped_manifest_count`.
* `--json`: Output as JSON * `--json`: Output as JSON
**`remote nuke`**: Delete every snapshot's metadata and every blob from the **`remote nuke`**: Delete every snapshot's metadata and every blob from the
@@ -530,14 +539,14 @@ complete annotated example also lives in
| Field | Default | Description | | Field | Default | Description |
|-------|---------|-------------| |-------|---------|-------------|
| `age_recipients` | (required) | Age public keys for encryption | | `age_recipients` | (required by `snapshot create`) | Age public keys for encryption. Other commands run without one, so a machine that only restores can leave it empty |
| `age_secret_key` | (unset) | Age private key for decryption (`snapshot restore`, `snapshot verify --deep`). Setting it in the config file places the private key on the backed-up host, defeating the public-key-only design (see "why" above). Prefer the `VAULTIK_AGE_SECRET_KEY` environment variable, supplied only on the machine you restore from. | | `age_secret_key` | (unset) | Age private key for decryption (`snapshot restore`, `snapshot verify --deep`). Setting it in the config file places the private key on the backed-up host, defeating the public-key-only design (see "why" above). Prefer the `VAULTIK_AGE_SECRET_KEY` environment variable, supplied only on the machine you restore from. |
| `snapshots` | (required) | Named snapshot definitions with paths and excludes | | `snapshots` | (required) | Named snapshot definitions with paths and excludes |
| `storage_url` | | Storage backend URL (`s3://`, `file://`, `rclone://`) | | `storage_url` | | Storage backend URL (`s3://`, `file://`, `rclone://`) |
| `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) | | `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) |
| `exclude` | | Global exclude patterns (applied to all snapshots) | | `exclude` | | Global exclude patterns (applied to all snapshots) |
| `chunk_size` | `10MB` | Average chunk size for content-defined chunking | | `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit | | `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` |
| `compression_level` | `3` | zstd compression level (1-19) | | `compression_level` | `3` | zstd compression level (1-19) |
| `hostname` | system hostname | Hostname used in snapshot IDs | | `hostname` | system hostname | Hostname used in snapshot IDs |
| `index_path` | platform data dir | Local SQLite index path | | `index_path` | platform data dir | Local SQLite index path |
@@ -645,8 +654,8 @@ Work planned after 1.0. Loosely ordered by priority.
## output style ## output style
Every command's user-facing output is governed by `internal/ui`, in one Every command's user-facing output is governed by `internal/ui`, in one
of two ways. Color is enabled when stdout is a TTY and the `NO_COLOR` of two ways. Color is enabled when the stream written to is a TTY and
environment variable is unset (https://no-color.org/). the `NO_COLOR` environment variable is unset (https://no-color.org/).
* **Status, progress, warnings, and errors** go through the `internal/ui` * **Status, progress, warnings, and errors** go through the `internal/ui`
message methods below: marker-prefixed, colored on a TTY, and — except message methods below: marker-prefixed, colored on a TTY, and — except
@@ -656,24 +665,26 @@ environment variable is unset (https://no-color.org/).
`config init`, `config set`, and `database delete`. `config init`, `config set`, and `database delete`.
* **The data a command exists to produce** is written plain, with no * **The data a command exists to produce** is written plain, with no
marker and no color, because a marker would corrupt a table or a marker and no color, because a marker would corrupt a table or a
parsed document. This covers the `version`, `info`, and `remote info` parsed document. This covers the `version`, `info`, `remote info` and
reports, the `snapshot list` table, `config get` values, and every `snapshot verify` reports, the `snapshot list` table, `config get`
`--json` document. `--quiet` silences the human reports and tables values, and every `--json` document. `--quiet` silences the human
(`version`, `info`, `remote info`, `snapshot list`) but never the reports and tables (`version`, `info`, `remote info`, `snapshot
machine-consumed `config get` value or the `--json` documents, which a verify`, `snapshot list`) but never the machine-consumed `config get`
script depends on. The `database delete` confirmation prompt is also value or the `--json` documents, which a script depends on. The
written this way and always shown: it is an interactive exchange the `database delete` confirmation prompt is also written this way and
operator must see. always shown: it is an interactive exchange the operator must see.
`internal/ui` writes to stdout; it is the output the user asked for. `internal/ui` writes to stdout; it is the output the user asked for. The
Structured log records are a different thing and go through exceptions are the startup banner and the error a failed command ends
`internal/log`, which writes to stderr (see "stdout and stderr" above). with, which go to stderr. Structured log records are a different thing
and go through `internal/log`, which writes to stderr (see "stdout and
stderr" above).
Message classes: Message classes:
| Class | Marker | Alignment | Use for | | Class | Marker | Alignment | Use for |
|-------|--------|-----------|---------| |-------|--------|-----------|---------|
| Banner | none | column 0 | The startup line printed once per invocation | | Banner | none | column 0 | The startup line printed once per invocation, on stderr |
| Begin | `》` (white) | column 0 | An operation is about to start (present-continuous verb) | | Begin | `》` (white) | column 0 | An operation is about to start (present-continuous verb) |
| Complete | `》` (green) | column 0 | An operation just finished (past-tense verb) | | Complete | `》` (green) | column 0 | An operation just finished (past-tense verb) |
| Info | `》` (white) | column 0 | Neutral status update | | Info | `》` (white) | column 0 | Neutral status update |
@@ -794,7 +805,8 @@ them. We provide:
in the `Dockerfile` together. in the `Dockerfile` together.
* `script/release` — cross-compile and publish the release artifacts * `script/release` — cross-compile and publish the release artifacts
with the pinned `goreleaser`. Refuses a `goreleaser` on `PATH` whose with the pinned `goreleaser`. Refuses a `goreleaser` on `PATH` whose
version is not the pinned one, on the same reasoning as `script/lint`. version is not the pinned one, because a different version would build
a different release from the same tag.
* `script/release-snapshot` — the same build with no publishing and no * `script/release-snapshot` — the same build with no publishing and no
tagging, into `./dist` tagging, into `./dist`
* `script/test` — run the test suite by building the `test` phase of * `script/test` — run the test suite by building the `test` phase of
@@ -904,14 +916,14 @@ It is passed to `goreleaser` as `GITEA_TOKEN`. The runner's automatic
token is deliberately not used: it is not guaranteed to carry release token is deliberately not used: it is not guaranteed to carry release
write access. write access.
The Go toolchain that compiles the released binaries comes from an The Go toolchain that compiles the released binaries is installed by
`actions/setup-go` step pinned by commit sha, reading its version from `script/install-go`, which downloads the version named by `go.mod`
`go.mod` (currently `1.26.1`, the same version the `Dockerfile` builder (currently `1.26.1`, the same version the `Dockerfile` builder stage
stage pins by digest). `goreleaser` shells out to `go` for every pins by digest) and refuses the archive unless its sha256 matches the
value committed in the script. `goreleaser` shells out to `go` for every
cross-compile, so without that step the release would either fail cross-compile, so without that step the release would either fail
outright or ship binaries built by whatever unpinned toolchain the outright or ship binaries built by whatever unpinned toolchain the
runner happened to carry — the one unpinned thing in an otherwise runner happened to carry.
hash-pinned release path.
To rehearse the whole build without publishing or tagging anything: To rehearse the whole build without publishing or tagging anything:
TODO.md
+199 -3
View File
@@ -22,6 +22,186 @@ the tag exists and is exercised; what is left is merging `next` to
# Completed Steps # Completed Steps
- 2026-10-07: Made an interrupted command exit 130 and say so
([issue #267](https://git.eeqj.de/sneak/vaultik/issues/267)). Ctrl-C
or SIGTERM during `snapshot create`, `snapshot restore` or `snapshot
verify` exited 0 with no error line (`snapshot verify --json` exited
1, also without one), so a `--cron` run that never finished looked
like a success. A command stopped by either signal now exits 130 and
prints `interrupted before the command finished` on stderr, under
`--cron` and `--json` too.
- 2026-10-07: Corrected documentation, help text and comments that were
false about the code
([issue #233](https://git.eeqj.de/sneak/vaultik/issues/233)). A blob
is not streamed to storage. The README, `ARCHITECTURE.md` and
`config.example.yml` now say a backup needs free temporary space,
because each blob is written whole to a temporary file before it is
uploaded (up to about `blob_size_limit`; an rclone destination that
cannot stream uploads needs about twice that) and the metadata export
writes copies of the local index. Temporary files go to `$TMPDIR`
(default `/tmp`); with `TMPDIR` unset, SQLite writes one of those
copies to `/var/tmp`. Also corrected: the snapshot ID format, what restore
reads and how incomplete snapshots are removed in `docs/DATAMODEL.md`,
what `source_path` holds, the `index_path` and config file defaults,
what `snapshot remove` cleans up, how the release gets its Go
toolchain, and the `script/release` and `script/fmt-check` comments.
- 2026-10-07: Cut the time the `internal/vaultik` and `internal/database`
tests take ([issue #235](https://git.eeqj.de/sneak/vaultik/issues/235)).
Most of the `internal/vaultik` time went to 24 tests that ran one at a
time only because they call `log.Initialize`; they now call it before
`t.Parallel()`, as the package's other tests do. `TestLargeDatasets`
committed each of its 1,500 inserts on its own and now makes them in
one transaction. `TestDedupOnlySnapshotRestores` gives its second
backup its own snapshot name instead of sleeping past the one-second
timestamp in the snapshot ID.
- 2026-10-07: Made two messages say only what is true
([issue #240](https://git.eeqj.de/sneak/vaultik/issues/240)). A config
file that others can read was warned about as containing S3
credentials even when it set none, as a `file://` config does. The
warning now says the file may contain S3 credentials only when
`s3.access_key_id` or `s3.secret_access_key` is set, since either may
come from a `${...}` reference rather than the file, and otherwise
says the file is readable by others. `snapshot purge` against a
destination store it could not list gave an error with
`listing remote snapshots:` in it twice; the prefix now appears once.
- 2026-10-07: Made `s3.part_size` set the multipart upload part size
([issue #232](https://git.eeqj.de/sneak/vaultik/issues/232)). It was
loaded and defaulted but never passed to the S3 client, whose uploader
used a fixed 10MiB part. It now reaches the uploader for `storage_url`
and for the `s3.*` fields, and a part size S3 refuses, below 5MiB or
above 5GiB, `0` included, fails at config load. A blob too large for
S3's limit of 10,000 parts at the configured size is uploaded in larger
parts. The docs gave the default as `5MB`, which the config file reads
as 5,000,000 bytes, below the minimum; they now say `5MiB`.
- 2026-10-07: Made per-name retention work when the hostname contains `_`
([issue #230](https://git.eeqj.de/sneak/vaultik/issues/230)). A
snapshot ID is `hostname_name_timestamp`, and the name was read as
everything between the first and the last `_`, so with
`hostname: my_host` the name `home` came out as `host_home`.
`snapshot purge --keep-latest --snapshot home` then printed "No
snapshots to delete", and `snapshot create --prune` purged nothing
without a message. The name is now read using the hostname the
`snapshots` table stores with each snapshot, cut at its first `.` as it
is in the ID.
- 2026-10-07: Made `remote info` stop reporting a snapshot's blobs as
orphaned when its manifest cannot be read, and stop printing raw
names from under `metadata/`
([issue #228](https://git.eeqj.de/sneak/vaultik/issues/228)). A
manifest it failed to read was skipped, so that snapshot's blobs were
counted as orphaned and the report advised running `vaultik prune`.
The orphan figures are now unknown in that case, with no prune
advice, and `--json` gives them as `null` with the unreadable remote
keys in `unreadable_manifests`. A name under `metadata/` that is not
64 lowercase hex characters is now skipped with a warning instead of
being printed, control characters included. A manifest under a
skipped name is then not read either, so it also leaves the orphan
figures unknown, and `--json` counts such manifests in
`skipped_manifest_count`. A directory with no manifest in it, as left
by an interrupted backup, leaves the figures known.
- 2026-10-07: Made `config set` keep a string that looks like a number
([issue #229](https://git.eeqj.de/sneak/vaultik/issues/229)). It wrote
every value unquoted, and `config.Load` reads the file through untyped
YAML, so an access key `00112233` loaded as `38043` and a hostname `007`
as `7`. A value for a string setting in `config.Config` is now tagged as
a YAML string, which the file quotes wherever YAML would read a number or
a boolean; other settings are still written unquoted.
- 2026-10-07: Made a backup notice a file rewritten with its size
unchanged and a new mtime in the same second as the one in the index
([issue #226](https://git.eeqj.de/sneak/vaultik/issues/226)). The
`files` table held mtime in whole seconds and the scanner compared
whole seconds, so every later snapshot kept the old content. A new
`mtime_nsec` column holds the nanoseconds within the second that
`mtime` holds, and the scanner compares the full mtime. A local index
created before the change lacks the column and is rebuilt with
`vaultik database delete` and a full backup.
- 2026-10-07: Made taking the process-wide lock atomic
([issue #227](https://git.eeqj.de/sneak/vaultik/issues/227)). The lock
read `vaultik.pid`, checked whether that PID was alive and then wrote
its own, so two writers started together could both pass the check and
both run. It is now an `flock` on `vaultik.pid`, held until the run
ends; the kernel drops it when the process exits, so a crash leaves no
lock behind. A clean exit now empties the file instead of deleting it,
because deleting it would let two later runs each lock a different
file.
- 2026-10-06: Made `snapshot remove --json` write only its document to
stdout when the destination store cannot be reached
([issue #251](https://git.eeqj.de/sneak/vaultik/issues/251)). Its
warning that the snapshot's metadata was left on the destination store
went to stdout ahead of the document, breaking `| jq` on a command that
exited 0. Under `--json` the warning now reaches stderr only, through
the logger. The warning, the README and the command's help said
`vaultik prune` would finish the cleanup, but `prune` never removes
snapshot metadata; they now say to run `vaultik snapshot remove` for the
snapshot again once the destination store is reachable.
- 2026-10-06: Made the backup summary and the `snapshots` row count each
file, byte and upload once
([issue #225](https://git.eeqj.de/sneak/vaultik/issues/225)). The
scanner added a file's bytes again for each new chunk and counted a
file as unchanged for each chunk already stored, so a first backup
reported twice its size and "backed up" could go negative. Upload
figures came from the progress reporter, which `--cron` turns off, and
`blob_count` counted earlier paths' blobs again for each later path.
The scanner now counts uploads itself; `blob_size`,
`blob_uncompressed_size` and `compression_ratio` describe the blobs
the snapshot references, and `docs/DATAMODEL.md` now says
`chunk_count` and `blob_count` count what the run added.
- 2026-10-06: Made command output follow the README's stdout and stderr
rules ([issue #224](https://git.eeqj.de/sneak/vaultik/issues/224)). The
startup banner went to stdout, so a `completion` script or a
`config get` value started with it; the banner now goes to stderr. A
failing `remote info`, `prune` or `snapshot remove` under `--json`
printed nothing on either stream, and now reports its error on stderr.
`snapshot verify --quiet` printed its whole report; it now prints
none, and a failure still reaches stderr with the same exit status.
- 2026-10-06: Made a backup without `--cron` of a snapshot with two or
more `paths` complete instead of panicking with `close of closed
channel` ([issue #253](https://git.eeqj.de/sneak/vaultik/issues/253)).
`Scan` runs once per path and started and stopped the progress
reporter each time, and a second stop panics. The reporter is now
started and stopped once per snapshot, around the scans of all its
paths.
- 2026-10-06: Made a restore path argument select only that path and
what is beneath it
([issue #223](https://git.eeqj.de/sneak/vaultik/issues/223)). The
lookup matched with SQL `LIKE`, so `/home/u/doc` also restored
`doc2`, `DOC` and `doc.txt.bak`, and a `_` or `%` in the path acted
as a wildcard. A backup used the same lookup to load the known files
of each configured path, so files of a longer sibling path were
counted as deleted. `FileRepository.ListUnderPath`, which replaces
`ListByPrefix`, returns the file at the path and every file whose
path starts with the path plus `/`, compared case-sensitively.
- 2026-10-06: Made restore return an error instead of panicking on a
malformed snapshot database
([issue #231](https://git.eeqj.de/sneak/vaultik/issues/231)). A chunk
hash shorter than 16 characters crashed the error message naming it,
and `--verify` dereferenced a missing `chunks` row and allocated
whatever chunk size the database gave. Those messages now go through
`shortHash`, a missing row is an error, and `--verify` rejects a
negative size and hashes each chunk as a stream.
- 2026-10-06: Made `s3://bucket/prefix` and `s3://bucket/prefix/` the same
destination ([issue #222](https://git.eeqj.de/sneak/vaultik/issues/222)).
The S3 client put the prefix directly in front of each key, so a prefix
without a trailing slash stored `prefixblobs/...`. A non-empty prefix is
now joined to every key with one `/`, giving the README's
`<bucket>/<prefix>/blobs/...` layout. The `s3.prefix` config setting goes
through the same client and gets the same join.
- 2026-10-06: Made restore apply owners, modes and times in an order - 2026-10-06: Made restore apply owners, modes and times in an order
that keeps them that keeps them
([issue #219](https://git.eeqj.de/sneak/vaultik/issues/219)). A ([issue #219](https://git.eeqj.de/sneak/vaultik/issues/219)). A
@@ -29,9 +209,25 @@ the tag exists and is exercised; what is left is merging `next` to
written, so a read-only directory came back without its files and a written, so a read-only directory came back without its files and a
non-empty one carried the time of the restore. Directories are now non-empty one carried the time of the restore. Directories are now
created owner-only and get their stored owner, mode and mtime once the created owner-only and get their stored owner, mode and mtime once the
restore loop is done, deepest first. A file's mode is applied after restore loop is done, each before its parent. A file's mode is applied
its chown, which on Linux clears setuid and setgid, and a symlink gets after its chown, which on Linux clears setuid and setgid, and a
its stored owner (as root) and mtime on the link itself. symlink gets its stored owner (as root) and mtime on the link itself.
- 2026-10-06: Made `go.mod` what `go mod tidy` writes, so the
pre-commit hook no longer stops every commit
([issue #246](https://git.eeqj.de/sneak/vaultik/issues/246)). A test
in `internal/cli` imports `github.com/spf13/pflag` directly, but
`go.mod` still marked it `// indirect`, and `script/precommit` fails
whenever the tidy changes `go.mod`. It is now in the direct `require`
block.
- 2026-10-06: Made the README's steps for restoring on another machine
work ([issue #221](https://git.eeqj.de/sneak/vaultik/issues/221)).
`config init` wrote a placeholder recipient that `config.Load`
rejects, so every command on the new machine failed before reaching
the store. The file now has an empty `age_recipients` list,
`config.Load` accepts an empty list, and `snapshot create` refuses to
run without a recipient.
- 2026-10-06: Made `snapshot restore --skip-errors` skip the files that - 2026-10-06: Made `snapshot restore --skip-errors` skip the files that
need a blob it cannot download need a blob it cannot download
config.example.yml
+17 -7
View File
@@ -3,7 +3,8 @@
# Copy this file and uncomment/modify the values you need # Copy this file and uncomment/modify the values you need
# Age recipient public keys for encryption # Age recipient public keys for encryption
# This is REQUIRED - backups are encrypted to these public keys # Backups are encrypted to these public keys. snapshot create needs at least
# one; listing, verifying and restoring do not
# Generate with: age-keygen | grep "public key" # Generate with: age-keygen | grep "public key"
age_recipients: age_recipients:
- age1cj2k2addawy294f6k2gr2mf9gps9r3syplryxca3nvxj3daqm96qfp84tz - age1cj2k2addawy294f6k2gr2mf9gps9r3syplryxca3nvxj3daqm96qfp84tz
@@ -286,15 +287,18 @@ storage_url: "rclone://myremote/path/to/backups"
# #use_ssl: true # #use_ssl: true
# #
# # Part size for multipart uploads # # Part size for multipart uploads
# # Minimum 5MB, affects memory usage during upload # # Minimum 5MiB, maximum 5GiB; affects memory usage during upload
# # Supports: 5MB, 10M, 100MiB, etc. # # A blob too large for 10,000 parts of this size gets larger parts
# # Default: 5MB # # Supports: 10MB, 16MiB, 100MiB, etc. (5MB is below the minimum)
# #part_size: 5MB # # Default: 5MiB
# #part_size: 5MiB
# Path to local SQLite index database # Path to local SQLite index database
# This database tracks file state for incremental backups # This database tracks file state for incremental backups
# Default: /var/lib/vaultik/index.sqlite # Default: the platform data directory, e.g.
#index_path: /var/lib/vaultik/index.sqlite # macOS: ~/Library/Application Support/vaultik/index.sqlite
# Linux: ~/.local/share/vaultik/index.sqlite
#index_path: /path/to/index.sqlite
# Average chunk size for content-defined chunking # Average chunk size for content-defined chunking
# Smaller chunks = better deduplication but more metadata # Smaller chunks = better deduplication but more metadata
@@ -309,6 +313,12 @@ storage_url: "rclone://myremote/path/to/backups"
# Chunking uses no secret (the FastCDC parameters are fixed and public). At a # Chunking uses no secret (the FastCDC parameters are fixed and public). At a
# large limit a blob holds hundreds of chunks, so individual chunk lengths are # large limit a blob holds hundreds of chunks, so individual chunk lengths are
# not visible in its size; lowering the limit toward chunk_size exposes them. # not visible in its size; lowering the limit toward chunk_size exposes them.
# A backup needs free temporary space, because each blob is written whole to
# a temporary file before it is uploaded (up to about blob_size_limit; an
# rclone destination that cannot stream uploads needs about twice that) and
# the metadata export writes copies of the local index. Temporary files go to
# $TMPDIR (default /tmp); with TMPDIR unset, SQLite writes one of those copies
# to /var/tmp.
# Supports: 1GB, 10G, 500MB, 1GiB, etc. # Supports: 1GB, 10G, 500MB, 1GiB, etc.
# Default: 10GB # Default: 10GB
#blob_size_limit: 10GB #blob_size_limit: 10GB
docs/DATAMODEL.md
+10 -10
View File
@@ -36,7 +36,8 @@ Stores metadata about files in the filesystem being backed up.
**Columns:** **Columns:**
- `id` (TEXT PRIMARY KEY) - UUID for the file record - `id` (TEXT PRIMARY KEY) - UUID for the file record
- `path` (TEXT NOT NULL UNIQUE) - Absolute file path - `path` (TEXT NOT NULL UNIQUE) - Absolute file path
- `mtime` (INTEGER NOT NULL) - Modification time as Unix timestamp - `mtime` (INTEGER NOT NULL) - Modification time, whole seconds since the Unix epoch
- `mtime_nsec` (INTEGER NOT NULL) - Nanoseconds within that second, 0 to 999999999
- `size` (INTEGER NOT NULL) - File size in bytes - `size` (INTEGER NOT NULL) - File size in bytes
- `mode` (INTEGER NOT NULL) - Unix file permissions and type - `mode` (INTEGER NOT NULL) - Unix file permissions and type
- `uid` (INTEGER NOT NULL) - User ID of file owner - `uid` (INTEGER NOT NULL) - User ID of file owner
@@ -110,17 +111,17 @@ Maps chunks to the blobs that contain them.
Tracks backup snapshots. Tracks backup snapshots.
**Columns:** **Columns:**
- `id` (TEXT PRIMARY KEY) - Snapshot ID (format: hostname-YYYYMMDD-HHMMSSZ) - `id` (TEXT PRIMARY KEY) - Snapshot ID (format: `hostname_name_timestamp`, e.g. `server1_home_2025-06-01T12:00:00Z`: the hostname up to its first `.`, the snapshot name, and an RFC 3339 UTC timestamp)
- `hostname` (TEXT) - Hostname where backup was created - `hostname` (TEXT) - Hostname where backup was created
- `vaultik_version` (TEXT) - Version of Vaultik used - `vaultik_version` (TEXT) - Version of Vaultik used
- `vaultik_git_revision` (TEXT) - Git revision of Vaultik used - `vaultik_git_revision` (TEXT) - Git revision of Vaultik used
- `started_at` (INTEGER) - Start timestamp - `started_at` (INTEGER) - Start timestamp
- `completed_at` (INTEGER) - Completion timestamp (NULL if in progress) - `completed_at` (INTEGER) - Completion timestamp (NULL if in progress)
- `file_count` (INTEGER) - Number of files in snapshot - `file_count` (INTEGER) - Number of files in snapshot
- `chunk_count` (INTEGER) - Number of unique chunks - `chunk_count` (INTEGER) - Number of chunks this snapshot stored that were not stored before
- `blob_count` (INTEGER) - Number of blobs referenced - `blob_count` (INTEGER) - Number of blobs this snapshot created
- `total_size` (INTEGER) - Total size of all files - `total_size` (INTEGER) - Total size of all files
- `blob_size` (INTEGER) - Total size of all blobs (compressed) - `blob_size` (INTEGER) - Total compressed size of all referenced blobs
- `blob_uncompressed_size` (INTEGER) - Total uncompressed size of all referenced blobs - `blob_uncompressed_size` (INTEGER) - Total uncompressed size of all referenced blobs
- `compression_ratio` (REAL) - Compression ratio achieved - `compression_ratio` (REAL) - Compression ratio achieved
- `compression_level` (INTEGER) - Compression level used for this snapshot - `compression_level` (INTEGER) - Compression level used for this snapshot
@@ -217,8 +218,8 @@ The `{remote-key}` directory name is a one-way hash of the human snapshot ID, so
### 4. Restore Process ### 4. Restore Process
The restore process doesn't use the local database. Instead: The restore process doesn't use the local database. Instead:
1. Downloads snapshot metadata from S3 1. Downloads and decrypts the snapshot's metadata database (`db.zst.age`) from S3
2. Downloads required blobs based on manifest 2. Downloads the blobs holding the chunks of the files being restored, found through that database's `blob_chunks` table; the manifest is not read
3. Reconstructs files from decrypted and decompressed chunks 3. Reconstructs files from decrypted and decompressed chunks
### 5. Pruning ### 5. Pruning
@@ -231,9 +232,8 @@ The restore process doesn't use the local database. Instead:
Before each backup: Before each backup:
1. Query incomplete snapshots (where `completed_at IS NULL`) 1. Query incomplete snapshots (where `completed_at IS NULL`)
2. Check if metadata exists in S3 2. Delete each one and all its associations, without checking S3 for its metadata
3. If no metadata, delete snapshot and all associations 3. Clean up orphaned files, chunks, and blobs
4. Clean up orphaned files, chunks, and blobs
## Repository Pattern ## Repository Pattern
go.mod
+1 -1
View File
@@ -20,6 +20,7 @@ require (
github.com/rclone/rclone v1.72.1 github.com/rclone/rclone v1.72.1
github.com/spf13/afero v1.15.0 github.com/spf13/afero v1.15.0
github.com/spf13/cobra v1.10.1 github.com/spf13/cobra v1.10.1
github.com/spf13/pflag v1.0.10
github.com/stretchr/testify v1.11.1 github.com/stretchr/testify v1.11.1
go.uber.org/fx v1.24.0 go.uber.org/fx v1.24.0
golang.org/x/sync v0.18.0 golang.org/x/sync v0.18.0
@@ -226,7 +227,6 @@ require (
github.com/smarty/assertions v1.16.0 // indirect github.com/smarty/assertions v1.16.0 // indirect
github.com/sony/gobreaker v1.0.0 // indirect github.com/sony/gobreaker v1.0.0 // indirect
github.com/spacemonkeygo/monkit/v3 v3.0.25-0.20251022131615-eb24eb109368 // indirect github.com/spacemonkeygo/monkit/v3 v3.0.25-0.20251022131615-eb24eb109368 // indirect
github.com/spf13/pflag v1.0.10 // indirect
github.com/t3rm1n4l/go-mega v0.0.0-20251031123324-a804aaa87491 // indirect github.com/t3rm1n4l/go-mega v0.0.0-20251031123324-a804aaa87491 // indirect
github.com/tidwall/gjson v1.18.0 // indirect github.com/tidwall/gjson v1.18.0 // indirect
github.com/tidwall/match v1.1.1 // indirect github.com/tidwall/match v1.1.1 // indirect
internal/cli/app.go
+48 -28
View File
@@ -200,12 +200,16 @@ func RunApp(ctx context.Context, app *fx.App) error {
} }
// errReported marks a failure the operation has already shown the user // errReported marks a failure the operation has already shown the user
// (and deliberately withheld under --json). Entry turns it into a // (or, under `snapshot verify --json`, put in its document). Entry
// non-zero exit status without printing anything further, so the error // turns it into a non-zero exit status without printing anything
// line is not doubled. It flows up from RunOperation through cobra to // further, so the error line is not doubled. It flows up from
// Entry. // RunOperation through cobra to Entry.
var errReported = errors.New("operation failed") var errReported = errors.New("operation failed")
// errInterrupted marks an operation that SIGINT or SIGTERM stopped
// before it finished. Entry shows it and returns exitCodeInterrupted.
var errInterrupted = errors.New("interrupted before the command finished")
// RunOperation runs op against the Vaultik instance inside the fx app // RunOperation runs op against the Vaultik instance inside the fx app
// and turns a failure into a returned error rather than an os.Exit from // and turns a failure into a returned error rather than an os.Exit from
// within the goroutine. An os.Exit there skipped main's deferred // within the goroutine. An os.Exit there skipped main's deferred
@@ -220,17 +224,21 @@ var errReported = errors.New("operation failed")
// interrupt OnStop cancels op and waits for the goroutine to return, so // interrupt OnStop cancels op and waits for the goroutine to return, so
// op's cleanup (removing decrypted scratch files) runs before the // op's cleanup (removing decrypted scratch files) runs before the
// process exits; the wait is bounded by shutdownTimeout. report is // process exits; the wait is bounded by shutdownTimeout. report is
// called with a non-canceled failure so the caller can log it (and // called with a failure so the caller can show it to the user before
// suppress it under --json) before it becomes errReported. A context // it becomes errReported.
// cancellation is the interrupt path, not a failure: it is neither //
// reported nor counted as one. // The run counts as interrupted unless op returned, without an
// interrupt having cancelled it, before RunWithApp returned. An
// interrupted op is not reported, whatever it returned; RunOperation
// returns errInterrupted instead.
func RunOperation( func RunOperation(
ctx context.Context, opts AppOptions, ctx context.Context, opts AppOptions,
op func(v *vaultik.Vaultik) error, report func(err error), op func(v *vaultik.Vaultik) error, report func(err error),
) error { ) error {
var ( var (
mu sync.Mutex mu sync.Mutex
failed bool finished bool // op returned before any interrupt cancelled it
failed bool // op finished with an error
) )
opts.Invokes = append(opts.Invokes, opts.Invokes = append(opts.Invokes,
@@ -241,11 +249,21 @@ func RunOperation(
OnStart: func(_ context.Context) error { OnStart: func(_ context.Context) error {
stop = v.StartOperation(func() { stop = v.StartOperation(func() {
err := op(v) err := op(v)
if err != nil && !errors.Is(err, context.Canceled) {
report(err) // Only stop, called from OnStop below, cancels the
// Vaultik context, so a live context means no
// interrupt cancelled op. Check the context, not
// err: an interrupted op need not return
// context.Canceled (`snapshot verify --json`
// returns a verification failure).
if v.Context().Err() == nil {
if err != nil {
report(err)
}
mu.Lock() mu.Lock()
failed = true finished = true
failed = err != nil
mu.Unlock() mu.Unlock()
} }
@@ -278,28 +296,34 @@ func RunOperation(
return err return err
} }
// The goroutine sets failed before triggering the shutdown that lets // RunWithApp returns only after the app was asked to stop, either by
// RunWithApp return, so the write is in place by the time we read it. // an interrupt or by the goroutine's Shutdown call. When op finished
// without being cancelled, the goroutine set finished before that
// call. So if finished is unset here, an interrupt stopped the app,
// and op either returned after it was cancelled or is still running
// because the shutdown timed out.
mu.Lock() mu.Lock()
defer mu.Unlock() defer mu.Unlock()
if failed { switch {
case !finished:
return errInterrupted
case failed:
return errReported return errReported
default:
return nil
} }
return nil
} }
// runVaultikApp runs the standard single-operation command lifecycle // runVaultikApp runs the standard single-operation command lifecycle
// shared by the snapshot list/purge/remove and remote nuke subcommands: // shared by the snapshot list/purge/remove and remote nuke subcommands:
// resolve the config, then run op against the Vaultik instance through // resolve the config, then run op against the Vaultik instance through
// RunOperation, reporting a failure prefixed with failMsg (suppressed // RunOperation, reporting a failure prefixed with failMsg on stderr. mode
// while suppressErrors is true, e.g. under --json). mode says whether the // says whether the command takes the PID lock. jsonOutput marks a command
// command takes the PID lock. jsonOutput marks a command whose stdout is a // whose stdout is a JSON document: it quiets the UI but, unlike Quiet,
// JSON document: it quiets the UI but, unlike Quiet, leaves the stderr log // leaves the stderr log level alone.
// level alone.
func runVaultikApp( func runVaultikApp(
cmd *cobra.Command, mode lockMode, jsonOutput, suppressErrors bool, cmd *cobra.Command, mode lockMode, jsonOutput bool,
failMsg string, op func(v *vaultik.Vaultik) error, failMsg string, op func(v *vaultik.Vaultik) error,
) error { ) error {
configPath, err := ResolveConfigPath() configPath, err := ResolveConfigPath()
@@ -319,10 +343,6 @@ func runVaultikApp(
}, },
Mode: mode, Mode: mode,
}, op, func(err error) { }, op, func(err error) {
if suppressErrors {
return
}
log.Error(failMsg, "error", err) log.Error(failMsg, "error", err)
ReportErrorf("%s: %v", failMsg, err) ReportErrorf("%s: %v", failMsg, err)
}) })
internal/cli/config.go
+63 -5
View File
@@ -7,11 +7,14 @@ import (
"os" "os"
"os/exec" "os/exec"
"path/filepath" "path/filepath"
"reflect"
"strconv" "strconv"
"strings" "strings"
"unicode/utf8"
"github.com/spf13/cobra" "github.com/spf13/cobra"
"gopkg.in/yaml.v3" "gopkg.in/yaml.v3"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/ui" "sneak.berlin/go/vaultik/internal/ui"
) )
@@ -31,6 +34,9 @@ const configDirMode = 0o755
// yaml.Marshal's 4-space default. // yaml.Marshal's 4-space default.
const configYAMLIndent = 2 const configYAMLIndent = 2
// yamlStringTag is YAML's tag for a string scalar.
const yamlStringTag = "!!str"
var ( var (
errConfigExists = errors.New("config file already exists") errConfigExists = errors.New("config file already exists")
errEmptyConfig = errors.New("empty config file") errEmptyConfig = errors.New("empty config file")
@@ -45,16 +51,19 @@ const defaultConfigTemplate = `# vaultik configuration
# ─── REQUIRED ──────────────────────────────────────────────────────────────── # ─── REQUIRED ────────────────────────────────────────────────────────────────
# Age recipient public keys for encryption. # Age recipient public keys for encryption. snapshot create needs at least
# one; listing, verifying and restoring do not, so a machine that only
# restores can leave this empty.
# Backups are encrypted to ALL listed recipients; any one of the corresponding # Backups are encrypted to ALL listed recipients; any one of the corresponding
# private keys can decrypt. Adding a recipient later does not re-encrypt data # private keys can decrypt. Adding a recipient later does not re-encrypt data
# already stored: deduplicated chunks and existing blobs stay encrypted to the # already stored: deduplicated chunks and existing blobs stay encrypted to the
# earlier recipients, so a newly added key cannot restore them on its own (see # earlier recipients, so a newly added key cannot restore them on its own (see
# docs/REPOSTRUCTURE.md, Accepted Risks). Generate a keypair with: # docs/REPOSTRUCTURE.md, Accepted Risks). Generate a keypair and add its
# public key with:
# age-keygen -o vaultik_backup_private_key.txt # age-keygen -o vaultik_backup_private_key.txt
# grep 'public key' vaultik_backup_private_key.txt # grep 'public key' vaultik_backup_private_key.txt
age_recipients: # vaultik config set age_recipients.0 age1...
- age1REPLACE_WITH_YOUR_PUBLIC_KEY age_recipients: []
# Named snapshots. Each snapshot backs up one or more paths and can have its # Named snapshots. Each snapshot backs up one or more paths and can have its
# own exclude patterns in addition to the global excludes below. # own exclude patterns in addition to the global excludes below.
@@ -196,7 +205,7 @@ storage_url: ""
# access_key_id: YOUR_ACCESS_KEY # access_key_id: YOUR_ACCESS_KEY
# secret_access_key: YOUR_SECRET_KEY # secret_access_key: YOUR_SECRET_KEY
# # region: us-east-1 # Default: us-east-1 # # region: us-east-1 # Default: us-east-1
# # part_size: 5MB # Multipart upload part size. Default: 5MB # # part_size: 5MiB # Upload part size, 5MiB to 5GiB. Default: 5MiB
# # For the s3:// form, disable TLS with ?ssl=false in the URL, not use_ssl. # # For the s3:// form, disable TLS with ?ssl=false in the URL, not use_ssl.
# ─── OPTIONAL ──────────────────────────────────────────────────────────────── # ─── OPTIONAL ────────────────────────────────────────────────────────────────
@@ -580,9 +589,58 @@ func yamlPathSet(root *yaml.Node, keys []string, value string) error {
} }
} }
// config.Load reads the file through untyped YAML, which turns an
// unquoted 00112233 into the number 38043 and 1e5 into 100000. Tagging
// a string setting as a string makes the encoder quote such a value.
// Other settings stay unquoted, so compression_level 9 is a number.
// The encoder refuses to write a value that is not valid UTF-8 as a
// string. Left untagged, such a value is written as base64 !!binary and
// loads back unchanged.
if configKeyIsString(keys) && utf8.ValidString(value) {
node.Tag = yamlStringTag
}
return nil return nil
} }
// configKeyIsString reports whether the dotted key names a string in
// config.Config, following the fields' yaml tags, as s3.access_key_id and
// snapshots.home.exclude.0 do.
func configKeyIsString(keys []string) bool {
typ := reflect.TypeFor[config.Config]()
for _, key := range keys {
switch {
case typ.Kind() == reflect.Map || typ.Kind() == reflect.Slice:
// The key is a snapshot name or a list index.
typ = typ.Elem()
case typ.Kind() == reflect.Struct:
field, ok := yamlField(typ, key)
if !ok {
return false
}
typ = field.Type
default:
return false
}
}
return typ.Kind() == reflect.String
}
// yamlField returns the field of struct type typ whose yaml tag names key.
func yamlField(typ reflect.Type, key string) (reflect.StructField, bool) {
for field := range typ.Fields() {
name, _, _ := strings.Cut(field.Tag.Get("yaml"), ",")
if name == key {
return field, true
}
}
return reflect.StructField{}, false
}
// yamlSetInMapping resolves (creating if needed) the value node for key // yamlSetInMapping resolves (creating if needed) the value node for key
// within a mapping node, setting it to value when it is the final path // within a mapping node, setting it to value when it is the final path
// element, and returns the node to descend into. // element, and returns the node to descend into.
internal/cli/config_test.go
+146 -2
View File
@@ -4,6 +4,7 @@ import (
"bytes" "bytes"
"os" "os"
"path/filepath" "path/filepath"
"strconv"
"strings" "strings"
"testing" "testing"
@@ -24,8 +25,10 @@ func TestDefaultConfigTemplateParses(t *testing.T) {
t.Fatalf("default config template is not valid YAML: %v", err) t.Fatalf("default config template is not valid YAML: %v", err)
} }
if len(cfg.AgeRecipients) != 1 { // A placeholder recipient would fail config.Load, so the template
t.Errorf("expected 1 placeholder age recipient, got %d", len(cfg.AgeRecipients)) // leaves the list empty.
if len(cfg.AgeRecipients) != 0 {
t.Errorf("expected no age recipients, got %d", len(cfg.AgeRecipients))
} }
home, ok := cfg.Snapshots["home"] home, ok := cfg.Snapshots["home"]
@@ -55,6 +58,147 @@ func TestDefaultConfigTemplateParses(t *testing.T) {
} }
} }
// TestConfigSetRecipientOnFreshConfig follows the README quickstart: on the
// file `config init` writes, `config set age_recipients.0` and
// `config set storage_url` give a config that loads with that recipient.
func TestConfigSetRecipientOnFreshConfig(t *testing.T) {
t.Parallel()
const recipient = "age1278m9q7dp3chsh2dcy82qk27v047zywyvtxwnj4cvt0z65jw6a7q5dqhfj"
path := filepath.Join(t.TempDir(), "config.yml")
err := os.WriteFile(path, []byte(defaultConfigTemplate), configFileMode)
if err != nil {
t.Fatalf("write config: %v", err)
}
out := ui.NewWithColor(&bytes.Buffer{}, false)
err = writeConfigSet(out, path, "age_recipients.0", recipient)
if err != nil {
t.Fatalf("config set age_recipients.0: %v", err)
}
err = writeConfigSet(out, path, "storage_url", "file:///mnt/backups")
if err != nil {
t.Fatalf("config set storage_url: %v", err)
}
cfg, err := config.Load(path)
if err != nil {
t.Fatalf("config.Load: %v", err)
}
if len(cfg.AgeRecipients) != 1 || cfg.AgeRecipients[0] != recipient {
t.Errorf("age_recipients = %v, want [%s]", cfg.AgeRecipients, recipient)
}
}
// TestConfigSetStringLooksLikeNumber sets string settings to values that
// YAML reads as numbers or booleans when they are unquoted, and checks that
// config.Load returns each one unchanged.
func TestConfigSetStringLooksLikeNumber(t *testing.T) {
t.Parallel()
tests := []struct {
key string
value string
field func(cfg *config.Config) string
}{
{"s3.access_key_id", "00112233",
func(cfg *config.Config) string { return cfg.S3.AccessKeyID }},
{"s3.secret_access_key", "12345678901234567890123456789012",
func(cfg *config.Config) string { return cfg.S3.SecretAccessKey }},
{"hostname", "007",
func(cfg *config.Config) string { return cfg.Hostname }},
{"s3.prefix", "1e5",
func(cfg *config.Config) string { return cfg.S3.Prefix }},
{"s3.bucket", "true",
func(cfg *config.Config) string { return cfg.S3.Bucket }},
{"s3.region", "FALSE",
func(cfg *config.Config) string { return cfg.S3.Region }},
{"snapshots.home.exclude.0", "1.10",
func(cfg *config.Config) string { return cfg.Snapshots["home"].Exclude[0] }},
}
for _, tt := range tests {
t.Run(tt.key+"="+tt.value, func(t *testing.T) {
t.Parallel()
cfg := loadAfterConfigSet(t, tt.key, tt.value)
got := tt.field(cfg)
if got != tt.value {
t.Errorf("%s = %q after config set %q", tt.key, got, tt.value)
}
})
}
}
// TestConfigSetNonUTF8Path checks that config set still accepts a value that
// is not valid UTF-8, such as a path with a Latin-1 file name, and that
// config.Load returns it unchanged.
func TestConfigSetNonUTF8Path(t *testing.T) {
t.Parallel()
const dir = "/srv/caf\xe9"
cfg := loadAfterConfigSet(t, "snapshots.home.paths.0", dir)
got := cfg.Snapshots["home"].Paths[0]
if got != dir {
t.Errorf("snapshots.home.paths.0 = %q, want %q", got, dir)
}
}
// TestConfigSetNumberStaysNumber checks that a number set for an integer
// setting is still read as a number, not as a quoted string.
func TestConfigSetNumberStaysNumber(t *testing.T) {
t.Parallel()
const level = 9
cfg := loadAfterConfigSet(t, "compression_level", strconv.Itoa(level))
if cfg.CompressionLevel != level {
t.Errorf("compression_level = %d, want %d", cfg.CompressionLevel, level)
}
}
// loadAfterConfigSet writes the file `config init` writes, sets storage_url
// to a local directory so that the file passes validation, applies
// `config set key value` and returns what config.Load reads back.
func loadAfterConfigSet(t *testing.T, key, value string) *config.Config {
t.Helper()
path := filepath.Join(t.TempDir(), "config.yml")
err := os.WriteFile(path, []byte(defaultConfigTemplate), configFileMode)
if err != nil {
t.Fatalf("write config: %v", err)
}
out := ui.NewWithColor(&bytes.Buffer{}, false)
err = writeConfigSet(out, path, "storage_url", "file:///mnt/backups")
if err != nil {
t.Fatalf("config set storage_url: %v", err)
}
err = writeConfigSet(out, path, key, value)
if err != nil {
t.Fatalf("config set %s: %v", key, err)
}
cfg, err := config.Load(path)
if err != nil {
t.Fatalf("config.Load: %v", err)
}
return cfg
}
const testYAML = `# top comment const testYAML = `# top comment
compression_level: 3 compression_level: 3
age_recipients: age_recipients:
internal/cli/entry.go
+25 -13
View File
@@ -15,17 +15,25 @@ import (
// the startup banner. // the startup banner.
const shortCommitLen = 12 const shortCommitLen = 12
// exitCodeInterrupted is the exit status of a command that SIGINT or
// SIGTERM stopped. It is 128 plus SIGINT's number, 2, which is what a
// shell reports for a command stopped by Ctrl-C.
const exitCodeInterrupted = 130
// Entry is the main entry point for the CLI application. // Entry is the main entry point for the CLI application.
// It prints the startup banner to stdout (unless a banner-suppressing // It prints the startup banner to stderr (unless a banner-suppressing
// flag is present in os.Args — see bannerSuppressedInArgs), executes the // flag is present in os.Args — see bannerSuppressedInArgs), executes the
// root cobra command, and routes any returned error through the // root cobra command, and routes any returned error through the
// ui.Writer so the user sees a properly formatted "🛑 ERROR:" line. // ui.Writer so the user sees a properly formatted "🛑 ERROR:" line.
// The banner goes to stderr because stdout carries only the output the
// user asked for, such as a completion script or a `config get` value.
// //
// It returns the process exit code (0 on success, 1 on error) rather // It returns the process exit code (0 on success, 130 when interrupted,
// than calling os.Exit, so that main's deferred profile writers run // 1 on any other error) rather than calling os.Exit, so that main's
// before the process ends. See run in cmd/vaultik/main.go. // deferred profile writers run before the process ends. See run in
// cmd/vaultik/main.go.
func Entry() int { func Entry() int {
emitStartupBanner(os.Args[1:], os.Stdout) emitStartupBanner(os.Args[1:], os.Stderr)
rootCmd := NewRootCommand() rootCmd := NewRootCommand()
rootCmd.SilenceErrors = true rootCmd.SilenceErrors = true
@@ -33,14 +41,20 @@ func Entry() int {
err := rootCmd.Execute() err := rootCmd.Execute()
if err != nil { if err != nil {
// An operation that ran inside the fx app has already reported // An operation that ran inside the fx app has already reported
// its own failure (and suppressed it under --json); errReported // its own failure (`snapshot verify --json` puts it in the
// says so. Printing it again here would double the error line. // document instead); errReported says so. Printing it again
// here would double the error line.
// Every other error — bad arguments, a config that would not // Every other error — bad arguments, a config that would not
// load — reaches Entry unreported, so it is shown here. // load, an interrupt — reaches Entry unreported, so it is shown
// here.
if !errors.Is(err, errReported) { if !errors.Is(err, errReported) {
ReportErrorf("%s", err.Error()) ReportErrorf("%s", err.Error())
} }
if errors.Is(err, errInterrupted) {
return exitCodeInterrupted
}
return 1 return 1
} }
@@ -49,9 +63,8 @@ func Entry() int {
// emitStartupBanner writes the startup banner to w unless args (the // emitStartupBanner writes the startup banner to w unless args (the
// argument vector with the program name already stripped) contains a // argument vector with the program name already stripped) contains a
// flag that suppresses it. Split out of Entry so that the decision — the // flag that suppresses it. Split out of Entry so that the decision is
// only thing standing between a --json invocation and a parseable // reachable from a test without running the whole CLI.
// stdout — is reachable from a test without running the whole CLI.
func emitStartupBanner(args []string, w io.Writer) { func emitStartupBanner(args []string, w io.Writer) {
if bannerSuppressedInArgs(args) { if bannerSuppressedInArgs(args) {
return return
@@ -86,8 +99,7 @@ func ReportErrorf(format string, args ...any) {
// --json is a subcommand flag rather than a persistent one, but so is // --json is a subcommand flag rather than a persistent one, but so is
// --cron (it exists only on `snapshot create`), so this adds no new // --cron (it exists only on `snapshot create`), so this adds no new
// class of imprecision. The only cost of a false positive is a missing // class of imprecision. The only cost of a false positive is a missing
// decorative banner; the cost of a false negative is a corrupt document // decorative banner.
// on stdout, so the scan errs deliberately in that direction.
func bannerSuppressedInArgs(args []string) bool { func bannerSuppressedInArgs(args []string) bool {
for _, a := range args { for _, a := range args {
if a == "--" { if a == "--" {
internal/cli/entry_banner_test.go
+16 -43
View File
@@ -36,23 +36,14 @@ const (
// strips it before scanning, so it has to be present. // strips it before scanning, so it has to be present.
programName = "vaultik" programName = "vaultik"
// someSnapshotID is any snapshot identifier: these tests never run // someSnapshotID only fills the positional argument; no test needs
// the command, so it only has to occupy the positional argument. // the snapshot to exist.
someSnapshotID = "host_2026-01-01T00:00:00Z" someSnapshotID = "host_2026-01-01T00:00:00Z"
) )
// placeholderJSONDocument stands in for whatever document a --json
// command writes to stdout. `snapshot list --json` with no snapshots
// prints exactly this; the other --json commands print an object rather
// than an array, but this test is not about their shape. It is about
// what is on stdout *before* them, which is the same for all of them
// because Entry prints the banner before cobra has parsed anything and
// therefore before it can know which command is running.
const placeholderJSONDocument = "[]\n"
// jsonArgumentVectors are the argument vectors of every --json // jsonArgumentVectors are the argument vectors of every --json
// invocation the CLI accepts, with the program name stripped exactly as // invocation the CLI accepts, with the program name stripped exactly as
// Entry strips it. Each one must leave stdout untouched by the banner. // Entry strips it. Each one must suppress the banner.
// //
//nolint:gochecknoglobals // read-only test fixture shared by two tests //nolint:gochecknoglobals // read-only test fixture shared by two tests
var jsonArgumentVectors = map[string][]string{ var jsonArgumentVectors = map[string][]string{
@@ -74,39 +65,23 @@ var jsonArgumentVectors = map[string][]string{
}, },
} }
// TestJSONInvocationStdoutIsExactlyOneDocument is the CLI-layer // TestJSONInvocationSuppressesBanner checks that every --json
// regression guard for issue #106: `vaultik snapshot list --json | jq` // invocation suppresses the startup banner, as the README says --json
// must work with no other flags. // does along with --quiet and --cron. The scan is over the raw argument
// // vector, so each position and spelling of --json is listed.
// internal/vaultik's TestListSnapshots_JSONStdoutIsOnlyTheDocument func TestJSONInvocationSuppressesBanner(t *testing.T) {
// guards the same contract one layer down, but it calls the library
// function directly and so cannot see Entry, which is where the
// contamination was: the startup banner is written to stdout before
// cobra parses anything, and the suppression scan did not know about
// --json. The two banner lines and the blank line landed ahead of the
// document and `jq` refused the result.
//
// The document is a constant here because this test is about the
// argument vectors, one per --json command; the one that runs a real
// command end to end is TestEntryJSONStdoutIsExactlyOneDocument below.
func TestJSONInvocationStdoutIsExactlyOneDocument(t *testing.T) {
t.Parallel() t.Parallel()
for name, argv := range jsonArgumentVectors { for name, argv := range jsonArgumentVectors {
t.Run(name, func(t *testing.T) { t.Run(name, func(t *testing.T) {
t.Parallel() t.Parallel()
var stdout bytes.Buffer var banner bytes.Buffer
emitStartupBanner(argv, &stdout) emitStartupBanner(argv, &banner)
require.Empty(t, stdout.String(), assert.Empty(t, banner.String(),
"nothing may reach stdout ahead of a --json document") "--json suppresses the banner")
_, err := stdout.WriteString(placeholderJSONDocument)
require.NoError(t, err)
requireExactlyOneJSONDocument(t, stdout.String())
}) })
} }
} }
@@ -127,11 +102,11 @@ func TestBannerStillPrintedWithoutSuppressingFlag(t *testing.T) {
t.Run(name, func(t *testing.T) { t.Run(name, func(t *testing.T) {
t.Parallel() t.Parallel()
var stdout bytes.Buffer var banner bytes.Buffer
emitStartupBanner(argv, &stdout) emitStartupBanner(argv, &banner)
assert.Contains(t, stdout.String(), "starting up at", assert.Contains(t, banner.String(), "starting up at",
"the banner belongs on invocations that did not opt out") "the banner belongs on invocations that did not opt out")
}) })
} }
@@ -247,9 +222,7 @@ func TestEntryJSONStdoutIsExactlyOneDocument(t *testing.T) {
// captureProcessStdout redirects the process's own stdout to a pipe for // captureProcessStdout redirects the process's own stdout to a pipe for
// the duration of fn and returns what was written to it. The redirection // the duration of fn and returns what was written to it. The redirection
// has to be at the file-descriptor level rather than through an injected // has to be at the file-descriptor level rather than through an injected
// writer, because the banner and the JSON encoder reach os.Stdout // writer, because the commands Entry runs reach os.Stdout directly.
// independently and the point of the test is that both land in the same
// place.
// //
// Not parallel-safe: os.Stdout is process-global. // Not parallel-safe: os.Stdout is process-global.
func captureProcessStdout(t *testing.T, fn func()) string { func captureProcessStdout(t *testing.T, fn func()) string {
internal/cli/entry_interrupt_test.go
+203
View File
@@ -0,0 +1,203 @@
package cli //nolint:testpackage // shares runEntry and the argument constants
import (
"fmt"
"net/http"
"net/http/httptest"
"os"
"os/signal"
"path/filepath"
"strings"
"testing"
"time"
"github.com/adrg/xdg"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// stalledStoreConfig is hermeticConfig with an s3:// destination store
// in place of the file:// one. The server behind it accepts any
// credentials.
const stalledStoreConfig = `age_recipients:
- age1278m9q7dp3chsh2dcy82qk27v047zywyvtxwnj4cvt0z65jw6a7q5dqhfj
snapshots:
test:
paths:
- %s
storage_url: s3://bucket?endpoint=%s&ssl=false
s3:
access_key_id: key
secret_access_key: secret
index_path: %s
hostname: test-host
`
// interruptRepeat is how often interruptOnFirstRequest sends SIGINT.
const interruptRepeat = 50 * time.Millisecond
// TestEntryInterruptedRun sends SIGINT to the test process while a
// command waits on the destination store, and checks that Entry returns
// 130 and prints one line on stderr saying the run was interrupted. The
// store is a local HTTP server that holds every request open, so the
// command is always mid-operation when the signal arrives. The two
// cases cover --cron and --json, which silence other output.
//
// Not parallel: it signals the process and replaces os.Args, os.Stdout,
// os.Stderr and the xdg globals.
//
//nolint:paralleltest // signals the process and replaces process globals
func TestEntryInterruptedRun(t *testing.T) {
for _, testCase := range []struct {
name string
args []string
}{
{
name: "snapshot create --cron",
args: []string{cmdSnapshot, cmdCreate, "--cron"},
},
{
name: "snapshot verify --json",
args: []string{cmdSnapshot, cmdVerify, someSnapshotID, flagJSON},
},
} {
t.Run(testCase.name, func(t *testing.T) {
endpoint, requestArrived := startStalledStore(t)
configPath := writeStalledStoreConfig(t, endpoint)
interruptOnFirstRequest(t, requestArrived)
code, _, stderr := runEntry(t,
append([]string{flagConfig, configPath}, testCase.args...)...)
assert.Equal(t, 130, code)
assert.Equal(t, 1,
strings.Count(stderr, errInterrupted.Error()), stderr)
})
}
}
// interruptOnFirstRequest sends SIGINT to the test process every
// interruptRepeat, from the first request to the destination store until
// the test ends. One signal is not enough: the command can reach the
// store before fx has started catching signals. The test catches SIGINT
// too, so that a signal fx is not catching does not kill the test
// binary.
func interruptOnFirstRequest(t *testing.T, requestArrived <-chan struct{}) {
t.Helper()
self, err := os.FindProcess(os.Getpid())
require.NoError(t, err)
caught := make(chan os.Signal, 1)
signal.Notify(caught, os.Interrupt)
testEnded := make(chan struct{})
senderDone := make(chan struct{})
// Stop catching SIGINT only after the sender has returned. The sender
// waits for each SIGINT it sends to arrive on caught; one still on
// its way after signal.Stop would kill the test binary.
t.Cleanup(func() {
close(testEnded)
<-senderDone
signal.Stop(caught)
})
go func() {
defer close(senderDone)
select {
case <-requestArrived:
case <-testEnded:
return
}
ticker := time.NewTicker(interruptRepeat)
defer ticker.Stop()
for {
// Empty caught, so that the receive below waits for this
// SIGINT rather than an earlier one.
select {
case <-caught:
default:
}
sendErr := self.Signal(os.Interrupt)
if sendErr != nil {
t.Errorf("sending SIGINT: %v", sendErr)
return
}
<-caught
select {
case <-testEnded:
return
case <-ticker.C:
}
}
}()
}
// startStalledStore starts an HTTP server that never answers: each
// request is held until the client gives up on it or the test ends.
// It returns the server's host:port and a channel that receives a value
// when the first request arrives.
func startStalledStore(t *testing.T) (string, <-chan struct{}) {
t.Helper()
requestArrived := make(chan struct{}, 1)
release := make(chan struct{})
server := httptest.NewServer(http.HandlerFunc(
func(_ http.ResponseWriter, r *http.Request) {
select {
case requestArrived <- struct{}{}:
default:
}
select {
case <-r.Context().Done():
case <-release:
}
}))
// Cleanups run last-registered first, so release lets any held
// request return before Close waits for it.
t.Cleanup(server.Close)
t.Cleanup(func() { close(release) })
return server.Listener.Addr().String(), requestArrived
}
// writeStalledStoreConfig writes a config whose destination store is the
// server at endpoint and whose snapshot source holds one small file, so
// that `snapshot create` has a blob to upload. Returns the config path.
func writeStalledStoreConfig(t *testing.T, endpoint string) string {
t.Helper()
dir := t.TempDir()
configPath := filepath.Join(dir, "config.yml")
sourceDir := filepath.Join(dir, "source")
require.NoError(t, os.Mkdir(sourceDir, 0o750))
require.NoError(t, os.WriteFile(filepath.Join(sourceDir, "file.txt"),
[]byte("contents"), 0o600))
contents := fmt.Sprintf(stalledStoreConfig,
sourceDir, endpoint, filepath.Join(dir, "index.sqlite"))
require.NoError(t,
os.WriteFile(configPath, []byte(contents), configFileMode))
// The PID lock lives under xdg.DataHome, which xdg resolves at
// package init; point it at the temp dir so the test neither
// touches nor collides with the real one.
t.Setenv("XDG_DATA_HOME", filepath.Join(dir, "data"))
xdg.Reload()
t.Cleanup(xdg.Reload)
return configPath
}
internal/cli/entry_status_test.go
+3 -3
View File
@@ -15,8 +15,8 @@ import (
// run, so a failing command must come back with a non-zero code rather // run, so a failing command must come back with a non-zero code rather
// than ending the process here. // than ending the process here.
// //
// Stdout is captured only to keep the banner and command output off the // Stdout and stderr are captured only to keep the banner and command
// test log; the assertion is on the returned code. // output off the test log; the assertion is on the returned code.
// //
//nolint:paralleltest // replaces os.Args and rootFlags //nolint:paralleltest // replaces os.Args and rootFlags
func TestEntryReturnsStatusCode(t *testing.T) { func TestEntryReturnsStatusCode(t *testing.T) {
@@ -50,7 +50,7 @@ func TestEntryReturnsStatusCode(t *testing.T) {
var code int var code int
_ = captureProcessStdout(t, func() { code = Entry() }) _, _ = captureProcessStdoutAndStderr(t, func() { code = Entry() })
assert.Equal(t, testCase.want, code) assert.Equal(t, testCase.want, code)
}) })
internal/cli/entry_stdout_stderr_test.go
+200
View File
@@ -0,0 +1,200 @@
package cli //nolint:testpackage // shares hermeticConfig and the capture helpers
import (
"context"
"encoding/json"
"fmt"
"log/slog"
"os"
"path/filepath"
"strings"
"testing"
"github.com/adrg/xdg"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/database"
)
// TestEntryCompletionStdoutIsTheScript runs `vaultik completion bash`,
// whose stdout the README tells the user to source. The script has to
// start on the first line.
//
//nolint:paralleltest // replaces os.Args, os.Stdout and os.Stderr
func TestEntryCompletionStdoutIsTheScript(t *testing.T) {
code, stdout, _ := runEntry(t, "completion", "bash")
require.Equal(t, 0, code)
firstLine, _, _ := strings.Cut(stdout, "\n")
assert.True(t, strings.HasPrefix(firstLine, "# bash completion"),
"the first line of stdout must be the script's, got %q", firstLine)
}
// TestEntryConfigGetStdoutIsTheValue runs `vaultik config get`, whose
// stdout a script reads as the value and nothing else.
//
//nolint:paralleltest // replaces os.Args, os.Stdout and os.Stderr
func TestEntryConfigGetStdoutIsTheValue(t *testing.T) {
configPath := filepath.Join(t.TempDir(), "config.yml")
require.NoError(t, os.WriteFile(configPath,
[]byte("hostname: test-host\n"), configFileMode))
code, stdout, _ := runEntry(t,
flagConfig, configPath, "config", "get", "hostname")
require.Equal(t, 0, code)
assert.Equal(t, "test-host\n", stdout)
}
// TestEntryJSONFailureIsReportedOnStderr runs each --json command that
// writes no document when it fails, against a destination it cannot
// use. The error must reach stderr, and stdout must stay empty.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntryJSONFailureIsReportedOnStderr(t *testing.T) {
for _, testCase := range []struct {
name string
args []string
wantOnStderr string
}{
{
name: "remote info",
args: []string{cmdRemote, cmdInfo, flagJSON},
wantOnStderr: "Failed to get remote info",
},
{
name: "prune",
args: []string{cmdPrune, flagJSON},
wantOnStderr: "Prune failed",
},
{
name: "snapshot remove",
args: []string{cmdSnapshot, cmdRemove, someSnapshotID, flagJSON},
wantOnStderr: "Failed to remove snapshot",
},
} {
t.Run(testCase.name, func(t *testing.T) {
configPath := writeUnusableDestinationConfig(t)
code, stdout, stderr := runEntry(t,
append([]string{flagConfig, configPath}, testCase.args...)...)
assert.Equal(t, 1, code)
assert.Empty(t, stdout,
"a failed --json command has no document to write")
assert.Contains(t, stderr, testCase.wantOnStderr,
"the failure must be reported on stderr")
})
}
}
// TestEntrySnapshotRemoveJSONWarningIsOnStderr runs `snapshot remove
// --json` on a snapshot in the local index, against a destination
// directory that does not exist. The command removes the snapshot from
// the local index and still exits 0. Its stdout must hold the document
// alone, with the warning about the destination store on stderr: the
// command to run again once it is reachable, and the snapshot's ID in
// the record's snapshot_id field.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotRemoveJSONWarningIsOnStderr(t *testing.T) {
configPath, indexPath := writeMissingDestinationConfig(t)
seedStaleSnapshotRecord(t, indexPath)
code, stdout, stderr := runEntry(t, flagConfig, configPath,
cmdSnapshot, cmdRemove, stalePruneSnapshotID, flagJSON)
require.Equal(t, 0, code)
requireExactlyOneJSONDocument(t, stdout)
// stderr is a pipe here, so the logger writes one JSON record a line.
var warning map[string]any
for line := range strings.Lines(stderr) {
if strings.Contains(line,
"Could not remove snapshot metadata from remote storage") {
require.NoError(t, json.Unmarshal([]byte(line), &warning))
}
}
require.NotNil(t, warning, "the warning must reach stderr")
assert.Contains(t, warning[slog.MessageKey],
"run 'vaultik snapshot remove' with the snapshot's ID again")
assert.Equal(t, stalePruneSnapshotID, warning["snapshot_id"])
}
// writeMissingDestinationConfig builds a config whose destination
// directory does not exist. Returns the config path and the path of
// its local index, which is not created here.
func writeMissingDestinationConfig(t *testing.T) (string, string) {
t.Helper()
dir := t.TempDir()
configPath := filepath.Join(dir, "config.yml")
indexPath := filepath.Join(dir, "index.sqlite")
contents := fmt.Sprintf(hermeticConfig,
filepath.Join(dir, "source"),
filepath.Join(dir, "missing-store"),
indexPath)
require.NoError(t,
os.WriteFile(configPath, []byte(contents), configFileMode))
// The PID lock lives under xdg.DataHome, which xdg resolves at
// package init; point it at the temp dir so the test neither
// touches nor collides with the real one.
t.Setenv("XDG_DATA_HOME", filepath.Join(dir, "data"))
xdg.Reload()
t.Cleanup(xdg.Reload)
return configPath, indexPath
}
// writeUnusableDestinationConfig builds a config whose destination
// directory does not exist, which fails `remote info`, and whose local
// index is bound to another destination, which fails `prune` and
// `snapshot remove` (a missing destination alone only makes `snapshot
// remove` warn). Returns the config path.
func writeUnusableDestinationConfig(t *testing.T) string {
t.Helper()
configPath, indexPath := writeMissingDestinationConfig(t)
ctx := context.Background()
db, err := database.New(ctx, indexPath)
require.NoError(t, err)
defer func() { require.NoError(t, db.Close()) }()
require.NoError(t, database.NewRepositories(db).LocalMeta.Set(ctx,
database.LocalMetaKeyStorageURL, "file://"+t.TempDir()))
return configPath
}
// runEntry runs Entry with args after the program name and returns its
// exit code and what it wrote to stdout and stderr.
//
// Not parallel-safe: it replaces os.Args, os.Stdout and os.Stderr.
func runEntry(t *testing.T, args ...string) (int, string, string) {
t.Helper()
previousArgs := os.Args
t.Cleanup(func() {
os.Args = previousArgs
rootFlags = RootFlags{}
})
os.Args = append([]string{programName}, args...)
var code int
stdout, stderr := captureProcessStdoutAndStderr(t,
func() { code = Entry() })
return code, stdout, stderr
}
internal/cli/prune.go
+5 -7
View File
@@ -22,9 +22,11 @@ scans every snapshot manifest in the destination store, builds the
set of still-referenced blob hashes, and deletes any blob not in that set of still-referenced blob hashes, and deletes any blob not in that
set. set.
Snapshot create --prune and snapshot remove run the same cleanup Snapshot create --prune runs the same cleanup automatically; this
automatically; this command is the manual entry point for the same command is the manual entry point for the same work (e.g. after a
work (e.g. after a crashed backup or to reclaim storage).`, crashed backup or to reclaim storage). Snapshot remove leaves blobs in
place; run this command afterwards to delete the ones no longer
referenced.`,
Args: cobra.NoArgs, Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error { RunE: func(cmd *cobra.Command, _ []string) error {
// Use unified config resolution // Use unified config resolution
@@ -48,10 +50,6 @@ work (e.g. after a crashed backup or to reclaim storage).`,
}, func(v *vaultik.Vaultik) error { }, func(v *vaultik.Vaultik) error {
return v.Prune(opts) return v.Prune(opts)
}, func(err error) { }, func(err error) {
if opts.JSON {
return
}
log.Error("Prune operation failed", "error", err) log.Error("Prune operation failed", "error", err)
ReportErrorf("Prune failed: %v", err) ReportErrorf("Prune failed: %v", err)
}) })
internal/cli/remote.go
+1 -5
View File
@@ -45,7 +45,7 @@ This is destructive and irreversible. Requires --force.`,
return errNukeNeedsForce return errNukeNeedsForce
} }
return runVaultikApp(cmd, mutating, false, false, "Remote nuke failed", return runVaultikApp(cmd, mutating, false, "Remote nuke failed",
func(v *vaultik.Vaultik) error { func(v *vaultik.Vaultik) error {
return v.NukeRemote(true) return v.NukeRemote(true)
}) })
@@ -92,10 +92,6 @@ func newRemoteInfoCommand() *cobra.Command {
}, func(v *vaultik.Vaultik) error { }, func(v *vaultik.Vaultik) error {
return v.RemoteInfo(jsonOutput) return v.RemoteInfo(jsonOutput)
}, func(err error) { }, func(err error) {
if jsonOutput {
return
}
log.Error("Failed to get remote info", "error", err) log.Error("Failed to get remote info", "error", err)
ReportErrorf("Failed to get remote info: %v", err) ReportErrorf("Failed to get remote info: %v", err)
}) })
internal/cli/snapshot.go
+9 -7
View File
@@ -66,8 +66,9 @@ func newSnapshotCreateCommand() *cobra.Command {
If snapshot names are provided, only those snapshots are created. If snapshot names are provided, only those snapshots are created.
If no names are provided, all configured snapshots are created. If no names are provided, all configured snapshots are created.
Config is located at /etc/vaultik/config.yml by default, but can be overridden by The config is read from the path given by --config or VAULTIK_CONFIG;
specifying a path using --config or by setting VAULTIK_CONFIG to a path.`, otherwise from the platform config directory (~/.config/vaultik/config.yml
on Linux), then /etc/vaultik/config.yml.`,
Args: cobra.ArbitraryArgs, Args: cobra.ArbitraryArgs,
RunE: func(cmd *cobra.Command, args []string) error { RunE: func(cmd *cobra.Command, args []string) error {
// Pass snapshot names from args // Pass snapshot names from args
@@ -126,7 +127,7 @@ func newSnapshotListCommand() *cobra.Command {
Long: "Lists all snapshots with their ID, timestamp, and compressed size", Long: "Lists all snapshots with their ID, timestamp, and compressed size",
Args: cobra.NoArgs, Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error { RunE: func(cmd *cobra.Command, _ []string) error {
return runVaultikApp(cmd, readOnly, false, false, return runVaultikApp(cmd, readOnly, false,
"Failed to list snapshots", "Failed to list snapshots",
func(v *vaultik.Vaultik) error { func(v *vaultik.Vaultik) error {
return v.ListSnapshots(jsonOutput) return v.ListSnapshots(jsonOutput)
@@ -162,7 +163,7 @@ restrict the operation to specific snapshot names.`,
return errPurgeCriteriaBoth return errPurgeCriteriaBoth
} }
return runVaultikApp(cmd, mutating, false, false, return runVaultikApp(cmd, mutating, false,
"Failed to purge snapshots", "Failed to purge snapshots",
func(v *vaultik.Vaultik) error { func(v *vaultik.Vaultik) error {
return v.PurgeSnapshotsWithOptions(opts) return v.PurgeSnapshotsWithOptions(opts)
@@ -258,14 +259,15 @@ Use --local-only to skip the remote half (e.g. when you want to forget a
snapshot locally without touching the destination store). snapshot locally without touching the destination store).
If the remote is unreachable, the local-database removal still completes If the remote is unreachable, the local-database removal still completes
and a warning is emitted; rerun 'vaultik prune' once the destination store and a warning is emitted; run 'vaultik snapshot remove <snapshot-id>' again
is reachable to finish remote cleanup. once the destination store is reachable to remove the snapshot's metadata
from it ('vaultik prune' does not).
To wipe the entire destination store and start over, use 'vaultik remote To wipe the entire destination store and start over, use 'vaultik remote
nuke --force' — it is the single supported entry point for that.`, nuke --force' — it is the single supported entry point for that.`,
Args: requireSnapshotIDArg, Args: requireSnapshotIDArg,
RunE: func(cmd *cobra.Command, args []string) error { RunE: func(cmd *cobra.Command, args []string) error {
return runVaultikApp(cmd, mutating, opts.JSON, opts.JSON, return runVaultikApp(cmd, mutating, opts.JSON,
"Failed to remove snapshot", "Failed to remove snapshot",
func(v *vaultik.Vaultik) error { func(v *vaultik.Vaultik) error {
_, err := v.RemoveSnapshot(args[0], opts) _, err := v.RemoveSnapshot(args[0], opts)
internal/config/config.go
+30 -16
View File
@@ -33,18 +33,19 @@ const secretKeyPrefix = "AGE-SECRET-KEY-"
const ( const (
defaultBlobSizeLimit = Size(10 * 1024 * 1024 * 1024) // 10GB defaultBlobSizeLimit = Size(10 * 1024 * 1024 * 1024) // 10GB
defaultChunkSize = Size(10 * 1024 * 1024) // 10MB defaultChunkSize = Size(10 * 1024 * 1024) // 10MB
defaultS3PartSize = Size(5 * 1024 * 1024) // 5MB defaultS3PartSize = Size(5 * 1024 * 1024) // 5MiB
defaultCompressionLevel = 3 defaultCompressionLevel = 3
minChunkSize = 1024 * 1024 // 1MB minChunkSize = 1024 * 1024 // 1MB
minCompressionLevel = 1 minCompressionLevel = 1
maxCompressionLevel = 19 maxCompressionLevel = 19
// S3 accepts a multipart upload part from 5MiB to 5GiB.
minS3PartSize = 5 * 1024 * 1024
maxS3PartSize = 5 * 1024 * 1024 * 1024
) )
// Sentinel validation errors. // Sentinel validation errors.
var ( var (
errNoConfigPath = errors.New("config path not provided") errNoConfigPath = errors.New("config path not provided")
errNoAgeRecipients = errors.New(
"at least one age_recipient is required (generate with: age-keygen)")
errRecipientIsSecretKey = errors.New( errRecipientIsSecretKey = errors.New(
"an age secret key was given where a public key (age1...) belongs") "an age secret key was given where a public key (age1...) belongs")
errRecipientNotX25519 = errors.New( errRecipientNotX25519 = errors.New(
@@ -57,6 +58,7 @@ var (
"blob_size_limit must be at least the largest chunk the chunker can " + "blob_size_limit must be at least the largest chunk the chunker can " +
"emit (chunk_size times the FastCDC size spread)") "emit (chunk_size times the FastCDC size spread)")
errBadCompression = errors.New("compression_level must be between 1 and 19") errBadCompression = errors.New("compression_level must be between 1 and 19")
errBadS3PartSize = errors.New("s3.part_size must be between 5MiB and 5GiB")
errBadStorageScheme = errors.New( errBadStorageScheme = errors.New(
"storage_url must start with s3://, file://, or rclone://") "storage_url must start with s3://, file://, or rclone://")
errStorageNotConfigured = errors.New( errStorageNotConfigured = errors.New(
@@ -246,6 +248,7 @@ func Load(path string) (*Config, error) {
ChunkSize: defaultChunkSize, ChunkSize: defaultChunkSize,
IndexPath: filepath.Join(xdg.DataHome, appName, "index.sqlite"), IndexPath: filepath.Join(xdg.DataHome, appName, "index.sqlite"),
CompressionLevel: defaultCompressionLevel, CompressionLevel: defaultCompressionLevel,
S3: S3Config{PartSize: defaultS3PartSize},
} }
// Convert smartconfig data to YAML then unmarshal // Convert smartconfig data to YAML then unmarshal
@@ -296,17 +299,13 @@ func Load(path string) (*Config, error) {
cfg.S3.Region = "us-east-1" cfg.S3.Region = "us-east-1"
} }
if cfg.S3.PartSize == 0 {
cfg.S3.PartSize = defaultS3PartSize
}
// Check config file permissions (warn if world or group readable) // Check config file permissions (warn if world or group readable)
//nolint:gosec // G703: config path is operator-supplied by design //nolint:gosec // G703: config path is operator-supplied by design
info, statErr := os.Stat(path) info, statErr := os.Stat(path)
if statErr == nil { if statErr == nil {
mode := info.Mode().Perm() mode := info.Mode().Perm()
if mode&0044 != 0 { // group or world readable if mode&0044 != 0 { // group or world readable
log.Warn("Config file has insecure permissions (contains S3 credentials)", log.Warn(cfg.readableByOthersWarning(),
"path", path, "path", path,
"mode", fmt.Sprintf("%04o", mode), "mode", fmt.Sprintf("%04o", mode),
"recommendation", "chmod 600 "+path) "recommendation", "chmod 600 "+path)
@@ -323,9 +322,10 @@ func Load(path string) (*Config, error) {
// Validate checks if the configuration is valid and complete. // Validate checks if the configuration is valid and complete.
// It ensures all required fields are present and have valid values: // It ensures all required fields are present and have valid values:
// - At least one age recipient must be specified, and every recipient must // - Every age recipient must parse as an X25519 age1... public key (so a
// parse as an X25519 age1... public key (so a bad entry fails at load, not // bad entry fails at load, not mid-backup); errors name the position,
// mid-backup); errors name the position, never the value // never the value. An empty list is accepted, because only snapshot
// create needs a recipient and it checks for one itself
// - At least one snapshot must be configured with at least one path // - At least one snapshot must be configured with at least one path
// - Storage must be configured (either storage_url or s3.* fields) // - Storage must be configured (either storage_url or s3.* fields)
// - Chunk size must be at least 1MB // - Chunk size must be at least 1MB
@@ -333,13 +333,10 @@ func Load(path string) (*Config, error) {
// (chunk_size times chunker.ChunkSizeSpread), so a single-chunk blob never // (chunk_size times chunker.ChunkSizeSpread), so a single-chunk blob never
// exceeds the configured limit // exceeds the configured limit
// - Compression level must be between 1 and 19 // - Compression level must be between 1 and 19
// - S3 part size must be between 5MiB and 5GiB, the part sizes S3 accepts
// //
// Returns an error describing the first validation failure encountered. // Returns an error describing the first validation failure encountered.
func (c *Config) Validate() error { func (c *Config) Validate() error {
if len(c.AgeRecipients) == 0 {
return errNoAgeRecipients
}
for i, recipient := range c.AgeRecipients { for i, recipient := range c.AgeRecipients {
err := validateAgeRecipient(recipient) err := validateAgeRecipient(recipient)
if err != nil { if err != nil {
@@ -381,6 +378,11 @@ func (c *Config) Validate() error {
return errBadCompression return errBadCompression
} }
if c.S3.PartSize.Int64() < minS3PartSize ||
c.S3.PartSize.Int64() > maxS3PartSize {
return errBadS3PartSize
}
return nil return nil
} }
@@ -416,6 +418,18 @@ func (c *Config) setAgeSecretKey() {
} }
} }
// readableByOthersWarning is the warning Load logs when others can read
// the config file. It says "may contain" because the S3 credentials are
// seen only after smartconfig has replaced any ${...} reference in the
// file with its value, so a set credential need not be in the file.
func (c *Config) readableByOthersWarning() string {
if c.S3.AccessKeyID != "" || c.S3.SecretAccessKey != "" {
return "Config file is readable by others and may contain S3 credentials"
}
return "Config file is readable by others"
}
// validateStorage validates storage configuration. // validateStorage validates storage configuration.
// If StorageURL is set, it takes precedence. S3 URLs require credentials. // If StorageURL is set, it takes precedence. S3 URLs require credentials.
// File URLs don't require any S3 configuration. // File URLs don't require any S3 configuration.
internal/config/config_test.go
+247 -2
View File
@@ -8,6 +8,7 @@ import (
"testing" "testing"
"sneak.berlin/go/vaultik/internal/chunker" "sneak.berlin/go/vaultik/internal/chunker"
"sneak.berlin/go/vaultik/internal/log"
) )
const ( const (
@@ -166,6 +167,7 @@ func TestValidateBlobSizeLimit(t *testing.T) {
ChunkSize: chunkSize, ChunkSize: chunkSize,
BlobSizeLimit: blobLimit, BlobSizeLimit: blobLimit,
CompressionLevel: 3, CompressionLevel: 3,
S3: S3Config{PartSize: defaultS3PartSize},
} }
} }
@@ -221,9 +223,124 @@ func TestValidateBlobSizeLimit(t *testing.T) {
} }
} }
// TestValidateS3PartSize checks that s3.part_size is held to the part sizes
// S3 accepts, 5MiB to 5GiB, by changing only the part size of the test
// config. "5MB" in the config file is 5,000,000 bytes, below the minimum.
func TestValidateS3PartSize(t *testing.T) {
t.Parallel()
base, err := Load(os.Getenv("VAULTIK_CONFIG"))
if err != nil {
t.Fatalf("Failed to load config: %v", err)
}
tests := []struct {
name string
partSize Size
wantErr bool
}{
{
name: "5MB is rejected",
partSize: 5_000_000,
wantErr: true,
},
{
name: "one byte below 5MiB is rejected",
partSize: minS3PartSize - 1,
wantErr: true,
},
{
name: "5MiB is accepted",
partSize: minS3PartSize,
wantErr: false,
},
{
name: "5GiB is accepted",
partSize: maxS3PartSize,
wantErr: false,
},
{
name: "one byte above 5GiB is rejected",
partSize: maxS3PartSize + 1,
wantErr: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
cfg := *base
cfg.S3.PartSize = tt.partSize
err := cfg.Validate()
if tt.wantErr {
if !errors.Is(err, errBadS3PartSize) {
t.Fatalf("Validate() error = %v, want errBadS3PartSize", err)
}
return
}
if err != nil {
t.Fatalf("Validate() unexpected error: %v", err)
}
})
}
}
// TestLoadS3PartSize checks that a config file without s3.part_size loads
// with the 5MiB default, and that an explicit 0 fails at load like any other
// part size S3 refuses.
func TestLoadS3PartSize(t *testing.T) {
t.Parallel()
const withoutPartSize = "snapshots:\n" +
" test:\n" +
" paths: [/tmp/vaultik-test-source]\n" +
"storage_url: file:///tmp/vaultik-test-storage\n"
writeConfig := func(t *testing.T, text string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "config.yml")
err := os.WriteFile(path, []byte(text), 0o600)
if err != nil {
t.Fatalf("write config: %v", err)
}
return path
}
t.Run("absent loads as 5MiB", func(t *testing.T) {
t.Parallel()
cfg, err := Load(writeConfig(t, withoutPartSize))
if err != nil {
t.Fatalf("Load() unexpected error: %v", err)
}
if cfg.S3.PartSize != defaultS3PartSize {
t.Errorf("s3.part_size = %d, want %d",
cfg.S3.PartSize, defaultS3PartSize)
}
})
t.Run("0 is rejected", func(t *testing.T) {
t.Parallel()
_, err := Load(writeConfig(t, withoutPartSize+"s3:\n part_size: 0\n"))
if !errors.Is(err, errBadS3PartSize) {
t.Fatalf("Load() error = %v, want errBadS3PartSize", err)
}
})
}
// TestValidateAgeRecipients checks that recipients are parsed at config load // TestValidateAgeRecipients checks that recipients are parsed at config load
// (a bad entry fails immediately, not mid-backup) and that no invalid entry — // (a bad entry fails immediately, not mid-backup) and that no invalid entry —
// least of all a pasted secret key — is echoed in the error. // least of all a pasted secret key — is echoed in the error. An empty list
// loads, because only snapshot create needs a recipient.
func TestValidateAgeRecipients(t *testing.T) { func TestValidateAgeRecipients(t *testing.T) {
t.Parallel() t.Parallel()
@@ -235,6 +352,7 @@ func TestValidateAgeRecipients(t *testing.T) {
ChunkSize: Size(10 * 1024 * 1024), ChunkSize: Size(10 * 1024 * 1024),
BlobSizeLimit: Size(10 * 1024 * 1024 * 1024), BlobSizeLimit: Size(10 * 1024 * 1024 * 1024),
CompressionLevel: 3, CompressionLevel: 3,
S3: S3Config{PartSize: defaultS3PartSize},
} }
} }
@@ -244,7 +362,12 @@ func TestValidateAgeRecipients(t *testing.T) {
wantErr bool wantErr bool
}{ }{
{ {
name: "config init placeholder is rejected", name: "no recipients is accepted",
recipients: nil,
wantErr: false,
},
{
name: "placeholder recipient is rejected",
recipients: []string{"age1REPLACE_WITH_YOUR_PUBLIC_KEY"}, recipients: []string{"age1REPLACE_WITH_YOUR_PUBLIC_KEY"},
wantErr: true, wantErr: true,
}, },
@@ -337,3 +460,125 @@ func TestAgeSecretKeySourceName(t *testing.T) {
}) })
} }
} }
// loadReadableConfig writes configYAML to a file that others can read,
// loads it, and returns what the logger wrote to stderr meanwhile. The
// logger writes to the os.Stderr it finds when it is initialized, so
// os.Stderr is pointed at a file first. Not parallel-safe: os.Stderr and
// the logger are process-global.
func loadReadableConfig(t *testing.T, configYAML string) string {
t.Helper()
dir := t.TempDir()
configPath := filepath.Join(dir, "config.yml")
stderrPath := filepath.Join(dir, "stderr")
err := os.WriteFile(configPath, []byte(configYAML), 0o600)
if err != nil {
t.Fatalf("writing config: %v", err)
}
//nolint:gosec // G302: the test needs a config file others can read
err = os.Chmod(configPath, 0o644)
if err != nil {
t.Fatalf("chmod config: %v", err)
}
stderrFile, err := os.Create(stderrPath) //nolint:gosec // G304: test temp path
if err != nil {
t.Fatalf("creating stderr file: %v", err)
}
previous := os.Stderr
os.Stderr = stderrFile
log.Initialize(log.Config{})
_, loadErr := Load(configPath)
os.Stderr = previous
log.Initialize(log.Config{})
_ = stderrFile.Close()
if loadErr != nil {
t.Fatalf("Load() error = %v", loadErr)
}
captured, err := os.ReadFile(stderrPath) //nolint:gosec // G304: test temp path
if err != nil {
t.Fatalf("reading stderr file: %v", err)
}
return string(captured)
}
// TestLoadWarnsReadableConfigWithoutS3Credentials checks that a config
// file others can read, holding no S3 credentials, is warned about
// without a claim that it holds them.
//
//nolint:paralleltest // loadReadableConfig replaces os.Stderr
func TestLoadWarnsReadableConfigWithoutS3Credentials(t *testing.T) {
stderr := loadReadableConfig(t, `
storage_url: file:///var/backups/vaultik
snapshots:
home:
paths:
- /home
`)
if !strings.Contains(stderr, "Config file is readable by others") {
t.Errorf("expected a warning that the file is readable by others, got %q",
stderr)
}
if strings.Contains(stderr, "S3 credentials") {
t.Errorf("warning names S3 credentials the file does not set: %q", stderr)
}
}
// TestLoadWarnsReadableConfigWithS3Credentials checks that a config file
// others can read and that sets S3 credentials, as values or as ${ENV:...}
// references, is warned about as one that may contain them.
//
//nolint:paralleltest // loadReadableConfig replaces os.Stderr
func TestLoadWarnsReadableConfigWithS3Credentials(t *testing.T) {
t.Setenv("VAULTIK_TEST_ACCESS_KEY_ID", "test-access-key")
t.Setenv("VAULTIK_TEST_SECRET_ACCESS_KEY", "test-secret-key")
configs := map[string]string{
"values": `
storage_url: s3://bucket/prefix?endpoint=s3.example.com
s3:
access_key_id: test-access-key
secret_access_key: test-secret-key
snapshots:
home:
paths:
- /home
`,
"references": `
storage_url: s3://bucket/prefix?endpoint=s3.example.com
s3:
access_key_id: ${ENV:VAULTIK_TEST_ACCESS_KEY_ID}
secret_access_key: ${ENV:VAULTIK_TEST_SECRET_ACCESS_KEY}
snapshots:
home:
paths:
- /home
`,
}
for name, configYAML := range configs {
t.Run(name, func(t *testing.T) {
stderr := loadReadableConfig(t, configYAML)
if !strings.Contains(stderr,
"Config file is readable by others and may contain S3 credentials") {
t.Errorf("expected a warning naming the S3 credentials, got %q",
stderr)
}
})
}
}
internal/database/files.go
+39 -25
View File
@@ -33,11 +33,13 @@ func (r *FileRepository) Create(ctx context.Context, tx *sql.Tx, file *File) err
} }
query := ` query := `
INSERT INTO files (id, path, source_path, mtime, size, mode, uid, gid, link_target) INSERT INTO files
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) (id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(path) DO UPDATE SET ON CONFLICT(path) DO UPDATE SET
source_path = excluded.source_path, source_path = excluded.source_path,
mtime = excluded.mtime, mtime = excluded.mtime,
mtime_nsec = excluded.mtime_nsec,
size = excluded.size, size = excluded.size,
mode = excluded.mode, mode = excluded.mode,
uid = excluded.uid, uid = excluded.uid,
@@ -54,16 +56,19 @@ func (r *FileRepository) Create(ctx context.Context, tx *sql.Tx, file *File) err
if tx != nil { if tx != nil {
LogSQL("Execute", query, LogSQL("Execute", query,
file.ID.String(), file.Path.String(), file.SourcePath.String(), file.ID.String(), file.Path.String(), file.SourcePath.String(),
file.MTime.Unix(), file.Size, file.Mode, file.UID, file.GID, file.MTime.Unix(), file.MTime.Nanosecond(),
file.Size, file.Mode, file.UID, file.GID,
file.LinkTarget.String()) file.LinkTarget.String())
err = tx.QueryRowContext(ctx, query, err = tx.QueryRowContext(ctx, query,
file.ID.String(), file.Path.String(), file.SourcePath.String(), file.ID.String(), file.Path.String(), file.SourcePath.String(),
file.MTime.Unix(), file.Size, file.Mode, file.UID, file.GID, file.MTime.Unix(), file.MTime.Nanosecond(),
file.Size, file.Mode, file.UID, file.GID,
file.LinkTarget.String()).Scan(&idStr) file.LinkTarget.String()).Scan(&idStr)
} else { } else {
err = r.db.QueryRowWithLog(ctx, query, err = r.db.QueryRowWithLog(ctx, query,
file.ID.String(), file.Path.String(), file.SourcePath.String(), file.ID.String(), file.Path.String(), file.SourcePath.String(),
file.MTime.Unix(), file.Size, file.Mode, file.UID, file.GID, file.MTime.Unix(), file.MTime.Nanosecond(),
file.Size, file.Mode, file.UID, file.GID,
file.LinkTarget.String()).Scan(&idStr) file.LinkTarget.String()).Scan(&idStr)
} }
@@ -84,7 +89,7 @@ func (r *FileRepository) Create(ctx context.Context, tx *sql.Tx, file *File) err
// in the index. // in the index.
func (r *FileRepository) GetByPath(ctx context.Context, path string) (*File, error) { func (r *FileRepository) GetByPath(ctx context.Context, path string) (*File, error) {
query := ` query := `
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
FROM files FROM files
WHERE path = ? WHERE path = ?
` `
@@ -104,7 +109,7 @@ func (r *FileRepository) GetByPath(ctx context.Context, path string) (*File, err
// GetByID retrieves a file by its UUID // GetByID retrieves a file by its UUID
func (r *FileRepository) GetByID(ctx context.Context, id types.FileID) (*File, error) { func (r *FileRepository) GetByID(ctx context.Context, id types.FileID) (*File, error) {
query := ` query := `
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
FROM files FROM files
WHERE id = ? WHERE id = ?
` `
@@ -127,7 +132,7 @@ func (r *FileRepository) GetByPathTx(
ctx context.Context, tx *sql.Tx, path string, ctx context.Context, tx *sql.Tx, path string,
) (*File, error) { ) (*File, error) {
query := ` query := `
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
FROM files FROM files
WHERE path = ? WHERE path = ?
` `
@@ -158,13 +163,14 @@ func (r *FileRepository) ListModifiedSince(
ctx context.Context, since time.Time, ctx context.Context, since time.Time,
) ([]*File, error) { ) ([]*File, error) {
query := ` query := `
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
FROM files FROM files
WHERE mtime >= ? WHERE (mtime, mtime_nsec) >= (?, ?)
ORDER BY path ORDER BY path
` `
rows, err := r.db.conn.QueryContext(ctx, query, since.Unix()) rows, err := r.db.conn.QueryContext(ctx, query,
since.Unix(), since.Nanosecond())
if err != nil { if err != nil {
return nil, fmt.Errorf("querying files: %w", err) return nil, fmt.Errorf("querying files: %w", err)
} }
@@ -228,19 +234,24 @@ func (r *FileRepository) DeleteByID(
return nil return nil
} }
// ListByPrefix returns all files whose path starts with prefix, ordered by // ListUnderPath returns the file at path and every file beneath it,
// path. // ordered by path. Paths are compared case-sensitively, and a trailing
func (r *FileRepository) ListByPrefix( // slash on path is ignored, so "/" lists every file.
ctx context.Context, prefix string, func (r *FileRepository) ListUnderPath(
ctx context.Context, path string,
) ([]*File, error) { ) ([]*File, error) {
path = strings.TrimRight(path, "/")
dirPrefix := path + "/"
// LIKE would ignore ASCII case and treat _ and % in path as wildcards.
query := ` query := `
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
FROM files FROM files
WHERE path LIKE ? || '%' WHERE path = ? OR substr(path, 1, length(?)) = ?
ORDER BY path ORDER BY path
` `
rows, err := r.db.conn.QueryContext(ctx, query, prefix) rows, err := r.db.conn.QueryContext(ctx, query, path, dirPrefix, dirPrefix)
if err != nil { if err != nil {
return nil, fmt.Errorf("querying files: %w", err) return nil, fmt.Errorf("querying files: %w", err)
} }
@@ -319,7 +330,7 @@ func (r *FileRepository) ListIDsWithChunksNotInUploadedBlobs(
// ListAll returns all files in the database // ListAll returns all files in the database
func (r *FileRepository) ListAll(ctx context.Context) ([]*File, error) { func (r *FileRepository) ListAll(ctx context.Context) ([]*File, error) {
query := ` query := `
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
FROM files FROM files
ORDER BY path ORDER BY path
` `
@@ -360,7 +371,7 @@ func (r *FileRepository) CreateBatch(
} }
// Each files row binds this many SQL variables. // Each files row binds this many SQL variables.
const fileCols = 9 const fileCols = 10
// Batch at 100 rows to be safe with SQLite's variable limit. // Batch at 100 rows to be safe with SQLite's variable limit.
const batchSize = 100 const batchSize = 100
@@ -371,7 +382,7 @@ func (r *FileRepository) CreateBatch(
batch := files[i:end] batch := files[i:end]
query := `INSERT INTO files query := `INSERT INTO files
(id, path, source_path, mtime, size, mode, uid, gid, link_target) (id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target)
VALUES ` VALUES `
args := make([]any, 0, len(batch)*fileCols) args := make([]any, 0, len(batch)*fileCols)
@@ -383,11 +394,12 @@ func (r *FileRepository) CreateBatch(
querySb325.WriteString(", ") querySb325.WriteString(", ")
} }
querySb325.WriteString("(?, ?, ?, ?, ?, ?, ?, ?, ?)") querySb325.WriteString("(?, ?, ?, ?, ?, ?, ?, ?, ?, ?)")
args = append(args, args = append(args,
f.ID.String(), f.Path.String(), f.SourcePath.String(), f.ID.String(), f.Path.String(), f.SourcePath.String(),
f.MTime.Unix(), f.Size, f.Mode, f.UID, f.GID, f.MTime.Unix(), f.MTime.Nanosecond(),
f.Size, f.Mode, f.UID, f.GID,
f.LinkTarget.String()) f.LinkTarget.String())
} }
@@ -396,6 +408,7 @@ func (r *FileRepository) CreateBatch(
query += ` ON CONFLICT(path) DO UPDATE SET query += ` ON CONFLICT(path) DO UPDATE SET
source_path = excluded.source_path, source_path = excluded.source_path,
mtime = excluded.mtime, mtime = excluded.mtime,
mtime_nsec = excluded.mtime_nsec,
size = excluded.size, size = excluded.size,
mode = excluded.mode, mode = excluded.mode,
uid = excluded.uid, uid = excluded.uid,
@@ -455,7 +468,7 @@ func (r *FileRepository) scanFileFrom(row fileRowScanner) (*File, error) {
var ( var (
file File file File
idStr, pathStr, sourcePathStr string idStr, pathStr, sourcePathStr string
mtimeUnix int64 mtimeUnix, mtimeNsec int64
linkTarget sql.NullString linkTarget sql.NullString
) )
@@ -464,6 +477,7 @@ func (r *FileRepository) scanFileFrom(row fileRowScanner) (*File, error) {
&pathStr, &pathStr,
&sourcePathStr, &sourcePathStr,
&mtimeUnix, &mtimeUnix,
&mtimeNsec,
&file.Size, &file.Size,
&file.Mode, &file.Mode,
&file.UID, &file.UID,
@@ -482,7 +496,7 @@ func (r *FileRepository) scanFileFrom(row fileRowScanner) (*File, error) {
file.Path = types.FilePath(pathStr) file.Path = types.FilePath(pathStr)
file.SourcePath = types.SourcePath(sourcePathStr) file.SourcePath = types.SourcePath(sourcePathStr)
file.MTime = time.Unix(mtimeUnix, 0).UTC() file.MTime = time.Unix(mtimeUnix, mtimeNsec).UTC()
if linkTarget.Valid { if linkTarget.Valid {
file.LinkTarget = types.FilePath(linkTarget.String) file.LinkTarget = types.FilePath(linkTarget.String)
} }
internal/database/files_test.go
+187
View File
@@ -5,10 +5,12 @@ import (
"database/sql" "database/sql"
"errors" "errors"
"os" "os"
"slices"
"testing" "testing"
"time" "time"
"sneak.berlin/go/vaultik/internal/database" "sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/types"
) )
// errTestRollback is the sentinel returned from transaction bodies to // errTestRollback is the sentinel returned from transaction bodies to
@@ -134,6 +136,82 @@ func TestFileRepositoryListDelete(t *testing.T) {
} }
} }
func TestFileRepositoryListUnderPath(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repo := database.NewFileRepository(db)
const (
docDir = "/home/u/doc"
docFile = "/home/u/doc/a.txt"
)
// In path order, so the root case can expect all of them as listed.
paths := []string{
"/home/u/50%/x.txt",
"/home/u/50percent/y.txt",
"/home/u/DOC/c.txt",
"/home/u/a_b/x.txt",
"/home/u/axb/y.txt",
docDir,
"/home/u/doc.txt.bak",
docFile,
"/home/u/doc/sub/b.txt",
"/home/u/doc2/b.txt",
}
for _, path := range paths {
err := repo.Create(ctx, nil, &database.File{
Path: types.FilePath(path),
MTime: time.Now().Truncate(time.Second),
Mode: 0644,
})
if err != nil {
t.Fatalf("failed to create %s: %v", path, err)
}
}
docTree := []string{docDir, docFile, "/home/u/doc/sub/b.txt"}
tests := []struct {
name string
path string
want []string
}{
{"directory", docDir, docTree},
{"directory with trailing slash", docDir + "/", docTree},
{"directory differing only in case", "/home/u/DOC",
[]string{"/home/u/DOC/c.txt"}},
{"file", docFile, []string{docFile}},
{"underscore is literal", "/home/u/a_b",
[]string{"/home/u/a_b/x.txt"}},
{"percent is literal", "/home/u/50%",
[]string{"/home/u/50%/x.txt"}},
{"root", "/", paths},
}
for _, tt := range tests {
files, err := repo.ListUnderPath(ctx, tt.path)
if err != nil {
t.Fatalf("%s: failed to list files: %v", tt.name, err)
}
got := make([]string, 0, len(files))
for _, f := range files {
got = append(got, f.Path.String())
}
if !slices.Equal(got, tt.want) {
t.Errorf("%s: listing %q got %q, want %q",
tt.name, tt.path, got, tt.want)
}
}
}
func TestFileRepositorySymlink(t *testing.T) { func TestFileRepositorySymlink(t *testing.T) {
t.Parallel() t.Parallel()
@@ -174,6 +252,115 @@ func TestFileRepositorySymlink(t *testing.T) {
} }
} }
// An mtime after 2262 or before 1678 does not fit in int64 nanoseconds
// since the epoch, and must still come back from the database unchanged.
func TestFileRepositoryMTimeOutsideInt64NanosecondRange(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repo := database.NewFileRepository(db)
mtimes := []time.Time{
time.Date(2300, time.January, 1, 0, 0, 0, 123456789, time.UTC),
time.Date(1601, time.January, 1, 0, 0, 0, 987654321, time.UTC),
}
for _, mtime := range mtimes {
created := &database.File{
Path: types.FilePath("/created-" + mtime.Format(time.RFC3339Nano)),
MTime: mtime,
}
err := repo.Create(ctx, nil, created)
if err != nil {
t.Fatalf("failed to create file: %v", err)
}
batched := &database.File{
ID: types.NewFileID(),
Path: types.FilePath("/batched-" + mtime.Format(time.RFC3339Nano)),
MTime: mtime,
}
err = repo.CreateBatch(ctx, nil, []*database.File{batched})
if err != nil {
t.Fatalf("failed to batch create file: %v", err)
}
for _, path := range []types.FilePath{created.Path, batched.Path} {
retrieved, err := repo.GetByPath(ctx, path.String())
if err != nil {
t.Fatalf("failed to get file: %v", err)
}
if !retrieved.MTime.Equal(mtime) {
t.Errorf("%s: mtime got %v, want %v",
path, retrieved.MTime, mtime)
}
}
}
}
// A file already in the index and rewritten within the same second must get
// its new nanoseconds stored, through both Create and CreateBatch.
func TestFileRepositoryUpsertMTimeInSameSecond(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repo := database.NewFileRepository(db)
indexed := time.Date(2026, time.October, 7, 12, 0, 0, 100000000, time.UTC)
rewritten := indexed.Add(800 * time.Millisecond)
tests := []struct {
name string
upsert func(file *database.File) error
}{
{"Create", func(file *database.File) error {
return repo.Create(ctx, nil, file)
}},
{"CreateBatch", func(file *database.File) error {
return repo.CreateBatch(ctx, nil, []*database.File{file})
}},
}
for _, tt := range tests {
file := &database.File{
ID: types.NewFileID(),
Path: types.FilePath("/" + tt.name),
MTime: indexed,
}
err := tt.upsert(file)
if err != nil {
t.Fatalf("%s: failed to create file: %v", tt.name, err)
}
file.MTime = rewritten
err = tt.upsert(file)
if err != nil {
t.Fatalf("%s: failed to update file: %v", tt.name, err)
}
retrieved, err := repo.GetByPath(ctx, file.Path.String())
if err != nil {
t.Fatalf("%s: failed to get file: %v", tt.name, err)
}
if !retrieved.MTime.Equal(rewritten) {
t.Errorf("%s: mtime got %v, want %v",
tt.name, retrieved.MTime, rewritten)
}
}
}
func TestFileRepositoryTransaction(t *testing.T) { func TestFileRepositoryTransaction(t *testing.T) {
t.Parallel() t.Parallel()
internal/database/models.go
+3 -4
View File
@@ -14,8 +14,7 @@ type File struct {
ID types.FileID // UUID primary key ID types.FileID // UUID primary key
Path types.FilePath // Absolute path of the file Path types.FilePath // Absolute path of the file
// SourcePath is the source directory this file came from (used for // SourcePath is the source directory this file came from.
// restore path stripping).
SourcePath types.SourcePath SourcePath types.SourcePath
MTime time.Time MTime time.Time
Size int64 Size int64
@@ -99,8 +98,8 @@ type Snapshot struct {
StartedAt time.Time StartedAt time.Time
CompletedAt *time.Time // nil if still in progress CompletedAt *time.Time // nil if still in progress
FileCount int64 FileCount int64
ChunkCount int64 ChunkCount int64 // Chunks this snapshot stored that were not stored before
BlobCount int64 BlobCount int64 // Blobs this snapshot created
TotalSize int64 // Total size of all referenced files TotalSize int64 // Total size of all referenced files
// BlobSize is the total size of all referenced blobs (compressed and // BlobSize is the total size of all referenced blobs (compressed and
internal/database/repository_comprehensive_test.go
+2 -2
View File
@@ -824,7 +824,7 @@ func TestTransactionIsolation(t *testing.T) {
} }
// Verify the file was not created (transaction rolled back) // Verify the file was not created (transaction rolled back)
files, err := repos.Files.ListByPrefix(ctx, "/tx-test") files, err := repos.Files.ListUnderPath(ctx, "/tx-test.txt")
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -916,7 +916,7 @@ func TestConcurrentOrphanedCleanup(t *testing.T) {
} }
// Verify correct files were deleted // Verify correct files were deleted
files, err := repos.Files.ListByPrefix(ctx, "/concurrent-") files, err := repos.Files.ListAll(ctx)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
internal/database/repository_debug_test.go
+1 -1
View File
@@ -147,7 +147,7 @@ func TestOrphanedFileCleanupDebug(t *testing.T) {
t.Logf("Files count after cleanup: %d", count) t.Logf("Files count after cleanup: %d", count)
// List remaining files // List remaining files
files, err := repos.Files.ListByPrefix(ctx, "/") files, err := repos.Files.ListUnderPath(ctx, "/")
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
internal/database/repository_edge_cases_test.go
+34 -27
View File
@@ -3,6 +3,7 @@ package database
import ( import (
"context" "context"
"database/sql"
"fmt" "fmt"
"strings" "strings"
"testing" "testing"
@@ -367,7 +368,7 @@ func verifyBlobNullUploadTS(
} }
// createLargeDatasetFiles creates fileCount files and adds every other // createLargeDatasetFiles creates fileCount files and adds every other
// one to the snapshot. // one to the snapshot, in one transaction as a backup writes them.
func createLargeDatasetFiles( func createLargeDatasetFiles(
t *testing.T, t *testing.T,
repos *Repositories, repos *Repositories,
@@ -376,31 +377,38 @@ func createLargeDatasetFiles(
) { ) {
t.Helper() t.Helper()
ctx := context.Background()
start := time.Now() start := time.Now()
for i := range fileCount { err := repos.WithTx(context.Background(),
file := &File{ func(ctx context.Context, tx *sql.Tx) error {
Path: types.FilePath(fmt.Sprintf("/large/file%05d.txt", i)), for i := range fileCount {
MTime: time.Now(), file := &File{
Size: int64(i * 1024), Path: types.FilePath(fmt.Sprintf("/large/file%05d.txt", i)),
Mode: 0644, MTime: time.Now(),
UID: uint32(1000 + (i % 10)), Size: int64(i * 1024),
GID: uint32(1000 + (i % 10)), Mode: 0644,
} UID: uint32(1000 + (i % 10)),
GID: uint32(1000 + (i % 10)),
}
err := repos.Files.Create(ctx, nil, file) err := repos.Files.Create(ctx, tx, file)
if err != nil { if err != nil {
t.Fatalf("failed to create file %d: %v", i, err) return fmt.Errorf("creating file %d: %w", i, err)
} }
// Add half to snapshot // Add half to snapshot
if i%2 == 0 { if i%2 == 0 {
err = repos.Snapshots.AddFileByID(ctx, nil, snapshotID, file.ID) err = repos.Snapshots.AddFileByID(ctx, tx, snapshotID, file.ID)
if err != nil { if err != nil {
t.Fatal(err) return err
}
}
} }
}
return nil
})
if err != nil {
t.Fatal(err)
} }
t.Logf("Created %d files in %v", fileCount, time.Since(start)) t.Logf("Created %d files in %v", fileCount, time.Since(start))
@@ -442,12 +450,12 @@ func TestLargeDatasets(t *testing.T) {
createLargeDatasetFiles(t, repos, snapshot.ID.String(), fileCount) createLargeDatasetFiles(t, repos, snapshot.ID.String(), fileCount)
}) })
// Test ListByPrefix performance // Test ListUnderPath performance
//nolint:paralleltest // phases share one database and are order-dependent //nolint:paralleltest // phases share one database and are order-dependent
t.Run("list by prefix performance", func(t *testing.T) { t.Run("list under path performance", func(t *testing.T) {
start := time.Now() start := time.Now()
files, err := repos.Files.ListByPrefix(ctx, "/large/") files, err := repos.Files.ListUnderPath(ctx, "/large/")
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -472,7 +480,7 @@ func TestLargeDatasets(t *testing.T) {
t.Logf("Cleaned up orphaned files in %v", time.Since(start)) t.Logf("Cleaned up orphaned files in %v", time.Since(start))
// Verify correct number remain // Verify correct number remain
files, err := repos.Files.ListByPrefix(ctx, "/large/") files, err := repos.Files.ListUnderPath(ctx, "/large/")
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -606,8 +614,7 @@ func TestTimezoneHandling(t *testing.T) {
t.Skip("timezone not available") t.Skip("timezone not available")
} }
// Use Truncate to remove sub-second precision since we store as Unix timestamps nyTime := time.Now().In(loc)
nyTime := time.Now().In(loc).Truncate(time.Second)
file := &File{ file := &File{
Path: "/timezone-test.txt", Path: "/timezone-test.txt",
MTime: nyTime, MTime: nyTime,
internal/database/schema/001.sql
+3 -2
View File
@@ -5,8 +5,9 @@
CREATE TABLE IF NOT EXISTS files ( CREATE TABLE IF NOT EXISTS files (
id TEXT PRIMARY KEY, -- UUID id TEXT PRIMARY KEY, -- UUID
path TEXT NOT NULL UNIQUE, path TEXT NOT NULL UNIQUE,
source_path TEXT NOT NULL DEFAULT '', -- The source directory this file came from (for restore path stripping) source_path TEXT NOT NULL DEFAULT '', -- The source directory this file came from
mtime INTEGER NOT NULL, mtime INTEGER NOT NULL, -- whole seconds since the Unix epoch
mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second, 0 to 999999999
size INTEGER NOT NULL, size INTEGER NOT NULL,
mode INTEGER NOT NULL, mode INTEGER NOT NULL,
uid INTEGER NOT NULL, uid INTEGER NOT NULL,
internal/database/snapshots.go
+28 -3
View File
@@ -127,6 +127,7 @@ func (r *SnapshotRepository) UpdateExtendedStats(
snapshotID string, snapshotID string,
blobUncompressedSize int64, blobUncompressedSize int64,
compressionLevel int, compressionLevel int,
uploadBytes int64,
uploadDurationMs int64, uploadDurationMs int64,
) error { ) error {
compressionRatio, err := r.extendedCompressionRatio( compressionRatio, err := r.extendedCompressionRatio(
@@ -141,7 +142,7 @@ func (r *SnapshotRepository) UpdateExtendedStats(
SET blob_uncompressed_size = ?, SET blob_uncompressed_size = ?,
compression_ratio = ?, compression_ratio = ?,
compression_level = ?, compression_level = ?,
upload_bytes = blob_size, upload_bytes = ?,
upload_duration_ms = ? upload_duration_ms = ?
WHERE id = ? WHERE id = ?
` `
@@ -149,11 +150,11 @@ func (r *SnapshotRepository) UpdateExtendedStats(
if tx != nil { if tx != nil {
_, err = tx.ExecContext(ctx, query, _, err = tx.ExecContext(ctx, query,
blobUncompressedSize, compressionRatio, compressionLevel, blobUncompressedSize, compressionRatio, compressionLevel,
uploadDurationMs, snapshotID) uploadBytes, uploadDurationMs, snapshotID)
} else { } else {
_, err = r.db.ExecWithLog(ctx, query, _, err = r.db.ExecWithLog(ctx, query,
blobUncompressedSize, compressionRatio, compressionLevel, blobUncompressedSize, compressionRatio, compressionLevel,
uploadDurationMs, snapshotID) uploadBytes, uploadDurationMs, snapshotID)
} }
if err != nil { if err != nil {
@@ -543,6 +544,30 @@ func (r *SnapshotRepository) GetSnapshotTotalCompressedSize(
return totalSize, nil return totalSize, nil
} }
// GetSnapshotBlobSizes returns the total compressed and uncompressed sizes
// of all blobs referenced by a snapshot.
func (r *SnapshotRepository) GetSnapshotBlobSizes(
ctx context.Context, snapshotID string,
) (int64, int64, error) {
query := `
SELECT COALESCE(SUM(b.compressed_size), 0),
COALESCE(SUM(b.uncompressed_size), 0)
FROM snapshot_blobs sb
JOIN blobs b ON sb.blob_hash = b.blob_hash
WHERE sb.snapshot_id = ?
`
var compressed, uncompressed int64
err := r.db.conn.QueryRowContext(ctx, query, snapshotID).Scan(
&compressed, &uncompressed)
if err != nil {
return 0, 0, fmt.Errorf("querying snapshot blob sizes: %w", err)
}
return compressed, uncompressed, nil
}
// GetSnapshotUncompressedChunkSize returns the sum of plaintext sizes of all unique // GetSnapshotUncompressedChunkSize returns the sum of plaintext sizes of all unique
// chunks referenced by a snapshot (via snapshot_files → file_chunks → chunks). // chunks referenced by a snapshot (via snapshot_files → file_chunks → chunks).
func (r *SnapshotRepository) GetSnapshotUncompressedChunkSize( func (r *SnapshotRepository) GetSnapshotUncompressedChunkSize(
internal/database/snapshots_test.go
+59
View File
@@ -145,6 +145,65 @@ func TestSnapshotRepositoryUpdateCounts(t *testing.T) {
} }
} }
// GetSnapshotBlobSizes totals the blobs the snapshot references, and only
// those.
func TestSnapshotRepositoryGetSnapshotBlobSizes(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repos := database.NewRepositories(db)
snapshot := &database.Snapshot{
ID: "2024-01-03T12:00:00Z",
Hostname: testHostname,
VaultikVersion: testVersion,
StartedAt: time.Now().Truncate(time.Second),
}
err := repos.Snapshots.Create(ctx, nil, snapshot)
if err != nil {
t.Fatalf("failed to create snapshot: %v", err)
}
blobs := []*database.Blob{
{Hash: "referenced-1", CompressedSize: 10, UncompressedSize: 100},
{Hash: "referenced-2", CompressedSize: 20, UncompressedSize: 200},
{Hash: "unreferenced", CompressedSize: 40, UncompressedSize: 400},
}
for _, blob := range blobs {
blob.ID = types.NewBlobID()
blob.CreatedTS = time.Now().Truncate(time.Second)
err = repos.Blobs.Create(ctx, nil, blob)
if err != nil {
t.Fatalf("failed to create blob %s: %v", blob.Hash, err)
}
}
for _, blob := range blobs[:2] {
err = repos.Snapshots.AddBlob(ctx, nil, snapshot.ID.String(),
blob.ID, blob.Hash)
if err != nil {
t.Fatalf("failed to add blob %s to snapshot: %v", blob.Hash, err)
}
}
compressed, uncompressed, err := repos.Snapshots.GetSnapshotBlobSizes(
ctx, snapshot.ID.String())
if err != nil {
t.Fatalf("failed to get snapshot blob sizes: %v", err)
}
if compressed != 30 || uncompressed != 300 {
t.Errorf("blob sizes: got %d and %d, want 30 and 300",
compressed, uncompressed)
}
}
func TestSnapshotRepositoryListRecent(t *testing.T) { func TestSnapshotRepositoryListRecent(t *testing.T) {
t.Parallel() t.Parallel()
internal/database/uploads.go
-16
View File
@@ -158,19 +158,3 @@ type UploadStats struct {
MinDurationMs int64 MinDurationMs int64
MaxDurationMs int64 MaxDurationMs int64
} }
// GetCountBySnapshot returns the count of uploads for a specific snapshot
func (r *UploadRepository) GetCountBySnapshot(
ctx context.Context, snapshotID string,
) (int64, error) {
query := `SELECT COUNT(*) FROM uploads WHERE snapshot_id = ?`
var count int64
err := r.conn.QueryRowContext(ctx, query, snapshotID).Scan(&count)
if err != nil {
return 0, err
}
return count, nil
}
internal/pidlock/pidlock.go
+67 -52
View File
@@ -10,15 +10,18 @@ import (
"path/filepath" "path/filepath"
"strconv" "strconv"
"strings" "strings"
"syscall"
"golang.org/x/sys/unix"
) )
// ErrAlreadyRunning indicates another vaultik instance is running. // ErrAlreadyRunning indicates another vaultik instance is running.
var ErrAlreadyRunning = errors.New("another vaultik instance is already running") var ErrAlreadyRunning = errors.New("another vaultik instance is already running")
// Lock represents an acquired PID lock. // Lock represents an acquired PID lock: an flock(2) on the PID file,
// held while the file stays open. The kernel drops it when the process
// exits, however it exits, so a crashed run never leaves the lock held.
type Lock struct { type Lock struct {
path string file *os.File
} }
const ( const (
@@ -29,10 +32,9 @@ const (
) )
// Acquire attempts to acquire a PID lock in the specified directory. // Acquire attempts to acquire a PID lock in the specified directory.
// If the lock file exists and the process is still running, it returns // If another process holds the lock, it returns ErrAlreadyRunning with
// ErrAlreadyRunning with details about the existing process. // that process's PID. On success, it writes the current PID to the lock
// On success, it writes the current PID to the lock file and returns // file and returns a Lock that must be released with Release().
// a Lock that must be released with Release().
func Acquire(lockDir string) (*Lock, error) { func Acquire(lockDir string) (*Lock, error) {
// Ensure lock directory exists // Ensure lock directory exists
err := os.MkdirAll(lockDir, lockDirPerm) err := os.MkdirAll(lockDir, lockDirPerm)
@@ -42,56 +44,82 @@ func Acquire(lockDir string) (*Lock, error) {
lockPath := filepath.Join(lockDir, "vaultik.pid") lockPath := filepath.Join(lockDir, "vaultik.pid")
// Check for existing lock // No O_TRUNC: the file may hold the PID of the process that has the
existingPID, err := readPIDFile(lockPath) // lock, which the error below reports.
if err == nil { file, err := os.OpenFile( //nolint:gosec // G304: path is our own lock file
// Lock file exists, check if process is running lockPath, os.O_RDWR|os.O_CREATE, pidFilePerm)
if isProcessRunning(existingPID) {
return nil, fmt.Errorf("%w (PID %d)", ErrAlreadyRunning, existingPID)
}
// Process is not running, stale lock file - we can take over
}
// Write our PID
pid := os.Getpid()
err = os.WriteFile(lockPath, []byte(strconv.Itoa(pid)), pidFilePerm)
if err != nil { if err != nil {
return nil, fmt.Errorf("writing PID file: %w", err) return nil, fmt.Errorf("opening PID file: %w", err)
} }
return &Lock{path: lockPath}, nil err = unix.Flock(int(file.Fd()), unix.LOCK_EX|unix.LOCK_NB)
if err != nil {
_ = file.Close()
if errors.Is(err, unix.EWOULDBLOCK) {
return nil, alreadyRunningError(lockPath)
}
return nil, fmt.Errorf("locking PID file: %w", err)
}
err = writePID(file)
if err != nil {
_ = file.Close()
return nil, err
}
return &Lock{file: file}, nil
} }
// Release removes the PID lock file. // Release empties the PID file and closes it, which drops the lock.
// It is safe to call Release multiple times. // It is safe to call Release multiple times.
func (l *Lock) Release() error { func (l *Lock) Release() error {
if l == nil || l.path == "" { if l == nil || l.file == nil {
return nil return nil
} }
// Verify we still own the lock (our PID is in the file) file := l.file
existingPID, err := readPIDFile(l.path) l.file = nil
// Do not remove the file here. A process that opened it a moment
// earlier could then lock the removed file while another creates and
// locks a new one, and both would run.
truncateErr := file.Truncate(0)
closeErr := file.Close()
return errors.Join(truncateErr, closeErr)
}
// writePID replaces the contents of the locked PID file with the current
// PID.
func writePID(file *os.File) error {
err := file.Truncate(0)
if err != nil { if err != nil {
// File already gone or unreadable - that's fine return fmt.Errorf("truncating PID file: %w", err)
return nil //nolint:nilerr // unreadable lock file means nothing to release
} }
if existingPID != os.Getpid() { _, err = file.WriteAt([]byte(strconv.Itoa(os.Getpid())), 0)
// Someone else wrote to our lock file - don't remove it if err != nil {
return nil return fmt.Errorf("writing PID file: %w", err)
} }
err = os.Remove(l.path)
if err != nil && !os.IsNotExist(err) {
return fmt.Errorf("removing PID file: %w", err)
}
l.path = "" // Prevent double-release
return nil return nil
} }
// alreadyRunningError reports that another process holds the lock,
// naming its PID when the file holds one. The holder writes its PID just
// after it locks, so the file can briefly be empty.
func alreadyRunningError(lockPath string) error {
pid, err := readPIDFile(lockPath)
if err != nil {
return ErrAlreadyRunning
}
return fmt.Errorf("%w (PID %d)", ErrAlreadyRunning, pid)
}
// readPIDFile reads and parses the PID from a lock file. // readPIDFile reads and parses the PID from a lock file.
func readPIDFile(path string) (int, error) { func readPIDFile(path string) (int, error) {
data, err := os.ReadFile(path) //nolint:gosec // G304: path is our own lock file data, err := os.ReadFile(path) //nolint:gosec // G304: path is our own lock file
@@ -106,16 +134,3 @@ func readPIDFile(path string) (int, error) {
return pid, nil return pid, nil
} }
// isProcessRunning checks if a process with the given PID is running.
func isProcessRunning(pid int) bool {
process, err := os.FindProcess(pid)
if err != nil {
return false
}
// On Unix, FindProcess always succeeds. We need to send signal 0 to check.
err = process.Signal(syscall.Signal(0))
return err == nil
}
internal/pidlock/pidlock_test.go
+63 -3
View File
@@ -4,6 +4,7 @@ import (
"os" "os"
"path/filepath" "path/filepath"
"strconv" "strconv"
"sync"
"testing" "testing"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
@@ -33,9 +34,10 @@ func TestAcquireAndRelease(t *testing.T) {
err = lock.Release() err = lock.Release()
require.NoError(t, err) require.NoError(t, err)
// Verify PID file is gone // Verify PID file is empty
_, err = os.Stat(pidPath) data, err = os.ReadFile(pidPath) //nolint:gosec // G304: test's own temp file
assert.True(t, os.IsNotExist(err)) require.NoError(t, err)
assert.Empty(t, data)
} }
func TestAcquireBlocksSecondInstance(t *testing.T) { func TestAcquireBlocksSecondInstance(t *testing.T) {
@@ -55,6 +57,64 @@ func TestAcquireBlocksSecondInstance(t *testing.T) {
lock2, err := pidlock.Acquire(tmpDir) lock2, err := pidlock.Acquire(tmpDir)
require.ErrorIs(t, err, pidlock.ErrAlreadyRunning) require.ErrorIs(t, err, pidlock.ErrAlreadyRunning)
assert.Nil(t, lock2) assert.Nil(t, lock2)
// Once the first lock is released, the next Acquire succeeds
require.NoError(t, lock1.Release())
lock3, err := pidlock.Acquire(tmpDir)
require.NoError(t, err)
require.NoError(t, lock3.Release())
}
// TestConcurrentAcquireAdmitsOne starts many Acquire calls at the same
// moment, as two cron entries firing together would, and checks that
// exactly one of them gets the lock.
func TestConcurrentAcquireAdmitsOne(t *testing.T) {
t.Parallel()
const callers = 50
tmpDir := t.TempDir()
start := make(chan struct{})
var (
mu sync.Mutex
acquired []*pidlock.Lock
failures []error
wg sync.WaitGroup
)
for range callers {
wg.Go(func() {
<-start
lock, err := pidlock.Acquire(tmpDir)
mu.Lock()
defer mu.Unlock()
if err != nil {
failures = append(failures, err)
return
}
acquired = append(acquired, lock)
})
}
close(start)
wg.Wait()
for _, lock := range acquired {
require.NoError(t, lock.Release())
}
assert.Len(t, acquired, 1, "exactly one caller should hold the lock")
for _, err := range failures {
require.ErrorIs(t, err, pidlock.ErrAlreadyRunning)
}
} }
func TestAcquireWithStaleLock(t *testing.T) { func TestAcquireWithStaleLock(t *testing.T) {
internal/s3/client.go
+34 -6
View File
@@ -6,6 +6,7 @@ import (
"context" "context"
"errors" "errors"
"io" "io"
"strings"
"sync/atomic" "sync/atomic"
"github.com/aws/aws-sdk-go-v2/aws" "github.com/aws/aws-sdk-go-v2/aws"
@@ -26,10 +27,14 @@ type Client struct {
bucket string bucket string
prefix string prefix string
endpoint string endpoint string
partSize int64
} }
// Config contains S3 client configuration. // Config contains S3 client configuration.
// All fields are required except Prefix, which defaults to an empty string. // All fields are required except Prefix, which defaults to an empty string,
// and PartSize, where zero means the SDK default of 5 MiB.
// A non-empty Prefix is joined to every key with one "/", whether or not
// it ends with one.
// The Endpoint field should include the protocol (http:// or https://). // The Endpoint field should include the protocol (http:// or https://).
type Config struct { type Config struct {
Endpoint string Endpoint string
@@ -38,6 +43,9 @@ type Config struct {
AccessKeyID string AccessKeyID string
SecretAccessKey string SecretAccessKey string
Region string Region string
// PartSize is the size in bytes of each part of a multipart upload.
// An upload too large for S3's limit of 10,000 parts gets larger parts.
PartSize int64
} }
// nopLogger is a logger that discards all output. // nopLogger is a logger that discards all output.
@@ -75,11 +83,19 @@ func NewClient(ctx context.Context, cfg Config) (*Client, error) {
s3Client := s3.NewFromConfig(awsCfg, s3Opts) s3Client := s3.NewFromConfig(awsCfg, s3Opts)
// Every method below builds a key as prefix + key, so the prefix
// must carry its own trailing "/".
prefix := strings.TrimRight(cfg.Prefix, "/")
if prefix != "" {
prefix += "/"
}
return &Client{ return &Client{
s3Client: s3Client, s3Client: s3Client,
bucket: cfg.Bucket, bucket: cfg.Bucket,
prefix: cfg.Prefix, prefix: prefix,
endpoint: cfg.Endpoint, endpoint: cfg.Endpoint,
partSize: cfg.PartSize,
}, nil }, nil
} }
@@ -113,12 +129,9 @@ func (c *Client) PutObjectWithProgress(
) error { ) error {
fullKey := c.prefix + key fullKey := c.prefix + key
// uploadPartSize is 10MB for better progress granularity.
const uploadPartSize = 10 * 1024 * 1024
// Create an uploader with the S3 client // Create an uploader with the S3 client
uploader := manager.NewUploader(c.s3Client, func(u *manager.Uploader) { uploader := manager.NewUploader(c.s3Client, func(u *manager.Uploader) {
u.PartSize = uploadPartSize u.PartSize = uploadPartSize(c.partSize, size)
}) })
// Create a progress reader that tracks upload progress // Create a progress reader that tracks upload progress
@@ -139,6 +152,21 @@ func (c *Client) PutObjectWithProgress(
return err return err
} }
// uploadPartSize returns the part size for an upload of size bytes: the
// configured part size (the SDK default when zero), raised where needed so
// the upload fits in S3's limit of 10,000 parts. The uploader cannot raise
// it itself, because it cannot seek the progress reader to learn its size.
func uploadPartSize(configured, size int64) int64 {
if configured == 0 {
configured = manager.DefaultUploadPartSize
}
maxParts := int64(manager.MaxUploadParts)
smallestThatFits := (size + maxParts - 1) / maxParts // rounded up
return max(configured, smallestThatFits)
}
// GetObject downloads an object from S3 with the specified key. // GetObject downloads an object from S3 with the specified key.
// The key is automatically prefixed with the configured prefix. // The key is automatically prefixed with the configured prefix.
// Returns a ReadCloser containing the object data. The caller must // Returns a ReadCloser containing the object data. The caller must
internal/s3/client_internal_test.go
+55
View File
@@ -0,0 +1,55 @@
package s3
import "testing"
// TestUploadPartSize checks that an upload too large for 10,000 parts of the
// configured size gets parts just large enough to fit in 10,000.
func TestUploadPartSize(t *testing.T) {
t.Parallel()
const mib = 1024 * 1024
tests := []struct {
name string
configured int64
size int64
want int64
}{
{
name: "an upload that fits keeps the configured size",
configured: 5 * mib,
size: 10 * 1024 * mib,
want: 5 * mib,
},
{
name: "exactly 10,000 parts keeps the configured size",
configured: 6 * mib,
size: 10_000 * 6 * mib,
want: 6 * mib,
},
{
name: "one byte more than 10,000 parts adds a byte to each",
configured: 6 * mib,
size: 10_000*6*mib + 1,
want: 6*mib + 1,
},
{
name: "zero means the SDK default of 5MiB",
configured: 0,
size: 1,
want: 5 * mib,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
got := uploadPartSize(tt.configured, tt.size)
if got != tt.want {
t.Errorf("uploadPartSize(%d, %d) = %d, want %d",
tt.configured, tt.size, got, tt.want)
}
})
}
}
internal/s3/module.go
+1
View File
@@ -28,6 +28,7 @@ func provideClient(lc fx.Lifecycle, cfg *config.Config) (*Client, error) {
AccessKeyID: cfg.S3.AccessKeyID, AccessKeyID: cfg.S3.AccessKeyID,
SecretAccessKey: cfg.S3.SecretAccessKey, SecretAccessKey: cfg.S3.SecretAccessKey,
Region: cfg.S3.Region, Region: cfg.S3.Region,
PartSize: cfg.S3.PartSize.Int64(),
}) })
if err != nil { if err != nil {
return nil, err return nil, err
internal/snapshot/backup_test.go
+1 -1
View File
@@ -71,7 +71,7 @@ func verifyBackupFiles(
) { ) {
t.Helper() t.Helper()
files, err := repos.Files.ListByPrefix(ctx, "") files, err := repos.Files.ListAll(ctx)
if err != nil { if err != nil {
t.Fatalf("Failed to list files: %v", err) t.Fatalf("Failed to list files: %v", err)
} }
internal/snapshot/progress.go
-4
View File
@@ -66,7 +66,6 @@ type ProgressStats struct {
BlobsCreated atomic.Int64 BlobsCreated atomic.Int64
BlobsUploaded atomic.Int64 BlobsUploaded atomic.Int64
BytesUploaded atomic.Int64 BytesUploaded atomic.Int64
UploadDurationMs atomic.Int64 // Total milliseconds spent uploading
CurrentFile atomic.Value // stores string CurrentFile atomic.Value // stores string
TotalSize atomic.Int64 // Total size to process (set after scan phase) TotalSize atomic.Int64 // Total size to process (set after scan phase)
TotalFiles atomic.Int64 // Total files to process in phase 2 TotalFiles atomic.Int64 // Total files to process in phase 2
@@ -231,9 +230,6 @@ func (pr *ProgressReporter) ReportUploadComplete(
// Clear current upload // Clear current upload
pr.stats.CurrentUpload.Store((*UploadInfo)(nil)) pr.stats.CurrentUpload.Store((*UploadInfo)(nil))
// Add to total upload duration
pr.stats.UploadDurationMs.Add(duration.Milliseconds())
// Calculate speed // Calculate speed
if duration < time.Millisecond { if duration < time.Millisecond {
duration = time.Millisecond duration = time.Millisecond
internal/snapshot/scanner.go
+57 -85
View File
@@ -58,8 +58,8 @@ type Scanner struct {
compressionLevel int compressionLevel int
ageRecipient string ageRecipient string
snapshotID string // Current snapshot being processed snapshotID string // Current snapshot being processed
// currentSourcePath is the source directory being scanned (used for // currentSourcePath is the source directory being scanned, stored with
// restore path stripping). // each file record.
currentSourcePath string currentSourcePath string
exclude []string // Glob patterns for files/directories to exclude exclude []string // Glob patterns for files/directories to exclude
compiledExclude []compiledPattern // Compiled glob patterns compiledExclude []compiledPattern // Compiled glob patterns
@@ -92,9 +92,6 @@ type Scanner struct {
// Mutex for coordinating blob creation // Mutex for coordinating blob creation
packerMu sync.Mutex // Blocks chunk production during blob creation packerMu sync.Mutex // Blocks chunk production during blob creation
// Context for cancellation
scanCtx context.Context //nolint:containedctx // set per-Scan for packer callbacks
} }
// Periodic status output intervals and thresholds for the scan and // Periodic status output intervals and thresholds for the scan and
@@ -134,18 +131,23 @@ type ScannerConfig struct {
SkipErrors bool SkipErrors bool
} }
// ScanResult contains the results of a scan operation // ScanResult contains the results of a scan operation. Files and bytes
// are counted per file: BytesScanned is the size of the new and changed
// files, BytesSkipped that of the unchanged ones.
type ScanResult struct { type ScanResult struct {
FilesScanned int FilesScanned int
FilesSkipped int FilesSkipped int
FilesDeleted int FilesDeleted int
BytesScanned int64 BytesScanned int64
BytesSkipped int64 BytesSkipped int64
BytesDeleted int64 BytesDeleted int64
ChunksCreated int ChunksCreated int
BlobsCreated int BlobsCreated int
StartTime time.Time BlobsUploaded int
EndTime time.Time BytesUploaded int64
UploadDuration time.Duration
StartTime time.Time
EndTime time.Time
} }
// NewScanner creates a new scanner instance // NewScanner creates a new scanner instance
@@ -209,9 +211,7 @@ func (s *Scanner) Scan(
ctx context.Context, path string, snapshotID string, ctx context.Context, path string, snapshotID string,
) (*ScanResult, error) { ) (*ScanResult, error) {
s.snapshotID = snapshotID s.snapshotID = snapshotID
// Store source path for file records (used during restore)
s.currentSourcePath = path s.currentSourcePath = path
s.scanCtx = ctx
result := &ScanResult{ result := &ScanResult{
StartTime: time.Now().UTC(), StartTime: time.Now().UTC(),
} }
@@ -219,17 +219,13 @@ func (s *Scanner) Scan(
// Set blob handler for concurrent upload // Set blob handler for concurrent upload
if s.storage != nil { if s.storage != nil {
log.Debug("Setting blob handler for storage uploads") log.Debug("Setting blob handler for storage uploads")
s.packer.SetBlobHandler(s.handleBlobReady) s.packer.SetBlobHandler(func(blobWithReader *blob.WithReader) error {
return s.handleBlobReady(ctx, blobWithReader, result)
})
} else { } else {
log.Debug("No storage configured, blobs will not be uploaded") log.Debug("No storage configured, blobs will not be uploaded")
} }
// Start progress reporting if enabled
if s.progress != nil {
s.progress.Start()
defer s.progress.Stop()
}
// Phase 0: Repair any state left by an interrupted previous run, then // Phase 0: Repair any state left by an interrupted previous run, then
// load known files and chunks from the database into memory for fast // load known files and chunks from the database into memory for fast
// lookup. // lookup.
@@ -294,13 +290,14 @@ func (s *Scanner) Scan(
log.Info("Phase 2/3: Skipping (no files need processing, metadata-only snapshot)") log.Info("Phase 2/3: Skipping (no files need processing, metadata-only snapshot)")
} }
// Finalize result with blob statistics result.EndTime = time.Now().UTC()
s.finalizeScanResult(ctx, result)
return result, nil return result, nil
} }
// GetProgress returns the progress reporter for this scanner // GetProgress returns the progress reporter for this scanner, or nil when
// progress is off. Scan neither starts nor stops it: the caller does,
// once for all the paths it scans, because a second Stop panics.
func (s *Scanner) GetProgress() *ProgressReporter { func (s *Scanner) GetProgress() *ProgressReporter {
return s.progress return s.progress
} }
@@ -435,35 +432,16 @@ func (s *Scanner) summarizeScanPhase(
s.ui.Completef("%s.", msg) s.ui.Completef("%s.", msg)
} }
// finalizeScanResult populates final blob statistics in the scan result // loadKnownFiles loads the known files at and beneath path from the
// by querying the packer and database for blob/upload counts // database into a map for fast lookup. Every loaded file the scan does
func (s *Scanner) finalizeScanResult(ctx context.Context, result *ScanResult) { // not find is counted as deleted. This avoids per-file database queries
blobs := s.packer.GetFinishedBlobs() // during the scan phase.
result.BlobsCreated += len(blobs)
// Query database for actual blob count created during this snapshot
// The database is authoritative, especially for concurrent blob uploads
// We count uploads rather than all snapshot_blobs to get only NEW blobs
if s.snapshotID != "" {
uploadCount, err := s.repos.Uploads.GetCountBySnapshot(ctx, s.snapshotID)
if err != nil {
log.Warn("Failed to query upload count from database", "error", err)
} else {
result.BlobsCreated = int(uploadCount)
}
}
result.EndTime = time.Now().UTC()
}
// loadKnownFiles loads all known files from the database into a map for fast lookup
// This avoids per-file database queries during the scan phase
func (s *Scanner) loadKnownFiles( func (s *Scanner) loadKnownFiles(
ctx context.Context, path string, ctx context.Context, path string,
) (map[string]*database.File, error) { ) (map[string]*database.File, error) {
files, err := s.repos.Files.ListByPrefix(ctx, path) files, err := s.repos.Files.ListUnderPath(ctx, path)
if err != nil { if err != nil {
return nil, fmt.Errorf("listing files by prefix: %w", err) return nil, fmt.Errorf("listing files under %s: %w", path, err)
} }
result := make(map[string]*database.File, len(files)) result := make(map[string]*database.File, len(files))
@@ -1219,9 +1197,8 @@ func (s *Scanner) checkFileInMemory(
} }
file := &database.File{ file := &database.File{
ID: fileID, ID: fileID,
Path: types.FilePath(path), Path: types.FilePath(path),
// Store source directory for restore path stripping
SourcePath: types.SourcePath(s.currentSourcePath), SourcePath: types.SourcePath(s.currentSourcePath),
MTime: info.ModTime(), MTime: info.ModTime(),
Size: info.Size(), Size: info.Size(),
@@ -1242,7 +1219,7 @@ func (s *Scanner) checkFileInMemory(
// Check if file has changed // Check if file has changed
if existingFile.Size != file.Size || if existingFile.Size != file.Size ||
existingFile.MTime.Unix() != file.MTime.Unix() || !existingFile.MTime.Equal(file.MTime) ||
existingFile.Mode != file.Mode || existingFile.Mode != file.Mode ||
existingFile.UID != file.UID || existingFile.UID != file.UID ||
existingFile.GID != file.GID { existingFile.GID != file.GID {
@@ -1513,24 +1490,24 @@ func (s *Scanner) finalizeProcessPhase(ctx context.Context, result *ScanResult)
} }
// handleBlobReady is called by the packer when a blob is finalized // handleBlobReady is called by the packer when a blob is finalized
func (s *Scanner) handleBlobReady(blobWithReader *blob.WithReader) error { func (s *Scanner) handleBlobReady(
ctx context.Context, blobWithReader *blob.WithReader, result *ScanResult,
) error {
startTime := time.Now().UTC() startTime := time.Now().UTC()
finishedBlob := blobWithReader.FinishedBlob finishedBlob := blobWithReader.FinishedBlob
result.BlobsCreated++
if s.progress != nil { if s.progress != nil {
s.progress.ReportUploadStart(finishedBlob.Hash, finishedBlob.Compressed) s.progress.ReportUploadStart(finishedBlob.Hash, finishedBlob.Compressed)
s.progress.GetStats().BlobsCreated.Add(1) s.progress.GetStats().BlobsCreated.Add(1)
} }
ctx := s.scanCtx
if ctx == nil {
ctx = context.Background()
}
blobPath := fmt.Sprintf("blobs/%s/%s/%s", blobPath := fmt.Sprintf("blobs/%s/%s/%s",
finishedBlob.Hash[:2], finishedBlob.Hash[2:4], finishedBlob.Hash) finishedBlob.Hash[:2], finishedBlob.Hash[2:4], finishedBlob.Hash)
blobExists, err := s.uploadBlobIfNeeded(ctx, blobPath, blobWithReader, startTime) blobExists, err := s.uploadBlobIfNeeded(
ctx, blobPath, blobWithReader, startTime, result)
if err != nil { if err != nil {
s.cleanupBlobTempFile(blobWithReader) s.cleanupBlobTempFile(blobWithReader)
@@ -1565,6 +1542,7 @@ func (s *Scanner) uploadBlobIfNeeded(
blobPath string, blobPath string,
blobWithReader *blob.WithReader, blobWithReader *blob.WithReader,
startTime time.Time, startTime time.Time,
result *ScanResult,
) (bool, error) { ) (bool, error) {
finishedBlob := blobWithReader.FinishedBlob finishedBlob := blobWithReader.FinishedBlob
@@ -1600,6 +1578,10 @@ func (s *Scanner) uploadBlobIfNeeded(
uploadDuration := time.Since(startTime) uploadDuration := time.Since(startTime)
uploadSpeedBps := float64(finishedBlob.Compressed) / uploadDuration.Seconds() uploadSpeedBps := float64(finishedBlob.Compressed) / uploadDuration.Seconds()
result.BlobsUploaded++
result.BytesUploaded += finishedBlob.Compressed
result.UploadDuration += uploadDuration
s.ui.Completef("Uploaded blob %s (%s) in %s at %s.", s.ui.Completef("Uploaded blob %s (%s) in %s at %s.",
s.ui.Hex(finishedBlob.Hash), s.ui.Hex(finishedBlob.Hash),
s.ui.Size(finishedBlob.Compressed), s.ui.Size(finishedBlob.Compressed),
@@ -1815,9 +1797,9 @@ func (s *Scanner) processFileStreaming(
size: chunk.Size, size: chunk.Size,
}) })
s.updateChunkStats(chunkExists, chunk.Size, result)
if !chunkExists { if !chunkExists {
s.updateChunkStats(chunk.Size, result)
err := s.addChunkToPacker(ctx, chunk) err := s.addChunkToPacker(ctx, chunk)
if err != nil { if err != nil {
// Mark as a packer error so --skip-errors cannot swallow it: // Mark as a packer error so --skip-errors cannot swallow it:
@@ -1845,26 +1827,16 @@ func (s *Scanner) processFileStreaming(
return nil return nil
} }
// updateChunkStats updates scan result and progress stats for a processed chunk // updateChunkStats counts a chunk that was not already stored. The scan
func (s *Scanner) updateChunkStats( // result's file counts, BytesScanned and BytesSkipped are not touched
chunkExists bool, chunkSize int64, result *ScanResult, // here: the scan phase counts each file once.
) { func (s *Scanner) updateChunkStats(chunkSize int64, result *ScanResult) {
if chunkExists { result.ChunksCreated++
result.FilesSkipped++
result.BytesSkipped += chunkSize if s.progress != nil {
if s.progress != nil { s.progress.GetStats().ChunksCreated.Add(1)
s.progress.GetStats().BytesSkipped.Add(chunkSize) s.progress.GetStats().BytesProcessed.Add(chunkSize)
} s.progress.UpdateChunkingActivity()
} else {
result.ChunksCreated++
result.BytesScanned += chunkSize
if s.progress != nil {
s.progress.GetStats().ChunksCreated.Add(1)
s.progress.GetStats().BytesProcessed.Add(chunkSize)
s.progress.UpdateChunkingActivity()
}
} }
} }
internal/snapshot/scanner_test.go
+1 -1
View File
@@ -71,7 +71,7 @@ func verifySimpleScanDatabase(
t.Helper() t.Helper()
// Verify files in database - includes regular files and directories // Verify files in database - includes regular files and directories
files, err := repos.Files.ListByPrefix(ctx, "/source") files, err := repos.Files.ListUnderPath(ctx, "/source")
if err != nil { if err != nil {
t.Fatalf("failed to list files: %v", err) t.Fatalf("failed to list files: %v", err)
} }
internal/snapshot/snapshot.go
+14 -28
View File
@@ -105,17 +105,21 @@ func (sm *SnapshotManager) CreateSnapshot(
return sm.CreateSnapshotWithName(ctx, hostname, "", version, gitRevision) return sm.CreateSnapshotWithName(ctx, hostname, "", version, gitRevision)
} }
// ShortHostname returns hostname up to its first dot. A snapshot ID starts
// with this form, while the snapshots table stores the full hostname.
func ShortHostname(hostname string) string {
short, _, _ := strings.Cut(hostname, ".")
return short
}
// CreateSnapshotWithName creates a new snapshot record with an optional // CreateSnapshotWithName creates a new snapshot record with an optional
// snapshot name. The snapshot ID format is: hostname_name_timestamp or // snapshot name. The snapshot ID format is: hostname_name_timestamp or
// hostname_timestamp if name is empty. // hostname_timestamp if name is empty.
func (sm *SnapshotManager) CreateSnapshotWithName( func (sm *SnapshotManager) CreateSnapshotWithName(
ctx context.Context, hostname, name, version, gitRevision string, ctx context.Context, hostname, name, version, gitRevision string,
) (string, error) { ) (string, error) {
// Use short hostname (strip domain if present) shortHostname := ShortHostname(hostname)
shortHostname := hostname
if before, _, ok := strings.Cut(hostname, "."); ok {
shortHostname = before
}
// Build snapshot ID with optional name // Build snapshot ID with optional name
timestamp := time.Now().UTC().Format("2006-01-02T15:04:05Z") timestamp := time.Now().UTC().Format("2006-01-02T15:04:05Z")
@@ -154,26 +158,6 @@ func (sm *SnapshotManager) CreateSnapshotWithName(
return snapshotID, nil return snapshotID, nil
} }
// UpdateSnapshotStats updates the statistics for a snapshot during backup
func (sm *SnapshotManager) UpdateSnapshotStats(
ctx context.Context, snapshotID string, stats BackupStats,
) error {
err := sm.repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
return sm.repos.Snapshots.UpdateCounts(ctx, tx, snapshotID,
int64(stats.FilesScanned),
int64(stats.ChunksCreated),
int64(stats.BlobsCreated),
stats.BytesScanned,
stats.BytesUploaded,
)
})
if err != nil {
return fmt.Errorf("updating snapshot stats: %w", err)
}
return nil
}
// UpdateSnapshotStatsExtended updates snapshot statistics with extended metrics. // UpdateSnapshotStatsExtended updates snapshot statistics with extended metrics.
// This includes compression level, uncompressed blob size, and upload duration. // This includes compression level, uncompressed blob size, and upload duration.
func (sm *SnapshotManager) UpdateSnapshotStatsExtended( func (sm *SnapshotManager) UpdateSnapshotStatsExtended(
@@ -185,8 +169,8 @@ func (sm *SnapshotManager) UpdateSnapshotStatsExtended(
int64(stats.FilesScanned), int64(stats.FilesScanned),
int64(stats.ChunksCreated), int64(stats.ChunksCreated),
int64(stats.BlobsCreated), int64(stats.BlobsCreated),
stats.BytesScanned, stats.TotalSize,
stats.BytesUploaded, stats.BlobSize,
) )
if err != nil { if err != nil {
return err return err
@@ -196,6 +180,7 @@ func (sm *SnapshotManager) UpdateSnapshotStatsExtended(
return sm.repos.Snapshots.UpdateExtendedStats(ctx, tx, snapshotID, return sm.repos.Snapshots.UpdateExtendedStats(ctx, tx, snapshotID,
stats.BlobUncompressedSize, stats.BlobUncompressedSize,
stats.CompressionLevel, stats.CompressionLevel,
stats.BytesUploaded,
stats.UploadDurationMs, stats.UploadDurationMs,
) )
}) })
@@ -890,7 +875,7 @@ func (sm *SnapshotManager) getFileSize(path string) int64 {
// BackupStats contains statistics from a backup operation // BackupStats contains statistics from a backup operation
type BackupStats struct { type BackupStats struct {
FilesScanned int FilesScanned int
BytesScanned int64 TotalSize int64 // Total size of all files examined
ChunksCreated int ChunksCreated int
BlobsCreated int BlobsCreated int
BytesUploaded int64 BytesUploaded int64
@@ -900,6 +885,7 @@ type BackupStats struct {
type ExtendedBackupStats struct { type ExtendedBackupStats struct {
BackupStats BackupStats
BlobSize int64 // Total compressed size of all referenced blobs
BlobUncompressedSize int64 // Total uncompressed size of all referenced blobs BlobUncompressedSize int64 // Total uncompressed size of all referenced blobs
CompressionLevel int // Compression level used for this snapshot CompressionLevel int // Compression level used for this snapshot
UploadDurationMs int64 // Total milliseconds spent uploading to S3 UploadDurationMs int64 // Total milliseconds spent uploading to S3
internal/storage/module.go
+2
View File
@@ -99,6 +99,7 @@ func storerFromParsedS3URL(parsed *URL, cfg *config.Config) (Storer, error) {
AccessKeyID: cfg.S3.AccessKeyID, AccessKeyID: cfg.S3.AccessKeyID,
SecretAccessKey: cfg.S3.SecretAccessKey, SecretAccessKey: cfg.S3.SecretAccessKey,
Region: region, Region: region,
PartSize: cfg.S3.PartSize.Int64(),
}) })
if err != nil { if err != nil {
return nil, fmt.Errorf("creating S3 client: %w", err) return nil, fmt.Errorf("creating S3 client: %w", err)
@@ -134,6 +135,7 @@ func storerFromLegacyS3Config(cfg *config.Config) (Storer, error) {
AccessKeyID: cfg.S3.AccessKeyID, AccessKeyID: cfg.S3.AccessKeyID,
SecretAccessKey: cfg.S3.SecretAccessKey, SecretAccessKey: cfg.S3.SecretAccessKey,
Region: region, Region: region,
PartSize: cfg.S3.PartSize.Int64(),
}) })
if err != nil { if err != nil {
return nil, fmt.Errorf("creating S3 client: %w", err) return nil, fmt.Errorf("creating S3 client: %w", err)
internal/storage/s3_test.go
+203
View File
@@ -1,14 +1,20 @@
package storage_test package storage_test
import ( import (
"bytes"
"context" "context"
"errors" "errors"
"net/http"
"net/http/httptest" "net/http/httptest"
"slices"
"strings"
"sync/atomic"
"testing" "testing"
"github.com/johannesboyne/gofakes3" "github.com/johannesboyne/gofakes3"
"github.com/johannesboyne/gofakes3/backend/s3mem" "github.com/johannesboyne/gofakes3/backend/s3mem"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/s3" "sneak.berlin/go/vaultik/internal/s3"
"sneak.berlin/go/vaultik/internal/storage" "sneak.berlin/go/vaultik/internal/storage"
) )
@@ -16,6 +22,13 @@ import (
// s3TestBucket is the bucket created for each in-process S3 server. // s3TestBucket is the bucket created for each in-process S3 server.
const s3TestBucket = "test-bucket" const s3TestBucket = "test-bucket"
// Credentials for the tests that build a storer from a config.Config. The
// in-process S3 server accepts any.
const (
s3TestAccessKeyID = "key"
s3TestSecretAccessKey = "secret"
)
// newS3Storer builds an s3:// backend backed by a fresh in-process // newS3Storer builds an s3:// backend backed by a fresh in-process
// S3 server. It reuses the same in-memory S3 harness (gofakes3 + s3mem // S3 server. It reuses the same in-memory S3 harness (gofakes3 + s3mem
// over httptest) that internal/s3 and the not-found regression test use, // over httptest) that internal/s3 and the not-found regression test use,
@@ -79,3 +92,193 @@ func TestS3StorerMissingKeyMapsToErrNotFound(t *testing.T) {
t.Errorf("Stat on missing key: got %v, want ErrNotFound", err) t.Errorf("Stat on missing key: got %v, want ErrNotFound", err)
} }
} }
// TestS3URLPrefixKeyLayout pins the bucket keys an s3:// URL reads and
// writes: the README's remote storage layout, with the prefix joined to
// each key by one "/". s3://b/p and s3://b/p/ must be the same
// destination, or a host that writes the URL the other way finds no
// snapshots. The listed object is put straight into the bucket, as
// another host would have written it. Both List and ListStream are
// checked: ListStream is what every snapshot listing goes through.
func TestS3URLPrefixKeyLayout(t *testing.T) {
t.Parallel()
const (
blobKey = "blobs/aa/bb/aabbccdd"
listPrefix = "metadata/"
manifestKey = listPrefix + "snap/manifest.json.zst"
manifestBody = "manifest"
)
cases := []struct {
urlPath string // URL path after the bucket name
keyPrefix string // what every key in the bucket must start with
}{
{urlPath: "/p", keyPrefix: "p/"},
{urlPath: "/p/", keyPrefix: "p/"},
{urlPath: "", keyPrefix: ""},
}
for _, tc := range cases {
storageURL := "s3://" + s3TestBucket + tc.urlPath
t.Run(storageURL, func(t *testing.T) {
t.Parallel()
backend := s3mem.New()
err := backend.CreateBucket(s3TestBucket)
if err != nil {
t.Fatalf("create bucket: %v", err)
}
srv := httptest.NewServer(gofakes3.New(backend).Server())
t.Cleanup(srv.Close)
storer, err := storage.NewStorer(&config.Config{
StorageURL: storageURL + "?endpoint=" + srv.URL,
S3: config.S3Config{
AccessKeyID: s3TestAccessKeyID,
SecretAccessKey: s3TestSecretAccessKey,
},
})
if err != nil {
t.Fatalf("NewStorer: %v", err)
}
ctx := context.Background()
err = storer.Put(ctx, blobKey, strings.NewReader("blob"))
if err != nil {
t.Fatalf("Put: %v", err)
}
_, err = backend.HeadObject(s3TestBucket, tc.keyPrefix+blobKey)
if err != nil {
t.Errorf("blob not stored at %q: %v", tc.keyPrefix+blobKey, err)
}
_, err = backend.PutObject(s3TestBucket, tc.keyPrefix+manifestKey,
nil, strings.NewReader(manifestBody), int64(len(manifestBody)))
if err != nil {
t.Fatalf("seed manifest: %v", err)
}
keys, err := storer.List(ctx, listPrefix)
if err != nil {
t.Fatalf("List: %v", err)
}
if !slices.Equal(keys, []string{manifestKey}) {
t.Errorf("List(%q) = %q, want [%q]", listPrefix, keys, manifestKey)
}
streamed := listStreamKeys(t, storer, listPrefix)
if !slices.Equal(streamed, []string{manifestKey}) {
t.Errorf("ListStream(%q) = %q, want [%q]", listPrefix, streamed, manifestKey)
}
})
}
}
// TestS3UploadUsesConfiguredPartSize checks that s3.part_size reaches the
// multipart uploader, through storage_url and through the s3.* fields. An
// object three parts long must arrive as three parts; at the SDK's default
// of 5 MiB it would arrive as four.
func TestS3UploadUsesConfiguredPartSize(t *testing.T) {
t.Parallel()
const (
partSize = 6 * 1024 * 1024
wantParts = 3
)
backend := s3mem.New()
err := backend.CreateBucket(s3TestBucket)
if err != nil {
t.Fatalf("create bucket: %v", err)
}
// Every part of a multipart upload is one request with a partNumber.
var parts atomic.Int32
fake := gofakes3.New(backend).Server()
srv := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
if r.URL.Query().Has("partNumber") {
parts.Add(1)
}
fake.ServeHTTP(w, r)
}))
t.Cleanup(srv.Close)
cases := []struct {
name string
cfg *config.Config
}{
{
name: "storage_url",
cfg: &config.Config{
StorageURL: "s3://" + s3TestBucket + "?endpoint=" + srv.URL,
S3: config.S3Config{
AccessKeyID: s3TestAccessKeyID,
SecretAccessKey: s3TestSecretAccessKey,
PartSize: partSize,
},
},
},
{
name: "s3.endpoint",
cfg: &config.Config{
S3: config.S3Config{
Endpoint: srv.URL,
Bucket: s3TestBucket,
AccessKeyID: s3TestAccessKeyID,
SecretAccessKey: s3TestSecretAccessKey,
PartSize: partSize,
},
},
},
}
for _, tc := range cases {
parts.Store(0)
storer, err := storage.NewStorer(tc.cfg)
if err != nil {
t.Fatalf("%s: NewStorer: %v", tc.name, err)
}
data := bytes.NewReader(make([]byte, wantParts*partSize))
err = storer.PutWithProgress(
context.Background(), "blob", data, data.Size(), nil)
if err != nil {
t.Fatalf("%s: PutWithProgress: %v", tc.name, err)
}
if got := parts.Load(); got != wantParts {
t.Errorf("%s: uploaded in %d parts, want %d", tc.name, got, wantParts)
}
}
}
// listStreamKeys returns the keys ListStream yields under a prefix, and
// fails the test on a listing error.
func listStreamKeys(t *testing.T, s storage.Storer, prefix string) []string {
t.Helper()
var keys []string
for obj := range s.ListStream(context.Background(), prefix) {
if obj.Err != nil {
t.Fatalf("ListStream %q: %v", prefix, obj.Err)
}
keys = append(keys, obj.Key)
}
return keys
}
internal/types/types.go
+2 -2
View File
@@ -155,8 +155,8 @@ type BlobHash string
// FilePath represents an absolute path to a file or directory. // FilePath represents an absolute path to a file or directory.
type FilePath string type FilePath string
// SourcePath represents the root directory from which files are backed up. // SourcePath is the source directory a scan found a file under, made
// Used during restore to strip the source prefix from paths. // absolute and with symlinks resolved.
type SourcePath string type SourcePath string
// Hostname identifies a host machine. // Hostname identifies a host machine.
internal/vaultik/changed_file_restore_test.go
+2 -4
View File
@@ -130,10 +130,9 @@ func assertThirdSnapshotRestores(
// up, and that snapshot is removed. The first snapshot keeps the file row, // up, and that snapshot is removed. The first snapshot keeps the file row,
// which now lists the appended content's chunks, while removal drops the // which now lists the appended content's chunks, while removal drops the
// blob that held them. // blob that held them.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestBackupAfterRemovingNewestSnapshotRestoresChangedFile(t *testing.T) { func TestBackupAfterRemovingNewestSnapshotRestoresChangedFile(t *testing.T) {
log.Initialize(log.Config{}) log.Initialize(log.Config{})
t.Parallel()
fs := afero.NewOsFs() fs := afero.NewOsFs()
tempDir := t.TempDir() tempDir := t.TempDir()
@@ -178,10 +177,9 @@ func TestBackupAfterRemovingNewestSnapshotRestoresChangedFile(t *testing.T) {
// The next run's prune drops that incomplete snapshot and its blob, while // The next run's prune drops that incomplete snapshot and its blob, while
// the first snapshot keeps the file row, which now lists the appended // the first snapshot keeps the file row, which now lists the appended
// content's chunks. // content's chunks.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestBackupAfterInterruptedRunRestoresChangedFile(t *testing.T) { func TestBackupAfterInterruptedRunRestoresChangedFile(t *testing.T) {
log.Initialize(log.Config{}) log.Initialize(log.Config{})
t.Parallel()
fs := afero.NewOsFs() fs := afero.NewOsFs()
tempDir := t.TempDir() tempDir := t.TempDir()
internal/vaultik/fault_injection_test.go
+15 -22
View File
@@ -38,11 +38,9 @@ import (
// (https://git.eeqj.de/sneak/vaultik/issues/130) and is not re-tested // (https://git.eeqj.de/sneak/vaultik/issues/130) and is not re-tested
// here; these tests target the layers above the backend. // here; these tests target the layers above the backend.
// //
// The tests run serially, not with t.Parallel: each calls // log.Initialize replaces the package-global logger that a running
// log.Initialize, which replaces the package-global logger, and a // backup or restore reads, so each test calls it before t.Parallel,
// backup or restore running concurrently reads that same logger. Under // while no parallel test is running yet.
// -race the two collide. Running one at a time is the same choice
// prune_count_test.go already makes for the same reason.
const ( const (
faultChunkSize = int64(64 * 1024) faultChunkSize = int64(64 * 1024)
@@ -165,17 +163,19 @@ func newReaderVaultik(
// Scenario 3: a stored blob's bytes are flipped before restore reads // Scenario 3: a stored blob's bytes are flipped before restore reads
// them. Restore must fail loudly, and no file must be left on the // them. Restore must fail loudly, and no file must be left on the
// restore target holding corrupt content. // restore target holding corrupt content.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestRestoreRejectsCorruptBlob(t *testing.T) { func TestRestoreRejectsCorruptBlob(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
assertRestoreRejectsDamagedBlob(t, faultstore.GetCorrupt, "corrupt") assertRestoreRejectsDamagedBlob(t, faultstore.GetCorrupt, "corrupt")
} }
// Scenario 4: a stored blob is truncated before restore reads it. Same // Scenario 4: a stored blob is truncated before restore reads it. Same
// contract as the corrupt case. // contract as the corrupt case.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestRestoreRejectsTruncatedBlob(t *testing.T) { func TestRestoreRejectsTruncatedBlob(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
assertRestoreRejectsDamagedBlob(t, faultstore.GetTruncate, "truncated") assertRestoreRejectsDamagedBlob(t, faultstore.GetTruncate, "truncated")
} }
@@ -188,7 +188,6 @@ func assertRestoreRejectsDamagedBlob(
t *testing.T, fault faultstore.GetFault, name string, t *testing.T, fault faultstore.GetFault, name string,
) { ) {
t.Helper() t.Helper()
log.Initialize(log.Config{})
fs := afero.NewOsFs() fs := afero.NewOsFs()
tempDir := t.TempDir() tempDir := t.TempDir()
@@ -232,10 +231,9 @@ func assertRestoreRejectsDamagedBlob(
// Scenario 6: the backend accepts blob uploads and reports success but // Scenario 6: the backend accepts blob uploads and reports success but
// stores nothing. verify --deep must catch it. // stores nothing. verify --deep must catch it.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestDeepVerifyCatchesLyingBackend(t *testing.T) { func TestDeepVerifyCatchesLyingBackend(t *testing.T) {
log.Initialize(log.Config{}) log.Initialize(log.Config{})
t.Parallel()
fs := afero.NewOsFs() fs := afero.NewOsFs()
tempDir := t.TempDir() tempDir := t.TempDir()
@@ -285,10 +283,9 @@ func TestDeepVerifyCatchesLyingBackend(t *testing.T) {
// Scenario 1a: a blob upload fails partway through. The interrupted run // Scenario 1a: a blob upload fails partway through. The interrupted run
// must not record the blob as uploaded, must not reference it from the // must not record the blob as uploaded, must not reference it from the
// snapshot, and must leave no blob object at the destination. // snapshot, and must leave no blob object at the destination.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestInterruptedBlobUploadRecordsNoUploadedBlob(t *testing.T) { func TestInterruptedBlobUploadRecordsNoUploadedBlob(t *testing.T) {
log.Initialize(log.Config{}) log.Initialize(log.Config{})
t.Parallel()
fs := afero.NewOsFs() fs := afero.NewOsFs()
tempDir := t.TempDir() tempDir := t.TempDir()
@@ -361,10 +358,9 @@ func TestInterruptedBlobUploadRecordsNoUploadedBlob(t *testing.T) {
// chunks in a blob that was actually uploaded, so the retry re-chunks and // chunks in a blob that was actually uploaded, so the retry re-chunks and
// re-uploads the affected data instead of silently referencing data that // re-uploads the affected data instead of silently referencing data that
// never reached storage. // never reached storage.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestBackupRetryAfterInterruptedUploadIsRestorable(t *testing.T) { func TestBackupRetryAfterInterruptedUploadIsRestorable(t *testing.T) {
log.Initialize(log.Config{}) log.Initialize(log.Config{})
t.Parallel()
fs := afero.NewOsFs() fs := afero.NewOsFs()
tempDir := t.TempDir() tempDir := t.TempDir()
@@ -426,10 +422,9 @@ func TestBackupRetryAfterInterruptedUploadIsRestorable(t *testing.T) {
// covered by TestBackupCompletesOnlyAfterMetadataExport // covered by TestBackupCompletesOnlyAfterMetadataExport
// (https://git.eeqj.de/sneak/vaultik/issues/177); this test exercises the // (https://git.eeqj.de/sneak/vaultik/issues/177); this test exercises the
// lower-level export path in isolation. // lower-level export path in isolation.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestBackupSurvivesMetadataExportInterruption(t *testing.T) { func TestBackupSurvivesMetadataExportInterruption(t *testing.T) {
log.Initialize(log.Config{}) log.Initialize(log.Config{})
t.Parallel()
fs := afero.NewOsFs() fs := afero.NewOsFs()
tempDir := t.TempDir() tempDir := t.TempDir()
@@ -509,10 +504,9 @@ func TestBackupSurvivesMetadataExportInterruption(t *testing.T) {
// destination. Rerunning the backup must then prune the incomplete // destination. Rerunning the backup must then prune the incomplete
// snapshot, produce a snapshot whose destination metadata and local index // snapshot, produce a snapshot whose destination metadata and local index
// agree, and restore. See https://git.eeqj.de/sneak/vaultik/issues/177. // agree, and restore. See https://git.eeqj.de/sneak/vaultik/issues/177.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestBackupCompletesOnlyAfterMetadataExport(t *testing.T) { func TestBackupCompletesOnlyAfterMetadataExport(t *testing.T) {
log.Initialize(log.Config{}) log.Initialize(log.Config{})
t.Parallel()
fs := afero.NewOsFs() fs := afero.NewOsFs()
tempDir := t.TempDir() tempDir := t.TempDir()
@@ -685,10 +679,9 @@ func faultScannerFactory(
// Scenario 5: the restore target runs out of space mid-file. Restore // Scenario 5: the restore target runs out of space mid-file. Restore
// must fail with an out-of-space error, and must not leave a truncated // must fail with an out-of-space error, and must not leave a truncated
// file at the target path presenting as a complete restore. // file at the target path presenting as a complete restore.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestRestoreReportsDiskFull(t *testing.T) { func TestRestoreReportsDiskFull(t *testing.T) {
log.Initialize(log.Config{}) log.Initialize(log.Config{})
t.Parallel()
osFS := afero.NewOsFs() osFS := afero.NewOsFs()
tempDir := t.TempDir() tempDir := t.TempDir()
internal/vaultik/helpers.go
+23 -18
View File
@@ -9,6 +9,7 @@ import (
"time" "time"
"github.com/dustin/go-humanize" "github.com/dustin/go-humanize"
"sneak.berlin/go/vaultik/internal/snapshot"
"sneak.berlin/go/vaultik/internal/types" "sneak.berlin/go/vaultik/internal/types"
) )
@@ -46,12 +47,9 @@ const (
year = 365 * day year = 365 * day
) )
// Snapshot IDs split on "_" into hostname, optional name parts, and a // A snapshot ID split on "_" has at least a hostname and a trailing
// trailing timestamp. // timestamp.
const ( const minSnapshotIDParts = 2
minSnapshotIDParts = 2
minSnapshotIDNameParts = 3
)
// SnapshotInfo contains information about a snapshot. // SnapshotInfo contains information about a snapshot.
// //
@@ -121,20 +119,27 @@ func parseSnapshotTimestamp(snapshotID string) (time.Time, error) {
return timestamp.UTC(), nil return timestamp.UTC(), nil
} }
// parseSnapshotName extracts the snapshot name from a snapshot ID. // parseSnapshotName extracts the snapshot name from a snapshot ID of the
// Format: hostname_snapshotname_timestamp — the middle part(s) between hostname // form hostname_name_timestamp, given the hostname stored with that
// and the RFC3339 timestamp are the snapshot name (may contain underscores). // snapshot. The hostname and the name may both contain underscores, so the
// Returns the snapshot name, or empty string if the ID is malformed. // name is what is left after removing the short hostname and its "_" from
func parseSnapshotName(snapshotID string) string { // the front and the last "_" and the timestamp from the end. Returns "" for
parts := strings.Split(snapshotID, "_") // an ID with no name (hostname_timestamp), and for an ID that does not start
if len(parts) < minSnapshotIDNameParts { // with that hostname, which CreateSnapshotWithName never writes.
// Format: hostname_timestamp — no snapshot name func parseSnapshotName(snapshotID, hostname string) string {
prefix := snapshot.ShortHostname(hostname) + "_"
rest, ok := strings.CutPrefix(snapshotID, prefix)
if !ok {
return "" return ""
} }
// Format: hostname_name_timestamp — middle parts are the name.
// The last part is the RFC3339 timestamp, the first part is the hostname, end := strings.LastIndex(rest, "_")
// everything in between is the snapshot name (which may itself contain underscores). if end < 0 {
return strings.Join(parts[1:len(parts)-1], "_") return ""
}
return rest[:end]
} }
// parseDuration parses a duration string with support for human-friendly units: // parseDuration parses a duration string with support for human-friendly units:
internal/vaultik/helpers_test.go
+25 -3
View File
@@ -11,33 +11,55 @@ func TestParseSnapshotName(t *testing.T) {
tests := []struct { tests := []struct {
name string name string
snapshotID string snapshotID string
hostname string
want string want string
}{ }{
{ {
name: "standard format with name", name: "standard format with name",
snapshotID: "myhost_home_2026-01-12T14:41:15Z", snapshotID: "myhost_home_2026-01-12T14:41:15Z",
hostname: "myhost",
want: "home", want: "home",
}, },
{ {
name: "standard format with different name", name: "standard format with different name",
snapshotID: "server1_system_2026-02-15T09:30:00Z", snapshotID: "server1_system_2026-02-15T09:30:00Z",
hostname: "server1",
want: "system", want: "system",
}, },
{ {
name: "name with underscores", name: "name with underscores",
snapshotID: "myhost_my_special_backup_2026-03-01T00:00:00Z", snapshotID: "myhost_my_special_backup_2026-03-01T00:00:00Z",
hostname: "myhost",
want: "my_special_backup", want: "my_special_backup",
}, },
{
name: "hostname with underscores",
snapshotID: "my_host_docs_2026-03-01T00:00:00Z",
hostname: "my_host",
want: "docs",
},
{
name: "stored hostname with domain",
snapshotID: "my_host_mail_2026-03-01T00:00:00Z",
hostname: "my_host.example.com",
want: "mail",
},
{
name: "no name",
snapshotID: "my_host_2026-03-01T00:00:00Z",
hostname: "my_host",
want: "",
},
} }
for _, tt := range tests { for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) { t.Run(tt.name, func(t *testing.T) {
t.Parallel() t.Parallel()
got := parseSnapshotName(tt.snapshotID) got := parseSnapshotName(tt.snapshotID, tt.hostname)
if got != tt.want { if got != tt.want {
t.Errorf("parseSnapshotName(%q) = %q, want %q", t.Errorf("parseSnapshotName(%q, %q) = %q, want %q",
tt.snapshotID, got, tt.want) tt.snapshotID, tt.hostname, got, tt.want)
} }
}) })
} }
internal/vaultik/info.go
+109 -25
View File
@@ -183,6 +183,11 @@ type SnapshotMetadataInfo struct {
TotalSize int64 `json:"total_size"` TotalSize int64 `json:"total_size"`
BlobCount int `json:"blob_count"` BlobCount int `json:"blob_count"`
BlobsSize int64 `json:"blobs_size"` BlobsSize int64 `json:"blobs_size"`
// Set when the listing holds this snapshot's manifest.json.zst. A
// backup interrupted before its manifest upload leaves a directory
// without one, which prune does not treat as a snapshot.
hasManifest bool
} }
// RemoteInfoResult contains all remote storage information // RemoteInfoResult contains all remote storage information
@@ -206,9 +211,20 @@ type RemoteInfoResult struct {
ReferencedBlobCount int `json:"referenced_blob_count"` ReferencedBlobCount int `json:"referenced_blob_count"`
ReferencedBlobSize int64 `json:"referenced_blob_size"` ReferencedBlobSize int64 `json:"referenced_blob_size"`
// Orphaned blobs // Orphaned blobs. Both stay nil (null in the JSON) when a manifest
OrphanedBlobCount int `json:"orphaned_blob_count"` // was listed but not read, since that snapshot's blobs would be
OrphanedBlobSize int64 `json:"orphaned_blob_size"` // counted as orphaned.
OrphanedBlobCount *int `json:"orphaned_blob_count"`
OrphanedBlobSize *int64 `json:"orphaned_blob_size"`
// Remote key of each snapshot whose manifest could not be read
UnreadableManifests []string `json:"unreadable_manifests,omitempty"`
// Number of manifests not read because the name above them under
// metadata/ is not a remote key. The names themselves are not
// reported: they come from the destination store and may hold
// control characters.
SkippedManifestCount int `json:"skipped_manifest_count,omitempty"`
} }
// RemoteInfo displays information about remote storage // RemoteInfo displays information about remote storage
@@ -234,16 +250,28 @@ func (v *Vaultik) RemoteInfo(jsonOutput bool) error {
v.stdoutf("Scanning snapshot metadata...\n") v.stdoutf("Scanning snapshot metadata...\n")
} }
snapshotMetadata, snapshotIDs, err := v.collectSnapshotMetadata() snapshotMetadata, snapshotIDs, skippedManifestCount, err := v.collectSnapshotMetadata()
if err != nil { if err != nil {
return err return err
} }
result.SkippedManifestCount = skippedManifestCount
if showText { if showText {
v.stdoutf("Downloading %d manifest(s)...\n", len(snapshotIDs)) manifestCount := 0
for _, info := range snapshotMetadata {
if info.hasManifest {
manifestCount++
}
}
v.stdoutf("Downloading %d manifest(s)...\n", manifestCount)
} }
referencedBlobs := v.collectReferencedBlobsFromManifests(snapshotIDs, snapshotMetadata) referencedBlobs, unreadableManifests := v.collectReferencedBlobsFromManifests(
snapshotIDs, snapshotMetadata)
result.UnreadableManifests = unreadableManifests
v.populateRemoteInfoResult(result, snapshotMetadata, snapshotIDs, referencedBlobs) v.populateRemoteInfoResult(result, snapshotMetadata, snapshotIDs, referencedBlobs)
@@ -256,7 +284,7 @@ func (v *Vaultik) RemoteInfo(jsonOutput bool) error {
"snapshots", result.TotalMetadataCount, "snapshots", result.TotalMetadataCount,
"total_blobs", result.TotalBlobCount, "total_blobs", result.TotalBlobCount,
"referenced_blobs", result.ReferencedBlobCount, "referenced_blobs", result.ReferencedBlobCount,
"orphaned_blobs", result.OrphanedBlobCount) "unreadable_manifests", len(result.UnreadableManifests))
if jsonOutput { if jsonOutput {
enc := json.NewEncoder(v.Stdout) enc := json.NewEncoder(v.Stdout)
@@ -273,16 +301,18 @@ func (v *Vaultik) RemoteInfo(jsonOutput bool) error {
} }
// collectSnapshotMetadata scans remote metadata and returns // collectSnapshotMetadata scans remote metadata and returns
// per-snapshot info and sorted IDs. // per-snapshot info, sorted IDs and the number of manifests it skipped
// because the name above them is not a remote key.
func (v *Vaultik) collectSnapshotMetadata() ( func (v *Vaultik) collectSnapshotMetadata() (
map[string]*SnapshotMetadataInfo, []string, error, map[string]*SnapshotMetadataInfo, []string, int, error,
) { ) {
snapshotMetadata := make(map[string]*SnapshotMetadataInfo) snapshotMetadata := make(map[string]*SnapshotMetadataInfo)
skippedManifestCount := 0
metadataCh := v.Storage.ListStream(v.ctx, "metadata/") metadataCh := v.Storage.ListStream(v.ctx, "metadata/")
for obj := range metadataCh { for obj := range metadataCh {
if obj.Err != nil { if obj.Err != nil {
return nil, nil, fmt.Errorf("listing metadata: %w", obj.Err) return nil, nil, 0, fmt.Errorf("listing metadata: %w", obj.Err)
} }
parts := strings.Split(obj.Key, "/") parts := strings.Split(obj.Key, "/")
@@ -291,6 +321,22 @@ func (v *Vaultik) collectSnapshotMetadata() (
} }
snapshotID := parts[1] snapshotID := parts[1]
filename := parts[2]
isManifest := filename == "manifest.json.zst"
// The name comes from the destination store, which is not
// trusted, and is printed in the report. Accept it only in the
// form of a remote key.
if !isBlobHash(snapshotID) {
log.Warn("Skipping non-conforming key under metadata/",
"key", obj.Key)
if isManifest {
skippedManifestCount++
}
continue
}
if _, exists := snapshotMetadata[snapshotID]; !exists { if _, exists := snapshotMetadata[snapshotID]; !exists {
snapshotMetadata[snapshotID] = &SnapshotMetadataInfo{SnapshotID: snapshotID} snapshotMetadata[snapshotID] = &SnapshotMetadataInfo{SnapshotID: snapshotID}
@@ -298,7 +344,10 @@ func (v *Vaultik) collectSnapshotMetadata() (
info := snapshotMetadata[snapshotID] info := snapshotMetadata[snapshotID]
filename := parts[2] if isManifest {
info.hasManifest = true
}
if strings.HasPrefix(filename, "manifest") { if strings.HasPrefix(filename, "manifest") {
info.ManifestSize = obj.Size info.ManifestSize = obj.Size
} else if strings.HasPrefix(filename, "db") { } else if strings.HasPrefix(filename, "db") {
@@ -315,17 +364,25 @@ func (v *Vaultik) collectSnapshotMetadata() (
sort.Strings(snapshotIDs) sort.Strings(snapshotIDs)
return snapshotMetadata, snapshotIDs, nil return snapshotMetadata, snapshotIDs, skippedManifestCount, nil
} }
// collectReferencedBlobsFromManifests downloads manifests and returns // collectReferencedBlobsFromManifests downloads the listed manifests
// referenced blob hashes with sizes. // and returns referenced blob hashes with sizes, and the remote keys
// of the manifests it could not read.
func (v *Vaultik) collectReferencedBlobsFromManifests( func (v *Vaultik) collectReferencedBlobsFromManifests(
snapshotIDs []string, snapshotMetadata map[string]*SnapshotMetadataInfo, snapshotIDs []string, snapshotMetadata map[string]*SnapshotMetadataInfo,
) map[string]int64 { ) (map[string]int64, []string) {
referencedBlobs := make(map[string]int64) referencedBlobs := make(map[string]int64)
var unreadable []string
for _, snapshotID := range snapshotIDs { for _, snapshotID := range snapshotIDs {
info := snapshotMetadata[snapshotID]
if !info.hasManifest {
continue
}
// snapshotIDs here are remote keys, taken straight from the // snapshotIDs here are remote keys, taken straight from the
// metadata/ listing. downloadManifestByKey is the single reader // metadata/ listing. downloadManifestByKey is the single reader
// for remote manifests; see its doc comment. // for remote manifests; see its doc comment.
@@ -333,10 +390,11 @@ func (v *Vaultik) collectReferencedBlobsFromManifests(
if err != nil { if err != nil {
log.Warn("Failed to read manifest", "snapshot", snapshotID, "error", err) log.Warn("Failed to read manifest", "snapshot", snapshotID, "error", err)
unreadable = append(unreadable, snapshotID)
continue continue
} }
info := snapshotMetadata[snapshotID]
info.BlobCount = manifest.BlobCount info.BlobCount = manifest.BlobCount
var blobsSize int64 var blobsSize int64
@@ -349,7 +407,7 @@ func (v *Vaultik) collectReferencedBlobsFromManifests(
info.BlobsSize = blobsSize info.BlobsSize = blobsSize
} }
return referencedBlobs return referencedBlobs, unreadable
} }
// populateRemoteInfoResult fills in the result's snapshot and // populateRemoteInfoResult fills in the result's snapshot and
@@ -378,8 +436,9 @@ func (v *Vaultik) populateRemoteInfoResult(
} }
// scanRemoteBlobStorage lists all blobs on remote and computes orphan // scanRemoteBlobStorage lists all blobs on remote and computes orphan
// stats. showText is true only when the human report is being printed // stats when every listed manifest was read. showText is true only
// (not --json, not --quiet), gating the progress line. // when the human report is being printed (not --json, not --quiet),
// gating the progress line.
func (v *Vaultik) scanRemoteBlobStorage( func (v *Vaultik) scanRemoteBlobStorage(
result *RemoteInfoResult, referencedBlobs map[string]int64, showText bool, result *RemoteInfoResult, referencedBlobs map[string]int64, showText bool,
) error { ) error {
@@ -406,13 +465,28 @@ func (v *Vaultik) scanRemoteBlobStorage(
result.TotalBlobSize += obj.Size result.TotalBlobSize += obj.Size
} }
// A blob named only by a manifest that could not be read, or by one
// under a skipped name, would be counted as orphaned, so the orphan
// figures stay unknown.
if len(result.UnreadableManifests) > 0 || result.SkippedManifestCount > 0 {
return nil
}
var (
orphanedCount int
orphanedSize int64
)
for hash, size := range allBlobs { for hash, size := range allBlobs {
if _, referenced := referencedBlobs[hash]; !referenced { if _, referenced := referencedBlobs[hash]; !referenced {
result.OrphanedBlobCount++ orphanedCount++
result.OrphanedBlobSize += size orphanedSize += size
} }
} }
result.OrphanedBlobCount = &orphanedCount
result.OrphanedBlobSize = &orphanedSize
return nil return nil
} }
@@ -465,11 +539,21 @@ func (v *Vaultik) printRemoteInfoTable(result *RemoteInfoResult) {
v.stdoutf("Referenced by snapshots: %s (%s)\n", v.stdoutf("Referenced by snapshots: %s (%s)\n",
humanize.Comma(int64(result.ReferencedBlobCount)), humanize.Comma(int64(result.ReferencedBlobCount)),
ubytes(result.ReferencedBlobSize)) ubytes(result.ReferencedBlobSize))
v.stdoutf("Orphaned (unreferenced): %s (%s)\n",
humanize.Comma(int64(result.OrphanedBlobCount)),
ubytes(result.OrphanedBlobSize))
if result.OrphanedBlobCount > 0 { if result.OrphanedBlobCount == nil {
v.stdoutf("Orphaned (unreferenced): unknown "+
"(%d manifest(s) could not be read, "+
"%d manifest(s) under a non-conforming name skipped)\n",
len(result.UnreadableManifests), result.SkippedManifestCount)
return
}
v.stdoutf("Orphaned (unreferenced): %s (%s)\n",
humanize.Comma(int64(*result.OrphanedBlobCount)),
ubytes(*result.OrphanedBlobSize))
if *result.OrphanedBlobCount > 0 {
v.stdoutf("\nRun 'vaultik prune' to remove orphaned blobs.\n") v.stdoutf("\nRun 'vaultik prune' to remove orphaned blobs.\n")
} }
} }
internal/vaultik/integration_test.go
+11 -12
View File
@@ -249,7 +249,7 @@ func verifyEndToEndBackupState(
assert.Positive(t, blobUploads, "Should upload at least one blob") assert.Positive(t, blobUploads, "Should upload at least one blob")
// Verify files in database // Verify files in database
files, err := repos.Files.ListByPrefix(ctx, "/home/user") files, err := repos.Files.ListUnderPath(ctx, "/home/user")
require.NoError(t, err) require.NoError(t, err)
// Count only regular files (not directories) // Count only regular files (not directories)
regularFiles := 0 regularFiles := 0
@@ -928,17 +928,17 @@ func setupDedupBackupEnv(
} }
} }
// runDedupSnapshot creates a "dedup" snapshot, scans dataDir into it, // runDedupSnapshot creates a snapshot with the given name, scans dataDir
// completes it, and exports its metadata, returning the snapshot ID and // into it, completes it, and exports its metadata, returning the snapshot
// scan result. // ID and scan result.
func runDedupSnapshot( func runDedupSnapshot(
ctx context.Context, t *testing.T, ctx context.Context, t *testing.T,
sm *snapshot.SnapshotManager, scanner *snapshot.Scanner, sm *snapshot.SnapshotManager, scanner *snapshot.Scanner,
hostname, dataDir, dbPath string, hostname, name, dataDir, dbPath string,
) (string, *snapshot.ScanResult) { ) (string, *snapshot.ScanResult) {
t.Helper() t.Helper()
id, err := sm.CreateSnapshotWithName(ctx, hostname, "dedup", "v", "g") id, err := sm.CreateSnapshotWithName(ctx, hostname, name, "v", "g")
require.NoError(t, err) require.NoError(t, err)
result, err := scanner.Scan(ctx, dataDir, id) result, err := scanner.Scan(ctx, dataDir, id)
@@ -980,16 +980,15 @@ func TestDedupOnlySnapshotRestores(t *testing.T) {
// First snapshot — uploads all blobs. // First snapshot — uploads all blobs.
_, r1 := runDedupSnapshot(ctx, t, sm, makeScanner(), _, r1 := runDedupSnapshot(ctx, t, sm, makeScanner(),
cfg.Hostname, dataDir, dbPath) cfg.Hostname, "first", dataDir, dbPath)
require.Positive(t, r1.BlobsCreated, require.Positive(t, r1.BlobsCreated,
"first snapshot should upload at least one blob") "first snapshot should upload at least one blob")
// Second snapshot — same data, every chunk dedups. Sleep past the // Second snapshot — same data, every chunk dedups. Its own name gives
// second-precision timestamp so the snapshot IDs differ. // it a different snapshot ID without waiting for the one-second
time.Sleep(1100 * time.Millisecond) // timestamp in the ID to tick over.
id2, r2 := runDedupSnapshot(ctx, t, sm, makeScanner(), id2, r2 := runDedupSnapshot(ctx, t, sm, makeScanner(),
cfg.Hostname, dataDir, dbPath) cfg.Hostname, "second", dataDir, dbPath)
require.Equal(t, 0, r2.BlobsCreated, require.Equal(t, 0, r2.BlobsCreated,
"second snapshot should upload zero new blobs (fully dedup'd)") "second snapshot should upload zero new blobs (fully dedup'd)")
internal/vaultik/missing_destination_test.go
+24 -8
View File
@@ -6,6 +6,7 @@ import (
"io/fs" "io/fs"
"os" "os"
"path/filepath" "path/filepath"
"strings"
"testing" "testing"
"github.com/spf13/afero" "github.com/spf13/afero"
@@ -77,10 +78,9 @@ func backUpThenUnplug(
// TestFirstBackupCreatesDestinationDirectory checks that a first backup // TestFirstBackupCreatesDestinationDirectory checks that a first backup
// to a destination directory that does not exist yet creates it, and // to a destination directory that does not exist yet creates it, and
// that the destination can be listed afterwards. // that the destination can be listed afterwards.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestFirstBackupCreatesDestinationDirectory(t *testing.T) { func TestFirstBackupCreatesDestinationDirectory(t *testing.T) {
log.Initialize(log.Config{}) log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background() ctx := context.Background()
storeDir := filepath.Join(t.TempDir(), "volume", "backup") storeDir := filepath.Join(t.TempDir(), "volume", "backup")
@@ -95,10 +95,9 @@ func TestFirstBackupCreatesDestinationDirectory(t *testing.T) {
// TestListSnapshotsWarnsWhenDestinationMissing checks that snapshot list // TestListSnapshotsWarnsWhenDestinationMissing checks that snapshot list
// warns and shows the local index alone, without reporting the local // warns and shows the local index alone, without reporting the local
// snapshot as missing from the destination. // snapshot as missing from the destination.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestListSnapshotsWarnsWhenDestinationMissing(t *testing.T) { func TestListSnapshotsWarnsWhenDestinationMissing(t *testing.T) {
log.Initialize(log.Config{}) log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background() ctx := context.Background()
v, repos, out := backUpThenUnplug(ctx, t) v, repos, out := backUpThenUnplug(ctx, t)
@@ -115,10 +114,9 @@ func TestListSnapshotsWarnsWhenDestinationMissing(t *testing.T) {
// TestRemoveSnapshotWarnsWhenDestinationMissing checks that snapshot // TestRemoveSnapshotWarnsWhenDestinationMissing checks that snapshot
// remove warns that the metadata could not be removed from the // remove warns that the metadata could not be removed from the
// destination, instead of reporting that it was. // destination, instead of reporting that it was.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestRemoveSnapshotWarnsWhenDestinationMissing(t *testing.T) { func TestRemoveSnapshotWarnsWhenDestinationMissing(t *testing.T) {
log.Initialize(log.Config{}) log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background() ctx := context.Background()
v, repos, out := backUpThenUnplug(ctx, t) v, repos, out := backUpThenUnplug(ctx, t)
@@ -137,10 +135,9 @@ func TestRemoveSnapshotWarnsWhenDestinationMissing(t *testing.T) {
// TestPruneKeepsLocalRecordsWhenDestinationMissing checks that prune // TestPruneKeepsLocalRecordsWhenDestinationMissing checks that prune
// fails on a destination it cannot list and deletes no local snapshot // fails on a destination it cannot list and deletes no local snapshot
// record. // record.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestPruneKeepsLocalRecordsWhenDestinationMissing(t *testing.T) { func TestPruneKeepsLocalRecordsWhenDestinationMissing(t *testing.T) {
log.Initialize(log.Config{}) log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background() ctx := context.Background()
v, repos, _ := backUpThenUnplug(ctx, t) v, repos, _ := backUpThenUnplug(ctx, t)
@@ -153,3 +150,22 @@ func TestPruneKeepsLocalRecordsWhenDestinationMissing(t *testing.T) {
require.NoError(t, err) require.NoError(t, err)
assert.Len(t, snapshots, 1, "prune must delete no local snapshot record") assert.Len(t, snapshots, 1, "prune must delete no local snapshot record")
} }
// TestPurgeSaysListingFailedOnceWhenDestinationMissing checks that
// snapshot purge fails on a destination it cannot list, with an error
// that says "listing remote snapshots" once.
func TestPurgeSaysListingFailedOnceWhenDestinationMissing(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
v, _, _ := backUpThenUnplug(ctx, t)
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
KeepLatest: true,
Force: true,
})
require.ErrorIs(t, err, fs.ErrNotExist)
assert.Equal(t, 1, strings.Count(err.Error(), "listing remote snapshots"),
err.Error())
}
internal/vaultik/prune_count_test.go
+1 -2
View File
@@ -14,10 +14,9 @@ import (
// the discarded-error bug: getTableCount for a table its query cannot // the discarded-error bug: getTableCount for a table its query cannot
// resolve must not silently become 0. A count that could not be read is // resolve must not silently become 0. A count that could not be read is
// reported as unknown, which a reader can tell apart from an empty table. // reported as unknown, which a reader can tell apart from an empty table.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestTableCountForReportSurfacesReadFailure(t *testing.T) { func TestTableCountForReportSurfacesReadFailure(t *testing.T) {
log.Initialize(log.Config{}) log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background() ctx := context.Background()
internal/vaultik/purge_local_remote_consistency_test.go
+1 -1
View File
@@ -42,7 +42,7 @@ func setupConsistencyTest(
completedAt := startedAt.Add(5 * time.Minute) completedAt := startedAt.Add(5 * time.Minute)
snap := &database.Snapshot{ snap := &database.Snapshot{
ID: types.SnapshotID(id), ID: types.SnapshotID(id),
Hostname: testHostname, Hostname: snapHostname,
VaultikVersion: testLabel, VaultikVersion: testLabel,
StartedAt: startedAt, StartedAt: startedAt,
CompletedAt: &completedAt, CompletedAt: &completedAt,
internal/vaultik/purge_per_name_test.go
+42 -12
View File
@@ -17,8 +17,10 @@ import (
"sneak.berlin/go/vaultik/internal/vaultik" "sneak.berlin/go/vaultik/internal/vaultik"
) )
// Snapshot IDs reused across the purge tests. // Snapshot IDs reused across the purge tests, and the hostname they were
// taken on.
const ( const (
snapHostname = "testhost"
snapSystemT0 = "testhost_system_2026-01-01T00:00:00Z" snapSystemT0 = "testhost_system_2026-01-01T00:00:00Z"
snapHomeT0 = "testhost_home_2026-01-01T00:00:00Z" snapHomeT0 = "testhost_home_2026-01-01T00:00:00Z"
snapHomeT1 = "testhost_home_2026-01-01T01:00:00Z" snapHomeT1 = "testhost_home_2026-01-01T01:00:00Z"
@@ -26,9 +28,12 @@ const (
) )
// setupPurgeTest creates a Vaultik instance with an in-memory database and mock // setupPurgeTest creates a Vaultik instance with an in-memory database and mock
// storage pre-populated with the given snapshot IDs. Each snapshot is marked as // storage pre-populated with the given snapshot IDs, all taken on hostname.
// completed. Remote metadata stubs are created so syncWithRemote keeps them. // Each snapshot is marked as completed. Remote metadata stubs are created so
func setupPurgeTest(t *testing.T, snapshotIDs []string) *vaultik.Vaultik { // syncWithRemote keeps them.
func setupPurgeTest(
t *testing.T, hostname string, snapshotIDs []string,
) *vaultik.Vaultik {
t.Helper() t.Helper()
ctx := context.Background() ctx := context.Background()
@@ -51,7 +56,7 @@ func setupPurgeTest(t *testing.T, snapshotIDs []string) *vaultik.Vaultik {
completedAt := startedAt.Add(5 * time.Minute) completedAt := startedAt.Add(5 * time.Minute)
snap := &database.Snapshot{ snap := &database.Snapshot{
ID: types.SnapshotID(id), ID: types.SnapshotID(id),
Hostname: "testhost", Hostname: types.Hostname(hostname),
VaultikVersion: testLabel, VaultikVersion: testLabel,
StartedAt: startedAt, StartedAt: startedAt,
CompletedAt: &completedAt, CompletedAt: &completedAt,
@@ -120,7 +125,7 @@ func TestPurgeKeepLatest_PerName(t *testing.T) {
"testhost_system_2026-01-01T04:00:00Z", "testhost_system_2026-01-01T04:00:00Z",
} }
v := setupPurgeTest(t, snapshotIDs) v := setupPurgeTest(t, snapHostname, snapshotIDs)
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{ err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
KeepLatest: true, KeepLatest: true,
@@ -148,7 +153,7 @@ func TestPurgeKeepLatest_SingleName(t *testing.T) {
"testhost_home_2026-01-01T02:00:00Z", "testhost_home_2026-01-01T02:00:00Z",
} }
v := setupPurgeTest(t, snapshotIDs) v := setupPurgeTest(t, snapHostname, snapshotIDs)
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{ err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
KeepLatest: true, KeepLatest: true,
@@ -176,7 +181,7 @@ func TestPurgeKeepLatest_WithNameFilter(t *testing.T) {
"testhost_home_2026-01-01T04:00:00Z", "testhost_home_2026-01-01T04:00:00Z",
} }
v := setupPurgeTest(t, snapshotIDs) v := setupPurgeTest(t, snapHostname, snapshotIDs)
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{ err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
KeepLatest: true, KeepLatest: true,
@@ -198,7 +203,7 @@ func TestPurgeKeepLatest_NoSnapshots(t *testing.T) {
log.Initialize(log.Config{}) log.Initialize(log.Config{})
t.Parallel() t.Parallel()
v := setupPurgeTest(t, nil) v := setupPurgeTest(t, snapHostname, nil)
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{ err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
KeepLatest: true, KeepLatest: true,
@@ -216,7 +221,7 @@ func TestPurgeKeepLatest_NameFilterNoMatch(t *testing.T) {
"testhost_system_2026-01-01T01:00:00Z", "testhost_system_2026-01-01T01:00:00Z",
} }
v := setupPurgeTest(t, snapshotIDs) v := setupPurgeTest(t, snapHostname, snapshotIDs)
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{ err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
KeepLatest: true, KeepLatest: true,
@@ -243,7 +248,7 @@ func TestPurgeOlderThan_WithNameFilter(t *testing.T) {
snapHomeT0, snapHomeT0,
} }
v := setupPurgeTest(t, snapshotIDs) v := setupPurgeTest(t, snapHostname, snapshotIDs)
// Purge only "home" snapshots older than 365 days // Purge only "home" snapshots older than 365 days
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{ err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
@@ -277,7 +282,7 @@ func TestPurgeKeepLatest_ThreeNames(t *testing.T) {
"testhost_home_2026-01-01T06:00:00Z", "testhost_home_2026-01-01T06:00:00Z",
} }
v := setupPurgeTest(t, snapshotIDs) v := setupPurgeTest(t, snapHostname, snapshotIDs)
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{ err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
KeepLatest: true, KeepLatest: true,
@@ -291,3 +296,28 @@ func TestPurgeKeepLatest_ThreeNames(t *testing.T) {
assert.Contains(t, remaining, "testhost_system_2026-01-01T04:00:00Z") assert.Contains(t, remaining, "testhost_system_2026-01-01T04:00:00Z")
assert.Contains(t, remaining, "testhost_media_2026-01-01T05:00:00Z") assert.Contains(t, remaining, "testhost_media_2026-01-01T05:00:00Z")
} }
// A hostname may contain underscores, so the snapshot name cannot be found
// by splitting the ID at them. A purge by name must still select "docs".
func TestPurgeKeepLatest_HostnameWithUnderscore(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
const (
system = "my_host_system_2026-01-01T00:00:00Z"
docsT1 = "my_host_docs_2026-01-01T01:00:00Z"
docsT2 = "my_host_docs_2026-01-01T02:00:00Z"
)
v := setupPurgeTest(t, "my_host", []string{system, docsT1, docsT2})
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
KeepLatest: true,
Force: true,
Names: []string{"docs"},
})
require.NoError(t, err)
assert.ElementsMatch(t, []string{system, docsT2},
listRemainingSnapshots(t, v))
}
internal/vaultik/remote_info_test.go
+159
View File
@@ -0,0 +1,159 @@
package vaultik_test
import (
"bytes"
"context"
"encoding/json"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot"
)
// testBlobHashB is a blob that the manifest written by addRemote does
// not reference.
const testBlobHashB = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" +
"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
// TestRemoteInfo_UnreadableManifestLeavesOrphansUnknown checks that a
// manifest remote info cannot read makes the orphan figures unknown. A
// blob referenced only by that snapshot would otherwise be counted as
// orphaned, and the report would advise running prune.
func TestRemoteInfo_UnreadableManifestLeavesOrphansUnknown(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newListEnv(t)
// The readable manifest references blob A only.
env.addRemote(t, listRemoteID, time.Date(2026, 3, 2, 0, 0, 0, 0, time.UTC))
addBlob(t, env.store.testStorer, testBlobHashA)
addBlob(t, env.store.testStorer, testBlobHashB)
// With every manifest readable, blob B is orphaned.
require.NoError(t, env.v.RemoteInfo(true))
var doc map[string]any
require.NoError(t, json.Unmarshal(env.stdout.Bytes(), &doc))
assert.InDelta(t, 1, doc["orphaned_blob_count"], 0)
// A second snapshot whose manifest cannot be decoded. Blob B may be
// one of its blobs.
unreadableKey := snapshot.RemoteSnapshotKey(listLocalID)
require.NoError(t, env.store.Put(context.Background(),
"metadata/"+unreadableKey+"/manifest.json.zst",
bytes.NewReader([]byte("not a valid manifest"))))
env.stdout.Reset()
require.NoError(t, env.v.RemoteInfo(false))
text := env.stdout.String()
assert.Contains(t, text, "Orphaned (unreferenced): unknown "+
"(1 manifest(s) could not be read, "+
"0 manifest(s) under a non-conforming name skipped)")
assert.NotContains(t, text, "vaultik prune")
env.stdout.Reset()
require.NoError(t, env.v.RemoteInfo(true))
doc = nil
require.NoError(t, json.Unmarshal(env.stdout.Bytes(), &doc))
assert.Contains(t, doc, "orphaned_blob_count")
assert.Nil(t, doc["orphaned_blob_count"])
assert.Contains(t, doc, "orphaned_blob_size")
assert.Nil(t, doc["orphaned_blob_size"])
assert.Equal(t, []any{unreadableKey}, doc["unreadable_manifests"])
}
// TestRemoteInfo_SkipsNonConformingMetadataName checks that a directory
// under metadata/ whose name is not a remote key is left out of the
// report, and that the orphan figures are unknown when it holds a
// manifest. The name comes from the destination store; printed raw, its
// control characters would reach the terminal. Its manifest is not
// read, so a blob only it references would otherwise be counted as
// orphaned.
func TestRemoteInfo_SkipsNonConformingMetadataName(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newListEnv(t)
env.addRemote(t, listRemoteID, time.Date(2026, 3, 2, 0, 0, 0, 0, time.UTC))
addBlob(t, env.store.testStorer, testBlobHashA)
addBlob(t, env.store.testStorer, testBlobHashB)
require.NoError(t, env.store.Put(context.Background(),
"metadata/\x1b[31mred/manifest.json.zst",
bytes.NewReader([]byte("not a valid manifest"))))
require.NoError(t, env.v.RemoteInfo(false))
text := env.stdout.String()
assert.NotContains(t, text, "\x1b")
assert.NotContains(t, text, "31mred")
assert.Contains(t, text, "Total (1 snapshots)")
assert.Contains(t, text, "Orphaned (unreferenced): unknown "+
"(0 manifest(s) could not be read, "+
"1 manifest(s) under a non-conforming name skipped)")
assert.NotContains(t, text, "vaultik prune")
env.stdout.Reset()
require.NoError(t, env.v.RemoteInfo(true))
out := env.stdout.String()
assert.NotContains(t, out, "31mred")
var doc map[string]any
require.NoError(t, json.Unmarshal([]byte(out), &doc))
assert.Contains(t, doc, "orphaned_blob_count")
assert.Nil(t, doc["orphaned_blob_count"])
assert.Contains(t, doc, "orphaned_blob_size")
assert.Nil(t, doc["orphaned_blob_size"])
assert.InDelta(t, 1, doc["skipped_manifest_count"], 0)
assert.NotContains(t, doc, "unreadable_manifests")
}
// TestRemoteInfo_DirectoryWithoutManifestLeavesOrphansKnown checks that
// a directory under metadata/ holding no manifest.json.zst, such as one
// left by a backup interrupted before its manifest upload, leaves the
// orphan figures known. prune does not treat such a directory as a
// snapshot and deletes the blobs the report lists as orphaned.
func TestRemoteInfo_DirectoryWithoutManifestLeavesOrphansKnown(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newListEnv(t)
env.addRemote(t, listRemoteID, time.Date(2026, 3, 2, 0, 0, 0, 0, time.UTC))
addBlob(t, env.store.testStorer, testBlobHashA)
addBlob(t, env.store.testStorer, testBlobHashB)
// One directory under a remote key and one under a non-conforming
// name, each holding only a database.
names := []string{snapshot.RemoteSnapshotKey(listLocalID), "\x1b[31mred"}
for _, name := range names {
require.NoError(t, env.store.Put(context.Background(),
"metadata/"+name+"/db.zst.age",
bytes.NewReader([]byte("not a valid database"))))
}
require.NoError(t, env.v.RemoteInfo(false))
text := env.stdout.String()
assert.NotContains(t, text, "\x1b")
assert.Contains(t, text, "Downloading 1 manifest(s)...")
assert.Contains(t, text, "Orphaned (unreferenced): 1 (")
assert.Contains(t, text, "Run 'vaultik prune' to remove orphaned blobs.")
env.stdout.Reset()
require.NoError(t, env.v.RemoteInfo(true))
var doc map[string]any
require.NoError(t, json.Unmarshal(env.stdout.Bytes(), &doc))
assert.InDelta(t, 1, doc["orphaned_blob_count"], 0)
assert.NotContains(t, doc, "unreadable_manifests")
assert.NotContains(t, doc, "skipped_manifest_count")
}
internal/vaultik/restore.go
+45 -22
View File
@@ -42,6 +42,7 @@ var (
errChunkNotInAnyBlob = errors.New("chunk not found in any blob") errChunkNotInAnyBlob = errors.New("chunk not found in any blob")
errBlobIDNotInHashIndex = errors.New("blob id missing from hash index") errBlobIDNotInHashIndex = errors.New("blob id missing from hash index")
errShortChunkRead = errors.New("short read") errShortChunkRead = errors.New("short read")
errChunkRowMissing = errors.New("chunk has no row in the chunks table")
errRestorePathEscapesTarget = errors.New( errRestorePathEscapesTarget = errors.New(
"refusing to restore path outside the target directory") "refusing to restore path outside the target directory")
errTrailingRestoreData = errors.New( errTrailingRestoreData = errors.New(
@@ -829,10 +830,9 @@ func (v *Vaultik) getFilesToRestore(
// Normalize the filter path // Normalize the filter path
filter = filepath.Clean(filter) filter = filepath.Clean(filter)
// Get files with this prefix files, err := repos.Files.ListUnderPath(ctx, filter)
files, err := repos.Files.ListByPrefix(ctx, filter)
if err != nil { if err != nil {
return nil, fmt.Errorf("listing files with prefix %s: %w", filter, err) return nil, fmt.Errorf("listing files under %s: %w", filter, err)
} }
for _, file := range files { for _, file := range files {
@@ -1054,11 +1054,12 @@ func (s *restoreSession) restoreDirectory(
} }
// applyDirectoryMetadata applies the stored owner, mtime and mode to every // applyDirectoryMetadata applies the stored owner, mtime and mode to every
// restored directory, deepest first, so a parent whose stored mode denies // restored directory, each before its parent, so a parent whose stored
// search does not block its children. Failures are logged at debug level // mode denies search does not block its children. Failures are logged at
// and do not abort the restore. // debug level and do not abort the restore.
func (s *restoreSession) applyDirectoryMetadata() { func (s *restoreSession) applyDirectoryMetadata() {
// A path sorts after its parent's, so reverse order is deepest first. // A path sorts after its parent's, so reverse order puts every
// directory before its parent.
slices.SortFunc(s.directories, func(a, b *database.File) int { slices.SortFunc(s.directories, func(a, b *database.File) int {
return strings.Compare(b.Path.String(), a.Path.String()) return strings.Compare(b.Path.String(), a.Path.String())
}) })
@@ -1073,6 +1074,17 @@ func (s *restoreSession) applyDirectoryMetadata() {
continue continue
} }
// A later entry can have put a symlink in the directory's place,
// for example one stored as "/d/" next to the directory "/d". The
// calls below follow symlinks, so they would change its target.
info, err := lstatIfPossible(s.v.Fs, targetPath)
if err != nil || !info.IsDir() {
log.Debug("Not setting directory metadata: no longer a directory",
"path", targetPath, "error", err)
continue
}
s.applyFileMetadata(dir, targetPath) s.applyFileMetadata(dir, targetPath)
err = s.v.Fs.Chmod(targetPath, os.FileMode(dir.Mode)) err = s.v.Fs.Chmod(targetPath, os.FileMode(dir.Mode))
@@ -1255,7 +1267,7 @@ func (s *restoreSession) writeFileChunks(
blobChunk, ok := s.chunkToBlobMap[chunkHashStr] blobChunk, ok := s.chunkToBlobMap[chunkHashStr]
if !ok { if !ok {
return bytesWritten, timings, fmt.Errorf( return bytesWritten, timings, fmt.Errorf(
"%w: %s", errChunkNotInAnyBlob, chunkHashStr[:16]) "%w: %s", errChunkNotInAnyBlob, shortHash(chunkHashStr))
} }
blobHash, ok := s.blobIDToHash[blobChunk.BlobID.String()] blobHash, ok := s.blobIDToHash[blobChunk.BlobID.String()]
@@ -1272,7 +1284,7 @@ func (s *restoreSession) writeFileChunks(
if err != nil { if err != nil {
return bytesWritten, timings, fmt.Errorf( return bytesWritten, timings, fmt.Errorf(
"reading chunk %s from cached blob %s: %w", "reading chunk %s from cached blob %s: %w",
fc.ChunkHash[:16], blobHash[:16], err) shortHash(chunkHashStr), shortHash(blobHash), err)
} }
t0 = time.Now() t0 = time.Now()
@@ -1470,33 +1482,44 @@ func (v *Vaultik) verifyFile(
chunk, err := repos.Chunks.GetByHash(ctx, fc.ChunkHash.String()) chunk, err := repos.Chunks.GetByHash(ctx, fc.ChunkHash.String())
if err != nil { if err != nil {
return bytesVerified, fmt.Errorf("getting chunk %s: %w", return bytesVerified, fmt.Errorf("getting chunk %s: %w",
fc.ChunkHash.String()[:16], err) shortHash(fc.ChunkHash.String()), err)
} }
// Read chunk data from file if chunk == nil {
chunkData := make([]byte, chunk.Size) return bytesVerified, fmt.Errorf("%w: %s",
errChunkRowMissing, shortHash(fc.ChunkHash.String()))
n, err := io.ReadFull(f, chunkData)
if err != nil {
return bytesVerified, fmt.Errorf("reading chunk data: %w", err)
} }
if int64(n) != chunk.Size { // chunk.Size comes from the snapshot database, which is not
// trusted: reject a negative size, and hash the chunk by
// streaming it rather than allocating that many bytes.
if chunk.Size < 0 {
return bytesVerified, fmt.Errorf("%w: chunk %d size %d",
errNegativeChunkLength, fc.Idx, chunk.Size)
}
hasher := sha256.New()
n, err := io.CopyN(hasher, f, chunk.Size)
if errors.Is(err, io.EOF) {
return bytesVerified, fmt.Errorf("%w: expected %d bytes, got %d", return bytesVerified, fmt.Errorf("%w: expected %d bytes, got %d",
errShortChunkRead, chunk.Size, n) errShortChunkRead, chunk.Size, n)
} }
// Calculate hash and compare if err != nil {
hash := sha256.Sum256(chunkData) return bytesVerified, fmt.Errorf("reading chunk data: %w", err)
actualHash := hex.EncodeToString(hash[:]) }
actualHash := hex.EncodeToString(hasher.Sum(nil))
expectedHash := fc.ChunkHash.String() expectedHash := fc.ChunkHash.String()
if actualHash != expectedHash { if actualHash != expectedHash {
return bytesVerified, fmt.Errorf("%w: chunk %d: expected %s, got %s", return bytesVerified, fmt.Errorf("%w: chunk %d: expected %s, got %s",
errChunkHashMismatch, fc.Idx, expectedHash[:16], actualHash[:16]) errChunkHashMismatch, fc.Idx,
shortHash(expectedHash), shortHash(actualHash))
} }
bytesVerified += int64(n) bytesVerified += n
} }
// The stored chunks account for the whole file, so the reader must // The stored chunks account for the whole file, so the reader must
internal/vaultik/restore_another_machine_test.go
+45 -12
View File
@@ -10,6 +10,7 @@ import (
"github.com/spf13/afero" "github.com/spf13/afero"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/cli"
"sneak.berlin/go/vaultik/internal/config" "sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/database" "sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log" "sneak.berlin/go/vaultik/internal/log"
@@ -27,10 +28,12 @@ import (
// //
// The backup half writes a snapshot with one index and hostname. The // The backup half writes a snapshot with one index and hostname. The
// restore half throws that index away entirely: a fresh, empty index and // restore half throws that index away entirely: a fresh, empty index and
// a config that shares nothing with the original but the storage location // a config written by `config init` and `config set storage_url`, as in
// and the secret key. If restore or verify needed the original local // the README's steps for restoring on another machine, which shares
// index — or the human snapshot ID that only that index holds — this test // nothing with the original but the storage location and the secret key.
// could not run, because the recovery host can know neither. // If restore or verify needed the original local index — or
// the human snapshot ID that only that index holds — this test could not
// run, because the recovery host can know neither.
func TestRestoreOnAnotherMachine(t *testing.T) { func TestRestoreOnAnotherMachine(t *testing.T) {
log.Initialize(log.Config{}) log.Initialize(log.Config{})
t.Parallel() t.Parallel()
@@ -58,7 +61,12 @@ func TestRestoreOnAnotherMachine(t *testing.T) {
// Recovery host: a fresh empty index, a different hostname, and no // Recovery host: a fresh empty index, a different hostname, and no
// age_recipients — only the secret key and the same storage location. // age_recipients — only the secret key and the same storage location.
recovery, stdout := newRecoveryHost(ctx, t, fs, storer) configPath := filepath.Join(tempDir, "config.yml")
runVaultikCommand(t, "--config", configPath, "config", "init")
runVaultikCommand(t, "--config", configPath,
"config", "set", "storage_url", "file://"+storeDir)
recovery, stdout := newRecoveryHost(ctx, t, fs, storer, configPath)
// The recovery index really is empty. This is the assertion that makes // The recovery index really is empty. This is the assertion that makes
// the test a guard against restore quietly depending on the original // the test a guard against restore quietly depending on the original
@@ -93,6 +101,10 @@ func TestRestoreOnAnotherMachine(t *testing.T) {
remote.RemoteKey, &vaultik.VerifyOptions{Deep: true})) remote.RemoteKey, &vaultik.VerifyOptions{Deep: true}))
assertRestoredTreeMatches(t, fs, restoreDir, sourceFiles) assertRestoredTreeMatches(t, fs, restoreDir, sourceFiles)
// With no public key configured, a backup must refuse to start.
err = recovery.CreateSnapshot(&vaultik.SnapshotCreateOptions{Cron: true})
require.ErrorContains(t, err, "age_recipients")
} }
// writeRecoverySourceTree writes a small source tree spanning several // writeRecoverySourceTree writes a small source tree spanning several
@@ -118,14 +130,24 @@ func writeRecoverySourceTree(
} }
// newRecoveryHost builds the Vaultik a replacement machine would run: an // newRecoveryHost builds the Vaultik a replacement machine would run: an
// empty in-memory index, a hostname different from the backup host, no // empty in-memory index, a hostname different from the backup host, and
// age_recipients, and only the secret key plus the shared storer. It // only the secret key plus the shared storer. Its config is read from
// returns the instance and the buffer its stdout is wired to. // configPath by config.Load, as every command reads it. It returns the
// instance and the buffer its stdout is wired to.
func newRecoveryHost( func newRecoveryHost(
ctx context.Context, t *testing.T, fs afero.Fs, storer storage.Storer, ctx context.Context, t *testing.T, fs afero.Fs, storer storage.Storer,
configPath string,
) (*vaultik.Vaultik, *bytes.Buffer) { ) (*vaultik.Vaultik, *bytes.Buffer) {
t.Helper() t.Helper()
cfg, err := config.Load(configPath)
require.NoError(t, err)
// Set directly rather than through VAULTIK_AGE_SECRET_KEY, which a
// parallel test cannot change.
cfg.AgeSecretKey = testAgeSecretKey
cfg.Hostname = "recovery-host"
recoveryDB, err := database.New(ctx, ":memory:") recoveryDB, err := database.New(ctx, ":memory:")
require.NoError(t, err) require.NoError(t, err)
t.Cleanup(func() { _ = recoveryDB.Close() }) t.Cleanup(func() { _ = recoveryDB.Close() })
@@ -133,10 +155,7 @@ func newRecoveryHost(
stdout := &bytes.Buffer{} stdout := &bytes.Buffer{}
recovery := &vaultik.Vaultik{ recovery := &vaultik.Vaultik{
Config: &config.Config{ Config: cfg,
AgeSecretKey: testAgeSecretKey,
Hostname: "recovery-host",
},
Storage: storer, Storage: storer,
Fs: fs, Fs: fs,
Repositories: database.NewRepositories(recoveryDB), Repositories: database.NewRepositories(recoveryDB),
@@ -150,6 +169,20 @@ func newRecoveryHost(
return recovery, stdout return recovery, stdout
} }
// runVaultikCommand runs one vaultik command line in-process and fails the
// test if it returns an error. The command writes the cli package's global
// flag variables, so it must not be called from two tests that run at the
// same time.
func runVaultikCommand(t *testing.T, args ...string) {
t.Helper()
cmd := cli.NewRootCommand()
cmd.SetArgs(args)
cmd.SetOut(io.Discard)
cmd.SetErr(io.Discard)
require.NoError(t, cmd.Execute())
}
// assertRestoredTreeMatches byte-compares every restored file against its // assertRestoredTreeMatches byte-compares every restored file against its
// source content. // source content.
func assertRestoredTreeMatches( func assertRestoredTreeMatches(
internal/vaultik/restore_interrupt_test.go
+1 -2
View File
@@ -164,10 +164,9 @@ func scratchEntries(t *testing.T, dir string) []string {
// restore while a blob download is in progress. The download fails only // restore while a blob download is in progress. The download fails only
// because of the cancel, so Restore must return context.Canceled without // because of the cancel, so Restore must return context.Canceled without
// reporting the file that needs the blob as failed. // reporting the file that needs the blob as failed.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestRestoreSkipErrorsCancelDuringBlobDownload(t *testing.T) { func TestRestoreSkipErrorsCancelDuringBlobDownload(t *testing.T) {
log.Initialize(log.Config{}) log.Initialize(log.Config{})
t.Parallel()
fs := afero.NewOsFs() fs := afero.NewOsFs()
tempDir := t.TempDir() tempDir := t.TempDir()
internal/vaultik/restore_malformed_db_test.go
+221
View File
@@ -0,0 +1,221 @@
package vaultik //nolint:testpackage // drives unexported restore and verify steps
import (
"context"
"math"
"path/filepath"
"strings"
"testing"
"time"
"github.com/spf13/afero"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/types"
)
// These tests feed restore and --verify a snapshot database written by
// hand, as a damaged or hostile store could serve one. Each malformed row
// must end in an error, not a panic.
// shortChunkHash is shorter than the hash prefix that error messages print.
const shortChunkHash = "abc"
// restoredFileContent is the content of the restored file under verify.
const restoredFileContent = "xyz"
// craftedSnapshotDB opens an empty snapshot database in a temp directory.
func craftedSnapshotDB(t *testing.T) (*database.DB, *database.Repositories) {
t.Helper()
db, err := database.New(context.Background(),
filepath.Join(t.TempDir(), "snapshot.db"))
require.NoError(t, err)
t.Cleanup(func() { _ = db.Close() })
return db, database.NewRepositories(db)
}
// craftedFile adds a regular file whose only chunk has the given hash.
// Adding the chunks row, if any, is left to the caller.
func craftedFile(
t *testing.T, repos *database.Repositories, chunkHash string,
) *database.File {
t.Helper()
ctx := context.Background()
file := &database.File{
Path: "/src/f",
MTime: time.Now().UTC(),
Size: int64(len(restoredFileContent)),
Mode: 0o644,
}
require.NoError(t, repos.Files.Create(ctx, nil, file))
require.NoError(t, repos.FileChunks.Create(ctx, nil, &database.FileChunk{
FileID: file.ID,
ChunkHash: types.ChunkHash(chunkHash),
}))
return file
}
// TestRestoreShortChunkHashInNoBlob proves a file whose short chunk hash
// has no blob_chunks row fails restore planning and the chunk write with
// an error.
func TestRestoreShortChunkHashInNoBlob(t *testing.T) {
t.Parallel()
ctx := context.Background()
_, repos := craftedSnapshotDB(t)
require.NoError(t, repos.Chunks.Create(ctx, nil,
&database.Chunk{ChunkHash: shortChunkHash, Size: 3}))
file := craftedFile(t, repos, shortChunkHash)
v := NewForTesting(nil)
chunkToBlobMap, err := v.buildChunkToBlobMap(ctx, repos)
require.NoError(t, err)
_, err = newRestorePlan(ctx, repos, []*database.File{file},
chunkToBlobMap, map[string]string{})
require.ErrorIs(t, err, errPlanChunkMissing)
fileChunks, err := repos.FileChunks.GetByFileID(ctx, file.ID)
require.NoError(t, err)
out, err := afero.NewMemMapFs().Create("out")
require.NoError(t, err)
session := &restoreSession{
v: v.Vaultik, ctx: ctx, chunkToBlobMap: chunkToBlobMap,
}
_, _, err = session.writeFileChunks(out, fileChunks)
require.ErrorIs(t, err, errChunkNotInAnyBlob)
}
// TestRestoreShortChunkHashReadPastBlobEnd proves a short chunk hash
// whose blob_chunks row reads past the end of its blob fails the chunk
// write with an error.
func TestRestoreShortChunkHashReadPastBlobEnd(t *testing.T) {
t.Parallel()
ctx := context.Background()
_, repos := craftedSnapshotDB(t)
blobHash := strings.Repeat("b", blobHashHexLen)
blob := &database.Blob{
ID: types.NewBlobID(),
Hash: types.BlobHash(blobHash),
CreatedTS: time.Now().UTC(),
}
require.NoError(t, repos.Blobs.Create(ctx, nil, blob))
require.NoError(t, repos.Chunks.Create(ctx, nil,
&database.Chunk{ChunkHash: shortChunkHash, Size: 3}))
require.NoError(t, repos.BlobChunks.Create(ctx, nil, &database.BlobChunk{
BlobID: blob.ID,
ChunkHash: shortChunkHash,
Length: 100,
}))
file := craftedFile(t, repos, shortChunkHash)
cache, err := newBlobDiskCache(1 << 20)
require.NoError(t, err)
t.Cleanup(func() { _ = cache.Close() })
require.NoError(t, cache.Put(blobHash, []byte("abc")))
v := NewForTesting(nil)
chunkToBlobMap, err := v.buildChunkToBlobMap(ctx, repos)
require.NoError(t, err)
_, blobIDToHash, err := v.buildBlobIndexes(repos)
require.NoError(t, err)
fileChunks, err := repos.FileChunks.GetByFileID(ctx, file.ID)
require.NoError(t, err)
out, err := afero.NewMemMapFs().Create("out")
require.NoError(t, err)
session := &restoreSession{
v: v.Vaultik,
ctx: ctx,
chunkToBlobMap: chunkToBlobMap,
blobIDToHash: blobIDToHash,
blobCache: cache,
}
_, _, err = session.writeFileChunks(out, fileChunks)
require.ErrorIs(t, err, errCacheReadBeyondBlob)
}
// TestVerifyFileMalformedChunkRow proves --verify returns an error for a
// chunk with no chunks row, a short chunk hash, and a chunk size from the
// database that is negative or larger than the restored file.
func TestVerifyFileMalformedChunkRow(t *testing.T) {
t.Parallel()
fullHash := types.ChunkHash(strings.Repeat("c", blobHashHexLen))
tests := []struct {
name string
hash types.ChunkHash
chunk *database.Chunk // nil adds no chunks row
want error
}{
{
name: "missing chunk row",
hash: fullHash,
want: errChunkRowMissing,
},
{
name: "short hash",
hash: shortChunkHash,
chunk: &database.Chunk{ChunkHash: shortChunkHash, Size: 3},
want: errChunkHashMismatch,
},
{
name: "size larger than the file",
hash: fullHash,
chunk: &database.Chunk{ChunkHash: fullHash, Size: math.MaxInt64},
want: errShortChunkRead,
},
{
name: "negative size",
hash: fullHash,
chunk: &database.Chunk{ChunkHash: fullHash, Size: -1},
want: errNegativeChunkLength,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
ctx := context.Background()
db, repos := craftedSnapshotDB(t)
if tt.chunk == nil {
// A crafted database need not satisfy its foreign keys.
_, err := db.Conn().ExecContext(ctx, "PRAGMA foreign_keys = OFF")
require.NoError(t, err)
} else {
require.NoError(t, repos.Chunks.Create(ctx, nil, tt.chunk))
}
file := craftedFile(t, repos, tt.hash.String())
v := NewForTesting(nil)
v.Fs = afero.NewMemMapFs()
require.NoError(t, afero.WriteFile(v.Fs, "/restore/f",
[]byte(restoredFileContent), 0o600))
_, err := v.verifyFile(ctx, repos, file, "/restore/f")
require.ErrorIs(t, err, tt.want)
})
}
}
internal/vaultik/restore_metadata_test.go
+168 -44
View File
@@ -17,8 +17,7 @@ import (
) )
// These tests check that restore applies each entry's owner, mode and // These tests check that restore applies each entry's owner, mode and
// mtime in an order that keeps them. They are written to fail as a normal // mtime in an order that keeps them.
// user; as root a read-only directory does not stop a write.
const ( const (
readOnlyDirMode = uint32(os.ModeDir | 0o555) readOnlyDirMode = uint32(os.ModeDir | 0o555)
@@ -29,33 +28,126 @@ const (
otherOwnerID = uint32(4321) otherOwnerID = uint32(4321)
writableTestMode = 0o755 writableTestMode = 0o755
symlinkTargetPath = "/nonexistent/target" symlinkTargetPath = "/nonexistent/target"
memTargetDir = "/restore"
// Owner bits a normal user needs on a directory to create an entry
// in it, and to change an entry in it.
ownerWriteAndSearch = os.FileMode(0o300)
ownerSearch = os.FileMode(0o100)
) )
// normalUserFs refuses what the kernel refuses a normal user. make test
// runs as root, which a read-only or unsearchable directory does not
// stop, so without it the tests below could not fail there. Creating an
// entry needs owner write and search on the directory holding it;
// changing an entry's mode or times needs owner search. Only that one
// directory is checked, not every ancestor.
type normalUserFs struct {
afero.Fs
}
//nolint:ireturn // afero.Fs.OpenFile is defined to return the interface
func (fs normalUserFs) OpenFile(
name string, flag int, perm os.FileMode,
) (afero.File, error) {
if flag&os.O_CREATE != 0 {
err := fs.checkParent(name, ownerWriteAndSearch)
if err != nil {
return nil, err
}
}
return fs.Fs.OpenFile(name, flag, perm)
}
func (fs normalUserFs) MkdirAll(path string, perm os.FileMode) error {
_, err := fs.Stat(path)
if err != nil {
err = fs.checkParent(path, ownerWriteAndSearch)
if err != nil {
return err
}
}
return fs.Fs.MkdirAll(path, perm)
}
func (fs normalUserFs) Chmod(name string, mode os.FileMode) error {
err := fs.checkParent(name, ownerSearch)
if err != nil {
return err
}
return fs.Fs.Chmod(name, mode)
}
func (fs normalUserFs) Chtimes(name string, atime, mtime time.Time) error {
err := fs.checkParent(name, ownerSearch)
if err != nil {
return err
}
return fs.Fs.Chtimes(name, atime, mtime)
}
// checkParent returns a permission error when the directory holding name
// exists and its owner bits lack any of need.
func (fs normalUserFs) checkParent(name string, need os.FileMode) error {
info, err := fs.Stat(filepath.Dir(name))
if err == nil && info.Mode().Perm()&need != need {
return &os.PathError{Op: "access", Path: name, Err: os.ErrPermission}
}
return nil
}
// TestRestoreFillsReadOnlyDirectory checks that a read-only directory // TestRestoreFillsReadOnlyDirectory checks that a read-only directory
// still receives the entries inside it, and that it ends with its stored // still receives the entries inside it, and ends with its stored mode.
// mode and mtime even though entries were written into it.
func TestRestoreFillsReadOnlyDirectory(t *testing.T) { func TestRestoreFillsReadOnlyDirectory(t *testing.T) {
log.Initialize(log.Config{}) log.Initialize(log.Config{})
t.Parallel() t.Parallel()
ctx := context.Background()
rows, repos := makeFiles(ctx, t, []*database.File{
{Path: "/ro", Mode: readOnlyDirMode},
{Path: "/ro/file", Mode: plainFileMode},
{Path: "/ro/sub", Mode: readOnlyDirMode},
{Path: "/ro/sub/file", Mode: plainFileMode},
})
fs := afero.NewMemMapFs()
v := newContainmentVaultik(ctx, normalUserFs{Fs: fs})
_, err := v.restoreAllFiles(rows, repos,
&RestoreOptions{TargetDir: memTargetDir}, nil, nil)
require.NoError(t, err)
for _, path := range []string{"/ro/file", "/ro/sub/file"} {
_, err := fs.Stat(filepath.Join(memTargetDir, path))
require.NoErrorf(t, err, "file inside a read-only directory: %s", path)
}
for _, dir := range []string{"/ro", "/ro/sub"} {
info, err := fs.Stat(filepath.Join(memTargetDir, dir))
require.NoError(t, err)
assert.Equalf(t, os.FileMode(0o555), info.Mode().Perm(), "mode of %s", dir)
}
}
// TestRestoreKeepsNonEmptyDirectoryMTime checks that a directory keeps
// its stored mtime although entries were written into it. It runs on the
// real filesystem, where writing an entry changes its directory's mtime.
func TestRestoreKeepsNonEmptyDirectoryMTime(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background() ctx := context.Background()
targetDir := t.TempDir() targetDir := t.TempDir()
mtime := time.Date(2001, time.February, 3, 4, 5, 6, 0, time.UTC) mtime := time.Date(2001, time.February, 3, 4, 5, 6, 0, time.UTC)
rows, repos := makeFiles(ctx, t, []*database.File{ rows, repos := makeFiles(ctx, t, []*database.File{
{Path: "/ro", Mode: readOnlyDirMode, MTime: mtime}, {Path: "/dir", Mode: plainDirMode, MTime: mtime},
{Path: "/ro/file", Mode: plainFileMode, MTime: mtime}, {Path: "/dir/file", Mode: plainFileMode, MTime: mtime},
{Path: "/ro/sub", Mode: readOnlyDirMode, MTime: mtime},
{Path: "/ro/sub/file", Mode: plainFileMode, MTime: mtime},
})
roDir := filepath.Join(targetDir, "ro")
subDir := filepath.Join(roDir, "sub")
// Let t.TempDir remove the tree afterwards.
t.Cleanup(func() {
_ = os.Chmod(roDir, writableTestMode)
_ = os.Chmod(subDir, writableTestMode)
}) })
v := newContainmentVaultik(ctx, afero.NewOsFs()) v := newContainmentVaultik(ctx, afero.NewOsFs())
@@ -63,20 +155,10 @@ func TestRestoreFillsReadOnlyDirectory(t *testing.T) {
&RestoreOptions{TargetDir: targetDir}, nil, nil) &RestoreOptions{TargetDir: targetDir}, nil, nil)
require.NoError(t, err) require.NoError(t, err)
for _, path := range []string{ info, err := os.Stat(filepath.Join(targetDir, "dir"))
filepath.Join(roDir, "file"), filepath.Join(subDir, "file"), require.NoError(t, err)
} { assert.Truef(t, info.ModTime().Equal(mtime),
_, err := os.Stat(path) "directory mtime is %s, stored %s", info.ModTime(), mtime)
require.NoErrorf(t, err, "file inside a read-only directory: %s", path)
}
for _, dir := range []string{roDir, subDir} {
info, err := os.Stat(dir)
require.NoError(t, err)
assert.Equalf(t, os.FileMode(0o555), info.Mode().Perm(), "mode of %s", dir)
assert.Truef(t, info.ModTime().Equal(mtime),
"mtime of %s is %s, stored %s", dir, info.ModTime(), mtime)
}
} }
// TestRestoreFinishesChildBeforeUnsearchableParent checks that a // TestRestoreFinishesChildBeforeUnsearchableParent checks that a
@@ -87,7 +169,6 @@ func TestRestoreFinishesChildBeforeUnsearchableParent(t *testing.T) {
t.Parallel() t.Parallel()
ctx := context.Background() ctx := context.Background()
targetDir := t.TempDir()
mtime := time.Date(2001, time.February, 3, 4, 5, 6, 0, time.UTC) mtime := time.Date(2001, time.February, 3, 4, 5, 6, 0, time.UTC)
rows, repos := makeFiles(ctx, t, []*database.File{ rows, repos := makeFiles(ctx, t, []*database.File{
@@ -95,30 +176,73 @@ func TestRestoreFinishesChildBeforeUnsearchableParent(t *testing.T) {
{Path: "/locked/sub", Mode: plainDirMode, MTime: mtime}, {Path: "/locked/sub", Mode: plainDirMode, MTime: mtime},
}) })
lockedDir := filepath.Join(targetDir, "locked") fs := afero.NewMemMapFs()
v := newContainmentVaultik(ctx, normalUserFs{Fs: fs})
// Let t.TempDir remove the tree afterwards.
t.Cleanup(func() { _ = os.Chmod(lockedDir, writableTestMode) })
v := newContainmentVaultik(ctx, afero.NewOsFs())
_, err := v.restoreAllFiles(rows, repos, _, err := v.restoreAllFiles(rows, repos,
&RestoreOptions{TargetDir: targetDir}, nil, nil) &RestoreOptions{TargetDir: memTargetDir}, nil, nil)
require.NoError(t, err) require.NoError(t, err)
info, err := os.Stat(lockedDir) info, err := fs.Stat(filepath.Join(memTargetDir, "locked"))
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, os.FileMode(0o600), info.Mode().Perm()) assert.Equal(t, os.FileMode(0o600), info.Mode().Perm())
// Open the parent so a normal user can look inside it. info, err = fs.Stat(filepath.Join(memTargetDir, "locked", "sub"))
require.NoError(t, os.Chmod(lockedDir, writableTestMode))
info, err = os.Stat(filepath.Join(lockedDir, "sub"))
require.NoError(t, err) require.NoError(t, err)
assert.Equal(t, os.FileMode(0o755), info.Mode().Perm()) assert.Equal(t, os.FileMode(0o755), info.Mode().Perm())
assert.Truef(t, info.ModTime().Equal(mtime), assert.Truef(t, info.ModTime().Equal(mtime),
"mtime of sub is %s, stored %s", info.ModTime(), mtime) "mtime of sub is %s, stored %s", info.ModTime(), mtime)
} }
// TestRestoreLeavesSymlinkedDirectoryTargetAlone checks that a directory
// whose place a later entry takes with a symlink does not hand its stored
// owner, mode and mtime to whatever the symlink points at. "/d" and "/d/"
// are different stored paths for the same place on disk.
func TestRestoreLeavesSymlinkedDirectoryTargetAlone(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
tempDir := t.TempDir()
targetDir := filepath.Join(tempDir, "target")
outsideDir := filepath.Join(tempDir, "outside")
require.NoError(t, os.Mkdir(outsideDir, writableTestMode))
before, err := os.Stat(outsideDir)
require.NoError(t, err)
mtime := time.Date(2001, time.February, 3, 4, 5, 6, 0, time.UTC)
rows, repos := makeFiles(ctx, t, []*database.File{
{
Path: "/d",
Mode: readOnlyDirMode,
UID: otherOwnerID,
GID: otherOwnerID,
MTime: mtime,
},
{Path: "/d/", LinkTarget: types.FilePath(outsideDir), MTime: mtime},
})
v := newContainmentVaultik(ctx, afero.NewOsFs())
_, err = v.restoreAllFiles(rows, repos,
&RestoreOptions{TargetDir: targetDir}, nil, nil)
require.NoError(t, err)
after, err := os.Stat(outsideDir)
require.NoError(t, err)
assert.Equal(t, before.Mode(), after.Mode())
assert.Truef(t, after.ModTime().Equal(before.ModTime()),
"mtime changed from %s to %s", before.ModTime(), after.ModTime())
beforeOwner, ok := before.Sys().(*syscall.Stat_t)
require.True(t, ok)
afterOwner, ok := after.Sys().(*syscall.Stat_t)
require.True(t, ok)
assert.Equal(t, beforeOwner.Uid, afterOwner.Uid)
assert.Equal(t, beforeOwner.Gid, afterOwner.Gid)
}
// TestRestoreKeepsSetuidThroughChown checks that a setuid file keeps the // TestRestoreKeepsSetuidThroughChown checks that a setuid file keeps the
// bit when restore changes its owner. Linux clears setuid on any chown of // bit when restore changes its owner. Linux clears setuid on any chown of
// a regular file, even one to its current owner, so the file is recorded // a regular file, even one to its current owner, so the file is recorded
internal/vaultik/restore_plan.go
+1 -1
View File
@@ -75,7 +75,7 @@ func newRestorePlan(
bc, ok := chunkToBlobMap[fc.ChunkHash.String()] bc, ok := chunkToBlobMap[fc.ChunkHash.String()]
if !ok { if !ok {
return nil, fmt.Errorf("planning %s: %w: %s", return nil, fmt.Errorf("planning %s: %w: %s",
f.Path, errPlanChunkMissing, fc.ChunkHash.String()[:16]) f.Path, errPlanChunkMissing, shortHash(fc.ChunkHash.String()))
} }
hash, ok := blobIDToHash[bc.BlobID.String()] hash, ok := blobIDToHash[bc.BlobID.String()]
internal/vaultik/restore_skip_errors_test.go
+6 -5
View File
@@ -45,9 +45,10 @@ type missingBlobBackup struct {
// after one blob of a two-blob snapshot was deleted. Every file stored in // after one blob of a two-blob snapshot was deleted. Every file stored in
// that blob must be reported as failed and left absent, every other file // that blob must be reported as failed and left absent, every other file
// must be restored intact, and Restore must still return an error. // must be restored intact, and Restore must still return an error.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestRestoreSkipErrorsSkipsFilesOfMissingBlob(t *testing.T) { func TestRestoreSkipErrorsSkipsFilesOfMissingBlob(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background() ctx := context.Background()
backup := backupThenDeleteOneBlob(ctx, t) backup := backupThenDeleteOneBlob(ctx, t)
@@ -85,9 +86,10 @@ func TestRestoreSkipErrorsSkipsFilesOfMissingBlob(t *testing.T) {
// TestRestoreMissingBlobAbortsWithoutSkipErrors checks that a deleted blob // TestRestoreMissingBlobAbortsWithoutSkipErrors checks that a deleted blob
// still ends the restore with an error when SkipErrors is not set. // still ends the restore with an error when SkipErrors is not set.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestRestoreMissingBlobAbortsWithoutSkipErrors(t *testing.T) { func TestRestoreMissingBlobAbortsWithoutSkipErrors(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background() ctx := context.Background()
backup := backupThenDeleteOneBlob(ctx, t) backup := backupThenDeleteOneBlob(ctx, t)
@@ -107,7 +109,6 @@ func backupThenDeleteOneBlob(
ctx context.Context, t *testing.T, ctx context.Context, t *testing.T,
) *missingBlobBackup { ) *missingBlobBackup {
t.Helper() t.Helper()
log.Initialize(log.Config{})
fs := afero.NewOsFs() fs := afero.NewOsFs()
tempDir := t.TempDir() tempDir := t.TempDir()
internal/vaultik/same_second_rewrite_test.go
+72
View File
@@ -0,0 +1,72 @@
package vaultik_test
import (
"bytes"
"context"
"path/filepath"
"testing"
"time"
"github.com/spf13/afero"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// A file rewritten with its size unchanged and a new mtime in the same
// second as the mtime the index holds must still be backed up. See
// https://git.eeqj.de/sneak/vaultik/issues/226.
func TestBackupOfSameSecondRewriteRestoresNewContent(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
fs := afero.NewOsFs()
tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "src")
storeDir := filepath.Join(tempDir, "remote")
restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite")
rewrittenPath := filepath.Join(dataDir, "small.txt")
ctx := context.Background()
files := writeFaultSourceTree(t, fs, dataDir)
cfg := changedFileConfig(dataDir, dbPath)
firstMTime := time.Date(2026, time.January, 2, 3, 4, 5, 0, time.UTC).
Add(100 * time.Millisecond)
secondMTime := firstMTime.Add(800 * time.Millisecond)
require.NoError(t, fs.Chtimes(rewrittenPath, firstMTime, firstMTime))
store, err := storage.NewFileStorer(storeDir)
require.NoError(t, err)
db, err := database.New(ctx, dbPath)
require.NoError(t, err)
repos := database.NewRepositories(db)
v := newBackupVaultik(ctx, cfg, store, repos, db, fs)
require.NoError(t, backUp(v, "first"))
// Upper-casing ASCII text keeps its size.
files[rewrittenPath] = bytes.ToUpper(files[rewrittenPath])
require.NoError(t, afero.WriteFile(fs, rewrittenPath, files[rewrittenPath], 0o644))
require.NoError(t, fs.Chtimes(rewrittenPath, secondMTime, secondMTime))
require.NoError(t, backUp(v, "second"))
id := localSnapshotID(ctx, t, repos, "second")
require.NoError(t, db.Close())
reader := newReaderVaultik(ctx, cfg, store, nil, fs)
require.NoError(t, reader.Restore(&vaultik.RestoreOptions{
SnapshotID: id,
TargetDir: restoreDir,
Verify: true,
}))
assertRestoredTree(t, fs, restoreDir, files)
}
internal/vaultik/snapshot.go
+101 -75
View File
@@ -14,6 +14,7 @@ import (
"sneak.berlin/go/vaultik/internal/log" "sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot" "sneak.berlin/go/vaultik/internal/snapshot"
"sneak.berlin/go/vaultik/internal/types"
) )
// Sentinel errors for snapshot management. // Sentinel errors for snapshot management.
@@ -23,6 +24,9 @@ var (
errSnapshotVerifyFailed = errors.New("verification failed") errSnapshotVerifyFailed = errors.New("verification failed")
errRemoveAllNeedsForce = errors.New("--all requires --force") errRemoveAllNeedsForce = errors.New("--all requires --force")
errInvalidTableName = errors.New("invalid table name") errInvalidTableName = errors.New("invalid table name")
errNoAgeRecipients = errors.New(
"creating a snapshot needs at least one public key in " +
"age_recipients (generate a keypair with: age-keygen)")
) )
// listRecentLimit caps how many snapshot rows are fetched from the // listRecentLimit caps how many snapshot rows are fetched from the
@@ -42,6 +46,12 @@ type SnapshotCreateOptions struct {
// CreateSnapshot executes the snapshot creation operation // CreateSnapshot executes the snapshot creation operation
func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error { func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
// config.Load accepts an empty list, since listing, verifying and
// restoring need no public key.
if len(v.Config.AgeRecipients) == 0 {
return errNoAgeRecipients
}
overallStartTime := time.Now() overallStartTime := time.Now()
log.Info("Starting snapshot creation", log.Info("Starting snapshot creation",
@@ -180,6 +190,11 @@ type snapshotStats struct {
totalBytesUploaded int64 totalBytesUploaded int64
totalBlobsUploaded int totalBlobsUploaded int
uploadDuration time.Duration uploadDuration time.Duration
// The sizes of all blobs the snapshot references, set by
// finalizeSnapshotMetadata once snapshot_blobs is populated.
blobSize int64
blobUncompressedSize int64
} }
// createNamedSnapshot creates a single named snapshot // createNamedSnapshot creates a single named snapshot
@@ -219,8 +234,6 @@ func (v *Vaultik) createNamedSnapshot(
return err return err
} }
v.collectUploadStats(scanner, stats)
err = v.finalizeSnapshotMetadata(snapshotID, stats) err = v.finalizeSnapshotMetadata(snapshotID, stats)
if err != nil { if err != nil {
return err return err
@@ -272,6 +285,11 @@ func (v *Vaultik) resolveSnapshotPaths(snapName string) ([]string, error) {
func (v *Vaultik) scanAllDirectories( func (v *Vaultik) scanAllDirectories(
scanner *snapshot.Scanner, resolvedDirs []string, snapshotID string, scanner *snapshot.Scanner, resolvedDirs []string, snapshotID string,
) (*snapshotStats, error) { ) (*snapshotStats, error) {
if progress := scanner.GetProgress(); progress != nil {
progress.Start()
defer progress.Stop()
}
stats := &snapshotStats{} stats := &snapshotStats{}
for i, dir := range resolvedDirs { for i, dir := range resolvedDirs {
@@ -300,6 +318,9 @@ func (v *Vaultik) scanAllDirectories(
stats.totalBytesSkipped += result.BytesSkipped stats.totalBytesSkipped += result.BytesSkipped
stats.totalFilesDeleted += result.FilesDeleted stats.totalFilesDeleted += result.FilesDeleted
stats.totalBytesDeleted += result.BytesDeleted stats.totalBytesDeleted += result.BytesDeleted
stats.totalBlobsUploaded += result.BlobsUploaded
stats.totalBytesUploaded += result.BytesUploaded
stats.uploadDuration += result.UploadDuration
log.Info("Directory scan complete", log.Info("Directory scan complete",
"path", dir, "path", dir,
@@ -315,18 +336,6 @@ func (v *Vaultik) scanAllDirectories(
return stats, nil return stats, nil
} }
// collectUploadStats gathers upload statistics from the scanner's
// progress reporter.
func (v *Vaultik) collectUploadStats(scanner *snapshot.Scanner, stats *snapshotStats) {
if s := scanner.GetProgress(); s != nil {
progressStats := s.GetStats()
stats.totalBytesUploaded = progressStats.BytesUploaded.Load()
stats.totalBlobsUploaded = int(progressStats.BlobsUploaded.Load())
stats.uploadDuration = time.Duration(
progressStats.UploadDurationMs.Load()) * time.Millisecond
}
}
// finalizeSnapshotMetadata updates stats, exports metadata, and only then // finalizeSnapshotMetadata updates stats, exports metadata, and only then
// marks the snapshot complete. Recording completion last is deliberate: an // marks the snapshot complete. Recording completion last is deliberate: an
// export interrupted by a crash leaves the snapshot incomplete rather than // export interrupted by a crash leaves the snapshot incomplete rather than
@@ -336,31 +345,39 @@ func (v *Vaultik) collectUploadStats(scanner *snapshot.Scanner, stats *snapshotS
func (v *Vaultik) finalizeSnapshotMetadata( func (v *Vaultik) finalizeSnapshotMetadata(
snapshotID string, stats *snapshotStats, snapshotID string, stats *snapshotStats,
) error { ) error {
// snapshot_blobs must be populated before the blob sizes below, which
// total the snapshot's blobs, and before the export, which builds the
// manifest and the trimmed metadata database from it.
err := v.SnapshotManager.PopulateSnapshotBlobs(v.ctx, snapshotID)
if err != nil {
return fmt.Errorf("populating snapshot blobs: %w", err)
}
stats.blobSize, stats.blobUncompressedSize, err =
v.Repositories.Snapshots.GetSnapshotBlobSizes(v.ctx, snapshotID)
if err != nil {
return fmt.Errorf("getting snapshot blob sizes: %w", err)
}
extStats := snapshot.ExtendedBackupStats{ extStats := snapshot.ExtendedBackupStats{
BackupStats: snapshot.BackupStats{ BackupStats: snapshot.BackupStats{
FilesScanned: stats.totalFiles, FilesScanned: stats.totalFiles,
BytesScanned: stats.totalBytes, TotalSize: stats.totalBytes + stats.totalBytesSkipped,
ChunksCreated: stats.totalChunks, ChunksCreated: stats.totalChunks,
BlobsCreated: stats.totalBlobs, BlobsCreated: stats.totalBlobs,
BytesUploaded: stats.totalBytesUploaded, BytesUploaded: stats.totalBytesUploaded,
}, },
BlobUncompressedSize: 0, BlobSize: stats.blobSize,
BlobUncompressedSize: stats.blobUncompressedSize,
CompressionLevel: v.Config.CompressionLevel, CompressionLevel: v.Config.CompressionLevel,
UploadDurationMs: stats.uploadDuration.Milliseconds(), UploadDurationMs: stats.uploadDuration.Milliseconds(),
} }
err := v.SnapshotManager.UpdateSnapshotStatsExtended(v.ctx, snapshotID, extStats) err = v.SnapshotManager.UpdateSnapshotStatsExtended(v.ctx, snapshotID, extStats)
if err != nil { if err != nil {
return fmt.Errorf("updating snapshot stats: %w", err) return fmt.Errorf("updating snapshot stats: %w", err)
} }
// snapshot_blobs must be populated before the export, which builds the
// manifest and the trimmed metadata database from it.
err = v.SnapshotManager.PopulateSnapshotBlobs(v.ctx, snapshotID)
if err != nil {
return fmt.Errorf("populating snapshot blobs: %w", err)
}
err = v.SnapshotManager.ExportSnapshotMetadata( err = v.SnapshotManager.ExportSnapshotMetadata(
v.ctx, v.Config.IndexPath, snapshotID) v.ctx, v.Config.IndexPath, snapshotID)
if err != nil { if err != nil {
@@ -395,12 +412,10 @@ func (v *Vaultik) printSnapshotSummary(
totalFilesChanged := stats.totalFiles - stats.totalFilesSkipped totalFilesChanged := stats.totalFiles - stats.totalFilesSkipped
totalBytesAll := stats.totalBytes + stats.totalBytesSkipped totalBytesAll := stats.totalBytes + stats.totalBytesSkipped
// Get total blob sizes from database
compressedSize, uncompressedSize := v.getSnapshotBlobSizes(snapshotID)
var compressionRatio float64 var compressionRatio float64
if uncompressedSize > 0 { if stats.blobUncompressedSize > 0 {
compressionRatio = float64(compressedSize) / float64(uncompressedSize) compressionRatio = float64(stats.blobSize) /
float64(stats.blobUncompressedSize)
} else { } else {
compressionRatio = 1.0 compressionRatio = 1.0
} }
@@ -428,8 +443,8 @@ func (v *Vaultik) printSnapshotSummary(
if stats.totalBlobsUploaded > 0 { if stats.totalBlobsUploaded > 0 {
v.UI.Detailf("Storage: %s compressed from %s (%.2fx ratio).", v.UI.Detailf("Storage: %s compressed from %s (%.2fx ratio).",
v.UI.Size(compressedSize), v.UI.Size(stats.blobSize),
v.UI.Size(uncompressedSize), v.UI.Size(stats.blobUncompressedSize),
compressionRatio) compressionRatio)
v.UI.Detailf("Upload: %d blobs, %s in %s (%s).", v.UI.Detailf("Upload: %d blobs, %s in %s (%s).",
stats.totalBlobsUploaded, stats.totalBlobsUploaded,
@@ -441,27 +456,6 @@ func (v *Vaultik) printSnapshotSummary(
v.UI.Detailf("Snapshot create duration: %s.", v.UI.Duration(snapshotDuration)) v.UI.Detailf("Snapshot create duration: %s.", v.UI.Duration(snapshotDuration))
} }
// getSnapshotBlobSizes returns total compressed and uncompressed blob
// sizes for a snapshot.
func (v *Vaultik) getSnapshotBlobSizes(snapshotID string) (int64, int64) {
var compressed, uncompressed int64
blobHashes, err := v.Repositories.Snapshots.GetBlobHashes(v.ctx, snapshotID)
if err != nil {
return 0, 0
}
for _, hash := range blobHashes {
blob, err := v.Repositories.Blobs.GetByHash(v.ctx, hash)
if err == nil && blob != nil {
compressed += blob.CompressedSize
uncompressed += blob.UncompressedSize
}
}
return compressed, uncompressed
}
// SnapshotPurgeOptions contains options for the snapshot purge command. // SnapshotPurgeOptions contains options for the snapshot purge command.
type SnapshotPurgeOptions struct { type SnapshotPurgeOptions struct {
KeepLatest bool // Keep only the most recent snapshot per name KeepLatest bool // Keep only the most recent snapshot per name
@@ -502,19 +496,23 @@ func (v *Vaultik) PurgeSnapshotsWithOptions(opts *SnapshotPurgeOptions) error {
nameFilter[n] = struct{}{} nameFilter[n] = struct{}{}
} }
// Collect completed snapshots, applying the name filter. // Collect completed snapshots and their names, applying the name filter.
snapshots := make([]SnapshotInfo, 0, len(dbSnapshots)) snapshots := make([]SnapshotInfo, 0, len(dbSnapshots))
names := make(map[types.SnapshotID]string, len(dbSnapshots))
for _, s := range dbSnapshots { for _, s := range dbSnapshots {
if s.CompletedAt == nil { if s.CompletedAt == nil {
continue continue
} }
name := parseSnapshotName(s.ID.String(), s.Hostname.String())
if len(nameFilter) > 0 { if len(nameFilter) > 0 {
if _, ok := nameFilter[parseSnapshotName(s.ID.String())]; !ok { if _, ok := nameFilter[name]; !ok {
continue continue
} }
} }
names[s.ID] = name
snapshots = append(snapshots, SnapshotInfo{ snapshots = append(snapshots, SnapshotInfo{
ID: s.ID, ID: s.ID,
Timestamp: s.StartedAt, Timestamp: s.StartedAt,
@@ -527,7 +525,7 @@ func (v *Vaultik) PurgeSnapshotsWithOptions(opts *SnapshotPurgeOptions) error {
return snapshots[i].Timestamp.After(snapshots[j].Timestamp) return snapshots[i].Timestamp.After(snapshots[j].Timestamp)
}) })
toDelete, err := selectSnapshotsToPurge(snapshots, opts) toDelete, err := selectSnapshotsToPurge(snapshots, names, opts)
if err != nil { if err != nil {
return err return err
} }
@@ -545,9 +543,11 @@ func (v *Vaultik) PurgeSnapshotsWithOptions(opts *SnapshotPurgeOptions) error {
// selectSnapshotsToPurge applies the purge retention criteria to the // selectSnapshotsToPurge applies the purge retention criteria to the
// newest-first sorted snapshot list and returns the deletion // newest-first sorted snapshot list and returns the deletion
// candidates. // candidates. names maps each snapshot's ID to its snapshot name.
func selectSnapshotsToPurge( func selectSnapshotsToPurge(
snapshots []SnapshotInfo, opts *SnapshotPurgeOptions, snapshots []SnapshotInfo,
names map[types.SnapshotID]string,
opts *SnapshotPurgeOptions,
) ([]SnapshotInfo, error) { ) ([]SnapshotInfo, error) {
var toDelete []SnapshotInfo var toDelete []SnapshotInfo
@@ -558,7 +558,7 @@ func selectSnapshotsToPurge(
seen := make(map[string]bool) seen := make(map[string]bool)
for _, snap := range snapshots { for _, snap := range snapshots {
name := parseSnapshotName(snap.ID.String()) name := names[snap.ID]
if seen[name] { if seen[name] {
toDelete = append(toDelete, snap) toDelete = append(toDelete, snap)
@@ -718,7 +718,7 @@ func (v *Vaultik) VerifySnapshotWithOptions(
result.BlobCount = manifest.BlobCount result.BlobCount = manifest.BlobCount
result.TotalSize = manifest.TotalCompressedSize result.TotalSize = manifest.TotalCompressedSize
if !opts.JSON { if !opts.JSON && !v.UI.Quiet() {
v.stdoutf("Snapshot information:\n") v.stdoutf("Snapshot information:\n")
v.stdoutf(" Blob count: %d\n", manifest.BlobCount) v.stdoutf(" Blob count: %d\n", manifest.BlobCount)
v.stdoutf(" Total size: %s\n", ubytes(manifest.TotalCompressedSize)) v.stdoutf(" Total size: %s\n", ubytes(manifest.TotalCompressedSize))
@@ -773,7 +773,7 @@ func (v *Vaultik) printVerifyHeader(snapshotID string, opts *VerifyOptions) {
snapshotTime = t snapshotTime = t
} }
if !opts.JSON { if !opts.JSON && !v.UI.Quiet() {
v.stdoutf("Verifying snapshot %s\n", snapshotID) v.stdoutf("Verifying snapshot %s\n", snapshotID)
if !snapshotTime.IsZero() { if !snapshotTime.IsZero() {
@@ -813,7 +813,7 @@ func (v *Vaultik) verifyManifestBlobs(
stat, err := v.Storage.Stat(v.ctx, blobPath) stat, err := v.Storage.Stat(v.ctx, blobPath)
switch { switch {
case err != nil: case err != nil:
if !opts.JSON { if !opts.JSON && !v.UI.Quiet() {
v.stdoutf(" Missing: %s (%s)\n", v.stdoutf(" Missing: %s (%s)\n",
blob.Hash, ubytes(blob.CompressedSize)) blob.Hash, ubytes(blob.CompressedSize))
} }
@@ -821,7 +821,7 @@ func (v *Vaultik) verifyManifestBlobs(
missing++ missing++
missingSize += blob.CompressedSize missingSize += blob.CompressedSize
case stat.Size != blob.CompressedSize: case stat.Size != blob.CompressedSize:
if !opts.JSON { if !opts.JSON && !v.UI.Quiet() {
v.stdoutf(" Wrong size: %s (store has %s, manifest lists %s)\n", v.stdoutf(" Wrong size: %s (store has %s, manifest lists %s)\n",
blob.Hash, ubytes(stat.Size), ubytes(blob.CompressedSize)) blob.Hash, ubytes(stat.Size), ubytes(blob.CompressedSize))
} }
@@ -853,6 +853,22 @@ func (v *Vaultik) formatVerifyResult(
return v.outputVerifyJSON(result) return v.outputVerifyJSON(result)
} }
// Under --quiet a failure is still returned, and the cli layer
// prints it on stderr.
if !v.UI.Quiet() {
v.printVerifySummary(result, failure)
}
if failure != "" {
return fmt.Errorf("%w: %s", errSnapshotVerifyFailed, failure)
}
return nil
}
// printVerifySummary prints the counts and the status line that end the
// human-readable shallow verify report. failure is empty when it passed.
func (v *Vaultik) printVerifySummary(result *VerifyResult, failure string) {
v.stdoutf("\nVerification complete:\n") v.stdoutf("\nVerification complete:\n")
v.stdoutf(" Present with listed size: %d blobs\n", result.Verified) v.stdoutf(" Present with listed size: %d blobs\n", result.Verified)
@@ -874,14 +890,12 @@ func (v *Vaultik) formatVerifyResult(
if failure != "" { if failure != "" {
v.stdoutf("FAILED - %s\n", failure) v.stdoutf("FAILED - %s\n", failure)
return fmt.Errorf("%w: %s", errSnapshotVerifyFailed, failure) return
} }
// Report only what was actually checked: presence and size, not contents. // Report only what was actually checked: presence and size, not contents.
v.stdoutf("OK - all %d blobs listed in the manifest are present with the "+ v.stdoutf("OK - all %d blobs listed in the manifest are present with the "+
"listed size; contents not checked (use --deep)\n", result.Verified) "listed size; contents not checked (use --deep)\n", result.Verified)
return nil
} }
// shallowVerifyFailure returns a human-readable description of everything // shallowVerifyFailure returns a human-readable description of everything
@@ -1038,7 +1052,7 @@ func (v *Vaultik) syncWithRemote() error {
// every local snapshot record (issue #160). // every local snapshot record (issue #160).
remoteKeys, err := v.listAllRemoteSnapshotKeys() remoteKeys, err := v.listAllRemoteSnapshotKeys()
if err != nil { if err != nil {
return fmt.Errorf("listing remote snapshots: %w", err) return err
} }
remoteKeySet := make(map[string]bool, len(remoteKeys)) remoteKeySet := make(map[string]bool, len(remoteKeys))
@@ -1103,6 +1117,12 @@ type RemoveResult struct {
// just-removed snapshot left behind on the destination store. // just-removed snapshot left behind on the destination store.
const pruneCommandHint = "vaultik prune" const pruneCommandHint = "vaultik prune"
// snapshotRemoveCommandHint is the command suggested, with the
// snapshot's ID, when a remove could not reach the destination store:
// running it again removes the snapshot's metadata there, which
// `vaultik prune` never does.
const snapshotRemoveCommandHint = "vaultik snapshot remove"
// RemoveSnapshot removes a snapshot from the local index database and, // RemoveSnapshot removes a snapshot from the local index database and,
// unless LocalOnly is set, also strips the snapshot's metadata from the // unless LocalOnly is set, also strips the snapshot's metadata from the
// destination store. Blobs are NOT touched: removing a snapshot's // destination store. Blobs are NOT touched: removing a snapshot's
@@ -1139,7 +1159,7 @@ func (v *Vaultik) RemoveSnapshot(
} }
if !opts.LocalOnly { if !opts.LocalOnly {
result.RemoteRemoved = v.removeSnapshotRemote(snapshotID) result.RemoteRemoved = v.removeSnapshotRemote(snapshotID, opts)
} }
if v.SnapshotManager != nil { if v.SnapshotManager != nil {
@@ -1222,9 +1242,11 @@ func (v *Vaultik) confirmRemoveSnapshot(snapshotID string, opts *RemoveOptions)
// removeSnapshotRemote strips the snapshot's metadata from the // removeSnapshotRemote strips the snapshot's metadata from the
// destination store, warning and proceeding on failure: the local-DB // destination store, warning and proceeding on failure: the local-DB
// removal has already happened, so the user is told the remote half // removal has already happened, so the user is told the remote half
// didn't finish and can retry with `vaultik prune` once the destination // didn't finish and to run `vaultik snapshot remove` for the snapshot
// store is reachable. Returns true when the remote removal succeeded. // again once the destination store is reachable (`vaultik prune` never
func (v *Vaultik) removeSnapshotRemote(snapshotID string) bool { // removes snapshot metadata). Returns true when the remote removal
// succeeded.
func (v *Vaultik) removeSnapshotRemote(snapshotID string, opts *RemoveOptions) bool {
log.Info("Removing snapshot metadata from remote storage", log.Info("Removing snapshot metadata from remote storage",
"snapshot_id", snapshotID) "snapshot_id", snapshotID)
@@ -1232,13 +1254,17 @@ func (v *Vaultik) removeSnapshotRemote(snapshotID string) bool {
err := v.deleteRemoteSnapshotByKey(remoteKey) err := v.deleteRemoteSnapshotByKey(remoteKey)
if err != nil { if err != nil {
log.Warn("Could not remove snapshot metadata from remote storage", log.Warn("Could not remove snapshot metadata from remote storage; "+
"error", err) "run '"+snapshotRemoveCommandHint+"' with the snapshot's ID "+
"again once the remote is reachable",
"snapshot_id", snapshotID, "error", err)
if v.UI != nil { // The UI writes to stdout, which under --json holds only the
// document; the log record above is the warning on stderr.
if v.UI != nil && !opts.JSON {
v.UI.Warningf("Could not remove snapshot metadata from remote: "+ v.UI.Warningf("Could not remove snapshot metadata from remote: "+
"%v. Run '%s' once the remote is reachable to finish cleanup.", "%v. Run '%s %s' again once the remote is reachable.",
err, pruneCommandHint) err, snapshotRemoveCommandHint, snapshotID)
} }
return false return false
internal/vaultik/snapshot_summary_test.go
+286
View File
@@ -0,0 +1,286 @@
package vaultik_test
import (
"bytes"
"context"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/storage/faultstore"
"sneak.berlin/go/vaultik/internal/ui"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// These tests cover https://git.eeqj.de/sneak/vaultik/issues/225: the
// summary printed after a backup, and the statistics stored in the
// snapshots table, count each file, byte and upload once, and a --cron
// run records its uploads.
// summaryUploadDelay slows every blob upload, so a run's upload time is
// at least this long per blob even on a local store.
const summaryUploadDelay = 20 * time.Millisecond
// summaryEnv is a backup setup whose user-facing output is kept in out.
type summaryEnv struct {
v *vaultik.Vaultik
db *database.DB
repos *database.Repositories
out *bytes.Buffer
// aPath is a.bin, whose content copy.bin repeats; aSize is its size
// and totalSize the size of all three source files.
aPath string
aSize int64
totalSize int64
}
// newSummaryEnv writes src/one/a.bin, src/one/small.txt and
// src/two/copy.bin, a copy of a.bin. Every chunk of copy.bin is therefore
// already stored by the time the backup reaches it.
//
// The snapshot names "first" and "second" back up src; "split" backs up
// src/one and src/two as two paths.
func newSummaryEnv(t *testing.T) *summaryEnv {
t.Helper()
fs := afero.NewOsFs()
tempDir := t.TempDir()
srcDir := filepath.Join(tempDir, "src")
dirOne := filepath.Join(srcDir, "one")
dirTwo := filepath.Join(srcDir, "two")
dbPath := filepath.Join(tempDir, "index.sqlite")
ctx := context.Background()
aContent := bytesPattern("a-", int(3*faultChunkSize))
smallContent := []byte("hello vaultik")
files := map[string][]byte{
filepath.Join(dirOne, "a.bin"): aContent,
filepath.Join(dirOne, "small.txt"): smallContent,
filepath.Join(dirTwo, "copy.bin"): aContent,
}
for path, content := range files {
require.NoError(t, fs.MkdirAll(filepath.Dir(path), 0o755))
require.NoError(t, afero.WriteFile(fs, path, content, 0o644))
}
cfg := faultTestConfig()
cfg.IndexPath = dbPath
cfg.ChunkSize = config.Size(faultChunkSize)
cfg.Snapshots = map[string]config.SnapshotConfig{
"first": {Paths: []string{srcDir}},
"second": {Paths: []string{srcDir}},
"split": {Paths: []string{dirOne, dirTwo}},
}
inner, err := storage.NewFileStorer(filepath.Join(tempDir, "remote"))
require.NoError(t, err)
store := faultstore.New(inner)
store.OnPut = func(key string) faultstore.PutAction {
if strings.HasPrefix(key, "blobs/") {
time.Sleep(summaryUploadDelay)
}
return faultstore.PutNormal
}
db, err := database.New(ctx, dbPath)
require.NoError(t, err)
t.Cleanup(func() { _ = db.Close() })
repos := database.NewRepositories(db)
out := &bytes.Buffer{}
v := newBackupVaultik(ctx, cfg, store, repos, db, fs)
v.UI = ui.NewWithColor(out, false)
return &summaryEnv{
v: v,
db: db,
repos: repos,
out: out,
aPath: filepath.Join(dirOne, "a.bin"),
aSize: int64(len(aContent)),
totalSize: int64(2*len(aContent) + len(smallContent)),
}
}
// backUp runs a backup of the named snapshot and returns its output.
func (e *summaryEnv) backUp(t *testing.T, name string, cron bool) string {
t.Helper()
e.out.Reset()
require.NoError(t, e.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
Cron: cron,
Snapshots: []string{name},
}))
return e.out.String()
}
// snapshot returns the local snapshots row of the snapshot named name.
func (e *summaryEnv) snapshot(t *testing.T, name string) *database.Snapshot {
t.Helper()
ctx := context.Background()
snap, err := e.repos.Snapshots.GetByID(ctx,
localSnapshotID(ctx, t, e.repos, name))
require.NoError(t, err)
require.NotNil(t, snap)
return snap
}
// uploads returns how many blobs the snapshot uploaded and their
// total size, as recorded in the uploads table.
func (e *summaryEnv) uploads(t *testing.T, snapshotID string) (int64, int64) {
t.Helper()
var count, size int64
err := e.db.Conn().QueryRowContext(context.Background(), `
SELECT COUNT(*), COALESCE(SUM(size), 0)
FROM uploads WHERE snapshot_id = ?`, snapshotID).Scan(&count, &size)
require.NoError(t, err)
return count, size
}
// referencedBlobSizes returns the compressed and uncompressed sizes of
// all blobs the snapshot references.
func (e *summaryEnv) referencedBlobSizes(
t *testing.T, snapshotID string,
) (int64, int64) {
t.Helper()
var compressed, uncompressed int64
err := e.db.Conn().QueryRowContext(context.Background(), `
SELECT COALESCE(SUM(b.compressed_size), 0),
COALESCE(SUM(b.uncompressed_size), 0)
FROM snapshot_blobs sb JOIN blobs b ON b.blob_hash = sb.blob_hash
WHERE sb.snapshot_id = ?`, snapshotID).Scan(&compressed, &uncompressed)
require.NoError(t, err)
return compressed, uncompressed
}
// filesLine returns the summary's line of file counts.
func filesLine(examined, backedUp, unchanged int) string {
return fmt.Sprintf("Files: %d examined, %d backed up, %d unchanged.",
examined, backedUp, unchanged)
}
// dataLine returns the summary's line of byte counts.
func (e *summaryEnv) dataLine(total, backedUp int64) string {
return fmt.Sprintf("Data: %s total (%s backed up).",
e.v.UI.Size(total), e.v.UI.Size(backedUp))
}
// A first backup stores copy.bin's chunks while backing up a.bin, so
// copy.bin's chunks are deduplicated within the run. Each file and byte
// is still counted once.
func TestSnapshotSummaryFirstRun(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newSummaryEnv(t)
summary := env.backUp(t, "first", false)
assert.Contains(t, summary, filesLine(3, 3, 0))
assert.Contains(t, summary, env.dataLine(env.totalSize, env.totalSize))
snap := env.snapshot(t, "first")
uploadCount, uploadBytes := env.uploads(t, snap.ID.String())
require.Positive(t, uploadCount)
assert.Contains(t, summary, fmt.Sprintf("Upload: %d blobs, %s in ",
uploadCount, env.v.UI.Size(uploadBytes)))
assert.Equal(t, int64(3), snap.FileCount)
assert.Equal(t, env.totalSize, snap.TotalSize)
assert.Equal(t, uploadCount, snap.BlobCount)
assert.Equal(t, uploadBytes, snap.UploadBytes)
}
// An incremental backup where a.bin's mtime changed but its content did
// not: a.bin is backed up again and every one of its chunks is already
// stored.
func TestSnapshotSummaryIncrementalRunWithDeduplicatedChunks(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newSummaryEnv(t)
env.backUp(t, "first", false)
later := time.Now().Add(time.Hour)
require.NoError(t, os.Chtimes(env.aPath, later, later))
summary := env.backUp(t, "second", false)
assert.Contains(t, summary, filesLine(3, 1, 2))
assert.Contains(t, summary, env.dataLine(env.totalSize, env.aSize))
assert.NotContains(t, summary, "Upload:")
snap := env.snapshot(t, "second")
compressed, uncompressed := env.referencedBlobSizes(t, snap.ID.String())
require.Positive(t, compressed)
assert.Equal(t, env.totalSize, snap.TotalSize)
assert.Zero(t, snap.ChunkCount)
assert.Zero(t, snap.BlobCount)
assert.Zero(t, snap.UploadBytes)
assert.Equal(t, compressed, snap.BlobSize,
"blob_size must total the blobs the snapshot references")
assert.Equal(t, uncompressed, snap.BlobUncompressedSize)
}
// Under --cron the progress reporter is off; the upload figures must
// still reach the summary and the snapshots row. The snapshot has two
// paths, each backed up by its own scan.
func TestSnapshotSummaryCronRunRecordsUploads(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newSummaryEnv(t)
summary := env.backUp(t, "split", true)
snap := env.snapshot(t, "split")
uploadCount, uploadBytes := env.uploads(t, snap.ID.String())
require.Positive(t, uploadCount)
assert.Contains(t, summary, filesLine(3, 3, 0))
assert.Contains(t, summary, env.dataLine(env.totalSize, env.totalSize))
assert.Contains(t, summary, fmt.Sprintf("Upload: %d blobs, %s in ",
uploadCount, env.v.UI.Size(uploadBytes)))
assert.Equal(t, env.totalSize, snap.TotalSize)
assert.Equal(t, uploadCount, snap.BlobCount,
"blob_count must count each blob once, however many paths the "+
"snapshot has")
assert.Equal(t, uploadBytes, snap.UploadBytes)
assert.GreaterOrEqual(t, snap.UploadDurationMs,
uploadCount*summaryUploadDelay.Milliseconds())
compressed, uncompressed := env.referencedBlobSizes(t, snap.ID.String())
require.Positive(t, uncompressed)
assert.Equal(t, compressed, snap.BlobSize)
assert.Equal(t, uncompressed, snap.BlobUncompressedSize)
assert.InDelta(t, float64(compressed)/float64(uncompressed),
snap.CompressionRatio, 1e-9)
}
internal/vaultik/two_path_snapshot_test.go
+70
View File
@@ -0,0 +1,70 @@
package vaultik_test
import (
"context"
"maps"
"path/filepath"
"testing"
"github.com/spf13/afero"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// A backup without --cron runs the progress reporter while one scanner
// scans each path of the snapshot in turn. See
// https://git.eeqj.de/sneak/vaultik/issues/253.
func TestBackupWithoutCronOfTwoPathSnapshotRestoresBothPaths(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
const snapshotName = "data"
fs := afero.NewOsFs()
tempDir := t.TempDir()
firstDir := filepath.Join(tempDir, "first")
secondDir := filepath.Join(tempDir, "second")
storeDir := filepath.Join(tempDir, "remote")
restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite")
ctx := context.Background()
files := writeFaultSourceTree(t, fs, firstDir)
maps.Copy(files, writeFaultSourceTree(t, fs, secondDir))
cfg := faultTestConfig()
cfg.IndexPath = dbPath
cfg.ChunkSize = config.Size(faultChunkSize)
cfg.Snapshots = map[string]config.SnapshotConfig{
snapshotName: {Paths: []string{firstDir, secondDir}},
}
store, err := storage.NewFileStorer(storeDir)
require.NoError(t, err)
db, err := database.New(ctx, dbPath)
require.NoError(t, err)
repos := database.NewRepositories(db)
v := newBackupVaultik(ctx, cfg, store, repos, db, fs)
require.NoError(t, v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
Snapshots: []string{snapshotName},
}))
id := localSnapshotID(ctx, t, repos, snapshotName)
require.NoError(t, db.Close())
reader := newReaderVaultik(ctx, cfg, store, nil, fs)
require.NoError(t, reader.Restore(&vaultik.RestoreOptions{
SnapshotID: id,
TargetDir: restoreDir,
Verify: true,
}))
assertRestoredTree(t, fs, restoreDir, files)
}
internal/vaultik/verify.go
+15 -12
View File
@@ -103,7 +103,7 @@ func (v *Vaultik) RunDeepVerify(snapshotID string, opts *VerifyOptions) error {
log.Info("Starting snapshot verification", "snapshot_id", snapshotID, "mode", "deep") log.Info("Starting snapshot verification", "snapshot_id", snapshotID, "mode", "deep")
if !opts.JSON { if !opts.JSON && !v.UI.Quiet() {
v.stdoutf("Deep verification of snapshot: %s\n\n", snapshotID) v.stdoutf("Deep verification of snapshot: %s\n\n", snapshotID)
} }
@@ -143,10 +143,13 @@ func (v *Vaultik) RunDeepVerify(snapshotID string, opts *VerifyOptions) error {
log.Info("✓ Verification completed successfully", log.Info("✓ Verification completed successfully",
"snapshot_id", snapshotID, "mode", "deep", "blobs_verified", len(dbBlobs)) "snapshot_id", snapshotID, "mode", "deep", "blobs_verified", len(dbBlobs))
v.stdoutf("\n✓ Verification completed successfully\n")
v.stdoutf(" Snapshot: %s\n", snapshotID) if !v.UI.Quiet() {
v.stdoutf(" Blobs verified: %d\n", len(dbBlobs)) v.stdoutf("\n✓ Verification completed successfully\n")
v.stdoutf(" Total size: %s\n", ubytes(totalSize)) v.stdoutf(" Snapshot: %s\n", snapshotID)
v.stdoutf(" Blobs verified: %d\n", len(dbBlobs))
v.stdoutf(" Total size: %s\n", ubytes(totalSize))
}
return nil return nil
} }
@@ -170,7 +173,7 @@ func (v *Vaultik) loadVerificationData(
// remote manifests; see its doc comment. // remote manifests; see its doc comment.
log.Info("Downloading manifest", "remote_key", remoteKey) log.Info("Downloading manifest", "remote_key", remoteKey)
if !opts.JSON { if !opts.JSON && !v.UI.Quiet() {
v.stdoutf("Downloading manifest...\n") v.stdoutf("Downloading manifest...\n")
} }
@@ -185,7 +188,7 @@ func (v *Vaultik) loadVerificationData(
"manifest_blob_count", manifest.BlobCount, "manifest_blob_count", manifest.BlobCount,
"manifest_total_size", ubytes(manifest.TotalCompressedSize)) "manifest_total_size", ubytes(manifest.TotalCompressedSize))
if !opts.JSON { if !opts.JSON && !v.UI.Quiet() {
v.stdoutf("Manifest loaded: %d blobs (%s)\n", v.stdoutf("Manifest loaded: %d blobs (%s)\n",
manifest.BlobCount, ubytes(manifest.TotalCompressedSize)) manifest.BlobCount, ubytes(manifest.TotalCompressedSize))
v.stdoutf("Downloading and decrypting database...\n") v.stdoutf("Downloading and decrypting database...\n")
@@ -215,7 +218,7 @@ func (v *Vaultik) loadVerificationData(
"db_blob_count", len(dbBlobs), "db_blob_count", len(dbBlobs),
"db_total_size", ubytes(dbTotalSize)) "db_total_size", ubytes(dbTotalSize))
if !opts.JSON { if !opts.JSON && !v.UI.Quiet() {
v.stdoutf("Database loaded: %d blobs (%s)\n", v.stdoutf("Database loaded: %d blobs (%s)\n",
len(dbBlobs), ubytes(dbTotalSize)) len(dbBlobs), ubytes(dbTotalSize))
} }
@@ -273,7 +276,7 @@ func (v *Vaultik) runVerificationSteps(
totalSize int64, totalSize int64,
identities []age.Identity, identities []age.Identity,
) error { ) error {
if !opts.JSON { if !opts.JSON && !v.UI.Quiet() {
v.stdoutf("Verifying manifest against database...\n") v.stdoutf("Verifying manifest against database...\n")
} }
@@ -282,7 +285,7 @@ func (v *Vaultik) runVerificationSteps(
return v.deepVerifyFailure(result, opts, err.Error(), err) return v.deepVerifyFailure(result, opts, err.Error(), err)
} }
if !opts.JSON { if !opts.JSON && !v.UI.Quiet() {
v.stdoutf("Manifest verified.\n") v.stdoutf("Manifest verified.\n")
v.stdoutf("Checking blob existence in remote storage...\n") v.stdoutf("Checking blob existence in remote storage...\n")
} }
@@ -292,7 +295,7 @@ func (v *Vaultik) runVerificationSteps(
return v.deepVerifyFailure(result, opts, err.Error(), err) return v.deepVerifyFailure(result, opts, err.Error(), err)
} }
if !opts.JSON { if !opts.JSON && !v.UI.Quiet() {
v.stdoutf("All blobs exist.\n") v.stdoutf("All blobs exist.\n")
v.stdoutf("Downloading and verifying blob contents (%d blobs, %s)...\n", v.stdoutf("Downloading and verifying blob contents (%d blobs, %s)...\n",
len(dbBlobs), ubytes(totalSize)) len(dbBlobs), ubytes(totalSize))
@@ -748,7 +751,7 @@ func (v *Vaultik) performDeepVerificationFromDB(
"eta", eta.Round(time.Second), "eta", eta.Round(time.Second),
) )
if !opts.JSON { if !opts.JSON && !v.UI.Quiet() {
v.stdoutf(" Verified %d/%d blobs (%d remaining) - %s/%s - elapsed %s, eta %s\n", v.stdoutf(" Verified %d/%d blobs (%d remaining) - %s/%s - elapsed %s, eta %s\n",
i+1, len(blobs), remaining, i+1, len(blobs), remaining,
ubytes(bytesProcessed), ubytes(bytesProcessed),
internal/vaultik/verify_quiet_test.go
+102
View File
@@ -0,0 +1,102 @@
package vaultik_test
import (
"bytes"
"context"
"io"
"os"
"path/filepath"
"testing"
"github.com/spf13/afero"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot"
"sneak.berlin/go/vaultik/internal/ui"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// TestVerify_QuietSuppressesReport is the --quiet contract for
// `snapshot verify`: neither shallow nor deep verify writes its report,
// a failed verify still returns its error (which the cli layer prints
// on stderr), and the --json document still emits.
func TestVerify_QuietSuppressesReport(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
fs := afero.NewOsFs()
tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "source")
storeDir := filepath.Join(tempDir, "remote")
dbPath := filepath.Join(tempDir, "index.sqlite")
chunkSize := int64(32 * 1024)
maxBlobSize := int64(128 * 1024)
require.NoError(t, fs.MkdirAll(dataDir, 0o755))
require.NoError(t, afero.WriteFile(fs,
filepath.Join(dataDir, "data.bin"),
bytesPattern("quiet-", int(maxBlobSize*2)), 0o644))
ctx := context.Background()
cfg, storer, snapshotID := runFileStorageBackup(
ctx, t, fs, dataDir, storeDir, dbPath, chunkSize, maxBlobSize)
// The UI writes to the same buffer as Stdout, as both write to the
// process's stdout in production.
var stdout bytes.Buffer
newQuietVerifier := func() *vaultik.Vaultik {
v := &vaultik.Vaultik{
Config: cfg,
Storage: storer,
Fs: fs,
Stdout: &stdout,
Stderr: io.Discard,
UI: ui.NewWithColor(&stdout, false),
}
v.SetContext(ctx)
v.UI.SetQuiet(true)
return v
}
require.NoError(t, newQuietVerifier().VerifySnapshotWithOptions(
snapshotID, &vaultik.VerifyOptions{}))
require.Empty(t, stdout.String(),
"shallow verify must write no report under --quiet")
require.NoError(t, newQuietVerifier().VerifySnapshotWithOptions(
snapshotID, &vaultik.VerifyOptions{Deep: true}))
require.Empty(t, stdout.String(),
"deep verify must write no report under --quiet")
require.NoError(t, newQuietVerifier().VerifySnapshotWithOptions(
snapshotID, &vaultik.VerifyOptions{JSON: true}))
require.Equal(t, "ok", decodeVerifyResult(t, stdout.Bytes()).Status,
"the --json document must still emit under --quiet")
// A snapshot without its encrypted database fails shallow verify. A
// failed report also lists each missing blob and each blob of the
// wrong size, so remove one blob and grow another.
require.NoError(t, os.Remove(filepath.Join(storeDir, "metadata",
snapshot.RemoteSnapshotKey(snapshotID), "db.zst.age")))
blobFiles, err := filepath.Glob(
filepath.Join(storeDir, "blobs", "*", "*", "*"))
require.NoError(t, err)
require.GreaterOrEqual(t, len(blobFiles), 2,
"the snapshot must span two blobs, one to remove and one to grow")
require.NoError(t, os.Remove(blobFiles[0]))
growOneBlob(t, fs, filepath.Join(storeDir, "blobs"))
stdout.Reset()
require.Error(t, newQuietVerifier().VerifySnapshotWithOptions(
snapshotID, &vaultik.VerifyOptions{}),
"--quiet must not change the outcome of a failed verify")
require.Empty(t, stdout.String(),
"a failed verify must write no report under --quiet")
}
script/fmt-check
+5 -2
View File
@@ -1,6 +1,9 @@
#!/bin/sh #!/bin/sh
# script/fmt-check: check formatting (read-only). Same scope as # script/fmt-check: check formatting (read-only). Fails instead of
# script/fmt, but fails instead of writing. # writing. It checks every Go file outside .tool, which is more than
# script/fmt formats: `go fmt ./...` skips `testdata` directories and
# files and directories whose names start with `.` or `_`. Fix a file
# only this reports with `gofmt -w`.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
script/release
+3 -4
View File
@@ -21,10 +21,9 @@ goreleaser_version() {
head -n 1 head -n 1
} }
# Resolve the goreleaser to run, on the same rule script/lint uses for # Resolve the goreleaser to run. A binary on PATH is accepted only when
# golangci-lint: a binary on PATH is accepted only when it is exactly # it is exactly the pinned version, because a differently versioned tool
# the pinned version, because a differently versioned tool would # would produce a differently built release from the same tag. Anything else
# produce a differently built release from the same tag. Anything else
# comes from .tool/bin, and a missing one is a loud failure naming the # comes from .tool/bin, and a missing one is a loud failure naming the
# script that installs it rather than a silent fallback. # script that installs it rather than a silent fallback.
resolve_goreleaser() { resolve_goreleaser() {
test/config.yaml
+1 -1
View File
@@ -19,7 +19,7 @@ s3:
secret_access_key: test-secret-key secret_access_key: test-secret-key
region: us-east-1 region: us-east-1
use_ssl: true use_ssl: true
part_size: 5242880 # 5MB part_size: 5242880 # 5MiB
index_path: /tmp/vaultik-test.sqlite index_path: /tmp/vaultik-test.sqlite
chunk_size: 10MB chunk_size: 10MB
blob_size_limit: 10GB blob_size_limit: 10GB