sneak/vaultik
1
0 Fork 0
Code Issues 6 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Compare commits

base: sneak/vaultik:docs/397-rewrite-prose-documents
sneak/vaultik:main sneak/vaultik:docs/397-rewrite-prose-documents sneak/vaultik:next sneak/vaultik:issue-337-purge-keeps-metadata
sneak/vaultik:v1.0.0-rc3 sneak/vaultik:v1.0.0-rc2 sneak/vaultik:v1.0.0-rc1
..
compare: sneak/vaultik:09dbe6f4c988995c98c3231737ea3d9e6ff4f84f
sneak/vaultik:docs/397-rewrite-prose-documents sneak/vaultik:next sneak/vaultik:issue-337-purge-keeps-metadata sneak/vaultik:main
sneak/vaultik:v1.0.0-rc3 sneak/vaultik:v1.0.0-rc2 sneak/vaultik:v1.0.0-rc1
1 Commits
docs/397-rewrite-prose-documents .. 09dbe6f4c9
Author SHA1 Message Date
sneak 09dbe6f4c9 Make script/cibuild unable to report an unearned green (closes #85)
check / check (pull_request) Successful in 2m57s
Details 
script/cibuild was a bare `docker build .` with no cache control. The
Dockerfile does `COPY . .` and then `RUN make fmt-check` / `RUN make
lint` in the lint stage and `COPY . .` / `RUN make test` in the builder
stage. On an unchanged tree Docker served those RUN layers from cache,
so the checks never executed, and the build still exited 0 -- the exit
code, which is the one signal automation trusts, was wrong, and wrong
in the direction that matters: the longer a branch sits unchanged, the
more likely its "verification" is a replay, which is exactly its state
just before a merge.

The fix is an `ARG CHECK_EPOCH` declared immediately above the check
RUNs, with script/cibuild passing a fresh value on every invocation.
ARG scope is per-stage in Docker, so the lint stage and the builder
stage each declare their own; covering only one would leave half the
gate fake.

Placement is the substance of the change. The ARG sits below the
`apk add`, `COPY go.mod go.sum`, and `go mod download` layers in both
stages, so only the check layers are invalidated: earlier and every
build would be cold, later and the checks would stay cached.

The epoch is assigned to its own variable rather than substituted
inline into the --build-arg:

    epoch="$(date +%s)"
    docker build --build-arg CHECK_EPOCH="$epoch" .

Under `set -eu` a command substitution that fails inside an argument
does not abort the script. Inline, a failing `date` would leave
CHECK_EPOCH an empty string; an empty string is a constant; and a
constant CHECK_EPOCH is precisely the cached-check false green this
commit exists to eliminate -- so the guard would have carried a silent
path to the defect it guards against. As a bare assignment, `set -e`
aborts before any build starts.

The guarantee is conditional, and README.md and the Dockerfile now say
so instead of claiming the check layers can never be cached. They are
keyed on CHECK_EPOCH, so they re-run for any value not yet built
against this tree -- but a build that omits --build-arg gets the empty
default, and on an unchanged tree every build after the first then
replays them, executes nothing, and exits 0. That state was produced
by measurement rather than reasoned about. Issue #91 tracks the
upstream hardening that would make the missing-arg case fail loudly,
along with the expanded ARG form, a per-invocation epoch, and
script/docker.

The bare unreferenced ARG form is kept deliberately, not because it
matches upstream -- upstream has since settled on expanding the value
into the check command. A declared-but-unreferenced ARG does enter
BuildKit's cache key, which is measured on this host rather than
assumed, and upstream records that repos on the bare form need no
rework. Moving to the expanded form is hardening, tracked in #91.

The script/cibuild header comment claimed the Dockerfile runs
script/check via make check. It does not: it runs make fmt-check and
make lint in the lint stage and make test in the builder stage.
Corrected.

Measurements are recorded once, in the verification comment on PR #89:
a back-to-back script/cibuild pair on an unchanged tree, and the
counterfactual that withholds --build-arg and reproduces the original
false green on its second run. They are deliberately not restated here
or in TODO.md, so there is a single record that cannot disagree with
itself.

.golangci.yml is unchanged (sha256 021cc83f4e6f...643346bcb), as is the
lint-stage FROM line that is the single source of truth for the linter
version, script/lint's pinned-image logic, and
.gitea/workflows/check.yml, whose only step is script/cibuild.
 2026-08-09 06:48:56 +00:00
246 changed files with 5983 additions and 32316 deletions
Expand all files Collapse all files

No files matched your search

.dockerignore
+8 -79
View File
@@ -1,79 +1,8 @@
# .dockerignore does NOT use .gitignore semantics. Docker matches with
# moby/patternmatcher: filepath.Match plus `**`, so `*` does not cross
# `/` and an unprefixed pattern is anchored at the context root. Every
# depth-independent pattern therefore needs `**/`, or `config/.env` and
# `certs/server.key` still ship while this file reads as solved. Only
# genuinely root-anchored entries go unprefixed. Never transplant these
# into .gitignore, where `**/` is wrong.
#
# Matching is case-sensitive, so secrets use character ranges rather
# than an ALL-CAPS twin, which would still miss `Server.Key`.
#
# Extend with this repo's own host-built artifacts, written anchored:
# `/myapp`, never `**/myapp`, which also matches `cmd/myapp/` and
# deletes the package directory from the context.
# .git is sent without its config. Without a VERSION build argument the
# stage that compiles runs `git describe --tags --always` on .git, which
# does not need .git/config; that file can hold a credential, such as a
# password in a remote URL or the token the CI checkout step stores there.
# Each submodule keeps a config with the same exposure in its git directory
# under .git/modules/, nested again for a submodule's own submodules, or in
# its own .git directory when it keeps one.
# KNOWN GAP: a submodule whose name has a `config` segment (`config`,
# `deploy/config`, `config/lib`) loses its whole git directory, because
# `**/.git/modules/**/config` also matches that segment's directory
# under .git/modules/. Go's version stamping then fails the build;
# nothing leaks. Name such a submodule without that segment:
# `git submodule add --name`.
**/.git/config
**/.git/modules/**/config
# Agent scratch: one full checkout of the repo per in-flight agent.
# Anchored because it occurs once where agents run at the repo root.
# KNOWN GAP: a repo running agents in subdirectories still ships
# `services/api/.claude/` and must add its own anchored entry.
.claude
# Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Re-include a committed template with a negation if the
# build needs one: `!docs/example.env`.
**/*.[eE][nN][vV]
**/.[eE][nN][vV].*
**/.[eE][nN][vV][rR][cC]
# Private keys and the bundles carrying them. Public certificates
# (*.crt, *.cer) are deliberately absent: they are legitimate inputs.
**/*.[pP][eE][mM]
**/*.[kK][eE][yY]
**/*.[pP]12
**/*.[pP][fF][xX]
**/[iI][dD]_[rR][sS][aA]
**/[iI][dD]_[dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
**/[iI][dD]_[eE][dD]25519
**/[iI][dD]_[eE][dD]25519_[sS][kK]
# Dependencies: restored inside the image, never copied in.
**/node_modules
# OS metadata.
**/.DS_Store
**/Thumbs.db
# Editor state: never a build input, and it churns COPY.
**/*.swp
**/*.swo
**/*~
**/*.bak
**/.idea
**/.vscode
**/*.sublime-*
# This repo's own host-built artifacts.
/vaultik
/dist
/.tool
/coverage.out
/coverage.html
.git
.gitea
*.md
LICENSE
vaultik
coverage.out
coverage.html
.DS_Store
.editorconfig
-3
View File
@@ -10,6 +10,3 @@ insert_final_newline = true
[Makefile]
indent_style = tab
[*.go]
indent_style = tab
.gitea/workflows/check.yml
+9 -4
View File
@@ -1,9 +1,14 @@
name: check
on: [push]
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
check:
runs-on: ubuntu-latest
steps:
# actions/checkout v4.2.2, 2026-02-22
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
- run: script/cibuild
# actions/checkout v4, 2024-09-16
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- name: Build and check
run: script/cibuild
.gitea/workflows/release.yml
-54
View File
@@ -1,54 +0,0 @@
name: release
on:
push:
tags: ["v*"]
jobs:
release:
runs-on: ubuntu-latest
steps:
# actions/checkout v4, 2024-09-16
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
# goreleaser needs the tags and the full history: the version
# it stamps comes from the tag, and the changelog comes from
# the commits since the previous one. A shallow checkout
# silently produces a mislabelled release.
fetch-depth: 0
# goreleaser is not a compiler: it shells out to `go` for the
# `before:` hook and for every one of the four cross-compiles.
# Without this step the release either fails at the before-hook or,
# worse, ships binaries built by whatever Go the runner happens to
# carry. check.yml's runner gets the same Go through
# script/bootstrap, which calls script/install-go, and uses it only
# for `go mod download` and gofmt; it compiles inside the
# digest-pinned Dockerfile images.
#
# actions/setup-go would pin the action by commit sha, but the Go
# tarball it downloads at runtime is verified against no value in
# this repo, and the action exposes no checksum input.
# REPO_POLICIES.md requires every external reference to be pinned
# by hash with no exceptions, and this is the compiler that
# produces the published binaries -- the input where a substituted
# artifact matters most. So Go is installed the way goreleaser is:
# script/install-go downloads the exact archive for go.mod's `go`
# directive and refuses it unless its sha256 matches the value
# committed in the script, then puts .tool/go/bin on PATH for the
# steps below.
- name: Install Go
run: script/install-go
- name: Install goreleaser
run: script/install-goreleaser
- name: Release
run: script/release
env:
# RELEASE_TOKEN is a repository Actions secret: a Gitea access
# token with write access to this repository's releases (scope
# write:repository), owned by an account that can publish here.
# It is deliberately not the runner's automatic token, which is
# not guaranteed to carry that scope.
GITEA_TOKEN: ${{ secrets.RELEASE_TOKEN }}
# Build with the toolchain install-go just verified, never a
# different one auto-downloaded from a `toolchain` directive:
# the point of the hash pin is that this exact compiler makes
# the release.
GOTOOLCHAIN: local
.gitignore
+16 -56
View File
@@ -1,62 +1,22 @@
# Binary
/vaultik
# Test artifacts
*.out
*.test
coverage.html
coverage.out
# IDE
.vscode/
.idea/
*.swp
*.swo
# OS
.DS_Store
Thumbs.db
# Editors
*.swp
*.swo
*~
*.bak
.idea/
.vscode/
*.sublime-*
# Agent scratch (worktrees of this repo, created and destroyed by
# in-flight tooling). Unanchored: .gitignore patterns already match at
# every depth, so no prefix is wanted here. This is not a .dockerignore
# entry and must not be given a `**/` prefix on the way into one.
.claude/
# Node
node_modules/
# Secrets. Unanchored like every entry above, so each matches at every
# depth. Matching is case-sensitive on Linux, so names use character
# ranges rather than a lowercase form that misses `Server.Key`.
# Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Only the templates `example.env` and `sample.env` are
# re-included below. A repository that commits any other template adds
# its own negation after these lines, for example `!.env.example`.
*.[eE][nN][vV]
.[eE][nN][vV].*
.[eE][nN][vV][rR][cC]
!example.env
!sample.env
# Private keys and the bundles carrying them.
*.[pP][eE][mM]
*.[kK][eE][yY]
*.[pP]12
*.[pP][fF][xX]
[iI][dD]_[rR][sS][aA]
[iI][dD]_[dD][sS][aA]
[iI][dD]_[eE][cC][dD][sS][aA]
[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
[iI][dD]_[eE][dD]25519
[iI][dD]_[eE][dD]25519_[sS][kK]
# Go build and test output.
*.log
*.out
*.test
coverage.html
# This repo's own host-built artifacts.
/vaultik
/dist/
/.tool/
# Local configs for development; they hold storage credentials.
# Local config for development
local-config.yaml
dev-config.yaml
.golangci.yml
+2 -71
View File
@@ -10,21 +10,14 @@ run:
linters:
default: all
enable:
# Successor to the deprecated gomodguard. Named explicitly, rather than
# left to `default: all`, because it carries the module policy below.
- gomodguard_v2
disable:
# Genuinely incompatible with project patterns
- exhaustruct # Requires all struct fields
- exhaustruct_v5 # Requires all struct fields (successor to exhaustruct)
- depguard # Dependency allow/block lists
- godot # Requires comments to end with periods
- wsl # Deprecated, replaced by wsl_v5
- wrapcheck # Too verbose for internal packages
- varnamelen # Short names like db, id are idiomatic Go
# Deprecated: the warning is attached to the old name, so it is
# silenced by disabling that name, not by enabling the successor.
- wsl # Deprecated, replaced by wsl_v5
- gomodguard # Deprecated, replaced by gomodguard_v2
settings:
lll:
line-length: 88
@@ -35,68 +28,6 @@ linters:
max-complexity: 15
dupl:
threshold: 100
depguard:
# Test-support code must not be compiled into the shipped binary. A
# test-support package exists to hand a test privileges the program
# itself must never have, so a file that is not a test must not import
# one. Test files, and the files inside a package whose directory name
# ends in `test`, are where that code belongs, and are exempt.
#
# The deny list below is the one part of this file a repository is
# expected to extend, and the only part it may. depguard matches an
# import path against a list of prefixes, so it cannot be told "any path
# whose last segment ends in test"; a repository's own test-support
# packages have to be named here one at a time, by full import path,
# under a module path that differs from repository to repository. Add
# them; change nothing else.
rules:
test-support:
list-mode: lax
files:
- "$all"
- "!$test"
- "!**/*test/**"
deny:
- pkg: net/http/httptest
desc: >-
Test-support code belongs in test files and in packages whose
directory name ends in test, not in the shipped binary.
- pkg: sneak.berlin/go/vaultik/internal/storage/faultstore
desc: >-
Test-support code belongs in test files and in packages whose
directory name ends in test, not in the shipped binary.
# Only decisions already recorded in the Go package defaults are
# listed here. Every entry matches the module path exactly.
gomodguard_v2:
blocked:
- module: github.com/rs/zerolog
recommendations:
- log/slog
reason: "Structured logging is stdlib log/slog."
# One entry per pre-fork module path, because the later releases
# are separate paths. A prefix match would be shorter but would
# also reach github.com/go-redis/redismock, the test double for
# the successor these entries recommend.
- module: github.com/go-redis/redis
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/go-redis/redis/v7
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/go-redis/redis/v8
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/sergi/go-diff
recommendations:
- github.com/aymanbagabas/go-udiff
reason: "No unified diff output; use go-udiff."
- module: github.com/hexops/gotextdiff
recommendations:
- github.com/aymanbagabas/go-udiff
reason: "Unmaintained fork; use go-udiff."
issues:
max-issues-per-linter: 0
.goreleaser.yaml
+1 -12
View File
@@ -2,13 +2,6 @@ version: 2
project_name: vaultik
# This repo lives on Gitea, not GitHub. Without this block goreleaser
# talks to the GitHub API by default and a `goreleaser release` either
# fails outright or publishes somewhere nobody is looking.
gitea_urls:
api: https://git.eeqj.de/api/v1
download: https://git.eeqj.de
before:
hooks:
- go mod tidy
@@ -44,12 +37,8 @@ checksum:
name_template: "checksums.txt"
algorithm: sha256
# A snapshot is not a release and must not name itself like one. The
# previous `{{ incpatch .Version }}-next` derived a plausible-looking
# release number from the last tag -- and with no tags in the repo at
# all, from goreleaser's fabricated v0.0.0.
snapshot:
version_template: "dev-{{ slice .FullCommit 0 12 }}"
version_template: "{{ incpatch .Version }}-next"
changelog:
sort: asc
AGENTS.md
+7 -27
View File
@@ -83,8 +83,8 @@ Version: 2025-06-08
possible to mock or stub these side-effects in tests.
9. Always use structured logging. Log any relevant state/context with the
messages (but do not log secrets). If the log stream is not a terminal,
output the structured logs in jsonl format.
messages (but do not log secrets). If stdout is not a terminal, output
the structured logs in jsonl format.
10. Avoid using bare strings or numbers in code, especially if they appear
anywhere more than once. Always define a constant (usually at the top
@@ -102,29 +102,9 @@ Version: 2025-06-08
build files are acceptable in the root, but source code and other files
should be organized in appropriate subdirectories.
13. Pre-1.0: NEVER add a database migration. Migrations are supported, but
nothing is installed anywhere yet, so there is nothing to migrate. To
change the schema, edit `internal/database/schema/001.sql` (and any
code that touches the affected tables) directly. After 1.0, each schema
change is a new numbered file in that directory and a released file is
never edited; an existing local database is then migrated when vaultik
is updated. Before 1.0, a local database left on an older schema is
deleted and re-created by a full backup. See
[`docs/DATAMODEL.md`](docs/DATAMODEL.md#schema-migrations).
13. Pre-1.0: NEVER write database migrations. There are no live databases
anywhere — every user's local index can be rebuilt from a fresh full
backup. When the schema changes, just change `schema.sql` (and any code
that touches the affected tables). The local index is disposable until
1.0 ships and is tagged.
14. Never use `git add -A`. Stage only the files you intentionally
changed.
15. Commit messages carry no attribution or advertising trailers for the
tool that helped write the code, or for its vendor: the owner is the
sole author of code written with a tool.
16. Run the whole test suite with `make test` every time, and read its full
output. Never run `go test`, a single test or a single package, and
never grep the output.
17. Do not stop working on a task until the definition of done given in the
initial instruction is met: all of the work, not part or most of it.
18. For estimates: backing up over 2.5Gbit/s ethernet to an S3 server
backed by 2000MB/sec SSD takes about 4 seconds per gigabyte.
ARCHITECTURE.md
+356 -378
View File
@@ -1,401 +1,379 @@
# vaultik architecture
# Vaultik Architecture
This document describes how the `vaultik` code is organised. It covers three
things:
This document describes the internal architecture of Vaultik, focusing on the data model, type instantiation, and the relationships between core modules.
- the packages
- how they are wired together
- what happens during a backup, a restore, a verify and a prune
## Overview
Two other documents cover the stored data:
Vaultik is a backup system that uses content-defined chunking for deduplication and packs chunks into large, compressed, encrypted blobs for efficient cloud storage. The system is built around dependency injection using [uber-go/fx](https://github.com/uber-go/fx).
- [docs/DATAMODEL.md](docs/DATAMODEL.md) describes the local index.
- [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md) describes the layout of the
destination.
## Terms
- **Snapshot**: one backup of the paths listed under one name in the config.
- **Chunk**: a piece of a file's content. A chunk is named by the SHA-256 hash
of its content.
- **Blob**: one object at the destination. It holds many chunks, compressed and
then encrypted.
- **Local index**: the SQLite database on the backed-up machine.
- **Destination**: the place the backups are stored.
- **Manifest**: the unencrypted list of the blobs one snapshot uses.
- **Remote key**: a hash of the snapshot ID. It names the snapshot's directory
at the destination.
## How a backup moves data
## Data Flow
```
source files
Source Files
│
▼
Scanner walks the paths and finds the files that changed
┌─────────────────┐
│ Scanner │ Walks directories, detects changed files
└────────┬────────┘
│
▼
Chunker cuts each changed file into chunks
┌─────────────────┐
│ Chunker │ Splits files into variable-size chunks (FastCDC)
└────────┬────────┘
│
▼
Packer packs new chunks into a blob, compressed and encrypted
┌─────────────────┐
│ Packer │ Accumulates chunks, compresses (zstd), encrypts (age)
└────────┬────────┘
│
▼
Storer uploads the blob to the destination
┌─────────────────┐
│ S3 Client │ Uploads blobs to remote storage
└─────────────────┘
```
## Packages
## Data Model
- `cmd/vaultik`: the `main` function. It starts optional profiling and calls
`cli.Entry`.
- `internal/cli`: the commands, their flags and their help text. It builds the
application, takes the lock and handles interrupts.
- `internal/vaultik`: the code behind each command. `Vaultik` holds every
dependency a command uses.
- `internal/snapshot`: the `Scanner`, which backs up one path, and the
`SnapshotManager`, which creates snapshot records and uploads their metadata.
- `internal/chunker`: cuts a stream into chunks with the FastCDC algorithm.
- `internal/blob`: the `Packer`, which collects chunks into blobs.
- `internal/blobgen`: the writer that hashes, compresses and encrypts a blob,
and the reader that reverses it.
- `internal/database`: the local index. It holds the schema and the repository
types that read and write the tables.
[docs/DATAMODEL.md](docs/DATAMODEL.md#code) says which type serves which
table.
- `internal/storage`: the `Storer` interface and its three implementations, for
S3, a local directory and rclone.
- `internal/s3`: the S3 client that the S3 `Storer` uses.
- `internal/config`: loads and checks the config file.
- `internal/ui`: formats the status lines, warnings and errors a user sees.
- `internal/log`: the logger. It writes to stderr.
- `internal/pidlock`: the lock that keeps two data-changing commands from
running at once.
- `internal/globals`: the version, commit and commit date, set when the binary
is built.
- `internal/types`: distinct types for IDs, hashes and paths.
### Core Entities
## Wiring
The database tracks five primary entities and their relationships:
The application is assembled with [uber-go/fx](https://github.com/uber-go/fx), a
dependency injection library. `cli.NewApp` lists the providers:
```
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Snapshot │────▶│ File │────▶│ Chunk │
└──────────────┘ └──────────────┘ └──────────────┘
│ │
│ │
▼ ▼
┌──────────────┐ ┌──────────────┐
│ Blob │◀─────────────────────────│ BlobChunk │
└──────────────┘ └──────────────┘
```
### Entity Descriptions
#### File (`database.File`)
Represents a file, directory, or symlink in the backup system. Stores metadata needed for restoration:
- Path, source_path (for restore path stripping), mtime
- Size, mode, ownership (uid, gid)
- Symlink target (if applicable)
#### Chunk (`database.Chunk`)
A content-addressed unit of data. Files are split into variable-size chunks using the FastCDC algorithm:
- `ChunkHash`: SHA256 hash of chunk content (primary key)
- `Size`: Chunk size in bytes
Chunk sizes vary between `avgChunkSize/4` and `avgChunkSize*4` (typically 16KB-256KB for 64KB average).
#### FileChunk (`database.FileChunk`)
Maps files to their constituent chunks:
- `FileID`: Reference to the file
- `Idx`: Position of this chunk within the file (0-indexed)
- `ChunkHash`: Reference to the chunk
#### Blob (`database.Blob`)
The final storage unit uploaded to S3. Contains many compressed and encrypted chunks:
- `ID`: UUID assigned at creation
- `Hash`: SHA256 of final compressed+encrypted content
- `UncompressedSize`: Total raw chunk data before compression
- `CompressedSize`: Size after zstd compression and age encryption
- `CreatedTS`, `FinishedTS`, `UploadedTS`: Lifecycle timestamps
Blob creation process:
1. Chunks are accumulated (up to MaxBlobSize, typically 10GB)
2. Compressed with zstd
3. Encrypted with age (recipients configured in config)
4. SHA256 hash computed → becomes filename in S3
5. Uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}`
#### BlobChunk (`database.BlobChunk`)
Maps chunks to their position within blobs:
- `BlobID`: Reference to the blob
- `ChunkHash`: Reference to the chunk
- `Offset`: Byte offset within the uncompressed blob
- `Length`: Chunk size
#### Snapshot (`database.Snapshot`)
Represents a point-in-time backup:
- `ID`: Format is `{hostname}_{snapshot-name}_{RFC3339}` (e.g. `server1_home_2025-06-01T12:00:00Z`)
- Tracks file count, chunk count, blob count, sizes, compression ratio
- `CompletedAt`: Null until snapshot finishes successfully
#### SnapshotFile / SnapshotBlob
Join tables linking snapshots to their files and blobs.
### Relationship Summary
```
Snapshot 1──────────▶ N SnapshotFile N ◀────────── 1 File
Snapshot 1──────────▶ N SnapshotBlob N ◀────────── 1 Blob
File 1──────────▶ N FileChunk N ◀────────── 1 Chunk
Blob 1──────────▶ N BlobChunk N ◀────────── 1 Chunk
```
## Type Instantiation
### Application Startup
The CLI uses fx for dependency injection. Here's the instantiation order:
```go
fx.Supply(config.Path(opts.ConfigPath)),
fx.Supply(opts.LogOptions),
fx.Provide(globals.New),
fx.Provide(log.New),
config.Module,
database.Module,
log.Module,
storage.Module,
snapshot.Module,
fx.Provide(vaultik.New),
fx.Invoke(setupGlobals),
fx.NopLogger,
// cli/app.go: NewApp()
fx.New(
fx.Supply(config.ConfigPath(opts.ConfigPath)), // 1. Config path
fx.Supply(opts.LogOptions), // 2. Log options
fx.Provide(globals.New), // 3. Globals
fx.Provide(log.New), // 4. Logger config
config.Module, // 5. Config
database.Module, // 6. Database + Repositories
log.Module, // 7. Logger initialization
storage.Module, // 8. Storage backend (S3/file/rclone)
snapshot.Module, // 9. SnapshotManager + ScannerFactory
fx.Provide(vaultik.New), // 10. Vaultik orchestrator
)
```
Each module provides these:
- `config.Module` provides `*config.Config`, from `config.Load`.
- `database.Module` provides `*database.DB` and `*database.Repositories`. It
creates the index file with mode `0600` when it does not exist.
- `storage.Module` provides the `storage.Storer` that the config selects.
- `snapshot.Module` provides the `snapshot.ScannerFactory` and the
`*snapshot.SnapshotManager`.
- `vaultik.New` provides `*vaultik.Vaultik`.
A command runs through `cli.RunOperation`. It works like this:
- `RunWithApp` takes the lock when the command changes data, and then starts the
application.
- The command's function runs in its own goroutine.
- When the function returns, it asks the application to shut down.
- On `SIGINT` or `SIGTERM`, the stop hook cancels the `Vaultik` context. It then
waits for the function to return, so that its temporary files are removed.
- The wait ends after 30 seconds at most.
The `config` commands, `database delete`, `completion` and `version` do not
build the application.
## Backup
`Vaultik.CreateSnapshot` runs a backup. It does these steps in order:
- `EnsureStorageBinding` checks that the local index belongs to the configured
destination.
- `markDestination` checks the marker file and then writes it.
- `PruneDatabase` removes unfinished snapshots and unused records from the
local index.
- `createNamedSnapshot` runs once for each snapshot name.
- With `--prune`, `runPostBackupPrune` removes old snapshots and unused data.
`CreateSnapshot` calls `skipPostBackupPrune` instead of `runPostBackupPrune` in
these cases:
- `--skip-errors` skipped a file or directory that could not be read.
- A configured path held nothing.
- An earlier snapshot of the same name found a mount point that this backup did
not find.
`PruneDatabase` must succeed before anything is scanned. An unfinished snapshot
left in the index would make the backup skip files that were never fully
stored.
### One snapshot
`createNamedSnapshot` does these steps:
- `resolveSnapshotPaths` makes each configured path absolute and resolves
symlinks.
- `SnapshotManager.CreateSnapshotWithName` creates the snapshot record. The
record is not yet marked complete.
- `Scanner.Scan` backs up each path in turn.
- `finalizeSnapshotMetadata` uploads the snapshot's metadata and marks the
snapshot complete.
A new `Scanner` is created for each snapshot, from `ScannerFactory`. It owns a
`Chunker` and a `Packer`.
### Scanning one path
`Scanner.Scan` works in these steps:
- `repairInterruptedBlobs` deletes the records of blobs whose upload never
finished. Their data is then stored again.
- `loadDatabaseState` loads the known files and chunks into memory.
- `scanPhase` walks the path. It sorts each entry into changed or unchanged.
- `detectDeletedFiles` counts the files that the previous snapshot of this name
had and that are now gone.
- `batchAddFilesToSnapshot` records the unchanged files in the new snapshot.
- `recordDirectories` records every directory, and the mount points found.
- `processPhase` reads each changed file and stores it.
The walk handles each kind of entry differently:
- A regular file or a symlink is compared with its record in the index.
- A directory is recorded with its metadata. It has no content to store.
- A device node, a named pipe or a socket is skipped.
- An entry that matches an exclude pattern is skipped. For a directory, so is
everything in it.
### Deduplication
Data is stored once at two levels.
- **Files.** A file is not read again when its record in the index still matches
it.
- **Chunks.** A chunk is named by the SHA-256 hash of its content. A chunk whose
hash is already in the index is not stored again.
A file's record matches when all of these are unchanged:
- the size
- the modification time
- the mode
- the owner and group
- the link target
One case overrides the file check. A file is read again when it lists a chunk
that no uploaded blob holds. This happens when the only snapshot that used the
blob holding that chunk is removed. `loadFilesToRechunk` finds these files.
### Chunks
`chunker.Chunker` cuts a file with the FastCDC algorithm. The cut points depend
on the content, so an insertion early in a file moves only nearby chunks.
`chunk_size` sets the average chunk size. No chunk is larger than four times
this size. Only a file's last chunk can be smaller than a quarter of it. With
the default of 10 MB, those limits are 40 MB and 2.5 MB.
`ChunkReaderStreaming` passes each chunk to a callback as it is found. The whole
file is never held in memory.
### Blobs
`blob.Packer` collects new chunks into a blob. For each blob it does this:
- It gives the blob a UUID and creates a temporary file.
- It adds each chunk's bytes to a running SHA-256 hash.
- In the same pass it compresses the bytes with zstd and encrypts them with
age. The result goes to the temporary file.
- It records each chunk's offset and length within the blob.
- It closes the blob when the next chunk would pass `blob_size_limit`, and at
the end of each path.
The blob's name is `hex(SHA256(SHA256(uncompressed contents)))`. It is computed
from the chunk data before compression and encryption.
[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#blobs) says what follows from
that.
When a blob is closed, `Scanner.handleBlobReady` stores it:
- It looks the blob up at the destination. A blob that is already there is not
uploaded again.
- Otherwise it uploads the temporary file to
`blobs/<hash[0:2]>/<hash[2:4]>/<hash>`.
- It records the upload in the index and links the blob to the snapshot.
- It deletes the temporary file.
A file's record is written to the index only after every blob that holds its
chunks is uploaded.
### Snapshot metadata
`finalizeSnapshotMetadata` does these steps in order:
- `PopulateSnapshotBlobs` records every blob the snapshot uses.
- `UpdateSnapshotStatsExtended` stores the snapshot's totals.
- `ExportSnapshotMetadata` uploads the metadata.
- `MarkSnapshotComplete` marks the snapshot complete.
Completion is recorded last on purpose. A backup that stops during the upload
leaves the snapshot unfinished, and the next backup removes it and stores its
data again.
`ExportSnapshotMetadata` works on a copy of the index, never on the index
itself:
- It copies the index into a temporary directory with `VACUUM INTO`.
- It deletes the rows of every other snapshot from the copy. It also deletes the
files and blobs that this snapshot does not use. It then deletes each chunk
that no remaining file and no remaining blob uses.
- It keeps the `local_meta` and `schema_migrations` tables as they are. The
uploaded database therefore names the destination, and lists the migrations
that have run.
- It runs `VACUUM`, so that no deleted row is left in the file.
- It compresses the copy with zstd and encrypts it with age.
- It builds the manifest from the copy.
- It uploads `metadata/<remote-key>/db.zst.age`, and then
`metadata/<remote-key>/manifest.json.zst`.
The manifest is uploaded last. A directory under `metadata/` with no manifest
therefore marks a backup that did not finish.
## Restore
`Vaultik.Restore` does these steps:
- `restoreIdentities` parses the age private key. A missing or malformed key
stops the restore before anything is downloaded.
- `downloadSnapshotDB` downloads and decrypts the snapshot's database into a
temporary directory.
- `getFilesToRestore` selects the files, from the paths given.
- `restoreAllFiles` downloads blobs and writes the files.
- With `--verify`, `verifyRestoredFiles` reads each restored file and checks its
chunk hashes.
The restore does not read the local index or the manifest. It finds each file's
chunks and blobs through the downloaded database.
### Checks on the downloaded database
Decryption shows that a database is readable. It does not show that the
database is honest. The restore therefore checks it:
- `database.OpenReadOnly` opens the file read-only. It refuses a file that has a
trigger, a view or a virtual table, or that lacks an expected table.
- `verifySnapshotDBIdentity` checks that the snapshot in the database hashes to
the remote key that was asked for.
- `containedRestorePath` refuses a stored path that would leave the target
directory. It also refuses to write through a symlink below the target.
- A blob is not decompressed beyond the size that the database gives for it.
- Each downloaded blob is hashed. The hash must match the blob's name.
### Order of work
`restorePlan` orders the files so that few blobs are on disk at once:
- Downloaded blobs are kept decrypted in a cache directory.
- A file is restored once every blob it needs is in the cache.
- When no file is ready, the restore downloads the blobs for the file that
needs the fewest.
- `restoreSweeper` deletes a cached blob once no remaining file needs it.
A regular file is created with mode `0600` and gets its stored mode after its
content is written. A directory gets its stored mode, owner and modification
time after the whole restore.
## Verify
`Vaultik.VerifySnapshotWithOptions` checks a snapshot without the private key:
- It downloads the manifest.
- It checks that the snapshot's encrypted database is present.
- It checks that each listed blob is present with the listed size.
`Vaultik.RunDeepVerify` needs the private key. It does these steps:
- It downloads the manifest and the database. It checks the database as a
restore does.
- `verifyManifestAgainstDatabase` checks that both list the same blobs with the
same sizes.
- `verifyBlobExistenceFromDB` checks that every blob is present with the
recorded size.
- `verifyBlob` downloads each blob, checks the hash of each chunk in it, and
checks the hash that names the blob.
## Prune
`Vaultik.Prune` does three steps:
- `CleanupLocalSnapshots` drops from the local index each snapshot whose
manifest is gone from the destination.
- `PruneDatabase` removes unfinished snapshots and unused records from the
local index.
- `PruneBlobs` deletes unused data at the destination.
`PruneBlobs` reads every manifest at the destination. It then deletes two kinds
of object, once they are older than `prune_grace_period`:
- a blob that no manifest lists
- a directory under `metadata/` that has no manifest, and whose snapshot is not
in the local index
It deletes nothing when any manifest cannot be read.
[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#pruning) explains why this is
safe, and where it is not.
## The destination
`storage.Storer` is the only way the code reaches the destination. It has these
methods:
- `Put` and `PutWithProgress` store an object.
- `Get` reads an object.
- `Stat` reports an object's size.
- `Delete` removes an object.
- `List` and `ListStream` list the objects under a prefix.
- `DeletePartialUploads` removes what an interrupted upload left behind.
- `Info` describes the destination.
`storage.NewStorer` picks the implementation from `storage_url`:
- `s3://` gives the S3 implementation.
- `file://` gives the local directory implementation.
- `rclone://` gives the rclone implementation.
With no `storage_url`, it builds the S3 implementation from the `s3` settings.
## The marker file and the destination binding
Two checks keep the local index and the destination in step.
- `checkDestinationMarker` looks for `vaultik-destination.txt`. It fails when
the file is reported missing while the local index holds snapshots.
- `EnsureStorageBinding` records the configured destination in the local index
on first use. Later it fails when the config names a different destination.
Every command that changes data calls `EnsureStorageBinding` first.
## Concurrency
- The local index is opened with one connection, so its queries run one at a
time.
- The index runs in SQLite's WAL mode. A second process can therefore read it
while a backup writes to it.
- `Packer` guards its state with a mutex.
- `Scanner` holds `packerMu` while a chunk is added to the packer and while a
blob is closed.
- `Repositories.WithTx` runs a function in one transaction. It commits when the
function succeeds and rolls back when it fails.
### Key Type Instantiation Points
#### 1. Config (`config.Config`)
- **Created by**: `config.Module` via `config.LoadConfig()`
- **When**: Application startup (fx DI)
- **Contains**: All configuration from YAML file (S3 credentials, encryption keys, paths, etc.)
#### 2. Database (`database.DB`)
- **Created by**: `database.Module` via `database.New()`
- **When**: Application startup (fx DI)
- **Contains**: SQLite connection, path reference
#### 3. Repositories (`database.Repositories`)
- **Created by**: `database.Module` via `database.NewRepositories()`
- **When**: Application startup (fx DI)
- **Contains**: All repository interfaces (Files, Chunks, Blobs, Snapshots, etc.)
#### 4. Vaultik (`vaultik.Vaultik`)
- **Created by**: `vaultik.New(VaultikParams)`
- **When**: Application startup (fx DI)
- **Contains**: All dependencies for backup operations
```go
type Vaultik struct {
Globals *globals.Globals
Config *config.Config
DB *database.DB
Repositories *database.Repositories
Storage storage.Storer
ScannerFactory snapshot.ScannerFactory
SnapshotManager *snapshot.SnapshotManager
Shutdowner fx.Shutdowner
Fs afero.Fs
ctx context.Context
cancel context.CancelFunc
}
```
#### 5. SnapshotManager (`snapshot.SnapshotManager`)
- **Created by**: `snapshot.Module` via `snapshot.NewSnapshotManager()`
- **When**: Application startup (fx DI)
- **Responsibility**: Creates/completes snapshots, exports metadata to S3
#### 6. Scanner (`snapshot.Scanner`)
- **Created by**: `ScannerFactory(ScannerParams)`
- **When**: Each `CreateSnapshot()` call
- **Contains**: Chunker, Packer, progress reporter
```go
// vaultik/snapshot.go: CreateSnapshot()
scanner := v.ScannerFactory(snapshot.ScannerParams{
EnableProgress: !opts.Cron,
Fs: v.Fs,
})
```
#### 7. Chunker (`chunker.Chunker`)
- **Created by**: `chunker.NewChunker(avgChunkSize)`
- **When**: Inside `snapshot.NewScanner()`
- **Configuration**:
- `avgChunkSize`: From config (typically 64KB)
- `minChunkSize`: avgChunkSize / 4
- `maxChunkSize`: avgChunkSize * 4
#### 8. Packer (`blob.Packer`)
- **Created by**: `blob.NewPacker(PackerConfig)`
- **When**: Inside `snapshot.NewScanner()`
- **Configuration**:
- `MaxBlobSize`: Maximum blob size before finalization (typically 10GB)
- `CompressionLevel`: zstd level (1-19)
- `Recipients`: age public keys for encryption
```go
// snapshot/scanner.go: NewScanner()
packerCfg := blob.PackerConfig{
MaxBlobSize: cfg.MaxBlobSize,
CompressionLevel: cfg.CompressionLevel,
Recipients: cfg.AgeRecipients,
Repositories: cfg.Repositories,
Fs: cfg.FS,
}
packer, err := blob.NewPacker(packerCfg)
```
## Module Responsibilities
### `internal/cli`
Entry point for fx application. Combines all modules and handles signal interrupts.
Key functions:
- `NewApp(AppOptions)` → Creates fx.App with all modules
- `RunApp(ctx, app)` → Starts app, handles graceful shutdown
- `RunWithApp(ctx, opts)` → Convenience wrapper
### `internal/vaultik`
Main orchestrator containing all dependencies and command implementations.
Key methods:
- `New(VaultikParams)` → Constructor (fx DI)
- `CreateSnapshot(opts)` → Main backup operation
- `ListSnapshots(jsonOutput)` → List available snapshots
- `VerifySnapshot(id, deep)` → Verify snapshot integrity
- `PurgeSnapshots(...)` → Remove old snapshots
### `internal/chunker`
Content-defined chunking using FastCDC algorithm.
Key types:
- `Chunk` → Hash, Data, Offset, Size
- `Chunker` → avgChunkSize, minChunkSize, maxChunkSize
Key methods:
- `NewChunker(avgChunkSize)` → Constructor
- `ChunkReaderStreaming(reader, callback)` → Stream chunks with callback (preferred)
- `ChunkReader(reader)` → Return all chunks at once (memory-intensive)
### `internal/blob`
Blob packing: accumulates chunks, compresses, encrypts, tracks metadata.
Key types:
- `Packer` → Thread-safe blob accumulator
- `ChunkRef` → Hash + Data for adding to packer
- `FinishedBlob` → Completed blob ready for upload
- `BlobWithReader` → FinishedBlob + io.Reader for streaming upload
Key methods:
- `NewPacker(PackerConfig)` → Constructor
- `AddChunk(ChunkRef)` → Add chunk to current blob
- `FinalizeBlob()` → Compress, encrypt, hash current blob
- `Flush()` → Finalize any in-progress blob
- `SetBlobHandler(func)` → Set callback for upload
### `internal/snapshot`
#### Scanner
Orchestrates the backup process for a directory.
Key methods:
- `NewScanner(ScannerConfig)` → Constructor (creates Chunker + Packer)
- `Scan(ctx, path, snapshotID)` → Main scan operation
Scan phases:
1. **Phase 0**: Detect deleted files from previous snapshots
2. **Phase 1**: Walk directory, identify files needing processing
3. **Phase 2**: Process files (chunk → pack → upload)
#### SnapshotManager
Manages snapshot lifecycle and metadata export.
Key methods:
- `CreateSnapshot(ctx, hostname, version, commit)` → Create snapshot record
- `CompleteSnapshot(ctx, snapshotID)` → Mark snapshot complete
- `ExportSnapshotMetadata(ctx, dbPath, snapshotID)` → Export to S3
- `CleanupIncompleteSnapshots(ctx, hostname)` → Remove failed snapshots
### `internal/database`
SQLite database for local index. Single-writer mode for thread safety.
Key types:
- `DB` → Database connection wrapper
- `Repositories` → Collection of all repository interfaces
Repository interfaces:
- `FilesRepository` → CRUD for File records
- `ChunksRepository` → CRUD for Chunk records
- `BlobsRepository` → CRUD for Blob records
- `SnapshotsRepository` → CRUD for Snapshot records
- Plus join table repositories (FileChunks, BlobChunks, etc.)
## Snapshot Creation Flow
```
CreateSnapshot(opts)
│
├─► CleanupIncompleteSnapshots() // Critical: avoid dedup errors
│
├─► SnapshotManager.CreateSnapshot() // Create DB record
│
├─► For each source directory:
│ │
│ ├─► scanner.Scan(ctx, path, snapshotID)
│ │ │
│ │ ├─► Phase 0: detectDeletedFiles()
│ │ │
│ │ ├─► Phase 1: scanPhase()
│ │ │ Walk directory
│ │ │ Check file metadata changes
│ │ │ Build list of files to process
│ │ │
│ │ └─► Phase 2: processPhase()
│ │ For each file:
│ │ chunker.ChunkReaderStreaming()
│ │ For each chunk:
│ │ packer.AddChunk()
│ │ If blob full → FinalizeBlob()
│ │ → handleBlobReady()
│ │ → s3Client.PutObjectWithProgress()
│ │ packer.Flush() // Final blob
│ │
│ └─► Accumulate statistics
│
├─► SnapshotManager.UpdateSnapshotStatsExtended()
│
├─► SnapshotManager.CompleteSnapshot()
│
└─► SnapshotManager.ExportSnapshotMetadata()
│
├─► Copy database to temp file
├─► Clean to only current snapshot data (VACUUM)
├─► Compress binary SQLite with zstd
├─► Encrypt with age
├─► Upload db.zst.age to storage
└─► Upload manifest.json.zst to storage
```
## Deduplication Strategy
1. **File-level**: Files unchanged since last backup are skipped (metadata comparison: size, mtime, mode, uid, gid)
2. **Chunk-level**: Chunks are content-addressed by SHA256 hash. If a chunk hash already exists in the database, the chunk data is not re-uploaded.
3. **Blob-level**: Blobs contain only unique chunks. Duplicate chunks within a blob are skipped.
## Storage Layout in S3
```
bucket/
├── blobs/
│ └── {hash[0:2]}/
│ └── {hash[2:4]}/
│ └── {full-hash} # Compressed+encrypted blob
│
└── metadata/
└── {snapshot-id}/
├── db.zst.age # Encrypted binary SQLite database
└── manifest.json.zst # Blob list (for pruning/verification)
```
## Thread Safety
- `Packer`: Thread-safe via mutex. Multiple goroutines can call `AddChunk()`.
- `Scanner`: Uses `packerMu` mutex to coordinate blob finalization.
- `Database`: Single-writer mode (`MaxOpenConns=1`) ensures SQLite thread safety.
- `Repositories.WithTx()`: Handles transaction lifecycle automatically.
CLAUDE.md
+44
View File
@@ -0,0 +1,44 @@
# Rules
Read the rules in AGENTS.md and follow them.
# Memory
* Claude is an inanimate tool. The spam that Claude attempts to insert into
commit messages (which it erroneously refers to as "attribution") is not
attribution, as I am the sole author of code created using Claude. It is
corporate advertising for Anthropic and is therefore completely
unacceptable in commit messages.
* NEVER use `git add -A`. Always add only the files you intentionally
changed.
* Tests should always be run before committing code. No commits should be
made that do not pass tests.
* Code should always be formatted before committing. Do not commit
unformatted code.
* Code should always be linted before committing. Do not commit
unlinted code.
* The test suite is fast and local. When running tests, don't run
individual parts of the test suite, always run the whole thing by running
"make test".
* Do not stop working on a task until you have reached the definition of
done provided to you in the initial instruction. Don't do part or most of
the work, do all of the work until the criteria for done are met.
* We do not need to support migrations; schema upgrades can be handled by
deleting the local state file and doing a full backup to re-create it.
* When testing on a 2.5Gbit/s ethernet to an s3 server backed by 2000MB/sec SSD,
estimate about 4 seconds per gigabyte of backup time.
* When running tests, don't run individual tests, or grep the output. run
the entire test suite every time and read the full output.
* When running tests, don't run individual tests, or try to grep the output.
never run "go test". only ever run "make test" to run the full test
suite, and examine the full output.
Dockerfile
+75 -85
View File
@@ -1,101 +1,91 @@
# Lint phase. The linter is invoked directly rather than through `make
# lint` or `script/lint`, which are themselves a docker build and would
# recurse into a daemon that does not exist in a build step.
# golangci/golangci-lint:v2.14.0, 2026-10-05
FROM golangci/golangci-lint:v2.14.0@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f AS lint
WORKDIR /src
COPY go.mod go.sum ./
# Without the time limit a stalled connection to the module proxy holds
# this step, and every build on the host sharing it, until the kernel
# drops the connection, which can take an hour. The limit is well above
# the three minutes a working download can take on a busy build host.
RUN for attempt in 1 2 3; do \
timeout 300 go mod download && exit 0; \
echo "go mod download attempt $attempt failed or ran over 300s" >&2; \
done; \
echo "module download failed: giving up after 3 attempts" >&2; \
exit 1
COPY . .
# `golangci-lint run` silently ignores an unknown top-level key in
# .golangci.yml, such as a misspelt `linters:`; `config verify` fails on it.
RUN golangci-lint config verify --config .golangci.yml
RUN golangci-lint run --config .golangci.yml ./...
# Lint stage
#
# This FROM line is the single source of truth for the linter version:
# script/lint parses the image reference out of it and runs that exact
# image, so a local `make lint` and CI use the same linter. Bump the
# linter here (tag AND digest) and nowhere else.
#
# golangci/golangci-lint:v2.12.2-alpine, 2026-08-07
FROM golangci/golangci-lint:v2.12.2-alpine@sha256:91b27804074a0bacea298707f016911e60cf0cdbc6c7bf5ccacb5f0606d18d60 AS lint
# Test phase. -race needs cgo and so a C compiler, which the Debian Go
# image ships and the alpine one does not.
# golang:1.26.1 (Debian trixie), 2026-10-05
FROM golang:1.26.1@sha256:cd78d88e00afadbedd272f977d375a6247455f3a4b1178f8ae8bbcb201743a8a AS test
WORKDIR /src
COPY go.mod go.sum ./
# Limited and retried as in the lint phase above.
RUN for attempt in 1 2 3; do \
timeout 300 go mod download && exit 0; \
echo "go mod download attempt $attempt failed or ran over 300s" >&2; \
done; \
echo "module download failed: giving up after 3 attempts" >&2; \
exit 1
COPY . .
RUN go test -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -timeout 90s -race -v ./...; exit 1; }
RUN apk add --no-cache make build-base
# Build stage. Nothing is wanted from either phase above; the copies
# are what make BuildKit build them first, so this stage cannot run
# unless lint and test passed.
WORKDIR /src
# Copy go mod files first for better layer caching
COPY go.mod go.sum ./
RUN go mod download
# Copy source code
COPY . .
# Run formatting check and linter.
#
# CHECK_EPOCH must stay immediately above these RUNs. These layers are
# keyed on its value, so they are cache-eligible only for a value
# already built against this same tree. script/cibuild passes a fresh
# value on every invocation, which is what makes its green mean the
# checks really executed.
#
# The guarantee is conditional on that fresh value, not absolute. A
# build that omits --build-arg -- a bare `docker build .` -- gets an
# empty CHECK_EPOCH, and an empty string is a constant: the first such
# build runs the checks, and every one after it on an unchanged tree
# replays these layers from cache, never executing a check and still
# exiting 0, a green nothing earned. Gate through script/cibuild.
# Making the missing-arg case fail loudly instead is tracked in #91.
#
# CHECK_EPOCH is deliberately not referenced by the commands below: a
# declared-but-unreferenced ARG does enter BuildKit's cache key, which
# is measured on this host rather than assumed (PR #89). Upstream
# sneak/prompts #26 prefers expanding the value into the command so
# that the miss is contractual rather than dependent on that behavior
# staying as it is; adopting that here is tracked in #91. Do not delete
# this ARG as dead code -- the gate depends on it.
#
# ARG scope is per-stage, so the builder stage declares its own.
# Everything above this line (apk, go.mod, `go mod download`) is
# deliberately outside the busted range and keeps caching.
ARG CHECK_EPOCH
RUN make fmt-check
RUN make lint
# Build stage
# golang:1.26.1-alpine, 2026-03-17
FROM golang:1.26.1-alpine@sha256:2389ebfa5b7f43eeafbd6be0c3700cc46690ef842ad962f6c5bd6be49ed82039 AS builder
# Depend on lint stage passing
COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
RUN apk add --no-cache git
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
ARG VERSION=dev
# Install build dependencies for CGO (mattn/go-sqlite3) and sqlite3 CLI (tests)
RUN apk add --no-cache make build-base sqlite
WORKDIR /src
# Copy go mod files first for better layer caching
COPY go.mod go.sum ./
# Limited and retried as in the lint phase above.
RUN for attempt in 1 2 3; do \
timeout 300 go mod download && exit 0; \
echo "go mod download attempt $attempt failed or ran over 300s" >&2; \
done; \
echo "module download failed: giving up after 3 attempts" >&2; \
exit 1
RUN go mod download
# Copy source code
COPY . .
# The VERSION build arg when one is given, otherwise
# `git describe --tags --always` on the .git in the build context. The
# commit and its date always come from that .git. With .git present, a
# version that is still empty, dev or unknown, or a commit or date that
# is unknown, fails the build: git is missing or could not read the
# checkout, as when .git is a file pointing outside the context. A
# context without .git, such as a source export, stamps "dev" and an
# "unknown" commit and date.
ARG VERSION
RUN VERSION="${VERSION:-$(git describe --tags --always || echo dev)}"; \
commit="$(git rev-parse HEAD || echo unknown)"; \
commit_date="$(git show -s --format=%cs HEAD || echo unknown)"; \
if [ -e .git ]; then \
case "$VERSION" in ""|dev|unknown) \
echo "version is '$VERSION' although .git is present" >&2; \
exit 1 ;; \
esac; \
if [ "$commit" = unknown ] || [ "$commit_date" = unknown ]; then \
echo "commit is '$commit' and its date '$commit_date'" \
"although .git is present" >&2; \
exit 1; \
fi; \
fi; \
globals=sneak.berlin/go/vaultik/internal/globals; \
CGO_ENABLED=0 go build -trimpath \
-ldflags="-s -w -X ${globals}.Version=${VERSION} \
-X ${globals}.Commit=${commit} \
-X ${globals}.CommitDate=${commit_date}" \
-o /vaultik ./cmd/vaultik
# Run tests. See the CHECK_EPOCH comment in the lint stage, including
# the conditions the guarantee depends on; ARG scope is per-stage, so
# this stage needs its own declaration, and it must stay immediately
# above the check RUN.
ARG CHECK_EPOCH
RUN make test
# Runtime stage, and the last one: a plain `docker build .` builds this
# stage's chain and nothing else.
# Build (pure Go, no CGO required since we use modernc.org/sqlite)
RUN CGO_ENABLED=0 go build -ldflags "-X 'sneak.berlin/go/vaultik/internal/globals.Version=${VERSION}' -X 'sneak.berlin/go/vaultik/internal/globals.Commit=$(git rev-parse HEAD 2>/dev/null || echo unknown)' -X 'sneak.berlin/go/vaultik/internal/globals.CommitDate=$(git show -s --format=%cs HEAD 2>/dev/null || echo unknown)'" -o /vaultik ./cmd/vaultik
# Runtime stage
# alpine:3.21, 2026-02-25
FROM alpine:3.21@sha256:c3f8e73fdb79deaebaa2037150150191b9dcbfba68b4a46d70103204c53f4709
RUN apk add --no-cache ca-certificates
RUN apk add --no-cache ca-certificates sqlite
# Copy binary from builder
COPY --from=builder /vaultik /usr/local/bin/vaultik
Makefile
+17 -50
View File
@@ -1,23 +1,7 @@
.PHONY: all bootstrap setup check test lint lint-fix fmt fmt-check build clean deps test-coverage local install release release-snapshot docker hooks
.PHONY: all bootstrap setup check test lint lint-fix fmt fmt-check build clean deps test-coverage test-integration local install release release-snapshot docker hooks
# Where script/bootstrap installs Go when the host has none.
export PATH := $(PATH):$(CURDIR)/.tool/go/bin
# Version number, derived from git by script/version (`git describe
# --tags --always --dirty`). This used to be a hardcoded
# constant, which meant every local build claimed to be a release that
# had never been tagged.
VERSION := $(shell script/version)
# $(shell) discards exit status, so a script/version that is missing,
# non-executable or broken would otherwise leave VERSION empty and every
# binary built here would print "vaultik " with no version at all. A
# build that cannot determine what it is must not produce an artifact.
ifeq ($(strip $(VERSION)),)
$(error script/version produced no version string; a build that cannot \
determine its version will not be made. Check that script/version exists \
and is executable)
endif
# Version number
VERSION := 1.0.0-rc.1
# Build variables
GIT_REVISION := $(shell git rev-parse HEAD 2>/dev/null || echo "unknown")
@@ -43,14 +27,7 @@ setup:
check:
@script/check
# Run tests only, by building the test phase of the Dockerfile. This
# runs the ENTIRE suite -- there is no separate integration target and
# no build-tagged subset held back. In particular
# internal/vaultik/integration_test.go, which does full
# chunk -> pack -> encrypt -> upload -> restore round-trips, runs here.
# A `test-integration` target used to exist and was removed: no file in
# the repo carried a build tag, so `-tags=integration` selected nothing
# extra and the target was an exact duplicate of this one.
# Run tests only.
test:
@script/test
@@ -70,18 +47,7 @@ lint:
lint-fix:
@script/lint-fix
# Build binary. `build` is the name the org convention reaches for and
# the one a caller checks the exit code of; `vaultik` is the file rule
# that does the work, so an unchanged tree still short-circuits.
#
# This alias is not decorative. `build` was listed in .PHONY with no
# rule, and a phony target with no prerequisites and no recipe is
# already satisfied: `make build` printed "Nothing to be done" and
# exited 0 without producing a binary (issue #110). Every name in
# .PHONY needs a rule for that reason; TestPhonyTargetsAllHaveRules in
# cmd/vaultik keeps it that way.
build: vaultik
# Build binary.
vaultik: internal/*/*.go cmd/vaultik/*.go
go build -ldflags "$(LDFLAGS)" -o $@ ./cmd/vaultik
@@ -91,21 +57,22 @@ clean:
go clean
# Install dependencies. The linter is deliberately not installed here:
# script/lint lints by building the lint phase of the Dockerfile, whose
# FROM line is the single source of truth for the linter version. A
# second, separately pinned copy on PATH could drift from it and make a
# local `make lint` disagree with CI.
# script/lint runs the digest-pinned golangci-lint image declared by the
# Dockerfile's lint stage, which is the single source of truth for the
# linter version. A second, separately pinned copy on PATH could drift
# from it and make a local `make lint` disagree with CI.
deps:
go mod download
# Run tests with coverage, on the host. -count=1 because without it an
# unchanged package is served from Go's test result cache, and a
# coverage profile assembled from cached results describes a run that
# did not happen.
# Run tests with coverage.
test-coverage:
go test -v -count=1 -coverprofile=coverage.out ./...
go test -v -coverprofile=coverage.out ./...
go tool cover -html=coverage.out -o coverage.html
# Run integration tests.
test-integration:
go test -v -tags=integration ./...
local:
VAULTIK_CONFIG=$(HOME)/etc/vaultik/config.yml ./vaultik snapshot --debug list 2>&1
VAULTIK_CONFIG=$(HOME)/etc/vaultik/config.yml ./vaultik snapshot --debug create 2>&1
@@ -115,11 +82,11 @@ install: vaultik
# Build and publish release artifacts (linux/darwin × amd64/arm64) via goreleaser.
release:
@script/release
goreleaser release --clean
# Dry-run a release build without publishing or tagging.
release-snapshot:
@script/release-snapshot
goreleaser release --clean --snapshot
# Build Docker image.
docker:
README.md
+547 -1282
View File
File diff suppressed because it is too large. Load diff
REPO_POLICIES.md
+82 -353
View File
@@ -1,6 +1,6 @@
---
title: Repository Policies
last_modified: 2026-10-04
last_modified: 2026-07-06
---
This document covers repository structure, tooling, and workflow standards. Code
@@ -60,28 +60,17 @@ style conventions are in separate documents:
prerequisite since nvm requires bash. yarn is then pinned via
`corepack prepare yarn@<version> --activate`. Never install "latest" or "lts";
always exact versions. `script/cibuild` runs the CI build: it changes to the
repo root, runs `script/bootstrap`, runs `script/check`, and builds the image
with the version; the Gitea workflow calls it. **`script/cibuild` runs
`script/bootstrap` first**, because the workflow checks out the repo and runs
nothing else, while `script/fmt-check` runs the formatter on the host: on a
pristine checkout with nothing installed the run dies there, after the
containerised gates have passed. **The bootstrap alone is not enough**:
`script/bootstrap` installs node and yarn under nvm and leaves neither on the
`PATH` of the shell that called it, so a bare `yarn` still exits 127. The host
entrypoints that need yarn — `script/fmt` and `script/fmt-check` — therefore
source nvm for the pinned node version before invoking it, exactly as
`script/bootstrap`'s own install step does. A runner carrying nothing but
docker and git then gets through `script/check`. Four further scripts are our
own extensions to the standard: `script/check` runs `script/test`,
`script/lint` and `script/fmt-check`; `script/precommit` is what the git
pre-commit hook runs, and it calls `script/check`; `script/install-precommit`
installs the git pre-commit hook (the `make hooks` target shims to it); and
`script/projectname` (literally that filename) simply outputs the project's
name. Scripts that need the name call `script/projectname` — e.g.
`script/docker` assembles its image tag from it — so those scripts stay
byte-identical across all repos. Repo-type-specific pre-commit extras (e.g.
`go mod tidy` verification in Go repos) belong in `script/precommit`, not in
the hook itself. Model scripts are at
repo root and runs `docker build .`; the Gitea workflow calls it. Four further
scripts are our own extensions to the standard: `script/check` runs
`script/test`, `script/lint`, and `script/fmt-check`; `script/precommit` is
what the git pre-commit hook runs, and it calls `script/check`;
`script/install-precommit` installs the git pre-commit hook (the `make hooks`
target shims to it); and `script/projectname` (literally that filename) simply
outputs the project's name. Scripts that need the name call
`script/projectname` — e.g. `script/docker` assembles its image tag from it —
so those scripts stay byte-identical across all repos. Repo-type-specific
pre-commit extras (e.g. `go mod tidy` verification in Go repos) belong in
`script/precommit`, not in the hook itself. Model scripts are at
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
must document the provided scripts in an **Entrypoints** section (see the
README requirements below).
@@ -100,198 +89,87 @@ style conventions are in separate documents:
contributor should be able to understand the entire development workflow by
reading the Makefile.
- Every repo should have a `Dockerfile`, and it carries the repo's gates: a
`lint` phase and a `test` phase, with the final stage depending on both so the
image cannot be built unless they pass. For non-server repos the final stage
brings up a development environment; for server repos it is the runtime image.
The gate phases and the build stage start from their pinned base images and
install what those images lack either inline, as the canonical Go `Dockerfile`
below does for `git`, or by running `script/bootstrap`, as the `prompts`
repo's own `Dockerfile` does for its yarn packages. The development
environment stage installs development prerequisites by running
`script/bootstrap` rather than duplicating its installs inline. A stage that
runs `script/bootstrap` COPYs `script/` and the dependency manifests
(`package.json` + `yarn.lock`, `go.mod` + `go.sum`, etc.) before running it.
- Every repo should have a `Dockerfile`. All Dockerfiles must run `make check`
as a build step so the build fails if the branch is not green. For non-server
repos, the Dockerfile should bring up a development environment and run
`make check`. For server repos, `make check` should run as an early build
stage before the final image is assembled. Dockerfiles install development
prerequisites by running `script/bootstrap` rather than duplicating installs
inline; COPY `script/` and the dependency manifests (`package.json` +
`yarn.lock`, `go.mod` + `go.sum`, etc.) before running it so the bootstrap
layer stays cached until dependencies change.
- **Linting and testing run in Docker, as phases of the `Dockerfile`.** There is
no separate lint file. `script/lint` and `script/test` each build one phase
and nothing else:
- **Dockerfiles must use a separate lint stage for fail-fast feedback.** Go
repos use a multistage build where linting runs in an independent stage based
on the `golangci/golangci-lint` image (pinned by hash). This stage runs
`make fmt-check` and `make lint` before the full build begins. The build stage
then declares an explicit dependency on the lint stage via
`COPY --from=lint /src/go.sum /dev/null`, which forces BuildKit to complete
linting before proceeding to compilation and tests. This ensures lint failures
surface in seconds rather than minutes, without blocking on dependency
download or compilation in the build stage.
```sh
docker build --no-cache --target lint -t "$(script/projectname)-lint" .
docker build --no-cache --target test -t "$(script/projectname)-test" .
```
**A stage that is not the last one in the file is built only when the final
stage's chain depends on it, or when `--target` names it.** That is why the
two gates are always invoked by name here, and why the final stage carries a
`COPY --from=` of a harmless file from each of them: without that edge a
plain `docker build .` builds the last stage alone and exits 0 having linted
and tested nothing.
**Every `docker build` in `script/` is tagged**, here and in
`script/cibuild` and `script/docker`. An untagged build leaves a dangling
image behind on every invocation, on every developer host and every CI
runner; a tagged one replaces the previous image.
Inside a phase the tool is invoked directly — `golangci-lint`, `go test`,
`eslint`, `prettier` — never through `make lint` or `script/test`, which are
themselves a `docker build` and would recurse into a daemon that does not
exist in a build step. Formatting is the exception and stays on the host:
`script/fmt` writes the working tree, and `script/fmt-check` is its
read-only twin.
**No lint verdict may come from a host invocation of the linter.** On a
shared host golangci-lint reads a result cache keyed on file content rather
than location, so a second checkout of the same content is served the first
one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
exit non-zero with `parallel golangci-lint is running` — a status a caller
cannot tell from real findings. Both have produced wrong verdicts in this
org, in both directions. A container has its own cache, its own `TMPDIR` and
a digest-pinned binary, so neither is reachable.
- **Any build that runs checks is built with `--no-cache`.** Docker invalidates
a `COPY` layer only when the copied content changes, so on an unchanged tree
the check `RUN` is served from cache, nothing executes, and the build still
exits 0. Every `docker build` in `script/` therefore passes `--no-cache`:
`script/lint`, `script/test`, `script/cibuild` and `script/docker` are the
four, and there is no fifth — `script/check` runs the two gate phases and
`script/fmt-check`, and builds no image of its own. A bare `docker build .` is
not evidence that anything ran: a sub-second build reporting success is a
cache hit, not a result. Never invalidate by pruning — `docker builder prune`
and friends destroy a build cache shared with every other build on the host.
When a check is added or changed, prove it works by planting a defect it must
catch and watching the run fail on it, then revert the defect. A green run
alone shows neither that the check ran nor that it covers what it should.
- **The gate phases are separate stages, and the build stage depends on both.**
The lint phase is based on the `golangci/golangci-lint` image (pinned by
hash), so lint failures surface in seconds rather than after a full compile,
and the test phase is based on the Debian Go image. The canonical Go repo
`Dockerfile`:
The standard pattern for a Go repo Dockerfile is:
```dockerfile
# Lint phase
# Lint stage — fast feedback on formatting and lint issues
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD
FROM golangci/golangci-lint@sha256:... AS lint
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN golangci-lint run --config .golangci.yml ./...
RUN make fmt-check
RUN make lint
# Test phase. -race needs cgo and so a C compiler, which the Debian Go
# image ships and the alpine one does not.
# golang:1.x, YYYY-MM-DD
FROM golang@sha256:... AS test
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN go test -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -timeout 90s -race -v ./...; exit 1; }
# Build stage. Nothing is wanted from either phase above; the copies
# are what make BuildKit build them first, so this stage cannot run
# unless lint and test passed.
# Build stage
# golang:1.x-alpine, YYYY-MM-DD
FROM golang@sha256:... AS builder
COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
RUN apk add --no-cache git
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
WORKDIR /src
# Force BuildKit to run the lint stage before proceeding
COPY --from=lint /src/go.sum /dev/null
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN make test
# The VERSION build arg when one is given, otherwise
# `git describe --tags --always` on the .git in the build context. With
# .git present, a version that is still empty, dev or unknown fails the
# build: git is missing or could not read the checkout.
ARG VERSION
RUN VERSION="${VERSION:-$(git describe --tags --always)}"; \
if [ -e .git ]; then \
case "$VERSION" in ""|dev|unknown) \
echo "version is '$VERSION' although .git is present" >&2; \
exit 1 ;; \
esac; \
fi; \
CGO_ENABLED=0 go build -trimpath \
ARG VERSION=dev
RUN CGO_ENABLED=0 go build -trimpath \
-ldflags="-s -w -X main.Version=${VERSION}" \
-o /app ./cmd/app/
# Runtime stage, and the last one
# Runtime stage
FROM alpine@sha256:...
COPY --from=builder /app /usr/local/bin/app
ENTRYPOINT ["app"]
```
Key points:
- The lint phase uses the `golangci/golangci-lint` image directly (it has
both Go and the linter), so nothing needs installing.
- `COPY --from=<phase> /src/go.sum /dev/null` is a no-op copy whose only
purpose is the ordering edge. BuildKit runs stages in parallel by default,
and a stage nothing depends on is not built at all, so without these two
lines a red gate would not fail the build.
- Keep the runtime stage last, and if you add a stage after it, give it the
same two copies. A plain `docker build .` builds the last stage's chain
and nothing else.
- The lint stage uses the `golangci/golangci-lint` image directly (it
includes both Go and the linter), so there is no need to install the
linter separately.
- `COPY --from=lint /src/go.sum /dev/null` is a no-op file copy that creates
a stage dependency. BuildKit runs stages in parallel by default; without
this line, the build stage would not wait for lint to finish and a lint
failure might not fail the overall build.
- If the project uses `//go:embed` directives that reference build artifacts
(e.g. a web frontend compiled in a separate stage), the lint phase must
(e.g. a web frontend compiled in a separate stage), the lint stage must
create placeholder files so the embed directives resolve. Example:
`RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`.
- If the project requires CGO or system libraries for linting, install them
in the lint phase. The `golangci/golangci-lint` image is Debian-based and
has no `apk`, so install with `apt-get` under the Debian package name
(`libvips-dev`, where alpine says `vips-dev`), and delete the package
lists in the same `RUN`, so the layer does not keep them:
```dockerfile
RUN apt-get update \
&& apt-get install -y --no-install-recommends libvips-dev \
&& rm -rf /var/lib/apt/lists/*
```
- `.dockerignore` lets `.git` into the build context. It keeps out every git
`config` at any depth (`**/.git/config`, `**/.git/modules/**/config`): the
repository's own, each submodule's under `.git/modules/`, and that of a
submodule keeping its own `.git` directory. `git describe` does not need
them, and each can hold a credential: a password in a remote URL, or the
token the CI checkout step stores there. A submodule whose name has a
`config` segment (`config`, `deploy/config`, `config/lib`) loses its whole
git directory to `**/.git/modules/**/config`, and Go's version stamping
then fails the build: give it a name without that segment
(`git submodule add --name`). The stage that compiles has `git` (the
Debian Go image has it; an alpine one needs `apk add --no-cache git`) and
takes the version from the `VERSION` build argument when one is given,
otherwise from `git describe --tags --always`. That gives the tag on a
tagged commit; on a later commit, the tag, the number of commits since it
and the short commit (`v1.2.3-4-gabc1234`); and the short commit when no
tag is reachable. The stage that compiles also marks its working directory
safe for git (`git config --system --add safe.directory /src`): a context
sent as a tar stream keeps the sender's file owners, and git refuses a
checkout owned by another user, so the version would come out empty.
`ARG VERSION` has no default, and the build fails if the context carries
`.git` and the version still comes out empty, `dev` or `unknown`. A plain
`docker build .` with no build arguments must succeed; a Dockerfile that
refuses an empty build argument drops that refusal and keeps the argument.
The lint stage should not depend on the actual build output — it exists to
fail fast.
- If the project requires CGO or system libraries for linting (e.g.
`vips-dev`), install them in the lint stage with `apk add`.
- The build stage runs `make test` after compilation setup. Tests run in the
build stage, not the lint stage, because they may require compiled
artifacts or heavier dependencies.
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
runs `script/cibuild` on push, and checks out the repo as its only other step.
That script bootstraps, runs the gate phases, and then builds the image, so a
successful run means every check passed; a bare `docker build .` does not
carry the same guarantee, because its gate phases may come from the cache. The
image build is uncached and so runs the gate phases a second time. That is the
price of the rule above, and it is worth paying: the image that ships is built
from a run of its own gates rather than from a cache entry. A separate
workflow limited to `main` by a `branches` list under `on: push` cannot be
checked by review: to try a change to it, add the feature branch to that list
and push, then remove the branch from the list again before merging. Keep any
job in it that publishes behind `if: github.ref_name == 'main'`, so the run
from the feature branch publishes nothing.
runs `script/cibuild` (which runs `docker build .`) on push. Since the
Dockerfile already runs `make check`, a successful build implies all checks
pass.
- Use platform-standard formatters: `black` for Python, `prettier` for
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
@@ -311,21 +189,14 @@ style conventions are in separate documents:
module under test to verify it compiles/parses. There is no excuse for
`make test` to be a no-op.
- `make test` must complete in under 60 seconds. That is the hard cap, and a
suite that exceeds it fails. Under 20 seconds is the target. A suite between
20 and 60 seconds is still green, but the overage must be filed as an
improvement bug against that repo. Add a 90-second timeout to the test
invocation (`go test -timeout 90s`). The backstop deliberately sits above the
hard cap so that it catches a genuinely hung test rather than a merely slow
one.
- `make test` must complete in under 20 seconds. Add a 30-second timeout in the
Makefile.
- **The test command should use the conditional verbose rerun pattern.** Run
tests without `-v` (verbose) first. If tests fail, automatically rerun with
`-v` to show full output. This keeps CI logs and `docker build` output clean
on success (just package/suite summaries) while providing full diagnostic
detail on failure (every test case, every assertion). The command lives in the
`test` phase of the `Dockerfile`, since `script/test` builds that phase; the
Makefile form below is the same pattern for any repo-local invocation:
- **`make test` should use the conditional verbose rerun pattern.** Run tests
without `-v` (verbose) first. If tests fail, automatically rerun with `-v` to
show full output. This keeps CI logs and `docker build` output clean on
success (just package/suite summaries) while providing full diagnostic detail
on failure (every test case, every assertion). The general shell pattern:
```makefile
test:
@@ -338,26 +209,11 @@ style conventions are in separate documents:
```makefile
test:
@go test -count=1 -timeout 90s -race -cover ./... || \
@go test -timeout 30s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -count=1 -timeout 90s -race -v ./...; exit 1; }
go test -timeout 30s -race -v ./...; exit 1; }
```
`-count=1` is required on both invocations: it defeats Go's test _result_
cache, so neither run can report a stored pass in place of running the
tests. It leaves the build cache alone, so it costs the runtime of the suite
and no recompilation.
That cache is Go's own, separate from Docker's layer cache. Go stores a
passing result in its cache directory (`GOCACHE`), and when the same tests
run again on unchanged code it prints that result, marked `(cached)`,
without running them. That matters on a developer's machine, where this
target runs and the directory lasts from one run to the next. The `test`
phase of the `Dockerfile` needs no `-count=1`: its base image holds no
result for this repo's tests and nothing before its `go test` step runs a
test, so there is nothing to replay. `--no-cache` (above) is what makes that
step run on an unchanged tree.
Python example:
```makefile
@@ -383,84 +239,10 @@ style conventions are in separate documents:
must be in `.gitignore`. No exceptions.
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`),
language build artifacts, and `node_modules/`. Fetch the standard `.gitignore`
from `https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when
setting up a new repo. These patterns are written to `.gitignore`'s own
semantics, in which an unanchored pattern already matches at every depth; they
are not a `.dockerignore` and must not be transplanted into one unmodified.
- **`.dockerignore` does not use `.gitignore` semantics, and copying patterns
across unmodified leaves secrets in the build context.** Docker matches with
`moby/patternmatcher`: `filepath.Match` semantics plus a `**` extension, so
`*` does not cross `/` and a pattern without a leading `**/` is anchored at
the build-context root. A `.dockerignore` listing `.env`, `*.pem` and `*.key`
therefore excludes only the copies at the repository root, while `config/.env`
and `certs/server.key` still reach the context and can land in an image layer
— which is more dangerous than a short file with no secret patterns at all,
because it reads as solved and stops anyone looking. Give every
depth-independent pattern the `**/` prefix and leave only genuinely
root-anchored entries unprefixed: `.claude`, and the repo's own host-built
binary, written `/myapp` and never `**/myapp`, which would also match
`cmd/myapp/` and delete the package directory from the context. Matching is
case-sensitive, and an ALL-CAPS twin per pattern still misses `Server.Key`, so
secret names use character ranges — `**/*.[kK][eE][yY]`, `**/*.[pP][eE][mM]`,
and likewise for `.envrc` and the extensionless SSH keys. Where such a pattern
also catches something the build needs, re-include it with a negation
(`!docs/example.env`); deleting the pattern reopens the exposure for every
other file it covers. Fetch the standard `.dockerignore` from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.dockerignore` and extend
it with the repo's own artifacts.
- **In-repo agent scratch belongs in both files, written to each file's own
semantics.** `.claude/` holds one worktree per in-flight agent — an entire
additional checkout of the repo — so under `COPY . .` the build context
inflates by a multiple of the repo and another session's unreviewed work can
be copied into an image layer. In `.gitignore` the entry is `.claude/`,
unanchored. In `.dockerignore` it is `.claude`, anchored and with **no** `**/`
prefix, because the prefixed form would also delete any nested directory of
that name from the build. Anchoring carries a known gap that the canonical
`.dockerignore` states in its own comment, since consuming repos receive the
file and not the tracker: the directory is created in the agent's working
directory, so a repo running agents in subdirectories still ships
`services/api/.claude/` and must add its own anchored entry there.
- **A plain `docker build .` of a clone stamps the version that
`git describe --tags --always` gives**, derived from the `.git` in the build
context as the canonical `Dockerfile` above shows. Without its failure check,
a missing `git` or an unreadable checkout would leave `-X main.Version=` empty
and the build would still exit 0. `script/docker` and `script/cibuild` pass
the version they compute on the host; it takes precedence. They do this
byte-identically across repos:
```sh
# Own line: a failing command substitution inside an argument does not
# trip `set -e`, so the inline form degrades to an empty constant.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$(script/projectname)" .
```
`--always` makes an untagged repo yield an abbreviated commit hash rather
than failing, and the `[ -n "$version" ]` line is the single place the
fallback is applied — a live check that fires on a build from an export with
no `.git` and on a repository with no commits yet. Do not fold it into the
substitution as `|| echo unknown`, which makes the guard unreachable. The
Dockerfile's side is `ARG VERSION` in the stage that compiles, declared
there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
the scripts stay byte-identical. One consequence for CI: the standard
checkout action clones shallow and fetches no tags, so a repo that embeds a
tag-derived version must set `fetch-depth: 0` on its checkout step.
- **Verify `.dockerignore` by enumerating the image, not by reading the
patterns.** Plant files at the root _and_ at least two directories deep, build
a probe image that does `COPY . .`, and list what actually landed
(`docker run --rm --entrypoint find IMAGE /app`). The `transferring context`
size is not a substitute: a nested secret is a few bytes, and BuildKit
transfers only the delta from the previous build.
editor files (`.swp`, `*~`), language build artifacts, and `node_modules/`.
Fetch the standard `.gitignore` from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when setting up
a new repo.
- **No build artifacts in version control.** Code-derived data (compiled
bundles, minified output, generated assets) must never be committed to the
@@ -476,56 +258,9 @@ style conventions are in separate documents:
- Make all changes on a feature branch. You can do whatever you want on a
feature branch.
- `.golangci.yml` is standardized. The vendored copy in a consuming repo must
_NEVER_ be modified by an agent: fetch it from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml` and keep it
byte-identical, so that no repo can quietly loosen its own linting. Linter
configuration changes are made to the canonical copy in the `prompts` repo and
reach consuming repos by re-vendoring; an agent may open a PR against
canonical, which only the user merges. One list is exempt from byte-identity,
because it cannot be written once for every repo: the `deny` list of the
`test-support` depguard rule, where a repo names its own test-support packages
by full import path. A repo adds entries there and changes nothing else, and a
re-vendor carries its entries forward. The canonical golangci-lint version is
v2.14.0 (released 2026-09-24), pinned as the digest of the lint phase's base
image
(`golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f`,
which reports `2.14.0 built with go1.27.0 from 114493f9`). A module's `go`
directive must not name a newer Go minor version than the one golangci-lint
was built with, or golangci-lint refuses to lint it: this release lints
`go 1.27.1` but not `go 1.28`. That digest is the only pin, since no repo
installs golangci-lint on the host. A repo sets the lint phase digest to the
one named here and re-vendors `.golangci.yml` in the same commit, whichever of
the two prompted the change: the canonical copy can name linters that an older
golangci-lint rejects, and a newer golangci-lint can add linters that
`default: all` switches on until the canonical copy disables them.
- **`script/bootstrap` installs a pinned tool by comparing versions, never by
testing presence.** An `if ! command -v <tool>; then install; fi` guard tests
`PATH` only, so on an already-provisioned machine the pin is inert and a
version bump is a silent no-op — while the Dockerfile, installing into a clean
image, gets the pinned version, so a local `make check` and `make docker` can
disagree about what the tool even is. The canonical form:
- compares the installed version against the pin over the **whole** version
token; a parser that stops at the first `-` reports `2.12.2` for a host
running `2.12.2-rc1` and skips the install;
- treats absent, non-zero, empty or unrecognised `--version` output as a
mismatch, so the failure direction is a redundant install and never a
skipped one;
- after installing, re-resolves the binary the way callers do — `hash -r`,
then through `PATH`, not through the directory the installer wrote to —
and fails naming the resolved path, since an install that a shadowing
binary hides succeeds while changing nothing any caller sees;
- is actually called, and prints the version on both success paths: a
function defined and never invoked has the same exit status and the same
empty output as one that worked.
Keep it POSIX sh: no arrays, no `[[`, no `grep -P`.
A Go tool a repo needs on the host is installed with `go install` pinned to
a commit hash (`go install <package>@<commit hash>`). It is never tracked as
a `go.mod` tool dependency or through a `tools.go` file, either of which
pulls the tool's own dependencies into the repo's `go.mod` and `go.sum`.
- `.golangci.yml` is standardized and must _NEVER_ be modified by an agent, only
manually by the user. Fetch from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml`.
- When pinning images or packages by hash, add a comment above the reference
with the version and date (YYYY-MM-DD).
@@ -639,14 +374,12 @@ style conventions are in separate documents:
settings.
- Avoid putting files in the repo root unless necessary. Root should contain
only project-level config files (`README.md`, `AGENTS.md`, `Makefile`,
`Dockerfile`, `LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`,
and language-specific config). Everything else goes in a subdirectory.
Canonical subdirectory names:
only project-level config files (`README.md`, `Makefile`, `Dockerfile`,
`LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`, and
language-specific config). Everything else goes in a subdirectory. Canonical
subdirectory names:
- `bin/` — executable scripts and tools
- `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose
body is a single call into `internal/` or `pkg/`, no project logic in
`cmd/`
- `cmd/` — Go command entrypoints
- `configs/` — configuration templates and examples
- `deploy/` — deployment manifests (k8s, compose, terraform)
- `docs/` — documentation and markdown (README.md stays in root)
@@ -673,7 +406,3 @@ style conventions are in separate documents:
- Go: `go.mod`, `go.sum`, `.golangci.yml`
- JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore`
- Python: `pyproject.toml`
- Guidance for coding agents lives in one `AGENTS.md` at the repository root. It
is never committed under a file or directory named after one agent tool, such
as `CLAUDE.md` or `.claude/`, and never split into separate memory files.
TODO.md
+6 -1236
View File
File diff suppressed because it is too large. Load diff
cmd/vaultik/dockermoddownload_test.go
-45
View File
@@ -1,45 +0,0 @@
package main_test
import (
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// TestDockerfileBoundsEveryModuleDownload fails unless every
// `go mod download` in the Dockerfile runs under a time limit, is tried
// a fixed number of times, and then fails the build with a message that
// names the module download (issue #351). Like the version tests, this
// parses the Dockerfile; that a stalled download really is cut off and
// retried was checked by hand.
func TestDockerfileBoundsEveryModuleDownload(t *testing.T) {
t.Parallel()
var downloads []string
for _, instruction := range instructions(t, productDockerfile) {
if strings.Contains(instruction, "go mod download") {
downloads = append(downloads, instruction)
}
}
require.NotEmpty(t, downloads, "%s must download modules",
productDockerfile)
for _, download := range downloads {
assert.Contains(t, download, "timeout 300 go mod download",
"%s: a module download must have a time limit: %s",
productDockerfile, download)
assert.Contains(t, download, "for attempt in 1 2 3",
"%s: a module download must be retried: %s",
productDockerfile, download)
assert.Contains(t, download, "module download failed",
"%s: a module download that never succeeds must say so: %s",
productDockerfile, download)
assert.True(t, strings.HasSuffix(download, "exit 1"),
"%s: a module download that never succeeds must fail the"+
" build: %s", productDockerfile, download)
}
}
cmd/vaultik/dockerversion_test.go
-149
View File
@@ -1,149 +0,0 @@
package main_test
import (
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// This file guards the version stamping of the product image (issue
// #75). The failure it protects against is silent: the image still
// builds and runs, but `vaultik version` inside it reports "commit:
// unknown" or a version of "dev", so an operator cannot tell which
// source produced a given backup. The build takes the version as a
// build arg, which script/docker computes on the host, and otherwise
// derives it from the .git in its context; the commit and its date
// always come from that .git.
//
// These are parses of the committed files, because shelling out to
// docker would nest a build inside `make test`. That `vaultik version`
// in the built image really prints the host's version is verified by
// hand.
// The files under guard, relative to the repository root.
const (
productDockerfile = "Dockerfile"
dockerScript = "script/docker"
)
// TestProductDockerfileTakesVersionAsBuildArg fails unless the build
// declares ARG VERSION, with no default, and stamps it into the binary
// whenever it is given, ahead of the value derived in the container.
func TestProductDockerfileTakesVersionAsBuildArg(t *testing.T) {
t.Parallel()
found := instructions(t, productDockerfile)
require.Contains(t, found, "ARG VERSION",
"%s must declare `ARG VERSION`, with no default, so the host can"+
" pass it in", productDockerfile)
assertLdflagReferences(t, found, "VERSION")
}
// TestProductDockerfileDerivesVersionFromGit fails unless a build given
// no VERSION, such as a plain `docker build .` of a clone, takes it from
// `git describe` of the .git in its context, stamps "dev" when the
// context has no .git, and fails rather than stamp "dev" when that .git
// yields no version. The commit and its date come from the same .git,
// and the build fails rather than stamp them "unknown" when it is
// present.
func TestProductDockerfileDerivesVersionFromGit(t *testing.T) {
t.Parallel()
found := instructions(t, productDockerfile)
buildAt := indexContaining(found, "go build")
require.GreaterOrEqual(t, buildAt, 0, "%s must build", productDockerfile)
assert.Contains(t, found[buildAt], "git describe --tags --always || echo dev",
"%s must derive the version from git when no VERSION is given,"+
" and stamp dev when the context has no .git", productDockerfile)
assert.Contains(t, found[buildAt], "[ -e .git ]",
"%s must fail when the context carries .git but yields no version",
productDockerfile)
assert.Contains(t, found[buildAt], "git rev-parse HEAD",
"%s must stamp the commit from git", productDockerfile)
assert.Contains(t, found[buildAt], "git show -s --format=%cs HEAD",
"%s must stamp the commit date from git", productDockerfile)
assert.Contains(t, found[buildAt],
`[ "$commit" = unknown ] || [ "$commit_date" = unknown ]`,
"%s must fail when the context carries .git but yields no commit"+
" or date", productDockerfile)
}
// TestDockerScriptComputesVersionOnTheHost fails unless script/docker
// passes the version it derives where .git exists as a build arg.
func TestDockerScriptComputesVersionOnTheHost(t *testing.T) {
t.Parallel()
script := readRepoFile(t, dockerScript)
assert.Contains(t, script, "--build-arg VERSION=",
"%s must pass --build-arg VERSION to the build", dockerScript)
}
// assertLdflagReferences fails unless the build instruction uses the
// named ARG whenever it is given (a ${arg:- reference), so a value
// passed in is not overridden by one derived inside the container.
func assertLdflagReferences(t *testing.T, found []string, arg string) {
t.Helper()
for _, instruction := range found {
if strings.HasPrefix(instruction, "RUN ") &&
strings.Contains(instruction, "go build") &&
strings.Contains(instruction, "${"+arg+":-") {
return
}
}
assert.Fail(t, "version arg is declared but never stamped",
"the go build in %s must use ${%s:-...}, or the arg is passed and"+
" discarded", productDockerfile, arg)
}
// instructions returns the Dockerfile's instructions, one per element,
// with comments and blank lines dropped and continuation lines joined,
// so a multi-line RUN is one string.
func instructions(t *testing.T, name string) []string {
t.Helper()
var (
out []string
continued string
isContinued bool
)
for line := range strings.SplitSeq(readRepoFile(t, name), "\n") {
trimmed := strings.TrimSpace(line)
if !isContinued && (trimmed == "" || strings.HasPrefix(trimmed, "#")) {
continue
}
isContinued = strings.HasSuffix(trimmed, `\`)
continued += strings.TrimSuffix(trimmed, `\`)
if isContinued {
continue
}
out = append(out, strings.Join(strings.Fields(continued), " "))
continued = ""
}
return out
}
// indexContaining returns the position of the first instruction
// containing want; -1 if there is none.
func indexContaining(found []string, want string) int {
for i, instruction := range found {
if strings.Contains(instruction, want) {
return i
}
}
return -1
}
cmd/vaultik/main.go
+1 -11
View File
@@ -10,16 +10,6 @@ import (
)
func main() {
os.Exit(run())
}
// run sets up optional profiling, runs the CLI, and returns the process
// exit code. os.Exit lives in main so it fires only after run's deferred
// profile writers have flushed. cli.Entry returns a status code rather
// than calling os.Exit itself: an os.Exit from inside it would skip
// these defers and truncate the profile of a failing command -- exactly
// the command one most often wants to profile.
func run() int {
// CPU profiling: set VAULTIK_CPUPROFILE=/path/to/cpu.prof
if cpuProfile := os.Getenv("VAULTIK_CPUPROFILE"); cpuProfile != "" {
f, err := os.Create(cpuProfile) //nolint:gosec // G304: operator-set path
@@ -56,5 +46,5 @@ func run() int {
}()
}
return cli.Entry()
cli.Entry()
}
cmd/vaultik/makefile_test.go
-187
View File
@@ -1,187 +0,0 @@
package main_test
import (
"os"
"path/filepath"
"regexp"
"slices"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// This file guards the Makefile that builds this program, which is why
// it lives beside it rather than in a package of its own.
//
// Issue #110: `build` was listed in .PHONY with no `build:` rule
// anywhere in the file. That combination is silently successful — make
// considers a phony target with no prerequisites and no recipe already
// satisfied, so `rm -f vaultik && make build` printed "Nothing to be
// done for 'build'" and exited 0 with no binary produced. Declaring the
// name phony is precisely what converts the "No rule to make target"
// error into a green.
//
// The guard is a parse of the Makefile rather than an invocation of
// make. `make test` is what runs these tests, so shelling back into
// `make build` here would nest a build inside the test run and drop a
// binary into the tree as a side effect of testing. The one property a
// parse cannot establish — that the recipe still fails when the build
// fails — is not testable from inside the build either; it is verified
// by hand against a deliberately broken tree.
// phonyDirective introduces the list of phony target names.
const phonyDirective = ".PHONY:"
// ruleLine matches a rule's target list: a target starts in column
// zero, so recipe lines (tab-indented) and the continuation lines of a
// variable assignment (space-indented) are excluded by construction.
//
// The trailing (?:[^=]|$) rejects `:=` assignments such as
// `VERSION := $(shell script/version)`, which are not rules. Directives
// and function calls (`.PHONY:`, `ifeq`, `$(error ...)`) do not match
// because a target here must begin with a letter, digit or underscore.
var ruleLine = regexp.MustCompile(`^([A-Za-z0-9_][A-Za-z0-9_./ -]*):(?:[^=]|$)`)
// TestPhonyTargetsAllHaveRules fails on any name in .PHONY that has no
// rule in the Makefile. Such a name is not a build target at all: it is
// a command that reports success without doing anything, which is worse
// than one that does not exist, because a caller checking the exit code
// cannot tell the difference.
func TestPhonyTargetsAllHaveRules(t *testing.T) {
t.Parallel()
makefile := readMakefile(t)
phony := phonyTargets(makefile)
require.NotEmpty(t, phony, "no .PHONY names found; the parser is broken")
rules := declaredRules(makefile)
// Sanity check on the rule parser before trusting its verdict: a
// parser that found nothing would pass this test by accident.
require.Contains(t, rules, "vaultik",
"the file rule that builds the binary must be recognized")
for _, target := range phony {
assert.Contains(t, rules, target,
"`.PHONY` lists %q but the Makefile declares no %q rule, so "+
"`make %s` exits 0 without doing anything", target, target, target)
}
}
// TestBuildTargetBuildsTheBinary pins the specific shape of issue #110:
// `make build` has to reach the rule that produces the binary. The test
// above would also pass if `build:` were given an empty recipe of its
// own, which would be the same silent success under a different
// spelling.
func TestBuildTargetBuildsTheBinary(t *testing.T) {
t.Parallel()
prerequisites := rulePrerequisites(readMakefile(t), "build")
require.NotNil(t, prerequisites, "the Makefile declares no `build` rule")
assert.Contains(t, prerequisites, "vaultik",
"`make build` must depend on the rule that builds the binary")
}
// readMakefile returns the contents of the repository's Makefile.
func readMakefile(t *testing.T) string {
t.Helper()
return readRepoFile(t, "Makefile")
}
// readRepoFile reads a file by its path relative to the repository
// root.
func readRepoFile(t *testing.T, name string) string {
t.Helper()
//nolint:gosec // G304: the path is a constant relative to this repo
contents, err := os.ReadFile(filepath.Join(repoRoot(t), name))
require.NoError(t, err)
return string(contents)
}
// repoRoot returns the repository root. The test binary runs with its
// package directory as the working directory, so the root is found by
// walking up until the module file appears.
func repoRoot(t *testing.T) string {
t.Helper()
dir, err := os.Getwd()
require.NoError(t, err)
for {
_, err = os.Stat(filepath.Join(dir, "go.mod"))
if err == nil {
return dir
}
parent := filepath.Dir(dir)
require.NotEqual(t, dir, parent,
"walked to the filesystem root without finding a go.mod")
dir = parent
}
}
// phonyTargets returns every name declared phony, across all .PHONY
// lines.
func phonyTargets(makefile string) []string {
var targets []string
for line := range strings.SplitSeq(makefile, "\n") {
if !strings.HasPrefix(line, phonyDirective) {
continue
}
targets = append(targets,
strings.Fields(strings.TrimPrefix(line, phonyDirective))...)
}
return targets
}
// declaredRules returns the set of target names that have a rule.
func declaredRules(makefile string) map[string]bool {
rules := make(map[string]bool)
for line := range strings.SplitSeq(makefile, "\n") {
match := ruleLine.FindStringSubmatch(line)
if match == nil {
continue
}
// One rule may name several targets: `a b: prereq`.
for target := range strings.FieldsSeq(match[1]) {
rules[target] = true
}
}
return rules
}
// rulePrerequisites returns the prerequisites of the named rule, or nil
// if no such rule exists. A rule with none returns an empty slice, so
// "declared with nothing to do" is distinguishable from "not declared".
func rulePrerequisites(makefile, target string) []string {
for line := range strings.SplitSeq(makefile, "\n") {
match := ruleLine.FindStringSubmatch(line)
if match == nil {
continue
}
if !slices.Contains(strings.Fields(match[1]), target) {
continue
}
_, after, _ := strings.Cut(line, ":")
return append([]string{}, strings.Fields(after)...)
}
return nil
}
config.example.yml
+16 -69
View File
@@ -1,31 +1,15 @@
# vaultik configuration file example
# This file shows every configuration option. The values are examples, not
# defaults; where an option has a default, its comment gives it.
# Copy this file and change the values you need
# This file shows all available configuration options with their default values
# Copy this file and uncomment/modify the values you need
# Age recipient public keys for encryption
# Backups are encrypted to these public keys. snapshot create needs at least
# one; listing, verifying and restoring do not
# Generate a key pair, and show its public key, with:
# age-keygen -o vaultik_backup_private_key.txt
# grep 'public key' vaultik_backup_private_key.txt
# Keep the private key file: restoring needs it. Do not pipe age-keygen
# straight into grep, which would discard the private key
# This is REQUIRED - backups are encrypted to these public keys
# Generate with: age-keygen | grep "public key"
age_recipients:
- age1cj2k2addawy294f6k2gr2mf9gps9r3syplryxca3nvxj3daqm96qfp84tz
# Age private key for decryption
# Only snapshot restore and snapshot verify --deep need it. The
# VAULTIK_AGE_SECRET_KEY environment variable overrides this setting
# Do not set it on a machine you back up: prefer the environment variable,
# supplied only on the machine you restore on
# Default: not set
#age_secret_key: YOUR_AGE_PRIVATE_KEY
# Named snapshots - each snapshot can contain multiple paths
# Each snapshot gets its own ID and can have snapshot-specific excludes
# Required: at least one snapshot with at least one path. Only the config
# commands, version and completion run without one
snapshots:
testing:
paths:
@@ -273,21 +257,16 @@ exclude:
# Storage URL - use either this OR the s3 section below
# Supports: s3://bucket/prefix, file:///path, rclone://remote/path
# An s3:// URL also needs s3.access_key_id and s3.secret_access_key below
# A file:// path must be absolute, and the directory must already exist:
# vaultik never creates it
# An rclone:// URL needs the remote to be defined in rclone's own config
# (rclone config)
storage_url: "rclone://myremote/path/to/backups"
storage_url: "rclone://las1stor1//srv/pool.2024.04/backups/heraklion"
# S3-compatible storage configuration
#s3:
# # S3-compatible endpoint URL
# # Examples: https://s3.amazonaws.com, https://storage.googleapis.com
# endpoint: https://s3.example.com
# endpoint: http://10.100.205.122:8333
#
# # Bucket name where backups will be stored
# bucket: mybucket
# bucket: testbucket
#
# # Prefix (folder) within the bucket for this host's backups
# # Useful for organizing backups from multiple hosts
@@ -295,54 +274,36 @@ storage_url: "rclone://myremote/path/to/backups"
# #prefix: "hosts/myserver/"
#
# # S3 access credentials
# access_key_id: YOUR_ACCESS_KEY
# secret_access_key: YOUR_SECRET_KEY
# access_key_id: Z9GT22M9YFU08WRMC5D4
# secret_access_key: Pi0tPKjFbN4rZlRhcA4zBtEkib04yy2WcIzI+AXk
#
# # S3 region
# # Default: us-east-1
# #region: us-east-1
#
# # Use SSL/TLS for S3 connections
# # Only read when storage_url is not set and the endpoint has no scheme;
# # with an s3:// storage_url, use ?ssl=false in the URL instead
# # Default: true
# #use_ssl: true
#
# # Part size for multipart uploads
# # Minimum 5MiB, maximum 5GiB; affects memory usage during upload
# # A blob too large for 10,000 parts of this size gets larger parts
# # Supports: 10MB, 16MiB, 100MiB, etc. (5MB is below the minimum)
# # Default: 5MiB
# #part_size: 5MiB
# # Minimum 5MB, affects memory usage during upload
# # Supports: 5MB, 10M, 100MiB, etc.
# # Default: 5MB
# #part_size: 5MB
# Path to local SQLite index database
# This database tracks file state for incremental backups
# The VAULTIK_INDEX_PATH environment variable overrides this setting
# Default: the platform data directory, e.g.
# macOS: ~/Library/Application Support/vaultik/index.sqlite
# Linux: ~/.local/share/vaultik/index.sqlite
#index_path: /path/to/index.sqlite
# Default: /var/lib/vaultik/index.sqlite
#index_path: /var/lib/vaultik/index.sqlite
# Average chunk size for content-defined chunking
# Smaller chunks = better deduplication but more metadata
# Minimum 1MB
# Supports: 10MB, 5M, 1GB, 64MiB, etc.
# Supports: 10MB, 5M, 1GB, 500KB, 64MiB, etc.
# Default: 10MB
#chunk_size: 10MB
# Maximum blob size
# Multiple chunks are packed into blobs up to this size
# Must be at least four times chunk_size (the largest chunk the chunker can
# emit); a smaller limit would let a single-chunk blob exceed it.
# Chunking uses no secret (the FastCDC parameters are fixed and public). At a
# large limit a blob holds hundreds of chunks, so individual chunk lengths are
# not visible in its size; lowering the limit toward chunk_size exposes them.
# A backup needs free temporary space, because each blob is written whole to
# a temporary file before it is uploaded (up to about blob_size_limit; an
# rclone destination that cannot stream uploads needs about twice that) and
# the metadata export writes copies of the local index. Temporary files go to
# $TMPDIR (default /tmp); with TMPDIR unset, SQLite writes one of those copies
# to /var/tmp.
# Supports: 1GB, 10G, 500MB, 1GiB, etc.
# Default: 10GB
#blob_size_limit: 10GB
@@ -351,20 +312,6 @@ storage_url: "rclone://myremote/path/to/backups"
# Higher = better compression but slower
# Default: 3
compression_level: 5
# How old a blob that no snapshot lists must be before prune deletes it, and
# how old every object in a directory under metadata/ with no manifest must
# be before prune deletes that directory
# A backup uploads its blobs and its db.zst.age before its manifest, so on a
# destination shared between hosts or users this must be longer than the
# longest backup run
# It does not protect a blob a backup reuses from an earlier run that stopped
# before its manifest: that blob keeps the time of its first upload, so a
# prune can delete it
# Supports: 36h, 90m, etc.; 0s deletes such blobs and directories at once
# Default: 24h
#prune_grace_period: 24h
# Hostname to use in backup metadata
# Default: system hostname
#hostname: myserver
docs/DATAMODEL.md
+248 -393
View File
@@ -1,418 +1,273 @@
# vaultik data model
# Vaultik Data Model
`vaultik` keeps a SQLite database on the backed-up machine, called the local
index. This document describes its tables and how each command uses them.
## Overview
Vaultik uses a local SQLite database to track file metadata, chunk mappings, and blob associations during the backup process. This database serves as an index for incremental backups and enables efficient deduplication.
**Important Notes:**
- **No Migration Support (pre-1.0)**: Vaultik does not support database schema
migrations. The local index is treated as disposable — if the schema changes,
delete the local SQLite database (`vaultik database delete`) and run a full
backup. The remote storage is unaffected; the new index will re-deduplicate
against existing remote blobs.
- **Version Compatibility**: In rare cases, you may need to use the same version
of Vaultik to restore a backup as was used to create it. This ensures
compatibility with the metadata format stored in S3.
## Database Tables
### 1. `files`
Stores metadata about files in the filesystem being backed up.
**Columns:**
- `id` (TEXT PRIMARY KEY) - UUID for the file record
- `path` (TEXT NOT NULL UNIQUE) - Absolute file path
- `mtime` (INTEGER NOT NULL) - Modification time as Unix timestamp
- `size` (INTEGER NOT NULL) - File size in bytes
- `mode` (INTEGER NOT NULL) - Unix file permissions and type
- `uid` (INTEGER NOT NULL) - User ID of file owner
- `gid` (INTEGER NOT NULL) - Group ID of file owner
- `link_target` (TEXT) - Symlink target path (NULL for regular files)
**Indexes:**
- `idx_files_path` on `path` for efficient lookups
**Purpose:** Tracks file metadata to detect changes between backup runs. Used for incremental backup decisions. The UUID primary key provides stable references that don't change if files are moved.
### 2. `chunks`
Stores information about content-defined chunks created from files.
**Columns:**
- `chunk_hash` (TEXT PRIMARY KEY) - SHA256 hash of chunk content
- `size` (INTEGER NOT NULL) - Chunk size in bytes
**Purpose:** Enables deduplication by tracking unique chunks across all files.
### 3. `file_chunks`
Maps files to their constituent chunks in order.
**Columns:**
- `file_id` (TEXT) - File ID (FK to files.id)
- `idx` (INTEGER) - Chunk index within file (0-based)
- `chunk_hash` (TEXT) - Chunk hash (FK to chunks.chunk_hash)
- PRIMARY KEY (`file_id`, `idx`)
The local index has two jobs:
**Purpose:** Allows reconstruction of files from chunks during restore.
### 4. `chunk_files`
Reverse mapping showing which files contain each chunk.
- It records which files, chunks and blobs are already stored. The next backup
then does not store them again.
- It is the source of each snapshot's own database. A backup uploads a trimmed,
encrypted copy of the index with every snapshot.
**Columns:**
- `chunk_hash` (TEXT) - Chunk hash (FK to chunks.chunk_hash)
- `file_id` (TEXT) - File ID (FK to files.id)
- `file_offset` (INTEGER) - Byte offset of chunk within file
- `length` (INTEGER) - Length of chunk in bytes
- PRIMARY KEY (`chunk_hash`, `file_id`)
**Purpose:** Supports efficient queries for chunk usage and deduplication statistics.
### 5. `blobs`
Stores information about packed, compressed, and encrypted blob files.
**Columns:**
- `id` (TEXT PRIMARY KEY) - UUID assigned when blob creation starts
- `blob_hash` (TEXT UNIQUE) - SHA256 hash of final blob (NULL until finalized)
- `created_ts` (INTEGER NOT NULL) - Creation timestamp
- `finished_ts` (INTEGER) - Finalization timestamp (NULL if in progress)
- `uncompressed_size` (INTEGER NOT NULL DEFAULT 0) - Total size of chunks before compression
- `compressed_size` (INTEGER NOT NULL DEFAULT 0) - Size after compression and encryption
- `uploaded_ts` (INTEGER) - Upload completion timestamp (NULL if not uploaded)
**Purpose:** Tracks blob lifecycle from creation through upload. The UUID primary key allows immediate association of chunks with blobs.
### 6. `blob_chunks`
Maps chunks to the blobs that contain them.
**Columns:**
- `blob_id` (TEXT) - Blob ID (FK to blobs.id)
- `chunk_hash` (TEXT) - Chunk hash (FK to chunks.chunk_hash)
- `offset` (INTEGER) - Byte offset of chunk within blob (before compression)
- `length` (INTEGER) - Length of chunk in bytes
- PRIMARY KEY (`blob_id`, `chunk_hash`)
**Purpose:** Enables chunk retrieval from blobs during restore operations.
### 7. `snapshots`
Tracks backup snapshots.
**Columns:**
- `id` (TEXT PRIMARY KEY) - Snapshot ID (format: hostname-YYYYMMDD-HHMMSSZ)
- `hostname` (TEXT) - Hostname where backup was created
- `vaultik_version` (TEXT) - Version of Vaultik used
- `vaultik_git_revision` (TEXT) - Git revision of Vaultik used
- `started_at` (INTEGER) - Start timestamp
- `completed_at` (INTEGER) - Completion timestamp (NULL if in progress)
- `file_count` (INTEGER) - Number of files in snapshot
- `chunk_count` (INTEGER) - Number of unique chunks
- `blob_count` (INTEGER) - Number of blobs referenced
- `total_size` (INTEGER) - Total size of all files
- `blob_size` (INTEGER) - Total size of all blobs (compressed)
- `blob_uncompressed_size` (INTEGER) - Total uncompressed size of all referenced blobs
- `compression_ratio` (REAL) - Compression ratio achieved
- `compression_level` (INTEGER) - Compression level used for this snapshot
- `upload_bytes` (INTEGER) - Total bytes uploaded during this snapshot
- `upload_duration_ms` (INTEGER) - Total milliseconds spent uploading to S3
A restore does not need the local index. It reads the copy that was uploaded
with the snapshot.
**Purpose:** Provides snapshot metadata and statistics including version tracking for compatibility.
### 8. `snapshot_files`
Maps snapshots to the files they contain.
**Columns:**
- `snapshot_id` (TEXT) - Snapshot ID (FK to snapshots.id)
- `file_id` (TEXT) - File ID (FK to files.id)
- PRIMARY KEY (`snapshot_id`, `file_id`)
## Terms
**Purpose:** Records which files are included in each snapshot.
- **Snapshot**: one backup of the paths listed under one name in the config.
- **Chunk**: a piece of a file's content, named by the SHA-256 hash of that
content.
- **Blob**: one object at the destination. It holds many chunks, compressed and
then encrypted.
- **Destination**: the place the backups are stored.
- **Manifest**: the unencrypted list of the blobs one snapshot uses.
- **Remote key**: a hash of the snapshot ID. It names the snapshot's directory
at the destination.
### 9. `snapshot_blobs`
Maps snapshots to the blobs they reference.
## Where the index is
**Columns:**
- `snapshot_id` (TEXT) - Snapshot ID (FK to snapshots.id)
- `blob_id` (TEXT) - Blob ID (FK to blobs.id)
- `blob_hash` (TEXT) - Denormalized blob hash for manifest generation
- PRIMARY KEY (`snapshot_id`, `blob_id`)
**Purpose:** Tracks blob dependencies for snapshots and enables manifest generation.
### 10. `uploads`
Tracks blob upload metrics.
**Columns:**
- `blob_hash` (TEXT PRIMARY KEY) - Hash of uploaded blob
- `snapshot_id` (TEXT NOT NULL) - The snapshot that triggered this upload (FK to snapshots.id)
- `uploaded_at` (INTEGER) - Upload timestamp
- `size` (INTEGER) - Size of uploaded blob
- `duration_ms` (INTEGER) - Upload duration in milliseconds
**Purpose:** Performance monitoring and tracking which blobs were newly created (uploaded) during each snapshot.
## Data Flow and Operations
### 1. Backup Process
1. **File Scanning**
- `INSERT OR REPLACE INTO files` - Update file metadata
- `SELECT * FROM files WHERE path = ?` - Check if file has changed
- `INSERT INTO snapshot_files` - Add file to current snapshot
2. **Chunking** (for changed files)
- `INSERT OR IGNORE INTO chunks` - Store new chunks
- `INSERT INTO file_chunks` - Map chunks to file
- `INSERT INTO chunk_files` - Create reverse mapping
3. **Blob Packing**
- `INSERT INTO blobs` - Create blob record with UUID (blob_hash NULL)
- `INSERT INTO blob_chunks` - Associate chunks with blob immediately
- `UPDATE blobs SET blob_hash = ?, finished_ts = ?` - Finalize blob after packing
4. **Upload**
- `UPDATE blobs SET uploaded_ts = ?` - Mark blob as uploaded
- `INSERT INTO uploads` - Record upload metrics with snapshot_id
- `INSERT INTO snapshot_blobs` - Associate blob with snapshot
5. **Snapshot Completion**
- `UPDATE snapshots SET completed_at = ?, stats...` - Finalize snapshot
- Generate and upload blob manifest from `snapshot_blobs`
### 2. Incremental Backup
1. **Change Detection**
- `SELECT * FROM files WHERE path = ?` - Get previous file metadata
- Compare mtime, size, mode to detect changes
- Skip unchanged files but still add to `snapshot_files`
2. **Chunk Reuse**
- `SELECT * FROM blob_chunks WHERE chunk_hash = ?` - Find existing chunks
- `INSERT INTO snapshot_blobs` - Reference existing blobs for unchanged files
### 3. Snapshot Metadata Export
After a snapshot is completed:
1. Copy database to temporary file
2. Clean temporary database to contain only current snapshot data
3. Export to SQL dump using sqlite3
4. Compress with zstd and encrypt with age
5. Upload to S3 as `metadata/{snapshot-id}/db.zst.age`
6. Generate blob manifest and upload as `metadata/{snapshot-id}/manifest.json.zst`
### 4. Restore Process
The restore process doesn't use the local database. Instead:
1. Downloads snapshot metadata from S3
2. Downloads required blobs based on manifest
3. Reconstructs files from decrypted and decompressed chunks
The index is the file `index.sqlite` in the platform data directory. That is
`~/.local/share/vaultik/` on Linux and `~/Library/Application Support/vaultik/`
on macOS. The `index_path` setting and the `VAULTIK_INDEX_PATH` environment
variable change the path.
### 5. Pruning
The file is readable only by its owner, because it lists every backed-up path.
`vaultik` sets that mode each time it opens the index.
1. **Identify Unreferenced Blobs**
- Query blobs not referenced by any remaining snapshot
- Delete from S3 and local database
## Schema migrations
### 6. Incomplete Snapshot Cleanup
`vaultik` supports schema migrations. They are the numbered SQL files in
`internal/database/schema/`, which are built into the binary.
Before each backup:
1. Query incomplete snapshots (where `completed_at IS NULL`)
2. Check if metadata exists in S3
3. If no metadata, delete snapshot and all associations
4. Clean up orphaned files, chunks, and blobs
- `000.sql` creates the `schema_migrations` table. That table records each
migration that has run.
- `001.sql` creates every other table.
## Repository Pattern
`database.New` opens the index. It then applies, in order, every migration that
the index has not recorded.
Vaultik uses a repository pattern for database access:
Before version 1.0, no migration is added, because nothing is installed anywhere
yet. A schema change edits `001.sql` directly, with the code that uses the
changed tables.
- `FileRepository` - CRUD operations for files and file metadata
- `ChunkRepository` - CRUD operations for content chunks
- `FileChunkRepository` - Manage file-to-chunk mappings
- `ChunkFileRepository` - Manage chunk-to-file reverse mappings
- `BlobRepository` - Manage blob lifecycle (creation, finalization, upload)
- `BlobChunkRepository` - Manage blob-to-chunk associations
- `SnapshotRepository` - Manage snapshots and their relationships
- `UploadRepository` - Track blob upload metrics
This has one consequence. An index created before such a change has already
recorded `001.sql` as run, so it keeps the old schema and can become unusable.
To rebuild it, run `vaultik database delete` and then a full backup.
Each repository provides methods like:
- `Create()` - Insert new record
- `GetByID()` / `GetByPath()` / `GetByHash()` - Retrieve records
- `Update()` - Update existing records
- `Delete()` - Remove records
- Specialized queries for each entity type (e.g., `DeleteOrphaned()`, `GetIncompleteByHostname()`)
After version 1.0, each schema change is a new numbered file. An existing index
is then migrated the first time an updated `vaultik` opens it. A file that was
part of a release is never edited.
## Transaction Management
Migrations apply to the local index only. A snapshot's database at the
destination is read as it was written. Each snapshot records the `vaultik`
version that wrote it.
## Tables
Times are whole seconds since the Unix epoch, in UTC. The two exceptions are
noted at their columns.
### `files`
One row for each backed-up path: a regular file, a symlink or a directory.
- `id`: a UUID. Primary key.
- `path`: the absolute path. Unique.
- `source_path`: the configured path that the scan found this entry under. A
restore does not read it.
- `mtime`: the modification time.
- `mtime_nsec`: the nanoseconds within that second.
- `size`: the size in bytes.
- `mode`: the permission bits and the file type.
- `uid`: the owner's user ID.
- `gid`: the owner's group ID.
- `link_target`: the target of a symlink. It is `NULL` for other entries.
`idx_files_path` indexes `path`.
There is one row for each path, and it is updated when the file changes. The
row is shared by every snapshot in the local index that lists the path. An
older snapshot's own view of the file is in that snapshot's database at the
destination.
### `chunks`
One row for each distinct chunk.
- `chunk_hash`: the SHA-256 hash of the chunk's content, in hex. Primary key.
- `size`: the size in bytes.
### `file_chunks`
The chunks that make up each file, in order.
- `file_id`: the file. References `files.id`.
- `idx`: the position of the chunk within the file, counted from 0.
- `chunk_hash`: the chunk. References `chunks.chunk_hash`.
The primary key is (`file_id`, `idx`). `idx_file_chunks_chunk_hash` indexes
`chunk_hash`.
### `chunk_files`
The reverse of `file_chunks`: the files that contain each chunk.
- `chunk_hash`: the chunk. References `chunks.chunk_hash`.
- `file_id`: the file. References `files.id`.
- `file_offset`: the byte offset of the chunk within the file.
- `length`: the length of the chunk in bytes.
The primary key is (`chunk_hash`, `file_id`). `idx_chunk_files_file_id` indexes
`file_id`.
### `blobs`
One row for each blob.
- `id`: a UUID, given when packing of the blob starts. Primary key.
- `blob_hash`: the name of the blob at the destination. Unique. Until the blob
is closed, it holds a placeholder that starts with `temp-placeholder-`.
- `created_ts`: when packing started.
- `finished_ts`: when the blob was closed. It is `NULL` before that.
- `uncompressed_size`: the total size of the blob's chunks.
- `compressed_size`: the size after compression and encryption.
- `uploaded_ts`: when the upload finished. It is `NULL` before that.
`blob_hash` is `hex(SHA256(SHA256(uncompressed contents)))`. See
[REPOSTRUCTURE.md](REPOSTRUCTURE.md#blobs).
Once a blob is uploaded, `compressed_size` is the size of the object at the
destination. A blob that was already there can have a different size from the
local copy. That happens when it was written with another compression level or
other recipients.
### `blob_chunks`
The chunks that each blob holds.
- `blob_id`: the blob. References `blobs.id`.
- `chunk_hash`: the chunk. References `chunks.chunk_hash`.
- `offset`: the byte offset of the chunk within the uncompressed blob.
- `length`: the length of the chunk in bytes.
The primary key is (`blob_id`, `chunk_hash`). `idx_blob_chunks_chunk_hash`
indexes `chunk_hash`.
A restore finds the blob for each chunk through this table.
### `snapshots`
One row for each snapshot.
- `id`: the snapshot ID. Primary key.
- `hostname`: the hostname of the machine that made the backup.
- `vaultik_version`: the `vaultik` version that made it.
- `vaultik_git_revision`: the git commit of that version.
- `started_at`: when the backup started.
- `completed_at`: when the backup finished. It is `NULL` until then.
- `file_count`: the number of files and symlinks in the snapshot. Directories
are not counted.
- `chunk_count`: the number of chunks this snapshot stored that were not stored
before.
- `new_chunk_size`: the total size of those chunks before compression.
- `blob_count`: the number of blobs this snapshot created.
- `total_size`: the total size of all files in the snapshot.
- `uncompressed_chunk_size`: the total size, before compression, of the
distinct chunks the snapshot's files use.
- `blob_size`: the total compressed size of the blobs the snapshot uses.
- `blob_uncompressed_size`: the total uncompressed size of those blobs.
- `compression_ratio`: the compression ratio of those blobs.
- `compression_level`: the compression level the backup used.
- `upload_bytes`: the bytes uploaded during the backup.
- `upload_duration_ms`: the time spent uploading, in milliseconds.
The snapshot ID has the form `<hostname>_<snapshot-name>_<timestamp>`. An
example is `server1_home_2025-06-01T12:00:00Z`. It has three parts:
- the hostname, up to its first `.`
- the snapshot name
- the start time, in UTC
`snapshot list` shows `uncompressed_chunk_size` as `UNCOMPRESSED SIZE` and
`new_chunk_size` as `NEW CHUNK SIZE`.
A snapshot with `completed_at` set to `NULL` is unfinished. `snapshot list`
does not show it. The next backup deletes it, and so does `prune`.
### `snapshot_files`
The files in each snapshot.
- `snapshot_id`: the snapshot. References `snapshots.id`.
- `file_id`: the file. References `files.id`.
The primary key is (`snapshot_id`, `file_id`). `idx_snapshot_files_file_id`
indexes `file_id`.
### `snapshot_blobs`
The blobs that each snapshot uses.
- `snapshot_id`: the snapshot. References `snapshots.id`.
- `blob_id`: the blob. References `blobs.id`.
- `blob_hash`: a copy of the blob's hash.
The primary key is (`snapshot_id`, `blob_id`). `idx_snapshot_blobs_blob_id`
indexes `blob_id`.
The manifest is built from this table.
### `snapshot_mount_points`
The mount points that a snapshot's backup found.
- `snapshot_id`: the snapshot. References `snapshots.id`.
- `path`: the absolute path of the directory.
The primary key is (`snapshot_id`, `path`).
A directory is recorded here when all of these hold:
- It is at or below a configured path.
- The backup found an entry in it that is not excluded.
- It is on a different filesystem from its parent directory.
The device number is read after the directory has been listed. Listing an
automount point mounts its volume, so the number is that of the volume. An
automount point whose volume cannot be mounted is not recorded, because the
backup finds no entry in it.
`snapshot create --prune` reads this table. It skips the prune when an earlier
snapshot of the same name recorded a mount point that the new backup did not
find. That happens when the volume is not mounted.
### `uploads`
One row for each blob that a backup uploaded.
- `blob_hash`: the blob. References `blobs.blob_hash`. Primary key.
- `snapshot_id`: the snapshot whose backup uploaded it. References
`snapshots.id`.
- `uploaded_at`: when the upload started. It is written as a date and time, not
as seconds.
- `size`: the size of the uploaded blob.
- `duration_ms`: the duration of the upload, in milliseconds.
`idx_uploads_snapshot_id` indexes `snapshot_id`.
A blob that was already at the destination gets no row here.
### `local_meta`
Settings that belong to this copy of the index.
- `key`: the name of the setting. Primary key.
- `value`: its value.
The only key is `storage_url`. It holds the destination that the index was
first used with. A command that changes data refuses to run when the config
names a different destination.
### `schema_migrations`
The migrations that have run.
- `version`: the number of the migration. Primary key.
- `applied_at`: when it ran. It is written as a date and time, not as seconds.
## How commands use the tables
### Backup
A backup works on the tables in this order:
- It deletes unfinished snapshots, and the rows that no snapshot uses.
- It creates a row in `snapshots`, with `completed_at` set to `NULL`.
- It deletes the `blobs` rows that were never uploaded.
- It walks each path and compares every entry with its row in `files`.
- It links each unchanged file to the snapshot in `snapshot_files`.
- It records each mount point in `snapshot_mount_points`.
- It reads each changed file and stores it. See below.
- It fills `snapshot_blobs` with every blob the snapshot uses.
- It stores the snapshot's totals in `snapshots`.
- It uploads the snapshot's metadata.
- It sets `completed_at`.
A file counts as changed when its row in `files` differs from the file on disk
in any of these:
- size
- modification time
- mode
- owner or group
- link target
A file is also read again when it lists a chunk that no uploaded blob holds.
Storing a changed file writes these rows:
- A row in `blobs`, when packing of a blob starts.
- When the blob is closed, in one transaction:
- a row in `chunks` for each new chunk in the blob
- the blob's rows in `blob_chunks`
- the blob's `blob_hash`, sizes and `finished_ts`
- When the blob is stored at the destination, in one transaction:
- `uploaded_ts` and `compressed_size` in `blobs`
- a row in `snapshot_blobs`
- a row in `uploads`, unless the blob was already at the destination
- The row in `files`, with its `file_chunks`, `chunk_files` and
`snapshot_files` rows.
The row in `files` is written last. It is written only after every blob that
holds the file's new chunks is stored.
### Snapshot metadata
The metadata of a snapshot is made from a copy of the index:
- The index is copied into a temporary directory with `VACUUM INTO`.
- The rows of every other snapshot are deleted from the copy. So are the files
and blobs that this snapshot does not use. A chunk is then deleted when no
remaining file and no remaining blob uses it.
- The `local_meta` and `schema_migrations` tables are kept as they are. The
uploaded database therefore names the destination, and lists the migrations
that have run.
- `VACUUM` rewrites the copy, so that no deleted row is left in the file.
- The copy is compressed with zstd and encrypted with age.
- The manifest is built from the copy's `snapshot_blobs` table.
- The copy is uploaded as `metadata/<remote-key>/db.zst.age`.
- The manifest is uploaded as `metadata/<remote-key>/manifest.json.zst`.
The copy is made through SQLite, not by copying the file. The index runs in WAL
mode, so rows that are already committed can still be in the `-wal` file beside
it.
### Restore
A restore does not use the local index.
- It downloads and decrypts `db.zst.age` into a temporary directory.
- It opens that file read-only. It applies no migration to it.
- It reads the files to restore from `files`.
- It finds each file's chunks in `file_chunks`.
- It finds the blob and offset of each chunk in `blob_chunks`.
- It finds the name of each blob in `blobs`.
It does not read the manifest.
### Prune
`prune` changes the local index in two ways:
- It deletes each snapshot whose manifest is gone from the destination.
- It deletes unfinished snapshots, and the rows that no snapshot uses.
The second step removes unused rows in this order:
- files that are in no snapshot
- blobs that are in no snapshot
- `blob_chunks` rows whose chunk is gone
- chunks that no file and no blob uses
`prune` then deletes unused data at the destination. That step reads the
manifests, not the local index. It reads the local index only to keep a
directory under `metadata/` whose snapshot is still recorded there.
## Code
Each repository type in `internal/database` serves one or more tables:
- `FileRepository`: `files`
- `ChunkRepository`: `chunks`
- `FileChunkRepository`: `file_chunks`
- `ChunkFileRepository`: `chunk_files`
- `BlobRepository`: `blobs`
- `BlobChunkRepository`: `blob_chunks`
- `SnapshotRepository`:
- `snapshots`
- `snapshot_files`
- `snapshot_blobs`
- `snapshot_mount_points`
- `UploadRepository`: `uploads`
- `LocalMetaRepository`: `local_meta`
No repository type serves `schema_migrations`. The code that applies the
migrations reads and writes it.
`Repositories` holds one of each type.
`Repositories.WithTx` runs a function in one transaction:
All database operations that modify multiple tables are wrapped in transactions:
```go
err := repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
// several repository calls that use tx
// Multiple repository operations using tx
})
```
The transaction commits when the function returns no error. Otherwise it is
rolled back.
This ensures consistency, especially important for operations like:
- Creating file-chunk mappings
- Associating chunks with blobs
- Updating snapshot statistics
## SQLite settings
## Performance Considerations
- The index runs in WAL mode. A command that only reads, such as
`snapshot list`, can read the index while a backup writes to it.
- A connection waits up to 10 seconds for a lock before it fails.
- Foreign keys are enforced.
- Each open database uses one connection, so its queries run one at a time.
- Queries pass their values as parameters.
1. **Indexes**:
- Primary keys are automatically indexed
- `idx_files_path` on `files(path)` for efficient file lookups
2. **Prepared Statements**: All queries use prepared statements for performance and security
3. **Batch Operations**: Where possible, operations are batched within transactions
4. **Write-Ahead Logging**: SQLite WAL mode is enabled for better concurrency
## Data Integrity
1. **Foreign Keys**: Enforced through CASCADE DELETE and application-level repository methods
2. **Unique Constraints**: Chunk hashes, file paths, and blob hashes are unique
3. **Null Handling**: Nullable fields clearly indicate in-progress operations
4. **Timestamp Tracking**: All major operations record timestamps for auditing
docs/REPOSTRUCTURE.md
+114 -386
View File
@@ -1,416 +1,144 @@
# vaultik destination layout
# Vaultik S3 Repository Structure
This document describes what `vaultik` stores at the destination. It covers the
layout, what the layout protects, and the risks it accepts.
This document describes the structure and organization of data stored in the S3 bucket by Vaultik.
The destination is the place the backups are stored. It is an S3 bucket and
prefix, a local directory or an rclone remote. The layout is the same on all
three.
## Overview
## Terms
Vaultik stores all backup data in an S3-compatible object store. The repository consists of two main components:
1. **Blobs** - The actual backup data (content-addressed, encrypted)
2. **Metadata** - Snapshot information and manifests (partially encrypted)
- **Snapshot**: one backup of the paths listed under one name in the config.
- **Snapshot ID**: the name of a snapshot, in the form
`<hostname>_<snapshot-name>_<timestamp>`.
- **Chunk**: a piece of a file's content, named by the SHA-256 hash of that
content.
- **Blob**: one object under `blobs/`. It holds many chunks.
- **Local index**: the SQLite database on the backed-up machine.
- **Recipient**: an age public key that backups are encrypted to.
## Layout
## Directory Structure
```
<destination>/
├── vaultik-destination.txt
<bucket>/<prefix>/
├── blobs/
│ └── <hash[0:2]>/
│ └── <hash[2:4]>/
│ └── <hash>
│ └── <full-hash>
└── metadata/
└── <remote-key>/
└── <snapshot-id>/
├── db.zst.age
└── manifest.json.zst
```
The destination holds three things:
## Blobs Directory (`blobs/`)
- `vaultik-destination.txt`, the marker file
- `blobs/`, which holds the file contents
- `metadata/`, which holds one directory for each snapshot
### Structure
- **Path format**: `blobs/<first-2-chars>/<next-2-chars>/<full-hash>`
- **Example**: `blobs/ca/fe/cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678`
- **Sharding**: The two-level directory structure (using the first 4 characters of the hash) prevents any single directory from containing too many objects
## The marker file
### Content
- **What it contains**: Packed collections of content-defined chunks from files
- **Format**: Zstandard compressed, then Age encrypted
- **Encryption**: Always encrypted with Age using the configured recipients
- **Naming**: Content-addressed using SHA256 hash of the encrypted blob
Every backup writes `vaultik-destination.txt` before it uploads anything else.
The file holds one fixed line of text. It names no host, path or time.
### Why Encrypted
Blobs contain the actual file data from backups and must be encrypted for security. The content-addressing ensures deduplication while the encryption ensures privacy.
The marker tells an emptied or missing destination apart from a new one. An
unplugged volume leaves an empty directory at its mount point. Without the
marker, that directory would look like a destination whose snapshots are all
gone.
## Metadata Directory (`metadata/`)
The check applies while the local index holds snapshots. When the destination
then answers that the marker does not exist, commands treat the destination as
not present:
Each snapshot has its own subdirectory named with the snapshot ID.
- `snapshot create` refuses to write to it.
- `prune`, `snapshot purge` and `remote nuke` fail.
- `snapshot list` shows the local index alone.
- `snapshot remove` deletes nothing at the destination.
### Snapshot ID Format
- **Format**: `<hostname>_<snapshot-name>_<RFC3339>` (or `<hostname>_<RFC3339>` if no
name was specified)
- **Example**: `laptop_home_2024-01-15T14:30:52Z`
- **Components**:
- Short hostname (everything before the first dot is stripped from the FQDN)
- Snapshot name from the configured `snapshots:` map (optional)
- RFC3339 UTC timestamp
The check is skipped when the look-up fails for any other reason. S3 refuses
the look-up to credentials that can only write. A machine with such credentials
therefore does not notice an emptied destination. Its next backup would not
upload data that its local index says is stored. After a destination is emptied,
run `vaultik database delete` on such a machine before it backs up again.
### Files in Each Snapshot Directory
`remote nuke` deletes the marker last.
## Blobs
A blob holds the content of many chunks, one after another.
- **Path**: `blobs/<first 2 characters>/<next 2 characters>/<full hash>`.
- **Example**:
`blobs/ca/fe/cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678`.
- **Format**: the chunk data, compressed with zstd and then encrypted with age.
- **Recipients**: the public keys in `age_recipients` when the blob was written.
The two directory levels keep any one directory from holding too many objects.
### Blob names
A blob's name is `hex(SHA256(SHA256(uncompressed contents)))`. The contents are
the chunk data, joined in order, before compression and encryption. The name is
not a hash of the stored bytes.
This has two consequences:
- Checking a blob against its name needs the private key. The blob must be
decrypted and decompressed first. `snapshot restore` and
`snapshot verify --deep` do this.
- Without the key, only a blob's presence and size can be checked.
`snapshot verify` does this.
`blobgen.DoubleSHA256` in `internal/blobgen` computes the name.
### Uploads
A backup looks a blob up by name before it uploads it. It does not upload a
blob that is already there.
When the destination refuses the look-up, the backup uploads the blob. An
object with that name is then replaced by one that holds the same chunk data.
## Metadata
Each snapshot has one directory under `metadata/`. The directory is named with
the snapshot's remote key, not with its snapshot ID.
### Snapshot IDs
A snapshot ID has the form `<hostname>_<snapshot-name>_<timestamp>`. An example
is `laptop_home_2024-01-15T14:30:52Z`. It has three parts:
- the hostname, up to its first `.`
- the snapshot name from the config
- the time the backup started, in UTC
The snapshot ID is used on the command line, in log records and in the local
index. It reveals the hostname, the snapshot name and the backup time. It is
therefore never written to the destination without encryption.
### Remote keys
The remote key is `hex(SHA256(SHA256("vaultik|" + snapshot ID)))`. It is 64 hex
characters long. `internal/snapshot/remotekey.go` computes it.
For example:
- Snapshot ID: `server1_home_2025-06-01T12:00:00Z`
- Remote key: `17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa`
- Directory:
`metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`
A listing of the destination therefore shows no hostname and no snapshot name.
The hash cannot be reversed, but it uses no secret. Someone who guesses a
snapshot ID can hash it the same way and see whether that snapshot exists. The
remote key keeps names out of a listing. It does not hide them from a guess.
### `db.zst.age`
The snapshot's database. A restore needs it.
- **Contents**: a SQLite database. It is a copy of the local index without the
rows of other snapshots. It lists every file in the snapshot with its
metadata, its chunks, and the blob that holds each chunk.
- **Format**: the database file, compressed with zstd and then encrypted with
age.
It is encrypted because it holds file paths, permissions and owners.
[DATAMODEL.md](DATAMODEL.md) describes its tables.
### `manifest.json.zst`
The list of the blobs the snapshot uses.
- **Format**: JSON, compressed with zstd. It is not encrypted.
#### `db.zst.age` - Encrypted Database
- **What it contains**: Pruned binary SQLite database for this snapshot
- **Format**: Binary SQLite → Zstandard compressed → Age encrypted
- **Encryption**: Encrypted with Age
- **Purpose**: Contains full file metadata, chunk mappings, and all relationships
- **Why encrypted**: Contains sensitive metadata like file paths, permissions, and ownership
#### `manifest.json.zst` - Unencrypted Blob Manifest
- **What it contains**: JSON list of all blob hashes referenced by this snapshot
- **Format**: JSON → Zstandard compressed (NOT encrypted)
- **Encryption**: NOT encrypted
- **Purpose**: Enables pruning operations without requiring decryption keys
- **Structure**:
```json
{
"snapshot_id": "17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa",
"timestamp": "2025-06-01T12:04:31Z",
"blob_count": 2,
"total_compressed_size": 57344,
"snapshot_id": "laptop_home_2024-01-15T14:30:52Z",
"timestamp": "2024-01-15T14:30:52Z",
"blob_count": 42,
"blobs": [
{
"hash": "cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678",
"compressed_size": 24576
},
{
"hash": "deadbeef1234567890abcdef1234567890abcdef1234567890abcdef12345678",
"compressed_size": 32768
}
"cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678",
"deadbeef1234567890abcdef1234567890abcdef1234567890abcdef12345678",
...
]
}
```
- `snapshot_id`: the remote key, not the snapshot ID.
- `timestamp`: when the manifest was written, at the end of the backup.
- `blob_count`: the number of blobs.
- `total_compressed_size`: the total size of those blobs, in bytes.
- `blobs`: the hash and the size of each blob.
The manifest holds no file name and no path.
The manifest is not encrypted so that these commands work without the private
key:
- `prune`, which must know which blobs are still in use
- `snapshot verify`, which checks that each blob is present
- `snapshot list` and `remote info`, which report what the destination holds
## What is protected
These are encrypted:
- file contents, in the blobs
- in `db.zst.age`:
- file names and paths
- permissions and owners
- modification times
- which chunks make up which file
These are not encrypted:
- the remote key of each snapshot
- the time each backup finished, in the manifest
- the hash and the compressed size of each blob, in the manifest
- the number of blobs and their total size for each snapshot, in the manifest
Someone who can read the destination therefore learns these:
- when each backup finished
- how many blobs each snapshot uses, and how large they are
- which blobs two snapshots share
- whether a snapshot ID they guess exists
They do not learn these:
- any hostname or snapshot name, unless they guess it
- file names or paths
- file contents
- permissions or owners
- the directory structure
- which chunks belong to which file
The times and sizes could not be hidden by encrypting the manifest. The
destination itself shows when each object was written and how large it is.
## Accepted risks
These are known properties of the format. They are written down so that you can
weigh them.
### No proof of who wrote the data
A restore and `snapshot verify --deep` prove two things. The data decrypts with
your private key, and it matches its hashes. They do not prove who wrote it.
The hashes use no secret. Someone who knows a recipient's public key and can
replace objects at the destination can put a snapshot of their own there. The
public key is not stored at the destination, but a backed-up machine that is
broken into has it.
The defences are on the destination's side:
- object versioning or object lock on the bucket
- credentials for the backed-up machine that cannot delete or overwrite
- running `prune` from a trusted machine
An S3 `PutObject` replaces an existing object. Permission to write alone
therefore does not stop overwrites.
### Sizes are visible
An object's size follows exactly from the size of its compressed contents.
All the new chunks in one blob share one zstd stream. The stream looks back 8
MiB, or 4 MiB at compression levels 1 and 2. A blob is closed at
`blob_size_limit` and at the end of each configured path. A chunk that is
already stored is never packed again.
Data that an outsider writes compresses smaller when it matches a secret in the
same chunk, so the blob gets smaller. The blob's size therefore shows the
outsider whether their data matched the secret.
This works only when all of these hold:
- The outsider can write into a backed-up file.
- The outsider can see the blob sizes at the destination.
- The outsider's data and the secret land in the same chunk.
- The file keeps changing.
Back up a directory that outsiders can write to as a snapshot of its own.
### Chunk boundaries use no secret
The FastCDC parameters are fixed and public. `chunk_size` sets the average chunk
size. No chunk is larger than four times this size. Only a file's last chunk can
be smaller than a quarter of it. With the default of 10 MB, those limits are 40
MB and 2.5 MB. A file of 2.5 MB or less is one chunk.
At the default `blob_size_limit` of 10 GB, a blob holds hundreds of chunks. The
size of the blob then does not show the size of any one chunk. A limit close to
`chunk_size` begins to show chunk sizes.
### Decrypted data on the local disk
These commands write unencrypted data to the temporary directory:
- `snapshot restore` writes decrypted blobs under `vaultik-blobcache-*/`. There
is no limit on their total size.
- `snapshot restore` writes the decrypted database to
`vaultik-restore-*/snapshot.db`.
- `snapshot verify --deep` writes the decrypted database to
`vaultik-verify-*/snapshot.db`.
- `snapshot create` writes a copy of the local index to
`vaultik-snapshot-*/snapshot.db`.
These files are readable only by their owner, and are removed when the command
ends. A `kill -9` or a power loss leaves them behind. Delete any leftover
`vaultik-*` entry in the temporary directory yourself.
Trust the temporary directory as much as the directory you restore into.
### Commands need different permissions
`snapshot create` needs only permission to write objects. Other commands need
more:
- `snapshot list`, `snapshot verify`, `snapshot restore` and `remote info` list
and read.
- `snapshot purge`, `snapshot remove` and `remote nuke` list and delete.
- `prune` lists, reads and deletes.
`snapshot create --prune` runs a prune on the backed-up machine. That machine
then holds credentials that can delete its own backups. To keep it to
write-only credentials, run `prune` from a separate, trusted machine.
### New recipients cannot read old data
Changing `age_recipients` does not re-encrypt what is already stored. A chunk
that is already stored is not written again, so it stays encrypted to the
earlier recipients.
A new snapshot that uses such chunks cannot be restored with a new recipient's
key alone. To make everything readable by a new key, run
`vaultik database delete` and take a full backup to a new destination.
### X25519 recipients only
`vaultik` accepts age X25519 public keys only. It rejects age SSH and plugin
recipients.
The encrypted data is held by a third party for a long time. There is no second
kind of recipient to fall back on if X25519 is broken, and no post-quantum
option.
## Order of writes
A backup writes in a fixed order:
- the marker file
- the blobs
- `db.zst.age`
- `manifest.json.zst`
A snapshot's metadata is written only after all its blobs are uploaded. The
manifest is written last. A directory under `metadata/` that has no manifest
therefore belongs to a backup that did not finish.
The local index marks a snapshot complete only after both metadata files are
uploaded. A backup that stops during that upload leaves the snapshot
unfinished. The next backup removes it from the local index and stores its data
again.
One short gap remains. A backup can stop after the upload and before the
snapshot is marked complete. The destination then holds a snapshot that can be
restored, and the local index does not list it. `snapshot list` shows it as
remote only.
An upload that is cut off leaves no partial object under the final name on S3
or on a local directory. The same holds for an rclone remote that can move
objects. An rclone remote that cannot move objects can keep a partial object.
## Pruning
`prune` deletes two kinds of object at the destination:
- a blob that no manifest lists
- a directory under `metadata/` that has no manifest
It reads the manifests, which are not encrypted, so it needs no private key.
These rules keep it from deleting data that is in use:
- It keeps every blob that any manifest lists.
- It deletes nothing when any manifest cannot be downloaded or decoded.
- It deletes a blob only once the blob is older than `prune_grace_period`.
- It deletes a directory only once every object in it is that old.
- It keeps a directory whose snapshot is in the local index.
- It leaves alone a directory whose name is not a remote key.
`prune_grace_period` defaults to `24h`. It protects a backup that is still
running. Such a backup has uploaded blobs and perhaps `db.zst.age`, but not yet
its manifest. A prune from this machine or another keeps those objects, as long
as no backup takes longer than the grace period.
The grace period does not protect one case. A backup does not upload a blob
that is already at the destination, so that blob keeps the time of its first
upload. The blob can be left over from an earlier backup that stopped before
its manifest. A prune that runs before the new manifest is written can then
delete that blob. The new snapshot cannot be restored afterwards.
To avoid this case, give each machine and each user its own destination.
`remote nuke` applies none of these rules.
## What a restore needs
- the age private key
- the snapshot's `db.zst.age`
- the blobs that hold the files to restore
A restore does not read the manifest and does not need the local index.
It works in these steps:
- It downloads and decrypts `db.zst.age`.
- It checks that the database is the one for the snapshot that was asked for.
- It downloads and decrypts the blobs it needs. It checks each blob against its
name.
- It writes each file from its chunks.
- It sets each file's permissions and times, and its owner when it runs as
root.
### Why Manifest is Unencrypted
The manifest must be readable without the private key to enable:
1. **Pruning operations** - Identifying unreferenced blobs for deletion
2. **Storage analysis** - Understanding space usage without decryption
3. **Verification** - Checking blob existence without decryption
4. **Cross-snapshot deduplication analysis** - Finding shared blobs between snapshots
The manifest only contains blob hashes, not file names or any other sensitive information.
## Security Considerations
### What's Encrypted
- **All file content** (in blobs)
- **All file metadata** (paths, permissions, timestamps, ownership in db.zst.age)
- **File-to-chunk mappings** (in db.zst.age)
### What's Not Encrypted
- **Blob hashes** (in manifest.json.zst)
- **Snapshot IDs** (directory names)
- **Blob count per snapshot** (in manifest.json.zst)
### Privacy Implications
From the unencrypted data, an observer can determine:
- When backups were taken (from snapshot IDs)
- Which hostname created backups (from snapshot IDs)
- How many blobs each snapshot references
- Which blobs are shared between snapshots (deduplication patterns)
- The size of each encrypted blob
An observer cannot determine:
- File names or paths
- File contents
- File permissions or ownership
- Directory structure
- Which chunks belong to which files
## Consistency Guarantees
1. **Blobs are immutable** - Once written, a blob is never modified
2. **Blobs are written before metadata** - A snapshot's metadata is only written after all its blobs are successfully uploaded
3. **Metadata is written atomically** - Both db.zst.age and manifest.json.zst are written as complete files
4. **Snapshots are marked complete in local DB only after metadata upload** - Ensures consistency between local and remote state
## Pruning Safety
The prune operation is safe because:
1. It only deletes blobs not referenced in any manifest
2. Manifests are unencrypted and can be read without keys
3. The operation compares the latest local DB snapshot with the latest S3 snapshot to ensure consistency
4. Pruning will fail if these don't match, preventing accidental deletion of needed blobs
## Restoration Requirements
To restore from a backup, you need:
1. **The Age private key** - To decrypt blobs and database
2. **The snapshot metadata** - Both files from the snapshot's metadata directory
3. **All referenced blobs** - As listed in the manifest
The restoration process:
1. Download and decrypt the database dump to understand file structure
2. Download and decrypt the required blobs
3. Reconstruct files from their chunks
4. Restore file metadata (permissions, timestamps, etc.)
go.mod
+3 -3
View File
@@ -20,12 +20,9 @@ require (
github.com/rclone/rclone v1.72.1
github.com/spf13/afero v1.15.0
github.com/spf13/cobra v1.10.1
github.com/spf13/pflag v1.0.10
github.com/stretchr/testify v1.11.1
go.uber.org/dig v1.19.0
go.uber.org/fx v1.24.0
golang.org/x/sync v0.18.0
golang.org/x/sys v0.38.0
golang.org/x/term v0.37.0
gopkg.in/yaml.v3 v3.0.1
modernc.org/sqlite v1.38.0
@@ -228,6 +225,7 @@ require (
github.com/smarty/assertions v1.16.0 // indirect
github.com/sony/gobreaker v1.0.0 // indirect
github.com/spacemonkeygo/monkit/v3 v3.0.25-0.20251022131615-eb24eb109368 // indirect
github.com/spf13/pflag v1.0.10 // indirect
github.com/t3rm1n4l/go-mega v0.0.0-20251031123324-a804aaa87491 // indirect
github.com/tidwall/gjson v1.18.0 // indirect
github.com/tidwall/match v1.1.1 // indirect
@@ -257,6 +255,7 @@ require (
go.opentelemetry.io/otel/metric v1.38.0 // indirect
go.opentelemetry.io/otel/trace v1.38.0 // indirect
go.shabbyrobe.org/gocovmerge v0.0.0-20230507111327-fa4f82cfbf4d // indirect
go.uber.org/dig v1.19.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
go.uber.org/zap v1.27.0 // indirect
go.yaml.in/yaml/v2 v2.4.3 // indirect
@@ -264,6 +263,7 @@ require (
golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546 // indirect
golang.org/x/net v0.47.0 // indirect
golang.org/x/oauth2 v0.33.0 // indirect
golang.org/x/sys v0.38.0 // indirect
golang.org/x/text v0.31.0 // indirect
golang.org/x/time v0.14.0 // indirect
golang.org/x/tools v0.38.0 // indirect
internal/blob/packer.go
+7 -20
View File
@@ -1,16 +1,14 @@
// Package blob handles the creation of blobs - the final storage units for Vaultik.
// A blob is a large file (up to 10GB) containing many compressed and encrypted chunks
// from multiple source files. Blobs are content-addressed: a blob's filename is
// hex(SHA256(SHA256(uncompressed blob contents))), computed from the concatenated
// chunk data before compression and encryption, not from the stored bytes. See
// blobgen.DoubleSHA256 and docs/REPOSTRUCTURE.md.
// from multiple source files. Blobs are content-addressed, meaning their filename
// is derived from the SHA256 hash of their compressed and encrypted content.
//
// The blob creation process:
// 1. Chunks are accumulated from multiple files
// 2. Each chunk's uncompressed bytes are fed to a running SHA-256 and, in the same
// pass, compressed with zstd and encrypted with age into the temp file
// 3. On finalize, the name is the double SHA-256 of that uncompressed content
// 4. The blob is uploaded to S3 using the name as the filename
// 2. The collection is compressed using zstd
// 3. The compressed data is encrypted using age
// 4. The encrypted blob is hashed to create its content-addressed name
// 5. The blob is uploaded to S3 using the hash as the filename
//
// This design optimizes storage efficiency by batching many small chunks into
// larger blobs, reducing the number of S3 operations and associated costs.
@@ -245,17 +243,6 @@ func (p *Packer) Flush(ctx context.Context) error {
return nil
}
// DiscardCurrentBlob closes and removes the temp file of the blob being
// packed, if any, and drops that blob. A caller that stops before Flush
// must call it, or the temp file is left behind. Thread-safe.
func (p *Packer) DiscardCurrentBlob() {
p.mu.Lock()
defer p.mu.Unlock()
p.cleanupTempFile()
p.currentBlob = nil
}
// FinalizeBlob finalizes the current blob being assembled.
// This compresses the accumulated chunks, encrypts the result, and computes
// the content-addressed hash. The finalized blob is either passed to the
@@ -500,7 +487,7 @@ func (p *Packer) closeBlobWriter() (string, int64, error) {
return "", 0, fmt.Errorf("seeking to start: %w", err)
}
finalHash := p.currentBlob.writer.ContentID()
finalHash := p.currentBlob.writer.Sum256()
return hex.EncodeToString(finalHash), finalSize, nil
}
internal/blobgen/compress.go
+89
View File
@@ -0,0 +1,89 @@
// Package blobgen implements the blob data pipeline: streaming zstd
// compression, age encryption, and SHA256 content hashing for blob
// creation, plus the matching decrypt/decompress/verify reader.
package blobgen
import (
"bytes"
"encoding/hex"
"fmt"
"io"
)
// CompressResult contains the results of compression
type CompressResult struct {
Data []byte
UncompressedSize int64
CompressedSize int64
SHA256 string
}
// CompressData compresses and encrypts data, returning the result with hash
func CompressData(
data []byte, compressionLevel int, recipients []string,
) (*CompressResult, error) {
var buf bytes.Buffer
// Create writer
w, err := NewWriter(&buf, compressionLevel, recipients)
if err != nil {
return nil, fmt.Errorf("creating writer: %w", err)
}
// Write data
_, err = w.Write(data)
if err != nil {
_ = w.Close()
return nil, fmt.Errorf("writing data: %w", err)
}
// Close to flush
err = w.Close()
if err != nil {
return nil, fmt.Errorf("closing writer: %w", err)
}
return &CompressResult{
Data: buf.Bytes(),
UncompressedSize: int64(len(data)),
CompressedSize: int64(buf.Len()),
SHA256: hex.EncodeToString(w.Sum256()),
}, nil
}
// CompressStream compresses and encrypts from reader to writer, returning
// the number of uncompressed bytes written and the content hash.
func CompressStream(
dst io.Writer, src io.Reader, compressionLevel int, recipients []string,
) (int64, string, error) {
// Create writer
w, err := NewWriter(dst, compressionLevel, recipients)
if err != nil {
return 0, "", fmt.Errorf("creating writer: %w", err)
}
closed := false
defer func() {
if !closed {
_ = w.Close()
}
}()
// Copy data
_, err = io.Copy(w, src)
if err != nil {
return 0, "", fmt.Errorf("copying data: %w", err)
}
// Close to flush
err = w.Close()
if err != nil {
return 0, "", fmt.Errorf("closing writer: %w", err)
}
closed = true
return w.BytesWritten(), hex.EncodeToString(w.Sum256()), nil
}
internal/blobgen/compress_test.go
+80
View File
@@ -0,0 +1,80 @@
package blobgen_test
import (
"bytes"
"crypto/rand"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// testRecipient is a static age recipient for tests.
const testRecipient = "age1cplgrwj77ta54dnmydvvmzn64ltk83ankxl5sww04mrtmu62kv3s89gmvv"
// TestCompressStreamNoDoubleClose is a regression test for issue #28.
// It verifies that CompressStream does not panic or return an error due to
// double-closing the underlying blobgen.Writer. Before the fix in PR #33,
// the explicit Close() on the happy path combined with defer Close() would
// cause a double close.
func TestCompressStreamNoDoubleClose(t *testing.T) {
t.Parallel()
input := []byte("regression test data for issue #28 double-close fix")
var buf bytes.Buffer
written, hash, err := blobgen.CompressStream(
&buf, bytes.NewReader(input), 3, []string{testRecipient})
require.NoError(t, err, "CompressStream should not return an error")
assert.Positive(t, written, "expected bytes written > 0")
assert.NotEmpty(t, hash, "expected non-empty hash")
assert.Positive(t, buf.Len(), "expected non-empty output")
}
// TestCompressStreamLargeInput exercises CompressStream with a larger payload
// to ensure no double-close issues surface under heavier I/O.
func TestCompressStreamLargeInput(t *testing.T) {
t.Parallel()
data := make([]byte, 512*1024) // 512 KB
_, err := rand.Read(data)
require.NoError(t, err)
var buf bytes.Buffer
written, hash, err := blobgen.CompressStream(
&buf, bytes.NewReader(data), 3, []string{testRecipient})
require.NoError(t, err)
assert.Positive(t, written)
assert.NotEmpty(t, hash)
}
// TestCompressStreamEmptyInput verifies CompressStream handles empty input
// without double-close issues.
func TestCompressStreamEmptyInput(t *testing.T) {
t.Parallel()
var buf bytes.Buffer
_, hash, err := blobgen.CompressStream(
&buf, strings.NewReader(""), 3, []string{testRecipient})
require.NoError(t, err)
assert.NotEmpty(t, hash)
}
// TestCompressDataNoDoubleClose mirrors the stream test for CompressData,
// ensuring the explicit Close + error-path Close pattern is also safe.
func TestCompressDataNoDoubleClose(t *testing.T) {
t.Parallel()
input := []byte("CompressData regression test for double-close")
result, err := blobgen.CompressData(input, 3, []string{testRecipient})
require.NoError(t, err)
assert.Positive(t, result.CompressedSize)
assert.Equal(t, result.UncompressedSize, int64(len(input)))
assert.NotEmpty(t, result.SHA256)
}
internal/blobgen/helpers_test.go
-119
View File
@@ -1,119 +0,0 @@
package blobgen_test
import (
"bytes"
"crypto/rand"
"errors"
"io"
"testing"
"filippo.io/age"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// ageChunkSize is age's STREAM plaintext chunk size (64 KiB); each encrypted
// chunk adds a 16-byte ChaCha20-Poly1305 tag.
const (
ageChunkSize = 64 * 1024
ageChunkTagSize = 16
ageSegmentSize = ageChunkSize + ageChunkTagSize
ageNonceSize = 16
)
// makeIdentity returns a fresh X25519 identity and its recipient string.
func makeIdentity(t *testing.T) (*age.X25519Identity, string) {
t.Helper()
id, err := age.GenerateX25519Identity()
require.NoError(t, err)
return id, id.Recipient().String()
}
// randomBytes returns n cryptographically random bytes, which do not compress
// so the encrypted payload spans multiple age segments.
func randomBytes(t *testing.T, n int) []byte {
t.Helper()
b := make([]byte, n)
_, err := rand.Read(b)
require.NoError(t, err)
return b
}
// compressibleBytes returns n bytes of a repeating pattern, which zstd packs
// down to a small payload.
func compressibleBytes(n int) []byte {
pattern := bytes.Repeat([]byte("compressible-"), n/13+1)
return pattern[:n]
}
// encryptBlob compresses, encrypts and returns a blob for plaintext at
// compression level 1.
func encryptBlob(t *testing.T, plaintext []byte, recipients ...string) []byte {
t.Helper()
var buf bytes.Buffer
w, err := blobgen.NewWriter(&buf, 1, recipients)
require.NoError(t, err)
_, err = w.Write(plaintext)
require.NoError(t, err)
require.NoError(t, w.Close())
return buf.Bytes()
}
// ageHeaderLen returns the byte length of blob's age header, i.e. the offset
// of the 16-byte payload nonce that follows it. The header ends with a MAC
// line "--- <mac>\n"; the nonce begins right after that newline.
func ageHeaderLen(t *testing.T, blob []byte) int {
t.Helper()
i := bytes.Index(blob, []byte("\n--- "))
require.GreaterOrEqual(t, i, 0, "age MAC footer line not found")
nl := bytes.IndexByte(blob[i+1:], '\n')
require.GreaterOrEqual(t, nl, 0, "newline ending MAC line not found")
return i + 1 + nl + 1
}
// requireBlobUnreadable asserts that data never decrypts to a plaintext with a
// nil error: either NewReader fails, or reading it does.
func requireBlobUnreadable(t *testing.T, data []byte, id age.Identity) {
t.Helper()
r, err := blobgen.NewReader(bytes.NewReader(data), id)
if err != nil {
return
}
_, err = io.ReadAll(r)
_ = r.Close()
require.Error(t, err, "reading a damaged blob must fail")
}
// errFailWriter is returned by failAfterWriter once its byte limit is passed.
var errFailWriter = errors.New("destination write failed")
// failAfterWriter accepts writes until more than limit bytes have been sent,
// then fails every write. It models a destination that dies mid-blob.
type failAfterWriter struct {
limit int
written int
}
func (f *failAfterWriter) Write(p []byte) (int, error) {
f.written += len(p)
if f.written > f.limit {
return 0, errFailWriter
}
return len(p), nil
}
internal/blobgen/limit.go
-49
View File
@@ -1,49 +0,0 @@
package blobgen
import (
"errors"
"io"
)
// ErrOutputTooLarge is returned by a reader from LimitReader once it has
// been asked for more than its limit. It bounds how far an untrusted
// compressed stream may expand, so a small, highly compressible object
// from the store cannot decompress without limit.
var ErrOutputTooLarge = errors.New("output exceeds size limit")
// LimitReader returns a reader that yields at most limit bytes from r and
// then fails with ErrOutputTooLarge. Unlike io.LimitReader, which reports
// a silent io.EOF at the limit (indistinguishable from a stream that
// simply ended), this fails, so a caller decoding or copying the stream
// sees an error rather than a truncated value. A stream of exactly limit
// bytes reads back cleanly to EOF; the first byte beyond it is the error.
func LimitReader(r io.Reader, limit int64) io.Reader {
// remaining counts down from limit+1: the extra byte is the one that,
// if it ever arrives, proves the stream is longer than the limit.
return &limitReader{r: r, remaining: limit + 1}
}
type limitReader struct {
r io.Reader
remaining int64
}
func (l *limitReader) Read(p []byte) (int, error) {
if l.remaining <= 0 {
return 0, ErrOutputTooLarge
}
if int64(len(p)) > l.remaining {
p = p[:l.remaining]
}
n, err := l.r.Read(p)
l.remaining -= int64(n)
if l.remaining <= 0 {
// The (limit+1)th byte was just read: the stream is too long.
return n, ErrOutputTooLarge
}
return n, err
}
internal/blobgen/limit_test.go
-43
View File
@@ -1,43 +0,0 @@
package blobgen_test
import (
"bytes"
"io"
"testing"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// TestLimitReaderPassesExactSize checks that a stream of exactly the limit
// reads back cleanly to EOF: the bound must not reject a legitimate blob
// whose plaintext equals its recorded size.
func TestLimitReaderPassesExactSize(t *testing.T) {
t.Parallel()
const n = 1000
r := blobgen.LimitReader(bytes.NewReader(bytes.Repeat([]byte("a"), n)), n)
got, err := io.ReadAll(r)
require.NoError(t, err)
require.Len(t, got, n)
}
// TestLimitReaderFailsPastLimit feeds a large, highly compressible run of
// zeros — the decompressed output a zip bomb would produce — through a
// small limit and checks it fails within the bound rather than passing
// the whole stream through.
func TestLimitReaderFailsPastLimit(t *testing.T) {
t.Parallel()
const limit = 1000
r := blobgen.LimitReader(
bytes.NewReader(bytes.Repeat([]byte{0}, limit*1000)), limit)
n, err := io.Copy(io.Discard, r)
require.ErrorIs(t, err, blobgen.ErrOutputTooLarge)
require.LessOrEqual(t, n, int64(limit)+1,
"reader must stop within one byte of the limit")
}
internal/blobgen/negative_test.go
-190
View File
@@ -1,190 +0,0 @@
package blobgen_test
import (
"bytes"
"fmt"
"testing"
"filippo.io/age"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// TestNewReaderWrongIdentity covers issue case 4: opening a blob with an
// identity other than the recipient reports no matching identity.
func TestNewReaderWrongIdentity(t *testing.T) {
t.Parallel()
_, recipient := makeIdentity(t)
other, _ := makeIdentity(t)
blob := encryptBlob(t, []byte("secret payload"), recipient)
_, err := blobgen.NewReader(bytes.NewReader(blob), other)
require.Error(t, err)
var noMatch *age.NoIdentityMatchError
assert.ErrorAs(t, err, &noMatch)
}
// TestNewReaderTruncated covers issue case 6: a multi-segment blob cut at
// several points must never read back as valid data. The point immediately
// after the header and nonce is intentionally excluded: it reads as a valid
// empty blob today and is the regression case for
// https://git.eeqj.de/sneak/vaultik/issues/152.
func TestNewReaderTruncated(t *testing.T) {
t.Parallel()
id, recipient := makeIdentity(t)
blob := encryptBlob(t, randomBytes(t, 4*65536+123), recipient)
h := ageHeaderLen(t, blob)
require.Greater(t, len(blob), h+ageNonceSize+ageSegmentSize,
"test needs a blob of at least two age segments")
cases := []struct {
name string
size int
}{
{"inside header", h / 2},
{"inside nonce", h + 8},
{"inside first segment", h + ageNonceSize + 100},
{"end of first full segment", h + ageNonceSize + ageSegmentSize},
{"last byte removed", len(blob) - 1},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
requireBlobUnreadable(t, blob[:tc.size], id)
})
}
}
// TestNewReaderCorrupted covers issue case 7: one flipped byte in each region
// of a multi-segment blob makes it unreadable.
func TestNewReaderCorrupted(t *testing.T) {
t.Parallel()
id, recipient := makeIdentity(t)
blob := encryptBlob(t, randomBytes(t, 4*65536+123), recipient)
h := ageHeaderLen(t, blob)
firstNL := bytes.IndexByte(blob, '\n')
require.Positive(t, firstNL, "header must have a version line")
cases := []struct {
name string
pos int
}{
{"header stanza", firstNL + 5},
{"header MAC line", h - 2},
{"nonce", h + 4},
{"body segment", h + ageNonceSize + 50},
{"final tag", len(blob) - 1},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
corrupt := append([]byte(nil), blob...)
corrupt[tc.pos] ^= 0xff
requireBlobUnreadable(t, corrupt, id)
})
}
}
// TestNewReaderTrailingAndGarbage covers issue case 8: bytes appended after a
// valid blob, empty input, and random garbage each fail to read.
func TestNewReaderTrailingAndGarbage(t *testing.T) {
t.Parallel()
id, recipient := makeIdentity(t)
valid := encryptBlob(t, []byte("small payload"), recipient)
appended := append(append([]byte(nil), valid...), []byte("trailing junk")...)
t.Run("appended bytes", func(t *testing.T) {
t.Parallel()
requireBlobUnreadable(t, appended, id)
})
t.Run("empty input", func(t *testing.T) {
t.Parallel()
requireBlobUnreadable(t, []byte{}, id)
})
t.Run("random garbage", func(t *testing.T) {
t.Parallel()
requireBlobUnreadable(t, randomBytes(t, 512), id)
})
}
// TestNewWriterInvalidLevel covers the rejected end of issue case 9: an
// out-of-range compression level errors and writes nothing to the destination.
func TestNewWriterInvalidLevel(t *testing.T) {
t.Parallel()
_, recipient := makeIdentity(t)
for _, level := range []int{0, -1, 20} {
t.Run(fmt.Sprintf("level%d", level), func(t *testing.T) {
t.Parallel()
var buf bytes.Buffer
w, err := blobgen.NewWriter(&buf, level, []string{recipient})
require.ErrorIs(t, err, blobgen.ErrInvalidCompressionLevel)
assert.Nil(t, w)
assert.Zero(t, buf.Len(), "nothing written on an invalid level")
})
}
}
// TestNewWriterInvalidRecipients covers issue case 10: nil and empty recipient
// lists and an unparsable recipient string each error.
func TestNewWriterInvalidRecipients(t *testing.T) {
t.Parallel()
cases := []struct {
name string
recipients []string
}{
{"nil list", nil},
{"empty list", []string{}},
{"invalid recipient string", []string{"not-a-recipient"}},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
var buf bytes.Buffer
w, err := blobgen.NewWriter(&buf, 1, tc.recipients)
require.Error(t, err)
assert.Nil(t, w)
})
}
}
// TestNewWriterFailingDestination covers issue case 11: a destination that
// fails mid-blob surfaces its error from Write or Close.
func TestNewWriterFailingDestination(t *testing.T) {
t.Parallel()
_, recipient := makeIdentity(t)
// The limit clears the age header and nonce so NewWriter succeeds, then
// trips once the compressed body starts flowing.
dst := &failAfterWriter{limit: 512}
w, err := blobgen.NewWriter(dst, 1, []string{recipient})
require.NoError(t, err)
_, writeErr := w.Write(randomBytes(t, 256*1024))
closeErr := w.Close()
assert.True(t, writeErr != nil || closeErr != nil,
"destination failure must surface from Write or Close")
}
internal/blobgen/reader.go
+4 -25
View File
@@ -2,7 +2,6 @@ package blobgen
import (
"crypto/sha256"
"errors"
"fmt"
"hash"
"io"
@@ -21,12 +20,10 @@ type Reader struct {
bytesRead int64
}
// NewReader creates a new Reader that decrypts, decompresses, and verifies
// data. Every supplied identity is offered to age.Decrypt, so a blob
// encrypted to any one of them can be read.
func NewReader(r io.Reader, identities ...age.Identity) (*Reader, error) {
// NewReader creates a new Reader that decrypts, decompresses, and verifies data
func NewReader(r io.Reader, identity age.Identity) (*Reader, error) {
// Create decryption reader
decReader, err := age.Decrypt(r, identities...)
decReader, err := age.Decrypt(r, identity)
if err != nil {
return nil, fmt.Errorf("creating decryption reader: %w", err)
}
@@ -57,22 +54,6 @@ func (r *Reader) Read(p []byte) (int, error) {
n, err := r.teeReader.Read(p)
r.bytesRead += int64(n)
// When the ciphertext is cut right after the age header plus its
// 16-byte nonce, the age reader's first read fails with
// io.ErrUnexpectedEOF, and the zstd decoder maps that to a clean
// io.EOF at frame start. That makes a truncated stream look like a
// valid empty one. Distinguish the two: on EOF, read once more from
// the age reader. A genuine end leaves it at (0, io.EOF); a truncated
// stream leaves its stored io.ErrUnexpectedEOF, which we surface.
if errors.Is(err, io.EOF) {
var probe [1]byte
m, ageErr := r.decryptor.Read(probe[:])
if m != 0 || !errors.Is(ageErr, io.EOF) {
return n, io.ErrUnexpectedEOF
}
}
return n, err
}
@@ -83,9 +64,7 @@ func (r *Reader) Close() error {
return nil
}
// Sum256 returns the single SHA-256 of the plaintext read so far. This is the
// first hash only; the stored object name is its double hash, which callers
// obtain by passing this digest to DoubleSHA256.
// Sum256 returns the SHA256 hash of all data read
func (r *Reader) Sum256() []byte {
return r.hasher.Sum(nil)
}
internal/blobgen/reader_test.go
-54
View File
@@ -1,54 +0,0 @@
package blobgen_test
import (
"bytes"
"io"
"testing"
"filippo.io/age"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// TestMultipleRecipients verifies that data written for several recipients can
// be read back by each recipient's identity. Moved from internal/crypto, which
// held the only multi-recipient test; blobgen is now the sole encryption path.
func TestMultipleRecipients(t *testing.T) {
t.Parallel()
identities := make([]*age.X25519Identity, 3)
recipients := make([]string, 3)
for i := range identities {
identity, err := age.GenerateX25519Identity()
require.NoError(t, err)
identities[i] = identity
recipients[i] = identity.Recipient().String()
}
plaintext := []byte("Secret message for multiple recipients")
var encrypted bytes.Buffer
writer, err := blobgen.NewWriter(&encrypted, 3, recipients)
require.NoError(t, err)
_, err = writer.Write(plaintext)
require.NoError(t, err)
require.NoError(t, writer.Close())
// Every recipient's identity must recover the original plaintext.
for i, identity := range identities {
reader, err := blobgen.NewReader(
bytes.NewReader(encrypted.Bytes()), identity)
require.NoError(t, err, "recipient %d should open the reader", i+1)
got, err := io.ReadAll(reader)
require.NoError(t, err, "recipient %d should read the plaintext", i+1)
require.NoError(t, reader.Close())
assert.Equal(t, plaintext, got,
"recipient %d should recover the original plaintext", i+1)
}
}
internal/blobgen/roundtrip_test.go
-132
View File
@@ -1,132 +0,0 @@
package blobgen_test
import (
"bytes"
"crypto/sha256"
"fmt"
"io"
"testing"
"filippo.io/age"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// checkRoundTrip writes input through a Writer, reads it back through a Reader,
// and verifies the plaintext, the byte counts, and the content hashes.
func checkRoundTrip(
t *testing.T, id *age.X25519Identity, recipient string,
level int, input []byte,
) {
t.Helper()
var buf bytes.Buffer
w, err := blobgen.NewWriter(&buf, level, []string{recipient})
require.NoError(t, err)
n, err := w.Write(input)
require.NoError(t, err)
assert.Equal(t, len(input), n)
require.NoError(t, w.Close())
require.Equal(t, int64(len(input)), w.BytesWritten())
r, err := blobgen.NewReader(bytes.NewReader(buf.Bytes()), id)
require.NoError(t, err)
got, err := io.ReadAll(r)
require.NoError(t, err)
require.NoError(t, r.Close())
assert.Equal(t, input, got, "decrypted output must equal input")
require.Equal(t, int64(len(input)), r.BytesRead())
// The hash values are checked by decrypting: the reader's single SHA-256
// is the hash of the plaintext, and hashing it once more (DoubleSHA256)
// gives the writer's ContentID.
single := sha256.Sum256(got)
assert.Equal(t, single[:], r.Sum256())
assert.Equal(t, blobgen.DoubleSHA256(r.Sum256()), w.ContentID())
}
// TestWriterReaderRoundTrip covers issue cases 1 and 2: every size round trips
// for both random and compressible data, and the reader hash, its double hash
// and the byte counts all agree.
func TestWriterReaderRoundTrip(t *testing.T) {
t.Parallel()
id, recipient := makeIdentity(t)
// Sizes exercise the age segment boundary (64 KiB) from just below to a
// few segments above it, plus the empty and single-byte edges.
sizes := []int{0, 1, 65535, 65536, 65537, 4*65536 + 123}
kinds := []struct {
name string
fill func(*testing.T, int) []byte
}{
{"random", randomBytes},
{"compressible", func(_ *testing.T, n int) []byte {
return compressibleBytes(n)
}},
}
for _, k := range kinds {
for _, size := range sizes {
name := fmt.Sprintf("%s/%d", k.name, size)
t.Run(name, func(t *testing.T) {
t.Parallel()
checkRoundTrip(t, id, recipient, 1, k.fill(t, size))
})
}
}
}
// TestZeroLengthNoWrite covers issue case 3: a Writer closed with no Write at
// all produces the double hash of the empty input, and the blob reads back as
// empty with no error.
func TestZeroLengthNoWrite(t *testing.T) {
t.Parallel()
id, recipient := makeIdentity(t)
var buf bytes.Buffer
w, err := blobgen.NewWriter(&buf, 1, []string{recipient})
require.NoError(t, err)
require.NoError(t, w.Close())
assert.Equal(t, int64(0), w.BytesWritten())
empty := sha256.Sum256(nil)
doubled := sha256.Sum256(empty[:])
assert.Equal(t, doubled[:], w.ContentID(),
"ContentID of empty input is SHA256(SHA256(\"\"))")
r, err := blobgen.NewReader(bytes.NewReader(buf.Bytes()), id)
require.NoError(t, err)
got, err := io.ReadAll(r)
require.NoError(t, err)
require.NoError(t, r.Close())
assert.Empty(t, got, "empty blob decrypts to empty output")
assert.Equal(t, int64(0), r.BytesRead())
assert.Equal(t, empty[:], r.Sum256())
}
// TestNewWriterValidLevelsRoundTrip covers the accepted end of issue case 9:
// the boundary compression levels 1 and 19 both round trip.
func TestNewWriterValidLevelsRoundTrip(t *testing.T) {
t.Parallel()
id, recipient := makeIdentity(t)
input := randomBytes(t, 4096)
for _, level := range []int{1, 19} {
t.Run(fmt.Sprintf("level%d", level), func(t *testing.T) {
t.Parallel()
checkRoundTrip(t, id, recipient, level, input)
})
}
}
internal/blobgen/truncation_test.go
-70
View File
@@ -1,70 +0,0 @@
package blobgen_test
import (
"bytes"
"io"
"testing"
"filippo.io/age"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/blobgen"
)
// TestReaderRejectsHeaderNonceTruncation guards against a stream cut right
// after the age header plus its 16-byte nonce. age.Decrypt still succeeds on
// such an object, and the zstd decoder maps the age reader's
// io.ErrUnexpectedEOF to a clean io.EOF at frame start, so without the extra
// check the truncated stream would read as a valid empty one. Reading it must
// now fail.
func TestReaderRejectsHeaderNonceTruncation(t *testing.T) {
t.Parallel()
identity, err := age.GenerateX25519Identity()
require.NoError(t, err)
// Encrypting empty plaintext yields header + nonce(16) + a single
// 16-byte final chunk tag. Dropping the trailing tag leaves exactly the
// age header plus its nonce — the truncation point that triggers the bug.
var full bytes.Buffer
w, err := age.Encrypt(&full, identity.Recipient())
require.NoError(t, err)
require.NoError(t, w.Close())
truncated := full.Bytes()[:full.Len()-16]
reader, err := blobgen.NewReader(bytes.NewReader(truncated), identity)
require.NoError(t, err)
defer func() { _ = reader.Close() }()
_, err = io.ReadAll(reader)
require.Error(t, err)
require.ErrorIs(t, err, io.ErrUnexpectedEOF)
}
// TestReaderReadsGenuinelyEmptyBlob confirms the truncation check does not
// reject a legitimately empty payload: a blob written with no data must round
// trip back to zero bytes with no error.
func TestReaderReadsGenuinelyEmptyBlob(t *testing.T) {
t.Parallel()
identity, err := age.GenerateX25519Identity()
require.NoError(t, err)
var encrypted bytes.Buffer
writer, err := blobgen.NewWriter(
&encrypted, 3, []string{identity.Recipient().String()})
require.NoError(t, err)
require.NoError(t, writer.Close())
reader, err := blobgen.NewReader(bytes.NewReader(encrypted.Bytes()), identity)
require.NoError(t, err)
defer func() { _ = reader.Close() }()
data, err := io.ReadAll(reader)
require.NoError(t, err)
require.Empty(t, data)
}
internal/blobgen/writer.go
+13 -36
View File
@@ -1,6 +1,3 @@
// Package blobgen implements the blob data pipeline: streaming zstd
// compression, age encryption, and SHA256 content hashing for blob
// creation, plus the matching decrypt/decompress/verify reader.
package blobgen
import (
@@ -15,24 +12,6 @@ import (
"github.com/klauspost/compress/zstd"
)
// DoubleSHA256 returns the double SHA-256 of content whose single SHA-256
// digest is sum: it hashes that digest once more. Stored objects — a blob, and
// the metadata database export — are named by this second hash.
//
// The second hash does not hide whether known content is stored: an attacker
// who can reproduce an object's entire plaintext computes the same name simply
// by hashing twice, exactly as this code does. What limits that is blob
// packing, not the double hash — a blob's name covers all of its concatenated
// chunk plaintext, so a name can be confirmed only by someone who can
// reproduce the whole blob (a snapshot made entirely of known content, or a
// known file large enough to fill blobs on its own). An ordinary file that
// shares a blob with other, unknown data cannot be confirmed this way.
func DoubleSHA256(sum []byte) []byte {
h := sha256.Sum256(sum)
return h[:]
}
// Zstd compression level bounds accepted by NewWriter.
const (
minCompressionLevel = 1
@@ -48,11 +27,6 @@ const reservedCompressionCPUs = 2
var ErrInvalidCompressionLevel = errors.New(
"invalid compression level: must be between 1 and 19")
// errInvalidRecipient is returned when a recipient string does not parse as
// an X25519 age1... public key. It omits the value, which can be sensitive.
var errInvalidRecipient = errors.New(
"not a valid X25519 age1... recipient")
// Writer wraps compression and encryption with SHA256 hashing.
// Data flows: input -> tee(hasher, compressor -> encryptor -> destination)
// The hash is computed on the uncompressed input for deterministic content-addressing.
@@ -83,12 +57,10 @@ func NewWriter(
// Parse recipients
var ageRecipients []age.Recipient
for i, recipient := range recipients {
// The recipient string can be sensitive (e.g. a secret key pasted by
// mistake), so the error names its position, never its value.
for _, recipient := range recipients {
r, err := age.ParseX25519Recipient(recipient)
if err != nil {
return nil, fmt.Errorf("%w: recipient %d", errInvalidRecipient, i)
return nil, fmt.Errorf("parsing recipient %s: %w", recipient, err)
}
ageRecipients = append(ageRecipients, r)
@@ -151,12 +123,17 @@ func (w *Writer) Close() error {
return nil
}
// ContentID returns the double SHA-256 of the uncompressed input data: the
// name under which this content is stored. It is the second hash of the
// running SHA-256, via DoubleSHA256; see that function for what naming content
// this way does and does not hide.
func (w *Writer) ContentID() []byte {
return DoubleSHA256(w.hasher.Sum(nil))
// Sum256 returns the double SHA256 hash of the uncompressed input data.
// Double hashing (SHA256(SHA256(data))) prevents information leakage about
// the plaintext - an attacker cannot confirm existence of known content
// by computing its hash and checking for a matching blob filename.
func (w *Writer) Sum256() []byte {
// First hash: SHA256(plaintext)
firstHash := w.hasher.Sum(nil)
// Second hash: SHA256(firstHash) - this is the blob ID
secondHash := sha256.Sum256(firstHash)
return secondHash[:]
}
// BytesWritten returns the number of uncompressed bytes written
internal/blobgen/writer_test.go
+9 -27
View File
@@ -12,10 +12,9 @@ import (
"sneak.berlin/go/vaultik/internal/blobgen"
)
// TestWriterHashIsDoubleHash verifies that Writer.ContentID() returns
// SHA256(SHA256(plaintext)). Stored objects are named by this second hash so a
// name is not the plaintext's own SHA-256; this does not stop someone who
// already holds the plaintext from confirming it.
// TestWriterHashIsDoubleHash verifies that Writer.Sum256() returns
// the double hash SHA256(SHA256(plaintext)) for security.
// Double hashing prevents attackers from confirming existence of known content.
func TestWriterHashIsDoubleHash(t *testing.T) {
t.Parallel()
@@ -44,7 +43,7 @@ func TestWriterHashIsDoubleHash(t *testing.T) {
require.NoError(t, err)
// Get the hash from the writer
writerHash := hex.EncodeToString(writer.ContentID())
writerHash := hex.EncodeToString(writer.Sum256())
// Calculate the expected double hash: SHA256(SHA256(plaintext))
firstHash := sha256.Sum256(testData)
@@ -61,11 +60,11 @@ func TestWriterHashIsDoubleHash(t *testing.T) {
// The writer hash should match the double hash
assert.Equal(t, expectedDoubleHash, writerHash,
"Writer.ContentID() must be SHA256(SHA256(plaintext))")
"Writer.Sum256() should return SHA256(SHA256(plaintext)) for security")
// It must be the second hash, not the plaintext's own SHA-256.
// Verify it's NOT the single hash (would leak information)
assert.NotEqual(t, singleHashStr, writerHash,
"Writer hash must be the double hash, not the single SHA-256")
"Writer hash should not be single hash (would allow content confirmation attacks)")
}
// TestWriterDeterministicHash verifies that the same input always produces
@@ -94,8 +93,8 @@ func TestWriterDeterministicHash(t *testing.T) {
require.NoError(t, err)
require.NoError(t, writer2.Close())
hash1 := hex.EncodeToString(writer1.ContentID())
hash2 := hex.EncodeToString(writer2.ContentID())
hash1 := hex.EncodeToString(writer1.Sum256())
hash2 := hex.EncodeToString(writer2.Sum256())
// Hashes should be identical (deterministic)
assert.Equal(t, hash1, hash2, "Same input should produce same hash")
@@ -109,20 +108,3 @@ func TestWriterDeterministicHash(t *testing.T) {
t.Logf("Encrypted size 1: %d bytes", buf1.Len())
t.Logf("Encrypted size 2: %d bytes", buf2.Len())
}
// TestNewWriterSecretKeyNotEchoed verifies that a secret key mistakenly passed
// as a recipient does not appear in the returned error. A recipient string can
// be sensitive, so the error must name only the position, not the value.
func TestNewWriterSecretKeyNotEchoed(t *testing.T) {
t.Parallel()
secretKey := "AGE-SECRET-KEY-19CR5YSFW59HM4TLD6GX" +
"VEDMZFTVVF7PPHKUT68TXSFPK7APHXA2QS2NJA5"
var buf bytes.Buffer
_, err := blobgen.NewWriter(&buf, 3, []string{secretKey})
require.Error(t, err)
assert.NotContains(t, err.Error(), secretKey,
"error must not echo the recipient value")
}
internal/chunker/chunker.go
+4 -5
View File
@@ -33,10 +33,9 @@ type Chunker struct {
maxChunkSize int
}
// ChunkSizeSpread is the FastCDC-recommended factor between the average
// chunkSizeSpread is the FastCDC-recommended factor between the average
// chunk size and the minimum (avg/spread) and maximum (avg*spread) sizes.
// The largest chunk the chunker can emit is therefore avg*ChunkSizeSpread.
const ChunkSizeSpread = 4
const chunkSizeSpread = 4
// NewChunker creates a new chunker with the specified average chunk size.
// The actual chunk sizes will vary between avgChunkSize/4 and avgChunkSize*4
@@ -46,8 +45,8 @@ func NewChunker(avgChunkSize int64) *Chunker {
// FastCDC recommends min = avg/4 and max = avg*4
return &Chunker{
avgChunkSize: int(avgChunkSize),
minChunkSize: int(avgChunkSize / ChunkSizeSpread),
maxChunkSize: int(avgChunkSize * ChunkSizeSpread),
minChunkSize: int(avgChunkSize / chunkSizeSpread),
maxChunkSize: int(avgChunkSize * chunkSizeSpread),
}
}
internal/cli/app.go
+113 -217
View File
@@ -7,13 +7,15 @@ import (
"context"
"errors"
"fmt"
"os"
"os/signal"
"path/filepath"
"sync"
"strings"
"syscall"
"time"
"github.com/adrg/xdg"
"github.com/spf13/cobra"
"go.uber.org/dig"
"go.uber.org/fx"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/database"
@@ -30,33 +32,14 @@ import (
// may take before we give up.
const shutdownTimeout = 30 * time.Second
// lockMode says whether a command mutates persistent state — the local
// index database or the remote store — and so must hold the process-wide
// PID lock, or only reads that state and may run alongside a mutator.
type lockMode int
const (
// mutating commands (snapshot create, snapshot purge, snapshot remove,
// prune, remote nuke) write the local index or the remote store. They
// hold the PID lock so that at most one runs at a time.
mutating lockMode = iota
// readOnly commands (info, snapshot list, snapshot verify, remote info,
// snapshot restore) do not write the local index or the remote store,
// so they run without the lock and are never blocked by a running
// mutator. restore writes only to the target directory it is given.
readOnly
)
// AppOptions contains common options for creating and running the fx
// application: the configuration file path, logging options, additional fx
// modules and invocations, and whether the command mutates persistent
// state (which decides whether it takes the PID lock).
// AppOptions contains common options for creating the fx application.
// It includes the configuration file path, logging options, and additional
// fx modules and invocations that should be included in the application.
type AppOptions struct {
ConfigPath string
LogOptions log.Options
Modules []fx.Option
Invokes []fx.Option
Mode lockMode
}
// setupGlobals records the startup time and, when an output-suppression
@@ -65,11 +48,6 @@ type AppOptions struct {
// silenced — per the documented convention that --quiet suppresses
// non-error output only. The startup banner is printed by Entry
// before cobra parses arguments, gated by the same arg-level check.
//
// --json quiets the UI here too, because stdout then carries a JSON
// document and human narration would corrupt it. Unlike Quiet it does
// not lower the stderr log level (issue #112), so --verbose/--debug
// still surface diagnostics alongside the document.
func setupGlobals(
lc fx.Lifecycle, g *globals.Globals, v *vaultik.Vaultik, opts log.Options,
) {
@@ -77,7 +55,7 @@ func setupGlobals(
OnStart: func(_ context.Context) error {
g.StartTime = time.Now().UTC()
if opts.Cron || opts.Quiet || opts.JSON {
if opts.Cron || opts.Quiet {
v.UI.SetQuiet(true)
}
@@ -146,180 +124,87 @@ func (e *startupError) Error() string {
// could not build arguments for function "Y" (file:line): failed to build U:
// received non-nil error from function "Z" (file:line): <real error>
//
// Users care about the real error, not the DI plumbing. dig.RootCause,
// from the library fx is built on, unwraps those layers and returns the
// real error whole. Cutting the text at "): " instead would also cut a
// real error that contains it, such as the blob_size_limit one.
// Users care about the real error, not the DI plumbing. We strip everything
// up through the last "): " (which is always the close-paren of an fx
// function-location annotation followed by the wrapped error).
func cleanStartupError(err error) error {
return &startupError{msg: dig.RootCause(err).Error()}
msg := err.Error()
if idx := strings.LastIndex(msg, "): "); idx >= 0 {
msg = msg[idx+3:]
}
// RunApp starts the fx application, blocks until it is asked to stop, and
// then stops it. The app is asked to stop either by an OS interrupt
// (SIGINT/SIGTERM — fx installs its own handler when app.Wait is called) or,
// on normal completion, by the finished operation calling
// Shutdowner.Shutdown(); both arrive on the app.Wait channel.
//
// Stopping runs the fx OnStop hooks, and RunApp does not return until Stop
// returns. On an interrupt the operation's OnStop hook cancels the running
// command and waits for it to unwind — removing its decrypted scratch files —
// so the process cannot proceed to exit mid-cleanup (issue #159). Waiting for
// Stop before returning is what makes that hook effective: routing the
// interrupt through app.Stop and not returning until it completes is required,
// because fx also fires the app.Wait channel on the signal, and an earlier
// version returned on that alone — unwinding to os.Exit while the concurrent
// cleanup still ran. The stop is bounded by shutdownTimeout. Returns an error
// if startup fails.
return &startupError{msg: msg}
}
// RunApp starts and stops the fx application within the given context.
// It handles graceful shutdown on interrupt signals (SIGINT, SIGTERM) and
// ensures the application stops cleanly. The function blocks until the
// application completes or is interrupted. Returns an error if startup fails.
func RunApp(ctx context.Context, app *fx.App) error {
// Set up signal handling for graceful shutdown
sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, os.Interrupt, syscall.SIGTERM)
// Create a context that will be cancelled on signal
ctx, cancel := context.WithCancel(ctx)
defer cancel()
// Start the app
err := app.Start(ctx)
if err != nil {
return cleanStartupError(err)
}
// Block until an interrupt or the finished operation's
// Shutdowner.Shutdown() arrives, then stop the app in this goroutine so we
// return only after its OnStop hooks — including the operation's cleanup
// wait — have run. Detach the stop from ctx's cancellation but keep its
// values, and bound it by shutdownTimeout.
<-app.Wait()
// Handle shutdown
shutdownComplete := make(chan struct{})
shutdownCtx, cancel := context.WithTimeout(
go func() {
defer close(shutdownComplete)
<-sigChan
log.Notice("Received interrupt signal, shutting down gracefully...")
// Create a timeout context for shutdown. The parent ctx is being
// cancelled, so detach from its cancellation but keep its values.
shutdownCtx, shutdownCancel := context.WithTimeout(
context.WithoutCancel(ctx), shutdownTimeout)
defer cancel()
defer shutdownCancel()
err = app.Stop(shutdownCtx)
err := app.Stop(shutdownCtx)
if err != nil {
log.Error("Error during shutdown", "error", err)
}
}()
// Wait for the signal handler to complete shutdown or the app to
// request shutdown.
select {
case <-shutdownComplete:
// Shutdown completed via signal
return nil
}
// errReported marks a failure the operation has already shown the user
// (or, under `snapshot verify --json`, put in its document). Entry
// turns it into a non-zero exit status without printing anything
// further, so the error line is not doubled. It flows up from
// RunOperation through cobra to Entry.
var errReported = errors.New("operation failed")
// errInterrupted marks an operation that SIGINT or SIGTERM stopped
// before it finished. Entry shows it and returns exitCodeInterrupted.
var errInterrupted = errors.New("interrupted before the command finished")
// RunOperation runs op against the Vaultik instance inside the fx app
// and turns a failure into a returned error rather than an os.Exit from
// within the goroutine. An os.Exit there skipped main's deferred
// profile writers -- so profiling a failing command yielded a truncated
// profile (issue #75) -- and RunWithApp's PID-lock release, and denied
// the app any graceful shutdown; returning the error to the top runs
// all three.
//
// op runs in a goroutine so OnStart returns promptly and an interrupt
// can still cancel through OnStop; when it finishes, success or failure,
// it triggers shutdown, which is what lets RunWithApp return. On an
// interrupt OnStop cancels op and waits for the goroutine to return, so
// op's cleanup (removing decrypted scratch files) runs before the
// process exits; the wait is bounded by shutdownTimeout. report is
// called with a failure so the caller can show it to the user before
// it becomes errReported.
//
// The run counts as interrupted unless op returned, without an
// interrupt having cancelled it, before RunWithApp returned. An
// interrupted op is not reported, whatever it returned; RunOperation
// returns errInterrupted instead.
func RunOperation(
ctx context.Context, opts AppOptions,
op func(v *vaultik.Vaultik) error, report func(err error),
) error {
var (
mu sync.Mutex
finished bool // op returned before any interrupt cancelled it
failed bool // op finished with an error
)
opts.Invokes = append(opts.Invokes,
fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
var stop func(context.Context) bool
lc.Append(fx.Hook{
OnStart: func(_ context.Context) error {
stop = v.StartOperation(func() {
err := op(v)
// Only stop, called from OnStop below, cancels the
// Vaultik context, so a live context means no
// interrupt cancelled op. Check the context, not
// err: an interrupted op need not return
// context.Canceled (`snapshot verify --json`
// returns a verification failure).
if v.Context().Err() == nil {
case <-ctx.Done():
// Context cancelled (shouldn't happen in normal operation)
err := app.Stop(context.WithoutCancel(ctx))
if err != nil {
report(err)
log.Error("Error stopping app", "error", err)
}
mu.Lock()
finished = true
failed = err != nil
mu.Unlock()
}
stopErr := v.Shutdowner.Shutdown()
if stopErr != nil {
log.Error("Failed to shutdown", "error", stopErr)
}
})
return nil
},
// On an interrupt, cancel the operation and wait for it to
// unwind so its cleanup defers (which remove decrypted
// scratch files from the temp directory) run before the
// process exits. The wait is bounded by ctx, the existing
// shutdownTimeout.
OnStop: func(ctx context.Context) error {
if !stop(ctx) {
log.Warn("Shutdown timed out before the operation " +
"finished; decrypted temporary files may remain")
}
return nil
},
})
}))
err := RunWithApp(ctx, opts)
if err != nil {
return err
}
// RunWithApp returns only after the app was asked to stop, either by
// an interrupt or by the goroutine's Shutdown call. When op finished
// without being cancelled, the goroutine set finished before that
// call. So if finished is unset here, an interrupt stopped the app,
// and op either returned after it was cancelled or is still running
// because the shutdown timed out.
mu.Lock()
defer mu.Unlock()
switch {
case !finished:
return errInterrupted
case failed:
return errReported
default:
return ctx.Err()
case <-app.Done():
// App finished running (e.g., backup completed)
return nil
}
}
// runVaultikApp runs the standard single-operation command lifecycle
// shared by the snapshot list/purge/remove and remote nuke subcommands:
// resolve the config, then run op against the Vaultik instance through
// RunOperation, reporting a failure prefixed with failMsg on stderr. mode
// says whether the command takes the PID lock. jsonOutput marks a command
// whose stdout is a JSON document: it quiets the UI but, unlike Quiet,
// leaves the stderr log level alone.
// shared by the list/purge/verify/remove/remote-info subcommands:
// resolve the config, start the fx app, run op against the Vaultik
// instance in a goroutine, report a failure prefixed with failMsg
// (suppressed while suppressErrors is true, e.g. under --json), then
// trigger shutdown. The operation is cancelled when the app stops.
// extraQuiet is OR-ed into LogOptions.Quiet (e.g. --json output modes).
func runVaultikApp(
cmd *cobra.Command, mode lockMode, jsonOutput bool,
cmd *cobra.Command, extraQuiet, suppressErrors bool,
failMsg string, op func(v *vaultik.Vaultik) error,
) error {
configPath, err := ResolveConfigPath()
@@ -329,64 +214,75 @@ func runVaultikApp(
rootFlags := GetRootFlags()
return RunOperation(cmd.Context(), AppOptions{
return RunWithApp(cmd.Context(), AppOptions{
ConfigPath: configPath,
LogOptions: log.Options{
Verbose: rootFlags.Verbose,
Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet,
JSON: jsonOutput,
Quiet: rootFlags.Quiet || extraQuiet,
},
Mode: mode,
}, op, func(err error) {
Modules: []fx.Option{},
Invokes: []fx.Option{
fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
lc.Append(fx.Hook{
OnStart: func(_ context.Context) error {
go func() {
err := op(v)
if err != nil {
if !errors.Is(err, context.Canceled) {
if !suppressErrors {
log.Error(failMsg, "error", err)
ReportErrorf("%s: %v", failMsg, err)
}
os.Exit(1)
}
}
err = v.Shutdowner.Shutdown()
if err != nil {
log.Error("Failed to shutdown", "error", err)
}
}()
return nil
},
OnStop: func(_ context.Context) error {
v.Cancel()
return nil
},
})
}),
},
})
}
// RunWithApp is a helper that creates and runs an fx app with the given options.
// It combines NewApp and RunApp into a single convenient function. This is the
// preferred way to run CLI commands that need the full application context.
// A mutating command takes the process-wide PID lock before starting so that
// only one runs at a time; a read-only command runs without it and is not
// blocked while a mutator holds the lock (opts.Mode).
// It acquires a PID lock before starting to prevent concurrent instances.
func RunWithApp(ctx context.Context, opts AppOptions) error {
release, err := acquireLockIfMutating(opts.Mode,
filepath.Join(xdg.DataHome, "vaultik"))
// Acquire PID lock to prevent concurrent instances
lockDir := filepath.Join(xdg.DataHome, "vaultik")
lock, err := pidlock.Acquire(lockDir)
if err != nil {
return err
if errors.Is(err, pidlock.ErrAlreadyRunning) {
return fmt.Errorf("cannot start: %w", err)
}
defer release()
return fmt.Errorf("failed to acquire lock: %w", err)
}
defer func() {
err := lock.Release()
if err != nil {
log.Warn("Failed to release PID lock", "error", err)
}
}()
app := NewApp(opts)
return RunApp(ctx, app)
}
// acquireLockIfMutating takes the process-wide PID lock in lockDir for a
// mutating command and returns a function that releases it. A read-only
// command takes no lock, so it returns a no-op release and is never blocked
// while a mutator holds the lock. ErrAlreadyRunning (another mutator holds
// the lock) is surfaced as a "cannot start" error.
func acquireLockIfMutating(mode lockMode, lockDir string) (func(), error) {
if mode != mutating {
return func() {}, nil
}
lock, err := pidlock.Acquire(lockDir)
if err != nil {
if errors.Is(err, pidlock.ErrAlreadyRunning) {
return nil, fmt.Errorf("cannot start: %w", err)
}
return nil, fmt.Errorf("failed to acquire lock: %w", err)
}
return func() {
err := lock.Release()
if err != nil {
log.Warn("Failed to release PID lock", "error", err)
}
}, nil
}
internal/cli/app_test.go
+39 -88
View File
@@ -2,103 +2,54 @@ package cli //nolint:testpackage // needs access to unexported cleanStartupError
import (
"errors"
"os"
"path/filepath"
"testing"
"go.uber.org/fx"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/pidlock"
)
// TestCleanStartupError starts a real fx app whose config has a
// blob_size_limit below the minimum. fx puts annotations naming the
// functions it was building in front of the config error; only those
// may be removed, although the config error itself contains "): ".
func TestCleanStartupError(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "config.yml")
err := os.WriteFile(path, []byte("snapshots:\n"+
" test:\n"+
" paths: [/tmp/vaultik-test-source]\n"+
"storage_url: file:///tmp/vaultik-test-storage\n"+
"chunk_size: 10MB\n"+
"blob_size_limit: 30MB\n"), 0o600)
if err != nil {
t.Fatalf("write config: %v", err)
tests := []struct {
name string
in string
want string
}{
{
name: "real fx error chain",
in: `could not build arguments for function ` +
`"sneak.berlin/go/vaultik/internal/cli".newSnapshotCreateCommand.func1.1 ` +
`(/Users/user/dev/vaultik/internal/cli/snapshot.go:71): ` +
`failed to build *vaultik.Vaultik: ` +
`could not build arguments for function ` +
`"sneak.berlin/go/vaultik/internal/vaultik".New ` +
`(/Users/user/dev/vaultik/internal/vaultik/vaultik.go:59): ` +
`failed to build storage.Storer: ` +
`received non-nil error from function ` +
`"sneak.berlin/go/vaultik/internal/storage".NewStorer ` +
`(/Users/user/dev/vaultik/internal/storage/module.go:23): ` +
`creating base path: mkdir /Volumes/BACKUPS: permission denied`,
want: `creating base path: mkdir /Volumes/BACKUPS: permission denied`,
},
{
name: "no fx wrapping",
in: "plain error",
want: "plain error",
},
{
name: "single fx wrapping",
in: `received non-nil error from function "foo" (file.go:1): underlying problem`,
want: "underlying problem",
},
}
app := fx.New(
fx.NopLogger,
fx.Supply(config.Path(path)),
config.Module,
fx.Invoke(func(*config.Config) {}),
)
err = app.Start(t.Context())
if err == nil {
t.Fatal("app started; want a blob_size_limit error")
}
want := "failed to load config: invalid config: blob_size_limit must " +
"be at least the largest chunk the chunker can emit (chunk_size " +
"times the FastCDC size spread): need at least 40000000 bytes"
got := cleanStartupError(err).Error()
if got != want {
t.Errorf("got %q, want %q", got, want)
}
}
// TestCleanStartupErrorNotFromFx checks that an error fx did not wrap is
// shown unchanged.
func TestCleanStartupErrorNotFromFx(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
//nolint:err113 // a one-off error standing in for a startup failure
got := cleanStartupError(errors.New("plain error")).Error()
if got != "plain error" {
t.Errorf("got %q, want %q", got, "plain error")
//nolint:err113 // test constructs errors from table input
got := cleanStartupError(errors.New(tt.in)).Error()
if got != tt.want {
t.Errorf("got %q, want %q", got, tt.want)
}
})
}
}
// TestLockScopedToMutatingCommands proves the partition the PID lock now
// enforces: a read-only command runs while a mutator holds the lock, and
// two mutating commands still mutually exclude.
func TestLockScopedToMutatingCommands(t *testing.T) {
t.Parallel()
lockDir := filepath.Join(t.TempDir(), "vaultik")
// A mutating command takes the process-wide lock.
releaseMutator, err := acquireLockIfMutating(mutating, lockDir)
if err != nil {
t.Fatalf("mutating command could not acquire lock: %v", err)
}
// A read-only command runs to completion even while the lock is held.
releaseReader, err := acquireLockIfMutating(readOnly, lockDir)
if err != nil {
t.Fatalf("read-only command was blocked by held lock: %v", err)
}
releaseReader()
// A second mutating command is refused while the first holds the lock.
_, err = acquireLockIfMutating(mutating, lockDir)
if !errors.Is(err, pidlock.ErrAlreadyRunning) {
t.Fatalf("second mutating command was not excluded, got: %v", err)
}
// Once the first mutator releases, another mutating command may run.
releaseMutator()
release, err := acquireLockIfMutating(mutating, lockDir)
if err != nil {
t.Fatalf("mutating command could not acquire released lock: %v", err)
}
release()
}
internal/cli/config.go
+55 -291
View File
@@ -1,22 +1,16 @@
package cli
import (
"bytes"
"errors"
"fmt"
"os"
"os/exec"
"path/filepath"
"reflect"
"slices"
"strconv"
"strings"
"unicode/utf8"
"github.com/spf13/cobra"
"gopkg.in/yaml.v3"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/ui"
)
// configFileMode is the permission set for freshly written config files;
@@ -30,14 +24,6 @@ const configSetArgs = 2
// parent config dirs (e.g. ~/.config) are conventionally traversable.
const configDirMode = 0o755
// configYAMLIndent matches the 2-space indentation of defaultConfigTemplate,
// so `config set` writes the file back with the same indentation rather than
// yaml.Marshal's 4-space default.
const configYAMLIndent = 2
// yamlStringTag is YAML's tag for a string scalar.
const yamlStringTag = "!!str"
var (
errConfigExists = errors.New("config file already exists")
errEmptyConfig = errors.New("empty config file")
@@ -45,9 +31,6 @@ var (
errNeedNumericIndex = errors.New("key is a list; use a numeric index")
errIndexOutOfRange = errors.New("index out of range")
errNotMapOrList = errors.New("key is not a map or list")
errWouldNotLoad = errors.New(
"vaultik would not load the config after this change, " +
"so the file was not changed")
)
const defaultConfigTemplate = `# vaultik configuration
@@ -55,19 +38,13 @@ const defaultConfigTemplate = `# vaultik configuration
# ─── REQUIRED ────────────────────────────────────────────────────────────────
# Age recipient public keys for encryption. snapshot create needs at least
# one; listing, verifying and restoring do not, so a machine that only
# restores can leave this empty.
# Backups are encrypted to ALL listed recipients; any one of the corresponding
# private keys can decrypt. Adding a recipient later does not re-encrypt data
# already stored: deduplicated chunks and existing blobs stay encrypted to the
# earlier recipients, so a newly added key cannot restore them on its own (see
# docs/REPOSTRUCTURE.md, Accepted Risks). Generate a keypair and add its
# public key with:
# Age recipient public keys for encryption.
# Backups are encrypted to ALL listed recipients. Any one of the corresponding
# private keys can decrypt. Generate a keypair with:
# age-keygen -o vaultik_backup_private_key.txt
# grep 'public key' vaultik_backup_private_key.txt
# vaultik config set age_recipients.0 age1...
age_recipients: []
age_recipients:
- age1REPLACE_WITH_YOUR_PUBLIC_KEY
# Named snapshots. Each snapshot backs up one or more paths and can have its
# own exclude patterns in addition to the global excludes below.
@@ -209,8 +186,8 @@ storage_url: ""
# access_key_id: YOUR_ACCESS_KEY
# secret_access_key: YOUR_SECRET_KEY
# # region: us-east-1 # Default: us-east-1
# # part_size: 5MiB # Upload part size, 5MiB to 5GiB. Default: 5MiB
# # For the s3:// form, disable TLS with ?ssl=false in the URL, not use_ssl.
# # use_ssl: true # Default: true
# # part_size: 5MB # Multipart upload part size. Default: 5MB
# ─── OPTIONAL ────────────────────────────────────────────────────────────────
@@ -224,13 +201,11 @@ storage_url: ""
# Average chunk size for content-defined chunking (FastCDC).
# Smaller = better deduplication but more metadata overhead.
# Minimum 1MB. Accepts: 1MB, 10M, 64MiB, etc.
# Accepts: 1MB, 10M, 64KB, etc.
# Default: 10MB
# chunk_size: 10MB
# Maximum blob size before splitting into a new blob.
# Must be at least four times chunk_size (the largest chunk the chunker can
# emit); a smaller limit would let a single-chunk blob exceed it.
# Accepts: 1GB, 10G, 500MB, etc.
# Default: 10GB
# blob_size_limit: 10GB
@@ -239,18 +214,6 @@ storage_url: ""
# Default: 3
# compression_level: 3
# How old a blob that no snapshot lists must be before prune deletes it,
# and how old every object in a directory under metadata/ with no manifest
# must be before prune deletes that directory. A backup uploads its blobs
# and its db.zst.age before its manifest, so on a destination shared
# between hosts or users this must be longer than the longest backup run.
# It does not protect a blob a backup reuses from an earlier run that
# stopped before its manifest: that blob keeps the time of its first
# upload, so a prune can delete it.
# Accepts: 36h, 90m, etc. 0s deletes such blobs and directories at once.
# Default: 24h
# prune_grace_period: 24h
# Hostname used in snapshot IDs. Default: system hostname.
# hostname: myserver
@@ -267,12 +230,6 @@ func NewConfigCommand() *cobra.Command {
Use: "config",
Short: "Manage the configuration file",
Long: "Commands for creating, editing, and querying the vaultik config file.",
// cobra checks Args only on a command that has a Run: without
// one, a mistyped subcommand printed help and exited 0.
Args: cobra.NoArgs,
Run: func(cmd *cobra.Command, _ []string) {
_ = cmd.Help()
},
}
cmd.AddCommand(newConfigInitCommand())
@@ -289,16 +246,14 @@ func newConfigInitCommand() *cobra.Command {
Use: "init",
Short: "Write a default config file",
Long: `Creates a default configuration file with commented explanations
for the settings it lists. If a config file already exists at the target
path, the command refuses to overwrite it. The file is created with mode
0600, because it can hold credentials.
for every setting. If a config file already exists at the target path,
the command refuses to overwrite it.
The config is written to the path from --config, $VAULTIK_CONFIG, or
config.yml in the platform default config directory
(~/Library/Application Support/vaultik/ on macOS, ~/.config/vaultik/ on
Linux, /etc/vaultik/ as root).`,
the platform default config directory (e.g. ~/Library/Application Support/
on macOS, ~/.config/ on Linux, /etc/vaultik/ as root).`,
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
RunE: func(_ *cobra.Command, _ []string) error {
path := configPathForInit()
_, err := os.Stat(path)
@@ -318,11 +273,8 @@ Linux, /etc/vaultik/ as root).`,
return fmt.Errorf("writing config file: %w", err)
}
// A written-confirmation, not scriptable output: route it
// through the UI so it is styled and --quiet silences it.
out := commandUI(cmd)
out.Infof("Config written to %s.", path)
out.Infof(
_, _ = fmt.Fprintf(os.Stdout, "Config written to %s\n", path)
_, _ = fmt.Fprintln(os.Stdout,
"Edit it to set your age_recipients, snapshots, and storage_url.")
return nil
@@ -334,7 +286,7 @@ Linux, /etc/vaultik/ as root).`,
func newConfigEditCommand() *cobra.Command {
return &cobra.Command{
Use: "edit",
Short: "Open the config file in $EDITOR (vi when it is not set)",
Short: "Open the config file in $EDITOR",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
path, err := ResolveConfigPath()
@@ -347,15 +299,8 @@ func newConfigEditCommand() *cobra.Command {
editor = "vi"
}
// Run $EDITOR through the shell, as git does, so a value
// with arguments such as "code --wait" works. The path
// goes in as an argument ("$@"), not into the command
// text, so a path with spaces or quotes arrives whole.
// sh -c takes the word after the command as $0, which
// is why editor appears twice.
//nolint:gosec // G204: launching the operator's own $EDITOR is the point
ed := exec.CommandContext(cmd.Context(),
"sh", "-c", editor+` "$@"`, editor, path)
ed := exec.CommandContext(cmd.Context(), editor, path)
ed.Stdin = os.Stdin
ed.Stdout = os.Stdout
ed.Stderr = os.Stderr
@@ -371,7 +316,7 @@ func newConfigGetCommand() *cobra.Command {
Use: "get <key>",
Short: "Print a config value by dotted path (e.g. storage_url, compression_level)",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
RunE: func(_ *cobra.Command, args []string) error {
path, err := ResolveConfigPath()
if err != nil {
return err
@@ -387,13 +332,8 @@ func newConfigGetCommand() *cobra.Command {
return err
}
// The value is scriptable output: it must stay machine-plain
// (no marker, no color) and is never silenced by --quiet, so it
// is written straight to stdout rather than through the UI.
w := cmd.OutOrStdout()
if node.Kind == yaml.ScalarNode {
_, _ = fmt.Fprintln(w, node.Value)
_, _ = fmt.Fprintln(os.Stdout, node.Value)
return nil
}
@@ -403,7 +343,7 @@ func newConfigGetCommand() *cobra.Command {
return fmt.Errorf("marshaling value: %w", err)
}
_, _ = fmt.Fprint(w, string(out))
_, _ = fmt.Fprint(os.Stdout, string(out))
return nil
},
@@ -416,17 +356,8 @@ func newConfigSetCommand() *cobra.Command {
Use: "set <key> <value>",
Short: "Set a config value by dotted path (e.g. compression_level 5)",
Long: `Sets a scalar config value addressed by dotted YAML path and writes
the file back, preserving comments and formatting. Missing keys along
the path are created: as a list when the next path element is a
number, otherwise as a map.
If the config loads, a change after which it would not, such as
compression_level 99, is refused and the file is left unchanged. A
config that does not load, such as the one config init writes before
storage_url is set or one with a ${ENV:...} reference to a variable
not set here, is checked one setting at a time: only a change after
which the changed setting cannot be read, such as one giving it the
wrong type, is refused.
the file back, preserving comments and formatting. Intermediate maps
are created as needed.
Examples:
vaultik config set storage_url "file:///mnt/backups"
@@ -434,154 +365,44 @@ Examples:
vaultik config set compression_level 9
vaultik config set s3.bucket mybucket # legacy S3 fields still supported`,
Args: cobra.ExactArgs(configSetArgs),
RunE: func(cmd *cobra.Command, args []string) error {
RunE: func(_ *cobra.Command, args []string) error {
path, err := ResolveConfigPath()
if err != nil {
return err
}
return writeConfigSet(commandUI(cmd), path, args[0], args[1])
},
}
}
// writeConfigSet applies key=value to the config at path, writes it back
// owner-only, and confirms the write by naming just the key through the
// UI writer (styled, and silenced by --quiet). The value is never
// echoed: it may be a secret such as s3.secret_access_key, and captured
// stdout or a pasted terminal would then leak it.
func writeConfigSet(out *ui.Writer, path, key, value string) error {
root, err := loadYAMLFile(path)
if err != nil {
return err
}
oldData, err := marshalConfigYAML(root)
if err != nil {
return fmt.Errorf("marshaling config: %w", err)
}
keys := strings.Split(key, ".")
err = yamlPathSet(root, keys, value)
err = yamlPathSet(root, strings.Split(args[0], "."), args[1])
if err != nil {
return err
}
data, err := marshalConfigYAML(root)
out, err := yaml.Marshal(root)
if err != nil {
return fmt.Errorf("marshaling config: %w", err)
}
setting, err := yamlPathOnly(root, keys)
if err != nil {
return err
mode := os.FileMode(configFileMode)
info, statErr := os.Stat(path)
if statErr == nil {
mode = info.Mode().Perm()
}
settingData, err := marshalConfigYAML(setting)
if err != nil {
return fmt.Errorf("marshaling config: %w", err)
}
err = checkConfigChange(oldData, data, settingData)
if err != nil {
return fmt.Errorf("%w: %w", errWouldNotLoad, err)
}
err = os.WriteFile(path, data, configFileMode)
err = os.WriteFile(path, out, mode)
if err != nil {
return fmt.Errorf("writing config file: %w", err)
}
// os.WriteFile does not change the mode of a file that already exists,
// so a config that was group- or world-readable stays that way. As it
// may hold S3 credentials, tighten it to owner-only after writing.
info, statErr := os.Stat(path)
if statErr == nil && info.Mode().Perm()&0o044 != 0 {
err = os.Chmod(path, configFileMode)
if err != nil {
return fmt.Errorf("tightening config file permissions: %w", err)
}
}
out.Infof("Set %s.", key)
_, _ = fmt.Fprintf(os.Stdout, "%s = %s\n", args[0], args[1])
return nil
},
}
// checkConfigChange returns the reason vaultik would not load newData, the
// config after a change, if the change brings that failure in. If oldData,
// the config before it, loads, the new one must load too. A config that
// does not load, such as one still being filled in or one with a ${...}
// reference that cannot be filled in here, is checked one setting at a
// time: settingData, the changed setting on its own, must decode into the
// config type.
func checkConfigChange(oldData, newData, settingData []byte) error {
oldCfg, err := config.Parse(oldData)
if err == nil && oldCfg.Validate() == nil {
newCfg, err := config.Parse(newData)
if err != nil {
return err
}
return newCfg.Validate()
}
_, err = config.Parse(settingData)
return err
}
// yamlPathOnly returns a document holding only the value at the dotted key
// path in root, inside the maps and lists that lead to it there.
func yamlPathOnly(root *yaml.Node, keys []string) (*yaml.Node, error) {
node, err := yamlPathGet(root, keys)
if err != nil {
return nil, err
}
for i, key := range slices.Backward(keys) {
parent, err := yamlPathGet(root, keys[:i])
if err != nil {
return nil, err
}
if parent.Kind == yaml.SequenceNode {
node = &yaml.Node{
Kind: yaml.SequenceNode, Content: []*yaml.Node{node},
}
} else {
keyNode := &yaml.Node{Kind: yaml.ScalarNode, Value: key}
node = &yaml.Node{
Kind: yaml.MappingNode, Content: []*yaml.Node{keyNode, node},
}
}
}
return &yaml.Node{Kind: yaml.DocumentNode, Content: []*yaml.Node{node}}, nil
}
// marshalConfigYAML renders a config document tree with 2-space indentation,
// matching defaultConfigTemplate. yaml.Marshal defaults to 4 spaces, which
// would reindent the whole file on the first `config set` despite the promise
// to preserve formatting.
func marshalConfigYAML(root *yaml.Node) ([]byte, error) {
var buf bytes.Buffer
enc := yaml.NewEncoder(&buf)
enc.SetIndent(configYAMLIndent)
err := enc.Encode(root)
if err != nil {
return nil, err
}
err = enc.Close()
if err != nil {
return nil, err
}
return buf.Bytes(), nil
}
// loadYAMLFile parses a YAML file into a yaml.Node document tree,
@@ -667,8 +488,8 @@ func yamlPathGet(root *yaml.Node, keys []string) (*yaml.Node, error) {
return node, nil
}
// yamlPathSet navigates a dotted key path, creating missing keys as needed,
// and sets the final key to the given scalar value. Numeric path
// yamlPathSet navigates a dotted key path, creating intermediate maps as
// needed, and sets the final key to the given scalar value. Numeric path
// components index into sequences; an index equal to the sequence length
// appends a new element (e.g. "age_recipients.1" on a 1-element list).
func yamlPathSet(root *yaml.Node, keys []string, value string) error {
@@ -681,12 +502,12 @@ func yamlPathSet(root *yaml.Node, keys []string, value string) error {
node = node.Content[0]
}
for i := range keys {
for i, key := range keys {
last := i == len(keys)-1
switch node.Kind {
case yaml.MappingNode:
node = yamlSetInMapping(node, keys, i, value, last)
node = yamlSetInMapping(node, key, value, last)
case yaml.SequenceNode:
next, err := yamlSetInSequence(node, keys, i, value, last)
if err != nil {
@@ -703,68 +524,17 @@ func yamlPathSet(root *yaml.Node, keys []string, value string) error {
}
}
// config.Load reads the file through untyped YAML, which turns an
// unquoted 00112233 into the number 38043 and 1e5 into 100000. Tagging
// a string setting as a string makes the encoder quote such a value.
// Other settings stay unquoted, so compression_level 9 is a number.
// The encoder refuses to write a value that is not valid UTF-8 as a
// string. Left untagged, such a value is written as base64 !!binary and
// loads back unchanged.
if configKeyIsString(keys) && utf8.ValidString(value) {
node.Tag = yamlStringTag
}
return nil
}
// configKeyIsString reports whether the dotted key names a string in
// config.Config, following the fields' yaml tags, as s3.access_key_id and
// snapshots.home.exclude.0 do.
func configKeyIsString(keys []string) bool {
typ := reflect.TypeFor[config.Config]()
for _, key := range keys {
switch {
case typ.Kind() == reflect.Map || typ.Kind() == reflect.Slice:
// The key is a snapshot name or a list index.
typ = typ.Elem()
case typ.Kind() == reflect.Struct:
field, ok := yamlField(typ, key)
if !ok {
return false
}
typ = field.Type
default:
return false
}
}
return typ.Kind() == reflect.String
}
// yamlField returns the field of struct type typ whose yaml tag names key.
func yamlField(typ reflect.Type, key string) (reflect.StructField, bool) {
for field := range typ.Fields() {
name, _, _ := strings.Cut(field.Tag.Get("yaml"), ",")
if name == key {
return field, true
}
}
return reflect.StructField{}, false
}
// yamlSetInMapping resolves (creating if needed) the value node for the
// path element keys[i] within a mapping node, setting it to value when it
// is the final path element, and returns the node to descend into.
func yamlSetInMapping(
node *yaml.Node, keys []string, i int, value string, last bool,
) *yaml.Node {
// yamlSetInMapping resolves (creating if needed) the value node for key
// within a mapping node, setting it to value when it is the final path
// element, and returns the node to descend into.
func yamlSetInMapping(node *yaml.Node, key, value string, last bool) *yaml.Node {
var valueNode *yaml.Node
for j := 0; j+1 < len(node.Content); j += 2 {
if node.Content[j].Value == keys[i] {
if node.Content[j].Value == key {
valueNode = node.Content[j+1]
break
@@ -772,8 +542,13 @@ func yamlSetInMapping(
}
if valueNode == nil {
keyNode := &yaml.Node{Kind: yaml.ScalarNode, Value: keys[i]}
valueNode = newPathNode(keys, i, value)
keyNode := &yaml.Node{Kind: yaml.ScalarNode, Value: key}
valueNode = &yaml.Node{Kind: yaml.MappingNode}
if last {
valueNode = &yaml.Node{Kind: yaml.ScalarNode, Value: value}
}
node.Content = append(node.Content, keyNode, valueNode)
} else if last {
setScalar(valueNode, value)
@@ -801,7 +576,12 @@ func yamlSetInSequence(
}
if idx == len(node.Content) {
node.Content = append(node.Content, newPathNode(keys, i, value))
newNode := &yaml.Node{Kind: yaml.MappingNode}
if last {
newNode = &yaml.Node{Kind: yaml.ScalarNode, Value: value}
}
node.Content = append(node.Content, newNode)
} else if last {
setScalar(node.Content[idx], value)
}
@@ -809,22 +589,6 @@ func yamlSetInSequence(
return node.Content[idx], nil
}
// newPathNode returns the node to create for the missing path element
// keys[i]: the value when it is the final path element, otherwise an empty
// list when the next path element is a number and an empty map when not.
func newPathNode(keys []string, i int, value string) *yaml.Node {
if i == len(keys)-1 {
return &yaml.Node{Kind: yaml.ScalarNode, Value: value}
}
_, err := strconv.Atoi(keys[i+1])
if err == nil {
return &yaml.Node{Kind: yaml.SequenceNode}
}
return &yaml.Node{Kind: yaml.MappingNode}
}
// setScalar overwrites a node in place with a plain scalar value.
func setScalar(n *yaml.Node, value string) {
n.Kind = yaml.ScalarNode
internal/cli/config_test.go
+2 -485
View File
@@ -1,16 +1,11 @@
package cli //nolint:testpackage // exercises unexported yamlPathGet/yamlPathSet
import (
"bytes"
"os"
"path/filepath"
"strconv"
"strings"
"testing"
"gopkg.in/yaml.v3"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/ui"
)
// TestDefaultConfigTemplateParses ensures the init template is valid YAML
@@ -25,10 +20,8 @@ func TestDefaultConfigTemplateParses(t *testing.T) {
t.Fatalf("default config template is not valid YAML: %v", err)
}
// A placeholder recipient would fail config.Load, so the template
// leaves the list empty.
if len(cfg.AgeRecipients) != 0 {
t.Errorf("expected no age recipients, got %d", len(cfg.AgeRecipients))
if len(cfg.AgeRecipients) != 1 {
t.Errorf("expected 1 placeholder age recipient, got %d", len(cfg.AgeRecipients))
}
home, ok := cfg.Snapshots["home"]
@@ -58,353 +51,6 @@ func TestDefaultConfigTemplateParses(t *testing.T) {
}
}
// testRecipient is a valid age public key.
const testRecipient = "age1278m9q7dp3chsh2dcy82qk27v047zywyvtxwnj4cvt0z65jw6a7q5dqhfj"
// TestConfigSetRecipientOnFreshConfig follows the README's getting started
// steps: on the file `config init` writes, `config set age_recipients.0` and
// `config set storage_url` give a config that loads with that recipient.
// Neither is refused, although the config does not load until both are set;
// once it loads, a recipient that is not a valid key is.
func TestConfigSetRecipientOnFreshConfig(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "config.yml")
err := os.WriteFile(path, []byte(defaultConfigTemplate), configFileMode)
if err != nil {
t.Fatalf("write config: %v", err)
}
out := ui.NewWithColor(&bytes.Buffer{}, false)
err = writeConfigSet(out, path, "age_recipients.0", testRecipient)
if err != nil {
t.Fatalf("config set age_recipients.0: %v", err)
}
err = writeConfigSet(out, path, "storage_url", "file:///mnt/backups")
if err != nil {
t.Fatalf("config set storage_url: %v", err)
}
cfg, err := config.Load(path)
if err != nil {
t.Fatalf("config.Load: %v", err)
}
if len(cfg.AgeRecipients) != 1 || cfg.AgeRecipients[0] != testRecipient {
t.Errorf("age_recipients = %v, want [%s]", cfg.AgeRecipients, testRecipient)
}
err = writeConfigSet(out, path, "age_recipients.0", "notakey")
if err == nil || !strings.Contains(err.Error(), "not a valid recipient") {
t.Errorf("config set age_recipients.0 notakey gives error %v, "+
"want one saying it is not a valid recipient", err)
}
_, err = config.Load(path)
if err != nil {
t.Errorf("config.Load after the refused change: %v", err)
}
}
// configWithoutRecipients loads, but has no age_recipients key and no
// snapshot named work.
const configWithoutRecipients = `storage_url: file:///mnt/backups
snapshots:
home:
paths:
- "~"
`
// configWithUnsetReference loads only where VAULTIK_TEST_UNSET_SECRET is
// set, as a backup job's environment might set it. No test sets it.
const configWithUnsetReference = configWithoutRecipients + `s3:
secret_access_key: ${ENV:VAULTIK_TEST_UNSET_SECRET}
`
// configWithLevelReference returns configWithoutRecipients with a ${FILE:...}
// reference on compression_level to a file holding 9, so that the config
// loads once the reference is filled in.
func configWithLevelReference(t *testing.T) string {
t.Helper()
levelPath := filepath.Join(t.TempDir(), "level")
err := os.WriteFile(levelPath, []byte("9"), configFileMode)
if err != nil {
t.Fatalf("write level: %v", err)
}
return configWithoutRecipients +
"compression_level: ${FILE:" + levelPath + "}\n"
}
// TestConfigSetIndexUnderMissingKey checks that a list index under a key
// the file does not have creates a list, and that the result loads.
func TestConfigSetIndexUnderMissingKey(t *testing.T) {
t.Parallel()
tests := []struct {
key string
value string
field func(cfg *config.Config) []string
}{
{"age_recipients.0", testRecipient,
func(cfg *config.Config) []string { return cfg.AgeRecipients }},
{"snapshots.work.paths.0", "/srv/work",
func(cfg *config.Config) []string { return cfg.Snapshots["work"].Paths }},
}
for _, tt := range tests {
t.Run(tt.key, func(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "config.yml")
err := os.WriteFile(path, []byte(configWithoutRecipients), configFileMode)
if err != nil {
t.Fatalf("write config: %v", err)
}
err = writeConfigSet(ui.NewWithColor(&bytes.Buffer{}, false),
path, tt.key, tt.value)
if err != nil {
t.Fatalf("config set %s: %v", tt.key, err)
}
cfg, err := config.Load(path)
if err != nil {
t.Fatalf("config.Load: %v", err)
}
got := tt.field(cfg)
if len(got) != 1 || got[0] != tt.value {
t.Errorf("%s gives %v, want [%s]", tt.key, got, tt.value)
}
})
}
}
// TestConfigSetKeepsFileThatWouldNotLoad checks that config set refuses a
// change after which the config would not load, with an error giving the
// reason, and leaves the file as it was. On the file `config init` writes,
// which does not load until storage_url is set, and on one that cannot be
// read here, only a value of the wrong type is refused.
func TestConfigSetKeepsFileThatWouldNotLoad(t *testing.T) {
t.Parallel()
levelReference := configWithLevelReference(t)
tests := []struct {
config string
key string
value string
reason string
}{
// Index 1 of the empty list the missing key gives.
{configWithoutRecipients, "age_recipients.1", testRecipient,
"index out of range"},
// A list set to a single value.
{configWithoutRecipients, "snapshots.home.paths", "/srv",
"into []string"},
{configWithoutRecipients, "chunk_size", "fast",
"invalid size format"},
// Values of the right type that fail validation.
{configWithoutRecipients, "compression_level", "99",
"compression_level must be between 1 and 19"},
{configWithoutRecipients, "chunk_size", "1KB",
"chunk_size must be at least 1MB"},
// With its reference filled in, the config loads.
{levelReference, "blob_size_limit", "1MB",
"blob_size_limit must be at least"},
// The type check on a config that does not load yet.
{defaultConfigTemplate, "snapshots.apps.paths", "/srv",
"into []string"},
// The type check on a config that cannot be read here.
{configWithUnsetReference, "snapshots.home.paths", "/x",
"cannot unmarshal !!str `/x`"},
}
for _, tt := range tests {
t.Run(tt.key+" "+tt.value, func(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "config.yml")
err := os.WriteFile(path, []byte(tt.config), configFileMode)
if err != nil {
t.Fatalf("write config: %v", err)
}
err = writeConfigSet(ui.NewWithColor(&bytes.Buffer{}, false),
path, tt.key, tt.value)
if err == nil || !strings.Contains(err.Error(), tt.reason) {
t.Errorf("config set %s %s gives error %v, want one saying %q",
tt.key, tt.value, err, tt.reason)
}
data, err := os.ReadFile(path) //nolint:gosec // G304: test-controlled path
if err != nil {
t.Fatalf("read config: %v", err)
}
if string(data) != tt.config {
t.Errorf("config set %s changed the file to:\n%s",
tt.key, data)
}
})
}
}
// TestConfigSetRefusesOnlyItsOwnFailure checks that setting another key
// works on a config with a ${...} reference on compression_level, which
// loads because loading fills the reference in, on one whose
// compression_level already fails to load, and on one with a reference
// that cannot be filled in here.
func TestConfigSetRefusesOnlyItsOwnFailure(t *testing.T) {
t.Parallel()
configs := map[string]string{
"reference": configWithLevelReference(t),
"failing": configWithoutRecipients + "compression_level: fast\n",
"unset reference": configWithUnsetReference,
}
for name, configYAML := range configs {
t.Run(name, func(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "config.yml")
err := os.WriteFile(path, []byte(configYAML), configFileMode)
if err != nil {
t.Fatalf("write config: %v", err)
}
err = writeConfigSet(ui.NewWithColor(&bytes.Buffer{}, false),
path, "hostname", "myhost")
if err != nil {
t.Fatalf("config set hostname: %v", err)
}
data, err := os.ReadFile(path) //nolint:gosec // G304: test-controlled path
if err != nil {
t.Fatalf("read config: %v", err)
}
if !strings.Contains(string(data), "hostname: myhost") {
t.Errorf("config set hostname wrote:\n%s", data)
}
})
}
}
// TestConfigSetStringLooksLikeNumber sets string settings to values that
// YAML reads as numbers or booleans when they are unquoted, and checks that
// config.Load returns each one unchanged.
func TestConfigSetStringLooksLikeNumber(t *testing.T) {
t.Parallel()
tests := []struct {
key string
value string
field func(cfg *config.Config) string
}{
{"s3.access_key_id", "00112233",
func(cfg *config.Config) string { return cfg.S3.AccessKeyID }},
{"s3.secret_access_key", "12345678901234567890123456789012",
func(cfg *config.Config) string { return cfg.S3.SecretAccessKey }},
{"hostname", "007",
func(cfg *config.Config) string { return cfg.Hostname }},
{"s3.prefix", "1e5",
func(cfg *config.Config) string { return cfg.S3.Prefix }},
{"s3.bucket", "true",
func(cfg *config.Config) string { return cfg.S3.Bucket }},
{"s3.region", "FALSE",
func(cfg *config.Config) string { return cfg.S3.Region }},
{"snapshots.home.exclude.0", "1.10",
func(cfg *config.Config) string { return cfg.Snapshots["home"].Exclude[0] }},
}
for _, tt := range tests {
t.Run(tt.key+"="+tt.value, func(t *testing.T) {
t.Parallel()
cfg := loadAfterConfigSet(t, tt.key, tt.value)
got := tt.field(cfg)
if got != tt.value {
t.Errorf("%s = %q after config set %q", tt.key, got, tt.value)
}
})
}
}
// TestConfigSetNonUTF8Path checks that config set still accepts a value that
// is not valid UTF-8, such as a path with a Latin-1 file name, and that
// config.Load returns it unchanged.
func TestConfigSetNonUTF8Path(t *testing.T) {
t.Parallel()
const dir = "/srv/caf\xe9"
cfg := loadAfterConfigSet(t, "snapshots.home.paths.0", dir)
got := cfg.Snapshots["home"].Paths[0]
if got != dir {
t.Errorf("snapshots.home.paths.0 = %q, want %q", got, dir)
}
}
// TestConfigSetNumberStaysNumber checks that a number set for an integer
// setting is still read as a number, not as a quoted string.
func TestConfigSetNumberStaysNumber(t *testing.T) {
t.Parallel()
const level = 9
cfg := loadAfterConfigSet(t, "compression_level", strconv.Itoa(level))
if cfg.CompressionLevel != level {
t.Errorf("compression_level = %d, want %d", cfg.CompressionLevel, level)
}
}
// loadAfterConfigSet writes the file `config init` writes, sets storage_url
// to a local directory so that the file passes validation, applies
// `config set key value` and returns what config.Load reads back.
func loadAfterConfigSet(t *testing.T, key, value string) *config.Config {
t.Helper()
path := filepath.Join(t.TempDir(), "config.yml")
err := os.WriteFile(path, []byte(defaultConfigTemplate), configFileMode)
if err != nil {
t.Fatalf("write config: %v", err)
}
out := ui.NewWithColor(&bytes.Buffer{}, false)
err = writeConfigSet(out, path, "storage_url", "file:///mnt/backups")
if err != nil {
t.Fatalf("config set storage_url: %v", err)
}
err = writeConfigSet(out, path, key, value)
if err != nil {
t.Fatalf("config set %s: %v", key, err)
}
cfg, err := config.Load(path)
if err != nil {
t.Fatalf("config.Load: %v", err)
}
return cfg
}
const testYAML = `# top comment
compression_level: 3
age_recipients:
@@ -542,135 +188,6 @@ func TestYAMLPathSet(t *testing.T) {
}
}
// TestConfigSetPreservesFormatting asserts the `config set` write path
// (marshalConfigYAML) round-trips a 2-space-indented file without reindenting
// it to yaml.Marshal's 4-space default, and keeps comments.
func TestConfigSetPreservesFormatting(t *testing.T) {
t.Parallel()
root := parseTestYAML(t)
err := yamlPathSet(root, splitPath("s3.bucket"), "newbucket")
if err != nil {
t.Fatalf("set s3.bucket: %v", err)
}
out, err := marshalConfigYAML(root)
if err != nil {
t.Fatalf("marshal: %v", err)
}
text := string(out)
for _, want := range []string{"# top comment", "# inline comment"} {
if !contains(text, want) {
t.Errorf("round-tripped YAML dropped comment %q:\n%s", want, text)
}
}
// Nested map keys stay at 2-space indent; the bug reindented them to 4.
if !contains(text, "\n bucket: newbucket") {
t.Errorf("expected 2-space indent for s3.bucket, got:\n%s", text)
}
if contains(text, "\n bucket:") {
t.Errorf("s3.bucket reindented to 4 spaces:\n%s", text)
}
// Sequence items under a key also stay at 2 spaces.
if !contains(text, "\n - age1aaa") {
t.Errorf("expected 2-space indent for sequence item, got:\n%s", text)
}
}
// TestWriteConfigSetHidesSecret checks that setting a secret key prints
// only the key name, never the value, to the confirmation output.
func TestWriteConfigSetHidesSecret(t *testing.T) {
t.Parallel()
const secret = "SUPERSECRETVALUE"
path := filepath.Join(t.TempDir(), "config.yaml")
err := os.WriteFile(path, []byte("version: 1\n"), 0o600)
if err != nil {
t.Fatalf("seed config: %v", err)
}
var buf bytes.Buffer
err = writeConfigSet(ui.NewWithColor(&buf, false), path,
"s3.secret_access_key", secret)
if err != nil {
t.Fatalf("writeConfigSet: %v", err)
}
if strings.Contains(buf.String(), secret) {
t.Errorf("output echoed the secret value: %q", buf.String())
}
if !strings.Contains(buf.String(), "s3.secret_access_key") {
t.Errorf("output did not confirm the key name: %q", buf.String())
}
}
// TestWriteConfigSetTightensMode checks that a pre-existing group- or
// world-readable config is tightened to owner-only after a set, since
// os.WriteFile leaves an existing file's mode untouched.
func TestWriteConfigSetTightensMode(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "config.yaml")
// Seed a world-readable config; the loose mode is the condition under
// test, so gosec's G306 is expected here.
err := os.WriteFile(path, []byte("version: 1\n"), 0o644) //nolint:gosec // G306
if err != nil {
t.Fatalf("seed config: %v", err)
}
var buf bytes.Buffer
err = writeConfigSet(ui.NewWithColor(&buf, false), path,
"compression_level", "9")
if err != nil {
t.Fatalf("writeConfigSet: %v", err)
}
info, err := os.Stat(path)
if err != nil {
t.Fatalf("stat config: %v", err)
}
if info.Mode().Perm() != 0o600 {
t.Errorf("config mode = %04o, want 0600", info.Mode().Perm())
}
}
// TestConfigEditEditorWithArgument checks that an $EDITOR carrying an
// argument works and that a config path with a space reaches it whole.
// The "editor" is cp with the replacement file as its argument, so the
// config ends up with the replacement's content only if both arrived.
func TestConfigEditEditorWithArgument(t *testing.T) {
dir := t.TempDir()
path := seedFile(t, dir, "my config.yml", "compression_level: 3\n")
edited := seedFile(t, dir, "edited.yml", "compression_level: 9\n")
setRootFlags(t, RootFlags{ConfigPath: path})
t.Setenv("EDITOR", "cp '"+edited+"'")
mustExecute(t, newConfigEditCommand())
data, err := os.ReadFile(path) //nolint:gosec // G304: test-controlled path
if err != nil {
t.Fatalf("reading config back: %v", err)
}
if string(data) != "compression_level: 9\n" {
t.Errorf("config after edit = %q, want the edited content", data)
}
}
func splitPath(s string) []string {
return strings.Split(s, ".")
}
internal/cli/database.go
+11 -18
View File
@@ -15,12 +15,6 @@ func NewDatabaseCommand() *cobra.Command {
Use: "database",
Short: "Manage the local state database",
Long: `Commands for managing the local SQLite state database.`,
// cobra checks Args only on a command that has a Run: without
// one, a mistyped subcommand printed help and exited 0.
Args: cobra.NoArgs,
Run: func(cmd *cobra.Command, _ []string) {
_ = cmd.Help()
},
}
cmd.AddCommand(
@@ -54,7 +48,7 @@ storage destination on that run.
Use --force to skip the confirmation prompt.`,
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
RunE: func(_ *cobra.Command, _ []string) error {
// Resolve config path
configPath, err := ResolveConfigPath()
if err != nil {
@@ -68,31 +62,26 @@ Use --force to skip the confirmation prompt.`,
}
dbPath := cfg.IndexPath
out := commandUI(cmd)
// Check if database exists
_, err = os.Stat(dbPath)
if os.IsNotExist(err) {
out.Infof("Local state database does not exist: %s.", dbPath)
_, _ = fmt.Fprintf(os.Stdout, "Database does not exist: %s\n", dbPath)
return nil
}
// Confirm unless --force. The prompt and its immediate result
// are an interactive exchange the operator must see, so they go
// straight to stdout rather than through the UI and --quiet does
// not silence them.
// Confirm unless --force
if !force {
w := cmd.OutOrStdout()
_, _ = fmt.Fprintf(w,
_, _ = fmt.Fprintf(os.Stdout,
"This will delete the local state database at:\n %s\n\n", dbPath)
_, _ = fmt.Fprint(w, "Are you sure? Type 'yes' to confirm: ")
_, _ = fmt.Fprint(os.Stdout, "Are you sure? Type 'yes' to confirm: ")
var confirm string
_, err = fmt.Scanln(&confirm)
if err != nil || confirm != "yes" {
_, _ = fmt.Fprintln(w, "Aborted.")
_, _ = fmt.Fprintln(os.Stdout, "Aborted.")
//nolint:nilerr // a failed/aborted confirmation is a clean abort
return nil
@@ -111,7 +100,11 @@ Use --force to skip the confirmation prompt.`,
_ = os.Remove(walPath) // Ignore errors - files may not exist
_ = os.Remove(shmPath)
out.Infof("Local state database deleted: %s.", dbPath)
rootFlags := GetRootFlags()
if !rootFlags.Quiet {
_, _ = fmt.Fprintf(os.Stdout, "Database deleted: %s\n", dbPath)
}
log.Info("Local state database deleted", "path", dbPath)
return nil
internal/cli/duration.go
+126
View File
@@ -0,0 +1,126 @@
package cli
import (
"errors"
"fmt"
"regexp"
"strconv"
"strings"
"time"
)
// Approximate lengths of the extended calendar units accepted by
// parseDuration.
const (
durationDay = 24 * time.Hour
durationWeek = 7 * durationDay
durationMonth = 30 * durationDay
durationYear = 365 * durationDay
)
var (
errNegativeDuration = errors.New("negative durations are not supported")
errInvalidDuration = errors.New("invalid duration format")
errUnknownTimeUnit = errors.New("unknown time unit")
)
// parseDuration parses duration strings. Supports standard Go duration format
// (e.g., "3h30m", "1h45m30s") as well as extended units:
// - d: days (e.g., "30d", "7d")
// - w: weeks (e.g., "2w", "4w")
// - mo: months (30 days) (e.g., "6mo", "1mo")
// - y: years (365 days) (e.g., "1y", "2y")
//
// Can combine units: "1y6mo", "2w3d", "1d12h30m"
func parseDuration(s string) (time.Duration, error) {
// First try standard Go duration parsing
d, err := time.ParseDuration(s)
if err == nil {
return d, nil
}
// Extended duration parsing
// Check for negative values
if strings.HasPrefix(strings.TrimSpace(s), "-") {
return 0, errNegativeDuration
}
// Pattern matches: number + unit, repeated
re := regexp.MustCompile(`(\d+(?:\.\d+)?)\s*([a-zA-Z]+)`)
matches := re.FindAllStringSubmatch(s, -1)
if len(matches) == 0 {
return 0, fmt.Errorf("%w: %q", errInvalidDuration, s)
}
var total time.Duration
for _, match := range matches {
valueStr := match[1]
unit := strings.ToLower(match[2])
value, err := strconv.ParseFloat(valueStr, 64)
if err != nil {
return 0, fmt.Errorf("invalid number %q: %w", valueStr, err)
}
d, err := durationForUnit(value, unit)
if err != nil {
return 0, err
}
total += d
}
return total, nil
}
// durationForUnit converts a value with a (case-normalized) unit suffix
// into a time.Duration, accepting Go's standard units plus the extended
// calendar units.
func durationForUnit(value float64, unit string) (time.Duration, error) {
switch unit {
// Standard time units
case "ns", "nanosecond", "nanoseconds":
return time.Duration(value), nil
case "us", "µs", "microsecond", "microseconds":
return time.Duration(value * float64(time.Microsecond)), nil
case "ms", "millisecond", "milliseconds":
return time.Duration(value * float64(time.Millisecond)), nil
case "s", "sec", "second", "seconds":
return time.Duration(value * float64(time.Second)), nil
case "m", "min", "minute", "minutes":
return time.Duration(value * float64(time.Minute)), nil
case "h", "hr", "hour", "hours":
return time.Duration(value * float64(time.Hour)), nil
// Extended units
case "d", "day", "days":
return time.Duration(value * float64(durationDay)), nil
case "w", "week", "weeks":
return time.Duration(value * float64(durationWeek)), nil
case "mo", "month", "months":
// Using 30 days as approximation
return time.Duration(value * float64(durationMonth)), nil
case "y", "year", "years":
// Using 365 days as approximation
return time.Duration(value * float64(durationYear)), nil
default:
// Try parsing as standard Go duration unit
testStr := "1" + unit
_, err := time.ParseDuration(testStr)
if err != nil {
return 0, fmt.Errorf("%w: %q", errUnknownTimeUnit, unit)
}
// It's a valid Go duration unit, parse the full value
fullStr := fmt.Sprintf("%g%s", value, unit)
d, err := time.ParseDuration(fullStr)
if err != nil {
return 0, fmt.Errorf("invalid duration %q: %w", fullStr, err)
}
return d, nil
}
}
internal/cli/duration_test.go
+299
View File
@@ -0,0 +1,299 @@
package cli //nolint:testpackage // needs access to unexported parseDuration
import (
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
type parseDurationCase struct {
name string
input string
expected time.Duration
wantErr bool
}
// runParseDurationCases executes a table of parseDuration cases as
// parallel subtests.
func runParseDurationCases(t *testing.T, tests []parseDurationCase) {
t.Helper()
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
got, err := parseDuration(tt.input)
if tt.wantErr {
require.Error(t, err, "expected error for input %q", tt.input)
return
}
require.NoError(t, err, "unexpected error for input %q", tt.input)
assert.Equal(t, tt.expected, got, "duration mismatch for input %q", tt.input)
})
}
}
func TestParseDurationStandard(t *testing.T) {
t.Parallel()
runParseDurationCases(t, []parseDurationCase{
{
name: "standard seconds",
input: "30s",
expected: 30 * time.Second,
},
{
name: "standard minutes",
input: "45m",
expected: 45 * time.Minute,
},
{
name: "standard hours",
input: "2h",
expected: 2 * time.Hour,
},
{
name: "standard combined",
input: "3h30m",
expected: 3*time.Hour + 30*time.Minute,
},
{
name: "standard complex",
input: "1h45m30s",
expected: 1*time.Hour + 45*time.Minute + 30*time.Second,
},
{
name: "standard with milliseconds",
input: "1s500ms",
expected: 1*time.Second + 500*time.Millisecond,
},
})
}
func TestParseDurationExtendedUnits(t *testing.T) {
t.Parallel()
runParseDurationCases(t, []parseDurationCase{
// Extended units - days
{
name: "single day",
input: "1d",
expected: 24 * time.Hour,
},
{
name: "multiple days",
input: "7d",
expected: 7 * 24 * time.Hour,
},
{
name: "fractional days",
input: "1.5d",
expected: 36 * time.Hour,
},
{
name: "days spelled out",
input: "3days",
expected: 3 * 24 * time.Hour,
},
// Extended units - weeks
{
name: "single week",
input: "1w",
expected: 7 * 24 * time.Hour,
},
{
name: "multiple weeks",
input: "4w",
expected: 4 * 7 * 24 * time.Hour,
},
{
name: "weeks spelled out",
input: "2weeks",
expected: 2 * 7 * 24 * time.Hour,
},
// Extended units - months
{
name: "single month",
input: "1mo",
expected: 30 * 24 * time.Hour,
},
{
name: "multiple months",
input: "6mo",
expected: 6 * 30 * 24 * time.Hour,
},
{
name: "months spelled out",
input: "3months",
expected: 3 * 30 * 24 * time.Hour,
},
// Extended units - years
{
name: "single year",
input: "1y",
expected: 365 * 24 * time.Hour,
},
{
name: "multiple years",
input: "2y",
expected: 2 * 365 * 24 * time.Hour,
},
{
name: "years spelled out",
input: "1year",
expected: 365 * 24 * time.Hour,
},
})
}
func TestParseDurationCombinedAndErrors(t *testing.T) {
t.Parallel()
runParseDurationCases(t, []parseDurationCase{
// Combined extended units
{
name: "weeks and days",
input: "2w3d",
expected: 2*7*24*time.Hour + 3*24*time.Hour,
},
{
name: "years and months",
input: "1y6mo",
expected: 365*24*time.Hour + 6*30*24*time.Hour,
},
{
name: "days and hours",
input: "1d12h",
expected: 24*time.Hour + 12*time.Hour,
},
{
name: "complex combination",
input: "1y2mo3w4d5h6m7s",
expected: 365*24*time.Hour + 2*30*24*time.Hour +
3*7*24*time.Hour + 4*24*time.Hour +
5*time.Hour + 6*time.Minute + 7*time.Second,
},
{
name: "with spaces",
input: "1d 12h 30m",
expected: 24*time.Hour + 12*time.Hour + 30*time.Minute,
},
// Edge cases
{
name: "zero duration",
input: "0s",
expected: 0,
},
{
name: "large duration",
input: "10y",
expected: 10 * 365 * 24 * time.Hour,
},
// Error cases
{
name: "empty string",
input: "",
wantErr: true,
},
{
name: "invalid format",
input: "abc",
wantErr: true,
},
{
name: "unknown unit",
input: "5x",
wantErr: true,
},
{
name: "invalid number",
input: "xyzd",
wantErr: true,
},
{
name: "negative not supported",
input: "-5d",
wantErr: true,
},
})
}
func TestParseDurationSpecialCases(t *testing.T) {
t.Parallel()
// Test that standard Go durations work exactly as expected
standardDurations := []string{
"300ms",
"1.5h",
"2h45m",
"72h",
"1us",
"1µs",
"1ns",
}
for _, d := range standardDurations {
expected, err := time.ParseDuration(d)
require.NoError(t, err)
got, err := parseDuration(d)
require.NoError(t, err)
assert.Equal(t, expected, got, "standard duration %q should parse identically", d)
}
}
func TestParseDurationRealWorldExamples(t *testing.T) {
t.Parallel()
// Test real-world snapshot purge scenarios
tests := []struct {
description string
input string
olderThan time.Duration
}{
{
description: "keep snapshots from last 30 days",
input: "30d",
olderThan: 30 * 24 * time.Hour,
},
{
description: "keep snapshots from last 6 months",
input: "6mo",
olderThan: 6 * 30 * 24 * time.Hour,
},
{
description: "keep snapshots from last year",
input: "1y",
olderThan: 365 * 24 * time.Hour,
},
{
description: "keep snapshots from last week and a half",
input: "1w3d",
olderThan: 10 * 24 * time.Hour,
},
{
description: "keep snapshots from last 90 days",
input: "90d",
olderThan: 90 * 24 * time.Hour,
},
}
for _, tt := range tests {
t.Run(tt.description, func(t *testing.T) {
t.Parallel()
got, err := parseDuration(tt.input)
require.NoError(t, err)
assert.Equal(t, tt.olderThan, got)
// Verify the duration makes sense for snapshot purging
assert.Greater(t, got, time.Hour,
"snapshot purge duration should be at least an hour")
})
}
}
internal/cli/entry.go
+18 -71
View File
@@ -1,8 +1,6 @@
package cli
import (
"errors"
"io"
"os"
"strings"
"time"
@@ -15,67 +13,28 @@ import (
// the startup banner.
const shortCommitLen = 12
// exitCodeInterrupted is the exit status of a command that SIGINT or
// SIGTERM stopped. It is 128 plus SIGINT's number, 2, which is what a
// shell reports for a command stopped by Ctrl-C.
const exitCodeInterrupted = 130
// Entry is the main entry point for the CLI application.
// It prints the startup banner to stderr (unless a banner-suppressing
// flag is present in os.Args — see bannerSuppressedInArgs), executes the
// root cobra command, and routes any returned error through the
// ui.Writer so the user sees a properly formatted "🛑 ERROR:" line.
// The banner goes to stderr because stdout carries only the output the
// user asked for, such as a completion script or a `config get` value.
//
// It returns the process exit code (0 on success, 130 when interrupted,
// 1 on any other error) rather than calling os.Exit, so that main's
// deferred profile writers run before the process ends. See run in
// cmd/vaultik/main.go.
func Entry() int {
emitStartupBanner(os.Args[1:], os.Stderr)
// It prints the startup banner (unless a quiet flag is present in os.Args),
// executes the root cobra command, and routes any returned error through
// the ui.Writer so the user sees a properly formatted "🛑 ERROR:" line.
func Entry() {
if !bannerSuppressedInArgs(os.Args[1:]) {
short := globals.Commit
if len(short) > shortCommitLen {
short = short[:shortCommitLen]
}
writeStartupBanner(ui.New(os.Stdout), time.Now().UTC(), short)
}
rootCmd := NewRootCommand()
rootCmd.SilenceErrors = true
err := rootCmd.Execute()
if err != nil {
// An operation that ran inside the fx app has already reported
// its own failure (`snapshot verify --json` puts it in the
// document instead); errReported says so. Printing it again
// here would double the error line.
// Every other error — bad arguments, a config that would not
// load, an interrupt — reaches Entry unreported, so it is shown
// here.
if !errors.Is(err, errReported) {
ReportErrorf("%s", err.Error())
os.Exit(1)
}
if errors.Is(err, errInterrupted) {
return exitCodeInterrupted
}
return 1
}
return 0
}
// emitStartupBanner writes the startup banner to w unless args (the
// argument vector with the program name already stripped) contains a
// flag that suppresses it. Split out of Entry so that the decision is
// reachable from a test without running the whole CLI.
func emitStartupBanner(args []string, w io.Writer) {
if bannerSuppressedInArgs(args) {
return
}
short := globals.Commit
if len(short) > shortCommitLen {
short = short[:shortCommitLen]
}
writeStartupBanner(ui.New(w), time.Now().UTC(), short)
}
// ReportErrorf emits a user-facing error to stderr in the standard
@@ -87,19 +46,9 @@ func ReportErrorf(format string, args ...any) {
}
// bannerSuppressedInArgs reports whether any of args is a flag that
// should suppress the startup banner (--quiet/-q/--cron/--json). Stops
// at the "--" argument terminator. Recognizes both long forms and short
// -q, including combined short flags like "-qv".
//
// This scans the raw argument vector because the banner is printed
// before cobra parses anything — deliberately, so that it still appears
// when cobra rejects the arguments and on --help. The consequence is
// that a flag is matched wherever it occurs in the vector, including
// positions where the command it belongs to would not accept it.
// --json is a subcommand flag rather than a persistent one, but so is
// --cron (it exists only on `snapshot create`), so this adds no new
// class of imprecision. The only cost of a false positive is a missing
// decorative banner.
// should suppress the startup banner (--quiet/-q/--cron). Stops at the
// "--" argument terminator. Recognizes both long forms and short -q,
// including combined short flags like "-qv".
func bannerSuppressedInArgs(args []string) bool {
for _, a := range args {
if a == "--" {
@@ -107,13 +56,11 @@ func bannerSuppressedInArgs(args []string) bool {
}
switch a {
case "--quiet", "-q", "--cron", "--json":
case "--quiet", "-q", "--cron":
return true
}
if strings.HasPrefix(a, "--quiet=") ||
strings.HasPrefix(a, "--cron=") ||
strings.HasPrefix(a, "--json=") {
if strings.HasPrefix(a, "--quiet=") || strings.HasPrefix(a, "--cron=") {
return true
}
// Combined short flags like -qv or -vq.
internal/cli/entry_banner_test.go
-276
View File
@@ -1,276 +0,0 @@
package cli //nolint:testpackage // needs access to unexported emitStartupBanner
import (
"bytes"
"encoding/json"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"testing"
"github.com/adrg/xdg"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// Command words and flags used to build argument vectors below. They are
// constants rather than repeated literals so that a rename shows up as a
// compile error in one place.
const (
cmdSnapshot = "snapshot"
cmdList = "list"
cmdCreate = "create"
cmdVerify = "verify"
cmdRemove = "remove"
cmdPrune = "prune"
cmdRemote = "remote"
cmdInfo = "info"
flagJSON = "--json"
flagQuiet = "--quiet"
flagConfig = "--config"
// programName is argv[0] as the real process receives it. Entry
// strips it before scanning, so it has to be present.
programName = "vaultik"
// someSnapshotID only fills the positional argument; no test needs
// the snapshot to exist.
someSnapshotID = "host_2026-01-01T00:00:00Z"
)
// jsonArgumentVectors are the argument vectors of every --json
// invocation the CLI accepts, with the program name stripped exactly as
// Entry strips it. Each one must suppress the banner.
//
//nolint:gochecknoglobals // read-only test fixture shared by two tests
var jsonArgumentVectors = map[string][]string{
"snapshot list": {cmdSnapshot, cmdList, flagJSON},
"snapshot verify": {cmdSnapshot, cmdVerify, someSnapshotID, flagJSON},
"snapshot remove": {cmdSnapshot, cmdRemove, someSnapshotID, flagJSON},
"prune": {cmdPrune, flagJSON},
"remote info": {cmdRemote, cmdInfo, flagJSON},
// --json before the subcommand, and with an explicit value: the
// scan is positional, so both forms have to be recognized.
"json first": {flagJSON, cmdSnapshot, cmdList},
"json with value": {cmdSnapshot, cmdList, flagJSON + "=true"},
// A --json invocation that also carries a flag with a value, so the
// scan cannot be fooled by an argument that consumes the next one.
"json with config": {
flagConfig, "/nonexistent/vaultik.yml", cmdSnapshot, cmdList, flagJSON,
},
}
// TestJSONInvocationSuppressesBanner checks that every --json
// invocation suppresses the startup banner, as the README says --json
// does along with --quiet and --cron. The scan is over the raw argument
// vector, so each position and spelling of --json is listed.
func TestJSONInvocationSuppressesBanner(t *testing.T) {
t.Parallel()
for name, argv := range jsonArgumentVectors {
t.Run(name, func(t *testing.T) {
t.Parallel()
var banner bytes.Buffer
emitStartupBanner(argv, &banner)
assert.Empty(t, banner.String(),
"--json suppresses the banner")
})
}
}
// TestBannerStillPrintedWithoutSuppressingFlag pins the other half of
// the contract. Without it, deleting the banner outright would satisfy
// the test above, and the banner is wanted on interactive invocations.
func TestBannerStillPrintedWithoutSuppressingFlag(t *testing.T) {
t.Parallel()
for name, argv := range map[string][]string{
"no flags": {cmdSnapshot, cmdList},
"verbose": {cmdSnapshot, cmdList, "--verbose"},
"after the terminator": {
cmdSnapshot, "restore", "--", flagJSON,
},
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
var banner bytes.Buffer
emitStartupBanner(argv, &banner)
assert.Contains(t, banner.String(), "starting up at",
"the banner belongs on invocations that did not opt out")
})
}
}
// TestBannerSuppressedInArgs covers the suppression scan directly,
// including the flags that suppressed the banner before --json joined
// them, so that adding --json cannot regress them.
func TestBannerSuppressedInArgs(t *testing.T) {
t.Parallel()
for name, testCase := range map[string]struct {
args []string
suppressed bool
}{
"quiet long": {[]string{cmdSnapshot, cmdCreate, flagQuiet}, true},
"quiet short": {[]string{cmdSnapshot, cmdCreate, "-q"}, true},
"quiet combined": {[]string{cmdSnapshot, cmdCreate, "-qv"}, true},
"cron": {[]string{cmdSnapshot, cmdCreate, "--cron"}, true},
"json": {[]string{cmdSnapshot, cmdList, flagJSON}, true},
"nothing": {[]string{cmdSnapshot, cmdList}, false},
"empty": {nil, false},
"json after dashes": {
[]string{cmdSnapshot, cmdList, "--", flagJSON}, false,
},
"quiet after dashes": {
[]string{cmdSnapshot, cmdCreate, "--", "-q"}, false,
},
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
assert.Equal(t, testCase.suppressed,
bannerSuppressedInArgs(testCase.args))
})
}
}
// hermeticConfig is a complete, valid config that needs no network and
// no credentials: file:// storage is exempt from the S3 credential
// checks. A test that lists the destination must create its directory
// first, because listing a directory that does not exist is an error.
// Chunk, blob and compression settings are filled in by config.Load.
const hermeticConfig = `age_recipients:
- age1278m9q7dp3chsh2dcy82qk27v047zywyvtxwnj4cvt0z65jw6a7q5dqhfj
snapshots:
test:
paths:
- %s
storage_url: file://%s
index_path: %s
hostname: test-host
`
// TestEntryJSONStdoutIsExactlyOneDocument runs the real thing: Entry,
// with a real argument vector, over the process's real stdout file
// descriptor, all the way through cobra and the fx graph to the
// document. It is the assertion the issue asks for — `vaultik snapshot
// list --json | jq .` with no other flags — with the pipe replaced by a
// decoder.
//
// `snapshot list` is the command chosen because it is the only --json
// command that reaches its document without a populated destination
// store: it reads the local index, streams `metadata/` (empty here),
// and treats an empty destination directory as an empty list rather
// than a failure.
//
// Not parallel: it replaces os.Args, os.Stdout and the xdg globals.
func TestEntryJSONStdoutIsExactlyOneDocument(t *testing.T) {
dir := t.TempDir()
configPath := filepath.Join(dir, "config.yml")
storeDir := filepath.Join(dir, "store")
contents := fmt.Sprintf(hermeticConfig,
filepath.Join(dir, "source"),
storeDir,
filepath.Join(dir, "index.sqlite"))
require.NoError(t,
os.WriteFile(configPath, []byte(contents), configFileMode))
require.NoError(t, os.Mkdir(storeDir, 0o750))
// The PID lock lives under xdg.DataHome, which xdg resolves at
// package init; point it at the temp dir so the test neither
// touches nor collides with the real one.
t.Setenv("XDG_DATA_HOME", filepath.Join(dir, "data"))
xdg.Reload()
t.Cleanup(xdg.Reload)
previousArgs := os.Args
t.Cleanup(func() {
os.Args = previousArgs
rootFlags = RootFlags{}
})
os.Args = []string{
programName, flagConfig, configPath, cmdSnapshot, cmdList, flagJSON,
}
stdout := captureProcessStdout(t, func() { _ = Entry() })
requireExactlyOneJSONDocument(t, stdout)
var snapshots []any
require.NoError(t, json.Unmarshal([]byte(stdout), &snapshots))
assert.Empty(t, snapshots,
"a destination store with no snapshots lists none")
}
// captureProcessStdout redirects the process's own stdout to a pipe for
// the duration of fn and returns what was written to it. The redirection
// has to be at the file-descriptor level rather than through an injected
// writer, because the commands Entry runs reach os.Stdout directly.
//
// Not parallel-safe: os.Stdout is process-global.
func captureProcessStdout(t *testing.T, fn func()) string {
t.Helper()
reader, writer, err := os.Pipe()
require.NoError(t, err)
previous := os.Stdout
os.Stdout = writer
captured := make(chan string, 1)
go func() {
var buf bytes.Buffer
_, _ = io.Copy(&buf, reader)
captured <- buf.String()
}()
fn()
os.Stdout = previous
require.NoError(t, writer.Close())
out := <-captured
require.NoError(t, reader.Close())
return out
}
// requireExactlyOneJSONDocument fails unless stdout decodes as a single
// JSON value with nothing before or after it — the property that makes
// `| jq` work.
func requireExactlyOneJSONDocument(t *testing.T, stdout string) {
t.Helper()
decoder := json.NewDecoder(strings.NewReader(stdout))
var document any
err := decoder.Decode(&document)
require.NoError(t, err,
"stdout must parse as JSON, got:\n%s", stdout)
_, err = decoder.Token()
require.ErrorIs(t, err, io.EOF,
"stdout must hold exactly one JSON document, got:\n%s", stdout)
}
internal/cli/entry_interrupt_test.go
-203
View File
@@ -1,203 +0,0 @@
package cli //nolint:testpackage // shares runEntry and the argument constants
import (
"fmt"
"net/http"
"net/http/httptest"
"os"
"os/signal"
"path/filepath"
"strings"
"testing"
"time"
"github.com/adrg/xdg"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// stalledStoreConfig is hermeticConfig with an s3:// destination store
// in place of the file:// one. The server behind it accepts any
// credentials.
const stalledStoreConfig = `age_recipients:
- age1278m9q7dp3chsh2dcy82qk27v047zywyvtxwnj4cvt0z65jw6a7q5dqhfj
snapshots:
test:
paths:
- %s
storage_url: s3://bucket?endpoint=%s&ssl=false
s3:
access_key_id: key
secret_access_key: secret
index_path: %s
hostname: test-host
`
// interruptRepeat is how often interruptOnFirstRequest sends SIGINT.
const interruptRepeat = 50 * time.Millisecond
// TestEntryInterruptedRun sends SIGINT to the test process while a
// command waits on the destination store, and checks that Entry returns
// 130 and prints one line on stderr saying the run was interrupted. The
// store is a local HTTP server that holds every request open, so the
// command is always mid-operation when the signal arrives. The two
// cases cover --cron and --json, which silence other output.
//
// Not parallel: it signals the process and replaces os.Args, os.Stdout,
// os.Stderr and the xdg globals.
//
//nolint:paralleltest // signals the process and replaces process globals
func TestEntryInterruptedRun(t *testing.T) {
for _, testCase := range []struct {
name string
args []string
}{
{
name: "snapshot create --cron",
args: []string{cmdSnapshot, cmdCreate, "--cron"},
},
{
name: "snapshot verify --json",
args: []string{cmdSnapshot, cmdVerify, someSnapshotID, flagJSON},
},
} {
t.Run(testCase.name, func(t *testing.T) {
endpoint, requestArrived := startStalledStore(t)
configPath := writeStalledStoreConfig(t, endpoint)
interruptOnFirstRequest(t, requestArrived)
code, _, stderr := runEntry(t,
append([]string{flagConfig, configPath}, testCase.args...)...)
assert.Equal(t, 130, code)
assert.Equal(t, 1,
strings.Count(stderr, errInterrupted.Error()), stderr)
})
}
}
// interruptOnFirstRequest sends SIGINT to the test process every
// interruptRepeat, from the first request to the destination store until
// the test ends. One signal is not enough: the command can reach the
// store before fx has started catching signals. The test catches SIGINT
// too, so that a signal fx is not catching does not kill the test
// binary.
func interruptOnFirstRequest(t *testing.T, requestArrived <-chan struct{}) {
t.Helper()
self, err := os.FindProcess(os.Getpid())
require.NoError(t, err)
caught := make(chan os.Signal, 1)
signal.Notify(caught, os.Interrupt)
testEnded := make(chan struct{})
senderDone := make(chan struct{})
// Stop catching SIGINT only after the sender has returned. The sender
// waits for each SIGINT it sends to arrive on caught; one still on
// its way after signal.Stop would kill the test binary.
t.Cleanup(func() {
close(testEnded)
<-senderDone
signal.Stop(caught)
})
go func() {
defer close(senderDone)
select {
case <-requestArrived:
case <-testEnded:
return
}
ticker := time.NewTicker(interruptRepeat)
defer ticker.Stop()
for {
// Empty caught, so that the receive below waits for this
// SIGINT rather than an earlier one.
select {
case <-caught:
default:
}
sendErr := self.Signal(os.Interrupt)
if sendErr != nil {
t.Errorf("sending SIGINT: %v", sendErr)
return
}
<-caught
select {
case <-testEnded:
return
case <-ticker.C:
}
}
}()
}
// startStalledStore starts an HTTP server that never answers: each
// request is held until the client gives up on it or the test ends.
// It returns the server's host:port and a channel that receives a value
// when the first request arrives.
func startStalledStore(t *testing.T) (string, <-chan struct{}) {
t.Helper()
requestArrived := make(chan struct{}, 1)
release := make(chan struct{})
server := httptest.NewServer(http.HandlerFunc(
func(_ http.ResponseWriter, r *http.Request) {
select {
case requestArrived <- struct{}{}:
default:
}
select {
case <-r.Context().Done():
case <-release:
}
}))
// Cleanups run last-registered first, so release lets any held
// request return before Close waits for it.
t.Cleanup(server.Close)
t.Cleanup(func() { close(release) })
return server.Listener.Addr().String(), requestArrived
}
// writeStalledStoreConfig writes a config whose destination store is the
// server at endpoint and whose snapshot source holds one small file, so
// that `snapshot create` has a blob to upload. Returns the config path.
func writeStalledStoreConfig(t *testing.T, endpoint string) string {
t.Helper()
dir := t.TempDir()
configPath := filepath.Join(dir, "config.yml")
sourceDir := filepath.Join(dir, "source")
require.NoError(t, os.Mkdir(sourceDir, 0o750))
require.NoError(t, os.WriteFile(filepath.Join(sourceDir, "file.txt"),
[]byte("contents"), 0o600))
contents := fmt.Sprintf(stalledStoreConfig,
sourceDir, endpoint, filepath.Join(dir, "index.sqlite"))
require.NoError(t,
os.WriteFile(configPath, []byte(contents), configFileMode))
// The PID lock lives under xdg.DataHome, which xdg resolves at
// package init; point it at the temp dir so the test neither
// touches nor collides with the real one.
t.Setenv("XDG_DATA_HOME", filepath.Join(dir, "data"))
xdg.Reload()
t.Cleanup(xdg.Reload)
return configPath
}
internal/cli/entry_prune_json_stderr_test.go
-140
View File
@@ -1,140 +0,0 @@
package cli //nolint:testpackage // shares the prune fixtures and capture helpers
import (
"bytes"
"io"
"os"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// staleRecordLogMessage is the local-cleanup audit line CleanupLocalSnapshots
// logs for each stale record. It is exactly the signal issue #112 says a
// machine consumer lost under --json: gated off stdout, and pinned below
// the log level on stderr because --json used to force Quiet.
const staleRecordLogMessage = "Removing stale local snapshot record"
// TestEntryPruneJSONStderrHonoursVerbosity is the end-to-end regression
// guard for issue #112. Under --json the log level must still follow
// --verbose/--debug rather than being pinned to WARN, so the
// local-cleanup records reach stderr under --verbose while stdout stays
// exactly one JSON document; without --verbose they stay below the
// level, as they do without --json.
//
// Both halves are asserted together on the same run, because the fix has
// to keep the document clean (issue #108) while freeing stderr.
//
// Not parallel: it replaces os.Args, os.Stdout, os.Stderr and the xdg
// globals.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntryPruneJSONStderrHonoursVerbosity(t *testing.T) {
for _, testCase := range []struct {
name string
verbose bool
wantOnStderr bool
}{
{
name: "verbose json surfaces the cleanup record on stderr",
verbose: true,
wantOnStderr: true,
},
{
name: "json alone keeps the cleanup record below the level",
verbose: false,
wantOnStderr: false,
},
} {
t.Run(testCase.name, func(t *testing.T) {
configPath := writeHermeticPruneConfig(t, true)
previousArgs := os.Args
t.Cleanup(func() {
os.Args = previousArgs
rootFlags = RootFlags{}
})
args := []string{
programName, flagConfig, configPath, cmdPrune, flagJSON,
}
if testCase.verbose {
args = append(args, "--verbose")
}
os.Args = args
stdout, stderr := captureProcessStdoutAndStderr(t,
func() { _ = Entry() })
// The document stays clean in both cases: freeing stderr must
// not regress issue #108.
requireExactlyOneJSONDocument(t, stdout)
if testCase.wantOnStderr {
assert.Contains(t, stderr, staleRecordLogMessage,
"--verbose --json must emit the cleanup record on stderr")
assert.Contains(t, stderr, stalePruneSnapshotID,
"the record must name the snapshot it removed")
} else {
assert.NotContains(t, stderr, staleRecordLogMessage,
"without --verbose the record stays below the log level")
}
})
}
}
// captureProcessStdoutAndStderr redirects both of the process's own
// standard streams to pipes for the duration of fn and returns what was
// written to each. The redirection is at the file-descriptor level
// because the logger binds os.Stderr when it initializes inside fn, and
// the JSON document reaches os.Stdout independently; the point is to see
// where each actually lands.
//
// Not parallel-safe: os.Stdout and os.Stderr are process-global.
func captureProcessStdoutAndStderr(t *testing.T, fn func()) (string, string) {
t.Helper()
outReader, outWriter, err := os.Pipe()
require.NoError(t, err)
errReader, errWriter, err := os.Pipe()
require.NoError(t, err)
previousOut, previousErr := os.Stdout, os.Stderr
os.Stdout, os.Stderr = outWriter, errWriter
capturedOut := drain(outReader)
capturedErr := drain(errReader)
fn()
os.Stdout, os.Stderr = previousOut, previousErr
require.NoError(t, outWriter.Close())
require.NoError(t, errWriter.Close())
out, errOut := <-capturedOut, <-capturedErr
require.NoError(t, outReader.Close())
require.NoError(t, errReader.Close())
return out, errOut
}
// drain copies a reader to a string on a goroutine and delivers the
// result once the writer end is closed.
func drain(reader io.Reader) <-chan string {
captured := make(chan string, 1)
go func() {
var buf bytes.Buffer
_, _ = io.Copy(&buf, reader)
captured <- buf.String()
}()
return captured
}
internal/cli/entry_prune_json_test.go
-176
View File
@@ -1,176 +0,0 @@
package cli //nolint:testpackage // shares hermeticConfig and the capture helpers
import (
"context"
"database/sql"
"encoding/json"
"fmt"
"os"
"path/filepath"
"testing"
"time"
"github.com/adrg/xdg"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/types"
)
// pruneJSONDocument is the shape `prune --json` writes: the
// PruneBlobsResult document, and nothing else.
//
//nolint:tagliatelle // snake_case is the established JSON output format
type pruneJSONDocument struct {
BlobsFound int `json:"blobs_found"`
BlobsDeleted int `json:"blobs_deleted"`
BytesFreed int64 `json:"bytes_freed"`
}
// stalePruneSnapshotID is seeded into the local index with no manifest
// on the destination store, which is exactly what makes it stale.
const stalePruneSnapshotID = "test-host_test_2026-04-01T09:00:00Z"
// TestEntryPruneJSONStdoutIsExactlyOneDocument is the end-to-end
// regression guard for issue #108: `vaultik prune --json | jq .` must
// work with no other flags.
//
// It runs Entry over the process's real stdout descriptor, through
// cobra and the fx graph, against a hermetic file:// destination store
// — the same construction TestEntryJSONStdoutIsExactlyOneDocument uses
// for `snapshot list`, with the pipe to jq replaced by a decoder.
//
// Both branches of the local-snapshot reconciliation are exercised
// because the three stdout writes that broke this covered all of them:
// one line per stale record and a summary when there were any, and a
// "No stale local snapshots found." line when there were none. No input
// avoided the contamination, so no single branch demonstrates the fix.
//
// Not parallel: it replaces os.Args, os.Stdout and the xdg globals.
//
//nolint:paralleltest // replaces os.Args, os.Stdout and the xdg globals
func TestEntryPruneJSONStdoutIsExactlyOneDocument(t *testing.T) {
for _, testCase := range []struct {
name string
seedStale bool
description string
}{
{
name: "no stale local records",
seedStale: false,
description: "the empty-index branch used to print a 'No stale' line",
},
{
name: "stale local records present",
seedStale: true,
description: "the removal branch used to print a line per record " +
"plus a summary",
},
} {
t.Run(testCase.name, func(t *testing.T) {
configPath := writeHermeticPruneConfig(t, testCase.seedStale)
previousArgs := os.Args
t.Cleanup(func() {
os.Args = previousArgs
rootFlags = RootFlags{}
})
os.Args = []string{
programName, flagConfig, configPath, cmdPrune, flagJSON,
}
stdout := captureProcessStdout(t, func() { _ = Entry() })
requireExactlyOneJSONDocument(t, stdout)
var document pruneJSONDocument
require.NoError(t, json.Unmarshal([]byte(stdout), &document),
testCase.description)
// A destination store with no blobs has none to prune. The
// assertion that matters is the one above; this one keeps the
// test honest about which document it decoded.
assert.Equal(t, 0, document.BlobsFound)
})
}
}
// writeHermeticPruneConfig builds a config over a temp directory with an
// empty destination directory and, if seedStale is set, creates the
// index database up front with one snapshot record that has no
// counterpart on the destination store, and gives the destination the
// marker a backup writes. Returns the config path.
func writeHermeticPruneConfig(t *testing.T, seedStale bool) string {
t.Helper()
dir := t.TempDir()
configPath := filepath.Join(dir, "config.yml")
indexPath := filepath.Join(dir, "index.sqlite")
storeDir := filepath.Join(dir, "store")
contents := fmt.Sprintf(hermeticConfig,
filepath.Join(dir, "source"),
storeDir,
indexPath)
require.NoError(t,
os.WriteFile(configPath, []byte(contents), configFileMode))
// prune fails on a destination directory that does not exist, so the
// empty store is created here rather than left to a first backup.
require.NoError(t, os.Mkdir(storeDir, 0o750))
// The PID lock lives under xdg.DataHome, which xdg resolves at
// package init; point it at the temp dir so the test neither
// touches nor collides with the real one.
t.Setenv("XDG_DATA_HOME", filepath.Join(dir, "data"))
xdg.Reload()
t.Cleanup(xdg.Reload)
if seedStale {
seedStaleSnapshotRecord(t, indexPath)
// While the local index holds a snapshot, prune fails on a
// destination that answers that it has no marker.
require.NoError(t, os.WriteFile(
filepath.Join(storeDir, "vaultik-destination.txt"), nil, 0o600))
}
return configPath
}
// seedStaleSnapshotRecord creates the index database at path and
// inserts one completed snapshot into it. Nothing is written to the
// destination store, so `prune` finds the record stale and removes it —
// the branch that printed a line per record.
func seedStaleSnapshotRecord(t *testing.T, path string) {
t.Helper()
ctx := context.Background()
db, err := database.New(ctx, path)
require.NoError(t, err)
defer func() { require.NoError(t, db.Close()) }()
startedAt := time.Date(2026, 4, 1, 9, 0, 0, 0, time.UTC)
completedAt := startedAt.Add(time.Minute)
snap := &database.Snapshot{
ID: types.SnapshotID(stalePruneSnapshotID),
Hostname: "test-host",
VaultikVersion: "test",
StartedAt: startedAt,
CompletedAt: &completedAt,
}
repos := database.NewRepositories(db)
err = repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
return repos.Snapshots.Create(ctx, tx, snap)
})
require.NoError(t, err)
}
internal/cli/entry_prune_quiet_test.go
-127
View File
@@ -1,127 +0,0 @@
package cli //nolint:testpackage // shares runEntry and the hermetic config helpers
import (
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// flagCron is used to build the argument vectors below.
const flagCron = "--cron"
// plantUnreferencedBlob writes a blob that no snapshot references to the
// destination store at storeDir, dated long before any prune grace period,
// so that a prune deletes it. Returns its path.
func plantUnreferencedBlob(t *testing.T, storeDir string) string {
t.Helper()
hash := strings.Repeat("b", 64)
path := filepath.Join(storeDir, "blobs", hash[:2], hash[2:4], hash)
require.NoError(t, os.MkdirAll(filepath.Dir(path), 0o750))
require.NoError(t, os.WriteFile(path, []byte("unreferenced"), 0o600))
longAgo := time.Unix(0, 0)
require.NoError(t, os.Chtimes(path, longAgo, longAgo))
return path
}
// TestEntrySnapshotCreatePruneIsSilent checks that a successful
// `snapshot create --prune` under --cron or --quiet prints nothing on
// stdout or stderr, whether or not the prune finds a blob to delete.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotCreatePruneIsSilent(t *testing.T) {
for _, testCase := range []struct {
name string
args []string
plantBlob bool
}{
{
name: "cron, no blob to delete",
args: []string{cmdSnapshot, cmdCreate, flagCron, flagPrune},
},
{
name: "cron, blob to delete",
args: []string{cmdSnapshot, cmdCreate, flagCron, flagPrune},
plantBlob: true,
},
{
name: "cron with keep-newer-than",
args: []string{
cmdSnapshot, cmdCreate, flagCron, flagPrune,
flagKeepNewerThan, "4w",
},
plantBlob: true,
},
{
name: "quiet",
args: []string{flagQuiet, cmdSnapshot, cmdCreate, flagPrune},
plantBlob: true,
},
} {
t.Run(testCase.name, func(t *testing.T) {
configPath, storeDir := writeBackupConfig(t)
var blob string
if testCase.plantBlob {
blob = plantUnreferencedBlob(t, storeDir)
}
code, stdout, stderr := runEntry(t,
append([]string{flagConfig, configPath}, testCase.args...)...)
assert.Equal(t, 0, code)
assert.Empty(t, stdout)
assert.Empty(t, stderr)
if testCase.plantBlob {
assert.NoFileExists(t, blob, "the prune must delete the blob")
}
})
}
}
// TestEntryPruneQuietIsSilent checks that a successful `prune --force`
// under --quiet prints nothing on stdout or stderr, both when there is
// nothing to remove and when it removes a stale local snapshot record
// and an unreferenced blob.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntryPruneQuietIsSilent(t *testing.T) {
for _, testCase := range []struct {
name string
staleRecordAndBlob bool
}{
{name: "nothing to remove"},
{name: "stale record and blob to remove", staleRecordAndBlob: true},
} {
t.Run(testCase.name, func(t *testing.T) {
configPath := writeHermeticPruneConfig(t, testCase.staleRecordAndBlob)
var blob string
if testCase.staleRecordAndBlob {
// writeHermeticPruneConfig puts the store beside the config.
blob = plantUnreferencedBlob(t,
filepath.Join(filepath.Dir(configPath), "store"))
}
code, stdout, stderr := runEntry(t,
flagConfig, configPath, flagQuiet, cmdPrune, flagForce)
assert.Equal(t, 0, code)
assert.Empty(t, stdout)
assert.Empty(t, stderr)
if testCase.staleRecordAndBlob {
assert.NoFileExists(t, blob, "the prune must delete the blob")
}
})
}
}
internal/cli/entry_snapshot_create_prune_test.go
-185
View File
@@ -1,185 +0,0 @@
package cli //nolint:testpackage // shares runEntry and the hermetic config helpers
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/database"
)
// Flags of `snapshot create` used to build argument vectors below.
const (
flagPrune = "--prune"
flagKeepNewerThan = "--keep-newer-than"
)
// TestEntrySnapshotCreateRejectsKeepNewerThanBeforeBackup checks that a
// --keep-newer-than the post-backup prune could not use stops `snapshot
// create` before the backup: the command fails and the destination
// store stays empty. It used to be read only by the prune, after the
// snapshot was uploaded, and without --prune it was ignored.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotCreateRejectsKeepNewerThanBeforeBackup(t *testing.T) {
for _, testCase := range []struct {
name string
args []string
}{
{
name: "decimal in a day count",
args: []string{flagPrune, flagKeepNewerThan, "1.5d"},
},
{
name: "unknown unit",
args: []string{flagPrune, flagKeepNewerThan, "4x"},
},
{
name: "without --prune",
args: []string{flagKeepNewerThan, "4w"},
},
} {
t.Run(testCase.name, func(t *testing.T) {
configPath, storeDir := writeBackupConfig(t)
code, _, _ := runEntry(t, append(
[]string{flagConfig, configPath, cmdSnapshot, cmdCreate},
testCase.args...)...)
assert.Equal(t, 1, code)
stored, err := os.ReadDir(storeDir)
require.NoError(t, err)
assert.Empty(t, stored, "nothing may be uploaded")
})
}
}
// TestEntrySnapshotCreateReportsPostBackupPruneFailure makes the
// post-backup prune fail after a good backup, by planting a manifest
// that cannot be decoded, and checks that the failure is reported as the
// prune's and not as a failed snapshot creation.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotCreateReportsPostBackupPruneFailure(t *testing.T) {
configPath, storeDir := writeBackupConfig(t)
// A remote key is 64 lowercase hex characters. The prune reads every
// manifest on the store and fails on one it cannot decode.
plantedDir := filepath.Join(storeDir, "metadata", strings.Repeat("a", 64))
require.NoError(t, os.MkdirAll(plantedDir, 0o750))
require.NoError(t, os.WriteFile(
filepath.Join(plantedDir, "manifest.json.zst"),
[]byte("not a valid manifest"), 0o600))
code, _, stderr := runEntry(t,
flagConfig, configPath, cmdSnapshot, cmdCreate, flagPrune)
assert.Equal(t, 1, code)
assert.Contains(t, stderr, "Backup completed, but post-backup prune failed")
assert.NotContains(t, stderr, "Snapshot creation failed")
snapshots, err := os.ReadDir(filepath.Join(storeDir, "metadata"))
require.NoError(t, err)
assert.Len(t, snapshots, 2,
"the new snapshot is on the store beside the planted one")
}
// TestEntrySnapshotCreatePruneExitsNonZeroWhenSourceIsEmpty backs up,
// empties the source directory, as a volume that is not mounted leaves
// its mount point, and runs `snapshot create --prune`. The command exits
// non-zero and reports the backup as completed, not as a failed snapshot
// creation. See https://git.eeqj.de/sneak/vaultik/issues/388.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotCreatePruneExitsNonZeroWhenSourceIsEmpty(t *testing.T) {
configPath, storeDir := writeBackupConfig(t)
code, _, _ := runEntry(t, flagConfig, configPath, cmdSnapshot, cmdCreate)
require.Equal(t, 0, code)
// writeBackupConfig puts the source directory beside the config file.
require.NoError(t, os.Remove(
filepath.Join(filepath.Dir(configPath), "source", "file.txt")))
code, _, stderr := runEntry(t,
flagConfig, configPath, cmdSnapshot, cmdCreate, flagPrune)
assert.Equal(t, 1, code)
assert.Contains(t, stderr,
"Backup completed, but 1 source path(s) were empty")
assert.NotContains(t, stderr, "Snapshot creation failed")
snapshots, err := os.ReadDir(filepath.Join(storeDir, "metadata"))
require.NoError(t, err)
assert.Len(t, snapshots, 2, "the older snapshot is kept")
}
// TestEntrySnapshotCreatePruneExitsNonZeroWhenMountPointIsMissing backs
// up, records in the local index that this backup found the source
// directory as a mount point, and runs `snapshot create --prune`, which
// does not find it as one. The command exits non-zero, reports the
// backup as completed, not as a failed snapshot creation, and keeps the
// older snapshot. See https://git.eeqj.de/sneak/vaultik/issues/395.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotCreatePruneExitsNonZeroWhenMountPointIsMissing(
t *testing.T,
) {
configPath, storeDir := writeBackupConfig(t)
code, _, _ := runEntry(t, flagConfig, configPath, cmdSnapshot, cmdCreate)
require.Equal(t, 0, code)
// writeBackupConfig puts the source directory and the local index
// beside the config file.
dir := filepath.Dir(configPath)
ctx := context.Background()
db, err := database.New(ctx, filepath.Join(dir, "index.sqlite"))
require.NoError(t, err)
repos := database.NewRepositories(db)
backedUp, err := repos.Snapshots.GetOnlySnapshot(ctx)
require.NoError(t, err)
require.NoError(t, repos.Snapshots.AddMountPoint(ctx, nil,
backedUp.ID.String(), filepath.Join(dir, "source")))
require.NoError(t, db.Close())
code, _, stderr := runEntry(t,
flagConfig, configPath, cmdSnapshot, cmdCreate, flagPrune)
assert.Equal(t, 1, code)
assert.Contains(t, stderr, "Backup completed, but 1 mount point(s) "+
"an earlier snapshot recorded were not found")
assert.NotContains(t, stderr, "Snapshot creation failed")
snapshots, err := os.ReadDir(filepath.Join(storeDir, "metadata"))
require.NoError(t, err)
assert.Len(t, snapshots, 2, "the older snapshot is kept")
}
// writeBackupConfig builds a hermetic config with one small file in its
// source directory and an empty destination store directory. Returns the
// config path and the store directory.
func writeBackupConfig(t *testing.T) (string, string) {
t.Helper()
configPath := writeHermeticPruneConfig(t, false)
// writeHermeticPruneConfig puts the source and store directories
// beside the config file.
dir := filepath.Dir(configPath)
sourceDir := filepath.Join(dir, "source")
require.NoError(t, os.Mkdir(sourceDir, 0o750))
require.NoError(t, os.WriteFile(
filepath.Join(sourceDir, "file.txt"), []byte("hello"), 0o600))
return configPath, filepath.Join(dir, "store")
}
internal/cli/entry_snapshot_create_skip_errors_test.go
-30
View File
@@ -1,30 +0,0 @@
package cli //nolint:testpackage // shares runEntry and the hermetic config helpers
import (
"testing"
"github.com/stretchr/testify/assert"
)
// TestEntrySnapshotCreateSkipErrorsExitsNonZeroWhenItSkips backs up,
// under --skip-errors, a source directory that does not exist, which the
// walk skips. The command exits non-zero and reports the backup as
// completed, not as a failed snapshot creation. See
// https://git.eeqj.de/sneak/vaultik/issues/310.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotCreateSkipErrorsExitsNonZeroWhenItSkips(t *testing.T) {
// writeBackupConfig would create the source directory.
configPath := writeHermeticPruneConfig(t, false)
code, stdout, stderr := runEntry(t,
flagConfig, configPath, "--skip-errors", cmdSnapshot, cmdCreate)
assert.Equal(t, 1, code)
assert.Contains(t, stdout,
"Files: 1 examined, 0 backed up, 0 unchanged, 1 failed.")
assert.NotContains(t, stdout, "Finished successfully.")
assert.Contains(t, stderr,
"Backup completed, but 1 file(s) could not be read")
assert.NotContains(t, stderr, "Snapshot creation failed")
}
internal/cli/entry_snapshot_purge_test.go
-75
View File
@@ -1,75 +0,0 @@
package cli //nolint:testpackage // shares runEntry and the hermetic config helpers
import (
"context"
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/snapshot"
)
// Words and flags of `snapshot purge` used to build argument vectors
// below.
const (
cmdPurge = "purge"
flagOlderThan = "--older-than"
flagForce = "--force"
)
// TestEntrySnapshotPurgeKeepsMetadataWhenLocalIndexDeleteFails makes
// the local index refuse to delete an old snapshot and checks that
// `snapshot purge` exits non-zero and leaves that snapshot's metadata on
// the destination store. It used to log the error, delete the metadata
// and exit 0.
//
// `snapshot create --prune` runs the same purge, but cannot be driven
// this way: the refusal also breaks the backup's metadata export, which
// deletes the other snapshots from its copy of the local index.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotPurgeKeepsMetadataWhenLocalIndexDeleteFails(t *testing.T) {
configPath := writeHermeticPruneConfig(t, false)
dir := filepath.Dir(configPath)
indexPath := filepath.Join(dir, "index.sqlite")
// seedStaleSnapshotRecord writes an old snapshot to the local index
// only; the manifest planted here puts it on the destination store
// too, so the purge selects it rather than dropping it as stale.
seedStaleSnapshotRecord(t, indexPath)
// While the local index holds a snapshot, purge fails on a
// destination that answers that it has no marker.
require.NoError(t, os.WriteFile(
filepath.Join(dir, "store", "vaultik-destination.txt"), nil, 0o600))
manifestDir := filepath.Join(dir, "store", "metadata",
snapshot.RemoteSnapshotKey(stalePruneSnapshotID))
require.NoError(t, os.MkdirAll(manifestDir, 0o750))
manifestPath := filepath.Join(manifestDir, "manifest.json.zst")
require.NoError(t, os.WriteFile(manifestPath, []byte("stub"), 0o600))
ctx := context.Background()
db, err := database.New(ctx, indexPath)
require.NoError(t, err)
_, err = db.Conn().ExecContext(ctx, `
CREATE TRIGGER refuse_snapshot_delete BEFORE DELETE ON snapshots
BEGIN SELECT RAISE(ABORT, 'simulated index error'); END`)
require.NoError(t, err)
require.NoError(t, db.Close())
code, _, stderr := runEntry(t, flagConfig, configPath,
cmdSnapshot, cmdPurge, flagOlderThan, "30d", flagForce)
assert.Equal(t, 1, code)
assert.Contains(t, stderr, "simulated index error",
"the purge must fail at the local-index delete")
assert.FileExists(t, manifestPath,
"the snapshot is still in the local index, so its metadata must stay")
}
internal/cli/entry_status_test.go
-58
View File
@@ -1,58 +0,0 @@
package cli //nolint:testpackage // shares programName and the capture helpers
import (
"os"
"testing"
"github.com/stretchr/testify/assert"
)
// TestEntryReturnsStatusCode pins the contract main() relies on for
// issue #75: Entry reports success or failure through its return value
// and never calls os.Exit. An os.Exit from inside Entry would skip
// main's deferred profile writers and truncate the profile of a failing
// command. main turns this code into os.Exit only after those defers
// run, so a failing command must come back with a non-zero code rather
// than ending the process here.
//
// Stdout and stderr are captured only to keep the banner and command
// output off the test log; the assertion is on the returned code.
//
//nolint:paralleltest // replaces os.Args and rootFlags
func TestEntryReturnsStatusCode(t *testing.T) {
for _, testCase := range []struct {
name string
args []string
want int
}{
{
// version is self-contained: it needs no config and no
// destination store, so it exercises the success path.
name: "successful command returns zero",
args: []string{programName, "version"},
want: 0,
},
{
name: "unknown command returns one",
args: []string{programName, "no-such-command"},
want: 1,
},
} {
t.Run(testCase.name, func(t *testing.T) {
previousArgs := os.Args
t.Cleanup(func() {
os.Args = previousArgs
rootFlags = RootFlags{}
})
os.Args = testCase.args
var code int
_, _ = captureProcessStdoutAndStderr(t, func() { code = Entry() })
assert.Equal(t, testCase.want, code)
})
}
}
internal/cli/entry_stdout_stderr_test.go
-200
View File
@@ -1,200 +0,0 @@
package cli //nolint:testpackage // shares hermeticConfig and the capture helpers
import (
"context"
"encoding/json"
"fmt"
"log/slog"
"os"
"path/filepath"
"strings"
"testing"
"github.com/adrg/xdg"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/database"
)
// TestEntryCompletionStdoutIsTheScript runs `vaultik completion bash`,
// whose stdout the README tells the user to source. The script has to
// start on the first line.
//
//nolint:paralleltest // replaces os.Args, os.Stdout and os.Stderr
func TestEntryCompletionStdoutIsTheScript(t *testing.T) {
code, stdout, _ := runEntry(t, "completion", "bash")
require.Equal(t, 0, code)
firstLine, _, _ := strings.Cut(stdout, "\n")
assert.True(t, strings.HasPrefix(firstLine, "# bash completion"),
"the first line of stdout must be the script's, got %q", firstLine)
}
// TestEntryConfigGetStdoutIsTheValue runs `vaultik config get`, whose
// stdout a script reads as the value and nothing else.
//
//nolint:paralleltest // replaces os.Args, os.Stdout and os.Stderr
func TestEntryConfigGetStdoutIsTheValue(t *testing.T) {
configPath := filepath.Join(t.TempDir(), "config.yml")
require.NoError(t, os.WriteFile(configPath,
[]byte("hostname: test-host\n"), configFileMode))
code, stdout, _ := runEntry(t,
flagConfig, configPath, "config", "get", "hostname")
require.Equal(t, 0, code)
assert.Equal(t, "test-host\n", stdout)
}
// TestEntryJSONFailureIsReportedOnStderr runs each --json command that
// writes no document when it fails, against a destination it cannot
// use. The error must reach stderr, and stdout must stay empty.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntryJSONFailureIsReportedOnStderr(t *testing.T) {
for _, testCase := range []struct {
name string
args []string
wantOnStderr string
}{
{
name: "remote info",
args: []string{cmdRemote, cmdInfo, flagJSON},
wantOnStderr: "Failed to get remote info",
},
{
name: "prune",
args: []string{cmdPrune, flagJSON},
wantOnStderr: "Prune failed",
},
{
name: "snapshot remove",
args: []string{cmdSnapshot, cmdRemove, someSnapshotID, flagJSON},
wantOnStderr: "Failed to remove snapshot",
},
} {
t.Run(testCase.name, func(t *testing.T) {
configPath := writeUnusableDestinationConfig(t)
code, stdout, stderr := runEntry(t,
append([]string{flagConfig, configPath}, testCase.args...)...)
assert.Equal(t, 1, code)
assert.Empty(t, stdout,
"a failed --json command has no document to write")
assert.Contains(t, stderr, testCase.wantOnStderr,
"the failure must be reported on stderr")
})
}
}
// TestEntrySnapshotRemoveJSONWarningIsOnStderr runs `snapshot remove
// --json` on a snapshot in the local index, against a destination
// directory that does not exist. The command removes the snapshot from
// the local index and still exits 0. Its stdout must hold the document
// alone, with the warning about the destination store on stderr: the
// command to run again once it is reachable, and the snapshot's ID in
// the record's snapshot_id field.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotRemoveJSONWarningIsOnStderr(t *testing.T) {
configPath, indexPath := writeMissingDestinationConfig(t)
seedStaleSnapshotRecord(t, indexPath)
code, stdout, stderr := runEntry(t, flagConfig, configPath,
cmdSnapshot, cmdRemove, stalePruneSnapshotID, flagJSON)
require.Equal(t, 0, code)
requireExactlyOneJSONDocument(t, stdout)
// stderr is a pipe here, so the logger writes one JSON record a line.
var warning map[string]any
for line := range strings.Lines(stderr) {
if strings.Contains(line,
"Could not remove snapshot metadata from remote storage") {
require.NoError(t, json.Unmarshal([]byte(line), &warning))
}
}
require.NotNil(t, warning, "the warning must reach stderr")
assert.Contains(t, warning[slog.MessageKey],
"run 'vaultik snapshot remove' with the snapshot's ID again")
assert.Equal(t, stalePruneSnapshotID, warning["snapshot_id"])
}
// writeMissingDestinationConfig builds a config whose destination
// directory does not exist. Returns the config path and the path of
// its local index, which is not created here.
func writeMissingDestinationConfig(t *testing.T) (string, string) {
t.Helper()
dir := t.TempDir()
configPath := filepath.Join(dir, "config.yml")
indexPath := filepath.Join(dir, "index.sqlite")
contents := fmt.Sprintf(hermeticConfig,
filepath.Join(dir, "source"),
filepath.Join(dir, "missing-store"),
indexPath)
require.NoError(t,
os.WriteFile(configPath, []byte(contents), configFileMode))
// The PID lock lives under xdg.DataHome, which xdg resolves at
// package init; point it at the temp dir so the test neither
// touches nor collides with the real one.
t.Setenv("XDG_DATA_HOME", filepath.Join(dir, "data"))
xdg.Reload()
t.Cleanup(xdg.Reload)
return configPath, indexPath
}
// writeUnusableDestinationConfig builds a config whose destination
// directory does not exist, which fails `remote info`, and whose local
// index is bound to another destination, which fails `prune` and
// `snapshot remove` (a missing destination alone only makes `snapshot
// remove` warn). Returns the config path.
func writeUnusableDestinationConfig(t *testing.T) string {
t.Helper()
configPath, indexPath := writeMissingDestinationConfig(t)
ctx := context.Background()
db, err := database.New(ctx, indexPath)
require.NoError(t, err)
defer func() { require.NoError(t, db.Close()) }()
require.NoError(t, database.NewRepositories(db).LocalMeta.Set(ctx,
database.LocalMetaKeyStorageURL, "file://"+t.TempDir()))
return configPath
}
// runEntry runs Entry with args after the program name and returns its
// exit code and what it wrote to stdout and stderr.
//
// Not parallel-safe: it replaces os.Args, os.Stdout and os.Stderr.
func runEntry(t *testing.T, args ...string) (int, string, string) {
t.Helper()
previousArgs := os.Args
t.Cleanup(func() {
os.Args = previousArgs
rootFlags = RootFlags{}
})
os.Args = append([]string{programName}, args...)
var code int
stdout, stderr := captureProcessStdoutAndStderr(t,
func() { code = Entry() })
return code, stdout, stderr
}
internal/cli/entry_unknown_subcommand_test.go
-43
View File
@@ -1,43 +0,0 @@
package cli //nolint:testpackage // shares runEntry and the command-word constants
import (
"testing"
"github.com/stretchr/testify/assert"
)
// TestEntryGroupRejectsUnknownSubcommand runs each group command with a
// mistyped subcommand, as a crontab line might carry one. It must fail
// with exit 1 and name the word, as `vaultik bogus` does at the top
// level, instead of printing the group's help and exiting 0 having done
// nothing. The group name alone still prints its help.
//
//nolint:paralleltest // replaces os.Args, os.Stdout and os.Stderr
func TestEntryGroupRejectsUnknownSubcommand(t *testing.T) {
for _, testCase := range []struct {
group string
mistyped string
}{
{group: cmdSnapshot, mistyped: "craete"},
{group: "config", mistyped: "inti"},
{group: cmdRemote, mistyped: "ifno"},
{group: "database", mistyped: "delte"},
{group: "completion", mistyped: "nosuchshell"},
} {
t.Run(testCase.group, func(t *testing.T) {
code, stdout, stderr := runEntry(t,
testCase.group, testCase.mistyped)
assert.Equal(t, 1, code)
assert.Empty(t, stdout)
assert.Contains(t, stderr,
`unknown command "`+testCase.mistyped+`"`)
code, stdout, _ = runEntry(t, testCase.group)
assert.Equal(t, 0, code)
assert.Contains(t, stdout,
programName+" "+testCase.group+" [command]")
})
}
}
internal/cli/info.go
+35 -5
View File
@@ -1,7 +1,12 @@
package cli
import (
"context"
"errors"
"os"
"github.com/spf13/cobra"
"go.uber.org/fx"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/vaultik"
)
@@ -28,19 +33,44 @@ func NewInfoCommand() *cobra.Command {
// Use the app framework
rootFlags := GetRootFlags()
return RunOperation(cmd.Context(), AppOptions{
return RunWithApp(cmd.Context(), AppOptions{
ConfigPath: configPath,
LogOptions: log.Options{
Verbose: rootFlags.Verbose,
Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet,
},
Mode: readOnly,
}, func(v *vaultik.Vaultik) error {
return v.ShowInfo()
}, func(err error) {
Modules: []fx.Option{},
Invokes: []fx.Option{
fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
lc.Append(fx.Hook{
OnStart: func(_ context.Context) error {
go func() {
err := v.ShowInfo()
if err != nil {
if !errors.Is(err, context.Canceled) {
log.Error("Failed to show info", "error", err)
ReportErrorf("Failed to show info: %v", err)
os.Exit(1)
}
}
err = v.Shutdowner.Shutdown()
if err != nil {
log.Error("Failed to shutdown", "error", err)
}
}()
return nil
},
OnStop: func(_ context.Context) error {
v.Cancel()
return nil
},
})
}),
},
})
},
}
internal/cli/json_help_test.go
-24
View File
@@ -1,24 +0,0 @@
package cli //nolint:testpackage // exercises the unexported command constructor
import (
"testing"
"github.com/spf13/cobra"
"github.com/stretchr/testify/assert"
)
// TestJSONHelpSaysConfirmationPromptIsSkipped checks that the --json help
// of `snapshot remove` and `prune` says the flag skips the confirmation
// prompt. Both delete without asking under --json, since a prompt on
// stdout would break the JSON document.
func TestJSONHelpSaysConfirmationPromptIsSkipped(t *testing.T) {
t.Parallel()
for _, cmd := range []*cobra.Command{
newSnapshotRemoveCommand(),
NewPruneCommand(),
} {
assert.Contains(t, cmd.Flags().Lookup("json").Usage,
"skips the confirmation prompt", cmd.Name())
}
}
internal/cli/prune.go
+51 -32
View File
@@ -1,7 +1,12 @@
package cli
import (
"context"
"errors"
"os"
"github.com/spf13/cobra"
"go.uber.org/fx"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/vaultik"
)
@@ -13,34 +18,18 @@ func NewPruneCommand() *cobra.Command {
cmd := &cobra.Command{
Use: "prune",
Short: "Tidy local database and remote storage",
Long: `Removes orphaned data from the local index database, and
unreferenced blobs and directories under metadata/ with no manifest
from the backup destination store.
Long: `Removes orphaned data from both the local index database and
unreferenced blobs from the backup destination store.
Needs permission to list, read and delete objects on the destination
store. It does not need the age private key.
Local cleanup drops snapshots whose manifest is gone from the
destination store, incomplete snapshots, and any files, chunks, or
Local cleanup drops incomplete snapshots and any files, chunks, or
blobs no longer referenced by a completed snapshot. Remote cleanup
scans every snapshot manifest in the destination store, builds the
set of still-referenced blob hashes, and deletes any blob not in that
set that is older than prune_grace_period (default 24h). A newer one
is kept: it may belong to a backup still running, on this host or
another sharing the destination store, that has not yet uploaded its
manifest. The grace period does not protect a blob such a backup
reuses from an earlier run that stopped before its manifest: that
blob keeps the time of its first upload, so it can be deleted.
Remote cleanup also deletes each directory under metadata/ with no
manifest, as a backup that stopped before its manifest leaves, once
all its objects are older than prune_grace_period and its snapshot is
not in the local index.
set.
Snapshot create --prune runs the same cleanup automatically; this
command is the manual entry point for the same work (e.g. after a
crashed backup or to reclaim storage). Snapshot remove leaves blobs in
place; run this command afterwards to delete the ones no longer
referenced.`,
Snapshot create --prune and snapshot remove run the same cleanup
automatically; this command is the manual entry point for the same
work (e.g. after a crashed backup or to reclaim storage).`,
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
// Use unified config resolution
@@ -52,27 +41,57 @@ referenced.`,
// Use the app framework like other commands
rootFlags := GetRootFlags()
return RunOperation(cmd.Context(), AppOptions{
return RunWithApp(cmd.Context(), AppOptions{
ConfigPath: configPath,
LogOptions: log.Options{
Verbose: rootFlags.Verbose,
Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet,
JSON: opts.JSON,
Quiet: rootFlags.Quiet || opts.JSON,
},
Mode: mutating,
}, func(v *vaultik.Vaultik) error {
return v.Prune(opts)
}, func(err error) {
Modules: []fx.Option{},
Invokes: []fx.Option{
fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
lc.Append(fx.Hook{
OnStart: func(_ context.Context) error {
// Start the prune operation in a goroutine
go func() {
// Run the prune operation
err := v.Prune(opts)
if err != nil {
if !errors.Is(err, context.Canceled) {
if !opts.JSON {
log.Error("Prune operation failed", "error", err)
ReportErrorf("Prune failed: %v", err)
}
os.Exit(1)
}
}
// Shutdown the app when prune completes
err = v.Shutdowner.Shutdown()
if err != nil {
log.Error("Failed to shutdown", "error", err)
}
}()
return nil
},
OnStop: func(_ context.Context) error {
log.Debug("Stopping prune operation")
v.Cancel()
return nil
},
})
}),
},
})
},
}
cmd.Flags().BoolVar(&opts.Force, "force", false, "Skip confirmation prompt")
cmd.Flags().BoolVar(&opts.JSON, "json", false,
"Output pruning stats as JSON; skips the confirmation prompt, as --force does")
cmd.Flags().BoolVar(&opts.JSON, "json", false, "Output pruning stats as JSON")
return cmd
}
internal/cli/quiet_test.go
-206
View File
@@ -1,206 +0,0 @@
package cli //nolint:testpackage // sets the unexported rootFlags directly
import (
"bytes"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"github.com/spf13/cobra"
)
// setRootFlags overrides the global rootFlags for the duration of one
// test and restores it afterward. These tests must not run in parallel:
// the flags are process-global, so the whole struct is saved and put
// back rather than left mutated for the next test.
func setRootFlags(t *testing.T, f RootFlags) {
t.Helper()
old := rootFlags
rootFlags = f
t.Cleanup(func() { rootFlags = old })
}
// seedFile writes content to a fresh file and returns its path.
func seedFile(t *testing.T, dir, name, content string) string {
t.Helper()
path := filepath.Join(dir, name)
err := os.WriteFile(path, []byte(content), 0o600)
if err != nil {
t.Fatalf("seeding %s: %v", name, err)
}
return path
}
// mustExecute runs a command with its output captured and fails the test
// if it errors, returning what the command printed.
func mustExecute(t *testing.T, cmd *cobra.Command, args ...string) string {
t.Helper()
var out bytes.Buffer
cmd.SetOut(&out)
cmd.SetArgs(args)
err := cmd.Execute()
if err != nil {
t.Fatalf("%s failed: %v", cmd.Name(), err)
}
return out.String()
}
// TestVersionQuietSuppressesReport checks that --quiet silences the whole
// version report: it is human-facing output, not a scriptable value.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestVersionQuietSuppressesReport(t *testing.T) {
setRootFlags(t, RootFlags{Quiet: true})
out := mustExecute(t, NewVersionCommand())
if out != "" {
t.Errorf("--quiet version printed %q, want nothing", out)
}
}
// TestConfigGetIgnoresQuiet checks that a config value is printed even
// under --quiet: it is scriptable output a caller depends on, so --quiet
// must not suppress it, and it stays machine-plain (no marker, no color).
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestConfigGetIgnoresQuiet(t *testing.T) {
dir := t.TempDir()
path := seedFile(t, dir, "config.yml", "storage_url: file:///mnt/x\n")
setRootFlags(t, RootFlags{Quiet: true, ConfigPath: path})
out := mustExecute(t, newConfigGetCommand(), "storage_url")
if out != "file:///mnt/x\n" {
t.Errorf("config get --quiet = %q, want the plain value", out)
}
}
// TestConfigSetQuietSuppressesConfirmation checks that --quiet silences
// the confirmation line while still writing the value to the file.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestConfigSetQuietSuppressesConfirmation(t *testing.T) {
dir := t.TempDir()
path := seedFile(t, dir, "config.yml", "compression_level: 3\n")
setRootFlags(t, RootFlags{Quiet: true, ConfigPath: path})
out := mustExecute(t, newConfigSetCommand(), "compression_level", "9")
if out != "" {
t.Errorf("--quiet config set printed %q, want nothing", out)
}
data, err := os.ReadFile(path) //nolint:gosec // G304: test-controlled path
if err != nil {
t.Fatalf("reading config back: %v", err)
}
if !strings.Contains(string(data), "compression_level: 9") {
t.Errorf("config set did not write the value under --quiet:\n%s", data)
}
}
// TestConfigSetConfirmsWhenNotQuiet checks that the confirmation names
// the key (styled) when --quiet is not set.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestConfigSetConfirmsWhenNotQuiet(t *testing.T) {
dir := t.TempDir()
path := seedFile(t, dir, "config.yml", "compression_level: 3\n")
setRootFlags(t, RootFlags{ConfigPath: path})
out := mustExecute(t, newConfigSetCommand(), "compression_level", "9")
if !strings.Contains(out, "compression_level") {
t.Errorf("config set did not confirm the key: %q", out)
}
}
// TestConfigInitQuietSuppressesConfirmation checks that --quiet silences
// the "config written" confirmation while still writing the file.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestConfigInitQuietSuppressesConfirmation(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "new-config.yml")
setRootFlags(t, RootFlags{Quiet: true, ConfigPath: path})
out := mustExecute(t, newConfigInitCommand())
if out != "" {
t.Errorf("--quiet config init printed %q, want nothing", out)
}
_, err := os.Stat(path)
if err != nil {
t.Errorf("config init did not write the file under --quiet: %v", err)
}
}
// seedDatabaseDeleteConfig writes a valid config whose index_path is a
// seeded database file, and returns both paths.
func seedDatabaseDeleteConfig(t *testing.T, dir string) (string, string) {
t.Helper()
dbPath := seedFile(t, dir, "index.sqlite", "not-a-real-db")
cfg := fmt.Sprintf(hermeticConfig,
filepath.Join(dir, "source"), filepath.Join(dir, "store"), dbPath)
cfgPath := seedFile(t, dir, "config.yml", cfg)
return dbPath, cfgPath
}
// TestDatabaseDeleteQuietSuppressesMessage checks that --quiet silences
// the "database deleted" line while still removing the file.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestDatabaseDeleteQuietSuppressesMessage(t *testing.T) {
dir := t.TempDir()
dbPath, cfgPath := seedDatabaseDeleteConfig(t, dir)
setRootFlags(t, RootFlags{Quiet: true, ConfigPath: cfgPath})
out := mustExecute(t, newDatabaseDeleteCommand(), "--force")
if out != "" {
t.Errorf("--quiet database delete printed %q, want nothing", out)
}
_, err := os.Stat(dbPath)
if !os.IsNotExist(err) {
t.Errorf("database delete did not remove the file: stat err = %v", err)
}
}
// TestDatabaseDeleteReportsWhenNotQuiet checks that the deletion is
// reported when --quiet is not set.
//
//nolint:paralleltest // mutates the process-global rootFlags
func TestDatabaseDeleteReportsWhenNotQuiet(t *testing.T) {
dir := t.TempDir()
_, cfgPath := seedDatabaseDeleteConfig(t, dir)
setRootFlags(t, RootFlags{ConfigPath: cfgPath})
out := mustExecute(t, newDatabaseDeleteCommand(), "--force")
if !strings.Contains(out, "deleted") {
t.Errorf("database delete did not report the deletion: %q", out)
}
}
internal/cli/remote.go
+43 -24
View File
@@ -1,9 +1,12 @@
package cli
import (
"context"
"errors"
"os"
"github.com/spf13/cobra"
"go.uber.org/fx"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/vaultik"
)
@@ -18,12 +21,6 @@ func NewRemoteCommand() *cobra.Command {
Use: "remote",
Short: "Remote storage management commands",
Long: "Commands for inspecting and managing remote storage",
// cobra checks Args only on a command that has a Run: without
// one, a mistyped subcommand printed help and exited 0.
Args: cobra.NoArgs,
Run: func(cmd *cobra.Command, _ []string) {
_ = cmd.Help()
},
}
// Add subcommands
@@ -40,20 +37,18 @@ func newRemoteNukeCommand() *cobra.Command {
cmd := &cobra.Command{
Use: "nuke",
Short: "Delete ALL snapshot metadata and blobs from the backup destination store",
Long: `Removes every snapshot's metadata, every blob and the destination
marker from the backup destination store, and each removed snapshot from
the local index. After this command completes successfully the bucket
prefix is empty and the next backup starts from scratch.
Long: `Removes every snapshot's metadata and every blob from remote
storage. After this command completes successfully the bucket prefix is
empty and the next backup starts from scratch.
This is destructive and irreversible. Requires --force, and permission
to list and delete objects on the destination store.`,
This is destructive and irreversible. Requires --force.`,
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
if !force {
return errNukeNeedsForce
}
return runVaultikApp(cmd, mutating, false, "Remote nuke failed",
return runVaultikApp(cmd, false, false, "Remote nuke failed",
func(v *vaultik.Vaultik) error {
return v.NukeRemote(true)
})
@@ -77,10 +72,7 @@ func newRemoteInfoCommand() *cobra.Command {
- Size of all snapshot metadata (per snapshot and total)
- Count and total size of all blobs
- Count and size of referenced blobs (from all manifests)
- Count and size of orphaned blobs (not referenced by any manifest)
Needs permission to list and read the destination store. It does not
need the age private key.`,
- Count and size of orphaned blobs (not referenced by any manifest)`,
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
// Use unified config resolution
@@ -91,20 +83,47 @@ need the age private key.`,
rootFlags := GetRootFlags()
return RunOperation(cmd.Context(), AppOptions{
return RunWithApp(cmd.Context(), AppOptions{
ConfigPath: configPath,
LogOptions: log.Options{
Verbose: rootFlags.Verbose,
Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet,
JSON: jsonOutput,
Quiet: rootFlags.Quiet || jsonOutput,
},
Mode: readOnly,
}, func(v *vaultik.Vaultik) error {
return v.RemoteInfo(jsonOutput)
}, func(err error) {
Modules: []fx.Option{},
Invokes: []fx.Option{
fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
lc.Append(fx.Hook{
OnStart: func(_ context.Context) error {
go func() {
err := v.RemoteInfo(jsonOutput)
if err != nil {
if !errors.Is(err, context.Canceled) {
if !jsonOutput {
log.Error("Failed to get remote info", "error", err)
ReportErrorf("Failed to get remote info: %v", err)
}
os.Exit(1)
}
}
err = v.Shutdowner.Shutdown()
if err != nil {
log.Error("Failed to shutdown", "error", err)
}
}()
return nil
},
OnStop: func(_ context.Context) error {
v.Cancel()
return nil
},
})
}),
},
})
},
}
internal/cli/root.go
+7 -42
View File
@@ -9,7 +9,6 @@ import (
"github.com/adrg/xdg"
"github.com/spf13/cobra"
"sneak.berlin/go/vaultik/internal/ui"
)
// errConfigNotFound is wrapped by all config-resolution failures.
@@ -35,10 +34,9 @@ func NewRootCommand() *cobra.Command {
cmd := &cobra.Command{
Use: "vaultik",
Short: "Secure incremental backup tool with asymmetric encryption",
Long: `vaultik is an incremental backup tool that encrypts data to age public
keys and uploads it to an S3-compatible object store, a local directory or
an rclone remote. The backed-up machine needs no private key: only
'snapshot restore' and 'snapshot verify --deep' need one.`,
Long: `vaultik is a secure incremental backup tool that encrypts data using age
public keys and uploads to S3-compatible storage. No private keys are needed
on the source system.`,
SilenceUsage: true,
// Bare 'vaultik' (no subcommand): print help. The banner is
// printed once at process startup by Entry, before cobra
@@ -51,20 +49,16 @@ an rclone remote. The backed-up machine needs no private key: only
// Add global flags
cmd.PersistentFlags().StringVar(&rootFlags.ConfigPath, "config", "",
"Path to config file (default: $VAULTIK_CONFIG, then the platform "+
"config directory, then /etc/vaultik/config.yml)")
"Path to config file (default: $VAULTIK_CONFIG or platform config dir)")
cmd.PersistentFlags().BoolVarP(&rootFlags.Verbose, "verbose", "v", false,
"Enable verbose output")
cmd.PersistentFlags().BoolVar(&rootFlags.Debug, "debug", false,
"Enable debug output")
cmd.PersistentFlags().BoolVarP(&rootFlags.Quiet, "quiet", "q", false,
"Suppress status output (warnings, errors, JSON documents, "+
"'config get' values and confirmation prompts are still printed)")
"Suppress non-error output")
cmd.PersistentFlags().BoolVar(&rootFlags.SkipErrors, "skip-errors", false,
"Skip files that cannot be read when creating a snapshot, or "+
"that cannot be restored when restoring, instead of aborting "+
"(packing and storage errors while creating a snapshot still "+
"abort)")
"Continue past per-file errors instead of aborting "+
"(applies to snapshot create and restore)")
// Add subcommands
cmd.AddCommand(
@@ -77,21 +71,6 @@ an rclone remote. The backed-up machine needs no private key: only
NewDatabaseCommand(),
)
// cobra adds the completion command only when Execute runs, and with
// no Run, so it skipped its own cobra.NoArgs and a mistyped shell name
// printed help and exited 0. Adding it here lets it have a Run. Its
// scripts go to os.Stdout as it is now: a later SetOut does not
// redirect them.
cmd.InitDefaultCompletionCmd()
for _, sub := range cmd.Commands() {
if sub.Name() == "completion" {
sub.Run = func(completion *cobra.Command, _ []string) {
_ = completion.Help()
}
}
}
return cmd
}
@@ -101,20 +80,6 @@ func GetRootFlags() RootFlags {
return rootFlags
}
// commandUI returns a UI writer for a command's stdout, in quiet mode
// when the global --quiet flag is set. This is how the pure-cli
// commands (version, config, database) reach internal/ui: color follows
// the writer (a TTY gets color, a captured test buffer does not), and
// --quiet silences the same message classes it silences everywhere else.
func commandUI(cmd *cobra.Command) *ui.Writer {
w := ui.New(cmd.OutOrStdout())
if GetRootFlags().Quiet {
w.SetQuiet(true)
}
return w
}
// ResolveConfigPath resolves the config file path from flags, environment, or default.
// Search order: --config flag, VAULTIK_CONFIG env, XDG config dir,
// /etc/vaultik/config.yml.
internal/cli/runapp_interrupt_test.go
-97
View File
@@ -1,97 +0,0 @@
package cli_test
import (
"context"
"os"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/require"
"go.uber.org/fx"
"sneak.berlin/go/vaultik/internal/cli"
)
// TestRunAppWaitsForOperationCleanupOnShutdown drives RunApp with an fx app
// wired the way RunOperation wires a command: a single lifecycle hook whose
// OnStart launches the operation in its own goroutine and whose OnStop cancels
// it and blocks until that goroutine returns. The operation stands in for a
// restore blocked mid-download — it holds a decrypted "scratch" file and only
// removes it as it unwinds on cancellation.
//
// The app is asked to stop once the operation is running (standing in for an
// OS interrupt; fx delivers a real signal and Shutdowner.Shutdown() on the
// same app.Wait channel, so both drive the identical shutdown path). RunApp
// must not return until app.Stop has run the OnStop hook, so the scratch file
// must be gone by the time RunApp returns. Before the fix RunApp returned as
// soon as the app.Wait/Done channel fired, without running app.Stop, so the
// cleanup never ran and this file would still be on disk (issue #159).
func TestRunAppWaitsForOperationCleanupOnShutdown(t *testing.T) {
t.Parallel()
scratch := filepath.Join(t.TempDir(), "decrypted-scratch")
require.NoError(t, os.WriteFile(scratch, []byte("secret"), 0o600))
// Cancel and reap the operation even if RunApp returns without doing so
// (the buggy path), so the goroutine cannot leak past the test.
opCtx, opCancel := context.WithCancel(context.Background())
t.Cleanup(opCancel)
var stop func(context.Context) bool
app := fx.New(
fx.NopLogger,
fx.Invoke(func(lc fx.Lifecycle, sh fx.Shutdowner) {
lc.Append(fx.Hook{
OnStart: func(_ context.Context) error {
done := make(chan struct{})
go func() {
defer close(done)
// Blocked mid-operation until cancelled, then run the
// cleanup an interrupted restore would run.
<-opCtx.Done()
_ = os.Remove(scratch)
}()
stop = func(ctx context.Context) bool {
opCancel()
select {
case <-done:
return true
case <-ctx.Done():
return false
}
}
// Ask the app to stop now that the operation is running.
go func() { _ = sh.Shutdown() }()
return nil
},
OnStop: func(ctx context.Context) error {
stop(ctx)
return nil
},
})
}),
)
done := make(chan error, 1)
go func() { done <- cli.RunApp(context.Background(), app) }()
select {
case err := <-done:
require.NoError(t, err)
case <-time.After(30 * time.Second):
t.Fatal("RunApp did not return after shutdown was requested")
}
_, err := os.Stat(scratch)
require.True(t, os.IsNotExist(err),
"RunApp returned before the operation removed its decrypted scratch file")
}
internal/cli/snapshot.go
+102 -141
View File
@@ -1,10 +1,13 @@
package cli
import (
"context"
"errors"
"fmt"
"os"
"github.com/spf13/cobra"
"go.uber.org/fx"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/vaultik"
)
@@ -41,12 +44,6 @@ func NewSnapshotCommand() *cobra.Command {
Use: "snapshot",
Short: "Snapshot management commands",
Long: "Commands for creating, listing, and managing snapshots",
// cobra checks Args only on a command that has a Run: without
// one, a mistyped subcommand printed help and exited 0.
Args: cobra.NoArgs,
Run: func(cmd *cobra.Command, _ []string) {
_ = cmd.Help()
},
}
// Add subcommands
@@ -60,21 +57,6 @@ func NewSnapshotCommand() *cobra.Command {
return cmd
}
// snapshotCreateLong is the long help text of 'snapshot create'.
const snapshotCreateLong = `Creates new snapshots of the configured directories.
If snapshot names are provided, only those snapshots are created.
If no names are provided, all configured snapshots are created.
Needs at least one public key in age_recipients, and permission to write
objects at the backup destination store. It does not need the age private
key. With --prune it also needs permission to list, read and delete objects
there.
The config is read from the path given by --config or VAULTIK_CONFIG;
otherwise from the platform config directory (~/.config/vaultik/config.yml
on Linux), then /etc/vaultik/config.yml.`
// newSnapshotCreateCommand creates the 'snapshot create' subcommand
func newSnapshotCreateCommand() *cobra.Command {
opts := &vaultik.SnapshotCreateOptions{}
@@ -82,14 +64,15 @@ func newSnapshotCreateCommand() *cobra.Command {
cmd := &cobra.Command{
Use: "create [snapshot-names...]",
Short: "Create new snapshots",
Long: snapshotCreateLong,
Long: `Creates new snapshots of the configured directories.
If snapshot names are provided, only those snapshots are created.
If no names are provided, all configured snapshots are created.
Config is located at /etc/vaultik/config.yml by default, but can be overridden by
specifying a path using --config or by setting VAULTIK_CONFIG to a path.`,
Args: cobra.ArbitraryArgs,
RunE: func(cmd *cobra.Command, args []string) error {
err := opts.Validate()
if err != nil {
return err
}
// Pass snapshot names from args
opts.Snapshots = args
// --skip-errors is a global flag on the root command.
@@ -103,8 +86,7 @@ func newSnapshotCreateCommand() *cobra.Command {
// Use the backup functionality from cli package
rootFlags := GetRootFlags()
// --cron suppression is wired through v.UI by setupGlobals.
return RunOperation(cmd.Context(), AppOptions{
return RunWithApp(cmd.Context(), AppOptions{
ConfigPath: configPath,
LogOptions: log.Options{
Verbose: rootFlags.Verbose,
@@ -112,46 +94,54 @@ func newSnapshotCreateCommand() *cobra.Command {
Cron: opts.Cron,
Quiet: rootFlags.Quiet,
},
Mode: mutating,
}, func(v *vaultik.Vaultik) error {
return v.CreateSnapshot(opts)
}, func(err error) {
if errors.Is(err, vaultik.ErrPostBackupPruneFailed) {
log.Error("Post-backup prune failed", "error", err)
ReportErrorf("Backup completed, but %v", err)
return
}
if errors.Is(err, vaultik.ErrFilesUnreadable) ||
errors.Is(err, vaultik.ErrSourcePathEmpty) ||
errors.Is(err, vaultik.ErrVolumeNotMounted) {
log.Error("Backup completed with an error", "error", err)
ReportErrorf("Backup completed, but %v", err)
return
}
Modules: []fx.Option{},
Invokes: []fx.Option{
fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
lc.Append(fx.Hook{
OnStart: func(_ context.Context) error {
// Start the snapshot creation in a goroutine
go func() {
// --cron suppression is wired through v.UI by setupGlobals.
err := v.CreateSnapshot(opts)
if err != nil {
if !errors.Is(err, context.Canceled) {
log.Error("Snapshot creation failed", "error", err)
ReportErrorf("Snapshot creation failed: %v", err)
os.Exit(1)
}
}
// Shutdown the app when snapshot completes
err = v.Shutdowner.Shutdown()
if err != nil {
log.Error("Failed to shutdown", "error", err)
}
}()
return nil
},
OnStop: func(_ context.Context) error {
log.Debug("Stopping snapshot creation")
// Cancel the Vaultik context
v.Cancel()
return nil
},
})
}),
},
})
},
}
cmd.Flags().BoolVar(&opts.Cron, "cron", false,
"Run in cron mode (silent unless warning or error)")
"Run in cron mode (silent unless error)")
cmd.Flags().BoolVar(&opts.Prune, "prune", false,
"After backup, drop older snapshots of the same name, remove "+
"orphaned blobs older than prune_grace_period, and delete each "+
"directory under metadata/ with no manifest once all its "+
"objects are older than prune_grace_period and its snapshot is "+
"not in the local index (not after a backup that skipped "+
"unreadable files, found a configured path empty, or did not "+
"find a mount point an earlier snapshot found)")
"After backup, drop older snapshots of the same name and remove "+
"orphaned blobs")
cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "",
"With --prune: keep snapshots newer than this duration "+
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+
"instead of only the latest")
"(e.g. 4w, 30d, 6mo) instead of only the latest")
return cmd
}
@@ -164,17 +154,10 @@ func newSnapshotListCommand() *cobra.Command {
Use: "list",
Aliases: []string{"ls"},
Short: "List all snapshots",
Long: `Lists the snapshots in the local index and on the backup destination
store, with their ID, timestamp, and sizes.
Needs permission to list and read the destination store. It does not need
the age private key. A snapshot found only on the destination store is
listed by its remote key, because its ID is stored only in the local index
and in the snapshot's encrypted database. If the destination store cannot
be listed, the command warns and lists the local index alone.`,
Long: "Lists all snapshots with their ID, timestamp, and compressed size",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
return runVaultikApp(cmd, readOnly, false,
return runVaultikApp(cmd, false, false,
"Failed to list snapshots",
func(v *vaultik.Vaultik) error {
return v.ListSnapshots(jsonOutput)
@@ -194,19 +177,11 @@ func newSnapshotPurgeCommand() *cobra.Command {
cmd := &cobra.Command{
Use: "purge",
Short: "Purge old snapshots",
Long: `Removes snapshots based on age or count criteria. Exactly one of
--keep-latest and --older-than is required.
Long: `Removes snapshots based on age or count criteria.
Retention is per-snapshot-name: --keep-latest keeps the latest snapshot of
each name, not the latest globally. Use --snapshot to restrict the
operation to specific snapshot names.
Only snapshots in the local index are removed. Each one is removed from
the local index and its metadata is deleted from the backup destination
store. Blobs are not touched: run 'vaultik prune' afterwards.
Needs permission to list and delete objects on the destination store. If
the destination store cannot be listed, nothing is removed.`,
Retention is per-snapshot-name: --keep-latest keeps the latest of each
configured snapshot name, not the latest globally. Use --snapshot to
restrict the operation to specific snapshot names.`,
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
// Validate flags
@@ -218,7 +193,7 @@ the destination store cannot be listed, nothing is removed.`,
return errPurgeCriteriaBoth
}
return runVaultikApp(cmd, mutating, false,
return runVaultikApp(cmd, false, false,
"Failed to purge snapshots",
func(v *vaultik.Vaultik) error {
return v.PurgeSnapshotsWithOptions(opts)
@@ -229,8 +204,7 @@ the destination store cannot be listed, nothing is removed.`,
cmd.Flags().BoolVar(&opts.KeepLatest, "keep-latest", false,
"Keep only the latest snapshot of each name")
cmd.Flags().StringVar(&opts.OlderThan, "older-than", "",
"Remove snapshots older than duration "+
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months)")
"Remove snapshots older than duration (e.g., 30d, 6m, 1y)")
cmd.Flags().BoolVar(&opts.Force, "force", false, "Skip confirmation prompt")
cmd.Flags().StringArrayVar(&opts.Names, "snapshot", nil,
"Restrict to snapshots with these names (repeat for multiple)")
@@ -244,30 +218,8 @@ func newSnapshotVerifyCommand() *cobra.Command {
cmd := &cobra.Command{
Use: "verify <snapshot-id>",
Short: "Check a snapshot's blobs are present with the listed size",
Long: `Checks that every blob the snapshot's manifest lists is present
in storage with the size the manifest records, and that the
snapshot's encrypted database is present. It does not read blob
contents and does not need the age private key.
Needs permission to read the backup destination store, and to list it
when a remote key is given.
Use --deep to download, decrypt, and re-hash every blob to detect
corruption -- integrity, not who wrote it.
--deep requires the age private key, in the VAULTIK_AGE_SECRET_KEY
environment variable or as age_secret_key in the config file; the
environment variable wins. The variable may hold the whole age-keygen
file, with its comments and all of its keys. Read the key from the
file rather than typing it, so that it does not land in your shell
history:
export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"
The snapshot may be named by its ID or, on a host with no local
index, by the remote key that 'snapshot list' prints for a
remote-only snapshot (an unambiguous leading part is enough).`,
Short: "Verify snapshot integrity",
Long: "Verifies that all blobs referenced in a snapshot exist",
Args: requireSnapshotIDArg,
RunE: func(cmd *cobra.Command, args []string) error {
snapshotID := args[0]
@@ -280,32 +232,52 @@ remote-only snapshot (an unambiguous leading part is enough).`,
rootFlags := GetRootFlags()
return RunOperation(cmd.Context(), AppOptions{
return RunWithApp(cmd.Context(), AppOptions{
ConfigPath: configPath,
LogOptions: log.Options{
Verbose: rootFlags.Verbose,
Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet,
JSON: opts.JSON,
Quiet: rootFlags.Quiet || opts.JSON,
},
Mode: readOnly,
}, func(v *vaultik.Vaultik) error {
return v.VerifySnapshotWithOptions(snapshotID, opts)
}, func(err error) {
if opts.JSON {
return
}
Modules: []fx.Option{},
Invokes: []fx.Option{
fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
lc.Append(fx.Hook{
OnStart: func(_ context.Context) error {
go func() {
err := v.VerifySnapshotWithOptions(snapshotID, opts)
if err != nil {
if !errors.Is(err, context.Canceled) {
if !opts.JSON {
log.Error("Verification failed", "error", err)
ReportErrorf("Verification failed: %v", err)
}
os.Exit(1)
}
}
err = v.Shutdowner.Shutdown()
if err != nil {
log.Error("Failed to shutdown", "error", err)
}
}()
return nil
},
OnStop: func(_ context.Context) error {
v.Cancel()
return nil
},
})
}),
},
})
},
}
cmd.Flags().BoolVar(&opts.Deep, "deep", false,
"Download, decrypt and re-hash every blob; requires the age private "+
"key in VAULTIK_AGE_SECRET_KEY, or as age_secret_key in the config "+
"file")
cmd.Flags().BoolVar(&opts.Deep, "deep", false, "Download and verify blob hashes")
cmd.Flags().BoolVar(&opts.JSON, "json", false, "Output verification results as JSON")
return cmd
@@ -325,31 +297,21 @@ By default, this removes the snapshot from the local index database and
strips the snapshot's metadata from the backup destination store. Blobs
are NOT touched: deleting them requires reading every remaining remote
manifest (the destination store may hold snapshots this host doesn't
know about), which is what 'vaultik prune' does. The command prints the
'vaultik prune' invocation to run as a follow-up only when it removed the
snapshot's metadata from the destination store, and never under --json.
The argument must be a snapshot ID: a remote key is not accepted. An ID
that names no snapshot in the local index or on the destination store is
an error. Needs permission to list and delete objects on the destination
store, unless --local-only is given.
know about), which is what 'vaultik prune' does. On success the command
prints the exact 'vaultik prune' invocation to run as a follow-up.
Use --local-only to skip the remote half (e.g. when you want to forget a
snapshot locally without touching the destination store); the snapshot
must then be in the local index.
snapshot locally without touching the destination store).
If removing the snapshot's metadata from the destination store fails (for
example because the remote is unreachable), the local-database removal
still completes and a warning is emitted; run 'vaultik snapshot remove
<snapshot-id>' again once the destination store is reachable to remove
the snapshot's metadata from it ('vaultik prune' does not). For a
snapshot that is only on the destination store, that failure is an error.
If the remote is unreachable, the local-database removal still completes
and a warning is emitted; rerun 'vaultik prune' once the destination store
is reachable to finish remote cleanup.
To wipe the entire destination store and start over, use 'vaultik remote
nuke --force' — it is the single supported entry point for that.`,
Args: requireSnapshotIDArg,
RunE: func(cmd *cobra.Command, args []string) error {
return runVaultikApp(cmd, mutating, opts.JSON,
return runVaultikApp(cmd, opts.JSON, opts.JSON,
"Failed to remove snapshot",
func(v *vaultik.Vaultik) error {
_, err := v.RemoveSnapshot(args[0], opts)
@@ -362,8 +324,7 @@ nuke --force' — it is the single supported entry point for that.`,
cmd.Flags().BoolVarP(&opts.Force, "force", "f", false, "Skip confirmation prompt")
cmd.Flags().BoolVar(&opts.DryRun, "dry-run", false,
"Show what would be removed without removing")
cmd.Flags().BoolVar(&opts.JSON, "json", false,
"Output result as JSON; skips the confirmation prompt, as --force does")
cmd.Flags().BoolVar(&opts.JSON, "json", false, "Output result as JSON")
cmd.Flags().BoolVar(&opts.LocalOnly, "local-only", false,
"Skip remote cleanup; only touch the local index")
internal/cli/snapshot_restore.go
+83 -29
View File
@@ -1,8 +1,16 @@
package cli
import (
"context"
"errors"
"os"
"github.com/spf13/cobra"
"go.uber.org/fx"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/globals"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/vaultik"
)
@@ -17,6 +25,15 @@ type RestoreOptions struct {
Verify bool // Verify restored files after restore
}
// RestoreApp contains all dependencies needed for restore
type RestoreApp struct {
Globals *globals.Globals
Config *config.Config
Storage storage.Storer
Vaultik *vaultik.Vaultik
Shutdowner fx.Shutdowner
}
// newSnapshotRestoreCommand creates the 'snapshot restore' subcommand
func newSnapshotRestoreCommand() *cobra.Command {
opts := &RestoreOptions{}
@@ -30,27 +47,9 @@ This command will restore files from the specified snapshot to the
target directory.
If no paths are specified, all files are restored.
If paths are specified, only matching files/directories are restored.
A path that matches nothing in the snapshot is named in a warning, and
the command exits non-zero after restoring the paths that match.
The snapshot may be named by its ID or, when restoring on a host with no
local index, by the remote key that 'snapshot list' prints for a
remote-only snapshot (an unambiguous leading part is enough).
Each file is restored under its original absolute path inside the target
directory. File owners are restored only when the command runs as root.
Requires the age private key, in the VAULTIK_AGE_SECRET_KEY environment
variable or as age_secret_key in the config file; the environment
variable wins. The variable may hold the whole age-keygen file, with its
comments and all of its keys. Read the key from the file rather than
typing it, so that it does not land in your shell history:
export VAULTIK_AGE_SECRET_KEY="$(cat vaultik_backup_private_key.txt)"
Also needs permission to read the backup destination store (and to list
it when a remote key is given), and free space in the temporary directory
for the decrypted blobs.
Requires the VAULTIK_AGE_SECRET_KEY environment variable to be set with
the age private key.
Examples:
# Restore entire snapshot
@@ -78,8 +77,7 @@ Examples:
return cmd
}
// runRestore parses arguments and runs the restore operation through the
// app framework.
// runRestore parses arguments and runs the restore operation through the app framework
func runRestore(cmd *cobra.Command, args []string, opts *RestoreOptions) error {
snapshotID := args[0]
@@ -88,31 +86,87 @@ func runRestore(cmd *cobra.Command, args []string, opts *RestoreOptions) error {
opts.Paths = args[restoreMinArgs:]
}
// Use unified config resolution
configPath, err := ResolveConfigPath()
if err != nil {
return err
}
// Use the app framework like other commands
rootFlags := GetRootFlags()
return RunOperation(cmd.Context(), AppOptions{
return RunWithApp(cmd.Context(), AppOptions{
ConfigPath: configPath,
LogOptions: log.Options{
Verbose: rootFlags.Verbose,
Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet,
},
Mode: readOnly,
}, func(v *vaultik.Vaultik) error {
return v.Restore(&vaultik.RestoreOptions{
Modules: buildRestoreModules(),
Invokes: buildRestoreInvokes(snapshotID, opts),
})
}
// buildRestoreModules returns the fx.Options for dependency injection in restore
func buildRestoreModules() []fx.Option {
return []fx.Option{
fx.Provide(fx.Annotate(
func(g *globals.Globals, cfg *config.Config,
storer storage.Storer, v *vaultik.Vaultik, shutdowner fx.Shutdowner) *RestoreApp {
return &RestoreApp{
Globals: g,
Config: cfg,
Storage: storer,
Vaultik: v,
Shutdowner: shutdowner,
}
},
)),
}
}
// buildRestoreInvokes returns the fx.Options that wire up the restore lifecycle
func buildRestoreInvokes(snapshotID string, opts *RestoreOptions) []fx.Option {
return []fx.Option{
fx.Invoke(func(app *RestoreApp, lc fx.Lifecycle) {
lc.Append(fx.Hook{
OnStart: func(_ context.Context) error {
// Start the restore operation in a goroutine
go func() {
// Run the restore operation
restoreOpts := &vaultik.RestoreOptions{
SnapshotID: snapshotID,
TargetDir: opts.TargetDir,
Paths: opts.Paths,
Verify: opts.Verify,
SkipErrors: rootFlags.SkipErrors,
})
}, func(err error) {
SkipErrors: GetRootFlags().SkipErrors,
}
err := app.Vaultik.Restore(restoreOpts)
if err != nil {
if !errors.Is(err, context.Canceled) {
log.Error("Restore operation failed", "error", err)
ReportErrorf("Restore failed: %v", err)
os.Exit(1)
}
}
// Shutdown the app when restore completes
err = app.Shutdowner.Shutdown()
if err != nil {
log.Error("Failed to shutdown", "error", err)
}
}()
return nil
},
OnStop: func(_ context.Context) error {
log.Debug("Stopping restore operation")
app.Vaultik.Cancel()
return nil
},
})
}),
}
}
internal/cli/snapshot_restore_test.go
-37
View File
@@ -1,37 +0,0 @@
package cli //nolint:testpackage // exercises the unexported command constructor
import (
"strings"
"testing"
"github.com/spf13/pflag"
)
// TestRestoreCommandDoesNotTakeKeyAsArgument guards the fix for the age
// key being echoed on the command line: restore must take the key only
// from the environment, never as a flag value, and its help must show the
// file-based form rather than a literal key that would land in shell
// history.
func TestRestoreCommandDoesNotTakeKeyAsArgument(t *testing.T) {
t.Parallel()
cmd := newSnapshotRestoreCommand()
cmd.Flags().VisitAll(func(f *pflag.Flag) {
lower := strings.ToLower(f.Name)
for _, banned := range []string{"key", "secret", "age", "identity"} {
if strings.Contains(lower, banned) {
t.Errorf("restore must not accept the key as a flag; found --%s", f.Name)
}
}
})
help := cmd.Long
if strings.Contains(help, "AGE-SECRET-KEY-") {
t.Error("restore help must not show a literal age private key to type")
}
if !strings.Contains(help, "$(cat ") {
t.Error("restore help should read the key from a file, e.g. $(cat ...)")
}
}
internal/cli/vaultik_snapshot_types.go
+12
View File
@@ -0,0 +1,12 @@
package cli
import "time"
// SnapshotInfo represents snapshot information for listing
//
//nolint:tagliatelle // snake_case is the established output format
type SnapshotInfo struct {
ID string `json:"id"`
Timestamp time.Time `json:"timestamp"`
CompressedSize int64 `json:"compressed_size"`
}
internal/cli/version.go
+21 -37
View File
@@ -2,11 +2,11 @@ package cli
import (
"fmt"
"os"
"runtime"
"github.com/spf13/cobra"
"sneak.berlin/go/vaultik/internal/globals"
"sneak.berlin/go/vaultik/internal/ui"
)
// NewVersionCommand creates the version command
@@ -16,44 +16,28 @@ func NewVersionCommand() *cobra.Command {
Short: "Print version information",
Long: `Print version, git commit, and build information for vaultik.`,
Args: cobra.NoArgs,
Run: func(cmd *cobra.Command, _ []string) {
writeVersion(commandUI(cmd))
Run: func(_ *cobra.Command, _ []string) {
_, _ = fmt.Fprintf(os.Stdout, "vaultik %s\n", globals.Version)
_, _ = fmt.Fprintf(os.Stdout, " commit: %s\n", globals.Commit)
_, _ = fmt.Fprintf(os.Stdout, " build date: %s\n", globals.CommitDate)
_, _ = fmt.Fprintf(os.Stdout, " go: %s\n", runtime.Version())
_, _ = fmt.Fprintf(os.Stdout, " os/arch: %s/%s\n",
runtime.GOOS, runtime.GOARCH)
_, _ = fmt.Fprintf(os.Stdout, " author: %s\n", globals.Author)
_, _ = fmt.Fprintf(os.Stdout, " homepage: %s\n", globals.Homepage)
_, _ = fmt.Fprintf(os.Stdout, " license: %s\n", globals.License)
if globals.Version == "dev" {
_, _ = fmt.Fprintln(os.Stdout)
_, _ = fmt.Fprintln(os.Stdout,
"This is a development build (no version information embedded).")
_, _ = fmt.Fprintln(os.Stdout,
"Build a release binary with 'make vaultik' or download from")
_, _ = fmt.Fprintln(os.Stdout,
"https://sneak.berlin/go/vaultik for embedded version metadata.")
}
},
}
return cmd
}
// writeVersion prints the version report through the UI writer. The
// report is the output this command exists to produce, so it is written
// plain (markers would corrupt the aligned report) via the writer's
// underlying stdout; --quiet silences it like any other non-error
// output.
func writeVersion(out *ui.Writer) {
if out.Quiet() {
return
}
w := out.Out()
_, _ = fmt.Fprintf(w, "vaultik %s\n", globals.Version)
_, _ = fmt.Fprintf(w, " commit: %s\n", globals.Commit)
_, _ = fmt.Fprintf(w, " build date: %s\n", globals.CommitDate)
_, _ = fmt.Fprintf(w, " go: %s\n", runtime.Version())
_, _ = fmt.Fprintf(w, " os/arch: %s/%s\n", runtime.GOOS, runtime.GOARCH)
_, _ = fmt.Fprintf(w, " author: %s\n", globals.Author)
_, _ = fmt.Fprintf(w, " homepage: %s\n", globals.Homepage)
_, _ = fmt.Fprintf(w, " license: %s\n", globals.License)
if globals.IsDevVersion(globals.Version) {
_, _ = fmt.Fprintln(w)
_, _ = fmt.Fprintln(w,
"This is a development build: it was not built from a tagged")
_, _ = fmt.Fprintln(w,
"commit, so it carries no release version. Released binaries")
_, _ = fmt.Fprintf(w,
"are published at %s\n", globals.ReleasesURL)
_, _ = fmt.Fprintln(w,
"and report their tag on the first line above.")
}
}
internal/cli/version_test.go
-77
View File
@@ -1,77 +0,0 @@
package cli_test
import (
"bytes"
"strings"
"testing"
"sneak.berlin/go/vaultik/internal/cli"
"sneak.berlin/go/vaultik/internal/globals"
)
// runVersionCommand executes `vaultik version` with its output
// captured, and returns what it printed.
func runVersionCommand(t *testing.T) string {
t.Helper()
cmd := cli.NewVersionCommand()
var out bytes.Buffer
cmd.SetOut(&out)
cmd.SetErr(&out)
cmd.SetArgs([]string{})
err := cmd.Execute()
if err != nil {
t.Fatalf("version command failed: %v", err)
}
return out.String()
}
// TestVersionCommandReportsBuildVersion checks that the first line of
// the report is the version the binary was actually built with. The
// test binary carries no -ldflags, so that is the "dev" default -- the
// same string an untagged `make vaultik` build stamps a prefix of.
//
//nolint:paralleltest // executes a command that reads the global rootFlags
func TestVersionCommandReportsBuildVersion(t *testing.T) {
out := runVersionCommand(t)
wantFirst := "vaultik " + globals.Version
if first, _, _ := strings.Cut(out, "\n"); first != wantFirst {
t.Errorf("first line = %q, want %q", first, wantFirst)
}
if !strings.Contains(out, "commit:") {
t.Error("output does not report the commit")
}
}
// TestVersionCommandFlagsDevelopmentBuild is the regression test for
// the thing this command exists to prevent: a build that is not a
// release must say so. The notice used to be gated on the version
// being exactly "dev", so once untagged builds started carrying their
// commit sha it would have gone silent and an unreleased binary would
// have looked like a release.
//
//nolint:paralleltest // executes a command that reads the global rootFlags
func TestVersionCommandFlagsDevelopmentBuild(t *testing.T) {
if !globals.IsDevVersion(globals.Version) {
t.Skipf("test binary was stamped with release version %q",
globals.Version)
}
out := runVersionCommand(t)
if !strings.Contains(out, "development build") {
t.Errorf("dev build did not print the development-build notice:\n%s",
out)
}
if !strings.Contains(out, globals.ReleasesURL) {
t.Errorf("development-build notice does not point at %s:\n%s",
globals.ReleasesURL, out)
}
}
internal/config/config.go
+65 -198
View File
@@ -4,73 +4,47 @@
package config
import (
"bytes"
"errors"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"time"
"filippo.io/age"
"git.eeqj.de/sneak/smartconfig"
"github.com/adrg/xdg"
"go.uber.org/fx"
"gopkg.in/yaml.v3"
"sneak.berlin/go/vaultik/internal/chunker"
"sneak.berlin/go/vaultik/internal/log"
)
const appName = "vaultik"
// secretKeyPrefix marks an age secret (private) key. It is compared
// case-insensitively so a recipient entry that is actually a private key is
// caught and never passed to age or echoed back.
//
//nolint:gosec // G101: marker for detecting a pasted secret key, not a credential
const secretKeyPrefix = "AGE-SECRET-KEY-"
// Defaults and validation bounds for tunable settings. MB and GB are
// 1000-based, as the size parser reads them; the S3 part sizes are S3's own
// limits, which are 1024-based (MiB, GiB).
// Defaults and validation bounds for tunable settings.
const (
defaultBlobSizeLimit = Size(10 * 1000 * 1000 * 1000) // 10GB
defaultChunkSize = Size(10 * 1000 * 1000) // 10MB
defaultS3PartSize = Size(5 * 1024 * 1024) // 5MiB
defaultBlobSizeLimit = Size(10 * 1024 * 1024 * 1024) // 10GB
defaultChunkSize = Size(10 * 1024 * 1024) // 10MB
defaultS3PartSize = Size(5 * 1024 * 1024) // 5MB
defaultCompressionLevel = 3
defaultPruneGracePeriod = 24 * time.Hour
minChunkSize = 1000 * 1000 // 1MB
minChunkSize = 1024 * 1024 // 1MB
minCompressionLevel = 1
maxCompressionLevel = 19
// S3 accepts a multipart upload part from 5MiB to 5GiB.
minS3PartSize = 5 * 1024 * 1024
maxS3PartSize = 5 * 1024 * 1024 * 1024
)
// Sentinel validation errors.
var (
errNoConfigPath = errors.New("config path not provided")
errRecipientIsSecretKey = errors.New(
"an age secret key was given where a public key (age1...) belongs")
errRecipientNotX25519 = errors.New(
"not a valid recipient; only X25519 age1... public keys are supported")
errNoAgeRecipients = errors.New(
"at least one age_recipient is required (generate with: age-keygen)")
errNoSnapshots = errors.New(
"at least one snapshot must be configured (see config.example.yml)")
errSnapshotNoPaths = errors.New("snapshot must have at least one path")
errChunkSizeTooSmall = errors.New("chunk_size must be at least 1MB")
errBlobSizeTooSmall = errors.New(
"blob_size_limit must be at least the largest chunk the chunker can " +
"emit (chunk_size times the FastCDC size spread)")
errBlobSizeTooSmall = errors.New("blob_size_limit must be at least chunk_size")
errBadCompression = errors.New("compression_level must be between 1 and 19")
errBadS3PartSize = errors.New("s3.part_size must be between 5MiB and 5GiB")
errBadStorageScheme = errors.New(
"storage_url must start with s3://, file://, or rclone://")
errFileURLNotAbsolute = errors.New(
"a file:// storage_url needs an absolute path, " +
"as in file:///path/to/backup")
errNegativePruneGracePeriod = errors.New(
"prune_grace_period must not be negative")
errStorageNotConfigured = errors.New(
"storage not configured; set storage_url or provide s3.endpoint + " +
"s3.bucket + credentials")
@@ -147,26 +121,6 @@ func (c *Config) SnapshotNames() []string {
return names
}
// Names of the two places the age secret key can be configured, used by
// AgeSecretKeySourceName for error messages that must not echo the value.
//
//nolint:gosec // G101: these are the names of the config sources, not a key
const (
ageSecretKeySourceEnv = "VAULTIK_AGE_SECRET_KEY"
ageSecretKeySourceConfig = "age_secret_key"
)
// AgeSecretKeySourceName returns the human name of where AgeSecretKey was
// configured. A Config built directly (as in tests) has no recorded
// source, so it reports the config-file field name.
func (c *Config) AgeSecretKeySourceName() string {
if c.AgeSecretKeySource != "" {
return c.AgeSecretKeySource
}
return ageSecretKeySourceConfig
}
// Config represents the application configuration for Vaultik.
// It defines all settings for backup operations, including source directories,
// encryption recipients, storage configuration, and performance tuning parameters.
@@ -176,11 +130,6 @@ func (c *Config) AgeSecretKeySourceName() string {
type Config struct {
AgeRecipients []string `yaml:"age_recipients"`
AgeSecretKey string `yaml:"age_secret_key"`
// AgeSecretKeySource names where AgeSecretKey was configured
// ("VAULTIK_AGE_SECRET_KEY" or "age_secret_key") so a later parse
// failure can name the source without echoing the secret value. It is
// set by Load and never read from or written to the config file.
AgeSecretKeySource string `yaml:"-"`
BlobSizeLimit Size `yaml:"blob_size_limit"`
ChunkSize Size `yaml:"chunk_size"`
// Exclude holds global excludes applied to all snapshots.
@@ -191,15 +140,6 @@ type Config struct {
Snapshots map[string]SnapshotConfig `yaml:"snapshots"`
CompressionLevel int `yaml:"compression_level"`
// PruneGracePeriod is how old a blob no manifest lists must be before
// a prune deletes it, and how old every object in a directory under
// metadata/ with no manifest must be before a prune deletes that
// directory. A backup writes its manifest only once it has uploaded
// all its blobs and its db.zst.age, so a younger blob or directory may
// belong to a backup still running, on this host or another sharing
// the destination store.
PruneGracePeriod time.Duration `yaml:"prune_grace_period"`
// StorageURL specifies the storage backend using a URL format.
// Takes precedence over S3Config if set.
// Supported formats:
@@ -222,9 +162,7 @@ type S3Config struct {
AccessKeyID string `yaml:"access_key_id"`
SecretAccessKey string `yaml:"secret_access_key"`
Region string `yaml:"region"`
// UseSSL selects HTTPS for a scheme-less endpoint. Omitted (nil) means
// the default, TLS; set it to false only to force plain HTTP.
UseSSL *bool `yaml:"use_ssl"`
UseSSL bool `yaml:"use_ssl"`
PartSize Size `yaml:"part_size"`
}
@@ -249,48 +187,14 @@ func New(path Path) (*Config, error) {
return cfg, nil
}
// Load reads the configuration file at path, parses it with Parse, warns
// when others can read the file, and validates the configuration.
// Returns an error if the file cannot be read, parsed, or if validation
// fails.
// Load reads and parses the configuration file from the specified path.
// It applies default values for optional fields, performs environment variable
// substitution using smartconfig, and validates the configuration.
// The configuration file should be in YAML format. Returns an error if the file
// cannot be read, parsed, or if validation fails.
func Load(path string) (*Config, error) {
data, err := os.ReadFile(path) //nolint:gosec // G304: config path is operator-supplied
if err != nil {
return nil, fmt.Errorf("failed to load config file: %w", err)
}
cfg, err := Parse(data)
if err != nil {
return nil, err
}
// Check config file permissions (warn if world or group readable)
//nolint:gosec // G703: config path is operator-supplied by design
info, statErr := os.Stat(path)
if statErr == nil {
mode := info.Mode().Perm()
if mode&0044 != 0 { // group or world readable
log.Warn(cfg.readableByOthersWarning(),
"path", path,
"mode", fmt.Sprintf("%04o", mode),
"recommendation", "chmod 600 "+path)
}
}
err = cfg.Validate()
if err != nil {
return nil, fmt.Errorf("invalid config: %w", err)
}
return cfg, nil
}
// Parse reads YAML config data as Load does, without validating it. It
// applies default values for optional fields, fills in ${...} references
// using smartconfig, and expands ~ in paths.
func Parse(data []byte) (*Config, error) {
// Load config using smartconfig for interpolation
sc, err := smartconfig.NewFromReader(bytes.NewReader(data))
sc, err := smartconfig.NewFromConfigPath(path)
if err != nil {
return nil, fmt.Errorf("failed to load config file: %w", err)
}
@@ -301,8 +205,6 @@ func Parse(data []byte) (*Config, error) {
ChunkSize: defaultChunkSize,
IndexPath: filepath.Join(xdg.DataHome, appName, "index.sqlite"),
CompressionLevel: defaultCompressionLevel,
PruneGracePeriod: defaultPruneGracePeriod,
S3: S3Config{PartSize: defaultS3PartSize},
}
// Convert smartconfig data to YAML then unmarshal
@@ -336,7 +238,10 @@ func Parse(data []byte) (*Config, error) {
cfg.IndexPath = expandTilde(envIndexPath)
}
cfg.setAgeSecretKey()
// Check for environment variable override for AgeSecretKey
if envAgeSecretKey := os.Getenv("VAULTIK_AGE_SECRET_KEY"); envAgeSecretKey != "" {
cfg.AgeSecretKey = extractAgeSecretKey(envAgeSecretKey)
}
// Get hostname if not set
if cfg.Hostname == "" {
@@ -353,32 +258,43 @@ func Parse(data []byte) (*Config, error) {
cfg.S3.Region = "us-east-1"
}
if cfg.S3.PartSize == 0 {
cfg.S3.PartSize = defaultS3PartSize
}
// Check config file permissions (warn if world or group readable)
//nolint:gosec // G703: config path is operator-supplied by design
info, statErr := os.Stat(path)
if statErr == nil {
mode := info.Mode().Perm()
if mode&0044 != 0 { // group or world readable
log.Warn("Config file has insecure permissions (contains S3 credentials)",
"path", path,
"mode", fmt.Sprintf("%04o", mode),
"recommendation", "chmod 600 "+path)
}
}
err = cfg.Validate()
if err != nil {
return nil, fmt.Errorf("invalid config: %w", err)
}
return cfg, nil
}
// Validate checks if the configuration is valid and complete.
// It ensures all required fields are present and have valid values:
// - Every age recipient must parse as an X25519 age1... public key (so a
// bad entry fails at load, not mid-backup); errors name the position,
// never the value. An empty list is accepted, because only snapshot
// create needs a recipient and it checks for one itself
// - At least one age recipient must be specified
// - At least one snapshot must be configured with at least one path
// - Storage must be configured (either storage_url or s3.* fields)
// - Chunk size must be at least 1MB
// - Blob size limit must be at least the largest chunk the chunker can emit
// (chunk_size times chunker.ChunkSizeSpread), so a single-chunk blob never
// exceeds the configured limit
// - Blob size limit must be at least the chunk size
// - Compression level must be between 1 and 19
// - S3 part size must be between 5MiB and 5GiB, the part sizes S3 accepts
// - Prune grace period must not be negative
//
// Returns an error describing the first validation failure encountered.
func (c *Config) Validate() error {
for i, recipient := range c.AgeRecipients {
err := validateAgeRecipient(recipient)
if err != nil {
return fmt.Errorf("age_recipients[%d]: %w", i, err)
}
if len(c.AgeRecipients) == 0 {
return errNoAgeRecipients
}
if len(c.Snapshots) == 0 {
@@ -401,13 +317,8 @@ func (c *Config) Validate() error {
return errChunkSizeTooSmall
}
// The chunker can emit chunks up to chunk_size * ChunkSizeSpread, and the
// packer places a single such chunk into an otherwise empty blob. A limit
// below that bound would let a blob exceed it, so reject it.
largestChunk := c.ChunkSize.Int64() * chunker.ChunkSizeSpread
if c.BlobSizeLimit.Int64() < largestChunk {
return fmt.Errorf("%w: need at least %d bytes",
errBlobSizeTooSmall, largestChunk)
if c.BlobSizeLimit.Int64() < c.ChunkSize.Int64() {
return errBlobSizeTooSmall
}
if c.CompressionLevel < minCompressionLevel ||
@@ -415,62 +326,9 @@ func (c *Config) Validate() error {
return errBadCompression
}
if c.S3.PartSize.Int64() < minS3PartSize ||
c.S3.PartSize.Int64() > maxS3PartSize {
return errBadS3PartSize
}
if c.PruneGracePeriod < 0 {
return errNegativePruneGracePeriod
}
return nil
}
// validateAgeRecipient parses one age_recipients entry with the age library
// and returns a value-free error on failure. A recipient string can be
// sensitive (an operator may paste a secret key by mistake), so neither the
// entry nor age's own error (which quotes its input) is ever included.
func validateAgeRecipient(recipient string) error {
if strings.HasPrefix(strings.ToUpper(recipient), secretKeyPrefix) {
return errRecipientIsSecretKey
}
_, err := age.ParseX25519Recipient(recipient)
if err != nil {
return errRecipientNotX25519
}
return nil
}
// setAgeSecretKey records the age secret key and where it came from. The
// value is stored raw and parsed only where decryption happens
// (internal/vaultik), so backup, list and prune keep working whatever the
// field holds. The environment variable overrides the config-file field.
func (c *Config) setAgeSecretKey() {
if c.AgeSecretKey != "" {
c.AgeSecretKeySource = ageSecretKeySourceConfig
}
if env := os.Getenv("VAULTIK_AGE_SECRET_KEY"); env != "" {
c.AgeSecretKey = env
c.AgeSecretKeySource = ageSecretKeySourceEnv
}
}
// readableByOthersWarning is the warning Load logs when others can read
// the config file. It says "may contain" because the S3 credentials are
// seen only after smartconfig has replaced any ${...} reference in the
// file with its value, so a set credential need not be in the file.
func (c *Config) readableByOthersWarning() string {
if c.S3.AccessKeyID != "" || c.S3.SecretAccessKey != "" {
return "Config file is readable by others and may contain S3 credentials"
}
return "Config file is readable by others"
}
// validateStorage validates storage configuration.
// If StorageURL is set, it takes precedence. S3 URLs require credentials.
// File URLs don't require any S3 configuration.
@@ -502,18 +360,11 @@ func (c *Config) validateStorage() error {
// validateStorageURL validates URL-based storage configuration. File and
// rclone URLs need no credentials; S3 URLs require the legacy s3.*
// credential fields. A file URL must name an absolute path: a relative one
// would resolve against the working directory, so the destination would
// move with it.
// credential fields.
func (c *Config) validateStorageURL() error {
switch {
case strings.HasPrefix(c.StorageURL, "file://"):
// storage.ParseStorageURL applies the same rule, but config
// cannot import storage, which imports config.
if !strings.HasPrefix(c.StorageURL, "file:///") {
return errFileURLNotAbsolute
}
// File storage doesn't need S3 credentials
return nil
case strings.HasPrefix(c.StorageURL, "rclone://"):
// Rclone storage uses rclone's own config
@@ -534,6 +385,22 @@ func (c *Config) validateStorageURL() error {
}
}
// extractAgeSecretKey extracts the AGE-SECRET-KEY from the input using
// the age library's parser, which handles comments and whitespace.
func extractAgeSecretKey(input string) string {
identities, err := age.ParseIdentities(strings.NewReader(input))
if err != nil || len(identities) == 0 {
// Fall back to trimmed input if parsing fails
return strings.TrimSpace(input)
}
// Return the string representation of the first identity
if id, ok := identities[0].(*age.X25519Identity); ok {
return id.String()
}
return strings.TrimSpace(input)
}
// Module exports the config module for fx dependency injection.
// It provides the Config type to other modules in the application.
//
internal/config/config_test.go
+38 -630
View File
@@ -1,15 +1,9 @@
package config //nolint:testpackage // exercises unexported source constants
package config //nolint:testpackage // exercises unexported extractAgeSecretKey
import (
"errors"
"os"
"path/filepath"
"strings"
"testing"
"time"
"sneak.berlin/go/vaultik/internal/chunker"
"sneak.berlin/go/vaultik/internal/log"
)
const (
@@ -89,48 +83,6 @@ func TestConfigLoad(t *testing.T) {
}
}
// TestExampleConfigIsScrubbedAndLoads checks that the shipped
// config.example.yml carries only neutral placeholders (no real credentials,
// private addresses, or internal host names) and still parses.
func TestExampleConfigIsScrubbedAndLoads(t *testing.T) {
t.Parallel()
examplePath := filepath.Join("..", "..", "config.example.yml")
cfg, err := Load(examplePath)
if err != nil {
t.Fatalf("Failed to load config.example.yml: %v", err)
}
if cfg.StorageURL != "rclone://myremote/path/to/backups" {
t.Errorf("Expected neutral storage_url, got '%s'", cfg.StorageURL)
}
//nolint:gosec // G304: examplePath is a fixed in-repo path, not user input
raw, err := os.ReadFile(examplePath)
if err != nil {
t.Fatalf("Failed to read config.example.yml: %v", err)
}
text := string(raw)
wantSubstrings := []string{
"YOUR_ACCESS_KEY",
"YOUR_SECRET_KEY",
"endpoint: https://",
}
for _, want := range wantSubstrings {
if !strings.Contains(text, want) {
t.Errorf("Expected config.example.yml to contain %q", want)
}
}
// A raw "http://" scheme would mean a plaintext, likely private endpoint.
if strings.Contains(text, "http://") {
t.Error("config.example.yml should not contain an http:// endpoint")
}
}
// TestConfigFromEnv tests loading config path from environment variable
func TestConfigFromEnv(t *testing.T) {
t.Parallel()
@@ -149,58 +101,53 @@ func TestConfigFromEnv(t *testing.T) {
}
}
// TestValidateBlobSizeLimit checks the blob_size_limit boundary: it must be at
// least the largest chunk the chunker can emit (chunk_size times
// chunker.ChunkSizeSpread), because the packer places a single such chunk into
// an otherwise empty blob. A limit between chunk_size and that bound is rejected.
func TestValidateBlobSizeLimit(t *testing.T) {
// TestExtractAgeSecretKey tests extraction of AGE-SECRET-KEY from various inputs
func TestExtractAgeSecretKey(t *testing.T) {
t.Parallel()
const chunkSize = Size(10 * 1024 * 1024) // 10MB
largestChunk := chunkSize.Int64() * chunker.ChunkSizeSpread
newConfig := func(blobLimit Size) *Config {
return &Config{
AgeRecipients: []string{testSneakAgePublicKey},
Snapshots: map[string]SnapshotConfig{"test": {Paths: []string{"/tmp/src"}}},
StorageURL: "file:///tmp/vaultik-test-store",
ChunkSize: chunkSize,
BlobSizeLimit: blobLimit,
CompressionLevel: 3,
S3: S3Config{PartSize: defaultS3PartSize},
}
}
tests := []struct {
name string
blobLimit Size
wantErr bool
input string
expected string
}{
{
name: "at chunk_size but below largest chunk is rejected",
blobLimit: chunkSize,
wantErr: true,
name: "plain key",
input: testIntegrationAgePrivateKey,
expected: testIntegrationAgePrivateKey,
},
{
name: "between chunk_size and largest chunk is rejected",
blobLimit: Size(chunkSize.Int64() * 2),
wantErr: true,
name: "key with trailing newline",
input: testIntegrationAgePrivateKey + "\n",
expected: testIntegrationAgePrivateKey,
},
{
name: "one byte below largest chunk is rejected",
blobLimit: Size(largestChunk - 1),
wantErr: true,
name: "full age-keygen output",
input: "# created: 2025-01-14T12:00:00Z\n" +
"# public key: " + testIntegrationAgePublicKey + "\n" +
testIntegrationAgePrivateKey + "\n",
expected: testIntegrationAgePrivateKey,
},
{
name: "exactly at largest chunk is accepted",
blobLimit: Size(largestChunk),
wantErr: false,
name: "age-keygen output with extra blank lines",
input: "# created: 2025-01-14T12:00:00Z\n" +
"# public key: " + testIntegrationAgePublicKey + "\n\n" +
testIntegrationAgePrivateKey + "\n\n",
expected: testIntegrationAgePrivateKey,
},
{
name: "above largest chunk is accepted",
blobLimit: Size(largestChunk * 100),
wantErr: false,
name: "key with leading whitespace",
input: " " + testIntegrationAgePrivateKey + " ",
expected: testIntegrationAgePrivateKey,
},
{
name: "empty input",
input: "",
expected: "",
},
{
name: "only comments",
input: "# this is a comment\n# another comment",
expected: "# this is a comment\n# another comment",
},
}
@@ -208,549 +155,10 @@ func TestValidateBlobSizeLimit(t *testing.T) {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
err := newConfig(tt.blobLimit).Validate()
if tt.wantErr {
if !errors.Is(err, errBlobSizeTooSmall) {
t.Fatalf("Validate() error = %v, want errBlobSizeTooSmall", err)
}
return
}
if err != nil {
t.Fatalf("Validate() unexpected error: %v", err)
}
})
}
}
// TestValidateS3PartSize checks that s3.part_size is held to the part sizes
// S3 accepts, 5MiB to 5GiB, by changing only the part size of the test
// config. "5MB" in the config file is 5,000,000 bytes, below the minimum.
func TestValidateS3PartSize(t *testing.T) {
t.Parallel()
base, err := Load(os.Getenv("VAULTIK_CONFIG"))
if err != nil {
t.Fatalf("Failed to load config: %v", err)
}
tests := []struct {
name string
partSize Size
wantErr bool
}{
{
name: "5MB is rejected",
partSize: 5_000_000,
wantErr: true,
},
{
name: "one byte below 5MiB is rejected",
partSize: minS3PartSize - 1,
wantErr: true,
},
{
name: "5MiB is accepted",
partSize: minS3PartSize,
wantErr: false,
},
{
name: "5GiB is accepted",
partSize: maxS3PartSize,
wantErr: false,
},
{
name: "one byte above 5GiB is rejected",
partSize: maxS3PartSize + 1,
wantErr: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
cfg := *base
cfg.S3.PartSize = tt.partSize
err := cfg.Validate()
if tt.wantErr {
if !errors.Is(err, errBadS3PartSize) {
t.Fatalf("Validate() error = %v, want errBadS3PartSize", err)
}
return
}
if err != nil {
t.Fatalf("Validate() unexpected error: %v", err)
}
})
}
}
// TestLoadS3PartSize checks that a config file without s3.part_size loads
// with the 5MiB default, and that an explicit 0 fails at load like any other
// part size S3 refuses.
func TestLoadS3PartSize(t *testing.T) {
t.Parallel()
const withoutPartSize = "snapshots:\n" +
" test:\n" +
" paths: [/tmp/vaultik-test-source]\n" +
"storage_url: file:///tmp/vaultik-test-storage\n"
writeConfig := func(t *testing.T, text string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "config.yml")
err := os.WriteFile(path, []byte(text), 0o600)
if err != nil {
t.Fatalf("write config: %v", err)
}
return path
}
t.Run("absent loads as 5MiB", func(t *testing.T) {
t.Parallel()
cfg, err := Load(writeConfig(t, withoutPartSize))
if err != nil {
t.Fatalf("Load() unexpected error: %v", err)
}
if cfg.S3.PartSize != defaultS3PartSize {
t.Errorf("s3.part_size = %d, want %d",
cfg.S3.PartSize, defaultS3PartSize)
}
})
t.Run("0 is rejected", func(t *testing.T) {
t.Parallel()
_, err := Load(writeConfig(t, withoutPartSize+"s3:\n part_size: 0\n"))
if !errors.Is(err, errBadS3PartSize) {
t.Fatalf("Load() error = %v, want errBadS3PartSize", err)
}
})
}
// TestDocumentedSizeDefaults checks that the defaults are the documented
// chunk_size 10MB and blob_size_limit 10GB as the parser reads those values
// (1000-based), so writing a default out in the config file changes nothing.
func TestDocumentedSizeDefaults(t *testing.T) {
t.Parallel()
cfg, err := Parse([]byte("chunk_size: 10MB\nblob_size_limit: 10GB\n"))
if err != nil {
t.Fatalf("Parse() unexpected error: %v", err)
}
if cfg.ChunkSize != defaultChunkSize {
t.Errorf("chunk_size: 10MB is %d bytes, the default is %d",
cfg.ChunkSize, defaultChunkSize)
}
if cfg.BlobSizeLimit != defaultBlobSizeLimit {
t.Errorf("blob_size_limit: 10GB is %d bytes, the default is %d",
cfg.BlobSizeLimit, defaultBlobSizeLimit)
}
}
// TestDocumentedSizeExamples checks the 1MB minimum chunk_size and that
// every size the README, config.example.yml and the config init template
// list as an example is accepted.
func TestDocumentedSizeExamples(t *testing.T) {
t.Parallel()
const withoutSizes = "snapshots:\n" +
" test:\n" +
" paths: [/tmp/vaultik-test-source]\n" +
"storage_url: file:///tmp/vaultik-test-storage\n"
tests := []struct {
setting string
wantErr error
}{
{"chunk_size: 1MB", nil},
{"chunk_size: 999999", errChunkSizeTooSmall},
{"chunk_size: 5M", nil},
{"chunk_size: 10M", nil},
{"chunk_size: 10MB", nil},
{"chunk_size: 64MiB", nil},
{"chunk_size: 1GB", nil},
// Four times the default chunk_size, the documented minimum.
{"blob_size_limit: 40MB", nil},
{"blob_size_limit: 500MB", nil},
{"blob_size_limit: 1GB", nil},
{"blob_size_limit: 1GiB", nil},
{"blob_size_limit: 10G", nil},
{"blob_size_limit: 10GB", nil},
{"s3: {part_size: 5MiB}", nil},
{"s3: {part_size: 10MB}", nil},
{"s3: {part_size: 16MiB}", nil},
{"s3: {part_size: 100MiB}", nil},
{"s3: {part_size: 5GiB}", nil},
}
for _, tt := range tests {
t.Run(tt.setting, func(t *testing.T) {
t.Parallel()
cfg, err := Parse([]byte(withoutSizes + tt.setting + "\n"))
if err != nil {
t.Fatalf("Parse() unexpected error: %v", err)
}
err = cfg.Validate()
if !errors.Is(err, tt.wantErr) {
t.Fatalf("Validate() error = %v, want %v", err, tt.wantErr)
}
})
}
}
// TestValidatePruneGracePeriod checks that a negative prune_grace_period
// is refused rather than acting like 0s, which deletes unreferenced blobs
// and directories under metadata/ with no manifest at once, and that 0s
// and the documented examples are accepted.
func TestValidatePruneGracePeriod(t *testing.T) {
t.Parallel()
const withoutGracePeriod = "snapshots:\n" +
" test:\n" +
" paths: [/tmp/vaultik-test-source]\n" +
"storage_url: file:///tmp/vaultik-test-storage\n"
tests := []struct {
setting string
wantErr error
}{
{"prune_grace_period: 0s", nil},
{"prune_grace_period: 90m", nil},
{"prune_grace_period: 36h", nil},
{"prune_grace_period: -1h", errNegativePruneGracePeriod},
}
for _, tt := range tests {
t.Run(tt.setting, func(t *testing.T) {
t.Parallel()
cfg, err := Parse([]byte(withoutGracePeriod + tt.setting + "\n"))
if err != nil {
t.Fatalf("Parse() unexpected error: %v", err)
}
err = cfg.Validate()
if !errors.Is(err, tt.wantErr) {
t.Fatalf("Validate() error = %v, want %v", err, tt.wantErr)
}
})
}
}
// TestLoadPruneGracePeriod checks that a config file without
// prune_grace_period loads with the documented 24h. The value is written out
// here rather than taken from defaultPruneGracePeriod, so that a change of the
// default fails this test.
func TestLoadPruneGracePeriod(t *testing.T) {
t.Parallel()
const withoutGracePeriod = "snapshots:\n" +
" test:\n" +
" paths: [/tmp/vaultik-test-source]\n" +
"storage_url: file:///tmp/vaultik-test-storage\n"
path := filepath.Join(t.TempDir(), "config.yml")
err := os.WriteFile(path, []byte(withoutGracePeriod), 0o600)
if err != nil {
t.Fatalf("write config: %v", err)
}
cfg, err := Load(path)
if err != nil {
t.Fatalf("Load() unexpected error: %v", err)
}
if cfg.PruneGracePeriod != 24*time.Hour {
t.Errorf("prune_grace_period = %s, want 24h", cfg.PruneGracePeriod)
}
}
// TestLoadRejectsRelativeFileURL checks that a file:// storage_url with a
// relative path fails at load. Accepted, it would resolve against the
// working directory, so a backup run from another directory would write
// to a different destination and still succeed.
func TestLoadRejectsRelativeFileURL(t *testing.T) {
t.Parallel()
const relativeConfig = "snapshots:\n" +
" test:\n" +
" paths: [/tmp/vaultik-test-source]\n" +
"storage_url: file://dest\n"
path := filepath.Join(t.TempDir(), "config.yml")
err := os.WriteFile(path, []byte(relativeConfig), 0o600)
if err != nil {
t.Fatalf("write config: %v", err)
}
_, err = Load(path)
if !errors.Is(err, errFileURLNotAbsolute) {
t.Fatalf("Load() error = %v, want errFileURLNotAbsolute", err)
}
if !strings.Contains(err.Error(), "absolute path") {
t.Errorf("error %q does not say to use an absolute path", err.Error())
}
}
// TestValidateAgeRecipients checks that recipients are parsed at config load
// (a bad entry fails immediately, not mid-backup) and that no invalid entry —
// least of all a pasted secret key — is echoed in the error. An empty list
// loads, because only snapshot create needs a recipient.
func TestValidateAgeRecipients(t *testing.T) {
t.Parallel()
baseConfig := func(recipients []string) *Config {
return &Config{
AgeRecipients: recipients,
Snapshots: map[string]SnapshotConfig{"test": {Paths: []string{"/tmp/src"}}},
StorageURL: "file:///tmp/vaultik-test-store",
ChunkSize: Size(10 * 1024 * 1024),
BlobSizeLimit: Size(10 * 1024 * 1024 * 1024),
CompressionLevel: 3,
S3: S3Config{PartSize: defaultS3PartSize},
}
}
tests := []struct {
name string
recipients []string
wantErr bool
}{
{
name: "no recipients is accepted",
recipients: nil,
wantErr: false,
},
{
name: "placeholder recipient is rejected",
recipients: []string{"age1REPLACE_WITH_YOUR_PUBLIC_KEY"},
wantErr: true,
},
{
name: "ssh-ed25519 recipient is rejected",
recipients: []string{"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIexamplekeydata"},
wantErr: true,
},
{
name: "truncated age1 string is rejected",
recipients: []string{"age1short"},
wantErr: true,
},
{
name: "secret key passed as recipient is rejected",
recipients: []string{testIntegrationAgePrivateKey},
wantErr: true,
},
{
name: "two valid recipients are accepted",
recipients: []string{testSneakAgePublicKey, testIntegrationAgePublicKey},
wantErr: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
err := baseConfig(tt.recipients).Validate()
if !tt.wantErr {
if err != nil {
t.Fatalf("Validate() unexpected error: %v", err)
}
return
}
if err == nil {
t.Fatal("Validate() returned nil, want error")
}
// The entry itself must never appear in the error, since a
// recipient string can be a secret key.
for _, recipient := range tt.recipients {
if strings.Contains(err.Error(), recipient) {
t.Fatalf("Validate() error echoed the recipient value: %v", err)
}
}
})
}
}
// TestAgeSecretKeySourceName checks the name reported for the configured
// age secret key: the recorded source when Load set one, and the
// config-file field name for a Config built directly (as in tests).
func TestAgeSecretKeySourceName(t *testing.T) {
t.Parallel()
tests := []struct {
name string
source string
want string
}{
{
name: "unset defaults to config field",
source: "",
want: ageSecretKeySourceConfig,
},
{
name: "environment source",
source: ageSecretKeySourceEnv,
want: ageSecretKeySourceEnv,
},
{
name: "config-file source",
source: ageSecretKeySourceConfig,
want: ageSecretKeySourceConfig,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
cfg := &Config{AgeSecretKeySource: tt.source}
if got := cfg.AgeSecretKeySourceName(); got != tt.want {
t.Errorf("AgeSecretKeySourceName() = %q, want %q", got, tt.want)
}
})
}
}
// loadReadableConfig writes configYAML to a file that others can read,
// loads it, and returns what the logger wrote to stderr meanwhile. The
// logger writes to the os.Stderr it finds when it is initialized, so
// os.Stderr is pointed at a file first. Not parallel-safe: os.Stderr and
// the logger are process-global.
func loadReadableConfig(t *testing.T, configYAML string) string {
t.Helper()
dir := t.TempDir()
configPath := filepath.Join(dir, "config.yml")
stderrPath := filepath.Join(dir, "stderr")
err := os.WriteFile(configPath, []byte(configYAML), 0o600)
if err != nil {
t.Fatalf("writing config: %v", err)
}
//nolint:gosec // G302: the test needs a config file others can read
err = os.Chmod(configPath, 0o644)
if err != nil {
t.Fatalf("chmod config: %v", err)
}
stderrFile, err := os.Create(stderrPath) //nolint:gosec // G304: test temp path
if err != nil {
t.Fatalf("creating stderr file: %v", err)
}
previous := os.Stderr
os.Stderr = stderrFile
log.Initialize(log.Config{})
_, loadErr := Load(configPath)
os.Stderr = previous
log.Initialize(log.Config{})
_ = stderrFile.Close()
if loadErr != nil {
t.Fatalf("Load() error = %v", loadErr)
}
captured, err := os.ReadFile(stderrPath) //nolint:gosec // G304: test temp path
if err != nil {
t.Fatalf("reading stderr file: %v", err)
}
return string(captured)
}
// TestLoadWarnsReadableConfigWithoutS3Credentials checks that a config
// file others can read, holding no S3 credentials, is warned about
// without a claim that it holds them.
//
//nolint:paralleltest // loadReadableConfig replaces os.Stderr
func TestLoadWarnsReadableConfigWithoutS3Credentials(t *testing.T) {
stderr := loadReadableConfig(t, `
storage_url: file:///var/backups/vaultik
snapshots:
home:
paths:
- /home
`)
if !strings.Contains(stderr, "Config file is readable by others") {
t.Errorf("expected a warning that the file is readable by others, got %q",
stderr)
}
if strings.Contains(stderr, "S3 credentials") {
t.Errorf("warning names S3 credentials the file does not set: %q", stderr)
}
}
// TestLoadWarnsReadableConfigWithS3Credentials checks that a config file
// others can read and that sets S3 credentials, as values or as ${ENV:...}
// references, is warned about as one that may contain them.
//
//nolint:paralleltest // loadReadableConfig replaces os.Stderr
func TestLoadWarnsReadableConfigWithS3Credentials(t *testing.T) {
t.Setenv("VAULTIK_TEST_ACCESS_KEY_ID", "test-access-key")
t.Setenv("VAULTIK_TEST_SECRET_ACCESS_KEY", "test-secret-key")
configs := map[string]string{
"values": `
storage_url: s3://bucket/prefix?endpoint=s3.example.com
s3:
access_key_id: test-access-key
secret_access_key: test-secret-key
snapshots:
home:
paths:
- /home
`,
"references": `
storage_url: s3://bucket/prefix?endpoint=s3.example.com
s3:
access_key_id: ${ENV:VAULTIK_TEST_ACCESS_KEY_ID}
secret_access_key: ${ENV:VAULTIK_TEST_SECRET_ACCESS_KEY}
snapshots:
home:
paths:
- /home
`,
}
for name, configYAML := range configs {
t.Run(name, func(t *testing.T) {
stderr := loadReadableConfig(t, configYAML)
if !strings.Contains(stderr,
"Config file is readable by others and may contain S3 credentials") {
t.Errorf("expected a warning naming the S3 credentials, got %q",
stderr)
result := extractAgeSecretKey(tt.input)
if result != tt.expected {
t.Errorf("extractAgeSecretKey(%q) = %q, want %q",
tt.input, result, tt.expected)
}
})
}
internal/config/size.go
+2
View File
@@ -16,6 +16,8 @@ var (
// Size represents a byte size that can be specified in configuration files.
// It can unmarshal from both numeric values (interpreted as bytes) and
// human-readable strings like "10MB", "2.5GB", or "1TB".
//
//nolint:recvcheck // UnmarshalYAML requires a pointer; String/Int64 are value reads
type Size int64
// UnmarshalYAML implements yaml.Unmarshaler for Size, allowing it to be
internal/crypto/encryption.go
+224
View File
@@ -0,0 +1,224 @@
// Package crypto provides thread-safe age encryption and decryption
// helpers used to protect blob and metadata content.
package crypto //nolint:revive,nolintlint // stdlib crypto unused; see #76
import (
"bytes"
"errors"
"fmt"
"io"
"sync"
"filippo.io/age"
"go.uber.org/fx"
)
// ErrNoRecipients is returned when an encryptor is created or updated
// without any recipient public keys.
var ErrNoRecipients = errors.New("at least one recipient is required")
// Encryptor provides thread-safe encryption using the age encryption library.
// It supports encrypting data for multiple recipients simultaneously, allowing
// any of the corresponding private keys to decrypt the data. This is useful
// for backup scenarios where multiple parties should be able to decrypt the data.
type Encryptor struct {
recipients []age.Recipient
mu sync.RWMutex
}
// NewEncryptor creates a new encryptor with the given age public keys.
// Each public key should be a valid age X25519 recipient string (e.g., "age1...")
// At least one recipient must be provided. Returns an error if any of the
// public keys are invalid or if no recipients are specified.
func NewEncryptor(publicKeys []string) (*Encryptor, error) {
if len(publicKeys) == 0 {
return nil, ErrNoRecipients
}
recipients := make([]age.Recipient, 0, len(publicKeys))
for _, key := range publicKeys {
recipient, err := age.ParseX25519Recipient(key)
if err != nil {
return nil, fmt.Errorf("parsing age recipient %s: %w", key, err)
}
recipients = append(recipients, recipient)
}
return &Encryptor{
recipients: recipients,
}, nil
}
// Encrypt encrypts data using age encryption for all configured recipients.
// The encrypted data can be decrypted by any of the corresponding private keys.
// This method is suitable for small to medium amounts of data that fit in memory.
// For large data streams, use EncryptStream or EncryptWriter instead.
func (e *Encryptor) Encrypt(data []byte) ([]byte, error) {
e.mu.RLock()
recipients := e.recipients
e.mu.RUnlock()
var buf bytes.Buffer
// Create encrypted writer for all recipients
w, err := age.Encrypt(&buf, recipients...)
if err != nil {
return nil, fmt.Errorf("creating encrypted writer: %w", err)
}
// Write data
_, err = w.Write(data)
if err != nil {
return nil, fmt.Errorf("writing encrypted data: %w", err)
}
// Close to flush
err = w.Close()
if err != nil {
return nil, fmt.Errorf("closing encrypted writer: %w", err)
}
return buf.Bytes(), nil
}
// EncryptStream encrypts data from reader to writer using age encryption.
// This method is suitable for encrypting large files or streams as it processes
// data in a streaming fashion without loading everything into memory.
// The encrypted data is written directly to the destination writer.
func (e *Encryptor) EncryptStream(dst io.Writer, src io.Reader) error {
e.mu.RLock()
recipients := e.recipients
e.mu.RUnlock()
// Create encrypted writer for all recipients
w, err := age.Encrypt(dst, recipients...)
if err != nil {
return fmt.Errorf("creating encrypted writer: %w", err)
}
// Copy data
_, err = io.Copy(w, src)
if err != nil {
return fmt.Errorf("copying encrypted data: %w", err)
}
// Close to flush
err = w.Close()
if err != nil {
return fmt.Errorf("closing encrypted writer: %w", err)
}
return nil
}
// EncryptWriter creates a writer that encrypts data written to it.
// All data written to the returned WriteCloser will be encrypted and written
// to the destination writer. The caller must call Close() on the returned
// writer to ensure all encrypted data is properly flushed and finalized.
// This is useful for integrating encryption into existing writer-based pipelines.
func (e *Encryptor) EncryptWriter(dst io.Writer) (io.WriteCloser, error) {
e.mu.RLock()
recipients := e.recipients
e.mu.RUnlock()
// Create encrypted writer for all recipients
w, err := age.Encrypt(dst, recipients...)
if err != nil {
return nil, fmt.Errorf("creating encrypted writer: %w", err)
}
return w, nil
}
// UpdateRecipients updates the recipients for future encryption operations.
// This method is thread-safe and can be called while other encryption operations
// are in progress. Existing encryption operations will continue with the old
// recipients. At least one recipient must be provided. Returns an error if any
// of the public keys are invalid or if no recipients are specified.
func (e *Encryptor) UpdateRecipients(publicKeys []string) error {
if len(publicKeys) == 0 {
return ErrNoRecipients
}
recipients := make([]age.Recipient, 0, len(publicKeys))
for _, key := range publicKeys {
recipient, err := age.ParseX25519Recipient(key)
if err != nil {
return fmt.Errorf("parsing age recipient %s: %w", key, err)
}
recipients = append(recipients, recipient)
}
e.mu.Lock()
e.recipients = recipients
e.mu.Unlock()
return nil
}
// Decryptor provides thread-safe decryption using the age encryption library.
// It uses a private key to decrypt data that was encrypted for the corresponding
// public key.
type Decryptor struct {
identity age.Identity
mu sync.RWMutex
}
// NewDecryptor creates a new decryptor with the given age private key.
// The private key should be a valid age X25519 identity string.
// Returns an error if the private key is invalid.
func NewDecryptor(privateKey string) (*Decryptor, error) {
identity, err := age.ParseX25519Identity(privateKey)
if err != nil {
return nil, fmt.Errorf("parsing age identity: %w", err)
}
return &Decryptor{
identity: identity,
}, nil
}
// Decrypt decrypts data using age decryption.
// This method is suitable for small to medium amounts of data that fit in memory.
// For large data streams, use DecryptStream instead.
func (d *Decryptor) Decrypt(data []byte) ([]byte, error) {
d.mu.RLock()
identity := d.identity
d.mu.RUnlock()
r, err := age.Decrypt(bytes.NewReader(data), identity)
if err != nil {
return nil, fmt.Errorf("creating decrypted reader: %w", err)
}
decrypted, err := io.ReadAll(r)
if err != nil {
return nil, fmt.Errorf("reading decrypted data: %w", err)
}
return decrypted, nil
}
// DecryptStream returns a reader that decrypts data from the provided reader.
// This method is suitable for decrypting large files or streams as it processes
// data in a streaming fashion without loading everything into memory.
// The caller should close the input reader when done.
func (d *Decryptor) DecryptStream(src io.Reader) (io.Reader, error) {
d.mu.RLock()
identity := d.identity
d.mu.RUnlock()
r, err := age.Decrypt(src, identity)
if err != nil {
return nil, fmt.Errorf("creating decrypted reader: %w", err)
}
return r, nil
}
// Module exports the crypto module for fx dependency injection.
//
//nolint:gochecknoglobals // fx module definitions are package globals
var Module = fx.Module("crypto")
internal/crypto/encryption_test.go
+178
View File
@@ -0,0 +1,178 @@
package crypto_test
import (
"bytes"
"testing"
"filippo.io/age"
"sneak.berlin/go/vaultik/internal/crypto"
)
func TestEncryptor(t *testing.T) {
t.Parallel()
// Generate a test key pair
identity, err := age.GenerateX25519Identity()
if err != nil {
t.Fatalf("failed to generate identity: %v", err)
}
publicKey := identity.Recipient().String()
// Create encryptor
enc, err := crypto.NewEncryptor([]string{publicKey})
if err != nil {
t.Fatalf("failed to create encryptor: %v", err)
}
// Test data
plaintext := []byte("Hello, World! This is a test message.")
// Encrypt
ciphertext, err := enc.Encrypt(plaintext)
if err != nil {
t.Fatalf("failed to encrypt: %v", err)
}
// Verify it's actually encrypted (should be larger and different)
if bytes.Equal(plaintext, ciphertext) {
t.Error("ciphertext equals plaintext")
}
// Decrypt to verify
r, err := age.Decrypt(bytes.NewReader(ciphertext), identity)
if err != nil {
t.Fatalf("failed to decrypt: %v", err)
}
var decrypted bytes.Buffer
_, err = decrypted.ReadFrom(r)
if err != nil {
t.Fatalf("failed to read decrypted data: %v", err)
}
if !bytes.Equal(plaintext, decrypted.Bytes()) {
t.Error("decrypted data doesn't match original")
}
}
func TestEncryptorMultipleRecipients(t *testing.T) {
t.Parallel()
// Generate three test key pairs
identity1, err := age.GenerateX25519Identity()
if err != nil {
t.Fatalf("failed to generate identity1: %v", err)
}
identity2, err := age.GenerateX25519Identity()
if err != nil {
t.Fatalf("failed to generate identity2: %v", err)
}
identity3, err := age.GenerateX25519Identity()
if err != nil {
t.Fatalf("failed to generate identity3: %v", err)
}
publicKeys := []string{
identity1.Recipient().String(),
identity2.Recipient().String(),
identity3.Recipient().String(),
}
// Create encryptor with multiple recipients
enc, err := crypto.NewEncryptor(publicKeys)
if err != nil {
t.Fatalf("failed to create encryptor: %v", err)
}
// Test data
plaintext := []byte("Secret message for multiple recipients")
// Encrypt
ciphertext, err := enc.Encrypt(plaintext)
if err != nil {
t.Fatalf("failed to encrypt: %v", err)
}
// Verify each recipient can decrypt
identities := []age.Identity{identity1, identity2, identity3}
for i, identity := range identities {
r, err := age.Decrypt(bytes.NewReader(ciphertext), identity)
if err != nil {
t.Fatalf("recipient %d failed to decrypt: %v", i+1, err)
}
var decrypted bytes.Buffer
_, err = decrypted.ReadFrom(r)
if err != nil {
t.Fatalf("recipient %d failed to read decrypted data: %v", i+1, err)
}
if !bytes.Equal(plaintext, decrypted.Bytes()) {
t.Errorf("recipient %d: decrypted data doesn't match original", i+1)
}
}
}
func TestEncryptorUpdateRecipients(t *testing.T) {
t.Parallel()
// Generate two identities
identity1, _ := age.GenerateX25519Identity()
identity2, _ := age.GenerateX25519Identity()
publicKey1 := identity1.Recipient().String()
publicKey2 := identity2.Recipient().String()
// Create encryptor with first key
enc, err := crypto.NewEncryptor([]string{publicKey1})
if err != nil {
t.Fatalf("failed to create encryptor: %v", err)
}
// Encrypt with first key
plaintext := []byte("test data")
ciphertext1, err := enc.Encrypt(plaintext)
if err != nil {
t.Fatalf("failed to encrypt: %v", err)
}
// Update to second key
err = enc.UpdateRecipients([]string{publicKey2})
if err != nil {
t.Fatalf("failed to update recipients: %v", err)
}
// Encrypt with second key
ciphertext2, err := enc.Encrypt(plaintext)
if err != nil {
t.Fatalf("failed to encrypt: %v", err)
}
// First ciphertext should only decrypt with first identity
_, err = age.Decrypt(bytes.NewReader(ciphertext1), identity1)
if err != nil {
t.Error("failed to decrypt with identity1")
}
_, err = age.Decrypt(bytes.NewReader(ciphertext1), identity2)
if err == nil {
t.Error("should not decrypt with identity2")
}
// Second ciphertext should only decrypt with second identity
_, err = age.Decrypt(bytes.NewReader(ciphertext2), identity2)
if err != nil {
t.Error("failed to decrypt with identity2")
}
_, err = age.Decrypt(bytes.NewReader(ciphertext2), identity1)
if err == nil {
t.Error("should not decrypt with identity1")
}
}
internal/database/blob_chunks.go
+6 -1
View File
@@ -58,7 +58,12 @@ func (r *BlobChunkRepository) GetByBlobID(
return nil, fmt.Errorf("querying blob chunks: %w", err)
}
defer func() { _ = rows.Close() }()
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
var blobChunks []*BlobChunk
internal/database/blobs.go
+11 -33
View File
@@ -83,7 +83,12 @@ func (r *BlobRepository) GetAll(ctx context.Context) (map[string]*Blob, error) {
return nil, fmt.Errorf("querying blobs: %w", err)
}
defer func() { _ = rows.Close() }()
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
out := make(map[string]*Blob)
@@ -154,16 +159,13 @@ func (r *BlobRepository) UpdateFinished(
return nil
}
// UpdateUploaded marks a blob as uploaded and records the size of the
// stored object as its compressed size. That size can differ from the
// local copy's when an identical blob was already stored with another
// compression level or other recipients.
// UpdateUploaded marks a blob as uploaded
func (r *BlobRepository) UpdateUploaded(
ctx context.Context, tx *sql.Tx, id string, storedSize int64,
ctx context.Context, tx *sql.Tx, id string,
) error {
query := `
UPDATE blobs
SET uploaded_ts = ?, compressed_size = ?
SET uploaded_ts = ?
WHERE id = ?
`
@@ -171,9 +173,9 @@ func (r *BlobRepository) UpdateUploaded(
var err error
if tx != nil {
_, err = tx.ExecContext(ctx, query, now, storedSize, id)
_, err = tx.ExecContext(ctx, query, now, id)
} else {
_, err = r.db.ExecWithLog(ctx, query, now, storedSize, id)
_, err = r.db.ExecWithLog(ctx, query, now, id)
}
if err != nil {
@@ -206,30 +208,6 @@ func (r *BlobRepository) DeleteOrphaned(ctx context.Context) error {
return nil
}
// DeleteUnuploaded deletes blob rows whose upload never completed
// (uploaded_ts IS NULL) and returns how many were removed. Their
// blob_chunks rows are removed by the ON DELETE CASCADE foreign key.
// A blob is only ever attached to a snapshot once its upload has been
// recorded, so an un-uploaded blob is never referenced by a completed
// snapshot: dropping it discards chunk rows that point at data which
// was never stored remotely, so the affected content is re-chunked and
// re-uploaded on the next run.
func (r *BlobRepository) DeleteUnuploaded(ctx context.Context) (int64, error) {
query := `DELETE FROM blobs WHERE uploaded_ts IS NULL`
result, err := r.db.ExecWithLog(ctx, query)
if err != nil {
return 0, fmt.Errorf("deleting un-uploaded blobs: %w", err)
}
rowsAffected, _ := result.RowsAffected()
if rowsAffected > 0 {
log.Debug("Deleted un-uploaded blobs", "count", rowsAffected)
}
return rowsAffected, nil
}
// getOne fetches a single blob row matched on the given column, or
// (nil, nil) when no row matches.
func (r *BlobRepository) getOne(
internal/database/blobs_test.go
+2 -6
View File
@@ -122,8 +122,8 @@ func TestBlobRepositoryUpdates(t *testing.T) {
t.Errorf("expected compressed size 500, got %d", updated.CompressedSize)
}
// Test UpdateUploaded, with a stored object larger than the local copy
err = repo.UpdateUploaded(ctx, nil, blob.ID.String(), 600)
// Test UpdateUploaded
err = repo.UpdateUploaded(ctx, nil, blob.ID.String())
if err != nil {
t.Fatalf("failed to update blob as uploaded: %v", err)
}
@@ -141,10 +141,6 @@ func TestBlobRepositoryUpdates(t *testing.T) {
if uploaded.UploadedTS.Before(now.Add(-1 * time.Second)) {
t.Error("uploaded timestamp should be around test time")
}
if uploaded.CompressedSize != 600 {
t.Errorf("expected the stored size 600, got %d", uploaded.CompressedSize)
}
}
func TestBlobRepositoryDuplicate(t *testing.T) {
internal/database/chunk_files.go
+19 -4
View File
@@ -61,7 +61,12 @@ func (r *ChunkFileRepository) GetByChunkHash(
return nil, fmt.Errorf("querying chunk files: %w", err)
}
defer func() { _ = rows.Close() }()
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
return r.scanChunkFiles(rows)
}
@@ -82,7 +87,12 @@ func (r *ChunkFileRepository) GetByFilePath(
return nil, fmt.Errorf("querying chunk files: %w", err)
}
defer func() { _ = rows.Close() }()
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
return r.scanChunkFiles(rows)
}
@@ -102,7 +112,12 @@ func (r *ChunkFileRepository) GetByFileID(
return nil, fmt.Errorf("querying chunk files: %w", err)
}
defer func() { _ = rows.Close() }()
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
return r.scanChunkFiles(rows)
}
@@ -205,7 +220,7 @@ func (r *ChunkFileRepository) CreateBatch(
cf.ChunkHash.String(), cf.FileID.String(), cf.FileOffset, cf.Length)
}
query += querySb183.String()
query += querySb183.String() //nolint:gosec // G202: appends "?" placeholders only
query += " ON CONFLICT(chunk_hash, file_id) DO NOTHING"
internal/database/chunks.go
+13 -3
View File
@@ -98,7 +98,7 @@ func (r *ChunkRepository) GetByHashes(
args[i] = hash
}
query += querySb75.String()
query += querySb75.String() //nolint:gosec // G202: appends "?" placeholders only
query += ") ORDER BY chunk_hash"
@@ -107,7 +107,12 @@ func (r *ChunkRepository) GetByHashes(
return nil, fmt.Errorf("querying chunks: %w", err)
}
defer func() { _ = rows.Close() }()
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
var chunks []*Chunk
@@ -147,7 +152,12 @@ func (r *ChunkRepository) ListUnpacked(
return nil, fmt.Errorf("querying unpacked chunks: %w", err)
}
defer func() { _ = rows.Close() }()
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
var chunks []*Chunk
internal/database/chunks_ext.go
+8 -23
View File
@@ -7,38 +7,23 @@ import (
// List returns every chunk in the index, ordered by chunk hash.
func (r *ChunkRepository) List(ctx context.Context) ([]*Chunk, error) {
return r.list(ctx, `
query := `
SELECT chunk_hash, size
FROM chunks
ORDER BY chunk_hash
`)
}
`
// ListInUploadedBlobs returns the chunks that are stored in a blob whose
// upload has completed (uploaded_ts set), ordered by chunk hash. These
// are the only chunks a backup may safely deduplicate against: a chunk
// recorded solely in a blob that was never uploaded refers to data that
// is not in remote storage, so trusting it would silently drop that data
// from later snapshots.
func (r *ChunkRepository) ListInUploadedBlobs(ctx context.Context) ([]*Chunk, error) {
return r.list(ctx, `
SELECT DISTINCT c.chunk_hash, c.size
FROM chunks c
JOIN blob_chunks bc ON c.chunk_hash = bc.chunk_hash
JOIN blobs b ON bc.blob_id = b.id
WHERE b.uploaded_ts IS NOT NULL
ORDER BY c.chunk_hash
`)
}
// list runs a chunk-selecting query and scans the (chunk_hash, size) rows.
func (r *ChunkRepository) list(ctx context.Context, query string) ([]*Chunk, error) {
rows, err := r.db.conn.QueryContext(ctx, query)
if err != nil {
return nil, fmt.Errorf("querying chunks: %w", err)
}
defer func() { _ = rows.Close() }()
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
var chunks []*Chunk
internal/database/database.go
+46 -173
View File
@@ -3,10 +3,8 @@
//
// Blobs in Vaultik are the final storage units uploaded to S3. Each blob is a
// large (up to 10GB) file containing many compressed and encrypted chunks from
// multiple source files. Blobs are content-addressed: the filename in S3 is
// hex(SHA256(SHA256(uncompressed blob contents))), computed from the chunk data
// before compression and encryption (not from the stored bytes). See
// blobgen.DoubleSHA256 and docs/REPOSTRUCTURE.md.
// multiple source files. Blobs are content-addressed, meaning their filename
// is derived from their SHA256 hash after compression and encryption.
//
// Schema is managed via numbered SQL migrations embedded in the schema/
// directory. Migration 000.sql bootstraps the schema_migrations tracking
@@ -19,7 +17,6 @@ import (
"embed"
"errors"
"fmt"
"net/url"
"os"
"path/filepath"
"sort"
@@ -42,10 +39,6 @@ var schemaFS embed.FS
// table itself. It is applied before the normal migration loop.
const bootstrapVersion = 0
// busyTimeoutMs is how long a connection to the index waits for another
// connection's lock before failing with "database is locked".
const busyTimeoutMs = 10000
// DB represents the Vaultik local index database connection.
// It uses SQLite to track file metadata, content-defined chunks, and blob associations.
// The database enables incremental backups by detecting changed files and
@@ -98,27 +91,10 @@ func ParseMigrationVersion(filename string) (int, error) {
return version, nil
}
// indexDSN returns the driver DSN that opens the database at path. The
// driver runs each _pragma parameter on every connection it opens and drops
// any parameter it does not know without an error, so a setting written in
// another form silently does nothing. In WAL mode one connection can write
// while others read; the busy timeout makes a connection wait for a lock
// instead of failing at once.
func indexDSN(path string) string {
return fmt.Sprintf(
"%s?_pragma=busy_timeout(%d)&_pragma=journal_mode(WAL)"+
"&_pragma=synchronous(NORMAL)&_pragma=foreign_keys(1)",
path, busyTimeoutMs)
}
// New creates a new database connection at the specified path.
// It creates the schema if needed. Every connection runs in WAL mode with
// a busy timeout and foreign keys on (see indexDSN), so a read-only command
// can read the index while a backup writes to it. Committed rows can sit in
// the -wal file beside the database until a checkpoint, so a copy of the
// database file alone may miss them.
// SQLite handles crash recovery automatically when opening a database with
// journal/WAL files present.
// It creates the schema if needed and configures SQLite with WAL mode for
// better concurrency. SQLite handles crash recovery automatically when
// opening a database with journal/WAL files present.
// The path parameter can be a file path for persistent storage or ":memory:"
// for an in-memory database (useful for testing).
func New(ctx context.Context, path string) (*DB, error) {
@@ -131,7 +107,11 @@ func New(ctx context.Context, path string) (*DB, error) {
// First attempt with standard WAL mode
log.Debug("Attempting to open database with WAL mode", "path", path)
conn, err := sql.Open("sqlite", indexDSN(path))
conn, err := sql.Open(
"sqlite",
path+"?_journal_mode=WAL&_synchronous=NORMAL&_busy_timeout=10000"+
"&_locking_mode=NORMAL&_foreign_keys=ON",
)
if err == nil {
configureConnPool(conn)
@@ -151,8 +131,8 @@ func New(ctx context.Context, path string) (*DB, error) {
_ = conn.Close()
}
// If the first attempt failed, try once more
return retryOpen(ctx, path)
// If first attempt failed, try with TRUNCATE mode to clear any locks
return openWithRecovery(ctx, path)
}
// configureConnPool serializes all database access through one connection.
@@ -164,12 +144,18 @@ func configureConnPool(conn *sql.DB) {
conn.SetMaxIdleConns(1)
}
// finishOpen wraps the connection and applies any pending migrations. On
// migration failure the connection is closed.
// finishOpen enables foreign keys, wraps the connection, and applies any
// pending migrations. On migration failure the connection is closed.
func finishOpen(ctx context.Context, conn *sql.DB, path string) (*DB, error) {
// Enable foreign keys explicitly
_, err := conn.ExecContext(ctx, "PRAGMA foreign_keys = ON")
if err != nil {
log.Warn("Failed to enable foreign keys", "path", path, "error", err)
}
db := &DB{conn: conn, path: path}
err := applyMigrations(ctx, conn)
err = applyMigrations(ctx, conn)
if err != nil {
_ = conn.Close()
@@ -179,15 +165,21 @@ func finishOpen(ctx context.Context, conn *sql.DB, path string) (*DB, error) {
return db, nil
}
// retryOpen makes a second attempt to open the database, with the same
// settings, after the first attempt failed, for example because another
// process held a lock for longer than the busy timeout.
func retryOpen(ctx context.Context, path string) (*DB, error) {
log.Info("Database appears locked, retrying open", "path", path)
// openWithRecovery retries opening the database in TRUNCATE journal mode to
// clear stale locks, then switches back to WAL mode.
func openWithRecovery(ctx context.Context, path string) (*DB, error) {
log.Info(
"Database appears locked, attempting recovery with TRUNCATE mode",
"path", path,
)
conn, err := sql.Open("sqlite", indexDSN(path))
conn, err := sql.Open(
"sqlite",
path+"?_journal_mode=TRUNCATE&_synchronous=NORMAL&_busy_timeout=10000"+
"&_foreign_keys=ON",
)
if err != nil {
return nil, fmt.Errorf("opening database on retry: %w", err)
return nil, fmt.Errorf("opening database in recovery mode: %w", err)
}
configureConnPool(conn)
@@ -195,18 +187,28 @@ func retryOpen(ctx context.Context, path string) (*DB, error) {
err = conn.PingContext(ctx)
if err != nil {
log.Debug(
"Failed to ping database on retry, closing",
"Failed to ping database in recovery mode, closing",
"path", path, "error", err,
)
_ = conn.Close()
return nil, fmt.Errorf(
"database still locked on retry: %w",
"database still locked after recovery attempt: %w",
err,
)
}
log.Debug("Database opened in TRUNCATE mode", "path", path)
// Switch back to WAL mode
log.Debug("Switching database back to WAL mode", "path", path)
_, err = conn.ExecContext(ctx, "PRAGMA journal_mode=WAL")
if err != nil {
log.Warn("Failed to switch back to WAL mode", "path", path, "error", err)
}
db, err := finishOpen(ctx, conn, path)
if err != nil {
return nil, err
@@ -217,135 +219,6 @@ func retryOpen(ctx context.Context, path string) (*DB, error) {
return db, nil
}
// errUntrustedSnapshotSchema is returned when a downloaded snapshot
// database carries schema objects the real schema never defines, or is
// missing a table the restore and deep-verify queries read.
var errUntrustedSnapshotSchema = errors.New(
"downloaded snapshot database has an untrusted schema")
// snapshotReadOnlyDSN builds the driver DSN that opens a materialized
// snapshot database file read-only. mode=ro opens the file read-only at
// the OS level, query_only rejects any write the engine is asked to make,
// and trusted_schema=OFF refuses to run application code named in the
// schema. The file: URI form is required for the driver to honour the
// mode parameter.
func snapshotReadOnlyDSN(path string) string {
u := url.URL{
Scheme: "file",
Path: path,
RawQuery: "mode=ro&_pragma=query_only(true)&_pragma=trusted_schema(false)",
}
return u.String()
}
// OpenReadOnly opens an already-materialized SQLite file for read-only
// querying of a snapshot database downloaded from the store, used by
// restore and deep verify. Unlike New it never applies schema migrations
// and never writes: the connection is opened read-only with query_only
// and trusted_schema=OFF. It refuses any file whose schema carries a
// trigger, view or virtual table, or lacks an expected table, so a forged
// file cannot redefine what the restore queries return. The caller owns
// the file and must remove it.
func OpenReadOnly(ctx context.Context, path string) (*DB, error) {
conn, err := sql.Open("sqlite", snapshotReadOnlyDSN(path))
if err != nil {
return nil, fmt.Errorf("opening read-only database: %w", err)
}
configureConnPool(conn)
err = conn.PingContext(ctx)
if err != nil {
_ = conn.Close()
return nil, fmt.Errorf("opening read-only database: %w", err)
}
err = verifySnapshotSchema(ctx, conn)
if err != nil {
_ = conn.Close()
return nil, err
}
return &DB{conn: conn, path: path}, nil
}
// verifySnapshotSchema rejects a downloaded database whose schema is not
// the plain table set the real schema defines. Any trigger, view or
// virtual table, or a missing expected table, fails the open.
func verifySnapshotSchema(ctx context.Context, conn *sql.DB) error {
// expectedSnapshotTables are the tables the restore and deep-verify
// queries read. A downloaded database missing any of them is not a
// genuine snapshot database and is refused.
expectedSnapshotTables := []string{
"blob_chunks",
"blobs",
"chunks",
"file_chunks",
"files",
}
rows, err := conn.QueryContext(
ctx, "SELECT type, name, sql FROM sqlite_master")
if err != nil {
return fmt.Errorf("reading snapshot schema: %w", err)
}
defer func() { _ = rows.Close() }()
present := make(map[string]struct{})
for rows.Next() {
var objType, name string
var objSQL sql.NullString
err = rows.Scan(&objType, &name, &objSQL)
if err != nil {
return fmt.Errorf("reading snapshot schema: %w", err)
}
switch objType {
case "trigger", "view":
return fmt.Errorf(
"%w: unexpected %s %q", errUntrustedSnapshotSchema, objType, name)
case "table":
if isVirtualTableSQL(objSQL.String) {
return fmt.Errorf(
"%w: unexpected virtual table %q",
errUntrustedSnapshotSchema, name)
}
present[name] = struct{}{}
}
}
err = rows.Err()
if err != nil {
return fmt.Errorf("reading snapshot schema: %w", err)
}
for _, table := range expectedSnapshotTables {
if _, ok := present[table]; !ok {
return fmt.Errorf(
"%w: missing table %q", errUntrustedSnapshotSchema, table)
}
}
return nil
}
// isVirtualTableSQL reports whether a sqlite_master row's SQL defines a
// virtual table. Virtual tables are recorded with type 'table' but a
// "CREATE VIRTUAL TABLE" definition and can run module code, so they are
// refused alongside triggers and views.
func isVirtualTableSQL(createSQL string) bool {
return strings.HasPrefix(
strings.ToUpper(strings.TrimSpace(createSQL)), "CREATE VIRTUAL TABLE")
}
// NewTestDB creates an in-memory SQLite database for testing purposes.
// The database is automatically initialized with the schema and is ready
// for use. Each call creates a new independent database instance.
internal/database/database_test.go
-83
View File
@@ -120,89 +120,6 @@ func TestDatabaseConcurrentAccess(t *testing.T) {
}
}
// TestNewSetsJournalModeAndBusyTimeout checks that the connection settings
// New passes reach SQLite. The driver drops a setting it does not recognise
// without an error, so only reading the value back shows it took effect.
func TestNewSetsJournalModeAndBusyTimeout(t *testing.T) {
t.Parallel()
ctx := context.Background()
db, err := New(ctx, filepath.Join(t.TempDir(), "index.db"))
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
var journalMode string
err = db.conn.QueryRowContext(ctx, "PRAGMA journal_mode").Scan(&journalMode)
if err != nil {
t.Fatalf("reading journal_mode: %v", err)
}
if journalMode != "wal" {
t.Errorf("journal_mode = %q, want %q", journalMode, "wal")
}
var busyTimeout int
err = db.conn.QueryRowContext(ctx, "PRAGMA busy_timeout").Scan(&busyTimeout)
if err != nil {
t.Fatalf("reading busy_timeout: %v", err)
}
if busyTimeout != busyTimeoutMs {
t.Errorf("busy_timeout = %d, want %d", busyTimeout, busyTimeoutMs)
}
}
// TestNewWriteSucceedsWhileAnotherHandleReads opens the same index twice,
// as a read-only command does while a backup runs, and checks that a write
// on one handle commits while the other is in the middle of a read.
func TestNewWriteSucceedsWhileAnotherHandleReads(t *testing.T) {
t.Parallel()
ctx := context.Background()
dbPath := filepath.Join(t.TempDir(), "index.db")
reader, err := New(ctx, dbPath)
if err != nil {
t.Fatalf("failed to open reading handle: %v", err)
}
defer func() { _ = reader.Close() }()
writer, err := New(ctx, dbPath)
if err != nil {
t.Fatalf("failed to open writing handle: %v", err)
}
defer func() { _ = writer.Close() }()
// The read lock taken by the SELECT is held until the transaction ends.
readTx, err := reader.BeginTx(ctx, nil)
if err != nil {
t.Fatalf("beginning read transaction: %v", err)
}
defer func() { _ = readTx.Rollback() }()
var count int
err = readTx.QueryRowContext(ctx, "SELECT COUNT(*) FROM chunks").Scan(&count)
if err != nil {
t.Fatalf("reading chunks: %v", err)
}
_, err = writer.ExecWithLog(ctx,
"INSERT INTO chunks (chunk_hash, size) VALUES (?, ?)", "hash", 1024)
if err != nil {
t.Fatalf("write while another handle reads: %v", err)
}
}
func TestParseMigrationVersion(t *testing.T) {
t.Parallel()
internal/database/errors.go
+12
View File
@@ -0,0 +1,12 @@
package database
import (
"fmt"
"os"
)
// Fatalf prints an error message to stderr and exits with status 1
func Fatalf(format string, args ...any) {
fmt.Fprintf(os.Stderr, "FATAL: "+format+"\n", args...)
os.Exit(1)
}
internal/database/file_chunks.go
+19 -4
View File
@@ -62,7 +62,12 @@ func (r *FileChunkRepository) GetByPath(
return nil, fmt.Errorf("querying file chunks: %w", err)
}
defer func() { _ = rows.Close() }()
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
return r.scanFileChunks(rows)
}
@@ -83,7 +88,12 @@ func (r *FileChunkRepository) GetByFileID(
return nil, fmt.Errorf("querying file chunks: %w", err)
}
defer func() { _ = rows.Close() }()
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
return r.scanFileChunks(rows)
}
@@ -107,7 +117,12 @@ func (r *FileChunkRepository) GetByPathTx(
return nil, fmt.Errorf("querying file chunks: %w", err)
}
defer func() { _ = rows.Close() }()
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
fileChunks, err := r.scanFileChunks(rows)
LogSQL("GetByPathTx", "Complete", path, "count", len(fileChunks))
@@ -238,7 +253,7 @@ func (r *FileChunkRepository) CreateBatch(
args = append(args, fc.FileID.String(), fc.Idx, fc.ChunkHash.String())
}
query += querySb211.String()
query += querySb211.String() //nolint:gosec // G202: appends "?" placeholders only
query += " ON CONFLICT(file_id, idx) DO NOTHING"
internal/database/files.go
+50 -133
View File
@@ -33,13 +33,11 @@ func (r *FileRepository) Create(ctx context.Context, tx *sql.Tx, file *File) err
}
query := `
INSERT INTO files
(id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
INSERT INTO files (id, path, source_path, mtime, size, mode, uid, gid, link_target)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(path) DO UPDATE SET
source_path = excluded.source_path,
mtime = excluded.mtime,
mtime_nsec = excluded.mtime_nsec,
size = excluded.size,
mode = excluded.mode,
uid = excluded.uid,
@@ -56,19 +54,16 @@ func (r *FileRepository) Create(ctx context.Context, tx *sql.Tx, file *File) err
if tx != nil {
LogSQL("Execute", query,
file.ID.String(), file.Path.String(), file.SourcePath.String(),
file.MTime.Unix(), file.MTime.Nanosecond(),
file.Size, file.Mode, file.UID, file.GID,
file.MTime.Unix(), file.Size, file.Mode, file.UID, file.GID,
file.LinkTarget.String())
err = tx.QueryRowContext(ctx, query,
file.ID.String(), file.Path.String(), file.SourcePath.String(),
file.MTime.Unix(), file.MTime.Nanosecond(),
file.Size, file.Mode, file.UID, file.GID,
file.MTime.Unix(), file.Size, file.Mode, file.UID, file.GID,
file.LinkTarget.String()).Scan(&idStr)
} else {
err = r.db.QueryRowWithLog(ctx, query,
file.ID.String(), file.Path.String(), file.SourcePath.String(),
file.MTime.Unix(), file.MTime.Nanosecond(),
file.Size, file.Mode, file.UID, file.GID,
file.MTime.Unix(), file.Size, file.Mode, file.UID, file.GID,
file.LinkTarget.String()).Scan(&idStr)
}
@@ -89,7 +84,7 @@ func (r *FileRepository) Create(ctx context.Context, tx *sql.Tx, file *File) err
// in the index.
func (r *FileRepository) GetByPath(ctx context.Context, path string) (*File, error) {
query := `
SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
FROM files
WHERE path = ?
`
@@ -109,7 +104,7 @@ func (r *FileRepository) GetByPath(ctx context.Context, path string) (*File, err
// GetByID retrieves a file by its UUID
func (r *FileRepository) GetByID(ctx context.Context, id types.FileID) (*File, error) {
query := `
SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
FROM files
WHERE id = ?
`
@@ -132,7 +127,7 @@ func (r *FileRepository) GetByPathTx(
ctx context.Context, tx *sql.Tx, path string,
) (*File, error) {
query := `
SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
FROM files
WHERE path = ?
`
@@ -163,19 +158,23 @@ func (r *FileRepository) ListModifiedSince(
ctx context.Context, since time.Time,
) ([]*File, error) {
query := `
SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
FROM files
WHERE (mtime, mtime_nsec) >= (?, ?)
WHERE mtime >= ?
ORDER BY path
`
rows, err := r.db.conn.QueryContext(ctx, query,
since.Unix(), since.Nanosecond())
rows, err := r.db.conn.QueryContext(ctx, query, since.Unix())
if err != nil {
return nil, fmt.Errorf("querying files: %w", err)
}
defer func() { _ = rows.Close() }()
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
var files []*File
@@ -229,103 +228,16 @@ func (r *FileRepository) DeleteByID(
return nil
}
// ListUnderPath returns the file at path and every file beneath it,
// ordered by path. Paths are compared case-sensitively, and a trailing
// slash on path is ignored, so "/" lists every file.
func (r *FileRepository) ListUnderPath(
ctx context.Context, path string,
) ([]*File, error) {
path = strings.TrimRight(path, "/")
dirPrefix := path + "/"
// LIKE would ignore ASCII case and treat _ and % in path as wildcards.
query := `
SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
FROM files
WHERE path = ? OR substr(path, 1, length(?)) = ?
ORDER BY path
`
rows, err := r.db.conn.QueryContext(ctx, query, path, dirPrefix, dirPrefix)
if err != nil {
return nil, fmt.Errorf("querying files: %w", err)
}
defer func() { _ = rows.Close() }()
var files []*File
for rows.Next() {
file, err := r.scanFileRows(rows)
if err != nil {
return nil, fmt.Errorf("scanning file: %w", err)
}
files = append(files, file)
}
return files, rows.Err()
}
// ListUnderPathInSnapshot returns the files that ListUnderPath returns
// for path and that the snapshot lists, in no particular order.
func (r *FileRepository) ListUnderPathInSnapshot(
ctx context.Context, path, snapshotID string,
) ([]*File, error) {
path = strings.TrimRight(path, "/")
dirPrefix := path + "/"
query := `
SELECT f.id, f.path, f.source_path, f.mtime, f.mtime_nsec, f.size,
f.mode, f.uid, f.gid, f.link_target
FROM files f
JOIN snapshot_files sf ON sf.file_id = f.id
WHERE sf.snapshot_id = ?
AND (f.path = ? OR substr(f.path, 1, length(?)) = ?)
`
rows, err := r.db.conn.QueryContext(ctx, query,
snapshotID, path, dirPrefix, dirPrefix)
if err != nil {
return nil, fmt.Errorf("querying files: %w", err)
}
defer func() { _ = rows.Close() }()
var files []*File
for rows.Next() {
file, err := r.scanFileRows(rows)
if err != nil {
return nil, fmt.Errorf("scanning file: %w", err)
}
files = append(files, file)
}
return files, rows.Err()
}
// ListIDsWithChunksNotInUploadedBlobs returns the IDs of the files whose
// path starts with prefix and that list at least one chunk held by no
// blob whose upload has completed (uploaded_ts set). A new snapshot
// cannot reference such a chunk, so a backup must not treat the file as
// unchanged even when its metadata matches the file on disk.
func (r *FileRepository) ListIDsWithChunksNotInUploadedBlobs(
// ListByPrefix returns all files whose path starts with prefix, ordered by
// path.
func (r *FileRepository) ListByPrefix(
ctx context.Context, prefix string,
) ([]types.FileID, error) {
) ([]*File, error) {
query := `
SELECT DISTINCT f.id
FROM files f
JOIN file_chunks fc ON fc.file_id = f.id
WHERE f.path LIKE ? || '%'
AND NOT EXISTS (
SELECT 1
FROM blob_chunks bc
JOIN blobs b ON bc.blob_id = b.id
WHERE bc.chunk_hash = fc.chunk_hash
AND b.uploaded_ts IS NOT NULL
)
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
FROM files
WHERE path LIKE ? || '%'
ORDER BY path
`
rows, err := r.db.conn.QueryContext(ctx, query, prefix)
@@ -333,28 +245,31 @@ func (r *FileRepository) ListIDsWithChunksNotInUploadedBlobs(
return nil, fmt.Errorf("querying files: %w", err)
}
defer func() { _ = rows.Close() }()
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
var ids []types.FileID
var files []*File
for rows.Next() {
var id types.FileID
err := rows.Scan(&id)
file, err := r.scanFileRows(rows)
if err != nil {
return nil, fmt.Errorf("scanning file ID: %w", err)
return nil, fmt.Errorf("scanning file: %w", err)
}
ids = append(ids, id)
files = append(files, file)
}
return ids, rows.Err()
return files, rows.Err()
}
// ListAll returns all files in the database
func (r *FileRepository) ListAll(ctx context.Context) ([]*File, error) {
query := `
SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
FROM files
ORDER BY path
`
@@ -364,7 +279,12 @@ func (r *FileRepository) ListAll(ctx context.Context) ([]*File, error) {
return nil, fmt.Errorf("querying files: %w", err)
}
defer func() { _ = rows.Close() }()
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
var files []*File
@@ -390,7 +310,7 @@ func (r *FileRepository) CreateBatch(
}
// Each files row binds this many SQL variables.
const fileCols = 10
const fileCols = 9
// Batch at 100 rows to be safe with SQLite's variable limit.
const batchSize = 100
@@ -401,7 +321,7 @@ func (r *FileRepository) CreateBatch(
batch := files[i:end]
query := `INSERT INTO files
(id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target)
(id, path, source_path, mtime, size, mode, uid, gid, link_target)
VALUES `
args := make([]any, 0, len(batch)*fileCols)
@@ -413,21 +333,19 @@ func (r *FileRepository) CreateBatch(
querySb325.WriteString(", ")
}
querySb325.WriteString("(?, ?, ?, ?, ?, ?, ?, ?, ?, ?)")
querySb325.WriteString("(?, ?, ?, ?, ?, ?, ?, ?, ?)")
args = append(args,
f.ID.String(), f.Path.String(), f.SourcePath.String(),
f.MTime.Unix(), f.MTime.Nanosecond(),
f.Size, f.Mode, f.UID, f.GID,
f.MTime.Unix(), f.Size, f.Mode, f.UID, f.GID,
f.LinkTarget.String())
}
query += querySb325.String()
query += querySb325.String() //nolint:gosec // G202: appends "?" placeholders only
query += ` ON CONFLICT(path) DO UPDATE SET
source_path = excluded.source_path,
mtime = excluded.mtime,
mtime_nsec = excluded.mtime_nsec,
size = excluded.size,
mode = excluded.mode,
uid = excluded.uid,
@@ -487,7 +405,7 @@ func (r *FileRepository) scanFileFrom(row fileRowScanner) (*File, error) {
var (
file File
idStr, pathStr, sourcePathStr string
mtimeUnix, mtimeNsec int64
mtimeUnix int64
linkTarget sql.NullString
)
@@ -496,7 +414,6 @@ func (r *FileRepository) scanFileFrom(row fileRowScanner) (*File, error) {
&pathStr,
&sourcePathStr,
&mtimeUnix,
&mtimeNsec,
&file.Size,
&file.Mode,
&file.UID,
@@ -515,7 +432,7 @@ func (r *FileRepository) scanFileFrom(row fileRowScanner) (*File, error) {
file.Path = types.FilePath(pathStr)
file.SourcePath = types.SourcePath(sourcePathStr)
file.MTime = time.Unix(mtimeUnix, mtimeNsec).UTC()
file.MTime = time.Unix(mtimeUnix, 0).UTC()
if linkTarget.Valid {
file.LinkTarget = types.FilePath(linkTarget.String)
}
internal/database/files_test.go
-246
View File
@@ -5,12 +5,10 @@ import (
"database/sql"
"errors"
"os"
"slices"
"testing"
"time"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/types"
)
// errTestRollback is the sentinel returned from transaction bodies to
@@ -136,141 +134,6 @@ func TestFileRepositoryListDelete(t *testing.T) {
}
}
func TestFileRepositoryListUnderPath(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repo := database.NewFileRepository(db)
const (
docDir = "/home/u/doc"
docFile = "/home/u/doc/a.txt"
)
// In path order, so the root case can expect all of them as listed.
paths := []string{
"/home/u/50%/x.txt",
"/home/u/50percent/y.txt",
"/home/u/DOC/c.txt",
"/home/u/a_b/x.txt",
"/home/u/axb/y.txt",
docDir,
"/home/u/doc.txt.bak",
docFile,
"/home/u/doc/sub/b.txt",
"/home/u/doc2/b.txt",
}
for _, path := range paths {
err := repo.Create(ctx, nil, &database.File{
Path: types.FilePath(path),
MTime: time.Now().Truncate(time.Second),
Mode: 0644,
})
if err != nil {
t.Fatalf("failed to create %s: %v", path, err)
}
}
docTree := []string{docDir, docFile, "/home/u/doc/sub/b.txt"}
tests := []struct {
name string
path string
want []string
}{
{"directory", docDir, docTree},
{"directory with trailing slash", docDir + "/", docTree},
{"directory differing only in case", "/home/u/DOC",
[]string{"/home/u/DOC/c.txt"}},
{"file", docFile, []string{docFile}},
{"underscore is literal", "/home/u/a_b",
[]string{"/home/u/a_b/x.txt"}},
{"percent is literal", "/home/u/50%",
[]string{"/home/u/50%/x.txt"}},
{"root", "/", paths},
}
for _, tt := range tests {
files, err := repo.ListUnderPath(ctx, tt.path)
if err != nil {
t.Fatalf("%s: failed to list files: %v", tt.name, err)
}
got := make([]string, 0, len(files))
for _, f := range files {
got = append(got, f.Path.String())
}
if !slices.Equal(got, tt.want) {
t.Errorf("%s: listing %q got %q, want %q",
tt.name, tt.path, got, tt.want)
}
}
}
// ListUnderPathInSnapshot leaves out files outside path and files the
// snapshot does not list, though another snapshot lists them.
func TestFileRepositoryListUnderPathInSnapshot(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
files := database.NewFileRepository(db)
snapshots := database.NewSnapshotRepository(db)
listed := map[string][]string{
"one": {"/src/a.txt", "/src2/c.txt"},
"two": {"/src/b.txt"},
}
for snapshotID, paths := range listed {
err := snapshots.Create(ctx, nil, &database.Snapshot{
ID: types.SnapshotID(snapshotID),
Hostname: "host",
StartedAt: time.Now(),
})
if err != nil {
t.Fatalf("failed to create snapshot %s: %v", snapshotID, err)
}
for _, path := range paths {
err := files.Create(ctx, nil, &database.File{
Path: types.FilePath(path),
MTime: time.Now().Truncate(time.Second),
Mode: 0644,
})
if err != nil {
t.Fatalf("failed to create %s: %v", path, err)
}
err = snapshots.AddFile(ctx, nil, snapshotID, path)
if err != nil {
t.Fatalf("failed to add %s to %s: %v", path, snapshotID, err)
}
}
}
found, err := files.ListUnderPathInSnapshot(ctx, "/src", "one")
if err != nil {
t.Fatalf("failed to list files: %v", err)
}
got := make([]string, 0, len(found))
for _, f := range found {
got = append(got, f.Path.String())
}
if !slices.Equal(got, []string{"/src/a.txt"}) {
t.Errorf("got %q, want only /src/a.txt", got)
}
}
func TestFileRepositorySymlink(t *testing.T) {
t.Parallel()
@@ -311,115 +174,6 @@ func TestFileRepositorySymlink(t *testing.T) {
}
}
// An mtime after 2262 or before 1678 does not fit in int64 nanoseconds
// since the epoch, and must still come back from the database unchanged.
func TestFileRepositoryMTimeOutsideInt64NanosecondRange(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repo := database.NewFileRepository(db)
mtimes := []time.Time{
time.Date(2300, time.January, 1, 0, 0, 0, 123456789, time.UTC),
time.Date(1601, time.January, 1, 0, 0, 0, 987654321, time.UTC),
}
for _, mtime := range mtimes {
created := &database.File{
Path: types.FilePath("/created-" + mtime.Format(time.RFC3339Nano)),
MTime: mtime,
}
err := repo.Create(ctx, nil, created)
if err != nil {
t.Fatalf("failed to create file: %v", err)
}
batched := &database.File{
ID: types.NewFileID(),
Path: types.FilePath("/batched-" + mtime.Format(time.RFC3339Nano)),
MTime: mtime,
}
err = repo.CreateBatch(ctx, nil, []*database.File{batched})
if err != nil {
t.Fatalf("failed to batch create file: %v", err)
}
for _, path := range []types.FilePath{created.Path, batched.Path} {
retrieved, err := repo.GetByPath(ctx, path.String())
if err != nil {
t.Fatalf("failed to get file: %v", err)
}
if !retrieved.MTime.Equal(mtime) {
t.Errorf("%s: mtime got %v, want %v",
path, retrieved.MTime, mtime)
}
}
}
}
// A file already in the index and rewritten within the same second must get
// its new nanoseconds stored, through both Create and CreateBatch.
func TestFileRepositoryUpsertMTimeInSameSecond(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repo := database.NewFileRepository(db)
indexed := time.Date(2026, time.October, 7, 12, 0, 0, 100000000, time.UTC)
rewritten := indexed.Add(800 * time.Millisecond)
tests := []struct {
name string
upsert func(file *database.File) error
}{
{"Create", func(file *database.File) error {
return repo.Create(ctx, nil, file)
}},
{"CreateBatch", func(file *database.File) error {
return repo.CreateBatch(ctx, nil, []*database.File{file})
}},
}
for _, tt := range tests {
file := &database.File{
ID: types.NewFileID(),
Path: types.FilePath("/" + tt.name),
MTime: indexed,
}
err := tt.upsert(file)
if err != nil {
t.Fatalf("%s: failed to create file: %v", tt.name, err)
}
file.MTime = rewritten
err = tt.upsert(file)
if err != nil {
t.Fatalf("%s: failed to update file: %v", tt.name, err)
}
retrieved, err := repo.GetByPath(ctx, file.Path.String())
if err != nil {
t.Fatalf("%s: failed to get file: %v", tt.name, err)
}
if !retrieved.MTime.Equal(rewritten) {
t.Errorf("%s: mtime got %v, want %v",
tt.name, retrieved.MTime, rewritten)
}
}
}
func TestFileRepositoryTransaction(t *testing.T) {
t.Parallel()
internal/database/models.go
+10 -9
View File
@@ -14,7 +14,8 @@ type File struct {
ID types.FileID // UUID primary key
Path types.FilePath // Absolute path of the file
// SourcePath is the source directory this file came from.
// SourcePath is the source directory this file came from (used for
// restore path stripping).
SourcePath types.SourcePath
MTime time.Time
Size int64
@@ -50,15 +51,15 @@ type Chunk struct {
// Blob represents a blob record in the database.
// A blob is Vaultik's final storage unit - a large file (up to 10GB) containing
// many compressed and encrypted chunks from multiple source files.
// Blobs are content-addressed: the filename in S3 is
// hex(SHA256(SHA256(uncompressed blob contents))), computed from the chunk data
// before compression and encryption (not from the stored bytes). See
// blobgen.DoubleSHA256 and docs/REPOSTRUCTURE.md.
// Blobs are content-addressed, meaning their filename in S3 is derived from
// the SHA256 hash of their compressed and encrypted content.
// The blob creation process is: chunks are accumulated -> compressed with zstd
// -> encrypted with age -> hashed -> uploaded to S3 with the hash as filename.
type Blob struct {
ID types.BlobID // UUID assigned when blob creation starts
// Hash is hex(SHA256(SHA256(uncompressed blob contents)))
// (empty until finalized); see the type comment above.
// Hash is the SHA256 of the final compressed+encrypted content
// (empty until finalized).
Hash types.BlobHash
CreatedTS time.Time // When blob creation started
FinishedTS *time.Time // When blob was finalized (nil if still packing)
@@ -98,8 +99,8 @@ type Snapshot struct {
StartedAt time.Time
CompletedAt *time.Time // nil if still in progress
FileCount int64
ChunkCount int64 // Chunks this snapshot stored that were not stored before
BlobCount int64 // Blobs this snapshot created
ChunkCount int64
BlobCount int64
TotalSize int64 // Total size of all referenced files
// BlobSize is the total size of all referenced blobs (compressed and
internal/database/module.go
-44
View File
@@ -2,7 +2,6 @@ package database
import (
"context"
"errors"
"fmt"
"os"
"path/filepath"
@@ -16,44 +15,6 @@ import (
// the index describes the backed-up file tree and must stay private.
const indexDirPerm = 0o700
// indexFilePerm restricts the index file to the owning user; it lists every
// backed-up path and chunk hash and must stay private.
const indexFilePerm = 0o600
// ensureIndexFileMode makes the index file owner-only before the SQLite
// driver opens it: it creates the file 0600 if absent, or chmods an existing
// one to 0600. Doing this first matters because SQLite creates its -wal and
// -shm side files with the mode of the main database file, so a private main
// file yields private side files. The driver treats a zero-byte file as an
// empty database, so pre-creating it here is safe.
func ensureIndexFileMode(path string) error {
info, err := os.Stat(path)
switch {
case err == nil:
if info.Mode().Perm() == indexFilePerm {
return nil
}
err = os.Chmod(path, indexFilePerm)
if err != nil {
return fmt.Errorf("restricting index file permissions: %w", err)
}
return nil
case errors.Is(err, os.ErrNotExist):
//nolint:gosec // G304: the index path is operator-configured by design
f, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY, indexFilePerm)
if err != nil {
return fmt.Errorf("creating index file: %w", err)
}
return f.Close()
default:
return fmt.Errorf("checking index file: %w", err)
}
}
// Module provides database dependencies
//
//nolint:gochecknoglobals // fx module definitions are package globals by convention
@@ -73,11 +34,6 @@ func provideDatabase(lc fx.Lifecycle, cfg *config.Config) (*DB, error) {
return nil, fmt.Errorf("creating index directory: %w", err)
}
err = ensureIndexFileMode(cfg.IndexPath)
if err != nil {
return nil, err
}
db, err := New(context.Background(), cfg.IndexPath)
if err != nil {
return nil, fmt.Errorf("opening database: %w", err)
internal/database/module_internal_test.go
-100
View File
@@ -1,100 +0,0 @@
package database
import (
"os"
"path/filepath"
"syscall"
"testing"
"go.uber.org/fx/fxtest"
"sneak.berlin/go/vaultik/internal/config"
)
// TestProvideDatabaseFreshIndexMode verifies that provideDatabase creates a
// missing index file owner-only (0600), even under a lenient 022 umask that
// would otherwise leave a freshly created file world-readable.
//
//nolint:paralleltest // syscall.Umask is process-global; parallel tests would clash
func TestProvideDatabaseFreshIndexMode(t *testing.T) {
restore := syscall.Umask(0o022)
defer syscall.Umask(restore)
indexPath := filepath.Join(t.TempDir(), "index.sqlite")
openIndex(t, indexPath)
assertPerm(t, indexPath, 0o600)
}
// TestProvideDatabaseExistingIndexMode verifies that provideDatabase tightens
// an existing world-readable index (0644) in a group/other-readable directory
// down to owner-only (0600).
//
//nolint:paralleltest // syscall.Umask is process-global; parallel tests would clash
func TestProvideDatabaseExistingIndexMode(t *testing.T) {
restore := syscall.Umask(0o022)
defer syscall.Umask(restore)
dir := filepath.Join(t.TempDir(), "data")
//nolint:gosec // G301: the test intentionally uses a 0755 directory
err := os.MkdirAll(dir, 0o755)
if err != nil {
t.Fatalf("creating index directory: %v", err)
}
//nolint:gosec // G302: the test intentionally uses a 0755 directory
err = os.Chmod(dir, 0o755)
if err != nil {
t.Fatalf("relaxing index directory permissions: %v", err)
}
indexPath := filepath.Join(dir, "index.sqlite")
//nolint:gosec // G306: the test intentionally starts from a 0644 index
err = os.WriteFile(indexPath, nil, 0o644)
if err != nil {
t.Fatalf("creating pre-existing index: %v", err)
}
//nolint:gosec // G302: the test intentionally starts from a 0644 index
err = os.Chmod(indexPath, 0o644)
if err != nil {
t.Fatalf("relaxing pre-existing index permissions: %v", err)
}
openIndex(t, indexPath)
assertPerm(t, indexPath, 0o600)
}
// openIndex runs provideDatabase against indexPath and closes the resulting
// database before returning.
func openIndex(t *testing.T, indexPath string) {
t.Helper()
cfg := &config.Config{IndexPath: indexPath}
db, err := provideDatabase(fxtest.NewLifecycle(t), cfg)
if err != nil {
t.Fatalf("provideDatabase: %v", err)
}
err = db.Close()
if err != nil {
t.Fatalf("closing database: %v", err)
}
}
// assertPerm fails the test unless path has exactly the given permission bits.
func assertPerm(t *testing.T, path string, want os.FileMode) {
t.Helper()
info, err := os.Stat(path)
if err != nil {
t.Fatalf("stat %s: %v", path, err)
}
got := info.Mode().Perm()
if got != want {
t.Fatalf("permissions of %s = %#o, want %#o", path, got, want)
}
}
internal/database/read_only_test.go
-145
View File
@@ -1,145 +0,0 @@
//nolint:testpackage // exercises unexported read-only open internals
package database
import (
"context"
"database/sql"
"errors"
"path/filepath"
"testing"
)
// genuineSnapshotDB writes a real snapshot database (the full schema
// applied) to a fresh file and returns its path.
func genuineSnapshotDB(t *testing.T) string {
t.Helper()
path := filepath.Join(t.TempDir(), "snapshot.db")
db, err := New(context.Background(), path)
if err != nil {
t.Fatalf("creating snapshot database: %v", err)
}
err = db.Close()
if err != nil {
t.Fatalf("closing snapshot database: %v", err)
}
return path
}
// forgedDB creates an empty database file and runs the given statements
// against it read-write, so a test can plant schema objects the real
// schema never defines.
func forgedDB(t *testing.T, stmts ...string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "forged.db")
db, err := sql.Open("sqlite", path)
if err != nil {
t.Fatalf("opening forged database: %v", err)
}
for _, stmt := range stmts {
_, err = db.ExecContext(context.Background(), stmt)
if err != nil {
t.Fatalf("executing %q: %v", stmt, err)
}
}
err = db.Close()
if err != nil {
t.Fatalf("closing forged database: %v", err)
}
return path
}
func TestOpenReadOnlyAcceptsGenuineSnapshot(t *testing.T) {
t.Parallel()
db, err := OpenReadOnly(context.Background(), genuineSnapshotDB(t))
if err != nil {
t.Fatalf("OpenReadOnly refused a genuine snapshot database: %v", err)
}
t.Cleanup(func() { _ = db.Close() })
}
func TestOpenReadOnlyRefusesWrites(t *testing.T) {
t.Parallel()
db, err := OpenReadOnly(context.Background(), genuineSnapshotDB(t))
if err != nil {
t.Fatalf("OpenReadOnly: %v", err)
}
t.Cleanup(func() { _ = db.Close() })
// A schema write depends on no table columns, so the only reason it
// can fail is that the database is open read-only.
_, err = db.Conn().ExecContext(context.Background(),
"CREATE TABLE probe_readonly (x)")
if err == nil {
t.Fatal("expected a write to a read-only snapshot database to fail")
}
}
func TestOpenReadOnlyRejectsView(t *testing.T) {
t.Parallel()
path := forgedDB(t, "CREATE VIEW files AS SELECT 1 AS path")
_, err := OpenReadOnly(context.Background(), path)
if !errors.Is(err, errUntrustedSnapshotSchema) {
t.Fatalf("expected a view named files to be refused, got %v", err)
}
}
func TestOpenReadOnlyRejectsTrigger(t *testing.T) {
t.Parallel()
path := forgedDB(t,
"CREATE TABLE files (path TEXT)",
"CREATE TRIGGER t AFTER INSERT ON files BEGIN SELECT 1; END")
_, err := OpenReadOnly(context.Background(), path)
if !errors.Is(err, errUntrustedSnapshotSchema) {
t.Fatalf("expected a trigger to be refused, got %v", err)
}
}
func TestOpenReadOnlyRejectsMissingTable(t *testing.T) {
t.Parallel()
// Only one of the expected tables is present.
path := forgedDB(t, "CREATE TABLE files (path TEXT)")
_, err := OpenReadOnly(context.Background(), path)
if !errors.Is(err, errUntrustedSnapshotSchema) {
t.Fatalf("expected a missing expected table to be refused, got %v", err)
}
}
func TestIsVirtualTableSQL(t *testing.T) {
t.Parallel()
cases := []struct {
sql string
want bool
}{
{"CREATE VIRTUAL TABLE t USING fts5(x)", true},
{" create virtual table t using fts5(x)", true},
{"CREATE TABLE t (x)", false},
{"CREATE VIEW t AS SELECT 1", false},
{"", false},
}
for _, c := range cases {
if got := isVirtualTableSQL(c.sql); got != c.want {
t.Errorf("isVirtualTableSQL(%q) = %v, want %v", c.sql, got, c.want)
}
}
}
internal/database/repositories.go
+18 -23
View File
@@ -3,7 +3,6 @@ package database
import (
"context"
"database/sql"
"errors"
"fmt"
)
@@ -61,24 +60,22 @@ func (r *Repositories) WithTx(ctx context.Context, fn TxFunc) error {
defer func() {
if p := recover(); p != nil {
// The panic is the failure to report, so a rollback error
// is dropped.
_ = tx.Rollback()
rollbackErr := tx.Rollback()
if rollbackErr != nil {
Fatalf("failed to rollback transaction: %v", rollbackErr)
}
panic(p)
} else if err != nil {
rollbackErr := tx.Rollback()
if rollbackErr != nil {
Fatalf("failed to rollback transaction: %v", rollbackErr)
}
}
}()
err = fn(ctx, tx)
if err != nil {
// Once ctx is cancelled, database/sql rolls the transaction
// back itself and Rollback returns sql.ErrTxDone.
rollbackErr := tx.Rollback()
if rollbackErr != nil && !errors.Is(rollbackErr, sql.ErrTxDone) {
return fmt.Errorf("%w; rolling back transaction: %w",
err, rollbackErr)
}
return err
}
@@ -106,24 +103,22 @@ func (r *Repositories) WithReadTx(ctx context.Context, fn TxFunc) error {
defer func() {
if p := recover(); p != nil {
// The panic is the failure to report, so a rollback error
// is dropped.
_ = tx.Rollback()
rollbackErr := tx.Rollback()
if rollbackErr != nil {
Fatalf("failed to rollback transaction: %v", rollbackErr)
}
panic(p)
} else if err != nil {
rollbackErr := tx.Rollback()
if rollbackErr != nil {
Fatalf("failed to rollback transaction: %v", rollbackErr)
}
}
}()
err = fn(ctx, tx)
if err != nil {
// Once ctx is cancelled, database/sql rolls the transaction
// back itself and Rollback returns sql.ErrTxDone.
rollbackErr := tx.Rollback()
if rollbackErr != nil && !errors.Is(rollbackErr, sql.ErrTxDone) {
return fmt.Errorf("%w; rolling back transaction: %w",
err, rollbackErr)
}
return err
}
internal/database/repositories_test.go
-158
View File
@@ -4,7 +4,6 @@ import (
"context"
"database/sql"
"errors"
"strings"
"testing"
"time"
@@ -236,125 +235,6 @@ func TestRepositoriesTransactionRollback(t *testing.T) {
}
}
// When the context is cancelled during the body, database/sql rolls the
// transaction back itself, so WithTx's own rollback finds it already
// done. WithTx must return the body's error alone, without that rollback.
func TestRepositoriesTransactionContextCancelled(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
repos := database.NewRepositories(db)
err := repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
file := &database.File{
Path: "/test/cancelled_file.txt",
MTime: time.Now().Truncate(time.Second),
Size: 1024,
Mode: 0644,
UID: 1000,
GID: 1000,
}
err := repos.Files.Create(ctx, tx, file)
if err != nil {
return err
}
cancel()
// database/sql rolls back from a goroutine of its own; wait
// until it has, so the test does not depend on that race.
for {
_, err = tx.ExecContext(context.WithoutCancel(ctx), "SELECT 1")
if errors.Is(err, sql.ErrTxDone) {
return ctx.Err()
}
time.Sleep(time.Millisecond)
}
})
if !errors.Is(err, context.Canceled) {
t.Fatalf("expected context.Canceled, got: %v", err)
}
if errors.Is(err, sql.ErrTxDone) {
t.Fatalf("the already-done rollback was reported: %v", err)
}
file, err := repos.Files.GetByPath(
context.Background(), "/test/cancelled_file.txt",
)
if err != nil {
t.Fatalf("error checking for file: %v", err)
}
if file != nil {
t.Error("file should not exist after the cancelled transaction")
}
}
// A body that ends the transaction itself with ROLLBACK makes WithTx's
// own rollback fail. WithTx must return that failure together with the
// body's error.
func TestRepositoriesTransactionRollbackFails(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repos := database.NewRepositories(db)
err := repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
_, err := tx.ExecContext(ctx, "ROLLBACK")
if err != nil {
return err
}
return errIntentionalRollback
})
if !errors.Is(err, errIntentionalRollback) {
t.Fatalf("expected the body's error, got: %v", err)
}
if !strings.Contains(err.Error(), "rolling back transaction") {
t.Errorf("expected the failed rollback in the error, got: %v", err)
}
}
// When the body ends the transaction itself with ROLLBACK and then
// panics, WithTx's own rollback fails; the panic must still reach the
// caller.
func TestRepositoriesTransactionPanicRollbackFails(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repos := database.NewRepositories(db)
defer func() {
if recover() == nil {
t.Error("expected the body's panic to reach the caller")
}
}()
_ = repos.WithTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
_, err := tx.ExecContext(ctx, "ROLLBACK")
if err != nil {
return err
}
panic(errIntentionalRollback)
})
}
func TestRepositoriesReadTransaction(t *testing.T) {
t.Parallel()
@@ -411,41 +291,3 @@ func TestRepositoriesReadTransaction(t *testing.T) {
t.Error("expected to retrieve file in read transaction")
}
}
// When the context is cancelled during a read transaction's body,
// database/sql rolls the transaction back itself, so WithReadTx's own
// rollback finds it already done. WithReadTx must return the body's error
// alone, without that rollback.
func TestRepositoriesReadTransactionContextCancelled(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
repos := database.NewRepositories(db)
err := repos.WithReadTx(ctx, func(ctx context.Context, tx *sql.Tx) error {
cancel()
// database/sql rolls back from a goroutine of its own; wait
// until it has, so the test does not depend on that race.
for {
_, err := tx.ExecContext(context.WithoutCancel(ctx), "SELECT 1")
if errors.Is(err, sql.ErrTxDone) {
return ctx.Err()
}
time.Sleep(time.Millisecond)
}
})
if !errors.Is(err, context.Canceled) {
t.Fatalf("expected context.Canceled, got: %v", err)
}
if errors.Is(err, sql.ErrTxDone) {
t.Fatalf("the already-done rollback was reported: %v", err)
}
}
internal/database/repository_comprehensive_test.go
+2 -2
View File
@@ -824,7 +824,7 @@ func TestTransactionIsolation(t *testing.T) {
}
// Verify the file was not created (transaction rolled back)
files, err := repos.Files.ListUnderPath(ctx, "/tx-test.txt")
files, err := repos.Files.ListByPrefix(ctx, "/tx-test")
if err != nil {
t.Fatal(err)
}
@@ -916,7 +916,7 @@ func TestConcurrentOrphanedCleanup(t *testing.T) {
}
// Verify correct files were deleted
files, err := repos.Files.ListAll(ctx)
files, err := repos.Files.ListByPrefix(ctx, "/concurrent-")
if err != nil {
t.Fatal(err)
}
internal/database/repository_debug_test.go
+1 -1
View File
@@ -147,7 +147,7 @@ func TestOrphanedFileCleanupDebug(t *testing.T) {
t.Logf("Files count after cleanup: %d", count)
// List remaining files
files, err := repos.Files.ListUnderPath(ctx, "/")
files, err := repos.Files.ListByPrefix(ctx, "/")
if err != nil {
t.Fatal(err)
}
internal/database/repository_edge_cases_test.go
+13 -20
View File
@@ -3,7 +3,6 @@ package database
import (
"context"
"database/sql"
"fmt"
"strings"
"testing"
@@ -368,7 +367,7 @@ func verifyBlobNullUploadTS(
}
// createLargeDatasetFiles creates fileCount files and adds every other
// one to the snapshot, in one transaction as a backup writes them.
// one to the snapshot.
func createLargeDatasetFiles(
t *testing.T,
repos *Repositories,
@@ -377,10 +376,9 @@ func createLargeDatasetFiles(
) {
t.Helper()
ctx := context.Background()
start := time.Now()
err := repos.WithTx(context.Background(),
func(ctx context.Context, tx *sql.Tx) error {
for i := range fileCount {
file := &File{
Path: types.FilePath(fmt.Sprintf("/large/file%05d.txt", i)),
@@ -391,25 +389,19 @@ func createLargeDatasetFiles(
GID: uint32(1000 + (i % 10)),
}
err := repos.Files.Create(ctx, tx, file)
err := repos.Files.Create(ctx, nil, file)
if err != nil {
return fmt.Errorf("creating file %d: %w", i, err)
t.Fatalf("failed to create file %d: %v", i, err)
}
// Add half to snapshot
if i%2 == 0 {
err = repos.Snapshots.AddFileByID(ctx, tx, snapshotID, file.ID)
if err != nil {
return err
}
}
}
return nil
})
err = repos.Snapshots.AddFileByID(ctx, nil, snapshotID, file.ID)
if err != nil {
t.Fatal(err)
}
}
}
t.Logf("Created %d files in %v", fileCount, time.Since(start))
}
@@ -450,12 +442,12 @@ func TestLargeDatasets(t *testing.T) {
createLargeDatasetFiles(t, repos, snapshot.ID.String(), fileCount)
})
// Test ListUnderPath performance
// Test ListByPrefix performance
//nolint:paralleltest // phases share one database and are order-dependent
t.Run("list under path performance", func(t *testing.T) {
t.Run("list by prefix performance", func(t *testing.T) {
start := time.Now()
files, err := repos.Files.ListUnderPath(ctx, "/large/")
files, err := repos.Files.ListByPrefix(ctx, "/large/")
if err != nil {
t.Fatal(err)
}
@@ -480,7 +472,7 @@ func TestLargeDatasets(t *testing.T) {
t.Logf("Cleaned up orphaned files in %v", time.Since(start))
// Verify correct number remain
files, err := repos.Files.ListUnderPath(ctx, "/large/")
files, err := repos.Files.ListByPrefix(ctx, "/large/")
if err != nil {
t.Fatal(err)
}
@@ -614,7 +606,8 @@ func TestTimezoneHandling(t *testing.T) {
t.Skip("timezone not available")
}
nyTime := time.Now().In(loc)
// Use Truncate to remove sub-second precision since we store as Unix timestamps
nyTime := time.Now().In(loc).Truncate(time.Second)
file := &File{
Path: "/timezone-test.txt",
MTime: nyTime,
internal/database/schema/001.sql
+2 -15
View File
@@ -5,9 +5,8 @@
CREATE TABLE IF NOT EXISTS files (
id TEXT PRIMARY KEY, -- UUID
path TEXT NOT NULL UNIQUE,
source_path TEXT NOT NULL DEFAULT '', -- The source directory this file came from
mtime INTEGER NOT NULL, -- whole seconds since the Unix epoch
mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second, 0 to 999999999
source_path TEXT NOT NULL DEFAULT '', -- The source directory this file came from (for restore path stripping)
mtime INTEGER NOT NULL,
size INTEGER NOT NULL,
mode INTEGER NOT NULL,
uid INTEGER NOT NULL,
@@ -86,10 +85,8 @@ CREATE TABLE IF NOT EXISTS snapshots (
completed_at INTEGER,
file_count INTEGER NOT NULL DEFAULT 0,
chunk_count INTEGER NOT NULL DEFAULT 0,
new_chunk_size INTEGER NOT NULL DEFAULT 0,
blob_count INTEGER NOT NULL DEFAULT 0,
total_size INTEGER NOT NULL DEFAULT 0,
uncompressed_chunk_size INTEGER NOT NULL DEFAULT 0,
blob_size INTEGER NOT NULL DEFAULT 0,
blob_uncompressed_size INTEGER NOT NULL DEFAULT 0,
compression_ratio REAL NOT NULL DEFAULT 1.0,
@@ -123,16 +120,6 @@ CREATE TABLE IF NOT EXISTS snapshot_blobs (
-- Index for efficient blob lookups (used in orphan detection)
CREATE INDEX IF NOT EXISTS idx_snapshot_blobs_blob_id ON snapshot_blobs(blob_id);
-- Snapshot mount points table: the directories at or below a configured
-- path that the snapshot's backup found on a different filesystem from
-- their parent directory, as a mount point is, and found an entry in
CREATE TABLE IF NOT EXISTS snapshot_mount_points (
snapshot_id TEXT NOT NULL,
path TEXT NOT NULL,
PRIMARY KEY (snapshot_id, path),
FOREIGN KEY (snapshot_id) REFERENCES snapshots(id) ON DELETE CASCADE
);
-- Uploads table: tracks blob upload metrics
CREATE TABLE IF NOT EXISTS uploads (
blob_hash TEXT PRIMARY KEY,
internal/database/snapshots.go
+84 -184
View File
@@ -11,21 +11,8 @@ import (
"sneak.berlin/go/vaultik/internal/types"
)
// Sentinel errors for the single-snapshot invariant that an exported
// per-snapshot metadata database must satisfy.
var (
// ErrNoSnapshotInDatabase means the metadata database has no snapshot
// row at all.
ErrNoSnapshotInDatabase = errors.New("database contains no snapshot")
// ErrMultipleSnapshotsInDatabase means the metadata database holds
// more than the single snapshot an export is supposed to contain.
ErrMultipleSnapshotsInDatabase = errors.New(
"database contains more than one snapshot")
)
// SnapshotRepository provides access to the snapshots table and its
// snapshot_files / snapshot_blobs association tables, and to the mount
// points each snapshot's backup found (snapshot_mount_points).
// snapshot_files / snapshot_blobs association tables.
type SnapshotRepository struct {
db *DB
}
@@ -121,20 +108,14 @@ func (r *SnapshotRepository) UpdateCounts(
return nil
}
// UpdateExtendedStats updates extended statistics for a snapshot. It also
// records the total plaintext size of the distinct chunks the snapshot's
// files reference. Only the snapshot's own backup can take that total: a
// file's row is shared by every snapshot that lists it, and a later
// backup of a changed file rewrites the row with the new chunks.
// UpdateExtendedStats updates extended statistics for a snapshot
func (r *SnapshotRepository) UpdateExtendedStats(
ctx context.Context,
tx *sql.Tx,
snapshotID string,
blobUncompressedSize int64,
compressionLevel int,
uploadBytes int64,
uploadDurationMs int64,
newChunkSize int64,
) error {
compressionRatio, err := r.extendedCompressionRatio(
ctx, tx, snapshotID, blobUncompressedSize,
@@ -148,30 +129,19 @@ func (r *SnapshotRepository) UpdateExtendedStats(
SET blob_uncompressed_size = ?,
compression_ratio = ?,
compression_level = ?,
upload_bytes = ?,
upload_duration_ms = ?,
new_chunk_size = ?,
uncompressed_chunk_size = (
SELECT COALESCE(SUM(c.size), 0)
FROM (
SELECT DISTINCT fc.chunk_hash
FROM snapshot_files sf
JOIN file_chunks fc ON sf.file_id = fc.file_id
WHERE sf.snapshot_id = snapshots.id
) sc
JOIN chunks c ON sc.chunk_hash = c.chunk_hash
)
upload_bytes = blob_size,
upload_duration_ms = ?
WHERE id = ?
`
if tx != nil {
_, err = tx.ExecContext(ctx, query,
blobUncompressedSize, compressionRatio, compressionLevel,
uploadBytes, uploadDurationMs, newChunkSize, snapshotID)
uploadDurationMs, snapshotID)
} else {
_, err = r.db.ExecWithLog(ctx, query,
blobUncompressedSize, compressionRatio, compressionLevel,
uploadBytes, uploadDurationMs, newChunkSize, snapshotID)
uploadDurationMs, snapshotID)
}
if err != nil {
@@ -236,43 +206,6 @@ func (r *SnapshotRepository) GetByID(
return &snapshot, nil
}
// GetOnlySnapshot returns the sole snapshot in an exported per-snapshot
// metadata database. The backup path writes each snapshot's database with
// exactly one snapshot row (see cleanSnapshotDB), so restore and deep
// verify expect exactly one. Zero rows return ErrNoSnapshotInDatabase and
// more than one returns ErrMultipleSnapshotsInDatabase; callers treat
// either as a failed identity check on the downloaded database.
func (r *SnapshotRepository) GetOnlySnapshot(ctx context.Context) (*Snapshot, error) {
query := `
SELECT id, hostname, vaultik_version, vaultik_git_revision,
started_at, completed_at, file_count, chunk_count, blob_count,
total_size, blob_size, compression_ratio
FROM snapshots
LIMIT 2
`
rows, err := r.db.conn.QueryContext(ctx, query)
if err != nil {
return nil, fmt.Errorf("querying snapshots: %w", err)
}
defer func() { _ = rows.Close() }()
snapshots, err := r.scanSnapshotRows(rows)
if err != nil {
return nil, err
}
switch len(snapshots) {
case 1:
return snapshots[0], nil
case 0:
return nil, ErrNoSnapshotInDatabase
default:
return nil, ErrMultipleSnapshotsInDatabase
}
}
// ListRecent returns up to limit snapshots, most recently started first.
func (r *SnapshotRepository) ListRecent(
ctx context.Context, limit int,
@@ -291,7 +224,12 @@ func (r *SnapshotRepository) ListRecent(
return nil, fmt.Errorf("querying snapshots: %w", err)
}
defer func() { _ = rows.Close() }()
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
return r.scanSnapshotRows(rows)
}
@@ -403,7 +341,7 @@ func (r *SnapshotRepository) AddFilesByIDBatch(
args = append(args, snapshotID, fileID.String())
}
query += querySb312.String()
query += querySb312.String() //nolint:gosec // G202: appends "?" placeholders only
var err error
if tx != nil {
@@ -501,57 +439,32 @@ func (r *SnapshotRepository) GetBlobHashes(
ORDER BY sb.blob_hash
`
blobs, err := r.queryStrings(ctx, query, snapshotID)
rows, err := r.db.conn.QueryContext(ctx, query, snapshotID)
if err != nil {
return nil, fmt.Errorf("querying blob hashes: %w", err)
}
return blobs, nil
}
// AddMountPoint records dir as a directory that the backup of a snapshot
// found on a different filesystem from its parent directory, as a mount
// point is, using tx when non-nil.
func (r *SnapshotRepository) AddMountPoint(
ctx context.Context, tx *sql.Tx, snapshotID string, dir string,
) error {
query := `
INSERT OR IGNORE INTO snapshot_mount_points (snapshot_id, path)
VALUES (?, ?)
`
var err error
if tx != nil {
_, err = tx.ExecContext(ctx, query, snapshotID, dir)
} else {
_, err = r.db.ExecWithLog(ctx, query, snapshotID, dir)
}
defer func() {
err := rows.Close()
if err != nil {
return fmt.Errorf("adding mount point to snapshot: %w", err)
Fatalf("failed to close rows: %v", err)
}
}()
return nil
}
var blobs []string
// GetMountPoints returns the directories AddMountPoint recorded for a
// snapshot, sorted.
func (r *SnapshotRepository) GetMountPoints(
ctx context.Context, snapshotID string,
) ([]string, error) {
query := `
SELECT path
FROM snapshot_mount_points
WHERE snapshot_id = ?
ORDER BY path
`
for rows.Next() {
var blobHash string
dirs, err := r.queryStrings(ctx, query, snapshotID)
err := rows.Scan(&blobHash)
if err != nil {
return nil, fmt.Errorf("querying mount points: %w", err)
return nil, fmt.Errorf("scanning blob hash: %w", err)
}
return dirs, nil
blobs = append(blobs, blobHash)
}
return blobs, rows.Err()
}
// GetSnapshotTotalCompressedSize returns the total compressed size of all
@@ -576,66 +489,71 @@ func (r *SnapshotRepository) GetSnapshotTotalCompressedSize(
return totalSize, nil
}
// GetSnapshotBlobSizes returns the total compressed and uncompressed sizes
// of all blobs referenced by a snapshot.
func (r *SnapshotRepository) GetSnapshotBlobSizes(
ctx context.Context, snapshotID string,
) (int64, int64, error) {
query := `
SELECT COALESCE(SUM(b.compressed_size), 0),
COALESCE(SUM(b.uncompressed_size), 0)
FROM snapshot_blobs sb
JOIN blobs b ON sb.blob_hash = b.blob_hash
WHERE sb.snapshot_id = ?
`
var compressed, uncompressed int64
err := r.db.conn.QueryRowContext(ctx, query, snapshotID).Scan(
&compressed, &uncompressed)
if err != nil {
return 0, 0, fmt.Errorf("querying snapshot blob sizes: %w", err)
}
return compressed, uncompressed, nil
}
// GetSnapshotUncompressedChunkSize returns the total plaintext size of the
// distinct chunks the snapshot's files reference, as the snapshot's backup
// recorded it (see UpdateExtendedStats).
// GetSnapshotUncompressedChunkSize returns the sum of plaintext sizes of all unique
// chunks referenced by a snapshot (via snapshot_files → file_chunks → chunks).
func (r *SnapshotRepository) GetSnapshotUncompressedChunkSize(
ctx context.Context, snapshotID string,
) (int64, error) {
query := `SELECT uncompressed_chunk_size FROM snapshots WHERE id = ?`
query := `
SELECT COALESCE(SUM(c.size), 0)
FROM (
SELECT DISTINCT fc.chunk_hash
FROM snapshot_files sf
JOIN file_chunks fc ON sf.file_id = fc.file_id
WHERE sf.snapshot_id = ?
) sc
JOIN chunks c ON sc.chunk_hash = c.chunk_hash
`
var uncompressedChunkSize int64
var totalSize int64
err := r.db.conn.QueryRowContext(ctx, query, snapshotID).Scan(&uncompressedChunkSize)
err := r.db.conn.QueryRowContext(ctx, query, snapshotID).Scan(&totalSize)
if err != nil {
return 0, fmt.Errorf("querying uncompressed chunk size: %w", err)
}
return uncompressedChunkSize, nil
return totalSize, nil
}
// GetSnapshotNewChunkSize returns the total plaintext size of the chunks the
// snapshot's backup stored that were not stored before, as the backup
// recorded it. It is not derived from the snapshot's files: a file's row is
// shared by every snapshot that lists it and is rewritten when the file
// changes, so it shows only the file's latest chunks.
// GetSnapshotNewChunkSize returns the sum of plaintext sizes of chunks that are
// referenced by this snapshot but not by any earlier completed snapshot known to
// the local database. The result is the marginal uncompressed data this snapshot
// added to the dedup pool — i.e., the delta from prior snapshots.
func (r *SnapshotRepository) GetSnapshotNewChunkSize(
ctx context.Context, snapshotID string,
) (int64, error) {
query := `SELECT new_chunk_size FROM snapshots WHERE id = ?`
query := `
WITH this_snap_chunks AS (
SELECT DISTINCT fc.chunk_hash
FROM snapshot_files sf
JOIN file_chunks fc ON sf.file_id = fc.file_id
WHERE sf.snapshot_id = ?
),
prior_chunks AS (
SELECT DISTINCT fc.chunk_hash
FROM snapshots s
JOIN snapshot_files sf ON sf.snapshot_id = s.id
JOIN file_chunks fc ON fc.file_id = sf.file_id
WHERE s.completed_at IS NOT NULL
AND s.id != ?
AND s.started_at < (SELECT started_at FROM snapshots WHERE id = ?)
)
SELECT COALESCE(SUM(c.size), 0)
FROM chunks c
JOIN this_snap_chunks t ON c.chunk_hash = t.chunk_hash
WHERE c.chunk_hash NOT IN (SELECT chunk_hash FROM prior_chunks)
`
var newChunkSize int64
var totalSize int64
err := r.db.conn.QueryRowContext(ctx, query, snapshotID).Scan(&newChunkSize)
err := r.db.conn.QueryRowContext(
ctx, query, snapshotID, snapshotID, snapshotID,
).Scan(&totalSize)
if err != nil {
return 0, fmt.Errorf("querying new chunk size: %w", err)
}
return newChunkSize, nil
return totalSize, nil
}
// GetIncompleteSnapshots returns all snapshots that haven't been completed
@@ -656,7 +574,12 @@ func (r *SnapshotRepository) GetIncompleteSnapshots(
return nil, fmt.Errorf("querying incomplete snapshots: %w", err)
}
defer func() { _ = rows.Close() }()
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
return r.scanSnapshotRows(rows)
}
@@ -679,7 +602,12 @@ func (r *SnapshotRepository) GetIncompleteByHostname(
return nil, fmt.Errorf("querying incomplete snapshots: %w", err)
}
defer func() { _ = rows.Close() }()
defer func() {
err := rows.Close()
if err != nil {
Fatalf("failed to close rows: %v", err)
}
}()
// Same column set as every other multi-row snapshot query, so the
// shared scanner applies — including its timestamp normalization.
@@ -740,34 +668,6 @@ func (r *SnapshotRepository) DeleteSnapshotUploads(
return nil
}
// queryStrings runs query, which selects one text column, and returns
// its values in the order of the rows.
func (r *SnapshotRepository) queryStrings(
ctx context.Context, query string, args ...any,
) ([]string, error) {
rows, err := r.db.conn.QueryContext(ctx, query, args...)
if err != nil {
return nil, err
}
defer func() { _ = rows.Close() }()
var values []string
for rows.Next() {
var value string
err := rows.Scan(&value)
if err != nil {
return nil, err
}
values = append(values, value)
}
return values, rows.Err()
}
// extendedCompressionRatio computes the compression ratio for a snapshot
// from its stored blob_size and the given uncompressed size. Returns 1.0
// when the uncompressed size is zero.
internal/database/snapshots_test.go
-109
View File
@@ -4,7 +4,6 @@ import (
"context"
"fmt"
"math"
"slices"
"testing"
"time"
@@ -146,65 +145,6 @@ func TestSnapshotRepositoryUpdateCounts(t *testing.T) {
}
}
// GetSnapshotBlobSizes totals the blobs the snapshot references, and only
// those.
func TestSnapshotRepositoryGetSnapshotBlobSizes(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repos := database.NewRepositories(db)
snapshot := &database.Snapshot{
ID: "2024-01-03T12:00:00Z",
Hostname: testHostname,
VaultikVersion: testVersion,
StartedAt: time.Now().Truncate(time.Second),
}
err := repos.Snapshots.Create(ctx, nil, snapshot)
if err != nil {
t.Fatalf("failed to create snapshot: %v", err)
}
blobs := []*database.Blob{
{Hash: "referenced-1", CompressedSize: 10, UncompressedSize: 100},
{Hash: "referenced-2", CompressedSize: 20, UncompressedSize: 200},
{Hash: "unreferenced", CompressedSize: 40, UncompressedSize: 400},
}
for _, blob := range blobs {
blob.ID = types.NewBlobID()
blob.CreatedTS = time.Now().Truncate(time.Second)
err = repos.Blobs.Create(ctx, nil, blob)
if err != nil {
t.Fatalf("failed to create blob %s: %v", blob.Hash, err)
}
}
for _, blob := range blobs[:2] {
err = repos.Snapshots.AddBlob(ctx, nil, snapshot.ID.String(),
blob.ID, blob.Hash)
if err != nil {
t.Fatalf("failed to add blob %s to snapshot: %v", blob.Hash, err)
}
}
compressed, uncompressed, err := repos.Snapshots.GetSnapshotBlobSizes(
ctx, snapshot.ID.String())
if err != nil {
t.Fatalf("failed to get snapshot blob sizes: %v", err)
}
if compressed != 30 || uncompressed != 300 {
t.Errorf("blob sizes: got %d and %d, want 30 and 300",
compressed, uncompressed)
}
}
func TestSnapshotRepositoryListRecent(t *testing.T) {
t.Parallel()
@@ -423,52 +363,3 @@ func TestSnapshotRepositoryDuplicate(t *testing.T) {
t.Error("expected error for duplicate snapshot")
}
}
// Deleting a snapshot from the local index deletes the mount points its
// backup recorded, and no other snapshot's. See
// https://git.eeqj.de/sneak/vaultik/issues/395.
func TestSnapshotRepositoryDeleteRemovesMountPoints(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repo := database.NewSnapshotRepository(db)
for _, id := range []types.SnapshotID{"kept", "deleted"} {
err := repo.Create(ctx, nil, &database.Snapshot{
ID: id,
Hostname: testHostname,
VaultikVersion: testVersion,
StartedAt: time.Now().Truncate(time.Second),
})
if err != nil {
t.Fatalf("failed to create snapshot: %v", err)
}
err = repo.AddMountPoint(ctx, nil, id.String(), "/data/nas")
if err != nil {
t.Fatalf("failed to add mount point: %v", err)
}
}
err := repo.Delete(ctx, "deleted")
if err != nil {
t.Fatalf("failed to delete snapshot: %v", err)
}
for id, want := range map[string][]string{
"kept": {"/data/nas"},
"deleted": nil,
} {
got, err := repo.GetMountPoints(ctx, id)
if err != nil {
t.Fatalf("failed to read mount points: %v", err)
}
if !slices.Equal(got, want) {
t.Errorf("mount points of %s: got %v, want %v", id, got, want)
}
}
}
internal/database/uploads.go
+16
View File
@@ -158,3 +158,19 @@ type UploadStats struct {
MinDurationMs int64
MaxDurationMs int64
}
// GetCountBySnapshot returns the count of uploads for a specific snapshot
func (r *UploadRepository) GetCountBySnapshot(
ctx context.Context, snapshotID string,
) (int64, error) {
query := `SELECT COUNT(*) FROM uploads WHERE snapshot_id = ?`
var count int64
err := r.conn.QueryRowContext(ctx, query, snapshotID).Scan(&count)
if err != nil {
return 0, err
}
return count, nil
}
internal/globals/globals.go
+3 -45
View File
@@ -3,34 +3,20 @@
package globals
import (
"regexp"
"strings"
"time"
)
// Appname is the application name, populated from main().
var Appname = "vaultik" //nolint:gochecknoglobals // set via -ldflags at build time
// DevVersion is the version a binary reports when it was built without
// git metadata: script/version emits it outside a git checkout, and an
// unstamped `go build` keeps it. goreleaser's snapshot template stamps
// it followed by "-" and the commit it was built from. It is
// deliberately not a number.
const DevVersion = "dev"
// Unknown is what Commit and CommitDate hold when the build did not
// stamp them, and the version script/docker and script/cibuild stamp
// when the host has no git checkout.
const Unknown = "unknown"
// Version is the application version, populated from main().
var Version = DevVersion //nolint:gochecknoglobals // set via -ldflags at build time
var Version = "dev" //nolint:gochecknoglobals // set via -ldflags at build time
// Commit is the git commit hash, populated from main().
var Commit = Unknown //nolint:gochecknoglobals // set via -ldflags at build time
var Commit = "unknown" //nolint:gochecknoglobals // set via -ldflags at build time
// CommitDate is the ISO-8601 date of the commit, populated from main().
var CommitDate = Unknown //nolint:gochecknoglobals // set via -ldflags at build time
var CommitDate = "unknown" //nolint:gochecknoglobals // set via -ldflags at build time
// Author identifies the upstream author of vaultik.
const Author = "Jeffrey Paul <sneak@sneak.berlin>"
@@ -38,9 +24,6 @@ const Author = "Jeffrey Paul <sneak@sneak.berlin>"
// Homepage is the canonical URL for vaultik.
const Homepage = "https://sneak.berlin/go/vaultik"
// ReleasesURL is where tagged release artifacts are published.
const ReleasesURL = "https://git.eeqj.de/sneak/vaultik/releases"
// License is the SPDX identifier for the project license.
const License = "MIT"
@@ -64,31 +47,6 @@ func New() (*Globals, error) {
}, nil
}
// IsDevVersion reports whether v names a development build rather than
// a release. "dev" and goreleaser's snapshot "dev-<sha>" count, and so
// does what `git describe --tags --always --dirty` gives a make or
// docker build of an untagged commit: the bare short commit, or
// tag-N-gHASH on a commit after a tag. Any version ending in "-dirty"
// counts, a modified checkout of a tag ("v1.0.0-dirty") included.
// A plain tag such as "v1.0.0" or "1.0.0" is a release.
//
// The empty string counts too. Nothing that knows its version reports
// no version, so an empty Version means the stamping failed, and the
// safe reading of "we could not establish that this is a release" is
// that it is not one. The Makefile refuses to build at all in that
// case; this is the second line of defence, for a binary linked by
// something other than the Makefile. Unknown counts for the same
// reason.
func IsDevVersion(v string) bool {
if v == "" || v == Unknown || v == DevVersion ||
strings.HasPrefix(v, DevVersion+"-") ||
strings.HasSuffix(v, "-dirty") {
return true
}
return regexp.MustCompile(`^(.+-[0-9]+-g)?[0-9a-f]+$`).MatchString(v)
}
// shortCommitLen is the number of commit-hash characters ShortCommit keeps.
const shortCommitLen = 12
Loaded 100 of 246 files, more files were not shown because too many files have changed in this diff. Show more