Author SHA1 Message Date
sneak 68f4ccf5b9 Accept a remote key for restore and verify, and document it (closes #124)
check / check (pull_request) Failing after 1s
A host restoring after the original is gone has no local index and cannot
know a snapshot's human ID; `snapshot list` shows such snapshots only by
their remote key. Restore and verify now resolve an identifier to that
remote key: a human ID is hashed as before, and a remote key (or an
unambiguous leading part of it, as the table prints) is used directly,
resolved against the store's metadata listing. Deep verify reads the one
snapshot in the exported database rather than filtering by the human ID.

Adds an integration test that backs up with one index and hostname, then
lists, restores, and deep-verifies from the store with a fresh empty
index, a different hostname, and no age_recipients — comparing restored
bytes to the source. The empty index is what makes it fail if restore
ever needed the original one.

Adds a "Restoring on another machine" README section walking the flow end
to end, and drops the now-done roadmap item.

Model: claude-opus-4-8
2026-09-21 20:00:22 +00:00
clawbot aab6a87f8c Reconcile the schema/migration docs with the code (closes #68)
check / check (pull_request) Failing after 1s
check / check (push) Successful in 2m46s
Four documents told different stories about the database schema. docs/DATAMODEL.md now owns the explanation and separates two things: the policy, which is unchanged (no supported upgrade path between versions; delete the local index with vaultik database delete and back up again), and the schema bootstrap that does exist (numbered files in internal/database/schema applied to a fresh database and recorded in schema_migrations).

README.md and AGENTS.md are reworded to match and link there. AGENTS.md names the real file to edit, internal/database/schema/001.sql, and notes that the pre-1.0 disposability clause expires on tagging. No code changed.

Judgement call: REPO_POLICIES.md still names a different schema file; it is cross-project policy and was left alone.

model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
2026-09-21 21:58:37 +02:00
clawbot c355ef4d25 Report a prune count that could not be read as unknown, not 0 (closes #96)
check / check (pull_request) Failing after 1s
check / check (push) Successful in 2m46s
Prune read table row counts before and after to report how many orphaned files, chunks and blobs it removed, and discarded the error from every read. A failed query therefore reported as a count of 0, and the summary showed plausible wrong numbers.

A count that cannot be read is now logged as a warning (on stderr, also under --json) and shown as "unknown"; a difference computed from an unknown count is itself unknown. 0 still means the table was empty. No --json document carries these counts, so none can show a false 0.

model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
2026-09-21 21:41:57 +02:00
clawbot 5927e1aa3d Write file:// blobs atomically via temp file and rename (closes #130)
check / check (push) Failing after 0s
check / check (pull_request) Failing after 0s
The file:// backend streamed each object straight to its final key, so an upload cut off mid-stream left a truncated object there. The next backup saw that Stat succeeded, recorded the blob as complete, and produced a snapshot that reported success but could not be restored.

Writes now go to a temporary file with a .partial suffix in the destination directory, are synced, then renamed onto the key. List and ListStream skip .partial files, so a leftover is never trusted as a blob and is overwritten when the key is written again. S3 PutObject is already atomic.

Disclosure: the containing directory is not synced after the rename, so a host crash right after it could still lose the object on some filesystems.

model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
2026-09-21 21:24:42 +02:00
clawbot 9ca962969a Map s3 not-found to storage.ErrNotFound in Get and Stat (closes #129)
check / check (push) Failing after 1s
check / check (pull_request) Successful in 3m50s
The Storer interface documents that Get and Stat return storage.ErrNotFound for a missing object. The file and rclone backends did; the s3 backend returned the raw SDK error, so callers testing for ErrNotFound behaved differently on s3.

S3Storer.Get and Stat now wrap ErrNotFound when the SDK reports a missing object and leave every other error untouched. The SDK reports a missing key two ways (NoSuchKey from Get, NotFound from Head); both are recognised in one helper, s3.IsNotFound, which HeadObject now also uses. The mapping lives in the storage package because internal/s3 cannot import it.

model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
2026-09-21 21:07:35 +02:00
clawbot 89ebfc78e2 Use one duration parser and fix the --older-than months example (closes #123)
check / check (push) Failing after 1s
check / check (pull_request) Failing after 1s
Two parseDuration functions existed with different grammars; only the one in internal/vaultik/helpers.go was reachable from a flag. The unused copy in internal/cli/duration.go is deleted, so no flag accepts anything it did not before.

The README gave 6m as the six-months example for snapshot purge --older-than, but m is minutes: that command removed every snapshot older than six minutes. The example is now 6mo, and the help for --older-than and --keep-newer-than states that m is minutes and mo is months.

The parser now rejects negative durations, which it used to accept or silently make positive.

model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)

Co-authored-by: clawbot <clawbot@noreply.example.org>
2026-09-21 20:58:30 +02:00
clawbot 07ef3a1c78 Drop the lint-guard shell scanner, keep the Dockerfile.lint checks (closes #121)
check / check (push) Failing after 0s
check / check (pull_request) Successful in 2m37s
The guard test in cmd/vaultik/lintdocker_test.go tried to prove that no script runs the linter outside the container by parsing shell scripts with a hand-written scanner. Four reviews each found another spelling it missed; such a parser cannot be complete, and nobody could follow it in one reading.

The scanner, its helpers and their tests are deleted. The plain Dockerfile.lint assertions stay: the linter image is pinned by digest, config verify runs before run, and the per-run value reaches both steps. TODO.md no longer claims a test proves the property; script/lint is the only lint entry point, and keeping it so is a review matter.

Judgement call: this drops a guard two reviewers asked to harden.

model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (decision, merge)
2026-09-21 20:41:33 +02:00
clawbot c423d13191 Hash the plaintext, not the encrypted bytes, in verify --deep (closes #131)
check / check (push) Failing after 0s
check / check (pull_request) Failing after 0s
The last step of verify --deep hashed the encrypted bytes it downloaded once with SHA256 and compared the result to the blob name. The name is the double SHA256 of the blob plaintext, so the two could never match and deep verification failed on every healthy blob with "blob hash mismatch".

It now hashes the decompressed plaintext as chunk verification streams it and compares the double SHA256 of that to the blob name, the same derivation the writer uses. A new test backs up a real snapshot, runs deep verify on it, then flips one byte in a stored blob and expects failure.

model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
2026-09-21 20:24:37 +02:00
clawbot 75a10d3a22 Hash-verify the Go toolchain in the release workflow (closes #105)
check / check (push) Failing after 0s
check / check (pull_request) Successful in 4m13s
The release workflow installed Go with actions/setup-go, which pins the action but not the Go archive it downloads, so the compiler that builds the published binaries was verified against nothing in this repo.

New script/install-go, modelled on script/install-goreleaser, downloads the go.dev archive for the version in go.mod and refuses it unless its sha256 matches the value committed in the script. It fails if its version disagrees with go.mod, and on any OS or architecture other than the Linux release runners. GOTOOLCHAIN=local on the release step keeps the verified toolchain from switching itself.

Judgement call: release path only; script/bootstrap still uses the host Go.

model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
2026-09-21 19:48:52 +02:00
clawbot 3d56dd7eb0 VACUUM snapshot metadata through the sqlite driver, not a CLI (closes #120)
check / check (push) Failing after 1s
check / check (pull_request) Successful in 2m57s
snapshot create compacted the metadata database by running a sqlite3 command-line binary, after every blob had already been uploaded. On a host without that binary, which includes anyone who installed with go install, the backup failed at the last step, and two tests failed the same way.

VACUUM now runs through the Go sqlite driver the program already uses, and its error is returned to the caller. The runtime Docker image no longer installs the sqlite package, since nothing in the binary calls it.

model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
2026-09-21 19:41:28 +02:00
clawbot bdce350041 Delete dead code and stale fixtures, fix config set reindent (closes #70)
check / check (push) Failing after 1s
check / check (pull_request) Failing after 1s
Removes code and fixtures nothing uses: the internal/models package and its test, a second SnapshotInfo type in package cli that had no references (the live one is in internal/vaultik), and two config fixtures, test-config.yml and test/integration-config.yml, whose keys the config loader no longer accepts. test/config.yaml stays; a test uses it.

Also fixes config set, which rewrote the whole file with 4-space indentation on the first set despite the documented promise to preserve formatting. It now encodes with 2-space indent like the default template, and a test asserts comments and indentation survive a set.

Deviation: one commit, not one per deletion as the issue asked.

model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
2026-09-21 19:25:01 +02:00
clawbot 753bc3ef60 Correct remote layout and privacy docs for hashed snapshot keys (closes #67)
check / check (pull_request) Failing after 1s
check / check (push) Successful in 3m11s
Three documents showed the remote layout with a plaintext snapshot ID as the metadata directory name, and docs/REPOSTRUCTURE.md blamed those IDs for the observable backup time. The store actually names each metadata directory with a one-way hash of the ID, so hostname and snapshot name are not visible; the backup time is, through the plaintext timestamp in the manifest, which is accepted behaviour.

README, ARCHITECTURE.md, docs/DATAMODEL.md and docs/REPOSTRUCTURE.md now show the hashed layout, the derivation is documented once, and the privacy section lists what the unencrypted manifest exposes. Two code comments that claimed the timestamp was hidden are corrected. No behaviour change.

Judgement call: docs/DATAMODEL.md was not named in the issue but had the same error.

model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
2026-09-21 19:24:44 +02:00
clawbot d2a0510cb4 Trigger CI on next, not only main (closes #122)
check / check (pull_request) Failing after 0s
check / check (push) Successful in 3m33s
`check.yml` ran only on push to `main` and on pull requests against `main`. Every unit is a PR based on `next`, and `next` is pushed on each squash-merge, so no unit PR and no push to `next` ever ran CI; a broken `next` would first surface on the milestone PR. `next` is added to both branch lists; nothing else in the workflow changes. The README Entrypoints section now says where CI runs.

Disclosure: the CI run on the PR itself fired (the proof the trigger works) but was red because the runner had no disk space left before any check step ran; the local gate was green.

Model: opus-4-8 (implementation and review)
model: claude-fable-5
2026-09-21 14:55:59 +02:00
clawbot 583f65040a Correct --cron flag help to name warnings as unsuppressed (closes #87)
check / check (pull_request) Failing after 0s
`--cron` sets the UI quiet, but `Warningf` and `Errorf` are unconditional, and the snapshot summary is routed through `Warningf` on purpose so cron delivers something on a successful run. The help string said `silent unless error`, so a user could read normal cron output as a failure. It now says `silent unless warning or error`, matching the README. String only; no behavior change.

Model: opus-4-8 (implementation and review)
model: claude-fable-5
2026-09-21 14:55:45 +02:00
sneak d257f8f658 Lint in a container as a build step, via Dockerfile.lint (closes #113)
check / check (pull_request) Successful in 2m58s
Every lint run now happens inside its own container, invoked through
script/lint, and linting is a build step rather than a container
command: a successful build of the new root Dockerfile.lint IS a clean
lint. That shape also works where the docker daemon is remote and bind
mounts are impossible.

Its FROM line -- golangci/golangci-lint:v2.12.2, pinned by digest -- is
now the only pin of the linter version in this repo.

A container per run has its own lint cache and its own golangci-lint
lock, both discarded with it, so neither cross-worktree contamination
nor lock contention exists any more. The machinery that defended
against them is therefore gone: the per-worktree cache directories, the
lock-retry loop, and script/lint-audit, which existed to catch findings
replayed from a cache that no longer exists. So is the host lint path
in its entirety -- the native escape hatch, its version detection, and
VAULTIK_LINT_IN_CONTAINER in both script/lint and the Dockerfile.
Nothing lints on the host, at any version.

A cached build lints nothing, so the CHECK_EPOCH mechanism the product
Dockerfile already used is what makes a green mean something:
ARG CHECK_EPOCH with no default, placed below the module layers so
dependency caching survives, a `RUN [ -n "$CHECK_EPOCH" ] || exit 1`
guard so a build that withholds the arg fails instead of replaying, and
the value expanded into the lint command itself. script/lint computes
`epoch="$(date +%s%N)$$"` as a bare assignment on its own line, because
inline in the argument a failing substitution does not abort under
`set -eu` and yields a constant empty epoch -- which is exactly the
false green being prevented.

The product Dockerfile loses its lint stage rather than gaining a
second linter pin. That stage ran `make lint`, which is now
`docker build`: docker-in-docker inside a BuildKit step with no daemon.
Calling golangci-lint directly there instead would have meant two
independently bumpable digests for one tool. `make fmt-check` moves
beside `make test` in the builder stage, and script/cibuild now builds
Dockerfile.lint and then Dockerfile, each with its own fresh epoch,
failing on either. Consequence, stated in comments rather than left to
be discovered: script/docker builds the product image only and no
longer lints; script/check and script/cibuild are the gates.

`golangci-lint config verify` runs as its own epoch-keyed layer, above
the lint. It is not belt-and-braces: `golangci-lint run` rejects a
config it cannot PARSE but silently IGNORES an unknown top-level KEY.
Renaming .golangci.yml's `linters:` to `linterz:` -- one character --
discards `default: all`, the disable list and every threshold, leaves
only the small default linter set running, and exits 0 reporting
`0 issues.` on a tree the real config fails with an lll finding, in a
run whose lint layer demonstrably executed. That is a set-but-
ineffective config falling back to defaults instead of failing loudly,
sitting in the gate's own configuration. `config verify` catches it and
does so with the network genuinely off at this pin: under
`docker run --network none` against the pinned digest it exits 0 on
this repo's config and exits 3 on the `linterz:` variant. It is keyed
on CHECK_EPOCH like the lint itself, because a cached validation
validates nothing.

script/lint-fix is kept, reimplemented as a bind-mounted
docker run against the image parsed out of Dockerfile.lint -- a build
step cannot write fixes back to the worktree -- and its header states
outright that it is a developer convenience, never a gate, and needs a
local daemon.

cmd/vaultik/lintdocker_test.go parses both Dockerfiles and both scripts
and fails if any part of the mechanism is dropped: the digest pin, the
defaultless ARG below `go mod download`, the emptiness guard, the
expansion of the epoch into each check command, the bare per-invocation
epoch assignment in both scripts, cibuild building both files, the
config verification running before the lint, and -- structurally, not
by searching for one retired variable name -- that no script invokes
golangci-lint except through docker. Every one of those losses is
silent: the build still exits 0 and nothing is checked, which is why
they are asserted rather than trusted. The scanner behind the last of
those has its own test, because a structural check that goes blind
passes on every tree, including a broken one.

script/lint takes no arguments now, and says so instead of dropping
them: a build step has no command line to pass linter flags to.
2026-08-10 13:38:45 +00:00
48 changed files with 2324 additions and 1375 deletions
+2 -2
View File
@@ -1,9 +1,9 @@
name: check
on:
push:
branches: [main]
branches: [main, next]
pull_request:
branches: [main]
branches: [main, next]
jobs:
check:
runs-on: ubuntu-latest
+18 -25
View File
@@ -20,33 +20,21 @@ jobs:
# check.yml runs script/cibuild, which does all of its work inside
# the digest-pinned Dockerfile images -- so without this step the
# release either fails at the before-hook or, worse, ships binaries
# built by whatever unpinned Go the runner happens to carry.
# REPO_POLICIES.md requires every external reference to be pinned,
# and script/release already refuses a goreleaser that is not the
# pinned build; the compiler that actually produces the artifacts
# is the last thing that should be exempt from that.
# built by whatever Go the runner happens to carry.
#
# go-version-file rather than a literal: go.mod's `go 1.26.1` is
# the single source of truth for the toolchain, the same way the
# Dockerfile FROM line is the single source of truth for the
# linter version that script/lint enforces. It is a three-component
# version, so setup-go resolves it exactly -- no silent drift onto
# a newer patch release.
#
# actions/setup-go v5.6.0, 2025-12-15. Pinned by commit sha, like
# the checkout above. v5.x is a node20 action, matching the node20
# actions/checkout v4 already in use here; the v6/v7 line requires
# a node24 runner, which this Gitea runner has never been asked
# for and cannot be assumed to provide.
# actions/setup-go would pin the action by commit sha, but the Go
# tarball it downloads at runtime is verified against no value in
# this repo, and the action exposes no checksum input.
# REPO_POLICIES.md requires every external reference to be pinned
# by hash with no exceptions, and this is the compiler that
# produces the published binaries -- the input where a substituted
# artifact matters most. So Go is installed the way goreleaser is:
# script/install-go downloads the exact archive for go.mod's `go`
# directive and refuses it unless its sha256 matches the value
# committed in the script, then puts .tool/go/bin on PATH for the
# steps below.
- name: Install Go
uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff
with:
go-version-file: go.mod
# setup-go's module cache needs a runner-side cache backend.
# A release is cut rarely and a cold module download costs
# seconds; a release failing because a cache service is absent
# costs a re-tag. Off, deliberately.
cache: false
run: script/install-go
- name: Install goreleaser
run: script/install-goreleaser
- name: Release
@@ -58,3 +46,8 @@ jobs:
# It is deliberately not the runner's automatic token, which is
# not guaranteed to carry that scope.
GITEA_TOKEN: ${{ secrets.RELEASE_TOKEN }}
# Build with the toolchain install-go just verified, never a
# different one auto-downloaded from a `toolchain` directive:
# the point of the hash pin is that this exact compiler makes
# the release.
GOTOOLCHAIN: local
+8 -3
View File
@@ -104,7 +104,12 @@ Version: 2025-06-08
13. Pre-1.0: NEVER write database migrations. There are no live databases
anywhere — every user's local index can be rebuilt from a fresh full
backup. When the schema changes, just change `schema.sql` (and any code
that touches the affected tables). The local index is disposable until
1.0 ships and is tagged.
backup. To change the schema, edit `internal/database/schema/001.sql`
(and any code that touches the affected tables) directly; do not add new
numbered schema files. Those numbered files and the `schema_migrations`
table they populate only bootstrap a fresh database — they are not an
upgrade path. The local index is disposable until 1.0 ships and is
tagged; once 1.0 is tagged that clause expires and the question of
upgrading existing indexes returns. See [`docs/DATAMODEL.md`](docs/DATAMODEL.md)
for the full explanation.
+7 -1
View File
@@ -366,11 +366,17 @@ bucket/
│ └── {full-hash} # Compressed+encrypted blob
└── metadata/
└── {snapshot-id}/
└── {remote-key}/
├── db.zst.age # Encrypted binary SQLite database
└── manifest.json.zst # Blob list (for pruning/verification)
```
The `{remote-key}` directory name is a one-way double SHA-256 hash of the human
snapshot ID, so the human ID (hostname, snapshot name, timestamp) is never
written to the store as a directory name. See
[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#remote-key-derivation) for the
derivation and a worked example.
## Thread Safety
- `Packer`: Thread-safe via mutex. Multiple goroutines can call `AddChunk()`.
+27 -49
View File
@@ -1,24 +1,32 @@
# Lint stage
# This file has no lint stage, deliberately.
#
# This FROM line is the single source of truth for the linter version:
# script/lint parses the image reference out of it and runs that exact
# image, so a local `make lint` and CI use the same linter. Bump the
# linter here (tag AND digest) and nowhere else.
# Linting lives in Dockerfile.lint, built by script/lint, and
# script/cibuild builds both. A lint stage here would have to either
# shell out to `make lint` -- which is now `docker build`, so
# docker-in-docker inside a BuildKit step with no daemon -- or call
# golangci-lint directly, which would mean a second, independently
# bumpable digest pin for the linter alongside the one in
# Dockerfile.lint. Two pins for one tool is the drift that
# https://git.eeqj.de/sneak/vaultik/issues/78 was filed over. See
# https://git.eeqj.de/sneak/vaultik/issues/113 for the ruling.
#
# golangci/golangci-lint:v2.12.2-alpine, 2026-08-07
FROM golangci/golangci-lint:v2.12.2-alpine@sha256:91b27804074a0bacea298707f016911e60cf0cdbc6c7bf5ccacb5f0606d18d60 AS lint
# Consequence, stated rather than left to be discovered: script/docker
# builds this file only and therefore does not lint. `make fmt-check`
# and `make test` still run here, so what a green build of this file
# means is "formatted, tested, and it compiles" -- the lint verdict
# comes from script/lint or script/cibuild.
# Build stage
# golang:1.26.1-alpine, 2026-03-17
FROM golang:1.26.1-alpine@sha256:2389ebfa5b7f43eeafbd6be0c3700cc46690ef842ad962f6c5bd6be49ed82039 AS builder
ARG VERSION=dev
# Build tooling: make, plus a C toolchain because `go test -race` needs cgo.
# The sqlite driver is pure Go (modernc.org/sqlite), so no sqlite library or
# CLI is required.
RUN apk add --no-cache make build-base
# The context signal for script/lint's native path. This stage runs
# `make lint` with no docker daemon available, so it is the one place
# that must run the golangci-lint on PATH directly. script/lint takes
# that path only when this is set AND the version matches the pin above;
# version equality alone would also admit a developer's locally
# installed copy on a host, bypassing the digest pin (issue #80).
# Nothing outside this stage sets it.
ENV VAULTIK_LINT_IN_CONTAINER=1
WORKDIR /src
# Copy go mod files first for better layer caching
@@ -28,7 +36,7 @@ RUN go mod download
# Copy source code
COPY . .
# Run formatting check and linter.
# Run the format check and the tests.
#
# CHECK_EPOCH must stay immediately above these RUNs. These layers are
# keyed on its value, so they are cache-eligible only for a value
@@ -47,45 +55,15 @@ COPY . .
# runs the checks and every one after it on an unchanged tree replays
# these layers from cache, executes nothing, and still exits 0. Failed
# steps are never cached, so the guard fails on EVERY invocation rather
# than once -- a bare `docker build .` is now a loud error, not a quiet
# than once -- a bare `docker build .` is a loud error, not a quiet
# green. Do not give CHECK_EPOCH a default value; a default would
# satisfy the guard with a constant and restore the hole.
#
# ARG scope is per-stage, so the builder stage declares its own.
# Everything above this line (apk, go.mod, `go mod download`) is
# deliberately outside the busted range and keeps caching.
ARG CHECK_EPOCH
RUN [ -n "$CHECK_EPOCH" ] || exit 1
RUN echo "check epoch: ${CHECK_EPOCH}" && make fmt-check
RUN echo "check epoch: ${CHECK_EPOCH}" && make lint
# Build stage
# golang:1.26.1-alpine, 2026-03-17
FROM golang:1.26.1-alpine@sha256:2389ebfa5b7f43eeafbd6be0c3700cc46690ef842ad962f6c5bd6be49ed82039 AS builder
# Depend on lint stage passing
COPY --from=lint /src/go.sum /dev/null
ARG VERSION=dev
# Install build dependencies for CGO (mattn/go-sqlite3) and sqlite3 CLI (tests)
RUN apk add --no-cache make build-base sqlite
WORKDIR /src
# Copy go mod files first for better layer caching
COPY go.mod go.sum ./
RUN go mod download
# Copy source code
COPY . .
# Run tests. See the CHECK_EPOCH comment in the lint stage for the
# mechanism; ARG scope is per-stage, so this stage needs its own
# declaration, its own guard, and its own expansion, and they must stay
# immediately above the check RUN.
ARG CHECK_EPOCH
RUN [ -n "$CHECK_EPOCH" ] || exit 1
RUN echo "check epoch: ${CHECK_EPOCH}" && make test
# Build (pure Go, no CGO required since we use modernc.org/sqlite)
@@ -95,7 +73,7 @@ RUN CGO_ENABLED=0 go build -ldflags "-X 'sneak.berlin/go/vaultik/internal/global
# alpine:3.21, 2026-02-25
FROM alpine:3.21@sha256:c3f8e73fdb79deaebaa2037150150191b9dcbfba68b4a46d70103204c53f4709
RUN apk add --no-cache ca-certificates sqlite
RUN apk add --no-cache ca-certificates
# Copy binary from builder
COPY --from=builder /vaultik /usr/local/bin/vaultik
+104
View File
@@ -0,0 +1,104 @@
# Lint image.
#
# Every lint run in this repo happens inside this image, invoked through
# script/lint, and linting is a BUILD STEP rather than a container
# command: a successful build of this file IS a clean lint. That shape
# also works where the docker daemon is remote and bind mounts are
# impossible, which `docker run` against a mounted worktree does not.
#
# This FROM line is the single source of truth for the linter version in
# this repo. Nothing else pins golangci-lint: the product Dockerfile has
# no lint stage, deliberately, so there is no second digest to bump and
# no pair of pins that can drift apart. Bump the tag AND the digest here
# and nowhere else.
#
# Note for readers coming from REPO_POLICIES.md: that document still
# describes the older pattern, a lint stage inside the product
# Dockerfile wired up with `COPY --from=lint /src/go.sum /dev/null`.
# That pattern is superseded here by the owner's ruling recorded in
# https://git.eeqj.de/sneak/vaultik/issues/113 -- lint runs in its own
# image, per run, with its own cache and its own lock, which is what
# makes concurrent runs on one host safe. The policy text is org-wide
# and is being amended separately; this file is what this repo does.
#
# golangci/golangci-lint:v2.12.2, 2026-08-10
FROM golangci/golangci-lint:v2.12.2@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240
WORKDIR /src
# Copy the dependency manifests first so the module download layer stays
# cached until they change. Everything above the ARG below is cacheable
# on purpose; a cold module download on every lint would make the inner
# loop unusable and buys nothing, because it is not what the gate is
# asserting.
COPY go.mod go.sum ./
RUN go mod download
COPY . .
# Force the check layers to execute on every invocation.
#
# CHECK_EPOCH must stay immediately above the RUNs below. Those layers
# are keyed on its value, so they are cache-eligible only for a value
# already built against this same tree; script/lint and script/cibuild
# each pass a fresh value on every invocation, which is what makes their
# green mean the linter really ran. Without it, `docker build -f
# Dockerfile.lint .` on an unchanged tree exits 0 in well under a second
# having linted nothing.
#
# The value is expanded into each check command itself rather than left
# to a bare declaration, so the cache miss does not depend on BuildKit's
# unreferenced-ARG handling staying as it is. It also puts the epoch in
# the build log, where a reader can see the layer was keyed fresh.
#
# The guard is what makes a build that omits --build-arg fail instead of
# lie. An unset ARG is an empty string, and an empty string is a
# perfectly stable cache key: without the guard the first such build
# lints and every one after it on an unchanged tree replays this layer,
# executes nothing, and still exits 0. Failed steps are never cached, so
# the guard fails on EVERY invocation rather than once. Do not give
# CHECK_EPOCH a default value; a default would satisfy the guard with a
# constant and restore the hole.
ARG CHECK_EPOCH
RUN [ -n "$CHECK_EPOCH" ] || exit 1
# Validate .golangci.yml before linting with it.
#
# This is not belt-and-braces; it closes a hole that `golangci-lint run`
# leaves wide open. `run` rejects YAML it cannot PARSE, but it silently
# IGNORES an unknown top-level KEY. Renaming `linters:` to `linterz:` --
# one character -- discards `default: all`, the whole disable list and
# every threshold, leaves only golangci-lint's small default linter set
# running, and exits 0 reporting `0 issues.` on a tree the real config
# fails. Demonstrated on this repo at this pin, recorded on
# https://git.eeqj.de/sneak/vaultik/pulls/114: with a planted
# over-length line, `script/lint` exits 1 naming the `revive` finding
# with `linters:` and exits 0 with `linterz:`. A set-but-ineffective
# config quietly falling back to defaults is precisely the false-green
# class this gate exists to eliminate, so it must not sit in the gate's
# own configuration.
#
# `config verify` catches it, and it does so OFFLINE at this pinned
# version -- verified, not assumed. Under `docker run --network none`
# against the pinned digest it exits 0 on this repo's config and exits 3
# on the `linterz:` variant with `additional properties 'linterz' not
# allowed`. An earlier revision of this file asserted the opposite, that
# the schema is fetched over live HTTPS from an unpinned URL, and used
# that to justify omitting this line. That claim was false at v2.12.2;
# the schema is embedded. If a future bump reintroduces a network fetch
# the failure is loud and this comment is where to record it.
#
# It is keyed on CHECK_EPOCH, like the lint run below, so it executes on
# every invocation. Content-addressing alone would arguably be enough --
# .golangci.yml arrives through `COPY . .`, so a cache hit here implies
# a byte-identical config was validated when the layer really ran. That
# argument is exactly the one that would also excuse caching the lint
# layer, and this repo has ruled it insufficient: a cached check layer
# checks nothing, and the cost of being wrong is silent. Forcing it costs
# milliseconds and puts the epoch in the log, where a reader can see that
# this validation ran rather than being replayed.
RUN echo "check epoch: ${CHECK_EPOCH}" && \
golangci-lint config verify --config .golangci.yml
RUN echo "check epoch: ${CHECK_EPOCH}" && \
golangci-lint run --config .golangci.yml ./...
+4 -4
View File
@@ -87,10 +87,10 @@ clean:
go clean
# Install dependencies. The linter is deliberately not installed here:
# script/lint runs the digest-pinned golangci-lint image declared by the
# Dockerfile's lint stage, which is the single source of truth for the
# linter version. A second, separately pinned copy on PATH could drift
# from it and make a local `make lint` disagree with CI.
# script/lint lints by building Dockerfile.lint, whose FROM line is the
# single source of truth for the linter version. A second, separately
# pinned copy on PATH could drift from it and make a local `make lint`
# disagree with CI.
deps:
go mod download
+154 -51
View File
@@ -84,6 +84,57 @@ VAULTIK_AGE_SECRET_KEY='AGE-SECRET-KEY-...' vaultik snapshot restore <snapshot-i
# 0 3 * * * vaultik snapshot create --cron --prune --keep-newer-than 4w
```
## restoring on another machine
Restoring on a host that never ran the backup — a replacement machine
after the original is gone — is the case vaultik is built for. That host
needs only three things: the `vaultik` binary, the age **private** key,
and the storage credentials for the destination. It does **not** need the
local index, the original config file, or the original hostname.
```sh
# install
go install sneak.berlin/go/vaultik/cmd/vaultik@latest
# create a config and point it at the ORIGINAL backup destination
vaultik config init
vaultik config set storage_url "s3://bucket/prefix?endpoint=https://s3.example.com"
vaultik config set s3.access_key_id "..."
vaultik config set s3.secret_access_key "..."
# see what is on the destination store
vaultik snapshot list
```
`snapshot list` reads the destination store without the private key. A
snapshot that is not in this host's (empty) local index is shown as
remote-only: its row is identified by `<remote only:...>` rather than by
a `hostname_name_timestamp` name, because the name lives only in the
local index and the encrypted database and cannot be recovered from the
store. Its timestamp and compressed size are real. (See the `snapshot
list` description under [command details](#command-details) for the full
explanation.)
Use that remote key — the hex printed inside `<remote only:...>`, or the
full `remote_key` from `snapshot list --json` — to restore and verify:
```sh
# restore everything to /tmp/restored, then check every restored file's
# chunk hashes
VAULTIK_AGE_SECRET_KEY='AGE-SECRET-KEY-...' \
vaultik snapshot restore --verify <remote-key> /tmp/restored
# optionally, deep-verify the snapshot against the store (downloads and
# cryptographically checks every blob)
VAULTIK_AGE_SECRET_KEY='AGE-SECRET-KEY-...' \
vaultik snapshot verify --deep <remote-key>
```
`age_recipients` (the public key) is not needed to restore — only the
private key in `VAULTIK_AGE_SECRET_KEY`. Both the abbreviated key printed
in the table and the full 64-character key from `--json` are accepted; a
leading part of the key is enough as long as it is unambiguous.
---
## cli
@@ -245,13 +296,16 @@ local index alone, and still exits zero.
* Default (shallow): checks that all blobs referenced in the manifest exist in storage
* `--deep`: Downloads and decrypts each blob, verifies chunk hashes against the
encrypted metadata database
* Accepts the same identifiers as `snapshot restore`: a snapshot ID, or a
remote-only snapshot's remote key (or an unambiguous leading part of it)
* `--json`: Output results as JSON
**`snapshot purge`**: Remove old snapshots based on criteria. Retention is
per-snapshot-name (`--keep-latest` keeps the latest of each name, not the
latest globally).
* `--keep-latest`: Keep only the most recent snapshot of each name
* `--older-than <duration>`: Remove snapshots older than duration (e.g. `30d`, `6m`, `1y`)
* `--older-than <duration>`: Remove snapshots older than duration (e.g. `30d`,
`4w`, `6mo`, `1y`; `m` is minutes, `mo` is months)
* `--snapshot <name>`: Restrict to specific snapshot names (repeat for multiple)
* `--force`: Skip confirmation prompt
@@ -274,6 +328,10 @@ on the destination in one go, use `vaultik remote nuke --force`.
**`snapshot restore`**: Restore files from a backup snapshot.
* Requires `VAULTIK_AGE_SECRET_KEY` environment variable
* Accepts a snapshot ID, or — for a snapshot only on the destination
store — its remote key (or an unambiguous leading part of it) as shown
by `snapshot list`. See
[restoring on another machine](#restoring-on-another-machine).
* Optional path arguments to restore specific files/directories (default: all)
* Preserves file permissions, timestamps, ownership (ownership requires root),
symlinks, and empty directories
@@ -344,7 +402,7 @@ both are set.
├── blobs/
│ └── <aa>/<bb>/<full_blob_hash>
└── metadata/
└── <snapshot_id>/
└── <remote-key>/
├── db.zst.age # Encrypted binary SQLite database
└── manifest.json.zst # Unencrypted blob list (for pruning)
```
@@ -355,8 +413,18 @@ both are set.
* `manifest.json.zst` is an unencrypted compressed JSON blob list, enabling
pruning without the private key
Snapshot IDs follow the format `<hostname>_<snapshot-name>_<RFC3339-timestamp>`
(e.g. `server1_home_2025-06-01T12:00:00Z`).
Snapshot IDs follow the human-readable format
`<hostname>_<snapshot-name>_<RFC3339-timestamp>` (e.g.
`server1_home_2025-06-01T12:00:00Z`), but this ID is never written to the
destination store in plaintext. Each snapshot's metadata directory is named
with its `<remote-key>`, a one-way double SHA-256 hash of the ID, so a listing
of the store reveals no hostname or snapshot name. The backup time is not
hidden: manifest.json.zst carries a plaintext timestamp, and object
modification times are visible at the storage layer regardless. For example,
`server1_home_2025-06-01T12:00:00Z` is stored under
`metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`.
See [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#remote-key-derivation) for the
derivation.
### data flow
@@ -373,7 +441,7 @@ Snapshot IDs follow the format `<hostname>_<snapshot-name>_<RFC3339-timestamp>`
**restore:**
1. Download and decrypt `metadata/<snapshot_id>/db.zst.age`
1. Download and decrypt `metadata/<remote-key>/db.zst.age`
2. Open the binary SQLite database
3. Query files (optionally filtered by paths)
4. Download and decrypt required blobs
@@ -446,9 +514,13 @@ Key fields:
sequentially. Restore speed is bound by single-stream throughput.
* **Device nodes, named pipes, and sockets are silently skipped.** Only
regular files, directories, and symlinks are backed up.
* **No database migrations.** If the local SQLite schema changes between
versions, delete the local database (`vaultik database delete`) and run
a full backup. Remote storage is unaffected.
* **No upgrade path between versions.** There is no supported way to carry
an existing local index across a schema change; if the local SQLite
schema changes between versions, delete the local database (`vaultik
database delete`) and run a full backup. Remote storage is unaffected.
(The binary does embed numbered schema files and a `schema_migrations`
table to bootstrap a fresh database — see [`docs/DATAMODEL.md`](docs/DATAMODEL.md)
— but that is not an upgrade path.)
* **Files that change during backup may be inconsistent.** There is no
filesystem snapshot or freeze. If a file is modified between the scan
and chunk phases, the backed-up copy may reflect a partial write.
@@ -514,14 +586,12 @@ priority.
### infrastructure
* **Cross-machine restore documentation.** The "restore from
another host" workflow works but isn't documented as a
first-class operation in this README. Worth a dedicated section
once it's settled.
* **Schema migrations.** Currently nonexistent — pre-1.0 schema
changes are handled by `vaultik database delete` plus a full
re-scan. Post-1.0 we'll need a migration story to keep existing
index databases usable across upgrades.
* **Cross-version schema upgrades.** There is no upgrade path between
released versions — pre-1.0 schema changes are handled by `vaultik
database delete` plus a full re-scan (see
[`docs/DATAMODEL.md`](docs/DATAMODEL.md)). Post-1.0 we'll need a
migration story to keep existing index databases usable across
upgrades.
* **Storage backend coverage tests.** S3, file://, and rclone://
all share the Storer interface but the rclone path is the least
exercised in CI.
@@ -598,11 +668,11 @@ regardless of color setting (emoji are not color).
* Go 1.26 or later
* Docker, with a reachable daemon, to lint, check, or commit:
`script/lint` runs the digest-pinned `golangci-lint` image declared by
the `Dockerfile` lint stage, and `make check` and the pre-commit hook
both run it. A `golangci-lint` installed on `PATH` is not a substitute
and is never used on a host, whatever its version.
* `sqlite3` CLI, which the test suite shells out to
`script/lint` lints by building `Dockerfile.lint`, which runs the
digest-pinned `golangci-lint` image as a build step, and `make check`
and the pre-commit hook both run it. A `golangci-lint` installed on
`PATH` is not a substitute and is never used on a host, whatever its
version.
* S3-compatible object storage (or local filesystem, or rclone remote)
## development workflow
@@ -633,8 +703,8 @@ standard: normalized scripts in `script/` are the entrypoints for the
development workflow, and the Makefile targets are thin shims that call
them. We provide:
* `script/bootstrap` — install all development dependencies (go, sqlite3,
Go module download). It deliberately does not install `golangci-lint`;
* `script/bootstrap` — install all development dependencies (go, Go
module download). It deliberately does not install `golangci-lint`;
see `script/lint` below.
* `script/setup` — make a fresh clone ready for development: runs
`script/bootstrap`, then `script/install-precommit`
@@ -648,6 +718,14 @@ them. We provide:
called by `script/bootstrap`; the release workflow calls it directly
because it needs `goreleaser` but not the Docker daemon
`script/bootstrap` insists on.
* `script/install-go` — install the Go toolchain named by `go.mod`'s
`go` directive into `.tool/go` from a sha256-verified `go.dev`
archive, and put it on `PATH`. Idempotent. Called only by the release
workflow, which needs a host Go for `goreleaser` to shell out to;
nothing else on the release runner does. `actions/setup-go` is not
used because it verifies the downloaded toolchain against no value in
this repo. Bumping Go edits `go.mod`, the checksum in this script, and
the `Dockerfile` `golang` digest together.
* `script/release` — cross-compile and publish the release artifacts
with the pinned `goreleaser`. Refuses a `goreleaser` on `PATH` whose
version is not the pinned one, on the same reasoning as `script/lint`.
@@ -671,46 +749,71 @@ them. We provide:
diverges from the 30s `REPO_POLICIES.md` mandates; the reasoning is in
the comment in the script, and issue #101 proposes amending the policy
text.
* `script/lint`run `golangci-lint run ./...` at the exact version CI
uses, by running the digest-pinned `golangci-lint` image declared by
the `Dockerfile` lint stage (requires Docker; it fails loudly rather
than falling back to a differently versioned `golangci-lint` on
`PATH`). That `FROM` line is the single source of truth for the linter
version — bump it there and nowhere else.
* `script/lint`lint by building `Dockerfile.lint`, which runs
`golangci-lint run --config .golangci.yml ./...` as a build step
inside the digest-pinned `golangci-lint` image, so a successful build
*is* a clean lint. Nothing lints on the host, at any version, ever;
the script requires Docker and fails loudly rather than falling back
to a `golangci-lint` on `PATH`. That `FROM` line is the single source
of truth for the linter version — bump it there and nowhere else.
It takes no arguments, because a build step has no command line to
pass flags to, and it passes a fresh `--build-arg CHECK_EPOCH` on
every invocation so the lint layer cannot be replayed from cache (see
`script/cibuild` below for what that mechanism defends against). To
watch the linter execute, run it as
`BUILDKIT_PROGRESS=plain script/lint` and check that the lint layer
says `RUN … golangci-lint` rather than `CACHED`.
One container per run means one lint cache and one `golangci-lint`
lock per run, both private to it and discarded with it, so concurrent
runs on one host cannot contaminate or block each other.
* `script/lint-fix` — apply the linter's autofixes (rewrites files),
using the same pinned linter
using the same pinned image, parsed out of `Dockerfile.lint`. It
cannot be a build step, because fixes have to land in the worktree, so
it bind-mounts the tree into a `docker run` and therefore needs a
*local* daemon. It is a developer convenience and never a gate: no
gate reads its exit status. Run `make lint` afterwards to find out
whether the tree is clean.
* `script/fmt` — format all code (writes)
* `script/fmt-check` — check formatting (read-only)
* `script/check` — run `script/test`, `script/lint`, and
`script/fmt-check`. This is authoritative *because* `script/lint` uses
the pinned linter: a local `make check` and CI cannot disagree about
lint findings.
`script/fmt-check`. This is authoritative *because* `script/lint`
builds `Dockerfile.lint`: a local `make check` and CI cannot disagree
about lint findings.
* `script/docker` — build the Docker image tagged via
`script/projectname`. Passes a fresh `--build-arg CHECK_EPOCH` for the
same reason `script/cibuild` does, so a local image build cannot be
green on checks it replayed from cache.
* `script/cibuild` — CI entrypoint: `docker build` (the `Dockerfile`
runs `make fmt-check` and `make lint` in its lint stage and `make
test` in its builder stage). This is the full CI-equivalent gate — it
runs the checks in the same containers CI does, from a clean copy of
the tree, so it also catches anything that depends on host state. It
passes a fresh `--build-arg CHECK_EPOCH`, unique per invocation, which
the `Dockerfile` declares immediately above the check `RUN`s in both
stages and expands into each check command. Those layers are keyed on
green on checks it replayed from cache. It builds the *product* image
only, and the product `Dockerfile` has no lint stage, so it does not
lint: a green here means formatted, tested, and it compiles.
* `script/cibuild` — CI entrypoint, and the full gate. Two builds, in
order: `Dockerfile.lint` (the linter, as a build step) and then
`Dockerfile` (`make fmt-check` and `make test` in its builder stage,
then the product image). Either failing fails the script. It runs the
checks in the same containers CI does, from a clean copy of the tree,
so it also catches anything that depends on host state.
`.gitea/workflows/check.yml` runs it on every push to `main` and
`next` and on every pull request against either.
It passes a fresh `--build-arg CHECK_EPOCH` to each build, unique per
invocation, which both files declare immediately above their check
`RUN`s and expand into each check command. Those layers are keyed on
that value, so a new value re-runs them even on a byte-identical tree,
and a green from this script means the checks executed. Dependency and
module layers sit above the `ARG` and still cache, so a build is not
cold.
A build that supplies no `CHECK_EPOCH` — a bare `docker build .`
fails rather than lying. An unset `ARG` is an empty string and an
empty string is a stable cache key, so without a guard such a build
would serve all three check layers from cache, execute nothing, and
still exit 0. Each check stage therefore asserts the value is
non-empty before running anything, and because failed steps are never
cached that assertion fires on every invocation rather than once. Use
`script/cibuild` (or `script/docker`, which passes the same arg); a
bare `docker build .` is now a loud error.
A build that supplies no `CHECK_EPOCH` — a bare `docker build .` or
`docker build -f Dockerfile.lint .` — fails rather than lying. An
unset `ARG` is an empty string and an empty string is a stable cache
key, so without a guard such a build would serve every check layer
from cache, execute nothing, and still exit 0. Each file therefore
asserts the value is non-empty before running anything, and because
failed steps are never cached that assertion fires on every
invocation rather than once. Use `script/lint`, `script/docker` or
`script/cibuild`, which pass the arg; a bare `docker build` is a loud
error.
* `script/precommit` — pre-commit gate: `go mod tidy` + `go fmt` (must
not change files), then `script/check`
* `script/install-precommit` — install the git pre-commit hook that
+133 -1
View File
@@ -25,6 +25,138 @@ release" is exactly the contradiction
# Completed Steps
- 2026-09-21: Stopped `prune` from reporting a failed row count as 0
([issue #96](https://git.eeqj.de/sneak/vaultik/issues/96)). The seven
`getTableCount` reads in `PruneDatabase` discarded their error, so a
query that could not run became a plausible `0` and the before/after
delta computed from it looked like real work. Each read now logs at
warn on failure and renders as `unknown`, never `0`, so an empty table
is distinguishable from one that could not be queried. The counts have
no `--json` representation — under `--json` the summary is suppressed
entirely — so nothing there can show a false `0`.
- 2026-09-21: Made the s3 storage backend report a missing object as
`storage.ErrNotFound`, like the `file` and `rclone` backends and as the
`Storer` interface documents. `S3Storer.Get` and `Stat` returned the raw
AWS SDK error, so `errors.Is(err, storage.ErrNotFound)` was false on s3
and callers branched differently per backend. Added a small `s3.IsNotFound`
helper (reused by `HeadObject`) and a test that a missing key maps to
`ErrNotFound`
([issue #129](https://git.eeqj.de/sneak/vaultik/issues/129)).
- 2026-09-21: Fixed `verify --deep` reporting healthy snapshots as
corrupt. Its final blob-integrity check hashed the encrypted
downloaded bytes with a single SHA256 and compared that to the blob
ID, which is the double SHA256 of the plaintext, so the two could
never match. It now hashes the decompressed plaintext and compares the
double SHA256. Added a test that backs up a real snapshot, deep-verifies
it, then flips a byte in one stored blob and confirms deep verification
then fails
([issue #131](https://git.eeqj.de/sneak/vaultik/issues/131)).
- 2026-09-21: Made `snapshot create` VACUUM the per-snapshot metadata
database through the `modernc.org/sqlite` driver instead of shelling
out to the external `sqlite` command-line binary (issue #120). A
backup no longer needs that binary on `PATH`, so `make check` passes
on a stock `go install` host; `script/bootstrap` and the `Dockerfile`
(both the test-build and the shipped runtime stage) no longer install
it, and a new test asserts the uploaded database keeps no pages from
deleted rows. Dropped the now-false note on the 2026-08-07 entry below
that said bootstrap installs it.
- 2026-09-21: Made `.gitea/workflows/check.yml` run on pushes to `main`
and `next` and on pull requests against either, so unit PRs (whose
base is `next`) and `next` itself get a CI run instead of relying on a
local `make check`
([issue #122](https://git.eeqj.de/sneak/vaultik/issues/122)).
- 2026-09-21: Hash-verified the Go toolchain in the release workflow
([issue #105](https://git.eeqj.de/sneak/vaultik/issues/105)). New
`script/install-go` downloads the exact `go.dev` archive for `go.mod`'s
`go` directive and refuses it unless its sha256 matches a value
committed in the script; `.gitea/workflows/release.yml` calls it
instead of `actions/setup-go`, which verified the downloaded toolchain
against nothing in the repo. `GOTOOLCHAIN: local` on the release step
keeps that exact compiler from auto-switching. Bumping Go now touches
`go.mod`, the checksum, and the `Dockerfile` `golang` digest together.
- 2026-09-21: Collapsed the two duration parsers into one and fixed the
`--older-than` months example
([issue #123](https://git.eeqj.de/sneak/vaultik/issues/123)). Two
functions named `parseDuration` existed with different grammars;
`snapshot purge --older-than` and `--keep-newer-than` both already went
through the one in `internal/vaultik`, while the richer copy in
`internal/cli/duration.go` was reachable only from its own test. Kept
the live-path parser and deleted the unused one, so no flag's accepted
grammar changes. The trap the issue was filed over: `README.md`
documented `6m` as the months example for `--older-than`, but `m` is
minutes, so the documented command deleted every snapshot older than
six minutes on a destructive flag. Corrected the doc to `6mo` and put
both flags' help text on one example list that states `m` is minutes
and `mo` is months. The surviving parser now rejects negatives, which
it previously accepted (`-5h`) or silently made positive (`-5d`).
Table-driven tests cover every unit, `6m` as six minutes, `6mo` as 180
days, and rejection of a bare number, an unknown unit, and a negative.
- 2026-08-10: Moved every lint run into its own container, as a build
step ([issue #113](https://git.eeqj.de/sneak/vaultik/issues/113)).
New root `Dockerfile.lint`, built by `script/lint`, runs
`golangci-lint run --config .golangci.yml ./...` as a `RUN`
instruction in the digest-pinned `golangci/golangci-lint` image: a
successful build of that file *is* a clean lint, and it works even
where the daemon is remote and bind mounts are impossible. That
`FROM` line is now the only pin of the linter version in the repo.
This supersedes the per-worktree cache isolation landed for
[issue #99](https://git.eeqj.de/sneak/vaultik/issues/99). Isolation
fixed cross-worktree contamination but not lock contention — two
concurrent runs with entirely separate cache directories still
collided. A container per run has its own cache and its own lock, so
the whole class is gone, and with it the per-worktree cache
machinery, the lock-retry loop, and `script/lint-audit`, which
existed to catch replayed findings from a cache that no longer
exists. The host lint path went too: no escape hatch, no
`VAULTIK_LINT_IN_CONTAINER`, no version detection. Nothing lints on
the host at any version.
A cached build lints nothing, so the same `CHECK_EPOCH` mechanism the
product `Dockerfile` already used is what makes the green mean
something: `ARG CHECK_EPOCH` with no default below the module layers,
a `RUN [ -n "$CHECK_EPOCH" ] || exit 1` guard, the value expanded
into each check command, and a fresh `$(date +%s%N)$$` per invocation
computed as a bare assignment. `cmd/vaultik/lintdocker_test.go`
parses both Dockerfiles and both scripts and fails if any part of
that is dropped, because every way of losing it is silent. No test
asserts that no script runs the host linter: `script/lint` is the one
lint entry point and runs `golangci-lint` only inside the container,
and keeping it that way is a review matter, not something a test
proves.
The product `Dockerfile` lost its lint stage rather than gaining a
second linter pin: `make lint` is now `docker build`, so the stage
would have been docker-in-docker with no daemon, and calling
`golangci-lint` directly there would have restored the two-pins drift
of [issue #78](https://git.eeqj.de/sneak/vaultik/issues/78).
`make fmt-check` moved beside `make test` in the builder stage, and
`script/cibuild` now builds `Dockerfile.lint` and then `Dockerfile`,
each with its own fresh epoch. Consequence, stated rather than left
to be found: `script/docker` builds the product image only and no
longer lints; the gates are `script/check` and `script/cibuild`.
`golangci-lint config verify` runs as its own epoch-keyed layer,
above the lint. `golangci-lint run` rejects a config it cannot parse
but silently ignores an unknown top-level *key*: renaming `linters:`
to `linterz:` discarded `default: all` and every threshold and still
exited 0 on a tree the real config fails. `config verify` catches
that, and it does so with the network off at this pin — checked under
`docker run --network none`, not assumed. An earlier revision omitted
it on the claim that it fetches its schema over live HTTPS; that
claim was false at v2.12.2.
`script/lint-fix` is kept, reimplemented as a
bind-mounted `docker run` against the image parsed out of
`Dockerfile.lint` — it cannot be a build step, because fixes have to
land in the worktree — and marked in its header as a developer
convenience that no gate reads.
- 2026-08-09: Finished the `--json` stdout contract and gave `make build`
a rule ([issue #108](https://git.eeqj.de/sneak/vaultik/issues/108),
[issue #110](https://git.eeqj.de/sneak/vaultik/issues/110)). Two
@@ -463,7 +595,7 @@ release" is exactly the contradiction
was green was wrong.
- 2026-08-07: Added the standard `.golangci.yml` and `.editorconfig`
(issue #59); lint findings under the new config are tracked in issue
#61. `script/bootstrap` now installs sqlite3 (needed by tests).
#61.
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints,
Makefile shims, README Entrypoints section
- 2026-07-02: Consolidated CLI verbs, retired overlapping commands; bound
+363
View File
@@ -0,0 +1,363 @@
package main_test
import (
"os"
"path/filepath"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// This file guards the shape of the lint gate. Every property asserted
// here is one whose loss is SILENT: the build still exits 0, the gate
// still looks green, and nothing was linted or tested.
//
// The gate is a build step. script/lint builds Dockerfile.lint, which
// runs golangci-lint as a RUN instruction, so a successful build is a
// clean lint. BuildKit will happily replay that RUN from cache on an
// unchanged tree in well under a second, which is why the check layers
// are keyed on a CHECK_EPOCH build arg that the calling script
// regenerates per invocation, and why an empty value is a hard error
// rather than a stable cache key.
//
// These are parses rather than invocations. Shelling out to docker from
// the test suite would nest a build inside `make test`, which itself
// runs inside a build in CI. The one property a parse cannot establish
// -- that a real finding actually fails the build -- is verified by
// hand against a deliberately broken tree, recorded on the pull
// request.
//
// One property is deliberately NOT tested here: that no script runs the
// linter on the host. script/lint is the only lint entry point, and it
// runs golangci-lint only inside the container; keeping it that way is a
// review matter, not something a test in this file establishes.
// The files under guard, relative to the repository root.
const (
lintDockerfile = "Dockerfile.lint"
productDockerfile = "Dockerfile"
lintScript = "script/lint"
cibuildScript = "script/cibuild"
)
// linterBinary is the linter's command name, used to locate the
// config-verify and lint steps in Dockerfile.lint.
const linterBinary = "golangci-lint"
// checkEpochARG is the declaration, with no default value. A default
// would satisfy the non-empty guard with a constant, and a constant is
// a stable cache key: the checks would be replayed from cache forever
// after the first build.
const checkEpochARG = "ARG CHECK_EPOCH"
// checkEpochGuard is what turns a build that omits --build-arg into a
// loud failure instead of a quiet green. Failed steps are never cached,
// so it fires on every such invocation rather than once.
const checkEpochGuard = `RUN [ -n "$CHECK_EPOCH" ] || exit 1`
// freshEpoch is the epoch computation the calling scripts must use, as
// a bare assignment on its own line. Inline in an argument, a failing
// `date` would not abort under `set -eu`; CHECK_EPOCH would become the
// empty string, and the guard above would be the only thing standing
// between that and a permanently cached green. `$$` is required because
// `date +%s` is second-granular and busybox silently drops `%N`, so
// without the pid two concurrent runs in one second can collide.
const freshEpoch = `epoch="$(date +%s%N)$$"`
// TestLintDockerfilePinsTheLinterByDigest fails if the lint image stops
// being pinned. An unpinned tag makes the gate's verdict depend on
// whatever the registry currently serves under that name.
func TestLintDockerfilePinsTheLinterByDigest(t *testing.T) {
t.Parallel()
from := ""
for _, instruction := range instructions(t, lintDockerfile) {
if strings.HasPrefix(instruction, "FROM ") {
from = instruction
break
}
}
require.NotEmpty(t, from, "%s declares no FROM", lintDockerfile)
assert.Contains(t, from, "golangci/golangci-lint",
"the lint image must be the golangci-lint image")
assert.Contains(t, from, "@sha256:",
"the lint image must be pinned by digest, not by tag alone")
}
// TestLintDockerfileCannotBeCachedGreen pins the whole cache-busting
// mechanism in the file that lints: the declaration with no default,
// the non-empty guard, and the value expanded into the lint command
// itself rather than merely declared.
func TestLintDockerfileCannotBeCachedGreen(t *testing.T) {
t.Parallel()
found := instructions(t, lintDockerfile)
argAt := indexOf(found, checkEpochARG)
require.GreaterOrEqual(t, argAt, 0,
"%s must declare `%s` with no default value",
lintDockerfile, checkEpochARG)
assert.GreaterOrEqual(t, indexOf(found, checkEpochGuard), argAt,
"%s must guard against an empty CHECK_EPOCH with `%s`",
lintDockerfile, checkEpochGuard)
assertEpochExpandedInto(t, found[argAt:], "golangci-lint run")
// Dependency layers must stay above the ARG, or every lint run
// re-downloads the module cache and the inner loop becomes
// unusable.
download := indexOf(found, "RUN go mod download")
require.GreaterOrEqual(t, download, 0,
"%s must download modules in their own layer", lintDockerfile)
assert.Less(t, download, argAt,
"`%s` must come after `go mod download` so dependency layers"+
" still cache", checkEpochARG)
}
// TestLintDockerfileVerifiesTheLinterConfig guards the validation of
// .golangci.yml itself. `golangci-lint run` rejects a config it cannot
// parse but silently IGNORES an unknown top-level key, so renaming
// `linters:` to `linterz:` discards `default: all` and every threshold
// and still exits 0 reporting no issues. `config verify` is what turns
// that into a failure, and it has to run BEFORE the lint, or the lint
// spends a minute reporting a verdict from a config already known to be
// wrong.
func TestLintDockerfileVerifiesTheLinterConfig(t *testing.T) {
t.Parallel()
found := instructions(t, lintDockerfile)
verify := linterBinary + " config verify"
verifyAt := indexContaining(found, verify)
require.GreaterOrEqual(t, verifyAt, 0,
"%s must run `%s --config .golangci.yml`: without it a typo'd"+
" top-level key in .golangci.yml is silently ignored and the"+
" gate passes with only the default linter set", lintDockerfile,
verify)
runAt := indexContaining(found, linterBinary+" run")
require.GreaterOrEqual(t, runAt, 0, "%s must lint", lintDockerfile)
assert.Less(t, verifyAt, runAt,
"%s must verify the config before linting with it", lintDockerfile)
// Keyed on the epoch like every other check layer, so it executes
// per invocation rather than being replayed. A cached validation
// validates nothing.
assertEpochExpandedInto(t, found, verify)
}
// TestProductDockerfileCannotBeCachedGreen holds the same line for the
// checks that remain in the product image build.
func TestProductDockerfileCannotBeCachedGreen(t *testing.T) {
t.Parallel()
found := instructions(t, productDockerfile)
argAt := indexOf(found, checkEpochARG)
require.GreaterOrEqual(t, argAt, 0,
"%s must declare `%s` with no default value",
productDockerfile, checkEpochARG)
assert.GreaterOrEqual(t, indexOf(found, checkEpochGuard), argAt,
"%s must guard against an empty CHECK_EPOCH", productDockerfile)
assertEpochExpandedInto(t, found[argAt:], "make fmt-check")
assertEpochExpandedInto(t, found[argAt:], "make test")
}
// TestProductDockerfileDoesNotLint records the split deliberately: the
// linter lives in Dockerfile.lint and nowhere else, so there is exactly
// one digest pinning it. A lint stage reintroduced here would either be
// docker-in-docker (`make lint` is now `docker build`) or a second,
// independently bumpable pin.
func TestProductDockerfileDoesNotLint(t *testing.T) {
t.Parallel()
contents := readRepoFile(t, productDockerfile)
for _, forbidden := range []string{"golangci", "make lint"} {
assert.NotContains(t, instructionText(contents), forbidden,
"%s must not lint: the linter is pinned once, in %s",
productDockerfile, lintDockerfile)
}
}
// TestLintScriptBuildsTheLintDockerfileWithAFreshEpoch is the other
// half of the mechanism. The Dockerfile's guard only rejects an EMPTY
// epoch; a constant non-empty one would satisfy it and still be served
// from cache forever.
func TestLintScriptBuildsTheLintDockerfileWithAFreshEpoch(t *testing.T) {
t.Parallel()
script := readRepoFile(t, lintScript)
assertBareEpochAssignment(t, script, lintScript)
assert.Contains(t, script, `--build-arg CHECK_EPOCH="$epoch"`,
"%s must pass the fresh epoch to the build", lintScript)
assert.Contains(t, script, lintDockerfile,
"%s must build %s", lintScript, lintDockerfile)
}
// TestCibuildBuildsBothDockerfilesWithFreshEpochs guards the CI gate:
// dropping either build silently removes a whole class of check from
// CI while leaving it green.
func TestCibuildBuildsBothDockerfilesWithFreshEpochs(t *testing.T) {
t.Parallel()
script := readRepoFile(t, cibuildScript)
assertBareEpochAssignment(t, script, cibuildScript)
assert.Equal(t, 2, strings.Count(script, freshEpoch),
"%s must compute a fresh epoch for each of its two builds",
cibuildScript)
assert.Equal(t, 2,
strings.Count(script, `--build-arg CHECK_EPOCH="$epoch"`),
"%s must pass a fresh epoch to both builds", cibuildScript)
assert.Contains(t, script, "-f Dockerfile.lint",
"%s must build %s", cibuildScript, lintDockerfile)
}
// assertEpochExpandedInto fails unless some instruction runs the named
// command with the epoch expanded into it. Expansion, not mere
// declaration: an ARG that no instruction references is not guaranteed
// to key the layer, and the expansion also puts the value in the build
// log where a reader can see the layer was keyed fresh.
func assertEpochExpandedInto(t *testing.T, found []string, command string) {
t.Helper()
for _, instruction := range found {
if !strings.HasPrefix(instruction, "RUN ") {
continue
}
if strings.Contains(instruction, command) &&
strings.Contains(instruction, "${CHECK_EPOCH}") {
return
}
}
assert.Fail(t, "no epoch-keyed layer runs the command",
"`%s` must run in a layer that expands ${CHECK_EPOCH}, or it"+
" will be replayed from cache without executing", command)
}
// assertBareEpochAssignment fails unless the script computes the epoch
// as a bare assignment on its own line.
func assertBareEpochAssignment(t *testing.T, script, name string) {
t.Helper()
for line := range strings.SplitSeq(script, "\n") {
if strings.TrimSpace(line) == freshEpoch {
return
}
}
assert.Fail(t, "no bare epoch assignment",
"%s must compute `%s` as a bare assignment on its own line, so"+
" `set -e` catches a failing date instead of quietly"+
" building with an empty epoch", name, freshEpoch)
}
// instructions returns the Dockerfile's instructions, one per element,
// with comments and blank lines dropped and continuation lines joined,
// so a multi-line RUN is one string.
func instructions(t *testing.T, name string) []string {
t.Helper()
return strings.Split(instructionText(readRepoFile(t, name)), "\n")
}
// instructionText is instructions' parse, before splitting: it is also
// what a "must not contain" assertion should look at, so that a word
// appearing only in a comment is not mistaken for behaviour.
func instructionText(contents string) string {
var (
out []string
continued string
isContinued bool
)
for line := range strings.SplitSeq(contents, "\n") {
trimmed := strings.TrimSpace(line)
if !isContinued && (trimmed == "" || strings.HasPrefix(trimmed, "#")) {
continue
}
isContinued = strings.HasSuffix(trimmed, `\`)
continued += strings.TrimSuffix(trimmed, `\`)
if isContinued {
continue
}
out = append(out, strings.Join(strings.Fields(continued), " "))
continued = ""
}
return strings.Join(out, "\n")
}
// indexOf returns the position of the first instruction equal to, or
// beginning with, want; -1 if there is none.
func indexOf(found []string, want string) int {
for i, instruction := range found {
if instruction == want || strings.HasPrefix(instruction, want+" ") {
return i
}
}
return -1
}
// indexContaining returns the position of the first instruction
// containing want; -1 if there is none.
func indexContaining(found []string, want string) int {
for i, instruction := range found {
if strings.Contains(instruction, want) {
return i
}
}
return -1
}
// readRepoFile reads a file by its path relative to the repository
// root.
func readRepoFile(t *testing.T, name string) string {
t.Helper()
//nolint:gosec // G304: the path is a constant relative to this repo
contents, err := os.ReadFile(filepath.Join(repoRoot(t), name))
require.NoError(t, err)
return string(contents)
}
// repoRoot returns the repository root. The test binary runs with its
// package directory as the working directory, so the root is found by
// walking up until the module file appears.
func repoRoot(t *testing.T) string {
t.Helper()
dir, err := os.Getwd()
require.NoError(t, err)
for {
_, err = os.Stat(filepath.Join(dir, "go.mod"))
if err == nil {
return dir
}
parent := filepath.Dir(dir)
require.NotEqual(t, dir, parent,
"walked to the filesystem root without finding a go.mod")
dir = parent
}
}
+2 -20
View File
@@ -1,8 +1,6 @@
package main_test
import (
"os"
"path/filepath"
"regexp"
"slices"
"strings"
@@ -87,27 +85,11 @@ func TestBuildTargetBuildsTheBinary(t *testing.T) {
}
// readMakefile returns the contents of the repository's Makefile. The
// test binary runs with its package directory as the working directory,
// so the root is found by walking up until the Makefile appears.
// root is located by the shared walk in lintdocker_test.go.
func readMakefile(t *testing.T) string {
t.Helper()
dir, err := os.Getwd()
require.NoError(t, err)
for {
//nolint:gosec // G304: the path is this test's own directory walk
contents, err := os.ReadFile(filepath.Join(dir, "Makefile"))
if err == nil {
return string(contents)
}
parent := filepath.Dir(dir)
require.NotEqual(t, dir, parent,
"walked to the filesystem root without finding a Makefile")
dir = parent
}
return readRepoFile(t, "Makefile")
}
// phonyTargets returns every name declared phony, across all .PHONY
+29 -8
View File
@@ -5,11 +5,30 @@
Vaultik uses a local SQLite database to track file metadata, chunk mappings, and blob associations during the backup process. This database serves as an index for incremental backups and enables efficient deduplication.
**Important Notes:**
- **No Migration Support (pre-1.0)**: Vaultik does not support database schema
migrations. The local index is treated as disposable — if the schema changes,
delete the local SQLite database (`vaultik database delete`) and run a full
backup. The remote storage is unaffected; the new index will re-deduplicate
against existing remote blobs.
This section is the authoritative explanation of the schema/migration story;
other documents (the README and `AGENTS.md`) link here.
- **No upgrade path between versions (pre-1.0)**: Vaultik has no supported way to
carry an existing local index across a schema change. The index is disposable
— if the on-disk schema changes between versions, delete the local SQLite
database (`vaultik database delete`) and run a full backup. Remote storage is
unaffected; the new index re-deduplicates against existing remote blobs. This
is the standing project policy, and it is separate from the schema bootstrap
described next.
- **Schema bootstrap**: a fresh database is populated from numbered SQL files
embedded in the binary under `internal/database/schema/`. `000.sql` creates the
`schema_migrations` table; `001.sql` creates the application tables. On opening
a database the code applies each numbered file that has not yet run and records
its version in `schema_migrations`. This bootstraps a new database; it does not
upgrade an existing one between released versions.
- **Changing the schema (pre-1.0)**: edit `internal/database/schema/001.sql` (and
the code that touches the affected tables) directly. Do not add new numbered
files — there is no installed base to migrate.
- **Disposability expires at 1.0**: the index is treated as disposable only until
1.0 ships and is tagged. Once 1.0 is tagged that clause expires and the
question of upgrading existing indexes returns. It is deliberately left open
here.
- **Version Compatibility**: In rare cases, you may need to use the same version
of Vaultik to restore a backup as was used to create it. This ensures
compatibility with the metadata format stored in S3.
@@ -192,10 +211,12 @@ Tracks blob upload metrics.
After a snapshot is completed:
1. Copy database to temporary file
2. Clean temporary database to contain only current snapshot data
3. Export to SQL dump using sqlite3
3. VACUUM the trimmed database so deleted rows leave no pages behind
4. Compress with zstd and encrypt with age
5. Upload to S3 as `metadata/{snapshot-id}/db.zst.age`
6. Generate blob manifest and upload as `metadata/{snapshot-id}/manifest.json.zst`
5. Upload to S3 as `metadata/{remote-key}/db.zst.age`
6. Generate blob manifest and upload as `metadata/{remote-key}/manifest.json.zst`
The `{remote-key}` directory name is a one-way hash of the human snapshot ID, so the ID is never written to the store in plaintext; see [REPOSTRUCTURE.md](REPOSTRUCTURE.md#remote-key-derivation).
### 4. Restore Process
+37 -17
View File
@@ -17,11 +17,13 @@ Vaultik stores all backup data in an S3-compatible object store. The repository
│ └── <hash[2:4]>/
│ └── <full-hash>
└── metadata/
└── <snapshot-id>/
└── <remote-key>/
├── db.zst.age
└── manifest.json.zst
```
The metadata subdirectory is named with the **remote key**, a one-way hash of the snapshot ID, not with the human-readable snapshot ID itself. See [Remote Key Derivation](#remote-key-derivation).
## Blobs Directory (`blobs/`)
### Structure
@@ -40,9 +42,11 @@ Blobs contain the actual file data from backups and must be encrypted for securi
## Metadata Directory (`metadata/`)
Each snapshot has its own subdirectory named with the snapshot ID.
Each snapshot has its own subdirectory. The directory is **not** named with the human-readable snapshot ID; it is named with the remote key — a one-way hash of that ID. The human ID is never written to the destination store as a directory name (see [Remote Key Derivation](#remote-key-derivation)).
### Snapshot ID Format
The human-readable snapshot ID is used in CLI arguments, log lines, and the local database. It is not written to the destination store.
- **Format**: `<hostname>_<snapshot-name>_<RFC3339>` (or `<hostname>_<RFC3339>` if no
name was specified)
- **Example**: `laptop_home_2024-01-15T14:30:52Z`
@@ -51,6 +55,19 @@ Each snapshot has its own subdirectory named with the snapshot ID.
- Snapshot name from the configured `snapshots:` map (optional)
- RFC3339 UTC timestamp
This ID reveals the hostname, the configured snapshot name, and the backup time, so it is never used as the on-disk directory name — the remote key is used instead.
### Remote Key Derivation
The remote key is `hex(SHA256(SHA256("vaultik|" + snapshot-id)))`: a double SHA-256 over the snapshot ID, with a `vaultik|` domain-separation prefix. The result is a 64-character hex string with no structure a remote observer can reverse. Implemented in `internal/snapshot/remotekey.go`.
Worked example:
- Snapshot ID: `server1_home_2025-06-01T12:00:00Z`
- Remote key: `17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa`
- Directory: `metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`
Because the hash is one-way, a listing of the destination store reveals neither the hostname nor the snapshot name of any backup. The same remote key is stored in the manifest's `snapshot_id` field.
### Files in Each Snapshot Directory
#### `db.zst.age` - Encrypted Database
@@ -68,16 +85,17 @@ Each snapshot has its own subdirectory named with the snapshot ID.
- **Structure**:
```json
{
"snapshot_id": "laptop_home_2024-01-15T14:30:52Z",
"timestamp": "2024-01-15T14:30:52Z",
"snapshot_id": "17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa",
"timestamp": "2025-06-01T12:00:00Z",
"blob_count": 42,
"total_compressed_size": 1048576,
"blobs": [
"cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678",
"deadbeef1234567890abcdef1234567890abcdef1234567890abcdef12345678",
...
{ "hash": "cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678", "compressed_size": 24576 },
{ "hash": "deadbeef1234567890abcdef1234567890abcdef1234567890abcdef12345678", "compressed_size": 32768 }
]
}
```
`snapshot_id` is the remote key (a hash), not the human ID; `timestamp` is written in the clear.
### Why Manifest is Unencrypted
The manifest must be readable without the private key to enable:
@@ -86,7 +104,7 @@ The manifest must be readable without the private key to enable:
3. **Verification** - Checking blob existence without decryption
4. **Cross-snapshot deduplication analysis** - Finding shared blobs between snapshots
The manifest only contains blob hashes, not file names or any other sensitive information.
The manifest contains the remote key, the backup timestamp, the blob count and total compressed size, and each blob's hash and compressed size. It contains no file names, paths, or other decrypted metadata.
## Security Considerations
@@ -96,19 +114,21 @@ The manifest only contains blob hashes, not file names or any other sensitive in
- **File-to-chunk mappings** (in db.zst.age)
### What's Not Encrypted
- **Blob hashes** (in manifest.json.zst)
- **Snapshot IDs** (directory names)
- **Blob count per snapshot** (in manifest.json.zst)
- **The remote key** — directory names and the manifest `snapshot_id`, a one-way hash of the snapshot ID (see [Remote Key Derivation](#remote-key-derivation))
- **The backup timestamp** (in manifest.json.zst)
- **Blob hashes and their compressed sizes** (in manifest.json.zst)
- **Blob count and total compressed size per snapshot** (in manifest.json.zst)
### Privacy Implications
From the unencrypted data, an observer can determine:
- When backups were taken (from snapshot IDs)
- Which hostname created backups (from snapshot IDs)
- How many blobs each snapshot references
- Which blobs are shared between snapshots (deduplication patterns)
- The size of each encrypted blob
From the unencrypted data, an observer of the destination store can determine:
- **When each backup was taken** — not from the directory name, which is a one-way hash, but from the plaintext `timestamp` field in manifest.json.zst, which is published in the clear
- How many blobs each snapshot references, and the total compressed size
- The compressed size of each blob, and which blobs are shared between snapshots (deduplication patterns)
Together these give an observer a timing-and-size profile of every snapshot. This is an accepted, documented property of the format, not a defect: the manifest is unencrypted so that pruning can run without the private key, and the timing channel could not be closed by encrypting it anyway — object creation times and per-object sizes stay visible at the storage layer on both `s3://` and `file://` destinations regardless.
An observer cannot determine:
- The hostname or snapshot name of any backup (the directory name and the manifest `snapshot_id` are one-way hashes of the human ID)
- File names or paths
- File contents
- File permissions or ownership
+30 -1
View File
@@ -1,6 +1,7 @@
package cli
import (
"bytes"
"errors"
"fmt"
"os"
@@ -24,6 +25,11 @@ const configSetArgs = 2
// parent config dirs (e.g. ~/.config) are conventionally traversable.
const configDirMode = 0o755
// configYAMLIndent matches the 2-space indentation of defaultConfigTemplate,
// so `config set` writes the file back with the same indentation rather than
// yaml.Marshal's 4-space default.
const configYAMLIndent = 2
var (
errConfigExists = errors.New("config file already exists")
errEmptyConfig = errors.New("empty config file")
@@ -381,7 +387,7 @@ Examples:
return err
}
out, err := yaml.Marshal(root)
out, err := marshalConfigYAML(root)
if err != nil {
return fmt.Errorf("marshaling config: %w", err)
}
@@ -405,6 +411,29 @@ Examples:
}
}
// marshalConfigYAML renders a config document tree with 2-space indentation,
// matching defaultConfigTemplate. yaml.Marshal defaults to 4 spaces, which
// would reindent the whole file on the first `config set` despite the promise
// to preserve formatting.
func marshalConfigYAML(root *yaml.Node) ([]byte, error) {
var buf bytes.Buffer
enc := yaml.NewEncoder(&buf)
enc.SetIndent(configYAMLIndent)
err := enc.Encode(root)
if err != nil {
return nil, err
}
err = enc.Close()
if err != nil {
return nil, err
}
return buf.Bytes(), nil
}
// loadYAMLFile parses a YAML file into a yaml.Node document tree,
// which preserves comments and ordering for round-tripping.
func loadYAMLFile(path string) (*yaml.Node, error) {
+41
View File
@@ -188,6 +188,47 @@ func TestYAMLPathSet(t *testing.T) {
}
}
// TestConfigSetPreservesFormatting asserts the `config set` write path
// (marshalConfigYAML) round-trips a 2-space-indented file without reindenting
// it to yaml.Marshal's 4-space default, and keeps comments.
func TestConfigSetPreservesFormatting(t *testing.T) {
t.Parallel()
root := parseTestYAML(t)
err := yamlPathSet(root, splitPath("s3.bucket"), "newbucket")
if err != nil {
t.Fatalf("set s3.bucket: %v", err)
}
out, err := marshalConfigYAML(root)
if err != nil {
t.Fatalf("marshal: %v", err)
}
text := string(out)
for _, want := range []string{"# top comment", "# inline comment"} {
if !contains(text, want) {
t.Errorf("round-tripped YAML dropped comment %q:\n%s", want, text)
}
}
// Nested map keys stay at 2-space indent; the bug reindented them to 4.
if !contains(text, "\n bucket: newbucket") {
t.Errorf("expected 2-space indent for s3.bucket, got:\n%s", text)
}
if contains(text, "\n bucket:") {
t.Errorf("s3.bucket reindented to 4 spaces:\n%s", text)
}
// Sequence items under a key also stay at 2 spaces.
if !contains(text, "\n - age1aaa") {
t.Errorf("expected 2-space indent for sequence item, got:\n%s", text)
}
}
func splitPath(s string) []string {
return strings.Split(s, ".")
}
-126
View File
@@ -1,126 +0,0 @@
package cli
import (
"errors"
"fmt"
"regexp"
"strconv"
"strings"
"time"
)
// Approximate lengths of the extended calendar units accepted by
// parseDuration.
const (
durationDay = 24 * time.Hour
durationWeek = 7 * durationDay
durationMonth = 30 * durationDay
durationYear = 365 * durationDay
)
var (
errNegativeDuration = errors.New("negative durations are not supported")
errInvalidDuration = errors.New("invalid duration format")
errUnknownTimeUnit = errors.New("unknown time unit")
)
// parseDuration parses duration strings. Supports standard Go duration format
// (e.g., "3h30m", "1h45m30s") as well as extended units:
// - d: days (e.g., "30d", "7d")
// - w: weeks (e.g., "2w", "4w")
// - mo: months (30 days) (e.g., "6mo", "1mo")
// - y: years (365 days) (e.g., "1y", "2y")
//
// Can combine units: "1y6mo", "2w3d", "1d12h30m"
func parseDuration(s string) (time.Duration, error) {
// First try standard Go duration parsing
d, err := time.ParseDuration(s)
if err == nil {
return d, nil
}
// Extended duration parsing
// Check for negative values
if strings.HasPrefix(strings.TrimSpace(s), "-") {
return 0, errNegativeDuration
}
// Pattern matches: number + unit, repeated
re := regexp.MustCompile(`(\d+(?:\.\d+)?)\s*([a-zA-Z]+)`)
matches := re.FindAllStringSubmatch(s, -1)
if len(matches) == 0 {
return 0, fmt.Errorf("%w: %q", errInvalidDuration, s)
}
var total time.Duration
for _, match := range matches {
valueStr := match[1]
unit := strings.ToLower(match[2])
value, err := strconv.ParseFloat(valueStr, 64)
if err != nil {
return 0, fmt.Errorf("invalid number %q: %w", valueStr, err)
}
d, err := durationForUnit(value, unit)
if err != nil {
return 0, err
}
total += d
}
return total, nil
}
// durationForUnit converts a value with a (case-normalized) unit suffix
// into a time.Duration, accepting Go's standard units plus the extended
// calendar units.
func durationForUnit(value float64, unit string) (time.Duration, error) {
switch unit {
// Standard time units
case "ns", "nanosecond", "nanoseconds":
return time.Duration(value), nil
case "us", "µs", "microsecond", "microseconds":
return time.Duration(value * float64(time.Microsecond)), nil
case "ms", "millisecond", "milliseconds":
return time.Duration(value * float64(time.Millisecond)), nil
case "s", "sec", "second", "seconds":
return time.Duration(value * float64(time.Second)), nil
case "m", "min", "minute", "minutes":
return time.Duration(value * float64(time.Minute)), nil
case "h", "hr", "hour", "hours":
return time.Duration(value * float64(time.Hour)), nil
// Extended units
case "d", "day", "days":
return time.Duration(value * float64(durationDay)), nil
case "w", "week", "weeks":
return time.Duration(value * float64(durationWeek)), nil
case "mo", "month", "months":
// Using 30 days as approximation
return time.Duration(value * float64(durationMonth)), nil
case "y", "year", "years":
// Using 365 days as approximation
return time.Duration(value * float64(durationYear)), nil
default:
// Try parsing as standard Go duration unit
testStr := "1" + unit
_, err := time.ParseDuration(testStr)
if err != nil {
return 0, fmt.Errorf("%w: %q", errUnknownTimeUnit, unit)
}
// It's a valid Go duration unit, parse the full value
fullStr := fmt.Sprintf("%g%s", value, unit)
d, err := time.ParseDuration(fullStr)
if err != nil {
return 0, fmt.Errorf("invalid duration %q: %w", fullStr, err)
}
return d, nil
}
}
-299
View File
@@ -1,299 +0,0 @@
package cli //nolint:testpackage // needs access to unexported parseDuration
import (
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
type parseDurationCase struct {
name string
input string
expected time.Duration
wantErr bool
}
// runParseDurationCases executes a table of parseDuration cases as
// parallel subtests.
func runParseDurationCases(t *testing.T, tests []parseDurationCase) {
t.Helper()
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
got, err := parseDuration(tt.input)
if tt.wantErr {
require.Error(t, err, "expected error for input %q", tt.input)
return
}
require.NoError(t, err, "unexpected error for input %q", tt.input)
assert.Equal(t, tt.expected, got, "duration mismatch for input %q", tt.input)
})
}
}
func TestParseDurationStandard(t *testing.T) {
t.Parallel()
runParseDurationCases(t, []parseDurationCase{
{
name: "standard seconds",
input: "30s",
expected: 30 * time.Second,
},
{
name: "standard minutes",
input: "45m",
expected: 45 * time.Minute,
},
{
name: "standard hours",
input: "2h",
expected: 2 * time.Hour,
},
{
name: "standard combined",
input: "3h30m",
expected: 3*time.Hour + 30*time.Minute,
},
{
name: "standard complex",
input: "1h45m30s",
expected: 1*time.Hour + 45*time.Minute + 30*time.Second,
},
{
name: "standard with milliseconds",
input: "1s500ms",
expected: 1*time.Second + 500*time.Millisecond,
},
})
}
func TestParseDurationExtendedUnits(t *testing.T) {
t.Parallel()
runParseDurationCases(t, []parseDurationCase{
// Extended units - days
{
name: "single day",
input: "1d",
expected: 24 * time.Hour,
},
{
name: "multiple days",
input: "7d",
expected: 7 * 24 * time.Hour,
},
{
name: "fractional days",
input: "1.5d",
expected: 36 * time.Hour,
},
{
name: "days spelled out",
input: "3days",
expected: 3 * 24 * time.Hour,
},
// Extended units - weeks
{
name: "single week",
input: "1w",
expected: 7 * 24 * time.Hour,
},
{
name: "multiple weeks",
input: "4w",
expected: 4 * 7 * 24 * time.Hour,
},
{
name: "weeks spelled out",
input: "2weeks",
expected: 2 * 7 * 24 * time.Hour,
},
// Extended units - months
{
name: "single month",
input: "1mo",
expected: 30 * 24 * time.Hour,
},
{
name: "multiple months",
input: "6mo",
expected: 6 * 30 * 24 * time.Hour,
},
{
name: "months spelled out",
input: "3months",
expected: 3 * 30 * 24 * time.Hour,
},
// Extended units - years
{
name: "single year",
input: "1y",
expected: 365 * 24 * time.Hour,
},
{
name: "multiple years",
input: "2y",
expected: 2 * 365 * 24 * time.Hour,
},
{
name: "years spelled out",
input: "1year",
expected: 365 * 24 * time.Hour,
},
})
}
func TestParseDurationCombinedAndErrors(t *testing.T) {
t.Parallel()
runParseDurationCases(t, []parseDurationCase{
// Combined extended units
{
name: "weeks and days",
input: "2w3d",
expected: 2*7*24*time.Hour + 3*24*time.Hour,
},
{
name: "years and months",
input: "1y6mo",
expected: 365*24*time.Hour + 6*30*24*time.Hour,
},
{
name: "days and hours",
input: "1d12h",
expected: 24*time.Hour + 12*time.Hour,
},
{
name: "complex combination",
input: "1y2mo3w4d5h6m7s",
expected: 365*24*time.Hour + 2*30*24*time.Hour +
3*7*24*time.Hour + 4*24*time.Hour +
5*time.Hour + 6*time.Minute + 7*time.Second,
},
{
name: "with spaces",
input: "1d 12h 30m",
expected: 24*time.Hour + 12*time.Hour + 30*time.Minute,
},
// Edge cases
{
name: "zero duration",
input: "0s",
expected: 0,
},
{
name: "large duration",
input: "10y",
expected: 10 * 365 * 24 * time.Hour,
},
// Error cases
{
name: "empty string",
input: "",
wantErr: true,
},
{
name: "invalid format",
input: "abc",
wantErr: true,
},
{
name: "unknown unit",
input: "5x",
wantErr: true,
},
{
name: "invalid number",
input: "xyzd",
wantErr: true,
},
{
name: "negative not supported",
input: "-5d",
wantErr: true,
},
})
}
func TestParseDurationSpecialCases(t *testing.T) {
t.Parallel()
// Test that standard Go durations work exactly as expected
standardDurations := []string{
"300ms",
"1.5h",
"2h45m",
"72h",
"1us",
"1µs",
"1ns",
}
for _, d := range standardDurations {
expected, err := time.ParseDuration(d)
require.NoError(t, err)
got, err := parseDuration(d)
require.NoError(t, err)
assert.Equal(t, expected, got, "standard duration %q should parse identically", d)
}
}
func TestParseDurationRealWorldExamples(t *testing.T) {
t.Parallel()
// Test real-world snapshot purge scenarios
tests := []struct {
description string
input string
olderThan time.Duration
}{
{
description: "keep snapshots from last 30 days",
input: "30d",
olderThan: 30 * 24 * time.Hour,
},
{
description: "keep snapshots from last 6 months",
input: "6mo",
olderThan: 6 * 30 * 24 * time.Hour,
},
{
description: "keep snapshots from last year",
input: "1y",
olderThan: 365 * 24 * time.Hour,
},
{
description: "keep snapshots from last week and a half",
input: "1w3d",
olderThan: 10 * 24 * time.Hour,
},
{
description: "keep snapshots from last 90 days",
input: "90d",
olderThan: 90 * 24 * time.Hour,
},
}
for _, tt := range tests {
t.Run(tt.description, func(t *testing.T) {
t.Parallel()
got, err := parseDuration(tt.input)
require.NoError(t, err)
assert.Equal(t, tt.olderThan, got)
// Verify the duration makes sense for snapshot purging
assert.Greater(t, got, time.Hour,
"snapshot purge duration should be at least an hour")
})
}
}
+9 -4
View File
@@ -135,13 +135,14 @@ specifying a path using --config or by setting VAULTIK_CONFIG to a path.`,
}
cmd.Flags().BoolVar(&opts.Cron, "cron", false,
"Run in cron mode (silent unless error)")
"Run in cron mode (silent unless warning or error)")
cmd.Flags().BoolVar(&opts.Prune, "prune", false,
"After backup, drop older snapshots of the same name and remove "+
"orphaned blobs")
cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "",
"With --prune: keep snapshots newer than this duration "+
"(e.g. 4w, 30d, 6mo) instead of only the latest")
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+
"instead of only the latest")
return cmd
}
@@ -204,7 +205,8 @@ restrict the operation to specific snapshot names.`,
cmd.Flags().BoolVar(&opts.KeepLatest, "keep-latest", false,
"Keep only the latest snapshot of each name")
cmd.Flags().StringVar(&opts.OlderThan, "older-than", "",
"Remove snapshots older than duration (e.g., 30d, 6m, 1y)")
"Remove snapshots older than duration "+
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months)")
cmd.Flags().BoolVar(&opts.Force, "force", false, "Skip confirmation prompt")
cmd.Flags().StringArrayVar(&opts.Names, "snapshot", nil,
"Restrict to snapshots with these names (repeat for multiple)")
@@ -219,7 +221,10 @@ func newSnapshotVerifyCommand() *cobra.Command {
cmd := &cobra.Command{
Use: "verify <snapshot-id>",
Short: "Verify snapshot integrity",
Long: "Verifies that all blobs referenced in a snapshot exist",
Long: "Verifies that all blobs referenced in a snapshot exist.\n\n" +
"The snapshot may be named by its ID or, on a host with no local\n" +
"index, by the remote key that 'snapshot list' prints for a\n" +
"remote-only snapshot (an unambiguous leading part is enough).",
Args: requireSnapshotIDArg,
RunE: func(cmd *cobra.Command, args []string) error {
snapshotID := args[0]
+4
View File
@@ -48,6 +48,10 @@ target directory.
If no paths are specified, all files are restored.
If paths are specified, only matching files/directories are restored.
The snapshot may be named by its ID or, when restoring on a host with no
local index, by the remote key that 'snapshot list' prints for a
remote-only snapshot (an unambiguous leading part is enough).
Requires the VAULTIK_AGE_SECRET_KEY environment variable to be set with
the age private key.
-12
View File
@@ -1,12 +0,0 @@
package cli
import "time"
// SnapshotInfo represents snapshot information for listing
//
//nolint:tagliatelle // snake_case is the established output format
type SnapshotInfo struct {
ID string `json:"id"`
Timestamp time.Time `json:"timestamp"`
CompressedSize int64 `json:"compressed_size"`
}
-67
View File
@@ -1,67 +0,0 @@
// Package models defines shared value types describing files, chunks,
// blobs, and snapshots as they move through the backup pipeline.
package models
import (
"time"
)
// FileInfo represents a file in the backup system
type FileInfo struct {
Path string
MTime time.Time
Size int64
}
// ChunkInfo represents a content-addressed chunk
type ChunkInfo struct {
Hash string // SHA256 hash
Size int64
Offset int64 // Offset within source file
}
// ChunkRef represents a reference to a chunk in a blob or file
type ChunkRef struct {
ChunkHash string
Offset int64
Length int64
}
// BlobInfo represents an encrypted blob containing multiple chunks
type BlobInfo struct {
Hash string // SHA256 hash of the blob content (content-addressable)
CreatedAt time.Time
Size int64
ChunkCount int
}
// Snapshot represents a backup snapshot
type Snapshot struct {
ID string // ISO8601 timestamp
Hostname string
Version string
CreatedAt time.Time
FileCount int64
ChunkCount int64
BlobCount int64
TotalSize int64
MetadataSize int64
}
// SnapshotMetadata contains the full metadata for a snapshot
type SnapshotMetadata struct {
Snapshot *Snapshot
Files map[string]*FileInfo
Chunks map[string]*ChunkInfo
Blobs map[string]*BlobInfo
FileChunks map[string][]*ChunkRef // path -> chunks
BlobChunks map[string][]*ChunkRef // blob hash -> chunks
}
// Chunk represents a data chunk for processing
type Chunk struct {
Data []byte
Hash string
Offset int64
Length int64
}
-58
View File
@@ -1,58 +0,0 @@
package models_test
import (
"testing"
"time"
"sneak.berlin/go/vaultik/internal/models"
)
// TestModelsCompilation ensures all model types can be instantiated
func TestModelsCompilation(t *testing.T) {
t.Parallel()
// This test primarily serves as a compilation test
// to ensure all types are properly defined
// Test FileInfo
fi := &models.FileInfo{
Path: "/test/file.txt",
MTime: time.Now(),
Size: 1024,
}
if fi.Path != "/test/file.txt" {
t.Errorf("FileInfo.Path not set correctly")
}
// Test ChunkInfo
ci := &models.ChunkInfo{
Hash: "abc123",
Size: 512,
Offset: 0,
}
if ci.Hash != "abc123" {
t.Errorf("ChunkInfo.Hash not set correctly")
}
// Test BlobInfo
bi := &models.BlobInfo{
Hash: "blob123",
CreatedAt: time.Now(),
Size: 1024,
ChunkCount: 2,
}
if bi.Hash != "blob123" {
t.Errorf("BlobInfo.Hash not set correctly")
}
// Test Snapshot
s := &models.Snapshot{
ID: "2024-01-01T00:00:00Z",
Hostname: "test-host",
Version: "1.0.0",
CreatedAt: time.Now(),
}
if s.ID != "2024-01-01T00:00:00Z" {
t.Errorf("Snapshot.ID not set correctly")
}
}
+13 -5
View File
@@ -219,11 +219,7 @@ func (c *Client) HeadObject(ctx context.Context, key string) (bool, error) {
Key: aws.String(fullKey),
})
if err != nil {
var (
notFound *s3types.NotFound
noSuchKey *s3types.NoSuchKey
)
if errors.As(err, &notFound) || errors.As(err, &noSuchKey) {
if IsNotFound(err) {
return false, nil
}
@@ -233,6 +229,18 @@ func (c *Client) HeadObject(ctx context.Context, key string) (bool, error) {
return true, nil
}
// IsNotFound reports whether err indicates that an object does not exist.
// Head and Get requests surface a missing object as different SDK types,
// so both are checked here.
func IsNotFound(err error) bool {
var (
notFound *s3types.NotFound
noSuchKey *s3types.NoSuchKey
)
return errors.As(err, &notFound) || errors.As(err, &noSuchKey)
}
// ObjectInfo contains information about an S3 object.
// It is used by ListObjectsStream to return object metadata
// along with any errors encountered during listing.
+3 -2
View File
@@ -22,8 +22,9 @@ const remoteKeyPrefix = "vaultik|"
//
// - the "metadata/<remote-key>/..." subdirectory on the storage
// backend so a directory listing of the bucket / file:// dest
// doesn't reveal hostnames, configured snapshot names, or backup
// timestamps;
// doesn't reveal hostnames or configured snapshot names. (The
// backup time is not hidden: the manifest.json.zst inside that
// directory carries a plaintext RFC3339 timestamp.)
// - the `snapshot_id` field of the unencrypted manifest.json.zst
// for the same reason;
// - any code path that needs to translate a known local snapshot ID
+25 -7
View File
@@ -44,7 +44,6 @@ import (
"errors"
"fmt"
"io"
"os/exec"
"path/filepath"
"strings"
"time"
@@ -669,14 +668,31 @@ func (sm *SnapshotManager) collectCleanupStats(
// vacuumDatabase runs VACUUM on the database to remove deleted data and compact
// This is critical for security - ensures no stale/deleted data pages are uploaded
//
// VACUUM runs through the modernc.org/sqlite driver, on a freshly opened
// connection with no transaction in flight (VACUUM cannot run inside one).
// The database opens in WAL mode, so VACUUM's rewrite lands in the WAL; the
// checkpoint on Close flushes it into the main file, which is the file we
// then compress and upload.
func (sm *SnapshotManager) vacuumDatabase(ctx context.Context, dbPath string) error {
log.Debug("Running VACUUM on database", "path", dbPath)
//nolint:gosec // G204: fixed argv; dbPath is our own temp file path
cmd := exec.CommandContext(ctx, "sqlite3", dbPath, "VACUUM;")
output, err := cmd.CombinedOutput()
db, err := database.New(ctx, dbPath)
if err != nil {
return fmt.Errorf("running VACUUM: %w (output: %s)", err, string(output))
return fmt.Errorf("opening database for VACUUM: %w", err)
}
defer func() {
cerr := db.Close()
if cerr != nil {
log.Debug("Failed to close database after VACUUM",
"path", dbPath, "error", cerr)
}
}()
_, err = db.ExecWithLog(ctx, "VACUUM")
if err != nil {
return fmt.Errorf("running VACUUM: %w", err)
}
return nil
@@ -840,8 +856,10 @@ func (sm *SnapshotManager) generateBlobManifest(
}
// Create manifest. SnapshotID in the unencrypted manifest is the
// double-SHA256 remote key, not the human ID, so the public bytes
// don't reveal hostname/snapshot-name/timestamp metadata.
// double-SHA256 remote key (see RemoteSnapshotKey), not the human ID,
// so neither this field nor the directory name reveals the hostname or
// snapshot name. Timestamp below is written in the clear, so the backup
// time is observable to anyone who can read the manifest.
manifest := &Manifest{
SnapshotID: RemoteSnapshotKey(snapshotID),
Timestamp: time.Now().UTC().Format(time.RFC3339),
+92
View File
@@ -2,6 +2,7 @@
package snapshot
import (
"bytes"
"context"
"database/sql"
"io"
@@ -96,6 +97,97 @@ func verifyCleanedDB(
}
}
// TestVacuumDatabaseRemovesDeletedData proves the export path uploads a
// compacted database: after rows carrying a recognizable marker are deleted
// and vacuumDatabase runs, no page holding that marker survives in the file
// on disk (the file compressFile later reads for upload).
func TestVacuumDatabaseRemovesDeletedData(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
fs := afero.NewOsFs()
tempDir := t.TempDir()
dbPath := filepath.Join(tempDir, "snapshot.db")
db, err := database.New(ctx, dbPath)
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
// A marker distinctive enough that its presence in the raw file can only
// come from the rows inserted below.
marker := []byte("VACUUM_PROBE_DEADBEEF_DELETED_ROW")
payload := bytes.Repeat(marker, 128) // ~4 KiB per row
_, err = db.Conn().ExecContext(ctx,
"CREATE TABLE vacuum_probe (id INTEGER PRIMARY KEY, payload BLOB)")
if err != nil {
t.Fatalf("failed to create probe table: %v", err)
}
for range 512 {
_, err = db.Conn().ExecContext(ctx,
"INSERT INTO vacuum_probe (payload) VALUES (?)", payload)
if err != nil {
t.Fatalf("failed to insert probe row: %v", err)
}
}
_, err = db.Conn().ExecContext(ctx, "DELETE FROM vacuum_probe")
if err != nil {
t.Fatalf("failed to delete probe rows: %v", err)
}
// Close so the deletes reach the main file, mirroring the state
// prepareExportDB hands to vacuumDatabase.
err = db.Close()
if err != nil {
t.Fatalf("failed to close database: %v", err)
}
beforeInfo, err := fs.Stat(dbPath)
if err != nil {
t.Fatalf("failed to stat database before vacuum: %v", err)
}
beforeBytes, err := afero.ReadFile(fs, dbPath)
if err != nil {
t.Fatalf("failed to read database before vacuum: %v", err)
}
if !bytes.Contains(beforeBytes, marker) {
t.Fatalf("expected deleted-row data to linger before vacuum")
}
sm := &SnapshotManager{fs: fs}
err = sm.vacuumDatabase(ctx, dbPath)
if err != nil {
t.Fatalf("vacuumDatabase failed: %v", err)
}
afterBytes, err := afero.ReadFile(fs, dbPath)
if err != nil {
t.Fatalf("failed to read database after vacuum: %v", err)
}
if bytes.Contains(afterBytes, marker) {
t.Fatalf("deleted-row data survived vacuum in the uploaded file")
}
afterInfo, err := fs.Stat(dbPath)
if err != nil {
t.Fatalf("failed to stat database after vacuum: %v", err)
}
if afterInfo.Size() >= beforeInfo.Size() {
t.Fatalf("expected vacuum to shrink the file: before=%d after=%d",
beforeInfo.Size(), afterInfo.Size())
}
}
func TestCleanSnapshotDBEmptySnapshot(t *testing.T) {
// Initialize logger
log.Initialize(log.Config{})
+79 -54
View File
@@ -46,31 +46,18 @@ func (f *FileStorer) SetFilesystem(fs afero.Fs) {
// storage base path.
const storageDirPerm = 0o755
// tempSuffix marks a partially written object. writeAtomic streams into a
// temp file carrying this suffix and only renames it onto the real key once
// the whole object is on disk, so an interrupted write can never leave a
// truncated object at the key a later run would Stat and trust as a complete
// blob. List and ListStream skip these files, so a leftover from an
// interrupted write is never listed or trusted as a blob; it is otherwise
// harmless and is overwritten when the same key is written again.
const tempSuffix = ".partial"
// Put stores data at the specified key.
func (f *FileStorer) Put(_ context.Context, key string, data io.Reader) error {
path := f.fullPath(key)
// Create parent directories
dir := filepath.Dir(path)
err := f.fs.MkdirAll(dir, storageDirPerm)
if err != nil {
return fmt.Errorf("creating directories: %w", err)
}
file, err := f.fs.Create(path)
if err != nil {
return fmt.Errorf("creating file: %w", err)
}
defer func() { _ = file.Close() }()
_, err = io.Copy(file, data)
if err != nil {
return fmt.Errorf("writing file: %w", err)
}
return nil
return f.writeAtomic(key, data, nil)
}
// PutWithProgress stores data with progress reporting.
@@ -78,35 +65,7 @@ func (f *FileStorer) PutWithProgress(
_ context.Context, key string, data io.Reader,
_ int64, progress ProgressCallback,
) error {
path := f.fullPath(key)
// Create parent directories
dir := filepath.Dir(path)
err := f.fs.MkdirAll(dir, storageDirPerm)
if err != nil {
return fmt.Errorf("creating directories: %w", err)
}
file, err := f.fs.Create(path)
if err != nil {
return fmt.Errorf("creating file: %w", err)
}
defer func() { _ = file.Close() }()
// Wrap with progress tracking
pw := &progressWriter{
writer: file,
callback: progress,
}
_, err = io.Copy(pw, data)
if err != nil {
return fmt.Errorf("writing file: %w", err)
}
return nil
return f.writeAtomic(key, data, progress)
}
// Get retrieves data from the specified key.
@@ -188,7 +147,7 @@ func (f *FileStorer) List(ctx context.Context, prefix string) ([]string, error)
default:
}
if !info.IsDir() {
if !info.IsDir() && !strings.HasSuffix(info.Name(), tempSuffix) {
// Convert back to key (relative path from basePath)
relPath, err := filepath.Rel(f.basePath, path)
if err != nil {
@@ -245,7 +204,7 @@ func (f *FileStorer) ListStream(ctx context.Context, prefix string) <-chan Objec
return nil //nolint:nilerr // continue walking despite errors
}
if !info.IsDir() {
if !info.IsDir() && !strings.HasSuffix(info.Name(), tempSuffix) {
relPath, err := filepath.Rel(f.basePath, path)
if err != nil {
ch <- ObjectInfo{Err: fmt.Errorf("computing relative path: %w", err)}
@@ -275,6 +234,72 @@ func (f *FileStorer) Info() Info {
}
}
// writeAtomic streams data into a temp file in the destination directory,
// fsyncs it, and renames it onto the final key. The key therefore appears
// only once the whole object has been durably written; a failure part-way
// leaves a temp file (removed here on the failing path) rather than a
// truncated object at the key.
func (f *FileStorer) writeAtomic(
key string, data io.Reader, progress ProgressCallback,
) error {
path := f.fullPath(key)
dir := filepath.Dir(path)
err := f.fs.MkdirAll(dir, storageDirPerm)
if err != nil {
return fmt.Errorf("creating directories: %w", err)
}
tmp, err := afero.TempFile(f.fs, dir, filepath.Base(path)+"-*"+tempSuffix)
if err != nil {
return fmt.Errorf("creating temp file: %w", err)
}
tmpPath := tmp.Name()
// Remove the temp file unless the rename below claims it. On the success
// path renamed is true, so the deferred Close and Remove are harmless
// no-ops on a name that no longer exists.
renamed := false
defer func() {
_ = tmp.Close()
if !renamed {
_ = f.fs.Remove(tmpPath)
}
}()
var w io.Writer = tmp
if progress != nil {
w = &progressWriter{writer: tmp, callback: progress}
}
_, err = io.Copy(w, data)
if err != nil {
return fmt.Errorf("writing file: %w", err)
}
err = tmp.Sync()
if err != nil {
return fmt.Errorf("syncing temp file: %w", err)
}
err = tmp.Close()
if err != nil {
return fmt.Errorf("closing temp file: %w", err)
}
err = f.fs.Rename(tmpPath, path)
if err != nil {
return fmt.Errorf("renaming temp file: %w", err)
}
renamed = true
return nil
}
// fullPath returns the full filesystem path for a key.
func (f *FileStorer) fullPath(key string) string {
return filepath.Join(f.basePath, key)
+119
View File
@@ -0,0 +1,119 @@
package storage_test
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"sneak.berlin/go/vaultik/internal/storage"
)
// errStreamInterrupted stands in for an upload cut off mid-stream.
var errStreamInterrupted = errors.New("connection reset mid-upload")
// failingReader yields its data once, then fails.
type failingReader struct {
data []byte
done bool
}
func (r *failingReader) Read(p []byte) (int, error) {
if r.done {
return 0, errStreamInterrupted
}
n := copy(p, r.data)
r.done = true
return n, nil
}
// TestFileStorer_InterruptedWriteLeavesNoTrustedObject checks that a write
// cut off mid-stream leaves nothing at the destination key, so a later run
// cannot Stat a truncated object and trust it as a complete blob.
func TestFileStorer_InterruptedWriteLeavesNoTrustedObject(t *testing.T) {
t.Parallel()
f, err := storage.NewFileStorer(t.TempDir())
if err != nil {
t.Fatalf("NewFileStorer: %v", err)
}
ctx := context.Background()
key := "blobs/aa/bb/aabbccddeeff"
err = f.PutWithProgress(ctx, key, &failingReader{data: []byte("partial")}, 4096, nil)
if err == nil {
t.Fatal("expected the interrupted write to fail, got nil")
}
_, err = f.Stat(ctx, key)
if !errors.Is(err, storage.ErrNotFound) {
t.Fatalf("expected key absent after interrupted write, got Stat err %v", err)
}
keys, err := f.List(ctx, "blobs/")
if err != nil {
t.Fatalf("List: %v", err)
}
if len(keys) != 0 {
t.Fatalf("expected no keys listed after interrupted write, got %v", keys)
}
}
// TestFileStorer_ListSkipsPartialFiles checks that a leftover temp file (the
// storage layer names them with a ".partial" suffix) is never surfaced as a
// key by List or ListStream.
func TestFileStorer_ListSkipsPartialFiles(t *testing.T) {
t.Parallel()
base := t.TempDir()
f, err := storage.NewFileStorer(base)
if err != nil {
t.Fatalf("NewFileStorer: %v", err)
}
ctx := context.Background()
realKey := "blobs/aa/bb/aabbccddeeff"
err = f.Put(ctx, realKey, strings.NewReader("blob-bytes"))
if err != nil {
t.Fatalf("Put: %v", err)
}
// A stray temp file, as an interrupted write would leave behind.
leftover := filepath.Join(base, "blobs/aa/bb/aabbccddeeff-123456.partial")
err = os.WriteFile(leftover, []byte("half"), 0o600)
if err != nil {
t.Fatalf("writing leftover temp file: %v", err)
}
keys, err := f.List(ctx, "blobs/")
if err != nil {
t.Fatalf("List: %v", err)
}
if len(keys) != 1 || keys[0] != realKey {
t.Fatalf("List should return only the real key, got %v", keys)
}
var streamed []string
for obj := range f.ListStream(ctx, "blobs/") {
if obj.Err != nil {
t.Fatalf("ListStream: %v", obj.Err)
}
streamed = append(streamed, obj.Key)
}
if len(streamed) != 1 || streamed[0] != realKey {
t.Fatalf("ListStream should return only the real key, got %v", streamed)
}
}
+16 -1
View File
@@ -38,14 +38,29 @@ func (s *S3Storer) PutWithProgress(
}
// Get retrieves data from the specified key.
// Returns ErrNotFound if the object does not exist.
func (s *S3Storer) Get(ctx context.Context, key string) (io.ReadCloser, error) {
return s.client.GetObject(ctx, key)
rc, err := s.client.GetObject(ctx, key)
if err != nil {
if s3.IsNotFound(err) {
return nil, fmt.Errorf("get %q: %w", key, ErrNotFound)
}
return nil, err
}
return rc, nil
}
// Stat returns metadata about an object without retrieving its contents.
// Returns ErrNotFound if the object does not exist.
func (s *S3Storer) Stat(ctx context.Context, key string) (*ObjectInfo, error) {
info, err := s.client.StatObject(ctx, key)
if err != nil {
if s3.IsNotFound(err) {
return nil, fmt.Errorf("stat %q: %w", key, ErrNotFound)
}
return nil, err
}
+59
View File
@@ -0,0 +1,59 @@
package storage_test
import (
"context"
"errors"
"net/http/httptest"
"testing"
"github.com/johannesboyne/gofakes3"
"github.com/johannesboyne/gofakes3/backend/s3mem"
"sneak.berlin/go/vaultik/internal/s3"
"sneak.berlin/go/vaultik/internal/storage"
)
// TestS3StorerMissingKeyMapsToErrNotFound verifies that the s3 backend reports
// a missing object as storage.ErrNotFound, matching the file and rclone
// backends and the Storer contract. Without the mapping, Get and Stat leak the
// raw SDK error and errors.Is(err, storage.ErrNotFound) is false.
//
//nolint:paralleltest // shares an in-process S3 server via t.Cleanup
func TestS3StorerMissingKeyMapsToErrNotFound(t *testing.T) {
const bucket = "test-bucket"
backend := s3mem.New()
err := backend.CreateBucket(bucket)
if err != nil {
t.Fatalf("create bucket: %v", err)
}
srv := httptest.NewServer(gofakes3.New(backend).Server())
t.Cleanup(srv.Close)
ctx := context.Background()
client, err := s3.NewClient(ctx, s3.Config{
Endpoint: srv.URL,
Bucket: bucket,
AccessKeyID: "test",
SecretAccessKey: "test",
Region: "us-east-1",
})
if err != nil {
t.Fatalf("new client: %v", err)
}
storer := storage.NewS3Storer(client)
_, err = storer.Get(ctx, "does-not-exist")
if !errors.Is(err, storage.ErrNotFound) {
t.Errorf("Get on missing key: got %v, want ErrNotFound", err)
}
_, err = storer.Stat(ctx, "does-not-exist")
if !errors.Is(err, storage.ErrNotFound) {
t.Errorf("Stat on missing key: got %v, want ErrNotFound", err)
}
}
+108
View File
@@ -0,0 +1,108 @@
package vaultik_test
import (
"context"
"io"
"os"
"path/filepath"
"testing"
"github.com/spf13/afero"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/ui"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// TestDeepVerifyAcceptsHealthyAndRejectsCorruptBlob backs up a real
// snapshot with the on-disk storage backend, runs deep verification on
// it, then flips a byte inside one stored blob and runs deep
// verification again. A healthy snapshot must pass; a corrupted blob
// must fail. The healthy case is the regression guard: deep
// verification used to hash the encrypted blob bytes and compare them
// to the blob's ID (the double SHA256 of the plaintext), so it reported
// every healthy blob as corrupt.
func TestDeepVerifyAcceptsHealthyAndRejectsCorruptBlob(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
fs := afero.NewOsFs()
tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "source")
storeDir := filepath.Join(tempDir, "remote")
dbPath := filepath.Join(tempDir, "index.sqlite")
chunkSize := int64(64 * 1024)
maxBlobSize := int64(512 * 1024)
// One file large enough to span several chunks within a single blob.
require.NoError(t, fs.MkdirAll(dataDir, 0o755))
require.NoError(t, afero.WriteFile(fs,
filepath.Join(dataDir, "data.bin"),
bytesPattern("deep-", int(chunkSize*3)), 0o644))
ctx := context.Background()
// runFileStorageBackup writes a real snapshot to storeDir and closes
// the source index, so verification runs from remote bytes only.
cfg, storer, snapshotID := runFileStorageBackup(
ctx, t, fs, dataDir, storeDir, dbPath, chunkSize, maxBlobSize)
newVerifier := func() *vaultik.Vaultik {
v := &vaultik.Vaultik{
Config: cfg,
Storage: storer,
Fs: fs,
Stdout: io.Discard,
Stderr: io.Discard,
UI: ui.NewWithColor(io.Discard, false),
}
v.SetContext(ctx)
return v
}
require.NoError(t,
newVerifier().RunDeepVerify(snapshotID, &vaultik.VerifyOptions{Deep: true}),
"deep verify should pass on a healthy snapshot")
// Flip a byte inside one blob without changing its length, so the
// blob-existence and size checks still pass and verification reaches
// the blob-content stage.
corruptOneBlob(t, fs, filepath.Join(storeDir, "blobs"))
require.Error(t,
newVerifier().RunDeepVerify(snapshotID, &vaultik.VerifyOptions{Deep: true}),
"deep verify should fail on a corrupted blob")
}
// corruptOneBlob flips a middle byte of the first blob file found under
// blobsDir, leaving the file length unchanged.
func corruptOneBlob(t *testing.T, fs afero.Fs, blobsDir string) {
t.Helper()
var blobPath string
err := afero.Walk(fs, blobsDir,
func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
if blobPath == "" && !info.IsDir() {
blobPath = path
}
return nil
})
require.NoError(t, err)
require.NotEmpty(t, blobPath, "expected at least one blob on disk")
data, err := afero.ReadFile(fs, blobPath)
require.NoError(t, err)
require.NotEmpty(t, data)
data[len(data)/2] ^= 0xff
require.NoError(t, afero.WriteFile(fs, blobPath, data, 0o644))
}
+7 -1
View File
@@ -35,6 +35,7 @@ var (
"invalid snapshot ID format: expected hostname_snapshotname_timestamp")
errInvalidDuration = errors.New("invalid duration")
errUnknownTimeUnit = errors.New("unknown time unit")
errNegativeDuration = errors.New("negative durations are not supported")
)
// Time-unit lengths used by parseDuration.
@@ -138,8 +139,13 @@ func parseSnapshotName(snapshotID string) string {
// parseDuration parses a duration string with support for human-friendly units:
// d/day/days, w/week/weeks, mo/month/months, y/year/years, plus standard Go
// duration units (h, m, s).
// duration units. Following Go, m is minutes and mo is months. A bare number,
// an unknown unit, and a negative value are all rejected.
func parseDuration(s string) (time.Duration, error) {
if strings.HasPrefix(strings.TrimSpace(s), "-") {
return 0, errNegativeDuration
}
d, err := time.ParseDuration(s)
if err == nil {
return d, nil
+25 -6
View File
@@ -51,13 +51,32 @@ func TestParseDuration(t *testing.T) {
want time.Duration
err bool
}{
{"30d", 30 * 24 * time.Hour, false},
{"4w", 4 * 7 * 24 * time.Hour, false},
{"6mo", 6 * 30 * 24 * time.Hour, false},
{"1y", 365 * 24 * time.Hour, false},
{"2w3d", 2*7*24*time.Hour + 3*24*time.Hour, false},
{"1h", time.Hour, false},
// Go units, including the m-is-minutes / mo-is-months distinction
// that this parser exists to keep straight.
{"10ns", 10 * time.Nanosecond, false},
{"10us", 10 * time.Microsecond, false},
{"500ms", 500 * time.Millisecond, false},
{"30s", 30 * time.Second, false},
{"6m", 6 * time.Minute, false},
{"1h", time.Hour, false},
// Extended calendar units.
{"30d", 30 * 24 * time.Hour, false},
{"3days", 3 * 24 * time.Hour, false},
{"4w", 4 * 7 * 24 * time.Hour, false},
{"2weeks", 2 * 7 * 24 * time.Hour, false},
{"6mo", 180 * 24 * time.Hour, false},
{"1month", 30 * 24 * time.Hour, false},
{"1y", 365 * 24 * time.Hour, false},
{"2years", 2 * 365 * 24 * time.Hour, false},
// Combined units.
{"2w3d", 2*7*24*time.Hour + 3*24*time.Hour, false},
{"1y6mo", 365*24*time.Hour + 180*24*time.Hour, false},
// Rejected inputs.
{"6", 0, true}, // bare number, no unit
{"5x", 0, true}, // unknown unit
{"-5d", 0, true}, // negative, extended unit
{"-5h", 0, true}, // negative, Go unit
{"", 0, true}, // empty
{"garbage", 0, true},
}
+79
View File
@@ -0,0 +1,79 @@
package vaultik //nolint:testpackage // exercises unexported count helpers
import (
"context"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
)
// TestTableCountForReportSurfacesReadFailure is the regression guard for
// the discarded-error bug: getTableCount for a table its query cannot
// resolve must not silently become 0. A count that could not be read is
// reported as unknown, which a reader can tell apart from an empty table.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestTableCountForReportSurfacesReadFailure(t *testing.T) {
log.Initialize(log.Config{})
ctx := context.Background()
db, err := database.New(ctx, ":memory:")
require.NoError(t, err)
t.Cleanup(func() { _ = db.Close() })
v := &Vaultik{DB: db}
v.SetContext(ctx)
// A table present in the schema reads as a real count.
blobs := v.tableCountForReport("blobs")
require.NotNil(t, blobs, "an existing table must read as a real count")
assert.Equal(t, int64(0), *blobs)
// A syntactically valid name the sanitizer accepts but whose table
// the query cannot resolve is the exact shape #96 describes: a
// would-be loud failure that used to be discarded into a 0.
_, err = v.getTableCount("snapshots_missing")
require.Error(t, err, "a query against a nonexistent table must fail")
missing := v.tableCountForReport("snapshots_missing")
assert.Nil(t, missing, "a failed read is unknown, not a count")
// The rendered count for a failed read must say unknown, never 0.
assert.Equal(t, countUnknown, countText(missing))
assert.NotEqual(t, "0", countText(missing))
}
// TestCountTextDistinguishesEmptyFromUnknown pins the distinction the
// output has to preserve: 0 means the table was empty, "unknown" means
// the count could not be read.
func TestCountTextDistinguishesEmptyFromUnknown(t *testing.T) {
t.Parallel()
zero := int64(0)
seven := int64(7)
assert.Equal(t, "0", countText(&zero))
assert.Equal(t, "7", countText(&seven))
assert.Equal(t, countUnknown, countText(nil))
}
// TestCountDiffUnknownWhenEitherSideUnknown checks that a delta computed
// from an unreadable count is itself unknown rather than a plausible
// number.
func TestCountDiffUnknownWhenEitherSideUnknown(t *testing.T) {
t.Parallel()
before := int64(10)
after := int64(3)
require.NotNil(t, countDiff(&before, &after))
assert.Equal(t, int64(7), *countDiff(&before, &after))
assert.Nil(t, countDiff(nil, &after), "unknown before yields unknown delta")
assert.Nil(t, countDiff(&before, nil), "unknown after yields unknown delta")
assert.Nil(t, countDiff(nil, nil))
}
+10 -5
View File
@@ -18,7 +18,6 @@ import (
"sneak.berlin/go/vaultik/internal/blobgen"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot"
"sneak.berlin/go/vaultik/internal/types"
)
@@ -577,14 +576,20 @@ func (v *Vaultik) handleRestoreVerification(
}
// downloadSnapshotDB downloads and decrypts the snapshot metadata
// database. The snapshotID is the human ID; we hash it to the remote
// key for the storage path.
// database. The identifier is resolved to the snapshot's remote key: a
// human ID is hashed, and a remote key (or its abbreviation, as printed
// for a remote-only snapshot) is used as-is, so a host with no local
// index can restore the snapshots it can only see on the store.
func (v *Vaultik) downloadSnapshotDB(
snapshotID string, identity age.Identity,
) (*database.DB, error) {
remoteKey, err := v.resolveSnapshotRemoteKey(snapshotID)
if err != nil {
return nil, err
}
// Download encrypted database from storage
dbKey := fmt.Sprintf("metadata/%s/db.zst.age",
snapshot.RemoteSnapshotKey(snapshotID))
dbKey := fmt.Sprintf("metadata/%s/db.zst.age", remoteKey)
reader, err := v.Storage.Get(v.ctx, dbKey)
if err != nil {
@@ -0,0 +1,167 @@
package vaultik_test
import (
"bytes"
"context"
"io"
"path/filepath"
"testing"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/ui"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// TestRestoreOnAnotherMachine proves the disaster-recovery path: a host
// that has only the vaultik binary, the age secret key, and the storage
// credentials — no local index, a different hostname, and no
// age_recipients configured — can list, restore, and verify a snapshot
// straight from the destination store.
//
// The backup half writes a snapshot with one index and hostname. The
// restore half throws that index away entirely: a fresh, empty index and
// a config that shares nothing with the original but the storage location
// and the secret key. If restore or verify needed the original local
// index — or the human snapshot ID that only that index holds — this test
// could not run, because the recovery host can know neither.
func TestRestoreOnAnotherMachine(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
fs := afero.NewOsFs()
tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "source")
storeDir := filepath.Join(tempDir, "remote")
restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite")
chunkSize := int64(64 * 1024)
maxBlobSize := int64(512 * 1024)
sourceFiles := writeRecoverySourceTree(t, fs, dataDir, chunkSize)
ctx := context.Background()
// Backup host: one index, hostname test-host, age_recipients set.
// runFileStorageBackup closes the index before returning, so nothing
// below can lean on it.
_, storer, originalID := runFileStorageBackup(
ctx, t, fs, dataDir, storeDir, dbPath, chunkSize, maxBlobSize)
// Recovery host: a fresh empty index, a different hostname, and no
// age_recipients — only the secret key and the same storage location.
recovery, stdout := newRecoveryHost(ctx, t, fs, storer)
// The recovery index really is empty. This is the assertion that makes
// the test a guard against restore quietly depending on the original
// index: if it did, an empty index would make restore fail.
localSnaps, err := recovery.Repositories.Snapshots.ListRecent(ctx, 100)
require.NoError(t, err)
require.Empty(t, localSnaps, "recovery host must start with no local index")
// List: the snapshot shows up as remote-only, identified by its remote
// key, with no recoverable human ID.
require.NoError(t, recovery.ListSnapshots(true))
rows := decodeListJSON(t, stdout.String())
require.Len(t, rows, 1)
remote := rows[0]
assert.False(t, remote.LocallyTracked, "snapshot must be remote-only here")
assert.Empty(t, remote.ID, "the human ID is unknown to the recovery host")
require.Len(t, remote.RemoteKey, 64)
assert.Equal(t, snapshot.RemoteSnapshotKey(originalID), remote.RemoteKey,
"the listed key is the hashed snapshot ID")
// Restore driven by the abbreviated identifier the table prints (the
// first 12 hex of the remote key), then deep-verify from the store
// keyed by the full remote key. Both are what a recovery host can know.
require.NoError(t, recovery.Restore(&vaultik.RestoreOptions{
SnapshotID: remote.RemoteKey[:12],
TargetDir: restoreDir,
Verify: true,
}))
require.NoError(t, recovery.RunDeepVerify(
remote.RemoteKey, &vaultik.VerifyOptions{Deep: true}))
assertRestoredTreeMatches(t, fs, restoreDir, sourceFiles)
}
// writeRecoverySourceTree writes a small source tree spanning several
// chunks (so restore reassembles real multi-chunk files) and returns the
// content keyed by absolute path.
func writeRecoverySourceTree(
t *testing.T, fs afero.Fs, dataDir string, chunkSize int64,
) map[string][]byte {
t.Helper()
sourceFiles := map[string][]byte{
filepath.Join(dataDir, "notes.txt"): []byte("recover me"),
filepath.Join(dataDir, "sub", "big.bin"): bytesPattern("big-", int(chunkSize*3)),
filepath.Join(dataDir, "sub", "small.bin"): bytesPattern("small-", 128),
}
for path, content := range sourceFiles {
require.NoError(t, fs.MkdirAll(filepath.Dir(path), 0o755))
require.NoError(t, afero.WriteFile(fs, path, content, 0o644))
}
return sourceFiles
}
// newRecoveryHost builds the Vaultik a replacement machine would run: an
// empty in-memory index, a hostname different from the backup host, no
// age_recipients, and only the secret key plus the shared storer. It
// returns the instance and the buffer its stdout is wired to.
func newRecoveryHost(
ctx context.Context, t *testing.T, fs afero.Fs, storer storage.Storer,
) (*vaultik.Vaultik, *bytes.Buffer) {
t.Helper()
recoveryDB, err := database.New(ctx, ":memory:")
require.NoError(t, err)
t.Cleanup(func() { _ = recoveryDB.Close() })
stdout := &bytes.Buffer{}
recovery := &vaultik.Vaultik{
Config: &config.Config{
AgeSecretKey: testAgeSecretKey,
Hostname: "recovery-host",
},
Storage: storer,
Fs: fs,
Repositories: database.NewRepositories(recoveryDB),
DB: recoveryDB,
Stdout: stdout,
Stderr: io.Discard,
UI: ui.NewWithColor(io.Discard, false),
}
recovery.SetContext(ctx)
return recovery, stdout
}
// assertRestoredTreeMatches byte-compares every restored file against its
// source content.
func assertRestoredTreeMatches(
t *testing.T, fs afero.Fs, restoreDir string, sourceFiles map[string][]byte,
) {
t.Helper()
for origPath, expected := range sourceFiles {
restored := filepath.Join(restoreDir, origPath)
got, err := afero.ReadFile(fs, restored)
require.NoErrorf(t, err, "restored file missing: %s", restored)
require.Truef(t, bytes.Equal(got, expected),
"byte mismatch for %s", origPath)
}
}
+84 -29
View File
@@ -8,6 +8,7 @@ import (
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
"time"
@@ -669,9 +670,11 @@ func (v *Vaultik) VerifySnapshotWithOptions(
v.printVerifyHeader(snapshotID, opts)
// Download and parse manifest. The caller supplies a human
// snapshot ID; we hash it to address remote storage.
manifest, err := v.downloadManifestByKey(snapshot.RemoteSnapshotKey(snapshotID))
// Resolve the identifier to the snapshot's remote key and download the
// manifest. A human ID is hashed; a remote key (or its abbreviation,
// as printed for a remote-only snapshot) is used as-is, so a host with
// no local index can verify a snapshot it can only see on the store.
manifest, err := v.resolveAndDownloadManifest(snapshotID)
if err != nil {
if opts.JSON {
result.Status = verifyStatusFailed
@@ -1540,12 +1543,17 @@ func (v *Vaultik) outputRemoveJSON(result *RemoveResult) error {
return encoder.Encode(result)
}
// PruneResult contains statistics about the prune operation
// PruneResult contains statistics about the prune operation.
// SnapshotsDeleted counts snapshots actually deleted. FilesDeleted,
// ChunksDeleted, and BlobsDeleted are derived from before/after row
// counts of the local index; each is nil when a count could not be read,
// so an unreadable count is reported as unknown rather than silently
// as 0.
type PruneResult struct {
SnapshotsDeleted int64
FilesDeleted int64
ChunksDeleted int64
BlobsDeleted int64
FilesDeleted *int64
ChunksDeleted *int64
BlobsDeleted *int64
}
// PruneDatabase removes incomplete snapshots and orphaned files, chunks,
@@ -1560,7 +1568,7 @@ func (v *Vaultik) PruneDatabase() (*PruneResult, error) {
result := &PruneResult{}
// Snapshot counts before deletion of incompletes.
snapshotCountBefore, _ := v.getTableCount("snapshots")
snapshotCountBefore := v.tableCountForReport("snapshots")
// First, delete any incomplete snapshots
incompleteSnapshots, err := v.Repositories.Snapshots.GetIncompleteSnapshots(v.ctx)
@@ -1575,9 +1583,9 @@ func (v *Vaultik) PruneDatabase() (*PruneResult, error) {
}
// Get counts before cleanup for reporting
fileCountBefore, _ := v.getTableCount("files")
chunkCountBefore, _ := v.getTableCount("chunks")
blobCountBefore, _ := v.getTableCount("blobs")
fileCountBefore := v.tableCountForReport("files")
chunkCountBefore := v.tableCountForReport("chunks")
blobCountBefore := v.tableCountForReport("blobs")
// Run the cleanup
err = v.SnapshotManager.CleanupOrphanedData(v.ctx)
@@ -1586,36 +1594,83 @@ func (v *Vaultik) PruneDatabase() (*PruneResult, error) {
}
// Get counts after cleanup
fileCountAfter, _ := v.getTableCount("files")
chunkCountAfter, _ := v.getTableCount("chunks")
blobCountAfter, _ := v.getTableCount("blobs")
fileCountAfter := v.tableCountForReport("files")
chunkCountAfter := v.tableCountForReport("chunks")
blobCountAfter := v.tableCountForReport("blobs")
result.FilesDeleted = fileCountBefore - fileCountAfter
result.ChunksDeleted = chunkCountBefore - chunkCountAfter
result.BlobsDeleted = blobCountBefore - blobCountAfter
result.FilesDeleted = countDiff(fileCountBefore, fileCountAfter)
result.ChunksDeleted = countDiff(chunkCountBefore, chunkCountAfter)
result.BlobsDeleted = countDiff(blobCountBefore, blobCountAfter)
log.Info("Local database prune complete",
"incomplete_snapshots", result.SnapshotsDeleted,
"orphaned_files", result.FilesDeleted,
"orphaned_chunks", result.ChunksDeleted,
"orphaned_blobs", result.BlobsDeleted,
"orphaned_files", countText(result.FilesDeleted),
"orphaned_chunks", countText(result.ChunksDeleted),
"orphaned_blobs", countText(result.BlobsDeleted),
)
snapshotCountAfter := snapshotCountBefore - result.SnapshotsDeleted
// Snapshots remaining after removing the incomplete ones; unknown if
// the pre-prune snapshot count could not be read.
snapshotsRemain := countDiff(snapshotCountBefore, &result.SnapshotsDeleted)
v.UI.Completef("Pruned local index database.")
v.UI.Detailf("Incomplete snapshots: %d removed (%d remain).",
result.SnapshotsDeleted, snapshotCountAfter)
v.UI.Detailf("Orphaned files: %d removed (%d remain).",
result.FilesDeleted, fileCountAfter)
v.UI.Detailf("Orphaned chunks: %d removed (%d remain).",
result.ChunksDeleted, chunkCountAfter)
v.UI.Detailf("Orphaned blobs: %d removed (%d remain).",
result.BlobsDeleted, blobCountAfter)
v.UI.Detailf("Incomplete snapshots: %s removed (%s remain).",
countText(&result.SnapshotsDeleted), countText(snapshotsRemain))
v.UI.Detailf("Orphaned files: %s removed (%s remain).",
countText(result.FilesDeleted), countText(fileCountAfter))
v.UI.Detailf("Orphaned chunks: %s removed (%s remain).",
countText(result.ChunksDeleted), countText(chunkCountAfter))
v.UI.Detailf("Orphaned blobs: %s removed (%s remain).",
countText(result.BlobsDeleted), countText(blobCountAfter))
return result, nil
}
// countUnknown is what a count reads as when its query could not be run,
// distinct from "0", which means the table really was empty.
const countUnknown = "unknown"
// tableCountForReport returns the row count of a table for the prune
// summary, or nil if the count could not be read. A read failure is
// logged at warn — visible even under --json, which routes warnings to
// stderr — and then rendered as unknown rather than silently becoming 0,
// so a broken query is a visible failure instead of a plausible wrong
// number.
func (v *Vaultik) tableCountForReport(tableName string) *int64 {
count, err := v.getTableCount(tableName)
if err != nil {
log.Warn("could not read table row count for prune summary",
"table", tableName, "error", err)
return nil
}
return &count
}
// countDiff returns before-after, or nil if either count is unknown so
// that an unreadable count does not collapse into a plausible delta.
func countDiff(before, after *int64) *int64 {
if before == nil || after == nil {
return nil
}
diff := *before - *after
return &diff
}
// countText renders a count that may be unknown: nil (the read failed)
// becomes "unknown", never "0", so a reader can tell an empty table from
// one that could not be queried.
func countText(count *int64) string {
if count == nil {
return countUnknown
}
return strconv.FormatInt(*count, 10)
}
// validTableNameRe matches table names containing only lowercase
// alphanumeric characters and underscores.
var validTableNameRe = regexp.MustCompile(`^[a-z0-9_]+$`)
+101
View File
@@ -0,0 +1,101 @@
package vaultik
import (
"errors"
"fmt"
"strings"
"sneak.berlin/go/vaultik/internal/snapshot"
)
// remoteKeyHexLen is the length of a full remote snapshot key: a SHA256
// digest rendered as lowercase hex.
const remoteKeyHexLen = 64
// Sentinel errors for resolving a snapshot identifier against the store.
var (
errSnapshotKeyNotFound = errors.New(
"no snapshot on the destination store matches this identifier")
errSnapshotKeyAmbiguous = errors.New(
"identifier matches more than one snapshot on the destination store")
)
// resolveSnapshotRemoteKey turns a snapshot identifier supplied on the
// command line into the remote key that names the snapshot's metadata
// directory on the destination store. Every remote path a restore or
// verify reads is built from that key.
//
// Two forms are accepted, matching the two things a host can know:
//
// - A human snapshot ID (hostname_name_timestamp), which a host holding
// the local index has. It is hashed to its remote key; the store is
// not consulted.
// - A remote key, or the leading part of one, which is all a host with
// no local index can know — it is exactly what `snapshot list` prints
// for a remote-only snapshot (see formatRemoteOnlyID). It is resolved
// against the destination store's metadata listing; an identifier that
// matches no snapshot, or more than one, is an error.
//
// The two are told apart by shape: a remote key is lowercase hex, and a
// human snapshot ID never is (it carries a hostname, underscores, and an
// RFC3339 timestamp).
func (v *Vaultik) resolveSnapshotRemoteKey(identifier string) (string, error) {
if !isRemoteKeyOrPrefix(identifier) {
return snapshot.RemoteSnapshotKey(identifier), nil
}
keys, err := v.listAllRemoteSnapshotKeys()
if err != nil {
return "", fmt.Errorf(
"listing destination store to resolve %q: %w", identifier, err)
}
var matches []string
for _, key := range keys {
if strings.HasPrefix(key, identifier) {
matches = append(matches, key)
}
}
switch len(matches) {
case 1:
return matches[0], nil
case 0:
return "", fmt.Errorf("%w: %s", errSnapshotKeyNotFound, identifier)
default:
return "", fmt.Errorf("%w: %s (%d matches)",
errSnapshotKeyAmbiguous, identifier, len(matches))
}
}
// resolveAndDownloadManifest resolves a snapshot identifier to its remote
// key (see resolveSnapshotRemoteKey) and downloads that snapshot's
// manifest.
func (v *Vaultik) resolveAndDownloadManifest(
identifier string,
) (*snapshot.Manifest, error) {
remoteKey, err := v.resolveSnapshotRemoteKey(identifier)
if err != nil {
return nil, err
}
return v.downloadManifestByKey(remoteKey)
}
// isRemoteKeyOrPrefix reports whether s is a full remote key or the
// leading part of one: 1 to 64 lowercase hex characters. A human snapshot
// ID is never all hex, so this shape test is enough to tell the two apart.
func isRemoteKeyOrPrefix(s string) bool {
if s == "" || len(s) > remoteKeyHexLen {
return false
}
for _, r := range s {
if (r < '0' || r > '9') && (r < 'a' || r > 'f') {
return false
}
}
return true
}
+37 -23
View File
@@ -138,8 +138,15 @@ func (v *Vaultik) RunDeepVerify(snapshotID string, opts *VerifyOptions) error {
func (v *Vaultik) loadVerificationData(
snapshotID string, opts *VerifyOptions, result *VerifyResult,
) (*snapshot.Manifest, *tempDB, []snapshot.BlobInfo, error) {
// All remote paths use the hashed key derived from the human ID.
remoteKey := snapshot.RemoteSnapshotKey(snapshotID)
// Resolve the identifier to the snapshot's remote key. A human ID is
// hashed; a remote key (or its abbreviation, as printed for a
// remote-only snapshot) is used as-is, so a host with no local index
// can verify a snapshot it can only see on the store.
remoteKey, err := v.resolveSnapshotRemoteKey(snapshotID)
if err != nil {
return nil, nil, nil, v.deepVerifyFailure(result, opts,
fmt.Sprintf("resolving snapshot identifier: %v", err), err)
}
// Download manifest. downloadManifestByKey is the single reader for
// remote manifests; see its doc comment.
@@ -186,7 +193,7 @@ func (v *Vaultik) loadVerificationData(
fmt.Errorf("failed to decrypt database: %w", err))
}
dbBlobs, err := v.getBlobsFromDatabase(snapshotID, tdb.DB)
dbBlobs, err := v.getBlobsFromDatabase(tdb.DB)
if err != nil {
_ = tdb.Close()
@@ -344,12 +351,8 @@ func (v *Vaultik) verifyBlob(blobInfo snapshot.BlobInfo, db *sql.DB) error {
return fmt.Errorf("failed to get decryptor: %w", err)
}
// Hash the encrypted blob data as it streams through to decryption
blobHasher := sha256.New()
teeReader := io.TeeReader(reader, blobHasher)
// Decrypt blob (reading through teeReader to hash encrypted data)
decryptedReader, err := decryptor.DecryptStream(teeReader)
// Decrypt blob
decryptedReader, err := decryptor.DecryptStream(reader)
if err != nil {
return fmt.Errorf("failed to decrypt: %w", err)
}
@@ -361,12 +364,19 @@ func (v *Vaultik) verifyBlob(blobInfo snapshot.BlobInfo, db *sql.DB) error {
}
defer decompressor.Close()
chunkCount, err := v.verifyBlobChunks(db, blobInfo.Hash, decompressor)
// A blob's hash — its remote name — is the double SHA256 of its
// decompressed plaintext (see blobgen.Writer.Sum256), not of the
// encrypted bytes. Hash the plaintext as chunk verification streams
// it, then compare on completion.
plaintextHasher := sha256.New()
hashedStream := io.TeeReader(decompressor, plaintextHasher)
chunkCount, err := v.verifyBlobChunks(db, blobInfo.Hash, hashedStream)
if err != nil {
return err
}
err = v.verifyBlobFinalIntegrity(decompressor, blobHasher, blobInfo.Hash)
err = v.verifyBlobFinalIntegrity(hashedStream, plaintextHasher, blobInfo.Hash)
if err != nil {
return err
}
@@ -470,14 +480,13 @@ func (v *Vaultik) verifyBlobChunks(
}
// verifyBlobFinalIntegrity checks that no trailing data exists in the
// decompressed stream and that the encrypted blob hash matches the
// expected value.
// decompressed stream and that the blob hash matches the expected value.
func (v *Vaultik) verifyBlobFinalIntegrity(
decompressor io.Reader, blobHasher hash.Hash, expectedHash string,
plaintext io.Reader, plaintextHasher hash.Hash, expectedHash string,
) error {
// Verify no remaining data in blob - if the chunk list is accurate,
// the blob should be fully consumed.
remaining, err := io.Copy(io.Discard, decompressor)
remaining, err := io.Copy(io.Discard, plaintext)
if err != nil {
return fmt.Errorf("failed to check for remaining blob data: %w", err)
}
@@ -486,8 +495,11 @@ func (v *Vaultik) verifyBlobFinalIntegrity(
return fmt.Errorf("%w: %d bytes", errTrailingBlobData, remaining)
}
// Verify blob hash matches the encrypted data we downloaded
calculatedBlobHash := hex.EncodeToString(blobHasher.Sum(nil))
// The blob hash is the double SHA256 of its plaintext content.
firstHash := plaintextHasher.Sum(nil)
secondHash := sha256.Sum256(firstHash)
calculatedBlobHash := hex.EncodeToString(secondHash[:])
if calculatedBlobHash != expectedHash {
return fmt.Errorf("%w: calculated %s, expected %s",
errBlobHashMismatch, calculatedBlobHash, expectedHash)
@@ -496,19 +508,21 @@ func (v *Vaultik) verifyBlobFinalIntegrity(
return nil
}
// getBlobsFromDatabase gets all blobs for the snapshot from the database
func (v *Vaultik) getBlobsFromDatabase(
snapshotID string, db *sql.DB,
) ([]snapshot.BlobInfo, error) {
// getBlobsFromDatabase gets all blobs for the snapshot from the database.
//
// The exported per-snapshot database holds exactly one snapshot's data
// (see cleanSnapshotDB), so every row in snapshot_blobs belongs to it.
// We select them directly rather than filtering by the human snapshot ID,
// which a host restoring from the store alone does not have.
func (v *Vaultik) getBlobsFromDatabase(db *sql.DB) ([]snapshot.BlobInfo, error) {
query := `
SELECT b.blob_hash, b.compressed_size
FROM snapshot_blobs sb
JOIN blobs b ON sb.blob_hash = b.blob_hash
WHERE sb.snapshot_id = ?
ORDER BY b.blob_hash
`
rows, err := db.QueryContext(v.ctx, query, snapshotID)
rows, err := db.QueryContext(v.ctx, query)
if err != nil {
return nil, fmt.Errorf("failed to query snapshot blobs: %w", err)
}
+18 -18
View File
@@ -48,11 +48,12 @@ missing() {
! command -v "$1" >/dev/null 2>&1
}
# Docker is a hard requirement, not a nice-to-have: script/lint runs the
# digest-pinned golangci-lint image from the Dockerfile's lint stage, and
# script/check and script/precommit both run script/lint. A bootstrap
# that prints "bootstrap complete" on a machine where `make check` cannot
# run is a false success, so this fails instead.
# Docker is a hard requirement, not a nice-to-have: script/lint lints by
# building Dockerfile.lint, whose digest-pinned golangci-lint image is
# the only place the linter runs, and script/check and script/precommit
# both run script/lint. A bootstrap that prints "bootstrap complete" on a
# machine where `make check` cannot run is a false success, so this fails
# instead.
#
# Installing docker from here was considered and rejected: it needs root,
# a running daemon, and on macOS a GUI cask, so an attempt would itself
@@ -79,13 +80,15 @@ bootstrap: FAILED - $reason.
Docker is required to develop this repo. Without it these do not work:
script/lint runs the digest-pinned golangci-lint image declared
by the Dockerfile's lint stage, which is the single
source of truth for the linter version
script/lint builds Dockerfile.lint, which runs the linter as a
build step in a digest-pinned golangci-lint image.
That FROM line is the single source of truth for the
linter version
script/check runs script/lint
script/precommit runs script/check, so commits are blocked by the
pre-commit hook installed by script/setup
script/cibuild builds the Dockerfile, which is what CI runs
script/cibuild builds Dockerfile.lint and Dockerfile, which is what
CI runs
Install docker (and start the daemon, checking DOCKER_HOST and your
group membership), then re-run script/bootstrap. golangci-lint on PATH
@@ -104,15 +107,12 @@ main() {
# Go toolchain
if missing go; then pkg_install go golang go go; fi
# golangci-lint is deliberately NOT installed: script/lint runs the
# digest-pinned golangci-lint image from the Dockerfile's lint stage,
# so whatever a package manager happens to ship would only be a
# shadow of the pinned version that could drift from CI. script/lint
# will not use a PATH binary on a host at any version, so installing
# one here would buy nothing.
# sqlite3 CLI: the test suite shells out to it (VACUUM).
if missing sqlite3; then pkg_install sqlite sqlite3 sqlite sqlite; fi
# golangci-lint is deliberately NOT installed: script/lint lints by
# building Dockerfile.lint, whose digest-pinned image is the only
# place the linter runs, so whatever a package manager happens to
# ship would only be a shadow of the pinned version that could drift
# from CI. Nothing on the host is ever used as a linter, at any
# version, so installing one here would buy nothing.
# goreleaser, at the version pinned by script/install-goreleaser and
# verified against a hardcoded sha256. Package managers are not used
+32 -11
View File
@@ -1,22 +1,31 @@
#!/bin/sh
# script/cibuild: run the CI build. The Dockerfile does not run
# script/check; it runs `make fmt-check` and `make lint` in its lint
# stage and `make test` in its builder stage. A successful build
# implies those three passed, provided they actually ran -- which is
# what the CHECK_EPOCH below is for.
# Generic: needs no adaptation. The Gitea workflow runs this on push.
# script/cibuild: run the CI build. This is the full gate, and it is two
# builds, in this order:
#
# Dockerfile.lint the linter, as a build step (a clean build IS a
# clean lint)
# Dockerfile `make fmt-check` and `make test` in the builder
# stage, then the product image
#
# Either one failing fails this script. Note what follows from the
# split: script/docker builds only the product image and so no longer
# lints -- this script and script/check (which runs script/lint) are the
# things that decide whether the tree is clean.
#
# Generic apart from the two Dockerfiles: the Gitea workflow runs this
# on push.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
# The Dockerfile's check layers are keyed on CHECK_EPOCH, so a
# fresh value here is what forces them to re-run: without it an
# Both Dockerfiles key their check layers on CHECK_EPOCH, so a fresh
# value is what forces those layers to re-run: without it an
# unchanged tree replays them from cache, the checks never execute,
# and the build still exits 0. The ARG sits immediately above the
# check RUNs, so dependency and module layers still cache. The
# Dockerfile also refuses to build at all when CHECK_EPOCH is empty,
# and the build still exits 0. Each ARG sits immediately above the
# check RUNs, so dependency and module layers still cache. Both
# Dockerfiles also refuse to build at all when CHECK_EPOCH is empty,
# so a missing value fails loudly here rather than passing quietly.
#
# The value must be unique per invocation, not per second. `date +%s`
@@ -35,6 +44,18 @@ main() {
# script exists to prevent -- so the guard would disarm itself and
# still exit 0. As a bare assignment, `set -e` catches a failing
# `date` and no build starts.
#
# A separate value per build, because they are separate builds: one
# `date` shared between them would still be fresh, but reusing it
# invites the two to be collapsed into a single value that is
# computed somewhere else and passed in.
epoch="$(date +%s%N)$$"
# cacheonly for the lint build: its verdict is the exit status and
# the image is never run, so exporting it is pure cost. See
# script/lint.
docker build --output=type=cacheonly \
--build-arg CHECK_EPOCH="$epoch" -f Dockerfile.lint .
epoch="$(date +%s%N)$$"
docker build --build-arg CHECK_EPOCH="$epoch" .
}
+7
View File
@@ -2,6 +2,13 @@
# script/docker: build the Docker image tagged with the project name.
# Identical in all repos; the tag comes from script/projectname.
# Generic: needs no adaptation.
#
# This builds the PRODUCT image only, and the product Dockerfile has no
# lint stage: linting lives in Dockerfile.lint and is run by
# script/lint. So a green here means `make fmt-check` and `make test`
# passed and the image built -- it says nothing about lint. The gates
# are script/check (which runs script/lint) and script/cibuild (which
# builds both files).
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+161
View File
@@ -0,0 +1,161 @@
#!/bin/sh
# script/install-go: install the Go toolchain pinned by go.mod into the
# repo-local tool directory, verified against a committed sha256. Our
# own extension to scripts-to-rule-them-all. Idempotent: exits at once
# when the pinned toolchain is already installed.
#
# Only .gitea/workflows/release.yml calls this. goreleaser is not a
# compiler: it shells out to `go` for the `before:` hook and for every
# one of the four cross-compiles, so the release runner needs a Go
# toolchain on PATH. check.yml never does -- it builds inside the
# digest-pinned Dockerfile images -- so this is the release path's only
# host Go, and per REPO_POLICIES.md it must be pinned by hash.
# actions/setup-go exposes no checksum input, so Go is installed the way
# script/install-goreleaser installs goreleaser: download the exact
# archive from go.dev and refuse it unless its sha256 matches the value
# committed below.
#
# The version is go.mod's `go` directive, the single source of truth for
# the toolchain. GO_VERSION below MUST equal it, and this script fails
# when they disagree -- so bumping Go is one reviewed change touching
# go.mod, the checksum here, and the Dockerfile golang digest together.
#
# Linux only, because that is what the release runner is. A darwin dev
# building a snapshot uses their own Go; supporting an OS means adding
# its checksums.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Go 1.26.1, 2026-09-21. Checksums are the sha256 values go.dev publishes
# for each archive at https://go.dev/dl/ (also in its ?mode=json
# manifest).
GO_VERSION="1.26.1"
SHA256_LINUX_AMD64="031f088e5d955bab8657ede27ad4e3bc5b7c1ba281f05f245bcc304f327c987a"
SHA256_LINUX_ARM64="a290581cfe4fe28ddd737dde3095f3dbeb7f2e4065cab4eae44dfc53b760c2f7"
GOROOT_DIR="$ROOT/.tool/go"
GOCMD="$GOROOT_DIR/bin/go"
# The `go` directive in go.mod, e.g. "1.26.1" from `go 1.26.1`.
gomod_go_version() {
sed -n 's/^go \([0-9][0-9.]*\).*/\1/p' "$ROOT/go.mod" | head -n 1
}
# Print the version of the go at $1 as "1.26.1", or nothing if it is not
# usable. `go version` prints "go version go1.26.1 linux/amd64".
go_version() {
[ -x "$1" ] || return 0
"$1" version 2>/dev/null |
sed -n 's/^go version go\([0-9][0-9.]*\) .*/\1/p' |
head -n 1
}
verify_sha256() {
file="$1"
want="$2"
if command -v sha256sum >/dev/null 2>&1; then
got="$(sha256sum "$file" | cut -d' ' -f1)"
elif command -v shasum >/dev/null 2>&1; then
got="$(shasum -a 256 "$file" | cut -d' ' -f1)"
else
echo "install-go: no sha256sum or shasum available" >&2
return 1
fi
if [ "$got" != "$want" ]; then
echo "install-go: checksum mismatch for $file" >&2
echo " expected: $want" >&2
echo " actual: $got" >&2
return 1
fi
}
# On a Gitea/GitHub Actions runner, put the toolchain on PATH for the
# steps that follow by appending to the file named by $GITHUB_PATH. A
# no-op off CI, where the caller manages its own PATH.
export_ci_path() {
[ -n "${GITHUB_PATH:-}" ] || return 0
echo "$GOROOT_DIR/bin" >>"$GITHUB_PATH"
}
main() {
cd "$ROOT"
want="$(gomod_go_version)"
if [ "$want" != "$GO_VERSION" ]; then
echo "install-go: go.mod says go $want but this script pins" \
"$GO_VERSION." >&2
echo " Update GO_VERSION and the checksums in this script to" \
"match go.mod." >&2
exit 1
fi
# Already installed from a previous run? Then just fix PATH and stop.
if [ "$(go_version "$GOCMD")" = "$GO_VERSION" ]; then
echo "go $GO_VERSION already installed in .tool/go"
export_ci_path
return 0
fi
os="$(uname -s)"
arch="$(uname -m)"
case "$os" in
Linux) os="linux" ;;
*)
echo "install-go: unsupported OS $os (release runner is Linux)" >&2
exit 1
;;
esac
case "$arch" in
x86_64 | amd64)
arch="amd64"
sum="$SHA256_LINUX_AMD64"
;;
arm64 | aarch64)
arch="arm64"
sum="$SHA256_LINUX_ARM64"
;;
*)
echo "install-go: no pinned checksum for architecture $arch" >&2
exit 1
;;
esac
archive="go${GO_VERSION}.${os}-${arch}.tar.gz"
url="https://go.dev/dl/${archive}"
if ! command -v curl >/dev/null 2>&1; then
echo "install-go: curl is required" >&2
exit 1
fi
dl="$(mktemp -d)"
mkdir -p "$ROOT/.tool"
stage="$(mktemp -d "$ROOT/.tool/.go-install.XXXXXX")"
# shellcheck disable=SC2064 # expand the paths now, not at trap time
trap "rm -rf '$dl' '$stage'" EXIT INT TERM
echo "installing go $GO_VERSION for ${os}-${arch}"
curl -fsSL --retry 3 -o "$dl/$archive" "$url"
verify_sha256 "$dl/$archive" "$sum"
# The archive unpacks to a top-level `go/` directory. Extract it into
# a staging directory on the same filesystem as the destination, then
# rename it into place so a concurrent run never observes a
# half-written toolchain.
tar -xzf "$dl/$archive" -C "$stage"
rm -rf "$GOROOT_DIR"
mv "$stage/go" "$GOROOT_DIR"
installed="$(go_version "$GOCMD")"
if [ "$installed" != "$GO_VERSION" ]; then
echo "install-go: installed toolchain reports '$installed'," \
"expected '$GO_VERSION'" >&2
exit 1
fi
echo "go $GO_VERSION installed to .tool/go"
export_ci_path
}
main "$@"
+64 -291
View File
@@ -1,110 +1,42 @@
#!/bin/sh
# script/lint: run the linter.
#
# The linter always runs at the version pinned by the Dockerfile's lint
# stage, so a local run and a CI run of the same tree cannot disagree.
# That FROM line (image tag plus digest) is the single source of truth
# for the linter version in this repo: bump it there and nothing else
# needs editing.
# The linter runs inside the image built by Dockerfile.lint, and it runs
# there as a BUILD STEP: a successful build of that file IS a clean
# lint. Nothing lints on the host, at any version, ever. That FROM line
# is the single source of truth for the linter version in this repo, so
# a local run and a CI run of the same tree cannot disagree.
#
# Normally that means running the pinned image with docker. The one
# exception is running INSIDE that image: the Dockerfile's lint stage
# runs `make lint`, and there is no docker daemon in there. That stage
# sets VAULTIK_LINT_IN_CONTAINER=1, and only when that variable is set
# is a golangci-lint on PATH used directly - and then only if its
# version is exactly the pin. Version equality alone is deliberately NOT
# enough: it also matches a developer's locally installed copy of the
# same version, which is a different build with a different Go
# toolchain, reached by a different code path, and it would bypass the
# digest pin this script exists to enforce. /.dockerenv was considered
# as the context signal and rejected: dockerd creates it for `docker
# run`, but it is not reliably present during a BuildKit `docker build`,
# which is exactly the case the exception exists for.
# One container per run means one lint cache and one golangci-lint lock
# per run, both private to that run and thrown away with it. That is
# what makes concurrent runs on a shared host safe, and it is why this
# script no longer carries per-worktree cache directories, a lock-retry
# loop, or an output audit: there is no shared state left for them to
# defend (issue https://git.eeqj.de/sneak/vaultik/issues/113).
#
# The linter's output is checked before it is believed: every run is
# audited by script/lint-audit for findings that cannot belong to this
# tree, and a run refused by golangci-lint's cross-process lock is
# retried rather than reported as a verdict. See the lock-retry loop in
# main and the header of script/lint-audit.
# To watch the linter execute, set BUILDKIT_PROGRESS=plain, which docker
# honours directly:
#
# Extra arguments are passed through to `golangci-lint run`, before
# `./...` (see script/lint-fix).
# BUILDKIT_PROGRESS=plain script/lint
#
# The check layers -- `golangci-lint config verify` and then
# `golangci-lint run` -- must appear as executing rather than CACHED on
# every run; see the CHECK_EPOCH comment in Dockerfile.lint.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
DOCKERFILE="$ROOT/Dockerfile"
# golangci-lint takes a cross-process lock and refuses to start while
# another instance holds it. That refusal is not a lint result, and
# exiting non-zero on it is indistinguishable to a caller from real
# findings - so it is retried rather than reported. Bounded, because a
# lock that is never released must fail rather than hang.
LOCK_MESSAGE="parallel golangci-lint is running"
LOCK_ATTEMPTS=6
LOCK_SLEEP=15
# The image reference of the Dockerfile's lint stage, tag and digest
# included, e.g.
# golangci/golangci-lint:v2.12.2-alpine@sha256:91b2...
lint_image() {
awk '$1 == "FROM" && $3 == "AS" && $4 == "lint" { print $2; exit }' \
"$DOCKERFILE"
}
# The bare version that image reference pins, e.g. 2.12.2
pinned_version() {
lint_image | sed -e 's/@.*//' -e 's/.*://' -e 's/^v//' -e 's/-.*//'
}
# The version of the golangci-lint on PATH, if any, e.g. 2.12.2
#
# `version --short` prints the bare version and is the interface meant
# for this (checked against 2.10.1 and 2.12.2). The banner scrape below
# it is a fallback for a release where --short is absent or silent; the
# banner's exact wording is not a stable interface, which is why it is
# no longer the primary parse.
installed_version() {
command -v golangci-lint >/dev/null 2>&1 || return 0
short="$(golangci-lint version --short 2>/dev/null |
tr -d '[:space:]' | sed -e 's/^v//')"
case "$short" in
*[0-9].[0-9]*.[0-9]*)
echo "$short"
return 0
;;
esac
golangci-lint version 2>/dev/null | awk '
{
for (i = 1; i <= NF; i++) {
if ($i ~ /^[0-9]+\.[0-9]+\.[0-9]+$/) {
print $i
exit
}
}
}'
}
# True inside the Dockerfile's lint stage, which sets this. Nothing else
# sets it: setting it by hand on a host is an explicit, visible decision
# to lint with an unpinned binary, not something reached by accident.
in_lint_container() {
[ "${VAULTIK_LINT_IN_CONTAINER:-}" = "1" ]
}
DOCKERFILE="$ROOT/Dockerfile.lint"
require_docker() {
image="$1"
if ! command -v docker >/dev/null 2>&1; then
cat >&2 <<EOF
lint: docker is required to run the pinned linter.
pinned image: $image
lint image declared by: $DOCKERFILE
Install docker. Linting with any other golangci-lint is not supported:
it is what lets a local run pass while CI fails. An installed
golangci-lint on PATH is not used, whatever its version; only the lint
stage of the Dockerfile itself runs the linter natively.
it is what lets a local run pass while CI fails. A golangci-lint on
PATH is never used, whatever its version.
EOF
exit 1
fi
@@ -113,7 +45,7 @@ EOF
lint: the docker daemon is not reachable, so the pinned linter cannot
run.
pinned image: $image
lint image declared by: $DOCKERFILE
Start the daemon (and check DOCKER_HOST / your group membership). This
script will not fall back to a different linter version or to an
@@ -123,213 +55,54 @@ EOF
fi
}
# Where the per-worktree caches live.
cache_home() {
echo "${XDG_CACHE_HOME:-${HOME:-/tmp}/.cache}/vaultik-lint"
}
usage() {
cat >&2 <<EOF
usage: $(basename "$0")
# A short, stable digest of this worktree's path.
path_digest() {
if command -v sha256sum >/dev/null 2>&1; then
printf '%s' "$ROOT" | sha256sum | cut -c1-12
elif command -v shasum >/dev/null 2>&1; then
printf '%s' "$ROOT" | shasum -a 256 | cut -c1-12
else
printf '%s' "$ROOT" | cksum | tr -cd '0-9' | cut -c1-12
fi
}
# Caches for the containerized linter, private to THIS worktree.
#
# Keeping them out of the repo and persisting them between runs is what
# keeps the inner loop fast: a warm run costs about the same as a native
# one plus container startup. Keeping them keyed on the worktree path is
# what keeps them correct. One shared cache for the whole repo was the
# defect in issue #99: two worktrees of this repo have identical file
# contents, so their cache keys collide, and golangci-lint replays the
# stored results - including the file paths recorded when they were
# produced. That silently reports one worktree's findings, or one
# worktree's clean bill of health, for another.
cache_dir() {
slug="$(printf '%s' "$(basename "$ROOT")" | tr -c 'A-Za-z0-9._-' '-')"
echo "$(cache_home)/$slug-$(path_digest)"
}
# One cache per worktree means throwaway worktrees would otherwise leave
# caches behind forever. Each cache records the worktree it belongs to,
# and any cache whose worktree no longer exists is collected here, so
# growth is bounded by the number of worktrees that actually exist. The
# whole tree also sits under XDG_CACHE_HOME (~/.cache by default), so it
# is disposable by definition: `rm -rf "${XDG_CACHE_HOME:-~/.cache}/vaultik-lint"`
# costs nothing but the next run's cold cache.
prune_dead_caches() {
home="$(cache_home)"
if [ ! -d "$home" ]; then
return 0
fi
for dir in "$home"/*; do
if [ ! -f "$dir/worktree" ]; then
continue
fi
owner="$(cat "$dir/worktree")"
if [ -z "$owner" ]; then
continue
fi
if [ ! -d "$owner" ]; then
# The Go module cache inside is deliberately read-only, and
# rm(1) cannot unlink a file out of a directory it may not
# write, so the tree has to be made writable first. And
# failing to tidy up is a housekeeping problem, never a
# reason to fail a lint: without the fallback below, `set
# -e` turns a stale cache that will not delete into a lint
# error, which is a gate failing for a reason that has
# nothing to do with the code. (Observed, not theorised.)
chmod -R u+w "$dir" 2>/dev/null || true
if ! rm -rf "$dir" 2>/dev/null; then
# Restore the marker on a partial removal: an
# unmarked leftover would be skipped by every future
# run and never collected.
mkdir -p "$dir" 2>/dev/null || true
echo "$owner" >"$dir/worktree" 2>/dev/null || true
echo "lint: could not remove stale cache $dir" >&2
fi
fi
done
}
prepare_cache() {
cache="$1"
mkdir -p "$cache/go-build" "$cache/go-mod" "$cache/golangci-lint"
echo "$ROOT" >"$cache/worktree"
}
# Run the linter, wherever it is that this script is allowed to run it.
run_linter() {
if in_lint_container; then
golangci-lint run "$@" ./...
return $?
fi
docker run --rm \
--user "$(id -u):$(id -g)" \
--env HOME=/tmp \
--env GOFLAGS=-buildvcs=false \
--env GOCACHE=/cache/go-build \
--env GOMODCACHE=/cache/go-mod \
--env GOLANGCI_LINT_CACHE=/cache/golangci-lint \
--volume "$ROOT:/src" \
--volume "$CACHE:/cache" \
--workdir /src \
"$IMAGE" \
golangci-lint run "$@" ./...
}
# Run the linter, streaming its combined output while also capturing it,
# and hand back its exit status. The output has to be inspected before
# it is believed, which is why this script no longer just execs the
# linter. `tee` would swallow the status, so it is smuggled out through
# a file: there is no pipefail in POSIX sh.
run_capture() {
capture="$1"
shift
rm -f "$capture.status"
{
rc=0
# `set -e` is in force inside this subshell too, so the status
# has to be caught here: an unguarded non-zero exit (which is
# what "the linter found something" looks like) would abort the
# subshell before the status was ever written.
run_linter "$@" 2>&1 || rc=$?
echo "$rc" >"$capture.status"
} | tee "$capture"
if [ ! -s "$capture.status" ]; then
echo "lint: the linter did not report an exit status" >&2
exit 1
fi
read -r captured_status <"$capture.status"
rm -f "$capture.status"
return "$captured_status"
}
# Reject output that cannot describe this tree. See script/lint-audit
# for what that means and why: in short, a finding citing a file that is
# not here means the result being reported was produced somewhere else,
# and a PASS built out of another checkout's analysis is silent (issue
# #99). The audit therefore runs on clean output as well.
audit_output() {
capture="$1"
if ! "$ROOT/script/lint-audit" "$capture"; then
if [ -n "$CACHE" ]; then
echo " this tree's lint cache: $CACHE" >&2
fi
exit 1
fi
script/lint takes no arguments. The linter runs as a build step, so
there is no command line to pass flags to; anything accepted here would
have to be silently dropped. To apply autofixes, use script/lint-fix,
which runs the same pinned image as a container for exactly this
reason.
EOF
exit 2
}
main() {
[ "$#" -eq 0 ] || usage
cd "$ROOT"
require_docker
IMAGE="$(lint_image)"
if [ -z "$IMAGE" ]; then
echo "lint: no lint stage found in $DOCKERFILE" >&2
exit 1
fi
# A fresh epoch per invocation is what forces the check layers to
# execute; the layers above the ARG in Dockerfile.lint still cache,
# so a run is not cold. The value must be unique per invocation, not
# per second: `date +%s` is second-granular, so two concurrent
# invocations in the same second would get identical epochs and the
# later one could be served from cache -- the false green in
# miniature. `%N` alone does not fix it either, because busybox
# silently drops %N, exits 0, and hands back second granularity with
# no warning. `$$` is what makes this correct regardless, since
# concurrent invocations have different pids.
#
# Assign it on its own line rather than inline in the argument.
# Under `set -eu` a command substitution that fails inside an
# argument does NOT abort the script: CHECK_EPOCH would become an
# empty string, an empty string is a constant, and a constant epoch
# is exactly the cached-lint false green this guards against. As a
# bare assignment, `set -e` catches a failing `date` and no build
# starts.
epoch="$(date +%s%N)$$"
CACHE=""
if in_lint_container; then
# No docker daemon in here, so there is no fallback: a mismatch
# is a hard error rather than a quiet substitution.
installed="$(installed_version)"
pinned="$(pinned_version)"
if [ -z "$installed" ] || [ "$installed" != "$pinned" ]; then
cat >&2 <<EOF
lint: VAULTIK_LINT_IN_CONTAINER is set, so this is expected to be
running inside the Dockerfile's pinned lint image, but the golangci-lint
on PATH does not match the pin.
pinned: $pinned ($IMAGE)
installed: ${installed:-<none>}
EOF
exit 1
fi
else
require_docker "$IMAGE"
prune_dead_caches
CACHE="$(cache_dir)"
prepare_cache "$CACHE"
fi
capture="$(mktemp "${TMPDIR:-/tmp}/vaultik-lint.XXXXXX")"
trap 'rm -f "$capture" "$capture.status"' EXIT HUP INT TERM
attempt=1
while :; do
status=0
run_capture "$capture" "$@" || status=$?
if grep -Fq "$LOCK_MESSAGE" "$capture"; then
if [ "$attempt" -lt "$LOCK_ATTEMPTS" ]; then
echo "lint: another golangci-lint holds the lock;" \
"retrying in ${LOCK_SLEEP}s" \
"(attempt $attempt of $LOCK_ATTEMPTS)" >&2
sleep "$LOCK_SLEEP"
attempt=$((attempt + 1))
continue
fi
cat >&2 <<EOF
lint: gave up after $LOCK_ATTEMPTS attempts, each blocked by another
golangci-lint holding the cross-process lock. This is NOT a lint
verdict: the tree was never analysed. Re-run when the other run has
finished.
EOF
exit 1
fi
audit_output "$capture"
exit "$status"
done
# cacheonly: the lint verdict is the build's exit status, and the
# image it would otherwise produce is never run. Exporting it costs
# most of the wall time of a warm run and leaves a dangling image
# behind on every invocation, on a host that may be running many.
docker build \
--output=type=cacheonly \
--build-arg CHECK_EPOCH="$epoch" \
-f "$DOCKERFILE" \
"$ROOT"
}
main "$@"
-113
View File
@@ -1,113 +0,0 @@
#!/bin/sh
# script/lint-audit: audit a captured golangci-lint run for output that
# cannot describe this tree. Called by script/lint on every run; usable
# on its own against any saved lint output.
#
# script/lint-audit <capture-file>
#
# Exits 0 when every finding cites a file in this tree, 1 when any does
# not. It NEVER certifies that a lint run passed - it has no idea
# whether the run found issues, and does not look. It only rejects
# output that is impossible for this tree, which is a different and much
# weaker claim. Do not use it as a gate; use script/lint.
#
# Why this exists (issue #99): golangci-lint caches analysis results,
# and a cache shared between two checkouts of this repo can serve one
# checkout's stored findings for another, file paths included. The
# failure is symmetric and only one direction is loud - a clean tree
# failed by a dirty sibling gets investigated, while a dirty tree passed
# by a clean sibling is silent. This turns the silent direction into a
# hard error, which is why it runs on clean output too.
#
# The primary fix is that script/lint now keys its cache on the worktree
# path so the collision cannot happen. This is the backstop, because a
# backstop that only runs when we already believe things are fine is
# worth more than one more assumption.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Where script/lint bind-mounts the tree inside the pinned image. A
# containerized run that prints absolute paths (`--path-mode abs`)
# prints them under this, so they are this tree's files under another
# name. Note the consequence, and why the cache key rather than this
# check is the real fix: two containerized runs of different checkouts
# both call themselves /src, so contamination between two container
# runs is not distinguishable by path alone.
CONTAINER_ROOT="/src"
usage() {
echo "usage: $(basename "$0") <capture-file>" >&2
exit 2
}
# Every path cited by a finding that is not a file in this tree.
#
# The linter runs with the tree root as its working directory, so a
# legitimate finding cites either a relative path that resolves inside
# the tree or an absolute path under the root. A path that escapes
# (absolute and elsewhere, or with a `..` component) or that names a
# file which is not here describes something this run did not analyse.
foreign_paths() {
capture="$1"
awk -F: '$1 ~ /\.go$/ && $2 ~ /^[0-9]+$/ { print $1 }' "$capture" |
sort -u |
while IFS= read -r path; do
case "$path" in
"$ROOT"/*)
path="${path#"$ROOT"/}"
;;
"$CONTAINER_ROOT"/*)
path="${path#"$CONTAINER_ROOT"/}"
;;
/*)
printf '%s\n' "$path"
continue
;;
../* | */../*)
printf '%s\n' "$path"
continue
;;
esac
if [ ! -e "$ROOT/$path" ]; then
printf '%s\n' "$path"
fi
done
}
main() {
[ "$#" -eq 1 ] || usage
capture="$1"
if [ ! -f "$capture" ]; then
echo "lint-audit: no such capture file: $capture" >&2
exit 2
fi
foreign="$(foreign_paths "$capture")"
if [ -z "$foreign" ]; then
exit 0
fi
cat >&2 <<EOF
lint: REJECTED - the linter reported findings for files that are not in
this tree, so its output does not describe the tree that was linted.
This result is void, whichever way it went: a pass here would be a pass
earned by analysing someone else's code.
tree: $ROOT
Paths reported that are not in this tree:
EOF
printf '%s\n' "$foreign" | sed -e 's/^/ /' >&2
cat >&2 <<EOF
This is the signature of analysis replayed from a cache belonging to
another checkout (issue #99). Clear this tree's lint cache and re-run:
rm -rf "\${XDG_CACHE_HOME:-\$HOME/.cache}/vaultik-lint"
EOF
exit 1
}
main "$@"
+43 -7
View File
@@ -1,18 +1,54 @@
#!/bin/sh
# script/lint-fix: run the linter's autofixer. Rewrites files in place
# for every finding the enabled linters know how to fix; findings
# without an autofix are reported but left alone (exit status is
# nonzero while any remain).
# without an autofix are reported but left alone.
#
# Delegates to script/lint so the autofixer is the same pinned linter
# version that script/lint and CI use - fixes written by a different
# version are not necessarily fixes for the version that gates.
# THIS IS A DEVELOPER CONVENIENCE AND NEVER A GATE. Nothing in
# script/check, script/precommit or script/cibuild calls it, and no gate
# reads its exit status. The gate is script/lint, which builds
# Dockerfile.lint; run that afterwards to find out whether the tree is
# actually clean.
#
# Unlike script/lint this cannot be a build step: a build step writes
# into an image, and fixes have to land in the worktree. So it runs the
# same pinned image as a container with the tree bind-mounted, which
# means it needs a LOCAL docker daemon -- a remote daemon has no access
# to these files, and this script will appear to do nothing there. The
# image reference is parsed out of Dockerfile.lint's FROM line, so the
# autofixer is always the same version as the linter that gates; fixes
# written by a different version are not necessarily fixes for the
# version that decides.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
DOCKERFILE="$ROOT/Dockerfile.lint"
# The image reference from Dockerfile.lint, tag and digest included.
lint_image() {
awk '$1 == "FROM" { print $2; exit }' "$DOCKERFILE"
}
main() {
exec "$SCRIPT_DIR/lint" --fix "$@"
cd "$ROOT"
image="$(lint_image)"
if [ -z "$image" ]; then
echo "lint-fix: no FROM line found in $DOCKERFILE" >&2
exit 1
fi
# Run as the invoking user so the rewritten files stay owned by
# them. HOME is set because the Go and golangci-lint caches default
# under it and that user has no home inside the container; those
# caches are per-container and discarded with it.
docker run --rm \
--user "$(id -u):$(id -g)" \
--env HOME=/tmp \
--env GOFLAGS=-buildvcs=false \
--volume "$ROOT:/src" \
--workdir /src \
"$image" \
golangci-lint run --config .golangci.yml --fix "$@" ./...
}
main "$@"
-27
View File
@@ -1,27 +0,0 @@
# Vaultik test configuration
hostname: test-host
index_path: /tmp/vaultik-test/index.db
source_dirs:
- /tmp/vaultik-test/source
# S3 configuration
s3:
endpoint: http://localhost:19000 # gofakes3 test endpoint
bucket: test-bucket
prefix: test-
access_key_id: test-key
secret_access_key: test-secret
region: us-east-1
# Chunking configuration
chunk_size: 65536 # 64KB average chunk size
min_chunk_size: 32768 # 32KB minimum
max_chunk_size: 131072 # 128KB maximum
blob_size: 1048576 # 1MB blobs for testing
# Compression
compression_level: 3
# Encryption
# age_recipients:
# - age1qyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqs3mw88h
-24
View File
@@ -1,24 +0,0 @@
age_recipients:
- age1278m9q7dp3chsh2dcy82qk27v047zywyvtxwnj4cvt0z65jw6a7q5dqhfj # sneak's long term age key
- age1ezrjmfpwsc95svdg0y54mums3zevgzu0x0ecq2f7tp8a05gl0sjq9q9wjg # insecure integration test key
source_dirs:
- /tmp/vaultik-test-source
exclude:
- '*.log'
- '*.tmp'
- '.git'
- 'node_modules'
s3:
endpoint: http://ber1app1.local:3900/
bucket: vaultik-integration-test
prefix: test-host/
access_key_id: GKbc8e6d35fdf50847f155aca5
secret_access_key: 217046bee47c050301e3cc13e3cba1a8a943cf5f37f8c7979c349c5254441d18
region: us-east-1
use_ssl: false
part_size: 5242880 # 5MB
index_path: /tmp/vaultik-integration-test.sqlite
chunk_size: 10MB
blob_size_limit: 10GB
compression_level: 3
hostname: test-host