10 Commits

Author SHA1 Message Date
2aaeeb4966 Refresh vendored REPO_POLICIES.md
All checks were successful
check / check (push) Successful in 4s
2026-07-07 01:53:18 +02:00
43346e62db Adopt scripts-to-rule-them-all: script/ entrypoints, Makefile shims 2026-07-07 01:53:18 +02:00
df975bb8f0 Add standard Workflow section to TODO.md
All checks were successful
check / check (push) Successful in 6s
2026-07-06 21:06:38 +02:00
fc56b0cb30 Add TODO.md
All checks were successful
check / check (push) Successful in 6s
2026-07-06 20:35:44 +02:00
1f32820607 Merge branch 'refactor/cli-verb-consolidation'
All checks were successful
check / check (push) Successful in 1m59s
2026-07-02 16:42:24 +02:00
34a4d163f2 Consolidate CLI verbs; retire overlapping commands
The verb surface accumulated overlapping cleanup commands. Consolidate
so each cleanup verb has one meaning:

- Rename 'database purge' -> 'database delete'. The command removes the
  SQLite file entirely; "purge" wrongly suggested purging contents.
- Fold 'snapshot cleanup' into 'prune'. Prune now runs three passes:
  reconcile local snapshots against the remote (previously the
  standalone cleanup command), drop orphaned local rows, then delete
  unreferenced remote blobs. One command, one mental model.
- Delete 'store info'. Its output was a strict subset of 'remote info',
  which already prints storage type + location. Any user reaching for
  either should reach for 'remote info'.
- Drop 'snapshot remove --all'. It duplicated 'remote nuke --force'.
  'remote nuke' is the single supported entry point for wiping the
  destination store.

Also update the storage-binding error message to reference the new
'vaultik database delete' name.
2026-07-02 16:42:20 +02:00
9497a31d0f Merge branch 'fix/bind-local-index-to-storage-url'
All checks were successful
check / check (push) Successful in 2m18s
2026-07-02 16:35:44 +02:00
d330f9f031 Bind the local index to its backup destination
The local index tracks which chunks and blobs already exist at the
backup destination. Nothing was recording *which* destination, so
changing storage_url and running a backup left the scanner treating
every already-seen chunk as still-present at the new (empty) location.
Uploads were skipped silently and the resulting snapshots pointed at
blobs that don't exist at the new destination.

Fix: record storage_url in a new local_meta key-value table on first
mutating command, and refuse to proceed when the configured URL later
differs from the stored one. The error explains the two recovery
paths (revert the config, or run 'vaultik database purge' to discard
the index and rebuild from a fresh full backup).

Wired into snapshot create / prune / snapshot remove / snapshot purge
/ snapshot cleanup. Read-only inspection commands (snapshot list,
remote info, store info) are exempt.
2026-07-02 16:35:41 +02:00
fda6d7a7eb Merge branch 'fix/restore-skip-chown-non-root'
All checks were successful
check / check (push) Successful in 2m0s
2026-06-28 07:48:32 +02:00
1a97a80a81 Skip chown when restore runs as non-root; warn at end
chown(2) requires root on every Unix-ish kernel. Restoring 39k files
as a non-root user produces 39k EPERM syscalls plus 39k matching debug
log lines, all for an operation that can't possibly succeed. Skip the
syscall entirely when euid != 0, and emit one warning at the end of the
restore so the user knows the on-disk UID/GID will reflect the running
user rather than the original owner.
2026-06-28 07:48:28 +02:00
33 changed files with 1134 additions and 298 deletions

View File

@@ -11,4 +11,4 @@ jobs:
# actions/checkout v4, 2024-09-16 # actions/checkout v4, 2024-09-16
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- name: Build and check - name: Build and check
run: docker build . run: script/cibuild

View File

@@ -1,4 +1,4 @@
.PHONY: all check test lint fmt fmt-check build clean deps test-coverage test-integration local install release release-snapshot docker hooks .PHONY: all bootstrap setup check test lint fmt fmt-check build clean deps test-coverage test-integration local install release release-snapshot docker hooks
# Version number # Version number
VERSION := 1.0.0-rc.1 VERSION := 1.0.0-rc.1
@@ -15,24 +15,33 @@ LDFLAGS := -X 'sneak.berlin/go/vaultik/internal/globals.Version=$(VERSION)' \
# Default target # Default target
all: vaultik all: vaultik
# Combined pre-commit/CI gate: lint, format check, then tests. # Install all development dependencies.
check: lint fmt-check test bootstrap:
@script/bootstrap
# Prepare a fresh clone: bootstrap plus pre-commit hook.
setup:
@script/setup
# Combined pre-commit/CI gate: tests, lint, format check.
check:
@script/check
# Run tests only. # Run tests only.
test: test:
go test -race -timeout 30s ./... @script/test
# Check if code is formatted (read-only). # Check if code is formatted (read-only).
fmt-check: fmt-check:
@test -z "$$(gofmt -l .)" || (echo "Files not formatted:" && gofmt -l . && exit 1) @script/fmt-check
# Format code. # Format code.
fmt: fmt:
go fmt ./... @script/fmt
# Run linter only. # Run linter only.
lint: lint:
golangci-lint run ./... @script/lint
# Build binary. # Build binary.
vaultik: internal/*/*.go cmd/vaultik/*.go vaultik: internal/*/*.go cmd/vaultik/*.go
@@ -74,11 +83,8 @@ release-snapshot:
# Build Docker image. # Build Docker image.
docker: docker:
docker build -t vaultik . @script/docker
# Install pre-commit hook. # Install pre-commit hook.
hooks: hooks:
@printf '#!/bin/sh\nset -e\n' > .git/hooks/pre-commit @script/install-precommit
@printf 'go mod tidy\ngo fmt ./...\ngit diff --exit-code -- go.mod go.sum || { echo "go mod tidy changed files; please stage and retry"; exit 1; }\n' >> .git/hooks/pre-commit
@printf 'make check\n' >> .git/hooks/pre-commit
@chmod +x .git/hooks/pre-commit

View File

@@ -99,15 +99,13 @@ vaultik [--config <path>] snapshot create [snapshot-names...] [--cron] [--prune]
vaultik [--config <path>] snapshot list [--json] vaultik [--config <path>] snapshot list [--json]
vaultik [--config <path>] snapshot verify <snapshot-id> [--deep] [--json] vaultik [--config <path>] snapshot verify <snapshot-id> [--deep] [--json]
vaultik [--config <path>] snapshot purge [--keep-latest | --older-than <duration>] [--snapshot <name>...] [--force] vaultik [--config <path>] snapshot purge [--keep-latest | --older-than <duration>] [--snapshot <name>...] [--force]
vaultik [--config <path>] snapshot remove <snapshot-id|--all> [--dry-run] [--force] [--local-only] [--json] vaultik [--config <path>] snapshot remove <snapshot-id> [--dry-run] [--force] [--local-only] [--json]
vaultik [--config <path>] snapshot cleanup
vaultik [--config <path>] snapshot restore <snapshot-id> <target-dir> [paths...] [--verify] vaultik [--config <path>] snapshot restore <snapshot-id> <target-dir> [paths...] [--verify]
vaultik [--config <path>] prune [--force] [--json] vaultik [--config <path>] prune [--force] [--json]
vaultik [--config <path>] info vaultik [--config <path>] info
vaultik [--config <path>] remote info [--json] vaultik [--config <path>] remote info [--json]
vaultik [--config <path>] remote nuke --force vaultik [--config <path>] remote nuke --force
vaultik [--config <path>] store info vaultik [--config <path>] database delete [--force]
vaultik [--config <path>] database purge [--force]
vaultik completion <bash|zsh|fish|powershell> vaultik completion <bash|zsh|fish|powershell>
vaultik version vaultik version
``` ```
@@ -198,7 +196,7 @@ latest globally).
* `--snapshot <name>`: Restrict to specific snapshot names (repeat for multiple) * `--snapshot <name>`: Restrict to specific snapshot names (repeat for multiple)
* `--force`: Skip confirmation prompt * `--force`: Skip confirmation prompt
**`snapshot remove`**: Remove a snapshot. By default this removes the **`snapshot remove`**: Remove one snapshot. By default this removes the
snapshot from the local index and strips the snapshot's metadata from snapshot from the local index and strips the snapshot's metadata from
the backup destination store. Blobs are NOT touched — deleting blobs the backup destination store. Blobs are NOT touched — deleting blobs
requires reading every remaining remote manifest (the destination store requires reading every remaining remote manifest (the destination store
@@ -208,17 +206,13 @@ prune` invocation to run as a follow-up. Local row cleanup (files,
chunks, blobs the snapshot was the last referrer for) runs chunks, blobs the snapshot was the last referrer for) runs
automatically. If the destination store is unreachable, the local-DB automatically. If the destination store is unreachable, the local-DB
removal still completes and a warning is emitted; rerun `vaultik prune` removal still completes and a warning is emitted; rerun `vaultik prune`
once the store is reachable to finish remote cleanup. once the store is reachable to finish remote cleanup. To wipe everything
on the destination in one go, use `vaultik remote nuke --force`.
* `--local-only`: Skip remote cleanup; only touch the local index * `--local-only`: Skip remote cleanup; only touch the local index
* `--all`: Remove all snapshots (requires `--force`)
* `--dry-run`: Show what would be deleted without deleting * `--dry-run`: Show what would be deleted without deleting
* `--force`: Skip confirmation prompt * `--force`: Skip confirmation prompt
* `--json`: Output result as JSON * `--json`: Output result as JSON
**`snapshot cleanup`**: Remove stale local snapshot records that have no
corresponding metadata in remote storage. These are typically left behind
by incomplete or interrupted backups. Does not touch remote storage.
**`snapshot restore`**: Restore files from a backup snapshot. **`snapshot restore`**: Restore files from a backup snapshot.
* Requires `VAULTIK_AGE_SECRET_KEY` environment variable * Requires `VAULTIK_AGE_SECRET_KEY` environment variable
* Optional path arguments to restore specific files/directories (default: all) * Optional path arguments to restore specific files/directories (default: all)
@@ -226,31 +220,38 @@ by incomplete or interrupted backups. Does not touch remote storage.
symlinks, and empty directories symlinks, and empty directories
* `--verify`: After restoring, verify every file's chunk hashes match * `--verify`: After restoring, verify every file's chunk hashes match
**`prune`**: Tidy up everything that isn't needed. Removes orphaned local **`prune`**: Tidy up everything that isn't needed. Runs three passes:
database rows (files, chunks, blobs no longer referenced by any completed (1) reconcile the local index against the destination store — any
snapshot) AND deletes unreferenced blobs from remote storage. `snapshot local snapshot whose remote metadata is missing is dropped from the
create --prune`, `snapshot remove`, and `snapshot purge` run the same local index; (2) delete orphaned local rows (files, chunks, blobs no
cleanup automatically; this is the manual entry point for the same work. longer referenced by any completed snapshot); (3) list every remote
manifest on the destination store to compute the still-referenced blob
set and delete any blob not in that set. Step (3) reads all remote
manifests — network cost scales with the number of snapshots. `snapshot
create --prune` runs the same cleanup automatically; this is the
manual entry point for the same work.
* `--force`: Skip confirmation prompt * `--force`: Skip confirmation prompt
* `--json`: Output stats as JSON * `--json`: Output stats as JSON
**`info`**: Display system configuration, storage settings, encryption **`info`**: Display system configuration, storage settings, encryption
recipients, and local database statistics. recipients, and local database statistics.
**`remote info`**: Show detailed remote storage information including per-snapshot **`remote info`**: Show storage backend type and location plus detailed
metadata sizes, blob counts, and orphaned blob detection. remote storage inventory: per-snapshot metadata sizes, blob counts, and
orphaned blob detection.
* `--json`: Output as JSON * `--json`: Output as JSON
**`remote nuke`**: Delete every snapshot's metadata and every blob from the **`remote nuke`**: Delete every snapshot's metadata and every blob from the
backup destination store, leaving the bucket prefix empty. Destructive and backup destination store, leaving the bucket prefix empty. Destructive and
irreversible. irreversible. This is the single supported way to wipe the entire
destination store.
* `--force`: Required to confirm destruction. * `--force`: Required to confirm destruction.
**`store info`**: Display storage backend type and statistics. **`database delete`**: Delete the local SQLite state database file
entirely. Remote storage is unaffected; the next backup will do a full
**`database purge`**: Delete the local SQLite state database entirely. Remote scan and re-deduplicate against existing remote blobs, and the local
storage is unaffected; the next backup will do a full scan and re-deduplicate index will re-bind to the currently configured storage destination.
against existing remote blobs. Use this after changing `storage_url` to a different destination.
* `--force`: Skip confirmation prompt * `--force`: Skip confirmation prompt
--- ---
@@ -387,7 +388,7 @@ Key fields:
* **Device nodes, named pipes, and sockets are silently skipped.** Only * **Device nodes, named pipes, and sockets are silently skipped.** Only
regular files, directories, and symlinks are backed up. regular files, directories, and symlinks are backed up.
* **No database migrations.** If the local SQLite schema changes between * **No database migrations.** If the local SQLite schema changes between
versions, delete the local database (`vaultik database purge`) and run versions, delete the local database (`vaultik database delete`) and run
a full backup. Remote storage is unaffected. a full backup. Remote storage is unaffected.
* **Files that change during backup may be inconsistent.** There is no * **Files that change during backup may be inconsistent.** There is no
filesystem snapshot or freeze. If a file is modified between the scan filesystem snapshot or freeze. If a file is modified between the scan
@@ -459,7 +460,7 @@ priority.
first-class operation in this README. Worth a dedicated section first-class operation in this README. Worth a dedicated section
once it's settled. once it's settled.
* **Schema migrations.** Currently nonexistent — pre-1.0 schema * **Schema migrations.** Currently nonexistent — pre-1.0 schema
changes are handled by `vaultik database purge` plus a full changes are handled by `vaultik database delete` plus a full
re-scan. Post-1.0 we'll need a migration story to keep existing re-scan. Post-1.0 we'll need a migration story to keep existing
index databases usable across upgrades. index databases usable across upgrades.
* **Storage backend coverage tests.** S3, file://, and rclone:// * **Storage backend coverage tests.** S3, file://, and rclone://
@@ -555,6 +556,35 @@ Do not commit directly to `main`. Do not skip steps.
Repository policies for AI agents are in [`AGENTS.md`](AGENTS.md). Repository policies for AI agents are in [`AGENTS.md`](AGENTS.md).
## Entrypoints
This repository adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard: normalized scripts in `script/` are the entrypoints for the
development workflow, and the Makefile targets are thin shims that call
them. We provide:
* `script/bootstrap` — install all development dependencies (go,
golangci-lint, Go module download)
* `script/setup` — make a fresh clone ready for development: runs
`script/bootstrap`, then `script/install-precommit`
* `script/projectname` — print the project name (used for the Docker
image tag)
* `script/test` — run the test suite (verbose rerun on failure)
* `script/lint` — run `golangci-lint run ./...`
* `script/fmt` — format all code (writes)
* `script/fmt-check` — check formatting (read-only)
* `script/check` — run `script/test`, `script/lint`, and
`script/fmt-check`
* `script/docker` — build the Docker image tagged via
`script/projectname`
* `script/cibuild` — CI entrypoint: `docker build .` (the Dockerfile
runs the checks)
* `script/precommit` — pre-commit gate: `go mod tidy` + `go fmt` (must
not change files), then `script/check`
* `script/install-precommit` — install the git pre-commit hook that
runs `script/precommit`
## license ## license
[MIT](https://opensource.org/license/mit/) [MIT](https://opensource.org/license/mit/)

408
REPO_POLICIES.md Normal file
View File

@@ -0,0 +1,408 @@
---
title: Repository Policies
last_modified: 2026-07-06
---
This document covers repository structure, tooling, and workflow standards. Code
style conventions are in separate documents:
- [Code Styleguide](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/CODE_STYLEGUIDE.md)
(general, bash, Docker)
- [Go](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/CODE_STYLEGUIDE_GO.md)
- [JavaScript](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/CODE_STYLEGUIDE_JS.md)
- [Python](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/CODE_STYLEGUIDE_PYTHON.md)
- [Go HTTP Server Conventions](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/GO_HTTP_SERVER_CONVENTIONS.md)
---
- Cross-project documentation (such as this file) must include
`last_modified: YYYY-MM-DD` in the YAML front matter so it can be kept in sync
with the authoritative source as policies evolve.
- **ALL external references must be pinned by cryptographic hash.** This
includes Docker base images, Go modules, npm packages, GitHub Actions, and
anything else fetched from a remote source. Version tags (`@v4`, `@latest`,
`:3.21`, etc.) are server-mutable and therefore remote code execution
vulnerabilities. The ONLY acceptable way to reference an external dependency
is by its content hash (Docker `@sha256:...`, Go module hash in `go.sum`, npm
integrity hash in lockfile, GitHub Actions `@<commit-sha>`). No exceptions.
This also means never `curl | bash` to install tools like pyenv, nvm, rustup,
etc. Instead, download a specific release archive from GitHub, verify its hash
(hardcoded in the Dockerfile or script), and only then install. Unverified
install scripts are arbitrary remote code execution. This is the single most
important rule in this document. Double-check every external reference in
every file before committing. There are zero exceptions to this rule.
- Every repo with software must have a root `Makefile` with these targets:
`make bootstrap`, `make setup`, `make test`, `make lint`, `make fmt` (writes),
`make fmt-check` (read-only), `make check` (runs `test`, `lint`, `fmt-check`),
`make docker`, and `make hooks` (installs pre-commit hook). A model Makefile
is at `https://git.eeqj.de/sneak/prompts/raw/branch/main/Makefile`.
- Repos follow the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
pattern: the implementation of each Makefile target lives in an executable
script in `script/` (`script/bootstrap`, `script/setup`, `script/test`,
`script/lint`, `script/fmt`, `script/fmt-check`, `script/check`,
`script/docker`), and the Makefile targets are thin shims that call them. The
scripts must be POSIX sh (`#!/bin/sh`, `set -eu`, no bashisms) so they run in
minimal containers (e.g. alpine images have no bash); locate the repo root
with `$(cd "$(dirname "$0")/.." && pwd -P)` and `cd` there before acting. From
the standard's canonical set we use `bootstrap`, `setup` (make the repo ready
for development after a fresh clone: runs `bootstrap`, then
`install-precommit`, plus any repo-specific initialization), `test`, and
`cibuild`. `script/bootstrap` installs all dependencies idempotently and
assumes nothing is present: base tools come from nix, apt, brew, or apk
(detected in that order; apt runs noninteractive). For node it uses the
installed node if present; otherwise it installs a PINNED node version via
nvm, first installing nvm itself if missing — from a hash-verified GitHub
release archive (never `curl | sh`), with bash installed as an explicit
prerequisite since nvm requires bash. yarn is then pinned via
`corepack prepare yarn@<version> --activate`. Never install "latest" or "lts";
always exact versions. `script/cibuild` runs the CI build: it changes to the
repo root and runs `docker build .`; the Gitea workflow calls it. Four further
scripts are our own extensions to the standard: `script/check` runs
`script/test`, `script/lint`, and `script/fmt-check`; `script/precommit` is
what the git pre-commit hook runs, and it calls `script/check`;
`script/install-precommit` installs the git pre-commit hook (the `make hooks`
target shims to it); and `script/projectname` (literally that filename) simply
outputs the project's name. Scripts that need the name call
`script/projectname` — e.g. `script/docker` assembles its image tag from it —
so those scripts stay byte-identical across all repos. Repo-type-specific
pre-commit extras (e.g. `go mod tidy` verification in Go repos) belong in
`script/precommit`, not in the hook itself. Model scripts are at
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
must document the provided scripts in an **Entrypoints** section (see the
README requirements below).
- Always use Makefile targets (`make fmt`, `make test`, `make lint`, etc.)
instead of invoking the underlying tools directly. The Makefile is the single
source of truth for how these operations are run.
- The Makefile is authoritative documentation for how the repo is used. Beyond
the required targets above, it should have targets for every common operation:
running a local development server (`make run`, `make dev`), re-initializing
or migrating the database (`make db-reset`, `make migrate`), building
artifacts (`make build`), generating code, seeding data, or anything else a
developer would do regularly. If someone checks out the repo and types
`make<tab>`, they should see every meaningful operation available. A new
contributor should be able to understand the entire development workflow by
reading the Makefile.
- Every repo should have a `Dockerfile`. All Dockerfiles must run `make check`
as a build step so the build fails if the branch is not green. For non-server
repos, the Dockerfile should bring up a development environment and run
`make check`. For server repos, `make check` should run as an early build
stage before the final image is assembled. Dockerfiles install development
prerequisites by running `script/bootstrap` rather than duplicating installs
inline; COPY `script/` and the dependency manifests (`package.json` +
`yarn.lock`, `go.mod` + `go.sum`, etc.) before running it so the bootstrap
layer stays cached until dependencies change.
- **Dockerfiles must use a separate lint stage for fail-fast feedback.** Go
repos use a multistage build where linting runs in an independent stage based
on the `golangci/golangci-lint` image (pinned by hash). This stage runs
`make fmt-check` and `make lint` before the full build begins. The build stage
then declares an explicit dependency on the lint stage via
`COPY --from=lint /src/go.sum /dev/null`, which forces BuildKit to complete
linting before proceeding to compilation and tests. This ensures lint failures
surface in seconds rather than minutes, without blocking on dependency
download or compilation in the build stage.
The standard pattern for a Go repo Dockerfile is:
```dockerfile
# Lint stage — fast feedback on formatting and lint issues
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD
FROM golangci/golangci-lint@sha256:... AS lint
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN make fmt-check
RUN make lint
# Build stage
# golang:1.x-alpine, YYYY-MM-DD
FROM golang@sha256:... AS builder
WORKDIR /src
# Force BuildKit to run the lint stage before proceeding
COPY --from=lint /src/go.sum /dev/null
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN make test
ARG VERSION=dev
RUN CGO_ENABLED=0 go build -trimpath \
-ldflags="-s -w -X main.Version=${VERSION}" \
-o /app ./cmd/app/
# Runtime stage
FROM alpine@sha256:...
COPY --from=builder /app /usr/local/bin/app
ENTRYPOINT ["app"]
```
Key points:
- The lint stage uses the `golangci/golangci-lint` image directly (it
includes both Go and the linter), so there is no need to install the
linter separately.
- `COPY --from=lint /src/go.sum /dev/null` is a no-op file copy that creates
a stage dependency. BuildKit runs stages in parallel by default; without
this line, the build stage would not wait for lint to finish and a lint
failure might not fail the overall build.
- If the project uses `//go:embed` directives that reference build artifacts
(e.g. a web frontend compiled in a separate stage), the lint stage must
create placeholder files so the embed directives resolve. Example:
`RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`.
The lint stage should not depend on the actual build output — it exists to
fail fast.
- If the project requires CGO or system libraries for linting (e.g.
`vips-dev`), install them in the lint stage with `apk add`.
- The build stage runs `make test` after compilation setup. Tests run in the
build stage, not the lint stage, because they may require compiled
artifacts or heavier dependencies.
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
runs `script/cibuild` (which runs `docker build .`) on push. Since the
Dockerfile already runs `make check`, a successful build implies all checks
pass.
- Use platform-standard formatters: `black` for Python, `prettier` for
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
two exceptions: four-space indents (except Go), and `proseWrap: always` for
Markdown (hard-wrap at 80 columns). Documentation and writing repos (Markdown,
HTML, CSS) should also have `.prettierrc` and `.prettierignore`.
- Pre-commit hook: runs `script/precommit`, which calls `script/check`. If local
testing is not possible in the repo, `script/precommit` may skip `script/test`
and run only `script/lint` and `script/fmt-check`. The hook is installed by
`script/install-precommit`; the Makefile must provide a `make hooks` target
that shims to it.
- All repos with software must have tests that run via the platform-standard
test framework (`go test`, `pytest`, `jest`/`vitest`, etc.). If no meaningful
tests exist yet, add the most minimal test possible — e.g. importing the
module under test to verify it compiles/parses. There is no excuse for
`make test` to be a no-op.
- `make test` must complete in under 20 seconds. Add a 30-second timeout in the
Makefile.
- **`make test` should use the conditional verbose rerun pattern.** Run tests
without `-v` (verbose) first. If tests fail, automatically rerun with `-v` to
show full output. This keeps CI logs and `docker build` output clean on
success (just package/suite summaries) while providing full diagnostic detail
on failure (every test case, every assertion). The general shell pattern:
```makefile
test:
@<test-command> || \
{ echo "--- Rerunning with -v for details ---"; \
<test-command-with-v>; exit 1; }
```
Go example:
```makefile
test:
@go test -timeout 30s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -timeout 30s -race -v ./...; exit 1; }
```
Python example:
```makefile
test:
@python -m pytest || \
{ echo "--- Rerunning with -v for details ---"; \
python -m pytest -v; exit 1; }
```
The `exit 1` ensures the target always fails after a rerun — the first run
already proved the tests are broken, so the build must not pass even if a
flaky test happens to succeed on the second attempt. The rerun exists solely
for diagnostic output.
- Docker builds must complete in under 5 minutes.
- `make check` must not modify any files in the repo. Tests may use temporary
directories.
- `main` must always pass `make check`, no exceptions.
- Never commit secrets. `.env` files, credentials, API keys, and private keys
must be in `.gitignore`. No exceptions.
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
editor files (`.swp`, `*~`), language build artifacts, and `node_modules/`.
Fetch the standard `.gitignore` from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when setting up
a new repo.
- **No build artifacts in version control.** Code-derived data (compiled
bundles, minified output, generated assets) must never be committed to the
repository if it can be avoided. The build process (e.g. Dockerfile, Makefile)
should generate these at build time. Notable exception: Go protobuf generated
files (`.pb.go`) ARE committed because repos need to work with `go get`, which
downloads code but does not execute code generation.
- Never use `git add -A` or `git add .`. Always stage files explicitly by name.
- Never force-push to `main`.
- Make all changes on a feature branch. You can do whatever you want on a
feature branch.
- `.golangci.yml` is standardized and must _NEVER_ be modified by an agent, only
manually by the user. Fetch from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml`.
- When pinning images or packages by hash, add a comment above the reference
with the version and date (YYYY-MM-DD).
- Use `yarn`, not `npm`.
- Write all dates as YYYY-MM-DD (ISO 8601).
- Simple projects should be configured with environment variables.
- Dockerized web services listen on port 8080 by default, overridable with
`PORT`.
- **HTTP/web services must be hardened for production internet exposure before
tagging 1.0.** This means full compliance with security best practices
including, without limitation, all of the following:
- **Security headers** on every response:
- `Strict-Transport-Security` (HSTS) with `max-age` of at least one year
and `includeSubDomains`.
- `Content-Security-Policy` (CSP) with a restrictive default policy
(`default-src 'self'` as a baseline, tightened per-resource as
needed). Never use `unsafe-inline` or `unsafe-eval` unless
unavoidable, and document the reason.
- `X-Frame-Options: DENY` (or `SAMEORIGIN` if framing is required).
Prefer the `frame-ancestors` CSP directive as the primary control.
- `X-Content-Type-Options: nosniff`.
- `Referrer-Policy: strict-origin-when-cross-origin` (or stricter).
- `Permissions-Policy` restricting access to browser features the
application does not use (camera, microphone, geolocation, etc.).
- **Request and response limits:**
- Maximum request body size enforced on all endpoints (e.g. Go
`http.MaxBytesReader`). Choose a sane default per-route; never accept
unbounded input.
- Maximum response body size where applicable (e.g. paginated APIs).
- `ReadTimeout` and `ReadHeaderTimeout` on the `http.Server` to defend
against slowloris attacks.
- `WriteTimeout` on the `http.Server`.
- `IdleTimeout` on the `http.Server`.
- Per-handler execution time limits via `context.WithTimeout` or
chi/stdlib `middleware.Timeout`.
- **Authentication and session security:**
- Rate limiting on password-based authentication endpoints. API keys are
high-entropy and not susceptible to brute force, so they are exempt.
- CSRF tokens on all state-mutating HTML forms. API endpoints
authenticated via `Authorization` header (Bearer token, API key) are
exempt because the browser does not attach these automatically.
- Passwords stored using bcrypt, scrypt, or argon2 — never plain-text,
MD5, or SHA.
- Session cookies set with `HttpOnly`, `Secure`, and `SameSite=Lax` (or
`Strict`) attributes.
- **Reverse proxy awareness:**
- True client IP detection when behind a reverse proxy
(`X-Forwarded-For`, `X-Real-IP`). The application must accept
forwarded headers only from a configured set of trusted proxy
addresses — never trust `X-Forwarded-For` unconditionally.
- **CORS:**
- Authenticated endpoints must restrict `Access-Control-Allow-Origin` to
an explicit allowlist of known origins. Wildcard (`*`) is acceptable
only for public, unauthenticated read-only APIs.
- **Error handling:**
- Internal errors must never leak stack traces, SQL queries, file paths,
or other implementation details to the client. Return generic error
messages in production; detailed errors only when `DEBUG` is enabled.
- **TLS:**
- Services never terminate TLS directly. They are always deployed behind
a TLS-terminating reverse proxy. The service itself listens on plain
HTTP. However, HSTS headers and `Secure` cookie flags must still be
set by the application so that the browser enforces HTTPS end-to-end.
This list is non-exhaustive. Apply defense-in-depth: if a standard security
hardening measure exists for HTTP services and is not listed here, it is
still expected. When in doubt, harden.
- `README.md` is the primary documentation. Required sections:
- **Description**: First line must include the project name, purpose,
category (web server, SPA, CLI tool, etc.), license, and author. Example:
"µPaaS is an MIT-licensed Go web application by @sneak that receives
git-frontend webhooks and deploys applications via Docker in realtime."
- **Getting Started**: Copy-pasteable install/usage code block.
- **Entrypoints**: Opens by stating that the repo adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard (with that link), then documents each provided `script/`
entrypoint and its purpose.
- **Rationale**: Why does this exist?
- **Design**: How is the program structured?
- **TODO**: Update meticulously, even between commits. When planning, put
the todo list in the README so a new agent can pick up where the last one
left off.
- **License**: MIT, GPL, or WTFPL. Ask the user for new projects. Include a
`LICENSE` file in the repo root and a License section in the README.
- **Author**: [@sneak](https://sneak.berlin).
- First commit of a new repo should contain only `README.md`.
- Go module root: `sneak.berlin/go/<name>`. Always run `go mod tidy` before
committing.
- Use SemVer.
- Database migrations live in `internal/db/migrations/` and must be embedded in
the binary.
- `000_migration.sql` — contains ONLY the creation of the migrations
tracking table itself. Nothing else.
- `001_schema.sql` — the full application schema.
- **Pre-1.0.0:** never add additional migration files (002, 003, etc.).
There is no installed base to migrate. Edit `001_schema.sql` directly.
- **Post-1.0.0:** add new numbered migration files for each schema change.
Never edit existing migrations after release.
- All repos should have an `.editorconfig` enforcing the project's indentation
settings.
- Avoid putting files in the repo root unless necessary. Root should contain
only project-level config files (`README.md`, `Makefile`, `Dockerfile`,
`LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`, and
language-specific config). Everything else goes in a subdirectory. Canonical
subdirectory names:
- `bin/` — executable scripts and tools
- `cmd/` — Go command entrypoints
- `configs/` — configuration templates and examples
- `deploy/` — deployment manifests (k8s, compose, terraform)
- `docs/` — documentation and markdown (README.md stays in root)
- `internal/` — Go internal packages
- `internal/db/migrations/` — database migrations
- `pkg/` — Go library packages
- `share/` — systemd units, data files
- `static/` — static assets (images, fonts, etc.)
- `web/` — web frontend source
- When setting up a new repo, files from the `prompts` repo may be used as
templates. Fetch them from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/<path>`.
- New repos must contain at minimum:
- `README.md`, `.git`, `.gitignore`, `.editorconfig`
- `LICENSE`, `REPO_POLICIES.md` (copy from the `prompts` repo)
- `Makefile`
- `script/` entrypoints (`bootstrap`, `setup`, `projectname`, `test`,
`lint`, `fmt`, `fmt-check`, `check`, `docker`, `cibuild`, `precommit`,
`install-precommit`)
- `Dockerfile`, `.dockerignore`
- `.gitea/workflows/check.yml`
- Go: `go.mod`, `go.sum`, `.golangci.yml`
- JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore`
- Python: `pyproject.toml`

50
TODO.md Normal file
View File

@@ -0,0 +1,50 @@
# Workflow
* branch (from `main`)
* do the work in Next Step
* move Next Step to the top of Completed Steps
* move the top item of Future Steps into Next Step
* commit (`TODO.md` changes in the same commit as the work)
* merge to `main` if the branch is not protected, otherwise open a PR
* push
# Status
pre-1.0
# Next Step
Bring the repo into policy compliance in one commit: add REPO_POLICIES.md
(copied from sneak/prompts), .editorconfig, and .golangci.yml. Verify
`make check` stays green with the new lint config.
# Completed Steps
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints,
Makefile shims, README Entrypoints section
- 2026-07-02: Consolidated CLI verbs, retired overlapping commands; bound
the local index to its backup destination URL.
- 2026-06-28: snapshot rm now removes metadata only and prints the prune
command; restore skips chown when running as non-root.
- 2026-06-26: Snapshot IDs hashed at the storage boundary; snapshot list
made resilient to bad remote entries.
- 2026-06-24: Collapsed snapshot prune into vaultik prune; restore streams
blobs to disk and restores files in blob-locality order; cron output
fixes.
- 2026-06-17: Restore overhaul: ReadAt chunk reads from cached blobs,
reference-counted blob sweeper, integration tests; new internal/ui
output layer, banner, and progress lines.
- 2025-12-18: Added ARCHITECTURE.md and godoc coverage for exported API.
- 2025-07-26: End-to-end integration tests; manifest format refactor;
renamed backup to snapshot; afero filesystem abstraction.
- 2025-07-20: Initial design and implementation: cobra + fx CLI skeleton,
SQLite index database, UUID blob storage with streaming chunking.
# Future Steps
- Add REPO_POLICIES.md, .editorconfig, .golangci.yml (the Next Step).
- Reconcile the uncommitted ARCHITECTURE.md edits on main: finish and
commit, or revert.
- Review stale local branches (add-godoc-to-cli-package,
feature/pluggable-storage-backend) and merge or delete them.
- Define remaining scope for a first tagged release and cut v0.1.0.

View File

@@ -7,7 +7,7 @@ Vaultik uses a local SQLite database to track file metadata, chunk mappings, and
**Important Notes:** **Important Notes:**
- **No Migration Support (pre-1.0)**: Vaultik does not support database schema - **No Migration Support (pre-1.0)**: Vaultik does not support database schema
migrations. The local index is treated as disposable — if the schema changes, migrations. The local index is treated as disposable — if the schema changes,
delete the local SQLite database (`vaultik database purge`) and run a full delete the local SQLite database (`vaultik database delete`) and run a full
backup. The remote storage is unaffected; the new index will re-deduplicate backup. The remote storage is unaffected; the new index will re-deduplicate
against existing remote blobs. against existing remote blobs.
- **Version Compatibility**: In rare cases, you may need to use the same version - **Version Compatibility**: In rare cases, you may need to use the same version

View File

@@ -18,28 +18,33 @@ func NewDatabaseCommand() *cobra.Command {
} }
cmd.AddCommand( cmd.AddCommand(
newDatabasePurgeCommand(), newDatabaseDeleteCommand(),
) )
return cmd return cmd
} }
// newDatabasePurgeCommand creates the database purge command // newDatabaseDeleteCommand creates the database delete command.
func newDatabasePurgeCommand() *cobra.Command { // (Renamed from "purge"; the operation removes the SQLite file
// entirely, which is a delete, not a purge of content.)
func newDatabaseDeleteCommand() *cobra.Command {
var force bool var force bool
cmd := &cobra.Command{ cmd := &cobra.Command{
Use: "purge", Use: "delete",
Short: "Delete the local state database", Short: "Delete the local state database file",
Long: `Completely removes the local SQLite state database. Long: `Completely removes the local SQLite state database.
This will erase all local tracking of: This will erase all local tracking of:
- File metadata and change detection state - File metadata and change detection state
- Chunk and blob mappings - Chunk and blob mappings
- Local snapshot records - Local snapshot records
- The storage-binding record
The remote storage is NOT affected. After purging, the next backup will The remote storage is NOT affected. After deletion, the next backup
perform a full scan and re-deduplicate against existing remote blobs. will perform a full scan and re-deduplicate against existing remote
blobs, and the local index will re-bind to the currently configured
storage destination on that run.
Use --force to skip the confirmation prompt.`, Use --force to skip the confirmation prompt.`,
Args: cobra.NoArgs, Args: cobra.NoArgs,
@@ -88,10 +93,10 @@ Use --force to skip the confirmation prompt.`,
rootFlags := GetRootFlags() rootFlags := GetRootFlags()
if !rootFlags.Quiet { if !rootFlags.Quiet {
fmt.Printf("Database purged: %s\n", dbPath) fmt.Printf("Database deleted: %s\n", dbPath)
} }
log.Info("Local state database purged", "path", dbPath) log.Info("Local state database deleted", "path", dbPath)
return nil return nil
}, },
} }

View File

@@ -18,7 +18,7 @@ func TestCLIEntry(t *testing.T) {
} }
// Verify all subcommands are registered // Verify all subcommands are registered
expectedCommands := []string{"config", "snapshot", "store", "prune", "info", "version", "remote", "database"} expectedCommands := []string{"config", "snapshot", "prune", "info", "version", "remote", "database"}
for _, expected := range expectedCommands { for _, expected := range expectedCommands {
found := false found := false
for _, cmd := range cmd.Commands() { for _, cmd := range cmd.Commands() {
@@ -38,7 +38,7 @@ func TestCLIEntry(t *testing.T) {
t.Errorf("Failed to find snapshot command: %v", err) t.Errorf("Failed to find snapshot command: %v", err)
} else { } else {
// Check snapshot subcommands // Check snapshot subcommands
expectedSubCommands := []string{"create", "list", "purge", "verify", "cleanup", "restore"} expectedSubCommands := []string{"create", "list", "purge", "verify", "remove", "restore"}
for _, expected := range expectedSubCommands { for _, expected := range expectedSubCommands {
found := false found := false
for _, subcmd := range snapshotCmd.Commands() { for _, subcmd := range snapshotCmd.Commands() {

View File

@@ -53,7 +53,6 @@ on the source system.`,
cmd.AddCommand( cmd.AddCommand(
NewConfigCommand(), NewConfigCommand(),
NewPruneCommand(), NewPruneCommand(),
NewStoreCommand(),
NewSnapshotCommand(), NewSnapshotCommand(),
NewInfoCommand(), NewInfoCommand(),
NewVersionCommand(), NewVersionCommand(),

View File

@@ -25,7 +25,6 @@ func NewSnapshotCommand() *cobra.Command {
cmd.AddCommand(newSnapshotPurgeCommand()) cmd.AddCommand(newSnapshotPurgeCommand())
cmd.AddCommand(newSnapshotVerifyCommand()) cmd.AddCommand(newSnapshotVerifyCommand())
cmd.AddCommand(newSnapshotRemoveCommand()) cmd.AddCommand(newSnapshotRemoveCommand())
cmd.AddCommand(newSnapshotCleanupCommand())
cmd.AddCommand(newSnapshotRestoreCommand()) cmd.AddCommand(newSnapshotRestoreCommand())
return cmd return cmd
@@ -322,7 +321,7 @@ func newSnapshotRemoveCommand() *cobra.Command {
opts := &vaultik.RemoveOptions{} opts := &vaultik.RemoveOptions{}
cmd := &cobra.Command{ cmd := &cobra.Command{
Use: "remove [snapshot-id]", Use: "remove <snapshot-id>",
Aliases: []string{"rm"}, Aliases: []string{"rm"},
Short: "Remove a snapshot from local index and remote metadata", Short: "Remove a snapshot from local index and remote metadata",
Long: `Removes a snapshot. Long: `Removes a snapshot.
@@ -341,20 +340,13 @@ If the remote is unreachable, the local-database removal still completes
and a warning is emitted; rerun 'vaultik prune' once the destination store and a warning is emitted; rerun 'vaultik prune' once the destination store
is reachable to finish remote cleanup. is reachable to finish remote cleanup.
Use --all --force to remove all snapshots.`, To wipe the entire destination store and start over, use 'vaultik remote
nuke --force' — it is the single supported entry point for that.`,
Args: func(cmd *cobra.Command, args []string) error { Args: func(cmd *cobra.Command, args []string) error {
all, _ := cmd.Flags().GetBool("all")
if all {
if len(args) > 0 {
_ = cmd.Help()
return fmt.Errorf("--all cannot be used with a snapshot ID")
}
return nil
}
if len(args) != 1 { if len(args) != 1 {
_ = cmd.Help() _ = cmd.Help()
if len(args) == 0 { if len(args) == 0 {
return fmt.Errorf("snapshot ID required (or use --all --force)") return fmt.Errorf("snapshot ID required")
} }
return fmt.Errorf("expected 1 argument, got %d", len(args)) return fmt.Errorf("expected 1 argument, got %d", len(args))
} }
@@ -381,12 +373,7 @@ Use --all --force to remove all snapshots.`,
lc.Append(fx.Hook{ lc.Append(fx.Hook{
OnStart: func(ctx context.Context) error { OnStart: func(ctx context.Context) error {
go func() { go func() {
var err error _, err := v.RemoveSnapshot(args[0], opts)
if opts.All {
_, err = v.RemoveAllSnapshots(opts)
} else {
_, err = v.RemoveSnapshot(args[0], opts)
}
if err != nil { if err != nil {
if err != context.Canceled { if err != context.Canceled {
if !opts.JSON { if !opts.JSON {
@@ -417,65 +404,6 @@ Use --all --force to remove all snapshots.`,
cmd.Flags().BoolVar(&opts.DryRun, "dry-run", false, "Show what would be removed without removing") cmd.Flags().BoolVar(&opts.DryRun, "dry-run", false, "Show what would be removed without removing")
cmd.Flags().BoolVar(&opts.JSON, "json", false, "Output result as JSON") cmd.Flags().BoolVar(&opts.JSON, "json", false, "Output result as JSON")
cmd.Flags().BoolVar(&opts.LocalOnly, "local-only", false, "Skip remote cleanup; only touch the local index") cmd.Flags().BoolVar(&opts.LocalOnly, "local-only", false, "Skip remote cleanup; only touch the local index")
cmd.Flags().BoolVar(&opts.All, "all", false, "Remove all snapshots (requires --force)")
return cmd
}
// newSnapshotCleanupCommand creates the 'snapshot cleanup' subcommand
func newSnapshotCleanupCommand() *cobra.Command {
cmd := &cobra.Command{
Use: "cleanup",
Short: "Remove stale local snapshot records not found in remote storage",
Long: `Removes local database records for snapshots whose metadata no longer
exists in remote storage. These are typically left behind by incomplete
or interrupted backups.
This command does not delete anything from remote storage.`,
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, args []string) error {
configPath, err := ResolveConfigPath()
if err != nil {
return err
}
rootFlags := GetRootFlags()
return RunWithApp(cmd.Context(), AppOptions{
ConfigPath: configPath,
LogOptions: log.LogOptions{
Verbose: rootFlags.Verbose,
Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet,
},
Modules: []fx.Option{},
Invokes: []fx.Option{
fx.Invoke(func(v *vaultik.Vaultik, lc fx.Lifecycle) {
lc.Append(fx.Hook{
OnStart: func(ctx context.Context) error {
go func() {
if err := v.CleanupLocalSnapshots(); err != nil {
if err != context.Canceled {
log.Error("Cleanup failed", "error", err)
ReportError("Cleanup failed: %v", err)
os.Exit(1)
}
}
if err := v.Shutdowner.Shutdown(); err != nil {
log.Error("Failed to shutdown", "error", err)
}
}()
return nil
},
OnStop: func(ctx context.Context) error {
v.Cancel()
return nil
},
})
}),
},
})
},
}
return cmd return cmd
} }

View File

@@ -1,158 +0,0 @@
package cli
import (
"context"
"fmt"
"strings"
"time"
"github.com/spf13/cobra"
"go.uber.org/fx"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage"
)
// StoreApp contains dependencies for store commands
type StoreApp struct {
Storage storage.Storer
Shutdowner fx.Shutdowner
}
// NewStoreCommand creates the store command and subcommands
func NewStoreCommand() *cobra.Command {
cmd := &cobra.Command{
Use: "store",
Short: "Storage information commands",
Long: "Commands for viewing information about the storage backend",
}
// Add subcommands
cmd.AddCommand(newStoreInfoCommand())
return cmd
}
// newStoreInfoCommand creates the 'store info' subcommand
func newStoreInfoCommand() *cobra.Command {
return &cobra.Command{
Use: "info",
Short: "Display storage information",
Long: "Shows storage configuration and statistics including snapshots and blobs",
RunE: func(cmd *cobra.Command, args []string) error {
return runWithApp(cmd.Context(), func(app *StoreApp) error {
return app.Info(cmd.Context())
})
},
}
}
// Info displays storage information
func (app *StoreApp) Info(ctx context.Context) error {
// Get storage info
storageInfo := app.Storage.Info()
fmt.Printf("Storage Information\n")
fmt.Printf("==================\n\n")
fmt.Printf("Storage Configuration:\n")
fmt.Printf(" Type: %s\n", storageInfo.Type)
fmt.Printf(" Location: %s\n\n", storageInfo.Location)
// Count snapshots by listing metadata/ prefix
snapshotCount := 0
snapshotCh := app.Storage.ListStream(ctx, "metadata/")
snapshotDirs := make(map[string]bool)
for object := range snapshotCh {
if object.Err != nil {
return fmt.Errorf("listing snapshots: %w", object.Err)
}
// Extract snapshot ID from path like metadata/2024-01-15-143052-hostname/
parts := strings.Split(object.Key, "/")
if len(parts) >= 2 && parts[0] == "metadata" && parts[1] != "" {
snapshotDirs[parts[1]] = true
}
}
snapshotCount = len(snapshotDirs)
// Count blobs and calculate total size by listing blobs/ prefix
blobCount := 0
var totalSize int64
blobCh := app.Storage.ListStream(ctx, "blobs/")
for object := range blobCh {
if object.Err != nil {
return fmt.Errorf("listing blobs: %w", object.Err)
}
if !strings.HasSuffix(object.Key, "/") { // Skip directories
blobCount++
totalSize += object.Size
}
}
fmt.Printf("Storage Statistics:\n")
fmt.Printf(" Snapshots: %d\n", snapshotCount)
fmt.Printf(" Blobs: %d\n", blobCount)
fmt.Printf(" Total Size: %s\n", formatBytes(totalSize))
return nil
}
// formatBytes formats bytes into human-readable format
func formatBytes(bytes int64) string {
const unit = 1024
if bytes < unit {
return fmt.Sprintf("%d B", bytes)
}
div, exp := int64(unit), 0
for n := bytes / unit; n >= unit; n /= unit {
div *= unit
exp++
}
return fmt.Sprintf("%.1f %cB", float64(bytes)/float64(div), "KMGTPE"[exp])
}
// runWithApp creates the FX app and runs the given function
func runWithApp(ctx context.Context, fn func(*StoreApp) error) error {
var result error
rootFlags := GetRootFlags()
// Use unified config resolution
configPath, err := ResolveConfigPath()
if err != nil {
return err
}
err = RunWithApp(ctx, AppOptions{
ConfigPath: configPath,
LogOptions: log.LogOptions{
Verbose: rootFlags.Verbose,
Debug: rootFlags.Debug,
Quiet: rootFlags.Quiet,
},
Modules: []fx.Option{
fx.Provide(func(storer storage.Storer, shutdowner fx.Shutdowner) *StoreApp {
return &StoreApp{
Storage: storer,
Shutdowner: shutdowner,
}
}),
},
Invokes: []fx.Option{
fx.Invoke(func(app *StoreApp, shutdowner fx.Shutdowner) {
result = fn(app)
// Shutdown after command completes
go func() {
time.Sleep(100 * time.Millisecond) // Brief delay to ensure clean shutdown
if err := shutdowner.Shutdown(); err != nil {
log.Error("Failed to shutdown", "error", err)
}
}()
}),
},
})
if err != nil {
return err
}
return result
}

View File

@@ -0,0 +1,53 @@
package database
import (
"context"
"database/sql"
"errors"
"fmt"
)
// LocalMetaKeyStorageURL is the key under which the destination store's
// URL is recorded when a mutating command first binds the local index
// to a specific backup destination.
const LocalMetaKeyStorageURL = "storage_url"
// LocalMetaRepository provides keyed access to host-local settings
// stored in the local_meta table.
type LocalMetaRepository struct {
db *DB
}
func NewLocalMetaRepository(db *DB) *LocalMetaRepository {
return &LocalMetaRepository{db: db}
}
// Get returns the value stored at key, or the empty string if the key
// is not set. A missing key is not an error — the caller distinguishes
// "unset" (bind on first use) from "set to something" (compare).
func (r *LocalMetaRepository) Get(ctx context.Context, key string) (string, error) {
var value string
err := r.db.conn.QueryRowContext(ctx,
"SELECT value FROM local_meta WHERE key = ?", key,
).Scan(&value)
if errors.Is(err, sql.ErrNoRows) {
return "", nil
}
if err != nil {
return "", fmt.Errorf("reading local_meta %q: %w", key, err)
}
return value, nil
}
// Set writes key=value, replacing any prior value.
func (r *LocalMetaRepository) Set(ctx context.Context, key, value string) error {
_, err := r.db.ExecWithLog(ctx,
`INSERT INTO local_meta (key, value) VALUES (?, ?)
ON CONFLICT(key) DO UPDATE SET value = excluded.value`,
key, value,
)
if err != nil {
return fmt.Errorf("writing local_meta %q: %w", key, err)
}
return nil
}

View File

@@ -0,0 +1,52 @@
package database_test
import (
"context"
"testing"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/database"
)
func TestLocalMetaEmptyOnFresh(t *testing.T) {
db, err := database.NewTestDB()
require.NoError(t, err)
defer func() { _ = db.Close() }()
repos := database.NewRepositories(db)
got, err := repos.LocalMeta.Get(context.Background(), database.LocalMetaKeyStorageURL)
require.NoError(t, err)
require.Equal(t, "", got, "fresh DB must return empty for unset keys, not error")
}
func TestLocalMetaSetGetRoundTrip(t *testing.T) {
db, err := database.NewTestDB()
require.NoError(t, err)
defer func() { _ = db.Close() }()
repos := database.NewRepositories(db)
ctx := context.Background()
require.NoError(t, repos.LocalMeta.Set(ctx, database.LocalMetaKeyStorageURL, "file:///mnt/backups"))
got, err := repos.LocalMeta.Get(ctx, database.LocalMetaKeyStorageURL)
require.NoError(t, err)
require.Equal(t, "file:///mnt/backups", got)
}
func TestLocalMetaSetOverwrites(t *testing.T) {
db, err := database.NewTestDB()
require.NoError(t, err)
defer func() { _ = db.Close() }()
repos := database.NewRepositories(db)
ctx := context.Background()
require.NoError(t, repos.LocalMeta.Set(ctx, database.LocalMetaKeyStorageURL, "s3://old"))
require.NoError(t, repos.LocalMeta.Set(ctx, database.LocalMetaKeyStorageURL, "s3://new"))
got, err := repos.LocalMeta.Get(ctx, database.LocalMetaKeyStorageURL)
require.NoError(t, err)
require.Equal(t, "s3://new", got)
}

View File

@@ -19,6 +19,7 @@ type Repositories struct {
ChunkFiles *ChunkFileRepository ChunkFiles *ChunkFileRepository
Snapshots *SnapshotRepository Snapshots *SnapshotRepository
Uploads *UploadRepository Uploads *UploadRepository
LocalMeta *LocalMetaRepository
} }
// NewRepositories creates a new Repositories instance with all repository types. // NewRepositories creates a new Repositories instance with all repository types.
@@ -34,6 +35,7 @@ func NewRepositories(db *DB) *Repositories {
ChunkFiles: NewChunkFileRepository(db), ChunkFiles: NewChunkFileRepository(db),
Snapshots: NewSnapshotRepository(db), Snapshots: NewSnapshotRepository(db),
Uploads: NewUploadRepository(db.conn), Uploads: NewUploadRepository(db.conn),
LocalMeta: NewLocalMetaRepository(db),
} }
} }

View File

@@ -133,3 +133,17 @@ CREATE TABLE IF NOT EXISTS uploads (
-- Index for efficient snapshot lookups -- Index for efficient snapshot lookups
CREATE INDEX IF NOT EXISTS idx_uploads_snapshot_id ON uploads(snapshot_id); CREATE INDEX IF NOT EXISTS idx_uploads_snapshot_id ON uploads(snapshot_id);
-- Local metadata: keyed, host-local settings that bind the state of the
-- local index database to external context. The primary use is
-- storage_url: once a backup writes blobs to a destination, the local
-- index is only valid against that destination — if the configured
-- storage_url later changes, the scanner would silently think already-
-- known chunks are still on the new (empty) destination and skip
-- uploading them, corrupting future snapshots. On every mutating
-- command startup, we compare the configured storage_url to the stored
-- one and refuse to proceed on mismatch.
CREATE TABLE IF NOT EXISTS local_meta (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
);

View File

@@ -55,6 +55,17 @@ type PruneBlobsResult struct {
// prefer this method over PruneDatabase or PruneBlobs individually // prefer this method over PruneDatabase or PruneBlobs individually
// unless it specifically wants one half. // unless it specifically wants one half.
func (v *Vaultik) Prune(opts *PruneOptions) error { func (v *Vaultik) Prune(opts *PruneOptions) error {
if err := v.EnsureStorageBinding(); err != nil {
return err
}
// First reconcile local snapshot records against remote metadata:
// any local snapshot whose manifest is missing from the destination
// store is treated as gone. This used to be the separate 'snapshot
// cleanup' command and is now folded in so a single 'vaultik prune'
// gets the local index fully back in sync with the destination.
if err := v.CleanupLocalSnapshots(); err != nil {
return fmt.Errorf("reconciling local snapshots with remote: %w", err)
}
if _, err := v.PruneDatabase(); err != nil { if _, err := v.PruneDatabase(); err != nil {
return fmt.Errorf("pruning local database: %w", err) return fmt.Errorf("pruning local database: %w", err)
} }

View File

@@ -291,7 +291,7 @@ func TestRemoveAllSnapshots_RequiresForce(t *testing.T) {
tv := vaultik.NewForTesting(store) tv := vaultik.NewForTesting(store)
opts := &vaultik.RemoveOptions{All: true} // No Force opts := &vaultik.RemoveOptions{} // No Force
_, err := tv.RemoveAllSnapshots(opts) _, err := tv.RemoveAllSnapshots(opts)
assert.Error(t, err) assert.Error(t, err)
@@ -310,7 +310,7 @@ func TestRemoveAllSnapshots_WithForce(t *testing.T) {
tv := vaultik.NewForTesting(store) tv := vaultik.NewForTesting(store)
opts := &vaultik.RemoveOptions{All: true, Force: true} opts := &vaultik.RemoveOptions{Force: true}
result, err := tv.RemoveAllSnapshots(opts) result, err := tv.RemoveAllSnapshots(opts)
require.NoError(t, err) require.NoError(t, err)
@@ -342,7 +342,7 @@ func TestRemoveAllSnapshots_DryRun(t *testing.T) {
// Default (no LocalOnly) enumerates the orphan remote keys, which // Default (no LocalOnly) enumerates the orphan remote keys, which
// matches what NewForTesting has — local DB is empty, so the two // matches what NewForTesting has — local DB is empty, so the two
// addManifest calls land as orphan remote keys. // addManifest calls land as orphan remote keys.
opts := &vaultik.RemoveOptions{All: true, Force: true, DryRun: true} opts := &vaultik.RemoveOptions{Force: true, DryRun: true}
result, err := tv.RemoveAllSnapshots(opts) result, err := tv.RemoveAllSnapshots(opts)
require.NoError(t, err) require.NoError(t, err)
@@ -362,7 +362,7 @@ func TestRemoveAllSnapshots_NoSnapshots(t *testing.T) {
tv := vaultik.NewForTesting(store) tv := vaultik.NewForTesting(store)
opts := &vaultik.RemoveOptions{All: true, Force: true} opts := &vaultik.RemoveOptions{Force: true}
result, err := tv.RemoveAllSnapshots(opts) result, err := tv.RemoveAllSnapshots(opts)
require.NoError(t, err) require.NoError(t, err)

View File

@@ -126,6 +126,10 @@ func (v *Vaultik) Restore(opts *RestoreOptions) error {
v.UI.Duration(result.Duration), v.UI.Duration(result.Duration),
) )
if os.Geteuid() != 0 {
v.UI.Warning("Restore did not preserve file ownership: chown(2) requires root. Re-run as root (e.g. with sudo) if you need original UID/GID preserved.")
}
if result.FilesFailed > 0 { if result.FilesFailed > 0 {
v.UI.Warning("%d file(s) failed to restore:", result.FilesFailed) v.UI.Warning("%d file(s) failed to restore:", result.FilesFailed)
for _, path := range result.FailedFiles { for _, path := range result.FailedFiles {
@@ -247,6 +251,7 @@ func (v *Vaultik) restoreAllFiles(
blobCache: blobCache, blobCache: blobCache,
sweeper: sweeper, sweeper: sweeper,
result: result, result: result,
runningAsRoot: os.Geteuid() == 0,
} }
// Periodic progress output, matching the snapshot create cadence. // Periodic progress output, matching the snapshot create cadence.
@@ -536,6 +541,13 @@ type restoreSession struct {
blobCache *blobDiskCache blobCache *blobDiskCache
sweeper *restoreSweeper sweeper *restoreSweeper
result *RestoreResult result *RestoreResult
// runningAsRoot gates chown(2). On every Unix-ish kernel, only
// root can chown a file to an arbitrary UID/GID — non-root chown
// always fails with EPERM. Attempting it anyway produces N
// guaranteed-failed syscalls + N noisy debug lines, so we skip
// the call entirely as non-root and emit one warning at the end
// of the restore explaining that ownership was not preserved.
runningAsRoot bool
} }
// restoreFile dispatches to the right per-kind restorer. // restoreFile dispatches to the right per-kind restorer.
@@ -580,9 +592,11 @@ func (s *restoreSession) restoreDirectory(file *database.File, targetPath string
if err := s.v.Fs.Chmod(targetPath, os.FileMode(file.Mode)); err != nil { if err := s.v.Fs.Chmod(targetPath, os.FileMode(file.Mode)); err != nil {
log.Debug("Failed to set directory permissions", "path", targetPath, "error", err) log.Debug("Failed to set directory permissions", "path", targetPath, "error", err)
} }
if _, ok := s.v.Fs.(*afero.OsFs); ok { if s.runningAsRoot {
if err := os.Chown(targetPath, int(file.UID), int(file.GID)); err != nil { if _, ok := s.v.Fs.(*afero.OsFs); ok {
log.Debug("Failed to set directory ownership", "path", targetPath, "error", err) if err := os.Chown(targetPath, int(file.UID), int(file.GID)); err != nil {
log.Debug("Failed to set directory ownership", "path", targetPath, "error", err)
}
} }
} }
if err := s.v.Fs.Chtimes(targetPath, file.MTime, file.MTime); err != nil { if err := s.v.Fs.Chtimes(targetPath, file.MTime, file.MTime); err != nil {
@@ -671,9 +685,11 @@ func (s *restoreSession) restoreRegularFile(file *database.File, targetPath stri
if err := s.v.Fs.Chmod(targetPath, os.FileMode(file.Mode)); err != nil { if err := s.v.Fs.Chmod(targetPath, os.FileMode(file.Mode)); err != nil {
log.Debug("Failed to set file permissions", "path", targetPath, "error", err) log.Debug("Failed to set file permissions", "path", targetPath, "error", err)
} }
if _, ok := s.v.Fs.(*afero.OsFs); ok { if s.runningAsRoot {
if err := os.Chown(targetPath, int(file.UID), int(file.GID)); err != nil { if _, ok := s.v.Fs.(*afero.OsFs); ok {
log.Debug("Failed to set file ownership", "path", targetPath, "error", err) if err := os.Chown(targetPath, int(file.UID), int(file.GID)); err != nil {
log.Debug("Failed to set file ownership", "path", targetPath, "error", err)
}
} }
} }
if err := s.v.Fs.Chtimes(targetPath, file.MTime, file.MTime); err != nil { if err := s.v.Fs.Chtimes(targetPath, file.MTime, file.MTime); err != nil {

View File

@@ -36,6 +36,10 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
"index_path", v.Config.IndexPath, "index_path", v.Config.IndexPath,
) )
if err := v.EnsureStorageBinding(); err != nil {
return err
}
// Clean up incomplete snapshots FIRST, before any scanning // Clean up incomplete snapshots FIRST, before any scanning
// This is critical for data safety - see CleanupIncompleteSnapshots for details // This is critical for data safety - see CleanupIncompleteSnapshots for details
hostname := v.Config.Hostname hostname := v.Config.Hostname
@@ -581,6 +585,9 @@ type SnapshotPurgeOptions struct {
// snapshot name, not the latest globally. This prevents `home` and `system` // snapshot name, not the latest globally. This prevents `home` and `system`
// snapshots from cannibalizing each other. // snapshots from cannibalizing each other.
func (v *Vaultik) PurgeSnapshotsWithOptions(opts *SnapshotPurgeOptions) error { func (v *Vaultik) PurgeSnapshotsWithOptions(opts *SnapshotPurgeOptions) error {
if err := v.EnsureStorageBinding(); err != nil {
return err
}
// Sync with remote first // Sync with remote first
if err := v.syncWithRemote(); err != nil { if err := v.syncWithRemote(); err != nil {
return fmt.Errorf("syncing with remote: %w", err) return fmt.Errorf("syncing with remote: %w", err)
@@ -861,6 +868,9 @@ func (v *Vaultik) outputVerifyJSON(result *VerifyResult) error {
// human ID is hashed via RemoteSnapshotKey and compared against the // human ID is hashed via RemoteSnapshotKey and compared against the
// remote listing. // remote listing.
func (v *Vaultik) CleanupLocalSnapshots() error { func (v *Vaultik) CleanupLocalSnapshots() error {
if err := v.EnsureStorageBinding(); err != nil {
return err
}
remoteKeys, err := v.listAllRemoteSnapshotKeys() remoteKeys, err := v.listAllRemoteSnapshotKeys()
if err != nil { if err != nil {
return err return err
@@ -977,7 +987,6 @@ type RemoveOptions struct {
DryRun bool DryRun bool
JSON bool JSON bool
LocalOnly bool // Skip remote cleanup; only touch the local index LocalOnly bool // Skip remote cleanup; only touch the local index
All bool // Remove all snapshots (requires Force)
} }
// RemoveResult contains the result of a snapshot removal // RemoveResult contains the result of a snapshot removal
@@ -1006,6 +1015,10 @@ func (v *Vaultik) RemoveSnapshot(snapshotID string, opts *RemoveOptions) (*Remov
SnapshotID: snapshotID, SnapshotID: snapshotID,
} }
if err := v.EnsureStorageBinding(); err != nil {
return result, err
}
if opts.DryRun { if opts.DryRun {
result.DryRun = true result.DryRun = true
if !opts.JSON { if !opts.JSON {
@@ -1086,6 +1099,9 @@ func (v *Vaultik) RemoveSnapshot(snapshotID string, opts *RemoveOptions) (*Remov
// "remove --all" leaves nothing behind, even when the local DB and // "remove --all" leaves nothing behind, even when the local DB and
// remote storage have diverged. // remote storage have diverged.
func (v *Vaultik) RemoveAllSnapshots(opts *RemoveOptions) (*RemoveResult, error) { func (v *Vaultik) RemoveAllSnapshots(opts *RemoveOptions) (*RemoveResult, error) {
if err := v.EnsureStorageBinding(); err != nil {
return nil, err
}
localSnaps, err := v.localSnapshotIDs() localSnaps, err := v.localSnapshotIDs()
if err != nil { if err != nil {
return nil, fmt.Errorf("listing local snapshots: %w", err) return nil, fmt.Errorf("listing local snapshots: %w", err)

View File

@@ -0,0 +1,91 @@
package vaultik
import (
"fmt"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
)
// EnsureStorageBinding guarantees that the local index database is
// bound to the currently-configured storage destination. Every mutating
// command must call this before touching either the local index or the
// destination store, because the two live in lockstep: the local index
// records which chunks/blobs already exist at the destination, and
// mismatched destination + local index produces silent corruption (the
// scanner sees "known" chunks and skips uploads, then writes snapshots
// whose manifests reference blobs that aren't on the new destination).
//
// Behaviour:
// - On first use (empty stored value), record the configured
// storage_url and log the binding.
// - When the stored value matches the configured storage_url, do
// nothing and return nil.
// - When the two differ, refuse with an error that tells the user
// how to recover (revert the config, or run `vaultik database
// purge` to discard the local index and rebuild against the new
// destination on the next backup).
//
// Read-only inspection commands (remote info, snapshot list, etc.)
// deliberately don't call this: they can be run against a bare
// destination store without any binding state.
func (v *Vaultik) EnsureStorageBinding() error {
if v.Repositories == nil || v.Config == nil {
// NewForTesting builds a Vaultik with no DB or config;
// there's nothing to bind and no binding to check. Callers
// exercising the bind path use a real DB and populate Config
// explicitly (see storage_bind_test.go).
return nil
}
configured := v.Config.StorageURL
if configured == "" {
// Some legacy configs still use the split s3.* keys instead of
// storage_url. Falling back to a synthetic URL for those would
// hide the fact that the binding is loose. Instead, treat
// unset as "nothing to bind against" — the check is
// necessarily best-effort for those configs.
return nil
}
stored, err := v.Repositories.LocalMeta.Get(v.ctx, database.LocalMetaKeyStorageURL)
if err != nil {
return fmt.Errorf("reading local storage binding: %w", err)
}
if stored == "" {
if err := v.Repositories.LocalMeta.Set(v.ctx, database.LocalMetaKeyStorageURL, configured); err != nil {
return fmt.Errorf("recording local storage binding: %w", err)
}
log.Info("Bound local index to storage destination", "storage_url", configured)
return nil
}
if stored == configured {
return nil
}
return fmt.Errorf("%s", buildBindingMismatchMessage(stored, configured))
}
// buildBindingMismatchMessage assembles the multi-line explanation
// shown when the local index is bound to a different destination than
// the currently-configured one. Kept as a separate function so the
// lint-flagged multi-line format string is expressed as a plain string
// literal rather than a fmt.Errorf argument (staticcheck ST1005
// disallows trailing punctuation on error format strings).
func buildBindingMismatchMessage(stored, configured string) string {
return "local index is bound to a different backup destination\n" +
" local index bound to: " + stored + "\n" +
" currently configured: " + configured + "\n" +
"\n" +
"The local index database tracks which chunks and blobs already exist at the\n" +
"destination store. Using it against a different destination would silently\n" +
"skip uploads (the scanner would treat every chunk as already present), leaving\n" +
"future snapshots referencing blobs that don't exist at the new destination.\n" +
"\n" +
"To proceed, either:\n" +
" - revert storage_url in your config to the bound destination, or\n" +
" - run 'vaultik database delete' to discard the local index and rebuild it\n" +
" from a fresh full backup against the new destination"
}

View File

@@ -0,0 +1,72 @@
package vaultik_test
import (
"context"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// buildBindTestVaultik returns a minimal Vaultik wired with a real
// in-memory DB and a config carrying the given StorageURL. Enough to
// exercise EnsureStorageBinding without spinning up storage or fx.
func buildBindTestVaultik(t *testing.T, storageURL string) (*vaultik.TestVaultik, *database.Repositories) {
t.Helper()
db, err := database.NewTestDB()
require.NoError(t, err)
t.Cleanup(func() { _ = db.Close() })
repos := database.NewRepositories(db)
tv := vaultik.NewForTesting(nil)
tv.Repositories = repos
tv.DB = db
tv.Config = &config.Config{StorageURL: storageURL}
tv.SetContext(context.Background())
return tv, repos
}
func TestEnsureStorageBinding_FreshDBRecordsURL(t *testing.T) {
log.Initialize(log.Config{})
tv, repos := buildBindTestVaultik(t, "file:///mnt/backups/new")
require.NoError(t, tv.EnsureStorageBinding())
got, err := repos.LocalMeta.Get(context.Background(), database.LocalMetaKeyStorageURL)
require.NoError(t, err)
assert.Equal(t, "file:///mnt/backups/new", got, "first call must record the configured URL")
}
func TestEnsureStorageBinding_MatchingURLPasses(t *testing.T) {
log.Initialize(log.Config{})
tv, repos := buildBindTestVaultik(t, "s3://bucket/prefix")
require.NoError(t, repos.LocalMeta.Set(context.Background(),
database.LocalMetaKeyStorageURL, "s3://bucket/prefix"))
require.NoError(t, tv.EnsureStorageBinding())
}
func TestEnsureStorageBinding_MismatchRefuses(t *testing.T) {
log.Initialize(log.Config{})
tv, repos := buildBindTestVaultik(t, "file:///mnt/backups/new")
require.NoError(t, repos.LocalMeta.Set(context.Background(),
database.LocalMetaKeyStorageURL, "file:///mnt/backups/old"))
err := tv.EnsureStorageBinding()
require.Error(t, err)
assert.Contains(t, err.Error(), "file:///mnt/backups/old")
assert.Contains(t, err.Error(), "file:///mnt/backups/new")
assert.Contains(t, err.Error(), "vaultik database delete")
}

73
script/bootstrap Executable file
View File

@@ -0,0 +1,73 @@
#!/bin/sh
# script/bootstrap: install all dependencies needed to build and develop
# this repo. Idempotent: every install is guarded by a check so already
# installed tools are skipped. Base tooling comes from nix, apt, brew,
# or apk (detected in that order); assumes NOTHING is present (not git,
# make, or go).
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
PKGMGR=""
SUDO=""
detect_pkgmgr() {
[ -n "$PKGMGR" ] && return 0
if command -v nix-env >/dev/null 2>&1; then
PKGMGR="nix"
elif command -v apt-get >/dev/null 2>&1; then
PKGMGR="apt"
elif command -v brew >/dev/null 2>&1; then
PKGMGR="brew"
elif command -v apk >/dev/null 2>&1; then
PKGMGR="apk"
else
echo "bootstrap: no supported package manager (nix, apt, brew, apk)" >&2
exit 1
fi
if [ "$PKGMGR" = "apt" ]; then
export DEBIAN_FRONTEND=noninteractive
if [ "$(id -u)" != "0" ]; then
SUDO="sudo"
fi
fi
}
# pkg_install <nix-attr> <apt-pkg> <brew-formula> <apk-pkg>
pkg_install() {
detect_pkgmgr
case "$PKGMGR" in
nix) nix-env -iA "nixpkgs.$1" ;;
apt) $SUDO env DEBIAN_FRONTEND=noninteractive apt-get install -y "$2" ;;
brew) brew install "$3" ;;
apk) apk add --no-cache "$4" ;;
esac
}
missing() {
! command -v "$1" >/dev/null 2>&1
}
main() {
cd "$ROOT"
# Base tooling (every repo)
if missing git; then pkg_install git git git git; fi
if missing make; then pkg_install gnumake make make make; fi
# Go toolchain
if missing go; then pkg_install go golang go go; fi
# golangci-lint: packaged in nix, brew, and apk. There is no apt
# package; on apt systems install it manually from a hash-verified
# GitHub release archive (never curl | sh).
if missing golangci-lint; then
pkg_install golangci-lint golangci-lint golangci-lint golangci-lint
fi
go mod download
echo "bootstrap complete"
}
main "$@"

15
script/check Executable file
View File

@@ -0,0 +1,15 @@
#!/bin/sh
# script/check: run all checks (test, lint, fmt-check). Our own
# extension to scripts-to-rule-them-all. Must not modify any files.
# Generic: usually needs no adaptation.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
main() {
"$SCRIPT_DIR/test"
"$SCRIPT_DIR/lint"
"$SCRIPT_DIR/fmt-check"
}
main "$@"

14
script/cibuild Executable file
View File

@@ -0,0 +1,14 @@
#!/bin/sh
# script/cibuild: run the CI build. The Dockerfile runs script/check
# (via make check), so a successful build implies all checks pass.
# Generic: needs no adaptation. The Gitea workflow runs this on push.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
docker build .
}
main "$@"

15
script/docker Executable file
View File

@@ -0,0 +1,15 @@
#!/bin/sh
# script/docker: build the Docker image tagged with the project name.
# Identical in all repos; the tag comes from script/projectname.
# Generic: needs no adaptation.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
docker build -t "$("$SCRIPT_DIR/projectname")" .
}
main "$@"

12
script/fmt Executable file
View File

@@ -0,0 +1,12 @@
#!/bin/sh
# script/fmt: format all files (writes).
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
go fmt ./...
}
main "$@"

18
script/fmt-check Executable file
View File

@@ -0,0 +1,18 @@
#!/bin/sh
# script/fmt-check: check formatting (read-only). Same scope as
# script/fmt, but fails instead of writing.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
unformatted="$(gofmt -l .)"
if [ -n "$unformatted" ]; then
echo "Files not formatted:" >&2
echo "$unformatted" >&2
exit 1
fi
}
main "$@"

17
script/install-precommit Executable file
View File

@@ -0,0 +1,17 @@
#!/bin/sh
# script/install-precommit: install the git pre-commit hook that runs
# script/precommit. Our own extension to scripts-to-rule-them-all.
# Generic: needs no adaptation.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
hook=".git/hooks/pre-commit"
printf '#!/bin/sh\nset -e\nscript/precommit\n' > .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit
echo "pre-commit hook installed: runs script/precommit"
}
main "$@"

12
script/lint Executable file
View File

@@ -0,0 +1,12 @@
#!/bin/sh
# script/lint: run the linter.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
golangci-lint run ./...
}
main "$@"

21
script/precommit Executable file
View File

@@ -0,0 +1,21 @@
#!/bin/sh
# script/precommit: run by the git pre-commit hook; fails the commit if
# checks fail. Our own extension to scripts-to-rule-them-all. Go repo
# extras: go mod tidy and go fmt must leave the tree unchanged.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
go mod tidy
go fmt ./...
git diff --exit-code -- go.mod go.sum || {
echo "go mod tidy changed files; please stage and retry"
exit 1
}
"$SCRIPT_DIR/check"
}
main "$@"

12
script/projectname Executable file
View File

@@ -0,0 +1,12 @@
#!/bin/sh
# script/projectname: output the name of this project. Our own
# extension to scripts-to-rule-them-all. Other scripts that need the
# name (e.g. script/docker) call this, so they can stay identical
# across all repos.
set -eu
main() {
echo "vaultik"
}
main "$@"

14
script/setup Executable file
View File

@@ -0,0 +1,14 @@
#!/bin/sh
# script/setup: set up the repo for development after a fresh clone:
# installs dependencies (script/bootstrap) and the git pre-commit hook.
# Add any repo-specific initialization (db init, .env template) here.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
main() {
"$SCRIPT_DIR/bootstrap"
"$SCRIPT_DIR/install-precommit"
}
main "$@"

18
script/test Executable file
View File

@@ -0,0 +1,18 @@
#!/bin/sh
# script/test: run the test suite. Quiet on success; on failure, rerun
# verbosely for full diagnostic output (the exit 1 ensures the rerun
# never turns a failure into a pass).
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
go test -race -timeout 30s ./... || {
echo "--- Rerunning with -v for details ---"
go test -race -timeout 30s -v ./...
exit 1
}
}
main "$@"