3 Commits
Author SHA1 Message Date
sneak d199ff53ce Harden the lint-guard shell scanner against silent evasions (closes #121)
check / check (pull_request) Failing after 1s
The guard test's shell scanner was weaker than its commit message
claimed. Two holes are closed.

shellCode now treats `<<` as a here-document only when it is a real
redirection: outside single and double quotes, not past an unquoted
word-initial `#` that begins an inline comment, and followed by a
delimiter word. A `<<` inside a quoted string or an inline comment no
longer opens a phantom here-document that swallows the rest of the
file -- including the silent case where the fake terminator recurs
later as a line of its own -- and a here-document still open at end of
file is a loud error rather than a silent truncation.

assertLinterIsContainerised now cuts the joined line into the simple
commands the shell would run -- on `;`, `&&`, `||` and `|` -- and
requires the command that names the linter to begin with docker. So
`docker info; golangci-lint run` and `docker info || golangci-lint run`
are rejected, while script/lint-fix's `docker run ... golangci-lint`
still passes.

The scanner comment now names the inline-comment exception alongside
the quoted-string and arithmetic ones, and the inherent limits of a
text scan. Dockerfile.lint's citation is corrected from `lll` to
`revive`, the finding the recorded evidence actually named.

model: claude-opus-4-8
2026-09-21 17:34:41 +00:00
clawbot bdce350041 Delete dead code and stale fixtures, fix config set reindent (closes #70)
check / check (push) Failing after 1s
check / check (pull_request) Failing after 1s
Removes code and fixtures nothing uses: the internal/models package and its test, a second SnapshotInfo type in package cli that had no references (the live one is in internal/vaultik), and two config fixtures, test-config.yml and test/integration-config.yml, whose keys the config loader no longer accepts. test/config.yaml stays; a test uses it.

Also fixes config set, which rewrote the whole file with 4-space indentation on the first set despite the documented promise to preserve formatting. It now encodes with 2-space indent like the default template, and a test asserts comments and indentation survive a set.

Deviation: one commit, not one per deletion as the issue asked.

model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
2026-09-21 19:25:01 +02:00
clawbot 753bc3ef60 Correct remote layout and privacy docs for hashed snapshot keys (closes #67)
check / check (pull_request) Failing after 1s
check / check (push) Successful in 3m11s
Three documents showed the remote layout with a plaintext snapshot ID as the metadata directory name, and docs/REPOSTRUCTURE.md blamed those IDs for the observable backup time. The store actually names each metadata directory with a one-way hash of the ID, so hostname and snapshot name are not visible; the backup time is, through the plaintext timestamp in the manifest, which is accepted behaviour.

README, ARCHITECTURE.md, docs/DATAMODEL.md and docs/REPOSTRUCTURE.md now show the hashed layout, the derivation is documented once, and the privacy section lists what the unencrypted manifest exposes. Two code comments that claimed the timestamp was hidden are corrected. No behaviour change.

Judgement call: docs/DATAMODEL.md was not named in the issue but had the same error.

model: claude-opus-4-8 (implementation, review); claude-fable-5-1 (merge)
2026-09-21 19:24:44 +02:00
15 changed files with 357 additions and 252 deletions
+7 -1
View File
@@ -366,11 +366,17 @@ bucket/
│ └── {full-hash} # Compressed+encrypted blob │ └── {full-hash} # Compressed+encrypted blob
└── metadata/ └── metadata/
└── {snapshot-id}/ └── {remote-key}/
├── db.zst.age # Encrypted binary SQLite database ├── db.zst.age # Encrypted binary SQLite database
└── manifest.json.zst # Blob list (for pruning/verification) └── manifest.json.zst # Blob list (for pruning/verification)
``` ```
The `{remote-key}` directory name is a one-way double SHA-256 hash of the human
snapshot ID, so the human ID (hostname, snapshot name, timestamp) is never
written to the store as a directory name. See
[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#remote-key-derivation) for the
derivation and a worked example.
## Thread Safety ## Thread Safety
- `Packer`: Thread-safe via mutex. Multiple goroutines can call `AddChunk()`. - `Packer`: Thread-safe via mutex. Multiple goroutines can call `AddChunk()`.
+5 -5
View File
@@ -72,11 +72,11 @@ RUN [ -n "$CHECK_EPOCH" ] || exit 1
# running, and exits 0 reporting `0 issues.` on a tree the real config # running, and exits 0 reporting `0 issues.` on a tree the real config
# fails. Demonstrated on this repo at this pin, recorded on # fails. Demonstrated on this repo at this pin, recorded on
# https://git.eeqj.de/sneak/vaultik/pulls/114: with a planted # https://git.eeqj.de/sneak/vaultik/pulls/114: with a planted
# over-length line, `script/lint` exits 1 naming the `lll` finding with # over-length line, `script/lint` exits 1 naming the `revive` finding
# `linters:` and exits 0 with `linterz:`. A set-but-ineffective config # with `linters:` and exits 0 with `linterz:`. A set-but-ineffective
# quietly falling back to defaults is precisely the false-green class # config quietly falling back to defaults is precisely the false-green
# this gate exists to eliminate, so it must not sit in the gate's own # class this gate exists to eliminate, so it must not sit in the gate's
# configuration. # own configuration.
# #
# `config verify` catches it, and it does so OFFLINE at this pinned # `config verify` catches it, and it does so OFFLINE at this pinned
# version -- verified, not assumed. Under `docker run --network none` # version -- verified, not assumed. Under `docker run --network none`
+14 -4
View File
@@ -344,7 +344,7 @@ both are set.
├── blobs/ ├── blobs/
│ └── <aa>/<bb>/<full_blob_hash> │ └── <aa>/<bb>/<full_blob_hash>
└── metadata/ └── metadata/
└── <snapshot_id>/ └── <remote-key>/
├── db.zst.age # Encrypted binary SQLite database ├── db.zst.age # Encrypted binary SQLite database
└── manifest.json.zst # Unencrypted blob list (for pruning) └── manifest.json.zst # Unencrypted blob list (for pruning)
``` ```
@@ -355,8 +355,18 @@ both are set.
* `manifest.json.zst` is an unencrypted compressed JSON blob list, enabling * `manifest.json.zst` is an unencrypted compressed JSON blob list, enabling
pruning without the private key pruning without the private key
Snapshot IDs follow the format `<hostname>_<snapshot-name>_<RFC3339-timestamp>` Snapshot IDs follow the human-readable format
(e.g. `server1_home_2025-06-01T12:00:00Z`). `<hostname>_<snapshot-name>_<RFC3339-timestamp>` (e.g.
`server1_home_2025-06-01T12:00:00Z`), but this ID is never written to the
destination store in plaintext. Each snapshot's metadata directory is named
with its `<remote-key>`, a one-way double SHA-256 hash of the ID, so a listing
of the store reveals no hostname or snapshot name. The backup time is not
hidden: manifest.json.zst carries a plaintext timestamp, and object
modification times are visible at the storage layer regardless. For example,
`server1_home_2025-06-01T12:00:00Z` is stored under
`metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`.
See [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#remote-key-derivation) for the
derivation.
### data flow ### data flow
@@ -373,7 +383,7 @@ Snapshot IDs follow the format `<hostname>_<snapshot-name>_<RFC3339-timestamp>`
**restore:** **restore:**
1. Download and decrypt `metadata/<snapshot_id>/db.zst.age` 1. Download and decrypt `metadata/<remote-key>/db.zst.age`
2. Open the binary SQLite database 2. Open the binary SQLite database
3. Query files (optionally filtered by paths) 3. Query files (optionally filtered by paths)
4. Download and decrypt required blobs 4. Download and decrypt required blobs
+212 -30
View File
@@ -1,6 +1,8 @@
package main_test package main_test
import ( import (
"errors"
"fmt"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
@@ -252,29 +254,64 @@ func TestNoHostLintPathRemains(t *testing.T) {
} }
name := filepath.Join("script", entry.Name()) name := filepath.Join("script", entry.Name())
for _, line := range shellCode(readRepoFile(t, name)) {
lines, err := shellCode(readRepoFile(t, name))
require.NoError(t, err, "scanning %s", name)
for _, line := range lines {
assertLinterIsContainerised(t, name, line) assertLinterIsContainerised(t, name, line)
} }
} }
} }
// assertLinterIsContainerised fails if the line runs the linter without // assertLinterIsContainerised fails unless every command that names the
// handing it to docker first. Position matters: docker has to come // linter on this joined line is a docker command. Merely mentioning
// before the binary, or the line is running the host linter and merely // docker somewhere on the line is not enough; see linterRunsInDocker.
// mentioning docker afterwards.
func assertLinterIsContainerised(t *testing.T, name, line string) { func assertLinterIsContainerised(t *testing.T, name, line string) {
t.Helper() t.Helper()
at := strings.Index(line, linterBinary) assert.True(t, linterRunsInDocker(line),
if at < 0 { "%s runs %s outside a container; every command that names the"+
return " linter must begin with docker (line: %s)", name, linterBinary,
line)
}
// linterRunsInDocker reports whether the linter, wherever it appears on
// this joined shell line, is only ever the argument of a docker command.
// The line is cut into the simple commands the shell would run -- on
// `;`, `&&`, `||` and `|` -- and every command that names the linter
// must begin with `docker`. This is what distinguishes the one
// legitimate invocation, script/lint-fix's `docker run ... golangci-lint
// run ...`, from evasions like `docker info; golangci-lint run` or
// `docker info || golangci-lint run`, where the linter sits in a command
// of its own that docker does not introduce.
func linterRunsInDocker(line string) bool {
for _, command := range splitShellCommands(line) {
if !strings.Contains(command, linterBinary) {
continue
}
if !strings.HasPrefix(strings.TrimSpace(command), "docker") {
return false
}
} }
docker := strings.Index(line, "docker") return true
}
assert.True(t, docker >= 0 && docker < at, // splitShellCommands breaks a joined shell line into the separate simple
"%s runs %s on the host; every lint run happens in a container"+ // commands the shell would run, cutting at the `;`, `&&`, `||` and `|`
" (line: %s)", name, linterBinary, line) // operators (`||` before `|`, so the two-character operator is not split
// twice). It is deliberately blind to quoting and to `$(...)`: no line
// under guard puts one of these operators inside a string, and a scan
// that tried to account for that would be the kind of half-parser this
// file avoids.
func splitShellCommands(line string) []string {
for _, op := range []string{"&&", "||", "|", ";"} {
line = strings.ReplaceAll(line, op, "\n")
}
return strings.Split(line, "\n")
} }
// TestShellCodeSeesCodeAndNotProse keeps the scanner above honest. It // TestShellCodeSeesCodeAndNotProse keeps the scanner above honest. It
@@ -287,20 +324,94 @@ func assertLinterIsContainerised(t *testing.T, name, line string) {
func TestShellCodeSeesCodeAndNotProse(t *testing.T) { func TestShellCodeSeesCodeAndNotProse(t *testing.T) {
t.Parallel() t.Parallel()
// A `<<` inside quotes is not a here-document, so the code after it
// is still scanned; a real `<<EOF` opens one and its body is dropped.
script := strings.Join([]string{ script := strings.Join([]string{
"#!/bin/sh", "#!/bin/sh",
"# a comment naming golangci-lint", "# a comment naming golangci-lint",
"cat >&2 <<EOF", "cat >&2 <<EOF",
"prose naming golangci-lint, printed not executed", "prose naming golangci-lint, printed not executed",
"EOF", "EOF",
`echo "a left shift << is not a here-document"`,
"docker run --rm \\", "docker run --rm \\",
" \"$image\" \\", " \"$image\" \\",
" golangci-lint run ./...", " golangci-lint run ./...",
}, "\n") }, "\n")
lines, err := shellCode(script)
require.NoError(t, err)
assert.Equal(t, assert.Equal(t,
[]string{"cat >&2 <<EOF", `docker run --rm "$image" golangci-lint run ./...`}, []string{
shellCode(script)) "cat >&2 <<EOF",
`echo "a left shift << is not a here-document"`,
`docker run --rm "$image" golangci-lint run ./...`,
},
lines)
// A `<<` inside an inline comment is not a here-document either: an
// unquoted, word-initial `#` begins a comment that runs to end of
// line, so the `<< STOP` is prose. The code that follows is still
// scanned -- here, a host golangci-lint that TestNoHostLintPathRemains
// must then see rather than have swallowed. The fake terminator even
// recurs later as a line of its own; a phantom here-document would
// swallow everything up to it silently, past the end-of-file error
// that only catches a terminator which never recurs.
inlineComment := strings.Join([]string{
": # housekeeping marker << STOP",
"golangci-lint run --config .golangci.yml ./...",
"STOP",
}, "\n")
lines, err = shellCode(inlineComment)
require.NoError(t, err)
assert.Equal(t,
[]string{
": # housekeeping marker << STOP",
"golangci-lint run --config .golangci.yml ./...",
"STOP",
},
lines)
// A here-document still open at end of file must be a loud error,
// not a silent truncation of everything the scanner has yet to see.
unterminated := strings.Join([]string{
"cat <<EOF",
"body line naming golangci-lint, no terminator follows",
}, "\n")
_, err = shellCode(unterminated)
require.Error(t, err)
}
// TestLinterCommandMustBeginWithDocker pins the property that a mention
// of docker somewhere on the line is not enough: the command that
// actually runs the linter has to be a docker command. The two evasions
// from the issue place the linter in a command of its own, joined to a
// harmless docker command by `;` or `||`; both must be rejected. The
// containerised invocation script/lint-fix writes -- docker run with the
// linter as its argument -- must still be accepted.
func TestLinterCommandMustBeginWithDocker(t *testing.T) {
t.Parallel()
rejected := []string{
"docker info >/dev/null; golangci-lint run ./...",
"docker info || golangci-lint run ./...",
"docker build . && golangci-lint run ./... | tee log",
}
for _, line := range rejected {
assert.False(t, linterRunsInDocker(line),
"a linter command docker does not introduce must be rejected: %s",
line)
}
accepted := []string{
`docker run --rm "$image" golangci-lint run ./...`,
`docker run --rm --user x --volume "$ROOT:/src" img golangci-lint run --fix ./...`,
}
for _, line := range accepted {
assert.True(t, linterRunsInDocker(line),
"a docker-introduced linter command must be accepted: %s", line)
}
} }
// assertEpochExpandedInto fails unless some instruction runs the named // assertEpochExpandedInto fails unless some instruction runs the named
@@ -410,7 +521,9 @@ func indexContaining(found []string, want string) int {
// shellCode returns a POSIX shell script's executable lines: comments // shellCode returns a POSIX shell script's executable lines: comments
// dropped, here-document bodies dropped, and backslash continuations // dropped, here-document bodies dropped, and backslash continuations
// joined so a multi-line command is a single string. Whitespace is // joined so a multi-line command is a single string. Whitespace is
// collapsed, as it is for Dockerfile instructions. // collapsed, as it is for Dockerfile instructions. A here-document left
// open at end of file is an error rather than a silent truncation of
// everything after its opener.
// //
// Both exclusions are load-bearing rather than tidiness. The scripts // Both exclusions are load-bearing rather than tidiness. The scripts
// name golangci-lint in prose to state that the host binary is never // name golangci-lint in prose to state that the host binary is never
@@ -418,7 +531,11 @@ func indexContaining(found []string, want string) int {
// container invocation -- script/lint-fix's `docker run`, whose linter // container invocation -- script/lint-fix's `docker run`, whose linter
// command sits several lines below the word `docker` -- be recognised // command sits several lines below the word `docker` -- be recognised
// as containerised. // as containerised.
func shellCode(contents string) []string { //
// This is a text scan, not a shell: it cannot see a linter name
// assembled at runtime, one split across a continuation, a script in a
// subdirectory of script/, or anything in the Makefile.
func shellCode(contents string) ([]string, error) {
var ( var (
out []string out []string
joined string joined string
@@ -452,23 +569,88 @@ func shellCode(contents string) []string {
joined = "" joined = ""
} }
return out if terminate != "" {
} return nil, fmt.Errorf("%w: terminator %q", errUnterminatedHeredoc,
terminate)
// heredocTerminator returns the terminator of the here-document a
// command opens, or "" if it opens none. Only the first on a line is
// recognised; nothing in script/ opens two.
func heredocTerminator(line string) string {
_, after, opens := strings.Cut(line, "<<")
if !opens {
return ""
} }
// `<<-` strips leading tabs from the body; the terminator word is return out, nil
// the same either way, and callers compare against trimmed lines. }
word, _, _ := strings.Cut(strings.TrimPrefix(after, "-"), " ")
return strings.Trim(word, `'"`) // errUnterminatedHeredoc is what shellCode returns when a here-document
// is still open at end of file. Its callers require its absence, so an
// unterminated body -- which would otherwise be swallowed silently --
// fails the guard loudly.
var errUnterminatedHeredoc = errors.New(
"here-document opened but never closed before end of file")
// heredocTerminator returns the delimiter word of the here-document the
// command opens, or "" if it opens none. A `<<` only opens one when it
// is a real redirection: outside single and double quotes, not in an
// inline comment, and followed by a delimiter word. A `<<` inside a
// quoted string, past an unquoted word-initial `#` (which begins a
// comment that runs to end of line), or in an arithmetic left shift
// like `$((x << 2))`, is not a here-document; the first two are cases
// this guards, the last appears in no script here. Only the first
// opener on a line is recognised; nothing in script/ opens two.
func heredocTerminator(line string) string {
var quote byte // 0 when outside quotes, else '\'' or '"'
for i := 0; i+1 < len(line); i++ {
c := line[i]
switch {
case quote != 0:
if c == quote {
quote = 0
}
case c == '\'' || c == '"':
quote = c
case c == '#' && (i == 0 || line[i-1] == ' '):
// A word-initial `#` starts a comment; the rest of the
// line, `<<` included, is prose, not a redirection.
return ""
case c == '<' && line[i+1] == '<':
return heredocWord(line[i+2:])
}
}
return ""
}
// heredocWord extracts the delimiter that follows `<<` or `<<-`: it drops
// an optional `-`, skips blanks, then reads the delimiter -- quoted or
// bare -- and returns it with quotes removed. `<<-'EOF'` and `<< EOF`
// both yield "EOF". It returns "" when no word follows, so a bare `<<`
// opens nothing.
func heredocWord(after string) string {
after = strings.TrimLeft(strings.TrimPrefix(after, "-"), " \t")
var (
word strings.Builder
quote byte
)
for i := range len(after) {
c := after[i]
switch {
case quote != 0:
if c == quote {
quote = 0
} else {
word.WriteByte(c)
}
case c == '\'' || c == '"':
quote = c
case c == ' ' || c == '\t':
return word.String()
default:
word.WriteByte(c)
}
}
return word.String()
} }
// readRepoFile reads a file by its path relative to the repository // readRepoFile reads a file by its path relative to the repository
+4 -2
View File
@@ -194,8 +194,10 @@ After a snapshot is completed:
2. Clean temporary database to contain only current snapshot data 2. Clean temporary database to contain only current snapshot data
3. Export to SQL dump using sqlite3 3. Export to SQL dump using sqlite3
4. Compress with zstd and encrypt with age 4. Compress with zstd and encrypt with age
5. Upload to S3 as `metadata/{snapshot-id}/db.zst.age` 5. Upload to S3 as `metadata/{remote-key}/db.zst.age`
6. Generate blob manifest and upload as `metadata/{snapshot-id}/manifest.json.zst` 6. Generate blob manifest and upload as `metadata/{remote-key}/manifest.json.zst`
The `{remote-key}` directory name is a one-way hash of the human snapshot ID, so the ID is never written to the store in plaintext; see [REPOSTRUCTURE.md](REPOSTRUCTURE.md#remote-key-derivation).
### 4. Restore Process ### 4. Restore Process
+37 -17
View File
@@ -17,11 +17,13 @@ Vaultik stores all backup data in an S3-compatible object store. The repository
│ └── <hash[2:4]>/ │ └── <hash[2:4]>/
│ └── <full-hash> │ └── <full-hash>
└── metadata/ └── metadata/
└── <snapshot-id>/ └── <remote-key>/
├── db.zst.age ├── db.zst.age
└── manifest.json.zst └── manifest.json.zst
``` ```
The metadata subdirectory is named with the **remote key**, a one-way hash of the snapshot ID, not with the human-readable snapshot ID itself. See [Remote Key Derivation](#remote-key-derivation).
## Blobs Directory (`blobs/`) ## Blobs Directory (`blobs/`)
### Structure ### Structure
@@ -40,9 +42,11 @@ Blobs contain the actual file data from backups and must be encrypted for securi
## Metadata Directory (`metadata/`) ## Metadata Directory (`metadata/`)
Each snapshot has its own subdirectory named with the snapshot ID. Each snapshot has its own subdirectory. The directory is **not** named with the human-readable snapshot ID; it is named with the remote key — a one-way hash of that ID. The human ID is never written to the destination store as a directory name (see [Remote Key Derivation](#remote-key-derivation)).
### Snapshot ID Format ### Snapshot ID Format
The human-readable snapshot ID is used in CLI arguments, log lines, and the local database. It is not written to the destination store.
- **Format**: `<hostname>_<snapshot-name>_<RFC3339>` (or `<hostname>_<RFC3339>` if no - **Format**: `<hostname>_<snapshot-name>_<RFC3339>` (or `<hostname>_<RFC3339>` if no
name was specified) name was specified)
- **Example**: `laptop_home_2024-01-15T14:30:52Z` - **Example**: `laptop_home_2024-01-15T14:30:52Z`
@@ -51,6 +55,19 @@ Each snapshot has its own subdirectory named with the snapshot ID.
- Snapshot name from the configured `snapshots:` map (optional) - Snapshot name from the configured `snapshots:` map (optional)
- RFC3339 UTC timestamp - RFC3339 UTC timestamp
This ID reveals the hostname, the configured snapshot name, and the backup time, so it is never used as the on-disk directory name — the remote key is used instead.
### Remote Key Derivation
The remote key is `hex(SHA256(SHA256("vaultik|" + snapshot-id)))`: a double SHA-256 over the snapshot ID, with a `vaultik|` domain-separation prefix. The result is a 64-character hex string with no structure a remote observer can reverse. Implemented in `internal/snapshot/remotekey.go`.
Worked example:
- Snapshot ID: `server1_home_2025-06-01T12:00:00Z`
- Remote key: `17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa`
- Directory: `metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`
Because the hash is one-way, a listing of the destination store reveals neither the hostname nor the snapshot name of any backup. The same remote key is stored in the manifest's `snapshot_id` field.
### Files in Each Snapshot Directory ### Files in Each Snapshot Directory
#### `db.zst.age` - Encrypted Database #### `db.zst.age` - Encrypted Database
@@ -68,16 +85,17 @@ Each snapshot has its own subdirectory named with the snapshot ID.
- **Structure**: - **Structure**:
```json ```json
{ {
"snapshot_id": "laptop_home_2024-01-15T14:30:52Z", "snapshot_id": "17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa",
"timestamp": "2024-01-15T14:30:52Z", "timestamp": "2025-06-01T12:00:00Z",
"blob_count": 42, "blob_count": 42,
"total_compressed_size": 1048576,
"blobs": [ "blobs": [
"cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678", { "hash": "cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678", "compressed_size": 24576 },
"deadbeef1234567890abcdef1234567890abcdef1234567890abcdef12345678", { "hash": "deadbeef1234567890abcdef1234567890abcdef1234567890abcdef12345678", "compressed_size": 32768 }
...
] ]
} }
``` ```
`snapshot_id` is the remote key (a hash), not the human ID; `timestamp` is written in the clear.
### Why Manifest is Unencrypted ### Why Manifest is Unencrypted
The manifest must be readable without the private key to enable: The manifest must be readable without the private key to enable:
@@ -86,7 +104,7 @@ The manifest must be readable without the private key to enable:
3. **Verification** - Checking blob existence without decryption 3. **Verification** - Checking blob existence without decryption
4. **Cross-snapshot deduplication analysis** - Finding shared blobs between snapshots 4. **Cross-snapshot deduplication analysis** - Finding shared blobs between snapshots
The manifest only contains blob hashes, not file names or any other sensitive information. The manifest contains the remote key, the backup timestamp, the blob count and total compressed size, and each blob's hash and compressed size. It contains no file names, paths, or other decrypted metadata.
## Security Considerations ## Security Considerations
@@ -96,19 +114,21 @@ The manifest only contains blob hashes, not file names or any other sensitive in
- **File-to-chunk mappings** (in db.zst.age) - **File-to-chunk mappings** (in db.zst.age)
### What's Not Encrypted ### What's Not Encrypted
- **Blob hashes** (in manifest.json.zst) - **The remote key** — directory names and the manifest `snapshot_id`, a one-way hash of the snapshot ID (see [Remote Key Derivation](#remote-key-derivation))
- **Snapshot IDs** (directory names) - **The backup timestamp** (in manifest.json.zst)
- **Blob count per snapshot** (in manifest.json.zst) - **Blob hashes and their compressed sizes** (in manifest.json.zst)
- **Blob count and total compressed size per snapshot** (in manifest.json.zst)
### Privacy Implications ### Privacy Implications
From the unencrypted data, an observer can determine: From the unencrypted data, an observer of the destination store can determine:
- When backups were taken (from snapshot IDs) - **When each backup was taken** — not from the directory name, which is a one-way hash, but from the plaintext `timestamp` field in manifest.json.zst, which is published in the clear
- Which hostname created backups (from snapshot IDs) - How many blobs each snapshot references, and the total compressed size
- How many blobs each snapshot references - The compressed size of each blob, and which blobs are shared between snapshots (deduplication patterns)
- Which blobs are shared between snapshots (deduplication patterns)
- The size of each encrypted blob Together these give an observer a timing-and-size profile of every snapshot. This is an accepted, documented property of the format, not a defect: the manifest is unencrypted so that pruning can run without the private key, and the timing channel could not be closed by encrypting it anyway — object creation times and per-object sizes stay visible at the storage layer on both `s3://` and `file://` destinations regardless.
An observer cannot determine: An observer cannot determine:
- The hostname or snapshot name of any backup (the directory name and the manifest `snapshot_id` are one-way hashes of the human ID)
- File names or paths - File names or paths
- File contents - File contents
- File permissions or ownership - File permissions or ownership
+30 -1
View File
@@ -1,6 +1,7 @@
package cli package cli
import ( import (
"bytes"
"errors" "errors"
"fmt" "fmt"
"os" "os"
@@ -24,6 +25,11 @@ const configSetArgs = 2
// parent config dirs (e.g. ~/.config) are conventionally traversable. // parent config dirs (e.g. ~/.config) are conventionally traversable.
const configDirMode = 0o755 const configDirMode = 0o755
// configYAMLIndent matches the 2-space indentation of defaultConfigTemplate,
// so `config set` writes the file back with the same indentation rather than
// yaml.Marshal's 4-space default.
const configYAMLIndent = 2
var ( var (
errConfigExists = errors.New("config file already exists") errConfigExists = errors.New("config file already exists")
errEmptyConfig = errors.New("empty config file") errEmptyConfig = errors.New("empty config file")
@@ -381,7 +387,7 @@ Examples:
return err return err
} }
out, err := yaml.Marshal(root) out, err := marshalConfigYAML(root)
if err != nil { if err != nil {
return fmt.Errorf("marshaling config: %w", err) return fmt.Errorf("marshaling config: %w", err)
} }
@@ -405,6 +411,29 @@ Examples:
} }
} }
// marshalConfigYAML renders a config document tree with 2-space indentation,
// matching defaultConfigTemplate. yaml.Marshal defaults to 4 spaces, which
// would reindent the whole file on the first `config set` despite the promise
// to preserve formatting.
func marshalConfigYAML(root *yaml.Node) ([]byte, error) {
var buf bytes.Buffer
enc := yaml.NewEncoder(&buf)
enc.SetIndent(configYAMLIndent)
err := enc.Encode(root)
if err != nil {
return nil, err
}
err = enc.Close()
if err != nil {
return nil, err
}
return buf.Bytes(), nil
}
// loadYAMLFile parses a YAML file into a yaml.Node document tree, // loadYAMLFile parses a YAML file into a yaml.Node document tree,
// which preserves comments and ordering for round-tripping. // which preserves comments and ordering for round-tripping.
func loadYAMLFile(path string) (*yaml.Node, error) { func loadYAMLFile(path string) (*yaml.Node, error) {
+41
View File
@@ -188,6 +188,47 @@ func TestYAMLPathSet(t *testing.T) {
} }
} }
// TestConfigSetPreservesFormatting asserts the `config set` write path
// (marshalConfigYAML) round-trips a 2-space-indented file without reindenting
// it to yaml.Marshal's 4-space default, and keeps comments.
func TestConfigSetPreservesFormatting(t *testing.T) {
t.Parallel()
root := parseTestYAML(t)
err := yamlPathSet(root, splitPath("s3.bucket"), "newbucket")
if err != nil {
t.Fatalf("set s3.bucket: %v", err)
}
out, err := marshalConfigYAML(root)
if err != nil {
t.Fatalf("marshal: %v", err)
}
text := string(out)
for _, want := range []string{"# top comment", "# inline comment"} {
if !contains(text, want) {
t.Errorf("round-tripped YAML dropped comment %q:\n%s", want, text)
}
}
// Nested map keys stay at 2-space indent; the bug reindented them to 4.
if !contains(text, "\n bucket: newbucket") {
t.Errorf("expected 2-space indent for s3.bucket, got:\n%s", text)
}
if contains(text, "\n bucket:") {
t.Errorf("s3.bucket reindented to 4 spaces:\n%s", text)
}
// Sequence items under a key also stay at 2 spaces.
if !contains(text, "\n - age1aaa") {
t.Errorf("expected 2-space indent for sequence item, got:\n%s", text)
}
}
func splitPath(s string) []string { func splitPath(s string) []string {
return strings.Split(s, ".") return strings.Split(s, ".")
} }
-12
View File
@@ -1,12 +0,0 @@
package cli
import "time"
// SnapshotInfo represents snapshot information for listing
//
//nolint:tagliatelle // snake_case is the established output format
type SnapshotInfo struct {
ID string `json:"id"`
Timestamp time.Time `json:"timestamp"`
CompressedSize int64 `json:"compressed_size"`
}
-67
View File
@@ -1,67 +0,0 @@
// Package models defines shared value types describing files, chunks,
// blobs, and snapshots as they move through the backup pipeline.
package models
import (
"time"
)
// FileInfo represents a file in the backup system
type FileInfo struct {
Path string
MTime time.Time
Size int64
}
// ChunkInfo represents a content-addressed chunk
type ChunkInfo struct {
Hash string // SHA256 hash
Size int64
Offset int64 // Offset within source file
}
// ChunkRef represents a reference to a chunk in a blob or file
type ChunkRef struct {
ChunkHash string
Offset int64
Length int64
}
// BlobInfo represents an encrypted blob containing multiple chunks
type BlobInfo struct {
Hash string // SHA256 hash of the blob content (content-addressable)
CreatedAt time.Time
Size int64
ChunkCount int
}
// Snapshot represents a backup snapshot
type Snapshot struct {
ID string // ISO8601 timestamp
Hostname string
Version string
CreatedAt time.Time
FileCount int64
ChunkCount int64
BlobCount int64
TotalSize int64
MetadataSize int64
}
// SnapshotMetadata contains the full metadata for a snapshot
type SnapshotMetadata struct {
Snapshot *Snapshot
Files map[string]*FileInfo
Chunks map[string]*ChunkInfo
Blobs map[string]*BlobInfo
FileChunks map[string][]*ChunkRef // path -> chunks
BlobChunks map[string][]*ChunkRef // blob hash -> chunks
}
// Chunk represents a data chunk for processing
type Chunk struct {
Data []byte
Hash string
Offset int64
Length int64
}
-58
View File
@@ -1,58 +0,0 @@
package models_test
import (
"testing"
"time"
"sneak.berlin/go/vaultik/internal/models"
)
// TestModelsCompilation ensures all model types can be instantiated
func TestModelsCompilation(t *testing.T) {
t.Parallel()
// This test primarily serves as a compilation test
// to ensure all types are properly defined
// Test FileInfo
fi := &models.FileInfo{
Path: "/test/file.txt",
MTime: time.Now(),
Size: 1024,
}
if fi.Path != "/test/file.txt" {
t.Errorf("FileInfo.Path not set correctly")
}
// Test ChunkInfo
ci := &models.ChunkInfo{
Hash: "abc123",
Size: 512,
Offset: 0,
}
if ci.Hash != "abc123" {
t.Errorf("ChunkInfo.Hash not set correctly")
}
// Test BlobInfo
bi := &models.BlobInfo{
Hash: "blob123",
CreatedAt: time.Now(),
Size: 1024,
ChunkCount: 2,
}
if bi.Hash != "blob123" {
t.Errorf("BlobInfo.Hash not set correctly")
}
// Test Snapshot
s := &models.Snapshot{
ID: "2024-01-01T00:00:00Z",
Hostname: "test-host",
Version: "1.0.0",
CreatedAt: time.Now(),
}
if s.ID != "2024-01-01T00:00:00Z" {
t.Errorf("Snapshot.ID not set correctly")
}
}
+3 -2
View File
@@ -22,8 +22,9 @@ const remoteKeyPrefix = "vaultik|"
// //
// - the "metadata/<remote-key>/..." subdirectory on the storage // - the "metadata/<remote-key>/..." subdirectory on the storage
// backend so a directory listing of the bucket / file:// dest // backend so a directory listing of the bucket / file:// dest
// doesn't reveal hostnames, configured snapshot names, or backup // doesn't reveal hostnames or configured snapshot names. (The
// timestamps; // backup time is not hidden: the manifest.json.zst inside that
// directory carries a plaintext RFC3339 timestamp.)
// - the `snapshot_id` field of the unencrypted manifest.json.zst // - the `snapshot_id` field of the unencrypted manifest.json.zst
// for the same reason; // for the same reason;
// - any code path that needs to translate a known local snapshot ID // - any code path that needs to translate a known local snapshot ID
+4 -2
View File
@@ -840,8 +840,10 @@ func (sm *SnapshotManager) generateBlobManifest(
} }
// Create manifest. SnapshotID in the unencrypted manifest is the // Create manifest. SnapshotID in the unencrypted manifest is the
// double-SHA256 remote key, not the human ID, so the public bytes // double-SHA256 remote key (see RemoteSnapshotKey), not the human ID,
// don't reveal hostname/snapshot-name/timestamp metadata. // so neither this field nor the directory name reveals the hostname or
// snapshot name. Timestamp below is written in the clear, so the backup
// time is observable to anyone who can read the manifest.
manifest := &Manifest{ manifest := &Manifest{
SnapshotID: RemoteSnapshotKey(snapshotID), SnapshotID: RemoteSnapshotKey(snapshotID),
Timestamp: time.Now().UTC().Format(time.RFC3339), Timestamp: time.Now().UTC().Format(time.RFC3339),
-27
View File
@@ -1,27 +0,0 @@
# Vaultik test configuration
hostname: test-host
index_path: /tmp/vaultik-test/index.db
source_dirs:
- /tmp/vaultik-test/source
# S3 configuration
s3:
endpoint: http://localhost:19000 # gofakes3 test endpoint
bucket: test-bucket
prefix: test-
access_key_id: test-key
secret_access_key: test-secret
region: us-east-1
# Chunking configuration
chunk_size: 65536 # 64KB average chunk size
min_chunk_size: 32768 # 32KB minimum
max_chunk_size: 131072 # 128KB maximum
blob_size: 1048576 # 1MB blobs for testing
# Compression
compression_level: 3
# Encryption
# age_recipients:
# - age1qyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqs3mw88h
-24
View File
@@ -1,24 +0,0 @@
age_recipients:
- age1278m9q7dp3chsh2dcy82qk27v047zywyvtxwnj4cvt0z65jw6a7q5dqhfj # sneak's long term age key
- age1ezrjmfpwsc95svdg0y54mums3zevgzu0x0ecq2f7tp8a05gl0sjq9q9wjg # insecure integration test key
source_dirs:
- /tmp/vaultik-test-source
exclude:
- '*.log'
- '*.tmp'
- '.git'
- 'node_modules'
s3:
endpoint: http://ber1app1.local:3900/
bucket: vaultik-integration-test
prefix: test-host/
access_key_id: GKbc8e6d35fdf50847f155aca5
secret_access_key: 217046bee47c050301e3cc13e3cba1a8a943cf5f37f8c7979c349c5254441d18
region: us-east-1
use_ssl: false
part_size: 5242880 # 5MB
index_path: /tmp/vaultik-integration-test.sqlite
chunk_size: 10MB
blob_size_limit: 10GB
compression_level: 3
hostname: test-host