VACUUM snapshot metadata through the sqlite driver, not a CLI (closes #120 )

snapshot create vacuumed the exported metadata database by shelling out to a sqlite3 binary, so a backup failed at the very end on any host without that CLI. vacuumDatabase now opens the database with the modernc.org/sqlite driver and runs VACUUM through it, outside any transaction; internal/snapshot no longer imports os/exec. The database opens in WAL mode, so the close after VACUUM checkpoints the rewrite into the main file that is compressed and uploaded. A new test deletes marked rows, vacuums, and asserts the file shrank and no longer holds the deleted bytes. script/bootstrap and the Dockerfile test image stop installing the CLI. Disclosure: the Dockerfile test-image change was not exercised by local make check, which builds only the lint image. Model: opus-4-8
2026-09-21 13:11:22 +00:00
10 changed files with 157 additions and 84 deletions
@@ -366,17 +366,11 @@ bucket/
 │           └── {full-hash}          # Compressed+encrypted blob
 │
 └── metadata/
-    └── {remote-key}/
+    └── {snapshot-id}/
        ├── db.zst.age               # Encrypted binary SQLite database
        └── manifest.json.zst        # Blob list (for pruning/verification)
 ```

-The `{remote-key}` directory name is a one-way double SHA-256 hash of the human
-snapshot ID, so the human ID (hostname, snapshot name, timestamp) is never
-written to the store as a directory name. See
-[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#remote-key-derivation) for the
-derivation and a worked example.
-
 ## Thread Safety

 - `Packer`: Thread-safe via mutex. Multiple goroutines can call `AddChunk()`.
@@ -22,8 +22,10 @@ FROM golang:1.26.1-alpine@sha256:2389ebfa5b7f43eeafbd6be0c3700cc46690ef842ad962f

 ARG VERSION=dev

-# Install build dependencies for CGO (mattn/go-sqlite3) and sqlite3 CLI (tests)
-RUN apk add --no-cache make build-base sqlite
+# Build tooling: make, plus a C toolchain because `go test -race` needs cgo.
+# The sqlite driver is pure Go (modernc.org/sqlite), so no sqlite library or
+# CLI is required.
+RUN apk add --no-cache make build-base

 WORKDIR /src

@@ -344,7 +344,7 @@ both are set.
 ├── blobs/
 │   └── <aa>/<bb>/<full_blob_hash>
 └── metadata/
-    └── <remote-key>/
+    └── <snapshot_id>/
        ├── db.zst.age          # Encrypted binary SQLite database
        └── manifest.json.zst   # Unencrypted blob list (for pruning)
 ```
@@ -355,18 +355,8 @@ both are set.
 * `manifest.json.zst` is an unencrypted compressed JSON blob list, enabling
  pruning without the private key

-Snapshot IDs follow the human-readable format
-`<hostname>_<snapshot-name>_<RFC3339-timestamp>` (e.g.
-`server1_home_2025-06-01T12:00:00Z`), but this ID is never written to the
-destination store in plaintext. Each snapshot's metadata directory is named
-with its `<remote-key>`, a one-way double SHA-256 hash of the ID, so a listing
-of the store reveals no hostname or snapshot name. The backup time is not
-hidden: manifest.json.zst carries a plaintext timestamp, and object
-modification times are visible at the storage layer regardless. For example,
-`server1_home_2025-06-01T12:00:00Z` is stored under
-`metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`.
-See [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#remote-key-derivation) for the
-derivation.
+Snapshot IDs follow the format `<hostname>_<snapshot-name>_<RFC3339-timestamp>`
+(e.g. `server1_home_2025-06-01T12:00:00Z`).

 ### data flow

@@ -383,7 +373,7 @@ derivation.

 **restore:**

-1. Download and decrypt `metadata/<remote-key>/db.zst.age`
+1. Download and decrypt `metadata/<snapshot_id>/db.zst.age`
 2. Open the binary SQLite database
 3. Query files (optionally filtered by paths)
 4. Download and decrypt required blobs
@@ -613,7 +603,6 @@ regardless of color setting (emoji are not color).
  and the pre-commit hook both run it. A `golangci-lint` installed on
  `PATH` is not a substitute and is never used on a host, whatever its
  version.
-* `sqlite3` CLI, which the test suite shells out to
 * S3-compatible object storage (or local filesystem, or rclone remote)

 ## development workflow
@@ -644,8 +633,8 @@ standard: normalized scripts in `script/` are the entrypoints for the
 development workflow, and the Makefile targets are thin shims that call
 them. We provide:

-* `script/bootstrap` — install all development dependencies (go, sqlite3,
-  Go module download). It deliberately does not install `golangci-lint`;
+* `script/bootstrap` — install all development dependencies (go, Go
+  module download). It deliberately does not install `golangci-lint`;
  see `script/lint` below.
 * `script/setup` — make a fresh clone ready for development: runs
  `script/bootstrap`, then `script/install-precommit`
@@ -25,6 +25,14 @@ release" is exactly the contradiction

 # Completed Steps

+- 2026-09-21: Made `snapshot create` VACUUM the per-snapshot metadata
+  database through the `modernc.org/sqlite` driver instead of shelling
+  out to the external `sqlite` command-line binary (issue #120). A
+  backup no longer needs that binary on `PATH`, so `make check` passes
+  on a stock `go install` host; `script/bootstrap` and the `Dockerfile`
+  test image no longer install it, and a new test asserts the uploaded
+  database keeps no pages from deleted rows. Dropped the now-false note
+  on the 2026-08-07 entry below that said bootstrap installs it.
 - 2026-09-21: Made `.gitea/workflows/check.yml` run on pushes to `main`
  and `next` and on pull requests against either, so unit PRs (whose
  base is `next`) and `next` itself get a CI run instead of relying on a
@@ -529,7 +537,7 @@ release" is exactly the contradiction
  was green was wrong.
 - 2026-08-07: Added the standard `.golangci.yml` and `.editorconfig`
  (issue #59); lint findings under the new config are tracked in issue
-  #61. `script/bootstrap` now installs sqlite3 (needed by tests).
+  #61.
 - 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints,
  Makefile shims, README Entrypoints section
 - 2026-07-02: Consolidated CLI verbs, retired overlapping commands; bound
@@ -192,12 +192,10 @@ Tracks blob upload metrics.
 After a snapshot is completed:
 1. Copy database to temporary file
 2. Clean temporary database to contain only current snapshot data
-3. Export to SQL dump using sqlite3
+3. VACUUM the trimmed database so deleted rows leave no pages behind
 4. Compress with zstd and encrypt with age
-5. Upload to S3 as `metadata/{remote-key}/db.zst.age`
-6. Generate blob manifest and upload as `metadata/{remote-key}/manifest.json.zst`
-
-The `{remote-key}` directory name is a one-way hash of the human snapshot ID, so the ID is never written to the store in plaintext; see [REPOSTRUCTURE.md](REPOSTRUCTURE.md#remote-key-derivation).
+5. Upload to S3 as `metadata/{snapshot-id}/db.zst.age`
+6. Generate blob manifest and upload as `metadata/{snapshot-id}/manifest.json.zst`

 ### 4. Restore Process

@@ -17,13 +17,11 @@ Vaultik stores all backup data in an S3-compatible object store. The repository
 │       └── <hash[2:4]>/
 │           └── <full-hash>
 └── metadata/
-    └── <remote-key>/
+    └── <snapshot-id>/
        ├── db.zst.age
        └── manifest.json.zst
 ```

-The metadata subdirectory is named with the **remote key**, a one-way hash of the snapshot ID, not with the human-readable snapshot ID itself. See [Remote Key Derivation](#remote-key-derivation).
-
 ## Blobs Directory (`blobs/`)

 ### Structure
@@ -42,11 +40,9 @@ Blobs contain the actual file data from backups and must be encrypted for securi

 ## Metadata Directory (`metadata/`)

-Each snapshot has its own subdirectory. The directory is **not** named with the human-readable snapshot ID; it is named with the remote key — a one-way hash of that ID. The human ID is never written to the destination store as a directory name (see [Remote Key Derivation](#remote-key-derivation)).
+Each snapshot has its own subdirectory named with the snapshot ID.

 ### Snapshot ID Format
-
-The human-readable snapshot ID is used in CLI arguments, log lines, and the local database. It is not written to the destination store.
 - **Format**: `<hostname>_<snapshot-name>_<RFC3339>` (or `<hostname>_<RFC3339>` if no
  name was specified)
 - **Example**: `laptop_home_2024-01-15T14:30:52Z`
@@ -55,19 +51,6 @@ The human-readable snapshot ID is used in CLI arguments, log lines, and the loca
  - Snapshot name from the configured `snapshots:` map (optional)
  - RFC3339 UTC timestamp

-This ID reveals the hostname, the configured snapshot name, and the backup time, so it is never used as the on-disk directory name — the remote key is used instead.
-
-### Remote Key Derivation
-
-The remote key is `hex(SHA256(SHA256("vaultik|" + snapshot-id)))`: a double SHA-256 over the snapshot ID, with a `vaultik|` domain-separation prefix. The result is a 64-character hex string with no structure a remote observer can reverse. Implemented in `internal/snapshot/remotekey.go`.
-
-Worked example:
- Snapshot ID: `server1_home_2025-06-01T12:00:00Z`
- Remote key: `17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa`
- Directory: `metadata/17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa/`
-
-Because the hash is one-way, a listing of the destination store reveals neither the hostname nor the snapshot name of any backup. The same remote key is stored in the manifest's `snapshot_id` field.
-
 ### Files in Each Snapshot Directory

 #### `db.zst.age` - Encrypted Database
@@ -85,17 +68,16 @@ Because the hash is one-way, a listing of the destination store reveals neither
 - **Structure**:
 ```json
 {
-  "snapshot_id": "17f97bcde958748af076b926af59823943db59e80ce7170b40f124dfa28f64aa",
-  "timestamp": "2025-06-01T12:00:00Z",
+  "snapshot_id": "laptop_home_2024-01-15T14:30:52Z",
+  "timestamp": "2024-01-15T14:30:52Z",
  "blob_count": 42,
-  "total_compressed_size": 1048576,
  "blobs": [
-    { "hash": "cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678", "compressed_size": 24576 },
-    { "hash": "deadbeef1234567890abcdef1234567890abcdef1234567890abcdef12345678", "compressed_size": 32768 }
+    "cafebabe1234567890abcdef1234567890abcdef1234567890abcdef12345678",
+    "deadbeef1234567890abcdef1234567890abcdef1234567890abcdef12345678",
+    ...
  ]
 }
 ```
-`snapshot_id` is the remote key (a hash), not the human ID; `timestamp` is written in the clear.

 ### Why Manifest is Unencrypted
 The manifest must be readable without the private key to enable:
@@ -104,7 +86,7 @@ The manifest must be readable without the private key to enable:
 3. **Verification** - Checking blob existence without decryption
 4. **Cross-snapshot deduplication analysis** - Finding shared blobs between snapshots

-The manifest contains the remote key, the backup timestamp, the blob count and total compressed size, and each blob's hash and compressed size. It contains no file names, paths, or other decrypted metadata.
+The manifest only contains blob hashes, not file names or any other sensitive information.

 ## Security Considerations

@@ -114,21 +96,19 @@ The manifest contains the remote key, the backup timestamp, the blob count and t
 - **File-to-chunk mappings** (in db.zst.age)

 ### What's Not Encrypted
- **The remote key** — directory names and the manifest `snapshot_id`, a one-way hash of the snapshot ID (see [Remote Key Derivation](#remote-key-derivation))
- **The backup timestamp** (in manifest.json.zst)
- **Blob hashes and their compressed sizes** (in manifest.json.zst)
- **Blob count and total compressed size per snapshot** (in manifest.json.zst)
+- **Blob hashes** (in manifest.json.zst)
+- **Snapshot IDs** (directory names)
+- **Blob count per snapshot** (in manifest.json.zst)

 ### Privacy Implications
-From the unencrypted data, an observer of the destination store can determine:
- **When each backup was taken** — not from the directory name, which is a one-way hash, but from the plaintext `timestamp` field in manifest.json.zst, which is published in the clear
- How many blobs each snapshot references, and the total compressed size
- The compressed size of each blob, and which blobs are shared between snapshots (deduplication patterns)
-
-Together these give an observer a timing-and-size profile of every snapshot. This is an accepted, documented property of the format, not a defect: the manifest is unencrypted so that pruning can run without the private key, and the timing channel could not be closed by encrypting it anyway — object creation times and per-object sizes stay visible at the storage layer on both `s3://` and `file://` destinations regardless.
+From the unencrypted data, an observer can determine:
+- When backups were taken (from snapshot IDs)
+- Which hostname created backups (from snapshot IDs)
+- How many blobs each snapshot references
+- Which blobs are shared between snapshots (deduplication patterns)
+- The size of each encrypted blob

 An observer cannot determine:
- The hostname or snapshot name of any backup (the directory name and the manifest `snapshot_id` are one-way hashes of the human ID)
 - File names or paths
 - File contents
 - File permissions or ownership
@@ -22,9 +22,8 @@ const remoteKeyPrefix = "vaultik|"
 //
 //   - the "metadata/<remote-key>/..." subdirectory on the storage
 //     backend so a directory listing of the bucket / file:// dest
-//     doesn't reveal hostnames or configured snapshot names. (The
-//     backup time is not hidden: the manifest.json.zst inside that
-//     directory carries a plaintext RFC3339 timestamp.)
+//     doesn't reveal hostnames, configured snapshot names, or backup
+//     timestamps;
 //   - the `snapshot_id` field of the unencrypted manifest.json.zst
 //     for the same reason;
 //   - any code path that needs to translate a known local snapshot ID
@@ -44,7 +44,6 @@ import (
 	"errors"
 	"fmt"
 	"io"
-	"os/exec"
 	"path/filepath"
 	"strings"
 	"time"
@@ -669,14 +668,31 @@ func (sm *SnapshotManager) collectCleanupStats(

 // vacuumDatabase runs VACUUM on the database to remove deleted data and compact
 // This is critical for security - ensures no stale/deleted data pages are uploaded
+//
+// VACUUM runs through the modernc.org/sqlite driver, on a freshly opened
+// connection with no transaction in flight (VACUUM cannot run inside one).
+// The database opens in WAL mode, so VACUUM's rewrite lands in the WAL; the
+// checkpoint on Close flushes it into the main file, which is the file we
+// then compress and upload.
 func (sm *SnapshotManager) vacuumDatabase(ctx context.Context, dbPath string) error {
 	log.Debug("Running VACUUM on database", "path", dbPath)
-	//nolint:gosec // G204: fixed argv; dbPath is our own temp file path
-	cmd := exec.CommandContext(ctx, "sqlite3", dbPath, "VACUUM;")

-	output, err := cmd.CombinedOutput()
+	db, err := database.New(ctx, dbPath)
 	if err != nil {
-		return fmt.Errorf("running VACUUM: %w (output: %s)", err, string(output))
+		return fmt.Errorf("opening database for VACUUM: %w", err)
+	}
+
+	defer func() {
+		cerr := db.Close()
+		if cerr != nil {
+			log.Debug("Failed to close database after VACUUM",
+				"path", dbPath, "error", cerr)
+		}
+	}()
+
+	_, err = db.ExecWithLog(ctx, "VACUUM")
+	if err != nil {
+		return fmt.Errorf("running VACUUM: %w", err)
 	}

 	return nil
@@ -840,10 +856,8 @@ func (sm *SnapshotManager) generateBlobManifest(
 	}

 	// Create manifest. SnapshotID in the unencrypted manifest is the
-	// double-SHA256 remote key (see RemoteSnapshotKey), not the human ID,
-	// so neither this field nor the directory name reveals the hostname or
-	// snapshot name. Timestamp below is written in the clear, so the backup
-	// time is observable to anyone who can read the manifest.
+	// double-SHA256 remote key, not the human ID, so the public bytes
+	// don't reveal hostname/snapshot-name/timestamp metadata.
 	manifest := &Manifest{
 		SnapshotID:          RemoteSnapshotKey(snapshotID),
 		Timestamp:           time.Now().UTC().Format(time.RFC3339),
@@ -2,6 +2,7 @@
 package snapshot

 import (
+	"bytes"
 	"context"
 	"database/sql"
 	"io"
@@ -96,6 +97,97 @@ func verifyCleanedDB(
 	}
 }

+// TestVacuumDatabaseRemovesDeletedData proves the export path uploads a
+// compacted database: after rows carrying a recognizable marker are deleted
+// and vacuumDatabase runs, no page holding that marker survives in the file
+// on disk (the file compressFile later reads for upload).
+func TestVacuumDatabaseRemovesDeletedData(t *testing.T) {
+	log.Initialize(log.Config{})
+	t.Parallel()
+
+	ctx := context.Background()
+	fs := afero.NewOsFs()
+
+	tempDir := t.TempDir()
+	dbPath := filepath.Join(tempDir, "snapshot.db")
+
+	db, err := database.New(ctx, dbPath)
+	if err != nil {
+		t.Fatalf("failed to create database: %v", err)
+	}
+
+	// A marker distinctive enough that its presence in the raw file can only
+	// come from the rows inserted below.
+	marker := []byte("VACUUM_PROBE_DEADBEEF_DELETED_ROW")
+	payload := bytes.Repeat(marker, 128) // ~4 KiB per row
+
+	_, err = db.Conn().ExecContext(ctx,
+		"CREATE TABLE vacuum_probe (id INTEGER PRIMARY KEY, payload BLOB)")
+	if err != nil {
+		t.Fatalf("failed to create probe table: %v", err)
+	}
+
+	for range 512 {
+		_, err = db.Conn().ExecContext(ctx,
+			"INSERT INTO vacuum_probe (payload) VALUES (?)", payload)
+		if err != nil {
+			t.Fatalf("failed to insert probe row: %v", err)
+		}
+	}
+
+	_, err = db.Conn().ExecContext(ctx, "DELETE FROM vacuum_probe")
+	if err != nil {
+		t.Fatalf("failed to delete probe rows: %v", err)
+	}
+
+	// Close so the deletes reach the main file, mirroring the state
+	// prepareExportDB hands to vacuumDatabase.
+	err = db.Close()
+	if err != nil {
+		t.Fatalf("failed to close database: %v", err)
+	}
+
+	beforeInfo, err := fs.Stat(dbPath)
+	if err != nil {
+		t.Fatalf("failed to stat database before vacuum: %v", err)
+	}
+
+	beforeBytes, err := afero.ReadFile(fs, dbPath)
+	if err != nil {
+		t.Fatalf("failed to read database before vacuum: %v", err)
+	}
+
+	if !bytes.Contains(beforeBytes, marker) {
+		t.Fatalf("expected deleted-row data to linger before vacuum")
+	}
+
+	sm := &SnapshotManager{fs: fs}
+
+	err = sm.vacuumDatabase(ctx, dbPath)
+	if err != nil {
+		t.Fatalf("vacuumDatabase failed: %v", err)
+	}
+
+	afterBytes, err := afero.ReadFile(fs, dbPath)
+	if err != nil {
+		t.Fatalf("failed to read database after vacuum: %v", err)
+	}
+
+	if bytes.Contains(afterBytes, marker) {
+		t.Fatalf("deleted-row data survived vacuum in the uploaded file")
+	}
+
+	afterInfo, err := fs.Stat(dbPath)
+	if err != nil {
+		t.Fatalf("failed to stat database after vacuum: %v", err)
+	}
+
+	if afterInfo.Size() >= beforeInfo.Size() {
+		t.Fatalf("expected vacuum to shrink the file: before=%d after=%d",
+			beforeInfo.Size(), afterInfo.Size())
+	}
+}
+
 func TestCleanSnapshotDBEmptySnapshot(t *testing.T) {
 	// Initialize logger
 	log.Initialize(log.Config{})
@@ -114,9 +114,6 @@ main() {
    # from CI. Nothing on the host is ever used as a linter, at any
    # version, so installing one here would buy nothing.

-    # sqlite3 CLI: the test suite shells out to it (VACUUM).
-    if missing sqlite3; then pkg_install sqlite sqlite3 sqlite sqlite; fi
-
    # goreleaser, at the version pinned by script/install-goreleaser and
    # verified against a hardcoded sha256. Package managers are not used
    # for it: they ship whatever version they happen to carry, and the