Correct doc and help sentences that are false about the code (closes #233 )

A blob is written in full to a temporary file in $TMPDIR and uploaded once finished, not streamed to storage; the README and config.example.yml now say a backup needs about blob_size_limit of free space there. Also corrected: the snapshot ID format, what restore reads and how incomplete snapshots are removed in docs/DATAMODEL.md, what source_path is for, the index_path default, the config search order in the snapshot create help, what snapshot remove cleans up in the prune help, how the release installs Go, and the script/release and script/fmt-check comments. The source_path claim is also corrected in models.go, scanner.go and 001.sql, which the issue did not list. Model: opus-5-5
2026-10-07 12:35:07 +00:00
17 changed files with 60 additions and 209 deletions
@@ -54,7 +54,7 @@ The database tracks five primary entities and their relationships:

 #### File (`database.File`)
 Represents a file, directory, or symlink in the backup system. Stores metadata needed for restoration:
- Path, source_path (for restore path stripping), mtime
+- Path, source_path (the configured snapshot path the scan found it under), mtime
 - Size, mode, ownership (uid, gid)
 - Symlink target (if applicable)

@@ -82,9 +82,9 @@ The final storage unit uploaded to S3. Contains many compressed and encrypted ch
 Blob creation process:
 1. Chunks are accumulated (up to MaxBlobSize, typically 10GB)
 2. As each chunk is added, its uncompressed bytes are fed to a running SHA-256
-3. Concurrently, the same bytes are compressed with zstd, then encrypted with age (recipients configured in config), and streamed to storage
+3. Concurrently, the same bytes are compressed with zstd, then encrypted with age (recipients configured in config), and written to a temporary file in `$TMPDIR` (`/tmp` when unset)
 4. On finalize, the blob's name is the double SHA-256 of the uncompressed contents — `hex(SHA256(SHA256(...)))` — not a hash of the compressed, encrypted bytes
-5. Uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}`
+5. The finished file is uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}` and then deleted, so a backup needs about `blob_size_limit` of free space in `$TMPDIR`

 #### BlobChunk (`database.BlobChunk`)
 Maps chunks to their position within blobs:
@@ -546,7 +546,7 @@ complete annotated example also lives in
 | `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) |
 | `exclude` | | Global exclude patterns (applied to all snapshots) |
 | `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
-| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit |
+| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. Each blob is written in full to a temporary file in `$TMPDIR` (`/tmp` when unset) before it is uploaded, so a backup needs about this much free space there |
 | `compression_level` | `3` | zstd compression level (1-19) |
 | `hostname` | system hostname | Hostname used in snapshot IDs |
 | `index_path` | platform data dir | Local SQLite index path |
@@ -805,7 +805,8 @@ them. We provide:
  in the `Dockerfile` together.
 * `script/release` — cross-compile and publish the release artifacts
  with the pinned `goreleaser`. Refuses a `goreleaser` on `PATH` whose
-  version is not the pinned one, on the same reasoning as `script/lint`.
+  version is not the pinned one, because a different version would build
+  a different release from the same tag.
 * `script/release-snapshot` — the same build with no publishing and no
  tagging, into `./dist`
 * `script/test` — run the test suite by building the `test` phase of
@@ -915,14 +916,14 @@ It is passed to `goreleaser` as `GITEA_TOKEN`. The runner's automatic
 token is deliberately not used: it is not guaranteed to carry release
 write access.

-The Go toolchain that compiles the released binaries comes from an
-`actions/setup-go` step pinned by commit sha, reading its version from
-`go.mod` (currently `1.26.1`, the same version the `Dockerfile` builder
-stage pins by digest). `goreleaser` shells out to `go` for every
+The Go toolchain that compiles the released binaries is installed by
+`script/install-go`, which downloads the version named by `go.mod`
+(currently `1.26.1`, the same version the `Dockerfile` builder stage
+pins by digest) and refuses the archive unless its sha256 matches the
+value committed in the script. `goreleaser` shells out to `go` for every
 cross-compile, so without that step the release would either fail
 outright or ship binaries built by whatever unpinned toolchain the
-runner happened to carry — the one unpinned thing in an otherwise
-hash-pinned release path.
+runner happened to carry.

 To rehearse the whole build without publishing or tagging anything:

@@ -22,16 +22,17 @@ the tag exists and is exercised; what is left is merging `next` to

 # Completed Steps

- 2026-10-07: Made two messages say only what is true
-  ([issue #240](https://git.eeqj.de/sneak/vaultik/issues/240)). A config
-  file that others can read was warned about as containing S3
-  credentials even when it set none, as a `file://` config does. The
-  warning now says the file may contain S3 credentials only when
-  `s3.access_key_id` or `s3.secret_access_key` is set, since either may
-  come from a `${...}` reference rather than the file, and otherwise
-  says the file is readable by others. `snapshot purge` against a
-  destination store it could not list gave an error with
-  `listing remote snapshots:` in it twice; the prefix now appears once.
+- 2026-10-07: Corrected documentation, help text and comments that were
+  false about the code
+  ([issue #233](https://git.eeqj.de/sneak/vaultik/issues/233)). A blob
+  is not streamed to storage; it is written in full to a temporary file
+  in `$TMPDIR` and uploaded once finished, so the README and
+  `config.example.yml` now say a backup needs about `blob_size_limit` of
+  free space there. Also corrected: the snapshot ID format, what restore
+  reads and how incomplete snapshots are removed in `docs/DATAMODEL.md`,
+  what `source_path` is for, the `index_path` and config file defaults,
+  what `snapshot remove` cleans up, how the release gets its Go
+  toolchain, and the `script/release` and `script/fmt-check` comments.

 - 2026-10-07: Made `s3.part_size` set the multipart upload part size
  ([issue #232](https://git.eeqj.de/sneak/vaultik/issues/232)). It was
@@ -295,8 +295,10 @@ storage_url: "rclone://myremote/path/to/backups"

 # Path to local SQLite index database
 # This database tracks file state for incremental backups
-# Default: /var/lib/vaultik/index.sqlite
-#index_path: /var/lib/vaultik/index.sqlite
+# Default: the platform data directory, e.g.
+#   macOS: ~/Library/Application Support/vaultik/index.sqlite
+#   Linux: ~/.local/share/vaultik/index.sqlite
+#index_path: /path/to/index.sqlite

 # Average chunk size for content-defined chunking
 # Smaller chunks = better deduplication but more metadata
@@ -311,6 +313,9 @@ storage_url: "rclone://myremote/path/to/backups"
 # Chunking uses no secret (the FastCDC parameters are fixed and public). At a
 # large limit a blob holds hundreds of chunks, so individual chunk lengths are
 # not visible in its size; lowering the limit toward chunk_size exposes them.
+# Each blob is written in full to a temporary file in $TMPDIR (/tmp when
+# unset) before it is uploaded, so a backup needs about this much free space
+# there.
 # Supports: 1GB, 10G, 500MB, 1GiB, etc.
 # Default: 10GB
 #blob_size_limit: 10GB
@@ -111,7 +111,7 @@ Maps chunks to the blobs that contain them.
 Tracks backup snapshots.

 **Columns:**
- `id` (TEXT PRIMARY KEY) - Snapshot ID (format: hostname-YYYYMMDD-HHMMSSZ)
+- `id` (TEXT PRIMARY KEY) - Snapshot ID (format: `hostname_name_timestamp`, e.g. `server1_home_2025-06-01T12:00:00Z`: the hostname up to its first `.`, the snapshot name, and an RFC 3339 UTC timestamp)
 - `hostname` (TEXT) - Hostname where backup was created
 - `vaultik_version` (TEXT) - Version of Vaultik used
 - `vaultik_git_revision` (TEXT) - Git revision of Vaultik used
@@ -218,8 +218,8 @@ The `{remote-key}` directory name is a one-way hash of the human snapshot ID, so
 ### 4. Restore Process

 The restore process doesn't use the local database. Instead:
-1. Downloads snapshot metadata from S3
-2. Downloads required blobs based on manifest
+1. Downloads and decrypts the snapshot's metadata database (`db.zst.age`) from S3
+2. Downloads the blobs holding the chunks of the files being restored, found through that database's `blob_chunks` table; the manifest is not read
 3. Reconstructs files from decrypted and decompressed chunks

 ### 5. Pruning
@@ -232,9 +232,8 @@ The restore process doesn't use the local database. Instead:

 Before each backup:
 1. Query incomplete snapshots (where `completed_at IS NULL`)
-2. Check if metadata exists in S3
-3. If no metadata, delete snapshot and all associations
-4. Clean up orphaned files, chunks, and blobs
+2. Delete each one and all its associations, without checking S3 for its metadata
+3. Clean up orphaned files, chunks, and blobs

 ## Repository Pattern

@@ -22,9 +22,11 @@ scans every snapshot manifest in the destination store, builds the
 set of still-referenced blob hashes, and deletes any blob not in that
 set.

-Snapshot create --prune and snapshot remove run the same cleanup
-automatically; this command is the manual entry point for the same
-work (e.g. after a crashed backup or to reclaim storage).`,
+Snapshot create --prune runs the same cleanup automatically; this
+command is the manual entry point for the same work (e.g. after a
+crashed backup or to reclaim storage). Snapshot remove leaves blobs in
+place; run this command afterwards to delete the ones no longer
+referenced.`,
 		Args: cobra.NoArgs,
 		RunE: func(cmd *cobra.Command, _ []string) error {
 			// Use unified config resolution
@@ -66,8 +66,9 @@ func newSnapshotCreateCommand() *cobra.Command {
 If snapshot names are provided, only those snapshots are created.
 If no names are provided, all configured snapshots are created.

-Config is located at /etc/vaultik/config.yml by default, but can be overridden by
-specifying a path using --config or by setting VAULTIK_CONFIG to a path.`,
+The config is read from the path given by --config or VAULTIK_CONFIG;
+otherwise from the platform config directory (~/.config/vaultik/config.yml
+on Linux), then /etc/vaultik/config.yml.`,
 		Args: cobra.ArbitraryArgs,
 		RunE: func(cmd *cobra.Command, args []string) error {
 			// Pass snapshot names from args
@@ -305,7 +305,7 @@ func Load(path string) (*Config, error) {
 	if statErr == nil {
 		mode := info.Mode().Perm()
 		if mode&0044 != 0 { // group or world readable
-			log.Warn(cfg.readableByOthersWarning(),
+			log.Warn("Config file has insecure permissions (contains S3 credentials)",
 				"path", path,
 				"mode", fmt.Sprintf("%04o", mode),
 				"recommendation", "chmod 600 "+path)
@@ -418,18 +418,6 @@ func (c *Config) setAgeSecretKey() {
 	}
 }

-// readableByOthersWarning is the warning Load logs when others can read
-// the config file. It says "may contain" because the S3 credentials are
-// seen only after smartconfig has replaced any ${...} reference in the
-// file with its value, so a set credential need not be in the file.
-func (c *Config) readableByOthersWarning() string {
-	if c.S3.AccessKeyID != "" || c.S3.SecretAccessKey != "" {
-		return "Config file is readable by others and may contain S3 credentials"
-	}
-
-	return "Config file is readable by others"
-}
-
 // validateStorage validates storage configuration.
 // If StorageURL is set, it takes precedence. S3 URLs require credentials.
 // File URLs don't require any S3 configuration.
@@ -8,7 +8,6 @@ import (
 	"testing"

 	"sneak.berlin/go/vaultik/internal/chunker"
-	"sneak.berlin/go/vaultik/internal/log"
 )

 const (
@@ -460,125 +459,3 @@ func TestAgeSecretKeySourceName(t *testing.T) {
 		})
 	}
 }
-
-// loadReadableConfig writes configYAML to a file that others can read,
-// loads it, and returns what the logger wrote to stderr meanwhile. The
-// logger writes to the os.Stderr it finds when it is initialized, so
-// os.Stderr is pointed at a file first. Not parallel-safe: os.Stderr and
-// the logger are process-global.
-func loadReadableConfig(t *testing.T, configYAML string) string {
-	t.Helper()
-
-	dir := t.TempDir()
-	configPath := filepath.Join(dir, "config.yml")
-	stderrPath := filepath.Join(dir, "stderr")
-
-	err := os.WriteFile(configPath, []byte(configYAML), 0o600)
-	if err != nil {
-		t.Fatalf("writing config: %v", err)
-	}
-
-	//nolint:gosec // G302: the test needs a config file others can read
-	err = os.Chmod(configPath, 0o644)
-	if err != nil {
-		t.Fatalf("chmod config: %v", err)
-	}
-
-	stderrFile, err := os.Create(stderrPath) //nolint:gosec // G304: test temp path
-	if err != nil {
-		t.Fatalf("creating stderr file: %v", err)
-	}
-
-	previous := os.Stderr
-	os.Stderr = stderrFile
-
-	log.Initialize(log.Config{})
-
-	_, loadErr := Load(configPath)
-
-	os.Stderr = previous
-
-	log.Initialize(log.Config{})
-
-	_ = stderrFile.Close()
-
-	if loadErr != nil {
-		t.Fatalf("Load() error = %v", loadErr)
-	}
-
-	captured, err := os.ReadFile(stderrPath) //nolint:gosec // G304: test temp path
-	if err != nil {
-		t.Fatalf("reading stderr file: %v", err)
-	}
-
-	return string(captured)
-}
-
-// TestLoadWarnsReadableConfigWithoutS3Credentials checks that a config
-// file others can read, holding no S3 credentials, is warned about
-// without a claim that it holds them.
-//
-//nolint:paralleltest // loadReadableConfig replaces os.Stderr
-func TestLoadWarnsReadableConfigWithoutS3Credentials(t *testing.T) {
-	stderr := loadReadableConfig(t, `
-storage_url: file:///var/backups/vaultik
-snapshots:
-  home:
-    paths:
-      - /home
-`)
-
-	if !strings.Contains(stderr, "Config file is readable by others") {
-		t.Errorf("expected a warning that the file is readable by others, got %q",
-			stderr)
-	}
-
-	if strings.Contains(stderr, "S3 credentials") {
-		t.Errorf("warning names S3 credentials the file does not set: %q", stderr)
-	}
-}
-
-// TestLoadWarnsReadableConfigWithS3Credentials checks that a config file
-// others can read and that sets S3 credentials, as values or as ${ENV:...}
-// references, is warned about as one that may contain them.
-//
-//nolint:paralleltest // loadReadableConfig replaces os.Stderr
-func TestLoadWarnsReadableConfigWithS3Credentials(t *testing.T) {
-	t.Setenv("VAULTIK_TEST_ACCESS_KEY_ID", "test-access-key")
-	t.Setenv("VAULTIK_TEST_SECRET_ACCESS_KEY", "test-secret-key")
-
-	configs := map[string]string{
-		"values": `
-storage_url: s3://bucket/prefix?endpoint=s3.example.com
-s3:
-  access_key_id: test-access-key
-  secret_access_key: test-secret-key
-snapshots:
-  home:
-    paths:
-      - /home
-`,
-		"references": `
-storage_url: s3://bucket/prefix?endpoint=s3.example.com
-s3:
-  access_key_id: ${ENV:VAULTIK_TEST_ACCESS_KEY_ID}
-  secret_access_key: ${ENV:VAULTIK_TEST_SECRET_ACCESS_KEY}
-snapshots:
-  home:
-    paths:
-      - /home
-`,
-	}
-
-	for name, configYAML := range configs {
-		t.Run(name, func(t *testing.T) {
-			stderr := loadReadableConfig(t, configYAML)
-
-			if !strings.Contains(stderr,
-				"Config file is readable by others and may contain S3 credentials") {
-				t.Errorf("expected a warning naming the S3 credentials, got %q",
-					stderr)
-			}
-		})
-	}
-}
@@ -14,8 +14,7 @@ type File struct {
 	ID   types.FileID   // UUID primary key
 	Path types.FilePath // Absolute path of the file

-	// SourcePath is the source directory this file came from (used for
-	// restore path stripping).
+	// SourcePath is the source directory this file came from.
 	SourcePath types.SourcePath
 	MTime      time.Time
 	Size       int64
@@ -5,7 +5,7 @@
 CREATE TABLE IF NOT EXISTS files (
    id TEXT PRIMARY KEY,  -- UUID
    path TEXT NOT NULL UNIQUE,
-    source_path TEXT NOT NULL DEFAULT '',  -- The source directory this file came from (for restore path stripping)
+    source_path TEXT NOT NULL DEFAULT '',  -- The source directory this file came from
    mtime INTEGER NOT NULL,  -- whole seconds since the Unix epoch
    mtime_nsec INTEGER NOT NULL,  -- nanoseconds within that second, 0 to 999999999
    size INTEGER NOT NULL,
@@ -58,8 +58,8 @@ type Scanner struct {
 	compressionLevel int
 	ageRecipient     string
 	snapshotID       string // Current snapshot being processed
-	// currentSourcePath is the source directory being scanned (used for
-	// restore path stripping).
+	// currentSourcePath is the source directory being scanned, stored with
+	// each file record.
 	currentSourcePath string
 	exclude           []string          // Glob patterns for files/directories to exclude
 	compiledExclude   []compiledPattern // Compiled glob patterns
@@ -211,7 +211,6 @@ func (s *Scanner) Scan(
 	ctx context.Context, path string, snapshotID string,
 ) (*ScanResult, error) {
 	s.snapshotID = snapshotID
-	// Store source path for file records (used during restore)
 	s.currentSourcePath = path
 	result := &ScanResult{
 		StartTime: time.Now().UTC(),
@@ -1198,9 +1197,8 @@ func (s *Scanner) checkFileInMemory(
 	}

 	file := &database.File{
-		ID:   fileID,
-		Path: types.FilePath(path),
-		// Store source directory for restore path stripping
+		ID:         fileID,
+		Path:       types.FilePath(path),
 		SourcePath: types.SourcePath(s.currentSourcePath),
 		MTime:      info.ModTime(),
 		Size:       info.Size(),
@@ -155,8 +155,7 @@ type BlobHash string
 // FilePath represents an absolute path to a file or directory.
 type FilePath string

-// SourcePath represents the root directory from which files are backed up.
-// Used during restore to strip the source prefix from paths.
+// SourcePath is the configured snapshot path a scan found a file under.
 type SourcePath string

 // Hostname identifies a host machine.
@@ -6,7 +6,6 @@ import (
 	"io/fs"
 	"os"
 	"path/filepath"
-	"strings"
 	"testing"

 	"github.com/spf13/afero"
@@ -154,23 +153,3 @@ func TestPruneKeepsLocalRecordsWhenDestinationMissing(t *testing.T) {
 	require.NoError(t, err)
 	assert.Len(t, snapshots, 1, "prune must delete no local snapshot record")
 }
-
-// TestPurgeSaysListingFailedOnceWhenDestinationMissing checks that
-// snapshot purge fails on a destination it cannot list, with an error
-// that says "listing remote snapshots" once.
-//
-//nolint:paralleltest // installs the global logger via log.Initialize
-func TestPurgeSaysListingFailedOnceWhenDestinationMissing(t *testing.T) {
-	log.Initialize(log.Config{})
-
-	ctx := context.Background()
-	v, _, _ := backUpThenUnplug(ctx, t)
-
-	err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
-		KeepLatest: true,
-		Force:      true,
-	})
-	require.ErrorIs(t, err, fs.ErrNotExist)
-	assert.Equal(t, 1, strings.Count(err.Error(), "listing remote snapshots"),
-		err.Error())
-}
@@ -1052,7 +1052,7 @@ func (v *Vaultik) syncWithRemote() error {
 	// every local snapshot record (issue #160).
 	remoteKeys, err := v.listAllRemoteSnapshotKeys()
 	if err != nil {
-		return err
+		return fmt.Errorf("listing remote snapshots: %w", err)
 	}

 	remoteKeySet := make(map[string]bool, len(remoteKeys))
@@ -1,6 +1,9 @@
 #!/bin/sh
-# script/fmt-check: check formatting (read-only). Same scope as
-# script/fmt, but fails instead of writing.
+# script/fmt-check: check formatting (read-only). Fails instead of
+# writing. It checks every Go file outside .tool, which is more than
+# script/fmt formats: `go fmt ./...` skips `testdata` directories and
+# files and directories whose names start with `.` or `_`. Fix a file
+# only this reports with `gofmt -w`.
 set -eu

 ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
@@ -21,10 +21,9 @@ goreleaser_version() {
        head -n 1
 }

-# Resolve the goreleaser to run, on the same rule script/lint uses for
-# golangci-lint: a binary on PATH is accepted only when it is exactly
-# the pinned version, because a differently versioned tool would
-# produce a differently built release from the same tag. Anything else
+# Resolve the goreleaser to run. A binary on PATH is accepted only when
+# it is exactly the pinned version, because a differently versioned tool
+# would produce a differently built release from the same tag. Anything else
 # comes from .tool/bin, and a missing one is a loud failure naming the
 # script that installs it rather than a silent fallback.
 resolve_goreleaser() {