Compare commits

..
2 Commits
Author SHA1 Message Date
sneak 72f9a8f8a0 Correct doc and help sentences that are false about the code (closes #233)
check / check (push) Canceled after 0s
A blob is written in full to a temporary file in $TMPDIR and uploaded
once finished, not streamed to storage, and the metadata export works on
a copy of the local index there. The README, ARCHITECTURE.md and
config.example.yml now say a backup needs free space there of the larger
of blob_size_limit and about three times the size of the local index.
Also corrected: the snapshot ID format, what restore reads and how
incomplete snapshots are removed in docs/DATAMODEL.md, what source_path
holds, the index_path default, the config search order in the snapshot
create help, what snapshot remove cleans up in the prune help, how the
release installs Go, and the script/release and script/fmt-check
comments.

Model: opus-5-5
2026-10-07 14:04:10 +00:00
clawbot d53202eb86 Fix two misleading messages (closes #240)
check / check (push) Canceled after 0s
The warning for a config file that others can read always said the
file contained S3 credentials, so a file:// config with none got a
false claim. When s3.access_key_id or s3.secret_access_key is set it
now says the file may contain them, because Load sees the values only
after smartconfig has replaced any ${...} reference, so a set
credential need not be in the file. Otherwise it says the file is
readable by others.

snapshot purge wrapped the listing error, which already starts with
"listing remote snapshots:", in that prefix a second time.
syncWithRemote now returns it unwrapped, as CleanupLocalSnapshots
does.

Model: opus-5-5
2026-10-07 15:29:07 +02:00
9 changed files with 188 additions and 12 deletions
+6 -2
View File
@@ -54,10 +54,12 @@ The database tracks five primary entities and their relationships:
#### File (`database.File`)
Represents a file, directory, or symlink in the backup system. Stores metadata needed for restoration:
- Path, source_path (the configured snapshot path the scan found it under), mtime
- Path, mtime
- Size, mode, ownership (uid, gid)
- Symlink target (if applicable)
It also stores `source_path`, the source directory the scan found it under, made absolute and with symlinks resolved. Restore does not read it.
#### Chunk (`database.Chunk`)
A content-addressed unit of data. Files are split into variable-size chunks using the FastCDC algorithm:
- `ChunkHash`: SHA256 hash of chunk content (primary key)
@@ -84,7 +86,9 @@ Blob creation process:
2. As each chunk is added, its uncompressed bytes are fed to a running SHA-256
3. Concurrently, the same bytes are compressed with zstd, then encrypted with age (recipients configured in config), and written to a temporary file in `$TMPDIR` (`/tmp` when unset)
4. On finalize, the blob's name is the double SHA-256 of the uncompressed contents — `hex(SHA256(SHA256(...)))` — not a hash of the compressed, encrypted bytes
5. The finished file is uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}` and then deleted, so a backup needs about `blob_size_limit` of free space in `$TMPDIR`
5. The finished file is uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}` and then deleted
The metadata export at the end of a backup also uses `$TMPDIR`: it copies the local index there and runs `VACUUM` on the copy, which writes another temporary copy and a write-ahead log. A backup therefore needs free space in `$TMPDIR` of the larger of `blob_size_limit` and about three times the size of the local index.
#### BlobChunk (`database.BlobChunk`)
Maps chunks to their position within blobs:
+1 -1
View File
@@ -546,7 +546,7 @@ complete annotated example also lives in
| `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) |
| `exclude` | | Global exclude patterns (applied to all snapshots) |
| `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. Each blob is written in full to a temporary file in `$TMPDIR` (`/tmp` when unset) before it is uploaded, so a backup needs about this much free space there |
| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. Each blob is written in full to a temporary file in `$TMPDIR` (`/tmp` when unset) before it is uploaded, and the metadata export at the end of a backup works on a copy of the local index there. A backup needs free space there of the larger of this limit and about three times the size of the local index |
| `compression_level` | `3` | zstd compression level (1-19) |
| `hostname` | system hostname | Hostname used in snapshot IDs |
| `index_path` | platform data dir | Local SQLite index path |
+17 -4
View File
@@ -26,14 +26,27 @@ the tag exists and is exercised; what is left is merging `next` to
false about the code
([issue #233](https://git.eeqj.de/sneak/vaultik/issues/233)). A blob
is not streamed to storage; it is written in full to a temporary file
in `$TMPDIR` and uploaded once finished, so the README and
`config.example.yml` now say a backup needs about `blob_size_limit` of
free space there. Also corrected: the snapshot ID format, what restore
in `$TMPDIR` and uploaded once finished, and the metadata export works
on a copy of the local index there. The README and
`config.example.yml` now say a backup needs free space there of the
larger of `blob_size_limit` and about three times the size of the
local index. Also corrected: the snapshot ID format, what restore
reads and how incomplete snapshots are removed in `docs/DATAMODEL.md`,
what `source_path` is for, the `index_path` and config file defaults,
what `source_path` holds, the `index_path` and config file defaults,
what `snapshot remove` cleans up, how the release gets its Go
toolchain, and the `script/release` and `script/fmt-check` comments.
- 2026-10-07: Made two messages say only what is true
([issue #240](https://git.eeqj.de/sneak/vaultik/issues/240)). A config
file that others can read was warned about as containing S3
credentials even when it set none, as a `file://` config does. The
warning now says the file may contain S3 credentials only when
`s3.access_key_id` or `s3.secret_access_key` is set, since either may
come from a `${...}` reference rather than the file, and otherwise
says the file is readable by others. `snapshot purge` against a
destination store it could not list gave an error with
`listing remote snapshots:` in it twice; the prefix now appears once.
- 2026-10-07: Made `s3.part_size` set the multipart upload part size
([issue #232](https://git.eeqj.de/sneak/vaultik/issues/232)). It was
loaded and defaulted but never passed to the S3 client, whose uploader
+4 -2
View File
@@ -314,8 +314,10 @@ storage_url: "rclone://myremote/path/to/backups"
# large limit a blob holds hundreds of chunks, so individual chunk lengths are
# not visible in its size; lowering the limit toward chunk_size exposes them.
# Each blob is written in full to a temporary file in $TMPDIR (/tmp when
# unset) before it is uploaded, so a backup needs about this much free space
# there.
# unset) before it is uploaded, and the metadata export at the end of a
# backup works on a copy of the local index there. A backup needs free space
# there of the larger of this limit and about three times the size of the
# local index.
# Supports: 1GB, 10G, 500MB, 1GiB, etc.
# Default: 10GB
#blob_size_limit: 10GB
+13 -1
View File
@@ -305,7 +305,7 @@ func Load(path string) (*Config, error) {
if statErr == nil {
mode := info.Mode().Perm()
if mode&0044 != 0 { // group or world readable
log.Warn("Config file has insecure permissions (contains S3 credentials)",
log.Warn(cfg.readableByOthersWarning(),
"path", path,
"mode", fmt.Sprintf("%04o", mode),
"recommendation", "chmod 600 "+path)
@@ -418,6 +418,18 @@ func (c *Config) setAgeSecretKey() {
}
}
// readableByOthersWarning is the warning Load logs when others can read
// the config file. It says "may contain" because the S3 credentials are
// seen only after smartconfig has replaced any ${...} reference in the
// file with its value, so a set credential need not be in the file.
func (c *Config) readableByOthersWarning() string {
if c.S3.AccessKeyID != "" || c.S3.SecretAccessKey != "" {
return "Config file is readable by others and may contain S3 credentials"
}
return "Config file is readable by others"
}
// validateStorage validates storage configuration.
// If StorageURL is set, it takes precedence. S3 URLs require credentials.
// File URLs don't require any S3 configuration.
+123
View File
@@ -8,6 +8,7 @@ import (
"testing"
"sneak.berlin/go/vaultik/internal/chunker"
"sneak.berlin/go/vaultik/internal/log"
)
const (
@@ -459,3 +460,125 @@ func TestAgeSecretKeySourceName(t *testing.T) {
})
}
}
// loadReadableConfig writes configYAML to a file that others can read,
// loads it, and returns what the logger wrote to stderr meanwhile. The
// logger writes to the os.Stderr it finds when it is initialized, so
// os.Stderr is pointed at a file first. Not parallel-safe: os.Stderr and
// the logger are process-global.
func loadReadableConfig(t *testing.T, configYAML string) string {
t.Helper()
dir := t.TempDir()
configPath := filepath.Join(dir, "config.yml")
stderrPath := filepath.Join(dir, "stderr")
err := os.WriteFile(configPath, []byte(configYAML), 0o600)
if err != nil {
t.Fatalf("writing config: %v", err)
}
//nolint:gosec // G302: the test needs a config file others can read
err = os.Chmod(configPath, 0o644)
if err != nil {
t.Fatalf("chmod config: %v", err)
}
stderrFile, err := os.Create(stderrPath) //nolint:gosec // G304: test temp path
if err != nil {
t.Fatalf("creating stderr file: %v", err)
}
previous := os.Stderr
os.Stderr = stderrFile
log.Initialize(log.Config{})
_, loadErr := Load(configPath)
os.Stderr = previous
log.Initialize(log.Config{})
_ = stderrFile.Close()
if loadErr != nil {
t.Fatalf("Load() error = %v", loadErr)
}
captured, err := os.ReadFile(stderrPath) //nolint:gosec // G304: test temp path
if err != nil {
t.Fatalf("reading stderr file: %v", err)
}
return string(captured)
}
// TestLoadWarnsReadableConfigWithoutS3Credentials checks that a config
// file others can read, holding no S3 credentials, is warned about
// without a claim that it holds them.
//
//nolint:paralleltest // loadReadableConfig replaces os.Stderr
func TestLoadWarnsReadableConfigWithoutS3Credentials(t *testing.T) {
stderr := loadReadableConfig(t, `
storage_url: file:///var/backups/vaultik
snapshots:
home:
paths:
- /home
`)
if !strings.Contains(stderr, "Config file is readable by others") {
t.Errorf("expected a warning that the file is readable by others, got %q",
stderr)
}
if strings.Contains(stderr, "S3 credentials") {
t.Errorf("warning names S3 credentials the file does not set: %q", stderr)
}
}
// TestLoadWarnsReadableConfigWithS3Credentials checks that a config file
// others can read and that sets S3 credentials, as values or as ${ENV:...}
// references, is warned about as one that may contain them.
//
//nolint:paralleltest // loadReadableConfig replaces os.Stderr
func TestLoadWarnsReadableConfigWithS3Credentials(t *testing.T) {
t.Setenv("VAULTIK_TEST_ACCESS_KEY_ID", "test-access-key")
t.Setenv("VAULTIK_TEST_SECRET_ACCESS_KEY", "test-secret-key")
configs := map[string]string{
"values": `
storage_url: s3://bucket/prefix?endpoint=s3.example.com
s3:
access_key_id: test-access-key
secret_access_key: test-secret-key
snapshots:
home:
paths:
- /home
`,
"references": `
storage_url: s3://bucket/prefix?endpoint=s3.example.com
s3:
access_key_id: ${ENV:VAULTIK_TEST_ACCESS_KEY_ID}
secret_access_key: ${ENV:VAULTIK_TEST_SECRET_ACCESS_KEY}
snapshots:
home:
paths:
- /home
`,
}
for name, configYAML := range configs {
t.Run(name, func(t *testing.T) {
stderr := loadReadableConfig(t, configYAML)
if !strings.Contains(stderr,
"Config file is readable by others and may contain S3 credentials") {
t.Errorf("expected a warning naming the S3 credentials, got %q",
stderr)
}
})
}
}
+2 -1
View File
@@ -155,7 +155,8 @@ type BlobHash string
// FilePath represents an absolute path to a file or directory.
type FilePath string
// SourcePath is the configured snapshot path a scan found a file under.
// SourcePath is the source directory a scan found a file under, made
// absolute and with symlinks resolved.
type SourcePath string
// Hostname identifies a host machine.
@@ -6,6 +6,7 @@ import (
"io/fs"
"os"
"path/filepath"
"strings"
"testing"
"github.com/spf13/afero"
@@ -153,3 +154,23 @@ func TestPruneKeepsLocalRecordsWhenDestinationMissing(t *testing.T) {
require.NoError(t, err)
assert.Len(t, snapshots, 1, "prune must delete no local snapshot record")
}
// TestPurgeSaysListingFailedOnceWhenDestinationMissing checks that
// snapshot purge fails on a destination it cannot list, with an error
// that says "listing remote snapshots" once.
//
//nolint:paralleltest // installs the global logger via log.Initialize
func TestPurgeSaysListingFailedOnceWhenDestinationMissing(t *testing.T) {
log.Initialize(log.Config{})
ctx := context.Background()
v, _, _ := backUpThenUnplug(ctx, t)
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
KeepLatest: true,
Force: true,
})
require.ErrorIs(t, err, fs.ErrNotExist)
assert.Equal(t, 1, strings.Count(err.Error(), "listing remote snapshots"),
err.Error())
}
+1 -1
View File
@@ -1052,7 +1052,7 @@ func (v *Vaultik) syncWithRemote() error {
// every local snapshot record (issue #160).
remoteKeys, err := v.listAllRemoteSnapshotKeys()
if err != nil {
return fmt.Errorf("listing remote snapshots: %w", err)
return err
}
remoteKeySet := make(map[string]bool, len(remoteKeys))