Compare commits
11
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
08340faaa4 | ||
|
|
b5389a62b5 | ||
|
|
d87202fb70 | ||
|
|
e161343eac | ||
|
|
d53202eb86 | ||
|
|
8b22ae8d42 | ||
|
|
3fc8a8f2f4 | ||
|
|
7696f83258 | ||
|
|
5d1118d143 | ||
|
|
b57ce2277d | ||
|
|
8496404d8b |
+7
-3
@@ -54,10 +54,12 @@ The database tracks five primary entities and their relationships:
|
||||
|
||||
#### File (`database.File`)
|
||||
Represents a file, directory, or symlink in the backup system. Stores metadata needed for restoration:
|
||||
- Path, source_path (for restore path stripping), mtime
|
||||
- Path, mtime
|
||||
- Size, mode, ownership (uid, gid)
|
||||
- Symlink target (if applicable)
|
||||
|
||||
It also stores `source_path`, the source directory the scan found it under, made absolute and with symlinks resolved. Restore does not read it.
|
||||
|
||||
#### Chunk (`database.Chunk`)
|
||||
A content-addressed unit of data. Files are split into variable-size chunks using the FastCDC algorithm:
|
||||
- `ChunkHash`: SHA256 hash of chunk content (primary key)
|
||||
@@ -82,9 +84,11 @@ The final storage unit uploaded to S3. Contains many compressed and encrypted ch
|
||||
Blob creation process:
|
||||
1. Chunks are accumulated (up to MaxBlobSize, typically 10GB)
|
||||
2. As each chunk is added, its uncompressed bytes are fed to a running SHA-256
|
||||
3. Concurrently, the same bytes are compressed with zstd, then encrypted with age (recipients configured in config), and streamed to storage
|
||||
3. Concurrently, the same bytes are compressed with zstd, then encrypted with age (recipients configured in config), and written to a temporary file
|
||||
4. On finalize, the blob's name is the double SHA-256 of the uncompressed contents — `hex(SHA256(SHA256(...)))` — not a hash of the compressed, encrypted bytes
|
||||
5. Uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}`
|
||||
5. The finished file is uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}` and then deleted
|
||||
|
||||
A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp`.
|
||||
|
||||
#### BlobChunk (`database.BlobChunk`)
|
||||
Maps chunks to their position within blobs:
|
||||
|
||||
@@ -390,7 +390,13 @@ recipients, and local database statistics.
|
||||
|
||||
**`remote info`**: Show storage backend type and location plus detailed
|
||||
remote storage inventory: per-snapshot metadata sizes, blob counts, and
|
||||
orphaned blob detection.
|
||||
orphaned blob detection. A name under `metadata/` that is not a remote
|
||||
key is skipped with a warning and is not printed. If a listed
|
||||
`manifest.json.zst` cannot be read, or sits under a skipped name, the
|
||||
orphaned blob figures are reported as unknown; `--json` gives them as
|
||||
`null`, lists the remote key of each unreadable manifest in
|
||||
`unreadable_manifests` and counts the manifests under skipped names in
|
||||
`skipped_manifest_count`.
|
||||
* `--json`: Output as JSON
|
||||
|
||||
**`remote nuke`**: Delete every snapshot's metadata and every blob from the
|
||||
@@ -540,7 +546,7 @@ complete annotated example also lives in
|
||||
| `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) |
|
||||
| `exclude` | | Global exclude patterns (applied to all snapshots) |
|
||||
| `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
|
||||
| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit |
|
||||
| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` |
|
||||
| `compression_level` | `3` | zstd compression level (1-19) |
|
||||
| `hostname` | system hostname | Hostname used in snapshot IDs |
|
||||
| `index_path` | platform data dir | Local SQLite index path |
|
||||
@@ -799,7 +805,8 @@ them. We provide:
|
||||
in the `Dockerfile` together.
|
||||
* `script/release` — cross-compile and publish the release artifacts
|
||||
with the pinned `goreleaser`. Refuses a `goreleaser` on `PATH` whose
|
||||
version is not the pinned one, on the same reasoning as `script/lint`.
|
||||
version is not the pinned one, because a different version would build
|
||||
a different release from the same tag.
|
||||
* `script/release-snapshot` — the same build with no publishing and no
|
||||
tagging, into `./dist`
|
||||
* `script/test` — run the test suite by building the `test` phase of
|
||||
@@ -909,14 +916,14 @@ It is passed to `goreleaser` as `GITEA_TOKEN`. The runner's automatic
|
||||
token is deliberately not used: it is not guaranteed to carry release
|
||||
write access.
|
||||
|
||||
The Go toolchain that compiles the released binaries comes from an
|
||||
`actions/setup-go` step pinned by commit sha, reading its version from
|
||||
`go.mod` (currently `1.26.1`, the same version the `Dockerfile` builder
|
||||
stage pins by digest). `goreleaser` shells out to `go` for every
|
||||
The Go toolchain that compiles the released binaries is installed by
|
||||
`script/install-go`, which downloads the version named by `go.mod`
|
||||
(currently `1.26.1`, the same version the `Dockerfile` builder stage
|
||||
pins by digest) and refuses the archive unless its sha256 matches the
|
||||
value committed in the script. `goreleaser` shells out to `go` for every
|
||||
cross-compile, so without that step the release would either fail
|
||||
outright or ship binaries built by whatever unpinned toolchain the
|
||||
runner happened to carry — the one unpinned thing in an otherwise
|
||||
hash-pinned release path.
|
||||
runner happened to carry.
|
||||
|
||||
To rehearse the whole build without publishing or tagging anything:
|
||||
|
||||
|
||||
@@ -22,6 +22,124 @@ the tag exists and is exercised; what is left is merging `next` to
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-07: Made a symlink whose target cannot be read stop the backup
|
||||
([issue #269](https://git.eeqj.de/sneak/vaultik/issues/269)). It was
|
||||
left out of the snapshot with only a debug log line, even without
|
||||
`--skip-errors`. It now aborts the run, or with `--skip-errors` is
|
||||
skipped with the `Failed to access` error line that any other entry
|
||||
the scan cannot read gets.
|
||||
|
||||
- 2026-10-07: Made an interrupted command exit 130 and say so
|
||||
([issue #267](https://git.eeqj.de/sneak/vaultik/issues/267)). Ctrl-C
|
||||
or SIGTERM during `snapshot create`, `snapshot restore` or `snapshot
|
||||
verify` exited 0 with no error line (`snapshot verify --json` exited
|
||||
1, also without one), so a `--cron` run that never finished looked
|
||||
like a success. A command stopped by either signal now exits 130 and
|
||||
prints `interrupted before the command finished` on stderr, under
|
||||
`--cron` and `--json` too.
|
||||
|
||||
- 2026-10-07: Corrected documentation, help text and comments that were
|
||||
false about the code
|
||||
([issue #233](https://git.eeqj.de/sneak/vaultik/issues/233)). A blob
|
||||
is not streamed to storage. The README, `ARCHITECTURE.md` and
|
||||
`config.example.yml` now say a backup needs free temporary space,
|
||||
because each blob is written whole to a temporary file before it is
|
||||
uploaded (up to about `blob_size_limit`; an rclone destination that
|
||||
cannot stream uploads needs about twice that) and the metadata export
|
||||
writes copies of the local index. Temporary files go to `$TMPDIR`
|
||||
(default `/tmp`); with `TMPDIR` unset, SQLite writes one of those
|
||||
copies to `/var/tmp`. Also corrected: the snapshot ID format, what restore
|
||||
reads and how incomplete snapshots are removed in `docs/DATAMODEL.md`,
|
||||
what `source_path` holds, the `index_path` and config file defaults,
|
||||
what `snapshot remove` cleans up, how the release gets its Go
|
||||
toolchain, and the `script/release` and `script/fmt-check` comments.
|
||||
|
||||
- 2026-10-07: Cut the time the `internal/vaultik` and `internal/database`
|
||||
tests take ([issue #235](https://git.eeqj.de/sneak/vaultik/issues/235)).
|
||||
Most of the `internal/vaultik` time went to 24 tests that ran one at a
|
||||
time only because they call `log.Initialize`; they now call it before
|
||||
`t.Parallel()`, as the package's other tests do. `TestLargeDatasets`
|
||||
committed each of its 1,500 inserts on its own and now makes them in
|
||||
one transaction. `TestDedupOnlySnapshotRestores` gives its second
|
||||
backup its own snapshot name instead of sleeping past the one-second
|
||||
timestamp in the snapshot ID.
|
||||
|
||||
- 2026-10-07: Made two messages say only what is true
|
||||
([issue #240](https://git.eeqj.de/sneak/vaultik/issues/240)). A config
|
||||
file that others can read was warned about as containing S3
|
||||
credentials even when it set none, as a `file://` config does. The
|
||||
warning now says the file may contain S3 credentials only when
|
||||
`s3.access_key_id` or `s3.secret_access_key` is set, since either may
|
||||
come from a `${...}` reference rather than the file, and otherwise
|
||||
says the file is readable by others. `snapshot purge` against a
|
||||
destination store it could not list gave an error with
|
||||
`listing remote snapshots:` in it twice; the prefix now appears once.
|
||||
|
||||
- 2026-10-07: Made `s3.part_size` set the multipart upload part size
|
||||
([issue #232](https://git.eeqj.de/sneak/vaultik/issues/232)). It was
|
||||
loaded and defaulted but never passed to the S3 client, whose uploader
|
||||
used a fixed 10MiB part. It now reaches the uploader for `storage_url`
|
||||
and for the `s3.*` fields, and a part size S3 refuses, below 5MiB or
|
||||
above 5GiB, `0` included, fails at config load. A blob too large for
|
||||
S3's limit of 10,000 parts at the configured size is uploaded in larger
|
||||
parts. The docs gave the default as `5MB`, which the config file reads
|
||||
as 5,000,000 bytes, below the minimum; they now say `5MiB`.
|
||||
|
||||
- 2026-10-07: Made per-name retention work when the hostname contains `_`
|
||||
([issue #230](https://git.eeqj.de/sneak/vaultik/issues/230)). A
|
||||
snapshot ID is `hostname_name_timestamp`, and the name was read as
|
||||
everything between the first and the last `_`, so with
|
||||
`hostname: my_host` the name `home` came out as `host_home`.
|
||||
`snapshot purge --keep-latest --snapshot home` then printed "No
|
||||
snapshots to delete", and `snapshot create --prune` purged nothing
|
||||
without a message. The name is now read using the hostname the
|
||||
`snapshots` table stores with each snapshot, cut at its first `.` as it
|
||||
is in the ID.
|
||||
|
||||
- 2026-10-07: Made `remote info` stop reporting a snapshot's blobs as
|
||||
orphaned when its manifest cannot be read, and stop printing raw
|
||||
names from under `metadata/`
|
||||
([issue #228](https://git.eeqj.de/sneak/vaultik/issues/228)). A
|
||||
manifest it failed to read was skipped, so that snapshot's blobs were
|
||||
counted as orphaned and the report advised running `vaultik prune`.
|
||||
The orphan figures are now unknown in that case, with no prune
|
||||
advice, and `--json` gives them as `null` with the unreadable remote
|
||||
keys in `unreadable_manifests`. A name under `metadata/` that is not
|
||||
64 lowercase hex characters is now skipped with a warning instead of
|
||||
being printed, control characters included. A manifest under a
|
||||
skipped name is then not read either, so it also leaves the orphan
|
||||
figures unknown, and `--json` counts such manifests in
|
||||
`skipped_manifest_count`. A directory with no manifest in it, as left
|
||||
by an interrupted backup, leaves the figures known.
|
||||
|
||||
- 2026-10-07: Made `config set` keep a string that looks like a number
|
||||
([issue #229](https://git.eeqj.de/sneak/vaultik/issues/229)). It wrote
|
||||
every value unquoted, and `config.Load` reads the file through untyped
|
||||
YAML, so an access key `00112233` loaded as `38043` and a hostname `007`
|
||||
as `7`. A value for a string setting in `config.Config` is now tagged as
|
||||
a YAML string, which the file quotes wherever YAML would read a number or
|
||||
a boolean; other settings are still written unquoted.
|
||||
|
||||
- 2026-10-07: Made a backup notice a file rewritten with its size
|
||||
unchanged and a new mtime in the same second as the one in the index
|
||||
([issue #226](https://git.eeqj.de/sneak/vaultik/issues/226)). The
|
||||
`files` table held mtime in whole seconds and the scanner compared
|
||||
whole seconds, so every later snapshot kept the old content. A new
|
||||
`mtime_nsec` column holds the nanoseconds within the second that
|
||||
`mtime` holds, and the scanner compares the full mtime. A local index
|
||||
created before the change lacks the column and is rebuilt with
|
||||
`vaultik database delete` and a full backup.
|
||||
|
||||
- 2026-10-07: Made taking the process-wide lock atomic
|
||||
([issue #227](https://git.eeqj.de/sneak/vaultik/issues/227)). The lock
|
||||
read `vaultik.pid`, checked whether that PID was alive and then wrote
|
||||
its own, so two writers started together could both pass the check and
|
||||
both run. It is now an `flock` on `vaultik.pid`, held until the run
|
||||
ends; the kernel drops it when the process exits, so a crash leaves no
|
||||
lock behind. A clean exit now empties the file instead of deleting it,
|
||||
because deleting it would let two later runs each lock a different
|
||||
file.
|
||||
|
||||
- 2026-10-06: Made `snapshot remove --json` write only its document to
|
||||
stdout when the destination store cannot be reached
|
||||
([issue #251](https://git.eeqj.de/sneak/vaultik/issues/251)). Its
|
||||
|
||||
+15
-6
@@ -287,15 +287,18 @@ storage_url: "rclone://myremote/path/to/backups"
|
||||
# #use_ssl: true
|
||||
#
|
||||
# # Part size for multipart uploads
|
||||
# # Minimum 5MB, affects memory usage during upload
|
||||
# # Supports: 5MB, 10M, 100MiB, etc.
|
||||
# # Default: 5MB
|
||||
# #part_size: 5MB
|
||||
# # Minimum 5MiB, maximum 5GiB; affects memory usage during upload
|
||||
# # A blob too large for 10,000 parts of this size gets larger parts
|
||||
# # Supports: 10MB, 16MiB, 100MiB, etc. (5MB is below the minimum)
|
||||
# # Default: 5MiB
|
||||
# #part_size: 5MiB
|
||||
|
||||
# Path to local SQLite index database
|
||||
# This database tracks file state for incremental backups
|
||||
# Default: /var/lib/vaultik/index.sqlite
|
||||
#index_path: /var/lib/vaultik/index.sqlite
|
||||
# Default: the platform data directory, e.g.
|
||||
# macOS: ~/Library/Application Support/vaultik/index.sqlite
|
||||
# Linux: ~/.local/share/vaultik/index.sqlite
|
||||
#index_path: /path/to/index.sqlite
|
||||
|
||||
# Average chunk size for content-defined chunking
|
||||
# Smaller chunks = better deduplication but more metadata
|
||||
@@ -310,6 +313,12 @@ storage_url: "rclone://myremote/path/to/backups"
|
||||
# Chunking uses no secret (the FastCDC parameters are fixed and public). At a
|
||||
# large limit a blob holds hundreds of chunks, so individual chunk lengths are
|
||||
# not visible in its size; lowering the limit toward chunk_size exposes them.
|
||||
# A backup needs free temporary space, because each blob is written whole to
|
||||
# a temporary file before it is uploaded (up to about blob_size_limit; an
|
||||
# rclone destination that cannot stream uploads needs about twice that) and
|
||||
# the metadata export writes copies of the local index. Temporary files go to
|
||||
# $TMPDIR (default /tmp); with TMPDIR unset, SQLite writes one of those copies
|
||||
# to /var/tmp.
|
||||
# Supports: 1GB, 10G, 500MB, 1GiB, etc.
|
||||
# Default: 10GB
|
||||
#blob_size_limit: 10GB
|
||||
|
||||
+7
-7
@@ -36,7 +36,8 @@ Stores metadata about files in the filesystem being backed up.
|
||||
**Columns:**
|
||||
- `id` (TEXT PRIMARY KEY) - UUID for the file record
|
||||
- `path` (TEXT NOT NULL UNIQUE) - Absolute file path
|
||||
- `mtime` (INTEGER NOT NULL) - Modification time as Unix timestamp
|
||||
- `mtime` (INTEGER NOT NULL) - Modification time, whole seconds since the Unix epoch
|
||||
- `mtime_nsec` (INTEGER NOT NULL) - Nanoseconds within that second, 0 to 999999999
|
||||
- `size` (INTEGER NOT NULL) - File size in bytes
|
||||
- `mode` (INTEGER NOT NULL) - Unix file permissions and type
|
||||
- `uid` (INTEGER NOT NULL) - User ID of file owner
|
||||
@@ -110,7 +111,7 @@ Maps chunks to the blobs that contain them.
|
||||
Tracks backup snapshots.
|
||||
|
||||
**Columns:**
|
||||
- `id` (TEXT PRIMARY KEY) - Snapshot ID (format: hostname-YYYYMMDD-HHMMSSZ)
|
||||
- `id` (TEXT PRIMARY KEY) - Snapshot ID (format: `hostname_name_timestamp`, e.g. `server1_home_2025-06-01T12:00:00Z`: the hostname up to its first `.`, the snapshot name, and an RFC 3339 UTC timestamp)
|
||||
- `hostname` (TEXT) - Hostname where backup was created
|
||||
- `vaultik_version` (TEXT) - Version of Vaultik used
|
||||
- `vaultik_git_revision` (TEXT) - Git revision of Vaultik used
|
||||
@@ -217,8 +218,8 @@ The `{remote-key}` directory name is a one-way hash of the human snapshot ID, so
|
||||
### 4. Restore Process
|
||||
|
||||
The restore process doesn't use the local database. Instead:
|
||||
1. Downloads snapshot metadata from S3
|
||||
2. Downloads required blobs based on manifest
|
||||
1. Downloads and decrypts the snapshot's metadata database (`db.zst.age`) from S3
|
||||
2. Downloads the blobs holding the chunks of the files being restored, found through that database's `blob_chunks` table; the manifest is not read
|
||||
3. Reconstructs files from decrypted and decompressed chunks
|
||||
|
||||
### 5. Pruning
|
||||
@@ -231,9 +232,8 @@ The restore process doesn't use the local database. Instead:
|
||||
|
||||
Before each backup:
|
||||
1. Query incomplete snapshots (where `completed_at IS NULL`)
|
||||
2. Check if metadata exists in S3
|
||||
3. If no metadata, delete snapshot and all associations
|
||||
4. Clean up orphaned files, chunks, and blobs
|
||||
2. Delete each one and all its associations, without checking S3 for its metadata
|
||||
3. Clean up orphaned files, chunks, and blobs
|
||||
|
||||
## Repository Pattern
|
||||
|
||||
|
||||
+39
-14
@@ -206,6 +206,10 @@ func RunApp(ctx context.Context, app *fx.App) error {
|
||||
// RunOperation through cobra to Entry.
|
||||
var errReported = errors.New("operation failed")
|
||||
|
||||
// errInterrupted marks an operation that SIGINT or SIGTERM stopped
|
||||
// before it finished. Entry shows it and returns exitCodeInterrupted.
|
||||
var errInterrupted = errors.New("interrupted before the command finished")
|
||||
|
||||
// RunOperation runs op against the Vaultik instance inside the fx app
|
||||
// and turns a failure into a returned error rather than an os.Exit from
|
||||
// within the goroutine. An os.Exit there skipped main's deferred
|
||||
@@ -220,17 +224,21 @@ var errReported = errors.New("operation failed")
|
||||
// interrupt OnStop cancels op and waits for the goroutine to return, so
|
||||
// op's cleanup (removing decrypted scratch files) runs before the
|
||||
// process exits; the wait is bounded by shutdownTimeout. report is
|
||||
// called with a non-canceled failure so the caller can show it to the
|
||||
// user before it becomes errReported. A context cancellation is the
|
||||
// interrupt path, not a failure: it is neither reported nor counted as
|
||||
// one.
|
||||
// called with a failure so the caller can show it to the user before
|
||||
// it becomes errReported.
|
||||
//
|
||||
// The run counts as interrupted unless op returned, without an
|
||||
// interrupt having cancelled it, before RunWithApp returned. An
|
||||
// interrupted op is not reported, whatever it returned; RunOperation
|
||||
// returns errInterrupted instead.
|
||||
func RunOperation(
|
||||
ctx context.Context, opts AppOptions,
|
||||
op func(v *vaultik.Vaultik) error, report func(err error),
|
||||
) error {
|
||||
var (
|
||||
mu sync.Mutex
|
||||
failed bool
|
||||
mu sync.Mutex
|
||||
finished bool // op returned before any interrupt cancelled it
|
||||
failed bool // op finished with an error
|
||||
)
|
||||
|
||||
opts.Invokes = append(opts.Invokes,
|
||||
@@ -241,11 +249,21 @@ func RunOperation(
|
||||
OnStart: func(_ context.Context) error {
|
||||
stop = v.StartOperation(func() {
|
||||
err := op(v)
|
||||
if err != nil && !errors.Is(err, context.Canceled) {
|
||||
report(err)
|
||||
|
||||
// Only stop, called from OnStop below, cancels the
|
||||
// Vaultik context, so a live context means no
|
||||
// interrupt cancelled op. Check the context, not
|
||||
// err: an interrupted op need not return
|
||||
// context.Canceled (`snapshot verify --json`
|
||||
// returns a verification failure).
|
||||
if v.Context().Err() == nil {
|
||||
if err != nil {
|
||||
report(err)
|
||||
}
|
||||
|
||||
mu.Lock()
|
||||
failed = true
|
||||
finished = true
|
||||
failed = err != nil
|
||||
mu.Unlock()
|
||||
}
|
||||
|
||||
@@ -278,16 +296,23 @@ func RunOperation(
|
||||
return err
|
||||
}
|
||||
|
||||
// The goroutine sets failed before triggering the shutdown that lets
|
||||
// RunWithApp return, so the write is in place by the time we read it.
|
||||
// RunWithApp returns only after the app was asked to stop, either by
|
||||
// an interrupt or by the goroutine's Shutdown call. When op finished
|
||||
// without being cancelled, the goroutine set finished before that
|
||||
// call. So if finished is unset here, an interrupt stopped the app,
|
||||
// and op either returned after it was cancelled or is still running
|
||||
// because the shutdown timed out.
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
|
||||
if failed {
|
||||
switch {
|
||||
case !finished:
|
||||
return errInterrupted
|
||||
case failed:
|
||||
return errReported
|
||||
default:
|
||||
return nil
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// runVaultikApp runs the standard single-operation command lifecycle
|
||||
|
||||
+56
-1
@@ -7,11 +7,14 @@ import (
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strconv"
|
||||
"strings"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
"gopkg.in/yaml.v3"
|
||||
"sneak.berlin/go/vaultik/internal/config"
|
||||
"sneak.berlin/go/vaultik/internal/ui"
|
||||
)
|
||||
|
||||
@@ -31,6 +34,9 @@ const configDirMode = 0o755
|
||||
// yaml.Marshal's 4-space default.
|
||||
const configYAMLIndent = 2
|
||||
|
||||
// yamlStringTag is YAML's tag for a string scalar.
|
||||
const yamlStringTag = "!!str"
|
||||
|
||||
var (
|
||||
errConfigExists = errors.New("config file already exists")
|
||||
errEmptyConfig = errors.New("empty config file")
|
||||
@@ -199,7 +205,7 @@ storage_url: ""
|
||||
# access_key_id: YOUR_ACCESS_KEY
|
||||
# secret_access_key: YOUR_SECRET_KEY
|
||||
# # region: us-east-1 # Default: us-east-1
|
||||
# # part_size: 5MB # Multipart upload part size. Default: 5MB
|
||||
# # part_size: 5MiB # Upload part size, 5MiB to 5GiB. Default: 5MiB
|
||||
# # For the s3:// form, disable TLS with ?ssl=false in the URL, not use_ssl.
|
||||
|
||||
# ─── OPTIONAL ────────────────────────────────────────────────────────────────
|
||||
@@ -583,9 +589,58 @@ func yamlPathSet(root *yaml.Node, keys []string, value string) error {
|
||||
}
|
||||
}
|
||||
|
||||
// config.Load reads the file through untyped YAML, which turns an
|
||||
// unquoted 00112233 into the number 38043 and 1e5 into 100000. Tagging
|
||||
// a string setting as a string makes the encoder quote such a value.
|
||||
// Other settings stay unquoted, so compression_level 9 is a number.
|
||||
// The encoder refuses to write a value that is not valid UTF-8 as a
|
||||
// string. Left untagged, such a value is written as base64 !!binary and
|
||||
// loads back unchanged.
|
||||
if configKeyIsString(keys) && utf8.ValidString(value) {
|
||||
node.Tag = yamlStringTag
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// configKeyIsString reports whether the dotted key names a string in
|
||||
// config.Config, following the fields' yaml tags, as s3.access_key_id and
|
||||
// snapshots.home.exclude.0 do.
|
||||
func configKeyIsString(keys []string) bool {
|
||||
typ := reflect.TypeFor[config.Config]()
|
||||
|
||||
for _, key := range keys {
|
||||
switch {
|
||||
case typ.Kind() == reflect.Map || typ.Kind() == reflect.Slice:
|
||||
// The key is a snapshot name or a list index.
|
||||
typ = typ.Elem()
|
||||
case typ.Kind() == reflect.Struct:
|
||||
field, ok := yamlField(typ, key)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
|
||||
typ = field.Type
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
return typ.Kind() == reflect.String
|
||||
}
|
||||
|
||||
// yamlField returns the field of struct type typ whose yaml tag names key.
|
||||
func yamlField(typ reflect.Type, key string) (reflect.StructField, bool) {
|
||||
for field := range typ.Fields() {
|
||||
name, _, _ := strings.Cut(field.Tag.Get("yaml"), ",")
|
||||
if name == key {
|
||||
return field, true
|
||||
}
|
||||
}
|
||||
|
||||
return reflect.StructField{}, false
|
||||
}
|
||||
|
||||
// yamlSetInMapping resolves (creating if needed) the value node for key
|
||||
// within a mapping node, setting it to value when it is the final path
|
||||
// element, and returns the node to descend into.
|
||||
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"bytes"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
@@ -94,6 +95,110 @@ func TestConfigSetRecipientOnFreshConfig(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestConfigSetStringLooksLikeNumber sets string settings to values that
|
||||
// YAML reads as numbers or booleans when they are unquoted, and checks that
|
||||
// config.Load returns each one unchanged.
|
||||
func TestConfigSetStringLooksLikeNumber(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
key string
|
||||
value string
|
||||
field func(cfg *config.Config) string
|
||||
}{
|
||||
{"s3.access_key_id", "00112233",
|
||||
func(cfg *config.Config) string { return cfg.S3.AccessKeyID }},
|
||||
{"s3.secret_access_key", "12345678901234567890123456789012",
|
||||
func(cfg *config.Config) string { return cfg.S3.SecretAccessKey }},
|
||||
{"hostname", "007",
|
||||
func(cfg *config.Config) string { return cfg.Hostname }},
|
||||
{"s3.prefix", "1e5",
|
||||
func(cfg *config.Config) string { return cfg.S3.Prefix }},
|
||||
{"s3.bucket", "true",
|
||||
func(cfg *config.Config) string { return cfg.S3.Bucket }},
|
||||
{"s3.region", "FALSE",
|
||||
func(cfg *config.Config) string { return cfg.S3.Region }},
|
||||
{"snapshots.home.exclude.0", "1.10",
|
||||
func(cfg *config.Config) string { return cfg.Snapshots["home"].Exclude[0] }},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.key+"="+tt.value, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cfg := loadAfterConfigSet(t, tt.key, tt.value)
|
||||
|
||||
got := tt.field(cfg)
|
||||
if got != tt.value {
|
||||
t.Errorf("%s = %q after config set %q", tt.key, got, tt.value)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestConfigSetNonUTF8Path checks that config set still accepts a value that
|
||||
// is not valid UTF-8, such as a path with a Latin-1 file name, and that
|
||||
// config.Load returns it unchanged.
|
||||
func TestConfigSetNonUTF8Path(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const dir = "/srv/caf\xe9"
|
||||
|
||||
cfg := loadAfterConfigSet(t, "snapshots.home.paths.0", dir)
|
||||
|
||||
got := cfg.Snapshots["home"].Paths[0]
|
||||
if got != dir {
|
||||
t.Errorf("snapshots.home.paths.0 = %q, want %q", got, dir)
|
||||
}
|
||||
}
|
||||
|
||||
// TestConfigSetNumberStaysNumber checks that a number set for an integer
|
||||
// setting is still read as a number, not as a quoted string.
|
||||
func TestConfigSetNumberStaysNumber(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const level = 9
|
||||
|
||||
cfg := loadAfterConfigSet(t, "compression_level", strconv.Itoa(level))
|
||||
|
||||
if cfg.CompressionLevel != level {
|
||||
t.Errorf("compression_level = %d, want %d", cfg.CompressionLevel, level)
|
||||
}
|
||||
}
|
||||
|
||||
// loadAfterConfigSet writes the file `config init` writes, sets storage_url
|
||||
// to a local directory so that the file passes validation, applies
|
||||
// `config set key value` and returns what config.Load reads back.
|
||||
func loadAfterConfigSet(t *testing.T, key, value string) *config.Config {
|
||||
t.Helper()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "config.yml")
|
||||
|
||||
err := os.WriteFile(path, []byte(defaultConfigTemplate), configFileMode)
|
||||
if err != nil {
|
||||
t.Fatalf("write config: %v", err)
|
||||
}
|
||||
|
||||
out := ui.NewWithColor(&bytes.Buffer{}, false)
|
||||
|
||||
err = writeConfigSet(out, path, "storage_url", "file:///mnt/backups")
|
||||
if err != nil {
|
||||
t.Fatalf("config set storage_url: %v", err)
|
||||
}
|
||||
|
||||
err = writeConfigSet(out, path, key, value)
|
||||
if err != nil {
|
||||
t.Fatalf("config set %s: %v", key, err)
|
||||
}
|
||||
|
||||
cfg, err := config.Load(path)
|
||||
if err != nil {
|
||||
t.Fatalf("config.Load: %v", err)
|
||||
}
|
||||
|
||||
return cfg
|
||||
}
|
||||
|
||||
const testYAML = `# top comment
|
||||
compression_level: 3
|
||||
age_recipients:
|
||||
|
||||
+15
-4
@@ -15,6 +15,11 @@ import (
|
||||
// the startup banner.
|
||||
const shortCommitLen = 12
|
||||
|
||||
// exitCodeInterrupted is the exit status of a command that SIGINT or
|
||||
// SIGTERM stopped. It is 128 plus SIGINT's number, 2, which is what a
|
||||
// shell reports for a command stopped by Ctrl-C.
|
||||
const exitCodeInterrupted = 130
|
||||
|
||||
// Entry is the main entry point for the CLI application.
|
||||
// It prints the startup banner to stderr (unless a banner-suppressing
|
||||
// flag is present in os.Args — see bannerSuppressedInArgs), executes the
|
||||
@@ -23,9 +28,10 @@ const shortCommitLen = 12
|
||||
// The banner goes to stderr because stdout carries only the output the
|
||||
// user asked for, such as a completion script or a `config get` value.
|
||||
//
|
||||
// It returns the process exit code (0 on success, 1 on error) rather
|
||||
// than calling os.Exit, so that main's deferred profile writers run
|
||||
// before the process ends. See run in cmd/vaultik/main.go.
|
||||
// It returns the process exit code (0 on success, 130 when interrupted,
|
||||
// 1 on any other error) rather than calling os.Exit, so that main's
|
||||
// deferred profile writers run before the process ends. See run in
|
||||
// cmd/vaultik/main.go.
|
||||
func Entry() int {
|
||||
emitStartupBanner(os.Args[1:], os.Stderr)
|
||||
|
||||
@@ -39,11 +45,16 @@ func Entry() int {
|
||||
// document instead); errReported says so. Printing it again
|
||||
// here would double the error line.
|
||||
// Every other error — bad arguments, a config that would not
|
||||
// load — reaches Entry unreported, so it is shown here.
|
||||
// load, an interrupt — reaches Entry unreported, so it is shown
|
||||
// here.
|
||||
if !errors.Is(err, errReported) {
|
||||
ReportErrorf("%s", err.Error())
|
||||
}
|
||||
|
||||
if errors.Is(err, errInterrupted) {
|
||||
return exitCodeInterrupted
|
||||
}
|
||||
|
||||
return 1
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,203 @@
|
||||
package cli //nolint:testpackage // shares runEntry and the argument constants
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"os/signal"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/adrg/xdg"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// stalledStoreConfig is hermeticConfig with an s3:// destination store
|
||||
// in place of the file:// one. The server behind it accepts any
|
||||
// credentials.
|
||||
const stalledStoreConfig = `age_recipients:
|
||||
- age1278m9q7dp3chsh2dcy82qk27v047zywyvtxwnj4cvt0z65jw6a7q5dqhfj
|
||||
snapshots:
|
||||
test:
|
||||
paths:
|
||||
- %s
|
||||
storage_url: s3://bucket?endpoint=%s&ssl=false
|
||||
s3:
|
||||
access_key_id: key
|
||||
secret_access_key: secret
|
||||
index_path: %s
|
||||
hostname: test-host
|
||||
`
|
||||
|
||||
// interruptRepeat is how often interruptOnFirstRequest sends SIGINT.
|
||||
const interruptRepeat = 50 * time.Millisecond
|
||||
|
||||
// TestEntryInterruptedRun sends SIGINT to the test process while a
|
||||
// command waits on the destination store, and checks that Entry returns
|
||||
// 130 and prints one line on stderr saying the run was interrupted. The
|
||||
// store is a local HTTP server that holds every request open, so the
|
||||
// command is always mid-operation when the signal arrives. The two
|
||||
// cases cover --cron and --json, which silence other output.
|
||||
//
|
||||
// Not parallel: it signals the process and replaces os.Args, os.Stdout,
|
||||
// os.Stderr and the xdg globals.
|
||||
//
|
||||
//nolint:paralleltest // signals the process and replaces process globals
|
||||
func TestEntryInterruptedRun(t *testing.T) {
|
||||
for _, testCase := range []struct {
|
||||
name string
|
||||
args []string
|
||||
}{
|
||||
{
|
||||
name: "snapshot create --cron",
|
||||
args: []string{cmdSnapshot, cmdCreate, "--cron"},
|
||||
},
|
||||
{
|
||||
name: "snapshot verify --json",
|
||||
args: []string{cmdSnapshot, cmdVerify, someSnapshotID, flagJSON},
|
||||
},
|
||||
} {
|
||||
t.Run(testCase.name, func(t *testing.T) {
|
||||
endpoint, requestArrived := startStalledStore(t)
|
||||
configPath := writeStalledStoreConfig(t, endpoint)
|
||||
interruptOnFirstRequest(t, requestArrived)
|
||||
|
||||
code, _, stderr := runEntry(t,
|
||||
append([]string{flagConfig, configPath}, testCase.args...)...)
|
||||
|
||||
assert.Equal(t, 130, code)
|
||||
assert.Equal(t, 1,
|
||||
strings.Count(stderr, errInterrupted.Error()), stderr)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// interruptOnFirstRequest sends SIGINT to the test process every
|
||||
// interruptRepeat, from the first request to the destination store until
|
||||
// the test ends. One signal is not enough: the command can reach the
|
||||
// store before fx has started catching signals. The test catches SIGINT
|
||||
// too, so that a signal fx is not catching does not kill the test
|
||||
// binary.
|
||||
func interruptOnFirstRequest(t *testing.T, requestArrived <-chan struct{}) {
|
||||
t.Helper()
|
||||
|
||||
self, err := os.FindProcess(os.Getpid())
|
||||
require.NoError(t, err)
|
||||
|
||||
caught := make(chan os.Signal, 1)
|
||||
signal.Notify(caught, os.Interrupt)
|
||||
|
||||
testEnded := make(chan struct{})
|
||||
senderDone := make(chan struct{})
|
||||
|
||||
// Stop catching SIGINT only after the sender has returned. The sender
|
||||
// waits for each SIGINT it sends to arrive on caught; one still on
|
||||
// its way after signal.Stop would kill the test binary.
|
||||
t.Cleanup(func() {
|
||||
close(testEnded)
|
||||
<-senderDone
|
||||
signal.Stop(caught)
|
||||
})
|
||||
|
||||
go func() {
|
||||
defer close(senderDone)
|
||||
|
||||
select {
|
||||
case <-requestArrived:
|
||||
case <-testEnded:
|
||||
return
|
||||
}
|
||||
|
||||
ticker := time.NewTicker(interruptRepeat)
|
||||
defer ticker.Stop()
|
||||
|
||||
for {
|
||||
// Empty caught, so that the receive below waits for this
|
||||
// SIGINT rather than an earlier one.
|
||||
select {
|
||||
case <-caught:
|
||||
default:
|
||||
}
|
||||
|
||||
sendErr := self.Signal(os.Interrupt)
|
||||
if sendErr != nil {
|
||||
t.Errorf("sending SIGINT: %v", sendErr)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
<-caught
|
||||
|
||||
select {
|
||||
case <-testEnded:
|
||||
return
|
||||
case <-ticker.C:
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
// startStalledStore starts an HTTP server that never answers: each
|
||||
// request is held until the client gives up on it or the test ends.
|
||||
// It returns the server's host:port and a channel that receives a value
|
||||
// when the first request arrives.
|
||||
func startStalledStore(t *testing.T) (string, <-chan struct{}) {
|
||||
t.Helper()
|
||||
|
||||
requestArrived := make(chan struct{}, 1)
|
||||
release := make(chan struct{})
|
||||
|
||||
server := httptest.NewServer(http.HandlerFunc(
|
||||
func(_ http.ResponseWriter, r *http.Request) {
|
||||
select {
|
||||
case requestArrived <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
|
||||
select {
|
||||
case <-r.Context().Done():
|
||||
case <-release:
|
||||
}
|
||||
}))
|
||||
|
||||
// Cleanups run last-registered first, so release lets any held
|
||||
// request return before Close waits for it.
|
||||
t.Cleanup(server.Close)
|
||||
t.Cleanup(func() { close(release) })
|
||||
|
||||
return server.Listener.Addr().String(), requestArrived
|
||||
}
|
||||
|
||||
// writeStalledStoreConfig writes a config whose destination store is the
|
||||
// server at endpoint and whose snapshot source holds one small file, so
|
||||
// that `snapshot create` has a blob to upload. Returns the config path.
|
||||
func writeStalledStoreConfig(t *testing.T, endpoint string) string {
|
||||
t.Helper()
|
||||
|
||||
dir := t.TempDir()
|
||||
configPath := filepath.Join(dir, "config.yml")
|
||||
sourceDir := filepath.Join(dir, "source")
|
||||
|
||||
require.NoError(t, os.Mkdir(sourceDir, 0o750))
|
||||
require.NoError(t, os.WriteFile(filepath.Join(sourceDir, "file.txt"),
|
||||
[]byte("contents"), 0o600))
|
||||
|
||||
contents := fmt.Sprintf(stalledStoreConfig,
|
||||
sourceDir, endpoint, filepath.Join(dir, "index.sqlite"))
|
||||
|
||||
require.NoError(t,
|
||||
os.WriteFile(configPath, []byte(contents), configFileMode))
|
||||
|
||||
// The PID lock lives under xdg.DataHome, which xdg resolves at
|
||||
// package init; point it at the temp dir so the test neither
|
||||
// touches nor collides with the real one.
|
||||
t.Setenv("XDG_DATA_HOME", filepath.Join(dir, "data"))
|
||||
xdg.Reload()
|
||||
t.Cleanup(xdg.Reload)
|
||||
|
||||
return configPath
|
||||
}
|
||||
@@ -22,9 +22,11 @@ scans every snapshot manifest in the destination store, builds the
|
||||
set of still-referenced blob hashes, and deletes any blob not in that
|
||||
set.
|
||||
|
||||
Snapshot create --prune and snapshot remove run the same cleanup
|
||||
automatically; this command is the manual entry point for the same
|
||||
work (e.g. after a crashed backup or to reclaim storage).`,
|
||||
Snapshot create --prune runs the same cleanup automatically; this
|
||||
command is the manual entry point for the same work (e.g. after a
|
||||
crashed backup or to reclaim storage). Snapshot remove leaves blobs in
|
||||
place; run this command afterwards to delete the ones no longer
|
||||
referenced.`,
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
// Use unified config resolution
|
||||
|
||||
@@ -66,8 +66,9 @@ func newSnapshotCreateCommand() *cobra.Command {
|
||||
If snapshot names are provided, only those snapshots are created.
|
||||
If no names are provided, all configured snapshots are created.
|
||||
|
||||
Config is located at /etc/vaultik/config.yml by default, but can be overridden by
|
||||
specifying a path using --config or by setting VAULTIK_CONFIG to a path.`,
|
||||
The config is read from the path given by --config or VAULTIK_CONFIG;
|
||||
otherwise from the platform config directory (~/.config/vaultik/config.yml
|
||||
on Linux), then /etc/vaultik/config.yml.`,
|
||||
Args: cobra.ArbitraryArgs,
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
// Pass snapshot names from args
|
||||
|
||||
@@ -33,11 +33,14 @@ const secretKeyPrefix = "AGE-SECRET-KEY-"
|
||||
const (
|
||||
defaultBlobSizeLimit = Size(10 * 1024 * 1024 * 1024) // 10GB
|
||||
defaultChunkSize = Size(10 * 1024 * 1024) // 10MB
|
||||
defaultS3PartSize = Size(5 * 1024 * 1024) // 5MB
|
||||
defaultS3PartSize = Size(5 * 1024 * 1024) // 5MiB
|
||||
defaultCompressionLevel = 3
|
||||
minChunkSize = 1024 * 1024 // 1MB
|
||||
minCompressionLevel = 1
|
||||
maxCompressionLevel = 19
|
||||
// S3 accepts a multipart upload part from 5MiB to 5GiB.
|
||||
minS3PartSize = 5 * 1024 * 1024
|
||||
maxS3PartSize = 5 * 1024 * 1024 * 1024
|
||||
)
|
||||
|
||||
// Sentinel validation errors.
|
||||
@@ -55,6 +58,7 @@ var (
|
||||
"blob_size_limit must be at least the largest chunk the chunker can " +
|
||||
"emit (chunk_size times the FastCDC size spread)")
|
||||
errBadCompression = errors.New("compression_level must be between 1 and 19")
|
||||
errBadS3PartSize = errors.New("s3.part_size must be between 5MiB and 5GiB")
|
||||
errBadStorageScheme = errors.New(
|
||||
"storage_url must start with s3://, file://, or rclone://")
|
||||
errStorageNotConfigured = errors.New(
|
||||
@@ -244,6 +248,7 @@ func Load(path string) (*Config, error) {
|
||||
ChunkSize: defaultChunkSize,
|
||||
IndexPath: filepath.Join(xdg.DataHome, appName, "index.sqlite"),
|
||||
CompressionLevel: defaultCompressionLevel,
|
||||
S3: S3Config{PartSize: defaultS3PartSize},
|
||||
}
|
||||
|
||||
// Convert smartconfig data to YAML then unmarshal
|
||||
@@ -294,17 +299,13 @@ func Load(path string) (*Config, error) {
|
||||
cfg.S3.Region = "us-east-1"
|
||||
}
|
||||
|
||||
if cfg.S3.PartSize == 0 {
|
||||
cfg.S3.PartSize = defaultS3PartSize
|
||||
}
|
||||
|
||||
// Check config file permissions (warn if world or group readable)
|
||||
//nolint:gosec // G703: config path is operator-supplied by design
|
||||
info, statErr := os.Stat(path)
|
||||
if statErr == nil {
|
||||
mode := info.Mode().Perm()
|
||||
if mode&0044 != 0 { // group or world readable
|
||||
log.Warn("Config file has insecure permissions (contains S3 credentials)",
|
||||
log.Warn(cfg.readableByOthersWarning(),
|
||||
"path", path,
|
||||
"mode", fmt.Sprintf("%04o", mode),
|
||||
"recommendation", "chmod 600 "+path)
|
||||
@@ -332,6 +333,7 @@ func Load(path string) (*Config, error) {
|
||||
// (chunk_size times chunker.ChunkSizeSpread), so a single-chunk blob never
|
||||
// exceeds the configured limit
|
||||
// - Compression level must be between 1 and 19
|
||||
// - S3 part size must be between 5MiB and 5GiB, the part sizes S3 accepts
|
||||
//
|
||||
// Returns an error describing the first validation failure encountered.
|
||||
func (c *Config) Validate() error {
|
||||
@@ -376,6 +378,11 @@ func (c *Config) Validate() error {
|
||||
return errBadCompression
|
||||
}
|
||||
|
||||
if c.S3.PartSize.Int64() < minS3PartSize ||
|
||||
c.S3.PartSize.Int64() > maxS3PartSize {
|
||||
return errBadS3PartSize
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -411,6 +418,18 @@ func (c *Config) setAgeSecretKey() {
|
||||
}
|
||||
}
|
||||
|
||||
// readableByOthersWarning is the warning Load logs when others can read
|
||||
// the config file. It says "may contain" because the S3 credentials are
|
||||
// seen only after smartconfig has replaced any ${...} reference in the
|
||||
// file with its value, so a set credential need not be in the file.
|
||||
func (c *Config) readableByOthersWarning() string {
|
||||
if c.S3.AccessKeyID != "" || c.S3.SecretAccessKey != "" {
|
||||
return "Config file is readable by others and may contain S3 credentials"
|
||||
}
|
||||
|
||||
return "Config file is readable by others"
|
||||
}
|
||||
|
||||
// validateStorage validates storage configuration.
|
||||
// If StorageURL is set, it takes precedence. S3 URLs require credentials.
|
||||
// File URLs don't require any S3 configuration.
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/vaultik/internal/chunker"
|
||||
"sneak.berlin/go/vaultik/internal/log"
|
||||
)
|
||||
|
||||
const (
|
||||
@@ -166,6 +167,7 @@ func TestValidateBlobSizeLimit(t *testing.T) {
|
||||
ChunkSize: chunkSize,
|
||||
BlobSizeLimit: blobLimit,
|
||||
CompressionLevel: 3,
|
||||
S3: S3Config{PartSize: defaultS3PartSize},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -221,6 +223,120 @@ func TestValidateBlobSizeLimit(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestValidateS3PartSize checks that s3.part_size is held to the part sizes
|
||||
// S3 accepts, 5MiB to 5GiB, by changing only the part size of the test
|
||||
// config. "5MB" in the config file is 5,000,000 bytes, below the minimum.
|
||||
func TestValidateS3PartSize(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
base, err := Load(os.Getenv("VAULTIK_CONFIG"))
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to load config: %v", err)
|
||||
}
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
partSize Size
|
||||
wantErr bool
|
||||
}{
|
||||
{
|
||||
name: "5MB is rejected",
|
||||
partSize: 5_000_000,
|
||||
wantErr: true,
|
||||
},
|
||||
{
|
||||
name: "one byte below 5MiB is rejected",
|
||||
partSize: minS3PartSize - 1,
|
||||
wantErr: true,
|
||||
},
|
||||
{
|
||||
name: "5MiB is accepted",
|
||||
partSize: minS3PartSize,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "5GiB is accepted",
|
||||
partSize: maxS3PartSize,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "one byte above 5GiB is rejected",
|
||||
partSize: maxS3PartSize + 1,
|
||||
wantErr: true,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cfg := *base
|
||||
cfg.S3.PartSize = tt.partSize
|
||||
|
||||
err := cfg.Validate()
|
||||
if tt.wantErr {
|
||||
if !errors.Is(err, errBadS3PartSize) {
|
||||
t.Fatalf("Validate() error = %v, want errBadS3PartSize", err)
|
||||
}
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
t.Fatalf("Validate() unexpected error: %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestLoadS3PartSize checks that a config file without s3.part_size loads
|
||||
// with the 5MiB default, and that an explicit 0 fails at load like any other
|
||||
// part size S3 refuses.
|
||||
func TestLoadS3PartSize(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const withoutPartSize = "snapshots:\n" +
|
||||
" test:\n" +
|
||||
" paths: [/tmp/vaultik-test-source]\n" +
|
||||
"storage_url: file:///tmp/vaultik-test-storage\n"
|
||||
|
||||
writeConfig := func(t *testing.T, text string) string {
|
||||
t.Helper()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "config.yml")
|
||||
|
||||
err := os.WriteFile(path, []byte(text), 0o600)
|
||||
if err != nil {
|
||||
t.Fatalf("write config: %v", err)
|
||||
}
|
||||
|
||||
return path
|
||||
}
|
||||
|
||||
t.Run("absent loads as 5MiB", func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cfg, err := Load(writeConfig(t, withoutPartSize))
|
||||
if err != nil {
|
||||
t.Fatalf("Load() unexpected error: %v", err)
|
||||
}
|
||||
|
||||
if cfg.S3.PartSize != defaultS3PartSize {
|
||||
t.Errorf("s3.part_size = %d, want %d",
|
||||
cfg.S3.PartSize, defaultS3PartSize)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("0 is rejected", func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := Load(writeConfig(t, withoutPartSize+"s3:\n part_size: 0\n"))
|
||||
if !errors.Is(err, errBadS3PartSize) {
|
||||
t.Fatalf("Load() error = %v, want errBadS3PartSize", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// TestValidateAgeRecipients checks that recipients are parsed at config load
|
||||
// (a bad entry fails immediately, not mid-backup) and that no invalid entry —
|
||||
// least of all a pasted secret key — is echoed in the error. An empty list
|
||||
@@ -236,6 +352,7 @@ func TestValidateAgeRecipients(t *testing.T) {
|
||||
ChunkSize: Size(10 * 1024 * 1024),
|
||||
BlobSizeLimit: Size(10 * 1024 * 1024 * 1024),
|
||||
CompressionLevel: 3,
|
||||
S3: S3Config{PartSize: defaultS3PartSize},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -343,3 +460,125 @@ func TestAgeSecretKeySourceName(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// loadReadableConfig writes configYAML to a file that others can read,
|
||||
// loads it, and returns what the logger wrote to stderr meanwhile. The
|
||||
// logger writes to the os.Stderr it finds when it is initialized, so
|
||||
// os.Stderr is pointed at a file first. Not parallel-safe: os.Stderr and
|
||||
// the logger are process-global.
|
||||
func loadReadableConfig(t *testing.T, configYAML string) string {
|
||||
t.Helper()
|
||||
|
||||
dir := t.TempDir()
|
||||
configPath := filepath.Join(dir, "config.yml")
|
||||
stderrPath := filepath.Join(dir, "stderr")
|
||||
|
||||
err := os.WriteFile(configPath, []byte(configYAML), 0o600)
|
||||
if err != nil {
|
||||
t.Fatalf("writing config: %v", err)
|
||||
}
|
||||
|
||||
//nolint:gosec // G302: the test needs a config file others can read
|
||||
err = os.Chmod(configPath, 0o644)
|
||||
if err != nil {
|
||||
t.Fatalf("chmod config: %v", err)
|
||||
}
|
||||
|
||||
stderrFile, err := os.Create(stderrPath) //nolint:gosec // G304: test temp path
|
||||
if err != nil {
|
||||
t.Fatalf("creating stderr file: %v", err)
|
||||
}
|
||||
|
||||
previous := os.Stderr
|
||||
os.Stderr = stderrFile
|
||||
|
||||
log.Initialize(log.Config{})
|
||||
|
||||
_, loadErr := Load(configPath)
|
||||
|
||||
os.Stderr = previous
|
||||
|
||||
log.Initialize(log.Config{})
|
||||
|
||||
_ = stderrFile.Close()
|
||||
|
||||
if loadErr != nil {
|
||||
t.Fatalf("Load() error = %v", loadErr)
|
||||
}
|
||||
|
||||
captured, err := os.ReadFile(stderrPath) //nolint:gosec // G304: test temp path
|
||||
if err != nil {
|
||||
t.Fatalf("reading stderr file: %v", err)
|
||||
}
|
||||
|
||||
return string(captured)
|
||||
}
|
||||
|
||||
// TestLoadWarnsReadableConfigWithoutS3Credentials checks that a config
|
||||
// file others can read, holding no S3 credentials, is warned about
|
||||
// without a claim that it holds them.
|
||||
//
|
||||
//nolint:paralleltest // loadReadableConfig replaces os.Stderr
|
||||
func TestLoadWarnsReadableConfigWithoutS3Credentials(t *testing.T) {
|
||||
stderr := loadReadableConfig(t, `
|
||||
storage_url: file:///var/backups/vaultik
|
||||
snapshots:
|
||||
home:
|
||||
paths:
|
||||
- /home
|
||||
`)
|
||||
|
||||
if !strings.Contains(stderr, "Config file is readable by others") {
|
||||
t.Errorf("expected a warning that the file is readable by others, got %q",
|
||||
stderr)
|
||||
}
|
||||
|
||||
if strings.Contains(stderr, "S3 credentials") {
|
||||
t.Errorf("warning names S3 credentials the file does not set: %q", stderr)
|
||||
}
|
||||
}
|
||||
|
||||
// TestLoadWarnsReadableConfigWithS3Credentials checks that a config file
|
||||
// others can read and that sets S3 credentials, as values or as ${ENV:...}
|
||||
// references, is warned about as one that may contain them.
|
||||
//
|
||||
//nolint:paralleltest // loadReadableConfig replaces os.Stderr
|
||||
func TestLoadWarnsReadableConfigWithS3Credentials(t *testing.T) {
|
||||
t.Setenv("VAULTIK_TEST_ACCESS_KEY_ID", "test-access-key")
|
||||
t.Setenv("VAULTIK_TEST_SECRET_ACCESS_KEY", "test-secret-key")
|
||||
|
||||
configs := map[string]string{
|
||||
"values": `
|
||||
storage_url: s3://bucket/prefix?endpoint=s3.example.com
|
||||
s3:
|
||||
access_key_id: test-access-key
|
||||
secret_access_key: test-secret-key
|
||||
snapshots:
|
||||
home:
|
||||
paths:
|
||||
- /home
|
||||
`,
|
||||
"references": `
|
||||
storage_url: s3://bucket/prefix?endpoint=s3.example.com
|
||||
s3:
|
||||
access_key_id: ${ENV:VAULTIK_TEST_ACCESS_KEY_ID}
|
||||
secret_access_key: ${ENV:VAULTIK_TEST_SECRET_ACCESS_KEY}
|
||||
snapshots:
|
||||
home:
|
||||
paths:
|
||||
- /home
|
||||
`,
|
||||
}
|
||||
|
||||
for name, configYAML := range configs {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
stderr := loadReadableConfig(t, configYAML)
|
||||
|
||||
if !strings.Contains(stderr,
|
||||
"Config file is readable by others and may contain S3 credentials") {
|
||||
t.Errorf("expected a warning naming the S3 credentials, got %q",
|
||||
stderr)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
+28
-19
@@ -33,11 +33,13 @@ func (r *FileRepository) Create(ctx context.Context, tx *sql.Tx, file *File) err
|
||||
}
|
||||
|
||||
query := `
|
||||
INSERT INTO files (id, path, source_path, mtime, size, mode, uid, gid, link_target)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
INSERT INTO files
|
||||
(id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(path) DO UPDATE SET
|
||||
source_path = excluded.source_path,
|
||||
mtime = excluded.mtime,
|
||||
mtime_nsec = excluded.mtime_nsec,
|
||||
size = excluded.size,
|
||||
mode = excluded.mode,
|
||||
uid = excluded.uid,
|
||||
@@ -54,16 +56,19 @@ func (r *FileRepository) Create(ctx context.Context, tx *sql.Tx, file *File) err
|
||||
if tx != nil {
|
||||
LogSQL("Execute", query,
|
||||
file.ID.String(), file.Path.String(), file.SourcePath.String(),
|
||||
file.MTime.Unix(), file.Size, file.Mode, file.UID, file.GID,
|
||||
file.MTime.Unix(), file.MTime.Nanosecond(),
|
||||
file.Size, file.Mode, file.UID, file.GID,
|
||||
file.LinkTarget.String())
|
||||
err = tx.QueryRowContext(ctx, query,
|
||||
file.ID.String(), file.Path.String(), file.SourcePath.String(),
|
||||
file.MTime.Unix(), file.Size, file.Mode, file.UID, file.GID,
|
||||
file.MTime.Unix(), file.MTime.Nanosecond(),
|
||||
file.Size, file.Mode, file.UID, file.GID,
|
||||
file.LinkTarget.String()).Scan(&idStr)
|
||||
} else {
|
||||
err = r.db.QueryRowWithLog(ctx, query,
|
||||
file.ID.String(), file.Path.String(), file.SourcePath.String(),
|
||||
file.MTime.Unix(), file.Size, file.Mode, file.UID, file.GID,
|
||||
file.MTime.Unix(), file.MTime.Nanosecond(),
|
||||
file.Size, file.Mode, file.UID, file.GID,
|
||||
file.LinkTarget.String()).Scan(&idStr)
|
||||
}
|
||||
|
||||
@@ -84,7 +89,7 @@ func (r *FileRepository) Create(ctx context.Context, tx *sql.Tx, file *File) err
|
||||
// in the index.
|
||||
func (r *FileRepository) GetByPath(ctx context.Context, path string) (*File, error) {
|
||||
query := `
|
||||
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
|
||||
SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
|
||||
FROM files
|
||||
WHERE path = ?
|
||||
`
|
||||
@@ -104,7 +109,7 @@ func (r *FileRepository) GetByPath(ctx context.Context, path string) (*File, err
|
||||
// GetByID retrieves a file by its UUID
|
||||
func (r *FileRepository) GetByID(ctx context.Context, id types.FileID) (*File, error) {
|
||||
query := `
|
||||
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
|
||||
SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
|
||||
FROM files
|
||||
WHERE id = ?
|
||||
`
|
||||
@@ -127,7 +132,7 @@ func (r *FileRepository) GetByPathTx(
|
||||
ctx context.Context, tx *sql.Tx, path string,
|
||||
) (*File, error) {
|
||||
query := `
|
||||
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
|
||||
SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
|
||||
FROM files
|
||||
WHERE path = ?
|
||||
`
|
||||
@@ -158,13 +163,14 @@ func (r *FileRepository) ListModifiedSince(
|
||||
ctx context.Context, since time.Time,
|
||||
) ([]*File, error) {
|
||||
query := `
|
||||
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
|
||||
SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
|
||||
FROM files
|
||||
WHERE mtime >= ?
|
||||
WHERE (mtime, mtime_nsec) >= (?, ?)
|
||||
ORDER BY path
|
||||
`
|
||||
|
||||
rows, err := r.db.conn.QueryContext(ctx, query, since.Unix())
|
||||
rows, err := r.db.conn.QueryContext(ctx, query,
|
||||
since.Unix(), since.Nanosecond())
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("querying files: %w", err)
|
||||
}
|
||||
@@ -239,7 +245,7 @@ func (r *FileRepository) ListUnderPath(
|
||||
|
||||
// LIKE would ignore ASCII case and treat _ and % in path as wildcards.
|
||||
query := `
|
||||
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
|
||||
SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
|
||||
FROM files
|
||||
WHERE path = ? OR substr(path, 1, length(?)) = ?
|
||||
ORDER BY path
|
||||
@@ -324,7 +330,7 @@ func (r *FileRepository) ListIDsWithChunksNotInUploadedBlobs(
|
||||
// ListAll returns all files in the database
|
||||
func (r *FileRepository) ListAll(ctx context.Context) ([]*File, error) {
|
||||
query := `
|
||||
SELECT id, path, source_path, mtime, size, mode, uid, gid, link_target
|
||||
SELECT id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target
|
||||
FROM files
|
||||
ORDER BY path
|
||||
`
|
||||
@@ -365,7 +371,7 @@ func (r *FileRepository) CreateBatch(
|
||||
}
|
||||
|
||||
// Each files row binds this many SQL variables.
|
||||
const fileCols = 9
|
||||
const fileCols = 10
|
||||
|
||||
// Batch at 100 rows to be safe with SQLite's variable limit.
|
||||
const batchSize = 100
|
||||
@@ -376,7 +382,7 @@ func (r *FileRepository) CreateBatch(
|
||||
batch := files[i:end]
|
||||
|
||||
query := `INSERT INTO files
|
||||
(id, path, source_path, mtime, size, mode, uid, gid, link_target)
|
||||
(id, path, source_path, mtime, mtime_nsec, size, mode, uid, gid, link_target)
|
||||
VALUES `
|
||||
|
||||
args := make([]any, 0, len(batch)*fileCols)
|
||||
@@ -388,11 +394,12 @@ func (r *FileRepository) CreateBatch(
|
||||
querySb325.WriteString(", ")
|
||||
}
|
||||
|
||||
querySb325.WriteString("(?, ?, ?, ?, ?, ?, ?, ?, ?)")
|
||||
querySb325.WriteString("(?, ?, ?, ?, ?, ?, ?, ?, ?, ?)")
|
||||
|
||||
args = append(args,
|
||||
f.ID.String(), f.Path.String(), f.SourcePath.String(),
|
||||
f.MTime.Unix(), f.Size, f.Mode, f.UID, f.GID,
|
||||
f.MTime.Unix(), f.MTime.Nanosecond(),
|
||||
f.Size, f.Mode, f.UID, f.GID,
|
||||
f.LinkTarget.String())
|
||||
}
|
||||
|
||||
@@ -401,6 +408,7 @@ func (r *FileRepository) CreateBatch(
|
||||
query += ` ON CONFLICT(path) DO UPDATE SET
|
||||
source_path = excluded.source_path,
|
||||
mtime = excluded.mtime,
|
||||
mtime_nsec = excluded.mtime_nsec,
|
||||
size = excluded.size,
|
||||
mode = excluded.mode,
|
||||
uid = excluded.uid,
|
||||
@@ -460,7 +468,7 @@ func (r *FileRepository) scanFileFrom(row fileRowScanner) (*File, error) {
|
||||
var (
|
||||
file File
|
||||
idStr, pathStr, sourcePathStr string
|
||||
mtimeUnix int64
|
||||
mtimeUnix, mtimeNsec int64
|
||||
linkTarget sql.NullString
|
||||
)
|
||||
|
||||
@@ -469,6 +477,7 @@ func (r *FileRepository) scanFileFrom(row fileRowScanner) (*File, error) {
|
||||
&pathStr,
|
||||
&sourcePathStr,
|
||||
&mtimeUnix,
|
||||
&mtimeNsec,
|
||||
&file.Size,
|
||||
&file.Mode,
|
||||
&file.UID,
|
||||
@@ -487,7 +496,7 @@ func (r *FileRepository) scanFileFrom(row fileRowScanner) (*File, error) {
|
||||
file.Path = types.FilePath(pathStr)
|
||||
file.SourcePath = types.SourcePath(sourcePathStr)
|
||||
|
||||
file.MTime = time.Unix(mtimeUnix, 0).UTC()
|
||||
file.MTime = time.Unix(mtimeUnix, mtimeNsec).UTC()
|
||||
if linkTarget.Valid {
|
||||
file.LinkTarget = types.FilePath(linkTarget.String)
|
||||
}
|
||||
|
||||
@@ -252,6 +252,115 @@ func TestFileRepositorySymlink(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// An mtime after 2262 or before 1678 does not fit in int64 nanoseconds
|
||||
// since the epoch, and must still come back from the database unchanged.
|
||||
func TestFileRepositoryMTimeOutsideInt64NanosecondRange(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
db, cleanup := setupTestDB(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
repo := database.NewFileRepository(db)
|
||||
|
||||
mtimes := []time.Time{
|
||||
time.Date(2300, time.January, 1, 0, 0, 0, 123456789, time.UTC),
|
||||
time.Date(1601, time.January, 1, 0, 0, 0, 987654321, time.UTC),
|
||||
}
|
||||
|
||||
for _, mtime := range mtimes {
|
||||
created := &database.File{
|
||||
Path: types.FilePath("/created-" + mtime.Format(time.RFC3339Nano)),
|
||||
MTime: mtime,
|
||||
}
|
||||
|
||||
err := repo.Create(ctx, nil, created)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to create file: %v", err)
|
||||
}
|
||||
|
||||
batched := &database.File{
|
||||
ID: types.NewFileID(),
|
||||
Path: types.FilePath("/batched-" + mtime.Format(time.RFC3339Nano)),
|
||||
MTime: mtime,
|
||||
}
|
||||
|
||||
err = repo.CreateBatch(ctx, nil, []*database.File{batched})
|
||||
if err != nil {
|
||||
t.Fatalf("failed to batch create file: %v", err)
|
||||
}
|
||||
|
||||
for _, path := range []types.FilePath{created.Path, batched.Path} {
|
||||
retrieved, err := repo.GetByPath(ctx, path.String())
|
||||
if err != nil {
|
||||
t.Fatalf("failed to get file: %v", err)
|
||||
}
|
||||
|
||||
if !retrieved.MTime.Equal(mtime) {
|
||||
t.Errorf("%s: mtime got %v, want %v",
|
||||
path, retrieved.MTime, mtime)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A file already in the index and rewritten within the same second must get
|
||||
// its new nanoseconds stored, through both Create and CreateBatch.
|
||||
func TestFileRepositoryUpsertMTimeInSameSecond(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
db, cleanup := setupTestDB(t)
|
||||
defer cleanup()
|
||||
|
||||
ctx := context.Background()
|
||||
repo := database.NewFileRepository(db)
|
||||
|
||||
indexed := time.Date(2026, time.October, 7, 12, 0, 0, 100000000, time.UTC)
|
||||
rewritten := indexed.Add(800 * time.Millisecond)
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
upsert func(file *database.File) error
|
||||
}{
|
||||
{"Create", func(file *database.File) error {
|
||||
return repo.Create(ctx, nil, file)
|
||||
}},
|
||||
{"CreateBatch", func(file *database.File) error {
|
||||
return repo.CreateBatch(ctx, nil, []*database.File{file})
|
||||
}},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
file := &database.File{
|
||||
ID: types.NewFileID(),
|
||||
Path: types.FilePath("/" + tt.name),
|
||||
MTime: indexed,
|
||||
}
|
||||
|
||||
err := tt.upsert(file)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: failed to create file: %v", tt.name, err)
|
||||
}
|
||||
|
||||
file.MTime = rewritten
|
||||
|
||||
err = tt.upsert(file)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: failed to update file: %v", tt.name, err)
|
||||
}
|
||||
|
||||
retrieved, err := repo.GetByPath(ctx, file.Path.String())
|
||||
if err != nil {
|
||||
t.Fatalf("%s: failed to get file: %v", tt.name, err)
|
||||
}
|
||||
|
||||
if !retrieved.MTime.Equal(rewritten) {
|
||||
t.Errorf("%s: mtime got %v, want %v",
|
||||
tt.name, retrieved.MTime, rewritten)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestFileRepositoryTransaction(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -14,8 +14,7 @@ type File struct {
|
||||
ID types.FileID // UUID primary key
|
||||
Path types.FilePath // Absolute path of the file
|
||||
|
||||
// SourcePath is the source directory this file came from (used for
|
||||
// restore path stripping).
|
||||
// SourcePath is the source directory this file came from.
|
||||
SourcePath types.SourcePath
|
||||
MTime time.Time
|
||||
Size int64
|
||||
|
||||
@@ -3,6 +3,7 @@ package database
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
@@ -367,7 +368,7 @@ func verifyBlobNullUploadTS(
|
||||
}
|
||||
|
||||
// createLargeDatasetFiles creates fileCount files and adds every other
|
||||
// one to the snapshot.
|
||||
// one to the snapshot, in one transaction as a backup writes them.
|
||||
func createLargeDatasetFiles(
|
||||
t *testing.T,
|
||||
repos *Repositories,
|
||||
@@ -376,31 +377,38 @@ func createLargeDatasetFiles(
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
ctx := context.Background()
|
||||
start := time.Now()
|
||||
|
||||
for i := range fileCount {
|
||||
file := &File{
|
||||
Path: types.FilePath(fmt.Sprintf("/large/file%05d.txt", i)),
|
||||
MTime: time.Now(),
|
||||
Size: int64(i * 1024),
|
||||
Mode: 0644,
|
||||
UID: uint32(1000 + (i % 10)),
|
||||
GID: uint32(1000 + (i % 10)),
|
||||
}
|
||||
err := repos.WithTx(context.Background(),
|
||||
func(ctx context.Context, tx *sql.Tx) error {
|
||||
for i := range fileCount {
|
||||
file := &File{
|
||||
Path: types.FilePath(fmt.Sprintf("/large/file%05d.txt", i)),
|
||||
MTime: time.Now(),
|
||||
Size: int64(i * 1024),
|
||||
Mode: 0644,
|
||||
UID: uint32(1000 + (i % 10)),
|
||||
GID: uint32(1000 + (i % 10)),
|
||||
}
|
||||
|
||||
err := repos.Files.Create(ctx, nil, file)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to create file %d: %v", i, err)
|
||||
}
|
||||
err := repos.Files.Create(ctx, tx, file)
|
||||
if err != nil {
|
||||
return fmt.Errorf("creating file %d: %w", i, err)
|
||||
}
|
||||
|
||||
// Add half to snapshot
|
||||
if i%2 == 0 {
|
||||
err = repos.Snapshots.AddFileByID(ctx, nil, snapshotID, file.ID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
// Add half to snapshot
|
||||
if i%2 == 0 {
|
||||
err = repos.Snapshots.AddFileByID(ctx, tx, snapshotID, file.ID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
t.Logf("Created %d files in %v", fileCount, time.Since(start))
|
||||
@@ -606,8 +614,7 @@ func TestTimezoneHandling(t *testing.T) {
|
||||
t.Skip("timezone not available")
|
||||
}
|
||||
|
||||
// Use Truncate to remove sub-second precision since we store as Unix timestamps
|
||||
nyTime := time.Now().In(loc).Truncate(time.Second)
|
||||
nyTime := time.Now().In(loc)
|
||||
file := &File{
|
||||
Path: "/timezone-test.txt",
|
||||
MTime: nyTime,
|
||||
|
||||
@@ -5,8 +5,9 @@
|
||||
CREATE TABLE IF NOT EXISTS files (
|
||||
id TEXT PRIMARY KEY, -- UUID
|
||||
path TEXT NOT NULL UNIQUE,
|
||||
source_path TEXT NOT NULL DEFAULT '', -- The source directory this file came from (for restore path stripping)
|
||||
mtime INTEGER NOT NULL,
|
||||
source_path TEXT NOT NULL DEFAULT '', -- The source directory this file came from
|
||||
mtime INTEGER NOT NULL, -- whole seconds since the Unix epoch
|
||||
mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second, 0 to 999999999
|
||||
size INTEGER NOT NULL,
|
||||
mode INTEGER NOT NULL,
|
||||
uid INTEGER NOT NULL,
|
||||
|
||||
+67
-52
@@ -10,15 +10,18 @@ import (
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
|
||||
"golang.org/x/sys/unix"
|
||||
)
|
||||
|
||||
// ErrAlreadyRunning indicates another vaultik instance is running.
|
||||
var ErrAlreadyRunning = errors.New("another vaultik instance is already running")
|
||||
|
||||
// Lock represents an acquired PID lock.
|
||||
// Lock represents an acquired PID lock: an flock(2) on the PID file,
|
||||
// held while the file stays open. The kernel drops it when the process
|
||||
// exits, however it exits, so a crashed run never leaves the lock held.
|
||||
type Lock struct {
|
||||
path string
|
||||
file *os.File
|
||||
}
|
||||
|
||||
const (
|
||||
@@ -29,10 +32,9 @@ const (
|
||||
)
|
||||
|
||||
// Acquire attempts to acquire a PID lock in the specified directory.
|
||||
// If the lock file exists and the process is still running, it returns
|
||||
// ErrAlreadyRunning with details about the existing process.
|
||||
// On success, it writes the current PID to the lock file and returns
|
||||
// a Lock that must be released with Release().
|
||||
// If another process holds the lock, it returns ErrAlreadyRunning with
|
||||
// that process's PID. On success, it writes the current PID to the lock
|
||||
// file and returns a Lock that must be released with Release().
|
||||
func Acquire(lockDir string) (*Lock, error) {
|
||||
// Ensure lock directory exists
|
||||
err := os.MkdirAll(lockDir, lockDirPerm)
|
||||
@@ -42,56 +44,82 @@ func Acquire(lockDir string) (*Lock, error) {
|
||||
|
||||
lockPath := filepath.Join(lockDir, "vaultik.pid")
|
||||
|
||||
// Check for existing lock
|
||||
existingPID, err := readPIDFile(lockPath)
|
||||
if err == nil {
|
||||
// Lock file exists, check if process is running
|
||||
if isProcessRunning(existingPID) {
|
||||
return nil, fmt.Errorf("%w (PID %d)", ErrAlreadyRunning, existingPID)
|
||||
}
|
||||
// Process is not running, stale lock file - we can take over
|
||||
}
|
||||
|
||||
// Write our PID
|
||||
pid := os.Getpid()
|
||||
|
||||
err = os.WriteFile(lockPath, []byte(strconv.Itoa(pid)), pidFilePerm)
|
||||
// No O_TRUNC: the file may hold the PID of the process that has the
|
||||
// lock, which the error below reports.
|
||||
file, err := os.OpenFile( //nolint:gosec // G304: path is our own lock file
|
||||
lockPath, os.O_RDWR|os.O_CREATE, pidFilePerm)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("writing PID file: %w", err)
|
||||
return nil, fmt.Errorf("opening PID file: %w", err)
|
||||
}
|
||||
|
||||
return &Lock{path: lockPath}, nil
|
||||
err = unix.Flock(int(file.Fd()), unix.LOCK_EX|unix.LOCK_NB)
|
||||
if err != nil {
|
||||
_ = file.Close()
|
||||
|
||||
if errors.Is(err, unix.EWOULDBLOCK) {
|
||||
return nil, alreadyRunningError(lockPath)
|
||||
}
|
||||
|
||||
return nil, fmt.Errorf("locking PID file: %w", err)
|
||||
}
|
||||
|
||||
err = writePID(file)
|
||||
if err != nil {
|
||||
_ = file.Close()
|
||||
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &Lock{file: file}, nil
|
||||
}
|
||||
|
||||
// Release removes the PID lock file.
|
||||
// Release empties the PID file and closes it, which drops the lock.
|
||||
// It is safe to call Release multiple times.
|
||||
func (l *Lock) Release() error {
|
||||
if l == nil || l.path == "" {
|
||||
if l == nil || l.file == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Verify we still own the lock (our PID is in the file)
|
||||
existingPID, err := readPIDFile(l.path)
|
||||
file := l.file
|
||||
l.file = nil
|
||||
|
||||
// Do not remove the file here. A process that opened it a moment
|
||||
// earlier could then lock the removed file while another creates and
|
||||
// locks a new one, and both would run.
|
||||
truncateErr := file.Truncate(0)
|
||||
closeErr := file.Close()
|
||||
|
||||
return errors.Join(truncateErr, closeErr)
|
||||
}
|
||||
|
||||
// writePID replaces the contents of the locked PID file with the current
|
||||
// PID.
|
||||
func writePID(file *os.File) error {
|
||||
err := file.Truncate(0)
|
||||
if err != nil {
|
||||
// File already gone or unreadable - that's fine
|
||||
return nil //nolint:nilerr // unreadable lock file means nothing to release
|
||||
return fmt.Errorf("truncating PID file: %w", err)
|
||||
}
|
||||
|
||||
if existingPID != os.Getpid() {
|
||||
// Someone else wrote to our lock file - don't remove it
|
||||
return nil
|
||||
_, err = file.WriteAt([]byte(strconv.Itoa(os.Getpid())), 0)
|
||||
if err != nil {
|
||||
return fmt.Errorf("writing PID file: %w", err)
|
||||
}
|
||||
|
||||
err = os.Remove(l.path)
|
||||
if err != nil && !os.IsNotExist(err) {
|
||||
return fmt.Errorf("removing PID file: %w", err)
|
||||
}
|
||||
|
||||
l.path = "" // Prevent double-release
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// alreadyRunningError reports that another process holds the lock,
|
||||
// naming its PID when the file holds one. The holder writes its PID just
|
||||
// after it locks, so the file can briefly be empty.
|
||||
func alreadyRunningError(lockPath string) error {
|
||||
pid, err := readPIDFile(lockPath)
|
||||
if err != nil {
|
||||
return ErrAlreadyRunning
|
||||
}
|
||||
|
||||
return fmt.Errorf("%w (PID %d)", ErrAlreadyRunning, pid)
|
||||
}
|
||||
|
||||
// readPIDFile reads and parses the PID from a lock file.
|
||||
func readPIDFile(path string) (int, error) {
|
||||
data, err := os.ReadFile(path) //nolint:gosec // G304: path is our own lock file
|
||||
@@ -106,16 +134,3 @@ func readPIDFile(path string) (int, error) {
|
||||
|
||||
return pid, nil
|
||||
}
|
||||
|
||||
// isProcessRunning checks if a process with the given PID is running.
|
||||
func isProcessRunning(pid int) bool {
|
||||
process, err := os.FindProcess(pid)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
|
||||
// On Unix, FindProcess always succeeds. We need to send signal 0 to check.
|
||||
err = process.Signal(syscall.Signal(0))
|
||||
|
||||
return err == nil
|
||||
}
|
||||
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
@@ -33,9 +34,10 @@ func TestAcquireAndRelease(t *testing.T) {
|
||||
err = lock.Release()
|
||||
require.NoError(t, err)
|
||||
|
||||
// Verify PID file is gone
|
||||
_, err = os.Stat(pidPath)
|
||||
assert.True(t, os.IsNotExist(err))
|
||||
// Verify PID file is empty
|
||||
data, err = os.ReadFile(pidPath) //nolint:gosec // G304: test's own temp file
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, data)
|
||||
}
|
||||
|
||||
func TestAcquireBlocksSecondInstance(t *testing.T) {
|
||||
@@ -55,6 +57,64 @@ func TestAcquireBlocksSecondInstance(t *testing.T) {
|
||||
lock2, err := pidlock.Acquire(tmpDir)
|
||||
require.ErrorIs(t, err, pidlock.ErrAlreadyRunning)
|
||||
assert.Nil(t, lock2)
|
||||
|
||||
// Once the first lock is released, the next Acquire succeeds
|
||||
require.NoError(t, lock1.Release())
|
||||
|
||||
lock3, err := pidlock.Acquire(tmpDir)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, lock3.Release())
|
||||
}
|
||||
|
||||
// TestConcurrentAcquireAdmitsOne starts many Acquire calls at the same
|
||||
// moment, as two cron entries firing together would, and checks that
|
||||
// exactly one of them gets the lock.
|
||||
func TestConcurrentAcquireAdmitsOne(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const callers = 50
|
||||
|
||||
tmpDir := t.TempDir()
|
||||
start := make(chan struct{})
|
||||
|
||||
var (
|
||||
mu sync.Mutex
|
||||
acquired []*pidlock.Lock
|
||||
failures []error
|
||||
wg sync.WaitGroup
|
||||
)
|
||||
|
||||
for range callers {
|
||||
wg.Go(func() {
|
||||
<-start
|
||||
|
||||
lock, err := pidlock.Acquire(tmpDir)
|
||||
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
|
||||
if err != nil {
|
||||
failures = append(failures, err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
acquired = append(acquired, lock)
|
||||
})
|
||||
}
|
||||
|
||||
close(start)
|
||||
wg.Wait()
|
||||
|
||||
for _, lock := range acquired {
|
||||
require.NoError(t, lock.Release())
|
||||
}
|
||||
|
||||
assert.Len(t, acquired, 1, "exactly one caller should hold the lock")
|
||||
|
||||
for _, err := range failures {
|
||||
require.ErrorIs(t, err, pidlock.ErrAlreadyRunning)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAcquireWithStaleLock(t *testing.T) {
|
||||
|
||||
+23
-5
@@ -27,10 +27,12 @@ type Client struct {
|
||||
bucket string
|
||||
prefix string
|
||||
endpoint string
|
||||
partSize int64
|
||||
}
|
||||
|
||||
// Config contains S3 client configuration.
|
||||
// All fields are required except Prefix, which defaults to an empty string.
|
||||
// All fields are required except Prefix, which defaults to an empty string,
|
||||
// and PartSize, where zero means the SDK default of 5 MiB.
|
||||
// A non-empty Prefix is joined to every key with one "/", whether or not
|
||||
// it ends with one.
|
||||
// The Endpoint field should include the protocol (http:// or https://).
|
||||
@@ -41,6 +43,9 @@ type Config struct {
|
||||
AccessKeyID string
|
||||
SecretAccessKey string
|
||||
Region string
|
||||
// PartSize is the size in bytes of each part of a multipart upload.
|
||||
// An upload too large for S3's limit of 10,000 parts gets larger parts.
|
||||
PartSize int64
|
||||
}
|
||||
|
||||
// nopLogger is a logger that discards all output.
|
||||
@@ -90,6 +95,7 @@ func NewClient(ctx context.Context, cfg Config) (*Client, error) {
|
||||
bucket: cfg.Bucket,
|
||||
prefix: prefix,
|
||||
endpoint: cfg.Endpoint,
|
||||
partSize: cfg.PartSize,
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -123,12 +129,9 @@ func (c *Client) PutObjectWithProgress(
|
||||
) error {
|
||||
fullKey := c.prefix + key
|
||||
|
||||
// uploadPartSize is 10MB for better progress granularity.
|
||||
const uploadPartSize = 10 * 1024 * 1024
|
||||
|
||||
// Create an uploader with the S3 client
|
||||
uploader := manager.NewUploader(c.s3Client, func(u *manager.Uploader) {
|
||||
u.PartSize = uploadPartSize
|
||||
u.PartSize = uploadPartSize(c.partSize, size)
|
||||
})
|
||||
|
||||
// Create a progress reader that tracks upload progress
|
||||
@@ -149,6 +152,21 @@ func (c *Client) PutObjectWithProgress(
|
||||
return err
|
||||
}
|
||||
|
||||
// uploadPartSize returns the part size for an upload of size bytes: the
|
||||
// configured part size (the SDK default when zero), raised where needed so
|
||||
// the upload fits in S3's limit of 10,000 parts. The uploader cannot raise
|
||||
// it itself, because it cannot seek the progress reader to learn its size.
|
||||
func uploadPartSize(configured, size int64) int64 {
|
||||
if configured == 0 {
|
||||
configured = manager.DefaultUploadPartSize
|
||||
}
|
||||
|
||||
maxParts := int64(manager.MaxUploadParts)
|
||||
smallestThatFits := (size + maxParts - 1) / maxParts // rounded up
|
||||
|
||||
return max(configured, smallestThatFits)
|
||||
}
|
||||
|
||||
// GetObject downloads an object from S3 with the specified key.
|
||||
// The key is automatically prefixed with the configured prefix.
|
||||
// Returns a ReadCloser containing the object data. The caller must
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
package s3
|
||||
|
||||
import "testing"
|
||||
|
||||
// TestUploadPartSize checks that an upload too large for 10,000 parts of the
|
||||
// configured size gets parts just large enough to fit in 10,000.
|
||||
func TestUploadPartSize(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const mib = 1024 * 1024
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
configured int64
|
||||
size int64
|
||||
want int64
|
||||
}{
|
||||
{
|
||||
name: "an upload that fits keeps the configured size",
|
||||
configured: 5 * mib,
|
||||
size: 10 * 1024 * mib,
|
||||
want: 5 * mib,
|
||||
},
|
||||
{
|
||||
name: "exactly 10,000 parts keeps the configured size",
|
||||
configured: 6 * mib,
|
||||
size: 10_000 * 6 * mib,
|
||||
want: 6 * mib,
|
||||
},
|
||||
{
|
||||
name: "one byte more than 10,000 parts adds a byte to each",
|
||||
configured: 6 * mib,
|
||||
size: 10_000*6*mib + 1,
|
||||
want: 6*mib + 1,
|
||||
},
|
||||
{
|
||||
name: "zero means the SDK default of 5MiB",
|
||||
configured: 0,
|
||||
size: 1,
|
||||
want: 5 * mib,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
got := uploadPartSize(tt.configured, tt.size)
|
||||
if got != tt.want {
|
||||
t.Errorf("uploadPartSize(%d, %d) = %d, want %d",
|
||||
tt.configured, tt.size, got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -28,6 +28,7 @@ func provideClient(lc fx.Lifecycle, cfg *config.Config) (*Client, error) {
|
||||
AccessKeyID: cfg.S3.AccessKeyID,
|
||||
SecretAccessKey: cfg.S3.SecretAccessKey,
|
||||
Region: cfg.S3.Region,
|
||||
PartSize: cfg.S3.PartSize.Int64(),
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
|
||||
@@ -58,8 +58,8 @@ type Scanner struct {
|
||||
compressionLevel int
|
||||
ageRecipient string
|
||||
snapshotID string // Current snapshot being processed
|
||||
// currentSourcePath is the source directory being scanned (used for
|
||||
// restore path stripping).
|
||||
// currentSourcePath is the source directory being scanned, stored with
|
||||
// each file record.
|
||||
currentSourcePath string
|
||||
exclude []string // Glob patterns for files/directories to exclude
|
||||
compiledExclude []compiledPattern // Compiled glob patterns
|
||||
@@ -211,7 +211,6 @@ func (s *Scanner) Scan(
|
||||
ctx context.Context, path string, snapshotID string,
|
||||
) (*ScanResult, error) {
|
||||
s.snapshotID = snapshotID
|
||||
// Store source path for file records (used during restore)
|
||||
s.currentSourcePath = path
|
||||
result := &ScanResult{
|
||||
StartTime: time.Now().UTC(),
|
||||
@@ -890,9 +889,10 @@ func (s *Scanner) scanPhase(
|
||||
}
|
||||
|
||||
// Handle symlinks and directories
|
||||
if handled := s.recordSpecialEntry(
|
||||
filePath, info, existingFiles, collector, result); handled {
|
||||
return nil
|
||||
handled, err := s.recordSpecialEntry(
|
||||
filePath, info, existingFiles, collector, result)
|
||||
if handled {
|
||||
return err
|
||||
}
|
||||
|
||||
// Skip other non-regular files (devices, sockets, etc.)
|
||||
@@ -934,22 +934,25 @@ func (s *Scanner) scanPhase(
|
||||
}
|
||||
|
||||
// recordSpecialEntry records symlinks and directories (which have no
|
||||
// data to chunk) and reports whether it handled the entry.
|
||||
// data to chunk) and reports whether it handled the entry. For a symlink
|
||||
// whose target cannot be read it returns handleWalkError's result.
|
||||
func (s *Scanner) recordSpecialEntry(
|
||||
filePath string, info os.FileInfo,
|
||||
existingFiles map[string]struct{},
|
||||
collector *scanCollector, result *ScanResult,
|
||||
) bool {
|
||||
) (bool, error) {
|
||||
// Handle symlinks
|
||||
if info.Mode()&os.ModeSymlink != 0 {
|
||||
file := s.buildSymlinkEntry(filePath, info)
|
||||
if file != nil {
|
||||
existingFiles[filePath] = struct{}{}
|
||||
collector.addToProcess(filePath, info, file)
|
||||
s.updateScanEntryStats(result, true, info)
|
||||
file, err := s.buildSymlinkEntry(filePath, info)
|
||||
if err != nil {
|
||||
return true, s.handleWalkError(filePath, err)
|
||||
}
|
||||
|
||||
return true
|
||||
existingFiles[filePath] = struct{}{}
|
||||
collector.addToProcess(filePath, info, file)
|
||||
s.updateScanEntryStats(result, true, info)
|
||||
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// Handle directories (record for permission/ownership preservation
|
||||
@@ -959,10 +962,10 @@ func (s *Scanner) recordSpecialEntry(
|
||||
existingFiles[filePath] = struct{}{}
|
||||
collector.addToProcess(filePath, info, file)
|
||||
|
||||
return true
|
||||
return true, nil
|
||||
}
|
||||
|
||||
return false
|
||||
return false, nil
|
||||
}
|
||||
|
||||
// handleWalkError deals with a filesystem error surfaced by the walk:
|
||||
@@ -1112,13 +1115,12 @@ func (s *Scanner) printScanProgressLine(
|
||||
}
|
||||
|
||||
// buildSymlinkEntry creates a File record for a symlink.
|
||||
// Returns nil if the link target cannot be read.
|
||||
func (s *Scanner) buildSymlinkEntry(path string, info os.FileInfo) *database.File {
|
||||
func (s *Scanner) buildSymlinkEntry(
|
||||
path string, info os.FileInfo,
|
||||
) (*database.File, error) {
|
||||
target, err := os.Readlink(path)
|
||||
if err != nil {
|
||||
log.Debug("Cannot read symlink target", "path", path, "error", err)
|
||||
|
||||
return nil
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var uid, gid uint32
|
||||
@@ -1137,7 +1139,7 @@ func (s *Scanner) buildSymlinkEntry(path string, info os.FileInfo) *database.Fil
|
||||
UID: uid,
|
||||
GID: gid,
|
||||
LinkTarget: types.FilePath(target),
|
||||
}
|
||||
}, nil
|
||||
}
|
||||
|
||||
// buildDirectoryEntry creates a File record for a directory.
|
||||
@@ -1198,9 +1200,8 @@ func (s *Scanner) checkFileInMemory(
|
||||
}
|
||||
|
||||
file := &database.File{
|
||||
ID: fileID,
|
||||
Path: types.FilePath(path),
|
||||
// Store source directory for restore path stripping
|
||||
ID: fileID,
|
||||
Path: types.FilePath(path),
|
||||
SourcePath: types.SourcePath(s.currentSourcePath),
|
||||
MTime: info.ModTime(),
|
||||
Size: info.Size(),
|
||||
@@ -1221,7 +1222,7 @@ func (s *Scanner) checkFileInMemory(
|
||||
|
||||
// Check if file has changed
|
||||
if existingFile.Size != file.Size ||
|
||||
existingFile.MTime.Unix() != file.MTime.Unix() ||
|
||||
!existingFile.MTime.Equal(file.MTime) ||
|
||||
existingFile.Mode != file.Mode ||
|
||||
existingFile.UID != file.UID ||
|
||||
existingFile.GID != file.GID {
|
||||
|
||||
@@ -3,6 +3,7 @@ package snapshot_test
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
@@ -13,6 +14,7 @@ import (
|
||||
"github.com/spf13/afero"
|
||||
"sneak.berlin/go/vaultik/internal/database"
|
||||
"sneak.berlin/go/vaultik/internal/snapshot"
|
||||
"sneak.berlin/go/vaultik/internal/ui"
|
||||
)
|
||||
|
||||
// errSimTempFail is the one-time temp-file creation failure blobTempFailFs
|
||||
@@ -80,6 +82,30 @@ func (f *readFailFs) Open(name string) (afero.File, error) {
|
||||
return file, nil
|
||||
}
|
||||
|
||||
// linkRemovedAfterLstatFs is the real filesystem, except that the symlink at
|
||||
// target is removed right after the walk lstats it, as happens when a link is
|
||||
// deleted during a backup. The scanner's readlink of it then fails.
|
||||
type linkRemovedAfterLstatFs struct {
|
||||
afero.OsFs
|
||||
|
||||
t *testing.T
|
||||
target string
|
||||
}
|
||||
|
||||
func (f *linkRemovedAfterLstatFs) LstatIfPossible(
|
||||
name string,
|
||||
) (os.FileInfo, bool, error) {
|
||||
info, lstatCalled, err := f.OsFs.LstatIfPossible(name)
|
||||
if err == nil && name == f.target {
|
||||
rmErr := os.Remove(name)
|
||||
if rmErr != nil {
|
||||
f.t.Errorf("removing %s: %v", name, rmErr)
|
||||
}
|
||||
}
|
||||
|
||||
return info, lstatCalled, err
|
||||
}
|
||||
|
||||
// writeSkipErrorTestFile writes one file into fs with a fixed mtime.
|
||||
func writeSkipErrorTestFile(t *testing.T, fs afero.Fs, path, content string) {
|
||||
t.Helper()
|
||||
@@ -102,10 +128,11 @@ func writeSkipErrorTestFile(t *testing.T, fs afero.Fs, path, content string) {
|
||||
}
|
||||
}
|
||||
|
||||
// runSkipErrorScan scans /source on fs with the given skip-errors setting and
|
||||
// returns the repositories (for inspection) and the scan error.
|
||||
// runSkipErrorScan scans source on fs with the given skip-errors setting,
|
||||
// printing user-facing messages to uiw (nil discards them), and returns the
|
||||
// repositories (for inspection) and the scan error.
|
||||
func runSkipErrorScan(
|
||||
t *testing.T, fs afero.Fs, skipErrors bool,
|
||||
t *testing.T, fs afero.Fs, source string, skipErrors bool, uiw *ui.Writer,
|
||||
) (*database.Repositories, error) {
|
||||
t.Helper()
|
||||
|
||||
@@ -130,6 +157,7 @@ func runSkipErrorScan(
|
||||
MaxBlobSize: int64(1024 * 1024),
|
||||
CompressionLevel: 3,
|
||||
AgeRecipients: []string{testAgePublicKey},
|
||||
UI: uiw,
|
||||
SkipErrors: skipErrors,
|
||||
})
|
||||
|
||||
@@ -137,7 +165,7 @@ func runSkipErrorScan(
|
||||
snapshotID := "test-snapshot-skip-errors"
|
||||
createTestSnapshotRecord(ctx, t, repos, snapshotID)
|
||||
|
||||
_, err = scanner.Scan(ctx, "/source", snapshotID)
|
||||
_, err = scanner.Scan(ctx, source, snapshotID)
|
||||
|
||||
return repos, err
|
||||
}
|
||||
@@ -157,7 +185,7 @@ func TestScannerPackingFailureAbortsUnderSkipErrors(t *testing.T) {
|
||||
writeSkipErrorTestFile(t, fs, "/source/file1.txt", "first file content")
|
||||
writeSkipErrorTestFile(t, fs, "/source/file2.txt", "second file content")
|
||||
|
||||
repos, err := runSkipErrorScan(t, fs, true)
|
||||
repos, err := runSkipErrorScan(t, fs, "/source", true, nil)
|
||||
if err == nil {
|
||||
t.Fatal("expected scan to abort on the packer error, got nil")
|
||||
}
|
||||
@@ -184,7 +212,7 @@ func TestScannerReadErrorAbortsWithoutSkipErrors(t *testing.T) {
|
||||
fs := &readFailFs{Fs: afero.NewMemMapFs(), target: target}
|
||||
writeSkipErrorTestFile(t, fs, target, "content that cannot be read")
|
||||
|
||||
_, err := runSkipErrorScan(t, fs, false)
|
||||
_, err := runSkipErrorScan(t, fs, "/source", false, nil)
|
||||
if err == nil {
|
||||
t.Fatal("expected scan to fail on the read error, got nil")
|
||||
}
|
||||
@@ -200,7 +228,7 @@ func TestScannerReadErrorSkippedWithSkipErrors(t *testing.T) {
|
||||
fs := &readFailFs{Fs: afero.NewMemMapFs(), target: target}
|
||||
writeSkipErrorTestFile(t, fs, target, "content that cannot be read")
|
||||
|
||||
repos, err := runSkipErrorScan(t, fs, true)
|
||||
repos, err := runSkipErrorScan(t, fs, "/source", true, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("expected scan to complete with --skip-errors, got %v", err)
|
||||
}
|
||||
@@ -214,3 +242,63 @@ func TestScannerReadErrorSkippedWithSkipErrors(t *testing.T) {
|
||||
t.Fatalf("expected unreadable file skipped, got %d chunks", len(chunks))
|
||||
}
|
||||
}
|
||||
|
||||
// writeSymlinkSource creates a source directory on disk holding one symlink
|
||||
// and returns the directory and the symlink's path.
|
||||
func writeSymlinkSource(t *testing.T) (string, string) {
|
||||
t.Helper()
|
||||
|
||||
sourceDir := t.TempDir()
|
||||
linkPath := filepath.Join(sourceDir, "link")
|
||||
|
||||
err := os.Symlink("target.txt", linkPath)
|
||||
if err != nil {
|
||||
t.Fatalf("creating symlink: %v", err)
|
||||
}
|
||||
|
||||
return sourceDir, linkPath
|
||||
}
|
||||
|
||||
// TestScannerUnreadableSymlinkAbortsWithoutSkipErrors checks that a symlink
|
||||
// whose target cannot be read aborts the run when --skip-errors is not set.
|
||||
func TestScannerUnreadableSymlinkAbortsWithoutSkipErrors(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
sourceDir, linkPath := writeSymlinkSource(t)
|
||||
fs := &linkRemovedAfterLstatFs{t: t, target: linkPath}
|
||||
|
||||
_, err := runSkipErrorScan(t, fs, sourceDir, false, nil)
|
||||
if !errors.Is(err, os.ErrNotExist) {
|
||||
t.Fatalf("expected scan to fail on the removed symlink, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestScannerUnreadableSymlinkSkippedWithSkipErrors checks that a symlink
|
||||
// whose target cannot be read is skipped with an error line, and the run
|
||||
// completes, when --skip-errors is set.
|
||||
func TestScannerUnreadableSymlinkSkippedWithSkipErrors(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
sourceDir, linkPath := writeSymlinkSource(t)
|
||||
fs := &linkRemovedAfterLstatFs{t: t, target: linkPath}
|
||||
uiw := ui.NewWithColor(io.Discard, false)
|
||||
|
||||
repos, err := runSkipErrorScan(t, fs, sourceDir, true, uiw)
|
||||
if err != nil {
|
||||
t.Fatalf("expected scan to complete with --skip-errors, got %v", err)
|
||||
}
|
||||
|
||||
if uiw.ErrorCount() != 1 {
|
||||
t.Fatalf("expected one error line for the symlink, got %d",
|
||||
uiw.ErrorCount())
|
||||
}
|
||||
|
||||
file, err := repos.Files.GetByPath(context.Background(), linkPath)
|
||||
if err != nil {
|
||||
t.Fatalf("getting %s: %v", linkPath, err)
|
||||
}
|
||||
|
||||
if file != nil {
|
||||
t.Fatalf("expected %s not to be recorded", linkPath)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -105,17 +105,21 @@ func (sm *SnapshotManager) CreateSnapshot(
|
||||
return sm.CreateSnapshotWithName(ctx, hostname, "", version, gitRevision)
|
||||
}
|
||||
|
||||
// ShortHostname returns hostname up to its first dot. A snapshot ID starts
|
||||
// with this form, while the snapshots table stores the full hostname.
|
||||
func ShortHostname(hostname string) string {
|
||||
short, _, _ := strings.Cut(hostname, ".")
|
||||
|
||||
return short
|
||||
}
|
||||
|
||||
// CreateSnapshotWithName creates a new snapshot record with an optional
|
||||
// snapshot name. The snapshot ID format is: hostname_name_timestamp or
|
||||
// hostname_timestamp if name is empty.
|
||||
func (sm *SnapshotManager) CreateSnapshotWithName(
|
||||
ctx context.Context, hostname, name, version, gitRevision string,
|
||||
) (string, error) {
|
||||
// Use short hostname (strip domain if present)
|
||||
shortHostname := hostname
|
||||
if before, _, ok := strings.Cut(hostname, "."); ok {
|
||||
shortHostname = before
|
||||
}
|
||||
shortHostname := ShortHostname(hostname)
|
||||
|
||||
// Build snapshot ID with optional name
|
||||
timestamp := time.Now().UTC().Format("2006-01-02T15:04:05Z")
|
||||
|
||||
@@ -99,6 +99,7 @@ func storerFromParsedS3URL(parsed *URL, cfg *config.Config) (Storer, error) {
|
||||
AccessKeyID: cfg.S3.AccessKeyID,
|
||||
SecretAccessKey: cfg.S3.SecretAccessKey,
|
||||
Region: region,
|
||||
PartSize: cfg.S3.PartSize.Int64(),
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("creating S3 client: %w", err)
|
||||
@@ -134,6 +135,7 @@ func storerFromLegacyS3Config(cfg *config.Config) (Storer, error) {
|
||||
AccessKeyID: cfg.S3.AccessKeyID,
|
||||
SecretAccessKey: cfg.S3.SecretAccessKey,
|
||||
Region: region,
|
||||
PartSize: cfg.S3.PartSize.Int64(),
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("creating S3 client: %w", err)
|
||||
|
||||
@@ -1,11 +1,14 @@
|
||||
package storage_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
|
||||
"github.com/johannesboyne/gofakes3"
|
||||
@@ -19,6 +22,13 @@ import (
|
||||
// s3TestBucket is the bucket created for each in-process S3 server.
|
||||
const s3TestBucket = "test-bucket"
|
||||
|
||||
// Credentials for the tests that build a storer from a config.Config. The
|
||||
// in-process S3 server accepts any.
|
||||
const (
|
||||
s3TestAccessKeyID = "key"
|
||||
s3TestSecretAccessKey = "secret"
|
||||
)
|
||||
|
||||
// newS3Storer builds an s3:// backend backed by a fresh in-process
|
||||
// S3 server. It reuses the same in-memory S3 harness (gofakes3 + s3mem
|
||||
// over httptest) that internal/s3 and the not-found regression test use,
|
||||
@@ -128,8 +138,8 @@ func TestS3URLPrefixKeyLayout(t *testing.T) {
|
||||
storer, err := storage.NewStorer(&config.Config{
|
||||
StorageURL: storageURL + "?endpoint=" + srv.URL,
|
||||
S3: config.S3Config{
|
||||
AccessKeyID: "key",
|
||||
SecretAccessKey: "secret",
|
||||
AccessKeyID: s3TestAccessKeyID,
|
||||
SecretAccessKey: s3TestSecretAccessKey,
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
@@ -171,6 +181,90 @@ func TestS3URLPrefixKeyLayout(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestS3UploadUsesConfiguredPartSize checks that s3.part_size reaches the
|
||||
// multipart uploader, through storage_url and through the s3.* fields. An
|
||||
// object three parts long must arrive as three parts; at the SDK's default
|
||||
// of 5 MiB it would arrive as four.
|
||||
func TestS3UploadUsesConfiguredPartSize(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
partSize = 6 * 1024 * 1024
|
||||
wantParts = 3
|
||||
)
|
||||
|
||||
backend := s3mem.New()
|
||||
|
||||
err := backend.CreateBucket(s3TestBucket)
|
||||
if err != nil {
|
||||
t.Fatalf("create bucket: %v", err)
|
||||
}
|
||||
|
||||
// Every part of a multipart upload is one request with a partNumber.
|
||||
var parts atomic.Int32
|
||||
|
||||
fake := gofakes3.New(backend).Server()
|
||||
srv := httptest.NewServer(http.HandlerFunc(
|
||||
func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Query().Has("partNumber") {
|
||||
parts.Add(1)
|
||||
}
|
||||
|
||||
fake.ServeHTTP(w, r)
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
cfg *config.Config
|
||||
}{
|
||||
{
|
||||
name: "storage_url",
|
||||
cfg: &config.Config{
|
||||
StorageURL: "s3://" + s3TestBucket + "?endpoint=" + srv.URL,
|
||||
S3: config.S3Config{
|
||||
AccessKeyID: s3TestAccessKeyID,
|
||||
SecretAccessKey: s3TestSecretAccessKey,
|
||||
PartSize: partSize,
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "s3.endpoint",
|
||||
cfg: &config.Config{
|
||||
S3: config.S3Config{
|
||||
Endpoint: srv.URL,
|
||||
Bucket: s3TestBucket,
|
||||
AccessKeyID: s3TestAccessKeyID,
|
||||
SecretAccessKey: s3TestSecretAccessKey,
|
||||
PartSize: partSize,
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
parts.Store(0)
|
||||
|
||||
storer, err := storage.NewStorer(tc.cfg)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: NewStorer: %v", tc.name, err)
|
||||
}
|
||||
|
||||
data := bytes.NewReader(make([]byte, wantParts*partSize))
|
||||
|
||||
err = storer.PutWithProgress(
|
||||
context.Background(), "blob", data, data.Size(), nil)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: PutWithProgress: %v", tc.name, err)
|
||||
}
|
||||
|
||||
if got := parts.Load(); got != wantParts {
|
||||
t.Errorf("%s: uploaded in %d parts, want %d", tc.name, got, wantParts)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// listStreamKeys returns the keys ListStream yields under a prefix, and
|
||||
// fails the test on a listing error.
|
||||
func listStreamKeys(t *testing.T, s storage.Storer, prefix string) []string {
|
||||
|
||||
@@ -155,8 +155,8 @@ type BlobHash string
|
||||
// FilePath represents an absolute path to a file or directory.
|
||||
type FilePath string
|
||||
|
||||
// SourcePath represents the root directory from which files are backed up.
|
||||
// Used during restore to strip the source prefix from paths.
|
||||
// SourcePath is the source directory a scan found a file under, made
|
||||
// absolute and with symlinks resolved.
|
||||
type SourcePath string
|
||||
|
||||
// Hostname identifies a host machine.
|
||||
|
||||
@@ -130,10 +130,9 @@ func assertThirdSnapshotRestores(
|
||||
// up, and that snapshot is removed. The first snapshot keeps the file row,
|
||||
// which now lists the appended content's chunks, while removal drops the
|
||||
// blob that held them.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestBackupAfterRemovingNewestSnapshotRestoresChangedFile(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
@@ -178,10 +177,9 @@ func TestBackupAfterRemovingNewestSnapshotRestoresChangedFile(t *testing.T) {
|
||||
// The next run's prune drops that incomplete snapshot and its blob, while
|
||||
// the first snapshot keeps the file row, which now lists the appended
|
||||
// content's chunks.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestBackupAfterInterruptedRunRestoresChangedFile(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
|
||||
@@ -38,11 +38,9 @@ import (
|
||||
// (https://git.eeqj.de/sneak/vaultik/issues/130) and is not re-tested
|
||||
// here; these tests target the layers above the backend.
|
||||
//
|
||||
// The tests run serially, not with t.Parallel: each calls
|
||||
// log.Initialize, which replaces the package-global logger, and a
|
||||
// backup or restore running concurrently reads that same logger. Under
|
||||
// -race the two collide. Running one at a time is the same choice
|
||||
// prune_count_test.go already makes for the same reason.
|
||||
// log.Initialize replaces the package-global logger that a running
|
||||
// backup or restore reads, so each test calls it before t.Parallel,
|
||||
// while no parallel test is running yet.
|
||||
|
||||
const (
|
||||
faultChunkSize = int64(64 * 1024)
|
||||
@@ -165,17 +163,19 @@ func newReaderVaultik(
|
||||
// Scenario 3: a stored blob's bytes are flipped before restore reads
|
||||
// them. Restore must fail loudly, and no file must be left on the
|
||||
// restore target holding corrupt content.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestRestoreRejectsCorruptBlob(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
assertRestoreRejectsDamagedBlob(t, faultstore.GetCorrupt, "corrupt")
|
||||
}
|
||||
|
||||
// Scenario 4: a stored blob is truncated before restore reads it. Same
|
||||
// contract as the corrupt case.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestRestoreRejectsTruncatedBlob(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
assertRestoreRejectsDamagedBlob(t, faultstore.GetTruncate, "truncated")
|
||||
}
|
||||
|
||||
@@ -188,7 +188,6 @@ func assertRestoreRejectsDamagedBlob(
|
||||
t *testing.T, fault faultstore.GetFault, name string,
|
||||
) {
|
||||
t.Helper()
|
||||
log.Initialize(log.Config{})
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
@@ -232,10 +231,9 @@ func assertRestoreRejectsDamagedBlob(
|
||||
|
||||
// Scenario 6: the backend accepts blob uploads and reports success but
|
||||
// stores nothing. verify --deep must catch it.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestDeepVerifyCatchesLyingBackend(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
@@ -285,10 +283,9 @@ func TestDeepVerifyCatchesLyingBackend(t *testing.T) {
|
||||
// Scenario 1a: a blob upload fails partway through. The interrupted run
|
||||
// must not record the blob as uploaded, must not reference it from the
|
||||
// snapshot, and must leave no blob object at the destination.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestInterruptedBlobUploadRecordsNoUploadedBlob(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
@@ -361,10 +358,9 @@ func TestInterruptedBlobUploadRecordsNoUploadedBlob(t *testing.T) {
|
||||
// chunks in a blob that was actually uploaded, so the retry re-chunks and
|
||||
// re-uploads the affected data instead of silently referencing data that
|
||||
// never reached storage.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestBackupRetryAfterInterruptedUploadIsRestorable(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
@@ -426,10 +422,9 @@ func TestBackupRetryAfterInterruptedUploadIsRestorable(t *testing.T) {
|
||||
// covered by TestBackupCompletesOnlyAfterMetadataExport
|
||||
// (https://git.eeqj.de/sneak/vaultik/issues/177); this test exercises the
|
||||
// lower-level export path in isolation.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestBackupSurvivesMetadataExportInterruption(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
@@ -509,10 +504,9 @@ func TestBackupSurvivesMetadataExportInterruption(t *testing.T) {
|
||||
// destination. Rerunning the backup must then prune the incomplete
|
||||
// snapshot, produce a snapshot whose destination metadata and local index
|
||||
// agree, and restore. See https://git.eeqj.de/sneak/vaultik/issues/177.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestBackupCompletesOnlyAfterMetadataExport(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
@@ -685,10 +679,9 @@ func faultScannerFactory(
|
||||
// Scenario 5: the restore target runs out of space mid-file. Restore
|
||||
// must fail with an out-of-space error, and must not leave a truncated
|
||||
// file at the target path presenting as a complete restore.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestRestoreReportsDiskFull(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
osFS := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
|
||||
+23
-18
@@ -9,6 +9,7 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/dustin/go-humanize"
|
||||
"sneak.berlin/go/vaultik/internal/snapshot"
|
||||
"sneak.berlin/go/vaultik/internal/types"
|
||||
)
|
||||
|
||||
@@ -46,12 +47,9 @@ const (
|
||||
year = 365 * day
|
||||
)
|
||||
|
||||
// Snapshot IDs split on "_" into hostname, optional name parts, and a
|
||||
// trailing timestamp.
|
||||
const (
|
||||
minSnapshotIDParts = 2
|
||||
minSnapshotIDNameParts = 3
|
||||
)
|
||||
// A snapshot ID split on "_" has at least a hostname and a trailing
|
||||
// timestamp.
|
||||
const minSnapshotIDParts = 2
|
||||
|
||||
// SnapshotInfo contains information about a snapshot.
|
||||
//
|
||||
@@ -121,20 +119,27 @@ func parseSnapshotTimestamp(snapshotID string) (time.Time, error) {
|
||||
return timestamp.UTC(), nil
|
||||
}
|
||||
|
||||
// parseSnapshotName extracts the snapshot name from a snapshot ID.
|
||||
// Format: hostname_snapshotname_timestamp — the middle part(s) between hostname
|
||||
// and the RFC3339 timestamp are the snapshot name (may contain underscores).
|
||||
// Returns the snapshot name, or empty string if the ID is malformed.
|
||||
func parseSnapshotName(snapshotID string) string {
|
||||
parts := strings.Split(snapshotID, "_")
|
||||
if len(parts) < minSnapshotIDNameParts {
|
||||
// Format: hostname_timestamp — no snapshot name
|
||||
// parseSnapshotName extracts the snapshot name from a snapshot ID of the
|
||||
// form hostname_name_timestamp, given the hostname stored with that
|
||||
// snapshot. The hostname and the name may both contain underscores, so the
|
||||
// name is what is left after removing the short hostname and its "_" from
|
||||
// the front and the last "_" and the timestamp from the end. Returns "" for
|
||||
// an ID with no name (hostname_timestamp), and for an ID that does not start
|
||||
// with that hostname, which CreateSnapshotWithName never writes.
|
||||
func parseSnapshotName(snapshotID, hostname string) string {
|
||||
prefix := snapshot.ShortHostname(hostname) + "_"
|
||||
|
||||
rest, ok := strings.CutPrefix(snapshotID, prefix)
|
||||
if !ok {
|
||||
return ""
|
||||
}
|
||||
// Format: hostname_name_timestamp — middle parts are the name.
|
||||
// The last part is the RFC3339 timestamp, the first part is the hostname,
|
||||
// everything in between is the snapshot name (which may itself contain underscores).
|
||||
return strings.Join(parts[1:len(parts)-1], "_")
|
||||
|
||||
end := strings.LastIndex(rest, "_")
|
||||
if end < 0 {
|
||||
return ""
|
||||
}
|
||||
|
||||
return rest[:end]
|
||||
}
|
||||
|
||||
// parseDuration parses a duration string with support for human-friendly units:
|
||||
|
||||
@@ -11,33 +11,55 @@ func TestParseSnapshotName(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
snapshotID string
|
||||
hostname string
|
||||
want string
|
||||
}{
|
||||
{
|
||||
name: "standard format with name",
|
||||
snapshotID: "myhost_home_2026-01-12T14:41:15Z",
|
||||
hostname: "myhost",
|
||||
want: "home",
|
||||
},
|
||||
{
|
||||
name: "standard format with different name",
|
||||
snapshotID: "server1_system_2026-02-15T09:30:00Z",
|
||||
hostname: "server1",
|
||||
want: "system",
|
||||
},
|
||||
{
|
||||
name: "name with underscores",
|
||||
snapshotID: "myhost_my_special_backup_2026-03-01T00:00:00Z",
|
||||
hostname: "myhost",
|
||||
want: "my_special_backup",
|
||||
},
|
||||
{
|
||||
name: "hostname with underscores",
|
||||
snapshotID: "my_host_docs_2026-03-01T00:00:00Z",
|
||||
hostname: "my_host",
|
||||
want: "docs",
|
||||
},
|
||||
{
|
||||
name: "stored hostname with domain",
|
||||
snapshotID: "my_host_mail_2026-03-01T00:00:00Z",
|
||||
hostname: "my_host.example.com",
|
||||
want: "mail",
|
||||
},
|
||||
{
|
||||
name: "no name",
|
||||
snapshotID: "my_host_2026-03-01T00:00:00Z",
|
||||
hostname: "my_host",
|
||||
want: "",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
got := parseSnapshotName(tt.snapshotID)
|
||||
got := parseSnapshotName(tt.snapshotID, tt.hostname)
|
||||
if got != tt.want {
|
||||
t.Errorf("parseSnapshotName(%q) = %q, want %q",
|
||||
tt.snapshotID, got, tt.want)
|
||||
t.Errorf("parseSnapshotName(%q, %q) = %q, want %q",
|
||||
tt.snapshotID, tt.hostname, got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
+109
-25
@@ -183,6 +183,11 @@ type SnapshotMetadataInfo struct {
|
||||
TotalSize int64 `json:"total_size"`
|
||||
BlobCount int `json:"blob_count"`
|
||||
BlobsSize int64 `json:"blobs_size"`
|
||||
|
||||
// Set when the listing holds this snapshot's manifest.json.zst. A
|
||||
// backup interrupted before its manifest upload leaves a directory
|
||||
// without one, which prune does not treat as a snapshot.
|
||||
hasManifest bool
|
||||
}
|
||||
|
||||
// RemoteInfoResult contains all remote storage information
|
||||
@@ -206,9 +211,20 @@ type RemoteInfoResult struct {
|
||||
ReferencedBlobCount int `json:"referenced_blob_count"`
|
||||
ReferencedBlobSize int64 `json:"referenced_blob_size"`
|
||||
|
||||
// Orphaned blobs
|
||||
OrphanedBlobCount int `json:"orphaned_blob_count"`
|
||||
OrphanedBlobSize int64 `json:"orphaned_blob_size"`
|
||||
// Orphaned blobs. Both stay nil (null in the JSON) when a manifest
|
||||
// was listed but not read, since that snapshot's blobs would be
|
||||
// counted as orphaned.
|
||||
OrphanedBlobCount *int `json:"orphaned_blob_count"`
|
||||
OrphanedBlobSize *int64 `json:"orphaned_blob_size"`
|
||||
|
||||
// Remote key of each snapshot whose manifest could not be read
|
||||
UnreadableManifests []string `json:"unreadable_manifests,omitempty"`
|
||||
|
||||
// Number of manifests not read because the name above them under
|
||||
// metadata/ is not a remote key. The names themselves are not
|
||||
// reported: they come from the destination store and may hold
|
||||
// control characters.
|
||||
SkippedManifestCount int `json:"skipped_manifest_count,omitempty"`
|
||||
}
|
||||
|
||||
// RemoteInfo displays information about remote storage
|
||||
@@ -234,16 +250,28 @@ func (v *Vaultik) RemoteInfo(jsonOutput bool) error {
|
||||
v.stdoutf("Scanning snapshot metadata...\n")
|
||||
}
|
||||
|
||||
snapshotMetadata, snapshotIDs, err := v.collectSnapshotMetadata()
|
||||
snapshotMetadata, snapshotIDs, skippedManifestCount, err := v.collectSnapshotMetadata()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
result.SkippedManifestCount = skippedManifestCount
|
||||
|
||||
if showText {
|
||||
v.stdoutf("Downloading %d manifest(s)...\n", len(snapshotIDs))
|
||||
manifestCount := 0
|
||||
|
||||
for _, info := range snapshotMetadata {
|
||||
if info.hasManifest {
|
||||
manifestCount++
|
||||
}
|
||||
}
|
||||
|
||||
v.stdoutf("Downloading %d manifest(s)...\n", manifestCount)
|
||||
}
|
||||
|
||||
referencedBlobs := v.collectReferencedBlobsFromManifests(snapshotIDs, snapshotMetadata)
|
||||
referencedBlobs, unreadableManifests := v.collectReferencedBlobsFromManifests(
|
||||
snapshotIDs, snapshotMetadata)
|
||||
result.UnreadableManifests = unreadableManifests
|
||||
|
||||
v.populateRemoteInfoResult(result, snapshotMetadata, snapshotIDs, referencedBlobs)
|
||||
|
||||
@@ -256,7 +284,7 @@ func (v *Vaultik) RemoteInfo(jsonOutput bool) error {
|
||||
"snapshots", result.TotalMetadataCount,
|
||||
"total_blobs", result.TotalBlobCount,
|
||||
"referenced_blobs", result.ReferencedBlobCount,
|
||||
"orphaned_blobs", result.OrphanedBlobCount)
|
||||
"unreadable_manifests", len(result.UnreadableManifests))
|
||||
|
||||
if jsonOutput {
|
||||
enc := json.NewEncoder(v.Stdout)
|
||||
@@ -273,16 +301,18 @@ func (v *Vaultik) RemoteInfo(jsonOutput bool) error {
|
||||
}
|
||||
|
||||
// collectSnapshotMetadata scans remote metadata and returns
|
||||
// per-snapshot info and sorted IDs.
|
||||
// per-snapshot info, sorted IDs and the number of manifests it skipped
|
||||
// because the name above them is not a remote key.
|
||||
func (v *Vaultik) collectSnapshotMetadata() (
|
||||
map[string]*SnapshotMetadataInfo, []string, error,
|
||||
map[string]*SnapshotMetadataInfo, []string, int, error,
|
||||
) {
|
||||
snapshotMetadata := make(map[string]*SnapshotMetadataInfo)
|
||||
skippedManifestCount := 0
|
||||
|
||||
metadataCh := v.Storage.ListStream(v.ctx, "metadata/")
|
||||
for obj := range metadataCh {
|
||||
if obj.Err != nil {
|
||||
return nil, nil, fmt.Errorf("listing metadata: %w", obj.Err)
|
||||
return nil, nil, 0, fmt.Errorf("listing metadata: %w", obj.Err)
|
||||
}
|
||||
|
||||
parts := strings.Split(obj.Key, "/")
|
||||
@@ -291,6 +321,22 @@ func (v *Vaultik) collectSnapshotMetadata() (
|
||||
}
|
||||
|
||||
snapshotID := parts[1]
|
||||
filename := parts[2]
|
||||
isManifest := filename == "manifest.json.zst"
|
||||
|
||||
// The name comes from the destination store, which is not
|
||||
// trusted, and is printed in the report. Accept it only in the
|
||||
// form of a remote key.
|
||||
if !isBlobHash(snapshotID) {
|
||||
log.Warn("Skipping non-conforming key under metadata/",
|
||||
"key", obj.Key)
|
||||
|
||||
if isManifest {
|
||||
skippedManifestCount++
|
||||
}
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
if _, exists := snapshotMetadata[snapshotID]; !exists {
|
||||
snapshotMetadata[snapshotID] = &SnapshotMetadataInfo{SnapshotID: snapshotID}
|
||||
@@ -298,7 +344,10 @@ func (v *Vaultik) collectSnapshotMetadata() (
|
||||
|
||||
info := snapshotMetadata[snapshotID]
|
||||
|
||||
filename := parts[2]
|
||||
if isManifest {
|
||||
info.hasManifest = true
|
||||
}
|
||||
|
||||
if strings.HasPrefix(filename, "manifest") {
|
||||
info.ManifestSize = obj.Size
|
||||
} else if strings.HasPrefix(filename, "db") {
|
||||
@@ -315,17 +364,25 @@ func (v *Vaultik) collectSnapshotMetadata() (
|
||||
|
||||
sort.Strings(snapshotIDs)
|
||||
|
||||
return snapshotMetadata, snapshotIDs, nil
|
||||
return snapshotMetadata, snapshotIDs, skippedManifestCount, nil
|
||||
}
|
||||
|
||||
// collectReferencedBlobsFromManifests downloads manifests and returns
|
||||
// referenced blob hashes with sizes.
|
||||
// collectReferencedBlobsFromManifests downloads the listed manifests
|
||||
// and returns referenced blob hashes with sizes, and the remote keys
|
||||
// of the manifests it could not read.
|
||||
func (v *Vaultik) collectReferencedBlobsFromManifests(
|
||||
snapshotIDs []string, snapshotMetadata map[string]*SnapshotMetadataInfo,
|
||||
) map[string]int64 {
|
||||
) (map[string]int64, []string) {
|
||||
referencedBlobs := make(map[string]int64)
|
||||
|
||||
var unreadable []string
|
||||
|
||||
for _, snapshotID := range snapshotIDs {
|
||||
info := snapshotMetadata[snapshotID]
|
||||
if !info.hasManifest {
|
||||
continue
|
||||
}
|
||||
|
||||
// snapshotIDs here are remote keys, taken straight from the
|
||||
// metadata/ listing. downloadManifestByKey is the single reader
|
||||
// for remote manifests; see its doc comment.
|
||||
@@ -333,10 +390,11 @@ func (v *Vaultik) collectReferencedBlobsFromManifests(
|
||||
if err != nil {
|
||||
log.Warn("Failed to read manifest", "snapshot", snapshotID, "error", err)
|
||||
|
||||
unreadable = append(unreadable, snapshotID)
|
||||
|
||||
continue
|
||||
}
|
||||
|
||||
info := snapshotMetadata[snapshotID]
|
||||
info.BlobCount = manifest.BlobCount
|
||||
|
||||
var blobsSize int64
|
||||
@@ -349,7 +407,7 @@ func (v *Vaultik) collectReferencedBlobsFromManifests(
|
||||
info.BlobsSize = blobsSize
|
||||
}
|
||||
|
||||
return referencedBlobs
|
||||
return referencedBlobs, unreadable
|
||||
}
|
||||
|
||||
// populateRemoteInfoResult fills in the result's snapshot and
|
||||
@@ -378,8 +436,9 @@ func (v *Vaultik) populateRemoteInfoResult(
|
||||
}
|
||||
|
||||
// scanRemoteBlobStorage lists all blobs on remote and computes orphan
|
||||
// stats. showText is true only when the human report is being printed
|
||||
// (not --json, not --quiet), gating the progress line.
|
||||
// stats when every listed manifest was read. showText is true only
|
||||
// when the human report is being printed (not --json, not --quiet),
|
||||
// gating the progress line.
|
||||
func (v *Vaultik) scanRemoteBlobStorage(
|
||||
result *RemoteInfoResult, referencedBlobs map[string]int64, showText bool,
|
||||
) error {
|
||||
@@ -406,13 +465,28 @@ func (v *Vaultik) scanRemoteBlobStorage(
|
||||
result.TotalBlobSize += obj.Size
|
||||
}
|
||||
|
||||
// A blob named only by a manifest that could not be read, or by one
|
||||
// under a skipped name, would be counted as orphaned, so the orphan
|
||||
// figures stay unknown.
|
||||
if len(result.UnreadableManifests) > 0 || result.SkippedManifestCount > 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
var (
|
||||
orphanedCount int
|
||||
orphanedSize int64
|
||||
)
|
||||
|
||||
for hash, size := range allBlobs {
|
||||
if _, referenced := referencedBlobs[hash]; !referenced {
|
||||
result.OrphanedBlobCount++
|
||||
result.OrphanedBlobSize += size
|
||||
orphanedCount++
|
||||
orphanedSize += size
|
||||
}
|
||||
}
|
||||
|
||||
result.OrphanedBlobCount = &orphanedCount
|
||||
result.OrphanedBlobSize = &orphanedSize
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -465,11 +539,21 @@ func (v *Vaultik) printRemoteInfoTable(result *RemoteInfoResult) {
|
||||
v.stdoutf("Referenced by snapshots: %s (%s)\n",
|
||||
humanize.Comma(int64(result.ReferencedBlobCount)),
|
||||
ubytes(result.ReferencedBlobSize))
|
||||
v.stdoutf("Orphaned (unreferenced): %s (%s)\n",
|
||||
humanize.Comma(int64(result.OrphanedBlobCount)),
|
||||
ubytes(result.OrphanedBlobSize))
|
||||
|
||||
if result.OrphanedBlobCount > 0 {
|
||||
if result.OrphanedBlobCount == nil {
|
||||
v.stdoutf("Orphaned (unreferenced): unknown "+
|
||||
"(%d manifest(s) could not be read, "+
|
||||
"%d manifest(s) under a non-conforming name skipped)\n",
|
||||
len(result.UnreadableManifests), result.SkippedManifestCount)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
v.stdoutf("Orphaned (unreferenced): %s (%s)\n",
|
||||
humanize.Comma(int64(*result.OrphanedBlobCount)),
|
||||
ubytes(*result.OrphanedBlobSize))
|
||||
|
||||
if *result.OrphanedBlobCount > 0 {
|
||||
v.stdoutf("\nRun 'vaultik prune' to remove orphaned blobs.\n")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -928,17 +928,17 @@ func setupDedupBackupEnv(
|
||||
}
|
||||
}
|
||||
|
||||
// runDedupSnapshot creates a "dedup" snapshot, scans dataDir into it,
|
||||
// completes it, and exports its metadata, returning the snapshot ID and
|
||||
// scan result.
|
||||
// runDedupSnapshot creates a snapshot with the given name, scans dataDir
|
||||
// into it, completes it, and exports its metadata, returning the snapshot
|
||||
// ID and scan result.
|
||||
func runDedupSnapshot(
|
||||
ctx context.Context, t *testing.T,
|
||||
sm *snapshot.SnapshotManager, scanner *snapshot.Scanner,
|
||||
hostname, dataDir, dbPath string,
|
||||
hostname, name, dataDir, dbPath string,
|
||||
) (string, *snapshot.ScanResult) {
|
||||
t.Helper()
|
||||
|
||||
id, err := sm.CreateSnapshotWithName(ctx, hostname, "dedup", "v", "g")
|
||||
id, err := sm.CreateSnapshotWithName(ctx, hostname, name, "v", "g")
|
||||
require.NoError(t, err)
|
||||
|
||||
result, err := scanner.Scan(ctx, dataDir, id)
|
||||
@@ -980,16 +980,15 @@ func TestDedupOnlySnapshotRestores(t *testing.T) {
|
||||
|
||||
// First snapshot — uploads all blobs.
|
||||
_, r1 := runDedupSnapshot(ctx, t, sm, makeScanner(),
|
||||
cfg.Hostname, dataDir, dbPath)
|
||||
cfg.Hostname, "first", dataDir, dbPath)
|
||||
require.Positive(t, r1.BlobsCreated,
|
||||
"first snapshot should upload at least one blob")
|
||||
|
||||
// Second snapshot — same data, every chunk dedups. Sleep past the
|
||||
// second-precision timestamp so the snapshot IDs differ.
|
||||
time.Sleep(1100 * time.Millisecond)
|
||||
|
||||
// Second snapshot — same data, every chunk dedups. Its own name gives
|
||||
// it a different snapshot ID without waiting for the one-second
|
||||
// timestamp in the ID to tick over.
|
||||
id2, r2 := runDedupSnapshot(ctx, t, sm, makeScanner(),
|
||||
cfg.Hostname, dataDir, dbPath)
|
||||
cfg.Hostname, "second", dataDir, dbPath)
|
||||
require.Equal(t, 0, r2.BlobsCreated,
|
||||
"second snapshot should upload zero new blobs (fully dedup'd)")
|
||||
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/spf13/afero"
|
||||
@@ -77,10 +78,9 @@ func backUpThenUnplug(
|
||||
// TestFirstBackupCreatesDestinationDirectory checks that a first backup
|
||||
// to a destination directory that does not exist yet creates it, and
|
||||
// that the destination can be listed afterwards.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestFirstBackupCreatesDestinationDirectory(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
ctx := context.Background()
|
||||
storeDir := filepath.Join(t.TempDir(), "volume", "backup")
|
||||
@@ -95,10 +95,9 @@ func TestFirstBackupCreatesDestinationDirectory(t *testing.T) {
|
||||
// TestListSnapshotsWarnsWhenDestinationMissing checks that snapshot list
|
||||
// warns and shows the local index alone, without reporting the local
|
||||
// snapshot as missing from the destination.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestListSnapshotsWarnsWhenDestinationMissing(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
ctx := context.Background()
|
||||
v, repos, out := backUpThenUnplug(ctx, t)
|
||||
@@ -115,10 +114,9 @@ func TestListSnapshotsWarnsWhenDestinationMissing(t *testing.T) {
|
||||
// TestRemoveSnapshotWarnsWhenDestinationMissing checks that snapshot
|
||||
// remove warns that the metadata could not be removed from the
|
||||
// destination, instead of reporting that it was.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestRemoveSnapshotWarnsWhenDestinationMissing(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
ctx := context.Background()
|
||||
v, repos, out := backUpThenUnplug(ctx, t)
|
||||
@@ -137,10 +135,9 @@ func TestRemoveSnapshotWarnsWhenDestinationMissing(t *testing.T) {
|
||||
// TestPruneKeepsLocalRecordsWhenDestinationMissing checks that prune
|
||||
// fails on a destination it cannot list and deletes no local snapshot
|
||||
// record.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestPruneKeepsLocalRecordsWhenDestinationMissing(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
ctx := context.Background()
|
||||
v, repos, _ := backUpThenUnplug(ctx, t)
|
||||
@@ -153,3 +150,22 @@ func TestPruneKeepsLocalRecordsWhenDestinationMissing(t *testing.T) {
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, snapshots, 1, "prune must delete no local snapshot record")
|
||||
}
|
||||
|
||||
// TestPurgeSaysListingFailedOnceWhenDestinationMissing checks that
|
||||
// snapshot purge fails on a destination it cannot list, with an error
|
||||
// that says "listing remote snapshots" once.
|
||||
func TestPurgeSaysListingFailedOnceWhenDestinationMissing(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
ctx := context.Background()
|
||||
v, _, _ := backUpThenUnplug(ctx, t)
|
||||
|
||||
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
||||
KeepLatest: true,
|
||||
Force: true,
|
||||
})
|
||||
require.ErrorIs(t, err, fs.ErrNotExist)
|
||||
assert.Equal(t, 1, strings.Count(err.Error(), "listing remote snapshots"),
|
||||
err.Error())
|
||||
}
|
||||
|
||||
@@ -14,10 +14,9 @@ import (
|
||||
// the discarded-error bug: getTableCount for a table its query cannot
|
||||
// resolve must not silently become 0. A count that could not be read is
|
||||
// reported as unknown, which a reader can tell apart from an empty table.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestTableCountForReportSurfacesReadFailure(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
ctx := context.Background()
|
||||
|
||||
|
||||
@@ -42,7 +42,7 @@ func setupConsistencyTest(
|
||||
completedAt := startedAt.Add(5 * time.Minute)
|
||||
snap := &database.Snapshot{
|
||||
ID: types.SnapshotID(id),
|
||||
Hostname: testHostname,
|
||||
Hostname: snapHostname,
|
||||
VaultikVersion: testLabel,
|
||||
StartedAt: startedAt,
|
||||
CompletedAt: &completedAt,
|
||||
|
||||
@@ -17,8 +17,10 @@ import (
|
||||
"sneak.berlin/go/vaultik/internal/vaultik"
|
||||
)
|
||||
|
||||
// Snapshot IDs reused across the purge tests.
|
||||
// Snapshot IDs reused across the purge tests, and the hostname they were
|
||||
// taken on.
|
||||
const (
|
||||
snapHostname = "testhost"
|
||||
snapSystemT0 = "testhost_system_2026-01-01T00:00:00Z"
|
||||
snapHomeT0 = "testhost_home_2026-01-01T00:00:00Z"
|
||||
snapHomeT1 = "testhost_home_2026-01-01T01:00:00Z"
|
||||
@@ -26,9 +28,12 @@ const (
|
||||
)
|
||||
|
||||
// setupPurgeTest creates a Vaultik instance with an in-memory database and mock
|
||||
// storage pre-populated with the given snapshot IDs. Each snapshot is marked as
|
||||
// completed. Remote metadata stubs are created so syncWithRemote keeps them.
|
||||
func setupPurgeTest(t *testing.T, snapshotIDs []string) *vaultik.Vaultik {
|
||||
// storage pre-populated with the given snapshot IDs, all taken on hostname.
|
||||
// Each snapshot is marked as completed. Remote metadata stubs are created so
|
||||
// syncWithRemote keeps them.
|
||||
func setupPurgeTest(
|
||||
t *testing.T, hostname string, snapshotIDs []string,
|
||||
) *vaultik.Vaultik {
|
||||
t.Helper()
|
||||
|
||||
ctx := context.Background()
|
||||
@@ -51,7 +56,7 @@ func setupPurgeTest(t *testing.T, snapshotIDs []string) *vaultik.Vaultik {
|
||||
completedAt := startedAt.Add(5 * time.Minute)
|
||||
snap := &database.Snapshot{
|
||||
ID: types.SnapshotID(id),
|
||||
Hostname: "testhost",
|
||||
Hostname: types.Hostname(hostname),
|
||||
VaultikVersion: testLabel,
|
||||
StartedAt: startedAt,
|
||||
CompletedAt: &completedAt,
|
||||
@@ -120,7 +125,7 @@ func TestPurgeKeepLatest_PerName(t *testing.T) {
|
||||
"testhost_system_2026-01-01T04:00:00Z",
|
||||
}
|
||||
|
||||
v := setupPurgeTest(t, snapshotIDs)
|
||||
v := setupPurgeTest(t, snapHostname, snapshotIDs)
|
||||
|
||||
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
||||
KeepLatest: true,
|
||||
@@ -148,7 +153,7 @@ func TestPurgeKeepLatest_SingleName(t *testing.T) {
|
||||
"testhost_home_2026-01-01T02:00:00Z",
|
||||
}
|
||||
|
||||
v := setupPurgeTest(t, snapshotIDs)
|
||||
v := setupPurgeTest(t, snapHostname, snapshotIDs)
|
||||
|
||||
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
||||
KeepLatest: true,
|
||||
@@ -176,7 +181,7 @@ func TestPurgeKeepLatest_WithNameFilter(t *testing.T) {
|
||||
"testhost_home_2026-01-01T04:00:00Z",
|
||||
}
|
||||
|
||||
v := setupPurgeTest(t, snapshotIDs)
|
||||
v := setupPurgeTest(t, snapHostname, snapshotIDs)
|
||||
|
||||
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
||||
KeepLatest: true,
|
||||
@@ -198,7 +203,7 @@ func TestPurgeKeepLatest_NoSnapshots(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
v := setupPurgeTest(t, nil)
|
||||
v := setupPurgeTest(t, snapHostname, nil)
|
||||
|
||||
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
||||
KeepLatest: true,
|
||||
@@ -216,7 +221,7 @@ func TestPurgeKeepLatest_NameFilterNoMatch(t *testing.T) {
|
||||
"testhost_system_2026-01-01T01:00:00Z",
|
||||
}
|
||||
|
||||
v := setupPurgeTest(t, snapshotIDs)
|
||||
v := setupPurgeTest(t, snapHostname, snapshotIDs)
|
||||
|
||||
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
||||
KeepLatest: true,
|
||||
@@ -243,7 +248,7 @@ func TestPurgeOlderThan_WithNameFilter(t *testing.T) {
|
||||
snapHomeT0,
|
||||
}
|
||||
|
||||
v := setupPurgeTest(t, snapshotIDs)
|
||||
v := setupPurgeTest(t, snapHostname, snapshotIDs)
|
||||
|
||||
// Purge only "home" snapshots older than 365 days
|
||||
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
||||
@@ -277,7 +282,7 @@ func TestPurgeKeepLatest_ThreeNames(t *testing.T) {
|
||||
"testhost_home_2026-01-01T06:00:00Z",
|
||||
}
|
||||
|
||||
v := setupPurgeTest(t, snapshotIDs)
|
||||
v := setupPurgeTest(t, snapHostname, snapshotIDs)
|
||||
|
||||
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
||||
KeepLatest: true,
|
||||
@@ -291,3 +296,28 @@ func TestPurgeKeepLatest_ThreeNames(t *testing.T) {
|
||||
assert.Contains(t, remaining, "testhost_system_2026-01-01T04:00:00Z")
|
||||
assert.Contains(t, remaining, "testhost_media_2026-01-01T05:00:00Z")
|
||||
}
|
||||
|
||||
// A hostname may contain underscores, so the snapshot name cannot be found
|
||||
// by splitting the ID at them. A purge by name must still select "docs".
|
||||
func TestPurgeKeepLatest_HostnameWithUnderscore(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
system = "my_host_system_2026-01-01T00:00:00Z"
|
||||
docsT1 = "my_host_docs_2026-01-01T01:00:00Z"
|
||||
docsT2 = "my_host_docs_2026-01-01T02:00:00Z"
|
||||
)
|
||||
|
||||
v := setupPurgeTest(t, "my_host", []string{system, docsT1, docsT2})
|
||||
|
||||
err := v.PurgeSnapshotsWithOptions(&vaultik.SnapshotPurgeOptions{
|
||||
KeepLatest: true,
|
||||
Force: true,
|
||||
Names: []string{"docs"},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.ElementsMatch(t, []string{system, docsT2},
|
||||
listRemainingSnapshots(t, v))
|
||||
}
|
||||
|
||||
@@ -0,0 +1,159 @@
|
||||
package vaultik_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/vaultik/internal/log"
|
||||
"sneak.berlin/go/vaultik/internal/snapshot"
|
||||
)
|
||||
|
||||
// testBlobHashB is a blob that the manifest written by addRemote does
|
||||
// not reference.
|
||||
const testBlobHashB = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" +
|
||||
"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
|
||||
|
||||
// TestRemoteInfo_UnreadableManifestLeavesOrphansUnknown checks that a
|
||||
// manifest remote info cannot read makes the orphan figures unknown. A
|
||||
// blob referenced only by that snapshot would otherwise be counted as
|
||||
// orphaned, and the report would advise running prune.
|
||||
func TestRemoteInfo_UnreadableManifestLeavesOrphansUnknown(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newListEnv(t)
|
||||
|
||||
// The readable manifest references blob A only.
|
||||
env.addRemote(t, listRemoteID, time.Date(2026, 3, 2, 0, 0, 0, 0, time.UTC))
|
||||
addBlob(t, env.store.testStorer, testBlobHashA)
|
||||
addBlob(t, env.store.testStorer, testBlobHashB)
|
||||
|
||||
// With every manifest readable, blob B is orphaned.
|
||||
require.NoError(t, env.v.RemoteInfo(true))
|
||||
|
||||
var doc map[string]any
|
||||
|
||||
require.NoError(t, json.Unmarshal(env.stdout.Bytes(), &doc))
|
||||
assert.InDelta(t, 1, doc["orphaned_blob_count"], 0)
|
||||
|
||||
// A second snapshot whose manifest cannot be decoded. Blob B may be
|
||||
// one of its blobs.
|
||||
unreadableKey := snapshot.RemoteSnapshotKey(listLocalID)
|
||||
require.NoError(t, env.store.Put(context.Background(),
|
||||
"metadata/"+unreadableKey+"/manifest.json.zst",
|
||||
bytes.NewReader([]byte("not a valid manifest"))))
|
||||
|
||||
env.stdout.Reset()
|
||||
require.NoError(t, env.v.RemoteInfo(false))
|
||||
|
||||
text := env.stdout.String()
|
||||
assert.Contains(t, text, "Orphaned (unreferenced): unknown "+
|
||||
"(1 manifest(s) could not be read, "+
|
||||
"0 manifest(s) under a non-conforming name skipped)")
|
||||
assert.NotContains(t, text, "vaultik prune")
|
||||
|
||||
env.stdout.Reset()
|
||||
require.NoError(t, env.v.RemoteInfo(true))
|
||||
|
||||
doc = nil
|
||||
require.NoError(t, json.Unmarshal(env.stdout.Bytes(), &doc))
|
||||
assert.Contains(t, doc, "orphaned_blob_count")
|
||||
assert.Nil(t, doc["orphaned_blob_count"])
|
||||
assert.Contains(t, doc, "orphaned_blob_size")
|
||||
assert.Nil(t, doc["orphaned_blob_size"])
|
||||
assert.Equal(t, []any{unreadableKey}, doc["unreadable_manifests"])
|
||||
}
|
||||
|
||||
// TestRemoteInfo_SkipsNonConformingMetadataName checks that a directory
|
||||
// under metadata/ whose name is not a remote key is left out of the
|
||||
// report, and that the orphan figures are unknown when it holds a
|
||||
// manifest. The name comes from the destination store; printed raw, its
|
||||
// control characters would reach the terminal. Its manifest is not
|
||||
// read, so a blob only it references would otherwise be counted as
|
||||
// orphaned.
|
||||
func TestRemoteInfo_SkipsNonConformingMetadataName(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newListEnv(t)
|
||||
env.addRemote(t, listRemoteID, time.Date(2026, 3, 2, 0, 0, 0, 0, time.UTC))
|
||||
addBlob(t, env.store.testStorer, testBlobHashA)
|
||||
addBlob(t, env.store.testStorer, testBlobHashB)
|
||||
require.NoError(t, env.store.Put(context.Background(),
|
||||
"metadata/\x1b[31mred/manifest.json.zst",
|
||||
bytes.NewReader([]byte("not a valid manifest"))))
|
||||
|
||||
require.NoError(t, env.v.RemoteInfo(false))
|
||||
|
||||
text := env.stdout.String()
|
||||
assert.NotContains(t, text, "\x1b")
|
||||
assert.NotContains(t, text, "31mred")
|
||||
assert.Contains(t, text, "Total (1 snapshots)")
|
||||
assert.Contains(t, text, "Orphaned (unreferenced): unknown "+
|
||||
"(0 manifest(s) could not be read, "+
|
||||
"1 manifest(s) under a non-conforming name skipped)")
|
||||
assert.NotContains(t, text, "vaultik prune")
|
||||
|
||||
env.stdout.Reset()
|
||||
require.NoError(t, env.v.RemoteInfo(true))
|
||||
|
||||
out := env.stdout.String()
|
||||
assert.NotContains(t, out, "31mred")
|
||||
|
||||
var doc map[string]any
|
||||
|
||||
require.NoError(t, json.Unmarshal([]byte(out), &doc))
|
||||
assert.Contains(t, doc, "orphaned_blob_count")
|
||||
assert.Nil(t, doc["orphaned_blob_count"])
|
||||
assert.Contains(t, doc, "orphaned_blob_size")
|
||||
assert.Nil(t, doc["orphaned_blob_size"])
|
||||
assert.InDelta(t, 1, doc["skipped_manifest_count"], 0)
|
||||
assert.NotContains(t, doc, "unreadable_manifests")
|
||||
}
|
||||
|
||||
// TestRemoteInfo_DirectoryWithoutManifestLeavesOrphansKnown checks that
|
||||
// a directory under metadata/ holding no manifest.json.zst, such as one
|
||||
// left by a backup interrupted before its manifest upload, leaves the
|
||||
// orphan figures known. prune does not treat such a directory as a
|
||||
// snapshot and deletes the blobs the report lists as orphaned.
|
||||
func TestRemoteInfo_DirectoryWithoutManifestLeavesOrphansKnown(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newListEnv(t)
|
||||
env.addRemote(t, listRemoteID, time.Date(2026, 3, 2, 0, 0, 0, 0, time.UTC))
|
||||
addBlob(t, env.store.testStorer, testBlobHashA)
|
||||
addBlob(t, env.store.testStorer, testBlobHashB)
|
||||
|
||||
// One directory under a remote key and one under a non-conforming
|
||||
// name, each holding only a database.
|
||||
names := []string{snapshot.RemoteSnapshotKey(listLocalID), "\x1b[31mred"}
|
||||
for _, name := range names {
|
||||
require.NoError(t, env.store.Put(context.Background(),
|
||||
"metadata/"+name+"/db.zst.age",
|
||||
bytes.NewReader([]byte("not a valid database"))))
|
||||
}
|
||||
|
||||
require.NoError(t, env.v.RemoteInfo(false))
|
||||
|
||||
text := env.stdout.String()
|
||||
assert.NotContains(t, text, "\x1b")
|
||||
assert.Contains(t, text, "Downloading 1 manifest(s)...")
|
||||
assert.Contains(t, text, "Orphaned (unreferenced): 1 (")
|
||||
assert.Contains(t, text, "Run 'vaultik prune' to remove orphaned blobs.")
|
||||
|
||||
env.stdout.Reset()
|
||||
require.NoError(t, env.v.RemoteInfo(true))
|
||||
|
||||
var doc map[string]any
|
||||
|
||||
require.NoError(t, json.Unmarshal(env.stdout.Bytes(), &doc))
|
||||
assert.InDelta(t, 1, doc["orphaned_blob_count"], 0)
|
||||
assert.NotContains(t, doc, "unreadable_manifests")
|
||||
assert.NotContains(t, doc, "skipped_manifest_count")
|
||||
}
|
||||
@@ -164,10 +164,9 @@ func scratchEntries(t *testing.T, dir string) []string {
|
||||
// restore while a blob download is in progress. The download fails only
|
||||
// because of the cancel, so Restore must return context.Canceled without
|
||||
// reporting the file that needs the blob as failed.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestRestoreSkipErrorsCancelDuringBlobDownload(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
|
||||
@@ -45,9 +45,10 @@ type missingBlobBackup struct {
|
||||
// after one blob of a two-blob snapshot was deleted. Every file stored in
|
||||
// that blob must be reported as failed and left absent, every other file
|
||||
// must be restored intact, and Restore must still return an error.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestRestoreSkipErrorsSkipsFilesOfMissingBlob(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
ctx := context.Background()
|
||||
backup := backupThenDeleteOneBlob(ctx, t)
|
||||
|
||||
@@ -85,9 +86,10 @@ func TestRestoreSkipErrorsSkipsFilesOfMissingBlob(t *testing.T) {
|
||||
|
||||
// TestRestoreMissingBlobAbortsWithoutSkipErrors checks that a deleted blob
|
||||
// still ends the restore with an error when SkipErrors is not set.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestRestoreMissingBlobAbortsWithoutSkipErrors(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
ctx := context.Background()
|
||||
backup := backupThenDeleteOneBlob(ctx, t)
|
||||
|
||||
@@ -107,7 +109,6 @@ func backupThenDeleteOneBlob(
|
||||
ctx context.Context, t *testing.T,
|
||||
) *missingBlobBackup {
|
||||
t.Helper()
|
||||
log.Initialize(log.Config{})
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
package vaultik_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/afero"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/vaultik/internal/database"
|
||||
"sneak.berlin/go/vaultik/internal/log"
|
||||
"sneak.berlin/go/vaultik/internal/storage"
|
||||
"sneak.berlin/go/vaultik/internal/vaultik"
|
||||
)
|
||||
|
||||
// A file rewritten with its size unchanged and a new mtime in the same
|
||||
// second as the mtime the index holds must still be backed up. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/226.
|
||||
func TestBackupOfSameSecondRewriteRestoresNewContent(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
dataDir := filepath.Join(tempDir, "src")
|
||||
storeDir := filepath.Join(tempDir, "remote")
|
||||
restoreDir := filepath.Join(tempDir, "restored")
|
||||
dbPath := filepath.Join(tempDir, "index.sqlite")
|
||||
rewrittenPath := filepath.Join(dataDir, "small.txt")
|
||||
|
||||
ctx := context.Background()
|
||||
files := writeFaultSourceTree(t, fs, dataDir)
|
||||
cfg := changedFileConfig(dataDir, dbPath)
|
||||
|
||||
firstMTime := time.Date(2026, time.January, 2, 3, 4, 5, 0, time.UTC).
|
||||
Add(100 * time.Millisecond)
|
||||
secondMTime := firstMTime.Add(800 * time.Millisecond)
|
||||
|
||||
require.NoError(t, fs.Chtimes(rewrittenPath, firstMTime, firstMTime))
|
||||
|
||||
store, err := storage.NewFileStorer(storeDir)
|
||||
require.NoError(t, err)
|
||||
|
||||
db, err := database.New(ctx, dbPath)
|
||||
require.NoError(t, err)
|
||||
|
||||
repos := database.NewRepositories(db)
|
||||
v := newBackupVaultik(ctx, cfg, store, repos, db, fs)
|
||||
|
||||
require.NoError(t, backUp(v, "first"))
|
||||
|
||||
// Upper-casing ASCII text keeps its size.
|
||||
files[rewrittenPath] = bytes.ToUpper(files[rewrittenPath])
|
||||
require.NoError(t, afero.WriteFile(fs, rewrittenPath, files[rewrittenPath], 0o644))
|
||||
require.NoError(t, fs.Chtimes(rewrittenPath, secondMTime, secondMTime))
|
||||
|
||||
require.NoError(t, backUp(v, "second"))
|
||||
|
||||
id := localSnapshotID(ctx, t, repos, "second")
|
||||
require.NoError(t, db.Close())
|
||||
|
||||
reader := newReaderVaultik(ctx, cfg, store, nil, fs)
|
||||
require.NoError(t, reader.Restore(&vaultik.RestoreOptions{
|
||||
SnapshotID: id,
|
||||
TargetDir: restoreDir,
|
||||
Verify: true,
|
||||
}))
|
||||
|
||||
assertRestoredTree(t, fs, restoreDir, files)
|
||||
}
|
||||
@@ -14,6 +14,7 @@ import (
|
||||
|
||||
"sneak.berlin/go/vaultik/internal/log"
|
||||
"sneak.berlin/go/vaultik/internal/snapshot"
|
||||
"sneak.berlin/go/vaultik/internal/types"
|
||||
)
|
||||
|
||||
// Sentinel errors for snapshot management.
|
||||
@@ -495,19 +496,23 @@ func (v *Vaultik) PurgeSnapshotsWithOptions(opts *SnapshotPurgeOptions) error {
|
||||
nameFilter[n] = struct{}{}
|
||||
}
|
||||
|
||||
// Collect completed snapshots, applying the name filter.
|
||||
// Collect completed snapshots and their names, applying the name filter.
|
||||
snapshots := make([]SnapshotInfo, 0, len(dbSnapshots))
|
||||
names := make(map[types.SnapshotID]string, len(dbSnapshots))
|
||||
|
||||
for _, s := range dbSnapshots {
|
||||
if s.CompletedAt == nil {
|
||||
continue
|
||||
}
|
||||
|
||||
name := parseSnapshotName(s.ID.String(), s.Hostname.String())
|
||||
if len(nameFilter) > 0 {
|
||||
if _, ok := nameFilter[parseSnapshotName(s.ID.String())]; !ok {
|
||||
if _, ok := nameFilter[name]; !ok {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
names[s.ID] = name
|
||||
snapshots = append(snapshots, SnapshotInfo{
|
||||
ID: s.ID,
|
||||
Timestamp: s.StartedAt,
|
||||
@@ -520,7 +525,7 @@ func (v *Vaultik) PurgeSnapshotsWithOptions(opts *SnapshotPurgeOptions) error {
|
||||
return snapshots[i].Timestamp.After(snapshots[j].Timestamp)
|
||||
})
|
||||
|
||||
toDelete, err := selectSnapshotsToPurge(snapshots, opts)
|
||||
toDelete, err := selectSnapshotsToPurge(snapshots, names, opts)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -538,9 +543,11 @@ func (v *Vaultik) PurgeSnapshotsWithOptions(opts *SnapshotPurgeOptions) error {
|
||||
|
||||
// selectSnapshotsToPurge applies the purge retention criteria to the
|
||||
// newest-first sorted snapshot list and returns the deletion
|
||||
// candidates.
|
||||
// candidates. names maps each snapshot's ID to its snapshot name.
|
||||
func selectSnapshotsToPurge(
|
||||
snapshots []SnapshotInfo, opts *SnapshotPurgeOptions,
|
||||
snapshots []SnapshotInfo,
|
||||
names map[types.SnapshotID]string,
|
||||
opts *SnapshotPurgeOptions,
|
||||
) ([]SnapshotInfo, error) {
|
||||
var toDelete []SnapshotInfo
|
||||
|
||||
@@ -551,7 +558,7 @@ func selectSnapshotsToPurge(
|
||||
seen := make(map[string]bool)
|
||||
|
||||
for _, snap := range snapshots {
|
||||
name := parseSnapshotName(snap.ID.String())
|
||||
name := names[snap.ID]
|
||||
if seen[name] {
|
||||
toDelete = append(toDelete, snap)
|
||||
|
||||
@@ -1045,7 +1052,7 @@ func (v *Vaultik) syncWithRemote() error {
|
||||
// every local snapshot record (issue #160).
|
||||
remoteKeys, err := v.listAllRemoteSnapshotKeys()
|
||||
if err != nil {
|
||||
return fmt.Errorf("listing remote snapshots: %w", err)
|
||||
return err
|
||||
}
|
||||
|
||||
remoteKeySet := make(map[string]bool, len(remoteKeys))
|
||||
|
||||
@@ -54,8 +54,6 @@ type summaryEnv struct {
|
||||
func newSummaryEnv(t *testing.T) *summaryEnv {
|
||||
t.Helper()
|
||||
|
||||
log.Initialize(log.Config{})
|
||||
|
||||
fs := afero.NewOsFs()
|
||||
tempDir := t.TempDir()
|
||||
srcDir := filepath.Join(tempDir, "src")
|
||||
@@ -193,9 +191,10 @@ func (e *summaryEnv) dataLine(total, backedUp int64) string {
|
||||
// A first backup stores copy.bin's chunks while backing up a.bin, so
|
||||
// copy.bin's chunks are deduplicated within the run. Each file and byte
|
||||
// is still counted once.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestSnapshotSummaryFirstRun(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newSummaryEnv(t)
|
||||
|
||||
summary := env.backUp(t, "first", false)
|
||||
@@ -219,9 +218,10 @@ func TestSnapshotSummaryFirstRun(t *testing.T) {
|
||||
// An incremental backup where a.bin's mtime changed but its content did
|
||||
// not: a.bin is backed up again and every one of its chunks is already
|
||||
// stored.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestSnapshotSummaryIncrementalRunWithDeduplicatedChunks(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newSummaryEnv(t)
|
||||
|
||||
env.backUp(t, "first", false)
|
||||
@@ -251,9 +251,10 @@ func TestSnapshotSummaryIncrementalRunWithDeduplicatedChunks(t *testing.T) {
|
||||
// Under --cron the progress reporter is off; the upload figures must
|
||||
// still reach the summary and the snapshots row. The snapshot has two
|
||||
// paths, each backed up by its own scan.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestSnapshotSummaryCronRunRecordsUploads(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
env := newSummaryEnv(t)
|
||||
|
||||
summary := env.backUp(t, "split", true)
|
||||
|
||||
@@ -18,10 +18,9 @@ import (
|
||||
// A backup without --cron runs the progress reporter while one scanner
|
||||
// scans each path of the snapshot in turn. See
|
||||
// https://git.eeqj.de/sneak/vaultik/issues/253.
|
||||
//
|
||||
//nolint:paralleltest // installs the global logger via log.Initialize
|
||||
func TestBackupWithoutCronOfTwoPathSnapshotRestoresBothPaths(t *testing.T) {
|
||||
log.Initialize(log.Config{})
|
||||
t.Parallel()
|
||||
|
||||
const snapshotName = "data"
|
||||
|
||||
|
||||
+5
-2
@@ -1,6 +1,9 @@
|
||||
#!/bin/sh
|
||||
# script/fmt-check: check formatting (read-only). Same scope as
|
||||
# script/fmt, but fails instead of writing.
|
||||
# script/fmt-check: check formatting (read-only). Fails instead of
|
||||
# writing. It checks every Go file outside .tool, which is more than
|
||||
# script/fmt formats: `go fmt ./...` skips `testdata` directories and
|
||||
# files and directories whose names start with `.` or `_`. Fix a file
|
||||
# only this reports with `gofmt -w`.
|
||||
set -eu
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||
|
||||
+3
-4
@@ -21,10 +21,9 @@ goreleaser_version() {
|
||||
head -n 1
|
||||
}
|
||||
|
||||
# Resolve the goreleaser to run, on the same rule script/lint uses for
|
||||
# golangci-lint: a binary on PATH is accepted only when it is exactly
|
||||
# the pinned version, because a differently versioned tool would
|
||||
# produce a differently built release from the same tag. Anything else
|
||||
# Resolve the goreleaser to run. A binary on PATH is accepted only when
|
||||
# it is exactly the pinned version, because a differently versioned tool
|
||||
# would produce a differently built release from the same tag. Anything else
|
||||
# comes from .tool/bin, and a missing one is a loud failure naming the
|
||||
# script that installs it rather than a silent fallback.
|
||||
resolve_goreleaser() {
|
||||
|
||||
+1
-1
@@ -19,7 +19,7 @@ s3:
|
||||
secret_access_key: test-secret-key
|
||||
region: us-east-1
|
||||
use_ssl: true
|
||||
part_size: 5242880 # 5MB
|
||||
part_size: 5242880 # 5MiB
|
||||
index_path: /tmp/vaultik-test.sqlite
|
||||
chunk_size: 10MB
|
||||
blob_size_limit: 10GB
|
||||
|
||||
Reference in New Issue
Block a user