Author SHA1 Message Date
sneak cd86db7416 Name the remote and rclone config file in rclone setup errors (closes #467)
check / check (push) Failing after 14m33s
A remote missing from the config and a remote that fails to open now
both name the remote and the rclone config file vaultik read, and say
when that file does not exist.

Model: opus-5-5
2026-11-19 07:30:57 +00:00
38 changed files with 406 additions and 1058 deletions

No files matched your search

+2 -9
View File
@@ -35,16 +35,9 @@ RUN for attempt in 1 2 3; do \
echo "module download failed: giving up after 3 attempts" >&2; \
exit 1
COPY . .
# -race also turns on checkptr, which checks every pointer conversion. In
# the SQLite library's translated C code that is about a third of the test
# CPU time, much of it under one lock all SQLite work shares, and the
# tests that use SQLite time out on a busy host. The gcflags turn it off
# for that library only; our own packages keep it.
RUN go test -timeout 90s -race -gcflags='modernc.org/...=-d=checkptr=0' \
-cover ./... || \
RUN go test -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -timeout 90s -race -gcflags='modernc.org/...=-d=checkptr=0' \
-v ./...; exit 1; }
go test -timeout 90s -race -v ./...; exit 1; }
# Build stage. Nothing is wanted from either phase above; the copies
# are what make BuildKit build them first, so this stage cannot run
+52 -79
View File
@@ -183,7 +183,6 @@ vaultik [--config <path>] remote nuke --force
vaultik [--config <path>] database delete [--force]
vaultik completion <bash|zsh|fish|powershell>
vaultik version
vaultik --version
```
### global flags
@@ -197,10 +196,9 @@ vaultik --version
that cannot be restored when restoring, instead of aborting. Packing and
storage errors while creating a snapshot (which would leave a chunk recorded
but not stored) still abort the run. A run that skipped anything still
finishes but exits `3` (see [exit status](#exit-status)): `snapshot create`
keeps the snapshot and counts each file or directory it skipped as failed. A
file removed during the backup is also counted as failed but does not change
the exit status.
finishes but exits non-zero: `snapshot create` keeps the snapshot and counts
each file or directory it skipped as failed. A file removed during the backup
is also counted as failed but does not change the exit status.
### locking
@@ -273,19 +271,6 @@ Every `--json` command therefore pipes on its own, with no additional flag:
`vaultik snapshot list --json | jq .` and `vaultik prune --json | jq .` both
work as written.
### exit status
- `0`: the command succeeded.
- `1`: the command failed.
- `3`: `snapshot create` or `snapshot restore` finished, but `--skip-errors`
skipped files it could not read or restore. `restic backup` uses the same
status for this case.
- `130`: Ctrl-C, `SIGINT` or `SIGTERM` stopped the command.
A cron job can page on `1` and only log `3`. A run that skipped files and also
failed in another way exits `1`. For example, a restore that skipped files and
was asked for a path the snapshot does not hold exits `1`.
### environment variables
- `VAULTIK_AGE_SECRET_KEY`: Age private key for decryption (required for
@@ -355,29 +340,31 @@ In such a config, an unknown key or a value of the wrong type is refused.
error that explains how to fix it
- `--cron`: Silent on total success; warnings and errors are still printed (for
crontab)
- `--prune`: After backup, drop older snapshots of each backed-up name, then
delete from the destination store what `prune` deletes there. By default keeps
only the latest snapshot per name; use `--keep-newer-than` for a rolling
window. When `--skip-errors` skipped a file or directory it could not read or
list, the prune does not run and every older snapshot is kept. One of them may
be the only one that holds what was skipped. The command says so and exits
`3`. The prune does not run either when the backup found nothing in a
configured path, as in the mount point of a volume that is not mounted. Every
older snapshot is kept, and the command names the path and exits `1`, on every
run until the path holds something again. Take a path that is empty on purpose
out of the config, or remove the older snapshots by hand with
`snapshot purge`. The prune does not run either when this backup does not find
a volume that an earlier snapshot of the same name in the local index found.
The volume's mount point is a directory at or below a configured path. That
directory is on a different filesystem from its parent and holds an entry the
backup does not exclude. Every older snapshot is kept, and the command names
the directory and exits `1`, on every run until the volume is mounted again. A
backup also does not find the volume when it finds no such entry in its
directory, as when an automounted volume cannot be mounted. Nor does it find
the volume when that directory is no longer backed up, because it is excluded
or its configured path was removed. If any of this is on purpose, as for a
retired volume, remove the older snapshots by hand with `snapshot purge`. A
file removed during the backup does not stop the prune.
- `--prune`: After backup, drop older snapshots of each backed-up name, remove
orphaned blobs older than `prune_grace_period` from remote storage, and delete
each directory under `metadata/` that has no `manifest.json.zst` once every
object in it is older than `prune_grace_period` and its snapshot is not in the
local index, as `prune` does. By default keeps only the latest snapshot per
name; use `--keep-newer-than` for a rolling window. When `--skip-errors`
skipped a file or directory it could not read or list, the prune does not run
and every older snapshot is kept, since one of them may be the only one that
holds what was skipped; the command says so and exits non-zero. When the
backup found nothing in a configured path, as in the mount point of a volume
that is not mounted, the prune does not run either: every older snapshot is
kept, and the command names the path and exits non-zero, on every run until
the path holds something again. Take a path that is empty on purpose out of
the config, or remove the older snapshots by hand with `snapshot purge`. When
an earlier snapshot of the same name in the local index found a volume mounted
at or below a configured path (a directory on a different filesystem from its
parent, with an entry in it that the backup does not exclude) and this backup
does not, the prune does not run either: every older snapshot is kept, and the
command names the directory and exits non-zero, on every run until the volume
is mounted again. A backup also does not find the volume when it finds no such
entry in its directory, as when an automounted volume cannot be mounted, or
when that directory is no longer backed up, because it is excluded or its
configured path was removed. If any of this is on purpose, as for a retired
volume, remove the older snapshots by hand with `snapshot purge`. A file
removed during the backup does not stop the prune.
- `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than this
duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`). Without
`--prune`, or with a value that is not a valid duration, the command fails
@@ -533,16 +520,14 @@ deleted. Step (3) also deletes each directory under `metadata/` that has no
`db.zst.age` but before its manifest, once every object in it is older than
`prune_grace_period` and its snapshot is not in the local index. An object the
listing gives no time for keeps its directory, and a directory whose name is not
a remote key is left alone. Step (3) also deletes each `.partial` object under
`metadata/` and `blobs/` left by a killed upload, once it is older than
`prune_grace_period`. Step (3) reads all remote manifests — network cost scales
with the number of snapshots. `snapshot create --prune` runs the same cleanup
automatically; this is the manual entry point for the same work. If the
a remote key is left alone. Step (3) reads all remote manifests — network cost
scales with the number of snapshots. `snapshot create --prune` runs the same
cleanup automatically; this is the manual entry point for the same work. If the
destination store cannot be listed, or the local index holds snapshots and the
destination answers that it has no `vaultik-destination.txt` (an unplugged
volume's empty mount point), prune fails and deletes nothing. `snapshot purge`
does the same. If a blob, such a directory or such a `.partial` object cannot be
deleted, the command (and `snapshot create --prune`) exits non-zero.
does the same. If a blob or such a directory cannot be deleted, the command (and
`snapshot create --prune`) exits non-zero.
- `--force`: Skip confirmation prompt
- `--json`: Output stats as JSON. Also skips the confirmation prompt, as
@@ -552,10 +537,7 @@ deleted, the command (and `snapshot create --prune`) exits non-zero.
gives no time for it, is counted in `blobs_too_new` and not in `blobs_found`.
A directory under `metadata/` with no manifest is counted in
`metadata_dirs_deleted` once deleted, or in `metadata_dirs_failed` if it
cannot be. A `.partial` object is counted in `partial_uploads_deleted` once
deleted, or in `partial_uploads_failed` if it cannot be. One kept for being
newer than `prune_grace_period`, or of unknown age, is counted in
`partial_uploads_too_new`.
cannot be.
**`info`**: Display system configuration, storage settings, encryption
recipients, and local database statistics.
@@ -570,9 +552,7 @@ in `unreadable_manifests` and counts the manifests under skipped names in
`skipped_manifest_count`. An unreadable manifest also leaves its snapshot's blob
count and blob size unknown, `null` in `--json`. A file under `blobs/` whose
name is not a blob hash is not counted as a blob, since `prune` never deletes
it. A leftover `.partial` object under `metadata/` or `blobs/` is not counted as
a blob either. Such objects are counted on their own line, and in
`partial_upload_count` and `partial_upload_size` in `--json`.
it.
- `--json`: Output as JSON
@@ -622,11 +602,7 @@ only `vaultik-destination.txt` stops such a backup, and only while the local
index holds snapshots.
**Rclone** (`rclone://remote/path`): Uses rclone's 70+ supported cloud
providers. Requires rclone to be configured separately (`rclone config`). The
path after the remote name is relative: rclone resolves it on the remote, for
example against the login directory on sftp. A second slash makes it absolute,
as in `rclone://remote//srv/backup`. A remote of type `local` needs an absolute
path; a relative one is refused.
providers. Requires rclone to be configured separately (`rclone config`).
An upload cut off part-way leaves nothing under the object's name on S3, which
shows an object only once its upload has completed, and on the local filesystem
@@ -635,9 +611,8 @@ with a server-side move (local and sftp among them) are written under a
temporary name ending in `.partial` and moved into place. Rclone remotes without
one are written in place: where such a remote shows a file while it is still
being written, a killed upload can leave a truncated object under its name,
which a later backup takes for complete. A backup ignores a leftover `.partial`
file. `prune` deletes it once it is older than `prune_grace_period`, and
`remote info` counts it.
which a later backup takes for complete. A leftover `.partial` file is ignored
and can be deleted.
A destination shared between hosts or users, each with its own local index,
relies on `prune_grace_period` to keep one's prune from deleting the blobs of
@@ -746,8 +721,6 @@ format does and does not protect.
5. Delete each directory under `metadata/` with no manifest whose objects are
all older than `prune_grace_period`, unless its snapshot is in the local
index
6. Delete each `.partial` object under `metadata/` and `blobs/` older than
`prune_grace_period`
### chunking and deduplication
@@ -781,20 +754,20 @@ Run `vaultik config init` to generate a fully commented config file; a complete
annotated example also lives in
[`configs/config.example.yml`](configs/config.example.yml). Key fields:
| Field | Default | Description |
| -------------------- | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `age_recipients` | (required by `snapshot create`) | Age public keys for encryption. Other commands run without one, so a machine that only restores can leave it empty |
| `age_secret_key` | (unset) | Age private key for decryption (`snapshot restore`, `snapshot files`, `snapshot verify --deep`). Setting it in the config file places the private key on the backed-up host, defeating the public-key-only design (see "why" above). Prefer the `VAULTIK_AGE_SECRET_KEY` environment variable, supplied only on the machine you restore from. |
| `snapshots` | (required) | Named snapshot definitions with paths and excludes |
| `storage_url` | | Storage backend URL (`s3://`, `file://`, `rclone://`) |
| `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) |
| `exclude` | | Global exclude patterns (applied to all snapshots) |
| `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` |
| `compression_level` | `3` | zstd compression level (1-19) |
| `prune_grace_period` | `24h` | How old a blob that no snapshot lists must be before `prune` deletes it, and how old every object in a directory under `metadata/` with no manifest must be before `prune` deletes that directory. `prune` also deletes a `.partial` object left by a killed upload only once it is this old. A backup uploads its blobs and its `db.zst.age` before its manifest, so on a destination shared between hosts or users this must be longer than the longest backup run (see [locking](#locking)). It does not protect a blob a backup reuses from an earlier run that stopped before its manifest: that blob keeps the time of its first upload, so a prune can delete it. A duration such as `36h` or `90m`; `0s` deletes such blobs and directories at once |
| `hostname` | system hostname | Hostname used in snapshot IDs |
| `index_path` | platform data dir | Local SQLite index path |
| Field | Default | Description |
| -------------------- | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `age_recipients` | (required by `snapshot create`) | Age public keys for encryption. Other commands run without one, so a machine that only restores can leave it empty |
| `age_secret_key` | (unset) | Age private key for decryption (`snapshot restore`, `snapshot files`, `snapshot verify --deep`). Setting it in the config file places the private key on the backed-up host, defeating the public-key-only design (see "why" above). Prefer the `VAULTIK_AGE_SECRET_KEY` environment variable, supplied only on the machine you restore from. |
| `snapshots` | (required) | Named snapshot definitions with paths and excludes |
| `storage_url` | | Storage backend URL (`s3://`, `file://`, `rclone://`) |
| `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) |
| `exclude` | | Global exclude patterns (applied to all snapshots) |
| `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` |
| `compression_level` | `3` | zstd compression level (1-19) |
| `prune_grace_period` | `24h` | How old a blob that no snapshot lists must be before `prune` deletes it, and how old every object in a directory under `metadata/` with no manifest must be before `prune` deletes that directory. A backup uploads its blobs and its `db.zst.age` before its manifest, so on a destination shared between hosts or users this must be longer than the longest backup run (see [locking](#locking)). It does not protect a blob a backup reuses from an earlier run that stopped before its manifest: that blob keeps the time of its first upload, so a prune can delete it. A duration such as `36h` or `90m`; `0s` deletes such blobs and directories at once |
| `hostname` | system hostname | Hostname used in snapshot IDs |
| `index_path` | platform data dir | Local SQLite index path |
---
-2
View File
@@ -333,8 +333,6 @@ compression_level: 5
# How old a blob that no snapshot lists must be before prune deletes it, and
# how old every object in a directory under metadata/ with no manifest must
# be before prune deletes that directory
# prune also deletes a .partial object left by a killed upload only once it is
# this old
# A backup uploads its blobs and its db.zst.age before its manifest, so on a
# destination shared between hosts or users this must be longer than the
# longest backup run
-3
View File
@@ -314,9 +314,6 @@ The restore process doesn't use the local database. Instead:
- Delete each directory under `metadata/` that has no `manifest.json.zst`
and no local snapshot record, once every object in it is older than
`prune_grace_period`
3. **Delete Partial Uploads**
- Delete each `.partial` object under `metadata/` or `blobs/` left by a
killed upload, once it is older than `prune_grace_period`
### 6. Incomplete Snapshot Cleanup
+3 -5
View File
@@ -329,8 +329,7 @@ The prune operation is safe because:
1. It keeps every blob listed in any snapshot's manifest and deletes only blobs
that no manifest references. Under `metadata/` it deletes only a directory
that has no `manifest.json.zst`, whose name is a remote key, and that no
local snapshot record claims. It also deletes the `.partial` objects left by
killed uploads, which no backup or restore reads
local snapshot record claims
2. Manifests are unencrypted and can be read without keys
3. If any manifest cannot be downloaded or decoded, prune deletes nothing and
exits with an error, rather than treating that snapshot's blobs as
@@ -344,9 +343,8 @@ The prune operation is safe because:
a backup reuses from an earlier run that stopped before its manifest: the
backup skips the upload of a blob already on the destination, so that blob
keeps the time of its first upload, and a prune before the backup's manifest
is written can delete it; the new snapshot then cannot be restored. Prune
deletes a `.partial` object only once it is older than `prune_grace_period`,
so an upload still running keeps its object. `remote nuke` has no such check
is written can delete it; the new snapshot then cannot be restored.
`remote nuke` has no such check
## Restoration Requirements
+6 -7
View File
@@ -198,8 +198,7 @@ func RunApp(ctx context.Context, app *fx.App) error {
// (or, under `snapshot verify --json`, put in its document). Entry
// turns it into a non-zero exit status without printing anything
// further, so the error line is not doubled. It flows up from
// RunOperation through cobra to Entry, wrapping the operation's error
// so that Entry can choose the exit status from it.
// RunOperation through cobra to Entry.
var errReported = errors.New("operation failed")
// errInterrupted marks an operation that SIGINT or SIGTERM stopped
@@ -233,8 +232,8 @@ func RunOperation(
) error {
var (
mu sync.Mutex
finished bool // op returned before any interrupt cancelled it
opErr error // what op returned, when it finished
finished bool // op returned before any interrupt cancelled it
failed bool // op finished with an error
)
opts.Invokes = append(opts.Invokes,
@@ -259,7 +258,7 @@ func RunOperation(
mu.Lock()
finished = true
opErr = err
failed = err != nil
mu.Unlock()
}
@@ -304,8 +303,8 @@ func RunOperation(
switch {
case !finished:
return errInterrupted
case opErr != nil:
return fmt.Errorf("%w: %w", errReported, opErr)
case failed:
return errReported
default:
return nil
}
+3 -5
View File
@@ -269,11 +269,9 @@ storage_url: ""
# How old a blob that no snapshot lists must be before prune deletes it,
# and how old every object in a directory under metadata/ with no manifest
# must be before prune deletes that directory. prune also deletes a
# .partial object left by a killed upload only once it is this old. A
# backup uploads its blobs and its db.zst.age before its manifest, so on
# a destination shared between hosts or users this must be longer than
# the longest backup run.
# must be before prune deletes that directory. A backup uploads its blobs
# and its db.zst.age before its manifest, so on a destination shared
# between hosts or users this must be longer than the longest backup run.
# It does not protect a blob a backup reuses from an earlier run that
# stopped before its manifest: that blob keeps the time of its first
# upload, so a prune can delete it.
+2 -16
View File
@@ -9,7 +9,6 @@ import (
"sneak.berlin/go/vaultik/internal/globals"
"sneak.berlin/go/vaultik/internal/ui"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// shortCommitLen is the number of git commit hash characters shown in
@@ -21,13 +20,6 @@ const shortCommitLen = 12
// shell reports for a command stopped by Ctrl-C.
const exitCodeInterrupted = 130
// exitCodeSkippedFiles is the exit status of a `snapshot create` or
// `snapshot restore` that finished but skipped files under
// --skip-errors. It is the status `restic backup` uses for a snapshot
// that could not read some files, so a cron job can tell such a run
// from one that made no backup.
const exitCodeSkippedFiles = 3
// Entry is the main entry point for the CLI application.
// It first calls globals.FillUnsetFromBuildInfo, which fills in the
// version, commit and commit date that the linker flags did not set.
@@ -38,9 +30,8 @@ const exitCodeSkippedFiles = 3
// The banner goes to stderr because stdout carries only the output the
// user asked for, such as a completion script or a `config get` value.
//
// It returns the process exit code (0 on success, 3 when files were
// skipped, 130 when interrupted, 1 on any other error) rather than
// calling os.Exit, so that main's
// It returns the process exit code (0 on success, 130 when interrupted,
// 1 on any other error) rather than calling os.Exit, so that main's
// deferred profile writers run before the process ends. See run in
// cmd/vaultik/main.go.
func Entry() int {
@@ -67,11 +58,6 @@ func Entry() int {
return exitCodeInterrupted
}
if errors.Is(err, vaultik.ErrFilesUnreadable) ||
errors.Is(err, vaultik.ErrFilesFailedRestore) {
return exitCodeSkippedFiles
}
return 1
}
@@ -1,83 +0,0 @@
package cli //nolint:testpackage // shares runEntry and the hermetic config helpers
import (
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
const flagSkipErrors = "--skip-errors"
// TestEntrySnapshotCreateSkipErrorsExitsNonZeroWhenItSkips backs up,
// under --skip-errors, a source directory that does not exist, which the
// walk skips. The command exits with exitCodeSkippedFiles and reports
// the backup as completed, not as a failed snapshot creation. See
// https://git.eeqj.de/sneak/vaultik/issues/310 and
// https://git.eeqj.de/sneak/vaultik/issues/439.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotCreateSkipErrorsExitsNonZeroWhenItSkips(t *testing.T) {
// writeBackupConfig would create the source directory.
configPath := writeHermeticPruneConfig(t, false)
code, stdout, stderr := runEntry(t,
flagConfig, configPath, flagSkipErrors, cmdSnapshot, cmdCreate)
assert.Equal(t, exitCodeSkippedFiles, code)
assert.Contains(t, stdout,
"Files: 1 examined, 0 backed up, 0 unchanged, 1 failed.")
assert.NotContains(t, stdout, "Finished successfully.")
assert.Contains(t, stderr,
"Backup completed, but 1 file(s) could not be read")
assert.NotContains(t, stderr, "Snapshot creation failed")
}
// TestEntrySnapshotRestoreSkipErrorsExitStatus backs up, deletes the
// blobs from the destination, and restores under --skip-errors, so every
// file is skipped. A restore that only skipped files exits with
// exitCodeSkippedFiles. One that was also asked for a path the snapshot
// does not hold exits 1. See
// https://git.eeqj.de/sneak/vaultik/issues/439.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotRestoreSkipErrorsExitStatus(t *testing.T) {
for _, testCase := range []struct {
name string
missingPaths []string
want int
}{
{name: "skipped files only", want: exitCodeSkippedFiles},
{
name: "and a path that matches nothing",
missingPaths: []string{"/no/such/path"},
want: 1,
},
} {
t.Run(testCase.name, func(t *testing.T) {
configPath, storeDir := writeBackupConfigWithPrivateKey(t)
code, _, _ := runEntry(t,
flagConfig, configPath, cmdSnapshot, cmdCreate)
require.Equal(t, 0, code)
require.NoError(t, os.RemoveAll(filepath.Join(storeDir, blobsDir)))
// writeBackupConfig puts the source directory beside the
// config file.
sourceDir := filepath.Join(filepath.Dir(configPath), "source")
args := append([]string{
flagConfig, configPath, flagSkipErrors, cmdSnapshot,
cmdRestore, onlySnapshotID(t, configPath), t.TempDir(),
sourceDir,
}, testCase.missingPaths...)
code, _, stderr := runEntry(t, args...)
assert.Equal(t, testCase.want, code)
assert.Contains(t, stderr, "1 file(s) failed to restore")
})
}
}
@@ -0,0 +1,30 @@
package cli //nolint:testpackage // shares runEntry and the hermetic config helpers
import (
"testing"
"github.com/stretchr/testify/assert"
)
// TestEntrySnapshotCreateSkipErrorsExitsNonZeroWhenItSkips backs up,
// under --skip-errors, a source directory that does not exist, which the
// walk skips. The command exits non-zero and reports the backup as
// completed, not as a failed snapshot creation. See
// https://git.eeqj.de/sneak/vaultik/issues/310.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotCreateSkipErrorsExitsNonZeroWhenItSkips(t *testing.T) {
// writeBackupConfig would create the source directory.
configPath := writeHermeticPruneConfig(t, false)
code, stdout, stderr := runEntry(t,
flagConfig, configPath, "--skip-errors", cmdSnapshot, cmdCreate)
assert.Equal(t, 1, code)
assert.Contains(t, stdout,
"Files: 1 examined, 0 backed up, 0 unchanged, 1 failed.")
assert.NotContains(t, stdout, "Finished successfully.")
assert.Contains(t, stderr,
"Backup completed, but 1 file(s) could not be read")
assert.NotContains(t, stderr, "Snapshot creation failed")
}
+4 -6
View File
@@ -13,10 +13,9 @@ func NewPruneCommand() *cobra.Command {
cmd := &cobra.Command{
Use: "prune",
Short: "Tidy local database and remote storage",
Long: `Removes orphaned data from the local index database. From the
backup destination store it removes unreferenced blobs, directories
under metadata/ with no manifest, and .partial objects left by killed
uploads.
Long: `Removes orphaned data from the local index database, and
unreferenced blobs and directories under metadata/ with no manifest
from the backup destination store.
Local cleanup drops incomplete snapshots and any files, chunks, or
blobs no longer referenced by a completed snapshot. Remote cleanup
@@ -31,8 +30,7 @@ blob keeps the time of its first upload, so it can be deleted.
Remote cleanup also deletes each directory under metadata/ with no
manifest, as a backup that stopped before its manifest leaves, once
all its objects are older than prune_grace_period and its snapshot is
not in the local index. It also deletes each .partial object under
metadata/ or blobs/ once it is older than prune_grace_period.
not in the local index.
Snapshot create --prune runs the same cleanup automatically; this
command is the manual entry point for the same work (e.g. after a
+1 -2
View File
@@ -75,8 +75,7 @@ func newRemoteInfoCommand() *cobra.Command {
- Size of all snapshot metadata (per snapshot and total)
- Count and total size of all blobs
- Count and size of referenced blobs (from all manifests)
- Count and size of orphaned blobs (not referenced by any manifest)
- Count and size of .partial objects left by killed uploads`,
- Count and size of orphaned blobs (not referenced by any manifest)`,
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error {
// Use unified config resolution
+4 -18
View File
@@ -32,8 +32,6 @@ var rootFlags RootFlags
// It sets up the command structure, global flags, and adds all subcommands.
// This is the main entry point for the CLI command hierarchy.
func NewRootCommand() *cobra.Command {
var showVersion bool
cmd := &cobra.Command{
Use: "vaultik",
Short: "Secure incremental backup tool with asymmetric encryption",
@@ -41,27 +39,15 @@ func NewRootCommand() *cobra.Command {
public keys and uploads to S3-compatible storage. No private keys are needed
on the source system.`,
SilenceUsage: true,
// Bare 'vaultik' (no subcommand): print help, or with --version
// the report of the version command. The banner is printed once
// at process startup by Entry, before cobra parses arguments, so
// it appears even when cobra rejects args (e.g. "requires at
// least 2 arg(s)") and on --help.
// Bare 'vaultik' (no subcommand): print help. The banner is
// printed once at process startup by Entry, before cobra
// parses arguments, so it appears even when cobra rejects
// args (e.g. "requires at least 2 arg(s)") and on --help.
Run: func(cmd *cobra.Command, _ []string) {
if showVersion {
writeVersion(commandUI(cmd))
return
}
_ = cmd.Help()
},
}
// cobra's own --version prints a template, not the version report,
// so vaultik defines the flag itself.
cmd.Flags().BoolVar(&showVersion, "version", false,
"Print version information (same as 'vaultik version')")
// Add global flags
cmd.PersistentFlags().StringVar(&rootFlags.ConfigPath, "config", "",
"Path to config file (default: $VAULTIK_CONFIG or platform config dir)")
+7 -5
View File
@@ -138,11 +138,13 @@ on Linux), then /etc/vaultik/config.yml.`,
cmd.Flags().BoolVar(&opts.Cron, "cron", false,
"Run in cron mode (silent unless warning or error)")
cmd.Flags().BoolVar(&opts.Prune, "prune", false,
"After backup, drop older snapshots of the same name, then delete "+
"from the destination store what prune deletes there (not "+
"after a backup that skipped unreadable files, found a "+
"configured path empty, or did not find a mount point an "+
"earlier snapshot found)")
"After backup, drop older snapshots of the same name, remove "+
"orphaned blobs older than prune_grace_period, and delete each "+
"directory under metadata/ with no manifest once all its "+
"objects are older than prune_grace_period and its snapshot is "+
"not in the local index (not after a backup that skipped "+
"unreadable files, found a configured path empty, or did not "+
"find a mount point an earlier snapshot found)")
cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "",
"With --prune: keep snapshots newer than this duration "+
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+
-24
View File
@@ -86,30 +86,6 @@ func TestVersionCommandReportLines(t *testing.T) {
}
}
// TestVersionFlagPrintsVersionReport checks that `vaultik --version`
// prints exactly the report of `vaultik version`.
//
//nolint:paralleltest // executes commands that read the global rootFlags
func TestVersionFlagPrintsVersionReport(t *testing.T) {
cmd := cli.NewRootCommand()
var out bytes.Buffer
cmd.SetOut(&out)
cmd.SetErr(&out)
cmd.SetArgs([]string{"--version"})
err := cmd.Execute()
if err != nil {
t.Fatalf("vaultik --version failed: %v", err)
}
want := runVersionCommand(t)
if out.String() != want {
t.Errorf("vaultik --version printed:\n%s\nwant:\n%s", out.String(), want)
}
}
// TestVersionCommandFlagsDevelopmentBuild is the regression test for
// the thing this command exists to prevent: a build that is not a
// release must say so. The notice used to be gated on the version
+1 -2
View File
@@ -206,8 +206,7 @@ type Config struct {
// directory. A backup writes its manifest only once it has uploaded
// all its blobs and its db.zst.age, so a younger blob or directory may
// belong to a backup still running, on this host or another sharing
// the destination store. A prune also deletes a `.partial` object left
// by a killed upload only once it is this old.
// the destination store.
PruneGracePeriod time.Duration `yaml:"prune_grace_period"`
// StorageURL specifies the storage backend using a URL format.
-35
View File
@@ -4,10 +4,8 @@ package snapshot
import (
"os"
"path/filepath"
"syscall"
"testing"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@@ -40,36 +38,3 @@ func TestDeviceNumberFromStat(t *testing.T) {
// /dev is a different filesystem from the temp dir on Linux, macOS and in Docker.
assert.NotEqual(t, deviceOf(dir), deviceOf("/dev"))
}
// fileInfoWithStat is a file's info whose Sys returns stat.
type fileInfoWithStat struct {
os.FileInfo
stat *syscall.Stat_t
}
func (i fileInfoWithStat) Sys() any { return i.stat }
// The in-memory test filesystem has no device numbers. The pinned afero
// gives no stat; its unreleased master gives a stat holding only uid, gid
// and size. See https://git.eeqj.de/sneak/vaultik/issues/462.
func TestDeviceNumberFromStatInMemory(t *testing.T) {
t.Parallel()
fs := afero.NewMemMapFs()
require.NoError(t, afero.WriteFile(fs, "/file.txt", []byte("data"), 0o600))
info, err := fs.Stat("/file.txt")
require.NoError(t, err)
_, ok := deviceNumberFromStat(info)
assert.False(t, ok, "device number read from a file info with no stat")
zeroed := fileInfoWithStat{
FileInfo: info,
stat: &syscall.Stat_t{Uid: 1000, Gid: 1000, Size: 4},
}
_, ok = deviceNumberFromStat(zeroed)
assert.False(t, ok, "device number read from a stat with no device or inode number")
}
+3 -4
View File
@@ -1438,12 +1438,11 @@ func (s *Scanner) deviceNumberAt(path string) (uint64, bool) {
}
// deviceNumberFromStat returns the number of the device that holds the
// file info describes, and false when info carries no stat from a real
// filesystem. The in-memory test filesystem gives either no stat or one
// whose device and inode numbers are both zero, which no real file has.
// file info describes, and false when info carries no stat, as from the
// in-memory test filesystem.
func deviceNumberFromStat(info os.FileInfo) (uint64, bool) {
stat, ok := info.Sys().(*syscall.Stat_t)
if !ok || (stat.Dev == 0 && stat.Ino == 0) {
if !ok {
return 0, false
}
+4 -9
View File
@@ -118,22 +118,17 @@ func ShortHostname(hostname string) string {
// hostname_timestamp if name is empty. The timestamp is in whole seconds.
// If the local index already has a snapshot with that ID, from a run of the
// same name that started in the same second, it waits a second and takes a
// new timestamp. The snapshot's start time is the moment in its ID, so a
// later run of one name always has a later start time.
// new timestamp.
func (sm *SnapshotManager) CreateSnapshotWithName(
ctx context.Context, hostname, name, version, gitRevision string,
) (string, error) {
shortHostname := ShortHostname(hostname)
var (
snapshotID string
startedAt time.Time
)
var snapshotID string
for {
// Build snapshot ID with optional name
startedAt = time.Now().UTC()
timestamp := startedAt.Format("2006-01-02T15:04:05Z")
timestamp := time.Now().UTC().Format("2006-01-02T15:04:05Z")
if name != "" {
snapshotID = fmt.Sprintf("%s_%s_%s", shortHostname, name, timestamp)
@@ -158,7 +153,7 @@ func (sm *SnapshotManager) CreateSnapshotWithName(
Hostname: types.Hostname(hostname),
VaultikVersion: types.Version(version),
VaultikGitRevision: types.GitRevision(gitRevision),
StartedAt: startedAt,
StartedAt: time.Now().UTC(),
CompletedAt: nil, // Not completed yet
FileCount: 0,
ChunkCount: 0,
-52
View File
@@ -492,55 +492,3 @@ func TestCreateSnapshotWithNameTwiceBackToBack(t *testing.T) {
t.Fatalf("both creates returned snapshot ID %s", first)
}
}
// Two creates of one snapshot name back to back list newest first. The
// start time is the moment in the ID, so the two cannot share a start
// time even when the clock crosses a second while the first is created.
// See https://git.eeqj.de/sneak/vaultik/issues/482.
func TestCreateSnapshotWithNameTwiceBackToBackListsNewestFirst(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
const created = 2
ctx := context.Background()
db, err := database.New(ctx, filepath.Join(t.TempDir(), "index.sqlite"))
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
defer func() { _ = db.Close() }()
repos := database.NewRepositories(db)
sm := &SnapshotManager{repos: repos}
first, err := sm.CreateSnapshotWithName(ctx, "test-host", "data", "v", "g")
if err != nil {
t.Fatalf("first create failed: %v", err)
}
second, err := sm.CreateSnapshotWithName(ctx, "test-host", "data", "v", "g")
if err != nil {
t.Fatalf("second create failed: %v", err)
}
snapshots, err := repos.Snapshots.ListRecent(ctx, created)
if err != nil {
t.Fatalf("listing snapshots failed: %v", err)
}
if len(snapshots) != created {
t.Fatalf("listed %d snapshots, want %d", len(snapshots), created)
}
if !snapshots[0].StartedAt.After(snapshots[1].StartedAt) {
t.Fatalf("start times %v and %v are not in order",
snapshots[1].StartedAt, snapshots[0].StartedAt)
}
if snapshots[0].ID.String() != second || snapshots[1].ID.String() != first {
t.Fatalf("listed %s, %s; want %s, %s",
snapshots[0].ID, snapshots[1].ID, second, first)
}
}
+3 -5
View File
@@ -200,11 +200,9 @@ func (f *Storer) ListStream(
return f.inner.ListStream(ctx, prefix)
}
// ListPartialUploads delegates unchanged.
func (f *Storer) ListPartialUploads(
ctx context.Context, prefix string,
) ([]storage.ObjectInfo, error) {
return f.inner.ListPartialUploads(ctx, prefix)
// DeletePartialUploads delegates unchanged.
func (f *Storer) DeletePartialUploads(ctx context.Context, prefix string) error {
return f.inner.DeletePartialUploads(ctx, prefix)
}
// Info delegates unchanged.
+13 -26
View File
@@ -56,8 +56,8 @@ const storageDirPerm = 0o755
// truncated object at the key a later run would Stat and trust as a complete
// blob. The rclone backend's upload does the same on remotes with a
// server-side move. List and ListStream skip these files, so a leftover from
// an interrupted write is never listed or trusted as a blob; prune deletes
// it once it is older than the prune grace period.
// an interrupted write is never listed or trusted as a blob; it is otherwise
// harmless.
const tempSuffix = ".partial"
// Put stores data at the specified key.
@@ -248,24 +248,22 @@ func (f *FileStorer) ListStream(ctx context.Context, prefix string) <-chan Objec
return ch
}
// ListPartialUploads returns each file under prefix whose name ends in
// DeletePartialUploads removes every file under prefix whose name ends in
// tempSuffix. Like List, it reads through symlinked directories and fails
// on one whose target is missing. A missing prefix has none.
func (f *FileStorer) ListPartialUploads(
ctx context.Context, prefix string,
) ([]ObjectInfo, error) {
// on one whose target is missing. A missing prefix has none to remove.
func (f *FileStorer) DeletePartialUploads(ctx context.Context, prefix string) error {
basePath := f.fullPath(prefix)
exists, err := f.prefixExists(prefix)
if err != nil {
return nil, fmt.Errorf("checking path: %w", err)
return fmt.Errorf("checking path: %w", err)
}
if !exists {
return nil, nil
return nil
}
var partial []ObjectInfo
err = f.walk(f.fullPath(prefix), func(path string, info os.FileInfo, err error) error {
err = f.walk(basePath, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
@@ -278,24 +276,13 @@ func (f *FileStorer) ListPartialUploads(
return nil
}
relPath, err := filepath.Rel(f.basePath, path)
if err != nil {
return fmt.Errorf("computing relative path: %w", err)
}
partial = append(partial, ObjectInfo{
Key: strings.ReplaceAll(relPath, string(filepath.Separator), "/"),
Size: info.Size(),
ModTime: info.ModTime(),
})
return nil
return f.fs.Remove(path)
})
if err != nil {
return nil, fmt.Errorf("walking directory: %w", err)
return fmt.Errorf("walking directory: %w", err)
}
return partial, nil
return nil
}
// Info returns human-readable storage location information.
+8 -32
View File
@@ -121,10 +121,9 @@ func TestFileStorer_ListSkipsPartialFiles(t *testing.T) {
}
}
// TestFileStorer_ListPartialUploads checks that a leftover temp file is
// listed with its size and time, that the object at the real key is not,
// and that Delete removes the leftover by the listed key.
func TestFileStorer_ListPartialUploads(t *testing.T) {
// TestFileStorer_DeletePartialUploads checks that a leftover temp file is
// removed and the object at the real key is kept.
func TestFileStorer_DeletePartialUploads(t *testing.T) {
t.Parallel()
base := t.TempDir()
@@ -134,46 +133,23 @@ func TestFileStorer_ListPartialUploads(t *testing.T) {
t.Fatalf("NewFileStorer: %v", err)
}
checkListPartialUploads(t, f, base)
}
// checkListPartialUploads stores an object and writes a leftover temp
// file next to it in the directory base that s stores into. It checks
// that ListPartialUploads lists only the leftover, with its size and a
// time, and that Delete removes it by the listed key and keeps the
// object.
func checkListPartialUploads(t *testing.T, s storage.Storer, base string) {
t.Helper()
ctx := context.Background()
err := s.Put(ctx, testBlobKey, strings.NewReader("blob-bytes"))
err = f.Put(ctx, testBlobKey, strings.NewReader("blob-bytes"))
if err != nil {
t.Fatalf("Put: %v", err)
}
leftoverKey := testBlobKey + "-123456.partial"
leftover := filepath.Join(base, leftoverKey)
leftover := filepath.Join(base, testBlobKey+"-123456.partial")
err = os.WriteFile(leftover, []byte("half"), 0o600)
if err != nil {
t.Fatalf("writing leftover temp file: %v", err)
}
partial, err := s.ListPartialUploads(ctx, "blobs/")
err = f.DeletePartialUploads(ctx, "blobs/")
if err != nil {
t.Fatalf("ListPartialUploads: %v", err)
}
if len(partial) != 1 || partial[0].Key != leftoverKey ||
partial[0].Size != int64(len("half")) || partial[0].ModTime.IsZero() {
t.Fatalf("ListPartialUploads = %+v, want only %s, 4 bytes, "+
"with a time", partial, leftoverKey)
}
err = s.Delete(ctx, partial[0].Key)
if err != nil {
t.Fatalf("Delete: %v", err)
t.Fatalf("DeletePartialUploads: %v", err)
}
_, err = os.Stat(leftover)
@@ -181,7 +157,7 @@ func checkListPartialUploads(t *testing.T, s storage.Storer, base string) {
t.Errorf("leftover temp file was not removed: %v", err)
}
_, err = s.Stat(ctx, testBlobKey)
_, err = f.Stat(ctx, testBlobKey)
if err != nil {
t.Errorf("Stat of the real key: %v", err)
}
+18 -13
View File
@@ -239,11 +239,11 @@ func TestFileStorerListFailsOnSymlinkLoop(t *testing.T) {
}
}
// TestFileStorerListPartialUploadsThroughSymlinkedDirectory checks that
// ListPartialUploads, which prune and remote nuke use, lists a leftover
// temp file behind a symlinked metadata/ or blobs/ directory under its
// key, and leaves out the object at the real key.
func TestFileStorerListPartialUploadsThroughSymlinkedDirectory(t *testing.T) {
// TestFileStorerDeletePartialUploadsThroughSymlinkedDirectory checks that
// DeletePartialUploads, which remote nuke uses, removes a leftover temp
// file behind a symlinked metadata/ or blobs/ directory and keeps the
// object at the real key.
func TestFileStorerDeletePartialUploadsThroughSymlinkedDirectory(t *testing.T) {
t.Parallel()
ctx := context.Background()
@@ -265,24 +265,29 @@ func TestFileStorerListPartialUploadsThroughSymlinkedDirectory(t *testing.T) {
t.Fatalf("Put: %v", err)
}
leftoverKey := key + "-123456.partial"
leftover := filepath.Join(base, key+"-123456.partial")
err = os.WriteFile(filepath.Join(base, leftoverKey),
[]byte("half"), 0o600)
err = os.WriteFile(leftover, []byte("half"), 0o600)
if err != nil {
t.Fatalf("writing leftover temp file: %v", err)
}
moveBehindSymlink(t, base, dir)
partial, err := s.ListPartialUploads(ctx, dir+"/")
err = s.DeletePartialUploads(ctx, dir+"/")
if err != nil {
t.Fatalf("ListPartialUploads: %v", err)
t.Fatalf("DeletePartialUploads: %v", err)
}
if len(partial) != 1 || partial[0].Key != leftoverKey {
t.Errorf("ListPartialUploads = %+v, want only %s",
partial, leftoverKey)
// leftover runs through the link.
_, err = os.Stat(leftover)
if !os.IsNotExist(err) {
t.Errorf("leftover temp file was not removed: %v", err)
}
_, err = s.Stat(ctx, key)
if err != nil {
t.Errorf("Stat of the real key: %v", err)
}
})
}
+34 -38
View File
@@ -8,7 +8,6 @@ import (
"fmt"
"io"
"os"
"path/filepath"
"strings"
"time"
@@ -24,15 +23,6 @@ import (
// ErrRemoteNotFound is returned when an rclone remote is not configured.
var ErrRemoteNotFound = errors.New("rclone remote not found in config")
// ErrRcloneRelativeLocalPath is returned when the path on an rclone remote
// of type local is relative. rclone would resolve it against the working
// directory, so the destination would move with it.
var ErrRcloneRelativeLocalPath = errors.New(
"the path on an rclone remote of type local must be absolute")
// rcloneLocalBackend is the type of rclone's local disk backend.
const rcloneLocalBackend = "local"
// RcloneStorer implements Storer using rclone's filesystem abstraction.
// This allows vaultik to use any of rclone's 70+ supported storage providers.
type RcloneStorer struct {
@@ -63,19 +53,6 @@ func NewRcloneStorer(ctx context.Context, remote, path string) (*RcloneStorer, e
rclonePath += path
}
// A remote rclone cannot find is reported by fs.NewFs below. Only the
// backend of what fs.ParseRemote returns is needed here.
backend, _, _, _, err := fs.ParseRemote(rclonePath) //nolint:dogsled
if err == nil && backend.Name == rcloneLocalBackend && !filepath.IsAbs(path) {
example := path
if example == "" {
example = "path/to/backup"
}
return nil, fmt.Errorf("%w: use rclone://%s//%s",
ErrRcloneRelativeLocalPath, remote, example)
}
// On S3, Swift and some other remotes rclone keeps a time of its own in
// each object's metadata, which costs one request per object to read.
// rclone's use_server_modtime setting makes ModTime the time the
@@ -90,10 +67,12 @@ func NewRcloneStorer(ctx context.Context, remote, path string) (*RcloneStorer, e
// Check for remote not found error
if strings.Contains(err.Error(), "didn't find section in config file") ||
strings.Contains(err.Error(), "failed to find remote") {
return nil, fmt.Errorf("%w: %s", ErrRemoteNotFound, remote)
return nil, fmt.Errorf("%w: %q (%s)",
ErrRemoteNotFound, remote, describeRcloneConfigFile())
}
return nil, fmt.Errorf("creating rclone filesystem: %w", err)
return nil, fmt.Errorf("opening rclone remote %q (%s): %w",
remote, describeRcloneConfigFile(), err)
}
return &RcloneStorer{
@@ -256,28 +235,29 @@ func (r *RcloneStorer) ListStream(
return ch
}
// ListPartialUploads returns each object under prefix whose name ends in
// tempSuffix.
func (r *RcloneStorer) ListPartialUploads(
ctx context.Context, prefix string,
) ([]ObjectInfo, error) {
var partial []ObjectInfo
// DeletePartialUploads removes every object under prefix whose name ends
// in tempSuffix.
func (r *RcloneStorer) DeletePartialUploads(ctx context.Context, prefix string) error {
var partial []fs.Object
err := operations.ListFn(ctx, r.fsys, func(obj fs.Object) {
key := obj.Remote()
if strings.HasPrefix(key, prefix) && strings.HasSuffix(key, tempSuffix) {
partial = append(partial, ObjectInfo{
Key: key,
Size: obj.Size(),
ModTime: obj.ModTime(ctx),
})
partial = append(partial, obj)
}
})
if err != nil {
return nil, fmt.Errorf("listing objects: %w", err)
return fmt.Errorf("listing objects: %w", err)
}
return partial, nil
for _, obj := range partial {
err = obj.Remove(ctx)
if err != nil {
return fmt.Errorf("removing object: %w", err)
}
}
return nil
}
// Info returns human-readable storage location information.
@@ -346,6 +326,22 @@ func (r *RcloneStorer) upload(ctx context.Context, key string, data io.Reader) e
return nil
}
// describeRcloneConfigFile names the rclone config file for an error
// message, and says so when that file does not exist.
func describeRcloneConfigFile() string {
path := config.GetConfigPath()
if path == "" {
return "no config file"
}
_, err := os.Stat(path)
if errors.Is(err, os.ErrNotExist) {
return fmt.Sprintf("config file %q, which does not exist", path)
}
return fmt.Sprintf("config file %q", path)
}
// progressReader wraps an io.Reader to track read progress.
type progressReader struct {
reader io.Reader
+114 -87
View File
@@ -205,21 +205,45 @@ func TestRcloneStorerListSkipsPartialFiles(t *testing.T) {
}
}
// TestRcloneStorerListPartialUploads checks that a temporary file left by
// a killed upload is listed with its size and time, that the object at
// the real key is not, and that Delete removes the leftover by the listed
// key.
// TestRcloneStorerDeletePartialUploads checks that a temporary file left
// by a killed upload is removed and the object at the real key is kept.
//
//nolint:paralleltest // NewRcloneStorer installs the process-global rclone config
func TestRcloneStorerListPartialUploads(t *testing.T) {
func TestRcloneStorerDeletePartialUploads(t *testing.T) {
dir := t.TempDir()
ctx := context.Background()
s, err := storage.NewRcloneStorer(context.Background(), ":local", dir)
s, err := storage.NewRcloneStorer(ctx, ":local", dir)
if err != nil {
t.Fatalf("NewRcloneStorer: %v", err)
}
checkListPartialUploads(t, s, dir)
err = s.Put(ctx, testBlobKey, strings.NewReader("blob-bytes"))
if err != nil {
t.Fatalf("Put: %v", err)
}
leftover := filepath.Join(dir, testBlobKey+"-123456.partial")
err = os.WriteFile(leftover, []byte("half"), 0o600)
if err != nil {
t.Fatalf("writing leftover temp file: %v", err)
}
err = s.DeletePartialUploads(ctx, "blobs/")
if err != nil {
t.Fatalf("DeletePartialUploads: %v", err)
}
_, err = os.Stat(leftover)
if !os.IsNotExist(err) {
t.Errorf("leftover temp file was not removed: %v", err)
}
_, err = s.Stat(ctx, testBlobKey)
if err != nil {
t.Errorf("Stat of the real key: %v", err)
}
}
// newRcloneStorerOnWrappedLocal registers name as rclone's local backend
@@ -374,24 +398,100 @@ func TestNewRcloneStorerUnknownRemote(t *testing.T) {
}
}
// TestNewRcloneStorerReadsRcloneConfigEnv checks that a remote defined only
// in the file RCLONE_CONFIG names is found and written to, as rclone itself
// would.
func TestNewRcloneStorerReadsRcloneConfigEnv(t *testing.T) {
// setRcloneConfigEnv points RCLONE_CONFIG at path for the rest of the test.
func setRcloneConfigEnv(t *testing.T, path string) {
t.Helper()
// rclone keeps the config file path in a process-global variable, which
// NewRcloneStorer leaves pointing at this test's file.
// NewRcloneStorer leaves pointing at path.
previous := config.GetConfigPath()
t.Cleanup(func() { _ = config.SetConfigPath(previous) })
t.Setenv("RCLONE_CONFIG", path)
}
// writeRcloneConfigFile writes contents to a new rclone config file and
// returns its path.
func writeRcloneConfigFile(t *testing.T, contents string) string {
t.Helper()
confFile := filepath.Join(t.TempDir(), "rclone.conf")
err := os.WriteFile(confFile, []byte("[loc]\ntype = local\n"), 0o600)
err := os.WriteFile(confFile, []byte(contents), 0o600)
if err != nil {
t.Fatalf("writing rclone config: %v", err)
}
t.Setenv("RCLONE_CONFIG", confFile)
return confFile
}
// TestNewRcloneStorerErrorsNameRemoteAndConfigFile checks that each error
// from opening a remote names the remote and the rclone config file, and
// says when that file does not exist.
//
//nolint:paralleltest // sets RCLONE_CONFIG and the process-global rclone config
func TestNewRcloneStorerErrorsNameRemoteAndConfigFile(t *testing.T) {
const doesNotExist = "which does not exist"
missingFile := filepath.Join(t.TempDir(), "missing.conf")
emptyFile := writeRcloneConfigFile(t, "")
badRemoteFile := writeRcloneConfigFile(t,
"[bad]\ntype = vaultik-no-such-backend\n")
tests := []struct {
name string
configFile string
remote string
wantNotFound bool
wantNoSuchFile bool
}{
{"config file missing", missingFile, "dav", true, true},
{"remote not in config file", emptyFile, "dav", true, false},
{"remote fails to open", badRemoteFile, "bad", false, false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
setRcloneConfigEnv(t, tt.configFile)
_, err := storage.NewRcloneStorer(
context.Background(), tt.remote, "path")
if err == nil {
t.Fatal("NewRcloneStorer succeeded, want an error")
}
msg := err.Error()
if errors.Is(err, storage.ErrRemoteNotFound) != tt.wantNotFound {
t.Errorf("error = %v, want ErrRemoteNotFound: %v",
err, tt.wantNotFound)
}
if !strings.Contains(msg, `"`+tt.remote+`"`) {
t.Errorf("error %q does not name remote %q", msg, tt.remote)
}
if !strings.Contains(msg, tt.configFile) {
t.Errorf("error %q does not name config file %q",
msg, tt.configFile)
}
if strings.Contains(msg, doesNotExist) != tt.wantNoSuchFile {
t.Errorf("error %q, want it to say the file does not exist: %v",
msg, tt.wantNoSuchFile)
}
})
}
}
// TestNewRcloneStorerReadsRcloneConfigEnv checks that a remote defined only
// in the file RCLONE_CONFIG names is found and written to, as rclone itself
// would.
//
//nolint:paralleltest // sets RCLONE_CONFIG and the process-global rclone config
func TestNewRcloneStorerReadsRcloneConfigEnv(t *testing.T) {
setRcloneConfigEnv(t, writeRcloneConfigFile(t, "[loc]\ntype = local\n"))
dir := t.TempDir()
ctx := context.Background()
@@ -412,79 +512,6 @@ func TestNewRcloneStorerReadsRcloneConfigEnv(t *testing.T) {
}
}
// useRcloneConfig points rclone at a config file holding contents for the
// rest of the test.
func useRcloneConfig(t *testing.T, contents string) {
t.Helper()
previous := config.GetConfigPath()
t.Cleanup(func() { _ = config.SetConfigPath(previous) })
confFile := filepath.Join(t.TempDir(), "rclone.conf")
err := os.WriteFile(confFile, []byte(contents), 0o600)
if err != nil {
t.Fatalf("writing rclone config: %v", err)
}
t.Setenv("RCLONE_CONFIG", confFile)
}
// TestNewRcloneStorerRefusesRelativeLocalPath checks that a relative path
// on a remote of type local is refused, with the absolute form in the
// error, and that nothing is created under the working directory. rclone
// would otherwise resolve the path against the working directory.
//
//nolint:paralleltest // changes the working directory and the rclone config
func TestNewRcloneStorerRefusesRelativeLocalPath(t *testing.T) {
useRcloneConfig(t, "[localfs]\ntype = local\n")
workdir := t.TempDir()
t.Chdir(workdir)
for path, wantInError := range map[string]string{
"srv/backup": "rclone://localfs//srv/backup",
"": "rclone://localfs//path/to/backup",
} {
_, err := storage.NewRcloneStorer(context.Background(), "localfs", path)
if !errors.Is(err, storage.ErrRcloneRelativeLocalPath) {
t.Errorf("path %q: error = %v, want ErrRcloneRelativeLocalPath",
path, err)
continue
}
if !strings.Contains(err.Error(), wantInError) {
t.Errorf("path %q: error %q does not show %q",
path, err, wantInError)
}
}
entries, err := os.ReadDir(workdir)
if err != nil {
t.Fatalf("reading working directory: %v", err)
}
if len(entries) != 0 {
t.Errorf("working directory holds %d entries, want none", len(entries))
}
}
// TestNewRcloneStorerKeepsRelativePathOnOtherRemotes checks that a relative
// path is accepted on a remote not of type local, where rclone resolves it
// on the remote itself, as sftp does against the login directory.
//
//nolint:paralleltest // changes the process-global rclone config
func TestNewRcloneStorerKeepsRelativePathOnOtherRemotes(t *testing.T) {
useRcloneConfig(t, "[mem]\ntype = memory\n")
_, err := storage.NewRcloneStorer(context.Background(), "mem", "backups")
if err != nil {
t.Errorf("NewRcloneStorer: %v", err)
}
}
// TestRcloneListStreamOnS3MakesNoRequestPerObject checks that listing an
// rclone S3 remote takes each object's time from the listing itself. By
// default rclone reads the time it keeps in each object's metadata, one
+4 -6
View File
@@ -100,12 +100,10 @@ func (s *S3Storer) ListStream(ctx context.Context, prefix string) <-chan ObjectI
return ch
}
// ListPartialUploads finds none: S3 shows an object only once its upload
// has completed, so an upload cut off part-way leaves none.
func (s *S3Storer) ListPartialUploads(
_ context.Context, _ string,
) ([]ObjectInfo, error) {
return nil, nil
// DeletePartialUploads has nothing to remove: S3 shows an object only once
// its upload has completed, so an upload cut off part-way leaves none.
func (s *S3Storer) DeletePartialUploads(_ context.Context, _ string) error {
return nil
}
// Info returns human-readable storage location information.
+5 -6
View File
@@ -77,12 +77,11 @@ type Storer interface {
// If an error occurs during listing, the final item will have Err set.
ListStream(ctx context.Context, prefix string) <-chan ObjectInfo
// ListPartialUploads returns each object under prefix whose name ends
// in `.partial`, as a killed upload leaves it, with its key, size and
// modification time. The file and rclone
// backends' List and ListStream skip such an object; Delete removes
// it.
ListPartialUploads(ctx context.Context, prefix string) ([]ObjectInfo, error)
// DeletePartialUploads removes every object under prefix that an
// upload cut off part-way left under a temporary name ending in
// `.partial`. The file and rclone backends' List and ListStream skip
// such an object.
DeletePartialUploads(ctx context.Context, prefix string) error
// Info returns human-readable storage location information.
Info() Info
@@ -258,10 +258,8 @@ func (s *stubLister) List(_ context.Context, _ string) ([]string, error) {
return nil, errStubUnused
}
func (s *stubLister) ListPartialUploads(
_ context.Context, _ string,
) ([]storage.ObjectInfo, error) {
return nil, errStubUnused
func (s *stubLister) DeletePartialUploads(_ context.Context, _ string) error {
return errStubUnused
}
func (s *stubLister) Info() storage.Info {
+7 -41
View File
@@ -216,11 +216,6 @@ type RemoteInfoResult struct {
OrphanedBlobCount *int `json:"orphaned_blob_count"`
OrphanedBlobSize *int64 `json:"orphaned_blob_size"`
// `.partial` objects under metadata/ and blobs/ left by killed
// uploads. The blob figures above leave them out.
PartialUploadCount int `json:"partial_upload_count"`
PartialUploadSize int64 `json:"partial_upload_size"`
// Remote key of each snapshot whose manifest could not be read
UnreadableManifests []string `json:"unreadable_manifests,omitempty"`
@@ -284,16 +279,6 @@ func (v *Vaultik) RemoteInfo(jsonOutput bool) error {
return err
}
partialUploads, err := v.listPartialUploads()
if err != nil {
return err
}
for _, object := range partialUploads {
result.PartialUploadCount++
result.PartialUploadSize += object.Size
}
log.Info("Remote info complete",
"snapshots", result.TotalMetadataCount,
"total_blobs", result.TotalBlobCount,
@@ -554,12 +539,6 @@ func (v *Vaultik) printRemoteInfoTable(result *RemoteInfoResult) {
"", "", ubytes(result.TotalMetadataSize))
}
v.printRemoteBlobStorage(result)
}
// printRemoteBlobStorage renders the blob storage section of the
// human-readable remote info output.
func (v *Vaultik) printRemoteBlobStorage(result *RemoteInfoResult) {
v.stdoutf("\n=== Blob Storage ===\n")
v.stdoutf("Total blobs on remote: %s (%s)\n",
humanize.Comma(int64(result.TotalBlobCount)),
@@ -573,29 +552,16 @@ func (v *Vaultik) printRemoteBlobStorage(result *RemoteInfoResult) {
"(%d manifest(s) could not be read, "+
"%d manifest(s) under a non-conforming name skipped)\n",
len(result.UnreadableManifests), result.SkippedManifestCount)
} else {
v.stdoutf("Orphaned (unreferenced): %s (%s)\n",
humanize.Comma(int64(*result.OrphanedBlobCount)),
ubytes(*result.OrphanedBlobSize))
return
}
v.stdoutf("Partial uploads: %s (%s)\n",
humanize.Comma(int64(result.PartialUploadCount)),
ubytes(result.PartialUploadSize))
v.stdoutf("Orphaned (unreferenced): %s (%s)\n",
humanize.Comma(int64(*result.OrphanedBlobCount)),
ubytes(*result.OrphanedBlobSize))
var toRemove []string
if result.OrphanedBlobCount != nil && *result.OrphanedBlobCount > 0 {
toRemove = append(toRemove, "orphaned blobs")
}
if result.PartialUploadCount > 0 {
toRemove = append(toRemove, "partial uploads")
}
if len(toRemove) > 0 {
v.stdoutf("\nRun 'vaultik prune' to remove %s.\n",
strings.Join(toRemove, " and "))
if *result.OrphanedBlobCount > 0 {
v.stdoutf("\nRun 'vaultik prune' to remove orphaned blobs.\n")
}
}
+4 -6
View File
@@ -155,12 +155,10 @@ func (m *MockStorer) ListStream(
return ch
}
// ListPartialUploads finds none: Put stores each object under its key at
// once.
func (m *MockStorer) ListPartialUploads(
_ context.Context, _ string,
) ([]storage.ObjectInfo, error) {
return nil, nil
// DeletePartialUploads has nothing to remove: Put stores each object
// under its key at once.
func (m *MockStorer) DeletePartialUploads(_ context.Context, _ string) error {
return nil
}
func (m *MockStorer) Info() storage.Info {
+45 -212
View File
@@ -31,11 +31,6 @@ var errBlobDeleteFailed = errors.New("blob(s) could not be deleted")
var errMetadataDirDeleteFailed = errors.New(
"metadata director(ies) with no manifest could not be deleted")
// errPartialUploadDeleteFailed reports `.partial` objects prune could not
// delete.
var errPartialUploadDeleteFailed = errors.New(
"partial upload(s) could not be deleted")
// errObjectsLeft reports objects remote nuke still finds on the backup
// destination store after deleting.
var errObjectsLeft = errors.New(
@@ -143,9 +138,9 @@ func (v *Vaultik) emptyDestinationStore() error {
// that fails is logged and the others still run; that object then
// shows up in the listing afterwards.
func (v *Vaultik) deleteEverythingUnder(prefix string) ([]string, error) {
keys, err := v.listEverythingUnder(prefix)
keys, err := v.Storage.List(v.ctx, prefix)
if err != nil {
return nil, err
return nil, fmt.Errorf("listing %s: %w", prefix, err)
}
for _, key := range keys {
@@ -155,47 +150,19 @@ func (v *Vaultik) deleteEverythingUnder(prefix string) ([]string, error) {
}
}
return v.listEverythingUnder(prefix)
}
// The file and rclone listings skip `.partial` objects, so the
// deletes above never reach them.
err = v.Storage.DeletePartialUploads(v.ctx, prefix)
if err != nil {
return nil, fmt.Errorf("deleting partial uploads: %w", err)
}
// listEverythingUnder returns the key of every object under prefix,
// including the `.partial` objects left by killed uploads, which the file
// and rclone backends' List skips.
func (v *Vaultik) listEverythingUnder(prefix string) ([]string, error) {
keys, err := v.Storage.List(v.ctx, prefix)
left, err := v.Storage.List(v.ctx, prefix)
if err != nil {
return nil, fmt.Errorf("listing %s: %w", prefix, err)
}
partialUploads, err := v.Storage.ListPartialUploads(v.ctx, prefix)
if err != nil {
return nil, fmt.Errorf("listing partial uploads under %s: %w",
prefix, err)
}
for _, object := range partialUploads {
keys = append(keys, object.Key)
}
return keys, nil
}
// listPartialUploads returns the `.partial` objects under metadata/ and
// blobs/ left by killed uploads.
func (v *Vaultik) listPartialUploads() ([]storage.ObjectInfo, error) {
var partialUploads []storage.ObjectInfo
for _, prefix := range []string{"metadata/", "blobs/"} {
objects, err := v.Storage.ListPartialUploads(v.ctx, prefix)
if err != nil {
return nil, fmt.Errorf("listing partial uploads under %s: %w",
prefix, err)
}
partialUploads = append(partialUploads, objects...)
}
return partialUploads, nil
return left, nil
}
// PruneBlobsResult contains the result of a blob prune operation
@@ -215,19 +182,11 @@ type PruneBlobsResult struct {
// not delete.
MetadataDirsDeleted int `json:"metadata_dirs_deleted"`
MetadataDirsFailed int `json:"metadata_dirs_failed,omitempty"`
// PartialUploadsDeleted counts the `.partial` objects prune deleted,
// and PartialUploadsFailed those it could not delete.
// PartialUploadsTooNew counts those kept for being newer than the
// prune grace period, or of unknown age.
PartialUploadsDeleted int `json:"partial_uploads_deleted"`
PartialUploadsFailed int `json:"partial_uploads_failed,omitempty"`
PartialUploadsTooNew int `json:"partial_uploads_too_new"`
}
// Prune removes orphaned data from the local index database AND, from the
// backup destination store, unreferenced blobs, the directories under
// metadata/ with no manifest and the `.partial` objects left by killed
// uploads. This is the single user-facing prune entry
// backup destination store, unreferenced blobs and the directories under
// metadata/ with no manifest. This is the single user-facing prune entry
// point — the split between local and remote cleanup is an
// implementation detail. Calling code should prefer this method over
// PruneDatabase or PruneBlobs individually unless it specifically wants
@@ -255,9 +214,8 @@ func (v *Vaultik) Prune(opts *PruneOptions) error {
return v.PruneBlobs(opts)
}
// PruneBlobs removes unreferenced blobs from storage, each directory
// under metadata/ that a backup left with no manifest, and each
// `.partial` object left by a killed upload.
// PruneBlobs removes unreferenced blobs from storage, and each directory
// under metadata/ that a backup left with no manifest.
func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
log.Info("Starting prune operation")
@@ -284,21 +242,14 @@ func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
return err
}
partialUploads, partialTooNew, err := v.findOldPartialUploads()
if err != nil {
return err
}
result := &PruneBlobsResult{
BlobsFound: len(unreferencedBlobs),
BlobsTooNew: tooNew,
PartialUploadsTooNew: partialTooNew,
BlobsFound: len(unreferencedBlobs),
BlobsTooNew: tooNew,
}
v.reportTooNew(result, opts)
v.reportBlobsTooNew(tooNew, opts)
if len(unreferencedBlobs) == 0 && len(dirsWithoutManifest) == 0 &&
len(partialUploads) == 0 {
if len(unreferencedBlobs) == 0 && len(dirsWithoutManifest) == 0 {
log.Info("No unreferenced blobs found")
if opts.JSON {
@@ -312,17 +263,15 @@ func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
log.Info("Found unreferenced blobs",
"count", len(unreferencedBlobs), "total_size", ubytes(totalSize),
"metadata_dirs_without_manifest", len(dirsWithoutManifest),
"partial_uploads", len(partialUploads))
"metadata_dirs_without_manifest", len(dirsWithoutManifest))
if !v.confirmPrune(opts, len(unreferencedBlobs), totalSize,
dirsWithoutManifest, partialUploads) {
dirsWithoutManifest) {
return ErrNotConfirmed
}
v.deleteUnreferencedBlobs(unreferencedBlobs, allBlobs, result)
v.deleteMetadataDirs(dirsWithoutManifest, result)
v.deletePartialUploads(partialUploads, result)
if opts.JSON {
err = v.outputPruneBlobsJSON(result)
@@ -330,16 +279,9 @@ func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
return err
}
} else {
v.reportPruneDeletions(result, len(dirsWithoutManifest),
len(partialUploads))
v.reportPruneDeletions(result, len(dirsWithoutManifest))
}
return pruneDeleteFailures(result)
}
// pruneDeleteFailures returns an error for each kind of object prune
// could not delete, or nil when every delete succeeded.
func pruneDeleteFailures(result *PruneBlobsResult) error {
var failures []error
if result.BlobsFailed > 0 {
@@ -352,41 +294,26 @@ func pruneDeleteFailures(result *PruneBlobsResult) error {
result.MetadataDirsFailed, errMetadataDirDeleteFailed))
}
if result.PartialUploadsFailed > 0 {
failures = append(failures, fmt.Errorf("%d %w",
result.PartialUploadsFailed, errPartialUploadDeleteFailed))
}
return errors.Join(failures...)
}
// confirmPrune states what prune found to delete, unreferenced blobs,
// confirmPrune states what prune found to delete, unreferenced blobs and
// directories under metadata/ with no manifest, naming each directory by
// its remote key, and `.partial` objects, and, unless --force is given,
// asks the operator to confirm. Under --json it prints nothing and does
// not ask. It reports whether to go ahead.
// its remote key, and, unless --force is given, asks the operator to
// confirm. Under --json it prints nothing and does not ask. It reports
// whether to go ahead.
func (v *Vaultik) confirmPrune(
opts *PruneOptions, blobCount int, blobsSize int64, dirs []string,
partialUploads []storage.ObjectInfo,
) bool {
if opts.JSON {
return true
}
var partialSize int64
for _, object := range partialUploads {
partialSize += object.Size
}
dirCount := len(dirs)
partialCount := len(partialUploads)
blobsFound := fmt.Sprintf("Found %d unreferenced blob(s) totaling %s.",
blobCount, ubytes(blobsSize))
dirsFound := fmt.Sprintf("Found %d metadata director(ies) left by "+
"backups that stopped before their manifest:", dirCount)
partialFound := fmt.Sprintf("Found %d partial upload(s) totaling %s, "+
"left by uploads cut off part-way.", partialCount, ubytes(partialSize))
if opts.Force {
if blobCount > 0 {
@@ -401,10 +328,6 @@ func (v *Vaultik) confirmPrune(
}
}
if partialCount > 0 {
v.UI.Infof("%s", partialFound)
}
return true
}
@@ -423,13 +346,16 @@ func (v *Vaultik) confirmPrune(
}
}
if partialCount > 0 {
v.stdoutf("%s\n", partialFound)
switch {
case dirCount == 0:
v.stdoutf("\nDelete %d unreferenced blob(s)? [y/N] ", blobCount)
case blobCount == 0:
v.stdoutf("\nDelete %d metadata director(ies)? [y/N] ", dirCount)
default:
v.stdoutf("\nDelete %d unreferenced blob(s) and %d metadata "+
"director(ies)? [y/N] ", blobCount, dirCount)
}
v.stdoutf("\nDelete %s? [y/N] ",
pruneTargets(blobCount, dirCount, partialCount))
var confirm string
// A failure to read the answer means no confirmation.
@@ -443,36 +369,9 @@ func (v *Vaultik) confirmPrune(
return true
}
// pruneTargets names what the prune prompt offers to delete, such as
// "2 unreferenced blob(s) and 1 partial upload(s)", leaving out each kind
// with a count of zero.
func pruneTargets(blobCount, dirCount, partialCount int) string {
var targets []string
if blobCount > 0 {
targets = append(targets,
fmt.Sprintf("%d unreferenced blob(s)", blobCount))
}
if dirCount > 0 {
targets = append(targets,
fmt.Sprintf("%d metadata director(ies)", dirCount))
}
if partialCount > 0 {
targets = append(targets,
fmt.Sprintf("%d partial upload(s)", partialCount))
}
return strings.Join(targets, " and ")
}
// reportPruneDeletions says how many blobs, metadata directories and
// partial uploads prune deleted, naming each kind only when it found some
// to delete.
func (v *Vaultik) reportPruneDeletions(
result *PruneBlobsResult, dirCount, partialCount int,
) {
// reportPruneDeletions says how many blobs and metadata directories
// prune deleted, naming each kind only when it found some to delete.
func (v *Vaultik) reportPruneDeletions(result *PruneBlobsResult, dirCount int) {
if result.BlobsFound > 0 {
v.UI.Infof("Deleted %d blob(s) totaling %s.",
result.BlobsDeleted, ubytes(result.BytesFreed))
@@ -482,11 +381,6 @@ func (v *Vaultik) reportPruneDeletions(
v.UI.Infof("Deleted %d metadata director(ies) with no manifest.",
result.MetadataDirsDeleted)
}
if partialCount > 0 {
v.UI.Infof("Deleted %d partial upload(s).",
result.PartialUploadsDeleted)
}
}
// listMetadataObjects lists every object under metadata/ on the
@@ -717,79 +611,18 @@ func (v *Vaultik) deleteMetadataDirs(
}
}
// reportTooNew says how many unreferenced blobs and partial uploads prune
// kept for being newer than the grace period or of unknown age. Under
// --json the document carries the counts instead.
func (v *Vaultik) reportTooNew(result *PruneBlobsResult, opts *PruneOptions) {
if opts.JSON {
// reportBlobsTooNew says how many unreferenced blobs prune kept for being
// newer than the grace period or of unknown age. Under --json the
// document carries the count instead.
func (v *Vaultik) reportBlobsTooNew(tooNew int, opts *PruneOptions) {
if tooNew == 0 || opts.JSON {
return
}
if result.BlobsTooNew > 0 {
v.UI.Infof("Kept %d unreferenced blob(s) newer than "+
"prune_grace_period (%s), or of unknown age: a backup still "+
"running may need them.",
result.BlobsTooNew, v.Config.PruneGracePeriod)
}
if result.PartialUploadsTooNew > 0 {
v.UI.Infof("Kept %d partial upload(s) newer than "+
"prune_grace_period (%s), or of unknown age: an upload may "+
"still be writing them.",
result.PartialUploadsTooNew, v.Config.PruneGracePeriod)
}
}
// findOldPartialUploads returns the `.partial` objects under metadata/
// and blobs/ that are older than the prune grace period. It also returns
// how many it left out for being newer than that, or of unknown age
// because the listing gave no time for them.
func (v *Vaultik) findOldPartialUploads() ([]storage.ObjectInfo, int, error) {
partialUploads, err := v.listPartialUploads()
if err != nil {
return nil, 0, err
}
cutoff := time.Now().UTC().Add(-v.Config.PruneGracePeriod)
var (
old []storage.ObjectInfo
tooNew int
)
for _, object := range partialUploads {
if object.ModTime.IsZero() || object.ModTime.After(cutoff) {
tooNew++
continue
}
old = append(old, object)
}
return old, tooNew, nil
}
// deletePartialUploads deletes the given `.partial` objects and counts
// in the result those deleted and those it could not delete.
func (v *Vaultik) deletePartialUploads(
partialUploads []storage.ObjectInfo, result *PruneBlobsResult,
) {
for _, object := range partialUploads {
err := v.Storage.Delete(v.ctx, object.Key)
if err != nil {
log.Error("Failed to delete partial upload",
"key", object.Key, "error", err)
result.PartialUploadsFailed++
continue
}
log.Info("Deleted partial upload", "key", object.Key)
result.PartialUploadsDeleted++
}
v.UI.Infof("Kept %d unreferenced blob(s) newer than "+
"prune_grace_period (%s), or of unknown age: a backup still "+
"running may need them.",
tooNew, v.Config.PruneGracePeriod)
}
// deleteUnreferencedBlobs deletes the given blobs from storage and
@@ -1,157 +0,0 @@
package vaultik_test
import (
"bytes"
"context"
"encoding/json"
"os"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/storage/faultstore"
"sneak.berlin/go/vaultik/internal/ui"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// partialUploadGracePeriod is the prune grace period of the vaultik
// newPartialUploadStore returns.
const partialUploadGracePeriod = time.Hour
// newPartialUploadStore returns a vaultik on a file:// destination
// holding `.partial` files left by killed uploads: one under blobs/
// and one under metadata/ older than the prune grace period, and one
// under blobs/ newer than that. It also returns the buffer the vaultik
// writes its output to, the paths of the old files and the path of the
// new one.
func newPartialUploadStore(
t *testing.T,
) (*vaultik.Vaultik, *bytes.Buffer, []string, string) {
t.Helper()
storeDir := t.TempDir()
store, err := storage.NewFileStorer(storeDir)
require.NoError(t, err)
hash := testBlobHashA
oldFiles := []string{
filepath.Join(storeDir, "blobs", hash[:2], hash[2:4],
hash+"-123456.partial"),
filepath.Join(storeDir, metadataObjectKey(
"testhost_home_2026-01-01T00:00:00Z", "db.zst.age-123456.partial")),
}
newFile := filepath.Join(storeDir, "blobs", hash[:2], hash[2:4],
hash+"-654321.partial")
beforeGracePeriod := time.Now().Add(-partialUploadGracePeriod - time.Minute)
for _, path := range append([]string{newFile}, oldFiles...) {
require.NoError(t, os.MkdirAll(filepath.Dir(path), 0o750))
require.NoError(t, os.WriteFile(path, []byte("half an upload"), 0o600))
}
for _, path := range oldFiles {
require.NoError(t, os.Chtimes(path, beforeGracePeriod, beforeGracePeriod))
}
stdout := &bytes.Buffer{}
v := &vaultik.Vaultik{
Config: &config.Config{PruneGracePeriod: partialUploadGracePeriod},
Storage: store,
Stdout: stdout,
UI: ui.NewWithColor(stdout, false),
}
v.SetContext(context.Background())
return v, stdout, oldFiles, newFile
}
// TestPruneDeletesPartialUploadsOlderThanGracePeriod checks that prune
// deletes a `.partial` file under blobs/ or metadata/ left by a killed
// upload once it is older than the grace period, keeps a newer one,
// and counts both in the --json document.
func TestPruneDeletesPartialUploadsOlderThanGracePeriod(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
v, stdout, oldFiles, newFile := newPartialUploadStore(t)
require.NoError(t, v.PruneBlobs(&vaultik.PruneOptions{JSON: true}))
for _, path := range oldFiles {
_, err := os.Stat(path)
require.ErrorIs(t, err, os.ErrNotExist, "%s must be deleted", path)
}
_, err := os.Stat(newFile)
require.NoError(t, err, "a partial upload newer than the grace period "+
"must be kept")
var result vaultik.PruneBlobsResult
require.NoError(t, json.Unmarshal(stdout.Bytes(), &result))
assert.Equal(t, 2, result.PartialUploadsDeleted)
assert.Equal(t, 1, result.PartialUploadsTooNew)
}
// TestPruneFailedPartialUploadDeleteReturnsError checks that prune
// returns an error, and counts the failures in the --json document, when
// the `.partial` files it found cannot be deleted.
func TestPruneFailedPartialUploadDeleteReturnsError(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
v, stdout, oldFiles, _ := newPartialUploadStore(t)
store := faultstore.New(v.Storage)
store.OnDelete = func(string) bool { return true }
v.Storage = store
err := v.PruneBlobs(&vaultik.PruneOptions{JSON: true})
require.ErrorContains(t, err, "2 partial upload(s) could not be deleted")
for _, path := range oldFiles {
_, statErr := os.Stat(path)
require.NoError(t, statErr, "%s must still exist", path)
}
var result vaultik.PruneBlobsResult
require.NoError(t, json.Unmarshal(stdout.Bytes(), &result),
"stdout must hold the JSON document, got:\n%s", stdout.String())
assert.Equal(t, 2, result.PartialUploadsFailed)
assert.Equal(t, 0, result.PartialUploadsDeleted)
}
// TestRemoteInfoCountsPartialUploads checks that remote info counts
// every `.partial` file under blobs/ and metadata/, whatever its age, and
// advises running prune.
func TestRemoteInfoCountsPartialUploads(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
v, stdout, _, _ := newPartialUploadStore(t)
require.NoError(t, v.RemoteInfo(true))
var result vaultik.RemoteInfoResult
require.NoError(t, json.Unmarshal(stdout.Bytes(), &result))
assert.Equal(t, 3, result.PartialUploadCount)
assert.Equal(t, int64(3*len("half an upload")), result.PartialUploadSize)
assert.Equal(t, 0, result.TotalBlobCount)
stdout.Reset()
require.NoError(t, v.RemoteInfo(false))
text := stdout.String()
assert.Contains(t, text, "Partial uploads: 3 (")
assert.Contains(t, text, "Run 'vaultik prune' to remove partial uploads.")
}
+4 -6
View File
@@ -136,12 +136,10 @@ func (s *testStorer) ListStream(
return ch
}
// ListPartialUploads finds none: Put stores each object under its key at
// once.
func (s *testStorer) ListPartialUploads(
_ context.Context, _ string,
) ([]storage.ObjectInfo, error) {
return nil, nil
// DeletePartialUploads has nothing to remove: Put stores each object
// under its key at once.
func (s *testStorer) DeletePartialUploads(_ context.Context, _ string) error {
return nil
}
func (s *testStorer) Info() storage.Info {
+8 -19
View File
@@ -25,13 +25,9 @@ import (
"sneak.berlin/go/vaultik/internal/types"
)
// ErrFilesFailedRestore is returned when --skip-errors skipped files the
// restore could not write. Every other file was restored, so the caller
// can report it apart from a failed restore.
var ErrFilesFailedRestore = errors.New("file(s) failed to restore")
// Sentinel errors for restore failures.
var (
errFilesFailedRestore = errors.New("file(s) failed to restore")
errPathsNotInSnapshot = errors.New("requested path(s) not in the snapshot")
errFilesFailedVerify = errors.New("files failed verification")
errDecryptionKeyRequired = errors.New(
@@ -205,9 +201,9 @@ func (v *Vaultik) Restore(opts *RestoreOptions) error {
}
// finishRestore prints the restore summary and the post-restore
// warnings, runs optional verification, and returns the error the
// restore ends with: one for the files --skip-errors skipped, one for
// unmatchedErr, the requested paths that matched nothing, or nil.
// warnings, runs optional verification, and converts any failed-file
// count into an error. Otherwise it returns unmatchedErr, the error for
// the requested paths that matched nothing (nil when there are none).
func (v *Vaultik) finishRestore(
repos *database.Repositories,
files []*database.File,
@@ -254,18 +250,11 @@ func (v *Vaultik) finishRestore(
}
}
switch {
case result.FilesFailed > 0 && unmatchedErr != nil:
// A requested path that matched nothing outweighs skipped files:
// the error wraps only unmatchedErr, so the command does not exit
// as a restore that only skipped files.
return fmt.Errorf("%w; %d %s",
unmatchedErr, result.FilesFailed, ErrFilesFailedRestore.Error())
case result.FilesFailed > 0:
return fmt.Errorf("%d %w", result.FilesFailed, ErrFilesFailedRestore)
default:
return unmatchedErr
if result.FilesFailed > 0 {
return fmt.Errorf("%d %w", result.FilesFailed, errFilesFailedRestore)
}
return unmatchedErr
}
// warnIfOwnershipNotPreserved warns once if some entry restored under
+1 -1
View File
@@ -80,7 +80,7 @@ func TestRestoreFreesBlobsNoPendingFileNeeds(t *testing.T) {
TargetDir: restoreDir,
SkipErrors: true,
})
require.ErrorIs(t, err, ErrFilesFailedRestore)
require.ErrorIs(t, err, errFilesFailedRestore)
requireRestoredFiles(t, fs, restoreDir, originals)
assert.LessOrEqualf(t, peak, 1,
+11 -24
View File
@@ -49,7 +49,8 @@ var ErrFilesUnreadable = errors.New("file(s) could not be read")
// prune because the scan found nothing in a configured path of a
// snapshot, as in the mount point of a volume that is not mounted. The
// snapshots are complete and kept, so the caller can report it apart
// from a failed backup.
// from a failed backup. A run that also skipped unreadable files
// returns ErrFilesUnreadable instead.
var ErrSourcePathEmpty = errors.New("source path(s) were empty")
// ErrVolumeNotMounted is returned when --prune skipped the post-backup
@@ -58,8 +59,8 @@ var ErrSourcePathEmpty = errors.New("source path(s) were empty")
// mounted there is not mounted, the run found no entry in it, or the
// directory is no longer backed up. The snapshots are complete and
// kept, so the caller can report it apart from a failed backup. A run
// that also found a configured path empty returns ErrSourcePathEmpty
// instead.
// that also skipped unreadable files or found a configured path empty
// returns ErrFilesUnreadable or ErrSourcePathEmpty instead.
var ErrVolumeNotMounted = errors.New(
"mount point(s) an earlier snapshot recorded were not found")
@@ -242,29 +243,15 @@ func (v *Vaultik) skipPostBackupPrune(
if filesUnreadable > 0 {
v.UI.Warningf("Skipped the post-backup prune because %d file(s) "+
"could not be read; older snapshots are kept.", filesUnreadable)
}
var err error
switch {
case len(emptyPaths) > 0:
err = fmt.Errorf("%d %w", len(emptyPaths), ErrSourcePathEmpty)
case len(unmountedPaths) > 0:
err = fmt.Errorf("%d %w", len(unmountedPaths), ErrVolumeNotMounted)
default:
return fmt.Errorf("%d %w", filesUnreadable, ErrFilesUnreadable)
}
// An empty path or a missing volume can mean a whole volume is not
// in the backup, which outweighs skipped files: the error wraps only
// the former, so the command does not exit as a backup that only
// skipped files.
if filesUnreadable > 0 {
return fmt.Errorf("%w; %d %s",
err, filesUnreadable, ErrFilesUnreadable.Error())
if len(emptyPaths) > 0 {
return fmt.Errorf("%d %w", len(emptyPaths), ErrSourcePathEmpty)
}
return err
return fmt.Errorf("%d %w", len(unmountedPaths), ErrVolumeNotMounted)
}
// finishCreateSnapshot ends a run whose snapshots are all complete and
@@ -315,10 +302,10 @@ func (v *Vaultik) snapshotNamesToCreate(requested []string) ([]string, error) {
}
// runPostBackupPrune drops older snapshots of the given names and removes
// from remote storage the orphan blobs, the directories under metadata/
// with no manifest and the `.partial` objects, as PruneBlobs does. If
// keepNewerThan is set (e.g. "4w"), snapshots newer than that duration are
// kept. Otherwise only the latest snapshot of each name is kept.
// from remote storage the orphan blobs and the directories under
// metadata/ with no manifest, as PruneBlobs does. If keepNewerThan is set
// (e.g. "4w"), snapshots newer than that duration are kept. Otherwise only
// the latest snapshot of each name is kept.
func (v *Vaultik) runPostBackupPrune(
snapshotNames []string, keepNewerThan string,
) error {
+3 -9
View File
@@ -176,11 +176,8 @@ func TestBackupThroughSymlinkWhoseTargetIsGoneDoesNotPrune(t *testing.T) {
// A backup under --skip-errors --prune that both skipped a file it could
// not read and found nothing in a configured path names the empty path
// too when it skips the prune. It ends with the empty path's error, which
// names the unreadable file too but does not wrap ErrFilesUnreadable, so
// the command does not exit as a backup that only skipped files. See
// https://git.eeqj.de/sneak/vaultik/issues/388 and
// https://git.eeqj.de/sneak/vaultik/issues/439.
// too when it skips the prune. See
// https://git.eeqj.de/sneak/vaultik/issues/388.
func TestBackupWithEmptySourcePathAndUnreadableFileNamesBoth(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
@@ -210,10 +207,7 @@ func TestBackupWithEmptySourcePathAndUnreadableFileNamesBoth(t *testing.T) {
SkipErrors: true,
Snapshots: []string{snapshotName},
})
require.ErrorIs(t, err, vaultik.ErrSourcePathEmpty)
require.NotErrorIs(t, err, vaultik.ErrFilesUnreadable)
require.EqualError(t, err,
"1 source path(s) were empty; 1 file(s) could not be read")
require.ErrorIs(t, err, vaultik.ErrFilesUnreadable)
assert.Contains(t, env.out.String(),
"Skipped the post-backup prune because 1 file(s) could not be read")
assert.Contains(t, env.out.String(),