Compare commits

..
Author SHA1 Message Date
sneak 59664f219e Test that a failed .partial delete fails prune
check / check (push) Waiting to run
Nothing tested that prune exits non-zero and counts
partial_uploads_failed when a .partial object cannot be deleted, so
dropping that branch from pruneDeleteFailures passed the suite. The new
test fails every delete through faultstore and checks both.

Model: opus-5-5
2026-11-19 07:53:03 +00:00
sneak 92296daf52 Mention .partial objects in the help and config init text
prune --help, remote info --help, the snapshot create --prune flag, the
config init template's prune_grace_period comment and the
runPostBackupPrune comment each listed what prune deletes or remote info
counts, and none named the .partial objects this change adds. They do
now; the --prune flag refers to prune instead of repeating its list.

Model: opus-5-5
2026-11-19 07:47:00 +00:00
sneak 332c2d30a2 Shorten the README's opening sentence for --prune
Adding .partial objects made the first sentence of the --prune bullet
about 70 words long, chaining three cleanup steps with "and ..., and".
It now says the flag deletes from the destination store what prune
deletes there; the prune section lists what that is.

Model: opus-5-5
2026-11-19 07:46:03 +00:00
sneak 18961fa52f Name prune in the docs sentence on .partial age
In docs/REPOSTRUCTURE.md the sentence on the age of a .partial object
began with "It" right after a sentence about the new snapshot, so it
read as the snapshot deleting the object. It now says "Prune".

Model: opus-5-5
2026-11-19 07:45:42 +00:00
sneak 3d0222d644 Reword the .partial sentences that stack clauses
The sentences added for .partial objects in README.md,
docs/REPOSTRUCTURE.md, the example config and the PruneGracePeriod
comment read "a .partial object an upload cut off part-way left" and
"how old a .partial object a killed upload left must be": the verb
arrives late and "cut off" reads as the verb. They now say "left by a
killed upload".

Model: opus-5-5
2026-11-19 07:45:34 +00:00
sneak 52fada5236 Let prune delete old .partial uploads and remote info count them (closes #466)
check / check (push) Waiting to run
A killed upload leaves a `.partial` object that only remote nuke
removed, and that remote info never showed.

The storage interface's DeletePartialUploads becomes ListPartialUploads,
which returns each such object's key, size and time; Delete removes it.
prune deletes those under metadata/ and blobs/ once older than
prune_grace_period, and reports kept, deleted and failed counts. remote
info counts them on their own line and in --json. remote nuke lists and
deletes them like any other object, and now names any it could not
delete.

Model: opus-5-5
2026-11-19 07:31:08 +00:00
29 changed files with 663 additions and 702 deletions

No files matched your search

+45 -48
View File
@@ -169,7 +169,7 @@ vaultik [--config <path>] config init
vaultik [--config <path>] config edit vaultik [--config <path>] config edit
vaultik [--config <path>] config get <key> vaultik [--config <path>] config get <key>
vaultik [--config <path>] config set <key> <value> vaultik [--config <path>] config set <key> <value>
vaultik [--config <path>] snapshot create [snapshot-names...] [--cron] [--prune] [--keep-newer-than <duration>] [--dry-run] vaultik [--config <path>] snapshot create [snapshot-names...] [--cron] [--prune] [--keep-newer-than <duration>]
vaultik [--config <path>] snapshot list [--json] # alias: ls vaultik [--config <path>] snapshot list [--json] # alias: ls
vaultik [--config <path>] snapshot files <snapshot-id> [path] [--json] vaultik [--config <path>] snapshot files <snapshot-id> [path] [--json]
vaultik [--config <path>] snapshot verify <snapshot-id> [--deep] [--json] vaultik [--config <path>] snapshot verify <snapshot-id> [--deep] [--json]
@@ -340,46 +340,33 @@ In such a config, an unknown key or a value of the wrong type is refused.
error that explains how to fix it error that explains how to fix it
- `--cron`: Silent on total success; warnings and errors are still printed (for - `--cron`: Silent on total success; warnings and errors are still printed (for
crontab) crontab)
- `--prune`: After backup, drop older snapshots of each backed-up name, remove - `--prune`: After backup, drop older snapshots of each backed-up name, then
orphaned blobs older than `prune_grace_period` from remote storage, and delete delete from the destination store what `prune` deletes there. By default keeps
each directory under `metadata/` that has no `manifest.json.zst` once every only the latest snapshot per name; use `--keep-newer-than` for a rolling
object in it is older than `prune_grace_period` and its snapshot is not in the window. When `--skip-errors` skipped a file or directory it could not read or
local index, as `prune` does. By default keeps only the latest snapshot per list, the prune does not run and every older snapshot is kept, since one of
name; use `--keep-newer-than` for a rolling window. When `--skip-errors` them may be the only one that holds what was skipped; the command says so and
skipped a file or directory it could not read or list, the prune does not run exits non-zero. When the backup found nothing in a configured path, as in the
and every older snapshot is kept, since one of them may be the only one that mount point of a volume that is not mounted, the prune does not run either:
holds what was skipped; the command says so and exits non-zero. When the every older snapshot is kept, and the command names the path and exits
backup found nothing in a configured path, as in the mount point of a volume non-zero, on every run until the path holds something again. Take a path that
that is not mounted, the prune does not run either: every older snapshot is is empty on purpose out of the config, or remove the older snapshots by hand
kept, and the command names the path and exits non-zero, on every run until with `snapshot purge`. When an earlier snapshot of the same name in the local
the path holds something again. Take a path that is empty on purpose out of index found a volume mounted at or below a configured path (a directory on a
the config, or remove the older snapshots by hand with `snapshot purge`. When different filesystem from its parent, with an entry in it that the backup does
an earlier snapshot of the same name in the local index found a volume mounted not exclude) and this backup does not, the prune does not run either: every
at or below a configured path (a directory on a different filesystem from its older snapshot is kept, and the command names the directory and exits
parent, with an entry in it that the backup does not exclude) and this backup non-zero, on every run until the volume is mounted again. A backup also does
does not, the prune does not run either: every older snapshot is kept, and the not find the volume when it finds no such entry in its directory, as when an
command names the directory and exits non-zero, on every run until the volume automounted volume cannot be mounted, or when that directory is no longer
is mounted again. A backup also does not find the volume when it finds no such backed up, because it is excluded or its configured path was removed. If any
entry in its directory, as when an automounted volume cannot be mounted, or of this is on purpose, as for a retired volume, remove the older snapshots by
when that directory is no longer backed up, because it is excluded or its hand with `snapshot purge`. A file removed during the backup does not stop the
configured path was removed. If any of this is on purpose, as for a retired prune.
volume, remove the older snapshots by hand with `snapshot purge`. A file
removed during the backup does not stop the prune.
- `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than this - `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than this
duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`). Without duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`). Without
`--prune`, or with a value that is not a valid duration, the command fails `--prune`, or with a value that is not a valid duration, the command fails
before the backup starts. before the backup starts.
- `--dry-run`: Scan the source paths and print what a backup would store,
without storing it. The summary gives the files examined, to back up and
unchanged, their sizes, and the count of excluded entries. An excluded
directory counts once, since the scan does not enter it. With `--verbose`, it
also prints each excluded entry and the exclude pattern that matched it. It
writes nothing to the destination and records nothing in the local index, so
it does not bind the index to the destination. Like the other commands that
read the local index, it creates an empty one if there is none. When the local
index holds an incomplete snapshot from an interrupted backup, it says so: the
next backup drops that snapshot first and may back up more files than the dry
run counts. It cannot be combined with `--prune`.
- Fails before it writes anything when a `file://` destination directory does - Fails before it writes anything when a `file://` destination directory does
not exist, or when the local index holds snapshots and the destination answers not exist, or when the local index holds snapshots and the destination answers
that it has no `vaultik-destination.txt`. When that look-up fails for any that it has no `vaultik-destination.txt`. When that look-up fails for any
@@ -531,14 +518,16 @@ deleted. Step (3) also deletes each directory under `metadata/` that has no
`db.zst.age` but before its manifest, once every object in it is older than `db.zst.age` but before its manifest, once every object in it is older than
`prune_grace_period` and its snapshot is not in the local index. An object the `prune_grace_period` and its snapshot is not in the local index. An object the
listing gives no time for keeps its directory, and a directory whose name is not listing gives no time for keeps its directory, and a directory whose name is not
a remote key is left alone. Step (3) reads all remote manifests — network cost a remote key is left alone. Step (3) also deletes each `.partial` object under
scales with the number of snapshots. `snapshot create --prune` runs the same `metadata/` and `blobs/` left by a killed upload, once it is older than
cleanup automatically; this is the manual entry point for the same work. If the `prune_grace_period`. Step (3) reads all remote manifests — network cost scales
with the number of snapshots. `snapshot create --prune` runs the same cleanup
automatically; this is the manual entry point for the same work. If the
destination store cannot be listed, or the local index holds snapshots and the destination store cannot be listed, or the local index holds snapshots and the
destination answers that it has no `vaultik-destination.txt` (an unplugged destination answers that it has no `vaultik-destination.txt` (an unplugged
volume's empty mount point), prune fails and deletes nothing. `snapshot purge` volume's empty mount point), prune fails and deletes nothing. `snapshot purge`
does the same. If a blob or such a directory cannot be deleted, the command (and does the same. If a blob, such a directory or such a `.partial` object cannot be
`snapshot create --prune`) exits non-zero. deleted, the command (and `snapshot create --prune`) exits non-zero.
- `--force`: Skip confirmation prompt - `--force`: Skip confirmation prompt
- `--json`: Output stats as JSON. Also skips the confirmation prompt, as - `--json`: Output stats as JSON. Also skips the confirmation prompt, as
@@ -548,7 +537,10 @@ does the same. If a blob or such a directory cannot be deleted, the command (and
gives no time for it, is counted in `blobs_too_new` and not in `blobs_found`. gives no time for it, is counted in `blobs_too_new` and not in `blobs_found`.
A directory under `metadata/` with no manifest is counted in A directory under `metadata/` with no manifest is counted in
`metadata_dirs_deleted` once deleted, or in `metadata_dirs_failed` if it `metadata_dirs_deleted` once deleted, or in `metadata_dirs_failed` if it
cannot be. cannot be. A `.partial` object is counted in `partial_uploads_deleted` once
deleted, or in `partial_uploads_failed` if it cannot be. One kept for being
newer than `prune_grace_period`, or of unknown age, is counted in
`partial_uploads_too_new`.
**`info`**: Display system configuration, storage settings, encryption **`info`**: Display system configuration, storage settings, encryption
recipients, and local database statistics. recipients, and local database statistics.
@@ -563,7 +555,9 @@ in `unreadable_manifests` and counts the manifests under skipped names in
`skipped_manifest_count`. An unreadable manifest also leaves its snapshot's blob `skipped_manifest_count`. An unreadable manifest also leaves its snapshot's blob
count and blob size unknown, `null` in `--json`. A file under `blobs/` whose count and blob size unknown, `null` in `--json`. A file under `blobs/` whose
name is not a blob hash is not counted as a blob, since `prune` never deletes name is not a blob hash is not counted as a blob, since `prune` never deletes
it. it. A leftover `.partial` object under `metadata/` or `blobs/` is not counted as
a blob either. Such objects are counted on their own line, and in
`partial_upload_count` and `partial_upload_size` in `--json`.
- `--json`: Output as JSON - `--json`: Output as JSON
@@ -622,8 +616,9 @@ with a server-side move (local and sftp among them) are written under a
temporary name ending in `.partial` and moved into place. Rclone remotes without temporary name ending in `.partial` and moved into place. Rclone remotes without
one are written in place: where such a remote shows a file while it is still one are written in place: where such a remote shows a file while it is still
being written, a killed upload can leave a truncated object under its name, being written, a killed upload can leave a truncated object under its name,
which a later backup takes for complete. A leftover `.partial` file is ignored which a later backup takes for complete. A backup ignores a leftover `.partial`
and can be deleted. file. `prune` deletes it once it is older than `prune_grace_period`, and
`remote info` counts it.
A destination shared between hosts or users, each with its own local index, A destination shared between hosts or users, each with its own local index,
relies on `prune_grace_period` to keep one's prune from deleting the blobs of relies on `prune_grace_period` to keep one's prune from deleting the blobs of
@@ -732,6 +727,8 @@ format does and does not protect.
5. Delete each directory under `metadata/` with no manifest whose objects are 5. Delete each directory under `metadata/` with no manifest whose objects are
all older than `prune_grace_period`, unless its snapshot is in the local all older than `prune_grace_period`, unless its snapshot is in the local
index index
6. Delete each `.partial` object under `metadata/` and `blobs/` older than
`prune_grace_period`
### chunking and deduplication ### chunking and deduplication
@@ -766,7 +763,7 @@ annotated example also lives in
[`configs/config.example.yml`](configs/config.example.yml). Key fields: [`configs/config.example.yml`](configs/config.example.yml). Key fields:
| Field | Default | Description | | Field | Default | Description |
| -------------------- | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | -------------------- | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `age_recipients` | (required by `snapshot create`) | Age public keys for encryption. Other commands run without one, so a machine that only restores can leave it empty | | `age_recipients` | (required by `snapshot create`) | Age public keys for encryption. Other commands run without one, so a machine that only restores can leave it empty |
| `age_secret_key` | (unset) | Age private key for decryption (`snapshot restore`, `snapshot files`, `snapshot verify --deep`). Setting it in the config file places the private key on the backed-up host, defeating the public-key-only design (see "why" above). Prefer the `VAULTIK_AGE_SECRET_KEY` environment variable, supplied only on the machine you restore from. | | `age_secret_key` | (unset) | Age private key for decryption (`snapshot restore`, `snapshot files`, `snapshot verify --deep`). Setting it in the config file places the private key on the backed-up host, defeating the public-key-only design (see "why" above). Prefer the `VAULTIK_AGE_SECRET_KEY` environment variable, supplied only on the machine you restore from. |
| `snapshots` | (required) | Named snapshot definitions with paths and excludes | | `snapshots` | (required) | Named snapshot definitions with paths and excludes |
@@ -776,7 +773,7 @@ annotated example also lives in
| `chunk_size` | `10MB` | Average chunk size for content-defined chunking | | `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` | | `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` |
| `compression_level` | `3` | zstd compression level (1-19) | | `compression_level` | `3` | zstd compression level (1-19) |
| `prune_grace_period` | `24h` | How old a blob that no snapshot lists must be before `prune` deletes it, and how old every object in a directory under `metadata/` with no manifest must be before `prune` deletes that directory. A backup uploads its blobs and its `db.zst.age` before its manifest, so on a destination shared between hosts or users this must be longer than the longest backup run (see [locking](#locking)). It does not protect a blob a backup reuses from an earlier run that stopped before its manifest: that blob keeps the time of its first upload, so a prune can delete it. A duration such as `36h` or `90m`; `0s` deletes such blobs and directories at once | | `prune_grace_period` | `24h` | How old a blob that no snapshot lists must be before `prune` deletes it, and how old every object in a directory under `metadata/` with no manifest must be before `prune` deletes that directory. `prune` also deletes a `.partial` object left by a killed upload only once it is this old. A backup uploads its blobs and its `db.zst.age` before its manifest, so on a destination shared between hosts or users this must be longer than the longest backup run (see [locking](#locking)). It does not protect a blob a backup reuses from an earlier run that stopped before its manifest: that blob keeps the time of its first upload, so a prune can delete it. A duration such as `36h` or `90m`; `0s` deletes such blobs and directories at once |
| `hostname` | system hostname | Hostname used in snapshot IDs | | `hostname` | system hostname | Hostname used in snapshot IDs |
| `index_path` | platform data dir | Local SQLite index path | | `index_path` | platform data dir | Local SQLite index path |
+2
View File
@@ -333,6 +333,8 @@ compression_level: 5
# How old a blob that no snapshot lists must be before prune deletes it, and # How old a blob that no snapshot lists must be before prune deletes it, and
# how old every object in a directory under metadata/ with no manifest must # how old every object in a directory under metadata/ with no manifest must
# be before prune deletes that directory # be before prune deletes that directory
# prune also deletes a .partial object left by a killed upload only once it is
# this old
# A backup uploads its blobs and its db.zst.age before its manifest, so on a # A backup uploads its blobs and its db.zst.age before its manifest, so on a
# destination shared between hosts or users this must be longer than the # destination shared between hosts or users this must be longer than the
# longest backup run # longest backup run
+3
View File
@@ -314,6 +314,9 @@ The restore process doesn't use the local database. Instead:
- Delete each directory under `metadata/` that has no `manifest.json.zst` - Delete each directory under `metadata/` that has no `manifest.json.zst`
and no local snapshot record, once every object in it is older than and no local snapshot record, once every object in it is older than
`prune_grace_period` `prune_grace_period`
3. **Delete Partial Uploads**
- Delete each `.partial` object a killed upload left under `metadata/` or
`blobs/`, once it is older than `prune_grace_period`
### 6. Incomplete Snapshot Cleanup ### 6. Incomplete Snapshot Cleanup
+5 -3
View File
@@ -329,7 +329,8 @@ The prune operation is safe because:
1. It keeps every blob listed in any snapshot's manifest and deletes only blobs 1. It keeps every blob listed in any snapshot's manifest and deletes only blobs
that no manifest references. Under `metadata/` it deletes only a directory that no manifest references. Under `metadata/` it deletes only a directory
that has no `manifest.json.zst`, whose name is a remote key, and that no that has no `manifest.json.zst`, whose name is a remote key, and that no
local snapshot record claims local snapshot record claims. It also deletes the `.partial` objects left by
killed uploads, which no backup or restore reads
2. Manifests are unencrypted and can be read without keys 2. Manifests are unencrypted and can be read without keys
3. If any manifest cannot be downloaded or decoded, prune deletes nothing and 3. If any manifest cannot be downloaded or decoded, prune deletes nothing and
exits with an error, rather than treating that snapshot's blobs as exits with an error, rather than treating that snapshot's blobs as
@@ -343,8 +344,9 @@ The prune operation is safe because:
a backup reuses from an earlier run that stopped before its manifest: the a backup reuses from an earlier run that stopped before its manifest: the
backup skips the upload of a blob already on the destination, so that blob backup skips the upload of a blob already on the destination, so that blob
keeps the time of its first upload, and a prune before the backup's manifest keeps the time of its first upload, and a prune before the backup's manifest
is written can delete it; the new snapshot then cannot be restored. is written can delete it; the new snapshot then cannot be restored. Prune
`remote nuke` has no such check deletes a `.partial` object only once it is older than `prune_grace_period`,
so an upload still running keeps its object. `remote nuke` has no such check
## Restoration Requirements ## Restoration Requirements
+4 -4
View File
@@ -39,10 +39,10 @@ const (
// hold the PID lock so that at most one runs at a time. // hold the PID lock so that at most one runs at a time.
mutating lockMode = iota mutating lockMode = iota
// readOnly commands (info, snapshot list, snapshot files, snapshot // readOnly commands (info, snapshot list, snapshot files, snapshot
// verify, remote info, snapshot restore, snapshot create --dry-run) // verify, remote info, snapshot restore) do not write the local index
// do not write the local index or the remote store, so they run // or the remote store, so they run without the lock and are never
// without the lock and are never blocked by a running mutator. // blocked by a running mutator. restore writes only to the target
// restore writes only to the target directory it is given. // directory it is given.
readOnly readOnly
) )
+5 -3
View File
@@ -269,9 +269,11 @@ storage_url: ""
# How old a blob that no snapshot lists must be before prune deletes it, # How old a blob that no snapshot lists must be before prune deletes it,
# and how old every object in a directory under metadata/ with no manifest # and how old every object in a directory under metadata/ with no manifest
# must be before prune deletes that directory. A backup uploads its blobs # must be before prune deletes that directory. prune also deletes a
# and its db.zst.age before its manifest, so on a destination shared # .partial object left by a killed upload only once it is this old. A
# between hosts or users this must be longer than the longest backup run. # backup uploads its blobs and its db.zst.age before its manifest, so on
# a destination shared between hosts or users this must be longer than
# the longest backup run.
# It does not protect a blob a backup reuses from an earlier run that # It does not protect a blob a backup reuses from an earlier run that
# stopped before its manifest: that blob keeps the time of its first # stopped before its manifest: that blob keeps the time of its first
# upload, so a prune can delete it. # upload, so a prune can delete it.
@@ -1,89 +0,0 @@
package cli //nolint:testpackage // shares runEntry and the hermetic config helpers
import (
"context"
"io"
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/ui"
)
// TestEntrySnapshotCreateDryRun runs `snapshot create --dry-run
// --verbose` on a source directory that holds one kept file, one file an
// unanchored pattern excludes and one directory an anchored pattern
// excludes. The summary counts the kept file and two excluded entries,
// the excluded ones are listed with their patterns, and nothing is
// written to the destination or recorded in the local index. See
// https://git.eeqj.de/sneak/vaultik/issues/436.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotCreateDryRun(t *testing.T) {
configPath, storeDir := writeBackupConfig(t)
sourceDir := filepath.Join(filepath.Dir(configPath), "source")
cacheDir := filepath.Join(sourceDir, "cache")
require.NoError(t, os.WriteFile(
filepath.Join(sourceDir, "debug.log"), []byte("log"), 0o600))
require.NoError(t, os.Mkdir(cacheDir, 0o750))
for _, name := range []string{"a", "b"} {
require.NoError(t, os.WriteFile(
filepath.Join(cacheDir, name), []byte(name), 0o600))
}
out := ui.NewWithColor(io.Discard, false)
require.NoError(t, writeConfigSet(out, configPath, "exclude.0", "*.log"))
require.NoError(t, writeConfigSet(out, configPath, "exclude.1", "/cache"))
code, stdout, stderr := runEntry(t, flagConfig, configPath, "--verbose",
cmdSnapshot, cmdCreate, "--dry-run")
require.Equal(t, 0, code, stderr)
assert.Contains(t, stdout, "Excluded "+filepath.Join(sourceDir, "debug.log")+
`: matches exclude pattern "*.log".`)
assert.Contains(t, stdout, "Excluded "+cacheDir+
`: matches exclude pattern "/cache".`)
assert.NotContains(t, stdout, filepath.Join(cacheDir, "a"))
assert.Contains(t, stdout, "Files: 1 examined, 1 to back up, 0 unchanged.")
assert.Contains(t, stdout, "Excluded: 2 entries.")
entries, err := os.ReadDir(storeDir)
require.NoError(t, err)
assert.Empty(t, entries, "a dry run writes nothing to the destination")
ctx := context.Background()
db, err := database.New(ctx,
filepath.Join(filepath.Dir(configPath), "index.sqlite"))
require.NoError(t, err)
defer func() { require.NoError(t, db.Close()) }()
repos := database.NewRepositories(db)
snapshots, err := repos.Snapshots.ListRecent(ctx, 1)
require.NoError(t, err)
assert.Empty(t, snapshots, "a dry run records no snapshot")
binding, err := repos.LocalMeta.Get(ctx, database.LocalMetaKeyStorageURL)
require.NoError(t, err)
assert.Empty(t, binding, "a dry run does not bind the local index")
}
// TestEntrySnapshotCreateDryRunRefusesPrune checks that --dry-run with
// --prune fails before anything runs.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotCreateDryRunRefusesPrune(t *testing.T) {
configPath, _ := writeBackupConfig(t)
code, _, stderr := runEntry(t, flagConfig, configPath,
cmdSnapshot, cmdCreate, "--dry-run", "--prune")
assert.NotEqual(t, 0, code)
assert.Contains(t, stderr, "--dry-run cannot be used with --prune")
}
+6 -4
View File
@@ -13,9 +13,10 @@ func NewPruneCommand() *cobra.Command {
cmd := &cobra.Command{ cmd := &cobra.Command{
Use: "prune", Use: "prune",
Short: "Tidy local database and remote storage", Short: "Tidy local database and remote storage",
Long: `Removes orphaned data from the local index database, and Long: `Removes orphaned data from the local index database. From the
unreferenced blobs and directories under metadata/ with no manifest backup destination store it removes unreferenced blobs, directories
from the backup destination store. under metadata/ with no manifest, and .partial objects left by killed
uploads.
Local cleanup drops incomplete snapshots and any files, chunks, or Local cleanup drops incomplete snapshots and any files, chunks, or
blobs no longer referenced by a completed snapshot. Remote cleanup blobs no longer referenced by a completed snapshot. Remote cleanup
@@ -30,7 +31,8 @@ blob keeps the time of its first upload, so it can be deleted.
Remote cleanup also deletes each directory under metadata/ with no Remote cleanup also deletes each directory under metadata/ with no
manifest, as a backup that stopped before its manifest leaves, once manifest, as a backup that stopped before its manifest leaves, once
all its objects are older than prune_grace_period and its snapshot is all its objects are older than prune_grace_period and its snapshot is
not in the local index. not in the local index. It also deletes each .partial object under
metadata/ or blobs/ once it is older than prune_grace_period.
Snapshot create --prune runs the same cleanup automatically; this Snapshot create --prune runs the same cleanup automatically; this
command is the manual entry point for the same work (e.g. after a command is the manual entry point for the same work (e.g. after a
+2 -1
View File
@@ -75,7 +75,8 @@ func newRemoteInfoCommand() *cobra.Command {
- Size of all snapshot metadata (per snapshot and total) - Size of all snapshot metadata (per snapshot and total)
- Count and total size of all blobs - Count and total size of all blobs
- Count and size of referenced blobs (from all manifests) - Count and size of referenced blobs (from all manifests)
- Count and size of orphaned blobs (not referenced by any manifest)`, - Count and size of orphaned blobs (not referenced by any manifest)
- Count and size of .partial objects left by killed uploads`,
Args: cobra.NoArgs, Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error { RunE: func(cmd *cobra.Command, _ []string) error {
// Use unified config resolution // Use unified config resolution
+24 -42
View File
@@ -99,13 +99,6 @@ on Linux), then /etc/vaultik/config.yml.`,
// Use the backup functionality from cli package // Use the backup functionality from cli package
rootFlags := GetRootFlags() rootFlags := GetRootFlags()
opts.ListExcluded = rootFlags.Verbose
// A dry run writes nothing, so it does not take the lock.
mode := mutating
if opts.DryRun {
mode = readOnly
}
// --cron suppression is wired through v.UI by setupGlobals. // --cron suppression is wired through v.UI by setupGlobals.
return RunOperation(cmd.Context(), AppOptions{ return RunOperation(cmd.Context(), AppOptions{
@@ -116,11 +109,28 @@ on Linux), then /etc/vaultik/config.yml.`,
Cron: opts.Cron, Cron: opts.Cron,
Quiet: rootFlags.Quiet, Quiet: rootFlags.Quiet,
}, },
Mode: mode, Mode: mutating,
}, func(v *vaultik.Vaultik) error { }, func(v *vaultik.Vaultik) error {
return v.CreateSnapshot(opts) return v.CreateSnapshot(opts)
}, func(err error) { }, func(err error) {
reportSnapshotCreateFailure(opts.DryRun, err) if errors.Is(err, vaultik.ErrPostBackupPruneFailed) {
log.Info("Post-backup prune failed", "error", err)
ReportErrorf("Backup completed, but %v", err)
return
}
if errors.Is(err, vaultik.ErrFilesUnreadable) ||
errors.Is(err, vaultik.ErrSourcePathEmpty) ||
errors.Is(err, vaultik.ErrVolumeNotMounted) {
log.Info("Backup completed with an error", "error", err)
ReportErrorf("Backup completed, but %v", err)
return
}
log.Info("Snapshot creation failed", "error", err)
ReportErrorf("Snapshot creation failed: %v", err)
}) })
}, },
} }
@@ -128,47 +138,19 @@ on Linux), then /etc/vaultik/config.yml.`,
cmd.Flags().BoolVar(&opts.Cron, "cron", false, cmd.Flags().BoolVar(&opts.Cron, "cron", false,
"Run in cron mode (silent unless warning or error)") "Run in cron mode (silent unless warning or error)")
cmd.Flags().BoolVar(&opts.Prune, "prune", false, cmd.Flags().BoolVar(&opts.Prune, "prune", false,
"After backup, drop older snapshots of the same name, remove "+ "After backup, drop older snapshots of the same name, then delete "+
"orphaned blobs older than prune_grace_period, and delete each "+ "from the destination store what prune deletes there (not "+
"directory under metadata/ with no manifest once all its "+ "after a backup that skipped unreadable files, found a "+
"objects are older than prune_grace_period and its snapshot is "+ "configured path empty, or did not find a mount point an "+
"not in the local index (not after a backup that skipped "+ "earlier snapshot found)")
"unreadable files, found a configured path empty, or did not "+
"find a mount point an earlier snapshot found)")
cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "", cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "",
"With --prune: keep snapshots newer than this duration "+ "With --prune: keep snapshots newer than this duration "+
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+ "(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+
"instead of only the latest") "instead of only the latest")
cmd.Flags().BoolVar(&opts.DryRun, "dry-run", false,
"Scan the snapshot source paths and print what a backup would store, "+
"writing nothing to the local index or the backup destination; "+
"with --verbose, also print each excluded entry and the exclude "+
"pattern that matched it")
return cmd return cmd
} }
// reportSnapshotCreateFailure shows the user the error `snapshot create`
// failed with. dryRun says whether it was given --dry-run.
func reportSnapshotCreateFailure(dryRun bool, err error) {
switch {
case dryRun:
log.Info("Dry run failed", "error", err)
ReportErrorf("Dry run failed: %v", err)
case errors.Is(err, vaultik.ErrPostBackupPruneFailed):
log.Info("Post-backup prune failed", "error", err)
ReportErrorf("Backup completed, but %v", err)
case errors.Is(err, vaultik.ErrFilesUnreadable) ||
errors.Is(err, vaultik.ErrSourcePathEmpty) ||
errors.Is(err, vaultik.ErrVolumeNotMounted):
log.Info("Backup completed with an error", "error", err)
ReportErrorf("Backup completed, but %v", err)
default:
log.Info("Snapshot creation failed", "error", err)
ReportErrorf("Snapshot creation failed: %v", err)
}
}
// newSnapshotListCommand creates the 'snapshot list' subcommand // newSnapshotListCommand creates the 'snapshot list' subcommand
func newSnapshotListCommand() *cobra.Command { func newSnapshotListCommand() *cobra.Command {
var jsonOutput bool var jsonOutput bool
+2 -1
View File
@@ -206,7 +206,8 @@ type Config struct {
// directory. A backup writes its manifest only once it has uploaded // directory. A backup writes its manifest only once it has uploaded
// all its blobs and its db.zst.age, so a younger blob or directory may // all its blobs and its db.zst.age, so a younger blob or directory may
// belong to a backup still running, on this host or another sharing // belong to a backup still running, on this host or another sharing
// the destination store. // the destination store. A prune also deletes a `.partial` object left
// by a killed upload only once it is this old.
PruneGracePeriod time.Duration `yaml:"prune_grace_period"` PruneGracePeriod time.Duration `yaml:"prune_grace_period"`
// StorageURL specifies the storage backend using a URL format. // StorageURL specifies the storage backend using a URL format.
+12 -102
View File
@@ -71,9 +71,6 @@ type Scanner struct {
// indexAndLockFiles holds the paths of vaultik's own working files, // indexAndLockFiles holds the paths of vaultik's own working files,
// which the scan treats as excluded (ScannerConfig.IndexAndLockFiles). // which the scan treats as excluded (ScannerConfig.IndexAndLockFiles).
indexAndLockFiles map[string]struct{} indexAndLockFiles map[string]struct{}
// listExcluded prints each excluded entry and why it is excluded
// (DryRun).
listExcluded bool
progress *ProgressReporter progress *ProgressReporter
// skipErrors skips files that cannot be opened or read (logged loudly); // skipErrors skips files that cannot be opened or read (logged loudly);
// packer, database, encryption, and upload errors still abort the run. // packer, database, encryption, and upload errors still abort the run.
@@ -193,11 +190,6 @@ type ScanResult struct {
// itself when it is a directory. It is zero for an empty directory, // itself when it is a directory. It is zero for an empty directory,
// such as the mount point of a volume that is not mounted. // such as the mount point of a volume that is not mounted.
EntriesFound int EntriesFound int
// EntriesExcluded counts the entries the walk found that an exclude
// pattern matches or that are vaultik's own files. An excluded
// directory counts once: the walk does not enter it.
EntriesExcluded int
} }
// NewScanner creates a new scanner instance // NewScanner creates a new scanner instance
@@ -368,42 +360,6 @@ func (s *Scanner) Scan(
return result, nil return result, nil
} }
// DryRun walks path as Scan does and counts what a backup would store,
// without storing it. It reads the local index to tell unchanged files
// from new and changed ones, and writes nothing to the local index or
// the backup destination store. With listExcluded it prints each
// excluded entry and the exclude pattern that matched it.
func (s *Scanner) DryRun(
ctx context.Context, path string, listExcluded bool,
) (*ScanResult, error) {
s.currentSourcePath = path
s.listExcluded = listExcluded
result := &ScanResult{
StartTime: time.Now().UTC(),
}
knownFiles, err := s.loadDatabaseState(ctx, path)
if err != nil {
return nil, err
}
existingFiles := make(map[string]struct{})
_, err = s.scanPhase(ctx, path, result, existingFiles, knownFiles)
if err != nil {
return nil, fmt.Errorf("scan phase failed: %w", err)
}
err = s.detectDeletedFiles(ctx, path, existingFiles, result)
if err != nil {
return nil, fmt.Errorf("detecting deleted files: %w", err)
}
result.EndTime = time.Now().UTC()
return result, nil
}
// GetProgress returns the progress reporter for this scanner, or nil when // GetProgress returns the progress reporter for this scanner, or nil when
// progress is off. Scan neither starts nor stops it: the caller does, // progress is off. Scan neither starts nor stops it: the caller does,
// once for all the paths it scans, because a second Stop panics. // once for all the paths it scans, because a second Stop panics.
@@ -988,8 +944,12 @@ func (s *Scanner) scanPhase(
// or cannot be looked up is neither counted nor reported. For // or cannot be looked up is neither counted nor reported. For
// directories, skip the entire subtree. info is nil when the // directories, skip the entire subtree. info is nil when the
// walk could not look the entry up. // walk could not look the entry up.
if excluded, walkErr := s.skipExcluded(filePath, path, info, result); excluded { if s.shouldExclude(filePath, path) {
return walkErr if info != nil && info.IsDir() {
return filepath.SkipDir
}
return nil
} }
// The source path is in no directory listing, so its absence is // The source path is in no directory listing, so its absence is
@@ -1056,32 +1016,6 @@ func (s *Scanner) scanPhase(
}, nil }, nil
} }
// skipExcluded reports whether the walk skips filePath, under rootPath,
// because it is excluded, and counts it if so. The error is what the walk
// function returns for it: filepath.SkipDir for a directory, so the walk
// does not enter it, and nil otherwise. info is nil when the walk could
// not look the entry up.
func (s *Scanner) skipExcluded(
filePath, rootPath string, info os.FileInfo, result *ScanResult,
) (bool, error) {
pattern, excluded := s.excludingPattern(filePath, rootPath)
if !excluded {
return false, nil
}
result.EntriesExcluded++
if s.listExcluded {
s.printExcluded(filePath, pattern)
}
if info != nil && info.IsDir() {
return true, filepath.SkipDir
}
return true, nil
}
// countNonDirectories returns how many of files are not directories. // countNonDirectories returns how many of files are not directories.
func countNonDirectories(files map[string]*database.File) int64 { func countNonDirectories(files map[string]*database.File) int64 {
var count int64 var count int64
@@ -2417,32 +2351,23 @@ func compileExcludePatterns(patterns []string) ([]compiledPattern, error) {
// filePath is the full path to the file // filePath is the full path to the file
// rootPath is the root of the backup source directory // rootPath is the root of the backup source directory
func (s *Scanner) shouldExclude(filePath, rootPath string) bool { func (s *Scanner) shouldExclude(filePath, rootPath string) bool {
_, excluded := s.excludingPattern(filePath, rootPath)
return excluded
}
// excludingPattern reports whether shouldExclude excludes filePath and
// returns the exclude pattern, as configured, that matches it. The
// pattern is "" for one of vaultik's own working files.
func (s *Scanner) excludingPattern(filePath, rootPath string) (string, bool) {
if _, ok := s.indexAndLockFiles[filePath]; ok { if _, ok := s.indexAndLockFiles[filePath]; ok {
return "", true return true
} }
if len(s.compiledExclude) == 0 { if len(s.compiledExclude) == 0 {
return "", false return false
} }
// Get the relative path from root // Get the relative path from root
relPath, err := filepath.Rel(rootPath, filePath) relPath, err := filepath.Rel(rootPath, filePath)
if err != nil { if err != nil {
return "", false return false
} }
// Never exclude the root directory itself // Never exclude the root directory itself
if relPath == "." { if relPath == "." {
return "", false return false
} }
// Normalize path separators // Normalize path separators
@@ -2451,26 +2376,11 @@ func (s *Scanner) excludingPattern(filePath, rootPath string) (string, bool) {
// Check each pattern // Check each pattern
for _, cp := range s.compiledExclude { for _, cp := range s.compiledExclude {
if cp.matches(relPath) { if cp.matches(relPath) {
return cp.original, true return true
} }
} }
return "", false return false
}
// printExcluded prints that the walk excluded filePath and why: pattern
// is the exclude pattern that matched it, "" for one of vaultik's own
// working files.
func (s *Scanner) printExcluded(filePath, pattern string) {
if pattern == "" {
s.ui.Infof("Excluded %s: vaultik's own index or lock file.",
s.ui.Path(filePath))
return
}
s.ui.Infof("Excluded %s: matches exclude pattern %q.",
s.ui.Path(filePath), pattern)
} }
// matches reports whether the compiled pattern matches the given // matches reports whether the compiled pattern matches the given
+5 -3
View File
@@ -200,9 +200,11 @@ func (f *Storer) ListStream(
return f.inner.ListStream(ctx, prefix) return f.inner.ListStream(ctx, prefix)
} }
// DeletePartialUploads delegates unchanged. // ListPartialUploads delegates unchanged.
func (f *Storer) DeletePartialUploads(ctx context.Context, prefix string) error { func (f *Storer) ListPartialUploads(
return f.inner.DeletePartialUploads(ctx, prefix) ctx context.Context, prefix string,
) ([]storage.ObjectInfo, error) {
return f.inner.ListPartialUploads(ctx, prefix)
} }
// Info delegates unchanged. // Info delegates unchanged.
+26 -13
View File
@@ -56,8 +56,8 @@ const storageDirPerm = 0o755
// truncated object at the key a later run would Stat and trust as a complete // truncated object at the key a later run would Stat and trust as a complete
// blob. The rclone backend's upload does the same on remotes with a // blob. The rclone backend's upload does the same on remotes with a
// server-side move. List and ListStream skip these files, so a leftover from // server-side move. List and ListStream skip these files, so a leftover from
// an interrupted write is never listed or trusted as a blob; it is otherwise // an interrupted write is never listed or trusted as a blob; prune deletes
// harmless. // it once it is older than the prune grace period.
const tempSuffix = ".partial" const tempSuffix = ".partial"
// Put stores data at the specified key. // Put stores data at the specified key.
@@ -248,22 +248,24 @@ func (f *FileStorer) ListStream(ctx context.Context, prefix string) <-chan Objec
return ch return ch
} }
// DeletePartialUploads removes every file under prefix whose name ends in // ListPartialUploads returns each file under prefix whose name ends in
// tempSuffix. Like List, it reads through symlinked directories and fails // tempSuffix. Like List, it reads through symlinked directories and fails
// on one whose target is missing. A missing prefix has none to remove. // on one whose target is missing. A missing prefix has none.
func (f *FileStorer) DeletePartialUploads(ctx context.Context, prefix string) error { func (f *FileStorer) ListPartialUploads(
basePath := f.fullPath(prefix) ctx context.Context, prefix string,
) ([]ObjectInfo, error) {
exists, err := f.prefixExists(prefix) exists, err := f.prefixExists(prefix)
if err != nil { if err != nil {
return fmt.Errorf("checking path: %w", err) return nil, fmt.Errorf("checking path: %w", err)
} }
if !exists { if !exists {
return nil return nil, nil
} }
err = f.walk(basePath, func(path string, info os.FileInfo, err error) error { var partial []ObjectInfo
err = f.walk(f.fullPath(prefix), func(path string, info os.FileInfo, err error) error {
if err != nil { if err != nil {
return err return err
} }
@@ -276,13 +278,24 @@ func (f *FileStorer) DeletePartialUploads(ctx context.Context, prefix string) er
return nil return nil
} }
return f.fs.Remove(path) relPath, err := filepath.Rel(f.basePath, path)
})
if err != nil { if err != nil {
return fmt.Errorf("walking directory: %w", err) return fmt.Errorf("computing relative path: %w", err)
} }
partial = append(partial, ObjectInfo{
Key: strings.ReplaceAll(relPath, string(filepath.Separator), "/"),
Size: info.Size(),
ModTime: info.ModTime(),
})
return nil return nil
})
if err != nil {
return nil, fmt.Errorf("walking directory: %w", err)
}
return partial, nil
} }
// Info returns human-readable storage location information. // Info returns human-readable storage location information.
+32 -8
View File
@@ -121,9 +121,10 @@ func TestFileStorer_ListSkipsPartialFiles(t *testing.T) {
} }
} }
// TestFileStorer_DeletePartialUploads checks that a leftover temp file is // TestFileStorer_ListPartialUploads checks that a leftover temp file is
// removed and the object at the real key is kept. // listed with its size and time, that the object at the real key is not,
func TestFileStorer_DeletePartialUploads(t *testing.T) { // and that Delete removes the leftover by the listed key.
func TestFileStorer_ListPartialUploads(t *testing.T) {
t.Parallel() t.Parallel()
base := t.TempDir() base := t.TempDir()
@@ -133,23 +134,46 @@ func TestFileStorer_DeletePartialUploads(t *testing.T) {
t.Fatalf("NewFileStorer: %v", err) t.Fatalf("NewFileStorer: %v", err)
} }
checkListPartialUploads(t, f, base)
}
// checkListPartialUploads stores an object and writes a leftover temp
// file next to it in the directory base that s stores into. It checks
// that ListPartialUploads lists only the leftover, with its size and a
// time, and that Delete removes it by the listed key and keeps the
// object.
func checkListPartialUploads(t *testing.T, s storage.Storer, base string) {
t.Helper()
ctx := context.Background() ctx := context.Background()
err = f.Put(ctx, testBlobKey, strings.NewReader("blob-bytes")) err := s.Put(ctx, testBlobKey, strings.NewReader("blob-bytes"))
if err != nil { if err != nil {
t.Fatalf("Put: %v", err) t.Fatalf("Put: %v", err)
} }
leftover := filepath.Join(base, testBlobKey+"-123456.partial") leftoverKey := testBlobKey + "-123456.partial"
leftover := filepath.Join(base, leftoverKey)
err = os.WriteFile(leftover, []byte("half"), 0o600) err = os.WriteFile(leftover, []byte("half"), 0o600)
if err != nil { if err != nil {
t.Fatalf("writing leftover temp file: %v", err) t.Fatalf("writing leftover temp file: %v", err)
} }
err = f.DeletePartialUploads(ctx, "blobs/") partial, err := s.ListPartialUploads(ctx, "blobs/")
if err != nil { if err != nil {
t.Fatalf("DeletePartialUploads: %v", err) t.Fatalf("ListPartialUploads: %v", err)
}
if len(partial) != 1 || partial[0].Key != leftoverKey ||
partial[0].Size != int64(len("half")) || partial[0].ModTime.IsZero() {
t.Fatalf("ListPartialUploads = %+v, want only %s, 4 bytes, "+
"with a time", partial, leftoverKey)
}
err = s.Delete(ctx, partial[0].Key)
if err != nil {
t.Fatalf("Delete: %v", err)
} }
_, err = os.Stat(leftover) _, err = os.Stat(leftover)
@@ -157,7 +181,7 @@ func TestFileStorer_DeletePartialUploads(t *testing.T) {
t.Errorf("leftover temp file was not removed: %v", err) t.Errorf("leftover temp file was not removed: %v", err)
} }
_, err = f.Stat(ctx, testBlobKey) _, err = s.Stat(ctx, testBlobKey)
if err != nil { if err != nil {
t.Errorf("Stat of the real key: %v", err) t.Errorf("Stat of the real key: %v", err)
} }
+13 -18
View File
@@ -239,11 +239,11 @@ func TestFileStorerListFailsOnSymlinkLoop(t *testing.T) {
} }
} }
// TestFileStorerDeletePartialUploadsThroughSymlinkedDirectory checks that // TestFileStorerListPartialUploadsThroughSymlinkedDirectory checks that
// DeletePartialUploads, which remote nuke uses, removes a leftover temp // ListPartialUploads, which prune and remote nuke use, lists a leftover
// file behind a symlinked metadata/ or blobs/ directory and keeps the // temp file behind a symlinked metadata/ or blobs/ directory under its
// object at the real key. // key, and leaves out the object at the real key.
func TestFileStorerDeletePartialUploadsThroughSymlinkedDirectory(t *testing.T) { func TestFileStorerListPartialUploadsThroughSymlinkedDirectory(t *testing.T) {
t.Parallel() t.Parallel()
ctx := context.Background() ctx := context.Background()
@@ -265,29 +265,24 @@ func TestFileStorerDeletePartialUploadsThroughSymlinkedDirectory(t *testing.T) {
t.Fatalf("Put: %v", err) t.Fatalf("Put: %v", err)
} }
leftover := filepath.Join(base, key+"-123456.partial") leftoverKey := key + "-123456.partial"
err = os.WriteFile(leftover, []byte("half"), 0o600) err = os.WriteFile(filepath.Join(base, leftoverKey),
[]byte("half"), 0o600)
if err != nil { if err != nil {
t.Fatalf("writing leftover temp file: %v", err) t.Fatalf("writing leftover temp file: %v", err)
} }
moveBehindSymlink(t, base, dir) moveBehindSymlink(t, base, dir)
err = s.DeletePartialUploads(ctx, dir+"/") partial, err := s.ListPartialUploads(ctx, dir+"/")
if err != nil { if err != nil {
t.Fatalf("DeletePartialUploads: %v", err) t.Fatalf("ListPartialUploads: %v", err)
} }
// leftover runs through the link. if len(partial) != 1 || partial[0].Key != leftoverKey {
_, err = os.Stat(leftover) t.Errorf("ListPartialUploads = %+v, want only %s",
if !os.IsNotExist(err) { partial, leftoverKey)
t.Errorf("leftover temp file was not removed: %v", err)
}
_, err = s.Stat(ctx, key)
if err != nil {
t.Errorf("Stat of the real key: %v", err)
} }
}) })
} }
+13 -14
View File
@@ -233,29 +233,28 @@ func (r *RcloneStorer) ListStream(
return ch return ch
} }
// DeletePartialUploads removes every object under prefix whose name ends // ListPartialUploads returns each object under prefix whose name ends in
// in tempSuffix. // tempSuffix.
func (r *RcloneStorer) DeletePartialUploads(ctx context.Context, prefix string) error { func (r *RcloneStorer) ListPartialUploads(
var partial []fs.Object ctx context.Context, prefix string,
) ([]ObjectInfo, error) {
var partial []ObjectInfo
err := operations.ListFn(ctx, r.fsys, func(obj fs.Object) { err := operations.ListFn(ctx, r.fsys, func(obj fs.Object) {
key := obj.Remote() key := obj.Remote()
if strings.HasPrefix(key, prefix) && strings.HasSuffix(key, tempSuffix) { if strings.HasPrefix(key, prefix) && strings.HasSuffix(key, tempSuffix) {
partial = append(partial, obj) partial = append(partial, ObjectInfo{
Key: key,
Size: obj.Size(),
ModTime: obj.ModTime(ctx),
})
} }
}) })
if err != nil { if err != nil {
return fmt.Errorf("listing objects: %w", err) return nil, fmt.Errorf("listing objects: %w", err)
} }
for _, obj := range partial { return partial, nil
err = obj.Remove(ctx)
if err != nil {
return fmt.Errorf("removing object: %w", err)
}
}
return nil
} }
// Info returns human-readable storage location information. // Info returns human-readable storage location information.
+7 -31
View File
@@ -205,45 +205,21 @@ func TestRcloneStorerListSkipsPartialFiles(t *testing.T) {
} }
} }
// TestRcloneStorerDeletePartialUploads checks that a temporary file left // TestRcloneStorerListPartialUploads checks that a temporary file left by
// by a killed upload is removed and the object at the real key is kept. // a killed upload is listed with its size and time, that the object at
// the real key is not, and that Delete removes the leftover by the listed
// key.
// //
//nolint:paralleltest // NewRcloneStorer installs the process-global rclone config //nolint:paralleltest // NewRcloneStorer installs the process-global rclone config
func TestRcloneStorerDeletePartialUploads(t *testing.T) { func TestRcloneStorerListPartialUploads(t *testing.T) {
dir := t.TempDir() dir := t.TempDir()
ctx := context.Background()
s, err := storage.NewRcloneStorer(ctx, ":local", dir) s, err := storage.NewRcloneStorer(context.Background(), ":local", dir)
if err != nil { if err != nil {
t.Fatalf("NewRcloneStorer: %v", err) t.Fatalf("NewRcloneStorer: %v", err)
} }
err = s.Put(ctx, testBlobKey, strings.NewReader("blob-bytes")) checkListPartialUploads(t, s, dir)
if err != nil {
t.Fatalf("Put: %v", err)
}
leftover := filepath.Join(dir, testBlobKey+"-123456.partial")
err = os.WriteFile(leftover, []byte("half"), 0o600)
if err != nil {
t.Fatalf("writing leftover temp file: %v", err)
}
err = s.DeletePartialUploads(ctx, "blobs/")
if err != nil {
t.Fatalf("DeletePartialUploads: %v", err)
}
_, err = os.Stat(leftover)
if !os.IsNotExist(err) {
t.Errorf("leftover temp file was not removed: %v", err)
}
_, err = s.Stat(ctx, testBlobKey)
if err != nil {
t.Errorf("Stat of the real key: %v", err)
}
} }
// newRcloneStorerOnWrappedLocal registers name as rclone's local backend // newRcloneStorerOnWrappedLocal registers name as rclone's local backend
+6 -4
View File
@@ -100,10 +100,12 @@ func (s *S3Storer) ListStream(ctx context.Context, prefix string) <-chan ObjectI
return ch return ch
} }
// DeletePartialUploads has nothing to remove: S3 shows an object only once // ListPartialUploads finds none: S3 shows an object only once its upload
// its upload has completed, so an upload cut off part-way leaves none. // has completed, so an upload cut off part-way leaves none.
func (s *S3Storer) DeletePartialUploads(_ context.Context, _ string) error { func (s *S3Storer) ListPartialUploads(
return nil _ context.Context, _ string,
) ([]ObjectInfo, error) {
return nil, nil
} }
// Info returns human-readable storage location information. // Info returns human-readable storage location information.
+6 -5
View File
@@ -77,11 +77,12 @@ type Storer interface {
// If an error occurs during listing, the final item will have Err set. // If an error occurs during listing, the final item will have Err set.
ListStream(ctx context.Context, prefix string) <-chan ObjectInfo ListStream(ctx context.Context, prefix string) <-chan ObjectInfo
// DeletePartialUploads removes every object under prefix that an // ListPartialUploads returns each object under prefix that an upload
// upload cut off part-way left under a temporary name ending in // cut off part-way left under a temporary name ending in `.partial`,
// `.partial`. The file and rclone backends' List and ListStream skip // with its key, size and modification time. The file and rclone
// such an object. // backends' List and ListStream skip such an object; Delete removes
DeletePartialUploads(ctx context.Context, prefix string) error // it.
ListPartialUploads(ctx context.Context, prefix string) ([]ObjectInfo, error)
// Info returns human-readable storage location information. // Info returns human-readable storage location information.
Info() Info Info() Info
@@ -258,8 +258,10 @@ func (s *stubLister) List(_ context.Context, _ string) ([]string, error) {
return nil, errStubUnused return nil, errStubUnused
} }
func (s *stubLister) DeletePartialUploads(_ context.Context, _ string) error { func (s *stubLister) ListPartialUploads(
return errStubUnused _ context.Context, _ string,
) ([]storage.ObjectInfo, error) {
return nil, errStubUnused
} }
func (s *stubLister) Info() storage.Info { func (s *stubLister) Info() storage.Info {
+40 -6
View File
@@ -216,6 +216,11 @@ type RemoteInfoResult struct {
OrphanedBlobCount *int `json:"orphaned_blob_count"` OrphanedBlobCount *int `json:"orphaned_blob_count"`
OrphanedBlobSize *int64 `json:"orphaned_blob_size"` OrphanedBlobSize *int64 `json:"orphaned_blob_size"`
// `.partial` objects under metadata/ and blobs/ that uploads cut off
// part-way left. The blob figures above leave them out.
PartialUploadCount int `json:"partial_upload_count"`
PartialUploadSize int64 `json:"partial_upload_size"`
// Remote key of each snapshot whose manifest could not be read // Remote key of each snapshot whose manifest could not be read
UnreadableManifests []string `json:"unreadable_manifests,omitempty"` UnreadableManifests []string `json:"unreadable_manifests,omitempty"`
@@ -279,6 +284,16 @@ func (v *Vaultik) RemoteInfo(jsonOutput bool) error {
return err return err
} }
partialUploads, err := v.listPartialUploads()
if err != nil {
return err
}
for _, object := range partialUploads {
result.PartialUploadCount++
result.PartialUploadSize += object.Size
}
log.Info("Remote info complete", log.Info("Remote info complete",
"snapshots", result.TotalMetadataCount, "snapshots", result.TotalMetadataCount,
"total_blobs", result.TotalBlobCount, "total_blobs", result.TotalBlobCount,
@@ -539,6 +554,12 @@ func (v *Vaultik) printRemoteInfoTable(result *RemoteInfoResult) {
"", "", ubytes(result.TotalMetadataSize)) "", "", ubytes(result.TotalMetadataSize))
} }
v.printRemoteBlobStorage(result)
}
// printRemoteBlobStorage renders the blob storage section of the
// human-readable remote info output.
func (v *Vaultik) printRemoteBlobStorage(result *RemoteInfoResult) {
v.stdoutf("\n=== Blob Storage ===\n") v.stdoutf("\n=== Blob Storage ===\n")
v.stdoutf("Total blobs on remote: %s (%s)\n", v.stdoutf("Total blobs on remote: %s (%s)\n",
humanize.Comma(int64(result.TotalBlobCount)), humanize.Comma(int64(result.TotalBlobCount)),
@@ -552,16 +573,29 @@ func (v *Vaultik) printRemoteInfoTable(result *RemoteInfoResult) {
"(%d manifest(s) could not be read, "+ "(%d manifest(s) could not be read, "+
"%d manifest(s) under a non-conforming name skipped)\n", "%d manifest(s) under a non-conforming name skipped)\n",
len(result.UnreadableManifests), result.SkippedManifestCount) len(result.UnreadableManifests), result.SkippedManifestCount)
} else {
return
}
v.stdoutf("Orphaned (unreferenced): %s (%s)\n", v.stdoutf("Orphaned (unreferenced): %s (%s)\n",
humanize.Comma(int64(*result.OrphanedBlobCount)), humanize.Comma(int64(*result.OrphanedBlobCount)),
ubytes(*result.OrphanedBlobSize)) ubytes(*result.OrphanedBlobSize))
}
if *result.OrphanedBlobCount > 0 { v.stdoutf("Partial uploads: %s (%s)\n",
v.stdoutf("\nRun 'vaultik prune' to remove orphaned blobs.\n") humanize.Comma(int64(result.PartialUploadCount)),
ubytes(result.PartialUploadSize))
var toRemove []string
if result.OrphanedBlobCount != nil && *result.OrphanedBlobCount > 0 {
toRemove = append(toRemove, "orphaned blobs")
}
if result.PartialUploadCount > 0 {
toRemove = append(toRemove, "partial uploads")
}
if len(toRemove) > 0 {
v.stdoutf("\nRun 'vaultik prune' to remove %s.\n",
strings.Join(toRemove, " and "))
} }
} }
+6 -4
View File
@@ -155,10 +155,12 @@ func (m *MockStorer) ListStream(
return ch return ch
} }
// DeletePartialUploads has nothing to remove: Put stores each object // ListPartialUploads finds none: Put stores each object under its key at
// under its key at once. // once.
func (m *MockStorer) DeletePartialUploads(_ context.Context, _ string) error { func (m *MockStorer) ListPartialUploads(
return nil _ context.Context, _ string,
) ([]storage.ObjectInfo, error) {
return nil, nil
} }
func (m *MockStorer) Info() storage.Info { func (m *MockStorer) Info() storage.Info {
+207 -40
View File
@@ -31,6 +31,11 @@ var errBlobDeleteFailed = errors.New("blob(s) could not be deleted")
var errMetadataDirDeleteFailed = errors.New( var errMetadataDirDeleteFailed = errors.New(
"metadata director(ies) with no manifest could not be deleted") "metadata director(ies) with no manifest could not be deleted")
// errPartialUploadDeleteFailed reports `.partial` objects prune could not
// delete.
var errPartialUploadDeleteFailed = errors.New(
"partial upload(s) could not be deleted")
// errObjectsLeft reports objects remote nuke still finds on the backup // errObjectsLeft reports objects remote nuke still finds on the backup
// destination store after deleting. // destination store after deleting.
var errObjectsLeft = errors.New( var errObjectsLeft = errors.New(
@@ -138,9 +143,9 @@ func (v *Vaultik) emptyDestinationStore() error {
// that fails is logged and the others still run; that object then // that fails is logged and the others still run; that object then
// shows up in the listing afterwards. // shows up in the listing afterwards.
func (v *Vaultik) deleteEverythingUnder(prefix string) ([]string, error) { func (v *Vaultik) deleteEverythingUnder(prefix string) ([]string, error) {
keys, err := v.Storage.List(v.ctx, prefix) keys, err := v.listEverythingUnder(prefix)
if err != nil { if err != nil {
return nil, fmt.Errorf("listing %s: %w", prefix, err) return nil, err
} }
for _, key := range keys { for _, key := range keys {
@@ -150,19 +155,47 @@ func (v *Vaultik) deleteEverythingUnder(prefix string) ([]string, error) {
} }
} }
// The file and rclone listings skip `.partial` objects, so the return v.listEverythingUnder(prefix)
// deletes above never reach them. }
err = v.Storage.DeletePartialUploads(v.ctx, prefix)
if err != nil {
return nil, fmt.Errorf("deleting partial uploads: %w", err)
}
left, err := v.Storage.List(v.ctx, prefix) // listEverythingUnder returns the key of every object under prefix,
// including the `.partial` objects uploads cut off part-way left, which
// the file and rclone backends' List skips.
func (v *Vaultik) listEverythingUnder(prefix string) ([]string, error) {
keys, err := v.Storage.List(v.ctx, prefix)
if err != nil { if err != nil {
return nil, fmt.Errorf("listing %s: %w", prefix, err) return nil, fmt.Errorf("listing %s: %w", prefix, err)
} }
return left, nil partialUploads, err := v.Storage.ListPartialUploads(v.ctx, prefix)
if err != nil {
return nil, fmt.Errorf("listing partial uploads under %s: %w",
prefix, err)
}
for _, object := range partialUploads {
keys = append(keys, object.Key)
}
return keys, nil
}
// listPartialUploads returns the `.partial` objects under metadata/ and
// blobs/ that uploads cut off part-way left.
func (v *Vaultik) listPartialUploads() ([]storage.ObjectInfo, error) {
var partialUploads []storage.ObjectInfo
for _, prefix := range []string{"metadata/", "blobs/"} {
objects, err := v.Storage.ListPartialUploads(v.ctx, prefix)
if err != nil {
return nil, fmt.Errorf("listing partial uploads under %s: %w",
prefix, err)
}
partialUploads = append(partialUploads, objects...)
}
return partialUploads, nil
} }
// PruneBlobsResult contains the result of a blob prune operation // PruneBlobsResult contains the result of a blob prune operation
@@ -182,11 +215,19 @@ type PruneBlobsResult struct {
// not delete. // not delete.
MetadataDirsDeleted int `json:"metadata_dirs_deleted"` MetadataDirsDeleted int `json:"metadata_dirs_deleted"`
MetadataDirsFailed int `json:"metadata_dirs_failed,omitempty"` MetadataDirsFailed int `json:"metadata_dirs_failed,omitempty"`
// PartialUploadsDeleted counts the `.partial` objects uploads cut off
// part-way left that prune deleted, and PartialUploadsFailed those it
// could not delete. PartialUploadsTooNew counts those kept for being
// newer than the prune grace period, or of unknown age.
PartialUploadsDeleted int `json:"partial_uploads_deleted"`
PartialUploadsFailed int `json:"partial_uploads_failed,omitempty"`
PartialUploadsTooNew int `json:"partial_uploads_too_new"`
} }
// Prune removes orphaned data from the local index database AND, from the // Prune removes orphaned data from the local index database AND, from the
// backup destination store, unreferenced blobs and the directories under // backup destination store, unreferenced blobs, the directories under
// metadata/ with no manifest. This is the single user-facing prune entry // metadata/ with no manifest and the `.partial` objects uploads cut off
// part-way left. This is the single user-facing prune entry
// point — the split between local and remote cleanup is an // point — the split between local and remote cleanup is an
// implementation detail. Calling code should prefer this method over // implementation detail. Calling code should prefer this method over
// PruneDatabase or PruneBlobs individually unless it specifically wants // PruneDatabase or PruneBlobs individually unless it specifically wants
@@ -214,8 +255,9 @@ func (v *Vaultik) Prune(opts *PruneOptions) error {
return v.PruneBlobs(opts) return v.PruneBlobs(opts)
} }
// PruneBlobs removes unreferenced blobs from storage, and each directory // PruneBlobs removes unreferenced blobs from storage, each directory
// under metadata/ that a backup left with no manifest. // under metadata/ that a backup left with no manifest, and each
// `.partial` object an upload cut off part-way left.
func (v *Vaultik) PruneBlobs(opts *PruneOptions) error { func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
log.Info("Starting prune operation") log.Info("Starting prune operation")
@@ -242,14 +284,21 @@ func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
return err return err
} }
partialUploads, partialTooNew, err := v.findOldPartialUploads()
if err != nil {
return err
}
result := &PruneBlobsResult{ result := &PruneBlobsResult{
BlobsFound: len(unreferencedBlobs), BlobsFound: len(unreferencedBlobs),
BlobsTooNew: tooNew, BlobsTooNew: tooNew,
PartialUploadsTooNew: partialTooNew,
} }
v.reportBlobsTooNew(tooNew, opts) v.reportTooNew(result, opts)
if len(unreferencedBlobs) == 0 && len(dirsWithoutManifest) == 0 { if len(unreferencedBlobs) == 0 && len(dirsWithoutManifest) == 0 &&
len(partialUploads) == 0 {
log.Info("No unreferenced blobs found") log.Info("No unreferenced blobs found")
if opts.JSON { if opts.JSON {
@@ -263,15 +312,17 @@ func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
log.Info("Found unreferenced blobs", log.Info("Found unreferenced blobs",
"count", len(unreferencedBlobs), "total_size", ubytes(totalSize), "count", len(unreferencedBlobs), "total_size", ubytes(totalSize),
"metadata_dirs_without_manifest", len(dirsWithoutManifest)) "metadata_dirs_without_manifest", len(dirsWithoutManifest),
"partial_uploads", len(partialUploads))
if !v.confirmPrune(opts, len(unreferencedBlobs), totalSize, if !v.confirmPrune(opts, len(unreferencedBlobs), totalSize,
dirsWithoutManifest) { dirsWithoutManifest, partialUploads) {
return ErrNotConfirmed return ErrNotConfirmed
} }
v.deleteUnreferencedBlobs(unreferencedBlobs, allBlobs, result) v.deleteUnreferencedBlobs(unreferencedBlobs, allBlobs, result)
v.deleteMetadataDirs(dirsWithoutManifest, result) v.deleteMetadataDirs(dirsWithoutManifest, result)
v.deletePartialUploads(partialUploads, result)
if opts.JSON { if opts.JSON {
err = v.outputPruneBlobsJSON(result) err = v.outputPruneBlobsJSON(result)
@@ -279,9 +330,16 @@ func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
return err return err
} }
} else { } else {
v.reportPruneDeletions(result, len(dirsWithoutManifest)) v.reportPruneDeletions(result, len(dirsWithoutManifest),
len(partialUploads))
} }
return pruneDeleteFailures(result)
}
// pruneDeleteFailures returns an error for each kind of object prune
// could not delete, or nil when every delete succeeded.
func pruneDeleteFailures(result *PruneBlobsResult) error {
var failures []error var failures []error
if result.BlobsFailed > 0 { if result.BlobsFailed > 0 {
@@ -294,26 +352,41 @@ func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
result.MetadataDirsFailed, errMetadataDirDeleteFailed)) result.MetadataDirsFailed, errMetadataDirDeleteFailed))
} }
if result.PartialUploadsFailed > 0 {
failures = append(failures, fmt.Errorf("%d %w",
result.PartialUploadsFailed, errPartialUploadDeleteFailed))
}
return errors.Join(failures...) return errors.Join(failures...)
} }
// confirmPrune states what prune found to delete, unreferenced blobs and // confirmPrune states what prune found to delete, unreferenced blobs,
// directories under metadata/ with no manifest, naming each directory by // directories under metadata/ with no manifest, naming each directory by
// its remote key, and, unless --force is given, asks the operator to // its remote key, and `.partial` objects, and, unless --force is given,
// confirm. Under --json it prints nothing and does not ask. It reports // asks the operator to confirm. Under --json it prints nothing and does
// whether to go ahead. // not ask. It reports whether to go ahead.
func (v *Vaultik) confirmPrune( func (v *Vaultik) confirmPrune(
opts *PruneOptions, blobCount int, blobsSize int64, dirs []string, opts *PruneOptions, blobCount int, blobsSize int64, dirs []string,
partialUploads []storage.ObjectInfo,
) bool { ) bool {
if opts.JSON { if opts.JSON {
return true return true
} }
var partialSize int64
for _, object := range partialUploads {
partialSize += object.Size
}
dirCount := len(dirs) dirCount := len(dirs)
partialCount := len(partialUploads)
blobsFound := fmt.Sprintf("Found %d unreferenced blob(s) totaling %s.", blobsFound := fmt.Sprintf("Found %d unreferenced blob(s) totaling %s.",
blobCount, ubytes(blobsSize)) blobCount, ubytes(blobsSize))
dirsFound := fmt.Sprintf("Found %d metadata director(ies) left by "+ dirsFound := fmt.Sprintf("Found %d metadata director(ies) left by "+
"backups that stopped before their manifest:", dirCount) "backups that stopped before their manifest:", dirCount)
partialFound := fmt.Sprintf("Found %d partial upload(s) totaling %s, "+
"left by uploads cut off part-way.", partialCount, ubytes(partialSize))
if opts.Force { if opts.Force {
if blobCount > 0 { if blobCount > 0 {
@@ -328,6 +401,10 @@ func (v *Vaultik) confirmPrune(
} }
} }
if partialCount > 0 {
v.UI.Infof("%s", partialFound)
}
return true return true
} }
@@ -346,16 +423,13 @@ func (v *Vaultik) confirmPrune(
} }
} }
switch { if partialCount > 0 {
case dirCount == 0: v.stdoutf("%s\n", partialFound)
v.stdoutf("\nDelete %d unreferenced blob(s)? [y/N] ", blobCount)
case blobCount == 0:
v.stdoutf("\nDelete %d metadata director(ies)? [y/N] ", dirCount)
default:
v.stdoutf("\nDelete %d unreferenced blob(s) and %d metadata "+
"director(ies)? [y/N] ", blobCount, dirCount)
} }
v.stdoutf("\nDelete %s? [y/N] ",
pruneTargets(blobCount, dirCount, partialCount))
var confirm string var confirm string
// A failure to read the answer means no confirmation. // A failure to read the answer means no confirmation.
@@ -369,9 +443,36 @@ func (v *Vaultik) confirmPrune(
return true return true
} }
// reportPruneDeletions says how many blobs and metadata directories // pruneTargets names what the prune prompt offers to delete, such as
// prune deleted, naming each kind only when it found some to delete. // "2 unreferenced blob(s) and 1 partial upload(s)", leaving out each kind
func (v *Vaultik) reportPruneDeletions(result *PruneBlobsResult, dirCount int) { // with a count of zero.
func pruneTargets(blobCount, dirCount, partialCount int) string {
var targets []string
if blobCount > 0 {
targets = append(targets,
fmt.Sprintf("%d unreferenced blob(s)", blobCount))
}
if dirCount > 0 {
targets = append(targets,
fmt.Sprintf("%d metadata director(ies)", dirCount))
}
if partialCount > 0 {
targets = append(targets,
fmt.Sprintf("%d partial upload(s)", partialCount))
}
return strings.Join(targets, " and ")
}
// reportPruneDeletions says how many blobs, metadata directories and
// partial uploads prune deleted, naming each kind only when it found some
// to delete.
func (v *Vaultik) reportPruneDeletions(
result *PruneBlobsResult, dirCount, partialCount int,
) {
if result.BlobsFound > 0 { if result.BlobsFound > 0 {
v.UI.Infof("Deleted %d blob(s) totaling %s.", v.UI.Infof("Deleted %d blob(s) totaling %s.",
result.BlobsDeleted, ubytes(result.BytesFreed)) result.BlobsDeleted, ubytes(result.BytesFreed))
@@ -381,6 +482,11 @@ func (v *Vaultik) reportPruneDeletions(result *PruneBlobsResult, dirCount int) {
v.UI.Infof("Deleted %d metadata director(ies) with no manifest.", v.UI.Infof("Deleted %d metadata director(ies) with no manifest.",
result.MetadataDirsDeleted) result.MetadataDirsDeleted)
} }
if partialCount > 0 {
v.UI.Infof("Deleted %d partial upload(s).",
result.PartialUploadsDeleted)
}
} }
// listMetadataObjects lists every object under metadata/ on the // listMetadataObjects lists every object under metadata/ on the
@@ -611,18 +717,79 @@ func (v *Vaultik) deleteMetadataDirs(
} }
} }
// reportBlobsTooNew says how many unreferenced blobs prune kept for being // reportTooNew says how many unreferenced blobs and partial uploads prune
// newer than the grace period or of unknown age. Under --json the // kept for being newer than the grace period or of unknown age. Under
// document carries the count instead. // --json the document carries the counts instead.
func (v *Vaultik) reportBlobsTooNew(tooNew int, opts *PruneOptions) { func (v *Vaultik) reportTooNew(result *PruneBlobsResult, opts *PruneOptions) {
if tooNew == 0 || opts.JSON { if opts.JSON {
return return
} }
if result.BlobsTooNew > 0 {
v.UI.Infof("Kept %d unreferenced blob(s) newer than "+ v.UI.Infof("Kept %d unreferenced blob(s) newer than "+
"prune_grace_period (%s), or of unknown age: a backup still "+ "prune_grace_period (%s), or of unknown age: a backup still "+
"running may need them.", "running may need them.",
tooNew, v.Config.PruneGracePeriod) result.BlobsTooNew, v.Config.PruneGracePeriod)
}
if result.PartialUploadsTooNew > 0 {
v.UI.Infof("Kept %d partial upload(s) newer than "+
"prune_grace_period (%s), or of unknown age: an upload may "+
"still be writing them.",
result.PartialUploadsTooNew, v.Config.PruneGracePeriod)
}
}
// findOldPartialUploads returns the `.partial` objects under metadata/
// and blobs/ that are older than the prune grace period. It also returns
// how many it left out for being newer than that, or of unknown age
// because the listing gave no time for them.
func (v *Vaultik) findOldPartialUploads() ([]storage.ObjectInfo, int, error) {
partialUploads, err := v.listPartialUploads()
if err != nil {
return nil, 0, err
}
cutoff := time.Now().UTC().Add(-v.Config.PruneGracePeriod)
var (
old []storage.ObjectInfo
tooNew int
)
for _, object := range partialUploads {
if object.ModTime.IsZero() || object.ModTime.After(cutoff) {
tooNew++
continue
}
old = append(old, object)
}
return old, tooNew, nil
}
// deletePartialUploads deletes the given `.partial` objects and counts
// in the result those deleted and those it could not delete.
func (v *Vaultik) deletePartialUploads(
partialUploads []storage.ObjectInfo, result *PruneBlobsResult,
) {
for _, object := range partialUploads {
err := v.Storage.Delete(v.ctx, object.Key)
if err != nil {
log.Error("Failed to delete partial upload",
"key", object.Key, "error", err)
result.PartialUploadsFailed++
continue
}
log.Info("Deleted partial upload", "key", object.Key)
result.PartialUploadsDeleted++
}
} }
// deleteUnreferencedBlobs deletes the given blobs from storage and // deleteUnreferencedBlobs deletes the given blobs from storage and
@@ -0,0 +1,157 @@
package vaultik_test
import (
"bytes"
"context"
"encoding/json"
"os"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/storage/faultstore"
"sneak.berlin/go/vaultik/internal/ui"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// partialUploadGracePeriod is the prune grace period of the vaultik
// newPartialUploadStore returns.
const partialUploadGracePeriod = time.Hour
// newPartialUploadStore returns a vaultik on a file:// destination
// holding the `.partial` files killed uploads leave: one under blobs/
// and one under metadata/ older than the prune grace period, and one
// under blobs/ newer than that. It also returns the buffer the vaultik
// writes its output to, the paths of the old files and the path of the
// new one.
func newPartialUploadStore(
t *testing.T,
) (*vaultik.Vaultik, *bytes.Buffer, []string, string) {
t.Helper()
storeDir := t.TempDir()
store, err := storage.NewFileStorer(storeDir)
require.NoError(t, err)
hash := testBlobHashA
oldFiles := []string{
filepath.Join(storeDir, "blobs", hash[:2], hash[2:4],
hash+"-123456.partial"),
filepath.Join(storeDir, metadataObjectKey(
"testhost_home_2026-01-01T00:00:00Z", "db.zst.age-123456.partial")),
}
newFile := filepath.Join(storeDir, "blobs", hash[:2], hash[2:4],
hash+"-654321.partial")
beforeGracePeriod := time.Now().Add(-partialUploadGracePeriod - time.Minute)
for _, path := range append([]string{newFile}, oldFiles...) {
require.NoError(t, os.MkdirAll(filepath.Dir(path), 0o750))
require.NoError(t, os.WriteFile(path, []byte("half an upload"), 0o600))
}
for _, path := range oldFiles {
require.NoError(t, os.Chtimes(path, beforeGracePeriod, beforeGracePeriod))
}
stdout := &bytes.Buffer{}
v := &vaultik.Vaultik{
Config: &config.Config{PruneGracePeriod: partialUploadGracePeriod},
Storage: store,
Stdout: stdout,
UI: ui.NewWithColor(stdout, false),
}
v.SetContext(context.Background())
return v, stdout, oldFiles, newFile
}
// TestPruneDeletesPartialUploadsOlderThanGracePeriod checks that prune
// deletes a `.partial` file a killed upload left under blobs/ or
// metadata/ once it is older than the grace period, keeps a newer one,
// and counts both in the --json document.
func TestPruneDeletesPartialUploadsOlderThanGracePeriod(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
v, stdout, oldFiles, newFile := newPartialUploadStore(t)
require.NoError(t, v.PruneBlobs(&vaultik.PruneOptions{JSON: true}))
for _, path := range oldFiles {
_, err := os.Stat(path)
require.ErrorIs(t, err, os.ErrNotExist, "%s must be deleted", path)
}
_, err := os.Stat(newFile)
require.NoError(t, err, "a partial upload newer than the grace period "+
"must be kept")
var result vaultik.PruneBlobsResult
require.NoError(t, json.Unmarshal(stdout.Bytes(), &result))
assert.Equal(t, 2, result.PartialUploadsDeleted)
assert.Equal(t, 1, result.PartialUploadsTooNew)
}
// TestPruneFailedPartialUploadDeleteReturnsError checks that prune
// returns an error, and counts the failures in the --json document, when
// the `.partial` files it found cannot be deleted.
func TestPruneFailedPartialUploadDeleteReturnsError(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
v, stdout, oldFiles, _ := newPartialUploadStore(t)
store := faultstore.New(v.Storage)
store.OnDelete = func(string) bool { return true }
v.Storage = store
err := v.PruneBlobs(&vaultik.PruneOptions{JSON: true})
require.ErrorContains(t, err, "2 partial upload(s) could not be deleted")
for _, path := range oldFiles {
_, statErr := os.Stat(path)
require.NoError(t, statErr, "%s must still exist", path)
}
var result vaultik.PruneBlobsResult
require.NoError(t, json.Unmarshal(stdout.Bytes(), &result),
"stdout must hold the JSON document, got:\n%s", stdout.String())
assert.Equal(t, 2, result.PartialUploadsFailed)
assert.Equal(t, 0, result.PartialUploadsDeleted)
}
// TestRemoteInfoCountsPartialUploads checks that remote info counts
// every `.partial` file under blobs/ and metadata/, whatever its age, and
// advises running prune.
func TestRemoteInfoCountsPartialUploads(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
v, stdout, _, _ := newPartialUploadStore(t)
require.NoError(t, v.RemoteInfo(true))
var result vaultik.RemoteInfoResult
require.NoError(t, json.Unmarshal(stdout.Bytes(), &result))
assert.Equal(t, 3, result.PartialUploadCount)
assert.Equal(t, int64(3*len("half an upload")), result.PartialUploadSize)
assert.Equal(t, 0, result.TotalBlobCount)
stdout.Reset()
require.NoError(t, v.RemoteInfo(false))
text := stdout.String()
assert.Contains(t, text, "Partial uploads: 3 (")
assert.Contains(t, text, "Run 'vaultik prune' to remove partial uploads.")
}
+6 -4
View File
@@ -136,10 +136,12 @@ func (s *testStorer) ListStream(
return ch return ch
} }
// DeletePartialUploads has nothing to remove: Put stores each object // ListPartialUploads finds none: Put stores each object under its key at
// under its key at once. // once.
func (s *testStorer) DeletePartialUploads(_ context.Context, _ string) error { func (s *testStorer) ListPartialUploads(
return nil _ context.Context, _ string,
) ([]storage.ObjectInfo, error) {
return nil, nil
} }
func (s *testStorer) Info() storage.Info { func (s *testStorer) Info() storage.Info {
+25 -51
View File
@@ -33,7 +33,6 @@ var (
"creating a snapshot needs at least one public key in " + "creating a snapshot needs at least one public key in " +
"age_recipients (generate a keypair with: age-keygen)") "age_recipients (generate a keypair with: age-keygen)")
errKeepNewerThanNeedsPrune = errors.New("--keep-newer-than requires --prune") errKeepNewerThanNeedsPrune = errors.New("--keep-newer-than requires --prune")
errDryRunWithPrune = errors.New("--dry-run cannot be used with --prune")
) )
// ErrPostBackupPruneFailed wraps a failure of the post-backup prune, // ErrPostBackupPruneFailed wraps a failure of the post-backup prune,
@@ -78,24 +77,12 @@ type SnapshotCreateOptions struct {
KeepNewerThan string KeepNewerThan string
SkipErrors bool // Skip file read errors (log them loudly but continue) SkipErrors bool // Skip file read errors (log them loudly but continue)
Snapshots []string // Optional list of snapshot names to process (empty = all) Snapshots []string // Optional list of snapshot names to process (empty = all)
// DryRun scans the snapshot source paths and prints what a backup
// would store, writing nothing to the local index or the backup
// destination store.
DryRun bool
// ListExcluded makes a dry run print each excluded entry and the
// exclude pattern that matched it. A backup ignores it.
ListExcluded bool
} }
// Validate fails when DryRun is set with Prune, or when KeepNewerThan is // Validate fails when KeepNewerThan is set without Prune or is not a
// set without Prune or is not a valid duration. Only the post-backup // valid duration. Only the post-backup prune reads KeepNewerThan, after
// prune reads KeepNewerThan, after the backup, so the caller checks it // the backup, so the caller checks it before starting one.
// before starting one.
func (opts *SnapshotCreateOptions) Validate() error { func (opts *SnapshotCreateOptions) Validate() error {
if opts.DryRun && opts.Prune {
return errDryRunWithPrune
}
if opts.KeepNewerThan == "" { if opts.KeepNewerThan == "" {
return nil return nil
} }
@@ -120,10 +107,6 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
return errNoAgeRecipients return errNoAgeRecipients
} }
if opts.DryRun {
return v.dryRunSnapshots(opts)
}
overallStartTime := time.Now() overallStartTime := time.Now()
log.Info("Starting snapshot creation", log.Info("Starting snapshot creation",
@@ -319,10 +302,10 @@ func (v *Vaultik) snapshotNamesToCreate(requested []string) ([]string, error) {
} }
// runPostBackupPrune drops older snapshots of the given names and removes // runPostBackupPrune drops older snapshots of the given names and removes
// from remote storage the orphan blobs and the directories under // from remote storage the orphan blobs, the directories under metadata/
// metadata/ with no manifest, as PruneBlobs does. If keepNewerThan is set // with no manifest and the `.partial` objects, as PruneBlobs does. If
// (e.g. "4w"), snapshots newer than that duration are kept. Otherwise only // keepNewerThan is set (e.g. "4w"), snapshots newer than that duration are
// the latest snapshot of each name is kept. // kept. Otherwise only the latest snapshot of each name is kept.
func (v *Vaultik) runPostBackupPrune( func (v *Vaultik) runPostBackupPrune(
snapshotNames []string, keepNewerThan string, snapshotNames []string, keepNewerThan string,
) error { ) error {
@@ -375,9 +358,6 @@ type snapshotStats struct {
// could not be read or listed. // could not be read or listed.
totalFilesUnreadable int totalFilesUnreadable int
// The entries the scan excluded; an excluded directory counts once.
totalEntriesExcluded int
// The configured paths in which the scan found nothing. // The configured paths in which the scan found nothing.
emptyPaths []string emptyPaths []string
@@ -391,29 +371,6 @@ type snapshotStats struct {
blobUncompressedSize int64 blobUncompressedSize int64
} }
// add adds the result of scanning the configured path dir to the totals.
func (stats *snapshotStats) add(dir string, result *snapshot.ScanResult) {
stats.totalFiles += result.FilesScanned
stats.totalBytes += result.BytesScanned
stats.totalChunks += result.ChunksCreated
stats.newChunkSize += result.NewChunkSize
stats.totalBlobs += result.BlobsCreated
stats.totalFilesSkipped += result.FilesSkipped
stats.totalFilesFailed += result.FilesFailed
stats.totalFilesUnreadable += result.FilesUnreadable
stats.totalBytesSkipped += result.BytesSkipped
stats.totalFilesDeleted += result.FilesDeleted
stats.totalBytesDeleted += result.BytesDeleted
stats.totalBlobsUploaded += result.BlobsUploaded
stats.totalBytesUploaded += result.BytesUploaded
stats.uploadDuration += result.UploadDuration
stats.totalEntriesExcluded += result.EntriesExcluded
if result.EntriesFound == 0 {
stats.emptyPaths = append(stats.emptyPaths, dir)
}
}
// createNamedSnapshot creates a single named snapshot and returns the // createNamedSnapshot creates a single named snapshot and returns the
// statistics of its scan. // statistics of its scan.
func (v *Vaultik) createNamedSnapshot( func (v *Vaultik) createNamedSnapshot(
@@ -607,7 +564,24 @@ func (v *Vaultik) scanAllDirectories(
return nil, fmt.Errorf("failed to scan %s: %w", dir, err) return nil, fmt.Errorf("failed to scan %s: %w", dir, err)
} }
stats.add(dir, result) stats.totalFiles += result.FilesScanned
stats.totalBytes += result.BytesScanned
stats.totalChunks += result.ChunksCreated
stats.newChunkSize += result.NewChunkSize
stats.totalBlobs += result.BlobsCreated
stats.totalFilesSkipped += result.FilesSkipped
stats.totalFilesFailed += result.FilesFailed
stats.totalFilesUnreadable += result.FilesUnreadable
stats.totalBytesSkipped += result.BytesSkipped
stats.totalFilesDeleted += result.FilesDeleted
stats.totalBytesDeleted += result.BytesDeleted
stats.totalBlobsUploaded += result.BlobsUploaded
stats.totalBytesUploaded += result.BytesUploaded
stats.uploadDuration += result.UploadDuration
if result.EntriesFound == 0 {
stats.emptyPaths = append(stats.emptyPaths, dir)
}
log.Info("Directory scan complete", log.Info("Directory scan complete",
"path", dir, "path", dir,
-133
View File
@@ -1,133 +0,0 @@
package vaultik
import (
"fmt"
"sneak.berlin/go/vaultik/internal/snapshot"
)
// dryRunSnapshots scans the source paths of each snapshot named in
// opts, or of every configured one, and prints what a backup would
// store. It writes nothing to the local index or the backup destination
// store. It first makes the read-only checks a backup makes, so it fails
// where a backup would: a local index bound to a different destination,
// or a destination that answers it has no marker while the local index
// holds snapshots stored there.
func (v *Vaultik) dryRunSnapshots(opts *SnapshotCreateOptions) error {
err := v.checkStorageBinding()
if err != nil {
return err
}
err = v.checkDestinationMarker()
if err != nil {
return err
}
// A backup first drops incomplete snapshots and the file records only
// they list (PruneDatabase), so it backs up those files again; the
// dry run counts them as unchanged.
incomplete, err := v.Repositories.Snapshots.GetIncompleteSnapshots(v.ctx)
if err != nil {
return fmt.Errorf("getting incomplete snapshots: %w", err)
}
if len(incomplete) > 0 {
v.UI.Noticef("The local index holds %s incomplete snapshot(s) from "+
"an interrupted backup. The next backup drops them first and may "+
"back up more files than this dry run counts.",
v.UI.Count(len(incomplete)))
}
snapshotNames, err := v.snapshotNamesToCreate(opts.Snapshots)
if err != nil {
return err
}
for _, snapName := range snapshotNames {
err = v.dryRunSnapshot(opts, snapName)
if err != nil {
return err
}
}
return nil
}
// dryRunSnapshot scans the source paths of the snapshot named snapName
// and prints what a backup of it would store.
func (v *Vaultik) dryRunSnapshot(
opts *SnapshotCreateOptions, snapName string,
) error {
resolvedDirs, err := v.resolveSnapshotPaths(snapName)
if err != nil {
return err
}
previousSnapshotID, err := v.previousSnapshotID(snapName)
if err != nil {
return err
}
scanner := v.ScannerFactory(snapshot.ScannerParams{
UI: v.UI,
Fs: v.Fs,
Exclude: v.Config.GetExcludes(snapName),
SkipErrors: opts.SkipErrors,
PreviousSnapshotID: previousSnapshotID,
})
v.UI.Beginf("Dry run of snapshot %s.", snapName)
stats := &snapshotStats{}
for i, dir := range resolvedDirs {
v.UI.Beginf("Enumerating snapshot source files in %s (%d of %d).",
v.UI.Path(dir), i+1, len(resolvedDirs))
result, err := scanner.DryRun(v.ctx, dir, opts.ListExcluded)
if err != nil {
return fmt.Errorf("failed to scan %s: %w", dir, err)
}
stats.add(dir, result)
}
v.printDryRunSummary(snapName, stats)
return nil
}
// printDryRunSummary prints what a backup of the snapshot named snapName
// would store, in the terms of printSnapshotSummary.
func (v *Vaultik) printDryRunSummary(snapName string, stats *snapshotStats) {
filesToBackUp := stats.totalFiles - stats.totalFilesSkipped -
stats.totalFilesFailed
v.UI.Completef("Dry run of snapshot %s complete; nothing was stored.",
snapName)
filesMsg := fmt.Sprintf("Files: %s examined, %s to back up, %s unchanged",
v.UI.Count(stats.totalFiles),
v.UI.Count(filesToBackUp),
v.UI.Count(stats.totalFilesSkipped))
if stats.totalFilesFailed > 0 {
filesMsg += fmt.Sprintf(", %s failed", v.UI.Count(stats.totalFilesFailed))
}
if stats.totalFilesDeleted > 0 {
filesMsg += fmt.Sprintf(", %s deleted", v.UI.Count(stats.totalFilesDeleted))
}
v.UI.Detailf("%s.", filesMsg)
dataMsg := fmt.Sprintf("Data: %s total (%s to back up)",
v.UI.Size(stats.totalBytes+stats.totalBytesSkipped),
v.UI.Size(stats.totalBytes))
if stats.totalBytesDeleted > 0 {
dataMsg += fmt.Sprintf(", %s deleted", v.UI.Size(stats.totalBytesDeleted))
}
v.UI.Detailf("%s.", dataMsg)
v.UI.Detailf("Excluded: %s entries.", v.UI.Count(stats.totalEntriesExcluded))
}
-69
View File
@@ -1,69 +0,0 @@
package vaultik_test
import (
"context"
"os"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// dryRunSnapshotName is the newSummaryEnv snapshot the dry run test
// backs up and then dry-runs.
const dryRunSnapshotName = "first"
// TestDryRunCountsWhatABackupWouldStore runs a dry run after a backup:
// the unchanged files count as unchanged, a rewritten one as to back up,
// and the dry run records no snapshot. See
// https://git.eeqj.de/sneak/vaultik/issues/436.
func TestDryRunCountsWhatABackupWouldStore(t *testing.T) {
t.Parallel()
env := newSummaryEnv(t)
env.backUp(t, dryRunSnapshotName, false)
dryRun := func() string {
env.out.Reset()
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
DryRun: true,
Snapshots: []string{dryRunSnapshotName},
}))
return env.out.String()
}
assert.Contains(t, dryRun(),
"Files: 3 examined, 0 to back up, 3 unchanged.")
require.NoError(t, os.WriteFile(env.aPath, []byte("rewritten"), 0o600))
assert.Contains(t, dryRun(),
"Files: 3 examined, 1 to back up, 2 unchanged.")
// localSnapshotID fails unless exactly one snapshot has the name.
localSnapshotID(context.Background(), t, env.repos, dryRunSnapshotName)
}
// TestDryRunNamesIncompleteSnapshots checks that a dry run says so when
// the local index holds an incomplete snapshot, which the next backup
// drops before it scans.
func TestDryRunNamesIncompleteSnapshots(t *testing.T) {
t.Parallel()
env := newSummaryEnv(t)
env.backUp(t, dryRunSnapshotName, false)
_, err := env.v.SnapshotManager.CreateSnapshotWithName(
context.Background(), testHostname, "split", "v", "g")
require.NoError(t, err)
env.out.Reset()
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
DryRun: true,
Snapshots: []string{dryRunSnapshotName},
}))
assert.Contains(t, env.out.String(),
"The local index holds 1 incomplete snapshot(s)")
}