2 Commits
Author SHA1 Message Date
clawbot cea4b0dd05 Do not prune after a backup that found a volume not mounted (closes #395)
check / check (push) Canceled after 0s
snapshot create --prune purged the older snapshots holding a volume's
files when the volume, mounted below a configured path, was not
mounted: its empty mount point read as deleted files.

The scan now records in the local index each directory at or below a
configured path that is on a different device from its parent
directory. Under --prune, a run that does not find such a directory
recorded by an earlier snapshot of that name skips the prune, keeps
every older snapshot, names the directory and its snapshot, and exits
non-zero, as for an empty configured path. A volume retired on purpose
is cleared by removing the older snapshots with snapshot purge.

Model: opus-5-5
2026-10-15 01:51:28 +00:00
clawbot ff50a137dd Delete metadata directories a backup left without a manifest (closes #390)
check / check (push) Canceled after 0s
A backup that stopped between uploading db.zst.age and its manifest
left that directory under metadata/ on the destination for good:
prune deleted nothing there.

The blob step of prune, which snapshot create --prune also runs, now
deletes each directory under metadata/ that has no manifest.json.zst,
that no local snapshot record claims, and all of whose objects are
older than prune_grace_period; an object listed with no time keeps its
directory, and only names that are remote keys are deleted. The prompt
and the --force output name each such directory, the --json result
counts them (metadata_dirs_deleted, metadata_dirs_failed), and a failed
delete fails the run. Every description of the grace period now
covers these directories.

Model: opus-5-5
2026-10-15 03:49:19 +02:00
26 changed files with 1233 additions and 163 deletions
+53 -29
View File
@@ -98,11 +98,12 @@ vaultik snapshot restore <snapshot-id> ~/vaultik-restore
On a destination shared between hosts or users, the prune in that cron line On a destination shared between hosts or users, the prune in that cron line
relies on `prune_grace_period` (default `24h`) to leave the blobs of a backup relies on `prune_grace_period` (default `24h`) to leave the blobs of a backup
another one is still running, so it must be longer than the longest backup another one is still running, and its directory under `metadata/` that has no
run. It does not protect a blob that backup reuses from an earlier run that manifest yet, so it must be longer than the longest backup run. It does not
stopped before its manifest: the prune can delete such a blob, and the new protect a blob that backup reuses from an earlier run that stopped before its
snapshot then cannot be restored. To not depend on the grace period, give each manifest: the prune can delete such a blob, and the new snapshot then cannot
host its own prefix or directory. See [locking](#locking). be restored. To not depend on the grace period, give each host its own prefix
or directory. See [locking](#locking).
## restoring on another machine ## restoring on another machine
@@ -204,18 +205,19 @@ corrupt the local index.
The lock covers one user on one machine. It does not stop another user, The lock covers one user on one machine. It does not stop another user,
or another host, from writing to the same destination store at the same or another host, from writing to the same destination store at the same
time. A backup uploads its blobs before the manifest that lists them, so a time. A backup uploads its blobs, and then its `db.zst.age`, before its
prune from elsewhere at that moment finds blobs no snapshot lists. It manifest, so a prune from elsewhere at that moment finds blobs no snapshot
deletes such a blob only once it is older than `prune_grace_period` lists and a directory under `metadata/` with no manifest. It
(default `24h`), which therefore must be longer than the longest backup run deletes such a blob only once it is older than `prune_grace_period` (default
of any host or user sharing the destination. The grace period does not `24h`), and such a directory only once every object in it is, so the grace
protect a blob a backup reuses from an earlier run that stopped before its period must be longer than the longest backup run of any host or user sharing
manifest: a backup skips the upload of a blob already on the destination, so the destination. The grace period does not protect a blob a backup reuses
that blob keeps the time of its first upload, and a prune from elsewhere from an earlier run that stopped before its manifest: a backup skips the
during the backup can delete it; the new snapshot then cannot be restored. upload of a blob already on the destination, so that blob keeps the time of
`remote nuke` deletes everything regardless. To not depend on the grace its first upload, and a prune from elsewhere during the backup can delete it;
period, give each host or user its own prefix (or directory) on the the new snapshot then cannot be restored. `remote nuke` deletes everything
destination. regardless. To not depend on the grace period, give each host or user its own
prefix (or directory) on the destination.
Read-only commands — `info`, `snapshot list`, `snapshot verify`, and Read-only commands — `info`, `snapshot list`, `snapshot verify`, and
`remote info` — do not take the lock and are never blocked, so they run `remote info` — do not take the lock and are never blocked, so they run
@@ -318,9 +320,12 @@ value where a list belongs or a word where a number belongs), is refused.
error that explains how to fix it error that explains how to fix it
* `--cron`: Silent on total success; warnings and errors are still printed * `--cron`: Silent on total success; warnings and errors are still printed
(for crontab) (for crontab)
* `--prune`: After backup, drop older snapshots of each backed-up name and * `--prune`: After backup, drop older snapshots of each backed-up name,
remove orphaned blobs older than `prune_grace_period` from remote storage, remove orphaned blobs older than `prune_grace_period` from remote storage,
as `prune` does. By default keeps only the latest and delete each directory under `metadata/` that has no
`manifest.json.zst` once every object in it is older than
`prune_grace_period` and its snapshot is not in the local index, as `prune`
does. By default keeps only the latest
snapshot per name; use `--keep-newer-than` for a rolling window. When snapshot per name; use `--keep-newer-than` for a rolling window. When
`--skip-errors` skipped a file or directory it could not read or list, the `--skip-errors` skipped a file or directory it could not read or list, the
prune does not run and every older snapshot is kept, since one of them may prune does not run and every older snapshot is kept, since one of them may
@@ -330,8 +335,16 @@ value where a list belongs or a word where a number belongs), is refused.
either: every older snapshot is kept, and the command names the path and either: every older snapshot is kept, and the command names the path and
exits non-zero, on every run until the path holds something again. Take a exits non-zero, on every run until the path holds something again. Take a
path that is empty on purpose out of the config, or remove the older path that is empty on purpose out of the config, or remove the older
snapshots by hand with `snapshot purge`. A file removed during the backup snapshots by hand with `snapshot purge`. When an earlier snapshot of the
does not stop the prune. same name in the local index found a volume mounted at or below a
configured path (a directory on a different filesystem from its parent)
and this backup finds none mounted there, the prune does not run either:
every older snapshot is kept, and the command names the directory and
exits non-zero, on every run until the volume is mounted again. The same
happens when that directory is no longer backed up, because it is excluded
or its configured path was removed. For a volume retired on purpose, remove
the older snapshots by hand with `snapshot purge`. A file removed during
the backup does not stop the prune.
* `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than * `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than
this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`). this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`).
Without `--prune`, or with a value that is not a valid duration, the Without `--prune`, or with a value that is not a valid duration, the
@@ -463,22 +476,29 @@ may belong to a backup still running, on this host or another sharing the
destination store, that has not yet uploaded its manifest. The grace period destination store, that has not yet uploaded its manifest. The grace period
does not protect a blob such a backup reuses from an earlier run that stopped does not protect a blob such a backup reuses from an earlier run that stopped
before its manifest: that blob keeps the time of its first upload, so it can before its manifest: that blob keeps the time of its first upload, so it can
be deleted. Step (3) reads all remote manifests — network cost scales with be deleted. Step (3) also deletes each directory under `metadata/` that has
the number of snapshots. no `manifest.json.zst`, as a backup leaves when it stops after uploading
`db.zst.age` but before its manifest, once every object in it is older than
`prune_grace_period` and its snapshot is not in the local index. An object the
listing gives no time for keeps its directory, and a directory whose name is
not a remote key is left alone. Step (3) reads all remote manifests — network
cost scales with the number of snapshots.
`snapshot create --prune` runs the same cleanup automatically; this is the `snapshot create --prune` runs the same cleanup automatically; this is the
manual entry point for the same work. If the destination store cannot manual entry point for the same work. If the destination store cannot
be listed, or the local index holds snapshots and the destination answers be listed, or the local index holds snapshots and the destination answers
that it has no `vaultik-destination.txt` (an unplugged volume's empty mount that it has no `vaultik-destination.txt` (an unplugged volume's empty mount
point), prune fails and deletes nothing. `snapshot purge` does the same. If a point), prune fails and deletes nothing. `snapshot purge` does the same. If a
blob cannot be deleted, the command (and `snapshot create --prune`) blob or such a directory cannot be deleted, the command (and
exits non-zero. `snapshot create --prune`) exits non-zero.
* `--force`: Skip confirmation prompt * `--force`: Skip confirmation prompt
* `--json`: Output stats as JSON. Also skips the confirmation prompt, as * `--json`: Output stats as JSON. Also skips the confirmation prompt, as
`--force` does. A blob that cannot be deleted is counted in `--force` does. A blob that cannot be deleted is counted in
`blobs_failed`, and the document is printed before the non-zero exit. `blobs_failed`, and the document is printed before the non-zero exit.
A blob kept for being newer than `prune_grace_period`, or because the A blob kept for being newer than `prune_grace_period`, or because the
listing of the destination gives no time for it, is counted in listing of the destination gives no time for it, is counted in
`blobs_too_new` and not in `blobs_found`. `blobs_too_new` and not in `blobs_found`. A directory under `metadata/`
with no manifest is counted in `metadata_dirs_deleted` once deleted, or
in `metadata_dirs_failed` if it cannot be.
**`info`**: Display system configuration, storage settings, encryption **`info`**: Display system configuration, storage settings, encryption
recipients, and local database statistics. recipients, and local database statistics.
@@ -555,8 +575,9 @@ and can be deleted.
A destination shared between hosts or users, each with its own local index, A destination shared between hosts or users, each with its own local index,
relies on `prune_grace_period` to keep one's prune from deleting the blobs of relies on `prune_grace_period` to keep one's prune from deleting the blobs of
another's backup still running, so it must be longer than the longest backup another's backup still running, or its directory under `metadata/` that has
run. It does not protect a blob that backup reuses from an earlier run that no manifest yet, so it must be longer than the longest backup run. It does not
protect a blob that backup reuses from an earlier run that
stopped before its manifest, which the prune can delete. To not depend on the stopped before its manifest, which the prune can delete. To not depend on the
grace period, give each host its own prefix (or directory). See grace period, give each host its own prefix (or directory). See
[locking](#locking). [locking](#locking).
@@ -656,6 +677,9 @@ format does and does not protect.
3. List all blobs in storage 3. List all blobs in storage
4. Delete any blob not in the referenced set that is older than 4. Delete any blob not in the referenced set that is older than
`prune_grace_period` `prune_grace_period`
5. Delete each directory under `metadata/` with no manifest whose objects
are all older than `prune_grace_period`, unless its snapshot is in the
local index
### chunking and deduplication ### chunking and deduplication
@@ -700,7 +724,7 @@ complete annotated example also lives in
| `chunk_size` | `10MB` | Average chunk size for content-defined chunking | | `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` | | `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` |
| `compression_level` | `3` | zstd compression level (1-19) | | `compression_level` | `3` | zstd compression level (1-19) |
| `prune_grace_period` | `24h` | How old a blob that no snapshot lists must be before `prune` deletes it. A backup uploads its blobs before the manifest that lists them, so on a destination shared between hosts or users this must be longer than the longest backup run (see [locking](#locking)). It does not protect a blob a backup reuses from an earlier run that stopped before its manifest: that blob keeps the time of its first upload, so a prune can delete it. A duration such as `36h` or `90m`; `0s` deletes such blobs at once | | `prune_grace_period` | `24h` | How old a blob that no snapshot lists must be before `prune` deletes it, and how old every object in a directory under `metadata/` with no manifest must be before `prune` deletes that directory. A backup uploads its blobs and its `db.zst.age` before its manifest, so on a destination shared between hosts or users this must be longer than the longest backup run (see [locking](#locking)). It does not protect a blob a backup reuses from an earlier run that stopped before its manifest: that blob keeps the time of its first upload, so a prune can delete it. A duration such as `36h` or `90m`; `0s` deletes such blobs and directories at once |
| `hostname` | system hostname | Hostname used in snapshot IDs | | `hostname` | system hostname | Hostname used in snapshot IDs |
| `index_path` | platform data dir | Local SQLite index path | | `index_path` | platform data dir | Local SQLite index path |
+8
View File
@@ -24,6 +24,14 @@ owner's; a `v*` tag runs the release build in
# Completed Steps # Completed Steps
- 2026-10-15: Stopped `snapshot create --prune` pruning after a backup
in which a volume mounted below a configured path in an earlier
snapshot of the same name was not mounted
([issue #395](https://git.eeqj.de/sneak/vaultik/issues/395)). Such a
prune could delete every snapshot still holding the volume's files.
Each snapshot now records in the local index the directories it found
on a different filesystem from their parent.
- 2026-10-14: Stopped `snapshot create --prune` pruning after a backup - 2026-10-14: Stopped `snapshot create --prune` pruning after a backup
that found nothing in a configured path that found nothing in a configured path
([issue #388](https://git.eeqj.de/sneak/vaultik/issues/388)), as in ([issue #388](https://git.eeqj.de/sneak/vaultik/issues/388)), as in
+5 -3
View File
@@ -329,14 +329,16 @@ storage_url: "rclone://myremote/path/to/backups"
# Default: 3 # Default: 3
compression_level: 5 compression_level: 5
# How old a blob that no snapshot lists must be before prune deletes it # How old a blob that no snapshot lists must be before prune deletes it, and
# A backup uploads its blobs before the manifest that lists them, so on a # how old every object in a directory under metadata/ with no manifest must
# be before prune deletes that directory
# A backup uploads its blobs and its db.zst.age before its manifest, so on a
# destination shared between hosts or users this must be longer than the # destination shared between hosts or users this must be longer than the
# longest backup run # longest backup run
# It does not protect a blob a backup reuses from an earlier run that stopped # It does not protect a blob a backup reuses from an earlier run that stopped
# before its manifest: that blob keeps the time of its first upload, so a # before its manifest: that blob keeps the time of its first upload, so a
# prune can delete it # prune can delete it
# Supports: 36h, 90m, etc.; 0s deletes them at once # Supports: 36h, 90m, etc.; 0s deletes such blobs and directories at once
# Default: 24h # Default: 24h
#prune_grace_period: 24h #prune_grace_period: 24h
+16 -1
View File
@@ -153,7 +153,17 @@ Maps snapshots to the blobs they reference.
**Purpose:** Tracks blob dependencies for snapshots and enables manifest generation. **Purpose:** Tracks blob dependencies for snapshots and enables manifest generation.
### 10. `uploads` ### 10. `snapshot_mount_points`
The directories at or below a configured path that a snapshot's backup found on a different filesystem from their parent directory (a different device number), as a mount point is, the configured path itself included.
**Columns:**
- `snapshot_id` (TEXT) - Snapshot ID (FK to snapshots.id)
- `path` (TEXT) - Absolute path of the directory
- PRIMARY KEY (`snapshot_id`, `path`)
**Purpose:** Lets `snapshot create --prune` skip the prune when a volume an earlier snapshot of the same name found mounted is not mounted now.
### 11. `uploads`
Tracks blob upload metrics. Tracks blob upload metrics.
**Columns:** **Columns:**
@@ -173,6 +183,7 @@ Tracks blob upload metrics.
- `INSERT OR REPLACE INTO files` - Update file metadata - `INSERT OR REPLACE INTO files` - Update file metadata
- `SELECT * FROM files WHERE path = ?` - Check if file has changed - `SELECT * FROM files WHERE path = ?` - Check if file has changed
- `INSERT INTO snapshot_files` - Add file to current snapshot - `INSERT INTO snapshot_files` - Add file to current snapshot
- `INSERT INTO snapshot_mount_points` - Record each directory on a different filesystem from its parent
2. **Chunking** (for changed files) 2. **Chunking** (for changed files)
- `INSERT OR IGNORE INTO chunks` - Store new chunks - `INSERT OR IGNORE INTO chunks` - Store new chunks
@@ -230,6 +241,10 @@ The restore process doesn't use the local database. Instead:
- Query blobs not referenced by any remaining snapshot - Query blobs not referenced by any remaining snapshot
- Delete from S3, once older than `prune_grace_period`, and from the local - Delete from S3, once older than `prune_grace_period`, and from the local
database database
2. **Delete Metadata Directories With No Manifest**
- Delete each directory under `metadata/` that has no `manifest.json.zst`
and no local snapshot record, once every object in it is older than
`prune_grace_period`
### 6. Incomplete Snapshot Cleanup ### 6. Incomplete Snapshot Cleanup
+2 -2
View File
@@ -165,10 +165,10 @@ These are known, deliberate properties of the format and the tooling, recorded s
## Pruning Safety ## Pruning Safety
The prune operation is safe because: The prune operation is safe because:
1. It keeps every blob listed in any snapshot's manifest and deletes only blobs that no manifest references 1. It keeps every blob listed in any snapshot's manifest and deletes only blobs that no manifest references. Under `metadata/` it deletes only a directory that has no `manifest.json.zst`, whose name is a remote key, and that no local snapshot record claims
2. Manifests are unencrypted and can be read without keys 2. Manifests are unencrypted and can be read without keys
3. If any manifest cannot be downloaded or decoded, prune deletes nothing and exits with an error, rather than treating that snapshot's blobs as unreferenced 3. If any manifest cannot be downloaded or decoded, prune deletes nothing and exits with an error, rather than treating that snapshot's blobs as unreferenced
4. It deletes an unreferenced blob only once the blob is older than `prune_grace_period` (default `24h`). A backup uploads its blobs before the manifest that lists them, so a prune running meanwhile, on this host or another sharing the destination, leaves them as long as no backup run takes longer than the grace period. The grace period does not protect a blob a backup reuses from an earlier run that stopped before its manifest: the backup skips the upload of a blob already on the destination, so that blob keeps the time of its first upload, and a prune before the backup's manifest is written can delete it; the new snapshot then cannot be restored. `remote nuke` has no such check 4. It deletes an unreferenced blob only once the blob is older than `prune_grace_period` (default `24h`), and a directory under `metadata/` with no manifest only once every object in it is. A backup uploads its blobs and its `db.zst.age` before its manifest, so a prune running meanwhile, on this host or another sharing the destination, leaves them as long as no backup run takes longer than the grace period. The grace period does not protect a blob a backup reuses from an earlier run that stopped before its manifest: the backup skips the upload of a blob already on the destination, so that blob keeps the time of its first upload, and a prune before the backup's manifest is written can delete it; the new snapshot then cannot be restored. `remote nuke` has no such check
## Restoration Requirements ## Restoration Requirements
+9 -7
View File
@@ -239,13 +239,15 @@ storage_url: ""
# Default: 3 # Default: 3
# compression_level: 3 # compression_level: 3
# How old a blob that no snapshot lists must be before prune deletes it. # How old a blob that no snapshot lists must be before prune deletes it,
# A backup uploads its blobs before the manifest that lists them, so on a # and how old every object in a directory under metadata/ with no manifest
# destination shared between hosts or users this must be longer than the # must be before prune deletes that directory. A backup uploads its blobs
# longest backup run. It does not protect a blob a backup reuses from an # and its db.zst.age before its manifest, so on a destination shared
# earlier run that stopped before its manifest: that blob keeps the time of # between hosts or users this must be longer than the longest backup run.
# its first upload, so a prune can delete it. # It does not protect a blob a backup reuses from an earlier run that
# Accepts: 36h, 90m, etc. 0s deletes them at once. # stopped before its manifest: that blob keeps the time of its first
# upload, so a prune can delete it.
# Accepts: 36h, 90m, etc. 0s deletes such blobs and directories at once.
# Default: 24h # Default: 24h
# prune_grace_period: 24h # prune_grace_period: 24h
+7 -2
View File
@@ -13,8 +13,9 @@ func NewPruneCommand() *cobra.Command {
cmd := &cobra.Command{ cmd := &cobra.Command{
Use: "prune", Use: "prune",
Short: "Tidy local database and remote storage", Short: "Tidy local database and remote storage",
Long: `Removes orphaned data from both the local index database and Long: `Removes orphaned data from the local index database, and
unreferenced blobs from the backup destination store. unreferenced blobs and directories under metadata/ with no manifest
from the backup destination store.
Local cleanup drops incomplete snapshots and any files, chunks, or Local cleanup drops incomplete snapshots and any files, chunks, or
blobs no longer referenced by a completed snapshot. Remote cleanup blobs no longer referenced by a completed snapshot. Remote cleanup
@@ -26,6 +27,10 @@ another sharing the destination store, that has not yet uploaded its
manifest. The grace period does not protect a blob such a backup manifest. The grace period does not protect a blob such a backup
reuses from an earlier run that stopped before its manifest: that reuses from an earlier run that stopped before its manifest: that
blob keeps the time of its first upload, so it can be deleted. blob keeps the time of its first upload, so it can be deleted.
Remote cleanup also deletes each directory under metadata/ with no
manifest, as a backup that stopped before its manifest leaves, once
all its objects are older than prune_grace_period and its snapshot is
not in the local index.
Snapshot create --prune runs the same cleanup automatically; this Snapshot create --prune runs the same cleanup automatically; this
command is the manual entry point for the same work (e.g. after a command is the manual entry point for the same work (e.g. after a
+9 -5
View File
@@ -116,7 +116,8 @@ on Linux), then /etc/vaultik/config.yml.`,
} }
if errors.Is(err, vaultik.ErrFilesUnreadable) || if errors.Is(err, vaultik.ErrFilesUnreadable) ||
errors.Is(err, vaultik.ErrSourcePathEmpty) { errors.Is(err, vaultik.ErrSourcePathEmpty) ||
errors.Is(err, vaultik.ErrVolumeNotMounted) {
log.Error("Backup completed with an error", "error", err) log.Error("Backup completed with an error", "error", err)
ReportErrorf("Backup completed, but %v", err) ReportErrorf("Backup completed, but %v", err)
@@ -132,10 +133,13 @@ on Linux), then /etc/vaultik/config.yml.`,
cmd.Flags().BoolVar(&opts.Cron, "cron", false, cmd.Flags().BoolVar(&opts.Cron, "cron", false,
"Run in cron mode (silent unless warning or error)") "Run in cron mode (silent unless warning or error)")
cmd.Flags().BoolVar(&opts.Prune, "prune", false, cmd.Flags().BoolVar(&opts.Prune, "prune", false,
"After backup, drop older snapshots of the same name and remove "+ "After backup, drop older snapshots of the same name, remove "+
"orphaned blobs older than prune_grace_period (not after a "+ "orphaned blobs older than prune_grace_period, and delete each "+
"backup that skipped unreadable files or found a configured "+ "directory under metadata/ with no manifest once all its "+
"path empty)") "objects are older than prune_grace_period and its snapshot is "+
"not in the local index (not after a backup that skipped "+
"unreadable files, found a configured path empty, or found "+
"no volume where an earlier snapshot found one mounted)")
cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "", cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "",
"With --prune: keep snapshots newer than this duration "+ "With --prune: keep snapshots newer than this duration "+
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+ "(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+
+6 -4
View File
@@ -192,10 +192,12 @@ type Config struct {
CompressionLevel int `yaml:"compression_level"` CompressionLevel int `yaml:"compression_level"`
// PruneGracePeriod is how old a blob no manifest lists must be before // PruneGracePeriod is how old a blob no manifest lists must be before
// a prune deletes it. A backup lists its blobs in a manifest only // a prune deletes it, and how old every object in a directory under
// once it has uploaded them all, so a younger one may belong to a // metadata/ with no manifest must be before a prune deletes that
// backup still running, on this host or another sharing the // directory. A backup writes its manifest only once it has uploaded
// destination store. // all its blobs and its db.zst.age, so a younger blob or directory may
// belong to a backup still running, on this host or another sharing
// the destination store.
PruneGracePeriod time.Duration `yaml:"prune_grace_period"` PruneGracePeriod time.Duration `yaml:"prune_grace_period"`
// StorageURL specifies the storage backend using a URL format. // StorageURL specifies the storage backend using a URL format.
+2 -1
View File
@@ -415,7 +415,8 @@ func TestDocumentedSizeExamples(t *testing.T) {
// TestValidatePruneGracePeriod checks that a negative prune_grace_period // TestValidatePruneGracePeriod checks that a negative prune_grace_period
// is refused rather than acting like 0s, which deletes unreferenced blobs // is refused rather than acting like 0s, which deletes unreferenced blobs
// at once, and that 0s and the documented examples are accepted. // and directories under metadata/ with no manifest at once, and that 0s
// and the documented examples are accepted.
func TestValidatePruneGracePeriod(t *testing.T) { func TestValidatePruneGracePeriod(t *testing.T) {
t.Parallel() t.Parallel()
+10
View File
@@ -123,6 +123,16 @@ CREATE TABLE IF NOT EXISTS snapshot_blobs (
-- Index for efficient blob lookups (used in orphan detection) -- Index for efficient blob lookups (used in orphan detection)
CREATE INDEX IF NOT EXISTS idx_snapshot_blobs_blob_id ON snapshot_blobs(blob_id); CREATE INDEX IF NOT EXISTS idx_snapshot_blobs_blob_id ON snapshot_blobs(blob_id);
-- Snapshot mount points table: the directories at or below a configured
-- path that the snapshot's backup found on a different filesystem from
-- their parent directory, as a mount point is
CREATE TABLE IF NOT EXISTS snapshot_mount_points (
snapshot_id TEXT NOT NULL,
path TEXT NOT NULL,
PRIMARY KEY (snapshot_id, path),
FOREIGN KEY (snapshot_id) REFERENCES snapshots(id) ON DELETE CASCADE
);
-- Uploads table: tracks blob upload metrics -- Uploads table: tracks blob upload metrics
CREATE TABLE IF NOT EXISTS uploads ( CREATE TABLE IF NOT EXISTS uploads (
blob_hash TEXT PRIMARY KEY, blob_hash TEXT PRIMARY KEY,
+74 -15
View File
@@ -24,7 +24,8 @@ var (
) )
// SnapshotRepository provides access to the snapshots table and its // SnapshotRepository provides access to the snapshots table and its
// snapshot_files / snapshot_blobs association tables. // snapshot_files / snapshot_blobs association tables, and to the mount
// points each snapshot's backup found (snapshot_mount_points).
type SnapshotRepository struct { type SnapshotRepository struct {
db *DB db *DB
} }
@@ -494,33 +495,63 @@ func (r *SnapshotRepository) GetBlobHashes(
ctx context.Context, snapshotID string, ctx context.Context, snapshotID string,
) ([]string, error) { ) ([]string, error) {
query := ` query := `
SELECT sb.blob_hash SELECT sb.blob_hash
FROM snapshot_blobs sb FROM snapshot_blobs sb
WHERE sb.snapshot_id = ? WHERE sb.snapshot_id = ?
ORDER BY sb.blob_hash ORDER BY sb.blob_hash
` `
rows, err := r.db.conn.QueryContext(ctx, query, snapshotID) blobs, err := r.queryStrings(ctx, query, snapshotID)
if err != nil { if err != nil {
return nil, fmt.Errorf("querying blob hashes: %w", err) return nil, fmt.Errorf("querying blob hashes: %w", err)
} }
defer func() { _ = rows.Close() }() return blobs, nil
}
var blobs []string // AddMountPoint records dir as a directory that the backup of a snapshot
// found on a different filesystem from its parent directory, as a mount
// point is, using tx when non-nil.
func (r *SnapshotRepository) AddMountPoint(
ctx context.Context, tx *sql.Tx, snapshotID string, dir string,
) error {
query := `
INSERT OR IGNORE INTO snapshot_mount_points (snapshot_id, path)
VALUES (?, ?)
`
for rows.Next() { var err error
var blobHash string if tx != nil {
_, err = tx.ExecContext(ctx, query, snapshotID, dir)
err := rows.Scan(&blobHash) } else {
if err != nil { _, err = r.db.ExecWithLog(ctx, query, snapshotID, dir)
return nil, fmt.Errorf("scanning blob hash: %w", err)
}
blobs = append(blobs, blobHash)
} }
return blobs, rows.Err() if err != nil {
return fmt.Errorf("adding mount point to snapshot: %w", err)
}
return nil
}
// GetMountPoints returns the directories AddMountPoint recorded for a
// snapshot, sorted.
func (r *SnapshotRepository) GetMountPoints(
ctx context.Context, snapshotID string,
) ([]string, error) {
query := `
SELECT path
FROM snapshot_mount_points
WHERE snapshot_id = ?
ORDER BY path
`
dirs, err := r.queryStrings(ctx, query, snapshotID)
if err != nil {
return nil, fmt.Errorf("querying mount points: %w", err)
}
return dirs, nil
} }
// GetSnapshotTotalCompressedSize returns the total compressed size of all // GetSnapshotTotalCompressedSize returns the total compressed size of all
@@ -709,6 +740,34 @@ func (r *SnapshotRepository) DeleteSnapshotUploads(
return nil return nil
} }
// queryStrings runs query, which selects one text column, and returns
// its values in the order of the rows.
func (r *SnapshotRepository) queryStrings(
ctx context.Context, query string, args ...any,
) ([]string, error) {
rows, err := r.db.conn.QueryContext(ctx, query, args...)
if err != nil {
return nil, err
}
defer func() { _ = rows.Close() }()
var values []string
for rows.Next() {
var value string
err := rows.Scan(&value)
if err != nil {
return nil, err
}
values = append(values, value)
}
return values, rows.Err()
}
// extendedCompressionRatio computes the compression ratio for a snapshot // extendedCompressionRatio computes the compression ratio for a snapshot
// from its stored blob_size and the given uncompressed size. Returns 1.0 // from its stored blob_size and the given uncompressed size. Returns 1.0
// when the uncompressed size is zero. // when the uncompressed size is zero.
+50
View File
@@ -4,6 +4,7 @@ import (
"context" "context"
"fmt" "fmt"
"math" "math"
"slices"
"testing" "testing"
"time" "time"
@@ -422,3 +423,52 @@ func TestSnapshotRepositoryDuplicate(t *testing.T) {
t.Error("expected error for duplicate snapshot") t.Error("expected error for duplicate snapshot")
} }
} }
// Deleting a snapshot from the local index deletes the mount points its
// backup recorded, and no other snapshot's. See
// https://git.eeqj.de/sneak/vaultik/issues/395.
func TestSnapshotRepositoryDeleteRemovesMountPoints(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repo := database.NewSnapshotRepository(db)
for _, id := range []types.SnapshotID{"kept", "deleted"} {
err := repo.Create(ctx, nil, &database.Snapshot{
ID: id,
Hostname: testHostname,
VaultikVersion: testVersion,
StartedAt: time.Now().Truncate(time.Second),
})
if err != nil {
t.Fatalf("failed to create snapshot: %v", err)
}
err = repo.AddMountPoint(ctx, nil, id.String(), "/data/nas")
if err != nil {
t.Fatalf("failed to add mount point: %v", err)
}
}
err := repo.Delete(ctx, "deleted")
if err != nil {
t.Fatalf("failed to delete snapshot: %v", err)
}
for id, want := range map[string][]string{
"kept": {"/data/nas"},
"deleted": nil,
} {
got, err := repo.GetMountPoints(ctx, id)
if err != nil {
t.Fatalf("failed to read mount points: %v", err)
}
if !slices.Equal(got, want) {
t.Errorf("mount points of %s: got %v, want %v", id, got, want)
}
}
}
+6
View File
@@ -1,6 +1,8 @@
package snapshot package snapshot
import ( import (
"os"
"github.com/spf13/afero" "github.com/spf13/afero"
"go.uber.org/fx" "go.uber.org/fx"
"sneak.berlin/go/vaultik/internal/config" "sneak.berlin/go/vaultik/internal/config"
@@ -19,6 +21,9 @@ type ScannerParams struct {
// PreviousSnapshotID is the most recent completed snapshot of the // PreviousSnapshotID is the most recent completed snapshot of the
// same name, "" for none. // same name, "" for none.
PreviousSnapshotID string PreviousSnapshotID string
// DeviceNumber is ScannerConfig.DeviceNumber; nil reads it from the
// file's stat.
DeviceNumber func(info os.FileInfo) (uint64, bool)
} }
// Module exports backup functionality as an fx module. // Module exports backup functionality as an fx module.
@@ -59,6 +64,7 @@ func provideScannerFactory(
Exclude: excludes, Exclude: excludes,
SkipErrors: params.SkipErrors, SkipErrors: params.SkipErrors,
PreviousSnapshotID: params.PreviousSnapshotID, PreviousSnapshotID: params.PreviousSnapshotID,
DeviceNumber: params.DeviceNumber,
}) })
} }
} }
+97
View File
@@ -0,0 +1,97 @@
package snapshot_test
import (
"context"
"os"
"path/filepath"
"slices"
"testing"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/snapshot"
)
// A scan records with the snapshot each directory on a different device
// from its parent directory, as a mount point is, the scanned path
// included. A filesystem that gives no device numbers, as the in-memory
// one, records none. See https://git.eeqj.de/sneak/vaultik/issues/395.
func TestScanRecordsMountPoints(t *testing.T) {
t.Parallel()
const (
scannedPath = "/data/source"
snapshotID = "mount-points"
)
tests := []struct {
name string
// onVolume names the directories on the volume's device; every
// other directory is on another one. nil leaves the device
// numbers to the in-memory filesystem, which has none.
onVolume []string
want []string
}{
{
name: "volume below the scanned path",
onVolume: []string{"nas"},
want: []string{scannedPath + "/nas"},
},
{
name: "volume at the scanned path",
onVolume: []string{"source", "nas"},
want: []string{scannedPath},
},
{name: "no device numbers", want: nil},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
ctx := context.Background()
fs := afero.NewMemMapFs()
for _, path := range []string{
scannedPath + "/notes.txt", scannedPath + "/nas/a.txt",
} {
require.NoError(t, fs.MkdirAll(filepath.Dir(path), 0o755))
require.NoError(t, afero.WriteFile(fs, path, []byte(path), 0o644))
}
db, err := database.NewTestDB()
require.NoError(t, err)
t.Cleanup(func() { _ = db.Close() })
repos := database.NewRepositories(db)
createTestSnapshotRecord(ctx, t, repos, snapshotID)
cfg := snapshot.ScannerConfig{
FS: fs,
ChunkSize: int64(1024 * 16),
Repositories: repos,
MaxBlobSize: int64(1024 * 1024),
CompressionLevel: 3,
AgeRecipients: []string{testAgePublicKey},
}
if tt.onVolume != nil {
cfg.DeviceNumber = func(info os.FileInfo) (uint64, bool) {
if slices.Contains(tt.onVolume, info.Name()) {
return 2, true
}
return 1, true
}
}
_, err = snapshot.NewScanner(cfg).Scan(ctx, scannedPath, snapshotID)
require.NoError(t, err)
got, err := repos.Snapshots.GetMountPoints(ctx, snapshotID)
require.NoError(t, err)
assert.Equal(t, tt.want, got)
})
}
}
+107 -4
View File
@@ -73,6 +73,9 @@ type Scanner struct {
skipErrors bool skipErrors bool
// ui is the user-facing output; never nil (defaults to a discarding writer). // ui is the user-facing output; never nil (defaults to a discarding writer).
ui *ui.Writer ui *ui.Writer
// deviceNumber returns the number of the device that holds a file;
// never nil (defaults to deviceNumberFromStat).
deviceNumber func(info os.FileInfo) (uint64, bool)
// In-memory cache of known chunk hashes for fast existence checks // In-memory cache of known chunk hashes for fast existence checks
knownChunks map[string]struct{} knownChunks map[string]struct{}
@@ -136,6 +139,10 @@ type ScannerConfig struct {
// PreviousSnapshotID is the most recent completed snapshot of the // PreviousSnapshotID is the most recent completed snapshot of the
// same name, "" for none. // same name, "" for none.
PreviousSnapshotID string PreviousSnapshotID string
// DeviceNumber returns the number of the device that holds a file,
// and false when there is none to read. nil reads it from the file's
// stat; tests replace it, since the in-memory test filesystem has none.
DeviceNumber func(info os.FileInfo) (uint64, bool)
} }
// ScanResult contains the results of a scan operation. Files and bytes // ScanResult contains the results of a scan operation. Files and bytes
@@ -214,6 +221,11 @@ func NewScanner(cfg ScannerConfig) *Scanner {
uiw = ui.NewWithColor(io.Discard, false) uiw = ui.NewWithColor(io.Discard, false)
} }
deviceNumber := cfg.DeviceNumber
if deviceNumber == nil {
deviceNumber = deviceNumberFromStat
}
return &Scanner{ return &Scanner{
fs: cfg.FS, fs: cfg.FS,
chunker: chunker.NewChunker(cfg.ChunkSize), chunker: chunker.NewChunker(cfg.ChunkSize),
@@ -229,6 +241,7 @@ func NewScanner(cfg ScannerConfig) *Scanner {
skipErrors: cfg.SkipErrors, skipErrors: cfg.SkipErrors,
previousSnapshotID: cfg.PreviousSnapshotID, previousSnapshotID: cfg.PreviousSnapshotID,
ui: uiw, ui: uiw,
deviceNumber: deviceNumber,
pendingChunkHashes: make(map[string]struct{}), pendingChunkHashes: make(map[string]struct{}),
} }
} }
@@ -1290,9 +1303,9 @@ func (s *Scanner) buildDirectoryEntry(path string, info os.FileInfo) *database.F
} }
} }
// recordDirectories records every directory the walk found. A directory // recordDirectories records every directory the walk found, and its
// has no data to back up, so it is not among the files counted and // mount points (see recordMountPoints). A directory has no data to back
// backed up. // up, so it is not among the files counted and backed up.
func (s *Scanner) recordDirectories( func (s *Scanner) recordDirectories(
ctx context.Context, directories []*FileToProcess, ctx context.Context, directories []*FileToProcess,
) error { ) error {
@@ -1303,7 +1316,97 @@ func (s *Scanner) recordDirectories(
} }
} }
return nil return s.recordMountPoints(ctx, directories)
}
// recordMountPoints records with the snapshot each directory the walk
// found on a different device from its parent directory, as a mount
// point is, the scanned path included.
func (s *Scanner) recordMountPoints(
ctx context.Context, directories []*FileToProcess,
) error {
mountPoints, err := s.findMountPoints(directories)
if err != nil {
return err
}
return s.repos.WithTx(ctx, func(txCtx context.Context, tx *sql.Tx) error {
for _, dir := range mountPoints {
log.Info("Found a mount point", "path", dir)
err := s.repos.Snapshots.AddMountPoint(txCtx, tx, s.snapshotID, dir)
if err != nil {
return fmt.Errorf("recording mount point %s: %w", dir, err)
}
}
return nil
})
}
// findMountPoints returns the directories, among those the walk found,
// that are on a different device from their parent directory. The
// scanned path is compared with the directory that holds it, which the
// walk did not reach.
func (s *Scanner) findMountPoints(
directories []*FileToProcess,
) ([]string, error) {
walked := make(map[string]os.FileInfo, len(directories))
for _, dir := range directories {
walked[dir.Path] = dir.FileInfo
}
var mountPoints []string
for _, dir := range directories {
parentPath := filepath.Dir(dir.Path)
parent, ok := walked[parentPath]
if !ok {
var err error
parent, err = s.fs.Stat(parentPath)
if err != nil {
return nil, fmt.Errorf(
"looking up the parent of %s: %w", dir.Path, err)
}
}
if s.onOtherDevice(dir.FileInfo, parent) {
mountPoints = append(mountPoints, dir.Path)
}
}
return mountPoints, nil
}
// onOtherDevice reports whether dir is on a different device from
// parent. A file whose device number cannot be read counts as on the
// same device as its parent.
func (s *Scanner) onOtherDevice(dir, parent os.FileInfo) bool {
dirDevice, ok := s.deviceNumber(dir)
if !ok {
return false
}
parentDevice, ok := s.deviceNumber(parent)
if !ok {
return false
}
return dirDevice != parentDevice
}
// deviceNumberFromStat returns the number of the device that holds the
// file info describes, and false when info carries no stat, as from the
// in-memory test filesystem.
func deviceNumberFromStat(info os.FileInfo) (uint64, bool) {
stat, ok := info.Sys().(*syscall.Stat_t)
if !ok {
return 0, false
}
return uint64(stat.Dev), true //nolint:unconvert // Dev is an int32 on darwin
} }
// recordNonRegularFile writes a symlink or directory entry to the database // recordNonRegularFile writes a symlink or directory entry to the database
+17 -4
View File
@@ -518,8 +518,8 @@ type CleanupStats struct {
// //
// The cleanup is performed in a specific order to maintain referential integrity: // The cleanup is performed in a specific order to maintain referential integrity:
// 1. Delete other snapshots // 1. Delete other snapshots
// 2. Delete orphaned snapshot associations (snapshot_files, snapshot_blobs) // 2. Delete orphaned snapshot associations (snapshot_files, snapshot_blobs,
// for deleted snapshots // snapshot_mount_points) for deleted snapshots
// 3. Delete orphaned files (not in the current snapshot) // 3. Delete orphaned files (not in the current snapshot)
// 4. Delete orphaned chunk-to-file mappings (references to deleted files) // 4. Delete orphaned chunk-to-file mappings (references to deleted files)
// 5. Delete orphaned blobs (not in the current snapshot) // 5. Delete orphaned blobs (not in the current snapshot)
@@ -948,8 +948,8 @@ func (sm *SnapshotManager) deleteOtherSnapshots(
return nil return nil
} }
// deleteOrphanedSnapshotAssociations deletes snapshot_files and // deleteOrphanedSnapshotAssociations deletes snapshot_files,
// snapshot_blobs for deleted snapshots // snapshot_blobs and snapshot_mount_points for deleted snapshots
func (sm *SnapshotManager) deleteOrphanedSnapshotAssociations( func (sm *SnapshotManager) deleteOrphanedSnapshotAssociations(
ctx context.Context, tx *sql.Tx, currentSnapshotID string, ctx context.Context, tx *sql.Tx, currentSnapshotID string,
) error { ) error {
@@ -983,6 +983,19 @@ func (sm *SnapshotManager) deleteOrphanedSnapshotAssociations(
log.Debug("[Temp DB Cleanup] Deleted snapshot_blobs associations", log.Debug("[Temp DB Cleanup] Deleted snapshot_blobs associations",
"count", rowsAffected) "count", rowsAffected)
// Delete the mount points other snapshots recorded
query := "DELETE FROM snapshot_mount_points WHERE snapshot_id != ?"
database.LogSQL("Execute", query, currentSnapshotID)
result, err = tx.ExecContext(ctx, query, currentSnapshotID)
if err != nil {
return fmt.Errorf("deleting orphaned snapshot_mount_points: %w", err)
}
rowsAffected, _ = result.RowsAffected()
log.Debug("[Temp DB Cleanup] Deleted snapshot_mount_points rows",
"count", rowsAffected)
return nil return nil
} }
+73
View File
@@ -7,12 +7,14 @@ import (
"database/sql" "database/sql"
"io" "io"
"path/filepath" "path/filepath"
"slices"
"testing" "testing"
"github.com/spf13/afero" "github.com/spf13/afero"
"sneak.berlin/go/vaultik/internal/config" "sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/database" "sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log" "sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/types"
) )
const ( const (
@@ -387,6 +389,77 @@ func TestCleanSnapshotDBNonExistentSnapshot(t *testing.T) {
} }
} }
// The exported metadata database keeps only the mount points that the
// exported snapshot recorded. See
// https://git.eeqj.de/sneak/vaultik/issues/395.
func TestCleanSnapshotDBKeepsOnlyTheSnapshotsMountPoints(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
dbPath := filepath.Join(t.TempDir(), "export.db")
db, err := database.New(ctx, dbPath)
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
repos := database.NewRepositories(db)
for _, id := range []types.SnapshotID{"exported", "other"} {
err = repos.Snapshots.Create(ctx, nil, &database.Snapshot{ID: id})
if err != nil {
t.Fatalf("failed to create snapshot: %v", err)
}
err = repos.Snapshots.AddMountPoint(ctx, nil, id.String(), "/data/"+id.String())
if err != nil {
t.Fatalf("failed to add mount point: %v", err)
}
}
err = db.Close()
if err != nil {
t.Fatalf("failed to close database: %v", err)
}
sm := &SnapshotManager{
config: &config.Config{
CompressionLevel: 3,
AgeRecipients: []string{testAgeRecipient},
},
fs: afero.NewOsFs(),
}
_, err = sm.cleanSnapshotDB(ctx, dbPath, "exported")
if err != nil {
t.Fatalf("failed to clean snapshot database: %v", err)
}
cleaned, err := database.New(ctx, dbPath)
if err != nil {
t.Fatalf("failed to open cleaned database: %v", err)
}
defer func() { _ = cleaned.Close() }()
cleanedRepos := database.NewRepositories(cleaned)
for id, want := range map[string][]string{
"exported": {"/data/exported"},
"other": nil,
} {
got, err := cleanedRepos.Snapshots.GetMountPoints(ctx, id)
if err != nil {
t.Fatalf("failed to read mount points: %v", err)
}
if !slices.Equal(got, want) {
t.Errorf("mount points of %s: got %v, want %v", id, got, want)
}
}
}
// Two creates of one snapshot name back to back start within one second, // Two creates of one snapshot name back to back start within one second,
// the resolution of the timestamp in a snapshot ID. See // the resolution of the timestamp in a snapshot ID. See
// https://git.eeqj.de/sneak/vaultik/issues/270. // https://git.eeqj.de/sneak/vaultik/issues/270.
+3 -2
View File
@@ -100,8 +100,9 @@ func TestS3StorerMissingKeyMapsToErrNotFound(t *testing.T) {
// TestS3ListingCutShortIsAnError checks that List and ListStream fail on a // TestS3ListingCutShortIsAnError checks that List and ListStream fail on a
// listing page marked truncated that carries no continuation token, as a // listing page marked truncated that carries no continuation token, as a
// server answering V2 listings V1-style sends it. Taken as the last page, // server answering V2 listings V1-style sends it. Taken as the last page,
// it would hide every later key, and prune would treat the blobs of the // it would hide every later key: prune would treat the blobs of the
// snapshots listed there as unreferenced. // snapshots listed there as unreferenced, and a directory under metadata/
// whose manifest is listed there as one with no manifest.
func TestS3ListingCutShortIsAnError(t *testing.T) { func TestS3ListingCutShortIsAnError(t *testing.T) {
t.Parallel() t.Parallel()
+1
View File
@@ -679,6 +679,7 @@ func faultScannerFactory(
Exclude: params.Exclude, Exclude: params.Exclude,
SkipErrors: params.SkipErrors, SkipErrors: params.SkipErrors,
PreviousSnapshotID: params.PreviousSnapshotID, PreviousSnapshotID: params.PreviousSnapshotID,
DeviceNumber: params.DeviceNumber,
}) })
} }
} }
+1 -1
View File
@@ -76,7 +76,7 @@ func TestNukeRemoteLeavesNoFiles(t *testing.T) {
// TestNukeRemoteFailsWhenADeleteFails checks that remote nuke returns an // TestNukeRemoteFailsWhenADeleteFails checks that remote nuke returns an
// error when a snapshot's metadata or a blob cannot be deleted, does not // error when a snapshot's metadata or a blob cannot be deleted, does not
// report the store empty, and advises running nuke again: prune deletes // report the store empty, and advises running nuke again: prune deletes
// no metadata. // no snapshot metadata that has its manifest.
func TestNukeRemoteFailsWhenADeleteFails(t *testing.T) { func TestNukeRemoteFailsWhenADeleteFails(t *testing.T) {
log.Initialize(log.Config{}) log.Initialize(log.Config{})
t.Parallel() t.Parallel()
+244 -60
View File
@@ -4,10 +4,12 @@ import (
"encoding/json" "encoding/json"
"errors" "errors"
"fmt" "fmt"
"sort"
"strings" "strings"
"time" "time"
"sneak.berlin/go/vaultik/internal/log" "sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot"
"sneak.berlin/go/vaultik/internal/storage" "sneak.berlin/go/vaultik/internal/storage"
) )
@@ -24,6 +26,11 @@ var errNukeRequiresForce = errors.New(
// errBlobDeleteFailed reports unreferenced blobs prune could not delete. // errBlobDeleteFailed reports unreferenced blobs prune could not delete.
var errBlobDeleteFailed = errors.New("blob(s) could not be deleted") var errBlobDeleteFailed = errors.New("blob(s) could not be deleted")
// errMetadataDirDeleteFailed reports directories under metadata/ with no
// manifest that prune could not delete.
var errMetadataDirDeleteFailed = errors.New(
"metadata director(ies) with no manifest could not be deleted")
// errObjectsLeft reports objects remote nuke still finds on the backup // errObjectsLeft reports objects remote nuke still finds on the backup
// destination store after deleting. // destination store after deleting.
var errObjectsLeft = errors.New( var errObjectsLeft = errors.New(
@@ -31,7 +38,8 @@ var errObjectsLeft = errors.New(
// nukeCommandHint is the command suggested when a remote nuke could not // nukeCommandHint is the command suggested when a remote nuke could not
// delete everything: running it again deletes what is left, which // delete everything: running it again deletes what is left, which
// `vaultik prune` does not do for snapshot metadata. // `vaultik prune` does not do for snapshot metadata that has its
// manifest.
const nukeCommandHint = "vaultik remote nuke --force" const nukeCommandHint = "vaultik remote nuke --force"
// metadataDirName is the top-level remote directory holding snapshot // metadataDirName is the top-level remote directory holding snapshot
@@ -169,14 +177,20 @@ type PruneBlobsResult struct {
// newer than the prune grace period, or of unknown age. BlobsFound // newer than the prune grace period, or of unknown age. BlobsFound
// does not include them. // does not include them.
BlobsTooNew int `json:"blobs_too_new"` BlobsTooNew int `json:"blobs_too_new"`
// MetadataDirsDeleted counts the directories under metadata/ with no
// manifest that prune deleted, and MetadataDirsFailed those it could
// not delete.
MetadataDirsDeleted int `json:"metadata_dirs_deleted"`
MetadataDirsFailed int `json:"metadata_dirs_failed,omitempty"`
} }
// Prune removes orphaned data from the local index database AND // Prune removes orphaned data from the local index database AND, from the
// unreferenced blobs from the backup destination store. This is the // backup destination store, unreferenced blobs and the directories under
// single user-facing prune entry point — the split between local and // metadata/ with no manifest. This is the single user-facing prune entry
// remote cleanup is an implementation detail. Calling code should // point — the split between local and remote cleanup is an
// prefer this method over PruneDatabase or PruneBlobs individually // implementation detail. Calling code should prefer this method over
// unless it specifically wants one half. // PruneDatabase or PruneBlobs individually unless it specifically wants
// one half.
func (v *Vaultik) Prune(opts *PruneOptions) error { func (v *Vaultik) Prune(opts *PruneOptions) error {
err := v.EnsureStorageBinding() err := v.EnsureStorageBinding()
if err != nil { if err != nil {
@@ -200,11 +214,17 @@ func (v *Vaultik) Prune(opts *PruneOptions) error {
return v.PruneBlobs(opts) return v.PruneBlobs(opts)
} }
// PruneBlobs removes unreferenced blobs from storage // PruneBlobs removes unreferenced blobs from storage, and each directory
// under metadata/ that a backup left with no manifest.
func (v *Vaultik) PruneBlobs(opts *PruneOptions) error { func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
log.Info("Starting prune operation") log.Info("Starting prune operation")
allBlobsReferenced, err := v.collectReferencedBlobs() metadataObjects, err := v.listMetadataObjects()
if err != nil {
return err
}
allBlobsReferenced, err := v.collectReferencedBlobs(metadataObjects)
if err != nil { if err != nil {
return err return err
} }
@@ -217,6 +237,11 @@ func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
unreferencedBlobs, totalSize, tooNew := v.findUnreferencedBlobs( unreferencedBlobs, totalSize, tooNew := v.findUnreferencedBlobs(
allBlobs, allBlobsReferenced) allBlobs, allBlobsReferenced)
dirsWithoutManifest, err := v.findMetadataDirsWithoutManifest(metadataObjects)
if err != nil {
return err
}
result := &PruneBlobsResult{ result := &PruneBlobsResult{
BlobsFound: len(unreferencedBlobs), BlobsFound: len(unreferencedBlobs),
BlobsTooNew: tooNew, BlobsTooNew: tooNew,
@@ -224,7 +249,7 @@ func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
v.reportBlobsTooNew(tooNew, opts) v.reportBlobsTooNew(tooNew, opts)
if len(unreferencedBlobs) == 0 { if len(unreferencedBlobs) == 0 && len(dirsWithoutManifest) == 0 {
log.Info("No unreferenced blobs found") log.Info("No unreferenced blobs found")
if opts.JSON { if opts.JSON {
@@ -237,38 +262,16 @@ func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
} }
log.Info("Found unreferenced blobs", log.Info("Found unreferenced blobs",
"count", len(unreferencedBlobs), "total_size", ubytes(totalSize)) "count", len(unreferencedBlobs), "total_size", ubytes(totalSize),
"metadata_dirs_without_manifest", len(dirsWithoutManifest))
if opts.Force && !opts.JSON { if !v.confirmPrune(opts, len(unreferencedBlobs), totalSize,
v.UI.Infof("Found %d unreferenced blob(s) totaling %s.", dirsWithoutManifest) {
len(unreferencedBlobs), ubytes(totalSize)) return nil
}
if !opts.Force && !opts.JSON {
// The count, the prompt and its answer bypass the UI: they are an
// interactive exchange the operator must see, so --quiet does not
// silence them.
v.stdoutf("Found %d unreferenced blob(s) totaling %s.\n",
len(unreferencedBlobs), ubytes(totalSize))
v.stdoutf("\nDelete %d unreferenced blob(s)? [y/N] ", len(unreferencedBlobs))
var confirm string
_, err = v.scanStdin(&confirm)
if err != nil {
v.printlnStdout("Cancelled")
return nil //nolint:nilerr // read failure means no confirmation
}
if strings.ToLower(confirm) != "y" {
v.printlnStdout("Cancelled")
return nil
}
} }
v.deleteUnreferencedBlobs(unreferencedBlobs, allBlobs, result) v.deleteUnreferencedBlobs(unreferencedBlobs, allBlobs, result)
v.deleteMetadataDirs(dirsWithoutManifest, result)
if opts.JSON { if opts.JSON {
err = v.outputPruneBlobsJSON(result) err = v.outputPruneBlobsJSON(result)
@@ -276,33 +279,142 @@ func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
return err return err
} }
} else { } else {
v.reportPruneDeletions(result, len(dirsWithoutManifest))
}
var failures []error
if result.BlobsFailed > 0 {
failures = append(failures,
fmt.Errorf("%d %w", result.BlobsFailed, errBlobDeleteFailed))
}
if result.MetadataDirsFailed > 0 {
failures = append(failures, fmt.Errorf("%d %w",
result.MetadataDirsFailed, errMetadataDirDeleteFailed))
}
return errors.Join(failures...)
}
// confirmPrune states what prune found to delete, unreferenced blobs and
// directories under metadata/ with no manifest, naming each directory by
// its remote key, and, unless --force is given, asks the operator to
// confirm. Under --json it prints nothing and does not ask. It reports
// whether to go ahead.
func (v *Vaultik) confirmPrune(
opts *PruneOptions, blobCount int, blobsSize int64, dirs []string,
) bool {
if opts.JSON {
return true
}
dirCount := len(dirs)
blobsFound := fmt.Sprintf("Found %d unreferenced blob(s) totaling %s.",
blobCount, ubytes(blobsSize))
dirsFound := fmt.Sprintf("Found %d metadata director(ies) left by "+
"backups that stopped before their manifest:", dirCount)
if opts.Force {
if blobCount > 0 {
v.UI.Infof("%s", blobsFound)
}
if dirCount > 0 {
v.UI.Infof("%s", dirsFound)
for _, remoteKey := range dirs {
v.UI.Detailf("metadata/%s/", remoteKey)
}
}
return true
}
// The counts, the names, the prompt and its answer bypass the UI:
// they are an interactive exchange the operator must see, so --quiet
// does not silence them.
if blobCount > 0 {
v.stdoutf("%s\n", blobsFound)
}
if dirCount > 0 {
v.stdoutf("%s\n", dirsFound)
for _, remoteKey := range dirs {
v.stdoutf(" metadata/%s/\n", remoteKey)
}
}
switch {
case dirCount == 0:
v.stdoutf("\nDelete %d unreferenced blob(s)? [y/N] ", blobCount)
case blobCount == 0:
v.stdoutf("\nDelete %d metadata director(ies)? [y/N] ", dirCount)
default:
v.stdoutf("\nDelete %d unreferenced blob(s) and %d metadata "+
"director(ies)? [y/N] ", blobCount, dirCount)
}
var confirm string
// A failure to read the answer means no confirmation.
_, err := v.scanStdin(&confirm)
if err != nil || strings.ToLower(confirm) != "y" {
v.printlnStdout("Cancelled")
return false
}
return true
}
// reportPruneDeletions says how many blobs and metadata directories
// prune deleted, naming each kind only when it found some to delete.
func (v *Vaultik) reportPruneDeletions(result *PruneBlobsResult, dirCount int) {
if result.BlobsFound > 0 {
v.UI.Infof("Deleted %d blob(s) totaling %s.", v.UI.Infof("Deleted %d blob(s) totaling %s.",
result.BlobsDeleted, ubytes(result.BytesFreed)) result.BlobsDeleted, ubytes(result.BytesFreed))
} }
if result.BlobsFailed > 0 { if dirCount > 0 {
return fmt.Errorf("%d %w", result.BlobsFailed, errBlobDeleteFailed) v.UI.Infof("Deleted %d metadata director(ies) with no manifest.",
result.MetadataDirsDeleted)
} }
return nil
} }
// collectReferencedBlobs downloads all manifests and returns the set of // listMetadataObjects lists every object under metadata/ on the
// referenced blob hashes. // destination store.
func (v *Vaultik) listMetadataObjects() ([]storage.ObjectInfo, error) {
log.Info("Listing remote snapshots")
var objects []storage.ObjectInfo
for object := range v.Storage.ListStream(v.ctx, "metadata/") {
if object.Err != nil {
return nil, fmt.Errorf("listing metadata objects: %w", object.Err)
}
objects = append(objects, object)
}
return objects, nil
}
// collectReferencedBlobs downloads every manifest among the listed
// metadata objects and returns the set of referenced blob hashes.
// //
// Every manifest must be read successfully. A manifest that cannot be // Every manifest must be read successfully. A manifest that cannot be
// downloaded or decoded means its snapshot's blobs are unknown, so // downloaded or decoded means its snapshot's blobs are unknown, so
// treating them as unreferenced would let prune delete data a snapshot // treating them as unreferenced would let prune delete data a snapshot
// still needs. Rather than risk that silent loss, any failure returns an // still needs. Rather than risk that silent loss, any failure returns an
// error naming the remote key and prune deletes nothing. // error naming the remote key and prune deletes nothing.
func (v *Vaultik) collectReferencedBlobs() (map[string]bool, error) { func (v *Vaultik) collectReferencedBlobs(
log.Info("Listing remote snapshots") metadataObjects []storage.ObjectInfo,
// IDs returned by listUniqueSnapshotIDs are remote keys (hashed ) (map[string]bool, error) {
// IDs returned by uniqueSnapshotIDs are remote keys (hashed
// subdirectories under metadata/), not human snapshot IDs. // subdirectories under metadata/), not human snapshot IDs.
remoteKeys, err := v.listUniqueSnapshotIDs() remoteKeys := uniqueSnapshotIDs(metadataObjects)
if err != nil {
return nil, fmt.Errorf("listing snapshot keys: %w", err)
}
log.Info("Found manifests in remote storage", "count", len(remoteKeys)) log.Info("Found manifests in remote storage", "count", len(remoteKeys))
@@ -327,18 +439,14 @@ func (v *Vaultik) collectReferencedBlobs() (map[string]bool, error) {
return allBlobsReferenced, nil return allBlobsReferenced, nil
} }
// listUniqueSnapshotIDs returns deduplicated snapshot IDs from remote metadata // uniqueSnapshotIDs returns deduplicated snapshot IDs from the listed
func (v *Vaultik) listUniqueSnapshotIDs() ([]string, error) { // metadata objects.
objectCh := v.Storage.ListStream(v.ctx, "metadata/") func uniqueSnapshotIDs(metadataObjects []storage.ObjectInfo) []string {
seen := make(map[string]bool) seen := make(map[string]bool)
var snapshotIDs []string var snapshotIDs []string
for object := range objectCh { for _, object := range metadataObjects {
if object.Err != nil {
return nil, fmt.Errorf("listing metadata objects: %w", object.Err)
}
parts := strings.Split(object.Key, "/") parts := strings.Split(object.Key, "/")
if len(parts) >= minSnapshotIDParts && if len(parts) >= minSnapshotIDParts &&
parts[0] == metadataDirName && parts[1] != "" { parts[0] == metadataDirName && parts[1] != "" {
@@ -353,7 +461,7 @@ func (v *Vaultik) listUniqueSnapshotIDs() ([]string, error) {
} }
} }
return snapshotIDs, nil return snapshotIDs
} }
// listAllRemoteBlobs returns a map of all blob hashes in remote storage // listAllRemoteBlobs returns a map of all blob hashes in remote storage
@@ -427,6 +535,82 @@ func (v *Vaultik) findUnreferencedBlobs(
return unreferenced, totalSize, tooNew return unreferenced, totalSize, tooNew
} }
// findMetadataDirsWithoutManifest returns the remote key of each
// directory under metadata/ that has no manifest.json.zst, such as one a
// backup leaves when its manifest upload fails after db.zst.age was
// uploaded. It leaves out a directory a local snapshot record claims,
// and one holding an object newer than the prune grace period, or of
// unknown age because the listing gave no time for it: a backup still
// running may be writing there. A name that is not a remote key is
// never returned.
func (v *Vaultik) findMetadataDirsWithoutManifest(
metadataObjects []storage.ObjectInfo,
) ([]string, error) {
localIDs, err := v.localSnapshotIDs()
if err != nil {
return nil, fmt.Errorf("listing local snapshots: %w", err)
}
keep := make(map[string]bool, len(localIDs))
for _, id := range localIDs {
keep[snapshot.RemoteSnapshotKey(id)] = true
}
cutoff := time.Now().UTC().Add(-v.Config.PruneGracePeriod)
listed := make(map[string]bool)
for _, object := range metadataObjects {
parts := strings.Split(object.Key, "/")
if len(parts) < metadataKeyParts || !isBlobHash(parts[1]) {
continue
}
remoteKey := parts[1]
listed[remoteKey] = true
if parts[2] == "manifest.json.zst" ||
object.ModTime.IsZero() || object.ModTime.After(cutoff) {
keep[remoteKey] = true
}
}
var remoteKeys []string
for remoteKey := range listed {
if !keep[remoteKey] {
remoteKeys = append(remoteKeys, remoteKey)
}
}
sort.Strings(remoteKeys)
return remoteKeys, nil
}
// deleteMetadataDirs deletes everything in the given directories under
// metadata/ and counts in the result the directories deleted and those
// it could not delete.
func (v *Vaultik) deleteMetadataDirs(
remoteKeys []string, result *PruneBlobsResult,
) {
for _, remoteKey := range remoteKeys {
_, err := v.deleteRemoteSnapshotByKey(remoteKey)
if err != nil {
log.Error("Failed to delete metadata directory with no manifest",
"remote_key", remoteKey, "error", err)
result.MetadataDirsFailed++
continue
}
log.Info("Deleted metadata directory with no manifest",
"remote_key", remoteKey)
result.MetadataDirsDeleted++
}
}
// reportBlobsTooNew says how many unreferenced blobs prune kept for being // reportBlobsTooNew says how many unreferenced blobs prune kept for being
// newer than the grace period or of unknown age. Under --json the // newer than the grace period or of unknown age. Under --json the
// document carries the count instead. // document carries the count instead.
+5 -3
View File
@@ -24,8 +24,9 @@ import (
// TestPruneDuringBackupKeepsItsBlobs runs a prune from another host // TestPruneDuringBackupKeepsItsBlobs runs a prune from another host
// sharing the destination store at the moment a backup has uploaded its // sharing the destination store at the moment a backup has uploaded its
// blobs but not yet its manifest, so no manifest lists those blobs. The // blobs and its db.zst.age but not yet its manifest, so no manifest lists
// prune must leave them, and the backup must then restore. // those blobs. The prune must leave the blobs and the db.zst.age, and the
// backup must then restore.
func TestPruneDuringBackupKeepsItsBlobs(t *testing.T) { func TestPruneDuringBackupKeepsItsBlobs(t *testing.T) {
log.Initialize(log.Config{}) log.Initialize(log.Config{})
t.Parallel() t.Parallel()
@@ -79,7 +80,8 @@ func TestPruneDuringBackupKeepsItsBlobs(t *testing.T) {
SnapshotID: id, SnapshotID: id,
TargetDir: restoreDir, TargetDir: restoreDir,
Verify: true, Verify: true,
}), "a prune during the backup must not delete the backup's blobs") }), "a prune during the backup must not delete the backup's blobs "+
"or its db.zst.age")
assertRestoredTree(t, fs, restoreDir, testFiles) assertRestoredTree(t, fs, restoreDir, testFiles)
} }
+225
View File
@@ -0,0 +1,225 @@
package vaultik_test
import (
"bytes"
"context"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/storage/faultstore"
"sneak.berlin/go/vaultik/internal/ui"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// metadataObjectKey returns the key of the named file in the metadata
// directory of the snapshot with the given human ID.
func metadataObjectKey(snapshotID, name string) string {
return "metadata/" + snapshot.RemoteSnapshotKey(snapshotID) + "/" + name
}
// TestPruneDeletesMetadataDirWithoutManifestOlderThanGracePeriod checks
// that prune deletes a directory under metadata/ holding only
// db.zst.age, as a backup leaves when its manifest upload fails, once it
// is older than the grace period, and keeps one newer than that. A
// directory with a manifest is never deleted, however old.
func TestPruneDeletesMetadataDirWithoutManifestOlderThanGracePeriod(
t *testing.T,
) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
storeDir := t.TempDir()
store, err := storage.NewFileStorer(storeDir)
require.NoError(t, err)
const (
oldID = "testhost_home_2026-01-01T00:00:00Z"
newID = "testhost_home_2026-01-02T00:00:00Z"
completeID = "testhost_home_2026-01-03T00:00:00Z"
)
oldDB := metadataObjectKey(oldID, "db.zst.age")
newDB := metadataObjectKey(newID, "db.zst.age")
completeDB := metadataObjectKey(completeID, "db.zst.age")
completeManifest := metadataObjectKey(completeID, "manifest.json.zst")
manifest, err := snapshot.EncodeManifest(&snapshot.Manifest{
SnapshotID: snapshot.RemoteSnapshotKey(completeID),
}, 3)
require.NoError(t, err)
for _, key := range []string{oldDB, newDB, completeDB} {
require.NoError(t, store.Put(ctx, key, strings.NewReader("db")))
}
require.NoError(t, store.Put(ctx, completeManifest,
bytes.NewReader(manifest)))
gracePeriod := time.Hour
beforeGracePeriod := time.Now().Add(-gracePeriod - time.Minute)
for _, key := range []string{oldDB, completeDB, completeManifest} {
require.NoError(t, os.Chtimes(filepath.Join(storeDir, key),
beforeGracePeriod, beforeGracePeriod))
}
stdout := &bytes.Buffer{}
v := &vaultik.Vaultik{
Config: &config.Config{PruneGracePeriod: gracePeriod},
Storage: store,
Stdout: stdout,
UI: ui.NewWithColor(stdout, false),
}
v.SetContext(ctx)
require.NoError(t, v.PruneBlobs(&vaultik.PruneOptions{JSON: true}))
_, err = store.Stat(ctx, oldDB)
require.ErrorIs(t, err, storage.ErrNotFound,
"a directory with no manifest older than the grace period "+
"must be deleted")
for _, key := range []string{newDB, completeDB, completeManifest} {
_, err = store.Stat(ctx, key)
require.NoError(t, err, "%s must be kept", key)
}
var result vaultik.PruneBlobsResult
require.NoError(t, json.Unmarshal(stdout.Bytes(), &result))
assert.Equal(t, 1, result.MetadataDirsDeleted)
}
// TestPruneKeepsMetadataDirClaimedUnknownAgeOrNotRemoteKey checks that
// prune --force keeps a directory under metadata/ with no manifest when a
// local snapshot record claims it, when the listing gives no time for an
// object in it, or when its name is not a remote key. A fourth such
// directory is named, deleted and reported, although there are no
// unreferenced blobs to delete.
func TestPruneKeepsMetadataDirClaimedUnknownAgeOrNotRemoteKey(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
env := newListEnv(t)
const (
claimedID = "testhost_home_2026-01-01T00:00:00Z"
leftID = "testhost_home_2026-01-03T00:00:00Z"
)
env.addLocal(t, claimedID, time.Now().Add(-time.Hour))
claimedDB := metadataObjectKey(claimedID, "db.zst.age")
unknownAgeDB := metadataObjectKey("testhost_home_2026-01-02T00:00:00Z",
"db.zst.age")
leftDB := metadataObjectKey(leftID, "db.zst.age")
notRemoteKeyDB := "metadata/not-a-remote-key/db.zst.age"
for _, key := range []string{claimedDB, leftDB, notRemoteKeyDB} {
require.NoError(t, env.store.Put(ctx, key, strings.NewReader("db")))
}
// Written into the mock store's map directly rather than through
// Put, so its listing gives no time.
env.store.data[unknownAgeDB] = []byte("db")
require.NoError(t, env.v.PruneBlobs(&vaultik.PruneOptions{Force: true}))
assert.True(t, env.store.hasKey(claimedDB),
"a directory a local snapshot record claims must be kept")
assert.True(t, env.store.hasKey(unknownAgeDB),
"a directory holding an object of unknown age must be kept")
assert.True(t, env.store.hasKey(notRemoteKeyDB),
"a directory whose name is not a remote key must be kept")
assert.False(t, env.store.hasKey(leftDB),
"a directory with no manifest, unclaimed and old, must be deleted")
output := env.stdout.String()
assert.Contains(t, output, "Found 1 metadata director(ies) left by "+
"backups that stopped before their manifest:")
assert.Contains(t, output,
"metadata/"+snapshot.RemoteSnapshotKey(leftID)+"/")
assert.NotContains(t, output, "not-a-remote-key")
assert.Contains(t, output,
"Deleted 1 metadata director(ies) with no manifest.")
assert.NotContains(t, output, "No unreferenced blobs to remove.")
}
// TestPruneDeclinedKeepsMetadataDirWithoutManifest checks that the prompt
// names a directory under metadata/ with no manifest that prune is about
// to delete, and that answering no keeps it.
func TestPruneDeclinedKeepsMetadataDirWithoutManifest(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newListEnv(t)
const leftID = "testhost_home_2026-01-01T00:00:00Z"
leftDB := metadataObjectKey(leftID, "db.zst.age")
require.NoError(t, env.store.Put(context.Background(), leftDB,
strings.NewReader("db")))
env.v.Stdin = strings.NewReader("n\n")
require.NoError(t, env.v.PruneBlobs(&vaultik.PruneOptions{}))
assert.True(t, env.store.hasKey(leftDB),
"a declined prune must keep the directory")
output := env.stdout.String()
assert.Contains(t, output,
" metadata/"+snapshot.RemoteSnapshotKey(leftID)+"/\n")
assert.Contains(t, output, "Delete 1 metadata director(ies)? [y/N] ")
assert.Contains(t, output, "Cancelled")
assert.NotContains(t, output, "Deleted")
}
// TestPruneFailedMetadataDirDeleteReturnsError checks that prune --json
// counts a directory under metadata/ with no manifest that it cannot
// delete, and then returns an error, so the command exits non-zero.
func TestPruneFailedMetadataDirDeleteReturnsError(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newListEnv(t)
leftDB := metadataObjectKey("testhost_home_2026-01-01T00:00:00Z",
"db.zst.age")
require.NoError(t, env.store.Put(context.Background(), leftDB,
strings.NewReader("db")))
store := faultstore.New(env.store)
store.OnDelete = func(key string) bool {
return strings.HasPrefix(key, "metadata/")
}
env.v.Storage = store
err := env.v.PruneBlobs(&vaultik.PruneOptions{JSON: true})
require.ErrorContains(t, err,
"1 metadata director(ies) with no manifest could not be deleted")
assert.True(t, env.store.hasKey(leftDB))
var result vaultik.PruneBlobsResult
require.NoError(t, json.Unmarshal(env.stdout.Bytes(), &result),
"stdout must hold the JSON document, got:\n%s", env.stdout.String())
assert.Equal(t, 1, result.MetadataDirsFailed)
assert.Equal(t, 0, result.MetadataDirsDeleted)
}
+107 -20
View File
@@ -7,6 +7,7 @@ import (
"os" "os"
"path/filepath" "path/filepath"
"regexp" "regexp"
"slices"
"sort" "sort"
"strconv" "strconv"
"strings" "strings"
@@ -52,6 +53,15 @@ var ErrFilesUnreadable = errors.New("file(s) could not be read")
// returns ErrFilesUnreadable instead. // returns ErrFilesUnreadable instead.
var ErrSourcePathEmpty = errors.New("source path(s) were empty") var ErrSourcePathEmpty = errors.New("source path(s) were empty")
// ErrVolumeNotMounted is returned when --prune skipped the post-backup
// prune because a directory that an earlier snapshot of the same name
// found as a mount point was not one in this run, as when the volume
// mounted there is not mounted. The snapshots are complete and kept, so
// the caller can report it apart from a failed backup. A run that also
// skipped unreadable files or found a configured path empty returns
// ErrFilesUnreadable or ErrSourcePathEmpty instead.
var ErrVolumeNotMounted = errors.New("volume(s) were not mounted")
// listRecentLimit caps how many snapshot rows are fetched from the // listRecentLimit caps how many snapshot rows are fetched from the
// local index for listing and purge operations. // local index for listing and purge operations.
const listRecentLimit = 10000 const listRecentLimit = 10000
@@ -142,7 +152,7 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
return err return err
} }
filesUnreadable, emptyPaths, err := v.createNamedSnapshots( filesUnreadable, emptyPaths, unmountedPaths, err := v.createNamedSnapshots(
opts, hostname, snapshotNames) opts, hostname, snapshotNames)
if err != nil { if err != nil {
return err return err
@@ -155,8 +165,10 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
} }
switch { switch {
case opts.Prune && (filesUnreadable > 0 || len(emptyPaths) > 0): case opts.Prune && (filesUnreadable > 0 || len(emptyPaths) > 0 ||
return v.skipPostBackupPrune(filesUnreadable, emptyPaths) len(unmountedPaths) > 0):
return v.skipPostBackupPrune(
filesUnreadable, emptyPaths, unmountedPaths)
case opts.Prune: case opts.Prune:
err = v.runPostBackupPrune(snapshotNames, opts.KeepNewerThan) err = v.runPostBackupPrune(snapshotNames, opts.KeepNewerThan)
if err != nil { if err != nil {
@@ -169,20 +181,22 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
// createNamedSnapshots creates each named snapshot in turn. It returns // createNamedSnapshots creates each named snapshot in turn. It returns
// the number of entries --skip-errors skipped because it could not read // the number of entries --skip-errors skipped because it could not read
// or list them, and each configured path in which the scan found // or list them, each configured path in which the scan found nothing,
// nothing, named with its snapshot. // and, under --prune, each directory where an earlier snapshot found a
// volume mounted and this run did not, the paths named with their
// snapshot.
func (v *Vaultik) createNamedSnapshots( func (v *Vaultik) createNamedSnapshots(
opts *SnapshotCreateOptions, hostname string, snapshotNames []string, opts *SnapshotCreateOptions, hostname string, snapshotNames []string,
) (int, []string, error) { ) (int, []string, []string, error) {
filesUnreadable := 0 filesUnreadable := 0
var emptyPaths []string var emptyPaths, unmountedPaths []string
for snapIdx, snapName := range snapshotNames { for snapIdx, snapName := range snapshotNames {
stats, err := v.createNamedSnapshot( stats, err := v.createNamedSnapshot(
opts, hostname, snapName, snapIdx+1, len(snapshotNames)) opts, hostname, snapName, snapIdx+1, len(snapshotNames))
if err != nil { if err != nil {
return 0, nil, err return 0, nil, nil, err
} }
filesUnreadable += stats.totalFilesUnreadable filesUnreadable += stats.totalFilesUnreadable
@@ -190,21 +204,27 @@ func (v *Vaultik) createNamedSnapshots(
for _, dir := range stats.emptyPaths { for _, dir := range stats.emptyPaths {
emptyPaths = append(emptyPaths, dir+" of snapshot "+snapName) emptyPaths = append(emptyPaths, dir+" of snapshot "+snapName)
} }
for _, dir := range stats.unmountedPaths {
unmountedPaths = append(unmountedPaths,
dir+" of snapshot "+snapName)
}
} }
return filesUnreadable, emptyPaths, nil return filesUnreadable, emptyPaths, unmountedPaths, nil
} }
// skipPostBackupPrune says why the post-backup prune does not run, naming // skipPostBackupPrune says why the post-backup prune does not run, naming
// every cause, and returns the error the run ends with. An older snapshot // every cause, and returns the error the run ends with. An older snapshot
// may be the only one that still holds a file this run could not read, // may be the only one that still holds a file this run could not read,
// or what an empty path held. The previous snapshot is no guide: after // or what an empty path or a volume that is not mounted held. The
// one run with an empty path it is empty too. // previous snapshot is no guide: after one such run it lacks them too.
func (v *Vaultik) skipPostBackupPrune( func (v *Vaultik) skipPostBackupPrune(
filesUnreadable int, emptyPaths []string, filesUnreadable int, emptyPaths, unmountedPaths []string,
) error { ) error {
log.Warn("Skipping post-backup prune", log.Warn("Skipping post-backup prune",
"files_unreadable", filesUnreadable, "empty_paths", emptyPaths) "files_unreadable", filesUnreadable, "empty_paths", emptyPaths,
"unmounted_paths", unmountedPaths)
if len(emptyPaths) > 0 { if len(emptyPaths) > 0 {
v.UI.Warningf("Skipped the post-backup prune because nothing was "+ v.UI.Warningf("Skipped the post-backup prune because nothing was "+
@@ -212,6 +232,15 @@ func (v *Vaultik) skipPostBackupPrune(
strings.Join(emptyPaths, ", ")) strings.Join(emptyPaths, ", "))
} }
if len(unmountedPaths) > 0 {
v.UI.Warningf("Skipped the post-backup prune because no volume "+
"is mounted at %s, where an earlier snapshot found one; older "+
"snapshots are kept. If a volume was retired on purpose, "+
"remove the older snapshots by hand with "+
"'vaultik snapshot purge'.",
strings.Join(unmountedPaths, ", "))
}
if filesUnreadable > 0 { if filesUnreadable > 0 {
v.UI.Warningf("Skipped the post-backup prune because %d file(s) "+ v.UI.Warningf("Skipped the post-backup prune because %d file(s) "+
"could not be read; older snapshots are kept.", filesUnreadable) "could not be read; older snapshots are kept.", filesUnreadable)
@@ -219,7 +248,11 @@ func (v *Vaultik) skipPostBackupPrune(
return fmt.Errorf("%d %w", filesUnreadable, ErrFilesUnreadable) return fmt.Errorf("%d %w", filesUnreadable, ErrFilesUnreadable)
} }
return fmt.Errorf("%d %w", len(emptyPaths), ErrSourcePathEmpty) if len(emptyPaths) > 0 {
return fmt.Errorf("%d %w", len(emptyPaths), ErrSourcePathEmpty)
}
return fmt.Errorf("%d %w", len(unmountedPaths), ErrVolumeNotMounted)
} }
// finishCreateSnapshot ends a run whose snapshots are all complete and // finishCreateSnapshot ends a run whose snapshots are all complete and
@@ -270,9 +303,10 @@ func (v *Vaultik) snapshotNamesToCreate(requested []string) ([]string, error) {
} }
// runPostBackupPrune drops older snapshots of the given names and removes // runPostBackupPrune drops older snapshots of the given names and removes
// orphan blobs from remote storage. If keepNewerThan is set (e.g. "4w"), // from remote storage the orphan blobs and the directories under
// snapshots newer than that duration are kept. Otherwise only the latest // metadata/ with no manifest, as PruneBlobs does. If keepNewerThan is set
// snapshot of each name is kept. // (e.g. "4w"), snapshots newer than that duration are kept. Otherwise only
// the latest snapshot of each name is kept.
func (v *Vaultik) runPostBackupPrune( func (v *Vaultik) runPostBackupPrune(
snapshotNames []string, keepNewerThan string, snapshotNames []string, keepNewerThan string,
) error { ) error {
@@ -299,7 +333,7 @@ func (v *Vaultik) runPostBackupPrune(
err = v.PruneBlobs(&PruneOptions{Force: true}) err = v.PruneBlobs(&PruneOptions{Force: true})
if err != nil { if err != nil {
return fmt.Errorf("pruning orphaned blobs: %w", err) return fmt.Errorf("pruning the destination store: %w", err)
} }
return nil return nil
@@ -328,6 +362,10 @@ type snapshotStats struct {
// The configured paths in which the scan found nothing. // The configured paths in which the scan found nothing.
emptyPaths []string emptyPaths []string
// Under --prune, the directories where an earlier snapshot of the
// same name found a volume mounted and this one did not.
unmountedPaths []string
// The sizes of all blobs the snapshot references, set by // The sizes of all blobs the snapshot references, set by
// finalizeSnapshotMetadata once snapshot_blobs is populated. // finalizeSnapshotMetadata once snapshot_blobs is populated.
blobSize int64 blobSize int64
@@ -393,9 +431,57 @@ func (v *Vaultik) createNamedSnapshot(
v.printSnapshotSummary(snapshotID, snapshotStartTime, stats) v.printSnapshotSummary(snapshotID, snapshotStartTime, stats)
if opts.Prune {
stats.unmountedPaths, err = v.findUnmountedPaths(snapName, snapshotID)
if err != nil {
return nil, err
}
}
return stats, nil return stats, nil
} }
// findUnmountedPaths returns each directory that an earlier snapshot
// named snapName in the local index recorded as a mount point and
// snapshot snapshotID did not, as when the volume mounted there is not
// mounted.
func (v *Vaultik) findUnmountedPaths(
snapName, snapshotID string,
) ([]string, error) {
found, err := v.Repositories.Snapshots.GetMountPoints(v.ctx, snapshotID)
if err != nil {
return nil, err
}
snapshots, err := v.Repositories.Snapshots.ListRecent(v.ctx, listRecentLimit)
if err != nil {
return nil, fmt.Errorf("listing snapshots: %w", err)
}
var unmounted []string
for _, s := range snapshots {
if s.ID.String() == snapshotID ||
parseSnapshotName(s.ID.String(), s.Hostname.String()) != snapName {
continue
}
recorded, err := v.Repositories.Snapshots.GetMountPoints(
v.ctx, s.ID.String())
if err != nil {
return nil, err
}
for _, dir := range recorded {
if !slices.Contains(found, dir) && !slices.Contains(unmounted, dir) {
unmounted = append(unmounted, dir)
}
}
}
return unmounted, nil
}
// previousSnapshotID returns the ID of the most recent completed snapshot // previousSnapshotID returns the ID of the most recent completed snapshot
// named snapName in the local index, or "" when there is none. // named snapName in the local index, or "" when there is none.
func (v *Vaultik) previousSnapshotID(snapName string) (string, error) { func (v *Vaultik) previousSnapshotID(snapName string) (string, error) {
@@ -1365,7 +1451,7 @@ const pruneCommandHint = "vaultik prune"
// snapshotRemoveCommandHint is the command suggested, with the // snapshotRemoveCommandHint is the command suggested, with the
// snapshot's ID, when a remove could not reach the destination store: // snapshot's ID, when a remove could not reach the destination store:
// running it again removes the snapshot's metadata there, which // running it again removes the snapshot's metadata there, which
// `vaultik prune` never does. // `vaultik prune` never does while the manifest is there.
const snapshotRemoveCommandHint = "vaultik snapshot remove" const snapshotRemoveCommandHint = "vaultik snapshot remove"
// RemoveSnapshot removes a snapshot from the local index database and, // RemoveSnapshot removes a snapshot from the local index database and,
@@ -1568,7 +1654,8 @@ func (v *Vaultik) confirmRemoveSnapshot(snapshotID string, opts *RemoveOptions)
// the local index, its removal there has already happened, so a failure // the local index, its removal there has already happened, so a failure
// is only a warning telling the user to run `vaultik snapshot remove` // is only a warning telling the user to run `vaultik snapshot remove`
// for the snapshot again once the destination store is reachable // for the snapshot again once the destination store is reachable
// (`vaultik prune` never removes snapshot metadata). // (`vaultik prune` never removes snapshot metadata that has its
// manifest).
func (v *Vaultik) removeSnapshotRemote( func (v *Vaultik) removeSnapshotRemote(
snapshotID string, inLocalIndex bool, markerErr error, snapshotID string, inLocalIndex bool, markerErr error,
opts *RemoveOptions, opts *RemoveOptions,
+96
View File
@@ -10,6 +10,7 @@ import (
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/config" "sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/log" "sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot"
"sneak.berlin/go/vaultik/internal/vaultik" "sneak.berlin/go/vaultik/internal/vaultik"
) )
@@ -259,3 +260,98 @@ func TestBackupWithNoEmptySourcePathPrunes(t *testing.T) {
}) })
} }
} }
// A backup under --prune in which a volume that an earlier snapshot found
// mounted below the configured path is not mounted does not prune: run
// after run, the snapshot that holds the volume's files is kept with
// everything it stored, and the run fails. Once the volume is mounted
// again, the prune runs. See https://git.eeqj.de/sneak/vaultik/issues/395.
func TestBackupWithVolumeNotMountedDoesNotPrune(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
// volume is the directory, below the configured path, where the
// volume is mounted.
const (
snapshotName = "data"
volume = "nas"
)
ctx := context.Background()
env := newDeletedCountEnv(t, map[string][]byte{
volume + "/a.txt": []byte("file a"),
"notes.txt": []byte("notes"),
})
env.v.Config.Snapshots[snapshotName] = config.SnapshotConfig{
Paths: []string{env.src},
}
prune := &vaultik.SnapshotCreateOptions{
Prune: true, Snapshots: []string{snapshotName},
}
// While the volume is mounted, its directory is on a device of its
// own.
mounted := true
newScanner := env.v.ScannerFactory
env.v.ScannerFactory = func(params snapshot.ScannerParams) *snapshot.Scanner {
params.DeviceNumber = func(info os.FileInfo) (uint64, bool) {
if mounted && info.Name() == volume {
return 2, true
}
return 1, true
}
return newScanner(params)
}
env.backUp(t, snapshotName)
firstID := localSnapshotID(ctx, t, env.v.Repositories, snapshotName)
stored, err := env.v.Storage.List(ctx, "")
require.NoError(t, err)
// A volume that is not mounted leaves its mount point behind as an
// empty directory on its parent's device.
volumeDir := filepath.Join(env.src, volume)
require.NoError(t, os.RemoveAll(volumeDir))
require.NoError(t, os.Mkdir(volumeDir, 0o750))
mounted = false
for range 2 {
env.out.Reset()
err := env.v.CreateSnapshot(prune)
require.ErrorIs(t, err, vaultik.ErrVolumeNotMounted)
assert.Contains(t, env.out.String(),
"no volume is mounted at "+volumeDir+" of snapshot "+snapshotName)
assert.Contains(t, env.out.String(), "'vaultik snapshot purge'")
assert.NotContains(t, env.out.String(), "Running post-backup prune.")
}
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
ctx, listRecentTestLimit)
require.NoError(t, err)
require.Len(t, snapshots, 3)
assert.Equal(t, firstID, snapshots[2].ID.String(),
"the snapshot that holds the volume's files is kept")
for _, key := range stored {
_, err := env.v.Storage.Stat(ctx, key)
require.NoError(t, err, "%s is kept", key)
}
// The volume is mounted again.
require.NoError(t, os.WriteFile(
filepath.Join(volumeDir, "a.txt"), []byte("file a"), 0o600))
mounted = true
require.NoError(t, env.v.CreateSnapshot(prune))
snapshots, err = env.v.Repositories.Snapshots.ListRecent(
ctx, listRecentTestLimit)
require.NoError(t, err)
require.Len(t, snapshots, 1, "the older snapshots are pruned")
}