1 Commits
Author SHA1 Message Date
sneak 63c824220c Keep unreferenced blobs newer than prune_grace_period (closes #387)
check / check (push) Failing after 5m50s
A backup uploads its blobs before the manifest that lists them, and the
process lock covers one user on one machine, so a prune from another host
or user sharing the destination deleted a running backup's blobs.

Prune now deletes an unreferenced blob only once its listed time is older
than the new config key prune_grace_period (default 24h; a negative value
is refused). A newer blob, or one listed with no time, is kept and counted
(blobs_too_new). A backup uploads a blob already on the destination again
instead of skipping it, so it gets a fresh time. The rclone backend takes
each object's time from the listing (use_server_modtime).

Cost: after database delete, the next backup uploads all its data again.
Side effect: a rebuilt blob replaces the stored copy; if compression_level
or the recipients changed since, older snapshots listing it fail verify.

Model: opus-5-5
2026-10-14 08:57:35 +00:00
37 changed files with 374 additions and 2190 deletions
+10 -15
View File
@@ -312,7 +312,7 @@ Repository interfaces:
``` ```
CreateSnapshot(opts) CreateSnapshot(opts)
│ │
├─► markDestination() // Check the marker (see Storage Layout in S3), then write it ├─► markDestination() // Refuse a destination without the marker, then write it
│ │
├─► PruneDatabase() // Critical: avoid dedup errors ├─► PruneDatabase() // Critical: avoid dedup errors
│ │
@@ -382,20 +382,15 @@ bucket/
└── manifest.json.zst # Blob list (for pruning/verification) └── manifest.json.zst # Blob list (for pruning/verification)
``` ```
A destination that answers a look-up of `vaultik-destination.txt` with not A destination without `vaultik-destination.txt` is treated as not present
found is treated as not present while the local index holds snapshots stored while the local index holds snapshots stored there (`checkDestinationMarker`
there (`checkDestinationMarker` in `internal/vaultik/destination_marker.go`): in `internal/vaultik/destination_marker.go`): `snapshot create` refuses to
`snapshot create` refuses to write to it, `prune`, `snapshot purge` and write to it, `prune`, `snapshot purge` and `remote nuke` fail, and
`remote nuke` fail, and `snapshot list` and `snapshot remove` behave as for a `snapshot list` and `snapshot remove` behave as for a destination that cannot
destination that cannot be listed. This keeps those commands from taking an be listed. This keeps those commands from taking an unplugged volume's empty
unplugged volume's empty mount point for a destination whose snapshots are mount point for a destination whose snapshots are all gone. The `file://`
all gone. When the look-up fails for any reason other than not found, for backend never creates its destination directory, so a write to a missing one
example when S3 refuses it to credentials that allow only PUT, the check is fails.
skipped, so a host with such credentials does not notice an emptied
destination; see
[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#marker-file-vaultik-destinationtxt).
The `file://` backend never creates its destination directory, so a write to a
missing one fails.
The `{remote-key}` directory name is a one-way double SHA-256 hash of the human The `{remote-key}` directory name is a one-way double SHA-256 hash of the human
snapshot ID, so the human ID (hostname, snapshot name, timestamp) is never snapshot ID, so the human ID (hostname, snapshot name, timestamp) is never
+41 -97
View File
@@ -98,12 +98,9 @@ vaultik snapshot restore <snapshot-id> ~/vaultik-restore
On a destination shared between hosts or users, the prune in that cron line On a destination shared between hosts or users, the prune in that cron line
relies on `prune_grace_period` (default `24h`) to leave the blobs of a backup relies on `prune_grace_period` (default `24h`) to leave the blobs of a backup
another one is still running, and its directory under `metadata/` that has no another one is still running, so it must be longer than the longest backup
manifest yet, so it must be longer than the longest backup run. It does not run; otherwise give each host its own prefix or directory. See
protect a blob that backup reuses from an earlier run that stopped before its [locking](#locking).
manifest: the prune can delete such a blob, and the new snapshot then cannot
be restored. To not depend on the grace period, give each host its own prefix
or directory. See [locking](#locking).
## restoring on another machine ## restoring on another machine
@@ -205,19 +202,13 @@ corrupt the local index.
The lock covers one user on one machine. It does not stop another user, The lock covers one user on one machine. It does not stop another user,
or another host, from writing to the same destination store at the same or another host, from writing to the same destination store at the same
time. A backup uploads its blobs, and then its `db.zst.age`, before its time. A backup uploads its blobs before the manifest that lists them, so a
manifest, so a prune from elsewhere at that moment finds blobs no snapshot prune from elsewhere at that moment finds blobs no snapshot lists. It
lists and a directory under `metadata/` with no manifest. It deletes such a blob only once it is older than `prune_grace_period`
deletes such a blob only once it is older than `prune_grace_period` (default (default `24h`), which therefore must be longer than the longest backup run
`24h`), and such a directory only once every object in it is, so the grace of any host or user sharing the destination. `remote nuke` deletes
period must be longer than the longest backup run of any host or user sharing everything regardless. To not depend on the grace period, give each host
the destination. The grace period does not protect a blob a backup reuses or user its own prefix (or directory) on the destination.
from an earlier run that stopped before its manifest: a backup skips the
upload of a blob already on the destination, so that blob keeps the time of
its first upload, and a prune from elsewhere during the backup can delete it;
the new snapshot then cannot be restored. `remote nuke` deletes everything
regardless. To not depend on the grace period, give each host or user its own
prefix (or directory) on the destination.
Read-only commands — `info`, `snapshot list`, `snapshot verify`, and Read-only commands — `info`, `snapshot list`, `snapshot verify`, and
`remote info` — do not take the lock and are never blocked, so they run `remote info` — do not take the lock and are never blocked, so they run
@@ -320,46 +311,22 @@ value where a list belongs or a word where a number belongs), is refused.
error that explains how to fix it error that explains how to fix it
* `--cron`: Silent on total success; warnings and errors are still printed * `--cron`: Silent on total success; warnings and errors are still printed
(for crontab) (for crontab)
* `--prune`: After backup, drop older snapshots of each backed-up name, * `--prune`: After backup, drop older snapshots of each backed-up name and
remove orphaned blobs older than `prune_grace_period` from remote storage, remove orphaned blobs older than `prune_grace_period` from remote storage,
and delete each directory under `metadata/` that has no as `prune` does. By default keeps only the latest
`manifest.json.zst` once every object in it is older than
`prune_grace_period` and its snapshot is not in the local index, as `prune`
does. By default keeps only the latest
snapshot per name; use `--keep-newer-than` for a rolling window. When snapshot per name; use `--keep-newer-than` for a rolling window. When
`--skip-errors` skipped a file or directory it could not read or list, the `--skip-errors` skipped a file or directory it could not read or list, the
prune does not run and every older snapshot is kept, since one of them may prune does not run and every older snapshot is kept, since one of them may
be the only one that holds what was skipped; the command says so and exits be the only one that holds what was skipped; the command says so and exits
non-zero. When the backup found nothing in a configured path, as in the non-zero. A file removed during the backup does not stop the prune.
mount point of a volume that is not mounted, the prune does not run
either: every older snapshot is kept, and the command names the path and
exits non-zero, on every run until the path holds something again. Take a
path that is empty on purpose out of the config, or remove the older
snapshots by hand with `snapshot purge`. When an earlier snapshot of the
same name in the local index found a volume mounted at or below a
configured path (a directory on a different filesystem from its parent,
with an entry in it that the backup does not exclude) and this backup does
not, the prune does not run either: every older snapshot is kept, and the
command names the directory and exits non-zero, on every run until the
volume is mounted again. A backup also does not find the volume when it
finds no such entry in its directory, as when an automounted volume cannot
be mounted, or when that directory is no longer backed up, because it is
excluded or its configured path was removed. If any of this is on purpose,
as for a retired volume, remove the older snapshots by hand with
`snapshot purge`. A file removed during the backup does not stop the
prune.
* `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than * `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than
this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`). this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`).
Without `--prune`, or with a value that is not a valid duration, the Without `--prune`, or with a value that is not a valid duration, the
command fails before the backup starts. command fails before the backup starts.
* Fails before it writes anything when a `file://` destination directory * Fails before it writes anything when a `file://` destination directory
does not exist, or when the local index holds snapshots and the does not exist, or when the local index holds snapshots and the
destination answers that it has no `vaultik-destination.txt`. When that destination has no `vaultik-destination.txt` (see
look-up fails for any reason other than not found, for example when S3
refuses it to credentials that allow only PUT, the check is skipped (see
[remote storage layout](#remote-storage-layout)) [remote storage layout](#remote-storage-layout))
* Fails, before anything is pruned, when a configured path is a symlink
whose target does not exist, as one into a volume that is not mounted
**`snapshot list`**: Show every snapshot known to this host — the union **`snapshot list`**: Show every snapshot known to this host — the union
of the local index and the backup destination store — with timestamps of the local index and the backup destination store — with timestamps
@@ -394,9 +361,9 @@ gives: revert the configuration, or run `vaultik database delete`.
If the destination store cannot be listed (unmounted volume, If the destination store cannot be listed (unmounted volume,
permission denied, network down), the command warns, falls back to the permission denied, network down), the command warns, falls back to the
local index alone, and still exits zero. It does the same when the local index alone, and still exits zero. It does the same when the
destination answers that it has no `vaultik-destination.txt` while the destination has no `vaultik-destination.txt` while the local index
local index holds snapshots stored there, as when a volume mounted at holds snapshots stored there, as when a volume mounted at the
the destination is unplugged and leaves its empty mount point behind. destination is unplugged and leaves its empty mount point behind.
* `--json`: Output in JSON format. Each entry carries `locally_tracked` * `--json`: Output in JSON format. Each entry carries `locally_tracked`
(whether the snapshot is in the local index), `remote_key` (the full (whether the snapshot is in the local index), `remote_key` (the full
64-character storage key), and `remote_present` (whether it was seen 64-character storage key), and `remote_present` (whether it was seen
@@ -439,8 +406,8 @@ with `--local-only`, the snapshot must be in the local index. Local row
cleanup (files, chunks, blobs the snapshot was the last referrer for) cleanup (files, chunks, blobs the snapshot was the last referrer for)
runs automatically. If removing the snapshot's metadata from the runs automatically. If removing the snapshot's metadata from the
destination store fails (for example because the store is unreachable, destination store fails (for example because the store is unreachable,
or the local index holds snapshots and the destination answers that it or the local index holds snapshots and the destination has no
has no `vaultik-destination.txt`), the local-DB removal still completes and a `vaultik-destination.txt`), the local-DB removal still completes and a
warning is emitted; run `vaultik snapshot remove <snapshot-id>` again warning is emitted; run `vaultik snapshot remove <snapshot-id>` again
once the store is reachable to remove the snapshot's metadata from it once the store is reachable to remove the snapshot's metadata from it
(`vaultik prune` does not). For a snapshot that is only on the (`vaultik prune` does not). For a snapshot that is only on the
@@ -476,32 +443,22 @@ manifest on the destination store to compute the still-referenced blob
set and delete any blob not in that set that is older than set and delete any blob not in that set that is older than
`prune_grace_period` (default `24h`). A newer one is kept and counted: it `prune_grace_period` (default `24h`). A newer one is kept and counted: it
may belong to a backup still running, on this host or another sharing the may belong to a backup still running, on this host or another sharing the
destination store, that has not yet uploaded its manifest. The grace period destination store, that has not yet uploaded its manifest. Step (3) reads
does not protect a blob such a backup reuses from an earlier run that stopped all remote manifests — network cost scales with the number of snapshots.
before its manifest: that blob keeps the time of its first upload, so it can
be deleted. Step (3) also deletes each directory under `metadata/` that has
no `manifest.json.zst`, as a backup leaves when it stops after uploading
`db.zst.age` but before its manifest, once every object in it is older than
`prune_grace_period` and its snapshot is not in the local index. An object the
listing gives no time for keeps its directory, and a directory whose name is
not a remote key is left alone. Step (3) reads all remote manifests — network
cost scales with the number of snapshots.
`snapshot create --prune` runs the same cleanup automatically; this is the `snapshot create --prune` runs the same cleanup automatically; this is the
manual entry point for the same work. If the destination store cannot manual entry point for the same work. If the destination store cannot
be listed, or the local index holds snapshots and the destination answers be listed, or the local index holds snapshots and the destination has
that it has no `vaultik-destination.txt` (an unplugged volume's empty mount no `vaultik-destination.txt` (an unplugged volume's empty mount point),
point), prune fails and deletes nothing. `snapshot purge` does the same. If a prune fails and deletes nothing. `snapshot purge` does the same. If a
blob or such a directory cannot be deleted, the command (and blob cannot be deleted, the command (and `snapshot create --prune`)
`snapshot create --prune`) exits non-zero. exits non-zero.
* `--force`: Skip confirmation prompt * `--force`: Skip confirmation prompt
* `--json`: Output stats as JSON. Also skips the confirmation prompt, as * `--json`: Output stats as JSON. Also skips the confirmation prompt, as
`--force` does. A blob that cannot be deleted is counted in `--force` does. A blob that cannot be deleted is counted in
`blobs_failed`, and the document is printed before the non-zero exit. `blobs_failed`, and the document is printed before the non-zero exit.
A blob kept for being newer than `prune_grace_period`, or because the A blob kept for being newer than `prune_grace_period`, or because the
listing of the destination gives no time for it, is counted in listing of the destination gives no time for it, is counted in
`blobs_too_new` and not in `blobs_found`. A directory under `metadata/` `blobs_too_new` and not in `blobs_found`.
with no manifest is counted in `metadata_dirs_deleted` once deleted, or
in `metadata_dirs_failed` if it cannot be.
**`info`**: Display system configuration, storage settings, encryption **`info`**: Display system configuration, storage settings, encryption
recipients, and local database statistics. recipients, and local database statistics.
@@ -527,8 +484,8 @@ and irreversible. This is the single supported way to wipe the entire
destination store. A snapshot is removed from the local index only once destination store. A snapshot is removed from the local index only once
its metadata is deleted from the destination store. If the destination its metadata is deleted from the destination store. If the destination
store cannot be listed, or the local index holds snapshots and the store cannot be listed, or the local index holds snapshots and the
destination answers that it has no `vaultik-destination.txt` (an unplugged destination has no `vaultik-destination.txt` (an unplugged volume's
volume's empty mount point), the command fails before it removes anything. If a empty mount point), the command fails before it removes anything. If a
listing afterwards still finds anything under `metadata/` or `blobs/`, listing afterwards still finds anything under `metadata/` or `blobs/`,
the command names each object left and exits non-zero; running it again the command names each object left and exits non-zero; running it again
deletes what is left. deletes what is left.
@@ -536,8 +493,8 @@ deletes what is left.
**`database delete`**: Delete the local SQLite state database file **`database delete`**: Delete the local SQLite state database file
entirely. Remote storage is unaffected; the next backup will do a full entirely. Remote storage is unaffected; the next backup will do a full
scan and re-deduplicate against existing remote blobs, and the local scan and upload all the data again, and the local index will re-bind to
index will re-bind to the currently configured storage destination. the currently configured storage destination.
Use this after changing `storage_url`, or `s3.endpoint`, `s3.bucket` or Use this after changing `storage_url`, or `s3.endpoint`, `s3.bucket` or
`s3.prefix`, to a different destination. `s3.prefix`, to a different destination.
* `--force`: Skip confirmation prompt * `--force`: Skip confirmation prompt
@@ -578,11 +535,8 @@ and can be deleted.
A destination shared between hosts or users, each with its own local index, A destination shared between hosts or users, each with its own local index,
relies on `prune_grace_period` to keep one's prune from deleting the blobs of relies on `prune_grace_period` to keep one's prune from deleting the blobs of
another's backup still running, or its directory under `metadata/` that has another's backup still running, so it must be longer than the longest backup
no manifest yet, so it must be longer than the longest backup run. It does not run; otherwise give each host its own prefix (or directory). See
protect a blob that backup reuses from an earlier run that
stopped before its manifest, which the prune can delete. To not depend on the
grace period, give each host its own prefix (or directory). See
[locking](#locking). [locking](#locking).
Legacy S3 configuration via `s3.*` fields (endpoint, bucket, prefix, etc.) is Legacy S3 configuration via `s3.*` fields (endpoint, bucket, prefix, etc.) is
@@ -611,22 +565,15 @@ For an implementation-level view of the internals — the data model, the
``` ```
* `vaultik-destination.txt` holds one fixed line of text, with no host * `vaultik-destination.txt` holds one fixed line of text, with no host
name, path or time. While the local index holds snapshots stored name, path or time. A destination without it is treated as not
there, a destination that answers a look-up of it with not found is present while the local index holds snapshots stored there:
treated as not present: `snapshot create` refuses to write to it, `snapshot create` refuses to write to it, `prune`, `snapshot purge`
`prune`, `snapshot purge` and `remote nuke` fail, and `snapshot list` and `remote nuke` fail, and `snapshot list` and `snapshot remove`
and `snapshot remove` behave as for a destination that cannot be behave as for a destination that cannot be listed. This keeps an
listed. This keeps an unplugged volume's empty mount point from being unplugged volume's empty mount point from being taken for a
taken for a destination whose snapshots are all gone. Mount destination whose snapshots are all gone. Mount
the volume if it is removable; if the destination was emptied on the volume if it is removable; if the destination was emptied on
purpose, run `vaultik database delete` and back up again. purpose, run `vaultik database delete` and back up again.
When the look-up fails for any reason other than not found, the check
is skipped and the command goes on. S3 refuses it on every run to
credentials that allow only PUT, so a host with such credentials does
not notice a destination emptied since its last backup, and its next
backup would leave out the data its local index records as already
stored. After the destination is emptied, run
`vaultik database delete` on such a host before it backs up again.
* Blobs are two-level directory sharded using the first 4 hex chars of the blob hash * Blobs are two-level directory sharded using the first 4 hex chars of the blob hash
* `db.zst.age` is a binary SQLite database (zstd compressed, age encrypted) * `db.zst.age` is a binary SQLite database (zstd compressed, age encrypted)
containing all file metadata, chunk mappings, and relationships for the snapshot containing all file metadata, chunk mappings, and relationships for the snapshot
@@ -680,9 +627,6 @@ format does and does not protect.
3. List all blobs in storage 3. List all blobs in storage
4. Delete any blob not in the referenced set that is older than 4. Delete any blob not in the referenced set that is older than
`prune_grace_period` `prune_grace_period`
5. Delete each directory under `metadata/` with no manifest whose objects
are all older than `prune_grace_period`, unless its snapshot is in the
local index
### chunking and deduplication ### chunking and deduplication
@@ -727,7 +671,7 @@ complete annotated example also lives in
| `chunk_size` | `10MB` | Average chunk size for content-defined chunking | | `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` | | `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` |
| `compression_level` | `3` | zstd compression level (1-19) | | `compression_level` | `3` | zstd compression level (1-19) |
| `prune_grace_period` | `24h` | How old a blob that no snapshot lists must be before `prune` deletes it, and how old every object in a directory under `metadata/` with no manifest must be before `prune` deletes that directory. A backup uploads its blobs and its `db.zst.age` before its manifest, so on a destination shared between hosts or users this must be longer than the longest backup run (see [locking](#locking)). It does not protect a blob a backup reuses from an earlier run that stopped before its manifest: that blob keeps the time of its first upload, so a prune can delete it. A duration such as `36h` or `90m`; `0s` deletes such blobs and directories at once | | `prune_grace_period` | `24h` | How old a blob that no snapshot lists must be before `prune` deletes it. A backup uploads its blobs before the manifest that lists them, so on a destination shared between hosts or users this must be longer than the longest backup run (see [locking](#locking)). A duration such as `36h` or `90m`; `0s` deletes such blobs at once |
| `hostname` | system hostname | Hostname used in snapshot IDs | | `hostname` | system hostname | Hostname used in snapshot IDs |
| `index_path` | platform data dir | Local SQLite index path | | `index_path` | platform data dir | Local SQLite index path |
-17
View File
@@ -24,23 +24,6 @@ owner's; a `v*` tag runs the release build in
# Completed Steps # Completed Steps
- 2026-10-15: Stopped `snapshot create --prune` pruning after a backup
in which a volume mounted below a configured path in an earlier
snapshot of the same name was not mounted
([issue #395](https://git.eeqj.de/sneak/vaultik/issues/395)). Such a
prune could delete every snapshot still holding the volume's files.
Each snapshot now records in the local index the directories it found
on a different filesystem from their parent and found an entry in that
the backup does not exclude.
- 2026-10-14: Stopped `snapshot create --prune` pruning after a backup
that found nothing in a configured path
([issue #388](https://git.eeqj.de/sneak/vaultik/issues/388)), as in
the mount point of a volume that is not mounted. Such a prune could
delete every snapshot still holding that path's files. A configured
path that is a symlink whose target is gone, as one into such a
volume, now fails the backup, with or without `--prune`.
- 2026-10-14: Stopped `snapshot create --prune` pruning after a backup - 2026-10-14: Stopped `snapshot create --prune` pruning after a backup
that skipped files it could not read that skipped files it could not read
([issue #382](https://git.eeqj.de/sneak/vaultik/issues/382)). Such a ([issue #382](https://git.eeqj.de/sneak/vaultik/issues/382)). Such a
+3 -8
View File
@@ -329,16 +329,11 @@ storage_url: "rclone://myremote/path/to/backups"
# Default: 3 # Default: 3
compression_level: 5 compression_level: 5
# How old a blob that no snapshot lists must be before prune deletes it, and # How old a blob that no snapshot lists must be before prune deletes it
# how old every object in a directory under metadata/ with no manifest must # A backup uploads its blobs before the manifest that lists them, so on a
# be before prune deletes that directory
# A backup uploads its blobs and its db.zst.age before its manifest, so on a
# destination shared between hosts or users this must be longer than the # destination shared between hosts or users this must be longer than the
# longest backup run # longest backup run
# It does not protect a blob a backup reuses from an earlier run that stopped # Supports: 36h, 90m, etc.; 0s deletes them at once
# before its manifest: that blob keeps the time of its first upload, so a
# prune can delete it
# Supports: 36h, 90m, etc.; 0s deletes such blobs and directories at once
# Default: 24h # Default: 24h
#prune_grace_period: 24h #prune_grace_period: 24h
+3 -18
View File
@@ -90,7 +90,7 @@ Stores information about packed, compressed, and encrypted blob files.
- `created_ts` (INTEGER NOT NULL) - Creation timestamp - `created_ts` (INTEGER NOT NULL) - Creation timestamp
- `finished_ts` (INTEGER) - Finalization timestamp (NULL if in progress) - `finished_ts` (INTEGER) - Finalization timestamp (NULL if in progress)
- `uncompressed_size` (INTEGER NOT NULL DEFAULT 0) - Total size of chunks before compression - `uncompressed_size` (INTEGER NOT NULL DEFAULT 0) - Total size of chunks before compression
- `compressed_size` (INTEGER NOT NULL DEFAULT 0) - Size after compression and encryption. Once uploaded, the size of the object in storage: when the blob was already stored, under another compression level or other recipients, this can differ from the size of the local copy - `compressed_size` (INTEGER NOT NULL DEFAULT 0) - Size after compression and encryption. Once uploaded, the size of the object in storage
- `uploaded_ts` (INTEGER) - Upload completion timestamp (NULL if not uploaded) - `uploaded_ts` (INTEGER) - Upload completion timestamp (NULL if not uploaded)
**Purpose:** Tracks blob lifecycle from creation through upload. The UUID primary key allows immediate association of chunks with blobs. **Purpose:** Tracks blob lifecycle from creation through upload. The UUID primary key allows immediate association of chunks with blobs.
@@ -153,17 +153,7 @@ Maps snapshots to the blobs they reference.
**Purpose:** Tracks blob dependencies for snapshots and enables manifest generation. **Purpose:** Tracks blob dependencies for snapshots and enables manifest generation.
### 10. `snapshot_mount_points` ### 10. `uploads`
The directories at or below a configured path, the configured path itself included, that a snapshot's backup found an entry in and, once it had listed them, found on a different filesystem from their parent directory (a different device number), as a mount point is. An automount point whose volume cannot be mounted is not recorded, because the backup finds no entry in it.
**Columns:**
- `snapshot_id` (TEXT) - Snapshot ID (FK to snapshots.id)
- `path` (TEXT) - Absolute path of the directory
- PRIMARY KEY (`snapshot_id`, `path`)
**Purpose:** Lets `snapshot create --prune` skip the prune when a backup does not find a mount point that an earlier snapshot of the same name found, as when the volume is not mounted.
### 11. `uploads`
Tracks blob upload metrics. Tracks blob upload metrics.
**Columns:** **Columns:**
@@ -183,7 +173,6 @@ Tracks blob upload metrics.
- `INSERT OR REPLACE INTO files` - Update file metadata - `INSERT OR REPLACE INTO files` - Update file metadata
- `SELECT * FROM files WHERE path = ?` - Check if file has changed - `SELECT * FROM files WHERE path = ?` - Check if file has changed
- `INSERT INTO snapshot_files` - Add file to current snapshot - `INSERT INTO snapshot_files` - Add file to current snapshot
- `INSERT OR IGNORE INTO snapshot_mount_points` - Record each directory on a different filesystem from its parent that the walk found an entry in
2. **Chunking** (for changed files) 2. **Chunking** (for changed files)
- `INSERT OR IGNORE INTO chunks` - Store new chunks - `INSERT OR IGNORE INTO chunks` - Store new chunks
@@ -196,7 +185,7 @@ Tracks blob upload metrics.
- `UPDATE blobs SET blob_hash = ?, finished_ts = ?` - Finalize blob after packing - `UPDATE blobs SET blob_hash = ?, finished_ts = ?` - Finalize blob after packing
4. **Upload** 4. **Upload**
- `UPDATE blobs SET uploaded_ts = ?, compressed_size = ?` - Mark blob as uploaded, with the size of the object in storage (the upload is skipped when the blob is already stored) - `UPDATE blobs SET uploaded_ts = ?, compressed_size = ?` - Mark blob as uploaded, with the size of the object in storage (a blob already stored is uploaded again, so that a prune does not take it for an unreferenced blob older than `prune_grace_period`)
- `INSERT INTO uploads` - Record upload metrics with snapshot_id - `INSERT INTO uploads` - Record upload metrics with snapshot_id
- `INSERT INTO snapshot_blobs` - Associate blob with snapshot - `INSERT INTO snapshot_blobs` - Associate blob with snapshot
@@ -241,10 +230,6 @@ The restore process doesn't use the local database. Instead:
- Query blobs not referenced by any remaining snapshot - Query blobs not referenced by any remaining snapshot
- Delete from S3, once older than `prune_grace_period`, and from the local - Delete from S3, once older than `prune_grace_period`, and from the local
database database
2. **Delete Metadata Directories With No Manifest**
- Delete each directory under `metadata/` that has no `manifest.json.zst`
and no local snapshot record, once every object in it is older than
`prune_grace_period`
### 6. Incomplete Snapshot Cleanup ### 6. Incomplete Snapshot Cleanup
+4 -6
View File
@@ -27,9 +27,7 @@ The metadata subdirectory is named with the **remote key**, a one-way hash of th
## Marker File (`vaultik-destination.txt`) ## Marker File (`vaultik-destination.txt`)
Every backup writes this file at the root before it uploads anything. It holds one fixed line of plain text and names no host, path or time. While the local index holds snapshots stored there, a destination that answers a look-up of the marker with not found is treated as not present: `snapshot create` refuses to write to it, and `prune`, `snapshot purge`, `snapshot remove`, `snapshot list` and `remote nuke` behave as for a destination that cannot be listed. An unplugged volume leaves an empty directory at its mount point, and without the marker that directory would read as a destination whose snapshots are all gone. `remote nuke` deletes the marker last. Every backup writes this file at the root before it uploads anything. It holds one fixed line of plain text and names no host, path or time. A destination without it is treated as not present while the local index holds snapshots stored there: `snapshot create` refuses to write to it, and `prune`, `snapshot purge`, `snapshot remove`, `snapshot list` and `remote nuke` behave as for a destination that cannot be listed. An unplugged volume leaves an empty directory at its mount point, and without the marker that directory would read as a destination whose snapshots are all gone. `remote nuke` deletes the marker last.
When the look-up of the marker fails for any reason other than not found, the check is skipped and the command goes on. S3 refuses it on every run to credentials that allow only PUT, so a host with such credentials does not notice a destination emptied since its last backup, and its next backup would leave out the data its local index records as already stored. After the destination is emptied, run `vaultik database delete` on such a host before it backs up again.
## Blobs Directory (`blobs/`) ## Blobs Directory (`blobs/`)
@@ -152,7 +150,7 @@ These are known, deliberate properties of the format and the tooling, recorded s
3. **Chunking uses no secret.** The FastCDC parameters are fixed and public. The default 10 MB average yields chunks between 2.5 MB and 40 MB, and any file of 2.5 MB or less is a single chunk. At the default 10 GB `blob_size_limit` a blob holds hundreds of chunks, so individual chunk lengths are not visible in the blob's size; lowering the limit toward the chunk size begins to expose them. 3. **Chunking uses no secret.** The FastCDC parameters are fixed and public. The default 10 MB average yields chunks between 2.5 MB and 40 MB, and any file of 2.5 MB or less is a single chunk. At the default 10 GB `blob_size_limit` a blob holds hundreds of chunks, so individual chunk lengths are not visible in the blob's size; lowering the limit toward the chunk size begins to expose them.
4. **Decrypted data on local disk.** Several commands stage plaintext under `$TMPDIR`: `snapshot restore` writes decrypted blobs under `vaultik-blobcache-*/` (no size cap) and the decrypted metadata database at `vaultik-restore-*/snapshot.db`; `verify --deep` writes that database at `vaultik-verify-*/snapshot.db`; `snapshot create` keeps a plaintext copy of the index at `vaultik-snapshot-*/snapshot.db`. These files are created `0600` and removed on success, but a `kill -9` or a power loss leaves them behind — delete any leftover `vaultik-*` directory under `$TMPDIR` by hand. `$TMPDIR` should be trusted to the same degree as the restore target. 4. **Decrypted data on local disk.** Several commands stage plaintext under `$TMPDIR`: `snapshot restore` writes decrypted blobs under `vaultik-blobcache-*/` (no size cap) and the decrypted metadata database at `vaultik-restore-*/snapshot.db`; `verify --deep` writes that database at `vaultik-verify-*/snapshot.db`; `snapshot create` keeps a plaintext copy of the index at `vaultik-snapshot-*/snapshot.db`. These files are created `0600` and removed on success, but a `kill -9` or a power loss leaves them behind — delete any leftover `vaultik-*` directory under `$TMPDIR` by hand. `$TMPDIR` should be trusted to the same degree as the restore target.
5. **Store permissions differ per command.** The backed-up host needs only PUT to run `snapshot create`: it writes blobs and metadata and neither reads nor deletes them. Other commands need more — `snapshot verify`, `snapshot restore`, and `prune` list and read; `prune`, `snapshot purge`, `snapshot remove`, and `remote nuke` also delete. The recommended cron line uses `--prune`, which runs `prune` on the backed-up host, so granting that host `--prune` gives it credentials that can delete its own backups. To keep the source host to PUT only, prune from a separate trusted host instead. 5. **Store permissions differ per command.** The backed-up host needs only PUT to run `snapshot create`: it writes blobs and metadata and neither reads nor deletes them. Other commands need more — `snapshot verify`, `snapshot restore`, and `prune` list and read; `prune`, `snapshot purge`, `snapshot remove`, and `remote nuke` also delete. The recommended cron line uses `--prune`, which runs `prune` on the backed-up host, so granting that host `--prune` gives it credentials that can delete its own backups. To keep the source host to PUT only, prune from a separate trusted host instead.
6. **Changing recipients does not re-encrypt existing data.** Deduplicated chunks and same-named blobs already on the destination stay encrypted to the recipients in force when they were written. A new snapshot that reuses them cannot be restored with a newly added recipient's key alone, because those reused objects were never encrypted to it. To make everything readable by a new key, run `vaultik database delete` and take a full backup to a fresh destination or prefix. 6. **Changing recipients does not re-encrypt existing data.** Deduplicated chunks already on the destination stay encrypted to the recipients in force when they were written. A new snapshot that reuses them cannot be restored with a newly added recipient's key alone, because those reused objects were never encrypted to it. To make everything readable by a new key, run `vaultik database delete` and take a full backup to a fresh destination or prefix.
7. **X25519 recipients only.** vaultik rejects age ssh and plugin recipients. Long-lived ciphertext held by a third party (the destination operator) has no fallback if X25519 is ever broken: there is no second recipient type and no post-quantum option. 7. **X25519 recipients only.** vaultik rejects age ssh and plugin recipients. Long-lived ciphertext held by a third party (the destination operator) has no fallback if X25519 is ever broken: there is no second recipient type and no post-quantum option.
## Consistency Guarantees ## Consistency Guarantees
@@ -165,10 +163,10 @@ These are known, deliberate properties of the format and the tooling, recorded s
## Pruning Safety ## Pruning Safety
The prune operation is safe because: The prune operation is safe because:
1. It keeps every blob listed in any snapshot's manifest and deletes only blobs that no manifest references. Under `metadata/` it deletes only a directory that has no `manifest.json.zst`, whose name is a remote key, and that no local snapshot record claims 1. It keeps every blob listed in any snapshot's manifest and deletes only blobs that no manifest references
2. Manifests are unencrypted and can be read without keys 2. Manifests are unencrypted and can be read without keys
3. If any manifest cannot be downloaded or decoded, prune deletes nothing and exits with an error, rather than treating that snapshot's blobs as unreferenced 3. If any manifest cannot be downloaded or decoded, prune deletes nothing and exits with an error, rather than treating that snapshot's blobs as unreferenced
4. It deletes an unreferenced blob only once the blob is older than `prune_grace_period` (default `24h`), and a directory under `metadata/` with no manifest only once every object in it is. A backup uploads its blobs and its `db.zst.age` before its manifest, so a prune running meanwhile, on this host or another sharing the destination, leaves them as long as no backup run takes longer than the grace period. The grace period does not protect a blob a backup reuses from an earlier run that stopped before its manifest: the backup skips the upload of a blob already on the destination, so that blob keeps the time of its first upload, and a prune before the backup's manifest is written can delete it; the new snapshot then cannot be restored. `remote nuke` has no such check 4. It deletes an unreferenced blob only once the blob is older than `prune_grace_period` (default `24h`). A backup uploads its blobs before the manifest that lists them, so a prune running meanwhile, on this host or another sharing the destination, leaves them as long as no backup run takes longer than the grace period. `remote nuke` has no such check
## Restoration Requirements ## Restoration Requirements
+4 -9
View File
@@ -239,15 +239,10 @@ storage_url: ""
# Default: 3 # Default: 3
# compression_level: 3 # compression_level: 3
# How old a blob that no snapshot lists must be before prune deletes it, # How old a blob that no snapshot lists must be before prune deletes it.
# and how old every object in a directory under metadata/ with no manifest # A backup uploads its blobs before the manifest that lists them, so on a
# must be before prune deletes that directory. A backup uploads its blobs # destination shared between hosts or users this must be longer than the
# and its db.zst.age before its manifest, so on a destination shared # longest backup run. Accepts: 36h, 90m, etc. 0s deletes them at once.
# between hosts or users this must be longer than the longest backup run.
# It does not protect a blob a backup reuses from an earlier run that
# stopped before its manifest: that blob keeps the time of its first
# upload, so a prune can delete it.
# Accepts: 36h, 90m, etc. 0s deletes such blobs and directories at once.
# Default: 24h # Default: 24h
# prune_grace_period: 24h # prune_grace_period: 24h
+3 -3
View File
@@ -48,9 +48,9 @@ This will erase all local tracking of:
- The storage-binding record - The storage-binding record
The remote storage is NOT affected. After deletion, the next backup The remote storage is NOT affected. After deletion, the next backup
will perform a full scan and re-deduplicate against existing remote will perform a full scan and upload all the data again, and the local
blobs, and the local index will re-bind to the currently configured index will re-bind to the currently configured storage destination on
storage destination on that run. that run.
Use --force to skip the confirmation prompt.`, Use --force to skip the confirmation prompt.`,
Args: cobra.NoArgs, Args: cobra.NoArgs,
+1 -1
View File
@@ -134,7 +134,7 @@ func writeHermeticPruneConfig(t *testing.T, seedStale bool) string {
seedStaleSnapshotRecord(t, indexPath) seedStaleSnapshotRecord(t, indexPath)
// While the local index holds a snapshot, prune fails on a // While the local index holds a snapshot, prune fails on a
// destination that answers that it has no marker. // destination without the marker a backup writes.
require.NoError(t, os.WriteFile( require.NoError(t, os.WriteFile(
filepath.Join(storeDir, "vaultik-destination.txt"), nil, 0o600)) filepath.Join(storeDir, "vaultik-destination.txt"), nil, 0o600))
} }
@@ -1,7 +1,6 @@
package cli //nolint:testpackage // shares runEntry and the hermetic config helpers package cli //nolint:testpackage // shares runEntry and the hermetic config helpers
import ( import (
"context"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
@@ -9,7 +8,6 @@ import (
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/database"
) )
// Flags of `snapshot create` used to build argument vectors below. // Flags of `snapshot create` used to build argument vectors below.
@@ -89,81 +87,6 @@ func TestEntrySnapshotCreateReportsPostBackupPruneFailure(t *testing.T) {
"the new snapshot is on the store beside the planted one") "the new snapshot is on the store beside the planted one")
} }
// TestEntrySnapshotCreatePruneExitsNonZeroWhenSourceIsEmpty backs up,
// empties the source directory, as a volume that is not mounted leaves
// its mount point, and runs `snapshot create --prune`. The command exits
// non-zero and reports the backup as completed, not as a failed snapshot
// creation. See https://git.eeqj.de/sneak/vaultik/issues/388.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotCreatePruneExitsNonZeroWhenSourceIsEmpty(t *testing.T) {
configPath, storeDir := writeBackupConfig(t)
code, _, _ := runEntry(t, flagConfig, configPath, cmdSnapshot, cmdCreate)
require.Equal(t, 0, code)
// writeBackupConfig puts the source directory beside the config file.
require.NoError(t, os.Remove(
filepath.Join(filepath.Dir(configPath), "source", "file.txt")))
code, _, stderr := runEntry(t,
flagConfig, configPath, cmdSnapshot, cmdCreate, flagPrune)
assert.Equal(t, 1, code)
assert.Contains(t, stderr,
"Backup completed, but 1 source path(s) were empty")
assert.NotContains(t, stderr, "Snapshot creation failed")
snapshots, err := os.ReadDir(filepath.Join(storeDir, "metadata"))
require.NoError(t, err)
assert.Len(t, snapshots, 2, "the older snapshot is kept")
}
// TestEntrySnapshotCreatePruneExitsNonZeroWhenMountPointIsMissing backs
// up, records in the local index that this backup found the source
// directory as a mount point, and runs `snapshot create --prune`, which
// does not find it as one. The command exits non-zero, reports the
// backup as completed, not as a failed snapshot creation, and keeps the
// older snapshot. See https://git.eeqj.de/sneak/vaultik/issues/395.
//
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
func TestEntrySnapshotCreatePruneExitsNonZeroWhenMountPointIsMissing(
t *testing.T,
) {
configPath, storeDir := writeBackupConfig(t)
code, _, _ := runEntry(t, flagConfig, configPath, cmdSnapshot, cmdCreate)
require.Equal(t, 0, code)
// writeBackupConfig puts the source directory and the local index
// beside the config file.
dir := filepath.Dir(configPath)
ctx := context.Background()
db, err := database.New(ctx, filepath.Join(dir, "index.sqlite"))
require.NoError(t, err)
repos := database.NewRepositories(db)
backedUp, err := repos.Snapshots.GetOnlySnapshot(ctx)
require.NoError(t, err)
require.NoError(t, repos.Snapshots.AddMountPoint(ctx, nil,
backedUp.ID.String(), filepath.Join(dir, "source")))
require.NoError(t, db.Close())
code, _, stderr := runEntry(t,
flagConfig, configPath, cmdSnapshot, cmdCreate, flagPrune)
assert.Equal(t, 1, code)
assert.Contains(t, stderr, "Backup completed, but 1 mount point(s) "+
"an earlier snapshot recorded were not found")
assert.NotContains(t, stderr, "Snapshot creation failed")
snapshots, err := os.ReadDir(filepath.Join(storeDir, "metadata"))
require.NoError(t, err)
assert.Len(t, snapshots, 2, "the older snapshot is kept")
}
// writeBackupConfig builds a hermetic config with one small file in its // writeBackupConfig builds a hermetic config with one small file in its
// source directory and an empty destination store directory. Returns the // source directory and an empty destination store directory. Returns the
// config path and the store directory. // config path and the store directory.
+1 -1
View File
@@ -42,7 +42,7 @@ func TestEntrySnapshotPurgeKeepsMetadataWhenLocalIndexDeleteFails(t *testing.T)
seedStaleSnapshotRecord(t, indexPath) seedStaleSnapshotRecord(t, indexPath)
// While the local index holds a snapshot, purge fails on a // While the local index holds a snapshot, purge fails on a
// destination that answers that it has no marker. // destination without the marker a backup writes.
require.NoError(t, os.WriteFile( require.NoError(t, os.WriteFile(
filepath.Join(dir, "store", "vaultik-destination.txt"), nil, 0o600)) filepath.Join(dir, "store", "vaultik-destination.txt"), nil, 0o600))
+3 -10
View File
@@ -13,9 +13,8 @@ func NewPruneCommand() *cobra.Command {
cmd := &cobra.Command{ cmd := &cobra.Command{
Use: "prune", Use: "prune",
Short: "Tidy local database and remote storage", Short: "Tidy local database and remote storage",
Long: `Removes orphaned data from the local index database, and Long: `Removes orphaned data from both the local index database and
unreferenced blobs and directories under metadata/ with no manifest unreferenced blobs from the backup destination store.
from the backup destination store.
Local cleanup drops incomplete snapshots and any files, chunks, or Local cleanup drops incomplete snapshots and any files, chunks, or
blobs no longer referenced by a completed snapshot. Remote cleanup blobs no longer referenced by a completed snapshot. Remote cleanup
@@ -24,13 +23,7 @@ set of still-referenced blob hashes, and deletes any blob not in that
set that is older than prune_grace_period (default 24h). A newer one set that is older than prune_grace_period (default 24h). A newer one
is kept: it may belong to a backup still running, on this host or is kept: it may belong to a backup still running, on this host or
another sharing the destination store, that has not yet uploaded its another sharing the destination store, that has not yet uploaded its
manifest. The grace period does not protect a blob such a backup manifest.
reuses from an earlier run that stopped before its manifest: that
blob keeps the time of its first upload, so it can be deleted.
Remote cleanup also deletes each directory under metadata/ with no
manifest, as a backup that stopped before its manifest leaves, once
all its objects are older than prune_grace_period and its snapshot is
not in the local index.
Snapshot create --prune runs the same cleanup automatically; this Snapshot create --prune runs the same cleanup automatically; this
command is the manual entry point for the same work (e.g. after a command is the manual entry point for the same work (e.g. after a
+5 -11
View File
@@ -115,10 +115,8 @@ on Linux), then /etc/vaultik/config.yml.`,
return return
} }
if errors.Is(err, vaultik.ErrFilesUnreadable) || if errors.Is(err, vaultik.ErrFilesUnreadable) {
errors.Is(err, vaultik.ErrSourcePathEmpty) || log.Error("Backup completed with unreadable files", "error", err)
errors.Is(err, vaultik.ErrVolumeNotMounted) {
log.Error("Backup completed with an error", "error", err)
ReportErrorf("Backup completed, but %v", err) ReportErrorf("Backup completed, but %v", err)
return return
@@ -133,13 +131,9 @@ on Linux), then /etc/vaultik/config.yml.`,
cmd.Flags().BoolVar(&opts.Cron, "cron", false, cmd.Flags().BoolVar(&opts.Cron, "cron", false,
"Run in cron mode (silent unless warning or error)") "Run in cron mode (silent unless warning or error)")
cmd.Flags().BoolVar(&opts.Prune, "prune", false, cmd.Flags().BoolVar(&opts.Prune, "prune", false,
"After backup, drop older snapshots of the same name, remove "+ "After backup, drop older snapshots of the same name and remove "+
"orphaned blobs older than prune_grace_period, and delete each "+ "orphaned blobs older than prune_grace_period (not after a "+
"directory under metadata/ with no manifest once all its "+ "backup that skipped unreadable files)")
"objects are older than prune_grace_period and its snapshot is "+
"not in the local index (not after a backup that skipped "+
"unreadable files, found a configured path empty, or did not "+
"find a mount point an earlier snapshot found)")
cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "", cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "",
"With --prune: keep snapshots newer than this duration "+ "With --prune: keep snapshots newer than this duration "+
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+ "(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+
+4 -6
View File
@@ -192,12 +192,10 @@ type Config struct {
CompressionLevel int `yaml:"compression_level"` CompressionLevel int `yaml:"compression_level"`
// PruneGracePeriod is how old a blob no manifest lists must be before // PruneGracePeriod is how old a blob no manifest lists must be before
// a prune deletes it, and how old every object in a directory under // a prune deletes it. A backup lists its blobs in a manifest only
// metadata/ with no manifest must be before a prune deletes that // once it has uploaded them all, so a younger one may belong to a
// directory. A backup writes its manifest only once it has uploaded // backup still running, on this host or another sharing the
// all its blobs and its db.zst.age, so a younger blob or directory may // destination store.
// belong to a backup still running, on this host or another sharing
// the destination store.
PruneGracePeriod time.Duration `yaml:"prune_grace_period"` PruneGracePeriod time.Duration `yaml:"prune_grace_period"`
// StorageURL specifies the storage backend using a URL format. // StorageURL specifies the storage backend using a URL format.
+1 -32
View File
@@ -6,7 +6,6 @@ import (
"path/filepath" "path/filepath"
"strings" "strings"
"testing" "testing"
"time"
"sneak.berlin/go/vaultik/internal/chunker" "sneak.berlin/go/vaultik/internal/chunker"
"sneak.berlin/go/vaultik/internal/log" "sneak.berlin/go/vaultik/internal/log"
@@ -415,8 +414,7 @@ func TestDocumentedSizeExamples(t *testing.T) {
// TestValidatePruneGracePeriod checks that a negative prune_grace_period // TestValidatePruneGracePeriod checks that a negative prune_grace_period
// is refused rather than acting like 0s, which deletes unreferenced blobs // is refused rather than acting like 0s, which deletes unreferenced blobs
// and directories under metadata/ with no manifest at once, and that 0s // at once, and that 0s and the documented examples are accepted.
// and the documented examples are accepted.
func TestValidatePruneGracePeriod(t *testing.T) { func TestValidatePruneGracePeriod(t *testing.T) {
t.Parallel() t.Parallel()
@@ -452,35 +450,6 @@ func TestValidatePruneGracePeriod(t *testing.T) {
} }
} }
// TestLoadPruneGracePeriod checks that a config file without
// prune_grace_period loads with the documented 24h. The value is written out
// here rather than taken from defaultPruneGracePeriod, so that a change of the
// default fails this test.
func TestLoadPruneGracePeriod(t *testing.T) {
t.Parallel()
const withoutGracePeriod = "snapshots:\n" +
" test:\n" +
" paths: [/tmp/vaultik-test-source]\n" +
"storage_url: file:///tmp/vaultik-test-storage\n"
path := filepath.Join(t.TempDir(), "config.yml")
err := os.WriteFile(path, []byte(withoutGracePeriod), 0o600)
if err != nil {
t.Fatalf("write config: %v", err)
}
cfg, err := Load(path)
if err != nil {
t.Fatalf("Load() unexpected error: %v", err)
}
if cfg.PruneGracePeriod != 24*time.Hour {
t.Errorf("prune_grace_period = %s, want 24h", cfg.PruneGracePeriod)
}
}
// TestLoadRejectsRelativeFileURL checks that a file:// storage_url with a // TestLoadRejectsRelativeFileURL checks that a file:// storage_url with a
// relative path fails at load. Accepted, it would resolve against the // relative path fails at load. Accepted, it would resolve against the
// working directory, so a backup run from another directory would write // working directory, so a backup run from another directory would write
+1 -3
View File
@@ -155,9 +155,7 @@ func (r *BlobRepository) UpdateFinished(
} }
// UpdateUploaded marks a blob as uploaded and records the size of the // UpdateUploaded marks a blob as uploaded and records the size of the
// stored object as its compressed size. That size can differ from the // stored object as its compressed size.
// local copy's when an identical blob was already stored with another
// compression level or other recipients.
func (r *BlobRepository) UpdateUploaded( func (r *BlobRepository) UpdateUploaded(
ctx context.Context, tx *sql.Tx, id string, storedSize int64, ctx context.Context, tx *sql.Tx, id string, storedSize int64,
) error { ) error {
-10
View File
@@ -123,16 +123,6 @@ CREATE TABLE IF NOT EXISTS snapshot_blobs (
-- Index for efficient blob lookups (used in orphan detection) -- Index for efficient blob lookups (used in orphan detection)
CREATE INDEX IF NOT EXISTS idx_snapshot_blobs_blob_id ON snapshot_blobs(blob_id); CREATE INDEX IF NOT EXISTS idx_snapshot_blobs_blob_id ON snapshot_blobs(blob_id);
-- Snapshot mount points table: the directories at or below a configured
-- path that the snapshot's backup found on a different filesystem from
-- their parent directory, as a mount point is, and found an entry in
CREATE TABLE IF NOT EXISTS snapshot_mount_points (
snapshot_id TEXT NOT NULL,
path TEXT NOT NULL,
PRIMARY KEY (snapshot_id, path),
FOREIGN KEY (snapshot_id) REFERENCES snapshots(id) ON DELETE CASCADE
);
-- Uploads table: tracks blob upload metrics -- Uploads table: tracks blob upload metrics
CREATE TABLE IF NOT EXISTS uploads ( CREATE TABLE IF NOT EXISTS uploads (
blob_hash TEXT PRIMARY KEY, blob_hash TEXT PRIMARY KEY,
+15 -74
View File
@@ -24,8 +24,7 @@ var (
) )
// SnapshotRepository provides access to the snapshots table and its // SnapshotRepository provides access to the snapshots table and its
// snapshot_files / snapshot_blobs association tables, and to the mount // snapshot_files / snapshot_blobs association tables.
// points each snapshot's backup found (snapshot_mount_points).
type SnapshotRepository struct { type SnapshotRepository struct {
db *DB db *DB
} }
@@ -495,63 +494,33 @@ func (r *SnapshotRepository) GetBlobHashes(
ctx context.Context, snapshotID string, ctx context.Context, snapshotID string,
) ([]string, error) { ) ([]string, error) {
query := ` query := `
SELECT sb.blob_hash SELECT sb.blob_hash
FROM snapshot_blobs sb FROM snapshot_blobs sb
WHERE sb.snapshot_id = ? WHERE sb.snapshot_id = ?
ORDER BY sb.blob_hash ORDER BY sb.blob_hash
` `
blobs, err := r.queryStrings(ctx, query, snapshotID) rows, err := r.db.conn.QueryContext(ctx, query, snapshotID)
if err != nil { if err != nil {
return nil, fmt.Errorf("querying blob hashes: %w", err) return nil, fmt.Errorf("querying blob hashes: %w", err)
} }
return blobs, nil defer func() { _ = rows.Close() }()
}
// AddMountPoint records dir as a directory that the backup of a snapshot var blobs []string
// found on a different filesystem from its parent directory, as a mount
// point is, using tx when non-nil.
func (r *SnapshotRepository) AddMountPoint(
ctx context.Context, tx *sql.Tx, snapshotID string, dir string,
) error {
query := `
INSERT OR IGNORE INTO snapshot_mount_points (snapshot_id, path)
VALUES (?, ?)
`
var err error for rows.Next() {
if tx != nil { var blobHash string
_, err = tx.ExecContext(ctx, query, snapshotID, dir)
} else { err := rows.Scan(&blobHash)
_, err = r.db.ExecWithLog(ctx, query, snapshotID, dir) if err != nil {
return nil, fmt.Errorf("scanning blob hash: %w", err)
}
blobs = append(blobs, blobHash)
} }
if err != nil { return blobs, rows.Err()
return fmt.Errorf("adding mount point to snapshot: %w", err)
}
return nil
}
// GetMountPoints returns the directories AddMountPoint recorded for a
// snapshot, sorted.
func (r *SnapshotRepository) GetMountPoints(
ctx context.Context, snapshotID string,
) ([]string, error) {
query := `
SELECT path
FROM snapshot_mount_points
WHERE snapshot_id = ?
ORDER BY path
`
dirs, err := r.queryStrings(ctx, query, snapshotID)
if err != nil {
return nil, fmt.Errorf("querying mount points: %w", err)
}
return dirs, nil
} }
// GetSnapshotTotalCompressedSize returns the total compressed size of all // GetSnapshotTotalCompressedSize returns the total compressed size of all
@@ -740,34 +709,6 @@ func (r *SnapshotRepository) DeleteSnapshotUploads(
return nil return nil
} }
// queryStrings runs query, which selects one text column, and returns
// its values in the order of the rows.
func (r *SnapshotRepository) queryStrings(
ctx context.Context, query string, args ...any,
) ([]string, error) {
rows, err := r.db.conn.QueryContext(ctx, query, args...)
if err != nil {
return nil, err
}
defer func() { _ = rows.Close() }()
var values []string
for rows.Next() {
var value string
err := rows.Scan(&value)
if err != nil {
return nil, err
}
values = append(values, value)
}
return values, rows.Err()
}
// extendedCompressionRatio computes the compression ratio for a snapshot // extendedCompressionRatio computes the compression ratio for a snapshot
// from its stored blob_size and the given uncompressed size. Returns 1.0 // from its stored blob_size and the given uncompressed size. Returns 1.0
// when the uncompressed size is zero. // when the uncompressed size is zero.
-50
View File
@@ -4,7 +4,6 @@ import (
"context" "context"
"fmt" "fmt"
"math" "math"
"slices"
"testing" "testing"
"time" "time"
@@ -423,52 +422,3 @@ func TestSnapshotRepositoryDuplicate(t *testing.T) {
t.Error("expected error for duplicate snapshot") t.Error("expected error for duplicate snapshot")
} }
} }
// Deleting a snapshot from the local index deletes the mount points its
// backup recorded, and no other snapshot's. See
// https://git.eeqj.de/sneak/vaultik/issues/395.
func TestSnapshotRepositoryDeleteRemovesMountPoints(t *testing.T) {
t.Parallel()
db, cleanup := setupTestDB(t)
defer cleanup()
ctx := context.Background()
repo := database.NewSnapshotRepository(db)
for _, id := range []types.SnapshotID{"kept", "deleted"} {
err := repo.Create(ctx, nil, &database.Snapshot{
ID: id,
Hostname: testHostname,
VaultikVersion: testVersion,
StartedAt: time.Now().Truncate(time.Second),
})
if err != nil {
t.Fatalf("failed to create snapshot: %v", err)
}
err = repo.AddMountPoint(ctx, nil, id.String(), "/data/nas")
if err != nil {
t.Fatalf("failed to add mount point: %v", err)
}
}
err := repo.Delete(ctx, "deleted")
if err != nil {
t.Fatalf("failed to delete snapshot: %v", err)
}
for id, want := range map[string][]string{
"kept": {"/data/nas"},
"deleted": nil,
} {
got, err := repo.GetMountPoints(ctx, id)
if err != nil {
t.Fatalf("failed to read mount points: %v", err)
}
if !slices.Equal(got, want) {
t.Errorf("mount points of %s: got %v, want %v", id, got, want)
}
}
}
-40
View File
@@ -1,40 +0,0 @@
//nolint:testpackage // exercises the unexported deviceNumberFromStat
package snapshot
import (
"os"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// The device number a real backup reads is the same for a directory and
// a file in it, and differs between the temp directory and /dev.
// Without it no mount point is ever found. See
// https://git.eeqj.de/sneak/vaultik/issues/395.
func TestDeviceNumberFromStat(t *testing.T) {
t.Parallel()
dir := t.TempDir()
file := filepath.Join(dir, "file.txt")
require.NoError(t, os.WriteFile(file, nil, 0o600))
deviceOf := func(path string) uint64 {
t.Helper()
info, err := os.Stat(path)
require.NoError(t, err)
device, ok := deviceNumberFromStat(info)
require.True(t, ok, "no device number read for %s", path)
return device
}
assert.Equal(t, deviceOf(dir), deviceOf(file))
// /dev is a different filesystem from the temp dir on Linux, macOS and in Docker.
assert.NotEqual(t, deviceOf(dir), deviceOf("/dev"))
}
-6
View File
@@ -1,8 +1,6 @@
package snapshot package snapshot
import ( import (
"os"
"github.com/spf13/afero" "github.com/spf13/afero"
"go.uber.org/fx" "go.uber.org/fx"
"sneak.berlin/go/vaultik/internal/config" "sneak.berlin/go/vaultik/internal/config"
@@ -21,9 +19,6 @@ type ScannerParams struct {
// PreviousSnapshotID is the most recent completed snapshot of the // PreviousSnapshotID is the most recent completed snapshot of the
// same name, "" for none. // same name, "" for none.
PreviousSnapshotID string PreviousSnapshotID string
// DeviceNumber is ScannerConfig.DeviceNumber; nil reads it from the
// file's stat.
DeviceNumber func(info os.FileInfo) (uint64, bool)
} }
// Module exports backup functionality as an fx module. // Module exports backup functionality as an fx module.
@@ -64,7 +59,6 @@ func provideScannerFactory(
Exclude: excludes, Exclude: excludes,
SkipErrors: params.SkipErrors, SkipErrors: params.SkipErrors,
PreviousSnapshotID: params.PreviousSnapshotID, PreviousSnapshotID: params.PreviousSnapshotID,
DeviceNumber: params.DeviceNumber,
}) })
} }
} }
-245
View File
@@ -1,245 +0,0 @@
package snapshot_test
import (
"context"
"os"
"path/filepath"
"slices"
"strings"
"syscall"
"testing"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/snapshot"
)
// A scan records with the snapshot each directory on a different device
// from its parent directory that the walk found an entry in, as a mount
// point is, the scanned path included. A filesystem that gives no device
// numbers, as the in-memory one, records none. See
// https://git.eeqj.de/sneak/vaultik/issues/395.
func TestScanRecordsMountPoints(t *testing.T) {
t.Parallel()
const (
scannedPath = "/data/source"
snapshotID = "mount-points"
)
tests := []struct {
name string
// onVolume names the directories on the volume's device; every
// other directory is on another one. nil leaves the device
// numbers to the in-memory filesystem, which has none.
onVolume []string
want []string
}{
{
name: "volume below the scanned path",
onVolume: []string{"nas"},
want: []string{scannedPath + "/nas"},
},
{
name: "volume at the scanned path",
onVolume: []string{"source", "nas"},
want: []string{scannedPath},
},
{name: "no device numbers", want: nil},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
ctx := context.Background()
fs := afero.NewMemMapFs()
for _, path := range []string{
scannedPath + "/notes.txt", scannedPath + "/nas/a.txt",
} {
require.NoError(t, fs.MkdirAll(filepath.Dir(path), 0o755))
require.NoError(t, afero.WriteFile(fs, path, []byte(path), 0o644))
}
db, err := database.NewTestDB()
require.NoError(t, err)
t.Cleanup(func() { _ = db.Close() })
repos := database.NewRepositories(db)
createTestSnapshotRecord(ctx, t, repos, snapshotID)
cfg := snapshot.ScannerConfig{
FS: fs,
ChunkSize: int64(1024 * 16),
Repositories: repos,
MaxBlobSize: int64(1024 * 1024),
CompressionLevel: 3,
AgeRecipients: []string{testAgePublicKey},
}
if tt.onVolume != nil {
cfg.DeviceNumber = func(info os.FileInfo) (uint64, bool) {
if slices.Contains(tt.onVolume, info.Name()) {
return 2, true
}
return 1, true
}
}
_, err = snapshot.NewScanner(cfg).Scan(ctx, scannedPath, snapshotID)
require.NoError(t, err)
got, err := repos.Snapshots.GetMountPoints(ctx, snapshotID)
require.NoError(t, err)
assert.Equal(t, tt.want, got)
})
}
}
// automountFs is an in-memory filesystem with a volume at mountPoint
// that is mounted on first access: opening mountPoint to list it mounts
// the volume or, if it cannot be mounted, fails with "no such file or
// directory", as with the autofs daemon. Stat records the device an
// entry is on at that moment, as a real stat does: unmountedDevice for
// mountPoint until the volume is mounted; the volume's, 2, for
// mountPoint once it is and for everything in it; 1 for everything else.
type automountFs struct {
afero.Fs
mountPoint string
unmountedDevice uint64
canMount bool
mounted bool
}
//nolint:ireturn // afero.Fs.Open is defined to return the interface.
func (fs *automountFs) Open(name string) (afero.File, error) {
if name == fs.mountPoint && !fs.canMount {
return nil, &os.PathError{Op: "open", Path: name, Err: syscall.ENOENT}
}
if name == fs.mountPoint {
fs.mounted = true
}
return fs.Fs.Open(name)
}
func (fs *automountFs) Stat(name string) (os.FileInfo, error) {
info, err := fs.Fs.Stat(name)
if err != nil {
return nil, err
}
device := uint64(1)
switch {
case name == fs.mountPoint && !fs.mounted:
device = fs.unmountedDevice
case name == fs.mountPoint || strings.HasPrefix(name, fs.mountPoint+"/"):
device = 2
}
return statWithDevice{FileInfo: info, device: device}, nil
}
// statWithDevice is an entry automountFs.Stat looked up, with the device
// it was on then.
type statWithDevice struct {
os.FileInfo
device uint64
}
// deviceAtLookup returns the device automountFs.Stat recorded for info.
func deviceAtLookup(info os.FileInfo) (uint64, bool) {
stat, ok := info.(statWithDevice)
return stat.device, ok
}
// A volume mounted on first access is mounted when the walk lists its
// mount point. Until then the mount point is on the automounter's own
// device (a systemd automount) or on its parent's (an autofs map that
// shows its mount points before mounting them). Either way only the
// mount point is recorded, by the backup that mounts the volume and by a
// later one that finds it mounted. Listing an automount point whose
// volume cannot be mounted fails; it is not recorded, because the walk
// finds no entry in it. See https://git.eeqj.de/sneak/vaultik/issues/395.
func TestScanRecordsOnlyTheMountPointOfAnAutomountedVolume(t *testing.T) {
t.Parallel()
const scannedPath = "/data/source"
mountPoint := scannedPath + "/nas"
tests := []struct {
name string
unmountedDevice uint64
canMount bool
want []string
}{
{
name: "on the automounter's device until listed",
unmountedDevice: 3, canMount: true, want: []string{mountPoint},
},
{
name: "on its parent's device until listed",
unmountedDevice: 1, canMount: true, want: []string{mountPoint},
},
{
name: "volume that cannot be mounted",
unmountedDevice: 1, canMount: false, want: nil,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
ctx := context.Background()
fs := &automountFs{
Fs: afero.NewMemMapFs(), mountPoint: mountPoint,
unmountedDevice: tt.unmountedDevice, canMount: tt.canMount,
}
require.NoError(t, fs.MkdirAll(mountPoint+"/photos", 0o755))
for _, path := range []string{
scannedPath + "/notes.txt", mountPoint + "/photos/a.jpg",
} {
require.NoError(t, afero.WriteFile(fs, path, []byte(path), 0o644))
}
db, err := database.NewTestDB()
require.NoError(t, err)
t.Cleanup(func() { _ = db.Close() })
repos := database.NewRepositories(db)
for _, snapshotID := range []string{"first", "later"} {
createTestSnapshotRecord(ctx, t, repos, snapshotID)
scanner := snapshot.NewScanner(snapshot.ScannerConfig{
FS: fs,
ChunkSize: int64(1024 * 16),
Repositories: repos,
MaxBlobSize: int64(1024 * 1024),
CompressionLevel: 3,
AgeRecipients: []string{testAgePublicKey},
DeviceNumber: deviceAtLookup,
})
_, err = scanner.Scan(ctx, scannedPath, snapshotID)
require.NoError(t, err)
got, err := repos.Snapshots.GetMountPoints(ctx, snapshotID)
require.NoError(t, err)
assert.Equal(t, tt.want, got, "mount points of the %s backup", snapshotID)
}
})
}
}
+43 -211
View File
@@ -73,9 +73,6 @@ type Scanner struct {
skipErrors bool skipErrors bool
// ui is the user-facing output; never nil (defaults to a discarding writer). // ui is the user-facing output; never nil (defaults to a discarding writer).
ui *ui.Writer ui *ui.Writer
// deviceNumber returns the number of the device that holds a file;
// never nil (defaults to deviceNumberFromStat).
deviceNumber func(info os.FileInfo) (uint64, bool)
// In-memory cache of known chunk hashes for fast existence checks // In-memory cache of known chunk hashes for fast existence checks
knownChunks map[string]struct{} knownChunks map[string]struct{}
@@ -139,10 +136,6 @@ type ScannerConfig struct {
// PreviousSnapshotID is the most recent completed snapshot of the // PreviousSnapshotID is the most recent completed snapshot of the
// same name, "" for none. // same name, "" for none.
PreviousSnapshotID string PreviousSnapshotID string
// DeviceNumber returns the number of the device that holds a file,
// and false when there is none to read. nil reads it from the file's
// stat; tests replace it, since the in-memory test filesystem has none.
DeviceNumber func(info os.FileInfo) (uint64, bool)
} }
// ScanResult contains the results of a scan operation. Files and bytes // ScanResult contains the results of a scan operation. Files and bytes
@@ -175,12 +168,6 @@ type ScanResult struct {
// skipped because it could not be read or listed; the rest are files // skipped because it could not be read or listed; the rest are files
// removed after the walk listed them. // removed after the walk listed them.
FilesUnreadable int FilesUnreadable int
// EntriesFound counts the entries the walk found and could look up
// that no exclude pattern matches, leaving out the scanned path
// itself when it is a directory. It is zero for an empty directory,
// such as the mount point of a volume that is not mounted.
EntriesFound int
} }
// NewScanner creates a new scanner instance // NewScanner creates a new scanner instance
@@ -221,11 +208,6 @@ func NewScanner(cfg ScannerConfig) *Scanner {
uiw = ui.NewWithColor(io.Discard, false) uiw = ui.NewWithColor(io.Discard, false)
} }
deviceNumber := cfg.DeviceNumber
if deviceNumber == nil {
deviceNumber = deviceNumberFromStat
}
return &Scanner{ return &Scanner{
fs: cfg.FS, fs: cfg.FS,
chunker: chunker.NewChunker(cfg.ChunkSize), chunker: chunker.NewChunker(cfg.ChunkSize),
@@ -241,7 +223,6 @@ func NewScanner(cfg ScannerConfig) *Scanner {
skipErrors: cfg.SkipErrors, skipErrors: cfg.SkipErrors,
previousSnapshotID: cfg.PreviousSnapshotID, previousSnapshotID: cfg.PreviousSnapshotID,
ui: uiw, ui: uiw,
deviceNumber: deviceNumber,
pendingChunkHashes: make(map[string]struct{}), pendingChunkHashes: make(map[string]struct{}),
} }
} }
@@ -310,7 +291,7 @@ func (s *Scanner) Scan(
} }
} }
err = s.recordDirectories(ctx, scanResult) err = s.recordDirectories(ctx, scanResult.Directories)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -888,11 +869,6 @@ type ScanPhaseResult struct {
FilesToProcess []*FileToProcess FilesToProcess []*FileToProcess
UnchangedFileIDs []types.FileID // IDs of unchanged files to associate with snapshot UnchangedFileIDs []types.FileID // IDs of unchanged files to associate with snapshot
Directories []*FileToProcess // Every directory found, to record Directories []*FileToProcess // Every directory found, to record
// MountPoints are the directories, the scanned path included, that
// the walk found an entry in and, once it had listed them, on a
// different device from their parent directory (see
// recordIfMountPoint).
MountPoints []string
} }
// scanPhase performs the initial directory scan to identify files to process // scanPhase performs the initial directory scan to identify files to process
@@ -912,7 +888,7 @@ func (s *Scanner) scanPhase(
// estimate leaves them out. // estimate leaves them out.
estimatedTotal := countNonDirectories(knownFiles) estimatedTotal := countNonDirectories(knownFiles)
collector := &scanCollector{dirsWithEntries: make(map[string]struct{})} collector := &scanCollector{}
// Set up periodic status output // Set up periodic status output
startTime := time.Now() startTime := time.Now()
@@ -952,13 +928,6 @@ func (s *Scanner) scanPhase(
default: default:
} }
// The scanned directory itself is not an entry found in it.
if filePath != path || !info.IsDir() {
result.EntriesFound++
}
s.recordIfMountPoint(filePath, path, collector)
// Handle symlinks and directories // Handle symlinks and directories
handled, err := s.recordSpecialEntry( handled, err := s.recordSpecialEntry(
filePath, info, existingFiles, knownFiles, collector, result) filePath, info, existingFiles, knownFiles, collector, result)
@@ -971,8 +940,18 @@ func (s *Scanner) scanPhase(
return nil return nil
} }
filesScanned, changedCount := s.recordRegularFile( // Track this file as existing (for deletion detection)
filePath, info, existingFiles, knownFiles, collector, result) existingFiles[filePath] = struct{}{}
// Check file against in-memory map (no DB query!)
file := s.buildRegularFileEntry(filePath, info)
needsProcessing := s.checkFileInMemory(file, knownFiles)
filesScanned, changedCount := collector.recordFile(
filePath, info, file, needsProcessing)
// Update result stats
s.updateScanEntryStats(result, needsProcessing, info)
// Output periodic status // Output periodic status
if time.Since(lastStatusTime) >= scanStatusInterval { if time.Since(lastStatusTime) >= scanStatusInterval {
@@ -993,7 +972,6 @@ func (s *Scanner) scanPhase(
FilesToProcess: collector.filesToProcess, FilesToProcess: collector.filesToProcess,
UnchangedFileIDs: collector.unchangedFileIDs, UnchangedFileIDs: collector.unchangedFileIDs,
Directories: collector.directories, Directories: collector.directories,
MountPoints: collector.mountPoints,
}, nil }, nil
} }
@@ -1049,30 +1027,6 @@ func (s *Scanner) recordSpecialEntry(
return false, nil return false, nil
} }
// recordRegularFile records a regular file as recordSpecialEntry does a
// symlink and returns the running scanned and changed counts.
func (s *Scanner) recordRegularFile(
filePath string, info os.FileInfo,
existingFiles map[string]struct{},
knownFiles map[string]*database.File,
collector *scanCollector, result *ScanResult,
) (int64, int) {
// Track this file as existing (for deletion detection)
existingFiles[filePath] = struct{}{}
// Check file against in-memory map (no DB query!)
file := s.buildRegularFileEntry(filePath, info)
needsProcessing := s.checkFileInMemory(file, knownFiles)
filesScanned, changedCount := collector.recordFile(
filePath, info, file, needsProcessing)
// Update result stats
s.updateScanEntryStats(result, needsProcessing, info)
return filesScanned, changedCount
}
// countRemovedDuringWalk counts an entry that was in its directory's // countRemovedDuringWalk counts an entry that was in its directory's
// listing but was gone when the walk looked it up, like a file removed // listing but was gone when the walk looked it up, like a file removed
// before phase 2 reads it: as examined and failed, and not as an error, // before phase 2 reads it: as examined and failed, and not as an error,
@@ -1120,29 +1074,6 @@ type scanCollector struct {
unchangedFileIDs []types.FileID // Just IDs - no new records needed unchangedFileIDs []types.FileID // Just IDs - no new records needed
directories []*FileToProcess directories []*FileToProcess
filesScanned int64 filesScanned int64
// dirsWithEntries holds each directory the walk has found an entry in.
dirsWithEntries map[string]struct{}
mountPoints []string
}
// firstEntryIn notes that the walk found an entry in directory dir and
// reports whether it is the first one.
func (c *scanCollector) firstEntryIn(dir string) bool {
c.mu.Lock()
defer c.mu.Unlock()
_, seen := c.dirsWithEntries[dir]
c.dirsWithEntries[dir] = struct{}{}
return !seen
}
// addMountPoint sets a mount point aside to be recorded.
func (c *scanCollector) addMountPoint(dir string) {
c.mu.Lock()
defer c.mu.Unlock()
c.mountPoints = append(c.mountPoints, dir)
} }
// addDirectory sets a directory aside to be recorded. It is not counted // addDirectory sets a directory aside to be recorded. It is not counted
@@ -1334,100 +1265,20 @@ func (s *Scanner) buildDirectoryEntry(path string, info os.FileInfo) *database.F
} }
} }
// recordDirectories records every directory the walk found, and its // recordDirectories records every directory the walk found. A directory
// mount points (see recordMountPoints). A directory has no data to back // has no data to back up, so it is not among the files counted and
// up, so it is not among the files counted and backed up. // backed up.
func (s *Scanner) recordDirectories( func (s *Scanner) recordDirectories(
ctx context.Context, scanned *ScanPhaseResult, ctx context.Context, directories []*FileToProcess,
) error { ) error {
for _, dir := range scanned.Directories { for _, dir := range directories {
err := s.recordNonRegularFile(ctx, dir) err := s.recordNonRegularFile(ctx, dir)
if err != nil { if err != nil {
return fmt.Errorf("recording directory %s: %w", dir.Path, err) return fmt.Errorf("recording directory %s: %w", dir.Path, err)
} }
} }
return s.recordMountPoints(ctx, scanned.MountPoints) return nil
}
// recordMountPoints records with the snapshot the mount points the walk
// found (see ScanPhaseResult.MountPoints).
func (s *Scanner) recordMountPoints(
ctx context.Context, mountPoints []string,
) error {
return s.repos.WithTx(ctx, func(txCtx context.Context, tx *sql.Tx) error {
for _, dir := range mountPoints {
log.Info("Found a mount point", "path", dir)
err := s.repos.Snapshots.AddMountPoint(txCtx, tx, s.snapshotID, dir)
if err != nil {
return fmt.Errorf("recording mount point %s: %w", dir, err)
}
}
return nil
})
}
// recordIfMountPoint sets the directory that holds entry, a path the
// walk found, aside to be recorded as a mount point if it is on a
// different device from its parent directory. The walk calls it for each
// entry it finds, and it decides on the first entry in a directory, so a
// directory the walk found no entry in is never recorded. By then the
// walk has listed the directory, which mounts a volume mounted there on
// first access, so the directory and its parent are looked up again:
// before the listing, the directory can be on the automounter's own
// device (a systemd automount) or on its parent's (an autofs map that
// shows its mount points before mounting them). The scanned path is in
// no listing the walk made, so it is skipped as an entry. A directory
// that cannot be looked up then, or whose parent cannot, is not
// recorded, so this snapshot does not stop the prune of a later backup
// that misses its volume.
func (s *Scanner) recordIfMountPoint(
entry, scannedPath string, collector *scanCollector,
) {
dir := filepath.Dir(entry)
if entry == scannedPath || !collector.firstEntryIn(dir) {
return
}
dirDevice, ok := s.deviceNumberAt(dir)
if !ok {
return
}
parentDevice, ok := s.deviceNumberAt(filepath.Dir(dir))
if ok && dirDevice != parentDevice {
collector.addMountPoint(dir)
}
}
// deviceNumberAt looks path up now and returns the number of the device
// that holds it, and false when the lookup fails (logged) or gives no
// device number.
func (s *Scanner) deviceNumberAt(path string) (uint64, bool) {
info, err := s.fs.Stat(path)
if err != nil {
log.Warn("Could not look up a directory to check for a mount point",
"path", path, "error", err)
return 0, false
}
return s.deviceNumber(info)
}
// deviceNumberFromStat returns the number of the device that holds the
// file info describes, and false when info carries no stat, as from the
// in-memory test filesystem.
func deviceNumberFromStat(info os.FileInfo) (uint64, bool) {
stat, ok := info.Sys().(*syscall.Stat_t)
if !ok {
return 0, false
}
return uint64(stat.Dev), true //nolint:unconvert // Dev is an int32 on darwin
} }
// recordNonRegularFile writes a symlink or directory entry to the database // recordNonRegularFile writes a symlink or directory entry to the database
@@ -1790,8 +1641,7 @@ func (s *Scanner) handleBlobReady(
blobPath := fmt.Sprintf("blobs/%s/%s/%s", blobPath := fmt.Sprintf("blobs/%s/%s/%s",
finishedBlob.Hash[:2], finishedBlob.Hash[2:4], finishedBlob.Hash) finishedBlob.Hash[:2], finishedBlob.Hash[2:4], finishedBlob.Hash)
storedSize, blobExists, err := s.uploadBlobIfNeeded( err := s.uploadBlob(ctx, blobPath, blobWithReader, startTime, result)
ctx, blobPath, blobWithReader, startTime, result)
if err != nil { if err != nil {
cleanupErr := s.cleanupBlobTempFile(blobWithReader) cleanupErr := s.cleanupBlobTempFile(blobWithReader)
@@ -1799,8 +1649,7 @@ func (s *Scanner) handleBlobReady(
fmt.Errorf("uploading blob %s: %w", finishedBlob.Hash, err), cleanupErr) fmt.Errorf("uploading blob %s: %w", finishedBlob.Hash, err), cleanupErr)
} }
err = s.recordBlobMetadata( err = s.recordBlobMetadata(ctx, finishedBlob, startTime)
ctx, finishedBlob, storedSize, blobExists, startTime)
if err != nil { if err != nil {
cleanupErr := s.cleanupBlobTempFile(blobWithReader) cleanupErr := s.cleanupBlobTempFile(blobWithReader)
@@ -1824,46 +1673,33 @@ func (s *Scanner) handleBlobReady(
return nil return nil
} }
// uploadBlobIfNeeded uploads the blob to storage if it doesn't already // uploadBlob uploads the blob to storage. A blob already there, left by
// exist. It returns the size of the object in storage, which for an // an earlier run stopped before its manifest, is uploaded again rather
// existing object can differ from the local copy's, and whether it // than skipped: keeping its old time would let a prune elsewhere delete
// already existed. // it as an unreferenced blob older than prune_grace_period before this
func (s *Scanner) uploadBlobIfNeeded( // backup's manifest lists it.
func (s *Scanner) uploadBlob(
ctx context.Context, ctx context.Context,
blobPath string, blobPath string,
blobWithReader *blob.WithReader, blobWithReader *blob.WithReader,
startTime time.Time, startTime time.Time,
result *ScanResult, result *ScanResult,
) (int64, bool, error) { ) error {
finishedBlob := blobWithReader.FinishedBlob finishedBlob := blobWithReader.FinishedBlob
// Check if blob already exists (deduplication after restart)
destination := s.storage.Info().Location destination := s.storage.Info().Location
stored, err := s.storage.Stat(ctx, blobPath)
if err == nil {
log.Info("Blob already exists in storage, skipping upload",
"hash", finishedBlob.Hash,
"size", humanize.Bytes(safeUint64(finishedBlob.Compressed)))
s.ui.Infof("Blob %s (%s) already exists at %s. Skipping upload.",
s.ui.Hex(finishedBlob.Hash), s.ui.Size(finishedBlob.Compressed),
s.ui.Path(destination))
return stored.Size, true, nil
}
s.ui.Beginf("Uploading blob %s (%s) to %s.", s.ui.Beginf("Uploading blob %s (%s) to %s.",
s.ui.Hex(finishedBlob.Hash), s.ui.Size(finishedBlob.Compressed), s.ui.Hex(finishedBlob.Hash), s.ui.Size(finishedBlob.Compressed),
s.ui.Path(destination)) s.ui.Path(destination))
progressCallback := s.makeUploadProgressCallback(ctx, finishedBlob, startTime) progressCallback := s.makeUploadProgressCallback(ctx, finishedBlob, startTime)
err = s.storage.PutWithProgress(ctx, blobPath, blobWithReader.Reader, err := s.storage.PutWithProgress(ctx, blobPath, blobWithReader.Reader,
finishedBlob.Compressed, progressCallback) finishedBlob.Compressed, progressCallback)
if err != nil { if err != nil {
log.Error("Failed to upload blob", "hash", finishedBlob.Hash, "error", err) log.Error("Failed to upload blob", "hash", finishedBlob.Hash, "error", err)
return 0, false, fmt.Errorf("uploading blob to storage: %w", err) return fmt.Errorf("uploading blob to storage: %w", err)
} }
uploadDuration := time.Since(startTime) uploadDuration := time.Since(startTime)
@@ -1894,7 +1730,7 @@ func (s *Scanner) uploadBlobIfNeeded(
stats.BytesUploaded.Add(finishedBlob.Compressed) stats.BytesUploaded.Add(finishedBlob.Compressed)
} }
return finishedBlob.Compressed, false, nil return nil
} }
// makeUploadProgressCallback creates a progress callback for blob uploads. // makeUploadProgressCallback creates a progress callback for blob uploads.
@@ -1964,8 +1800,6 @@ func (s *Scanner) makeUploadProgressCallback(
func (s *Scanner) recordBlobMetadata( func (s *Scanner) recordBlobMetadata(
ctx context.Context, ctx context.Context,
finishedBlob *blob.FinishedBlob, finishedBlob *blob.FinishedBlob,
storedSize int64,
blobExists bool,
startTime time.Time, startTime time.Time,
) error { ) error {
finishedBlobID, err := types.ParseBlobID(finishedBlob.ID) finishedBlobID, err := types.ParseBlobID(finishedBlob.ID)
@@ -1977,7 +1811,7 @@ func (s *Scanner) recordBlobMetadata(
return s.repos.WithTx(ctx, func(txCtx context.Context, tx *sql.Tx) error { return s.repos.WithTx(ctx, func(txCtx context.Context, tx *sql.Tx) error {
err := s.repos.Blobs.UpdateUploaded( err := s.repos.Blobs.UpdateUploaded(
txCtx, tx, finishedBlob.ID, storedSize) txCtx, tx, finishedBlob.ID, finishedBlob.Compressed)
if err != nil { if err != nil {
return fmt.Errorf("updating blob upload timestamp: %w", err) return fmt.Errorf("updating blob upload timestamp: %w", err)
} }
@@ -1988,19 +1822,17 @@ func (s *Scanner) recordBlobMetadata(
return fmt.Errorf("adding blob to snapshot: %w", err) return fmt.Errorf("adding blob to snapshot: %w", err)
} }
if !blobExists { upload := &database.Upload{
upload := &database.Upload{ BlobHash: finishedBlob.Hash,
BlobHash: finishedBlob.Hash, SnapshotID: s.snapshotID,
SnapshotID: s.snapshotID, UploadedAt: startTime,
UploadedAt: startTime, Size: finishedBlob.Compressed,
Size: finishedBlob.Compressed, DurationMs: uploadDuration.Milliseconds(),
DurationMs: uploadDuration.Milliseconds(), }
}
err := s.repos.Uploads.Create(txCtx, tx, upload) err = s.repos.Uploads.Create(txCtx, tx, upload)
if err != nil { if err != nil {
return fmt.Errorf("recording upload metrics: %w", err) return fmt.Errorf("recording upload metrics: %w", err)
}
} }
return nil return nil
+4 -17
View File
@@ -518,8 +518,8 @@ type CleanupStats struct {
// //
// The cleanup is performed in a specific order to maintain referential integrity: // The cleanup is performed in a specific order to maintain referential integrity:
// 1. Delete other snapshots // 1. Delete other snapshots
// 2. Delete orphaned snapshot associations (snapshot_files, snapshot_blobs, // 2. Delete orphaned snapshot associations (snapshot_files, snapshot_blobs)
// snapshot_mount_points) for deleted snapshots // for deleted snapshots
// 3. Delete orphaned files (not in the current snapshot) // 3. Delete orphaned files (not in the current snapshot)
// 4. Delete orphaned chunk-to-file mappings (references to deleted files) // 4. Delete orphaned chunk-to-file mappings (references to deleted files)
// 5. Delete orphaned blobs (not in the current snapshot) // 5. Delete orphaned blobs (not in the current snapshot)
@@ -948,8 +948,8 @@ func (sm *SnapshotManager) deleteOtherSnapshots(
return nil return nil
} }
// deleteOrphanedSnapshotAssociations deletes snapshot_files, // deleteOrphanedSnapshotAssociations deletes snapshot_files and
// snapshot_blobs and snapshot_mount_points for deleted snapshots // snapshot_blobs for deleted snapshots
func (sm *SnapshotManager) deleteOrphanedSnapshotAssociations( func (sm *SnapshotManager) deleteOrphanedSnapshotAssociations(
ctx context.Context, tx *sql.Tx, currentSnapshotID string, ctx context.Context, tx *sql.Tx, currentSnapshotID string,
) error { ) error {
@@ -983,19 +983,6 @@ func (sm *SnapshotManager) deleteOrphanedSnapshotAssociations(
log.Debug("[Temp DB Cleanup] Deleted snapshot_blobs associations", log.Debug("[Temp DB Cleanup] Deleted snapshot_blobs associations",
"count", rowsAffected) "count", rowsAffected)
// Delete the mount points other snapshots recorded
query := "DELETE FROM snapshot_mount_points WHERE snapshot_id != ?"
database.LogSQL("Execute", query, currentSnapshotID)
result, err = tx.ExecContext(ctx, query, currentSnapshotID)
if err != nil {
return fmt.Errorf("deleting orphaned snapshot_mount_points: %w", err)
}
rowsAffected, _ = result.RowsAffected()
log.Debug("[Temp DB Cleanup] Deleted snapshot_mount_points rows",
"count", rowsAffected)
return nil return nil
} }
-73
View File
@@ -7,14 +7,12 @@ import (
"database/sql" "database/sql"
"io" "io"
"path/filepath" "path/filepath"
"slices"
"testing" "testing"
"github.com/spf13/afero" "github.com/spf13/afero"
"sneak.berlin/go/vaultik/internal/config" "sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/database" "sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log" "sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/types"
) )
const ( const (
@@ -389,77 +387,6 @@ func TestCleanSnapshotDBNonExistentSnapshot(t *testing.T) {
} }
} }
// The exported metadata database keeps only the mount points that the
// exported snapshot recorded. See
// https://git.eeqj.de/sneak/vaultik/issues/395.
func TestCleanSnapshotDBKeepsOnlyTheSnapshotsMountPoints(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
dbPath := filepath.Join(t.TempDir(), "export.db")
db, err := database.New(ctx, dbPath)
if err != nil {
t.Fatalf("failed to create database: %v", err)
}
repos := database.NewRepositories(db)
for _, id := range []types.SnapshotID{"exported", "other"} {
err = repos.Snapshots.Create(ctx, nil, &database.Snapshot{ID: id})
if err != nil {
t.Fatalf("failed to create snapshot: %v", err)
}
err = repos.Snapshots.AddMountPoint(ctx, nil, id.String(), "/data/"+id.String())
if err != nil {
t.Fatalf("failed to add mount point: %v", err)
}
}
err = db.Close()
if err != nil {
t.Fatalf("failed to close database: %v", err)
}
sm := &SnapshotManager{
config: &config.Config{
CompressionLevel: 3,
AgeRecipients: []string{testAgeRecipient},
},
fs: afero.NewOsFs(),
}
_, err = sm.cleanSnapshotDB(ctx, dbPath, "exported")
if err != nil {
t.Fatalf("failed to clean snapshot database: %v", err)
}
cleaned, err := database.New(ctx, dbPath)
if err != nil {
t.Fatalf("failed to open cleaned database: %v", err)
}
defer func() { _ = cleaned.Close() }()
cleanedRepos := database.NewRepositories(cleaned)
for id, want := range map[string][]string{
"exported": {"/data/exported"},
"other": nil,
} {
got, err := cleanedRepos.Snapshots.GetMountPoints(ctx, id)
if err != nil {
t.Fatalf("failed to read mount points: %v", err)
}
if !slices.Equal(got, want) {
t.Errorf("mount points of %s: got %v, want %v", id, got, want)
}
}
}
// Two creates of one snapshot name back to back start within one second, // Two creates of one snapshot name back to back start within one second,
// the resolution of the timestamp in a snapshot ID. See // the resolution of the timestamp in a snapshot ID. See
// https://git.eeqj.de/sneak/vaultik/issues/270. // https://git.eeqj.de/sneak/vaultik/issues/270.
+8 -23
View File
@@ -2,17 +2,15 @@
// faults on demand, so tests can reproduce the failure modes a real // faults on demand, so tests can reproduce the failure modes a real
// backend exhibits: an upload that fails partway, a backend that reports // backend exhibits: an upload that fails partway, a backend that reports
// success while storing nothing, reads that return corrupt or truncated // success while storing nothing, reads that return corrupt or truncated
// bytes, and a delete or a stat the backend refuses. It is the seam // bytes, and a delete the backend refuses. It is the seam called for by
// called for by the fault-injection tests (sneak/vaultik issue 72) and // the fault-injection tests (sneak/vaultik issue 72) and is meant to be
// is meant to be reused by future tests rather than re-implemented per // reused by future tests rather than re-implemented per case.
// case.
// //
// The wrapper delegates every method to the inner Storer. Four hooks // The wrapper delegates every method to the inner Storer. Three hooks
// change that: OnPut decides the fate of each write, OnGet decides how // change that: OnPut decides the fate of each write, OnGet decides how
// each read's bytes are returned, OnDelete decides whether each delete // each read's bytes are returned, and OnDelete decides whether each
// fails, and OnStat whether each stat fails. All are keyed by the object // delete fails. All are keyed by the object key, so a test can fault
// key, so a test can fault only blobs, only metadata, or a single // only blobs, only metadata, or a single object.
// object.
package faultstore package faultstore
import ( import (
@@ -32,11 +30,6 @@ var ErrInjectedUpload = errors.New("faultstore: injected upload failure")
// fail. // fail.
var ErrInjectedDelete = errors.New("faultstore: injected delete failure") var ErrInjectedDelete = errors.New("faultstore: injected delete failure")
// ErrInjectedStat is returned by a Stat the OnStat hook chose to fail.
// It is not storage.ErrNotFound: the backend refused the request, as S3
// refuses a HEAD to credentials that allow only PUT.
var ErrInjectedStat = errors.New("faultstore: injected stat failure")
// PutAction is the disposition OnPut assigns to a write. // PutAction is the disposition OnPut assigns to a write.
type PutAction int type PutAction int
@@ -83,10 +76,6 @@ type Storer struct {
// object key. Returning true fails the delete and leaves the object // object key. Returning true fails the delete and leaves the object
// in place. // in place.
OnDelete func(key string) bool OnDelete func(key string) bool
// OnStat, when set, is consulted before every Stat with the object
// key. Returning true fails the stat with ErrInjectedStat.
OnStat func(key string) bool
} }
// New wraps inner. inner must be non-nil. // New wraps inner. inner must be non-nil.
@@ -170,12 +159,8 @@ func damage(fault GetFault, data []byte) []byte {
return out return out
} }
// Stat returns the object's info unless OnStat fails the stat. // Stat delegates unchanged.
func (f *Storer) Stat(ctx context.Context, key string) (*storage.ObjectInfo, error) { func (f *Storer) Stat(ctx context.Context, key string) (*storage.ObjectInfo, error) {
if f.OnStat != nil && f.OnStat(key) {
return nil, fmt.Errorf("%w for %q", ErrInjectedStat, key)
}
return f.inner.Stat(ctx, key) return f.inner.Stat(ctx, key)
} }
+2 -3
View File
@@ -100,9 +100,8 @@ func TestS3StorerMissingKeyMapsToErrNotFound(t *testing.T) {
// TestS3ListingCutShortIsAnError checks that List and ListStream fail on a // TestS3ListingCutShortIsAnError checks that List and ListStream fail on a
// listing page marked truncated that carries no continuation token, as a // listing page marked truncated that carries no continuation token, as a
// server answering V2 listings V1-style sends it. Taken as the last page, // server answering V2 listings V1-style sends it. Taken as the last page,
// it would hide every later key: prune would treat the blobs of the // it would hide every later key, and prune would treat the blobs of the
// snapshots listed there as unreferenced, and a directory under metadata/ // snapshots listed there as unreferenced.
// whose manifest is listed there as one with no manifest.
func TestS3ListingCutShortIsAnError(t *testing.T) { func TestS3ListingCutShortIsAnError(t *testing.T) {
t.Parallel() t.Parallel()
+10 -23
View File
@@ -5,16 +5,14 @@ import (
"fmt" "fmt"
"strings" "strings"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage" "sneak.berlin/go/vaultik/internal/storage"
) )
// destinationMarkerKey is the object every backup writes at the root of // destinationMarkerKey is the object every backup writes at the root of
// the backup destination store. While the local index holds a snapshot // the backup destination store. While the local index holds a snapshot
// stored there, a destination that answers a look-up of it with not // stored there, a destination without it is treated as not present: an
// found is treated as not present: an unplugged volume leaves an empty // unplugged volume leaves an empty directory at its mount point, which
// directory at its mount point, which would otherwise read as a store // would otherwise read as a store whose snapshots are all gone.
// whose snapshots are all gone.
const destinationMarkerKey = "vaultik-destination.txt" const destinationMarkerKey = "vaultik-destination.txt"
// destinationMarkerText is the whole content of the marker. It names no // destinationMarkerText is the whole content of the marker. It names no
@@ -22,24 +20,16 @@ const destinationMarkerKey = "vaultik-destination.txt"
const destinationMarkerText = "This is a vaultik backup destination store.\n" const destinationMarkerText = "This is a vaultik backup destination store.\n"
// errDestinationNotPresent is returned when the local index holds a // errDestinationNotPresent is returned when the local index holds a
// snapshot stored on the backup destination store and the store answers // snapshot stored on the backup destination store and the store has no
// that it has no marker. // marker.
var errDestinationNotPresent = errors.New("backup destination store is not present") var errDestinationNotPresent = errors.New("backup destination store is not present")
// checkDestinationMarker returns errDestinationNotPresent when the local // checkDestinationMarker returns errDestinationNotPresent when the local
// index holds a snapshot stored on the backup destination store and the // index holds a snapshot stored on the backup destination store and the
// store answers that it has no marker. With no snapshot in the local // store has no marker. With no snapshot in the local index, as on a
// index, as on a first backup or on another machine, there is nothing to // first backup or on another machine, there is nothing to check; nor
// check; nor when the local index is bound to a different destination // when the local index is bound to a different destination than the
// than the configured one, since its snapshots are stored there. // configured one, since its snapshots are stored there.
//
// A look-up of the marker that fails for any other reason than not
// found is logged and otherwise ignored, because the check could not be
// made: S3 refuses it to credentials that allow only PUT, which is all
// a backup needs. The check exists for a file:// volume's empty mount
// point, which always answers not found, and a destination that cannot
// be reached fails on the next request made to it, such as the backup's
// PUT of the marker.
func (v *Vaultik) checkDestinationMarker() error { func (v *Vaultik) checkDestinationMarker() error {
ids, err := v.localSnapshotIDs() ids, err := v.localSnapshotIDs()
if err != nil { if err != nil {
@@ -73,11 +63,8 @@ func (v *Vaultik) checkDestinationMarker() error {
destinationMarkerKey) destinationMarkerKey)
} }
// Info, not a warning: --cron must stay silent on a successful
// backup from a host whose credentials allow only PUT.
if err != nil { if err != nil {
log.Info("Could not check the backup destination store for its marker", return fmt.Errorf("checking for %s: %w", destinationMarkerKey, err)
"key", destinationMarkerKey, "error", err)
} }
return nil return nil
+3 -37
View File
@@ -13,16 +13,15 @@ import (
"sneak.berlin/go/vaultik/internal/database" "sneak.berlin/go/vaultik/internal/database"
"sneak.berlin/go/vaultik/internal/log" "sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage" "sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/storage/faultstore"
"sneak.berlin/go/vaultik/internal/vaultik" "sneak.berlin/go/vaultik/internal/vaultik"
) )
// These tests cover https://git.eeqj.de/sneak/vaultik/issues/239: a // These tests cover https://git.eeqj.de/sneak/vaultik/issues/239: a
// file:// destination at a volume's mount point leaves an empty directory // file:// destination at a volume's mount point leaves an empty directory
// behind when the volume is unplugged. While the local index holds // behind when the volume is unplugged. While the local index holds
// snapshots, a destination that answers that it has no // snapshots, a destination without vaultik-destination.txt is not present,
// vaultik-destination.txt is not present, so no command may write to it // so no command may write to it or conclude from it that the local
// or conclude from it that the local snapshots are gone. // snapshots are gone.
// markerFile is the name of the file a backup writes at the root of the // markerFile is the name of the file a backup writes at the root of the
// destination. // destination.
@@ -94,39 +93,6 @@ func TestBackupRefusesDestinationWithoutMarker(t *testing.T) {
assertUntouched(ctx, t, repos, storeDir) assertUntouched(ctx, t, repos, storeDir)
} }
// TestBackupGoesOnWhenMarkerCannotBeChecked checks that a backup goes on
// and writes the marker when looking it up fails with an error other
// than not found, as S3 answers credentials that allow only PUT
// (https://git.eeqj.de/sneak/vaultik/issues/389), and that a marker
// that is not found still stops it.
func TestBackupGoesOnWhenMarkerCannotBeChecked(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
storeDir := t.TempDir()
v, repos, _ := backUpToFileDestination(ctx, t, storeDir)
// Removed so the test sees the backup write it again: with the
// look-up failing, the backup cannot tell that it is gone.
markerPath := filepath.Join(storeDir, markerFile)
require.NoError(t, os.Remove(markerPath))
store := faultstore.New(v.Storage)
store.OnStat = func(key string) bool { return key == markerFile }
v.Storage = store
require.NoError(t, backUp(v, "second"))
localSnapshotID(ctx, t, repos, "second")
assert.FileExists(t, markerPath)
require.NoError(t, os.Remove(markerPath))
store.OnStat = nil
require.ErrorContains(t, backUp(v, "third"), markerFile)
}
// TestBackupChecksMarkerBeforeDroppingIncompleteSnapshots checks that // TestBackupChecksMarkerBeforeDroppingIncompleteSnapshots checks that
// snapshot create writes nothing when the local index holds only an // snapshot create writes nothing when the local index holds only an
// incomplete snapshot and the destination has no marker. The backup drops // incomplete snapshot and the destination has no marker. The backup drops
-1
View File
@@ -679,7 +679,6 @@ func faultScannerFactory(
Exclude: params.Exclude, Exclude: params.Exclude,
SkipErrors: params.SkipErrors, SkipErrors: params.SkipErrors,
PreviousSnapshotID: params.PreviousSnapshotID, PreviousSnapshotID: params.PreviousSnapshotID,
DeviceNumber: params.DeviceNumber,
}) })
} }
} }
+1 -1
View File
@@ -76,7 +76,7 @@ func TestNukeRemoteLeavesNoFiles(t *testing.T) {
// TestNukeRemoteFailsWhenADeleteFails checks that remote nuke returns an // TestNukeRemoteFailsWhenADeleteFails checks that remote nuke returns an
// error when a snapshot's metadata or a blob cannot be deleted, does not // error when a snapshot's metadata or a blob cannot be deleted, does not
// report the store empty, and advises running nuke again: prune deletes // report the store empty, and advises running nuke again: prune deletes
// no snapshot metadata that has its manifest. // no metadata.
func TestNukeRemoteFailsWhenADeleteFails(t *testing.T) { func TestNukeRemoteFailsWhenADeleteFails(t *testing.T) {
log.Initialize(log.Config{}) log.Initialize(log.Config{})
t.Parallel() t.Parallel()
+61 -245
View File
@@ -4,12 +4,10 @@ import (
"encoding/json" "encoding/json"
"errors" "errors"
"fmt" "fmt"
"sort"
"strings" "strings"
"time" "time"
"sneak.berlin/go/vaultik/internal/log" "sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot"
"sneak.berlin/go/vaultik/internal/storage" "sneak.berlin/go/vaultik/internal/storage"
) )
@@ -26,11 +24,6 @@ var errNukeRequiresForce = errors.New(
// errBlobDeleteFailed reports unreferenced blobs prune could not delete. // errBlobDeleteFailed reports unreferenced blobs prune could not delete.
var errBlobDeleteFailed = errors.New("blob(s) could not be deleted") var errBlobDeleteFailed = errors.New("blob(s) could not be deleted")
// errMetadataDirDeleteFailed reports directories under metadata/ with no
// manifest that prune could not delete.
var errMetadataDirDeleteFailed = errors.New(
"metadata director(ies) with no manifest could not be deleted")
// errObjectsLeft reports objects remote nuke still finds on the backup // errObjectsLeft reports objects remote nuke still finds on the backup
// destination store after deleting. // destination store after deleting.
var errObjectsLeft = errors.New( var errObjectsLeft = errors.New(
@@ -38,8 +31,7 @@ var errObjectsLeft = errors.New(
// nukeCommandHint is the command suggested when a remote nuke could not // nukeCommandHint is the command suggested when a remote nuke could not
// delete everything: running it again deletes what is left, which // delete everything: running it again deletes what is left, which
// `vaultik prune` does not do for snapshot metadata that has its // `vaultik prune` does not do for snapshot metadata.
// manifest.
const nukeCommandHint = "vaultik remote nuke --force" const nukeCommandHint = "vaultik remote nuke --force"
// metadataDirName is the top-level remote directory holding snapshot // metadataDirName is the top-level remote directory holding snapshot
@@ -66,8 +58,8 @@ func (v *Vaultik) NukeRemote(force bool) error {
// Checked before anything is deleted: running nuke again would fail // Checked before anything is deleted: running nuke again would fail
// the same way, so the advice below must not follow these errors. // the same way, so the advice below must not follow these errors.
// The listing also fails on a destination that answers that it has // The listing also fails on a destination without the marker while
// no marker while the local index holds snapshots. // the local index holds snapshots.
err := v.EnsureStorageBinding() err := v.EnsureStorageBinding()
if err != nil { if err != nil {
return err return err
@@ -177,20 +169,14 @@ type PruneBlobsResult struct {
// newer than the prune grace period, or of unknown age. BlobsFound // newer than the prune grace period, or of unknown age. BlobsFound
// does not include them. // does not include them.
BlobsTooNew int `json:"blobs_too_new"` BlobsTooNew int `json:"blobs_too_new"`
// MetadataDirsDeleted counts the directories under metadata/ with no
// manifest that prune deleted, and MetadataDirsFailed those it could
// not delete.
MetadataDirsDeleted int `json:"metadata_dirs_deleted"`
MetadataDirsFailed int `json:"metadata_dirs_failed,omitempty"`
} }
// Prune removes orphaned data from the local index database AND, from the // Prune removes orphaned data from the local index database AND
// backup destination store, unreferenced blobs and the directories under // unreferenced blobs from the backup destination store. This is the
// metadata/ with no manifest. This is the single user-facing prune entry // single user-facing prune entry point — the split between local and
// point — the split between local and remote cleanup is an // remote cleanup is an implementation detail. Calling code should
// implementation detail. Calling code should prefer this method over // prefer this method over PruneDatabase or PruneBlobs individually
// PruneDatabase or PruneBlobs individually unless it specifically wants // unless it specifically wants one half.
// one half.
func (v *Vaultik) Prune(opts *PruneOptions) error { func (v *Vaultik) Prune(opts *PruneOptions) error {
err := v.EnsureStorageBinding() err := v.EnsureStorageBinding()
if err != nil { if err != nil {
@@ -214,17 +200,11 @@ func (v *Vaultik) Prune(opts *PruneOptions) error {
return v.PruneBlobs(opts) return v.PruneBlobs(opts)
} }
// PruneBlobs removes unreferenced blobs from storage, and each directory // PruneBlobs removes unreferenced blobs from storage
// under metadata/ that a backup left with no manifest.
func (v *Vaultik) PruneBlobs(opts *PruneOptions) error { func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
log.Info("Starting prune operation") log.Info("Starting prune operation")
metadataObjects, err := v.listMetadataObjects() allBlobsReferenced, err := v.collectReferencedBlobs()
if err != nil {
return err
}
allBlobsReferenced, err := v.collectReferencedBlobs(metadataObjects)
if err != nil { if err != nil {
return err return err
} }
@@ -237,11 +217,6 @@ func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
unreferencedBlobs, totalSize, tooNew := v.findUnreferencedBlobs( unreferencedBlobs, totalSize, tooNew := v.findUnreferencedBlobs(
allBlobs, allBlobsReferenced) allBlobs, allBlobsReferenced)
dirsWithoutManifest, err := v.findMetadataDirsWithoutManifest(metadataObjects)
if err != nil {
return err
}
result := &PruneBlobsResult{ result := &PruneBlobsResult{
BlobsFound: len(unreferencedBlobs), BlobsFound: len(unreferencedBlobs),
BlobsTooNew: tooNew, BlobsTooNew: tooNew,
@@ -249,7 +224,7 @@ func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
v.reportBlobsTooNew(tooNew, opts) v.reportBlobsTooNew(tooNew, opts)
if len(unreferencedBlobs) == 0 && len(dirsWithoutManifest) == 0 { if len(unreferencedBlobs) == 0 {
log.Info("No unreferenced blobs found") log.Info("No unreferenced blobs found")
if opts.JSON { if opts.JSON {
@@ -262,16 +237,38 @@ func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
} }
log.Info("Found unreferenced blobs", log.Info("Found unreferenced blobs",
"count", len(unreferencedBlobs), "total_size", ubytes(totalSize), "count", len(unreferencedBlobs), "total_size", ubytes(totalSize))
"metadata_dirs_without_manifest", len(dirsWithoutManifest))
if !v.confirmPrune(opts, len(unreferencedBlobs), totalSize, if opts.Force && !opts.JSON {
dirsWithoutManifest) { v.UI.Infof("Found %d unreferenced blob(s) totaling %s.",
return nil len(unreferencedBlobs), ubytes(totalSize))
}
if !opts.Force && !opts.JSON {
// The count, the prompt and its answer bypass the UI: they are an
// interactive exchange the operator must see, so --quiet does not
// silence them.
v.stdoutf("Found %d unreferenced blob(s) totaling %s.\n",
len(unreferencedBlobs), ubytes(totalSize))
v.stdoutf("\nDelete %d unreferenced blob(s)? [y/N] ", len(unreferencedBlobs))
var confirm string
_, err = v.scanStdin(&confirm)
if err != nil {
v.printlnStdout("Cancelled")
return nil //nolint:nilerr // read failure means no confirmation
}
if strings.ToLower(confirm) != "y" {
v.printlnStdout("Cancelled")
return nil
}
} }
v.deleteUnreferencedBlobs(unreferencedBlobs, allBlobs, result) v.deleteUnreferencedBlobs(unreferencedBlobs, allBlobs, result)
v.deleteMetadataDirs(dirsWithoutManifest, result)
if opts.JSON { if opts.JSON {
err = v.outputPruneBlobsJSON(result) err = v.outputPruneBlobsJSON(result)
@@ -279,142 +276,33 @@ func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
return err return err
} }
} else { } else {
v.reportPruneDeletions(result, len(dirsWithoutManifest))
}
var failures []error
if result.BlobsFailed > 0 {
failures = append(failures,
fmt.Errorf("%d %w", result.BlobsFailed, errBlobDeleteFailed))
}
if result.MetadataDirsFailed > 0 {
failures = append(failures, fmt.Errorf("%d %w",
result.MetadataDirsFailed, errMetadataDirDeleteFailed))
}
return errors.Join(failures...)
}
// confirmPrune states what prune found to delete, unreferenced blobs and
// directories under metadata/ with no manifest, naming each directory by
// its remote key, and, unless --force is given, asks the operator to
// confirm. Under --json it prints nothing and does not ask. It reports
// whether to go ahead.
func (v *Vaultik) confirmPrune(
opts *PruneOptions, blobCount int, blobsSize int64, dirs []string,
) bool {
if opts.JSON {
return true
}
dirCount := len(dirs)
blobsFound := fmt.Sprintf("Found %d unreferenced blob(s) totaling %s.",
blobCount, ubytes(blobsSize))
dirsFound := fmt.Sprintf("Found %d metadata director(ies) left by "+
"backups that stopped before their manifest:", dirCount)
if opts.Force {
if blobCount > 0 {
v.UI.Infof("%s", blobsFound)
}
if dirCount > 0 {
v.UI.Infof("%s", dirsFound)
for _, remoteKey := range dirs {
v.UI.Detailf("metadata/%s/", remoteKey)
}
}
return true
}
// The counts, the names, the prompt and its answer bypass the UI:
// they are an interactive exchange the operator must see, so --quiet
// does not silence them.
if blobCount > 0 {
v.stdoutf("%s\n", blobsFound)
}
if dirCount > 0 {
v.stdoutf("%s\n", dirsFound)
for _, remoteKey := range dirs {
v.stdoutf(" metadata/%s/\n", remoteKey)
}
}
switch {
case dirCount == 0:
v.stdoutf("\nDelete %d unreferenced blob(s)? [y/N] ", blobCount)
case blobCount == 0:
v.stdoutf("\nDelete %d metadata director(ies)? [y/N] ", dirCount)
default:
v.stdoutf("\nDelete %d unreferenced blob(s) and %d metadata "+
"director(ies)? [y/N] ", blobCount, dirCount)
}
var confirm string
// A failure to read the answer means no confirmation.
_, err := v.scanStdin(&confirm)
if err != nil || strings.ToLower(confirm) != "y" {
v.printlnStdout("Cancelled")
return false
}
return true
}
// reportPruneDeletions says how many blobs and metadata directories
// prune deleted, naming each kind only when it found some to delete.
func (v *Vaultik) reportPruneDeletions(result *PruneBlobsResult, dirCount int) {
if result.BlobsFound > 0 {
v.UI.Infof("Deleted %d blob(s) totaling %s.", v.UI.Infof("Deleted %d blob(s) totaling %s.",
result.BlobsDeleted, ubytes(result.BytesFreed)) result.BlobsDeleted, ubytes(result.BytesFreed))
} }
if dirCount > 0 { if result.BlobsFailed > 0 {
v.UI.Infof("Deleted %d metadata director(ies) with no manifest.", return fmt.Errorf("%d %w", result.BlobsFailed, errBlobDeleteFailed)
result.MetadataDirsDeleted)
}
}
// listMetadataObjects lists every object under metadata/ on the
// destination store.
func (v *Vaultik) listMetadataObjects() ([]storage.ObjectInfo, error) {
log.Info("Listing remote snapshots")
var objects []storage.ObjectInfo
for object := range v.Storage.ListStream(v.ctx, "metadata/") {
if object.Err != nil {
return nil, fmt.Errorf("listing metadata objects: %w", object.Err)
}
objects = append(objects, object)
} }
return objects, nil return nil
} }
// collectReferencedBlobs downloads every manifest among the listed // collectReferencedBlobs downloads all manifests and returns the set of
// metadata objects and returns the set of referenced blob hashes. // referenced blob hashes.
// //
// Every manifest must be read successfully. A manifest that cannot be // Every manifest must be read successfully. A manifest that cannot be
// downloaded or decoded means its snapshot's blobs are unknown, so // downloaded or decoded means its snapshot's blobs are unknown, so
// treating them as unreferenced would let prune delete data a snapshot // treating them as unreferenced would let prune delete data a snapshot
// still needs. Rather than risk that silent loss, any failure returns an // still needs. Rather than risk that silent loss, any failure returns an
// error naming the remote key and prune deletes nothing. // error naming the remote key and prune deletes nothing.
func (v *Vaultik) collectReferencedBlobs( func (v *Vaultik) collectReferencedBlobs() (map[string]bool, error) {
metadataObjects []storage.ObjectInfo, log.Info("Listing remote snapshots")
) (map[string]bool, error) { // IDs returned by listUniqueSnapshotIDs are remote keys (hashed
// IDs returned by uniqueSnapshotIDs are remote keys (hashed
// subdirectories under metadata/), not human snapshot IDs. // subdirectories under metadata/), not human snapshot IDs.
remoteKeys := uniqueSnapshotIDs(metadataObjects) remoteKeys, err := v.listUniqueSnapshotIDs()
if err != nil {
return nil, fmt.Errorf("listing snapshot keys: %w", err)
}
log.Info("Found manifests in remote storage", "count", len(remoteKeys)) log.Info("Found manifests in remote storage", "count", len(remoteKeys))
@@ -439,14 +327,18 @@ func (v *Vaultik) collectReferencedBlobs(
return allBlobsReferenced, nil return allBlobsReferenced, nil
} }
// uniqueSnapshotIDs returns deduplicated snapshot IDs from the listed // listUniqueSnapshotIDs returns deduplicated snapshot IDs from remote metadata
// metadata objects. func (v *Vaultik) listUniqueSnapshotIDs() ([]string, error) {
func uniqueSnapshotIDs(metadataObjects []storage.ObjectInfo) []string { objectCh := v.Storage.ListStream(v.ctx, "metadata/")
seen := make(map[string]bool) seen := make(map[string]bool)
var snapshotIDs []string var snapshotIDs []string
for _, object := range metadataObjects { for object := range objectCh {
if object.Err != nil {
return nil, fmt.Errorf("listing metadata objects: %w", object.Err)
}
parts := strings.Split(object.Key, "/") parts := strings.Split(object.Key, "/")
if len(parts) >= minSnapshotIDParts && if len(parts) >= minSnapshotIDParts &&
parts[0] == metadataDirName && parts[1] != "" { parts[0] == metadataDirName && parts[1] != "" {
@@ -461,7 +353,7 @@ func uniqueSnapshotIDs(metadataObjects []storage.ObjectInfo) []string {
} }
} }
return snapshotIDs return snapshotIDs, nil
} }
// listAllRemoteBlobs returns a map of all blob hashes in remote storage // listAllRemoteBlobs returns a map of all blob hashes in remote storage
@@ -535,82 +427,6 @@ func (v *Vaultik) findUnreferencedBlobs(
return unreferenced, totalSize, tooNew return unreferenced, totalSize, tooNew
} }
// findMetadataDirsWithoutManifest returns the remote key of each
// directory under metadata/ that has no manifest.json.zst, such as one a
// backup leaves when its manifest upload fails after db.zst.age was
// uploaded. It leaves out a directory a local snapshot record claims,
// and one holding an object newer than the prune grace period, or of
// unknown age because the listing gave no time for it: a backup still
// running may be writing there. A name that is not a remote key is
// never returned.
func (v *Vaultik) findMetadataDirsWithoutManifest(
metadataObjects []storage.ObjectInfo,
) ([]string, error) {
localIDs, err := v.localSnapshotIDs()
if err != nil {
return nil, fmt.Errorf("listing local snapshots: %w", err)
}
keep := make(map[string]bool, len(localIDs))
for _, id := range localIDs {
keep[snapshot.RemoteSnapshotKey(id)] = true
}
cutoff := time.Now().UTC().Add(-v.Config.PruneGracePeriod)
listed := make(map[string]bool)
for _, object := range metadataObjects {
parts := strings.Split(object.Key, "/")
if len(parts) < metadataKeyParts || !isBlobHash(parts[1]) {
continue
}
remoteKey := parts[1]
listed[remoteKey] = true
if parts[2] == "manifest.json.zst" ||
object.ModTime.IsZero() || object.ModTime.After(cutoff) {
keep[remoteKey] = true
}
}
var remoteKeys []string
for remoteKey := range listed {
if !keep[remoteKey] {
remoteKeys = append(remoteKeys, remoteKey)
}
}
sort.Strings(remoteKeys)
return remoteKeys, nil
}
// deleteMetadataDirs deletes everything in the given directories under
// metadata/ and counts in the result the directories deleted and those
// it could not delete.
func (v *Vaultik) deleteMetadataDirs(
remoteKeys []string, result *PruneBlobsResult,
) {
for _, remoteKey := range remoteKeys {
_, err := v.deleteRemoteSnapshotByKey(remoteKey)
if err != nil {
log.Error("Failed to delete metadata directory with no manifest",
"remote_key", remoteKey, "error", err)
result.MetadataDirsFailed++
continue
}
log.Info("Deleted metadata directory with no manifest",
"remote_key", remoteKey)
result.MetadataDirsDeleted++
}
}
// reportBlobsTooNew says how many unreferenced blobs prune kept for being // reportBlobsTooNew says how many unreferenced blobs prune kept for being
// newer than the grace period or of unknown age. Under --json the // newer than the grace period or of unknown age. Under --json the
// document carries the count instead. // document carries the count instead.
+97 -5
View File
@@ -24,9 +24,8 @@ import (
// TestPruneDuringBackupKeepsItsBlobs runs a prune from another host // TestPruneDuringBackupKeepsItsBlobs runs a prune from another host
// sharing the destination store at the moment a backup has uploaded its // sharing the destination store at the moment a backup has uploaded its
// blobs and its db.zst.age but not yet its manifest, so no manifest lists // blobs but not yet its manifest, so no manifest lists those blobs. The
// those blobs. The prune must leave the blobs and the db.zst.age, and the // prune must leave them, and the backup must then restore.
// backup must then restore.
func TestPruneDuringBackupKeepsItsBlobs(t *testing.T) { func TestPruneDuringBackupKeepsItsBlobs(t *testing.T) {
log.Initialize(log.Config{}) log.Initialize(log.Config{})
t.Parallel() t.Parallel()
@@ -80,8 +79,101 @@ func TestPruneDuringBackupKeepsItsBlobs(t *testing.T) {
SnapshotID: id, SnapshotID: id,
TargetDir: restoreDir, TargetDir: restoreDir,
Verify: true, Verify: true,
}), "a prune during the backup must not delete the backup's blobs "+ }), "a prune during the backup must not delete the backup's blobs")
"or its db.zst.age")
assertRestoredTree(t, fs, restoreDir, testFiles)
}
// TestPruneDuringBackupKeepsBlobsOfKilledRun covers a backup that builds
// the blobs an earlier run, stopped before its manifest upload, left on
// the destination store, older than the grace period by the time of the
// backup. A prune from another host runs before the backup's manifest
// upload, and the backup must then restore.
func TestPruneDuringBackupKeepsBlobsOfKilledRun(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
fs := afero.NewOsFs()
tempDir := t.TempDir()
dataDir := filepath.Join(tempDir, "src")
storeDir := t.TempDir()
restoreDir := filepath.Join(tempDir, "restored")
dbPath := filepath.Join(tempDir, "index.sqlite")
ctx := context.Background()
testFiles := writeFaultSourceTree(t, fs, dataDir)
cfg := faultTestConfig()
cfg.IndexPath = dbPath
cfg.ChunkSize = config.Size(faultChunkSize)
cfg.Snapshots = map[string]config.SnapshotConfig{
"data": {Paths: []string{dataDir}},
}
inner, err := storage.NewFileStorer(storeDir)
require.NoError(t, err)
otherHostCfg := faultTestConfig()
otherHostCfg.PruneGracePeriod = time.Hour
otherHost := newReaderVaultik(ctx, otherHostCfg, inner, nil, fs)
var (
killedRun = true
pruned bool
pruneErr error
)
store := faultstore.New(inner)
store.OnPut = func(key string) faultstore.PutAction {
if !strings.HasSuffix(key, "/manifest.json.zst") {
return faultstore.PutNormal
}
if killedRun {
return faultstore.PutFail
}
pruned = true
pruneErr = otherHost.PruneBlobs(&vaultik.PruneOptions{Force: true})
return faultstore.PutNormal
}
db, err := database.New(ctx, dbPath)
require.NoError(t, err)
repos := database.NewRepositories(db)
v := newBackupVaultik(ctx, cfg, store, repos, db, fs)
opts := &vaultik.SnapshotCreateOptions{Cron: true}
require.Error(t, v.CreateSnapshot(opts),
"the first run must stop at its manifest upload")
beforeGracePeriod := time.Now().Add(-2 * otherHostCfg.PruneGracePeriod)
for object := range inner.ListStream(ctx, "blobs/") {
require.NoError(t, object.Err)
require.NoError(t, os.Chtimes(filepath.Join(storeDir, object.Key),
beforeGracePeriod, beforeGracePeriod))
}
killedRun = false
require.NoError(t, v.CreateSnapshot(opts))
require.True(t, pruned, "the prune must run before the manifest upload")
require.NoError(t, pruneErr)
local, err := repos.Snapshots.ListRecent(ctx, listRecentTestLimit)
require.NoError(t, err)
require.Len(t, local, 1)
require.NoError(t, db.Close())
reader := newReaderVaultik(ctx, cfg, inner, nil, fs)
require.NoError(t, reader.Restore(&vaultik.RestoreOptions{
SnapshotID: local[0].ID.String(),
TargetDir: restoreDir,
Verify: true,
}), "a prune during the backup must not delete the blobs it rebuilt")
assertRestoredTree(t, fs, restoreDir, testFiles) assertRestoredTree(t, fs, restoreDir, testFiles)
} }
-225
View File
@@ -1,225 +0,0 @@
package vaultik_test
import (
"bytes"
"context"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/storage/faultstore"
"sneak.berlin/go/vaultik/internal/ui"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// metadataObjectKey returns the key of the named file in the metadata
// directory of the snapshot with the given human ID.
func metadataObjectKey(snapshotID, name string) string {
return "metadata/" + snapshot.RemoteSnapshotKey(snapshotID) + "/" + name
}
// TestPruneDeletesMetadataDirWithoutManifestOlderThanGracePeriod checks
// that prune deletes a directory under metadata/ holding only
// db.zst.age, as a backup leaves when its manifest upload fails, once it
// is older than the grace period, and keeps one newer than that. A
// directory with a manifest is never deleted, however old.
func TestPruneDeletesMetadataDirWithoutManifestOlderThanGracePeriod(
t *testing.T,
) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
storeDir := t.TempDir()
store, err := storage.NewFileStorer(storeDir)
require.NoError(t, err)
const (
oldID = "testhost_home_2026-01-01T00:00:00Z"
newID = "testhost_home_2026-01-02T00:00:00Z"
completeID = "testhost_home_2026-01-03T00:00:00Z"
)
oldDB := metadataObjectKey(oldID, "db.zst.age")
newDB := metadataObjectKey(newID, "db.zst.age")
completeDB := metadataObjectKey(completeID, "db.zst.age")
completeManifest := metadataObjectKey(completeID, "manifest.json.zst")
manifest, err := snapshot.EncodeManifest(&snapshot.Manifest{
SnapshotID: snapshot.RemoteSnapshotKey(completeID),
}, 3)
require.NoError(t, err)
for _, key := range []string{oldDB, newDB, completeDB} {
require.NoError(t, store.Put(ctx, key, strings.NewReader("db")))
}
require.NoError(t, store.Put(ctx, completeManifest,
bytes.NewReader(manifest)))
gracePeriod := time.Hour
beforeGracePeriod := time.Now().Add(-gracePeriod - time.Minute)
for _, key := range []string{oldDB, completeDB, completeManifest} {
require.NoError(t, os.Chtimes(filepath.Join(storeDir, key),
beforeGracePeriod, beforeGracePeriod))
}
stdout := &bytes.Buffer{}
v := &vaultik.Vaultik{
Config: &config.Config{PruneGracePeriod: gracePeriod},
Storage: store,
Stdout: stdout,
UI: ui.NewWithColor(stdout, false),
}
v.SetContext(ctx)
require.NoError(t, v.PruneBlobs(&vaultik.PruneOptions{JSON: true}))
_, err = store.Stat(ctx, oldDB)
require.ErrorIs(t, err, storage.ErrNotFound,
"a directory with no manifest older than the grace period "+
"must be deleted")
for _, key := range []string{newDB, completeDB, completeManifest} {
_, err = store.Stat(ctx, key)
require.NoError(t, err, "%s must be kept", key)
}
var result vaultik.PruneBlobsResult
require.NoError(t, json.Unmarshal(stdout.Bytes(), &result))
assert.Equal(t, 1, result.MetadataDirsDeleted)
}
// TestPruneKeepsMetadataDirClaimedUnknownAgeOrNotRemoteKey checks that
// prune --force keeps a directory under metadata/ with no manifest when a
// local snapshot record claims it, when the listing gives no time for an
// object in it, or when its name is not a remote key. A fourth such
// directory is named, deleted and reported, although there are no
// unreferenced blobs to delete.
func TestPruneKeepsMetadataDirClaimedUnknownAgeOrNotRemoteKey(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
ctx := context.Background()
env := newListEnv(t)
const (
claimedID = "testhost_home_2026-01-01T00:00:00Z"
leftID = "testhost_home_2026-01-03T00:00:00Z"
)
env.addLocal(t, claimedID, time.Now().Add(-time.Hour))
claimedDB := metadataObjectKey(claimedID, "db.zst.age")
unknownAgeDB := metadataObjectKey("testhost_home_2026-01-02T00:00:00Z",
"db.zst.age")
leftDB := metadataObjectKey(leftID, "db.zst.age")
notRemoteKeyDB := "metadata/not-a-remote-key/db.zst.age"
for _, key := range []string{claimedDB, leftDB, notRemoteKeyDB} {
require.NoError(t, env.store.Put(ctx, key, strings.NewReader("db")))
}
// Written into the mock store's map directly rather than through
// Put, so its listing gives no time.
env.store.data[unknownAgeDB] = []byte("db")
require.NoError(t, env.v.PruneBlobs(&vaultik.PruneOptions{Force: true}))
assert.True(t, env.store.hasKey(claimedDB),
"a directory a local snapshot record claims must be kept")
assert.True(t, env.store.hasKey(unknownAgeDB),
"a directory holding an object of unknown age must be kept")
assert.True(t, env.store.hasKey(notRemoteKeyDB),
"a directory whose name is not a remote key must be kept")
assert.False(t, env.store.hasKey(leftDB),
"a directory with no manifest, unclaimed and old, must be deleted")
output := env.stdout.String()
assert.Contains(t, output, "Found 1 metadata director(ies) left by "+
"backups that stopped before their manifest:")
assert.Contains(t, output,
"metadata/"+snapshot.RemoteSnapshotKey(leftID)+"/")
assert.NotContains(t, output, "not-a-remote-key")
assert.Contains(t, output,
"Deleted 1 metadata director(ies) with no manifest.")
assert.NotContains(t, output, "No unreferenced blobs to remove.")
}
// TestPruneDeclinedKeepsMetadataDirWithoutManifest checks that the prompt
// names a directory under metadata/ with no manifest that prune is about
// to delete, and that answering no keeps it.
func TestPruneDeclinedKeepsMetadataDirWithoutManifest(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newListEnv(t)
const leftID = "testhost_home_2026-01-01T00:00:00Z"
leftDB := metadataObjectKey(leftID, "db.zst.age")
require.NoError(t, env.store.Put(context.Background(), leftDB,
strings.NewReader("db")))
env.v.Stdin = strings.NewReader("n\n")
require.NoError(t, env.v.PruneBlobs(&vaultik.PruneOptions{}))
assert.True(t, env.store.hasKey(leftDB),
"a declined prune must keep the directory")
output := env.stdout.String()
assert.Contains(t, output,
" metadata/"+snapshot.RemoteSnapshotKey(leftID)+"/\n")
assert.Contains(t, output, "Delete 1 metadata director(ies)? [y/N] ")
assert.Contains(t, output, "Cancelled")
assert.NotContains(t, output, "Deleted")
}
// TestPruneFailedMetadataDirDeleteReturnsError checks that prune --json
// counts a directory under metadata/ with no manifest that it cannot
// delete, and then returns an error, so the command exits non-zero.
func TestPruneFailedMetadataDirDeleteReturnsError(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newListEnv(t)
leftDB := metadataObjectKey("testhost_home_2026-01-01T00:00:00Z",
"db.zst.age")
require.NoError(t, env.store.Put(context.Background(), leftDB,
strings.NewReader("db")))
store := faultstore.New(env.store)
store.OnDelete = func(key string) bool {
return strings.HasPrefix(key, "metadata/")
}
env.v.Storage = store
err := env.v.PruneBlobs(&vaultik.PruneOptions{JSON: true})
require.ErrorContains(t, err,
"1 metadata director(ies) with no manifest could not be deleted")
assert.True(t, env.store.hasKey(leftDB))
var result vaultik.PruneBlobsResult
require.NoError(t, json.Unmarshal(env.stdout.Bytes(), &result),
"stdout must hold the JSON document, got:\n%s", env.stdout.String())
assert.Equal(t, 1, result.MetadataDirsFailed)
assert.Equal(t, 0, result.MetadataDirsDeleted)
}
+9 -9
View File
@@ -14,13 +14,13 @@ import (
"sneak.berlin/go/vaultik/internal/vaultik" "sneak.berlin/go/vaultik/internal/vaultik"
) )
// TestVerifyPassesWhenBackupReusesStoredBlob backs up a file, then backs it // TestVerifyPassesWhenBackupRebuildsStoredBlob backs up a file, then backs
// up again from an empty local index, as after `database delete`, with an // it up again from an empty local index, as after `database delete`, with
// extra age recipient. The second backup builds the same blob, finds it // an extra age recipient. The second backup builds the same blob, already
// already stored and skips the upload, but its own copy is larger, because // stored, and uploads it again; its copy is larger, because age writes a
// age writes a header entry for each recipient. The second snapshot must // header entry for each recipient. The second snapshot must record the
// record the size of the stored object, so shallow and deep verify pass. // size of the object now stored, so shallow and deep verify pass.
func TestVerifyPassesWhenBackupReusesStoredBlob(t *testing.T) { func TestVerifyPassesWhenBackupRebuildsStoredBlob(t *testing.T) {
log.Initialize(log.Config{}) log.Initialize(log.Config{})
t.Parallel() t.Parallel()
@@ -58,8 +58,8 @@ func TestVerifyPassesWhenBackupReusesStoredBlob(t *testing.T) {
snapshotID, result := runDedupSnapshot(ctx, t, env.sm, env.makeScanner(), snapshotID, result := runDedupSnapshot(ctx, t, env.sm, env.makeScanner(),
env.cfg.Hostname, "second", dataDir, secondDBPath) env.cfg.Hostname, "second", dataDir, secondDBPath)
require.Positive(t, result.BlobsCreated) require.Positive(t, result.BlobsCreated)
require.Zero(t, result.BlobsUploaded, require.Equal(t, result.BlobsCreated, result.BlobsUploaded,
"the second backup should find every blob already stored") "the second backup should upload every blob again")
require.NoError(t, env.db.Close()) require.NoError(t, env.db.Close())
+37 -192
View File
@@ -7,7 +7,6 @@ import (
"os" "os"
"path/filepath" "path/filepath"
"regexp" "regexp"
"slices"
"sort" "sort"
"strconv" "strconv"
"strings" "strings"
@@ -45,25 +44,6 @@ var ErrPostBackupPruneFailed = errors.New("post-backup prune failed")
// kept, so the caller can report it apart from a failed backup. // kept, so the caller can report it apart from a failed backup.
var ErrFilesUnreadable = errors.New("file(s) could not be read") var ErrFilesUnreadable = errors.New("file(s) could not be read")
// ErrSourcePathEmpty is returned when --prune skipped the post-backup
// prune because the scan found nothing in a configured path of a
// snapshot, as in the mount point of a volume that is not mounted. The
// snapshots are complete and kept, so the caller can report it apart
// from a failed backup. A run that also skipped unreadable files
// returns ErrFilesUnreadable instead.
var ErrSourcePathEmpty = errors.New("source path(s) were empty")
// ErrVolumeNotMounted is returned when --prune skipped the post-backup
// prune because a directory that an earlier snapshot of the same name
// found as a mount point was not found as one in this run: the volume
// mounted there is not mounted, the run found no entry in it, or the
// directory is no longer backed up. The snapshots are complete and
// kept, so the caller can report it apart from a failed backup. A run
// that also skipped unreadable files or found a configured path empty
// returns ErrFilesUnreadable or ErrSourcePathEmpty instead.
var ErrVolumeNotMounted = errors.New(
"mount point(s) an earlier snapshot recorded were not found")
// listRecentLimit caps how many snapshot rows are fetched from the // listRecentLimit caps how many snapshot rows are fetched from the
// local index for listing and purge operations. // local index for listing and purge operations.
const listRecentLimit = 10000 const listRecentLimit = 10000
@@ -154,10 +134,17 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
return err return err
} }
filesUnreadable, emptyPaths, unmountedPaths, err := v.createNamedSnapshots( // Process each named snapshot
opts, hostname, snapshotNames) filesUnreadable := 0
if err != nil {
return err for snapIdx, snapName := range snapshotNames {
unreadable, err := v.createNamedSnapshot(
opts, hostname, snapName, snapIdx+1, len(snapshotNames))
if err != nil {
return err
}
filesUnreadable += unreadable
} }
// Print overall summary if multiple snapshots // Print overall summary if multiple snapshots
@@ -167,10 +154,13 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
} }
switch { switch {
case opts.Prune && (filesUnreadable > 0 || len(emptyPaths) > 0 || case opts.Prune && filesUnreadable > 0:
len(unmountedPaths) > 0): // An older snapshot may be the only one that still holds a file
return v.skipPostBackupPrune( // this run could not read.
filesUnreadable, emptyPaths, unmountedPaths) log.Warn("Skipping post-backup prune",
"files_unreadable", filesUnreadable)
v.UI.Warningf("Skipped the post-backup prune because %d file(s) "+
"could not be read; older snapshots are kept.", filesUnreadable)
case opts.Prune: case opts.Prune:
err = v.runPostBackupPrune(snapshotNames, opts.KeepNewerThan) err = v.runPostBackupPrune(snapshotNames, opts.KeepNewerThan)
if err != nil { if err != nil {
@@ -181,84 +171,6 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
return v.finishCreateSnapshot(filesUnreadable) return v.finishCreateSnapshot(filesUnreadable)
} }
// createNamedSnapshots creates each named snapshot in turn. It returns
// the number of entries --skip-errors skipped because it could not read
// or list them, each configured path in which the scan found nothing,
// and, under --prune, each directory where an earlier snapshot found a
// volume mounted and this run did not, the paths named with their
// snapshot.
func (v *Vaultik) createNamedSnapshots(
opts *SnapshotCreateOptions, hostname string, snapshotNames []string,
) (int, []string, []string, error) {
filesUnreadable := 0
var emptyPaths, unmountedPaths []string
for snapIdx, snapName := range snapshotNames {
stats, err := v.createNamedSnapshot(
opts, hostname, snapName, snapIdx+1, len(snapshotNames))
if err != nil {
return 0, nil, nil, err
}
filesUnreadable += stats.totalFilesUnreadable
for _, dir := range stats.emptyPaths {
emptyPaths = append(emptyPaths, dir+" of snapshot "+snapName)
}
for _, dir := range stats.unmountedPaths {
unmountedPaths = append(unmountedPaths,
dir+" of snapshot "+snapName)
}
}
return filesUnreadable, emptyPaths, unmountedPaths, nil
}
// skipPostBackupPrune says why the post-backup prune does not run, naming
// every cause, and returns the error the run ends with. An older snapshot
// may be the only one that still holds a file this run could not read,
// or what an empty path or a volume that is not mounted held. The
// previous snapshot is no guide: after one such run it lacks them too.
func (v *Vaultik) skipPostBackupPrune(
filesUnreadable int, emptyPaths, unmountedPaths []string,
) error {
log.Warn("Skipping post-backup prune",
"files_unreadable", filesUnreadable, "empty_paths", emptyPaths,
"unmounted_paths", unmountedPaths)
if len(emptyPaths) > 0 {
v.UI.Warningf("Skipped the post-backup prune because nothing was "+
"found in %s; older snapshots are kept.",
strings.Join(emptyPaths, ", "))
}
if len(unmountedPaths) > 0 {
v.UI.Warningf("Skipped the post-backup prune because this backup "+
"did not find a mount point at %s, where an earlier snapshot "+
"found one: the volume is not mounted, this backup found no "+
"entry in it, or the directory is no longer backed up. Older "+
"snapshots are kept. Mount the volume again, or, if that is on "+
"purpose, remove the older snapshots by hand with "+
"'vaultik snapshot purge'.",
strings.Join(unmountedPaths, ", "))
}
if filesUnreadable > 0 {
v.UI.Warningf("Skipped the post-backup prune because %d file(s) "+
"could not be read; older snapshots are kept.", filesUnreadable)
return fmt.Errorf("%d %w", filesUnreadable, ErrFilesUnreadable)
}
if len(emptyPaths) > 0 {
return fmt.Errorf("%d %w", len(emptyPaths), ErrSourcePathEmpty)
}
return fmt.Errorf("%d %w", len(unmountedPaths), ErrVolumeNotMounted)
}
// finishCreateSnapshot ends a run whose snapshots are all complete and // finishCreateSnapshot ends a run whose snapshots are all complete and
// kept. Like finishRestore, it returns an error when --skip-errors // kept. Like finishRestore, it returns an error when --skip-errors
// skipped anything it could not read or list; otherwise it prints the // skipped anything it could not read or list; otherwise it prints the
@@ -307,10 +219,9 @@ func (v *Vaultik) snapshotNamesToCreate(requested []string) ([]string, error) {
} }
// runPostBackupPrune drops older snapshots of the given names and removes // runPostBackupPrune drops older snapshots of the given names and removes
// from remote storage the orphan blobs and the directories under // orphan blobs from remote storage. If keepNewerThan is set (e.g. "4w"),
// metadata/ with no manifest, as PruneBlobs does. If keepNewerThan is set // snapshots newer than that duration are kept. Otherwise only the latest
// (e.g. "4w"), snapshots newer than that duration are kept. Otherwise only // snapshot of each name is kept.
// the latest snapshot of each name is kept.
func (v *Vaultik) runPostBackupPrune( func (v *Vaultik) runPostBackupPrune(
snapshotNames []string, keepNewerThan string, snapshotNames []string, keepNewerThan string,
) error { ) error {
@@ -337,7 +248,7 @@ func (v *Vaultik) runPostBackupPrune(
err = v.PruneBlobs(&PruneOptions{Force: true}) err = v.PruneBlobs(&PruneOptions{Force: true})
if err != nil { if err != nil {
return fmt.Errorf("pruning the destination store: %w", err) return fmt.Errorf("pruning orphaned blobs: %w", err)
} }
return nil return nil
@@ -363,13 +274,6 @@ type snapshotStats struct {
// could not be read or listed. // could not be read or listed.
totalFilesUnreadable int totalFilesUnreadable int
// The configured paths in which the scan found nothing.
emptyPaths []string
// Under --prune, the directories where an earlier snapshot of the
// same name found a volume mounted and this one did not.
unmountedPaths []string
// The sizes of all blobs the snapshot references, set by // The sizes of all blobs the snapshot references, set by
// finalizeSnapshotMetadata once snapshot_blobs is populated. // finalizeSnapshotMetadata once snapshot_blobs is populated.
blobSize int64 blobSize int64
@@ -377,10 +281,11 @@ type snapshotStats struct {
} }
// createNamedSnapshot creates a single named snapshot and returns the // createNamedSnapshot creates a single named snapshot and returns the
// statistics of its scan. // number of entries --skip-errors skipped because it could not read or
// list them.
func (v *Vaultik) createNamedSnapshot( func (v *Vaultik) createNamedSnapshot(
opts *SnapshotCreateOptions, hostname, snapName string, idx, total int, opts *SnapshotCreateOptions, hostname, snapName string, idx, total int,
) (*snapshotStats, error) { ) (int, error) {
snapshotStartTime := time.Now() snapshotStartTime := time.Now()
if total > 1 { if total > 1 {
@@ -389,12 +294,12 @@ func (v *Vaultik) createNamedSnapshot(
resolvedDirs, err := v.resolveSnapshotPaths(snapName) resolvedDirs, err := v.resolveSnapshotPaths(snapName)
if err != nil { if err != nil {
return nil, err return 0, err
} }
previousSnapshotID, err := v.previousSnapshotID(snapName) previousSnapshotID, err := v.previousSnapshotID(snapName)
if err != nil { if err != nil {
return nil, err return 0, err
} }
scanner := v.ScannerFactory(snapshot.ScannerParams{ scanner := v.ScannerFactory(snapshot.ScannerParams{
@@ -409,7 +314,7 @@ func (v *Vaultik) createNamedSnapshot(
snapshotID, err := v.SnapshotManager.CreateSnapshotWithName( snapshotID, err := v.SnapshotManager.CreateSnapshotWithName(
v.ctx, hostname, snapName, v.Globals.Version, v.Globals.Commit) v.ctx, hostname, snapName, v.Globals.Version, v.Globals.Commit)
if err != nil { if err != nil {
return nil, fmt.Errorf("creating snapshot: %w", err) return 0, fmt.Errorf("creating snapshot: %w", err)
} }
log.Info("Beginning snapshot", "snapshot_id", snapshotID, "name", snapName) log.Info("Beginning snapshot", "snapshot_id", snapshotID, "name", snapName)
@@ -417,12 +322,12 @@ func (v *Vaultik) createNamedSnapshot(
stats, err := v.scanAllDirectories(scanner, resolvedDirs, snapshotID) stats, err := v.scanAllDirectories(scanner, resolvedDirs, snapshotID)
if err != nil { if err != nil {
return nil, err return 0, err
} }
err = v.finalizeSnapshotMetadata(snapshotID, stats) err = v.finalizeSnapshotMetadata(snapshotID, stats)
if err != nil { if err != nil {
return nil, err return 0, err
} }
log.Info("Snapshot complete", log.Info("Snapshot complete",
@@ -435,55 +340,7 @@ func (v *Vaultik) createNamedSnapshot(
v.printSnapshotSummary(snapshotID, snapshotStartTime, stats) v.printSnapshotSummary(snapshotID, snapshotStartTime, stats)
if opts.Prune { return stats.totalFilesUnreadable, nil
stats.unmountedPaths, err = v.findUnmountedPaths(snapName, snapshotID)
if err != nil {
return nil, err
}
}
return stats, nil
}
// findUnmountedPaths returns each directory that an earlier snapshot
// named snapName in the local index recorded as a mount point and
// snapshot snapshotID did not, as when the volume mounted there is not
// mounted.
func (v *Vaultik) findUnmountedPaths(
snapName, snapshotID string,
) ([]string, error) {
found, err := v.Repositories.Snapshots.GetMountPoints(v.ctx, snapshotID)
if err != nil {
return nil, err
}
snapshots, err := v.Repositories.Snapshots.ListRecent(v.ctx, listRecentLimit)
if err != nil {
return nil, fmt.Errorf("listing snapshots: %w", err)
}
var unmounted []string
for _, s := range snapshots {
if s.ID.String() == snapshotID ||
parseSnapshotName(s.ID.String(), s.Hostname.String()) != snapName {
continue
}
recorded, err := v.Repositories.Snapshots.GetMountPoints(
v.ctx, s.ID.String())
if err != nil {
return nil, err
}
for _, dir := range recorded {
if !slices.Contains(found, dir) && !slices.Contains(unmounted, dir) {
unmounted = append(unmounted, dir)
}
}
}
return unmounted, nil
} }
// previousSnapshotID returns the ID of the most recent completed snapshot // previousSnapshotID returns the ID of the most recent completed snapshot
@@ -505,8 +362,7 @@ func (v *Vaultik) previousSnapshotID(snapName string) (string, error) {
} }
// resolveSnapshotPaths resolves source directories to absolute paths // resolveSnapshotPaths resolves source directories to absolute paths
// with symlink resolution. A path that is a symlink whose target does // with symlink resolution.
// not exist is an error.
func (v *Vaultik) resolveSnapshotPaths(snapName string) ([]string, error) { func (v *Vaultik) resolveSnapshotPaths(snapName string) ([]string, error) {
snapConfig := v.Config.Snapshots[snapName] snapConfig := v.Config.Snapshots[snapName]
resolvedDirs := make([]string, 0, len(snapConfig.Paths)) resolvedDirs := make([]string, 0, len(snapConfig.Paths))
@@ -519,18 +375,12 @@ func (v *Vaultik) resolveSnapshotPaths(snapName string) ([]string, error) {
resolvedPath, err := filepath.EvalSymlinks(absPath) resolvedPath, err := filepath.EvalSymlinks(absPath)
if err != nil { if err != nil {
// A path that does not exist is left for the scan to report. if os.IsNotExist(err) {
// One that exists is a symlink whose target is gone, as one resolvedPath = absPath
// into a volume that is not mounted: the scan would record } else {
// only the link, and --prune would then drop the older
// snapshots that hold the files.
_, lstatErr := os.Lstat(absPath)
if !os.IsNotExist(err) || lstatErr == nil {
return nil, fmt.Errorf( return nil, fmt.Errorf(
"failed to resolve symlinks for %s: %w", absPath, err) "failed to resolve symlinks for %s: %w", absPath, err)
} }
resolvedPath = absPath
} }
resolvedDirs = append(resolvedDirs, resolvedPath) resolvedDirs = append(resolvedDirs, resolvedPath)
@@ -584,10 +434,6 @@ func (v *Vaultik) scanAllDirectories(
stats.totalBytesUploaded += result.BytesUploaded stats.totalBytesUploaded += result.BytesUploaded
stats.uploadDuration += result.UploadDuration stats.uploadDuration += result.UploadDuration
if result.EntriesFound == 0 {
stats.emptyPaths = append(stats.emptyPaths, dir)
}
log.Info("Directory scan complete", log.Info("Directory scan complete",
"path", dir, "path", dir,
"files", result.FilesScanned, "files", result.FilesScanned,
@@ -1455,7 +1301,7 @@ const pruneCommandHint = "vaultik prune"
// snapshotRemoveCommandHint is the command suggested, with the // snapshotRemoveCommandHint is the command suggested, with the
// snapshot's ID, when a remove could not reach the destination store: // snapshot's ID, when a remove could not reach the destination store:
// running it again removes the snapshot's metadata there, which // running it again removes the snapshot's metadata there, which
// `vaultik prune` never does while the manifest is there. // `vaultik prune` never does.
const snapshotRemoveCommandHint = "vaultik snapshot remove" const snapshotRemoveCommandHint = "vaultik snapshot remove"
// RemoveSnapshot removes a snapshot from the local index database and, // RemoveSnapshot removes a snapshot from the local index database and,
@@ -1658,8 +1504,7 @@ func (v *Vaultik) confirmRemoveSnapshot(snapshotID string, opts *RemoveOptions)
// the local index, its removal there has already happened, so a failure // the local index, its removal there has already happened, so a failure
// is only a warning telling the user to run `vaultik snapshot remove` // is only a warning telling the user to run `vaultik snapshot remove`
// for the snapshot again once the destination store is reachable // for the snapshot again once the destination store is reachable
// (`vaultik prune` never removes snapshot metadata that has its // (`vaultik prune` never removes snapshot metadata).
// manifest).
func (v *Vaultik) removeSnapshotRemote( func (v *Vaultik) removeSnapshotRemote(
snapshotID string, inLocalIndex bool, markerErr error, snapshotID string, inLocalIndex bool, markerErr error,
opts *RemoveOptions, opts *RemoveOptions,
-386
View File
@@ -1,18 +1,8 @@
package vaultik_test package vaultik_test
import ( import (
"context"
"os"
"path/filepath"
"syscall"
"testing" "testing"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/snapshot"
"sneak.berlin/go/vaultik/internal/vaultik" "sneak.berlin/go/vaultik/internal/vaultik"
) )
@@ -35,379 +25,3 @@ func TestSnapshotCreateOptions_PruneFlag(t *testing.T) {
t.Error("Expected Prune to be false") t.Error("Expected Prune to be false")
} }
} }
// A backup under --prune that finds nothing in a configured path, as in
// the mount point of a volume that is not mounted, does not prune: run
// after run, the snapshot that holds the files is kept with everything
// it stored, and the run fails. That holds whether the empty path is the
// snapshot's only path or one of two. See
// https://git.eeqj.de/sneak/vaultik/issues/388.
func TestBackupWithEmptySourcePathDoesNotPrune(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
// emptied is the path, relative to the source directory, that is
// emptied after the first backup.
const (
snapshotName = "media"
emptied = "music"
)
tests := []struct {
name string
paths []string // relative to the source directory
}{
{name: "only path", paths: []string{emptied}},
{name: "one of two paths", paths: []string{emptied, "docs"}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
ctx := context.Background()
env := newDeletedCountEnv(t, map[string][]byte{
emptied + "/a.mp3": []byte("song a"),
emptied + "/b.mp3": []byte("song b"),
"docs/notes.txt": []byte("notes"),
})
paths := make([]string, 0, len(tt.paths))
for _, rel := range tt.paths {
paths = append(paths, filepath.Join(env.src, rel))
}
env.v.Config.Snapshots[snapshotName] = config.SnapshotConfig{
Paths: paths,
}
env.backUp(t, snapshotName)
firstID := localSnapshotID(ctx, t, env.v.Repositories, snapshotName)
stored, err := env.v.Storage.List(ctx, "")
require.NoError(t, err)
// A volume that is not mounted leaves its mount point behind
// as an empty directory.
emptiedDir := filepath.Join(env.src, emptied)
require.NoError(t, os.RemoveAll(emptiedDir))
require.NoError(t, os.Mkdir(emptiedDir, 0o750))
for range 2 {
env.out.Reset()
err := env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
Prune: true,
Snapshots: []string{snapshotName},
})
require.ErrorIs(t, err, vaultik.ErrSourcePathEmpty)
assert.Contains(t, env.out.String(),
"Skipped the post-backup prune because nothing was found in ")
assert.Contains(t, env.out.String(),
emptied+" of snapshot "+snapshotName+";")
assert.NotContains(t, env.out.String(), "Running post-backup prune.")
}
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
ctx, listRecentTestLimit)
require.NoError(t, err)
require.Len(t, snapshots, 3)
assert.Equal(t, firstID, snapshots[2].ID.String(),
"the snapshot that holds the files is kept")
for _, key := range stored {
_, err := env.v.Storage.Stat(ctx, key)
require.NoError(t, err, "%s is kept", key)
}
})
}
}
// A backup under --prune of a configured path that is a symlink whose
// target is gone, as one into a volume that is not mounted, fails before
// anything is pruned: the snapshot that holds the files is kept with
// everything it stored. See https://git.eeqj.de/sneak/vaultik/issues/388.
func TestBackupThroughSymlinkWhoseTargetIsGoneDoesNotPrune(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
const snapshotName = "photos"
ctx := context.Background()
env := newDeletedCountEnv(t, map[string][]byte{
"volume/a.jpg": []byte("photo a"),
"volume/b.jpg": []byte("photo b"),
})
target := filepath.Join(env.src, "volume")
link := filepath.Join(env.src, "photos")
require.NoError(t, os.Symlink(target, link))
env.setSnapshot(snapshotName, "photos")
env.backUp(t, snapshotName)
firstID := localSnapshotID(ctx, t, env.v.Repositories, snapshotName)
stored, err := env.v.Storage.List(ctx, "")
require.NoError(t, err)
// The volume is not mounted, so the symlink points at nothing.
require.NoError(t, os.RemoveAll(target))
env.out.Reset()
err = env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
Prune: true,
Snapshots: []string{snapshotName},
})
require.ErrorIs(t, err, os.ErrNotExist)
assert.Contains(t, err.Error(), link)
assert.NotContains(t, env.out.String(), "Running post-backup prune.")
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
ctx, listRecentTestLimit)
require.NoError(t, err)
require.Len(t, snapshots, 1)
assert.Equal(t, firstID, snapshots[0].ID.String(),
"the snapshot that holds the files is kept")
for _, key := range stored {
_, err := env.v.Storage.Stat(ctx, key)
require.NoError(t, err, "%s is kept", key)
}
}
// A backup under --skip-errors --prune that both skipped a file it could
// not read and found nothing in a configured path names the empty path
// too when it skips the prune. See
// https://git.eeqj.de/sneak/vaultik/issues/388.
func TestBackupWithEmptySourcePathAndUnreadableFileNamesBoth(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
const snapshotName = "media"
env := newDeletedCountEnv(t, map[string][]byte{
"docs/notes.txt": []byte("notes"),
})
require.NoError(t, os.Mkdir(filepath.Join(env.src, "music"), 0o750))
env.v.Config.Snapshots[snapshotName] = config.SnapshotConfig{
Paths: []string{
filepath.Join(env.src, "docs"),
filepath.Join(env.src, "music"),
},
}
// A mode does not lock out root, who runs the tests, so opening the
// file is made to fail instead.
env.v.Fs = &openFailFs{
path: filepath.Join(env.src, "docs", "notes.txt"),
err: os.ErrPermission,
}
err := env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
Prune: true,
SkipErrors: true,
Snapshots: []string{snapshotName},
})
require.ErrorIs(t, err, vaultik.ErrFilesUnreadable)
assert.Contains(t, env.out.String(),
"Skipped the post-backup prune because 1 file(s) could not be read")
assert.Contains(t, env.out.String(),
"Skipped the post-backup prune because nothing was found in ")
assert.Contains(t, env.out.String(), "music of snapshot "+snapshotName+";")
assert.NotContains(t, env.out.String(), "Running post-backup prune.")
}
// A backup under --prune that finds something in each configured path
// prunes the previous snapshot, also when a path is a single file. See
// https://git.eeqj.de/sneak/vaultik/issues/388.
func TestBackupWithNoEmptySourcePathPrunes(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
const (
snapshotName = "album"
file = "pictures/one.jpg" // relative to the source directory
)
tests := []struct {
name string
path string // relative to the source directory
}{
{name: "directory", path: filepath.Dir(file)},
{name: "single file", path: file},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
ctx := context.Background()
env := newDeletedCountEnv(t, map[string][]byte{
file: []byte("picture"),
})
env.setSnapshot(snapshotName, tt.path)
env.backUp(t, snapshotName)
firstID := localSnapshotID(ctx, t, env.v.Repositories, snapshotName)
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
Prune: true,
Snapshots: []string{snapshotName},
}))
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
ctx, listRecentTestLimit)
require.NoError(t, err)
require.Len(t, snapshots, 1, "the previous snapshot is pruned")
assert.NotEqual(t, firstID, snapshots[0].ID.String())
})
}
}
// automountFs is the real filesystem with a volume at volumeDir that is
// mounted on first access, as by an autofs map that shows its mount
// points before mounting them: listing volumeDir mounts the volume or,
// while cannotMount, fails with "no such file or directory".
type automountFs struct {
afero.OsFs
volumeDir string
cannotMount bool
mounted bool
}
//nolint:ireturn // afero.Fs.Open is defined to return the interface.
func (f *automountFs) Open(name string) (afero.File, error) {
if name == f.volumeDir && f.cannotMount {
return nil, &os.PathError{Op: "open", Path: name, Err: syscall.ENOENT}
}
if name == f.volumeDir {
f.mounted = true
}
return f.OsFs.Open(name)
}
// A backup under --prune in which a volume that an earlier snapshot found
// mounted below the configured path is missing does not prune: run after
// run, the snapshot that holds the volume's files is kept with everything
// it stored, and the run fails. Once the volume is back, the prune runs.
// The volume is mounted on first access and every backup starts with it
// not mounted, so its directory is on its parent's device until the walk
// lists it. While the volume is missing, its directory is left empty, or
// listing it fails with "no such file or directory", as when an autofs
// volume cannot be mounted, which the walk counts as a directory removed
// during the backup. See https://git.eeqj.de/sneak/vaultik/issues/395.
func TestBackupWithVolumeNotMountedDoesNotPrune(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
tests := []struct {
name string
listingFails bool
}{
{name: "directory left empty", listingFails: false},
{name: "listing fails", listingFails: true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
backUpWithVolumeNotMounted(t, tt.listingFails)
})
}
}
// backUpWithVolumeNotMounted runs the backups of
// TestBackupWithVolumeNotMountedDoesNotPrune. While the volume is
// missing, listing its directory fails if listingFails; otherwise the
// directory is empty.
func backUpWithVolumeNotMounted(t *testing.T, listingFails bool) {
t.Helper()
// volume is the directory, below the configured path, where the
// volume is mounted.
const (
snapshotName = "data"
volume = "nas"
)
ctx := context.Background()
env := newDeletedCountEnv(t, map[string][]byte{
volume + "/a.txt": []byte("file a"),
"notes.txt": []byte("notes"),
})
env.setSnapshot(snapshotName, "")
prune := &vaultik.SnapshotCreateOptions{
Prune: true, Snapshots: []string{snapshotName},
}
// The volume's directory is on the volume's device, 2, while the
// volume is mounted. Everything else is on device 1.
volumeDir := filepath.Join(env.src, volume)
fs := &automountFs{volumeDir: volumeDir}
newScanner := env.v.ScannerFactory
env.v.ScannerFactory = func(params snapshot.ScannerParams) *snapshot.Scanner {
fs.mounted = false
params.Fs = fs
params.DeviceNumber = func(info os.FileInfo) (uint64, bool) {
if info.Name() == volume && fs.mounted {
return 2, true
}
return 1, true
}
return newScanner(params)
}
env.backUp(t, snapshotName)
firstID := localSnapshotID(ctx, t, env.v.Repositories, snapshotName)
stored, err := env.v.Storage.List(ctx, "")
require.NoError(t, err)
// The volume goes missing.
fs.cannotMount = listingFails
if !listingFails {
require.NoError(t, os.RemoveAll(volumeDir))
require.NoError(t, os.Mkdir(volumeDir, 0o750))
}
for range 2 {
env.out.Reset()
err := env.v.CreateSnapshot(prune)
require.ErrorIs(t, err, vaultik.ErrVolumeNotMounted)
assert.Contains(t, env.out.String(),
"did not find a mount point at "+volumeDir+" of snapshot "+snapshotName)
assert.Contains(t, env.out.String(), "'vaultik snapshot purge'")
assert.NotContains(t, env.out.String(), "Running post-backup prune.")
}
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
ctx, listRecentTestLimit)
require.NoError(t, err)
require.Len(t, snapshots, 3)
assert.Equal(t, firstID, snapshots[2].ID.String(),
"the snapshot that holds the volume's files is kept")
for _, key := range stored {
_, err := env.v.Storage.Stat(ctx, key)
require.NoError(t, err, "%s is kept", key)
}
// The volume is back.
fs.cannotMount = false
require.NoError(t, os.WriteFile(
filepath.Join(volumeDir, "a.txt"), []byte("file a"), 0o600))
require.NoError(t, env.v.CreateSnapshot(prune))
snapshots, err = env.v.Repositories.Snapshots.ListRecent(
ctx, listRecentTestLimit)
require.NoError(t, err)
require.Len(t, snapshots, 1, "the older snapshots are pruned")
}