Compare commits
3
Commits
v1.0.0-rc1
..
next
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e9b17cc5fd | ||
|
|
850e2de25a | ||
|
|
6a57750a7b |
+15
-10
@@ -312,7 +312,7 @@ Repository interfaces:
|
|||||||
```
|
```
|
||||||
CreateSnapshot(opts)
|
CreateSnapshot(opts)
|
||||||
│
|
│
|
||||||
├─► markDestination() // Refuse a destination without the marker, then write it
|
├─► markDestination() // Check the marker (see Storage Layout in S3), then write it
|
||||||
│
|
│
|
||||||
├─► PruneDatabase() // Critical: avoid dedup errors
|
├─► PruneDatabase() // Critical: avoid dedup errors
|
||||||
│
|
│
|
||||||
@@ -382,15 +382,20 @@ bucket/
|
|||||||
└── manifest.json.zst # Blob list (for pruning/verification)
|
└── manifest.json.zst # Blob list (for pruning/verification)
|
||||||
```
|
```
|
||||||
|
|
||||||
A destination without `vaultik-destination.txt` is treated as not present
|
A destination that answers a look-up of `vaultik-destination.txt` with not
|
||||||
while the local index holds snapshots stored there (`checkDestinationMarker`
|
found is treated as not present while the local index holds snapshots stored
|
||||||
in `internal/vaultik/destination_marker.go`): `snapshot create` refuses to
|
there (`checkDestinationMarker` in `internal/vaultik/destination_marker.go`):
|
||||||
write to it, `prune`, `snapshot purge` and `remote nuke` fail, and
|
`snapshot create` refuses to write to it, `prune`, `snapshot purge` and
|
||||||
`snapshot list` and `snapshot remove` behave as for a destination that cannot
|
`remote nuke` fail, and `snapshot list` and `snapshot remove` behave as for a
|
||||||
be listed. This keeps those commands from taking an unplugged volume's empty
|
destination that cannot be listed. This keeps those commands from taking an
|
||||||
mount point for a destination whose snapshots are all gone. The `file://`
|
unplugged volume's empty mount point for a destination whose snapshots are
|
||||||
backend never creates its destination directory, so a write to a missing one
|
all gone. When the look-up fails for any reason other than not found, for
|
||||||
fails.
|
example when S3 refuses it to credentials that allow only PUT, the check is
|
||||||
|
skipped, so a host with such credentials does not notice an emptied
|
||||||
|
destination; see
|
||||||
|
[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#marker-file-vaultik-destinationtxt).
|
||||||
|
The `file://` backend never creates its destination directory, so a write to a
|
||||||
|
missing one fails.
|
||||||
|
|
||||||
The `{remote-key}` directory name is a one-way double SHA-256 hash of the human
|
The `{remote-key}` directory name is a one-way double SHA-256 hash of the human
|
||||||
snapshot ID, so the human ID (hostname, snapshot name, timestamp) is never
|
snapshot ID, so the human ID (hostname, snapshot name, timestamp) is never
|
||||||
|
|||||||
@@ -96,6 +96,14 @@ vaultik snapshot restore <snapshot-id> ~/vaultik-restore
|
|||||||
# 0 3 * * * vaultik snapshot create --cron --prune --keep-newer-than 4w
|
# 0 3 * * * vaultik snapshot create --cron --prune --keep-newer-than 4w
|
||||||
```
|
```
|
||||||
|
|
||||||
|
On a destination shared between hosts or users, the prune in that cron line
|
||||||
|
relies on `prune_grace_period` (default `24h`) to leave the blobs of a backup
|
||||||
|
another one is still running, so it must be longer than the longest backup
|
||||||
|
run. It does not protect a blob that backup reuses from an earlier run that
|
||||||
|
stopped before its manifest: the prune can delete such a blob, and the new
|
||||||
|
snapshot then cannot be restored. To not depend on the grace period, give each
|
||||||
|
host its own prefix or directory. See [locking](#locking).
|
||||||
|
|
||||||
## restoring on another machine
|
## restoring on another machine
|
||||||
|
|
||||||
Restoring on a host that never ran the backup — a replacement machine
|
Restoring on a host that never ran the backup — a replacement machine
|
||||||
@@ -192,7 +200,22 @@ process-wide lock at `$XDG_DATA_HOME/vaultik/vaultik.pid`
|
|||||||
(`~/.local/share/vaultik/vaultik.pid` on Linux) for the whole run. Only
|
(`~/.local/share/vaultik/vaultik.pid` on Linux) for the whole run. Only
|
||||||
one of them runs at a time: a second one exits immediately with an
|
one of them runs at a time: a second one exits immediately with an
|
||||||
"already running" error rather than waiting, so two writers can never
|
"already running" error rather than waiting, so two writers can never
|
||||||
corrupt the local index or the destination store.
|
corrupt the local index.
|
||||||
|
|
||||||
|
The lock covers one user on one machine. It does not stop another user,
|
||||||
|
or another host, from writing to the same destination store at the same
|
||||||
|
time. A backup uploads its blobs before the manifest that lists them, so a
|
||||||
|
prune from elsewhere at that moment finds blobs no snapshot lists. It
|
||||||
|
deletes such a blob only once it is older than `prune_grace_period`
|
||||||
|
(default `24h`), which therefore must be longer than the longest backup run
|
||||||
|
of any host or user sharing the destination. The grace period does not
|
||||||
|
protect a blob a backup reuses from an earlier run that stopped before its
|
||||||
|
manifest: a backup skips the upload of a blob already on the destination, so
|
||||||
|
that blob keeps the time of its first upload, and a prune from elsewhere
|
||||||
|
during the backup can delete it; the new snapshot then cannot be restored.
|
||||||
|
`remote nuke` deletes everything regardless. To not depend on the grace
|
||||||
|
period, give each host or user its own prefix (or directory) on the
|
||||||
|
destination.
|
||||||
|
|
||||||
Read-only commands — `info`, `snapshot list`, `snapshot verify`, and
|
Read-only commands — `info`, `snapshot list`, `snapshot verify`, and
|
||||||
`remote info` — do not take the lock and are never blocked, so they run
|
`remote info` — do not take the lock and are never blocked, so they run
|
||||||
@@ -296,20 +319,31 @@ value where a list belongs or a word where a number belongs), is refused.
|
|||||||
* `--cron`: Silent on total success; warnings and errors are still printed
|
* `--cron`: Silent on total success; warnings and errors are still printed
|
||||||
(for crontab)
|
(for crontab)
|
||||||
* `--prune`: After backup, drop older snapshots of each backed-up name and
|
* `--prune`: After backup, drop older snapshots of each backed-up name and
|
||||||
remove orphaned blobs from remote storage. By default keeps only the latest
|
remove orphaned blobs older than `prune_grace_period` from remote storage,
|
||||||
|
as `prune` does. By default keeps only the latest
|
||||||
snapshot per name; use `--keep-newer-than` for a rolling window. When
|
snapshot per name; use `--keep-newer-than` for a rolling window. When
|
||||||
`--skip-errors` skipped a file or directory it could not read or list, the
|
`--skip-errors` skipped a file or directory it could not read or list, the
|
||||||
prune does not run and every older snapshot is kept, since one of them may
|
prune does not run and every older snapshot is kept, since one of them may
|
||||||
be the only one that holds what was skipped; the command says so and exits
|
be the only one that holds what was skipped; the command says so and exits
|
||||||
non-zero. A file removed during the backup does not stop the prune.
|
non-zero. When the backup found nothing in a configured path, as in the
|
||||||
|
mount point of a volume that is not mounted, the prune does not run
|
||||||
|
either: every older snapshot is kept, and the command names the path and
|
||||||
|
exits non-zero, on every run until the path holds something again. Take a
|
||||||
|
path that is empty on purpose out of the config, or remove the older
|
||||||
|
snapshots by hand with `snapshot purge`. A file removed during the backup
|
||||||
|
does not stop the prune.
|
||||||
* `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than
|
* `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than
|
||||||
this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`).
|
this duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`).
|
||||||
Without `--prune`, or with a value that is not a valid duration, the
|
Without `--prune`, or with a value that is not a valid duration, the
|
||||||
command fails before the backup starts.
|
command fails before the backup starts.
|
||||||
* Fails before it writes anything when a `file://` destination directory
|
* Fails before it writes anything when a `file://` destination directory
|
||||||
does not exist, or when the local index holds snapshots and the
|
does not exist, or when the local index holds snapshots and the
|
||||||
destination has no `vaultik-destination.txt` (see
|
destination answers that it has no `vaultik-destination.txt`. When that
|
||||||
|
look-up fails for any reason other than not found, for example when S3
|
||||||
|
refuses it to credentials that allow only PUT, the check is skipped (see
|
||||||
[remote storage layout](#remote-storage-layout))
|
[remote storage layout](#remote-storage-layout))
|
||||||
|
* Fails, before anything is pruned, when a configured path is a symlink
|
||||||
|
whose target does not exist, as one into a volume that is not mounted
|
||||||
|
|
||||||
**`snapshot list`**: Show every snapshot known to this host — the union
|
**`snapshot list`**: Show every snapshot known to this host — the union
|
||||||
of the local index and the backup destination store — with timestamps
|
of the local index and the backup destination store — with timestamps
|
||||||
@@ -344,9 +378,9 @@ gives: revert the configuration, or run `vaultik database delete`.
|
|||||||
If the destination store cannot be listed (unmounted volume,
|
If the destination store cannot be listed (unmounted volume,
|
||||||
permission denied, network down), the command warns, falls back to the
|
permission denied, network down), the command warns, falls back to the
|
||||||
local index alone, and still exits zero. It does the same when the
|
local index alone, and still exits zero. It does the same when the
|
||||||
destination has no `vaultik-destination.txt` while the local index
|
destination answers that it has no `vaultik-destination.txt` while the
|
||||||
holds snapshots stored there, as when a volume mounted at the
|
local index holds snapshots stored there, as when a volume mounted at
|
||||||
destination is unplugged and leaves its empty mount point behind.
|
the destination is unplugged and leaves its empty mount point behind.
|
||||||
* `--json`: Output in JSON format. Each entry carries `locally_tracked`
|
* `--json`: Output in JSON format. Each entry carries `locally_tracked`
|
||||||
(whether the snapshot is in the local index), `remote_key` (the full
|
(whether the snapshot is in the local index), `remote_key` (the full
|
||||||
64-character storage key), and `remote_present` (whether it was seen
|
64-character storage key), and `remote_present` (whether it was seen
|
||||||
@@ -389,8 +423,8 @@ with `--local-only`, the snapshot must be in the local index. Local row
|
|||||||
cleanup (files, chunks, blobs the snapshot was the last referrer for)
|
cleanup (files, chunks, blobs the snapshot was the last referrer for)
|
||||||
runs automatically. If removing the snapshot's metadata from the
|
runs automatically. If removing the snapshot's metadata from the
|
||||||
destination store fails (for example because the store is unreachable,
|
destination store fails (for example because the store is unreachable,
|
||||||
or the local index holds snapshots and the destination has no
|
or the local index holds snapshots and the destination answers that it
|
||||||
`vaultik-destination.txt`), the local-DB removal still completes and a
|
has no `vaultik-destination.txt`), the local-DB removal still completes and a
|
||||||
warning is emitted; run `vaultik snapshot remove <snapshot-id>` again
|
warning is emitted; run `vaultik snapshot remove <snapshot-id>` again
|
||||||
once the store is reachable to remove the snapshot's metadata from it
|
once the store is reachable to remove the snapshot's metadata from it
|
||||||
(`vaultik prune` does not). For a snapshot that is only on the
|
(`vaultik prune` does not). For a snapshot that is only on the
|
||||||
@@ -423,19 +457,28 @@ local snapshot whose remote metadata is missing is dropped from the
|
|||||||
local index; (2) delete orphaned local rows (files, chunks, blobs no
|
local index; (2) delete orphaned local rows (files, chunks, blobs no
|
||||||
longer referenced by any completed snapshot); (3) list every remote
|
longer referenced by any completed snapshot); (3) list every remote
|
||||||
manifest on the destination store to compute the still-referenced blob
|
manifest on the destination store to compute the still-referenced blob
|
||||||
set and delete any blob not in that set. Step (3) reads all remote
|
set and delete any blob not in that set that is older than
|
||||||
manifests — network cost scales with the number of snapshots. `snapshot
|
`prune_grace_period` (default `24h`). A newer one is kept and counted: it
|
||||||
create --prune` runs the same cleanup automatically; this is the
|
may belong to a backup still running, on this host or another sharing the
|
||||||
|
destination store, that has not yet uploaded its manifest. The grace period
|
||||||
|
does not protect a blob such a backup reuses from an earlier run that stopped
|
||||||
|
before its manifest: that blob keeps the time of its first upload, so it can
|
||||||
|
be deleted. Step (3) reads all remote manifests — network cost scales with
|
||||||
|
the number of snapshots.
|
||||||
|
`snapshot create --prune` runs the same cleanup automatically; this is the
|
||||||
manual entry point for the same work. If the destination store cannot
|
manual entry point for the same work. If the destination store cannot
|
||||||
be listed, or the local index holds snapshots and the destination has
|
be listed, or the local index holds snapshots and the destination answers
|
||||||
no `vaultik-destination.txt` (an unplugged volume's empty mount point),
|
that it has no `vaultik-destination.txt` (an unplugged volume's empty mount
|
||||||
prune fails and deletes nothing. `snapshot purge` does the same. If a
|
point), prune fails and deletes nothing. `snapshot purge` does the same. If a
|
||||||
blob cannot be deleted, the command (and `snapshot create --prune`)
|
blob cannot be deleted, the command (and `snapshot create --prune`)
|
||||||
exits non-zero.
|
exits non-zero.
|
||||||
* `--force`: Skip confirmation prompt
|
* `--force`: Skip confirmation prompt
|
||||||
* `--json`: Output stats as JSON. Also skips the confirmation prompt, as
|
* `--json`: Output stats as JSON. Also skips the confirmation prompt, as
|
||||||
`--force` does. A blob that cannot be deleted is counted in
|
`--force` does. A blob that cannot be deleted is counted in
|
||||||
`blobs_failed`, and the document is printed before the non-zero exit.
|
`blobs_failed`, and the document is printed before the non-zero exit.
|
||||||
|
A blob kept for being newer than `prune_grace_period`, or because the
|
||||||
|
listing of the destination gives no time for it, is counted in
|
||||||
|
`blobs_too_new` and not in `blobs_found`.
|
||||||
|
|
||||||
**`info`**: Display system configuration, storage settings, encryption
|
**`info`**: Display system configuration, storage settings, encryption
|
||||||
recipients, and local database statistics.
|
recipients, and local database statistics.
|
||||||
@@ -461,8 +504,8 @@ and irreversible. This is the single supported way to wipe the entire
|
|||||||
destination store. A snapshot is removed from the local index only once
|
destination store. A snapshot is removed from the local index only once
|
||||||
its metadata is deleted from the destination store. If the destination
|
its metadata is deleted from the destination store. If the destination
|
||||||
store cannot be listed, or the local index holds snapshots and the
|
store cannot be listed, or the local index holds snapshots and the
|
||||||
destination has no `vaultik-destination.txt` (an unplugged volume's
|
destination answers that it has no `vaultik-destination.txt` (an unplugged
|
||||||
empty mount point), the command fails before it removes anything. If a
|
volume's empty mount point), the command fails before it removes anything. If a
|
||||||
listing afterwards still finds anything under `metadata/` or `blobs/`,
|
listing afterwards still finds anything under `metadata/` or `blobs/`,
|
||||||
the command names each object left and exits non-zero; running it again
|
the command names each object left and exits non-zero; running it again
|
||||||
deletes what is left.
|
deletes what is left.
|
||||||
@@ -510,6 +553,14 @@ being written, a killed upload can leave a truncated object under its name,
|
|||||||
which a later backup takes for complete. A leftover `.partial` file is ignored
|
which a later backup takes for complete. A leftover `.partial` file is ignored
|
||||||
and can be deleted.
|
and can be deleted.
|
||||||
|
|
||||||
|
A destination shared between hosts or users, each with its own local index,
|
||||||
|
relies on `prune_grace_period` to keep one's prune from deleting the blobs of
|
||||||
|
another's backup still running, so it must be longer than the longest backup
|
||||||
|
run. It does not protect a blob that backup reuses from an earlier run that
|
||||||
|
stopped before its manifest, which the prune can delete. To not depend on the
|
||||||
|
grace period, give each host its own prefix (or directory). See
|
||||||
|
[locking](#locking).
|
||||||
|
|
||||||
Legacy S3 configuration via `s3.*` fields (endpoint, bucket, prefix, etc.) is
|
Legacy S3 configuration via `s3.*` fields (endpoint, bucket, prefix, etc.) is
|
||||||
still supported for backward compatibility. `storage_url` takes precedence if
|
still supported for backward compatibility. `storage_url` takes precedence if
|
||||||
both are set.
|
both are set.
|
||||||
@@ -536,15 +587,22 @@ For an implementation-level view of the internals — the data model, the
|
|||||||
```
|
```
|
||||||
|
|
||||||
* `vaultik-destination.txt` holds one fixed line of text, with no host
|
* `vaultik-destination.txt` holds one fixed line of text, with no host
|
||||||
name, path or time. A destination without it is treated as not
|
name, path or time. While the local index holds snapshots stored
|
||||||
present while the local index holds snapshots stored there:
|
there, a destination that answers a look-up of it with not found is
|
||||||
`snapshot create` refuses to write to it, `prune`, `snapshot purge`
|
treated as not present: `snapshot create` refuses to write to it,
|
||||||
and `remote nuke` fail, and `snapshot list` and `snapshot remove`
|
`prune`, `snapshot purge` and `remote nuke` fail, and `snapshot list`
|
||||||
behave as for a destination that cannot be listed. This keeps an
|
and `snapshot remove` behave as for a destination that cannot be
|
||||||
unplugged volume's empty mount point from being taken for a
|
listed. This keeps an unplugged volume's empty mount point from being
|
||||||
destination whose snapshots are all gone. Mount
|
taken for a destination whose snapshots are all gone. Mount
|
||||||
the volume if it is removable; if the destination was emptied on
|
the volume if it is removable; if the destination was emptied on
|
||||||
purpose, run `vaultik database delete` and back up again.
|
purpose, run `vaultik database delete` and back up again.
|
||||||
|
When the look-up fails for any reason other than not found, the check
|
||||||
|
is skipped and the command goes on. S3 refuses it on every run to
|
||||||
|
credentials that allow only PUT, so a host with such credentials does
|
||||||
|
not notice a destination emptied since its last backup, and its next
|
||||||
|
backup would leave out the data its local index records as already
|
||||||
|
stored. After the destination is emptied, run
|
||||||
|
`vaultik database delete` on such a host before it backs up again.
|
||||||
* Blobs are two-level directory sharded using the first 4 hex chars of the blob hash
|
* Blobs are two-level directory sharded using the first 4 hex chars of the blob hash
|
||||||
* `db.zst.age` is a binary SQLite database (zstd compressed, age encrypted)
|
* `db.zst.age` is a binary SQLite database (zstd compressed, age encrypted)
|
||||||
containing all file metadata, chunk mappings, and relationships for the snapshot
|
containing all file metadata, chunk mappings, and relationships for the snapshot
|
||||||
@@ -596,7 +654,8 @@ format does and does not protect.
|
|||||||
1. List all snapshot manifests
|
1. List all snapshot manifests
|
||||||
2. Build set of all referenced blob hashes
|
2. Build set of all referenced blob hashes
|
||||||
3. List all blobs in storage
|
3. List all blobs in storage
|
||||||
4. Delete any blob not in the referenced set
|
4. Delete any blob not in the referenced set that is older than
|
||||||
|
`prune_grace_period`
|
||||||
|
|
||||||
### chunking and deduplication
|
### chunking and deduplication
|
||||||
|
|
||||||
@@ -641,6 +700,7 @@ complete annotated example also lives in
|
|||||||
| `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
|
| `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
|
||||||
| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` |
|
| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` |
|
||||||
| `compression_level` | `3` | zstd compression level (1-19) |
|
| `compression_level` | `3` | zstd compression level (1-19) |
|
||||||
|
| `prune_grace_period` | `24h` | How old a blob that no snapshot lists must be before `prune` deletes it. A backup uploads its blobs before the manifest that lists them, so on a destination shared between hosts or users this must be longer than the longest backup run (see [locking](#locking)). It does not protect a blob a backup reuses from an earlier run that stopped before its manifest: that blob keeps the time of its first upload, so a prune can delete it. A duration such as `36h` or `90m`; `0s` deletes such blobs at once |
|
||||||
| `hostname` | system hostname | Hostname used in snapshot IDs |
|
| `hostname` | system hostname | Hostname used in snapshot IDs |
|
||||||
| `index_path` | platform data dir | Local SQLite index path |
|
| `index_path` | platform data dir | Local SQLite index path |
|
||||||
|
|
||||||
|
|||||||
@@ -24,6 +24,14 @@ owner's; a `v*` tag runs the release build in
|
|||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
|
- 2026-10-14: Stopped `snapshot create --prune` pruning after a backup
|
||||||
|
that found nothing in a configured path
|
||||||
|
([issue #388](https://git.eeqj.de/sneak/vaultik/issues/388)), as in
|
||||||
|
the mount point of a volume that is not mounted. Such a prune could
|
||||||
|
delete every snapshot still holding that path's files. A configured
|
||||||
|
path that is a symlink whose target is gone, as one into such a
|
||||||
|
volume, now fails the backup, with or without `--prune`.
|
||||||
|
|
||||||
- 2026-10-14: Stopped `snapshot create --prune` pruning after a backup
|
- 2026-10-14: Stopped `snapshot create --prune` pruning after a backup
|
||||||
that skipped files it could not read
|
that skipped files it could not read
|
||||||
([issue #382](https://git.eeqj.de/sneak/vaultik/issues/382)). Such a
|
([issue #382](https://git.eeqj.de/sneak/vaultik/issues/382)). Such a
|
||||||
|
|||||||
@@ -328,6 +328,18 @@ storage_url: "rclone://myremote/path/to/backups"
|
|||||||
# Higher = better compression but slower
|
# Higher = better compression but slower
|
||||||
# Default: 3
|
# Default: 3
|
||||||
compression_level: 5
|
compression_level: 5
|
||||||
|
|
||||||
|
# How old a blob that no snapshot lists must be before prune deletes it
|
||||||
|
# A backup uploads its blobs before the manifest that lists them, so on a
|
||||||
|
# destination shared between hosts or users this must be longer than the
|
||||||
|
# longest backup run
|
||||||
|
# It does not protect a blob a backup reuses from an earlier run that stopped
|
||||||
|
# before its manifest: that blob keeps the time of its first upload, so a
|
||||||
|
# prune can delete it
|
||||||
|
# Supports: 36h, 90m, etc.; 0s deletes them at once
|
||||||
|
# Default: 24h
|
||||||
|
#prune_grace_period: 24h
|
||||||
|
|
||||||
# Hostname to use in backup metadata
|
# Hostname to use in backup metadata
|
||||||
# Default: system hostname
|
# Default: system hostname
|
||||||
#hostname: myserver
|
#hostname: myserver
|
||||||
|
|||||||
+2
-1
@@ -228,7 +228,8 @@ The restore process doesn't use the local database. Instead:
|
|||||||
|
|
||||||
1. **Identify Unreferenced Blobs**
|
1. **Identify Unreferenced Blobs**
|
||||||
- Query blobs not referenced by any remaining snapshot
|
- Query blobs not referenced by any remaining snapshot
|
||||||
- Delete from S3 and local database
|
- Delete from S3, once older than `prune_grace_period`, and from the local
|
||||||
|
database
|
||||||
|
|
||||||
### 6. Incomplete Snapshot Cleanup
|
### 6. Incomplete Snapshot Cleanup
|
||||||
|
|
||||||
|
|||||||
@@ -27,7 +27,9 @@ The metadata subdirectory is named with the **remote key**, a one-way hash of th
|
|||||||
|
|
||||||
## Marker File (`vaultik-destination.txt`)
|
## Marker File (`vaultik-destination.txt`)
|
||||||
|
|
||||||
Every backup writes this file at the root before it uploads anything. It holds one fixed line of plain text and names no host, path or time. A destination without it is treated as not present while the local index holds snapshots stored there: `snapshot create` refuses to write to it, and `prune`, `snapshot purge`, `snapshot remove`, `snapshot list` and `remote nuke` behave as for a destination that cannot be listed. An unplugged volume leaves an empty directory at its mount point, and without the marker that directory would read as a destination whose snapshots are all gone. `remote nuke` deletes the marker last.
|
Every backup writes this file at the root before it uploads anything. It holds one fixed line of plain text and names no host, path or time. While the local index holds snapshots stored there, a destination that answers a look-up of the marker with not found is treated as not present: `snapshot create` refuses to write to it, and `prune`, `snapshot purge`, `snapshot remove`, `snapshot list` and `remote nuke` behave as for a destination that cannot be listed. An unplugged volume leaves an empty directory at its mount point, and without the marker that directory would read as a destination whose snapshots are all gone. `remote nuke` deletes the marker last.
|
||||||
|
|
||||||
|
When the look-up of the marker fails for any reason other than not found, the check is skipped and the command goes on. S3 refuses it on every run to credentials that allow only PUT, so a host with such credentials does not notice a destination emptied since its last backup, and its next backup would leave out the data its local index records as already stored. After the destination is emptied, run `vaultik database delete` on such a host before it backs up again.
|
||||||
|
|
||||||
## Blobs Directory (`blobs/`)
|
## Blobs Directory (`blobs/`)
|
||||||
|
|
||||||
@@ -166,7 +168,7 @@ The prune operation is safe because:
|
|||||||
1. It keeps every blob listed in any snapshot's manifest and deletes only blobs that no manifest references
|
1. It keeps every blob listed in any snapshot's manifest and deletes only blobs that no manifest references
|
||||||
2. Manifests are unencrypted and can be read without keys
|
2. Manifests are unencrypted and can be read without keys
|
||||||
3. If any manifest cannot be downloaded or decoded, prune deletes nothing and exits with an error, rather than treating that snapshot's blobs as unreferenced
|
3. If any manifest cannot be downloaded or decoded, prune deletes nothing and exits with an error, rather than treating that snapshot's blobs as unreferenced
|
||||||
4. Prune requires exclusive access to the destination: running it during a concurrent backup can race a snapshot whose manifest is not yet written, so do not prune while a backup is in progress
|
4. It deletes an unreferenced blob only once the blob is older than `prune_grace_period` (default `24h`). A backup uploads its blobs before the manifest that lists them, so a prune running meanwhile, on this host or another sharing the destination, leaves them as long as no backup run takes longer than the grace period. The grace period does not protect a blob a backup reuses from an earlier run that stopped before its manifest: the backup skips the upload of a blob already on the destination, so that blob keeps the time of its first upload, and a prune before the backup's manifest is written can delete it; the new snapshot then cannot be restored. `remote nuke` has no such check
|
||||||
|
|
||||||
## Restoration Requirements
|
## Restoration Requirements
|
||||||
|
|
||||||
|
|||||||
@@ -239,6 +239,16 @@ storage_url: ""
|
|||||||
# Default: 3
|
# Default: 3
|
||||||
# compression_level: 3
|
# compression_level: 3
|
||||||
|
|
||||||
|
# How old a blob that no snapshot lists must be before prune deletes it.
|
||||||
|
# A backup uploads its blobs before the manifest that lists them, so on a
|
||||||
|
# destination shared between hosts or users this must be longer than the
|
||||||
|
# longest backup run. It does not protect a blob a backup reuses from an
|
||||||
|
# earlier run that stopped before its manifest: that blob keeps the time of
|
||||||
|
# its first upload, so a prune can delete it.
|
||||||
|
# Accepts: 36h, 90m, etc. 0s deletes them at once.
|
||||||
|
# Default: 24h
|
||||||
|
# prune_grace_period: 24h
|
||||||
|
|
||||||
# Hostname used in snapshot IDs. Default: system hostname.
|
# Hostname used in snapshot IDs. Default: system hostname.
|
||||||
# hostname: myserver
|
# hostname: myserver
|
||||||
|
|
||||||
|
|||||||
@@ -134,7 +134,7 @@ func writeHermeticPruneConfig(t *testing.T, seedStale bool) string {
|
|||||||
seedStaleSnapshotRecord(t, indexPath)
|
seedStaleSnapshotRecord(t, indexPath)
|
||||||
|
|
||||||
// While the local index holds a snapshot, prune fails on a
|
// While the local index holds a snapshot, prune fails on a
|
||||||
// destination without the marker a backup writes.
|
// destination that answers that it has no marker.
|
||||||
require.NoError(t, os.WriteFile(
|
require.NoError(t, os.WriteFile(
|
||||||
filepath.Join(storeDir, "vaultik-destination.txt"), nil, 0o600))
|
filepath.Join(storeDir, "vaultik-destination.txt"), nil, 0o600))
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import (
|
|||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
@@ -14,8 +15,8 @@ import (
|
|||||||
const flagCron = "--cron"
|
const flagCron = "--cron"
|
||||||
|
|
||||||
// plantUnreferencedBlob writes a blob that no snapshot references to the
|
// plantUnreferencedBlob writes a blob that no snapshot references to the
|
||||||
// destination store at storeDir, so that a prune deletes it. Returns its
|
// destination store at storeDir, dated long before any prune grace period,
|
||||||
// path.
|
// so that a prune deletes it. Returns its path.
|
||||||
func plantUnreferencedBlob(t *testing.T, storeDir string) string {
|
func plantUnreferencedBlob(t *testing.T, storeDir string) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
@@ -25,6 +26,9 @@ func plantUnreferencedBlob(t *testing.T, storeDir string) string {
|
|||||||
require.NoError(t, os.MkdirAll(filepath.Dir(path), 0o750))
|
require.NoError(t, os.MkdirAll(filepath.Dir(path), 0o750))
|
||||||
require.NoError(t, os.WriteFile(path, []byte("unreferenced"), 0o600))
|
require.NoError(t, os.WriteFile(path, []byte("unreferenced"), 0o600))
|
||||||
|
|
||||||
|
longAgo := time.Unix(0, 0)
|
||||||
|
require.NoError(t, os.Chtimes(path, longAgo, longAgo))
|
||||||
|
|
||||||
return path
|
return path
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -87,6 +87,36 @@ func TestEntrySnapshotCreateReportsPostBackupPruneFailure(t *testing.T) {
|
|||||||
"the new snapshot is on the store beside the planted one")
|
"the new snapshot is on the store beside the planted one")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestEntrySnapshotCreatePruneExitsNonZeroWhenSourceIsEmpty backs up,
|
||||||
|
// empties the source directory, as a volume that is not mounted leaves
|
||||||
|
// its mount point, and runs `snapshot create --prune`. The command exits
|
||||||
|
// non-zero and reports the backup as completed, not as a failed snapshot
|
||||||
|
// creation. See https://git.eeqj.de/sneak/vaultik/issues/388.
|
||||||
|
//
|
||||||
|
//nolint:paralleltest // replaces os.Args, os.Stdout, os.Stderr and the xdg globals
|
||||||
|
func TestEntrySnapshotCreatePruneExitsNonZeroWhenSourceIsEmpty(t *testing.T) {
|
||||||
|
configPath, storeDir := writeBackupConfig(t)
|
||||||
|
|
||||||
|
code, _, _ := runEntry(t, flagConfig, configPath, cmdSnapshot, cmdCreate)
|
||||||
|
require.Equal(t, 0, code)
|
||||||
|
|
||||||
|
// writeBackupConfig puts the source directory beside the config file.
|
||||||
|
require.NoError(t, os.Remove(
|
||||||
|
filepath.Join(filepath.Dir(configPath), "source", "file.txt")))
|
||||||
|
|
||||||
|
code, _, stderr := runEntry(t,
|
||||||
|
flagConfig, configPath, cmdSnapshot, cmdCreate, flagPrune)
|
||||||
|
|
||||||
|
assert.Equal(t, 1, code)
|
||||||
|
assert.Contains(t, stderr,
|
||||||
|
"Backup completed, but 1 source path(s) were empty")
|
||||||
|
assert.NotContains(t, stderr, "Snapshot creation failed")
|
||||||
|
|
||||||
|
snapshots, err := os.ReadDir(filepath.Join(storeDir, "metadata"))
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Len(t, snapshots, 2, "the older snapshot is kept")
|
||||||
|
}
|
||||||
|
|
||||||
// writeBackupConfig builds a hermetic config with one small file in its
|
// writeBackupConfig builds a hermetic config with one small file in its
|
||||||
// source directory and an empty destination store directory. Returns the
|
// source directory and an empty destination store directory. Returns the
|
||||||
// config path and the store directory.
|
// config path and the store directory.
|
||||||
|
|||||||
@@ -42,7 +42,7 @@ func TestEntrySnapshotPurgeKeepsMetadataWhenLocalIndexDeleteFails(t *testing.T)
|
|||||||
seedStaleSnapshotRecord(t, indexPath)
|
seedStaleSnapshotRecord(t, indexPath)
|
||||||
|
|
||||||
// While the local index holds a snapshot, purge fails on a
|
// While the local index holds a snapshot, purge fails on a
|
||||||
// destination without the marker a backup writes.
|
// destination that answers that it has no marker.
|
||||||
require.NoError(t, os.WriteFile(
|
require.NoError(t, os.WriteFile(
|
||||||
filepath.Join(dir, "store", "vaultik-destination.txt"), nil, 0o600))
|
filepath.Join(dir, "store", "vaultik-destination.txt"), nil, 0o600))
|
||||||
|
|
||||||
|
|||||||
@@ -20,7 +20,12 @@ Local cleanup drops incomplete snapshots and any files, chunks, or
|
|||||||
blobs no longer referenced by a completed snapshot. Remote cleanup
|
blobs no longer referenced by a completed snapshot. Remote cleanup
|
||||||
scans every snapshot manifest in the destination store, builds the
|
scans every snapshot manifest in the destination store, builds the
|
||||||
set of still-referenced blob hashes, and deletes any blob not in that
|
set of still-referenced blob hashes, and deletes any blob not in that
|
||||||
set.
|
set that is older than prune_grace_period (default 24h). A newer one
|
||||||
|
is kept: it may belong to a backup still running, on this host or
|
||||||
|
another sharing the destination store, that has not yet uploaded its
|
||||||
|
manifest. The grace period does not protect a blob such a backup
|
||||||
|
reuses from an earlier run that stopped before its manifest: that
|
||||||
|
blob keeps the time of its first upload, so it can be deleted.
|
||||||
|
|
||||||
Snapshot create --prune runs the same cleanup automatically; this
|
Snapshot create --prune runs the same cleanup automatically; this
|
||||||
command is the manual entry point for the same work (e.g. after a
|
command is the manual entry point for the same work (e.g. after a
|
||||||
|
|||||||
@@ -115,8 +115,9 @@ on Linux), then /etc/vaultik/config.yml.`,
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
if errors.Is(err, vaultik.ErrFilesUnreadable) {
|
if errors.Is(err, vaultik.ErrFilesUnreadable) ||
|
||||||
log.Error("Backup completed with unreadable files", "error", err)
|
errors.Is(err, vaultik.ErrSourcePathEmpty) {
|
||||||
|
log.Error("Backup completed with an error", "error", err)
|
||||||
ReportErrorf("Backup completed, but %v", err)
|
ReportErrorf("Backup completed, but %v", err)
|
||||||
|
|
||||||
return
|
return
|
||||||
@@ -132,8 +133,9 @@ on Linux), then /etc/vaultik/config.yml.`,
|
|||||||
"Run in cron mode (silent unless warning or error)")
|
"Run in cron mode (silent unless warning or error)")
|
||||||
cmd.Flags().BoolVar(&opts.Prune, "prune", false,
|
cmd.Flags().BoolVar(&opts.Prune, "prune", false,
|
||||||
"After backup, drop older snapshots of the same name and remove "+
|
"After backup, drop older snapshots of the same name and remove "+
|
||||||
"orphaned blobs (not after a backup that skipped unreadable "+
|
"orphaned blobs older than prune_grace_period (not after a "+
|
||||||
"files)")
|
"backup that skipped unreadable files or found a configured "+
|
||||||
|
"path empty)")
|
||||||
cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "",
|
cmd.Flags().StringVar(&opts.KeepNewerThan, "keep-newer-than", "",
|
||||||
"With --prune: keep snapshots newer than this duration "+
|
"With --prune: keep snapshots newer than this duration "+
|
||||||
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+
|
"(e.g. 30d, 4w, 6mo, 1y; m is minutes, mo is months) "+
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ import (
|
|||||||
"path/filepath"
|
"path/filepath"
|
||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
"filippo.io/age"
|
"filippo.io/age"
|
||||||
"git.eeqj.de/sneak/smartconfig"
|
"git.eeqj.de/sneak/smartconfig"
|
||||||
@@ -38,6 +39,7 @@ const (
|
|||||||
defaultChunkSize = Size(10 * 1000 * 1000) // 10MB
|
defaultChunkSize = Size(10 * 1000 * 1000) // 10MB
|
||||||
defaultS3PartSize = Size(5 * 1024 * 1024) // 5MiB
|
defaultS3PartSize = Size(5 * 1024 * 1024) // 5MiB
|
||||||
defaultCompressionLevel = 3
|
defaultCompressionLevel = 3
|
||||||
|
defaultPruneGracePeriod = 24 * time.Hour
|
||||||
minChunkSize = 1000 * 1000 // 1MB
|
minChunkSize = 1000 * 1000 // 1MB
|
||||||
minCompressionLevel = 1
|
minCompressionLevel = 1
|
||||||
maxCompressionLevel = 19
|
maxCompressionLevel = 19
|
||||||
@@ -67,6 +69,8 @@ var (
|
|||||||
errFileURLNotAbsolute = errors.New(
|
errFileURLNotAbsolute = errors.New(
|
||||||
"a file:// storage_url needs an absolute path, " +
|
"a file:// storage_url needs an absolute path, " +
|
||||||
"as in file:///path/to/backup")
|
"as in file:///path/to/backup")
|
||||||
|
errNegativePruneGracePeriod = errors.New(
|
||||||
|
"prune_grace_period must not be negative")
|
||||||
errStorageNotConfigured = errors.New(
|
errStorageNotConfigured = errors.New(
|
||||||
"storage not configured; set storage_url or provide s3.endpoint + " +
|
"storage not configured; set storage_url or provide s3.endpoint + " +
|
||||||
"s3.bucket + credentials")
|
"s3.bucket + credentials")
|
||||||
@@ -187,6 +191,13 @@ type Config struct {
|
|||||||
Snapshots map[string]SnapshotConfig `yaml:"snapshots"`
|
Snapshots map[string]SnapshotConfig `yaml:"snapshots"`
|
||||||
CompressionLevel int `yaml:"compression_level"`
|
CompressionLevel int `yaml:"compression_level"`
|
||||||
|
|
||||||
|
// PruneGracePeriod is how old a blob no manifest lists must be before
|
||||||
|
// a prune deletes it. A backup lists its blobs in a manifest only
|
||||||
|
// once it has uploaded them all, so a younger one may belong to a
|
||||||
|
// backup still running, on this host or another sharing the
|
||||||
|
// destination store.
|
||||||
|
PruneGracePeriod time.Duration `yaml:"prune_grace_period"`
|
||||||
|
|
||||||
// StorageURL specifies the storage backend using a URL format.
|
// StorageURL specifies the storage backend using a URL format.
|
||||||
// Takes precedence over S3Config if set.
|
// Takes precedence over S3Config if set.
|
||||||
// Supported formats:
|
// Supported formats:
|
||||||
@@ -288,6 +299,7 @@ func Parse(data []byte) (*Config, error) {
|
|||||||
ChunkSize: defaultChunkSize,
|
ChunkSize: defaultChunkSize,
|
||||||
IndexPath: filepath.Join(xdg.DataHome, appName, "index.sqlite"),
|
IndexPath: filepath.Join(xdg.DataHome, appName, "index.sqlite"),
|
||||||
CompressionLevel: defaultCompressionLevel,
|
CompressionLevel: defaultCompressionLevel,
|
||||||
|
PruneGracePeriod: defaultPruneGracePeriod,
|
||||||
S3: S3Config{PartSize: defaultS3PartSize},
|
S3: S3Config{PartSize: defaultS3PartSize},
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -356,6 +368,7 @@ func Parse(data []byte) (*Config, error) {
|
|||||||
// exceeds the configured limit
|
// exceeds the configured limit
|
||||||
// - Compression level must be between 1 and 19
|
// - Compression level must be between 1 and 19
|
||||||
// - S3 part size must be between 5MiB and 5GiB, the part sizes S3 accepts
|
// - S3 part size must be between 5MiB and 5GiB, the part sizes S3 accepts
|
||||||
|
// - Prune grace period must not be negative
|
||||||
//
|
//
|
||||||
// Returns an error describing the first validation failure encountered.
|
// Returns an error describing the first validation failure encountered.
|
||||||
func (c *Config) Validate() error {
|
func (c *Config) Validate() error {
|
||||||
@@ -405,6 +418,10 @@ func (c *Config) Validate() error {
|
|||||||
return errBadS3PartSize
|
return errBadS3PartSize
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if c.PruneGracePeriod < 0 {
|
||||||
|
return errNegativePruneGracePeriod
|
||||||
|
}
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ import (
|
|||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/vaultik/internal/chunker"
|
"sneak.berlin/go/vaultik/internal/chunker"
|
||||||
"sneak.berlin/go/vaultik/internal/log"
|
"sneak.berlin/go/vaultik/internal/log"
|
||||||
@@ -412,6 +413,73 @@ func TestDocumentedSizeExamples(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestValidatePruneGracePeriod checks that a negative prune_grace_period
|
||||||
|
// is refused rather than acting like 0s, which deletes unreferenced blobs
|
||||||
|
// at once, and that 0s and the documented examples are accepted.
|
||||||
|
func TestValidatePruneGracePeriod(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const withoutGracePeriod = "snapshots:\n" +
|
||||||
|
" test:\n" +
|
||||||
|
" paths: [/tmp/vaultik-test-source]\n" +
|
||||||
|
"storage_url: file:///tmp/vaultik-test-storage\n"
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
setting string
|
||||||
|
wantErr error
|
||||||
|
}{
|
||||||
|
{"prune_grace_period: 0s", nil},
|
||||||
|
{"prune_grace_period: 90m", nil},
|
||||||
|
{"prune_grace_period: 36h", nil},
|
||||||
|
{"prune_grace_period: -1h", errNegativePruneGracePeriod},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.setting, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
cfg, err := Parse([]byte(withoutGracePeriod + tt.setting + "\n"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = cfg.Validate()
|
||||||
|
if !errors.Is(err, tt.wantErr) {
|
||||||
|
t.Fatalf("Validate() error = %v, want %v", err, tt.wantErr)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLoadPruneGracePeriod checks that a config file without
|
||||||
|
// prune_grace_period loads with the documented 24h. The value is written out
|
||||||
|
// here rather than taken from defaultPruneGracePeriod, so that a change of the
|
||||||
|
// default fails this test.
|
||||||
|
func TestLoadPruneGracePeriod(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const withoutGracePeriod = "snapshots:\n" +
|
||||||
|
" test:\n" +
|
||||||
|
" paths: [/tmp/vaultik-test-source]\n" +
|
||||||
|
"storage_url: file:///tmp/vaultik-test-storage\n"
|
||||||
|
|
||||||
|
path := filepath.Join(t.TempDir(), "config.yml")
|
||||||
|
|
||||||
|
err := os.WriteFile(path, []byte(withoutGracePeriod), 0o600)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("write config: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg, err := Load(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Load() unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if cfg.PruneGracePeriod != 24*time.Hour {
|
||||||
|
t.Errorf("prune_grace_period = %s, want 24h", cfg.PruneGracePeriod)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// TestLoadRejectsRelativeFileURL checks that a file:// storage_url with a
|
// TestLoadRejectsRelativeFileURL checks that a file:// storage_url with a
|
||||||
// relative path fails at load. Accepted, it would resolve against the
|
// relative path fails at load. Accepted, it would resolve against the
|
||||||
// working directory, so a backup run from another directory would write
|
// working directory, so a backup run from another directory would write
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ import (
|
|||||||
"io"
|
"io"
|
||||||
"strings"
|
"strings"
|
||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
|
"time"
|
||||||
|
|
||||||
"github.com/aws/aws-sdk-go-v2/aws"
|
"github.com/aws/aws-sdk-go-v2/aws"
|
||||||
"github.com/aws/aws-sdk-go-v2/config"
|
"github.com/aws/aws-sdk-go-v2/config"
|
||||||
@@ -302,7 +303,10 @@ func IsNotFound(err error) bool {
|
|||||||
type ObjectInfo struct {
|
type ObjectInfo struct {
|
||||||
Key string
|
Key string
|
||||||
Size int64
|
Size int64
|
||||||
Err error
|
// ModTime is the object's LastModified time. Only ListObjectsStream
|
||||||
|
// sets it.
|
||||||
|
ModTime time.Time
|
||||||
|
Err error
|
||||||
}
|
}
|
||||||
|
|
||||||
// ListObjectsStream lists objects with the given prefix and returns a channel.
|
// ListObjectsStream lists objects with the given prefix and returns a channel.
|
||||||
@@ -343,8 +347,9 @@ func (c *Client) ListObjectsStream(
|
|||||||
}
|
}
|
||||||
|
|
||||||
ch <- ObjectInfo{
|
ch <- ObjectInfo{
|
||||||
Key: key,
|
Key: key,
|
||||||
Size: *obj.Size,
|
Size: *obj.Size,
|
||||||
|
ModTime: aws.ToTime(obj.LastModified),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -168,6 +168,12 @@ type ScanResult struct {
|
|||||||
// skipped because it could not be read or listed; the rest are files
|
// skipped because it could not be read or listed; the rest are files
|
||||||
// removed after the walk listed them.
|
// removed after the walk listed them.
|
||||||
FilesUnreadable int
|
FilesUnreadable int
|
||||||
|
|
||||||
|
// EntriesFound counts the entries the walk found and could look up
|
||||||
|
// that no exclude pattern matches, leaving out the scanned path
|
||||||
|
// itself when it is a directory. It is zero for an empty directory,
|
||||||
|
// such as the mount point of a volume that is not mounted.
|
||||||
|
EntriesFound int
|
||||||
}
|
}
|
||||||
|
|
||||||
// NewScanner creates a new scanner instance
|
// NewScanner creates a new scanner instance
|
||||||
@@ -928,6 +934,11 @@ func (s *Scanner) scanPhase(
|
|||||||
default:
|
default:
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The scanned directory itself is not an entry found in it.
|
||||||
|
if filePath != path || !info.IsDir() {
|
||||||
|
result.EntriesFound++
|
||||||
|
}
|
||||||
|
|
||||||
// Handle symlinks and directories
|
// Handle symlinks and directories
|
||||||
handled, err := s.recordSpecialEntry(
|
handled, err := s.recordSpecialEntry(
|
||||||
filePath, info, existingFiles, knownFiles, collector, result)
|
filePath, info, existingFiles, knownFiles, collector, result)
|
||||||
@@ -940,18 +951,8 @@ func (s *Scanner) scanPhase(
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// Track this file as existing (for deletion detection)
|
filesScanned, changedCount := s.recordRegularFile(
|
||||||
existingFiles[filePath] = struct{}{}
|
filePath, info, existingFiles, knownFiles, collector, result)
|
||||||
|
|
||||||
// Check file against in-memory map (no DB query!)
|
|
||||||
file := s.buildRegularFileEntry(filePath, info)
|
|
||||||
needsProcessing := s.checkFileInMemory(file, knownFiles)
|
|
||||||
|
|
||||||
filesScanned, changedCount := collector.recordFile(
|
|
||||||
filePath, info, file, needsProcessing)
|
|
||||||
|
|
||||||
// Update result stats
|
|
||||||
s.updateScanEntryStats(result, needsProcessing, info)
|
|
||||||
|
|
||||||
// Output periodic status
|
// Output periodic status
|
||||||
if time.Since(lastStatusTime) >= scanStatusInterval {
|
if time.Since(lastStatusTime) >= scanStatusInterval {
|
||||||
@@ -1027,6 +1028,30 @@ func (s *Scanner) recordSpecialEntry(
|
|||||||
return false, nil
|
return false, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// recordRegularFile records a regular file as recordSpecialEntry does a
|
||||||
|
// symlink and returns the running scanned and changed counts.
|
||||||
|
func (s *Scanner) recordRegularFile(
|
||||||
|
filePath string, info os.FileInfo,
|
||||||
|
existingFiles map[string]struct{},
|
||||||
|
knownFiles map[string]*database.File,
|
||||||
|
collector *scanCollector, result *ScanResult,
|
||||||
|
) (int64, int) {
|
||||||
|
// Track this file as existing (for deletion detection)
|
||||||
|
existingFiles[filePath] = struct{}{}
|
||||||
|
|
||||||
|
// Check file against in-memory map (no DB query!)
|
||||||
|
file := s.buildRegularFileEntry(filePath, info)
|
||||||
|
needsProcessing := s.checkFileInMemory(file, knownFiles)
|
||||||
|
|
||||||
|
filesScanned, changedCount := collector.recordFile(
|
||||||
|
filePath, info, file, needsProcessing)
|
||||||
|
|
||||||
|
// Update result stats
|
||||||
|
s.updateScanEntryStats(result, needsProcessing, info)
|
||||||
|
|
||||||
|
return filesScanned, changedCount
|
||||||
|
}
|
||||||
|
|
||||||
// countRemovedDuringWalk counts an entry that was in its directory's
|
// countRemovedDuringWalk counts an entry that was in its directory's
|
||||||
// listing but was gone when the walk looked it up, like a file removed
|
// listing but was gone when the walk looked it up, like a file removed
|
||||||
// before phase 2 reads it: as examined and failed, and not as an error,
|
// before phase 2 reads it: as examined and failed, and not as an error,
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ import (
|
|||||||
"reflect"
|
"reflect"
|
||||||
"sort"
|
"sort"
|
||||||
"testing"
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/vaultik/internal/storage"
|
"sneak.berlin/go/vaultik/internal/storage"
|
||||||
)
|
)
|
||||||
@@ -19,15 +20,17 @@ import (
|
|||||||
// constructor, so the contract is defined once.
|
// constructor, so the contract is defined once.
|
||||||
//
|
//
|
||||||
// It exercises the public Storer interface: round-trip, stat, list with
|
// It exercises the public Storer interface: round-trip, stat, list with
|
||||||
// prefix filtering, overwrite, delete, delete-of-missing, and not-found on
|
// prefix filtering, the modification time ListStream reports, overwrite,
|
||||||
// Get and Stat. Each section takes its own fresh backend instance, so the
|
// delete, delete-of-missing, and not-found on Get and Stat. Each section
|
||||||
// order of sections never matters and no section sees another's objects.
|
// takes its own fresh backend instance, so the order of sections never
|
||||||
|
// matters and no section sees another's objects.
|
||||||
func runStorerConformance(t *testing.T, newStorer func(*testing.T) storage.Storer) {
|
func runStorerConformance(t *testing.T, newStorer func(*testing.T) storage.Storer) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
conformanceRoundTrip(t, newStorer(t))
|
conformanceRoundTrip(t, newStorer(t))
|
||||||
conformanceOverwrite(t, newStorer(t))
|
conformanceOverwrite(t, newStorer(t))
|
||||||
conformanceList(t, newStorer(t))
|
conformanceList(t, newStorer(t))
|
||||||
|
conformanceListStreamModTime(t, newStorer(t))
|
||||||
conformanceDelete(t, newStorer(t))
|
conformanceDelete(t, newStorer(t))
|
||||||
conformanceNotFound(t, newStorer(t))
|
conformanceNotFound(t, newStorer(t))
|
||||||
}
|
}
|
||||||
@@ -118,6 +121,35 @@ func conformanceList(t *testing.T, s storage.Storer) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// conformanceListStreamModTime checks that ListStream gives an object
|
||||||
|
// just written a modification time of about now, which prune compares
|
||||||
|
// with its grace period.
|
||||||
|
func conformanceListStreamModTime(t *testing.T, s storage.Storer) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
err := s.Put(ctx, "blobs/dd/written-now", bytes.NewReader([]byte("data")))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Put: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var listed []storage.ObjectInfo
|
||||||
|
|
||||||
|
for info := range s.ListStream(ctx, "blobs/") {
|
||||||
|
listed = append(listed, info)
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(listed) != 1 || listed[0].Err != nil {
|
||||||
|
t.Fatalf("ListStream(\"blobs/\") = %v, want the one object", listed)
|
||||||
|
}
|
||||||
|
|
||||||
|
if time.Since(listed[0].ModTime).Abs() > time.Minute {
|
||||||
|
t.Errorf("ListStream ModTime = %v, want within a minute of now",
|
||||||
|
listed[0].ModTime)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// conformanceDelete checks that Delete removes an object and that deleting
|
// conformanceDelete checks that Delete removes an object and that deleting
|
||||||
// a missing key is not an error.
|
// a missing key is not an error.
|
||||||
func conformanceDelete(t *testing.T, s storage.Storer) {
|
func conformanceDelete(t *testing.T, s storage.Storer) {
|
||||||
|
|||||||
@@ -2,15 +2,17 @@
|
|||||||
// faults on demand, so tests can reproduce the failure modes a real
|
// faults on demand, so tests can reproduce the failure modes a real
|
||||||
// backend exhibits: an upload that fails partway, a backend that reports
|
// backend exhibits: an upload that fails partway, a backend that reports
|
||||||
// success while storing nothing, reads that return corrupt or truncated
|
// success while storing nothing, reads that return corrupt or truncated
|
||||||
// bytes, and a delete the backend refuses. It is the seam called for by
|
// bytes, and a delete or a stat the backend refuses. It is the seam
|
||||||
// the fault-injection tests (sneak/vaultik issue 72) and is meant to be
|
// called for by the fault-injection tests (sneak/vaultik issue 72) and
|
||||||
// reused by future tests rather than re-implemented per case.
|
// is meant to be reused by future tests rather than re-implemented per
|
||||||
|
// case.
|
||||||
//
|
//
|
||||||
// The wrapper delegates every method to the inner Storer. Three hooks
|
// The wrapper delegates every method to the inner Storer. Four hooks
|
||||||
// change that: OnPut decides the fate of each write, OnGet decides how
|
// change that: OnPut decides the fate of each write, OnGet decides how
|
||||||
// each read's bytes are returned, and OnDelete decides whether each
|
// each read's bytes are returned, OnDelete decides whether each delete
|
||||||
// delete fails. All are keyed by the object key, so a test can fault
|
// fails, and OnStat whether each stat fails. All are keyed by the object
|
||||||
// only blobs, only metadata, or a single object.
|
// key, so a test can fault only blobs, only metadata, or a single
|
||||||
|
// object.
|
||||||
package faultstore
|
package faultstore
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -30,6 +32,11 @@ var ErrInjectedUpload = errors.New("faultstore: injected upload failure")
|
|||||||
// fail.
|
// fail.
|
||||||
var ErrInjectedDelete = errors.New("faultstore: injected delete failure")
|
var ErrInjectedDelete = errors.New("faultstore: injected delete failure")
|
||||||
|
|
||||||
|
// ErrInjectedStat is returned by a Stat the OnStat hook chose to fail.
|
||||||
|
// It is not storage.ErrNotFound: the backend refused the request, as S3
|
||||||
|
// refuses a HEAD to credentials that allow only PUT.
|
||||||
|
var ErrInjectedStat = errors.New("faultstore: injected stat failure")
|
||||||
|
|
||||||
// PutAction is the disposition OnPut assigns to a write.
|
// PutAction is the disposition OnPut assigns to a write.
|
||||||
type PutAction int
|
type PutAction int
|
||||||
|
|
||||||
@@ -76,6 +83,10 @@ type Storer struct {
|
|||||||
// object key. Returning true fails the delete and leaves the object
|
// object key. Returning true fails the delete and leaves the object
|
||||||
// in place.
|
// in place.
|
||||||
OnDelete func(key string) bool
|
OnDelete func(key string) bool
|
||||||
|
|
||||||
|
// OnStat, when set, is consulted before every Stat with the object
|
||||||
|
// key. Returning true fails the stat with ErrInjectedStat.
|
||||||
|
OnStat func(key string) bool
|
||||||
}
|
}
|
||||||
|
|
||||||
// New wraps inner. inner must be non-nil.
|
// New wraps inner. inner must be non-nil.
|
||||||
@@ -159,8 +170,12 @@ func damage(fault GetFault, data []byte) []byte {
|
|||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
// Stat delegates unchanged.
|
// Stat returns the object's info unless OnStat fails the stat.
|
||||||
func (f *Storer) Stat(ctx context.Context, key string) (*storage.ObjectInfo, error) {
|
func (f *Storer) Stat(ctx context.Context, key string) (*storage.ObjectInfo, error) {
|
||||||
|
if f.OnStat != nil && f.OnStat(key) {
|
||||||
|
return nil, fmt.Errorf("%w for %q", ErrInjectedStat, key)
|
||||||
|
}
|
||||||
|
|
||||||
return f.inner.Stat(ctx, key)
|
return f.inner.Stat(ctx, key)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -235,8 +235,9 @@ func (f *FileStorer) ListStream(ctx context.Context, prefix string) <-chan Objec
|
|||||||
// Normalize path separators
|
// Normalize path separators
|
||||||
relPath = strings.ReplaceAll(relPath, string(filepath.Separator), "/")
|
relPath = strings.ReplaceAll(relPath, string(filepath.Separator), "/")
|
||||||
ch <- ObjectInfo{
|
ch <- ObjectInfo{
|
||||||
Key: relPath,
|
Key: relPath,
|
||||||
Size: info.Size(),
|
Size: info.Size(),
|
||||||
|
ModTime: info.ModTime(),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -53,6 +53,14 @@ func NewRcloneStorer(ctx context.Context, remote, path string) (*RcloneStorer, e
|
|||||||
rclonePath += path
|
rclonePath += path
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// On S3, Swift and some other remotes rclone keeps a time of its own in
|
||||||
|
// each object's metadata, which costs one request per object to read.
|
||||||
|
// rclone's use_server_modtime setting makes ModTime the time the
|
||||||
|
// listing gives instead: when the object was stored, which is what
|
||||||
|
// prune compares with prune_grace_period.
|
||||||
|
ctx, ci := fs.AddConfig(ctx)
|
||||||
|
ci.UseServerModTime = true
|
||||||
|
|
||||||
// Create the rclone filesystem
|
// Create the rclone filesystem
|
||||||
fsys, err := fs.NewFs(ctx, rclonePath)
|
fsys, err := fs.NewFs(ctx, rclonePath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -211,8 +219,9 @@ func (r *RcloneStorer) ListStream(
|
|||||||
|
|
||||||
if prefix == "" || strings.HasPrefix(key, prefix) {
|
if prefix == "" || strings.HasPrefix(key, prefix) {
|
||||||
ch <- ObjectInfo{
|
ch <- ObjectInfo{
|
||||||
Key: key,
|
Key: key,
|
||||||
Size: obj.Size(),
|
Size: obj.Size(),
|
||||||
|
ModTime: obj.ModTime(ctx),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|||||||
@@ -5,11 +5,17 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
"io"
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
|
"sync/atomic"
|
||||||
"testing"
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/johannesboyne/gofakes3"
|
||||||
|
"github.com/johannesboyne/gofakes3/backend/s3mem"
|
||||||
"github.com/rclone/rclone/fs"
|
"github.com/rclone/rclone/fs"
|
||||||
"github.com/rclone/rclone/fs/config"
|
"github.com/rclone/rclone/fs/config"
|
||||||
"github.com/rclone/rclone/fs/config/configmap"
|
"github.com/rclone/rclone/fs/config/configmap"
|
||||||
@@ -429,3 +435,72 @@ func TestNewRcloneStorerReadsRcloneConfigEnv(t *testing.T) {
|
|||||||
t.Errorf("object is not in the remote's directory: %v", err)
|
t.Errorf("object is not in the remote's directory: %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestRcloneListStreamOnS3MakesNoRequestPerObject checks that listing an
|
||||||
|
// rclone S3 remote takes each object's time from the listing itself. By
|
||||||
|
// default rclone reads the time it keeps in each object's metadata, one
|
||||||
|
// request per object, and every command that lists the destination store
|
||||||
|
// would slow down as it grows.
|
||||||
|
//
|
||||||
|
//nolint:paralleltest // NewRcloneStorer installs the process-global rclone config
|
||||||
|
func TestRcloneListStreamOnS3MakesNoRequestPerObject(t *testing.T) {
|
||||||
|
backend := s3mem.New()
|
||||||
|
|
||||||
|
err := backend.CreateBucket(s3TestBucket)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("create bucket: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
const body = "blob-bytes"
|
||||||
|
|
||||||
|
_, err = backend.PutObject(s3TestBucket, testBlobKey, nil,
|
||||||
|
strings.NewReader(body), int64(len(body)))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("seed object: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var headRequests atomic.Int64
|
||||||
|
|
||||||
|
fakeS3 := gofakes3.New(backend).Server()
|
||||||
|
srv := httptest.NewServer(http.HandlerFunc(
|
||||||
|
func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if r.Method == http.MethodHead {
|
||||||
|
headRequests.Add(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
fakeS3.ServeHTTP(w, r)
|
||||||
|
}))
|
||||||
|
t.Cleanup(srv.Close)
|
||||||
|
|
||||||
|
remote := ":s3,provider=Other" +
|
||||||
|
",access_key_id=" + s3TestAccessKeyID +
|
||||||
|
",secret_access_key=" + s3TestSecretAccessKey +
|
||||||
|
",endpoint='" + srv.URL + "'"
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
s, err := storage.NewRcloneStorer(ctx, remote, s3TestBucket)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewRcloneStorer: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
headRequests.Store(0)
|
||||||
|
|
||||||
|
var listed []storage.ObjectInfo
|
||||||
|
|
||||||
|
for info := range s.ListStream(ctx, "blobs/") {
|
||||||
|
listed = append(listed, info)
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(listed) != 1 || listed[0].Err != nil {
|
||||||
|
t.Fatalf("ListStream(\"blobs/\") = %v, want the one object", listed)
|
||||||
|
}
|
||||||
|
|
||||||
|
if time.Since(listed[0].ModTime).Abs() > time.Minute {
|
||||||
|
t.Errorf("ListStream ModTime = %v, want within a minute of now",
|
||||||
|
listed[0].ModTime)
|
||||||
|
}
|
||||||
|
|
||||||
|
if n := headRequests.Load(); n != 0 {
|
||||||
|
t.Errorf("ListStream made %d HEAD request(s), want none", n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -89,9 +89,10 @@ func (s *S3Storer) ListStream(ctx context.Context, prefix string) <-chan ObjectI
|
|||||||
|
|
||||||
for info := range s.client.ListObjectsStream(ctx, prefix, false) {
|
for info := range s.client.ListObjectsStream(ctx, prefix, false) {
|
||||||
ch <- ObjectInfo{
|
ch <- ObjectInfo{
|
||||||
Key: info.Key,
|
Key: info.Key,
|
||||||
Size: info.Size,
|
Size: info.Size,
|
||||||
Err: info.Err,
|
ModTime: info.ModTime,
|
||||||
|
Err: info.Err,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}()
|
}()
|
||||||
|
|||||||
@@ -14,6 +14,7 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
"io"
|
"io"
|
||||||
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
// ErrNotFound is returned when an object does not exist.
|
// ErrNotFound is returned when an object does not exist.
|
||||||
@@ -27,7 +28,10 @@ type ProgressCallback func(bytesTransferred int64) error
|
|||||||
type ObjectInfo struct {
|
type ObjectInfo struct {
|
||||||
Key string // Object key/path
|
Key string // Object key/path
|
||||||
Size int64 // Size in bytes
|
Size int64 // Size in bytes
|
||||||
Err error // Error for streaming results (nil on success)
|
// ModTime is when the object was written, as the store reports it.
|
||||||
|
// Only ListStream sets it; Stat leaves it zero.
|
||||||
|
ModTime time.Time
|
||||||
|
Err error // Error for streaming results (nil on success)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Info provides human-readable storage configuration.
|
// Info provides human-readable storage configuration.
|
||||||
@@ -67,7 +71,8 @@ type Storer interface {
|
|||||||
// For large result sets, prefer ListStream.
|
// For large result sets, prefer ListStream.
|
||||||
List(ctx context.Context, prefix string) ([]string, error)
|
List(ctx context.Context, prefix string) ([]string, error)
|
||||||
|
|
||||||
// ListStream returns a channel of ObjectInfo for large result sets.
|
// ListStream returns a channel of ObjectInfo, with each object's key,
|
||||||
|
// size and modification time, for large result sets.
|
||||||
// The channel is closed when listing completes.
|
// The channel is closed when listing completes.
|
||||||
// If an error occurs during listing, the final item will have Err set.
|
// If an error occurs during listing, the final item will have Err set.
|
||||||
ListStream(ctx context.Context, prefix string) <-chan ObjectInfo
|
ListStream(ctx context.Context, prefix string) <-chan ObjectInfo
|
||||||
|
|||||||
@@ -5,14 +5,16 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
|
"sneak.berlin/go/vaultik/internal/log"
|
||||||
"sneak.berlin/go/vaultik/internal/storage"
|
"sneak.berlin/go/vaultik/internal/storage"
|
||||||
)
|
)
|
||||||
|
|
||||||
// destinationMarkerKey is the object every backup writes at the root of
|
// destinationMarkerKey is the object every backup writes at the root of
|
||||||
// the backup destination store. While the local index holds a snapshot
|
// the backup destination store. While the local index holds a snapshot
|
||||||
// stored there, a destination without it is treated as not present: an
|
// stored there, a destination that answers a look-up of it with not
|
||||||
// unplugged volume leaves an empty directory at its mount point, which
|
// found is treated as not present: an unplugged volume leaves an empty
|
||||||
// would otherwise read as a store whose snapshots are all gone.
|
// directory at its mount point, which would otherwise read as a store
|
||||||
|
// whose snapshots are all gone.
|
||||||
const destinationMarkerKey = "vaultik-destination.txt"
|
const destinationMarkerKey = "vaultik-destination.txt"
|
||||||
|
|
||||||
// destinationMarkerText is the whole content of the marker. It names no
|
// destinationMarkerText is the whole content of the marker. It names no
|
||||||
@@ -20,16 +22,24 @@ const destinationMarkerKey = "vaultik-destination.txt"
|
|||||||
const destinationMarkerText = "This is a vaultik backup destination store.\n"
|
const destinationMarkerText = "This is a vaultik backup destination store.\n"
|
||||||
|
|
||||||
// errDestinationNotPresent is returned when the local index holds a
|
// errDestinationNotPresent is returned when the local index holds a
|
||||||
// snapshot stored on the backup destination store and the store has no
|
// snapshot stored on the backup destination store and the store answers
|
||||||
// marker.
|
// that it has no marker.
|
||||||
var errDestinationNotPresent = errors.New("backup destination store is not present")
|
var errDestinationNotPresent = errors.New("backup destination store is not present")
|
||||||
|
|
||||||
// checkDestinationMarker returns errDestinationNotPresent when the local
|
// checkDestinationMarker returns errDestinationNotPresent when the local
|
||||||
// index holds a snapshot stored on the backup destination store and the
|
// index holds a snapshot stored on the backup destination store and the
|
||||||
// store has no marker. With no snapshot in the local index, as on a
|
// store answers that it has no marker. With no snapshot in the local
|
||||||
// first backup or on another machine, there is nothing to check; nor
|
// index, as on a first backup or on another machine, there is nothing to
|
||||||
// when the local index is bound to a different destination than the
|
// check; nor when the local index is bound to a different destination
|
||||||
// configured one, since its snapshots are stored there.
|
// than the configured one, since its snapshots are stored there.
|
||||||
|
//
|
||||||
|
// A look-up of the marker that fails for any other reason than not
|
||||||
|
// found is logged and otherwise ignored, because the check could not be
|
||||||
|
// made: S3 refuses it to credentials that allow only PUT, which is all
|
||||||
|
// a backup needs. The check exists for a file:// volume's empty mount
|
||||||
|
// point, which always answers not found, and a destination that cannot
|
||||||
|
// be reached fails on the next request made to it, such as the backup's
|
||||||
|
// PUT of the marker.
|
||||||
func (v *Vaultik) checkDestinationMarker() error {
|
func (v *Vaultik) checkDestinationMarker() error {
|
||||||
ids, err := v.localSnapshotIDs()
|
ids, err := v.localSnapshotIDs()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -63,8 +73,11 @@ func (v *Vaultik) checkDestinationMarker() error {
|
|||||||
destinationMarkerKey)
|
destinationMarkerKey)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Info, not a warning: --cron must stay silent on a successful
|
||||||
|
// backup from a host whose credentials allow only PUT.
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("checking for %s: %w", destinationMarkerKey, err)
|
log.Info("Could not check the backup destination store for its marker",
|
||||||
|
"key", destinationMarkerKey, "error", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
|
|||||||
@@ -13,15 +13,16 @@ import (
|
|||||||
"sneak.berlin/go/vaultik/internal/database"
|
"sneak.berlin/go/vaultik/internal/database"
|
||||||
"sneak.berlin/go/vaultik/internal/log"
|
"sneak.berlin/go/vaultik/internal/log"
|
||||||
"sneak.berlin/go/vaultik/internal/storage"
|
"sneak.berlin/go/vaultik/internal/storage"
|
||||||
|
"sneak.berlin/go/vaultik/internal/storage/faultstore"
|
||||||
"sneak.berlin/go/vaultik/internal/vaultik"
|
"sneak.berlin/go/vaultik/internal/vaultik"
|
||||||
)
|
)
|
||||||
|
|
||||||
// These tests cover https://git.eeqj.de/sneak/vaultik/issues/239: a
|
// These tests cover https://git.eeqj.de/sneak/vaultik/issues/239: a
|
||||||
// file:// destination at a volume's mount point leaves an empty directory
|
// file:// destination at a volume's mount point leaves an empty directory
|
||||||
// behind when the volume is unplugged. While the local index holds
|
// behind when the volume is unplugged. While the local index holds
|
||||||
// snapshots, a destination without vaultik-destination.txt is not present,
|
// snapshots, a destination that answers that it has no
|
||||||
// so no command may write to it or conclude from it that the local
|
// vaultik-destination.txt is not present, so no command may write to it
|
||||||
// snapshots are gone.
|
// or conclude from it that the local snapshots are gone.
|
||||||
|
|
||||||
// markerFile is the name of the file a backup writes at the root of the
|
// markerFile is the name of the file a backup writes at the root of the
|
||||||
// destination.
|
// destination.
|
||||||
@@ -93,6 +94,39 @@ func TestBackupRefusesDestinationWithoutMarker(t *testing.T) {
|
|||||||
assertUntouched(ctx, t, repos, storeDir)
|
assertUntouched(ctx, t, repos, storeDir)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestBackupGoesOnWhenMarkerCannotBeChecked checks that a backup goes on
|
||||||
|
// and writes the marker when looking it up fails with an error other
|
||||||
|
// than not found, as S3 answers credentials that allow only PUT
|
||||||
|
// (https://git.eeqj.de/sneak/vaultik/issues/389), and that a marker
|
||||||
|
// that is not found still stops it.
|
||||||
|
func TestBackupGoesOnWhenMarkerCannotBeChecked(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
storeDir := t.TempDir()
|
||||||
|
v, repos, _ := backUpToFileDestination(ctx, t, storeDir)
|
||||||
|
|
||||||
|
// Removed so the test sees the backup write it again: with the
|
||||||
|
// look-up failing, the backup cannot tell that it is gone.
|
||||||
|
markerPath := filepath.Join(storeDir, markerFile)
|
||||||
|
require.NoError(t, os.Remove(markerPath))
|
||||||
|
|
||||||
|
store := faultstore.New(v.Storage)
|
||||||
|
store.OnStat = func(key string) bool { return key == markerFile }
|
||||||
|
v.Storage = store
|
||||||
|
|
||||||
|
require.NoError(t, backUp(v, "second"))
|
||||||
|
localSnapshotID(ctx, t, repos, "second")
|
||||||
|
assert.FileExists(t, markerPath)
|
||||||
|
|
||||||
|
require.NoError(t, os.Remove(markerPath))
|
||||||
|
|
||||||
|
store.OnStat = nil
|
||||||
|
|
||||||
|
require.ErrorContains(t, backUp(v, "third"), markerFile)
|
||||||
|
}
|
||||||
|
|
||||||
// TestBackupChecksMarkerBeforeDroppingIncompleteSnapshots checks that
|
// TestBackupChecksMarkerBeforeDroppingIncompleteSnapshots checks that
|
||||||
// snapshot create writes nothing when the local index holds only an
|
// snapshot create writes nothing when the local index holds only an
|
||||||
// incomplete snapshot and the destination has no marker. The backup drops
|
// incomplete snapshot and the destination has no marker. The backup drops
|
||||||
|
|||||||
@@ -457,9 +457,9 @@ func (v *Vaultik) scanRemoteBlobStorage(
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, size := range allBlobs {
|
for _, blob := range allBlobs {
|
||||||
result.TotalBlobCount++
|
result.TotalBlobCount++
|
||||||
result.TotalBlobSize += size
|
result.TotalBlobSize += blob.Size
|
||||||
}
|
}
|
||||||
|
|
||||||
// A blob named only by a manifest that could not be read, or by one
|
// A blob named only by a manifest that could not be read, or by one
|
||||||
@@ -474,10 +474,10 @@ func (v *Vaultik) scanRemoteBlobStorage(
|
|||||||
orphanedSize int64
|
orphanedSize int64
|
||||||
)
|
)
|
||||||
|
|
||||||
for hash, size := range allBlobs {
|
for hash, blob := range allBlobs {
|
||||||
if _, referenced := referencedBlobs[hash]; !referenced {
|
if _, referenced := referencedBlobs[hash]; !referenced {
|
||||||
orphanedCount++
|
orphanedCount++
|
||||||
orphanedSize += size
|
orphanedSize += blob.Size
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+61
-19
@@ -5,8 +5,10 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"strings"
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
"sneak.berlin/go/vaultik/internal/log"
|
"sneak.berlin/go/vaultik/internal/log"
|
||||||
|
"sneak.berlin/go/vaultik/internal/storage"
|
||||||
)
|
)
|
||||||
|
|
||||||
// PruneOptions contains options for the prune command
|
// PruneOptions contains options for the prune command
|
||||||
@@ -56,8 +58,8 @@ func (v *Vaultik) NukeRemote(force bool) error {
|
|||||||
|
|
||||||
// Checked before anything is deleted: running nuke again would fail
|
// Checked before anything is deleted: running nuke again would fail
|
||||||
// the same way, so the advice below must not follow these errors.
|
// the same way, so the advice below must not follow these errors.
|
||||||
// The listing also fails on a destination without the marker while
|
// The listing also fails on a destination that answers that it has
|
||||||
// the local index holds snapshots.
|
// no marker while the local index holds snapshots.
|
||||||
err := v.EnsureStorageBinding()
|
err := v.EnsureStorageBinding()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
@@ -163,6 +165,10 @@ type PruneBlobsResult struct {
|
|||||||
BlobsDeleted int `json:"blobs_deleted"`
|
BlobsDeleted int `json:"blobs_deleted"`
|
||||||
BlobsFailed int `json:"blobs_failed,omitempty"`
|
BlobsFailed int `json:"blobs_failed,omitempty"`
|
||||||
BytesFreed int64 `json:"bytes_freed"`
|
BytesFreed int64 `json:"bytes_freed"`
|
||||||
|
// BlobsTooNew counts the unreferenced blobs kept because they are
|
||||||
|
// newer than the prune grace period, or of unknown age. BlobsFound
|
||||||
|
// does not include them.
|
||||||
|
BlobsTooNew int `json:"blobs_too_new"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// Prune removes orphaned data from the local index database AND
|
// Prune removes orphaned data from the local index database AND
|
||||||
@@ -208,9 +214,15 @@ func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
unreferencedBlobs, totalSize := v.findUnreferencedBlobs(allBlobs, allBlobsReferenced)
|
unreferencedBlobs, totalSize, tooNew := v.findUnreferencedBlobs(
|
||||||
|
allBlobs, allBlobsReferenced)
|
||||||
|
|
||||||
result := &PruneBlobsResult{BlobsFound: len(unreferencedBlobs)}
|
result := &PruneBlobsResult{
|
||||||
|
BlobsFound: len(unreferencedBlobs),
|
||||||
|
BlobsTooNew: tooNew,
|
||||||
|
}
|
||||||
|
|
||||||
|
v.reportBlobsTooNew(tooNew, opts)
|
||||||
|
|
||||||
if len(unreferencedBlobs) == 0 {
|
if len(unreferencedBlobs) == 0 {
|
||||||
log.Info("No unreferenced blobs found")
|
log.Info("No unreferenced blobs found")
|
||||||
@@ -344,11 +356,12 @@ func (v *Vaultik) listUniqueSnapshotIDs() ([]string, error) {
|
|||||||
return snapshotIDs, nil
|
return snapshotIDs, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// listAllRemoteBlobs returns a map of all blob hashes to their sizes in remote storage
|
// listAllRemoteBlobs returns a map of all blob hashes in remote storage
|
||||||
func (v *Vaultik) listAllRemoteBlobs() (map[string]int64, error) {
|
// to their listing entries.
|
||||||
|
func (v *Vaultik) listAllRemoteBlobs() (map[string]storage.ObjectInfo, error) {
|
||||||
log.Info("Listing all blobs in storage")
|
log.Info("Listing all blobs in storage")
|
||||||
|
|
||||||
allBlobs := make(map[string]int64)
|
allBlobs := make(map[string]storage.ObjectInfo)
|
||||||
blobObjectCh := v.Storage.ListStream(v.ctx, "blobs/")
|
blobObjectCh := v.Storage.ListStream(v.ctx, "blobs/")
|
||||||
|
|
||||||
for object := range blobObjectCh {
|
for object := range blobObjectCh {
|
||||||
@@ -372,7 +385,7 @@ func (v *Vaultik) listAllRemoteBlobs() (map[string]int64, error) {
|
|||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
allBlobs[parts[3]] = object.Size
|
allBlobs[parts[3]] = object
|
||||||
}
|
}
|
||||||
|
|
||||||
log.Info("Found blobs in storage", "count", len(allBlobs))
|
log.Info("Found blobs in storage", "count", len(allBlobs))
|
||||||
@@ -380,30 +393,59 @@ func (v *Vaultik) listAllRemoteBlobs() (map[string]int64, error) {
|
|||||||
return allBlobs, nil
|
return allBlobs, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// findUnreferencedBlobs returns blob hashes not referenced by any
|
// findUnreferencedBlobs returns the hashes of the blobs no manifest
|
||||||
// manifest and their total size.
|
// references that are older than the prune grace period, and their total
|
||||||
|
// size. It also returns how many unreferenced blobs it left out for
|
||||||
|
// being newer than that, or of unknown age because the listing gave no
|
||||||
|
// time for them.
|
||||||
func (v *Vaultik) findUnreferencedBlobs(
|
func (v *Vaultik) findUnreferencedBlobs(
|
||||||
allBlobs map[string]int64, referenced map[string]bool,
|
allBlobs map[string]storage.ObjectInfo, referenced map[string]bool,
|
||||||
) ([]string, int64) {
|
) ([]string, int64, int) {
|
||||||
var (
|
var (
|
||||||
unreferenced []string
|
unreferenced []string
|
||||||
totalSize int64
|
totalSize int64
|
||||||
|
tooNew int
|
||||||
)
|
)
|
||||||
|
|
||||||
for hash, size := range allBlobs {
|
cutoff := time.Now().UTC().Add(-v.Config.PruneGracePeriod)
|
||||||
if !referenced[hash] {
|
|
||||||
unreferenced = append(unreferenced, hash)
|
for hash, blob := range allBlobs {
|
||||||
totalSize += size
|
if referenced[hash] {
|
||||||
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if blob.ModTime.IsZero() || blob.ModTime.After(cutoff) {
|
||||||
|
tooNew++
|
||||||
|
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
unreferenced = append(unreferenced, hash)
|
||||||
|
totalSize += blob.Size
|
||||||
}
|
}
|
||||||
|
|
||||||
return unreferenced, totalSize
|
return unreferenced, totalSize, tooNew
|
||||||
|
}
|
||||||
|
|
||||||
|
// reportBlobsTooNew says how many unreferenced blobs prune kept for being
|
||||||
|
// newer than the grace period or of unknown age. Under --json the
|
||||||
|
// document carries the count instead.
|
||||||
|
func (v *Vaultik) reportBlobsTooNew(tooNew int, opts *PruneOptions) {
|
||||||
|
if tooNew == 0 || opts.JSON {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
v.UI.Infof("Kept %d unreferenced blob(s) newer than "+
|
||||||
|
"prune_grace_period (%s), or of unknown age: a backup still "+
|
||||||
|
"running may need them.",
|
||||||
|
tooNew, v.Config.PruneGracePeriod)
|
||||||
}
|
}
|
||||||
|
|
||||||
// deleteUnreferencedBlobs deletes the given blobs from storage and
|
// deleteUnreferencedBlobs deletes the given blobs from storage and
|
||||||
// populates the result.
|
// populates the result.
|
||||||
func (v *Vaultik) deleteUnreferencedBlobs(
|
func (v *Vaultik) deleteUnreferencedBlobs(
|
||||||
unreferencedBlobs []string, allBlobs map[string]int64, result *PruneBlobsResult,
|
unreferencedBlobs []string, allBlobs map[string]storage.ObjectInfo,
|
||||||
|
result *PruneBlobsResult,
|
||||||
) {
|
) {
|
||||||
log.Info("Deleting unreferenced blobs")
|
log.Info("Deleting unreferenced blobs")
|
||||||
|
|
||||||
@@ -418,7 +460,7 @@ func (v *Vaultik) deleteUnreferencedBlobs(
|
|||||||
}
|
}
|
||||||
|
|
||||||
result.BlobsDeleted++
|
result.BlobsDeleted++
|
||||||
result.BytesFreed += allBlobs[hash]
|
result.BytesFreed += allBlobs[hash].Size
|
||||||
|
|
||||||
if (i+1)%progressLogEvery == 0 || i == len(unreferencedBlobs)-1 {
|
if (i+1)%progressLogEvery == 0 || i == len(unreferencedBlobs)-1 {
|
||||||
log.Info("Deletion progress",
|
log.Info("Deletion progress",
|
||||||
|
|||||||
@@ -0,0 +1,166 @@
|
|||||||
|
package vaultik_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/spf13/afero"
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"sneak.berlin/go/vaultik/internal/config"
|
||||||
|
"sneak.berlin/go/vaultik/internal/database"
|
||||||
|
"sneak.berlin/go/vaultik/internal/log"
|
||||||
|
"sneak.berlin/go/vaultik/internal/storage"
|
||||||
|
"sneak.berlin/go/vaultik/internal/storage/faultstore"
|
||||||
|
"sneak.berlin/go/vaultik/internal/ui"
|
||||||
|
"sneak.berlin/go/vaultik/internal/vaultik"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestPruneDuringBackupKeepsItsBlobs runs a prune from another host
|
||||||
|
// sharing the destination store at the moment a backup has uploaded its
|
||||||
|
// blobs but not yet its manifest, so no manifest lists those blobs. The
|
||||||
|
// prune must leave them, and the backup must then restore.
|
||||||
|
func TestPruneDuringBackupKeepsItsBlobs(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
fs := afero.NewOsFs()
|
||||||
|
tempDir := t.TempDir()
|
||||||
|
dataDir := filepath.Join(tempDir, "src")
|
||||||
|
storeDir := t.TempDir()
|
||||||
|
restoreDir := filepath.Join(tempDir, "restored")
|
||||||
|
dbPath := filepath.Join(tempDir, "index.sqlite")
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
cfg := faultTestConfig()
|
||||||
|
testFiles := writeFaultSourceTree(t, fs, dataDir)
|
||||||
|
|
||||||
|
inner, err := storage.NewFileStorer(storeDir)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
otherHostCfg := faultTestConfig()
|
||||||
|
otherHostCfg.PruneGracePeriod = time.Hour
|
||||||
|
otherHost := newReaderVaultik(ctx, otherHostCfg, inner, nil, fs)
|
||||||
|
|
||||||
|
var (
|
||||||
|
pruned bool
|
||||||
|
pruneErr error
|
||||||
|
)
|
||||||
|
|
||||||
|
store := faultstore.New(inner)
|
||||||
|
store.OnPut = func(key string) faultstore.PutAction {
|
||||||
|
if strings.HasSuffix(key, "/manifest.json.zst") {
|
||||||
|
pruned = true
|
||||||
|
pruneErr = otherHost.PruneBlobs(&vaultik.PruneOptions{Force: true})
|
||||||
|
}
|
||||||
|
|
||||||
|
return faultstore.PutNormal
|
||||||
|
}
|
||||||
|
|
||||||
|
db, err := database.New(ctx, dbPath)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
repos := database.NewRepositories(db)
|
||||||
|
|
||||||
|
id := fullFaultBackup(ctx, t, fs, store, cfg, repos, dataDir, dbPath, "shared")
|
||||||
|
require.NoError(t, db.Close())
|
||||||
|
|
||||||
|
require.True(t, pruned, "the prune must run before the manifest upload")
|
||||||
|
require.NoError(t, pruneErr)
|
||||||
|
|
||||||
|
v := newReaderVaultik(ctx, cfg, inner, nil, fs)
|
||||||
|
require.NoError(t, v.Restore(&vaultik.RestoreOptions{
|
||||||
|
SnapshotID: id,
|
||||||
|
TargetDir: restoreDir,
|
||||||
|
Verify: true,
|
||||||
|
}), "a prune during the backup must not delete the backup's blobs")
|
||||||
|
|
||||||
|
assertRestoredTree(t, fs, restoreDir, testFiles)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestPruneKeepsUnreferencedBlobWithNoTime checks that a blob no
|
||||||
|
// manifest lists, whose listing gives no modification time (as an S3
|
||||||
|
// listing entry without LastModified does), is kept and counted with
|
||||||
|
// the blobs too new to delete: its age is unknown.
|
||||||
|
func TestPruneKeepsUnreferencedBlobWithNoTime(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
env := newListEnv(t)
|
||||||
|
|
||||||
|
// Written into the mock store's map directly rather than through
|
||||||
|
// Put, so its listing gives no time.
|
||||||
|
blobKey := "blobs/" + testBlobHashA[:2] + "/" + testBlobHashA[2:4] +
|
||||||
|
"/" + testBlobHashA
|
||||||
|
env.store.data[blobKey] = []byte("blob")
|
||||||
|
|
||||||
|
require.NoError(t, env.v.PruneBlobs(&vaultik.PruneOptions{JSON: true}))
|
||||||
|
|
||||||
|
assert.True(t, env.store.hasKey(blobKey),
|
||||||
|
"a blob of unknown age must be kept")
|
||||||
|
|
||||||
|
var result vaultik.PruneBlobsResult
|
||||||
|
|
||||||
|
require.NoError(t, json.Unmarshal(env.stdout.Bytes(), &result))
|
||||||
|
assert.Equal(t, 0, result.BlobsDeleted)
|
||||||
|
assert.Equal(t, 1, result.BlobsTooNew)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestPruneDeletesUnreferencedBlobOlderThanGracePeriod checks that the
|
||||||
|
// grace period only delays deletion: a blob no manifest lists is deleted
|
||||||
|
// once it is older than the grace period, and kept and counted until
|
||||||
|
// then.
|
||||||
|
func TestPruneDeletesUnreferencedBlobOlderThanGracePeriod(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
storeDir := t.TempDir()
|
||||||
|
|
||||||
|
store, err := storage.NewFileStorer(storeDir)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
oldKey := "blobs/" + testBlobHashA[:2] + "/" + testBlobHashA[2:4] +
|
||||||
|
"/" + testBlobHashA
|
||||||
|
newKey := "blobs/" + testBlobHashB[:2] + "/" + testBlobHashB[2:4] +
|
||||||
|
"/" + testBlobHashB
|
||||||
|
|
||||||
|
for _, key := range []string{oldKey, newKey} {
|
||||||
|
require.NoError(t, store.Put(ctx, key, strings.NewReader("blob")))
|
||||||
|
}
|
||||||
|
|
||||||
|
gracePeriod := time.Hour
|
||||||
|
beforeGracePeriod := time.Now().Add(-gracePeriod - time.Minute)
|
||||||
|
require.NoError(t, os.Chtimes(filepath.Join(storeDir, oldKey),
|
||||||
|
beforeGracePeriod, beforeGracePeriod))
|
||||||
|
|
||||||
|
stdout := &bytes.Buffer{}
|
||||||
|
v := &vaultik.Vaultik{
|
||||||
|
Config: &config.Config{PruneGracePeriod: gracePeriod},
|
||||||
|
Storage: store,
|
||||||
|
Stdout: stdout,
|
||||||
|
UI: ui.NewWithColor(stdout, false),
|
||||||
|
}
|
||||||
|
v.SetContext(ctx)
|
||||||
|
|
||||||
|
require.NoError(t, v.PruneBlobs(&vaultik.PruneOptions{JSON: true}))
|
||||||
|
|
||||||
|
_, err = store.Stat(ctx, oldKey)
|
||||||
|
require.ErrorIs(t, err, storage.ErrNotFound,
|
||||||
|
"a blob older than the grace period must be deleted")
|
||||||
|
|
||||||
|
_, err = store.Stat(ctx, newKey)
|
||||||
|
require.NoError(t, err, "a blob newer than the grace period must be kept")
|
||||||
|
|
||||||
|
var result vaultik.PruneBlobsResult
|
||||||
|
|
||||||
|
require.NoError(t, json.Unmarshal(stdout.Bytes(), &result))
|
||||||
|
assert.Equal(t, 1, result.BlobsDeleted)
|
||||||
|
assert.Equal(t, 1, result.BlobsTooNew)
|
||||||
|
}
|
||||||
@@ -26,11 +26,14 @@ import (
|
|||||||
type testStorer struct {
|
type testStorer struct {
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
data map[string][]byte
|
data map[string][]byte
|
||||||
|
// putTimes holds when Put stored each object; ListStream reports it.
|
||||||
|
putTimes map[string]time.Time
|
||||||
}
|
}
|
||||||
|
|
||||||
func newTestStorer() *testStorer {
|
func newTestStorer() *testStorer {
|
||||||
return &testStorer{
|
return &testStorer{
|
||||||
data: make(map[string][]byte),
|
data: make(map[string][]byte),
|
||||||
|
putTimes: make(map[string]time.Time),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -44,6 +47,7 @@ func (s *testStorer) Put(_ context.Context, key string, reader io.Reader) error
|
|||||||
}
|
}
|
||||||
|
|
||||||
s.data[key] = data
|
s.data[key] = data
|
||||||
|
s.putTimes[key] = time.Now().UTC()
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -87,6 +91,7 @@ func (s *testStorer) Delete(_ context.Context, key string) error {
|
|||||||
defer s.mu.Unlock()
|
defer s.mu.Unlock()
|
||||||
|
|
||||||
delete(s.data, key)
|
delete(s.data, key)
|
||||||
|
delete(s.putTimes, key)
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -120,8 +125,9 @@ func (s *testStorer) ListStream(
|
|||||||
for key, data := range s.data {
|
for key, data := range s.data {
|
||||||
if prefix == "" || strings.HasPrefix(key, prefix) {
|
if prefix == "" || strings.HasPrefix(key, prefix) {
|
||||||
ch <- storage.ObjectInfo{
|
ch <- storage.ObjectInfo{
|
||||||
Key: key,
|
Key: key,
|
||||||
Size: int64(len(data)),
|
Size: int64(len(data)),
|
||||||
|
ModTime: s.putTimes[key],
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -44,6 +44,14 @@ var ErrPostBackupPruneFailed = errors.New("post-backup prune failed")
|
|||||||
// kept, so the caller can report it apart from a failed backup.
|
// kept, so the caller can report it apart from a failed backup.
|
||||||
var ErrFilesUnreadable = errors.New("file(s) could not be read")
|
var ErrFilesUnreadable = errors.New("file(s) could not be read")
|
||||||
|
|
||||||
|
// ErrSourcePathEmpty is returned when --prune skipped the post-backup
|
||||||
|
// prune because the scan found nothing in a configured path of a
|
||||||
|
// snapshot, as in the mount point of a volume that is not mounted. The
|
||||||
|
// snapshots are complete and kept, so the caller can report it apart
|
||||||
|
// from a failed backup. A run that also skipped unreadable files
|
||||||
|
// returns ErrFilesUnreadable instead.
|
||||||
|
var ErrSourcePathEmpty = errors.New("source path(s) were empty")
|
||||||
|
|
||||||
// listRecentLimit caps how many snapshot rows are fetched from the
|
// listRecentLimit caps how many snapshot rows are fetched from the
|
||||||
// local index for listing and purge operations.
|
// local index for listing and purge operations.
|
||||||
const listRecentLimit = 10000
|
const listRecentLimit = 10000
|
||||||
@@ -134,17 +142,10 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
// Process each named snapshot
|
filesUnreadable, emptyPaths, err := v.createNamedSnapshots(
|
||||||
filesUnreadable := 0
|
opts, hostname, snapshotNames)
|
||||||
|
if err != nil {
|
||||||
for snapIdx, snapName := range snapshotNames {
|
return err
|
||||||
unreadable, err := v.createNamedSnapshot(
|
|
||||||
opts, hostname, snapName, snapIdx+1, len(snapshotNames))
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
filesUnreadable += unreadable
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Print overall summary if multiple snapshots
|
// Print overall summary if multiple snapshots
|
||||||
@@ -154,13 +155,8 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
switch {
|
switch {
|
||||||
case opts.Prune && filesUnreadable > 0:
|
case opts.Prune && (filesUnreadable > 0 || len(emptyPaths) > 0):
|
||||||
// An older snapshot may be the only one that still holds a file
|
return v.skipPostBackupPrune(filesUnreadable, emptyPaths)
|
||||||
// this run could not read.
|
|
||||||
log.Warn("Skipping post-backup prune",
|
|
||||||
"files_unreadable", filesUnreadable)
|
|
||||||
v.UI.Warningf("Skipped the post-backup prune because %d file(s) "+
|
|
||||||
"could not be read; older snapshots are kept.", filesUnreadable)
|
|
||||||
case opts.Prune:
|
case opts.Prune:
|
||||||
err = v.runPostBackupPrune(snapshotNames, opts.KeepNewerThan)
|
err = v.runPostBackupPrune(snapshotNames, opts.KeepNewerThan)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -171,6 +167,61 @@ func (v *Vaultik) CreateSnapshot(opts *SnapshotCreateOptions) error {
|
|||||||
return v.finishCreateSnapshot(filesUnreadable)
|
return v.finishCreateSnapshot(filesUnreadable)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// createNamedSnapshots creates each named snapshot in turn. It returns
|
||||||
|
// the number of entries --skip-errors skipped because it could not read
|
||||||
|
// or list them, and each configured path in which the scan found
|
||||||
|
// nothing, named with its snapshot.
|
||||||
|
func (v *Vaultik) createNamedSnapshots(
|
||||||
|
opts *SnapshotCreateOptions, hostname string, snapshotNames []string,
|
||||||
|
) (int, []string, error) {
|
||||||
|
filesUnreadable := 0
|
||||||
|
|
||||||
|
var emptyPaths []string
|
||||||
|
|
||||||
|
for snapIdx, snapName := range snapshotNames {
|
||||||
|
stats, err := v.createNamedSnapshot(
|
||||||
|
opts, hostname, snapName, snapIdx+1, len(snapshotNames))
|
||||||
|
if err != nil {
|
||||||
|
return 0, nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
filesUnreadable += stats.totalFilesUnreadable
|
||||||
|
|
||||||
|
for _, dir := range stats.emptyPaths {
|
||||||
|
emptyPaths = append(emptyPaths, dir+" of snapshot "+snapName)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return filesUnreadable, emptyPaths, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// skipPostBackupPrune says why the post-backup prune does not run, naming
|
||||||
|
// every cause, and returns the error the run ends with. An older snapshot
|
||||||
|
// may be the only one that still holds a file this run could not read,
|
||||||
|
// or what an empty path held. The previous snapshot is no guide: after
|
||||||
|
// one run with an empty path it is empty too.
|
||||||
|
func (v *Vaultik) skipPostBackupPrune(
|
||||||
|
filesUnreadable int, emptyPaths []string,
|
||||||
|
) error {
|
||||||
|
log.Warn("Skipping post-backup prune",
|
||||||
|
"files_unreadable", filesUnreadable, "empty_paths", emptyPaths)
|
||||||
|
|
||||||
|
if len(emptyPaths) > 0 {
|
||||||
|
v.UI.Warningf("Skipped the post-backup prune because nothing was "+
|
||||||
|
"found in %s; older snapshots are kept.",
|
||||||
|
strings.Join(emptyPaths, ", "))
|
||||||
|
}
|
||||||
|
|
||||||
|
if filesUnreadable > 0 {
|
||||||
|
v.UI.Warningf("Skipped the post-backup prune because %d file(s) "+
|
||||||
|
"could not be read; older snapshots are kept.", filesUnreadable)
|
||||||
|
|
||||||
|
return fmt.Errorf("%d %w", filesUnreadable, ErrFilesUnreadable)
|
||||||
|
}
|
||||||
|
|
||||||
|
return fmt.Errorf("%d %w", len(emptyPaths), ErrSourcePathEmpty)
|
||||||
|
}
|
||||||
|
|
||||||
// finishCreateSnapshot ends a run whose snapshots are all complete and
|
// finishCreateSnapshot ends a run whose snapshots are all complete and
|
||||||
// kept. Like finishRestore, it returns an error when --skip-errors
|
// kept. Like finishRestore, it returns an error when --skip-errors
|
||||||
// skipped anything it could not read or list; otherwise it prints the
|
// skipped anything it could not read or list; otherwise it prints the
|
||||||
@@ -274,6 +325,9 @@ type snapshotStats struct {
|
|||||||
// could not be read or listed.
|
// could not be read or listed.
|
||||||
totalFilesUnreadable int
|
totalFilesUnreadable int
|
||||||
|
|
||||||
|
// The configured paths in which the scan found nothing.
|
||||||
|
emptyPaths []string
|
||||||
|
|
||||||
// The sizes of all blobs the snapshot references, set by
|
// The sizes of all blobs the snapshot references, set by
|
||||||
// finalizeSnapshotMetadata once snapshot_blobs is populated.
|
// finalizeSnapshotMetadata once snapshot_blobs is populated.
|
||||||
blobSize int64
|
blobSize int64
|
||||||
@@ -281,11 +335,10 @@ type snapshotStats struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// createNamedSnapshot creates a single named snapshot and returns the
|
// createNamedSnapshot creates a single named snapshot and returns the
|
||||||
// number of entries --skip-errors skipped because it could not read or
|
// statistics of its scan.
|
||||||
// list them.
|
|
||||||
func (v *Vaultik) createNamedSnapshot(
|
func (v *Vaultik) createNamedSnapshot(
|
||||||
opts *SnapshotCreateOptions, hostname, snapName string, idx, total int,
|
opts *SnapshotCreateOptions, hostname, snapName string, idx, total int,
|
||||||
) (int, error) {
|
) (*snapshotStats, error) {
|
||||||
snapshotStartTime := time.Now()
|
snapshotStartTime := time.Now()
|
||||||
|
|
||||||
if total > 1 {
|
if total > 1 {
|
||||||
@@ -294,12 +347,12 @@ func (v *Vaultik) createNamedSnapshot(
|
|||||||
|
|
||||||
resolvedDirs, err := v.resolveSnapshotPaths(snapName)
|
resolvedDirs, err := v.resolveSnapshotPaths(snapName)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
previousSnapshotID, err := v.previousSnapshotID(snapName)
|
previousSnapshotID, err := v.previousSnapshotID(snapName)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
scanner := v.ScannerFactory(snapshot.ScannerParams{
|
scanner := v.ScannerFactory(snapshot.ScannerParams{
|
||||||
@@ -314,7 +367,7 @@ func (v *Vaultik) createNamedSnapshot(
|
|||||||
snapshotID, err := v.SnapshotManager.CreateSnapshotWithName(
|
snapshotID, err := v.SnapshotManager.CreateSnapshotWithName(
|
||||||
v.ctx, hostname, snapName, v.Globals.Version, v.Globals.Commit)
|
v.ctx, hostname, snapName, v.Globals.Version, v.Globals.Commit)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, fmt.Errorf("creating snapshot: %w", err)
|
return nil, fmt.Errorf("creating snapshot: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
log.Info("Beginning snapshot", "snapshot_id", snapshotID, "name", snapName)
|
log.Info("Beginning snapshot", "snapshot_id", snapshotID, "name", snapName)
|
||||||
@@ -322,12 +375,12 @@ func (v *Vaultik) createNamedSnapshot(
|
|||||||
|
|
||||||
stats, err := v.scanAllDirectories(scanner, resolvedDirs, snapshotID)
|
stats, err := v.scanAllDirectories(scanner, resolvedDirs, snapshotID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
err = v.finalizeSnapshotMetadata(snapshotID, stats)
|
err = v.finalizeSnapshotMetadata(snapshotID, stats)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
log.Info("Snapshot complete",
|
log.Info("Snapshot complete",
|
||||||
@@ -340,7 +393,7 @@ func (v *Vaultik) createNamedSnapshot(
|
|||||||
|
|
||||||
v.printSnapshotSummary(snapshotID, snapshotStartTime, stats)
|
v.printSnapshotSummary(snapshotID, snapshotStartTime, stats)
|
||||||
|
|
||||||
return stats.totalFilesUnreadable, nil
|
return stats, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// previousSnapshotID returns the ID of the most recent completed snapshot
|
// previousSnapshotID returns the ID of the most recent completed snapshot
|
||||||
@@ -362,7 +415,8 @@ func (v *Vaultik) previousSnapshotID(snapName string) (string, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// resolveSnapshotPaths resolves source directories to absolute paths
|
// resolveSnapshotPaths resolves source directories to absolute paths
|
||||||
// with symlink resolution.
|
// with symlink resolution. A path that is a symlink whose target does
|
||||||
|
// not exist is an error.
|
||||||
func (v *Vaultik) resolveSnapshotPaths(snapName string) ([]string, error) {
|
func (v *Vaultik) resolveSnapshotPaths(snapName string) ([]string, error) {
|
||||||
snapConfig := v.Config.Snapshots[snapName]
|
snapConfig := v.Config.Snapshots[snapName]
|
||||||
resolvedDirs := make([]string, 0, len(snapConfig.Paths))
|
resolvedDirs := make([]string, 0, len(snapConfig.Paths))
|
||||||
@@ -375,12 +429,18 @@ func (v *Vaultik) resolveSnapshotPaths(snapName string) ([]string, error) {
|
|||||||
|
|
||||||
resolvedPath, err := filepath.EvalSymlinks(absPath)
|
resolvedPath, err := filepath.EvalSymlinks(absPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if os.IsNotExist(err) {
|
// A path that does not exist is left for the scan to report.
|
||||||
resolvedPath = absPath
|
// One that exists is a symlink whose target is gone, as one
|
||||||
} else {
|
// into a volume that is not mounted: the scan would record
|
||||||
|
// only the link, and --prune would then drop the older
|
||||||
|
// snapshots that hold the files.
|
||||||
|
_, lstatErr := os.Lstat(absPath)
|
||||||
|
if !os.IsNotExist(err) || lstatErr == nil {
|
||||||
return nil, fmt.Errorf(
|
return nil, fmt.Errorf(
|
||||||
"failed to resolve symlinks for %s: %w", absPath, err)
|
"failed to resolve symlinks for %s: %w", absPath, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
resolvedPath = absPath
|
||||||
}
|
}
|
||||||
|
|
||||||
resolvedDirs = append(resolvedDirs, resolvedPath)
|
resolvedDirs = append(resolvedDirs, resolvedPath)
|
||||||
@@ -434,6 +494,10 @@ func (v *Vaultik) scanAllDirectories(
|
|||||||
stats.totalBytesUploaded += result.BytesUploaded
|
stats.totalBytesUploaded += result.BytesUploaded
|
||||||
stats.uploadDuration += result.UploadDuration
|
stats.uploadDuration += result.UploadDuration
|
||||||
|
|
||||||
|
if result.EntriesFound == 0 {
|
||||||
|
stats.emptyPaths = append(stats.emptyPaths, dir)
|
||||||
|
}
|
||||||
|
|
||||||
log.Info("Directory scan complete",
|
log.Info("Directory scan complete",
|
||||||
"path", dir,
|
"path", dir,
|
||||||
"files", result.FilesScanned,
|
"files", result.FilesScanned,
|
||||||
|
|||||||
@@ -1,8 +1,15 @@
|
|||||||
package vaultik_test
|
package vaultik_test
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"context"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"sneak.berlin/go/vaultik/internal/config"
|
||||||
|
"sneak.berlin/go/vaultik/internal/log"
|
||||||
"sneak.berlin/go/vaultik/internal/vaultik"
|
"sneak.berlin/go/vaultik/internal/vaultik"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -25,3 +32,230 @@ func TestSnapshotCreateOptions_PruneFlag(t *testing.T) {
|
|||||||
t.Error("Expected Prune to be false")
|
t.Error("Expected Prune to be false")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A backup under --prune that finds nothing in a configured path, as in
|
||||||
|
// the mount point of a volume that is not mounted, does not prune: run
|
||||||
|
// after run, the snapshot that holds the files is kept with everything
|
||||||
|
// it stored, and the run fails. That holds whether the empty path is the
|
||||||
|
// snapshot's only path or one of two. See
|
||||||
|
// https://git.eeqj.de/sneak/vaultik/issues/388.
|
||||||
|
func TestBackupWithEmptySourcePathDoesNotPrune(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
// emptied is the path, relative to the source directory, that is
|
||||||
|
// emptied after the first backup.
|
||||||
|
const (
|
||||||
|
snapshotName = "media"
|
||||||
|
emptied = "music"
|
||||||
|
)
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
paths []string // relative to the source directory
|
||||||
|
}{
|
||||||
|
{name: "only path", paths: []string{emptied}},
|
||||||
|
{name: "one of two paths", paths: []string{emptied, "docs"}},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
env := newDeletedCountEnv(t, map[string][]byte{
|
||||||
|
emptied + "/a.mp3": []byte("song a"),
|
||||||
|
emptied + "/b.mp3": []byte("song b"),
|
||||||
|
"docs/notes.txt": []byte("notes"),
|
||||||
|
})
|
||||||
|
|
||||||
|
paths := make([]string, 0, len(tt.paths))
|
||||||
|
|
||||||
|
for _, rel := range tt.paths {
|
||||||
|
paths = append(paths, filepath.Join(env.src, rel))
|
||||||
|
}
|
||||||
|
|
||||||
|
env.v.Config.Snapshots[snapshotName] = config.SnapshotConfig{
|
||||||
|
Paths: paths,
|
||||||
|
}
|
||||||
|
env.backUp(t, snapshotName)
|
||||||
|
|
||||||
|
firstID := localSnapshotID(ctx, t, env.v.Repositories, snapshotName)
|
||||||
|
stored, err := env.v.Storage.List(ctx, "")
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
// A volume that is not mounted leaves its mount point behind
|
||||||
|
// as an empty directory.
|
||||||
|
emptiedDir := filepath.Join(env.src, emptied)
|
||||||
|
require.NoError(t, os.RemoveAll(emptiedDir))
|
||||||
|
require.NoError(t, os.Mkdir(emptiedDir, 0o750))
|
||||||
|
|
||||||
|
for range 2 {
|
||||||
|
env.out.Reset()
|
||||||
|
|
||||||
|
err := env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||||
|
Prune: true,
|
||||||
|
Snapshots: []string{snapshotName},
|
||||||
|
})
|
||||||
|
require.ErrorIs(t, err, vaultik.ErrSourcePathEmpty)
|
||||||
|
assert.Contains(t, env.out.String(),
|
||||||
|
"Skipped the post-backup prune because nothing was found in ")
|
||||||
|
assert.Contains(t, env.out.String(),
|
||||||
|
emptied+" of snapshot "+snapshotName+";")
|
||||||
|
assert.NotContains(t, env.out.String(), "Running post-backup prune.")
|
||||||
|
}
|
||||||
|
|
||||||
|
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
|
||||||
|
ctx, listRecentTestLimit)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Len(t, snapshots, 3)
|
||||||
|
assert.Equal(t, firstID, snapshots[2].ID.String(),
|
||||||
|
"the snapshot that holds the files is kept")
|
||||||
|
|
||||||
|
for _, key := range stored {
|
||||||
|
_, err := env.v.Storage.Stat(ctx, key)
|
||||||
|
require.NoError(t, err, "%s is kept", key)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A backup under --prune of a configured path that is a symlink whose
|
||||||
|
// target is gone, as one into a volume that is not mounted, fails before
|
||||||
|
// anything is pruned: the snapshot that holds the files is kept with
|
||||||
|
// everything it stored. See https://git.eeqj.de/sneak/vaultik/issues/388.
|
||||||
|
func TestBackupThroughSymlinkWhoseTargetIsGoneDoesNotPrune(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const snapshotName = "photos"
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
env := newDeletedCountEnv(t, map[string][]byte{
|
||||||
|
"volume/a.jpg": []byte("photo a"),
|
||||||
|
"volume/b.jpg": []byte("photo b"),
|
||||||
|
})
|
||||||
|
|
||||||
|
target := filepath.Join(env.src, "volume")
|
||||||
|
link := filepath.Join(env.src, "photos")
|
||||||
|
require.NoError(t, os.Symlink(target, link))
|
||||||
|
env.setSnapshot(snapshotName, "photos")
|
||||||
|
env.backUp(t, snapshotName)
|
||||||
|
|
||||||
|
firstID := localSnapshotID(ctx, t, env.v.Repositories, snapshotName)
|
||||||
|
stored, err := env.v.Storage.List(ctx, "")
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
// The volume is not mounted, so the symlink points at nothing.
|
||||||
|
require.NoError(t, os.RemoveAll(target))
|
||||||
|
|
||||||
|
env.out.Reset()
|
||||||
|
err = env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||||
|
Prune: true,
|
||||||
|
Snapshots: []string{snapshotName},
|
||||||
|
})
|
||||||
|
require.ErrorIs(t, err, os.ErrNotExist)
|
||||||
|
assert.Contains(t, err.Error(), link)
|
||||||
|
assert.NotContains(t, env.out.String(), "Running post-backup prune.")
|
||||||
|
|
||||||
|
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
|
||||||
|
ctx, listRecentTestLimit)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Len(t, snapshots, 1)
|
||||||
|
assert.Equal(t, firstID, snapshots[0].ID.String(),
|
||||||
|
"the snapshot that holds the files is kept")
|
||||||
|
|
||||||
|
for _, key := range stored {
|
||||||
|
_, err := env.v.Storage.Stat(ctx, key)
|
||||||
|
require.NoError(t, err, "%s is kept", key)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A backup under --skip-errors --prune that both skipped a file it could
|
||||||
|
// not read and found nothing in a configured path names the empty path
|
||||||
|
// too when it skips the prune. See
|
||||||
|
// https://git.eeqj.de/sneak/vaultik/issues/388.
|
||||||
|
func TestBackupWithEmptySourcePathAndUnreadableFileNamesBoth(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const snapshotName = "media"
|
||||||
|
|
||||||
|
env := newDeletedCountEnv(t, map[string][]byte{
|
||||||
|
"docs/notes.txt": []byte("notes"),
|
||||||
|
})
|
||||||
|
require.NoError(t, os.Mkdir(filepath.Join(env.src, "music"), 0o750))
|
||||||
|
env.v.Config.Snapshots[snapshotName] = config.SnapshotConfig{
|
||||||
|
Paths: []string{
|
||||||
|
filepath.Join(env.src, "docs"),
|
||||||
|
filepath.Join(env.src, "music"),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
// A mode does not lock out root, who runs the tests, so opening the
|
||||||
|
// file is made to fail instead.
|
||||||
|
env.v.Fs = &openFailFs{
|
||||||
|
path: filepath.Join(env.src, "docs", "notes.txt"),
|
||||||
|
err: os.ErrPermission,
|
||||||
|
}
|
||||||
|
|
||||||
|
err := env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||||
|
Prune: true,
|
||||||
|
SkipErrors: true,
|
||||||
|
Snapshots: []string{snapshotName},
|
||||||
|
})
|
||||||
|
require.ErrorIs(t, err, vaultik.ErrFilesUnreadable)
|
||||||
|
assert.Contains(t, env.out.String(),
|
||||||
|
"Skipped the post-backup prune because 1 file(s) could not be read")
|
||||||
|
assert.Contains(t, env.out.String(),
|
||||||
|
"Skipped the post-backup prune because nothing was found in ")
|
||||||
|
assert.Contains(t, env.out.String(), "music of snapshot "+snapshotName+";")
|
||||||
|
assert.NotContains(t, env.out.String(), "Running post-backup prune.")
|
||||||
|
}
|
||||||
|
|
||||||
|
// A backup under --prune that finds something in each configured path
|
||||||
|
// prunes the previous snapshot, also when a path is a single file. See
|
||||||
|
// https://git.eeqj.de/sneak/vaultik/issues/388.
|
||||||
|
func TestBackupWithNoEmptySourcePathPrunes(t *testing.T) {
|
||||||
|
log.Initialize(log.Config{})
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const (
|
||||||
|
snapshotName = "album"
|
||||||
|
file = "pictures/one.jpg" // relative to the source directory
|
||||||
|
)
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
path string // relative to the source directory
|
||||||
|
}{
|
||||||
|
{name: "directory", path: filepath.Dir(file)},
|
||||||
|
{name: "single file", path: file},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
ctx := context.Background()
|
||||||
|
env := newDeletedCountEnv(t, map[string][]byte{
|
||||||
|
file: []byte("picture"),
|
||||||
|
})
|
||||||
|
env.setSnapshot(snapshotName, tt.path)
|
||||||
|
env.backUp(t, snapshotName)
|
||||||
|
|
||||||
|
firstID := localSnapshotID(ctx, t, env.v.Repositories, snapshotName)
|
||||||
|
|
||||||
|
require.NoError(t, env.v.CreateSnapshot(&vaultik.SnapshotCreateOptions{
|
||||||
|
Prune: true,
|
||||||
|
Snapshots: []string{snapshotName},
|
||||||
|
}))
|
||||||
|
|
||||||
|
snapshots, err := env.v.Repositories.Snapshots.ListRecent(
|
||||||
|
ctx, listRecentTestLimit)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.Len(t, snapshots, 1, "the previous snapshot is pruned")
|
||||||
|
assert.NotEqual(t, firstID, snapshots[0].ID.String())
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user