Compare commits

..
Author SHA1 Message Date
sneak 7cf82cc1d2 Report snapshot blobs missing from the destination in remote info (closes #425)
check / check (push) Waiting to run
remote info now counts the blobs that a manifest lists but the
destination does not hold, with their total size, and names each
snapshot that lists one, since it cannot be restored. When any blob is
missing the command exits non-zero after writing its report. --json
gives the figures in missing_blob_count, missing_blob_size and
snapshots_missing_blobs.

Model: opus-5-5
2026-11-19 06:54:34 +00:00
22 changed files with 335 additions and 564 deletions

No files matched your search

+48 -54
View File
@@ -344,29 +344,27 @@ In such a config, an unknown key or a value of the wrong type is refused.
orphaned blobs older than `prune_grace_period` from remote storage, and delete orphaned blobs older than `prune_grace_period` from remote storage, and delete
each directory under `metadata/` that has no `manifest.json.zst` once every each directory under `metadata/` that has no `manifest.json.zst` once every
object in it is older than `prune_grace_period` and its snapshot is not in the object in it is older than `prune_grace_period` and its snapshot is not in the
local index, and delete each `.partial` object older than local index, as `prune` does. By default keeps only the latest snapshot per
`prune_grace_period`, as `prune` does. By default keeps only the latest name; use `--keep-newer-than` for a rolling window. When `--skip-errors`
snapshot per name; use `--keep-newer-than` for a rolling window. When skipped a file or directory it could not read or list, the prune does not run
`--skip-errors` skipped a file or directory it could not read or list, the and every older snapshot is kept, since one of them may be the only one that
prune does not run and every older snapshot is kept, since one of them may be holds what was skipped; the command says so and exits non-zero. When the
the only one that holds what was skipped; the command says so and exits backup found nothing in a configured path, as in the mount point of a volume
non-zero. When the backup found nothing in a configured path, as in the mount that is not mounted, the prune does not run either: every older snapshot is
point of a volume that is not mounted, the prune does not run either: every kept, and the command names the path and exits non-zero, on every run until
older snapshot is kept, and the command names the path and exits non-zero, on the path holds something again. Take a path that is empty on purpose out of
every run until the path holds something again. Take a path that is empty on the config, or remove the older snapshots by hand with `snapshot purge`. When
purpose out of the config, or remove the older snapshots by hand with an earlier snapshot of the same name in the local index found a volume mounted
`snapshot purge`. When an earlier snapshot of the same name in the local index at or below a configured path (a directory on a different filesystem from its
found a volume mounted at or below a configured path (a directory on a parent, with an entry in it that the backup does not exclude) and this backup
different filesystem from its parent, with an entry in it that the backup does does not, the prune does not run either: every older snapshot is kept, and the
not exclude) and this backup does not, the prune does not run either: every command names the directory and exits non-zero, on every run until the volume
older snapshot is kept, and the command names the directory and exits is mounted again. A backup also does not find the volume when it finds no such
non-zero, on every run until the volume is mounted again. A backup also does entry in its directory, as when an automounted volume cannot be mounted, or
not find the volume when it finds no such entry in its directory, as when an when that directory is no longer backed up, because it is excluded or its
automounted volume cannot be mounted, or when that directory is no longer configured path was removed. If any of this is on purpose, as for a retired
backed up, because it is excluded or its configured path was removed. If any volume, remove the older snapshots by hand with `snapshot purge`. A file
of this is on purpose, as for a retired volume, remove the older snapshots by removed during the backup does not stop the prune.
hand with `snapshot purge`. A file removed during the backup does not stop the
prune.
- `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than this - `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than this
duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`). Without duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`). Without
`--prune`, or with a value that is not a valid duration, the command fails `--prune`, or with a value that is not a valid duration, the command fails
@@ -522,16 +520,14 @@ deleted. Step (3) also deletes each directory under `metadata/` that has no
`db.zst.age` but before its manifest, once every object in it is older than `db.zst.age` but before its manifest, once every object in it is older than
`prune_grace_period` and its snapshot is not in the local index. An object the `prune_grace_period` and its snapshot is not in the local index. An object the
listing gives no time for keeps its directory, and a directory whose name is not listing gives no time for keeps its directory, and a directory whose name is not
a remote key is left alone. Step (3) also deletes each `.partial` object under a remote key is left alone. Step (3) reads all remote manifests — network cost
`metadata/` and `blobs/` that an upload cut off part-way left, once it is older
than `prune_grace_period`. Step (3) reads all remote manifests — network cost
scales with the number of snapshots. `snapshot create --prune` runs the same scales with the number of snapshots. `snapshot create --prune` runs the same
cleanup automatically; this is the manual entry point for the same work. If the cleanup automatically; this is the manual entry point for the same work. If the
destination store cannot be listed, or the local index holds snapshots and the destination store cannot be listed, or the local index holds snapshots and the
destination answers that it has no `vaultik-destination.txt` (an unplugged destination answers that it has no `vaultik-destination.txt` (an unplugged
volume's empty mount point), prune fails and deletes nothing. `snapshot purge` volume's empty mount point), prune fails and deletes nothing. `snapshot purge`
does the same. If a blob, such a directory or such a `.partial` object cannot be does the same. If a blob or such a directory cannot be deleted, the command (and
deleted, the command (and `snapshot create --prune`) exits non-zero. `snapshot create --prune`) exits non-zero.
- `--force`: Skip confirmation prompt - `--force`: Skip confirmation prompt
- `--json`: Output stats as JSON. Also skips the confirmation prompt, as - `--json`: Output stats as JSON. Also skips the confirmation prompt, as
@@ -541,10 +537,7 @@ deleted, the command (and `snapshot create --prune`) exits non-zero.
gives no time for it, is counted in `blobs_too_new` and not in `blobs_found`. gives no time for it, is counted in `blobs_too_new` and not in `blobs_found`.
A directory under `metadata/` with no manifest is counted in A directory under `metadata/` with no manifest is counted in
`metadata_dirs_deleted` once deleted, or in `metadata_dirs_failed` if it `metadata_dirs_deleted` once deleted, or in `metadata_dirs_failed` if it
cannot be. A `.partial` object is counted in `partial_uploads_deleted` once cannot be.
deleted, or in `partial_uploads_failed` if it cannot be. One kept for being
newer than `prune_grace_period`, or of unknown age, is counted in
`partial_uploads_too_new`.
**`info`**: Display system configuration, storage settings, encryption **`info`**: Display system configuration, storage settings, encryption
recipients, and local database statistics. recipients, and local database statistics.
@@ -559,9 +552,13 @@ in `unreadable_manifests` and counts the manifests under skipped names in
`skipped_manifest_count`. An unreadable manifest also leaves its snapshot's blob `skipped_manifest_count`. An unreadable manifest also leaves its snapshot's blob
count and blob size unknown, `null` in `--json`. A file under `blobs/` whose count and blob size unknown, `null` in `--json`. A file under `blobs/` whose
name is not a blob hash is not counted as a blob, since `prune` never deletes name is not a blob hash is not counted as a blob, since `prune` never deletes
it. A `.partial` object an upload cut off part-way left under `metadata/` or it.
`blobs/` is not counted as a blob either. Such objects are counted on their own
line, and in `partial_upload_count` and `partial_upload_size` in `--json`. A blob that a manifest lists but the destination does not hold is reported as
missing, with a count and size. The report names each snapshot that lists a
missing blob, since that snapshot cannot be restored. The command then exits
non-zero. `--json` gives these in `missing_blob_count`, `missing_blob_size` and
`snapshots_missing_blobs`.
- `--json`: Output as JSON - `--json`: Output as JSON
@@ -620,9 +617,8 @@ with a server-side move (local and sftp among them) are written under a
temporary name ending in `.partial` and moved into place. Rclone remotes without temporary name ending in `.partial` and moved into place. Rclone remotes without
one are written in place: where such a remote shows a file while it is still one are written in place: where such a remote shows a file while it is still
being written, a killed upload can leave a truncated object under its name, being written, a killed upload can leave a truncated object under its name,
which a later backup takes for complete. A backup ignores a leftover `.partial` which a later backup takes for complete. A leftover `.partial` file is ignored
file. `prune` deletes it once it is older than `prune_grace_period`, and and can be deleted.
`remote info` counts it.
A destination shared between hosts or users, each with its own local index, A destination shared between hosts or users, each with its own local index,
relies on `prune_grace_period` to keep one's prune from deleting the blobs of relies on `prune_grace_period` to keep one's prune from deleting the blobs of
@@ -731,8 +727,6 @@ format does and does not protect.
5. Delete each directory under `metadata/` with no manifest whose objects are 5. Delete each directory under `metadata/` with no manifest whose objects are
all older than `prune_grace_period`, unless its snapshot is in the local all older than `prune_grace_period`, unless its snapshot is in the local
index index
6. Delete each `.partial` object under `metadata/` and `blobs/` older than
`prune_grace_period`
### chunking and deduplication ### chunking and deduplication
@@ -766,20 +760,20 @@ Run `vaultik config init` to generate a fully commented config file; a complete
annotated example also lives in annotated example also lives in
[`configs/config.example.yml`](configs/config.example.yml). Key fields: [`configs/config.example.yml`](configs/config.example.yml). Key fields:
| Field | Default | Description | | Field | Default | Description |
| -------------------- | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | -------------------- | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `age_recipients` | (required by `snapshot create`) | Age public keys for encryption. Other commands run without one, so a machine that only restores can leave it empty | | `age_recipients` | (required by `snapshot create`) | Age public keys for encryption. Other commands run without one, so a machine that only restores can leave it empty |
| `age_secret_key` | (unset) | Age private key for decryption (`snapshot restore`, `snapshot files`, `snapshot verify --deep`). Setting it in the config file places the private key on the backed-up host, defeating the public-key-only design (see "why" above). Prefer the `VAULTIK_AGE_SECRET_KEY` environment variable, supplied only on the machine you restore from. | | `age_secret_key` | (unset) | Age private key for decryption (`snapshot restore`, `snapshot files`, `snapshot verify --deep`). Setting it in the config file places the private key on the backed-up host, defeating the public-key-only design (see "why" above). Prefer the `VAULTIK_AGE_SECRET_KEY` environment variable, supplied only on the machine you restore from. |
| `snapshots` | (required) | Named snapshot definitions with paths and excludes | | `snapshots` | (required) | Named snapshot definitions with paths and excludes |
| `storage_url` | | Storage backend URL (`s3://`, `file://`, `rclone://`) | | `storage_url` | | Storage backend URL (`s3://`, `file://`, `rclone://`) |
| `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) | | `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) |
| `exclude` | | Global exclude patterns (applied to all snapshots) | | `exclude` | | Global exclude patterns (applied to all snapshots) |
| `chunk_size` | `10MB` | Average chunk size for content-defined chunking | | `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` | | `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` |
| `compression_level` | `3` | zstd compression level (1-19) | | `compression_level` | `3` | zstd compression level (1-19) |
| `prune_grace_period` | `24h` | How old a blob that no snapshot lists must be before `prune` deletes it, and how old every object in a directory under `metadata/` with no manifest must be before `prune` deletes that directory. It also sets how old a `.partial` object a killed upload left must be before `prune` deletes it. A backup uploads its blobs and its `db.zst.age` before its manifest, so on a destination shared between hosts or users this must be longer than the longest backup run (see [locking](#locking)). It does not protect a blob a backup reuses from an earlier run that stopped before its manifest: that blob keeps the time of its first upload, so a prune can delete it. A duration such as `36h` or `90m`; `0s` deletes such blobs and directories at once | | `prune_grace_period` | `24h` | How old a blob that no snapshot lists must be before `prune` deletes it, and how old every object in a directory under `metadata/` with no manifest must be before `prune` deletes that directory. A backup uploads its blobs and its `db.zst.age` before its manifest, so on a destination shared between hosts or users this must be longer than the longest backup run (see [locking](#locking)). It does not protect a blob a backup reuses from an earlier run that stopped before its manifest: that blob keeps the time of its first upload, so a prune can delete it. A duration such as `36h` or `90m`; `0s` deletes such blobs and directories at once |
| `hostname` | system hostname | Hostname used in snapshot IDs | | `hostname` | system hostname | Hostname used in snapshot IDs |
| `index_path` | platform data dir | Local SQLite index path | | `index_path` | platform data dir | Local SQLite index path |
--- ---
-2
View File
@@ -333,8 +333,6 @@ compression_level: 5
# How old a blob that no snapshot lists must be before prune deletes it, and # How old a blob that no snapshot lists must be before prune deletes it, and
# how old every object in a directory under metadata/ with no manifest must # how old every object in a directory under metadata/ with no manifest must
# be before prune deletes that directory # be before prune deletes that directory
# It is also how old a .partial object a killed upload left must be before
# prune deletes it
# A backup uploads its blobs and its db.zst.age before its manifest, so on a # A backup uploads its blobs and its db.zst.age before its manifest, so on a
# destination shared between hosts or users this must be longer than the # destination shared between hosts or users this must be longer than the
# longest backup run # longest backup run
-3
View File
@@ -314,9 +314,6 @@ The restore process doesn't use the local database. Instead:
- Delete each directory under `metadata/` that has no `manifest.json.zst` - Delete each directory under `metadata/` that has no `manifest.json.zst`
and no local snapshot record, once every object in it is older than and no local snapshot record, once every object in it is older than
`prune_grace_period` `prune_grace_period`
3. **Delete Partial Uploads**
- Delete each `.partial` object a killed upload left under `metadata/` or
`blobs/`, once it is older than `prune_grace_period`
### 6. Incomplete Snapshot Cleanup ### 6. Incomplete Snapshot Cleanup
+3 -5
View File
@@ -329,8 +329,7 @@ The prune operation is safe because:
1. It keeps every blob listed in any snapshot's manifest and deletes only blobs 1. It keeps every blob listed in any snapshot's manifest and deletes only blobs
that no manifest references. Under `metadata/` it deletes only a directory that no manifest references. Under `metadata/` it deletes only a directory
that has no `manifest.json.zst`, whose name is a remote key, and that no that has no `manifest.json.zst`, whose name is a remote key, and that no
local snapshot record claims. It also deletes the `.partial` objects that local snapshot record claims
uploads cut off part-way leave, which no backup or restore reads
2. Manifests are unencrypted and can be read without keys 2. Manifests are unencrypted and can be read without keys
3. If any manifest cannot be downloaded or decoded, prune deletes nothing and 3. If any manifest cannot be downloaded or decoded, prune deletes nothing and
exits with an error, rather than treating that snapshot's blobs as exits with an error, rather than treating that snapshot's blobs as
@@ -344,9 +343,8 @@ The prune operation is safe because:
a backup reuses from an earlier run that stopped before its manifest: the a backup reuses from an earlier run that stopped before its manifest: the
backup skips the upload of a blob already on the destination, so that blob backup skips the upload of a blob already on the destination, so that blob
keeps the time of its first upload, and a prune before the backup's manifest keeps the time of its first upload, and a prune before the backup's manifest
is written can delete it; the new snapshot then cannot be restored. It is written can delete it; the new snapshot then cannot be restored.
deletes a `.partial` object only once it is older than `prune_grace_period`, `remote nuke` has no such check
so an upload still running keeps its object. `remote nuke` has no such check
## Restoration Requirements ## Restoration Requirements
+3 -1
View File
@@ -75,7 +75,9 @@ func newRemoteInfoCommand() *cobra.Command {
- Size of all snapshot metadata (per snapshot and total) - Size of all snapshot metadata (per snapshot and total)
- Count and total size of all blobs - Count and total size of all blobs
- Count and size of referenced blobs (from all manifests) - Count and size of referenced blobs (from all manifests)
- Count and size of orphaned blobs (not referenced by any manifest)`, - Count and size of orphaned blobs (not referenced by any manifest)
- Count and size of referenced blobs missing from remote storage, and the
snapshots that list them; the command exits non-zero when any are missing`,
Args: cobra.NoArgs, Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, _ []string) error { RunE: func(cmd *cobra.Command, _ []string) error {
// Use unified config resolution // Use unified config resolution
+1 -2
View File
@@ -206,8 +206,7 @@ type Config struct {
// directory. A backup writes its manifest only once it has uploaded // directory. A backup writes its manifest only once it has uploaded
// all its blobs and its db.zst.age, so a younger blob or directory may // all its blobs and its db.zst.age, so a younger blob or directory may
// belong to a backup still running, on this host or another sharing // belong to a backup still running, on this host or another sharing
// the destination store. It is also how old a `.partial` object a // the destination store.
// killed upload left must be before a prune deletes it.
PruneGracePeriod time.Duration `yaml:"prune_grace_period"` PruneGracePeriod time.Duration `yaml:"prune_grace_period"`
// StorageURL specifies the storage backend using a URL format. // StorageURL specifies the storage backend using a URL format.
+3 -5
View File
@@ -200,11 +200,9 @@ func (f *Storer) ListStream(
return f.inner.ListStream(ctx, prefix) return f.inner.ListStream(ctx, prefix)
} }
// ListPartialUploads delegates unchanged. // DeletePartialUploads delegates unchanged.
func (f *Storer) ListPartialUploads( func (f *Storer) DeletePartialUploads(ctx context.Context, prefix string) error {
ctx context.Context, prefix string, return f.inner.DeletePartialUploads(ctx, prefix)
) ([]storage.ObjectInfo, error) {
return f.inner.ListPartialUploads(ctx, prefix)
} }
// Info delegates unchanged. // Info delegates unchanged.
+13 -26
View File
@@ -56,8 +56,8 @@ const storageDirPerm = 0o755
// truncated object at the key a later run would Stat and trust as a complete // truncated object at the key a later run would Stat and trust as a complete
// blob. The rclone backend's upload does the same on remotes with a // blob. The rclone backend's upload does the same on remotes with a
// server-side move. List and ListStream skip these files, so a leftover from // server-side move. List and ListStream skip these files, so a leftover from
// an interrupted write is never listed or trusted as a blob; prune deletes // an interrupted write is never listed or trusted as a blob; it is otherwise
// it once it is older than the prune grace period. // harmless.
const tempSuffix = ".partial" const tempSuffix = ".partial"
// Put stores data at the specified key. // Put stores data at the specified key.
@@ -248,24 +248,22 @@ func (f *FileStorer) ListStream(ctx context.Context, prefix string) <-chan Objec
return ch return ch
} }
// ListPartialUploads returns each file under prefix whose name ends in // DeletePartialUploads removes every file under prefix whose name ends in
// tempSuffix. Like List, it reads through symlinked directories and fails // tempSuffix. Like List, it reads through symlinked directories and fails
// on one whose target is missing. A missing prefix has none. // on one whose target is missing. A missing prefix has none to remove.
func (f *FileStorer) ListPartialUploads( func (f *FileStorer) DeletePartialUploads(ctx context.Context, prefix string) error {
ctx context.Context, prefix string, basePath := f.fullPath(prefix)
) ([]ObjectInfo, error) {
exists, err := f.prefixExists(prefix) exists, err := f.prefixExists(prefix)
if err != nil { if err != nil {
return nil, fmt.Errorf("checking path: %w", err) return fmt.Errorf("checking path: %w", err)
} }
if !exists { if !exists {
return nil, nil return nil
} }
var partial []ObjectInfo err = f.walk(basePath, func(path string, info os.FileInfo, err error) error {
err = f.walk(f.fullPath(prefix), func(path string, info os.FileInfo, err error) error {
if err != nil { if err != nil {
return err return err
} }
@@ -278,24 +276,13 @@ func (f *FileStorer) ListPartialUploads(
return nil return nil
} }
relPath, err := filepath.Rel(f.basePath, path) return f.fs.Remove(path)
if err != nil {
return fmt.Errorf("computing relative path: %w", err)
}
partial = append(partial, ObjectInfo{
Key: strings.ReplaceAll(relPath, string(filepath.Separator), "/"),
Size: info.Size(),
ModTime: info.ModTime(),
})
return nil
}) })
if err != nil { if err != nil {
return nil, fmt.Errorf("walking directory: %w", err) return fmt.Errorf("walking directory: %w", err)
} }
return partial, nil return nil
} }
// Info returns human-readable storage location information. // Info returns human-readable storage location information.
+8 -32
View File
@@ -121,10 +121,9 @@ func TestFileStorer_ListSkipsPartialFiles(t *testing.T) {
} }
} }
// TestFileStorer_ListPartialUploads checks that a leftover temp file is // TestFileStorer_DeletePartialUploads checks that a leftover temp file is
// listed with its size and time, that the object at the real key is not, // removed and the object at the real key is kept.
// and that Delete removes the leftover by the listed key. func TestFileStorer_DeletePartialUploads(t *testing.T) {
func TestFileStorer_ListPartialUploads(t *testing.T) {
t.Parallel() t.Parallel()
base := t.TempDir() base := t.TempDir()
@@ -134,46 +133,23 @@ func TestFileStorer_ListPartialUploads(t *testing.T) {
t.Fatalf("NewFileStorer: %v", err) t.Fatalf("NewFileStorer: %v", err)
} }
checkListPartialUploads(t, f, base)
}
// checkListPartialUploads stores an object and writes a leftover temp
// file next to it in the directory base that s stores into. It checks
// that ListPartialUploads lists only the leftover, with its size and a
// time, and that Delete removes it by the listed key and keeps the
// object.
func checkListPartialUploads(t *testing.T, s storage.Storer, base string) {
t.Helper()
ctx := context.Background() ctx := context.Background()
err := s.Put(ctx, testBlobKey, strings.NewReader("blob-bytes")) err = f.Put(ctx, testBlobKey, strings.NewReader("blob-bytes"))
if err != nil { if err != nil {
t.Fatalf("Put: %v", err) t.Fatalf("Put: %v", err)
} }
leftoverKey := testBlobKey + "-123456.partial" leftover := filepath.Join(base, testBlobKey+"-123456.partial")
leftover := filepath.Join(base, leftoverKey)
err = os.WriteFile(leftover, []byte("half"), 0o600) err = os.WriteFile(leftover, []byte("half"), 0o600)
if err != nil { if err != nil {
t.Fatalf("writing leftover temp file: %v", err) t.Fatalf("writing leftover temp file: %v", err)
} }
partial, err := s.ListPartialUploads(ctx, "blobs/") err = f.DeletePartialUploads(ctx, "blobs/")
if err != nil { if err != nil {
t.Fatalf("ListPartialUploads: %v", err) t.Fatalf("DeletePartialUploads: %v", err)
}
if len(partial) != 1 || partial[0].Key != leftoverKey ||
partial[0].Size != int64(len("half")) || partial[0].ModTime.IsZero() {
t.Fatalf("ListPartialUploads = %+v, want only %s, 4 bytes, "+
"with a time", partial, leftoverKey)
}
err = s.Delete(ctx, partial[0].Key)
if err != nil {
t.Fatalf("Delete: %v", err)
} }
_, err = os.Stat(leftover) _, err = os.Stat(leftover)
@@ -181,7 +157,7 @@ func checkListPartialUploads(t *testing.T, s storage.Storer, base string) {
t.Errorf("leftover temp file was not removed: %v", err) t.Errorf("leftover temp file was not removed: %v", err)
} }
_, err = s.Stat(ctx, testBlobKey) _, err = f.Stat(ctx, testBlobKey)
if err != nil { if err != nil {
t.Errorf("Stat of the real key: %v", err) t.Errorf("Stat of the real key: %v", err)
} }
+18 -13
View File
@@ -239,11 +239,11 @@ func TestFileStorerListFailsOnSymlinkLoop(t *testing.T) {
} }
} }
// TestFileStorerListPartialUploadsThroughSymlinkedDirectory checks that // TestFileStorerDeletePartialUploadsThroughSymlinkedDirectory checks that
// ListPartialUploads, which prune and remote nuke use, lists a leftover // DeletePartialUploads, which remote nuke uses, removes a leftover temp
// temp file behind a symlinked metadata/ or blobs/ directory under its // file behind a symlinked metadata/ or blobs/ directory and keeps the
// key, and leaves out the object at the real key. // object at the real key.
func TestFileStorerListPartialUploadsThroughSymlinkedDirectory(t *testing.T) { func TestFileStorerDeletePartialUploadsThroughSymlinkedDirectory(t *testing.T) {
t.Parallel() t.Parallel()
ctx := context.Background() ctx := context.Background()
@@ -265,24 +265,29 @@ func TestFileStorerListPartialUploadsThroughSymlinkedDirectory(t *testing.T) {
t.Fatalf("Put: %v", err) t.Fatalf("Put: %v", err)
} }
leftoverKey := key + "-123456.partial" leftover := filepath.Join(base, key+"-123456.partial")
err = os.WriteFile(filepath.Join(base, leftoverKey), err = os.WriteFile(leftover, []byte("half"), 0o600)
[]byte("half"), 0o600)
if err != nil { if err != nil {
t.Fatalf("writing leftover temp file: %v", err) t.Fatalf("writing leftover temp file: %v", err)
} }
moveBehindSymlink(t, base, dir) moveBehindSymlink(t, base, dir)
partial, err := s.ListPartialUploads(ctx, dir+"/") err = s.DeletePartialUploads(ctx, dir+"/")
if err != nil { if err != nil {
t.Fatalf("ListPartialUploads: %v", err) t.Fatalf("DeletePartialUploads: %v", err)
} }
if len(partial) != 1 || partial[0].Key != leftoverKey { // leftover runs through the link.
t.Errorf("ListPartialUploads = %+v, want only %s", _, err = os.Stat(leftover)
partial, leftoverKey) if !os.IsNotExist(err) {
t.Errorf("leftover temp file was not removed: %v", err)
}
_, err = s.Stat(ctx, key)
if err != nil {
t.Errorf("Stat of the real key: %v", err)
} }
}) })
} }
+14 -13
View File
@@ -233,28 +233,29 @@ func (r *RcloneStorer) ListStream(
return ch return ch
} }
// ListPartialUploads returns each object under prefix whose name ends in // DeletePartialUploads removes every object under prefix whose name ends
// tempSuffix. // in tempSuffix.
func (r *RcloneStorer) ListPartialUploads( func (r *RcloneStorer) DeletePartialUploads(ctx context.Context, prefix string) error {
ctx context.Context, prefix string, var partial []fs.Object
) ([]ObjectInfo, error) {
var partial []ObjectInfo
err := operations.ListFn(ctx, r.fsys, func(obj fs.Object) { err := operations.ListFn(ctx, r.fsys, func(obj fs.Object) {
key := obj.Remote() key := obj.Remote()
if strings.HasPrefix(key, prefix) && strings.HasSuffix(key, tempSuffix) { if strings.HasPrefix(key, prefix) && strings.HasSuffix(key, tempSuffix) {
partial = append(partial, ObjectInfo{ partial = append(partial, obj)
Key: key,
Size: obj.Size(),
ModTime: obj.ModTime(ctx),
})
} }
}) })
if err != nil { if err != nil {
return nil, fmt.Errorf("listing objects: %w", err) return fmt.Errorf("listing objects: %w", err)
} }
return partial, nil for _, obj := range partial {
err = obj.Remove(ctx)
if err != nil {
return fmt.Errorf("removing object: %w", err)
}
}
return nil
} }
// Info returns human-readable storage location information. // Info returns human-readable storage location information.
+31 -7
View File
@@ -205,21 +205,45 @@ func TestRcloneStorerListSkipsPartialFiles(t *testing.T) {
} }
} }
// TestRcloneStorerListPartialUploads checks that a temporary file left by // TestRcloneStorerDeletePartialUploads checks that a temporary file left
// a killed upload is listed with its size and time, that the object at // by a killed upload is removed and the object at the real key is kept.
// the real key is not, and that Delete removes the leftover by the listed
// key.
// //
//nolint:paralleltest // NewRcloneStorer installs the process-global rclone config //nolint:paralleltest // NewRcloneStorer installs the process-global rclone config
func TestRcloneStorerListPartialUploads(t *testing.T) { func TestRcloneStorerDeletePartialUploads(t *testing.T) {
dir := t.TempDir() dir := t.TempDir()
ctx := context.Background()
s, err := storage.NewRcloneStorer(context.Background(), ":local", dir) s, err := storage.NewRcloneStorer(ctx, ":local", dir)
if err != nil { if err != nil {
t.Fatalf("NewRcloneStorer: %v", err) t.Fatalf("NewRcloneStorer: %v", err)
} }
checkListPartialUploads(t, s, dir) err = s.Put(ctx, testBlobKey, strings.NewReader("blob-bytes"))
if err != nil {
t.Fatalf("Put: %v", err)
}
leftover := filepath.Join(dir, testBlobKey+"-123456.partial")
err = os.WriteFile(leftover, []byte("half"), 0o600)
if err != nil {
t.Fatalf("writing leftover temp file: %v", err)
}
err = s.DeletePartialUploads(ctx, "blobs/")
if err != nil {
t.Fatalf("DeletePartialUploads: %v", err)
}
_, err = os.Stat(leftover)
if !os.IsNotExist(err) {
t.Errorf("leftover temp file was not removed: %v", err)
}
_, err = s.Stat(ctx, testBlobKey)
if err != nil {
t.Errorf("Stat of the real key: %v", err)
}
} }
// newRcloneStorerOnWrappedLocal registers name as rclone's local backend // newRcloneStorerOnWrappedLocal registers name as rclone's local backend
+4 -6
View File
@@ -100,12 +100,10 @@ func (s *S3Storer) ListStream(ctx context.Context, prefix string) <-chan ObjectI
return ch return ch
} }
// ListPartialUploads finds none: S3 shows an object only once its upload // DeletePartialUploads has nothing to remove: S3 shows an object only once
// has completed, so an upload cut off part-way leaves none. // its upload has completed, so an upload cut off part-way leaves none.
func (s *S3Storer) ListPartialUploads( func (s *S3Storer) DeletePartialUploads(_ context.Context, _ string) error {
_ context.Context, _ string, return nil
) ([]ObjectInfo, error) {
return nil, nil
} }
// Info returns human-readable storage location information. // Info returns human-readable storage location information.
+5 -6
View File
@@ -77,12 +77,11 @@ type Storer interface {
// If an error occurs during listing, the final item will have Err set. // If an error occurs during listing, the final item will have Err set.
ListStream(ctx context.Context, prefix string) <-chan ObjectInfo ListStream(ctx context.Context, prefix string) <-chan ObjectInfo
// ListPartialUploads returns each object under prefix that an upload // DeletePartialUploads removes every object under prefix that an
// cut off part-way left under a temporary name ending in `.partial`, // upload cut off part-way left under a temporary name ending in
// with its key, size and modification time. The file and rclone // `.partial`. The file and rclone backends' List and ListStream skip
// backends' List and ListStream skip such an object; Delete removes // such an object.
// it. DeletePartialUploads(ctx context.Context, prefix string) error
ListPartialUploads(ctx context.Context, prefix string) ([]ObjectInfo, error)
// Info returns human-readable storage location information. // Info returns human-readable storage location information.
Info() Info Info() Info
@@ -258,10 +258,8 @@ func (s *stubLister) List(_ context.Context, _ string) ([]string, error) {
return nil, errStubUnused return nil, errStubUnused
} }
func (s *stubLister) ListPartialUploads( func (s *stubLister) DeletePartialUploads(_ context.Context, _ string) error {
_ context.Context, _ string, return errStubUnused
) ([]storage.ObjectInfo, error) {
return nil, errStubUnused
} }
func (s *stubLister) Info() storage.Info { func (s *stubLister) Info() storage.Info {
+68 -35
View File
@@ -9,6 +9,7 @@ import (
"github.com/dustin/go-humanize" "github.com/dustin/go-humanize"
"sneak.berlin/go/vaultik/internal/log" "sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage"
) )
// ShowInfo displays system and configuration information // ShowInfo displays system and configuration information
@@ -187,6 +188,9 @@ type SnapshotMetadataInfo struct {
// backup interrupted before its manifest upload leaves a directory // backup interrupted before its manifest upload leaves a directory
// without one, which prune does not treat as a snapshot. // without one, which prune does not treat as a snapshot.
hasManifest bool hasManifest bool
// Hashes of the blobs the snapshot's manifest lists.
blobHashes []string
} }
// RemoteInfoResult contains all remote storage information // RemoteInfoResult contains all remote storage information
@@ -216,10 +220,12 @@ type RemoteInfoResult struct {
OrphanedBlobCount *int `json:"orphaned_blob_count"` OrphanedBlobCount *int `json:"orphaned_blob_count"`
OrphanedBlobSize *int64 `json:"orphaned_blob_size"` OrphanedBlobSize *int64 `json:"orphaned_blob_size"`
// `.partial` objects under metadata/ and blobs/ that uploads cut off // Blobs that a manifest lists but the destination does not hold,
// part-way left. The blob figures above leave them out. // and the remote key of each snapshot whose manifest lists one.
PartialUploadCount int `json:"partial_upload_count"` // Those snapshots cannot be restored.
PartialUploadSize int64 `json:"partial_upload_size"` MissingBlobCount int `json:"missing_blob_count"`
MissingBlobSize int64 `json:"missing_blob_size"`
SnapshotsMissingBlobs []string `json:"snapshots_missing_blobs,omitempty"`
// Remote key of each snapshot whose manifest could not be read // Remote key of each snapshot whose manifest could not be read
UnreadableManifests []string `json:"unreadable_manifests,omitempty"` UnreadableManifests []string `json:"unreadable_manifests,omitempty"`
@@ -284,33 +290,34 @@ func (v *Vaultik) RemoteInfo(jsonOutput bool) error {
return err return err
} }
partialUploads, err := v.listPartialUploads()
if err != nil {
return err
}
for _, object := range partialUploads {
result.PartialUploadCount++
result.PartialUploadSize += object.Size
}
log.Info("Remote info complete", log.Info("Remote info complete",
"snapshots", result.TotalMetadataCount, "snapshots", result.TotalMetadataCount,
"total_blobs", result.TotalBlobCount, "total_blobs", result.TotalBlobCount,
"referenced_blobs", result.ReferencedBlobCount, "referenced_blobs", result.ReferencedBlobCount,
"missing_blobs", result.MissingBlobCount,
"unreadable_manifests", len(result.UnreadableManifests)) "unreadable_manifests", len(result.UnreadableManifests))
if jsonOutput { if jsonOutput {
enc := json.NewEncoder(v.Stdout) enc := json.NewEncoder(v.Stdout)
enc.SetIndent("", " ") enc.SetIndent("", " ")
return enc.Encode(result) err = enc.Encode(result)
if err != nil {
return fmt.Errorf("encoding JSON: %w", err)
}
} }
if showText { if showText {
v.printRemoteInfoTable(result) v.printRemoteInfoTable(result)
} }
// The report is written first, so the exit status adds to it.
if result.MissingBlobCount > 0 {
return fmt.Errorf("%w: %d blob(s) that snapshots list are missing; "+
"%d snapshot(s) cannot be restored", errProblemsInStorage,
result.MissingBlobCount, len(result.SnapshotsMissingBlobs))
}
return nil return nil
} }
@@ -420,6 +427,7 @@ func (v *Vaultik) collectReferencedBlobsFromManifests(
for _, blob := range manifest.Blobs { for _, blob := range manifest.Blobs {
referencedBlobs[blob.Hash] = blob.CompressedSize referencedBlobs[blob.Hash] = blob.CompressedSize
blobsSize += blob.CompressedSize blobsSize += blob.CompressedSize
info.blobHashes = append(info.blobHashes, blob.Hash)
} }
info.BlobCount = &blobCount info.BlobCount = &blobCount
@@ -477,6 +485,8 @@ func (v *Vaultik) scanRemoteBlobStorage(
result.TotalBlobSize += blob.Size result.TotalBlobSize += blob.Size
} }
countMissingBlobs(result, referencedBlobs, allBlobs)
// A blob named only by a manifest that could not be read, or by one // A blob named only by a manifest that could not be read, or by one
// under a skipped name, would be counted as orphaned, so the orphan // under a skipped name, would be counted as orphaned, so the orphan
// figures stay unknown. // figures stay unknown.
@@ -502,6 +512,33 @@ func (v *Vaultik) scanRemoteBlobStorage(
return nil return nil
} }
// countMissingBlobs records in result the referenced blobs that
// allBlobs, the listing of the destination, does not hold, and the
// snapshots whose manifests list one of them. It reads the snapshots
// from result.Snapshots, so populateRemoteInfoResult must run first.
func countMissingBlobs(
result *RemoteInfoResult, referencedBlobs map[string]int64,
allBlobs map[string]storage.ObjectInfo,
) {
for hash, size := range referencedBlobs {
if _, present := allBlobs[hash]; !present {
result.MissingBlobCount++
result.MissingBlobSize += size
}
}
for _, info := range result.Snapshots {
for _, hash := range info.blobHashes {
if _, present := allBlobs[hash]; !present {
result.SnapshotsMissingBlobs = append(
result.SnapshotsMissingBlobs, info.SnapshotID)
break
}
}
}
}
// printRemoteInfoTable renders the human-readable remote info output // printRemoteInfoTable renders the human-readable remote info output
func (v *Vaultik) printRemoteInfoTable(result *RemoteInfoResult) { func (v *Vaultik) printRemoteInfoTable(result *RemoteInfoResult) {
const ( const (
@@ -554,12 +591,12 @@ func (v *Vaultik) printRemoteInfoTable(result *RemoteInfoResult) {
"", "", ubytes(result.TotalMetadataSize)) "", "", ubytes(result.TotalMetadataSize))
} }
v.printRemoteBlobStorage(result) v.printBlobStorageSummary(result)
} }
// printRemoteBlobStorage renders the blob storage section of the // printBlobStorageSummary renders the blob storage section that ends
// human-readable remote info output. // the human-readable remote info output.
func (v *Vaultik) printRemoteBlobStorage(result *RemoteInfoResult) { func (v *Vaultik) printBlobStorageSummary(result *RemoteInfoResult) {
v.stdoutf("\n=== Blob Storage ===\n") v.stdoutf("\n=== Blob Storage ===\n")
v.stdoutf("Total blobs on remote: %s (%s)\n", v.stdoutf("Total blobs on remote: %s (%s)\n",
humanize.Comma(int64(result.TotalBlobCount)), humanize.Comma(int64(result.TotalBlobCount)),
@@ -567,6 +604,9 @@ func (v *Vaultik) printRemoteBlobStorage(result *RemoteInfoResult) {
v.stdoutf("Referenced by snapshots: %s (%s)\n", v.stdoutf("Referenced by snapshots: %s (%s)\n",
humanize.Comma(int64(result.ReferencedBlobCount)), humanize.Comma(int64(result.ReferencedBlobCount)),
ubytes(result.ReferencedBlobSize)) ubytes(result.ReferencedBlobSize))
v.stdoutf("Missing from remote: %s (%s)\n",
humanize.Comma(int64(result.MissingBlobCount)),
ubytes(result.MissingBlobSize))
if result.OrphanedBlobCount == nil { if result.OrphanedBlobCount == nil {
v.stdoutf("Orphaned (unreferenced): unknown "+ v.stdoutf("Orphaned (unreferenced): unknown "+
@@ -577,25 +617,18 @@ func (v *Vaultik) printRemoteBlobStorage(result *RemoteInfoResult) {
v.stdoutf("Orphaned (unreferenced): %s (%s)\n", v.stdoutf("Orphaned (unreferenced): %s (%s)\n",
humanize.Comma(int64(*result.OrphanedBlobCount)), humanize.Comma(int64(*result.OrphanedBlobCount)),
ubytes(*result.OrphanedBlobSize)) ubytes(*result.OrphanedBlobSize))
if *result.OrphanedBlobCount > 0 {
v.stdoutf("\nRun 'vaultik prune' to remove orphaned blobs.\n")
}
} }
v.stdoutf("Partial uploads: %s (%s)\n", if len(result.SnapshotsMissingBlobs) > 0 {
humanize.Comma(int64(result.PartialUploadCount)), v.stdoutf("\nSnapshots that cannot be restored (blobs missing):\n")
ubytes(result.PartialUploadSize))
var toRemove []string for _, id := range result.SnapshotsMissingBlobs {
v.stdoutf(" %s\n", id)
if result.OrphanedBlobCount != nil && *result.OrphanedBlobCount > 0 { }
toRemove = append(toRemove, "orphaned blobs")
}
if result.PartialUploadCount > 0 {
toRemove = append(toRemove, "partial uploads")
}
if len(toRemove) > 0 {
v.stdoutf("\nRun 'vaultik prune' to remove %s.\n",
strings.Join(toRemove, " and "))
} }
} }
+4 -6
View File
@@ -155,12 +155,10 @@ func (m *MockStorer) ListStream(
return ch return ch
} }
// ListPartialUploads finds none: Put stores each object under its key at // DeletePartialUploads has nothing to remove: Put stores each object
// once. // under its key at once.
func (m *MockStorer) ListPartialUploads( func (m *MockStorer) DeletePartialUploads(_ context.Context, _ string) error {
_ context.Context, _ string, return nil
) ([]storage.ObjectInfo, error) {
return nil, nil
} }
func (m *MockStorer) Info() storage.Info { func (m *MockStorer) Info() storage.Info {
+45 -212
View File
@@ -31,11 +31,6 @@ var errBlobDeleteFailed = errors.New("blob(s) could not be deleted")
var errMetadataDirDeleteFailed = errors.New( var errMetadataDirDeleteFailed = errors.New(
"metadata director(ies) with no manifest could not be deleted") "metadata director(ies) with no manifest could not be deleted")
// errPartialUploadDeleteFailed reports `.partial` objects prune could not
// delete.
var errPartialUploadDeleteFailed = errors.New(
"partial upload(s) could not be deleted")
// errObjectsLeft reports objects remote nuke still finds on the backup // errObjectsLeft reports objects remote nuke still finds on the backup
// destination store after deleting. // destination store after deleting.
var errObjectsLeft = errors.New( var errObjectsLeft = errors.New(
@@ -143,9 +138,9 @@ func (v *Vaultik) emptyDestinationStore() error {
// that fails is logged and the others still run; that object then // that fails is logged and the others still run; that object then
// shows up in the listing afterwards. // shows up in the listing afterwards.
func (v *Vaultik) deleteEverythingUnder(prefix string) ([]string, error) { func (v *Vaultik) deleteEverythingUnder(prefix string) ([]string, error) {
keys, err := v.listEverythingUnder(prefix) keys, err := v.Storage.List(v.ctx, prefix)
if err != nil { if err != nil {
return nil, err return nil, fmt.Errorf("listing %s: %w", prefix, err)
} }
for _, key := range keys { for _, key := range keys {
@@ -155,47 +150,19 @@ func (v *Vaultik) deleteEverythingUnder(prefix string) ([]string, error) {
} }
} }
return v.listEverythingUnder(prefix) // The file and rclone listings skip `.partial` objects, so the
} // deletes above never reach them.
err = v.Storage.DeletePartialUploads(v.ctx, prefix)
if err != nil {
return nil, fmt.Errorf("deleting partial uploads: %w", err)
}
// listEverythingUnder returns the key of every object under prefix, left, err := v.Storage.List(v.ctx, prefix)
// including the `.partial` objects uploads cut off part-way left, which
// the file and rclone backends' List skips.
func (v *Vaultik) listEverythingUnder(prefix string) ([]string, error) {
keys, err := v.Storage.List(v.ctx, prefix)
if err != nil { if err != nil {
return nil, fmt.Errorf("listing %s: %w", prefix, err) return nil, fmt.Errorf("listing %s: %w", prefix, err)
} }
partialUploads, err := v.Storage.ListPartialUploads(v.ctx, prefix) return left, nil
if err != nil {
return nil, fmt.Errorf("listing partial uploads under %s: %w",
prefix, err)
}
for _, object := range partialUploads {
keys = append(keys, object.Key)
}
return keys, nil
}
// listPartialUploads returns the `.partial` objects under metadata/ and
// blobs/ that uploads cut off part-way left.
func (v *Vaultik) listPartialUploads() ([]storage.ObjectInfo, error) {
var partialUploads []storage.ObjectInfo
for _, prefix := range []string{"metadata/", "blobs/"} {
objects, err := v.Storage.ListPartialUploads(v.ctx, prefix)
if err != nil {
return nil, fmt.Errorf("listing partial uploads under %s: %w",
prefix, err)
}
partialUploads = append(partialUploads, objects...)
}
return partialUploads, nil
} }
// PruneBlobsResult contains the result of a blob prune operation // PruneBlobsResult contains the result of a blob prune operation
@@ -215,19 +182,11 @@ type PruneBlobsResult struct {
// not delete. // not delete.
MetadataDirsDeleted int `json:"metadata_dirs_deleted"` MetadataDirsDeleted int `json:"metadata_dirs_deleted"`
MetadataDirsFailed int `json:"metadata_dirs_failed,omitempty"` MetadataDirsFailed int `json:"metadata_dirs_failed,omitempty"`
// PartialUploadsDeleted counts the `.partial` objects uploads cut off
// part-way left that prune deleted, and PartialUploadsFailed those it
// could not delete. PartialUploadsTooNew counts those kept for being
// newer than the prune grace period, or of unknown age.
PartialUploadsDeleted int `json:"partial_uploads_deleted"`
PartialUploadsFailed int `json:"partial_uploads_failed,omitempty"`
PartialUploadsTooNew int `json:"partial_uploads_too_new"`
} }
// Prune removes orphaned data from the local index database AND, from the // Prune removes orphaned data from the local index database AND, from the
// backup destination store, unreferenced blobs, the directories under // backup destination store, unreferenced blobs and the directories under
// metadata/ with no manifest and the `.partial` objects uploads cut off // metadata/ with no manifest. This is the single user-facing prune entry
// part-way left. This is the single user-facing prune entry
// point — the split between local and remote cleanup is an // point — the split between local and remote cleanup is an
// implementation detail. Calling code should prefer this method over // implementation detail. Calling code should prefer this method over
// PruneDatabase or PruneBlobs individually unless it specifically wants // PruneDatabase or PruneBlobs individually unless it specifically wants
@@ -255,9 +214,8 @@ func (v *Vaultik) Prune(opts *PruneOptions) error {
return v.PruneBlobs(opts) return v.PruneBlobs(opts)
} }
// PruneBlobs removes unreferenced blobs from storage, each directory // PruneBlobs removes unreferenced blobs from storage, and each directory
// under metadata/ that a backup left with no manifest, and each // under metadata/ that a backup left with no manifest.
// `.partial` object an upload cut off part-way left.
func (v *Vaultik) PruneBlobs(opts *PruneOptions) error { func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
log.Info("Starting prune operation") log.Info("Starting prune operation")
@@ -284,21 +242,14 @@ func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
return err return err
} }
partialUploads, partialTooNew, err := v.findOldPartialUploads()
if err != nil {
return err
}
result := &PruneBlobsResult{ result := &PruneBlobsResult{
BlobsFound: len(unreferencedBlobs), BlobsFound: len(unreferencedBlobs),
BlobsTooNew: tooNew, BlobsTooNew: tooNew,
PartialUploadsTooNew: partialTooNew,
} }
v.reportTooNew(result, opts) v.reportBlobsTooNew(tooNew, opts)
if len(unreferencedBlobs) == 0 && len(dirsWithoutManifest) == 0 && if len(unreferencedBlobs) == 0 && len(dirsWithoutManifest) == 0 {
len(partialUploads) == 0 {
log.Info("No unreferenced blobs found") log.Info("No unreferenced blobs found")
if opts.JSON { if opts.JSON {
@@ -312,17 +263,15 @@ func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
log.Info("Found unreferenced blobs", log.Info("Found unreferenced blobs",
"count", len(unreferencedBlobs), "total_size", ubytes(totalSize), "count", len(unreferencedBlobs), "total_size", ubytes(totalSize),
"metadata_dirs_without_manifest", len(dirsWithoutManifest), "metadata_dirs_without_manifest", len(dirsWithoutManifest))
"partial_uploads", len(partialUploads))
if !v.confirmPrune(opts, len(unreferencedBlobs), totalSize, if !v.confirmPrune(opts, len(unreferencedBlobs), totalSize,
dirsWithoutManifest, partialUploads) { dirsWithoutManifest) {
return ErrNotConfirmed return ErrNotConfirmed
} }
v.deleteUnreferencedBlobs(unreferencedBlobs, allBlobs, result) v.deleteUnreferencedBlobs(unreferencedBlobs, allBlobs, result)
v.deleteMetadataDirs(dirsWithoutManifest, result) v.deleteMetadataDirs(dirsWithoutManifest, result)
v.deletePartialUploads(partialUploads, result)
if opts.JSON { if opts.JSON {
err = v.outputPruneBlobsJSON(result) err = v.outputPruneBlobsJSON(result)
@@ -330,16 +279,9 @@ func (v *Vaultik) PruneBlobs(opts *PruneOptions) error {
return err return err
} }
} else { } else {
v.reportPruneDeletions(result, len(dirsWithoutManifest), v.reportPruneDeletions(result, len(dirsWithoutManifest))
len(partialUploads))
} }
return pruneDeleteFailures(result)
}
// pruneDeleteFailures returns an error for each kind of object prune
// could not delete, or nil when every delete succeeded.
func pruneDeleteFailures(result *PruneBlobsResult) error {
var failures []error var failures []error
if result.BlobsFailed > 0 { if result.BlobsFailed > 0 {
@@ -352,41 +294,26 @@ func pruneDeleteFailures(result *PruneBlobsResult) error {
result.MetadataDirsFailed, errMetadataDirDeleteFailed)) result.MetadataDirsFailed, errMetadataDirDeleteFailed))
} }
if result.PartialUploadsFailed > 0 {
failures = append(failures, fmt.Errorf("%d %w",
result.PartialUploadsFailed, errPartialUploadDeleteFailed))
}
return errors.Join(failures...) return errors.Join(failures...)
} }
// confirmPrune states what prune found to delete, unreferenced blobs, // confirmPrune states what prune found to delete, unreferenced blobs and
// directories under metadata/ with no manifest, naming each directory by // directories under metadata/ with no manifest, naming each directory by
// its remote key, and `.partial` objects, and, unless --force is given, // its remote key, and, unless --force is given, asks the operator to
// asks the operator to confirm. Under --json it prints nothing and does // confirm. Under --json it prints nothing and does not ask. It reports
// not ask. It reports whether to go ahead. // whether to go ahead.
func (v *Vaultik) confirmPrune( func (v *Vaultik) confirmPrune(
opts *PruneOptions, blobCount int, blobsSize int64, dirs []string, opts *PruneOptions, blobCount int, blobsSize int64, dirs []string,
partialUploads []storage.ObjectInfo,
) bool { ) bool {
if opts.JSON { if opts.JSON {
return true return true
} }
var partialSize int64
for _, object := range partialUploads {
partialSize += object.Size
}
dirCount := len(dirs) dirCount := len(dirs)
partialCount := len(partialUploads)
blobsFound := fmt.Sprintf("Found %d unreferenced blob(s) totaling %s.", blobsFound := fmt.Sprintf("Found %d unreferenced blob(s) totaling %s.",
blobCount, ubytes(blobsSize)) blobCount, ubytes(blobsSize))
dirsFound := fmt.Sprintf("Found %d metadata director(ies) left by "+ dirsFound := fmt.Sprintf("Found %d metadata director(ies) left by "+
"backups that stopped before their manifest:", dirCount) "backups that stopped before their manifest:", dirCount)
partialFound := fmt.Sprintf("Found %d partial upload(s) totaling %s, "+
"left by uploads cut off part-way.", partialCount, ubytes(partialSize))
if opts.Force { if opts.Force {
if blobCount > 0 { if blobCount > 0 {
@@ -401,10 +328,6 @@ func (v *Vaultik) confirmPrune(
} }
} }
if partialCount > 0 {
v.UI.Infof("%s", partialFound)
}
return true return true
} }
@@ -423,13 +346,16 @@ func (v *Vaultik) confirmPrune(
} }
} }
if partialCount > 0 { switch {
v.stdoutf("%s\n", partialFound) case dirCount == 0:
v.stdoutf("\nDelete %d unreferenced blob(s)? [y/N] ", blobCount)
case blobCount == 0:
v.stdoutf("\nDelete %d metadata director(ies)? [y/N] ", dirCount)
default:
v.stdoutf("\nDelete %d unreferenced blob(s) and %d metadata "+
"director(ies)? [y/N] ", blobCount, dirCount)
} }
v.stdoutf("\nDelete %s? [y/N] ",
pruneTargets(blobCount, dirCount, partialCount))
var confirm string var confirm string
// A failure to read the answer means no confirmation. // A failure to read the answer means no confirmation.
@@ -443,36 +369,9 @@ func (v *Vaultik) confirmPrune(
return true return true
} }
// pruneTargets names what the prune prompt offers to delete, such as // reportPruneDeletions says how many blobs and metadata directories
// "2 unreferenced blob(s) and 1 partial upload(s)", leaving out each kind // prune deleted, naming each kind only when it found some to delete.
// with a count of zero. func (v *Vaultik) reportPruneDeletions(result *PruneBlobsResult, dirCount int) {
func pruneTargets(blobCount, dirCount, partialCount int) string {
var targets []string
if blobCount > 0 {
targets = append(targets,
fmt.Sprintf("%d unreferenced blob(s)", blobCount))
}
if dirCount > 0 {
targets = append(targets,
fmt.Sprintf("%d metadata director(ies)", dirCount))
}
if partialCount > 0 {
targets = append(targets,
fmt.Sprintf("%d partial upload(s)", partialCount))
}
return strings.Join(targets, " and ")
}
// reportPruneDeletions says how many blobs, metadata directories and
// partial uploads prune deleted, naming each kind only when it found some
// to delete.
func (v *Vaultik) reportPruneDeletions(
result *PruneBlobsResult, dirCount, partialCount int,
) {
if result.BlobsFound > 0 { if result.BlobsFound > 0 {
v.UI.Infof("Deleted %d blob(s) totaling %s.", v.UI.Infof("Deleted %d blob(s) totaling %s.",
result.BlobsDeleted, ubytes(result.BytesFreed)) result.BlobsDeleted, ubytes(result.BytesFreed))
@@ -482,11 +381,6 @@ func (v *Vaultik) reportPruneDeletions(
v.UI.Infof("Deleted %d metadata director(ies) with no manifest.", v.UI.Infof("Deleted %d metadata director(ies) with no manifest.",
result.MetadataDirsDeleted) result.MetadataDirsDeleted)
} }
if partialCount > 0 {
v.UI.Infof("Deleted %d partial upload(s).",
result.PartialUploadsDeleted)
}
} }
// listMetadataObjects lists every object under metadata/ on the // listMetadataObjects lists every object under metadata/ on the
@@ -717,79 +611,18 @@ func (v *Vaultik) deleteMetadataDirs(
} }
} }
// reportTooNew says how many unreferenced blobs and partial uploads prune // reportBlobsTooNew says how many unreferenced blobs prune kept for being
// kept for being newer than the grace period or of unknown age. Under // newer than the grace period or of unknown age. Under --json the
// --json the document carries the counts instead. // document carries the count instead.
func (v *Vaultik) reportTooNew(result *PruneBlobsResult, opts *PruneOptions) { func (v *Vaultik) reportBlobsTooNew(tooNew int, opts *PruneOptions) {
if opts.JSON { if tooNew == 0 || opts.JSON {
return return
} }
if result.BlobsTooNew > 0 { v.UI.Infof("Kept %d unreferenced blob(s) newer than "+
v.UI.Infof("Kept %d unreferenced blob(s) newer than "+ "prune_grace_period (%s), or of unknown age: a backup still "+
"prune_grace_period (%s), or of unknown age: a backup still "+ "running may need them.",
"running may need them.", tooNew, v.Config.PruneGracePeriod)
result.BlobsTooNew, v.Config.PruneGracePeriod)
}
if result.PartialUploadsTooNew > 0 {
v.UI.Infof("Kept %d partial upload(s) newer than "+
"prune_grace_period (%s), or of unknown age: an upload may "+
"still be writing them.",
result.PartialUploadsTooNew, v.Config.PruneGracePeriod)
}
}
// findOldPartialUploads returns the `.partial` objects under metadata/
// and blobs/ that are older than the prune grace period. It also returns
// how many it left out for being newer than that, or of unknown age
// because the listing gave no time for them.
func (v *Vaultik) findOldPartialUploads() ([]storage.ObjectInfo, int, error) {
partialUploads, err := v.listPartialUploads()
if err != nil {
return nil, 0, err
}
cutoff := time.Now().UTC().Add(-v.Config.PruneGracePeriod)
var (
old []storage.ObjectInfo
tooNew int
)
for _, object := range partialUploads {
if object.ModTime.IsZero() || object.ModTime.After(cutoff) {
tooNew++
continue
}
old = append(old, object)
}
return old, tooNew, nil
}
// deletePartialUploads deletes the given `.partial` objects and counts
// in the result those deleted and those it could not delete.
func (v *Vaultik) deletePartialUploads(
partialUploads []storage.ObjectInfo, result *PruneBlobsResult,
) {
for _, object := range partialUploads {
err := v.Storage.Delete(v.ctx, object.Key)
if err != nil {
log.Error("Failed to delete partial upload",
"key", object.Key, "error", err)
result.PartialUploadsFailed++
continue
}
log.Info("Deleted partial upload", "key", object.Key)
result.PartialUploadsDeleted++
}
} }
// deleteUnreferencedBlobs deletes the given blobs from storage and // deleteUnreferencedBlobs deletes the given blobs from storage and
@@ -1,126 +0,0 @@
package vaultik_test
import (
"bytes"
"context"
"encoding/json"
"os"
"path/filepath"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/vaultik/internal/config"
"sneak.berlin/go/vaultik/internal/log"
"sneak.berlin/go/vaultik/internal/storage"
"sneak.berlin/go/vaultik/internal/ui"
"sneak.berlin/go/vaultik/internal/vaultik"
)
// partialUploadGracePeriod is the prune grace period of the vaultik
// newPartialUploadStore returns.
const partialUploadGracePeriod = time.Hour
// newPartialUploadStore returns a vaultik on a file:// destination
// holding the `.partial` files killed uploads leave: one under blobs/
// and one under metadata/ older than the prune grace period, and one
// under blobs/ newer than that. It also returns the buffer the vaultik
// writes its output to, the paths of the old files and the path of the
// new one.
func newPartialUploadStore(
t *testing.T,
) (*vaultik.Vaultik, *bytes.Buffer, []string, string) {
t.Helper()
storeDir := t.TempDir()
store, err := storage.NewFileStorer(storeDir)
require.NoError(t, err)
hash := testBlobHashA
oldFiles := []string{
filepath.Join(storeDir, "blobs", hash[:2], hash[2:4],
hash+"-123456.partial"),
filepath.Join(storeDir, metadataObjectKey(
"testhost_home_2026-01-01T00:00:00Z", "db.zst.age-123456.partial")),
}
newFile := filepath.Join(storeDir, "blobs", hash[:2], hash[2:4],
hash+"-654321.partial")
beforeGracePeriod := time.Now().Add(-partialUploadGracePeriod - time.Minute)
for _, path := range append([]string{newFile}, oldFiles...) {
require.NoError(t, os.MkdirAll(filepath.Dir(path), 0o750))
require.NoError(t, os.WriteFile(path, []byte("half an upload"), 0o600))
}
for _, path := range oldFiles {
require.NoError(t, os.Chtimes(path, beforeGracePeriod, beforeGracePeriod))
}
stdout := &bytes.Buffer{}
v := &vaultik.Vaultik{
Config: &config.Config{PruneGracePeriod: partialUploadGracePeriod},
Storage: store,
Stdout: stdout,
UI: ui.NewWithColor(stdout, false),
}
v.SetContext(context.Background())
return v, stdout, oldFiles, newFile
}
// TestPruneDeletesPartialUploadsOlderThanGracePeriod checks that prune
// deletes a `.partial` file a killed upload left under blobs/ or
// metadata/ once it is older than the grace period, keeps a newer one,
// and counts both in the --json document.
func TestPruneDeletesPartialUploadsOlderThanGracePeriod(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
v, stdout, oldFiles, newFile := newPartialUploadStore(t)
require.NoError(t, v.PruneBlobs(&vaultik.PruneOptions{JSON: true}))
for _, path := range oldFiles {
_, err := os.Stat(path)
require.ErrorIs(t, err, os.ErrNotExist, "%s must be deleted", path)
}
_, err := os.Stat(newFile)
require.NoError(t, err, "a partial upload newer than the grace period "+
"must be kept")
var result vaultik.PruneBlobsResult
require.NoError(t, json.Unmarshal(stdout.Bytes(), &result))
assert.Equal(t, 2, result.PartialUploadsDeleted)
assert.Equal(t, 1, result.PartialUploadsTooNew)
}
// TestRemoteInfoCountsPartialUploads checks that remote info counts
// every `.partial` file under blobs/ and metadata/, whatever its age, and
// advises running prune.
func TestRemoteInfoCountsPartialUploads(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
v, stdout, _, _ := newPartialUploadStore(t)
require.NoError(t, v.RemoteInfo(true))
var result vaultik.RemoteInfoResult
require.NoError(t, json.Unmarshal(stdout.Bytes(), &result))
assert.Equal(t, 3, result.PartialUploadCount)
assert.Equal(t, int64(3*len("half an upload")), result.PartialUploadSize)
assert.Equal(t, 0, result.TotalBlobCount)
stdout.Reset()
require.NoError(t, v.RemoteInfo(false))
text := stdout.String()
assert.Contains(t, text, "Partial uploads: 3 (")
assert.Contains(t, text, "Run 'vaultik prune' to remove partial uploads.")
}
+60
View File
@@ -83,6 +83,7 @@ func TestRemoteInfo_UnreadableManifestLeavesSnapshotBlobsUnknown(t *testing.T) {
env := newListEnv(t) env := newListEnv(t)
readableKey := env.addRemote(t, listRemoteID, readableKey := env.addRemote(t, listRemoteID,
time.Date(2026, 3, 2, 0, 0, 0, 0, time.UTC)) time.Date(2026, 3, 2, 0, 0, 0, 0, time.UTC))
addBlob(t, env.store.testStorer, testBlobHashA)
unreadableKey := snapshot.RemoteSnapshotKey(listLocalID) unreadableKey := snapshot.RemoteSnapshotKey(listLocalID)
require.NoError(t, env.store.Put(context.Background(), require.NoError(t, env.store.Put(context.Background(),
@@ -270,6 +271,7 @@ func TestRemoteInfo_StatusLineUsesUIMarker(t *testing.T) {
env := newListEnv(t) env := newListEnv(t)
env.addRemote(t, listRemoteID, time.Date(2026, 3, 2, 0, 0, 0, 0, time.UTC)) env.addRemote(t, listRemoteID, time.Date(2026, 3, 2, 0, 0, 0, 0, time.UTC))
addBlob(t, env.store.testStorer, testBlobHashA)
const statusLine = ui.Marker + " Scanning snapshot metadata...\n" const statusLine = ui.Marker + " Scanning snapshot metadata...\n"
@@ -281,3 +283,61 @@ func TestRemoteInfo_StatusLineUsesUIMarker(t *testing.T) {
require.NoError(t, env.v.RemoteInfo(false)) require.NoError(t, env.v.RemoteInfo(false))
assert.NotContains(t, env.stdout.String(), "Scanning snapshot metadata") assert.NotContains(t, env.stdout.String(), "Scanning snapshot metadata")
} }
// TestRemoteInfo_ReportsBlobsMissingFromDestination checks that a blob a
// manifest lists but the destination does not hold is counted as
// missing, that the snapshot listing it is named, and that remote info
// then fails after writing its report.
func TestRemoteInfo_ReportsBlobsMissingFromDestination(t *testing.T) {
log.Initialize(log.Config{})
t.Parallel()
env := newListEnv(t)
// The first snapshot lists blob A, which is not stored. The second
// lists blob B, which is.
brokenKey := env.addRemote(t, listRemoteID,
time.Date(2026, 3, 2, 0, 0, 0, 0, time.UTC))
addManifest(t, env.store.testStorer, listLocalID, []string{testBlobHashB})
addBlob(t, env.store.testStorer, testBlobHashB)
const wantError = "1 blob(s) that snapshots list are missing; " +
"1 snapshot(s) cannot be restored"
err := env.v.RemoteInfo(false)
require.ErrorContains(t, err, wantError)
text := env.stdout.String()
assert.Contains(t, text, "Missing from remote: 1 (5.0 MB)")
assert.Contains(t, text, "Snapshots that cannot be restored (blobs missing):\n"+
" "+brokenKey+"\n")
assert.NotContains(t, text, snapshot.RemoteSnapshotKey(listLocalID)+"\n")
env.stdout.Reset()
require.ErrorContains(t, env.v.RemoteInfo(true), wantError)
var doc map[string]any
require.NoError(t, json.Unmarshal(env.stdout.Bytes(), &doc))
assert.InDelta(t, 1, doc["missing_blob_count"], 0)
assert.InDelta(t, fiveMegabytes, doc["missing_blob_size"], 0)
assert.Equal(t, []any{brokenKey}, doc["snapshots_missing_blobs"])
// --quiet drops the report, not the failure.
env.stdout.Reset()
env.v.UI.SetQuiet(true)
require.ErrorContains(t, env.v.RemoteInfo(false), wantError)
assert.Empty(t, env.stdout.String())
env.v.UI.SetQuiet(false)
// With blob A stored again nothing is missing.
addBlob(t, env.store.testStorer, testBlobHashA)
env.stdout.Reset()
require.NoError(t, env.v.RemoteInfo(true))
doc = nil
require.NoError(t, json.Unmarshal(env.stdout.Bytes(), &doc))
assert.InDelta(t, 0, doc["missing_blob_count"], 0)
assert.InDelta(t, 0, doc["missing_blob_size"], 0)
assert.NotContains(t, doc, "snapshots_missing_blobs")
}
+4 -6
View File
@@ -136,12 +136,10 @@ func (s *testStorer) ListStream(
return ch return ch
} }
// ListPartialUploads finds none: Put stores each object under its key at // DeletePartialUploads has nothing to remove: Put stores each object
// once. // under its key at once.
func (s *testStorer) ListPartialUploads( func (s *testStorer) DeletePartialUploads(_ context.Context, _ string) error {
_ context.Context, _ string, return nil
) ([]storage.ObjectInfo, error) {
return nil, nil
} }
func (s *testStorer) Info() storage.Info { func (s *testStorer) Info() storage.Info {
+1
View File
@@ -349,6 +349,7 @@ func TestSnapshotListAndPurgePrintSizesLikeRemoteInfo(t *testing.T) {
env.addRemote(t, listLocalID, start) env.addRemote(t, listLocalID, start)
require.NoError(t, env.v.Repositories.Snapshots.UpdateCounts( require.NoError(t, env.v.Repositories.Snapshots.UpdateCounts(
context.Background(), nil, listLocalID, 0, 0, 0, 0, fiveMegabytes)) context.Background(), nil, listLocalID, 0, 0, 0, 0, fiveMegabytes))
addBlob(t, env.store.testStorer, testBlobHashA)
require.NoError(t, env.v.RemoteInfo(false)) require.NoError(t, env.v.RemoteInfo(false))
assert.Contains(t, findTableRow(t, env.stdout.String(), remoteKey[:16]), assert.Contains(t, findTableRow(t, env.stdout.String(), remoteKey[:16]),