Let prune delete old .partial uploads and remote info count them (closes #466) #474
No files matched your search
@@ -523,10 +523,10 @@ deleted. Step (3) also deletes each directory under `metadata/` that has no
|
||||
`prune_grace_period` and its snapshot is not in the local index. An object the
|
||||
listing gives no time for keeps its directory, and a directory whose name is not
|
||||
a remote key is left alone. Step (3) also deletes each `.partial` object under
|
||||
`metadata/` and `blobs/` that an upload cut off part-way left, once it is older
|
||||
than `prune_grace_period`. Step (3) reads all remote manifests — network cost
|
||||
scales with the number of snapshots. `snapshot create --prune` runs the same
|
||||
cleanup automatically; this is the manual entry point for the same work. If the
|
||||
`metadata/` and `blobs/` left by a killed upload, once it is older than
|
||||
`prune_grace_period`. Step (3) reads all remote manifests — network cost scales
|
||||
with the number of snapshots. `snapshot create --prune` runs the same cleanup
|
||||
automatically; this is the manual entry point for the same work. If the
|
||||
destination store cannot be listed, or the local index holds snapshots and the
|
||||
destination answers that it has no `vaultik-destination.txt` (an unplugged
|
||||
volume's empty mount point), prune fails and deletes nothing. `snapshot purge`
|
||||
@@ -559,9 +559,9 @@ in `unreadable_manifests` and counts the manifests under skipped names in
|
||||
`skipped_manifest_count`. An unreadable manifest also leaves its snapshot's blob
|
||||
count and blob size unknown, `null` in `--json`. A file under `blobs/` whose
|
||||
name is not a blob hash is not counted as a blob, since `prune` never deletes
|
||||
it. A `.partial` object an upload cut off part-way left under `metadata/` or
|
||||
`blobs/` is not counted as a blob either. Such objects are counted on their own
|
||||
line, and in `partial_upload_count` and `partial_upload_size` in `--json`.
|
||||
it. A leftover `.partial` object under `metadata/` or `blobs/` is not counted as
|
||||
a blob either. Such objects are counted on their own line, and in
|
||||
`partial_upload_count` and `partial_upload_size` in `--json`.
|
||||
|
||||
- `--json`: Output as JSON
|
||||
|
||||
@@ -766,20 +766,20 @@ Run `vaultik config init` to generate a fully commented config file; a complete
|
||||
annotated example also lives in
|
||||
[`configs/config.example.yml`](configs/config.example.yml). Key fields:
|
||||
|
||||
| Field | Default | Description |
|
||||
| -------------------- | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `age_recipients` | (required by `snapshot create`) | Age public keys for encryption. Other commands run without one, so a machine that only restores can leave it empty |
|
||||
| `age_secret_key` | (unset) | Age private key for decryption (`snapshot restore`, `snapshot files`, `snapshot verify --deep`). Setting it in the config file places the private key on the backed-up host, defeating the public-key-only design (see "why" above). Prefer the `VAULTIK_AGE_SECRET_KEY` environment variable, supplied only on the machine you restore from. |
|
||||
| `snapshots` | (required) | Named snapshot definitions with paths and excludes |
|
||||
| `storage_url` | | Storage backend URL (`s3://`, `file://`, `rclone://`) |
|
||||
| `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) |
|
||||
| `exclude` | | Global exclude patterns (applied to all snapshots) |
|
||||
| `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
|
||||
| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` |
|
||||
| `compression_level` | `3` | zstd compression level (1-19) |
|
||||
| `prune_grace_period` | `24h` | How old a blob that no snapshot lists must be before `prune` deletes it, and how old every object in a directory under `metadata/` with no manifest must be before `prune` deletes that directory. It also sets how old a `.partial` object a killed upload left must be before `prune` deletes it. A backup uploads its blobs and its `db.zst.age` before its manifest, so on a destination shared between hosts or users this must be longer than the longest backup run (see [locking](#locking)). It does not protect a blob a backup reuses from an earlier run that stopped before its manifest: that blob keeps the time of its first upload, so a prune can delete it. A duration such as `36h` or `90m`; `0s` deletes such blobs and directories at once |
|
||||
| `hostname` | system hostname | Hostname used in snapshot IDs |
|
||||
| `index_path` | platform data dir | Local SQLite index path |
|
||||
| Field | Default | Description |
|
||||
| -------------------- | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `age_recipients` | (required by `snapshot create`) | Age public keys for encryption. Other commands run without one, so a machine that only restores can leave it empty |
|
||||
| `age_secret_key` | (unset) | Age private key for decryption (`snapshot restore`, `snapshot files`, `snapshot verify --deep`). Setting it in the config file places the private key on the backed-up host, defeating the public-key-only design (see "why" above). Prefer the `VAULTIK_AGE_SECRET_KEY` environment variable, supplied only on the machine you restore from. |
|
||||
| `snapshots` | (required) | Named snapshot definitions with paths and excludes |
|
||||
| `storage_url` | | Storage backend URL (`s3://`, `file://`, `rclone://`) |
|
||||
| `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) |
|
||||
| `exclude` | | Global exclude patterns (applied to all snapshots) |
|
||||
| `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
|
||||
| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` |
|
||||
| `compression_level` | `3` | zstd compression level (1-19) |
|
||||
| `prune_grace_period` | `24h` | How old a blob that no snapshot lists must be before `prune` deletes it, and how old every object in a directory under `metadata/` with no manifest must be before `prune` deletes that directory. `prune` also deletes a `.partial` object left by a killed upload only once it is this old. A backup uploads its blobs and its `db.zst.age` before its manifest, so on a destination shared between hosts or users this must be longer than the longest backup run (see [locking](#locking)). It does not protect a blob a backup reuses from an earlier run that stopped before its manifest: that blob keeps the time of its first upload, so a prune can delete it. A duration such as `36h` or `90m`; `0s` deletes such blobs and directories at once |
|
||||
| `hostname` | system hostname | Hostname used in snapshot IDs |
|
||||
| `index_path` | platform data dir | Local SQLite index path |
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -333,8 +333,8 @@ compression_level: 5
|
||||
# How old a blob that no snapshot lists must be before prune deletes it, and
|
||||
# how old every object in a directory under metadata/ with no manifest must
|
||||
# be before prune deletes that directory
|
||||
# It is also how old a .partial object a killed upload left must be before
|
||||
# prune deletes it
|
||||
# prune also deletes a .partial object left by a killed upload only once it is
|
||||
# this old
|
||||
# A backup uploads its blobs and its db.zst.age before its manifest, so on a
|
||||
# destination shared between hosts or users this must be longer than the
|
||||
# longest backup run
|
||||
|
||||
@@ -329,8 +329,8 @@ The prune operation is safe because:
|
||||
1. It keeps every blob listed in any snapshot's manifest and deletes only blobs
|
||||
that no manifest references. Under `metadata/` it deletes only a directory
|
||||
that has no `manifest.json.zst`, whose name is a remote key, and that no
|
||||
local snapshot record claims. It also deletes the `.partial` objects that
|
||||
uploads cut off part-way leave, which no backup or restore reads
|
||||
local snapshot record claims. It also deletes the `.partial` objects left by
|
||||
killed uploads, which no backup or restore reads
|
||||
2. Manifests are unencrypted and can be read without keys
|
||||
3. If any manifest cannot be downloaded or decoded, prune deletes nothing and
|
||||
exits with an error, rather than treating that snapshot's blobs as
|
||||
|
||||
@@ -206,8 +206,8 @@ type Config struct {
|
||||
// directory. A backup writes its manifest only once it has uploaded
|
||||
// all its blobs and its db.zst.age, so a younger blob or directory may
|
||||
// belong to a backup still running, on this host or another sharing
|
||||
// the destination store. It is also how old a `.partial` object a
|
||||
// killed upload left must be before a prune deletes it.
|
||||
// the destination store. A prune also deletes a `.partial` object left
|
||||
// by a killed upload only once it is this old.
|
||||
PruneGracePeriod time.Duration `yaml:"prune_grace_period"`
|
||||
|
||||
// StorageURL specifies the storage backend using a URL format.
|
||||
|
||||
Reference in new issue
Block a user