Let prune delete old .partial uploads and remote info count them (closes #466)
check / check (push) Waiting to run
check / check (push) Waiting to run
A killed upload leaves a `.partial` object that only remote nuke removed, and that remote info never showed. The storage interface's DeletePartialUploads becomes ListPartialUploads, which returns each such object's key, size and time; Delete removes it. prune deletes those under metadata/ and blobs/ once older than prune_grace_period, and reports kept, deleted and failed counts. remote info counts them on their own line and in --json. remote nuke lists and deletes them like any other object, and now names any it could not delete. Model: opus-5-5
This commit is contained in:
1 parent
d747ddef3a
commit
52fada5236
19 files changed
+570
-205
No files matched your search
@@ -344,27 +344,29 @@ In such a config, an unknown key or a value of the wrong type is refused.
|
||||
orphaned blobs older than `prune_grace_period` from remote storage, and delete
|
||||
each directory under `metadata/` that has no `manifest.json.zst` once every
|
||||
object in it is older than `prune_grace_period` and its snapshot is not in the
|
||||
local index, as `prune` does. By default keeps only the latest snapshot per
|
||||
name; use `--keep-newer-than` for a rolling window. When `--skip-errors`
|
||||
skipped a file or directory it could not read or list, the prune does not run
|
||||
and every older snapshot is kept, since one of them may be the only one that
|
||||
holds what was skipped; the command says so and exits non-zero. When the
|
||||
backup found nothing in a configured path, as in the mount point of a volume
|
||||
that is not mounted, the prune does not run either: every older snapshot is
|
||||
kept, and the command names the path and exits non-zero, on every run until
|
||||
the path holds something again. Take a path that is empty on purpose out of
|
||||
the config, or remove the older snapshots by hand with `snapshot purge`. When
|
||||
an earlier snapshot of the same name in the local index found a volume mounted
|
||||
at or below a configured path (a directory on a different filesystem from its
|
||||
parent, with an entry in it that the backup does not exclude) and this backup
|
||||
does not, the prune does not run either: every older snapshot is kept, and the
|
||||
command names the directory and exits non-zero, on every run until the volume
|
||||
is mounted again. A backup also does not find the volume when it finds no such
|
||||
entry in its directory, as when an automounted volume cannot be mounted, or
|
||||
when that directory is no longer backed up, because it is excluded or its
|
||||
configured path was removed. If any of this is on purpose, as for a retired
|
||||
volume, remove the older snapshots by hand with `snapshot purge`. A file
|
||||
removed during the backup does not stop the prune.
|
||||
local index, and delete each `.partial` object older than
|
||||
`prune_grace_period`, as `prune` does. By default keeps only the latest
|
||||
snapshot per name; use `--keep-newer-than` for a rolling window. When
|
||||
`--skip-errors` skipped a file or directory it could not read or list, the
|
||||
prune does not run and every older snapshot is kept, since one of them may be
|
||||
the only one that holds what was skipped; the command says so and exits
|
||||
non-zero. When the backup found nothing in a configured path, as in the mount
|
||||
point of a volume that is not mounted, the prune does not run either: every
|
||||
older snapshot is kept, and the command names the path and exits non-zero, on
|
||||
every run until the path holds something again. Take a path that is empty on
|
||||
purpose out of the config, or remove the older snapshots by hand with
|
||||
`snapshot purge`. When an earlier snapshot of the same name in the local index
|
||||
found a volume mounted at or below a configured path (a directory on a
|
||||
different filesystem from its parent, with an entry in it that the backup does
|
||||
not exclude) and this backup does not, the prune does not run either: every
|
||||
older snapshot is kept, and the command names the directory and exits
|
||||
non-zero, on every run until the volume is mounted again. A backup also does
|
||||
not find the volume when it finds no such entry in its directory, as when an
|
||||
automounted volume cannot be mounted, or when that directory is no longer
|
||||
backed up, because it is excluded or its configured path was removed. If any
|
||||
of this is on purpose, as for a retired volume, remove the older snapshots by
|
||||
hand with `snapshot purge`. A file removed during the backup does not stop the
|
||||
prune.
|
||||
- `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than this
|
||||
duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`). Without
|
||||
`--prune`, or with a value that is not a valid duration, the command fails
|
||||
@@ -520,14 +522,16 @@ deleted. Step (3) also deletes each directory under `metadata/` that has no
|
||||
`db.zst.age` but before its manifest, once every object in it is older than
|
||||
`prune_grace_period` and its snapshot is not in the local index. An object the
|
||||
listing gives no time for keeps its directory, and a directory whose name is not
|
||||
a remote key is left alone. Step (3) reads all remote manifests — network cost
|
||||
a remote key is left alone. Step (3) also deletes each `.partial` object under
|
||||
`metadata/` and `blobs/` that an upload cut off part-way left, once it is older
|
||||
than `prune_grace_period`. Step (3) reads all remote manifests — network cost
|
||||
scales with the number of snapshots. `snapshot create --prune` runs the same
|
||||
cleanup automatically; this is the manual entry point for the same work. If the
|
||||
destination store cannot be listed, or the local index holds snapshots and the
|
||||
destination answers that it has no `vaultik-destination.txt` (an unplugged
|
||||
volume's empty mount point), prune fails and deletes nothing. `snapshot purge`
|
||||
does the same. If a blob or such a directory cannot be deleted, the command (and
|
||||
`snapshot create --prune`) exits non-zero.
|
||||
does the same. If a blob, such a directory or such a `.partial` object cannot be
|
||||
deleted, the command (and `snapshot create --prune`) exits non-zero.
|
||||
|
||||
- `--force`: Skip confirmation prompt
|
||||
- `--json`: Output stats as JSON. Also skips the confirmation prompt, as
|
||||
@@ -537,7 +541,10 @@ does the same. If a blob or such a directory cannot be deleted, the command (and
|
||||
gives no time for it, is counted in `blobs_too_new` and not in `blobs_found`.
|
||||
A directory under `metadata/` with no manifest is counted in
|
||||
`metadata_dirs_deleted` once deleted, or in `metadata_dirs_failed` if it
|
||||
cannot be.
|
||||
cannot be. A `.partial` object is counted in `partial_uploads_deleted` once
|
||||
deleted, or in `partial_uploads_failed` if it cannot be. One kept for being
|
||||
newer than `prune_grace_period`, or of unknown age, is counted in
|
||||
`partial_uploads_too_new`.
|
||||
|
||||
**`info`**: Display system configuration, storage settings, encryption
|
||||
recipients, and local database statistics.
|
||||
@@ -552,7 +559,9 @@ in `unreadable_manifests` and counts the manifests under skipped names in
|
||||
`skipped_manifest_count`. An unreadable manifest also leaves its snapshot's blob
|
||||
count and blob size unknown, `null` in `--json`. A file under `blobs/` whose
|
||||
name is not a blob hash is not counted as a blob, since `prune` never deletes
|
||||
it.
|
||||
it. A `.partial` object an upload cut off part-way left under `metadata/` or
|
||||
`blobs/` is not counted as a blob either. Such objects are counted on their own
|
||||
line, and in `partial_upload_count` and `partial_upload_size` in `--json`.
|
||||
|
||||
- `--json`: Output as JSON
|
||||
|
||||
@@ -611,8 +620,9 @@ with a server-side move (local and sftp among them) are written under a
|
||||
temporary name ending in `.partial` and moved into place. Rclone remotes without
|
||||
one are written in place: where such a remote shows a file while it is still
|
||||
being written, a killed upload can leave a truncated object under its name,
|
||||
which a later backup takes for complete. A leftover `.partial` file is ignored
|
||||
and can be deleted.
|
||||
which a later backup takes for complete. A backup ignores a leftover `.partial`
|
||||
file. `prune` deletes it once it is older than `prune_grace_period`, and
|
||||
`remote info` counts it.
|
||||
|
||||
A destination shared between hosts or users, each with its own local index,
|
||||
relies on `prune_grace_period` to keep one's prune from deleting the blobs of
|
||||
@@ -721,6 +731,8 @@ format does and does not protect.
|
||||
5. Delete each directory under `metadata/` with no manifest whose objects are
|
||||
all older than `prune_grace_period`, unless its snapshot is in the local
|
||||
index
|
||||
6. Delete each `.partial` object under `metadata/` and `blobs/` older than
|
||||
`prune_grace_period`
|
||||
|
||||
### chunking and deduplication
|
||||
|
||||
@@ -754,20 +766,20 @@ Run `vaultik config init` to generate a fully commented config file; a complete
|
||||
annotated example also lives in
|
||||
[`configs/config.example.yml`](configs/config.example.yml). Key fields:
|
||||
|
||||
| Field | Default | Description |
|
||||
| -------------------- | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `age_recipients` | (required by `snapshot create`) | Age public keys for encryption. Other commands run without one, so a machine that only restores can leave it empty |
|
||||
| `age_secret_key` | (unset) | Age private key for decryption (`snapshot restore`, `snapshot files`, `snapshot verify --deep`). Setting it in the config file places the private key on the backed-up host, defeating the public-key-only design (see "why" above). Prefer the `VAULTIK_AGE_SECRET_KEY` environment variable, supplied only on the machine you restore from. |
|
||||
| `snapshots` | (required) | Named snapshot definitions with paths and excludes |
|
||||
| `storage_url` | | Storage backend URL (`s3://`, `file://`, `rclone://`) |
|
||||
| `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) |
|
||||
| `exclude` | | Global exclude patterns (applied to all snapshots) |
|
||||
| `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
|
||||
| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` |
|
||||
| `compression_level` | `3` | zstd compression level (1-19) |
|
||||
| `prune_grace_period` | `24h` | How old a blob that no snapshot lists must be before `prune` deletes it, and how old every object in a directory under `metadata/` with no manifest must be before `prune` deletes that directory. A backup uploads its blobs and its `db.zst.age` before its manifest, so on a destination shared between hosts or users this must be longer than the longest backup run (see [locking](#locking)). It does not protect a blob a backup reuses from an earlier run that stopped before its manifest: that blob keeps the time of its first upload, so a prune can delete it. A duration such as `36h` or `90m`; `0s` deletes such blobs and directories at once |
|
||||
| `hostname` | system hostname | Hostname used in snapshot IDs |
|
||||
| `index_path` | platform data dir | Local SQLite index path |
|
||||
| Field | Default | Description |
|
||||
| -------------------- | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `age_recipients` | (required by `snapshot create`) | Age public keys for encryption. Other commands run without one, so a machine that only restores can leave it empty |
|
||||
| `age_secret_key` | (unset) | Age private key for decryption (`snapshot restore`, `snapshot files`, `snapshot verify --deep`). Setting it in the config file places the private key on the backed-up host, defeating the public-key-only design (see "why" above). Prefer the `VAULTIK_AGE_SECRET_KEY` environment variable, supplied only on the machine you restore from. |
|
||||
| `snapshots` | (required) | Named snapshot definitions with paths and excludes |
|
||||
| `storage_url` | | Storage backend URL (`s3://`, `file://`, `rclone://`) |
|
||||
| `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) |
|
||||
| `exclude` | | Global exclude patterns (applied to all snapshots) |
|
||||
| `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
|
||||
| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` |
|
||||
| `compression_level` | `3` | zstd compression level (1-19) |
|
||||
| `prune_grace_period` | `24h` | How old a blob that no snapshot lists must be before `prune` deletes it, and how old every object in a directory under `metadata/` with no manifest must be before `prune` deletes that directory. It also sets how old a `.partial` object a killed upload left must be before `prune` deletes it. A backup uploads its blobs and its `db.zst.age` before its manifest, so on a destination shared between hosts or users this must be longer than the longest backup run (see [locking](#locking)). It does not protect a blob a backup reuses from an earlier run that stopped before its manifest: that blob keeps the time of its first upload, so a prune can delete it. A duration such as `36h` or `90m`; `0s` deletes such blobs and directories at once |
|
||||
| `hostname` | system hostname | Hostname used in snapshot IDs |
|
||||
| `index_path` | platform data dir | Local SQLite index path |
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in new issue
Block a user