Let prune delete old .partial uploads and remote info count them (closes #466)
check / check (push) Waiting to run

A killed upload leaves a `.partial` object that only remote nuke
removed, and that remote info never showed.

The storage interface's DeletePartialUploads becomes ListPartialUploads,
which returns each such object's key, size and time; Delete removes it.
prune deletes those under metadata/ and blobs/ once older than
prune_grace_period, and reports kept, deleted and failed counts. remote
info counts them on their own line and in --json. remote nuke lists and
deletes them like any other object, and now names any it could not
delete.

Model: opus-5-5
This commit was merged in pull request #474.
This commit is contained in:
clawbot committed 2026-11-19 14:01:04 +01:00
1 parent aaa2219d6b
commit be362080d3
24 files changed
+625 -230

No files matched your search

+56 -48
View File
@@ -341,31 +341,29 @@ In such a config, an unknown key or a value of the wrong type is refused.
error that explains how to fix it
- `--cron`: Silent on total success; warnings and errors are still printed (for
crontab)
- `--prune`: After backup, drop older snapshots of each backed-up name, remove
orphaned blobs older than `prune_grace_period` from remote storage, and delete
each directory under `metadata/` that has no `manifest.json.zst` once every
object in it is older than `prune_grace_period` and its snapshot is not in the
local index, as `prune` does. By default keeps only the latest snapshot per
name; use `--keep-newer-than` for a rolling window. When `--skip-errors`
skipped a file or directory it could not read or list, the prune does not run
and every older snapshot is kept, since one of them may be the only one that
holds what was skipped; the command says so and exits non-zero. When the
backup found nothing in a configured path, as in the mount point of a volume
that is not mounted, the prune does not run either: every older snapshot is
kept, and the command names the path and exits non-zero, on every run until
the path holds something again. Take a path that is empty on purpose out of
the config, or remove the older snapshots by hand with `snapshot purge`. When
an earlier snapshot of the same name in the local index found a volume mounted
at or below a configured path (a directory on a different filesystem from its
parent, with an entry in it that the backup does not exclude) and this backup
does not, the prune does not run either: every older snapshot is kept, and the
command names the directory and exits non-zero, on every run until the volume
is mounted again. A backup also does not find the volume when it finds no such
entry in its directory, as when an automounted volume cannot be mounted, or
when that directory is no longer backed up, because it is excluded or its
configured path was removed. If any of this is on purpose, as for a retired
volume, remove the older snapshots by hand with `snapshot purge`. A file
removed during the backup does not stop the prune.
- `--prune`: After backup, drop older snapshots of each backed-up name, then
delete from the destination store what `prune` deletes there. By default keeps
only the latest snapshot per name; use `--keep-newer-than` for a rolling
window. When `--skip-errors` skipped a file or directory it could not read or
list, the prune does not run and every older snapshot is kept, since one of
them may be the only one that holds what was skipped; the command says so and
exits non-zero. When the backup found nothing in a configured path, as in the
mount point of a volume that is not mounted, the prune does not run either:
every older snapshot is kept, and the command names the path and exits
non-zero, on every run until the path holds something again. Take a path that
is empty on purpose out of the config, or remove the older snapshots by hand
with `snapshot purge`. When an earlier snapshot of the same name in the local
index found a volume mounted at or below a configured path (a directory on a
different filesystem from its parent, with an entry in it that the backup does
not exclude) and this backup does not, the prune does not run either: every
older snapshot is kept, and the command names the directory and exits
non-zero, on every run until the volume is mounted again. A backup also does
not find the volume when it finds no such entry in its directory, as when an
automounted volume cannot be mounted, or when that directory is no longer
backed up, because it is excluded or its configured path was removed. If any
of this is on purpose, as for a retired volume, remove the older snapshots by
hand with `snapshot purge`. A file removed during the backup does not stop the
prune.
- `--keep-newer-than <duration>`: With `--prune`, keep snapshots newer than this
duration instead of only the latest (e.g. `4w`, `30d`, `6mo`, `1y`). Without
`--prune`, or with a value that is not a valid duration, the command fails
@@ -521,14 +519,16 @@ deleted. Step (3) also deletes each directory under `metadata/` that has no
`db.zst.age` but before its manifest, once every object in it is older than
`prune_grace_period` and its snapshot is not in the local index. An object the
listing gives no time for keeps its directory, and a directory whose name is not
a remote key is left alone. Step (3) reads all remote manifests — network cost
scales with the number of snapshots. `snapshot create --prune` runs the same
cleanup automatically; this is the manual entry point for the same work. If the
a remote key is left alone. Step (3) also deletes each `.partial` object under
`metadata/` and `blobs/` left by a killed upload, once it is older than
`prune_grace_period`. Step (3) reads all remote manifests — network cost scales
with the number of snapshots. `snapshot create --prune` runs the same cleanup
automatically; this is the manual entry point for the same work. If the
destination store cannot be listed, or the local index holds snapshots and the
destination answers that it has no `vaultik-destination.txt` (an unplugged
volume's empty mount point), prune fails and deletes nothing. `snapshot purge`
does the same. If a blob or such a directory cannot be deleted, the command (and
`snapshot create --prune`) exits non-zero.
does the same. If a blob, such a directory or such a `.partial` object cannot be
deleted, the command (and `snapshot create --prune`) exits non-zero.
- `--force`: Skip confirmation prompt
- `--json`: Output stats as JSON. Also skips the confirmation prompt, as
@@ -538,7 +538,10 @@ does the same. If a blob or such a directory cannot be deleted, the command (and
gives no time for it, is counted in `blobs_too_new` and not in `blobs_found`.
A directory under `metadata/` with no manifest is counted in
`metadata_dirs_deleted` once deleted, or in `metadata_dirs_failed` if it
cannot be.
cannot be. A `.partial` object is counted in `partial_uploads_deleted` once
deleted, or in `partial_uploads_failed` if it cannot be. One kept for being
newer than `prune_grace_period`, or of unknown age, is counted in
`partial_uploads_too_new`.
**`info`**: Display system configuration, storage settings, encryption
recipients, and local database statistics.
@@ -553,7 +556,9 @@ in `unreadable_manifests` and counts the manifests under skipped names in
`skipped_manifest_count`. An unreadable manifest also leaves its snapshot's blob
count and blob size unknown, `null` in `--json`. A file under `blobs/` whose
name is not a blob hash is not counted as a blob, since `prune` never deletes
it.
it. A leftover `.partial` object under `metadata/` or `blobs/` is not counted as
a blob either. Such objects are counted on their own line, and in
`partial_upload_count` and `partial_upload_size` in `--json`.
- `--json`: Output as JSON
@@ -616,8 +621,9 @@ with a server-side move (local and sftp among them) are written under a
temporary name ending in `.partial` and moved into place. Rclone remotes without
one are written in place: where such a remote shows a file while it is still
being written, a killed upload can leave a truncated object under its name,
which a later backup takes for complete. A leftover `.partial` file is ignored
and can be deleted.
which a later backup takes for complete. A backup ignores a leftover `.partial`
file. `prune` deletes it once it is older than `prune_grace_period`, and
`remote info` counts it.
A destination shared between hosts or users, each with its own local index,
relies on `prune_grace_period` to keep one's prune from deleting the blobs of
@@ -726,6 +732,8 @@ format does and does not protect.
5. Delete each directory under `metadata/` with no manifest whose objects are
all older than `prune_grace_period`, unless its snapshot is in the local
index
6. Delete each `.partial` object under `metadata/` and `blobs/` older than
`prune_grace_period`
### chunking and deduplication
@@ -759,20 +767,20 @@ Run `vaultik config init` to generate a fully commented config file; a complete
annotated example also lives in
[`configs/config.example.yml`](configs/config.example.yml). Key fields:
| Field | Default | Description |
| -------------------- | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `age_recipients` | (required by `snapshot create`) | Age public keys for encryption. Other commands run without one, so a machine that only restores can leave it empty |
| `age_secret_key` | (unset) | Age private key for decryption (`snapshot restore`, `snapshot files`, `snapshot verify --deep`). Setting it in the config file places the private key on the backed-up host, defeating the public-key-only design (see "why" above). Prefer the `VAULTIK_AGE_SECRET_KEY` environment variable, supplied only on the machine you restore from. |
| `snapshots` | (required) | Named snapshot definitions with paths and excludes |
| `storage_url` | | Storage backend URL (`s3://`, `file://`, `rclone://`) |
| `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) |
| `exclude` | | Global exclude patterns (applied to all snapshots) |
| `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` |
| `compression_level` | `3` | zstd compression level (1-19) |
| `prune_grace_period` | `24h` | How old a blob that no snapshot lists must be before `prune` deletes it, and how old every object in a directory under `metadata/` with no manifest must be before `prune` deletes that directory. A backup uploads its blobs and its `db.zst.age` before its manifest, so on a destination shared between hosts or users this must be longer than the longest backup run (see [locking](#locking)). It does not protect a blob a backup reuses from an earlier run that stopped before its manifest: that blob keeps the time of its first upload, so a prune can delete it. A duration such as `36h` or `90m`; `0s` deletes such blobs and directories at once |
| `hostname` | system hostname | Hostname used in snapshot IDs |
| `index_path` | platform data dir | Local SQLite index path |
| Field | Default | Description |
| -------------------- | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `age_recipients` | (required by `snapshot create`) | Age public keys for encryption. Other commands run without one, so a machine that only restores can leave it empty |
| `age_secret_key` | (unset) | Age private key for decryption (`snapshot restore`, `snapshot files`, `snapshot verify --deep`). Setting it in the config file places the private key on the backed-up host, defeating the public-key-only design (see "why" above). Prefer the `VAULTIK_AGE_SECRET_KEY` environment variable, supplied only on the machine you restore from. |
| `snapshots` | (required) | Named snapshot definitions with paths and excludes |
| `storage_url` | | Storage backend URL (`s3://`, `file://`, `rclone://`) |
| `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) |
| `exclude` | | Global exclude patterns (applied to all snapshots) |
| `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` |
| `compression_level` | `3` | zstd compression level (1-19) |
| `prune_grace_period` | `24h` | How old a blob that no snapshot lists must be before `prune` deletes it, and how old every object in a directory under `metadata/` with no manifest must be before `prune` deletes that directory. `prune` also deletes a `.partial` object left by a killed upload only once it is this old. A backup uploads its blobs and its `db.zst.age` before its manifest, so on a destination shared between hosts or users this must be longer than the longest backup run (see [locking](#locking)). It does not protect a blob a backup reuses from an earlier run that stopped before its manifest: that blob keeps the time of its first upload, so a prune can delete it. A duration such as `36h` or `90m`; `0s` deletes such blobs and directories at once |
| `hostname` | system hostname | Hostname used in snapshot IDs |
| `index_path` | platform data dir | Local SQLite index path |
---