diff --git a/TODO.md b/TODO.md index 9160574..ad2ee6f 100644 --- a/TODO.md +++ b/TODO.md @@ -15,13 +15,407 @@ pre-1.0 # Next Step The 1.0 work is complete on `next`: the scope settled on -[issue #125](https://git.eeqj.de/sneak/vaultik/issues/125) was the work -already planned for 1.0, and all of it has landed. The mechanism to cut -the tag exists and is exercised; what is left is merging `next` to -`main` and tagging, both the owner's. +[issue #125](https://git.eeqj.de/sneak/vaultik/issues/125) has landed, +and so has every fix from the defect hunt before the release candidate +([issue #288](https://git.eeqj.de/sneak/vaultik/issues/288)). What is +left is merging `next` to `main` and tagging the release, both the +owner's; a `v*` tag runs the release build in +`.gitea/workflows/release.yml`. # Completed Steps +- 2026-10-14: Stopped `snapshot create --prune` pruning after a backup + that skipped files it could not read + ([issue #382](https://git.eeqj.de/sneak/vaultik/issues/382)). Under + `--skip-errors` the prune still ran, so by default it deleted the + previous snapshot of the name, the only one still holding a file that + had become unreadable, along with blobs only that snapshot used. When + `--skip-errors` skipped anything it could not read or list, the prune + now does not run and older snapshots are kept, a warning says why, and + the run still exits non-zero. + +- 2026-10-14: Made `snapshot create --skip-errors` exit non-zero when + it skips anything it cannot read + ([issue #310](https://git.eeqj.de/sneak/vaultik/issues/310)). The run + printed its skip errors, then `Finished successfully.`, and exited 0, + so a cron wrapper could not tell it from a complete backup. Such a + run now keeps its snapshots and then fails with + `Backup completed, but N file(s) could not be read`, and an entry the + walk could not read, such as a directory it could not list, now + counts as failed too. + +- 2026-10-14: Kept the local index when `remote nuke` cannot list or + find the destination store + ([issue #292](https://git.eeqj.de/sneak/vaultik/issues/292)). Nuke + took a store it could not list, or one without + `vaultik-destination.txt` while the local index held snapshots, as a + warning and cleared the local snapshot records anyway; for the empty + mount point of an unplugged volume it then reported the store empty + and exited 0. It now fails on either before it removes anything, and + deletes each snapshot's metadata on the destination store before its + local record, so a failed delete keeps the snapshot in the local + index. + +- 2026-10-14: Stopped counting directories and unchanged symlinks as + files to back up + ([issue #305](https://git.eeqj.de/sneak/vaultik/issues/305)). Every + backup queued every directory and symlink to be backed up, so the + line ending the walk counted directories and their sizes as files and + data to back up while its enumerated count left them out, and every + symlink was reported as backed up on every run. Directories are no + longer counted as files or data to back up, and a symlink is compared + with its record like a regular file, so an unchanged one counts as + unchanged. + +- 2026-10-14: Refused a backup destination that an unplugged volume + left empty, as the owner ruled + ([issue #239](https://git.eeqj.de/sneak/vaultik/issues/239)). A + `file://` destination at the mount point of an unplugged volume listed + as an empty store, so `prune` dropped every local snapshot record, and + `snapshot create` wrote a whole backup to the internal disk under the + empty mount point. Every backup now writes `vaultik-destination.txt` + at the destination root, and while the local index holds snapshots + stored there, a destination without it is treated as not present: + `snapshot create` refuses before writing, and `prune` deletes + nothing. The `file://` backend also never creates its destination + directory. + +- 2026-10-14: Warned about restored ownership only when an entry's + owner or group differs + ([issue #316](https://git.eeqj.de/sneak/vaultik/issues/316)). A + restore by a normal user warned that file ownership was not preserved + even when every entry came out with its recorded owner and group. The + owner and group the restored entries have on disk are now compared + with the recorded ones, and the warning is printed once if any differ. + +- 2026-10-14: Made `snapshot list` report a changed destination instead + of advising `prune` + ([issue #317](https://git.eeqj.de/sneak/vaultik/issues/317)). With the + local index bound to another destination than the configured one, + `snapshot list` reported its snapshots as missing from the destination + store and advised `prune`, which then refused. It now gives the + warning `prune`, `snapshot purge` and `snapshot remove` refuse with: + both destinations, and the choice of reverting the configuration or + running `vaultik database delete`. + +- 2026-10-14: Counted a file as deleted only if the previous snapshot + of its name had it + ([issue #304](https://git.eeqj.de/sneak/vaultik/issues/304)). A backup + counted as deleted every file in the local index under the source + path that its walk did not find: files the snapshot excludes but + another snapshot backs up, and files removed long ago, which every + later backup reported again. It now counts only files that the latest + completed snapshot of the same name lists and that are gone from disk. + +- 2026-10-14: Named each prefix once in restore, verify and purge errors + ([issue #327](https://git.eeqj.de/sneak/vaultik/issues/327)). A + missing blob during restore, a failed shallow verify and an invalid + purge duration each printed their prefix twice, and a deep verify log + line named S3 whatever the destination. Each prefix now appears once, + and that log line names storage instead. + +- 2026-10-14: Refused credentials in a storage URL's endpoint or rclone + remote name + ([issue #322](https://git.eeqj.de/sneak/vaultik/issues/322)). Userinfo + inside the `endpoint=` value of an `s3://` URL, and rclone + `remote,param=value` parameters, were accepted and then printed by + `info`, `remote info` and `snapshot create`. Both are now refused + without echoing the value, as is an `s3://` or `rclone://` query that + cannot be decoded. + +- 2026-10-14: Ran `$EDITOR` through the shell in `config edit` + ([issue #315](https://git.eeqj.de/sneak/vaultik/issues/315)). The + whole of `$EDITOR` was taken as the program name, so a value with + arguments such as `code --wait` or `emacsclient -t` failed with + `executable file not found`. It now runs as git runs it, through + `sh -c`, with the config path passed as a separate argument. + +- 2026-10-14: Made `snapshot remove` fail when it cannot delete the + metadata of a snapshot that is only on the destination store + ([issue #340](https://git.eeqj.de/sneak/vaultik/issues/340)). It + warned and exited 0 having removed nothing. That failure is now an + error; for a snapshot in the local index, which is still removed + there, it stays a warning. The README and the help now say the + `vaultik prune` follow-up is printed only when metadata was removed + from the destination store, and never under `--json`. + +- 2026-10-14: Labelled the totals in the local index cleanup before + each backup as totals + ([issue #306](https://git.eeqj.de/sneak/vaultik/issues/306)). Each + removed count was followed by `(N remain)`, which read as incomplete + or orphaned rows left over but counted every snapshot, file, chunk or + blob left in the local index, so removing the only incomplete snapshot + printed `1 removed (1 remain)`. Each line now names that figure as the + total left in the local index. + +- 2026-10-13: Freed a cached blob once no file still to restore needs + it ([issue #324](https://git.eeqj.de/sneak/vaultik/issues/324)). A + blob was freed only once every file in the snapshot that uses it had + been restored, which a file outside the selected paths, or one that + failed under `--skip-errors`, never is, so such a restore kept those + blobs in the temporary cache until it ended. A cached blob is now + freed once no file still pending in this restore uses it. + +- 2026-10-13: Kept the whole startup error when it contains `): ` + ([issue #320](https://git.eeqj.de/sneak/vaultik/issues/320)). A + startup error was cut at its last `): ` to drop the annotations `fx` + adds, so the `blob_size_limit` message, which itself contains `): `, + lost everything but `need at least N bytes`, the setting's name + included. Only those annotations are now removed, and the error + underneath is shown whole. + +- 2026-10-13: Silenced the status lines of `prune` and snapshot removal + under `--cron` and `--quiet` + ([issue #294](https://git.eeqj.de/sneak/vaultik/issues/294)). `prune`, + `snapshot purge`, `snapshot remove`, `remote nuke` and the prune step + of `snapshot create --prune` wrote their status lines straight to + stdout, so the README's cron line printed on every successful run and + `--quiet` did not silence them. Those lines are now dropped under both + flags; warnings, errors, `--json` documents and confirmation prompts + still print. + +- 2026-10-13: Read the rclone config file that `RCLONE_CONFIG` names + ([issue #321](https://git.eeqj.de/sneak/vaultik/issues/321)). rclone + reads that variable only through its `--config` flag, which vaultik + does not have, so a remote defined only in that file was reported as + not found. vaultik now sets rclone's config path from `RCLONE_CONFIG` + when it is set. + +- 2026-10-13: Made `/**/` in an exclude pattern match zero directories + ([issue #338](https://git.eeqj.de/sneak/vaultik/issues/338)). The + anchored `/**/x` missed an `x` directly under the snapshot path, and + `a/**/*.log` missed `a/x.log`. As in gitignore, a leading `**/` now + makes an anchored pattern match at every depth too, and each `/**/` + inside a pattern also matches a single `/`. + +- 2026-10-13: Made the `chunk_size` and `blob_size_limit` defaults and + the `chunk_size` minimum 1000-based + ([issue #313](https://git.eeqj.de/sneak/vaultik/issues/313)). They + were 1024-based while the size parser, the docs and `info` use + 1000-based units, so `chunk_size: 1MB` was rejected, writing out a + documented default changed it, and `info` showed an 11 GB blob size + limit. They are now 1000-based like the parser, and + `config.example.yml` and the `config init` template state the + minimum. + +- 2026-10-13: Kept a snapshot's remote metadata when `snapshot purge` + cannot remove it from the local index + ([issue #337](https://git.eeqj.de/sneak/vaultik/issues/337)). + `snapshot purge`, and the purge step of `snapshot create --prune`, + logged that failure, deleted the snapshot's metadata from the + destination store anyway, and exited 0. The purge now stops at that + snapshot with an error before touching its remote metadata. + +- 2026-10-13: Made `snapshot restore` fail for a requested path that + matches nothing + ([issue #309](https://git.eeqj.de/sneak/vaultik/issues/309)). A path + that matched no entry in the snapshot, such as a typo or a relative + path, was ignored and the command exited 0, even when nothing matched. + Each such path is now named in a warning, the paths that do match are + still restored, and the command exits non-zero. + +- 2026-10-13: Made `remote nuke` delete every object under `metadata/` + and `blobs/` + ([issue #291](https://git.eeqj.de/sneak/vaultik/issues/291)). It + deleted only the snapshot metadata and blobs it recognised, so a + snapshot database whose manifest was never uploaded, or a stray file + such as `.DS_Store` under `blobs/`, survived while nuke reported the + store empty. It now deletes everything listed under both, reports the + store empty only when a listing afterwards finds nothing, and + otherwise names each object left and fails. + +- 2026-10-13: Made `config set` create a list for an index under a + missing key + ([issue #308](https://git.eeqj.de/sneak/vaultik/issues/308)). + `config set age_recipients.0 age1...` on a config without + `age_recipients` wrote a map keyed `0`, after which no command could + load the config. A missing key is now created as a list when the next + path element is a number, and `config set` refuses a change that + would stop a config that loaded from loading. + +- 2026-10-13: Listed a `file://` destination through a symlinked + directory + ([issue #296](https://git.eeqj.de/sneak/vaultik/issues/296)). The + listing did not follow a symlinked `metadata/` or `blobs/`, while + reads and writes did, so what was behind the link listed as empty; + with `metadata/` linked, `prune` then dropped the local snapshot + records and deleted every blob. The listings now read through such a + link, and fail when its target is missing + ([issue #332](https://git.eeqj.de/sneak/vaultik/issues/332)). + +- 2026-10-13: Made `snapshot verify --deep --json` report its blob + counts ([issue #314](https://git.eeqj.de/sneak/vaultik/issues/314)). + Deep verify set `verified` only on success and never set `missing`, + so a failed run reported 0 for both. It now counts each blob whose + contents pass, and before reading any, counts missing and wrong-size + blobs as shallow verify does. + +- 2026-10-13: Limited and retried the module download in the Docker + build ([issue #351](https://git.eeqj.de/sneak/vaultik/issues/351)). + `go mod download` in the `Dockerfile` had no time limit, so a stalled + connection to the module proxy held the step, and every build on the + host sharing it, until the kernel gave up on the connection, once + after about 55 minutes. Each such step now runs under a five-minute + limit, is tried up to three times, and then fails the build with + `module download failed`. + +- 2026-10-13: Recorded each snapshot's uncompressed size when its + backup runs + ([issue #360](https://git.eeqj.de/sneak/vaultik/issues/360)). + `snapshot list` summed the chunks a snapshot reaches through file + rows shared by every snapshot that lists the file and that a changed + file rewrites, so backing up a changed file changed the UNCOMPRESSED + SIZE of every older snapshot that lists that file. The backup now + stores that total in the `snapshots` table's + `uncompressed_chunk_size`, which `snapshot list` reads. + +- 2026-10-13: Stopped `snapshot restore --verify` reporting a file that + failed to restore again as a checksum failure + ([issue #323](https://git.eeqj.de/sneak/vaultik/issues/323)). Under + `--skip-errors`, a file that failed to restore was verified anyway, + failed again because it was never written, and was listed twice as a + checksum mismatch. Verification now skips the files that failed to + restore and reports its own failures apart, each path once. + +- 2026-10-13: Recorded the stored size of a blob a backup finds already + stored ([issue #300](https://git.eeqj.de/sneak/vaultik/issues/300)). + The upload was skipped, but the local index and the manifest kept the + size of the local copy, which can differ from the stored object when + `compression_level` or the recipients changed, so `snapshot verify` + failed on a good snapshot. The size the destination store reports for + the existing object is now recorded. + +- 2026-10-13: Stopped the local index and the blob upload step ending + the process on a rollback after Ctrl-C, a failed close of query + results, or a failed temp file cleanup + ([issue #345](https://git.eeqj.de/sneak/vaultik/issues/345)). Each + ended the process at once and skipped the run's cleanup. + Read-only transactions now handle the rollback as other transactions + do, the query functions report a close failure through the error they + already return, and a failed close or removal of a blob temp file + fails the backup. + +- 2026-10-13: Stopped a stalled snapshot database read when restore or + deep verify is interrupted + ([issue #356](https://git.eeqj.de/sneak/vaultik/issues/356)). Both + read the snapshot database without the fix for stalled blob reads, so + after Ctrl-C a database read stalled on a slow device still left the + decrypted database in the temporary directory. That read is now + closed on cancel too. + +- 2026-10-13: Made `prune` and `remote nuke` fail when a delete fails + ([issue #290](https://git.eeqj.de/sneak/vaultik/issues/290)). `prune`, + and the prune step of `snapshot create --prune`, counted a blob it + could not delete and still exited 0, and `remote nuke` only warned + when it could not delete a snapshot's metadata, advised + `vaultik prune`, which never removes metadata, and still reported the + store empty. Both now exit non-zero, and nuke says to run + `vaultik remote nuke --force` again. + +- 2026-10-13: Made an S3 listing that the server cuts short an error + ([issue #297](https://git.eeqj.de/sneak/vaultik/issues/297)). A + listing page marked truncated but carrying no continuation token + ended the listing without an error, so `prune` saw only the snapshots + on the first page and treated the blobs of the rest as unreferenced. + Such a page now fails the listing, so `prune` stops before deleting + any blob. + +- 2026-10-13: Made shallow `snapshot verify` report a blob it could not + check apart from a missing one + ([issue #318](https://git.eeqj.de/sneak/vaultik/issues/318)). Every + error from checking a blob or the encrypted database counted as + missing and was printed nowhere, so a permission error or an S3 403 + read as data loss. Only a not-found answer now counts as missing; any + other error is reported with its cause as could not be checked, in + the text report and under `--json`, and the verify fails. + +- 2026-10-13: Restored the other files when a file's chunk is in no + blob under `--skip-errors` + ([issue #326](https://git.eeqj.de/sneak/vaultik/issues/326)). Restore + planning stopped at the first such file, before the flag could skip + it, so nothing was restored. That file now fails like any other file + that cannot be restored: skipped under the flag, and fatal without it. + +- 2026-10-13: Recorded each snapshot's new chunk size when its backup + runs ([issue #311](https://git.eeqj.de/sneak/vaultik/issues/311)). + `snapshot list` worked out NEW CHUNK SIZE from file rows shared by + every snapshot that lists the file and that a changed file rewrites, + so a snapshot whose change was a modified file showed 0 B and older + snapshots' figures moved after later backups. The backup now stores + the size of the chunks it newly stored in the `snapshots` table's + `new_chunk_size`, which `snapshot list` reads. + +- 2026-10-13: Made a backup interrupted during a local index write exit + as interrupted + ([issue #301](https://git.eeqj.de/sneak/vaultik/issues/301)). After + Ctrl-C the transaction was already rolled back, so the explicit + rollback failed and was treated as fatal: the run printed + `FATAL: failed to rollback transaction`, exited 1 instead of 130, and + skipped its cleanup. The rollback's report that the transaction was + already over is now ignored, and a rollback that really fails is + returned with the original error instead of ending the process. + +- 2026-10-13: Printed `snapshot list` and `snapshot purge` sizes like + every other command + ([issue #312](https://git.eeqj.de/sneak/vaultik/issues/312)). They, + and the `bytes` value in terminal log lines, divided by 1024 but + labelled the result KB or MB, so 6,329,198 bytes read 6.0 MB there and + 6.3 MB in other commands. All three now use 1000-based units, as other + commands do. + +- 2026-10-13: Stopped a stalled blob read when restore or deep verify + is interrupted + ([issue #325](https://git.eeqj.de/sneak/vaultik/issues/325)). Both + checked for a cancel only between blobs, and the `file://` backend's + reader ignores it, so after Ctrl-C a blob read stalled on a slow + device outlasted the 30-second shutdown timeout and the decrypted + snapshot database and blob cache were left in the temporary + directory. Each blob's storage reader is now closed as soon as the + command is cancelled, which ends a stalled read unless it is stuck in + the kernel. + +- 2026-10-13: Made a mistyped subcommand under a group command an error + ([issue #319](https://git.eeqj.de/sneak/vaultik/issues/319)). + `snapshot`, `config`, `remote`, `database` and `completion` printed + their help and exited 0 for a word that is not one of their + subcommands, so a mistyped cron line made no backup and reported + nothing. Each now prints its help only when given no argument, and + otherwise fails as an unknown command, as the top level does. + +- 2026-10-13: Made an exclude pattern starting with `**/` match + directly under the snapshot path + ([issue #303](https://git.eeqj.de/sneak/vaultik/issues/303)). + `**/node_modules` and `**/*.pyc` excluded `sub/node_modules` and + `sub/y.pyc` but backed up `node_modules/` and `x.pyc` at the top of + the path. The leading `**/` is now dropped when the pattern is + compiled, so the pattern matches at every depth, the top level + included, as in gitignore. + +- 2026-10-13: Refused a relative `file://` storage URL and decoded its + percent-escapes + ([issue #299](https://git.eeqj.de/sneak/vaultik/issues/299)). The path + after `file://` was used as written, so `file://dest` followed the + working directory and a backup run from elsewhere exited 0 and could + not be restored, and `my%20stick` made a directory of that literal + name. Config load and the storage URL parser now refuse a path that + is not absolute, and the parser decodes escapes. + +- 2026-10-13: Removed the blob temp file when a backup stops part-way + ([issue #302](https://git.eeqj.de/sneak/vaultik/issues/302)). When + `snapshot create` stopped on a file error, a full temporary directory + or Ctrl-C, the half-written `vaultik-blob-*.tmp` of the blob being + packed, up to `blob_size_limit` of encrypted data, stayed in + `$TMPDIR`. The backup now closes and removes it on the way out. + +- 2026-10-13: Checked `--keep-newer-than` before a backup starts + ([issue #307](https://git.eeqj.de/sneak/vaultik/issues/307)). An + invalid value was read only by the prune after the backup, so the + snapshot was uploaded, the run reported it as failed and retention + never ran; without `--prune` the flag was ignored. `snapshot create` + now refuses an invalid duration, or the flag without `--prune`, + before it starts, and a failed post-backup prune is reported as + `Backup completed, but post-backup prune failed`. + - 2026-10-13: Stopped `remote info` counting a file under `blobs/` that is not a blob as an orphaned blob ([issue #295](https://git.eeqj.de/sneak/vaultik/issues/295)). A stray