diff --git a/docs/TODO.md b/docs/TODO.md index fe2b554..3805561 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -14,1210 +14,7 @@ pre-1.0 # Next Step -The 1.0 work is complete on `next`: the scope settled on -[issue #125](https://git.eeqj.de/sneak/vaultik/issues/125) has landed, and so -has every fix from the defect hunt before the release candidate -([issue #288](https://git.eeqj.de/sneak/vaultik/issues/288)). What is left is -merging `next` to `main` and tagging the release, both the owner's; a `v*` tag -runs the release build in `.gitea/workflows/release.yml`. - -# Completed Steps - -- 2026-11-10: `script/fmt` and `script/fmt-check` now format and check all - Markdown with `prettier` - ([issue #411](https://git.eeqj.de/sneak/vaultik/issues/411)). The architecture - document and this file moved to `docs/`, and the example config to `configs/`. - -- 2026-11-10: Printed the status and progress lines of `remote info` and - `snapshot verify` through `internal/ui`, with its marker and color, as the - README describes ([issue #410](https://git.eeqj.de/sneak/vaultik/issues/410)). - Their reports stay plain. - -- 2026-11-10: Made `config init` write the `apps` snapshot, which backs up - `/Applications`, only on macOS - ([issue #408](https://git.eeqj.de/sneak/vaultik/issues/408)). On Linux the - first `snapshot create` stopped on the missing directory. - -- 2026-10-15: Stopped `snapshot create --prune` pruning after a backup in which - a volume mounted below a configured path in an earlier snapshot of the same - name was not mounted - ([issue #395](https://git.eeqj.de/sneak/vaultik/issues/395)). Such a prune - could delete every snapshot still holding the volume's files. Each snapshot - now records in the local index the directories it found on a different - filesystem from their parent and found an entry in that the backup does not - exclude. - -- 2026-10-14: Stopped `snapshot create --prune` pruning after a backup that - found nothing in a configured path - ([issue #388](https://git.eeqj.de/sneak/vaultik/issues/388)), as in the mount - point of a volume that is not mounted. Such a prune could delete every - snapshot still holding that path's files. A configured path that is a symlink - whose target is gone, as one into such a volume, now fails the backup, with or - without `--prune`. - -- 2026-10-14: Stopped `snapshot create --prune` pruning after a backup that - skipped files it could not read - ([issue #382](https://git.eeqj.de/sneak/vaultik/issues/382)). Such a prune - could delete the only snapshot still holding those files. - -- 2026-10-14: Made `snapshot create --skip-errors` exit non-zero, instead of 0, - when it skips anything it cannot read - ([issue #310](https://git.eeqj.de/sneak/vaultik/issues/310)). - -- 2026-10-14: Kept the local index when `remote nuke` cannot list or find the - destination store - ([issue #292](https://git.eeqj.de/sneak/vaultik/issues/292)). Nuke had cleared - the local snapshot records anyway; it now fails before removing anything. - -- 2026-10-14: Stopped counting directories and unchanged symlinks as files to - back up ([issue #305](https://git.eeqj.de/sneak/vaultik/issues/305)). - -- 2026-10-14: Refused a backup destination that an unplugged volume left empty, - as the owner ruled - ([issue #239](https://git.eeqj.de/sneak/vaultik/issues/239)). Every backup now - writes `vaultik-destination.txt` at the destination root, and while the local - index holds snapshots stored there, a destination without it is treated as not - present. - -- 2026-10-14: Warned about restored ownership only when an entry's owner or - group differs from the recorded one - ([issue #316](https://git.eeqj.de/sneak/vaultik/issues/316)); a restore by a - non-root user had warned even when none did. - -- 2026-10-14: Made `snapshot list` report a changed destination instead of - advising `prune`, which refuses in that case - ([issue #317](https://git.eeqj.de/sneak/vaultik/issues/317)). - -- 2026-10-14: Counted a file as deleted by a backup only if the previous - snapshot of the same name listed it - ([issue #304](https://git.eeqj.de/sneak/vaultik/issues/304)); files that - snapshot did not list had been counted too. - -- 2026-10-14: Named each prefix once in restore, verify and purge errors, some - of which repeated it - ([issue #327](https://git.eeqj.de/sneak/vaultik/issues/327)). A deep verify - log line that named S3 whatever the destination now names storage. - -- 2026-10-14: Refused credentials in a storage URL's endpoint or rclone remote - name, which were accepted and could then be printed - ([issue #322](https://git.eeqj.de/sneak/vaultik/issues/322)). - -- 2026-10-14: Ran `$EDITOR` through the shell in `config edit`, so a value with - arguments, such as `code --wait`, no longer fails - ([issue #315](https://git.eeqj.de/sneak/vaultik/issues/315)). - -- 2026-10-14: Made `snapshot remove` fail, instead of exiting 0, when it cannot - delete the metadata of a snapshot that is only on the destination store - ([issue #340](https://git.eeqj.de/sneak/vaultik/issues/340)). The README and - the help now say when it prints the `vaultik prune` follow-up. - -- 2026-10-14: Labelled the totals in the local index cleanup before each backup - as totals ([issue #306](https://git.eeqj.de/sneak/vaultik/issues/306)). Each - was shown as `(N remain)`, which read as rows left to clean up. - -- 2026-10-13: Freed a cached blob once no file still to restore needs it - ([issue #324](https://git.eeqj.de/sneak/vaultik/issues/324)); one also used by - a file outside the selected paths had stayed in the temporary cache until the - restore ended. - -- 2026-10-13: Kept the whole startup error when it contains `): ` - ([issue #320](https://git.eeqj.de/sneak/vaultik/issues/320)). Such an error - was cut at its last `): `, so the `blob_size_limit` one lost the setting's - name. - -- 2026-10-13: Silenced the status lines of `prune` and snapshot removal under - `--cron` and `--quiet` - ([issue #294](https://git.eeqj.de/sneak/vaultik/issues/294)). - -- 2026-10-13: Read the rclone config file that `RCLONE_CONFIG` names - ([issue #321](https://git.eeqj.de/sneak/vaultik/issues/321)); a remote defined - only there had been reported as not found. - -- 2026-10-13: Made `/**/` in an exclude pattern match zero directories, as in - gitignore ([issue #338](https://git.eeqj.de/sneak/vaultik/issues/338)); - `a/**/*.log` had missed `a/x.log`. - -- 2026-10-13: Made the `chunk_size` and `blob_size_limit` defaults and the - `chunk_size` minimum 1000-based, like the size parser and the docs - ([issue #313](https://git.eeqj.de/sneak/vaultik/issues/313)). - -- 2026-10-13: Kept a snapshot's remote metadata when `snapshot purge` cannot - remove it from the local index - ([issue #337](https://git.eeqj.de/sneak/vaultik/issues/337)); the purge had - deleted the metadata anyway and exited 0. - -- 2026-10-13: Made `snapshot restore` fail for a requested path that matches - nothing, which it had ignored - ([issue #309](https://git.eeqj.de/sneak/vaultik/issues/309)). - -- 2026-10-13: Made `remote nuke` delete every object under `metadata/` and - `blobs/` ([issue #291](https://git.eeqj.de/sneak/vaultik/issues/291)); it had - left some, such as a stray `.DS_Store`, while reporting the store empty. - -- 2026-10-13: Made `config set` create a list for an index under a missing key - ([issue #308](https://git.eeqj.de/sneak/vaultik/issues/308)); it had written a - map, after which no command could load the config. - -- 2026-10-13: Listed a `file://` destination through a symlinked `metadata/` or - `blobs/`, which had listed as empty - ([issue #296](https://git.eeqj.de/sneak/vaultik/issues/296)). A link to a - missing target now fails the listing - ([issue #332](https://git.eeqj.de/sneak/vaultik/issues/332)). - -- 2026-10-13: Made `snapshot verify --deep --json` report its blob counts - ([issue #314](https://git.eeqj.de/sneak/vaultik/issues/314)); a failed run had - reported 0 missing and 0 verified. - -- 2026-10-13: Limited and retried the module download in the Docker build - ([issue #351](https://git.eeqj.de/sneak/vaultik/issues/351)), which had hung - for as long as a stalled connection to the module proxy lasted. - -- 2026-10-13: Recorded each snapshot's uncompressed size when its backup runs - ([issue #360](https://git.eeqj.de/sneak/vaultik/issues/360)). A later backup - could change an older snapshot's UNCOMPRESSED SIZE in `snapshot list`. - -- 2026-10-13: Stopped `snapshot restore --skip-errors --verify` reporting a file - that failed to restore again as a checksum failure - ([issue #323](https://git.eeqj.de/sneak/vaultik/issues/323)). - -- 2026-10-13: Recorded the stored size of a blob a backup finds already stored, - not the size of its local copy, which can differ, so `snapshot verify` could - fail on a good snapshot - ([issue #300](https://git.eeqj.de/sneak/vaultik/issues/300)). - -- 2026-10-13: Stopped the local index and the blob upload step ending the - process on a failed close of query results or a failed temp file cleanup - ([issue #345](https://git.eeqj.de/sneak/vaultik/issues/345)); both are now - returned as errors. - -- 2026-10-13: Closed the snapshot database's storage reader too when restore or - deep verify is interrupted - ([issue #356](https://git.eeqj.de/sneak/vaultik/issues/356)). - -- 2026-10-13: Made `prune` and `remote nuke` fail when a delete on the backup - destination store fails - ([issue #290](https://git.eeqj.de/sneak/vaultik/issues/290)). - -- 2026-10-13: Made an S3 listing that the server cuts short an error - ([issue #297](https://git.eeqj.de/sneak/vaultik/issues/297)); `prune` had - taken it as complete and could delete blobs still in use. - -- 2026-10-13: Made shallow `snapshot verify` report a blob it could not check - apart from a missing one - ([issue #318](https://git.eeqj.de/sneak/vaultik/issues/318)); any error - checking a blob had counted as missing, its cause dropped. - -- 2026-10-13: Restored the other files when a file's chunk is in no blob under - `--skip-errors` ([issue #326](https://git.eeqj.de/sneak/vaultik/issues/326)); - that file had stopped the restore before anything was restored. - -- 2026-10-13: Recorded each snapshot's new chunk size when its backup runs - ([issue #311](https://git.eeqj.de/sneak/vaultik/issues/311)). `snapshot list` - could show a NEW CHUNK SIZE of 0 B for a snapshot whose change was a modified - file. - -- 2026-10-13: Made a backup interrupted during a local index write exit as - interrupted ([issue #301](https://git.eeqj.de/sneak/vaultik/issues/301)); it - could end with `FATAL: failed to rollback transaction` and exit 1 instead - of 130. - -- 2026-10-13: Printed `snapshot list` and `snapshot purge` sizes like every - other command ([issue #312](https://git.eeqj.de/sneak/vaultik/issues/312)). - They, and the `bytes` value in terminal log lines, were 1024-based but - labelled KB or MB, and now use 1000-based units. - -- 2026-10-13: Closed each blob's storage reader when restore or deep verify is - interrupted, which ends a stalled read unless it is stuck in the kernel - ([issue #325](https://git.eeqj.de/sneak/vaultik/issues/325)). - -- 2026-10-13: Made a mistyped subcommand under `snapshot`, `config`, `remote`, - `database` or `completion` an error - ([issue #319](https://git.eeqj.de/sneak/vaultik/issues/319)); each had printed - its help and exited 0. - -- 2026-10-13: Made an exclude pattern starting with `**/` match directly under - the snapshot path too, as in gitignore - ([issue #303](https://git.eeqj.de/sneak/vaultik/issues/303)). - -- 2026-10-13: Refused a relative `file://` storage URL and decoded its - percent-escapes ([issue #299](https://git.eeqj.de/sneak/vaultik/issues/299)); - a relative path had moved with the working directory. - -- 2026-10-13: Removed the blob temp file when a backup stops part-way - ([issue #302](https://git.eeqj.de/sneak/vaultik/issues/302)); the half-written - blob had stayed in `$TMPDIR`. - -- 2026-10-13: Checked `--keep-newer-than` before a backup starts - ([issue #307](https://git.eeqj.de/sneak/vaultik/issues/307)); an invalid value - had been read only by the prune after the backup, so retention never ran. - -- 2026-10-13: Stopped `remote info` counting a file under `blobs/` that is not a - blob as an orphaned blob - ([issue #295](https://git.eeqj.de/sneak/vaultik/issues/295)). A stray file - such as `.DS_Store` was reported as an orphaned blob and the report advised - running `prune`, which skips such a file and deleted nothing. `remote info` - now takes its blob list from the same listing `prune` uses, in the table and - in `--json`. - -- 2026-10-13: Made `snapshot remove` fail for a snapshot that does not exist - ([issue #293](https://git.eeqj.de/sneak/vaultik/issues/293)). The command - never checked the ID: it reported the snapshot removed from the local database - and from the destination store and exited 0, under `--json`, `--dry-run` and - `--local-only` too. An ID that names no snapshot in the local index or on the - destination store is now an error; with `--local-only`, the snapshot must be - in the local index. The command now reports a removal from either side only - when one happened, and `remote_removed` is true only when metadata was - deleted. - -- 2026-10-12: Bound the local index to the destination an `s3.*` config names - ([issue #298](https://git.eeqj.de/sneak/vaultik/issues/298)). Only - `storage_url` was checked, so after `s3.bucket` or `s3.prefix` changed, the - next backup treated every chunk as already stored, uploaded nothing, and - completed a snapshot that could not be restored. Without `storage_url`, the - local index is now bound to `s3.endpoint`, `s3.bucket` and `s3.prefix`, and a - change is refused as a changed `storage_url` is. - -- 2026-10-12: Rejected a retention duration longer than about 292 years - ([issue #289](https://git.eeqj.de/sneak/vaultik/issues/289)). The number given - to `--older-than` or `--keep-newer-than` was multiplied by the length of its - unit with no limit, so from `293y`, `15251w`, `3559mo` or `106752d`, or a sum - of parts such as `200y200y`, the result wrapped to a negative duration. The - cutoff then lay in the future, every snapshot of the name was deleted, the one - just made included, and the command exited 0. Such a duration is now rejected - as invalid and nothing is deleted. - -- 2026-10-08: Made a backup cancelled under `--skip-errors` stop at once - ([issue #286](https://git.eeqj.de/sneak/vaultik/issues/286)). After Ctrl-C or - SIGTERM, phase 2 treated the cancellation error from each remaining file like - an unreadable file: it opened the file, printed an error line for it, counted - it as failed and went on to the next one. The run now stops at the first file - after the cancellation, and no file is counted as failed because of it. - -- 2026-10-08: Made `remote nuke` delete the `.partial` files that uploads cut - off part-way leave on the destination store - ([issue #281](https://git.eeqj.de/sneak/vaultik/issues/281)). The file and - rclone listings skip such a file, so the command left it in place and still - reported the store empty. It now removes them under `metadata/` and `blobs/` - as its last step. - -- 2026-10-08: Made a local index error while recording a directory or symlink - stop a backup under `--skip-errors` - ([issue #284](https://git.eeqj.de/sneak/vaultik/issues/284)). Phase 2 only - records such an entry in the local index, since it has no data to open or - read, but an error doing so was skipped like an unreadable file. The snapshot - then completed without the entry, and a restore did not recreate it. The error - now stops the backup, as it does without the flag. - -- 2026-10-08: Counted a file that a backup could not store as failed - ([issue #280](https://git.eeqj.de/sneak/vaultik/issues/280)). A file that - phase 1 counted and phase 2 could not open, because it was unreadable under - `--skip-errors` or removed in between, was reported in the summary as - unchanged with its bytes as backed up, and the `snapshots` row's `file_count` - and `total_size` included it. The summary now counts it as failed, and its - data total and the row leave it out. - -- 2026-10-08: Made `remote info` report a snapshot's blob count and blob size as - unknown when its manifest cannot be read - ([issue #272](https://git.eeqj.de/sneak/vaultik/issues/272)). The orphan - figures were already unknown in that case, but the snapshot's row still gave 0 - blobs and 0 B, in the table and in `--json`. The row now reads `unknown` and - `--json` gives `null`. A directory with no manifest still shows 0, since the - orphan figures count its blobs as orphaned. - -- 2026-10-08: Kept the progress line of a snapshot with more than one path - within 100% ([issue #271](https://git.eeqj.de/sneak/vaultik/issues/271)). The - bytes and files processed counted every path, but the totals they were divided - by held only the current path's, so a second path smaller than the first - showed more than 100% and an ETA of `unknown`. The totals now add up over the - paths scanned so far. The rate behind the ETA restarts when each path's scan - phase ends, so that phase does not lower the rate while the path is processed; - until then the rate is the previous path's. - -- 2026-10-08: Made a second `snapshot create` of one name succeed when it starts - in the same second as the first - ([issue #270](https://git.eeqj.de/sneak/vaultik/issues/270)). The timestamp in - a snapshot ID is in whole seconds, so the second run got the first run's ID - and failed with `UNIQUE constraint failed: snapshots.id`. When the local index - already has a snapshot with the ID, the create now waits a second and takes a - new timestamp. - -- 2026-10-07: Documented that `--json` skips the confirmation prompt of - `snapshot remove` and `prune` - ([issue #268](https://git.eeqj.de/sneak/vaultik/issues/268)). Both commands - delete without asking under `--json`, since a prompt on stdout would break the - JSON document, but the help and the README described only `--force` as - skipping it. The `--json` help of both commands and their README entries now - say so. - -- 2026-10-07: Made a symlink whose target cannot be read stop the backup - ([issue #269](https://git.eeqj.de/sneak/vaultik/issues/269)). It was left out - of the snapshot with only a debug log line, even without `--skip-errors`. It - now aborts the run, or with `--skip-errors` is skipped with the - `Failed to access` error line that any other entry the scan cannot read gets. - -- 2026-10-07: Stopped a killed rclone upload from leaving a truncated object - under its key ([issue #266](https://git.eeqj.de/sneak/vaultik/issues/266)). - The rclone backend wrote each object straight to its key, so killing an upload - to a local or sftp remote left a truncated object there; the next backup found - the key with `Stat`, skipped the upload and recorded a snapshot that could not - be restored. On a remote with a server-side move, an object is now written - under a temporary name ending in `.partial` and moved into place, and listings - skip such names. Remotes without one are still written in place. - -- 2026-10-07: Made an interrupted command exit 130 and say so - ([issue #267](https://git.eeqj.de/sneak/vaultik/issues/267)). Ctrl-C or - SIGTERM during `snapshot create`, `snapshot restore` or `snapshot verify` - exited 0 with no error line (`snapshot verify --json` exited 1, also without - one), so a `--cron` run that never finished looked like a success. A command - stopped by either signal now exits 130 and prints - `interrupted before the command finished` on stderr, under `--cron` and - `--json` too. - -- 2026-10-07: Corrected documentation, help text and comments that were false - about the code ([issue #233](https://git.eeqj.de/sneak/vaultik/issues/233)). A - blob is not streamed to storage. The README, `ARCHITECTURE.md` and - `config.example.yml` now say a backup needs free temporary space, because each - blob is written whole to a temporary file before it is uploaded (up to about - `blob_size_limit`; an rclone destination that cannot stream uploads needs - about twice that) and the metadata export writes copies of the local index. - Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite - writes one of those copies to `/var/tmp`. Also corrected: the snapshot ID - format, what restore reads and how incomplete snapshots are removed in - `docs/DATAMODEL.md`, what `source_path` holds, the `index_path` and config - file defaults, what `snapshot remove` cleans up, how the release gets its Go - toolchain, and the `script/release` and `script/fmt-check` comments. - -- 2026-10-07: Cut the time the `internal/vaultik` and `internal/database` tests - take ([issue #235](https://git.eeqj.de/sneak/vaultik/issues/235)). Most of the - `internal/vaultik` time went to 24 tests that ran one at a time only because - they call `log.Initialize`; they now call it before `t.Parallel()`, as the - package's other tests do. `TestLargeDatasets` committed each of its 1,500 - inserts on its own and now makes them in one transaction. - `TestDedupOnlySnapshotRestores` gives its second backup its own snapshot name - instead of sleeping past the one-second timestamp in the snapshot ID. - -- 2026-10-07: Made two messages say only what is true - ([issue #240](https://git.eeqj.de/sneak/vaultik/issues/240)). A config file - that others can read was warned about as containing S3 credentials even when - it set none, as a `file://` config does. The warning now says the file may - contain S3 credentials only when `s3.access_key_id` or `s3.secret_access_key` - is set, since either may come from a `${...}` reference rather than the file, - and otherwise says the file is readable by others. `snapshot purge` against a - destination store it could not list gave an error with - `listing remote snapshots:` in it twice; the prefix now appears once. - -- 2026-10-07: Made `s3.part_size` set the multipart upload part size - ([issue #232](https://git.eeqj.de/sneak/vaultik/issues/232)). It was loaded - and defaulted but never passed to the S3 client, whose uploader used a fixed - 10MiB part. It now reaches the uploader for `storage_url` and for the `s3.*` - fields, and a part size S3 refuses, below 5MiB or above 5GiB, `0` included, - fails at config load. A blob too large for S3's limit of 10,000 parts at the - configured size is uploaded in larger parts. The docs gave the default as - `5MB`, which the config file reads as 5,000,000 bytes, below the minimum; they - now say `5MiB`. - -- 2026-10-07: Made per-name retention work when the hostname contains `_` - ([issue #230](https://git.eeqj.de/sneak/vaultik/issues/230)). A snapshot ID is - `hostname_name_timestamp`, and the name was read as everything between the - first and the last `_`, so with `hostname: my_host` the name `home` came out - as `host_home`. `snapshot purge --keep-latest --snapshot home` then printed - "No snapshots to delete", and `snapshot create --prune` purged nothing without - a message. The name is now read using the hostname the `snapshots` table - stores with each snapshot, cut at its first `.` as it is in the ID. - -- 2026-10-07: Made `remote info` stop reporting a snapshot's blobs as orphaned - when its manifest cannot be read, and stop printing raw names from under - `metadata/` ([issue #228](https://git.eeqj.de/sneak/vaultik/issues/228)). A - manifest it failed to read was skipped, so that snapshot's blobs were counted - as orphaned and the report advised running `vaultik prune`. The orphan figures - are now unknown in that case, with no prune advice, and `--json` gives them as - `null` with the unreadable remote keys in `unreadable_manifests`. A name under - `metadata/` that is not 64 lowercase hex characters is now skipped with a - warning instead of being printed, control characters included. A manifest - under a skipped name is then not read either, so it also leaves the orphan - figures unknown, and `--json` counts such manifests in - `skipped_manifest_count`. A directory with no manifest in it, as left by an - interrupted backup, leaves the figures known. - -- 2026-10-07: Made `config set` keep a string that looks like a number - ([issue #229](https://git.eeqj.de/sneak/vaultik/issues/229)). It wrote every - value unquoted, and `config.Load` reads the file through untyped YAML, so an - access key `00112233` loaded as `38043` and a hostname `007` as `7`. A value - for a string setting in `config.Config` is now tagged as a YAML string, which - the file quotes wherever YAML would read a number or a boolean; other settings - are still written unquoted. - -- 2026-10-07: Made a backup notice a file rewritten with its size unchanged and - a new mtime in the same second as the one in the index - ([issue #226](https://git.eeqj.de/sneak/vaultik/issues/226)). The `files` - table held mtime in whole seconds and the scanner compared whole seconds, so - every later snapshot kept the old content. A new `mtime_nsec` column holds the - nanoseconds within the second that `mtime` holds, and the scanner compares the - full mtime. A local index created before the change lacks the column and is - rebuilt with `vaultik database delete` and a full backup. - -- 2026-10-07: Made taking the process-wide lock atomic - ([issue #227](https://git.eeqj.de/sneak/vaultik/issues/227)). The lock read - `vaultik.pid`, checked whether that PID was alive and then wrote its own, so - two writers started together could both pass the check and both run. It is now - an `flock` on `vaultik.pid`, held until the run ends; the kernel drops it when - the process exits, so a crash leaves no lock behind. A clean exit now empties - the file instead of deleting it, because deleting it would let two later runs - each lock a different file. - -- 2026-10-06: Made `snapshot remove --json` write only its document to stdout - when the destination store cannot be reached - ([issue #251](https://git.eeqj.de/sneak/vaultik/issues/251)). Its warning that - the snapshot's metadata was left on the destination store went to stdout ahead - of the document, breaking `| jq` on a command that exited 0. Under `--json` - the warning now reaches stderr only, through the logger. The warning, the - README and the command's help said `vaultik prune` would finish the cleanup, - but `prune` never removes snapshot metadata; they now say to run - `vaultik snapshot remove` for the snapshot again once the destination store is - reachable. - -- 2026-10-06: Made the backup summary and the `snapshots` row count each file, - byte and upload once - ([issue #225](https://git.eeqj.de/sneak/vaultik/issues/225)). The scanner - added a file's bytes again for each new chunk and counted a file as unchanged - for each chunk already stored, so a first backup reported twice its size and - "backed up" could go negative. Upload figures came from the progress reporter, - which `--cron` turns off, and `blob_count` counted earlier paths' blobs again - for each later path. The scanner now counts uploads itself; `blob_size`, - `blob_uncompressed_size` and `compression_ratio` describe the blobs the - snapshot references, and `docs/DATAMODEL.md` now says `chunk_count` and - `blob_count` count what the run added. - -- 2026-10-06: Made command output follow the README's stdout and stderr rules - ([issue #224](https://git.eeqj.de/sneak/vaultik/issues/224)). The startup - banner went to stdout, so a `completion` script or a `config get` value - started with it; the banner now goes to stderr. A failing `remote info`, - `prune` or `snapshot remove` under `--json` printed nothing on either stream, - and now reports its error on stderr. `snapshot verify --quiet` printed its - whole report; it now prints none, and a failure still reaches stderr with the - same exit status. - -- 2026-10-06: Made a backup without `--cron` of a snapshot with two or more - `paths` complete instead of panicking with `close of closed channel` - ([issue #253](https://git.eeqj.de/sneak/vaultik/issues/253)). `Scan` runs once - per path and started and stopped the progress reporter each time, and a second - stop panics. The reporter is now started and stopped once per snapshot, around - the scans of all its paths. - -- 2026-10-06: Made a restore path argument select only that path and what is - beneath it ([issue #223](https://git.eeqj.de/sneak/vaultik/issues/223)). The - lookup matched with SQL `LIKE`, so `/home/u/doc` also restored `doc2`, `DOC` - and `doc.txt.bak`, and a `_` or `%` in the path acted as a wildcard. A backup - used the same lookup to load the known files of each configured path, so files - of a longer sibling path were counted as deleted. - `FileRepository.ListUnderPath`, which replaces `ListByPrefix`, returns the - file at the path and every file whose path starts with the path plus `/`, - compared case-sensitively. - -- 2026-10-06: Made restore return an error instead of panicking on a malformed - snapshot database - ([issue #231](https://git.eeqj.de/sneak/vaultik/issues/231)). A chunk hash - shorter than 16 characters crashed the error message naming it, and `--verify` - dereferenced a missing `chunks` row and allocated whatever chunk size the - database gave. Those messages now go through `shortHash`, a missing row is an - error, and `--verify` rejects a negative size and hashes each chunk as a - stream. - -- 2026-10-06: Made `s3://bucket/prefix` and `s3://bucket/prefix/` the same - destination ([issue #222](https://git.eeqj.de/sneak/vaultik/issues/222)). The - S3 client put the prefix directly in front of each key, so a prefix without a - trailing slash stored `prefixblobs/...`. A non-empty prefix is now joined to - every key with one `/`, giving the README's `//blobs/...` - layout. The `s3.prefix` config setting goes through the same client and gets - the same join. - -- 2026-10-06: Made restore apply owners, modes and times in an order that keeps - them ([issue #219](https://git.eeqj.de/sneak/vaultik/issues/219)). A directory - got its stored mode and mtime before its contents were written, so a read-only - directory came back without its files and a non-empty one carried the time of - the restore. Directories are now created owner-only and get their stored - owner, mode and mtime once the restore loop is done, each before its parent. A - file's mode is applied after its chown, which on Linux clears setuid and - setgid, and a symlink gets its stored owner (as root) and mtime on the link - itself. - -- 2026-10-06: Made `go.mod` what `go mod tidy` writes, so the pre-commit hook no - longer stops every commit - ([issue #246](https://git.eeqj.de/sneak/vaultik/issues/246)). A test in - `internal/cli` imports `github.com/spf13/pflag` directly, but `go.mod` still - marked it `// indirect`, and `script/precommit` fails whenever the tidy - changes `go.mod`. It is now in the direct `require` block. - -- 2026-10-06: Made the README's steps for restoring on another machine work - ([issue #221](https://git.eeqj.de/sneak/vaultik/issues/221)). `config init` - wrote a placeholder recipient that `config.Load` rejects, so every command on - the new machine failed before reaching the store. The file now has an empty - `age_recipients` list, `config.Load` accepts an empty list, and - `snapshot create` refuses to run without a recipient. - -- 2026-10-06: Made `snapshot restore --skip-errors` skip the files that need a - blob it cannot download - ([issue #218](https://git.eeqj.de/sneak/vaultik/issues/218)). A missing or - damaged blob ended the restore even with the flag, after restoring whichever - files happened to come first. Every file that needs such a blob is now - reported as failed, the rest are restored, and the command still exits - non-zero. Without the flag the blob error still aborts. - -- 2026-10-06: Re-vendored the canonical files from `sneak/prompts` at `dd4027b` - ([issue #213](https://git.eeqj.de/sneak/vaultik/issues/213)). Linting and - testing are now the `lint` and `test` phases of the `Dockerfile`, and the - build stage depends on both. `Dockerfile.lint`, `CHECK_EPOCH` and the tests - that checked them are gone; every `docker build` in `script/` passes - `--no-cache` instead. golangci-lint is v2.14.0, with its new findings fixed in - the code, and the rules in `CLAUDE.md` now live in `AGENTS.md`. - -- 2026-10-06: Made the local index actually run in WAL mode with a busy timeout - ([issue #217](https://git.eeqj.de/sneak/vaultik/issues/217)). The connection - settings were written in a form the SQLite driver ignores, so the index ran - without either and `snapshot list` or `info` during a backup could make the - backup's next write fail with `database is locked`. They are now `_pragma=` - parameters, and the metadata export copies the open index with `VACUUM INTO`, - because a copy of the file alone misses rows still in the `-wal` file. - -- 2026-10-06: Made a backup record the real uid and gid of files, directories - and symlinks ([issue #216](https://git.eeqj.de/sneak/vaultik/issues/216)). The - scanner asked the stat result for `Uid()` and `Gid()` methods, which - `*syscall.Stat_t` does not have, so every entry was stored as `0:0` and a - restore as root gave every file to root. It now reads the `Uid` and `Gid` - fields of `*syscall.Stat_t`. - -- 2026-10-06: Made a `file://` destination whose directory is missing count as - one that cannot be listed - ([issue #220](https://git.eeqj.de/sneak/vaultik/issues/220)). The file backend - listed a missing directory as an empty store, so with the quickstart's USB - stick unplugged `snapshot list` reported every local snapshot as missing from - the store, `snapshot remove` claimed to have removed metadata it never - reached, and `prune` dropped every local snapshot record. Listing a missing - destination directory is now an error; a first backup still creates the - directory. - -- 2026-10-06: Made `--older-than` and `--keep-newer-than` reject a duration with - characters outside its number-and-unit parts - ([issue #215](https://git.eeqj.de/sneak/vaultik/issues/215)). The parser - picked out the parts it recognised and skipped the rest, so `1.0y` became zero - and `--prune --keep-newer-than 1.0y` deleted every snapshot of the backed-up - names, the new one included. `1.0y`, `1,5y`, `30 days`, `x7d` and a bare `0` - are now errors; decimals still work in Go units such as `1.5h`. - -- 2026-10-06: Made a backup re-chunk a known file that lists a chunk no uploaded - blob holds ([issue #214](https://git.eeqj.de/sneak/vaultik/issues/214)). File - rows are shared by every snapshot and updated in place, so removing or pruning - the only snapshot that referenced a changed file's current blob left an older - snapshot keeping a row that matched the disk while no blob held its chunks. - The next backup skipped the file and completed a snapshot that could not - restore it. - -- 2026-09-22: Routed the last direct-to-stdout command output through - `internal/ui` ([issue #149](https://git.eeqj.de/sneak/vaultik/issues/149)). - The `version`, `info`, `remote info`, `config`, and `database delete` commands - wrote plain text straight to stdout, so they were unstyled and ignored - `--quiet`. Output now falls in two buckets, both governed by `internal/ui`: - status lines and confirmations go through its message methods (styled, and - `--quiet` silences them), while the data a command exists to produce — the - reports, the `snapshot list` table, `config get` values, and the `--json` - documents — is written plain. `--quiet` silences the human reports and tables - but never the `config get` value or the `--json` documents, which a script - depends on, and the `database delete` confirmation prompt is always shown. The - README output-style section now states this rule. - -- 2026-09-22: Validated blob hashes, offsets and lengths read back from the - destination before using them - ([issue #155](https://git.eeqj.de/sneak/vaultik/issues/155)). A blob hash - taken from the downloaded database or the store listing was trusted unchecked, - so a hostile remote could drive a decrypted blob to be written outside the - cache directory (a hash like `aa/../../etc`) or crash a command with a short - or negative value. `blobDiskCache.path` now refuses any key containing a path - separator, and `ReadAt` rejects a negative offset or length and bounds with - `length > size-offset` so a sum cannot overflow past the check. A new - `isBlobHash` helper (a plain function, not a method — the packer stores - `temp-placeholder-{uuid}` as a hash) gates `FetchBlob`, shallow and deep - verify; the `blobs/` and `metadata/` listings skip a non-conforming name with - a warning; and short-hash prefixes in log and error text go through a - `shortHash` helper that cannot panic. `verify`'s chunk reader also rejects a - negative `blob_chunks` length and streams the chunk instead of allocating a - database-supplied size. `restore.go` and `internal/database` were left - untouched to avoid colliding with the in-flight - [issue #156](https://git.eeqj.de/sneak/vaultik/issues/156) work; the - cache-path and `FetchBlob` guards already stop the unsafe write and fetch, so - restore's own `buildBlobIndexes` early check is deferred as fail-fast defense - in depth. - -- 2026-09-21: Stopped an interrupted blob upload from making a later backup - deduplicate against data that was never stored - ([issue #148](https://git.eeqj.de/sneak/vaultik/issues/148)). The packer - commits a blob's `chunks`, `blob_chunks`, and `blobs` rows before the upload - is attempted, so a failed upload left chunk rows behind and the next run - skipped re-uploading them, producing a snapshot that reported success but - could not be restored. A run now deduplicates only against chunks held by a - blob whose `uploaded_ts` is set, and at startup drops any un-uploaded blob - rows (and the chunks they orphan) so the affected data is re-chunked and - re-uploaded. Blobs recorded with no remote backend are marked uploaded so this - invariant holds uniformly. - -- 2026-09-22: Made restore refuse any snapshot path that would write outside the - target directory ([issue #154](https://git.eeqj.de/sneak/vaultik/issues/154)). - `restoreFile` and `verifyRestoredFiles` joined the stored path onto the target - with no containment check, so a `..` segment or an absolute path escaped the - target and a restored symlink could redirect a later child write anywhere on - disk. Every stored path is now rejected unless `filepath.IsLocal` accepts it - with the leading separator removed, and each existing ancestor directory below - the target is `Lstat`ed to refuse descending through a symlink; honest - symlinks pointing outside the tree are still written verbatim. age decryption - proves a snapshot is readable, not honest, and restore usually runs as root. - -- 2026-09-21: Stopped `--json` from silencing stderr diagnostics - ([issue #112](https://git.eeqj.de/sneak/vaultik/issues/112)). `--json` used to - be folded into `Quiet`, which pinned the log level to `WARN`, so - `prune --json` gave a machine consumer no record of the local index rows it - deleted even under `--verbose`. `--json` now quiets only the stdout UI (the - JSON document must stay clean, per - [issue #108](https://git.eeqj.de/sneak/vaultik/issues/108)); the stderr log - level follows `--verbose`/`--debug` again. The coupling was removed the same - way for `snapshot verify`, `snapshot remove`, and `remote info`, which carried - it for the same outdated reason. - -- 2026-09-21: Stopped `prune` from reporting a failed row count as 0 - ([issue #96](https://git.eeqj.de/sneak/vaultik/issues/96)). The seven - `getTableCount` reads in `PruneDatabase` discarded their error, so a query - that could not run became a plausible `0` and the before/after delta computed - from it looked like real work. Each read now logs at warn on failure and - renders as `unknown`, never `0`, so an empty table is distinguishable from one - that could not be queried. The counts have no `--json` representation — under - `--json` the summary is suppressed entirely — so nothing there can show a - false `0`. - -- 2026-09-21: Made the s3 storage backend report a missing object as - `storage.ErrNotFound`, like the `file` and `rclone` backends and as the - `Storer` interface documents. `S3Storer.Get` and `Stat` returned the raw AWS - SDK error, so `errors.Is(err, storage.ErrNotFound)` was false on s3 and - callers branched differently per backend. Added a small `s3.IsNotFound` helper - (reused by `HeadObject`) and a test that a missing key maps to `ErrNotFound` - ([issue #129](https://git.eeqj.de/sneak/vaultik/issues/129)). -- 2026-09-21: Fixed `verify --deep` reporting healthy snapshots as corrupt. Its - final blob-integrity check hashed the encrypted downloaded bytes with a single - SHA256 and compared that to the blob ID, which is the double SHA256 of the - plaintext, so the two could never match. It now hashes the decompressed - plaintext and compares the double SHA256. Added a test that backs up a real - snapshot, deep-verifies it, then flips a byte in one stored blob and confirms - deep verification then fails - ([issue #131](https://git.eeqj.de/sneak/vaultik/issues/131)). - -- 2026-09-21: Made `snapshot create` VACUUM the per-snapshot metadata database - through the `modernc.org/sqlite` driver instead of shelling out to the - external `sqlite` command-line binary (issue #120). A backup no longer needs - that binary on `PATH`, so `make check` passes on a stock `go install` host; - `script/bootstrap` and the `Dockerfile` (both the test-build and the shipped - runtime stage) no longer install it, and a new test asserts the uploaded - database keeps no pages from deleted rows. Dropped the now-false note on the - 2026-08-07 entry below that said bootstrap installs it. -- 2026-09-21: Made `.gitea/workflows/check.yml` run on pushes to `main` and - `next` and on pull requests against either, so unit PRs (whose base is `next`) - and `next` itself get a CI run instead of relying on a local `make check` - ([issue #122](https://git.eeqj.de/sneak/vaultik/issues/122)). - -- 2026-09-21: Hash-verified the Go toolchain in the release workflow - ([issue #105](https://git.eeqj.de/sneak/vaultik/issues/105)). New - `script/install-go` downloads the exact `go.dev` archive for `go.mod`'s `go` - directive and refuses it unless its sha256 matches a value committed in the - script; `.gitea/workflows/release.yml` calls it instead of `actions/setup-go`, - which verified the downloaded toolchain against nothing in the repo. - `GOTOOLCHAIN: local` on the release step keeps that exact compiler from - auto-switching. Bumping Go now touches `go.mod`, the checksum, and the - `Dockerfile` `golang` digest together. - -- 2026-09-21: Collapsed the two duration parsers into one and fixed the - `--older-than` months example - ([issue #123](https://git.eeqj.de/sneak/vaultik/issues/123)). Two functions - named `parseDuration` existed with different grammars; - `snapshot purge --older-than` and `--keep-newer-than` both already went - through the one in `internal/vaultik`, while the richer copy in - `internal/cli/duration.go` was reachable only from its own test. Kept the - live-path parser and deleted the unused one, so no flag's accepted grammar - changes. The trap the issue was filed over: `README.md` documented `6m` as the - months example for `--older-than`, but `m` is minutes, so the documented - command deleted every snapshot older than six minutes on a destructive flag. - Corrected the doc to `6mo` and put both flags' help text on one example list - that states `m` is minutes and `mo` is months. The surviving parser now - rejects negatives, which it previously accepted (`-5h`) or silently made - positive (`-5d`). Table-driven tests cover every unit, `6m` as six minutes, - `6mo` as 180 days, and rejection of a bare number, an unknown unit, and a - negative. - -- 2026-08-10: Moved every lint run into its own container, as a build step - ([issue #113](https://git.eeqj.de/sneak/vaultik/issues/113)). New root - `Dockerfile.lint`, built by `script/lint`, runs - `golangci-lint run --config .golangci.yml ./...` as a `RUN` instruction in the - digest-pinned `golangci/golangci-lint` image: a successful build of that file - _is_ a clean lint, and it works even where the daemon is remote and bind - mounts are impossible. That `FROM` line is now the only pin of the linter - version in the repo. - - This supersedes the per-worktree cache isolation landed for - [issue #99](https://git.eeqj.de/sneak/vaultik/issues/99). Isolation fixed - cross-worktree contamination but not lock contention — two concurrent runs - with entirely separate cache directories still collided. A container per run - has its own cache and its own lock, so the whole class is gone, and with it - the per-worktree cache machinery, the lock-retry loop, and - `script/lint-audit`, which existed to catch replayed findings from a cache - that no longer exists. The host lint path went too: no escape hatch, no - `VAULTIK_LINT_IN_CONTAINER`, no version detection. Nothing lints on the host - at any version. - - A cached build lints nothing, so the same `CHECK_EPOCH` mechanism the - product `Dockerfile` already used is what makes the green mean something: - `ARG CHECK_EPOCH` with no default below the module layers, a - `RUN [ -n "$CHECK_EPOCH" ] || exit 1` guard, the value expanded into each - check command, and a fresh `$(date +%s%N)$$` per invocation computed as a - bare assignment. `cmd/vaultik/lintdocker_test.go` parses both Dockerfiles - and both scripts and fails if any part of that is dropped, because every way - of losing it is silent. No test asserts that no script runs the host linter: - `script/lint` is the one lint entry point and runs `golangci-lint` only - inside the container, and keeping it that way is a review matter, not - something a test proves. - - The product `Dockerfile` lost its lint stage rather than gaining a second - linter pin: `make lint` is now `docker build`, so the stage would have been - docker-in-docker with no daemon, and calling `golangci-lint` directly there - would have restored the two-pins drift of - [issue #78](https://git.eeqj.de/sneak/vaultik/issues/78). `make fmt-check` - moved beside `make test` in the builder stage, and `script/cibuild` now - builds `Dockerfile.lint` and then `Dockerfile`, each with its own fresh - epoch. Consequence, stated rather than left to be found: `script/docker` - builds the product image only and no longer lints; the gates are - `script/check` and `script/cibuild`. - - `golangci-lint config verify` runs as its own epoch-keyed layer, above the - lint. `golangci-lint run` rejects a config it cannot parse but silently - ignores an unknown top-level _key_: renaming `linters:` to `linterz:` - discarded `default: all` and every threshold and still exited 0 on a tree - the real config fails. `config verify` catches that, and it does so with the - network off at this pin — checked under `docker run --network none`, not - assumed. An earlier revision omitted it on the claim that it fetches its - schema over live HTTPS; that claim was false at v2.12.2. - - `script/lint-fix` is kept, reimplemented as a bind-mounted `docker run` - against the image parsed out of `Dockerfile.lint` — it cannot be a build - step, because fixes have to land in the worktree — and marked in its header - as a developer convenience that no gate reads. - -- 2026-08-09: Finished the `--json` stdout contract and gave `make build` a rule - ([issue #108](https://git.eeqj.de/sneak/vaultik/issues/108), - [issue #110](https://git.eeqj.de/sneak/vaultik/issues/110)). Two unrelated - defects of the same shape — a command reporting something it did not do — - landed together because both are small. - - `CleanupLocalSnapshots` wrote three prose lines to stdout with no `--json` - awareness, covering every branch of the function, so no input avoided them - and `vaultik prune --json | jq` failed even after - [issue #106](https://git.eeqj.de/sneak/vaultik/issues/106) removed the - banner. `-q` never helped either: `printlnStdout` and `stdoutf` write - straight to `Vaultik.Stdout` and never consult `Vaultik.UI`, which is what - `SetQuiet` affects. The issue offered three fixes and asked for a decision. - Taken: thread `*PruneOptions` into the function and gate each write on - `!opts.JSON`, matching `PruneBlobs` (its sibling phase, which already takes - the same struct), `RemoveSnapshot` and `remote info`, so the package has one - pattern rather than two. Rejected: moving the lines to `log.Info`, because - the logger's default level is `slog.LevelWarn`, so that would not relocate - them to stderr — it would delete them from a plain `vaultik prune`, and the - removal of rows from the local index is not something to narrate only under - `--verbose`. Also rejected: putting the stale-record count into - `PruneBlobsResult`, whose every field is blob-scoped and which is produced - by the later phase; a prune document covering both phases is a reasonable - thing to want, but it is a schema design question and not a stream-hygiene - fix. The narration is duplicated as `log.Info` records, which `PruneBlobs` - already does alongside its own prints, so the events survive on stderr for - anyone running `--verbose`. - - `make build` printed "Nothing to be done for 'build'" and exited 0 without - producing a binary: `build` was listed in `.PHONY` with no `build:` rule - anywhere, and declaring a name phony is exactly what converts make's "No - rule to make target" error into a silent success. Fixed with - `build: vaultik`, keeping `vaultik:` as the file rule. The audit the issue - asked for covers all 19 `.PHONY` names; `build` was the only one without a - rule, and `vaultik` is correctly absent from `.PHONY`, being a real file - target. - - Tests, each verified to fail with the fix reverted rather than assumed to: - `CleanupLocalSnapshots` leaves stdout untouched under `--json` in all three - branches (stale records, none, empty index) and still emits every line - without it, so the guard cannot be satisfied by deleting the output; - `prune --json` run end to end through `Entry`, cobra and fx over the - process's real stdout descriptor against a `file://` store, asserting - exactly one JSON document, in both the stale and non-stale branches; and a - parse of the `Makefile` asserting every `.PHONY` name has a rule and that - `build` reaches the rule that produces the binary, which keeps the audit - true for names added later. That last one is a parse rather than an - invocation of `make`, since `make test` is what runs it and shelling back - into `make build` would nest a build inside the test run. The property a - parse cannot establish — that the recipe still fails when the build fails — - was verified by hand against a deliberately broken tree: `make build` exits - 2 and produces nothing. `cmd/vaultik` gains its first test file, so - `make test` now reports 16 packages `ok` where it reported 15. - -- 2026-08-09: Stopped the startup banner from contaminating `--json` documents - ([issue #106](https://git.eeqj.de/sneak/vaultik/issues/106)). `Entry` writes - the banner to stdout before cobra parses anything, and the flag scan that - suppresses it knew `--quiet`, `-q` and `--cron` but not `--json`, so every - `--json` document arrived behind two lines of prose and a blank line, and - `vaultik snapshot list --json | jq` failed. With the logger already on stderr - from [issue #82](https://git.eeqj.de/sneak/vaultik/issues/82), this was the - last writer that could put something on stdout that the caller did not ask - for. The design question the issue raised — extend the raw-argv scan, or move - the banner after parsing — is answered in favour of the scan: the banner is - printed first deliberately, so that it still appears when cobra rejects the - arguments and on `--help`, and after parsing there is no single place that - covers those paths. The stated cost of the scan, that `--json` is a subcommand - flag matched anywhere in the vector, is a cost `--cron` already carries — it - exists only on `snapshot create` — so this adds an instance of an accepted - imprecision rather than a new kind, and the two error directions are not - symmetric: a false positive loses a decorative banner, a false negative - corrupts a document. Regression tests at the CLI layer, where - `internal/vaultik`'s existing guard cannot reach: one runs `Entry` itself over - the process's real stdout descriptor, through cobra and fx to the document, - made hermetic by `file://` storage; a second covers the argument vectors of - all five `--json` commands; a third asserts the banner is still printed - without a suppressing flag, so the first cannot be satisfied by deleting the - banner. Also corrected `AGENTS.md` policy 9, which still keyed the - structured-log format on stdout's TTY-ness after #82 moved that decision to - stderr — a rules file that misdescribes the code misleads exactly the readers - who trust it most. Two smaller findings from the same review: `bytesAttrKey`'s - human-readable byte formatting silently stopped applying under an open group, - because the key reaching the comparison is group-qualified (`transfer.bytes`), - now matched on its final segment and tested both ways; and `listEnv.stderr` in - `snapshot_list_test.go`, assigned but never read since those tests began - capturing the process's stderr, is removed. `Vaultik.Stderr` is kept — nothing - writes to it today, which its comment now says outright. - -- 2026-08-09: Moved the logger to stderr and fixed `TTYHandler`'s discarded - attributes ([issue #82](https://git.eeqj.de/sneak/vaultik/issues/82), - [issue #97](https://git.eeqj.de/sneak/vaultik/issues/97)). Two defects in - `internal/log`, fixed together because both live in the handler construction - path. The first: both handlers were built over `os.Stdout`, and `WARN`/`ERROR` - are never suppressed, so a config file with group- or world-readable - permissions was enough to put a log record inside a `--json` document and - break `jq`. Diagnostics now go to stderr, and the TTY/JSON format choice - follows stderr rather than stdout — testing the wrong stream would colorize - records on a redirected stderr whenever stdout happened to be a terminal. This - is user-visible: `--verbose` and `--debug` output moves to stderr too, which - is documented in `README.md` under "stdout and stderr". It also let the local - workaround in `internal/vaultik/snapshot_list.go` go: `warnWhileListing` had - been hand-rolling structured-log formatting to reach a non-stdout writer, and - the `jsonOutput` parameter threaded through the remote-listing helpers existed - only to choose between the two writers. The collect-then-emit machinery around - `listingWarning` stays, but on its remaining merit — warnings emitted in key - order after `group.Wait()` are deterministic run to run, where emitting from - the fetch workers would order them by network timing. The second defect: - `TTYHandler.WithAttrs` and `WithGroup` discarded their arguments and returned - the receiver while their doc comments claimed otherwise, so `log.With` - attributes vanished on a terminal and appeared correctly in CI — failing - precisely when someone is debugging interactively. Both now return a new - handler (the receiver is never written to, since `slog` permits concurrent - derivation), attributes persist across records, and grouping is implemented as - dotted key prefixes, which is the only honest rendering for a format with - nowhere to nest. New tests cover both, including one that feeds the same - derivation chain to the TTY and JSON handlers and compares the attribute sets, - so the two paths cannot drift apart again. Found and filed while verifying: - the startup banner is written to stdout and `--json` does not suppress it - ([issue #106](https://git.eeqj.de/sneak/vaultik/issues/106)), which is a - separate writer on a separate path and the remaining source of stdout - contamination. - -- 2026-08-09: Made the tagged-release path actually work on Gitea - ([issue #65](https://git.eeqj.de/sneak/vaultik/issues/65)). Three independent - blockers, one of which was the whole release: `.goreleaser.yaml` had no - `gitea_urls:` block, so goreleaser defaulted to the GitHub API and a - `goreleaser release` from this repo would have failed or published where - nobody is looking. It now points at `https://git.eeqj.de/api/v1`. The version - is the second: it was a hardcoded `VERSION := 1.0.0-rc.1` in the `Makefile`, - so every local build claimed to be a release candidate that had never been - tagged and did not exist, while `git tag -l` was empty and `internal/globals` - defaulted to `dev`. Version now comes from git via the new `script/version` — - the exact tag with a leading `v` stripped (so a `make` build and a goreleaser - build of one commit report the same string, and it matches the archive names), - otherwise `dev-<12-char sha>`, with `-dirty` appended in either case when - tracked files are modified. Untracked files are deliberately not counted, - matching `git describe --dirty`. The same honesty was owed by the snapshot - path: `snapshot.version_template` was `{{ incpatch .Version }}-next`, which - manufactures a release number from the last tag and, with no tags at all, from - goreleaser's fabricated `v0.0.0`; it now emits the same `dev-`. The one - non-obvious consequence is that `internal/cli/version.go` gated its "this is a - development build" notice on the version being exactly `dev`, so the moment - untagged builds began carrying a commit sha that notice would have gone silent - and an unreleased binary would have read as a release — the gate is now - `globals.IsDevVersion`, which is a predicate over a string rather than a - comparison against a global precisely so it can be tested, and it is tested at - the boundary (`1.0.0-dev` is a release, `dev-` is not). Release - automation is the third blocker: a tag-triggered - `.gitea/workflows/release.yml` runs the build in CI rather than from a laptop, - with `fetch-depth: 0` because a shallow checkout has no tags and would - silently mislabel the release, and with the `RELEASE_TOKEN` repository secret - passed as `GITEA_TOKEN` (documented in `README.md`; the runner's automatic - token is not used because it is not guaranteed to carry release write scope). - `script/release` unsets any `GITHUB_TOKEN`/`GITLAB_TOKEN` it finds, since - goreleaser chooses its forge from whichever token variable is set and refuses - to run when it sees more than one — a runner-provided token must not get to - decide where these artifacts are published. `make release` and - `make release-snapshot`, the last two Makefile targets that were not shims, - now call `script/release` and `script/release-snapshot`, which resolve - goreleaser exactly the way `script/lint` resolves the linter: a `PATH` binary - is used only at the pinned version, never as a silent fallback. - `script/bootstrap` installs it, from a sha256-verified GitHub release archive - per `REPO_POLICIES.md`, via a separate `script/install-goreleaser` — separate - because `script/bootstrap` hard-fails without a usable Docker daemon by - design, and the release runner needs goreleaser without needing Docker. - Verified by running the thing rather than reading it: `make release-snapshot` - produced four archives and `checksums.txt`, and the linux/amd64 binary from - `dist/` reports `dev-` with the development-build notice. Tag handling - was exercised in a throwaway repository rather than by tagging this one; no - tag was created here, since that is the owner's call. Signing, SBOM, - reproducible builds, completions and a man page are out of scope by the issue. - -- 2026-08-09: Isolated the lint cache per worktree and context-gated the native - lint path (issues #99, #80). One defect seen twice: `script/lint` decided - whether it could skip the pinned image by asking what version was on `PATH` - rather than where it was running, and cache isolation is part of that same - question. The cache was one directory per repo, shared by every worktree on - the host, so two checkouts with identical Go file contents collided and - golangci-lint replayed the stored analysis — paths and all. The loud direction - of that failure (a clean tree failed by a dirty sibling) is the harmless one; - the silent direction, a dirty tree **passed** by a clean sibling, is a sixth - way for a gate here to report a green it did not earn. The cache is now keyed - on a digest of the worktree path, and every run is audited by the new - `script/lint-audit`, which rejects output citing any file that is not in the - tree being linted — a backstop that runs on clean output too, because that is - the case nobody investigates. Caches record the worktree they belong to and - are collected when it disappears, so throwaway worktrees do not accumulate - them; the whole tree lives under `XDG_CACHE_HOME` and is disposable. The - `parallel golangci-lint is running` refusal is now a bounded retry rather than - a verdict: it is not a lint result, and exiting non-zero on it is - indistinguishable to a caller from real findings (#88 showed a private cache - does not remove that contention). The native path now requires - `VAULTIK_LINT_IN_CONTAINER=1`, set only by the `Dockerfile` lint stage, in - addition to matching the pin, so a developer's locally installed 2.12.2 no - longer bypasses the digest pin; `/.dockerenv` was rejected as the signal - because `dockerd` creates it for `docker run` and it is not reliably present - during a BuildKit `docker build`, which is the case the exception exists for. - Version detection uses `golangci-lint version --short` with the old banner - scrape kept only as a fallback. `script/bootstrap` no longer prints - `bootstrap complete` on a machine that cannot run the gate: a missing docker, - or one whose daemon is unreachable, is a hard failure naming exactly what - breaks. Verification was by reproduction rather than inspection — two - concurrent lints from two worktrees of differing cleanliness, a real run made - to report an outside path, a matching linter shimmed onto `PATH`, and a `PATH` - with docker removed — and is recorded on the pull request. -- 2026-08-09: Closed the fifth false-green mechanism (issues #93, #69). - `script/test` omitted `-count=1`, so Go's test result cache could satisfy the - gate outright: a second back-to-back `make test` printed the full set of 14 - `ok` lines, every one marked `(cached)`, having executed no test at all. Since - `ok (cached)` is an `ok` line, the "14 `ok` lines means the suite ran" - signal this repo leans on was forgeable, one level below the Docker layer - cache that #85 addressed. Fixed with `-count=1` unconditionally rather than - only in the container, because the pre-commit hook runs the same script and a - gate honest only in CI is dishonest where people rely on it most; - `test-coverage` got the same flag, and `script/check` inherits it by calling - `script/test`. In the same area, `make test-integration` was deleted rather - than made real: no file in the repo carried a build tag, so - `-tags=integration` selected nothing and the target was an exact duplicate of - `make test`. Tagging a subset was rejected because the entire suite runs in - well under a minute, and a scheme whose failure mode is "some tests silently - stopped running" is a poor trade for those seconds in a repo with this - particular history. The `-timeout` was raised from 30s after measuring rather - than after assuming: the standing claim that cold-cache compilation is charged - against `-timeout` is **false**, disproved by a containerised run that spent - 46s compiling and still reported per-package durations within noise of a warm - host run. `-timeout` reaches the test binary as `-test.timeout` and its clock - starts inside `testing.M.Run`, after the build. The real exposure was margin, - not compilation. The 120s landed on is a **deliberate, documented divergence** - from `REPO_POLICIES.md:192`, which mandates 30s, and from that file's - canonical recipe at `:212-214`; the divergence is recorded in `script/test`'s - comment because `REPO_POLICIES.md` is org-canonical and not editable here, and - issue #101 proposes amending the policy text upstream. Numbers and the full - verification are recorded once, on the pull request, and are deliberately not - restated here. - -- 2026-08-09: Triaged all fifteen stale remote branches (issue #71) and deleted - fourteen of them; the full per-branch disposition with evidence is recorded on - that issue. Method mattered more than the outcome here: a three-dot - `git diff main...branch` diffs from the merge base, so it replays everything - that landed on `main` after the branch diverged and makes any old branch look - like it holds unlanded work. That artifact is what made `golangci-v2.12.2` - appear to carry 126 files of unpushed changes when its tree was byte-identical - to `main`'s. Every containment claim here therefore rests on two-dot tip - diffs, tree-hash equality, `git cherry`, and `git branch -r --merged`. Nine - branches were plain ancestors of `main` with zero `git cherry` `+` commits. - `golangci-v2.12.2` had landed squashed as `cc58583`, whose tree hash equals - the branch tip's exactly; note the hash recorded in the issue had gone stale - because `main` advanced, so the check had to be redone rather than repeated. - `fix/sync-snapshot-cleanup` was redundant, its one line already on `main` in - `syncWithRemote`. `feature/restore-progress-bar` was superseded by - `printRestoreProgress` and the disk-backed blob cache, and had become actively - regressive — it would have deleted `internal/blobgen/compress_test.go`, the - #28 regression test that landed separately. The two branches this issue was - filed for both turned out to be closed questions that `main` had already moved - past by a recorded decision, so neither was landed and no regression test was - owed: `ctime` no longer exists anywhere in the codebase after `1c72a37` - removed the column, the `File.CTime` field and every use (#54/#55), and change - detection compares size, mtime, mode, uid and gid only, exactly as - `ARCHITECTURE.md` documents — so the silently-skipped-file data-loss risk that - made this a 1.0 item does not exist. The SQL allow-list branch would have - reverted `bfd7334`, which replaced that very allow-list with regex - sanitisation on review feedback, and would have broken - `getTableCount("snapshots")` because its allow-list omits that table. - `feature/daemon-mode` is untouched and deferred to #94 pending an owner - decision, so it is the one branch besides `main` still on the remote. The - stale `TODO.md` entry named in the issue needed no fix: `e496aa3` had already - removed it. No product code changed. -- 2026-08-09: Adopted the remaining upstream `CHECK_EPOCH` hardening (issue - #91), closing the gap #85 knowingly left open. Four changes, all four decided - as adopt upstream in `sneak/prompts` #26. (1) Each check stage now asserts - `[ -n "$CHECK_EPOCH" ] || exit 1` before running anything, so a build that - supplies no `--build-arg` fails instead of lying. This is the item that - mattered: an unset `ARG` is an empty string and an empty string is a stable - cache key, so the second and every later bare `docker build .` on an unchanged - tree replayed all three check layers and still exited 0 — and `docker build .` - is the command `REPO_POLICIES.md` names verbatim as a thing that must be - green, so the documented command was precisely the one that lied. Failed steps - are never cached, which is what makes the guard fire on every invocation - rather than once. (2) The epoch is now expanded into each check command rather - than left as a bare declaration, so the cache miss no longer depends on - BuildKit's unreferenced-`ARG` handling staying as it is, and the value appears - in the build log. (3) `script/cibuild` uses `epoch="$(date +%s%N)$$"`, unique - per invocation rather than per second; `%N` alone is insufficient because - busybox drops it silently and exits 0, and `$$` is what makes the guarantee - hold regardless. The bare-assignment form is kept deliberately — inlined in an - argument, a failing substitution does not abort under `set -eu` and would - yield an empty constant epoch, restoring the exact false green being fixed. - (4) `script/docker` passes the same fresh arg, so the two entrypoints cannot - disagree about whether the tree is green; local builds are almost always warm, - which made it the likelier fooling in practice. The `ARG` placement from #85 - is unchanged, below `apk add`, `COPY go.mod go.sum` and `go mod download`, so - dependency layers still cache and the build is not cold. Verified by negative - control rather than inspection — a bare `docker build .` run twice back to - back, plus back-to-back pairs of both scripts and a host-side `make check`; - the measurements are recorded once, in the PR verification comment, rather - than restated here. `.golangci.yml`, the lint-stage `FROM` line and its - digest, `script/lint`, `REPO_POLICIES.md` and `.gitea/workflows/check.yml` are - all untouched. -- 2026-08-09: Stopped `script/cibuild` from reporting a green it did not earn - (issue #85). A bare `docker build .` let Docker serve the check layers from - the layer cache whenever the tree had not changed: the checks never executed - and the build still exited 0. The fix is an `ARG CHECK_EPOCH` declared - immediately above the check `RUN`s in both the lint stage and the builder - stage (`ARG` scope is per-stage, so each declares its own), with - `script/cibuild` assigning `epoch="$(date +%s)"` and passing - `--build-arg CHECK_EPOCH="$epoch"`. The assignment is separate on purpose: - under `set -eu` a command substitution that fails inside an argument does not - abort the script, which would leave an empty constant `CHECK_EPOCH` and - restore the very false green being fixed. Placement is the rest of the point — - the `ARG` sits below the `apk add`, `COPY go.mod go.sum`, and - `go mod download` layers, so only the checks are invalidated and the - dependency layers still cache. The guarantee is conditional on a fresh value - rather than absolute: a bare `docker build .` gets an empty `CHECK_EPOCH` and - can still serve the check layers from cache, which `README.md` and the - `Dockerfile` now say plainly, with issue #91 tracking the upstream hardening - (expanded `ARG` form, unset guard, per-invocation epoch, `script/docker`) that - would close it. Verified by re-running the reproduction plus the - withheld-`--build-arg` counterfactual; the measurements are recorded once, in - the PR #89 verification comment, rather than restated here. `.golangci.yml`, - the lint-stage `FROM` line and its digest, `script/lint`, and - `.gitea/workflows/check.yml` are all untouched. -- 2026-08-09: Corrected the `Vaultik.UI` doc comment (issue #84). It claimed the - cli layer replaces the writer with a discarding one in `--cron` mode; the - actual mechanism is `UI.SetQuiet(true)` in `setupGlobals`, which drops - Begin/Complete/Info/Notice/Detail/ Progress/Banner but still emits Warning and - Error. The `--cron` line in `README.md` said "Silent unless error", which - understated what survives, and now names warnings too. The other `--cron` - comments (`internal/log/log.go`, `internal/cli/snapshot.go`, - `internal/vaultik/snapshot.go`) were audited and already accurate. Comments - and docs only, no behavior change. -- 2026-08-09: Made `snapshot list` list the destination store without the - private key (issue #64). The listing is now the union of the local index and a - single streamed listing of the `metadata/` prefix, with no `age_secret_key` - gate — the manifest is unencrypted, so a host holding only the public key can - enumerate its own backups and a host that lost its local index can still see - them. A remote-only snapshot's hostname and name are deliberately not - recovered (they are not recoverable without the private key, and making them - so would undo the privacy property tracked in issue #81); such rows are - labelled by an abbreviation of their remote key and carry the real timestamp - and compressed size from the manifest, with `` in the two columns - that require the local index. Local-only snapshots are reported as drift, and - the hint now names `vaultik prune`, which exists, instead of - `vaultik snapshot cleanup`, which does not. `reportRemoteDrift` collapsed into - the merged view. Every remote manifest read in the codebase now goes through - `downloadManifestByKey`, so issue #81 has one call site to change. Review - rework: snapshot timestamps now normalize to UTC in `scanSnapshotRows`, the - one place they enter the domain, so the merged TIMESTAMP column cannot show - local time for a locally tracked row and UTC for a remote-only row on a - non-UTC host; `GetIncompleteByHostname` was folded onto that same scanner. - `--json` now reports the unreadable-manifest count and the 1000-row truncation - on stderr instead of returning a silently short document (the document's shape - is unchanged). The two per-snapshot `log.Warn` calls on the listing path now - route through the same JSON-aware writer as the existing workaround, so one - corrupt manifest can no longer put a log line on stdout ahead of the document - and break `| jq` — still a local workaround pending issue #82. Verified with - `script/cibuild` and with an uncached `make check` (`0 issues.`, no cached - test packages), plus end to end against a `file://` destination with no secret - key present. -- 2026-08-09: Closed the gap between `make lint` and CI (issue #78). - `script/lint` now runs the digest-pinned `golangci-lint` image taken from the - `Dockerfile` lint stage, which is the single source of truth for the linter - version; the duplicate pin in the `Makefile` `deps` target and the unpinned - `golangci-lint` install in `script/bootstrap` are gone. A `golangci-lint` on - `PATH` is used only when its version is exactly the pinned one (which is how - the lint stage runs it inside the container); anything else goes through - Docker, and a missing or unreachable Docker daemon is a hard error rather than - a silent fallback. Only the **lint** leg of `make check` became equivalent to - `script/cibuild`; its tests and `gofmt` still run on the host against the host - toolchain, as `README.md` states. An earlier version of this entry claimed - `make check` was "as trustworthy as `script/cibuild`" outright, which - overstated it; corrected under issue #80. -- 2026-08-09: Finished the lint remediation under the canonical `.golangci.yml` - (issue #61, which also unblocks issue #59). The remaining findings were fixed - behavior-preservingly: `wsl_v5` whitespace, `sqlclosecheck`, and `prealloc`. - The `sqlclosecheck` sites now close `sql.Rows` in a deferred closure instead - of via the `CloseRows` helper, which the linter could not see through. Only - the `revive` package-name findings remain suppressed, with per-site `//nolint` - directives; the package-rename question behind them is tracked in issue #76. - Verified with `script/cibuild`, which exits 0 — that is the only trustworthy - gate, because `script/lint` runs whatever `golangci-lint` happens to be on - `PATH` rather than the pinned v2.12.2 that CI and the `Dockerfile` use, so - `make check` can report green on findings CI still fails. That tooling gap is - tracked in issue #78. -- 2026-08-09: The earlier next step "reconcile the uncommitted `ARCHITECTURE.md` - edits on `main`" needed no work: the working tree is clean and - `ARCHITECTURE.md` is committed on `main`. -- 2026-08-07: Updated golangci-lint to v2.12.2 everywhere it is pinned - (`Dockerfile` lint stage, `Makefile` deps target), replaced `.golangci.yml` - with the canonical config (v2 schema, `default: all`), and remediated the bulk - of the lint findings it surfaced (issue #61): behavior-preserving fixes across - every package, 2,990 findings down to 80. `make test` and `make fmt-check` - were green at that point but `make lint` was still red; the commit message - claiming `make check` was green was wrong. -- 2026-08-07: Added the standard `.golangci.yml` and `.editorconfig` (issue - #59); lint findings under the new config are tracked in issue #61. -- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile - shims, README Entrypoints section -- 2026-07-02: Consolidated CLI verbs, retired overlapping commands; bound the - local index to its backup destination URL. -- 2026-06-28: snapshot rm now removes metadata only and prints the prune - command; restore skips chown when running as non-root. -- 2026-06-26: Snapshot IDs hashed at the storage boundary; snapshot list made - resilient to bad remote entries. -- 2026-06-24: Collapsed snapshot prune into vaultik prune; restore streams blobs - to disk and restores files in blob-locality order; cron output fixes. -- 2026-06-17: Restore overhaul: ReadAt chunk reads from cached blobs, - reference-counted blob sweeper, integration tests; new internal/ui output - layer, banner, and progress lines. -- 2025-12-18: Added ARCHITECTURE.md and godoc coverage for exported API. -- 2025-07-26: End-to-end integration tests; manifest format refactor; renamed - backup to snapshot; afero filesystem abstraction. -- 2025-07-20: Initial design and implementation: cobra + fx CLI skeleton, SQLite - index database, UUID blob storage with streaming chunking. +None for an agent. Merging `next` into `main` is the owner's. # Future Steps