README.md: config set keeps comments and can remove blank lines; a
config that does not load yet also has a value that is not a size
refused; vaultik version reports a tag and adds a development-build
note to every other version. docs/REPOSTRUCTURE.md and README.md:
someone who has a blob's whole contents can compute its name and see
whether it is stored. docs/DATAMODEL.md: snapshot list shows an
unfinished snapshot as remote only once its manifest is at the
destination; closing a full blob commits the first chunk of the next
one. internal/cli/config.go: config set --help says the same about
comments and blank lines.
Model: fable-5-1
The snapshot restore help splits its 40-word permissions sentence in
two. The snapshot-ID case leaves the parenthesis, and the temporary
space requirement gets its own sentence.
configs/config.example.yml gains a commented-out env entry, after
hostname, so its first line, which says the file shows every option,
is true again.
docs/REPOSTRUCTURE.md defines remote key in Terms, before the
marker-file section uses it.
Model: fable-5-1
- snapshot verify and snapshot restore also list the destination for a
snapshot ID that is not in the local index: README, REPOSTRUCTURE,
ARCHITECTURE, DATAMODEL and the two commands' help.
- prune on a separate machine deletes only unused data and does not
remove old snapshots.
- The --quiet usage string names everything the flag leaves out.
- A warning written as a log record stays on stderr.
- config set on a config that does not load refuses three things.
- No test is held back by a build tag.
- A plain docker build . adds no -dirty.
- The backup's order of work records directories.
- TODO.md says what parallel downloads would speed up and names
restorePlan and restoreSweeper.
Model: fable-5-1
- `README.md`, `docs/REPOSTRUCTURE.md`, `internal/cli/snapshot.go`: a
backup needs only write permission on S3; an rclone remote that can
move objects also needs read and delete, and an upload there stops
when its look-up is refused.
- `docs/DATAMODEL.md`, `docs/REPOSTRUCTURE.md`: a restore reads no
file, chunk or blob rows from the local index, but it does open it.
- `README.md`: confirmation prompts go to stdout.
- `internal/cli/remote.go`: `remote nuke` leaves the backup destination
store empty, not a bucket prefix.
- `README.md`: a TODO section, as `REPO_POLICIES.md` requires.
Model: fable-5-1
Fixes the four findings of the sixth review: `snapshot remove` with
`--local-only`, the two comma-chained permission lists, six undefined terms,
and the sentence about an unfinished snapshot in the local index.
Then reads every sentence of `README.md`, `ARCHITECTURE.md`,
`docs/DATAMODEL.md`, `docs/REPOSTRUCTURE.md` and the comments in
`config.example.yml` against the code for four faults: a command's needs or
behaviour stated without regard to a flag, a false absolute, three or more
things chained with commas, and a technical word not defined on first use.
Each sentence changed is listed on the pull request.
The help string of `snapshot remove --dry-run` now says it needs no
permission to delete.
Model: fable-5-1
A search of the four documents, the help text and config.example.yml
for the review's faults found these.
- prune help, and snapshot purge and snapshot list in README.md: a
snapshot is dropped or reported when its manifest is gone, not its
metadata or directory.
- README.md, version: the date printed is the commit date.
- README.md, Configuration: names the commands that start without a
config that loads.
- README.md, snapshot purge: "each such snapshot" named nothing.
- README.md and docs/DATAMODEL.md: a backup can upload unchanged
content again, so "uploads only what changed" was false.
- docs/REPOSTRUCTURE.md: db.zst.age holds more than one snapshot's
rows.
- ARCHITECTURE.md: "database records" read as a noun.
Model: fable-5-1
snapshot list says why a remote-only snapshot is listed by its remote
key. snapshot verify says what access to the destination it needs. The
note on the age-keygen file is two sentences in snapshot verify and
snapshot restore. remote nuke says "backup destination store".
config.example.yml names the commands that run without a snapshot.
Only help strings and comments change.
Model: fable-5-1
The rewrite removed the README's roadmap, which TODO.md linked to. The
roadmap items are now under Future Steps in TODO.md, in place of the
link.
A comment in internal/storage/s3_test.go cited the README for the
destination layout. That layout is in docs/REPOSTRUCTURE.md.
Model: fable-5-1
The first sentence now names the licence, the kind of program, the
language and the author. The README gains a Design section, and its
sections carry the names REPO_POLICIES.md asks for: Rationale and
Getting started, which now holds the install command. Entrypoints opens
with the Scripts to Rule Them All statement.
Corrected: when a file is read again, the chunk size limits, and the
first prune step, which looks for a snapshot's manifest. Two sentences
that misparsed are reworded.
A test comment that named the README quickstart follows the rename.
Model: fable-5-1
snapshot verify --help now says --deep needs the age private key, from
VAULTIK_AGE_SECRET_KEY or age_secret_key, as snapshot restore does.
Both now name the config key as well as the environment variable.
The help of snapshot create, list, purge, remove, prune, remote info
and remote nuke says which access to the destination it needs.
config.example.yml gains age_secret_key, and no longer tells the reader
to pipe age-keygen into grep, which discards the private key.
Only help strings and comments change.
Model: fable-5-1
A failed command printed its error twice on stderr: as an ERROR log record and as the final ERROR: line the README documents. The command handlers, a restore failure that ends the command and a failed blob upload now log that error at info level or not at all, so without --verbose or --debug only the final line appears.
Model: opus-5-5
config set storage_url to an s3:// URL was refused, without saying why, when s3.access_key_id and s3.secret_access_key were not yet set. The refusal now names both keys and says to set them first. config set still refuses any change that leaves the config unable to load, and clearing a key gets the plain refusal.
Model: opus-5-5
A non-numeric chunk_size or blob_size_limit failed with an error that named neither the setting nor the value. Every size setting now fails at load with its name and the quoted value, for example chunk_size: "banana" is not a size; use a number of bytes or a unit such as 10MB. A value too large for a size says so, with the setting name.
Model: opus-5-5
A relative index_path, or VAULTIK_INDEX_PATH, was accepted and resolved against whatever directory vaultik ran in. Loading the config now refuses a relative or empty value, quoting it, with the same wording as the file:// check; ~ is still expanded first.
Model: opus-5-5
The default config backs up the home directory, which holds vaultik's local index and PID file, so a backup stored a torn copy of the index it was writing. The scan now skips the index, its SQLite side files and the PID file whatever the config says, including when index_path or its directory is a symlink. database purge removes the same list of index files.
Model: opus-5-5
A binary from go install, the README's first command, reported itself as version dev, a development build, although Go had recorded the tag it was built from. When the linker flags leave the version, commit or commit date unset, they are now filled in from the Go build info, so such a binary reports its tag and counts as a release. Builds through make, script/docker and goreleaser are unchanged.
Model: opus-5-5
Pressing Ctrl-C or sending SIGTERM during a backup logged the interrupted blob upload as an ERROR before the interrupted message. The upload step now skips that record when the backup's context has been cancelled, whatever error the storage returned. An upload that fails for any other reason is still logged.
Model: opus-5-5
With a private key the backup was not encrypted to, restore and deep verify failed with age's "no identity matched any of the recipients" behind a doubled prefix. They now say in one sentence that the key from VAULTIK_AGE_SECRET_KEY or age_secret_key is not one of the keys the snapshot, or a blob, was encrypted to. A blob reused after a recipient change gets the same message.
Model: opus-5-5
A misspelled or unknown config key was accepted silently, so a typo in exclude backed up everything it meant to leave out. Loading now refuses a key that no setting has, at any depth, naming its dotted path and, when it is in the file, its line; keys brought in by ${...} references are checked too. config set refuses an unknown key. A top-level env: section stays accepted, since the config library uses it. Anchors that contain themselves fail with an error.
Model: opus-5-5
A blob that is not a valid age stream made restore print age's error, which quotes the blob's raw bytes, once for every file that needed it. The error now says the blob is not a valid age stream and names its hash, without quoting any bytes. Under --skip-errors a blob that failed is reported once, with the number of files it affected, and is not downloaded again for each file.
Model: opus-5-5
A first backup that failed before writing anything, on a missing bucket, a refused credential or missing recipients, still bound the local index to the configured destination, so correcting a mistyped storage_url was refused afterwards. A backup now records the binding only after it has written the marker file at the destination. prune, snapshot purge, snapshot remove and remote nuke only refuse a different binding and leave an unbound index unbound.
Model: opus-5-5
snapshot restore and snapshot verify given an ID that names no snapshot in the local index or at the destination failed with "object not found" on a hashed metadata path. They now look the ID up the way snapshot remove does and say that no such snapshot exists, before printing anything else.
Model: opus-5-5
prune, snapshot purge, snapshot remove and database delete exited 0 when their confirmation prompt was declined or left unanswered, so a script could not tell that nothing was deleted. They now end with "deletion not confirmed" and exit 1. The [y/N] prompts accept y or Y; database delete still asks for yes.
Model: opus-5-5
The internal/vaultik tests took 36 to 56 seconds under go test -race and ran past the 90-second timeout when the host was busy. Tests that built the same backup now copy one shared backup, and the interruption tests use smaller sources. No assertion is weakened and no test is removed. Measured back to back on next under the same load, the package takes 55% of its former time.
Model: opus-5-5
REPO_POLICIES.md names prettier as the Markdown formatter and keeps documents out of the root. script/fmt and script/fmt-check now run prettier over all Markdown (4-space indents, proseWrap always) besides gofmt, with node and yarn installed by script/bootstrap as in sneak/prompts. All Markdown is reformatted with no text change. ARCHITECTURE.md and TODO.md move to docs/, config.example.yml to configs/, and every reference follows.
Model: opus-5-5
vaultik version printed globals.CommitDate, the date of the commit the binary was built from, labelled "build date", and the startup banner called it "built on". Both now say "commit date". The test sets Commit and CommitDate to distinct values and checks the label, padding and value of every report line.
Model: opus-5-5
The status and progress lines of remote info and snapshot verify (shallow and deep) were written with stdoutf, so they had no marker and no color, unlike every other command's. They now go through Beginf, Completef and Progressf, as the README's output style section says. The reports stay plain, and --quiet and --json still drop these lines.
Model: opus-5-5
The default config backed up /Applications on every system. That directory exists only on macOS, so on Linux the first snapshot create stopped on the missing path. config init now writes the apps snapshot only when runtime.GOOS is darwin; the macOS file is unchanged.
Model: opus-5-5 (implementation, review); opus-5-5 (merge)
Two doc comments described behaviour that no longer exists. The comment on `ExportSnapshotMetadata` described an SQL dump uploaded under `snapshots/`; it now lists the steps the function performs and the two keys it writes. The comment on `EnsureStorageBinding` named `vaultik database purge`; the command is `vaultik database delete`. Comments only.
Model: opus-5-5
snapshot create --prune purged the older snapshots holding a volume's
files when the volume, mounted below a configured path, was not
mounted: its empty mount point read as deleted files.
The scan now records in the local index each directory at or below a
configured path that the walk found an entry in and that, once listed,
is on a different device from its parent; listing mounts an
automounted volume. Under --prune, when an earlier snapshot of that
name recorded such a directory and this run does not, the run skips
the prune, keeps every older snapshot and exits non-zero, naming the
directory. For a volume retired on purpose, snapshot purge removes the
older snapshots.
Model: opus-5-5
A backup that stopped between uploading db.zst.age and its manifest
left that directory under metadata/ on the destination for good:
prune deleted nothing there.
The blob step of prune, which snapshot create --prune also runs, now
deletes each directory under metadata/ that has no manifest.json.zst,
that no local snapshot record claims, and all of whose objects are
older than prune_grace_period; an object listed with no time keeps its
directory, and only names that are remote keys are deleted. The prompt
and the --force output name each such directory, the --json result
counts them (metadata_dirs_deleted, metadata_dirs_failed), and a failed
delete fails the run. Every description of the grace period now
covers these directories.
Model: opus-5-5
snapshot create --prune of a snapshot whose path was an empty
directory, as the mount point of a volume that is not mounted is,
recorded an empty snapshot and then purged the older snapshots that
held the data, with their blobs.
The scan now counts the entries it finds in each configured path.
Under --prune, a run in which a path of a snapshot it made held
nothing skips the prune, keeps every older snapshot, names each such
path with its snapshot, and exits non-zero, as
#382 does for unreadable
files. Only the current run is looked at. Without --prune this check
does nothing. A configured path that is a symlink whose target is
gone now fails every run, with or without --prune.
Model: opus-5-5
A source host whose credentials allow only PUT, which the documents
say is enough to back up, failed every backup after its first: the
marker check looks the marker up with an S3 HEAD, which those
credentials are refused, and any error other than not found was
fatal.
Such an error now means the check could not be made: it is logged at
info level, so --cron stays silent, and the command goes on. Not found
still stops it. The documents now say so, and that such a host does
not notice an emptied destination, so vaultik database delete must be
run on it after the destination is emptied. The fault-injecting test
store gains a Stat hook for the test.
Model: opus-5-5
A backup uploads its blobs before the manifest that lists them, and the
process lock covers one user on one machine, so a prune from another host
or user sharing the destination deleted a running backup's blobs.
Prune now deletes an unreferenced blob only once its listed time is older
than the new config key prune_grace_period (default 24h; a negative value
is refused). A newer blob, or one listed with no time, is kept and counted
(blobs_too_new). The rclone backend takes each object's time from the
listing (use_server_modtime).
Decision: a blob a backup reuses from an earlier run that stopped before
its manifest keeps its old time and is not protected; the docs and config
comments say so.
Model: opus-5-5
snapshot create --skip-errors --prune pruned even when files were
skipped, so by default it deleted the previous snapshot of that name,
the only one still holding a file that had become unreadable, along
with blobs only that snapshot referenced.
The prune is now skipped exactly when the run exits non-zero for
entries --skip-errors could not read or list: older snapshots are
kept, a warning says why, and the run still exits non-zero. A file
removed during the run does not count, so such a run still prunes.
The README's --prune text and the flag's help say so.
Model: opus-5-5
A backup under --skip-errors printed its skip errors, then "Finished
successfully.", and exited 0, so a cron wrapper could not tell it from
a complete backup.
Each skipped entry, including a directory the walk could not list, is
now counted as examined and failed. If --skip-errors skipped anything
it could not read or list, the run still keeps its snapshots and
prunes as before, then fails with "Backup completed, but N file(s)
could not be read". A file removed during the run, even before the
walk reaches it, counts once as failed and leaves the exit status
alone; without the flag it no longer stops the backup. Excluded
entries are never counted.
Model: opus-5-5
`remote nuke` took a destination store it could not list, or one
without `vaultik-destination.txt` while the local index held
snapshots, as a warning: it then cleared the local snapshot records,
and for an unplugged volume's empty mount point reported the store
empty and exited zero. Nuke now fails on either before it removes
anything, without its advice to run it again, which would fail the
same way. The bulk remove it runs fails on either too; only a
local-only remove still goes on with a warning. Nuke also deletes each
snapshot's remote metadata before its local record, so a failed delete
keeps the snapshot in the local index rather than leaving it on the
store as a remote-only snapshot.
Model: opus-5-5
Every backup queued every directory and symlink for the second phase.
The line ending the walk counted directories, and their stat sizes, as
files to back up while its enumerated count left them out, and a
symlink was counted as backed up on every run, its target length as
data.
The walk now sets directories aside: they are recorded before the files
are backed up and never counted as files or data, and the walk's
progress estimate leaves out the directories the local index knows. A
symlink is checked against its record like a regular file, its target
included, so an unchanged one counts as unchanged.
Judgement call: every directory is still written to the local index on
every backup, as before; no report shows it.
Model: opus-5-5
Every backup writes vaultik-destination.txt at the destination root.
While the local index holds snapshots stored there, a destination
without it is not present: snapshot create refuses before writing,
prune and snapshot purge fail, and snapshot list and snapshot remove
act as for a destination that cannot be listed. snapshot create and
snapshot remove check it first. remote nuke deletes the marker last.
The file:// backend never creates its destination directory, which
protects a directory on a removable volume, not a mount point.
Tests back up into directories that exist; fixtures for a written
destination carry the marker.
Judgement call: the marker is rewritten on every backup.
Deviation: for the complexity limit, two blocks moved, unchanged, into
snapshotNamesToCreate and confirmRemoveSnapshot.
Model: opus-5-5
A restore by a normal user always warned that file ownership was not
preserved, even when every entry came out with its recorded owner and
group. The owner and group each restored entry has on disk are now
compared with the recorded ones, and the warning is printed once if any
differ. The comparison is made on disk because on macOS, and in a setgid
directory on Linux, a new entry takes the group of its directory, not
the user's.
Judgement call: an entry that is not on disk after the restore is not compared.
Unverified: not run on a Mac; the directory-group test runs only as root.
Model: opus-5-5
When the local index is bound to a different destination than the
configured one, snapshot list now gives the warning prune, snapshot
purge and snapshot remove refuse with: both destinations and the two
ways out (revert the configuration, or run vaultik database delete).
Below the table it replaces the local-only note and its prune advice.
With --json it goes to stderr through the logger, so stdout stays the
document, whose local rows show remote_present false.
Reading the binding moves out of EnsureStorageBinding into
storageBinding, which records nothing, so the read-only list can call
it without binding a fresh index.
Model: opus-5-5
A backup counted as deleted every file in the local index under the
source path that the walk did not find: files the snapshot excludes
but another snapshot backs up, and files removed long ago, which every
later backup, and the first backup of another name over the same
path, reported as deleted again.
The backup now counts only files that the latest completed snapshot of
the same name lists, the walk did not find, and are gone from disk,
including a file whose directory became a regular file. Excluded files
and files the walk could not reach under --skip-errors are not
counted.
Only the backup's output changes; the snapshots row stores no deleted
figure.
Model: opus-5-5
A missing blob during restore, a failed shallow verify and an invalid
purge duration each printed their prefix twice. FetchBlob now returns
the storage error as is, since restore and deep verify name the blob
themselves; purge returns the duration parser's error unchanged; and a
failed shallow verify wraps "problems found in storage" instead of
"verification failed", which the command already prints. Deep verify
words a storage error other than not found as a blob it could not
check, and its log line no longer says S3.
Already fixed for #314: the
deep verify size message naming S3 and other errors called missing;
tests now check both.
Not tested: the log line.
Model: opus-5-5
storage_url refused only the outer URL's userinfo and unknown query
parameters. Userinfo inside the s3 endpoint= value, and rclone's
remote,param=value parameters in the remote name, were accepted and
then printed by info, remote info and snapshot create. Both are now
refused without echoing the value: a host name never contains "@", and
an rclone remote name never contains a comma. An s3 or rclone query
that cannot be decoded (a ";" or a bad "%" escape) is refused too,
because decoding drops that part before it can be checked. The comma
is checked before url.Parse, whose error for such a name prints part of
it. The rclone remote name case has its own error pointing at rclone
config.
Model: opus-5-5
config edit used the whole of $EDITOR as the program name, so a value
with arguments such as "code --wait" or "emacsclient -t" failed with
"executable file not found". It now runs $EDITOR the way git does:
sh -c '<editor> "$@"' with the config path as a separate argument, so
the editor's own arguments and quoting work and a path with spaces
reaches it whole. A test runs config edit with an $EDITOR carrying an
argument on a config path that contains a space.
Model: opus-5-5
When a snapshot was only on the destination store and deleting its
metadata there failed, snapshot remove warned and exited 0 having
removed nothing. That failure is now an error. A snapshot in the local
index keeps its behaviour: it is removed from the local index and the
failed delete on the destination store is a warning.
The README and the help text said the `vaultik prune` follow-up is
printed on every success; they now say it is printed only when metadata
was removed from the destination store, and never under --json. The
comments on RemoveSnapshot and removeSnapshotRemote describe both
cases; removeSnapshotRemote's also says it returns true only when the
delete finished without an error and removed at least one object.
Model: opus-5-5
The summary printed by the local index cleanup before each backup gave,
in brackets after each removed count, the number of snapshots, files,
chunks or blobs left in the local index, worded "(N remain)", which
read as incomplete or orphaned rows left over: removing the only
incomplete snapshot printed "1 removed (1 remain)". Each line now
labels that figure as the total of its kind left in the local index.
Model: opus-5-5
A restore of selected paths, or one where files failed under
--skip-errors, kept the blobs those files share in the temporary cache
until it ended: the sweeper freed a blob only once every file in the
snapshot database that uses it had been restored, and files outside the
selection or that failed never are.
The sweeper now asks the restore plan, through pendingFileUsesBlob,
whether any pending file still uses a cached blob. The plan holds only
this run's files and drops each one once it is restored or has failed.
The new test backs up 16 KiB files so it stays quick; the locality
backup helper takes the file size for that.
Model: opus-5-5
A startup error was cut at its last "): " to drop the annotations fx
adds, so the blob_size_limit message, which itself contains
"(... size spread): ", lost everything but "need at least N bytes".
cleanStartupError now uses dig.RootCause, from the library fx is built
on, which removes exactly those annotations and returns the error
underneath unchanged.
The test starts a real fx app on a config whose blob_size_limit is below
the minimum. The old cases fed it plain strings that only looked like fx
output; such an error is now rightly left whole, so they are replaced.
go.uber.org/dig, already required through fx, becomes a direct
requirement in go.mod.
Model: opus-5-5
PruneBlobs, CleanupLocalSnapshots, RemoveSnapshot, RemoveAllSnapshots
and snapshot purge wrote their status lines straight to stdout, so
`snapshot create --cron --prune` printed on every successful run and
--quiet did not silence `prune`, `snapshot purge`, `snapshot remove` or
`remote nuke`. Those lines now go through internal/ui, which drops them
under --cron and --quiet; warnings, errors and --json documents still
print.
The confirmation prompts still write straight to stdout, as the
`database delete` prompt does, together with what they ask to delete:
the snapshot list before the purge prompt, and the blob count and size
before the prune prompt. With --force those lines are status lines.
The two purge test fixtures gained the UI writer production always sets.
Model: opus-5-5
clawbot
clawbot
sneak