Read all five documents against the code and fix what the sixth review found
check / check (push) Successful in 12m5s
check / check (push) Successful in 12m5s
Fixes the four findings of the sixth review: `snapshot remove` with `--local-only`, the two comma-chained permission lists, six undefined terms, and the sentence about an unfinished snapshot in the local index. Then reads every sentence of `README.md`, `ARCHITECTURE.md`, `docs/DATAMODEL.md`, `docs/REPOSTRUCTURE.md` and the comments in `config.example.yml` against the code for four faults: a command's needs or behaviour stated without regard to a flag, a false absolute, three or more things chained with commas, and a technical word not defined on first use. Each sentence changed is listed on the pull request. The help string of `snapshot remove --dry-run` now says it needs no permission to delete. Model: fable-5-1
This commit is contained in:
1 parent
a133b8838e
commit
46b2a9fc78
6 files changed
+341
-193
No files matched your search
+51
-28
@@ -1,7 +1,8 @@
|
||||
# vaultik data model
|
||||
|
||||
`vaultik` keeps a SQLite database on the backed-up machine, called the local
|
||||
index. This document describes its tables and how each command uses them.
|
||||
index. This document describes its tables, and how a backup, a restore and
|
||||
`prune` use them.
|
||||
|
||||
The local index has two jobs:
|
||||
|
||||
@@ -27,10 +28,11 @@ with the snapshot.
|
||||
|
||||
## Where the index is
|
||||
|
||||
The index is the file `index.sqlite` in the platform data directory. That is
|
||||
`~/.local/share/vaultik/` on Linux and `~/Library/Application Support/vaultik/`
|
||||
on macOS. The `index_path` setting and the `VAULTIK_INDEX_PATH` environment
|
||||
variable change the path.
|
||||
The index is the file `index.sqlite` in the platform data directory. By default
|
||||
that is `~/.local/share/vaultik/` on Linux and
|
||||
`~/Library/Application Support/vaultik/` on macOS. The `index_path` setting and
|
||||
the `VAULTIK_INDEX_PATH` environment variable change the path. So does
|
||||
`XDG_DATA_HOME`.
|
||||
|
||||
The file is readable only by its owner, because it lists every backed-up path.
|
||||
`vaultik` sets that mode each time it opens the index.
|
||||
@@ -74,15 +76,15 @@ One row for each backed-up path: a regular file, a symlink or a directory.
|
||||
|
||||
- `id`: a UUID. Primary key.
|
||||
- `path`: the absolute path. Unique.
|
||||
- `source_path`: the configured path that the scan found this entry under. A
|
||||
restore does not read it.
|
||||
- `source_path`: the configured path under which the scan found this entry,
|
||||
made absolute with symlinks resolved. A restore does not read it.
|
||||
- `mtime`: the modification time.
|
||||
- `mtime_nsec`: the nanoseconds within that second.
|
||||
- `size`: the size in bytes.
|
||||
- `mode`: the permission bits and the file type.
|
||||
- `uid`: the owner's user ID.
|
||||
- `gid`: the owner's group ID.
|
||||
- `link_target`: the target of a symlink. It is `NULL` for other entries.
|
||||
- `link_target`: the target of a symlink. It is empty for other entries.
|
||||
|
||||
`idx_files_path` indexes `path`.
|
||||
|
||||
@@ -161,7 +163,8 @@ A restore finds the blob for each chunk through this table.
|
||||
One row for each snapshot.
|
||||
|
||||
- `id`: the snapshot ID. Primary key.
|
||||
- `hostname`: the hostname of the machine that made the backup.
|
||||
- `hostname`: the `hostname` setting of the config. When that setting is empty,
|
||||
it is the machine's hostname.
|
||||
- `vaultik_version`: the `vaultik` version that made it.
|
||||
- `vaultik_git_revision`: the git commit of that version.
|
||||
- `started_at`: when the backup started.
|
||||
@@ -189,8 +192,8 @@ example is `server1_home_2025-06-01T12:00:00Z`. It has three parts:
|
||||
- the snapshot name
|
||||
- the start time, in UTC
|
||||
|
||||
`snapshot list` shows `uncompressed_chunk_size` as `UNCOMPRESSED SIZE` and
|
||||
`new_chunk_size` as `NEW CHUNK SIZE`.
|
||||
The `snapshot list` table shows `uncompressed_chunk_size` as
|
||||
`UNCOMPRESSED SIZE` and `new_chunk_size` as `NEW CHUNK SIZE`.
|
||||
|
||||
A snapshot with `completed_at` set to `NULL` is unfinished. `snapshot list`
|
||||
does not show it. The next backup deletes it, and so does `prune`.
|
||||
@@ -233,10 +236,10 @@ A directory is recorded here when all of these hold:
|
||||
- The backup found an entry in it that is not excluded.
|
||||
- It is on a different filesystem from its parent directory.
|
||||
|
||||
The device number is read after the directory has been listed. Listing an
|
||||
automount point mounts its volume, so the number is that of the volume. An
|
||||
automount point whose volume cannot be mounted is not recorded, because the
|
||||
backup finds no entry in it.
|
||||
The device number is read after the directory has been listed. An automount
|
||||
point is a directory whose volume is mounted when it is first listed, so the
|
||||
number is that of the volume. An automount point whose volume cannot be mounted
|
||||
is not recorded, because the backup finds no entry in it.
|
||||
|
||||
`snapshot create --prune` reads this table. It skips the prune when an earlier
|
||||
snapshot of the same name recorded a mount point that the new backup did not
|
||||
@@ -266,8 +269,14 @@ Settings that belong to this copy of the index.
|
||||
- `value`: its value.
|
||||
|
||||
The only key is `storage_url`. It holds the destination that the index was
|
||||
first used with. A command that changes data refuses to run when the config
|
||||
names a different destination.
|
||||
first used with. These commands refuse to run when the config names a different
|
||||
destination:
|
||||
|
||||
- `snapshot create`
|
||||
- `snapshot purge`
|
||||
- `snapshot remove`
|
||||
- `prune`
|
||||
- `remote nuke`
|
||||
|
||||
### `schema_migrations`
|
||||
|
||||
@@ -282,13 +291,16 @@ The migrations that have run.
|
||||
|
||||
A backup works on the tables in this order:
|
||||
|
||||
- It saves the destination in `local_meta` when the index has none yet.
|
||||
- It deletes unfinished snapshots, and the rows that no snapshot uses.
|
||||
- It creates a row in `snapshots`, with `completed_at` set to `NULL`.
|
||||
- It deletes the `blobs` rows that were never uploaded.
|
||||
- It walks each path and compares every entry with its row in `files`.
|
||||
- It walks each path. It compares each regular file and symlink with its row in
|
||||
`files`.
|
||||
- It links each unchanged file to the snapshot in `snapshot_files`.
|
||||
- It records each mount point in `snapshot_mount_points`.
|
||||
- It reads each changed file and stores it. See below.
|
||||
- It reads each changed regular file and stores it. See below. A changed symlink
|
||||
is recorded without being read.
|
||||
- It fills `snapshot_blobs` with every blob the snapshot uses.
|
||||
- It stores the snapshot's totals in `snapshots`.
|
||||
- It uploads the snapshot's metadata.
|
||||
@@ -303,7 +315,8 @@ in any of these:
|
||||
- owner or group
|
||||
- link target
|
||||
|
||||
A file is also read again when it lists a chunk that no uploaded blob holds.
|
||||
A file is also read again when it lists a chunk that no uploaded blob holds. A
|
||||
file with no row in `files` is read too.
|
||||
|
||||
Storing a changed file writes these rows:
|
||||
|
||||
@@ -316,8 +329,10 @@ Storing a changed file writes these rows:
|
||||
- `uploaded_ts` and `compressed_size` in `blobs`
|
||||
- a row in `snapshot_blobs`
|
||||
- a row in `uploads`, unless the blob was already at the destination
|
||||
- The row in `files`, with its `file_chunks`, `chunk_files` and
|
||||
`snapshot_files` rows.
|
||||
- The row in `files`, and its rows in these tables:
|
||||
- `file_chunks`
|
||||
- `chunk_files`
|
||||
- `snapshot_files`
|
||||
|
||||
The row in `files` is written last. It is written only after every blob that
|
||||
holds the file's new chunks is stored.
|
||||
@@ -334,31 +349,38 @@ The metadata of a snapshot is made from a copy of the index:
|
||||
uploaded database therefore names the destination, and lists the migrations
|
||||
that have run.
|
||||
- `VACUUM` rewrites the copy, so that no deleted row is left in the file.
|
||||
- The copy is compressed with zstd and encrypted with age.
|
||||
- The copy is compressed with [zstd](https://github.com/facebook/zstd) and
|
||||
encrypted with [age](https://age-encryption.org/).
|
||||
- The manifest is built from the copy's `snapshot_blobs` table.
|
||||
- The copy is uploaded as `metadata/<remote-key>/db.zst.age`.
|
||||
- The manifest is uploaded as `metadata/<remote-key>/manifest.json.zst`.
|
||||
|
||||
The copy is made through SQLite, not by copying the file. The index runs in WAL
|
||||
mode, so rows that are already committed can still be in the `-wal` file beside
|
||||
it.
|
||||
The copy is made through SQLite, not by copying the file. The index runs in
|
||||
SQLite's write-ahead log (WAL) mode, so rows that are already committed can
|
||||
still be in the `-wal` file beside it.
|
||||
|
||||
### Restore
|
||||
|
||||
A restore does not use the local index.
|
||||
A restore reads no file, chunk or blob rows from the local index. Given a
|
||||
remote key, it reads `snapshots` and `local_meta` there to check the destination
|
||||
marker.
|
||||
|
||||
- It downloads and decrypts `db.zst.age` into a temporary directory.
|
||||
- It opens that file read-only. It applies no migration to it.
|
||||
- It reads the snapshot's row in `snapshots`, to check that this is the right
|
||||
database.
|
||||
- It reads the files to restore from `files`.
|
||||
- It finds each file's chunks in `file_chunks`.
|
||||
- It finds the blob and offset of each chunk in `blob_chunks`.
|
||||
- It finds the name of each blob in `blobs`.
|
||||
- With `--verify`, it reads each chunk's size from `chunks`.
|
||||
|
||||
It does not read the manifest.
|
||||
|
||||
### Prune
|
||||
|
||||
`prune` changes the local index in two ways:
|
||||
`prune` first saves the destination in `local_meta` when the index has none yet.
|
||||
It then changes the local index in two ways:
|
||||
|
||||
- It deletes each snapshot whose manifest is gone from the destination.
|
||||
- It deletes unfinished snapshots, and the rows that no snapshot uses.
|
||||
@@ -389,6 +411,7 @@ Each repository type in `internal/database` serves one or more tables:
|
||||
- `snapshot_files`
|
||||
- `snapshot_blobs`
|
||||
- `snapshot_mount_points`
|
||||
- `uploads`, where it deletes a snapshot's rows
|
||||
- `UploadRepository`: `uploads`
|
||||
- `LocalMetaRepository`: `local_meta`
|
||||
|
||||
|
||||
Reference in new issue
Block a user