check / check (push) Failing after 3m23s
The documents, the example config and the help text are rebased onto the fourteen fixes on next. The sentences those fixes made false now match the code: an unknown config key, a relative index_path and a value that is not a size are refused when the config loads; a backup saves the destination in the local index only after it writes the marker file, and no other command saves it; a backup skips the local index, its side files and the lock file; a binary from go install reports its tag and counts as a release; a declined deletion prompt exits 1 with "deletion not confirmed"; the S3 keys are set before an s3:// storage_url. No document quoted the old messages for an unknown snapshot ID, a wrong private key or an unreadable blob. Model: fable-5-1
414 lines
16 KiB
Markdown
414 lines
16 KiB
Markdown
# vaultik architecture
|
|
|
|
This document describes how the `vaultik` code is organised. It covers three
|
|
things:
|
|
|
|
- the packages
|
|
- how they are wired together
|
|
- what happens during a backup, a restore, a verify and a prune
|
|
|
|
Two other documents cover the stored data:
|
|
|
|
- [`DATAMODEL.md`](DATAMODEL.md) describes the local index.
|
|
- [`REPOSTRUCTURE.md`](REPOSTRUCTURE.md) describes the layout of the
|
|
destination.
|
|
|
|
## Terms
|
|
|
|
- **Snapshot**: one backup of the paths listed under one name in the config.
|
|
- **Chunk**: a piece of a file's content. A chunk is named by the SHA-256 hash
|
|
of its content.
|
|
- **Blob**: one object at the destination. It holds many chunks, compressed and
|
|
then encrypted.
|
|
- **Local index**: the SQLite database on the backed-up machine.
|
|
- **Destination**: the place the backups are stored.
|
|
- **Manifest**: the unencrypted list of the blobs one snapshot uses.
|
|
- **Remote key**: a hash of the snapshot ID. It names the snapshot's directory
|
|
at the destination.
|
|
|
|
## How a backup moves data
|
|
|
|
```
|
|
source files
|
|
│
|
|
▼
|
|
Scanner walks the paths and finds the files that changed
|
|
│
|
|
▼
|
|
Chunker cuts each changed file into chunks
|
|
│
|
|
▼
|
|
Packer packs new chunks into a blob, compressed and encrypted
|
|
│
|
|
▼
|
|
Storer uploads the blob to the destination
|
|
```
|
|
|
|
## Packages
|
|
|
|
- `cmd/vaultik`: the `main` function. It starts optional profiling and calls
|
|
`cli.Entry`.
|
|
- `internal/cli`: the commands, their flags and their help text. It builds the
|
|
application, takes the lock and handles interrupts.
|
|
- `internal/vaultik`: the code behind each command. `Vaultik` holds every
|
|
dependency a command uses.
|
|
- `internal/snapshot`: the `Scanner`, which backs up one path, and the
|
|
`SnapshotManager`, which creates snapshot records and uploads their metadata.
|
|
- `internal/chunker`: cuts a stream into chunks with the FastCDC algorithm.
|
|
- `internal/blob`: the `Packer`, which collects chunks into blobs.
|
|
- `internal/blobgen`: the writer that hashes, compresses and encrypts a blob,
|
|
and the reader that reverses it.
|
|
- `internal/database`: the local index. It holds the schema and the repository
|
|
types that read and write the tables. [`DATAMODEL.md`](DATAMODEL.md#code) says
|
|
which type serves which table.
|
|
- `internal/storage`: the `Storer` interface and its three implementations, for
|
|
S3, a local directory and [rclone](https://rclone.org/).
|
|
- `internal/s3`: the S3 client that the S3 `Storer` uses.
|
|
- `internal/config`: loads and checks the config file.
|
|
- `internal/ui`: formats the status lines, warnings and errors a user sees.
|
|
- `internal/log`: the logger. It writes to stderr.
|
|
- `internal/pidlock`: the lock that `RunWithApp` takes for a command that
|
|
changes data, so that only one runs at a time.
|
|
- `internal/globals`: the version, commit and commit date, set when the binary
|
|
is built.
|
|
- `internal/types`: distinct types for IDs, hashes and paths.
|
|
|
|
## Wiring
|
|
|
|
The application is assembled with [uber-go/fx](https://github.com/uber-go/fx), a
|
|
dependency injection library. `cli.NewApp` lists the providers:
|
|
|
|
```go
|
|
fx.Supply(config.Path(opts.ConfigPath)),
|
|
fx.Supply(opts.LogOptions),
|
|
fx.Provide(globals.New),
|
|
fx.Provide(log.New),
|
|
config.Module,
|
|
database.Module,
|
|
log.Module,
|
|
storage.Module,
|
|
snapshot.Module,
|
|
fx.Provide(vaultik.New),
|
|
fx.Invoke(setupGlobals),
|
|
fx.NopLogger,
|
|
```
|
|
|
|
Each module provides these:
|
|
|
|
- `config.Module` provides `*config.Config`, from `config.Load`.
|
|
- `database.Module` provides `*database.DB` and `*database.Repositories`. It
|
|
creates the index file with mode `0600` when it does not exist.
|
|
- `storage.Module` provides the `storage.Storer` that the config selects.
|
|
- `snapshot.Module` provides the `snapshot.ScannerFactory` and the
|
|
`*snapshot.SnapshotManager`.
|
|
- `vaultik.New` provides `*vaultik.Vaultik`.
|
|
|
|
A command runs through `cli.RunOperation`. It works like this:
|
|
|
|
- `RunWithApp` takes the lock when the command changes data, and then starts the
|
|
application.
|
|
- The command's function runs in its own goroutine.
|
|
- When the function returns, it asks the application to shut down.
|
|
- On `SIGINT` or `SIGTERM`, the stop hook cancels the `Vaultik` context. It then
|
|
waits for the function to return, so that its temporary files are removed.
|
|
- The wait ends after 30 seconds at most.
|
|
|
|
These commands do not build the application:
|
|
|
|
- the `config` commands
|
|
- `database delete`
|
|
- `completion`
|
|
- `version`
|
|
|
|
## Backup
|
|
|
|
`Vaultik.CreateSnapshot` runs a backup. It does these steps in order:
|
|
|
|
- `markDestination` checks that the local index is not bound to another
|
|
destination. It then checks the marker file, writes it, and binds the local
|
|
index to the configured destination.
|
|
- `PruneDatabase` removes unfinished snapshots and unused records from the local
|
|
index.
|
|
- `createNamedSnapshot` runs once for each snapshot name.
|
|
- With `--prune`, `runPostBackupPrune` removes old snapshots and unused data.
|
|
|
|
`CreateSnapshot` calls `skipPostBackupPrune` instead of `runPostBackupPrune` in
|
|
these cases:
|
|
|
|
- `--skip-errors` skipped a file or directory that could not be read.
|
|
- A configured path held nothing.
|
|
- An earlier snapshot of the same name found a mount point that this backup did
|
|
not find.
|
|
|
|
`PruneDatabase` must succeed before anything is scanned. An unfinished snapshot
|
|
left in the index would make the backup skip files that were never fully stored.
|
|
|
|
### One snapshot
|
|
|
|
`createNamedSnapshot` does these steps:
|
|
|
|
- `resolveSnapshotPaths` makes each configured path absolute and resolves
|
|
symlinks.
|
|
- `SnapshotManager.CreateSnapshotWithName` creates the snapshot record. The
|
|
record is not yet marked complete.
|
|
- `Scanner.Scan` backs up each path in turn.
|
|
- `finalizeSnapshotMetadata` uploads the snapshot's metadata and marks the
|
|
snapshot complete.
|
|
|
|
A new `Scanner` is created for each snapshot, from `ScannerFactory`. It owns a
|
|
`Chunker` and a `Packer`.
|
|
|
|
### Scanning one path
|
|
|
|
`Scanner.Scan` works in these steps:
|
|
|
|
- `repairInterruptedBlobs` deletes the records of blobs whose upload never
|
|
finished. Their data is then stored again.
|
|
- `loadDatabaseState` loads the known files and chunks into memory.
|
|
- `scanPhase` walks the path. It sorts each regular file and symlink into
|
|
changed or unchanged.
|
|
- `detectDeletedFiles` counts the files that the previous snapshot of this name
|
|
had and that are now gone.
|
|
- `batchAddFilesToSnapshot` records the unchanged files in the new snapshot.
|
|
- `recordDirectories` records every directory, and the mount points found.
|
|
- `processPhase` reads each changed regular file and stores it. It records each
|
|
changed symlink.
|
|
|
|
The walk handles each kind of entry differently:
|
|
|
|
- A regular file or a symlink is compared with its record in the index.
|
|
- A directory is recorded with its metadata. It has no content to store.
|
|
- A device node, a named pipe or a socket is skipped.
|
|
- An entry that matches an exclude pattern is skipped. For a directory, so is
|
|
everything in it.
|
|
- The local index, its SQLite side files and the lock file `vaultik.pid` are
|
|
skipped, whatever the exclude patterns say.
|
|
|
|
### Deduplication
|
|
|
|
Data is stored once at two levels.
|
|
|
|
- **Files.** A file is not read again when its record in the index still matches
|
|
it.
|
|
- **Chunks.** A chunk is named by the SHA-256 hash of its content. A chunk that
|
|
an uploaded blob already holds, according to the index, is not stored again.
|
|
|
|
A file's record matches when all of these are unchanged:
|
|
|
|
- the size
|
|
- the modification time
|
|
- the mode
|
|
- the owner and group
|
|
- the link target
|
|
|
|
One case overrides the file check. A file is read again when it lists a chunk
|
|
that no uploaded blob holds. This happens when the only snapshot that used the
|
|
blob holding that chunk is removed. `loadFilesToRechunk` finds these files.
|
|
|
|
### Chunks
|
|
|
|
`chunker.Chunker` cuts a file with the FastCDC algorithm. The cut points depend
|
|
on the content, so an insertion early in a file moves only nearby chunks.
|
|
|
|
`chunk_size` sets the average chunk size. No chunk is larger than four times
|
|
this size. Only a file's last chunk can be smaller than a quarter of it. With
|
|
the default of 10 MB, those limits are 40 MB and 2.5 MB.
|
|
|
|
`ChunkReaderStreaming` passes each chunk to a callback as it is found. The
|
|
chunker buffers at most twice the largest chunk size.
|
|
|
|
### Blobs
|
|
|
|
`blob.Packer` collects new chunks into a blob. For each blob it does this:
|
|
|
|
- It gives the blob a UUID and creates a temporary file.
|
|
- It adds each chunk's bytes to a running SHA-256 hash.
|
|
- In the same pass it compresses the bytes with
|
|
[zstd](https://github.com/facebook/zstd) and encrypts them with
|
|
[age](https://age-encryption.org/). The result goes to the temporary file.
|
|
- It records each chunk's offset and length within the blob.
|
|
- It closes the blob when the next chunk would pass `blob_size_limit`, and at
|
|
the end of each path.
|
|
|
|
The blob's name is `hex(SHA256(SHA256(uncompressed contents)))`. It is computed
|
|
from the chunk data before compression and encryption.
|
|
[`REPOSTRUCTURE.md`](REPOSTRUCTURE.md#blobs) says what follows from that.
|
|
|
|
When a blob is closed, `Scanner.handleBlobReady` stores it:
|
|
|
|
- It looks the blob up at the destination. A blob that is already there is not
|
|
uploaded again.
|
|
- Otherwise it uploads the temporary file to
|
|
`blobs/<hash[0:2]>/<hash[2:4]>/<hash>`.
|
|
- It records the upload in the index and links the blob to the snapshot.
|
|
- It deletes the temporary file.
|
|
|
|
A file's record is written to the index only after every blob that holds its
|
|
chunks is uploaded.
|
|
|
|
### Snapshot metadata
|
|
|
|
`finalizeSnapshotMetadata` does these steps in order:
|
|
|
|
- `PopulateSnapshotBlobs` records every blob the snapshot uses.
|
|
- `UpdateSnapshotStatsExtended` stores the snapshot's totals.
|
|
- `ExportSnapshotMetadata` uploads the metadata.
|
|
- `MarkSnapshotComplete` marks the snapshot complete.
|
|
|
|
Completion is recorded last on purpose. A backup that stops during the upload
|
|
leaves the snapshot unfinished, and the next backup removes it and backs up its
|
|
files again.
|
|
|
|
`ExportSnapshotMetadata` works on a copy of the index, never on the index
|
|
itself:
|
|
|
|
- It copies the index into a temporary directory with `VACUUM INTO`.
|
|
- It deletes the rows of every other snapshot from the copy. It also deletes the
|
|
files and blobs that this snapshot does not use. It then deletes each chunk
|
|
that no remaining file and no remaining blob uses.
|
|
- It keeps the `local_meta` and `schema_migrations` tables as they are. The
|
|
uploaded database therefore names the destination, and lists the migrations
|
|
that have run.
|
|
- It runs `VACUUM`, so that no deleted row is left in the file.
|
|
- It compresses the copy with zstd and encrypts it with age.
|
|
- It builds the manifest from the copy.
|
|
- It uploads `metadata/<remote-key>/db.zst.age`, and then
|
|
`metadata/<remote-key>/manifest.json.zst`.
|
|
|
|
The manifest is uploaded last. A directory under `metadata/` with no manifest
|
|
therefore marks a backup that did not finish.
|
|
|
|
## Restore
|
|
|
|
`Vaultik.Restore` does these steps:
|
|
|
|
- `restoreIdentities` parses the age private key. A missing or malformed key
|
|
stops the restore before anything is downloaded.
|
|
- `downloadSnapshotDB` downloads and decrypts the snapshot's database into a
|
|
temporary directory.
|
|
- `getFilesToRestore` selects the files, from the paths given.
|
|
- `restoreAllFiles` downloads blobs and writes the files.
|
|
- With `--verify`, `verifyRestoredFiles` reads each restored regular file and
|
|
checks its chunk hashes.
|
|
|
|
Given a snapshot ID, the restore does not read the local index. Given a remote
|
|
key, it reads the local index only to check the destination marker. It does not
|
|
read the manifest. It finds each file's chunks and blobs through the downloaded
|
|
database.
|
|
|
|
### Checks on the downloaded database
|
|
|
|
Decryption shows that a database is readable. It does not show that the database
|
|
is honest. The restore therefore checks it:
|
|
|
|
- `database.OpenReadOnly` opens the file read-only. It refuses a file that has a
|
|
trigger, a view or a virtual table, or that lacks an expected table.
|
|
- `verifySnapshotDBIdentity` checks that the snapshot in the database hashes to
|
|
the remote key that was asked for.
|
|
- `containedRestorePath` refuses a stored path that would leave the target
|
|
directory. It also refuses to write through a symlink below the target.
|
|
- A blob is not decompressed beyond the size that the database gives for it.
|
|
- Each downloaded blob is hashed. The hash must match the blob's name.
|
|
|
|
### Order of work
|
|
|
|
`restorePlan` orders the files so that few blobs are on disk at once:
|
|
|
|
- Downloaded blobs are kept decrypted in a cache directory.
|
|
- A file is restored once every blob it needs is in the cache.
|
|
- When no file is ready, the restore downloads the blobs for the file that needs
|
|
the fewest.
|
|
- `restoreSweeper` deletes a cached blob once no remaining file needs it.
|
|
|
|
A regular file is created with mode `0600` and gets its stored mode after its
|
|
content is written. A directory gets its stored mode and modification time after
|
|
the whole restore, and its owner when the command runs as root.
|
|
|
|
## Verify
|
|
|
|
Without `--deep`, `Vaultik.VerifySnapshotWithOptions` checks a snapshot without
|
|
the private key:
|
|
|
|
- It downloads the manifest.
|
|
- It checks that the snapshot's encrypted database is present.
|
|
- It checks that each listed blob is present with the listed size.
|
|
|
|
`Vaultik.RunDeepVerify` needs the private key. It does these steps:
|
|
|
|
- It downloads the manifest and the database. It checks the database as a
|
|
restore does.
|
|
- `verifyManifestAgainstDatabase` checks that both list the same blobs with the
|
|
same sizes.
|
|
- `verifyBlobExistenceFromDB` checks that every blob is present with the
|
|
recorded size.
|
|
- `verifyBlob` downloads each blob, checks the hash of each chunk in it, and
|
|
checks the hash that names the blob.
|
|
|
|
## Prune
|
|
|
|
`Vaultik.Prune` calls `checkStorageBinding` and then does three steps:
|
|
|
|
- `CleanupLocalSnapshots` drops from the local index each snapshot whose
|
|
manifest is gone from the destination.
|
|
- `PruneDatabase` removes unfinished snapshots and unused records from the local
|
|
index.
|
|
- `PruneBlobs` deletes unused data at the destination.
|
|
|
|
`PruneBlobs` reads every manifest at the destination. It then deletes two kinds
|
|
of object, once they are older than `prune_grace_period`:
|
|
|
|
- a blob that no manifest lists
|
|
- a directory under `metadata/` that has no manifest, and whose snapshot is not
|
|
in the local index
|
|
|
|
It deletes nothing when any manifest cannot be read.
|
|
[`REPOSTRUCTURE.md`](REPOSTRUCTURE.md#pruning) explains why this is safe, and
|
|
where it is not.
|
|
|
|
## The destination
|
|
|
|
`storage.Storer` is the only way the code reaches the destination. It has these
|
|
methods:
|
|
|
|
- `Put` and `PutWithProgress` store an object.
|
|
- `Get` reads an object.
|
|
- `Stat` reports an object's size.
|
|
- `Delete` removes an object.
|
|
- `List` and `ListStream` list the objects under a prefix.
|
|
- `DeletePartialUploads` removes what an interrupted upload left behind.
|
|
- `Info` describes the destination.
|
|
|
|
`storage.NewStorer` picks the implementation from `storage_url`:
|
|
|
|
- `s3://` gives the S3 implementation.
|
|
- `file://` gives the local directory implementation.
|
|
- `rclone://` gives the rclone implementation.
|
|
|
|
With no `storage_url`, it builds the S3 implementation from the `s3` settings.
|
|
|
|
## The marker file and the destination binding
|
|
|
|
These checks keep the local index and the destination in step.
|
|
|
|
- `checkDestinationMarker` looks for `vaultik-destination.txt`. It fails when
|
|
the file is reported missing while the local index holds snapshots. It skips
|
|
the check when the local index belongs to another destination.
|
|
- `checkStorageBinding` fails when the local index is bound to a different
|
|
destination from the configured one. It records nothing.
|
|
- `EnsureStorageBinding` binds an unbound local index to the configured
|
|
destination. Only `markDestination` calls it, after the marker file is
|
|
written, so a backup that writes nothing leaves the local index unbound.
|
|
|
|
Every command that takes the lock calls `checkStorageBinding` first.
|
|
|
|
## Concurrency
|
|
|
|
- Each open database uses one connection, so its queries run one at a time.
|
|
- The index runs in SQLite's write-ahead log (WAL) mode. A second process can
|
|
therefore read it while a backup writes to it.
|
|
- `Packer` guards its state with a mutex.
|
|
- `Scanner` holds `packerMu` while a chunk is added to the packer and while a
|
|
blob is closed.
|
|
- `Repositories.WithTx` runs a function in one transaction. It commits when the
|
|
function succeeds and rolls back when it fails.
|