vaultik/ARCHITECTURE.md

# vaultik architecture

This document describes how the `vaultik` code is organised. It covers three
things:

- the packages
- how they are wired together
- what happens during a backup, a restore, a verify and a prune

Two other documents cover the stored data:

- [`docs/DATAMODEL.md`](docs/DATAMODEL.md) describes the local index.
- [`docs/REPOSTRUCTURE.md`](docs/REPOSTRUCTURE.md) describes the layout of the
  destination.

## Terms

- **Snapshot**: one backup of the paths listed under one name in the config.
- **Chunk**: a piece of a file's content. A chunk is named by the SHA-256 hash
  of its content.
- **Blob**: one object at the destination. It holds many chunks, compressed and
  then encrypted.
- **Local index**: the SQLite database on the backed-up machine.
- **Destination**: the place the backups are stored.
- **Manifest**: the unencrypted list of the blobs one snapshot uses.
- **Remote key**: a hash of the snapshot ID. It names the snapshot's directory
  at the destination.

## How a backup moves data

```
source files
     │
     ▼
  Scanner    walks the paths and finds the files that changed
     │
     ▼
  Chunker    cuts each changed file into chunks
     │
     ▼
  Packer     packs new chunks into a blob, compressed and encrypted
     │
     ▼
  Storer     uploads the blob to the destination
```

## Packages

- `cmd/vaultik`: the `main` function. It starts optional profiling and calls
  `cli.Entry`.
- `internal/cli`: the commands, their flags and their help text. It builds the
  application, takes the lock and handles interrupts.
- `internal/vaultik`: the code behind each command. `Vaultik` holds every
  dependency a command uses.
- `internal/snapshot`: the `Scanner`, which backs up one path, and the
  `SnapshotManager`, which creates snapshot records and uploads their metadata.
- `internal/chunker`: cuts a stream into chunks with the FastCDC algorithm.
- `internal/blob`: the `Packer`, which collects chunks into blobs.
- `internal/blobgen`: the writer that hashes, compresses and encrypts a blob,
  and the reader that reverses it.
- `internal/database`: the local index. It holds the schema and the repository
  types that read and write the tables.
  [`docs/DATAMODEL.md`](docs/DATAMODEL.md#code) says which type serves which
  table.
- `internal/storage`: the `Storer` interface and its three implementations, for
  S3, a local directory and [rclone](https://rclone.org/).
- `internal/s3`: the S3 client that the S3 `Storer` uses.
- `internal/config`: loads and checks the config file.
- `internal/ui`: formats the status lines, warnings and errors a user sees.
- `internal/log`: the logger. It writes to stderr.
- `internal/pidlock`: the lock that `RunWithApp` takes for a command that
  changes data, so that only one runs at a time.
- `internal/globals`: the version, commit and commit date, set when the binary
  is built.
- `internal/types`: distinct types for IDs, hashes and paths.

## Wiring

The application is assembled with [uber-go/fx](https://github.com/uber-go/fx), a
dependency injection library. `cli.NewApp` lists the providers:

```go
fx.Supply(config.Path(opts.ConfigPath)),
fx.Supply(opts.LogOptions),
fx.Provide(globals.New),
fx.Provide(log.New),
config.Module,
database.Module,
log.Module,
storage.Module,
snapshot.Module,
fx.Provide(vaultik.New),
fx.Invoke(setupGlobals),
fx.NopLogger,
```

Each module provides these:

- `config.Module` provides `*config.Config`, from `config.Load`.
- `database.Module` provides `*database.DB` and `*database.Repositories`. It
  creates the index file with mode `0600` when it does not exist.
- `storage.Module` provides the `storage.Storer` that the config selects.
- `snapshot.Module` provides the `snapshot.ScannerFactory` and the
  `*snapshot.SnapshotManager`.
- `vaultik.New` provides `*vaultik.Vaultik`.

A command runs through `cli.RunOperation`. It works like this:

- `RunWithApp` takes the lock when the command changes data, and then starts the
  application.
- The command's function runs in its own goroutine.
- When the function returns, it asks the application to shut down.
- On `SIGINT` or `SIGTERM`, the stop hook cancels the `Vaultik` context. It then
  waits for the function to return, so that its temporary files are removed.
- The wait ends after 30 seconds at most.

These commands do not build the application:

- the `config` commands
- `database delete`
- `completion`
- `version`

## Backup

`Vaultik.CreateSnapshot` runs a backup. It does these steps in order:

- `EnsureStorageBinding` checks that the local index belongs to the configured
  destination.
- `markDestination` checks the marker file and then writes it.
- `PruneDatabase` removes unfinished snapshots and unused records from the
  local index.
- `createNamedSnapshot` runs once for each snapshot name.
- With `--prune`, `runPostBackupPrune` removes old snapshots and unused data.

`CreateSnapshot` calls `skipPostBackupPrune` instead of `runPostBackupPrune` in
these cases:

- `--skip-errors` skipped a file or directory that could not be read.
- A configured path held nothing.
- An earlier snapshot of the same name found a mount point that this backup did
  not find.

`PruneDatabase` must succeed before anything is scanned. An unfinished snapshot
left in the index would make the backup skip files that were never fully
stored.

### One snapshot

`createNamedSnapshot` does these steps:

- `resolveSnapshotPaths` makes each configured path absolute and resolves
  symlinks.
- `SnapshotManager.CreateSnapshotWithName` creates the snapshot record. The
  record is not yet marked complete.
- `Scanner.Scan` backs up each path in turn.
- `finalizeSnapshotMetadata` uploads the snapshot's metadata and marks the
  snapshot complete.

A new `Scanner` is created for each snapshot, from `ScannerFactory`. It owns a
`Chunker` and a `Packer`.

### Scanning one path

`Scanner.Scan` works in these steps:

- `repairInterruptedBlobs` deletes the records of blobs whose upload never
  finished. Their data is then stored again.
- `loadDatabaseState` loads the known files and chunks into memory.
- `scanPhase` walks the path. It sorts each regular file and symlink into
  changed or unchanged.
- `detectDeletedFiles` counts the files that the previous snapshot of this name
  had and that are now gone.
- `batchAddFilesToSnapshot` records the unchanged files in the new snapshot.
- `recordDirectories` records every directory, and the mount points found.
- `processPhase` reads each changed regular file and stores it. It records each
  changed symlink.

The walk handles each kind of entry differently:

- A regular file or a symlink is compared with its record in the index.
- A directory is recorded with its metadata. It has no content to store.
- A device node, a named pipe or a socket is skipped.
- An entry that matches an exclude pattern is skipped. For a directory, so is
  everything in it.

### Deduplication

Data is stored once at two levels.

- **Files.** A file is not read again when its record in the index still matches
  it.
- **Chunks.** A chunk is named by the SHA-256 hash of its content. A chunk that
  an uploaded blob already holds, according to the index, is not stored again.

A file's record matches when all of these are unchanged:

- the size
- the modification time
- the mode
- the owner and group
- the link target

One case overrides the file check. A file is read again when it lists a chunk
that no uploaded blob holds. This happens when the only snapshot that used the
blob holding that chunk is removed. `loadFilesToRechunk` finds these files.

### Chunks

`chunker.Chunker` cuts a file with the FastCDC algorithm. The cut points depend
on the content, so an insertion early in a file moves only nearby chunks.

`chunk_size` sets the average chunk size. No chunk is larger than four times
this size. Only a file's last chunk can be smaller than a quarter of it. With
the default of 10 MB, those limits are 40 MB and 2.5 MB.

`ChunkReaderStreaming` passes each chunk to a callback as it is found. The
chunker buffers at most twice the largest chunk size.

### Blobs

`blob.Packer` collects new chunks into a blob. For each blob it does this:

- It gives the blob a UUID and creates a temporary file.
- It adds each chunk's bytes to a running SHA-256 hash.
- In the same pass it compresses the bytes with
  [zstd](https://github.com/facebook/zstd) and encrypts them with
  [age](https://age-encryption.org/). The result goes to the temporary file.
- It records each chunk's offset and length within the blob.
- It closes the blob when the next chunk would pass `blob_size_limit`, and at
  the end of each path.

The blob's name is `hex(SHA256(SHA256(uncompressed contents)))`. It is computed
from the chunk data before compression and encryption.
[`docs/REPOSTRUCTURE.md`](docs/REPOSTRUCTURE.md#blobs) says what follows from
that.

When a blob is closed, `Scanner.handleBlobReady` stores it:

- It looks the blob up at the destination. A blob that is already there is not
  uploaded again.
- Otherwise it uploads the temporary file to
  `blobs/<hash[0:2]>/<hash[2:4]>/<hash>`.
- It records the upload in the index and links the blob to the snapshot.
- It deletes the temporary file.

A file's record is written to the index only after every blob that holds its
chunks is uploaded.

### Snapshot metadata

`finalizeSnapshotMetadata` does these steps in order:

- `PopulateSnapshotBlobs` records every blob the snapshot uses.
- `UpdateSnapshotStatsExtended` stores the snapshot's totals.
- `ExportSnapshotMetadata` uploads the metadata.
- `MarkSnapshotComplete` marks the snapshot complete.

Completion is recorded last on purpose. A backup that stops during the upload
leaves the snapshot unfinished, and the next backup removes it and backs up its
files again.

`ExportSnapshotMetadata` works on a copy of the index, never on the index
itself:

- It copies the index into a temporary directory with `VACUUM INTO`.
- It deletes the rows of every other snapshot from the copy. It also deletes the
  files and blobs that this snapshot does not use. It then deletes each chunk
  that no remaining file and no remaining blob uses.
- It keeps the `local_meta` and `schema_migrations` tables as they are. The
  uploaded database therefore names the destination, and lists the migrations
  that have run.
- It runs `VACUUM`, so that no deleted row is left in the file.
- It compresses the copy with zstd and encrypts it with age.
- It builds the manifest from the copy.
- It uploads `metadata/<remote-key>/db.zst.age`, and then
  `metadata/<remote-key>/manifest.json.zst`.

The manifest is uploaded last. A directory under `metadata/` with no manifest
therefore marks a backup that did not finish.

## Restore

`Vaultik.Restore` does these steps:

- `restoreIdentities` parses the age private key. A missing or malformed key
  stops the restore before anything is downloaded.
- `downloadSnapshotDB` downloads and decrypts the snapshot's database into a
  temporary directory.
- `getFilesToRestore` selects the files, from the paths given.
- `restoreAllFiles` downloads blobs and writes the files.
- With `--verify`, `verifyRestoredFiles` reads each restored regular file and
  checks its chunk hashes.

Given a snapshot ID, the restore does not read the local index. Given a remote
key, it reads the local index only to check the destination marker. It does not
read the manifest. It finds each file's chunks and blobs through the downloaded
database.

### Checks on the downloaded database

Decryption shows that a database is readable. It does not show that the
database is honest. The restore therefore checks it:

- `database.OpenReadOnly` opens the file read-only. It refuses a file that has a
  trigger, a view or a virtual table, or that lacks an expected table.
- `verifySnapshotDBIdentity` checks that the snapshot in the database hashes to
  the remote key that was asked for.
- `containedRestorePath` refuses a stored path that would leave the target
  directory. It also refuses to write through a symlink below the target.
- A blob is not decompressed beyond the size that the database gives for it.
- Each downloaded blob is hashed. The hash must match the blob's name.

### Order of work

`restorePlan` orders the files so that few blobs are on disk at once:

- Downloaded blobs are kept decrypted in a cache directory.
- A file is restored once every blob it needs is in the cache.
- When no file is ready, the restore downloads the blobs for the file that
  needs the fewest.
- `restoreSweeper` deletes a cached blob once no remaining file needs it.

A regular file is created with mode `0600` and gets its stored mode after its
content is written. A directory gets its stored mode and modification time after
the whole restore, and its owner when the command runs as root.

## Verify

Without `--deep`, `Vaultik.VerifySnapshotWithOptions` checks a snapshot without
the private key:

- It downloads the manifest.
- It checks that the snapshot's encrypted database is present.
- It checks that each listed blob is present with the listed size.

`Vaultik.RunDeepVerify` needs the private key. It does these steps:

- It downloads the manifest and the database. It checks the database as a
  restore does.
- `verifyManifestAgainstDatabase` checks that both list the same blobs with the
  same sizes.
- `verifyBlobExistenceFromDB` checks that every blob is present with the
  recorded size.
- `verifyBlob` downloads each blob, checks the hash of each chunk in it, and
  checks the hash that names the blob.

## Prune

`Vaultik.Prune` calls `EnsureStorageBinding` and then does three steps:

- `CleanupLocalSnapshots` drops from the local index each snapshot whose
  manifest is gone from the destination.
- `PruneDatabase` removes unfinished snapshots and unused records from the
  local index.
- `PruneBlobs` deletes unused data at the destination.

`PruneBlobs` reads every manifest at the destination. It then deletes two kinds
of object, once they are older than `prune_grace_period`:

- a blob that no manifest lists
- a directory under `metadata/` that has no manifest, and whose snapshot is not
  in the local index

It deletes nothing when any manifest cannot be read.
[`docs/REPOSTRUCTURE.md`](docs/REPOSTRUCTURE.md#pruning) explains why this is
safe, and where it is not.

## The destination

`storage.Storer` is the only way the code reaches the destination. It has these
methods:

- `Put` and `PutWithProgress` store an object.
- `Get` reads an object.
- `Stat` reports an object's size.
- `Delete` removes an object.
- `List` and `ListStream` list the objects under a prefix.
- `DeletePartialUploads` removes what an interrupted upload left behind.
- `Info` describes the destination.

`storage.NewStorer` picks the implementation from `storage_url`:

- `s3://` gives the S3 implementation.
- `file://` gives the local directory implementation.
- `rclone://` gives the rclone implementation.

With no `storage_url`, it builds the S3 implementation from the `s3` settings.

## The marker file and the destination binding

Two checks keep the local index and the destination in step.

- `checkDestinationMarker` looks for `vaultik-destination.txt`. It fails when
  the file is reported missing while the local index holds snapshots. It skips
  the check when the local index belongs to another destination.
- `EnsureStorageBinding` records the configured destination in the local index
  on first use. Later it fails when the config names a different destination.

Every command that takes the lock calls `EnsureStorageBinding` first.

## Concurrency

- Each open database uses one connection, so its queries run one at a time.
- The index runs in SQLite's write-ahead log (WAL) mode. A second process can
  therefore read it while a backup writes to it.
- `Packer` guards its state with a mutex.
- `Scanner` holds `packerMu` while a chunk is added to the packer and while a
  blob is closed.
- `Repositories.WithTx` runs a function in one transaction. It commits when the
  function succeeds and rolls back when it fails.