vaultik/ARCHITECTURE.md

# vaultik architecture

This document describes how the `vaultik` code is organised. It covers three
things:

- the packages
- how they are wired together
- what happens during a backup, a restore, a verify and a prune

Two other documents cover the stored data:

- [docs/DATAMODEL.md](docs/DATAMODEL.md) describes the local index.
- [docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md) describes the layout of the
  destination.

## Terms

- **Snapshot**: one backup of the paths listed under one name in the config.
- **Chunk**: a piece of a file's content. A chunk is named by the SHA-256 hash
  of its content.
- **Blob**: one object at the destination. It holds many chunks, compressed and
  then encrypted.
- **Local index**: the SQLite database on the backed-up machine.
- **Destination**: the place the backups are stored.
- **Manifest**: the unencrypted list of the blobs one snapshot uses.
- **Remote key**: a hash of the snapshot ID. It names the snapshot's directory
  at the destination.

## How a backup moves data

```
source files
     │
     ▼
  Scanner    walks the paths and finds the files that changed
     │
     ▼
  Chunker    cuts each changed file into chunks
     │
     ▼
  Packer     packs new chunks into a blob, compressed and encrypted
     │
     ▼
  Storer     uploads the blob to the destination
```

## Packages

- `cmd/vaultik`: the `main` function. It starts optional profiling and calls
  `cli.Entry`.
- `internal/cli`: the commands, their flags and their help text. It builds the
  application, takes the lock and handles interrupts.
- `internal/vaultik`: the code behind each command. `Vaultik` holds every
  dependency a command uses.
- `internal/snapshot`: the `Scanner`, which backs up one path, and the
  `SnapshotManager`, which creates snapshot records and uploads their metadata.
- `internal/chunker`: cuts a stream into chunks with the FastCDC algorithm.
- `internal/blob`: the `Packer`, which collects chunks into blobs.
- `internal/blobgen`: the writer that hashes, compresses and encrypts a blob,
  and the reader that reverses it.
- `internal/database`: the local index. It holds the schema and one repository
  type for each table.
- `internal/storage`: the `Storer` interface and its three implementations, for
  S3, a local directory and rclone.
- `internal/s3`: the S3 client that the S3 `Storer` uses.
- `internal/config`: loads and checks the config file.
- `internal/ui`: formats the status lines, warnings and errors a user sees.
- `internal/log`: the logger. It writes to stderr.
- `internal/pidlock`: the lock that keeps two data-changing commands from
  running at once.
- `internal/globals`: the version, commit and build date, set when the binary
  is built.
- `internal/types`: distinct types for IDs, hashes and paths.

## Wiring

The application is assembled with [uber-go/fx](https://github.com/uber-go/fx), a
dependency injection library. `cli.NewApp` lists the providers:

```go
fx.Supply(config.Path(opts.ConfigPath)),
fx.Supply(opts.LogOptions),
fx.Provide(globals.New),
fx.Provide(log.New),
config.Module,
database.Module,
log.Module,
storage.Module,
snapshot.Module,
fx.Provide(vaultik.New),
fx.Invoke(setupGlobals),
fx.NopLogger,
```

Each module provides these:

- `config.Module` provides `*config.Config`, from `config.Load`.
- `database.Module` provides `*database.DB` and `*database.Repositories`. It
  creates the index file with mode `0600` when it does not exist.
- `storage.Module` provides the `storage.Storer` that the config selects.
- `snapshot.Module` provides the `snapshot.ScannerFactory` and the
  `*snapshot.SnapshotManager`.
- `vaultik.New` provides `*vaultik.Vaultik`.

A command runs through `cli.RunOperation`. It works like this:

- `RunWithApp` takes the lock when the command changes data, and then starts the
  application.
- The command's function runs in its own goroutine.
- When the function returns, it asks the application to shut down.
- On `SIGINT` or `SIGTERM`, the stop hook cancels the `Vaultik` context. It then
  waits for the function to return, so that its temporary files are removed.
- The wait ends after 30 seconds at most.

The `config` commands, `database delete`, `completion` and `version` do not
build the application.

## Backup

`Vaultik.CreateSnapshot` runs a backup. It does these steps in order:

- `EnsureStorageBinding` checks that the local index belongs to the configured
  destination.
- `markDestination` checks the marker file and then writes it.
- `PruneDatabase` removes unfinished snapshots and unused records from the
  local index.
- `createNamedSnapshot` runs once for each snapshot name.
- With `--prune`, `runPostBackupPrune` removes old snapshots and unused data.

`PruneDatabase` must succeed before anything is scanned. An unfinished snapshot
left in the index would make the backup skip files that were never fully
stored.

### One snapshot

`createNamedSnapshot` does these steps:

- `resolveSnapshotPaths` makes each configured path absolute and resolves
  symlinks.
- `SnapshotManager.CreateSnapshotWithName` creates the snapshot record. The
  record is not yet marked complete.
- `Scanner.Scan` backs up each path in turn.
- `finalizeSnapshotMetadata` uploads the snapshot's metadata and marks the
  snapshot complete.

A new `Scanner` is created for each snapshot, from `ScannerFactory`. It owns a
`Chunker` and a `Packer`.

### Scanning one path

`Scanner.Scan` works in these steps:

- `repairInterruptedBlobs` deletes the records of blobs whose upload never
  finished. Their data is then stored again.
- `loadDatabaseState` loads the known files and chunks into memory.
- `scanPhase` walks the path. It sorts each entry into changed or unchanged.
- `detectDeletedFiles` counts the files that the previous snapshot of this name
  had and that are now gone.
- `batchAddFilesToSnapshot` records the unchanged files in the new snapshot.
- `recordDirectories` records every directory, and the mount points found.
- `processPhase` reads each changed file and stores it.

The walk handles each kind of entry differently:

- A regular file or a symlink is compared with its record in the index.
- A directory is recorded with its metadata. It has no content to store.
- A device node, a named pipe or a socket is skipped.
- An entry that matches an exclude pattern is skipped. For a directory, so is
  everything in it.

### Deduplication

Data is stored once at two levels.

- **Files.** A file is not read again when its record in the index still matches
  it.
- **Chunks.** A chunk is named by the SHA-256 hash of its content. A chunk whose
  hash is already in the index is not stored again.

A file's record matches when all of these are unchanged:

- the size
- the modification time
- the mode
- the owner and group
- the link target

One case overrides the file check. A file is read again when it lists a chunk
that no uploaded blob holds. This happens after the only snapshot that used
such a blob is removed. `loadFilesToRechunk` finds these files.

### Chunks

`chunker.Chunker` cuts a file with the FastCDC algorithm. The cut points depend
on the content, so an insertion early in a file moves only nearby chunks.

`chunk_size` sets the average chunk size. A chunk is between a quarter of that
size and four times that size. With the default of 10 MB, a chunk is between 2.5
MB and 40 MB.

`ChunkReaderStreaming` passes each chunk to a callback as it is found. The whole
file is never held in memory.

### Blobs

`blob.Packer` collects new chunks into a blob. For each blob it does this:

- It gives the blob a UUID and creates a temporary file.
- It adds each chunk's bytes to a running SHA-256 hash.
- In the same pass it compresses the bytes with zstd and encrypts them with
  age. The result goes to the temporary file.
- It records each chunk's offset and length within the blob.
- It closes the blob when the next chunk would pass `blob_size_limit`, and at
  the end of each path.

The blob's name is `hex(SHA256(SHA256(uncompressed contents)))`. It is computed
from the chunk data before compression and encryption.
[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#blobs) says what follows from
that.

When a blob is closed, `Scanner.handleBlobReady` stores it:

- It looks the blob up at the destination. A blob that is already there is not
  uploaded again.
- Otherwise it uploads the temporary file to
  `blobs/<hash[0:2]>/<hash[2:4]>/<hash>`.
- It records the upload in the index and links the blob to the snapshot.
- It deletes the temporary file.

A file's record is written to the index only after every blob that holds its
chunks is uploaded.

### Snapshot metadata

`finalizeSnapshotMetadata` does these steps in order:

- `PopulateSnapshotBlobs` records every blob the snapshot uses.
- `UpdateSnapshotStatsExtended` stores the snapshot's totals.
- `ExportSnapshotMetadata` uploads the metadata.
- `MarkSnapshotComplete` marks the snapshot complete.

Completion is recorded last on purpose. A backup that stops during the upload
leaves the snapshot unfinished, and the next backup removes it and stores its
data again.

`ExportSnapshotMetadata` works on a copy of the index, never on the index
itself:

- It copies the index into a temporary directory with `VACUUM INTO`.
- It deletes from the copy everything that does not belong to this snapshot.
- It runs `VACUUM`, so that no deleted row is left in the file.
- It compresses the copy with zstd and encrypts it with age.
- It builds the manifest from the copy.
- It uploads `metadata/<remote-key>/db.zst.age`, and then
  `metadata/<remote-key>/manifest.json.zst`.

The manifest is uploaded last. A directory under `metadata/` with no manifest
therefore marks a backup that did not finish.

## Restore

`Vaultik.Restore` does these steps:

- `restoreIdentities` parses the age private key. A missing or malformed key
  stops the restore before anything is downloaded.
- `downloadSnapshotDB` downloads and decrypts the snapshot's database into a
  temporary directory.
- `getFilesToRestore` selects the files, from the paths given.
- `restoreAllFiles` downloads blobs and writes the files.
- With `--verify`, `verifyRestoredFiles` reads each restored file and checks its
  chunk hashes.

The restore does not read the local index or the manifest. It finds each file's
chunks and blobs through the downloaded database.

### Checks on the downloaded database

Decryption shows that a database is readable. It does not show that the
database is honest. The restore therefore checks it:

- `database.OpenReadOnly` opens the file read-only. It refuses a file that has a
  trigger, a view or a virtual table, or that lacks an expected table.
- `verifySnapshotDBIdentity` checks that the snapshot in the database hashes to
  the remote key that was asked for.
- `containedRestorePath` refuses a stored path that would leave the target
  directory. It also refuses to write through a symlink below the target.
- A blob is not decompressed beyond the size the database records for it.
- Each downloaded blob is hashed. The hash must match the blob's name.

### Order of work

`restorePlan` orders the files so that few blobs are on disk at once:

- Downloaded blobs are kept decrypted in a cache directory.
- A file is restored once every blob it needs is in the cache.
- When no file is ready, the restore downloads the blobs for the file that
  needs the fewest.
- `restoreSweeper` deletes a cached blob once no remaining file needs it.

A regular file is created with mode `0600` and gets its stored mode after its
content is written. A directory gets its stored mode, owner and modification
time after the whole restore.

## Verify

`Vaultik.VerifySnapshotWithOptions` checks a snapshot without the private key:

- It downloads the manifest.
- It checks that the snapshot's encrypted database is present.
- It checks that each listed blob is present with the listed size.

`Vaultik.RunDeepVerify` needs the private key. It does these steps:

- It downloads the manifest and the database. It checks the database as a
  restore does.
- `verifyManifestAgainstDatabase` checks that both list the same blobs with the
  same sizes.
- `verifyBlobExistenceFromDB` checks that every blob is present with the
  recorded size.
- `verifyBlob` downloads each blob, checks the hash of each chunk in it, and
  checks the hash that names the blob.

## Prune

`Vaultik.Prune` does three steps:

- `CleanupLocalSnapshots` drops each snapshot from the local index whose
  directory is gone from the destination.
- `PruneDatabase` removes unfinished snapshots and unused records from the
  local index.
- `PruneBlobs` deletes unused data at the destination.

`PruneBlobs` reads every manifest at the destination. It then deletes two kinds
of object, once they are older than `prune_grace_period`:

- a blob that no manifest lists
- a directory under `metadata/` that has no manifest, and whose snapshot is not
  in the local index

It deletes nothing when any manifest cannot be read.
[docs/REPOSTRUCTURE.md](docs/REPOSTRUCTURE.md#pruning) explains why this is
safe, and where it is not.

## The destination

`storage.Storer` is the only way the code reaches the destination. It has these
methods:

- `Put` and `PutWithProgress` store an object.
- `Get` reads an object.
- `Stat` reports an object's size.
- `Delete` removes an object.
- `List` and `ListStream` list the objects under a prefix.
- `DeletePartialUploads` removes what an interrupted upload left behind.
- `Info` describes the destination.

`storage.NewStorer` picks the implementation from `storage_url`:

- `s3://` gives the S3 implementation.
- `file://` gives the local directory implementation.
- `rclone://` gives the rclone implementation.

With no `storage_url`, it builds the S3 implementation from the `s3` settings.

## The marker file and the destination binding

Two checks keep the local index and the destination in step.

- `checkDestinationMarker` looks for `vaultik-destination.txt`. It fails when
  the file is reported missing while the local index holds snapshots.
- `EnsureStorageBinding` records the configured destination in the local index
  on first use. Later it fails when the config names a different destination.

Every command that changes data calls `EnsureStorageBinding` first.

## Concurrency

- The local index is opened with one connection, so all access to it is in
  turn.
- The index runs in SQLite's WAL mode. A second process can therefore read it
  while a backup writes to it.
- `Packer` guards its state with a mutex.
- `Scanner` holds `packerMu` while a chunk is added to the packer and while a
  blob is closed.
- `Repositories.WithTx` runs a function in one transaction. It commits when the
  function succeeds and rolls back when it fails.