vaultik/ARCHITECTURE.md

vaultik architecture

This document describes how the vaultik code is organised. It covers three things:

• the packages
• how they are wired together
• what happens during a backup, a restore, a verify and a prune

Two other documents cover the stored data:

• docs/DATAMODEL.md describes the local index.
• docs/REPOSTRUCTURE.md describes the layout of the destination.

Terms

• Snapshot: one backup of the paths listed under one name in the config.
• Chunk: a piece of a file's content. A chunk is named by the SHA-256 hash of its content.
• Blob: one object at the destination. It holds many chunks, compressed and then encrypted.
• Local index: the SQLite database on the backed-up machine.
• Destination: the place the backups are stored.
• Manifest: the unencrypted list of the blobs one snapshot uses.
• Remote key: a hash of the snapshot ID. It names the snapshot's directory at the destination.

How a backup moves data

source files
     │
     ▼
  Scanner    walks the paths and finds the files that changed
     │
     ▼
  Chunker    cuts each changed file into chunks
     │
     ▼
  Packer     packs new chunks into a blob, compressed and encrypted
     │
     ▼
  Storer     uploads the blob to the destination

Packages

• cmd/vaultik: the main function. It starts optional profiling and calls cli.Entry.
• internal/cli: the commands, their flags and their help text. It builds the application, takes the lock and handles interrupts.
• internal/vaultik: the code behind each command. Vaultik holds every dependency a command uses.
• internal/snapshot: the Scanner, which backs up one path, and the SnapshotManager, which creates snapshot records and uploads their metadata.
• internal/chunker: cuts a stream into chunks with the FastCDC algorithm.
• internal/blob: the Packer, which collects chunks into blobs.
• internal/blobgen: the writer that hashes, compresses and encrypts a blob, and the reader that reverses it.
• internal/database: the local index. It holds the schema and the repository types that read and write the tables. docs/DATAMODEL.md says which type serves which table.
• internal/storage: the Storer interface and its three implementations, for S3, a local directory and rclone.
• internal/s3: the S3 client that the S3 Storer uses.
• internal/config: loads and checks the config file.
• internal/ui: formats the status lines, warnings and errors a user sees.
• internal/log: the logger. It writes to stderr.
• internal/pidlock: the lock that keeps two data-changing commands from running at once.
• internal/globals: the version, commit and commit date, set when the binary is built.
• internal/types: distinct types for IDs, hashes and paths.

Wiring

The application is assembled with uber-go/fx, a dependency injection library. cli.NewApp lists the providers:

fx.Supply(config.Path(opts.ConfigPath)),
fx.Supply(opts.LogOptions),
fx.Provide(globals.New),
fx.Provide(log.New),
config.Module,
database.Module,
log.Module,
storage.Module,
snapshot.Module,
fx.Provide(vaultik.New),
fx.Invoke(setupGlobals),
fx.NopLogger,

Each module provides these:

• config.Module provides *config.Config, from config.Load.
• database.Module provides *database.DB and *database.Repositories. It creates the index file with mode 0600 when it does not exist.
• storage.Module provides the storage.Storer that the config selects.
• snapshot.Module provides the snapshot.ScannerFactory and the *snapshot.SnapshotManager.
• vaultik.New provides *vaultik.Vaultik.

A command runs through cli.RunOperation. It works like this:

• RunWithApp takes the lock when the command changes data, and then starts the application.
• The command's function runs in its own goroutine.
• When the function returns, it asks the application to shut down.
• On SIGINT or SIGTERM, the stop hook cancels the Vaultik context. It then waits for the function to return, so that its temporary files are removed.
• The wait ends after 30 seconds at most.

The config commands, database delete, completion and version do not build the application.

Backup

Vaultik.CreateSnapshot runs a backup. It does these steps in order:

• EnsureStorageBinding checks that the local index belongs to the configured destination.
• markDestination checks the marker file and then writes it.
• PruneDatabase removes unfinished snapshots and unused records from the local index.
• createNamedSnapshot runs once for each snapshot name.
• With --prune, runPostBackupPrune removes old snapshots and unused data.

CreateSnapshot calls skipPostBackupPrune instead of runPostBackupPrune in these cases:

• --skip-errors skipped a file or directory that could not be read.
• A configured path held nothing.
• An earlier snapshot of the same name found a mount point that this backup did not find.

PruneDatabase must succeed before anything is scanned. An unfinished snapshot left in the index would make the backup skip files that were never fully stored.

One snapshot

createNamedSnapshot does these steps:

• resolveSnapshotPaths makes each configured path absolute and resolves symlinks.
• SnapshotManager.CreateSnapshotWithName creates the snapshot record. The record is not yet marked complete.
• Scanner.Scan backs up each path in turn.
• finalizeSnapshotMetadata uploads the snapshot's metadata and marks the snapshot complete.

A new Scanner is created for each snapshot, from ScannerFactory. It owns a Chunker and a Packer.

Scanning one path

Scanner.Scan works in these steps:

• repairInterruptedBlobs deletes the records of blobs whose upload never finished. Their data is then stored again.
• loadDatabaseState loads the known files and chunks into memory.
• scanPhase walks the path. It sorts each entry into changed or unchanged.
• detectDeletedFiles counts the files that the previous snapshot of this name had and that are now gone.
• batchAddFilesToSnapshot records the unchanged files in the new snapshot.
• recordDirectories records every directory, and the mount points found.
• processPhase reads each changed file and stores it.

The walk handles each kind of entry differently:

• A regular file or a symlink is compared with its record in the index.
• A directory is recorded with its metadata. It has no content to store.
• A device node, a named pipe or a socket is skipped.
• An entry that matches an exclude pattern is skipped. For a directory, so is everything in it.

Deduplication

Data is stored once at two levels.

• Files. A file is not read again when its record in the index still matches it.
• Chunks. A chunk is named by the SHA-256 hash of its content. A chunk whose hash is already in the index is not stored again.

A file's record matches when all of these are unchanged:

• the size
• the modification time
• the mode
• the owner and group
• the link target

One case overrides the file check. A file is read again when it lists a chunk that no uploaded blob holds. This happens when the only snapshot that used the blob holding that chunk is removed. loadFilesToRechunk finds these files.

Chunks

chunker.Chunker cuts a file with the FastCDC algorithm. The cut points depend on the content, so an insertion early in a file moves only nearby chunks.

chunk_size sets the average chunk size. No chunk is larger than four times this size. Only a file's last chunk can be smaller than a quarter of it. With the default of 10 MB, those limits are 40 MB and 2.5 MB.

ChunkReaderStreaming passes each chunk to a callback as it is found. The whole file is never held in memory.

Blobs

blob.Packer collects new chunks into a blob. For each blob it does this:

• It gives the blob a UUID and creates a temporary file.
• It adds each chunk's bytes to a running SHA-256 hash.
• In the same pass it compresses the bytes with zstd and encrypts them with age. The result goes to the temporary file.
• It records each chunk's offset and length within the blob.
• It closes the blob when the next chunk would pass blob_size_limit, and at the end of each path.

The blob's name is hex(SHA256(SHA256(uncompressed contents))). It is computed from the chunk data before compression and encryption. docs/REPOSTRUCTURE.md says what follows from that.

When a blob is closed, Scanner.handleBlobReady stores it:

• It looks the blob up at the destination. A blob that is already there is not uploaded again.
• Otherwise it uploads the temporary file to blobs/<hash[0:2]>/<hash[2:4]>/<hash>.
• It records the upload in the index and links the blob to the snapshot.
• It deletes the temporary file.

A file's record is written to the index only after every blob that holds its chunks is uploaded.

Snapshot metadata

finalizeSnapshotMetadata does these steps in order:

• PopulateSnapshotBlobs records every blob the snapshot uses.
• UpdateSnapshotStatsExtended stores the snapshot's totals.
• ExportSnapshotMetadata uploads the metadata.
• MarkSnapshotComplete marks the snapshot complete.

Completion is recorded last on purpose. A backup that stops during the upload leaves the snapshot unfinished, and the next backup removes it and stores its data again.

ExportSnapshotMetadata works on a copy of the index, never on the index itself:

• It copies the index into a temporary directory with VACUUM INTO.
• It deletes the rows of every other snapshot from the copy. It also deletes the files and blobs that this snapshot does not use. It then deletes each chunk that no remaining file and no remaining blob uses.
• It keeps the local_meta and schema_migrations tables as they are. The uploaded database therefore names the destination, and lists the migrations that have run.
• It runs VACUUM, so that no deleted row is left in the file.
• It compresses the copy with zstd and encrypts it with age.
• It builds the manifest from the copy.
• It uploads metadata/<remote-key>/db.zst.age, and then metadata/<remote-key>/manifest.json.zst.

The manifest is uploaded last. A directory under metadata/ with no manifest therefore marks a backup that did not finish.

Restore

Vaultik.Restore does these steps:

• restoreIdentities parses the age private key. A missing or malformed key stops the restore before anything is downloaded.
• downloadSnapshotDB downloads and decrypts the snapshot's database into a temporary directory.
• getFilesToRestore selects the files, from the paths given.
• restoreAllFiles downloads blobs and writes the files.
• With --verify, verifyRestoredFiles reads each restored file and checks its chunk hashes.

The restore does not read the local index or the manifest. It finds each file's chunks and blobs through the downloaded database.

Checks on the downloaded database

Decryption shows that a database is readable. It does not show that the database is honest. The restore therefore checks it:

• database.OpenReadOnly opens the file read-only. It refuses a file that has a trigger, a view or a virtual table, or that lacks an expected table.
• verifySnapshotDBIdentity checks that the snapshot in the database hashes to the remote key that was asked for.
• containedRestorePath refuses a stored path that would leave the target directory. It also refuses to write through a symlink below the target.
• A blob is not decompressed beyond the size that the database gives for it.
• Each downloaded blob is hashed. The hash must match the blob's name.

Order of work

restorePlan orders the files so that few blobs are on disk at once:

• Downloaded blobs are kept decrypted in a cache directory.
• A file is restored once every blob it needs is in the cache.
• When no file is ready, the restore downloads the blobs for the file that needs the fewest.
• restoreSweeper deletes a cached blob once no remaining file needs it.

A regular file is created with mode 0600 and gets its stored mode after its content is written. A directory gets its stored mode, owner and modification time after the whole restore.

Verify

Vaultik.VerifySnapshotWithOptions checks a snapshot without the private key:

• It downloads the manifest.
• It checks that the snapshot's encrypted database is present.
• It checks that each listed blob is present with the listed size.

Vaultik.RunDeepVerify needs the private key. It does these steps:

• It downloads the manifest and the database. It checks the database as a restore does.
• verifyManifestAgainstDatabase checks that both list the same blobs with the same sizes.
• verifyBlobExistenceFromDB checks that every blob is present with the recorded size.
• verifyBlob downloads each blob, checks the hash of each chunk in it, and checks the hash that names the blob.

Prune

Vaultik.Prune does three steps:

• CleanupLocalSnapshots drops from the local index each snapshot whose manifest is gone from the destination.
• PruneDatabase removes unfinished snapshots and unused records from the local index.
• PruneBlobs deletes unused data at the destination.

PruneBlobs reads every manifest at the destination. It then deletes two kinds of object, once they are older than prune_grace_period:

• a blob that no manifest lists
• a directory under metadata/ that has no manifest, and whose snapshot is not in the local index

It deletes nothing when any manifest cannot be read. docs/REPOSTRUCTURE.md explains why this is safe, and where it is not.

The destination

storage.Storer is the only way the code reaches the destination. It has these methods:

• Put and PutWithProgress store an object.
• Get reads an object.
• Stat reports an object's size.
• Delete removes an object.
• List and ListStream list the objects under a prefix.
• DeletePartialUploads removes what an interrupted upload left behind.
• Info describes the destination.

storage.NewStorer picks the implementation from storage_url:

• s3:// gives the S3 implementation.
• file:// gives the local directory implementation.
• rclone:// gives the rclone implementation.

With no storage_url, it builds the S3 implementation from the s3 settings.

The marker file and the destination binding

Two checks keep the local index and the destination in step.

• checkDestinationMarker looks for vaultik-destination.txt. It fails when the file is reported missing while the local index holds snapshots.
• EnsureStorageBinding records the configured destination in the local index on first use. Later it fails when the config names a different destination.

Every command that changes data calls EnsureStorageBinding first.

Concurrency

• The local index is opened with one connection, so its queries run one at a time.
• The index runs in SQLite's WAL mode. A second process can therefore read it while a backup writes to it.
• Packer guards its state with a mutex.
• Scanner holds packerMu while a chunk is added to the packer and while a blob is closed.
• Repositories.WithTx runs a function in one transaction. It commits when the function succeeds and rolls back when it fails.