diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index c91f95d..1fc5f72 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -58,8 +58,10 @@ source files - `internal/blob`: the `Packer`, which collects chunks into blobs. - `internal/blobgen`: the writer that hashes, compresses and encrypts a blob, and the reader that reverses it. -- `internal/database`: the local index. It holds the schema and one repository - type for each table. +- `internal/database`: the local index. It holds the schema and the repository + types that read and write the tables. + [docs/DATAMODEL.md](docs/DATAMODEL.md#code) says which type serves which + table. - `internal/storage`: the `Storer` interface and its three implementations, for S3, a local directory and rclone. - `internal/s3`: the S3 client that the S3 `Storer` uses. @@ -68,7 +70,7 @@ source files - `internal/log`: the logger. It writes to stderr. - `internal/pidlock`: the lock that keeps two data-changing commands from running at once. -- `internal/globals`: the version, commit and build date, set when the binary +- `internal/globals`: the version, commit and commit date, set when the binary is built. - `internal/types`: distinct types for IDs, hashes and paths. @@ -127,6 +129,14 @@ build the application. - `createNamedSnapshot` runs once for each snapshot name. - With `--prune`, `runPostBackupPrune` removes old snapshots and unused data. +`CreateSnapshot` calls `skipPostBackupPrune` instead of `runPostBackupPrune` in +these cases: + +- `--skip-errors` skipped a file or directory that could not be read. +- A configured path held nothing. +- An earlier snapshot of the same name found a mount point that this backup did + not find. + `PruneDatabase` must succeed before anything is scanned. An unfinished snapshot left in the index would make the backup skip files that were never fully stored. @@ -186,17 +196,17 @@ A file's record matches when all of these are unchanged: - the link target One case overrides the file check. A file is read again when it lists a chunk -that no uploaded blob holds. This happens after the only snapshot that used -such a blob is removed. `loadFilesToRechunk` finds these files. +that no uploaded blob holds. This happens when the only snapshot that used the +blob holding that chunk is removed. `loadFilesToRechunk` finds these files. ### Chunks `chunker.Chunker` cuts a file with the FastCDC algorithm. The cut points depend on the content, so an insertion early in a file moves only nearby chunks. -`chunk_size` sets the average chunk size. A chunk is between a quarter of that -size and four times that size. With the default of 10 MB, a chunk is between 2.5 -MB and 40 MB. +`chunk_size` sets the average chunk size. No chunk is larger than four times +this size. Only a file's last chunk can be smaller than a quarter of it. With +the default of 10 MB, those limits are 40 MB and 2.5 MB. `ChunkReaderStreaming` passes each chunk to a callback as it is found. The whole file is never held in memory. @@ -324,8 +334,8 @@ time after the whole restore. `Vaultik.Prune` does three steps: -- `CleanupLocalSnapshots` drops each snapshot from the local index whose - directory is gone from the destination. +- `CleanupLocalSnapshots` drops from the local index each snapshot whose + manifest is gone from the destination. - `PruneDatabase` removes unfinished snapshots and unused records from the local index. - `PruneBlobs` deletes unused data at the destination. @@ -375,8 +385,8 @@ Every command that changes data calls `EnsureStorageBinding` first. ## Concurrency -- The local index is opened with one connection, so all access to it is in - turn. +- The local index is opened with one connection, so its queries run one at a + time. - The index runs in SQLite's WAL mode. A second process can therefore read it while a backup writes to it. - `Packer` guards its state with a mutex. diff --git a/docs/DATAMODEL.md b/docs/DATAMODEL.md index f011a7a..090f93e 100644 --- a/docs/DATAMODEL.md +++ b/docs/DATAMODEL.md @@ -327,7 +327,12 @@ holds the file's new chunks is stored. The metadata of a snapshot is made from a copy of the index: - The index is copied into a temporary directory with `VACUUM INTO`. -- Every row that does not belong to this snapshot is deleted from the copy. +- The rows of every other snapshot are deleted from the copy. So are the files + and blobs that this snapshot does not use. A chunk is then deleted when no + remaining file and no remaining blob uses it. +- The `local_meta` and `schema_migrations` tables are kept as they are. The + uploaded database therefore names the destination, and lists the migrations + that have run. - `VACUUM` rewrites the copy, so that no deleted row is left in the file. - The copy is compressed with zstd and encrypted with age. - The manifest is built from the copy's `snapshot_blobs` table. @@ -347,6 +352,7 @@ A restore does not use the local index. - It reads the files to restore from `files`. - It finds each file's chunks in `file_chunks`. - It finds the blob and offset of each chunk in `blob_chunks`. +- It finds the name of each blob in `blobs`. It does not read the manifest. @@ -354,7 +360,7 @@ It does not read the manifest. `prune` changes the local index in two ways: -- It deletes each snapshot whose directory is gone from the destination. +- It deletes each snapshot whose manifest is gone from the destination. - It deletes unfinished snapshots, and the rows that no snapshot uses. The second step removes unused rows in this order: @@ -362,7 +368,7 @@ The second step removes unused rows in this order: - files that are in no snapshot - blobs that are in no snapshot - `blob_chunks` rows whose chunk is gone -- chunks that no file uses +- chunks that no file and no blob uses `prune` then deletes unused data at the destination. That step reads the manifests, not the local index. It reads the local index only to keep a @@ -370,19 +376,26 @@ directory under `metadata/` whose snapshot is still recorded there. ## Code -Each table has a repository type in `internal/database`: +Each repository type in `internal/database` serves one or more tables: -- `FileRepository` -- `ChunkRepository` -- `FileChunkRepository` -- `ChunkFileRepository` -- `BlobRepository` -- `BlobChunkRepository` -- `SnapshotRepository` -- `UploadRepository` -- `LocalMetaRepository` +- `FileRepository`: `files` +- `ChunkRepository`: `chunks` +- `FileChunkRepository`: `file_chunks` +- `ChunkFileRepository`: `chunk_files` +- `BlobRepository`: `blobs` +- `BlobChunkRepository`: `blob_chunks` +- `SnapshotRepository`: + - `snapshots` + - `snapshot_files` + - `snapshot_blobs` + - `snapshot_mount_points` +- `UploadRepository`: `uploads` +- `LocalMetaRepository`: `local_meta` -`Repositories` holds one of each. +No repository type serves `schema_migrations`. The code that applies the +migrations reads and writes it. + +`Repositories` holds one of each type. `Repositories.WithTx` runs a function in one transaction: @@ -401,5 +414,5 @@ rolled back. `snapshot list`, can read the index while a backup writes to it. - A connection waits up to 10 seconds for a lock before it fails. - Foreign keys are enforced. -- A process opens one connection, so all its access to the index is in turn. +- Each open database uses one connection, so its queries run one at a time. - Queries pass their values as parameters. diff --git a/docs/REPOSTRUCTURE.md b/docs/REPOSTRUCTURE.md index 9acf674..7fde2f0 100644 --- a/docs/REPOSTRUCTURE.md +++ b/docs/REPOSTRUCTURE.md @@ -253,17 +253,23 @@ therefore does not stop overwrites. ### Sizes are visible -age adds no padding. The size of an object is the exact size of its compressed -contents. +An object's size follows exactly from the size of its compressed contents. All the new chunks in one blob share one zstd stream. The stream looks back 8 MiB, or 4 MiB at compression levels 1 and 2. A blob is closed at `blob_size_limit` and at the end of each configured path. A chunk that is already stored is never packed again. -Someone who can write into a backed-up file and watch blob sizes can learn -something in one case. Their data and a secret must land in the same chunk, in -a file that keeps changing. +Data that an outsider writes compresses smaller when it matches a secret in the +same chunk, so the blob gets smaller. The blob's size therefore shows the +outsider whether their data matched the secret. + +This works only when all of these hold: + +- The outsider can write into a backed-up file. +- The outsider can see the blob sizes at the destination. +- The outsider's data and the secret land in the same chunk. +- The file keeps changing. Back up a directory that outsiders can write to as a snapshot of its own. @@ -363,10 +369,11 @@ objects. An rclone remote that cannot move objects can keep a partial object. - a blob that no manifest lists - a directory under `metadata/` that has no manifest +It reads the manifests, which are not encrypted, so it needs no private key. + These rules keep it from deleting data that is in use: - It keeps every blob that any manifest lists. -- It reads the manifests, which are not encrypted, so it needs no private key. - It deletes nothing when any manifest cannot be downloaded or decoded. - It deletes a blob only once the blob is older than `prune_grace_period`. - It deletes a directory only once every object in it is that old.