Compare commits

..
1 Commits
Author SHA1 Message Date
sneak c2ce69d258 Correct doc and help sentences that are false about the code (closes #233)
check / check (push) Canceled after 0s
A blob is written whole to a temporary file and uploaded once finished,
not streamed to storage. The README, ARCHITECTURE.md and
config.example.yml now say a backup needs free temporary space for each
blob (twice that for an rclone destination that cannot stream uploads)
and for the metadata export's copies of the local index, in $TMPDIR, or
partly in /var/tmp when TMPDIR is unset. Also corrected: the snapshot ID
format, what restore reads and how incomplete snapshots are removed in
docs/DATAMODEL.md, what source_path holds, the index_path default, the
config search order in the snapshot create help, what snapshot remove
cleans up in the prune help, how the release installs Go, and the
script/release and script/fmt-check comments.

Model: opus-5-5
2026-10-07 15:07:02 +00:00
4 changed files with 17 additions and 14 deletions
+2 -2
View File
@@ -84,11 +84,11 @@ The final storage unit uploaded to S3. Contains many compressed and encrypted ch
Blob creation process: Blob creation process:
1. Chunks are accumulated (up to MaxBlobSize, typically 10GB) 1. Chunks are accumulated (up to MaxBlobSize, typically 10GB)
2. As each chunk is added, its uncompressed bytes are fed to a running SHA-256 2. As each chunk is added, its uncompressed bytes are fed to a running SHA-256
3. Concurrently, the same bytes are compressed with zstd, then encrypted with age (recipients configured in config), and written to a temporary file in `$TMPDIR` (`/tmp` when unset) 3. Concurrently, the same bytes are compressed with zstd, then encrypted with age (recipients configured in config), and written to a temporary file
4. On finalize, the blob's name is the double SHA-256 of the uncompressed contents — `hex(SHA256(SHA256(...)))` — not a hash of the compressed, encrypted bytes 4. On finalize, the blob's name is the double SHA-256 of the uncompressed contents — `hex(SHA256(SHA256(...)))` — not a hash of the compressed, encrypted bytes
5. The finished file is uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}` and then deleted 5. The finished file is uploaded to `blobs/{hash[0:2]}/{hash[2:4]}/{hash}` and then deleted
The metadata export at the end of a backup also uses `$TMPDIR`: it copies the local index there and runs `VACUUM` on the copy, which writes another temporary copy and a write-ahead log. A backup therefore needs free space in `$TMPDIR` of the larger of `blob_size_limit` and about three times the size of the local index. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp`.
#### BlobChunk (`database.BlobChunk`) #### BlobChunk (`database.BlobChunk`)
Maps chunks to their position within blobs: Maps chunks to their position within blobs:
+1 -1
View File
@@ -546,7 +546,7 @@ complete annotated example also lives in
| `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) | | `s3.*` | | Legacy S3 configuration (endpoint, bucket, credentials) |
| `exclude` | | Global exclude patterns (applied to all snapshots) | | `exclude` | | Global exclude patterns (applied to all snapshots) |
| `chunk_size` | `10MB` | Average chunk size for content-defined chunking | | `chunk_size` | `10MB` | Average chunk size for content-defined chunking |
| `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. Each blob is written in full to a temporary file in `$TMPDIR` (`/tmp` when unset) before it is uploaded, and the metadata export at the end of a backup works on a copy of the local index there. A backup needs free space there of the larger of this limit and about three times the size of the local index | | `blob_size_limit` | `10GB` | Maximum blob size before splitting. Must be at least four times `chunk_size` (the largest chunk the chunker can emit), otherwise a single-chunk blob could exceed the limit. A backup needs free temporary space, because each blob is written whole to a temporary file before it is uploaded (up to about `blob_size_limit`; an rclone destination that cannot stream uploads needs about twice that) and the metadata export writes copies of the local index. Temporary files go to `$TMPDIR` (default `/tmp`); with `TMPDIR` unset, SQLite writes one of those copies to `/var/tmp` |
| `compression_level` | `3` | zstd compression level (1-19) | | `compression_level` | `3` | zstd compression level (1-19) |
| `hostname` | system hostname | Hostname used in snapshot IDs | | `hostname` | system hostname | Hostname used in snapshot IDs |
| `index_path` | platform data dir | Local SQLite index path | | `index_path` | platform data dir | Local SQLite index path |
+8 -6
View File
@@ -25,12 +25,14 @@ the tag exists and is exercised; what is left is merging `next` to
- 2026-10-07: Corrected documentation, help text and comments that were - 2026-10-07: Corrected documentation, help text and comments that were
false about the code false about the code
([issue #233](https://git.eeqj.de/sneak/vaultik/issues/233)). A blob ([issue #233](https://git.eeqj.de/sneak/vaultik/issues/233)). A blob
is not streamed to storage; it is written in full to a temporary file is not streamed to storage. The README, `ARCHITECTURE.md` and
in `$TMPDIR` and uploaded once finished, and the metadata export works `config.example.yml` now say a backup needs free temporary space,
on a copy of the local index there. The README and because each blob is written whole to a temporary file before it is
`config.example.yml` now say a backup needs free space there of the uploaded (up to about `blob_size_limit`; an rclone destination that
larger of `blob_size_limit` and about three times the size of the cannot stream uploads needs about twice that) and the metadata export
local index. Also corrected: the snapshot ID format, what restore writes copies of the local index. Temporary files go to `$TMPDIR`
(default `/tmp`); with `TMPDIR` unset, SQLite writes one of those
copies to `/var/tmp`. Also corrected: the snapshot ID format, what restore
reads and how incomplete snapshots are removed in `docs/DATAMODEL.md`, reads and how incomplete snapshots are removed in `docs/DATAMODEL.md`,
what `source_path` holds, the `index_path` and config file defaults, what `source_path` holds, the `index_path` and config file defaults,
what `snapshot remove` cleans up, how the release gets its Go what `snapshot remove` cleans up, how the release gets its Go
+6 -5
View File
@@ -313,11 +313,12 @@ storage_url: "rclone://myremote/path/to/backups"
# Chunking uses no secret (the FastCDC parameters are fixed and public). At a # Chunking uses no secret (the FastCDC parameters are fixed and public). At a
# large limit a blob holds hundreds of chunks, so individual chunk lengths are # large limit a blob holds hundreds of chunks, so individual chunk lengths are
# not visible in its size; lowering the limit toward chunk_size exposes them. # not visible in its size; lowering the limit toward chunk_size exposes them.
# Each blob is written in full to a temporary file in $TMPDIR (/tmp when # A backup needs free temporary space, because each blob is written whole to
# unset) before it is uploaded, and the metadata export at the end of a # a temporary file before it is uploaded (up to about blob_size_limit; an
# backup works on a copy of the local index there. A backup needs free space # rclone destination that cannot stream uploads needs about twice that) and
# there of the larger of this limit and about three times the size of the # the metadata export writes copies of the local index. Temporary files go to
# local index. # $TMPDIR (default /tmp); with TMPDIR unset, SQLite writes one of those copies
# to /var/tmp.
# Supports: 1GB, 10G, 500MB, 1GiB, etc. # Supports: 1GB, 10G, 500MB, 1GiB, etc.
# Default: 10GB # Default: 10GB
#blob_size_limit: 10GB #blob_size_limit: 10GB