Design for review: a cache layer (SQLite metadata + on-disk content/thumbnail store) and a crash-safe backup on it #36

New Issue

clawbot · 2026-09-10T13:17:05+02:00

clawbot commented

2026-09-10 13:17:05 +02:00

This is a plan for your review, not a finished spec — please critique and send rework requests. It covers the two goals you set: (1) a higher-level API over two long-lived caches — a SQLite metadata cache and an on-disk content/thumbnail cache — and (2) a backup tool that is extremely reliable and built on that API. Design rule applied throughout: the simplest thing that is fit for purpose, fewest moving parts, no tunables the goals don't need.

Current state (grounded)

No cache exists. Client holds only the session keys + token in memory. runBackup (src/backup.ts) and runMetadataBackup (src/metadata-backup.ts) re-enumerate and re-decrypt the whole account on every run. The only state that survives a run is the files on disk.
The incremental-sync machinery exists but is thrown away. Client.listCollections() always calls /collections/v2 with sinceTime: 0; listFiles() paginates /collections/v2/diff but always restarts at sinceTime: 0, and both drop isDeleted tombstones. So the diff/hasMore/updationTime loop is used only for within-run pagination, never to fetch just the changes, and deletions are invisible to callers.
backup <dir> layout (the only persistent state): originals/<fileID>.<ext> (content) + originals/<fileID>.json (metadata sidecar); collections/<name>/<title> symlinks into ../../originals; collections/<name>.json. Skip rule is existsSync && size > 0 — no integrity check. The sidecar is written only when absent, so edited metadata goes stale. Per-file download failures are logged, counted, and stepped over (errors[], exit 1 if any failed); that list is discarded at process exit.
Identity/versioning is already clean in the data: fileID identifies content (dedup across collections), (collectionID, fileID) is a membership, updationTime (µs) advances on any change. Your (collectionID, fileID, updationTime) key is exactly right. FileMetadata.hash carries a plaintext content hash (optional — not on every file). RawEnteFile.info.fileSize/thumbSize exist on the wire but decryptFile drops them (and drops isDeleted).
Reliability primitives already present and reusable: atomic temp-sibling+rename (writeAtomic), TAG_FINAL truncation detection in streamDecrypt, the retry classifier (withRetry/isRetryable/isSafeToReplay), per-request and per-body deadlines.

Design — the cache and its API

Where it lives. Inside the backup target directory (recommended): the SQLite file and the content/thumbnail blobs are subdirectories of the user's backup <dir>. One location, self-contained and portable, multiple independent mirrors, nothing hidden. (Alternative in the open questions: one managed store under env-paths.) "Long-lived" is a property of invalidation, not location: both caches are invalidated only by the server's own change feed, so they carry no TTL and can persist indefinitely wherever they live.

The SQLite metadata cache — what it stores, and why long-lived. Tables (minimal):

collection: id, ownerID, name, type, updationTime, isShared, the three magic-metadata layers (JSON), the decrypted collection key, deleted, and this collection's file-diff cursor sinceTime.
file: one row per membership, PK (collectionID, fileID), with updationTime, the basic + two magic metadata layers (JSON), fileHeader/thumbHeader, the decrypted file key, contentHash, fileSize/thumbSize.
content: one row per unique fileID — presence + verification state of the on-disk original and thumbnail, the stored contentHash, byte length, extension.
failure: durable per-fileID (× kind) ledger — classification (transient/permanent/unknown), message, attempts, last-tried time.
a small meta row: collections-list cursor, userID, schema version.

Populated from the diff endpoints, decrypted with the existing decryptCollection/decryptFile. Invalidation is server-driven, keyed on updationTime: a diff row newer than the stored one replaces the metadata; if fileHeader/contentHash changed, the content row is marked stale for re-download. Tombstones (isDeleted) remove the membership (or mark the collection deleted). It is long-lived because nothing else can make it stale — there is no speculative caching here; it is an authoritative local mirror, and keeping it turns every later run into an O(changes) diff instead of an O(account) full re-enumeration + re-decrypt.

Secrets note: this DB holds decrypted metadata (titles, GPS) and the decrypted collection/file keys, so it is as sensitive as session.json — create it 0600 in a 0700 dir and treat it like the password. (Alternative: store only the raw encryptedKey/nonce and re-derive keys from the in-memory master key on read; more moving parts for the same on-disk sensitivity — not recommended.)

The on-disk content + thumbnail cache — layout, keying, lifetime.

Keyed by fileID (content dedup — one download regardless of how many collections hold it; matches today). Layout: originals/<fileID>.<ext> and thumbnails/<fileID>.<ext>. Flat directories; no prefix sharding unless a real account proves it necessary.
Every write goes through writeAtomic (temp sibling + rename). The content row is flipped to "present + verified" only after the rename succeeds, so the DB can never claim a file the disk does not fully have.
Long-lived because Ente content is immutable per fileID — an edit yields a new version observable as a new updationTime (and a changed fileHeader/contentHash). A hash-verified original never needs re-fetching unless the diff reports a new version. No TTL.

The higher-level API — only what callers and the backup tool need. A cache object, constructed from a Client and a directory (the concrete class name is yours to choose). Surface:

sync() — pull each collection's diff from its stored cursor, update the DB, apply tombstones; returns what changed. This is the incremental enumeration thrown away today.
collections() / files(collectionID) — served from the DB, no network. (Also removes the O(collections) linear scan get/get-thumb do now.)
ensureOriginal(fileID) / ensureThumbnail(fileID) — idempotent: download + verify + store if absent-or-stale, else a no-op; returns the path.
verify(fileID) / verifyAll() — re-hash on-disk bytes against the stored hash; mark mismatches stale.
pending() — files whose content is absent, stale, or previously failed.
pathFor(fileID).

That is the whole surface. The only inputs are the Client and the directory — no tunables beyond what ApiClient already exposes. It needs small additions to Client (which owns the keys and decryption): resumable enumerators that accept a starting sinceTime, return the final cursor, and surface isDeleted rows — today's listCollections/listFiles reset to 0 and drop tombstones. ML/EXIF (backup-metadata) is out of scope for the core two caches; it can later read from the same DB.

Design — the crash-safe backup tool on the API

runBackup becomes: sync(), then ensureOriginal (and optionally ensureThumbnail) for each pending() file, then materialize the collections/ symlink views + collection JSON from the DB. Concretely:

Idempotency: a file present and verified in the DB is an O(1) no-op — no stat, no re-hash, no re-download. Each content version is downloaded exactly once.
Resumability after interruption: the DB is the durable progress ledger. A crash leaves content either fully renamed-in (recorded) or not (an orphan temp file, unrecorded) — never half. On restart, pending() is exactly the unfinished set and the persisted cursor means no full re-enumeration.
Atomic writes: writeAtomic for bytes; DB mutations in a transaction; the rule "rename the bytes, then record the row" keeps the DB from overstating the disk. The symlink tree and collection JSON are derived views, rebuildable from the DB, so they need no crash-safety of their own — and rebuilding the view also repairs the stale-sidecar and symlink-failure problems.
Integrity verification: TAG_FINAL already rejects truncation at decrypt time; after decrypt, compare the content hash to FileMetadata.hash when present and store it; verifyAll() re-checks disk against the DB on demand. This replaces today's size > 0. (Impl note: confirm the exact hash construction — algorithm, and the live-photo combined case — before relying on it; fall back to info.fileSize when hash is absent.)
Per-failure handling: reuse the retry classifier. A file that fails after retries is recorded in failure with its classification and attempt count, and the run continues (as today). The next run retries transient/unknown failures and can skip or de-prioritize permanent ones. Exit non-zero while unresolved failures remain. This turns today's lost in-memory errors[] into a durable, queryable ledger.
Deletions: when the diff reports a file/collection gone, keep the original on disk (it is a backup) and drop only the membership + its view. (Pruning to mirror the account is the alternative — open question.)

Ordered next steps (smallest set; reuse vs new)

All TDD per the repo workflow (tests first, red commit, branch off main). The external backup <dir> layout and the CLI contract stay unchanged.

Carry info.fileSize/thumbSize and isDeleted through decryptFile into EnteFile. Tiny; needed for integrity and deletion. Extend existing.
Resumable, tombstone-surfacing enumeration on Client: variants of listCollections/listFiles that take a starting sinceTime, return the final cursor, and include isDeleted rows. Extend existing; ApiClient already accepts an arbitrary sinceTime.
SQLite metadata cache + sync(): the schema above, cursor persistence, tombstone application, updationTime invalidation. New; reuses decrypt + enumeration.
On-disk content/thumbnail store: keyed by fileID, atomic write, hash verification, DB state recorded only post-rename, orphan temp reaping on open. New; reuses writeAtomic, streamDecrypt/TAG_FINAL, downloadFile/downloadThumbnail.
The cache API (sync, collections/files, ensureOriginal/ensureThumbnail, verify, pending, pathFor). Thin façade over 3–4.
Rewrite runBackup on the API + the durable failure ledger; port backup.test.ts; keep the layout and exit-code contract. Rewrite.

Sequencing vs v1.0.0 is yours to set — this subsumes some 1.0.0 items (notably the backup-robustness issue) and realizes the README/TODO "local cache (SQLite) … reliable" goal inside this repo, ahead of the desktop client.

Relationship to existing issues

#8 — runBackup symlink crash + partial originals: subsumed by step 6 (derived views cannot abort the run) and step 4 (atomic write + post-rename recording makes partial originals impossible).
#7 — listFiles infinite loop on a non-advancing server: the resumable enumerator in step 2 must handle it.
#22 — atomic-write durability + orphan reaping: the content store (step 4) is where reaping lands.
#21 — streamDecrypt buffering: open question 4 (streaming decrypt-to-disk) resolves the whole-file buffer too.
#24 — retry/timeout follow-ups: the per-failure ledger reuses and leans on the classifier.
#9 — filename sanitization: keying content by fileID removes the hazard for originals; the symlink view still sanitizes titles.
#10 — Client session/keys: step 2 touches Client; the cache depends on its keys.
#13 — README API reference rewrite: should wait for / incorporate the new surface.

Open questions

Cache location — inside backup <dir> (recommended: portable, self-contained, survives) vs one global store under env-paths (paths.cache is unused today; paths.data holds session.json). Global suits a future always-on desktop client; in-<dir> suits an explicit, movable backup. Recommend in-<dir> now and revisit for the desktop client.
SQLite driver — Node's built-in node:sqlite (zero new dependency, best for the hash-pinned supply chain, but still flagged experimental and needs a recent Node baseline) vs better-sqlite3 (mature, synchronous, a native build). The global "prefer stdlib" rule points to node:sqlite if your Node baseline supports it and experimental status is acceptable; otherwise better-sqlite3. Recommend node:sqlite, falling back to better-sqlite3.
Deleted files — keep originals on disk and drop only the collection view (recommended, backup semantics) vs prune to mirror the live account. A backup that silently deletes your removed photos is surprising; recommend keep.
Large-file memory — downloads currently buffer the whole plaintext in RAM (≈2× transiently), so a multi-gigabyte video can OOM the run. Add a streaming decrypt-to-temp path in ensureOriginal (recommended for "extremely reliable"; an extension of writeAtomic + streamDecrypt) vs keep whole-file buffering (simplest). Overlaps issue 21.

Model: opus-4-8

This is a plan for your review, not a finished spec — please critique and send rework requests. It covers the two goals you set: (1) a higher-level API over two long-lived caches — a SQLite metadata cache and an on-disk content/thumbnail cache — and (2) a backup tool that is extremely reliable and built on that API. Design rule applied throughout: the simplest thing that is fit for purpose, fewest moving parts, no tunables the goals don't need. ## Current state (grounded) - **No cache exists.** `Client` holds only the session keys + token in memory. `runBackup` (`src/backup.ts`) and `runMetadataBackup` (`src/metadata-backup.ts`) re-enumerate and re-decrypt the whole account on every run. The only state that survives a run is the files on disk. - **The incremental-sync machinery exists but is thrown away.** `Client.listCollections()` always calls `/collections/v2` with `sinceTime: 0`; `listFiles()` paginates `/collections/v2/diff` but always restarts at `sinceTime: 0`, and both **drop `isDeleted` tombstones**. So the `diff`/`hasMore`/`updationTime` loop is used only for within-run pagination, never to fetch just the changes, and deletions are invisible to callers. - **`backup <dir>` layout** (the only persistent state): `originals/<fileID>.<ext>` (content) + `originals/<fileID>.json` (metadata sidecar); `collections/<name>/<title>` symlinks into `../../originals`; `collections/<name>.json`. Skip rule is `existsSync && size > 0` — no integrity check. The sidecar is written only when absent, so edited metadata goes stale. Per-file download failures are logged, counted, and stepped over (`errors[]`, exit 1 if any failed); that list is discarded at process exit. - **Identity/versioning is already clean in the data:** `fileID` identifies content (dedup across collections), `(collectionID, fileID)` is a membership, `updationTime` (µs) advances on any change. Your `(collectionID, fileID, updationTime)` key is exactly right. `FileMetadata.hash` carries a plaintext content hash (optional — not on every file). `RawEnteFile.info.fileSize`/`thumbSize` exist on the wire but `decryptFile` drops them (and drops `isDeleted`). - **Reliability primitives already present and reusable:** atomic temp-sibling+rename (`writeAtomic`), `TAG_FINAL` truncation detection in `streamDecrypt`, the retry classifier (`withRetry`/`isRetryable`/`isSafeToReplay`), per-request and per-body deadlines. ## Design — the cache and its API **Where it lives.** Inside the backup target directory (recommended): the SQLite file and the content/thumbnail blobs are subdirectories of the user's `backup <dir>`. One location, self-contained and portable, multiple independent mirrors, nothing hidden. (Alternative in the open questions: one managed store under `env-paths`.) "Long-lived" is a property of invalidation, not location: both caches are invalidated only by the server's own change feed, so they carry no TTL and can persist indefinitely wherever they live. **The SQLite metadata cache — what it stores, and why long-lived.** Tables (minimal): - `collection`: `id`, `ownerID`, `name`, `type`, `updationTime`, `isShared`, the three magic-metadata layers (JSON), the decrypted collection key, `deleted`, and this collection's file-diff cursor `sinceTime`. - `file`: one row per membership, PK `(collectionID, fileID)`, with `updationTime`, the basic + two magic metadata layers (JSON), `fileHeader`/`thumbHeader`, the decrypted file key, `contentHash`, `fileSize`/`thumbSize`. - `content`: one row per unique `fileID` — presence + verification state of the on-disk original and thumbnail, the stored `contentHash`, byte length, extension. - `failure`: durable per-`fileID` (× kind) ledger — classification (transient/permanent/unknown), message, attempts, last-tried time. - a small `meta` row: collections-list cursor, `userID`, schema version. Populated from the diff endpoints, decrypted with the existing `decryptCollection`/`decryptFile`. **Invalidation is server-driven, keyed on `updationTime`:** a diff row newer than the stored one replaces the metadata; if `fileHeader`/`contentHash` changed, the `content` row is marked stale for re-download. Tombstones (`isDeleted`) remove the membership (or mark the collection deleted). It is _long-lived_ because nothing else can make it stale — there is no speculative caching here; it is an authoritative local mirror, and keeping it turns every later run into an O(changes) diff instead of an O(account) full re-enumeration + re-decrypt. Secrets note: this DB holds decrypted metadata (titles, GPS) and the decrypted collection/file keys, so it is as sensitive as `session.json` — create it `0600` in a `0700` dir and treat it like the password. (Alternative: store only the raw `encryptedKey`/nonce and re-derive keys from the in-memory master key on read; more moving parts for the same on-disk sensitivity — not recommended.) **The on-disk content + thumbnail cache — layout, keying, lifetime.** - Keyed by `fileID` (content dedup — one download regardless of how many collections hold it; matches today). Layout: `originals/<fileID>.<ext>` and `thumbnails/<fileID>.<ext>`. Flat directories; no prefix sharding unless a real account proves it necessary. - Every write goes through `writeAtomic` (temp sibling + rename). **The `content` row is flipped to "present + verified" only after the rename succeeds**, so the DB can never claim a file the disk does not fully have. - _Long-lived_ because Ente content is immutable per `fileID` — an edit yields a new version observable as a new `updationTime` (and a changed `fileHeader`/`contentHash`). A hash-verified original never needs re-fetching unless the diff reports a new version. No TTL. **The higher-level API — only what callers and the backup tool need.** A cache object, constructed from a `Client` and a directory (the concrete class name is yours to choose). Surface: - `sync()` — pull each collection's diff from its stored cursor, update the DB, apply tombstones; returns what changed. This is the incremental enumeration thrown away today. - `collections()` / `files(collectionID)` — served from the DB, no network. (Also removes the O(collections) linear scan `get`/`get-thumb` do now.) - `ensureOriginal(fileID)` / `ensureThumbnail(fileID)` — idempotent: download + verify + store if absent-or-stale, else a no-op; returns the path. - `verify(fileID)` / `verifyAll()` — re-hash on-disk bytes against the stored hash; mark mismatches stale. - `pending()` — files whose content is absent, stale, or previously failed. - `pathFor(fileID)`. That is the whole surface. The only inputs are the `Client` and the directory — no tunables beyond what `ApiClient` already exposes. It needs small additions to `Client` (which owns the keys and decryption): resumable enumerators that accept a starting `sinceTime`, return the final cursor, and surface `isDeleted` rows — today's `listCollections`/`listFiles` reset to 0 and drop tombstones. ML/EXIF (`backup-metadata`) is out of scope for the core two caches; it can later read from the same DB. ## Design — the crash-safe backup tool on the API `runBackup` becomes: `sync()`, then `ensureOriginal` (and optionally `ensureThumbnail`) for each `pending()` file, then materialize the `collections/` symlink views + collection JSON from the DB. Concretely: - **Idempotency:** a file present and verified in the DB is an O(1) no-op — no stat, no re-hash, no re-download. Each content version is downloaded exactly once. - **Resumability after interruption:** the DB is the durable progress ledger. A crash leaves content either fully renamed-in (recorded) or not (an orphan temp file, unrecorded) — never half. On restart, `pending()` is exactly the unfinished set and the persisted cursor means no full re-enumeration. - **Atomic writes:** `writeAtomic` for bytes; DB mutations in a transaction; the rule "rename the bytes, then record the row" keeps the DB from overstating the disk. The symlink tree and collection JSON are derived views, rebuildable from the DB, so they need no crash-safety of their own — and rebuilding the view also repairs the stale-sidecar and symlink-failure problems. - **Integrity verification:** `TAG_FINAL` already rejects truncation at decrypt time; after decrypt, compare the content hash to `FileMetadata.hash` when present and store it; `verifyAll()` re-checks disk against the DB on demand. This replaces today's `size > 0`. (Impl note: confirm the exact hash construction — algorithm, and the live-photo combined case — before relying on it; fall back to `info.fileSize` when `hash` is absent.) - **Per-failure handling:** reuse the retry classifier. A file that fails after retries is recorded in `failure` with its classification and attempt count, and the run continues (as today). The next run retries transient/unknown failures and can skip or de-prioritize permanent ones. Exit non-zero while unresolved failures remain. This turns today's lost in-memory `errors[]` into a durable, queryable ledger. - **Deletions:** when the diff reports a file/collection gone, keep the original on disk (it is a backup) and drop only the membership + its view. (Pruning to mirror the account is the alternative — open question.) ## Ordered next steps (smallest set; reuse vs new) All TDD per the repo workflow (tests first, red commit, branch off `main`). The external `backup <dir>` layout and the CLI contract stay unchanged. 1. **Carry `info.fileSize`/`thumbSize` and `isDeleted` through `decryptFile` into `EnteFile`.** Tiny; needed for integrity and deletion. _Extend existing._ 2. **Resumable, tombstone-surfacing enumeration on `Client`:** variants of `listCollections`/`listFiles` that take a starting `sinceTime`, return the final cursor, and include `isDeleted` rows. _Extend existing; `ApiClient` already accepts an arbitrary `sinceTime`._ 3. **SQLite metadata cache + `sync()`:** the schema above, cursor persistence, tombstone application, `updationTime` invalidation. _New; reuses decrypt + enumeration._ 4. **On-disk content/thumbnail store:** keyed by `fileID`, atomic write, hash verification, DB state recorded only post-rename, orphan temp reaping on open. _New; reuses `writeAtomic`, `streamDecrypt`/`TAG_FINAL`, `downloadFile`/`downloadThumbnail`._ 5. **The cache API** (`sync`, `collections`/`files`, `ensureOriginal`/`ensureThumbnail`, `verify`, `pending`, `pathFor`). _Thin façade over 3–4._ 6. **Rewrite `runBackup` on the API** + the durable `failure` ledger; port `backup.test.ts`; keep the layout and exit-code contract. _Rewrite._ Sequencing vs `v1.0.0` is yours to set — this subsumes some 1.0.0 items (notably the backup-robustness issue) and realizes the README/TODO "local cache (SQLite) … reliable" goal inside this repo, ahead of the desktop client. ## Relationship to existing issues - https://git.eeqj.de/sneak/quak/issues/8 — `runBackup` symlink crash + partial originals: subsumed by step 6 (derived views cannot abort the run) and step 4 (atomic write + post-rename recording makes partial originals impossible). - https://git.eeqj.de/sneak/quak/issues/7 — `listFiles` infinite loop on a non-advancing server: the resumable enumerator in step 2 must handle it. - https://git.eeqj.de/sneak/quak/issues/22 — atomic-write durability + orphan reaping: the content store (step 4) is where reaping lands. - https://git.eeqj.de/sneak/quak/issues/21 — `streamDecrypt` buffering: open question 4 (streaming decrypt-to-disk) resolves the whole-file buffer too. - https://git.eeqj.de/sneak/quak/issues/24 — retry/timeout follow-ups: the per-failure ledger reuses and leans on the classifier. - https://git.eeqj.de/sneak/quak/issues/9 — filename sanitization: keying content by `fileID` removes the hazard for originals; the symlink view still sanitizes titles. - https://git.eeqj.de/sneak/quak/issues/10 — `Client` session/keys: step 2 touches `Client`; the cache depends on its keys. - https://git.eeqj.de/sneak/quak/issues/13 — README API reference rewrite: should wait for / incorporate the new surface. ## Open questions 1. **Cache location** — inside `backup <dir>` (recommended: portable, self-contained, survives) vs one global store under `env-paths` (`paths.cache` is unused today; `paths.data` holds `session.json`). Global suits a future always-on desktop client; in-`<dir>` suits an explicit, movable backup. Recommend in-`<dir>` now and revisit for the desktop client. 2. **SQLite driver** — Node's built-in `node:sqlite` (zero new dependency, best for the hash-pinned supply chain, but still flagged experimental and needs a recent Node baseline) vs `better-sqlite3` (mature, synchronous, a native build). The global "prefer stdlib" rule points to `node:sqlite` if your Node baseline supports it and experimental status is acceptable; otherwise `better-sqlite3`. Recommend `node:sqlite`, falling back to `better-sqlite3`. 3. **Deleted files** — keep originals on disk and drop only the collection view (recommended, backup semantics) vs prune to mirror the live account. A backup that silently deletes your removed photos is surprising; recommend keep. 4. **Large-file memory** — downloads currently buffer the whole plaintext in RAM (≈2× transiently), so a multi-gigabyte video can OOM the run. Add a streaming decrypt-to-temp path in `ensureOriginal` (recommended for "extremely reliable"; an extension of `writeAtomic` + `streamDecrypt`) vs keep whole-file buffering (simplest). Overlaps issue 21. Model: opus-4-8

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: sneak/quak#36