sneak/quak
1
0
Fork 0
Code Issues 8 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

On-disk JSON metadata store for the local cache #41

New Issue
Closed
opened 2026-09-22 09:11:00 +02:00 by clawbot · 1 comment
clawbot commented 2026-09-22 09:11:00 +02:00
Collaborator
Copy Link
Copy Source

Cache/API implementation (#36), phase 2.
Depends on the decrypt-fields unit. New module.

Goal

The on-disk JSON metadata store the library reads and writes: one
metadata.json holding everything the server told us, loaded whole into RAM,
rewritten whole (temp-file + fsync + rename) only when a refresh changed
something.

Scope

  • metadata.json schema (owner-approved: JSON, never SQLite): userID,
    schemaVersion, the collections-list cursor, and per-collection records
    (including each album's file-diff cursor sinceTime, decrypted collection
    key, type, name, magic layers, updationTime, isShared) and per-membership
    file records ((collectionID, fileID), updationTime, metadata + magic
    layers, fileHeader/thumbHeader, decrypted file key, contentHash,
    fileSize/thumbSize).
  • Load: parse whole file, build in-memory Maps for id lookups. Missing/corrupt
    file is not an error — treat as empty (it is a cache; the refresh re-fetches).
  • Save: temp-file + fsync-before-rename + dir-fsync-after (reuse the exported
    atomic writer). Single rename covers cursors and the data they describe so
    they can never disagree.
  • Sensitivity: it holds decrypted titles, GPS, and keys — create the directory
    0700 and the file 0600, same as session.json.
  • No lock file (owner ruling). No sync() in any public surface.

Definition of done

  • Round-trip: write then read reproduces the model; id Maps correct.
  • Corrupt/missing file loads as empty without throwing.
  • Permissions 0700/0600 asserted; atomic write path tested.
  • make check green.

Grounding

New module under src/library/ (name the file plainly, e.g. store.ts). Reuse
the fsync atomic writer from the download-plumbing unit. This unit stores; the
refresh unit populates.

Dispatch notes: TDD; no scripted edits; no interactive questions; plain language
only. Squash subject ends (closes #<this issue>). End every message with
Model: opus-4-8.

Model: opus-4-8

Cache/API implementation (https://git.eeqj.de/sneak/quak/issues/36), phase 2. Depends on the decrypt-fields unit. New module. ## Goal The on-disk JSON metadata store the library reads and writes: one `metadata.json` holding everything the server told us, loaded whole into RAM, rewritten whole (temp-file + fsync + rename) only when a refresh changed something. ## Scope - `metadata.json` schema (owner-approved: JSON, never SQLite): `userID`, `schemaVersion`, the collections-list cursor, and per-collection records (including each album's file-diff cursor `sinceTime`, decrypted collection key, type, name, magic layers, `updationTime`, `isShared`) and per-membership file records (`(collectionID, fileID)`, `updationTime`, metadata + magic layers, `fileHeader`/`thumbHeader`, decrypted file key, `contentHash`, `fileSize`/`thumbSize`). - Load: parse whole file, build in-memory `Map`s for id lookups. Missing/corrupt file is not an error — treat as empty (it is a cache; the refresh re-fetches). - Save: temp-file + fsync-before-rename + dir-fsync-after (reuse the exported atomic writer). Single rename covers cursors and the data they describe so they can never disagree. - Sensitivity: it holds decrypted titles, GPS, and keys — create the directory `0700` and the file `0600`, same as `session.json`. - No lock file (owner ruling). No `sync()` in any public surface. ## Definition of done - Round-trip: write then read reproduces the model; id `Map`s correct. - Corrupt/missing file loads as empty without throwing. - Permissions `0700`/`0600` asserted; atomic write path tested. - `make check` green. ## Grounding New module under `src/library/` (name the file plainly, e.g. `store.ts`). Reuse the fsync atomic writer from the download-plumbing unit. This unit stores; the refresh unit populates. Dispatch notes: TDD; no scripted edits; no interactive questions; plain language only. Squash subject ends ` (closes #<this issue>)`. End every message with `Model: opus-4-8`. Model: opus-4-8
clawbot commented 2026-09-22 12:22:31 +02:00
Author
Collaborator
Copy Link
Copy Source

Implemented in #58 (branch issue-41-json-metadata-store, base next).

MetadataStore (src/library/store.ts): one metadata.json holding userID, schemaVersion, the collections cursor, and the decrypted collection and file records. Loaded whole into RAM with Maps for id lookups; rewritten whole through the download layer's exported fsync atomic writer. Binary keys are base64 on disk. Missing/corrupt/wrong-schema files load empty. Directory 0700, file 0600; no lock file, no sync() in public surfaces. File records are keyed per membership (collectionID, fileID). This unit only stores; the refresh unit (#42) populates it.

make check green (298 tests, 10 new).

Model: opus-4-8

Implemented in https://git.eeqj.de/sneak/quak/pulls/58 (branch `issue-41-json-metadata-store`, base `next`). `MetadataStore` (`src/library/store.ts`): one `metadata.json` holding `userID`, `schemaVersion`, the collections cursor, and the decrypted collection and file records. Loaded whole into RAM with `Map`s for id lookups; rewritten whole through the download layer's exported fsync atomic writer. Binary keys are base64 on disk. Missing/corrupt/wrong-schema files load empty. Directory `0700`, file `0600`; no lock file, no `sync()` in public surfaces. File records are keyed per membership `(collectionID, fileID)`. This unit only stores; the refresh unit (https://git.eeqj.de/sneak/quak/issues/42) populates it. `make check` green (298 tests, 10 new). Model: opus-4-8
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
merge-ready
Review passed; ready to merge
 needs-checks
CI / make check is not green on this PR
 needs-rebase
PR conflicts with main and must be rebased
 needs-review
PR is awaiting adversarial review
 needs-rework
Review failed; PR needs changes
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
clawbot sneak (Jeffrey Paul)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/quak#41