Design for review: a cache layer (SQLite metadata + on-disk content/thumbnail store) and a crash-safe backup on it #36
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: sneak/quak#36
Reference in New Issue
Block a user
This is a plan for your review, not a finished spec — please critique and send rework requests. It covers the two goals you set: (1) a higher-level API over two long-lived caches — a SQLite metadata cache and an on-disk content/thumbnail cache — and (2) a backup tool that is extremely reliable and built on that API. Design rule applied throughout: the simplest thing that is fit for purpose, fewest moving parts, no tunables the goals don't need.
Current state (grounded)
Clientholds only the session keys + token in memory.runBackup(src/backup.ts) andrunMetadataBackup(src/metadata-backup.ts) re-enumerate and re-decrypt the whole account on every run. The only state that survives a run is the files on disk.Client.listCollections()always calls/collections/v2withsinceTime: 0;listFiles()paginates/collections/v2/diffbut always restarts atsinceTime: 0, and both dropisDeletedtombstones. So thediff/hasMore/updationTimeloop is used only for within-run pagination, never to fetch just the changes, and deletions are invisible to callers.backup <dir>layout (the only persistent state):originals/<fileID>.<ext>(content) +originals/<fileID>.json(metadata sidecar);collections/<name>/<title>symlinks into../../originals;collections/<name>.json. Skip rule isexistsSync && size > 0— no integrity check. The sidecar is written only when absent, so edited metadata goes stale. Per-file download failures are logged, counted, and stepped over (errors[], exit 1 if any failed); that list is discarded at process exit.fileIDidentifies content (dedup across collections),(collectionID, fileID)is a membership,updationTime(µs) advances on any change. Your(collectionID, fileID, updationTime)key is exactly right.FileMetadata.hashcarries a plaintext content hash (optional — not on every file).RawEnteFile.info.fileSize/thumbSizeexist on the wire butdecryptFiledrops them (and dropsisDeleted).writeAtomic),TAG_FINALtruncation detection instreamDecrypt, the retry classifier (withRetry/isRetryable/isSafeToReplay), per-request and per-body deadlines.Design — the cache and its API
Where it lives. Inside the backup target directory (recommended): the SQLite file and the content/thumbnail blobs are subdirectories of the user's
backup <dir>. One location, self-contained and portable, multiple independent mirrors, nothing hidden. (Alternative in the open questions: one managed store underenv-paths.) "Long-lived" is a property of invalidation, not location: both caches are invalidated only by the server's own change feed, so they carry no TTL and can persist indefinitely wherever they live.The SQLite metadata cache — what it stores, and why long-lived. Tables (minimal):
collection:id,ownerID,name,type,updationTime,isShared, the three magic-metadata layers (JSON), the decrypted collection key,deleted, and this collection's file-diff cursorsinceTime.file: one row per membership, PK(collectionID, fileID), withupdationTime, the basic + two magic metadata layers (JSON),fileHeader/thumbHeader, the decrypted file key,contentHash,fileSize/thumbSize.content: one row per uniquefileID— presence + verification state of the on-disk original and thumbnail, the storedcontentHash, byte length, extension.failure: durable per-fileID(× kind) ledger — classification (transient/permanent/unknown), message, attempts, last-tried time.metarow: collections-list cursor,userID, schema version.Populated from the diff endpoints, decrypted with the existing
decryptCollection/decryptFile. Invalidation is server-driven, keyed onupdationTime: a diff row newer than the stored one replaces the metadata; iffileHeader/contentHashchanged, thecontentrow is marked stale for re-download. Tombstones (isDeleted) remove the membership (or mark the collection deleted). It is long-lived because nothing else can make it stale — there is no speculative caching here; it is an authoritative local mirror, and keeping it turns every later run into an O(changes) diff instead of an O(account) full re-enumeration + re-decrypt.Secrets note: this DB holds decrypted metadata (titles, GPS) and the decrypted collection/file keys, so it is as sensitive as
session.json— create it0600in a0700dir and treat it like the password. (Alternative: store only the rawencryptedKey/nonce and re-derive keys from the in-memory master key on read; more moving parts for the same on-disk sensitivity — not recommended.)The on-disk content + thumbnail cache — layout, keying, lifetime.
fileID(content dedup — one download regardless of how many collections hold it; matches today). Layout:originals/<fileID>.<ext>andthumbnails/<fileID>.<ext>. Flat directories; no prefix sharding unless a real account proves it necessary.writeAtomic(temp sibling + rename). Thecontentrow is flipped to "present + verified" only after the rename succeeds, so the DB can never claim a file the disk does not fully have.fileID— an edit yields a new version observable as a newupdationTime(and a changedfileHeader/contentHash). A hash-verified original never needs re-fetching unless the diff reports a new version. No TTL.The higher-level API — only what callers and the backup tool need. A cache object, constructed from a
Clientand a directory (the concrete class name is yours to choose). Surface:sync()— pull each collection's diff from its stored cursor, update the DB, apply tombstones; returns what changed. This is the incremental enumeration thrown away today.collections()/files(collectionID)— served from the DB, no network. (Also removes the O(collections) linear scanget/get-thumbdo now.)ensureOriginal(fileID)/ensureThumbnail(fileID)— idempotent: download + verify + store if absent-or-stale, else a no-op; returns the path.verify(fileID)/verifyAll()— re-hash on-disk bytes against the stored hash; mark mismatches stale.pending()— files whose content is absent, stale, or previously failed.pathFor(fileID).That is the whole surface. The only inputs are the
Clientand the directory — no tunables beyond whatApiClientalready exposes. It needs small additions toClient(which owns the keys and decryption): resumable enumerators that accept a startingsinceTime, return the final cursor, and surfaceisDeletedrows — today'slistCollections/listFilesreset to 0 and drop tombstones. ML/EXIF (backup-metadata) is out of scope for the core two caches; it can later read from the same DB.Design — the crash-safe backup tool on the API
runBackupbecomes:sync(), thenensureOriginal(and optionallyensureThumbnail) for eachpending()file, then materialize thecollections/symlink views + collection JSON from the DB. Concretely:pending()is exactly the unfinished set and the persisted cursor means no full re-enumeration.writeAtomicfor bytes; DB mutations in a transaction; the rule "rename the bytes, then record the row" keeps the DB from overstating the disk. The symlink tree and collection JSON are derived views, rebuildable from the DB, so they need no crash-safety of their own — and rebuilding the view also repairs the stale-sidecar and symlink-failure problems.TAG_FINALalready rejects truncation at decrypt time; after decrypt, compare the content hash toFileMetadata.hashwhen present and store it;verifyAll()re-checks disk against the DB on demand. This replaces today'ssize > 0. (Impl note: confirm the exact hash construction — algorithm, and the live-photo combined case — before relying on it; fall back toinfo.fileSizewhenhashis absent.)failurewith its classification and attempt count, and the run continues (as today). The next run retries transient/unknown failures and can skip or de-prioritize permanent ones. Exit non-zero while unresolved failures remain. This turns today's lost in-memoryerrors[]into a durable, queryable ledger.Ordered next steps (smallest set; reuse vs new)
All TDD per the repo workflow (tests first, red commit, branch off
main). The externalbackup <dir>layout and the CLI contract stay unchanged.info.fileSize/thumbSizeandisDeletedthroughdecryptFileintoEnteFile. Tiny; needed for integrity and deletion. Extend existing.Client: variants oflistCollections/listFilesthat take a startingsinceTime, return the final cursor, and includeisDeletedrows. Extend existing;ApiClientalready accepts an arbitrarysinceTime.sync(): the schema above, cursor persistence, tombstone application,updationTimeinvalidation. New; reuses decrypt + enumeration.fileID, atomic write, hash verification, DB state recorded only post-rename, orphan temp reaping on open. New; reuseswriteAtomic,streamDecrypt/TAG_FINAL,downloadFile/downloadThumbnail.sync,collections/files,ensureOriginal/ensureThumbnail,verify,pending,pathFor). Thin façade over 3–4.runBackupon the API + the durablefailureledger; portbackup.test.ts; keep the layout and exit-code contract. Rewrite.Sequencing vs
v1.0.0is yours to set — this subsumes some 1.0.0 items (notably the backup-robustness issue) and realizes the README/TODO "local cache (SQLite) … reliable" goal inside this repo, ahead of the desktop client.Relationship to existing issues
runBackupsymlink crash + partial originals: subsumed by step 6 (derived views cannot abort the run) and step 4 (atomic write + post-rename recording makes partial originals impossible).listFilesinfinite loop on a non-advancing server: the resumable enumerator in step 2 must handle it.streamDecryptbuffering: open question 4 (streaming decrypt-to-disk) resolves the whole-file buffer too.fileIDremoves the hazard for originals; the symlink view still sanitizes titles.Clientsession/keys: step 2 touchesClient; the cache depends on its keys.Open questions
backup <dir>(recommended: portable, self-contained, survives) vs one global store underenv-paths(paths.cacheis unused today;paths.dataholdssession.json). Global suits a future always-on desktop client; in-<dir>suits an explicit, movable backup. Recommend in-<dir>now and revisit for the desktop client.node:sqlite(zero new dependency, best for the hash-pinned supply chain, but still flagged experimental and needs a recent Node baseline) vsbetter-sqlite3(mature, synchronous, a native build). The global "prefer stdlib" rule points tonode:sqliteif your Node baseline supports it and experimental status is acceptable; otherwisebetter-sqlite3. Recommendnode:sqlite, falling back tobetter-sqlite3.ensureOriginal(recommended for "extremely reliable"; an extension ofwriteAtomic+streamDecrypt) vs keep whole-file buffering (simplest). Overlaps issue 21.Model: opus-4-8
I don't like the idea that the user (of the library) has to call sync - I want the cache to be transparent to the user, they should just use the API as if it were consulting the remote database as local.
Please write me 5 different blocks of example code, using the high-level library in five different ways, each being a different API surface/style of the high level API, so I can evaluate them and decide.
Why are we using SQLite instead of json? I expect all metadata to comfortably fit in 100MB. Reading entire files into ram and writing entire caches to disk on update will be trivial affairs. Keep our on-disk formats dead simple.
Use Fable.
Reading of your ruling: no
sync()anywhere in the public surface. Every readfirst asks the server for changes since the stored cursor (one small diff
request, usually empty), folds them into the local copy, saves it, then answers
from RAM. The caller never sees the cache. The existing
Client(
Client.fromJSON,listCollections,listFiles,downloadFile) stays as thelow-level layer underneath.
Each block below does the same task: open the library on a directory, find the
album "Iceland 2025", list its files, fetch one original, run a full backup,
report failed files.
EnteFile,Collection,BackupResult,BackupError,ClientSnapshotare today's real types; every other name is a placeholder andyours to choose.
Plain object of async methods
Flat and obvious, closest to today's
Client; nothing to learn. Tradeoff: idsget passed around by hand and the method list grows with every feature.
Collections of objects
Reads like a small object model: albums hold photos, photos know how to fetch
themselves. Pleasant to explore and hard to misuse (no loose ids). Tradeoff:
more classes to build and document, and objects held a long time can describe
a photo that has since changed on the server.
Directory-like tree
The account is presented as a read-only folder tree, the same shape as the
collections/andoriginals/layout on disk. Instantly familiar, and a FUSEmount or a file browser falls out of it later. Tradeoff: everything becomes a
path string, so metadata (GPS, dates, shared-with) has to be squeezed into a
statcall, and duplicate titles in one album need a naming rule.Async iterators and an event stream
Everything is a
for awaitloop; nothing large is ever built in memory by thecaller, and a backup reports progress as it happens instead of at the end.
Best fit for a CLI progress bar or a very large account. Tradeoff: simple
questions ("how many files?") need a loop or a helper, and error handling
inside loops is easy to get wrong.
Declare the wanted state, library makes it true
The caller says what should exist on disk and the library does whatever is
needed, every time, from any starting point (fresh, half-done, crashed). The
smallest surface of the five and the best match for "extremely reliable
backup". Tradeoff: it is a backup tool, not a library for browsing; listing an
album or grabbing one file has to be expressed as a narrower wanted state.
Why SQLite instead of JSON
No good reason under your constraint. It was chosen out of habit for "a durable
ledger with transactions", and the README/TODO wording said SQLite. With
metadata under 100MB, held in RAM and rewritten whole, JSON does the job and
the format is readable with
catandjq. Dropping SQLite also removes openquestion 2 (the driver choice) and a dependency. Accepted.
Simplest shape, inside the backup directory:
metadata.jsonholds everything the server told us: collections, files percollection, and the change cursors. One file, so the cursors can never
disagree with the data they describe: one rename covers both.
originals/<fileID>.<ext>andthumbnails/<fileID>.<ext>hold content, astoday. No "downloaded" flag is stored anywhere. Content only ever appears by
rename, so a file that exists is complete; the directory listing is the
record of what we have.
failures.jsonholds the failed-file list (small).collections/symlinks and per-collection JSON stay, rebuilt frommetadata.jsonon each run.What JSON costs here, plainly:
file would be absurd on a 50,000-file first backup. The shape above avoids it:
metadata.jsonis rewritten only when the server reports changes (once perrefresh, and during the first big enumeration at most once per collection or
every few seconds), never per download. Downloads touch only
failures.json,which is tiny.
or the new one, never half. Worst case we re-ask the server for a few pages
of changes. One gap: today's
writeAtomic(src/download/index.ts) does notfsync, so after a power cut the renamed file can be empty. The metadata
writer must fsync the file before the rename and the directory after it;
that is already tracked in #22.
writeAtomicis also private to the download module and needs exporting.metadata.jsonis not data loss: delete it and the nextread re-fetches everything from the server. It is a cache.
would have handled that). Answer: a lock file, one writer per directory, the
second process fails with a clear message.
second at startup. Acceptable for a backup tool; worth knowing for the
desktop client.
Maps built at load. No querylanguage is lost that we needed.
so
0600in a0700directory, same assession.json.One question the transparency ruling raises, for later: when the server is
unreachable, should reads fail, or answer from the local copy and say so? I
would answer from the local copy with a flag on the result; backups would still
fail loudly.
My pick: plain object of async methods for the library, with
backup()builtinternally the declare-the-state way, because it is the smallest step from
today's
Clientand the other styles can be layered on it later withoutbreaking it.
Disclosure: posted with the
teaCLI asclawbot; the Gitea MCP tool was notavailable in this session.
Model: fable-5-1
all these api shape designs use positional arguments which make it unclear what those paths mean in the second argument. use an args object with descriptive and unambiguous key names.
i like the collection of objects shape, but it needs a way of reporting progress in realtime, both when working a list or when working a single thing like a download. perhaps a progressCallback arg?
Revised API after your two rulings: every call takes one args object with named keys (no positional arguments anywhere in the new surface), and every long operation accepts an optional
onProgresscallback that fires in real time, both for list work and for a single download.Standing decisions, restated once:
sync()in the surface. Every read first asks the server for changes since the stored cursor (one small diff request, usually empty), folds them in, saves, then answers from RAM.metadata.json(collections, files, cursors),failures.json, plusoriginals/andthumbnails/, all written by temp-file-plus-rename.The surface
Client,ClientSnapshot,Collection,EnteFile,FileType,BackupResultare today's types (src/client.ts,src/model/types.ts,src/backup.ts).Library,Album,Photo,ProgressEvent,Failureare new; the names are yours to change.What
onProgressemits, per operation:albums.list,photos.list,byName,byID): arefreshstartedanddone(orfailed) pair first, since every read checks the server.albums.list:started,done.itemsTotalis known at once because/collections/v2answers in one response.photos.list:started, then oneprogressper diff page withitemsDonegrowing anditemsTotalabsent (the server pages withhasMore, so the total is unknown until the end), thendone. When the local copy is already current it is juststartedanddone.original()/thumbnail():startedwithphotoandbytesTotalwhen known,progressper chunk withbytesDone, thendone,skipped(already on disk) orfailedwitherror.backup():backup startedwithitemsTotal(known after the refresh), then for each file the same download events as above carryingphotoandalbum, then onebackup progresswithitemsDoneafter each file, thenbackup done. Per-item events and overall progress arrive through the one callback, so no second mechanism is needed.Worked examples
Find an album and list its files with progress:
Fetch one original with download progress:
Full backup with progress and the failures report:
Implementation notes, grounded in the current code
downloadFileanddownloadThumbnailinsrc/download/index.tsreportbytesWrittenonly at the end; the chunk loop needs a per-chunk hook sobytesDonecan be emitted while the body streams.bytesTotalisFileBlob.size, which comes fromRawEnteFile.info.fileSize;decryptFiledrops it today (step 1 of the original plan carries it through).ProgressCallbackinsrc/backup.tsis(message: string) => voidtoday; it is replaced by the typed event, and the CLI formats events into its log lines.Client(listFiles(collectionID, collectionKey),downloadFile(file, outPath?)) is left as is; the ruling is applied to the new surface, andClientcan be brought in line in a separate change if you want it.Open questions
refreshfailedevent (witherror), whilebackup()throws; the alternative is that every read fails when offline.Disclosure: posted with the tea CLI as clawbot; the Gitea MCP tool was not available in this session.
Model: fable-5-1
the cachedirectory should be an option and it should default to standard xda cache dirs. downloadDirectory should be an option as well.
how does this library work as a backend client for a GUI tool to replace the ente 1p desktop app? i want to build something very simple like autistmask/universal paperclips that just does a very plain minimum-styled list of photos (by week or month or something), albums, magic tags, etc. the ente 1p one is terribly laggy/slow and with all metadata on disk/in memory and a large thumbnail/data cache, we should be able to build one that's extremely performant (in electron). how does this api shape work for that use case? it seems good to me but i'd like a critical review along with any proposed changes that might make it work best for something like that.
also as part of this api shape implementation i'd like the cli quak tool updated to use the new api.
Reading of your three orders: both directories become options with the cache defaulting to the XDG cache location; the API gets a critical review as the backend of a minimal Electron photo app; the
quakCLI is ported to this API in the same work. Short verdict on the review: the object model is right for the CLI and for scripts, but as accepted it would reproduce the lag you want to get rid of, for three reasons: every read waits on the network, there is no read across all photos at all, and objects with methods cannot cross from the Electron main process to the window. Proposed changes below.Directory options
env-paths4.0.0 asbin/quak.tsalready calls it (envPaths("quak", { suffix: "" }), today used only forpaths.data/session.json):$XDG_CACHE_HOME/quak, else~/.cache/quak~/Library/Caches/quak%LOCALAPPDATA%\quak\Cache/<userID>subdirectory (client.whoami().userID), so two accounts never share onemetadata.json.cacheDirectoryholds only what can be deleted and fetched again,0700directory and0600files (it contains decrypted titles, GPS and file keys):metadata.json(albums, files, change cursors)failures.json(losing it loses only attempt counts)thumbnails/<fileID>.jpgoriginals/<fileID>.<ext>: originals fetched for viewing (photo.original())downloadDirectoryholds the backup the user keeps, in today'srunBackuplayout, unchanged:originals/<fileID>.<ext>, theoriginals/<fileID>.jsonsidecars,collections/<name>/symlinks,collections/<name>.json. The sidecars make it readable without the cache.backup()writes originals todownloadDirectory. An original already in the cache is copied, not downloaded again.photo.original()returns thedownloadDirectorycopy if there is one, else fetches into the cache.backup()with nodownloadDirectory(neither inopen()nor in the call) throws before any network traffic. It does not fall back to the cache: a backup in a directory that cache cleaners delete is not a backup.Critical review as an Electron backend
Startup: reads must not wait for the server
JSON.parseof a 100MBmetadata.jsonalso blocks its thread for about a second, and rewriting it blocks again; in the Electron main process that freezes every window.open(). The CLI keeps today's ruling; the GUI answers from RAM at once and is told when a refresh changed something. Still nosync()the caller must call."inBackground":open()returns as soon asmetadata.jsonis loaded (first paint is local only), a refresh starts at once and repeats every 60 seconds, and changes arrive throughsubscribe(below). On a first run with nometadata.json, reads return what has arrived so far and change events stream the rest in./collections/v2returns each album'supdationTime, so only albums whose time advanced need a/collections/v2/diffcall. Today'sClient.listFilesalways restarts atsinceTime: 0.utilityProcess, not in main, so the parse and the rewrite never block a window. That is the app's choice, but it forces the next point.Process split: plain data and ids, not objects
PhotoandAlbumhave methods, andEnteFilecarrieskey: Uint8Array. Across Electron IPC the methods are dropped, and file keys should never reach the window. One IPC call per photo (photo.thumbnail()200 times per screen) is also too chatty.Librarythat takes ids in arrays and returns plain records.AlbumandPhotostay as thin wrappers over those calls for in-process callers (the CLI, scripts).Photo:takenAtandtitleread onlymetadata, so a date or name the user corrected in Ente would be ignored and the timeline would sort wrongly.thumbnailPathon the record means cached thumbnails paint with no further call. The library knows what is cached from one directory read atopen().Timeline: there is no read across all photos
album.photos.list()andlib.photos.byID()only. A timeline would have to list every album and merge, and a file in three albums would appear three times (EnteFileis one row per album membership).Photo[]for 50,000 photos is fine in RAM but wrong over IPC. A virtualized list needs two things: the count per group (for total scroll height) and the ids per group; then records only for the rows on screen.Week-grouped timeline in use (main side;
sendis the app's IPC):Thumbnails at scroll speed
photo.thumbnail()is one request per call with no ordering, no limit on simultaneous requests and no way to stop. A fast scroll queues thousands of downloads for rows already gone.protocol.handle) that serves files out ofcacheDirectory/thumbnails; plainfile://is blocked for a page not itself loaded fromfile://.visiblebeforeaheadbeforebackground, the samefileIDrequested twice is downloaded once.Prefetch in use:
downloadFileholds the whole plaintext in RAM (streamDecryptinsrc/download/index.ts), so opening a multi-gigabyte video can kill the process. Streaming decrypt to disk (#21) is a precondition for the GUI, not an extra.Magic metadata and search
EnteFile.magicMetadataandpubMagicMetadataareRecord<string, unknown>(src/model/types.ts); the acceptedPhotoexposes none of it. The fields a GUI needs are promoted ontoPhotoRecordabove (caption,editedTime,editedName,w,hfrom the public layer;visibilityfrom the private one). Unverified: those are Ente's field names as I know them; confirm against the fixtures when implementing.PhotoFilter.text, in RAM. Nothing more is needed at 50,000 records.src/metadata-backup.tsalready fetches it (/files/data/fetch, typemldata, 200 ids per request): face detections and CLIP embeddings per file. Ente stores no tag strings server-side; its "magic search" embeds the typed text on the device and compares. Named people live behind an Ente endpoint quak does not implement. Embeddings for 50,000 files exceed 100MB as JSON, so they must never go intometadata.json; if wanted, a separatemldata.jsonloaded on demand.Change notification
quak backupon the same cache would always fail. Recommend no lock:metadata.jsoncarries its own cursors and is replaced by rename, so whichever process writes last leaves a consistent file and the other re-asks the server for a few changes.CLI
The
quakCLI is ported to this API in the same implementation; commands, flags, output and exit codes stay as they are, plus one new global option--cache-dir.collections:lib.albums.list().files --collection:lib.albums.byID()thenalbum.photos.list()get/get-thumb:lib.photos.byID()thenphoto.original()/photo.thumbnail(), copied to--out(default as today). This removes today's scan of every album per call;--collectionis accepted and ignored.backup <dir>:Library.open({ client, downloadDirectory: dir }),lib.backup({ onProgress }); the typed events are formatted into today's log lines;--jsonand exit 1 on failures unchanged.backup-metadata <dir>andhelper *-missing-thumbnails: enumerate throughlib.albums/album.photosinstead ofClient.listCollections/listFiles; the ML fetch, EXIF extraction and thumbnail upload stay where they are.--exifreadsphoto.original()instead of a temporary download.login,whoami,logout: unchanged.Open questions
open()optioncacheOriginalsMaxBytes; thumbnails are never removed.lib.refresh()for a reload button, never required: recommend yes.Disclosures: posted with the
teaCLI asclawbot(the Gitea MCP tool was not available). Nothing was built or run; Electron behaviour (protocol.handle,utilityProcess, IPC cloning) is from its documentation, not tested here. The 8-download limit, the 60-second interval and the 10 GiB default are judgement calls.Model: fable-5-1
magic tags means ML results - "magic" refers to ente's search by content feature.
10GiB cache is tiny. 100GiB is fine, disks are big. we can adapt it downward dynamically LRU if the disk has less than 50GiB free.
no lock file.
i would like a TTL of our local cache, so on launch we fetch things, but then we don't hit the network at all again for 1-3 seconds, so that subsequent calls serve entirely locally. this should simplify the UI side of things.
i want to use a lot of ram and disk and cpu in exchange for no lag ever. it's ok to load all of the metadata into ram on both sides of the IPC on startup, and to use dozens of gigabytes of disk space for local cache. i'd like to perhaps even aggressively locally precache thumbnails up to and including 100% of the account, for maximum local speed. we should probably locally precache originals for the latest week of items too, and all items in the favorites album.
let's move all of the caching/ttl/etc stuff into the library so the UI doesn't have to know much of anything about it.
suggests that all queries would hit the server for freshness. i don't want that. we can be 3-5 seconds behind the server for recent images, and 5-10 seconds behind the server for lists of albums, and probably 10 seconds behind the server for album membership.
i want separate parallelization for metadata fetches vs content fetches vs thumbnail fetches. metadata fetches should have max 10 req parallel, content fetches should have max 5, and thumbnail fetches should have max 25.
Revised design after #36 (comment) and #36 (comment). Reading of the seven rulings, then the design that follows from each. Everything accepted in #36 (comment) that is not named below stands unchanged (args objects everywhere,
onProgressevents, plain records over IPC,PhotoRecord,timeline.groups, JSON on disk, directory options, CLI port).Rulings, as read:
metadata.json.refresh: "beforeEachRead"mode and theserverReachableflag are gone.TTL and refresh (rulings 4 and 6)
open()loadsmetadata.jsoninto RAM, then fetches: one/collections/v2call with the stored cursor, then/collections/v2/difffrom each album's stored cursor for the albums whoseupdationTimeadvanced, then the ML data (below). On a first run there is no local copy, soopen()resolves only when this first fetch is complete. On later runs it resolves as soon as the local copy is loaded and the first refresh has been started, so a launch with the server down still paints at once./collections/v2?sinceTime=cursorrequest (empty answer when nothing changed), and a diff only for the albums it names. Every read (albums.list,photos.list,photos.records,timeline.groups,snapshot) is answered from RAM, always, with no refresh in front of it. That meets all three bounds with one small request per 3 s: new images, the album list and membership are each at most 3 s plus one round trip behind. There is no per-kind timer because the per-album diff is only sent for albums the collections call reported as changed, so the three bounds collapse into one interval. This rests on the server advancing an album'supdationTimewhen a file is added or removed from it, which is how Ente's own clients decide what to diff; unverified here, see the end.metadata.jsonis rewritten (temp file plus rename, fsync before and after) only when a refresh changed something. A refresh that fails is not an error a read can see: reads keep answering from RAM, the failure is reported once throughonProgress(operation: "refresh",status: "failed") and is visible instatus().LibraryChangelosesserverReachable;open()losesrefresh.refreshIntervalSeconds, default 3. A caller that wants the album list only every 10 s sets 10; there is no separate knob per kind.lib.refresh()is dropped: with a 3 s interval a reload button has nothing to do.Client.listCollectionsandClient.listFiles(src/client.ts) both start atsinceTime: 0and dropisDeletedrows; the refresh needs the cursor-taking, tombstone-surfacing variants from the original plan (step 2). Every request in this section goes through the metadata pool.All metadata in RAM on both sides (ruling 5)
snapshot()returns the whole account as plain records in one call, andsubscribethen delivers only what changed. The window keeps its own copy, groups and filters it locally, and never asks the library per row. 50,000PhotoRecords are roughly 15 MB over structured clone; a one-time cost at startup.PhotoRecordgainsoriginalPath?next tothumbnailPath?: set when the original is in the cache or indownloadDirectory, so the viewer opens pinned and recently viewed photos with no call at all.timeline.groupsandPhotoFilterstay for in-process callers (CLI, scripts); a window holding the snapshot does not need them.Startup in the app's library process:
Precache (rulings 5 and 6)
Both precaches start inside
open()and need nothing from the caller. They report through theonProgressgiven toopen()and throughstatus().thumbnails.ensure:visibleandaheadrequests go ahead of the precache, so scrolling is never behind the background fill. A file whose thumbnail already exists costs oneSetlookup, from the directory listing taken atopen().Collection.type === "favorites"), then every file whosetakenAtlies within the 7 days ending at the newesttakenAtin the account. These are the pinned set. When a refresh adds a favorite or a newer file, it joins the queue; when the week window moves or a favorite is removed, the file leaves the pinned set and becomes an ordinary cached original.photos.byID().original()andthumbnails.ensurekeep their shapes;original()on a cached or pinned file returns the path with oneskippedevent and no network.runBackupandfetchMLDataForFilesare sequential today;downloadFilebuffers the whole plaintext (streamDecrypt,src/download/index.ts), so 5 concurrent originals means 5 files in RAM at once and a 4 GB video on each. Streaming decrypt to disk (#21) is a precondition for the originals precache, as it was for the viewer.Originals cache limit and eviction (ruling 2)
cacheOriginalsMaxBytes, default 100 GiB. Before each original is written the library checks the volume holdingcacheDirectory(fs.statfs, Node 18.15+, this repo is on 22) and uses the lower of the configured limit andbytesUsedByOriginals + bytesFree - 50 GiB. So on a full disk the limit falls on its own and rises again as space returns;status().originalsLimitBytesshows the value in force.freeBelowBytes(default 50 GiB) is the second option; both areopen()options.utimes) when a read returns its path, so the order survives restarts and needs no ledger.atimeis not used because most volumes mountnoatimeorrelatime.cacheDirectory/originalsis ever evicted.downloadDirectoryis the backup; it is never counted and never touched.No lock file (ruling 3)
No lock and no detection.
metadata.jsoncarries its own cursors and is replaced by one rename, so whichever process writes last leaves a consistent file; the other keeps running on its RAM copy and its next refresh re-asks the server from its own cursor. The same holds for the ML files and for eviction: two processes evicting at once each remove least-recently-used files from the directory they both see, and the worst case is a file removed that the other would have kept, which is refetched on next use.ML data, the magic tags (ruling 1)
What exists:
src/metadata-backup.tsfetches/files/data/fetchwithtype: "mldata", 200 ids per request, decrypts each entry with the file key and gunzips it. The payload (test/cli/metadata-backup.test.ts) is{ face: { faces: [{ faceID, detection: { box, landmarks }, score, blur, embedding }] }, clip: { embedding } }. Nothing else server-side: Ente keeps no tag strings; search by content embeds the typed text on the device and ranks files by similarity toclip.embedding.Storage, its own part of the cache, never in
metadata.json:mldata/<fileID>.json: the decrypted, gunzipped payload as fetched, one file per fileID, written by rename. Present means complete, the same rule as originals.mldata/clip.f32plusmldata/clip.json: the derived index the search runs on.clip.jsonlists the fileIDs in order and the embedding length;clip.f32is those embeddings as oneFloat32Array, so 50,000 files at 512 floats load in one read as 100 MB and no parse. Rebuilt from the per-file JSON when missing or when it disagrees with the files present, appended to when new payloads arrive.Float32Arrayand the id list. Per-file payloads (face boxes, landmarks) are read from disk on demand, not held.metadata.jsonand absent frommldata/is requested, 200 per call, 10 calls in flight. That is the whole account on the first run and only new files afterwards. A file whoseupdationTimeadvanced is refetched, because the ML payload is rewritten when a client re-runs indexing. The fetch is reported throughonProgress(operation: "fetchMLData") andstatus().Surface:
Search by content itself needs a text encoder that turns the typed words into an embedding in the same space as
clip.embedding. quak has none.searchByEmbeddingtakes the vector so the app, or a later addition to the library, can supply it; see the open questions.Three request pools (ruling 7)
/collections/v2,/collections/v2/diff,/files/data/fetchgetFileStream(originals, for precache, viewer and backup)getThumbnailStreamvisiblethumbnails) goes before precache and backup work; the same fileID requested twice is fetched once. A pool that is idle does not lend its slots to another.open()optionsmetadataConcurrency,contentConcurrency,thumbnailConcurrency, defaults 10, 5, 25. Grounding:ApiClienthas no concurrency limit and none of its callers run more than one request at a time; the pools wrap calls intoClientrather than changingApiClient, so the low-level layer stays as it is andquak backupgets 5 parallel downloads throughlib.backup()without a change of its own.What the UI is left knowing
Open, take the snapshot, subscribe, tell the library which thumbnails are on screen (
thumbnails.ensurewithvisibleandahead), ask for an original's path when the viewer opens one, and hand the library a query embedding for search. It never sees a cursor, a TTL, a limit or a pool.Judgement calls and open questions
updationTimeon file add or remove, the fallback is a diff of every album every 10 s, which is O(albums) small requests per cycle. Please confirm which it is if you know; otherwise the implementation checks it against the live API first.open()not waiting for the first refresh when a local copy exists (above). The alternative, waiting, delays first paint by a round trip and by up to the 30 s request deadline when offline.takenAtin the account, not the last 7 calendar days, so an account with no new uploads still has a week of originals pinned. Say if you meant upload time or calendar days.metadata.jsonrewrite would take about 10 s of CPU per change, which argues for splitting it per album at that scale.searchByEmbedding(nothing new in quak; the app carries the model). Recommend the second now, the first as its own issue once the app exists.updationTimeadvanced. If Ente re-indexes without touching the file, that change is missed until the per-file JSON is deleted; a cheaper check would need a server endpoint I have not confirmed exists.Disclosures: posted through the Gitea REST API as
clawbot; the Gitea MCP tool was not loaded in this session. Nothing was built or run. Not checked: the server'supdationTimebehaviour above, the ML payload shape beyond the test fixture, andfs.statfson Windows.Model: fable-5-1
Implementation issues filed for this design, broken into small ordered units,
starting from the pieces every later unit needs. They will be driven one
issue-to-PR unit at a time, each independently reviewed before it lands on
next. The design here is treated as settled enough to start; amend any unit asneeded and later ones are fleshed out just before dispatch.
Order and dependencies:
Foundation (low-level; everything else builds on these)
fileSize/thumbSize/isDeletedthroughdecryptFile.Client(after #37). Subsumes #7.Metadata cache + library core
Library.open+ background refresh loop + staleness window (after #41, #38).Content/thumbnail cache
Machine-learning (magic) data
Backup, CLI, docs
quakCLI to the new API (after #51). Overlaps #17.Older issues these subsume or overlap (#7, #8, #9, #17, #21, #22) are cross-
referenced in the units above and closed by their landing commits where fully
covered; #24 (retry follow-ups) and #10 (session/keys hardening) stay separate.
Starting now with the foundation: #37 and #39 first (independent files, in
parallel), then #38 after #37 and #40 after #39.
Model: opus-4-8
I would like some additional asynchronous API calls that can be used sort of like how you would use a synchronous fetch so you will still await them but they don't actually resolve until such time a round trip to the server has been made and the local cache has been updated. I want the usual case to be that a fetch kicks off a background retrieval and then responds from disk. as has been designed presently. But what that means is that it's going to get stale data. And in certain cases, you are going to want to lag while it makes a round trip so that you can be sure that you have the latest data. So in that case, I would like some additional methods available that will not resolve until such time it knows that the data is synchronized. The CLI tools should then use these methods.
Reading of the amendment above, for the record and for the manager to fold into the design and the affected unit issues before they dispatch:
Manager: amend the read-surface and CLI unit issues accordingly (and any other unit this touches), then continue the normal one-unit-at-a-time flow.
(model: claude-fable-5)