quak backup writes each file's ML data into its JSON (closes #163)
check / check (push) Successful in 2m33s

lib.backup() now waits for an ML data fetch, joining the one its refresh
started or starting one, before it writes the per-file JSON, and each
file's JSON carries the cached payload as mlData. When the fetch fails,
each file with no cached ML data gets mlDataError and an entry in
failures.json, so the result counts it as failed and quak backup exits 1;
the next run fetches again.

Judgement call: the wait comes after the originals are downloaded, so the fetch runs alongside the downloads.
Judgement call: a failed ML fetch is recorded per file in failures.json, which is how the exit code goes non-zero without changing src/cli-commands.ts.

Model: opus-5-5
This commit is contained in:
2026-10-06 00:49:14 +00:00
parent 14ac7d05ef
commit b886943a6d
6 changed files with 214 additions and 35 deletions
+17 -9
View File
@@ -599,8 +599,8 @@ the smallest does not.
per unique file, two for a live photo: see
below)
YYYY-MM-DD.<fileID>.json the file's basic metadata fields quak
keeps, and its private and public magic
metadata
keeps, its private and public magic
metadata, and its ML data
YYYY-MM-DD.<fileID>.livephoto.json
which of a live photo's two files is which
collections/
@@ -619,6 +619,14 @@ name as uploaded, case kept, or `.bin` when it has none or it holds anything but
letters and digits. When the date or the time zone changes, the next run saves
the original at its new path and leaves the old copy where it is.
A file's JSON holds Ente's ML data for it (its faces and its CLIP embedding) as
`mlData`, the same payload `backup-metadata` writes; a file Ente has no ML data
for has no `mlData`. The backup waits for the library's ML data fetch to finish
before it writes the JSON files. If that fetch fails, each file whose ML data is
not in the cache gets the reason in `mlDataError` instead and counts as failed,
and the next run fetches it again. The JSON files are rewritten on every run, so
ML data that arrived since the last run appears.
`failures.json` records each failed file with the kind of failure, how many
times it has been tried and when it was last tried. A file leaves it once it
succeeds, or once it is no longer in the library or in the backup's scope. The
@@ -655,8 +663,7 @@ subsequent runs, existing originals are skipped. If a download fails, the error
is logged and the backup continues with the next file. The exit code is non-zero
if any files failed. `quak backup` opens its library with the thumbnail and
originals precache off, so the only file content it fetches is the originals the
backup stores. The library's ML data fetch still runs and fills the cache's
`mldata/`.
backup stores.
Each original is written to a temporary file in the same directory, synced to
disk, and renamed into place, so an original is either complete or absent, even
@@ -876,11 +883,12 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
- `await lib.backup(opts?)` → `BackupResult`. It waits for a refresh as
`fresh()` does, puts every in-scope original not already at its save path
there as `photo.download()` does (and, with `includeThumbnails`, fetches
thumbnails) through the content cache, and rebuilds the on-disk backup tree
with a durable failure ledger. A fetched original is written straight to its
save path and not into the cache, which then counts it as present; one the
cache already held is copied from there. `BackupOptions`: `downloadDirectory`
(falls back to the library's), `includeOriginals` (default `true`),
thumbnails) through the content cache, waits for an ML data fetch, and
rebuilds the on-disk backup tree, each file's JSON with its ML data, with a
durable failure ledger. A fetched original is written straight to its save
path and not into the cache, which then counts it as present; one the cache
already held is copied from there. `BackupOptions`: `downloadDirectory` (falls
back to the library's), `includeOriginals` (default `true`),
`includeThumbnails` (default `false`), `onlyAlbumNames`, and `onProgress`. See
Backup layout above for the tree it writes.