quak backup writes each file's ML data into its JSON (closes #163)
check / check (push) Successful in 2m33s
check / check (push) Successful in 2m33s
lib.backup() now waits for an ML data fetch, joining the one its refresh started or starting one, before it writes the per-file JSON, and each file's JSON carries the cached payload as mlData. When the fetch fails, each file with no cached ML data gets mlDataError and an entry in failures.json, so the result counts it as failed and quak backup exits 1; the next run fetches again. Judgement call: the wait comes after the originals are downloaded, so the fetch runs alongside the downloads. Judgement call: a failed ML fetch is recorded per file in failures.json, which is how the exit code goes non-zero without changing src/cli-commands.ts. Model: opus-5-5
This commit is contained in:
@@ -599,8 +599,8 @@ the smallest does not.
|
||||
per unique file, two for a live photo: see
|
||||
below)
|
||||
YYYY-MM-DD.<fileID>.json the file's basic metadata fields quak
|
||||
keeps, and its private and public magic
|
||||
metadata
|
||||
keeps, its private and public magic
|
||||
metadata, and its ML data
|
||||
YYYY-MM-DD.<fileID>.livephoto.json
|
||||
which of a live photo's two files is which
|
||||
collections/
|
||||
@@ -619,6 +619,14 @@ name as uploaded, case kept, or `.bin` when it has none or it holds anything but
|
||||
letters and digits. When the date or the time zone changes, the next run saves
|
||||
the original at its new path and leaves the old copy where it is.
|
||||
|
||||
A file's JSON holds Ente's ML data for it (its faces and its CLIP embedding) as
|
||||
`mlData`, the same payload `backup-metadata` writes; a file Ente has no ML data
|
||||
for has no `mlData`. The backup waits for the library's ML data fetch to finish
|
||||
before it writes the JSON files. If that fetch fails, each file whose ML data is
|
||||
not in the cache gets the reason in `mlDataError` instead and counts as failed,
|
||||
and the next run fetches it again. The JSON files are rewritten on every run, so
|
||||
ML data that arrived since the last run appears.
|
||||
|
||||
`failures.json` records each failed file with the kind of failure, how many
|
||||
times it has been tried and when it was last tried. A file leaves it once it
|
||||
succeeds, or once it is no longer in the library or in the backup's scope. The
|
||||
@@ -655,8 +663,7 @@ subsequent runs, existing originals are skipped. If a download fails, the error
|
||||
is logged and the backup continues with the next file. The exit code is non-zero
|
||||
if any files failed. `quak backup` opens its library with the thumbnail and
|
||||
originals precache off, so the only file content it fetches is the originals the
|
||||
backup stores. The library's ML data fetch still runs and fills the cache's
|
||||
`mldata/`.
|
||||
backup stores.
|
||||
|
||||
Each original is written to a temporary file in the same directory, synced to
|
||||
disk, and renamed into place, so an original is either complete or absent, even
|
||||
@@ -876,11 +883,12 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
|
||||
- `await lib.backup(opts?)` → `BackupResult`. It waits for a refresh as
|
||||
`fresh()` does, puts every in-scope original not already at its save path
|
||||
there as `photo.download()` does (and, with `includeThumbnails`, fetches
|
||||
thumbnails) through the content cache, and rebuilds the on-disk backup tree
|
||||
with a durable failure ledger. A fetched original is written straight to its
|
||||
save path and not into the cache, which then counts it as present; one the
|
||||
cache already held is copied from there. `BackupOptions`: `downloadDirectory`
|
||||
(falls back to the library's), `includeOriginals` (default `true`),
|
||||
thumbnails) through the content cache, waits for an ML data fetch, and
|
||||
rebuilds the on-disk backup tree, each file's JSON with its ML data, with a
|
||||
durable failure ledger. A fetched original is written straight to its save
|
||||
path and not into the cache, which then counts it as present; one the cache
|
||||
already held is copied from there. `BackupOptions`: `downloadDirectory` (falls
|
||||
back to the library's), `includeOriginals` (default `true`),
|
||||
`includeThumbnails` (default `false`), `onlyAlbumNames`, and `onProgress`. See
|
||||
Backup layout above for the tree it writes.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user