sneak/quak
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Save path ./photos/YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.fileID.ext; download() from the cache first (closes #143)
check / check (push) Successful in 1m25s
Details

Browse Source 
Originals are saved at `{downloadDirectory}/YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.{fileID}{ext}`. The date is the photo's `takenAt` in local time, and `downloadDirectory` defaults to `./photos`, resolved when the library opens. The old `originals/` layout is gone.

`photo.download()` writes the original to `savePath`. It copies from the cache when the cache holds the original, and fetches otherwise. `lib.backup()` uses the same path and rule, and every album in `collections/` links to it. `isLocal` is true only when the original is at `savePath`.

For a file in several albums, one rule picks the copy everything uses: the most recently synced, with the lowest album ID breaking a tie.

Model: opus-5-5
This commit was merged in pull request #146.
This commit is contained in:
clawbot
2026-10-01 23:20:58 +02:00
parent 67d554fb46
commit 10e1a9ef39
16 changed files with 975 additions and 499 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+89 -68
View File
@@ -528,25 +528,36 @@ the smallest does not.
```
<dir>/
originals/
<fileID>.<ext> actual file content (one per unique file,
two for a live photo: see below)
<fileID>.json the file's basic metadata fields quak
keeps, and its private and public magic
metadata
<fileID>.livephoto.json which of a live photo's two files is which
YYYY/YYYY-MM/YYYY-MM-DD/
YYYY-MM-DD.<fileID>.<ext> actual file content, at its save path (one
per unique file, two for a live photo: see
below)
YYYY-MM-DD.<fileID>.json the file's basic metadata fields quak
keeps, and its private and public magic
metadata
YYYY-MM-DD.<fileID>.livephoto.json
which of a live photo's two files is which
collections/
<name>/
<title> -> ../../originals/<fileID>.<ext> (symlink)
<name>.json collection metadata + file list
failures.json files that failed and have not yet succeeded
<title> -> ../../YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>
(symlink)
<name>.json collection metadata + file list
failures.json files that failed and have not yet succeeded
```
Each original is saved at its save path, the same path `photo.savePath` gives
and `photo.download()` writes (see Read surface below). The date is the photo's
`takenAt` (the date set in Ente if it was edited, else its creation time) in the
time zone of the machine running quak. The extension is the one in the file's
name as uploaded, case kept, or `.bin` when it has none or it holds anything but
letters and digits. When the date or the time zone changes, the next run saves
the original at its new path and leaves the old copy where it is.
`failures.json` records each failed file with the kind of failure, how many
times it has been tried and when it was last tried. A file leaves it once it
succeeds, or once it is no longer in the library or in the backup's scope. The
library's `lib.backup({ includeThumbnails: true })` also writes
`thumbnails/<fileID>.jpg` beside `originals/`; `quak backup` does not.
`thumbnails/<fileID>.jpg` beside `collections/`; `quak backup` does not.
A collection's directory and JSON are named after the collection, and a symlink
after the file's title, both with unsafe characters replaced. When two
@@ -558,22 +569,21 @@ name stays the same from run to run until such a clash appears or goes away.
A live photo, which Ente stores as one ZIP of its image and its video, is stored
as those two files, which a photo viewer can open: each is
`originals/<fileID>.<ext>` with the extension it has inside the ZIP (for example
`12345.heic` and `12345.mov`), and `<fileID>.livephoto.json` names the two. The
live photo counts as stored only when both files are present and not empty. Its
album folder links both, each named after the title with that file's extension
(`IMG_0001.heic` and `IMG_0001.mov`). A live photo that an earlier version of
quak stored as the ZIP, under the image's name, is replaced by its two files on
the next run, and the ZIP and its link are removed.
`YYYY-MM-DD.<fileID>.<ext>` with the extension it has inside the ZIP (for
example `2026-03-01.12345.heic` and `2026-03-01.12345.mov`), and
`YYYY-MM-DD.<fileID>.livephoto.json` names the two. The live photo counts as
stored only when both files are present and not empty. Its album folder links
both, each named after the title with that file's extension (`IMG_0001.heic` and
`IMG_0001.mov`).
Each run removes the symlinks into `originals/` that no longer belong in their
collection's directory, and the directories (and JSON) of collections that were
deleted or renamed. Nothing else in `collections/` is touched: a file or a
Each run removes the symlinks into the date folders that no longer belong in
their collection's directory, and the directories (and JSON) of collections that
were deleted or renamed. Nothing else in `collections/` is touched: a file or a
symlink you put there stays, and a directory that still holds one after its
symlinks are removed stays too, with its JSON.
Each file is downloaded exactly once regardless of how many collections it
appears in, and written once: straight into `originals/`, with no copy left in
appears in, and written once: straight to its save path, with no copy left in
the cache. An original the cache already held is copied from there instead. On
subsequent runs, existing originals are skipped. If a download fails, the error
is logged and the backup continues with the next file. The exit code is non-zero
@@ -586,14 +596,15 @@ Each original is written to a temporary file in the same directory, synced to
disk, and renamed into place, so an original is either complete or absent, even
after a power cut. A downloaded original's temporary file is named
`.quak-<pid>-<random>.tmp`, one copied from the cache
`.quak-backup-<fileID>.<ext>-<pid>-<random>.tmp`. A run that is killed can leave
one of these temporary files behind; the next backup deletes those whose process
is no longer running. The content cache uses the same scheme, and opening a
library deletes the temporary files in the cache whose process is no longer
running, so a download another process has in progress in the same cache is left
alone. The rename replaces whatever was at the destination rather than writing
through it: a symlink there is replaced, not followed, and the new file has the
temporary file's permissions, not those of the file it replaced.
`.quak-backup-YYYY-MM-DD.<fileID>.<ext>-<pid>-<random>.tmp`. A run that is
killed can leave one of these temporary files behind; the next backup deletes
those whose process is no longer running. The content cache uses the same
scheme, and opening a library deletes the temporary files in the cache whose
process is no longer running, so a download another process has in progress in
the same cache is left alone. The rename replaces whatever was at the
destination rather than writing through it: a symlink there is replaced, not
followed, and the new file has the temporary file's permissions, not those of
the file it replaced.
## TODO
@@ -635,21 +646,24 @@ background, so an unreachable server does not block opening.
`LibraryOptions`:
| Option | Default | Meaning |
| ------------------------ | -------------------------------------------------- | --------------------------------------------------------------------- |
| `client` | required | the account client (a `Client`, or any `LibraryClient`) |
| `cacheDirectory` | `quak/<userID>` under the per-user cache directory | where `metadata.json` and the content cache live |
| `downloadDirectory` | none | backup destination; an original already stored there counts as cached |
| `refreshIntervalSeconds` | `3` | background refresh cadence |
| `precacheThumbnails` | `true` | prefetch every thumbnail, newest first |
| `precacheOriginals` | `true` | prefetch the favorites album and the latest-window originals |
| `precacheOriginalsDays` | `7` | length in days of that latest window |
| `cacheOriginalsMaxBytes` | 100 GiB | size limit on the originals cache; pinned originals can exceed it |
| `freeBelowBytes` | 50 GiB | free space to protect on the volume; the effective limit adapts down |
| `isOriginalPinned` | none | extra predicate for originals that must never be evicted |
| `pools` | fresh `RequestPools` | the bounded request pools (sets concurrency) |
| `onProgress` | none | refresh/ML/precache progress callback (`RefreshEvent`) |
| `contentSource` | the client's own | override the byte source (mainly for tests) |
| Option | Default | Meaning |
| ------------------------ | -------------------------------------------------- | -------------------------------------------------------------------- |
| `client` | required | the account client (a `Client`, or any `LibraryClient`) |
| `cacheDirectory` | `quak/<userID>` under the per-user cache directory | where `metadata.json` and the content cache live |
| `downloadDirectory` | `photos` in the working directory | root of the save paths; an original stored there counts as cached |
| `refreshIntervalSeconds` | `3` | background refresh cadence |
| `precacheThumbnails` | `true` | prefetch every thumbnail, newest first |
| `precacheOriginals` | `true` | prefetch the favorites album and the latest-window originals |
| `precacheOriginalsDays` | `7` | length in days of that latest window |
| `cacheOriginalsMaxBytes` | 100 GiB | size limit on the originals cache; pinned originals can exceed it |
| `freeBelowBytes` | 50 GiB | free space to protect on the volume; the effective limit adapts down |
| `isOriginalPinned` | none | extra predicate for originals that must never be evicted |
| `pools` | fresh `RequestPools` | the bounded request pools (sets concurrency) |
| `onProgress` | none | refresh/ML/precache progress callback (`RefreshEvent`) |
| `contentSource` | the client's own | override the byte source (mainly for tests) |
The default `downloadDirectory` is resolved against the working directory once,
when the library opens; `lib.downloadDirectory` holds the result.
Concurrency is set through `pools`: construct
`new RequestPools({ metadataConcurrency, contentConcurrency, thumbnailConcurrency })`
@@ -702,20 +716,26 @@ and `originalPath`, and `photo.year`, the local-time year of `takenAt`, all as
synchronous getters read from RAM; `photo.record()` → `PhotoRecord`. Two more
synchronous getters look at the disk and never touch the network:
- `photo.savePath` → `string | undefined` — where `lib.backup()` writes the
original, `originals/<fileID>.<ext>` under the `downloadDirectory` the library
was opened with, whether or not it is there yet: for a live photo already
stored, its image. For a live photo not yet stored, it carries the title's
extension, and the backup may store the image under a different one, found
inside the live photo. `undefined` when the library has no `downloadDirectory`
or no content source.
- `photo.isLocal` → `boolean` — whether the whole original is at `savePath`.
- `photo.savePath` → `string` — where `photo.download()` and `lib.backup()` put
the original, whether or not it is there yet:
`YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>` under the library's
`downloadDirectory` (see Backup layout above for the date and the extension).
For a live photo already stored, its image. For a live photo not yet stored,
it carries the title's extension, and the image may be stored under a
different one, found inside the live photo. It needs no content source.
- `photo.isLocal` → `boolean` — whether the whole original is at `savePath`. A
copy only in the cache does not count.
Four async methods may download:
Five async methods may download:
- `await photo.original(opts?)` → `{ path, bytes, videoPath? }` — the
full-resolution file. For a live photo, `path` and `bytes` are its image's and
`videoPath` is its video.
- `await photo.download()` → `{ path, bytes, videoPath? }`, as `original()` —
puts the original at `savePath`, creating its folders. When it is already
there, nothing is written. When the cache holds it, it is copied from the
cache; otherwise it is fetched straight to `savePath`, with no copy left in
the cache. Afterwards `isLocal` is true.
- `await photo.thumbnail(opts?)` → `{ path, bytes }`.
- `await photo.content(opts?)` → `Uint8Array` — the original's bytes, read
through `original()`; for a live photo, its image's.
@@ -731,10 +751,11 @@ Four async methods may download:
They serve from the on-disk content cache when the bytes are present and
otherwise fetch through the pools; `original()`, `content()` and `exif()` also
serve an original the backup has already stored. `opts.onProgress` reports
serve an original already stored at its save path. `opts.onProgress` reports
per-file progress. They throw when the library was opened without a content
source. An original that `content()` or `exif()` downloads lands in the cache,
which does not make `isLocal` true; only `lib.backup()` does.
source. An original that `original()`, `content()` or `exif()` downloads lands
in the cache, which does not make `isLocal` true; only `download()` and
`lib.backup()` do.
Lower-level accessors that return decrypted model objects (which hold key
material) are also available: `listCollections()`, `getCollection(id)`,
@@ -777,15 +798,15 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes
a query vector the caller produced elsewhere.
- `await lib.backup(opts?)` → `BackupResult`. It waits for a refresh as
`fresh()` does, fetches every in-scope original not already in the backup
(and, with `includeThumbnails`, thumbnails) through the content cache, and
rebuilds the on-disk backup tree with a durable failure ledger. A fetched
original is written straight into the backup's `originals/` and not into the
cache, which then counts it as present; one the cache already held is copied
from there. `BackupOptions`: `downloadDirectory` (falls back to the one
`open()` was given), `includeOriginals` (default `true`), `includeThumbnails`
(default `false`), `onlyAlbumNames`, and `onProgress`. See Backup layout above
for the tree it writes.
`fresh()` does, puts every in-scope original not already at its save path
there as `photo.download()` does (and, with `includeThumbnails`, fetches
thumbnails) through the content cache, and rebuilds the on-disk backup tree
with a durable failure ledger. A fetched original is written straight to its
save path and not into the cache, which then counts it as present; one the
cache already held is copied from there. `BackupOptions`: `downloadDirectory`
(falls back to the library's), `includeOriginals` (default `true`),
`includeThumbnails` (default `false`), `onlyAlbumNames`, and `onProgress`. See
Backup layout above for the tree it writes.
### Request pools
@@ -816,7 +837,7 @@ When `metadata.json` belongs to a different account than the client's,
originals and thumbnails are kept; they are reached only through the files the
current account's records name.
A live photo's original is cached as in the backup: its image and its video,
A live photo's original is cached as at its save path: its image and its video,
each `originals/<fileID>.<ext>` with its own extension, and
`originals/<fileID>.livephoto.json` naming them; the two are evicted together. A
live photo that an earlier version cached as its ZIP is not served: the library
@@ -840,7 +861,7 @@ from a very old client, is stored unchecked.
- `src/library/index.ts`: `Library`, `LibraryOptions`, `LibraryStatus`,
`LibraryClient`, `RefreshEvent`
- `src/library/read.ts`: `Album`, `Photo`, `AlbumsAPI`, `PhotosAPI`,
`TimelineAPI`, `PhotoFilter`, `TimelineGroup`, `GroupBy`
`TimelineAPI`, `PhotoFilter`, `TimelineGroup`, `GroupBy`, `SavePathLookup`
- `src/library/content.ts`: `ContentResult`, `ContentOptions`, `ThumbnailsAPI`,
`EnsureOptions`, `EnsureResult`, `ContentSource`
- `src/library/records.ts`: `PhotoRecord`, `AlbumRecord`, `LibrarySnapshot`,