downloadFile, shared by quak get, the content cache and backup, hashes
the decrypted bytes (unkeyed BLAKE2b-512, standard base64) and stores
nothing on a mismatch, failing with an error naming the file ID. A live
photo ZIP is unpacked as it streams with fflate's Unzip, in small
slices so memory stays bounded however far an entry expands, and its
image and video hashed separately as <imageHash>:<videoHash>.
decryptFile reads the older imageHash/videoHash fields for live
photos. A file with no recorded hash is stored unchecked.
Model: opus-5-5
When metadata.json in the cache directory was written for a different,
non-zero user ID than the client's, Library.open deletes it and mldata/
and loads an empty store, so the first refresh enumerates from 0 and none
of the other account's collections, files, keys or ML results are served.
Only reachable with --cache-dir or an explicit cacheDirectory.
Model: opus-5-5
Each ML data request of up to 200 files is now tried on its own. A request
that still fails after its retries is logged, its files are written with the
reason in `mlDataError`, and the command exits 1 once the dump is complete.
`fetchMLData`, used only here, is removed in favour of a per-batch loop over
`fetchMLDataBatch`.
Model: opus-5-5
src/index.ts imports package.json for VERSION and bin/quak.ts passes
VERSION to commander, so package.json is the only place the version is
written. tsc copies package.json to dist/package.json, so the import
resolves from the built output; script/build runs the built CLI with
--version to prove it. A test checks VERSION and quak --version against
package.json.
Model: opus-5-5
The close() test that holds a precache fetch open now runs twice: once
with only the thumbnail fill and once with only the originals fill, so
removing either wait from Precache.close() fails a test.
Model: opus-5-5
Library.close() now returns a promise that resolves once the work it
started has finished: an in-flight refresh with its cache write, the ML
data fetch, and running precache sweeps. The interval test could see a
refresh's new state, close, and remove the directory while the write was
still running. Every library test and CLI command now awaits close(), and
tests hold each of the three writes open to prove close() waits for it.
The precache test waited for its stub source to be called, but the cache
records a file only after checking it on disk, so status() could lag. It
now waits for both fills to report "done".
Model: opus-5-5
The download idle deadline's timer is unref'd so it never holds the
process open, and a test checks no timer is left after a download
completes or fails. A test covers the rejection of "#" in a request
path. The EXIF scan accepts an APP1 segment only when its length is at
least 8, since a shorter one cannot hold the six-byte Exif header;
tests cover lengths 7 and 8.
Model: opus-5-5
downloadTimeoutMs now aborts a file or thumbnail download only when no
bytes have arrived for that long (default 60 s, was a 600 s cap on the
whole transfer), so a slow download that keeps making progress completes.
The abort reason is still a TimeoutError, so retry classification is
unchanged. A download that fails before reading the whole response body
cancels it, including when the temp file cannot be opened or the header
is malformed, so a failed file no longer holds its connection.
Model: opus-5-5
getJSON built its URL with new URL and then overwrote the host and
path, which dropped a base path in a self-hosted apiOrigin; postJSON,
putJSON and the file and thumbnail download URLs joined strings. All of
them now go through one function next to ApiClient that accepts a path
with or without a leading slash and an origin with or without a
trailing slash or base path, and percent-encodes query parameters. A
path containing "?" or "#" is rejected.
Model: opus-5-5
The command bodies in bin/quak.ts become functions in
src/cli-commands.ts that take their options and a context (output
streams, session directory, cache directory, session loader) and return
an exit code. bin/quak.ts wires them to commander and exits with that
code once stdout and stderr have drained; nothing below it calls
process.exit. test/cli/commands.test.ts drives the commands with a fake
client and temp directories. Output is unchanged.
Model: opus-5-5
The backup copy now fsyncs its temp file before the rename and the
directory after it, using the download writer's new fsyncPath helper.
Each backup run deletes .quak-backup-*.tmp files whose process is no
longer running, leaving those of a concurrent backup alone. The rename
sites and the README backup layout state that a symlink at the
destination is replaced and the new file takes the temp file's
permissions, and the README names the temp files. Adds tests for a
missing and an unwritable destination directory for downloadFile and
downloadThumbnail.
Model: opus-5-5
A POST or PUT is replayed only when every errno in the cause chain is a
connect errno and the walk reached the end of the chain, and
postJSON/putJSON no longer follow redirects, so a redirect is an
ApiError that is not retried. getRetryOptions() returns a copy. New
tests pin every errno the classifier names, the cause-chain depth
limit, a chain deeper than the limit, a two-error cycle, and a fresh
deadline per attempt for every retrying entry point. The README
endpoint list is now the one place naming the requests the replay rule
covers; code comments point to it.
Model: opus-5-5
The segment scan behind `backup-metadata --exif` now checks every
segment length against the bytes that remain and stops on lengths
under 2, so a truncated or corrupt original can neither throw nor loop.
A malformed or unparseable EXIF segment is recorded as
`imageMetadata.exifError`, and a failure to read the original as
`imageMetadataError` in the per-file JSON, instead of the field being
silently left out. Tests use short hand-built byte arrays.
Model: opus-5-5
vitest does not read .gitignore when finding tests, so a checkout nested
under .claude/ had its whole test/ tree run as part of this suite.
vitest.config.ts adds .claude/** to vitest's default excludes. The new
packaging test plants a nested checkout in a temp directory and fails if
vitest, run with this config, would collect it.
Model: opus-5-5
The package is an empty stub with no declarations; libsodium-wrappers-sumo
ships its own types. Removed with yarn remove, which regenerated yarn.lock.
Model: opus-5-5
Client.fromJSON checks every snapshot field and each key's decoded length
and throws an error naming the bad field. toJSON reads the token through a
new ApiClient.getAuthToken and throws when there is none. logout zeroes the
key buffers in place; collectionsSince re-checks for logout after its
request so it never decrypts with zeroed keys. The CLI now reports a
corrupt session file separately from a missing one.
Model: opus-5-5
A file title or album name decrypted from server data could name a path
outside the chosen directory (`../../.ssh/authorized_keys`). One module,
src/filename.ts, now makes such names safe for `quak get`/`get-thumb`
without `--out`, downloadFile/downloadThumbnail without outPath, and the
backup and metadata backup trees. Originals-cache extensions are limited to
letters and digits. A user-supplied path is still used as is. decryptFile
reads a missing or non-string title as "" and rejects metadata that is not
a JSON object.
Model: opus-5-5
Rewrites the README API reference to match the shipped library surface: Library.open options (XDG cache dir + userID, downloadDirectory, refreshIntervalSeconds, precache and cache-size options), default vs fresh reads (Library.fresh()), snapshot/subscribe, albums/photos/timeline, Photo.original/thumbnail, thumbnails.ensure, mldata search, backup, the request pools and on-disk layout, and the CLI (--cache-dir, fresh reads). Every documented signature verified against the code. Notes the deferred content-hash integrity check (#68). Also issue #13.
Model: opus-4-8
Ports the CLI to the library API. collections/files/get/get-thumb use the fresh read variants (Library.fresh(), current server state); backup runs lib.backup; backup-metadata and the missing-thumbnail helpers enumerate via the library. files/get output is byte-identical to the pre-port CLI — raw metadata.title, microsecond creationTime, pre-port row order — exit codes unchanged. Adds --cache-dir; fixes the helper JPEG-only assumption (closes#17).
Model: opus-4-8
Adds Library.fresh(): forces a refresh, awaits its completion and persist, then returns the albums/photos/timeline read namespaces now reflecting a completed server round trip — the caller awaits and is guaranteed current at resolve. Default reads and the background loop are unchanged (immediate-from-cache). Concurrent fresh reads coalesce to one in-flight refresh; a fresh read whose refresh fails rejects rather than answering stale. The CLI adopts fresh reads separately (#52).
Model: opus-4-8
Precaches aggressively inside open(): every thumbnail (newest first, through the shared thumbnail pool, never evicted; visible/ahead requests preempt the background fill) and the pinned originals — the favorites album plus every file within precacheOriginalsDays (default 7) of the newest takenAt — through the content pool. The pinned set integrates with the #47 eviction hook; when the window moves or a favorite is removed, files become ordinary evictable originals. open() options precacheThumbnails/precacheOriginals/precacheOriginalsDays; progress via onProgress/status().
Model: opus-4-8
Adds lib.mldata search over the CLIP index (#49): forFile returns a file's stored payload; similar ranks nearest files by cosine on the CLIP embedding; searchByEmbedding ranks the index against a caller-supplied query vector. All RAM-only, reusing the packed Float32Array index and id list. No text encoder is bundled — the caller provides the query embedding. (Redo of the reverted first attempt, now tsc-clean.)
Model: opus-4-8
Rewrites backup on the library API. backup() refreshes, fetches pending originals (and optionally thumbnails) through the pools reusing the content cache, then materialises the unchanged collections/ symlink views + per-collection JSON + sidecars from the model. A symlink failure no longer aborts the run (closes#8). failures.json reconciles against each run's attempted set — since-deleted/out-of-scope/resolved entries clear, still-failing retained, one attempt per file per run — and the exit-code contract is preserved.
Model: opus-4-8
Bounds the originals cache and evicts least-recently-used. cacheOriginalsMaxBytes default 100 GiB; the effective limit adapts down via fs.statfs to keep freeBelowBytes (default 50 GiB) free. Over-limit writes evict unpinned originals oldest-mtime-first (mtime touched on read); pinned files (favorites + latest week) are skipped and an over-budget on-demand fetch proceeds over-limit. Every in-flight write is excluded from eviction, so concurrent fetches never delete each other's just-stored file. Only cacheDirectory/originals is evicted; downloadDirectory and thumbnails never.
Model: opus-4-8
Adds the on-disk content and thumbnail cache keyed by fileID: originals/ and thumbnails/ under cacheDirectory, present-means-complete (streaming atomic rename), orphan temp reaping on open. Photo.original/thumbnail return a cached path with no network when present, else fetch through the shared request pool; thumbnails.ensure drives the thumbnail pool with priority, dedup and AbortSignal. One shared RequestPools set serves both the ML fetch and the content cache. Content-hash integrity is deferred (#68); authenticated streaming decrypt guarantees integrity now.
Model: opus-4-8
Reverts the #50 merge (224bd101): mlsearch.ts failed tsc in the CI build (make check does not run make build, so it slipped past review). next restored to green; #50 to be redone with make build in its gate.
Model: opus-4-8
Adds lib.mldata search over the CLIP index (#49): forFile returns a file's stored payload; similar ranks nearest files by cosine on the CLIP embedding; searchByEmbedding ranks the index against a caller-supplied query vector. All RAM-only, reusing the packed Float32Array index and id list. No text encoder is bundled — the caller provides the query embedding.
Model: opus-4-8
Adds the machine-learning (magic) data layer: fetches per-file ML payloads (face detections + CLIP embeddings) via the existing metadata-backup fetch through the metadata pool after each refresh, decrypts and gunzips them, and stores one mldata/<fileID>.json per file by rename (present-means-complete). A derived index (mldata/clip.f32 + clip.json) loads in one read and is rebuilt whenever it disagrees with the payloads on disk in either direction, so an interrupted backfill self-heals. Never in metadata.json; incremental on later refreshes; progress via onProgress/status.
Model: opus-4-8
Adds the in-process read surface, served from RAM with args-object signatures: albums (list/byName/byID), photos (byID/records -> plain PhotoRecord[]), thin Album/Photo wrappers, and timeline.groups (day/week/month) with a PhotoFilter (albumID/text/fileTypes/hasLocation/includeArchived; hidden excluded). Week keys use ISO YYYY-Www; each file appears once per group, newest first. Built on the #43 snapshot projection; no network.
Model: opus-4-8
Adds three independent bounded request pools — metadata (10 in flight), content (5), thumbnails (25), each overridable — with on-demand-before-background priority and in-flight dedup (a shared key runs once); an idle pool never lends slots, and retries run inside a slot. Self-contained module; the content cache wires it into the library later.
Model: opus-4-8
Adds a synchronous, key-free snapshot() returning LibrarySnapshot (one PhotoRecord per fileID, deduped, newest first) with edited-name/time precedence (pubMagicMetadata over basic metadata) in milliseconds, plus AlbumRecord (favorites identified by type). subscribe({onChange}) delivers LibraryChange (changed/removed albums and files, refreshedAt) only when a refresh changes something; unsubscribe stops delivery. Built on the existing refresh loop; served from RAM, safe to send over IPC.
Model: opus-4-8
streamDecrypt no longer recopies the whole accumulation buffer on every network read. Reads are queued with a running byte count and a contiguous buffer is materialised only at each ENC_CHUNK_SIZE boundary, with a straddling read split via a subarray view, so each byte is copied once instead of O(n^2). TAG_FINAL truncation detection, final-chunk handling, retry, and the per-chunk progress hook are unchanged. A new test feeds a multi-chunk body through a ReadableStream that yields many small pieces, exercising the fragmented-read path.
Model: opus-4-8
Library.open loads metadata.json and serves reads from RAM: an empty cache awaits the first refresh, an existing cache returns at once and refreshes in the background so an unreachable server never stalls open(). A background timer refreshes every refreshIntervalSeconds (default 3), diffing only changed albums via the resumable cursor+tombstone enumerators and rewriting metadata.json only when something changed. A refresh failure is invisible to reads and surfaced via status()/onProgress; a failed save keeps status().lastError set and retries until one lands, so a stale disk is never masked. No sync()/refresh()/serverReachable surface; status() and close() included.
Model: opus-4-8
Originals no longer buffer the whole decrypted file in RAM: streamDecrypt hands each secretstream chunk to a sink and the download path writes it to the staged temp file, so peak memory is one chunk regardless of file size. The atomic write moved inside the retry loop; only a TAG_FINAL-authenticated attempt renames; truncation leaves no destination file. Does not close#21 (the streamDecrypt accumulation-buffer recopy stays open).
Model: opus-4-8
Adds the metadata.json store: loads whole into RAM with id-lookup Maps, rewrites whole through the exported fsync atomic writer (temp, fsync, rename, dir fsync); a missing, unparseable, or wrong-schema file loads as empty (it is a cache); directory 0700, file 0600. No lock file, no public sync().
Model: opus-4-8
Adds collectionsSince/filesSince taking a starting cursor, returning the resumable max-updationTime cursor and a separate list of tombstoned ids; filesSince throws instead of looping when the server reports hasMore without advancing (closes#7). listCollections/listFiles stay as thin, unaffected wrappers.
Model: opus-4-8
The atomic writer fsyncs the staged temp file before rename and the directory after, and is exported for reuse. downloadFile/downloadThumbnail gain an optional per-chunk onProgress hook (non-decreasing, final equals bytesWritten; no-op when absent). Retry and TAG_FINAL checks unchanged.
Model: opus-4-8
decryptFile now sets file.size/thumbnail.size from raw.info (undefined when absent) and carries an optional isDeleted on EnteFile. Plain EnteFile return, no caller changes; listFiles keeps filtering tombstones. Tests cover the three fields and the size-absent case.
Model: opus-4-8
clawbot