From 34fd5faccda18aeb69a408d3c0d922350a1eac0a Mon Sep 17 00:00:00 2001 From: sneak Date: Wed, 11 Nov 2026 00:15:23 +0000 Subject: [PATCH] TODO.md: open work only (closes #235) Removes the Completed Steps section: dated accounts of finished work, review and merge history, and notes on past decisions. Git keeps the history. The Workflow, Status, Next Step and Future Steps sections are unchanged. Model: opus-5-5 --- TODO.md | 735 -------------------------------------------------------- 1 file changed, 735 deletions(-) diff --git a/TODO.md b/TODO.md index e4da039..a90c5f1 100644 --- a/TODO.md +++ b/TODO.md @@ -24,741 +24,6 @@ only sneak has, so it waits on him. Tagging and releases are decided by sneak alone, and happen only when he declares one. -# Completed Steps - -- 2026-11-10: The test suite passes on a host under heavy load - (https://git.eeqj.de/sneak/quak/issues/233). vitest runs at most four workers - instead of one per CPU core. `test/cli/bin.test.ts` compiles the sources - `bin/quak.ts` imports and starts libsodium once, before its tests run. The - fake accounts of `test/ente-server.ts` and - `test/thumbnails/thumbnails.test.ts` stretch their passwords at libsodium's - lowest Argon2id cost, since every login stretches the password again. - -- 2026-10-26: On an empty cache, `quak helper fix-missing-thumbnails --file` - stops fetching collections' files once it finds every file it names - (https://git.eeqj.de/sneak/quak/issues/223). It used to wait for a refresh, - which on an empty cache fetched the files of every collection. Now it reads - the collection list, then one collection's files at a time, the most recently - updated first. It stores none of the collections or file records it reads. It - fetches each original through `lib.originalOf()`, which keeps it in the cache. - Once the cache holds the account's collections, it still waits for a refresh. - -- 2026-10-26: On an empty cache, `quak get` and `quak get-thumb` stop fetching - collections' files once they find the file - (https://git.eeqj.de/sneak/quak/issues/222). They used to wait for a refresh, - which on an empty cache fetched the files of every collection. Now they read - the collection list, then one collection's files at a time. They store none of - the collections or file records they read, and fetch the original or thumbnail - through the new `lib.originalOf()` and `lib.thumbnailOf()`. Once the cache - holds the account's collections, they still wait for a refresh. - -- 2026-10-26: `quak files` with a collection ID fetches that collection's files - alone (https://git.eeqj.de/sneak/quak/issues/217). It used to wait for a - refresh, which on an empty cache fetched the files of every collection. It now - reads the collection list through `lib.freshCollections()`, then the one - collection's files through the new `lib.freshFiles()`. It stores nothing, and - the files print in the server's order. - -- 2026-10-22: The JSON `quak backup` and `quak backup-metadata` write for a file - holds `keywords` in place of the CLIP embedding in its ML data - (https://git.eeqj.de/sneak/quak/issues/218). The keywords are the titles of - the Discover categories of Ente's photos app that the file falls in, by the - app's rule: the cosine similarity of the file's CLIP embedding and the text - embedding of a category's query is at least the category's minimum score, and - videos stay out of the categories that leave them out. `src/clip-keywords.ts` - applies the rule. `src/clip-keyword-vectors.ts` holds the 16 categories with - their text embeddings. `bin/clip-keyword-vectors` writes it with Ente's CLIP - text model, and writes the CLIP embedding of a public-domain photo of a sunset - that the tests use. - -- 2026-10-18: `-v`/`--verbose` raises the CLI's log level once for each repeat, - and `-q`/`--quiet` prints only fatal errors - (https://git.eeqj.de/sneak/quak/issues/212). The levels are fatal, error, - warning, info, debug and trace; the default is info, everything quak printed - on stderr before, `-v` gives debug and `-vv` trace. `src/cli-log.ts` holds the - level and the log the commands write stderr lines through. Debug adds each - retry, which `withRetry` reports through the new `onRetry`, and the start and - end of each refresh and ML data fetch; trace adds each request, which - `ApiClient` reports through the new `onRequest`. `--json` no longer keeps the - progress lines off stderr, so `backup` and `--json` runs log a refresh's - progress too. Every line that reports a failed file or album is at error, - which the progress callbacks of `backup`, `backup-metadata` and the thumbnail - helpers now say, and a command that exits 1 because files or albums failed - ends with one fatal line saying how many and where to look. - -- 2026-10-18: `quak files` takes the collection ID as an optional argument - instead of the `--collection` option, which is gone - (https://git.eeqj.de/sneak/quak/issues/211). With no argument it lists every - file in the account once, collection by collection; a file in several - collections is listed with the first of them, which is its `collectionID` in - `--json`. The fresh read in `src/cli-read.ts` does this. - -- 2026-10-18: Names quak builds from an album name or a title fit in a file name - (https://git.eeqj.de/sneak/quak/issues/205). `shortenKeepingExtension` in - `src/filename.ts` cuts a name short to a number of bytes, keeping its - extension when it fits and always its first character, and can put text right - before the extension; the backup's `withID` uses it for an ID. - `defaultFileName` there gives the name a file is saved under when the caller - gives no path: a prefix such as `thumb_` and the title with unsafe characters - replaced, cut short with it to 255 bytes, and, for a live photo's image or - video, with that part's own extension before the cut. `quak get`, - `quak get-thumb`, `downloadFile` and `downloadThumbnail` use it without - `--out` or `outPath`. `backup-metadata` cuts an album's name short so that its - folder, `-`, fits in 255 bytes. `safeExtension` gives `.bin` for an - extension of more than 16 letters and digits, so a save path's name always - fits. - -- 2026-10-18: `quak collections` no longer waits for the files of every - collection (issue 210). On an empty cache it opened its library with a refresh - that fetched and decrypted the files of each collection in turn, printing - nothing and storing nothing until all were done, so on a large account it - seemed to hang. It now reads the collection list alone through the new - `lib.freshCollections()`, one request that stores nothing. The read commands - open their library with the new `refreshOnOpen: false`, so each makes only the - refresh it asks for. A refresh sends a `"progress"` event after each - collection whose files it fetched, and the commands that wait on one print it - on stderr, except with `--json`. - -- 2026-10-16: ML data the server sends that does not decrypt, gunzip or parse is - no longer taken for no ML data (issue 198). `fetchMLDataBatch` returns why for - each such file beside the payloads; ML data that parses to something other - than a JSON object, such as `null`, is one of them. The library keeps the - reasons from its last ML data fetch that succeeded in memory, not in the - cache, and gives them in `status().mlDataErrors` and in the `fetchMLData` - event with status `"done"`; a fetch still running or one that fails does not - change them. `quak backup` writes the reason as `mlDataError` in the file's - JSON when no ML data is cached for it, and records the file in `failures.json` - with an error starting `ML data: `; `backup-metadata` writes `mlDataError` - too. Both exit 1. Tests serve ML data encrypted under a key that is not the - file's. - -- 2026-10-16: With `--verify`, an original whose hash does not match is no - longer removed before its replacement is downloaded (issue 199). - `replaceOriginal` in `src/library/content.ts` downloads the replacement to a - temporary file beside it, a live photo's image and video each to its own, - hashes it and renames it over the original only when it matches. When the - download fails or does not match either, the original keeps its bytes and - links, and the file goes into `failures.json`. The replacement is always - downloaded, through the new `fetchOriginal` of the content cache, never copied - from the cache. - -- 2026-10-16: The `~/bin/quak` that `make install` builds retries a refused - connection and a name that does not resolve, and the backup records either as - transient (issue 206). Bun's `fetch` reports them with codes of its own, - `ConnectionRefused` and `FailedToOpenSocket`, which `src/retry.ts` now counts - as transport failures. A `POST` or `PUT` is not replayed on them, because bun - also gives them when it resends a request on a new connection after a - connection it kept open closed, and the server may already have the request. - Bun already gave `ECONNRESET` for a reset and a `TimeoutError` for a deadline. - `classify` in `src/backup.ts` now records any `TimeoutError` as transient, so - a stalled download, whose error says only that it stalled, is no longer - `unknown`. The shapes were found by running the binary against local - listeners. - -- 2026-10-15: An album whose folder or JSON `quak backup` cannot write no longer - ends the run before the other albums and `failures.json` are written (issue - 197). The run prints `FAILED album : `, lists the album's ID in - `failedAlbums` in the result and in `--json`, goes on, and exits 1. An album - whose folder would have the name of another album's JSON, as `x.json` beside - `x` would, gets its ID added. A folder or link name over 255 bytes is cut - short between two characters, by `shortenToBytes` in `src/filename.ts`, and - gets the ID. - -- 2026-10-15: A network failure is printed and recorded with its reason, not as - just `fetch failed` (issue 196). `describeError` in `src/errors.ts` gives an - error's message followed by the message of each error on its `cause` chain, - such as `fetch failed: connect ECONNREFUSED 1.2.3.4:443`, and each address's - reason when a host has several and none could be reached. `run` prints that, - and the backup logs it and records it in `failures.json`, where `classify` now - calls the failure transient when the chain holds an errno that - `hasTransportCode` in `src/retry.ts` recognises, for a failed ML data fetch - too. The walk along the chain moved from `src/retry.ts` to `src/errors.ts`, - which both use. Tests refuse every connection of a backup, then only one - file's download, and fail the ML data fetch. - -- 2026-10-15: The README says that `make install` needs nix (issue 200), in - Getting Started and in "Running the backup from cron": `make build-bin` runs - bun through `nix-shell -p bun`. The `Makefile` is unchanged. - -- 2026-10-14: `lib.backup()`, and so `quak backup`, works on up to the content - pool's limit of originals at once, 5 by default, instead of one at a time - (issue 182). That many workers take files from one shared list, each doing all - the work for one original, its `--verify` hashing included, so no more - originals than that are open at once. `BackupLibrary` says the limit as - `contentConcurrency`. A test holds each download at the fake Ente server to - count those in flight. Once a worker has thrown, as one does when `onProgress` - throws, or the lock is lost, no worker takes another file; the backup fails - once the originals under way are in place, and a lost lock stops it before it - fetches ML data. - -- 2026-10-14: Next Step said the cache design, issue 36, waits on sneak's - review, though that issue closed on 2026-10-03 (issue 194). It now names the - issues open on the tracker. - -- 2026-10-14: A backup whose lock is lost no longer dies with a stack trace - (issue 184). `lockBackupDirectory` gives proper-lockfile an `onCompromised` - that records why the lock was lost instead of throwing from its timer, and - returns the lock, which `quak backup` passes to `lib.backup()` as `lock`. The - backup then stops before it starts another file with - `lost the lock on DIR: REASON`, which `quak backup` prints as one line before - it exits 1. Releasing a lost lock leaves `backup.lock` alone. Tests in - `test/cli/backup-end-to-end.test.ts` remove the lock, or put another backup's - in its place, while a download is held open. - -- 2026-10-14: An album or file quak cannot decrypt, or whose metadata is not - JSON, no longer fails the refresh, so one bad record no longer stops - `quak backup` from storing anything (issue 181). The refresh leaves it out, an - album with its files, and the library keeps it with the reason in - `metadata.json` until the server sends a copy that decrypts or deletes it. - `quak backup` stores every other file, records each such file in - `failures.json` with an error starting `could not be decrypted:`, names each - such album's ID in a printed line, or with `--json` in `undecryptableAlbums`, - and exits 1 while any remain. A file whose copy in another album decrypts is - not recorded: its original is stored from that copy, and the album where it - does not decrypt neither links nor lists it. The read commands answer from - what decrypted. - -- 2026-10-14: `quak backup` no longer asks the server for changes every 3 - seconds while it runs (issue 183). It opens its library with - `refreshIntervalSeconds: Infinity`, which the library now takes as no - background timer, so the backup refreshes only as it starts and no timed - refresh starts another ML data pass during the run. A test holds a backup's - download open while two hours pass and counts its `/collections/v2` requests. - -- 2026-10-14: `test/cli/backup-end-to-end.test.ts` runs `quak backup` as cron - does (issue 186): `backupCommand` and `run`, the real `loadSession` reading a - session saved after logging in, and every request answered by a fake of the - Ente HTTP API serving data encrypted with libsodium. It covers the backup tree - of an account with two albums of its own and one shared with it, a second run - that downloads nothing, a download that succeeds after two 503s, a 404 that - lands in `failures.json` with exit 1, a refresh answered 401 with exit 3, and - `--verify` putting back an original changed on disk. The fake server and the - account it logs in moved from the backup-metadata test into - `test/ente-server.ts`, unchanged. - -- 2026-10-14: `files --collection`, `get`, `get-thumb` and - `helper fix-missing-thumbnails --file` take an ID only when it is written in - decimal digits and is a safe integer (issue 185). Before, `Number()` read - `0x10` as 16, `1e3` as 1000 and an empty string as 0, and `--file` dropped an - ID it could not read without a word. Anything else now prints - `Invalid file ID: ` or `Invalid collection ID: ` and exits 1 before - any request; one bad `--file` ID fails the whole command. `parseID` in - `src/cli-commands.ts` does the check for all four. - -- 2026-10-06: The README says how to run `quak backup` from cron (issue 170): - log in once as the job's user, a crontab with a backup every night and - `--verify` on Sundays appending to a log file, and the `HOME` and - `XDG_DATA_HOME` the job needs to find the saved session. A table gives each - exit code: 0, the backup is complete; 1, files are in `failures.json` or - another error stopped the run; 2, another backup of the directory is running; - 3, there is no usable session. The introduction lists everything the backup - keeps for each file. - -- 2026-10-06: `quak backup --verify` and `lib.backup({ verify: true })` hash - each original already at its save path as the download check does, streamed, a - live photo as `:` (issue 168). One that does not match - the content hash its metadata records is logged, removed (both files of a live - photo) and put back in the same run, downloaded or copied from the cache, and - what is put back is hashed too. One that still does not match, or whose - download fails, goes into `failures.json`. One with no recorded hash is left - alone. The result, `--json` and the summary gain `verified`, `mismatched` and - `unchecked`. Without `--verify` nothing is hashed. - -- 2026-10-06: Two backups of the same directory never run at once (issue 169). - `lib.backup()` takes a lock, `backup.lock` in its download directory, made - with `proper-lockfile`, before its refresh, and removes it when it ends, - whether it succeeds or fails. A second backup of the directory, from another - process or the same one, fails at once with an error naming the directory. - `quak backup` takes the lock before it opens its library, so a refused run - sends no request; it prints the error as one line and exits 2. A lock that has - gone 10 seconds untouched, left by a run that could not remove it, is taken - over by the next run. - -- 2026-10-06: `quak backup` writes each original's EXIF, XMP and dimensions into - the file's JSON as `imageMetadata`, what `backup-metadata --exif` records - (issue 167): for a live photo from its image, for a video nothing, and `{}` - for an original with none of them. A failed read puts the reason in - `imageMetadataError` and fails neither the file nor the run. An original is - read when the run stores it or when its JSON has neither field; otherwise the - field is taken from that JSON. The hand-built JPEGs moved to - `test/exif-jpeg.ts`, beside the HEIC. - -- 2026-10-06: When the server answers HTTP 401 and that ends a command that - loads the saved session, because the server no longer accepts its token, the - command prints one line, - `quak: the saved session is no longer valid; run "quak login"`, and exits 3 - (issue 164). `quak backup` meets that 401 on the refresh it starts with, - before it touches any file. For the same commands, a missing or corrupt - session file keeps its message and also exits 3, so a cron job can tell that - the user must log in again. `quak logout` is unchanged. `run` in - `src/cli-run.ts` recognises the 401, which reaches it unchanged. - -- 2026-10-06: `quak backup` retries a failed request for longer than the other - commands do (issue 165). `src/retry.ts` exports `UNATTENDED_RETRY_OPTIONS` - beside the unchanged default: 10 attempts, a 1 s base delay and a 60 s cap, so - a request that keeps failing waits at most 243 s before it gives up. - `bin/quak.ts` loads the backup's session with them, so its refresh, ML data - and downloads all use them. What is retried and the backoff formula are - unchanged. - -- 2026-10-06: The files this repository copies from `sneak/prompts` are copied - again from its commit `dd4027b` (issue 171). `.gitignore` and `.dockerignore` - keep out more secret files, and this repository's build artifacts follow the - copied content. `script/check` runs `script/fmt-check` again, `script/cibuild` - runs `script/bootstrap` and `script/check` before the image build, - `script/fmt` and `script/fmt-check` find the pinned yarn under nvm, and the - image's last stage marks `/app` safe for git. - -- 2026-10-06: `quak backup` writes the account and album records - `backup-metadata` writes (issue 166): `account.json` with the account's - `email` and `userID`, and in each album's JSON its `ownerID`, `isShared`, - `updationTime` and, when present, its three layers of magic metadata. Each - file's JSON gains its `updationTime`. - -- 2026-10-05: `quak backup` writes each file's ML data (its faces and CLIP - embedding) into the file's JSON as `mlData`, the payload - `lib.mldata.forFile()` returns (issue 163). `lib.backup()` waits for an ML - data fetch before it writes the JSON files. When that fetch fails, each file - whose ML data is not cached gets the reason in `mlDataError` and counts as - failed, and the next run fetches it again. - -- 2026-10-03: The download-albums example test no longer fails on a disk with - under 50 GiB free (issue 160). It opens its libraries with - `freeBelowBytes: 0`, so the free space of the disk it runs on cannot shrink - the cache, evict the original it cached and add a fetch to the count. No other - test's result depends on that free space. The 50 GiB default is unchanged. - -- 2026-10-02: `docker build .` stamps the commit's tag or short commit, not - `dev` (issue 154). `script/build` writes the version `script/version` prints - into `dist/package.json`: the `VERSION` environment variable or build arg when - one is given, otherwise `git describe --tags --always`, otherwise the version - `package.json` declares. `.dockerignore` sends `.git`, and a checkout with - `.git` whose version comes out empty, `dev` or `unknown` fails the build. - `make build-bin` bundles the built `dist/`, so the single binary reports the - same version. - -- 2026-10-02: `photo.exif()` returns every EXIF tag in the file as `ExifTags`, - keyed by tag name, each as exifreader decodes it, not only the thirteen common - fields (issue 156). The embedded thumbnail's tags are under `Thumbnail`, - without the thumbnail image. The thirteen typed methods stay, each picking its - field from the tags `exif()` returns, typed as in `PhotoExif`. The example - script's JSON files now carry every tag. - -- 2026-10-01: The content cache no longer looks for a live photo that an earlier - version cached as one ZIP (issue 151). When the cache opens, a live photo's - file that no JSON file names is now always left alone. - -- 2026-10-01: `examples/download-albums.ts` logs in, opens the library, and for - every album downloads each photo to its save path, writes the photo's record - and EXIF fields to a JSON file beside it, and writes the album's photos to - `albums/.json` (issue 144). The build compiles it to - `dist/examples/`; the README's "Examples" section says how to run it. - -- 2026-10-01: A `Photo` has one async method for each field of `exif()`, named - and typed as in `PhotoExif`: `make()`, `model()`, `lensModel()`, - `dateTimeOriginal()`, `offsetTimeOriginal()`, `exposureTime()`, `fNumber()`, - `iso()`, `focalLength()`, `orientation()`, `gpsLatitude()`, `gpsLongitude()` - and `gpsAltitude()` (issue 148). Each calls `exif()` and returns its one - field, or undefined when the file lacks it. `Photo` implements a type with one - method per `PhotoExif` field, so the build's type check fails when a field has - no method. A test checks, on the JPEG and the HEIC, that each method gives the - same value as `exif()`. - -- 2026-10-01: Each original's save path is - `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD..` under the library's - download directory, which defaults to `photos` in the working directory (issue - 143). The date is the photo's `takenAt` in the machine's time zone. - `photo.savePath` is always a string, with or without a content cache, and - `isLocal` is true only when the original is at its save path. - `photo.download()` puts the original there, copied from the cache when the - cache holds it and fetched otherwise. `lib.backup()` does the same for each - file, writes each file's JSON beside its original, and links `collections/` to - the save paths. - -- 2026-10-01: `photo.exif()` and `backup-metadata --exif` read EXIF from - HEIC/HEIF originals, a live photo's HEIC image included, as well as JPEG and - the other image formats `exifreader` reads (issue 145). `exifreader` replaces - `exif-reader` and the JPEG segment scan; `PhotoExif` is unchanged. The `exif` - field of `backup-metadata --exif` is now exifreader's tag output, and - `exifRaw` holds the whole EXIF block it could not read. The tests use a real - HEIC, `test/exif.heic`. - -- 2026-10-01: A `Photo` has `savePath`, `isLocal`, `content()`, `exif()`, - `modifiedAt`, `hash` and `year` (issue 141). `savePath` is where - `lib.backup()` writes the original under the library's download directory; for - a live photo not yet stored, it carries the title's extension, and the backup - may store the image under a different one. `isLocal` says whether the whole - original is there. Both look only at the disk. `content()` returns the - original's bytes and `exif()` the common EXIF fields of a JPEG; both may - download the original, and `exif()` downloads no video. `PhotoRecord` gains - `modifiedAt` and `hash`, and the JPEG EXIF scan moved to `src/exif.ts`. - -- 2026-09-29: The "Workflow" list at the top of this file now says to branch - from `next` and open a pull request that targets `next`, that the repository - manager squash-merges a reviewed pull request into `next`, and that only sneak - merges `next` into `main`, as the README does (issue 137). - -- 2026-09-29: The README's "Development workflow" and "For LLMs" sections now - say work branches from `next`, every pull request targets `next`, the - repository manager squash-merges reviewed pull requests into `next`, and only - sneak merges `next` into `main` (issue 135). - -- 2026-09-29: Brought the README and this file in line with `next` after the - milestone merge (issue 132). The Next Step says no implementation work is open - and the cache design (issue 36) waits on sneak's review. README corrections: - the Getting Started comments say when the library refreshes; most, not all, - Makefile targets call a script; `script/lint` and `script/test` have no host - path, though `yarn test` does; the SRP handshake uses `fast-srp-hap`, outside - `crypto/`, and a thumbnail upload's MD5 uses `node:crypto`; the SRP password - is the first 16 bytes of a 32-byte subkey; the key attributes and token come - after SRP, after the TOTP code SRP may ask for, or after the email OTP that - replaces SRP when the account has email MFA on, and quak cannot answer a - passkey; the auth token is sent as URL-safe base64 with padding; every - `TypeError` is retried; each download attempt writes its own temporary files, - two for a live photo, and only the attempt that completes renames them into - place; `runMetadataBackup` records a failed download in the file's JSON; a - second `client.logout()` does not throw; `quak logout` with no session exits - 0, and names the cache directory only when it knows it; `login`, `whoami` and - `logout` open no library; `--exif` records XMP and, for a JPEG, EXIF, and no - IPTC; which commands take `--json`; a failed ML data request may not have been - retried; the thumbnail fixer is not limited to baseline JPEG; `quak backup` - still fetches ML data; a backup's JSON holds the basic metadata fields quak - keeps and the private and public magic metadata, not every decrypted field, - and the README no longer lists what the magic metadata holds; the default - cache directory is the per-user one, not an XDG path on macOS; pinned - originals can exceed `cacheOriginalsMaxBytes`; which tests cover which - operations; a default read is only as current as the last refresh whose - requests all succeeded; `fresh()` and `lib.backup()` join a refresh already - running; a `Photo` has no `thumbnailPath` or `originalPath`; and `408` and - `429` are retried. - -- 2026-09-28: Tested the live-photo writer's fsyncs (issue 130). A test checks - that the image's and the video's temp files are fsynced before either is - renamed into place, and the directory after both renames, as the `writeAtomic` - test does for one file. - -- 2026-09-23: Live photos are stored as their image and their video (issue 107). - A live photo, which Ente stores as one ZIP, is unpacked as it downloads into - `.` for the image and for the video, each with its extension from - the ZIP, beside `.livephoto.json`, which names the two. Both are - checked against the recorded hash and renamed into place only when both are - complete. A ZIP holding a second image or video, or whose two files come to - more than 20 times its size plus 16 MiB, is refused. The backup and the - content cache count the live photo as stored only with both files, the album - folder links both, `quak get` writes both, and the content result gives the - video as `videoPath`. A ZIP an earlier version stored is replaced on the next - backup run, and removed from the cache when the library opens it. - -- 2026-09-23: Settled the package metadata (issue 6). quak is not published, so - `package.json` is marked `"private": true` and the `files` field is gone. - `engines.node` is `>=22`, the major version `script/bootstrap` and the - `Dockerfile` use. An `exports` map makes `.` and `./package.json` the only - importable paths; `runMetadataBackup`, the thumbnail helpers and their types - stay internal to the CLI. - -- 2026-09-23: Brought the README and this file in line with the tree (issue - 111). The layout lists `src/library/` and the other source files, the backup - layout names `failures.json` and the optional `thumbnails/`, "Opening a - library" says what happens when the first refresh fails, the Testing section - gives the 60-second hard cap and 20-second target for `make test` and names - the 90-second `timeout` in the `Dockerfile` as the backstop for a hung test, - and "Tag v1.0.0" is no longer listed as the next step. - -- 2026-09-23: Tested `quak login` and `backup-metadata --exif` (issue 110). - `loginCommand` takes its login function and its prompts from `CliContext`, and - `bin/quak.ts` passes `Client.login` and the terminal prompts. Tests cover a - login from `QUAK_EMAIL` and `QUAK_PASSWORD` with no prompt, the TOTP prompt, a - failed login, and the saved session's modes, and show that `--exif` and - `--all` each turn on EXIF extraction and that it is off without them. - -- 2026-09-23: `quak backup` writes each original once and no longer fills the - cache (issue 106). An original fetched for a backup is written by the download - writer straight into the backup's `originals/`, and the content cache records - it there instead of keeping its own copy; one the cache already held is still - copied. `quak backup` opens its library with the thumbnail and originals - precache off. - -- 2026-09-23: `backup-metadata`, `helper list-missing-thumbnails` and - `helper fix-missing-thumbnails` refresh before they answer (issue 100). Each - awaits `lib.fresh()` before reading, so a file added since the cache was - written is included, and a failed refresh prints one line and exits 1 instead - of answering from a stale or empty cache. The README lists them among the - commands that refresh first. - -- 2026-09-23: `quak backup` waits for the server refresh and fails when it fails - (issue 99). `lib.backup()` joins a refresh already running or starts one, as - `fresh()` does, and rejects before touching any file when it fails, leaving - `failures.json` as it was, so `quak backup` prints the error as one line and - exits 1 instead of backing up the previous run's file list, or nothing, and - exiting 0. - -- 2026-09-23: CLI errors print a message instead of a stack trace (issue 102). - An error a command throws is printed as one `quak: MESSAGE` line on stderr and - the CLI exits 1 once output has drained. The wrapper that does this moved from - `bin/quak.ts` to `src/cli-run.ts`, and `bin/quak.ts` now awaits - `program.parseAsync()`. - -- 2026-09-23: Opening a library no longer deletes another process's download in - progress (issue 105). The download writer's temp files are named - `.quak--.tmp`, and `removeLeftoverTempFiles`, moved from the - backup into the download module, deletes a `.quak-*.tmp` file only when the - process ID in its name is no longer running. The content cache calls it at - `open()` for `originals/` and `thumbnails/`, the backup as before. - -- 2026-09-23: Re-vendored the lint and test setup from the template (issue 96). - Linting and testing are the `lint` and `test` phases of the `Dockerfile`; - `script/lint` and `script/test` each build one with `--no-cache`, and the last - stage compiles and depends on both, so `script/cibuild` is one build. - `Dockerfile.lint`, `CHECK_EPOCH`, `LINT_EPOCH` and the tests that checked them - are gone; `REPO_POLICIES.md` is re-copied. - -- 2026-09-23: Stopped `helper fix-missing-thumbnails` retrying files the server - always refuses (issue 109). Both thumbnail helpers skip a file another account - owns without fetching it. The fixer skips a file whose recorded thumbnail size - is 0 or unknown before downloading it, and otherwise tries smaller encodings - (720 px quality 50 down to 160 px quality 20) until the encrypted thumbnail is - no larger than that size, skipping the file if none fits. - -- 2026-09-23: Tested the live-photo hash check's error paths (issue 117). Tests - download a live photo whose ZIP names an unknown compression method, one whose - ZIP has no image entry and one with no video entry, and check that nothing is - stored and the error names the file ID; the unreadable one is not retried. - -- 2026-09-23: `quak logout` ends the session on the server (issue 108). It calls - `POST /users/logout` through the new `Client.logoutOnServer()`, then deletes - `session.json` even when that call fails, says so and exits 1. It prints the - account's cache directory and says it still holds decrypted data. The default - cache path is now `defaultCacheDirectory()` in the library, shared with - `Library.open`. - -- 2026-09-23: Fixed the backup's per-collection folders (issue 103). Two files - in one collection with the same title, and two collections with the same name, - each get their ID added to the name (`IMG_0001 (12345).JPG`, `Trip (10)/`), so - none replaces another's symlink or JSON. Each run removes symlinks into - `originals/` for files no longer in the collection, and the folders of deleted - or renamed collections, leaving anything else in `collections/` alone. The - README backup layout states the naming rule. - -- 2026-09-23: Checked downloaded originals against their recorded content hash - (issue 68). `downloadFile`, which `quak get`, the content cache and backup all - use, hashes the decrypted bytes (unkeyed BLAKE2b-512, standard base64) and - stores nothing on a mismatch, failing with an error naming the file ID. A live - photo ZIP is unpacked as it streams with `fflate` and its image and video - hashed separately as `:`. `decryptFile` reads older - clients' `imageHash` and `videoHash` fields for live photos. A file with no - recorded hash is stored unchecked. - -- 2026-09-23: Kept one account's cache from mixing with another's (issue 104). - When `metadata.json` in the cache directory was written for a different, - non-zero user ID than the client's, `Library.open` deletes it and `mldata/` - and starts empty, so the first refresh enumerates from 0. This only happens - with `--cache-dir` or an explicit `cacheDirectory`; the default path already - includes the user ID. A test opens one account's cache as another account. - -- 2026-09-23: `backup-metadata` no longer stops on one failed ML data request - (issue 101). Each request of up to 200 files is tried on its own; a failed one - is logged, its files are written with the reason in `mlDataError`, and the - command exits 1 once the dump is complete. `fetchMLData`, which only this - command used, is gone; the command calls `fetchMLDataBatch` per batch. -- 2026-09-23: Single-sourced the version string (issue 5). `package.json` is the - only place it is written: `src/index.ts` imports it for `VERSION` and - `bin/quak.ts` passes `VERSION` to commander. tsc copies `package.json` to - `dist/package.json`, so the import resolves from the built output too, and - `script/build` runs the built CLI with `--version` to prove it. A test checks - that `VERSION` and `quak --version` both equal the `package.json` version. -- 2026-09-23: Tested that `Library.close()` waits for the originals precache - (issue 93). The test that holds a precache fetch open while `close()` runs now - runs once with only the thumbnail fill and once with only the originals fill, - so dropping either wait from `Precache.close()` fails a test. -- 2026-09-23: Fixed two intermittently failing library tests (issue 90). - `Library.close()` now returns a promise that resolves once an in-flight - refresh (including its cache write), the ML data fetch and running precache - sweeps have finished; the library tests await it, so `afterEach` no longer - removes the cache directory while something is still writing into it. The - precache test waits for both fills to report "done" instead of for its stub - source to be called, which happened before the cache recorded the file. -- 2026-09-23: Pinned three guards reviewers found untested (issue 89). The - download idle deadline's timer is unref'd, so it can never keep the process - alive, and a test checks no timer is left after a download completes or fails. - A test covers the rejection of `#` in a request path. The EXIF scan compares - the `Exif` header only in an APP1 segment of length 8 or more, so it never - reads the next segment's bytes, with a test for a short one. -- 2026-09-23: Made the download deadline an idle deadline (issue 24). - `downloadTimeoutMs` now aborts a file or thumbnail download only after no - bytes have arrived for that long, default 60 seconds, instead of bounding the - whole transfer at 10 minutes, so a slow download that keeps making progress - completes. A download that fails before reading the whole body cancels it, so - a failed file no longer holds its connection. -- 2026-09-23: Every `ApiClient` request URL is now built by one function next to - the class (issue 18), so a self-hosted `apiOrigin` with a base path keeps it - on every request, a path works with or without a leading slash, and query - parameters are percent-encoded. A path containing `?` or `#` is rejected with - an error instead of being silently cut. -- 2026-09-23: Made the CLI testable and tested it (issue 12). The command bodies - moved from `bin/quak.ts` into `src/cli-commands.ts` as functions that take - their options and a context (output streams, session directory, cache - directory, session loader) and return an exit code; `bin/quak.ts` only wires - them to commander and exits with the code once stdout and stderr have drained, - so nothing below it calls `process.exit`. `test/cli/commands.test.ts` drives - them with a fake client: session file modes, logout, the missing and corrupt - session paths, and the output and exit code of `whoami`, `collections`, - `files`, `get`, `get-thumb`, `backup` and `helper list-missing-thumbnails`. -- 2026-09-23: Hardened the backup tree's atomic copy (issue 22). `copyAtomic` - fsyncs its temp file before the rename and the directory after it, through the - download writer's `fsyncPath`; each backup run deletes `.quak-backup-*.tmp` - files whose process is no longer running. The README backup layout names the - temp files and states that the rename replaces a symlink and takes the temp - file's permissions. Added tests for a missing and an unwritable destination - directory for `downloadFile` and `downloadThumbnail`. -- 2026-09-23: Hardened the JPEG EXIF scan behind `backup-metadata --exif` (issue - 11). Every segment length is checked against the remaining bytes and lengths - under 2 stop the scan, so a truncated or corrupt original can neither throw - nor loop. A malformed or unparseable EXIF segment is recorded as - `imageMetadata.exifError`, and a failure to read the original as - `imageMetadataError` in the per-file JSON, instead of the field being left - out. -- 2026-09-22: Hardened the retry classifier (issue 80). A `POST` or `PUT` is - replayed only when every errno in the cause chain is a connect errno, and it - no longer follows redirects. `getRetryOptions()` returns a copy. Tests pin - every errno the classifier names, the cause-chain depth limit, cycle - termination, and a fresh deadline per attempt for every retrying entry point. - The README's endpoint list is the one place that names the requests the replay - rule covers. -- 2026-09-22: Stopped `make test` collecting tests from checkouts nested under - `.claude/` (issue 25). vitest ignores `.gitignore` when finding tests, so a - nested checkout ran the whole suite again; `vitest.config.ts` now adds - `.claude/**` to vitest's default excludes, and - `test/packaging/nested-checkout.test.ts` plants a nested checkout in a temp - directory and fails if vitest would collect it. -- 2026-09-22: Dropped the deprecated `@types/libsodium-wrappers-sumo` stub from - `devDependencies` (issue 27). It shipped no declarations; the types come from - `libsodium-wrappers-sumo` itself. `yarn.lock` regenerated by `yarn remove`. -- 2026-09-22: Hardened the client session lifecycle (issue 10). - `Client.fromJSON` checks every snapshot field and each key's decoded length - and names the bad field; `toJSON` reads the token through - `ApiClient.getAuthToken` and throws when there is none; `logout` zeroes the - key buffers, and `collectionsSince` re-checks for logout after its request so - it never decrypts with zeroed keys. The CLI reports a corrupt session file - separately from a missing one (`src/cli-session.ts`). -- 2026-09-22: Sanitized file names taken from server metadata (issue 9). A new - `src/filename.ts` holds the one sanitizer, used by `quak get`/`get-thumb` - without `--out`, `downloadFile`/`downloadThumbnail` without `outPath`, and the - backup and metadata backup trees; it removes separators, control characters, - leading dots and Windows device names, and falls back to a name built from the - ID for an empty title. Originals-cache extensions are letters and digits only, - else `.bin`. A user-supplied path is used as is. `decryptFile` reads a missing - or non-string title as "" and rejects metadata that is not a JSON object. -- 2026-09-22: Rewrote the README API reference (and the Getting Started / usage - snippets) to match the shipped cache/API library on `next` (issue 53, issue - 13). Documented `Library.open` and its options, the default-read vs `fresh()` - distinction (and that the CLI's read commands are fresh), the record types and - `snapshot()`/`subscribe()`, the `albums`/`photos`/`timeline` read surface, - `Photo` content methods, `thumbnails.ensure`, the `mldata` search surface, - `backup()`, the three request pools (10/5/25), and the on-disk cache layout; - noted the deferred content-hash integrity check (issue 68). Docs-only; no code - changed. -- 2026-09-22: Added resumable, deletion-aware enumeration to `Client` (issue 38, - closes issue 7). `collectionsSince`/`filesSince` take a starting cursor, - decrypt live records, surface tombstoned ids in a separate `deleted` list (a - tombstone has nothing to decrypt, so it is a bare id, not a hollow record), - and return the max `updationTime` seen as the cursor to resume from. - `filesSince` refuses to loop when the diff reports `hasMore` without advancing - the cursor (issue 7). `listCollections`/`listFiles` are now thin wrappers that - enumerate from `sinceTime: 0` and drop deletions, so existing callers are - unaffected. -- 2026-09-22: Carried file size, thumbnail size, and the deletion flag through - `decryptFile` (issue 37, foundation for the cache/API design). Live files now - populate `file.size`/`thumbnail.size` from the server's `info` (left - `undefined` when the server omits it), and `isDeleted` is carried from the - diff row onto `EnteFile`. No caller change: `listFiles` still filters deleted - rows before decrypting. Surfacing a tombstone through decryption belongs to - the enumeration unit (issue 38). -- 2026-08-10: Made `lint-once.test.ts` enforce what its header claims. It walked - `make check` only, so it never read `Dockerfile` — the image CI builds through - `script/cibuild` — and a second `prettier --check .` could be added there with - the suite staying green. The walk now also starts at - `.gitea/workflows/check.yml` and follows its `run:` steps, so the graph under - test is the one CI executes rather than the one someone assumed it executes. - The lockfile assertion was a substring check against the whole of - `script/bootstrap`, which has two install sites and so reported the branch the - containers never take; the two branches are now resolved separately and every - `yarn install` in each is required to be `--frozen-lockfile`. Prettier is - counted per occurrence instead of per line, so two invocations chained with - `&&` no longer read as one, and edges are followed on counted lines instead of - being skipped. Every way for the walk to reach nothing — an unknown target, an - unknown script, a missing file, a node with no commands, an unknown node kind - — is a thrown error rather than a quiet zero. Every assertion in the file was - mutation-tested individually. -- 2026-08-10: Stopped `make check` running `prettier --check .` twice. Since - linting moved into Docker, the duplicate was one container pass and one host - pass of the same check: `script/lint` builds `Dockerfile.lint`, which runs - prettier as a build step, and `script/check` then called `script/fmt-check` as - well. The host call is gone from `script/check` and from `script/precommit`; - the container keeps checking formatting, because a successful - `Dockerfile.lint` build is what CI treats as proof of a clean tree, and it is - also what still fails the pre-commit hook on a badly formatted tree. - `script/fmt-check` survives as a standalone entrypoint, whose verdict cannot - drift from the container's. A test walks the invocation graph from each - entrypoint — through the Makefile shims, the `script/` calls and the - `docker build` — and asserts the prettier count, so the duplication cannot - come back unnoticed. -- 2026-08-10: Moved all linting into Docker. `script/lint` builds a new root - `Dockerfile.lint`, which copies the repo into the digest-pinned node image and - runs eslint and prettier as build steps, so a successful build is a clean - lint; no host lint path remains and `yarn lint` is gone from `package.json`. A - fail-closed `LINT_EPOCH` guard stops Docker serving the linter layers from - cache, which is how a lint build returns success in under a second having - linted nothing. The lint stage inside `Dockerfile` and its `COPY --from=lint` - ordering hack are gone: that image now runs `make test` and `make build` only, - because `script/check` calls `script/lint` and running it in a container would - mean docker inside docker. `script/cibuild` builds the lint image first, then - the test and build image. -- 2026-08-09: Made `make docker` green and policy-conformant. Multi-stage - Dockerfile: a lint stage runs `make fmt-check` and `make lint`, and the check - stage takes a `COPY --from=lint` dependency on it before running `make check` - and `make build`. `CHECK_EPOCH` and a fail-closed guard stop Docker serving - those two layers from cache, which is what let a build report success without - running the suite. `script/projectname` says `quak`, so the image is tagged - `quak`; `script/bootstrap` updates apt lists before installing, so a Debian - base works; `.dockerignore` no longer ships the compiled binary, the caches or - agent worktrees into the build context, and keeps `.gitignore` in it for - prettier. -- 2026-08-09: Fixed the TypeScript build. `rootDir` is the repo root, so `bin/` - compiles alongside `src/` instead of failing with TS6059; output is - `dist/src/` and `dist/bin/`, which is where `main`, `types` and `bin.quak` now - point. `script/build` verifies the declared entrypoints exist after the - compiler runs and makes the CLI executable, the Dockerfile runs `make build` - as well as `make check`, and a `quak` script makes the README's - `yarn quak ` examples work. -- 2026-08-09: Retry policy: no retry on 4xx (except `408` and `429`), - exponential backoff with full jitter on 5xx, transport failures and truncated - transfers, under per-attempt deadlines that cover the response body as well as - the request. Downloads retry request, stream consumption and decryption as one - unit; `postJSON` and `putJSON` are replayed only when the connection was never - established. -- 2026-08-09: Downloads verify the secretstream terminated on `TAG_FINAL` and - write output atomically: a truncated body is rejected instead of landing on - disk as a short file, and plaintext is staged in a sibling temp file and - renamed into place, so a failed download leaves the destination untouched. -- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile - shims, README Entrypoints section -- 2026-06-10: Decrypted collections shared by other users (sealed-box keys); - listCollections drops deleted-collection tombstones. -- 2026-06-10: Login hardening: dual-2FA empty-string fields handled, TOTP - preferred when a passkey is also enrolled, interactive input via - @inquirer/prompts. -- 2026-06-10: Replaced sharp with pure JS (jpeg-js + exif-reader); added - single-binary bun build and make install. -- 2026-06-09: Added backup-metadata command (ML data always included, --exif - opt-in); rewrote README to match the implementation; added thumbnail helper - tests. -- 2026-05-13: Full CLI surface: login, backup with dedup symlink layout, - collections, files, get, get-thumb, thumbnail repair helpers. -- 2026-05-13: Client OO API with literate usage tests; file download and - decryption; all three metadata layers decrypted and persisted; renamed quack - to quak. -- 2026-05-11: SRP login flow (email OTP + TOTP) and ApiClient. - # Future Steps - Future desktop client, separate repo: