quak/TODO.md

Workflow

• branch (from main)
• do the work in Next Step
• move Next Step to the top of Completed Steps
• move the top item of Future Steps into Next Step
• commit (TODO.md changes in the same commit as the work)
• merge to main if the branch is not protected, otherwise open a PR
• push

Status

pre-1.0

Next Step

Tag v1.0.0.

Completed Steps

• 2026-09-23: Opening a library no longer deletes another process's download in progress (issue 105). The download writer's temp files are named .quak-<pid>-<random>.tmp, and removeLeftoverTempFiles, moved from the backup into the download module, deletes a .quak-*.tmp file only when the process ID in its name is no longer running. The content cache calls it at open() for originals/ and thumbnails/, the backup as before.

• 2026-09-23: Re-vendored the lint and test setup from the template (issue 96). Linting and testing are the lint and test phases of the Dockerfile; script/lint and script/test each build one with --no-cache, and the last stage compiles and depends on both, so script/cibuild is one build. Dockerfile.lint, CHECK_EPOCH, LINT_EPOCH and the tests that checked them are gone; REPO_POLICIES.md is re-copied.

• 2026-09-23: Stopped helper fix-missing-thumbnails retrying files the server always refuses (issue 109). Both thumbnail helpers skip a file another account owns without fetching it. The fixer skips a file whose recorded thumbnail size is 0 or unknown before downloading it, and otherwise tries smaller encodings (720 px quality 50 down to 160 px quality 20) until the encrypted thumbnail is no larger than that size, skipping the file if none fits.

• 2026-09-23: Tested the live-photo hash check's error paths (issue 117). Tests download a live photo whose ZIP names an unknown compression method, one whose ZIP has no image entry and one with no video entry, and check that nothing is stored and the error names the file ID; the unreadable one is not retried.

• 2026-09-23: quak logout ends the session on the server (issue 108). It calls POST /users/logout through the new Client.logoutOnServer(), then deletes session.json even when that call fails, says so and exits 1. It prints the account's cache directory and says it still holds decrypted data. The default cache path is now defaultCacheDirectory() in the library, shared with Library.open.

• 2026-09-23: Fixed the backup's per-collection folders (issue 103). Two files in one collection with the same title, and two collections with the same name, each get their ID added to the name (IMG_0001 (12345).JPG, Trip (10)/), so none replaces another's symlink or JSON. Each run removes symlinks into originals/ for files no longer in the collection, and the folders of deleted or renamed collections, leaving anything else in collections/ alone. The README backup layout states the naming rule.

• 2026-09-23: Checked downloaded originals against their recorded content hash (issue 68). downloadFile, which quak get, the content cache and backup all use, hashes the decrypted bytes (unkeyed BLAKE2b-512, standard base64) and stores nothing on a mismatch, failing with an error naming the file ID. A live photo ZIP is unpacked as it streams with fflate and its image and video hashed separately as <imageHash>:<videoHash>. decryptFile reads older clients' imageHash and videoHash fields for live photos. A file with no recorded hash is stored unchecked.

• 2026-09-23: Kept one account's cache from mixing with another's (issue 104). When metadata.json in the cache directory was written for a different, non-zero user ID than the client's, Library.open deletes it and mldata/ and starts empty, so the first refresh enumerates from 0. This only happens with --cache-dir or an explicit cacheDirectory; the default path already includes the user ID. A test opens one account's cache as another account.

• 2026-09-23: backup-metadata no longer stops on one failed ML data request (issue 101). Each request of up to 200 files is tried on its own; a failed one is logged, its files are written with the reason in mlDataError, and the command exits 1 once the dump is complete. fetchMLData, which only this command used, is gone; the command calls fetchMLDataBatch per batch.

• 2026-09-23: Single-sourced the version string (issue 5). package.json is the only place it is written: src/index.ts imports it for VERSION and bin/quak.ts passes VERSION to commander. tsc copies package.json to dist/package.json, so the import resolves from the built output too, and script/build runs the built CLI with --version to prove it. A test checks that VERSION and quak --version both equal the package.json version.

• 2026-09-23: Tested that Library.close() waits for the originals precache (issue 93). The test that holds a precache fetch open while close() runs now runs once with only the thumbnail fill and once with only the originals fill, so dropping either wait from Precache.close() fails a test.

• 2026-09-23: Fixed two intermittently failing library tests (issue 90). Library.close() now returns a promise that resolves once an in-flight refresh (including its cache write), the ML data fetch and running precache sweeps have finished; the library tests await it, so afterEach no longer removes the cache directory while something is still writing into it. The precache test waits for both fills to report "done" instead of for its stub source to be called, which happened before the cache recorded the file.

• 2026-09-23: Pinned three guards reviewers found untested (issue 89). The download idle deadline's timer is unref'd, so it can never keep the process alive, and a test checks no timer is left after a download completes or fails. A test covers the rejection of # in a request path. The EXIF scan compares the Exif header only in an APP1 segment of length 8 or more, so it never reads the next segment's bytes, with a test for a short one.

• 2026-09-23: Made the download deadline an idle deadline (issue 24). downloadTimeoutMs now aborts a file or thumbnail download only after no bytes have arrived for that long, default 60 seconds, instead of bounding the whole transfer at 10 minutes, so a slow download that keeps making progress completes. A download that fails before reading the whole body cancels it, so a failed file no longer holds its connection.

• 2026-09-23: Every ApiClient request URL is now built by one function next to the class (issue 18), so a self-hosted apiOrigin with a base path keeps it on every request, a path works with or without a leading slash, and query parameters are percent-encoded. A path containing ? or # is rejected with an error instead of being silently cut.

• 2026-09-23: Made the CLI testable and tested it (issue 12). The command bodies moved from bin/quak.ts into src/cli-commands.ts as functions that take their options and a context (output streams, session directory, cache directory, session loader) and return an exit code; bin/quak.ts only wires them to commander and exits with the code once stdout and stderr have drained, so nothing below it calls process.exit. test/cli/commands.test.ts drives them with a fake client: session file modes, logout, the missing and corrupt session paths, and the output and exit code of whoami, collections, files, get, get-thumb, backup and helper list-missing-thumbnails.

• 2026-09-23: Hardened the backup tree's atomic copy (issue 22). copyAtomic fsyncs its temp file before the rename and the directory after it, through the download writer's fsyncPath; each backup run deletes .quak-backup-*.tmp files whose process is no longer running. The README backup layout names the temp files and states that the rename replaces a symlink and takes the temp file's permissions. Added tests for a missing and an unwritable destination directory for downloadFile and downloadThumbnail.

• 2026-09-23: Hardened the JPEG EXIF scan behind backup-metadata --exif (issue 11). Every segment length is checked against the remaining bytes and lengths under 2 stop the scan, so a truncated or corrupt original can neither throw nor loop. A malformed or unparseable EXIF segment is recorded as imageMetadata.exifError, and a failure to read the original as imageMetadataError in the per-file JSON, instead of the field being left out.

• 2026-09-22: Hardened the retry classifier (issue 80). A POST or PUT is replayed only when every errno in the cause chain is a connect errno, and it no longer follows redirects. getRetryOptions() returns a copy. Tests pin every errno the classifier names, the cause-chain depth limit, cycle termination, and a fresh deadline per attempt for every retrying entry point. The README's endpoint list is the one place that names the requests the replay rule covers.

• 2026-09-22: Stopped make test collecting tests from checkouts nested under .claude/ (issue 25). vitest ignores .gitignore when finding tests, so a nested checkout ran the whole suite again; vitest.config.ts now adds .claude/** to vitest's default excludes, and test/packaging/nested-checkout.test.ts plants a nested checkout in a temp directory and fails if vitest would collect it.

• 2026-09-22: Dropped the deprecated @types/libsodium-wrappers-sumo stub from devDependencies (issue 27). It shipped no declarations; the types come from libsodium-wrappers-sumo itself. yarn.lock regenerated by yarn remove.

• 2026-09-22: Hardened the client session lifecycle (issue 10). Client.fromJSON checks every snapshot field and each key's decoded length and names the bad field; toJSON reads the token through ApiClient.getAuthToken and throws when there is none; logout zeroes the key buffers, and collectionsSince re-checks for logout after its request so it never decrypts with zeroed keys. The CLI reports a corrupt session file separately from a missing one (src/cli-session.ts).

• 2026-09-22: Sanitized file names taken from server metadata (issue 9). A new src/filename.ts holds the one sanitizer, used by quak get/get-thumb without --out, downloadFile/downloadThumbnail without outPath, and the backup and metadata backup trees; it removes separators, control characters, leading dots and Windows device names, and falls back to a name built from the ID for an empty title. Originals-cache extensions are letters and digits only, else .bin. A user-supplied path is used as is. decryptFile reads a missing or non-string title as "" and rejects metadata that is not a JSON object.

• 2026-09-22: Rewrote the README API reference (and the Getting Started / usage snippets) to match the shipped cache/API library on next (issue 53, issue 13). Documented Library.open and its options, the default-read vs fresh() distinction (and that the CLI's read commands are fresh), the record types and snapshot()/subscribe(), the albums/photos/timeline read surface, Photo content methods, thumbnails.ensure, the mldata search surface, backup(), the three request pools (10/5/25), and the on-disk cache layout; noted the deferred content-hash integrity check (issue 68). Docs-only; no code changed.

• 2026-09-22: Added resumable, deletion-aware enumeration to Client (issue 38, closes issue 7). collectionsSince/filesSince take a starting cursor, decrypt live records, surface tombstoned ids in a separate deleted list (a tombstone has nothing to decrypt, so it is a bare id, not a hollow record), and return the max updationTime seen as the cursor to resume from. filesSince refuses to loop when the diff reports hasMore without advancing the cursor (issue 7). listCollections/listFiles are now thin wrappers that enumerate from sinceTime: 0 and drop deletions, so existing callers are unaffected.

• 2026-09-22: Carried file size, thumbnail size, and the deletion flag through decryptFile (issue 37, foundation for the cache/API design). Live files now populate file.size/thumbnail.size from the server's info (left undefined when the server omits it), and isDeleted is carried from the diff row onto EnteFile. No caller change: listFiles still filters deleted rows before decrypting. Surfacing a tombstone through decryption belongs to the enumeration unit (issue 38).

• 2026-08-10: Made lint-once.test.ts enforce what its header claims. It walked make check only, so it never read Dockerfile — the image CI builds through script/cibuild — and a second prettier --check . could be added there with the suite staying green. The walk now also starts at .gitea/workflows/check.yml and follows its run: steps, so the graph under test is the one CI executes rather than the one someone assumed it executes. The lockfile assertion was a substring check against the whole of script/bootstrap, which has two install sites and so reported the branch the containers never take; the two branches are now resolved separately and every yarn install in each is required to be --frozen-lockfile. Prettier is counted per occurrence instead of per line, so two invocations chained with && no longer read as one, and edges are followed on counted lines instead of being skipped. Every way for the walk to reach nothing — an unknown target, an unknown script, a missing file, a node with no commands, an unknown node kind — is a thrown error rather than a quiet zero. Every assertion in the file was mutation-tested individually.

• 2026-08-10: Stopped make check running prettier --check . twice. Since linting moved into Docker, the duplicate was one container pass and one host pass of the same check: script/lint builds Dockerfile.lint, which runs prettier as a build step, and script/check then called script/fmt-check as well. The host call is gone from script/check and from script/precommit; the container keeps checking formatting, because a successful Dockerfile.lint build is what CI treats as proof of a clean tree, and it is also what still fails the pre-commit hook on a badly formatted tree. script/fmt-check survives as a standalone entrypoint, whose verdict cannot drift from the container's. A test walks the invocation graph from each entrypoint — through the Makefile shims, the script/ calls and the docker build — and asserts the prettier count, so the duplication cannot come back unnoticed.

• 2026-08-10: Moved all linting into Docker. script/lint builds a new root Dockerfile.lint, which copies the repo into the digest-pinned node image and runs eslint and prettier as build steps, so a successful build is a clean lint; no host lint path remains and yarn lint is gone from package.json. A fail-closed LINT_EPOCH guard stops Docker serving the linter layers from cache, which is how a lint build returns success in under a second having linted nothing. The lint stage inside Dockerfile and its COPY --from=lint ordering hack are gone: that image now runs make test and make build only, because script/check calls script/lint and running it in a container would mean docker inside docker. script/cibuild builds the lint image first, then the test and build image.

• 2026-08-09: Made make docker green and policy-conformant. Multi-stage Dockerfile: a lint stage runs make fmt-check and make lint, and the check stage takes a COPY --from=lint dependency on it before running make check and make build. CHECK_EPOCH and a fail-closed guard stop Docker serving those two layers from cache, which is what let a build report success without running the suite. script/projectname says quak, so the image is tagged quak; script/bootstrap updates apt lists before installing, so a Debian base works; .dockerignore no longer ships the compiled binary, the caches or agent worktrees into the build context, and keeps .gitignore in it for prettier.

• 2026-08-09: Fixed the TypeScript build. rootDir is the repo root, so bin/ compiles alongside src/ instead of failing with TS6059; output is dist/src/ and dist/bin/, which is where main, types and bin.quak now point. script/build verifies the declared entrypoints exist after the compiler runs and makes the CLI executable, the Dockerfile runs make build as well as make check, and a quak script makes the README's yarn quak <command> examples work.

• 2026-08-09: Retry policy: no retry on 4xx (except 408 and 429), exponential backoff with full jitter on 5xx, transport failures and truncated transfers, under per-attempt deadlines that cover the response body as well as the request. Downloads retry request, stream consumption and decryption as one unit; postJSON and putJSON are replayed only when the connection was never established.

• 2026-08-09: Downloads verify the secretstream terminated on TAG_FINAL and write output atomically: a truncated body is rejected instead of landing on disk as a short file, and plaintext is staged in a sibling temp file and renamed into place, so a failed download leaves the destination untouched.

• 2026-07-07 Adopted scripts-to-rule-them-all: script/ entrypoints, Makefile shims, README Entrypoints section

• 2026-06-10: Decrypted collections shared by other users (sealed-box keys); listCollections drops deleted-collection tombstones.

• 2026-06-10: Login hardening: dual-2FA empty-string fields handled, TOTP preferred when a passkey is also enrolled, interactive input via @inquirer/prompts.

• 2026-06-10: Replaced sharp with pure JS (jpeg-js + exif-reader); added single-binary bun build and make install.

• 2026-06-09: Added backup-metadata command (ML data always included, --exif opt-in); rewrote README to match the implementation; added thumbnail helper tests.

• 2026-05-13: Full CLI surface: login, backup with dedup symlink layout, collections, files, get, get-thumb, thumbnail repair helpers.

• 2026-05-13: Client OO API with literate usage tests; file download and decryption; all three metadata layers decrypted and persisted; renamed quack to quak.

• 2026-05-11: SRP login flow (email OTP + TOTP) and ApiClient.

Future Steps

• Future desktop client, separate repo:
  • Electron app skeleton consuming this library.
  • Local SQLite cache keyed on (collectionID, fileID, updationTime).
  • Background sync worker streaming new files into the cache.
  • Gallery UI: thumbnails, full-image view, basic search.
  • Upload, delete, and share operations in the library.