quak/TODO.md

# Workflow

- branch (from `main`)
- do the work in Next Step
- move Next Step to the top of Completed Steps
- move the top item of Future Steps into Next Step
- commit (`TODO.md` changes in the same commit as the work)
- merge to `main` if the branch is not protected, otherwise open a PR
- push

# Status

pre-1.0

# Next Step

Store live photos in a form a photo viewer can open
(https://git.eeqj.de/sneak/quak/issues/107). This waits on sneak's choice
between keeping the ZIP and unpacking it into the image and the video.

Tagging and releases are decided by sneak alone, and happen only when he
declares one.

# Completed Steps

- 2026-09-23: Brought the README and this file in line with the tree (issue
  111). The layout lists `src/library/` and the other source files, the backup
  layout names `failures.json` and the optional `thumbnails/`, "Opening a
  library" says what happens when the first refresh fails, the Testing section
  gives the 60-second hard cap and 20-second target for `make test` and names
  the 90-second `timeout` in the `Dockerfile` as the backstop for a hung test,
  and "Tag v1.0.0" is no longer listed as the next step.

- 2026-09-23: Tested `quak login` and `backup-metadata --exif` (issue 110).
  `loginCommand` takes its login function and its prompts from `CliContext`, and
  `bin/quak.ts` passes `Client.login` and the terminal prompts. Tests cover a
  login from `QUAK_EMAIL` and `QUAK_PASSWORD` with no prompt, the TOTP prompt, a
  failed login, and the saved session's modes, and show that `--exif` and
  `--all` each turn on EXIF extraction and that it is off without them.

- 2026-09-23: `quak backup` writes each original once and no longer fills the
  cache (issue 106). An original fetched for a backup is written by the download
  writer straight into the backup's `originals/`, and the content cache records
  it there instead of keeping its own copy; one the cache already held is still
  copied. `quak backup` opens its library with the thumbnail and originals
  precache off.

- 2026-09-23: `backup-metadata`, `helper list-missing-thumbnails` and
  `helper fix-missing-thumbnails` refresh before they answer (issue 100). Each
  awaits `lib.fresh()` before reading, so a file added since the cache was
  written is included, and a failed refresh prints one line and exits 1 instead
  of answering from a stale or empty cache. The README lists them among the
  commands that refresh first.

- 2026-09-23: `quak backup` waits for the server refresh and fails when it fails
  (issue 99). `lib.backup()` joins a refresh already running or starts one, as
  `fresh()` does, and rejects before touching any file when it fails, leaving
  `failures.json` as it was, so `quak backup` prints the error as one line and
  exits 1 instead of backing up the previous run's file list, or nothing, and
  exiting 0.

- 2026-09-23: CLI errors print a message instead of a stack trace (issue 102).
  An error a command throws is printed as one `quak: MESSAGE` line on stderr and
  the CLI exits 1 once output has drained. The wrapper that does this moved from
  `bin/quak.ts` to `src/cli-run.ts`, and `bin/quak.ts` now awaits
  `program.parseAsync()`.

- 2026-09-23: Opening a library no longer deletes another process's download in
  progress (issue 105). The download writer's temp files are named
  `.quak-<pid>-<random>.tmp`, and `removeLeftoverTempFiles`, moved from the
  backup into the download module, deletes a `.quak-*.tmp` file only when the
  process ID in its name is no longer running. The content cache calls it at
  `open()` for `originals/` and `thumbnails/`, the backup as before.

- 2026-09-23: Re-vendored the lint and test setup from the template (issue 96).
  Linting and testing are the `lint` and `test` phases of the `Dockerfile`;
  `script/lint` and `script/test` each build one with `--no-cache`, and the last
  stage compiles and depends on both, so `script/cibuild` is one build.
  `Dockerfile.lint`, `CHECK_EPOCH`, `LINT_EPOCH` and the tests that checked them
  are gone; `REPO_POLICIES.md` is re-copied.

- 2026-09-23: Stopped `helper fix-missing-thumbnails` retrying files the server
  always refuses (issue 109). Both thumbnail helpers skip a file another account
  owns without fetching it. The fixer skips a file whose recorded thumbnail size
  is 0 or unknown before downloading it, and otherwise tries smaller encodings
  (720 px quality 50 down to 160 px quality 20) until the encrypted thumbnail is
  no larger than that size, skipping the file if none fits.

- 2026-09-23: Tested the live-photo hash check's error paths (issue 117). Tests
  download a live photo whose ZIP names an unknown compression method, one whose
  ZIP has no image entry and one with no video entry, and check that nothing is
  stored and the error names the file ID; the unreadable one is not retried.

- 2026-09-23: `quak logout` ends the session on the server (issue 108). It calls
  `POST /users/logout` through the new `Client.logoutOnServer()`, then deletes
  `session.json` even when that call fails, says so and exits 1. It prints the
  account's cache directory and says it still holds decrypted data. The default
  cache path is now `defaultCacheDirectory()` in the library, shared with
  `Library.open`.

- 2026-09-23: Fixed the backup's per-collection folders (issue 103). Two files
  in one collection with the same title, and two collections with the same name,
  each get their ID added to the name (`IMG_0001 (12345).JPG`, `Trip (10)/`), so
  none replaces another's symlink or JSON. Each run removes symlinks into
  `originals/` for files no longer in the collection, and the folders of deleted
  or renamed collections, leaving anything else in `collections/` alone. The
  README backup layout states the naming rule.

- 2026-09-23: Checked downloaded originals against their recorded content hash
  (issue 68). `downloadFile`, which `quak get`, the content cache and backup all
  use, hashes the decrypted bytes (unkeyed BLAKE2b-512, standard base64) and
  stores nothing on a mismatch, failing with an error naming the file ID. A live
  photo ZIP is unpacked as it streams with `fflate` and its image and video
  hashed separately as `<imageHash>:<videoHash>`. `decryptFile` reads older
  clients' `imageHash` and `videoHash` fields for live photos. A file with no
  recorded hash is stored unchecked.

- 2026-09-23: Kept one account's cache from mixing with another's (issue 104).
  When `metadata.json` in the cache directory was written for a different,
  non-zero user ID than the client's, `Library.open` deletes it and `mldata/`
  and starts empty, so the first refresh enumerates from 0. This only happens
  with `--cache-dir` or an explicit `cacheDirectory`; the default path already
  includes the user ID. A test opens one account's cache as another account.

- 2026-09-23: `backup-metadata` no longer stops on one failed ML data request
  (issue 101). Each request of up to 200 files is tried on its own; a failed one
  is logged, its files are written with the reason in `mlDataError`, and the
  command exits 1 once the dump is complete. `fetchMLData`, which only this
  command used, is gone; the command calls `fetchMLDataBatch` per batch.
- 2026-09-23: Single-sourced the version string (issue 5). `package.json` is the
  only place it is written: `src/index.ts` imports it for `VERSION` and
  `bin/quak.ts` passes `VERSION` to commander. tsc copies `package.json` to
  `dist/package.json`, so the import resolves from the built output too, and
  `script/build` runs the built CLI with `--version` to prove it. A test checks
  that `VERSION` and `quak --version` both equal the `package.json` version.
- 2026-09-23: Tested that `Library.close()` waits for the originals precache
  (issue 93). The test that holds a precache fetch open while `close()` runs now
  runs once with only the thumbnail fill and once with only the originals fill,
  so dropping either wait from `Precache.close()` fails a test.
- 2026-09-23: Fixed two intermittently failing library tests (issue 90).
  `Library.close()` now returns a promise that resolves once an in-flight
  refresh (including its cache write), the ML data fetch and running precache
  sweeps have finished; the library tests await it, so `afterEach` no longer
  removes the cache directory while something is still writing into it. The
  precache test waits for both fills to report "done" instead of for its stub
  source to be called, which happened before the cache recorded the file.
- 2026-09-23: Pinned three guards reviewers found untested (issue 89). The
  download idle deadline's timer is unref'd, so it can never keep the process
  alive, and a test checks no timer is left after a download completes or fails.
  A test covers the rejection of `#` in a request path. The EXIF scan compares
  the `Exif` header only in an APP1 segment of length 8 or more, so it never
  reads the next segment's bytes, with a test for a short one.
- 2026-09-23: Made the download deadline an idle deadline (issue 24).
  `downloadTimeoutMs` now aborts a file or thumbnail download only after no
  bytes have arrived for that long, default 60 seconds, instead of bounding the
  whole transfer at 10 minutes, so a slow download that keeps making progress
  completes. A download that fails before reading the whole body cancels it, so
  a failed file no longer holds its connection.
- 2026-09-23: Every `ApiClient` request URL is now built by one function next to
  the class (issue 18), so a self-hosted `apiOrigin` with a base path keeps it
  on every request, a path works with or without a leading slash, and query
  parameters are percent-encoded. A path containing `?` or `#` is rejected with
  an error instead of being silently cut.
- 2026-09-23: Made the CLI testable and tested it (issue 12). The command bodies
  moved from `bin/quak.ts` into `src/cli-commands.ts` as functions that take
  their options and a context (output streams, session directory, cache
  directory, session loader) and return an exit code; `bin/quak.ts` only wires
  them to commander and exits with the code once stdout and stderr have drained,
  so nothing below it calls `process.exit`. `test/cli/commands.test.ts` drives
  them with a fake client: session file modes, logout, the missing and corrupt
  session paths, and the output and exit code of `whoami`, `collections`,
  `files`, `get`, `get-thumb`, `backup` and `helper list-missing-thumbnails`.
- 2026-09-23: Hardened the backup tree's atomic copy (issue 22). `copyAtomic`
  fsyncs its temp file before the rename and the directory after it, through the
  download writer's `fsyncPath`; each backup run deletes `.quak-backup-*.tmp`
  files whose process is no longer running. The README backup layout names the
  temp files and states that the rename replaces a symlink and takes the temp
  file's permissions. Added tests for a missing and an unwritable destination
  directory for `downloadFile` and `downloadThumbnail`.
- 2026-09-23: Hardened the JPEG EXIF scan behind `backup-metadata --exif` (issue
  11). Every segment length is checked against the remaining bytes and lengths
  under 2 stop the scan, so a truncated or corrupt original can neither throw
  nor loop. A malformed or unparseable EXIF segment is recorded as
  `imageMetadata.exifError`, and a failure to read the original as
  `imageMetadataError` in the per-file JSON, instead of the field being left
  out.
- 2026-09-22: Hardened the retry classifier (issue 80). A `POST` or `PUT` is
  replayed only when every errno in the cause chain is a connect errno, and it
  no longer follows redirects. `getRetryOptions()` returns a copy. Tests pin
  every errno the classifier names, the cause-chain depth limit, cycle
  termination, and a fresh deadline per attempt for every retrying entry point.
  The README's endpoint list is the one place that names the requests the replay
  rule covers.
- 2026-09-22: Stopped `make test` collecting tests from checkouts nested under
  `.claude/` (issue 25). vitest ignores `.gitignore` when finding tests, so a
  nested checkout ran the whole suite again; `vitest.config.ts` now adds
  `.claude/**` to vitest's default excludes, and
  `test/packaging/nested-checkout.test.ts` plants a nested checkout in a temp
  directory and fails if vitest would collect it.
- 2026-09-22: Dropped the deprecated `@types/libsodium-wrappers-sumo` stub from
  `devDependencies` (issue 27). It shipped no declarations; the types come from
  `libsodium-wrappers-sumo` itself. `yarn.lock` regenerated by `yarn remove`.
- 2026-09-22: Hardened the client session lifecycle (issue 10).
  `Client.fromJSON` checks every snapshot field and each key's decoded length
  and names the bad field; `toJSON` reads the token through
  `ApiClient.getAuthToken` and throws when there is none; `logout` zeroes the
  key buffers, and `collectionsSince` re-checks for logout after its request so
  it never decrypts with zeroed keys. The CLI reports a corrupt session file
  separately from a missing one (`src/cli-session.ts`).
- 2026-09-22: Sanitized file names taken from server metadata (issue 9). A new
  `src/filename.ts` holds the one sanitizer, used by `quak get`/`get-thumb`
  without `--out`, `downloadFile`/`downloadThumbnail` without `outPath`, and the
  backup and metadata backup trees; it removes separators, control characters,
  leading dots and Windows device names, and falls back to a name built from the
  ID for an empty title. Originals-cache extensions are letters and digits only,
  else `.bin`. A user-supplied path is used as is. `decryptFile` reads a missing
  or non-string title as "" and rejects metadata that is not a JSON object.
- 2026-09-22: Rewrote the README API reference (and the Getting Started / usage
  snippets) to match the shipped cache/API library on `next` (issue 53, issue
  13). Documented `Library.open` and its options, the default-read vs `fresh()`
  distinction (and that the CLI's read commands are fresh), the record types and
  `snapshot()`/`subscribe()`, the `albums`/`photos`/`timeline` read surface,
  `Photo` content methods, `thumbnails.ensure`, the `mldata` search surface,
  `backup()`, the three request pools (10/5/25), and the on-disk cache layout;
  noted the deferred content-hash integrity check (issue 68). Docs-only; no code
  changed.
- 2026-09-22: Added resumable, deletion-aware enumeration to `Client` (issue 38,
  closes issue 7). `collectionsSince`/`filesSince` take a starting cursor,
  decrypt live records, surface tombstoned ids in a separate `deleted` list (a
  tombstone has nothing to decrypt, so it is a bare id, not a hollow record),
  and return the max `updationTime` seen as the cursor to resume from.
  `filesSince` refuses to loop when the diff reports `hasMore` without advancing
  the cursor (issue 7). `listCollections`/`listFiles` are now thin wrappers that
  enumerate from `sinceTime: 0` and drop deletions, so existing callers are
  unaffected.
- 2026-09-22: Carried file size, thumbnail size, and the deletion flag through
  `decryptFile` (issue 37, foundation for the cache/API design). Live files now
  populate `file.size`/`thumbnail.size` from the server's `info` (left
  `undefined` when the server omits it), and `isDeleted` is carried from the
  diff row onto `EnteFile`. No caller change: `listFiles` still filters deleted
  rows before decrypting. Surfacing a tombstone through decryption belongs to
  the enumeration unit (issue 38).
- 2026-08-10: Made `lint-once.test.ts` enforce what its header claims. It walked
  `make check` only, so it never read `Dockerfile` — the image CI builds through
  `script/cibuild` — and a second `prettier --check .` could be added there with
  the suite staying green. The walk now also starts at
  `.gitea/workflows/check.yml` and follows its `run:` steps, so the graph under
  test is the one CI executes rather than the one someone assumed it executes.
  The lockfile assertion was a substring check against the whole of
  `script/bootstrap`, which has two install sites and so reported the branch the
  containers never take; the two branches are now resolved separately and every
  `yarn install` in each is required to be `--frozen-lockfile`. Prettier is
  counted per occurrence instead of per line, so two invocations chained with
  `&&` no longer read as one, and edges are followed on counted lines instead of
  being skipped. Every way for the walk to reach nothing — an unknown target, an
  unknown script, a missing file, a node with no commands, an unknown node kind
  — is a thrown error rather than a quiet zero. Every assertion in the file was
  mutation-tested individually.
- 2026-08-10: Stopped `make check` running `prettier --check .` twice. Since
  linting moved into Docker, the duplicate was one container pass and one host
  pass of the same check: `script/lint` builds `Dockerfile.lint`, which runs
  prettier as a build step, and `script/check` then called `script/fmt-check` as
  well. The host call is gone from `script/check` and from `script/precommit`;
  the container keeps checking formatting, because a successful
  `Dockerfile.lint` build is what CI treats as proof of a clean tree, and it is
  also what still fails the pre-commit hook on a badly formatted tree.
  `script/fmt-check` survives as a standalone entrypoint, whose verdict cannot
  drift from the container's. A test walks the invocation graph from each
  entrypoint — through the Makefile shims, the `script/` calls and the
  `docker build` — and asserts the prettier count, so the duplication cannot
  come back unnoticed.
- 2026-08-10: Moved all linting into Docker. `script/lint` builds a new root
  `Dockerfile.lint`, which copies the repo into the digest-pinned node image and
  runs eslint and prettier as build steps, so a successful build is a clean
  lint; no host lint path remains and `yarn lint` is gone from `package.json`. A
  fail-closed `LINT_EPOCH` guard stops Docker serving the linter layers from
  cache, which is how a lint build returns success in under a second having
  linted nothing. The lint stage inside `Dockerfile` and its `COPY --from=lint`
  ordering hack are gone: that image now runs `make test` and `make build` only,
  because `script/check` calls `script/lint` and running it in a container would
  mean docker inside docker. `script/cibuild` builds the lint image first, then
  the test and build image.
- 2026-08-09: Made `make docker` green and policy-conformant. Multi-stage
  Dockerfile: a lint stage runs `make fmt-check` and `make lint`, and the check
  stage takes a `COPY --from=lint` dependency on it before running `make check`
  and `make build`. `CHECK_EPOCH` and a fail-closed guard stop Docker serving
  those two layers from cache, which is what let a build report success without
  running the suite. `script/projectname` says `quak`, so the image is tagged
  `quak`; `script/bootstrap` updates apt lists before installing, so a Debian
  base works; `.dockerignore` no longer ships the compiled binary, the caches or
  agent worktrees into the build context, and keeps `.gitignore` in it for
  prettier.
- 2026-08-09: Fixed the TypeScript build. `rootDir` is the repo root, so `bin/`
  compiles alongside `src/` instead of failing with TS6059; output is
  `dist/src/` and `dist/bin/`, which is where `main`, `types` and `bin.quak` now
  point. `script/build` verifies the declared entrypoints exist after the
  compiler runs and makes the CLI executable, the Dockerfile runs `make build`
  as well as `make check`, and a `quak` script makes the README's
  `yarn quak <command>` examples work.
- 2026-08-09: Retry policy: no retry on 4xx (except `408` and `429`),
  exponential backoff with full jitter on 5xx, transport failures and truncated
  transfers, under per-attempt deadlines that cover the response body as well as
  the request. Downloads retry request, stream consumption and decryption as one
  unit; `postJSON` and `putJSON` are replayed only when the connection was never
  established.
- 2026-08-09: Downloads verify the secretstream terminated on `TAG_FINAL` and
  write output atomically: a truncated body is rejected instead of landing on
  disk as a short file, and plaintext is staged in a sibling temp file and
  renamed into place, so a failed download leaves the destination untouched.
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile
  shims, README Entrypoints section
- 2026-06-10: Decrypted collections shared by other users (sealed-box keys);
  listCollections drops deleted-collection tombstones.
- 2026-06-10: Login hardening: dual-2FA empty-string fields handled, TOTP
  preferred when a passkey is also enrolled, interactive input via
  @inquirer/prompts.
- 2026-06-10: Replaced sharp with pure JS (jpeg-js + exif-reader); added
  single-binary bun build and make install.
- 2026-06-09: Added backup-metadata command (ML data always included, --exif
  opt-in); rewrote README to match the implementation; added thumbnail helper
  tests.
- 2026-05-13: Full CLI surface: login, backup with dedup symlink layout,
  collections, files, get, get-thumb, thumbnail repair helpers.
- 2026-05-13: Client OO API with literate usage tests; file download and
  decryption; all three metadata layers decrypted and persisted; renamed quack
  to quak.
- 2026-05-11: SRP login flow (email OTP + TOTP) and ApiClient.

# Future Steps

- Future desktop client, separate repo:
    - Electron app skeleton consuming this library.
    - Local SQLite cache keyed on (collectionID, fileID, updationTime).
    - Background sync worker streaming new files into the cache.
    - Gallery UI: thumbnails, full-image view, basic search.
    - Upload, delete, and share operations in the library.