sneak/quak
1
0
Fork 0
Code Issues 5 Pull Requests 2 Actions 7 Packages Projects Releases Wiki Activity

One cron-safe command that mirrors the whole account, ML data and metadata included #162

New Issue
Open
opened 2026-10-06 01:10:29 +02:00 by clawbot · 3 comments
clawbot commented 2026-10-06 01:10:29 +02:00
Collaborator
Copy Link
Copy Source

sneak (chat, 2026-10-05 ~23:07 UTC):

i want a single command that i can run that uses the new cache apis that will download the entirety of the account - all source files, and especially all metadata, including AI tags/magic/etc - to a local directory, with resumability and integrity checking. something that is resilient and can be run from cron, that will retry with backoff on api failures (that should be in the library part), etc. something i can set and forget in a cronjob.

What next already has

quak backup <dir> runs on the cache-backed Library. It downloads every original once, skips any already on disk, and writes each file atomically (temporary file, sync, rename). Each file gets a JSON file with the basic fields and its private and public magic metadata. It keeps a failures.json ledger and exits non-zero when any file failed. The library retries with exponential backoff and jitter (src/retry.ts).

Gaps against the request

  • ML data and EXIF are not in the backup. Ente's ML data (faces, the CLIP embedding) and the EXIF/XMP read from each original are written only by a second command, quak backup-metadata --exif. quak backup only fills the cache's mldata/.
  • No integrity check of stored files. backup treats a file that exists as whole and never re-checks it against its content hash.
  • No guard against overlapping runs. Two cron runs at once would both write the same tree.
  • Short retry budget. At most 4 attempts and about 3.5 seconds of waiting per file; a longer outage fails the file until the next run.

Proposed definition of done (to be confirmed by sneak in chat before work starts)

  • One command, quak backup <dir>, writes everything backup-metadata --exif writes today, per file, beside each original: ML data, EXIF, XMP and dimensions, in addition to the basic fields and magic metadata.
  • Each new download is checked against the content hash Ente records before it is renamed into place. quak backup --verify <dir> re-hashes every stored original and downloads again any that is missing or does not match.
  • A lock in <dir>: a second run while one is running exits at once with a clear one-line message and a non-zero exit code.
  • Runs unattended from the saved session. An expired session fails with a one-line message saying to run quak login.
  • Retries stay in the library, with a longer budget suited to an unattended run.
  • README: a cron example and what each exit code means.
  • Tests for each of the above; lands on next after an independent review.

Model: opus-5-5

sneak (chat, 2026-10-05 ~23:07 UTC): > i want a single command that i can run that uses the new cache apis that will download the entirety of the account - all source files, and especially all metadata, including AI tags/magic/etc - to a local directory, with resumability and integrity checking. something that is resilient and can be run from cron, that will retry with backoff on api failures (that should be in the library part), etc. something i can set and forget in a cronjob. ## What `next` already has `quak backup <dir>` runs on the cache-backed `Library`. It downloads every original once, skips any already on disk, and writes each file atomically (temporary file, sync, rename). Each file gets a JSON file with the basic fields and its private and public magic metadata. It keeps a `failures.json` ledger and exits non-zero when any file failed. The library retries with exponential backoff and jitter (`src/retry.ts`). ## Gaps against the request - **ML data and EXIF are not in the backup.** Ente's ML data (faces, the CLIP embedding) and the EXIF/XMP read from each original are written only by a second command, `quak backup-metadata --exif`. `quak backup` only fills the cache's `mldata/`. - **No integrity check of stored files.** `backup` treats a file that exists as whole and never re-checks it against its content hash. - **No guard against overlapping runs.** Two cron runs at once would both write the same tree. - **Short retry budget.** At most 4 attempts and about 3.5 seconds of waiting per file; a longer outage fails the file until the next run. ## Proposed definition of done (to be confirmed by sneak in chat before work starts) - One command, `quak backup <dir>`, writes everything `backup-metadata --exif` writes today, per file, beside each original: ML data, EXIF, XMP and dimensions, in addition to the basic fields and magic metadata. - Each new download is checked against the content hash Ente records before it is renamed into place. `quak backup --verify <dir>` re-hashes every stored original and downloads again any that is missing or does not match. - A lock in `<dir>`: a second run while one is running exits at once with a clear one-line message and a non-zero exit code. - Runs unattended from the saved session. An expired session fails with a one-line message saying to run `quak login`. - Retries stay in the library, with a longer budget suited to an unattended run. - README: a cron example and what each exit code means. - Tests for each of the above; lands on `next` after an independent review. Model: opus-5-5
clawbot commented 2026-10-06 01:12:12 +02:00
Author
Collaborator
Copy Link
Copy Source

sneak confirmed the proposed definition of done above in chat (2026-10-05 ~23:12 UTC: "yes please"). It is the definition of done for this issue. The quak repo-manager takes it from here.

Model: opus-5-5

sneak confirmed the proposed definition of done above in chat (2026-10-05 ~23:12 UTC: "yes please"). It is the definition of done for this issue. The quak repo-manager takes it from here. Model: opus-5-5
clawbot commented 2026-10-06 01:17:44 +02:00
Author
Collaborator
Copy Link
Copy Source

Plan: eight units, one PR each to next, each gated by an independent reviewer. The brief for each is on its issue.

Run now, in parallel (they change different files):

  • #163 ML data in each file's JSON
  • #164 expired session: one line naming quak login, exit 3
  • #165 longer retry budget for quak backup, set in the library

Then, one at a time where they share files:

  • #166 account and album records (after 163)
  • #169 the lock in the backup directory, exit 2 (after 163 and 164)
  • #167 EXIF, XMP and dimensions in each file's JSON (after 166)
  • #168 --verify (after 167)
  • #170 README: cron example and exit codes (last)

Decisions taken:

  • Checking new downloads against the recorded hash is already on next (#68); only --verify remains.
  • Exit codes: 0 complete; 1 files failed or another error, the next run retries; 2 another backup of the directory is running; 3 no usable session, run quak login.
  • EXIF is read from images and a live photo's image only, as photo.exif() does; no video is read into memory.
  • backup-metadata stays.
  • A session revoked mid-run fails the remaining files into failures.json; the next run stops at once with exit 3.

Model: opus-5-5

Plan: eight units, one PR each to `next`, each gated by an independent reviewer. The brief for each is on its issue. Run now, in parallel (they change different files): - https://git.eeqj.de/sneak/quak/issues/163 ML data in each file's JSON - https://git.eeqj.de/sneak/quak/issues/164 expired session: one line naming `quak login`, exit 3 - https://git.eeqj.de/sneak/quak/issues/165 longer retry budget for `quak backup`, set in the library Then, one at a time where they share files: - https://git.eeqj.de/sneak/quak/issues/166 account and album records (after 163) - https://git.eeqj.de/sneak/quak/issues/169 the lock in the backup directory, exit 2 (after 163 and 164) - https://git.eeqj.de/sneak/quak/issues/167 EXIF, XMP and dimensions in each file's JSON (after 166) - https://git.eeqj.de/sneak/quak/issues/168 `--verify` (after 167) - https://git.eeqj.de/sneak/quak/issues/170 README: cron example and exit codes (last) Decisions taken: - Checking new downloads against the recorded hash is already on `next` (https://git.eeqj.de/sneak/quak/issues/68); only `--verify` remains. - Exit codes: 0 complete; 1 files failed or another error, the next run retries; 2 another backup of the directory is running; 3 no usable session, run `quak login`. - EXIF is read from images and a live photo's image only, as `photo.exif()` does; no video is read into memory. - `backup-metadata` stays. - A session revoked mid-run fails the remaining files into `failures.json`; the next run stops at once with exit 3. Model: opus-5-5
clawbot commented 2026-10-06 10:32:07 +02:00
Author
Collaborator
Copy Link
Copy Source

Progress and a process note for whoever manages this next.

Landed on next: #163 (ML data), #164 (expired session, exit 3), #165 (retry budget) and #166 (account and album records). In review: #177 (167). In progress: #169 (lock). Not started: #168 (after 167), #170 (last).

Every unit adds its TODO.md entry at the top of Completed Steps, so each merge leaves every open PR conflicting there and nowhere else. Reviewers now resolve that one conflict locally (both entries, the PR's on top), gate that tree and report its tree hash. After a pass, a worker rebases the PR the same way and gates the exact commit; it merges without another review when its patch matches the reviewed one except for TODO.md context (checked with git range-diff) and its parent is the current next.

Model: opus-5-5

Progress and a process note for whoever manages this next. Landed on `next`: https://git.eeqj.de/sneak/quak/issues/163 (ML data), https://git.eeqj.de/sneak/quak/issues/164 (expired session, exit 3), https://git.eeqj.de/sneak/quak/issues/165 (retry budget) and https://git.eeqj.de/sneak/quak/issues/166 (account and album records). In review: https://git.eeqj.de/sneak/quak/pulls/177 (167). In progress: https://git.eeqj.de/sneak/quak/issues/169 (lock). Not started: https://git.eeqj.de/sneak/quak/issues/168 (after 167), https://git.eeqj.de/sneak/quak/issues/170 (last). Every unit adds its `TODO.md` entry at the top of Completed Steps, so each merge leaves every open PR conflicting there and nowhere else. Reviewers now resolve that one conflict locally (both entries, the PR's on top), gate that tree and report its tree hash. After a pass, a worker rebases the PR the same way and gates the exact commit; it merges without another review when its patch matches the reviewed one except for `TODO.md` context (checked with `git range-diff`) and its parent is the current `next`. Model: opus-5-5
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
critical
Extremely important: data loss, a security hole, production down, or basic use impossible
 merge-ready
Review passed; ready to merge
 needs-checks
CI / make check is not green on this PR
 needs-rebase
PR conflicts with main and must be rebased
 needs-review
PR is awaiting adversarial review
 needs-rework
Review failed; PR needs changes
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
clawbot sneak (Jeffrey Paul)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/quak#162