diff --git a/README.md b/README.md index 62d3697..42ebd3b 100644 --- a/README.md +++ b/README.md @@ -127,19 +127,20 @@ matters. ### Exit codes -| Code | Meaning | -| ---- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `0` | The backup is complete: every original is at its save path, and no file failed. | -| `1` | The run finished with files in `failures.json`, or it stopped on another error, printed as one line `quak: `. The next run tries again. | -| `2` | Another backup of the same directory is running. This run sent no request and changed nothing. | -| `3` | There is no usable session: none is saved, the saved one is corrupt, or the server no longer accepts it. Run `quak login` as the user the cron job runs as. | +| Code | Meaning | +| ---- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `0` | The backup is complete: every original is at its save path, and no file failed. | +| `1` | The run finished with files in `failures.json` or an album it could not decrypt, or it stopped on another error, printed as one line `quak: `. The next run tries again. | +| `2` | Another backup of the same directory is running. This run sent no request and changed nothing. | +| `3` | There is no usable session: none is saved, the saved one is corrupt, or the server no longer accepts it. Run `quak login` as the user the cron job runs as. | After a `1`, the next run fetches each missing original again, fetches the ML -data that is not cached, and rebuilds the album links. An original that a -`--verify` run could not read, or put back with bytes that still do not match, -stays at its save path. Only a later `--verify` run checks it again: a run -without `--verify` leaves it as it is and takes the file out of `failures.json`, -so that run can exit 0. +data that is not cached, and rebuilds the album links. A file or album quak +could not decrypt keeps the exit code at `1` until the server's copy changes +(see "Backup layout"). An original that a `--verify` run could not read, or put +back with bytes that still do not match, stays at its save path. Only a later +`--verify` run checks it again: a run without `--verify` leaves it as it is and +takes the file out of `failures.json`, so that run can exit 0. ## Examples @@ -623,8 +624,10 @@ library. The read commands — `collections`, `files`, `get`, `get-thumb`, answer, so they report current account state rather than whatever the cache last held. If that round-trip fails, the command prints the error on one line and exits 1, or 3 when the server no longer accepts the saved session (see "Session -handling"). `--cache-dir` overrides where the cache lives; without it each -account gets its own directory under the per-user cache path. +handling"). An album or file quak cannot decrypt does not fail it; the command +answers without it (see "Default reads vs. fresh reads"). `--cache-dir` +overrides where the cache lives; without it each account gets its own directory +under the per-user cache path. `get` and `get-thumb` resolve the file by ID directly, so `--collection` is accepted for backward compatibility but ignored. For a live photo, `get` writes @@ -734,6 +737,17 @@ succeeds, or once it is no longer in the library or in the backup's scope. The library's `lib.backup({ includeThumbnails: true })` also writes `thumbnails/.jpg` beside `collections/`; `quak backup` does not. +A file the library could not decrypt (see "Default reads vs. fresh reads") goes +into `failures.json` with an empty title, since its title is encrypted too, and +an error that starts `could not be decrypted:` and gives the reason. A file in +two albums is backed up as usual if its copy in the other album decrypted. An +album the library could not decrypt has no name quak can read and no files it +can list. Each run prints one line for it, +`FAILED album : could not be decrypted: `, and `--json` and the +`BackupResult` list its ID in `undecryptableAlbums`. Every other file is backed +up. Either kind keeps the exit code at 1, run after run, until the server sends +a copy that decrypts or deletes it; the next run then drops it. + `backup.lock` keeps two backups of the same directory from running at once, such as a cron run that starts while the previous one is still going. A backup creates `` if it is missing and takes the lock before its refresh, and @@ -906,6 +920,15 @@ rejects the caller (default reads stay silent and keep serving the last good copy). The CLI's read commands use fresh reads (issue https://git.eeqj.de/sneak/quak/issues/75). +An album or file in the server's answer that quak cannot decrypt, or whose +metadata is not JSON, does not fail the refresh. The refresh leaves it out, an +album with all its files, and drops any older copy the cache held, so reads +answer from what did decrypt. The library keeps a list of what it left out, with +the reason, in `metadata.json`: the next refresh asks the server only for what +changed, so it would not see the record again. A record stays on that list until +a later refresh gets a copy that decrypts or learns that the server deleted it. +`lib.backup()` reports each one (see "Backup layout"). + ### Read surface - `lib.albums.list()` → `Album[]`, newest-updated first. diff --git a/TODO.md b/TODO.md index 2ab03da..74dcff8 100644 --- a/TODO.md +++ b/TODO.md @@ -25,6 +25,17 @@ declares one. # Completed Steps +- 2026-10-13: An album or file quak cannot decrypt, or whose metadata is not + JSON, no longer fails the refresh, so one bad record no longer stops + `quak backup` from storing anything (issue 181). The refresh leaves it out, an + album with its files, and the library keeps it with the reason in + `metadata.json` until the server sends a copy that decrypts or deletes it. The + backup stores every other file, records each such file in `failures.json` with + an error starting `could not be decrypted:`, prints one line naming each such + album's ID, lists those IDs in `undecryptableAlbums`, and exits 1 while any + remain. A file whose copy in another album decrypts is backed up as usual. The + read commands answer from what decrypted. + - 2026-10-06: The README says how to run `quak backup` from cron (issue 170): log in once as the job's user, a crontab with a backup every night and `--verify` on Sundays appending to a log file, and the `HOME` and diff --git a/src/backup.ts b/src/backup.ts index 184d466..662c1b0 100644 --- a/src/backup.ts +++ b/src/backup.ts @@ -47,6 +47,9 @@ // download, a failed symlink, or ML data missing because the ML data fetch // failed is caught, recorded in `failures.json` with a classification, a // running attempt count, and the last-tried time, and the run continues. +// A file the library could not decrypt is recorded there too, unless a copy of +// it in another album decrypted; an album it could not decrypt is logged by ID +// and listed in `result.undecryptableAlbums`, as its name is not known. // `result.failed` — and thus the CLI's exit code — stays non-zero while any // failure remains unresolved and clears once every one succeeds. Each run // reconciles the ledger against the files it attempted, so an entry for a file @@ -86,6 +89,7 @@ import { storedAtSavePath, } from "./library/content.js"; import { representative } from "./library/records.js"; +import type { Undecryptable } from "./library/store.js"; import { extractImageMetadata } from "./metadata-backup.js"; import type { MLData } from "./mldata-fetch.js"; import type { Collection, EnteFile, FileMetadata } from "./model/types.js"; @@ -141,6 +145,9 @@ export interface BackupResult { failed: number; // This run's per-file errors, in encounter order. errors: BackupError[]; + // The IDs of the albums the library could not decrypt, whose files it + // could not list; the CLI exits non-zero while there are any. + undecryptableAlbums: number[]; } // The slice of the library that backup drives. `Library` implements it; a test @@ -151,6 +158,9 @@ export interface BackupLibrary { refresh(): Promise; listCollections(): Collection[]; listFiles(collectionID: number): EnteFile[]; + // The albums, and files in albums, that the library could not decrypt, + // which `listCollections` and `listFiles` leave out. + undecryptable(): Undecryptable[]; // Get an original's bytes onto disk through the content cache/pools, // returning where they landed: `destination` when they were fetched now, // otherwise wherever they already were (the cache, or the library's save @@ -574,33 +584,54 @@ const runLockedBackup = async ( let unchecked = 0; const recordFailure = ( - file: EnteFile, + fileID: number, + title: string, collection: string, err: unknown, ): void => { // Count at most one attempt per file per run: a file whose original // and thumbnail both fail this run must not double its attempt count // or appear twice in errors. - if (failedThisRun.has(file.id)) return; + if (failedThisRun.has(fileID)) return; const error = errorMessage(err); - errors.push({ - fileID: file.id, - title: file.metadata.title, - collection, - error, - }); - const prior = ledger.get(file.id); - ledger.set(file.id, { - fileID: file.id, - title: file.metadata.title, + errors.push({ fileID, title, collection, error }); + const prior = ledger.get(fileID); + ledger.set(fileID, { + fileID, + title, classification: classify(err), attempts: (prior?.attempts ?? 0) + 1, lastTriedAt: now, error, }); - failedThisRun.add(file.id); + failedThisRun.add(fileID); }; + // What the library could not decrypt. An album is logged by its ID, as + // its name is not known, so a run limited to some albums logs it too. A + // file in an album in scope is recorded as failed, with an empty title as + // its metadata is not known, unless a copy of it in another album + // decrypted. + const scope = new Set(collections.map((c) => c.id)); + const undecryptableAlbums: number[] = []; + for (const record of lib.undecryptable()) { + const error = `could not be decrypted: ${record.error}`; + if (record.fileID === undefined) { + log(`FAILED album ${record.collectionID}: ${error}`); + undecryptableAlbums.push(record.collectionID); + } else if ( + scope.has(record.collectionID) && + !memberships.has(record.fileID) + ) { + recordFailure( + record.fileID, + "", + collectionName.get(record.collectionID) ?? "", + new Error(error), + ); + } + } + // Phase 1: get the bytes. Put each pending original at its save path // through the content cache/pools, as `Photo.download()` does, and fetch // the optional thumbnails; a present file is left as is. With `verify`, a @@ -636,7 +667,8 @@ const runLockedBackup = async ( `FAILED verifying original ${file.metadata.title}: ${errorMessage(err)}`, ); recordFailure( - file, + file.id, + file.metadata.title, collectionName.get(file.collectionID) ?? "", err, ); @@ -676,7 +708,8 @@ const runLockedBackup = async ( `FAILED original ${file.metadata.title}: ${errorMessage(err)}`, ); recordFailure( - file, + file.id, + file.metadata.title, collectionName.get(file.collectionID) ?? "", err, ); @@ -693,7 +726,8 @@ const runLockedBackup = async ( await copyAtomic(path, dest); } catch (err) { recordFailure( - file, + file.id, + file.metadata.title, collectionName.get(file.collectionID) ?? "", err, ); @@ -733,7 +767,8 @@ const runLockedBackup = async ( if (mlData === undefined && mlDataError !== undefined) { writeSidecar(path, file, { mlDataError }, image); recordFailure( - file, + file.id, + file.metadata.title, collectionName.get(file.collectionID) ?? "", new Error(`ML data: ${mlDataError}`), ); @@ -806,7 +841,12 @@ const runLockedBackup = async ( log( `FAILED symlink ${c.name}/${linkName}: ${errorMessage(err)}`, ); - recordFailure(link.file, c.name, err); + recordFailure( + link.file.id, + link.file.metadata.title, + c.name, + err, + ); } } @@ -837,6 +877,7 @@ const runLockedBackup = async ( unchecked, failed: ledger.size, errors, + undecryptableAlbums, }; }; diff --git a/src/cli-commands.ts b/src/cli-commands.ts index f519414..e2d536f 100644 --- a/src/cli-commands.ts +++ b/src/cli-commands.ts @@ -462,7 +462,9 @@ export const backupCommand = async ( } } - return result.failed > 0 ? 1 : 0; + return result.failed > 0 || result.undecryptableAlbums.length > 0 + ? 1 + : 0; } finally { await lib.close(); } diff --git a/src/client.ts b/src/client.ts index 62784c1..bcb3dac 100644 --- a/src/client.ts +++ b/src/client.ts @@ -44,20 +44,28 @@ export interface ClientSnapshot { // The result of a resumable enumeration. Live decrypted records and deleted // ids are kept apart on purpose: a tombstone carries no key or metadata to -// decrypt, so it is a bare id rather than a hollowed-out record. `cursor` is -// the max `updationTime` seen, to pass back into the next call. +// decrypt, so it is a bare id rather than a hollowed-out record. A live row +// that cannot be decrypted, or whose metadata cannot be parsed, is in +// `undecryptable` with the reason instead, so one bad row does not fail the +// call; absent means there was none. `cursor` is the max `updationTime` seen, +// those rows included, to pass back into the next call. export interface CollectionsPage { collections: Collection[]; deleted: number[]; + undecryptable?: { id: number; error: string }[]; cursor: number; } export interface FilesPage { files: EnteFile[]; deleted: number[]; + undecryptable?: { id: number; error: string }[]; cursor: number; } +const errorMessage = (err: unknown): string => + err instanceof Error ? err.message : String(err); + export class Client { private readonly api: ApiClient; private readonly email: string; @@ -237,11 +245,12 @@ export class Client { } // Enumerate collections changed since `sinceTime`. Live collections are - // decrypted; tombstoned ones (isDeleted) are surfaced as bare ids. The - // returned cursor is the max `updationTime` seen — including tombstones, so - // the next sync resumes past them — falling back to `sinceTime` when the - // response is empty. `/collections/v2` returns the whole changed set in one - // response, so there is no pagination here. + // decrypted; tombstoned ones (isDeleted) are surfaced as bare ids, and + // ones that do not decrypt as ids with the reason. The returned cursor is + // the max `updationTime` seen — including tombstones, so the next sync + // resumes past them — falling back to `sinceTime` when the response is + // empty. `/collections/v2` returns the whole changed set in one response, + // so there is no pagination here. async collectionsSince(args: { sinceTime: number; }): Promise { @@ -254,31 +263,40 @@ export class Client { const collections: Collection[] = []; const deleted: number[] = []; + const undecryptable: { id: number; error: string }[] = []; let cursor = args.sinceTime; for (const raw of raws) { if (raw.isDeleted) { deleted.push(raw.id); } else { - collections.push( - decryptCollection( - raw, - { - masterKey: this.masterKey, - publicKey: this.publicKey, - secretKey: this.secretKey, - }, - this.userID, - ), - ); + try { + collections.push( + decryptCollection( + raw, + { + masterKey: this.masterKey, + publicKey: this.publicKey, + secretKey: this.secretKey, + }, + this.userID, + ), + ); + } catch (err) { + undecryptable.push({ + id: raw.id, + error: errorMessage(err), + }); + } } if (raw.updationTime > cursor) cursor = raw.updationTime; } - return { collections, deleted, cursor }; + return { collections, deleted, undecryptable, cursor }; } // Enumerate a collection's files changed since `sinceTime`, paginating the // diff from that cursor. Live rows are decrypted; tombstoned ones are - // surfaced as bare ids. Returns the final cursor to resume from. + // surfaced as bare ids, and ones that do not decrypt or parse as ids with + // the reason. Returns the final cursor to resume from. async filesSince(args: { collectionID: number; collectionKey: Uint8Array; @@ -288,6 +306,7 @@ export class Client { const { collectionID, collectionKey } = args; const files: EnteFile[] = []; const deleted: number[] = []; + const undecryptable: { id: number; error: string }[] = []; let cursor = args.sinceTime; for (;;) { const { diff, hasMore } = await this.api.getJSON<{ @@ -300,7 +319,14 @@ export class Client { if (raw.isDeleted) { deleted.push(raw.id); } else { - files.push(decryptFile(raw, collectionKey)); + try { + files.push(decryptFile(raw, collectionKey)); + } catch (err) { + undecryptable.push({ + id: raw.id, + error: errorMessage(err), + }); + } } if (raw.updationTime > pageMax) pageMax = raw.updationTime; } @@ -321,18 +347,19 @@ export class Client { } cursor = pageMax; } - return { files, deleted, cursor }; + return { files, deleted, undecryptable, cursor }; } - // Whole-account listing: every live collection, deletions hidden. A thin - // wrapper over `collectionsSince` from the beginning of time. + // Whole-account listing: every live collection that decrypts, deletions + // hidden. A thin wrapper over `collectionsSince` from the beginning of + // time. async listCollections(): Promise { const { collections } = await this.collectionsSince({ sinceTime: 0 }); return collections; } - // Every live file in a collection, deletions hidden. A thin wrapper over - // `filesSince` from the beginning of time. + // Every live file in a collection that decrypts, deletions hidden. A thin + // wrapper over `filesSince` from the beginning of time. async listFiles( collectionID: number, collectionKey: Uint8Array, diff --git a/src/library/index.ts b/src/library/index.ts index 489dd88..ff0f1f3 100644 --- a/src/library/index.ts +++ b/src/library/index.ts @@ -598,6 +598,7 @@ export class Library { refresh: () => this.refreshNow(), listCollections: () => this.store.listCollections(), listFiles: (id) => this.store.listFiles(id), + undecryptable: () => this.store.listUndecryptable(), original: (fileID, destination) => cache!.backupOriginal(fileID, destination), thumbnail: (fileID) => cache!.thumbnail(fileID), @@ -699,7 +700,10 @@ export class Library { // Fetch every change since the stored cursor, then commit. All network // reads happen before any store mutation, so a fetch that throws leaves the - // store untouched and the previous snapshot intact. + // store untouched and the previous snapshot intact. A collection or file + // that does not decrypt is left out instead, as if deleted, with the reason + // kept in the store: the cursor moves past it, so no later refresh sees it + // again until the server's copy changes. private async refreshOnce(): Promise { const page = await this.client.collectionsSince({ sinceTime: this.store.collectionsSinceTime, @@ -736,9 +740,17 @@ export class Library { this.store.deleteCollection(id); changed = true; } + if (this.store.deleteUndecryptable(id)) changed = true; + } + // A collection left out takes its files with it. + for (const { id, error } of page.undecryptable ?? []) { + this.store.deleteCollection(id); + this.store.putUndecryptable({ collectionID: id, error }); + changed = true; } for (const collection of page.collections) { this.store.putCollection(collection); + this.store.deleteUndecryptable(collection.id); changed = true; } @@ -748,9 +760,22 @@ export class Library { this.store.deleteFile(collectionID, id); changed = true; } + if (this.store.deleteUndecryptable(collectionID, id)) { + changed = true; + } + } + for (const { id, error } of filePage.undecryptable ?? []) { + this.store.deleteFile(collectionID, id); + this.store.putUndecryptable({ + collectionID, + fileID: id, + error, + }); + changed = true; } for (const f of filePage.files) { this.store.putFile(f); + this.store.deleteUndecryptable(collectionID, f.id); changed = true; } } diff --git a/src/library/store.ts b/src/library/store.ts index e74f8e1..f45cb3c 100644 --- a/src/library/store.ts +++ b/src/library/store.ts @@ -2,10 +2,11 @@ // // The store keeps one `metadata.json` file holding the account's server // state: the user id, a schema version, the cursor for the incremental -// collections listing, and the decrypted collection and file records. The -// whole file is read into RAM on load and rewritten as a whole on save; there -// is no partial update and no lock file. A separate refresh unit populates the -// store from the server — this module only stores what it is given. +// collections listing, the decrypted collection and file records, and the +// collections and files the refresh could not decrypt. The whole file is read +// into RAM on load and rewritten as a whole on save; there is no partial +// update and no lock file. A separate refresh unit populates the store from +// the server — this module only stores what it is given. // // The file is a cache, so it is never trusted to exist or to be intact: a // missing or unreadable file loads as an empty store rather than an error, and @@ -33,12 +34,22 @@ const FILE_MODE = 0o600; type StoredCollection = Omit & { key: string }; type StoredFile = Omit & { key: string }; +// A collection, or with `fileID` a file in it, that the last refresh to fetch +// it could not decrypt or parse, and why. It is kept until a later refresh +// fetches it again, because a refresh fetches only what changed. +export interface Undecryptable { + collectionID: number; + fileID?: number; + error: string; +} + interface StoredMetadata { schemaVersion: number; userID: number; collectionsSinceTime: Microseconds; collections: StoredCollection[]; files: StoredFile[]; + undecryptable?: Undecryptable[]; } const encodeKey = (key: Uint8Array): string => @@ -67,6 +78,7 @@ export class MetadataStore { private readonly collections = new Map(); private readonly files = new Map(); + private undecryptable: Undecryptable[] = []; private constructor(path: string) { this.path = path; @@ -107,6 +119,7 @@ export class MetadataStore { }; store.files.set(fileKey(file.collectionID, file.id), file); } + store.undecryptable = parsed.undecryptable ?? []; } catch { // Any corruption discards the partial result: a half-read cache is // worse than an empty one, since the refresh unit will rebuild it. @@ -134,6 +147,7 @@ export class MetadataStore { ...f, key: encodeKey(f.key), })), + undecryptable: this.undecryptable, }; const dir = dirname(this.path); @@ -165,7 +179,8 @@ export class MetadataStore { this.collections.set(collection.id, collection); } - // Removing a collection also drops its file memberships: a file record is + // Removing a collection also drops its file memberships, and any record + // that it or one of its files could not be decrypted: a file record is // only meaningful as part of a collection the cache still knows about. deleteCollection(id: number): void { this.collections.delete(id); @@ -174,6 +189,9 @@ export class MetadataStore { this.files.delete(key); } } + this.undecryptable = this.undecryptable.filter( + (record) => record.collectionID !== id, + ); } getFile(collectionID: number, fileID: number): EnteFile | undefined { @@ -205,4 +223,26 @@ export class MetadataStore { deleteFile(collectionID: number, fileID: number): void { this.files.delete(fileKey(collectionID, fileID)); } + + listUndecryptable(): Undecryptable[] { + return [...this.undecryptable]; + } + + // Replaces any earlier record of the same collection or file. + putUndecryptable(record: Undecryptable): void { + this.deleteUndecryptable(record.collectionID, record.fileID); + this.undecryptable.push(record); + } + + // Forgets the record of the collection, or with `fileID` of that file in + // it. Returns whether there was one. + deleteUndecryptable(collectionID: number, fileID?: number): boolean { + const before = this.undecryptable.length; + this.undecryptable = this.undecryptable.filter( + (record) => + record.collectionID !== collectionID || + record.fileID !== fileID, + ); + return this.undecryptable.length < before; + } } diff --git a/test/cli/backup-undecryptable.test.ts b/test/cli/backup-undecryptable.test.ts new file mode 100644 index 0000000..72adc11 --- /dev/null +++ b/test/cli/backup-undecryptable.test.ts @@ -0,0 +1,427 @@ +/** + * `quak backup` over an account holding an album or a file quak cannot decrypt + * (issue 181). + * + * The server sends every album and file encrypted, and a refresh decrypts each + * one it fetches. A record whose key no key of the account opens, or whose + * metadata is not JSON, must not fail the refresh: a refresh that fails commits + * nothing, so `quak backup` would store nothing, on every run, until the + * server's copy changed. Instead the refresh leaves that record out, and the + * library keeps a list of what it left out in its cache. It has to: the next + * refresh asks the server only for what changed since the last one, so it will + * not see the record again. + * + * The backup stores every other file. It records each file it could not + * decrypt in `failures.json`, prints one line for each album it could not + * decrypt, and exits 1 while any remain. Once the server sends a copy that + * decrypts, or deletes the record, the next run drops it and can exit 0. The + * read commands answer from what did decrypt. + * + * These tests run the commands with a real saved session over a stand-in Ente + * server, which serves albums, files and originals encrypted with libsodium as + * the real server does. A record is broken the way a corrupt one arrives: its + * key replaced by bytes no key opens, or its metadata text that is not JSON. + */ + +import { existsSync, mkdtempSync, readFileSync, rmSync } from "node:fs"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; +import sodium from "libsodium-wrappers-sumo"; +import { afterEach, beforeAll, beforeEach, describe, expect, it } from "vitest"; + +import { + backupCommand, + collectionsCommand, + filesCommand, + saveSession, + type CliContext, +} from "../../src/cli-commands.js"; +import { loadSession } from "../../src/cli-session.js"; +import type { ClientSnapshot } from "../../src/client.js"; +import { encryptBlob, init, toBase64 } from "../../src/crypto/index.js"; + +const USER_ID = 42; + +// Every file is taken at noon local time on 2026-03-01, in microseconds as Ente +// stores times, so its original is saved in the folder `DAY` whatever the time +// zone of the machine running the tests. +const TAKEN = new Date(2026, 2, 1, 12).getTime() * 1000; +const DAY = join("2026", "2026-03", "2026-03-01"); + +type Row = { id: number; updationTime: number } & Record; + +// `plaintext` sealed under `key` with a fresh nonce, both base64, as Ente +// sends a key or an album name. +const secretbox = ( + plaintext: Uint8Array, + key: Uint8Array, +): { ciphertext: string; nonce: string } => { + const nonce = sodium.randombytes_buf(sodium.crypto_secretbox_NONCEBYTES); + return { + ciphertext: toBase64( + sodium.crypto_secretbox_easy(plaintext, nonce, key), + ), + nonce: toBase64(nonce), + }; +}; + +// What a corrupt row carries in place of its sealed key: random bytes of the +// same length, which no key opens. +const corruptKey = (): string => + toBase64( + sodium.randombytes_buf( + sodium.crypto_secretbox_KEYBYTES + sodium.crypto_secretbox_MACBYTES, + ), + ); + +const json = (body: unknown): Response => + new Response(JSON.stringify(body), { + status: 200, + headers: { "content-type": "application/json" }, + }); + +// A stand-in Ente server for one account. It holds each album, and each file in +// an album, as the row `/collections/v2` and `/collections/v2/diff` serve, and +// answers a request with the rows whose `updationTime` is past the request's +// `sinceTime`, as the real server does. A change to a file also moves its +// album's `updationTime`, which is what tells a client to list the album's +// files again. Originals are served by file ID, as the files CDN serves them. +class FakeEnte { + private readonly masterKey = sodium.crypto_secretbox_keygen(); + private readonly keyPair = sodium.crypto_box_keypair(); + private readonly albums = new Map(); + private readonly albumKeys = new Map(); + // Each album's file rows, by album ID and then file ID. + private readonly files = new Map>(); + // Each file's key and its original as served, by file ID. A file in two + // albums is one original under one key, sealed under each album's key. + private readonly originals = new Map< + number, + { key: Uint8Array; header: string; body: Uint8Array } + >(); + + // The session `quak login` would have saved for the account. + snapshot(): ClientSnapshot { + return { + email: "user@example.com", + userID: USER_ID, + token: "token", + masterKey: toBase64(this.masterKey), + secretKey: toBase64(this.keyPair.privateKey), + publicKey: toBase64(this.keyPair.publicKey), + }; + } + + // Album `id`, named `name`, as of time `t`. With `corrupt`, its key cannot + // be decrypted. An album keeps its key from one call to the next, so its + // files' rows still decrypt once it is put back without `corrupt`. + putAlbum(id: number, name: string, t: number, corrupt = false): void { + const key = this.albumKeys.get(id) ?? sodium.crypto_secretbox_keygen(); + this.albumKeys.set(id, key); + const sealedKey = secretbox(key, this.masterKey); + const sealedName = secretbox(new TextEncoder().encode(name), key); + this.albums.set(id, { + id, + owner: { id: USER_ID }, + encryptedKey: corrupt ? corruptKey() : sealedKey.ciphertext, + keyDecryptionNonce: sealedKey.nonce, + encryptedName: sealedName.ciphertext, + nameDecryptionNonce: sealedName.nonce, + type: "album", + updationTime: t, + }); + if (!this.files.has(id)) this.files.set(id, new Map()); + } + + // File `id`, titled `photo-.jpg`, in album `albumID` as of time `t`. + // With `corruptKey`, its key cannot be decrypted. With `metadata`, its + // metadata decrypts to that text instead of the JSON a client uploads. + putFile( + albumID: number, + id: number, + t: number, + opts: { corruptKey?: boolean; metadata?: string } = {}, + ): void { + const original = this.originalOf(id); + const sealedKey = secretbox(original.key, this.albumKeys.get(albumID)!); + const metadata = + opts.metadata ?? + JSON.stringify({ + title: `photo-${id}.jpg`, + fileType: 0, + creationTime: TAKEN, + modificationTime: TAKEN, + }); + const sealedMetadata = encryptBlob( + new TextEncoder().encode(metadata), + original.key, + ); + this.files.get(albumID)!.set(id, { + id, + collectionID: albumID, + ownerID: USER_ID, + encryptedKey: opts.corruptKey ? corruptKey() : sealedKey.ciphertext, + keyDecryptionNonce: sealedKey.nonce, + metadata: { + encryptedData: toBase64(sealedMetadata.ciphertext), + decryptionHeader: toBase64(sealedMetadata.header), + }, + file: { decryptionHeader: original.header }, + thumbnail: { decryptionHeader: original.header }, + updationTime: t, + }); + this.moveAlbum(albumID, t); + } + + // A deleted record stays on the server as a tombstone holding only its + // ID, its time and `isDeleted`. + deleteFile(albumID: number, id: number, t: number): void { + this.files + .get(albumID)! + .set(id, { id, updationTime: t, isDeleted: true }); + this.moveAlbum(albumID, t); + } + + deleteAlbum(id: number, t: number): void { + this.albums.set(id, { id, updationTime: t, isDeleted: true }); + } + + // The `fetch` the saved session's client is given. + readonly fetch = (async (input: string | URL | Request) => { + const url = new URL(input instanceof Request ? input.url : input); + const since = Number(url.searchParams.get("sinceTime")); + const changed = (rows: Iterable): Row[] => + [...rows].filter((row) => row.updationTime > since); + switch (url.pathname) { + case "/collections/v2": + return json({ collections: changed(this.albums.values()) }); + case "/collections/v2/diff": { + const albumID = Number(url.searchParams.get("collectionID")); + const rows = this.files.get(albumID)?.values() ?? []; + return json({ diff: changed(rows), hasMore: false }); + } + case "/files/data/fetch": + // No file has ML data. + return json({ data: [] }); + default: { + const fileID = Number(url.searchParams.get("fileID")); + const original = this.originals.get(fileID); + return original + ? new Response(original.body) + : new Response("not found", { status: 404 }); + } + } + }) as typeof globalThis.fetch; + + private originalOf(id: number): { + key: Uint8Array; + header: string; + body: Uint8Array; + } { + const known = this.originals.get(id); + if (known) return known; + const key = sodium.crypto_secretbox_keygen(); + const { header, ciphertext } = encryptBlob( + new TextEncoder().encode(`original of ${id}`), + key, + ); + const original = { key, header: toBase64(header), body: ciphertext }; + this.originals.set(id, original); + return original; + } + + private moveAlbum(albumID: number, t: number): void { + this.albums.set(albumID, { + ...this.albums.get(albumID)!, + updationTime: t, + }); + } +} + +// Album 1, "Trip", holds file 10, which decrypts, file 11, whose key does not, +// and file 12, whose metadata is not JSON. Album 2, "Family", holds file 20, +// but the album's own key does not decrypt. All of it as of time 1. +const brokenAccount = (): FakeEnte => { + const ente = new FakeEnte(); + ente.putAlbum(1, "Trip", 1); + ente.putFile(1, 10, 1); + ente.putFile(1, 11, 1, { corruptKey: true }); + ente.putFile(1, 12, 1, { metadata: "not json" }); + ente.putAlbum(2, "Family", 1, true); + ente.putFile(2, 20, 1); + return ente; +}; + +// Collects everything written to it. +class Output { + text = ""; + write(text: string): void { + this.text += text; + } +} + +let root: string; +let stdout: Output; +let stderr: Output; + +beforeAll(async () => { + await init(); + await sodium.ready; +}); + +beforeEach(() => { + root = mkdtempSync(join(tmpdir(), "quak-undecryptable-")); +}); + +afterEach(() => { + rmSync(root, { recursive: true, force: true }); +}); + +// A context for one command, with fresh output, whose saved session talks to +// `ente`. Every run shares one cache directory, as runs of `quak` do. +const context = (ente: FakeEnte): CliContext => { + stdout = new Output(); + stderr = new Output(); + saveSession(join(root, "session"), ente.snapshot()); + return { + stdout, + stderr, + sessionDir: join(root, "session"), + cacheDir: join(root, "cache"), + loadSession: (path) => + loadSession(path, { fetch: ente.fetch, retry: { attempts: 1 } }), + login: async () => { + throw new Error("login not expected"); + }, + prompt: async () => { + throw new Error("prompt not expected"); + }, + promptSecret: async () => { + throw new Error("prompt not expected"); + }, + }; +}; + +const backupDir = (): string => join(root, "backup"); + +// Runs `quak backup` into the backup directory and returns its exit code. +const backup = (ente: FakeEnte, opts: { json?: boolean } = {}) => + backupCommand(context(ente), backupDir(), opts); + +// The backup's `failures.json` entries by file ID, or {} when it has none. +const failures = (): Record => { + const path = join(backupDir(), "failures.json"); + return existsSync(path) + ? JSON.parse(readFileSync(path, "utf-8")).files + : {}; +}; + +// Whether file `id`'s original is at its save path in the backup. +const stored = (id: number): boolean => + existsSync(join(backupDir(), DAY, `2026-03-01.${id}.jpg`)); + +describe("quak backup with records it cannot decrypt", () => { + it("backs up every other file, records each file it cannot decrypt in failures.json, names each such album, and exits 1", async () => { + expect(await backup(brokenAccount())).toBe(1); + + expect(stored(10)).toBe(true); + // Without its album's key, file 20 cannot even be listed. + expect(stored(20)).toBe(false); + expect(failures()).toEqual({ + "11": expect.objectContaining({ + fileID: 11, + error: expect.stringMatching( + /^could not be decrypted: .*wrong secret key/, + ), + }), + "12": expect.objectContaining({ + fileID: 12, + error: expect.stringMatching(/^could not be decrypted: .*JSON/), + }), + }); + expect(stderr.text).toMatch( + /^FAILED album 2: could not be decrypted: .*wrong secret key/m, + ); + expect(stderr.text).toContain(" Failed: 2\n"); + }); + + it("names each album it cannot decrypt in the --json result", async () => { + expect(await backup(brokenAccount(), { json: true })).toBe(1); + + expect(JSON.parse(stdout.text)).toMatchObject({ + downloaded: 1, + failed: 2, + undecryptableAlbums: [2], + }); + }); + + it("still reports them on the next run, though the server does not send them again", async () => { + const ente = brokenAccount(); + await backup(ente); + + // Nothing changed on the server, so this run's refresh gets no rows at + // all. What it reports, it remembered from the first run. + expect(await backup(ente)).toBe(1); + + expect(Object.keys(failures())).toEqual(["11", "12"]); + expect(failures()["11"]!.attempts).toBe(2); + expect(stderr.text).toMatch(/^FAILED album 2: could not be decrypted/m); + }); + + it("drops each once the server sends a copy that decrypts, and then exits 0", async () => { + const ente = brokenAccount(); + await backup(ente); + + // At time 2 files 11 and 12 and album 2 change on the server, and now + // each decrypts. Album 2 brings file 20 with it. + ente.putFile(1, 11, 2); + ente.putFile(1, 12, 2); + ente.putAlbum(2, "Family", 2); + expect(await backup(ente)).toBe(0); + + expect([11, 12, 20].map(stored)).toEqual([true, true, true]); + expect(existsSync(join(backupDir(), "failures.json"))).toBe(false); + expect(stderr.text).not.toContain("FAILED"); + }); + + it("drops each once the server deletes it, and then exits 0", async () => { + const ente = brokenAccount(); + await backup(ente); + + ente.deleteFile(1, 11, 2); + ente.deleteFile(1, 12, 2); + ente.deleteAlbum(2, 2); + expect(await backup(ente)).toBe(0); + + expect(existsSync(join(backupDir(), "failures.json"))).toBe(false); + expect(stderr.text).not.toContain("FAILED"); + }); + + it("backs up a file that does not decrypt in one album but does in another, and does not report it", async () => { + // File 30 is in albums 1 and 3, each holding its key sealed under the + // album's own key. In album 1 that is corrupt; in album 3 it is not, + // and one copy that decrypts is all the backup needs. + const ente = new FakeEnte(); + ente.putAlbum(1, "Trip", 1); + ente.putFile(1, 30, 1, { corruptKey: true }); + ente.putAlbum(3, "Favorites", 1); + ente.putFile(3, 30, 1); + + expect(await backup(ente)).toBe(0); + + expect(stored(30)).toBe(true); + expect(failures()).toEqual({}); + }); +}); + +describe("the read commands with records quak cannot decrypt", () => { + it("quak collections and quak files answer from what decrypts", async () => { + const ente = brokenAccount(); + + expect(await collectionsCommand(context(ente), {})).toBe(0); + expect(stdout.text).toBe("1\talbum\tTrip\n"); + + expect(await filesCommand(context(ente), { collection: "1" })).toBe(0); + expect(stdout.text).toBe("10\timage\tphoto-10.jpg\n"); + }); +}); diff --git a/test/cli/backup.test.ts b/test/cli/backup.test.ts index 977a584..065a2c5 100644 --- a/test/cli/backup.test.ts +++ b/test/cli/backup.test.ts @@ -1482,6 +1482,7 @@ describe("backup album folders", () => { listCollections: () => albums.map((a) => a.collection), listFiles: (id) => albums.find((a) => a.collection.id === id)?.files ?? [], + undecryptable: () => [], original: async (fileID) => { const path = join(root, `source-${fileID}`); writeFileSync(path, `original ${fileID}`); diff --git a/test/client/enumeration.test.ts b/test/client/enumeration.test.ts index 363f27d..49fcb07 100644 --- a/test/client/enumeration.test.ts +++ b/test/client/enumeration.test.ts @@ -10,7 +10,9 @@ * The return shape keeps live records and tombstones apart — `collections` / * `files` are decrypted live records, `deleted` is a plain list of the ids the * server tombstoned. A tombstone carries no decryptable key or metadata, so it - * is a bare id rather than a hollowed-out `Collection` / `EnteFile`. + * is a bare id rather than a hollowed-out `Collection` / `EnteFile`. A live row + * that does not decrypt, or whose metadata is not JSON, is listed under + * `undecryptable` with the reason, so one bad row does not fail the call. * * All tests inject a fake `fetch` and drive a real `Client` (built with * `Client.fromJSON`) so the decryption path runs for real. Live rows are built @@ -91,27 +93,38 @@ const ownedCollectionRow = ( }; }; -/** A live file row inside a collection, keyed under that collection's key. */ +/** + * A live file row inside a collection, keyed under that collection's key. Its + * metadata decrypts to `metadata` when given, otherwise to the JSON a client + * uploads for `title`. + */ const fileRow = ( collectionKey: Uint8Array, - opts: { id: number; title: string; updationTime: number }, + opts: { + id: number; + title: string; + updationTime: number; + metadata?: string; + }, ): Record => { const fileKey = sodium.crypto_secretbox_keygen(); const { ciphertext: encFileKey, nonce: fileKeyNonce } = secretboxEncrypt( fileKey, collectionKey, ); - const metadata = { - title: opts.title, - fileType: 0, - creationTime: opts.updationTime, - modificationTime: opts.updationTime, - }; + const metadata = + opts.metadata ?? + JSON.stringify({ + title: opts.title, + fileType: 0, + creationTime: opts.updationTime, + modificationTime: opts.updationTime, + }); const push = sodium.crypto_secretstream_xchacha20poly1305_init_push(fileKey); const encMeta = sodium.crypto_secretstream_xchacha20poly1305_push( push.state, - new TextEncoder().encode(JSON.stringify(metadata)), + new TextEncoder().encode(metadata), null, sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL, ); @@ -131,6 +144,21 @@ const fileRow = ( }; }; +/** + * `row` with its sealed key replaced by random bytes of the same length, as a + * corrupt row arrives: no key opens it, so the row cannot be decrypted. + */ +const withCorruptKey = ( + row: Record, +): Record => ({ + ...row, + encryptedKey: toBase64( + sodium.randombytes_buf( + sodium.crypto_secretbox_KEYBYTES + sodium.crypto_secretbox_MACBYTES, + ), + ), +}); + /** A tombstone row. Never decrypted, so only these fields are ever read. */ const tombstoneRow = ( id: number, @@ -278,6 +306,58 @@ describe("Client.filesSince", () => { // It gave up after the first page rather than looping. expect(sinceTimes).toEqual(["50"]); }); + + it("leaves out each file it cannot decrypt or parse, with the reason, and decrypts the rest", async () => { + const keys = buildKeys(); + const collectionKey = sodium.crypto_secretbox_keygen(); + + // File 1002's key is sealed under no key quak holds, and file 1003's + // metadata decrypts to text that is not JSON. Neither may cost the + // caller file 1001: each is left out of `files` and reported under + // `undecryptable` with the reason instead of failing the call. + const { fetch } = recordingFetch( + jsonResponse({ + diff: [ + fileRow(collectionKey, { + id: 1001, + title: "fine.jpg", + updationTime: 100, + }), + withCorruptKey( + fileRow(collectionKey, { + id: 1002, + title: "bad-key.jpg", + updationTime: 200, + }), + ), + fileRow(collectionKey, { + id: 1003, + title: "bad-metadata.jpg", + updationTime: 300, + metadata: "not json", + }), + ], + hasMore: false, + }), + ); + const client = Client.fromJSON(snapshotFor(keys), { fetch }); + + const { files, undecryptable, cursor } = await client.filesSince({ + collectionID: 1, + collectionKey, + sinceTime: 0, + }); + + expect(files.map((f) => f.id)).toEqual([1001]); + expect(undecryptable).toEqual([ + { id: 1002, error: expect.stringContaining("wrong secret key") }, + { id: 1003, error: expect.stringContaining("JSON") }, + ]); + // The cursor moves past the two left out, as past any other row, so + // the server does not send them again until its copy changes. The + // caller has to keep what it was told here. + expect(cursor).toBe(300); + }); }); describe("Client.collectionsSince", () => { @@ -331,6 +411,42 @@ describe("Client.collectionsSince", () => { expect(result.cursor).toBe(150); expect(sinceTimes).toEqual(["150"]); }); + + it("leaves out an album it cannot decrypt, with the reason, and decrypts the rest", async () => { + const keys = buildKeys(); + + // Album 2's key is sealed under no key quak holds. Without its key + // neither its name nor its files can be read, so it is left out and + // reported under `undecryptable`; album 1 is unaffected. + const { fetch } = recordingFetch( + jsonResponse({ + collections: [ + ownedCollectionRow(keys.masterKey, { + id: 1, + name: "Vacation", + updationTime: 100, + }), + withCorruptKey( + ownedCollectionRow(keys.masterKey, { + id: 2, + name: "Broken", + updationTime: 200, + }), + ), + ], + }), + ); + const client = Client.fromJSON(snapshotFor(keys), { fetch }); + + const { collections, undecryptable, cursor } = + await client.collectionsSince({ sinceTime: 0 }); + + expect(collections.map((c) => c.name)).toEqual(["Vacation"]); + expect(undecryptable).toEqual([ + { id: 2, error: expect.stringContaining("wrong secret key") }, + ]); + expect(cursor).toBe(200); + }); }); describe("Client list wrappers still hide deletions", () => {