Compare commits

..
6 Commits
Author SHA1 Message Date
sneak c6bc1a6265 Test: the exit code when an album is the only record left out
check / check (push) Successful in 3m11s
A backup whose only record quak cannot decrypt is an album exits 1 with
nothing in failures.json, and 0 once the album decrypts or is deleted.
The test of a file in two albums now also checks that the album where its
copy does not decrypt neither links the original nor lists the file.

Model: opus-5-5
2026-10-14 03:03:27 +00:00
sneak 1833009c08 README: say which records quak could not decrypt lib.backup() reports
It reports each such album, and each such file in its scope that has no
copy that decrypted, not every record the library left out.

Model: opus-5-5
2026-10-14 02:54:22 +00:00
sneak 564ec13b69 Back up from a copy that decrypted even outside a run's albums; README wording
A file whose copy in an album in scope does not decrypt is now backed up
from a copy that decrypted in any album, not only one in scope; with no
such copy it is still recorded as failed. The README now says the exit
code stays at 1 until the server sends a copy that decrypts or deletes it,
and that only a run without --json prints the FAILED album line.
StoredMetadata.undecryptable is required, like the other fields.

Model: opus-5-5
2026-10-14 02:54:21 +00:00
sneak cfb3d31047 Test: a backup limited to one album stores a file whose copy there does not decrypt
A run of lib.backup() limited to "Trip" must store a file whose copy in
Trip does not decrypt when its copy in an album the run skips does. Today
the run neither stores it nor records it as failed.

Model: opus-5-5
2026-10-14 02:54:21 +00:00
sneak 349e364109 One album or file quak cannot decrypt no longer stops a refresh or a backup (closes #181)
collectionsSince and filesSince put a row that does not decrypt or parse in
an undecryptable list with the reason instead of throwing. The refresh
leaves it out, an album with its files, and the store keeps the list in
metadata.json until the server sends a copy that decrypts or deletes it,
since the next refresh asks only for changes. lib.backup() records each
such file in failures.json as "could not be decrypted: <reason>", unless a
copy in another album decrypted, logs one line per such album and lists
their IDs in undecryptableAlbums; quak backup exits 1 while any remain.
Tests use a stand-in Ente server with real encryption.

Model: opus-5-5
2026-10-14 02:54:21 +00:00
clawbot 288825b11a quak backup no longer refreshes its library on a timer (closes #183)
check / check (push) Successful in 3m39s
The backup opened its library with the default 3-second refresh interval,
so for its whole run it asked the server for changes every few seconds,
and each refresh started another ML data pass. It now opens it with
refreshIntervalSeconds: Infinity. The library takes Infinity as no
background timer, since setTimeout would turn it into 1 ms. A refresh
interval of 3600, as the read commands use, would still refresh once an
hour in a backup that runs for hours.

Model: opus-5-5
2026-10-14 04:10:14 +02:00
6 changed files with 215 additions and 5 deletions
+3 -2
View File
@@ -803,7 +803,8 @@ subsequent runs, existing originals are skipped. If a download fails, the error
is logged and the backup continues with the next file. The exit code is non-zero
if any files failed. `quak backup` opens its library with the thumbnail and
originals precache off, so the only file content it fetches is the originals the
backup stores.
backup stores. It also opens it with no background timer, so it asks the server
for changes only as it starts, not again while it downloads.
With `--verify`, or `lib.backup({ verify: true })`, a run also hashes each
original already at its save path the way a download is checked (see "On-disk
@@ -884,7 +885,7 @@ background, so an unreachable server does not block opening.
| `client` | required | the account client (a `Client`, or any `LibraryClient`) |
| `cacheDirectory` | `quak/<userID>` under the per-user cache directory | where `metadata.json` and the content cache live |
| `downloadDirectory` | `photos` in the working directory | root of the save paths; an original stored there counts as cached |
| `refreshIntervalSeconds` | `3` | background refresh cadence |
| `refreshIntervalSeconds` | `3` | background refresh cadence; `Infinity` for no background timer |
| `precacheThumbnails` | `true` | prefetch every thumbnail, newest first |
| `precacheOriginals` | `true` | prefetch the favorites album and the latest-window originals |
| `precacheOriginalsDays` | `7` | length in days of that latest window |
+7
View File
@@ -36,6 +36,13 @@ declares one.
remain. A file whose copy in another album decrypts is backed up as usual. The
read commands answer from what decrypted.
- 2026-10-14: `quak backup` no longer asks the server for changes every 3
seconds while it runs (issue 183). It opens its library with
`refreshIntervalSeconds: Infinity`, which the library now takes as no
background timer, so the backup refreshes only as it starts and no timed
refresh starts another ML data pass during the run. A test holds a backup's
download open while two hours pass and counts its `/collections/v2` requests.
- 2026-10-14: `test/cli/backup-end-to-end.test.ts` runs `quak backup` as cron
does (issue 186): `backupCommand` and `run`, the real `loadSession` reading a
session saved after logging in, and every request answered by a fake of the
+3
View File
@@ -427,10 +427,13 @@ export const backupCommand = async (
try {
// The precache is off: the backup fetches what it needs, and must not
// also fill the cache with every thumbnail and the recent originals.
// No refresh starts on a timer: the backup lists its files once, from
// the refresh it waits for, and can run for hours.
const lib = await Library.open({
client,
downloadDirectory: dir,
cacheDirectory: ctx.cacheDir,
refreshIntervalSeconds: Infinity,
precacheThumbnails: false,
precacheOriginals: false,
});
+5 -1
View File
@@ -179,6 +179,9 @@ export interface LibraryOptions {
// directory at open. The content cache treats an original already stored
// at its save path as present.
downloadDirectory?: string;
// How long after one refresh ends the background timer starts the next.
// `Infinity` starts none: the library then refreshes only as it opens and
// when `fresh()` or `backup()` asks.
refreshIntervalSeconds?: number;
onProgress?: RefreshProgressCallback;
// The bounded request pools (issue #45), shared by the ML-data fetch (the
@@ -628,7 +631,8 @@ export class Library {
}
private scheduleNext(): void {
if (this.closed) return;
// `setTimeout` would turn an infinite delay into 1 ms.
if (this.closed || this.intervalMs === Infinity) return;
this.timer = setTimeout(() => {
void this.runRefresh().then(() => this.scheduleNext());
}, this.intervalMs);
+55 -1
View File
@@ -23,7 +23,13 @@
* key replaced by bytes no key opens, or its metadata text that is not JSON.
*/
import { existsSync, mkdtempSync, readFileSync, rmSync } from "node:fs";
import {
existsSync,
mkdtempSync,
readdirSync,
readFileSync,
rmSync,
} from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import sodium from "libsodium-wrappers-sumo";
@@ -253,6 +259,18 @@ const brokenAccount = (): FakeEnte => {
return ente;
};
// Album 1, "Trip", holds file 10, and album 2, "Family", holds file 20. Both
// files decrypt, but album 2's own key does not, so the album is the only
// record left out. All of it as of time 1.
const brokenAlbumAccount = (): FakeEnte => {
const ente = new FakeEnte();
ente.putAlbum(1, "Trip", 1);
ente.putFile(1, 10, 1);
ente.putAlbum(2, "Family", 1, true);
ente.putFile(2, 20, 1);
return ente;
};
// Collects everything written to it.
class Output {
text = "";
@@ -321,6 +339,12 @@ const failures = (): Record<string, { error: string; attempts: number }> => {
const stored = (id: number): boolean =>
existsSync(join(backupDir(), DAY, `2026-03-01.${id}.jpg`));
// The IDs of the files that album `name`'s JSON lists.
const albumFiles = (name: string): number[] =>
JSON.parse(
readFileSync(join(backupDir(), "collections", `${name}.json`), "utf-8"),
).files.map((f: { id: number }) => f.id);
describe("quak backup with records it cannot decrypt", () => {
it("backs up every other file, records each file it cannot decrypt in failures.json, names each such album, and exits 1", async () => {
expect(await backup(brokenAccount())).toBe(1);
@@ -398,6 +422,29 @@ describe("quak backup with records it cannot decrypt", () => {
expect(stderr.text).not.toContain("FAILED");
});
it("exits 1 when the only record it cannot decrypt is an album, and 0 once the album decrypts", async () => {
const ente = brokenAlbumAccount();
// No file is in `failures.json`: the album alone keeps the exit code
// at 1.
expect(await backup(ente)).toBe(1);
expect(failures()).toEqual({});
ente.putAlbum(2, "Family", 2);
expect(await backup(ente)).toBe(0);
expect(stored(20)).toBe(true);
});
it("exits 1 when the only record it cannot decrypt is an album, and 0 once the album is deleted", async () => {
const ente = brokenAlbumAccount();
expect(await backup(ente)).toBe(1);
expect(failures()).toEqual({});
ente.deleteAlbum(2, 2);
expect(await backup(ente)).toBe(0);
});
it("backs up a file that does not decrypt in one album but does in another, and does not report it", async () => {
// File 30 is in albums 1 and 3, each holding its key sealed under the
// album's own key. In album 1 that is corrupt; in album 3 it is not,
@@ -412,6 +459,13 @@ describe("quak backup with records it cannot decrypt", () => {
expect(stored(30)).toBe(true);
expect(failures()).toEqual({});
// Album 1 has no copy of file 30 that decrypts, so it neither links
// the original nor lists the file; album 3 lists it.
expect(readdirSync(join(backupDir(), "collections", "Trip"))).toEqual(
[],
);
expect(albumFiles("Trip")).toEqual([]);
expect(albumFiles("Favorites")).toEqual([30]);
});
it("backs up that file in a run limited to the album where it does not decrypt", async () => {
+142 -1
View File
@@ -25,6 +25,7 @@ import { join } from "node:path";
import { tmpdir } from "node:os";
import { PassThrough } from "node:stream";
import * as jpegJs from "jpeg-js";
import sodium from "libsodium-wrappers-sumo";
import {
describe,
it,
@@ -55,7 +56,7 @@ import { loadSession } from "../../src/cli-session.js";
import type { Client, ClientSnapshot, LoginOptions } from "../../src/client.js";
import { savePath, type ContentSource } from "../../src/library/content.js";
import type { Collection, EnteFile } from "../../src/model/types.js";
import { init, toBase64 } from "../../src/crypto/index.js";
import { encryptBlob, init, toBase64 } from "../../src/crypto/index.js";
import { defaultCacheDirectory } from "../../src/library/index.js";
import { HEIC_WITH_EXIF } from "../exif-heic.js";
import {
@@ -903,6 +904,146 @@ describe("backup", () => {
// Opening the library would have made its cache directory.
expect(existsSync(join(root, "cache"))).toBe(false);
});
// A backup runs for as long as its downloads take, hours on a large
// account. It lists its files from the refresh it waits for as it starts,
// so it must not ask the server for changes again on a timer while it
// runs: each refresh is a request, and each also starts a pass that asks
// again for the ML data of every file Ente holds none for.
//
// A saved session is read back by `loadSession`, its `fetch` a fake of the
// Ente HTTP API serving one album with one photo, encrypted as the server
// serves them. The fake holds the photo's download open while the clock
// moves on by two hours, then serves it. Only `setTimeout` and
// `clearTimeout` are faked: the library's refresh timer runs on them.
it("asks the server for changes only as it starts, however long it runs", async () => {
const masterKey = new Uint8Array(32).fill(1);
const albumKey = new Uint8Array(32).fill(2);
const photoKey = new Uint8Array(32).fill(3);
saveSession(join(root, "session"), {
email: "cli@example.com",
userID: USER_ID,
token: "token",
masterKey: toBase64(masterKey),
secretKey: toBase64(new Uint8Array(32)),
publicKey: toBase64(new Uint8Array(32)),
});
// A secretbox as the server sends one: ciphertext and nonce, base64.
const box = (plaintext: Uint8Array, key: Uint8Array) => {
const nonce = sodium.randombytes_buf(
sodium.crypto_secretbox_NONCEBYTES,
);
return {
ciphertext: toBase64(
sodium.crypto_secretbox_easy(plaintext, nonce, key),
),
nonce: toBase64(nonce),
};
};
const albumKeyBox = box(albumKey, masterKey);
const albumName = box(new TextEncoder().encode("Vacation"), albumKey);
const photoKeyBox = box(photoKey, albumKey);
const metadata = encryptBlob(
new TextEncoder().encode(
JSON.stringify({
title: "beach.jpg",
fileType: 0,
creationTime: 1_700_000_000_000_000,
modificationTime: 1_700_000_000_000_000,
}),
),
photoKey,
);
const content = encryptBlob(
new TextEncoder().encode("the photo's bytes"),
photoKey,
);
const album = {
id: 1,
owner: { id: USER_ID },
encryptedKey: albumKeyBox.ciphertext,
keyDecryptionNonce: albumKeyBox.nonce,
encryptedName: albumName.ciphertext,
nameDecryptionNonce: albumName.nonce,
type: "album",
updationTime: 1,
};
const photo = {
id: 100,
collectionID: 1,
ownerID: USER_ID,
encryptedKey: photoKeyBox.ciphertext,
keyDecryptionNonce: photoKeyBox.nonce,
metadata: {
encryptedData: toBase64(metadata.ciphertext),
decryptionHeader: toBase64(metadata.header),
},
file: { decryptionHeader: toBase64(content.header) },
// The backup fetches no thumbnail.
thumbnail: { decryptionHeader: "" },
updationTime: 1,
};
// Each refresh starts with `collectionsSince`, which sends one
// `/collections/v2` request.
let refreshes = 0;
let downloadRequested!: () => void;
const requested = new Promise<void>((resolve) => {
downloadRequested = resolve;
});
let releaseDownload!: () => void;
const released = new Promise<void>((resolve) => {
releaseDownload = resolve;
});
const json = (body: unknown): Response =>
new Response(JSON.stringify(body), {
headers: { "content-type": "application/json" },
});
const fetch = async (url: string | URL): Promise<Response> => {
const { hostname, pathname } = new URL(String(url));
if (hostname === "files.ente.io") {
downloadRequested();
await released;
return new Response(content.ciphertext);
}
if (pathname === "/collections/v2") {
refreshes++;
return json({ collections: [album] });
}
if (pathname === "/collections/v2/diff") {
return json({ diff: [photo], hasMore: false });
}
// Ente holds no ML data for the photo.
if (pathname === "/files/data/fetch") return json({ data: [] });
return new Response("not found", { status: 404 });
};
const ctx = {
...context(),
// The download's idle deadline is set past the two hours, or quak
// would give up the held download as stalled.
loadSession: (path: string) =>
loadSession(path, {
fetch: fetch as typeof globalThis.fetch,
downloadTimeoutMs: 3 * 3_600_000,
}),
};
vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] });
try {
const backup = backupCommand(ctx, join(root, "backup"), {});
await requested;
vi.advanceTimersByTime(2 * 3_600_000);
releaseDownload();
expect(await backup).toBe(0);
} finally {
vi.useRealTimers();
}
// Two refreshes, both before the download: the library opens onto the
// empty cache with one, and the backup waits for the other before it
// lists its files.
expect(refreshes).toBe(2);
});
});
describe("helper list-missing-thumbnails", () => {