Retry transient network failures with exponential backoff (closes #2)

No retry on 4xx, backoff on 5xx and transport failures, and a deadline on
every request. Before this, one transient 503 or TCP reset failed a file
for good, and a CDN connection that went quiet after accepting the request
blocked `quak backup` forever, because there was no timeout anywhere.

src/retry.ts holds the policy: a classifier that decides whether another
attempt could produce a different answer, and a loop that acts on it with
exponential backoff and full jitter. Retried: 5xx, 408, 429, transport
failures (the errno is read out of the cause chain, which is where Node's
fetch puts it), deadline aborts, and truncated transfers. Not retried:
every other 4xx, and anything unrecognised — a wrongly retried permanent
failure delays every remaining file, while a wrongly abandoned transient
one costs a single file the next run picks up. Attempt count, delays,
sleep and jitter source are all configurable through ApiClientOptions;
sleep being injectable is what lets the suite exercise the policy without
waiting.

Truncation needed a type before it could be classified. streamDecrypt
threw plain Errors whose messages began "download: stream truncated", and
classifying on message text would mean the next reword silently turned
every truncated download into a permanent failure. It now throws
TruncatedStreamError, which lives in src/errors.ts alongside ApiError so
the classifier can recognise both without importing the modules that
import it; api/client.ts re-exports ApiError, so it stays one class and
every existing import path still resolves.

Downloads retry the request, the stream consumption and the decryption
together. Only the first of those happens inside ApiClient: a socket
reset after the headers arrived throws in streamDecrypt, and retrying the
request alone would never see it. The client's own retry is switched off
for those two calls so the budgets do not multiply into sixteen requests
per file, and the atomic write stays outside the loop so a download that
took three attempts still performs one write and one rename.

Non-idempotent requests are not blindly replayed. postJSON and putJSON
reach create-session, two-factor/verify — which burns one of a few
second-factor attempts — and files/thumbnail, so they retry only when the
connection was never established and the server provably never saw the
request. putFile is exempt and retries fully: a presigned PUT stores one
whole object at one key, with no partial state to damage. It now throws
ApiError with the status, as do the two null-body paths, which previously
threw bare Errors that nothing could classify.

Timeouts come from AbortSignal.timeout(), renewed per attempt: 30s for
JSON and upload calls, 10 minutes for file bodies, since a value short
enough to keep a hung API call from stalling a backup would cancel a
legitimate multi-gigabyte download. The download deadline is enforced
over the body rather than only the headers, by racing each read against
the signal, so the guarantee does not depend on the fetch implementation
tearing down a stream it already handed over.

listMissingThumbnails now separates a genuine 404 from an exhausted
retry. Its bare catch reported both as missing, which after this change
would have let a few minutes of 500s talk fix-missing-thumbnails into
regenerating and re-uploading thumbnails that were fine. runBackup and
runMetadataBackup are untouched: the retry sits below them and their
per-file resilience is unchanged.

src/api/client.ts

@@ -1,8 +1,33 @@
+import { ApiError } from "../errors.js";
+import {
+    isSafeToReplay,
+    resolveRetryOptions,
+    withRetry,
+    type ResolvedRetryOptions,
+    type RetryOptions,
+} from "../retry.js";
+
+// `ApiError` is defined in `src/errors.ts` so that the retry classifier can
+// recognise it without importing this module, which imports the classifier.
+// It is re-exported here because this is where callers have always imported it
+// from, and it must remain one class: a second copy would make `instanceof`
+// fail in the classifier and every 5xx would look permanent.
+export { ApiError };
+
 const DEFAULT_API_ORIGIN = "https://api.ente.io";
 const DEFAULT_FILES_ORIGIN = "https://files.ente.io";
 const DEFAULT_THUMBS_ORIGIN = "https://thumbnails.ente.io";
 const CLIENT_PACKAGE = "berlin.sneak.quak";
 
+// Two deadlines rather than one, because a single number cannot serve both
+// jobs. Thirty seconds is generous for a JSON call and short enough that a
+// hung API connection cannot stall a backup for long. A file body is a
+// different shape of problem: the deadline has to cover the whole transfer,
+// which for a large video on a slow link is minutes, so a value sane for JSON
+// would cancel legitimate downloads.
+export const DEFAULT_REQUEST_TIMEOUT_MS = 30_000;
+export const DEFAULT_DOWNLOAD_TIMEOUT_MS = 600_000;
+
 export interface ApiClientOptions {
     apiOrigin?: string;
     filesOrigin?: string;
@@ -10,33 +35,78 @@ export interface ApiClientOptions {
     authToken?: string;
     fetch?: typeof globalThis.fetch;
     userAgent?: string;
+    retry?: RetryOptions;
+    requestTimeoutMs?: number;
+    downloadTimeoutMs?: number;
 }
 
-export class ApiError extends Error {
-    readonly status: number;
-    readonly code?: string;
-    readonly requestID?: string;
-    readonly body?: unknown;
-    constructor(
-        message: string,
-        status: number,
-        opts?: { code?: string; requestID?: string; body?: unknown },
-    ) {
-        super(message);
-        this.name = "ApiError";
-        this.status = status;
-        this.code = opts?.code;
-        this.requestID = opts?.requestID;
-        this.body = opts?.body;
-    }
+export interface StreamOptions {
+    // Opt out of this client's own retry. Exactly one caller wants that: the
+    // download layer, which retries the request, the stream consumption and
+    // the decryption as one unit. Leaving both layers enabled would multiply
+    // the budgets — four attempts each becoming sixteen requests per file.
+    retry?: boolean;
 }
 
+// Enforce a deadline over a response body, not merely over its headers.
+//
+// `getFileStream` returns as soon as headers arrive; the bytes are pulled
+// later, in the download layer. Whether the signal passed to `fetch` also
+// tears down the body afterwards is up to the fetch implementation, so this
+// wrapper makes it a property of quak instead: every read races the signal,
+// and an abort errors the stream with the abort reason — which the retry
+// classifier recognises.
+const deadlineStream = (
+    body: ReadableStream<Uint8Array>,
+    signal: AbortSignal,
+): ReadableStream<Uint8Array> => {
+    const reader = body.getReader();
+    let rejectOnAbort: (reason: unknown) => void = () => undefined;
+    const aborted = new Promise<never>((_resolve, reject) => {
+        rejectOnAbort = reject;
+    });
+    // The abort may fire when nothing is awaiting `aborted` — after the body
+    // has been read in full, say. Without this, that rejection would surface
+    // as an unhandled rejection and take the process down.
+    void aborted.catch(() => undefined);
+
+    const onAbort = (): void => rejectOnAbort(signal.reason);
+    if (signal.aborted) onAbort();
+    else signal.addEventListener("abort", onAbort, { once: true });
+    const release = (): void => signal.removeEventListener("abort", onAbort);
+
+    return new ReadableStream<Uint8Array>({
+        async pull(controller) {
+            try {
+                const next = await Promise.race([reader.read(), aborted]);
+                if (next.done) {
+                    release();
+                    controller.close();
+                    return;
+                }
+                controller.enqueue(next.value);
+            } catch (err) {
+                release();
+                await reader.cancel(err).catch(() => undefined);
+                throw err;
+            }
+        },
+        async cancel(reason) {
+            release();
+            await reader.cancel(reason);
+        },
+    });
+};
+
 export class ApiClient {
     private readonly apiOrigin: string;
     private readonly isCustomOrigin: boolean;
     private readonly filesOrigin: string;
     private readonly thumbsOrigin: string;
     private readonly _fetch: typeof globalThis.fetch;
+    private readonly retry: ResolvedRetryOptions;
+    private readonly requestTimeoutMs: number;
+    private readonly downloadTimeoutMs: number;
     private token: string | undefined;
 
     constructor(opts?: ApiClientOptions) {
@@ -53,6 +123,11 @@ export class ApiClient {
             opts?.thumbsOrigin ?? DEFAULT_THUMBS_ORIGIN
         ).replace(/\/+$/, "");
         this._fetch = opts?.fetch ?? globalThis.fetch;
+        this.retry = resolveRetryOptions(opts?.retry);
+        this.requestTimeoutMs =
+            opts?.requestTimeoutMs ?? DEFAULT_REQUEST_TIMEOUT_MS;
+        this.downloadTimeoutMs =
+            opts?.downloadTimeoutMs ?? DEFAULT_DOWNLOAD_TIMEOUT_MS;
         this.token = opts?.authToken;
     }
 
@@ -64,6 +139,13 @@ export class ApiClient {
         this.token = undefined;
     }
 
+    // The policy this client was configured with, so that a caller wrapping a
+    // whole operation in its own `withRetry` — the download layer — runs under
+    // the same settings rather than under the library defaults.
+    getRetryOptions(): ResolvedRetryOptions {
+        return this.retry;
+    }
+
     private headers(extra?: Record<string, string>): Record<string, string> {
         const h: Record<string, string> = {
             "X-Client-Package": CLIENT_PACKAGE,
@@ -128,38 +210,53 @@ export class ApiClient {
                 }
             }
         }
-        const resp = await this._fetch(url.href, {
-            method: "GET",
-            headers: this.headers(),
-        });
-        await this.throwIfError(resp);
-        return (await resp.json()) as T;
+        // A GET changes nothing, so it is retried under the full policy.
+        return withRetry(async () => {
+            const resp = await this._fetch(url.href, {
+                method: "GET",
+                headers: this.headers(),
+                signal: AbortSignal.timeout(this.requestTimeoutMs),
+            });
+            await this.throwIfError(resp);
+            return (await resp.json()) as T;
+        }, this.retry);
     }
 
     async postJSON<T>(path: string, body: unknown): Promise<T> {
         const url = `${this.apiOrigin}${path}`;
-        const resp = await this._fetch(url, {
-            method: "POST",
-            headers: this.headers({ "Content-Type": "application/json" }),
-            body: JSON.stringify(body),
-        });
-        await this.throwIfError(resp);
-        return (await resp.json()) as T;
+        // Idempotency: this reaches `/users/srp/create-session`,
+        // `/users/two-factor/verify` and `/users/ott`, all of which change
+        // server state — verifying a second factor consumes one of a small
+        // number of attempts. So a POST is replayed only when the failure
+        // proves the request never reached the server, which in practice means
+        // the connection was never established. A 5xx, a mid-flight reset and
+        // a timeout are all left to the caller, because each of them can occur
+        // after the server has already acted.
+        return withRetry(
+            async () => {
+                const resp = await this._fetch(url, {
+                    method: "POST",
+                    headers: this.headers({
+                        "Content-Type": "application/json",
+                    }),
+                    body: JSON.stringify(body),
+                    signal: AbortSignal.timeout(this.requestTimeoutMs),
+                });
+                await this.throwIfError(resp);
+                return (await resp.json()) as T;
+            },
+            { ...this.retry, isRetryable: isSafeToReplay },
+        );
     }
 
-    async getFileStream(fileID: number): Promise<ReadableStream<Uint8Array>> {
+    async getFileStream(
+        fileID: number,
+        opts?: StreamOptions,
+    ): Promise<ReadableStream<Uint8Array>> {
         const url = this.isCustomOrigin
             ? `${this.apiOrigin}/files/download/${fileID}`
             : `${this.filesOrigin}/?fileID=${fileID}`;
-        const resp = await this._fetch(url, {
-            method: "GET",
-            headers: this.headers(),
-        });
-        await this.throwIfError(resp);
-        if (!resp.body) {
-            throw new Error("response body is null");
-        }
-        return resp.body;
+        return this.streamRequest(url, opts);
     }
 
     async getUploadURL(
@@ -173,28 +270,52 @@ export class ApiClient {
     }
 
     async putFile(presignedURL: string, data: Uint8Array): Promise<void> {
-        const resp = await this._fetch(presignedURL, {
-            method: "PUT",
-            headers: {
-                "Content-Type": "application/octet-stream",
-                "Content-Length": String(data.length),
-            },
-            body: data,
-        });
-        if (!resp.ok) {
-            throw new Error(`PUT to presigned URL failed: HTTP ${resp.status}`);
-        }
+        // Idempotent despite being a write: a presigned PUT stores one whole
+        // object at one key in one request, so replaying it either overwrites
+        // the same bytes or lands them for the first time. There is no partial
+        // state to protect, hence the full policy rather than the POST rule.
+        await withRetry(async () => {
+            const resp = await this._fetch(presignedURL, {
+                method: "PUT",
+                headers: {
+                    "Content-Type": "application/octet-stream",
+                    "Content-Length": String(data.length),
+                },
+                body: data,
+                signal: AbortSignal.timeout(this.requestTimeoutMs),
+            });
+            if (!resp.ok) {
+                // An ApiError, not a bare Error: without the status on the
+                // error the upload path cannot be classified at all, and a
+                // 503 from S3 would be indistinguishable from a bug.
+                throw new ApiError(
+                    `PUT to presigned URL failed: HTTP ${resp.status}`,
+                    resp.status,
+                );
+            }
+        }, this.retry);
     }
 
     async putJSON<T>(path: string, body: unknown): Promise<T> {
         const url = `${this.apiOrigin}${path}`;
-        const resp = await this._fetch(url, {
-            method: "PUT",
-            headers: this.headers({ "Content-Type": "application/json" }),
-            body: JSON.stringify(body),
-        });
-        await this.throwIfError(resp);
-        return (await resp.json()) as T;
+        // Same idempotency rule as `postJSON`, for the same reason: this
+        // reaches `/files/thumbnail`, which registers an uploaded thumbnail
+        // against a file.
+        return withRetry(
+            async () => {
+                const resp = await this._fetch(url, {
+                    method: "PUT",
+                    headers: this.headers({
+                        "Content-Type": "application/json",
+                    }),
+                    body: JSON.stringify(body),
+                    signal: AbortSignal.timeout(this.requestTimeoutMs),
+                });
+                await this.throwIfError(resp);
+                return (await resp.json()) as T;
+            },
+            { ...this.retry, isRetryable: isSafeToReplay },
+        );
     }
 
     async updateThumbnail(
@@ -210,18 +331,36 @@ export class ApiClient {
 
     async getThumbnailStream(
         fileID: number,
+        opts?: StreamOptions,
     ): Promise<ReadableStream<Uint8Array>> {
         const url = this.isCustomOrigin
             ? `${this.apiOrigin}/files/preview/${fileID}`
             : `${this.thumbsOrigin}/?fileID=${fileID}`;
-        const resp = await this._fetch(url, {
-            method: "GET",
-            headers: this.headers(),
-        });
-        await this.throwIfError(resp);
-        if (!resp.body) {
-            throw new Error("response body is null");
-        }
-        return resp.body;
+        return this.streamRequest(url, opts);
+    }
+
+    private async streamRequest(
+        url: string,
+        opts?: StreamOptions,
+    ): Promise<ReadableStream<Uint8Array>> {
+        const once = async (): Promise<ReadableStream<Uint8Array>> => {
+            // A fresh deadline per attempt, so a retry gets the whole budget
+            // rather than the remainder of the one that just expired.
+            const signal = AbortSignal.timeout(this.downloadTimeoutMs);
+            const resp = await this._fetch(url, {
+                method: "GET",
+                headers: this.headers(),
+                signal,
+            });
+            await this.throwIfError(resp);
+            if (!resp.body) {
+                // Carries the status, and is not retryable: a response that
+                // arrived without a body is malformed, and asking again
+                // produces the same malformed response.
+                throw new ApiError("response body is null", resp.status);
+            }
+            return deadlineStream(resp.body, signal);
+        };
+        return opts?.retry === false ? once() : withRetry(once, this.retry);
     }
 }