Retry transient network failures with exponential backoff (closes #2)

No retry on 4xx, backoff on 5xx and transport failures, and a deadline on
every request. Before this, one transient 503 or TCP reset failed a file
for good, and a CDN connection that went quiet after accepting the request
blocked `quak backup` forever, because there was no timeout anywhere.

src/retry.ts holds the policy: a classifier that decides whether another
attempt could produce a different answer, and a loop that acts on it with
exponential backoff and full jitter. Retried: 5xx, 408, 429, transport
failures (the errno is read out of the cause chain, which is where Node's
fetch puts it), deadline aborts, and truncated transfers. Not retried:
every other 4xx, and anything unrecognised — a wrongly retried permanent
failure delays every remaining file, while a wrongly abandoned transient
one costs a single file the next run picks up. Attempt count, delays,
sleep and jitter source are all configurable through ApiClientOptions;
sleep being injectable is what lets the suite exercise the policy without
waiting.

Truncation needed a type before it could be classified. streamDecrypt
threw plain Errors whose messages began "download: stream truncated", and
classifying on message text would mean the next reword silently turned
every truncated download into a permanent failure. It now throws
TruncatedStreamError, which lives in src/errors.ts alongside ApiError so
the classifier can recognise both without importing the modules that
import it; api/client.ts re-exports ApiError, so it stays one class and
every existing import path still resolves.

Downloads retry the request, the stream consumption and the decryption
together. Only the first of those happens inside ApiClient: a socket
reset after the headers arrived throws in streamDecrypt, and retrying the
request alone would never see it. The client's own retry is switched off
for those two calls so the budgets do not multiply into sixteen requests
per file, and the atomic write stays outside the loop so a download that
took three attempts still performs one write and one rename.

Non-idempotent requests are not blindly replayed. postJSON and putJSON
reach create-session, two-factor/verify — which burns one of a few
second-factor attempts — and files/thumbnail, so they retry only when the
connection was never established and the server provably never saw the
request. putFile is exempt and retries fully: a presigned PUT stores one
whole object at one key, with no partial state to damage. It now throws
ApiError with the status, as do the two null-body paths, which previously
threw bare Errors that nothing could classify.

Timeouts come from AbortSignal.timeout(), renewed per attempt: 30s for
JSON and upload calls, 10 minutes for file bodies, since a value short
enough to keep a hung API call from stalling a backup would cancel a
legitimate multi-gigabyte download. The download deadline is enforced
over the body rather than only the headers, by racing each read against
the signal, so the guarantee does not depend on the fetch implementation
tearing down a stream it already handed over.

listMissingThumbnails now separates a genuine 404 from an exhausted
retry. Its bare catch reported both as missing, which after this change
would have let a few minutes of 500s talk fix-missing-thumbnails into
regenerating and re-uploading thumbnails that were fine. runBackup and
runMetadataBackup are untouched: the retry sits below them and their
per-file resilience is unchanged.

src/download/index.ts

@@ -9,6 +9,8 @@ import {
     STREAM_CHUNK_SIZE,
     streamTagFinal,
 } from "../crypto/index.js";
+import { TruncatedStreamError } from "../errors.js";
+import { withRetry } from "../retry.js";
 import type { ApiClient } from "../api/client.js";
 import type { EnteFile } from "../model/types.js";
 
@@ -65,7 +67,7 @@ const streamDecrypt = async (
                 try {
                     pulled = pullStreamChunk(state, buffer);
                 } catch (err) {
-                    throw new Error(
+                    throw new TruncatedStreamError(
                         `download: stream truncated: response body ended with ${buffer.length} trailing bytes that did not authenticate as a final chunk (transfer stopped mid-chunk, or the data is corrupt)`,
                         { cause: err },
                     );
@@ -85,13 +87,13 @@ const streamDecrypt = async (
     // Returning a short plaintext here would put a corrupt file on disk that
     // later backup runs would treat as complete.
     if (chunksPulled === 0) {
-        throw new Error(
+        throw new TruncatedStreamError(
             "download: stream truncated: response body contained no secretstream chunks",
         );
     }
     const tagFinal = streamTagFinal();
     if (lastTag !== tagFinal) {
-        throw new Error(
+        throw new TruncatedStreamError(
             `download: stream truncated: last chunk tag ${lastTag}, expected TAG_FINAL (${tagFinal})`,
         );
     }
@@ -128,15 +130,49 @@ const writeAtomic = async (
     }
 };
 
+// Fetch a stream and decrypt it, retrying the whole sequence.
+//
+// The request is only the first third of a download. `getXStream` returns as
+// soon as headers arrive, and the bytes are pulled here, so a socket reset
+// mid-body — the dominant failure mode for multi-megabyte photos over a CDN —
+// throws in `streamDecrypt` and never reaches `ApiClient` at all. Retrying the
+// request alone would miss it entirely.
+//
+// The client's own retry is therefore switched off for these two calls: with
+// both layers active the budgets would multiply, and the library default of
+// four attempts would mean sixteen requests for one file. The policy comes
+// from the client so a caller that configured one gets it here too.
+//
+// A retry starts the file over from byte zero: the secretstream pull state is
+// not resumable and there is no Range support on these endpoints.
+const fetchAndDecrypt = async (
+    api: ApiClient,
+    openStream: () => Promise<ReadableStream<Uint8Array>>,
+    header: Uint8Array,
+    key: Uint8Array,
+): Promise<Uint8Array> =>
+    withRetry(async () => {
+        const stream = await openStream();
+        return streamDecrypt(stream, header, key);
+    }, api.getRetryOptions());
+
 export const downloadFile = async (
     api: ApiClient,
     file: EnteFile,
     outPath?: string,
 ): Promise<DownloadResult> => {
     const resolvedPath = outPath ?? file.metadata.title;
-    const stream = await api.getFileStream(file.id);
     const header = fromBase64(file.file.decryptionHeader);
-    const plaintext = await streamDecrypt(stream, header, file.key);
+    const plaintext = await fetchAndDecrypt(
+        api,
+        () => api.getFileStream(file.id, { retry: false }),
+        header,
+        file.key,
+    );
+    // Outside the retry, deliberately: only the attempt that produced a
+    // complete, authenticated plaintext gets to stage a temporary file, so a
+    // download that needed three tries still performs exactly one write and
+    // one rename.
     await writeAtomic(resolvedPath, plaintext);
     return { path: resolvedPath, bytesWritten: plaintext.length };
 };
@@ -147,9 +183,13 @@ export const downloadThumbnail = async (
     outPath?: string,
 ): Promise<DownloadResult> => {
     const resolvedPath = outPath ?? `thumb_${file.metadata.title}`;
-    const stream = await api.getThumbnailStream(file.id);
     const header = fromBase64(file.thumbnail.decryptionHeader);
-    const plaintext = await streamDecrypt(stream, header, file.key);
+    const plaintext = await fetchAndDecrypt(
+        api,
+        () => api.getThumbnailStream(file.id, { retry: false }),
+        header,
+        file.key,
+    );
     await writeAtomic(resolvedPath, plaintext);
     return { path: resolvedPath, bytesWritten: plaintext.length };
 };