Compare commits

..
3 Commits
Author SHA1 Message Date
clawbot 4c9a42db49 The binary make install builds retries a refused or unresolvable connection (closes #206)
check / check (push) Waiting to run
Bun's `fetch`, which `~/bin/quak` runs under, rejects with a plain `Error`
that has no `cause` and a code of bun's own. Running the built binary against
local listeners gave `ConnectionRefused` for a refused connection and for a
failed TLS handshake, and `ConnectionRefused` or `FailedToOpenSocket` for a
name that does not resolve. Both codes join `TRANSPORT_CODES` and
`CONNECT_CODES` in `src/retry.ts`, so they are retried, replayed, and called
transient by `classify`. A reset already came as `ECONNRESET` and a deadline
as a `TimeoutError`; the tests pin every shape found.

Judgement call: a failed TLS handshake is replayed, as no request byte was
sent.
Not changed: a certificate bun rejects is not retried.

Model: opus-5-5
2026-10-16 01:43:48 +00:00
clawbot d016dbfaa7 A network failure is printed and recorded with its reason (closes #196)
check / check (push) Waiting to run
Node's `fetch` rejects every failure below HTTP as `fetch failed` and puts
the reason on `cause`. `describeError` in `src/errors.ts` gives the message
followed by the messages on the cause chain, and for an `AggregateError`,
which Node gives when every address of a host fails, the message of each
address. `run` prints it, and the backup logs and records it. `classify`
reads that text and calls a failure transient when `hasTransportCode`, now
exported by `src/retry.ts`, finds a transport errno on the chain. A failed
ML data fetch keeps its error as the cause. The bounded walk along the
chain moved from `src/retry.ts` to `src/errors.ts`.

A truncated download's message leaves the authentication failure to its
cause.

Model: opus-5-5
2026-10-16 03:23:16 +02:00
clawbot 5a940d4e4e README says that make install needs nix (closes #200)
check / check (push) Waiting to run
`make install` runs `make build-bin`, which runs bun through
`nix-shell -p bun`, so on a machine without nix it stops with
`nix-shell: not found`, and the README named no such prerequisite.
Getting Started now says what its commands need and that `make install`
also needs nix, with a link to its download page. "Running the backup
from cron" says the same where it introduces `make install`, and that a
bun already installed does not help. The Makefile is unchanged, as the
plan on the issue decided. TODO.md's Next Step names the four bugs that
are open instead of saying none is.

Judgement call: "node 22 or later" is the `engines` entry in package.json.

Model: opus-5-5
2026-10-16 01:40:19 +02:00
11 changed files with 591 additions and 74 deletions
+51 -15
View File
@@ -19,6 +19,10 @@ encrypting and uploading them back to the server.
## Getting Started ## Getting Started
The commands below need git, node 22 or later, and yarn. `make install`, which
builds the single binary that "Running the backup from cron" uses, also needs
[nix](https://nixos.org/download).
```bash ```bash
git clone https://git.eeqj.de/sneak/quak.git git clone https://git.eeqj.de/sneak/quak.git
cd quak cd quak
@@ -86,9 +90,12 @@ enumeration/download calls) is exported too and documented under Design below.
## Running the backup from cron ## Running the backup from cron
`quak backup` never prompts, so cron can run it. `make install` builds quak as a `quak backup` never prompts, so cron can run it. `make install` builds quak as a
single binary and copies it to `~/bin/quak`. Log in once with it, as the user single binary and copies it to `~/bin/quak`. Besides what Getting Started needs,
the cron job will run as; the session is saved in that user's data directory it needs [nix](https://nixos.org/download), even where bun is already installed:
(see "Session handling"): `make install` runs `make build-bin`, which bundles the built `dist/` with bun
run through `nix-shell -p bun`. Log in once with `~/bin/quak`, as the user the
cron job will run as; the session is saved in that user's data directory (see
"Session handling"):
```bash ```bash
~/bin/quak login ~/bin/quak login
@@ -134,6 +141,20 @@ matters.
| `2` | Another backup of the same directory is running. This run sent no request and changed nothing. | | `2` | Another backup of the same directory is running. This run sent no request and changed nothing. |
| `3` | There is no usable session: none is saved, the saved one is corrupt, or the server no longer accepts it. Run `quak login` as the user the cron job runs as. | | `3` | There is no usable session: none is saved, the saved one is corrupt, or the server no longer accepts it. Run `quak login` as the user the cron job runs as. |
`<message>` is the error's message, followed by the message of each error that
caused it. Under Node, as `yarn quak` runs quak, `fetch` reports every failure
below HTTP as just `fetch failed` and gives the reason as its cause, so a run
that could not reach the server prints, for example,
`quak: fetch failed: connect ECONNREFUSED 1.2.3.4:443`. When the server has
several addresses and none of them could be reached, the line gives the reason
for each, separated by `, `. A file whose download failed that way is in
`failures.json` with that error, as a transient failure. So is each file with no
ML data cached when the ML data fetch failed that way, its error starting
`ML data: `. The `~/bin/quak` that `make install` builds runs under bun, and
prints bun's own wording for such a failure, with no reason after it, such as
`quak: Unable to connect. Is the computer able to access the url?` for a refused
connection. It records such a failure in `failures.json` as transient too.
After a `1`, the next run fetches each missing original again, fetches the ML After a `1`, the next run fetches each missing original again, fetches the ML
data that is not cached, and rebuilds the album links. An album quak could not data that is not cached, and rebuilds the album links. An album quak could not
decrypt, or a file in `failures.json` because quak could not decrypt it, keeps decrypt, or a file in `failures.json` because quak could not decrypt it, keeps
@@ -347,7 +368,8 @@ quak/
exif.ts EXIF read from an image's bytes with exifreader exif.ts EXIF read from an image's bytes with exifreader
mldata-fetch.ts fetch + decrypt per-file ML data mldata-fetch.ts fetch + decrypt per-file ML data
filename.ts safe file names from server metadata filename.ts safe file names from server metadata
errors.ts error types shared across layers errors.ts error types shared across layers, and how an error
is described
retry.ts retry classifier + exponential backoff with jitter retry.ts retry classifier + exponential backoff with jitter
thumbnails.ts detect + regenerate missing thumbnails thumbnails.ts detect + regenerate missing thumbnails
client.ts high-level Client class assembled from the above client.ts high-level Client class assembled from the above
@@ -474,6 +496,17 @@ request is repeated only when repeating it could produce a different answer:
declining to retry a transient one costs a single file that the next run picks declining to retry a transient one costs a single file that the next run picks
up. up.
The `~/bin/quak` that `make install` builds runs under bun, whose `fetch`
rejects with a plain `Error` that has no `cause` and carries a code of bun's
own. quak retries a refused connection, a name that does not resolve, a reset
connection and a deadline under bun as it does under Node. Bun reports a refused
connection, and a TLS handshake that failed, as `ConnectionRefused`. For a name
that does not resolve it gives `ConnectionRefused` on some requests and
`FailedToOpenSocket` on others. A connection reset or closed before the response
ended is `ECONNRESET`, as under Node, and a deadline is the same `TimeoutError`.
A certificate bun rejects, reported with a code such as
`DEPTH_ZERO_SELF_SIGNED_CERT`, is not retried.
Backoff is exponential with full jitter: the delay before retry _n_ is Backoff is exponential with full jitter: the delay before retry _n_ is
`random() * min(maxDelayMs, baseDelayMs * 2 ** (n - 1))`. The exponential term `random() * min(maxDelayMs, baseDelayMs * 2 ** (n - 1))`. The exponential term
is the ceiling and the wait is drawn below it, so a client that lost many is the ceiling and the wait is drawn below it, so a client that lost many
@@ -521,17 +554,20 @@ second-factor attempts. They are retried only when every errno in the error's
`cause` chain is one of the three that establish no TCP connection to the server `cause` chain is one of the three that establish no TCP connection to the server
ever existed, so no request byte can have been transmitted: `ENOTFOUND` and ever existed, so no request byte can have been transmitted: `ENOTFOUND` and
`EAI_AGAIN` (name resolution produced no address) and `ECONNREFUSED` (the peer `EAI_AGAIN` (name resolution produced no address) and `ECONNREFUSED` (the peer
refused the connection). A 5xx, a mid-flight reset and a deadline are all left refused the connection). Under bun these failures are `ConnectionRefused` and
to the caller, because each of them can happen after the server has already `FailedToOpenSocket`. Bun also gives `ConnectionRefused` for a TLS handshake
acted. These two do not follow redirects either: a redirect means the server that failed; the TCP connection existed then, but the request is only sent once
already received the request, so it is reported as an error and not retried. The the handshake is done. A 5xx, a mid-flight reset and a deadline are all left to
routing errnos `EHOSTUNREACH`, `ENETUNREACH` and `ENETDOWN` are excluded for the the caller, because each of them can happen after the server has already acted.
same reason, despite looking like connect-time failures: on Linux an ICMP These two do not follow redirects either: a redirect means the server already
unreachable arriving mid-flight, or a local interface going down after the received the request, so it is reported as an error and not retried. The routing
request was written, delivers them on an already-established socket. They stay errnos `EHOSTUNREACH`, `ENETUNREACH` and `ENETDOWN` are excluded for the same
retryable for the idempotent calls. `putFile` is exempt: a presigned PUT stores reason, despite looking like connect-time failures: on Linux an ICMP unreachable
one whole object at one key in one request, so replaying it has no partial state arriving mid-flight, or a local interface going down after the request was
to damage. written, delivers them on an already-established socket. They stay retryable for
the idempotent calls. `putFile` is exempt: a presigned PUT stores one whole
object at one key in one request, so replaying it has no partial state to
damage.
A download is retried as a whole — request, stream consumption, and decryption — A download is retried as a whole — request, stream consumption, and decryption —
because a socket reset after the response headers have arrived surfaces in the because a socket reset after the response headers have arrived surfaces in the
+28 -1
View File
@@ -17,7 +17,10 @@ pre-1.0
# Next Step # Next Step
None: no implementation work is open. The check of quak against a live Ente Four bugs are open: https://git.eeqj.de/sneak/quak/issues/197,
https://git.eeqj.de/sneak/quak/issues/198,
https://git.eeqj.de/sneak/quak/issues/199 and
https://git.eeqj.de/sneak/quak/issues/205. The check of quak against a live Ente
account, https://git.eeqj.de/sneak/quak/issues/187, needs an account only sneak account, https://git.eeqj.de/sneak/quak/issues/187, needs an account only sneak
has, so it waits on him. has, so it waits on him.
@@ -26,6 +29,30 @@ declares one.
# Completed Steps # Completed Steps
- 2026-10-16: The `~/bin/quak` that `make install` builds retries a refused
connection and a name that does not resolve, and the backup records either as
transient (issue 206). Bun's `fetch` reports them with codes of its own,
`ConnectionRefused` and `FailedToOpenSocket`, which `src/retry.ts` now counts
as transport failures that are safe to replay. Bun already gave `ECONNRESET`
for a reset and a `TimeoutError` for a deadline. The shapes were found by
running the binary against local listeners.
- 2026-10-15: A network failure is printed and recorded with its reason, not as
just `fetch failed` (issue 196). `describeError` in `src/errors.ts` gives an
error's message followed by the message of each error on its `cause` chain,
such as `fetch failed: connect ECONNREFUSED 1.2.3.4:443`, and each address's
reason when a host has several and none could be reached. `run` prints that,
and the backup logs it and records it in `failures.json`, where `classify` now
calls the failure transient when the chain holds an errno that
`hasTransportCode` in `src/retry.ts` recognises, for a failed ML data fetch
too. The walk along the chain moved from `src/retry.ts` to `src/errors.ts`,
which both use. Tests refuse every connection of a backup, then only one
file's download, and fail the ML data fetch.
- 2026-10-15: The README says that `make install` needs nix (issue 200), in
Getting Started and in "Running the backup from cron": `make build-bin` runs
bun through `nix-shell -p bun`. The `Makefile` is unchanged.
- 2026-10-14: `lib.backup()`, and so `quak backup`, works on up to the content - 2026-10-14: `lib.backup()`, and so `quak backup`, works on up to the content
pool's limit of originals at once, 5 by default, instead of one at a time pool's limit of originals at once, 5 by default, instead of one at a time
(issue 182). That many workers take files from one shared list, each doing all (issue 182). That many workers take files from one shared list, each doing all
+22 -15
View File
@@ -82,6 +82,7 @@ import {
init, init,
} from "./crypto/index.js"; } from "./crypto/index.js";
import { removeLeftoverTempFiles } from "./download/index.js"; import { removeLeftoverTempFiles } from "./download/index.js";
import { describeError } from "./errors.js";
import { sanitizeFileName, withExtension } from "./filename.js"; import { sanitizeFileName, withExtension } from "./filename.js";
import { import {
copyAtomic, copyAtomic,
@@ -94,6 +95,7 @@ import type { Undecryptable } from "./library/store.js";
import { extractImageMetadata } from "./metadata-backup.js"; import { extractImageMetadata } from "./metadata-backup.js";
import type { MLData } from "./mldata-fetch.js"; import type { MLData } from "./mldata-fetch.js";
import type { Collection, EnteFile, FileMetadata } from "./model/types.js"; import type { Collection, EnteFile, FileMetadata } from "./model/types.js";
import { hasTransportCode } from "./retry.js";
export type ProgressCallback = (message: string) => void; export type ProgressCallback = (message: string) => void;
@@ -219,11 +221,13 @@ const isPresent = (path: string): boolean => {
// Best-effort classification for the ledger. Retryable server/network problems // Best-effort classification for the ledger. Retryable server/network problems
// are transient; refusals and local filesystem/decrypt errors are permanent; // are transient; refusals and local filesystem/decrypt errors are permanent;
// anything else is unknown. Both the error code and message are inspected. // anything else is unknown. The error code and the error as `describeError`
// gives it are inspected, and so is each errno on the error's cause chain,
// where Node's `fetch` puts it.
const classify = (err: unknown): FailureClass => { const classify = (err: unknown): FailureClass => {
if (hasTransportCode(err)) return "transient";
const e = err as NodeJS.ErrnoException; const e = err as NodeJS.ErrnoException;
const text = const text = `${e?.code ?? ""} ${describeError(err)}`.toLowerCase();
`${e?.code ?? ""} ${err instanceof Error ? err.message : String(err)}`.toLowerCase();
if ( if (
/timeout|timed out|econnreset|econnrefused|econnaborted|network|socket|eai_again|throttl|temporarily|429|500|502|503|504/.test( /timeout|timed out|econnreset|econnrefused|econnaborted|network|socket|eai_again|throttl|temporarily|429|500|502|503|504/.test(
text, text,
@@ -241,9 +245,6 @@ const classify = (err: unknown): FailureClass => {
return "unknown"; return "unknown";
}; };
const errorMessage = (err: unknown): string =>
err instanceof Error ? err.message : String(err);
// Ensure `linkPath` is a symlink to `target`, rebuilding a missing, wrong, or // Ensure `linkPath` is a symlink to `target`, rebuilding a missing, wrong, or
// non-symlink entry. Throws on failure (a directory in the way, no permission) // non-symlink entry. Throws on failure (a directory in the way, no permission)
// so the caller records it and moves on rather than aborting the run. // so the caller records it and moves on rather than aborting the run.
@@ -470,7 +471,7 @@ const imageMetadataFor = async (
const bytes = await readFile(originalPath); const bytes = await readFile(originalPath);
return { imageMetadata: extractImageMetadata(bytes) ?? {} }; return { imageMetadata: extractImageMetadata(bytes) ?? {} };
} catch (err) { } catch (err) {
return { imageMetadataError: errorMessage(err) }; return { imageMetadataError: describeError(err) };
} }
}; };
@@ -610,7 +611,7 @@ const runLockedBackup = async (
// and thumbnail both fail this run must not double its attempt count // and thumbnail both fail this run must not double its attempt count
// or appear twice in errors. // or appear twice in errors.
if (failedThisRun.has(fileID)) return; if (failedThisRun.has(fileID)) return;
const error = errorMessage(err); const error = describeError(err);
errors.push({ fileID, title, collection, error }); errors.push({ fileID, title, collection, error });
const prior = ledger.get(fileID); const prior = ledger.get(fileID);
ledger.set(fileID, { ledger.set(fileID, {
@@ -681,7 +682,7 @@ const runLockedBackup = async (
} }
} catch (err) { } catch (err) {
log( log(
`FAILED verifying original ${file.metadata.title}: ${errorMessage(err)}`, `FAILED verifying original ${file.metadata.title}: ${describeError(err)}`,
); );
recordFailure( recordFailure(
file.id, file.id,
@@ -717,7 +718,9 @@ const runLockedBackup = async (
} }
downloaded++; downloaded++;
} catch (err) { } catch (err) {
log(`FAILED original ${file.metadata.title}: ${errorMessage(err)}`); log(
`FAILED original ${file.metadata.title}: ${describeError(err)}`,
);
recordFailure( recordFailure(
file.id, file.id,
file.metadata.title, file.metadata.title,
@@ -785,12 +788,14 @@ const runLockedBackup = async (
// instead and is recorded as failed. The next run fetches its ML data // instead and is recorded as failed. The next run fetches its ML data
// again because none is cached. // again because none is cached.
if (includeOriginals) { if (includeOriginals) {
let mlDataFailure: unknown;
let mlDataError: string | undefined; let mlDataError: string | undefined;
try { try {
log("Fetching ML data..."); log("Fetching ML data...");
await lib.fetchMLData(); await lib.fetchMLData();
} catch (err) { } catch (err) {
mlDataError = errorMessage(err); mlDataFailure = err;
mlDataError = describeError(err);
log(`FAILED ML data: ${mlDataError}`); log(`FAILED ML data: ${mlDataError}`);
} }
for (const file of distinct.values()) { for (const file of distinct.values()) {
@@ -814,7 +819,7 @@ const runLockedBackup = async (
file.id, file.id,
file.metadata.title, file.metadata.title,
collectionName.get(file.collectionID) ?? "", collectionName.get(file.collectionID) ?? "",
new Error(`ML data: ${mlDataError}`), new Error("ML data", { cause: mlDataFailure }),
); );
} else { } else {
writeSidecar(path, file, { mlData }, image); writeSidecar(path, file, { mlData }, image);
@@ -846,7 +851,7 @@ const runLockedBackup = async (
downloadDirectory, downloadDirectory,
); );
} catch (err) { } catch (err) {
log(`FAILED removing old album directories: ${errorMessage(err)}`); log(`FAILED removing old album directories: ${describeError(err)}`);
} }
for (const c of collections) { for (const c of collections) {
@@ -868,7 +873,9 @@ const runLockedBackup = async (
try { try {
removeStaleLinks(colDir, new Set(linkNames), downloadDirectory); removeStaleLinks(colDir, new Set(linkNames), downloadDirectory);
} catch (err) { } catch (err) {
log(`FAILED removing old links in ${c.name}: ${errorMessage(err)}`); log(
`FAILED removing old links in ${c.name}: ${describeError(err)}`,
);
} }
const metaFiles = files.map((f) => ({ const metaFiles = files.map((f) => ({
@@ -885,7 +892,7 @@ const runLockedBackup = async (
); );
} catch (err) { } catch (err) {
log( log(
`FAILED symlink ${c.name}/${linkName}: ${errorMessage(err)}`, `FAILED symlink ${c.name}/${linkName}: ${describeError(err)}`,
); );
recordFailure( recordFailure(
link.file.id, link.file.id,
+7 -6
View File
@@ -2,14 +2,17 @@
import type { Writable } from "node:stream"; import type { Writable } from "node:stream";
import { ApiError } from "./api/client.js"; import { ApiError } from "./api/client.js";
import { describeError } from "./errors.js";
// Run a command and exit with its code once stdout/stderr have drained. // Run a command and exit with its code once stdout/stderr have drained.
// Exiting before the drain can truncate piped output, and the library can keep // Exiting before the drain can truncate piped output, and the library can keep
// the event loop alive after a command returns, so a plain return could hang. // the event loop alive after a command returns, so a plain return could hang.
// An error the command throws is printed as one `quak: MESSAGE` line, without // An error the command throws is printed as one `quak: MESSAGE` line, without
// the stack trace, and exits 1. A 401 from the server means it no longer // the stack trace, and exits 1. MESSAGE is the error as `describeError` gives
// accepts the saved session: that prints one line saying to log in again and // it, so it carries the reason Node's `fetch` puts on `cause`. A 401 from the
// exits 3, as a missing or corrupt session file does. // server means it no longer accepts the saved session: that prints one line
// saying to log in again and exits 3, as a missing or corrupt session file
// does.
export const run = async ( export const run = async (
command: Promise<number>, command: Promise<number>,
stdout: Writable, stdout: Writable,
@@ -26,9 +29,7 @@ export const run = async (
); );
code = 3; code = 3;
} else { } else {
stderr.write( stderr.write(`quak: ${describeError(err)}\n`);
`quak: ${err instanceof Error ? err.message : String(err)}\n`,
);
code = 1; code = 1;
} }
} }
+5 -3
View File
@@ -135,15 +135,17 @@ const streamDecrypt = async (
// cannot tell a partial chunk from a corrupt one, so this // cannot tell a partial chunk from a corrupt one, so this
// is reported as the truncation it almost always is, with // is reported as the truncation it almost always is, with
// the authentication failure kept as the error's cause. // the authentication failure kept as the error's cause.
// Only the pull is guarded: a sink failure on a chunk // The message leaves that failure to the cause, which
// that did authenticate is a disk error, not a // `describeError` appends to what `quak backup` prints and
// records. Only the pull is guarded: a sink failure on a
// chunk that did authenticate is a disk error, not a
// truncation. // truncation.
let pulled; let pulled;
try { try {
pulled = pullStreamChunk(state, buffer); pulled = pullStreamChunk(state, buffer);
} catch (err) { } catch (err) {
throw new TruncatedStreamError( throw new TruncatedStreamError(
`download: stream truncated: response body ended with ${buffer.length} trailing bytes that did not authenticate as a final chunk (transfer stopped mid-chunk, or the data is corrupt)`, `download: stream truncated: response body ended with ${buffer.length} trailing bytes (transfer stopped mid-chunk, or the data is corrupt)`,
{ cause: err }, { cause: err },
); );
} }
+71 -3
View File
@@ -1,10 +1,13 @@
// Error types that more than one layer of quak needs to recognise. // Error types that more than one layer of quak needs to recognise, and how an
// error is described in what quak prints and in `failures.json`.
// //
// They live here rather than beside the code that throws them so that the // The types live here rather than beside the code that throws them so that the
// retry classifier can identify them without importing the HTTP client or the // retry classifier can identify them without importing the HTTP client or the
// download layer — both of which import the classifier. `ApiError` is // download layer — both of which import the classifier. `ApiError` is
// re-exported from `src/api/client.ts`, which is where callers have always // re-exported from `src/api/client.ts`, which is where callers have always
// imported it from and where it still belongs conceptually. // imported it from and where it still belongs conceptually. The walk along an
// error's `cause` chain is here too, because the retry classifier reads the
// chain for errnos and `describeError` for messages.
export class ApiError extends Error { export class ApiError extends Error {
readonly status: number; readonly status: number;
@@ -43,3 +46,68 @@ export class TruncatedStreamError extends Error {
this.name = "TruncatedStreamError"; this.name = "TruncatedStreamError";
} }
} }
// `cause` is an arbitrary user-settable property and nothing prevents it from
// forming a cycle, so the walk is bounded. Hanging the process would be worse
// than misclassifying an error or describing only part of it.
const MAX_CAUSE_DEPTH = 8;
// The error and each error on its `cause` chain, top first, at most
// `MAX_CAUSE_DEPTH` of them. `complete` is false when the walk stopped at the
// depth limit with more of the chain still below it.
export const causeChain = (
err: unknown,
): { chain: object[]; complete: boolean } => {
const chain: object[] = [];
let current: unknown = err;
for (let depth = 0; depth < MAX_CAUSE_DEPTH; depth++) {
if (current === null || typeof current !== "object") {
return { chain, complete: true };
}
chain.push(current);
const { cause } = current as { cause?: unknown };
if (cause === current) return { chain, complete: true };
current = cause;
}
return { chain, complete: current === null || typeof current !== "object" };
};
// An error as quak prints it and records it in `failures.json`: its message,
// then the message of each error on its `cause` chain, joined with ": ". Node's
// `fetch` rejects with just "fetch failed" and puts the reason on `cause`, so
// this gives "fetch failed: connect ECONNREFUSED 1.2.3.4:443". When the host
// has several addresses and every one fails, the cause is an `AggregateError`
// with an empty message and one error per address in `errors`, so an
// `AggregateError` is given as the messages of its errors, joined with ", ".
// A cause that is not an object, such as a thrown string, is given as
// `String(cause)`. Each message is trimmed of the white space around it, as
// OpenSSL ends its own with a newline. A message the text already holds is
// left out.
export const describeError = (err: unknown): string => {
if (!(err instanceof Error)) return String(err);
const { chain } = causeChain(err);
const messages = chain
.slice(1)
.map((cause) =>
cause instanceof AggregateError
? cause.errors
.map((e: unknown) =>
(e instanceof Error ? e.message : String(e)).trim(),
)
.join(", ")
: (cause as { message?: unknown }).message,
);
// `causeChain` stops at a cause that is not an object, so that one is read
// from the last error on the chain.
const last = chain[chain.length - 1] as { cause?: unknown };
if (last.cause !== undefined && typeof last.cause !== "object") {
messages.push(String(last.cause));
}
let text = err.message.trim();
for (const message of messages) {
if (typeof message !== "string") continue;
const trimmed = message.trim();
if (!text.includes(trimmed)) text += `: ${trimmed}`;
}
return text;
};
+47 -30
View File
@@ -10,7 +10,7 @@
// declining to retry a transient one costs a single file that the next run // declining to retry a transient one costs a single file that the next run
// picks up anyway. When the evidence is ambiguous, fail fast. // picks up anyway. When the evidence is ambiguous, fail fast.
import { ApiError, TruncatedStreamError } from "./errors.js"; import { ApiError, causeChain, TruncatedStreamError } from "./errors.js";
export interface RetryOptions { export interface RetryOptions {
// Total calls, not retries: `attempts: 1` disables retrying. // Total calls, not retries: `attempts: 1` disables retrying.
@@ -58,6 +58,13 @@ export const resolveRetryOptions = (
}); });
// Transport failures: the request did not complete, for reasons below HTTP. // Transport failures: the request did not complete, for reasons below HTTP.
//
// The errnos are Node's. The binary `make install` builds runs under bun,
// whose `fetch` rejects with a plain `Error` that has no `cause` and a `code`
// of bun's own; `CONNECT_CODES` says what bun's two codes mean. For a
// connection reset or closed before the response ended, bun gives Node's
// `ECONNRESET`. These were found by running that binary, built with bun
// 1.3.13, against listeners on 127.0.0.1.
const TRANSPORT_CODES = new Set([ const TRANSPORT_CODES = new Set([
"ECONNRESET", "ECONNRESET",
"ECONNABORTED", "ECONNABORTED",
@@ -70,12 +77,19 @@ const TRANSPORT_CODES = new Set([
"ENETUNREACH", "ENETUNREACH",
"ENETRESET", "ENETRESET",
"ENETDOWN", "ENETDOWN",
// Bun's.
"ConnectionRefused",
"FailedToOpenSocket",
]); ]);
// The subset of the above that can only be reported before a TCP connection // The subset of the above that can only be reported before any request byte
// exists, and therefore before any request byte could have been written: name // could have been written. Node reports these before a TCP connection exists:
// resolution produced no address (`ENOTFOUND`, `EAI_AGAIN`) or the peer // name resolution produced no address (`ENOTFOUND`, `EAI_AGAIN`) or the peer
// refused the connection with an RST to the SYN (`ECONNREFUSED`). // refused the connection with an RST to the SYN (`ECONNREFUSED`). Bun gives
// `ConnectionRefused` for a refused connection, and `ConnectionRefused` or
// `FailedToOpenSocket` for a name that does not resolve. It also gives
// `ConnectionRefused` for a TLS handshake that failed, after the TCP
// connection exists, but the request is only sent once the handshake is done.
// //
// The routing errnos — `EHOSTUNREACH`, `ENETUNREACH`, `ENETDOWN` — are // The routing errnos — `EHOSTUNREACH`, `ENETUNREACH`, `ENETDOWN` — are
// deliberately absent even though they look like connect-time failures. On // deliberately absent even though they look like connect-time failures. On
@@ -87,12 +101,13 @@ const TRANSPORT_CODES = new Set([
// ambiguity this set exists to exclude. They stay in `TRANSPORT_CODES`, so // ambiguity this set exists to exclude. They stay in `TRANSPORT_CODES`, so
// they remain retryable for idempotent calls; only replay eligibility is // they remain retryable for idempotent calls; only replay eligibility is
// narrowed. See `isSafeToReplay`. // narrowed. See `isSafeToReplay`.
const CONNECT_CODES = new Set(["ENOTFOUND", "EAI_AGAIN", "ECONNREFUSED"]); const CONNECT_CODES = new Set([
"ENOTFOUND",
// `cause` is an arbitrary user-settable property and nothing prevents it from "EAI_AGAIN",
// forming a cycle, so the walk is bounded. Hanging the process would be a "ECONNREFUSED",
// worse outcome than any misclassification. "ConnectionRefused",
const MAX_CAUSE_DEPTH = 8; "FailedToOpenSocket",
]);
// Collect every `code` in an error's cause chain. undici does not put the // Collect every `code` in an error's cause chain. undici does not put the
// errno on the error it throws — it hangs the underlying socket error off // errno on the error it throws — it hangs the underlying socket error off
@@ -101,20 +116,21 @@ const MAX_CAUSE_DEPTH = 8;
// connection permanent. `complete` is false when the walk stopped at the // connection permanent. `complete` is false when the walk stopped at the
// depth limit with more of the chain still below it. // depth limit with more of the chain still below it.
const causeCodes = (err: unknown): { codes: string[]; complete: boolean } => { const causeCodes = (err: unknown): { codes: string[]; complete: boolean } => {
const { chain, complete } = causeChain(err);
const codes: string[] = []; const codes: string[] = [];
let current: unknown = err; for (const link of chain) {
for (let depth = 0; depth < MAX_CAUSE_DEPTH; depth++) { const { code } = link as { code?: unknown };
if (current === null || typeof current !== "object") {
return { codes, complete: true };
}
const { code, cause } = current as { code?: unknown; cause?: unknown };
if (typeof code === "string") codes.push(code); if (typeof code === "string") codes.push(code);
if (cause === current) return { codes, complete: true };
current = cause;
} }
return { codes, complete: current === null || typeof current !== "object" }; return { codes, complete };
}; };
// Does the error's cause chain hold the code of a transport failure, one of
// `TRANSPORT_CODES`? `isRetryable` retries such an error, and `classify` in
// `src/backup.ts` calls it transient.
export const hasTransportCode = (err: unknown): boolean =>
causeCodes(err).codes.some((code) => TRANSPORT_CODES.has(code));
const isAbort = (err: unknown): boolean => { const isAbort = (err: unknown): boolean => {
if (err === null || typeof err !== "object") return false; if (err === null || typeof err !== "object") return false;
const { name } = err as { name?: unknown }; const { name } = err as { name?: unknown };
@@ -156,9 +172,10 @@ export const isRetryable = (err: unknown): boolean => {
// deliberately literal. Demanding a recognised `cause` instead would // deliberately literal. Demanding a recognised `cause` instead would
// classify real network failures as permanent and fail backups that should // classify real network failures as permanent and fail backups that should
// have succeeded; the cost of the imprecision is bounded by the attempt // have succeeded; the cost of the imprecision is bounded by the attempt
// count. // count. Bun's fetch rejects with a plain `Error` instead, which is
// recognised by its code (see `TRANSPORT_CODES`).
if (err instanceof TypeError) return true; if (err instanceof TypeError) return true;
return causeCodes(err).codes.some((code) => TRANSPORT_CODES.has(code)); return hasTransportCode(err);
}; };
// Could the first attempt already have taken effect on the server? // Could the first attempt already have taken effect on the server?
@@ -167,18 +184,18 @@ export const isRetryable = (err: unknown): boolean => {
// `postJSON` and `putJSON` use this for every `POST` and `PUT` listed in the // `postJSON` and `putJSON` use this for every `POST` and `PUT` listed in the
// README under "Endpoints used"; verifying a second factor, for one, consumes // README under "Endpoints used"; verifying a second factor, for one, consumes
// one of a small number of attempts. They are replayed only on the failures in // one of a small number of attempts. They are replayed only on the failures in
// `CONNECT_CODES`, which establish that no TCP connection to the server ever // `CONNECT_CODES`, which establish that no request byte was ever transmitted:
// existed: there was no address to connect to, or the peer refused the // there was no address to connect to, the peer refused the connection
// connection outright. A request byte cannot have been transmitted, so the // outright, or, under bun, the TLS handshake failed. The server cannot have
// server cannot have acted. // acted.
// //
// Everything else is ambiguous. A 5xx proves the server did process the // Everything else is ambiguous. A 5xx proves the server did process the
// request. A reset or a broken pipe can arrive after it was fully sent and // request. A reset or a broken pipe can arrive after it was fully sent and
// acted on. A routing errno can be delivered on an established socket. A // acted on. A routing errno can be delivered on an established socket. A
// deadline says nothing at all about the server's state. So every errno in the // deadline says nothing at all about the server's state. So every code in the
// cause chain must be a connect errno: one other errno anywhere in the chain // cause chain must be one of `CONNECT_CODES`: one other code anywhere in the
// is doubt, and doubt is not replayed. A chain longer than the walk is doubt // chain is doubt, and doubt is not replayed. A chain longer than the walk is
// too: the links below the limit were never read. // doubt too: the links below the limit were never read.
export const isSafeToReplay = (err: unknown): boolean => { export const isSafeToReplay = (err: unknown): boolean => {
const { codes, complete } = causeCodes(err); const { codes, complete } = causeCodes(err);
return ( return (
+188
View File
@@ -329,6 +329,41 @@ const downloadOf = (input: RequestInfo | URL): number | undefined => {
: undefined; : undefined;
}; };
// The error Node gives for a connection to `address` that was refused.
const refusedAt = (address: string): Error =>
Object.assign(new Error(`connect ECONNREFUSED ${address}`), {
code: "ECONNREFUSED",
});
// What Node's `fetch` rejects with when the server refuses the connection, as
// for every failure below HTTP: a TypeError that says only "fetch failed",
// with the reason, and its errno, as its cause. When the server has several
// addresses and each one refuses, the cause is an AggregateError with an empty
// message, one error per address and the errno of the first. `reason` is the
// error as quak prints it and records it in `failures.json`.
const REFUSED = [
{
addresses: "its one address",
error: (): TypeError =>
new TypeError("fetch failed", { cause: refusedAt("1.2.3.4:443") }),
reason: "fetch failed: connect ECONNREFUSED 1.2.3.4:443",
},
{
addresses: "each of its two addresses",
error: (): TypeError =>
new TypeError("fetch failed", {
cause: Object.assign(
new AggregateError([
refusedAt("2001:db8::1:443"),
refusedAt("1.2.3.4:443"),
]),
{ code: "ECONNREFUSED" },
),
}),
reason: "fetch failed: connect ECONNREFUSED 2001:db8::1:443, connect ECONNREFUSED 1.2.3.4:443",
},
];
// The fake server, pushing onto `downloads` the ID of each file whose original // The fake server, pushing onto `downloads` the ID of each file whose original
// is downloaded. // is downloaded.
const recordingDownloads = const recordingDownloads =
@@ -619,6 +654,159 @@ describe("quak backup", () => {
expect(readdirSync(dir)).toEqual([]); expect(readdirSync(dir)).toEqual([]);
}); });
// The server refuses every connection, at its one address or at each of
// several. quak tries each request again, here without waiting, and then
// the run stops at the refresh it starts with. The line it prints gives
// the reason Node's `fetch` put on the error's cause, so a cron log tells
// a refused connection from a name that did not resolve, and the exit
// code is 1.
it.each(REFUSED)(
"prints why the refresh could not reach the server at $addresses, and exits 1",
async ({ error, reason }) => {
const dir = mkdtempSync(join(root, "backup-"));
const fetch: typeof globalThis.fetch = async () => {
throw error();
};
const { code, stderr } = await backup(dir, {}, fetch);
expect(code).toBe(1);
expect(stderr).toBe(
"Starting backup...\n" +
"Refreshing library...\n" +
`quak: ${reason}\n`,
);
},
);
// The files CDN refuses every connection for toast.mp4, at its one
// address or at each of several. Once its retries are spent, the file
// goes into `failures.json` with the reason, as a transient failure,
// since a later run may well reach the CDN, and the lines the backup
// prints for it give the same reason. The rest of the account is backed
// up, and the exit code is 1.
it.each(REFUSED)(
"records why a download could not reach the server at $addresses in failures.json, as transient, and exits 1",
async ({ error, reason }) => {
let attempts = 0;
const fetch: typeof globalThis.fetch = async (input, init) => {
if (downloadOf(input) === 200) {
attempts++;
throw error();
}
return server(input, init);
};
const dir = mkdtempSync(join(root, "backup-"));
const { code, stderr } = await backup(dir, {}, fetch);
expect(code).toBe(1);
expect(attempts).toBe(UNATTENDED_RETRY_OPTIONS.attempts);
const failures = readJSON(join(dir, "failures.json"));
expect(Object.keys(failures.files)).toEqual(["200"]);
expect(failures.files["200"]).toMatchObject({
fileID: 200,
title: "toast.mp4",
classification: "transient",
attempts: 1,
error: reason,
});
expect(stderr).toContain(`FAILED original toast.mp4: ${reason}\n`);
expect(stderr).toContain(
`[Friend's Wedding] toast.mp4 (id 200): ${reason}\n`,
);
expect(readFileSync(join(dir, DAY, "2024-03-01.100.jpg"))).toEqual(
Buffer.from(JPEG_WITH_EXIF),
);
expect(existsSync(join(dir, DAY, "2024-03-01.200.mp4"))).toBe(
false,
);
},
);
// The download of toast.mp4 fails as bun's `fetch` reports a failure, in
// the binary `make install` builds: a plain `Error` with bun's own code
// and message, and no cause. These are what bun gave for a refused
// connection, a name that did not resolve and a reset connection. The
// file is tried as often as under Node, and goes into `failures.json`
// with that message, as a transient failure.
it.each([
[
"ConnectionRefused",
"Unable to connect. Is the computer able to access the url?",
],
["FailedToOpenSocket", "Was there a typo in the url or port?"],
[
"ECONNRESET",
"The socket connection was closed unexpectedly. For more information, pass `verbose: true` in the second argument to fetch()",
],
])(
"records a download that failed with bun's %s in failures.json, as transient",
async (bunCode, message) => {
let attempts = 0;
const fetch: typeof globalThis.fetch = async (input, init) => {
if (downloadOf(input) === 200) {
attempts++;
throw Object.assign(new Error(message), {
code: bunCode,
errno: 0,
});
}
return server(input, init);
};
const dir = mkdtempSync(join(root, "backup-"));
const { code } = await backup(dir, {}, fetch);
expect(code).toBe(1);
expect(attempts).toBe(UNATTENDED_RETRY_OPTIONS.attempts);
const failures = readJSON(join(dir, "failures.json"));
expect(failures.files["200"]).toMatchObject({
classification: "transient",
error: message,
});
},
);
// The name of Ente's API server does not resolve when quak fetches the ML
// data, after every original has been backed up. No file has ML data
// cached yet, so each goes into `failures.json` with the reason, as a
// transient failure, as a download that could not reach the server does,
// and the exit code is 1.
it("records why the ML data fetch could not reach the server in failures.json, as transient, and exits 1", async () => {
const fetch: typeof globalThis.fetch = async (input, init) => {
if (new URL(String(input)).pathname === "/files/data/fetch") {
throw new TypeError("fetch failed", {
cause: Object.assign(
new Error("getaddrinfo ENOTFOUND api.ente.io"),
{ code: "ENOTFOUND" },
),
});
}
return server(input, init);
};
const dir = mkdtempSync(join(root, "backup-"));
const { code, stderr } = await backup(dir, {}, fetch);
expect(code).toBe(1);
const error =
"ML data: fetch failed: getaddrinfo ENOTFOUND api.ente.io";
expect(stderr).toContain(`FAILED ${error}\n`);
const failures = readJSON(join(dir, "failures.json"));
expect(
Object.values(failures.files).map((f) => [
f.fileID,
f.classification,
f.error,
]),
).toEqual([
[100, "transient", error],
[200, "transient", error],
[300, "transient", error],
]);
});
// Something changed beach.jpg on disk after it was backed up. A run with // Something changed beach.jpg on disk after it was backed up. A run with
// --verify hashes every stored original, finds that one no longer matches // --verify hashes every stored original, finds that one no longer matches
// the hash Ente recorded, says so, and downloads it again; it put back the // the hash Ente recorded, says so, and downloads it again; it put back the
+80
View File
@@ -48,6 +48,78 @@ describe("run", () => {
); );
}); });
// Node's `fetch` rejects every failure below HTTP with a TypeError that
// says only "fetch failed", and hangs the reason, with its errno, off
// `cause`. The printed line carries the reason, so a cron log tells a
// refused connection from a name that did not resolve.
it("prints the reason a fetch failed, from the error's cause", async () => {
const refused = new TypeError("fetch failed", {
cause: Object.assign(
new Error("connect ECONNREFUSED 1.2.3.4:443"),
{ code: "ECONNREFUSED" },
),
});
const result = await runToExit(Promise.reject(refused));
expect(result).toEqual({
code: 1,
stdout: "",
stderr: "quak: fetch failed: connect ECONNREFUSED 1.2.3.4:443\n",
});
});
// When the host has several addresses and every one refuses, Node's
// `fetch` gives as the cause an AggregateError with an empty message and
// one error per address. The printed line gives each address's reason.
it("prints the reason for each address a fetch could not reach", async () => {
const refusedAt = (address: string): Error =>
Object.assign(new Error(`connect ECONNREFUSED ${address}`), {
code: "ECONNREFUSED",
});
const refused = new TypeError("fetch failed", {
cause: Object.assign(
new AggregateError([
refusedAt("2001:db8::1:443"),
refusedAt("1.2.3.4:443"),
]),
{ code: "ECONNREFUSED" },
),
});
const result = await runToExit(Promise.reject(refused));
expect(result).toEqual({
code: 1,
stdout: "",
stderr: "quak: fetch failed: connect ECONNREFUSED 2001:db8::1:443, connect ECONNREFUSED 1.2.3.4:443\n",
});
});
// An error that already quotes its cause's message is printed as it is.
it("does not print a cause's message twice", async () => {
const reset = new Error("read ECONNRESET");
const wrapped = new Error(`download: file 7: ${reset.message}`, {
cause: reset,
});
const result = await runToExit(Promise.reject(wrapped));
expect(result.stderr).toBe("quak: download: file 7: read ECONNRESET\n");
});
// OpenSSL ends its message for a failed TLS connection, such as one to a
// server that answers in plain HTTP, with a newline. The printed line is
// still one line.
it("prints a cause's message without the white space around it", async () => {
const wrongVersion = new TypeError("fetch failed", {
cause: Object.assign(
new Error(
"C0B73A55A47F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:354:\n",
),
{ code: "ERR_SSL_WRONG_VERSION_NUMBER" },
),
});
const result = await runToExit(Promise.reject(wrongVersion));
expect(result.stderr).toBe(
"quak: fetch failed: C0B73A55A47F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:354:\n",
);
});
it("prints a thrown value that is not an Error", async () => { it("prints a thrown value that is not an Error", async () => {
const result = await runToExit(Promise.reject("offline")); const result = await runToExit(Promise.reject("offline"));
expect(result).toEqual({ expect(result).toEqual({
@@ -57,6 +129,14 @@ describe("run", () => {
}); });
}); });
// A cause that is not an Error, such as a thrown string, is printed as a
// thrown value that is not an Error is.
it("prints a cause that is not an Error", async () => {
const wrapped = new Error("ML data", { cause: "offline" });
const result = await runToExit(Promise.reject(wrapped));
expect(result.stderr).toBe("quak: ML data: offline\n");
});
it("on a 401 from the server says to run quak login, on one line, and exits 3", async () => { it("on a 401 from the server says to run quak login, on one line, and exits 3", async () => {
const result = await runToExit( const result = await runToExit(
Promise.reject(new ApiError("unauthorized", 401)), Promise.reject(new ApiError("unauthorized", 401)),
+10 -1
View File
@@ -74,7 +74,11 @@ import {
} from "vitest"; } from "vitest";
import { init, toBase64, STREAM_CHUNK_SIZE } from "../../src/crypto/index.js"; import { init, toBase64, STREAM_CHUNK_SIZE } from "../../src/crypto/index.js";
import { ApiClient } from "../../src/api/client.js"; import { ApiClient } from "../../src/api/client.js";
import { ApiError, TruncatedStreamError } from "../../src/errors.js"; import {
ApiError,
describeError,
TruncatedStreamError,
} from "../../src/errors.js";
import type { RetryOptions } from "../../src/retry.js"; import type { RetryOptions } from "../../src/retry.js";
import { import {
downloadFile, downloadFile,
@@ -834,6 +838,11 @@ describe.each(entryPoints)(
expect(((err as Error).cause as Error).message).toMatch( expect(((err as Error).cause as Error).message).toMatch(
/authentication failed/i, /authentication failed/i,
); );
// As quak prints it and records it in `failures.json`, the error
// states the authentication failure once, from its cause.
expect(describeError(err)).toMatch(
/^download: stream truncated: response body ended with \d+ trailing bytes \(transfer stopped mid-chunk, or the data is corrupt\): secretstream chunk authentication failed$/,
);
// And, as with every other failure, the destination is untouched // And, as with every other failure, the destination is untouched
// and no staged temp file survives. // and no staged temp file survives.
+82
View File
@@ -77,6 +77,14 @@ const recordingSleep = (): {
const errnoError = (code: string, message = code): Error => const errnoError = (code: string, message = code): Error =>
Object.assign(new Error(message), { code }); Object.assign(new Error(message), { code });
/**
* An error shaped like what bun's `fetch` rejects with for a failure below
* HTTP: a plain `Error` with bun's own message, a `code`, an `errno` of 0, and
* no `cause`.
*/
const bunError = (code: string, message: string): Error =>
Object.assign(new Error(message), { code, errno: 0 });
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Classification // Classification
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -244,6 +252,80 @@ describe("isRetryable: transport failures", () => {
}); });
}); });
describe("isRetryable and isSafeToReplay: bun's fetch errors", () => {
// The binary `make install` builds runs quak under bun, whose `fetch`
// does not reject as Node's does. Each shape here was found by running
// that binary, built with bun 1.3.13, against listeners on 127.0.0.1 and
// a name under `.invalid`.
it("retries and replays a refused connection", () => {
// Bun gives the same error when the TLS handshake fails. The TCP
// connection existed then, but the request is only sent once the
// handshake is done, so replaying it is as safe.
const refused = bunError(
"ConnectionRefused",
"Unable to connect. Is the computer able to access the url?",
);
expect(isRetryable(refused)).toBe(true);
expect(isSafeToReplay(refused)).toBe(true);
});
it("retries and replays a name that does not resolve, with either code bun gives", () => {
// For the same name bun gave `ConnectionRefused` on some requests and
// `FailedToOpenSocket` on others.
for (const unresolved of [
bunError(
"ConnectionRefused",
"Unable to connect. Is the computer able to access the url?",
),
bunError(
"FailedToOpenSocket",
"Was there a typo in the url or port?",
),
]) {
expect(isRetryable(unresolved)).toBe(true);
expect(isSafeToReplay(unresolved)).toBe(true);
}
});
it("retries a connection closed before the response ended, and does not replay it", () => {
// Bun gives Node's errno, `ECONNRESET`, for a reset and for a close,
// before the response or part-way through the body. The request may
// already have reached the server.
const reset = bunError(
"ECONNRESET",
"The socket connection was closed unexpectedly. For more information, pass `verbose: true` in the second argument to fetch()",
);
expect(isRetryable(reset)).toBe(true);
expect(isSafeToReplay(reset)).toBe(false);
});
it("retries a request that ran out of time, and does not replay it", () => {
// What bun's fetch rejects with when `AbortSignal.timeout()`, which
// bounds each JSON request, fires.
const timedOut = new DOMException(
"The operation timed out.",
"TimeoutError",
);
expect(isRetryable(timedOut)).toBe(true);
expect(isSafeToReplay(timedOut)).toBe(false);
});
it("does not retry a certificate bun rejects", () => {
// A certificate that fails verification fails the same way however
// often it is asked. Under Node it is retried only because every
// `TypeError` is.
expect(
isRetryable(
bunError(
"DEPTH_ZERO_SELF_SIGNED_CERT",
"self signed certificate",
),
),
).toBe(false);
});
});
describe("isRetryable: stream truncation versus corruption", () => { describe("isRetryable: stream truncation versus corruption", () => {
it("retries a truncated stream", () => { it("retries a truncated stream", () => {
// Truncation is a transfer that stopped early. The bytes that did // Truncation is a transfer that stopped early. The bytes that did