The binary make install builds retries a refused or unresolvable connection (closes #206)
check / check (push) Waiting to run

Bun's `fetch`, which `~/bin/quak` runs under, rejects with a plain `Error`
that has no `cause` and a code of bun's own. Running the built binary against
local listeners gave `ConnectionRefused` for a refused connection and for a
failed TLS handshake, and `ConnectionRefused` or `FailedToOpenSocket` for a
name that does not resolve. Both codes join `TRANSPORT_CODES` and
`CONNECT_CODES` in `src/retry.ts`, so they are retried, replayed, and called
transient by `classify`. A reset already came as `ECONNRESET` and a deadline
as a `TimeoutError`; the tests pin every shape found.

Judgement call: a failed TLS handshake is replayed, as no request byte was
sent.
Not changed: a certificate bun rejects is not retried.

Model: opus-5-5
This commit is contained in:
2026-10-16 01:43:48 +00:00
parent d016dbfaa7
commit 4c9a42db49
5 changed files with 201 additions and 31 deletions
+28 -12
View File
@@ -151,7 +151,9 @@ for each, separated by `, `. A file whose download failed that way is in
`failures.json` with that error, as a transient failure. So is each file with no `failures.json` with that error, as a transient failure. So is each file with no
ML data cached when the ML data fetch failed that way, its error starting ML data cached when the ML data fetch failed that way, its error starting
`ML data: `. The `~/bin/quak` that `make install` builds runs under bun, and `ML data: `. The `~/bin/quak` that `make install` builds runs under bun, and
prints bun's own wording for such a failure. prints bun's own wording for such a failure, with no reason after it, such as
`quak: Unable to connect. Is the computer able to access the url?` for a refused
connection. It records such a failure in `failures.json` as transient too.
After a `1`, the next run fetches each missing original again, fetches the ML After a `1`, the next run fetches each missing original again, fetches the ML
data that is not cached, and rebuilds the album links. An album quak could not data that is not cached, and rebuilds the album links. An album quak could not
@@ -494,6 +496,17 @@ request is repeated only when repeating it could produce a different answer:
declining to retry a transient one costs a single file that the next run picks declining to retry a transient one costs a single file that the next run picks
up. up.
The `~/bin/quak` that `make install` builds runs under bun, whose `fetch`
rejects with a plain `Error` that has no `cause` and carries a code of bun's
own. quak retries a refused connection, a name that does not resolve, a reset
connection and a deadline under bun as it does under Node. Bun reports a refused
connection, and a TLS handshake that failed, as `ConnectionRefused`. For a name
that does not resolve it gives `ConnectionRefused` on some requests and
`FailedToOpenSocket` on others. A connection reset or closed before the response
ended is `ECONNRESET`, as under Node, and a deadline is the same `TimeoutError`.
A certificate bun rejects, reported with a code such as
`DEPTH_ZERO_SELF_SIGNED_CERT`, is not retried.
Backoff is exponential with full jitter: the delay before retry _n_ is Backoff is exponential with full jitter: the delay before retry _n_ is
`random() * min(maxDelayMs, baseDelayMs * 2 ** (n - 1))`. The exponential term `random() * min(maxDelayMs, baseDelayMs * 2 ** (n - 1))`. The exponential term
is the ceiling and the wait is drawn below it, so a client that lost many is the ceiling and the wait is drawn below it, so a client that lost many
@@ -541,17 +554,20 @@ second-factor attempts. They are retried only when every errno in the error's
`cause` chain is one of the three that establish no TCP connection to the server `cause` chain is one of the three that establish no TCP connection to the server
ever existed, so no request byte can have been transmitted: `ENOTFOUND` and ever existed, so no request byte can have been transmitted: `ENOTFOUND` and
`EAI_AGAIN` (name resolution produced no address) and `ECONNREFUSED` (the peer `EAI_AGAIN` (name resolution produced no address) and `ECONNREFUSED` (the peer
refused the connection). A 5xx, a mid-flight reset and a deadline are all left refused the connection). Under bun these failures are `ConnectionRefused` and
to the caller, because each of them can happen after the server has already `FailedToOpenSocket`. Bun also gives `ConnectionRefused` for a TLS handshake
acted. These two do not follow redirects either: a redirect means the server that failed; the TCP connection existed then, but the request is only sent once
already received the request, so it is reported as an error and not retried. The the handshake is done. A 5xx, a mid-flight reset and a deadline are all left to
routing errnos `EHOSTUNREACH`, `ENETUNREACH` and `ENETDOWN` are excluded for the the caller, because each of them can happen after the server has already acted.
same reason, despite looking like connect-time failures: on Linux an ICMP These two do not follow redirects either: a redirect means the server already
unreachable arriving mid-flight, or a local interface going down after the received the request, so it is reported as an error and not retried. The routing
request was written, delivers them on an already-established socket. They stay errnos `EHOSTUNREACH`, `ENETUNREACH` and `ENETDOWN` are excluded for the same
retryable for the idempotent calls. `putFile` is exempt: a presigned PUT stores reason, despite looking like connect-time failures: on Linux an ICMP unreachable
one whole object at one key in one request, so replaying it has no partial state arriving mid-flight, or a local interface going down after the request was
to damage. written, delivers them on an already-established socket. They stay retryable for
the idempotent calls. `putFile` is exempt: a presigned PUT stores one whole
object at one key in one request, so replaying it has no partial state to
damage.
A download is retried as a whole — request, stream consumption, and decryption — A download is retried as a whole — request, stream consumption, and decryption —
because a socket reset after the response headers have arrived surfaces in the because a socket reset after the response headers have arrived surfaces in the
+11 -4
View File
@@ -17,11 +17,10 @@ pre-1.0
# Next Step # Next Step
Five bugs are open: https://git.eeqj.de/sneak/quak/issues/197, Four bugs are open: https://git.eeqj.de/sneak/quak/issues/197,
https://git.eeqj.de/sneak/quak/issues/198, https://git.eeqj.de/sneak/quak/issues/198,
https://git.eeqj.de/sneak/quak/issues/199, https://git.eeqj.de/sneak/quak/issues/199 and
https://git.eeqj.de/sneak/quak/issues/205 and https://git.eeqj.de/sneak/quak/issues/205. The check of quak against a live Ente
https://git.eeqj.de/sneak/quak/issues/206. The check of quak against a live Ente
account, https://git.eeqj.de/sneak/quak/issues/187, needs an account only sneak account, https://git.eeqj.de/sneak/quak/issues/187, needs an account only sneak
has, so it waits on him. has, so it waits on him.
@@ -30,6 +29,14 @@ declares one.
# Completed Steps # Completed Steps
- 2026-10-16: The `~/bin/quak` that `make install` builds retries a refused
connection and a name that does not resolve, and the backup records either as
transient (issue 206). Bun's `fetch` reports them with codes of its own,
`ConnectionRefused` and `FailedToOpenSocket`, which `src/retry.ts` now counts
as transport failures that are safe to replay. Bun already gave `ECONNRESET`
for a reset and a `TimeoutError` for a deadline. The shapes were found by
running the binary against local listeners.
- 2026-10-15: A network failure is printed and recorded with its reason, not as - 2026-10-15: A network failure is printed and recorded with its reason, not as
just `fetch failed` (issue 196). `describeError` in `src/errors.ts` gives an just `fetch failed` (issue 196). `describeError` in `src/errors.ts` gives an
error's message followed by the message of each error on its `cause` chain, error's message followed by the message of each error on its `cause` chain,
+36 -15
View File
@@ -58,6 +58,13 @@ export const resolveRetryOptions = (
}); });
// Transport failures: the request did not complete, for reasons below HTTP. // Transport failures: the request did not complete, for reasons below HTTP.
//
// The errnos are Node's. The binary `make install` builds runs under bun,
// whose `fetch` rejects with a plain `Error` that has no `cause` and a `code`
// of bun's own; `CONNECT_CODES` says what bun's two codes mean. For a
// connection reset or closed before the response ended, bun gives Node's
// `ECONNRESET`. These were found by running that binary, built with bun
// 1.3.13, against listeners on 127.0.0.1.
const TRANSPORT_CODES = new Set([ const TRANSPORT_CODES = new Set([
"ECONNRESET", "ECONNRESET",
"ECONNABORTED", "ECONNABORTED",
@@ -70,12 +77,19 @@ const TRANSPORT_CODES = new Set([
"ENETUNREACH", "ENETUNREACH",
"ENETRESET", "ENETRESET",
"ENETDOWN", "ENETDOWN",
// Bun's.
"ConnectionRefused",
"FailedToOpenSocket",
]); ]);
// The subset of the above that can only be reported before a TCP connection // The subset of the above that can only be reported before any request byte
// exists, and therefore before any request byte could have been written: name // could have been written. Node reports these before a TCP connection exists:
// resolution produced no address (`ENOTFOUND`, `EAI_AGAIN`) or the peer // name resolution produced no address (`ENOTFOUND`, `EAI_AGAIN`) or the peer
// refused the connection with an RST to the SYN (`ECONNREFUSED`). // refused the connection with an RST to the SYN (`ECONNREFUSED`). Bun gives
// `ConnectionRefused` for a refused connection, and `ConnectionRefused` or
// `FailedToOpenSocket` for a name that does not resolve. It also gives
// `ConnectionRefused` for a TLS handshake that failed, after the TCP
// connection exists, but the request is only sent once the handshake is done.
// //
// The routing errnos — `EHOSTUNREACH`, `ENETUNREACH`, `ENETDOWN` — are // The routing errnos — `EHOSTUNREACH`, `ENETUNREACH`, `ENETDOWN` — are
// deliberately absent even though they look like connect-time failures. On // deliberately absent even though they look like connect-time failures. On
@@ -87,7 +101,13 @@ const TRANSPORT_CODES = new Set([
// ambiguity this set exists to exclude. They stay in `TRANSPORT_CODES`, so // ambiguity this set exists to exclude. They stay in `TRANSPORT_CODES`, so
// they remain retryable for idempotent calls; only replay eligibility is // they remain retryable for idempotent calls; only replay eligibility is
// narrowed. See `isSafeToReplay`. // narrowed. See `isSafeToReplay`.
const CONNECT_CODES = new Set(["ENOTFOUND", "EAI_AGAIN", "ECONNREFUSED"]); const CONNECT_CODES = new Set([
"ENOTFOUND",
"EAI_AGAIN",
"ECONNREFUSED",
"ConnectionRefused",
"FailedToOpenSocket",
]);
// Collect every `code` in an error's cause chain. undici does not put the // Collect every `code` in an error's cause chain. undici does not put the
// errno on the error it throws — it hangs the underlying socket error off // errno on the error it throws — it hangs the underlying socket error off
@@ -105,7 +125,7 @@ const causeCodes = (err: unknown): { codes: string[]; complete: boolean } => {
return { codes, complete }; return { codes, complete };
}; };
// Does the error's cause chain hold the errno of a transport failure, one of // Does the error's cause chain hold the code of a transport failure, one of
// `TRANSPORT_CODES`? `isRetryable` retries such an error, and `classify` in // `TRANSPORT_CODES`? `isRetryable` retries such an error, and `classify` in
// `src/backup.ts` calls it transient. // `src/backup.ts` calls it transient.
export const hasTransportCode = (err: unknown): boolean => export const hasTransportCode = (err: unknown): boolean =>
@@ -152,7 +172,8 @@ export const isRetryable = (err: unknown): boolean => {
// deliberately literal. Demanding a recognised `cause` instead would // deliberately literal. Demanding a recognised `cause` instead would
// classify real network failures as permanent and fail backups that should // classify real network failures as permanent and fail backups that should
// have succeeded; the cost of the imprecision is bounded by the attempt // have succeeded; the cost of the imprecision is bounded by the attempt
// count. // count. Bun's fetch rejects with a plain `Error` instead, which is
// recognised by its code (see `TRANSPORT_CODES`).
if (err instanceof TypeError) return true; if (err instanceof TypeError) return true;
return hasTransportCode(err); return hasTransportCode(err);
}; };
@@ -163,18 +184,18 @@ export const isRetryable = (err: unknown): boolean => {
// `postJSON` and `putJSON` use this for every `POST` and `PUT` listed in the // `postJSON` and `putJSON` use this for every `POST` and `PUT` listed in the
// README under "Endpoints used"; verifying a second factor, for one, consumes // README under "Endpoints used"; verifying a second factor, for one, consumes
// one of a small number of attempts. They are replayed only on the failures in // one of a small number of attempts. They are replayed only on the failures in
// `CONNECT_CODES`, which establish that no TCP connection to the server ever // `CONNECT_CODES`, which establish that no request byte was ever transmitted:
// existed: there was no address to connect to, or the peer refused the // there was no address to connect to, the peer refused the connection
// connection outright. A request byte cannot have been transmitted, so the // outright, or, under bun, the TLS handshake failed. The server cannot have
// server cannot have acted. // acted.
// //
// Everything else is ambiguous. A 5xx proves the server did process the // Everything else is ambiguous. A 5xx proves the server did process the
// request. A reset or a broken pipe can arrive after it was fully sent and // request. A reset or a broken pipe can arrive after it was fully sent and
// acted on. A routing errno can be delivered on an established socket. A // acted on. A routing errno can be delivered on an established socket. A
// deadline says nothing at all about the server's state. So every errno in the // deadline says nothing at all about the server's state. So every code in the
// cause chain must be a connect errno: one other errno anywhere in the chain // cause chain must be one of `CONNECT_CODES`: one other code anywhere in the
// is doubt, and doubt is not replayed. A chain longer than the walk is doubt // chain is doubt, and doubt is not replayed. A chain longer than the walk is
// too: the links below the limit were never read. // doubt too: the links below the limit were never read.
export const isSafeToReplay = (err: unknown): boolean => { export const isSafeToReplay = (err: unknown): boolean => {
const { codes, complete } = causeCodes(err); const { codes, complete } = causeCodes(err);
return ( return (
+44
View File
@@ -724,6 +724,50 @@ describe("quak backup", () => {
}, },
); );
// The download of toast.mp4 fails as bun's `fetch` reports a failure, in
// the binary `make install` builds: a plain `Error` with bun's own code
// and message, and no cause. These are what bun gave for a refused
// connection, a name that did not resolve and a reset connection. The
// file is tried as often as under Node, and goes into `failures.json`
// with that message, as a transient failure.
it.each([
[
"ConnectionRefused",
"Unable to connect. Is the computer able to access the url?",
],
["FailedToOpenSocket", "Was there a typo in the url or port?"],
[
"ECONNRESET",
"The socket connection was closed unexpectedly. For more information, pass `verbose: true` in the second argument to fetch()",
],
])(
"records a download that failed with bun's %s in failures.json, as transient",
async (bunCode, message) => {
let attempts = 0;
const fetch: typeof globalThis.fetch = async (input, init) => {
if (downloadOf(input) === 200) {
attempts++;
throw Object.assign(new Error(message), {
code: bunCode,
errno: 0,
});
}
return server(input, init);
};
const dir = mkdtempSync(join(root, "backup-"));
const { code } = await backup(dir, {}, fetch);
expect(code).toBe(1);
expect(attempts).toBe(UNATTENDED_RETRY_OPTIONS.attempts);
const failures = readJSON(join(dir, "failures.json"));
expect(failures.files["200"]).toMatchObject({
classification: "transient",
error: message,
});
},
);
// The name of Ente's API server does not resolve when quak fetches the ML // The name of Ente's API server does not resolve when quak fetches the ML
// data, after every original has been backed up. No file has ML data // data, after every original has been backed up. No file has ML data
// cached yet, so each goes into `failures.json` with the reason, as a // cached yet, so each goes into `failures.json` with the reason, as a
+82
View File
@@ -77,6 +77,14 @@ const recordingSleep = (): {
const errnoError = (code: string, message = code): Error => const errnoError = (code: string, message = code): Error =>
Object.assign(new Error(message), { code }); Object.assign(new Error(message), { code });
/**
* An error shaped like what bun's `fetch` rejects with for a failure below
* HTTP: a plain `Error` with bun's own message, a `code`, an `errno` of 0, and
* no `cause`.
*/
const bunError = (code: string, message: string): Error =>
Object.assign(new Error(message), { code, errno: 0 });
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Classification // Classification
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -244,6 +252,80 @@ describe("isRetryable: transport failures", () => {
}); });
}); });
describe("isRetryable and isSafeToReplay: bun's fetch errors", () => {
// The binary `make install` builds runs quak under bun, whose `fetch`
// does not reject as Node's does. Each shape here was found by running
// that binary, built with bun 1.3.13, against listeners on 127.0.0.1 and
// a name under `.invalid`.
it("retries and replays a refused connection", () => {
// Bun gives the same error when the TLS handshake fails. The TCP
// connection existed then, but the request is only sent once the
// handshake is done, so replaying it is as safe.
const refused = bunError(
"ConnectionRefused",
"Unable to connect. Is the computer able to access the url?",
);
expect(isRetryable(refused)).toBe(true);
expect(isSafeToReplay(refused)).toBe(true);
});
it("retries and replays a name that does not resolve, with either code bun gives", () => {
// For the same name bun gave `ConnectionRefused` on some requests and
// `FailedToOpenSocket` on others.
for (const unresolved of [
bunError(
"ConnectionRefused",
"Unable to connect. Is the computer able to access the url?",
),
bunError(
"FailedToOpenSocket",
"Was there a typo in the url or port?",
),
]) {
expect(isRetryable(unresolved)).toBe(true);
expect(isSafeToReplay(unresolved)).toBe(true);
}
});
it("retries a connection closed before the response ended, and does not replay it", () => {
// Bun gives Node's errno, `ECONNRESET`, for a reset and for a close,
// before the response or part-way through the body. The request may
// already have reached the server.
const reset = bunError(
"ECONNRESET",
"The socket connection was closed unexpectedly. For more information, pass `verbose: true` in the second argument to fetch()",
);
expect(isRetryable(reset)).toBe(true);
expect(isSafeToReplay(reset)).toBe(false);
});
it("retries a request that ran out of time, and does not replay it", () => {
// What bun's fetch rejects with when `AbortSignal.timeout()`, which
// bounds each JSON request, fires.
const timedOut = new DOMException(
"The operation timed out.",
"TimeoutError",
);
expect(isRetryable(timedOut)).toBe(true);
expect(isSafeToReplay(timedOut)).toBe(false);
});
it("does not retry a certificate bun rejects", () => {
// A certificate that fails verification fails the same way however
// often it is asked. Under Node it is retried only because every
// `TypeError` is.
expect(
isRetryable(
bunError(
"DEPTH_ZERO_SELF_SIGNED_CERT",
"self signed certificate",
),
),
).toBe(false);
});
});
describe("isRetryable: stream truncation versus corruption", () => { describe("isRetryable: stream truncation versus corruption", () => {
it("retries a truncated stream", () => { it("retries a truncated stream", () => {
// Truncation is a transfer that stopped early. The bytes that did // Truncation is a transfer that stopped early. The bytes that did