Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4c9a42db49 | ||
|
|
d016dbfaa7 | ||
|
|
5a940d4e4e |
@@ -19,6 +19,10 @@ encrypting and uploading them back to the server.
|
|||||||
|
|
||||||
## Getting Started
|
## Getting Started
|
||||||
|
|
||||||
|
The commands below need git, node 22 or later, and yarn. `make install`, which
|
||||||
|
builds the single binary that "Running the backup from cron" uses, also needs
|
||||||
|
[nix](https://nixos.org/download).
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
git clone https://git.eeqj.de/sneak/quak.git
|
git clone https://git.eeqj.de/sneak/quak.git
|
||||||
cd quak
|
cd quak
|
||||||
@@ -86,9 +90,12 @@ enumeration/download calls) is exported too and documented under Design below.
|
|||||||
## Running the backup from cron
|
## Running the backup from cron
|
||||||
|
|
||||||
`quak backup` never prompts, so cron can run it. `make install` builds quak as a
|
`quak backup` never prompts, so cron can run it. `make install` builds quak as a
|
||||||
single binary and copies it to `~/bin/quak`. Log in once with it, as the user
|
single binary and copies it to `~/bin/quak`. Besides what Getting Started needs,
|
||||||
the cron job will run as; the session is saved in that user's data directory
|
it needs [nix](https://nixos.org/download), even where bun is already installed:
|
||||||
(see "Session handling"):
|
`make install` runs `make build-bin`, which bundles the built `dist/` with bun
|
||||||
|
run through `nix-shell -p bun`. Log in once with `~/bin/quak`, as the user the
|
||||||
|
cron job will run as; the session is saved in that user's data directory (see
|
||||||
|
"Session handling"):
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
~/bin/quak login
|
~/bin/quak login
|
||||||
@@ -134,6 +141,20 @@ matters.
|
|||||||
| `2` | Another backup of the same directory is running. This run sent no request and changed nothing. |
|
| `2` | Another backup of the same directory is running. This run sent no request and changed nothing. |
|
||||||
| `3` | There is no usable session: none is saved, the saved one is corrupt, or the server no longer accepts it. Run `quak login` as the user the cron job runs as. |
|
| `3` | There is no usable session: none is saved, the saved one is corrupt, or the server no longer accepts it. Run `quak login` as the user the cron job runs as. |
|
||||||
|
|
||||||
|
`<message>` is the error's message, followed by the message of each error that
|
||||||
|
caused it. Under Node, as `yarn quak` runs quak, `fetch` reports every failure
|
||||||
|
below HTTP as just `fetch failed` and gives the reason as its cause, so a run
|
||||||
|
that could not reach the server prints, for example,
|
||||||
|
`quak: fetch failed: connect ECONNREFUSED 1.2.3.4:443`. When the server has
|
||||||
|
several addresses and none of them could be reached, the line gives the reason
|
||||||
|
for each, separated by `, `. A file whose download failed that way is in
|
||||||
|
`failures.json` with that error, as a transient failure. So is each file with no
|
||||||
|
ML data cached when the ML data fetch failed that way, its error starting
|
||||||
|
`ML data: `. The `~/bin/quak` that `make install` builds runs under bun, and
|
||||||
|
prints bun's own wording for such a failure, with no reason after it, such as
|
||||||
|
`quak: Unable to connect. Is the computer able to access the url?` for a refused
|
||||||
|
connection. It records such a failure in `failures.json` as transient too.
|
||||||
|
|
||||||
After a `1`, the next run fetches each missing original again, fetches the ML
|
After a `1`, the next run fetches each missing original again, fetches the ML
|
||||||
data that is not cached, and rebuilds the album links. An album quak could not
|
data that is not cached, and rebuilds the album links. An album quak could not
|
||||||
decrypt, or a file in `failures.json` because quak could not decrypt it, keeps
|
decrypt, or a file in `failures.json` because quak could not decrypt it, keeps
|
||||||
@@ -347,7 +368,8 @@ quak/
|
|||||||
exif.ts EXIF read from an image's bytes with exifreader
|
exif.ts EXIF read from an image's bytes with exifreader
|
||||||
mldata-fetch.ts fetch + decrypt per-file ML data
|
mldata-fetch.ts fetch + decrypt per-file ML data
|
||||||
filename.ts safe file names from server metadata
|
filename.ts safe file names from server metadata
|
||||||
errors.ts error types shared across layers
|
errors.ts error types shared across layers, and how an error
|
||||||
|
is described
|
||||||
retry.ts retry classifier + exponential backoff with jitter
|
retry.ts retry classifier + exponential backoff with jitter
|
||||||
thumbnails.ts detect + regenerate missing thumbnails
|
thumbnails.ts detect + regenerate missing thumbnails
|
||||||
client.ts high-level Client class assembled from the above
|
client.ts high-level Client class assembled from the above
|
||||||
@@ -474,6 +496,17 @@ request is repeated only when repeating it could produce a different answer:
|
|||||||
declining to retry a transient one costs a single file that the next run picks
|
declining to retry a transient one costs a single file that the next run picks
|
||||||
up.
|
up.
|
||||||
|
|
||||||
|
The `~/bin/quak` that `make install` builds runs under bun, whose `fetch`
|
||||||
|
rejects with a plain `Error` that has no `cause` and carries a code of bun's
|
||||||
|
own. quak retries a refused connection, a name that does not resolve, a reset
|
||||||
|
connection and a deadline under bun as it does under Node. Bun reports a refused
|
||||||
|
connection, and a TLS handshake that failed, as `ConnectionRefused`. For a name
|
||||||
|
that does not resolve it gives `ConnectionRefused` on some requests and
|
||||||
|
`FailedToOpenSocket` on others. A connection reset or closed before the response
|
||||||
|
ended is `ECONNRESET`, as under Node, and a deadline is the same `TimeoutError`.
|
||||||
|
A certificate bun rejects, reported with a code such as
|
||||||
|
`DEPTH_ZERO_SELF_SIGNED_CERT`, is not retried.
|
||||||
|
|
||||||
Backoff is exponential with full jitter: the delay before retry _n_ is
|
Backoff is exponential with full jitter: the delay before retry _n_ is
|
||||||
`random() * min(maxDelayMs, baseDelayMs * 2 ** (n - 1))`. The exponential term
|
`random() * min(maxDelayMs, baseDelayMs * 2 ** (n - 1))`. The exponential term
|
||||||
is the ceiling and the wait is drawn below it, so a client that lost many
|
is the ceiling and the wait is drawn below it, so a client that lost many
|
||||||
@@ -521,17 +554,20 @@ second-factor attempts. They are retried only when every errno in the error's
|
|||||||
`cause` chain is one of the three that establish no TCP connection to the server
|
`cause` chain is one of the three that establish no TCP connection to the server
|
||||||
ever existed, so no request byte can have been transmitted: `ENOTFOUND` and
|
ever existed, so no request byte can have been transmitted: `ENOTFOUND` and
|
||||||
`EAI_AGAIN` (name resolution produced no address) and `ECONNREFUSED` (the peer
|
`EAI_AGAIN` (name resolution produced no address) and `ECONNREFUSED` (the peer
|
||||||
refused the connection). A 5xx, a mid-flight reset and a deadline are all left
|
refused the connection). Under bun these failures are `ConnectionRefused` and
|
||||||
to the caller, because each of them can happen after the server has already
|
`FailedToOpenSocket`. Bun also gives `ConnectionRefused` for a TLS handshake
|
||||||
acted. These two do not follow redirects either: a redirect means the server
|
that failed; the TCP connection existed then, but the request is only sent once
|
||||||
already received the request, so it is reported as an error and not retried. The
|
the handshake is done. A 5xx, a mid-flight reset and a deadline are all left to
|
||||||
routing errnos `EHOSTUNREACH`, `ENETUNREACH` and `ENETDOWN` are excluded for the
|
the caller, because each of them can happen after the server has already acted.
|
||||||
same reason, despite looking like connect-time failures: on Linux an ICMP
|
These two do not follow redirects either: a redirect means the server already
|
||||||
unreachable arriving mid-flight, or a local interface going down after the
|
received the request, so it is reported as an error and not retried. The routing
|
||||||
request was written, delivers them on an already-established socket. They stay
|
errnos `EHOSTUNREACH`, `ENETUNREACH` and `ENETDOWN` are excluded for the same
|
||||||
retryable for the idempotent calls. `putFile` is exempt: a presigned PUT stores
|
reason, despite looking like connect-time failures: on Linux an ICMP unreachable
|
||||||
one whole object at one key in one request, so replaying it has no partial state
|
arriving mid-flight, or a local interface going down after the request was
|
||||||
to damage.
|
written, delivers them on an already-established socket. They stay retryable for
|
||||||
|
the idempotent calls. `putFile` is exempt: a presigned PUT stores one whole
|
||||||
|
object at one key in one request, so replaying it has no partial state to
|
||||||
|
damage.
|
||||||
|
|
||||||
A download is retried as a whole — request, stream consumption, and decryption —
|
A download is retried as a whole — request, stream consumption, and decryption —
|
||||||
because a socket reset after the response headers have arrived surfaces in the
|
because a socket reset after the response headers have arrived surfaces in the
|
||||||
|
|||||||
@@ -17,7 +17,10 @@ pre-1.0
|
|||||||
|
|
||||||
# Next Step
|
# Next Step
|
||||||
|
|
||||||
None: no implementation work is open. The check of quak against a live Ente
|
Four bugs are open: https://git.eeqj.de/sneak/quak/issues/197,
|
||||||
|
https://git.eeqj.de/sneak/quak/issues/198,
|
||||||
|
https://git.eeqj.de/sneak/quak/issues/199 and
|
||||||
|
https://git.eeqj.de/sneak/quak/issues/205. The check of quak against a live Ente
|
||||||
account, https://git.eeqj.de/sneak/quak/issues/187, needs an account only sneak
|
account, https://git.eeqj.de/sneak/quak/issues/187, needs an account only sneak
|
||||||
has, so it waits on him.
|
has, so it waits on him.
|
||||||
|
|
||||||
@@ -26,6 +29,30 @@ declares one.
|
|||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
|
- 2026-10-16: The `~/bin/quak` that `make install` builds retries a refused
|
||||||
|
connection and a name that does not resolve, and the backup records either as
|
||||||
|
transient (issue 206). Bun's `fetch` reports them with codes of its own,
|
||||||
|
`ConnectionRefused` and `FailedToOpenSocket`, which `src/retry.ts` now counts
|
||||||
|
as transport failures that are safe to replay. Bun already gave `ECONNRESET`
|
||||||
|
for a reset and a `TimeoutError` for a deadline. The shapes were found by
|
||||||
|
running the binary against local listeners.
|
||||||
|
|
||||||
|
- 2026-10-15: A network failure is printed and recorded with its reason, not as
|
||||||
|
just `fetch failed` (issue 196). `describeError` in `src/errors.ts` gives an
|
||||||
|
error's message followed by the message of each error on its `cause` chain,
|
||||||
|
such as `fetch failed: connect ECONNREFUSED 1.2.3.4:443`, and each address's
|
||||||
|
reason when a host has several and none could be reached. `run` prints that,
|
||||||
|
and the backup logs it and records it in `failures.json`, where `classify` now
|
||||||
|
calls the failure transient when the chain holds an errno that
|
||||||
|
`hasTransportCode` in `src/retry.ts` recognises, for a failed ML data fetch
|
||||||
|
too. The walk along the chain moved from `src/retry.ts` to `src/errors.ts`,
|
||||||
|
which both use. Tests refuse every connection of a backup, then only one
|
||||||
|
file's download, and fail the ML data fetch.
|
||||||
|
|
||||||
|
- 2026-10-15: The README says that `make install` needs nix (issue 200), in
|
||||||
|
Getting Started and in "Running the backup from cron": `make build-bin` runs
|
||||||
|
bun through `nix-shell -p bun`. The `Makefile` is unchanged.
|
||||||
|
|
||||||
- 2026-10-14: `lib.backup()`, and so `quak backup`, works on up to the content
|
- 2026-10-14: `lib.backup()`, and so `quak backup`, works on up to the content
|
||||||
pool's limit of originals at once, 5 by default, instead of one at a time
|
pool's limit of originals at once, 5 by default, instead of one at a time
|
||||||
(issue 182). That many workers take files from one shared list, each doing all
|
(issue 182). That many workers take files from one shared list, each doing all
|
||||||
|
|||||||
+22
-15
@@ -82,6 +82,7 @@ import {
|
|||||||
init,
|
init,
|
||||||
} from "./crypto/index.js";
|
} from "./crypto/index.js";
|
||||||
import { removeLeftoverTempFiles } from "./download/index.js";
|
import { removeLeftoverTempFiles } from "./download/index.js";
|
||||||
|
import { describeError } from "./errors.js";
|
||||||
import { sanitizeFileName, withExtension } from "./filename.js";
|
import { sanitizeFileName, withExtension } from "./filename.js";
|
||||||
import {
|
import {
|
||||||
copyAtomic,
|
copyAtomic,
|
||||||
@@ -94,6 +95,7 @@ import type { Undecryptable } from "./library/store.js";
|
|||||||
import { extractImageMetadata } from "./metadata-backup.js";
|
import { extractImageMetadata } from "./metadata-backup.js";
|
||||||
import type { MLData } from "./mldata-fetch.js";
|
import type { MLData } from "./mldata-fetch.js";
|
||||||
import type { Collection, EnteFile, FileMetadata } from "./model/types.js";
|
import type { Collection, EnteFile, FileMetadata } from "./model/types.js";
|
||||||
|
import { hasTransportCode } from "./retry.js";
|
||||||
|
|
||||||
export type ProgressCallback = (message: string) => void;
|
export type ProgressCallback = (message: string) => void;
|
||||||
|
|
||||||
@@ -219,11 +221,13 @@ const isPresent = (path: string): boolean => {
|
|||||||
|
|
||||||
// Best-effort classification for the ledger. Retryable server/network problems
|
// Best-effort classification for the ledger. Retryable server/network problems
|
||||||
// are transient; refusals and local filesystem/decrypt errors are permanent;
|
// are transient; refusals and local filesystem/decrypt errors are permanent;
|
||||||
// anything else is unknown. Both the error code and message are inspected.
|
// anything else is unknown. The error code and the error as `describeError`
|
||||||
|
// gives it are inspected, and so is each errno on the error's cause chain,
|
||||||
|
// where Node's `fetch` puts it.
|
||||||
const classify = (err: unknown): FailureClass => {
|
const classify = (err: unknown): FailureClass => {
|
||||||
|
if (hasTransportCode(err)) return "transient";
|
||||||
const e = err as NodeJS.ErrnoException;
|
const e = err as NodeJS.ErrnoException;
|
||||||
const text =
|
const text = `${e?.code ?? ""} ${describeError(err)}`.toLowerCase();
|
||||||
`${e?.code ?? ""} ${err instanceof Error ? err.message : String(err)}`.toLowerCase();
|
|
||||||
if (
|
if (
|
||||||
/timeout|timed out|econnreset|econnrefused|econnaborted|network|socket|eai_again|throttl|temporarily|429|500|502|503|504/.test(
|
/timeout|timed out|econnreset|econnrefused|econnaborted|network|socket|eai_again|throttl|temporarily|429|500|502|503|504/.test(
|
||||||
text,
|
text,
|
||||||
@@ -241,9 +245,6 @@ const classify = (err: unknown): FailureClass => {
|
|||||||
return "unknown";
|
return "unknown";
|
||||||
};
|
};
|
||||||
|
|
||||||
const errorMessage = (err: unknown): string =>
|
|
||||||
err instanceof Error ? err.message : String(err);
|
|
||||||
|
|
||||||
// Ensure `linkPath` is a symlink to `target`, rebuilding a missing, wrong, or
|
// Ensure `linkPath` is a symlink to `target`, rebuilding a missing, wrong, or
|
||||||
// non-symlink entry. Throws on failure (a directory in the way, no permission)
|
// non-symlink entry. Throws on failure (a directory in the way, no permission)
|
||||||
// so the caller records it and moves on rather than aborting the run.
|
// so the caller records it and moves on rather than aborting the run.
|
||||||
@@ -470,7 +471,7 @@ const imageMetadataFor = async (
|
|||||||
const bytes = await readFile(originalPath);
|
const bytes = await readFile(originalPath);
|
||||||
return { imageMetadata: extractImageMetadata(bytes) ?? {} };
|
return { imageMetadata: extractImageMetadata(bytes) ?? {} };
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
return { imageMetadataError: errorMessage(err) };
|
return { imageMetadataError: describeError(err) };
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -610,7 +611,7 @@ const runLockedBackup = async (
|
|||||||
// and thumbnail both fail this run must not double its attempt count
|
// and thumbnail both fail this run must not double its attempt count
|
||||||
// or appear twice in errors.
|
// or appear twice in errors.
|
||||||
if (failedThisRun.has(fileID)) return;
|
if (failedThisRun.has(fileID)) return;
|
||||||
const error = errorMessage(err);
|
const error = describeError(err);
|
||||||
errors.push({ fileID, title, collection, error });
|
errors.push({ fileID, title, collection, error });
|
||||||
const prior = ledger.get(fileID);
|
const prior = ledger.get(fileID);
|
||||||
ledger.set(fileID, {
|
ledger.set(fileID, {
|
||||||
@@ -681,7 +682,7 @@ const runLockedBackup = async (
|
|||||||
}
|
}
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
log(
|
log(
|
||||||
`FAILED verifying original ${file.metadata.title}: ${errorMessage(err)}`,
|
`FAILED verifying original ${file.metadata.title}: ${describeError(err)}`,
|
||||||
);
|
);
|
||||||
recordFailure(
|
recordFailure(
|
||||||
file.id,
|
file.id,
|
||||||
@@ -717,7 +718,9 @@ const runLockedBackup = async (
|
|||||||
}
|
}
|
||||||
downloaded++;
|
downloaded++;
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
log(`FAILED original ${file.metadata.title}: ${errorMessage(err)}`);
|
log(
|
||||||
|
`FAILED original ${file.metadata.title}: ${describeError(err)}`,
|
||||||
|
);
|
||||||
recordFailure(
|
recordFailure(
|
||||||
file.id,
|
file.id,
|
||||||
file.metadata.title,
|
file.metadata.title,
|
||||||
@@ -785,12 +788,14 @@ const runLockedBackup = async (
|
|||||||
// instead and is recorded as failed. The next run fetches its ML data
|
// instead and is recorded as failed. The next run fetches its ML data
|
||||||
// again because none is cached.
|
// again because none is cached.
|
||||||
if (includeOriginals) {
|
if (includeOriginals) {
|
||||||
|
let mlDataFailure: unknown;
|
||||||
let mlDataError: string | undefined;
|
let mlDataError: string | undefined;
|
||||||
try {
|
try {
|
||||||
log("Fetching ML data...");
|
log("Fetching ML data...");
|
||||||
await lib.fetchMLData();
|
await lib.fetchMLData();
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
mlDataError = errorMessage(err);
|
mlDataFailure = err;
|
||||||
|
mlDataError = describeError(err);
|
||||||
log(`FAILED ML data: ${mlDataError}`);
|
log(`FAILED ML data: ${mlDataError}`);
|
||||||
}
|
}
|
||||||
for (const file of distinct.values()) {
|
for (const file of distinct.values()) {
|
||||||
@@ -814,7 +819,7 @@ const runLockedBackup = async (
|
|||||||
file.id,
|
file.id,
|
||||||
file.metadata.title,
|
file.metadata.title,
|
||||||
collectionName.get(file.collectionID) ?? "",
|
collectionName.get(file.collectionID) ?? "",
|
||||||
new Error(`ML data: ${mlDataError}`),
|
new Error("ML data", { cause: mlDataFailure }),
|
||||||
);
|
);
|
||||||
} else {
|
} else {
|
||||||
writeSidecar(path, file, { mlData }, image);
|
writeSidecar(path, file, { mlData }, image);
|
||||||
@@ -846,7 +851,7 @@ const runLockedBackup = async (
|
|||||||
downloadDirectory,
|
downloadDirectory,
|
||||||
);
|
);
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
log(`FAILED removing old album directories: ${errorMessage(err)}`);
|
log(`FAILED removing old album directories: ${describeError(err)}`);
|
||||||
}
|
}
|
||||||
|
|
||||||
for (const c of collections) {
|
for (const c of collections) {
|
||||||
@@ -868,7 +873,9 @@ const runLockedBackup = async (
|
|||||||
try {
|
try {
|
||||||
removeStaleLinks(colDir, new Set(linkNames), downloadDirectory);
|
removeStaleLinks(colDir, new Set(linkNames), downloadDirectory);
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
log(`FAILED removing old links in ${c.name}: ${errorMessage(err)}`);
|
log(
|
||||||
|
`FAILED removing old links in ${c.name}: ${describeError(err)}`,
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
const metaFiles = files.map((f) => ({
|
const metaFiles = files.map((f) => ({
|
||||||
@@ -885,7 +892,7 @@ const runLockedBackup = async (
|
|||||||
);
|
);
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
log(
|
log(
|
||||||
`FAILED symlink ${c.name}/${linkName}: ${errorMessage(err)}`,
|
`FAILED symlink ${c.name}/${linkName}: ${describeError(err)}`,
|
||||||
);
|
);
|
||||||
recordFailure(
|
recordFailure(
|
||||||
link.file.id,
|
link.file.id,
|
||||||
|
|||||||
+7
-6
@@ -2,14 +2,17 @@
|
|||||||
|
|
||||||
import type { Writable } from "node:stream";
|
import type { Writable } from "node:stream";
|
||||||
import { ApiError } from "./api/client.js";
|
import { ApiError } from "./api/client.js";
|
||||||
|
import { describeError } from "./errors.js";
|
||||||
|
|
||||||
// Run a command and exit with its code once stdout/stderr have drained.
|
// Run a command and exit with its code once stdout/stderr have drained.
|
||||||
// Exiting before the drain can truncate piped output, and the library can keep
|
// Exiting before the drain can truncate piped output, and the library can keep
|
||||||
// the event loop alive after a command returns, so a plain return could hang.
|
// the event loop alive after a command returns, so a plain return could hang.
|
||||||
// An error the command throws is printed as one `quak: MESSAGE` line, without
|
// An error the command throws is printed as one `quak: MESSAGE` line, without
|
||||||
// the stack trace, and exits 1. A 401 from the server means it no longer
|
// the stack trace, and exits 1. MESSAGE is the error as `describeError` gives
|
||||||
// accepts the saved session: that prints one line saying to log in again and
|
// it, so it carries the reason Node's `fetch` puts on `cause`. A 401 from the
|
||||||
// exits 3, as a missing or corrupt session file does.
|
// server means it no longer accepts the saved session: that prints one line
|
||||||
|
// saying to log in again and exits 3, as a missing or corrupt session file
|
||||||
|
// does.
|
||||||
export const run = async (
|
export const run = async (
|
||||||
command: Promise<number>,
|
command: Promise<number>,
|
||||||
stdout: Writable,
|
stdout: Writable,
|
||||||
@@ -26,9 +29,7 @@ export const run = async (
|
|||||||
);
|
);
|
||||||
code = 3;
|
code = 3;
|
||||||
} else {
|
} else {
|
||||||
stderr.write(
|
stderr.write(`quak: ${describeError(err)}\n`);
|
||||||
`quak: ${err instanceof Error ? err.message : String(err)}\n`,
|
|
||||||
);
|
|
||||||
code = 1;
|
code = 1;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -135,15 +135,17 @@ const streamDecrypt = async (
|
|||||||
// cannot tell a partial chunk from a corrupt one, so this
|
// cannot tell a partial chunk from a corrupt one, so this
|
||||||
// is reported as the truncation it almost always is, with
|
// is reported as the truncation it almost always is, with
|
||||||
// the authentication failure kept as the error's cause.
|
// the authentication failure kept as the error's cause.
|
||||||
// Only the pull is guarded: a sink failure on a chunk
|
// The message leaves that failure to the cause, which
|
||||||
// that did authenticate is a disk error, not a
|
// `describeError` appends to what `quak backup` prints and
|
||||||
|
// records. Only the pull is guarded: a sink failure on a
|
||||||
|
// chunk that did authenticate is a disk error, not a
|
||||||
// truncation.
|
// truncation.
|
||||||
let pulled;
|
let pulled;
|
||||||
try {
|
try {
|
||||||
pulled = pullStreamChunk(state, buffer);
|
pulled = pullStreamChunk(state, buffer);
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
throw new TruncatedStreamError(
|
throw new TruncatedStreamError(
|
||||||
`download: stream truncated: response body ended with ${buffer.length} trailing bytes that did not authenticate as a final chunk (transfer stopped mid-chunk, or the data is corrupt)`,
|
`download: stream truncated: response body ended with ${buffer.length} trailing bytes (transfer stopped mid-chunk, or the data is corrupt)`,
|
||||||
{ cause: err },
|
{ cause: err },
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|||||||
+71
-3
@@ -1,10 +1,13 @@
|
|||||||
// Error types that more than one layer of quak needs to recognise.
|
// Error types that more than one layer of quak needs to recognise, and how an
|
||||||
|
// error is described in what quak prints and in `failures.json`.
|
||||||
//
|
//
|
||||||
// They live here rather than beside the code that throws them so that the
|
// The types live here rather than beside the code that throws them so that the
|
||||||
// retry classifier can identify them without importing the HTTP client or the
|
// retry classifier can identify them without importing the HTTP client or the
|
||||||
// download layer — both of which import the classifier. `ApiError` is
|
// download layer — both of which import the classifier. `ApiError` is
|
||||||
// re-exported from `src/api/client.ts`, which is where callers have always
|
// re-exported from `src/api/client.ts`, which is where callers have always
|
||||||
// imported it from and where it still belongs conceptually.
|
// imported it from and where it still belongs conceptually. The walk along an
|
||||||
|
// error's `cause` chain is here too, because the retry classifier reads the
|
||||||
|
// chain for errnos and `describeError` for messages.
|
||||||
|
|
||||||
export class ApiError extends Error {
|
export class ApiError extends Error {
|
||||||
readonly status: number;
|
readonly status: number;
|
||||||
@@ -43,3 +46,68 @@ export class TruncatedStreamError extends Error {
|
|||||||
this.name = "TruncatedStreamError";
|
this.name = "TruncatedStreamError";
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// `cause` is an arbitrary user-settable property and nothing prevents it from
|
||||||
|
// forming a cycle, so the walk is bounded. Hanging the process would be worse
|
||||||
|
// than misclassifying an error or describing only part of it.
|
||||||
|
const MAX_CAUSE_DEPTH = 8;
|
||||||
|
|
||||||
|
// The error and each error on its `cause` chain, top first, at most
|
||||||
|
// `MAX_CAUSE_DEPTH` of them. `complete` is false when the walk stopped at the
|
||||||
|
// depth limit with more of the chain still below it.
|
||||||
|
export const causeChain = (
|
||||||
|
err: unknown,
|
||||||
|
): { chain: object[]; complete: boolean } => {
|
||||||
|
const chain: object[] = [];
|
||||||
|
let current: unknown = err;
|
||||||
|
for (let depth = 0; depth < MAX_CAUSE_DEPTH; depth++) {
|
||||||
|
if (current === null || typeof current !== "object") {
|
||||||
|
return { chain, complete: true };
|
||||||
|
}
|
||||||
|
chain.push(current);
|
||||||
|
const { cause } = current as { cause?: unknown };
|
||||||
|
if (cause === current) return { chain, complete: true };
|
||||||
|
current = cause;
|
||||||
|
}
|
||||||
|
return { chain, complete: current === null || typeof current !== "object" };
|
||||||
|
};
|
||||||
|
|
||||||
|
// An error as quak prints it and records it in `failures.json`: its message,
|
||||||
|
// then the message of each error on its `cause` chain, joined with ": ". Node's
|
||||||
|
// `fetch` rejects with just "fetch failed" and puts the reason on `cause`, so
|
||||||
|
// this gives "fetch failed: connect ECONNREFUSED 1.2.3.4:443". When the host
|
||||||
|
// has several addresses and every one fails, the cause is an `AggregateError`
|
||||||
|
// with an empty message and one error per address in `errors`, so an
|
||||||
|
// `AggregateError` is given as the messages of its errors, joined with ", ".
|
||||||
|
// A cause that is not an object, such as a thrown string, is given as
|
||||||
|
// `String(cause)`. Each message is trimmed of the white space around it, as
|
||||||
|
// OpenSSL ends its own with a newline. A message the text already holds is
|
||||||
|
// left out.
|
||||||
|
export const describeError = (err: unknown): string => {
|
||||||
|
if (!(err instanceof Error)) return String(err);
|
||||||
|
const { chain } = causeChain(err);
|
||||||
|
const messages = chain
|
||||||
|
.slice(1)
|
||||||
|
.map((cause) =>
|
||||||
|
cause instanceof AggregateError
|
||||||
|
? cause.errors
|
||||||
|
.map((e: unknown) =>
|
||||||
|
(e instanceof Error ? e.message : String(e)).trim(),
|
||||||
|
)
|
||||||
|
.join(", ")
|
||||||
|
: (cause as { message?: unknown }).message,
|
||||||
|
);
|
||||||
|
// `causeChain` stops at a cause that is not an object, so that one is read
|
||||||
|
// from the last error on the chain.
|
||||||
|
const last = chain[chain.length - 1] as { cause?: unknown };
|
||||||
|
if (last.cause !== undefined && typeof last.cause !== "object") {
|
||||||
|
messages.push(String(last.cause));
|
||||||
|
}
|
||||||
|
let text = err.message.trim();
|
||||||
|
for (const message of messages) {
|
||||||
|
if (typeof message !== "string") continue;
|
||||||
|
const trimmed = message.trim();
|
||||||
|
if (!text.includes(trimmed)) text += `: ${trimmed}`;
|
||||||
|
}
|
||||||
|
return text;
|
||||||
|
};
|
||||||
|
|||||||
+47
-30
@@ -10,7 +10,7 @@
|
|||||||
// declining to retry a transient one costs a single file that the next run
|
// declining to retry a transient one costs a single file that the next run
|
||||||
// picks up anyway. When the evidence is ambiguous, fail fast.
|
// picks up anyway. When the evidence is ambiguous, fail fast.
|
||||||
|
|
||||||
import { ApiError, TruncatedStreamError } from "./errors.js";
|
import { ApiError, causeChain, TruncatedStreamError } from "./errors.js";
|
||||||
|
|
||||||
export interface RetryOptions {
|
export interface RetryOptions {
|
||||||
// Total calls, not retries: `attempts: 1` disables retrying.
|
// Total calls, not retries: `attempts: 1` disables retrying.
|
||||||
@@ -58,6 +58,13 @@ export const resolveRetryOptions = (
|
|||||||
});
|
});
|
||||||
|
|
||||||
// Transport failures: the request did not complete, for reasons below HTTP.
|
// Transport failures: the request did not complete, for reasons below HTTP.
|
||||||
|
//
|
||||||
|
// The errnos are Node's. The binary `make install` builds runs under bun,
|
||||||
|
// whose `fetch` rejects with a plain `Error` that has no `cause` and a `code`
|
||||||
|
// of bun's own; `CONNECT_CODES` says what bun's two codes mean. For a
|
||||||
|
// connection reset or closed before the response ended, bun gives Node's
|
||||||
|
// `ECONNRESET`. These were found by running that binary, built with bun
|
||||||
|
// 1.3.13, against listeners on 127.0.0.1.
|
||||||
const TRANSPORT_CODES = new Set([
|
const TRANSPORT_CODES = new Set([
|
||||||
"ECONNRESET",
|
"ECONNRESET",
|
||||||
"ECONNABORTED",
|
"ECONNABORTED",
|
||||||
@@ -70,12 +77,19 @@ const TRANSPORT_CODES = new Set([
|
|||||||
"ENETUNREACH",
|
"ENETUNREACH",
|
||||||
"ENETRESET",
|
"ENETRESET",
|
||||||
"ENETDOWN",
|
"ENETDOWN",
|
||||||
|
// Bun's.
|
||||||
|
"ConnectionRefused",
|
||||||
|
"FailedToOpenSocket",
|
||||||
]);
|
]);
|
||||||
|
|
||||||
// The subset of the above that can only be reported before a TCP connection
|
// The subset of the above that can only be reported before any request byte
|
||||||
// exists, and therefore before any request byte could have been written: name
|
// could have been written. Node reports these before a TCP connection exists:
|
||||||
// resolution produced no address (`ENOTFOUND`, `EAI_AGAIN`) or the peer
|
// name resolution produced no address (`ENOTFOUND`, `EAI_AGAIN`) or the peer
|
||||||
// refused the connection with an RST to the SYN (`ECONNREFUSED`).
|
// refused the connection with an RST to the SYN (`ECONNREFUSED`). Bun gives
|
||||||
|
// `ConnectionRefused` for a refused connection, and `ConnectionRefused` or
|
||||||
|
// `FailedToOpenSocket` for a name that does not resolve. It also gives
|
||||||
|
// `ConnectionRefused` for a TLS handshake that failed, after the TCP
|
||||||
|
// connection exists, but the request is only sent once the handshake is done.
|
||||||
//
|
//
|
||||||
// The routing errnos — `EHOSTUNREACH`, `ENETUNREACH`, `ENETDOWN` — are
|
// The routing errnos — `EHOSTUNREACH`, `ENETUNREACH`, `ENETDOWN` — are
|
||||||
// deliberately absent even though they look like connect-time failures. On
|
// deliberately absent even though they look like connect-time failures. On
|
||||||
@@ -87,12 +101,13 @@ const TRANSPORT_CODES = new Set([
|
|||||||
// ambiguity this set exists to exclude. They stay in `TRANSPORT_CODES`, so
|
// ambiguity this set exists to exclude. They stay in `TRANSPORT_CODES`, so
|
||||||
// they remain retryable for idempotent calls; only replay eligibility is
|
// they remain retryable for idempotent calls; only replay eligibility is
|
||||||
// narrowed. See `isSafeToReplay`.
|
// narrowed. See `isSafeToReplay`.
|
||||||
const CONNECT_CODES = new Set(["ENOTFOUND", "EAI_AGAIN", "ECONNREFUSED"]);
|
const CONNECT_CODES = new Set([
|
||||||
|
"ENOTFOUND",
|
||||||
// `cause` is an arbitrary user-settable property and nothing prevents it from
|
"EAI_AGAIN",
|
||||||
// forming a cycle, so the walk is bounded. Hanging the process would be a
|
"ECONNREFUSED",
|
||||||
// worse outcome than any misclassification.
|
"ConnectionRefused",
|
||||||
const MAX_CAUSE_DEPTH = 8;
|
"FailedToOpenSocket",
|
||||||
|
]);
|
||||||
|
|
||||||
// Collect every `code` in an error's cause chain. undici does not put the
|
// Collect every `code` in an error's cause chain. undici does not put the
|
||||||
// errno on the error it throws — it hangs the underlying socket error off
|
// errno on the error it throws — it hangs the underlying socket error off
|
||||||
@@ -101,20 +116,21 @@ const MAX_CAUSE_DEPTH = 8;
|
|||||||
// connection permanent. `complete` is false when the walk stopped at the
|
// connection permanent. `complete` is false when the walk stopped at the
|
||||||
// depth limit with more of the chain still below it.
|
// depth limit with more of the chain still below it.
|
||||||
const causeCodes = (err: unknown): { codes: string[]; complete: boolean } => {
|
const causeCodes = (err: unknown): { codes: string[]; complete: boolean } => {
|
||||||
|
const { chain, complete } = causeChain(err);
|
||||||
const codes: string[] = [];
|
const codes: string[] = [];
|
||||||
let current: unknown = err;
|
for (const link of chain) {
|
||||||
for (let depth = 0; depth < MAX_CAUSE_DEPTH; depth++) {
|
const { code } = link as { code?: unknown };
|
||||||
if (current === null || typeof current !== "object") {
|
|
||||||
return { codes, complete: true };
|
|
||||||
}
|
|
||||||
const { code, cause } = current as { code?: unknown; cause?: unknown };
|
|
||||||
if (typeof code === "string") codes.push(code);
|
if (typeof code === "string") codes.push(code);
|
||||||
if (cause === current) return { codes, complete: true };
|
|
||||||
current = cause;
|
|
||||||
}
|
}
|
||||||
return { codes, complete: current === null || typeof current !== "object" };
|
return { codes, complete };
|
||||||
};
|
};
|
||||||
|
|
||||||
|
// Does the error's cause chain hold the code of a transport failure, one of
|
||||||
|
// `TRANSPORT_CODES`? `isRetryable` retries such an error, and `classify` in
|
||||||
|
// `src/backup.ts` calls it transient.
|
||||||
|
export const hasTransportCode = (err: unknown): boolean =>
|
||||||
|
causeCodes(err).codes.some((code) => TRANSPORT_CODES.has(code));
|
||||||
|
|
||||||
const isAbort = (err: unknown): boolean => {
|
const isAbort = (err: unknown): boolean => {
|
||||||
if (err === null || typeof err !== "object") return false;
|
if (err === null || typeof err !== "object") return false;
|
||||||
const { name } = err as { name?: unknown };
|
const { name } = err as { name?: unknown };
|
||||||
@@ -156,9 +172,10 @@ export const isRetryable = (err: unknown): boolean => {
|
|||||||
// deliberately literal. Demanding a recognised `cause` instead would
|
// deliberately literal. Demanding a recognised `cause` instead would
|
||||||
// classify real network failures as permanent and fail backups that should
|
// classify real network failures as permanent and fail backups that should
|
||||||
// have succeeded; the cost of the imprecision is bounded by the attempt
|
// have succeeded; the cost of the imprecision is bounded by the attempt
|
||||||
// count.
|
// count. Bun's fetch rejects with a plain `Error` instead, which is
|
||||||
|
// recognised by its code (see `TRANSPORT_CODES`).
|
||||||
if (err instanceof TypeError) return true;
|
if (err instanceof TypeError) return true;
|
||||||
return causeCodes(err).codes.some((code) => TRANSPORT_CODES.has(code));
|
return hasTransportCode(err);
|
||||||
};
|
};
|
||||||
|
|
||||||
// Could the first attempt already have taken effect on the server?
|
// Could the first attempt already have taken effect on the server?
|
||||||
@@ -167,18 +184,18 @@ export const isRetryable = (err: unknown): boolean => {
|
|||||||
// `postJSON` and `putJSON` use this for every `POST` and `PUT` listed in the
|
// `postJSON` and `putJSON` use this for every `POST` and `PUT` listed in the
|
||||||
// README under "Endpoints used"; verifying a second factor, for one, consumes
|
// README under "Endpoints used"; verifying a second factor, for one, consumes
|
||||||
// one of a small number of attempts. They are replayed only on the failures in
|
// one of a small number of attempts. They are replayed only on the failures in
|
||||||
// `CONNECT_CODES`, which establish that no TCP connection to the server ever
|
// `CONNECT_CODES`, which establish that no request byte was ever transmitted:
|
||||||
// existed: there was no address to connect to, or the peer refused the
|
// there was no address to connect to, the peer refused the connection
|
||||||
// connection outright. A request byte cannot have been transmitted, so the
|
// outright, or, under bun, the TLS handshake failed. The server cannot have
|
||||||
// server cannot have acted.
|
// acted.
|
||||||
//
|
//
|
||||||
// Everything else is ambiguous. A 5xx proves the server did process the
|
// Everything else is ambiguous. A 5xx proves the server did process the
|
||||||
// request. A reset or a broken pipe can arrive after it was fully sent and
|
// request. A reset or a broken pipe can arrive after it was fully sent and
|
||||||
// acted on. A routing errno can be delivered on an established socket. A
|
// acted on. A routing errno can be delivered on an established socket. A
|
||||||
// deadline says nothing at all about the server's state. So every errno in the
|
// deadline says nothing at all about the server's state. So every code in the
|
||||||
// cause chain must be a connect errno: one other errno anywhere in the chain
|
// cause chain must be one of `CONNECT_CODES`: one other code anywhere in the
|
||||||
// is doubt, and doubt is not replayed. A chain longer than the walk is doubt
|
// chain is doubt, and doubt is not replayed. A chain longer than the walk is
|
||||||
// too: the links below the limit were never read.
|
// doubt too: the links below the limit were never read.
|
||||||
export const isSafeToReplay = (err: unknown): boolean => {
|
export const isSafeToReplay = (err: unknown): boolean => {
|
||||||
const { codes, complete } = causeCodes(err);
|
const { codes, complete } = causeCodes(err);
|
||||||
return (
|
return (
|
||||||
|
|||||||
@@ -329,6 +329,41 @@ const downloadOf = (input: RequestInfo | URL): number | undefined => {
|
|||||||
: undefined;
|
: undefined;
|
||||||
};
|
};
|
||||||
|
|
||||||
|
// The error Node gives for a connection to `address` that was refused.
|
||||||
|
const refusedAt = (address: string): Error =>
|
||||||
|
Object.assign(new Error(`connect ECONNREFUSED ${address}`), {
|
||||||
|
code: "ECONNREFUSED",
|
||||||
|
});
|
||||||
|
|
||||||
|
// What Node's `fetch` rejects with when the server refuses the connection, as
|
||||||
|
// for every failure below HTTP: a TypeError that says only "fetch failed",
|
||||||
|
// with the reason, and its errno, as its cause. When the server has several
|
||||||
|
// addresses and each one refuses, the cause is an AggregateError with an empty
|
||||||
|
// message, one error per address and the errno of the first. `reason` is the
|
||||||
|
// error as quak prints it and records it in `failures.json`.
|
||||||
|
const REFUSED = [
|
||||||
|
{
|
||||||
|
addresses: "its one address",
|
||||||
|
error: (): TypeError =>
|
||||||
|
new TypeError("fetch failed", { cause: refusedAt("1.2.3.4:443") }),
|
||||||
|
reason: "fetch failed: connect ECONNREFUSED 1.2.3.4:443",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
addresses: "each of its two addresses",
|
||||||
|
error: (): TypeError =>
|
||||||
|
new TypeError("fetch failed", {
|
||||||
|
cause: Object.assign(
|
||||||
|
new AggregateError([
|
||||||
|
refusedAt("2001:db8::1:443"),
|
||||||
|
refusedAt("1.2.3.4:443"),
|
||||||
|
]),
|
||||||
|
{ code: "ECONNREFUSED" },
|
||||||
|
),
|
||||||
|
}),
|
||||||
|
reason: "fetch failed: connect ECONNREFUSED 2001:db8::1:443, connect ECONNREFUSED 1.2.3.4:443",
|
||||||
|
},
|
||||||
|
];
|
||||||
|
|
||||||
// The fake server, pushing onto `downloads` the ID of each file whose original
|
// The fake server, pushing onto `downloads` the ID of each file whose original
|
||||||
// is downloaded.
|
// is downloaded.
|
||||||
const recordingDownloads =
|
const recordingDownloads =
|
||||||
@@ -619,6 +654,159 @@ describe("quak backup", () => {
|
|||||||
expect(readdirSync(dir)).toEqual([]);
|
expect(readdirSync(dir)).toEqual([]);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// The server refuses every connection, at its one address or at each of
|
||||||
|
// several. quak tries each request again, here without waiting, and then
|
||||||
|
// the run stops at the refresh it starts with. The line it prints gives
|
||||||
|
// the reason Node's `fetch` put on the error's cause, so a cron log tells
|
||||||
|
// a refused connection from a name that did not resolve, and the exit
|
||||||
|
// code is 1.
|
||||||
|
it.each(REFUSED)(
|
||||||
|
"prints why the refresh could not reach the server at $addresses, and exits 1",
|
||||||
|
async ({ error, reason }) => {
|
||||||
|
const dir = mkdtempSync(join(root, "backup-"));
|
||||||
|
const fetch: typeof globalThis.fetch = async () => {
|
||||||
|
throw error();
|
||||||
|
};
|
||||||
|
|
||||||
|
const { code, stderr } = await backup(dir, {}, fetch);
|
||||||
|
|
||||||
|
expect(code).toBe(1);
|
||||||
|
expect(stderr).toBe(
|
||||||
|
"Starting backup...\n" +
|
||||||
|
"Refreshing library...\n" +
|
||||||
|
`quak: ${reason}\n`,
|
||||||
|
);
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
// The files CDN refuses every connection for toast.mp4, at its one
|
||||||
|
// address or at each of several. Once its retries are spent, the file
|
||||||
|
// goes into `failures.json` with the reason, as a transient failure,
|
||||||
|
// since a later run may well reach the CDN, and the lines the backup
|
||||||
|
// prints for it give the same reason. The rest of the account is backed
|
||||||
|
// up, and the exit code is 1.
|
||||||
|
it.each(REFUSED)(
|
||||||
|
"records why a download could not reach the server at $addresses in failures.json, as transient, and exits 1",
|
||||||
|
async ({ error, reason }) => {
|
||||||
|
let attempts = 0;
|
||||||
|
const fetch: typeof globalThis.fetch = async (input, init) => {
|
||||||
|
if (downloadOf(input) === 200) {
|
||||||
|
attempts++;
|
||||||
|
throw error();
|
||||||
|
}
|
||||||
|
return server(input, init);
|
||||||
|
};
|
||||||
|
const dir = mkdtempSync(join(root, "backup-"));
|
||||||
|
|
||||||
|
const { code, stderr } = await backup(dir, {}, fetch);
|
||||||
|
|
||||||
|
expect(code).toBe(1);
|
||||||
|
expect(attempts).toBe(UNATTENDED_RETRY_OPTIONS.attempts);
|
||||||
|
const failures = readJSON(join(dir, "failures.json"));
|
||||||
|
expect(Object.keys(failures.files)).toEqual(["200"]);
|
||||||
|
expect(failures.files["200"]).toMatchObject({
|
||||||
|
fileID: 200,
|
||||||
|
title: "toast.mp4",
|
||||||
|
classification: "transient",
|
||||||
|
attempts: 1,
|
||||||
|
error: reason,
|
||||||
|
});
|
||||||
|
expect(stderr).toContain(`FAILED original toast.mp4: ${reason}\n`);
|
||||||
|
expect(stderr).toContain(
|
||||||
|
`[Friend's Wedding] toast.mp4 (id 200): ${reason}\n`,
|
||||||
|
);
|
||||||
|
expect(readFileSync(join(dir, DAY, "2024-03-01.100.jpg"))).toEqual(
|
||||||
|
Buffer.from(JPEG_WITH_EXIF),
|
||||||
|
);
|
||||||
|
expect(existsSync(join(dir, DAY, "2024-03-01.200.mp4"))).toBe(
|
||||||
|
false,
|
||||||
|
);
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
// The download of toast.mp4 fails as bun's `fetch` reports a failure, in
|
||||||
|
// the binary `make install` builds: a plain `Error` with bun's own code
|
||||||
|
// and message, and no cause. These are what bun gave for a refused
|
||||||
|
// connection, a name that did not resolve and a reset connection. The
|
||||||
|
// file is tried as often as under Node, and goes into `failures.json`
|
||||||
|
// with that message, as a transient failure.
|
||||||
|
it.each([
|
||||||
|
[
|
||||||
|
"ConnectionRefused",
|
||||||
|
"Unable to connect. Is the computer able to access the url?",
|
||||||
|
],
|
||||||
|
["FailedToOpenSocket", "Was there a typo in the url or port?"],
|
||||||
|
[
|
||||||
|
"ECONNRESET",
|
||||||
|
"The socket connection was closed unexpectedly. For more information, pass `verbose: true` in the second argument to fetch()",
|
||||||
|
],
|
||||||
|
])(
|
||||||
|
"records a download that failed with bun's %s in failures.json, as transient",
|
||||||
|
async (bunCode, message) => {
|
||||||
|
let attempts = 0;
|
||||||
|
const fetch: typeof globalThis.fetch = async (input, init) => {
|
||||||
|
if (downloadOf(input) === 200) {
|
||||||
|
attempts++;
|
||||||
|
throw Object.assign(new Error(message), {
|
||||||
|
code: bunCode,
|
||||||
|
errno: 0,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return server(input, init);
|
||||||
|
};
|
||||||
|
const dir = mkdtempSync(join(root, "backup-"));
|
||||||
|
|
||||||
|
const { code } = await backup(dir, {}, fetch);
|
||||||
|
|
||||||
|
expect(code).toBe(1);
|
||||||
|
expect(attempts).toBe(UNATTENDED_RETRY_OPTIONS.attempts);
|
||||||
|
const failures = readJSON(join(dir, "failures.json"));
|
||||||
|
expect(failures.files["200"]).toMatchObject({
|
||||||
|
classification: "transient",
|
||||||
|
error: message,
|
||||||
|
});
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
// The name of Ente's API server does not resolve when quak fetches the ML
|
||||||
|
// data, after every original has been backed up. No file has ML data
|
||||||
|
// cached yet, so each goes into `failures.json` with the reason, as a
|
||||||
|
// transient failure, as a download that could not reach the server does,
|
||||||
|
// and the exit code is 1.
|
||||||
|
it("records why the ML data fetch could not reach the server in failures.json, as transient, and exits 1", async () => {
|
||||||
|
const fetch: typeof globalThis.fetch = async (input, init) => {
|
||||||
|
if (new URL(String(input)).pathname === "/files/data/fetch") {
|
||||||
|
throw new TypeError("fetch failed", {
|
||||||
|
cause: Object.assign(
|
||||||
|
new Error("getaddrinfo ENOTFOUND api.ente.io"),
|
||||||
|
{ code: "ENOTFOUND" },
|
||||||
|
),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return server(input, init);
|
||||||
|
};
|
||||||
|
const dir = mkdtempSync(join(root, "backup-"));
|
||||||
|
|
||||||
|
const { code, stderr } = await backup(dir, {}, fetch);
|
||||||
|
|
||||||
|
expect(code).toBe(1);
|
||||||
|
const error =
|
||||||
|
"ML data: fetch failed: getaddrinfo ENOTFOUND api.ente.io";
|
||||||
|
expect(stderr).toContain(`FAILED ${error}\n`);
|
||||||
|
const failures = readJSON(join(dir, "failures.json"));
|
||||||
|
expect(
|
||||||
|
Object.values(failures.files).map((f) => [
|
||||||
|
f.fileID,
|
||||||
|
f.classification,
|
||||||
|
f.error,
|
||||||
|
]),
|
||||||
|
).toEqual([
|
||||||
|
[100, "transient", error],
|
||||||
|
[200, "transient", error],
|
||||||
|
[300, "transient", error],
|
||||||
|
]);
|
||||||
|
});
|
||||||
|
|
||||||
// Something changed beach.jpg on disk after it was backed up. A run with
|
// Something changed beach.jpg on disk after it was backed up. A run with
|
||||||
// --verify hashes every stored original, finds that one no longer matches
|
// --verify hashes every stored original, finds that one no longer matches
|
||||||
// the hash Ente recorded, says so, and downloads it again; it put back the
|
// the hash Ente recorded, says so, and downloads it again; it put back the
|
||||||
|
|||||||
@@ -48,6 +48,78 @@ describe("run", () => {
|
|||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// Node's `fetch` rejects every failure below HTTP with a TypeError that
|
||||||
|
// says only "fetch failed", and hangs the reason, with its errno, off
|
||||||
|
// `cause`. The printed line carries the reason, so a cron log tells a
|
||||||
|
// refused connection from a name that did not resolve.
|
||||||
|
it("prints the reason a fetch failed, from the error's cause", async () => {
|
||||||
|
const refused = new TypeError("fetch failed", {
|
||||||
|
cause: Object.assign(
|
||||||
|
new Error("connect ECONNREFUSED 1.2.3.4:443"),
|
||||||
|
{ code: "ECONNREFUSED" },
|
||||||
|
),
|
||||||
|
});
|
||||||
|
const result = await runToExit(Promise.reject(refused));
|
||||||
|
expect(result).toEqual({
|
||||||
|
code: 1,
|
||||||
|
stdout: "",
|
||||||
|
stderr: "quak: fetch failed: connect ECONNREFUSED 1.2.3.4:443\n",
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// When the host has several addresses and every one refuses, Node's
|
||||||
|
// `fetch` gives as the cause an AggregateError with an empty message and
|
||||||
|
// one error per address. The printed line gives each address's reason.
|
||||||
|
it("prints the reason for each address a fetch could not reach", async () => {
|
||||||
|
const refusedAt = (address: string): Error =>
|
||||||
|
Object.assign(new Error(`connect ECONNREFUSED ${address}`), {
|
||||||
|
code: "ECONNREFUSED",
|
||||||
|
});
|
||||||
|
const refused = new TypeError("fetch failed", {
|
||||||
|
cause: Object.assign(
|
||||||
|
new AggregateError([
|
||||||
|
refusedAt("2001:db8::1:443"),
|
||||||
|
refusedAt("1.2.3.4:443"),
|
||||||
|
]),
|
||||||
|
{ code: "ECONNREFUSED" },
|
||||||
|
),
|
||||||
|
});
|
||||||
|
const result = await runToExit(Promise.reject(refused));
|
||||||
|
expect(result).toEqual({
|
||||||
|
code: 1,
|
||||||
|
stdout: "",
|
||||||
|
stderr: "quak: fetch failed: connect ECONNREFUSED 2001:db8::1:443, connect ECONNREFUSED 1.2.3.4:443\n",
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// An error that already quotes its cause's message is printed as it is.
|
||||||
|
it("does not print a cause's message twice", async () => {
|
||||||
|
const reset = new Error("read ECONNRESET");
|
||||||
|
const wrapped = new Error(`download: file 7: ${reset.message}`, {
|
||||||
|
cause: reset,
|
||||||
|
});
|
||||||
|
const result = await runToExit(Promise.reject(wrapped));
|
||||||
|
expect(result.stderr).toBe("quak: download: file 7: read ECONNRESET\n");
|
||||||
|
});
|
||||||
|
|
||||||
|
// OpenSSL ends its message for a failed TLS connection, such as one to a
|
||||||
|
// server that answers in plain HTTP, with a newline. The printed line is
|
||||||
|
// still one line.
|
||||||
|
it("prints a cause's message without the white space around it", async () => {
|
||||||
|
const wrongVersion = new TypeError("fetch failed", {
|
||||||
|
cause: Object.assign(
|
||||||
|
new Error(
|
||||||
|
"C0B73A55A47F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:354:\n",
|
||||||
|
),
|
||||||
|
{ code: "ERR_SSL_WRONG_VERSION_NUMBER" },
|
||||||
|
),
|
||||||
|
});
|
||||||
|
const result = await runToExit(Promise.reject(wrongVersion));
|
||||||
|
expect(result.stderr).toBe(
|
||||||
|
"quak: fetch failed: C0B73A55A47F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:354:\n",
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
it("prints a thrown value that is not an Error", async () => {
|
it("prints a thrown value that is not an Error", async () => {
|
||||||
const result = await runToExit(Promise.reject("offline"));
|
const result = await runToExit(Promise.reject("offline"));
|
||||||
expect(result).toEqual({
|
expect(result).toEqual({
|
||||||
@@ -57,6 +129,14 @@ describe("run", () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// A cause that is not an Error, such as a thrown string, is printed as a
|
||||||
|
// thrown value that is not an Error is.
|
||||||
|
it("prints a cause that is not an Error", async () => {
|
||||||
|
const wrapped = new Error("ML data", { cause: "offline" });
|
||||||
|
const result = await runToExit(Promise.reject(wrapped));
|
||||||
|
expect(result.stderr).toBe("quak: ML data: offline\n");
|
||||||
|
});
|
||||||
|
|
||||||
it("on a 401 from the server says to run quak login, on one line, and exits 3", async () => {
|
it("on a 401 from the server says to run quak login, on one line, and exits 3", async () => {
|
||||||
const result = await runToExit(
|
const result = await runToExit(
|
||||||
Promise.reject(new ApiError("unauthorized", 401)),
|
Promise.reject(new ApiError("unauthorized", 401)),
|
||||||
|
|||||||
@@ -74,7 +74,11 @@ import {
|
|||||||
} from "vitest";
|
} from "vitest";
|
||||||
import { init, toBase64, STREAM_CHUNK_SIZE } from "../../src/crypto/index.js";
|
import { init, toBase64, STREAM_CHUNK_SIZE } from "../../src/crypto/index.js";
|
||||||
import { ApiClient } from "../../src/api/client.js";
|
import { ApiClient } from "../../src/api/client.js";
|
||||||
import { ApiError, TruncatedStreamError } from "../../src/errors.js";
|
import {
|
||||||
|
ApiError,
|
||||||
|
describeError,
|
||||||
|
TruncatedStreamError,
|
||||||
|
} from "../../src/errors.js";
|
||||||
import type { RetryOptions } from "../../src/retry.js";
|
import type { RetryOptions } from "../../src/retry.js";
|
||||||
import {
|
import {
|
||||||
downloadFile,
|
downloadFile,
|
||||||
@@ -834,6 +838,11 @@ describe.each(entryPoints)(
|
|||||||
expect(((err as Error).cause as Error).message).toMatch(
|
expect(((err as Error).cause as Error).message).toMatch(
|
||||||
/authentication failed/i,
|
/authentication failed/i,
|
||||||
);
|
);
|
||||||
|
// As quak prints it and records it in `failures.json`, the error
|
||||||
|
// states the authentication failure once, from its cause.
|
||||||
|
expect(describeError(err)).toMatch(
|
||||||
|
/^download: stream truncated: response body ended with \d+ trailing bytes \(transfer stopped mid-chunk, or the data is corrupt\): secretstream chunk authentication failed$/,
|
||||||
|
);
|
||||||
|
|
||||||
// And, as with every other failure, the destination is untouched
|
// And, as with every other failure, the destination is untouched
|
||||||
// and no staged temp file survives.
|
// and no staged temp file survives.
|
||||||
|
|||||||
@@ -77,6 +77,14 @@ const recordingSleep = (): {
|
|||||||
const errnoError = (code: string, message = code): Error =>
|
const errnoError = (code: string, message = code): Error =>
|
||||||
Object.assign(new Error(message), { code });
|
Object.assign(new Error(message), { code });
|
||||||
|
|
||||||
|
/**
|
||||||
|
* An error shaped like what bun's `fetch` rejects with for a failure below
|
||||||
|
* HTTP: a plain `Error` with bun's own message, a `code`, an `errno` of 0, and
|
||||||
|
* no `cause`.
|
||||||
|
*/
|
||||||
|
const bunError = (code: string, message: string): Error =>
|
||||||
|
Object.assign(new Error(message), { code, errno: 0 });
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// Classification
|
// Classification
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
@@ -244,6 +252,80 @@ describe("isRetryable: transport failures", () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
describe("isRetryable and isSafeToReplay: bun's fetch errors", () => {
|
||||||
|
// The binary `make install` builds runs quak under bun, whose `fetch`
|
||||||
|
// does not reject as Node's does. Each shape here was found by running
|
||||||
|
// that binary, built with bun 1.3.13, against listeners on 127.0.0.1 and
|
||||||
|
// a name under `.invalid`.
|
||||||
|
|
||||||
|
it("retries and replays a refused connection", () => {
|
||||||
|
// Bun gives the same error when the TLS handshake fails. The TCP
|
||||||
|
// connection existed then, but the request is only sent once the
|
||||||
|
// handshake is done, so replaying it is as safe.
|
||||||
|
const refused = bunError(
|
||||||
|
"ConnectionRefused",
|
||||||
|
"Unable to connect. Is the computer able to access the url?",
|
||||||
|
);
|
||||||
|
expect(isRetryable(refused)).toBe(true);
|
||||||
|
expect(isSafeToReplay(refused)).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("retries and replays a name that does not resolve, with either code bun gives", () => {
|
||||||
|
// For the same name bun gave `ConnectionRefused` on some requests and
|
||||||
|
// `FailedToOpenSocket` on others.
|
||||||
|
for (const unresolved of [
|
||||||
|
bunError(
|
||||||
|
"ConnectionRefused",
|
||||||
|
"Unable to connect. Is the computer able to access the url?",
|
||||||
|
),
|
||||||
|
bunError(
|
||||||
|
"FailedToOpenSocket",
|
||||||
|
"Was there a typo in the url or port?",
|
||||||
|
),
|
||||||
|
]) {
|
||||||
|
expect(isRetryable(unresolved)).toBe(true);
|
||||||
|
expect(isSafeToReplay(unresolved)).toBe(true);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it("retries a connection closed before the response ended, and does not replay it", () => {
|
||||||
|
// Bun gives Node's errno, `ECONNRESET`, for a reset and for a close,
|
||||||
|
// before the response or part-way through the body. The request may
|
||||||
|
// already have reached the server.
|
||||||
|
const reset = bunError(
|
||||||
|
"ECONNRESET",
|
||||||
|
"The socket connection was closed unexpectedly. For more information, pass `verbose: true` in the second argument to fetch()",
|
||||||
|
);
|
||||||
|
expect(isRetryable(reset)).toBe(true);
|
||||||
|
expect(isSafeToReplay(reset)).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("retries a request that ran out of time, and does not replay it", () => {
|
||||||
|
// What bun's fetch rejects with when `AbortSignal.timeout()`, which
|
||||||
|
// bounds each JSON request, fires.
|
||||||
|
const timedOut = new DOMException(
|
||||||
|
"The operation timed out.",
|
||||||
|
"TimeoutError",
|
||||||
|
);
|
||||||
|
expect(isRetryable(timedOut)).toBe(true);
|
||||||
|
expect(isSafeToReplay(timedOut)).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not retry a certificate bun rejects", () => {
|
||||||
|
// A certificate that fails verification fails the same way however
|
||||||
|
// often it is asked. Under Node it is retried only because every
|
||||||
|
// `TypeError` is.
|
||||||
|
expect(
|
||||||
|
isRetryable(
|
||||||
|
bunError(
|
||||||
|
"DEPTH_ZERO_SELF_SIGNED_CERT",
|
||||||
|
"self signed certificate",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
).toBe(false);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
describe("isRetryable: stream truncation versus corruption", () => {
|
describe("isRetryable: stream truncation versus corruption", () => {
|
||||||
it("retries a truncated stream", () => {
|
it("retries a truncated stream", () => {
|
||||||
// Truncation is a transfer that stopped early. The bytes that did
|
// Truncation is a transfer that stopped early. The bytes that did
|
||||||
|
|||||||
Reference in New Issue
Block a user