quak backup retries failed requests for longer (closes #165)
check / check (push) Successful in 2m34s

src/retry.ts exports UNATTENDED_RETRY_OPTIONS beside the unchanged
default: 10 attempts, a 1 s base delay and a 60 s cap, so a request that
keeps failing waits at most 243 s before it gives up. bin/quak.ts loads
the backup's session with them, so its refresh, ML data and downloads
all use them; every other command keeps the default. What is retried
and the backoff formula are unchanged.

Model: opus-5-5
This commit is contained in:
2026-10-06 01:02:43 +00:00
committed by sneak
parent 14ac7d05ef
commit 22682fa7cf
8 changed files with 168 additions and 11 deletions
+15 -10
View File
@@ -420,18 +420,23 @@ Backoff is exponential with full jitter: the delay before retry _n_ is
`random() * min(maxDelayMs, baseDelayMs * 2 ** (n - 1))`. The exponential term `random() * min(maxDelayMs, baseDelayMs * 2 ** (n - 1))`. The exponential term
is the ceiling and the wait is drawn below it, so a client that lost many is the ceiling and the wait is drawn below it, so a client that lost many
parallel downloads to one CDN blip does not send them all again at the same parallel downloads to one CDN blip does not send them all again at the same
instant. Defaults, configurable through `ApiClientOptions.retry`: instant. The numbers, configurable through `ApiClientOptions.retry`:
| Option | Default | Meaning | | Option | Default | `quak backup` | Meaning |
| ------------- | ------- | ----------------------------------- | | ------------- | ------- | ------------- | ----------------------------------- |
| `attempts` | `4` | total calls, not retries | | `attempts` | `4` | `10` | total calls, not retries |
| `baseDelayMs` | `500` | ceiling for the first retry's delay | | `baseDelayMs` | `500` | `1000` | ceiling for the first retry's delay |
| `maxDelayMs` | `10000` | upper bound on that ceiling | | `maxDelayMs` | `10000` | `60000` | upper bound on that ceiling |
With those defaults a file that is going to fail gives up after at most three With the defaults a file that is going to fail gives up after at most three and
and a half seconds of waiting. `sleep` and `random` are injectable through the a half seconds of waiting. `quak backup` usually runs from cron with nobody
same option, which is how the test suite exercises the whole policy without watching, so every request it makes uses the second column instead, exported as
waiting. `UNATTENDED_RETRY_OPTIONS`: a request that keeps failing gives up after at most
243 seconds of waiting, and usually after about half that, since each wait is
drawn at random below its ceiling. Every other command uses the defaults. A
library user gets the same budget by passing `UNATTENDED_RETRY_OPTIONS` as
`ApiClientOptions.retry`. `sleep` and `random` are injectable through the same
option, which is how the test suite exercises the whole policy without waiting.
Two deadlines, renewed for each attempt: Two deadlines, renewed for each attempt:
+8
View File
@@ -25,6 +25,14 @@ declares one.
# Completed Steps # Completed Steps
- 2026-10-05: `quak backup` retries a failed request for longer than the other
commands do (issue 165). `src/retry.ts` exports `UNATTENDED_RETRY_OPTIONS`
beside the unchanged default: 10 attempts, a 1 s base delay and a 60 s cap, so
a request that keeps failing waits at most 243 s before it gives up.
`bin/quak.ts` loads the backup's session with them, so its refresh, ML data
and downloads all use them. What is retried and the backoff formula are
unchanged.
- 2026-10-03: The download-albums example test no longer fails on a disk with - 2026-10-03: The download-albums example test no longer fails on a disk with
under 50 GiB free (issue 160). It opens its libraries with under 50 GiB free (issue 160). It opens its libraries with
`freeBelowBytes: 0`, so the free space of the disk it runs on cannot shrink `freeBelowBytes: 0`, so the free space of the disk it runs on cannot shrink
+14 -1
View File
@@ -23,6 +23,7 @@ import { run as runCommand } from "../src/cli-run.js";
import { loadSession } from "../src/cli-session.js"; import { loadSession } from "../src/cli-session.js";
import { Client } from "../src/client.js"; import { Client } from "../src/client.js";
import { VERSION } from "../src/index.js"; import { VERSION } from "../src/index.js";
import { UNATTENDED_RETRY_OPTIONS } from "../src/retry.js";
const paths = envPaths("quak", { suffix: "" }); const paths = envPaths("quak", { suffix: "" });
@@ -129,8 +130,20 @@ program
) )
.argument("<dir>", "Output directory") .argument("<dir>", "Output directory")
.option("--json", "Print result as JSON instead of human-readable summary") .option("--json", "Print result as JSON instead of human-readable summary")
// A backup usually runs from cron with nobody watching, so every request
// it makes retries for longer than the other commands' requests do.
.action((dir: string, opts: { json?: boolean }) => .action((dir: string, opts: { json?: boolean }) =>
run(backupCommand(context(), dir, opts)), run(
backupCommand(
{
...context(),
loadSession: (path) =>
loadSession(path, { retry: UNATTENDED_RETRY_OPTIONS }),
},
dir,
opts,
),
),
); );
const helper = program const helper = program
+1
View File
@@ -27,6 +27,7 @@ export {
isRetryable, isRetryable,
isSafeToReplay, isSafeToReplay,
resolveRetryOptions, resolveRetryOptions,
UNATTENDED_RETRY_OPTIONS,
withRetry, withRetry,
type ResolvedRetryOptions, type ResolvedRetryOptions,
type RetryOptions, type RetryOptions,
+10
View File
@@ -37,6 +37,16 @@ export const DEFAULT_RETRY_OPTIONS: ResolvedRetryOptions = {
random: Math.random, random: Math.random,
}; };
// For a run nobody is watching, such as `quak backup` from cron. A request that
// keeps failing waits at most 243 s in all before it gives up, usually about
// half that, since each wait is drawn at random below its ceiling.
export const UNATTENDED_RETRY_OPTIONS: ResolvedRetryOptions = {
...DEFAULT_RETRY_OPTIONS,
attempts: 10,
baseDelayMs: 1_000,
maxDelayMs: 60_000,
};
export const resolveRetryOptions = ( export const resolveRetryOptions = (
opts?: RetryOptions, opts?: RetryOptions,
): ResolvedRetryOptions => ({ ): ResolvedRetryOptions => ({
+71
View File
@@ -0,0 +1,71 @@
/**
* Tests for the retry options `bin/quak.ts` loads the saved session with:
* `quak backup` gets the unattended ones, every other command the default.
*
* Each test runs `bin/quak.ts`, as the smoke test does, with its session loader
* replaced by one that records the options it is given and reports no session,
* so the command stops before it makes any request.
*/
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { afterAll, afterEach, describe, expect, it, vi } from "vitest";
import type { ApiClientOptions } from "../../src/api/client.js";
import { UNATTENDED_RETRY_OPTIONS } from "../../src/retry.js";
const loaded = vi.hoisted(() => [] as (ApiClientOptions | undefined)[]);
vi.mock("../../src/cli-session.js", () => ({
loadSession: (_path: string, apiOptions?: ApiClientOptions) => {
loaded.push(apiOptions);
return null;
},
}));
const argv = process.argv;
const dir = mkdtempSync(join(tmpdir(), "quak-bin-test-"));
// Run `quak <args>` to completion. The "Not logged in" message and the exit
// are swallowed.
const quak = async (...args: string[]): Promise<void> => {
vi.spyOn(process.stderr, "write").mockImplementation(() => true);
vi.spyOn(process, "exit").mockImplementation(() => undefined as never);
process.argv = ["node", "quak", ...args];
vi.resetModules();
await import("../../bin/quak.js");
};
afterEach(() => {
process.argv = argv;
loaded.length = 0;
vi.restoreAllMocks();
});
afterAll(() => {
rmSync(dir, { recursive: true, force: true });
});
describe("bin/quak.ts session loading", () => {
it("loads the session for backup with the unattended retry options", async () => {
await quak("backup", dir);
// `vi.resetModules()` gave `bin/quak.ts` its own copy of
// `src/retry.ts`, whose `sleep` is a different function, so the
// options are compared by their numbers.
const { attempts, baseDelayMs, maxDelayMs } = UNATTENDED_RETRY_OPTIONS;
expect(loaded).toEqual([
{
retry: expect.objectContaining({
attempts,
baseDelayMs,
maxDelayMs,
}),
},
]);
});
it("loads the session for another command with the default options", async () => {
await quak("collections");
expect(loaded).toEqual([undefined]);
});
});
+18
View File
@@ -12,6 +12,10 @@ import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { init, toBase64 } from "../../src/crypto/index.js"; import { init, toBase64 } from "../../src/crypto/index.js";
import { Client, type ClientSnapshot } from "../../src/client.js"; import { Client, type ClientSnapshot } from "../../src/client.js";
import { loadSession } from "../../src/cli-session.js"; import { loadSession } from "../../src/cli-session.js";
import {
DEFAULT_RETRY_OPTIONS,
UNATTENDED_RETRY_OPTIONS,
} from "../../src/retry.js";
const validSnapshot = (): ClientSnapshot => { const validSnapshot = (): ClientSnapshot => {
const kp = sodium.crypto_box_keypair(); const kp = sodium.crypto_box_keypair();
@@ -176,6 +180,20 @@ describe("loadSession", () => {
}); });
}); });
it("gives the restored client the retry options it is passed", () => {
const path = join(dir, "retry.json");
writeFileSync(path, JSON.stringify(validSnapshot()));
const retryOptions = (client: Client | null) =>
client!.getApiClient().getRetryOptions();
expect(
retryOptions(
loadSession(path, { retry: UNATTENDED_RETRY_OPTIONS }),
),
).toEqual(UNATTENDED_RETRY_OPTIONS);
expect(retryOptions(loadSession(path))).toEqual(DEFAULT_RETRY_OPTIONS);
});
it("says the file is corrupt when it is not JSON", () => { it("says the file is corrupt when it is not JSON", () => {
const path = join(dir, "truncated.json"); const path = join(dir, "truncated.json");
writeFileSync(path, '{"email": "user@exa'); writeFileSync(path, '{"email": "user@exa');
+31
View File
@@ -43,6 +43,7 @@ import {
isRetryable, isRetryable,
isSafeToReplay, isSafeToReplay,
resolveRetryOptions, resolveRetryOptions,
UNATTENDED_RETRY_OPTIONS,
withRetry, withRetry,
} from "../../src/retry.js"; } from "../../src/retry.js";
import { ApiError, TruncatedStreamError } from "../../src/errors.js"; import { ApiError, TruncatedStreamError } from "../../src/errors.js";
@@ -639,3 +640,33 @@ describe("retry defaults", () => {
} }
}); });
}); });
describe("unattended retry options", () => {
it("allow ten attempts, a 1 s base delay and a 60 s cap", () => {
// The numbers the README documents for `quak backup`.
expect(UNATTENDED_RETRY_OPTIONS.attempts).toBe(10);
expect(UNATTENDED_RETRY_OPTIONS.baseDelayMs).toBe(1_000);
expect(UNATTENDED_RETRY_OPTIONS.maxDelayMs).toBe(60_000);
});
it("give up on a request that keeps failing after at most 243 s of waiting", async () => {
// `random: () => 1` makes every wait its ceiling, the worst case.
const { sleep, delays } = recordingSleep();
let calls = 0;
await expect(
withRetry(
() => {
calls++;
return Promise.reject(new ApiError("HTTP 503", 503));
},
{ ...UNATTENDED_RETRY_OPTIONS, sleep, random: () => 1 },
),
).rejects.toThrow("HTTP 503");
expect(calls).toBe(10);
expect(delays).toEqual([
1_000, 2_000, 4_000, 8_000, 16_000, 32_000, 60_000, 60_000, 60_000,
]);
expect(delays.reduce((sum, ms) => sum + ms, 0)).toBe(243_000);
});
});