Retry transient network failures with exponential backoff (closes #2)
All checks were successful
check / check (push) Successful in 20s
All checks were successful
check / check (push) Successful in 20s
No retry on 4xx, backoff on 5xx and transport failures, and a deadline on every request. Before this, one transient 503 or TCP reset failed a file for good, and a CDN connection that went quiet after accepting the request blocked `quak backup` forever, because there was no timeout anywhere. src/retry.ts holds the policy: a classifier that decides whether another attempt could produce a different answer, and a loop that acts on it with exponential backoff and full jitter. Retried: 5xx, 408, 429, transport failures (the errno is read out of the cause chain, which is where Node's fetch puts it), deadline aborts, and truncated transfers. Not retried: every other 4xx, and anything unrecognised — a wrongly retried permanent failure delays every remaining file, while a wrongly abandoned transient one costs a single file the next run picks up. Attempt count, delays, sleep and jitter source are all configurable through ApiClientOptions; sleep being injectable is what lets the suite exercise the policy without waiting. Truncation needed a type before it could be classified. streamDecrypt threw plain Errors whose messages began "download: stream truncated", and classifying on message text would mean the next reword silently turned every truncated download into a permanent failure. It now throws TruncatedStreamError, which lives in src/errors.ts alongside ApiError so the classifier can recognise both without importing the modules that import it; api/client.ts re-exports ApiError, so it stays one class and every existing import path still resolves. Downloads retry the request, the stream consumption and the decryption together. Only the first of those happens inside ApiClient: a socket reset after the headers arrived throws in streamDecrypt, and retrying the request alone would never see it. The client's own retry is switched off for those two calls so the budgets do not multiply into sixteen requests per file, and the atomic write stays outside the loop so a download that took three attempts still performs one write and one rename. Non-idempotent requests are not blindly replayed. postJSON and putJSON reach create-session, two-factor/verify — which burns one of a few second-factor attempts — and files/thumbnail, so they retry only when the connection was never established and the server provably never saw the request. putFile is exempt and retries fully: a presigned PUT stores one whole object at one key, with no partial state to damage. It now throws ApiError with the status, as do the two null-body paths, which previously threw bare Errors that nothing could classify. Timeouts come from AbortSignal.timeout(), renewed per attempt: 30s for JSON and upload calls, 10 minutes for file bodies, since a value short enough to keep a hung API call from stalling a backup would cancel a legitimate multi-gigabyte download. The download deadline is enforced over the body rather than only the headers, by racing each read against the signal, so the guarantee does not depend on the fetch implementation tearing down a stream it already handed over. listMissingThumbnails now separates a genuine 404 from an exhausted retry. Its bare catch reported both as missing, which after this change would have let a few minutes of 500s talk fix-missing-thumbnails into regenerating and re-uploading thumbnails that were fine. runBackup and runMetadataBackup are untouched: the retry sits below them and their per-file resilience is unchanged.
This commit is contained in:
13
TODO.md
13
TODO.md
@@ -14,12 +14,16 @@ pre-1.0
|
||||
|
||||
# Next Step
|
||||
|
||||
Implement the download retry policy from the README TODO: no retry on 4xx,
|
||||
exponential backoff on 5xx and network errors. Apply it to file and thumbnail
|
||||
downloads, cover it with mock-server tests, and update the README TODO checkbox.
|
||||
Update the README API reference section to match the current implementation.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-08-09: Retry policy: no retry on 4xx (except `408` and `429`),
|
||||
exponential backoff with full jitter on 5xx, transport failures and truncated
|
||||
transfers, under per-attempt deadlines that cover the response body as well as
|
||||
the request. Downloads retry request, stream consumption and decryption as one
|
||||
unit; `postJSON` and `putJSON` are replayed only when the connection was never
|
||||
established.
|
||||
- 2026-08-09: Downloads verify the secretstream terminated on `TAG_FINAL` and
|
||||
write output atomically: a truncated body is rejected instead of landing on
|
||||
disk as a short file, and plaintext is staged in a sibling temp file and
|
||||
@@ -45,9 +49,6 @@ downloads, cover it with mock-server tests, and update the README TODO checkbox.
|
||||
|
||||
# Future Steps
|
||||
|
||||
- Retry policy: no retry on 4xx, exponential backoff on 5xx and network errors
|
||||
(the Next Step).
|
||||
- Update the README API reference section to match the current implementation.
|
||||
- Make `make docker` green.
|
||||
- Tag v1.0.0.
|
||||
- Future desktop client, separate repo:
|
||||
|
||||
Reference in New Issue
Block a user