Retry transient network failures with exponential backoff (closes #2)
All checks were successful
check / check (push) Successful in 20s
All checks were successful
check / check (push) Successful in 20s
No retry on 4xx, backoff on 5xx and transport failures, and a deadline on every request. Before this, one transient 503 or TCP reset failed a file for good, and a CDN connection that went quiet after accepting the request blocked `quak backup` forever, because there was no timeout anywhere. src/retry.ts holds the policy: a classifier that decides whether another attempt could produce a different answer, and a loop that acts on it with exponential backoff and full jitter. Retried: 5xx, 408, 429, transport failures (the errno is read out of the cause chain, which is where Node's fetch puts it), deadline aborts, and truncated transfers. Not retried: every other 4xx, and anything unrecognised — a wrongly retried permanent failure delays every remaining file, while a wrongly abandoned transient one costs a single file the next run picks up. Attempt count, delays, sleep and jitter source are all configurable through ApiClientOptions; sleep being injectable is what lets the suite exercise the policy without waiting. Truncation needed a type before it could be classified. streamDecrypt threw plain Errors whose messages began "download: stream truncated", and classifying on message text would mean the next reword silently turned every truncated download into a permanent failure. It now throws TruncatedStreamError, which lives in src/errors.ts alongside ApiError so the classifier can recognise both without importing the modules that import it; api/client.ts re-exports ApiError, so it stays one class and every existing import path still resolves. Downloads retry the request, the stream consumption and the decryption together. Only the first of those happens inside ApiClient: a socket reset after the headers arrived throws in streamDecrypt, and retrying the request alone would never see it. The client's own retry is switched off for those two calls so the budgets do not multiply into sixteen requests per file, and the atomic write stays outside the loop so a download that took three attempts still performs one write and one rename. Non-idempotent requests are not blindly replayed. postJSON and putJSON reach create-session, two-factor/verify — which burns one of a few second-factor attempts — and files/thumbnail, so they retry only when the connection was never established and the server provably never saw the request. putFile is exempt and retries fully: a presigned PUT stores one whole object at one key, with no partial state to damage. It now throws ApiError with the status, as do the two null-body paths, which previously threw bare Errors that nothing could classify. Timeouts come from AbortSignal.timeout(), renewed per attempt: 30s for JSON and upload calls, 10 minutes for file bodies, since a value short enough to keep a hung API call from stalling a backup would cancel a legitimate multi-gigabyte download. The download deadline is enforced over the body rather than only the headers, by racing each read against the signal, so the guarantee does not depend on the fetch implementation tearing down a stream it already handed over. listMissingThumbnails now separates a genuine 404 from an exhausted retry. Its bare catch reported both as missing, which after this change would have let a few minutes of 500s talk fix-missing-thumbnails into regenerating and re-uploading thumbnails that were fine. runBackup and runMetadataBackup are untouched: the retry sits below them and their per-file resilience is unchanged.
This commit is contained in:
90
README.md
90
README.md
@@ -169,6 +169,8 @@ quak/
|
||||
model/ decrypted Collection, File, Metadata types + decrypt fns
|
||||
download/ streaming file/thumbnail download + decryption
|
||||
backup.ts resilient full-account backup with dedup
|
||||
errors.ts error types shared across layers
|
||||
retry.ts retry classifier + exponential backoff with jitter
|
||||
thumbnails.ts detect + regenerate missing thumbnails
|
||||
client.ts high-level Client class assembled from the above
|
||||
index.ts public library exports
|
||||
@@ -250,6 +252,87 @@ Endpoints used:
|
||||
- `POST /files/upload-url`: mint a presigned upload URL (for thumbnail repair).
|
||||
- `PUT /files/thumbnail`: register an uploaded thumbnail's object key.
|
||||
|
||||
### Retries and timeouts
|
||||
|
||||
Every request in the library goes through one policy, in `src/retry.ts`. A
|
||||
request is repeated only when repeating it could produce a different answer:
|
||||
|
||||
- `ApiError` with a 5xx status: retried. So are `408` and `429`, the two 4xx
|
||||
codes that are statements about timing rather than about the request.
|
||||
- Every other 4xx: not retried. A 404 in particular is an answer, and
|
||||
`listMissingThumbnails` depends on getting it promptly and once.
|
||||
- Transport failures — a `fetch` rejection, `ECONNRESET`, `ETIMEDOUT`, a DNS or
|
||||
TLS failure — and deadline aborts: retried. The errno is looked for in the
|
||||
error's `cause` chain, because that is where Node's `fetch` puts it.
|
||||
- A truncated download: retried.
|
||||
- Anything else, including a secretstream authentication failure that is not
|
||||
truncation: not retried. The default answer is no. For a backup tool, retrying
|
||||
a permanent failure spends round trips and delays every remaining file, while
|
||||
declining to retry a transient one costs a single file that the next run picks
|
||||
up.
|
||||
|
||||
Backoff is exponential with full jitter: the delay before retry _n_ is
|
||||
`random() * min(maxDelayMs, baseDelayMs * 2 ** (n - 1))`. The exponential term
|
||||
is the ceiling and the wait is drawn below it, so a client that lost many
|
||||
parallel downloads to one CDN blip does not send them all again at the same
|
||||
instant. Defaults, configurable through `ApiClientOptions.retry`:
|
||||
|
||||
| Option | Default | Meaning |
|
||||
| ------------- | ------- | ----------------------------------- |
|
||||
| `attempts` | `4` | total calls, not retries |
|
||||
| `baseDelayMs` | `500` | ceiling for the first retry's delay |
|
||||
| `maxDelayMs` | `10000` | upper bound on that ceiling |
|
||||
|
||||
With those defaults a file that is going to fail gives up after at most three
|
||||
and a half seconds of waiting. `sleep` and `random` are injectable through the
|
||||
same option, which is how the test suite exercises the whole policy without
|
||||
waiting.
|
||||
|
||||
Two deadlines, applied with `AbortSignal.timeout()` and renewed for each
|
||||
attempt:
|
||||
|
||||
| Option | Default | Applies to |
|
||||
| ------------------- | -------- | ------------------------------------------- |
|
||||
| `requestTimeoutMs` | `30000` | `getJSON`, `postJSON`, `putJSON`, `putFile` |
|
||||
| `downloadTimeoutMs` | `600000` | file and thumbnail body transfers |
|
||||
|
||||
They are separate because one number cannot serve both: a value short enough to
|
||||
keep a hung API call from stalling a backup would cancel a legitimate
|
||||
multi-gigabyte download. The download deadline covers the body, not just the
|
||||
headers — `getFileStream` returns as soon as headers arrive, so a deadline that
|
||||
only guarded the initial request would leave the same hang one layer down.
|
||||
|
||||
**Non-idempotent requests are not blindly replayed.** `postJSON` and `putJSON`
|
||||
reach `/users/srp/create-session`, `/users/two-factor/verify` — which consumes
|
||||
one of a small number of second-factor attempts — and `/files/thumbnail`. They
|
||||
are retried only on failures that prove no request byte reached the server,
|
||||
which means the connection was never established (`ECONNREFUSED`, `ENOTFOUND`,
|
||||
and the like). A 5xx, a mid-flight reset and a deadline are all left to the
|
||||
caller, because each of them can happen after the server has already acted.
|
||||
`putFile` is exempt: a presigned PUT stores one whole object at one key in one
|
||||
request, so replaying it has no partial state to damage.
|
||||
|
||||
A download is retried as a whole — request, stream consumption, and decryption —
|
||||
because a socket reset after the response headers have arrived surfaces in the
|
||||
download layer rather than in `ApiClient`, and that is the common failure for
|
||||
multi-megabyte photos over a CDN. The secretstream pull state is not resumable
|
||||
and these endpoints have no Range support, so a retry starts the file over. The
|
||||
atomic write stays outside the retry, so a download that needed three attempts
|
||||
still performs exactly one write and one rename. `runBackup` and
|
||||
`runMetadataBackup` are unchanged: the retry sits below them, and a file that
|
||||
fails after exhausting it is still logged, counted, and stepped over.
|
||||
|
||||
One imprecision is deliberate and worth knowing about. When a body ends part-way
|
||||
through a secretstream chunk, Poly1305 fails and carries no framing signal, so a
|
||||
cut connection and genuinely corrupt bytes are indistinguishable. quak reports
|
||||
that as truncation, which means it is retried. For a body of more than one chunk
|
||||
the distinction is real — a chunk that failed while the stream carried on past
|
||||
it stays an authentication failure and is not retried — but for a single-chunk
|
||||
body, which is most thumbnails and every small file, a wrong key, server-side
|
||||
corruption and a mid-chunk cutoff all present alike and all get retried. The
|
||||
cost is bounded by the attempt count, and it buys never silently keeping a
|
||||
truncated file.
|
||||
|
||||
### Session handling
|
||||
|
||||
The `Client` class holds the auth token, master key, secret key, and public key
|
||||
@@ -308,7 +391,7 @@ code is non-zero if any files failed.
|
||||
|
||||
## TODO
|
||||
|
||||
- [ ] Retry policy: no retry on 4xx, exponential backoff on 5xx and network
|
||||
- [x] Retry policy: no retry on 4xx, exponential backoff on 5xx and network
|
||||
errors
|
||||
- [ ] Update the API reference section below to match the current implementation
|
||||
- [ ] `make docker` green
|
||||
@@ -333,7 +416,10 @@ are correct.
|
||||
The key types and their actual signatures can be found in:
|
||||
|
||||
- `src/client.ts`: `Client`, `LoginOptions`, `ClientSnapshot`
|
||||
- `src/api/client.ts`: `ApiClient`, `ApiClientOptions`, `ApiError`
|
||||
- `src/api/client.ts`: `ApiClient`, `ApiClientOptions`, `ApiError`,
|
||||
`StreamOptions`
|
||||
- `src/errors.ts`: `ApiError`, `TruncatedStreamError`
|
||||
- `src/retry.ts`: `withRetry`, `isRetryable`, `isSafeToReplay`, `RetryOptions`
|
||||
- `src/auth/types.ts`: `KeyAttributes`, `SRPAttributes`,
|
||||
`AuthorizationResponse`, `LoginChallenge`
|
||||
- `src/model/types.ts`: `Collection`, `EnteFile`, `FileMetadata`, `FileBlob`,
|
||||
|
||||
Reference in New Issue
Block a user