Compare commits

..
1 Commits
Author SHA1 Message Date
sneak 9ceb82fe72 Harden the backup tree's atomic copy (closes #22)
check / check (push) Successful in 35s
The backup copy now fsyncs its temp file before the rename and the
directory after it, using the download writer's new fsyncPath helper.
Each backup run deletes .quak-backup-*.tmp files whose process is no
longer running, leaving those of a concurrent backup alone. The rename
sites and the README backup layout state that a symlink at the
destination is replaced and the new file takes the temp file's
permissions, and the README names the temp files. Adds tests for a
missing and an unwritable destination directory for downloadFile and
downloadThumbnail.

Model: opus-5-5
2026-09-23 00:25:32 +00:00
9 changed files with 108 additions and 457 deletions
+14 -18
View File
@@ -323,7 +323,6 @@ Endpoints used:
- `GET /collections/v2/diff?collectionID=<id>&sinceTime=<usec>`: list files in a - `GET /collections/v2/diff?collectionID=<id>&sinceTime=<usec>`: list files in a
collection; paginate while `hasMore` is true. collection; paginate while `hasMore` is true.
- `GET https://files.ente.io/?fileID=<id>`: download encrypted file bytes. - `GET https://files.ente.io/?fileID=<id>`: download encrypted file bytes.
- `POST /files/data/fetch`: fetch encrypted ML data for a batch of files.
- `POST /files/upload-url`: mint a presigned upload URL (for thumbnail repair). - `POST /files/upload-url`: mint a presigned upload URL (for thumbnail repair).
- `PUT /files/thumbnail`: register an uploaded thumbnail's object key. - `PUT /files/thumbnail`: register an uploaded thumbnail's object key.
@@ -378,23 +377,20 @@ headers — `getFileStream` returns as soon as headers arrive, so a deadline tha
only guarded the initial request would leave the same hang one layer down. only guarded the initial request would leave the same hang one layer down.
**Non-idempotent requests are not blindly replayed.** `postJSON` and `putJSON` **Non-idempotent requests are not blindly replayed.** `postJSON` and `putJSON`
send every `POST` and `PUT` in the endpoint list above; some of them change reach `/users/srp/create-session`, `/users/two-factor/verify` — which consumes
server state, and `/users/two-factor/verify` consumes one of a small number of one of a small number of second-factor attempts — and `/files/thumbnail`. They
second-factor attempts. They are retried only when every errno in the error's are retried only on the three failures that establish no TCP connection to the
`cause` chain is one of the three that establish no TCP connection to the server server ever existed, so no request byte can have been transmitted: `ENOTFOUND`
ever existed, so no request byte can have been transmitted: `ENOTFOUND` and and `EAI_AGAIN` (name resolution produced no address) and `ECONNREFUSED` (the
`EAI_AGAIN` (name resolution produced no address) and `ECONNREFUSED` (the peer peer refused the connection). A 5xx, a mid-flight reset and a deadline are all
refused the connection). A 5xx, a mid-flight reset and a deadline are all left left to the caller, because each of them can happen after the server has already
to the caller, because each of them can happen after the server has already acted. The routing errnos `EHOSTUNREACH`, `ENETUNREACH` and `ENETDOWN` are
acted. These two do not follow redirects either: a redirect means the server excluded for the same reason, despite looking like connect-time failures: on
already received the request, so it is reported as an error and not retried. The Linux an ICMP unreachable arriving mid-flight, or a local interface going down
routing errnos `EHOSTUNREACH`, `ENETUNREACH` and `ENETDOWN` are excluded for the after the request was written, delivers them on an already-established socket.
same reason, despite looking like connect-time failures: on Linux an ICMP They stay retryable for the idempotent calls. `putFile` is exempt: a presigned
unreachable arriving mid-flight, or a local interface going down after the PUT stores one whole object at one key in one request, so replaying it has no
request was written, delivers them on an already-established socket. They stay partial state to damage.
retryable for the idempotent calls. `putFile` is exempt: a presigned PUT stores
one whole object at one key in one request, so replaying it has no partial state
to damage.
A download is retried as a whole — request, stream consumption, and decryption — A download is retried as a whole — request, stream consumption, and decryption —
because a socket reset after the response headers have arrived surfaces in the because a socket reset after the response headers have arrived surfaces in the
-14
View File
@@ -25,20 +25,6 @@ Tag v1.0.0.
temp files and states that the rename replaces a symlink and takes the temp temp files and states that the rename replaces a symlink and takes the temp
file's permissions. Added tests for a missing and an unwritable destination file's permissions. Added tests for a missing and an unwritable destination
directory for `downloadFile` and `downloadThumbnail`. directory for `downloadFile` and `downloadThumbnail`.
- 2026-09-23: Hardened the JPEG EXIF scan behind `backup-metadata --exif` (issue
11). Every segment length is checked against the remaining bytes and lengths
under 2 stop the scan, so a truncated or corrupt original can neither throw
nor loop. A malformed or unparseable EXIF segment is recorded as
`imageMetadata.exifError`, and a failure to read the original as
`imageMetadataError` in the per-file JSON, instead of the field being left
out.
- 2026-09-22: Hardened the retry classifier (issue 80). A `POST` or `PUT` is
replayed only when every errno in the cause chain is a connect errno, and it
no longer follows redirects. `getRetryOptions()` returns a copy. Tests pin
every errno the classifier names, the cause-chain depth limit, cycle
termination, and a fresh deadline per attempt for every retrying entry point.
The README's endpoint list is the one place that names the requests the replay
rule covers.
- 2026-09-22: Stopped `make test` collecting tests from checkouts nested under - 2026-09-22: Stopped `make test` collecting tests from checkouts nested under
`.claude/` (issue 25). vitest ignores `.gitignore` when finding tests, so a `.claude/` (issue 25). vitest ignores `.gitignore` when finding tests, so a
nested checkout ran the whole suite again; `vitest.config.ts` now adds nested checkout ran the whole suite again; `vitest.config.ts` now adds
+13 -14
View File
@@ -146,9 +146,8 @@ export class ApiClient {
// The policy this client was configured with, so that a caller wrapping a // The policy this client was configured with, so that a caller wrapping a
// whole operation in its own `withRetry` — the download layer — runs under // whole operation in its own `withRetry` — the download layer — runs under
// the same settings rather than under the library defaults. // the same settings rather than under the library defaults.
// A copy, so the caller cannot change this client's settings through it.
getRetryOptions(): ResolvedRetryOptions { getRetryOptions(): ResolvedRetryOptions {
return { ...this.retry }; return this.retry;
} }
private headers(extra?: Record<string, string>): Record<string, string> { private headers(extra?: Record<string, string>): Record<string, string> {
@@ -229,15 +228,15 @@ export class ApiClient {
async postJSON<T>(path: string, body: unknown): Promise<T> { async postJSON<T>(path: string, body: unknown): Promise<T> {
const url = `${this.apiOrigin}${path}`; const url = `${this.apiOrigin}${path}`;
// Not idempotent: a POST is replayed only when `isSafeToReplay` // Idempotency: this reaches `/users/srp/create-session`,
// says no request byte can have reached the server. The endpoints // `/users/two-factor/verify` and `/users/ott`, all of which change
// this covers are listed in the README under "Endpoints used". // server state — verifying a second factor consumes one of a small
// // number of attempts. So a POST is replayed only on a failure that
// Redirects are not followed. The origin has already received the // establishes no TCP connection to the server ever existed: DNS
// request when it answers with one, so a connection refused by the // produced no address, or the peer refused the connection. A 5xx, a
// redirect target would look replay-safe when it is not. The API has // mid-flight reset, a routing errno (which Linux also delivers on an
// no legitimate redirect, so one surfaces as an `ApiError` with its // established socket) and a timeout are all left to the caller,
// 3xx status, which is not retried. // because each of them can occur after the server has already acted.
return withRetry( return withRetry(
async () => { async () => {
const resp = await this._fetch(url, { const resp = await this._fetch(url, {
@@ -246,7 +245,6 @@ export class ApiClient {
"Content-Type": "application/json", "Content-Type": "application/json",
}), }),
body: JSON.stringify(body), body: JSON.stringify(body),
redirect: "manual",
signal: AbortSignal.timeout(this.requestTimeoutMs), signal: AbortSignal.timeout(this.requestTimeoutMs),
}); });
await this.throwIfError(resp); await this.throwIfError(resp);
@@ -305,7 +303,9 @@ export class ApiClient {
async putJSON<T>(path: string, body: unknown): Promise<T> { async putJSON<T>(path: string, body: unknown): Promise<T> {
const url = `${this.apiOrigin}${path}`; const url = `${this.apiOrigin}${path}`;
// Same replay and redirect rules as `postJSON`, for the same reasons. // Same idempotency rule as `postJSON`, for the same reason: this
// reaches `/files/thumbnail`, which registers an uploaded thumbnail
// against a file.
return withRetry( return withRetry(
async () => { async () => {
const resp = await this._fetch(url, { const resp = await this._fetch(url, {
@@ -314,7 +314,6 @@ export class ApiClient {
"Content-Type": "application/json", "Content-Type": "application/json",
}), }),
body: JSON.stringify(body), body: JSON.stringify(body),
redirect: "manual",
signal: AbortSignal.timeout(this.requestTimeoutMs), signal: AbortSignal.timeout(this.requestTimeoutMs),
}); });
await this.throwIfError(resp); await this.throwIfError(resp);
+62 -85
View File
@@ -15,35 +15,18 @@ export interface MetadataBackupOptions {
onProgress?: ProgressCallback; onProgress?: ProgressCallback;
} }
// Find the raw EXIF APP1 segment in JPEG bytes. Returns `exif` (the segment // Extract the raw EXIF APP1 segment from JPEG bytes. Returns the EXIF
// data, starting at the "Exif\0\0" header) when there is one, nothing when the // data buffer (starting after the APP1 length field, at the "Exif\0\0"
// bytes are not a JPEG or carry no EXIF, and `error` when the segment layout is // header) or undefined if no APP1 marker is found.
// malformed. Each segment length is checked against the bytes that remain and const extractExifFromJpeg = (buf: Uint8Array): Buffer | undefined => {
// each step moves forward by at least 4 bytes, so the scan ends on any input. if (buf[0] !== 0xff || buf[1] !== 0xd8) return undefined;
export const extractExifFromJpeg = (
buf: Uint8Array,
): { exif?: Buffer; error?: string } => {
if (buf[0] !== 0xff || buf[1] !== 0xd8) return {};
let offset = 2; let offset = 2;
while (offset < buf.length) { while (offset < buf.length - 1) {
if (offset + 2 > buf.length) if (buf[offset] !== 0xff) return undefined;
return { error: `truncated segment marker at byte ${offset}` };
if (buf[offset] !== 0xff)
return { error: `no segment marker at byte ${offset}` };
const marker = buf[offset + 1]!; const marker = buf[offset + 1]!;
if (marker === 0xda) return {}; // start of scan, no more markers if (marker === 0xda) break; // start of scan, no more markers
if (offset + 4 > buf.length) if (offset + 3 >= buf.length) break;
return { error: `truncated segment length at byte ${offset}` };
const len = (buf[offset + 2]! << 8) | buf[offset + 3]!; const len = (buf[offset + 2]! << 8) | buf[offset + 3]!;
// The length counts its own two bytes, so anything under 2 is invalid.
if (len < 2)
return {
error: `segment length ${len} at byte ${offset} is too small`,
};
if (offset + 2 + len > buf.length)
return {
error: `segment length ${len} at byte ${offset} runs past the end of the file`,
};
if (marker === 0xe1) { if (marker === 0xe1) {
// APP1 — check for "Exif\0\0" header // APP1 — check for "Exif\0\0" header
if ( if (
@@ -52,70 +35,65 @@ export const extractExifFromJpeg = (
buf[offset + 6] === 0x69 && buf[offset + 6] === 0x69 &&
buf[offset + 7] === 0x66 buf[offset + 7] === 0x66
) { ) {
return { return Buffer.from(
exif: Buffer.from( buf.buffer,
buf.buffer, buf.byteOffset + offset + 4,
buf.byteOffset + offset + 4, len - 2,
len - 2, );
),
};
} }
} }
offset += 2 + len; offset += 2 + len;
} }
return { error: "file ends before the image data" }; return undefined;
}; };
// Extract dimensions, EXIF and XMP from a file's bytes. When the EXIF segment const extractImageMetadata = (
// is malformed or cannot be parsed, the record carries the reason in
// `exifError`.
export const extractImageMetadata = (
fileBytes: Uint8Array, fileBytes: Uint8Array,
): Record<string, unknown> | undefined => { ): Record<string, unknown> | undefined => {
const result: Record<string, unknown> = {};
// Try to get dimensions from JPEG decode
try { try {
const decoded = jpeg.decode(fileBytes, { const result: Record<string, unknown> = {};
useTArray: true,
formatAsRGBA: false,
});
result.format = "jpeg";
result.width = decoded.width;
result.height = decoded.height;
} catch {
// Not every original is a JPEG (PNG, HEIC, video), so a failed decode
// is expected and only means no dimensions; a malformed JPEG is still
// reported below through `exifError`.
}
const { exif, error } = extractExifFromJpeg(fileBytes); // Try to get dimensions from JPEG decode
if (error) result.exifError = error;
if (exif) {
try { try {
result.exif = exifReader(exif); const decoded = jpeg.decode(fileBytes, {
} catch (err) { useTArray: true,
result.exifRaw = exif.toString("base64"); formatAsRGBA: false,
result.exifError = err instanceof Error ? err.message : String(err); });
result.format = "jpeg";
result.width = decoded.width;
result.height = decoded.height;
} catch {
// Not a JPEG or corrupt; still try EXIF extraction
} }
}
// Extract XMP (look for "http://ns.adobe.com/xap" in the bytes) const exifBuf = extractExifFromJpeg(fileBytes);
const xmpStart = Buffer.from(fileBytes).indexOf("<?xpacket begin"); if (exifBuf) {
if (xmpStart !== -1) { try {
const xmpEnd = Buffer.from(fileBytes).indexOf( result.exif = exifReader(exifBuf);
"<?xpacket end", } catch {
xmpStart, result.exifRaw = exifBuf.toString("base64");
); }
if (xmpEnd !== -1) {
const end = Buffer.from(fileBytes).indexOf("?>", xmpEnd);
result.xmp = Buffer.from(fileBytes)
.subarray(xmpStart, end !== -1 ? end + 2 : xmpEnd + 50)
.toString("utf-8");
} }
}
return Object.keys(result).length > 0 ? result : undefined; // Extract XMP (look for "http://ns.adobe.com/xap" in the bytes)
const xmpStart = Buffer.from(fileBytes).indexOf("<?xpacket begin");
if (xmpStart !== -1) {
const xmpEnd = Buffer.from(fileBytes).indexOf(
"<?xpacket end",
xmpStart,
);
if (xmpEnd !== -1) {
const end = Buffer.from(fileBytes).indexOf("?>", xmpEnd);
result.xmp = Buffer.from(fileBytes)
.subarray(xmpStart, end !== -1 ? end + 2 : xmpEnd + 50)
.toString("utf-8");
}
}
return Object.keys(result).length > 0 ? result : undefined;
} catch {
return undefined;
}
}; };
// Read a file's original bytes through the library's content cache and extract // Read a file's original bytes through the library's content cache and extract
@@ -125,9 +103,13 @@ export const extractImageMetadata = (
const extractExif = async ( const extractExif = async (
photo: Photo, photo: Photo,
): Promise<Record<string, unknown> | undefined> => { ): Promise<Record<string, unknown> | undefined> => {
const { path } = await photo.original(); try {
const fileBytes = new Uint8Array(readFileSync(path)); const { path } = await photo.original();
return extractImageMetadata(fileBytes); const fileBytes = new Uint8Array(readFileSync(path));
return extractImageMetadata(fileBytes);
} catch {
return undefined;
}
}; };
// Dump every decrypted metadata layer the account holds into a directory tree // Dump every decrypted metadata layer the account holds into a directory tree
@@ -233,13 +215,8 @@ export const runMetadataBackup = async (
if (wantExif && !writtenFileIDs.has(file.id)) { if (wantExif && !writtenFileIDs.has(file.id)) {
log(`[${file.metadata.title}] Extracting EXIF...`); log(`[${file.metadata.title}] Extracting EXIF...`);
try { const exifData = await extractExif(photo);
const exifData = await extractExif(photo); if (exifData) fileMeta.imageMetadata = exifData;
if (exifData) fileMeta.imageMetadata = exifData;
} catch (err) {
fileMeta.imageMetadataError =
err instanceof Error ? err.message : String(err);
}
} }
writtenFileIDs.add(file.id); writtenFileIDs.add(file.id);
+12 -25
View File
@@ -88,21 +88,18 @@ const MAX_CAUSE_DEPTH = 8;
// errno on the error it throws — it hangs the underlying socket error off // errno on the error it throws — it hangs the underlying socket error off
// `cause`, sometimes more than one level down — so a classifier that only read // `cause`, sometimes more than one level down — so a classifier that only read
// the top-level error would see a bare `Error` and call every dropped // the top-level error would see a bare `Error` and call every dropped
// connection permanent. `complete` is false when the walk stopped at the // connection permanent.
// depth limit with more of the chain still below it. const causeCodes = (err: unknown): string[] => {
const causeCodes = (err: unknown): { codes: string[]; complete: boolean } => {
const codes: string[] = []; const codes: string[] = [];
let current: unknown = err; let current: unknown = err;
for (let depth = 0; depth < MAX_CAUSE_DEPTH; depth++) { for (let depth = 0; depth < MAX_CAUSE_DEPTH; depth++) {
if (current === null || typeof current !== "object") { if (current === null || typeof current !== "object") break;
return { codes, complete: true };
}
const { code, cause } = current as { code?: unknown; cause?: unknown }; const { code, cause } = current as { code?: unknown; cause?: unknown };
if (typeof code === "string") codes.push(code); if (typeof code === "string") codes.push(code);
if (cause === current) return { codes, complete: true }; if (cause === current) break;
current = cause; current = cause;
} }
return { codes, complete: current === null || typeof current !== "object" }; return codes;
}; };
const isAbort = (err: unknown): boolean => { const isAbort = (err: unknown): boolean => {
@@ -148,15 +145,15 @@ export const isRetryable = (err: unknown): boolean => {
// have succeeded; the cost of the imprecision is bounded by the attempt // have succeeded; the cost of the imprecision is bounded by the attempt
// count. // count.
if (err instanceof TypeError) return true; if (err instanceof TypeError) return true;
return causeCodes(err).codes.some((code) => TRANSPORT_CODES.has(code)); return causeCodes(err).some((code) => TRANSPORT_CODES.has(code));
}; };
// Could the first attempt already have taken effect on the server? // Could the first attempt already have taken effect on the server?
// //
// `isRetryable` is the wrong question for a request that changes state. // `isRetryable` is the wrong question for a request that changes state.
// `postJSON` and `putJSON` use this for every `POST` and `PUT` listed in the // quak's non-idempotent calls are `/users/srp/create-session`,
// README under "Endpoints used"; verifying a second factor, for one, consumes // `/users/two-factor/verify` — which consumes one of a small number of 2FA
// one of a small number of attempts. They are replayed only on the failures in // attempts — and `/files/thumbnail`. They are replayed only on the failures in
// `CONNECT_CODES`, which establish that no TCP connection to the server ever // `CONNECT_CODES`, which establish that no TCP connection to the server ever
// existed: there was no address to connect to, or the peer refused the // existed: there was no address to connect to, or the peer refused the
// connection outright. A request byte cannot have been transmitted, so the // connection outright. A request byte cannot have been transmitted, so the
@@ -165,19 +162,9 @@ export const isRetryable = (err: unknown): boolean => {
// Everything else is ambiguous. A 5xx proves the server did process the // Everything else is ambiguous. A 5xx proves the server did process the
// request. A reset or a broken pipe can arrive after it was fully sent and // request. A reset or a broken pipe can arrive after it was fully sent and
// acted on. A routing errno can be delivered on an established socket. A // acted on. A routing errno can be delivered on an established socket. A
// deadline says nothing at all about the server's state. So every errno in the // deadline says nothing at all about the server's state.
// cause chain must be a connect errno: one other errno anywhere in the chain export const isSafeToReplay = (err: unknown): boolean =>
// is doubt, and doubt is not replayed. A chain longer than the walk is doubt isRetryable(err) && causeCodes(err).some((code) => CONNECT_CODES.has(code));
// too: the links below the limit were never read.
export const isSafeToReplay = (err: unknown): boolean => {
const { codes, complete } = causeCodes(err);
return (
isRetryable(err) &&
complete &&
codes.length > 0 &&
codes.every((code) => CONNECT_CODES.has(code))
);
};
export interface WithRetryOptions extends RetryOptions { export interface WithRetryOptions extends RetryOptions {
isRetryable?: (err: unknown) => boolean; isRetryable?: (err: unknown) => boolean;
+3 -91
View File
@@ -639,24 +639,6 @@ describe("ApiClient retries", () => {
expect(policy.baseDelayMs).toBe(7); expect(policy.baseDelayMs).toBe(7);
expect(policy.maxDelayMs).toBe(11); expect(policy.maxDelayMs).toBe(11);
}); });
it("does not let a caller change its settings through that policy", async () => {
const { fetch, calls } = scriptedFetch(
textResponse("boom", 500),
textResponse("boom", 500),
textResponse("boom", 500),
);
const client = new ApiClient({
fetch,
retry: { ...noWait, attempts: 2 },
});
client.getRetryOptions().attempts = 3;
expect(client.getRetryOptions().attempts).toBe(2);
await expect(client.getJSON("/x")).rejects.toBeInstanceOf(ApiError);
expect(calls).toHaveLength(2);
});
}); });
describe("ApiClient timeouts", () => { describe("ApiClient timeouts", () => {
@@ -711,51 +693,6 @@ describe("ApiClient timeouts", () => {
expect(new Set(signals).size).toBe(3); expect(new Set(signals).size).toBe(3);
}, 5000); }, 5000);
it("gives every retrying entry point a fresh deadline per attempt", async () => {
// A refused connection is retried by every entry point, the
// non-idempotent ones included. If the deadline were created once,
// outside the retry, both attempts would carry the same signal.
const entryPoints: [
string,
() => Response,
(c: ApiClient) => unknown,
][] = [
["getJSON", () => jsonResponse({}), (c) => c.getJSON("/a")],
["postJSON", () => jsonResponse({}), (c) => c.postJSON("/b", {})],
["putJSON", () => jsonResponse({}), (c) => c.putJSON("/c", {})],
[
"putFile",
() => new Response(null, { status: 200 }),
(c) => c.putFile("https://s3.example/x", new Uint8Array([1])),
],
[
"getFileStream",
() => streamResponse(new Uint8Array([1])),
(c) => c.getFileStream(1),
],
[
"getThumbnailStream",
() => streamResponse(new Uint8Array([1])),
(c) => c.getThumbnailStream(1),
],
];
for (const [name, success, call] of entryPoints) {
const { fetch, calls } = scriptedFetch(
errnoError("ECONNREFUSED", "connect ECONNREFUSED"),
success(),
);
const client = new ApiClient({ fetch, retry: noWait });
await call(client);
expect(calls, name).toHaveLength(2);
const [first, second] = calls.map((c) => c.init?.signal);
expect(first, name).toBeInstanceOf(AbortSignal);
expect(second, name).toBeInstanceOf(AbortSignal);
expect(second, name).not.toBe(first);
}
});
it("recovers when a later attempt answers in time", async () => { it("recovers when a later attempt answers in time", async () => {
const { fetch, calls } = scriptedFetch(HANG, jsonResponse({ ok: 1 })); const { fetch, calls } = scriptedFetch(HANG, jsonResponse({ ok: 1 }));
const client = new ApiClient({ const client = new ApiClient({
@@ -883,8 +820,9 @@ describe("ApiClient error typing", () => {
describe("ApiClient non-idempotent requests", () => { describe("ApiClient non-idempotent requests", () => {
/** /**
* `postJSON` and `putJSON` carry quak's requests that can change server * `postJSON` and `putJSON` carry quak's only requests that change server
* state; the README lists them under "Endpoints used". * state: `/users/srp/create-session`, `/users/two-factor/verify` — which
* consumes one of a small number of 2FA attempts — and `/files/thumbnail`.
* *
* They are retried only on a failure that establishes no TCP connection to * They are retried only on a failure that establishes no TCP connection to
* the server ever existed — DNS produced no address, or the peer refused * the server ever existed — DNS produced no address, or the peer refused
@@ -984,30 +922,4 @@ describe("ApiClient non-idempotent requests", () => {
await refusedClient.updateThumbnail(1, "key", "header"); await refusedClient.updateThumbnail(1, "key", "header");
expect(refused.calls).toHaveLength(2); expect(refused.calls).toHaveLength(2);
}); });
it("does not follow or replay a redirect on POST or PUT", async () => {
// The origin has already received a request it answers with a
// redirect, so following it would let a refused connection to the
// redirect target pass for a request that never went out.
for (const send of [
(c: ApiClient) => c.postJSON("/users/ott", {}),
(c: ApiClient) => c.putJSON("/files/thumbnail", {}),
]) {
const { fetch, calls } = scriptedFetch(
new Response(null, {
status: 307,
headers: { location: "https://elsewhere.example/" },
}),
jsonResponse({}),
);
const client = new ApiClient({ fetch, retry: noWait });
const err: unknown = await send(client).catch((e: unknown) => e);
expect(calls[0]?.init?.redirect).toBe("manual");
expect(err).toBeInstanceOf(ApiError);
expect((err as ApiError).status).toBe(307);
expect(calls).toHaveLength(1);
}
});
}); });
-13
View File
@@ -621,18 +621,5 @@ describe("quak backup-metadata", () => {
expect(fileMeta.imageMetadata.format).toBe("jpeg"); expect(fileMeta.imageMetadata.format).toBe("jpeg");
expect(fileMeta.imageMetadata.width).toBe(100); expect(fileMeta.imageMetadata.width).toBe(100);
expect(fileMeta.imageMetadata.height).toBe(80); expect(fileMeta.imageMetadata.height).toBe(80);
expect(fileMeta.imageMetadataError).toBeUndefined();
// File 200 has no original on the mock server, so extraction fails
// and the reason is recorded instead of the field being left out.
const workDir = collDirs.find((d) => d.includes("Work"))!;
const failedMeta = JSON.parse(
readFileSync(
join(outDir, "collections", workDir, "200.json"),
"utf-8",
),
);
expect(failedMeta.imageMetadata).toBeUndefined();
expect(failedMeta.imageMetadataError).toEqual(expect.any(String));
}); });
}); });
-122
View File
@@ -1,122 +0,0 @@
/**
* Tests for the JPEG EXIF scan behind `quak backup-metadata --exif`.
*
* The originals come from users' libraries, so a truncated or corrupt JPEG
* must neither hang the scan nor throw out of it, and a malformed file must be
* told apart from one that simply has no EXIF: the record carries the reason in
* `exifError`. Each input below is a short hand-built byte array.
*/
import { describe, expect, it } from "vitest";
import {
extractExifFromJpeg,
extractImageMetadata,
} from "../../src/metadata-backup.js";
const SOI = [0xff, 0xd8]; // start of image
const SOS = [0xff, 0xda, 0x00, 0x02]; // start of scan, where the scan stops
const EXIF_HEADER = [0x45, 0x78, 0x69, 0x66, 0x00, 0x00]; // "Exif\0\0"
// A big-endian TIFF block with one IFD entry: Orientation (0x0112), SHORT, 6.
const TIFF_ORIENTATION_6 = [
0x4d, 0x4d, 0x00, 0x2a, 0x00, 0x00, 0x00, 0x08, 0x00, 0x01, 0x01, 0x12,
0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x00, 0x06, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00,
];
// An APP1 segment whose length field matches its data.
const app1 = (data: number[]): number[] => {
const len = data.length + 2;
return [0xff, 0xe1, len >> 8, len & 0xff, ...data];
};
const bytes = (...parts: number[][]): Uint8Array =>
new Uint8Array(parts.flat());
describe("extractExifFromJpeg", () => {
it("returns the EXIF segment of a valid JPEG", () => {
const data = [...EXIF_HEADER, ...TIFF_ORIENTATION_6];
const scan = extractExifFromJpeg(bytes(SOI, app1(data), SOS));
expect(scan.error).toBeUndefined();
expect([...scan.exif!]).toEqual(data);
});
it("returns nothing for a file that is not a JPEG", () => {
const png = bytes([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]);
expect(extractExifFromJpeg(png)).toEqual({});
});
it("returns nothing for a JPEG without EXIF", () => {
const app0 = [0xff, 0xe0, 0x00, 0x04, 0x00, 0x00];
expect(extractExifFromJpeg(bytes(SOI, app0, SOS))).toEqual({});
});
it("reports a JPEG truncated inside a segment header", () => {
const scan = extractExifFromJpeg(bytes(SOI, [0xff, 0xe1, 0x00]));
expect(scan.exif).toBeUndefined();
expect(scan.error).toMatch(/truncated segment length/);
});
it("reports a JPEG that ends before the image data", () => {
const app0 = [0xff, 0xe0, 0x00, 0x04, 0x00, 0x00];
const scan = extractExifFromJpeg(bytes(SOI, app0));
expect(scan.error).toMatch(/ends before the image data/);
});
it("stops on a zero-length segment instead of looping", () => {
// A length of 0 would otherwise step the scan by 2 bytes at a time
// through the rest of the file, reading garbage as markers.
const zero = [0xff, 0xe0, 0x00, 0x00];
const scan = extractExifFromJpeg(
bytes(SOI, zero, zero, zero, zero, SOS),
);
expect(scan.error).toMatch(/segment length 0 at byte 2 is too small/);
});
it("stops on a segment length of 1", () => {
const scan = extractExifFromJpeg(
bytes(SOI, [0xff, 0xe0, 0x00, 0x01], SOS),
);
expect(scan.error).toMatch(/segment length 1 at byte 2 is too small/);
});
it("reports a segment length that runs past the end of the file", () => {
// APP1 claims 0x4000 bytes but only the "Exif\0\0" header follows.
const scan = extractExifFromJpeg(
bytes(SOI, [0xff, 0xe1, 0x40, 0x00], EXIF_HEADER),
);
expect(scan.exif).toBeUndefined();
expect(scan.error).toMatch(/runs past the end of the file/);
});
});
describe("extractImageMetadata", () => {
it("parses EXIF from a valid JPEG", () => {
const meta = extractImageMetadata(
bytes(SOI, app1([...EXIF_HEADER, ...TIFF_ORIENTATION_6]), SOS),
);
expect(meta?.exifError).toBeUndefined();
expect(meta?.exif).toMatchObject({ Image: { Orientation: 6 } });
});
it("returns nothing for a file that is not a JPEG", () => {
const text = new TextEncoder().encode("just some text, not an image");
expect(extractImageMetadata(text)).toBeUndefined();
});
it("records the reason when the JPEG is malformed", () => {
const meta = extractImageMetadata(
bytes(SOI, [0xff, 0xe1, 0x40, 0x00], EXIF_HEADER),
);
expect(meta?.exif).toBeUndefined();
expect(meta?.exifError).toMatch(/runs past the end of the file/);
});
it("keeps the raw bytes and the reason when EXIF cannot be parsed", () => {
const data = [...EXIF_HEADER, 0x58, 0x58];
const meta = extractImageMetadata(bytes(SOI, app1(data), SOS));
expect(meta?.exif).toBeUndefined();
expect(meta?.exifRaw).toBe(Buffer.from(data).toString("base64"));
expect(meta?.exifError).toEqual(expect.any(String));
});
});
+4 -75
View File
@@ -149,11 +149,8 @@ describe("isRetryable: transport failures", () => {
}); });
it("retries an errno carried on the error itself", () => { it("retries an errno carried on the error itself", () => {
// Every errno the classifier names, so none can be reclassified
// unnoticed.
for (const code of [ for (const code of [
"ECONNRESET", "ECONNRESET",
"ECONNABORTED",
"ETIMEDOUT", "ETIMEDOUT",
"EPIPE", "EPIPE",
"ENOTFOUND", "ENOTFOUND",
@@ -161,8 +158,6 @@ describe("isRetryable: transport failures", () => {
"ECONNREFUSED", "ECONNREFUSED",
"EHOSTUNREACH", "EHOSTUNREACH",
"ENETUNREACH", "ENETUNREACH",
"ENETRESET",
"ENETDOWN",
]) { ]) {
expect(isRetryable(errnoError(code))).toBe(true); expect(isRetryable(errnoError(code))).toBe(true);
} }
@@ -220,27 +215,6 @@ describe("isRetryable: transport failures", () => {
looped.cause = looped; looped.cause = looped;
expect(isRetryable(looped)).toBe(false); expect(isRetryable(looped)).toBe(false);
}); });
it("terminates on a cause chain that loops through two errors", () => {
const first: Error & { cause?: unknown } = new Error("first");
const second = new Error("second", { cause: first });
first.cause = second;
expect(isRetryable(first)).toBe(false);
});
it("reads the error and at most seven causes below it", () => {
// The walk is bounded at eight links. An errno at the eighth link is
// found; one at the ninth is not.
const buried = (causes: number): Error => {
let err = errnoError("ECONNRESET");
for (let i = 0; i < causes; i++) {
err = new Error(`wrapper ${i}`, { cause: err });
}
return err;
};
expect(isRetryable(buried(7))).toBe(true);
expect(isRetryable(buried(8))).toBe(false);
});
}); });
describe("isRetryable: stream truncation versus corruption", () => { describe("isRetryable: stream truncation versus corruption", () => {
@@ -305,9 +279,10 @@ describe("isSafeToReplay", () => {
* that is not the whole question: the other half is "could the first * that is not the whole question: the other half is "could the first
* attempt already have taken effect on the server?". * attempt already have taken effect on the server?".
* *
* The calls this guards are the `POST` and `PUT` requests listed in the * quak's non-idempotent calls are `/users/srp/create-session`,
* README under "Endpoints used". A blind replay of some of them can do * `/users/two-factor/verify` (which consumes one of a limited number of
* real damage, so they retry only on the failures that establish no TCP * 2FA attempts) and `/files/thumbnail`. A blind replay of any of them can
* do real damage, so they retry only on the failures that establish no TCP
* connection to the server ever existed — DNS produced no address, or the * connection to the server ever existed — DNS produced no address, or the
* peer refused the connection — and therefore that no request byte can * peer refused the connection — and therefore that no request byte can
* have been transmitted. * have been transmitted.
@@ -358,52 +333,6 @@ describe("isSafeToReplay", () => {
).toBe(false); ).toBe(false);
expect(isSafeToReplay(new TypeError("fetch failed"))).toBe(false); expect(isSafeToReplay(new TypeError("fetch failed"))).toBe(false);
}); });
it("does not replay any other errno the classifier names", () => {
for (const code of [
"ECONNRESET",
"ECONNABORTED",
"ETIMEDOUT",
"EPIPE",
"EHOSTUNREACH",
"ENETUNREACH",
"ENETRESET",
"ENETDOWN",
]) {
expect(isSafeToReplay(errnoError(code))).toBe(false);
}
});
it("does not replay a chain that also shows the request may have gone out", () => {
// A connect errno somewhere in the chain is not enough: any other
// errno beside it is doubt, and doubt is not replayed.
const reset = Object.assign(
new Error("read ECONNRESET", { cause: errnoError("ECONNREFUSED") }),
{ code: "ECONNRESET" },
);
const mixed = new TypeError("fetch failed", { cause: reset });
expect(isRetryable(mixed)).toBe(true);
expect(isSafeToReplay(mixed)).toBe(false);
});
it("does not replay a chain longer than the walk reads", () => {
// Eight connect errnos, then a reset at the ninth link, below the
// limit. The walk never sees the reset, so it cannot rule it out.
const refusedChain = (below: Error | undefined): Error => {
let err = below;
for (let i = 0; i < 8; i++) {
err = Object.assign(new Error(`refused ${i}`, { cause: err }), {
code: "ECONNREFUSED",
});
}
return err as Error;
};
expect(isSafeToReplay(refusedChain(errnoError("ECONNRESET")))).toBe(
false,
);
// The same eight links with nothing below them are replayable.
expect(isSafeToReplay(refusedChain(undefined))).toBe(true);
});
}); });
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------