Accumulate streamDecrypt reads linearly, not quadratically (closes #21)
check / check (push) Successful in 35s
check / check (push) Successful in 35s
streamDecrypt reallocated and recopied its whole accumulation buffer on every network read, so a chunk delivered as N reads cost O(chunk^2/read) bytes copied — for a 4 MiB chunk, about a second of memory churn against ~20 ms of libsodium work, and the dominant cost of a large backup. Incoming reads are now queued as-is with a running byte count and only stitched into a contiguous buffer at each ENC_CHUNK_SIZE boundary, so each received byte is copied once regardless of file size. A read that straddles a boundary is split with a subarray view, its tail requeued. Wire format, chunk framing, truncation detection, retry semantics, and the per-chunk progress hook are unchanged; behaviour is identical. A new test feeds a multi-chunk body through a ReadableStream in many small 1000-byte pieces — the fragmented-read path the other fixtures, which deliver each body in one read, never exercised — and asserts the plaintext is byte-identical and the chunk framing intact. Model: opus-4-8
This commit is contained in:
@@ -1226,6 +1226,86 @@ describe("download retries: corruption is not retried", () => {
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Fragmented network reads
|
||||
//
|
||||
// A CDN does not hand the body over one secretstream chunk at a time; it
|
||||
// arrives in whatever pieces the socket produces, many of them far smaller than
|
||||
// a chunk and most straddling a chunk boundary. `streamDecrypt` reassembles
|
||||
// those pieces before decrypting, copying each received byte once rather than
|
||||
// recopying the whole accumulator on every read. This is the path the other
|
||||
// fixtures never take — their mock fetch delivers each body as a single
|
||||
// `Response` value, i.e. one read — so it is exercised explicitly here.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* A fetch that serves `body` through a `ReadableStream` sliced into many
|
||||
* fixed-size pieces, imitating a socket that trickles bytes in. `pieceSize` is
|
||||
* chosen not to divide the chunk framing evenly, so pieces straddle the
|
||||
* `ENC_CHUNK_SIZE` boundary the downloader splits on — the case a single-value
|
||||
* body can never produce. `emitted` reports how many pieces were yielded, so a
|
||||
* test can assert the body really was fragmented and not delivered whole.
|
||||
*/
|
||||
const mockFetchForFragmentedBody = (
|
||||
body: Uint8Array,
|
||||
pieceSize: number,
|
||||
): { fetch: typeof globalThis.fetch; emitted: () => number } => {
|
||||
let pieces = 0;
|
||||
const fake = async (): Promise<Response> =>
|
||||
new Response(
|
||||
new ReadableStream<Uint8Array>({
|
||||
start(controller) {
|
||||
for (let off = 0; off < body.length; off += pieceSize) {
|
||||
controller.enqueue(body.subarray(off, off + pieceSize));
|
||||
pieces++;
|
||||
}
|
||||
controller.close();
|
||||
},
|
||||
}),
|
||||
{ status: 200 },
|
||||
);
|
||||
return { fetch: fake as typeof globalThis.fetch, emitted: () => pieces };
|
||||
};
|
||||
|
||||
describe("streamDecrypt fragmented reads", () => {
|
||||
it("decrypts a multi-chunk body delivered in many small pieces", async () => {
|
||||
// The multi-chunk fixture (one full 4 MiB chunk plus a small final
|
||||
// chunk) delivered in 1000-byte pieces: several thousand reads, with
|
||||
// the piece that spans the 4 MiB + 17 byte chunk boundary split across
|
||||
// two chunks by the reassembler. The plaintext must come out
|
||||
// byte-identical to the single-read case, and the chunk framing must be
|
||||
// untouched: exactly two writes, `STREAM_CHUNK_SIZE` then the final
|
||||
// chunk, the same as when the body arrives whole. If the boundary
|
||||
// handling were off by a byte under fragmentation, either the pull
|
||||
// would fail to authenticate or the write sizes would shift.
|
||||
const { fetch, emitted } = mockFetchForFragmentedBody(
|
||||
multiChunk.body,
|
||||
1000,
|
||||
);
|
||||
const api = new ApiClient({ fetch });
|
||||
const file = buildMockEnteFile(
|
||||
multiChunkKey,
|
||||
multiChunk.header,
|
||||
multiChunk.header,
|
||||
);
|
||||
const dir = mkdtempSync(join(testDir, "fragmented-"));
|
||||
const outPath = join(dir, "fragmented.bin");
|
||||
|
||||
const result = await downloadFile(api, file, outPath);
|
||||
|
||||
// The body really was trickled in, not handed over whole.
|
||||
expect(emitted()).toBeGreaterThan(1000);
|
||||
|
||||
const writes = writeHook.writes.filter((w) => w.path.endsWith(".tmp"));
|
||||
expect(writes.map((w) => w.length)).toEqual([
|
||||
STREAM_CHUNK_SIZE,
|
||||
multiChunk.plaintext.length - STREAM_CHUNK_SIZE,
|
||||
]);
|
||||
expect(result.bytesWritten).toBe(multiChunk.plaintext.length);
|
||||
expectSameBytes(readFileSync(outPath), multiChunk.plaintext);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Durable atomic writes
|
||||
//
|
||||
|
||||
Reference in New Issue
Block a user