Accumulate streamDecrypt reads linearly, not quadratically (closes #21)
check / check (push) Successful in 35s

streamDecrypt reallocated and recopied its whole accumulation buffer on
every network read, so a chunk delivered as N reads cost O(chunk^2/read)
bytes copied — for a 4 MiB chunk, about a second of memory churn against
~20 ms of libsodium work, and the dominant cost of a large backup.

Incoming reads are now queued as-is with a running byte count and only
stitched into a contiguous buffer at each ENC_CHUNK_SIZE boundary, so
each received byte is copied once regardless of file size. A read that
straddles a boundary is split with a subarray view, its tail requeued.

Wire format, chunk framing, truncation detection, retry semantics, and
the per-chunk progress hook are unchanged; behaviour is identical. A new
test feeds a multi-chunk body through a ReadableStream in many small
1000-byte pieces — the fragmented-read path the other fixtures, which
deliver each body in one read, never exercised — and asserts the
plaintext is byte-identical and the chunk framing intact.

Model: opus-4-8
This commit is contained in:
2026-09-22 12:57:59 +00:00
parent 7570055a5b
commit 880bd26d67
2 changed files with 118 additions and 10 deletions
+80
View File
@@ -1226,6 +1226,86 @@ describe("download retries: corruption is not retried", () => {
});
});
// ---------------------------------------------------------------------------
// Fragmented network reads
//
// A CDN does not hand the body over one secretstream chunk at a time; it
// arrives in whatever pieces the socket produces, many of them far smaller than
// a chunk and most straddling a chunk boundary. `streamDecrypt` reassembles
// those pieces before decrypting, copying each received byte once rather than
// recopying the whole accumulator on every read. This is the path the other
// fixtures never take — their mock fetch delivers each body as a single
// `Response` value, i.e. one read — so it is exercised explicitly here.
// ---------------------------------------------------------------------------
/**
* A fetch that serves `body` through a `ReadableStream` sliced into many
* fixed-size pieces, imitating a socket that trickles bytes in. `pieceSize` is
* chosen not to divide the chunk framing evenly, so pieces straddle the
* `ENC_CHUNK_SIZE` boundary the downloader splits on — the case a single-value
* body can never produce. `emitted` reports how many pieces were yielded, so a
* test can assert the body really was fragmented and not delivered whole.
*/
const mockFetchForFragmentedBody = (
body: Uint8Array,
pieceSize: number,
): { fetch: typeof globalThis.fetch; emitted: () => number } => {
let pieces = 0;
const fake = async (): Promise<Response> =>
new Response(
new ReadableStream<Uint8Array>({
start(controller) {
for (let off = 0; off < body.length; off += pieceSize) {
controller.enqueue(body.subarray(off, off + pieceSize));
pieces++;
}
controller.close();
},
}),
{ status: 200 },
);
return { fetch: fake as typeof globalThis.fetch, emitted: () => pieces };
};
describe("streamDecrypt fragmented reads", () => {
it("decrypts a multi-chunk body delivered in many small pieces", async () => {
// The multi-chunk fixture (one full 4 MiB chunk plus a small final
// chunk) delivered in 1000-byte pieces: several thousand reads, with
// the piece that spans the 4 MiB + 17 byte chunk boundary split across
// two chunks by the reassembler. The plaintext must come out
// byte-identical to the single-read case, and the chunk framing must be
// untouched: exactly two writes, `STREAM_CHUNK_SIZE` then the final
// chunk, the same as when the body arrives whole. If the boundary
// handling were off by a byte under fragmentation, either the pull
// would fail to authenticate or the write sizes would shift.
const { fetch, emitted } = mockFetchForFragmentedBody(
multiChunk.body,
1000,
);
const api = new ApiClient({ fetch });
const file = buildMockEnteFile(
multiChunkKey,
multiChunk.header,
multiChunk.header,
);
const dir = mkdtempSync(join(testDir, "fragmented-"));
const outPath = join(dir, "fragmented.bin");
const result = await downloadFile(api, file, outPath);
// The body really was trickled in, not handed over whole.
expect(emitted()).toBeGreaterThan(1000);
const writes = writeHook.writes.filter((w) => w.path.endsWith(".tmp"));
expect(writes.map((w) => w.length)).toEqual([
STREAM_CHUNK_SIZE,
multiChunk.plaintext.length - STREAM_CHUNK_SIZE,
]);
expect(result.bytesWritten).toBe(multiChunk.plaintext.length);
expectSameBytes(readFileSync(outPath), multiChunk.plaintext);
});
});
// ---------------------------------------------------------------------------
// Durable atomic writes
//