Stream decrypted downloads to disk instead of buffering whole files in memory #40

New Issue

2026-09-22T09:10:59+02:00

clawbot commented

2026-09-22 09:10:59 +02:00

Foundation unit for the cache/API design (#36).
Depends on the download-plumbing unit (the fsync atomic writer). This is a
precondition for precaching originals and for the GUI viewer, per the design.

Goal

Stop buffering a whole decrypted file in RAM. Stream the decrypted plaintext
straight to the temp file, chunk by chunk, then rename it into place. A
multi-gigabyte video must not need ~2× its size in memory, and five concurrent
originals must not mean five whole files resident.

Current state (grounded, `src/download/index.ts`)

streamDecrypt accumulates every plaintext chunk in plainChunks[] and
returns one Uint8Array; downloadFile/downloadThumbnail then hand that
whole buffer to the atomic writer. RAM cost scales with file size (this is
#21).
The TAG_FINAL truncation guarantee must be preserved: a body cut short must
never leave a complete-looking file on disk.

Scope

Add a streaming decrypt-to-temp path: pull each secretstream chunk and write
its plaintext to the temp file as it is produced (reuse the exported fsync
atomic writer's temp-then-rename discipline). The rename happens only after
the stream authenticates as terminated on TAG_FINAL; a truncated stream
deletes the temp file and throws exactly as today.
Preserve retry semantics: a retry restarts from byte zero (secretstream pull
state is not resumable) and must truncate/replace the temp file cleanly.
Keep the per-chunk progress hook working through the streaming path.
Memory during a download must be bounded by the chunk size, not the file size.
A whole-buffer helper may remain for small payloads (thumbnails, metadata) if
it keeps the code simpler, but originals must stream.

Definition of done

Originals decrypt-and-write with memory bounded by chunk size (tested with a
multi-chunk fixture; assert the temp file grows and the buffer is not retained
— e.g. via a spy on writes, or a large synthetic stream under a heap check if
practical).
Truncation still rejected and leaves no destination file; TAG_FINAL check
intact; retry restarts cleanly. All covered by tests.
make check green. Closes #21.

Grounding

Files: src/download/index.ts, src/crypto/stream.ts (if the pull API needs a
streaming accessor), test/download/download.test.ts, test/crypto/stream.test.ts.
Subsumes #21.

Dispatch notes: TDD; no scripted edits; no interactive questions. Squash subject
ends (closes #<this issue>). End every message with Model: opus-4-8.

Model: opus-4-8

Foundation unit for the cache/API design (https://git.eeqj.de/sneak/quak/issues/36). Depends on the download-plumbing unit (the fsync atomic writer). This is a precondition for precaching originals and for the GUI viewer, per the design. ## Goal Stop buffering a whole decrypted file in RAM. Stream the decrypted plaintext straight to the temp file, chunk by chunk, then rename it into place. A multi-gigabyte video must not need ~2× its size in memory, and five concurrent originals must not mean five whole files resident. ## Current state (grounded, `src/download/index.ts`) - `streamDecrypt` accumulates every plaintext chunk in `plainChunks[]` and returns one `Uint8Array`; `downloadFile`/`downloadThumbnail` then hand that whole buffer to the atomic writer. RAM cost scales with file size (this is https://git.eeqj.de/sneak/quak/issues/21). - The `TAG_FINAL` truncation guarantee must be preserved: a body cut short must never leave a complete-looking file on disk. ## Scope - Add a streaming decrypt-to-temp path: pull each secretstream chunk and write its plaintext to the temp file as it is produced (reuse the exported fsync atomic writer's temp-then-rename discipline). The rename happens only after the stream authenticates as terminated on `TAG_FINAL`; a truncated stream deletes the temp file and throws exactly as today. - Preserve retry semantics: a retry restarts from byte zero (secretstream pull state is not resumable) and must truncate/replace the temp file cleanly. - Keep the per-chunk progress hook working through the streaming path. - Memory during a download must be bounded by the chunk size, not the file size. A whole-buffer helper may remain for small payloads (thumbnails, metadata) if it keeps the code simpler, but originals must stream. ## Definition of done - Originals decrypt-and-write with memory bounded by chunk size (tested with a multi-chunk fixture; assert the temp file grows and the buffer is not retained — e.g. via a spy on writes, or a large synthetic stream under a heap check if practical). - Truncation still rejected and leaves no destination file; `TAG_FINAL` check intact; retry restarts cleanly. All covered by tests. - `make check` green. Closes https://git.eeqj.de/sneak/quak/issues/21. ## Grounding Files: `src/download/index.ts`, `src/crypto/stream.ts` (if the pull API needs a streaming accessor), `test/download/download.test.ts`, `test/crypto/stream.test.ts`. Subsumes https://git.eeqj.de/sneak/quak/issues/21. Dispatch notes: TDD; no scripted edits; no interactive questions. Squash subject ends ` (closes #<this issue>)`. End every message with `Model: opus-4-8`. Model: opus-4-8

clawbot referenced this issue

2026-09-22 09:11:56 +02:00

Design for review: a cache layer (SQLite metadata + on-disk content/thumbnail store) and a crash-safe backup on it #36

clawbot referenced this issue from a commit

2026-09-22 12:41:54 +02:00

Stream decrypted downloads to disk with bounded memory (closes #40)

clawbot referenced a pull request that will close this issue

2026-09-22 12:42:27 +02:00

Stream decrypted downloads to disk with bounded memory (closes #40) #59

clawbot commented

2026-09-22 12:42:44 +02:00

Implemented in #59 (base next, needs-review).

Decrypted downloads now stream straight to disk: each secretstream chunk is written to the staged temp file as it decrypts and dropped, and the file is renamed into place only after the stream authenticates on TAG_FINAL. Peak memory is one 4 MiB chunk regardless of file size. Truncation is still rejected with no destination file left behind, and a retry restarts from byte zero into a fresh temp file. Also closes #21.

Model: opus-4-8

Implemented in https://git.eeqj.de/sneak/quak/pulls/59 (base `next`, needs-review). Decrypted downloads now stream straight to disk: each secretstream chunk is written to the staged temp file as it decrypts and dropped, and the file is renamed into place only after the stream authenticates on `TAG_FINAL`. Peak memory is one 4 MiB chunk regardless of file size. Truncation is still rejected with no destination file left behind, and a retry restarts from byte zero into a fresh temp file. Also closes https://git.eeqj.de/sneak/quak/issues/21. Model: opus-4-8

clawbot referenced this issue

2026-09-22 13:08:25 +02:00

Stream decrypted downloads to disk with bounded memory (closes #40) #59

clawbot referenced a pull request that will close this issue

2026-09-22 13:14:24 +02:00

Stream decrypted downloads to disk with bounded memory (closes #40) #59

clawbot referenced this issue from a commit

2026-09-22 13:48:37 +02:00

Stream decrypted downloads to disk with bounded memory (closes #40)

clawbot referenced this issue

2026-09-22 13:49:13 +02:00

Stream decrypted downloads to disk with bounded memory (closes #40) #59