Download module: shared atomic writer with fsync, and per-chunk progress #39

New Issue

2026-09-22T09:10:58+02:00

clawbot commented

2026-09-22 09:10:58 +02:00

Foundation unit for the cache/API design (#36).
Independent of the decrypt-fields and enumeration units (touches only the
download module), so it can run in parallel with them.

Goal

Two small download-module changes the cache and the GUI need:

A shared atomic file writer that survives a power cut (fsync the file before
the rename, fsync the directory after), exported for the cache to use.
A per-chunk progress hook on downloadFile/downloadThumbnail so callers
can report bytes-written as the body streams (feeds the design's
onProgress bytesDone).

Current state (grounded, `src/download/index.ts`)

writeAtomic(destination, plaintext) stages a temp sibling and renames, but
is module-private and does NOT fsync — after a crash the renamed file can be
empty (tracked as #22).
downloadFile/downloadThumbnail report bytesWritten only in the final
DownloadResult; there is no signal while the body streams.

Scope

Add fsync to the atomic writer: open the temp file, write, fsync it, close,
rename, then fsync the containing directory. Keep the existing best-effort
temp cleanup on error. Export it (its own small module or a named export from
the download module) so the metadata store can reuse one writer.
Add an optional onProgress?: (bytesDone: number) => void (or a richer typed
event if that is cleaner — align with the design's ProgressEvent) to
downloadFile/downloadThumbnail, called as decrypted plaintext accumulates.
For now, whole-file buffering stays (streaming-to-disk is the next unit); the
hook can fire per pulled chunk inside streamDecrypt, threaded out through the
download functions. Keep the callback optional and side-effect-free on the
happy path when absent.
Do not change retry semantics or the TAG_FINAL truncation checks.

Definition of done

Atomic writer fsyncs file and directory; exported and covered by a test
(a test may assert the temp-then-rename path and that no partial file is left
on a simulated write failure).
Progress callback fires with monotonically increasing byte counts and a final
value equal to bytesWritten; tested.
Existing download tests still pass; make check green.
Overlaps #22 (fsync durability). Note it
in the landing commit; close #22 only if this unit fully covers its scope,
otherwise cross-reference and leave orphan-reaping to the content-store unit.

Grounding

Files: src/download/index.ts (+ possibly a new small atomic-write module),
test/download/download.test.ts.

Dispatch notes: TDD; no scripted edits; no interactive questions. Squash subject
ends (closes #<this issue>). End every message with Model: opus-4-8.

Model: opus-4-8

Foundation unit for the cache/API design (https://git.eeqj.de/sneak/quak/issues/36). Independent of the decrypt-fields and enumeration units (touches only the download module), so it can run in parallel with them. ## Goal Two small download-module changes the cache and the GUI need: 1. A shared atomic file writer that survives a power cut (fsync the file before the rename, fsync the directory after), exported for the cache to use. 2. A per-chunk progress hook on `downloadFile`/`downloadThumbnail` so callers can report bytes-written as the body streams (feeds the design's `onProgress` `bytesDone`). ## Current state (grounded, `src/download/index.ts`) - `writeAtomic(destination, plaintext)` stages a temp sibling and renames, but is module-private and does NOT fsync — after a crash the renamed file can be empty (tracked as https://git.eeqj.de/sneak/quak/issues/22). - `downloadFile`/`downloadThumbnail` report `bytesWritten` only in the final `DownloadResult`; there is no signal while the body streams. ## Scope - Add fsync to the atomic writer: open the temp file, write, `fsync` it, close, `rename`, then `fsync` the containing directory. Keep the existing best-effort temp cleanup on error. Export it (its own small module or a named export from the download module) so the metadata store can reuse one writer. - Add an optional `onProgress?: (bytesDone: number) => void` (or a richer typed event if that is cleaner — align with the design's `ProgressEvent`) to `downloadFile`/`downloadThumbnail`, called as decrypted plaintext accumulates. For now, whole-file buffering stays (streaming-to-disk is the next unit); the hook can fire per pulled chunk inside `streamDecrypt`, threaded out through the download functions. Keep the callback optional and side-effect-free on the happy path when absent. - Do not change retry semantics or the `TAG_FINAL` truncation checks. ## Definition of done - Atomic writer fsyncs file and directory; exported and covered by a test (a test may assert the temp-then-rename path and that no partial file is left on a simulated write failure). - Progress callback fires with monotonically increasing byte counts and a final value equal to `bytesWritten`; tested. - Existing download tests still pass; `make check` green. - Overlaps https://git.eeqj.de/sneak/quak/issues/22 (fsync durability). Note it in the landing commit; close #22 only if this unit fully covers its scope, otherwise cross-reference and leave orphan-reaping to the content-store unit. ## Grounding Files: `src/download/index.ts` (+ possibly a new small atomic-write module), `test/download/download.test.ts`. Dispatch notes: TDD; no scripted edits; no interactive questions. Squash subject ends ` (closes #<this issue>)`. End every message with `Model: opus-4-8`. Model: opus-4-8

clawbot referenced this issue

2026-09-22 09:11:56 +02:00

Design for review: a cache layer (SQLite metadata + on-disk content/thumbnail store) and a crash-safe backup on it #36

clawbot referenced this issue

2026-09-22 09:51:51 +02:00

Owner decision: quak (priority repo) is starved of worker capacity #54

clawbot referenced this issue from a commit

2026-09-22 11:35:40 +02:00

feat(download): durable atomic writer and per-chunk progress hook (closes #39)

clawbot referenced a pull request that will close this issue

2026-09-22 11:36:16 +02:00

feat(download): durable atomic writer and per-chunk progress hook (closes #39) #56

clawbot commented

2026-09-22 11:36:32 +02:00

Implemented in #56 (branch issue-39-atomic-fsync-progress, base next).

writeAtomic fsyncs the staged temp file before the rename and the containing
directory after, and is now exported for the metadata store to reuse.
downloadFile/downloadThumbnail take an optional onProgress(bytesDone)
hook fired within streamDecrypt; values are non-decreasing and the last
equals bytesWritten, and it is a no-op when omitted.
Whole-file buffering, retry semantics, and the TAG_FINAL truncation checks are
unchanged.

Tests added for the two fsyncs and their order, the temp-file cleanup on a failed
rename, and monotonic progress ending at bytesWritten; existing download tests
pass and make check is green.

This covers only the durability gap (area 1) of
#22; orphan reaping and the rest there
remain for a later unit, so it is referenced, not closed.

Model: opus-4-8

Implemented in https://git.eeqj.de/sneak/quak/pulls/56 (branch `issue-39-atomic-fsync-progress`, base `next`). - `writeAtomic` fsyncs the staged temp file before the rename and the containing directory after, and is now exported for the metadata store to reuse. - `downloadFile`/`downloadThumbnail` take an optional `onProgress(bytesDone)` hook fired within `streamDecrypt`; values are non-decreasing and the last equals `bytesWritten`, and it is a no-op when omitted. - Whole-file buffering, retry semantics, and the TAG_FINAL truncation checks are unchanged. Tests added for the two fsyncs and their order, the temp-file cleanup on a failed rename, and monotonic progress ending at `bytesWritten`; existing download tests pass and `make check` is green. This covers only the durability gap (area 1) of https://git.eeqj.de/sneak/quak/issues/22; orphan reaping and the rest there remain for a later unit, so it is referenced, not closed. Model: opus-4-8

clawbot closed this issue

2026-09-22 12:01:06 +02:00

clawbot referenced this issue from a commit

2026-09-22 12:01:07 +02:00

Durable atomic writer with fsync and per-chunk download progress (closes #39)

clawbot referenced this issue

2026-09-22 12:33:36 +02:00

On-disk JSON metadata store for the local cache (closes #41) #58

clawbot referenced this issue

2026-09-23 02:08:53 +02:00

Atomic download writes: durability, orphan reaping, and untested failure paths #22

clawbot referenced this issue