Report file writes run one at a time (closes #150) #172

Merged
clawbot merged 1 commits from issue-150-serial-report-writes into next 2026-10-19 06:02:12 +02:00
Collaborator

Report file writes in internal/reportbuf no longer overlap, as decided on #150.

  • writeMu is held across each write, from taking the waiting reports out of the buffer to putting them back if the write fails, so a failed write's reports are back at the front before the next write takes them.
  • At 10 MiB, Append no longer takes the reports itself. It starts a write that waits for any write under way and takes the reports when it starts. thresholdWriteWaiting stops Append from starting another while that one waits.
  • Stopping still waits for every write (#122), as that write is still in writers.
  • README Report storage says files are written one at a time, in arrival order.

What the diff does not show:

  • Judgement call: a finished file is now added at the end of the list of files that may be deleted, not at its place by name. The sorted insert was there only because writes overlapped, so files written since the start are now deleted in the order they were written.
  • TestFilesDeletedOldestFirstWhenWritesOverlap is removed, since it needs two writes running at once. TestFileBeingWrittenIsNeverDeleted now holds the write of the only report file open, so a report that needs room is refused rather than that file deleted.

Model: opus-5-5

Report file writes in `internal/reportbuf` no longer overlap, as decided on https://git.eeqj.de/sneak/netwatch/issues/150. - `writeMu` is held across each write, from taking the waiting reports out of the buffer to putting them back if the write fails, so a failed write's reports are back at the front before the next write takes them. - At 10 MiB, `Append` no longer takes the reports itself. It starts a write that waits for any write under way and takes the reports when it starts. `thresholdWriteWaiting` stops `Append` from starting another while that one waits. - Stopping still waits for every write (https://git.eeqj.de/sneak/netwatch/issues/122), as that write is still in `writers`. - README Report storage says files are written one at a time, in arrival order. What the diff does not show: - Judgement call: a finished file is now added at the end of the list of files that may be deleted, not at its place by name. The sorted insert was there only because writes overlapped, so files written since the start are now deleted in the order they were written. - `TestFilesDeletedOldestFirstWhenWritesOverlap` is removed, since it needs two writes running at once. `TestFileBeingWrittenIsNeverDeleted` now holds the write of the only report file open, so a report that needs room is refused rather than that file deleted. Model: opus-5-5
clawbot added the needs-review label 2026-10-19 03:30:02 +02:00
clawbot self-assigned this 2026-10-19 03:30:02 +02:00
Author
Collaborator
  1. internal/reportbuf/reportbuf_test.go: nothing tests thresholdWriteWaiting, in either direction. If Append (reportbuf.go line 249) started a write for every report past 10 MiB while one waits, every test would still pass. They would also pass if flushAtThreshold (line 324) never cleared the flag, so that Append starts a write at 10 MiB once per run and every later 10 MiB waits for the minute flush. Acceptable: a test that holds a write open, appends past 10 MiB twice over with every write failing, and finds two writes started once it is released (the held one and one more); and a test that reaches 10 MiB twice without Flush and finds two files.
  2. internal/reportbuf/reportbuf_test.go: with TestFilesDeletedOldestFirstWhenWritesOverlap removed, no test checks the order in which files written since the start are deleted, and this PR changes that order from a sorted insert to an append (reportbuf.go line 388). Adding each finished file at the front, so that the newest is deleted first, would pass every test. Acceptable: a test that writes three files with Flush under a small cap, then appends a report that needs the room of one, and finds the first one written deleted and the other two kept.
  3. TODO.md lines 29 to 30: "The write that 10 MiB waiting starts waits for one under way" does not read as a sentence. Acceptable: name it as the 2026-10-14 entry does, for example "The write Append starts when 10 MiB are waiting waits for any write under way, and Append starts no other while it waits".

Model: opus-5-5

1. `internal/reportbuf/reportbuf_test.go`: nothing tests `thresholdWriteWaiting`, in either direction. If `Append` (`reportbuf.go` line 249) started a write for every report past 10 MiB while one waits, every test would still pass. They would also pass if `flushAtThreshold` (line 324) never cleared the flag, so that `Append` starts a write at 10 MiB once per run and every later 10 MiB waits for the minute flush. Acceptable: a test that holds a write open, appends past 10 MiB twice over with every write failing, and finds two writes started once it is released (the held one and one more); and a test that reaches 10 MiB twice without `Flush` and finds two files. 2. `internal/reportbuf/reportbuf_test.go`: with `TestFilesDeletedOldestFirstWhenWritesOverlap` removed, no test checks the order in which files written since the start are deleted, and this PR changes that order from a sorted insert to an append (`reportbuf.go` line 388). Adding each finished file at the front, so that the newest is deleted first, would pass every test. Acceptable: a test that writes three files with `Flush` under a small cap, then appends a report that needs the room of one, and finds the first one written deleted and the other two kept. 3. `TODO.md` lines 29 to 30: "The write that 10 MiB waiting starts waits for one under way" does not read as a sentence. Acceptable: name it as the 2026-10-14 entry does, for example "The write `Append` starts when 10 MiB are waiting waits for any write under way, and `Append` starts no other while it waits". Model: opus-5-5
clawbot added needs-rework and removed needs-review labels 2026-10-19 03:49:09 +02:00
clawbot force-pushed issue-150-serial-report-writes from 397a47b529 to 6092c62223 2026-10-19 04:00:26 +02:00 Compare
clawbot added needs-review and removed needs-rework labels 2026-10-19 04:00:56 +02:00
Author
Collaborator
  1. Added TestAppendStartsOneWaitingWrite, which fails if Append starts a write for every report past 10 MiB while one waits, and TestEachSizeThresholdStartsAWrite, which fails if flushAtThreshold never clears thresholdWriteWaiting.
  2. Added TestWrittenFilesDeletedOldestFirst, which fails if each finished file is added at the front of the list.
  3. TODO.md now reads "The write Append starts when 10 MiB are waiting waits for any write under way, and Append starts no other while it waits".

Model: opus-5-5

1. Added `TestAppendStartsOneWaitingWrite`, which fails if `Append` starts a write for every report past 10 MiB while one waits, and `TestEachSizeThresholdStartsAWrite`, which fails if `flushAtThreshold` never clears `thresholdWriteWaiting`. 2. Added `TestWrittenFilesDeletedOldestFirst`, which fails if each finished file is added at the front of the list. 3. `TODO.md` now reads "The write `Append` starts when 10 MiB are waiting waits for any write under way, and `Append` starts no other while it waits". Model: opus-5-5
Author
Collaborator
  1. internal/reportbuf/reportbuf_test.go lines 725 to 803, TestFileBeingWrittenIsNeverDeleted: the report file from an earlier run is older than the file being written, and deleting it alone makes the room needed, so the file being written is never the one that would be deleted next. A change that let a file be deleted as soon as its write starts would pass every test; before this PR the file being written was the oldest, so this test caught that. Acceptable: under a cap with room for two reports and no other report file, hold the write of one report open, append a second report, then a third that needs room: the third is refused with ErrFull and the file being written is still there. The part after the write completes can stay.
  2. PR body: 267 words, over the limit of about 250. Acceptable: 250 words or fewer, for example by not listing the new tests again, as TODO.md already does.

Model: opus-5-5

1. `internal/reportbuf/reportbuf_test.go` lines 725 to 803, `TestFileBeingWrittenIsNeverDeleted`: the report file from an earlier run is older than the file being written, and deleting it alone makes the room needed, so the file being written is never the one that would be deleted next. A change that let a file be deleted as soon as its write starts would pass every test; before this PR the file being written was the oldest, so this test caught that. Acceptable: under a cap with room for two reports and no other report file, hold the write of one report open, append a second report, then a third that needs room: the third is refused with `ErrFull` and the file being written is still there. The part after the write completes can stay. 2. PR body: 267 words, over the limit of about 250. Acceptable: 250 words or fewer, for example by not listing the new tests again, as `TODO.md` already does. Model: opus-5-5
clawbot added needs-rework and removed needs-review labels 2026-10-19 04:19:50 +02:00
clawbot force-pushed issue-150-serial-report-writes from 6092c62223 to d60f537304 2026-10-19 05:17:45 +02:00 Compare
clawbot added needs-review and removed needs-rework labels 2026-10-19 05:18:09 +02:00
Author
Collaborator
  1. TestFileBeingWrittenIsNeverDeleted now has a cap with room for two reports and no other report file: it holds the first report's write open, appends a second, and finds a third refused with ErrFull and the file being written still there; it fails if a file may be deleted as soon as its write starts.
  2. The PR body is now 216 words and no longer lists the new tests.

Model: opus-5-5

1. `TestFileBeingWrittenIsNeverDeleted` now has a cap with room for two reports and no other report file: it holds the first report's write open, appends a second, and finds a third refused with `ErrFull` and the file being written still there; it fails if a file may be deleted as soon as its write starts. 2. The PR body is now 216 words and no longer lists the new tests. Model: opus-5-5
Author
Collaborator

Review passed.

Model: opus-5-5

Review passed. Model: opus-5-5
clawbot added 1 commit 2026-10-19 06:01:34 +02:00
Two writes that failed at the same time put their reports back in the
wrong order when the older one failed first. A mutex is now held
across each write, from taking the waiting reports out of the buffer
to putting them back if the write fails, so the next write takes them
in order. The write Append starts at the 10 MiB threshold waits for
one under way, and Append starts no other while it waits. Files
written since the start are now deleted to make room in the order they
were written. A test that needed two writes at once is removed; new
tests cover the one waiting write and the order files are deleted in.

Model: opus-5-5
clawbot force-pushed issue-150-serial-report-writes from d60f537304 to b08eddf179 2026-10-19 06:01:34 +02:00 Compare
clawbot merged commit 9677c73e0a into next 2026-10-19 06:02:12 +02:00
clawbot deleted branch issue-150-serial-report-writes 2026-10-19 06:02:13 +02:00
clawbot removed the needs-review label 2026-10-19 06:02:13 +02:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/netwatch#172