The state file is renamed into place without being synced, so a crash can leave it empty and stop the next start #264

Closed
opened 2026-10-13 12:32:08 +02:00 by clawbot · 0 comments
Collaborator

What is wrong. Save writes state.json.tmp with os.WriteFile and renames it over state.json without syncing the file (or the directory) first. After a power loss or kernel crash the rename can be on disk while the file's contents are not, leaving state.json empty or truncated. Load then fails with parsing state file: ..., OnStart returns the error, and dnswatcher exits at every start until someone deletes the file. Monitoring stays down until a person acts.

Where (at e8b4006): internal/state/state.go, Save lines 255-266; Load lines 210-215 returns an error for a file that does not parse; New line 178 makes that a startup error.

Scenario. The host loses power shortly after a check saved state. The file is rewritten after every check, so the window comes round every interval. On boot the container starts, logs parsing state file: unexpected end of JSON input, and exits; the restart loop never gets past it. Not reproduced: the missing sync is confirmed by reading, and the empty-file outcome is the known behaviour of write-then-rename without a sync (XFS, and ext4 outside its rename heuristic).

Correct. README, State Management: "State is written atomically (write to temp file, then rename) to prevent corruption." The temp file is synced before the rename, and the directory after it, so the rename never publishes data that is not on disk. Whether an unreadable state file should also be moved aside and logged rather than stop the start is a separate call for the owner; my reading is no, a start that silently drops state is worse than one that stops.

Done when: Save syncs the temp file before Rename and the directory after it.

Severity: major

Model: fable-5-1

**What is wrong.** `Save` writes `state.json.tmp` with `os.WriteFile` and renames it over `state.json` without syncing the file (or the directory) first. After a power loss or kernel crash the rename can be on disk while the file's contents are not, leaving `state.json` empty or truncated. `Load` then fails with `parsing state file: ...`, `OnStart` returns the error, and dnswatcher exits at every start until someone deletes the file. Monitoring stays down until a person acts. **Where** (at `e8b4006`): `internal/state/state.go`, `Save` lines 255-266; `Load` lines 210-215 returns an error for a file that does not parse; `New` line 178 makes that a startup error. **Scenario.** The host loses power shortly after a check saved state. The file is rewritten after every check, so the window comes round every interval. On boot the container starts, logs `parsing state file: unexpected end of JSON input`, and exits; the restart loop never gets past it. Not reproduced: the missing sync is confirmed by reading, and the empty-file outcome is the known behaviour of write-then-rename without a sync (XFS, and ext4 outside its rename heuristic). **Correct.** README, State Management: "State is written atomically (write to temp file, then rename) to prevent corruption." The temp file is synced before the rename, and the directory after it, so the rename never publishes data that is not on disk. Whether an unreadable state file should also be moved aside and logged rather than stop the start is a separate call for the owner; my reading is no, a start that silently drops state is worse than one that stops. **Done when:** `Save` syncs the temp file before `Rename` and the directory after it. Severity: major Model: fable-5-1
clawbot added this to the 1.0 milestone 2026-10-13 12:32:08 +02:00
clawbot self-assigned this 2026-10-13 12:32:08 +02:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/dnswatcher#264