Observation, not a defect: one unreproducible SQLITE_IOERR_SHORT_READ storm under load on ZFS-backed storage #295

New Issue

clawbot · 2026-08-24T06:17:39+02:00

clawbot commented

2026-08-24 06:17:39 +02:00

Filed as a durable record, NOT as a defect. Nothing here is attributed to webhooker's code, and no fix is proposed. If this ever recurs on a real host, this note should make it diagnosable in minutes rather than a night.

What was seen, once

During the pre-tag end-to-end verification, in one run out of eight: 200 events at roughly 25/s with a concurrent sqlite3 <db> .dump loop produced 167 of 200 inbound webhooks answered 500 and 305 occurrences of disk I/O error (522).

522 is SQLITE_IOERR_SHORT_READ — a filesystem-level error. It is NOT SQLITE_BUSY, and it is not the lock-contention class that #256 addressed. It self-cleared, and a restart recovered every stranded delivery with nothing lost.

Why it is not attributed to this codebase

Seven further attempts failed to reproduce it, including deliberately harsher conditions than the run that triggered it:

a tighter reader loop (4,384 open/close cycles)
a sqlite3 .backup loop instead of .dump
500 events at 50/s with a continuous .dump running throughout — 9,000 deliveries, completely clean

The test host runs ZFS-backed Docker storage. SQLite's WAL mode memory-maps the -shm file, and mmap over ZFS is a known source of exactly this error code. That is the leading explanation and it is environmental, not a property of this program.

What the engine did while it was happening, which is the reassuring part

It behaved the way #256 intends:

logged loudly rather than continuing quietly
refused to write a status it could not stand behind, leaving rows recoverable rather than falsely terminal
lost no data — a restart recovered every stranded delivery

The residual cost was duplicate deliveries after the restart, which is the documented at-least-once behaviour, not a fault.

If this recurs

Check the storage layer FIRST, before the application:

Is DATA_DIR on ZFS, or on any filesystem where mmap semantics differ from a plain local disk (network filesystems, overlay filesystems, some container storage drivers)?
Error code 522 / SQLITE_IOERR_SHORT_READ points at the filesystem, not at locking. If the errors are SQLITE_BUSY or database is locked instead, that is a different problem and belongs with the durability work in #256.
The -shm file is the mmap'd one. Moving DATA_DIR to a plainly-backed local filesystem is the first thing to try.

SQLite documents WAL as requiring shared memory that behaves like POSIX shared memory, and warns specifically against WAL on network filesystems. If webhooker is ever deployed somewhere with unusual mmap semantics, that constraint is the thing to check.

Worth doing at some point, not now

The README's backup and deployment sections say nothing about filesystem requirements for DATA_DIR. A sentence stating that WAL needs a filesystem with working mmap, and naming network filesystems as unsupported, would be cheap and would prevent a genuinely confusing failure. Not urgent: the default deployment is a local volume or bind mount, which is fine.

Filed as a durable record, NOT as a defect. Nothing here is attributed to webhooker's code, and no fix is proposed. If this ever recurs on a real host, this note should make it diagnosable in minutes rather than a night. ## What was seen, once During the pre-tag end-to-end verification, in **one run out of eight**: 200 events at roughly 25/s with a concurrent `sqlite3 <db> .dump` loop produced **167 of 200 inbound webhooks answered `500`** and **305 occurrences of `disk I/O error (522)`**. `522` is `SQLITE_IOERR_SHORT_READ` — a filesystem-level error. It is NOT `SQLITE_BUSY`, and it is not the lock-contention class that https://git.eeqj.de/sneak/webhooker/issues/256 addressed. It self-cleared, and a restart recovered every stranded delivery with nothing lost. ## Why it is not attributed to this codebase Seven further attempts failed to reproduce it, including deliberately harsher conditions than the run that triggered it: - a tighter reader loop (4,384 open/close cycles) - a `sqlite3 .backup` loop instead of `.dump` - 500 events at 50/s with a continuous `.dump` running throughout — 9,000 deliveries, completely clean The test host runs **ZFS-backed Docker storage**. SQLite's WAL mode memory-maps the `-shm` file, and mmap over ZFS is a known source of exactly this error code. That is the leading explanation and it is environmental, not a property of this program. ## What the engine did while it was happening, which is the reassuring part It behaved the way https://git.eeqj.de/sneak/webhooker/issues/256 intends: - logged loudly rather than continuing quietly - refused to write a status it could not stand behind, leaving rows recoverable rather than falsely terminal - lost no data — a restart recovered every stranded delivery The residual cost was duplicate deliveries after the restart, which is the documented at-least-once behaviour, not a fault. ## If this recurs Check the storage layer FIRST, before the application: - Is `DATA_DIR` on ZFS, or on any filesystem where mmap semantics differ from a plain local disk (network filesystems, overlay filesystems, some container storage drivers)? - Error code `522` / `SQLITE_IOERR_SHORT_READ` points at the filesystem, not at locking. If the errors are `SQLITE_BUSY` or `database is locked` instead, that is a different problem and belongs with the durability work in #256. - The `-shm` file is the mmap'd one. Moving `DATA_DIR` to a plainly-backed local filesystem is the first thing to try. SQLite documents WAL as requiring shared memory that behaves like POSIX shared memory, and warns specifically against WAL on network filesystems. If webhooker is ever deployed somewhere with unusual mmap semantics, that constraint is the thing to check. ## Worth doing at some point, not now The README's backup and deployment sections say nothing about filesystem requirements for `DATA_DIR`. A sentence stating that WAL needs a filesystem with working mmap, and naming network filesystems as unsupported, would be cheap and would prevent a genuinely confusing failure. Not urgent: the default deployment is a local volume or bind mount, which is fine.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: sneak/webhooker#295