No lock on DATA_DIR: two instances both run delivery recovery and both deliver #201

New Issue

clawbot · 2026-08-20T05:47:02+02:00

clawbot commented

2026-08-20 05:47:02 +02:00

Nothing prevents two processes from opening the same DATA_DIR. Both open the same per-webhook SQLite files and both run delivery recovery over the same rows, so both deliver.

Reproduced. Attempt counts per delivery, before and after a second process started:

attempt_num | count
1|1  2|1  3|1  4|1
5|2  6|2  7|2  8|2

Every attempt from the moment the second process came up was delivered twice to the destination. The second process in this reproduction was the non-serving zombie from the listen-failure issue filed alongside this one, but any double start or overlapping deploy does the same thing.

Duplicate delivery to a customer endpoint is the worst failure this product can have, and the trigger is a mundane operator slip.

Definition of done:

an exclusive lock is taken on DATA_DIR at startup, held for the process lifetime, released on exit
a second instance pointed at the same DATA_DIR refuses to start and exits non-zero with a clear message naming the directory
the lock is not left stale by kill -9: verify a hard-killed process's lock does not block the next start (advisory flock on an open fd gives this for free; a bare pidfile does not)
a regression test covers both: second instance refuses, and restart after kill -9 succeeds

Nothing prevents two processes from opening the same `DATA_DIR`. Both open the same per-webhook SQLite files and both run delivery recovery over the same rows, so both deliver. Reproduced. Attempt counts per delivery, before and after a second process started: ``` attempt_num | count 1|1 2|1 3|1 4|1 5|2 6|2 7|2 8|2 ``` Every attempt from the moment the second process came up was delivered twice to the destination. The second process in this reproduction was the non-serving zombie from the listen-failure issue filed alongside this one, but any double start or overlapping deploy does the same thing. Duplicate delivery to a customer endpoint is the worst failure this product can have, and the trigger is a mundane operator slip. Definition of done: - an exclusive lock is taken on `DATA_DIR` at startup, held for the process lifetime, released on exit - a second instance pointed at the same `DATA_DIR` refuses to start and exits non-zero with a clear message naming the directory - the lock is not left stale by `kill -9`: verify a hard-killed process's lock does not block the next start (advisory `flock` on an open fd gives this for free; a bare pidfile does not) - a regression test covers both: second instance refuses, and restart after `kill -9` succeeds

clawbot added this to the 1.0.0 milestone 2026-08-20 05:47:02 +02:00

clawbot referenced this issue

2026-08-20 05:49:42 +02:00

1.0/mvp #33

clawbot referenced this issue

2026-08-20 05:56:52 +02:00

Milestone 1.0.0: internet-facing readiness #111

clawbot referenced this issue from a commit

2026-08-20 06:22:49 +02:00

Lock DATA_DIR against a second instance (closes #201)

clawbot referenced a pull request that will close this issue

2026-08-20 06:23:16 +02:00