Lock DATA_DIR against a second instance (closes #201)
All checks were successful
check / check (push) Successful in 6m35s

Nothing stopped two processes opening the same DATA_DIR. Both open the
same per-webhook databases, both run delivery recovery over the same
rows, and both deliver: every pending delivery reaches the destination
twice, from nothing worse than an overlapping deploy.

The entry point now takes an exclusive advisory flock(2) on
{DATA_DIR}/webhooker.lock before anything opens a database, and holds it
for the process lifetime. A second process pointed at the same directory
prints a message naming that directory and exits 1. The lock is the
kernel's, not the file's, so a process killed with SIGKILL leaves a lock
file that blocks nothing -- which is what a pidfile would get wrong. The
file is never unlinked: doing so would let the next process lock a fresh
inode while a third still held the old one.

Acquisition lives in internal/datadir rather than in the server's fx
graph, so any entry point touching DATA_DIR takes it the same way, and
ErrLocked lets a caller tell a live deployment from any other failure.
config.DataDir() resolves DATA_DIR once, for both the lock and Config,
so the two cannot disagree.

Regression coverage: a real second process is refused, and a restart
after kill -9 succeeds with the stale lock file in place.

github.com/gofrs/flock carries the lock; its own module minimums pull
testify to v1.11.1 and golang.org/x/sys to v0.37.0.
This commit is contained in:
2026-08-20 04:13:56 +00:00
parent 10c8dd2331
commit 445ef57ada
9 changed files with 534 additions and 22 deletions

View File

@@ -115,6 +115,32 @@ TTY detection, and security headers are always applied.
| `RECEIVER_RATE_LIMIT` | Receiver requests/minute per IP per entrypoint (10x that per IP across the route) | `120` |
| `TRUSTED_PROXIES` | CIDRs whose forwarded headers are trusted (unset: all clients behind a proxy share one rate-limit bucket; a correct login password is never throttled either way) | `""` (none) |
#### Single-instance lock
Exactly one webhooker process may use a `DATA_DIR` at a time. Two
processes sharing one open the same databases and each run delivery
recovery over the same rows, so every pending delivery goes out twice —
duplicate delivery to your endpoints, from nothing worse than an
overlapping deploy or a double start.
At startup, before anything opens a database, the process takes an
exclusive advisory lock (`flock(2)`) on `{DATA_DIR}/webhooker.lock` and
holds it for its lifetime. A second process pointed at the same
directory prints a message naming it and exits non-zero:
```
webhooker: data directory is already in use by another instance: /var/lib/webhooker (/var/lib/webhooker/webhooker.lock). Only one webhooker may use a data directory: two both run delivery recovery over the same rows and both deliver
```
The lock is the kernel's, not the file's: it is released when the
process exits, including `kill -9`, so a leftover `webhooker.lock`
never blocks a restart and must not be deleted by hand. The file is
also left in place on a clean shutdown, deliberately — unlinking it
would let the next process lock a fresh inode while a third still held
the old one.
To run two webhookers on one host, give each its own `DATA_DIR`.
#### Trusted proxies
`TRUSTED_PROXIES` is a comma-separated list of CIDR blocks (a bare
@@ -283,10 +309,12 @@ is both the simplest and the only complete rule:
`{webhook_uuid}` is the webhook's UUID primary key in its canonical
36-character hyphenated form, so a real filename looks like
`events-3f2a1c9e-....db`. Nothing else is written to `DATA_DIR`, and no
`-wal` or `-shm` files are produced (see below); a transient
`{name}.db-journal` may exist beside a database while a write is in
flight and is not part of the backup set.
`events-3f2a1c9e-....db`. The only other file is `webhooker.lock`, the
always-empty [single-instance lock](#single-instance-lock); it holds no
state and is not part of the backup set — a copied one is stale and
blocks nothing. No `-wal` or `-shm` files are produced (see below); a
transient `{name}.db-journal` may exist beside a database while a write
is in flight and is not part of the backup set either.
Configuration is **not** in `DATA_DIR` — it comes from the environment
and from a `.env` file read out of the process working directory. Back
@@ -1709,7 +1737,7 @@ imports. The entry point is `cmd/webhooker/main.go`.
```
webhooker/
├── cmd/webhooker/
│ └── main.go # Entry point: sets globals, wires fx
│ └── main.go # Entry point: sets globals, locks DATA_DIR, wires fx
├── internal/
│ ├── config/
│ │ └── config.go # Configuration loading from environment variables
@@ -1730,6 +1758,8 @@ webhooker/
│ │ ├── retention.go # Retention reaper (per-webhook event expiry)
│ │ ├── testing.go # NewTestDatabase: wrapper for tests, no fx lifecycle
│ │ └── webhook_db_manager.go # Per-webhook DB lifecycle manager
│ ├── datadir/
│ │ └── lock.go # Exclusive advisory lock on DATA_DIR (one instance)
│ ├── globals/
│ │ └── globals.go # Build-time variables (appname, version, arch)
│ ├── gormlog/