Lock DATA_DIR against a second instance (closes #201)
All checks were successful
check / check (push) Successful in 6m35s
All checks were successful
check / check (push) Successful in 6m35s
Nothing stopped two processes opening the same DATA_DIR. Both open the
same per-webhook databases, both run delivery recovery over the same
rows, and both deliver: every pending delivery reaches the destination
twice, from nothing worse than an overlapping deploy.
The entry point now takes an exclusive advisory flock(2) on
{DATA_DIR}/webhooker.lock before anything opens a database, and holds it
for the process lifetime. A second process pointed at the same directory
prints a message naming that directory and exits 1. The lock is the
kernel's, not the file's, so a process killed with SIGKILL leaves a lock
file that blocks nothing -- which is what a pidfile would get wrong. The
file is never unlinked: doing so would let the next process lock a fresh
inode while a third still held the old one.
Acquisition lives in internal/datadir rather than in the server's fx
graph, so any entry point touching DATA_DIR takes it the same way, and
ErrLocked lets a caller tell a live deployment from any other failure.
config.DataDir() resolves DATA_DIR once, for both the lock and Config,
so the two cannot disagree.
Regression coverage: a real second process is refused, and a restart
after kill -9 succeeds with the stale lock file in place.
github.com/gofrs/flock carries the lock; its own module minimums pull
testify to v1.11.1 and golang.org/x/sys to v0.37.0.
This commit is contained in:
40
README.md
40
README.md
@@ -115,6 +115,32 @@ TTY detection, and security headers are always applied.
|
||||
| `RECEIVER_RATE_LIMIT` | Receiver requests/minute per IP per entrypoint (10x that per IP across the route) | `120` |
|
||||
| `TRUSTED_PROXIES` | CIDRs whose forwarded headers are trusted (unset: all clients behind a proxy share one rate-limit bucket; a correct login password is never throttled either way) | `""` (none) |
|
||||
|
||||
#### Single-instance lock
|
||||
|
||||
Exactly one webhooker process may use a `DATA_DIR` at a time. Two
|
||||
processes sharing one open the same databases and each run delivery
|
||||
recovery over the same rows, so every pending delivery goes out twice —
|
||||
duplicate delivery to your endpoints, from nothing worse than an
|
||||
overlapping deploy or a double start.
|
||||
|
||||
At startup, before anything opens a database, the process takes an
|
||||
exclusive advisory lock (`flock(2)`) on `{DATA_DIR}/webhooker.lock` and
|
||||
holds it for its lifetime. A second process pointed at the same
|
||||
directory prints a message naming it and exits non-zero:
|
||||
|
||||
```
|
||||
webhooker: data directory is already in use by another instance: /var/lib/webhooker (/var/lib/webhooker/webhooker.lock). Only one webhooker may use a data directory: two both run delivery recovery over the same rows and both deliver
|
||||
```
|
||||
|
||||
The lock is the kernel's, not the file's: it is released when the
|
||||
process exits, including `kill -9`, so a leftover `webhooker.lock`
|
||||
never blocks a restart and must not be deleted by hand. The file is
|
||||
also left in place on a clean shutdown, deliberately — unlinking it
|
||||
would let the next process lock a fresh inode while a third still held
|
||||
the old one.
|
||||
|
||||
To run two webhookers on one host, give each its own `DATA_DIR`.
|
||||
|
||||
#### Trusted proxies
|
||||
|
||||
`TRUSTED_PROXIES` is a comma-separated list of CIDR blocks (a bare
|
||||
@@ -283,10 +309,12 @@ is both the simplest and the only complete rule:
|
||||
|
||||
`{webhook_uuid}` is the webhook's UUID primary key in its canonical
|
||||
36-character hyphenated form, so a real filename looks like
|
||||
`events-3f2a1c9e-....db`. Nothing else is written to `DATA_DIR`, and no
|
||||
`-wal` or `-shm` files are produced (see below); a transient
|
||||
`{name}.db-journal` may exist beside a database while a write is in
|
||||
flight and is not part of the backup set.
|
||||
`events-3f2a1c9e-....db`. The only other file is `webhooker.lock`, the
|
||||
always-empty [single-instance lock](#single-instance-lock); it holds no
|
||||
state and is not part of the backup set — a copied one is stale and
|
||||
blocks nothing. No `-wal` or `-shm` files are produced (see below); a
|
||||
transient `{name}.db-journal` may exist beside a database while a write
|
||||
is in flight and is not part of the backup set either.
|
||||
|
||||
Configuration is **not** in `DATA_DIR` — it comes from the environment
|
||||
and from a `.env` file read out of the process working directory. Back
|
||||
@@ -1709,7 +1737,7 @@ imports. The entry point is `cmd/webhooker/main.go`.
|
||||
```
|
||||
webhooker/
|
||||
├── cmd/webhooker/
|
||||
│ └── main.go # Entry point: sets globals, wires fx
|
||||
│ └── main.go # Entry point: sets globals, locks DATA_DIR, wires fx
|
||||
├── internal/
|
||||
│ ├── config/
|
||||
│ │ └── config.go # Configuration loading from environment variables
|
||||
@@ -1730,6 +1758,8 @@ webhooker/
|
||||
│ │ ├── retention.go # Retention reaper (per-webhook event expiry)
|
||||
│ │ ├── testing.go # NewTestDatabase: wrapper for tests, no fx lifecycle
|
||||
│ │ └── webhook_db_manager.go # Per-webhook DB lifecycle manager
|
||||
│ ├── datadir/
|
||||
│ │ └── lock.go # Exclusive advisory lock on DATA_DIR (one instance)
|
||||
│ ├── globals/
|
||||
│ │ └── globals.go # Build-time variables (appname, version, arch)
|
||||
│ ├── gormlog/
|
||||
|
||||
Reference in New Issue
Block a user