Lock DATA_DIR against a second instance (closes #201)

Nothing stopped two processes opening the same DATA_DIR. Both open the
same per-webhook databases, both run delivery recovery over the same
rows, and both deliver: every pending delivery reaches the destination
twice, from nothing worse than an overlapping deploy.

The entry point now takes an exclusive advisory flock(2) on
{DATA_DIR}/webhooker.lock before anything opens a database, and holds it
for the process lifetime. A second process pointed at the same directory
prints a message naming that directory and exits 1. The lock is the
kernel's, not the file's, so a process killed with SIGKILL leaves a lock
file that blocks nothing -- which is what a pidfile would get wrong. The
file is never unlinked: doing so would let the next process lock a fresh
inode while a third still held the old one.

Acquisition lives in internal/datadir rather than in the server's fx
graph, so any entry point touching DATA_DIR takes it the same way, and
ErrLocked lets a caller tell a live deployment from any other failure.
config.DataDir() resolves DATA_DIR once, for both the lock and Config,
so the two cannot disagree.

Regression coverage: a real second process is refused, and a restart
after kill -9 succeeds with the stale lock file in place.

github.com/gofrs/flock carries the lock; its own module minimums pull
testify to v1.11.1 and golang.org/x/sys to v0.37.0.

cmd/webhooker/main.go

@@ -2,11 +2,15 @@
 package main
 
 import (
+	"fmt"
+	"io"
+	"os"
 	"time"
 
 	"go.uber.org/fx"
 	"sneak.berlin/go/webhooker/internal/config"
 	"sneak.berlin/go/webhooker/internal/database"
+	"sneak.berlin/go/webhooker/internal/datadir"
 	"sneak.berlin/go/webhooker/internal/delivery"
 	"sneak.berlin/go/webhooker/internal/globals"
 	"sneak.berlin/go/webhooker/internal/handlers"
@@ -56,7 +60,32 @@ func main() {
 	globals.Appname = appname
 	globals.Version = version
 
+	os.Exit(run(os.Stderr))
+}
+
+// run takes the exclusive DATA_DIR lock, then runs the application
+// under it, and returns the process exit status.
+//
+// The lock is taken here rather than inside the fx graph because it has
+// to be held before anything opens a database, and because a refusal
+// has to reach the operator as a plain line on standard error rather
+// than as one entry in an fx failure dump. It is released by the defer
+// on a clean shutdown, and by the kernel closing the descriptor on any
+// other exit — including the one fx performs itself when a start or
+// stop hook fails, which skips deferred calls.
+func run(stderr io.Writer) int {
+	lock, err := datadir.Acquire(config.DataDir())
+	if err != nil {
+		_, _ = fmt.Fprintf(stderr, "%s: %v\n", appname, err)
+
+		return 1
+	}
+
+	defer func() { _ = lock.Release() }()
+
 	newApp().Run()
+
+	return 0
 }
 
 // newApp builds the application graph. It is separate from main so

cmd/webhooker/main_test.go

@@ -1,10 +1,13 @@
 package main
 
 import (
+	"bytes"
 	"testing"
 	"time"
 
+	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
+	"sneak.berlin/go/webhooker/internal/datadir"
 	"sneak.berlin/go/webhooker/internal/server"
 )
 
@@ -33,6 +36,38 @@ func TestNewApp_StopTimeout(t *testing.T) {
 	require.Less(t, got, dockerStopGrace)
 }
 
+// TestRunRefusesLockedDataDir pins what an operator's second start
+// does. The entry point must refuse before it builds the fx graph —
+// nothing may open a database in a DATA_DIR another process holds —
+// and must exit non-zero with a message naming the directory rather
+// than starting a second delivery engine over the same rows.
+//
+// flock(2) locks descriptors independently, so holding the lock here
+// is the same denial a separate process gets; internal/datadir pins
+// that property and covers the real two-process case.
+func TestRunRefusesLockedDataDir(t *testing.T) {
+	dir := t.TempDir()
+	t.Setenv("DATA_DIR", dir)
+
+	lock, err := datadir.Acquire(dir)
+	require.NoError(t, err)
+
+	defer func() { _ = lock.Release() }()
+
+	var stderr bytes.Buffer
+
+	code := run(&stderr)
+
+	require.Equal(
+		t, 1, code, "a second instance must exit non-zero",
+	)
+	assert.Contains(
+		t, stderr.String(), dir,
+		"the refusal must name the directory",
+	)
+	assert.Contains(t, stderr.String(), "another instance")
+}
+
 // tailHeadroom is the slack the fx stop budget must keep beyond the
 // server stop hook. The hooks that run after the server — the
 // delivery engine, the healthcheck, the webhook DB manager and the