All checks were successful
check / check (push) Successful in 3m37s
Nothing stopped two processes opening the same DATA_DIR. Both open the
same per-webhook databases, both run delivery recovery over the same
rows, and both deliver: every pending delivery reaches the destination
twice, from nothing worse than an overlapping deploy.
The entry point now takes an exclusive advisory flock(2) on
{DATA_DIR}/webhooker.lock before anything opens a database, and holds it
for the process lifetime. A second process pointed at the same directory
prints a message naming that directory and exits 1. The lock is the
kernel's, not the file's, so a process killed with SIGKILL leaves a lock
file that blocks nothing -- which is what a pidfile would get wrong. The
file is never unlinked: doing so would let the next process lock a fresh
inode while a third still held the old one.
Acquisition lives in internal/datadir rather than in the server's fx
graph, so any entry point touching DATA_DIR takes it the same way, and
ErrLocked lets a caller tell a live deployment from any other failure.
config.DataDir() resolves DATA_DIR once, for both the lock and Config,
so the two cannot disagree.
Regression coverage: a real second process is refused, and a restart
after kill -9 succeeds with the stale lock file in place.
github.com/gofrs/flock carries the lock; its own module minimums pull
testify to v1.11.1 and golang.org/x/sys to v0.37.0.
127 lines
3.7 KiB
Go
127 lines
3.7 KiB
Go
// Package datadir guards exclusive access to the directory holding
|
|
// every SQLite database webhooker writes.
|
|
//
|
|
// Two processes sharing a DATA_DIR each open the same per-webhook
|
|
// event databases and each run delivery recovery over the same rows,
|
|
// so every pending delivery goes out twice. SQLite's own locking does
|
|
// not prevent that: both writers are serialised correctly and both
|
|
// deliver. The only thing that prevents it is refusing to be the
|
|
// second process.
|
|
//
|
|
// The lock lives here rather than in the server's fx graph so that any
|
|
// entry point which touches DATA_DIR — the server, or a CLI
|
|
// subcommand that must not operate on a live deployment's data — takes
|
|
// it the same way.
|
|
package datadir
|
|
|
|
import (
|
|
"errors"
|
|
"fmt"
|
|
"os"
|
|
"path/filepath"
|
|
|
|
"github.com/gofrs/flock"
|
|
)
|
|
|
|
// LockFileName is the advisory lock file created inside DATA_DIR. Its
|
|
// contents are never read: the lock is the flock(2) held on the open
|
|
// descriptor, not the file's existence, so a leftover file from a
|
|
// process that was killed with SIGKILL blocks nothing.
|
|
const LockFileName = "webhooker.lock"
|
|
|
|
// dirPerm is the mode Acquire creates DATA_DIR with. It matches what
|
|
// internal/database uses, since whichever runs first creates it.
|
|
const dirPerm = 0o750
|
|
|
|
// ErrLocked reports that another live process holds the data
|
|
// directory. Callers that need to know whether a deployment is running
|
|
// — rather than merely failing to start — test for this with
|
|
// errors.Is.
|
|
var ErrLocked = errors.New(
|
|
"data directory is already in use by another instance",
|
|
)
|
|
|
|
// ErrNoDir reports that Acquire was given an empty directory.
|
|
var ErrNoDir = errors.New("no data directory given")
|
|
|
|
// Lock is a held exclusive advisory lock on a data directory. It is
|
|
// valid only while the process that took it lives: the kernel drops it
|
|
// when the descriptor closes, whether that is Release, a normal exit,
|
|
// or a SIGKILL.
|
|
type Lock struct {
|
|
dir string
|
|
file *flock.Flock
|
|
}
|
|
|
|
// Acquire takes the exclusive advisory lock on dir, creating dir if it
|
|
// does not exist. It never waits: if another process holds the lock it
|
|
// returns an error wrapping ErrLocked and naming dir.
|
|
//
|
|
// The returned Lock must be held for as long as the caller intends to
|
|
// use dir.
|
|
func Acquire(dir string) (*Lock, error) {
|
|
if dir == "" {
|
|
return nil, ErrNoDir
|
|
}
|
|
|
|
err := os.MkdirAll(dir, dirPerm)
|
|
if err != nil {
|
|
return nil, fmt.Errorf(
|
|
"creating data directory %s: %w", dir, err,
|
|
)
|
|
}
|
|
|
|
path := filepath.Join(dir, LockFileName)
|
|
fl := flock.New(path)
|
|
|
|
held, err := fl.TryLock()
|
|
if err != nil {
|
|
return nil, fmt.Errorf(
|
|
"locking data directory %s: %w", dir, err,
|
|
)
|
|
}
|
|
|
|
if !held {
|
|
// A no-op on flock v0.13.0, which closes its own descriptor on
|
|
// a failed TryLock; kept so no version can leak one.
|
|
_ = fl.Close()
|
|
|
|
return nil, fmt.Errorf(
|
|
"%w: %s (%s). Only one webhooker may use a data "+
|
|
"directory: two both run delivery recovery over the "+
|
|
"same rows and both deliver",
|
|
ErrLocked, dir, path,
|
|
)
|
|
}
|
|
|
|
return &Lock{dir: dir, file: fl}, nil
|
|
}
|
|
|
|
// Dir returns the locked directory.
|
|
func (l *Lock) Dir() string {
|
|
return l.dir
|
|
}
|
|
|
|
// Path returns the lock file backing the lock.
|
|
func (l *Lock) Path() string {
|
|
return l.file.Path()
|
|
}
|
|
|
|
// Release drops the lock and closes the descriptor. It is safe to call
|
|
// more than once.
|
|
//
|
|
// The lock file is deliberately left on disk. Unlinking it would let
|
|
// the next process create and lock a fresh inode while a third still
|
|
// holds the old one, which is the one outcome this package exists to
|
|
// prevent.
|
|
func (l *Lock) Release() error {
|
|
err := l.file.Unlock()
|
|
if err != nil {
|
|
return fmt.Errorf(
|
|
"releasing lock on data directory %s: %w", l.dir, err,
|
|
)
|
|
}
|
|
|
|
return nil
|
|
}
|