Report or refuse each unusable file webhooker reads (closes #290) #460
+46
-15
@@ -79,7 +79,8 @@ directory, read once at startup before anything else looks at the
|
|||||||
environment.
|
environment.
|
||||||
|
|
||||||
The file is optional and having none is the normal case for a
|
The file is optional and having none is the normal case for a
|
||||||
deployment. A file that is there but cannot be parsed aborts startup
|
deployment. An empty file is the same as none: it has nothing in it to
|
||||||
|
apply. A file that is there but cannot be parsed aborts startup
|
||||||
with a message naming it, because a single malformed line makes none
|
with a message naming it, because a single malformed line makes none
|
||||||
of the file apply: every variable in it silently reverts to its
|
of the file apply: every variable in it silently reverts to its
|
||||||
default, which is exactly the failure [Invalid values abort
|
default, which is exactly the failure [Invalid values abort
|
||||||
@@ -564,11 +565,13 @@ its Argon2id hash. There is no second account and no forgot-password
|
|||||||
flow, so the banner and the reset command below are the only two ways
|
flow, so the banner and the reset command below are the only two ways
|
||||||
in.
|
in.
|
||||||
|
|
||||||
A start that finds no `webhooker.db` in `DATA_DIR` also logs
|
A start that finds no `webhooker.db` in `DATA_DIR`, or a zero-length
|
||||||
|
one (which SQLite opens as an empty database), also logs
|
||||||
`created a new, empty database` at `WARN`, with the file's path,
|
`created a new, empty database` at `WARN`, with the file's path,
|
||||||
shortly before the banner. On a deployment that has run before, that
|
shortly before the banner. On a deployment that has run before, that
|
||||||
line means `DATA_DIR` was empty, most often because its volume is not
|
line means `webhooker.db` was lost: either the file was missing, most
|
||||||
mounted.
|
often because the volume holding `DATA_DIR` is not mounted, or it was
|
||||||
|
zero-length, as a truncated copy leaves it.
|
||||||
|
|
||||||
#### Recovering a lost admin password
|
#### Recovering a lost admin password
|
||||||
|
|
||||||
@@ -612,7 +615,8 @@ What it will not do:
|
|||||||
the old password, so a reset underneath it would report a change the
|
the old password, so a reset underneath it would report a change the
|
||||||
service does not honour.
|
service does not honour.
|
||||||
- **Create anything.** A `DATA_DIR` that does not exist, or that holds
|
- **Create anything.** A `DATA_DIR` that does not exist, or that holds
|
||||||
no `webhooker.db`, is an error rather than a new empty deployment —
|
no `webhooker.db` or a zero-length one, is an error naming the path
|
||||||
|
rather than a new empty deployment —
|
||||||
a mistyped path must not be built out and then reported as a success.
|
a mistyped path must not be built out and then reported as a success.
|
||||||
- **Create an account.** A username that does not exist is an error.
|
- **Create an account.** A username that does not exist is an error.
|
||||||
`resetpw` changes an existing account's password and nothing else.
|
`resetpw` changes an existing account's password and nothing else.
|
||||||
@@ -993,6 +997,16 @@ its sidecars; a killed or crashed instance leaves them, and they must be
|
|||||||
carried with the `.db`. An archive the service has not opened since a
|
carried with the `.db`. An archive the service has not opened since a
|
||||||
crash keeps that crash's sidecars, even across a later clean stop.
|
crash keeps that crash's sidecars, even across a later clean stop.
|
||||||
|
|
||||||
|
A missing sidecar is therefore normal, and SQLite makes new ones, so a
|
||||||
|
`-wal` lost from a copy cannot be reported: the transactions it held
|
||||||
|
are simply gone. SQLite reads a `-wal` up to its first damaged frame,
|
||||||
|
as after a crash, and rebuilds a damaged `-shm`. A sidecar with the
|
||||||
|
wrong mode is set back to `0600` when its database is opened. A
|
||||||
|
directory in place of either is refused then, with an error naming it:
|
||||||
|
for `webhooker.db` the server and `webhooker resetpw` stop, and an event
|
||||||
|
or archive database fails as a damaged one does (see
|
||||||
|
[Database Architecture](#database-architecture)).
|
||||||
|
|
||||||
Configuration is **not** in `DATA_DIR` — it comes from the environment
|
Configuration is **not** in `DATA_DIR` — it comes from the environment
|
||||||
and from a `.env` file read out of the process working directory. Back
|
and from a `.env` file read out of the process working directory. Back
|
||||||
that up with your deployment config, separately.
|
that up with your deployment config, separately.
|
||||||
@@ -1076,13 +1090,19 @@ with any `-wal`/`-shm` beside it, or wait until there are none.
|
|||||||
1. Stop the service.
|
1. Stop the service.
|
||||||
|
|
||||||
2. Restore the **whole set together**: `webhooker.db` *and* every
|
2. Restore the **whole set together**: `webhooker.db` *and* every
|
||||||
`events-*.db` *and* every `archive-*.db`. A partial restore fails
|
`events-*.db` *and* every `archive-*.db`. A restore that leaves out
|
||||||
quietly rather than loudly. Every database is opened `mode=rwc`, so a
|
`webhooker.db` or an `events-*.db` is reported, not refused; one
|
||||||
missing `events-{uuid}.db` is **created empty** on first access
|
that leaves out an `archive-*.db` or a `-wal` (step 3) is not
|
||||||
instead of erroring — the webhook comes back with its configuration
|
reported at all. Every database is opened `mode=rwc`, so a
|
||||||
intact and its entire event history silently gone. Event databases
|
missing `events-{uuid}.db` is **created empty**: the webhook comes
|
||||||
restored without `webhooker.db` are simply orphaned; nothing
|
back with its configuration intact and its entire event history
|
||||||
references their UUIDs.
|
gone. The first start after the restore logs
|
||||||
|
`created a new, empty database` at `WARN` for each such file, with
|
||||||
|
its path, as it does for a missing `webhooker.db`. A missing
|
||||||
|
`archive-*.db` is recreated at its target's next delivery without a
|
||||||
|
warning, since moving one away is a supported workflow. Event
|
||||||
|
databases restored without `webhooker.db` are simply orphaned;
|
||||||
|
nothing references their UUIDs.
|
||||||
|
|
||||||
3. Carry any `*.db-wal` and `*.db-shm` files that are in the backup.
|
3. Carry any `*.db-wal` and `*.db-shm` files that are in the backup.
|
||||||
They are part of the database, and dropping a `-wal` silently
|
They are part of the database, and dropping a `-wal` silently
|
||||||
@@ -1945,10 +1965,19 @@ encryption key is generated and stored, and an `admin` user is created.
|
|||||||
the deliveries per target, kept through retention
|
the deliveries per target, kept through retention
|
||||||
|
|
||||||
Per-webhook databases are created automatically when a webhook is
|
Per-webhook databases are created automatically when a webhook is
|
||||||
created (and lazily on first access for webhooks that predate this
|
created. They are managed by the `WebhookDBManager` component, which
|
||||||
feature). They are managed by the `WebhookDBManager` component, which
|
|
||||||
handles connection pooling, lazy opening, migrations, and cleanup.
|
handles connection pooling, lazy opening, migrations, and cleanup.
|
||||||
|
|
||||||
|
A per-webhook database that is missing or zero-length later means its
|
||||||
|
webhook's events and pending deliveries are gone. The next time it is
|
||||||
|
opened, an empty one is created in its place, so the webhook keeps
|
||||||
|
receiving, and `created a new, empty database` is logged at `WARN` with
|
||||||
|
the file's path. Every webhook's database is opened when the service
|
||||||
|
starts, so this appears at the latest at the first start after the
|
||||||
|
file was lost. A file there that SQLite cannot open fails that
|
||||||
|
webhook alone, with an `ERROR` naming the webhook on every access and a
|
||||||
|
500 to its senders, so one damaged file does not stop the others.
|
||||||
|
|
||||||
This separation provides:
|
This separation provides:
|
||||||
|
|
||||||
- **Isolation** — a high-volume webhook won't cause lock contention or
|
- **Isolation** — a high-volume webhook won't cause lock contention or
|
||||||
@@ -2013,7 +2042,9 @@ After each write the archive handle is closed
|
|||||||
and reopened, debounced to at most once per second, so an operator can
|
and reopened, debounced to at most once per second, so an operator can
|
||||||
move the archive file away for offline archiving without stopping the
|
move the archive file away for offline archiving without stopping the
|
||||||
service; a moved or removed archive file is recreated automatically on
|
service; a moved or removed archive file is recreated automatically on
|
||||||
the next write. An optional `expiry` in the target's config JSON (e.g.
|
the next write. A zero-length archive file is written to as a new
|
||||||
|
archive: SQLite opens it as an empty database, so it holds nothing to
|
||||||
|
lose. An optional `expiry` in the target's config JSON (e.g.
|
||||||
`{"expiry":"720h"}`) is validated when the target is created — the
|
`{"expiry":"720h"}`) is validated when the target is created — the
|
||||||
default (unset or the literal `never`) keeps rows forever — and rows
|
default (unset or the literal `never`) keeps rows forever — and rows
|
||||||
older than the expiry are pruned each time the archive is (re)opened. An
|
older than the expiry are pruned each time the archive is (re)opened. An
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ import (
|
|||||||
"bytes"
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
@@ -119,3 +120,26 @@ func TestNewDatabase_IsLoggedWithItsPath(t *testing.T) {
|
|||||||
t, second, created, "an existing database is not new",
|
t, second, created, "an existing database is not new",
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestZeroLengthDatabase_IsLoggedAsNew covers what
|
||||||
|
// https://git.eeqj.de/sneak/webhooker/issues/290 found: SQLite opens a
|
||||||
|
// zero-length file as an empty database, so a start on one is a first
|
||||||
|
// start, and it must say so exactly as a start with no file does.
|
||||||
|
func TestZeroLengthDatabase_IsLoggedAsNew(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
path := filepath.Join(dir, database.MainDBFileName)
|
||||||
|
require.NoError(t, os.WriteFile(path, nil, database.SQLiteFilePerm))
|
||||||
|
|
||||||
|
var out bytes.Buffer
|
||||||
|
|
||||||
|
db, err := database.Open(dir, slog.New(slog.NewTextHandler(&out, nil)))
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.NoError(t, db.Close())
|
||||||
|
|
||||||
|
assert.Contains(
|
||||||
|
t, out.String(),
|
||||||
|
`level=WARN msg="created a new, empty database" path=`+path,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|||||||
@@ -8,7 +8,6 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"io/fs"
|
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
@@ -203,8 +202,7 @@ func (d *Database) connectTo(dataDir string) error {
|
|||||||
// Checked before opening, which creates the file. A DATA_DIR that
|
// Checked before opening, which creates the file. A DATA_DIR that
|
||||||
// is unexpectedly empty -- its volume not mounted, say -- looks
|
// is unexpectedly empty -- its volume not mounted, say -- looks
|
||||||
// exactly like a first start, so a new database is a warning.
|
// exactly like a first start, so a new database is a warning.
|
||||||
_, statErr := os.Stat(dbPath)
|
created := missingOrEmpty(dbPath)
|
||||||
created := errors.Is(statErr, fs.ErrNotExist)
|
|
||||||
|
|
||||||
// Opened through OpenSQLite so this handle carries the same WAL
|
// Opened through OpenSQLite so this handle carries the same WAL
|
||||||
// journaling, busy timeout, immediate-transaction locking, and pool
|
// journaling, busy timeout, immediate-transaction locking, and pool
|
||||||
@@ -213,13 +211,15 @@ func (d *Database) connectTo(dataDir string) error {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
d.log.Error(
|
d.log.Error(
|
||||||
"failed to open database",
|
"failed to open database",
|
||||||
|
"path", dbPath,
|
||||||
"error", err,
|
"error", err,
|
||||||
)
|
)
|
||||||
|
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
// Then use it with GORM
|
// Then use it with GORM. Its errors are SQLite's alone and name no
|
||||||
|
// file, so the path is added to them here.
|
||||||
db, err := gorm.Open(sqlite.Dialector{
|
db, err := gorm.Open(sqlite.Dialector{
|
||||||
Conn: sqlDB,
|
Conn: sqlDB,
|
||||||
}, &gorm.Config{
|
}, &gorm.Config{
|
||||||
@@ -229,10 +229,11 @@ func (d *Database) connectTo(dataDir string) error {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
d.log.Error(
|
d.log.Error(
|
||||||
"failed to connect to database",
|
"failed to connect to database",
|
||||||
|
"path", dbPath,
|
||||||
"error", err,
|
"error", err,
|
||||||
)
|
)
|
||||||
|
|
||||||
return err
|
return fmt.Errorf("connecting to %s: %w", dbPath, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
d.db = db
|
d.db = db
|
||||||
@@ -243,8 +244,12 @@ func (d *Database) connectTo(dataDir string) error {
|
|||||||
d.log.Info("connected to database", "path", dbPath)
|
d.log.Info("connected to database", "path", dbPath)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Run migrations
|
err = d.migrate()
|
||||||
return d.migrate()
|
if err != nil {
|
||||||
|
return fmt.Errorf("migrating %s: %w", dbPath, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (d *Database) migrate() error {
|
func (d *Database) migrate() error {
|
||||||
|
|||||||
@@ -1,9 +1,15 @@
|
|||||||
package database_test
|
package database_test
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
|
"log/slog"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
"go.uber.org/fx/fxtest"
|
"go.uber.org/fx/fxtest"
|
||||||
"sneak.berlin/go/webhooker/internal/config"
|
"sneak.berlin/go/webhooker/internal/config"
|
||||||
"sneak.berlin/go/webhooker/internal/database"
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
@@ -100,3 +106,22 @@ func TestDatabaseConnection(t *testing.T) {
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestOpen_UnreadableDatabaseIsNamed pins
|
||||||
|
// https://git.eeqj.de/sneak/webhooker/issues/459: when SQLite cannot
|
||||||
|
// read webhooker.db, the error that stops the server and `webhooker
|
||||||
|
// resetpw` names the file, not only SQLite's own message.
|
||||||
|
func TestOpen_UnreadableDatabaseIsNamed(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
path := filepath.Join(dir, database.MainDBFileName)
|
||||||
|
require.NoError(t, os.WriteFile(
|
||||||
|
path, bytes.Repeat([]byte("junk"), 1024), database.SQLiteFilePerm,
|
||||||
|
))
|
||||||
|
|
||||||
|
_, err := database.Open(dir, slog.New(slog.DiscardHandler))
|
||||||
|
require.Error(t, err)
|
||||||
|
assert.Contains(t, err.Error(), path)
|
||||||
|
assert.Contains(t, err.Error(), "file is not a database")
|
||||||
|
}
|
||||||
|
|||||||
@@ -184,8 +184,8 @@ func (r *RetentionReaper) sweep(ctx context.Context) {
|
|||||||
|
|
||||||
wh := webhooks[i]
|
wh := webhooks[i]
|
||||||
|
|
||||||
// Nothing to reap if the per-webhook database has never
|
// A missing database has nothing to reap. Restart recovery
|
||||||
// been created.
|
// reports a lost one (see WebhookDBManager.GetDB).
|
||||||
if !r.dbManager.DBExists(wh.ID) {
|
if !r.dbManager.DBExists(wh.ID) {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -182,6 +182,29 @@ func TestOpenSQLiteTightensFilesLeftWorldReadable(t *testing.T) {
|
|||||||
requireDatabaseSetOwnerOnly(t, path)
|
requireDatabaseSetOwnerOnly(t, path)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestOpenSQLiteRefusesADirectorySidecar covers a directory in place
|
||||||
|
// of -wal or -shm. Beside a -shm directory SQLite opens the database
|
||||||
|
// read-only without a word, and every write then fails naming no file,
|
||||||
|
// so the open must stop instead, naming the directory.
|
||||||
|
func TestOpenSQLiteRefusesADirectorySidecar(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
for _, suffix := range []string{"-wal", "-shm"} {
|
||||||
|
t.Run(suffix, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
path := filepath.Join(t.TempDir(), database.MainDBFileName)
|
||||||
|
require.NoError(t, os.Mkdir(path+suffix, 0o700))
|
||||||
|
|
||||||
|
_, err := database.OpenSQLite(
|
||||||
|
path, database.SQLiteModeCreate,
|
||||||
|
)
|
||||||
|
require.Error(t, err)
|
||||||
|
assert.Contains(t, err.Error(), path+suffix)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// TestOpenSQLiteExistingModeDoesNotCreateTheFile guards the mechanism
|
// TestOpenSQLiteExistingModeDoesNotCreateTheFile guards the mechanism
|
||||||
// the fix uses: OpenSQLite now creates the database file itself, and
|
// the fix uses: OpenSQLite now creates the database file itself, and
|
||||||
// must not do so for a caller that asked for an existing database. An
|
// must not do so for a caller that asked for an existing database. An
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ import (
|
|||||||
"io/fs"
|
"io/fs"
|
||||||
"net/url"
|
"net/url"
|
||||||
"os"
|
"os"
|
||||||
|
"syscall"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
_ "modernc.org/sqlite" // Pure Go SQLite driver
|
_ "modernc.org/sqlite" // Pure Go SQLite driver
|
||||||
@@ -93,7 +94,8 @@ const (
|
|||||||
const SQLiteFilePerm fs.FileMode = 0o600
|
const SQLiteFilePerm fs.FileMode = 0o600
|
||||||
|
|
||||||
// reserveSQLiteFile puts path at SQLiteFilePerm before the driver ever
|
// reserveSQLiteFile puts path at SQLiteFilePerm before the driver ever
|
||||||
// touches it, and tightens any sidecar already on disk.
|
// touches it, and tightens any sidecar already on disk. A directory in
|
||||||
|
// place of any of them is an error naming it.
|
||||||
//
|
//
|
||||||
// The mode has to be settled here rather than by a chmod after opening,
|
// The mode has to be settled here rather than by a chmod after opening,
|
||||||
// because SQLite picks it: robust_open substitutes
|
// because SQLite picks it: robust_open substitutes
|
||||||
@@ -143,7 +145,15 @@ func reserveSQLiteFile(path string, create bool) error {
|
|||||||
for _, p := range append(
|
for _, p := range append(
|
||||||
[]string{path}, sqliteSidecarPaths(path)...,
|
[]string{path}, sqliteSidecarPaths(path)...,
|
||||||
) {
|
) {
|
||||||
err := os.Chmod(p, SQLiteFilePerm)
|
// Chmod accepts a directory, and SQLite opens a database whose
|
||||||
|
// -shm is one read-only, without a word: every write then
|
||||||
|
// fails naming no file.
|
||||||
|
info, err := os.Stat(p)
|
||||||
|
if err == nil && info.IsDir() {
|
||||||
|
return fmt.Errorf("securing %s: %w", p, syscall.EISDIR)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = os.Chmod(p, SQLiteFilePerm)
|
||||||
if err != nil && !errors.Is(err, fs.ErrNotExist) {
|
if err != nil && !errors.Is(err, fs.ErrNotExist) {
|
||||||
return fmt.Errorf("securing %s: %w", p, err)
|
return fmt.Errorf("securing %s: %w", p, err)
|
||||||
}
|
}
|
||||||
@@ -152,6 +162,20 @@ func reserveSQLiteFile(path string, create bool) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// missingOrEmpty reports whether opening path in SQLiteModeCreate
|
||||||
|
// would start a new, empty database: the file is not there, or it is
|
||||||
|
// zero-length, which SQLite opens as an empty database. A file left at
|
||||||
|
// zero length by an interrupted first start or a truncated copy holds
|
||||||
|
// as little as a missing one, and must be reported the same way.
|
||||||
|
func missingOrEmpty(path string) bool {
|
||||||
|
info, err := os.Stat(path)
|
||||||
|
if errors.Is(err, fs.ErrNotExist) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
return err == nil && info.Size() == 0
|
||||||
|
}
|
||||||
|
|
||||||
// sqliteSidecarPaths returns the files SQLite maintains beside a
|
// sqliteSidecarPaths returns the files SQLite maintains beside a
|
||||||
// database under WAL. They carry the same rows as the database itself,
|
// database under WAL. They carry the same rows as the database itself,
|
||||||
// so a fix that tightens only the main file has fixed nothing.
|
// so a fix that tightens only the main file has fixed nothing.
|
||||||
|
|||||||
@@ -98,34 +98,37 @@ func NewWebhookDBManager(
|
|||||||
return m, nil
|
return m, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// GetDB returns the database connection for a webhook,
|
// GetDB returns the database connection for a webhook, opening it on
|
||||||
// creating the database file lazily if it doesn't exist.
|
// first use.
|
||||||
|
//
|
||||||
|
// The file is made by CreateDB when the webhook is created. One that is
|
||||||
|
// missing or zero-length here means the webhook's events and pending
|
||||||
|
// deliveries are gone: an empty database is created in its place so
|
||||||
|
// the webhook keeps receiving, and that is logged as a warning naming
|
||||||
|
// the file, as a new main database is.
|
||||||
func (m *WebhookDBManager) GetDB(
|
func (m *WebhookDBManager) GetDB(
|
||||||
webhookID string,
|
webhookID string,
|
||||||
) (*gorm.DB, error) {
|
) (*gorm.DB, error) {
|
||||||
// Fast path: already open
|
return m.getDB(webhookID, false)
|
||||||
if val, ok := m.dbs.Load(webhookID); ok {
|
}
|
||||||
return asGormDB(val, webhookID)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Slow path: open the database under the lock, looking in the
|
// GetDBIf is GetDB, done only when check reports true. check runs under
|
||||||
// cache again first. A caller that raced another one here then
|
// the lock DeleteDB holds while it removes the files, so a caller can
|
||||||
// waits for its handle instead of opening a second one.
|
// confirm the webhook still exists and open its database with no delete
|
||||||
|
// in between. The handle is nil when check reports false. check must
|
||||||
|
// not call the manager.
|
||||||
|
func (m *WebhookDBManager) GetDBIf(
|
||||||
|
webhookID string, check func() (bool, error),
|
||||||
|
) (*gorm.DB, error) {
|
||||||
m.mu.Lock()
|
m.mu.Lock()
|
||||||
defer m.mu.Unlock()
|
defer m.mu.Unlock()
|
||||||
|
|
||||||
if val, ok := m.dbs.Load(webhookID); ok {
|
ok, err := check()
|
||||||
return asGormDB(val, webhookID)
|
if err != nil || !ok {
|
||||||
}
|
|
||||||
|
|
||||||
db, err := m.openDB(webhookID)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
m.dbs.Store(webhookID, db)
|
return m.getDBLocked(webhookID, false)
|
||||||
|
|
||||||
return db, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// asGormDB returns a value read from the cache as the database
|
// asGormDB returns a value read from the cache as the database
|
||||||
@@ -143,12 +146,12 @@ func asGormDB(val any, webhookID string) (*gorm.DB, error) {
|
|||||||
return db, nil
|
return db, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// CreateDB explicitly creates a new per-webhook database file
|
// CreateDB creates a new webhook's database file and runs
|
||||||
// and runs migrations.
|
// migrations.
|
||||||
func (m *WebhookDBManager) CreateDB(
|
func (m *WebhookDBManager) CreateDB(
|
||||||
webhookID string,
|
webhookID string,
|
||||||
) error {
|
) error {
|
||||||
_, err := m.GetDB(webhookID)
|
_, err := m.getDB(webhookID, true)
|
||||||
|
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -266,6 +269,54 @@ func (m *WebhookDBManager) DBPath(
|
|||||||
return m.dbPath(webhookID)
|
return m.dbPath(webhookID)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// getDB is GetDB, and CreateDB when isNew is true: the webhook has just
|
||||||
|
// been created, so a missing file is expected rather than lost.
|
||||||
|
func (m *WebhookDBManager) getDB(
|
||||||
|
webhookID string, isNew bool,
|
||||||
|
) (*gorm.DB, error) {
|
||||||
|
// Fast path: already open
|
||||||
|
if val, ok := m.dbs.Load(webhookID); ok {
|
||||||
|
return asGormDB(val, webhookID)
|
||||||
|
}
|
||||||
|
|
||||||
|
m.mu.Lock()
|
||||||
|
defer m.mu.Unlock()
|
||||||
|
|
||||||
|
return m.getDBLocked(webhookID, isNew)
|
||||||
|
}
|
||||||
|
|
||||||
|
// getDBLocked is getDB's slow path, run with m.mu held. It looks in the
|
||||||
|
// cache again first: a caller that raced another one to the lock then
|
||||||
|
// gets its handle instead of opening a second one.
|
||||||
|
func (m *WebhookDBManager) getDBLocked(
|
||||||
|
webhookID string, isNew bool,
|
||||||
|
) (*gorm.DB, error) {
|
||||||
|
if val, ok := m.dbs.Load(webhookID); ok {
|
||||||
|
return asGormDB(val, webhookID)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Checked before opening, which creates the file. See GetDB.
|
||||||
|
path := m.dbPath(webhookID)
|
||||||
|
replaced := !isNew && missingOrEmpty(path)
|
||||||
|
|
||||||
|
db, err := m.openDB(webhookID)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
if replaced {
|
||||||
|
m.log.Warn(
|
||||||
|
"created a new, empty database",
|
||||||
|
"webhook_id", webhookID,
|
||||||
|
"path", path,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
m.dbs.Store(webhookID, db)
|
||||||
|
|
||||||
|
return db, nil
|
||||||
|
}
|
||||||
|
|
||||||
func (m *WebhookDBManager) dbPath(
|
func (m *WebhookDBManager) dbPath(
|
||||||
webhookID string,
|
webhookID string,
|
||||||
) string {
|
) string {
|
||||||
|
|||||||
@@ -289,6 +289,75 @@ func TestWebhookDBManager_LazyCreation(t *testing.T) {
|
|||||||
assert.True(t, mgr.DBExists(webhookID))
|
assert.True(t, mgr.DBExists(webhookID))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A webhook's database is made by CreateDB along with the webhook. One
|
||||||
|
// that GetDB finds missing or zero-length has lost the webhook's events
|
||||||
|
// and pending deliveries, so the empty database made in its place is
|
||||||
|
// logged as a warning naming the file
|
||||||
|
// (https://git.eeqj.de/sneak/webhooker/issues/290). CreateDB, and
|
||||||
|
// reopening a database that is there, log no such warning.
|
||||||
|
func TestWebhookDBManager_LostDatabaseIsLogged(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const created = `level=WARN msg="created a new, empty database"`
|
||||||
|
|
||||||
|
open := func(
|
||||||
|
t *testing.T, prepare func(*database.WebhookDBManager, string),
|
||||||
|
) (string, string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
var logs bytes.Buffer
|
||||||
|
|
||||||
|
mgr := database.NewTestWebhookDBManagerWithLogger(
|
||||||
|
t.TempDir(),
|
||||||
|
slog.New(slog.NewTextHandler(&logs, nil)),
|
||||||
|
)
|
||||||
|
|
||||||
|
webhookID := uuid.New().String()
|
||||||
|
prepare(mgr, webhookID)
|
||||||
|
|
||||||
|
_, err := mgr.GetDB(webhookID)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.NoError(t, mgr.CloseAll())
|
||||||
|
|
||||||
|
return logs.String(),
|
||||||
|
" webhook_id=" + webhookID + " path=" + mgr.DBPath(webhookID)
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Run("missing", func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
logs, fields := open(
|
||||||
|
t, func(*database.WebhookDBManager, string) {},
|
||||||
|
)
|
||||||
|
assert.Contains(t, logs, created+fields)
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("zero-length", func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
logs, fields := open(
|
||||||
|
t, func(mgr *database.WebhookDBManager, webhookID string) {
|
||||||
|
require.NoError(t, os.WriteFile(
|
||||||
|
mgr.DBPath(webhookID), nil, database.SQLiteFilePerm,
|
||||||
|
))
|
||||||
|
},
|
||||||
|
)
|
||||||
|
assert.Contains(t, logs, created+fields)
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("created with the webhook, then reopened", func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
logs, _ := open(
|
||||||
|
t, func(mgr *database.WebhookDBManager, webhookID string) {
|
||||||
|
require.NoError(t, mgr.CreateDB(webhookID))
|
||||||
|
require.NoError(t, mgr.CloseAll())
|
||||||
|
},
|
||||||
|
)
|
||||||
|
assert.NotContains(t, logs, created)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
func TestWebhookDBManager_DeliveryWorkflow(t *testing.T) {
|
func TestWebhookDBManager_DeliveryWorkflow(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
@@ -699,10 +699,9 @@ func (e *Engine) recoverInFlight(ctx context.Context) {
|
|||||||
default:
|
default:
|
||||||
}
|
}
|
||||||
|
|
||||||
if !e.dbManager.DBExists(webhookID) {
|
// Opened even when its file is missing, so that a lost
|
||||||
continue
|
// database is reported at start, not when the webhook next
|
||||||
}
|
// receives an event, which for a quiet webhook may be never.
|
||||||
|
|
||||||
e.recoverWebhookDeliveries(ctx, webhookID)
|
e.recoverWebhookDeliveries(ctx, webhookID)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -710,7 +709,24 @@ func (e *Engine) recoverInFlight(ctx context.Context) {
|
|||||||
func (e *Engine) recoverWebhookDeliveries(
|
func (e *Engine) recoverWebhookDeliveries(
|
||||||
ctx context.Context, webhookID string,
|
ctx context.Context, webhookID string,
|
||||||
) {
|
) {
|
||||||
webhookDB, err := e.dbManager.GetDB(webhookID)
|
// The web interface is already serving, so the webhook may have
|
||||||
|
// been deleted since the list was read. Opening its database then
|
||||||
|
// would create the file again after the delete removed it.
|
||||||
|
stillExists := func() (bool, error) {
|
||||||
|
var count int64
|
||||||
|
|
||||||
|
err := e.database.DB().
|
||||||
|
Model(&database.Webhook{}).
|
||||||
|
Where("id = ?", webhookID).
|
||||||
|
Count(&count).Error
|
||||||
|
if err != nil {
|
||||||
|
return false, fmt.Errorf("confirming webhook exists: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return count > 0, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
webhookDB, err := e.dbManager.GetDBIf(webhookID, stillExists)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
e.log.Error(
|
e.log.Error(
|
||||||
"failed to get webhook database for recovery",
|
"failed to get webhook database for recovery",
|
||||||
@@ -721,6 +737,10 @@ func (e *Engine) recoverWebhookDeliveries(
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if webhookDB == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
e.recoverPendingDeliveries(
|
e.recoverPendingDeliveries(
|
||||||
ctx, webhookDB, webhookID,
|
ctx, webhookDB, webhookID,
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
package delivery_test
|
package delivery_test
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
@@ -1136,6 +1137,85 @@ func TestRecoverInFlight_WithPendingDeliveries(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestRecoverInFlight_ReportsAMissingWebhookDatabase covers a webhook
|
||||||
|
// whose database file is gone, after a partial restore say. Restart
|
||||||
|
// recovery opens every webhook's database, so the empty one made in its
|
||||||
|
// place is reported at start, naming the file
|
||||||
|
// (https://git.eeqj.de/sneak/webhooker/issues/290).
|
||||||
|
func TestRecoverInFlight_ReportsAMissingWebhookDatabase(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
mainDB := iMainDB(t)
|
||||||
|
webhookID := uuid.New().String()
|
||||||
|
iCreateWebhook(t, mainDB, webhookID, "lost-database")
|
||||||
|
|
||||||
|
var logs bytes.Buffer
|
||||||
|
|
||||||
|
dbMgr := database.NewTestWebhookDBManagerWithLogger(
|
||||||
|
t.TempDir(), slog.New(slog.NewTextHandler(&logs, nil)),
|
||||||
|
)
|
||||||
|
t.Cleanup(func() { _ = dbMgr.CloseAll() })
|
||||||
|
|
||||||
|
engine := delivery.NewTestEngineWithDB(
|
||||||
|
database.NewTestDatabase(mainDB), dbMgr,
|
||||||
|
slog.New(slog.DiscardHandler),
|
||||||
|
&http.Client{Timeout: 5 * time.Second}, 1,
|
||||||
|
)
|
||||||
|
|
||||||
|
engine.ExportRecoverInFlight(context.Background())
|
||||||
|
|
||||||
|
assert.Contains(
|
||||||
|
t, logs.String(),
|
||||||
|
`level=WARN msg="created a new, empty database" webhook_id=`+
|
||||||
|
webhookID+" path="+dbMgr.DBPath(webhookID),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRecoverInFlight_SkipsAWebhookDeletedAfterTheListIsRead covers a
|
||||||
|
// webhook deleted from the web interface while restart recovery runs.
|
||||||
|
// Its database file is gone, and recovery must not create it again.
|
||||||
|
func TestRecoverInFlight_SkipsAWebhookDeletedAfterTheListIsRead(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
mainDB := iMainDB(t)
|
||||||
|
webhookID := uuid.New().String()
|
||||||
|
iCreateWebhook(t, mainDB, webhookID, "deleted-during-recovery")
|
||||||
|
|
||||||
|
// The first query to return is recovery's read of the list of
|
||||||
|
// webhooks. Deleting the webhook right after it puts the delete
|
||||||
|
// between that read and the opening of the webhook's database.
|
||||||
|
deleted := false
|
||||||
|
|
||||||
|
require.NoError(t, mainDB.Callback().Query().After("gorm:query").
|
||||||
|
Register("delete-after-list", func(*gorm.DB) {
|
||||||
|
if deleted {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
deleted = true
|
||||||
|
|
||||||
|
require.NoError(t, mainDB.Delete(
|
||||||
|
&database.Webhook{}, "id = ?", webhookID,
|
||||||
|
).Error)
|
||||||
|
}))
|
||||||
|
|
||||||
|
dbMgr := database.NewTestWebhookDBManager(t.TempDir())
|
||||||
|
t.Cleanup(func() { _ = dbMgr.CloseAll() })
|
||||||
|
|
||||||
|
engine := delivery.NewTestEngineWithDB(
|
||||||
|
database.NewTestDatabase(mainDB), dbMgr,
|
||||||
|
slog.New(slog.DiscardHandler),
|
||||||
|
&http.Client{Timeout: 5 * time.Second}, 1,
|
||||||
|
)
|
||||||
|
|
||||||
|
engine.ExportRecoverInFlight(context.Background())
|
||||||
|
|
||||||
|
require.True(t, deleted)
|
||||||
|
assert.False(t, dbMgr.DBExists(webhookID))
|
||||||
|
}
|
||||||
|
|
||||||
// --- HTTP Config with custom headers ---
|
// --- HTTP Config with custom headers ---
|
||||||
|
|
||||||
func TestDeliverHTTP_CustomTargetHeaders(t *testing.T) {
|
func TestDeliverHTTP_CustomTargetHeaders(t *testing.T) {
|
||||||
|
|||||||
@@ -573,6 +573,8 @@ func TestRecoverPending_TargetDeleted(t *testing.T) {
|
|||||||
|
|
||||||
s := newISetup(t)
|
s := newISetup(t)
|
||||||
|
|
||||||
|
iCreateWebhook(t, s.MainDB, s.WebhookID, "pending-recovery")
|
||||||
|
|
||||||
deliveryID := tSeedDeletedTarget(
|
deliveryID := tSeedDeletedTarget(
|
||||||
t, s, "gone-while-pending", "http://example.com/hook",
|
t, s, "gone-while-pending", "http://example.com/hook",
|
||||||
database.DeliveryStatusPending,
|
database.DeliveryStatusPending,
|
||||||
@@ -612,6 +614,8 @@ func TestRecoverPending_TargetDeleted_LeavesAnOwnedDeliveryAlone(
|
|||||||
|
|
||||||
s := newISetup(t)
|
s := newISetup(t)
|
||||||
|
|
||||||
|
iCreateWebhook(t, s.MainDB, s.WebhookID, "owned-recovery")
|
||||||
|
|
||||||
deliveryID := tSeedDeletedTarget(
|
deliveryID := tSeedDeletedTarget(
|
||||||
t, s, "gone-but-owned", "http://example.com/hook",
|
t, s, "gone-but-owned", "http://example.com/hook",
|
||||||
database.DeliveryStatusPending,
|
database.DeliveryStatusPending,
|
||||||
|
|||||||
@@ -308,7 +308,7 @@ func checkDataDir(dir string) error {
|
|||||||
|
|
||||||
dbPath := filepath.Join(dir, database.MainDBFileName)
|
dbPath := filepath.Join(dir, database.MainDBFileName)
|
||||||
|
|
||||||
_, err = os.Stat(dbPath)
|
dbInfo, err := os.Stat(dbPath)
|
||||||
|
|
||||||
switch {
|
switch {
|
||||||
case errors.Is(err, fs.ErrNotExist):
|
case errors.Is(err, fs.ErrNotExist):
|
||||||
@@ -319,6 +319,15 @@ func checkDataDir(dir string) error {
|
|||||||
)
|
)
|
||||||
case err != nil:
|
case err != nil:
|
||||||
return fmt.Errorf("checking %s: %w", dbPath, err)
|
return fmt.Errorf("checking %s: %w", dbPath, err)
|
||||||
|
case dbInfo.Size() == 0:
|
||||||
|
// SQLite opens a zero-length file as an empty database, so
|
||||||
|
// it holds no deployment either, and opening it would write
|
||||||
|
// an empty schema into it.
|
||||||
|
return fmt.Errorf(
|
||||||
|
"%w: %s is zero-length. The admin account is created by "+
|
||||||
|
"the first server start",
|
||||||
|
ErrNoDatabase, dbPath,
|
||||||
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
|
|||||||
@@ -377,6 +377,33 @@ func TestMissingDatabaseCreatesNothing(t *testing.T) {
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestZeroLengthDatabaseCreatesNothing covers a webhooker.db left at
|
||||||
|
// zero length, as a truncated copy leaves it. SQLite would open it as
|
||||||
|
// an empty database, so it is refused like a missing one and left as
|
||||||
|
// it is.
|
||||||
|
func TestZeroLengthDatabaseCreatesNothing(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
t.Setenv("DATA_DIR", dir)
|
||||||
|
|
||||||
|
dbPath := filepath.Join(dir, database.MainDBFileName)
|
||||||
|
require.NoError(
|
||||||
|
t, os.WriteFile(dbPath, nil, database.SQLiteFilePerm),
|
||||||
|
)
|
||||||
|
|
||||||
|
code, _, stderr := run(t, newPassword+"\n", operatorUser)
|
||||||
|
|
||||||
|
require.Equal(t, exitFailure, code)
|
||||||
|
assert.Contains(t, stderr, dbPath)
|
||||||
|
|
||||||
|
entries, err := os.ReadDir(dir)
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Len(t, entries, 1, "nothing may be created beside it")
|
||||||
|
|
||||||
|
info, err := os.Stat(dbPath)
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Zero(t, info.Size(), "nothing may be written into it")
|
||||||
|
}
|
||||||
|
|
||||||
// TestUnknownUserFails states the decision: resetpw changes an
|
// TestUnknownUserFails states the decision: resetpw changes an
|
||||||
// existing account's password and never creates an account. A typo in
|
// existing account's password and never creates an account. A typo in
|
||||||
// the username must say so rather than quietly adding a second user.
|
// the username must say so rather than quietly adding a second user.
|
||||||
|
|||||||
Atsaukties uz šo jaunā problēmā
Block a user