Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
49def467b1 | ||
|
|
12e6c3bf3e | ||
|
|
2dc6f1f077 | ||
|
|
237f131367 |
@@ -44,9 +44,9 @@ make bootstrap
|
|||||||
# Run all checks (test, lint, format check)
|
# Run all checks (test, lint, format check)
|
||||||
make check
|
make check
|
||||||
|
|
||||||
# Run in development mode. DATA_DIR defaults to /var/lib/webhooker in
|
# Run the server from the clone. DATA_DIR defaults to
|
||||||
# every environment, so set it (in .env or the shell) to a writable
|
# /var/lib/webhooker in every environment, so set it (in .env or the
|
||||||
# directory when running from a clone.
|
# shell) to a writable directory.
|
||||||
DATA_DIR=./data make dev
|
DATA_DIR=./data make dev
|
||||||
|
|
||||||
# Build Docker image
|
# Build Docker image
|
||||||
@@ -92,7 +92,8 @@ them at once. A variable already present in the real environment wins
|
|||||||
over the file's value for the same name.
|
over the file's value for the same name.
|
||||||
|
|
||||||
The environment is selected by setting `WEBHOOKER_ENVIRONMENT` to `dev`
|
The environment is selected by setting `WEBHOOKER_ENVIRONMENT` to `dev`
|
||||||
or `prod` (default: `dev`). The setting controls exactly one behavior:
|
or `prod` (default: `prod`; `dev` must be set explicitly). The setting
|
||||||
|
controls exactly one behavior:
|
||||||
|
|
||||||
| Behavior | `dev` | `prod` |
|
| Behavior | `dev` | `prod` |
|
||||||
| -------- | ----------------------- | ---------------- |
|
| -------- | ----------------------- | ---------------- |
|
||||||
@@ -134,7 +135,7 @@ TTY detection, and security headers are always applied.
|
|||||||
|
|
||||||
| Variable | Description | Default |
|
| Variable | Description | Default |
|
||||||
| ----------------------- | ----------------------------------- | -------- |
|
| ----------------------- | ----------------------------------- | -------- |
|
||||||
| `WEBHOOKER_ENVIRONMENT` | `dev` or `prod` | `dev` |
|
| `WEBHOOKER_ENVIRONMENT` | `dev` or `prod` | `prod` |
|
||||||
| `PORT` | HTTP listen port | `8080` |
|
| `PORT` | HTTP listen port | `8080` |
|
||||||
| `BIND_ADDRESS` | IP address the HTTP listener binds. Loopback by default, so the cleartext listener is not published on every interface. The Docker image ships `0.0.0.0` instead. See [Bind address](#bind-address) | `127.0.0.1` (image: `0.0.0.0`) |
|
| `BIND_ADDRESS` | IP address the HTTP listener binds. Loopback by default, so the cleartext listener is not published on every interface. The Docker image ships `0.0.0.0` instead. See [Bind address](#bind-address) | `127.0.0.1` (image: `0.0.0.0`) |
|
||||||
| `DATA_DIR` | Directory for all SQLite databases | `/var/lib/webhooker` |
|
| `DATA_DIR` | Directory for all SQLite databases | `/var/lib/webhooker` |
|
||||||
@@ -399,10 +400,9 @@ the bucket is. See [Rate Limiting](#rate-limiting).
|
|||||||
|
|
||||||
The remedy is to set `TRUSTED_PROXIES` to your reverse proxy's
|
The remedy is to set `TRUSTED_PROXIES` to your reverse proxy's
|
||||||
address, which restores per-client buckets. webhooker logs a warning
|
address, which restores per-client buckets. webhooker logs a warning
|
||||||
at startup whenever `TRUSTED_PROXIES` is empty, in every environment —
|
at startup whenever `TRUSTED_PROXIES` is empty, in every environment,
|
||||||
not only when `WEBHOOKER_ENVIRONMENT=prod`, because that variable
|
because behind a proxy every client shares one bucket in `dev` and
|
||||||
defaults to `dev` and an operator who never set it is precisely the
|
`prod` alike. The warning is informational when nothing proxies to the
|
||||||
one at risk. The warning is informational when nothing proxies to the
|
|
||||||
process: with no proxy in front, the peer address is the client's own
|
process: with no proxy in front, the peer address is the client's own
|
||||||
and the buckets are already per-client. See
|
and the buckets are already per-client. See
|
||||||
[Rate Limiting](#rate-limiting) for what each limit shares.
|
[Rate Limiting](#rate-limiting) for what each limit shares.
|
||||||
@@ -638,7 +638,6 @@ decision:
|
|||||||
docker run -d \
|
docker run -d \
|
||||||
-p 127.0.0.1:8080:8080 \
|
-p 127.0.0.1:8080:8080 \
|
||||||
-v /path/to/data:/var/lib/webhooker \
|
-v /path/to/data:/var/lib/webhooker \
|
||||||
-e WEBHOOKER_ENVIRONMENT=prod \
|
|
||||||
-e BIND_ADDRESS=0.0.0.0 \
|
-e BIND_ADDRESS=0.0.0.0 \
|
||||||
webhooker:latest
|
webhooker:latest
|
||||||
```
|
```
|
||||||
@@ -812,17 +811,18 @@ reports.
|
|||||||
serves the admin login form and the unauthenticated receiver with
|
serves the admin login form and the unauthenticated receiver with
|
||||||
no TLS at all, and the proxy in front of it changes nothing about
|
no TLS at all, and the proxy in front of it changes nothing about
|
||||||
that.
|
that.
|
||||||
2. **Set `WEBHOOKER_ENVIRONMENT=prod`, and make sure the proxy sends
|
2. **Make sure the environment is not `dev` (leave
|
||||||
`X-Forwarded-Proto`.** These are two requirements, not one. The
|
`WEBHOOKER_ENVIRONMENT` unset or set it to `prod`), and make sure
|
||||||
environment setting decides CORS and nothing else: the default
|
the proxy sends `X-Forwarded-Proto`.** These are two requirements,
|
||||||
|
not one. The environment setting decides CORS and nothing else:
|
||||||
`dev` answers every origin with `Access-Control-Allow-Origin: *`
|
`dev` answers every origin with `Access-Control-Allow-Origin: *`
|
||||||
(without credentials), which a server-rendered production
|
(without credentials), which a server-rendered production
|
||||||
deployment has no use for. Cookie `Secure` and the strict
|
deployment has no use for, and `prod` — the default — disables it.
|
||||||
Origin/Referer mode are **not** tied to it — they are decided per
|
Cookie `Secure` and the strict Origin/Referer mode are **not** tied
|
||||||
request from the transport, which behind a proxy means the
|
to it — they are decided per request from the transport, which
|
||||||
`X-Forwarded-Proto` header. The block below sets it; without it
|
behind a proxy means the `X-Forwarded-Proto` header. The block below
|
||||||
every request is read as plaintext and cookies ship without
|
sets it; without it every request is read as plaintext and cookies
|
||||||
`Secure`. See [Configuration](#configuration).
|
ship without `Secure`. See [Configuration](#configuration).
|
||||||
3. **Set `TRUSTED_PROXIES` to the proxy's address.** Unset, every rate
|
3. **Set `TRUSTED_PROXIES` to the proxy's address.** Unset, every rate
|
||||||
limiter keys on the connecting peer, which behind a proxy is the
|
limiter keys on the connecting peer, which behind a proxy is the
|
||||||
proxy on every request: all clients collapse into one global bucket
|
proxy on every request: all clients collapse into one global bucket
|
||||||
@@ -914,7 +914,6 @@ sent — `$scheme` above does.
|
|||||||
With that block, webhooker's environment is:
|
With that block, webhooker's environment is:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
WEBHOOKER_ENVIRONMENT=prod
|
|
||||||
BIND_ADDRESS=127.0.0.1 # the default; stated here to be explicit
|
BIND_ADDRESS=127.0.0.1 # the default; stated here to be explicit
|
||||||
TRUSTED_PROXIES=127.0.0.1
|
TRUSTED_PROXIES=127.0.0.1
|
||||||
```
|
```
|
||||||
@@ -1762,19 +1761,25 @@ retries) is individually logged for full observability.
|
|||||||
|
|
||||||
#### Event-tier indexes
|
#### Event-tier indexes
|
||||||
|
|
||||||
Beyond the primary keys, the per-webhook event databases carry secondary
|
These indexes on the per-webhook event databases are declared in the model
|
||||||
indexes on the columns the background work reads by, each created by
|
tags, so `AutoMigrate` creates them on a fresh and on an existing database:
|
||||||
`AutoMigrate` on a fresh and on an existing database:
|
|
||||||
|
|
||||||
| Column | Serves |
|
| Table | Columns | Serves |
|
||||||
| ------------------------------ | ------ |
|
| ------------------ | --------------------------- | ------ |
|
||||||
| `deliveries.status` | The recovery and sweep queries that select deliveries by status once a minute |
|
| `deliveries` | `status`, `deleted_at` | Startup recovery, the retry and pending sweeps every 60 seconds and the queue-depth sampler every 30 seconds, which select deliveries by status |
|
||||||
| `deliveries.event_id` | Loading a page of the event log, which reads deliveries by event |
|
| `deliveries` | `event_id`, `deleted_at` | The event log, which loads each event's deliveries, and retention, which selects and deletes the deliveries of expired events |
|
||||||
| `delivery_results.delivery_id` | Loading a page of the event log, which reads results by delivery |
|
| `delivery_results` | `delivery_id`, `deleted_at` | The event log, which loads the attempts of a page's deliveries, and retention, which deletes the attempts of expired events |
|
||||||
| `events.created_at` | Retention, which deletes events by age |
|
| `events` | `deleted_at`, `created_at` | Retention, which selects expired events by age |
|
||||||
|
| `events` | `created_at` | Retention's delete of the expired events themselves |
|
||||||
|
|
||||||
The `events.resubmitted_from_id` column is also indexed, to resolve the
|
GORM's soft delete adds `deleted_at IS NULL` to these queries; retention's
|
||||||
resubmit relationship both ways in the event log.
|
deletes leave it out, but their lookups of expired rows keep it. SQLite keeps
|
||||||
|
no statistics on these tables, and without them it rates the `deleted_at`
|
||||||
|
index, which every live row matches, above an index on a column matched
|
||||||
|
against several values or compared with `<`. So every index but the last also
|
||||||
|
covers `deleted_at`. It comes second, so that retention's deletes can use the
|
||||||
|
index without it, except in `events`, where `created_at` is compared with `<`
|
||||||
|
and SQLite narrows by a `<` only on the last column it uses.
|
||||||
|
|
||||||
#### Common Fields
|
#### Common Fields
|
||||||
|
|
||||||
|
|||||||
@@ -585,12 +585,14 @@ func resolveMetricsAuth() (string, string, error) {
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
// resolveEnvironment reads WEBHOOKER_ENVIRONMENT, defaulting to
|
// resolveEnvironment reads WEBHOOKER_ENVIRONMENT, defaulting to prod
|
||||||
// dev, and rejects unrecognised values.
|
// when it is unset so a deployment that forgets the variable is not
|
||||||
|
// silently permissive; dev must be set explicitly. It rejects
|
||||||
|
// unrecognised values.
|
||||||
func resolveEnvironment() (string, error) {
|
func resolveEnvironment() (string, error) {
|
||||||
environment := os.Getenv("WEBHOOKER_ENVIRONMENT")
|
environment := os.Getenv("WEBHOOKER_ENVIRONMENT")
|
||||||
if environment == "" {
|
if environment == "" {
|
||||||
environment = EnvironmentDev
|
environment = EnvironmentProd
|
||||||
}
|
}
|
||||||
|
|
||||||
if environment != EnvironmentDev &&
|
if environment != EnvironmentDev &&
|
||||||
@@ -772,10 +774,8 @@ func (c *Config) warnEgressAllowlist(log *slog.Logger) {
|
|||||||
// everyone else's wrong passwords, and the receiver's limits become
|
// everyone else's wrong passwords, and the receiver's limits become
|
||||||
// service-wide ceilings.
|
// service-wide ceilings.
|
||||||
//
|
//
|
||||||
// The warning is deliberately not gated on WEBHOOKER_ENVIRONMENT. That
|
// The warning is deliberately not gated on WEBHOOKER_ENVIRONMENT:
|
||||||
// variable defaults to dev, so gating on it would silence the warning
|
// behind a proxy every client shares one bucket in dev and prod alike.
|
||||||
// for exactly the operator who forgot to configure the deployment —
|
|
||||||
// the case it exists to catch.
|
|
||||||
//
|
//
|
||||||
// The default of trusting nobody is deliberate — trusting forwarded
|
// The default of trusting nobody is deliberate — trusting forwarded
|
||||||
// headers from arbitrary peers lets any client choose its own bucket —
|
// headers from arbitrary peers lets any client choose its own bucket —
|
||||||
|
|||||||
@@ -44,9 +44,9 @@ func TestEnvironmentConfig(t *testing.T) {
|
|||||||
isProd bool
|
isProd bool
|
||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
name: "default is dev",
|
name: "default is prod",
|
||||||
isDev: true,
|
isDev: false,
|
||||||
isProd: false,
|
isProd: true,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "explicit dev",
|
name: "explicit dev",
|
||||||
@@ -848,10 +848,9 @@ func TestEgressAllowlistWarning(t *testing.T) {
|
|||||||
// tells an operator a deployment behind a reverse proxy shares one
|
// tells an operator a deployment behind a reverse proxy shares one
|
||||||
// rate-limit bucket between every client, which turns the receiver
|
// rate-limit bucket between every client, which turns the receiver
|
||||||
// limits into service-wide ceilings and collapses login failure
|
// limits into service-wide ceilings and collapses login failure
|
||||||
// counting. It must fire whenever TRUSTED_PROXIES is empty,
|
// counting. It must fire whenever TRUSTED_PROXIES is empty, in any
|
||||||
// in any environment: WEBHOOKER_ENVIRONMENT defaults to dev, so gating
|
// environment, because behind a proxy every client shares one bucket
|
||||||
// on it would silence the warning for exactly the operator who never
|
// in dev and prod alike. It stays quiet once proxies are named.
|
||||||
// configured the deployment. It stays quiet once proxies are named.
|
|
||||||
func TestSharedRateLimitBucketWarning(t *testing.T) {
|
func TestSharedRateLimitBucketWarning(t *testing.T) {
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
@@ -871,10 +870,6 @@ func TestSharedRateLimitBucketWarning(t *testing.T) {
|
|||||||
expectWarning: false,
|
expectWarning: false,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
// The default environment. An internet-exposed
|
|
||||||
// deployment whose operator never set
|
|
||||||
// WEBHOOKER_ENVIRONMENT lands here and has exactly
|
|
||||||
// the exposure the warning announces.
|
|
||||||
name: "dev without trusted proxies warns",
|
name: "dev without trusted proxies warns",
|
||||||
environment: config.EnvironmentDev,
|
environment: config.EnvironmentDev,
|
||||||
expectWarning: true,
|
expectWarning: true,
|
||||||
|
|||||||
@@ -2,37 +2,33 @@ package database_test
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"fmt"
|
||||||
"testing"
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
"github.com/google/uuid"
|
"github.com/google/uuid"
|
||||||
"github.com/stretchr/testify/assert"
|
"github.com/stretchr/testify/assert"
|
||||||
"github.com/stretchr/testify/require"
|
"github.com/stretchr/testify/require"
|
||||||
|
"gorm.io/gorm"
|
||||||
"sneak.berlin/go/webhooker/internal/database"
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
)
|
)
|
||||||
|
|
||||||
// indexedColumn names a secondary index by the model and struct field
|
|
||||||
// GORM derives the index name from.
|
|
||||||
type indexedColumn struct {
|
|
||||||
model any
|
|
||||||
field string
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestWebhookDBManager_OpenAddsEventTierIndexes verifies that opening a
|
// TestWebhookDBManager_OpenAddsEventTierIndexes verifies that opening a
|
||||||
// per-webhook database that predates these indexes creates them, so the
|
// per-webhook database that predates these indexes creates them. It
|
||||||
// queries that read by those columns stop scanning whole tables. It
|
|
||||||
// stands in for an older database file by dropping the indexes
|
// stands in for an older database file by dropping the indexes
|
||||||
// AutoMigrate just created, then reopening the same file.
|
// AutoMigrate just created, then reopening the same file.
|
||||||
func TestWebhookDBManager_OpenAddsEventTierIndexes(t *testing.T) {
|
func TestWebhookDBManager_OpenAddsEventTierIndexes(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
// The columns the background work reads by: the recovery and sweep
|
indexes := []struct {
|
||||||
// queries (status), the event log (event_id and delivery_id) and
|
model any
|
||||||
// retention (created_at).
|
name string
|
||||||
eventTierIndexes := []indexedColumn{
|
}{
|
||||||
{&database.Delivery{}, "Status"},
|
{&database.Delivery{}, "idx_deliveries_status"},
|
||||||
{&database.Delivery{}, "EventID"},
|
{&database.Delivery{}, "idx_deliveries_event_id"},
|
||||||
{&database.DeliveryResult{}, "DeliveryID"},
|
{&database.DeliveryResult{}, "idx_delivery_results_delivery_id"},
|
||||||
{&database.Event{}, "CreatedAt"},
|
{&database.Event{}, "idx_events_deleted_at_created_at"},
|
||||||
|
{&database.Event{}, "idx_events_created_at"},
|
||||||
}
|
}
|
||||||
|
|
||||||
mgr, lc := setupTestWebhookDBManager(t)
|
mgr, lc := setupTestWebhookDBManager(t)
|
||||||
@@ -47,14 +43,14 @@ func TestWebhookDBManager_OpenAddsEventTierIndexes(t *testing.T) {
|
|||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
|
|
||||||
// A fresh database has them.
|
// A fresh database has them.
|
||||||
for _, ix := range eventTierIndexes {
|
for _, ix := range indexes {
|
||||||
require.True(t, db.Migrator().HasIndex(ix.model, ix.field))
|
require.True(t, db.Migrator().HasIndex(ix.model, ix.name))
|
||||||
}
|
}
|
||||||
|
|
||||||
// Stand in for a database file created before the indexes existed.
|
// Stand in for a database file created before the indexes existed.
|
||||||
for _, ix := range eventTierIndexes {
|
for _, ix := range indexes {
|
||||||
require.NoError(t, db.Migrator().DropIndex(ix.model, ix.field))
|
require.NoError(t, db.Migrator().DropIndex(ix.model, ix.name))
|
||||||
require.False(t, db.Migrator().HasIndex(ix.model, ix.field))
|
require.False(t, db.Migrator().HasIndex(ix.model, ix.name))
|
||||||
}
|
}
|
||||||
|
|
||||||
// Drop the cached connection so the next open reopens the file and
|
// Drop the cached connection so the next open reopens the file and
|
||||||
@@ -64,9 +60,107 @@ func TestWebhookDBManager_OpenAddsEventTierIndexes(t *testing.T) {
|
|||||||
db, err = mgr.GetDB(webhookID)
|
db, err = mgr.GetDB(webhookID)
|
||||||
require.NoError(t, err)
|
require.NoError(t, err)
|
||||||
|
|
||||||
for _, ix := range eventTierIndexes {
|
for _, ix := range indexes {
|
||||||
assert.True(t, db.Migrator().HasIndex(ix.model, ix.field),
|
assert.True(t, db.Migrator().HasIndex(ix.model, ix.name),
|
||||||
"opening the existing database should create the index on %s",
|
"opening the existing database should create %s", ix.name)
|
||||||
ix.field)
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestEventTierQueriesUseTheirIndexes verifies that the statements the
|
||||||
|
// indexes are for use them. GORM builds each statement in a dry run as
|
||||||
|
// the code named above it does, soft-delete condition included, and
|
||||||
|
// SQLite, which keeps no statistics on these tables, must plan to seek
|
||||||
|
// on each index listed by the columns in parentheses.
|
||||||
|
func TestEventTierQueriesUseTheirIndexes(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
mgr, lc := setupTestWebhookDBManager(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
require.NoError(t, lc.Start(ctx))
|
||||||
|
|
||||||
|
defer func() { require.NoError(t, lc.Stop(ctx)) }()
|
||||||
|
|
||||||
|
db, err := mgr.GetDB(uuid.New().String())
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
dry := db.Session(&gorm.Session{DryRun: true})
|
||||||
|
ids := []string{
|
||||||
|
uuid.New().String(), uuid.New().String(), uuid.New().String(),
|
||||||
|
}
|
||||||
|
cutoff := time.Now()
|
||||||
|
|
||||||
|
var (
|
||||||
|
deliveries []database.Delivery
|
||||||
|
results []database.DeliveryResult
|
||||||
|
depths []struct{ Depth int }
|
||||||
|
)
|
||||||
|
|
||||||
|
byStatus := "idx_deliveries_status (status=? AND deleted_at=?)"
|
||||||
|
byEvent := "idx_deliveries_event_id (event_id=? AND deleted_at=?)"
|
||||||
|
byAge := "idx_events_deleted_at_created_at (deleted_at=? AND created_at<?)"
|
||||||
|
|
||||||
|
// The delivery engine: recovery and the retry sweep, the sweep for
|
||||||
|
// stranded pending deliveries, and the queue depth count.
|
||||||
|
assertPlanUses(t, db, dry.Where(
|
||||||
|
"status = ?", database.DeliveryStatusRetrying,
|
||||||
|
).Find(&deliveries), byStatus)
|
||||||
|
assertPlanUses(t, db, dry.Where(
|
||||||
|
"status = ? AND updated_at < ?",
|
||||||
|
database.DeliveryStatusPending, cutoff,
|
||||||
|
).Limit(500).Find(&deliveries), byStatus)
|
||||||
|
assertPlanUses(t, db, dry.Model(&database.Delivery{}).
|
||||||
|
Select("target_id", "status", "count(*) as depth").
|
||||||
|
Where("status IN ?", []database.DeliveryStatus{
|
||||||
|
database.DeliveryStatusPending,
|
||||||
|
database.DeliveryStatusRetrying,
|
||||||
|
}).Group("target_id, status").Find(&depths), byStatus)
|
||||||
|
|
||||||
|
// The event log: each event's deliveries, then their attempts
|
||||||
|
// (loadEventsWithDeliveries, loadDeliveryResults).
|
||||||
|
assertPlanUses(t, db, dry.Where("event_id = ?", ids[0]).
|
||||||
|
Find(&deliveries), byEvent)
|
||||||
|
assertPlanUses(t, db, dry.Where("delivery_id IN ?", ids).
|
||||||
|
Order("attempt_num ASC").Find(&results),
|
||||||
|
"idx_delivery_results_delivery_id (delivery_id=? AND deleted_at=?)")
|
||||||
|
|
||||||
|
// Retention's three deletes (reapExpired), whose subqueries are built
|
||||||
|
// afresh for each statement as it builds them.
|
||||||
|
expiredEventIDs := func() *gorm.DB {
|
||||||
|
return dry.Model(&database.Event{}).Select("id").
|
||||||
|
Where("created_at < ?", cutoff)
|
||||||
|
}
|
||||||
|
|
||||||
|
assertPlanUses(t, db, dry.Unscoped().Where(
|
||||||
|
"delivery_id IN (?)", dry.Model(&database.Delivery{}).
|
||||||
|
Select("id").Where("event_id IN (?)", expiredEventIDs()),
|
||||||
|
).Delete(&database.DeliveryResult{}),
|
||||||
|
"idx_delivery_results_delivery_id (delivery_id=?)", byEvent, byAge)
|
||||||
|
assertPlanUses(t, db, dry.Unscoped().Where(
|
||||||
|
"event_id IN (?)", expiredEventIDs(),
|
||||||
|
).Delete(&database.Delivery{}),
|
||||||
|
"idx_deliveries_event_id (event_id=?)", byAge)
|
||||||
|
assertPlanUses(t, db, dry.Unscoped().Where(
|
||||||
|
"created_at < ?", cutoff,
|
||||||
|
).Delete(&database.Event{}), "idx_events_created_at (created_at<?)")
|
||||||
|
}
|
||||||
|
|
||||||
|
// assertPlanUses asserts that SQLite's plan for a statement GORM built
|
||||||
|
// in a dry run, run with the same SQL and arguments GORM would send,
|
||||||
|
// names each of the given indexes.
|
||||||
|
func assertPlanUses(
|
||||||
|
t *testing.T, db, built *gorm.DB, indexes ...string,
|
||||||
|
) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
var plan []struct{ Detail string }
|
||||||
|
|
||||||
|
require.NoError(t, db.Raw(
|
||||||
|
"EXPLAIN QUERY PLAN "+built.Statement.SQL.String(),
|
||||||
|
built.Statement.Vars...,
|
||||||
|
).Scan(&plan).Error)
|
||||||
|
|
||||||
|
for _, index := range indexes {
|
||||||
|
assert.Contains(t, fmt.Sprint(plan), index,
|
||||||
|
built.Statement.SQL.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,5 +1,7 @@
|
|||||||
package database
|
package database
|
||||||
|
|
||||||
|
import "gorm.io/gorm"
|
||||||
|
|
||||||
// DeliveryStatus represents the status of a delivery
|
// DeliveryStatus represents the status of a delivery
|
||||||
type DeliveryStatus string
|
type DeliveryStatus string
|
||||||
|
|
||||||
@@ -29,12 +31,19 @@ func (s DeliveryStatus) Terminal() bool {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Delivery represents a delivery attempt for an event to a target
|
// Delivery represents a delivery attempt for an event to a target
|
||||||
|
//
|
||||||
|
//nolint:lll // a struct tag cannot wrap
|
||||||
type Delivery struct {
|
type Delivery struct {
|
||||||
BaseModel
|
BaseModel
|
||||||
|
|
||||||
EventID string `gorm:"type:uuid;not null;index" json:"eventId"`
|
EventID string `gorm:"type:uuid;not null;index:idx_deliveries_event_id,priority:1" json:"eventId"`
|
||||||
TargetID string `gorm:"type:uuid;not null" json:"targetId"`
|
TargetID string `gorm:"type:uuid;not null" json:"targetId"`
|
||||||
Status DeliveryStatus `gorm:"not null;default:'pending';index" json:"status"`
|
Status DeliveryStatus `gorm:"not null;default:'pending';index:idx_deliveries_status,priority:1" json:"status"`
|
||||||
|
|
||||||
|
// DeletedAt repeats the BaseModel field only to be the second column
|
||||||
|
// of the event_id and status indexes, for the reason DeliveryResult
|
||||||
|
// gives.
|
||||||
|
DeletedAt gorm.DeletedAt `gorm:"index:idx_deliveries_event_id,priority:2;index:idx_deliveries_status,priority:2" json:"deletedAt,omitzero"`
|
||||||
|
|
||||||
// Relations
|
// Relations
|
||||||
Event Event `json:"event,omitzero"`
|
Event Event `json:"event,omitzero"`
|
||||||
|
|||||||
@@ -1,14 +1,25 @@
|
|||||||
package database
|
package database
|
||||||
|
|
||||||
|
import "gorm.io/gorm"
|
||||||
|
|
||||||
// DeliveryResult represents the result of a delivery attempt
|
// DeliveryResult represents the result of a delivery attempt
|
||||||
|
//
|
||||||
|
//nolint:lll // a struct tag cannot wrap
|
||||||
type DeliveryResult struct {
|
type DeliveryResult struct {
|
||||||
BaseModel
|
BaseModel
|
||||||
|
|
||||||
DeliveryID string `gorm:"type:uuid;not null;index" json:"deliveryId"`
|
// DeliveryID and DeletedAt make up one index, in that order.
|
||||||
AttemptNum int `gorm:"not null" json:"attemptNum"`
|
// DeletedAt repeats the BaseModel field only to join it: GORM adds
|
||||||
|
// "deleted_at IS NULL" to almost every query, and where a column is
|
||||||
|
// matched against several values SQLite otherwise reads through the
|
||||||
|
// deleted_at index, which every live row matches.
|
||||||
|
DeliveryID string `gorm:"type:uuid;not null;index:idx_delivery_results_delivery_id,priority:1" json:"deliveryId"`
|
||||||
|
DeletedAt gorm.DeletedAt `gorm:"index:idx_delivery_results_delivery_id,priority:2" json:"deletedAt,omitzero"`
|
||||||
|
|
||||||
|
AttemptNum int `gorm:"not null" json:"attemptNum"`
|
||||||
Success bool `json:"success"`
|
Success bool `json:"success"`
|
||||||
StatusCode int `json:"statusCode,omitempty"`
|
StatusCode int `json:"statusCode,omitempty"`
|
||||||
ResponseBody string `gorm:"type:text" json:"responseBody,omitempty"`
|
ResponseBody string `gorm:"type:text" json:"responseBody,omitempty"`
|
||||||
Error string `json:"error,omitempty"`
|
Error string `json:"error,omitempty"`
|
||||||
Duration int64 `json:"durationMs"` // Duration in milliseconds
|
Duration int64 `json:"durationMs"` // Duration in milliseconds
|
||||||
|
|
||||||
|
|||||||
@@ -1,16 +1,26 @@
|
|||||||
package database
|
package database
|
||||||
|
|
||||||
import "time"
|
import (
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gorm.io/gorm"
|
||||||
|
)
|
||||||
|
|
||||||
// Event represents a captured webhook event
|
// Event represents a captured webhook event
|
||||||
|
//
|
||||||
|
//nolint:lll // a struct tag cannot wrap
|
||||||
type Event struct {
|
type Event struct {
|
||||||
BaseModel
|
BaseModel
|
||||||
|
|
||||||
// CreatedAt overrides BaseModel.CreatedAt only to add an index:
|
// CreatedAt and DeletedAt repeat the BaseModel fields only to index
|
||||||
// retention deletes events by age, so events.created_at is queried
|
// them for retention, which finds events by age. Its lookups carry
|
||||||
// on every sweep. The other tables keep the unindexed BaseModel
|
// GORM's "deleted_at IS NULL" (see DeliveryResult) and compare
|
||||||
// field.
|
// created_at with <, so their index has deleted_at first: SQLite
|
||||||
CreatedAt time.Time `gorm:"index" json:"createdAt"`
|
// narrows by a < only on the last column it uses. Its final delete
|
||||||
|
// has no deleted_at condition and uses the index on created_at
|
||||||
|
// alone. The other tables keep the unindexed BaseModel created_at.
|
||||||
|
CreatedAt time.Time `gorm:"index;index:idx_events_deleted_at_created_at,priority:2" json:"createdAt"`
|
||||||
|
DeletedAt gorm.DeletedAt `gorm:"index:idx_events_deleted_at_created_at,priority:1" json:"deletedAt,omitzero"`
|
||||||
|
|
||||||
WebhookID string `gorm:"type:uuid;not null" json:"webhookId"`
|
WebhookID string `gorm:"type:uuid;not null" json:"webhookId"`
|
||||||
EntrypointID string `gorm:"type:uuid;not null" json:"entrypointId"`
|
EntrypointID string `gorm:"type:uuid;not null" json:"entrypointId"`
|
||||||
|
|||||||
@@ -380,9 +380,8 @@ func csrfTookStrictPath(
|
|||||||
|
|
||||||
// TestCSRF_ForwardedProtoSpellingsTakeStrictPath runs the header
|
// TestCSRF_ForwardedProtoSpellingsTakeStrictPath runs the header
|
||||||
// spellings a real proxy emits through the middleware. The environment
|
// spellings a real proxy emits through the middleware. The environment
|
||||||
// is dev -- the DEFAULT when WEBHOOKER_ENVIRONMENT is unset -- to pin
|
// is set to dev -- the permissive setting -- to pin that the routing is
|
||||||
// that the routing is a per-request transport decision and owes
|
// a per-request transport decision and owes nothing to configuration.
|
||||||
// nothing to configuration.
|
|
||||||
func TestCSRF_ForwardedProtoSpellingsTakeStrictPath(t *testing.T) {
|
func TestCSRF_ForwardedProtoSpellingsTakeStrictPath(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
@@ -5,7 +5,7 @@
|
|||||||
// several packages, by hand, and the answers disagreed. The session
|
// several packages, by hand, and the answers disagreed. The session
|
||||||
// cookie's Secure attribute was decided at startup from the configured
|
// cookie's Secure attribute was decided at startup from the configured
|
||||||
// environment while the CSRF cookie's was decided per-request, so a
|
// environment while the CSRF cookie's was decided per-request, so a
|
||||||
// deployment behind a TLS proxy in the default environment emitted one
|
// deployment behind a TLS proxy in the dev environment emitted one
|
||||||
// Secure cookie and one non-Secure cookie on the same response.
|
// Secure cookie and one non-Secure cookie on the same response.
|
||||||
// Everything kept working, which is exactly why nobody noticed.
|
// Everything kept working, which is exactly why nobody noticed.
|
||||||
//
|
//
|
||||||
|
|||||||
@@ -146,8 +146,8 @@ func newStore(key []byte) *sessions.CookieStore {
|
|||||||
//
|
//
|
||||||
// This is decided per-request, not once at startup. Deciding it at
|
// This is decided per-request, not once at startup. Deciding it at
|
||||||
// startup from the configured environment is what this replaces, and
|
// startup from the configured environment is what this replaces, and
|
||||||
// it got the DEFAULT posture wrong: "dev" is the environment when
|
// it got the DEFAULT posture wrong: "dev" was then the environment when
|
||||||
// WEBHOOKER_ENVIRONMENT is unset, so a deployment terminating TLS at a
|
// WEBHOOKER_ENVIRONMENT was unset, so a deployment terminating TLS at a
|
||||||
// proxy without also setting the environment emitted the
|
// proxy without also setting the environment emitted the
|
||||||
// authentication cookie with no Secure attribute -- silently, and on
|
// authentication cookie with no Secure attribute -- silently, and on
|
||||||
// the same response as a CSRF cookie that did have one.
|
// the same response as a CSRF cookie that did have one.
|
||||||
|
|||||||
@@ -990,8 +990,8 @@ func sessionCookieFrom(
|
|||||||
|
|
||||||
// TestSave_SecureFollowsRequestTransport is the regression test for
|
// TestSave_SecureFollowsRequestTransport is the regression test for
|
||||||
// the defect this replaces: Secure was fixed at startup from the
|
// the defect this replaces: Secure was fixed at startup from the
|
||||||
// configured environment, and "dev" is the environment when
|
// configured environment, and "dev" was then the environment when
|
||||||
// WEBHOOKER_ENVIRONMENT is unset. A deployment behind a TLS proxy in
|
// WEBHOOKER_ENVIRONMENT was unset. A deployment behind a TLS proxy in
|
||||||
// that DEFAULT posture shipped the authentication cookie with no
|
// that DEFAULT posture shipped the authentication cookie with no
|
||||||
// Secure attribute and said nothing about it.
|
// Secure attribute and said nothing about it.
|
||||||
//
|
//
|
||||||
|
|||||||
Reference in New Issue
Block a user