Show each entrypoint's last event and event count on the webhook page (closes #393)
check / check (push) Successful in 3m8s
check / check (push) Successful in 3m8s
Each entrypoint in the webhook page's entrypoint list shows when the last event arrived on its URL (relative, with the full UTC time on hover), or "never" if none ever has, and how many events arrived on it within the webhook's retention period. The last event is kept per entrypoint in a new EntrypointTotals row in the event database, written in the transaction that stores the event, so retention leaves it in place. The count is one query per page, grouped by entrypoint, over a new events index on entrypoint_id, deleted_at, resubmitted_from_id and created_at. Resubmitted copies did not arrive on the URL and count in neither. Pre-1.0: the table and index go into the schema in place. Model: opus-5-5
This commit is contained in:
@@ -1127,7 +1127,7 @@ unconditionally against whatever files it finds:
|
|||||||
- the main database on connect — `Setting`, `User`, `APIKey`, `Webhook`,
|
- the main database on connect — `Setting`, `User`, `APIKey`, `Webhook`,
|
||||||
`Entrypoint`, `Target`
|
`Entrypoint`, `Target`
|
||||||
- each event database when it is lazily opened — `Event`, `Delivery`,
|
- each event database when it is lazily opened — `Event`, `Delivery`,
|
||||||
`DeliveryResult`, `EventTotals`, `TargetTotals`
|
`DeliveryResult`, `EventTotals`, `TargetTotals`, `EntrypointTotals`
|
||||||
- each archive database on every open and reopen
|
- each archive database on every open and reopen
|
||||||
|
|
||||||
There is no schema version table, no migration ledger, and no down
|
There is no schema version table, no migration ledger, and no down
|
||||||
@@ -1531,6 +1531,9 @@ tier** (event ingestion, delivery, and logging).
|
|||||||
│ ┌──────────────┐ (one row per target: running counts │
|
│ ┌──────────────┐ (one row per target: running counts │
|
||||||
│ │ TargetTotals │ of its deliveries) │
|
│ │ TargetTotals │ of its deliveries) │
|
||||||
│ └──────────────┘ │
|
│ └──────────────┘ │
|
||||||
|
│ ┌──────────────────┐ (one row per entrypoint: when the │
|
||||||
|
│ │ EntrypointTotals │ last event arrived on its URL) │
|
||||||
|
│ └──────────────────┘ │
|
||||||
└─────────────────────────────────────────────────────────────┘
|
└─────────────────────────────────────────────────────────────┘
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -1644,6 +1647,11 @@ different event sources that all feed into the same processing pipeline
|
|||||||
(e.g., one entrypoint for GitHub, another for Stripe, both routing to
|
(e.g., one entrypoint for GitHub, another for Stripe, both routing to
|
||||||
the same targets).
|
the same targets).
|
||||||
|
|
||||||
|
The webhook page shows, for each entrypoint, when the last event arrived
|
||||||
|
on its URL, which retention leaves in place, or "never" if none ever has,
|
||||||
|
and how many events arrived on it within the webhook's retention period.
|
||||||
|
A resubmitted event did not arrive on the URL and counts in neither.
|
||||||
|
|
||||||
#### Target
|
#### Target
|
||||||
|
|
||||||
A delivery destination for events. Each target defines where and how
|
A delivery destination for events. Each target defines where and how
|
||||||
@@ -1853,10 +1861,11 @@ retries) is individually logged for full observability.
|
|||||||
|
|
||||||
**Relations:** Belongs to Delivery.
|
**Relations:** Belongs to Delivery.
|
||||||
|
|
||||||
#### EventTotals and TargetTotals
|
#### EventTotals, TargetTotals and EntrypointTotals
|
||||||
|
|
||||||
Running counts in each event database, read by the statistics pane at the
|
Running counts in each event database, read by the statistics pane at the
|
||||||
top of the webhook page and by the webhook list. `EventTotals` is one row:
|
top of the webhook page and by the webhook list, and each entrypoint's last
|
||||||
|
event, read by the webhook page's entrypoint list. `EventTotals` is one row:
|
||||||
|
|
||||||
| Field | Type | Description |
|
| Field | Type | Description |
|
||||||
| ---------------- | --------- | ----------- |
|
| ---------------- | --------- | ----------- |
|
||||||
@@ -1875,18 +1884,26 @@ top of the webhook page and by the webhook list. `EventTotals` is one row:
|
|||||||
| `deliveries_removed` | integer | Its deliveries retention has deleted |
|
| `deliveries_removed` | integer | Its deliveries retention has deleted |
|
||||||
| `failed_removed` | integer | Its failed deliveries retention has deleted |
|
| `failed_removed` | integer | Its failed deliveries retention has deleted |
|
||||||
|
|
||||||
Each count changes in the transaction that writes or deletes the rows it
|
`EntrypointTotals` is one row per entrypoint, created by the first event
|
||||||
counts. The pane's lifetime events are `events`, and its lifetime
|
that arrives on its URL:
|
||||||
deliveries and failures are `deliveries` and `failed` summed over the
|
|
||||||
targets; each figure within retention is the same less what retention
|
| Field | Type | Description |
|
||||||
removed, so neither needs the rows themselves. Its last event is
|
| --------------- | --------- | ----------- |
|
||||||
`last_event_at`, written in the transaction that stores the event, so it
|
| `entrypoint_id` | UUID | The entrypoint (primary key) |
|
||||||
still shows once retention has removed every event. Its last-10-minutes and
|
| `last_event_at` | timestamp | When the newest event arrived on its URL; a resubmitted event leaves it as it is, and so does retention |
|
||||||
last-24-hours figures are counted from the `events` and `deliveries`
|
|
||||||
indexes over just that window, the deliveries in one query grouped by
|
Each count changes in the transaction that writes or deletes the rows it counts,
|
||||||
target. Its failure percentage for a window is the deliveries that became
|
and each `last_event_at` in the transaction that stores the event. The pane's
|
||||||
`failed` in it out of all that became `delivered` or `failed` in it, and
|
lifetime events are `events`, and its lifetime deliveries and failures are
|
||||||
a dash when none did.
|
`deliveries` and `failed` summed over the targets; each figure within retention
|
||||||
|
is the same less what retention removed, so neither needs the rows themselves.
|
||||||
|
Its last event is `last_event_at` in `EventTotals`, so it still shows once
|
||||||
|
retention has removed every event; each entrypoint's last event, from
|
||||||
|
`EntrypointTotals`, does too. Its last-10-minutes and last-24-hours figures are
|
||||||
|
counted from the `events` and `deliveries` indexes over just that window, the
|
||||||
|
deliveries in one query grouped by target. Its failure percentage for a window
|
||||||
|
is the deliveries that became `failed` in it out of all that became `delivered`
|
||||||
|
or `failed` in it, and a dash when none did.
|
||||||
|
|
||||||
The webhook list at `/hooks` shows three of the pane's figures for each
|
The webhook list at `/hooks` shows three of the pane's figures for each
|
||||||
webhook: its events within retention and its last event, both from
|
webhook: its events within retention and its last event, both from
|
||||||
@@ -1908,26 +1925,31 @@ tags, so `AutoMigrate` creates them on a fresh database:
|
|||||||
| `delivery_results` | `delivery_id`, `deleted_at` | The event log, which loads the attempts of a page's deliveries, and retention, which deletes the attempts of expired events |
|
| `delivery_results` | `delivery_id`, `deleted_at` | The event log, which loads the attempts of a page's deliveries, and retention, which deletes the attempts of expired events |
|
||||||
| `events` | `deleted_at`, `created_at` | The webhook page's statistics, which count recent events |
|
| `events` | `deleted_at`, `created_at` | The webhook page's statistics, which count recent events |
|
||||||
| `events` | `resubmitted_from_id`, `deleted_at` | The event log, which counts the events resubmitted from each event on a page |
|
| `events` | `resubmitted_from_id`, `deleted_at` | The event log, which counts the events resubmitted from each event on a page |
|
||||||
|
| `events` | `entrypoint_id`, `deleted_at`, `resubmitted_from_id`, `created_at` | The webhook page's entrypoint list, which counts the events that arrived on each entrypoint's URL within the retention period |
|
||||||
| `events` | `created_at` | Retention, which selects expired events by age |
|
| `events` | `created_at` | Retention, which selects expired events by age |
|
||||||
|
|
||||||
GORM's soft delete adds `deleted_at IS NULL` to these queries; retention
|
GORM's soft delete adds `deleted_at IS NULL` to these queries; retention leaves
|
||||||
leaves it out. SQLite keeps no statistics on these tables, and without them it
|
it out. SQLite keeps no statistics on these tables, and without them it rates
|
||||||
rates the `deleted_at` index, which every live row matches, above an index on
|
the `deleted_at` index, which every live row matches, above an index on a column
|
||||||
a column matched against several values or compared with a range. So every
|
matched against several values or compared with a range. So every index but the
|
||||||
index but the last also covers `deleted_at`. It comes second in the `event_id`
|
last also covers `deleted_at`. It comes second in the `event_id` and
|
||||||
and `delivery_id` indexes, so that retention can use them without it. The event
|
`delivery_id` indexes, so that retention can use them without it. The event
|
||||||
log's count, the one query on the `resubmitted_from_id` index, always carries
|
log's count, the one query on the `resubmitted_from_id` index, always carries
|
||||||
`deleted_at IS NULL` and uses both columns. In the statistics' `events` index
|
`deleted_at IS NULL` and uses both columns. The entrypoint list's count, the one
|
||||||
`deleted_at` comes first, because they compare `created_at` with a range (`>=`)
|
query on the `entrypoint_id` index, uses all four, `resubmitted_from_id IS NULL`
|
||||||
and SQLite narrows by a range only on the last column it uses.
|
leaving out resubmitted copies and `created_at` last because it compares it with
|
||||||
|
a range (`>=`). In the statistics' `events` index `deleted_at` comes first,
|
||||||
|
because they compare `created_at` with a range (`>=`) and SQLite narrows by a
|
||||||
|
range only on the last column it uses.
|
||||||
|
|
||||||
#### Common Fields
|
#### Common Fields
|
||||||
|
|
||||||
Every entity except `Setting`, `EventTotals` and `TargetTotals` includes
|
Every entity except `Setting`, `EventTotals`, `TargetTotals` and
|
||||||
these fields from `BaseModel`. `Setting` is a bare key-value row with no
|
`EntrypointTotals` includes these fields from `BaseModel`. `Setting` is a bare
|
||||||
`id`, no timestamps and no soft delete, and the two totals tables hold
|
key-value row with no `id`, no timestamps and no soft delete. Of the three
|
||||||
counts, plus `last_event_at` in `event_totals`, keyed by a numeric `id`
|
totals tables, `event_totals` holds counts and `last_event_at`, keyed by a
|
||||||
and by `target_id`:
|
numeric `id`; `target_totals` holds counts, keyed by `target_id`; and
|
||||||
|
`entrypoint_totals` holds `last_event_at`, keyed by `entrypoint_id`:
|
||||||
|
|
||||||
| Field | Type | Description |
|
| Field | Type | Description |
|
||||||
| ------------ | --------- | ----------- |
|
| ------------ | --------- | ----------- |
|
||||||
@@ -1969,8 +1991,9 @@ encryption key is generated and stored, and an `admin` user is created.
|
|||||||
- **Events** — captured incoming webhook payloads
|
- **Events** — captured incoming webhook payloads
|
||||||
- **Deliveries** — event-to-target pairings and their status
|
- **Deliveries** — event-to-target pairings and their status
|
||||||
- **DeliveryResults** — individual delivery attempt logs
|
- **DeliveryResults** — individual delivery attempt logs
|
||||||
- **EventTotals** and **TargetTotals** — running counts of the above,
|
- **EventTotals**, **TargetTotals** and **EntrypointTotals** — running
|
||||||
the deliveries per target, kept through retention
|
counts of the above, the deliveries per target, and each entrypoint's
|
||||||
|
last event, kept through retention
|
||||||
|
|
||||||
Per-webhook databases are created automatically when a webhook is
|
Per-webhook databases are created automatically when a webhook is
|
||||||
created. They are managed by the `WebhookDBManager` component, which
|
created. They are managed by the `WebhookDBManager` component, which
|
||||||
@@ -3035,7 +3058,7 @@ webhooker/
|
|||||||
│ │ ├── model_event.go # Event entity (per-webhook DB)
|
│ │ ├── model_event.go # Event entity (per-webhook DB)
|
||||||
│ │ ├── model_delivery.go # Delivery entity (per-webhook DB)
|
│ │ ├── model_delivery.go # Delivery entity (per-webhook DB)
|
||||||
│ │ ├── model_delivery_result.go # DeliveryResult entity (per-webhook DB)
|
│ │ ├── model_delivery_result.go # DeliveryResult entity (per-webhook DB)
|
||||||
│ │ ├── model_totals.go # EventTotals and TargetTotals (per-webhook DB)
|
│ │ ├── model_totals.go # EventTotals, TargetTotals and EntrypointTotals (per-webhook DB)
|
||||||
│ │ ├── model_apikey.go # APIKey entity
|
│ │ ├── model_apikey.go # APIKey entity
|
||||||
│ │ ├── password.go # Argon2id hashing and verification
|
│ │ ├── password.go # Argon2id hashing and verification
|
||||||
│ │ ├── retention.go # Retention reaper (per-webhook event expiry)
|
│ │ ├── retention.go # Retention reaper (per-webhook event expiry)
|
||||||
@@ -3311,9 +3334,9 @@ check, see [The login endpoint](#the-login-endpoint).
|
|||||||
`ENTRYPOINT` script, which sets the data directory's owner and mode
|
`ENTRYPOINT` script, which sets the data directory's owner and mode
|
||||||
before the app starts; the image's health check; and `docker exec`,
|
before the app starts; the image's health check; and `docker exec`,
|
||||||
unless given `--user`
|
unless given `--user`
|
||||||
- GORM soft deletes on every entity that carries `BaseModel`, which is
|
- GORM soft deletes on every entity that carries `BaseModel`, which is all of
|
||||||
all of them but `Setting`, `EventTotals` and `TargetTotals` (data
|
them but `Setting`, `EventTotals`, `TargetTotals` and `EntrypointTotals`
|
||||||
preserved for audit)
|
(data preserved for audit)
|
||||||
|
|
||||||
### Shutdown
|
### Shutdown
|
||||||
|
|
||||||
|
|||||||
@@ -233,6 +233,43 @@ func TestResubmitCountUsesItsIndex(t *testing.T) {
|
|||||||
"(resubmitted_from_id=? AND deleted_at=?)")
|
"(resubmitted_from_id=? AND deleted_at=?)")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestEntrypointEventsUseTheirIndex does the same for the webhook
|
||||||
|
// page's count, for each entrypoint, of the events that arrived on its
|
||||||
|
// URL since the retention cutoff (addEntrypointEvents in the
|
||||||
|
// handlers), which must come from the index alone. It passes 25
|
||||||
|
// entrypoints, as TestResubmitCountUsesItsIndex passes 25 events.
|
||||||
|
func TestEntrypointEventsUseTheirIndex(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
mgr, lc := setupTestWebhookDBManager(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
require.NoError(t, lc.Start(ctx))
|
||||||
|
|
||||||
|
defer func() { require.NoError(t, lc.Stop(ctx)) }()
|
||||||
|
|
||||||
|
db, err := mgr.GetDB(uuid.New().String())
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
dry := db.Session(&gorm.Session{DryRun: true})
|
||||||
|
|
||||||
|
entrypoints := make([]string, 25)
|
||||||
|
for i := range entrypoints {
|
||||||
|
entrypoints[i] = uuid.New().String()
|
||||||
|
}
|
||||||
|
|
||||||
|
var rows []struct{ Events int }
|
||||||
|
|
||||||
|
assertPlanUses(t, db, dry.Model(&database.Event{}).
|
||||||
|
Select("entrypoint_id, count(*) AS events").
|
||||||
|
Where("entrypoint_id IN ? AND resubmitted_from_id IS NULL",
|
||||||
|
entrypoints).
|
||||||
|
Where("created_at >= ?", time.Now()).
|
||||||
|
Group("entrypoint_id").Find(&rows),
|
||||||
|
"COVERING INDEX idx_events_entrypoint_id "+
|
||||||
|
"(entrypoint_id=? AND deleted_at=? AND "+
|
||||||
|
"resubmitted_from_id=? AND created_at>?)")
|
||||||
|
}
|
||||||
|
|
||||||
// assertPlanUses asserts that SQLite's plan for a statement GORM built
|
// assertPlanUses asserts that SQLite's plan for a statement GORM built
|
||||||
// in a dry run, run with the same SQL and arguments GORM would send,
|
// in a dry run, run with the same SQL and arguments GORM would send,
|
||||||
// names each of the given indexes.
|
// names each of the given indexes.
|
||||||
|
|||||||
@@ -20,12 +20,15 @@ type Event struct {
|
|||||||
// has no deleted_at condition and uses the index on created_at
|
// has no deleted_at condition and uses the index on created_at
|
||||||
// alone. The other tables keep the unindexed BaseModel created_at.
|
// alone. The other tables keep the unindexed BaseModel created_at.
|
||||||
// DeletedAt is also the second column of the resubmitted_from_id
|
// DeletedAt is also the second column of the resubmitted_from_id
|
||||||
// index, for the reason DeliveryResult gives.
|
// index, for the reason DeliveryResult gives. The entrypoint_id
|
||||||
CreatedAt time.Time `gorm:"index;index:idx_events_deleted_at_created_at,priority:2" json:"createdAt"`
|
// index, for the webhook page's entrypoint list, has it second too,
|
||||||
DeletedAt gorm.DeletedAt `gorm:"index:idx_events_deleted_at_created_at,priority:1;index:idx_events_resubmitted_from_id,priority:2" json:"deletedAt,omitzero"`
|
// resubmitted_from_id third, and created_at last, which the list
|
||||||
|
// compares with a range.
|
||||||
|
CreatedAt time.Time `gorm:"index;index:idx_events_deleted_at_created_at,priority:2;index:idx_events_entrypoint_id,priority:4" json:"createdAt"`
|
||||||
|
DeletedAt gorm.DeletedAt `gorm:"index:idx_events_deleted_at_created_at,priority:1;index:idx_events_resubmitted_from_id,priority:2;index:idx_events_entrypoint_id,priority:2" json:"deletedAt,omitzero"`
|
||||||
|
|
||||||
WebhookID string `gorm:"type:uuid;not null" json:"webhookId"`
|
WebhookID string `gorm:"type:uuid;not null" json:"webhookId"`
|
||||||
EntrypointID string `gorm:"type:uuid;not null" json:"entrypointId"`
|
EntrypointID string `gorm:"type:uuid;not null;index:idx_events_entrypoint_id,priority:1" json:"entrypointId"`
|
||||||
|
|
||||||
// Request data
|
// Request data
|
||||||
Method string `gorm:"not null" json:"method"`
|
Method string `gorm:"not null" json:"method"`
|
||||||
@@ -44,7 +47,7 @@ type Event struct {
|
|||||||
// existed. It is not a foreign key: the source event can be
|
// existed. It is not a foreign key: the source event can be
|
||||||
// reaped by retention while its copies remain, and the id is
|
// reaped by retention while its copies remain, and the id is
|
||||||
// kept as the record of where the copy came from either way.
|
// kept as the record of where the copy came from either way.
|
||||||
ResubmittedFromID *string `gorm:"type:uuid;index:idx_events_resubmitted_from_id,priority:1" json:"resubmittedFromId,omitempty"`
|
ResubmittedFromID *string `gorm:"type:uuid;index:idx_events_resubmitted_from_id,priority:1;index:idx_events_entrypoint_id,priority:3" json:"resubmittedFromId,omitempty"`
|
||||||
|
|
||||||
// Relations. No model marshals the record it belongs to, so
|
// Relations. No model marshals the record it belongs to, so
|
||||||
// Webhook and Entrypoint are left out of the JSON.
|
// Webhook and Entrypoint are left out of the JSON.
|
||||||
|
|||||||
@@ -52,6 +52,21 @@ func (TargetTotals) TableName() string {
|
|||||||
return "target_totals"
|
return "target_totals"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// EntrypointTotals is one row per entrypoint, created by the first
|
||||||
|
// event that arrives on its URL: when the newest such event arrived,
|
||||||
|
// which retention leaves as it is. A resubmitted copy did not arrive
|
||||||
|
// on the URL and does not change it.
|
||||||
|
type EntrypointTotals struct {
|
||||||
|
EntrypointID string `gorm:"type:uuid;primaryKey"`
|
||||||
|
|
||||||
|
LastEventAt time.Time `gorm:"not null"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// TableName names the table AddEntrypointTotals updates.
|
||||||
|
func (EntrypointTotals) TableName() string {
|
||||||
|
return "entrypoint_totals"
|
||||||
|
}
|
||||||
|
|
||||||
// AddEventTotals adds each count in add to the webhook's event totals,
|
// AddEventTotals adds each count in add to the webhook's event totals,
|
||||||
// and records add.LastEventAt as when the newest event arrived if it is
|
// and records add.LastEventAt as when the newest event arrived if it is
|
||||||
// set. Call it on the transaction that writes or deletes the events it
|
// set. Call it on the transaction that writes or deletes the events it
|
||||||
@@ -97,3 +112,25 @@ func AddTargetTotals(tx *gorm.DB, add TargetTotals) error {
|
|||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// AddEntrypointTotals records add.LastEventAt as when the newest event
|
||||||
|
// arrived on the URL of the entrypoint add.EntrypointID names, creating
|
||||||
|
// its row the first time. Call it on the transaction that stores the
|
||||||
|
// event.
|
||||||
|
func AddEntrypointTotals(tx *gorm.DB, add EntrypointTotals) error {
|
||||||
|
err := tx.Exec(
|
||||||
|
`INSERT INTO entrypoint_totals (entrypoint_id, last_event_at)
|
||||||
|
VALUES (?, ?)
|
||||||
|
ON CONFLICT (entrypoint_id) DO UPDATE SET
|
||||||
|
last_event_at = excluded.last_event_at`,
|
||||||
|
add.EntrypointID, add.LastEventAt,
|
||||||
|
).Error
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf(
|
||||||
|
"adding to totals of entrypoint %s: %w",
|
||||||
|
add.EntrypointID, err,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|||||||
@@ -111,6 +111,13 @@ func (w *Webhook) RetainsForever() bool {
|
|||||||
return retainsForever(w.RetentionDays)
|
return retainsForever(w.RetentionDays)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// RetentionCutoff returns the time before which this webhook's events
|
||||||
|
// have expired, as the reaper computes it, and false when the webhook
|
||||||
|
// retains them forever.
|
||||||
|
func (w *Webhook) RetentionCutoff(now time.Time) (time.Time, bool) {
|
||||||
|
return retentionCutoff(now, w.RetentionDays)
|
||||||
|
}
|
||||||
|
|
||||||
// RetentionLabel returns the webhook's retention policy as display
|
// RetentionLabel returns the webhook's retention policy as display
|
||||||
// text, so that no template has to know about the sentinel value.
|
// text, so that no template has to know about the sentinel value.
|
||||||
func (w *Webhook) RetentionLabel() string {
|
func (w *Webhook) RetentionLabel() string {
|
||||||
|
|||||||
@@ -3,7 +3,7 @@ package database
|
|||||||
// Migrate runs database migrations for the main application database.
|
// Migrate runs database migrations for the main application database.
|
||||||
// Only configuration-tier models are stored in the main database.
|
// Only configuration-tier models are stored in the main database.
|
||||||
// Event-tier models (Event, Delivery, DeliveryResult, EventTotals,
|
// Event-tier models (Event, Delivery, DeliveryResult, EventTotals,
|
||||||
// TargetTotals) live in
|
// TargetTotals, EntrypointTotals) live in
|
||||||
// per-webhook dedicated databases managed by WebhookDBManager.
|
// per-webhook dedicated databases managed by WebhookDBManager.
|
||||||
func (d *Database) Migrate() error {
|
func (d *Database) Migrate() error {
|
||||||
return d.db.AutoMigrate(
|
return d.db.AutoMigrate(
|
||||||
|
|||||||
@@ -49,7 +49,7 @@ var ErrSidecarNotRemoved = errors.New(
|
|||||||
// WebhookDBManager manages per-webhook SQLite database files
|
// WebhookDBManager manages per-webhook SQLite database files
|
||||||
// for event storage. Each webhook gets its own dedicated
|
// for event storage. Each webhook gets its own dedicated
|
||||||
// database containing Events, Deliveries, DeliveryResults and the
|
// database containing Events, Deliveries, DeliveryResults and the
|
||||||
// running totals of them (EventTotals, TargetTotals).
|
// running totals of them (EventTotals, TargetTotals, EntrypointTotals).
|
||||||
// Database connections are opened lazily and cached.
|
// Database connections are opened lazily and cached.
|
||||||
type WebhookDBManager struct {
|
type WebhookDBManager struct {
|
||||||
dataDir string
|
dataDir string
|
||||||
@@ -381,7 +381,7 @@ func (m *WebhookDBManager) openDB(
|
|||||||
// Run migrations for event-tier models only
|
// Run migrations for event-tier models only
|
||||||
err = db.AutoMigrate(
|
err = db.AutoMigrate(
|
||||||
&Event{}, &Delivery{}, &DeliveryResult{},
|
&Event{}, &Delivery{}, &DeliveryResult{},
|
||||||
&EventTotals{}, &TargetTotals{},
|
&EventTotals{}, &TargetTotals{}, &EntrypointTotals{},
|
||||||
)
|
)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
_ = sqlDB.Close()
|
_ = sqlDB.Close()
|
||||||
|
|||||||
@@ -1,6 +1,11 @@
|
|||||||
package handlers
|
package handlers
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"fmt"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/dustin/go-humanize"
|
||||||
|
"gorm.io/gorm"
|
||||||
"sneak.berlin/go/webhooker/internal/database"
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -11,6 +16,21 @@ type EntrypointView struct {
|
|||||||
Path string
|
Path string
|
||||||
Description string
|
Description string
|
||||||
Active bool
|
Active bool
|
||||||
|
|
||||||
|
// Events is how many events arrived on the entrypoint's URL within
|
||||||
|
// the webhook's retention period. LastEvent is when the newest
|
||||||
|
// event ever to arrive on it did, relative, and LastEventUTC the
|
||||||
|
// full time; both are empty when none ever did.
|
||||||
|
Events int64
|
||||||
|
LastEvent string
|
||||||
|
LastEventUTC string
|
||||||
|
}
|
||||||
|
|
||||||
|
// entrypointEvents is one entrypoint's count read by
|
||||||
|
// addEntrypointEvents.
|
||||||
|
type entrypointEvents struct {
|
||||||
|
EntrypointID string
|
||||||
|
Events int64
|
||||||
}
|
}
|
||||||
|
|
||||||
// NewEntrypointViews projects entrypoints for rendering.
|
// NewEntrypointViews projects entrypoints for rendering.
|
||||||
@@ -32,3 +52,60 @@ func NewEntrypointViews(
|
|||||||
|
|
||||||
return views
|
return views
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// addEntrypointEvents fills in each view's event figures from the
|
||||||
|
// webhook's event database: when the last event arrived on its URL,
|
||||||
|
// from its EntrypointTotals row, and how many events arrived on it
|
||||||
|
// since the webhook's retention cutoff, counted in one query over the
|
||||||
|
// events' entrypoint_id index. Resubmitted copies did not arrive on
|
||||||
|
// the URL and are left out of both.
|
||||||
|
func addEntrypointEvents(
|
||||||
|
webhookDB *gorm.DB,
|
||||||
|
webhook *database.Webhook,
|
||||||
|
views []EntrypointView,
|
||||||
|
now time.Time,
|
||||||
|
) error {
|
||||||
|
ids := make([]string, len(views))
|
||||||
|
byID := make(map[string]*EntrypointView, len(views))
|
||||||
|
|
||||||
|
for i := range views {
|
||||||
|
ids[i] = views[i].ID
|
||||||
|
byID[views[i].ID] = &views[i]
|
||||||
|
}
|
||||||
|
|
||||||
|
var totals []database.EntrypointTotals
|
||||||
|
|
||||||
|
err := webhookDB.Where("entrypoint_id IN ?", ids).Find(&totals).Error
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("reading entrypoint totals: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
query := webhookDB.Model(&database.Event{}).
|
||||||
|
Select("entrypoint_id, count(*) AS events").
|
||||||
|
Where("entrypoint_id IN ? AND resubmitted_from_id IS NULL", ids)
|
||||||
|
|
||||||
|
cutoff, finite := webhook.RetentionCutoff(now)
|
||||||
|
if finite {
|
||||||
|
query = query.Where("created_at >= ?", cutoff)
|
||||||
|
}
|
||||||
|
|
||||||
|
var counts []entrypointEvents
|
||||||
|
|
||||||
|
err = query.Group("entrypoint_id").Find(&counts).Error
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("counting events by entrypoint: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, row := range totals {
|
||||||
|
view := byID[row.EntrypointID]
|
||||||
|
view.LastEvent = humanize.Time(row.LastEventAt)
|
||||||
|
view.LastEventUTC =
|
||||||
|
row.LastEventAt.UTC().Format(time.DateTime) + " UTC"
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, row := range counts {
|
||||||
|
byID[row.EntrypointID].Events = row.Events
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,197 @@
|
|||||||
|
package handlers_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/http"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/google/uuid"
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"gorm.io/gorm/clause"
|
||||||
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
|
"sneak.berlin/go/webhooker/internal/handlers"
|
||||||
|
"sneak.berlin/go/webhooker/internal/logger"
|
||||||
|
"sneak.berlin/go/webhooker/internal/session"
|
||||||
|
)
|
||||||
|
|
||||||
|
// entrypointRow returns the part of a rendered webhook page from an
|
||||||
|
// entrypoint's URL to the next entrypoint's, which holds its figures.
|
||||||
|
func entrypointRow(t *testing.T, page, entrypointID string) string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
_, row, found := strings.Cut(page, `id="entrypoint-url-`+entrypointID+`"`)
|
||||||
|
require.True(t, found)
|
||||||
|
|
||||||
|
row, _, _ = strings.Cut(row, `id="entrypoint-url-`)
|
||||||
|
|
||||||
|
return row
|
||||||
|
}
|
||||||
|
|
||||||
|
// lastEventShown matches an entrypoint row's last event arriving at at.
|
||||||
|
func lastEventShown(at time.Time) string {
|
||||||
|
return `Last Event:</span>\s*<span title="` +
|
||||||
|
at.UTC().Format(time.DateTime) + ` UTC">[^<]+</span>`
|
||||||
|
}
|
||||||
|
|
||||||
|
// eventsShown matches an entrypoint row's count of n events.
|
||||||
|
func eventsShown(n int) string {
|
||||||
|
return `Events Within Retention:</span>\s*<span>` +
|
||||||
|
strconv.Itoa(n) + `</span>`
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleSourceDetail_ShowsEntrypointEvents proves each entrypoint
|
||||||
|
// on the webhook page shows its own figures: how many events arrived
|
||||||
|
// through it within the webhook's retention period, leaving out one
|
||||||
|
// older than that, and when the newest arrived, or "never" for an
|
||||||
|
// entrypoint with none.
|
||||||
|
func TestHandleSourceDetail_ShowsEntrypointEvents(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
var (
|
||||||
|
h *handlers.Handlers
|
||||||
|
sess *session.Session
|
||||||
|
db *database.Database
|
||||||
|
dbMgr *database.WebhookDBManager
|
||||||
|
)
|
||||||
|
|
||||||
|
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||||
|
app.RequireStart()
|
||||||
|
|
||||||
|
t.Cleanup(app.RequireStop)
|
||||||
|
|
||||||
|
wh := &database.Webhook{
|
||||||
|
UserID: deleteTestUserID, Name: "figures", RetentionDays: 7,
|
||||||
|
}
|
||||||
|
require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
|
||||||
|
|
||||||
|
webhookDB, err := dbMgr.GetDB(wh.ID)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
entrypoint := func() *database.Entrypoint {
|
||||||
|
ep := &database.Entrypoint{
|
||||||
|
WebhookID: wh.ID, Path: uuid.New().String(), Active: true,
|
||||||
|
}
|
||||||
|
require.NoError(t,
|
||||||
|
db.DB().Omit(clause.Associations).Create(ep).Error)
|
||||||
|
|
||||||
|
return ep
|
||||||
|
}
|
||||||
|
|
||||||
|
// event stores an event that arrived on ep's URL age ago and
|
||||||
|
// records it as ep's last event, as the receiver does.
|
||||||
|
event := func(ep *database.Entrypoint, age time.Duration) time.Time {
|
||||||
|
e := &database.Event{
|
||||||
|
WebhookID: wh.ID,
|
||||||
|
EntrypointID: ep.ID,
|
||||||
|
Method: http.MethodPost,
|
||||||
|
}
|
||||||
|
e.CreatedAt = time.Now().Add(-age)
|
||||||
|
require.NoError(t,
|
||||||
|
webhookDB.Omit(clause.Associations).Create(e).Error)
|
||||||
|
require.NoError(t, database.AddEntrypointTotals(webhookDB,
|
||||||
|
database.EntrypointTotals{
|
||||||
|
EntrypointID: ep.ID, LastEventAt: e.CreatedAt,
|
||||||
|
}))
|
||||||
|
|
||||||
|
return e.CreatedAt
|
||||||
|
}
|
||||||
|
|
||||||
|
busy, quiet, unused := entrypoint(), entrypoint(), entrypoint()
|
||||||
|
|
||||||
|
event(busy, 8*24*time.Hour) // older than the 7 days kept
|
||||||
|
event(busy, 3*time.Hour)
|
||||||
|
busyLast := event(busy, time.Hour)
|
||||||
|
quietLast := event(quiet, 2*24*time.Hour)
|
||||||
|
|
||||||
|
body := renderSourceDetailPage(t, h, sess, wh.ID)
|
||||||
|
|
||||||
|
assert.Regexp(t, lastEventShown(busyLast), entrypointRow(t, body, busy.ID))
|
||||||
|
assert.Regexp(t, eventsShown(2), entrypointRow(t, body, busy.ID))
|
||||||
|
assert.Regexp(t, lastEventShown(quietLast), entrypointRow(t, body, quiet.ID))
|
||||||
|
assert.Regexp(t, eventsShown(1), entrypointRow(t, body, quiet.ID))
|
||||||
|
assert.Regexp(t, `Last Event:</span>\s*<span>never</span>`,
|
||||||
|
entrypointRow(t, body, unused.ID))
|
||||||
|
assert.Regexp(t, eventsShown(0), entrypointRow(t, body, unused.ID))
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleSourceDetail_EntrypointLastEventSurvivesRetention checks
|
||||||
|
// that once retention has removed every event that arrived on an
|
||||||
|
// entrypoint's URL, the entrypoint still shows when the last one
|
||||||
|
// arrived rather than "never".
|
||||||
|
func TestHandleSourceDetail_EntrypointLastEventSurvivesRetention(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
var (
|
||||||
|
h *handlers.Handlers
|
||||||
|
sess *session.Session
|
||||||
|
db *database.Database
|
||||||
|
dbMgr *database.WebhookDBManager
|
||||||
|
log *logger.Logger
|
||||||
|
)
|
||||||
|
|
||||||
|
app := newTestApp(t, &h, &sess, &db, &dbMgr, &log)
|
||||||
|
app.RequireStart()
|
||||||
|
|
||||||
|
t.Cleanup(app.RequireStop)
|
||||||
|
|
||||||
|
wh := &database.Webhook{
|
||||||
|
UserID: deleteTestUserID, Name: "swept", RetentionDays: 1,
|
||||||
|
}
|
||||||
|
require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
|
||||||
|
|
||||||
|
ep := seedEntrypoint(t, db, wh.ID)
|
||||||
|
webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, ep.Path, 1)
|
||||||
|
arrived := events[0].CreatedAt
|
||||||
|
|
||||||
|
statsAge(t, webhookDB, events[0].ID, time.Now().Add(-50*time.Hour))
|
||||||
|
statsPrune(t, db, dbMgr, log, webhookDB)
|
||||||
|
require.Empty(t, listEvents(t, webhookDB))
|
||||||
|
|
||||||
|
row := entrypointRow(t, renderSourceDetailPage(t, h, sess, wh.ID), ep.ID)
|
||||||
|
assert.Regexp(t, lastEventShown(arrived), row)
|
||||||
|
assert.Regexp(t, eventsShown(0), row)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleSourceDetail_ResubmitLeavesEntrypointFigures checks that a
|
||||||
|
// resubmitted copy, which did not arrive on the entrypoint's URL,
|
||||||
|
// changes neither the entrypoint's last event nor its count.
|
||||||
|
func TestHandleSourceDetail_ResubmitLeavesEntrypointFigures(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
var (
|
||||||
|
h *handlers.Handlers
|
||||||
|
sess *session.Session
|
||||||
|
db *database.Database
|
||||||
|
dbMgr *database.WebhookDBManager
|
||||||
|
)
|
||||||
|
|
||||||
|
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||||
|
app.RequireStart()
|
||||||
|
|
||||||
|
t.Cleanup(app.RequireStop)
|
||||||
|
|
||||||
|
wh := seedWebhook(t, db)
|
||||||
|
ep := seedEntrypoint(t, db, wh.ID)
|
||||||
|
webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, ep.Path, 1)
|
||||||
|
arrived := events[0].CreatedAt
|
||||||
|
|
||||||
|
require.Equal(t, http.StatusSeeOther,
|
||||||
|
postResubmit(t, h, sess, wh.ID, events[0].ID).Code)
|
||||||
|
require.Len(t, listEvents(t, webhookDB), 2)
|
||||||
|
|
||||||
|
var totals database.EntrypointTotals
|
||||||
|
|
||||||
|
require.NoError(t, webhookDB.Take(&totals).Error)
|
||||||
|
assert.True(t, arrived.Equal(totals.LastEventAt))
|
||||||
|
|
||||||
|
row := entrypointRow(t, renderSourceDetailPage(t, h, sess, wh.ID), ep.ID)
|
||||||
|
assert.Regexp(t, lastEventShown(arrived), row)
|
||||||
|
assert.Regexp(t, eventsShown(1), row)
|
||||||
|
}
|
||||||
@@ -485,6 +485,8 @@ func (h *Handlers) renderSourceDetail(
|
|||||||
"webhook_id = ?", webhook.ID,
|
"webhook_id = ?", webhook.ID,
|
||||||
).Find(&targets)
|
).Find(&targets)
|
||||||
|
|
||||||
|
entrypointViews := NewEntrypointViews(entrypoints)
|
||||||
|
|
||||||
var events []RecentEventView
|
var events []RecentEventView
|
||||||
|
|
||||||
if h.dbMgr.DBExists(webhook.ID) {
|
if h.dbMgr.DBExists(webhook.ID) {
|
||||||
@@ -503,6 +505,15 @@ func (h *Handlers) renderSourceDetail(
|
|||||||
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
err = addEntrypointEvents(
|
||||||
|
webhookDB, &webhook, entrypointViews, time.Now(),
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
h.serverError(w, r, "failed to count entrypoint events", err)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
scheme := "http"
|
scheme := "http"
|
||||||
@@ -523,7 +534,7 @@ func (h *Handlers) renderSourceDetail(
|
|||||||
// Targets are projected to a display-safe view: a
|
// Targets are projected to a display-safe view: a
|
||||||
// target's stored config blob holds a credential, and it
|
// target's stored config blob holds a credential, and it
|
||||||
// must never reach a template.
|
// must never reach a template.
|
||||||
"Entrypoints": NewEntrypointViews(entrypoints),
|
"Entrypoints": entrypointViews,
|
||||||
"Targets": h.targetRows(&webhook, targets),
|
"Targets": h.targetRows(&webhook, targets),
|
||||||
"Events": events,
|
"Events": events,
|
||||||
"BaseURL": baseURL,
|
"BaseURL": baseURL,
|
||||||
|
|||||||
@@ -326,6 +326,19 @@ func (h *Handlers) createAndFanOut(
|
|||||||
return nil, nil, err
|
return nil, nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A resubmitted copy did not arrive on its entrypoint's URL, so it
|
||||||
|
// leaves the entrypoint's last event as it is.
|
||||||
|
if src.ResubmittedFromID == nil {
|
||||||
|
err = database.AddEntrypointTotals(tx, database.EntrypointTotals{
|
||||||
|
EntrypointID: event.EntrypointID, LastEventAt: event.CreatedAt,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
tx.Rollback()
|
||||||
|
|
||||||
|
return nil, nil, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
err = tx.Commit().Error
|
err = tx.Commit().Error
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, nil, fmt.Errorf(
|
return nil, nil, fmt.Errorf(
|
||||||
|
|||||||
@@ -97,6 +97,18 @@
|
|||||||
</div>
|
</div>
|
||||||
<!-- The URL above is the entrypoint's credential:
|
<!-- The URL above is the entrypoint's credential:
|
||||||
anyone holding it can submit events. -->
|
anyone holding it can submit events. -->
|
||||||
|
<div class="text-xs text-gray-500 mt-1">
|
||||||
|
<span class="font-medium text-gray-700">Last Event:</span>
|
||||||
|
{{if .LastEvent}}
|
||||||
|
<span title="{{.LastEventUTC}}">{{.LastEvent}}</span>
|
||||||
|
{{else}}
|
||||||
|
<span>never</span>
|
||||||
|
{{end}}
|
||||||
|
</div>
|
||||||
|
<div class="text-xs text-gray-500 mt-1">
|
||||||
|
<span class="font-medium text-gray-700">Events Within Retention:</span>
|
||||||
|
<span>{{.Events}}</span>
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
{{else}}
|
{{else}}
|
||||||
<div class="p-4 text-sm text-gray-500">No entrypoints configured.</div>
|
<div class="p-4 text-sm text-gray-500">No entrypoints configured.</div>
|
||||||
|
|||||||
Reference in New Issue
Block a user