Show each entrypoint's last event and event count on the webhook page (closes #393) #470
@@ -1127,7 +1127,7 @@ unconditionally against whatever files it finds:
|
|||||||
- the main database on connect — `Setting`, `User`, `APIKey`, `Webhook`,
|
- the main database on connect — `Setting`, `User`, `APIKey`, `Webhook`,
|
||||||
`Entrypoint`, `Target`
|
`Entrypoint`, `Target`
|
||||||
- each event database when it is lazily opened — `Event`, `Delivery`,
|
- each event database when it is lazily opened — `Event`, `Delivery`,
|
||||||
`DeliveryResult`, `EventTotals`, `TargetTotals`
|
`DeliveryResult`, `EventTotals`, `TargetTotals`, `EntrypointTotals`
|
||||||
- each archive database on every open and reopen
|
- each archive database on every open and reopen
|
||||||
|
|
||||||
There is no schema version table, no migration ledger, and no down
|
There is no schema version table, no migration ledger, and no down
|
||||||
@@ -1531,6 +1531,9 @@ tier** (event ingestion, delivery, and logging).
|
|||||||
│ ┌──────────────┐ (one row per target: running counts │
|
│ ┌──────────────┐ (one row per target: running counts │
|
||||||
│ │ TargetTotals │ of its deliveries) │
|
│ │ TargetTotals │ of its deliveries) │
|
||||||
│ └──────────────┘ │
|
│ └──────────────┘ │
|
||||||
|
│ ┌──────────────────┐ (one row per entrypoint: when the │
|
||||||
|
│ │ EntrypointTotals │ last event arrived on its URL) │
|
||||||
|
│ └──────────────────┘ │
|
||||||
└─────────────────────────────────────────────────────────────┘
|
└─────────────────────────────────────────────────────────────┘
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -1644,6 +1647,11 @@ different event sources that all feed into the same processing pipeline
|
|||||||
(e.g., one entrypoint for GitHub, another for Stripe, both routing to
|
(e.g., one entrypoint for GitHub, another for Stripe, both routing to
|
||||||
the same targets).
|
the same targets).
|
||||||
|
|
||||||
|
The webhook page shows, for each entrypoint, when the last event arrived
|
||||||
|
on its URL, which retention leaves in place, or "never" if none ever has,
|
||||||
|
and how many events arrived on it within the webhook's retention period.
|
||||||
|
A resubmitted event did not arrive on the URL and counts in neither.
|
||||||
|
|
||||||
#### Target
|
#### Target
|
||||||
|
|
||||||
A delivery destination for events. Each target defines where and how
|
A delivery destination for events. Each target defines where and how
|
||||||
@@ -1853,10 +1861,11 @@ retries) is individually logged for full observability.
|
|||||||
|
|
||||||
**Relations:** Belongs to Delivery.
|
**Relations:** Belongs to Delivery.
|
||||||
|
|
||||||
#### EventTotals and TargetTotals
|
#### EventTotals, TargetTotals and EntrypointTotals
|
||||||
|
|
||||||
Running counts in each event database, read by the statistics pane at the
|
Running counts in each event database, read by the statistics pane at the
|
||||||
top of the webhook page and by the webhook list. `EventTotals` is one row:
|
top of the webhook page and by the webhook list, and each entrypoint's last
|
||||||
|
event, read by the webhook page's entrypoint list. `EventTotals` is one row:
|
||||||
|
|
||||||
| Field | Type | Description |
|
| Field | Type | Description |
|
||||||
| ---------------- | --------- | ----------- |
|
| ---------------- | --------- | ----------- |
|
||||||
@@ -1875,18 +1884,26 @@ top of the webhook page and by the webhook list. `EventTotals` is one row:
|
|||||||
| `deliveries_removed` | integer | Its deliveries retention has deleted |
|
| `deliveries_removed` | integer | Its deliveries retention has deleted |
|
||||||
| `failed_removed` | integer | Its failed deliveries retention has deleted |
|
| `failed_removed` | integer | Its failed deliveries retention has deleted |
|
||||||
|
|
||||||
Each count changes in the transaction that writes or deletes the rows it
|
`EntrypointTotals` is one row per entrypoint, created by the first event
|
||||||
counts. The pane's lifetime events are `events`, and its lifetime
|
that arrives on its URL:
|
||||||
deliveries and failures are `deliveries` and `failed` summed over the
|
|
||||||
targets; each figure within retention is the same less what retention
|
| Field | Type | Description |
|
||||||
removed, so neither needs the rows themselves. Its last event is
|
| --------------- | --------- | ----------- |
|
||||||
`last_event_at`, written in the transaction that stores the event, so it
|
| `entrypoint_id` | UUID | The entrypoint (primary key) |
|
||||||
still shows once retention has removed every event. Its last-10-minutes and
|
| `last_event_at` | timestamp | When the newest event arrived on its URL; a resubmitted event leaves it as it is, and so does retention |
|
||||||
last-24-hours figures are counted from the `events` and `deliveries`
|
|
||||||
indexes over just that window, the deliveries in one query grouped by
|
Each count changes in the transaction that writes or deletes the rows it counts,
|
||||||
target. Its failure percentage for a window is the deliveries that became
|
and each `last_event_at` in the transaction that stores the event. The pane's
|
||||||
`failed` in it out of all that became `delivered` or `failed` in it, and
|
lifetime events are `events`, and its lifetime deliveries and failures are
|
||||||
a dash when none did.
|
`deliveries` and `failed` summed over the targets; each figure within retention
|
||||||
|
is the same less what retention removed, so neither needs the rows themselves.
|
||||||
|
Its last event is `last_event_at` in `EventTotals`, so it still shows once
|
||||||
|
retention has removed every event; each entrypoint's last event, from
|
||||||
|
`EntrypointTotals`, does too. Its last-10-minutes and last-24-hours figures are
|
||||||
|
counted from the `events` and `deliveries` indexes over just that window, the
|
||||||
|
deliveries in one query grouped by target. Its failure percentage for a window
|
||||||
|
is the deliveries that became `failed` in it out of all that became `delivered`
|
||||||
|
or `failed` in it, and a dash when none did.
|
||||||
|
|
||||||
The webhook list at `/hooks` shows three of the pane's figures for each
|
The webhook list at `/hooks` shows three of the pane's figures for each
|
||||||
webhook: its events within retention and its last event, both from
|
webhook: its events within retention and its last event, both from
|
||||||
@@ -1908,26 +1925,31 @@ tags, so `AutoMigrate` creates them on a fresh database:
|
|||||||
| `delivery_results` | `delivery_id`, `deleted_at` | The event log, which loads the attempts of a page's deliveries, and retention, which deletes the attempts of expired events |
|
| `delivery_results` | `delivery_id`, `deleted_at` | The event log, which loads the attempts of a page's deliveries, and retention, which deletes the attempts of expired events |
|
||||||
| `events` | `deleted_at`, `created_at` | The webhook page's statistics, which count recent events |
|
| `events` | `deleted_at`, `created_at` | The webhook page's statistics, which count recent events |
|
||||||
| `events` | `resubmitted_from_id`, `deleted_at` | The event log, which counts the events resubmitted from each event on a page |
|
| `events` | `resubmitted_from_id`, `deleted_at` | The event log, which counts the events resubmitted from each event on a page |
|
||||||
|
| `events` | `entrypoint_id`, `deleted_at`, `resubmitted_from_id`, `created_at` | The webhook page's entrypoint list, which counts the events that arrived on each entrypoint's URL within the retention period |
|
||||||
| `events` | `created_at` | Retention, which selects expired events by age |
|
| `events` | `created_at` | Retention, which selects expired events by age |
|
||||||
|
|
||||||
GORM's soft delete adds `deleted_at IS NULL` to these queries; retention
|
GORM's soft delete adds `deleted_at IS NULL` to these queries; retention leaves
|
||||||
leaves it out. SQLite keeps no statistics on these tables, and without them it
|
it out. SQLite keeps no statistics on these tables, and without them it rates
|
||||||
rates the `deleted_at` index, which every live row matches, above an index on
|
the `deleted_at` index, which every live row matches, above an index on a column
|
||||||
a column matched against several values or compared with a range. So every
|
matched against several values or compared with a range. So every index but the
|
||||||
index but the last also covers `deleted_at`. It comes second in the `event_id`
|
last also covers `deleted_at`. It comes second in the `event_id` and
|
||||||
and `delivery_id` indexes, so that retention can use them without it. The event
|
`delivery_id` indexes, so that retention can use them without it. The event
|
||||||
log's count, the one query on the `resubmitted_from_id` index, always carries
|
log's count, the one query on the `resubmitted_from_id` index, always carries
|
||||||
`deleted_at IS NULL` and uses both columns. In the statistics' `events` index
|
`deleted_at IS NULL` and uses both columns. The entrypoint list's count, the one
|
||||||
`deleted_at` comes first, because they compare `created_at` with a range (`>=`)
|
query on the `entrypoint_id` index, uses all four, `resubmitted_from_id IS NULL`
|
||||||
and SQLite narrows by a range only on the last column it uses.
|
leaving out resubmitted copies and `created_at` last because it compares it with
|
||||||
|
a range (`>=`). In the statistics' `events` index `deleted_at` comes first,
|
||||||
|
because they compare `created_at` with a range (`>=`) and SQLite narrows by a
|
||||||
|
range only on the last column it uses.
|
||||||
|
|
||||||
#### Common Fields
|
#### Common Fields
|
||||||
|
|
||||||
Every entity except `Setting`, `EventTotals` and `TargetTotals` includes
|
Every entity except `Setting`, `EventTotals`, `TargetTotals` and
|
||||||
these fields from `BaseModel`. `Setting` is a bare key-value row with no
|
`EntrypointTotals` includes these fields from `BaseModel`. `Setting` is a bare
|
||||||
`id`, no timestamps and no soft delete, and the two totals tables hold
|
key-value row with no `id`, no timestamps and no soft delete. Of the three
|
||||||
counts, plus `last_event_at` in `event_totals`, keyed by a numeric `id`
|
totals tables, `event_totals` holds counts and `last_event_at`, keyed by a
|
||||||
and by `target_id`:
|
numeric `id`; `target_totals` holds counts, keyed by `target_id`; and
|
||||||
|
`entrypoint_totals` holds `last_event_at`, keyed by `entrypoint_id`:
|
||||||
|
|
||||||
| Field | Type | Description |
|
| Field | Type | Description |
|
||||||
| ------------ | --------- | ----------- |
|
| ------------ | --------- | ----------- |
|
||||||
@@ -1969,8 +1991,9 @@ encryption key is generated and stored, and an `admin` user is created.
|
|||||||
- **Events** — captured incoming webhook payloads
|
- **Events** — captured incoming webhook payloads
|
||||||
- **Deliveries** — event-to-target pairings and their status
|
- **Deliveries** — event-to-target pairings and their status
|
||||||
- **DeliveryResults** — individual delivery attempt logs
|
- **DeliveryResults** — individual delivery attempt logs
|
||||||
- **EventTotals** and **TargetTotals** — running counts of the above,
|
- **EventTotals**, **TargetTotals** and **EntrypointTotals** — running
|
||||||
the deliveries per target, kept through retention
|
counts of the above, the deliveries per target, and each entrypoint's
|
||||||
|
last event, kept through retention
|
||||||
|
|
||||||
Per-webhook databases are created automatically when a webhook is
|
Per-webhook databases are created automatically when a webhook is
|
||||||
created. They are managed by the `WebhookDBManager` component, which
|
created. They are managed by the `WebhookDBManager` component, which
|
||||||
@@ -3035,7 +3058,7 @@ webhooker/
|
|||||||
│ │ ├── model_event.go # Event entity (per-webhook DB)
|
│ │ ├── model_event.go # Event entity (per-webhook DB)
|
||||||
│ │ ├── model_delivery.go # Delivery entity (per-webhook DB)
|
│ │ ├── model_delivery.go # Delivery entity (per-webhook DB)
|
||||||
│ │ ├── model_delivery_result.go # DeliveryResult entity (per-webhook DB)
|
│ │ ├── model_delivery_result.go # DeliveryResult entity (per-webhook DB)
|
||||||
│ │ ├── model_totals.go # EventTotals and TargetTotals (per-webhook DB)
|
│ │ ├── model_totals.go # EventTotals, TargetTotals and EntrypointTotals (per-webhook DB)
|
||||||
│ │ ├── model_apikey.go # APIKey entity
|
│ │ ├── model_apikey.go # APIKey entity
|
||||||
│ │ ├── password.go # Argon2id hashing and verification
|
│ │ ├── password.go # Argon2id hashing and verification
|
||||||
│ │ ├── retention.go # Retention reaper (per-webhook event expiry)
|
│ │ ├── retention.go # Retention reaper (per-webhook event expiry)
|
||||||
@@ -3311,9 +3334,9 @@ check, see [The login endpoint](#the-login-endpoint).
|
|||||||
`ENTRYPOINT` script, which sets the data directory's owner and mode
|
`ENTRYPOINT` script, which sets the data directory's owner and mode
|
||||||
before the app starts; the image's health check; and `docker exec`,
|
before the app starts; the image's health check; and `docker exec`,
|
||||||
unless given `--user`
|
unless given `--user`
|
||||||
- GORM soft deletes on every entity that carries `BaseModel`, which is
|
- GORM soft deletes on every entity that carries `BaseModel`, which is all of
|
||||||
all of them but `Setting`, `EventTotals` and `TargetTotals` (data
|
them but `Setting`, `EventTotals`, `TargetTotals` and `EntrypointTotals`
|
||||||
preserved for audit)
|
(data preserved for audit)
|
||||||
|
|
||||||
### Shutdown
|
### Shutdown
|
||||||
|
|
||||||
|
|||||||
@@ -233,6 +233,43 @@ func TestResubmitCountUsesItsIndex(t *testing.T) {
|
|||||||
"(resubmitted_from_id=? AND deleted_at=?)")
|
"(resubmitted_from_id=? AND deleted_at=?)")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestEntrypointEventsUseTheirIndex does the same for the webhook
|
||||||
|
// page's count, for each entrypoint, of the events that arrived on its
|
||||||
|
// URL since the retention cutoff (addEntrypointEvents in the
|
||||||
|
// handlers), which must come from the index alone. It passes 25
|
||||||
|
// entrypoints, as TestResubmitCountUsesItsIndex passes 25 events.
|
||||||
|
func TestEntrypointEventsUseTheirIndex(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
mgr, lc := setupTestWebhookDBManager(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
require.NoError(t, lc.Start(ctx))
|
||||||
|
|
||||||
|
defer func() { require.NoError(t, lc.Stop(ctx)) }()
|
||||||
|
|
||||||
|
db, err := mgr.GetDB(uuid.New().String())
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
dry := db.Session(&gorm.Session{DryRun: true})
|
||||||
|
|
||||||
|
entrypoints := make([]string, 25)
|
||||||
|
for i := range entrypoints {
|
||||||
|
entrypoints[i] = uuid.New().String()
|
||||||
|
}
|
||||||
|
|
||||||
|
var rows []struct{ Events int }
|
||||||
|
|
||||||
|
assertPlanUses(t, db, dry.Model(&database.Event{}).
|
||||||
|
Select("entrypoint_id, count(*) AS events").
|
||||||
|
Where("entrypoint_id IN ? AND resubmitted_from_id IS NULL",
|
||||||
|
entrypoints).
|
||||||
|
Where("created_at >= ?", time.Now()).
|
||||||
|
Group("entrypoint_id").Find(&rows),
|
||||||
|
"COVERING INDEX idx_events_entrypoint_id "+
|
||||||
|
"(entrypoint_id=? AND deleted_at=? AND "+
|
||||||
|
"resubmitted_from_id=? AND created_at>?)")
|
||||||
|
}
|
||||||
|
|
||||||
// assertPlanUses asserts that SQLite's plan for a statement GORM built
|
// assertPlanUses asserts that SQLite's plan for a statement GORM built
|
||||||
// in a dry run, run with the same SQL and arguments GORM would send,
|
// in a dry run, run with the same SQL and arguments GORM would send,
|
||||||
// names each of the given indexes.
|
// names each of the given indexes.
|
||||||
|
|||||||
@@ -20,12 +20,15 @@ type Event struct {
|
|||||||
// has no deleted_at condition and uses the index on created_at
|
// has no deleted_at condition and uses the index on created_at
|
||||||
// alone. The other tables keep the unindexed BaseModel created_at.
|
// alone. The other tables keep the unindexed BaseModel created_at.
|
||||||
// DeletedAt is also the second column of the resubmitted_from_id
|
// DeletedAt is also the second column of the resubmitted_from_id
|
||||||
// index, for the reason DeliveryResult gives.
|
// index, for the reason DeliveryResult gives. The entrypoint_id
|
||||||
CreatedAt time.Time `gorm:"index;index:idx_events_deleted_at_created_at,priority:2" json:"createdAt"`
|
// index, for the webhook page's entrypoint list, has it second too,
|
||||||
DeletedAt gorm.DeletedAt `gorm:"index:idx_events_deleted_at_created_at,priority:1;index:idx_events_resubmitted_from_id,priority:2" json:"deletedAt,omitzero"`
|
// resubmitted_from_id third, and created_at last, which the list
|
||||||
|
// compares with a range.
|
||||||
|
CreatedAt time.Time `gorm:"index;index:idx_events_deleted_at_created_at,priority:2;index:idx_events_entrypoint_id,priority:4" json:"createdAt"`
|
||||||
|
DeletedAt gorm.DeletedAt `gorm:"index:idx_events_deleted_at_created_at,priority:1;index:idx_events_resubmitted_from_id,priority:2;index:idx_events_entrypoint_id,priority:2" json:"deletedAt,omitzero"`
|
||||||
|
|
||||||
WebhookID string `gorm:"type:uuid;not null" json:"webhookId"`
|
WebhookID string `gorm:"type:uuid;not null" json:"webhookId"`
|
||||||
EntrypointID string `gorm:"type:uuid;not null" json:"entrypointId"`
|
EntrypointID string `gorm:"type:uuid;not null;index:idx_events_entrypoint_id,priority:1" json:"entrypointId"`
|
||||||
|
|
||||||
// Request data
|
// Request data
|
||||||
Method string `gorm:"not null" json:"method"`
|
Method string `gorm:"not null" json:"method"`
|
||||||
@@ -44,7 +47,7 @@ type Event struct {
|
|||||||
// existed. It is not a foreign key: the source event can be
|
// existed. It is not a foreign key: the source event can be
|
||||||
// reaped by retention while its copies remain, and the id is
|
// reaped by retention while its copies remain, and the id is
|
||||||
// kept as the record of where the copy came from either way.
|
// kept as the record of where the copy came from either way.
|
||||||
ResubmittedFromID *string `gorm:"type:uuid;index:idx_events_resubmitted_from_id,priority:1" json:"resubmittedFromId,omitempty"`
|
ResubmittedFromID *string `gorm:"type:uuid;index:idx_events_resubmitted_from_id,priority:1;index:idx_events_entrypoint_id,priority:3" json:"resubmittedFromId,omitempty"`
|
||||||
|
|
||||||
// Relations. No model marshals the record it belongs to, so
|
// Relations. No model marshals the record it belongs to, so
|
||||||
// Webhook and Entrypoint are left out of the JSON.
|
// Webhook and Entrypoint are left out of the JSON.
|
||||||
|
|||||||
@@ -52,6 +52,21 @@ func (TargetTotals) TableName() string {
|
|||||||
return "target_totals"
|
return "target_totals"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// EntrypointTotals is one row per entrypoint, created by the first
|
||||||
|
// event that arrives on its URL: when the newest such event arrived,
|
||||||
|
// which retention leaves as it is. A resubmitted copy did not arrive
|
||||||
|
// on the URL and does not change it.
|
||||||
|
type EntrypointTotals struct {
|
||||||
|
EntrypointID string `gorm:"type:uuid;primaryKey"`
|
||||||
|
|
||||||
|
LastEventAt time.Time `gorm:"not null"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// TableName names the table AddEntrypointTotals updates.
|
||||||
|
func (EntrypointTotals) TableName() string {
|
||||||
|
return "entrypoint_totals"
|
||||||
|
}
|
||||||
|
|
||||||
// AddEventTotals adds each count in add to the webhook's event totals,
|
// AddEventTotals adds each count in add to the webhook's event totals,
|
||||||
// and records add.LastEventAt as when the newest event arrived if it is
|
// and records add.LastEventAt as when the newest event arrived if it is
|
||||||
// set. Call it on the transaction that writes or deletes the events it
|
// set. Call it on the transaction that writes or deletes the events it
|
||||||
@@ -97,3 +112,25 @@ func AddTargetTotals(tx *gorm.DB, add TargetTotals) error {
|
|||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// AddEntrypointTotals records add.LastEventAt as when the newest event
|
||||||
|
// arrived on the URL of the entrypoint add.EntrypointID names, creating
|
||||||
|
// its row the first time. Call it on the transaction that stores the
|
||||||
|
// event.
|
||||||
|
func AddEntrypointTotals(tx *gorm.DB, add EntrypointTotals) error {
|
||||||
|
err := tx.Exec(
|
||||||
|
`INSERT INTO entrypoint_totals (entrypoint_id, last_event_at)
|
||||||
|
VALUES (?, ?)
|
||||||
|
ON CONFLICT (entrypoint_id) DO UPDATE SET
|
||||||
|
last_event_at = excluded.last_event_at`,
|
||||||
|
add.EntrypointID, add.LastEventAt,
|
||||||
|
).Error
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf(
|
||||||
|
"adding to totals of entrypoint %s: %w",
|
||||||
|
add.EntrypointID, err,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|||||||
@@ -111,6 +111,13 @@ func (w *Webhook) RetainsForever() bool {
|
|||||||
return retainsForever(w.RetentionDays)
|
return retainsForever(w.RetentionDays)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// RetentionCutoff returns the time before which this webhook's events
|
||||||
|
// have expired, as the reaper computes it, and false when the webhook
|
||||||
|
// retains them forever.
|
||||||
|
func (w *Webhook) RetentionCutoff(now time.Time) (time.Time, bool) {
|
||||||
|
return retentionCutoff(now, w.RetentionDays)
|
||||||
|
}
|
||||||
|
|
||||||
// RetentionLabel returns the webhook's retention policy as display
|
// RetentionLabel returns the webhook's retention policy as display
|
||||||
// text, so that no template has to know about the sentinel value.
|
// text, so that no template has to know about the sentinel value.
|
||||||
func (w *Webhook) RetentionLabel() string {
|
func (w *Webhook) RetentionLabel() string {
|
||||||
|
|||||||
@@ -3,7 +3,7 @@ package database
|
|||||||
// Migrate runs database migrations for the main application database.
|
// Migrate runs database migrations for the main application database.
|
||||||
// Only configuration-tier models are stored in the main database.
|
// Only configuration-tier models are stored in the main database.
|
||||||
// Event-tier models (Event, Delivery, DeliveryResult, EventTotals,
|
// Event-tier models (Event, Delivery, DeliveryResult, EventTotals,
|
||||||
// TargetTotals) live in
|
// TargetTotals, EntrypointTotals) live in
|
||||||
// per-webhook dedicated databases managed by WebhookDBManager.
|
// per-webhook dedicated databases managed by WebhookDBManager.
|
||||||
func (d *Database) Migrate() error {
|
func (d *Database) Migrate() error {
|
||||||
return d.db.AutoMigrate(
|
return d.db.AutoMigrate(
|
||||||
|
|||||||
@@ -49,7 +49,7 @@ var ErrSidecarNotRemoved = errors.New(
|
|||||||
// WebhookDBManager manages per-webhook SQLite database files
|
// WebhookDBManager manages per-webhook SQLite database files
|
||||||
// for event storage. Each webhook gets its own dedicated
|
// for event storage. Each webhook gets its own dedicated
|
||||||
// database containing Events, Deliveries, DeliveryResults and the
|
// database containing Events, Deliveries, DeliveryResults and the
|
||||||
// running totals of them (EventTotals, TargetTotals).
|
// running totals of them (EventTotals, TargetTotals, EntrypointTotals).
|
||||||
// Database connections are opened lazily and cached.
|
// Database connections are opened lazily and cached.
|
||||||
type WebhookDBManager struct {
|
type WebhookDBManager struct {
|
||||||
dataDir string
|
dataDir string
|
||||||
@@ -381,7 +381,7 @@ func (m *WebhookDBManager) openDB(
|
|||||||
// Run migrations for event-tier models only
|
// Run migrations for event-tier models only
|
||||||
err = db.AutoMigrate(
|
err = db.AutoMigrate(
|
||||||
&Event{}, &Delivery{}, &DeliveryResult{},
|
&Event{}, &Delivery{}, &DeliveryResult{},
|
||||||
&EventTotals{}, &TargetTotals{},
|
&EventTotals{}, &TargetTotals{}, &EntrypointTotals{},
|
||||||
)
|
)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
_ = sqlDB.Close()
|
_ = sqlDB.Close()
|
||||||
|
|||||||
@@ -1,6 +1,11 @@
|
|||||||
package handlers
|
package handlers
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"fmt"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/dustin/go-humanize"
|
||||||
|
"gorm.io/gorm"
|
||||||
"sneak.berlin/go/webhooker/internal/database"
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -11,6 +16,21 @@ type EntrypointView struct {
|
|||||||
Path string
|
Path string
|
||||||
Description string
|
Description string
|
||||||
Active bool
|
Active bool
|
||||||
|
|
||||||
|
// Events is how many events arrived on the entrypoint's URL within
|
||||||
|
// the webhook's retention period. LastEvent is when the newest
|
||||||
|
// event ever to arrive on it did, relative, and LastEventUTC the
|
||||||
|
// full time; both are empty when none ever did.
|
||||||
|
Events int64
|
||||||
|
LastEvent string
|
||||||
|
LastEventUTC string
|
||||||
|
}
|
||||||
|
|
||||||
|
// entrypointEvents is one entrypoint's count read by
|
||||||
|
// addEntrypointEvents.
|
||||||
|
type entrypointEvents struct {
|
||||||
|
EntrypointID string
|
||||||
|
Events int64
|
||||||
}
|
}
|
||||||
|
|
||||||
// NewEntrypointViews projects entrypoints for rendering.
|
// NewEntrypointViews projects entrypoints for rendering.
|
||||||
@@ -32,3 +52,60 @@ func NewEntrypointViews(
|
|||||||
|
|
||||||
return views
|
return views
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// addEntrypointEvents fills in each view's event figures from the
|
||||||
|
// webhook's event database: when the last event arrived on its URL,
|
||||||
|
// from its EntrypointTotals row, and how many events arrived on it
|
||||||
|
// since the webhook's retention cutoff, counted in one query over the
|
||||||
|
// events' entrypoint_id index. Resubmitted copies did not arrive on
|
||||||
|
// the URL and are left out of both.
|
||||||
|
func addEntrypointEvents(
|
||||||
|
webhookDB *gorm.DB,
|
||||||
|
webhook *database.Webhook,
|
||||||
|
views []EntrypointView,
|
||||||
|
now time.Time,
|
||||||
|
) error {
|
||||||
|
ids := make([]string, len(views))
|
||||||
|
byID := make(map[string]*EntrypointView, len(views))
|
||||||
|
|
||||||
|
for i := range views {
|
||||||
|
ids[i] = views[i].ID
|
||||||
|
byID[views[i].ID] = &views[i]
|
||||||
|
}
|
||||||
|
|
||||||
|
var totals []database.EntrypointTotals
|
||||||
|
|
||||||
|
err := webhookDB.Where("entrypoint_id IN ?", ids).Find(&totals).Error
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("reading entrypoint totals: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
query := webhookDB.Model(&database.Event{}).
|
||||||
|
Select("entrypoint_id, count(*) AS events").
|
||||||
|
Where("entrypoint_id IN ? AND resubmitted_from_id IS NULL", ids)
|
||||||
|
|
||||||
|
cutoff, finite := webhook.RetentionCutoff(now)
|
||||||
|
if finite {
|
||||||
|
query = query.Where("created_at >= ?", cutoff)
|
||||||
|
}
|
||||||
|
|
||||||
|
var counts []entrypointEvents
|
||||||
|
|
||||||
|
err = query.Group("entrypoint_id").Find(&counts).Error
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("counting events by entrypoint: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, row := range totals {
|
||||||
|
view := byID[row.EntrypointID]
|
||||||
|
view.LastEvent = humanize.Time(row.LastEventAt)
|
||||||
|
view.LastEventUTC =
|
||||||
|
row.LastEventAt.UTC().Format(time.DateTime) + " UTC"
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, row := range counts {
|
||||||
|
byID[row.EntrypointID].Events = row.Events
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,197 @@
|
|||||||
|
package handlers_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/http"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/google/uuid"
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"gorm.io/gorm/clause"
|
||||||
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
|
"sneak.berlin/go/webhooker/internal/handlers"
|
||||||
|
"sneak.berlin/go/webhooker/internal/logger"
|
||||||
|
"sneak.berlin/go/webhooker/internal/session"
|
||||||
|
)
|
||||||
|
|
||||||
|
// entrypointRow returns the part of a rendered webhook page from an
|
||||||
|
// entrypoint's URL to the next entrypoint's, which holds its figures.
|
||||||
|
func entrypointRow(t *testing.T, page, entrypointID string) string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
_, row, found := strings.Cut(page, `id="entrypoint-url-`+entrypointID+`"`)
|
||||||
|
require.True(t, found)
|
||||||
|
|
||||||
|
row, _, _ = strings.Cut(row, `id="entrypoint-url-`)
|
||||||
|
|
||||||
|
return row
|
||||||
|
}
|
||||||
|
|
||||||
|
// lastEventShown matches an entrypoint row's last event arriving at at.
|
||||||
|
func lastEventShown(at time.Time) string {
|
||||||
|
return `Last Event:</span>\s*<span title="` +
|
||||||
|
at.UTC().Format(time.DateTime) + ` UTC">[^<]+</span>`
|
||||||
|
}
|
||||||
|
|
||||||
|
// eventsShown matches an entrypoint row's count of n events.
|
||||||
|
func eventsShown(n int) string {
|
||||||
|
return `Events Within Retention:</span>\s*<span>` +
|
||||||
|
strconv.Itoa(n) + `</span>`
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleSourceDetail_ShowsEntrypointEvents proves each entrypoint
|
||||||
|
// on the webhook page shows its own figures: how many events arrived
|
||||||
|
// through it within the webhook's retention period, leaving out one
|
||||||
|
// older than that, and when the newest arrived, or "never" for an
|
||||||
|
// entrypoint with none.
|
||||||
|
func TestHandleSourceDetail_ShowsEntrypointEvents(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
var (
|
||||||
|
h *handlers.Handlers
|
||||||
|
sess *session.Session
|
||||||
|
db *database.Database
|
||||||
|
dbMgr *database.WebhookDBManager
|
||||||
|
)
|
||||||
|
|
||||||
|
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||||
|
app.RequireStart()
|
||||||
|
|
||||||
|
t.Cleanup(app.RequireStop)
|
||||||
|
|
||||||
|
wh := &database.Webhook{
|
||||||
|
UserID: deleteTestUserID, Name: "figures", RetentionDays: 7,
|
||||||
|
}
|
||||||
|
require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
|
||||||
|
|
||||||
|
webhookDB, err := dbMgr.GetDB(wh.ID)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
entrypoint := func() *database.Entrypoint {
|
||||||
|
ep := &database.Entrypoint{
|
||||||
|
WebhookID: wh.ID, Path: uuid.New().String(), Active: true,
|
||||||
|
}
|
||||||
|
require.NoError(t,
|
||||||
|
db.DB().Omit(clause.Associations).Create(ep).Error)
|
||||||
|
|
||||||
|
return ep
|
||||||
|
}
|
||||||
|
|
||||||
|
// event stores an event that arrived on ep's URL age ago and
|
||||||
|
// records it as ep's last event, as the receiver does.
|
||||||
|
event := func(ep *database.Entrypoint, age time.Duration) time.Time {
|
||||||
|
e := &database.Event{
|
||||||
|
WebhookID: wh.ID,
|
||||||
|
EntrypointID: ep.ID,
|
||||||
|
Method: http.MethodPost,
|
||||||
|
}
|
||||||
|
e.CreatedAt = time.Now().Add(-age)
|
||||||
|
require.NoError(t,
|
||||||
|
webhookDB.Omit(clause.Associations).Create(e).Error)
|
||||||
|
require.NoError(t, database.AddEntrypointTotals(webhookDB,
|
||||||
|
database.EntrypointTotals{
|
||||||
|
EntrypointID: ep.ID, LastEventAt: e.CreatedAt,
|
||||||
|
}))
|
||||||
|
|
||||||
|
return e.CreatedAt
|
||||||
|
}
|
||||||
|
|
||||||
|
busy, quiet, unused := entrypoint(), entrypoint(), entrypoint()
|
||||||
|
|
||||||
|
event(busy, 8*24*time.Hour) // older than the 7 days kept
|
||||||
|
event(busy, 3*time.Hour)
|
||||||
|
busyLast := event(busy, time.Hour)
|
||||||
|
quietLast := event(quiet, 2*24*time.Hour)
|
||||||
|
|
||||||
|
body := renderSourceDetailPage(t, h, sess, wh.ID)
|
||||||
|
|
||||||
|
assert.Regexp(t, lastEventShown(busyLast), entrypointRow(t, body, busy.ID))
|
||||||
|
assert.Regexp(t, eventsShown(2), entrypointRow(t, body, busy.ID))
|
||||||
|
assert.Regexp(t, lastEventShown(quietLast), entrypointRow(t, body, quiet.ID))
|
||||||
|
assert.Regexp(t, eventsShown(1), entrypointRow(t, body, quiet.ID))
|
||||||
|
assert.Regexp(t, `Last Event:</span>\s*<span>never</span>`,
|
||||||
|
entrypointRow(t, body, unused.ID))
|
||||||
|
assert.Regexp(t, eventsShown(0), entrypointRow(t, body, unused.ID))
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleSourceDetail_EntrypointLastEventSurvivesRetention checks
|
||||||
|
// that once retention has removed every event that arrived on an
|
||||||
|
// entrypoint's URL, the entrypoint still shows when the last one
|
||||||
|
// arrived rather than "never".
|
||||||
|
func TestHandleSourceDetail_EntrypointLastEventSurvivesRetention(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
var (
|
||||||
|
h *handlers.Handlers
|
||||||
|
sess *session.Session
|
||||||
|
db *database.Database
|
||||||
|
dbMgr *database.WebhookDBManager
|
||||||
|
log *logger.Logger
|
||||||
|
)
|
||||||
|
|
||||||
|
app := newTestApp(t, &h, &sess, &db, &dbMgr, &log)
|
||||||
|
app.RequireStart()
|
||||||
|
|
||||||
|
t.Cleanup(app.RequireStop)
|
||||||
|
|
||||||
|
wh := &database.Webhook{
|
||||||
|
UserID: deleteTestUserID, Name: "swept", RetentionDays: 1,
|
||||||
|
}
|
||||||
|
require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
|
||||||
|
|
||||||
|
ep := seedEntrypoint(t, db, wh.ID)
|
||||||
|
webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, ep.Path, 1)
|
||||||
|
arrived := events[0].CreatedAt
|
||||||
|
|
||||||
|
statsAge(t, webhookDB, events[0].ID, time.Now().Add(-50*time.Hour))
|
||||||
|
statsPrune(t, db, dbMgr, log, webhookDB)
|
||||||
|
require.Empty(t, listEvents(t, webhookDB))
|
||||||
|
|
||||||
|
row := entrypointRow(t, renderSourceDetailPage(t, h, sess, wh.ID), ep.ID)
|
||||||
|
assert.Regexp(t, lastEventShown(arrived), row)
|
||||||
|
assert.Regexp(t, eventsShown(0), row)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleSourceDetail_ResubmitLeavesEntrypointFigures checks that a
|
||||||
|
// resubmitted copy, which did not arrive on the entrypoint's URL,
|
||||||
|
// changes neither the entrypoint's last event nor its count.
|
||||||
|
func TestHandleSourceDetail_ResubmitLeavesEntrypointFigures(
|
||||||
|
t *testing.T,
|
||||||
|
) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
var (
|
||||||
|
h *handlers.Handlers
|
||||||
|
sess *session.Session
|
||||||
|
db *database.Database
|
||||||
|
dbMgr *database.WebhookDBManager
|
||||||
|
)
|
||||||
|
|
||||||
|
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||||
|
app.RequireStart()
|
||||||
|
|
||||||
|
t.Cleanup(app.RequireStop)
|
||||||
|
|
||||||
|
wh := seedWebhook(t, db)
|
||||||
|
ep := seedEntrypoint(t, db, wh.ID)
|
||||||
|
webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, ep.Path, 1)
|
||||||
|
arrived := events[0].CreatedAt
|
||||||
|
|
||||||
|
require.Equal(t, http.StatusSeeOther,
|
||||||
|
postResubmit(t, h, sess, wh.ID, events[0].ID).Code)
|
||||||
|
require.Len(t, listEvents(t, webhookDB), 2)
|
||||||
|
|
||||||
|
var totals database.EntrypointTotals
|
||||||
|
|
||||||
|
require.NoError(t, webhookDB.Take(&totals).Error)
|
||||||
|
assert.True(t, arrived.Equal(totals.LastEventAt))
|
||||||
|
|
||||||
|
row := entrypointRow(t, renderSourceDetailPage(t, h, sess, wh.ID), ep.ID)
|
||||||
|
assert.Regexp(t, lastEventShown(arrived), row)
|
||||||
|
assert.Regexp(t, eventsShown(1), row)
|
||||||
|
}
|
||||||
@@ -485,6 +485,8 @@ func (h *Handlers) renderSourceDetail(
|
|||||||
"webhook_id = ?", webhook.ID,
|
"webhook_id = ?", webhook.ID,
|
||||||
).Find(&targets)
|
).Find(&targets)
|
||||||
|
|
||||||
|
entrypointViews := NewEntrypointViews(entrypoints)
|
||||||
|
|
||||||
var events []RecentEventView
|
var events []RecentEventView
|
||||||
|
|
||||||
if h.dbMgr.DBExists(webhook.ID) {
|
if h.dbMgr.DBExists(webhook.ID) {
|
||||||
@@ -503,6 +505,15 @@ func (h *Handlers) renderSourceDetail(
|
|||||||
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
err = addEntrypointEvents(
|
||||||
|
webhookDB, &webhook, entrypointViews, time.Now(),
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
h.serverError(w, r, "failed to count entrypoint events", err)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
scheme := "http"
|
scheme := "http"
|
||||||
@@ -523,7 +534,7 @@ func (h *Handlers) renderSourceDetail(
|
|||||||
// Targets are projected to a display-safe view: a
|
// Targets are projected to a display-safe view: a
|
||||||
// target's stored config blob holds a credential, and it
|
// target's stored config blob holds a credential, and it
|
||||||
// must never reach a template.
|
// must never reach a template.
|
||||||
"Entrypoints": NewEntrypointViews(entrypoints),
|
"Entrypoints": entrypointViews,
|
||||||
"Targets": h.targetRows(&webhook, targets),
|
"Targets": h.targetRows(&webhook, targets),
|
||||||
"Events": events,
|
"Events": events,
|
||||||
"BaseURL": baseURL,
|
"BaseURL": baseURL,
|
||||||
|
|||||||
@@ -326,6 +326,19 @@ func (h *Handlers) createAndFanOut(
|
|||||||
return nil, nil, err
|
return nil, nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A resubmitted copy did not arrive on its entrypoint's URL, so it
|
||||||
|
// leaves the entrypoint's last event as it is.
|
||||||
|
if src.ResubmittedFromID == nil {
|
||||||
|
err = database.AddEntrypointTotals(tx, database.EntrypointTotals{
|
||||||
|
EntrypointID: event.EntrypointID, LastEventAt: event.CreatedAt,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
tx.Rollback()
|
||||||
|
|
||||||
|
return nil, nil, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
err = tx.Commit().Error
|
err = tx.Commit().Error
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, nil, fmt.Errorf(
|
return nil, nil, fmt.Errorf(
|
||||||
|
|||||||
@@ -97,6 +97,18 @@
|
|||||||
</div>
|
</div>
|
||||||
<!-- The URL above is the entrypoint's credential:
|
<!-- The URL above is the entrypoint's credential:
|
||||||
anyone holding it can submit events. -->
|
anyone holding it can submit events. -->
|
||||||
|
<div class="text-xs text-gray-500 mt-1">
|
||||||
|
<span class="font-medium text-gray-700">Last Event:</span>
|
||||||
|
{{if .LastEvent}}
|
||||||
|
<span title="{{.LastEventUTC}}">{{.LastEvent}}</span>
|
||||||
|
{{else}}
|
||||||
|
<span>never</span>
|
||||||
|
{{end}}
|
||||||
|
</div>
|
||||||
|
<div class="text-xs text-gray-500 mt-1">
|
||||||
|
<span class="font-medium text-gray-700">Events Within Retention:</span>
|
||||||
|
<span>{{.Events}}</span>
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
{{else}}
|
{{else}}
|
||||||
<div class="p-4 text-sm text-gray-500">No entrypoints configured.</div>
|
<div class="p-4 text-sm text-gray-500">No entrypoints configured.</div>
|
||||||
|
|||||||
Reference in New Issue
Block a user