Compare commits
2
Commits
78cda941b4
...
46399d7d7b
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
46399d7d7b | ||
|
|
ff24638ba4 |
No files matched your search
@@ -1127,7 +1127,7 @@ unconditionally against whatever files it finds:
|
||||
- the main database on connect — `Setting`, `User`, `APIKey`, `Webhook`,
|
||||
`Entrypoint`, `Target`
|
||||
- each event database when it is lazily opened — `Event`, `Delivery`,
|
||||
`DeliveryResult`, `EventTotals`, `TargetTotals`
|
||||
`DeliveryResult`, `EventTotals`, `TargetTotals`, `EntrypointTotals`
|
||||
- each archive database on every open and reopen
|
||||
|
||||
There is no schema version table, no migration ledger, and no down
|
||||
@@ -1531,6 +1531,9 @@ tier** (event ingestion, delivery, and logging).
|
||||
│ ┌──────────────┐ (one row per target: running counts │
|
||||
│ │ TargetTotals │ of its deliveries) │
|
||||
│ └──────────────┘ │
|
||||
│ ┌──────────────────┐ (one row per entrypoint: when the │
|
||||
│ │ EntrypointTotals │ last event arrived on its URL) │
|
||||
│ └──────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
@@ -1577,6 +1580,13 @@ more entrypoints (receiver URLs) and one or more targets (delivery
|
||||
destinations) into a logical unit. A user creates a webhook to set up
|
||||
event routing.
|
||||
|
||||
The new webhook form can also give the webhook its first targets: an
|
||||
optional HTTP target URL creates an `http` target named `HTTP`, and the
|
||||
archive checkbox creates a `database` target named `Archive` whose
|
||||
`expiry` is the pruning chosen beside it (never, 1h, 12h, 24h, 30d, 90d
|
||||
or 365d). Both are validated as on the add target form, and the webhook
|
||||
and its targets are created together or not at all.
|
||||
|
||||
| Field | Type | Description |
|
||||
| ---------------- | ------- | ----------- |
|
||||
| `id` | UUID | Primary key |
|
||||
@@ -1644,6 +1654,11 @@ different event sources that all feed into the same processing pipeline
|
||||
(e.g., one entrypoint for GitHub, another for Stripe, both routing to
|
||||
the same targets).
|
||||
|
||||
The webhook page shows, for each entrypoint, when the last event arrived
|
||||
on its URL, which retention leaves in place, or "never" if none ever has,
|
||||
and how many events arrived on it within the webhook's retention period.
|
||||
A resubmitted event did not arrive on the URL and counts in neither.
|
||||
|
||||
#### Target
|
||||
|
||||
A delivery destination for events. Each target defines where and how
|
||||
@@ -1853,10 +1868,11 @@ retries) is individually logged for full observability.
|
||||
|
||||
**Relations:** Belongs to Delivery.
|
||||
|
||||
#### EventTotals and TargetTotals
|
||||
#### EventTotals, TargetTotals and EntrypointTotals
|
||||
|
||||
Running counts in each event database, read by the statistics pane at the
|
||||
top of the webhook page and by the webhook list. `EventTotals` is one row:
|
||||
top of the webhook page and by the webhook list, and each entrypoint's last
|
||||
event, read by the webhook page's entrypoint list. `EventTotals` is one row:
|
||||
|
||||
| Field | Type | Description |
|
||||
| ---------------- | --------- | ----------- |
|
||||
@@ -1875,18 +1891,26 @@ top of the webhook page and by the webhook list. `EventTotals` is one row:
|
||||
| `deliveries_removed` | integer | Its deliveries retention has deleted |
|
||||
| `failed_removed` | integer | Its failed deliveries retention has deleted |
|
||||
|
||||
Each count changes in the transaction that writes or deletes the rows it
|
||||
counts. The pane's lifetime events are `events`, and its lifetime
|
||||
deliveries and failures are `deliveries` and `failed` summed over the
|
||||
targets; each figure within retention is the same less what retention
|
||||
removed, so neither needs the rows themselves. Its last event is
|
||||
`last_event_at`, written in the transaction that stores the event, so it
|
||||
still shows once retention has removed every event. Its last-10-minutes and
|
||||
last-24-hours figures are counted from the `events` and `deliveries`
|
||||
indexes over just that window, the deliveries in one query grouped by
|
||||
target. Its failure percentage for a window is the deliveries that became
|
||||
`failed` in it out of all that became `delivered` or `failed` in it, and
|
||||
a dash when none did.
|
||||
`EntrypointTotals` is one row per entrypoint, created by the first event
|
||||
that arrives on its URL:
|
||||
|
||||
| Field | Type | Description |
|
||||
| --------------- | --------- | ----------- |
|
||||
| `entrypoint_id` | UUID | The entrypoint (primary key) |
|
||||
| `last_event_at` | timestamp | When the newest event arrived on its URL; a resubmitted event leaves it as it is, and so does retention |
|
||||
|
||||
Each count changes in the transaction that writes or deletes the rows it counts,
|
||||
and each `last_event_at` in the transaction that stores the event. The pane's
|
||||
lifetime events are `events`, and its lifetime deliveries and failures are
|
||||
`deliveries` and `failed` summed over the targets; each figure within retention
|
||||
is the same less what retention removed, so neither needs the rows themselves.
|
||||
Its last event is `last_event_at` in `EventTotals`, so it still shows once
|
||||
retention has removed every event; each entrypoint's last event, from
|
||||
`EntrypointTotals`, does too. Its last-10-minutes and last-24-hours figures are
|
||||
counted from the `events` and `deliveries` indexes over just that window, the
|
||||
deliveries in one query grouped by target. Its failure percentage for a window
|
||||
is the deliveries that became `failed` in it out of all that became `delivered`
|
||||
or `failed` in it, and a dash when none did.
|
||||
|
||||
The webhook list at `/hooks` shows three of the pane's figures for each
|
||||
webhook: its events within retention and its last event, both from
|
||||
@@ -1908,26 +1932,31 @@ tags, so `AutoMigrate` creates them on a fresh database:
|
||||
| `delivery_results` | `delivery_id`, `deleted_at` | The event log, which loads the attempts of a page's deliveries, and retention, which deletes the attempts of expired events |
|
||||
| `events` | `deleted_at`, `created_at` | The webhook page's statistics, which count recent events |
|
||||
| `events` | `resubmitted_from_id`, `deleted_at` | The event log, which counts the events resubmitted from each event on a page |
|
||||
| `events` | `entrypoint_id`, `deleted_at`, `resubmitted_from_id`, `created_at` | The webhook page's entrypoint list, which counts the events that arrived on each entrypoint's URL within the retention period |
|
||||
| `events` | `created_at` | Retention, which selects expired events by age |
|
||||
|
||||
GORM's soft delete adds `deleted_at IS NULL` to these queries; retention
|
||||
leaves it out. SQLite keeps no statistics on these tables, and without them it
|
||||
rates the `deleted_at` index, which every live row matches, above an index on
|
||||
a column matched against several values or compared with a range. So every
|
||||
index but the last also covers `deleted_at`. It comes second in the `event_id`
|
||||
and `delivery_id` indexes, so that retention can use them without it. The event
|
||||
GORM's soft delete adds `deleted_at IS NULL` to these queries; retention leaves
|
||||
it out. SQLite keeps no statistics on these tables, and without them it rates
|
||||
the `deleted_at` index, which every live row matches, above an index on a column
|
||||
matched against several values or compared with a range. So every index but the
|
||||
last also covers `deleted_at`. It comes second in the `event_id` and
|
||||
`delivery_id` indexes, so that retention can use them without it. The event
|
||||
log's count, the one query on the `resubmitted_from_id` index, always carries
|
||||
`deleted_at IS NULL` and uses both columns. In the statistics' `events` index
|
||||
`deleted_at` comes first, because they compare `created_at` with a range (`>=`)
|
||||
and SQLite narrows by a range only on the last column it uses.
|
||||
`deleted_at IS NULL` and uses both columns. The entrypoint list's count, the one
|
||||
query on the `entrypoint_id` index, uses all four, `resubmitted_from_id IS NULL`
|
||||
leaving out resubmitted copies and `created_at` last because it compares it with
|
||||
a range (`>=`). In the statistics' `events` index `deleted_at` comes first,
|
||||
because they compare `created_at` with a range (`>=`) and SQLite narrows by a
|
||||
range only on the last column it uses.
|
||||
|
||||
#### Common Fields
|
||||
|
||||
Every entity except `Setting`, `EventTotals` and `TargetTotals` includes
|
||||
these fields from `BaseModel`. `Setting` is a bare key-value row with no
|
||||
`id`, no timestamps and no soft delete, and the two totals tables hold
|
||||
counts, plus `last_event_at` in `event_totals`, keyed by a numeric `id`
|
||||
and by `target_id`:
|
||||
Every entity except `Setting`, `EventTotals`, `TargetTotals` and
|
||||
`EntrypointTotals` includes these fields from `BaseModel`. `Setting` is a bare
|
||||
key-value row with no `id`, no timestamps and no soft delete. Of the three
|
||||
totals tables, `event_totals` holds counts and `last_event_at`, keyed by a
|
||||
numeric `id`; `target_totals` holds counts, keyed by `target_id`; and
|
||||
`entrypoint_totals` holds `last_event_at`, keyed by `entrypoint_id`:
|
||||
|
||||
| Field | Type | Description |
|
||||
| ------------ | --------- | ----------- |
|
||||
@@ -1969,8 +1998,9 @@ encryption key is generated and stored, and an `admin` user is created.
|
||||
- **Events** — captured incoming webhook payloads
|
||||
- **Deliveries** — event-to-target pairings and their status
|
||||
- **DeliveryResults** — individual delivery attempt logs
|
||||
- **EventTotals** and **TargetTotals** — running counts of the above,
|
||||
the deliveries per target, kept through retention
|
||||
- **EventTotals**, **TargetTotals** and **EntrypointTotals** — running
|
||||
counts of the above, the deliveries per target, and each entrypoint's
|
||||
last event, kept through retention
|
||||
|
||||
Per-webhook databases are created automatically when a webhook is
|
||||
created. They are managed by the `WebhookDBManager` component, which
|
||||
@@ -3035,7 +3065,7 @@ webhooker/
|
||||
│ │ ├── model_event.go # Event entity (per-webhook DB)
|
||||
│ │ ├── model_delivery.go # Delivery entity (per-webhook DB)
|
||||
│ │ ├── model_delivery_result.go # DeliveryResult entity (per-webhook DB)
|
||||
│ │ ├── model_totals.go # EventTotals and TargetTotals (per-webhook DB)
|
||||
│ │ ├── model_totals.go # EventTotals, TargetTotals and EntrypointTotals (per-webhook DB)
|
||||
│ │ ├── model_apikey.go # APIKey entity
|
||||
│ │ ├── password.go # Argon2id hashing and verification
|
||||
│ │ ├── retention.go # Retention reaper (per-webhook event expiry)
|
||||
@@ -3311,9 +3341,9 @@ check, see [The login endpoint](#the-login-endpoint).
|
||||
`ENTRYPOINT` script, which sets the data directory's owner and mode
|
||||
before the app starts; the image's health check; and `docker exec`,
|
||||
unless given `--user`
|
||||
- GORM soft deletes on every entity that carries `BaseModel`, which is
|
||||
all of them but `Setting`, `EventTotals` and `TargetTotals` (data
|
||||
preserved for audit)
|
||||
- GORM soft deletes on every entity that carries `BaseModel`, which is all of
|
||||
them but `Setting`, `EventTotals`, `TargetTotals` and `EntrypointTotals`
|
||||
(data preserved for audit)
|
||||
|
||||
### Shutdown
|
||||
|
||||
|
||||
@@ -233,6 +233,43 @@ func TestResubmitCountUsesItsIndex(t *testing.T) {
|
||||
"(resubmitted_from_id=? AND deleted_at=?)")
|
||||
}
|
||||
|
||||
// TestEntrypointEventsUseTheirIndex does the same for the webhook
|
||||
// page's count, for each entrypoint, of the events that arrived on its
|
||||
// URL since the retention cutoff (addEntrypointEvents in the
|
||||
// handlers), which must come from the index alone. It passes 25
|
||||
// entrypoints, as TestResubmitCountUsesItsIndex passes 25 events.
|
||||
func TestEntrypointEventsUseTheirIndex(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
mgr, lc := setupTestWebhookDBManager(t)
|
||||
ctx := context.Background()
|
||||
require.NoError(t, lc.Start(ctx))
|
||||
|
||||
defer func() { require.NoError(t, lc.Stop(ctx)) }()
|
||||
|
||||
db, err := mgr.GetDB(uuid.New().String())
|
||||
require.NoError(t, err)
|
||||
|
||||
dry := db.Session(&gorm.Session{DryRun: true})
|
||||
|
||||
entrypoints := make([]string, 25)
|
||||
for i := range entrypoints {
|
||||
entrypoints[i] = uuid.New().String()
|
||||
}
|
||||
|
||||
var rows []struct{ Events int }
|
||||
|
||||
assertPlanUses(t, db, dry.Model(&database.Event{}).
|
||||
Select("entrypoint_id, count(*) AS events").
|
||||
Where("entrypoint_id IN ? AND resubmitted_from_id IS NULL",
|
||||
entrypoints).
|
||||
Where("created_at >= ?", time.Now()).
|
||||
Group("entrypoint_id").Find(&rows),
|
||||
"COVERING INDEX idx_events_entrypoint_id "+
|
||||
"(entrypoint_id=? AND deleted_at=? AND "+
|
||||
"resubmitted_from_id=? AND created_at>?)")
|
||||
}
|
||||
|
||||
// assertPlanUses asserts that SQLite's plan for a statement GORM built
|
||||
// in a dry run, run with the same SQL and arguments GORM would send,
|
||||
// names each of the given indexes.
|
||||
|
||||
@@ -20,12 +20,15 @@ type Event struct {
|
||||
// has no deleted_at condition and uses the index on created_at
|
||||
// alone. The other tables keep the unindexed BaseModel created_at.
|
||||
// DeletedAt is also the second column of the resubmitted_from_id
|
||||
// index, for the reason DeliveryResult gives.
|
||||
CreatedAt time.Time `gorm:"index;index:idx_events_deleted_at_created_at,priority:2" json:"createdAt"`
|
||||
DeletedAt gorm.DeletedAt `gorm:"index:idx_events_deleted_at_created_at,priority:1;index:idx_events_resubmitted_from_id,priority:2" json:"deletedAt,omitzero"`
|
||||
// index, for the reason DeliveryResult gives. The entrypoint_id
|
||||
// index, for the webhook page's entrypoint list, has it second too,
|
||||
// resubmitted_from_id third, and created_at last, which the list
|
||||
// compares with a range.
|
||||
CreatedAt time.Time `gorm:"index;index:idx_events_deleted_at_created_at,priority:2;index:idx_events_entrypoint_id,priority:4" json:"createdAt"`
|
||||
DeletedAt gorm.DeletedAt `gorm:"index:idx_events_deleted_at_created_at,priority:1;index:idx_events_resubmitted_from_id,priority:2;index:idx_events_entrypoint_id,priority:2" json:"deletedAt,omitzero"`
|
||||
|
||||
WebhookID string `gorm:"type:uuid;not null" json:"webhookId"`
|
||||
EntrypointID string `gorm:"type:uuid;not null" json:"entrypointId"`
|
||||
WebhookID string `gorm:"type:uuid;not null" json:"webhookId"`
|
||||
EntrypointID string `gorm:"type:uuid;not null;index:idx_events_entrypoint_id,priority:1" json:"entrypointId"`
|
||||
|
||||
// Request data
|
||||
Method string `gorm:"not null" json:"method"`
|
||||
@@ -44,7 +47,7 @@ type Event struct {
|
||||
// existed. It is not a foreign key: the source event can be
|
||||
// reaped by retention while its copies remain, and the id is
|
||||
// kept as the record of where the copy came from either way.
|
||||
ResubmittedFromID *string `gorm:"type:uuid;index:idx_events_resubmitted_from_id,priority:1" json:"resubmittedFromId,omitempty"`
|
||||
ResubmittedFromID *string `gorm:"type:uuid;index:idx_events_resubmitted_from_id,priority:1;index:idx_events_entrypoint_id,priority:3" json:"resubmittedFromId,omitempty"`
|
||||
|
||||
// Relations. No model marshals the record it belongs to, so
|
||||
// Webhook and Entrypoint are left out of the JSON.
|
||||
|
||||
@@ -52,6 +52,21 @@ func (TargetTotals) TableName() string {
|
||||
return "target_totals"
|
||||
}
|
||||
|
||||
// EntrypointTotals is one row per entrypoint, created by the first
|
||||
// event that arrives on its URL: when the newest such event arrived,
|
||||
// which retention leaves as it is. A resubmitted copy did not arrive
|
||||
// on the URL and does not change it.
|
||||
type EntrypointTotals struct {
|
||||
EntrypointID string `gorm:"type:uuid;primaryKey"`
|
||||
|
||||
LastEventAt time.Time `gorm:"not null"`
|
||||
}
|
||||
|
||||
// TableName names the table AddEntrypointTotals updates.
|
||||
func (EntrypointTotals) TableName() string {
|
||||
return "entrypoint_totals"
|
||||
}
|
||||
|
||||
// AddEventTotals adds each count in add to the webhook's event totals,
|
||||
// and records add.LastEventAt as when the newest event arrived if it is
|
||||
// set. Call it on the transaction that writes or deletes the events it
|
||||
@@ -97,3 +112,25 @@ func AddTargetTotals(tx *gorm.DB, add TargetTotals) error {
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// AddEntrypointTotals records add.LastEventAt as when the newest event
|
||||
// arrived on the URL of the entrypoint add.EntrypointID names, creating
|
||||
// its row the first time. Call it on the transaction that stores the
|
||||
// event.
|
||||
func AddEntrypointTotals(tx *gorm.DB, add EntrypointTotals) error {
|
||||
err := tx.Exec(
|
||||
`INSERT INTO entrypoint_totals (entrypoint_id, last_event_at)
|
||||
VALUES (?, ?)
|
||||
ON CONFLICT (entrypoint_id) DO UPDATE SET
|
||||
last_event_at = excluded.last_event_at`,
|
||||
add.EntrypointID, add.LastEventAt,
|
||||
).Error
|
||||
if err != nil {
|
||||
return fmt.Errorf(
|
||||
"adding to totals of entrypoint %s: %w",
|
||||
add.EntrypointID, err,
|
||||
)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -111,6 +111,13 @@ func (w *Webhook) RetainsForever() bool {
|
||||
return retainsForever(w.RetentionDays)
|
||||
}
|
||||
|
||||
// RetentionCutoff returns the time before which this webhook's events
|
||||
// have expired, as the reaper computes it, and false when the webhook
|
||||
// retains them forever.
|
||||
func (w *Webhook) RetentionCutoff(now time.Time) (time.Time, bool) {
|
||||
return retentionCutoff(now, w.RetentionDays)
|
||||
}
|
||||
|
||||
// RetentionLabel returns the webhook's retention policy as display
|
||||
// text, so that no template has to know about the sentinel value.
|
||||
func (w *Webhook) RetentionLabel() string {
|
||||
|
||||
@@ -3,7 +3,7 @@ package database
|
||||
// Migrate runs database migrations for the main application database.
|
||||
// Only configuration-tier models are stored in the main database.
|
||||
// Event-tier models (Event, Delivery, DeliveryResult, EventTotals,
|
||||
// TargetTotals) live in
|
||||
// TargetTotals, EntrypointTotals) live in
|
||||
// per-webhook dedicated databases managed by WebhookDBManager.
|
||||
func (d *Database) Migrate() error {
|
||||
return d.db.AutoMigrate(
|
||||
|
||||
@@ -49,7 +49,7 @@ var ErrSidecarNotRemoved = errors.New(
|
||||
// WebhookDBManager manages per-webhook SQLite database files
|
||||
// for event storage. Each webhook gets its own dedicated
|
||||
// database containing Events, Deliveries, DeliveryResults and the
|
||||
// running totals of them (EventTotals, TargetTotals).
|
||||
// running totals of them (EventTotals, TargetTotals, EntrypointTotals).
|
||||
// Database connections are opened lazily and cached.
|
||||
type WebhookDBManager struct {
|
||||
dataDir string
|
||||
@@ -381,7 +381,7 @@ func (m *WebhookDBManager) openDB(
|
||||
// Run migrations for event-tier models only
|
||||
err = db.AutoMigrate(
|
||||
&Event{}, &Delivery{}, &DeliveryResult{},
|
||||
&EventTotals{}, &TargetTotals{},
|
||||
&EventTotals{}, &TargetTotals{}, &EntrypointTotals{},
|
||||
)
|
||||
if err != nil {
|
||||
_ = sqlDB.Close()
|
||||
|
||||
@@ -1,6 +1,11 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/dustin/go-humanize"
|
||||
"gorm.io/gorm"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
)
|
||||
|
||||
@@ -11,6 +16,21 @@ type EntrypointView struct {
|
||||
Path string
|
||||
Description string
|
||||
Active bool
|
||||
|
||||
// Events is how many events arrived on the entrypoint's URL within
|
||||
// the webhook's retention period. LastEvent is when the newest
|
||||
// event ever to arrive on it did, relative, and LastEventUTC the
|
||||
// full time; both are empty when none ever did.
|
||||
Events int64
|
||||
LastEvent string
|
||||
LastEventUTC string
|
||||
}
|
||||
|
||||
// entrypointEvents is one entrypoint's count read by
|
||||
// addEntrypointEvents.
|
||||
type entrypointEvents struct {
|
||||
EntrypointID string
|
||||
Events int64
|
||||
}
|
||||
|
||||
// NewEntrypointViews projects entrypoints for rendering.
|
||||
@@ -32,3 +52,60 @@ func NewEntrypointViews(
|
||||
|
||||
return views
|
||||
}
|
||||
|
||||
// addEntrypointEvents fills in each view's event figures from the
|
||||
// webhook's event database: when the last event arrived on its URL,
|
||||
// from its EntrypointTotals row, and how many events arrived on it
|
||||
// since the webhook's retention cutoff, counted in one query over the
|
||||
// events' entrypoint_id index. Resubmitted copies did not arrive on
|
||||
// the URL and are left out of both.
|
||||
func addEntrypointEvents(
|
||||
webhookDB *gorm.DB,
|
||||
webhook *database.Webhook,
|
||||
views []EntrypointView,
|
||||
now time.Time,
|
||||
) error {
|
||||
ids := make([]string, len(views))
|
||||
byID := make(map[string]*EntrypointView, len(views))
|
||||
|
||||
for i := range views {
|
||||
ids[i] = views[i].ID
|
||||
byID[views[i].ID] = &views[i]
|
||||
}
|
||||
|
||||
var totals []database.EntrypointTotals
|
||||
|
||||
err := webhookDB.Where("entrypoint_id IN ?", ids).Find(&totals).Error
|
||||
if err != nil {
|
||||
return fmt.Errorf("reading entrypoint totals: %w", err)
|
||||
}
|
||||
|
||||
query := webhookDB.Model(&database.Event{}).
|
||||
Select("entrypoint_id, count(*) AS events").
|
||||
Where("entrypoint_id IN ? AND resubmitted_from_id IS NULL", ids)
|
||||
|
||||
cutoff, finite := webhook.RetentionCutoff(now)
|
||||
if finite {
|
||||
query = query.Where("created_at >= ?", cutoff)
|
||||
}
|
||||
|
||||
var counts []entrypointEvents
|
||||
|
||||
err = query.Group("entrypoint_id").Find(&counts).Error
|
||||
if err != nil {
|
||||
return fmt.Errorf("counting events by entrypoint: %w", err)
|
||||
}
|
||||
|
||||
for _, row := range totals {
|
||||
view := byID[row.EntrypointID]
|
||||
view.LastEvent = humanize.Time(row.LastEventAt)
|
||||
view.LastEventUTC =
|
||||
row.LastEventAt.UTC().Format(time.DateTime) + " UTC"
|
||||
}
|
||||
|
||||
for _, row := range counts {
|
||||
byID[row.EntrypointID].Events = row.Events
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,197 @@
|
||||
package handlers_test
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/gorm/clause"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/handlers"
|
||||
"sneak.berlin/go/webhooker/internal/logger"
|
||||
"sneak.berlin/go/webhooker/internal/session"
|
||||
)
|
||||
|
||||
// entrypointRow returns the part of a rendered webhook page from an
|
||||
// entrypoint's URL to the next entrypoint's, which holds its figures.
|
||||
func entrypointRow(t *testing.T, page, entrypointID string) string {
|
||||
t.Helper()
|
||||
|
||||
_, row, found := strings.Cut(page, `id="entrypoint-url-`+entrypointID+`"`)
|
||||
require.True(t, found)
|
||||
|
||||
row, _, _ = strings.Cut(row, `id="entrypoint-url-`)
|
||||
|
||||
return row
|
||||
}
|
||||
|
||||
// lastEventShown matches an entrypoint row's last event arriving at at.
|
||||
func lastEventShown(at time.Time) string {
|
||||
return `Last Event:</span>\s*<span title="` +
|
||||
at.UTC().Format(time.DateTime) + ` UTC">[^<]+</span>`
|
||||
}
|
||||
|
||||
// eventsShown matches an entrypoint row's count of n events.
|
||||
func eventsShown(n int) string {
|
||||
return `Events Within Retention:</span>\s*<span>` +
|
||||
strconv.Itoa(n) + `</span>`
|
||||
}
|
||||
|
||||
// TestHandleSourceDetail_ShowsEntrypointEvents proves each entrypoint
|
||||
// on the webhook page shows its own figures: how many events arrived
|
||||
// through it within the webhook's retention period, leaving out one
|
||||
// older than that, and when the newest arrived, or "never" for an
|
||||
// entrypoint with none.
|
||||
func TestHandleSourceDetail_ShowsEntrypointEvents(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
wh := &database.Webhook{
|
||||
UserID: deleteTestUserID, Name: "figures", RetentionDays: 7,
|
||||
}
|
||||
require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
|
||||
|
||||
webhookDB, err := dbMgr.GetDB(wh.ID)
|
||||
require.NoError(t, err)
|
||||
|
||||
entrypoint := func() *database.Entrypoint {
|
||||
ep := &database.Entrypoint{
|
||||
WebhookID: wh.ID, Path: uuid.New().String(), Active: true,
|
||||
}
|
||||
require.NoError(t,
|
||||
db.DB().Omit(clause.Associations).Create(ep).Error)
|
||||
|
||||
return ep
|
||||
}
|
||||
|
||||
// event stores an event that arrived on ep's URL age ago and
|
||||
// records it as ep's last event, as the receiver does.
|
||||
event := func(ep *database.Entrypoint, age time.Duration) time.Time {
|
||||
e := &database.Event{
|
||||
WebhookID: wh.ID,
|
||||
EntrypointID: ep.ID,
|
||||
Method: http.MethodPost,
|
||||
}
|
||||
e.CreatedAt = time.Now().Add(-age)
|
||||
require.NoError(t,
|
||||
webhookDB.Omit(clause.Associations).Create(e).Error)
|
||||
require.NoError(t, database.AddEntrypointTotals(webhookDB,
|
||||
database.EntrypointTotals{
|
||||
EntrypointID: ep.ID, LastEventAt: e.CreatedAt,
|
||||
}))
|
||||
|
||||
return e.CreatedAt
|
||||
}
|
||||
|
||||
busy, quiet, unused := entrypoint(), entrypoint(), entrypoint()
|
||||
|
||||
event(busy, 8*24*time.Hour) // older than the 7 days kept
|
||||
event(busy, 3*time.Hour)
|
||||
busyLast := event(busy, time.Hour)
|
||||
quietLast := event(quiet, 2*24*time.Hour)
|
||||
|
||||
body := renderSourceDetailPage(t, h, sess, wh.ID)
|
||||
|
||||
assert.Regexp(t, lastEventShown(busyLast), entrypointRow(t, body, busy.ID))
|
||||
assert.Regexp(t, eventsShown(2), entrypointRow(t, body, busy.ID))
|
||||
assert.Regexp(t, lastEventShown(quietLast), entrypointRow(t, body, quiet.ID))
|
||||
assert.Regexp(t, eventsShown(1), entrypointRow(t, body, quiet.ID))
|
||||
assert.Regexp(t, `Last Event:</span>\s*<span>never</span>`,
|
||||
entrypointRow(t, body, unused.ID))
|
||||
assert.Regexp(t, eventsShown(0), entrypointRow(t, body, unused.ID))
|
||||
}
|
||||
|
||||
// TestHandleSourceDetail_EntrypointLastEventSurvivesRetention checks
|
||||
// that once retention has removed every event that arrived on an
|
||||
// entrypoint's URL, the entrypoint still shows when the last one
|
||||
// arrived rather than "never".
|
||||
func TestHandleSourceDetail_EntrypointLastEventSurvivesRetention(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
log *logger.Logger
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr, &log)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
wh := &database.Webhook{
|
||||
UserID: deleteTestUserID, Name: "swept", RetentionDays: 1,
|
||||
}
|
||||
require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
|
||||
|
||||
ep := seedEntrypoint(t, db, wh.ID)
|
||||
webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, ep.Path, 1)
|
||||
arrived := events[0].CreatedAt
|
||||
|
||||
statsAge(t, webhookDB, events[0].ID, time.Now().Add(-50*time.Hour))
|
||||
statsPrune(t, db, dbMgr, log, webhookDB)
|
||||
require.Empty(t, listEvents(t, webhookDB))
|
||||
|
||||
row := entrypointRow(t, renderSourceDetailPage(t, h, sess, wh.ID), ep.ID)
|
||||
assert.Regexp(t, lastEventShown(arrived), row)
|
||||
assert.Regexp(t, eventsShown(0), row)
|
||||
}
|
||||
|
||||
// TestHandleSourceDetail_ResubmitLeavesEntrypointFigures checks that a
|
||||
// resubmitted copy, which did not arrive on the entrypoint's URL,
|
||||
// changes neither the entrypoint's last event nor its count.
|
||||
func TestHandleSourceDetail_ResubmitLeavesEntrypointFigures(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
wh := seedWebhook(t, db)
|
||||
ep := seedEntrypoint(t, db, wh.ID)
|
||||
webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, ep.Path, 1)
|
||||
arrived := events[0].CreatedAt
|
||||
|
||||
require.Equal(t, http.StatusSeeOther,
|
||||
postResubmit(t, h, sess, wh.ID, events[0].ID).Code)
|
||||
require.Len(t, listEvents(t, webhookDB), 2)
|
||||
|
||||
var totals database.EntrypointTotals
|
||||
|
||||
require.NoError(t, webhookDB.Take(&totals).Error)
|
||||
assert.True(t, arrived.Equal(totals.LastEventAt))
|
||||
|
||||
row := entrypointRow(t, renderSourceDetailPage(t, h, sess, wh.ID), ep.ID)
|
||||
assert.Regexp(t, lastEventShown(arrived), row)
|
||||
assert.Regexp(t, eventsShown(1), row)
|
||||
}
|
||||
@@ -0,0 +1,207 @@
|
||||
package handlers_test
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"html"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/gorm"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
)
|
||||
|
||||
// submitCreateForm posts the new webhook form and returns the
|
||||
// recorder.
|
||||
func submitCreateForm(
|
||||
env *sourceTestEnv, form url.Values,
|
||||
) *httptest.ResponseRecorder {
|
||||
req := formRequest("/hooks/new", env.cookies, form, nil)
|
||||
w := httptest.NewRecorder()
|
||||
|
||||
env.handlers.HandleSourceCreateSubmit().ServeHTTP(w, req)
|
||||
|
||||
return w
|
||||
}
|
||||
|
||||
// assertNothingCreated checks that the main database holds no webhook,
|
||||
// entrypoint or target.
|
||||
func assertNothingCreated(t *testing.T, db *database.Database) {
|
||||
t.Helper()
|
||||
|
||||
for _, model := range []any{
|
||||
&database.Webhook{}, &database.Entrypoint{}, &database.Target{},
|
||||
} {
|
||||
var count int64
|
||||
|
||||
require.NoError(t, db.DB().Model(model).Count(&count).Error)
|
||||
assert.Zerof(t, count, "%T rows were created", model)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHandleSourceCreateSubmit_CreatesRequestedTargets submits the new
|
||||
// webhook form with the HTTP target URL filled in or empty, and with
|
||||
// the archive checkbox off or on with each pruning choice. The webhook
|
||||
// gets an HTTP target only for a URL and a database target only for a
|
||||
// checked archive. The pruning choice is always submitted, as the
|
||||
// browser submits it while it is hidden, and is ignored when archive
|
||||
// is off.
|
||||
func TestHandleSourceCreateSubmit_CreatesRequestedTargets(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupSourceTest(t)
|
||||
|
||||
// Each value the archive pruning choice submits, after an empty
|
||||
// one that stands for the archive checkbox left off.
|
||||
expiries := []string{
|
||||
"", "never", "1h", "12h", "24h", "720h", "2160h", "8760h",
|
||||
}
|
||||
|
||||
for _, httpURL := range []string{"", editOriginalURL} {
|
||||
for _, expiry := range expiries {
|
||||
name := "url=" + httpURL + " archive=" + expiry
|
||||
|
||||
t.Run(name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
form := url.Values{}
|
||||
form.Set("name", name)
|
||||
form.Set("http_url", httpURL)
|
||||
form.Set("archive_expiry", "720h")
|
||||
|
||||
if expiry != "" {
|
||||
form.Set("archive", "on")
|
||||
form.Set("archive_expiry", expiry)
|
||||
}
|
||||
|
||||
w := submitCreateForm(env, form)
|
||||
require.Equal(t, http.StatusSeeOther, w.Code, w.Body.String())
|
||||
|
||||
var webhook database.Webhook
|
||||
|
||||
require.NoError(t, env.db.DB().
|
||||
Where("name = ?", name).First(&webhook).Error)
|
||||
|
||||
byType := map[database.TargetType]database.Target{}
|
||||
for _, target := range targetsForWebhook(t, env.db, webhook.ID) {
|
||||
byType[target.Type] = target
|
||||
}
|
||||
|
||||
wantCount := 0
|
||||
|
||||
if httpURL != "" {
|
||||
wantCount++
|
||||
|
||||
assert.Equal(t, "HTTP", byType[database.TargetTypeHTTP].Name)
|
||||
assert.JSONEq(t, `{"url":"`+httpURL+`"}`,
|
||||
byType[database.TargetTypeHTTP].Config)
|
||||
}
|
||||
|
||||
if expiry != "" {
|
||||
wantCount++
|
||||
|
||||
assert.Equal(t, "Archive",
|
||||
byType[database.TargetTypeDatabase].Name)
|
||||
assert.JSONEq(t, `{"expiry":"`+expiry+`"}`,
|
||||
byType[database.TargetTypeDatabase].Config)
|
||||
}
|
||||
|
||||
assert.Len(t, byType, wantCount)
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestHandleSourceCreateSubmit_RefusedFormKeepsEveryValue refuses the
|
||||
// new webhook form for an invalid HTTP target URL and for an invalid
|
||||
// retention, each with archive on. Nothing is created, and the form
|
||||
// comes back with the reason and every value entered: name,
|
||||
// description, retention, URL, the checked archive box and the pruning
|
||||
// choice.
|
||||
func TestHandleSourceCreateSubmit_RefusedFormKeepsEveryValue(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
const badURL = "Invalid target URL"
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
retention string
|
||||
httpURL string
|
||||
reason string
|
||||
}{
|
||||
{"blocked url", "7", editBlockedURL, badURL},
|
||||
{"unsupported scheme", "7", "ftp://93.184.216.34/hook", badURL},
|
||||
{"bad retention", "-5", editOriginalURL, "Retention must be"},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupSourceTest(t)
|
||||
|
||||
form := url.Values{}
|
||||
form.Set("name", "kept name")
|
||||
form.Set("description", "kept description")
|
||||
form.Set("retention_days", tc.retention)
|
||||
form.Set("http_url", tc.httpURL)
|
||||
form.Set("archive", "on")
|
||||
form.Set("archive_expiry", "2160h")
|
||||
|
||||
w := submitCreateForm(env, form)
|
||||
require.Equal(t, http.StatusBadRequest, w.Code)
|
||||
|
||||
page := w.Body.String()
|
||||
assert.Contains(t, page, tc.reason)
|
||||
assert.Contains(t, page, `value="kept name"`)
|
||||
assert.Contains(t, page, `>kept description</textarea>`)
|
||||
assert.Contains(t, page, `value="`+tc.retention+`"`)
|
||||
assert.Contains(t, page,
|
||||
`value="`+html.EscapeString(tc.httpURL)+`"`)
|
||||
assert.Contains(t, page, `name="archive" value="on" checked`)
|
||||
assert.Contains(t, page, `x-data="collapsible" data-open`)
|
||||
assert.Contains(t, page, `<option value="2160h" selected>`)
|
||||
|
||||
assertNothingCreated(t, env.db)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// errInjectedTargetCreate is the failure a test makes the insert of a
|
||||
// target report.
|
||||
var errInjectedTargetCreate = errors.New("injected target create failure")
|
||||
|
||||
// TestHandleSourceCreateSubmit_FailedTargetInsertCreatesNothing makes
|
||||
// inserting a target fail after the webhook and its entrypoint were
|
||||
// inserted, and checks that neither is left behind.
|
||||
func TestHandleSourceCreateSubmit_FailedTargetInsertCreatesNothing(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupSourceTest(t)
|
||||
|
||||
require.NoError(t, env.db.DB().Callback().Create().
|
||||
Before("gorm:create").
|
||||
Register("test:fail_target_create", func(tx *gorm.DB) {
|
||||
if tx.Statement.Table == "targets" {
|
||||
_ = tx.AddError(errInjectedTargetCreate)
|
||||
}
|
||||
}),
|
||||
)
|
||||
|
||||
form := url.Values{}
|
||||
form.Set("name", "rolled back")
|
||||
form.Set("archive", "on")
|
||||
form.Set("archive_expiry", "never")
|
||||
|
||||
w := submitCreateForm(env, form)
|
||||
require.Equal(t, http.StatusInternalServerError, w.Code)
|
||||
|
||||
assertNothingCreated(t, env.db)
|
||||
}
|
||||
@@ -276,27 +276,41 @@ func (h *Handlers) HandleSourceCreate() http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
h.renderTemplate(
|
||||
w, r, "sources_new.html",
|
||||
newSourceFormData("", "", ""),
|
||||
newSourceFormData("", sourceFormInput{
|
||||
RetentionDays: strconv.Itoa(
|
||||
database.DefaultRetentionDays,
|
||||
),
|
||||
}),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// sourceFormInput carries the raw values of the new webhook form. A
|
||||
// refused submission is shown again from it, so every value entered
|
||||
// comes back, retention included.
|
||||
type sourceFormInput struct {
|
||||
Name string
|
||||
Description string
|
||||
RetentionDays string
|
||||
// HTTPURL, when not empty, asks for an HTTP target with this
|
||||
// destination.
|
||||
HTTPURL string
|
||||
// Archive asks for a database (archive) target, whose rows expire
|
||||
// after ArchiveExpiry.
|
||||
Archive bool
|
||||
ArchiveExpiry string
|
||||
}
|
||||
|
||||
// newSourceFormData builds the template data for the webhook creation
|
||||
// form.
|
||||
//
|
||||
// It carries the retention default so the pre-filled value comes from
|
||||
// database.DefaultRetentionDays rather than being a third hardcoded
|
||||
// copy of the same policy, and it carries the submitted name and
|
||||
// description so that re-rendering the form after a validation failure
|
||||
// gives the user their input back instead of a blank form. The edit
|
||||
// form already behaves that way; create now matches it.
|
||||
// form. It carries the retention default, which the form's help text
|
||||
// names, from database.DefaultRetentionDays rather than a hardcoded
|
||||
// copy of the same policy.
|
||||
func newSourceFormData(
|
||||
errMsg, name, description string,
|
||||
errMsg string, in sourceFormInput,
|
||||
) map[string]any {
|
||||
return map[string]any{
|
||||
tmplKeyError: errMsg,
|
||||
"Name": name,
|
||||
"Description": description,
|
||||
"Form": in,
|
||||
"DefaultRetentionDays": database.DefaultRetentionDays,
|
||||
}
|
||||
}
|
||||
@@ -323,57 +337,112 @@ func (h *Handlers) HandleSourceCreateSubmit() http.HandlerFunc {
|
||||
return
|
||||
}
|
||||
|
||||
name := r.PostFormValue("name")
|
||||
description := r.PostFormValue("description")
|
||||
retentionStr := r.PostFormValue("retention_days")
|
||||
in := sourceFormInput{
|
||||
Name: r.PostFormValue("name"),
|
||||
Description: r.PostFormValue("description"),
|
||||
RetentionDays: r.PostFormValue("retention_days"),
|
||||
HTTPURL: r.PostFormValue("http_url"),
|
||||
Archive: r.PostFormValue("archive") != "",
|
||||
ArchiveExpiry: r.PostFormValue("archive_expiry"),
|
||||
}
|
||||
|
||||
if name == "" {
|
||||
refuse := func(errMsg string) {
|
||||
h.renderTemplateStatus(
|
||||
w, r, "sources_new.html",
|
||||
newSourceFormData(
|
||||
"Name is required", name, description,
|
||||
),
|
||||
newSourceFormData(errMsg, in),
|
||||
http.StatusBadRequest,
|
||||
)
|
||||
}
|
||||
|
||||
if in.Name == "" {
|
||||
refuse("Name is required")
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
retentionDays, errMsg := parseRetentionDays(
|
||||
retentionStr, database.DefaultRetentionDays,
|
||||
in.RetentionDays, database.DefaultRetentionDays,
|
||||
)
|
||||
if errMsg != "" {
|
||||
h.renderTemplateStatus(
|
||||
w, r, "sources_new.html",
|
||||
newSourceFormData(errMsg, name, description),
|
||||
http.StatusBadRequest,
|
||||
)
|
||||
refuse(errMsg)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
h.createWebhookWithEntrypoint(
|
||||
w, r, userID, name, description, retentionDays,
|
||||
)
|
||||
targets, errMsg, err := h.newWebhookTargets(r.Context(), in)
|
||||
if err != nil {
|
||||
h.serverError(w, r, "failed to encode target config", err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
if errMsg != "" {
|
||||
refuse(errMsg)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
h.createWebhookWithEntrypoint(w, r, &database.Webhook{
|
||||
UserID: userID,
|
||||
Name: in.Name,
|
||||
Description: in.Description,
|
||||
RetentionDays: retentionDays,
|
||||
}, targets)
|
||||
}
|
||||
}
|
||||
|
||||
// createWebhookWithEntrypoint creates a webhook and its default
|
||||
// entrypoint in a transaction.
|
||||
// newWebhookTargets validates the targets the new webhook form asks
|
||||
// for and returns the rows to create with the webhook, or the message
|
||||
// the form shows for the first one it refuses. A filled-in HTTP URL
|
||||
// asks for an HTTP target named "HTTP", and the archive checkbox for a
|
||||
// database target named "Archive". Each goes through newTarget, as on
|
||||
// the webhook page's add target form. The rows have no WebhookID yet:
|
||||
// the webhook has no ID until it is created.
|
||||
func (h *Handlers) newWebhookTargets(
|
||||
ctx context.Context,
|
||||
in sourceFormInput,
|
||||
) ([]*database.Target, string, error) {
|
||||
var requested []targetFormInput
|
||||
|
||||
if in.HTTPURL != "" {
|
||||
requested = append(requested, targetFormInput{
|
||||
Name: "HTTP",
|
||||
Type: database.TargetTypeHTTP,
|
||||
URL: in.HTTPURL,
|
||||
})
|
||||
}
|
||||
|
||||
if in.Archive {
|
||||
requested = append(requested, targetFormInput{
|
||||
Name: "Archive",
|
||||
Type: database.TargetTypeDatabase,
|
||||
Expiry: in.ArchiveExpiry,
|
||||
})
|
||||
}
|
||||
|
||||
targets := make([]*database.Target, 0, len(requested))
|
||||
|
||||
for _, form := range requested {
|
||||
target, errMsg, err := h.newTarget(ctx, "", form)
|
||||
if err != nil || errMsg != "" {
|
||||
return nil, errMsg, err
|
||||
}
|
||||
|
||||
targets = append(targets, target)
|
||||
}
|
||||
|
||||
return targets, "", nil
|
||||
}
|
||||
|
||||
// createWebhookWithEntrypoint creates a webhook, its default
|
||||
// entrypoint and the given targets in a transaction.
|
||||
func (h *Handlers) createWebhookWithEntrypoint(
|
||||
w http.ResponseWriter,
|
||||
r *http.Request,
|
||||
userID, name, description string,
|
||||
retentionDays int,
|
||||
webhook *database.Webhook,
|
||||
targets []*database.Target,
|
||||
) {
|
||||
webhook := &database.Webhook{
|
||||
UserID: userID,
|
||||
Name: name,
|
||||
Description: description,
|
||||
RetentionDays: retentionDays,
|
||||
}
|
||||
|
||||
err := h.commitWebhook(webhook)
|
||||
err := h.commitWebhook(webhook, targets)
|
||||
if err != nil {
|
||||
h.serverError(w, r, "failed to create webhook", err)
|
||||
|
||||
@@ -390,7 +459,7 @@ func (h *Handlers) createWebhookWithEntrypoint(
|
||||
|
||||
h.log.Info("webhook created",
|
||||
"webhook_id", webhook.ID,
|
||||
"name", name, "user_id", userID,
|
||||
"name", webhook.Name, "user_id", webhook.UserID,
|
||||
)
|
||||
|
||||
http.Redirect(
|
||||
@@ -399,10 +468,12 @@ func (h *Handlers) createWebhookWithEntrypoint(
|
||||
)
|
||||
}
|
||||
|
||||
// commitWebhook creates a webhook and default entrypoint in
|
||||
// a transaction. Returns an error on failure (rolls back).
|
||||
// commitWebhook creates a webhook, its default entrypoint and the
|
||||
// given targets in a transaction. Returns an error on failure (rolls
|
||||
// back).
|
||||
func (h *Handlers) commitWebhook(
|
||||
webhook *database.Webhook,
|
||||
targets []*database.Target,
|
||||
) error {
|
||||
tx := h.db.DB().Begin()
|
||||
if tx.Error != nil {
|
||||
@@ -430,6 +501,17 @@ func (h *Handlers) commitWebhook(
|
||||
return err
|
||||
}
|
||||
|
||||
for _, target := range targets {
|
||||
target.WebhookID = webhook.ID
|
||||
|
||||
err = tx.Create(target).Error
|
||||
if err != nil {
|
||||
tx.Rollback()
|
||||
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return tx.Commit().Error
|
||||
}
|
||||
|
||||
@@ -485,6 +567,8 @@ func (h *Handlers) renderSourceDetail(
|
||||
"webhook_id = ?", webhook.ID,
|
||||
).Find(&targets)
|
||||
|
||||
entrypointViews := NewEntrypointViews(entrypoints)
|
||||
|
||||
var events []RecentEventView
|
||||
|
||||
if h.dbMgr.DBExists(webhook.ID) {
|
||||
@@ -503,6 +587,15 @@ func (h *Handlers) renderSourceDetail(
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
err = addEntrypointEvents(
|
||||
webhookDB, &webhook, entrypointViews, time.Now(),
|
||||
)
|
||||
if err != nil {
|
||||
h.serverError(w, r, "failed to count entrypoint events", err)
|
||||
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
scheme := "http"
|
||||
@@ -523,7 +616,7 @@ func (h *Handlers) renderSourceDetail(
|
||||
// Targets are projected to a display-safe view: a
|
||||
// target's stored config blob holds a credential, and it
|
||||
// must never reach a template.
|
||||
"Entrypoints": NewEntrypointViews(entrypoints),
|
||||
"Entrypoints": entrypointViews,
|
||||
"Targets": h.targetRows(&webhook, targets),
|
||||
"Events": events,
|
||||
"BaseURL": baseURL,
|
||||
|
||||
@@ -196,8 +196,6 @@ func TestCreateFormRetentionCopyMatchesBehaviour(t *testing.T) {
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
body := renderPage(t, h, sess, "sources_new.html", map[string]any{
|
||||
"Name": "",
|
||||
"Description": "",
|
||||
"DefaultRetentionDays": database.DefaultRetentionDays,
|
||||
dataKeyError: "",
|
||||
})
|
||||
|
||||
@@ -326,6 +326,19 @@ func (h *Handlers) createAndFanOut(
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
// A resubmitted copy did not arrive on its entrypoint's URL, so it
|
||||
// leaves the entrypoint's last event as it is.
|
||||
if src.ResubmittedFromID == nil {
|
||||
err = database.AddEntrypointTotals(tx, database.EntrypointTotals{
|
||||
EntrypointID: event.EntrypointID, LastEventAt: event.CreatedAt,
|
||||
})
|
||||
if err != nil {
|
||||
tx.Rollback()
|
||||
|
||||
return nil, nil, err
|
||||
}
|
||||
}
|
||||
|
||||
err = tx.Commit().Error
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf(
|
||||
|
||||
@@ -122,6 +122,9 @@ func TestAlpineRunsUnderTheSecurityPolicy(t *testing.T) {
|
||||
checkEntrypointEdit(ctx, t, page, page+"/events")
|
||||
checkRecentEvents(ctx, t, page)
|
||||
checkEventLog(ctx, t, page+"/events", event.ID, target.Name)
|
||||
checkArchiveChoice(ctx, t, srv.URL+"/hooks/new", page)
|
||||
checkNewWebhookTargets(ctx, t, env, srv.URL+"/hooks/new")
|
||||
checkRefusedNewWebhook(ctx, t, srv.URL+"/hooks/new")
|
||||
checkMobileMenu(ctx, t, page)
|
||||
|
||||
assert.Empty(t, problems(), "the browser reported problems")
|
||||
@@ -642,6 +645,199 @@ func checkEventLog(
|
||||
"clicking the event again does not collapse it")
|
||||
}
|
||||
|
||||
// The parts of the new webhook page the checks below find and click.
|
||||
const (
|
||||
archiveBox = `//input[@name="archive"]`
|
||||
archiveIsOn = `document.querySelector('input[name="archive"]').checked`
|
||||
pruningChoice = `//select[@name="archive_expiry"]`
|
||||
createButton = `//button[text()="Create Webhook"]`
|
||||
)
|
||||
|
||||
// checkArchiveChoice loads the new webhook page and checks that the
|
||||
// archive pruning choice stays hidden until the archive box is checked
|
||||
// and hides again when it is unchecked; and that after checking it,
|
||||
// opening the page at elsewhere and going back, the page again shows
|
||||
// the box unchecked and the choice hidden.
|
||||
func checkArchiveChoice(
|
||||
ctx context.Context, t *testing.T, url, elsewhere string,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
require.NoError(t, chromedp.Run(ctx, loadPage(url)))
|
||||
|
||||
assert.True(t, hidden(ctx, pruningChoice),
|
||||
"the pruning choice shows before archive is checked")
|
||||
|
||||
click(ctx, t, archiveBox)
|
||||
assert.True(t, shown(ctx, pruningChoice),
|
||||
"checking archive does not show the pruning choice")
|
||||
|
||||
click(ctx, t, archiveBox)
|
||||
assert.True(t, hidden(ctx, pruningChoice),
|
||||
"unchecking archive does not hide the pruning choice")
|
||||
|
||||
var (
|
||||
loaded string
|
||||
checked bool
|
||||
)
|
||||
|
||||
click(ctx, t, archiveBox)
|
||||
require.NoError(t, chromedp.Run(
|
||||
ctx,
|
||||
loadPage(elsewhere),
|
||||
chromedp.NavigateBack(),
|
||||
chromedp.WaitNotPresent("[x-cloak]", chromedp.ByQuery),
|
||||
chromedp.Evaluate(
|
||||
`performance.getEntriesByType("navigation")[0].type`, &loaded,
|
||||
),
|
||||
chromedp.Evaluate(archiveIsOn, &checked),
|
||||
))
|
||||
require.Equal(
|
||||
t, "back_forward", loaded,
|
||||
"going back, the browser did not load the page again",
|
||||
)
|
||||
|
||||
assert.False(t, checked, "going back leaves archive checked")
|
||||
assert.True(t, hidden(ctx, pruningChoice),
|
||||
"going back shows the pruning choice")
|
||||
}
|
||||
|
||||
// checkNewWebhookTargets submits the new webhook page with the HTTP
|
||||
// target URL filled in or empty, and with archive left off or checked
|
||||
// with each pruning choice, and checks that each webhook is created
|
||||
// with exactly the targets asked for.
|
||||
func checkNewWebhookTargets(
|
||||
ctx context.Context, t *testing.T, env *testEnv, url string,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
// Each value the pruning choice submits, after an empty one that
|
||||
// stands for archive left off.
|
||||
expiries := []string{
|
||||
"", "never", "1h", "12h", "24h", "720h", "2160h", "8760h",
|
||||
}
|
||||
|
||||
for _, httpURL := range []string{"", publicTargetURL} {
|
||||
for _, expiry := range expiries {
|
||||
name := "url=" + httpURL + " archive=" + expiry
|
||||
want := map[database.TargetType]string{}
|
||||
|
||||
require.NoError(t, chromedp.Run(
|
||||
ctx,
|
||||
loadPage(url),
|
||||
chromedp.SetValue("#name", name, chromedp.ByQuery),
|
||||
))
|
||||
|
||||
if httpURL != "" {
|
||||
require.NoError(t, chromedp.Run(ctx, chromedp.SetValue(
|
||||
"#http_url", httpURL, chromedp.ByQuery,
|
||||
)))
|
||||
|
||||
want[database.TargetTypeHTTP] = `{"url":"` + httpURL + `"}`
|
||||
}
|
||||
|
||||
if expiry != "" {
|
||||
// The choice showing moves Create down, so it is
|
||||
// waited for before Create is clicked.
|
||||
click(ctx, t, archiveBox)
|
||||
require.Truef(t, shown(ctx, pruningChoice),
|
||||
"%s: checking archive does not show the pruning choice",
|
||||
name)
|
||||
require.NoError(t, chromedp.Run(ctx, chromedp.SetValue(
|
||||
pruningChoice, expiry, chromedp.BySearch,
|
||||
)))
|
||||
|
||||
want[database.TargetTypeDatabase] = `{"expiry":"` + expiry + `"}`
|
||||
}
|
||||
|
||||
click(ctx, t, createButton)
|
||||
require.Truef(t, shown(ctx, `//h1[text()="`+name+`"]`),
|
||||
"%s: the new webhook's page does not open", name)
|
||||
|
||||
assert.Equalf(t, want, targetConfigs(t, env, name),
|
||||
"%s: the webhook does not have the targets asked for", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// targetConfigs reads the targets of the webhook named name, and
|
||||
// returns each one's stored configuration by its type.
|
||||
func targetConfigs(
|
||||
t *testing.T, env *testEnv, name string,
|
||||
) map[database.TargetType]string {
|
||||
t.Helper()
|
||||
|
||||
var (
|
||||
webhook database.Webhook
|
||||
targets []database.Target
|
||||
)
|
||||
|
||||
require.NoError(t, env.db.DB().
|
||||
Where("name = ?", name).First(&webhook).Error)
|
||||
require.NoError(t, env.db.DB().
|
||||
Where("webhook_id = ?", webhook.ID).Find(&targets).Error)
|
||||
|
||||
configs := map[database.TargetType]string{}
|
||||
for _, target := range targets {
|
||||
configs[target.Type] = target.Config
|
||||
}
|
||||
|
||||
return configs
|
||||
}
|
||||
|
||||
// checkRefusedNewWebhook submits the new webhook page with archive
|
||||
// checked and an HTTP target URL the server refuses, a loopback
|
||||
// destination, and checks that the page comes back with the reason and
|
||||
// every value entered, archive still checked and its pruning choice
|
||||
// showing.
|
||||
func checkRefusedNewWebhook(ctx context.Context, t *testing.T, url string) {
|
||||
t.Helper()
|
||||
|
||||
const refusedURL = "http://127.0.0.1/hook"
|
||||
|
||||
require.NoError(t, chromedp.Run(
|
||||
ctx,
|
||||
loadPage(url),
|
||||
chromedp.SetValue("#name", "refused", chromedp.ByQuery),
|
||||
chromedp.SetValue("#description", "kept", chromedp.ByQuery),
|
||||
chromedp.SetValue("#retention_days", "7", chromedp.ByQuery),
|
||||
chromedp.SetValue("#http_url", refusedURL, chromedp.ByQuery),
|
||||
))
|
||||
click(ctx, t, archiveBox)
|
||||
require.True(t, shown(ctx, pruningChoice),
|
||||
"checking archive does not show the pruning choice")
|
||||
require.NoError(t, chromedp.Run(ctx, chromedp.SetValue(
|
||||
pruningChoice, "2160h", chromedp.BySearch,
|
||||
)))
|
||||
click(ctx, t, createButton)
|
||||
|
||||
assert.True(t, shown(ctx, `//div[@class="alert-error"]`),
|
||||
"a refused webhook does not show the reason")
|
||||
|
||||
var (
|
||||
name, description, retention, typed, expiry string
|
||||
checked bool
|
||||
)
|
||||
|
||||
require.NoError(t, chromedp.Run(
|
||||
ctx,
|
||||
chromedp.Value("#name", &name, chromedp.ByQuery),
|
||||
chromedp.Value("#description", &description, chromedp.ByQuery),
|
||||
chromedp.Value("#retention_days", &retention, chromedp.ByQuery),
|
||||
chromedp.Value("#http_url", &typed, chromedp.ByQuery),
|
||||
chromedp.Value("#archive_expiry", &expiry, chromedp.ByQuery),
|
||||
chromedp.Evaluate(archiveIsOn, &checked),
|
||||
))
|
||||
|
||||
assert.Equal(t, "refused", name, "the name entered is lost")
|
||||
assert.Equal(t, "kept", description, "the description entered is lost")
|
||||
assert.Equal(t, "7", retention, "the retention entered is lost")
|
||||
assert.Equal(t, refusedURL, typed, "the url entered is lost")
|
||||
assert.True(t, checked, "archive is no longer checked")
|
||||
assert.True(t, shown(ctx, pruningChoice), "the pruning choice is hidden")
|
||||
assert.Equal(t, "2160h", expiry, "the pruning chosen is lost")
|
||||
}
|
||||
|
||||
// checkMobileMenu loads a page in a phone-sized window and checks that
|
||||
// the menu button opens and closes the mobile menu.
|
||||
func checkMobileMenu(ctx context.Context, t *testing.T, url string) {
|
||||
|
||||
+3
-2
@@ -71,8 +71,9 @@ document.addEventListener("alpine:init", function () {
|
||||
|
||||
// Something a click shows and hides: the mobile menu, an add form,
|
||||
// an entrypoint's edit form, an event in the event log or in the
|
||||
// recent events, a delivery's attempts. It starts hidden, or shown
|
||||
// when its element has the data-open attribute.
|
||||
// recent events, a delivery's attempts, the new webhook page's
|
||||
// archive pruning choice. It starts hidden, or shown when its
|
||||
// element has the data-open attribute.
|
||||
window.Alpine.data("collapsible", function () {
|
||||
return {
|
||||
open: false,
|
||||
|
||||
@@ -97,6 +97,18 @@
|
||||
</div>
|
||||
<!-- The URL above is the entrypoint's credential:
|
||||
anyone holding it can submit events. -->
|
||||
<div class="text-xs text-gray-500 mt-1">
|
||||
<span class="font-medium text-gray-700">Last Event:</span>
|
||||
{{if .LastEvent}}
|
||||
<span title="{{.LastEventUTC}}">{{.LastEvent}}</span>
|
||||
{{else}}
|
||||
<span>never</span>
|
||||
{{end}}
|
||||
</div>
|
||||
<div class="text-xs text-gray-500 mt-1">
|
||||
<span class="font-medium text-gray-700">Events Within Retention:</span>
|
||||
<span>{{.Events}}</span>
|
||||
</div>
|
||||
</div>
|
||||
{{else}}
|
||||
<div class="p-4 text-sm text-gray-500">No entrypoints configured.</div>
|
||||
|
||||
@@ -18,20 +18,50 @@
|
||||
<input type="hidden" name="csrf_token" value="{{.CSRFToken}}">
|
||||
<div class="form-group">
|
||||
<label for="name" class="label">Name</label>
|
||||
<input type="text" id="name" name="name" value="{{.Name}}" required autofocus placeholder="My Webhook" class="input">
|
||||
<input type="text" id="name" name="name" value="{{.Form.Name}}" required autofocus placeholder="My Webhook" class="input">
|
||||
</div>
|
||||
|
||||
<div class="form-group">
|
||||
<label for="description" class="label">Description</label>
|
||||
<textarea id="description" name="description" rows="3" placeholder="Optional description" class="input">{{.Description}}</textarea>
|
||||
<textarea id="description" name="description" rows="3" placeholder="Optional description" class="input">{{.Form.Description}}</textarea>
|
||||
</div>
|
||||
|
||||
<div class="form-group">
|
||||
<label for="retention_days" class="label">Retention (days)</label>
|
||||
<input type="number" id="retention_days" name="retention_days" value="{{.DefaultRetentionDays}}" min="0" class="input">
|
||||
<input type="number" id="retention_days" name="retention_days" value="{{.Form.RetentionDays}}" min="0" class="input">
|
||||
<p class="text-xs text-gray-500 mt-1">A periodic cleanup permanently deletes events older than this, along with their delivery records. Enter 0 to retain events forever; leave blank to use the default of {{.DefaultRetentionDays}} days.</p>
|
||||
</div>
|
||||
|
||||
<div class="form-group">
|
||||
<label for="http_url" class="label">HTTP target URL</label>
|
||||
<input type="url" id="http_url" name="http_url" value="{{.Form.HTTPURL}}" placeholder="https://example.com/webhook" class="input">
|
||||
<p class="text-xs text-gray-500 mt-1">Optional. When filled in, the webhook is created with an HTTP target that delivers each event to this URL.</p>
|
||||
</div>
|
||||
|
||||
<!-- The checkbox shows the pruning choice while checked. With
|
||||
autocomplete="off", going back to the page does not
|
||||
check the box again with the choice hidden. -->
|
||||
<div class="form-group" x-data="collapsible"{{if .Form.Archive}} data-open{{end}}>
|
||||
<label class="flex items-center gap-2 text-sm font-medium text-gray-700">
|
||||
<input type="checkbox" name="archive" value="on"{{if .Form.Archive}} checked{{end}} autocomplete="off" @change="toggle" class="h-4 w-4">
|
||||
Archive
|
||||
</label>
|
||||
<p class="text-xs text-gray-500 mt-1">When checked, the webhook is created with a database target that keeps a copy of every event.</p>
|
||||
<div x-show="open" x-cloak class="mt-3">
|
||||
<label for="archive_expiry" class="label">Archive pruning</label>
|
||||
<select id="archive_expiry" name="archive_expiry" class="input">
|
||||
<option value="never"{{if eq .Form.ArchiveExpiry "never"}} selected{{end}}>never</option>
|
||||
<option value="1h"{{if eq .Form.ArchiveExpiry "1h"}} selected{{end}}>1h</option>
|
||||
<option value="12h"{{if eq .Form.ArchiveExpiry "12h"}} selected{{end}}>12h</option>
|
||||
<option value="24h"{{if eq .Form.ArchiveExpiry "24h"}} selected{{end}}>24h</option>
|
||||
<option value="720h"{{if eq .Form.ArchiveExpiry "720h"}} selected{{end}}>30d</option>
|
||||
<option value="2160h"{{if eq .Form.ArchiveExpiry "2160h"}} selected{{end}}>90d</option>
|
||||
<option value="8760h"{{if eq .Form.ArchiveExpiry "8760h"}} selected{{end}}>365d</option>
|
||||
</select>
|
||||
<p class="text-xs text-gray-500 mt-1">Archived events older than this are deleted from the archive; never keeps them all.</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="flex gap-3">
|
||||
<button type="submit" class="btn-primary">Create Webhook</button>
|
||||
<a href="/hooks" class="btn-secondary">Cancel</a>
|
||||
|
||||
Reference in new issue
Block a user