Compare commits
3
Commits
5de316bb7a
...
3e36c966ba
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3e36c966ba | ||
|
|
9305af4f85 | ||
|
|
ff24638ba4 |
@@ -1127,7 +1127,7 @@ unconditionally against whatever files it finds:
|
||||
- the main database on connect — `Setting`, `User`, `APIKey`, `Webhook`,
|
||||
`Entrypoint`, `Target`
|
||||
- each event database when it is lazily opened — `Event`, `Delivery`,
|
||||
`DeliveryResult`, `EventTotals`, `TargetTotals`
|
||||
`DeliveryResult`, `EventTotals`, `TargetTotals`, `EntrypointTotals`
|
||||
- each archive database on every open and reopen
|
||||
|
||||
There is no schema version table, no migration ledger, and no down
|
||||
@@ -1343,23 +1343,25 @@ markup. The CSP build runs no expressions, so every Alpine directive in
|
||||
`static/js/app.js`: `x-data="collapsible"` and `@click="toggle"`, never
|
||||
`x-data="{ open: false }"` or `@click="open = !open"`.
|
||||
|
||||
A browser test in `internal/server` loads the webhook page and the event log
|
||||
under the real policy and checks that: the add entrypoint form stays hidden
|
||||
until Add is clicked; for every target type, the targets section's Add shows
|
||||
only a choice of type with Next and Cancel, Next shows only that type's fields
|
||||
(no url field for `database` or `log`), Cancel at either step closes the form,
|
||||
and saving adds the target; a refused target comes back with its form open, the
|
||||
values entered and the reason, and after Cancel the next Add starts with an
|
||||
empty form and no reason; the Copy button beside an entrypoint URL reads
|
||||
"Copied" once clicked; an entrypoint's Edit button shows its edit form in place
|
||||
of its description and hides until the form closes, Cancel hides the form and
|
||||
drops what was typed, as does leaving the page and going back to it, and Save
|
||||
changes the description; of the recent events on the webhook page only the
|
||||
newest starts expanded, each expands and collapses, and Open leads to the
|
||||
event's own page; an event in the event log expands and collapses, and so do a
|
||||
delivery's attempts inside it; and at phone width the menu button opens and
|
||||
closes the mobile menu. It also fails if the browser reports a console warning
|
||||
or error, an uncaught exception, or anything the policy refused. `make check`
|
||||
A browser test in `internal/server` loads the webhook page, its edit pages and
|
||||
the event log under the real policy and checks that: the add entrypoint form
|
||||
stays hidden until Add is clicked; for every target type, the targets section's
|
||||
Add shows only a choice of type with Next and Cancel, Next shows only that
|
||||
type's fields (no url field for `database` or `log`), Cancel at either step
|
||||
closes the form, and saving adds the target; a refused target comes back with
|
||||
its form open, the values entered and the reason, and after Cancel the next Add
|
||||
starts with an empty form and no reason; a refused save on the target edit page
|
||||
and on the webhook edit page comes back with the reason and every value
|
||||
entered; the Copy button beside an entrypoint URL reads "Copied" once clicked;
|
||||
an entrypoint's Edit button shows its edit form in place of its description and
|
||||
hides until the form closes, Cancel hides the form and drops what was typed, as
|
||||
does leaving the page and going back to it, and Save changes the description;
|
||||
of the recent events on the webhook page only the newest starts expanded, each
|
||||
expands and collapses, and Open leads to the event's own page; an event in the
|
||||
event log expands and collapses, and so do a delivery's attempts inside it; and
|
||||
at phone width the menu button opens and closes the mobile menu. It also fails
|
||||
if the browser reports a console warning or error, an uncaught exception, or
|
||||
anything the policy refused. `make check`
|
||||
and the image build lint it but do not run it, and `make test` leaves it out
|
||||
(its file is built only with the `browser` build tag). Run it with
|
||||
`make test-browser` after changing `templates/` or `static/js/`: that builds
|
||||
@@ -1531,6 +1533,9 @@ tier** (event ingestion, delivery, and logging).
|
||||
│ ┌──────────────┐ (one row per target: running counts │
|
||||
│ │ TargetTotals │ of its deliveries) │
|
||||
│ └──────────────┘ │
|
||||
│ ┌──────────────────┐ (one row per entrypoint: when the │
|
||||
│ │ EntrypointTotals │ last event arrived on its URL) │
|
||||
│ └──────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
@@ -1644,6 +1649,11 @@ different event sources that all feed into the same processing pipeline
|
||||
(e.g., one entrypoint for GitHub, another for Stripe, both routing to
|
||||
the same targets).
|
||||
|
||||
The webhook page shows, for each entrypoint, when the last event arrived
|
||||
on its URL, which retention leaves in place, or "never" if none ever has,
|
||||
and how many events arrived on it within the webhook's retention period.
|
||||
A resubmitted event did not arrive on the URL and counts in neither.
|
||||
|
||||
#### Target
|
||||
|
||||
A delivery destination for events. Each target defines where and how
|
||||
@@ -1853,10 +1863,11 @@ retries) is individually logged for full observability.
|
||||
|
||||
**Relations:** Belongs to Delivery.
|
||||
|
||||
#### EventTotals and TargetTotals
|
||||
#### EventTotals, TargetTotals and EntrypointTotals
|
||||
|
||||
Running counts in each event database, read by the statistics pane at the
|
||||
top of the webhook page and by the webhook list. `EventTotals` is one row:
|
||||
top of the webhook page and by the webhook list, and each entrypoint's last
|
||||
event, read by the webhook page's entrypoint list. `EventTotals` is one row:
|
||||
|
||||
| Field | Type | Description |
|
||||
| ---------------- | --------- | ----------- |
|
||||
@@ -1875,18 +1886,26 @@ top of the webhook page and by the webhook list. `EventTotals` is one row:
|
||||
| `deliveries_removed` | integer | Its deliveries retention has deleted |
|
||||
| `failed_removed` | integer | Its failed deliveries retention has deleted |
|
||||
|
||||
Each count changes in the transaction that writes or deletes the rows it
|
||||
counts. The pane's lifetime events are `events`, and its lifetime
|
||||
deliveries and failures are `deliveries` and `failed` summed over the
|
||||
targets; each figure within retention is the same less what retention
|
||||
removed, so neither needs the rows themselves. Its last event is
|
||||
`last_event_at`, written in the transaction that stores the event, so it
|
||||
still shows once retention has removed every event. Its last-10-minutes and
|
||||
last-24-hours figures are counted from the `events` and `deliveries`
|
||||
indexes over just that window, the deliveries in one query grouped by
|
||||
target. Its failure percentage for a window is the deliveries that became
|
||||
`failed` in it out of all that became `delivered` or `failed` in it, and
|
||||
a dash when none did.
|
||||
`EntrypointTotals` is one row per entrypoint, created by the first event
|
||||
that arrives on its URL:
|
||||
|
||||
| Field | Type | Description |
|
||||
| --------------- | --------- | ----------- |
|
||||
| `entrypoint_id` | UUID | The entrypoint (primary key) |
|
||||
| `last_event_at` | timestamp | When the newest event arrived on its URL; a resubmitted event leaves it as it is, and so does retention |
|
||||
|
||||
Each count changes in the transaction that writes or deletes the rows it counts,
|
||||
and each `last_event_at` in the transaction that stores the event. The pane's
|
||||
lifetime events are `events`, and its lifetime deliveries and failures are
|
||||
`deliveries` and `failed` summed over the targets; each figure within retention
|
||||
is the same less what retention removed, so neither needs the rows themselves.
|
||||
Its last event is `last_event_at` in `EventTotals`, so it still shows once
|
||||
retention has removed every event; each entrypoint's last event, from
|
||||
`EntrypointTotals`, does too. Its last-10-minutes and last-24-hours figures are
|
||||
counted from the `events` and `deliveries` indexes over just that window, the
|
||||
deliveries in one query grouped by target. Its failure percentage for a window
|
||||
is the deliveries that became `failed` in it out of all that became `delivered`
|
||||
or `failed` in it, and a dash when none did.
|
||||
|
||||
The webhook list at `/hooks` shows three of the pane's figures for each
|
||||
webhook: its events within retention and its last event, both from
|
||||
@@ -1896,6 +1915,12 @@ counted with the pane's query. It opens each webhook's event database once
|
||||
with the number of webhooks and, for each, with the deliveries that
|
||||
finished in the last 24 hours, never with the events stored.
|
||||
|
||||
The target list on the webhook page shows, for each target, its
|
||||
`delivered` and `failed` totals, which retention does not reduce, and its
|
||||
deliveries that became `delivered` and `failed` in the last 24 hours,
|
||||
counted with the pane's query. Deliveries still `pending` or `retrying`
|
||||
count in neither.
|
||||
|
||||
#### Event-tier indexes
|
||||
|
||||
These indexes on the per-webhook event databases are declared in the model
|
||||
@@ -1903,31 +1928,36 @@ tags, so `AutoMigrate` creates them on a fresh database:
|
||||
|
||||
| Table | Columns | Serves |
|
||||
| ------------------ | --------------------------- | ------ |
|
||||
| `deliveries` | `status`, `deleted_at`, `finished_at`, `target_id` | Startup recovery, the retry and pending sweeps every 60 seconds and the queue-depth sampler every 30 seconds, which select deliveries by status, and the webhook page's statistics and the webhook list, which count each target's deliveries by status and when they finished |
|
||||
| `deliveries` | `status`, `deleted_at`, `finished_at`, `target_id` | Startup recovery, the retry and pending sweeps every 60 seconds and the queue-depth sampler every 30 seconds, which select deliveries by status, and the webhook page's statistics and target list and the webhook list, which count each target's deliveries by status and when they finished |
|
||||
| `deliveries` | `event_id`, `deleted_at` | The event log, which loads each event's deliveries, and retention, which counts and deletes the deliveries of expired events |
|
||||
| `delivery_results` | `delivery_id`, `deleted_at` | The event log, which loads the attempts of a page's deliveries, and retention, which deletes the attempts of expired events |
|
||||
| `events` | `deleted_at`, `created_at` | The webhook page's statistics, which count recent events |
|
||||
| `events` | `resubmitted_from_id`, `deleted_at` | The event log, which counts the events resubmitted from each event on a page |
|
||||
| `events` | `entrypoint_id`, `deleted_at`, `resubmitted_from_id`, `created_at` | The webhook page's entrypoint list, which counts the events that arrived on each entrypoint's URL within the retention period |
|
||||
| `events` | `created_at` | Retention, which selects expired events by age |
|
||||
|
||||
GORM's soft delete adds `deleted_at IS NULL` to these queries; retention
|
||||
leaves it out. SQLite keeps no statistics on these tables, and without them it
|
||||
rates the `deleted_at` index, which every live row matches, above an index on
|
||||
a column matched against several values or compared with a range. So every
|
||||
index but the last also covers `deleted_at`. It comes second in the `event_id`
|
||||
and `delivery_id` indexes, so that retention can use them without it. The event
|
||||
GORM's soft delete adds `deleted_at IS NULL` to these queries; retention leaves
|
||||
it out. SQLite keeps no statistics on these tables, and without them it rates
|
||||
the `deleted_at` index, which every live row matches, above an index on a column
|
||||
matched against several values or compared with a range. So every index but the
|
||||
last also covers `deleted_at`. It comes second in the `event_id` and
|
||||
`delivery_id` indexes, so that retention can use them without it. The event
|
||||
log's count, the one query on the `resubmitted_from_id` index, always carries
|
||||
`deleted_at IS NULL` and uses both columns. In the statistics' `events` index
|
||||
`deleted_at` comes first, because they compare `created_at` with a range (`>=`)
|
||||
and SQLite narrows by a range only on the last column it uses.
|
||||
`deleted_at IS NULL` and uses both columns. The entrypoint list's count, the one
|
||||
query on the `entrypoint_id` index, uses all four, `resubmitted_from_id IS NULL`
|
||||
leaving out resubmitted copies and `created_at` last because it compares it with
|
||||
a range (`>=`). In the statistics' `events` index `deleted_at` comes first,
|
||||
because they compare `created_at` with a range (`>=`) and SQLite narrows by a
|
||||
range only on the last column it uses.
|
||||
|
||||
#### Common Fields
|
||||
|
||||
Every entity except `Setting`, `EventTotals` and `TargetTotals` includes
|
||||
these fields from `BaseModel`. `Setting` is a bare key-value row with no
|
||||
`id`, no timestamps and no soft delete, and the two totals tables hold
|
||||
counts, plus `last_event_at` in `event_totals`, keyed by a numeric `id`
|
||||
and by `target_id`:
|
||||
Every entity except `Setting`, `EventTotals`, `TargetTotals` and
|
||||
`EntrypointTotals` includes these fields from `BaseModel`. `Setting` is a bare
|
||||
key-value row with no `id`, no timestamps and no soft delete. Of the three
|
||||
totals tables, `event_totals` holds counts and `last_event_at`, keyed by a
|
||||
numeric `id`; `target_totals` holds counts, keyed by `target_id`; and
|
||||
`entrypoint_totals` holds `last_event_at`, keyed by `entrypoint_id`:
|
||||
|
||||
| Field | Type | Description |
|
||||
| ------------ | --------- | ----------- |
|
||||
@@ -1969,8 +1999,9 @@ encryption key is generated and stored, and an `admin` user is created.
|
||||
- **Events** — captured incoming webhook payloads
|
||||
- **Deliveries** — event-to-target pairings and their status
|
||||
- **DeliveryResults** — individual delivery attempt logs
|
||||
- **EventTotals** and **TargetTotals** — running counts of the above,
|
||||
the deliveries per target, kept through retention
|
||||
- **EventTotals**, **TargetTotals** and **EntrypointTotals** — running
|
||||
counts of the above, the deliveries per target, and each entrypoint's
|
||||
last event, kept through retention
|
||||
|
||||
Per-webhook databases are created automatically when a webhook is
|
||||
created. They are managed by the `WebhookDBManager` component, which
|
||||
@@ -3035,7 +3066,7 @@ webhooker/
|
||||
│ │ ├── model_event.go # Event entity (per-webhook DB)
|
||||
│ │ ├── model_delivery.go # Delivery entity (per-webhook DB)
|
||||
│ │ ├── model_delivery_result.go # DeliveryResult entity (per-webhook DB)
|
||||
│ │ ├── model_totals.go # EventTotals and TargetTotals (per-webhook DB)
|
||||
│ │ ├── model_totals.go # EventTotals, TargetTotals and EntrypointTotals (per-webhook DB)
|
||||
│ │ ├── model_apikey.go # APIKey entity
|
||||
│ │ ├── password.go # Argon2id hashing and verification
|
||||
│ │ ├── retention.go # Retention reaper (per-webhook event expiry)
|
||||
@@ -3311,9 +3342,9 @@ check, see [The login endpoint](#the-login-endpoint).
|
||||
`ENTRYPOINT` script, which sets the data directory's owner and mode
|
||||
before the app starts; the image's health check; and `docker exec`,
|
||||
unless given `--user`
|
||||
- GORM soft deletes on every entity that carries `BaseModel`, which is
|
||||
all of them but `Setting`, `EventTotals` and `TargetTotals` (data
|
||||
preserved for audit)
|
||||
- GORM soft deletes on every entity that carries `BaseModel`, which is all of
|
||||
them but `Setting`, `EventTotals`, `TargetTotals` and `EntrypointTotals`
|
||||
(data preserved for audit)
|
||||
|
||||
### Shutdown
|
||||
|
||||
|
||||
@@ -233,6 +233,43 @@ func TestResubmitCountUsesItsIndex(t *testing.T) {
|
||||
"(resubmitted_from_id=? AND deleted_at=?)")
|
||||
}
|
||||
|
||||
// TestEntrypointEventsUseTheirIndex does the same for the webhook
|
||||
// page's count, for each entrypoint, of the events that arrived on its
|
||||
// URL since the retention cutoff (addEntrypointEvents in the
|
||||
// handlers), which must come from the index alone. It passes 25
|
||||
// entrypoints, as TestResubmitCountUsesItsIndex passes 25 events.
|
||||
func TestEntrypointEventsUseTheirIndex(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
mgr, lc := setupTestWebhookDBManager(t)
|
||||
ctx := context.Background()
|
||||
require.NoError(t, lc.Start(ctx))
|
||||
|
||||
defer func() { require.NoError(t, lc.Stop(ctx)) }()
|
||||
|
||||
db, err := mgr.GetDB(uuid.New().String())
|
||||
require.NoError(t, err)
|
||||
|
||||
dry := db.Session(&gorm.Session{DryRun: true})
|
||||
|
||||
entrypoints := make([]string, 25)
|
||||
for i := range entrypoints {
|
||||
entrypoints[i] = uuid.New().String()
|
||||
}
|
||||
|
||||
var rows []struct{ Events int }
|
||||
|
||||
assertPlanUses(t, db, dry.Model(&database.Event{}).
|
||||
Select("entrypoint_id, count(*) AS events").
|
||||
Where("entrypoint_id IN ? AND resubmitted_from_id IS NULL",
|
||||
entrypoints).
|
||||
Where("created_at >= ?", time.Now()).
|
||||
Group("entrypoint_id").Find(&rows),
|
||||
"COVERING INDEX idx_events_entrypoint_id "+
|
||||
"(entrypoint_id=? AND deleted_at=? AND "+
|
||||
"resubmitted_from_id=? AND created_at>?)")
|
||||
}
|
||||
|
||||
// assertPlanUses asserts that SQLite's plan for a statement GORM built
|
||||
// in a dry run, run with the same SQL and arguments GORM would send,
|
||||
// names each of the given indexes.
|
||||
|
||||
@@ -20,12 +20,15 @@ type Event struct {
|
||||
// has no deleted_at condition and uses the index on created_at
|
||||
// alone. The other tables keep the unindexed BaseModel created_at.
|
||||
// DeletedAt is also the second column of the resubmitted_from_id
|
||||
// index, for the reason DeliveryResult gives.
|
||||
CreatedAt time.Time `gorm:"index;index:idx_events_deleted_at_created_at,priority:2" json:"createdAt"`
|
||||
DeletedAt gorm.DeletedAt `gorm:"index:idx_events_deleted_at_created_at,priority:1;index:idx_events_resubmitted_from_id,priority:2" json:"deletedAt,omitzero"`
|
||||
// index, for the reason DeliveryResult gives. The entrypoint_id
|
||||
// index, for the webhook page's entrypoint list, has it second too,
|
||||
// resubmitted_from_id third, and created_at last, which the list
|
||||
// compares with a range.
|
||||
CreatedAt time.Time `gorm:"index;index:idx_events_deleted_at_created_at,priority:2;index:idx_events_entrypoint_id,priority:4" json:"createdAt"`
|
||||
DeletedAt gorm.DeletedAt `gorm:"index:idx_events_deleted_at_created_at,priority:1;index:idx_events_resubmitted_from_id,priority:2;index:idx_events_entrypoint_id,priority:2" json:"deletedAt,omitzero"`
|
||||
|
||||
WebhookID string `gorm:"type:uuid;not null" json:"webhookId"`
|
||||
EntrypointID string `gorm:"type:uuid;not null" json:"entrypointId"`
|
||||
WebhookID string `gorm:"type:uuid;not null" json:"webhookId"`
|
||||
EntrypointID string `gorm:"type:uuid;not null;index:idx_events_entrypoint_id,priority:1" json:"entrypointId"`
|
||||
|
||||
// Request data
|
||||
Method string `gorm:"not null" json:"method"`
|
||||
@@ -44,7 +47,7 @@ type Event struct {
|
||||
// existed. It is not a foreign key: the source event can be
|
||||
// reaped by retention while its copies remain, and the id is
|
||||
// kept as the record of where the copy came from either way.
|
||||
ResubmittedFromID *string `gorm:"type:uuid;index:idx_events_resubmitted_from_id,priority:1" json:"resubmittedFromId,omitempty"`
|
||||
ResubmittedFromID *string `gorm:"type:uuid;index:idx_events_resubmitted_from_id,priority:1;index:idx_events_entrypoint_id,priority:3" json:"resubmittedFromId,omitempty"`
|
||||
|
||||
// Relations. No model marshals the record it belongs to, so
|
||||
// Webhook and Entrypoint are left out of the JSON.
|
||||
|
||||
@@ -52,6 +52,21 @@ func (TargetTotals) TableName() string {
|
||||
return "target_totals"
|
||||
}
|
||||
|
||||
// EntrypointTotals is one row per entrypoint, created by the first
|
||||
// event that arrives on its URL: when the newest such event arrived,
|
||||
// which retention leaves as it is. A resubmitted copy did not arrive
|
||||
// on the URL and does not change it.
|
||||
type EntrypointTotals struct {
|
||||
EntrypointID string `gorm:"type:uuid;primaryKey"`
|
||||
|
||||
LastEventAt time.Time `gorm:"not null"`
|
||||
}
|
||||
|
||||
// TableName names the table AddEntrypointTotals updates.
|
||||
func (EntrypointTotals) TableName() string {
|
||||
return "entrypoint_totals"
|
||||
}
|
||||
|
||||
// AddEventTotals adds each count in add to the webhook's event totals,
|
||||
// and records add.LastEventAt as when the newest event arrived if it is
|
||||
// set. Call it on the transaction that writes or deletes the events it
|
||||
@@ -97,3 +112,25 @@ func AddTargetTotals(tx *gorm.DB, add TargetTotals) error {
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// AddEntrypointTotals records add.LastEventAt as when the newest event
|
||||
// arrived on the URL of the entrypoint add.EntrypointID names, creating
|
||||
// its row the first time. Call it on the transaction that stores the
|
||||
// event.
|
||||
func AddEntrypointTotals(tx *gorm.DB, add EntrypointTotals) error {
|
||||
err := tx.Exec(
|
||||
`INSERT INTO entrypoint_totals (entrypoint_id, last_event_at)
|
||||
VALUES (?, ?)
|
||||
ON CONFLICT (entrypoint_id) DO UPDATE SET
|
||||
last_event_at = excluded.last_event_at`,
|
||||
add.EntrypointID, add.LastEventAt,
|
||||
).Error
|
||||
if err != nil {
|
||||
return fmt.Errorf(
|
||||
"adding to totals of entrypoint %s: %w",
|
||||
add.EntrypointID, err,
|
||||
)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -111,6 +111,13 @@ func (w *Webhook) RetainsForever() bool {
|
||||
return retainsForever(w.RetentionDays)
|
||||
}
|
||||
|
||||
// RetentionCutoff returns the time before which this webhook's events
|
||||
// have expired, as the reaper computes it, and false when the webhook
|
||||
// retains them forever.
|
||||
func (w *Webhook) RetentionCutoff(now time.Time) (time.Time, bool) {
|
||||
return retentionCutoff(now, w.RetentionDays)
|
||||
}
|
||||
|
||||
// RetentionLabel returns the webhook's retention policy as display
|
||||
// text, so that no template has to know about the sentinel value.
|
||||
func (w *Webhook) RetentionLabel() string {
|
||||
|
||||
@@ -3,7 +3,7 @@ package database
|
||||
// Migrate runs database migrations for the main application database.
|
||||
// Only configuration-tier models are stored in the main database.
|
||||
// Event-tier models (Event, Delivery, DeliveryResult, EventTotals,
|
||||
// TargetTotals) live in
|
||||
// TargetTotals, EntrypointTotals) live in
|
||||
// per-webhook dedicated databases managed by WebhookDBManager.
|
||||
func (d *Database) Migrate() error {
|
||||
return d.db.AutoMigrate(
|
||||
|
||||
@@ -49,7 +49,7 @@ var ErrSidecarNotRemoved = errors.New(
|
||||
// WebhookDBManager manages per-webhook SQLite database files
|
||||
// for event storage. Each webhook gets its own dedicated
|
||||
// database containing Events, Deliveries, DeliveryResults and the
|
||||
// running totals of them (EventTotals, TargetTotals).
|
||||
// running totals of them (EventTotals, TargetTotals, EntrypointTotals).
|
||||
// Database connections are opened lazily and cached.
|
||||
type WebhookDBManager struct {
|
||||
dataDir string
|
||||
@@ -381,7 +381,7 @@ func (m *WebhookDBManager) openDB(
|
||||
// Run migrations for event-tier models only
|
||||
err = db.AutoMigrate(
|
||||
&Event{}, &Delivery{}, &DeliveryResult{},
|
||||
&EventTotals{}, &TargetTotals{},
|
||||
&EventTotals{}, &TargetTotals{}, &EntrypointTotals{},
|
||||
)
|
||||
if err != nil {
|
||||
_ = sqlDB.Close()
|
||||
|
||||
@@ -1,6 +1,11 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/dustin/go-humanize"
|
||||
"gorm.io/gorm"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
)
|
||||
|
||||
@@ -11,6 +16,21 @@ type EntrypointView struct {
|
||||
Path string
|
||||
Description string
|
||||
Active bool
|
||||
|
||||
// Events is how many events arrived on the entrypoint's URL within
|
||||
// the webhook's retention period. LastEvent is when the newest
|
||||
// event ever to arrive on it did, relative, and LastEventUTC the
|
||||
// full time; both are empty when none ever did.
|
||||
Events int64
|
||||
LastEvent string
|
||||
LastEventUTC string
|
||||
}
|
||||
|
||||
// entrypointEvents is one entrypoint's count read by
|
||||
// addEntrypointEvents.
|
||||
type entrypointEvents struct {
|
||||
EntrypointID string
|
||||
Events int64
|
||||
}
|
||||
|
||||
// NewEntrypointViews projects entrypoints for rendering.
|
||||
@@ -32,3 +52,60 @@ func NewEntrypointViews(
|
||||
|
||||
return views
|
||||
}
|
||||
|
||||
// addEntrypointEvents fills in each view's event figures from the
|
||||
// webhook's event database: when the last event arrived on its URL,
|
||||
// from its EntrypointTotals row, and how many events arrived on it
|
||||
// since the webhook's retention cutoff, counted in one query over the
|
||||
// events' entrypoint_id index. Resubmitted copies did not arrive on
|
||||
// the URL and are left out of both.
|
||||
func addEntrypointEvents(
|
||||
webhookDB *gorm.DB,
|
||||
webhook *database.Webhook,
|
||||
views []EntrypointView,
|
||||
now time.Time,
|
||||
) error {
|
||||
ids := make([]string, len(views))
|
||||
byID := make(map[string]*EntrypointView, len(views))
|
||||
|
||||
for i := range views {
|
||||
ids[i] = views[i].ID
|
||||
byID[views[i].ID] = &views[i]
|
||||
}
|
||||
|
||||
var totals []database.EntrypointTotals
|
||||
|
||||
err := webhookDB.Where("entrypoint_id IN ?", ids).Find(&totals).Error
|
||||
if err != nil {
|
||||
return fmt.Errorf("reading entrypoint totals: %w", err)
|
||||
}
|
||||
|
||||
query := webhookDB.Model(&database.Event{}).
|
||||
Select("entrypoint_id, count(*) AS events").
|
||||
Where("entrypoint_id IN ? AND resubmitted_from_id IS NULL", ids)
|
||||
|
||||
cutoff, finite := webhook.RetentionCutoff(now)
|
||||
if finite {
|
||||
query = query.Where("created_at >= ?", cutoff)
|
||||
}
|
||||
|
||||
var counts []entrypointEvents
|
||||
|
||||
err = query.Group("entrypoint_id").Find(&counts).Error
|
||||
if err != nil {
|
||||
return fmt.Errorf("counting events by entrypoint: %w", err)
|
||||
}
|
||||
|
||||
for _, row := range totals {
|
||||
view := byID[row.EntrypointID]
|
||||
view.LastEvent = humanize.Time(row.LastEventAt)
|
||||
view.LastEventUTC =
|
||||
row.LastEventAt.UTC().Format(time.DateTime) + " UTC"
|
||||
}
|
||||
|
||||
for _, row := range counts {
|
||||
byID[row.EntrypointID].Events = row.Events
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,197 @@
|
||||
package handlers_test
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/gorm/clause"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/handlers"
|
||||
"sneak.berlin/go/webhooker/internal/logger"
|
||||
"sneak.berlin/go/webhooker/internal/session"
|
||||
)
|
||||
|
||||
// entrypointRow returns the part of a rendered webhook page from an
|
||||
// entrypoint's URL to the next entrypoint's, which holds its figures.
|
||||
func entrypointRow(t *testing.T, page, entrypointID string) string {
|
||||
t.Helper()
|
||||
|
||||
_, row, found := strings.Cut(page, `id="entrypoint-url-`+entrypointID+`"`)
|
||||
require.True(t, found)
|
||||
|
||||
row, _, _ = strings.Cut(row, `id="entrypoint-url-`)
|
||||
|
||||
return row
|
||||
}
|
||||
|
||||
// lastEventShown matches an entrypoint row's last event arriving at at.
|
||||
func lastEventShown(at time.Time) string {
|
||||
return `Last Event:</span>\s*<span title="` +
|
||||
at.UTC().Format(time.DateTime) + ` UTC">[^<]+</span>`
|
||||
}
|
||||
|
||||
// eventsShown matches an entrypoint row's count of n events.
|
||||
func eventsShown(n int) string {
|
||||
return `Events Within Retention:</span>\s*<span>` +
|
||||
strconv.Itoa(n) + `</span>`
|
||||
}
|
||||
|
||||
// TestHandleSourceDetail_ShowsEntrypointEvents proves each entrypoint
|
||||
// on the webhook page shows its own figures: how many events arrived
|
||||
// through it within the webhook's retention period, leaving out one
|
||||
// older than that, and when the newest arrived, or "never" for an
|
||||
// entrypoint with none.
|
||||
func TestHandleSourceDetail_ShowsEntrypointEvents(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
wh := &database.Webhook{
|
||||
UserID: deleteTestUserID, Name: "figures", RetentionDays: 7,
|
||||
}
|
||||
require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
|
||||
|
||||
webhookDB, err := dbMgr.GetDB(wh.ID)
|
||||
require.NoError(t, err)
|
||||
|
||||
entrypoint := func() *database.Entrypoint {
|
||||
ep := &database.Entrypoint{
|
||||
WebhookID: wh.ID, Path: uuid.New().String(), Active: true,
|
||||
}
|
||||
require.NoError(t,
|
||||
db.DB().Omit(clause.Associations).Create(ep).Error)
|
||||
|
||||
return ep
|
||||
}
|
||||
|
||||
// event stores an event that arrived on ep's URL age ago and
|
||||
// records it as ep's last event, as the receiver does.
|
||||
event := func(ep *database.Entrypoint, age time.Duration) time.Time {
|
||||
e := &database.Event{
|
||||
WebhookID: wh.ID,
|
||||
EntrypointID: ep.ID,
|
||||
Method: http.MethodPost,
|
||||
}
|
||||
e.CreatedAt = time.Now().Add(-age)
|
||||
require.NoError(t,
|
||||
webhookDB.Omit(clause.Associations).Create(e).Error)
|
||||
require.NoError(t, database.AddEntrypointTotals(webhookDB,
|
||||
database.EntrypointTotals{
|
||||
EntrypointID: ep.ID, LastEventAt: e.CreatedAt,
|
||||
}))
|
||||
|
||||
return e.CreatedAt
|
||||
}
|
||||
|
||||
busy, quiet, unused := entrypoint(), entrypoint(), entrypoint()
|
||||
|
||||
event(busy, 8*24*time.Hour) // older than the 7 days kept
|
||||
event(busy, 3*time.Hour)
|
||||
busyLast := event(busy, time.Hour)
|
||||
quietLast := event(quiet, 2*24*time.Hour)
|
||||
|
||||
body := renderSourceDetailPage(t, h, sess, wh.ID)
|
||||
|
||||
assert.Regexp(t, lastEventShown(busyLast), entrypointRow(t, body, busy.ID))
|
||||
assert.Regexp(t, eventsShown(2), entrypointRow(t, body, busy.ID))
|
||||
assert.Regexp(t, lastEventShown(quietLast), entrypointRow(t, body, quiet.ID))
|
||||
assert.Regexp(t, eventsShown(1), entrypointRow(t, body, quiet.ID))
|
||||
assert.Regexp(t, `Last Event:</span>\s*<span>never</span>`,
|
||||
entrypointRow(t, body, unused.ID))
|
||||
assert.Regexp(t, eventsShown(0), entrypointRow(t, body, unused.ID))
|
||||
}
|
||||
|
||||
// TestHandleSourceDetail_EntrypointLastEventSurvivesRetention checks
|
||||
// that once retention has removed every event that arrived on an
|
||||
// entrypoint's URL, the entrypoint still shows when the last one
|
||||
// arrived rather than "never".
|
||||
func TestHandleSourceDetail_EntrypointLastEventSurvivesRetention(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
log *logger.Logger
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr, &log)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
wh := &database.Webhook{
|
||||
UserID: deleteTestUserID, Name: "swept", RetentionDays: 1,
|
||||
}
|
||||
require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
|
||||
|
||||
ep := seedEntrypoint(t, db, wh.ID)
|
||||
webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, ep.Path, 1)
|
||||
arrived := events[0].CreatedAt
|
||||
|
||||
statsAge(t, webhookDB, events[0].ID, time.Now().Add(-50*time.Hour))
|
||||
statsPrune(t, db, dbMgr, log, webhookDB)
|
||||
require.Empty(t, listEvents(t, webhookDB))
|
||||
|
||||
row := entrypointRow(t, renderSourceDetailPage(t, h, sess, wh.ID), ep.ID)
|
||||
assert.Regexp(t, lastEventShown(arrived), row)
|
||||
assert.Regexp(t, eventsShown(0), row)
|
||||
}
|
||||
|
||||
// TestHandleSourceDetail_ResubmitLeavesEntrypointFigures checks that a
|
||||
// resubmitted copy, which did not arrive on the entrypoint's URL,
|
||||
// changes neither the entrypoint's last event nor its count.
|
||||
func TestHandleSourceDetail_ResubmitLeavesEntrypointFigures(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
wh := seedWebhook(t, db)
|
||||
ep := seedEntrypoint(t, db, wh.ID)
|
||||
webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, ep.Path, 1)
|
||||
arrived := events[0].CreatedAt
|
||||
|
||||
require.Equal(t, http.StatusSeeOther,
|
||||
postResubmit(t, h, sess, wh.ID, events[0].ID).Code)
|
||||
require.Len(t, listEvents(t, webhookDB), 2)
|
||||
|
||||
var totals database.EntrypointTotals
|
||||
|
||||
require.NoError(t, webhookDB.Take(&totals).Error)
|
||||
assert.True(t, arrived.Equal(totals.LastEventAt))
|
||||
|
||||
row := entrypointRow(t, renderSourceDetailPage(t, h, sess, wh.ID), ep.ID)
|
||||
assert.Regexp(t, lastEventShown(arrived), row)
|
||||
assert.Regexp(t, eventsShown(1), row)
|
||||
}
|
||||
@@ -485,6 +485,8 @@ func (h *Handlers) renderSourceDetail(
|
||||
"webhook_id = ?", webhook.ID,
|
||||
).Find(&targets)
|
||||
|
||||
entrypointViews := NewEntrypointViews(entrypoints)
|
||||
|
||||
var events []RecentEventView
|
||||
|
||||
if h.dbMgr.DBExists(webhook.ID) {
|
||||
@@ -503,6 +505,15 @@ func (h *Handlers) renderSourceDetail(
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
err = addEntrypointEvents(
|
||||
webhookDB, &webhook, entrypointViews, time.Now(),
|
||||
)
|
||||
if err != nil {
|
||||
h.serverError(w, r, "failed to count entrypoint events", err)
|
||||
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
scheme := "http"
|
||||
@@ -523,13 +534,13 @@ func (h *Handlers) renderSourceDetail(
|
||||
// Targets are projected to a display-safe view: a
|
||||
// target's stored config blob holds a credential, and it
|
||||
// must never reach a template.
|
||||
"Entrypoints": NewEntrypointViews(entrypoints),
|
||||
"Targets": h.targetRows(&webhook, targets),
|
||||
"Events": events,
|
||||
"BaseURL": baseURL,
|
||||
"Stats": h.loadWebhookStats(webhook.ID, entrypoints, targets),
|
||||
"TargetForm": targetForm,
|
||||
"TargetError": targetErr,
|
||||
"Entrypoints": entrypointViews,
|
||||
"Targets": h.targetRows(&webhook, targets),
|
||||
"Events": events,
|
||||
"BaseURL": baseURL,
|
||||
"Stats": h.loadWebhookStats(webhook.ID, entrypoints, targets),
|
||||
tmplKeyTargetForm: targetForm,
|
||||
"TargetError": targetErr,
|
||||
}
|
||||
|
||||
status := http.StatusOK
|
||||
@@ -565,12 +576,12 @@ func (h *Handlers) HandleSourceEdit() http.HandlerFunc {
|
||||
return
|
||||
}
|
||||
|
||||
data := map[string]any{
|
||||
tmplKeyWebhook: &webhook,
|
||||
tmplKeyError: "",
|
||||
}
|
||||
|
||||
h.renderTemplate(w, r, "source_edit.html", data)
|
||||
h.renderWebhookEdit(
|
||||
w, r, &webhook,
|
||||
webhook.Name, webhook.Description,
|
||||
strconv.Itoa(webhook.RetentionDays),
|
||||
"", http.StatusOK,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -616,7 +627,8 @@ func (h *Handlers) HandleSourceEditSubmit() http.HandlerFunc {
|
||||
}
|
||||
}
|
||||
|
||||
// applyWebhookEdit validates and saves webhook edits.
|
||||
// applyWebhookEdit validates and saves webhook edits. A refused save
|
||||
// shows the edit form again with the values submitted and the reason.
|
||||
func (h *Handlers) applyWebhookEdit(
|
||||
w http.ResponseWriter,
|
||||
r *http.Request,
|
||||
@@ -625,52 +637,52 @@ func (h *Handlers) applyWebhookEdit(
|
||||
// The body size cap is enforced by the MaxBodySize middleware,
|
||||
// which runs before CSRF parses the form.
|
||||
name := r.PostFormValue("name")
|
||||
if name == "" {
|
||||
data := map[string]any{
|
||||
tmplKeyWebhook: webhook,
|
||||
tmplKeyError: "Name is required",
|
||||
}
|
||||
description := r.PostFormValue("description")
|
||||
retention := r.PostFormValue("retention_days")
|
||||
|
||||
h.renderTemplateStatus(w, r, "source_edit.html", data, http.StatusBadRequest)
|
||||
if name == "" {
|
||||
h.renderWebhookEdit(
|
||||
w, r, webhook, name, description, retention,
|
||||
"Name is required", http.StatusBadRequest,
|
||||
)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
oldName := webhook.Name
|
||||
webhook.Name = name
|
||||
webhook.Description = r.PostFormValue("description")
|
||||
|
||||
// An empty field falls back to the stored value, so submitting the
|
||||
// form without touching retention leaves the policy alone.
|
||||
retentionDays, errMsg := parseRetentionDays(
|
||||
r.PostFormValue("retention_days"), webhook.RetentionDays,
|
||||
retention, webhook.RetentionDays,
|
||||
)
|
||||
if errMsg != "" {
|
||||
data := map[string]any{
|
||||
tmplKeyWebhook: webhook,
|
||||
tmplKeyError: errMsg,
|
||||
}
|
||||
|
||||
h.renderTemplateStatus(w, r, "source_edit.html", data, http.StatusBadRequest)
|
||||
h.renderWebhookEdit(
|
||||
w, r, webhook, name, description, retention,
|
||||
errMsg, http.StatusBadRequest,
|
||||
)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
webhook.RetentionDays = retentionDays
|
||||
// edited is the webhook as the submission leaves it; webhook stays
|
||||
// as stored, for the page shown again when the save is refused.
|
||||
edited := *webhook
|
||||
edited.Name = name
|
||||
edited.Description = description
|
||||
edited.RetentionDays = retentionDays
|
||||
|
||||
// A new name renames the archive files before it is saved (see
|
||||
// delivery.Engine.Rename). If either step fails, the same targets'
|
||||
// archives go back to the name that is still stored, without
|
||||
// reading the main database again.
|
||||
targets, err := h.renameWebhookArchives(
|
||||
webhook.ID, oldName, webhook.Name,
|
||||
webhook.ID, webhook.Name, edited.Name,
|
||||
)
|
||||
if err == nil {
|
||||
err = h.db.DB().Save(webhook).Error
|
||||
err = h.db.DB().Save(&edited).Error
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
restoreErr := h.renameArchives(targets, oldName)
|
||||
restoreErr := h.renameArchives(targets, webhook.Name)
|
||||
if restoreErr != nil {
|
||||
h.log.Error(
|
||||
"failed to rename archives back",
|
||||
@@ -680,15 +692,14 @@ func (h *Handlers) applyWebhookEdit(
|
||||
}
|
||||
|
||||
if errors.Is(err, delivery.ErrArchiveNameTaken) {
|
||||
data := map[string]any{
|
||||
tmplKeyWebhook: webhook,
|
||||
tmplKeyError: "Not saved: " + err.Error() +
|
||||
". Move that archive out of the data directory, " +
|
||||
"its .db together with any -wal and -shm beside " +
|
||||
h.renderWebhookEdit(
|
||||
w, r, webhook, name, description, retention,
|
||||
"Not saved: "+err.Error()+
|
||||
". Move that archive out of the data directory, "+
|
||||
"its .db together with any -wal and -shm beside "+
|
||||
"it, then save again.",
|
||||
}
|
||||
|
||||
h.renderTemplateStatus(w, r, "source_edit.html", data, http.StatusConflict)
|
||||
http.StatusConflict,
|
||||
)
|
||||
|
||||
return
|
||||
}
|
||||
@@ -704,6 +715,27 @@ func (h *Handlers) applyWebhookEdit(
|
||||
)
|
||||
}
|
||||
|
||||
// renderWebhookEdit renders the webhook edit page for the webhook as
|
||||
// stored, its form showing name, description and retentionDays, with
|
||||
// an optional error message above it.
|
||||
func (h *Handlers) renderWebhookEdit(
|
||||
w http.ResponseWriter,
|
||||
r *http.Request,
|
||||
webhook *database.Webhook,
|
||||
name, description, retentionDays, errMsg string,
|
||||
status int,
|
||||
) {
|
||||
data := map[string]any{
|
||||
tmplKeyWebhook: webhook,
|
||||
tmplKeyError: errMsg,
|
||||
"Name": name,
|
||||
"Description": description,
|
||||
"RetentionDays": retentionDays,
|
||||
}
|
||||
|
||||
h.renderTemplateStatus(w, r, "source_edit.html", data, status)
|
||||
}
|
||||
|
||||
// HandleSourceDelete handles webhook deletion.
|
||||
func (h *Handlers) HandleSourceDelete() http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
@@ -1575,49 +1607,57 @@ func (h *Handlers) newTarget(
|
||||
webhookID string,
|
||||
in targetFormInput,
|
||||
) (*database.Target, string, error) {
|
||||
if in.Name == "" {
|
||||
return nil, "Name is required", nil
|
||||
target := &database.Target{
|
||||
WebhookID: webhookID,
|
||||
Type: in.Type,
|
||||
Active: true,
|
||||
}
|
||||
|
||||
if !isValidTargetType(in.Type) {
|
||||
return nil, "Invalid target type", nil
|
||||
}
|
||||
|
||||
configJSON, errMsg, err := h.buildTargetConfig(ctx, in.Type, in)
|
||||
errMsg, err := h.setTargetFromForm(ctx, target, in)
|
||||
if err != nil || errMsg != "" {
|
||||
return nil, errMsg, err
|
||||
}
|
||||
|
||||
// A new target has no stored retry count, so an absent field
|
||||
// takes the fire-and-forget default. A field the operator filled
|
||||
// in with something invalid is refused rather than becoming
|
||||
// that default.
|
||||
maxRetries, err := parseMaxRetries(in.MaxRetries, 0)
|
||||
if err != nil {
|
||||
return nil, "Invalid max retries: " + retriesErrorMessage(err), nil
|
||||
}
|
||||
|
||||
return &database.Target{
|
||||
WebhookID: webhookID,
|
||||
Name: in.Name,
|
||||
Type: in.Type,
|
||||
Active: true,
|
||||
Config: configJSON,
|
||||
MaxRetries: maxRetries,
|
||||
}, "", nil
|
||||
return target, "", nil
|
||||
}
|
||||
|
||||
// isValidTargetType checks whether the target type is supported.
|
||||
func isValidTargetType(tt database.TargetType) bool {
|
||||
switch tt {
|
||||
case database.TargetTypeHTTP,
|
||||
database.TargetTypeDatabase,
|
||||
database.TargetTypeLog,
|
||||
database.TargetTypeSlack:
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
// setTargetFromForm validates a target form against the target's type
|
||||
// and, when it accepts it, sets the target's name, configuration and
|
||||
// retry count from it. It returns the message the form shows for
|
||||
// anything it refuses, an unknown type among them, and then leaves the
|
||||
// target unchanged; an error is the server's fault, as for newTarget.
|
||||
// The add target form and the target edit form both go through here,
|
||||
// so the two cannot come to disagree about what a target may be.
|
||||
func (h *Handlers) setTargetFromForm(
|
||||
ctx context.Context,
|
||||
target *database.Target,
|
||||
in targetFormInput,
|
||||
) (string, error) {
|
||||
if in.Name == "" {
|
||||
return "Name is required", nil
|
||||
}
|
||||
|
||||
configJSON, errMsg, err := h.buildTargetConfig(ctx, target.Type, in)
|
||||
if err != nil || errMsg != "" {
|
||||
return errMsg, err
|
||||
}
|
||||
|
||||
// An empty max_retries keeps the target's count: the
|
||||
// fire-and-forget default of 0 for a new target, and the stored
|
||||
// count for an edited one, since the forms for target types that
|
||||
// do not retry have no such field. A value that is filled in but
|
||||
// invalid is refused rather than becoming that count, so a typo
|
||||
// cannot destroy the count a target is delivering with.
|
||||
maxRetries, err := parseMaxRetries(in.MaxRetries, target.MaxRetries)
|
||||
if err != nil {
|
||||
return "Invalid max retries: " + retriesErrorMessage(err), nil
|
||||
}
|
||||
|
||||
target.Name = in.Name
|
||||
target.Config = configJSON
|
||||
target.MaxRetries = maxRetries
|
||||
|
||||
return "", nil
|
||||
}
|
||||
|
||||
// pageOrFirst parses a paginated page number, answering 1 for
|
||||
@@ -1639,9 +1679,10 @@ func pageOrFirst(s string) int {
|
||||
}
|
||||
|
||||
// targetFormInput carries the raw values of a target form. Both the
|
||||
// create and the edit path fill one and hand it to buildTargetConfig,
|
||||
// so neither can come to validate a destination differently from the
|
||||
// other. A refused add target form is shown again from it.
|
||||
// create and the edit path fill one and hand it to setTargetFromForm,
|
||||
// so neither can come to validate a target differently from the
|
||||
// other. Both forms are filled from one: the edit form with the
|
||||
// stored values, and a refused form with the values submitted.
|
||||
type targetFormInput struct {
|
||||
// Name is the target's name.
|
||||
Name string
|
||||
|
||||
@@ -509,6 +509,51 @@ func TestHandleSourceEditSubmit_InvalidRetentionIsRejected(
|
||||
)
|
||||
}
|
||||
|
||||
// TestHandleSourceEditSubmit_RefusedFormComesBack refuses an edit for
|
||||
// each reason the form can give and checks that the form comes back
|
||||
// with the reason and the name, description and retention submitted,
|
||||
// that the page still reports the stored retention, and that nothing
|
||||
// is saved.
|
||||
func TestHandleSourceEditSubmit_RefusedFormComesBack(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupSourceTest(t)
|
||||
|
||||
refused := func(name, retention, reason string) {
|
||||
t.Helper()
|
||||
|
||||
wh := seedWebhookWithRetention(t, env.db, 30)
|
||||
|
||||
submitted := wh
|
||||
submitted.Name = name
|
||||
submitted.Description = "a description worth keeping"
|
||||
|
||||
w := submitEdit(t, env, submitted, retention)
|
||||
assert.Equal(t, http.StatusBadRequest, w.Code)
|
||||
|
||||
page := w.Body.String()
|
||||
assert.Contains(t, page, `class="alert-error">`+reason)
|
||||
assert.Contains(t, page, `name="name" value="`+name+`"`)
|
||||
assert.Contains(t, page, ">a description worth keeping</textarea>")
|
||||
assert.Contains(
|
||||
t, page, `name="retention_days" value="`+retention+`"`,
|
||||
)
|
||||
assert.Contains(t, page, "Currently 30 days.")
|
||||
|
||||
var stored database.Webhook
|
||||
|
||||
require.NoError(
|
||||
t, env.db.DB().First(&stored, "id = ?", wh.ID).Error,
|
||||
)
|
||||
assert.Equal(t, wh.Name, stored.Name)
|
||||
assert.Empty(t, stored.Description)
|
||||
assert.Equal(t, 30, stored.RetentionDays)
|
||||
}
|
||||
|
||||
refused("", "45", "Name is required")
|
||||
refused("kept-name", "nonsense", "Retention must be")
|
||||
}
|
||||
|
||||
func TestHandleSourceEditSubmit_EmptyRetentionLeavesValueUnchanged(
|
||||
t *testing.T,
|
||||
) {
|
||||
@@ -809,6 +854,10 @@ func TestHandleSourceEditSubmit_ArchiveNameTaken(t *testing.T) {
|
||||
w := submitEdit(t, env, wh, "")
|
||||
require.Equal(t, http.StatusConflict, w.Code)
|
||||
assert.Contains(t, w.Body.String(), "archive-taken.db")
|
||||
assert.Contains(
|
||||
t, w.Body.String(), `name="name" value="`+renamedWebhookName+`"`,
|
||||
"the form comes back with the name submitted",
|
||||
)
|
||||
|
||||
var stored database.Webhook
|
||||
|
||||
|
||||
@@ -3,6 +3,7 @@ package handlers
|
||||
import (
|
||||
"errors"
|
||||
"net/http"
|
||||
"strconv"
|
||||
|
||||
"github.com/go-chi/chi"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
@@ -13,10 +14,13 @@ import (
|
||||
const targetEditTemplate = "target_edit.html"
|
||||
|
||||
// tmplKeyTarget is the template data key for the target being
|
||||
// edited, and tmplKeyMaxTimeout for the timeout ceiling the form
|
||||
// tells the user about.
|
||||
// edited, tmplKeyTargetForm for the values its form shows, and
|
||||
// tmplKeyMaxTimeout for the timeout ceiling the form tells the user
|
||||
// about. The add target form on the webhook page takes its values
|
||||
// under the same key as the edit form.
|
||||
const (
|
||||
tmplKeyTarget = "Target"
|
||||
tmplKeyTargetForm = "TargetForm"
|
||||
tmplKeyMaxTimeout = "MaxTimeout"
|
||||
)
|
||||
|
||||
@@ -28,20 +32,19 @@ const configUnreadableMessage = "The stored configuration for this " +
|
||||
"target could not be read. Enter the values below; saving " +
|
||||
"replaces the stored configuration."
|
||||
|
||||
// targetEditView is the display model for the target edit page.
|
||||
// targetEditView is the display model for the target edit page: the
|
||||
// target's row fields as stored. The values the form shows, the
|
||||
// UNMASKED configuration among them, come separately, as a
|
||||
// targetFormInput.
|
||||
//
|
||||
// It carries the target's row fields alongside its UNMASKED
|
||||
// configuration, and deliberately omits database.Target's raw
|
||||
// Config blob: the form renders named fields, and giving the
|
||||
// template the blob as well would put an unreviewed second path to
|
||||
// the credential on the page.
|
||||
// It deliberately omits database.Target's raw Config blob: the form
|
||||
// renders named fields, and giving the template the blob as well
|
||||
// would put an unreviewed second path to the credential on the page.
|
||||
type targetEditView struct {
|
||||
ID string
|
||||
Name string
|
||||
Type database.TargetType
|
||||
Active bool
|
||||
MaxRetries int
|
||||
Config delivery.TargetConfigForm
|
||||
ID string
|
||||
Name string
|
||||
Type database.TargetType
|
||||
Active bool
|
||||
}
|
||||
|
||||
// HandleTargetEdit shows the form to edit a target.
|
||||
@@ -73,7 +76,18 @@ func (h *Handlers) HandleTargetEdit() http.HandlerFunc {
|
||||
msg = configUnreadableMessage
|
||||
}
|
||||
|
||||
h.renderTargetEdit(w, r, webhook, target, cfg, msg)
|
||||
form := targetFormInput{
|
||||
Name: target.Name,
|
||||
URL: cfg.URL,
|
||||
Headers: cfg.Headers,
|
||||
Timeout: cfg.Timeout,
|
||||
MaxRetries: strconv.Itoa(target.MaxRetries),
|
||||
Expiry: cfg.Expiry,
|
||||
}
|
||||
|
||||
h.renderTargetEdit(
|
||||
w, r, webhook, target, form, msg, http.StatusOK,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -101,11 +115,12 @@ func (h *Handlers) HandleTargetEditSubmit() http.HandlerFunc {
|
||||
}
|
||||
}
|
||||
|
||||
// applyTargetEdit validates and saves target edits.
|
||||
// applyTargetEdit validates and saves target edits. A refused save
|
||||
// shows the edit form again with the values submitted and the reason.
|
||||
//
|
||||
// The submitted configuration goes through buildTargetConfig, the
|
||||
// same builder the create path uses, so an edited destination is
|
||||
// SSRF-validated exactly as a new one is.
|
||||
// The submission goes through setTargetFromForm, as a new target
|
||||
// does, so an edited destination is SSRF-validated exactly as a new
|
||||
// one is.
|
||||
//
|
||||
// The target's type is not editable. Each type stores a different
|
||||
// configuration shape and its delivery history is recorded against
|
||||
@@ -118,16 +133,13 @@ func (h *Handlers) applyTargetEdit(
|
||||
webhook database.Webhook,
|
||||
target *database.Target,
|
||||
) {
|
||||
name := r.PostFormValue("name")
|
||||
if name == "" {
|
||||
http.Error(w, "Name is required", http.StatusBadRequest)
|
||||
in := targetFormInputFrom(r)
|
||||
|
||||
return
|
||||
}
|
||||
// edited is the target as the submission leaves it; target stays
|
||||
// as stored, for the page shown again when the save is refused.
|
||||
edited := *target
|
||||
|
||||
configJSON, errMsg, err := h.buildTargetConfig(
|
||||
r.Context(), target.Type, targetFormInputFrom(r),
|
||||
)
|
||||
errMsg, err := h.setTargetFromForm(r.Context(), &edited, in)
|
||||
if err != nil {
|
||||
h.serverError(w, r, "failed to encode target config", err)
|
||||
|
||||
@@ -135,45 +147,26 @@ func (h *Handlers) applyTargetEdit(
|
||||
}
|
||||
|
||||
if errMsg != "" {
|
||||
http.Error(w, errMsg, http.StatusBadRequest)
|
||||
h.renderTargetEdit(
|
||||
w, r, webhook, target, in, errMsg, http.StatusBadRequest,
|
||||
)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
// Retries are offered only by the forms for target types that
|
||||
// retry, so an absent field means "this form does not edit
|
||||
// retries" rather than "set them to zero". Reading it
|
||||
// unconditionally would silently disable retries on any target
|
||||
// saved from a form that does not render the input.
|
||||
//
|
||||
// A field that IS submitted but does not parse is a 400, through
|
||||
// the same validator the create path uses. It is rejected before
|
||||
// anything is written, so a typo cannot destroy the retry count
|
||||
// the target is already delivering with.
|
||||
if r.PostForm.Has("max_retries") {
|
||||
retries, ok := targetMaxRetries(w, r, target.MaxRetries)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
target.MaxRetries = retries
|
||||
}
|
||||
|
||||
oldName := target.Name
|
||||
target.Name = name
|
||||
target.Config = configJSON
|
||||
|
||||
// A new name renames the archive file before it is saved (see
|
||||
// delivery.Engine.Rename). If either step fails, it goes back to
|
||||
// the name that is still stored.
|
||||
err = h.renameTargetArchive(target, webhook.Name, oldName, name)
|
||||
err = h.renameTargetArchive(
|
||||
target, webhook.Name, target.Name, edited.Name,
|
||||
)
|
||||
if err == nil {
|
||||
err = h.db.DB().Save(target).Error
|
||||
err = h.db.DB().Save(&edited).Error
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
restoreErr := h.renameTargetArchive(
|
||||
target, webhook.Name, name, oldName,
|
||||
target, webhook.Name, edited.Name, target.Name,
|
||||
)
|
||||
if restoreErr != nil {
|
||||
h.log.Error(
|
||||
@@ -184,8 +177,8 @@ func (h *Handlers) applyTargetEdit(
|
||||
}
|
||||
|
||||
if errors.Is(err, delivery.ErrArchiveNameTaken) {
|
||||
http.Error(
|
||||
w,
|
||||
h.renderTargetEdit(
|
||||
w, r, webhook, target, in,
|
||||
"Not saved: "+err.Error()+
|
||||
". Move that archive out of the data directory, "+
|
||||
"its .db together with any -wal and -shm beside "+
|
||||
@@ -222,15 +215,17 @@ func (h *Handlers) renameTargetArchive(
|
||||
return h.archives.Rename(target.ID, webhookName, newName)
|
||||
}
|
||||
|
||||
// renderTargetEdit renders the target edit page with an optional
|
||||
// error message.
|
||||
// renderTargetEdit renders the target edit page for the target as
|
||||
// stored, its form showing form's values, with an optional error
|
||||
// message above it.
|
||||
func (h *Handlers) renderTargetEdit(
|
||||
w http.ResponseWriter,
|
||||
r *http.Request,
|
||||
webhook database.Webhook,
|
||||
target *database.Target,
|
||||
cfg delivery.TargetConfigForm,
|
||||
form targetFormInput,
|
||||
errMsg string,
|
||||
status int,
|
||||
) {
|
||||
// The template calls Webhook methods, which take pointer
|
||||
// receivers; html/template cannot address a value stored in a
|
||||
@@ -238,18 +233,17 @@ func (h *Handlers) renderTargetEdit(
|
||||
data := map[string]any{
|
||||
tmplKeyWebhook: &webhook,
|
||||
tmplKeyTarget: targetEditView{
|
||||
ID: target.ID,
|
||||
Name: target.Name,
|
||||
Type: target.Type,
|
||||
Active: target.Active,
|
||||
MaxRetries: target.MaxRetries,
|
||||
Config: cfg,
|
||||
ID: target.ID,
|
||||
Name: target.Name,
|
||||
Type: target.Type,
|
||||
Active: target.Active,
|
||||
},
|
||||
tmplKeyTargetForm: form,
|
||||
tmplKeyMaxTimeout: delivery.MaxTargetTimeoutSeconds,
|
||||
tmplKeyError: errMsg,
|
||||
}
|
||||
|
||||
h.renderTemplate(w, r, targetEditTemplate, data)
|
||||
h.renderTemplateStatus(w, r, targetEditTemplate, data, status)
|
||||
}
|
||||
|
||||
// ownedTarget resolves the request's sourceID and targetID
|
||||
|
||||
@@ -565,6 +565,73 @@ func assertEditRejectsTimeout(
|
||||
)
|
||||
}
|
||||
|
||||
// TestHandleTargetEditSubmit_RefusedFormComesBack refuses an edit of
|
||||
// a target of each type and checks that the edit form comes back with
|
||||
// the reason and every value submitted, and that nothing is saved.
|
||||
func TestHandleTargetEditSubmit_RefusedFormComesBack(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupSourceTest(t)
|
||||
|
||||
// fields is what the operator submitted, as a query string.
|
||||
cases := []struct {
|
||||
targetType database.TargetType
|
||||
fields string
|
||||
reason string
|
||||
}{
|
||||
{
|
||||
database.TargetTypeHTTP,
|
||||
"name=edited&url=" + editBlockedURL +
|
||||
"&headers=X-Edited:+kept&timeout=12&max_retries=3",
|
||||
"Invalid target URL",
|
||||
},
|
||||
{
|
||||
database.TargetTypeSlack,
|
||||
"name=edited&url=" + editOriginalURL + "&max_retries=25",
|
||||
"Invalid max retries",
|
||||
},
|
||||
{
|
||||
database.TargetTypeDatabase, "name=edited&expiry=7d",
|
||||
"Invalid archive expiry",
|
||||
},
|
||||
{database.TargetTypeLog, "name=", "Name is required"},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(string(tc.targetType), func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
webhook := seedWebhookWithRetention(t, env.db, 30)
|
||||
target := seedTarget(t, env.db, webhook.ID, tc.targetType)
|
||||
|
||||
form, err := url.ParseQuery(tc.fields)
|
||||
require.NoError(t, err)
|
||||
|
||||
w := submitTargetEdit(env, webhook.ID, target.ID, form)
|
||||
assert.Equal(t, http.StatusBadRequest, w.Code)
|
||||
|
||||
page := w.Body.String()
|
||||
assert.Contains(t, page, `class="alert-error">`+tc.reason)
|
||||
|
||||
// headers is the form's one textarea; every other field is
|
||||
// an input.
|
||||
for field := range form {
|
||||
shown := `name="` + field + `" value="` + form.Get(field) + `"`
|
||||
if field == "headers" {
|
||||
shown = ">" + form.Get(field) + "</textarea>"
|
||||
}
|
||||
|
||||
assert.Contains(t, page, shown)
|
||||
}
|
||||
|
||||
assert.Equal(
|
||||
t, target.Name, storedTarget(t, env, target.ID).Name,
|
||||
"a refused edit must save nothing",
|
||||
)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestHandleTargetEdit_Scoping keeps the edit routes scoped the way
|
||||
// the delete and toggle routes are: ownership is decided by the
|
||||
// webhook, and the target is then scoped to it.
|
||||
@@ -700,6 +767,10 @@ func TestHandleTargetEditSubmit_RenamesArchive(t *testing.T) {
|
||||
w = submitTargetEdit(env, wh.ID, archive.ID, again)
|
||||
require.Equal(t, http.StatusConflict, w.Code)
|
||||
assert.Contains(t, w.Body.String(), "archive-taken.db")
|
||||
assert.Contains(
|
||||
t, w.Body.String(), `name="name" value="Again"`,
|
||||
"the form comes back with the name submitted",
|
||||
)
|
||||
assert.Equal(
|
||||
t, renamedTargetName, storedTarget(t, env, archive.ID).Name,
|
||||
)
|
||||
|
||||
@@ -2,11 +2,13 @@ package handlers
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
"github.com/dustin/go-humanize"
|
||||
"gorm.io/gorm"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
)
|
||||
@@ -15,11 +17,27 @@ import (
|
||||
type TargetRowView struct {
|
||||
delivery.TargetView
|
||||
|
||||
// Deliveries counts the target's delivered and failed deliveries,
|
||||
// and is nil when the webhook's event database could not be read.
|
||||
Deliveries *TargetDeliveries
|
||||
|
||||
// Archive is a database target's archive file, and nil for a target
|
||||
// of any other type.
|
||||
Archive *ArchiveFileView
|
||||
}
|
||||
|
||||
// TargetDeliveries is how many of a target's deliveries became
|
||||
// delivered and how many failed: in total, which retention does not
|
||||
// reduce, and in the last 24 hours. Deliveries still pending or
|
||||
// retrying count in neither.
|
||||
type TargetDeliveries struct {
|
||||
Delivered int64
|
||||
Failed int64
|
||||
|
||||
DeliveredLast24Hours int64
|
||||
FailedLast24Hours int64
|
||||
}
|
||||
|
||||
// ArchiveFileView is what a database target's row shows about its
|
||||
// archive file.
|
||||
type ArchiveFileView struct {
|
||||
@@ -45,10 +63,24 @@ func (h *Handlers) targetRows(
|
||||
views := delivery.NewTargetViews(targets)
|
||||
rows := make([]TargetRowView, len(views))
|
||||
|
||||
deliveries, err := h.loadTargetDeliveries(webhook.ID)
|
||||
if err != nil {
|
||||
h.log.Error(
|
||||
"failed to read target delivery counts",
|
||||
"webhook_id", webhook.ID,
|
||||
"error", err,
|
||||
)
|
||||
}
|
||||
|
||||
// NewTargetViews returns one view per target, in order.
|
||||
for i := range views {
|
||||
rows[i].TargetView = views[i]
|
||||
|
||||
if err == nil {
|
||||
counts := deliveries[targets[i].ID]
|
||||
rows[i].Deliveries = &counts
|
||||
}
|
||||
|
||||
if targets[i].Type == database.TargetTypeDatabase {
|
||||
rows[i].Archive = h.archiveFileView(webhook, &targets[i])
|
||||
}
|
||||
@@ -57,6 +89,64 @@ func (h *Handlers) targetRows(
|
||||
return rows
|
||||
}
|
||||
|
||||
// loadTargetDeliveries reads the delivery counts of a webhook's targets
|
||||
// from its event database, keyed by target. A target with no deliveries
|
||||
// is left out, and so is every target when the event database does not
|
||||
// exist yet, since opening it would create it.
|
||||
func (h *Handlers) loadTargetDeliveries(
|
||||
webhookID string,
|
||||
) (map[string]TargetDeliveries, error) {
|
||||
if !h.dbMgr.DBExists(webhookID) {
|
||||
return map[string]TargetDeliveries{}, nil
|
||||
}
|
||||
|
||||
webhookDB, err := h.dbMgr.GetDB(webhookID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return readTargetDeliveries(webhookDB, time.Now())
|
||||
}
|
||||
|
||||
// readTargetDeliveries counts each target's deliveries that became
|
||||
// delivered and those that failed: in total from the targets' running
|
||||
// totals, and in the 24 hours before now from the deliveries' status
|
||||
// index. Each is one query for all the targets, and neither reads every
|
||||
// stored delivery.
|
||||
func readTargetDeliveries(
|
||||
db *gorm.DB, now time.Time,
|
||||
) (map[string]TargetDeliveries, error) {
|
||||
var totals []database.TargetTotals
|
||||
|
||||
err := db.Find(&totals).Error
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("reading target totals: %w", err)
|
||||
}
|
||||
|
||||
lastDay, err := finishedByTarget(db, now.Add(-longWindow))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
byTarget := make(map[string]TargetDeliveries, len(totals))
|
||||
|
||||
for _, total := range totals {
|
||||
byTarget[total.TargetID] = TargetDeliveries{
|
||||
Delivered: total.Delivered,
|
||||
Failed: total.Failed,
|
||||
}
|
||||
}
|
||||
|
||||
for _, finished := range lastDay {
|
||||
counts := byTarget[finished.TargetID]
|
||||
counts.DeliveredLast24Hours = finished.Delivered
|
||||
counts.FailedLast24Hours = finished.Failed
|
||||
byTarget[finished.TargetID] = counts
|
||||
}
|
||||
|
||||
return byTarget, nil
|
||||
}
|
||||
|
||||
// archiveFileView describes a database target's archive file from the
|
||||
// file's metadata alone; the archive is never opened. The file is found
|
||||
// by the name the archive writer uses, so it follows a rename of the
|
||||
|
||||
@@ -3,6 +3,7 @@ package handlers_test
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -12,6 +13,7 @@ import (
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
"sneak.berlin/go/webhooker/internal/handlers"
|
||||
"sneak.berlin/go/webhooker/internal/logger"
|
||||
"sneak.berlin/go/webhooker/internal/session"
|
||||
)
|
||||
|
||||
@@ -69,3 +71,104 @@ func TestHandleSourceDetail_ShowsArchiveFile(t *testing.T) {
|
||||
assert.Contains(t, body, "not created yet")
|
||||
assert.NotContains(t, body, "Archive Size:")
|
||||
}
|
||||
|
||||
// targetList returns the text of the targets section in a rendered
|
||||
// webhook page, from its heading to the next heading, with the markup
|
||||
// taken out and each run of space made one space. Each target's row
|
||||
// then reads as its name, type, state and buttons, followed by the
|
||||
// lines below them.
|
||||
func targetList(t *testing.T, page string) string {
|
||||
t.Helper()
|
||||
|
||||
_, list, found := strings.Cut(page, ">Targets</h2>")
|
||||
require.True(t, found, "the page has no targets section")
|
||||
|
||||
list, _, _ = strings.Cut(list, "<h2")
|
||||
list = regexp.MustCompile(`<[^>]*>`).ReplaceAllString(list, " ")
|
||||
|
||||
return strings.Join(strings.Fields(list), " ")
|
||||
}
|
||||
|
||||
// TestHandleSourceDetail_ShowsTargetDeliveries checks each target row's
|
||||
// delivered and failed deliveries, in total and in the last 24 hours,
|
||||
// for the history seedStatsHistory builds, before and after the real
|
||||
// retention reaper removes the oldest event. The http target has one
|
||||
// delivered, one of them in the last 24 hours, and three failed, one of
|
||||
// them in the last 24 hours and one of them the oldest event's, which
|
||||
// retention removes without changing the total. The active log target
|
||||
// has two failed, both in the last 24 hours, and its pending and
|
||||
// retrying deliveries count in neither. The four inactive log targets
|
||||
// have none.
|
||||
func TestHandleSourceDetail_ShowsTargetDeliveries(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
log *logger.Logger
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr, &log)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
hist := seedStatsHistory(t, h, sess, db, dbMgr)
|
||||
|
||||
const (
|
||||
httpRow = "Delivered: 1 in total, 1 in the last 24 hours " +
|
||||
"Failed: 3 in total, 1 in the last 24 hours"
|
||||
activeLogRow = "t-log log Active Edit Deactivate Delete " +
|
||||
"Delivered: 0 in total, 0 in the last 24 hours " +
|
||||
"Failed: 2 in total, 2 in the last 24 hours"
|
||||
inactiveLogRow = "t-log log Inactive Edit Activate Delete " +
|
||||
"Delivered: 0 in total, 0 in the last 24 hours " +
|
||||
"Failed: 0 in total, 0 in the last 24 hours"
|
||||
)
|
||||
|
||||
list := targetList(t, renderSourceDetailPage(t, h, sess, hist.webhook.ID))
|
||||
assert.Equal(t, 1, strings.Count(list, httpRow))
|
||||
assert.Equal(t, 1, strings.Count(list, activeLogRow))
|
||||
assert.Equal(t, 4, strings.Count(list, inactiveLogRow))
|
||||
|
||||
statsPrune(t, db, dbMgr, log, hist.webhookDB)
|
||||
|
||||
list = targetList(t, renderSourceDetailPage(t, h, sess, hist.webhook.ID))
|
||||
assert.Equal(t, 1, strings.Count(list, httpRow))
|
||||
assert.Equal(t, 1, strings.Count(list, activeLogRow))
|
||||
assert.Equal(t, 4, strings.Count(list, inactiveLogRow))
|
||||
}
|
||||
|
||||
// TestHandleSourceDetail_TargetDeliveriesUnreadable checks that when the
|
||||
// webhook's event database cannot be read, each target's row says so
|
||||
// instead of showing zeros.
|
||||
func TestHandleSourceDetail_TargetDeliveriesUnreadable(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
wh := seedWebhook(t, db)
|
||||
seedTarget(t, db, wh.ID, database.TargetTypeLog)
|
||||
|
||||
webhookDB, err := dbMgr.GetDB(wh.ID)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t,
|
||||
webhookDB.Migrator().DropTable(&database.TargetTotals{}))
|
||||
|
||||
list := targetList(t, renderSourceDetailPage(t, h, sess, wh.ID))
|
||||
assert.Contains(t, list, "t-log log Active Edit Deactivate Delete "+
|
||||
"The delivery counts could not be read.")
|
||||
assert.NotContains(t, list, "Delivered:")
|
||||
}
|
||||
|
||||
@@ -44,9 +44,9 @@ func TestTargetRefusal_PrivateDestinationSaysHowToAllowIt(
|
||||
form.Set("type", string(targetType))
|
||||
form.Set("url", editBlockedURL)
|
||||
|
||||
// A refused add shows the webhook page again, where
|
||||
// the hint is HTML-escaped; a refused edit answers in
|
||||
// plain text.
|
||||
// A refused add shows the webhook page again, and a
|
||||
// refused edit the edit page, where the hint is
|
||||
// HTML-escaped.
|
||||
added := serveTarget(
|
||||
env, http.MethodPost, targetsPath, form,
|
||||
)
|
||||
@@ -76,7 +76,8 @@ func TestTargetRefusal_PrivateDestinationSaysHowToAllowIt(
|
||||
)
|
||||
assert.Equal(t, http.StatusBadRequest, edited.Code)
|
||||
assert.Contains(
|
||||
t, edited.Body.String(), privateRefusalHint,
|
||||
t, edited.Body.String(),
|
||||
html.EscapeString(privateRefusalHint),
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
@@ -2,7 +2,6 @@ package handlers
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
@@ -89,32 +88,3 @@ func retriesErrorMessage(err error) string {
|
||||
return errRetriesInvalid.Error() +
|
||||
", or 0 for fire-and-forget"
|
||||
}
|
||||
|
||||
// targetMaxRetries reads and validates max_retries from a target edit
|
||||
// submission, answering the request with a 400 and reporting false
|
||||
// when the value is set but invalid.
|
||||
//
|
||||
// It and the create path (newTarget) both use parseMaxRetries and
|
||||
// retriesErrorMessage, so the two cannot come to disagree about what a
|
||||
// valid retry count is. The wording matches the timeout control on
|
||||
// the same submission.
|
||||
func targetMaxRetries(
|
||||
w http.ResponseWriter,
|
||||
r *http.Request,
|
||||
fallback int,
|
||||
) (int, bool) {
|
||||
retries, err := parseMaxRetries(
|
||||
r.PostFormValue("max_retries"), fallback,
|
||||
)
|
||||
if err != nil {
|
||||
http.Error(
|
||||
w,
|
||||
"Invalid max retries: "+retriesErrorMessage(err),
|
||||
http.StatusBadRequest,
|
||||
)
|
||||
|
||||
return 0, false
|
||||
}
|
||||
|
||||
return retries, true
|
||||
}
|
||||
|
||||
@@ -398,21 +398,21 @@ func TestTargetFormMaxRetriesCopyMatchesBehaviour(t *testing.T) {
|
||||
)
|
||||
|
||||
// A slack target exercises the same max_retries field while needing
|
||||
// only Config.URL from the edit template, so the test data stays
|
||||
// minimal. The Target key mirrors the field names the template reads
|
||||
// off the handler's view value.
|
||||
// only a URL from the edit template, so the test data stays
|
||||
// minimal. The Target and TargetForm keys mirror the field names
|
||||
// the template reads off the handler's values.
|
||||
editBody := renderPage(
|
||||
t, h, sess, "target_edit.html", map[string]any{
|
||||
dataKeyWebhook: webhook,
|
||||
"Target": map[string]any{
|
||||
"ID": "tg-1",
|
||||
"Name": "t",
|
||||
"Type": "slack",
|
||||
"Active": true,
|
||||
"MaxRetries": 3,
|
||||
"Config": map[string]any{
|
||||
"URL": "https://hooks.slack.com/services/x",
|
||||
},
|
||||
"ID": "tg-1",
|
||||
"Name": "t",
|
||||
"Type": "slack",
|
||||
"Active": true,
|
||||
},
|
||||
"TargetForm": map[string]any{
|
||||
"URL": "https://hooks.slack.com/services/x",
|
||||
"MaxRetries": "3",
|
||||
},
|
||||
dataKeyError: "",
|
||||
},
|
||||
|
||||
@@ -326,6 +326,19 @@ func (h *Handlers) createAndFanOut(
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
// A resubmitted copy did not arrive on its entrypoint's URL, so it
|
||||
// leaves the entrypoint's last event as it is.
|
||||
if src.ResubmittedFromID == nil {
|
||||
err = database.AddEntrypointTotals(tx, database.EntrypointTotals{
|
||||
EntrypointID: event.EntrypointID, LastEventAt: event.CreatedAt,
|
||||
})
|
||||
if err != nil {
|
||||
tx.Rollback()
|
||||
|
||||
return nil, nil, err
|
||||
}
|
||||
}
|
||||
|
||||
err = tx.Commit().Error
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf(
|
||||
|
||||
@@ -47,7 +47,8 @@ const (
|
||||
// TestAlpineRunsUnderTheSecurityPolicy loads the webhook page and the
|
||||
// event log in a headless browser, served by the real router and so
|
||||
// under the real Content-Security-Policy, and checks that the pages'
|
||||
// Alpine.js directives and the copy control work.
|
||||
// Alpine.js directives and the copy control work, and that a target's
|
||||
// row shows its delivery counts.
|
||||
func TestAlpineRunsUnderTheSecurityPolicy(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -118,6 +119,10 @@ func TestAlpineRunsUnderTheSecurityPolicy(t *testing.T) {
|
||||
}
|
||||
|
||||
checkRefusedTarget(ctx, t, page)
|
||||
checkTargetDeliveries(ctx, t, page, target.Name,
|
||||
"0 in total, 0 in the last 24 hours",
|
||||
"1 in total, 1 in the last 24 hours")
|
||||
checkRefusedEdits(ctx, t, page, target.ID)
|
||||
checkCopy(ctx, t, page)
|
||||
checkEntrypointEdit(ctx, t, page, page+"/events")
|
||||
checkRecentEvents(ctx, t, page)
|
||||
@@ -452,6 +457,86 @@ func checkRefusedTarget(ctx context.Context, t *testing.T, url string) {
|
||||
assert.Empty(t, typed, "after Cancel, the next Add keeps the url entered")
|
||||
}
|
||||
|
||||
// checkTargetDeliveries loads a webhook page and checks that the row of
|
||||
// the target named name shows delivered and failed beside its
|
||||
// "Delivered:" and "Failed:" labels.
|
||||
func checkTargetDeliveries(
|
||||
ctx context.Context, t *testing.T, url, name, delivered, failed string,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
row := `//span[text()="` + name + `"]/ancestor::div[@class="p-4"][1]`
|
||||
figure := func(label, value string) string {
|
||||
return row + `//span[text()="` + label +
|
||||
`"]/following-sibling::span[text()="` + value + `"]`
|
||||
}
|
||||
|
||||
require.NoError(t, chromedp.Run(ctx, loadPage(url)))
|
||||
|
||||
assert.Truef(t, shown(ctx, figure("Delivered:", delivered)),
|
||||
"the row of %s does not show %q delivered", name, delivered)
|
||||
assert.Truef(t, shown(ctx, figure("Failed:", failed)),
|
||||
"the row of %s does not show %q failed", name, failed)
|
||||
}
|
||||
|
||||
// checkRefusedEdits fills in the target edit page and the webhook edit
|
||||
// page of a webhook page with values the server refuses, a loopback
|
||||
// destination and a retention above the longest finite one, which the
|
||||
// browser lets through. It saves each and checks that the page comes
|
||||
// back with the reason and every value still in its field. The values
|
||||
// are keyed by the id of their field.
|
||||
func checkRefusedEdits(
|
||||
ctx context.Context, t *testing.T, page, targetID string,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
const reason = `//div[@class="alert-error"]`
|
||||
|
||||
edits := []struct {
|
||||
url string
|
||||
values map[string]string
|
||||
}{
|
||||
{page + "/targets/" + targetID + "/edit", map[string]string{
|
||||
"#name": "edited-target",
|
||||
"#url": "http://127.0.0.1/hook",
|
||||
"#headers": "X-Edited: kept",
|
||||
"#timeout": "12",
|
||||
"#max_retries": "3",
|
||||
}},
|
||||
{page + "/edit", map[string]string{
|
||||
"#name": "edited-webhook",
|
||||
"#description": "kept description",
|
||||
"#retention_days": "200000",
|
||||
}},
|
||||
}
|
||||
|
||||
for _, edit := range edits {
|
||||
require.NoError(t, chromedp.Run(ctx, loadPage(edit.url)))
|
||||
|
||||
for field, value := range edit.values {
|
||||
require.NoError(t, chromedp.Run(
|
||||
ctx, chromedp.SetValue(field, value, chromedp.ByQuery),
|
||||
))
|
||||
}
|
||||
|
||||
click(ctx, t, `//button[text()="Save Changes"]`)
|
||||
|
||||
assert.Truef(t, shown(ctx, reason),
|
||||
"%s: a refused save does not show the reason", edit.url)
|
||||
|
||||
for field, value := range edit.values {
|
||||
var kept string
|
||||
|
||||
require.NoError(t, chromedp.Run(
|
||||
ctx, chromedp.Value(field, &kept, chromedp.ByQuery),
|
||||
))
|
||||
assert.Equalf(t, value, kept,
|
||||
"%s: a refused save does not keep the %s entered",
|
||||
edit.url, field)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// checkCopy loads a webhook page and checks that the Copy control beside
|
||||
// its entrypoint's URL is a button, and that clicking it copies the URL
|
||||
// and says so: the button reads "Copied" only once the copy succeeded.
|
||||
|
||||
@@ -11,6 +11,7 @@ import (
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/stretchr/testify/assert"
|
||||
@@ -439,7 +440,8 @@ func (e *testEnv) storedEntrypoint(
|
||||
}
|
||||
|
||||
// seedFailedDelivery records a terminally failed delivery of an event
|
||||
// to a target in the webhook's own database.
|
||||
// to a target in the webhook's own database, as the delivery engine
|
||||
// leaves one: finished now, and counted in its target's totals.
|
||||
func (e *testEnv) seedFailedDelivery(
|
||||
t *testing.T,
|
||||
webhookID, eventID, targetID string,
|
||||
@@ -449,16 +451,21 @@ func (e *testEnv) seedFailedDelivery(
|
||||
webhookDB, err := e.dbMgr.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
|
||||
finishedAt := time.Now()
|
||||
dlv := &database.Delivery{
|
||||
EventID: eventID,
|
||||
TargetID: targetID,
|
||||
Status: database.DeliveryStatusFailed,
|
||||
EventID: eventID,
|
||||
TargetID: targetID,
|
||||
Status: database.DeliveryStatusFailed,
|
||||
FinishedAt: &finishedAt,
|
||||
}
|
||||
|
||||
require.NoError(
|
||||
t,
|
||||
webhookDB.Omit(clause.Associations).Create(dlv).Error,
|
||||
)
|
||||
require.NoError(t, database.AddTargetTotals(webhookDB,
|
||||
database.TargetTotals{TargetID: targetID, Deliveries: 1, Failed: 1},
|
||||
))
|
||||
|
||||
return dlv
|
||||
}
|
||||
|
||||
@@ -97,6 +97,18 @@
|
||||
</div>
|
||||
<!-- The URL above is the entrypoint's credential:
|
||||
anyone holding it can submit events. -->
|
||||
<div class="text-xs text-gray-500 mt-1">
|
||||
<span class="font-medium text-gray-700">Last Event:</span>
|
||||
{{if .LastEvent}}
|
||||
<span title="{{.LastEventUTC}}">{{.LastEvent}}</span>
|
||||
{{else}}
|
||||
<span>never</span>
|
||||
{{end}}
|
||||
</div>
|
||||
<div class="text-xs text-gray-500 mt-1">
|
||||
<span class="font-medium text-gray-700">Events Within Retention:</span>
|
||||
<span>{{.Events}}</span>
|
||||
</div>
|
||||
</div>
|
||||
{{else}}
|
||||
<div class="p-4 text-sm text-gray-500">No entrypoints configured.</div>
|
||||
@@ -259,6 +271,18 @@
|
||||
</div>
|
||||
{{end}}
|
||||
{{end}}
|
||||
{{with .Deliveries}}
|
||||
<div class="text-xs text-gray-500 mt-1">
|
||||
<span class="font-medium text-gray-700">Delivered:</span>
|
||||
<span>{{.Delivered}} in total, {{.DeliveredLast24Hours}} in the last 24 hours</span>
|
||||
</div>
|
||||
<div class="text-xs text-gray-500 mt-1">
|
||||
<span class="font-medium text-gray-700">Failed:</span>
|
||||
<span>{{.Failed}} in total, {{.FailedLast24Hours}} in the last 24 hours</span>
|
||||
</div>
|
||||
{{else}}
|
||||
<div class="text-xs text-gray-500 mt-1">The delivery counts could not be read.</div>
|
||||
{{end}}
|
||||
</div>
|
||||
{{else}}
|
||||
<div class="p-4 text-sm text-gray-500">No targets configured.</div>
|
||||
|
||||
@@ -18,17 +18,17 @@
|
||||
<input type="hidden" name="csrf_token" value="{{.CSRFToken}}">
|
||||
<div class="form-group">
|
||||
<label for="name" class="label">Name</label>
|
||||
<input type="text" id="name" name="name" value="{{.Webhook.Name}}" required class="input">
|
||||
<input type="text" id="name" name="name" value="{{.Name}}" required class="input">
|
||||
</div>
|
||||
|
||||
<div class="form-group">
|
||||
<label for="description" class="label">Description</label>
|
||||
<textarea id="description" name="description" rows="3" class="input">{{.Webhook.Description}}</textarea>
|
||||
<textarea id="description" name="description" rows="3" class="input">{{.Description}}</textarea>
|
||||
</div>
|
||||
|
||||
<div class="form-group">
|
||||
<label for="retention_days" class="label">Retention (days)</label>
|
||||
<input type="number" id="retention_days" name="retention_days" value="{{.Webhook.RetentionDays}}" min="0" class="input">
|
||||
<input type="number" id="retention_days" name="retention_days" value="{{.RetentionDays}}" min="0" class="input">
|
||||
<p class="text-xs text-gray-500 mt-1">Currently {{.Webhook.RetentionLabel}}.{{if .Webhook.RetainsForever}} No events are deleted while retention is set to forever.{{else}} A periodic cleanup permanently deletes events older than this, along with their delivery records.{{end}} Enter 0 to retain events forever; leave blank to keep the current setting.</p>
|
||||
</div>
|
||||
|
||||
|
||||
@@ -17,7 +17,7 @@
|
||||
|
||||
{{if or (eq .Target.Type "http") (eq .Target.Type "slack")}}
|
||||
<div class="mb-6 rounded-md bg-gray-50 p-4 text-sm text-gray-700">
|
||||
This form shows the target's stored destination in full, including any credential carried in its URL or headers. It is the only page that does; everywhere else the value is masked.
|
||||
This form shows the target's destination in full, including any credential carried in its URL or headers. It is the only page that does; everywhere else the value is masked.
|
||||
</div>
|
||||
{{end}}
|
||||
|
||||
@@ -26,25 +26,25 @@
|
||||
|
||||
<div class="form-group">
|
||||
<label for="name" class="label">Name</label>
|
||||
<input type="text" id="name" name="name" value="{{.Target.Name}}" required class="input">
|
||||
<input type="text" id="name" name="name" value="{{.TargetForm.Name}}" required class="input">
|
||||
</div>
|
||||
|
||||
{{if eq .Target.Type "http"}}
|
||||
<div class="form-group">
|
||||
<label for="url" class="label">Destination URL</label>
|
||||
<input type="url" id="url" name="url" value="{{.Target.Config.URL}}" required class="input">
|
||||
<input type="url" id="url" name="url" value="{{.TargetForm.URL}}" required class="input">
|
||||
<p class="text-xs text-gray-500 mt-1">Revalidated on save; destinations that resolve to private or link-local addresses are rejected.</p>
|
||||
</div>
|
||||
|
||||
<div class="form-group">
|
||||
<label for="headers" class="label">Headers</label>
|
||||
<textarea id="headers" name="headers" rows="4" class="input" placeholder="Authorization: Bearer ...">{{.Target.Config.Headers}}</textarea>
|
||||
<textarea id="headers" name="headers" rows="4" class="input" placeholder="Authorization: Bearer ...">{{.TargetForm.Headers}}</textarea>
|
||||
<p class="text-xs text-gray-500 mt-1">One <code>Name: value</code> per line, sent with every delivery. Leave blank for none. <code>Host</code>, <code>Content-Length</code>, <code>Transfer-Encoding</code>, <code>Connection</code>, <code>Trailer</code> and <code>User-Agent</code> are set by the delivery engine and are rejected here rather than silently ignored. Headers set here are dropped if a redirect leaves the destination's own origin, so a credential cannot follow one to another host.</p>
|
||||
</div>
|
||||
|
||||
<div class="form-group">
|
||||
<label for="timeout" class="label">Timeout (seconds)</label>
|
||||
<input type="number" id="timeout" name="timeout" value="{{.Target.Config.Timeout}}" min="0" max="{{.MaxTimeout}}" class="input">
|
||||
<input type="number" id="timeout" name="timeout" value="{{.TargetForm.Timeout}}" min="0" max="{{.MaxTimeout}}" class="input">
|
||||
<p class="text-xs text-gray-500 mt-1">Per-request timeout, at most {{.MaxTimeout}} seconds. Leave blank to use the default.</p>
|
||||
</div>
|
||||
{{end}}
|
||||
@@ -52,7 +52,7 @@
|
||||
{{if eq .Target.Type "slack"}}
|
||||
<div class="form-group">
|
||||
<label for="url" class="label">Webhook URL</label>
|
||||
<input type="url" id="url" name="url" value="{{.Target.Config.URL}}" required class="input">
|
||||
<input type="url" id="url" name="url" value="{{.TargetForm.URL}}" required class="input">
|
||||
<p class="text-xs text-gray-500 mt-1">Slack or Mattermost incoming webhook URL. Revalidated on save.</p>
|
||||
</div>
|
||||
{{end}}
|
||||
@@ -60,7 +60,7 @@
|
||||
{{if eq .Target.Type "database"}}
|
||||
<div class="form-group">
|
||||
<label for="expiry" class="label">Archive Expiry</label>
|
||||
<input type="text" id="expiry" name="expiry" value="{{.Target.Config.Expiry}}" placeholder="never" class="input">
|
||||
<input type="text" id="expiry" name="expiry" value="{{.TargetForm.Expiry}}" placeholder="never" class="input">
|
||||
<p class="text-xs text-gray-500 mt-1">"never" (the default when blank) keeps archived rows forever, or a Go duration like "720h" prunes older rows.</p>
|
||||
</div>
|
||||
{{end}}
|
||||
@@ -68,7 +68,7 @@
|
||||
{{if or (eq .Target.Type "http") (eq .Target.Type "slack")}}
|
||||
<div class="form-group">
|
||||
<label for="max_retries" class="label">Max retries</label>
|
||||
<input type="number" id="max_retries" name="max_retries" value="{{.Target.MaxRetries}}" min="0" max="20" class="input">
|
||||
<input type="number" id="max_retries" name="max_retries" value="{{.TargetForm.MaxRetries}}" min="0" max="20" class="input">
|
||||
<p class="text-xs text-gray-500 mt-1">This is the total number of delivery attempts, not retries on top of the first: a value of 3 makes three attempts in all. 0 means a single attempt with no retries and no circuit breaker.</p>
|
||||
</div>
|
||||
{{end}}
|
||||
|
||||
Reference in New Issue
Block a user