Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 clawbot
|
4bc7a1bce4 |
@@ -7,7 +7,7 @@ services, durably stores them, and delivers them to configured targets
|
||||
with retry support, logging, and observability. Category: infrastructure
|
||||
/ web service. License: MIT.
|
||||
|
||||
Each entrypoint is a version 4 UUID served at `/h/{uuid}`, and
|
||||
Each entrypoint is a version 4 UUID served at `/webhook/{uuid}`, and
|
||||
that UUID is the entrypoint's only credential. webhooker does not use
|
||||
shared secrets, HMAC signatures or token headers on the receiver, and
|
||||
will not add them — read
|
||||
@@ -1075,7 +1075,7 @@ unconditionally against whatever files it finds:
|
||||
- the main database on connect — `Setting`, `User`, `APIKey`, `Webhook`,
|
||||
`Entrypoint`, `Target`
|
||||
- each event database when it is lazily opened — `Event`, `Delivery`,
|
||||
`DeliveryResult`, `EventTotals`, `TargetTotals`
|
||||
`DeliveryResult`
|
||||
- each archive database on every open and reopen
|
||||
|
||||
There is no schema version table, no migration ledger, and no down
|
||||
@@ -1188,7 +1188,7 @@ backups at rest and restrict who can read them.
|
||||
|
||||
**The entrypoint UUID is the credential, and it is the only one.**
|
||||
webhooker mints a version 4 UUID per entrypoint and serves it at
|
||||
`/h/{uuid}`. Possession of that URL is the authentication:
|
||||
`/webhook/{uuid}`. Possession of that URL is the authentication:
|
||||
anyone who holds it can submit events to the entrypoint, and the
|
||||
receiver verifies nothing else about the sender.
|
||||
|
||||
@@ -1392,7 +1392,7 @@ The codebase uses consistent naming throughout (rename completed in
|
||||
|
||||
### Data Model
|
||||
|
||||
webhooker's data model has eleven entities organized into two tiers: the
|
||||
webhooker's data model has nine entities organized into two tiers: the
|
||||
**application tier** (user and webhook configuration) and the **event
|
||||
tier** (event ingestion, delivery, and logging).
|
||||
|
||||
@@ -1421,13 +1421,6 @@ tier** (event ingestion, delivery, and logging).
|
||||
│ ┌──────────┐ ┌──────────┐ ┌─────────────────┐ │
|
||||
│ │ Event │──1:N──│ Delivery │──1:N──│ DeliveryResult │ │
|
||||
│ └──────────┘ └──────────┘ └─────────────────┘ │
|
||||
│ │
|
||||
│ ┌──────────────┐ (one row: running counts of events) │
|
||||
│ │ EventTotals │ │
|
||||
│ └──────────────┘ │
|
||||
│ ┌──────────────┐ (one row per target: running counts │
|
||||
│ │ TargetTotals │ of its deliveries) │
|
||||
│ └──────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
@@ -1530,7 +1523,7 @@ the full request and creates an Event.
|
||||
| -------------- | ------- | ----------- |
|
||||
| `id` | UUID | Primary key |
|
||||
| `webhook_id` | UUID | Foreign key → Webhook |
|
||||
| `path` | string | Unique bare UUID, generated at creation. The `/h/` prefix is route only and is not stored: the receiver matches this column against the raw `{uuid}` path segment. It is also the entrypoint's credential; see [The entrypoint URL is the authentication secret](#the-entrypoint-url-is-the-authentication-secret) |
|
||||
| `path` | string | Unique bare UUID, generated at creation. The `/webhook/` prefix is route only and is not stored: the receiver matches this column against the raw `{uuid}` path segment. It is also the entrypoint's credential; see [The entrypoint URL is the authentication secret](#the-entrypoint-url-is-the-authentication-secret) |
|
||||
| `description` | string | Optional description |
|
||||
| `active` | boolean | Whether this entrypoint accepts events (default: true) |
|
||||
|
||||
@@ -1681,7 +1674,6 @@ status across potentially multiple attempts.
|
||||
| `event_id` | UUID | Foreign key → Event |
|
||||
| `target_id`| UUID | Foreign key → Target |
|
||||
| `status` | DeliveryStatus | One of: `pending`, `delivered`, `failed`, `retrying` |
|
||||
| `finished_at` | timestamp | When the delivery became `delivered` or `failed` (nullable; empty while `pending` or `retrying`) |
|
||||
|
||||
**Relations:** Belongs to Event. Belongs to Target. Has many
|
||||
DeliveryResults.
|
||||
@@ -1749,78 +1741,33 @@ retries) is individually logged for full observability.
|
||||
|
||||
**Relations:** Belongs to Delivery.
|
||||
|
||||
#### EventTotals and TargetTotals
|
||||
|
||||
Running counts in each event database, read by the statistics pane at the
|
||||
top of the webhook page and by the webhook list. `EventTotals` is one row:
|
||||
|
||||
| Field | Type | Description |
|
||||
| ---------------- | --------- | ----------- |
|
||||
| `events` | integer | Events ever stored, resubmitted copies included |
|
||||
| `events_removed` | integer | Events retention has deleted |
|
||||
| `last_event_at` | timestamp | When the newest event arrived (nullable; empty before the first); retention leaves it as it is |
|
||||
|
||||
`TargetTotals` is one row per target, created by the first delivery to it:
|
||||
|
||||
| Field | Type | Description |
|
||||
| -------------------- | ------- | ----------- |
|
||||
| `target_id` | UUID | The target (primary key) |
|
||||
| `deliveries` | integer | Deliveries to it ever created, replays included |
|
||||
| `delivered` | integer | Of those, how many became `delivered` |
|
||||
| `failed` | integer | Of those, how many became `failed` |
|
||||
| `deliveries_removed` | integer | Its deliveries retention has deleted |
|
||||
| `failed_removed` | integer | Its failed deliveries retention has deleted |
|
||||
|
||||
Each count changes in the transaction that writes or deletes the rows it
|
||||
counts. The pane's lifetime events are `events`, and its lifetime
|
||||
deliveries and failures are `deliveries` and `failed` summed over the
|
||||
targets; each figure within retention is the same less what retention
|
||||
removed, so neither needs the rows themselves. Its last event is
|
||||
`last_event_at`, written in the transaction that stores the event, so it
|
||||
still shows once retention has removed every event. Its last-10-minutes and
|
||||
last-24-hours figures are counted from the `events` and `deliveries`
|
||||
indexes over just that window, the deliveries in one query grouped by
|
||||
target. Its failure percentage for a window is the deliveries that became
|
||||
`failed` in it out of all that became `delivered` or `failed` in it, and
|
||||
a dash when none did.
|
||||
|
||||
The webhook list at `/hooks` shows three of the pane's figures for each
|
||||
webhook: its events within retention and its last event, both from
|
||||
`EventTotals`, and its deliveries that failed in the last 24 hours,
|
||||
counted with the pane's query. It opens each webhook's event database once
|
||||
(the handle stays open) and runs those two reads there, so its cost grows
|
||||
with the number of webhooks and, for each, with the deliveries that
|
||||
finished in the last 24 hours, never with the events stored.
|
||||
|
||||
#### Event-tier indexes
|
||||
|
||||
These indexes on the per-webhook event databases are declared in the model
|
||||
tags, so `AutoMigrate` creates them on a fresh database:
|
||||
tags, so `AutoMigrate` creates them on a fresh and on an existing database:
|
||||
|
||||
| Table | Columns | Serves |
|
||||
| ------------------ | --------------------------- | ------ |
|
||||
| `deliveries` | `status`, `deleted_at`, `finished_at`, `target_id` | Startup recovery, the retry and pending sweeps every 60 seconds and the queue-depth sampler every 30 seconds, which select deliveries by status, and the webhook page's statistics and the webhook list, which count each target's deliveries by status and when they finished |
|
||||
| `deliveries` | `event_id`, `deleted_at` | The event log, which loads each event's deliveries, and retention, which counts and deletes the deliveries of expired events |
|
||||
| `deliveries` | `status`, `deleted_at` | Startup recovery, the retry and pending sweeps every 60 seconds and the queue-depth sampler every 30 seconds, which select deliveries by status |
|
||||
| `deliveries` | `event_id`, `deleted_at` | The event log, which loads each event's deliveries, and retention, which selects and deletes the deliveries of expired events |
|
||||
| `delivery_results` | `delivery_id`, `deleted_at` | The event log, which loads the attempts of a page's deliveries, and retention, which deletes the attempts of expired events |
|
||||
| `events` | `deleted_at`, `created_at` | The webhook page's statistics, which count recent events |
|
||||
| `events` | `created_at` | Retention, which selects expired events by age |
|
||||
| `events` | `deleted_at`, `created_at` | Retention, which selects expired events by age |
|
||||
| `events` | `created_at` | Retention's delete of the expired events themselves |
|
||||
|
||||
GORM's soft delete adds `deleted_at IS NULL` to these queries; retention
|
||||
leaves it out. SQLite keeps no statistics on these tables, and without them it
|
||||
rates the `deleted_at` index, which every live row matches, above an index on
|
||||
a column matched against several values or compared with a range. So every
|
||||
index but the last also covers `deleted_at`. It comes second, so that
|
||||
retention can use the index without it, except in `events`, where the
|
||||
statistics compare `created_at` with a range (`>=`) and SQLite narrows by a
|
||||
range only on the last column it uses.
|
||||
GORM's soft delete adds `deleted_at IS NULL` to these queries; retention's
|
||||
deletes leave it out, but their lookups of expired rows keep it. SQLite keeps
|
||||
no statistics on these tables, and without them it rates the `deleted_at`
|
||||
index, which every live row matches, above an index on a column matched
|
||||
against several values or compared with `<`. So every index but the last also
|
||||
covers `deleted_at`. It comes second, so that retention's deletes can use the
|
||||
index without it, except in `events`, where `created_at` is compared with `<`
|
||||
and SQLite narrows by a `<` only on the last column it uses.
|
||||
|
||||
#### Common Fields
|
||||
|
||||
Every entity except `Setting`, `EventTotals` and `TargetTotals` includes
|
||||
these fields from `BaseModel`. `Setting` is a bare key-value row with no
|
||||
`id`, no timestamps and no soft delete, and the two totals tables hold
|
||||
counts, plus `last_event_at` in `event_totals`, keyed by a numeric `id`
|
||||
and by `target_id`:
|
||||
Every entity except `Setting` includes these fields from `BaseModel`.
|
||||
`Setting` is a bare key-value row with no `id`, no timestamps and no
|
||||
soft delete:
|
||||
|
||||
| Field | Type | Description |
|
||||
| ------------ | --------- | ----------- |
|
||||
@@ -1862,8 +1809,6 @@ encryption key is generated and stored, and an `admin` user is created.
|
||||
- **Events** — captured incoming webhook payloads
|
||||
- **Deliveries** — event-to-target pairings and their status
|
||||
- **DeliveryResults** — individual delivery attempt logs
|
||||
- **EventTotals** and **TargetTotals** — running counts of the above,
|
||||
the deliveries per target, kept through retention
|
||||
|
||||
Per-webhook databases are created automatically when a webhook is
|
||||
created (and lazily on first access for webhooks that predate this
|
||||
@@ -1960,7 +1905,7 @@ runtime, though CGO is required at build time due to the transitive
|
||||
```
|
||||
External Service
|
||||
│
|
||||
│ POST /h/{uuid}
|
||||
│ POST /webhook/{uuid}
|
||||
▼
|
||||
┌─────────────┐ ┌──────────────┐ ┌──────────────┐
|
||||
│ chi Router │────►│ Middleware │────►│ Webhook │
|
||||
@@ -2186,7 +2131,7 @@ The middleware records three more on the same registry:
|
||||
Two of those labels are written once per request from bytes the client
|
||||
chose, so both are bounded to something this service registers:
|
||||
|
||||
- `handler` is the chi route pattern — `/h/{uuid}`, never the
|
||||
- `handler` is the chi route pattern — `/webhook/{uuid}`, never the
|
||||
concrete path. A request matching no route carries `(unmatched)`,
|
||||
and no entrypoint UUID ever reaches a label.
|
||||
- `method` is the request method when the router can route it, and
|
||||
@@ -2216,7 +2161,7 @@ unpredictable rates, and blanket limits shared with other routes would
|
||||
cause legitimate deliveries to be dropped.
|
||||
|
||||
The receiver instead has its own dedicated abuse limit, scoped to the
|
||||
`/h/{uuid}` route only and keyed per client IP per request path
|
||||
`/webhook/{uuid}` route only and keyed per client IP per request path
|
||||
(`httprate.KeyByEndpoint`): one misbehaving sender is throttled without
|
||||
affecting other senders of the same entrypoint or the same sender's
|
||||
other entrypoints. Keying on the path rather than on the entrypoint
|
||||
@@ -2253,7 +2198,7 @@ log spends. The access log is bounded by neither limit: every request
|
||||
is recorded once at `INFO`, served or rejected alike.
|
||||
|
||||
What the access log does bound is the _content_ of those lines. A 3xx
|
||||
or 4xx response logs the chi route pattern — `/h/{uuid}`,
|
||||
or 4xx response logs the chi route pattern — `/webhook/{uuid}`,
|
||||
`/user/{username}//`, or the literal `(unmatched)` when the request hit
|
||||
no route at all — in place of the concrete URL. Those are the outcomes
|
||||
an unauthenticated client can drive for free: 404 and 429 on any
|
||||
@@ -2287,12 +2232,12 @@ reduces the headers to a fixed allowlist — `Accept`, `Content-Length`,
|
||||
|
||||
The same hook rewrites the request URL. The SDK builds it as
|
||||
`scheme://host/path` from the concrete path, which on the receiver
|
||||
route is `/h/<uuid>` in full — and that UUID is a write
|
||||
route is `/webhook/<uuid>` in full — and that UUID is a write
|
||||
capability, not an identifier: anyone holding it can post events this
|
||||
service accepts and its targets then deliver. A tracker has its own
|
||||
retention, access control and deletion policy, so the rule the access
|
||||
log follows above does not carry across that boundary. What is sent is
|
||||
the chi route pattern instead: `http://host/h/{uuid}`.
|
||||
the chi route pattern instead: `http://host/webhook/{uuid}`.
|
||||
|
||||
The scheme and the host are kept, and everything else in the URL is
|
||||
discarded rather than edited, so a future SDK version that starts
|
||||
@@ -2336,8 +2281,8 @@ fallback is never the concrete path. The path becomes the literal
|
||||
rewrite cannot parse into a scheme is withheld whole. A transaction
|
||||
event additionally carries the SDK's own `METHOD /path` name, built
|
||||
from the concrete path as well; it is rewritten on the same terms, to
|
||||
`POST /h/{uuid}` where the pattern is known and `POST /(redacted)`
|
||||
where it is not.
|
||||
`POST /webhook/{uuid}` where the pattern is known and `POST
|
||||
/(redacted)` where it is not.
|
||||
|
||||
The headers are an allowlist for the same reason the rules above are
|
||||
unconditional: the SDK's own filter removes four names and passes
|
||||
@@ -2472,7 +2417,7 @@ logger printed the fully interpolated SQL — parameters and all — to
|
||||
standard output on every statement that returned an error, including a
|
||||
plain record-not-found, at a level no operator setting reached. Two of
|
||||
this service's lookups miss by design on unauthenticated routes: the
|
||||
entrypoint lookup behind `/h/{uuid}` and the user lookup behind
|
||||
entrypoint lookup behind `/webhook/{uuid}` and the user lookup behind
|
||||
the login form, whose path segment and submitted username the client
|
||||
picks outright. Every
|
||||
`gorm.Open` in the service now installs the adapter in
|
||||
@@ -2749,48 +2694,44 @@ abuse limit later; they are tracked as future work.
|
||||
|
||||
| Method | Path | Description |
|
||||
| ------ | --------------------------- | ----------- |
|
||||
| `GET` | `/` | Root redirect, 303 (authenticated → `/hooks`, unauthenticated → `/pages/login`) |
|
||||
| `GET` | `/` | Root redirect, 303 (authenticated → `/sources`, unauthenticated → `/pages/login`) |
|
||||
| `GET` | `/.well-known/healthcheck` | Health check (JSON: `status`, `now`, `uptimeSeconds`, `uptimeHuman`, `version`, `appname`, `maintenanceMode`) |
|
||||
| `GET`, `HEAD` | `/s/*` | Static file serving (embedded CSS, JS). `GET` and `HEAD` only — `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, `TRACE` and `CONNECT` are answered `405 Method Not Allowed` with `Allow: GET, HEAD`. Any other method (such as `PROPFIND`) is refused by chi before it reaches this route, and gets `405` without an `Allow` header. Pinned by `TestStaticServesOnlyGetAndHead` |
|
||||
| `POST` | `/h/{uuid}` | Webhook receiver endpoint. `POST` only — every other method is answered `405 Method Not Allowed` with `Allow: POST`. Rate limited (see [Rate Limiting](#rate-limiting)) |
|
||||
| `POST` | `/webhook/{uuid}` | Webhook receiver endpoint. `POST` only — every other method is answered `405 Method Not Allowed` with `Allow: POST`. Rate limited (see [Rate Limiting](#rate-limiting)) |
|
||||
|
||||
#### Authentication Endpoints
|
||||
|
||||
| Method | Path | Description |
|
||||
| ------ | --------------- | ----------- |
|
||||
| `GET` | `/pages/login` | Login page (not rate limited). Its `next` parameter names the page to return to after login; anything but a path on this site is replaced with `/` |
|
||||
| `POST` | `/pages/login` | Login form submission. On success, redirects to the form's `next` when it is a path on this site, otherwise to `/`. Credentials are verified before any limit is consulted, so a correct password is never throttled; 5 FAILED attempts per minute per bucket per submitted username, then `429`. `503` if no verification slot frees up within 5s, or immediately if 16 requests are already queued for one (see [Rate Limiting](#rate-limiting)) |
|
||||
| `GET` | `/pages/login` | Login page (not rate limited) |
|
||||
| `POST` | `/pages/login` | Login form submission. Credentials are verified before any limit is consulted, so a correct password is never throttled; 5 FAILED attempts per minute per bucket per submitted username, then `429`. `503` if no verification slot frees up within 5s, or immediately if 16 requests are already queued for one (see [Rate Limiting](#rate-limiting)) |
|
||||
| `POST` | `/pages/logout` | Logout (destroys session) |
|
||||
|
||||
#### Authenticated Endpoints
|
||||
|
||||
A logged-out `GET` of any of these is redirected to `/pages/login` with
|
||||
its path and query as `next` when they fit in 2048 bytes, so logging in
|
||||
returns to the page that was asked for.
|
||||
|
||||
| Method | Path | Description |
|
||||
| ------ | ------------------------ | ----------- |
|
||||
| `GET` | `/user/{username}` | User profile page |
|
||||
| `POST` | `/user/{username}/password` | Change the user's password (5 per minute per bucket, then `429`; `503` if no verification slot frees up within 5s, or immediately if 16 requests are already queued for one) |
|
||||
| `GET` | `/hooks` | List user's webhooks |
|
||||
| `GET` | `/hooks/new` | Create webhook form |
|
||||
| `POST` | `/hooks/new` | Create webhook submission |
|
||||
| `GET` | `/hook/{id}` | Webhook detail view |
|
||||
| `GET` | `/hook/{id}/edit` | Edit webhook form |
|
||||
| `POST` | `/hook/{id}/edit` | Edit webhook submission |
|
||||
| `POST` | `/hook/{id}/delete` | Delete webhook |
|
||||
| `GET` | `/hook/{id}/events` | Full Event Log |
|
||||
| `GET` | `/hook/{id}/events/{eventID}/body` | Download an event's full stored body. The log page renders each body only up to its cap, so this is the only route that serves a whole one; it is offered wherever a body is shown truncated |
|
||||
| `POST` | `/hook/{id}/deliveries/{deliveryID}/replay` | Replay a finished delivery: creates a new delivery for the same event against the target's current configuration (30 per minute per bucket, then `429`) |
|
||||
| `POST` | `/hook/{id}/events/{eventID}/resubmit` | Resubmit a stored event: creates a new event copying it and fans that out to every currently active target (30 per minute per bucket, then `429`) |
|
||||
| `POST` | `/hook/{id}/entrypoints` | Add entrypoint to webhook |
|
||||
| `POST` | `/hook/{id}/entrypoints/{entrypointID}/delete` | Delete an entrypoint |
|
||||
| `POST` | `/hook/{id}/entrypoints/{entrypointID}/toggle` | Enable or disable an entrypoint |
|
||||
| `POST` | `/hook/{id}/targets` | Add target to webhook |
|
||||
| `GET` | `/hook/{id}/targets/{targetID}/edit` | Edit target form. The one page that renders a target's destination URL and header values in full, rather than masked |
|
||||
| `POST` | `/hook/{id}/targets/{targetID}/edit` | Edit target submission |
|
||||
| `POST` | `/hook/{id}/targets/{targetID}/delete` | Delete a target |
|
||||
| `POST` | `/hook/{id}/targets/{targetID}/toggle` | Enable or disable a target |
|
||||
| `GET` | `/sources` | List user's webhooks |
|
||||
| `GET` | `/sources/new` | Create webhook form |
|
||||
| `POST` | `/sources/new` | Create webhook submission |
|
||||
| `GET` | `/source/{id}` | Webhook detail view |
|
||||
| `GET` | `/source/{id}/edit` | Edit webhook form |
|
||||
| `POST` | `/source/{id}/edit` | Edit webhook submission |
|
||||
| `POST` | `/source/{id}/delete` | Delete webhook |
|
||||
| `GET` | `/source/{id}/logs` | Webhook event logs |
|
||||
| `GET` | `/source/{id}/logs/{eventID}/body` | Download an event's full stored body. The log page renders each body only up to its cap, so this is the only route that serves a whole one; it is offered wherever a body is shown truncated |
|
||||
| `POST` | `/source/{id}/deliveries/{deliveryID}/replay` | Replay a finished delivery: creates a new delivery for the same event against the target's current configuration (30 per minute per bucket, then `429`) |
|
||||
| `POST` | `/source/{id}/events/{eventID}/resubmit` | Resubmit a stored event: creates a new event copying it and fans that out to every currently active target (30 per minute per bucket, then `429`) |
|
||||
| `POST` | `/source/{id}/entrypoints` | Add entrypoint to webhook |
|
||||
| `POST` | `/source/{id}/entrypoints/{entrypointID}/delete` | Delete an entrypoint |
|
||||
| `POST` | `/source/{id}/entrypoints/{entrypointID}/toggle` | Enable or disable an entrypoint |
|
||||
| `POST` | `/source/{id}/targets` | Add target to webhook |
|
||||
| `GET` | `/source/{id}/targets/{targetID}/edit` | Edit target form. The one page that renders a target's destination URL and header values in full, rather than masked |
|
||||
| `POST` | `/source/{id}/targets/{targetID}/edit` | Edit target submission |
|
||||
| `POST` | `/source/{id}/targets/{targetID}/delete` | Delete a target |
|
||||
| `POST` | `/source/{id}/targets/{targetID}/toggle` | Enable or disable a target |
|
||||
|
||||
#### Infrastructure Endpoints
|
||||
|
||||
@@ -2847,7 +2788,6 @@ webhooker/
|
||||
│ │ ├── model_event.go # Event entity (per-webhook DB)
|
||||
│ │ ├── model_delivery.go # Delivery entity (per-webhook DB)
|
||||
│ │ ├── model_delivery_result.go # DeliveryResult entity (per-webhook DB)
|
||||
│ │ ├── model_totals.go # EventTotals and TargetTotals (per-webhook DB)
|
||||
│ │ ├── model_apikey.go # APIKey entity
|
||||
│ │ ├── password.go # Argon2id hashing and verification
|
||||
│ │ ├── retention.go # Retention reaper (per-webhook event expiry)
|
||||
@@ -2943,13 +2883,15 @@ Components are wired via Uber fx in this order:
|
||||
7. `healthcheck.New` — Health check service
|
||||
8. `session.New` — Cookie-based session manager (key from database)
|
||||
9. `handlers.New` — HTTP handlers
|
||||
10. `middleware.New` — HTTP middleware
|
||||
11. `delivery.New` — Event-driven delivery engine
|
||||
12. `delivery.NewArchiveSweeper` — Periodic pruning of idle archives
|
||||
13. `delivery.Engine` → `delivery.Notifier` — interface bridge
|
||||
14. `delivery.Engine` → `delivery.WebhookEvictor` — interface bridge so
|
||||
10. `metrics.NewRegistry` — The registry `/metrics` serves
|
||||
11. `metrics.New` — The delivery collectors, registered on that registry
|
||||
12. `middleware.New` — HTTP middleware
|
||||
13. `delivery.New` — Event-driven delivery engine
|
||||
14. `delivery.NewArchiveSweeper` — Periodic pruning of idle archives
|
||||
15. `delivery.Engine` → `delivery.Notifier` — interface bridge
|
||||
16. `delivery.Engine` → `delivery.WebhookEvictor` — interface bridge so
|
||||
deleting a webhook releases its archive writer
|
||||
15. `server.New` — HTTP server and router
|
||||
17. `server.New` — HTTP server and router
|
||||
|
||||
The server starts via `fx.Invoke(func(*server.Server, *delivery.Engine,
|
||||
*database.RetentionReaper, *delivery.ArchiveSweeper) {})`, which
|
||||
@@ -2992,8 +2934,8 @@ local record instead of nothing. What that placement gives up is
|
||||
recovery of a panic in the six entries above it, none of which does
|
||||
more than set a header or start a timer.
|
||||
|
||||
Additionally, form endpoints (`/pages`, `/user/*`, `/hooks`,
|
||||
`/hook/*`) apply a **MaxBodySize** middleware that limits
|
||||
Additionally, form endpoints (`/pages`, `/user/*`, `/sources`,
|
||||
`/source/*`) apply a **MaxBodySize** middleware that limits
|
||||
POST/PUT/PATCH request bodies to 1 MB. It is registered ahead of the
|
||||
CSRF middleware in every one of those route groups, because
|
||||
gorilla/csrf parses the form; if the cap were installed after it, form
|
||||
@@ -3017,7 +2959,7 @@ Those same four route groups then apply **CSRF** and **NoCache**
|
||||
`/pages` applies **RequireAuth**. The rate limiters are per-route
|
||||
rather than global: **PasswordChangeRateLimit** on
|
||||
`/user/{username}/password` and **ReceiverRateLimit** on
|
||||
`/h/{uuid}`. There is deliberately none on `/pages/login` — that
|
||||
`/webhook/{uuid}`. There is deliberately none on `/pages/login` — that
|
||||
endpoint counts failures inside the handler, after the credential
|
||||
check, see [The login endpoint](#the-login-endpoint).
|
||||
|
||||
@@ -3058,8 +3000,8 @@ check, see [The login endpoint](#the-login-endpoint).
|
||||
by middleware that runs before CSRF parses the form
|
||||
- **CSRF protection** via [gorilla/csrf](https://github.com/gorilla/csrf)
|
||||
on all state-changing forms (cookie-based double-submit tokens with
|
||||
HMAC authentication). Applied to `/pages`, `/hooks`, `/hook`, and
|
||||
`/user` routes. Excluded from `/h` (inbound webhook POSTs) and
|
||||
HMAC authentication). Applied to `/pages`, `/sources`, `/source`, and
|
||||
`/user` routes. Excluded from `/webhook` (inbound webhook POSTs) and
|
||||
`/api` (stateless API). The middleware detects TLS per-request through
|
||||
`internal/reqtls.IsTLS` — the same predicate the session cookie uses —
|
||||
to set appropriate cookie security flags and Origin/Referer validation
|
||||
@@ -3112,8 +3054,7 @@ check, see [The login endpoint](#the-login-endpoint).
|
||||
before the app starts; the image's health check; and `docker exec`,
|
||||
unless given `--user`
|
||||
- GORM soft deletes on every entity that carries `BaseModel`, which is
|
||||
all of them but `Setting`, `EventTotals` and `TargetTotals` (data
|
||||
preserved for audit)
|
||||
all of them but `Setting` (data preserved for audit)
|
||||
|
||||
### Shutdown
|
||||
|
||||
|
||||
@@ -387,7 +387,7 @@ point of the branch.
|
||||
- 2026-03-05 security headers middleware, session regeneration on
|
||||
login, request body size limits (#41)
|
||||
- 2026-03-04 tests for delivery, middleware, and session packages
|
||||
(#32); removed the build-architecture global (#31)
|
||||
(#32); removed globals.Buildarch (#31)
|
||||
- 2026-03-04 1.0 MVP merge: Webhook/Entrypoint/Target rename, core
|
||||
delivery engine with bounded worker pool and circuit breaker,
|
||||
parallel fan-out, per-webhook event databases, management UI (#16)
|
||||
|
||||
@@ -16,6 +16,7 @@ import (
|
||||
"sneak.berlin/go/webhooker/internal/handlers"
|
||||
"sneak.berlin/go/webhooker/internal/healthcheck"
|
||||
"sneak.berlin/go/webhooker/internal/logger"
|
||||
"sneak.berlin/go/webhooker/internal/metrics"
|
||||
"sneak.berlin/go/webhooker/internal/middleware"
|
||||
"sneak.berlin/go/webhooker/internal/resetpw"
|
||||
"sneak.berlin/go/webhooker/internal/server"
|
||||
@@ -177,6 +178,10 @@ func newApp() *fx.App {
|
||||
healthcheck.New,
|
||||
session.New,
|
||||
handlers.New,
|
||||
// The registry /metrics serves, and the delivery
|
||||
// collectors registered on it.
|
||||
metrics.NewRegistry,
|
||||
metrics.New,
|
||||
middleware.New,
|
||||
// The one SSRF guard both target-creation validation
|
||||
// and the delivery dialer consult, so they cannot
|
||||
|
||||
@@ -93,11 +93,11 @@ func TestEventTierQueriesUseTheirIndexes(t *testing.T) {
|
||||
deliveries []database.Delivery
|
||||
results []database.DeliveryResult
|
||||
depths []struct{ Depth int }
|
||||
removed []database.TargetTotals
|
||||
)
|
||||
|
||||
byStatus := "idx_deliveries_status (status=? AND deleted_at=?)"
|
||||
byEvent := "idx_deliveries_event_id (event_id=? AND deleted_at=?)"
|
||||
byAge := "idx_events_deleted_at_created_at (deleted_at=? AND created_at<?)"
|
||||
|
||||
// The delivery engine: recovery and the retry sweep, the sweep for
|
||||
// stranded pending deliveries, and the queue depth count.
|
||||
@@ -123,80 +123,25 @@ func TestEventTierQueriesUseTheirIndexes(t *testing.T) {
|
||||
Order("attempt_num ASC").Find(&results),
|
||||
"idx_delivery_results_delivery_id (delivery_id=? AND deleted_at=?)")
|
||||
|
||||
// Retention (reapExpired, deleteEvents): one batch of expired
|
||||
// events, then their attempts, deliveries and the events.
|
||||
var expired []string
|
||||
// Retention's three deletes (reapExpired), whose subqueries are built
|
||||
// afresh for each statement as it builds them.
|
||||
expiredEventIDs := func() *gorm.DB {
|
||||
return dry.Model(&database.Event{}).Select("id").
|
||||
Where("created_at < ?", cutoff)
|
||||
}
|
||||
|
||||
assertPlanUses(t, db, dry.Unscoped().Model(&database.Event{}).
|
||||
Where("created_at < ?", cutoff).
|
||||
Limit(database.ExportReapBatchSize).Pluck("id", &expired),
|
||||
"idx_events_created_at (created_at<?)")
|
||||
assertPlanUses(t, db, dry.Unscoped().Where(
|
||||
"delivery_id IN (?)", dry.Unscoped().Model(&database.Delivery{}).
|
||||
Select("id").Where("event_id IN ?", ids),
|
||||
"delivery_id IN (?)", dry.Model(&database.Delivery{}).
|
||||
Select("id").Where("event_id IN (?)", expiredEventIDs()),
|
||||
).Delete(&database.DeliveryResult{}),
|
||||
"idx_delivery_results_delivery_id (delivery_id=?)",
|
||||
"idx_deliveries_event_id (event_id=?)")
|
||||
assertPlanUses(t, db, dry.Unscoped().Model(&database.Delivery{}).
|
||||
Select("target_id, count(*) AS deliveries_removed, "+
|
||||
"count(CASE WHEN status = ? THEN 1 END) AS failed_removed",
|
||||
database.DeliveryStatusFailed).
|
||||
Where("event_id IN ?", ids).Group("target_id").Find(&removed),
|
||||
"idx_deliveries_event_id (event_id=?)")
|
||||
assertPlanUses(t, db, dry.Unscoped().Where("event_id IN ?", ids).
|
||||
Delete(&database.Delivery{}), "idx_deliveries_event_id (event_id=?)")
|
||||
assertPlanUses(t, db, dry.Unscoped().Where("id IN ?", ids).
|
||||
Delete(&database.Event{}), "sqlite_autoindex_events_1 (id=?)")
|
||||
}
|
||||
|
||||
// TestStatisticsQueriesUseTheirIndexes does the same for the webhook
|
||||
// page's statistics (readEventStats in the handlers): deliveries in
|
||||
// progress, each target's deliveries finished since a time, which must
|
||||
// come from the index alone, and events received since a time.
|
||||
func TestStatisticsQueriesUseTheirIndexes(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
mgr, lc := setupTestWebhookDBManager(t)
|
||||
ctx := context.Background()
|
||||
require.NoError(t, lc.Start(ctx))
|
||||
|
||||
defer func() { require.NoError(t, lc.Stop(ctx)) }()
|
||||
|
||||
db, err := mgr.GetDB(uuid.New().String())
|
||||
require.NoError(t, err)
|
||||
|
||||
dry := db.Session(&gorm.Session{DryRun: true})
|
||||
since := time.Now()
|
||||
|
||||
var (
|
||||
count int64
|
||||
byTarget []struct{ TargetID string }
|
||||
)
|
||||
|
||||
assertPlanUses(t, db, dry.Model(&database.Delivery{}).
|
||||
Where("status IN ?", []database.DeliveryStatus{
|
||||
database.DeliveryStatusPending,
|
||||
database.DeliveryStatusRetrying,
|
||||
}).Count(&count),
|
||||
"idx_deliveries_status (status=? AND deleted_at=?)")
|
||||
assertPlanUses(t, db, dry.Model(&database.Delivery{}).
|
||||
Select("target_id, "+
|
||||
"count(CASE WHEN status = ? THEN 1 END) AS delivered, "+
|
||||
"count(CASE WHEN status = ? THEN 1 END) AS failed",
|
||||
database.DeliveryStatusDelivered,
|
||||
database.DeliveryStatusFailed).
|
||||
Where("status IN ? AND finished_at >= ?",
|
||||
[]database.DeliveryStatus{
|
||||
database.DeliveryStatusDelivered,
|
||||
database.DeliveryStatusFailed,
|
||||
}, since).
|
||||
Group("target_id").Find(&byTarget),
|
||||
"COVERING INDEX idx_deliveries_status "+
|
||||
"(status=? AND deleted_at=? AND finished_at>?)")
|
||||
assertPlanUses(t, db, dry.Model(&database.Event{}).
|
||||
Where("created_at >= ?", since).Count(&count),
|
||||
"idx_events_deleted_at_created_at "+
|
||||
"(deleted_at=? AND created_at>?)")
|
||||
"idx_delivery_results_delivery_id (delivery_id=?)", byEvent, byAge)
|
||||
assertPlanUses(t, db, dry.Unscoped().Where(
|
||||
"event_id IN (?)", expiredEventIDs(),
|
||||
).Delete(&database.Delivery{}),
|
||||
"idx_deliveries_event_id (event_id=?)", byAge)
|
||||
assertPlanUses(t, db, dry.Unscoped().Where(
|
||||
"created_at < ?", cutoff,
|
||||
).Delete(&database.Event{}), "idx_events_created_at (created_at<?)")
|
||||
}
|
||||
|
||||
// assertPlanUses asserts that SQLite's plan for a statement GORM built
|
||||
|
||||
@@ -5,7 +5,6 @@ import (
|
||||
"io"
|
||||
"log/slog"
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"go.uber.org/fx"
|
||||
@@ -29,10 +28,6 @@ func NewTestRetentionReaper(
|
||||
}
|
||||
}
|
||||
|
||||
// ExportReapBatchSize exposes how many expired events one retention
|
||||
// transaction deletes.
|
||||
const ExportReapBatchSize = reapBatchSize
|
||||
|
||||
// ExportSweep runs a single retention sweep synchronously for tests.
|
||||
func (r *RetentionReaper) ExportSweep(ctx context.Context) {
|
||||
r.sweep(ctx)
|
||||
@@ -84,14 +79,3 @@ func (d *Database) ExportSetBannerOut(w io.Writer) {
|
||||
func DummyPasswordHashForTest() string {
|
||||
return dummyPasswordHash()
|
||||
}
|
||||
|
||||
// HashAtShippedCostForTest makes HashPassword hash at the shipped
|
||||
// memory cost until t ends. t must not run in parallel with other
|
||||
// tests, which would hash at that cost alongside it.
|
||||
func HashAtShippedCostForTest(t *testing.T) {
|
||||
t.Helper()
|
||||
|
||||
hashAtShippedCostInTest = true
|
||||
|
||||
t.Cleanup(func() { hashAtShippedCostInTest = false })
|
||||
}
|
||||
|
||||
@@ -1,10 +1,6 @@
|
||||
package database
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
import "gorm.io/gorm"
|
||||
|
||||
// DeliveryStatus represents the status of a delivery
|
||||
type DeliveryStatus string
|
||||
@@ -41,7 +37,7 @@ type Delivery struct {
|
||||
BaseModel
|
||||
|
||||
EventID string `gorm:"type:uuid;not null;index:idx_deliveries_event_id,priority:1" json:"eventId"`
|
||||
TargetID string `gorm:"type:uuid;not null;index:idx_deliveries_status,priority:4" json:"targetId"`
|
||||
TargetID string `gorm:"type:uuid;not null" json:"targetId"`
|
||||
Status DeliveryStatus `gorm:"not null;default:'pending';index:idx_deliveries_status,priority:1" json:"status"`
|
||||
|
||||
// DeletedAt repeats the BaseModel field only to be the second column
|
||||
@@ -49,13 +45,6 @@ type Delivery struct {
|
||||
// gives.
|
||||
DeletedAt gorm.DeletedAt `gorm:"index:idx_deliveries_event_id,priority:2;index:idx_deliveries_status,priority:2" json:"deletedAt,omitzero"`
|
||||
|
||||
// FinishedAt is when the delivery became delivered or failed, and
|
||||
// nil while it is pending or retrying. It and then TargetID end the
|
||||
// status index, so the webhook page counts each target's deliveries
|
||||
// that finished in a recent window by reading just that window from
|
||||
// the index.
|
||||
FinishedAt *time.Time `gorm:"index:idx_deliveries_status,priority:3" json:"finishedAt,omitempty"`
|
||||
|
||||
// Relations
|
||||
Event Event `json:"event,omitzero"`
|
||||
Target Target `json:"target,omitzero"`
|
||||
|
||||
@@ -1,99 +0,0 @@
|
||||
package database
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
// The running totals in a webhook's event database keep the webhook
|
||||
// page's lifetime figures right after retention has removed the rows
|
||||
// they count, and let the page show them without counting every row.
|
||||
// Each total changes in the transaction that writes or deletes the
|
||||
// rows it counts.
|
||||
|
||||
// EventTotals is the single row counting a webhook's events: every
|
||||
// event ever stored, how many of them retention has deleted, and when
|
||||
// the newest arrived, which retention leaves as it is.
|
||||
type EventTotals struct {
|
||||
ID int64 `gorm:"primaryKey"`
|
||||
|
||||
Events int64 `gorm:"not null"`
|
||||
EventsRemoved int64 `gorm:"not null"`
|
||||
|
||||
// LastEventAt is when the newest event arrived, or nil before the
|
||||
// first.
|
||||
LastEventAt *time.Time
|
||||
}
|
||||
|
||||
// TableName names the table AddEventTotals updates.
|
||||
func (EventTotals) TableName() string {
|
||||
return "event_totals"
|
||||
}
|
||||
|
||||
// TargetTotals is one row per target counting its deliveries: every
|
||||
// delivery ever created, how many became delivered and how many
|
||||
// failed, and how many deliveries and failed deliveries retention has
|
||||
// deleted. The webhook's delivery figures are these rows summed.
|
||||
type TargetTotals struct {
|
||||
TargetID string `gorm:"type:uuid;primaryKey"`
|
||||
|
||||
Deliveries int64 `gorm:"not null"`
|
||||
Delivered int64 `gorm:"not null"`
|
||||
Failed int64 `gorm:"not null"`
|
||||
|
||||
DeliveriesRemoved int64 `gorm:"not null"`
|
||||
FailedRemoved int64 `gorm:"not null"`
|
||||
}
|
||||
|
||||
// TableName names the table AddTargetTotals updates.
|
||||
func (TargetTotals) TableName() string {
|
||||
return "target_totals"
|
||||
}
|
||||
|
||||
// AddEventTotals adds each count in add to the webhook's event totals,
|
||||
// and records add.LastEventAt as when the newest event arrived if it is
|
||||
// set. Call it on the transaction that writes or deletes the events it
|
||||
// counts.
|
||||
func AddEventTotals(tx *gorm.DB, add EventTotals) error {
|
||||
err := tx.Exec(
|
||||
`UPDATE event_totals SET
|
||||
events = events + ?,
|
||||
events_removed = events_removed + ?,
|
||||
last_event_at = coalesce(?, last_event_at)`,
|
||||
add.Events, add.EventsRemoved, add.LastEventAt,
|
||||
).Error
|
||||
if err != nil {
|
||||
return fmt.Errorf("adding to event totals: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// AddTargetTotals adds each count in add to the totals of the target
|
||||
// add.TargetID names, creating its row the first time. Call it on the
|
||||
// transaction that writes or deletes the deliveries it counts.
|
||||
func AddTargetTotals(tx *gorm.DB, add TargetTotals) error {
|
||||
err := tx.Exec(
|
||||
`INSERT INTO target_totals (target_id, deliveries, delivered,
|
||||
failed, deliveries_removed, failed_removed)
|
||||
VALUES (?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT (target_id) DO UPDATE SET
|
||||
deliveries = deliveries + excluded.deliveries,
|
||||
delivered = delivered + excluded.delivered,
|
||||
failed = failed + excluded.failed,
|
||||
deliveries_removed =
|
||||
deliveries_removed + excluded.deliveries_removed,
|
||||
failed_removed = failed_removed + excluded.failed_removed`,
|
||||
add.TargetID, add.Deliveries, add.Delivered,
|
||||
add.Failed, add.DeliveriesRemoved, add.FailedRemoved,
|
||||
).Error
|
||||
if err != nil {
|
||||
return fmt.Errorf(
|
||||
"adding to totals of target %s: %w", add.TargetID, err,
|
||||
)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -2,8 +2,7 @@ package database
|
||||
|
||||
// Migrate runs database migrations for the main application database.
|
||||
// Only configuration-tier models are stored in the main database.
|
||||
// Event-tier models (Event, Delivery, DeliveryResult, EventTotals,
|
||||
// TargetTotals) live in
|
||||
// Event-tier models (Event, Delivery, DeliveryResult) live in
|
||||
// per-webhook dedicated databases managed by WebhookDBManager.
|
||||
func (d *Database) Migrate() error {
|
||||
return d.db.AutoMigrate(
|
||||
|
||||
@@ -9,7 +9,6 @@ import (
|
||||
"math/big"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"golang.org/x/crypto/argon2"
|
||||
)
|
||||
@@ -64,30 +63,10 @@ func DefaultPasswordConfig() *PasswordConfig {
|
||||
}
|
||||
}
|
||||
|
||||
// testArgon2Memory is the Argon2id memory cost, in KiB, that a test
|
||||
// binary hashes with: 1 MB instead of the shipped 64 MB. Every test
|
||||
// that starts a database hashes the bootstrap admin password, dozens
|
||||
// of them run in parallel, and under the race detector each 64 MB hash
|
||||
// holds about 150 MB. VerifyPassword reads the cost from the hash it
|
||||
// checks, so verification follows.
|
||||
const testArgon2Memory = 1024
|
||||
|
||||
// hashAtShippedCostInTest makes a test binary hash at the shipped
|
||||
// memory cost. Only TestHashPassword_ShippedParameters sets it.
|
||||
//
|
||||
//nolint:gochecknoglobals // set by one test, see above
|
||||
var hashAtShippedCostInTest bool
|
||||
|
||||
// HashPassword generates an Argon2id hash of the password. A binary
|
||||
// built by go test hashes at testArgon2Memory; one built by go build
|
||||
// always hashes at the defaults.
|
||||
// HashPassword generates an Argon2id hash of the password
|
||||
func HashPassword(password string) (string, error) {
|
||||
config := DefaultPasswordConfig()
|
||||
|
||||
if testing.Testing() && !hashAtShippedCostInTest {
|
||||
config.Memory = testArgon2Memory
|
||||
}
|
||||
|
||||
// Generate a salt
|
||||
salt := make([]byte, config.SaltLen)
|
||||
|
||||
|
||||
@@ -192,39 +192,6 @@ func TestHashPasswordUniqueness(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestHashPassword_ShippedParameters hashes and verifies through
|
||||
// HashPassword at the shipped Argon2id parameters. Every other test
|
||||
// hashes at the lower memory cost a test binary uses, so this is the
|
||||
// one that keeps production hashing covered. One hash and one
|
||||
// verification: each costs 64 MB.
|
||||
//
|
||||
//nolint:paralleltest // changes the hashing cost for the whole binary
|
||||
func TestHashPassword_ShippedParameters(t *testing.T) {
|
||||
database.HashAtShippedCostForTest(t)
|
||||
|
||||
password := "correct horse battery staple"
|
||||
|
||||
hash, err := database.HashPassword(password)
|
||||
if err != nil {
|
||||
t.Fatalf("hashing with the shipped parameters: %v", err)
|
||||
}
|
||||
|
||||
const shipped = "$argon2id$v=19$m=65536,t=1,p=4$"
|
||||
|
||||
if !strings.HasPrefix(hash, shipped) {
|
||||
t.Errorf("hash = %q, want prefix %q", hash, shipped)
|
||||
}
|
||||
|
||||
valid, err := database.VerifyPassword(password, hash)
|
||||
if err != nil {
|
||||
t.Fatalf("VerifyPassword() error = %v", err)
|
||||
}
|
||||
|
||||
if !valid {
|
||||
t.Error("VerifyPassword() returned false for correct password")
|
||||
}
|
||||
}
|
||||
|
||||
// TestVerifyDummyPassword_DoesRealWork covers the anti-enumeration
|
||||
// path. Login charges an unknown username a verification against a
|
||||
// dummy hash so that a nonexistent account is not answered in
|
||||
|
||||
+46
-111
@@ -18,19 +18,6 @@ import (
|
||||
// computation.
|
||||
const hoursPerDay = 24
|
||||
|
||||
// reapBatchSize is how many expired events one retention transaction
|
||||
// deletes. A transaction holds the event database's write lock, which
|
||||
// the receiver and the delivery workers wait for, so a large prune is
|
||||
// split into transactions each short enough to finish well inside the
|
||||
// busy timeout.
|
||||
const reapBatchSize = 1000
|
||||
|
||||
// reapBatchPause is how long retention waits after one batch before
|
||||
// starting the next. A writer waiting for the write lock checks for it
|
||||
// again after at most 100 ms, so a longer pause lets it in between two
|
||||
// batches instead of only after the whole prune.
|
||||
const reapBatchPause = 200 * time.Millisecond
|
||||
|
||||
// RetentionReaperParams holds the fx dependencies for the
|
||||
// RetentionReaper.
|
||||
type RetentionReaperParams struct {
|
||||
@@ -200,15 +187,13 @@ func (r *RetentionReaper) sweep(ctx context.Context) {
|
||||
continue
|
||||
}
|
||||
|
||||
r.reapWebhook(ctx, wh.ID, wh.RetentionDays)
|
||||
r.reapWebhook(wh.ID, wh.RetentionDays)
|
||||
}
|
||||
}
|
||||
|
||||
// reapWebhook removes every expired event (and its dependents) from a
|
||||
// single webhook's database, or as many as it reaches before ctx is
|
||||
// cancelled.
|
||||
// single webhook's database.
|
||||
func (r *RetentionReaper) reapWebhook(
|
||||
ctx context.Context,
|
||||
webhookID string,
|
||||
retentionDays int,
|
||||
) {
|
||||
@@ -228,7 +213,7 @@ func (r *RetentionReaper) reapWebhook(
|
||||
return
|
||||
}
|
||||
|
||||
deleted, err := reapExpired(ctx, db, cutoff)
|
||||
deleted, err := reapExpired(db, cutoff)
|
||||
if err != nil {
|
||||
r.log.Error(
|
||||
"retention sweep: failed to reap expired events",
|
||||
@@ -280,107 +265,57 @@ func retentionCutoff(
|
||||
), true
|
||||
}
|
||||
|
||||
// reapExpired hard-deletes the events older than cutoff, with their
|
||||
// deliveries and delivery results, reapBatchSize events per
|
||||
// transaction with reapBatchPause between transactions, until none is
|
||||
// left. Once ctx is cancelled it returns after the batch in hand,
|
||||
// leaving the rest to the next sweep, so stopping the app does not
|
||||
// wait for a long prune. It returns the number of events deleted.
|
||||
func reapExpired(
|
||||
ctx context.Context, db *gorm.DB, cutoff time.Time,
|
||||
) (int64, error) {
|
||||
var total int64
|
||||
|
||||
for {
|
||||
var eventIDs []string
|
||||
|
||||
err := db.Transaction(func(tx *gorm.DB) error {
|
||||
err := tx.Unscoped().Model(&Event{}).
|
||||
Where("created_at < ?", cutoff).
|
||||
Limit(reapBatchSize).
|
||||
Pluck("id", &eventIDs).Error
|
||||
if err != nil {
|
||||
return fmt.Errorf("selecting expired events: %w", err)
|
||||
}
|
||||
|
||||
if len(eventIDs) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
return deleteEvents(tx, eventIDs)
|
||||
})
|
||||
if err != nil {
|
||||
return total, err
|
||||
}
|
||||
|
||||
total += int64(len(eventIDs))
|
||||
|
||||
if len(eventIDs) < reapBatchSize {
|
||||
return total, nil
|
||||
}
|
||||
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return total, nil
|
||||
case <-time.After(reapBatchPause):
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// deleteEvents hard-deletes the given events and, in foreign-key-safe
|
||||
// order before them, their delivery results and deliveries, then adds
|
||||
// what it deleted to the running totals. It runs on reapExpired's
|
||||
// transaction, so the totals change exactly when the rows do. Deletes
|
||||
// are unscoped so rows are physically removed rather than
|
||||
// soft-deleted, reclaiming disk.
|
||||
func deleteEvents(tx *gorm.DB, eventIDs []string) error {
|
||||
// 1. The delivery results of the events' deliveries.
|
||||
err := tx.Unscoped().
|
||||
Where("delivery_id IN (?)", tx.Unscoped().Model(&Delivery{}).
|
||||
// reapExpired hard-deletes, in foreign-key-safe order, the delivery
|
||||
// results, deliveries, and events associated with events older than
|
||||
// cutoff. Deletes are unscoped so rows are physically removed rather
|
||||
// than soft-deleted, reclaiming disk. It returns the number of events
|
||||
// deleted.
|
||||
func reapExpired(db *gorm.DB, cutoff time.Time) (int64, error) {
|
||||
// Fresh subqueries are built per statement to avoid reusing a
|
||||
// mutated builder across executions.
|
||||
expiredEventIDs := func() *gorm.DB {
|
||||
return db.Model(&Event{}).
|
||||
Select("id").
|
||||
Where("event_id IN ?", eventIDs)).
|
||||
Delete(&DeliveryResult{}).Error
|
||||
if err != nil {
|
||||
return fmt.Errorf("deleting expired delivery results: %w", err)
|
||||
Where("created_at < ?", cutoff)
|
||||
}
|
||||
expiredDeliveryIDs := func() *gorm.DB {
|
||||
return db.Model(&Delivery{}).
|
||||
Select("id").
|
||||
Where("event_id IN (?)", expiredEventIDs())
|
||||
}
|
||||
|
||||
// 2. The events' deliveries, after counting them, and the failed
|
||||
// ones among them, per target. The status is tested in the select
|
||||
// list rather than the WHERE clause: there, SQLite would read every
|
||||
// failed delivery the webhook has through the status index,
|
||||
// instead of only these through the event_id index.
|
||||
var removed []TargetTotals
|
||||
|
||||
err = tx.Unscoped().Model(&Delivery{}).
|
||||
Select("target_id, count(*) AS deliveries_removed, "+
|
||||
"count(CASE WHEN status = ? THEN 1 END) AS failed_removed",
|
||||
DeliveryStatusFailed).
|
||||
Where("event_id IN ?", eventIDs).
|
||||
Group("target_id").
|
||||
Find(&removed).Error
|
||||
if err != nil {
|
||||
return fmt.Errorf("counting expired deliveries: %w", err)
|
||||
// 1. Delivery results whose delivery belongs to an expired event.
|
||||
res := db.Unscoped().
|
||||
Where("delivery_id IN (?)", expiredDeliveryIDs()).
|
||||
Delete(&DeliveryResult{})
|
||||
if res.Error != nil {
|
||||
return 0, fmt.Errorf(
|
||||
"deleting expired delivery results: %w",
|
||||
res.Error,
|
||||
)
|
||||
}
|
||||
|
||||
err = tx.Unscoped().
|
||||
Where("event_id IN ?", eventIDs).
|
||||
Delete(&Delivery{}).Error
|
||||
if err != nil {
|
||||
return fmt.Errorf("deleting expired deliveries: %w", err)
|
||||
// 2. Deliveries belonging to an expired event.
|
||||
del := db.Unscoped().
|
||||
Where("event_id IN (?)", expiredEventIDs()).
|
||||
Delete(&Delivery{})
|
||||
if del.Error != nil {
|
||||
return 0, fmt.Errorf(
|
||||
"deleting expired deliveries: %w",
|
||||
del.Error,
|
||||
)
|
||||
}
|
||||
|
||||
// 3. The events themselves.
|
||||
ev := tx.Unscoped().Where("id IN ?", eventIDs).Delete(&Event{})
|
||||
// 3. The expired events themselves.
|
||||
ev := db.Unscoped().
|
||||
Where("created_at < ?", cutoff).
|
||||
Delete(&Event{})
|
||||
if ev.Error != nil {
|
||||
return fmt.Errorf("deleting expired events: %w", ev.Error)
|
||||
return 0, fmt.Errorf(
|
||||
"deleting expired events: %w",
|
||||
ev.Error,
|
||||
)
|
||||
}
|
||||
|
||||
for i := range removed {
|
||||
err = AddTargetTotals(tx, removed[i])
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return AddEventTotals(tx, EventTotals{EventsRemoved: ev.RowsAffected})
|
||||
return ev.RowsAffected, nil
|
||||
}
|
||||
|
||||
@@ -1,410 +0,0 @@
|
||||
package database_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/gorm"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
)
|
||||
|
||||
// readEventTotals reads a webhook database's row of event totals,
|
||||
// asserting that it has exactly one.
|
||||
func readEventTotals(t *testing.T, db *gorm.DB) database.EventTotals {
|
||||
t.Helper()
|
||||
|
||||
var rows []database.EventTotals
|
||||
|
||||
require.NoError(t, db.Find(&rows).Error)
|
||||
require.Len(t, rows, 1)
|
||||
|
||||
return rows[0]
|
||||
}
|
||||
|
||||
// readTargetTotals reads a webhook database's target totals, keyed by
|
||||
// target.
|
||||
func readTargetTotals(
|
||||
t *testing.T, db *gorm.DB,
|
||||
) map[string]database.TargetTotals {
|
||||
t.Helper()
|
||||
|
||||
var rows []database.TargetTotals
|
||||
|
||||
require.NoError(t, db.Find(&rows).Error)
|
||||
|
||||
byTarget := make(map[string]database.TargetTotals, len(rows))
|
||||
for _, row := range rows {
|
||||
byTarget[row.TargetID] = row
|
||||
}
|
||||
|
||||
return byTarget
|
||||
}
|
||||
|
||||
// TestWebhookDBManager_TotalsSurviveReopen verifies that a new event
|
||||
// database starts with one row of zero event totals and no target
|
||||
// totals, that adding to a target twice adds to the one row, and that
|
||||
// opening the database again keeps everything added.
|
||||
func TestWebhookDBManager_TotalsSurviveReopen(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
mgr, lc := setupTestWebhookDBManager(t)
|
||||
ctx := context.Background()
|
||||
require.NoError(t, lc.Start(ctx))
|
||||
|
||||
defer func() { require.NoError(t, lc.Stop(ctx)) }()
|
||||
|
||||
webhookID := uuid.New().String()
|
||||
|
||||
db, err := mgr.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
|
||||
fresh := readEventTotals(t, db)
|
||||
assert.Equal(t, database.EventTotals{ID: fresh.ID}, fresh)
|
||||
assert.Empty(t, readTargetTotals(t, db))
|
||||
|
||||
first, second := uuid.New().String(), uuid.New().String()
|
||||
|
||||
require.NoError(t, database.AddEventTotals(db, database.EventTotals{
|
||||
Events: 2,
|
||||
}))
|
||||
require.NoError(t, database.AddTargetTotals(db, database.TargetTotals{
|
||||
TargetID: first, Deliveries: 2, Delivered: 1,
|
||||
}))
|
||||
require.NoError(t, database.AddTargetTotals(db, database.TargetTotals{
|
||||
TargetID: first, Failed: 1,
|
||||
}))
|
||||
require.NoError(t, database.AddTargetTotals(db, database.TargetTotals{
|
||||
TargetID: second, Deliveries: 1,
|
||||
}))
|
||||
|
||||
// Drop the cached connection so the next open reopens the file,
|
||||
// as a restart would.
|
||||
require.NoError(t, mgr.CloseAll())
|
||||
|
||||
db, err = mgr.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Equal(t, database.EventTotals{ID: fresh.ID, Events: 2},
|
||||
readEventTotals(t, db))
|
||||
assert.Equal(t, map[string]database.TargetTotals{
|
||||
first: {
|
||||
TargetID: first, Deliveries: 2, Delivered: 1, Failed: 1,
|
||||
},
|
||||
second: {TargetID: second, Deliveries: 1},
|
||||
}, readTargetTotals(t, db))
|
||||
}
|
||||
|
||||
// seedExpiredEvents stores count events created at the given time,
|
||||
// each with a delivered delivery to one target and a failed delivery
|
||||
// to the other, and one attempt for each delivery.
|
||||
func seedExpiredEvents(
|
||||
t *testing.T,
|
||||
db *gorm.DB,
|
||||
webhookID string,
|
||||
count int,
|
||||
createdAt time.Time,
|
||||
delivered, failed string,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
events := make([]database.Event, count)
|
||||
deliveries := make([]database.Delivery, 0, 2*count)
|
||||
|
||||
for i := range events {
|
||||
events[i] = database.Event{
|
||||
WebhookID: webhookID,
|
||||
EntrypointID: uuid.New().String(),
|
||||
Method: http.MethodPost,
|
||||
}
|
||||
events[i].ID = uuid.New().String()
|
||||
events[i].CreatedAt = createdAt
|
||||
|
||||
deliveries = append(deliveries,
|
||||
database.Delivery{
|
||||
EventID: events[i].ID,
|
||||
TargetID: delivered,
|
||||
Status: database.DeliveryStatusDelivered,
|
||||
},
|
||||
database.Delivery{
|
||||
EventID: events[i].ID,
|
||||
TargetID: failed,
|
||||
Status: database.DeliveryStatusFailed,
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
require.NoError(t, db.CreateInBatches(events, 500).Error)
|
||||
require.NoError(t, db.CreateInBatches(deliveries, 500).Error)
|
||||
|
||||
results := make([]database.DeliveryResult, len(deliveries))
|
||||
for i := range deliveries {
|
||||
results[i] = database.DeliveryResult{
|
||||
DeliveryID: deliveries[i].ID, AttemptNum: 1,
|
||||
}
|
||||
}
|
||||
|
||||
require.NoError(t, db.CreateInBatches(results, 500).Error)
|
||||
}
|
||||
|
||||
// seedBareEvents stores count events created at the given time, with
|
||||
// no deliveries.
|
||||
func seedBareEvents(
|
||||
t *testing.T,
|
||||
db *gorm.DB,
|
||||
webhookID string,
|
||||
count int,
|
||||
createdAt time.Time,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
events := make([]database.Event, count)
|
||||
for i := range events {
|
||||
events[i] = database.Event{
|
||||
WebhookID: webhookID,
|
||||
EntrypointID: uuid.New().String(),
|
||||
Method: http.MethodPost,
|
||||
}
|
||||
events[i].CreatedAt = createdAt
|
||||
}
|
||||
|
||||
require.NoError(t, db.CreateInBatches(events, 500).Error)
|
||||
}
|
||||
|
||||
// TestRetentionReaper_PrunesMoreThanOneBatch verifies that a prune
|
||||
// larger than one transaction's batch removes every expired event with
|
||||
// its deliveries and delivery results, keeps the recent event, and
|
||||
// adds what it removed to the event and target totals, so the totals
|
||||
// within retention match the rows still stored.
|
||||
func TestRetentionReaper_PrunesMoreThanOneBatch(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupRetentionTest(t)
|
||||
|
||||
webhookID := createWebhook(t, env.mainDB.DB(), 30)
|
||||
|
||||
db, err := env.mgr.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
|
||||
expired := database.ExportReapBatchSize + 1
|
||||
delivered, failed := uuid.New().String(), uuid.New().String()
|
||||
seedExpiredEvents(t, db, webhookID, expired,
|
||||
time.Now().Add(-40*24*time.Hour), delivered, failed)
|
||||
|
||||
// One recent event, delivered to the first target.
|
||||
recent := seedEventChain(t, db, webhookID, time.Now())
|
||||
require.NoError(t, db.Model(&database.Delivery{}).
|
||||
Where("id = ?", recent.deliveryID).
|
||||
Update("target_id", delivered).Error)
|
||||
|
||||
// The totals storing those rows would have left.
|
||||
n := int64(expired)
|
||||
require.NoError(t, database.AddEventTotals(db, database.EventTotals{
|
||||
Events: n + 1,
|
||||
}))
|
||||
require.NoError(t, database.AddTargetTotals(db, database.TargetTotals{
|
||||
TargetID: delivered, Deliveries: n + 1, Delivered: n + 1,
|
||||
}))
|
||||
require.NoError(t, database.AddTargetTotals(db, database.TargetTotals{
|
||||
TargetID: failed, Deliveries: n, Failed: n,
|
||||
}))
|
||||
|
||||
env.reaper.ExportSweep(context.Background())
|
||||
|
||||
// Only the recent event's rows are left.
|
||||
for _, model := range []any{
|
||||
&database.Event{}, &database.Delivery{}, &database.DeliveryResult{},
|
||||
} {
|
||||
var count int64
|
||||
|
||||
require.NoError(t, db.Model(model).Count(&count).Error)
|
||||
assert.Equal(t, int64(1), count, "%T rows left", model)
|
||||
}
|
||||
|
||||
assertChainPresent(t, db, recent)
|
||||
|
||||
eventTotals := readEventTotals(t, db)
|
||||
assert.Equal(t, database.EventTotals{
|
||||
ID: eventTotals.ID, Events: n + 1, EventsRemoved: n,
|
||||
}, eventTotals)
|
||||
|
||||
targetTotals := readTargetTotals(t, db)
|
||||
assert.Equal(t, map[string]database.TargetTotals{
|
||||
delivered: {
|
||||
TargetID: delivered, Deliveries: n + 1, Delivered: n + 1,
|
||||
DeliveriesRemoved: n,
|
||||
},
|
||||
failed: {
|
||||
TargetID: failed, Deliveries: n, Failed: n,
|
||||
DeliveriesRemoved: n, FailedRemoved: n,
|
||||
},
|
||||
}, targetTotals)
|
||||
|
||||
// A sweep with nothing left to remove changes nothing.
|
||||
env.reaper.ExportSweep(context.Background())
|
||||
|
||||
assert.Equal(t, eventTotals, readEventTotals(t, db))
|
||||
assert.Equal(t, targetTotals, readTargetTotals(t, db))
|
||||
}
|
||||
|
||||
// TestRetentionReaper_WriteDuringPruneSucceeds verifies that a prune
|
||||
// of several batches lets other writers in between its batches: an
|
||||
// event stored once the first batch is deleted is stored while expired
|
||||
// events are still left, not only after the prune has finished.
|
||||
func TestRetentionReaper_WriteDuringPruneSucceeds(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupRetentionTest(t)
|
||||
|
||||
webhookID := createWebhook(t, env.mainDB.DB(), 30)
|
||||
|
||||
db, err := env.mgr.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Three batches of expired events, with nothing else stored: only
|
||||
// the number of batches matters here.
|
||||
expired := 3 * database.ExportReapBatchSize
|
||||
seedBareEvents(t, db, webhookID, expired,
|
||||
time.Now().Add(-40*24*time.Hour))
|
||||
|
||||
cutoff := time.Now().Add(-30 * 24 * time.Hour)
|
||||
countExpired := func() int64 {
|
||||
var count int64
|
||||
|
||||
require.NoError(t, db.Model(&database.Event{}).
|
||||
Where("created_at < ?", cutoff).
|
||||
Count(&count).Error)
|
||||
|
||||
return count
|
||||
}
|
||||
|
||||
pruned := make(chan struct{})
|
||||
|
||||
go func() {
|
||||
defer close(pruned)
|
||||
|
||||
env.reaper.ExportSweep(context.Background())
|
||||
}()
|
||||
|
||||
t.Cleanup(func() { <-pruned })
|
||||
|
||||
// Every stored event is expired until the write below.
|
||||
require.Eventually(t, func() bool {
|
||||
var count int64
|
||||
|
||||
err := db.Model(&database.Event{}).Count(&count).Error
|
||||
|
||||
return err == nil && count < int64(expired)
|
||||
}, 10*time.Second, 10*time.Millisecond)
|
||||
|
||||
event := &database.Event{
|
||||
WebhookID: webhookID,
|
||||
EntrypointID: uuid.New().String(),
|
||||
Method: http.MethodPost,
|
||||
}
|
||||
require.NoError(t, db.Create(event).Error)
|
||||
|
||||
assert.Positive(t, countExpired(),
|
||||
"the event was stored only after the whole prune")
|
||||
|
||||
<-pruned
|
||||
|
||||
assert.Zero(t, countExpired())
|
||||
|
||||
var stored database.Event
|
||||
|
||||
require.NoError(t, db.First(&stored, "id = ?", event.ID).Error)
|
||||
}
|
||||
|
||||
// TestRetentionReaper_StopDuringPruneLeavesTheRest verifies that
|
||||
// stopping the reaper during a prune of several batches returns
|
||||
// between two batches, well inside the stop timeout, leaving the
|
||||
// remaining expired events for the next sweep, and that the totals
|
||||
// match the rows left.
|
||||
func TestRetentionReaper_StopDuringPruneLeavesTheRest(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupRetentionTest(t)
|
||||
|
||||
webhookID := createWebhook(t, env.mainDB.DB(), 30)
|
||||
|
||||
db, err := env.mgr.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Two batches and one more of expired events, a few of them with a
|
||||
// delivered and a failed delivery for the target totals to count.
|
||||
// Most carry nothing else, to keep the test quick.
|
||||
const withDeliveries = 10
|
||||
|
||||
expiredAt := time.Now().Add(-40 * 24 * time.Hour)
|
||||
delivered, failed := uuid.New().String(), uuid.New().String()
|
||||
seedExpiredEvents(t, db, webhookID, withDeliveries, expiredAt,
|
||||
delivered, failed)
|
||||
seedBareEvents(t, db, webhookID,
|
||||
2*database.ExportReapBatchSize+1-withDeliveries, expiredAt)
|
||||
|
||||
n := int64(2*database.ExportReapBatchSize + 1)
|
||||
require.NoError(t, database.AddEventTotals(db, database.EventTotals{
|
||||
Events: n,
|
||||
}))
|
||||
require.NoError(t, database.AddTargetTotals(db, database.TargetTotals{
|
||||
TargetID: delivered, Deliveries: withDeliveries,
|
||||
Delivered: withDeliveries,
|
||||
}))
|
||||
require.NoError(t, database.AddTargetTotals(db, database.TargetTotals{
|
||||
TargetID: failed, Deliveries: withDeliveries,
|
||||
Failed: withDeliveries,
|
||||
}))
|
||||
|
||||
env.reaper.ExportSetInterval(time.Millisecond)
|
||||
env.reaper.ExportStart()
|
||||
|
||||
// Stop once the first batch is deleted. The stop lands in the pause
|
||||
// after it, or at worst during the second batch, so at least the
|
||||
// last event is left.
|
||||
require.Eventually(t, func() bool {
|
||||
var count int64
|
||||
|
||||
err := db.Model(&database.Event{}).Count(&count).Error
|
||||
|
||||
return err == nil && count < n
|
||||
}, 10*time.Second, 10*time.Millisecond)
|
||||
|
||||
// The app's stop timeout.
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
|
||||
require.NoError(t, env.reaper.ExportStop(ctx))
|
||||
|
||||
var events int64
|
||||
|
||||
require.NoError(t, db.Model(&database.Event{}).Count(&events).Error)
|
||||
assert.Positive(t, events, "the stop waited for the whole prune")
|
||||
|
||||
eventTotals := readEventTotals(t, db)
|
||||
assert.Equal(t, events, eventTotals.Events-eventTotals.EventsRemoved)
|
||||
|
||||
targetTotals := readTargetTotals(t, db)
|
||||
require.Len(t, targetTotals, 2)
|
||||
|
||||
for target, totals := range targetTotals {
|
||||
var deliveries, failures int64
|
||||
|
||||
require.NoError(t, db.Model(&database.Delivery{}).
|
||||
Where("target_id = ?", target).
|
||||
Count(&deliveries).Error)
|
||||
require.NoError(t, db.Model(&database.Delivery{}).
|
||||
Where("target_id = ? AND status = ?",
|
||||
target, database.DeliveryStatusFailed).
|
||||
Count(&failures).Error)
|
||||
|
||||
assert.Equal(t, deliveries,
|
||||
totals.Deliveries-totals.DeliveriesRemoved, target)
|
||||
assert.Equal(t, failures, totals.Failed-totals.FailedRemoved,
|
||||
target)
|
||||
}
|
||||
}
|
||||
@@ -35,8 +35,7 @@ var errInvalidCachedDBType = errors.New(
|
||||
|
||||
// WebhookDBManager manages per-webhook SQLite database files
|
||||
// for event storage. Each webhook gets its own dedicated
|
||||
// database containing Events, Deliveries, DeliveryResults and the
|
||||
// running totals of them (EventTotals, TargetTotals).
|
||||
// database containing Events, Deliveries, and DeliveryResults.
|
||||
// Database connections are opened lazily and cached.
|
||||
type WebhookDBManager struct {
|
||||
dataDir string
|
||||
@@ -296,7 +295,6 @@ func (m *WebhookDBManager) openDB(
|
||||
// Run migrations for event-tier models only
|
||||
err = db.AutoMigrate(
|
||||
&Event{}, &Delivery{}, &DeliveryResult{},
|
||||
&EventTotals{}, &TargetTotals{},
|
||||
)
|
||||
if err != nil {
|
||||
_ = sqlDB.Close()
|
||||
@@ -307,18 +305,6 @@ func (m *WebhookDBManager) openDB(
|
||||
)
|
||||
}
|
||||
|
||||
// A new database gets its row of event totals, all zero. Target
|
||||
// totals rows are created by the first delivery to each target.
|
||||
err = db.FirstOrCreate(&EventTotals{}).Error
|
||||
if err != nil {
|
||||
_ = sqlDB.Close()
|
||||
|
||||
return nil, fmt.Errorf(
|
||||
"creating event totals for webhook database %s: %w",
|
||||
webhookID, err,
|
||||
)
|
||||
}
|
||||
|
||||
m.log.Info(
|
||||
"opened per-webhook database",
|
||||
"webhook_id", webhookID,
|
||||
|
||||
@@ -1,177 +0,0 @@
|
||||
package delivery_test
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/gorm"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
)
|
||||
|
||||
// targetTotals reads one target's totals from a webhook database, all
|
||||
// zero when it has no row.
|
||||
func targetTotals(
|
||||
t *testing.T, db *gorm.DB, targetID string,
|
||||
) database.TargetTotals {
|
||||
t.Helper()
|
||||
|
||||
var rows []database.TargetTotals
|
||||
|
||||
require.NoError(t, db.Where("target_id = ?", targetID).
|
||||
Find(&rows).Error)
|
||||
|
||||
if len(rows) == 0 {
|
||||
return database.TargetTotals{TargetID: targetID}
|
||||
}
|
||||
|
||||
return rows[0]
|
||||
}
|
||||
|
||||
// TestUpdateDeliveryStatus_FinishTimeAndTargetTotals pins what a status
|
||||
// write records for the webhook page's statistics: the time a delivery
|
||||
// finished, set only when it becomes delivered or failed, and one more
|
||||
// on its target's delivered or failed total.
|
||||
func TestUpdateDeliveryStatus_FinishTimeAndTargetTotals(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
status database.DeliveryStatus
|
||||
finished bool
|
||||
delivered int64
|
||||
failed int64
|
||||
}{
|
||||
{database.DeliveryStatusRetrying, false, 0, 0},
|
||||
{database.DeliveryStatusDelivered, true, 1, 0},
|
||||
{database.DeliveryStatusFailed, true, 0, 1},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(string(tt.status), func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
db := testWebhookDB(t)
|
||||
e := testEngine(t, 1)
|
||||
event := seedEvent(t, db, `{}`)
|
||||
targetID := uuid.New().String()
|
||||
d := seedDelivery(
|
||||
t, db, event.ID, targetID,
|
||||
database.DeliveryStatusPending,
|
||||
)
|
||||
|
||||
before := time.Now()
|
||||
|
||||
require.NoError(t, e.ExportUpdateDeliveryStatus(
|
||||
db, &d, tt.status,
|
||||
))
|
||||
|
||||
var stored database.Delivery
|
||||
|
||||
require.NoError(t, db.First(&stored, "id = ?", d.ID).Error)
|
||||
assert.Equal(t, tt.status, stored.Status)
|
||||
|
||||
if tt.finished {
|
||||
require.NotNil(t, stored.FinishedAt)
|
||||
assert.False(t, stored.FinishedAt.Before(before))
|
||||
} else {
|
||||
assert.Nil(t, stored.FinishedAt)
|
||||
}
|
||||
|
||||
assert.Equal(t, database.TargetTotals{
|
||||
TargetID: targetID,
|
||||
Delivered: tt.delivered,
|
||||
Failed: tt.failed,
|
||||
}, targetTotals(t, db, targetID))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestUpdateDeliveryStatus_DeletedDeliveryIsNotCounted covers a
|
||||
// delivery retention deleted while the engine still held it. Failing
|
||||
// it afterwards writes no row, so it adds no failure either: retention
|
||||
// has already counted what it removed.
|
||||
func TestUpdateDeliveryStatus_DeletedDeliveryIsNotCounted(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
db := testWebhookDB(t)
|
||||
e := testEngine(t, 1)
|
||||
event := seedEvent(t, db, `{}`)
|
||||
targetID := uuid.New().String()
|
||||
d := seedDelivery(
|
||||
t, db, event.ID, targetID,
|
||||
database.DeliveryStatusRetrying,
|
||||
)
|
||||
|
||||
require.NoError(t, db.Unscoped().
|
||||
Delete(&database.Delivery{}, "id = ?", d.ID).Error)
|
||||
|
||||
require.NoError(t, e.ExportUpdateDeliveryStatus(
|
||||
db, &d, database.DeliveryStatusFailed,
|
||||
))
|
||||
|
||||
assert.Equal(t, database.TargetTotals{TargetID: targetID},
|
||||
targetTotals(t, db, targetID))
|
||||
}
|
||||
|
||||
// TestUpdateDeliveryStatus_FinishedDeliveryIsNotSettledAgain covers a
|
||||
// delivery settled a second time, as recovery can do when a worker has
|
||||
// settled it since recovery read it. Neither status writes over the
|
||||
// first, and the totals do not move.
|
||||
func TestUpdateDeliveryStatus_FinishedDeliveryIsNotSettledAgain(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
finished := []database.DeliveryStatus{
|
||||
database.DeliveryStatusDelivered,
|
||||
database.DeliveryStatusFailed,
|
||||
}
|
||||
|
||||
for _, first := range finished {
|
||||
t.Run(string(first), func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
db := testWebhookDB(t)
|
||||
e := testEngine(t, 1)
|
||||
event := seedEvent(t, db, `{}`)
|
||||
targetID := uuid.New().String()
|
||||
d := seedDelivery(
|
||||
t, db, event.ID, targetID,
|
||||
database.DeliveryStatusRetrying,
|
||||
)
|
||||
|
||||
// The delivery as recovery read it, before the worker
|
||||
// settled it.
|
||||
readBefore := d
|
||||
|
||||
require.NoError(t, e.ExportUpdateDeliveryStatus(
|
||||
db, &d, first,
|
||||
))
|
||||
|
||||
var settled database.Delivery
|
||||
|
||||
require.NoError(t, db.First(&settled, "id = ?", d.ID).Error)
|
||||
require.NotNil(t, settled.FinishedAt)
|
||||
|
||||
totals := targetTotals(t, db, targetID)
|
||||
|
||||
for _, again := range finished {
|
||||
stale := readBefore
|
||||
|
||||
require.NoError(t, e.ExportUpdateDeliveryStatus(
|
||||
db, &stale, again,
|
||||
))
|
||||
}
|
||||
|
||||
var stored database.Delivery
|
||||
|
||||
require.NoError(t, db.First(&stored, "id = ?", d.ID).Error)
|
||||
assert.Equal(t, first, stored.Status)
|
||||
require.NotNil(t, stored.FinishedAt)
|
||||
assert.True(t, settled.FinishedAt.Equal(*stored.FinishedAt))
|
||||
assert.Equal(t, totals, targetTotals(t, db, targetID))
|
||||
})
|
||||
}
|
||||
}
|
||||
+10
-50
@@ -148,6 +148,7 @@ type EngineParams struct {
|
||||
DBManager *database.WebhookDBManager
|
||||
Logger *logger.Logger
|
||||
SSRFGuard *Guard
|
||||
Metrics *metrics.Set
|
||||
}
|
||||
|
||||
// Engine processes queued deliveries in the background
|
||||
@@ -167,10 +168,10 @@ type Engine struct {
|
||||
retryCh chan Task
|
||||
workers int
|
||||
|
||||
// mtr is the delivery metric set. Production wires the
|
||||
// process-wide one; a test can substitute a set registered on
|
||||
// a private registry so its assertions are not disturbed by
|
||||
// deliveries other tests are making at the same time.
|
||||
// mtr is the delivery metric set. Production wires the one
|
||||
// registered on the registry /metrics serves; a test can
|
||||
// substitute a set registered on a registry it holds, so it can
|
||||
// gather what its own deliveries recorded.
|
||||
mtr *metrics.Set
|
||||
|
||||
// targets maps each target type to its implementation.
|
||||
@@ -204,7 +205,7 @@ func New(
|
||||
deliveryCh: make(chan Task, deliveryChannelSize),
|
||||
retryCh: make(chan Task, retryChannelSize),
|
||||
workers: defaultWorkers,
|
||||
mtr: metrics.Default(),
|
||||
mtr: params.Metrics,
|
||||
}
|
||||
|
||||
e.initTargets(&http.Client{
|
||||
@@ -531,11 +532,6 @@ func (e *Engine) processRetryTask(
|
||||
return
|
||||
}
|
||||
|
||||
// Set before anything below can fail the delivery: the failure is
|
||||
// added to this target's totals.
|
||||
d.EventID = task.EventID
|
||||
d.TargetID = task.TargetID
|
||||
|
||||
if d.Status != database.DeliveryStatusRetrying {
|
||||
e.log.Debug(
|
||||
"skipping retry for delivery "+
|
||||
@@ -567,6 +563,8 @@ func (e *Engine) processRetryTask(
|
||||
}
|
||||
|
||||
target := buildTargetFromTask(task)
|
||||
d.EventID = task.EventID
|
||||
d.TargetID = task.TargetID
|
||||
d.Event = event
|
||||
d.Target = target
|
||||
|
||||
@@ -1557,9 +1555,8 @@ func (e *Engine) updateDeliveryStatus(
|
||||
targetType database.TargetType,
|
||||
status database.DeliveryStatus,
|
||||
) error {
|
||||
err := webhookDB.Transaction(func(tx *gorm.DB) error {
|
||||
return writeDeliveryStatus(tx, d, status)
|
||||
})
|
||||
err := webhookDB.Model(d).
|
||||
Update("status", status).Error
|
||||
if err != nil {
|
||||
return fmt.Errorf(
|
||||
"updating delivery %s to status %s: %w",
|
||||
@@ -1578,43 +1575,6 @@ func (e *Engine) updateDeliveryStatus(
|
||||
return nil
|
||||
}
|
||||
|
||||
// writeDeliveryStatus writes a delivery's new status. A delivery that
|
||||
// becomes delivered or failed also gets the time it finished, and is
|
||||
// added to its target's delivered or failed total. That write changes
|
||||
// only a delivery not yet delivered or failed, and the total moves
|
||||
// only when it changed a row: retention may have deleted the delivery
|
||||
// while the engine was working on it, and a recovery path may settle
|
||||
// a delivery that a worker has already settled.
|
||||
func writeDeliveryStatus(
|
||||
tx *gorm.DB,
|
||||
d *database.Delivery,
|
||||
status database.DeliveryStatus,
|
||||
) error {
|
||||
if !status.Terminal() {
|
||||
return tx.Model(d).Update("status", status).Error
|
||||
}
|
||||
|
||||
res := tx.Model(d).
|
||||
Where("status NOT IN ?", []database.DeliveryStatus{
|
||||
database.DeliveryStatusDelivered,
|
||||
database.DeliveryStatusFailed,
|
||||
}).
|
||||
Updates(map[string]any{
|
||||
"status": status,
|
||||
"finished_at": time.Now(),
|
||||
})
|
||||
if res.Error != nil || res.RowsAffected == 0 {
|
||||
return res.Error
|
||||
}
|
||||
|
||||
add := database.TargetTotals{TargetID: d.TargetID, Delivered: 1}
|
||||
if status == database.DeliveryStatusFailed {
|
||||
add = database.TargetTotals{TargetID: d.TargetID, Failed: 1}
|
||||
}
|
||||
|
||||
return database.AddTargetTotals(tx, add)
|
||||
}
|
||||
|
||||
// settleStatus moves a delivery to its outcome status and reports a
|
||||
// failed write through bookkeepingFailed, which leaves the row
|
||||
// recoverable. It exists so the target call sites read as one
|
||||
|
||||
@@ -57,10 +57,7 @@ func testWebhookDB(t *testing.T) *gorm.DB {
|
||||
&database.Event{},
|
||||
&database.Delivery{},
|
||||
&database.DeliveryResult{},
|
||||
&database.EventTotals{},
|
||||
&database.TargetTotals{},
|
||||
))
|
||||
require.NoError(t, db.Create(&database.EventTotals{}).Error)
|
||||
|
||||
return db
|
||||
}
|
||||
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"net/url"
|
||||
"time"
|
||||
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"go.uber.org/fx"
|
||||
"gorm.io/gorm"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
@@ -40,6 +41,11 @@ const (
|
||||
ExportPendingSweepMinAge = pendingSweepMinAge
|
||||
)
|
||||
|
||||
// ExportIsBlockedIP exposes isBlockedIP for testing.
|
||||
func ExportIsBlockedIP(ip net.IP) bool {
|
||||
return isBlockedIP(ip)
|
||||
}
|
||||
|
||||
// NewTestGuard builds an SSRF Guard from an explicit egress
|
||||
// allowlist, without going through config. Passing no prefixes
|
||||
// yields the default guard, which blocks every private/reserved
|
||||
@@ -65,11 +71,6 @@ func ExportBlockedNetworks() []*net.IPNet {
|
||||
return blockedNetworks
|
||||
}
|
||||
|
||||
// ExportBlockedPublicNetworks exposes blockedPublicNetworks.
|
||||
func ExportBlockedPublicNetworks() []*net.IPNet {
|
||||
return blockedPublicNetworks
|
||||
}
|
||||
|
||||
// ExportIsForwardableHeader exposes isForwardableHeader.
|
||||
func ExportIsForwardableHeader(name string) bool {
|
||||
return isForwardableHeader(name)
|
||||
@@ -150,16 +151,6 @@ func (e *Engine) ExportDeliverSlack(
|
||||
)
|
||||
}
|
||||
|
||||
// ExportUpdateDeliveryStatus exposes updateDeliveryStatus. It passes no
|
||||
// target type, so no metric moves.
|
||||
func (e *Engine) ExportUpdateDeliveryStatus(
|
||||
webhookDB *gorm.DB,
|
||||
d *database.Delivery,
|
||||
status database.DeliveryStatus,
|
||||
) error {
|
||||
return e.updateDeliveryStatus(webhookDB, d, "", status)
|
||||
}
|
||||
|
||||
// ExportProcessNewTask exposes processNewTask.
|
||||
func (e *Engine) ExportProcessNewTask(
|
||||
ctx context.Context, task *Task,
|
||||
@@ -399,7 +390,7 @@ func NewTestEngine(
|
||||
deliveryCh: make(chan Task, deliveryChannelSize),
|
||||
retryCh: make(chan Task, retryChannelSize),
|
||||
workers: workers,
|
||||
mtr: metrics.Default(),
|
||||
mtr: metrics.New(prometheus.NewRegistry()),
|
||||
}
|
||||
e.initTargets(client)
|
||||
|
||||
@@ -414,7 +405,7 @@ func NewTestEngineSmallRetry(
|
||||
e := &Engine{
|
||||
log: log,
|
||||
retryCh: make(chan Task, 1),
|
||||
mtr: metrics.Default(),
|
||||
mtr: metrics.New(prometheus.NewRegistry()),
|
||||
}
|
||||
e.initTargets(nil)
|
||||
|
||||
@@ -437,7 +428,7 @@ func NewTestEngineWithDB(
|
||||
deliveryCh: make(chan Task, deliveryChannelSize),
|
||||
retryCh: make(chan Task, retryChannelSize),
|
||||
workers: workers,
|
||||
mtr: metrics.Default(),
|
||||
mtr: metrics.New(prometheus.NewRegistry()),
|
||||
}
|
||||
e.initTargets(client)
|
||||
|
||||
@@ -445,8 +436,7 @@ func NewTestEngineWithDB(
|
||||
}
|
||||
|
||||
// ExportSetMetrics substitutes the engine's metric set, so a test can
|
||||
// assert on collectors registered on a private registry instead of
|
||||
// the process-wide ones every other test is also moving.
|
||||
// assert on collectors registered on a registry it holds.
|
||||
func (e *Engine) ExportSetMetrics(mtr *metrics.Set) {
|
||||
e.mtr = mtr
|
||||
}
|
||||
|
||||
@@ -35,9 +35,8 @@ const (
|
||||
)
|
||||
|
||||
// mIsolate gives the setup's engine a metric set registered on a
|
||||
// private registry. The process-wide collectors are moved by every
|
||||
// other delivery test running in parallel, so exact assertions are
|
||||
// only possible against a registry this test owns.
|
||||
// registry this test holds, so its exact assertions can gather from
|
||||
// it.
|
||||
func mIsolate(
|
||||
t *testing.T, s iSetup,
|
||||
) *prometheus.Registry {
|
||||
|
||||
+24
-45
@@ -25,16 +25,8 @@ var (
|
||||
errNoIPs = errors.New(
|
||||
"hostname resolved to no IP addresses",
|
||||
)
|
||||
// ErrBlockedPrivateOrReservedIP reports an address in the
|
||||
// default blocklist's private and reserved ranges,
|
||||
// blockedNetworks.
|
||||
ErrBlockedPrivateOrReservedIP = errors.New(
|
||||
"blocked private or reserved address",
|
||||
)
|
||||
// errBlockedPublicMetadata reports a public address on the
|
||||
// default blocklist, one in blockedPublicNetworks.
|
||||
errBlockedPublicMetadata = errors.New(
|
||||
"blocked cloud metadata address",
|
||||
errBlockedIP = errors.New(
|
||||
"blocked private, reserved or cloud metadata address",
|
||||
)
|
||||
errBlockedMetadata = errors.New(
|
||||
"blocked link-local or cloud instance metadata " +
|
||||
@@ -45,31 +37,21 @@ var (
|
||||
)
|
||||
)
|
||||
|
||||
// blockedNetworks and blockedPublicNetworks together are the
|
||||
// default blocklist: the private and reserved IP ranges, plus
|
||||
// the public cloud metadata addresses, that are blocked to
|
||||
// prevent SSRF attacks. An operator can permit specific blocks
|
||||
// out of this set with ALLOWED_EGRESS_CIDRS; see Guard.
|
||||
//
|
||||
// blockedNetworks holds the private and reserved IP ranges.
|
||||
//
|
||||
//nolint:gochecknoglobals // package-level network list is appropriate here
|
||||
var blockedNetworks []*net.IPNet
|
||||
|
||||
// blockedPublicNetworks holds the default blocklist's public
|
||||
// addresses, kept apart from blockedNetworks so that they are
|
||||
// refused as cloud metadata addresses, never as private or
|
||||
// reserved ones.
|
||||
// blockedNetworks is the default blocklist: the private and
|
||||
// reserved IP ranges, plus the public cloud metadata addresses,
|
||||
// that are blocked to prevent SSRF attacks. An operator can
|
||||
// permit specific blocks out of this set with
|
||||
// ALLOWED_EGRESS_CIDRS; see Guard.
|
||||
//
|
||||
// A public address belongs on the default blocklist only if it
|
||||
// hands credentials, user data or bootstrap material to whatever
|
||||
// can reach it, without the caller presenting anything; it goes
|
||||
// in this list. A provider's other public addresses are not
|
||||
// refused, since reaching them can be legitimate and no list of
|
||||
// them could be complete.
|
||||
// can reach it, without the caller presenting anything. A
|
||||
// provider's other public addresses are not refused, since
|
||||
// reaching them can be legitimate and no list of them could be
|
||||
// complete.
|
||||
//
|
||||
//nolint:gochecknoglobals // package-level network list is appropriate here
|
||||
var blockedPublicNetworks []*net.IPNet
|
||||
var blockedNetworks []*net.IPNet
|
||||
|
||||
// alwaysBlockedNetworks are the ranges no configuration can
|
||||
// open: the link-local blocks and the cloud instance metadata
|
||||
@@ -106,8 +88,8 @@ var blockedPublicNetworks []*net.IPNet
|
||||
// when it clears both halves. Nothing in this list can be
|
||||
// reopened, so putting a public address here leaves the operator
|
||||
// no escape hatch at all — the condition ALLOWED_EGRESS_CIDRS
|
||||
// exists to remove. Default-block it in blockedPublicNetworks
|
||||
// instead, which an allowlist can override.
|
||||
// exists to remove. Default-block it in blockedNetworks instead,
|
||||
// which an allowlist can override.
|
||||
//
|
||||
// This is a criterion, not an enumeration of every metadata
|
||||
// address in existence.
|
||||
@@ -148,9 +130,6 @@ func init() {
|
||||
"::1/128",
|
||||
"fc00::/7",
|
||||
"fe80::/10",
|
||||
})
|
||||
|
||||
blockedPublicNetworks = mustParseCIDRs([]string{
|
||||
// Azure WireServer, a public address that serves VM credentials.
|
||||
"168.63.129.16/32",
|
||||
})
|
||||
@@ -246,6 +225,13 @@ func matchesAny(networks []*net.IPNet, ip net.IP) bool {
|
||||
return false
|
||||
}
|
||||
|
||||
// isBlockedIP checks whether an IP address falls within
|
||||
// the default blocklist, before any operator allowlist is
|
||||
// considered.
|
||||
func isBlockedIP(ip net.IP) bool {
|
||||
return matchesAny(blockedNetworks, ip)
|
||||
}
|
||||
|
||||
// Guard makes every SSRF decision in the process.
|
||||
//
|
||||
// It holds the operator's ALLOWED_EGRESS_CIDRS allowlist and
|
||||
@@ -346,8 +332,7 @@ func (g *Guard) allows(ip net.IP) bool {
|
||||
// consulted, so no configured CIDR reaches link-local or a
|
||||
// cloud metadata endpoint at a non-public address.
|
||||
// 2. The allowlist is consulted next, so a listed private
|
||||
// network, or a listed public address on the default
|
||||
// blocklist, becomes reachable.
|
||||
// network becomes reachable.
|
||||
// 3. Everything else keeps the default blocklist's answer.
|
||||
func (g *Guard) checkIP(ip net.IP) error {
|
||||
if matchesAny(alwaysBlockedNetworks, ip) {
|
||||
@@ -360,15 +345,9 @@ func (g *Guard) checkIP(ip net.IP) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
if matchesAny(blockedNetworks, ip) {
|
||||
if isBlockedIP(ip) {
|
||||
return fmt.Errorf(
|
||||
"target IP %s: %w", ip, ErrBlockedPrivateOrReservedIP,
|
||||
)
|
||||
}
|
||||
|
||||
if matchesAny(blockedPublicNetworks, ip) {
|
||||
return fmt.Errorf(
|
||||
"target IP %s: %w", ip, errBlockedPublicMetadata,
|
||||
"target IP %s: %w", ip, errBlockedIP,
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
@@ -23,10 +23,6 @@ const (
|
||||
metadataIP = "169.254.169.254"
|
||||
metadataURL = "http://" + metadataIP + "/latest/meta-data/"
|
||||
|
||||
// linkLocalIPv4 is the IPv4 link-local block, which holds
|
||||
// metadataIP.
|
||||
linkLocalIPv4 = "169.254.0.0/16"
|
||||
|
||||
// loopbackHookURL is a target on this host: blocked by
|
||||
// default, reachable only once an operator allowlists
|
||||
// loopback.
|
||||
@@ -241,7 +237,7 @@ func linkLocalRefusedCases() []metadataAlwaysRefusedCase {
|
||||
},
|
||||
{
|
||||
name: "whole link-local block",
|
||||
allow: linkLocalIPv4,
|
||||
allow: "169.254.0.0/16",
|
||||
target: metadataURL,
|
||||
},
|
||||
{
|
||||
@@ -416,9 +412,6 @@ func TestGuardAllowlist_AzureWireServerReopenable(t *testing.T) {
|
||||
"WireServer must be refused by the default blocklist, "+
|
||||
"which an allowlist can override",
|
||||
)
|
||||
require.NotErrorIs(t, err, delivery.ErrBlockedPrivateOrReservedIP,
|
||||
"WireServer is public, not private or reserved",
|
||||
)
|
||||
|
||||
assertDialRefused(t, defaultGuard, target)
|
||||
|
||||
@@ -503,7 +496,7 @@ func TestAlwaysBlockedNetworks_PinnedSet(t *testing.T) {
|
||||
want := []string{
|
||||
// IPv4 link-local: the 169.254.169.254 metadata
|
||||
// service on AWS, Azure and others.
|
||||
linkLocalIPv4,
|
||||
"169.254.0.0/16",
|
||||
// IPv6 link-local.
|
||||
"fe80::/10",
|
||||
// AWS IPv6 IMDS, inside the ULA space an operator may
|
||||
@@ -533,90 +526,6 @@ func TestAlwaysBlockedNetworks_PinnedSet(t *testing.T) {
|
||||
assert.Equal(t, want, got)
|
||||
}
|
||||
|
||||
// TestDefaultBlocklist_PinnedSet pins each list of the default
|
||||
// blocklist on its own, the private and reserved ranges in
|
||||
// blockedNetworks and the public addresses in
|
||||
// blockedPublicNetworks, so moving an entry from one list to the
|
||||
// other fails it. For the first address of each entry it then
|
||||
// checks that the default guard refuses it, and that listing the
|
||||
// entry in ALLOWED_EGRESS_CIDRS opens it unless the unconditional
|
||||
// set holds that address.
|
||||
func TestDefaultBlocklist_PinnedSet(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// public marks an entry of blockedPublicNetworks; every other
|
||||
// entry belongs in blockedNetworks.
|
||||
tests := []struct {
|
||||
cidr string
|
||||
public bool
|
||||
reopenable bool
|
||||
}{
|
||||
{cidr: "127.0.0.0/8", reopenable: true},
|
||||
{cidr: "10.0.0.0/8", reopenable: true},
|
||||
{cidr: "172.16.0.0/12", reopenable: true},
|
||||
{cidr: "192.168.0.0/16", reopenable: true},
|
||||
{cidr: linkLocalIPv4, reopenable: false},
|
||||
{cidr: "0.0.0.0/8", reopenable: true},
|
||||
{cidr: "100.64.0.0/10", reopenable: true},
|
||||
{cidr: "192.0.0.0/24", reopenable: true},
|
||||
{cidr: "192.0.2.0/24", reopenable: true},
|
||||
{cidr: "198.18.0.0/15", reopenable: true},
|
||||
{cidr: "198.51.100.0/24", reopenable: true},
|
||||
{cidr: "203.0.113.0/24", reopenable: true},
|
||||
{cidr: "224.0.0.0/4", reopenable: true},
|
||||
{cidr: "240.0.0.0/4", reopenable: true},
|
||||
{cidr: "::1/128", reopenable: true},
|
||||
{cidr: "fc00::/7", reopenable: true},
|
||||
{cidr: "fe80::/10", reopenable: false},
|
||||
{cidr: "168.63.129.16/32", public: true, reopenable: true},
|
||||
}
|
||||
|
||||
wantPrivate := make([]string, 0, len(tests))
|
||||
wantPublic := make([]string, 0, len(tests))
|
||||
|
||||
for _, tt := range tests {
|
||||
if tt.public {
|
||||
wantPublic = append(wantPublic, tt.cidr)
|
||||
} else {
|
||||
wantPrivate = append(wantPrivate, tt.cidr)
|
||||
}
|
||||
}
|
||||
|
||||
gotPrivate := make([]string, 0, len(tests))
|
||||
for _, n := range delivery.ExportBlockedNetworks() {
|
||||
gotPrivate = append(gotPrivate, n.String())
|
||||
}
|
||||
|
||||
gotPublic := make([]string, 0, len(tests))
|
||||
for _, n := range delivery.ExportBlockedPublicNetworks() {
|
||||
gotPublic = append(gotPublic, n.String())
|
||||
}
|
||||
|
||||
assert.ElementsMatch(t, wantPrivate, gotPrivate, "blockedNetworks")
|
||||
assert.ElementsMatch(t, wantPublic, gotPublic, "blockedPublicNetworks")
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.cidr, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
prefix := netip.MustParsePrefix(tt.cidr)
|
||||
ip := net.IP(prefix.Addr().AsSlice())
|
||||
|
||||
require.Error(t,
|
||||
delivery.NewTestGuard().ExportCheckIP(ip),
|
||||
"the default guard must refuse %s", ip,
|
||||
)
|
||||
|
||||
err := delivery.NewTestGuard(prefix).ExportCheckIP(ip)
|
||||
if tt.reopenable {
|
||||
assert.NoError(t, err, "listing %s must open it", tt.cidr)
|
||||
} else {
|
||||
assert.Error(t, err, "listing %s must not open it", tt.cidr)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// requireLoopback fails the test unless rawURL's host is a
|
||||
// loopback address, so the allowlist test cannot silently stop
|
||||
// exercising a blocked range.
|
||||
|
||||
@@ -10,7 +10,7 @@ import (
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
)
|
||||
|
||||
func TestGuardCheckIP_PrivateRanges(t *testing.T) {
|
||||
func TestIsBlockedIP_PrivateRanges(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
@@ -56,14 +56,12 @@ func TestGuardCheckIP_PrivateRanges(t *testing.T) {
|
||||
"failed to parse IP %s", tt.ip,
|
||||
)
|
||||
|
||||
refused := delivery.NewTestGuard().ExportCheckIP(ip) != nil
|
||||
|
||||
assert.Equal(t,
|
||||
tt.blocked,
|
||||
refused,
|
||||
"default guard refuses %s = %v, want %v",
|
||||
delivery.ExportIsBlockedIP(ip),
|
||||
"isBlockedIP(%s) = %v, want %v",
|
||||
tt.ip,
|
||||
refused,
|
||||
delivery.ExportIsBlockedIP(ip),
|
||||
tt.blocked,
|
||||
)
|
||||
})
|
||||
|
||||
@@ -418,38 +418,6 @@ func TestProcessRetryTask_TargetDeleted_MakesNoAttempt(
|
||||
assert.Zero(t, s.Engine.ExportInflightHeld())
|
||||
}
|
||||
|
||||
// TestProcessRetryTask_TargetDeleted_CountsFailureOnTarget verifies
|
||||
// that the failure of a retry abandoned because its target is gone is
|
||||
// added to that target's own totals, not to a row with no target.
|
||||
func TestProcessRetryTask_TargetDeleted_CountsFailureOnTarget(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
s := newISetup(t)
|
||||
|
||||
var hits atomic.Int64
|
||||
|
||||
task, targetID := tRetryChainSetup(
|
||||
t, s, "gone-counted", &hits,
|
||||
)
|
||||
|
||||
require.NoError(t, s.MainDB.Delete(
|
||||
&database.Target{}, "id = ?", targetID,
|
||||
).Error)
|
||||
|
||||
s.Engine.ExportProcessRetryTask(
|
||||
context.Background(), &task,
|
||||
)
|
||||
|
||||
var rows []database.TargetTotals
|
||||
|
||||
require.NoError(t, s.WebhookDB.Find(&rows).Error)
|
||||
assert.Equal(t, []database.TargetTotals{
|
||||
{TargetID: targetID, Failed: 1},
|
||||
}, rows)
|
||||
}
|
||||
|
||||
// TestProcessRetryTask_TargetPresent_StillDelivers is the guard's
|
||||
// mutation check: a liveness check that refused every retry would pass
|
||||
// the test above and break every retry there is.
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
// SQL — parameters and all — for every statement that returns an
|
||||
// error, including gorm.ErrRecordNotFound. Two of this service's
|
||||
// lookups miss by design on unauthenticated routes: the entrypoint
|
||||
// lookup on /h/{uuid}, whose path segment the client picks
|
||||
// lookup on /webhook/{uuid}, whose path segment the client picks
|
||||
// outright, and the user lookup behind the login form, whose username
|
||||
// the client picks outright. Under the default logger each of those
|
||||
// misses printed an unbounded, attacker-chosen string, at no level the
|
||||
|
||||
@@ -2,56 +2,19 @@ package handlers
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"strings"
|
||||
"unicode"
|
||||
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/logfield"
|
||||
"sneak.berlin/go/webhooker/internal/middleware"
|
||||
)
|
||||
|
||||
// loginDestination returns where a successful login sends the
|
||||
// browser: next when it is a path on this site, otherwise "/", which
|
||||
// leads to the webhook list.
|
||||
//
|
||||
// A browser reads "//host" as another site, reads "\" as "/", and
|
||||
// drops tabs and newlines before reading at all. So the value must
|
||||
// start with exactly one "/" and hold no "\" or control character
|
||||
// anywhere: http.Redirect cleans "/a/../\host" down to "/\host". It
|
||||
// is checked after percent-decoding, so an encoded form of any of
|
||||
// these is refused too.
|
||||
func loginDestination(next string) string {
|
||||
if len(next) > middleware.MaxNextBytes {
|
||||
return "/"
|
||||
}
|
||||
|
||||
decoded, err := url.PathUnescape(next)
|
||||
if err != nil ||
|
||||
!strings.HasPrefix(decoded, "/") ||
|
||||
strings.HasPrefix(decoded, "//") ||
|
||||
strings.Contains(decoded, `\`) ||
|
||||
strings.ContainsFunc(decoded, unicode.IsControl) {
|
||||
return "/"
|
||||
}
|
||||
|
||||
return next
|
||||
}
|
||||
|
||||
// HandleLoginPage returns a handler for the login page (GET)
|
||||
func (h *Handlers) HandleLoginPage() http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
next := loginDestination(
|
||||
r.URL.Query().Get(middleware.NextParam),
|
||||
)
|
||||
|
||||
// Check if already logged in
|
||||
sess, err := h.session.Get(r)
|
||||
if err == nil && h.session.IsAuthenticated(sess) {
|
||||
http.Redirect( //nolint:gosec // checked by loginDestination
|
||||
w, r, next, http.StatusSeeOther,
|
||||
)
|
||||
http.Redirect(w, r, "/", http.StatusSeeOther)
|
||||
|
||||
return
|
||||
}
|
||||
@@ -59,7 +22,6 @@ func (h *Handlers) HandleLoginPage() http.HandlerFunc {
|
||||
// Render login page
|
||||
data := map[string]any{
|
||||
tmplKeyError: "",
|
||||
tmplKeyNext: next,
|
||||
}
|
||||
|
||||
h.renderTemplate(w, r, "login.html", data)
|
||||
@@ -115,13 +77,8 @@ func (h *Handlers) HandleLoginSubmit() http.HandlerFunc {
|
||||
"user_id", user.ID,
|
||||
)
|
||||
|
||||
// The form value is the client's to set, so it is checked
|
||||
// again here rather than trusted from the rendered page.
|
||||
http.Redirect( //nolint:gosec // checked by loginDestination
|
||||
w, r,
|
||||
loginDestination(r.PostFormValue(middleware.NextParam)),
|
||||
http.StatusSeeOther,
|
||||
)
|
||||
// Redirect to home page
|
||||
http.Redirect(w, r, "/", http.StatusSeeOther)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -134,9 +91,6 @@ func (h *Handlers) renderLoginError(
|
||||
) {
|
||||
data := map[string]any{
|
||||
tmplKeyError: msg,
|
||||
tmplKeyNext: loginDestination(
|
||||
r.PostFormValue(middleware.NextParam),
|
||||
),
|
||||
}
|
||||
|
||||
w.WriteHeader(status)
|
||||
|
||||
@@ -454,202 +454,6 @@ func TestLogin_SuccessCreatesSession(t *testing.T) {
|
||||
)
|
||||
}
|
||||
|
||||
// TestLogin_ReturnsOnlyToAPathOnThisSite is the security half of
|
||||
// https://git.eeqj.de/sneak/webhooker/issues/384: the page a login
|
||||
// returns to is client-chosen, so anything that is not a path on this
|
||||
// site, plain or percent-encoded, must land on "/", the webhook list.
|
||||
func TestLogin_ReturnsOnlyToAPathOnThisSite(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
db *database.Database
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &db)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
seedOperator(t, db)
|
||||
|
||||
cases := []struct{ next, want string }{
|
||||
{"/hook/abc/events?page=2", "/hook/abc/events?page=2"},
|
||||
{"", "/"},
|
||||
{"https://evil.example/", "/"},
|
||||
{"https%3A%2F%2Fevil.example%2F", "/"},
|
||||
{"//evil.example/", "/"},
|
||||
{"%2F%2Fevil.example/", "/"},
|
||||
{"/%2Fevil.example/", "/"},
|
||||
{`/\evil.example/`, "/"},
|
||||
{"%2F%5Cevil.example/", "/"},
|
||||
{"/%5Cevil.example/", "/"},
|
||||
{`/a/../\evil.example/`, "/"},
|
||||
{"/\t/evil.example/", "/"},
|
||||
{"/%09/evil.example/", "/"},
|
||||
{"/\n/evil.example/", "/"},
|
||||
{"/%0A/evil.example/", "/"},
|
||||
{"/\r/evil.example/", "/"},
|
||||
{"/%0D/evil.example/", "/"},
|
||||
{"/%00/evil.example/", "/"},
|
||||
{"/%7F/evil.example/", "/"},
|
||||
{"%252F%252Fevil.example/", "/"},
|
||||
{"https%253A%252F%252Fevil.example%252F", "/"},
|
||||
{"/" + strings.Repeat("a", 4096), "/"},
|
||||
}
|
||||
|
||||
for _, c := range cases {
|
||||
form := url.Values{}
|
||||
form.Set("username", operatorUser)
|
||||
form.Set("password", operatorPassword)
|
||||
form.Set("next", c.next)
|
||||
|
||||
req := httptest.NewRequestWithContext(
|
||||
context.Background(),
|
||||
http.MethodPost,
|
||||
"/pages/login",
|
||||
strings.NewReader(form.Encode()),
|
||||
)
|
||||
req.Header.Set(
|
||||
"Content-Type", "application/x-www-form-urlencoded",
|
||||
)
|
||||
req.RemoteAddr = sharedProxyPeer
|
||||
|
||||
w := httptest.NewRecorder()
|
||||
h.HandleLoginSubmit().ServeHTTP(w, req)
|
||||
|
||||
assert.Equal(t, http.StatusSeeOther, w.Code, "next %q", c.next)
|
||||
assert.Equal(
|
||||
t, c.want, w.Header().Get("Location"), "next %q", c.next,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// TestLogin_WrongPasswordKeepsTheRequestedPage: after a wrong
|
||||
// password the login page is shown again with the same next, so the
|
||||
// next attempt still returns to the page that was asked for.
|
||||
func TestLogin_WrongPasswordKeepsTheRequestedPage(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
db *database.Database
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &db)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
seedOperator(t, db)
|
||||
|
||||
form := url.Values{}
|
||||
form.Set("username", operatorUser)
|
||||
form.Set("password", "wrong")
|
||||
form.Set("next", "/hook/abc")
|
||||
|
||||
req := httptest.NewRequestWithContext(
|
||||
context.Background(),
|
||||
http.MethodPost,
|
||||
"/pages/login",
|
||||
strings.NewReader(form.Encode()),
|
||||
)
|
||||
req.Header.Set(
|
||||
"Content-Type", "application/x-www-form-urlencoded",
|
||||
)
|
||||
req.RemoteAddr = sharedProxyPeer
|
||||
|
||||
w := httptest.NewRecorder()
|
||||
h.HandleLoginSubmit().ServeHTTP(w, req)
|
||||
|
||||
assert.Equal(t, http.StatusUnauthorized, w.Code)
|
||||
assert.Contains(
|
||||
t, w.Body.String(), `name="next" value="/hook/abc"`,
|
||||
)
|
||||
}
|
||||
|
||||
// loginPageGet renders the login page as a GET with the given next
|
||||
// value and cookies.
|
||||
func loginPageGet(
|
||||
h *handlers.Handlers, next string, cookies []*http.Cookie,
|
||||
) *httptest.ResponseRecorder {
|
||||
req := httptest.NewRequestWithContext(
|
||||
context.Background(), http.MethodGet,
|
||||
"/pages/login?"+url.Values{"next": {next}}.Encode(), nil,
|
||||
)
|
||||
|
||||
for _, c := range cookies {
|
||||
req.AddCookie(c)
|
||||
}
|
||||
|
||||
w := httptest.NewRecorder()
|
||||
h.HandleLoginPage().ServeHTTP(w, req)
|
||||
|
||||
return w
|
||||
}
|
||||
|
||||
// TestLoginPage_CarriesOnlyAPathOnThisSite covers the login page
|
||||
// itself: its form carries the requested page only when it is a path
|
||||
// on this site, and a browser already logged in goes straight there,
|
||||
// or to "/" when it is not.
|
||||
func TestLoginPage_CarriesOnlyAPathOnThisSite(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
sess *session.Session
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
assert.Contains(
|
||||
t, loginPageGet(h, "/hook/abc", nil).Body.String(),
|
||||
`name="next" value="/hook/abc"`,
|
||||
)
|
||||
assert.Contains(
|
||||
t, loginPageGet(h, "//evil.example/", nil).Body.String(),
|
||||
`name="next" value="/"`,
|
||||
)
|
||||
|
||||
cookies := authenticatedCookies(t, sess, "test-user-id", "testuser")
|
||||
|
||||
cases := []struct{ next, want string }{
|
||||
{"/hook/abc", "/hook/abc"},
|
||||
{"//evil.example/", "/"},
|
||||
{`/\evil.example/`, "/"},
|
||||
}
|
||||
|
||||
for _, c := range cases {
|
||||
w := loginPageGet(h, c.next, cookies)
|
||||
|
||||
assert.Equal(t, http.StatusSeeOther, w.Code, "next %q", c.next)
|
||||
assert.Equal(
|
||||
t, c.want, w.Header().Get("Location"), "next %q", c.next,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// TestLoginPage_HasNoLinkToItself: the navigation bar on the login
|
||||
// page offers no link to the login page.
|
||||
func TestLoginPage_HasNoLinkToItself(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var h *handlers.Handlers
|
||||
|
||||
app := newTestApp(t, &h)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
w := loginPageGet(h, "", nil)
|
||||
|
||||
require.Equal(t, http.StatusOK, w.Code)
|
||||
assert.NotContains(t, w.Body.String(), `href="/pages/login"`)
|
||||
}
|
||||
|
||||
// TestLogin_UsernameAtLimitCanLogIn shows that a username of exactly
|
||||
// database.MaxUsernameBytes still fits in the session cookie. Past
|
||||
// what the cookie can carry, a correct login answers 500.
|
||||
|
||||
@@ -299,9 +299,8 @@ func countInFlightDeliveries(
|
||||
return count, err
|
||||
}
|
||||
|
||||
// createReplayDelivery writes the new pending delivery row, adds it to
|
||||
// its target's totals in the same transaction, and returns the task
|
||||
// that carries it to the delivery engine.
|
||||
// createReplayDelivery writes the new pending delivery row and returns
|
||||
// the task that carries it to the delivery engine.
|
||||
//
|
||||
// The row is written with associations omitted, and neither Event nor
|
||||
// Target is populated on it: GORM's SaveBeforeAssociations would
|
||||
@@ -320,16 +319,7 @@ func createReplayDelivery(
|
||||
Status: database.DeliveryStatusPending,
|
||||
}
|
||||
|
||||
err := webhookDB.Transaction(func(tx *gorm.DB) error {
|
||||
err := tx.Omit(clause.Associations).Create(dlv).Error
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return database.AddTargetTotals(tx, database.TargetTotals{
|
||||
TargetID: dlv.TargetID, Deliveries: 1,
|
||||
})
|
||||
})
|
||||
err := webhookDB.Omit(clause.Associations).Create(dlv).Error
|
||||
if err != nil {
|
||||
return delivery.Task{}, err
|
||||
}
|
||||
@@ -372,7 +362,7 @@ func (h *Handlers) finishReplay(
|
||||
webhook database.Webhook,
|
||||
code replayOutcomeCode,
|
||||
) {
|
||||
dest := "/hook/" + webhook.ID + "/events?" +
|
||||
dest := "/source/" + webhook.ID + "/logs?" +
|
||||
replayOutcomeParam + "=" + string(code)
|
||||
|
||||
// The page is read from the form rather than the query string:
|
||||
|
||||
@@ -138,7 +138,7 @@ func postReplay(
|
||||
t.Helper()
|
||||
|
||||
req := postRequest(
|
||||
"/hook/"+webhookID+"/deliveries/"+
|
||||
"/source/"+webhookID+"/deliveries/"+
|
||||
deliveryID+"/replay",
|
||||
authenticatedCookies(
|
||||
t, sess, deleteTestUserID, deleteTestUsername,
|
||||
@@ -212,7 +212,7 @@ func TestHandleDeliveryReplay_AppendsDeliveryAndLeavesOriginal(
|
||||
require.Equal(t, http.StatusSeeOther, w.Code)
|
||||
assert.Equal(
|
||||
t,
|
||||
"/hook/"+wh.ID+"/events?replay=queued",
|
||||
"/source/"+wh.ID+"/logs?replay=queued",
|
||||
w.Header().Get("Location"),
|
||||
)
|
||||
|
||||
@@ -362,7 +362,7 @@ func TestHandleDeliveryReplay_RefusesDeletedTarget(t *testing.T) {
|
||||
require.Equal(t, http.StatusSeeOther, w.Code)
|
||||
assert.Equal(
|
||||
t,
|
||||
"/hook/"+wh.ID+"/events?replay=target-deleted",
|
||||
"/source/"+wh.ID+"/logs?replay=target-deleted",
|
||||
w.Header().Get("Location"),
|
||||
)
|
||||
|
||||
@@ -390,7 +390,7 @@ func TestHandleDeliveryReplay_RefusesDeletedTarget(t *testing.T) {
|
||||
require.Equal(t, http.StatusSeeOther, missing.Code)
|
||||
assert.Equal(
|
||||
t,
|
||||
"/hook/"+wh.ID+"/events?replay=target-missing",
|
||||
"/source/"+wh.ID+"/logs?replay=target-missing",
|
||||
missing.Header().Get("Location"),
|
||||
)
|
||||
}
|
||||
@@ -431,7 +431,7 @@ func TestHandleDeliveryReplay_RefusesWhileEarlierReplayInFlight(
|
||||
require.Equal(t, http.StatusSeeOther, first.Code)
|
||||
require.Equal(
|
||||
t,
|
||||
"/hook/"+wh.ID+"/events?replay=queued",
|
||||
"/source/"+wh.ID+"/logs?replay=queued",
|
||||
first.Header().Get("Location"),
|
||||
)
|
||||
|
||||
@@ -439,7 +439,7 @@ func TestHandleDeliveryReplay_RefusesWhileEarlierReplayInFlight(
|
||||
require.Equal(t, http.StatusSeeOther, second.Code)
|
||||
assert.Equal(
|
||||
t,
|
||||
"/hook/"+wh.ID+"/events?replay=in-flight",
|
||||
"/source/"+wh.ID+"/logs?replay=in-flight",
|
||||
second.Header().Get("Location"),
|
||||
)
|
||||
|
||||
@@ -465,7 +465,7 @@ func TestHandleDeliveryReplay_RefusesWhileEarlierReplayInFlight(
|
||||
require.Equal(t, http.StatusSeeOther, pending.Code)
|
||||
assert.Equal(
|
||||
t,
|
||||
"/hook/"+wh.ID+"/events?replay=not-terminal",
|
||||
"/source/"+wh.ID+"/logs?replay=not-terminal",
|
||||
pending.Header().Get("Location"),
|
||||
)
|
||||
}
|
||||
@@ -501,7 +501,7 @@ func TestHandleSourceLogs_RendersReplayControlAndBanner(t *testing.T) {
|
||||
|
||||
assert.Contains(
|
||||
t, body,
|
||||
`action="/hook/`+wh.ID+`/deliveries/`+
|
||||
`action="/source/`+wh.ID+`/deliveries/`+
|
||||
original.ID+`/replay"`,
|
||||
)
|
||||
assert.Contains(t, body, `method="POST"`)
|
||||
|
||||
@@ -64,8 +64,8 @@ func fetchEventBody(
|
||||
req := httptest.NewRequestWithContext(
|
||||
context.Background(),
|
||||
http.MethodGet,
|
||||
"/hook/"+url.PathEscape(sourceID)+
|
||||
"/events/"+url.PathEscape(eventID)+"/body",
|
||||
"/source/"+url.PathEscape(sourceID)+
|
||||
"/logs/"+url.PathEscape(eventID)+"/body",
|
||||
nil,
|
||||
)
|
||||
|
||||
@@ -490,7 +490,7 @@ func TestHandleSourceLogs_TruncationMarkerLinksToDownload(
|
||||
page := renderSourceLogsPage(t, h, sess, big.ID)
|
||||
assert.Contains(
|
||||
t, page,
|
||||
"/hook/"+big.ID+"/events/"+bigEvt.ID+"/body",
|
||||
"/source/"+big.ID+"/logs/"+bigEvt.ID+"/body",
|
||||
)
|
||||
|
||||
small := seedWebhook(t, db)
|
||||
@@ -501,6 +501,6 @@ func TestHandleSourceLogs_TruncationMarkerLinksToDownload(
|
||||
page = renderSourceLogsPage(t, h, sess, small.ID)
|
||||
assert.NotContains(
|
||||
t, page,
|
||||
"/hook/"+small.ID+"/events/"+smallEvt.ID+"/body",
|
||||
"/source/"+small.ID+"/logs/"+smallEvt.ID+"/body",
|
||||
)
|
||||
}
|
||||
|
||||
@@ -257,7 +257,7 @@ func (h *Handlers) finishResubmit(
|
||||
webhook database.Webhook,
|
||||
code resubmitOutcomeCode,
|
||||
) {
|
||||
dest := "/hook/" + webhook.ID + "/events?" +
|
||||
dest := "/source/" + webhook.ID + "/logs?" +
|
||||
resubmitOutcomeParam + "=" + string(code)
|
||||
|
||||
// The page is read from the form rather than the query string:
|
||||
|
||||
@@ -65,7 +65,7 @@ func postResubmit(
|
||||
t.Helper()
|
||||
|
||||
req := postRequest(
|
||||
"/hook/"+webhookID+"/events/"+eventID+"/resubmit",
|
||||
"/source/"+webhookID+"/events/"+eventID+"/resubmit",
|
||||
authenticatedCookies(
|
||||
t, sess, deleteTestUserID, deleteTestUsername,
|
||||
),
|
||||
@@ -154,7 +154,7 @@ func TestHandleEventResubmit_DeliversToTargetCreatedAfterTheEvent(
|
||||
require.Equal(t, http.StatusSeeOther, w.Code)
|
||||
assert.Equal(
|
||||
t,
|
||||
"/hook/"+wh.ID+"/events?resubmit=queued",
|
||||
"/source/"+wh.ID+"/logs?resubmit=queued",
|
||||
w.Header().Get("Location"),
|
||||
)
|
||||
|
||||
@@ -282,7 +282,7 @@ func TestHandleEventResubmit_IsRepeatable(t *testing.T) {
|
||||
require.Equal(t, http.StatusSeeOther, w.Code)
|
||||
assert.Equal(
|
||||
t,
|
||||
"/hook/"+wh.ID+"/events?resubmit=queued",
|
||||
"/source/"+wh.ID+"/logs?resubmit=queued",
|
||||
w.Header().Get("Location"),
|
||||
"a resubmit must not be refused while an earlier "+
|
||||
"one is in flight",
|
||||
@@ -436,7 +436,7 @@ func TestHandleEventResubmit_SkipsInactiveTarget(t *testing.T) {
|
||||
require.Equal(t, http.StatusSeeOther, w.Code)
|
||||
assert.Equal(
|
||||
t,
|
||||
"/hook/"+wh.ID+"/events?resubmit=queued",
|
||||
"/source/"+wh.ID+"/logs?resubmit=queued",
|
||||
w.Header().Get("Location"),
|
||||
"an inactive target is skipped, not an error",
|
||||
)
|
||||
@@ -482,7 +482,7 @@ func TestHandleEventResubmit_NoActiveTargetsStillStoresEvent(
|
||||
require.Equal(t, http.StatusSeeOther, w.Code)
|
||||
assert.Equal(
|
||||
t,
|
||||
"/hook/"+wh.ID+"/events?resubmit=no-targets",
|
||||
"/source/"+wh.ID+"/logs?resubmit=no-targets",
|
||||
w.Header().Get("Location"),
|
||||
)
|
||||
|
||||
@@ -598,7 +598,7 @@ func TestHandleSourceLogs_ShowsResubmitProvenance(t *testing.T) {
|
||||
)
|
||||
assert.Contains(
|
||||
t, body,
|
||||
"/hook/"+wh.ID+"/events/"+original.ID+"/resubmit",
|
||||
"/source/"+wh.ID+"/events/"+original.ID+"/resubmit",
|
||||
"the log must offer the resubmit action per event",
|
||||
)
|
||||
}
|
||||
|
||||
@@ -4,9 +4,7 @@ import (
|
||||
"html/template"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"time"
|
||||
|
||||
"gorm.io/gorm"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
)
|
||||
|
||||
@@ -71,29 +69,6 @@ func (s *Handlers) LoadEventLogViewsForTest(
|
||||
return views
|
||||
}
|
||||
|
||||
// WebhookStatsForTest returns the figures the statistics pane on a
|
||||
// webhook's page shows, from the webhook's entrypoints and targets
|
||||
// loaded as that page loads them.
|
||||
func (s *Handlers) WebhookStatsForTest(webhookID string) *WebhookStats {
|
||||
var entrypoints []database.Entrypoint
|
||||
|
||||
s.db.DB().Where("webhook_id = ?", webhookID).Find(&entrypoints)
|
||||
|
||||
var targets []database.Target
|
||||
|
||||
s.db.DB().Where("webhook_id = ?", webhookID).Find(&targets)
|
||||
|
||||
return s.loadWebhookStats(webhookID, entrypoints, targets)
|
||||
}
|
||||
|
||||
// FinishedByTargetForTest exposes finishedByTarget for use in the
|
||||
// handlers_test package.
|
||||
func FinishedByTargetForTest(
|
||||
webhookDB *gorm.DB, since time.Time,
|
||||
) ([]TargetFinished, error) {
|
||||
return finishedByTarget(webhookDB, since)
|
||||
}
|
||||
|
||||
// AddTemplateForTest registers a template under a page name so that
|
||||
// the handlers_test package can drive the render path with a
|
||||
// template of its own.
|
||||
|
||||
@@ -306,7 +306,7 @@ func postWebhook(
|
||||
t.Helper()
|
||||
|
||||
req := httptest.NewRequestWithContext(
|
||||
context.Background(), http.MethodPost, "/h/x",
|
||||
context.Background(), http.MethodPost, "/webhook/x",
|
||||
strings.NewReader("{}"),
|
||||
)
|
||||
|
||||
|
||||
@@ -12,6 +12,7 @@ import (
|
||||
"net/http"
|
||||
"sync/atomic"
|
||||
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"go.uber.org/fx"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
@@ -36,9 +37,6 @@ const (
|
||||
tmplKeyError = "Error"
|
||||
// tmplKeyWebhook is the template data key for a webhook.
|
||||
tmplKeyWebhook = "Webhook"
|
||||
// tmplKeyNext is the template data key for the page to return
|
||||
// to after login.
|
||||
tmplKeyNext = "Next"
|
||||
)
|
||||
|
||||
// errInvalidPassword is returned when a password does not match.
|
||||
@@ -64,6 +62,8 @@ type HandlersParams struct {
|
||||
Notifier delivery.Notifier
|
||||
Evictor delivery.WebhookEvictor
|
||||
SSRFGuard *delivery.Guard
|
||||
Metrics *metrics.Set
|
||||
Registry *prometheus.Registry
|
||||
}
|
||||
|
||||
// Handlers provides HTTP handler methods for all application
|
||||
@@ -94,22 +94,18 @@ type Handlers struct {
|
||||
|
||||
// parsePageTemplate parses a page-specific template set from the
|
||||
// embedded FS. Each page template is combined with the shared
|
||||
// base, htmlheader, and navbar templates, and with any further files
|
||||
// the page includes. The page file must be listed first so that its
|
||||
// root action ({{template "base" .}}) becomes the template set's entry
|
||||
// point.
|
||||
func parsePageTemplate(
|
||||
pageFile string, included ...string,
|
||||
) *template.Template {
|
||||
files := append([]string{
|
||||
pageFile,
|
||||
"base.html",
|
||||
"htmlheader.html",
|
||||
"navbar.html",
|
||||
}, included...)
|
||||
|
||||
// base, htmlheader, and navbar templates. The page file must be
|
||||
// listed first so that its root action ({{template "base" .}})
|
||||
// becomes the template set's entry point.
|
||||
func parsePageTemplate(pageFile string) *template.Template {
|
||||
return template.Must(
|
||||
template.ParseFS(templates.Templates, files...),
|
||||
template.ParseFS(
|
||||
templates.Templates,
|
||||
pageFile,
|
||||
"base.html",
|
||||
"htmlheader.html",
|
||||
"navbar.html",
|
||||
),
|
||||
)
|
||||
}
|
||||
|
||||
@@ -129,7 +125,7 @@ func New(
|
||||
s.mw = params.Middleware
|
||||
s.notifier = params.Notifier
|
||||
s.evictor = params.Evictor
|
||||
s.mtr = metrics.Default()
|
||||
s.mtr = params.Metrics
|
||||
s.ssrf = params.SSRFGuard
|
||||
|
||||
// Parse all page templates once at startup
|
||||
@@ -138,7 +134,7 @@ func New(
|
||||
"profile.html": parsePageTemplate("profile.html"),
|
||||
"sources_list.html": parsePageTemplate("sources_list.html"),
|
||||
"sources_new.html": parsePageTemplate("sources_new.html"),
|
||||
"source_detail.html": parsePageTemplate("source_detail.html", "webhook_stats.html"),
|
||||
"source_detail.html": parsePageTemplate("source_detail.html"),
|
||||
"source_edit.html": parsePageTemplate("source_edit.html"),
|
||||
"source_logs.html": parsePageTemplate("source_logs.html"),
|
||||
"target_edit.html": parsePageTemplate("target_edit.html"),
|
||||
|
||||
@@ -20,6 +20,7 @@ import (
|
||||
"sneak.berlin/go/webhooker/internal/handlers"
|
||||
"sneak.berlin/go/webhooker/internal/healthcheck"
|
||||
"sneak.berlin/go/webhooker/internal/logger"
|
||||
"sneak.berlin/go/webhooker/internal/metrics"
|
||||
"sneak.berlin/go/webhooker/internal/middleware"
|
||||
"sneak.berlin/go/webhooker/internal/session"
|
||||
)
|
||||
@@ -109,6 +110,8 @@ func newTestApp(
|
||||
func(r *recordingEvictor) delivery.WebhookEvictor {
|
||||
return r
|
||||
},
|
||||
metrics.NewRegistry,
|
||||
metrics.New,
|
||||
middleware.New,
|
||||
delivery.NewGuard,
|
||||
handlers.New,
|
||||
@@ -176,7 +179,7 @@ func TestHandleIndex_Authenticated(t *testing.T) {
|
||||
|
||||
assert.Equal(t, http.StatusSeeOther, w2.Code)
|
||||
assert.Equal(
|
||||
t, "/hooks", w2.Header().Get("Location"),
|
||||
t, "/sources", w2.Header().Get("Location"),
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
@@ -5,13 +5,13 @@ import (
|
||||
)
|
||||
|
||||
// HandleIndex returns a handler for the root path that redirects
|
||||
// based on authentication state: authenticated users go to /hooks
|
||||
// based on authentication state: authenticated users go to /sources
|
||||
// (the dashboard), unauthenticated users go to the login page.
|
||||
func (s *Handlers) HandleIndex() http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
sess, err := s.session.Get(r)
|
||||
if err == nil && s.session.IsAuthenticated(sess) {
|
||||
http.Redirect(w, r, "/hooks", http.StatusSeeOther)
|
||||
http.Redirect(w, r, "/sources", http.StatusSeeOther)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
@@ -4,7 +4,7 @@ package handlers_test
|
||||
// this package reach a value an UNAUTHENTICATED client picks outright
|
||||
// and of a length it picks outright:
|
||||
//
|
||||
// - the unknown-entrypoint DEBUG line on /h/{uuid}, whose
|
||||
// - the unknown-entrypoint DEBUG line on /webhook/{uuid}, whose
|
||||
// path segment matched no stored entrypoint and so is bounded by
|
||||
// nothing;
|
||||
// - the failed-login DEBUG lines, whose username is a form field.
|
||||
@@ -190,12 +190,12 @@ func assertNoClientText(t *testing.T, buf *bytes.Buffer) {
|
||||
// route pattern.
|
||||
func receiverRouter(h *handlers.Handlers) *chi.Mux {
|
||||
router := chi.NewRouter()
|
||||
router.Post("/h/{uuid}", h.HandleWebhook())
|
||||
router.Post("/webhook/{uuid}", h.HandleWebhook())
|
||||
|
||||
return router
|
||||
}
|
||||
|
||||
// postReceiver sends one POST at /h/<segment>.
|
||||
// postReceiver sends one POST at /webhook/<segment>.
|
||||
//
|
||||
// RawPath is cleared after parsing so chi routes on the decoded path
|
||||
// and the handler sees the raw bytes rather than their percent-escaped
|
||||
@@ -210,7 +210,7 @@ func postReceiver(
|
||||
req := httptest.NewRequestWithContext(
|
||||
context.Background(),
|
||||
http.MethodPost,
|
||||
"/h/"+url.PathEscape(segment),
|
||||
"/webhook/"+url.PathEscape(segment),
|
||||
strings.NewReader(""),
|
||||
)
|
||||
req.URL.RawPath = ""
|
||||
@@ -507,7 +507,7 @@ func TestVerificationCapacity_LogLineDoesNotTrackPathSize(
|
||||
http.StatusServiceUnavailable,
|
||||
postLoginAtPath(
|
||||
t, h,
|
||||
"/hook/"+url.PathEscape(
|
||||
"/source/"+url.PathEscape(
|
||||
oversizedFill(fill),
|
||||
)+"/login",
|
||||
),
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
|
||||
"github.com/prometheus/client_golang/prometheus/promhttp"
|
||||
)
|
||||
|
||||
// HandleMetrics returns the Prometheus scrape handler for the
|
||||
// registry built by metrics.NewRegistry, which the HTTP, delivery, Go
|
||||
// runtime and process collectors register on. It is what
|
||||
// promhttp.Handler builds for the global default registry, including
|
||||
// the promhttp_metric_handler_* series that count scrapes, pointed at
|
||||
// that registry instead.
|
||||
func (s *Handlers) HandleMetrics() http.HandlerFunc {
|
||||
reg := s.params.Registry
|
||||
|
||||
return promhttp.InstrumentMetricHandler(
|
||||
reg, promhttp.HandlerFor(reg, promhttp.HandlerOpts{}),
|
||||
).ServeHTTP
|
||||
}
|
||||
@@ -160,10 +160,7 @@ func TestUserRoute_Unauthenticated_RedirectedByMiddleware(t *testing.T) {
|
||||
"handler must not be reached for unauthenticated request",
|
||||
)
|
||||
assert.Equal(t, http.StatusSeeOther, w.Code)
|
||||
assert.Equal(
|
||||
t, "/pages/login?next=%2Fuser%2Ftestuser",
|
||||
w.Header().Get("Location"),
|
||||
)
|
||||
assert.Equal(t, "/pages/login", w.Header().Get("Location"))
|
||||
}
|
||||
|
||||
// passwordChangeRequest builds a POST request to the password-change
|
||||
|
||||
@@ -313,7 +313,7 @@ func TestHandleWebhook_RecordsBodySize(t *testing.T) {
|
||||
body := strings.Repeat("é", 1024)
|
||||
|
||||
req := httptest.NewRequestWithContext(
|
||||
context.Background(), http.MethodPost, "/h/x",
|
||||
context.Background(), http.MethodPost, "/webhook/x",
|
||||
strings.NewReader(body),
|
||||
)
|
||||
|
||||
|
||||
@@ -220,7 +220,7 @@ func TestHandleSourceDelete_EvictsArchiveWriter(t *testing.T) {
|
||||
)
|
||||
|
||||
req := postRequest(
|
||||
"/hook/"+wh.ID+"/delete",
|
||||
"/source/"+wh.ID+"/delete",
|
||||
cookies,
|
||||
map[string]string{paramSourceID: wh.ID},
|
||||
)
|
||||
@@ -267,7 +267,7 @@ func TestHandleSourceDelete_KeepsArchiveFile(t *testing.T) {
|
||||
)
|
||||
|
||||
req := postRequest(
|
||||
"/hook/"+wh.ID+"/delete",
|
||||
"/source/"+wh.ID+"/delete",
|
||||
cookies,
|
||||
map[string]string{paramSourceID: wh.ID},
|
||||
)
|
||||
@@ -323,7 +323,7 @@ func TestHandleSourceDelete_FailedDeleteKeepsEverything(
|
||||
)
|
||||
|
||||
req := postRequest(
|
||||
"/hook/"+wh.ID+"/delete",
|
||||
"/source/"+wh.ID+"/delete",
|
||||
cookies,
|
||||
map[string]string{paramSourceID: wh.ID},
|
||||
)
|
||||
@@ -337,7 +337,7 @@ func TestHandleSourceDelete_FailedDeleteKeepsEverything(
|
||||
)
|
||||
assert.Empty(
|
||||
t, w.Header().Get("Location"),
|
||||
"a failed deletion must not redirect to /hooks",
|
||||
"a failed deletion must not redirect to /sources",
|
||||
)
|
||||
|
||||
assert.Equal(
|
||||
@@ -402,7 +402,7 @@ func TestHandleSourceDelete_RemovesConfigAndEventDatabase(
|
||||
)
|
||||
|
||||
req := postRequest(
|
||||
"/hook/"+wh.ID+"/delete",
|
||||
"/source/"+wh.ID+"/delete",
|
||||
cookies,
|
||||
map[string]string{paramSourceID: wh.ID},
|
||||
)
|
||||
@@ -411,7 +411,7 @@ func TestHandleSourceDelete_RemovesConfigAndEventDatabase(
|
||||
h.HandleSourceDelete().ServeHTTP(w, req)
|
||||
|
||||
require.Equal(t, http.StatusSeeOther, w.Code)
|
||||
assert.Equal(t, "/hooks", w.Header().Get("Location"))
|
||||
assert.Equal(t, "/sources", w.Header().Get("Location"))
|
||||
|
||||
assert.Equal(
|
||||
t, int64(0),
|
||||
@@ -465,7 +465,7 @@ func TestHandleTargetDelete_EvictsWhenLastDatabaseTargetGone(
|
||||
)
|
||||
|
||||
req := postRequest(
|
||||
"/hook/"+wh.ID+"/targets/"+tgt.ID+"/delete",
|
||||
"/source/"+wh.ID+"/targets/"+tgt.ID+"/delete",
|
||||
cookies,
|
||||
map[string]string{
|
||||
paramSourceID: wh.ID,
|
||||
@@ -515,7 +515,7 @@ func TestHandleTargetDelete_KeepsWriterWhenDatabaseTargetRemains(
|
||||
)
|
||||
|
||||
req := postRequest(
|
||||
"/hook/"+wh.ID+"/targets/"+doomed.ID+"/delete",
|
||||
"/source/"+wh.ID+"/targets/"+doomed.ID+"/delete",
|
||||
cookies,
|
||||
map[string]string{
|
||||
paramSourceID: wh.ID,
|
||||
@@ -563,7 +563,7 @@ func TestHandleTargetDelete_KeepsWriterWhenOtherTypeDeleted(
|
||||
)
|
||||
|
||||
req := postRequest(
|
||||
"/hook/"+wh.ID+"/targets/"+other.ID+"/delete",
|
||||
"/source/"+wh.ID+"/targets/"+other.ID+"/delete",
|
||||
cookies,
|
||||
map[string]string{
|
||||
paramSourceID: wh.ID,
|
||||
|
||||
@@ -81,7 +81,7 @@ func (f *baseURLFixture) entrypointURL(
|
||||
req := httptest.NewRequestWithContext(
|
||||
context.Background(),
|
||||
http.MethodGet,
|
||||
"/hook/"+f.webhook,
|
||||
"/source/"+f.webhook,
|
||||
nil,
|
||||
)
|
||||
req.Host = host
|
||||
@@ -213,7 +213,7 @@ func TestSourceDetailBaseURL_ForwardedProtoSpellings(t *testing.T) {
|
||||
|
||||
assert.Equal(
|
||||
t,
|
||||
tc.scheme+"://"+host+"/h/"+fixture.path,
|
||||
tc.scheme+"://"+host+"/webhook/"+fixture.path,
|
||||
fixture.entrypointURL(
|
||||
t, host, forwardedProto(tc.header),
|
||||
),
|
||||
@@ -244,7 +244,7 @@ func TestSourceDetailBaseURL_DirectTLSBeatsPlaintextHeader(
|
||||
|
||||
assert.Equal(
|
||||
t,
|
||||
"https://"+host+"/h/"+fixture.path,
|
||||
"https://"+host+"/webhook/"+fixture.path,
|
||||
got,
|
||||
"a connection this process terminated with TLS "+
|
||||
"outranks a header claiming plaintext",
|
||||
@@ -272,7 +272,7 @@ func TestSourceDetailBaseURL_KeepsHostAuthority(t *testing.T) {
|
||||
|
||||
assert.Equal(
|
||||
t,
|
||||
"https://"+host+"/h/"+fixture.path,
|
||||
"https://"+host+"/webhook/"+fixture.path,
|
||||
fixture.entrypointURL(
|
||||
t, host, forwardedProto("HTTPS"),
|
||||
),
|
||||
|
||||
@@ -82,7 +82,7 @@ func serveSourceDetailPage(
|
||||
req := httptest.NewRequestWithContext(
|
||||
context.Background(),
|
||||
http.MethodGet,
|
||||
"/hook/"+webhookID,
|
||||
"/source/"+webhookID,
|
||||
nil,
|
||||
)
|
||||
|
||||
|
||||
@@ -1,317 +0,0 @@
|
||||
package handlers_test
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/clause"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/handlers"
|
||||
"sneak.berlin/go/webhooker/internal/logger"
|
||||
"sneak.berlin/go/webhooker/internal/session"
|
||||
)
|
||||
|
||||
// failedHighlight is how the list marks a number of failed deliveries
|
||||
// that is not zero.
|
||||
const failedHighlight = `class="font-medium text-red-600"`
|
||||
|
||||
// listWebhook adds a webhook with the given name, owned by the test
|
||||
// user.
|
||||
func listWebhook(
|
||||
t *testing.T, db *database.Database, name string,
|
||||
) *database.Webhook {
|
||||
t.Helper()
|
||||
|
||||
wh := &database.Webhook{UserID: deleteTestUserID, Name: name}
|
||||
require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
|
||||
|
||||
return wh
|
||||
}
|
||||
|
||||
// renderWebhookList runs the real webhook list handler as the test user
|
||||
// and returns the rendered page.
|
||||
func renderWebhookList(
|
||||
t *testing.T, h *handlers.Handlers, sess *session.Session,
|
||||
) string {
|
||||
t.Helper()
|
||||
|
||||
cookies := authenticatedCookies(
|
||||
t, sess, deleteTestUserID, deleteTestUsername,
|
||||
)
|
||||
|
||||
w := httptest.NewRecorder()
|
||||
h.HandleSourceList().ServeHTTP(
|
||||
w, getRequest(t, "/hooks", cookies, nil),
|
||||
)
|
||||
require.Equal(t, http.StatusOK, w.Code)
|
||||
|
||||
return w.Body.String()
|
||||
}
|
||||
|
||||
// listCard returns one webhook's entry in a rendered webhook list, its
|
||||
// markup as rendered and its text with the markup taken out and each
|
||||
// run of space made one space.
|
||||
func listCard(t *testing.T, page, webhookID string) (string, string) {
|
||||
t.Helper()
|
||||
|
||||
_, card, found := strings.Cut(page, `href="/hook/`+webhookID+`"`)
|
||||
require.True(t, found, "the list has no entry for %s", webhookID)
|
||||
|
||||
card, _, _ = strings.Cut(card, "</a>")
|
||||
text := regexp.MustCompile(`<[^>]*>`).ReplaceAllString(card, " ")
|
||||
|
||||
return card, strings.Join(strings.Fields(text), " ")
|
||||
}
|
||||
|
||||
// receiveEvents posts the given number of events to an entrypoint
|
||||
// through the real receiver, and returns the webhook's event database
|
||||
// and its events, oldest first.
|
||||
func receiveEvents(
|
||||
t *testing.T,
|
||||
h *handlers.Handlers,
|
||||
dbMgr *database.WebhookDBManager,
|
||||
webhookID, path string,
|
||||
count int,
|
||||
) (*gorm.DB, []database.Event) {
|
||||
t.Helper()
|
||||
|
||||
router := receiverRouter(h)
|
||||
|
||||
for range count {
|
||||
require.Equal(t, http.StatusOK, postReceiver(t, router, path))
|
||||
}
|
||||
|
||||
webhookDB, err := dbMgr.GetDB(webhookID)
|
||||
require.NoError(t, err)
|
||||
|
||||
events := listEvents(t, webhookDB)
|
||||
require.Len(t, events, count)
|
||||
|
||||
return webhookDB, events
|
||||
}
|
||||
|
||||
// seedFailingWebhook adds a webhook with two entrypoints, one inactive,
|
||||
// and four targets, one inactive. Three events each reach the three
|
||||
// active targets. Two deliveries failed in the last 24 hours, one 30
|
||||
// hours ago, and one was delivered. It returns the webhook and its
|
||||
// newest event.
|
||||
func seedFailingWebhook(
|
||||
t *testing.T,
|
||||
h *handlers.Handlers,
|
||||
db *database.Database,
|
||||
dbMgr *database.WebhookDBManager,
|
||||
) (*database.Webhook, database.Event) {
|
||||
t.Helper()
|
||||
|
||||
wh := listWebhook(t, db, "failing")
|
||||
path := statsEntrypoint(t, db, wh.ID, true)
|
||||
|
||||
statsEntrypoint(t, db, wh.ID, false)
|
||||
|
||||
first := seedTarget(t, db, wh.ID, database.TargetTypeLog)
|
||||
second := seedTarget(t, db, wh.ID, database.TargetTypeLog)
|
||||
|
||||
seedTarget(t, db, wh.ID, database.TargetTypeLog)
|
||||
|
||||
inactive := seedTarget(t, db, wh.ID, database.TargetTypeLog)
|
||||
require.NoError(t, db.DB().Model(inactive).
|
||||
Update("active", false).Error)
|
||||
|
||||
webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, path, 3)
|
||||
now := time.Now()
|
||||
|
||||
statsFinish(t, webhookDB,
|
||||
statsDelivery(t, webhookDB, events[0].ID, first.ID),
|
||||
database.DeliveryStatusFailed, now.Add(-30*time.Hour))
|
||||
statsFinish(t, webhookDB,
|
||||
statsDelivery(t, webhookDB, events[1].ID, first.ID),
|
||||
database.DeliveryStatusFailed, now.Add(-time.Hour))
|
||||
statsFinish(t, webhookDB,
|
||||
statsDelivery(t, webhookDB, events[2].ID, first.ID),
|
||||
database.DeliveryStatusFailed, now.Add(-time.Minute))
|
||||
statsFinish(t, webhookDB,
|
||||
statsDelivery(t, webhookDB, events[2].ID, second.ID),
|
||||
database.DeliveryStatusDelivered, now.Add(-time.Minute))
|
||||
|
||||
return wh, events[2]
|
||||
}
|
||||
|
||||
// seedHealthyWebhook adds a webhook with one entrypoint and one target,
|
||||
// both active, and two events, both delivered. It returns the webhook
|
||||
// and its newest event.
|
||||
func seedHealthyWebhook(
|
||||
t *testing.T,
|
||||
h *handlers.Handlers,
|
||||
db *database.Database,
|
||||
dbMgr *database.WebhookDBManager,
|
||||
) (*database.Webhook, database.Event) {
|
||||
t.Helper()
|
||||
|
||||
wh := listWebhook(t, db, "healthy")
|
||||
path := statsEntrypoint(t, db, wh.ID, true)
|
||||
target := seedTarget(t, db, wh.ID, database.TargetTypeLog)
|
||||
|
||||
webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, path, 2)
|
||||
|
||||
for _, ev := range events {
|
||||
statsFinish(t, webhookDB,
|
||||
statsDelivery(t, webhookDB, ev.ID, target.ID),
|
||||
database.DeliveryStatusDelivered, time.Now())
|
||||
}
|
||||
|
||||
return wh, events[1]
|
||||
}
|
||||
|
||||
// lastEventText is how the list shows the arrival of an event.
|
||||
func lastEventText(ev database.Event) string {
|
||||
return ev.CreatedAt.UTC().Format("2006-01-02 15:04:05 UTC")
|
||||
}
|
||||
|
||||
// TestSourceList_ShowsActivityOfEachWebhook checks the figures the list
|
||||
// shows for a webhook with recent failures, a healthy one, a new one
|
||||
// that has received no event, and one without an event database.
|
||||
func TestSourceList_ShowsActivityOfEachWebhook(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
failing, failingNewest := seedFailingWebhook(t, h, db, dbMgr)
|
||||
healthy, healthyNewest := seedHealthyWebhook(t, h, db, dbMgr)
|
||||
|
||||
// Creating a webhook creates its event database.
|
||||
fresh := listWebhook(t, db, "fresh")
|
||||
require.NoError(t, dbMgr.CreateDB(fresh.ID))
|
||||
|
||||
quiet := listWebhook(t, db, "quiet")
|
||||
|
||||
page := renderWebhookList(t, h, sess)
|
||||
|
||||
card, text := listCard(t, page, failing.ID)
|
||||
assert.Contains(t, text, "2 entrypoints, 1 inactive "+
|
||||
"4 targets, 1 inactive "+
|
||||
"3 events within retention "+
|
||||
"Last event "+lastEventText(failingNewest)+" "+
|
||||
"2 failed deliveries in the last 24 hours")
|
||||
assert.Contains(t, card,
|
||||
failedHighlight+">2 failed deliveries in the last 24 hours<")
|
||||
|
||||
card, text = listCard(t, page, healthy.ID)
|
||||
assert.Contains(t, text, "1 entrypoint "+
|
||||
"1 target "+
|
||||
"2 events within retention "+
|
||||
"Last event "+lastEventText(healthyNewest)+" "+
|
||||
"0 failed deliveries in the last 24 hours")
|
||||
assert.NotContains(t, text, "inactive")
|
||||
assert.NotContains(t, card, failedHighlight)
|
||||
|
||||
card, text = listCard(t, page, fresh.ID)
|
||||
assert.Contains(t, text, "0 entrypoints "+
|
||||
"0 targets "+
|
||||
"0 events within retention "+
|
||||
"No events yet "+
|
||||
"0 failed deliveries in the last 24 hours")
|
||||
assert.NotContains(t, card, failedHighlight)
|
||||
|
||||
card, text = listCard(t, page, quiet.ID)
|
||||
assert.Contains(t, text, "0 entrypoints "+
|
||||
"0 targets "+
|
||||
"0 events within retention "+
|
||||
"No events yet "+
|
||||
"0 failed deliveries in the last 24 hours")
|
||||
assert.NotContains(t, card, failedHighlight)
|
||||
assert.False(t, dbMgr.DBExists(quiet.ID),
|
||||
"showing the list must not create an event database")
|
||||
}
|
||||
|
||||
// TestSourceList_CountsOnlyEventsWithinRetention checks that once
|
||||
// retention has removed one of a webhook's three events, the list
|
||||
// counts the two still stored.
|
||||
func TestSourceList_CountsOnlyEventsWithinRetention(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
log *logger.Logger
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr, &log)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
wh := &database.Webhook{
|
||||
UserID: deleteTestUserID, Name: "pruned", RetentionDays: 14,
|
||||
}
|
||||
require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
|
||||
|
||||
path := statsEntrypoint(t, db, wh.ID, true)
|
||||
webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, path, 3)
|
||||
|
||||
statsAge(t, webhookDB, events[0].ID, time.Now().Add(-15*24*time.Hour))
|
||||
statsPrune(t, db, dbMgr, log, webhookDB)
|
||||
require.Len(t, listEvents(t, webhookDB), 2)
|
||||
|
||||
_, text := listCard(t, renderWebhookList(t, h, sess), wh.ID)
|
||||
assert.Contains(t, text,
|
||||
"1 entrypoint 0 targets 2 events within retention")
|
||||
}
|
||||
|
||||
// TestSourceList_UnreadableEventDatabase checks that a webhook whose
|
||||
// event database cannot be read says so in its entry instead of
|
||||
// showing zeros, and that the rest of the list is still shown.
|
||||
func TestSourceList_UnreadableEventDatabase(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
broken := listWebhook(t, db, "broken")
|
||||
statsEntrypoint(t, db, broken.ID, true)
|
||||
|
||||
brokenDB, err := dbMgr.GetDB(broken.ID)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t,
|
||||
brokenDB.Migrator().DropTable(&database.EventTotals{}))
|
||||
|
||||
quiet := listWebhook(t, db, "quiet")
|
||||
|
||||
page := renderWebhookList(t, h, sess)
|
||||
|
||||
_, text := listCard(t, page, broken.ID)
|
||||
assert.Contains(t, text,
|
||||
"1 entrypoint 0 targets The event figures could not be read.")
|
||||
assert.NotContains(t, text, "events")
|
||||
assert.NotContains(t, text, "failed")
|
||||
|
||||
_, text = listCard(t, page, quiet.ID)
|
||||
assert.Contains(t, text, "No events yet")
|
||||
}
|
||||
@@ -28,7 +28,7 @@ func deleteTargetThroughHandler(
|
||||
t.Helper()
|
||||
|
||||
req := postRequest(
|
||||
"/hook/"+webhookID+"/targets/"+targetID+"/delete",
|
||||
"/source/"+webhookID+"/targets/"+targetID+"/delete",
|
||||
authenticatedCookies(
|
||||
t, sess, deleteTestUserID, deleteTestUsername,
|
||||
),
|
||||
|
||||
@@ -84,7 +84,7 @@ func renderSourceLogsPageWithQuery(
|
||||
req := httptest.NewRequestWithContext(
|
||||
context.Background(),
|
||||
http.MethodGet,
|
||||
"/hook/"+webhookID+"/events"+query,
|
||||
"/source/"+webhookID+"/logs"+query,
|
||||
nil,
|
||||
)
|
||||
|
||||
|
||||
@@ -3,12 +3,10 @@ package handlers
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/go-chi/chi"
|
||||
"github.com/google/uuid"
|
||||
@@ -22,20 +20,9 @@ import (
|
||||
type WebhookListItem struct {
|
||||
database.Webhook
|
||||
|
||||
EntrypointCount int
|
||||
InactiveEntrypointCount int
|
||||
TargetCount int
|
||||
InactiveTargetCount int
|
||||
|
||||
// EventCount is how many events the webhook holds, LastEventAt
|
||||
// when the newest arrived (nil before the first), and
|
||||
// FailedLast24Hours how many of its deliveries failed in the last
|
||||
// 24 hours. When the webhook's event database could not be read,
|
||||
// EventsUnreadable is set and these three are not known.
|
||||
EventCount int64
|
||||
LastEventAt *time.Time
|
||||
FailedLast24Hours int64
|
||||
EventsUnreadable bool
|
||||
EntrypointCount int64
|
||||
TargetCount int64
|
||||
EventCount int64
|
||||
}
|
||||
|
||||
// errMissingURL signals that a required URL was not provided.
|
||||
@@ -173,18 +160,7 @@ func (h *Handlers) HandleSourceList() http.HandlerFunc {
|
||||
return
|
||||
}
|
||||
|
||||
items, err := h.buildWebhookListItems(webhooks)
|
||||
if err != nil {
|
||||
h.log.Error(
|
||||
"failed to list webhooks", "error", err,
|
||||
)
|
||||
http.Error(
|
||||
w, "Internal server error",
|
||||
http.StatusInternalServerError,
|
||||
)
|
||||
|
||||
return
|
||||
}
|
||||
items := h.buildWebhookListItems(webhooks)
|
||||
|
||||
data := map[string]any{
|
||||
"Webhooks": items,
|
||||
@@ -194,115 +170,36 @@ func (h *Handlers) HandleSourceList() http.HandlerFunc {
|
||||
}
|
||||
}
|
||||
|
||||
// buildWebhookListItems builds the list's entry for each webhook. It
|
||||
// fails when the main database cannot be read. A webhook whose event
|
||||
// database cannot be read is marked on its own entry, and the error is
|
||||
// logged.
|
||||
// buildWebhookListItems builds list items with counts.
|
||||
func (h *Handlers) buildWebhookListItems(
|
||||
webhooks []database.Webhook,
|
||||
) ([]WebhookListItem, error) {
|
||||
) []WebhookListItem {
|
||||
items := make([]WebhookListItem, len(webhooks))
|
||||
since := time.Now().Add(-longWindow)
|
||||
|
||||
for i := range webhooks {
|
||||
item := &items[i]
|
||||
item.Webhook = webhooks[i]
|
||||
items[i].Webhook = webhooks[i]
|
||||
|
||||
var err error
|
||||
h.db.DB().Model(&database.Entrypoint{}).Where(
|
||||
"webhook_id = ?", webhooks[i].ID,
|
||||
).Count(&items[i].EntrypointCount)
|
||||
|
||||
item.EntrypointCount, item.InactiveEntrypointCount, err =
|
||||
h.countWithInactive(&database.Entrypoint{}, item.ID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
h.db.DB().Model(&database.Target{}).Where(
|
||||
"webhook_id = ?", webhooks[i].ID,
|
||||
).Count(&items[i].TargetCount)
|
||||
|
||||
item.TargetCount, item.InactiveTargetCount, err =
|
||||
h.countWithInactive(&database.Target{}, item.ID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Opening an event database that does not exist would create
|
||||
// it, and it would hold nothing to count.
|
||||
if !h.dbMgr.DBExists(item.ID) {
|
||||
continue
|
||||
}
|
||||
|
||||
err = h.readListEventFigures(item, since)
|
||||
if err != nil {
|
||||
h.log.Error(
|
||||
"failed to read webhook list figures",
|
||||
"webhook_id", item.ID,
|
||||
"error", err,
|
||||
if h.dbMgr.DBExists(webhooks[i].ID) {
|
||||
webhookDB, err := h.dbMgr.GetDB(
|
||||
webhooks[i].ID,
|
||||
)
|
||||
|
||||
item.EventsUnreadable = true
|
||||
if err == nil {
|
||||
webhookDB.Model(
|
||||
&database.Event{},
|
||||
).Count(&items[i].EventCount)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return items, nil
|
||||
}
|
||||
|
||||
// countWithInactive returns how many entrypoints or targets, as model
|
||||
// says, a webhook has, and how many of them are inactive.
|
||||
func (h *Handlers) countWithInactive(
|
||||
model any, webhookID string,
|
||||
) (int, int, error) {
|
||||
var active []bool
|
||||
|
||||
err := h.db.DB().Model(model).
|
||||
Where("webhook_id = ?", webhookID).
|
||||
Pluck("active", &active).Error
|
||||
if err != nil {
|
||||
return 0, 0, fmt.Errorf(
|
||||
"reading active flags of webhook %s: %w", webhookID, err,
|
||||
)
|
||||
}
|
||||
|
||||
inactive := 0
|
||||
|
||||
for _, a := range active {
|
||||
if !a {
|
||||
inactive++
|
||||
}
|
||||
}
|
||||
|
||||
return len(active), inactive, nil
|
||||
}
|
||||
|
||||
// readListEventFigures fills in the figures the list shows from the
|
||||
// webhook's event database, with the statistics pane's own queries:
|
||||
// the event count and last arrival from the event totals row, and the
|
||||
// deliveries that failed since the given time from the deliveries'
|
||||
// status index.
|
||||
func (h *Handlers) readListEventFigures(
|
||||
item *WebhookListItem, since time.Time,
|
||||
) error {
|
||||
webhookDB, err := h.dbMgr.GetDB(item.ID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var totals database.EventTotals
|
||||
|
||||
err = webhookDB.Take(&totals).Error
|
||||
if err != nil {
|
||||
return fmt.Errorf("reading event totals: %w", err)
|
||||
}
|
||||
|
||||
item.EventCount = totals.Events - totals.EventsRemoved
|
||||
item.LastEventAt = totals.LastEventAt
|
||||
|
||||
byTarget, err := finishedByTarget(webhookDB, since)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for _, f := range byTarget {
|
||||
item.FailedLast24Hours += f.Failed
|
||||
}
|
||||
|
||||
return nil
|
||||
return items
|
||||
}
|
||||
|
||||
// HandleSourceCreate shows the form to create a new webhook.
|
||||
@@ -433,7 +330,7 @@ func (h *Handlers) createWebhookWithEntrypoint(
|
||||
)
|
||||
|
||||
http.Redirect(
|
||||
w, r, "/hook/"+webhook.ID, http.StatusSeeOther,
|
||||
w, r, "/source/"+webhook.ID, http.StatusSeeOther,
|
||||
)
|
||||
}
|
||||
|
||||
@@ -560,7 +457,6 @@ func (h *Handlers) renderSourceDetail(
|
||||
"Targets": delivery.NewTargetViews(targets),
|
||||
"Events": events,
|
||||
"BaseURL": baseURL,
|
||||
"Stats": h.loadWebhookStats(webhook.ID, entrypoints, targets),
|
||||
}
|
||||
|
||||
h.renderTemplate(w, r, "source_detail.html", data)
|
||||
@@ -692,7 +588,7 @@ func (h *Handlers) applyWebhookEdit(
|
||||
}
|
||||
|
||||
http.Redirect(
|
||||
w, r, "/hook/"+webhook.ID, http.StatusSeeOther,
|
||||
w, r, "/source/"+webhook.ID, http.StatusSeeOther,
|
||||
)
|
||||
}
|
||||
|
||||
@@ -775,7 +671,7 @@ func (h *Handlers) deleteWebhookResources(
|
||||
return
|
||||
}
|
||||
|
||||
http.Redirect(w, r, "/hooks", http.StatusSeeOther)
|
||||
http.Redirect(w, r, "/sources", http.StatusSeeOther)
|
||||
}
|
||||
|
||||
// commitWebhookDeletion soft-deletes a webhook's entrypoints,
|
||||
@@ -1368,7 +1264,7 @@ func (h *Handlers) HandleEntrypointCreate() http.HandlerFunc {
|
||||
}
|
||||
|
||||
http.Redirect(
|
||||
w, r, "/hook/"+webhook.ID, http.StatusSeeOther,
|
||||
w, r, "/source/"+webhook.ID, http.StatusSeeOther,
|
||||
)
|
||||
}
|
||||
}
|
||||
@@ -1424,7 +1320,7 @@ func (h *Handlers) processTargetCreate(
|
||||
//
|
||||
// Every field here is read with PostFormValue, not FormValue.
|
||||
// FormValue falls back to the query string, which would let
|
||||
// `POST /hook/{id}/targets?url=https://hooks.slack.com/...`
|
||||
// `POST /source/{id}/targets?url=https://hooks.slack.com/...`
|
||||
// configure a target from a value the request line carries — and
|
||||
// the request line, unlike the body, is what logs, proxies,
|
||||
// Referer headers and error trackers record.
|
||||
@@ -1481,7 +1377,7 @@ func (h *Handlers) processTargetCreate(
|
||||
}
|
||||
|
||||
http.Redirect(
|
||||
w, r, "/hook/"+webhook.ID, http.StatusSeeOther,
|
||||
w, r, "/source/"+webhook.ID, http.StatusSeeOther,
|
||||
)
|
||||
}
|
||||
|
||||
@@ -1539,7 +1435,7 @@ type targetFormInput struct {
|
||||
//
|
||||
// Every field is read with PostFormValue, not FormValue. FormValue
|
||||
// falls back to the query string, which would let
|
||||
// `POST /hook/{id}/targets?url=https://hooks.slack.com/...`
|
||||
// `POST /source/{id}/targets?url=https://hooks.slack.com/...`
|
||||
// configure a target from a value the request line carries — and the
|
||||
// request line, unlike the body, is what logs, proxies, Referer
|
||||
// headers and error trackers record. The headers field is under the
|
||||
@@ -1681,22 +1577,11 @@ func (h *Handlers) validateTargetURL(
|
||||
"url", delivery.MaskURL(targetURL),
|
||||
"error", err,
|
||||
)
|
||||
|
||||
msg := "Invalid target URL: " + err.Error()
|
||||
|
||||
// Only a private or reserved address's refusal says how
|
||||
// to allow it. Metadata refusals never do: link-local and
|
||||
// the other unconditional metadata addresses cannot be
|
||||
// opened, and the default blocklist's public addresses,
|
||||
// which listing does open, hand out credentials.
|
||||
if errors.Is(err, delivery.ErrBlockedPrivateOrReservedIP) {
|
||||
msg += ". Private and reserved addresses are refused " +
|
||||
"by default; the server's ALLOWED_EGRESS_CIDRS " +
|
||||
"setting allows named networks (see \"Allowing " +
|
||||
"egress to your own network\" in the README)."
|
||||
}
|
||||
|
||||
http.Error(w, msg, http.StatusBadRequest)
|
||||
http.Error(
|
||||
w,
|
||||
"Invalid target URL: "+err.Error(),
|
||||
http.StatusBadRequest,
|
||||
)
|
||||
|
||||
return err
|
||||
}
|
||||
@@ -1829,7 +1714,7 @@ func (h *Handlers) deleteChildResource(
|
||||
|
||||
http.Redirect(
|
||||
w, r,
|
||||
"/hook/"+webhook.ID,
|
||||
"/source/"+webhook.ID,
|
||||
http.StatusSeeOther,
|
||||
)
|
||||
}
|
||||
@@ -1926,7 +1811,7 @@ func (h *Handlers) toggleChildResource(
|
||||
|
||||
http.Redirect(
|
||||
w, r,
|
||||
"/hook/"+webhook.ID,
|
||||
"/source/"+webhook.ID,
|
||||
http.StatusSeeOther,
|
||||
)
|
||||
}
|
||||
|
||||
@@ -105,7 +105,7 @@ func submitCreate(
|
||||
form.Set("retention_days", *retention)
|
||||
}
|
||||
|
||||
req := formRequest("/hooks/new", cookies, form, nil)
|
||||
req := formRequest("/sources/new", cookies, form, nil)
|
||||
w := httptest.NewRecorder()
|
||||
|
||||
h.HandleSourceCreateSubmit().ServeHTTP(w, req)
|
||||
@@ -265,7 +265,7 @@ func TestHandleSourceCreate_PrefillsDefaultFromConstant(t *testing.T) {
|
||||
|
||||
w := httptest.NewRecorder()
|
||||
env.handlers.HandleSourceCreate().ServeHTTP(
|
||||
w, getRequest(t, "/hooks/new", env.cookies, nil),
|
||||
w, getRequest(t, "/sources/new", env.cookies, nil),
|
||||
)
|
||||
|
||||
require.Equal(t, http.StatusOK, w.Code)
|
||||
@@ -402,7 +402,7 @@ func TestHandleSourceCreateSubmit_RejectedFormKeepsUserInput(
|
||||
form.Set("description", description)
|
||||
form.Set("retention_days", "nonsense")
|
||||
|
||||
req := formRequest("/hooks/new", env.cookies, form, nil)
|
||||
req := formRequest("/sources/new", env.cookies, form, nil)
|
||||
w := httptest.NewRecorder()
|
||||
|
||||
env.handlers.HandleSourceCreateSubmit().ServeHTTP(w, req)
|
||||
@@ -430,7 +430,7 @@ func submitEdit(
|
||||
form.Set("retention_days", retention)
|
||||
|
||||
req := formRequest(
|
||||
"/hook/"+wh.ID+"/edit",
|
||||
"/source/"+wh.ID+"/edit",
|
||||
env.cookies,
|
||||
form,
|
||||
map[string]string{sourceIDParam: wh.ID},
|
||||
@@ -512,7 +512,7 @@ func TestSourceEditForm_ForeverWebhookRoundTrips(t *testing.T) {
|
||||
)
|
||||
|
||||
req := getRequest(
|
||||
t, "/hook/"+wh.ID+"/edit", env.cookies,
|
||||
t, "/source/"+wh.ID+"/edit", env.cookies,
|
||||
map[string]string{sourceIDParam: wh.ID},
|
||||
)
|
||||
w := httptest.NewRecorder()
|
||||
@@ -567,7 +567,7 @@ func TestSourceListAndDetail_ShowForeverNotTheSentinelNumber(
|
||||
|
||||
listW := httptest.NewRecorder()
|
||||
env.handlers.HandleSourceList().ServeHTTP(
|
||||
listW, getRequest(t, "/hooks", env.cookies, nil),
|
||||
listW, getRequest(t, "/sources", env.cookies, nil),
|
||||
)
|
||||
|
||||
require.Equal(t, http.StatusOK, listW.Code)
|
||||
@@ -578,7 +578,7 @@ func TestSourceListAndDetail_ShowForeverNotTheSentinelNumber(
|
||||
env.handlers.HandleSourceDetail().ServeHTTP(
|
||||
detailW,
|
||||
getRequest(
|
||||
t, "/hook/"+wh.ID, env.cookies,
|
||||
t, "/source/"+wh.ID, env.cookies,
|
||||
map[string]string{sourceIDParam: wh.ID},
|
||||
),
|
||||
)
|
||||
|
||||
@@ -76,11 +76,11 @@ func postTargetCreate(
|
||||
router := chi.NewRouter()
|
||||
router.Use(mw.Logging())
|
||||
router.Post(
|
||||
"/hook/{sourceID}/targets",
|
||||
"/source/{sourceID}/targets",
|
||||
env.handlers.HandleTargetCreate(),
|
||||
)
|
||||
|
||||
target := "/hook/" + webhookID + "/targets"
|
||||
target := "/source/" + webhookID + "/targets"
|
||||
if query != "" {
|
||||
target += "?" + query
|
||||
}
|
||||
@@ -114,7 +114,7 @@ func postTargetCreate(
|
||||
// regression test for the ingress leak. r.FormValue falls back to the
|
||||
// query string when a field is absent from the POST body, so
|
||||
//
|
||||
// POST /hook/{id}/targets?url=https://hooks.slack.com/services/...
|
||||
// POST /source/{id}/targets?url=https://hooks.slack.com/services/...
|
||||
//
|
||||
// with an empty url field used to create a working target from a value
|
||||
// carried on the request line — where logs, proxies, Referer headers
|
||||
|
||||
@@ -47,7 +47,7 @@ type targetEditView struct {
|
||||
//
|
||||
// This page is the one place the full destination URL and header
|
||||
// values are shown. It is reachable only through the
|
||||
// /hook/{sourceID} route group, which supplies RequireAuth and
|
||||
// /source/{sourceID} route group, which supplies RequireAuth and
|
||||
// NoCache, and only for a target of a webhook the session's user
|
||||
// owns; masking (delivery.TargetView) is unchanged everywhere else.
|
||||
func (h *Handlers) HandleTargetEdit() http.HandlerFunc {
|
||||
@@ -163,7 +163,7 @@ func (h *Handlers) applyTargetEdit(
|
||||
}
|
||||
|
||||
http.Redirect(
|
||||
w, r, "/hook/"+webhook.ID, http.StatusSeeOther,
|
||||
w, r, "/source/"+webhook.ID, http.StatusSeeOther,
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
@@ -42,15 +42,15 @@ const (
|
||||
func targetRouter(env *sourceTestEnv) *chi.Mux {
|
||||
router := chi.NewRouter()
|
||||
router.Post(
|
||||
"/hook/{sourceID}/targets",
|
||||
"/source/{sourceID}/targets",
|
||||
env.handlers.HandleTargetCreate(),
|
||||
)
|
||||
router.Get(
|
||||
"/hook/{sourceID}/targets/{targetID}/edit",
|
||||
"/source/{sourceID}/targets/{targetID}/edit",
|
||||
env.handlers.HandleTargetEdit(),
|
||||
)
|
||||
router.Post(
|
||||
"/hook/{sourceID}/targets/{targetID}/edit",
|
||||
"/source/{sourceID}/targets/{targetID}/edit",
|
||||
env.handlers.HandleTargetEditSubmit(),
|
||||
)
|
||||
|
||||
@@ -117,7 +117,7 @@ func seedHTTPTarget(
|
||||
|
||||
w := serveTarget(
|
||||
env, http.MethodPost,
|
||||
"/hook/"+webhook.ID+"/targets", form,
|
||||
"/source/"+webhook.ID+"/targets", form,
|
||||
)
|
||||
require.Equal(t, http.StatusSeeOther, w.Code, w.Body.String())
|
||||
|
||||
@@ -188,7 +188,7 @@ func submitTargetEdit(
|
||||
) *httptest.ResponseRecorder {
|
||||
return serveTarget(
|
||||
env, http.MethodPost,
|
||||
"/hook/"+webhookID+"/targets/"+targetID+"/edit",
|
||||
"/source/"+webhookID+"/targets/"+targetID+"/edit",
|
||||
form,
|
||||
)
|
||||
}
|
||||
@@ -401,7 +401,7 @@ func TestHandleTargetEdit_PrefillsTheStoredValuesUnmasked(
|
||||
|
||||
w := serveTarget(
|
||||
env, http.MethodGet,
|
||||
"/hook/"+webhook.ID+"/targets/"+target.ID+"/edit",
|
||||
"/source/"+webhook.ID+"/targets/"+target.ID+"/edit",
|
||||
nil,
|
||||
)
|
||||
require.Equal(t, http.StatusOK, w.Code)
|
||||
@@ -508,7 +508,7 @@ func assertEditIgnoresQueryString(
|
||||
|
||||
w := serveTarget(
|
||||
env, http.MethodPost,
|
||||
"/hook/"+webhook.ID+"/targets/"+target.ID+
|
||||
"/source/"+webhook.ID+"/targets/"+target.ID+
|
||||
"/edit?url="+url.QueryEscape(editReplacedURL)+
|
||||
"&headers="+url.QueryEscape(editAuthHeader),
|
||||
form,
|
||||
@@ -592,7 +592,7 @@ func assertTargetOfAnotherWebhook404s(
|
||||
|
||||
get := serveTarget(
|
||||
env, http.MethodGet,
|
||||
"/hook/"+mine.ID+"/targets/"+target.ID+"/edit", nil,
|
||||
"/source/"+mine.ID+"/targets/"+target.ID+"/edit", nil,
|
||||
)
|
||||
assert.Equal(t, http.StatusNotFound, get.Code)
|
||||
|
||||
@@ -630,7 +630,7 @@ func assertWebhookOfAnotherUser404s(
|
||||
|
||||
w := serveTarget(
|
||||
env, http.MethodGet,
|
||||
"/hook/"+other.ID+"/targets/"+target.ID+"/edit", nil,
|
||||
"/source/"+other.ID+"/targets/"+target.ID+"/edit", nil,
|
||||
)
|
||||
|
||||
assert.Equal(t, http.StatusNotFound, w.Code)
|
||||
|
||||
@@ -1,116 +0,0 @@
|
||||
package handlers_test
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/url"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
)
|
||||
|
||||
// privateRefusalHint is the sentence that tells an operator a private
|
||||
// destination is refused on purpose, and how to allow one.
|
||||
const privateRefusalHint = "Private and reserved addresses are " +
|
||||
"refused by default; the server's ALLOWED_EGRESS_CIDRS setting " +
|
||||
"allows named networks (see \"Allowing egress to your own " +
|
||||
"network\" in the README)."
|
||||
|
||||
// TestTargetRefusal_PrivateDestinationSaysHowToAllowIt covers both
|
||||
// target types that take a URL, on add and on edit.
|
||||
func TestTargetRefusal_PrivateDestinationSaysHowToAllowIt(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupSourceTest(t)
|
||||
|
||||
targetTypes := []database.TargetType{
|
||||
database.TargetTypeHTTP,
|
||||
database.TargetTypeSlack,
|
||||
}
|
||||
|
||||
for _, targetType := range targetTypes {
|
||||
t.Run(string(targetType), func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
webhook := seedWebhookWithRetention(t, env.db, 30)
|
||||
targetsPath := "/hook/" + webhook.ID + "/targets"
|
||||
|
||||
form := url.Values{}
|
||||
form.Set("name", "private")
|
||||
form.Set("type", string(targetType))
|
||||
form.Set("url", editBlockedURL)
|
||||
|
||||
added := serveTarget(
|
||||
env, http.MethodPost, targetsPath, form,
|
||||
)
|
||||
assert.Equal(t, http.StatusBadRequest, added.Code)
|
||||
assert.Contains(
|
||||
t, added.Body.String(), privateRefusalHint,
|
||||
)
|
||||
|
||||
form.Set("url", editOriginalURL)
|
||||
|
||||
created := serveTarget(
|
||||
env, http.MethodPost, targetsPath, form,
|
||||
)
|
||||
require.Equal(
|
||||
t, http.StatusSeeOther, created.Code,
|
||||
created.Body.String(),
|
||||
)
|
||||
|
||||
targets := targetsForWebhook(t, env.db, webhook.ID)
|
||||
require.Len(t, targets, 1)
|
||||
|
||||
form.Set("url", editBlockedURL)
|
||||
|
||||
edited := submitTargetEdit(
|
||||
env, webhook.ID, targets[0].ID, form,
|
||||
)
|
||||
assert.Equal(t, http.StatusBadRequest, edited.Code)
|
||||
assert.Contains(
|
||||
t, edited.Body.String(), privateRefusalHint,
|
||||
)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestTargetRefusal_MetadataDestinationDoesNotSayHowToAllowIt: no
|
||||
// setting opens a link-local address, and Azure's WireServer hands out
|
||||
// VM credentials, so neither refusal points at the setting.
|
||||
func TestTargetRefusal_MetadataDestinationDoesNotSayHowToAllowIt(
|
||||
t *testing.T,
|
||||
) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupSourceTest(t)
|
||||
|
||||
metadataURLs := map[string]string{
|
||||
"link-local": "http://169.254.169.254/latest/meta-data/",
|
||||
"wireserver": "http://168.63.129.16/?comp=versions",
|
||||
}
|
||||
|
||||
for name, metadataURL := range metadataURLs {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
webhook := seedWebhookWithRetention(t, env.db, 30)
|
||||
|
||||
form := url.Values{}
|
||||
form.Set("name", "metadata")
|
||||
form.Set("type", string(database.TargetTypeHTTP))
|
||||
form.Set("url", metadataURL)
|
||||
|
||||
w := serveTarget(
|
||||
env, http.MethodPost,
|
||||
"/hook/"+webhook.ID+"/targets", form,
|
||||
)
|
||||
assert.Equal(t, http.StatusBadRequest, w.Code)
|
||||
assert.NotContains(
|
||||
t, w.Body.String(), privateRefusalHint,
|
||||
)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -102,7 +102,7 @@ func createWithRetries(
|
||||
|
||||
w := serveTarget(
|
||||
env, http.MethodPost,
|
||||
"/hook/"+webhook.ID+"/targets",
|
||||
"/source/"+webhook.ID+"/targets",
|
||||
createRetriesForm(retries),
|
||||
)
|
||||
|
||||
|
||||
@@ -54,7 +54,8 @@ func renderPage(
|
||||
}
|
||||
|
||||
// TestNavbarUsesWebhookTerminology pins the user-visible navigation
|
||||
// label to "Webhooks" and its link to the webhook list at /hooks.
|
||||
// label to "Webhooks". The /sources route is deliberately unchanged, so
|
||||
// the assertion targets the link text rather than the href.
|
||||
func TestNavbarUsesWebhookTerminology(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -94,11 +95,15 @@ func TestNavbarUsesWebhookTerminology(t *testing.T) {
|
||||
t, body, ">Sources<",
|
||||
"no user-visible element may still be labelled Sources",
|
||||
)
|
||||
assert.Contains(t, body, `href="/hooks"`)
|
||||
assert.Contains(
|
||||
t, body, `href="/sources"`,
|
||||
"the /sources route itself must not change",
|
||||
)
|
||||
}
|
||||
|
||||
// TestEditPageUsesWebhookTerminology pins the edit page's heading and
|
||||
// its back link to the webhook page at /hook/{id}.
|
||||
// its back link. The link's href still points at /source/{id}, which is
|
||||
// intentional: only user-visible copy changes.
|
||||
func TestEditPageUsesWebhookTerminology(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -125,57 +130,7 @@ func TestEditPageUsesWebhookTerminology(t *testing.T) {
|
||||
|
||||
assert.Contains(t, body, "Edit Webhook")
|
||||
assert.NotContains(t, body, ">Sources<")
|
||||
assert.Contains(t, body, `href="/hook/wh-1"`)
|
||||
}
|
||||
|
||||
// TestEventLogPageIsCalledFullEventLog pins the one name the event log
|
||||
// page at /hook/{id}/events goes by: both links to it on the webhook
|
||||
// page, and its own heading, read "Full Event Log".
|
||||
func TestEventLogPageIsCalledFullEventLog(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var h *handlers.Handlers
|
||||
|
||||
var sess *session.Session
|
||||
|
||||
app := newTestApp(t, &h, &sess)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
// A pointer, as in the handlers: source_detail.html calls
|
||||
// Webhook.RetentionLabel, a pointer method. Both pages only range
|
||||
// over their lists, and a list left out renders as empty, so the
|
||||
// lists are left out.
|
||||
webhook := &database.Webhook{Name: "wh", RetentionDays: 14}
|
||||
webhook.ID = testWebhookID
|
||||
|
||||
detailBody := renderPage(
|
||||
t, h, sess, "source_detail.html", map[string]any{
|
||||
dataKeyWebhook: webhook,
|
||||
},
|
||||
)
|
||||
|
||||
assert.Contains(
|
||||
t, detailBody,
|
||||
`<a href="/hook/wh-1/events" class="btn-secondary">Full Event Log</a>`,
|
||||
"the button at the top of the webhook page",
|
||||
)
|
||||
assert.Contains(
|
||||
t, detailBody,
|
||||
`<a href="/hook/wh-1/events" class="btn-text text-sm">Full Event Log</a>`,
|
||||
"the link under recent events",
|
||||
)
|
||||
|
||||
logBody := renderPage(t, h, sess, "source_logs.html", map[string]any{
|
||||
dataKeyWebhook: webhook,
|
||||
"TotalEvents": int64(0),
|
||||
})
|
||||
|
||||
assert.Contains(
|
||||
t, logBody,
|
||||
`<h1 class="text-2xl font-medium text-gray-900">Full Event Log</h1>`,
|
||||
)
|
||||
assert.Contains(t, body, `href="/source/wh-1"`)
|
||||
}
|
||||
|
||||
// TestCreateFormRetentionCopyMatchesBehaviour pins the create form's
|
||||
@@ -328,7 +283,7 @@ func TestEntrypointCopyButtonIsProgressiveEnhancement(t *testing.T) {
|
||||
t, body,
|
||||
`<code id="entrypoint-url-ep-1"`,
|
||||
)
|
||||
assert.Contains(t, body, "https://hooks.example.com/h/abc123")
|
||||
assert.Contains(t, body, "https://hooks.example.com/webhook/abc123")
|
||||
assert.Contains(
|
||||
t, body,
|
||||
`hidden data-copy-target="entrypoint-url-ep-1"`,
|
||||
|
||||
@@ -131,7 +131,7 @@ func (h *Handlers) lookupEntrypoint(
|
||||
"path = ?", entrypointUUID,
|
||||
).First(&entrypoint)
|
||||
if result.Error != nil {
|
||||
// The receiver is unauthenticated and /h/{uuid}
|
||||
// The receiver is unauthenticated and /webhook/{uuid}
|
||||
// matches any single segment, so this value is entirely
|
||||
// client-chosen on exactly the branch where the lookup
|
||||
// failed. DEBUG is off by default; the cap is what keeps
|
||||
@@ -253,12 +253,11 @@ func requestEventSource(
|
||||
}
|
||||
}
|
||||
|
||||
// createAndFanOut writes the event and one pending delivery per target,
|
||||
// and adds them to the webhook's running totals, in a single
|
||||
// transaction, then hands the tasks to the delivery engine. It is the
|
||||
// only path by which an event and its deliveries are created, so a
|
||||
// resubmitted event is retried, SSRF-guarded and circuit-broken
|
||||
// exactly as a received one is.
|
||||
// createAndFanOut writes the event and one pending delivery per target
|
||||
// in a single transaction, then hands the tasks to the delivery
|
||||
// engine. It is the only path by which an event and its deliveries are
|
||||
// created, so a resubmitted event is retried, SSRF-guarded and
|
||||
// circuit-broken exactly as a received one is.
|
||||
//
|
||||
// The tasks are returned as well as queued, so a caller can report how
|
||||
// many targets the event went to.
|
||||
@@ -298,15 +297,6 @@ func (h *Handlers) createAndFanOut(
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
err = database.AddEventTotals(tx, database.EventTotals{
|
||||
Events: 1, LastEventAt: &event.CreatedAt,
|
||||
})
|
||||
if err != nil {
|
||||
tx.Rollback()
|
||||
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
err = tx.Commit().Error
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf(
|
||||
@@ -365,9 +355,8 @@ func (h *Handlers) finishWebhookResponse(
|
||||
}
|
||||
|
||||
// buildDeliveryTasks creates one pending delivery per target in the
|
||||
// transaction, adds each to its target's totals, and returns the tasks
|
||||
// for the delivery engine. The caller owns the transaction and rolls
|
||||
// it back on error.
|
||||
// transaction and returns the tasks for the delivery engine. The
|
||||
// caller owns the transaction and rolls it back on error.
|
||||
func buildDeliveryTasks(
|
||||
tx *gorm.DB,
|
||||
event *database.Event,
|
||||
@@ -391,13 +380,6 @@ func buildDeliveryTasks(
|
||||
)
|
||||
}
|
||||
|
||||
err = database.AddTargetTotals(tx, database.TargetTotals{
|
||||
TargetID: targets[i].ID, Deliveries: 1,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
tasks = append(tasks, delivery.Task{
|
||||
DeliveryID: dlv.ID,
|
||||
EventID: event.ID,
|
||||
|
||||
@@ -1,259 +0,0 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"gorm.io/gorm"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
)
|
||||
|
||||
// The spans of the two recent windows the statistics pane reports on:
|
||||
// the last 10 minutes and the last 24 hours.
|
||||
const (
|
||||
shortWindow = 10 * time.Minute
|
||||
longWindow = 24 * time.Hour
|
||||
)
|
||||
|
||||
// percent turns a fraction into a percentage.
|
||||
const percent = 100
|
||||
|
||||
// WebhookStats holds the figures in the statistics pane at the top of
|
||||
// the webhook page.
|
||||
type WebhookStats struct {
|
||||
Entrypoints int
|
||||
ActiveEntrypoints int
|
||||
Targets int
|
||||
ActiveTargets int
|
||||
|
||||
// Lifetime counts every event, delivery and failure the webhook
|
||||
// has had, and WithinRetention those still stored.
|
||||
Lifetime Counts
|
||||
WithinRetention Counts
|
||||
|
||||
// InProgress counts the deliveries still pending or retrying.
|
||||
InProgress int64
|
||||
|
||||
// LastEventAt is when the newest event arrived, or nil when none
|
||||
// has. Retention does not change it.
|
||||
LastEventAt *time.Time
|
||||
|
||||
Last10Minutes RecentWindow
|
||||
Last24Hours RecentWindow
|
||||
}
|
||||
|
||||
// Counts holds a number of events, of deliveries and of failed
|
||||
// deliveries.
|
||||
type Counts struct {
|
||||
Events int64
|
||||
Deliveries int64
|
||||
Failures int64
|
||||
}
|
||||
|
||||
// RecentWindow holds what happened in one recent window: the events
|
||||
// received in it, and the deliveries that became delivered or failed in
|
||||
// it.
|
||||
type RecentWindow struct {
|
||||
Events int64
|
||||
Delivered int64
|
||||
Failed int64
|
||||
}
|
||||
|
||||
// TargetFinished is how many of one target's deliveries became
|
||||
// delivered, and how many failed, in a recent window.
|
||||
type TargetFinished struct {
|
||||
TargetID string
|
||||
Delivered int64
|
||||
Failed int64
|
||||
}
|
||||
|
||||
// FailurePercent is the share of the deliveries finished in the window
|
||||
// that failed, or a dash when none finished. Deliveries still pending
|
||||
// or retrying are not counted either way.
|
||||
func (w RecentWindow) FailurePercent() string {
|
||||
finished := w.Delivered + w.Failed
|
||||
if finished == 0 {
|
||||
return "—"
|
||||
}
|
||||
|
||||
return fmt.Sprintf(
|
||||
"%.1f%%", percent*float64(w.Failed)/float64(finished),
|
||||
)
|
||||
}
|
||||
|
||||
// loadWebhookStats gathers the figures for the statistics pane from the
|
||||
// webhook's entrypoints and targets, as the page has already loaded
|
||||
// them, and from its event database. It returns nil, and logs why, when
|
||||
// the event database cannot be read.
|
||||
func (h *Handlers) loadWebhookStats(
|
||||
webhookID string,
|
||||
entrypoints []database.Entrypoint,
|
||||
targets []database.Target,
|
||||
) *WebhookStats {
|
||||
stats := &WebhookStats{
|
||||
Entrypoints: len(entrypoints),
|
||||
Targets: len(targets),
|
||||
}
|
||||
|
||||
for i := range entrypoints {
|
||||
if entrypoints[i].Active {
|
||||
stats.ActiveEntrypoints++
|
||||
}
|
||||
}
|
||||
|
||||
for i := range targets {
|
||||
if targets[i].Active {
|
||||
stats.ActiveTargets++
|
||||
}
|
||||
}
|
||||
|
||||
// Opening an event database that does not exist would create it,
|
||||
// and it would hold nothing to count.
|
||||
if !h.dbMgr.DBExists(webhookID) {
|
||||
return stats
|
||||
}
|
||||
|
||||
webhookDB, err := h.dbMgr.GetDB(webhookID)
|
||||
if err == nil {
|
||||
err = readEventStats(webhookDB, time.Now(), stats)
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
h.log.Error(
|
||||
"failed to read webhook statistics",
|
||||
"webhook_id", webhookID,
|
||||
"error", err,
|
||||
)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
return stats
|
||||
}
|
||||
|
||||
// readEventStats fills in the figures that come from the webhook's
|
||||
// event database. None of them reads every stored row: the totals are
|
||||
// one row for the events and one per target for the deliveries, and
|
||||
// every other figure is read from an index, over only the rows it
|
||||
// counts.
|
||||
func readEventStats(
|
||||
db *gorm.DB, now time.Time, stats *WebhookStats,
|
||||
) error {
|
||||
err := readTotals(db, stats)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
err = db.Model(&database.Delivery{}).
|
||||
Where("status IN ?", []database.DeliveryStatus{
|
||||
database.DeliveryStatusPending,
|
||||
database.DeliveryStatusRetrying,
|
||||
}).
|
||||
Count(&stats.InProgress).Error
|
||||
if err != nil {
|
||||
return fmt.Errorf("counting deliveries in progress: %w", err)
|
||||
}
|
||||
|
||||
stats.Last10Minutes, err = readRecentWindow(
|
||||
db, now.Add(-shortWindow),
|
||||
)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
stats.Last24Hours, err = readRecentWindow(
|
||||
db, now.Add(-longWindow),
|
||||
)
|
||||
|
||||
return err
|
||||
}
|
||||
|
||||
// readTotals fills in the lifetime and within-retention figures, and
|
||||
// when the last event arrived, from the running totals: the events'
|
||||
// row, and the targets' rows summed.
|
||||
func readTotals(db *gorm.DB, stats *WebhookStats) error {
|
||||
var events database.EventTotals
|
||||
|
||||
err := db.Take(&events).Error
|
||||
if err != nil {
|
||||
return fmt.Errorf("reading event totals: %w", err)
|
||||
}
|
||||
|
||||
var targets []database.TargetTotals
|
||||
|
||||
err = db.Find(&targets).Error
|
||||
if err != nil {
|
||||
return fmt.Errorf("reading target totals: %w", err)
|
||||
}
|
||||
|
||||
stats.Lifetime.Events = events.Events
|
||||
stats.WithinRetention.Events = events.Events - events.EventsRemoved
|
||||
stats.LastEventAt = events.LastEventAt
|
||||
|
||||
for _, t := range targets {
|
||||
stats.Lifetime.Deliveries += t.Deliveries
|
||||
stats.Lifetime.Failures += t.Failed
|
||||
stats.WithinRetention.Deliveries += t.Deliveries - t.DeliveriesRemoved
|
||||
stats.WithinRetention.Failures += t.Failed - t.FailedRemoved
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// readRecentWindow counts the events received, and the deliveries that
|
||||
// became delivered or failed, since the given time.
|
||||
func readRecentWindow(
|
||||
db *gorm.DB, since time.Time,
|
||||
) (RecentWindow, error) {
|
||||
var w RecentWindow
|
||||
|
||||
err := db.Model(&database.Event{}).
|
||||
Where("created_at >= ?", since).
|
||||
Count(&w.Events).Error
|
||||
if err != nil {
|
||||
return w, fmt.Errorf("counting recent events: %w", err)
|
||||
}
|
||||
|
||||
byTarget, err := finishedByTarget(db, since)
|
||||
if err != nil {
|
||||
return w, err
|
||||
}
|
||||
|
||||
for _, f := range byTarget {
|
||||
w.Delivered += f.Delivered
|
||||
w.Failed += f.Failed
|
||||
}
|
||||
|
||||
return w, nil
|
||||
}
|
||||
|
||||
// finishedByTarget counts, for each target, the deliveries that became
|
||||
// delivered and those that failed since the given time, in one query
|
||||
// over just that window of the deliveries' status index. A target with
|
||||
// neither is left out.
|
||||
func finishedByTarget(
|
||||
db *gorm.DB, since time.Time,
|
||||
) ([]TargetFinished, error) {
|
||||
var byTarget []TargetFinished
|
||||
|
||||
err := db.Model(&database.Delivery{}).
|
||||
Select("target_id, "+
|
||||
"count(CASE WHEN status = ? THEN 1 END) AS delivered, "+
|
||||
"count(CASE WHEN status = ? THEN 1 END) AS failed",
|
||||
database.DeliveryStatusDelivered,
|
||||
database.DeliveryStatusFailed).
|
||||
Where("status IN ? AND finished_at >= ?",
|
||||
[]database.DeliveryStatus{
|
||||
database.DeliveryStatusDelivered,
|
||||
database.DeliveryStatusFailed,
|
||||
}, since).
|
||||
Group("target_id").
|
||||
Find(&byTarget).Error
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf(
|
||||
"counting deliveries finished by target: %w", err,
|
||||
)
|
||||
}
|
||||
|
||||
return byTarget, nil
|
||||
}
|
||||
@@ -1,569 +0,0 @@
|
||||
package handlers_test
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"go.uber.org/fx/fxtest"
|
||||
"gorm.io/gorm"
|
||||
"gorm.io/gorm/clause"
|
||||
"sneak.berlin/go/webhooker/internal/config"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/handlers"
|
||||
"sneak.berlin/go/webhooker/internal/logger"
|
||||
"sneak.berlin/go/webhooker/internal/session"
|
||||
)
|
||||
|
||||
// statsEntrypoint adds an entrypoint to a webhook and returns its path.
|
||||
func statsEntrypoint(
|
||||
t *testing.T, db *database.Database, webhookID string, active bool,
|
||||
) string {
|
||||
t.Helper()
|
||||
|
||||
ep := &database.Entrypoint{
|
||||
WebhookID: webhookID,
|
||||
Path: uuid.New().String(),
|
||||
}
|
||||
|
||||
require.NoError(t, db.DB().Omit(clause.Associations).Create(ep).Error)
|
||||
require.NoError(t, db.DB().Model(ep).Update("active", active).Error)
|
||||
|
||||
return ep.Path
|
||||
}
|
||||
|
||||
// statsDelivery returns an event's delivery to a target.
|
||||
func statsDelivery(
|
||||
t *testing.T, webhookDB *gorm.DB, eventID, targetID string,
|
||||
) database.Delivery {
|
||||
t.Helper()
|
||||
|
||||
var d database.Delivery
|
||||
|
||||
require.NoError(t, webhookDB.Where(
|
||||
"event_id = ? AND target_id = ?", eventID, targetID,
|
||||
).First(&d).Error)
|
||||
|
||||
return d
|
||||
}
|
||||
|
||||
// statsFinish settles a delivery as the delivery engine does: its
|
||||
// final status and the time it finished, and one more on its target's
|
||||
// delivered or failed total, in one transaction.
|
||||
func statsFinish(
|
||||
t *testing.T,
|
||||
webhookDB *gorm.DB,
|
||||
d database.Delivery,
|
||||
status database.DeliveryStatus,
|
||||
at time.Time,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
add := database.TargetTotals{TargetID: d.TargetID, Delivered: 1}
|
||||
if status == database.DeliveryStatusFailed {
|
||||
add = database.TargetTotals{TargetID: d.TargetID, Failed: 1}
|
||||
}
|
||||
|
||||
require.NoError(t, webhookDB.Transaction(func(tx *gorm.DB) error {
|
||||
err := tx.Model(&database.Delivery{}).
|
||||
Where("id = ?", d.ID).
|
||||
Updates(map[string]any{"status": status, "finished_at": at}).
|
||||
Error
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return database.AddTargetTotals(tx, add)
|
||||
}))
|
||||
}
|
||||
|
||||
// statsAge moves an event's arrival back to the given time.
|
||||
func statsAge(
|
||||
t *testing.T, webhookDB *gorm.DB, eventID string, at time.Time,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
require.NoError(t, webhookDB.Model(&database.Event{}).
|
||||
Where("id = ?", eventID).
|
||||
Update("created_at", at).Error)
|
||||
}
|
||||
|
||||
// statsTargetTotals reads a webhook database's target totals, keyed by
|
||||
// target.
|
||||
func statsTargetTotals(
|
||||
t *testing.T, webhookDB *gorm.DB,
|
||||
) map[string]database.TargetTotals {
|
||||
t.Helper()
|
||||
|
||||
var rows []database.TargetTotals
|
||||
|
||||
require.NoError(t, webhookDB.Find(&rows).Error)
|
||||
|
||||
byTarget := make(map[string]database.TargetTotals, len(rows))
|
||||
for _, row := range rows {
|
||||
byTarget[row.TargetID] = row
|
||||
}
|
||||
|
||||
return byTarget
|
||||
}
|
||||
|
||||
// statsHistory is the webhook seedStatsHistory builds: its event
|
||||
// database, its newest event, and its two active targets.
|
||||
type statsHistory struct {
|
||||
webhook *database.Webhook
|
||||
webhookDB *gorm.DB
|
||||
newest database.Event
|
||||
first, second string
|
||||
}
|
||||
|
||||
// seedStatsHistory builds the webhook the statistics test checks: 14
|
||||
// days of retention, twelve entrypoints (one inactive) and six targets
|
||||
// (four inactive). Ten events arrive through the receiver, and so each
|
||||
// has a delivery to the two active targets. The oldest event is past
|
||||
// retention, the next 30 hours old, the next six hours old, the other
|
||||
// seven just in. Six deliveries are settled as the delivery engine
|
||||
// would, two of them inside a recent window though their event arrived
|
||||
// before it. The newest event's delivery to the second target is
|
||||
// retrying, the rest are left pending, and a replay adds a pending
|
||||
// delivery to the oldest event. Once retention has removed the oldest
|
||||
// event, every figure in the pane differs from every other.
|
||||
func seedStatsHistory(
|
||||
t *testing.T,
|
||||
h *handlers.Handlers,
|
||||
sess *session.Session,
|
||||
db *database.Database,
|
||||
dbMgr *database.WebhookDBManager,
|
||||
) statsHistory {
|
||||
t.Helper()
|
||||
|
||||
wh := &database.Webhook{UserID: deleteTestUserID, Name: "stats", RetentionDays: 14}
|
||||
require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
|
||||
|
||||
path := statsEntrypoint(t, db, wh.ID, true)
|
||||
for range 10 {
|
||||
statsEntrypoint(t, db, wh.ID, true)
|
||||
}
|
||||
|
||||
statsEntrypoint(t, db, wh.ID, false)
|
||||
|
||||
first := seedConfiguredTarget(
|
||||
t, db, wh.ID, database.TargetTypeHTTP,
|
||||
`{"url":"`+replayTargetURL+`"}`,
|
||||
)
|
||||
second := seedTarget(t, db, wh.ID, database.TargetTypeLog)
|
||||
|
||||
for range 4 {
|
||||
inactive := seedTarget(t, db, wh.ID, database.TargetTypeLog)
|
||||
require.NoError(t, db.DB().Model(inactive).
|
||||
Update("active", false).Error)
|
||||
}
|
||||
|
||||
router := receiverRouter(h)
|
||||
|
||||
for range 10 {
|
||||
require.Equal(t, http.StatusOK, postReceiver(t, router, path))
|
||||
}
|
||||
|
||||
webhookDB, err := dbMgr.GetDB(wh.ID)
|
||||
require.NoError(t, err)
|
||||
|
||||
events := listEvents(t, webhookDB)
|
||||
require.Len(t, events, 10)
|
||||
|
||||
oldest, yesterday, middle, newest := events[0], events[1], events[2], events[9]
|
||||
now := time.Now()
|
||||
|
||||
statsAge(t, webhookDB, oldest.ID, now.Add(-15*24*time.Hour))
|
||||
statsAge(t, webhookDB, yesterday.ID, now.Add(-30*time.Hour))
|
||||
statsAge(t, webhookDB, middle.ID, now.Add(-6*time.Hour))
|
||||
|
||||
oldestFailure := statsDelivery(t, webhookDB, oldest.ID, first.ID)
|
||||
statsFinish(t, webhookDB, oldestFailure,
|
||||
database.DeliveryStatusFailed, now.Add(-14*24*time.Hour))
|
||||
statsFinish(t, webhookDB,
|
||||
statsDelivery(t, webhookDB, yesterday.ID, first.ID),
|
||||
database.DeliveryStatusFailed, now.Add(-29*time.Hour))
|
||||
statsFinish(t, webhookDB,
|
||||
statsDelivery(t, webhookDB, yesterday.ID, second.ID),
|
||||
database.DeliveryStatusFailed, now.Add(-23*time.Hour))
|
||||
statsFinish(t, webhookDB,
|
||||
statsDelivery(t, webhookDB, middle.ID, second.ID),
|
||||
database.DeliveryStatusFailed, now.Add(-5*time.Hour))
|
||||
statsFinish(t, webhookDB,
|
||||
statsDelivery(t, webhookDB, middle.ID, first.ID),
|
||||
database.DeliveryStatusFailed, now.Add(-time.Minute))
|
||||
statsFinish(t, webhookDB,
|
||||
statsDelivery(t, webhookDB, newest.ID, first.ID),
|
||||
database.DeliveryStatusDelivered, now.Add(-2*time.Minute))
|
||||
|
||||
retrying := statsDelivery(t, webhookDB, newest.ID, second.ID)
|
||||
require.NoError(t, webhookDB.Model(&retrying).
|
||||
Update("status", database.DeliveryStatusRetrying).Error)
|
||||
|
||||
require.Equal(t, http.StatusSeeOther,
|
||||
postReplay(t, h, sess, wh.ID, oldestFailure.ID).Code)
|
||||
|
||||
return statsHistory{
|
||||
webhook: wh, webhookDB: webhookDB, newest: newest,
|
||||
first: first.ID, second: second.ID,
|
||||
}
|
||||
}
|
||||
|
||||
// statsPrune runs the real retention reaper until it has removed one
|
||||
// event from the webhook's database, then stops it.
|
||||
func statsPrune(
|
||||
t *testing.T,
|
||||
db *database.Database,
|
||||
dbMgr *database.WebhookDBManager,
|
||||
log *logger.Logger,
|
||||
webhookDB *gorm.DB,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
lc := fxtest.NewLifecycle(t)
|
||||
database.NewRetentionReaper(lc, database.RetentionReaperParams{
|
||||
Config: &config.Config{
|
||||
RetentionSweepInterval: 10 * time.Millisecond,
|
||||
},
|
||||
Database: db,
|
||||
DBManager: dbMgr,
|
||||
Logger: log,
|
||||
})
|
||||
|
||||
lc.RequireStart()
|
||||
|
||||
require.Eventually(t, func() bool {
|
||||
var totals database.EventTotals
|
||||
|
||||
err := webhookDB.Take(&totals).Error
|
||||
|
||||
return err == nil && totals.EventsRemoved == 1
|
||||
}, 10*time.Second, 10*time.Millisecond)
|
||||
|
||||
lc.RequireStop()
|
||||
}
|
||||
|
||||
// statsPane returns the text of the statistics pane in a rendered
|
||||
// webhook page, everything from its heading to the next heading on the
|
||||
// page, with the markup taken out and each run of space made one
|
||||
// space. A table then reads header by header and row by row, each
|
||||
// row's label followed by its figures in column order.
|
||||
func statsPane(t *testing.T, page string) string {
|
||||
t.Helper()
|
||||
|
||||
_, pane, found := strings.Cut(page, ">Statistics</h2>")
|
||||
require.True(t, found, "the page has no statistics pane")
|
||||
|
||||
pane, _, _ = strings.Cut(pane, "<h2")
|
||||
pane = regexp.MustCompile(`<[^>]*>`).ReplaceAllString(pane, " ")
|
||||
|
||||
return strings.Join(strings.Fields(pane), " ")
|
||||
}
|
||||
|
||||
// assertStatsTargets checks, for the history seedStatsHistory builds,
|
||||
// each target's totals and its deliveries finished in the last 24
|
||||
// hours. The first target has ten deliveries and the replay, the
|
||||
// second ten; the inactive targets have none and so no row.
|
||||
func assertStatsTargets(t *testing.T, hist statsHistory) {
|
||||
t.Helper()
|
||||
|
||||
first, second := hist.first, hist.second
|
||||
|
||||
assert.Equal(t, map[string]database.TargetTotals{
|
||||
first: {TargetID: first, Deliveries: 11, Delivered: 1, Failed: 3},
|
||||
second: {TargetID: second, Deliveries: 10, Failed: 2},
|
||||
}, statsTargetTotals(t, hist.webhookDB))
|
||||
|
||||
lastDay, err := handlers.FinishedByTargetForTest(
|
||||
hist.webhookDB, time.Now().Add(-24*time.Hour),
|
||||
)
|
||||
require.NoError(t, err)
|
||||
assert.ElementsMatch(t, []handlers.TargetFinished{
|
||||
{TargetID: first, Delivered: 1, Failed: 1},
|
||||
{TargetID: second, Failed: 2},
|
||||
}, lastDay)
|
||||
}
|
||||
|
||||
// assertStatsPaneAfterPrune checks the rendered statistics pane for the
|
||||
// history seedStatsHistory builds, once retention has removed the
|
||||
// oldest event: each figure after its label, in its column.
|
||||
func assertStatsPaneAfterPrune(
|
||||
t *testing.T,
|
||||
h *handlers.Handlers,
|
||||
sess *session.Session,
|
||||
hist statsHistory,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
pane := statsPane(t, renderSourceDetailPage(t, h, sess, hist.webhook.ID))
|
||||
lastEvent := hist.newest.CreatedAt.UTC().Format("2006-01-02 15:04:05 UTC")
|
||||
|
||||
assert.Contains(t, pane, "Entrypoints 12 (11 active) "+
|
||||
"Targets 6 (2 active) "+
|
||||
"Deliveries in progress 13 "+
|
||||
"Last event "+lastEvent+" "+
|
||||
"Retention 14 days")
|
||||
assert.Contains(t, pane, "Lifetime Within retention "+
|
||||
"Events 10 9 "+
|
||||
"Deliveries 21 18 "+
|
||||
"Failures 5 4")
|
||||
assert.Contains(t, pane, "Last 10 minutes Last 24 hours "+
|
||||
"Events 7 8 "+
|
||||
"Failures 1 3 "+
|
||||
"Failure percentage 50.0% 75.0%")
|
||||
}
|
||||
|
||||
// TestWebhookStats_EveryFigureAcrossRetentionPrune checks every figure
|
||||
// the statistics pane shows for the history seedStatsHistory builds,
|
||||
// and each target's totals and recent figures, before and after the
|
||||
// real retention reaper removes the oldest event.
|
||||
func TestWebhookStats_EveryFigureAcrossRetentionPrune(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
log *logger.Logger
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr, &log)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
hist := seedStatsHistory(t, h, sess, db, dbMgr)
|
||||
first, second := hist.first, hist.second
|
||||
|
||||
stats := h.WebhookStatsForTest(hist.webhook.ID)
|
||||
require.NotNil(t, stats)
|
||||
|
||||
assert.Equal(t, 12, stats.Entrypoints)
|
||||
assert.Equal(t, 11, stats.ActiveEntrypoints)
|
||||
assert.Equal(t, 6, stats.Targets)
|
||||
assert.Equal(t, 2, stats.ActiveTargets)
|
||||
assert.Equal(t, handlers.Counts{Events: 10, Deliveries: 21, Failures: 5},
|
||||
stats.Lifetime)
|
||||
assert.Equal(t, stats.Lifetime, stats.WithinRetention)
|
||||
assert.Equal(t, int64(15), stats.InProgress)
|
||||
require.NotNil(t, stats.LastEventAt)
|
||||
assert.True(t, hist.newest.CreatedAt.Equal(*stats.LastEventAt))
|
||||
assert.Equal(t, handlers.RecentWindow{
|
||||
Events: 7, Delivered: 1, Failed: 1,
|
||||
}, stats.Last10Minutes)
|
||||
assert.Equal(t, handlers.RecentWindow{
|
||||
Events: 8, Delivered: 1, Failed: 3,
|
||||
}, stats.Last24Hours)
|
||||
assert.Equal(t, "50.0%", stats.Last10Minutes.FailurePercent())
|
||||
assert.Equal(t, "75.0%", stats.Last24Hours.FailurePercent())
|
||||
|
||||
assertStatsTargets(t, hist)
|
||||
|
||||
// Retention removes the oldest event with its three deliveries:
|
||||
// the first target's failed one and the pending replay, and the
|
||||
// second target's pending one.
|
||||
statsPrune(t, db, dbMgr, log, hist.webhookDB)
|
||||
|
||||
after := h.WebhookStatsForTest(hist.webhook.ID)
|
||||
require.NotNil(t, after)
|
||||
|
||||
assert.Equal(t, stats.Lifetime, after.Lifetime)
|
||||
assert.Equal(t, handlers.Counts{Events: 9, Deliveries: 18, Failures: 4},
|
||||
after.WithinRetention)
|
||||
assert.Equal(t, int64(13), after.InProgress)
|
||||
assert.Equal(t, stats.LastEventAt, after.LastEventAt)
|
||||
assert.Equal(t, stats.Last10Minutes, after.Last10Minutes)
|
||||
assert.Equal(t, stats.Last24Hours, after.Last24Hours)
|
||||
|
||||
assert.Equal(t, map[string]database.TargetTotals{
|
||||
first: {
|
||||
TargetID: first, Deliveries: 11, Delivered: 1, Failed: 3,
|
||||
DeliveriesRemoved: 2, FailedRemoved: 1,
|
||||
},
|
||||
second: {
|
||||
TargetID: second, Deliveries: 10, Failed: 2,
|
||||
DeliveriesRemoved: 1,
|
||||
},
|
||||
}, statsTargetTotals(t, hist.webhookDB))
|
||||
|
||||
assertStatsPaneAfterPrune(t, h, sess, hist)
|
||||
}
|
||||
|
||||
// TestWebhookStats_LastEventSurvivesPruningEveryEvent checks that once
|
||||
// retention has removed every event, the pane still shows when the last
|
||||
// one arrived rather than "none".
|
||||
func TestWebhookStats_LastEventSurvivesPruningEveryEvent(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
log *logger.Logger
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr, &log)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
wh := &database.Webhook{
|
||||
UserID: deleteTestUserID, Name: "pruned", RetentionDays: 1,
|
||||
}
|
||||
require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
|
||||
|
||||
path := statsEntrypoint(t, db, wh.ID, true)
|
||||
require.Equal(t, http.StatusOK,
|
||||
postReceiver(t, receiverRouter(h), path))
|
||||
|
||||
webhookDB, err := dbMgr.GetDB(wh.ID)
|
||||
require.NoError(t, err)
|
||||
|
||||
events := listEvents(t, webhookDB)
|
||||
require.Len(t, events, 1)
|
||||
|
||||
arrived := events[0].CreatedAt
|
||||
|
||||
statsAge(t, webhookDB, events[0].ID, time.Now().Add(-50*time.Hour))
|
||||
statsPrune(t, db, dbMgr, log, webhookDB)
|
||||
require.Empty(t, listEvents(t, webhookDB))
|
||||
|
||||
stats := h.WebhookStatsForTest(wh.ID)
|
||||
require.NotNil(t, stats)
|
||||
require.NotNil(t, stats.LastEventAt)
|
||||
assert.True(t, arrived.Equal(*stats.LastEventAt))
|
||||
|
||||
pane := statsPane(t, renderSourceDetailPage(t, h, sess, wh.ID))
|
||||
assert.Contains(t, pane,
|
||||
"Last event "+arrived.UTC().Format("2006-01-02 15:04:05 UTC"))
|
||||
}
|
||||
|
||||
// TestWebhookStats_LastEventInUTC checks that the pane shows when the
|
||||
// last event arrived in UTC, as the event list does, when the time was
|
||||
// stored in another zone, as it is on a host whose local time is not
|
||||
// UTC.
|
||||
func TestWebhookStats_LastEventInUTC(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
wh := seedWebhook(t, db)
|
||||
|
||||
webhookDB, err := dbMgr.GetDB(wh.ID)
|
||||
require.NoError(t, err)
|
||||
|
||||
arrived := time.Date(2026, time.March, 4, 22, 30, 0, 0,
|
||||
time.FixedZone("EST", -5*60*60))
|
||||
require.NoError(t, database.AddEventTotals(webhookDB,
|
||||
database.EventTotals{Events: 1, LastEventAt: &arrived}))
|
||||
|
||||
pane := statsPane(t, renderSourceDetailPage(t, h, sess, wh.ID))
|
||||
assert.Contains(t, pane, "Last event 2026-03-05 03:30:00 UTC")
|
||||
}
|
||||
|
||||
// TestWebhookStats_PaneShowsRetentionPeriod checks that the statistics
|
||||
// pane itself, not only the line at the foot of the page, shows the
|
||||
// webhook's retention period, for a finite one and for forever.
|
||||
func TestWebhookStats_PaneShowsRetentionPeriod(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
tests := []struct {
|
||||
retentionDays int
|
||||
want string
|
||||
}{
|
||||
{30, "30 days"},
|
||||
{database.RetentionForeverDays, "forever"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
wh := &database.Webhook{
|
||||
UserID: deleteTestUserID,
|
||||
Name: "retention",
|
||||
RetentionDays: tt.retentionDays,
|
||||
}
|
||||
require.NoError(t,
|
||||
db.DB().Omit(clause.Associations).Create(wh).Error)
|
||||
|
||||
pane := statsPane(t, renderSourceDetailPage(t, h, sess, wh.ID))
|
||||
assert.Contains(t, pane, "Retention "+tt.want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestWebhookStats_WebhookWithNoEvents covers a webhook whose event
|
||||
// database has never been opened: every count is zero, the
|
||||
// percentages are a dash, and showing the page does not create the
|
||||
// database.
|
||||
func TestWebhookStats_WebhookWithNoEvents(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
wh := seedWebhook(t, db)
|
||||
|
||||
assert.Equal(t, &handlers.WebhookStats{}, h.WebhookStatsForTest(wh.ID))
|
||||
assert.Equal(t, "—", handlers.RecentWindow{}.FailurePercent())
|
||||
|
||||
pane := statsPane(t, renderSourceDetailPage(t, h, sess, wh.ID))
|
||||
assert.Contains(t, pane, "Last event none")
|
||||
assert.Contains(t, pane, "Failure percentage — —")
|
||||
assert.False(t, dbMgr.DBExists(wh.ID))
|
||||
}
|
||||
|
||||
// TestRecentWindow_FailurePercent pins the percentage: failed
|
||||
// deliveries out of all that finished in the window.
|
||||
func TestRecentWindow_FailurePercent(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
window handlers.RecentWindow
|
||||
want string
|
||||
}{
|
||||
{handlers.RecentWindow{}, "—"},
|
||||
{handlers.RecentWindow{Events: 4}, "—"},
|
||||
{handlers.RecentWindow{Delivered: 3, Failed: 1}, "25.0%"},
|
||||
{handlers.RecentWindow{Failed: 2}, "100.0%"},
|
||||
{handlers.RecentWindow{Delivered: 2}, "0.0%"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
assert.Equal(t, tt.want, tt.window.FailurePercent(), tt.window)
|
||||
}
|
||||
}
|
||||
@@ -201,7 +201,7 @@ func TestTruncate_LeavesShortValuesAlone(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, s := range []string{
|
||||
"", "GET", "/hook/abc/edit", "Mozilla/5.0 (X11)",
|
||||
"", "GET", "/source/abc/edit", "Mozilla/5.0 (X11)",
|
||||
} {
|
||||
assert.Equal(t, s, logfield.Truncate(s, budget))
|
||||
}
|
||||
|
||||
+27
-20
@@ -3,17 +3,18 @@
|
||||
// deliveries are attempted, how they end, how long they take, how
|
||||
// deep the queues are, and how many circuit breakers are open.
|
||||
//
|
||||
// The inbound HTTP metrics come from the go-http-metrics recorder in
|
||||
// internal/middleware and land on prometheus.DefaultRegisterer. These
|
||||
// collectors register there too, so both surfaces are gathered by the
|
||||
// one promhttp handler mounted on the authenticated /metrics route.
|
||||
// It also builds the registry the authenticated /metrics route
|
||||
// serves. In production, these collectors, the inbound HTTP metrics
|
||||
// recorded in internal/middleware, and the Go runtime and process
|
||||
// collectors all register on that one registry, never on Prometheus's
|
||||
// global default.
|
||||
package metrics
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"github.com/prometheus/client_golang/prometheus/collectors"
|
||||
"github.com/prometheus/client_golang/prometheus/promauto"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
)
|
||||
@@ -57,25 +58,31 @@ var knownTargetTypes = []database.TargetType{
|
||||
database.TargetTypeSlack,
|
||||
}
|
||||
|
||||
// defaultSet is the process-wide metric set, registered on the same
|
||||
// registry the HTTP middleware and the /metrics handler already use.
|
||||
// It is built on first use rather than in an init so that a test
|
||||
// binary that never touches metrics never registers them.
|
||||
// NewRegistry returns the registry /metrics serves, carrying the Go
|
||||
// runtime and process collectors that Prometheus's global default
|
||||
// registry carries, so the go_* and process_* series stay in the
|
||||
// scrape.
|
||||
//
|
||||
//nolint:gochecknoglobals // one process-wide registration, by design
|
||||
var defaultSet = sync.OnceValue(func() *Set {
|
||||
return New(prometheus.DefaultRegisterer)
|
||||
})
|
||||
// A registry of its own, rather than the global default, is what lets
|
||||
// two dependency graphs in one process — two tests, say — each
|
||||
// register their collectors without the second registration
|
||||
// panicking.
|
||||
func NewRegistry() *prometheus.Registry {
|
||||
reg := prometheus.NewRegistry()
|
||||
reg.MustRegister(
|
||||
collectors.NewGoCollector(),
|
||||
collectors.NewProcessCollector(
|
||||
collectors.ProcessCollectorOpts{},
|
||||
),
|
||||
)
|
||||
|
||||
// Default returns the process-wide metric set.
|
||||
func Default() *Set {
|
||||
return defaultSet()
|
||||
return reg
|
||||
}
|
||||
|
||||
// Set is one registered group of webhooker's delivery collectors.
|
||||
// Production uses the single Default set; tests build their own
|
||||
// against a private registry so assertions are not disturbed by
|
||||
// deliveries other tests are making concurrently.
|
||||
// Production builds one on the registry /metrics serves; tests build
|
||||
// one on a registry of their own so they can gather what their own
|
||||
// deliveries recorded.
|
||||
type Set struct {
|
||||
eventsReceived prometheus.Counter
|
||||
deliveryAttempts *prometheus.CounterVec
|
||||
@@ -93,7 +100,7 @@ type Set struct {
|
||||
// New registers a full set of delivery collectors on reg and returns
|
||||
// it. It panics if reg already holds them, which is the intended
|
||||
// behaviour for a duplicate registration.
|
||||
func New(reg prometheus.Registerer) *Set {
|
||||
func New(reg *prometheus.Registry) *Set {
|
||||
factory := promauto.With(reg)
|
||||
|
||||
s := &Set{
|
||||
|
||||
@@ -119,7 +119,7 @@ func accessLogRouter(m *middleware.Middleware) *chi.Mux {
|
||||
)
|
||||
|
||||
router.HandleFunc(
|
||||
"/h/{uuid}",
|
||||
"/webhook/{uuid}",
|
||||
func(w http.ResponseWriter, r *http.Request) {
|
||||
// Stands in for the real handler: an unknown entrypoint
|
||||
// UUID 404s, a known one succeeds.
|
||||
@@ -271,11 +271,11 @@ func TestAccessLog_InventedReceiverPathsLogRoutePattern(t *testing.T) {
|
||||
assertFloodIsBounded(
|
||||
t,
|
||||
func(i int) string {
|
||||
return "/h/" + attackerMarker +
|
||||
return "/webhook/" + attackerMarker +
|
||||
strings.Repeat("x", i) + "?q=" + attackerMarker
|
||||
},
|
||||
http.StatusNotFound,
|
||||
"/h/{uuid}",
|
||||
"/webhook/{uuid}",
|
||||
)
|
||||
}
|
||||
|
||||
@@ -346,10 +346,10 @@ type sizeCase struct {
|
||||
func lineSizeCases() map[string]sizeCase {
|
||||
cases := map[string]sizeCase{
|
||||
"oversized path segment": {
|
||||
target: "/h/" + attackerMarker +
|
||||
target: "/webhook/" + attackerMarker +
|
||||
strings.Repeat("x", oversizedSegmentBytes),
|
||||
wantStatus: http.StatusNotFound,
|
||||
wantURL: "/h/{uuid}",
|
||||
wantURL: "/webhook/{uuid}",
|
||||
bound: maxLineBytes,
|
||||
},
|
||||
// /.well-known/healthcheck answers 200 to anyone and has no
|
||||
@@ -605,14 +605,14 @@ func TestAccessLog_SuccessKeepsConcretePathAndRedactsQuery(
|
||||
router := accessLogRouter(m)
|
||||
|
||||
assert.Equal(
|
||||
t, http.StatusOK, get(t, router, "/h/known?src=ci"),
|
||||
t, http.StatusOK, get(t, router, "/webhook/known?src=ci"),
|
||||
)
|
||||
|
||||
// The path resolved against a stored entrypoint, so it stays. The
|
||||
// query never does: see TestAccessLog_UnauthenticatedSuccess...
|
||||
entries := accessLogEntries(t, buf)
|
||||
require.Len(t, entries, 1)
|
||||
assert.Equal(t, "/h/known?(redacted)", entries[0]["url"])
|
||||
assert.Equal(t, "/webhook/known?(redacted)", entries[0]["url"])
|
||||
assert.NotContains(t, buf.String(), "src=ci")
|
||||
}
|
||||
|
||||
@@ -640,7 +640,7 @@ func TestAccessLog_RetainsEveryOtherField(t *testing.T) {
|
||||
assert.Equal(
|
||||
t,
|
||||
http.StatusNotFound,
|
||||
get(t, router, "/h/"+attackerMarker),
|
||||
get(t, router, "/webhook/"+attackerMarker),
|
||||
)
|
||||
|
||||
entries := accessLogEntries(t, buf)
|
||||
|
||||
@@ -41,7 +41,7 @@ func (m *Middleware) CSRF() func(http.Handler) http.Handler {
|
||||
// CSRF is registered ahead of RequireAuth on every route
|
||||
// group that uses it, so this WARN is reachable by an
|
||||
// unauthenticated client: a POST with no token to
|
||||
// /hook/<any length of any text>/edit lands here. The
|
||||
// /source/<any length of any text>/edit lands here. The
|
||||
// method and path are capped against the same budgets as
|
||||
// the access log. remote_addr is set by net/http from the
|
||||
// accepted connection rather than by the client, and
|
||||
|
||||
@@ -10,8 +10,7 @@ import (
|
||||
|
||||
// MetricsMiddlewareForTest builds the metrics recording middleware
|
||||
// against a caller-supplied recorder, so a test can gather from its
|
||||
// own Prometheus registry rather than the process-wide default one
|
||||
// that Middleware.Metrics uses.
|
||||
// own Prometheus registry without building a whole Middleware.
|
||||
func MetricsMiddlewareForTest(
|
||||
rec httpmetrics.Recorder,
|
||||
) func(http.Handler) http.Handler {
|
||||
|
||||
@@ -383,7 +383,7 @@ func TestLogLines_ClientChosenPathDoesNotSizeTheLine(t *testing.T) {
|
||||
t, newHandler,
|
||||
)
|
||||
|
||||
path := "/hook/" +
|
||||
path := "/source/" +
|
||||
oversizedPathSegment(fill) + "/edit"
|
||||
|
||||
assert.Equal(
|
||||
@@ -434,7 +434,7 @@ func TestLoginThrottle_LogLineDoesNotTrackPathSize(t *testing.T) {
|
||||
req := httptest.NewRequestWithContext(
|
||||
context.Background(),
|
||||
http.MethodPost,
|
||||
"/hook/"+
|
||||
"/source/"+
|
||||
oversizedPathSegment(fill)+"/login",
|
||||
nil,
|
||||
)
|
||||
@@ -499,7 +499,7 @@ func TestMaxBodySize_FloodOfOversizePathsDoesNotGrowTheLog(
|
||||
http.StatusRequestEntityTooLarge,
|
||||
postOversize(
|
||||
h,
|
||||
"/hook/"+segment(i)+"/edit",
|
||||
"/source/"+segment(i)+"/edit",
|
||||
),
|
||||
)
|
||||
}
|
||||
|
||||
@@ -7,7 +7,6 @@ import (
|
||||
|
||||
"github.com/go-chi/chi"
|
||||
httpmetrics "github.com/slok/go-http-metrics/metrics"
|
||||
prommetrics "github.com/slok/go-http-metrics/metrics/prometheus"
|
||||
ghmm "github.com/slok/go-http-metrics/middleware"
|
||||
"github.com/slok/go-http-metrics/middleware/std"
|
||||
)
|
||||
@@ -40,7 +39,7 @@ const unmatchedMethod = unmatchedRoute
|
||||
//
|
||||
// The pattern is what bounds the label's domain to the routes the
|
||||
// service registers. The path does not bound it at all — every byte
|
||||
// after /h/ is client-chosen, so labelling by path lets any
|
||||
// after /webhook/ is client-chosen, so labelling by path lets any
|
||||
// unauthenticated client mint permanent series at will, and publishes
|
||||
// the entrypoint UUID (the receiver's only credential) in the scrape
|
||||
// while doing it.
|
||||
@@ -151,17 +150,17 @@ func (r boundedLabelRecorder) AddInflightRequests(
|
||||
|
||||
var _ httpmetrics.Recorder = boundedLabelRecorder{}
|
||||
|
||||
// Metrics returns middleware that records Prometheus HTTP metrics on
|
||||
// the default registry, which is the one the /metrics route gathers.
|
||||
// Metrics returns middleware that records Prometheus HTTP metrics
|
||||
// with the Middleware's one recorder, which New builds on the registry
|
||||
// the /metrics route serves and NewForTest on a registry of its own.
|
||||
// Every call reuses that recorder, so any number of routers can
|
||||
// install it.
|
||||
func (s *Middleware) Metrics() func(http.Handler) http.Handler {
|
||||
return metricsMiddleware(
|
||||
prommetrics.NewRecorder(prommetrics.Config{}),
|
||||
)
|
||||
return metricsMiddleware(s.metricsRecorder)
|
||||
}
|
||||
|
||||
// metricsMiddleware builds the recording middleware against a given
|
||||
// recorder, so tests can gather from a registry of their own instead
|
||||
// of the process-wide default.
|
||||
// recorder, so tests can gather from a registry of their own.
|
||||
func metricsMiddleware(
|
||||
rec httpmetrics.Recorder,
|
||||
) func(http.Handler) http.Handler {
|
||||
|
||||
@@ -50,7 +50,7 @@ func realMethods() []string {
|
||||
// dimension varying, so any series growth a probe produces is the
|
||||
// method label's and nothing else's.
|
||||
func methodProbePath() string {
|
||||
return "/h/" + uuid.NewString()
|
||||
return "/webhook/" + uuid.NewString()
|
||||
}
|
||||
|
||||
// inventedMethods returns n distinct RFC 9110 method tokens that no
|
||||
|
||||
@@ -28,7 +28,7 @@ const (
|
||||
|
||||
// receiverRoutePattern is the one handler label every receiver
|
||||
// request must produce, however the client varies the path.
|
||||
receiverRoutePattern = "/h/{uuid}"
|
||||
receiverRoutePattern = "/webhook/{uuid}"
|
||||
|
||||
// okRoute is a static route used to pin that the response-writer
|
||||
// interceptor still reports status and size after the handler id
|
||||
@@ -57,9 +57,8 @@ const (
|
||||
// Server.setupWebhookRoutes inside it. That ordering is the whole
|
||||
// defect, so a test that flattens it would prove nothing.
|
||||
//
|
||||
// The recorder writes to a registry of the test's own rather than the
|
||||
// process-wide default one, so each test observes only its own
|
||||
// traffic.
|
||||
// The recorder writes to a registry of the test's own, so each test
|
||||
// observes only its own traffic.
|
||||
func metricsTestRouter(
|
||||
t *testing.T,
|
||||
receiverLimit int,
|
||||
@@ -143,13 +142,13 @@ func drivePaths(
|
||||
return drive(t, h, probes)
|
||||
}
|
||||
|
||||
// receiverPaths returns n distinct /h/ paths, each naming a
|
||||
// receiverPaths returns n distinct /webhook/ paths, each naming a
|
||||
// fresh UUID exactly as an unauthenticated flood would.
|
||||
func receiverPaths(n int) []string {
|
||||
paths := make([]string, 0, n)
|
||||
|
||||
for range n {
|
||||
paths = append(paths, "/h/"+uuid.NewString())
|
||||
paths = append(paths, "/webhook/"+uuid.NewString())
|
||||
}
|
||||
|
||||
return paths
|
||||
@@ -220,7 +219,7 @@ func keys(set map[string]struct{}) []string {
|
||||
|
||||
// TestMetrics_DistinctReceiverPathsMintOneLabelSet is the direct
|
||||
// assertion the issue asks for: N requests to N distinct
|
||||
// /h/<uuid> paths must produce exactly ONE handler label, the
|
||||
// /webhook/<uuid> paths must produce exactly ONE handler label, the
|
||||
// route pattern. Before the fix this produced N of them.
|
||||
func TestMetrics_DistinctReceiverPathsMintOneLabelSet(t *testing.T) {
|
||||
t.Parallel()
|
||||
@@ -250,7 +249,7 @@ func TestMetrics_DistinctReceiverPathsMintOneLabelSet(t *testing.T) {
|
||||
// The scrape must not republish the UUIDs it was driven with.
|
||||
// They are the receiver's only credential.
|
||||
for _, p := range paths {
|
||||
id := strings.TrimPrefix(p, "/h/")
|
||||
id := strings.TrimPrefix(p, "/webhook/")
|
||||
for label := range labels {
|
||||
assert.NotContains(
|
||||
t, label, id,
|
||||
@@ -354,7 +353,7 @@ func TestMetrics_UnmatchedPathsCollapseToTheSentinel(t *testing.T) {
|
||||
if i%2 == 0 {
|
||||
paths = append(paths, "/"+id)
|
||||
} else {
|
||||
paths = append(paths, "/h/"+id+"/"+id)
|
||||
paths = append(paths, "/webhook/"+id+"/"+id)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -455,3 +454,29 @@ func TestMetrics_StatusAndSizeStillRecorded(t *testing.T) {
|
||||
"the interceptor must still count written bytes",
|
||||
)
|
||||
}
|
||||
|
||||
// TestMetrics_WorksOnNewForTestMiddleware pins that a Middleware built
|
||||
// by NewForTest has a recorder of its own: its Metrics() serves a
|
||||
// request instead of panicking, and a second one does not collide
|
||||
// with the first.
|
||||
func TestMetrics_WorksOnNewForTestMiddleware(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
log := slog.New(slog.DiscardHandler)
|
||||
cfg := &config.Config{Environment: "prod"}
|
||||
ok := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
_, _ = w.Write([]byte(okBody))
|
||||
})
|
||||
|
||||
for range 2 {
|
||||
h := middleware.NewForTest(log, cfg, nil).Metrics()(ok)
|
||||
|
||||
req := httptest.NewRequestWithContext(
|
||||
t.Context(), http.MethodGet, okRoute, nil,
|
||||
)
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, req)
|
||||
|
||||
assert.Equal(t, http.StatusOK, w.Code)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -6,7 +6,6 @@ import (
|
||||
"log/slog"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
@@ -14,6 +13,9 @@ import (
|
||||
"github.com/go-chi/chi"
|
||||
"github.com/go-chi/chi/middleware"
|
||||
"github.com/go-chi/cors"
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
httpmetrics "github.com/slok/go-http-metrics/metrics"
|
||||
prommetrics "github.com/slok/go-http-metrics/metrics/prometheus"
|
||||
"go.uber.org/fx"
|
||||
"sneak.berlin/go/webhooker/internal/config"
|
||||
"sneak.berlin/go/webhooker/internal/globals"
|
||||
@@ -149,10 +151,11 @@ const (
|
||||
type MiddlewareParams struct {
|
||||
fx.In
|
||||
|
||||
Logger *logger.Logger
|
||||
Globals *globals.Globals
|
||||
Config *config.Config
|
||||
Session *session.Session
|
||||
Logger *logger.Logger
|
||||
Globals *globals.Globals
|
||||
Config *config.Config
|
||||
Session *session.Session
|
||||
Registry *prometheus.Registry
|
||||
}
|
||||
|
||||
// Middleware provides HTTP middleware for logging, CORS, auth, and
|
||||
@@ -162,6 +165,14 @@ type Middleware struct {
|
||||
params *MiddlewareParams
|
||||
session *session.Session
|
||||
|
||||
// metricsRecorder records the inbound HTTP metrics. New builds
|
||||
// it on the registry /metrics serves, NewForTest on a registry
|
||||
// of its own. Either way it is built once per Middleware and
|
||||
// Metrics reuses it, because building it registers its
|
||||
// collectors, and a second registration on the same registry
|
||||
// panics.
|
||||
metricsRecorder httpmetrics.Recorder
|
||||
|
||||
// loginGuard counts failed credential verifications and bounds
|
||||
// concurrent password hashing. It is built on first use so that
|
||||
// every construction path gets one; see guard().
|
||||
@@ -180,6 +191,9 @@ func New(
|
||||
s.params = ¶ms
|
||||
s.log = params.Logger.Get()
|
||||
s.session = params.Session
|
||||
s.metricsRecorder = prommetrics.NewRecorder(
|
||||
prommetrics.Config{Registry: params.Registry},
|
||||
)
|
||||
|
||||
return s, nil
|
||||
}
|
||||
@@ -258,7 +272,7 @@ func concreteLogURL(r *http.Request) string {
|
||||
//
|
||||
// 3xx and 4xx responses get the chi route pattern instead. Those are
|
||||
// the outcomes an unauthenticated client drives for free: 404 or 429
|
||||
// on any invented /h/ path, 303 to the login page on any
|
||||
// on any invented /webhook/ path, 303 to the login page on any
|
||||
// invented /user/ path. Logging the concrete URL there lets a flood
|
||||
// write attacker-chosen text, of attacker-chosen length, into the
|
||||
// operator's log at one line per request. The pattern comes from the
|
||||
@@ -367,30 +381,6 @@ func (s *Middleware) CORS() func(http.Handler) http.Handler {
|
||||
}
|
||||
}
|
||||
|
||||
// NextParam is the query parameter on the login redirect, and the
|
||||
// login form field, that holds the page to return to after login.
|
||||
const NextParam = "next"
|
||||
|
||||
// MaxNextBytes bounds the NextParam value. The login page writes it
|
||||
// into its form, and every page is rendered into a buffer first, so
|
||||
// without a bound a request would choose the size of that buffer.
|
||||
const MaxNextBytes = 2048
|
||||
|
||||
// loginURL is the login page RequireAuth redirects to. A GET carries
|
||||
// its own path and query in NextParam so that logging in returns to
|
||||
// it, unless they are longer than MaxNextBytes; loginDestination in
|
||||
// the handlers package checks whether that value is safe to follow.
|
||||
// Other methods carry nothing, since a redirect cannot repeat them.
|
||||
func loginURL(r *http.Request) string {
|
||||
next := r.URL.RequestURI()
|
||||
|
||||
if r.Method != http.MethodGet || len(next) > MaxNextBytes {
|
||||
return "/pages/login"
|
||||
}
|
||||
|
||||
return "/pages/login?" + url.Values{NextParam: {next}}.Encode()
|
||||
}
|
||||
|
||||
// RequireAuth returns middleware that checks for a valid session.
|
||||
// Unauthenticated users are redirected to the login page.
|
||||
func (s *Middleware) RequireAuth() func(http.Handler) http.Handler {
|
||||
@@ -406,7 +396,7 @@ func (s *Middleware) RequireAuth() func(http.Handler) http.Handler {
|
||||
"error", err,
|
||||
)
|
||||
http.Redirect(
|
||||
w, r, loginURL(r), http.StatusSeeOther,
|
||||
w, r, "/pages/login", http.StatusSeeOther,
|
||||
)
|
||||
|
||||
return
|
||||
@@ -434,7 +424,7 @@ func (s *Middleware) RequireAuth() func(http.Handler) http.Handler {
|
||||
),
|
||||
)
|
||||
http.Redirect(
|
||||
w, r, loginURL(r), http.StatusSeeOther,
|
||||
w, r, "/pages/login", http.StatusSeeOther,
|
||||
)
|
||||
|
||||
return
|
||||
@@ -585,7 +575,7 @@ func (s *Middleware) MaxBodySize(
|
||||
// internal/server/routes.go), so an
|
||||
// unauthenticated client reaches it with a path
|
||||
// of its own choosing and its own length —
|
||||
// POST /hook/<8 KB>/edit with an oversize
|
||||
// POST /source/<8 KB>/edit with an oversize
|
||||
// declared Content-Length costs nothing to
|
||||
// send. At WARN, on by default, that is a
|
||||
// write into the operator's log sized by the
|
||||
|
||||
@@ -338,76 +338,6 @@ func TestRequireAuth_NoSession_RedirectsToLogin(t *testing.T) {
|
||||
"unauthenticated request",
|
||||
)
|
||||
assert.Equal(t, http.StatusSeeOther, w.Code)
|
||||
assert.Equal(
|
||||
t, "/pages/login?next=%2Fdashboard", w.Header().Get("Location"),
|
||||
)
|
||||
}
|
||||
|
||||
// TestRequireAuth_LoginRedirectCarriesOnlyAGet pins what the login
|
||||
// redirect carries: a GET's path and query, so logging in can return
|
||||
// there, and nothing for a POST, which a redirect cannot repeat.
|
||||
func TestRequireAuth_LoginRedirectCarriesOnlyAGet(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
m, _ := testMiddleware(t, config.EnvironmentDev)
|
||||
|
||||
handler := m.RequireAuth()(http.HandlerFunc(
|
||||
func(_ http.ResponseWriter, _ *http.Request) {},
|
||||
))
|
||||
|
||||
get := httptest.NewRequestWithContext(
|
||||
context.Background(),
|
||||
http.MethodGet, "/hook/abc/events?page=2", nil,
|
||||
)
|
||||
w := httptest.NewRecorder()
|
||||
handler.ServeHTTP(w, get)
|
||||
|
||||
assert.Equal(
|
||||
t, "/pages/login?next=%2Fhook%2Fabc%2Fevents%3Fpage%3D2",
|
||||
w.Header().Get("Location"),
|
||||
)
|
||||
|
||||
post := httptest.NewRequestWithContext(
|
||||
context.Background(),
|
||||
http.MethodPost, "/hook/abc/delete", nil,
|
||||
)
|
||||
w = httptest.NewRecorder()
|
||||
handler.ServeHTTP(w, post)
|
||||
|
||||
assert.Equal(t, "/pages/login", w.Header().Get("Location"))
|
||||
}
|
||||
|
||||
// TestRequireAuth_LoginRedirectLeavesOutALongURL: a GET whose path
|
||||
// and query are longer than the login page accepts goes to the plain
|
||||
// login page, so a long URL does not make the redirect long.
|
||||
func TestRequireAuth_LoginRedirectLeavesOutALongURL(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
m, _ := testMiddleware(t, config.EnvironmentDev)
|
||||
|
||||
handler := m.RequireAuth()(http.HandlerFunc(
|
||||
func(_ http.ResponseWriter, _ *http.Request) {},
|
||||
))
|
||||
|
||||
atLimit := "/" + strings.Repeat("a", middleware.MaxNextBytes-1)
|
||||
|
||||
get := httptest.NewRequestWithContext(
|
||||
context.Background(), http.MethodGet, atLimit, nil,
|
||||
)
|
||||
w := httptest.NewRecorder()
|
||||
handler.ServeHTTP(w, get)
|
||||
|
||||
assert.Equal(
|
||||
t, "/pages/login?next=%2F"+atLimit[1:],
|
||||
w.Header().Get("Location"),
|
||||
)
|
||||
|
||||
get = httptest.NewRequestWithContext(
|
||||
context.Background(), http.MethodGet, atLimit+"a", nil,
|
||||
)
|
||||
w = httptest.NewRecorder()
|
||||
handler.ServeHTTP(w, get)
|
||||
|
||||
assert.Equal(t, "/pages/login", w.Header().Get("Location"))
|
||||
}
|
||||
|
||||
@@ -513,9 +443,7 @@ func TestRequireAuth_UnauthenticatedSession_RedirectsToLogin(
|
||||
"unauthenticated session",
|
||||
)
|
||||
assert.Equal(t, http.StatusSeeOther, w.Code)
|
||||
assert.Equal(
|
||||
t, "/pages/login?next=%2Fdashboard", w.Header().Get("Location"),
|
||||
)
|
||||
assert.Equal(t, "/pages/login", w.Header().Get("Location"))
|
||||
}
|
||||
|
||||
// --- RequireAuth Session Expiry Tests ---
|
||||
@@ -613,9 +541,7 @@ func TestRequireAuth_IdleExpiredSession_RedirectsToLogin(
|
||||
"handler should not run for an idle-expired session",
|
||||
)
|
||||
assert.Equal(t, http.StatusSeeOther, w.Code)
|
||||
assert.Equal(
|
||||
t, "/pages/login?next=%2Fdashboard", w.Header().Get("Location"),
|
||||
)
|
||||
assert.Equal(t, "/pages/login", w.Header().Get("Location"))
|
||||
assert.Empty(
|
||||
t, sessionCookies(w),
|
||||
"an expired session must not be refreshed",
|
||||
@@ -714,7 +640,7 @@ func TestNoCache_SetsHeaders(t *testing.T) {
|
||||
|
||||
req := httptest.NewRequestWithContext(
|
||||
context.Background(),
|
||||
http.MethodGet, "/hooks", nil,
|
||||
http.MethodGet, "/sources", nil,
|
||||
)
|
||||
w := httptest.NewRecorder()
|
||||
|
||||
|
||||
@@ -63,7 +63,7 @@ const (
|
||||
|
||||
// receiverAggregateMultiplier scales the configured
|
||||
// per-entrypoint receiver limit into the aggregate limit one
|
||||
// client IP may spend across the whole /h/* route. Ten
|
||||
// client IP may spend across the whole /webhook/* route. Ten
|
||||
// entrypoints' worth lets a single sender address drive several
|
||||
// entrypoints at their full rate, while still capping what one
|
||||
// address costs the unauthenticated receiver.
|
||||
@@ -389,7 +389,7 @@ func (m *Middleware) postRateLimit(
|
||||
// It is Config.ReceiverRateLimit requests per minute.
|
||||
//
|
||||
// That limit alone bounds nothing in aggregate. The route pattern
|
||||
// /h/{uuid} matches any single segment, so a client that
|
||||
// /webhook/{uuid} matches any single segment, so a client that
|
||||
// invents a fresh path per request mints a fresh bucket per request
|
||||
// and never refills one — and every such request still reaches the
|
||||
// handler's entrypoint lookup before it 404s. The outer limit is
|
||||
|
||||
@@ -275,7 +275,7 @@ func TestReceiverRateLimit_LimitsPerIPAndPath(t *testing.T) {
|
||||
// pass.
|
||||
for i := range limit {
|
||||
w := receiverPost(
|
||||
handler, "9.9.9.9:1234", "/h/uuid-a",
|
||||
handler, "9.9.9.9:1234", "/webhook/uuid-a",
|
||||
)
|
||||
assert.Equal(
|
||||
t, http.StatusOK, w.Code,
|
||||
@@ -286,7 +286,7 @@ func TestReceiverRateLimit_LimitsPerIPAndPath(t *testing.T) {
|
||||
// The next request over the limit is rejected with a 429
|
||||
// carrying a Retry-After header.
|
||||
w := receiverPost(
|
||||
handler, "9.9.9.9:1234", "/h/uuid-a",
|
||||
handler, "9.9.9.9:1234", "/webhook/uuid-a",
|
||||
)
|
||||
assert.Equal(t, http.StatusTooManyRequests, w.Code)
|
||||
assert.NotEmpty(
|
||||
@@ -296,7 +296,7 @@ func TestReceiverRateLimit_LimitsPerIPAndPath(t *testing.T) {
|
||||
|
||||
// The same IP is not limited on a different entrypoint.
|
||||
w = receiverPost(
|
||||
handler, "9.9.9.9:1234", "/h/uuid-b",
|
||||
handler, "9.9.9.9:1234", "/webhook/uuid-b",
|
||||
)
|
||||
assert.Equal(
|
||||
t, http.StatusOK, w.Code,
|
||||
@@ -305,7 +305,7 @@ func TestReceiverRateLimit_LimitsPerIPAndPath(t *testing.T) {
|
||||
|
||||
// A different IP is not limited on the same entrypoint.
|
||||
w = receiverPost(
|
||||
handler, "8.8.8.8:1234", "/h/uuid-a",
|
||||
handler, "8.8.8.8:1234", "/webhook/uuid-a",
|
||||
)
|
||||
assert.Equal(
|
||||
t, http.StatusOK, w.Code,
|
||||
@@ -322,7 +322,7 @@ func TestReceiverRateLimit_CountsEveryMethod(t *testing.T) {
|
||||
const (
|
||||
limit = 2
|
||||
ip = "7.7.7.7:1234"
|
||||
path = "/h/uuid-c"
|
||||
path = "/webhook/uuid-c"
|
||||
)
|
||||
|
||||
handler := receiverLimitedHandler(t, limit)
|
||||
@@ -715,7 +715,7 @@ func TestReceiverRateLimit_LimitsAggregateAcrossInventedPaths(
|
||||
// none of them shares a per-entrypoint bucket with another.
|
||||
for i := range aggregate {
|
||||
w := receiverPost(
|
||||
handler, ip, fmt.Sprintf("/h/invented-%d", i),
|
||||
handler, ip, fmt.Sprintf("/webhook/invented-%d", i),
|
||||
)
|
||||
assert.Equal(
|
||||
t, http.StatusOK, w.Code,
|
||||
@@ -724,17 +724,17 @@ func TestReceiverRateLimit_LimitsAggregateAcrossInventedPaths(
|
||||
}
|
||||
|
||||
w := receiverPost(
|
||||
handler, ip, fmt.Sprintf("/h/invented-%d", aggregate),
|
||||
handler, ip, fmt.Sprintf("/webhook/invented-%d", aggregate),
|
||||
)
|
||||
assert.Equal(
|
||||
t, http.StatusTooManyRequests, w.Code,
|
||||
"a client must not be able to raise its aggregate rate "+
|
||||
"against /h/* by varying the path",
|
||||
"against /webhook/* by varying the path",
|
||||
)
|
||||
|
||||
// The aggregate limit is still per client IP: exhausting one
|
||||
// address must not throttle another.
|
||||
w = receiverPost(handler, "6.6.6.7:1234", "/h/invented-0")
|
||||
w = receiverPost(handler, "6.6.6.7:1234", "/webhook/invented-0")
|
||||
assert.Equal(
|
||||
t, http.StatusOK, w.Code,
|
||||
"a different client IP must not be affected",
|
||||
@@ -771,7 +771,7 @@ func TestReceiverRateLimit_RejectedRequestsCountTowardAggregate(
|
||||
// limit requests are served; the rest are rejected by the
|
||||
// per-entrypoint limiter but still count against the aggregate.
|
||||
for i := range aggregate {
|
||||
w := receiverPost(handler, ip, "/h/exhausted")
|
||||
w := receiverPost(handler, ip, "/webhook/exhausted")
|
||||
|
||||
want := http.StatusTooManyRequests
|
||||
if i < limit {
|
||||
@@ -784,7 +784,7 @@ func TestReceiverRateLimit_RejectedRequestsCountTowardAggregate(
|
||||
)
|
||||
}
|
||||
|
||||
w := receiverPost(handler, ip, "/h/never-used")
|
||||
w := receiverPost(handler, ip, "/webhook/never-used")
|
||||
assert.Equal(
|
||||
t, http.StatusTooManyRequests, w.Code,
|
||||
"requests rejected per entrypoint must still count "+
|
||||
@@ -823,7 +823,7 @@ func TestReceiverRateLimit_IgnoresForwardedFromUntrustedPeer(
|
||||
const (
|
||||
limit = 3
|
||||
peer = "203.0.113.10:44444"
|
||||
path = "/h/uuid-d"
|
||||
path = "/webhook/uuid-d"
|
||||
)
|
||||
|
||||
handler := receiverLimitedHandler(t, limit)
|
||||
|
||||
@@ -3,12 +3,17 @@ package middleware
|
||||
import (
|
||||
"log/slog"
|
||||
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
prommetrics "github.com/slok/go-http-metrics/metrics/prometheus"
|
||||
"sneak.berlin/go/webhooker/internal/config"
|
||||
"sneak.berlin/go/webhooker/internal/session"
|
||||
)
|
||||
|
||||
// NewForTest creates a Middleware with the minimum dependencies
|
||||
// needed for testing. This bypasses the fx lifecycle.
|
||||
//
|
||||
// Its metrics recorder writes to a fresh registry of its own, so
|
||||
// Metrics() works on it and two of them never collide.
|
||||
func NewForTest(
|
||||
log *slog.Logger,
|
||||
cfg *config.Config,
|
||||
@@ -20,5 +25,8 @@ func NewForTest(
|
||||
Config: cfg,
|
||||
},
|
||||
session: sess,
|
||||
metricsRecorder: prommetrics.NewRecorder(
|
||||
prommetrics.Config{Registry: prometheus.NewRegistry()},
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -24,6 +24,7 @@ import (
|
||||
"sneak.berlin/go/webhooker/internal/handlers"
|
||||
"sneak.berlin/go/webhooker/internal/healthcheck"
|
||||
"sneak.berlin/go/webhooker/internal/logger"
|
||||
"sneak.berlin/go/webhooker/internal/metrics"
|
||||
"sneak.berlin/go/webhooker/internal/middleware"
|
||||
"sneak.berlin/go/webhooker/internal/resetpw"
|
||||
"sneak.berlin/go/webhooker/internal/session"
|
||||
@@ -140,7 +141,7 @@ func (n *noopEvictor) EvictWebhook(string) {}
|
||||
// and the database, exactly as internal/handlers builds them.
|
||||
//
|
||||
// One application per test function, not per case: every start that
|
||||
// finds no account seeds one with an Argon2id hash, and this package's
|
||||
// finds no account seeds one at 64 MB of Argon2id, and this package's
|
||||
// budget is not the place to spend that repeatedly.
|
||||
func newServerApp(
|
||||
t *testing.T, dir string,
|
||||
@@ -163,6 +164,8 @@ func newServerApp(
|
||||
session.New,
|
||||
func() delivery.Notifier { return &noopNotifier{} },
|
||||
func() delivery.WebhookEvictor { return &noopEvictor{} },
|
||||
metrics.NewRegistry,
|
||||
metrics.New,
|
||||
middleware.New,
|
||||
delivery.NewGuard,
|
||||
handlers.New,
|
||||
|
||||
@@ -7,7 +7,6 @@ import (
|
||||
sentryhttp "github.com/getsentry/sentry-go/http"
|
||||
"github.com/go-chi/chi"
|
||||
"github.com/go-chi/chi/middleware"
|
||||
"github.com/prometheus/client_golang/prometheus/promhttp"
|
||||
"sneak.berlin/go/webhooker/static"
|
||||
)
|
||||
|
||||
@@ -130,12 +129,7 @@ func (s *Server) setupRoutes() {
|
||||
if s.params.Config.MetricsAuthEnabled() {
|
||||
s.router.Group(func(r chi.Router) {
|
||||
r.Use(s.mw.MetricsAuth())
|
||||
r.Get(
|
||||
"/metrics",
|
||||
http.HandlerFunc(
|
||||
promhttp.Handler().ServeHTTP,
|
||||
),
|
||||
)
|
||||
r.Get("/metrics", s.h.HandleMetrics())
|
||||
})
|
||||
}
|
||||
|
||||
@@ -183,7 +177,7 @@ func (s *Server) setupUserRoutes() {
|
||||
}
|
||||
|
||||
func (s *Server) setupSourceRoutes() {
|
||||
s.router.Route("/hooks", func(r chi.Router) {
|
||||
s.router.Route("/sources", func(r chi.Router) {
|
||||
// MaxBodySize precedes CSRF and RequireAuth deliberately;
|
||||
// see maxFormBodySize for why, and for what it costs.
|
||||
r.Use(s.mw.MaxBodySize(maxFormBodySize))
|
||||
@@ -195,7 +189,7 @@ func (s *Server) setupSourceRoutes() {
|
||||
r.Post("/new", s.h.HandleSourceCreateSubmit())
|
||||
})
|
||||
|
||||
s.router.Route("/hook/{sourceID}", func(r chi.Router) {
|
||||
s.router.Route("/source/{sourceID}", func(r chi.Router) {
|
||||
// MaxBodySize precedes CSRF and RequireAuth deliberately;
|
||||
// see maxFormBodySize for why, and for what it costs.
|
||||
r.Use(s.mw.MaxBodySize(maxFormBodySize))
|
||||
@@ -206,14 +200,14 @@ func (s *Server) setupSourceRoutes() {
|
||||
r.Get("/edit", s.h.HandleSourceEdit())
|
||||
r.Post("/edit", s.h.HandleSourceEditSubmit())
|
||||
r.Post("/delete", s.h.HandleSourceDelete())
|
||||
r.Get("/events", s.h.HandleSourceLogs())
|
||||
r.Get("/logs", s.h.HandleSourceLogs())
|
||||
// The log page renders each body only up to its cap, so
|
||||
// this is the only route that serves a whole one. It
|
||||
// belongs to this group for its RequireAuth and
|
||||
// NoCache; see HandleEventBodyDownload for the headers
|
||||
// that keep the bytes it returns inert.
|
||||
r.Get(
|
||||
"/events/{eventID}/body",
|
||||
"/logs/{eventID}/body",
|
||||
s.h.HandleEventBodyDownload(),
|
||||
)
|
||||
// Replay is the one page action that queues outbound work:
|
||||
@@ -280,7 +274,7 @@ func (s *Server) setupSourceRoutes() {
|
||||
|
||||
func (s *Server) setupWebhookRoutes() {
|
||||
s.router.With(s.mw.ReceiverRateLimit()).HandleFunc(
|
||||
"/h/{uuid}",
|
||||
"/webhook/{uuid}",
|
||||
s.h.HandleWebhook(),
|
||||
)
|
||||
}
|
||||
|
||||
+57
-462
@@ -24,6 +24,7 @@ import (
|
||||
"sneak.berlin/go/webhooker/internal/handlers"
|
||||
"sneak.berlin/go/webhooker/internal/healthcheck"
|
||||
"sneak.berlin/go/webhooker/internal/logger"
|
||||
"sneak.berlin/go/webhooker/internal/metrics"
|
||||
"sneak.berlin/go/webhooker/internal/middleware"
|
||||
"sneak.berlin/go/webhooker/internal/server"
|
||||
"sneak.berlin/go/webhooker/internal/session"
|
||||
@@ -47,8 +48,8 @@ type noopNotifier struct{}
|
||||
func (n *noopNotifier) Notify([]delivery.Task) {}
|
||||
|
||||
// noopEvictor satisfies handlers.New's delivery.WebhookEvictor
|
||||
// dependency. No test here checks what gets evicted, so it records
|
||||
// nothing.
|
||||
// dependency. These tests never delete a webhook, so there is
|
||||
// nothing to record.
|
||||
type noopEvictor struct{}
|
||||
|
||||
func (e *noopEvictor) EvictWebhook(string) {}
|
||||
@@ -113,6 +114,8 @@ func newTestEnvWithConfig(
|
||||
session.New,
|
||||
func() delivery.Notifier { return &noopNotifier{} },
|
||||
func() delivery.WebhookEvictor { return &noopEvictor{} },
|
||||
metrics.NewRegistry,
|
||||
metrics.New,
|
||||
middleware.New,
|
||||
delivery.NewGuard,
|
||||
handlers.New,
|
||||
@@ -240,26 +243,6 @@ func (e *testEnv) csrfFrom(
|
||||
return token, combined
|
||||
}
|
||||
|
||||
// urlFrom renders the page at path and returns the link or form
|
||||
// action that pattern's one group captures, so a test requests the
|
||||
// URL the template emitted rather than one it wrote itself.
|
||||
func (e *testEnv) urlFrom(
|
||||
t *testing.T,
|
||||
path, pattern string,
|
||||
cookies []*http.Cookie,
|
||||
) string {
|
||||
t.Helper()
|
||||
|
||||
w := e.get(path, cookies)
|
||||
require.Equal(t, http.StatusOK, w.Code)
|
||||
|
||||
match := regexp.MustCompile(pattern).
|
||||
FindStringSubmatch(w.Body.String())
|
||||
require.Len(t, match, 2, "%s should render %s", path, pattern)
|
||||
|
||||
return html.UnescapeString(match[1])
|
||||
}
|
||||
|
||||
// authCookies forges an authenticated session for the given user.
|
||||
func (e *testEnv) authCookies(
|
||||
t *testing.T,
|
||||
@@ -694,50 +677,12 @@ func TestPagesLogin_CookiesFromAnEarlierDatabase(t *testing.T) {
|
||||
|
||||
require.NotNil(t, fresh, "login must set a session cookie")
|
||||
assert.Equal(
|
||||
t, "/hooks",
|
||||
t, "/sources",
|
||||
env.get("/", []*http.Cookie{fresh}).Header().Get("Location"),
|
||||
"the new session cookie must authenticate",
|
||||
)
|
||||
}
|
||||
|
||||
// TestPagesLogin_ReturnsToTheRequestedPage is
|
||||
// https://git.eeqj.de/sneak/webhooker/issues/384: a page opened while
|
||||
// logged out leads to the login page, and logging in from there lands
|
||||
// on that page, query included.
|
||||
func TestPagesLogin_ReturnsToTheRequestedPage(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
username = "operator"
|
||||
password = "correct-horse-battery-staple"
|
||||
)
|
||||
|
||||
env := newTestEnv(t)
|
||||
userID, _ := env.seedUser(t, username, password)
|
||||
asked := "/hook/" + env.seedWebhook(t, userID).ID + "/events?page=2"
|
||||
|
||||
bounced := env.get(asked, nil)
|
||||
require.Equal(t, http.StatusSeeOther, bounced.Code)
|
||||
|
||||
loginPage := bounced.Header().Get("Location")
|
||||
|
||||
match := regexp.MustCompile(`name="next" value="([^"]*)"`).
|
||||
FindStringSubmatch(env.get(loginPage, nil).Body.String())
|
||||
require.Len(t, match, 2, "the login form must carry the page")
|
||||
|
||||
token, cookies := env.csrfFrom(t, loginPage, nil)
|
||||
|
||||
form := url.Values{}
|
||||
form.Set("csrf_token", token)
|
||||
form.Set("username", username)
|
||||
form.Set("password", password)
|
||||
form.Set("next", html.UnescapeString(match[1]))
|
||||
|
||||
w := env.post("/pages/login", form, cookies)
|
||||
require.Equal(t, http.StatusSeeOther, w.Code)
|
||||
assert.Equal(t, asked, w.Header().Get("Location"))
|
||||
}
|
||||
|
||||
// --- /user/{username} group ---
|
||||
|
||||
// TestPasswordChange_OversizeBody_RejectedAndPasswordUnchanged
|
||||
@@ -799,344 +744,7 @@ func TestPasswordChange_UnderLimit_Succeeds(t *testing.T) {
|
||||
)
|
||||
}
|
||||
|
||||
// --- /hooks group ---
|
||||
|
||||
// TestHooks_ListAndNewWebhookForm gets the webhook list through the
|
||||
// production router, follows both of its links to the new-webhook
|
||||
// form, then submits the form to the action and with the token the
|
||||
// page rendered. A mistyped route, link or form action fails here;
|
||||
// the handler tests cannot catch any of them, because they never
|
||||
// route a request.
|
||||
func TestHooks_ListAndNewWebhookForm(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := newTestEnv(t)
|
||||
|
||||
userID, _ := env.seedUser(t, "lister", "somepassword")
|
||||
cookies := env.authCookies(t, userID, "lister")
|
||||
|
||||
// The list shows its "Create Webhook" link only while it is empty.
|
||||
createLink := env.urlFrom(
|
||||
t, "/hooks", `href="([^"]+)"[^>]*>Create Webhook<`, cookies,
|
||||
)
|
||||
|
||||
existing := env.seedWebhook(t, userID)
|
||||
|
||||
list := env.get("/hooks", cookies)
|
||||
require.Equal(t, http.StatusOK, list.Code)
|
||||
assert.Contains(
|
||||
t, list.Body.String(), `href="/hook/`+existing.ID+`"`,
|
||||
"the list should link the user's webhook",
|
||||
)
|
||||
|
||||
// The "New Webhook" link has an icon between its href and its text.
|
||||
newLink := env.urlFrom(
|
||||
t, "/hooks", `href="([^"]+)"[^>]*>(?:\s*<[^>]*>)*\s*New Webhook`,
|
||||
cookies,
|
||||
)
|
||||
|
||||
token, cookies := env.csrfFrom(t, newLink, cookies)
|
||||
action := env.urlFrom(t, newLink, `action="(/hooks[^"]*)"`, cookies)
|
||||
assert.Equal(
|
||||
t, action,
|
||||
env.urlFrom(t, createLink, `action="(/hooks[^"]*)"`, cookies),
|
||||
"both links should open the new-webhook form",
|
||||
)
|
||||
|
||||
form := url.Values{}
|
||||
form.Set("csrf_token", token)
|
||||
form.Set("name", "created")
|
||||
|
||||
w := env.post(action, form, cookies)
|
||||
require.Equal(t, http.StatusSeeOther, w.Code)
|
||||
|
||||
var created database.Webhook
|
||||
|
||||
require.NoError(t,
|
||||
env.db.DB().Where("name = ?", "created").First(&created).Error,
|
||||
)
|
||||
assert.Equal(
|
||||
t, "/hook/"+created.ID, w.Header().Get("Location"),
|
||||
"creating a webhook should redirect to its page",
|
||||
)
|
||||
}
|
||||
|
||||
// --- /hook/{sourceID} group ---
|
||||
|
||||
// TestHook_EditFormAndDelete follows the webhook page's Edit link to
|
||||
// the edit form and submits it, then deletes the webhook with the
|
||||
// form on its page, every URL and token taken from the rendered
|
||||
// pages.
|
||||
func TestHook_EditFormAndDelete(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := newTestEnv(t)
|
||||
|
||||
userID, _ := env.seedUser(t, "editor", "somepassword")
|
||||
cookies := env.authCookies(t, userID, "editor")
|
||||
wh := env.seedWebhook(t, userID)
|
||||
page := "/hook/" + wh.ID
|
||||
|
||||
editPage := env.urlFrom(t, page, `href="(/hook/[^/"]+/edit)"`, cookies)
|
||||
token, cookies := env.csrfFrom(t, editPage, cookies)
|
||||
|
||||
form := url.Values{}
|
||||
form.Set("csrf_token", token)
|
||||
form.Set("name", "renamed")
|
||||
|
||||
w := env.post(
|
||||
env.urlFrom(t, editPage, `action="(/hook/[^/"]+/edit)"`, cookies),
|
||||
form, cookies,
|
||||
)
|
||||
require.Equal(t, http.StatusSeeOther, w.Code)
|
||||
assert.Equal(t, page, w.Header().Get("Location"))
|
||||
|
||||
var edited database.Webhook
|
||||
|
||||
require.NoError(t, env.db.DB().First(&edited, "id = ?", wh.ID).Error)
|
||||
assert.Equal(t, "renamed", edited.Name)
|
||||
|
||||
form = url.Values{}
|
||||
form.Set("csrf_token", token)
|
||||
|
||||
w = env.post(
|
||||
env.urlFrom(t, page, `action="(/hook/[^/"]+/delete)"`, cookies),
|
||||
form, cookies,
|
||||
)
|
||||
require.Equal(t, http.StatusSeeOther, w.Code)
|
||||
assert.Equal(t, "/hooks", w.Header().Get("Location"))
|
||||
assert.Equal(
|
||||
t, http.StatusNotFound, env.get(page, cookies).Code,
|
||||
"a deleted webhook's page should be gone",
|
||||
)
|
||||
}
|
||||
|
||||
// TestHook_EntrypointActions adds, deactivates and deletes an
|
||||
// entrypoint with the forms on the webhook page, each submitted to
|
||||
// the action and with the token the page rendered.
|
||||
func TestHook_EntrypointActions(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := newTestEnv(t)
|
||||
|
||||
userID, _ := env.seedUser(t, "epuser", "somepassword")
|
||||
cookies := env.authCookies(t, userID, "epuser")
|
||||
wh := env.seedWebhook(t, userID)
|
||||
page := "/hook/" + wh.ID
|
||||
|
||||
token, cookies := env.csrfFrom(t, page, cookies)
|
||||
|
||||
form := url.Values{}
|
||||
form.Set("csrf_token", token)
|
||||
|
||||
// submit posts the webhook page's form whose action pattern
|
||||
// captures, and requires the redirect back to that page.
|
||||
submit := func(pattern string) {
|
||||
t.Helper()
|
||||
|
||||
w := env.post(env.urlFrom(t, page, pattern, cookies), form, cookies)
|
||||
require.Equal(t, http.StatusSeeOther, w.Code)
|
||||
require.Equal(t, page, w.Header().Get("Location"))
|
||||
}
|
||||
|
||||
submit(`action="(/hook/[^/"]+/entrypoints)"`)
|
||||
|
||||
var added database.Entrypoint
|
||||
|
||||
require.NoError(t,
|
||||
env.db.DB().First(&added, "webhook_id = ?", wh.ID).Error,
|
||||
)
|
||||
require.True(t, added.Active)
|
||||
|
||||
submit(`action="(/hook/[^/"]+/entrypoints/[^/"]+/toggle)"`)
|
||||
|
||||
var toggled database.Entrypoint
|
||||
|
||||
require.NoError(t,
|
||||
env.db.DB().First(&toggled, "id = ?", added.ID).Error,
|
||||
)
|
||||
assert.False(t, toggled.Active, "the toggle should deactivate it")
|
||||
|
||||
submit(`action="(/hook/[^/"]+/entrypoints/[^/"]+/delete)"`)
|
||||
|
||||
var left int64
|
||||
|
||||
require.NoError(t, env.db.DB().Model(&database.Entrypoint{}).
|
||||
Where("webhook_id = ?", wh.ID).Count(&left).Error)
|
||||
assert.Zero(t, left, "the delete should remove the entrypoint")
|
||||
}
|
||||
|
||||
// TestHook_TargetActions adds a target with the form on the webhook
|
||||
// page, follows its Edit link to the target edit form and submits
|
||||
// it, then deactivates and deletes it, every URL and token taken from
|
||||
// the rendered pages.
|
||||
func TestHook_TargetActions(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := newTestEnv(t)
|
||||
|
||||
userID, _ := env.seedUser(t, "tgtuser", "somepassword")
|
||||
cookies := env.authCookies(t, userID, "tgtuser")
|
||||
wh := env.seedWebhook(t, userID)
|
||||
page := "/hook/" + wh.ID
|
||||
|
||||
token, cookies := env.csrfFrom(t, page, cookies)
|
||||
|
||||
// submit posts form, with the token, to the action pattern
|
||||
// captures on the page at from, and requires the redirect back to
|
||||
// the webhook page.
|
||||
submit := func(from, pattern string, form url.Values) {
|
||||
t.Helper()
|
||||
|
||||
form.Set("csrf_token", token)
|
||||
|
||||
w := env.post(env.urlFrom(t, from, pattern, cookies), form, cookies)
|
||||
require.Equal(t, http.StatusSeeOther, w.Code)
|
||||
require.Equal(t, page, w.Header().Get("Location"))
|
||||
}
|
||||
|
||||
submit(page, `action="(/hook/[^/"]+/targets)"`, url.Values{
|
||||
"name": {"added"},
|
||||
"type": {string(database.TargetTypeLog)},
|
||||
})
|
||||
|
||||
editPage := env.urlFrom(
|
||||
t, page, `href="(/hook/[^/"]+/targets/[^/"]+/edit)"`, cookies,
|
||||
)
|
||||
submit(editPage, `action="(/hook/[^/"]+/targets/[^/"]+/edit)"`,
|
||||
url.Values{"name": {"renamed"}})
|
||||
|
||||
var edited database.Target
|
||||
|
||||
require.NoError(t,
|
||||
env.db.DB().First(&edited, "webhook_id = ?", wh.ID).Error,
|
||||
)
|
||||
assert.Equal(t, "renamed", edited.Name)
|
||||
require.True(t, edited.Active)
|
||||
|
||||
submit(page, `action="(/hook/[^/"]+/targets/[^/"]+/toggle)"`,
|
||||
url.Values{})
|
||||
|
||||
var toggled database.Target
|
||||
|
||||
require.NoError(t,
|
||||
env.db.DB().First(&toggled, "id = ?", edited.ID).Error,
|
||||
)
|
||||
assert.False(t, toggled.Active, "the toggle should deactivate it")
|
||||
|
||||
submit(page, `action="(/hook/[^/"]+/targets/[^/"]+/delete)"`,
|
||||
url.Values{})
|
||||
|
||||
var left int64
|
||||
|
||||
require.NoError(t, env.db.DB().Model(&database.Target{}).
|
||||
Where("webhook_id = ?", wh.ID).Count(&left).Error)
|
||||
assert.Zero(t, left, "the delete should remove the target")
|
||||
}
|
||||
|
||||
// TestHook_ResubmitFromEventLog follows the webhook page's "Full
|
||||
// Event Log" link, then resubmits a stored event with the form on
|
||||
// that page, submitted to the action and with the token the page
|
||||
// rendered.
|
||||
func TestHook_ResubmitFromEventLog(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := newTestEnv(t)
|
||||
|
||||
userID, _ := env.seedUser(t, "resubmitter", "somepassword")
|
||||
cookies := env.authCookies(t, userID, "resubmitter")
|
||||
wh := env.seedWebhook(t, userID)
|
||||
env.seedEvent(t, wh.ID, `{"resubmit":"me"}`)
|
||||
|
||||
logsPath := env.urlFrom(
|
||||
t, "/hook/"+wh.ID, `href="([^"]+)"[^>]*>Full Event Log<`, cookies,
|
||||
)
|
||||
|
||||
token, cookies := env.csrfFrom(t, logsPath, cookies)
|
||||
|
||||
form := url.Values{}
|
||||
form.Set("csrf_token", token)
|
||||
|
||||
w := env.post(
|
||||
env.urlFrom(t, logsPath, `action="(/hook/[^"]+/resubmit)"`, cookies),
|
||||
form, cookies,
|
||||
)
|
||||
require.Equal(t, http.StatusSeeOther, w.Code)
|
||||
assert.Equal(
|
||||
t, logsPath+"?resubmit=no-targets", w.Header().Get("Location"),
|
||||
)
|
||||
|
||||
webhookDB, err := env.dbMgr.GetDB(wh.ID)
|
||||
require.NoError(t, err)
|
||||
|
||||
var events int64
|
||||
|
||||
require.NoError(t,
|
||||
webhookDB.Model(&database.Event{}).Count(&events).Error,
|
||||
)
|
||||
assert.Equal(t, int64(2), events, "the resubmit stores a new event")
|
||||
}
|
||||
|
||||
// TestHook_LinksBetweenPages follows each link to a webhook page that
|
||||
// the tests above do not: the navbar's "Webhooks" links, the back and
|
||||
// Cancel links, the list's link to a webhook, the "Full Event Log"
|
||||
// link beside the recent events, and the event log's page links. Each
|
||||
// must point where it should, and that page must render.
|
||||
func TestHook_LinksBetweenPages(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := newTestEnv(t)
|
||||
|
||||
userID, _ := env.seedUser(t, "navigator", "somepassword")
|
||||
cookies := env.authCookies(t, userID, "navigator")
|
||||
wh := env.seedWebhook(t, userID)
|
||||
tgt := env.seedTarget(t, wh.ID)
|
||||
|
||||
// The event log shows 25 events a page; one more gives it a second
|
||||
// page, so it renders its Next and Previous links.
|
||||
for range 26 {
|
||||
env.seedEvent(t, wh.ID, "paged")
|
||||
}
|
||||
|
||||
list := "/hooks"
|
||||
newForm := list + "/new"
|
||||
page := "/hook/" + wh.ID
|
||||
targetEdit := page + "/targets/" + tgt.ID + "/edit"
|
||||
events := page + "/events"
|
||||
back := `href="([^"]+)"[^>]*>← Back to `
|
||||
cancel := `href="([^"]+)"[^>]*>Cancel<`
|
||||
|
||||
for _, link := range []struct{ from, pattern, want string }{
|
||||
// The navbar on the profile page: its desktop link, then its
|
||||
// mobile menu link.
|
||||
{
|
||||
"/user/navigator/",
|
||||
`href="([^"]+)" class="btn-text">Webhooks<`,
|
||||
list,
|
||||
},
|
||||
{
|
||||
"/user/navigator/",
|
||||
`href="([^"]+)" class="btn-text w-full[^"]*">Webhooks<`,
|
||||
list,
|
||||
},
|
||||
{list, `href="(/hook/[^"]+)"`, page},
|
||||
{newForm, back, list},
|
||||
{newForm, cancel, list},
|
||||
{page, back, list},
|
||||
{page, `Recent Events</h2>\s*<a href="([^"]+)"`, events},
|
||||
{page + "/edit", back, page},
|
||||
{page + "/edit", cancel, page},
|
||||
{targetEdit, back, page},
|
||||
{targetEdit, cancel, page},
|
||||
{events, back, page},
|
||||
{events, `href="([^"]+)"[^>]*>Next →<`, events + "?page=2"},
|
||||
{events + "?page=2", `href="([^"]+)"[^>]*>← Previous<`, events + "?page=1"},
|
||||
} {
|
||||
got := env.urlFrom(t, link.from, link.pattern, cookies)
|
||||
assert.Equal(t, link.want, got, "%s: %s", link.from, link.pattern)
|
||||
assert.Equal(t, http.StatusOK, env.get(got, cookies).Code, got)
|
||||
}
|
||||
}
|
||||
// --- /source/{sourceID} group ---
|
||||
|
||||
// TestSourceLogs_TruncationLinkDownloadsTheBody walks the whole
|
||||
// feature the way a user does: render the event log page through
|
||||
@@ -1164,11 +772,11 @@ func TestSourceLogs_TruncationLinkDownloadsTheBody(t *testing.T) {
|
||||
wh := env.seedWebhook(t, userID)
|
||||
env.seedEvent(t, wh.ID, stored)
|
||||
|
||||
page := env.get("/hook/"+wh.ID+"/events", cookies)
|
||||
page := env.get("/source/"+wh.ID+"/logs", cookies)
|
||||
require.Equal(t, http.StatusOK, page.Code)
|
||||
|
||||
link := regexp.MustCompile(
|
||||
`href="(/hook/[^"]+/body)"`,
|
||||
`href="(/source/[^"]+/body)"`,
|
||||
).FindStringSubmatch(page.Body.String())
|
||||
require.Len(
|
||||
t, link, 2,
|
||||
@@ -1214,7 +822,7 @@ func TestSourceLogsBody_OtherUser404s(t *testing.T) {
|
||||
const payload = "OWNERS-PAYLOAD-77c1"
|
||||
|
||||
evt := env.seedEvent(t, wh.ID, payload)
|
||||
path := "/hook/" + wh.ID + "/events/" + evt.ID + "/body"
|
||||
path := "/source/" + wh.ID + "/logs/" + evt.ID + "/body"
|
||||
|
||||
intruderID, _ := env.seedUser(t, "intruder", "somepassword")
|
||||
intruder := env.authCookies(t, intruderID, "intruder")
|
||||
@@ -1225,10 +833,7 @@ func TestSourceLogsBody_OtherUser404s(t *testing.T) {
|
||||
|
||||
anon := env.get(path, nil)
|
||||
assert.Equal(t, http.StatusSeeOther, anon.Code)
|
||||
assert.Equal(
|
||||
t, "/pages/login?next="+url.QueryEscape(path),
|
||||
anon.Header().Get("Location"),
|
||||
)
|
||||
assert.Equal(t, "/pages/login", anon.Header().Get("Location"))
|
||||
}
|
||||
|
||||
// TestDeliveryReplay_PostOnlyAndCSRFProtected walks the replay action
|
||||
@@ -1251,7 +856,7 @@ func TestDeliveryReplay_PostOnlyAndCSRFProtected(t *testing.T) {
|
||||
evt := env.seedEvent(t, wh.ID, `{"replay":"me"}`)
|
||||
dlv := env.seedFailedDelivery(t, wh.ID, evt.ID, tgt.ID)
|
||||
|
||||
path := "/hook/" + wh.ID + "/deliveries/" + dlv.ID +
|
||||
path := "/source/" + wh.ID + "/deliveries/" + dlv.ID +
|
||||
"/replay"
|
||||
|
||||
assert.Equal(
|
||||
@@ -1277,7 +882,7 @@ func TestDeliveryReplay_PostOnlyAndCSRFProtected(t *testing.T) {
|
||||
// The token and the action URL both come out of the rendered
|
||||
// page, so a typo in either the route pattern or the template
|
||||
// fails here.
|
||||
logsPath := "/hook/" + wh.ID + "/events"
|
||||
logsPath := "/source/" + wh.ID + "/logs"
|
||||
|
||||
token, cookies := env.csrfFrom(t, logsPath, cookies)
|
||||
|
||||
@@ -1285,7 +890,7 @@ func TestDeliveryReplay_PostOnlyAndCSRFProtected(t *testing.T) {
|
||||
require.Equal(t, http.StatusOK, page.Code)
|
||||
|
||||
action := regexp.MustCompile(
|
||||
`action="(/hook/[^"]+/replay)"`,
|
||||
`action="(/source/[^"]+/replay)"`,
|
||||
).FindStringSubmatch(page.Body.String())
|
||||
require.Len(
|
||||
t, action, 2,
|
||||
@@ -1310,59 +915,6 @@ func TestDeliveryReplay_PostOnlyAndCSRFProtected(t *testing.T) {
|
||||
)
|
||||
}
|
||||
|
||||
// --- /h/{uuid} receiver ---
|
||||
|
||||
// TestReceiver_EntrypointURLIsRateLimited takes the entrypoint URL
|
||||
// the webhook page shows and posts to it through the production
|
||||
// router until the receiver rate limit refuses it. The URL has to
|
||||
// reach the receiver, and the limit has to apply to it.
|
||||
func TestReceiver_EntrypointURLIsRateLimited(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const limit = 2
|
||||
|
||||
env := newTestEnvWithConfig(t, &config.Config{
|
||||
DataDir: t.TempDir(),
|
||||
Environment: config.EnvironmentDev,
|
||||
ReceiverRateLimit: limit,
|
||||
})
|
||||
|
||||
userID, _ := env.seedUser(t, "receiver", "somepassword")
|
||||
cookies := env.authCookies(t, userID, "receiver")
|
||||
|
||||
wh := env.seedWebhook(t, userID)
|
||||
require.NoError(t, env.db.DB().Omit(clause.Associations).Create(
|
||||
&database.Entrypoint{
|
||||
WebhookID: wh.ID,
|
||||
Path: "6f1e2a9c-4b7d-4e3a-9c2f-1d8b5a7e3c60",
|
||||
Active: true,
|
||||
},
|
||||
).Error)
|
||||
|
||||
page := env.get("/hook/"+wh.ID, cookies)
|
||||
require.Equal(t, http.StatusOK, page.Code)
|
||||
|
||||
shown := regexp.MustCompile(`(/h/[^<]+)</code>`).
|
||||
FindStringSubmatch(page.Body.String())
|
||||
require.Len(
|
||||
t, shown, 2, "the webhook page should show the entrypoint URL",
|
||||
)
|
||||
|
||||
for i := range limit {
|
||||
assert.Equal(
|
||||
t, http.StatusOK,
|
||||
env.post(shown[1], url.Values{}, nil).Code,
|
||||
"request %d should reach the receiver", i,
|
||||
)
|
||||
}
|
||||
|
||||
assert.Equal(
|
||||
t, http.StatusTooManyRequests,
|
||||
env.post(shown[1], url.Values{}, nil).Code,
|
||||
"the receiver rate limit must apply to the entrypoint URL",
|
||||
)
|
||||
}
|
||||
|
||||
// metricsConfig is a Config differing from the routing default only
|
||||
// in the two /metrics credentials.
|
||||
func metricsConfig(
|
||||
@@ -1478,3 +1030,46 @@ func TestMetricsRouteUnmountedOnHalfSetConfig(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestTwoMetricsRoutersInOneProcess pins
|
||||
// https://git.eeqj.de/sneak/webhooker/issues/227: a second
|
||||
// metrics-enabled router in one process used to panic, because the
|
||||
// HTTP metrics registered on Prometheus's global default registry.
|
||||
// Two routers are built over separate dependency graphs and a third
|
||||
// over the first graph again, and each must still serve the HTTP,
|
||||
// delivery, Go runtime and process series, and the series counting
|
||||
// scrapes of /metrics itself.
|
||||
func TestTwoMetricsRoutersInOneProcess(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
first := newTestEnvWithConfig(
|
||||
t, metricsConfig(t, metricsUser, metricsAuthValue),
|
||||
)
|
||||
second := newTestEnvWithConfig(
|
||||
t, metricsConfig(t, metricsUser, metricsAuthValue),
|
||||
)
|
||||
third := &testEnv{
|
||||
router: server.NewRouterForTest(
|
||||
first.log.Get(), first.cfg, first.mw, first.hnd,
|
||||
),
|
||||
}
|
||||
|
||||
for _, env := range []*testEnv{first, second, third} {
|
||||
env.get("/", nil)
|
||||
|
||||
scrape := env.metricsRequest(metricsUser, metricsAuthValue)
|
||||
require.Equal(t, http.StatusOK, scrape.Code)
|
||||
|
||||
for _, series := range []string{
|
||||
"http_request_duration_seconds",
|
||||
"http_response_size_bytes",
|
||||
"http_requests_inflight",
|
||||
"webhooker_events_received_total",
|
||||
"go_goroutines",
|
||||
"process_start_time_seconds",
|
||||
"promhttp_metric_handler_requests_total",
|
||||
} {
|
||||
assert.Contains(t, scrape.Body.String(), series)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -55,7 +55,7 @@ func sentryClientOptions(dsn, release string) sentry.ClientOptions {
|
||||
//
|
||||
// URL is the third such field. NewRequest builds it as
|
||||
// scheme://host/path (interfaces.go:183), and on the receiver route
|
||||
// that path is /h/<uuid> in full — a write capability, not an
|
||||
// that path is /webhook/<uuid> in full — a write capability, not an
|
||||
// identifier. It is rebuilt here from the chi route pattern, on every
|
||||
// route, keeping the scheme and the host.
|
||||
//
|
||||
|
||||
@@ -153,7 +153,7 @@ func (c sentryCase) router() http.Handler {
|
||||
sentryhttp.New(sentryhttp.Options{Repanic: true}).Handle,
|
||||
)
|
||||
router.HandleFunc("/pages/login", handler)
|
||||
router.HandleFunc("/h/{uuid}", handler)
|
||||
router.HandleFunc("/webhook/{uuid}", handler)
|
||||
|
||||
return router
|
||||
}
|
||||
@@ -191,7 +191,7 @@ func sentryLoginRequest(client *sentry.Client) *http.Request {
|
||||
// concrete path carries the entrypoint capability.
|
||||
func sentryReceiverRequest(client *sentry.Client) *http.Request {
|
||||
return sentryRequest(
|
||||
client, "/h/"+sentryReceiverUUID, "payload=hello",
|
||||
client, "/webhook/"+sentryReceiverUUID, "payload=hello",
|
||||
)
|
||||
}
|
||||
|
||||
@@ -316,7 +316,7 @@ func TestSentryScrub_ReplacesTheCapabilityPathWithTheRoutePattern(
|
||||
t, marshalEvent(t, event), sentryReceiverUUID,
|
||||
)
|
||||
assert.Equal(
|
||||
t, "http://example.com/h/{uuid}", event.Request.URL,
|
||||
t, "http://example.com/webhook/{uuid}", event.Request.URL,
|
||||
)
|
||||
}
|
||||
|
||||
@@ -401,7 +401,7 @@ func TestSentryScrub_TransactionDispatchIsUnscrubbedWithoutTheHook(
|
||||
func TestSentryScrub_FallsBackWithoutARoutePattern(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
concrete := "https://example.com/h/" + sentryReceiverUUID
|
||||
concrete := "https://example.com/webhook/" + sentryReceiverUUID
|
||||
|
||||
// A request with no chi routing context on it at all, which is
|
||||
// what an event captured outside the router would carry.
|
||||
@@ -426,7 +426,7 @@ func TestSentryScrub_FallsBackWithoutARoutePattern(t *testing.T) {
|
||||
|
||||
event := sentry.NewEvent()
|
||||
event.Request = &sentry.Request{URL: concrete}
|
||||
event.Transaction = "POST /h/" +
|
||||
event.Transaction = "POST /webhook/" +
|
||||
sentryReceiverUUID
|
||||
|
||||
scrubbed := server.ScrubSentryRequestForTest(
|
||||
@@ -459,9 +459,9 @@ func TestSentryScrub_WithholdsUnparseableValues(t *testing.T) {
|
||||
|
||||
event := sentry.NewEvent()
|
||||
event.Request = &sentry.Request{
|
||||
URL: "/h/" + sentryReceiverUUID,
|
||||
URL: "/webhook/" + sentryReceiverUUID,
|
||||
}
|
||||
event.Transaction = "/h/" + sentryReceiverUUID
|
||||
event.Transaction = "/webhook/" + sentryReceiverUUID
|
||||
|
||||
scrubbed := server.ScrubSentryRequestForTest(event, nil)
|
||||
require.NotNil(t, scrubbed)
|
||||
|
||||
+1
-11
@@ -22,16 +22,6 @@
|
||||
# The one figure above 90s is GOMAXPROCS 1, a synthetic core floor rather than
|
||||
# a condition CI runs under. If a CPU-limited runner ever puts a real run near
|
||||
# 67s, that is the datum to revisit the org figure with.
|
||||
#
|
||||
# -p 4 -parallel 8 keep the run under 2 GB of memory: at most four test
|
||||
# binaries build or run at once, each with at most eight parallel tests. Under
|
||||
# -race every test binary and every link costs a few hundred MB, so the
|
||||
# defaults (one per core) add up to several GB on a many-core host.
|
||||
#
|
||||
# No -v: the Docker build cuts each step's log off at 2 MiB, and verbose output
|
||||
# from the whole suite passes that before a failure is printed. Without it, go
|
||||
# test prints one result line per package and, for a package that fails,
|
||||
# everything its tests wrote, application log lines included.
|
||||
set -eu
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||
@@ -39,7 +29,7 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||
main() {
|
||||
cd "$ROOT"
|
||||
"$ROOT/script/assets"
|
||||
go test -race -p 4 -parallel 8 -timeout 90s ./...
|
||||
go test -v -race -timeout 90s ./...
|
||||
}
|
||||
|
||||
main "$@"
|
||||
|
||||
@@ -24,7 +24,6 @@
|
||||
|
||||
<form method="POST" action="/pages/login" class="space-y-6">
|
||||
<input type="hidden" name="csrf_token" value="{{.CSRFToken}}">
|
||||
<input type="hidden" name="next" value="{{.Next}}">
|
||||
<div class="form-group">
|
||||
<label for="username" class="label">Username</label>
|
||||
<input
|
||||
|
||||
@@ -6,19 +6,17 @@
|
||||
</div>
|
||||
|
||||
<!-- Mobile menu button -->
|
||||
{{if .User}}
|
||||
<button @click="open = !open" class="md:hidden p-2 rounded-md text-gray-500 hover:bg-gray-100">
|
||||
<svg class="w-6 h-6" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
||||
<path x-show="!open" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M4 6h16M4 12h16M4 18h16"/>
|
||||
<path x-show="open" x-cloak stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M6 18L18 6M6 6l12 12"/>
|
||||
</svg>
|
||||
</button>
|
||||
{{end}}
|
||||
|
||||
<!-- Desktop navigation -->
|
||||
<div class="hidden md:flex items-center gap-4">
|
||||
{{if .User}}
|
||||
<a href="/hooks" class="btn-text">Webhooks</a>
|
||||
<a href="/sources" class="btn-text">Webhooks</a>
|
||||
<a href="/user/{{.User.Username}}" class="btn-text">
|
||||
<svg class="w-5 h-5 mr-1" fill="currentColor" viewBox="0 0 16 16">
|
||||
<path d="M11 6a3 3 0 1 1-6 0 3 3 0 0 1 6 0z"/>
|
||||
@@ -30,6 +28,8 @@
|
||||
<input type="hidden" name="csrf_token" value="{{.CSRFToken}}">
|
||||
<button type="submit" class="btn-text">Logout</button>
|
||||
</form>
|
||||
{{else}}
|
||||
<a href="/pages/login" class="btn-primary">Login</a>
|
||||
{{end}}
|
||||
</div>
|
||||
</div>
|
||||
@@ -38,12 +38,14 @@
|
||||
<div x-show="open" x-cloak x-transition class="md:hidden mt-4 pt-4 border-t border-gray-200">
|
||||
<div class="flex flex-col gap-2">
|
||||
{{if .User}}
|
||||
<a href="/hooks" class="btn-text w-full text-left">Webhooks</a>
|
||||
<a href="/sources" class="btn-text w-full text-left">Webhooks</a>
|
||||
<a href="/user/{{.User.Username}}" class="btn-text w-full text-left">Profile</a>
|
||||
<form method="POST" action="/pages/logout">
|
||||
<input type="hidden" name="csrf_token" value="{{.CSRFToken}}">
|
||||
<button type="submit" class="btn-text w-full text-left">Logout</button>
|
||||
</form>
|
||||
{{else}}
|
||||
<a href="/pages/login" class="btn-primary w-full">Login</a>
|
||||
{{end}}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
{{define "content"}}
|
||||
<div class="max-w-6xl mx-auto px-6 py-8" x-data="{ showAddEntrypoint: false, showAddTarget: false }">
|
||||
<div class="mb-6">
|
||||
<a href="/hooks" class="text-sm text-primary-600 hover:text-primary-700">← Back to webhooks</a>
|
||||
<a href="/sources" class="text-sm text-primary-600 hover:text-primary-700">← Back to webhooks</a>
|
||||
<div class="flex justify-between items-center mt-2">
|
||||
<div>
|
||||
<h1 class="text-2xl font-medium text-gray-900">{{.Webhook.Name}}</h1>
|
||||
@@ -14,9 +14,9 @@
|
||||
{{end}}
|
||||
</div>
|
||||
<div class="flex gap-2">
|
||||
<a href="/hook/{{.Webhook.ID}}/events" class="btn-secondary">Full Event Log</a>
|
||||
<a href="/hook/{{.Webhook.ID}}/edit" class="btn-secondary">Edit</a>
|
||||
<form method="POST" action="/hook/{{.Webhook.ID}}/delete" onsubmit="return confirm('Delete this webhook and all its data?')">
|
||||
<a href="/source/{{.Webhook.ID}}/logs" class="btn-secondary">Event Log</a>
|
||||
<a href="/source/{{.Webhook.ID}}/edit" class="btn-secondary">Edit</a>
|
||||
<form method="POST" action="/source/{{.Webhook.ID}}/delete" onsubmit="return confirm('Delete this webhook and all its data?')">
|
||||
<input type="hidden" name="csrf_token" value="{{.CSRFToken}}">
|
||||
<button type="submit" class="btn-danger">Delete</button>
|
||||
</form>
|
||||
@@ -24,8 +24,6 @@
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{{template "webhook_stats" .}}
|
||||
|
||||
<div class="grid grid-cols-1 lg:grid-cols-2 gap-6">
|
||||
<!-- Entrypoints -->
|
||||
<div class="card">
|
||||
@@ -41,7 +39,7 @@
|
||||
|
||||
<!-- Add entrypoint form -->
|
||||
<div x-show="showAddEntrypoint" x-cloak class="p-4 bg-gray-50 border-b border-gray-200">
|
||||
<form method="POST" action="/hook/{{.Webhook.ID}}/entrypoints" class="flex gap-2">
|
||||
<form method="POST" action="/source/{{.Webhook.ID}}/entrypoints" class="flex gap-2">
|
||||
<input type="hidden" name="csrf_token" value="{{.CSRFToken}}">
|
||||
<input type="text" name="description" placeholder="Description (optional)" class="input text-sm flex-1">
|
||||
<button type="submit" class="btn-primary text-sm">Add</button>
|
||||
@@ -59,20 +57,20 @@
|
||||
{{else}}
|
||||
<span class="badge-error">Inactive</span>
|
||||
{{end}}
|
||||
<form method="POST" action="/hook/{{$.Webhook.ID}}/entrypoints/{{.ID}}/toggle" class="inline">
|
||||
<form method="POST" action="/source/{{$.Webhook.ID}}/entrypoints/{{.ID}}/toggle" class="inline">
|
||||
<input type="hidden" name="csrf_token" value="{{$.CSRFToken}}">
|
||||
<button type="submit" class="text-xs text-gray-500 hover:text-primary-600" title="{{if .Active}}Deactivate{{else}}Activate{{end}}">
|
||||
{{if .Active}}Deactivate{{else}}Activate{{end}}
|
||||
</button>
|
||||
</form>
|
||||
<form method="POST" action="/hook/{{$.Webhook.ID}}/entrypoints/{{.ID}}/delete" onsubmit="return confirm('Delete this entrypoint?')" class="inline">
|
||||
<form method="POST" action="/source/{{$.Webhook.ID}}/entrypoints/{{.ID}}/delete" onsubmit="return confirm('Delete this entrypoint?')" class="inline">
|
||||
<input type="hidden" name="csrf_token" value="{{$.CSRFToken}}">
|
||||
<button type="submit" class="text-xs text-red-500 hover:text-red-700" title="Delete">Delete</button>
|
||||
</form>
|
||||
</div>
|
||||
</div>
|
||||
<div class="flex items-start gap-2 mt-1">
|
||||
<code id="entrypoint-url-{{.ID}}" class="text-xs text-gray-500 break-all block flex-1">{{$.BaseURL}}/h/{{.Path}}</code>
|
||||
<code id="entrypoint-url-{{.ID}}" class="text-xs text-gray-500 break-all block flex-1">{{$.BaseURL}}/webhook/{{.Path}}</code>
|
||||
<!-- Hidden until app.js reveals it; without the
|
||||
script the URL above stays selectable. -->
|
||||
<button type="button" hidden data-copy-target="entrypoint-url-{{.ID}}" class="text-xs text-gray-500 hover:text-primary-600">Copy</button>
|
||||
@@ -100,7 +98,7 @@
|
||||
|
||||
<!-- Add target form -->
|
||||
<div x-show="showAddTarget" x-cloak class="p-4 bg-gray-50 border-b border-gray-200">
|
||||
<form method="POST" action="/hook/{{.Webhook.ID}}/targets" x-data="{ targetType: 'http' }" class="space-y-3">
|
||||
<form method="POST" action="/source/{{.Webhook.ID}}/targets" x-data="{ targetType: 'http' }" class="space-y-3">
|
||||
<input type="hidden" name="csrf_token" value="{{.CSRFToken}}">
|
||||
<div class="flex gap-2">
|
||||
<input type="text" name="name" placeholder="Target name" required class="input text-sm flex-1">
|
||||
@@ -153,14 +151,14 @@
|
||||
{{else}}
|
||||
<span class="badge-error">Inactive</span>
|
||||
{{end}}
|
||||
<a href="/hook/{{$.Webhook.ID}}/targets/{{.ID}}/edit" class="text-xs text-gray-500 hover:text-primary-600" title="Edit">Edit</a>
|
||||
<form method="POST" action="/hook/{{$.Webhook.ID}}/targets/{{.ID}}/toggle" class="inline">
|
||||
<a href="/source/{{$.Webhook.ID}}/targets/{{.ID}}/edit" class="text-xs text-gray-500 hover:text-primary-600" title="Edit">Edit</a>
|
||||
<form method="POST" action="/source/{{$.Webhook.ID}}/targets/{{.ID}}/toggle" class="inline">
|
||||
<input type="hidden" name="csrf_token" value="{{$.CSRFToken}}">
|
||||
<button type="submit" class="text-xs text-gray-500 hover:text-primary-600" title="{{if .Active}}Deactivate{{else}}Activate{{end}}">
|
||||
{{if .Active}}Deactivate{{else}}Activate{{end}}
|
||||
</button>
|
||||
</form>
|
||||
<form method="POST" action="/hook/{{$.Webhook.ID}}/targets/{{.ID}}/delete" onsubmit="return confirm('Delete this target?')" class="inline">
|
||||
<form method="POST" action="/source/{{$.Webhook.ID}}/targets/{{.ID}}/delete" onsubmit="return confirm('Delete this target?')" class="inline">
|
||||
<input type="hidden" name="csrf_token" value="{{$.CSRFToken}}">
|
||||
<button type="submit" class="text-xs text-red-500 hover:text-red-700" title="Delete">Delete</button>
|
||||
</form>
|
||||
@@ -184,7 +182,7 @@
|
||||
<div class="card mt-6">
|
||||
<div class="p-4 border-b border-gray-200 flex justify-between items-center">
|
||||
<h2 class="text-lg font-medium text-gray-900">50 Most Recent Events</h2>
|
||||
<a href="/hook/{{.Webhook.ID}}/events" class="btn-text text-sm">Full Event Log</a>
|
||||
<a href="/source/{{.Webhook.ID}}/logs" class="btn-text text-sm">View All</a>
|
||||
</div>
|
||||
<div class="divide-y divide-gray-100">
|
||||
{{range .Events}}
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
{{define "content"}}
|
||||
<div class="max-w-2xl mx-auto px-6 py-8">
|
||||
<div class="mb-6">
|
||||
<a href="/hook/{{.Webhook.ID}}" class="text-sm text-primary-600 hover:text-primary-700">← Back to {{.Webhook.Name}}</a>
|
||||
<a href="/source/{{.Webhook.ID}}" class="text-sm text-primary-600 hover:text-primary-700">← Back to {{.Webhook.Name}}</a>
|
||||
<h1 class="text-2xl font-medium text-gray-900 mt-2">Edit Webhook</h1>
|
||||
</div>
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
<div class="alert-error">{{.Error}}</div>
|
||||
{{end}}
|
||||
|
||||
<form method="POST" action="/hook/{{.Webhook.ID}}/edit" class="space-y-6">
|
||||
<form method="POST" action="/source/{{.Webhook.ID}}/edit" class="space-y-6">
|
||||
<input type="hidden" name="csrf_token" value="{{.CSRFToken}}">
|
||||
<div class="form-group">
|
||||
<label for="name" class="label">Name</label>
|
||||
@@ -34,7 +34,7 @@
|
||||
|
||||
<div class="flex gap-3">
|
||||
<button type="submit" class="btn-primary">Save Changes</button>
|
||||
<a href="/hook/{{.Webhook.ID}}" class="btn-secondary">Cancel</a>
|
||||
<a href="/source/{{.Webhook.ID}}" class="btn-secondary">Cancel</a>
|
||||
</div>
|
||||
</form>
|
||||
</div>
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
{{template "base" .}}
|
||||
|
||||
{{define "title"}}Full Event Log - {{.Webhook.Name}} - Webhooker{{end}}
|
||||
{{define "title"}}Event Log - {{.Webhook.Name}} - Webhooker{{end}}
|
||||
|
||||
{{define "content"}}
|
||||
<div class="max-w-6xl mx-auto px-6 py-8">
|
||||
<div class="mb-6">
|
||||
<a href="/hook/{{.Webhook.ID}}" class="text-sm text-primary-600 hover:text-primary-700">← Back to {{.Webhook.Name}}</a>
|
||||
<a href="/source/{{.Webhook.ID}}" class="text-sm text-primary-600 hover:text-primary-700">← Back to {{.Webhook.Name}}</a>
|
||||
<div class="flex justify-between items-center mt-2">
|
||||
<h1 class="text-2xl font-medium text-gray-900">Full Event Log</h1>
|
||||
<h1 class="text-2xl font-medium text-gray-900">Event Log</h1>
|
||||
<span class="text-sm text-gray-500">{{.TotalEvents}} total event{{if ne .TotalEvents 1}}s{{end}}</span>
|
||||
</div>
|
||||
</div>
|
||||
@@ -55,7 +55,7 @@
|
||||
{{if .ResubmittedFrom}}Resubmitted from event <span class="font-mono">{{.ResubmittedFromID}}</span>.{{end}}
|
||||
{{if .ResubmitCount}}Resubmitted as {{.ResubmitCount}} new event{{if ne .ResubmitCount 1}}s{{end}}.{{end}}
|
||||
</div>
|
||||
<form method="POST" action="/hook/{{$.Webhook.ID}}/events/{{.ID}}/resubmit" class="inline">
|
||||
<form method="POST" action="/source/{{$.Webhook.ID}}/events/{{.ID}}/resubmit" class="inline">
|
||||
<input type="hidden" name="csrf_token" value="{{$.CSRFToken}}">
|
||||
<input type="hidden" name="page" value="{{$.Page}}">
|
||||
<button type="submit" class="text-xs text-primary-600 hover:text-primary-700" title="Submit this event again as a new event, to every currently active target">Resubmit</button>
|
||||
@@ -63,7 +63,7 @@
|
||||
</div>
|
||||
<pre class="text-xs text-gray-700 overflow-x-auto whitespace-pre-wrap break-all">{{.Body}}</pre>
|
||||
{{if .BodyTruncated}}
|
||||
<p class="mt-2 text-xs text-gray-500">Body truncated for display: showing {{.BodyShownBytes}} of {{.BodyBytes}} bytes. The stored body is unchanged — <a href="/hook/{{$.Webhook.ID}}/events/{{.ID}}/body" class="text-primary-600 hover:text-primary-700 underline">download the full body</a>.</p>
|
||||
<p class="mt-2 text-xs text-gray-500">Body truncated for display: showing {{.BodyShownBytes}} of {{.BodyBytes}} bytes. The stored body is unchanged — <a href="/source/{{$.Webhook.ID}}/logs/{{.ID}}/body" class="text-primary-600 hover:text-primary-700 underline">download the full body</a>.</p>
|
||||
{{end}}
|
||||
|
||||
{{if .Deliveries}}
|
||||
@@ -79,7 +79,7 @@
|
||||
</div>
|
||||
<div class="flex items-center gap-3">
|
||||
{{if .Status.Terminal}}
|
||||
<form method="POST" action="/hook/{{$.Webhook.ID}}/deliveries/{{.ID}}/replay" class="inline" @click.stop>
|
||||
<form method="POST" action="/source/{{$.Webhook.ID}}/deliveries/{{.ID}}/replay" class="inline" @click.stop>
|
||||
<input type="hidden" name="csrf_token" value="{{$.CSRFToken}}">
|
||||
<input type="hidden" name="page" value="{{$.Page}}">
|
||||
<button type="submit" class="text-xs text-primary-600 hover:text-primary-700" title="Send this event to the target again">Replay</button>
|
||||
@@ -139,11 +139,11 @@
|
||||
{{if or .HasPrev .HasNext}}
|
||||
<div class="flex justify-center gap-2 mt-6">
|
||||
{{if .HasPrev}}
|
||||
<a href="/hook/{{.Webhook.ID}}/events?page={{.PrevPage}}" class="btn-secondary text-sm">← Previous</a>
|
||||
<a href="/source/{{.Webhook.ID}}/logs?page={{.PrevPage}}" class="btn-secondary text-sm">← Previous</a>
|
||||
{{end}}
|
||||
<span class="inline-flex items-center px-4 py-2 text-sm text-gray-500">Page {{.Page}} of {{.TotalPages}}</span>
|
||||
{{if .HasNext}}
|
||||
<a href="/hook/{{.Webhook.ID}}/events?page={{.NextPage}}" class="btn-secondary text-sm">Next →</a>
|
||||
<a href="/source/{{.Webhook.ID}}/logs?page={{.NextPage}}" class="btn-secondary text-sm">Next →</a>
|
||||
{{end}}
|
||||
</div>
|
||||
{{end}}
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
<div class="max-w-6xl mx-auto px-6 py-8">
|
||||
<div class="flex justify-between items-center mb-6">
|
||||
<h1 class="text-2xl font-medium text-gray-900">Webhooks</h1>
|
||||
<a href="/hooks/new" class="btn-primary">
|
||||
<a href="/sources/new" class="btn-primary">
|
||||
<svg class="w-5 h-5 mr-1" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
||||
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M12 4v16m8-8H4"/>
|
||||
</svg>
|
||||
@@ -17,7 +17,7 @@
|
||||
{{if .Webhooks}}
|
||||
<div class="grid gap-4">
|
||||
{{range .Webhooks}}
|
||||
<a href="/hook/{{.ID}}" class="card-elevated p-6 block">
|
||||
<a href="/source/{{.ID}}" class="card-elevated p-6 block">
|
||||
<div class="flex justify-between items-start">
|
||||
<div>
|
||||
<h2 class="text-lg font-medium text-gray-900">{{.Name}}</h2>
|
||||
@@ -27,16 +27,10 @@
|
||||
</div>
|
||||
<span class="badge-info">Retention: {{.RetentionLabel}}</span>
|
||||
</div>
|
||||
<div class="flex flex-wrap gap-6 mt-4 text-sm text-gray-500">
|
||||
<span>{{.EntrypointCount}} entrypoint{{if ne .EntrypointCount 1}}s{{end}}{{if .InactiveEntrypointCount}}, {{.InactiveEntrypointCount}} inactive{{end}}</span>
|
||||
<span>{{.TargetCount}} target{{if ne .TargetCount 1}}s{{end}}{{if .InactiveTargetCount}}, {{.InactiveTargetCount}} inactive{{end}}</span>
|
||||
{{if .EventsUnreadable}}
|
||||
<span class="text-red-600">The event figures could not be read.</span>
|
||||
{{else}}
|
||||
<span>{{.EventCount}} event{{if ne .EventCount 1}}s{{end}} within retention</span>
|
||||
<span>{{with .LastEventAt}}Last event {{.UTC.Format "2006-01-02 15:04:05 UTC"}}{{else}}No events yet{{end}}</span>
|
||||
<span class="{{if .FailedLast24Hours}}font-medium text-red-600{{end}}">{{.FailedLast24Hours}} failed deliver{{if eq .FailedLast24Hours 1}}y{{else}}ies{{end}} in the last 24 hours</span>
|
||||
{{end}}
|
||||
<div class="flex gap-6 mt-4 text-sm text-gray-500">
|
||||
<span>{{.EntrypointCount}} entrypoint{{if ne .EntrypointCount 1}}s{{end}}</span>
|
||||
<span>{{.TargetCount}} target{{if ne .TargetCount 1}}s{{end}}</span>
|
||||
<span>{{.EventCount}} event{{if ne .EventCount 1}}s{{end}}</span>
|
||||
</div>
|
||||
</a>
|
||||
{{end}}
|
||||
@@ -48,7 +42,7 @@
|
||||
</svg>
|
||||
<h2 class="text-lg font-medium text-gray-900 mb-2">No webhooks yet</h2>
|
||||
<p class="text-gray-500 mb-6">Create your first webhook to start receiving and forwarding events.</p>
|
||||
<a href="/hooks/new" class="btn-primary">Create Webhook</a>
|
||||
<a href="/sources/new" class="btn-primary">Create Webhook</a>
|
||||
</div>
|
||||
{{end}}
|
||||
</div>
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
{{define "content"}}
|
||||
<div class="max-w-2xl mx-auto px-6 py-8">
|
||||
<div class="mb-6">
|
||||
<a href="/hooks" class="text-sm text-primary-600 hover:text-primary-700">← Back to webhooks</a>
|
||||
<a href="/sources" class="text-sm text-primary-600 hover:text-primary-700">← Back to webhooks</a>
|
||||
<h1 class="text-2xl font-medium text-gray-900 mt-2">Create Webhook</h1>
|
||||
</div>
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
<div class="alert-error">{{.Error}}</div>
|
||||
{{end}}
|
||||
|
||||
<form method="POST" action="/hooks/new" class="space-y-6">
|
||||
<form method="POST" action="/sources/new" class="space-y-6">
|
||||
<input type="hidden" name="csrf_token" value="{{.CSRFToken}}">
|
||||
<div class="form-group">
|
||||
<label for="name" class="label">Name</label>
|
||||
@@ -34,7 +34,7 @@
|
||||
|
||||
<div class="flex gap-3">
|
||||
<button type="submit" class="btn-primary">Create Webhook</button>
|
||||
<a href="/hooks" class="btn-secondary">Cancel</a>
|
||||
<a href="/sources" class="btn-secondary">Cancel</a>
|
||||
</div>
|
||||
</form>
|
||||
</div>
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
{{define "content"}}
|
||||
<div class="max-w-2xl mx-auto px-6 py-8">
|
||||
<div class="mb-6">
|
||||
<a href="/hook/{{.Webhook.ID}}" class="text-sm text-primary-600 hover:text-primary-700">← Back to {{.Webhook.Name}}</a>
|
||||
<a href="/source/{{.Webhook.ID}}" class="text-sm text-primary-600 hover:text-primary-700">← Back to {{.Webhook.Name}}</a>
|
||||
<h1 class="text-2xl font-medium text-gray-900 mt-2">Edit Target</h1>
|
||||
<p class="text-sm text-gray-500 mt-1">Type: {{.Target.Type}}. A target's type cannot be changed; create a new target to deliver a different way.</p>
|
||||
</div>
|
||||
@@ -21,7 +21,7 @@
|
||||
</div>
|
||||
{{end}}
|
||||
|
||||
<form method="POST" action="/hook/{{.Webhook.ID}}/targets/{{.Target.ID}}/edit" class="space-y-6">
|
||||
<form method="POST" action="/source/{{.Webhook.ID}}/targets/{{.Target.ID}}/edit" class="space-y-6">
|
||||
<input type="hidden" name="csrf_token" value="{{.CSRFToken}}">
|
||||
|
||||
<div class="form-group">
|
||||
@@ -75,7 +75,7 @@
|
||||
|
||||
<div class="flex gap-3">
|
||||
<button type="submit" class="btn-primary">Save Changes</button>
|
||||
<a href="/hook/{{.Webhook.ID}}" class="btn-secondary">Cancel</a>
|
||||
<a href="/source/{{.Webhook.ID}}" class="btn-secondary">Cancel</a>
|
||||
</div>
|
||||
</form>
|
||||
</div>
|
||||
|
||||
@@ -1,95 +0,0 @@
|
||||
{{define "webhook_stats"}}
|
||||
<!-- Statistics pane at the top of the webhook page. -->
|
||||
<div class="card mb-6">
|
||||
<div class="p-4 border-b border-gray-200">
|
||||
<h2 class="text-lg font-medium text-gray-900">Statistics</h2>
|
||||
</div>
|
||||
{{with .Stats}}
|
||||
<div class="p-4 flex flex-wrap gap-6 text-sm border-b border-gray-200">
|
||||
<div>
|
||||
<span class="text-gray-500">Entrypoints</span>
|
||||
<span class="font-medium text-gray-900">{{.Entrypoints}}</span>
|
||||
<span class="text-gray-500">({{.ActiveEntrypoints}} active)</span>
|
||||
</div>
|
||||
<div>
|
||||
<span class="text-gray-500">Targets</span>
|
||||
<span class="font-medium text-gray-900">{{.Targets}}</span>
|
||||
<span class="text-gray-500">({{.ActiveTargets}} active)</span>
|
||||
</div>
|
||||
<div>
|
||||
<span class="text-gray-500">Deliveries in progress</span>
|
||||
<span class="font-medium text-gray-900">{{.InProgress}}</span>
|
||||
</div>
|
||||
<div>
|
||||
<span class="text-gray-500">Last event</span>
|
||||
<span class="font-medium text-gray-900">{{with .LastEventAt}}{{.UTC.Format "2006-01-02 15:04:05 UTC"}}{{else}}none{{end}}</span>
|
||||
</div>
|
||||
<div>
|
||||
<span class="text-gray-500">Retention</span>
|
||||
<span class="font-medium text-gray-900">{{$.Webhook.RetentionLabel}}</span>
|
||||
</div>
|
||||
</div>
|
||||
<div class="p-4 grid grid-cols-1 lg:grid-cols-2 gap-6 text-sm">
|
||||
<div>
|
||||
<table class="w-full text-center text-gray-900">
|
||||
<thead>
|
||||
<tr class="border-b border-gray-200 text-xs text-gray-500 uppercase tracking-wide">
|
||||
<th></th>
|
||||
<th class="py-2 font-medium">Lifetime</th>
|
||||
<th class="py-2 font-medium">Within retention</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td class="py-2 text-left text-gray-600">Events</td>
|
||||
<td class="py-2">{{.Lifetime.Events}}</td>
|
||||
<td class="py-2">{{.WithinRetention.Events}}</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="py-2 text-left text-gray-600">Deliveries</td>
|
||||
<td class="py-2">{{.Lifetime.Deliveries}}</td>
|
||||
<td class="py-2">{{.WithinRetention.Deliveries}}</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="py-2 text-left text-gray-600">Failures</td>
|
||||
<td class="py-2">{{.Lifetime.Failures}}</td>
|
||||
<td class="py-2">{{.WithinRetention.Failures}}</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<div>
|
||||
<table class="w-full text-center text-gray-900">
|
||||
<thead>
|
||||
<tr class="border-b border-gray-200 text-xs text-gray-500 uppercase tracking-wide">
|
||||
<th></th>
|
||||
<th class="py-2 font-medium">Last 10 minutes</th>
|
||||
<th class="py-2 font-medium">Last 24 hours</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td class="py-2 text-left text-gray-600">Events</td>
|
||||
<td class="py-2">{{.Last10Minutes.Events}}</td>
|
||||
<td class="py-2">{{.Last24Hours.Events}}</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="py-2 text-left text-gray-600">Failures</td>
|
||||
<td class="py-2">{{.Last10Minutes.Failed}}</td>
|
||||
<td class="py-2">{{.Last24Hours.Failed}}</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="py-2 text-left text-gray-600">Failure percentage</td>
|
||||
<td class="py-2">{{.Last10Minutes.FailurePercent}}</td>
|
||||
<td class="py-2">{{.Last24Hours.FailurePercent}}</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p class="mt-2 text-xs text-gray-500">Failure percentage is the failed deliveries out of all deliveries that finished in the window. Deliveries still pending or retrying are not counted.</p>
|
||||
</div>
|
||||
</div>
|
||||
{{else}}
|
||||
<div class="p-4 text-sm text-gray-500">The statistics could not be read.</div>
|
||||
{{end}}
|
||||
</div>
|
||||
{{end}}
|
||||
Reference in New Issue
Block a user