Author SHA1 Message Date
clawbot 7287e163e7 Add a statistics pane to the webhook page (closes #368)
check / check (push) Successful in 3m58s
Each webhook's event database keeps one row of running totals: events,
deliveries and failures, and how many of each retention removed.
Storing an event, creating a delivery, a delivery becoming failed and
the retention sweep each update it in the transaction that writes or
deletes the rows it counts. Deliveries get a finished_at column, the
last column of the status index, so the last-10-minutes and
last-24-hours figures are index-range counts. The pane is its own
template, included at the top of the page.

The schema changes in place with nothing back-filled, so an existing
database must be recreated.

Model: opus-5-5
2026-10-01 19:49:18 +00:00
46 changed files with 1563 additions and 907 deletions
+8 -7
View File
@@ -1,15 +1,16 @@
# .git is deliberately NOT excluded: the build derives the version it stamps
# into the binary from it (script/version). Nor is any tracked file: git in
# the build would see it as deleted and mark the version -dirty. Only
# untracked files belong here.
#
# .ci-fingerprint is deliberately NOT excluded: it is the CI cache barrier
# that keeps the check stages from replaying a cached pass. See the lint
# stage of the Dockerfile.
.git/
bin/
# Extracted from 3p/ by `make assets` inside the build; a host copy is not
# needed. The tarball in 3p/ must stay in the context.
# Third-party browser assets are fetched and hash-verified inside the build by
# script/fetch-assets. Excluding any host copy keeps a developer's working tree
# from supplying the bytes that get shipped. The script and its
# static/vendor.sha256 manifest stay in the context.
static/js/alpine.min.js
*.md
LICENSE
.editorconfig
.env
.env.*
*.db
+6 -5
View File
@@ -28,11 +28,12 @@ jobs:
run: script/ci-mark-superseded
- name: Fingerprint the build context
# Every commit that changes more than docs writes a new fingerprint
# into the context, which invalidates the `COPY . .` layer of both
# check stages: a commit that was never linted, formatted-checked,
# tested and built cannot report success from cache. Docs-only
# commits rebuild too, since the context also carries `.git`.
# `.dockerignore` keeps docs out of the build context, so a docs-only
# commit legitimately replays the whole image from cache and stays
# cheap. Every other commit writes a new fingerprint into the context,
# which invalidates the `COPY . .` layer of both check stages: a
# commit that was never linted, formatted-checked, tested and built
# cannot report success from cache.
run: |
set -eu
fp="$(git log -1 --format=%H -- . ':!*.md' ':!LICENSE' ':!.editorconfig')"
+3 -2
View File
@@ -46,6 +46,7 @@ temp/
# CI cache barrier, written into the build context by the check workflow
.ci-fingerprint
# Alpine.js, extracted by `make assets` from its tarball in 3p/, which is
# what is committed.
# Third-party browser assets, fetched and hash-verified by
# script/fetch-assets against static/vendor.sha256. Not committed:
# REPO_POLICIES.md forbids minified bundles in version control.
/static/js/alpine.min.js
Binary file not shown.
+18 -19
View File
@@ -38,8 +38,8 @@ FROM golang:1.26.1-bookworm@sha256:4465644228bc2857a954b092167e12aa59c006a349228
COPY --from=lint /src/go.sum /dev/null
# jq is a runtime dependency of script/ci-mark-superseded, which the test
# suite executes. git is what script/version derives the version with.
RUN apt-get update && apt-get install -y --no-install-recommends make curl ca-certificates jq git && rm -rf /var/lib/apt/lists/*
# suite executes.
RUN apt-get update && apt-get install -y --no-install-recommends make curl ca-certificates jq && rm -rf /var/lib/apt/lists/*
WORKDIR /build
@@ -51,26 +51,25 @@ RUN go mod download
# the lint stage above.
COPY . .
# Run tests and build. Both first run script/assets, which extracts Alpine.js
# from its tarball in 3p/.
# Fetch the third-party browser assets the UI serves. They are not committed
# (REPO_POLICIES.md forbids minified bundles in version control) and
# .dockerignore keeps any host copy out of the build context, so this step is
# the only way they enter the image. Each download is checked against a
# hardcoded sha256 and the build fails on mismatch; make test re-checks the
# hashes against the bytes go:embed actually put in the binary.
RUN script/fetch-assets
# Run tests and build
RUN make test
# Version stamped into the binary: the VERSION build arg when one is
# given, otherwise what script/version derives from the .git the build
# context carries, so any `docker build .` of a clone stamps its commit.
# With neither, as from a source tarball, it is "unknown".
# Version stamped into the binary. .dockerignore excludes .git/, so
# nothing in this stage can derive it: script/docker resolves it on the
# host and passes it in. The default is what a bare `docker build .`
# with no --build-arg gets, and it names no tag the tree may not be at.
#
# Declared here, below the test step, so a changed version does not
# invalidate its cached layer.
ARG VERSION
# A context that carries .git must not stamp "unknown": that means git is
# missing here or refused to read the checkout, and the image could not be
# traced back to its commit.
RUN if [ -d .git ] && [ "$(make version VERSION="$VERSION")" = unknown ]; then \
echo "version is unknown although the build context carries .git" >&2; \
exit 1; \
fi
# Declared here, below the test and asset steps, so a changed version
# does not invalidate their cached layers.
ARG VERSION=unknown
RUN make build VERSION="$VERSION"
+7 -7
View File
@@ -4,12 +4,12 @@
.DEFAULT_GOAL := check
# Version stamped into the binary. Derived from git by script/version;
# override it (`make build VERSION=v1.2.3`) to stamp a given value, which is
# how the Dockerfile passes its build arg in.
# override it (`make build VERSION=v1.2.3`) where git metadata is
# unavailable, which is how the Dockerfile passes its build arg in.
VERSION ?= $(shell script/version)
# An empty override (`make build VERSION=`, or the Dockerfile's `make build
# VERSION="$VERSION"` when no VERSION build arg was given) means unset,
# An empty override (`make build VERSION=`, or a `--build-arg VERSION=`
# landing on the Dockerfile's `make build VERSION="$VERSION"`) means unset,
# exactly as it does in script/version -- stamping "" would leave the binary
# reporting no version and the footer back on its "dev" fallback. `override`
# is required: a plain assignment loses to the command-line definition it
@@ -28,7 +28,7 @@ setup:
@script/setup
assets:
@script/assets
@script/fetch-assets
test:
@script/test
@@ -45,13 +45,13 @@ fmt-check:
check:
@script/check
build: assets
build:
go build -ldflags '$(strip -X main.version=$(VERSION) $(GO_LDFLAGS))' -o bin/webhooker ./cmd/webhooker
run: build
./bin/webhooker
dev: assets
dev:
go run ./cmd/webhooker
deps:
+90 -69
View File
@@ -21,6 +21,9 @@ before deploying one.
- Go 1.26.1+ (the version in `go.mod`)
- Docker (for linting, for the test stage of the CI gate, and for
containerized deployment)
- `curl`, used by `script/fetch-assets` to download the third-party
browser assets, which are not committed (`make bootstrap` installs
it if missing)
golangci-lint is not a prerequisite and must not be installed on the
host: `script/bootstrap` does not install it, and `make lint` runs the
@@ -33,7 +36,9 @@ digest-pinned linter image via `Dockerfile.lint`.
git clone https://git.eeqj.de/sneak/webhooker.git
cd webhooker
# Install the Go toolchain if missing, and the Go dependencies
# Install Go dependencies and the third-party browser assets.
# `make deps` alone is not enough: it only runs go mod download/tidy,
# and the checks below need the fetched assets.
make bootstrap
# Run all checks (test, lint, format check)
@@ -53,7 +58,7 @@ make docker
```bash
make bootstrap # Install all dependencies (idempotent)
make setup # Bootstrap + install git pre-commit hook
make assets # Extract Alpine.js from 3p/ (test, check, build, dev run it)
make assets # Fetch + verify third-party browser assets
make fmt # Format code (gofmt + goimports)
make fmt-check # Fail if gofmt would change anything (writes nothing)
make lint # Run golangci-lint in Docker (Dockerfile.lint)
@@ -1065,7 +1070,7 @@ unconditionally against whatever files it finds:
- the main database on connect — `Setting`, `User`, `APIKey`, `Webhook`,
`Entrypoint`, `Target`
- each event database when it is lazily opened — `Event`, `Delivery`,
`DeliveryResult`
`DeliveryResult`, `Totals`
- each archive database on every open and reopen
There is no schema version table, no migration ledger, and no down
@@ -1123,21 +1128,13 @@ build itself.
| Uncommitted changes | the above with a `-dirty` suffix |
| No git metadata | `unknown` |
The image derives it the same way, from the `.git` that the build
context carries, so any `docker build .` of a clone stamps the commit it
was built from; a shallow clone of one branch has no tags and stamps the
short SHA. `.dockerignore` must therefore leave out neither `.git` nor
any tracked file, which git in the build would see as deleted, marking
the version `-dirty`. A `VERSION` build arg (`--build-arg VERSION=...`)
takes precedence; `script/docker` (and so `make docker`) passes the one
`script/version` resolves on the host. The image build fails if its
context carries `.git` and the version still comes out `unknown`, which
means git in the build could not read the checkout.
`unknown` is what a source tarball, or a `docker build` with no `.git`
in its context and no `VERSION` build arg, reports. A build that reports
`unknown` is a build nobody told what it was; it is not a failure, but
it cannot be traced back to a commit.
`unknown` is what a source tarball or a `docker build .` with no
`--build-arg VERSION=...` reports. `.dockerignore` excludes `.git/`, so
the build context carries no git metadata and the image cannot derive
the version itself: `script/docker` (and so `make docker`) resolves it
on the host and passes it in as the `VERSION` build arg. A build that
reports `unknown` is a build nobody told what it was; it is not a
failure, but it cannot be traced back to a commit.
`make version` prints what the current checkout would stamp, and
`make build VERSION=v1.2.3` overrides it. An empty override — from
@@ -1224,15 +1221,14 @@ This repository adheres to the
standard: normalized scripts in `script/` are the entrypoints for the
development workflow. Ten of the Makefile's seventeen targets are thin
shims that call them; `build`, `run`, `dev`, `deps`, `clean`, `css` and
`version` are inline commands with no script behind them, though `build`,
`run` and `dev` first run `script/assets`, and `build` and `version` both
take their value from `script/version`.
`version` are inline commands with no script behind them, though
`build` and `version` both take their value from `script/version`.
`script/test`, `make build` and `make dev` each run `script/assets`
first, which writes the ignored `static/js/alpine.min.js` (see
[Third-party browser assets](#third-party-browser-assets)), so
`make test`, `make check` and the pre-commit hook work on a fresh clone
without a separate step.
`make check` needs the third-party browser assets in `static/`, which
are not committed, so run `make bootstrap` (or just `make assets`) once
after cloning. Without them the tests fail with a message naming that
remedy. `make check` does not fetch them itself because it must not
change any files in the repo.
We provide:
@@ -1240,8 +1236,8 @@ We provide:
- `script/setup` — make a fresh clone ready for development
(bootstrap, then install-precommit)
- `script/projectname` — output the project name ("webhooker")
- `script/assets` — extract Alpine.js from its tarball in `3p/` (see
[Third-party browser assets](#third-party-browser-assets))
- `script/fetch-assets` — download the third-party browser assets into
`static/`, verifying each against its pinned sha256
- `script/test` — run the test suite
- `script/lint` — run golangci-lint in Docker (see Linting below)
- `script/fmt` — format all code (writes)
@@ -1263,25 +1259,24 @@ We provide:
## Third-party browser assets
The web UI serves one third-party script, Alpine.js. Its npm package tarball
is committed as `3p/alpinejs-3.14.9.tgz`, byte for byte as the npm registry
publishes it. It is a dependency, not this repo's build output, so
`REPO_POLICIES.md`'s rule against committed build artifacts does not apply.
The directory is `3p/` rather than `vendor/` because Go treats a root
`vendor/` directory as its module vendor directory.
The web UI serves one third-party script, Alpine.js. It is **not** committed:
a minified bundle in the tree is unreviewable, and `REPO_POLICIES.md` bars
both committed build artifacts and unpinned external references.
`script/assets` (`make assets`) extracts the browser build,
`package/dist/cdn.min.js`, from the tarball to `static/js/alpine.min.js`,
where `go:embed` picks it up. `script/test`, `make build` and `make dev` run
it first, and the Dockerfile builds through `make test` and `make build`, so
nothing downloads Alpine.js. The extracted file is not committed, and
`.dockerignore` keeps any host copy out of the build context.
Instead `script/fetch-assets` downloads it from a pinned URL, checks the
download against a hardcoded sha256, and installs it under `static/`. The
sha256 of every installed asset is recorded in `static/vendor.sha256`, and
`static/vendor_test.go` re-hashes the bytes `go:embed` put in the binary
against that manifest — so the pin is enforced on what actually ships, not
merely written down. Any mismatch fails the build.
To move to a new version: download
`https://registry.npmjs.org/alpinejs/-/alpinejs-<version>.tgz`, check it
against the `dist.integrity` hash listed at
`https://registry.npmjs.org/alpinejs/<version>`, replace the tarball in `3p/`
with it, update its file name in `script/assets`, and run `make check`.
`make bootstrap` runs the fetch for local development, and the Dockerfile
runs it in the build stage; `.gitignore` and `.dockerignore` keep the
artifact out of both the repo and the build context.
To move to a new version: update the version, URL, and tarball sha256 in
`script/fetch-assets` and the asset sha256 in `static/vendor.sha256`, then
run `make assets && make check`.
## Rationale
@@ -1389,7 +1384,7 @@ The codebase uses consistent naming throughout (rename completed in
### Data Model
webhooker's data model has nine entities organized into two tiers: the
webhooker's data model has ten entities organized into two tiers: the
**application tier** (user and webhook configuration) and the **event
tier** (event ingestion, delivery, and logging).
@@ -1418,6 +1413,10 @@ tier** (event ingestion, delivery, and logging).
│ ┌──────────┐ ┌──────────┐ ┌─────────────────┐ │
│ │ Event │──1:N──│ Delivery │──1:N──│ DeliveryResult │ │
│ └──────────┘ └──────────┘ └─────────────────┘ │
│ │
│ ┌──────────┐ │
│ │ Totals │ (one row of running counts) │
│ └──────────┘ │
└─────────────────────────────────────────────────────────────┘
```
@@ -1650,7 +1649,6 @@ data for auditing, for replay, and for resubmission.
| `headers` | JSON | Complete request headers |
| `body` | text | Raw request body |
| `content_type` | string | Content-Type header value |
| `body_bytes` | integer | The body's size in bytes, recorded when the event is stored, on receipt and on resubmit |
| `resubmitted_from_id` | UUID | The event this one was copied from by a resubmit (nullable; empty for an event that arrived on the receiver). Not a foreign key: the source event can be reaped by retention while its copies remain |
**Relations:** Belongs to Webhook. Belongs to Entrypoint. Has many
@@ -1671,6 +1669,7 @@ status across potentially multiple attempts.
| `event_id` | UUID | Foreign key → Event |
| `target_id`| UUID | Foreign key → Target |
| `status` | DeliveryStatus | One of: `pending`, `delivered`, `failed`, `retrying` |
| `finished_at` | timestamp | When the delivery became `delivered` or `failed` (nullable; empty while `pending` or `retrying`) |
**Relations:** Belongs to Event. Belongs to Target. Has many
DeliveryResults.
@@ -1738,6 +1737,29 @@ retries) is individually logged for full observability.
**Relations:** Belongs to Delivery.
#### Totals
The one row of running counts in each event database, read by the
statistics pane at the top of the webhook page.
| Field | Type | Description |
| -------------------- | ------- | ----------- |
| `events` | integer | Events ever stored, resubmitted copies included |
| `deliveries` | integer | Deliveries ever created, replays included |
| `failures` | integer | Deliveries that ever became `failed` |
| `events_removed` | integer | Events retention has deleted |
| `deliveries_removed` | integer | Deliveries retention has deleted |
| `failures_removed` | integer | Failed deliveries retention has deleted |
Each count changes in the transaction that writes or deletes the rows it
counts. The pane shows each of the first three as a lifetime figure, and
less what retention removed as the figure within retention, so neither
needs the rows themselves. Its last-10-minutes and last-24-hours figures
are counted from the `events` and `deliveries` indexes over just that
window. Its failure percentage for a window is the deliveries that became
`failed` in it out of all that became `delivered` or `failed` in it, and
a dash when none did.
#### Event-tier indexes
These indexes on the per-webhook event databases are declared in the model
@@ -1745,10 +1767,10 @@ tags, so `AutoMigrate` creates them on a fresh and on an existing database:
| Table | Columns | Serves |
| ------------------ | --------------------------- | ------ |
| `deliveries` | `status`, `deleted_at` | Startup recovery, the retry and pending sweeps every 60 seconds and the queue-depth sampler every 30 seconds, which select deliveries by status |
| `deliveries` | `status`, `deleted_at`, `finished_at` | Startup recovery, the retry and pending sweeps every 60 seconds and the queue-depth sampler every 30 seconds, which select deliveries by status, and the webhook page's statistics, which count deliveries by status and when they finished |
| `deliveries` | `event_id`, `deleted_at` | The event log, which loads each event's deliveries, and retention, which selects and deletes the deliveries of expired events |
| `delivery_results` | `delivery_id`, `deleted_at` | The event log, which loads the attempts of a page's deliveries, and retention, which deletes the attempts of expired events |
| `events` | `deleted_at`, `created_at` | Retention, which selects expired events by age |
| `events` | `deleted_at`, `created_at` | Retention, which selects expired events by age, and the webhook page's statistics, which count recent events and find the newest |
| `events` | `created_at` | Retention's delete of the expired events themselves |
GORM's soft delete adds `deleted_at IS NULL` to these queries; retention's
@@ -1762,9 +1784,10 @@ and SQLite narrows by a `<` only on the last column it uses.
#### Common Fields
Every entity except `Setting` includes these fields from `BaseModel`.
`Setting` is a bare key-value row with no `id`, no timestamps and no
soft delete:
Every entity except `Setting` and `Totals` includes these fields from
`BaseModel`. `Setting` is a bare key-value row with no `id`, no
timestamps and no soft delete, and `Totals` is a single row of counts
with only a numeric `id`:
| Field | Type | Description |
| ------------ | --------- | ----------- |
@@ -1806,6 +1829,7 @@ encryption key is generated and stored, and an `admin` user is created.
- **Events** — captured incoming webhook payloads
- **Deliveries** — event-to-target pairings and their status
- **DeliveryResults** — individual delivery attempt logs
- **Totals** — running counts of the above, kept through retention
Per-webhook databases are created automatically when a webhook is
created (and lazily on first access for webhooks that predate this
@@ -2761,8 +2785,6 @@ imports. The entry point is `cmd/webhooker/main.go`.
```
webhooker/
├── 3p/
│ └── alpinejs-3.14.9.tgz # Alpine.js npm package, extracted by make assets
├── cmd/webhooker/
│ └── main.go # Entry point: subcommand dispatch; no args locks DATA_DIR and wires fx
├── internal/
@@ -2856,7 +2878,8 @@ webhooker/
│ ├── css/tailwind.css # Generated stylesheet the pages load
│ ├── css/style.css # Older hand-written stylesheet, no longer loaded
│ ├── js/app.js # Progressive-enhancement copy-to-clipboard
│ └── js/alpine.min.js # Alpine.js, extracted from 3p/ by make assets, not committed
│ ├── js/alpine.min.js # Alpine.js, fetched by script/fetch-assets, not committed
│ └── vendor.sha256 # Pinned hashes the fetched assets are verified against
├── templates/ # Go HTML templates (base, login, sources, etc.)
├── script/ # Scripts to Rule Them All entrypoints
├── Dockerfile # Three stages: lint, test+build, Alpine runtime
@@ -3168,15 +3191,14 @@ version is fixed independently of the compiler's:
`make fmt-check`, then `golangci-lint config verify` and
`golangci-lint run`, both with `--network=none`.
2. **Builder stage** (`golang:1.26.1-bookworm`) — depends on the lint
stage passing (it copies a file from it), runs `make test` and
`make build` (both extract Alpine.js from `3p/` first), and finally
rebuilds the binary with `CGO_ENABLED=1` and static linking so it
runs on musl. Both builds go through `make build`, the relink adding
its `-extldflags` via `GO_LDFLAGS`, so neither can drop the `-X` that
stamps the version. The version is the `VERSION` build arg if one is
given, otherwise derived from the `.git` in the context, and the
stage fails if a context with `.git` would stamp `unknown` (see
[Version stamping](#version-stamping)).
stage passing (it copies a file from it), runs `script/fetch-assets`
to download and verify the third-party browser assets, then runs
`make test` and `make build`, and finally rebuilds the binary with
`CGO_ENABLED=1` and static linking so it runs on musl. Both builds
go through `make build`, the relink adding its `-extldflags` via
`GO_LDFLAGS`, so neither can drop the `-X` that stamps the version.
The version arrives as the `VERSION` build arg, since the context
has no `.git` (see [Version stamping](#version-stamping)).
3. **Runtime stage** (`alpine:3.21`) — copies the static binary and
`deploy/docker-entrypoint.sh`, creates the `/var/lib/webhooker`
directory for all SQLite databases, exposes port 8080, and includes
@@ -3207,17 +3229,16 @@ A layer cache lets `docker build .` exit 0 in seconds with the lint and
test stages replayed rather than executed, which would make a green
check meaningless. The `check` workflow therefore writes
`.ci-fingerprint` into the build context before building. Its value is
the hash of the last commit that touched anything other than `*.md`,
`LICENSE` and `.editorconfig`, so:
the hash of the last commit that touched the build context, so:
- Any commit that changes code (including a squash merge whose tree
matches an already-built branch) gets a new fingerprint, invalidates
the `COPY . .` layer of both check stages, and really runs
`make fmt-check`, `golangci-lint`, `make test`, and `make build`. A
run that reports success ran them.
- A docs-only commit leaves the fingerprint unchanged, but it still
rebuilds in full: the context also carries `.git`, which changes with
every commit (see [Version stamping](#version-stamping)).
- A docs-only commit leaves the fingerprint unchanged — `.dockerignore`
excludes `*.md`, `LICENSE` and `.editorconfig` from the context
anyway — so the image replays from cache and costs seconds.
The module download layer sits above `COPY . .` and stays cached either
way.
+1 -1
View File
@@ -4,7 +4,6 @@ go 1.26.1
require (
github.com/99designs/basicauth-go v0.0.0-20230316000542-bf6f9cbbf0f8
github.com/dustin/go-humanize v1.0.1
github.com/getsentry/sentry-go v0.25.0
github.com/go-chi/chi v1.5.5
github.com/go-chi/cors v1.2.1
@@ -30,6 +29,7 @@ require (
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/gorilla/securecookie v1.1.2 // indirect
github.com/jinzhu/inflection v1.0.0 // indirect
github.com/jinzhu/now v1.1.5 // indirect
+69 -5
View File
@@ -93,6 +93,7 @@ func TestEventTierQueriesUseTheirIndexes(t *testing.T) {
deliveries []database.Delivery
results []database.DeliveryResult
depths []struct{ Depth int }
failed struct{ Count int64 }
)
byStatus := "idx_deliveries_status (status=? AND deleted_at=?)"
@@ -123,7 +124,7 @@ func TestEventTierQueriesUseTheirIndexes(t *testing.T) {
Order("attempt_num ASC").Find(&results),
"idx_delivery_results_delivery_id (delivery_id=? AND deleted_at=?)")
// Retention's three deletes (reapExpired), whose subqueries are built
// Retention's deletes (deleteExpired), whose subqueries are built
// afresh for each statement as it builds them.
expiredEventIDs := func() *gorm.DB {
return dry.Model(&database.Event{}).Select("id").
@@ -135,6 +136,12 @@ func TestEventTierQueriesUseTheirIndexes(t *testing.T) {
Select("id").Where("event_id IN (?)", expiredEventIDs()),
).Delete(&database.DeliveryResult{}),
"idx_delivery_results_delivery_id (delivery_id=?)", byEvent, byAge)
assertPlanUses(t, db, dry.Unscoped().Model(&database.Delivery{}).
Select("count(CASE WHEN status = ? THEN 1 END) AS count",
database.DeliveryStatusFailed).
Where("event_id IN (?)", expiredEventIDs()).
Take(&failed),
"idx_deliveries_event_id (event_id=?)", byAge)
assertPlanUses(t, db, dry.Unscoped().Where(
"event_id IN (?)", expiredEventIDs(),
).Delete(&database.Delivery{}),
@@ -144,6 +151,54 @@ func TestEventTierQueriesUseTheirIndexes(t *testing.T) {
).Delete(&database.Event{}), "idx_events_created_at (created_at<?)")
}
// TestStatisticsQueriesUseTheirIndexes does the same for the webhook
// page's statistics (readEventStats in the handlers): deliveries in
// progress, deliveries finished and events received since a time, and
// the newest event, which must come straight off an index rather than
// from sorting every event.
func TestStatisticsQueriesUseTheirIndexes(t *testing.T) {
t.Parallel()
mgr, lc := setupTestWebhookDBManager(t)
ctx := context.Background()
require.NoError(t, lc.Start(ctx))
defer func() { require.NoError(t, lc.Stop(ctx)) }()
db, err := mgr.GetDB(uuid.New().String())
require.NoError(t, err)
dry := db.Session(&gorm.Session{DryRun: true})
since := time.Now()
var (
count int64
newest []time.Time
)
assertPlanUses(t, db, dry.Model(&database.Delivery{}).
Where("status IN ?", []database.DeliveryStatus{
database.DeliveryStatusPending,
database.DeliveryStatusRetrying,
}).Count(&count),
"idx_deliveries_status (status=? AND deleted_at=?)")
assertPlanUses(t, db, dry.Model(&database.Delivery{}).
Where("status = ? AND finished_at >= ?",
database.DeliveryStatusFailed, since).Count(&count),
"idx_deliveries_status "+
"(status=? AND deleted_at=? AND finished_at>?)")
assertPlanUses(t, db, dry.Model(&database.Event{}).
Where("created_at >= ?", since).Count(&count),
"idx_events_deleted_at_created_at "+
"(deleted_at=? AND created_at>?)")
newestEvent := dry.Model(&database.Event{}).
Order("created_at DESC").Limit(1).Pluck("created_at", &newest)
assertPlanUses(t, db, newestEvent,
"idx_events_deleted_at_created_at (deleted_at=?)")
assert.NotContains(t, queryPlan(t, db, newestEvent), "TEMP B-TREE")
}
// assertPlanUses asserts that SQLite's plan for a statement GORM built
// in a dry run, run with the same SQL and arguments GORM would send,
// names each of the given indexes.
@@ -152,6 +207,18 @@ func assertPlanUses(
) {
t.Helper()
plan := queryPlan(t, db, built)
for _, index := range indexes {
assert.Contains(t, plan, index, built.Statement.SQL.String())
}
}
// queryPlan returns SQLite's plan for a statement GORM built in a dry
// run, run with the same SQL and arguments GORM would send.
func queryPlan(t *testing.T, db, built *gorm.DB) string {
t.Helper()
var plan []struct{ Detail string }
require.NoError(t, db.Raw(
@@ -159,8 +226,5 @@ func assertPlanUses(
built.Statement.Vars...,
).Scan(&plan).Error)
for _, index := range indexes {
assert.Contains(t, fmt.Sprint(plan), index,
built.Statement.SQL.String())
}
return fmt.Sprint(plan)
}
+11 -1
View File
@@ -1,6 +1,10 @@
package database
import "gorm.io/gorm"
import (
"time"
"gorm.io/gorm"
)
// DeliveryStatus represents the status of a delivery
type DeliveryStatus string
@@ -45,6 +49,12 @@ type Delivery struct {
// gives.
DeletedAt gorm.DeletedAt `gorm:"index:idx_deliveries_event_id,priority:2;index:idx_deliveries_status,priority:2" json:"deletedAt,omitzero"`
// FinishedAt is when the delivery became delivered or failed, and
// nil while it is pending or retrying. It ends the status index,
// so the webhook page counts the deliveries that finished in a
// recent window by reading that window from the index.
FinishedAt *time.Time `gorm:"index:idx_deliveries_status,priority:3" json:"finishedAt,omitempty"`
// Relations
Event Event `json:"event,omitzero"`
Target Target `json:"target,omitzero"`
-5
View File
@@ -31,11 +31,6 @@ type Event struct {
Body string `gorm:"type:text" json:"body"`
ContentType string `json:"contentType"`
// BodyBytes is the size of Body in bytes, recorded when the event
// is stored so the recent events list can show it without reading
// the body.
BodyBytes int64 `gorm:"not null" json:"bodyBytes"`
// ResubmittedFromID names the event this one was copied from by
// an operator resubmit. It is nil for an event that arrived on
// the receiver, which is every event created before the column
+73
View File
@@ -0,0 +1,73 @@
package database
import (
"fmt"
"gorm.io/gorm"
)
// Totals is the single row of running totals in a webhook's event
// database. It is what keeps the webhook page's lifetime figures right
// after retention has removed the rows they count, and what lets the
// page show them without counting every row.
//
// Storing an event, creating a delivery and failing a delivery each
// add one, and retention adds what it deletes to the Removed columns.
// Every addition goes through AddTotals, in the transaction that
// writes or deletes the rows it counts.
type Totals struct {
ID int64 `gorm:"primaryKey"`
Events int64 `gorm:"not null"`
Deliveries int64 `gorm:"not null"`
Failures int64 `gorm:"not null"`
EventsRemoved int64 `gorm:"not null"`
DeliveriesRemoved int64 `gorm:"not null"`
FailuresRemoved int64 `gorm:"not null"`
}
// TableName names the table AddTotals updates.
func (Totals) TableName() string {
return "totals"
}
// EventsWithinRetention is how many of the webhook's events are still
// stored.
func (t Totals) EventsWithinRetention() int64 {
return t.Events - t.EventsRemoved
}
// DeliveriesWithinRetention is how many of the webhook's deliveries
// are still stored.
func (t Totals) DeliveriesWithinRetention() int64 {
return t.Deliveries - t.DeliveriesRemoved
}
// FailuresWithinRetention is how many of the webhook's failed
// deliveries are still stored.
func (t Totals) FailuresWithinRetention() int64 {
return t.Failures - t.FailuresRemoved
}
// AddTotals adds each count in add to the webhook's running totals.
// Call it on the transaction that writes or deletes the rows it
// counts, so the totals change exactly when those rows do.
func AddTotals(tx *gorm.DB, add Totals) error {
err := tx.Exec(
`UPDATE totals SET
events = events + ?,
deliveries = deliveries + ?,
failures = failures + ?,
events_removed = events_removed + ?,
deliveries_removed = deliveries_removed + ?,
failures_removed = failures_removed + ?`,
add.Events, add.Deliveries, add.Failures,
add.EventsRemoved, add.DeliveriesRemoved, add.FailuresRemoved,
).Error
if err != nil {
return fmt.Errorf("adding to running totals: %w", err)
}
return nil
}
+1 -1
View File
@@ -2,7 +2,7 @@ package database
// Migrate runs database migrations for the main application database.
// Only configuration-tier models are stored in the main database.
// Event-tier models (Event, Delivery, DeliveryResult) live in
// Event-tier models (Event, Delivery, DeliveryResult, Totals) live in
// per-webhook dedicated databases managed by WebhookDBManager.
func (d *Database) Migrate() error {
return d.db.AutoMigrate(
+59 -13
View File
@@ -267,55 +267,101 @@ func retentionCutoff(
// reapExpired hard-deletes, in foreign-key-safe order, the delivery
// results, deliveries, and events associated with events older than
// cutoff. Deletes are unscoped so rows are physically removed rather
// than soft-deleted, reclaiming disk. It returns the number of events
// deleted.
// cutoff, and adds what it deleted to the running totals, all in one
// transaction. Deletes are unscoped so rows are physically removed
// rather than soft-deleted, reclaiming disk. It returns the number of
// events deleted.
func reapExpired(db *gorm.DB, cutoff time.Time) (int64, error) {
var removed Totals
err := db.Transaction(func(tx *gorm.DB) error {
var err error
removed, err = deleteExpired(tx, cutoff)
if err != nil {
return err
}
return AddTotals(tx, removed)
})
if err != nil {
return 0, err
}
return removed.EventsRemoved, nil
}
// deleteExpired runs reapExpired's deletes and returns how many
// events, deliveries and failed deliveries they removed.
func deleteExpired(tx *gorm.DB, cutoff time.Time) (Totals, error) {
var removed Totals
// Fresh subqueries are built per statement to avoid reusing a
// mutated builder across executions.
expiredEventIDs := func() *gorm.DB {
return db.Model(&Event{}).
return tx.Model(&Event{}).
Select("id").
Where("created_at < ?", cutoff)
}
expiredDeliveryIDs := func() *gorm.DB {
return db.Model(&Delivery{}).
return tx.Model(&Delivery{}).
Select("id").
Where("event_id IN (?)", expiredEventIDs())
}
// 1. Delivery results whose delivery belongs to an expired event.
res := db.Unscoped().
res := tx.Unscoped().
Where("delivery_id IN (?)", expiredDeliveryIDs()).
Delete(&DeliveryResult{})
if res.Error != nil {
return 0, fmt.Errorf(
return removed, fmt.Errorf(
"deleting expired delivery results: %w",
res.Error,
)
}
// 2. Deliveries belonging to an expired event.
del := db.Unscoped().
// 2. Deliveries belonging to an expired event, after counting the
// failed ones among them. The status is tested in the select list
// rather than the WHERE clause: there, SQLite would read every
// failed delivery the webhook has through the status index,
// instead of only the expired ones through the event_id index.
var failed struct{ Count int64 }
err := tx.Unscoped().Model(&Delivery{}).
Select("count(CASE WHEN status = ? THEN 1 END) AS count",
DeliveryStatusFailed).
Where("event_id IN (?)", expiredEventIDs()).
Take(&failed).Error
if err != nil {
return removed, fmt.Errorf(
"counting expired failed deliveries: %w", err,
)
}
del := tx.Unscoped().
Where("event_id IN (?)", expiredEventIDs()).
Delete(&Delivery{})
if del.Error != nil {
return 0, fmt.Errorf(
return removed, fmt.Errorf(
"deleting expired deliveries: %w",
del.Error,
)
}
// 3. The expired events themselves.
ev := db.Unscoped().
ev := tx.Unscoped().
Where("created_at < ?", cutoff).
Delete(&Event{})
if ev.Error != nil {
return 0, fmt.Errorf(
return removed, fmt.Errorf(
"deleting expired events: %w",
ev.Error,
)
}
return ev.RowsAffected, nil
removed.EventsRemoved = ev.RowsAffected
removed.DeliveriesRemoved = del.RowsAffected
removed.FailuresRemoved = failed.Count
return removed, nil
}
+126
View File
@@ -0,0 +1,126 @@
package database_test
import (
"context"
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// readTotals reads a webhook database's row of running totals,
// asserting that it has exactly one.
func readTotals(t *testing.T, db *gorm.DB) database.Totals {
t.Helper()
var rows []database.Totals
require.NoError(t, db.Find(&rows).Error)
require.Len(t, rows, 1)
return rows[0]
}
// TestWebhookDBManager_TotalsRowSurvivesReopen verifies that a new
// event database starts with one row of zero totals, and that opening
// it again keeps that row and what was added to it.
func TestWebhookDBManager_TotalsRowSurvivesReopen(t *testing.T) {
t.Parallel()
mgr, lc := setupTestWebhookDBManager(t)
ctx := context.Background()
require.NoError(t, lc.Start(ctx))
defer func() { require.NoError(t, lc.Stop(ctx)) }()
webhookID := uuid.New().String()
db, err := mgr.GetDB(webhookID)
require.NoError(t, err)
fresh := readTotals(t, db)
assert.Equal(t, database.Totals{ID: fresh.ID}, fresh)
require.NoError(t, database.AddTotals(db, database.Totals{
Events: 2, Deliveries: 3, Failures: 1,
}))
// Drop the cached connection so the next open reopens the file,
// as a restart would.
require.NoError(t, mgr.CloseAll())
db, err = mgr.GetDB(webhookID)
require.NoError(t, err)
assert.Equal(t, database.Totals{
ID: fresh.ID, Events: 2, Deliveries: 3, Failures: 1,
}, readTotals(t, db))
}
// TestRetentionReaper_AddsWhatItRemovesToTotals verifies that a sweep
// leaves the lifetime totals alone and adds the events, deliveries and
// failed deliveries it deletes to the removed totals, so the totals
// within retention match the rows still stored.
func TestRetentionReaper_AddsWhatItRemovesToTotals(t *testing.T) {
t.Parallel()
env := setupRetentionTest(t)
webhookID := createWebhook(t, env.mainDB.DB(), 30)
db, err := env.mgr.GetDB(webhookID)
require.NoError(t, err)
now := time.Now()
expired := now.Add(-40 * 24 * time.Hour)
seedEventChain(t, db, webhookID, expired)
expiredFailure := seedEventChain(t, db, webhookID, expired)
recentFailure := seedEventChain(
t, db, webhookID, now.Add(-24*time.Hour),
)
for _, id := range []string{
expiredFailure.deliveryID, recentFailure.deliveryID,
} {
require.NoError(t, db.Model(&database.Delivery{}).
Where("id = ?", id).
Update("status", database.DeliveryStatusFailed).Error)
}
// The totals storing those rows would have left.
require.NoError(t, database.AddTotals(db, database.Totals{
Events: 3, Deliveries: 3, Failures: 2,
}))
env.reaper.ExportSweep(context.Background())
totals := readTotals(t, db)
assert.Equal(t, database.Totals{
ID: totals.ID,
Events: 3, Deliveries: 3, Failures: 2,
EventsRemoved: 2, DeliveriesRemoved: 2, FailuresRemoved: 1,
}, totals)
var events, deliveries, failures int64
require.NoError(t, db.Model(&database.Event{}).Count(&events).Error)
require.NoError(t, db.Model(&database.Delivery{}).
Count(&deliveries).Error)
require.NoError(t, db.Model(&database.Delivery{}).
Where("status = ?", database.DeliveryStatusFailed).
Count(&failures).Error)
assert.Equal(t, events, totals.EventsWithinRetention())
assert.Equal(t, deliveries, totals.DeliveriesWithinRetention())
assert.Equal(t, failures, totals.FailuresWithinRetention())
// A sweep with nothing left to remove changes nothing.
env.reaper.ExportSweep(context.Background())
assert.Equal(t, totals, readTotals(t, db))
}
+14 -2
View File
@@ -35,7 +35,8 @@ var errInvalidCachedDBType = errors.New(
// WebhookDBManager manages per-webhook SQLite database files
// for event storage. Each webhook gets its own dedicated
// database containing Events, Deliveries, and DeliveryResults.
// database containing Events, Deliveries, DeliveryResults and the
// running Totals of them.
// Database connections are opened lazily and cached.
type WebhookDBManager struct {
dataDir string
@@ -294,7 +295,7 @@ func (m *WebhookDBManager) openDB(
// Run migrations for event-tier models only
err = db.AutoMigrate(
&Event{}, &Delivery{}, &DeliveryResult{},
&Event{}, &Delivery{}, &DeliveryResult{}, &Totals{},
)
if err != nil {
_ = sqlDB.Close()
@@ -305,6 +306,17 @@ func (m *WebhookDBManager) openDB(
)
}
// A new database gets its row of running totals, all zero.
err = db.FirstOrCreate(&Totals{}).Error
if err != nil {
_ = sqlDB.Close()
return nil, fmt.Errorf(
"creating running totals for webhook database %s: %w",
webhookID, err,
)
}
m.log.Info(
"opened per-webhook database",
"webhook_id", webhookID,
+100
View File
@@ -0,0 +1,100 @@
package delivery_test
import (
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// failureTotal reads the running failure total of a webhook database.
func failureTotal(t *testing.T, db *gorm.DB) int64 {
t.Helper()
var totals database.Totals
require.NoError(t, db.Take(&totals).Error)
return totals.Failures
}
// TestUpdateDeliveryStatus_FinishTimeAndFailureTotal pins what a status
// write records for the webhook page's statistics: the time a delivery
// finished, set only when it becomes delivered or failed, and one more
// on the failure total when it fails.
func TestUpdateDeliveryStatus_FinishTimeAndFailureTotal(t *testing.T) {
t.Parallel()
tests := []struct {
status database.DeliveryStatus
finished bool
failures int64
}{
{database.DeliveryStatusRetrying, false, 0},
{database.DeliveryStatusDelivered, true, 0},
{database.DeliveryStatusFailed, true, 1},
}
for _, tt := range tests {
t.Run(string(tt.status), func(t *testing.T) {
t.Parallel()
db := testWebhookDB(t)
e := testEngine(t, 1)
event := seedEvent(t, db, `{}`)
d := seedDelivery(
t, db, event.ID, uuid.New().String(),
database.DeliveryStatusPending,
)
before := time.Now()
require.NoError(t, e.ExportUpdateDeliveryStatus(
db, &d, tt.status,
))
var stored database.Delivery
require.NoError(t, db.First(&stored, "id = ?", d.ID).Error)
assert.Equal(t, tt.status, stored.Status)
if tt.finished {
require.NotNil(t, stored.FinishedAt)
assert.False(t, stored.FinishedAt.Before(before))
} else {
assert.Nil(t, stored.FinishedAt)
}
assert.Equal(t, tt.failures, failureTotal(t, db))
})
}
}
// TestUpdateDeliveryStatus_DeletedDeliveryIsNotCounted covers a
// delivery retention deleted while the engine still held it. Failing
// it afterwards writes no row, so it adds no failure either: retention
// has already counted what it removed.
func TestUpdateDeliveryStatus_DeletedDeliveryIsNotCounted(t *testing.T) {
t.Parallel()
db := testWebhookDB(t)
e := testEngine(t, 1)
event := seedEvent(t, db, `{}`)
d := seedDelivery(
t, db, event.ID, uuid.New().String(),
database.DeliveryStatusRetrying,
)
require.NoError(t, db.Unscoped().
Delete(&database.Delivery{}, "id = ?", d.ID).Error)
require.NoError(t, e.ExportUpdateDeliveryStatus(
db, &d, database.DeliveryStatusFailed,
))
assert.Zero(t, failureTotal(t, db))
}
+30 -2
View File
@@ -1554,8 +1554,9 @@ func (e *Engine) updateDeliveryStatus(
targetType database.TargetType,
status database.DeliveryStatus,
) error {
err := webhookDB.Model(d).
Update("status", status).Error
err := webhookDB.Transaction(func(tx *gorm.DB) error {
return writeDeliveryStatus(tx, d, status)
})
if err != nil {
return fmt.Errorf(
"updating delivery %s to status %s: %w",
@@ -1574,6 +1575,33 @@ func (e *Engine) updateDeliveryStatus(
return nil
}
// writeDeliveryStatus writes a delivery's new status. A delivery that
// becomes delivered or failed also gets the time it finished, and a
// failed one is added to the webhook's running failure total. The
// failure is counted only if the row was still there to update:
// retention may have deleted it while the engine was working on it.
func writeDeliveryStatus(
tx *gorm.DB,
d *database.Delivery,
status database.DeliveryStatus,
) error {
columns := map[string]any{"status": status}
if status.Terminal() {
columns["finished_at"] = time.Now()
}
res := tx.Model(d).Updates(columns)
if res.Error != nil {
return res.Error
}
if status == database.DeliveryStatusFailed && res.RowsAffected > 0 {
return database.AddTotals(tx, database.Totals{Failures: 1})
}
return nil
}
// settleStatus moves a delivery to its outcome status and reports a
// failed write through bookkeepingFailed, which leaves the row
// recoverable. It exists so the target call sites read as one
+2
View File
@@ -57,7 +57,9 @@ func testWebhookDB(t *testing.T) *gorm.DB {
&database.Event{},
&database.Delivery{},
&database.DeliveryResult{},
&database.Totals{},
))
require.NoError(t, db.Create(&database.Totals{}).Error)
return db
}
+10
View File
@@ -150,6 +150,16 @@ func (e *Engine) ExportDeliverSlack(
)
}
// ExportUpdateDeliveryStatus exposes updateDeliveryStatus. It passes no
// target type, so no metric moves.
func (e *Engine) ExportUpdateDeliveryStatus(
webhookDB *gorm.DB,
d *database.Delivery,
status database.DeliveryStatus,
) error {
return e.updateDeliveryStatus(webhookDB, d, "", status)
}
// ExportProcessNewTask exposes processNewTask.
func (e *Engine) ExportProcessNewTask(
ctx context.Context, task *Task,
+10 -2
View File
@@ -299,7 +299,8 @@ func countInFlightDeliveries(
return count, err
}
// createReplayDelivery writes the new pending delivery row and returns
// createReplayDelivery writes the new pending delivery row, adds it to
// the webhook's running totals in the same transaction, and returns
// the task that carries it to the delivery engine.
//
// The row is written with associations omitted, and neither Event nor
@@ -319,7 +320,14 @@ func createReplayDelivery(
Status: database.DeliveryStatusPending,
}
err := webhookDB.Omit(clause.Associations).Create(dlv).Error
err := webhookDB.Transaction(func(tx *gorm.DB) error {
err := tx.Omit(clause.Associations).Create(dlv).Error
if err != nil {
return err
}
return database.AddTotals(tx, database.Totals{Deliveries: 1})
})
if err != nil {
return delivery.Task{}, err
}
-1
View File
@@ -204,7 +204,6 @@ func assertEventCopy(
assert.Equal(t, original.Method, fresh.Method)
assert.Equal(t, original.Headers, fresh.Headers)
assert.Equal(t, original.Body, fresh.Body)
assert.Equal(t, int64(len(original.Body)), fresh.BodyBytes)
assert.Equal(t, original.ContentType, fresh.ContentType)
assert.Equal(t, original.EntrypointID, fresh.EntrypointID)
assert.Equal(t, original.WebhookID, fresh.WebhookID)
+15
View File
@@ -69,6 +69,21 @@ func (s *Handlers) LoadEventLogViewsForTest(
return views
}
// WebhookStatsForTest returns the figures the statistics pane on a
// webhook's page shows, from the webhook's entrypoints and targets
// loaded as that page loads them.
func (s *Handlers) WebhookStatsForTest(webhookID string) *WebhookStats {
var entrypoints []database.Entrypoint
s.db.DB().Where("webhook_id = ?", webhookID).Find(&entrypoints)
var targets []database.Target
s.db.DB().Where("webhook_id = ?", webhookID).Find(&targets)
return s.loadWebhookStats(webhookID, entrypoints, targets)
}
// AddTemplateForTest registers a template under a page name so that
// the handlers_test package can drive the render path with a
// template of its own.
+14 -10
View File
@@ -28,7 +28,7 @@ const (
// maxBodyShift is the bit shift for 1 MB body limit.
maxBodyShift = 20
// recentEventLimit is the number of recent events to show.
recentEventLimit = 50
recentEventLimit = 20
// paginationPerPage is the number of items per page.
paginationPerPage = 25
@@ -91,18 +91,22 @@ type Handlers struct {
// parsePageTemplate parses a page-specific template set from the
// embedded FS. Each page template is combined with the shared
// base, htmlheader, and navbar templates. The page file must be
// listed first so that its root action ({{template "base" .}})
// becomes the template set's entry point.
func parsePageTemplate(pageFile string) *template.Template {
return template.Must(
template.ParseFS(
templates.Templates,
// base, htmlheader, and navbar templates, and with any further files
// the page includes. The page file must be listed first so that its
// root action ({{template "base" .}}) becomes the template set's entry
// point.
func parsePageTemplate(
pageFile string, included ...string,
) *template.Template {
files := append([]string{
pageFile,
"base.html",
"htmlheader.html",
"navbar.html",
),
}, included...)
return template.Must(
template.ParseFS(templates.Templates, files...),
)
}
@@ -131,7 +135,7 @@ func New(
"profile.html": parsePageTemplate("profile.html"),
"sources_list.html": parsePageTemplate("sources_list.html"),
"sources_new.html": parsePageTemplate("sources_new.html"),
"source_detail.html": parsePageTemplate("source_detail.html"),
"source_detail.html": parsePageTemplate("source_detail.html", "webhook_stats.html"),
"source_edit.html": parsePageTemplate("source_edit.html"),
"source_logs.html": parsePageTemplate("source_logs.html"),
"target_edit.html": parsePageTemplate("target_edit.html"),
-2
View File
@@ -88,8 +88,6 @@ func TestHandleProfile_OwnProfile_OK(t *testing.T) {
h.HandleProfile().ServeHTTP(w, req)
assert.Equal(t, http.StatusOK, w.Code)
assert.Contains(t, w.Body.String(), "Account Information")
assert.NotContains(t, w.Body.String(), "Account Type")
}
func TestHandleProfile_OtherProfile_Forbidden(t *testing.T) {
-293
View File
@@ -1,293 +0,0 @@
package handlers
import (
"net/http"
"slices"
"strconv"
"time"
"github.com/dustin/go-humanize"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// recentEventColumns is the recent events list's projection. It
// leaves out the body, for the reason maxRenderedBodyBytes gives,
// and reads its size from body_bytes, recorded when the event was
// stored.
const recentEventColumns = "id, created_at, method, content_type, " +
"resubmitted_from_id, body_bytes"
// recentAttemptColumns is the part of a recorded attempt the list
// uses. The event log's deliveryResultColumns also reads response
// bodies, which the list does not show.
const recentAttemptColumns = "delivery_id, status_code, created_at"
// RecentEventView is one row of the recent events list on a
// webhook's page.
type RecentEventView struct {
Method string
ContentType string
// ResubmittedFromID names the event this one was copied from,
// empty for an event that arrived on the receiver.
ResubmittedFromID string
// Received is how long ago the event arrived, and ReceivedUTC
// the full timestamp the page shows on hover.
Received string
ReceivedUTC string
// Size is the size of the stored body.
Size string
// ProcessingTime is how long the event's slowest delivery
// took; see processingTime.
ProcessingTime string
// Status is what the webhook's HTTP target answered, and
// StatusClass its colour; see targetStatus. Both are empty
// unless the webhook has exactly one HTTP target.
Status string
StatusClass string
}
// recentEventRow is one row of recentEventColumns.
type recentEventRow struct {
ID string
CreatedAt time.Time
Method string
ContentType string
ResubmittedFromID *string
BodyBytes uint64
}
// recentAttemptRow is one row of recentAttemptColumns. CreatedAt is
// when the attempt's result was recorded, which is when the attempt
// finished.
type recentAttemptRow struct {
DeliveryID string
StatusCode int
CreatedAt time.Time
}
// singleHTTPTargetID returns the ID of the webhook's HTTP target
// when it has exactly one, and "" when it has none or several.
func singleHTTPTargetID(targets []database.Target) string {
id := ""
count := 0
for i := range targets {
if targets[i].Type == database.TargetTypeHTTP {
id = targets[i].ID
count++
}
}
if count != 1 {
return ""
}
return id
}
// loadRecentEvents loads the webhook's recentEventLimit newest
// events for its page, newest first. statusTargetID is the
// webhook's only HTTP target, or "" when the list shows no status.
func loadRecentEvents(
webhookDB *gorm.DB, webhookID, statusTargetID string,
) ([]RecentEventView, error) {
var rows []recentEventRow
err := webhookDB.Model(&database.Event{}).
Select(recentEventColumns).
Where("webhook_id = ?", webhookID).
Order("created_at DESC").
Limit(recentEventLimit).
Find(&rows).Error
if err != nil {
return nil, err
}
eventIDs := make([]string, len(rows))
for i := range rows {
eventIDs[i] = rows[i].ID
}
// Oldest first, so an event's last delivery to a target is its
// newest: a replay adds a delivery rather than changing the
// earlier one.
var deliveries []database.Delivery
err = webhookDB.
Select("id, event_id, target_id, status, created_at").
Where("event_id IN ?", eventIDs).
Order("created_at ASC").
Find(&deliveries).Error
if err != nil {
return nil, err
}
byEvent := make(map[string][]database.Delivery, len(rows))
deliveryIDs := make([]string, len(deliveries))
for i := range deliveries {
eventID := deliveries[i].EventID
byEvent[eventID] = append(byEvent[eventID], deliveries[i])
deliveryIDs[i] = deliveries[i].ID
}
attempts, err := loadRecentAttempts(webhookDB, deliveryIDs)
if err != nil {
return nil, err
}
views := make([]RecentEventView, len(rows))
for i := range rows {
views[i] = rows[i].view(
byEvent[rows[i].ID], attempts, statusTargetID,
)
}
return views, nil
}
// loadRecentAttempts loads the recorded attempts of the listed
// events' deliveries, keyed by delivery ID, each delivery's in
// attempt order. The IDs go in chunks for the reason
// deliveryIDChunkSize gives.
func loadRecentAttempts(
webhookDB *gorm.DB, deliveryIDs []string,
) (map[string][]recentAttemptRow, error) {
byDelivery := make(map[string][]recentAttemptRow)
for chunk := range slices.Chunk(deliveryIDs, deliveryIDChunkSize) {
var rows []recentAttemptRow
err := webhookDB.Model(&database.DeliveryResult{}).
Select(recentAttemptColumns).
Where("delivery_id IN ?", chunk).
Order("attempt_num ASC").
Find(&rows).Error
if err != nil {
return nil, err
}
for i := range rows {
id := rows[i].DeliveryID
byDelivery[id] = append(byDelivery[id], rows[i])
}
}
return byDelivery, nil
}
// view projects a loaded row for rendering. deliveries is the
// event's deliveries, oldest first, and attempts their recorded
// attempts keyed by delivery ID.
func (r *recentEventRow) view(
deliveries []database.Delivery,
attempts map[string][]recentAttemptRow,
statusTargetID string,
) RecentEventView {
v := RecentEventView{
Method: r.Method,
ContentType: r.ContentType,
Received: humanize.Time(r.CreatedAt),
ReceivedUTC: r.CreatedAt.UTC().Format(time.DateTime) + " UTC",
Size: humanize.Bytes(r.BodyBytes),
ProcessingTime: processingTime(deliveries, attempts),
}
if r.ResubmittedFromID != nil {
v.ResubmittedFromID = *r.ResubmittedFromID
}
if statusTargetID != "" {
v.Status, v.StatusClass = targetStatus(
deliveries, attempts, statusTargetID,
)
}
return v
}
// processingTime is how long the event's slowest delivery took,
// from being queued to its last recorded attempt, time spent
// waiting between retries included. A delivery is queued when its
// event is received, or when an operator replays it, so a replay
// is timed from the replay rather than from the event's arrival.
// It is "in progress" while any delivery is pending or retrying,
// and empty for an event with no deliveries.
func processingTime(
deliveries []database.Delivery,
attempts map[string][]recentAttemptRow,
) string {
if len(deliveries) == 0 {
return ""
}
var slowest time.Duration
for i := range deliveries {
if !deliveries[i].Status.Terminal() {
return "in progress"
}
tries := attempts[deliveries[i].ID]
if len(tries) == 0 {
continue
}
last := tries[len(tries)-1].CreatedAt
slowest = max(slowest, last.Sub(deliveries[i].CreatedAt))
}
return slowest.Round(time.Millisecond).String()
}
// targetStatus is what the target answered for the event, and the
// colour to show it in: the HTTP status code of the last attempt of
// the event's newest delivery to the target. Without a code it is
// "no response" when that attempt failed before a response
// arrived, the delivery's status ("pending") before any attempt,
// and "not sent" when the event has no delivery to the target.
func targetStatus(
deliveries []database.Delivery,
attempts map[string][]recentAttemptRow,
targetID string,
) (string, string) {
newest := -1
for i := range deliveries {
if deliveries[i].TargetID == targetID {
newest = i
}
}
if newest < 0 {
return "not sent", "text-gray-400"
}
tries := attempts[deliveries[newest].ID]
if len(tries) == 0 {
return string(deliveries[newest].Status), "text-gray-400"
}
code := tries[len(tries)-1].StatusCode
switch {
case code == 0:
return "no response", "text-red-600"
case code >= http.StatusInternalServerError:
return strconv.Itoa(code), "text-red-600"
case code >= http.StatusBadRequest:
return strconv.Itoa(code), "text-yellow-600"
case code >= http.StatusMultipleChoices:
return strconv.Itoa(code), "text-gray-500"
case code >= http.StatusOK:
return strconv.Itoa(code), "text-green-600"
default:
return strconv.Itoa(code), "text-gray-500"
}
}
-362
View File
@@ -1,362 +0,0 @@
package handlers_test
import (
"context"
"fmt"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"github.com/go-chi/chi"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm"
"gorm.io/gorm/clause"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/session"
)
// statusTitle marks the status column's cell in a recent events
// row; it is absent from the page when the column is not shown.
const statusTitle = `title="HTTP status from the HTTP target"`
// recentEventsFixture is one started app and a webhook whose
// recent events list a test fills.
type recentEventsFixture struct {
h *handlers.Handlers
sess *session.Session
db *database.Database
webhook *database.Webhook
webhookDB *gorm.DB
}
func newRecentEventsFixture(t *testing.T) *recentEventsFixture {
t.Helper()
f := &recentEventsFixture{}
var dbMgr *database.WebhookDBManager
app := newTestApp(t, &f.h, &f.sess, &f.db, &dbMgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
f.webhook = seedWebhook(t, f.db)
webhookDB, err := dbMgr.GetDB(f.webhook.ID)
require.NoError(t, err)
f.webhookDB = webhookDB
return f
}
func (f *recentEventsFixture) render(t *testing.T) string {
t.Helper()
return renderSourceDetailPage(t, f.h, f.sess, f.webhook.ID)
}
// event records an event received at receivedAt, with its body's
// size as the receiver records it.
func (f *recentEventsFixture) event(
t *testing.T, contentType, body string, receivedAt time.Time,
) *database.Event {
t.Helper()
event := &database.Event{
WebhookID: f.webhook.ID,
Method: http.MethodPost,
Body: body,
BodyBytes: int64(len(body)),
ContentType: contentType,
}
event.CreatedAt = receivedAt
require.NoError(t, f.webhookDB.Omit(
clause.Associations,
).Create(event).Error)
return event
}
// delivery records a delivery of the event to the target, queued
// when the event was received.
func (f *recentEventsFixture) delivery(
t *testing.T,
event *database.Event,
targetID string,
status database.DeliveryStatus,
) *database.Delivery {
t.Helper()
return f.deliveryQueuedAt(
t, event, targetID, status, event.CreatedAt,
)
}
// deliveryQueuedAt records a delivery of the event to the target,
// queued at queuedAt, as a replay is.
func (f *recentEventsFixture) deliveryQueuedAt(
t *testing.T,
event *database.Event,
targetID string,
status database.DeliveryStatus,
queuedAt time.Time,
) *database.Delivery {
t.Helper()
dlv := &database.Delivery{
EventID: event.ID,
TargetID: targetID,
Status: status,
}
dlv.CreatedAt = queuedAt
require.NoError(t, f.webhookDB.Omit(
clause.Associations,
).Create(dlv).Error)
return dlv
}
// attempt records one attempt of the delivery that finished took
// after the delivery was queued, with HTTP status code (0 for no
// response).
func (f *recentEventsFixture) attempt(
t *testing.T, dlv *database.Delivery, code int, took time.Duration,
) {
t.Helper()
result := &database.DeliveryResult{
DeliveryID: dlv.ID,
AttemptNum: 1,
StatusCode: code,
}
result.CreatedAt = dlv.CreatedAt.Add(took)
require.NoError(t, f.webhookDB.Omit(
clause.Associations,
).Create(result).Error)
}
// statusCell is the status column's cell as the page renders it.
func statusCell(class, text string) string {
return `<span class="font-medium ` + class + `" ` + statusTitle +
`>` + text + `</span>`
}
// TestHandleSourceDetail_ShowsFiftyNewestEvents proves the list is
// headed "50 Most Recent Events" and holds the 50 newest events,
// newest first, and not one more.
func TestHandleSourceDetail_ShowsFiftyNewestEvents(t *testing.T) {
t.Parallel()
f := newRecentEventsFixture(t)
base := time.Now().Add(-time.Hour)
for i := range 51 {
f.event(
t, fmt.Sprintf("application/x-recent-%02d", i), "{}",
base.Add(time.Duration(i)*time.Second),
)
}
body := f.render(t)
assert.Contains(t, body, ">50 Most Recent Events</h2>")
assert.Equal(t, 50, strings.Count(body, `title="Body size"`))
assert.NotContains(t, body, "application/x-recent-00")
assert.Contains(t, body, "application/x-recent-01")
assert.Less(
t,
strings.Index(body, "application/x-recent-50"),
strings.Index(body, "application/x-recent-49"),
)
}
// TestHandleSourceDetail_RecentEventColumns proves a row shows its
// time relative with the UTC timestamp on hover, its body size,
// and its processing time once every delivery has finished.
func TestHandleSourceDetail_RecentEventColumns(t *testing.T) {
t.Parallel()
f := newRecentEventsFixture(t)
logTarget := seedTarget(t, f.db, f.webhook.ID, database.TargetTypeLog)
receivedAt := time.Now().Add(-210 * time.Second).
UTC().Truncate(time.Second)
done := f.event(
t, contentTypeJSON, strings.Repeat("x", 2048), receivedAt,
)
f.attempt(
t,
f.delivery(t, done, logTarget.ID, database.DeliveryStatusDelivered),
0, 1500*time.Millisecond,
)
waiting := f.event(t, "text/plain", "{}", receivedAt)
f.delivery(t, waiting, logTarget.ID, database.DeliveryStatusPending)
body := f.render(t)
assert.Contains(
t, body,
`<span title="`+receivedAt.Format(time.DateTime)+
` UTC">3 minutes ago</span>`,
)
assert.Contains(t, body, `<span title="Body size">2.0 kB</span>`)
assert.Contains(t, body, ">1.5s</span>")
assert.Contains(t, body, ">in progress</span>")
}
// TestHandleSourceDetail_StatusWithSingleHTTPTarget proves that a
// webhook with exactly one HTTP target shows, colour-coded, what
// that target answered for each event. The log target beside it
// does not count against "exactly one".
func TestHandleSourceDetail_StatusWithSingleHTTPTarget(t *testing.T) {
t.Parallel()
f := newRecentEventsFixture(t)
target := seedTarget(t, f.db, f.webhook.ID, database.TargetTypeHTTP)
seedTarget(t, f.db, f.webhook.ID, database.TargetTypeLog)
now := time.Now()
for _, code := range []int{204, 302, 404, 503, 0} {
dlv := f.delivery(
t, f.event(t, contentTypeJSON, "{}", now), target.ID,
database.DeliveryStatusDelivered,
)
f.attempt(t, dlv, code, time.Second)
}
f.delivery(
t, f.event(t, contentTypeJSON, "{}", now), target.ID,
database.DeliveryStatusPending,
)
f.event(t, contentTypeJSON, "{}", now)
// A replay is a newer delivery, and its answer is the one shown.
replayed := f.event(t, contentTypeJSON, "{}", now)
f.attempt(t, f.delivery(
t, replayed, target.ID, database.DeliveryStatusFailed,
), 502, time.Second)
f.attempt(t, f.deliveryQueuedAt(
t, replayed, target.ID, database.DeliveryStatusDelivered,
now.Add(time.Minute),
), 200, time.Second)
body := f.render(t)
assert.Contains(t, body, statusCell("text-green-600", "204"))
assert.Contains(t, body, statusCell("text-gray-500", "302"))
assert.Contains(t, body, statusCell("text-yellow-600", "404"))
assert.Contains(t, body, statusCell("text-red-600", "503"))
assert.Contains(t, body, statusCell("text-red-600", "no response"))
assert.Contains(t, body, statusCell("text-gray-400", "pending"))
assert.Contains(t, body, statusCell("text-gray-400", "not sent"))
assert.Contains(t, body, statusCell("text-green-600", "200"))
assert.NotContains(t, body, ">502<")
}
// TestHandleSourceDetail_NoStatusWithoutSingleHTTPTarget proves the
// status column is absent when the webhook has no HTTP target or
// more than one.
func TestHandleSourceDetail_NoStatusWithoutSingleHTTPTarget(
t *testing.T,
) {
t.Parallel()
cases := map[string][]database.TargetType{
"none": {database.TargetTypeLog},
"several": {database.TargetTypeHTTP, database.TargetTypeHTTP},
}
for name, types := range cases {
t.Run(name, func(t *testing.T) {
t.Parallel()
f := newRecentEventsFixture(t)
event := f.event(t, contentTypeJSON, "{}", time.Now())
for _, tt := range types {
target := seedTarget(t, f.db, f.webhook.ID, tt)
f.attempt(t, f.delivery(
t, event, target.ID,
database.DeliveryStatusDelivered,
), 200, time.Second)
}
body := f.render(t)
assert.Contains(t, body, `title="Body size"`)
assert.NotContains(t, body, statusTitle)
})
}
}
// TestHandleWebhook_RecordsBodySize proves the receiver records the
// body's size in bytes, not characters, with the event it stores.
func TestHandleWebhook_RecordsBodySize(t *testing.T) {
t.Parallel()
f := newRecentEventsFixture(t)
seedEntrypoint(t, f.db, f.webhook.ID)
// Two bytes per character.
body := strings.Repeat("é", 1024)
req := httptest.NewRequestWithContext(
context.Background(), http.MethodPost, "/webhook/x",
strings.NewReader(body),
)
rctx := chi.NewRouteContext()
rctx.URLParams.Add("uuid", "ep-"+f.webhook.ID)
req = req.WithContext(context.WithValue(
req.Context(), chi.RouteCtxKey, rctx,
))
w := httptest.NewRecorder()
f.h.HandleWebhook().ServeHTTP(w, req)
require.Equal(t, http.StatusOK, w.Code)
var stored database.Event
require.NoError(t, f.webhookDB.First(&stored).Error)
assert.Equal(t, int64(2048), stored.BodyBytes)
}
// TestHandleSourceDetail_FailedLoadIsAnError proves that when the
// list cannot be loaded the page answers with an error, rather than
// an empty list claiming the webhook has no events.
func TestHandleSourceDetail_FailedLoadIsAnError(t *testing.T) {
t.Parallel()
f := newRecentEventsFixture(t)
target := seedTarget(t, f.db, f.webhook.ID, database.TargetTypeHTTP)
f.attempt(t, f.delivery(
t, f.event(t, contentTypeJSON, "{}", time.Now()), target.ID,
database.DeliveryStatusDelivered,
), 200, time.Second)
// The attempts are the list's last query, so its events and
// deliveries have already loaded when it fails.
require.NoError(t, f.webhookDB.Exec(
"DROP TABLE delivery_results",
).Error)
w := serveSourceDetailPage(t, f.h, f.sess, f.webhook.ID)
assert.Equal(t, http.StatusInternalServerError, w.Code)
assert.NotContains(t, w.Body.String(), "No events received yet.")
}
+3 -18
View File
@@ -62,23 +62,6 @@ func renderSourceDetailPage(
) string {
t.Helper()
w := serveSourceDetailPage(t, h, sess, webhookID)
require.Equal(t, http.StatusOK, w.Code)
return w.Body.String()
}
// serveSourceDetailPage runs the real source detail handler for a
// webhook and returns its response, whatever its status.
func serveSourceDetailPage(
t *testing.T,
h *handlers.Handlers,
sess *session.Session,
webhookID string,
) *httptest.ResponseRecorder {
t.Helper()
req := httptest.NewRequestWithContext(
context.Background(),
http.MethodGet,
@@ -104,7 +87,9 @@ func serveSourceDetailPage(
w := httptest.NewRecorder()
h.HandleSourceDetail().ServeHTTP(w, req)
return w
require.Equal(t, http.StatusOK, w.Code)
return w.Body.String()
}
// TestHandleSourceDetail_MasksSlackWebhookURL is the
+9 -15
View File
@@ -415,23 +415,16 @@ func (h *Handlers) renderSourceDetail(
"webhook_id = ?", webhook.ID,
).Find(&targets)
var events []RecentEventView
var events []database.Event
if h.dbMgr.DBExists(webhook.ID) {
webhookDB, err := h.dbMgr.GetDB(webhook.ID)
if err != nil {
h.serverError(w, "failed to get webhook database", err)
return
}
events, err = loadRecentEvents(
webhookDB, webhook.ID, singleHTTPTargetID(targets),
)
if err != nil {
h.serverError(w, "failed to load recent events", err)
return
webhookDB, dbErr := h.dbMgr.GetDB(webhook.ID)
if dbErr == nil {
webhookDB.Where(
"webhook_id = ?", webhook.ID,
).Order("created_at DESC").Limit(
recentEventLimit,
).Find(&events)
}
}
@@ -457,6 +450,7 @@ func (h *Handlers) renderSourceDetail(
"Targets": delivery.NewTargetViews(targets),
"Events": events,
"BaseURL": baseURL,
"Stats": h.loadWebhookStats(webhook.ID, entrypoints, targets),
}
h.renderTemplate(w, r, "source_detail.html", data)
+16 -6
View File
@@ -230,7 +230,6 @@ func (s eventSource) event() *database.Event {
Method: s.Method,
Headers: s.HeadersJSON,
Body: string(s.Body),
BodyBytes: int64(len(s.Body)),
ContentType: s.ContentType,
ResubmittedFromID: s.ResubmittedFromID,
}
@@ -253,11 +252,12 @@ func requestEventSource(
}
}
// createAndFanOut writes the event and one pending delivery per target
// in a single transaction, then hands the tasks to the delivery
// engine. It is the only path by which an event and its deliveries are
// created, so a resubmitted event is retried, SSRF-guarded and
// circuit-broken exactly as a received one is.
// createAndFanOut writes the event and one pending delivery per target,
// and adds them to the webhook's running totals, in a single
// transaction, then hands the tasks to the delivery engine. It is the
// only path by which an event and its deliveries are created, so a
// resubmitted event is retried, SSRF-guarded and circuit-broken
// exactly as a received one is.
//
// The tasks are returned as well as queued, so a caller can report how
// many targets the event went to.
@@ -297,6 +297,16 @@ func (h *Handlers) createAndFanOut(
return nil, nil, err
}
err = database.AddTotals(tx, database.Totals{
Events: 1,
Deliveries: int64(len(tasks)),
})
if err != nil {
tx.Rollback()
return nil, nil, err
}
err = tx.Commit().Error
if err != nil {
return nil, nil, fmt.Errorf(
+212
View File
@@ -0,0 +1,212 @@
package handlers
import (
"fmt"
"time"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// The spans of the two recent windows the statistics pane reports on:
// the last 10 minutes and the last 24 hours.
const (
shortWindow = 10 * time.Minute
longWindow = 24 * time.Hour
)
// percent turns a fraction into a percentage.
const percent = 100
// WebhookStats holds the figures in the statistics pane at the top of
// the webhook page.
type WebhookStats struct {
Entrypoints int
ActiveEntrypoints int
Targets int
ActiveTargets int
// Totals holds the lifetime counts of events, deliveries and
// failures, and how many of each retention has removed.
Totals database.Totals
// InProgress counts the deliveries still pending or retrying.
InProgress int64
// LastEventAt is when the newest stored event arrived, or nil when
// none is stored.
LastEventAt *time.Time
Last10Minutes RecentWindow
Last24Hours RecentWindow
}
// RecentWindow holds what happened in one recent window: the events
// received in it, and the deliveries that became delivered or failed in
// it.
type RecentWindow struct {
Events int64
Delivered int64
Failed int64
}
// FailurePercent is the share of the deliveries finished in the window
// that failed, or a dash when none finished. Deliveries still pending
// or retrying are not counted either way.
func (w RecentWindow) FailurePercent() string {
finished := w.Delivered + w.Failed
if finished == 0 {
return "—"
}
return fmt.Sprintf(
"%.1f%%", percent*float64(w.Failed)/float64(finished),
)
}
// loadWebhookStats gathers the figures for the statistics pane from the
// webhook's entrypoints and targets, as the page has already loaded
// them, and from its event database. It returns nil, and logs why, when
// the event database cannot be read.
func (h *Handlers) loadWebhookStats(
webhookID string,
entrypoints []database.Entrypoint,
targets []database.Target,
) *WebhookStats {
stats := &WebhookStats{
Entrypoints: len(entrypoints),
Targets: len(targets),
}
for i := range entrypoints {
if entrypoints[i].Active {
stats.ActiveEntrypoints++
}
}
for i := range targets {
if targets[i].Active {
stats.ActiveTargets++
}
}
// Opening an event database that does not exist would create it,
// and it would hold nothing to count.
if !h.dbMgr.DBExists(webhookID) {
return stats
}
webhookDB, err := h.dbMgr.GetDB(webhookID)
if err == nil {
err = readEventStats(webhookDB, time.Now(), stats)
}
if err != nil {
h.log.Error(
"failed to read webhook statistics",
"webhook_id", webhookID,
"error", err,
)
return nil
}
return stats
}
// readEventStats fills in the figures that come from the webhook's
// event database. None of them reads every stored row: the totals are
// one row, and every other figure is read from an index, over only the
// rows it counts.
func readEventStats(
db *gorm.DB, now time.Time, stats *WebhookStats,
) error {
err := db.Take(&stats.Totals).Error
if err != nil {
return fmt.Errorf("reading running totals: %w", err)
}
err = db.Model(&database.Delivery{}).
Where("status IN ?", []database.DeliveryStatus{
database.DeliveryStatusPending,
database.DeliveryStatusRetrying,
}).
Count(&stats.InProgress).Error
if err != nil {
return fmt.Errorf("counting deliveries in progress: %w", err)
}
var newest []time.Time
err = db.Model(&database.Event{}).
Order("created_at DESC").
Limit(1).
Pluck("created_at", &newest).Error
if err != nil {
return fmt.Errorf("reading newest event time: %w", err)
}
if len(newest) > 0 {
stats.LastEventAt = &newest[0]
}
stats.Last10Minutes, err = readRecentWindow(
db, now.Add(-shortWindow),
)
if err != nil {
return err
}
stats.Last24Hours, err = readRecentWindow(
db, now.Add(-longWindow),
)
return err
}
// readRecentWindow counts the events received, and the deliveries that
// became delivered or failed, since the given time.
func readRecentWindow(
db *gorm.DB, since time.Time,
) (RecentWindow, error) {
var w RecentWindow
err := db.Model(&database.Event{}).
Where("created_at >= ?", since).
Count(&w.Events).Error
if err != nil {
return w, fmt.Errorf("counting recent events: %w", err)
}
w.Delivered, err = countFinishedSince(
db, database.DeliveryStatusDelivered, since,
)
if err != nil {
return w, err
}
w.Failed, err = countFinishedSince(
db, database.DeliveryStatusFailed, since,
)
return w, err
}
// countFinishedSince counts the deliveries that reached the given
// final status since the given time.
func countFinishedSince(
db *gorm.DB, status database.DeliveryStatus, since time.Time,
) (int64, error) {
var n int64
err := db.Model(&database.Delivery{}).
Where("status = ? AND finished_at >= ?", status, since).
Count(&n).Error
if err != nil {
return 0, fmt.Errorf(
"counting deliveries %s recently: %w", status, err,
)
}
return n, nil
}
+328
View File
@@ -0,0 +1,328 @@
package handlers_test
import (
"net/http"
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.uber.org/fx/fxtest"
"gorm.io/gorm"
"gorm.io/gorm/clause"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/logger"
"sneak.berlin/go/webhooker/internal/session"
)
// statsEntrypoint adds an entrypoint to a webhook and returns its path.
func statsEntrypoint(
t *testing.T, db *database.Database, webhookID string, active bool,
) string {
t.Helper()
ep := &database.Entrypoint{
WebhookID: webhookID,
Path: uuid.New().String(),
}
require.NoError(t, db.DB().Omit(clause.Associations).Create(ep).Error)
require.NoError(t, db.DB().Model(ep).Update("active", active).Error)
return ep.Path
}
// statsDelivery returns the id of an event's delivery to a target.
func statsDelivery(
t *testing.T, webhookDB *gorm.DB, eventID, targetID string,
) string {
t.Helper()
var d database.Delivery
require.NoError(t, webhookDB.Where(
"event_id = ? AND target_id = ?", eventID, targetID,
).First(&d).Error)
return d.ID
}
// statsFinish settles a delivery as the delivery engine does: its
// final status and the time it finished, and for a failure one more on
// the webhook's failure total, in one transaction.
func statsFinish(
t *testing.T,
webhookDB *gorm.DB,
deliveryID string,
status database.DeliveryStatus,
at time.Time,
) {
t.Helper()
require.NoError(t, webhookDB.Transaction(func(tx *gorm.DB) error {
err := tx.Model(&database.Delivery{}).
Where("id = ?", deliveryID).
Updates(map[string]any{"status": status, "finished_at": at}).
Error
if err != nil || status != database.DeliveryStatusFailed {
return err
}
return database.AddTotals(tx, database.Totals{Failures: 1})
}))
}
// statsAge moves an event's arrival back to the given time.
func statsAge(
t *testing.T, webhookDB *gorm.DB, eventID string, at time.Time,
) {
t.Helper()
require.NoError(t, webhookDB.Model(&database.Event{}).
Where("id = ?", eventID).
Update("created_at", at).Error)
}
// seedStatsHistory builds the webhook the statistics test checks: one
// day of retention, two entrypoints (one inactive) and three targets
// (one inactive). Three events arrive through the receiver, and so
// each has a delivery to the two active targets. The oldest event is
// past retention, the middle one six hours old, the newest just in.
// Their deliveries are settled as the delivery engine would, and a
// replay adds a pending delivery to the oldest event. It returns the
// webhook, its event database and the newest event.
func seedStatsHistory(
t *testing.T,
h *handlers.Handlers,
sess *session.Session,
db *database.Database,
dbMgr *database.WebhookDBManager,
) (*database.Webhook, *gorm.DB, database.Event) {
t.Helper()
wh := &database.Webhook{
UserID: deleteTestUserID, Name: "stats", RetentionDays: 1,
}
require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
path := statsEntrypoint(t, db, wh.ID, true)
statsEntrypoint(t, db, wh.ID, false)
first := seedConfiguredTarget(
t, db, wh.ID, database.TargetTypeHTTP,
`{"url":"`+replayTargetURL+`"}`,
)
second := seedTarget(t, db, wh.ID, database.TargetTypeLog)
inactive := seedTarget(t, db, wh.ID, database.TargetTypeLog)
require.NoError(t, db.DB().Model(inactive).
Update("active", false).Error)
router := receiverRouter(h)
for range 3 {
require.Equal(t, http.StatusOK, postReceiver(t, router, path))
}
webhookDB, err := dbMgr.GetDB(wh.ID)
require.NoError(t, err)
events := listEvents(t, webhookDB)
require.Len(t, events, 3)
oldest, middle, newest := events[0], events[1], events[2]
now := time.Now()
statsAge(t, webhookDB, oldest.ID, now.Add(-50*time.Hour))
statsAge(t, webhookDB, middle.ID, now.Add(-6*time.Hour))
oldestFailure := statsDelivery(t, webhookDB, oldest.ID, first.ID)
statsFinish(t, webhookDB, oldestFailure,
database.DeliveryStatusFailed, now.Add(-49*time.Hour))
statsFinish(t, webhookDB,
statsDelivery(t, webhookDB, oldest.ID, second.ID),
database.DeliveryStatusDelivered, now.Add(-49*time.Hour))
statsFinish(t, webhookDB,
statsDelivery(t, webhookDB, middle.ID, first.ID),
database.DeliveryStatusFailed, now.Add(-5*time.Hour))
statsFinish(t, webhookDB,
statsDelivery(t, webhookDB, middle.ID, second.ID),
database.DeliveryStatusFailed, now.Add(-time.Minute))
statsFinish(t, webhookDB,
statsDelivery(t, webhookDB, newest.ID, first.ID),
database.DeliveryStatusDelivered, now.Add(-2*time.Minute))
require.Equal(t, http.StatusSeeOther,
postReplay(t, h, sess, wh.ID, oldestFailure).Code)
return wh, webhookDB, newest
}
// statsPrune runs the real retention reaper until it has removed one
// event from the webhook's database, then stops it.
func statsPrune(
t *testing.T,
db *database.Database,
dbMgr *database.WebhookDBManager,
log *logger.Logger,
webhookDB *gorm.DB,
) {
t.Helper()
lc := fxtest.NewLifecycle(t)
database.NewRetentionReaper(lc, database.RetentionReaperParams{
Config: &config.Config{
RetentionSweepInterval: 10 * time.Millisecond,
},
Database: db,
DBManager: dbMgr,
Logger: log,
})
lc.RequireStart()
require.Eventually(t, func() bool {
var totals database.Totals
err := webhookDB.Take(&totals).Error
return err == nil && totals.EventsRemoved == 1
}, 10*time.Second, 10*time.Millisecond)
lc.RequireStop()
}
// assertStatsTotals checks the lifetime events, deliveries and
// failures, and those within retention.
func assertStatsTotals(
t *testing.T, totals database.Totals, lifetime, within [3]int64,
) {
t.Helper()
gotLifetime := [3]int64{
totals.Events, totals.Deliveries, totals.Failures,
}
gotWithin := [3]int64{
totals.EventsWithinRetention(),
totals.DeliveriesWithinRetention(),
totals.FailuresWithinRetention(),
}
assert.Equal(t, lifetime, gotLifetime,
"lifetime events, deliveries, failures")
assert.Equal(t, within, gotWithin,
"events, deliveries, failures within retention")
}
// TestWebhookStats_EveryFigureAcrossRetentionPrune checks every figure
// the statistics pane shows for the history seedStatsHistory builds,
// before and after the real retention reaper removes the oldest event.
func TestWebhookStats_EveryFigureAcrossRetentionPrune(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
log *logger.Logger
)
app := newTestApp(t, &h, &sess, &db, &dbMgr, &log)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh, webhookDB, newest := seedStatsHistory(t, h, sess, db, dbMgr)
stats := h.WebhookStatsForTest(wh.ID)
require.NotNil(t, stats)
assert.Equal(t, 2, stats.Entrypoints)
assert.Equal(t, 1, stats.ActiveEntrypoints)
assert.Equal(t, 3, stats.Targets)
assert.Equal(t, 2, stats.ActiveTargets)
assertStatsTotals(t, stats.Totals, [3]int64{3, 7, 3}, [3]int64{3, 7, 3})
assert.Equal(t, int64(2), stats.InProgress)
require.NotNil(t, stats.LastEventAt)
assert.True(t, newest.CreatedAt.Equal(*stats.LastEventAt))
assert.Equal(t, handlers.RecentWindow{
Events: 1, Delivered: 1, Failed: 1,
}, stats.Last10Minutes)
assert.Equal(t, handlers.RecentWindow{
Events: 2, Delivered: 1, Failed: 2,
}, stats.Last24Hours)
assert.Equal(t, "50.0%", stats.Last10Minutes.FailurePercent())
assert.Equal(t, "66.7%", stats.Last24Hours.FailurePercent())
// Retention removes the oldest event with its three deliveries,
// one of them failed and one the pending replay.
statsPrune(t, db, dbMgr, log, webhookDB)
after := h.WebhookStatsForTest(wh.ID)
require.NotNil(t, after)
assertStatsTotals(t, after.Totals, [3]int64{3, 7, 3}, [3]int64{2, 4, 2})
assert.Equal(t, int64(1), after.InProgress)
assert.Equal(t, stats.LastEventAt, after.LastEventAt)
assert.Equal(t, stats.Last10Minutes, after.Last10Minutes)
assert.Equal(t, stats.Last24Hours, after.Last24Hours)
body := renderSourceDetailPage(t, h, sess, wh.ID)
assert.Contains(t, body, "Statistics")
assert.Contains(t, body, "Within retention")
assert.Contains(t, body, "50.0%")
assert.Contains(t, body, "66.7%")
}
// TestWebhookStats_WebhookWithNoEvents covers a webhook whose event
// database has never been opened: every count is zero, the
// percentages are a dash, and showing the page does not create the
// database.
func TestWebhookStats_WebhookWithNoEvents(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
assert.Equal(t, &handlers.WebhookStats{}, h.WebhookStatsForTest(wh.ID))
assert.Equal(t, "—", handlers.RecentWindow{}.FailurePercent())
body := renderSourceDetailPage(t, h, sess, wh.ID)
assert.Contains(t, body, "Statistics")
assert.False(t, dbMgr.DBExists(wh.ID))
}
// TestRecentWindow_FailurePercent pins the percentage: failed
// deliveries out of all that finished in the window.
func TestRecentWindow_FailurePercent(t *testing.T) {
t.Parallel()
tests := []struct {
window handlers.RecentWindow
want string
}{
{handlers.RecentWindow{}, "—"},
{handlers.RecentWindow{Events: 4}, "—"},
{handlers.RecentWindow{Delivered: 3, Failed: 1}, "25.0%"},
{handlers.RecentWindow{Failed: 2}, "100.0%"},
{handlers.RecentWindow{Delivered: 2}, "0.0%"},
}
for _, tt := range tests {
assert.Equal(t, tt.want, tt.window.FailurePercent(), tt.window)
}
}
+3 -3
View File
@@ -13,9 +13,9 @@ import (
// TestBaseTemplateScriptsAreServed walks every /s/ script the base
// template loads on each page and fetches it through the real router.
// Alpine.js is extracted from its tarball in 3p/ at build time, so the
// file is not in the tree: this is the check that the page still gets
// the JavaScript it asks for.
// Alpine.js is fetched at build time rather than committed, so nothing
// in the repo guarantees it is present: this is the check that the page
// still gets the JavaScript it asks for.
func TestBaseTemplateScriptsAreServed(t *testing.T) {
t.Parallel()
@@ -115,8 +115,8 @@ func TestVersion_EnclosingRepositoryIsNotUsed(t *testing.T) {
require.Equal(t, unknown, runScript(t, inner, nil))
}
// An explicit VERSION, such as the Dockerfile's build arg, wins over
// anything derivable.
// The Docker build has no git metadata, so the version arrives as an
// environment override. It wins over anything derivable.
func TestVersion_EnvironmentOverrideWins(t *testing.T) {
t.Parallel()
@@ -128,8 +128,8 @@ func TestVersion_EnvironmentOverrideWins(t *testing.T) {
}
// An empty VERSION is treated as unset rather than stamping an empty
// string: a caller exporting VERSION= must not produce a binary
// reporting "".
// string: the Dockerfile's build arg has a non-empty default, but a
// caller exporting VERSION= must not produce a binary reporting "".
func TestVersion_EmptyOverrideFallsBackToGit(t *testing.T) {
t.Parallel()
@@ -168,8 +168,8 @@ func TestMakefile_BuildComposesVersionAndExtraFlags(t *testing.T) {
}
// A caller can define VERSION as the empty string -- `make build
// VERSION=`, or the Dockerfile's `make build VERSION="$VERSION"` when no
// VERSION build arg was given. script/version's own guard does not cover
// VERSION=`, or a `--build-arg VERSION=` reaching the Dockerfile's `make
// build VERSION="$VERSION"`. script/version's own guard does not cover
// that: the value never passes through the script. Stamping "" would
// leave the binary reporting no version and the footer on "dev", which
// is the defect this package exists for.
@@ -231,7 +231,7 @@ func TestDockerfile_BuildsThroughTheMakeTarget(t *testing.T) {
require.NotContains(t, dockerfile, "go build",
"a raw go build bypasses the Makefile's -X flag")
require.Contains(t, dockerfile, "ARG VERSION")
require.Contains(t, dockerfile, "ARG VERSION=")
require.Contains(t, dockerfile,
`make build VERSION="$VERSION" GO_LDFLAGS='-extldflags "-static"'`)
}
-16
View File
@@ -1,16 +0,0 @@
#!/bin/sh
# script/assets: extract Alpine.js from its npm package tarball, committed
# in 3p/, to static/js/alpine.min.js, where go:embed reads it. The
# extracted file is not committed. script/test, make build and make dev run
# this first.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
tar -xzOf 3p/alpinejs-3.14.9.tgz package/dist/cdn.min.js \
>static/js/alpine.min.js
}
main "$@"
+8 -1
View File
@@ -4,7 +4,9 @@
# installed tools are skipped. Base tooling comes from nix, apt, brew,
# or apk (detected in that order); assumes NOTHING is present (not git,
# make, or go). golangci-lint is deliberately not installed: linting runs
# only in docker, via script/lint and Dockerfile.lint.
# only in docker, via script/lint and Dockerfile.lint. Finishes by running
# script/fetch-assets, which installs the hash-pinned third-party browser
# assets the repo does not commit.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
@@ -67,6 +69,11 @@ main() {
go mod download
# Third-party browser assets are not committed; fetch and verify them
# so a fresh clone can build and test.
if missing curl; then pkg_install curl curl curl curl; fi
"$ROOT/script/fetch-assets"
echo "bootstrap complete"
}
+1 -2
View File
@@ -1,7 +1,6 @@
#!/bin/sh
# script/check: run all checks (test, lint, fmt-check). Our own
# extension to scripts-to-rule-them-all.
# Writes only the ignored static/js/alpine.min.js, through script/test.
# extension to scripts-to-rule-them-all. Must not modify any files.
# Generic: usually needs no adaptation.
set -eu
+3 -3
View File
@@ -2,9 +2,9 @@
# script/docker: build the Docker image tagged with the project name.
# The tag comes from script/projectname.
#
# The version script/version resolves here goes in as the VERSION build
# arg, which takes precedence over what the build would derive from the
# .git in its context.
# .dockerignore excludes .git/, so the builder stage cannot derive the
# version itself. It is resolved here, where the checkout is, and passed
# in as a build arg; without it the image would stamp itself "unknown".
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+104
View File
@@ -0,0 +1,104 @@
#!/bin/sh
# script/fetch-assets: download the third-party browser assets the web UI
# ships and install them under static/. Minified bundles are not committed
# (REPO_POLICIES.md: no build artifacts in version control), so the build
# fetches them here. Every download is verified against a hardcoded sha256
# before it is installed, and any mismatch aborts. Idempotent: an asset
# already present with its pinned hash is left alone.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# The sha256 of each installed asset lives in static/vendor.sha256, in
# sha256sum(1) format, with paths relative to static/. That file is the
# single source of truth: this script verifies against it, and
# static/vendor_test.go asserts the bytes embedded into the binary match
# it, so the hash cannot rot into a value nothing checks.
MANIFEST="static/vendor.sha256"
# Alpine.js 3.14.9, 2026-08-17. Fetched from registry.npmjs.org, the
# publisher of record; the jsDelivr and unpkg copies are mirrors of this
# same tarball. dist/cdn.min.js is the browser build Alpine publishes for
# a <script> tag.
ALPINE_VERSION="3.14.9"
ALPINE_URL="https://registry.npmjs.org/alpinejs/-/alpinejs-${ALPINE_VERSION}.tgz"
# sha256 of alpinejs-3.14.9.tgz
ALPINE_TARBALL_SHA256="97dad7c0c81e659cfc8e7700055da9770f8186187cb9a8a76efb57e00d5ce52a"
ALPINE_MEMBER="package/dist/cdn.min.js"
ALPINE_DEST="js/alpine.min.js"
sha256_of() {
if command -v sha256sum >/dev/null 2>&1; then
sha256sum "$1" | cut -d' ' -f1
else
shasum -a 256 "$1" | cut -d' ' -f1
fi
}
# expected_sha256 <path-relative-to-static>
expected_sha256() {
awk -v want="$1" '$2 == want { print $1; found = 1 }
END { if (!found) exit 1 }' "$ROOT/$MANIFEST"
}
# verify <file> <expected-sha256> <what>
verify() {
actual="$(sha256_of "$1")"
if [ "$actual" != "$2" ]; then
echo "fetch-assets: sha256 mismatch for $3" >&2
echo " expected: $2" >&2
echo " actual: $actual" >&2
exit 1
fi
}
# up_to_date <path-relative-to-static> <expected-sha256>
up_to_date() {
[ -f "$ROOT/static/$1" ] || return 1
[ "$(sha256_of "$ROOT/static/$1")" = "$2" ]
}
fetch_alpine() {
want="$(expected_sha256 "$ALPINE_DEST")"
if up_to_date "$ALPINE_DEST" "$want"; then
echo "fetch-assets: static/$ALPINE_DEST already at $want"
return 0
fi
echo "fetch-assets: fetching Alpine.js $ALPINE_VERSION from $ALPINE_URL"
tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' EXIT INT TERM
curl -fsSL -o "$tmp/alpine.tgz" "$ALPINE_URL"
verify "$tmp/alpine.tgz" "$ALPINE_TARBALL_SHA256" "alpinejs-${ALPINE_VERSION}.tgz"
tar -xzOf "$tmp/alpine.tgz" "$ALPINE_MEMBER" >"$tmp/alpine.min.js"
verify "$tmp/alpine.min.js" "$want" "$ALPINE_MEMBER from alpinejs-${ALPINE_VERSION}.tgz"
mkdir -p "$(dirname "$ROOT/static/$ALPINE_DEST")"
cp "$tmp/alpine.min.js" "$ROOT/static/$ALPINE_DEST"
rm -rf "$tmp"
trap - EXIT INT TERM
echo "fetch-assets: installed static/$ALPINE_DEST ($want)"
}
# Re-check every manifest entry against what is now on disk, so an entry
# no script installs fails loudly instead of passing silently.
verify_manifest() {
while read -r want path; do
case "$want" in '' | '#'*) continue ;; esac
if [ ! -f "$ROOT/static/$path" ]; then
echo "fetch-assets: $MANIFEST lists static/$path, which is missing" >&2
exit 1
fi
verify "$ROOT/static/$path" "$want" "static/$path"
done <"$ROOT/$MANIFEST"
}
main() {
cd "$ROOT"
fetch_alpine
verify_manifest
echo "fetch-assets: all assets in $MANIFEST verified"
}
main "$@"
-1
View File
@@ -28,7 +28,6 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
"$ROOT/script/assets"
go test -v -race -timeout 90s ./...
}
+7 -5
View File
@@ -7,16 +7,18 @@
#
# Order of precedence:
#
# 1. $VERSION, if set and non-empty: an explicit value, such as the
# Dockerfile's VERSION build arg.
# 1. $VERSION, if set and non-empty. This is how the value reaches a
# build that cannot derive it: .dockerignore excludes .git/, so the
# builder stage has no git metadata and the Dockerfile takes the
# value as a build arg instead.
# 2. `git describe --tags --always --dirty` against this checkout. At
# a clean tagged commit that is exactly the tag; otherwise it
# carries the short SHA, the commit distance when a tag is
# reachable, and a -dirty suffix for uncommitted changes.
# 3. "unknown", for a tree with no git metadata and no $VERSION -- a
# source tarball, or a `docker build` with no .git in its context
# and no VERSION build arg. That case must not fail the build and
# must not name a tag the tree may not be at, so it names nothing.
# source tarball, or `docker build .` with no --build-arg. That
# case must not fail the build and must not name a tag the tree may
# not be at, so it names nothing.
#
# The git step insists the enclosing repository is this checkout, not
# merely some repository above it: an unpacked tarball sitting inside an
+1
View File
@@ -0,0 +1 @@
3ed1eed252488921df65e363d6715deb04d7f92aaedb9e52199fdf73cb1e0ad3 js/alpine.min.js
+92
View File
@@ -0,0 +1,92 @@
package static_test
import (
"bufio"
"crypto/sha256"
"encoding/hex"
"os"
"strings"
"testing"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/static"
)
const manifestPath = "vendor.sha256"
// fetchHint is appended to every failure here: the assets the manifest
// covers are fetched by the build, not committed, so a fresh clone that
// has not run script/fetch-assets fails this test and should be told why.
const fetchHint = "run `script/fetch-assets` (or `make assets`) to install " +
"the pinned third-party assets"
// TestVendoredAssetsMatchManifest asserts that every asset listed in
// static/vendor.sha256 is embedded in the binary with exactly the pinned
// bytes. script/fetch-assets verifies the same hashes at download time;
// this test verifies them again on what actually ships, so a build that
// skipped, cached, or subverted the fetch cannot produce a binary serving
// unpinned third-party JavaScript.
func TestVendoredAssetsMatchManifest(t *testing.T) {
t.Parallel()
entries := readManifest(t)
require.NotEmpty(t, entries, "%s lists no assets", manifestPath)
for path, want := range entries {
t.Run(path, func(t *testing.T) {
t.Parallel()
data, err := static.Static.ReadFile(path)
require.NoErrorf(
t, err,
"%s is listed in %s but is not embedded; %s",
path, manifestPath, fetchHint,
)
sum := sha256.Sum256(data)
got := hex.EncodeToString(sum[:])
require.Equalf(
t, want, got,
"embedded %s does not match its pinned sha256 in %s; %s",
path, manifestPath, fetchHint,
)
})
}
}
// readManifest parses static/vendor.sha256, which is in sha256sum(1)
// format with paths relative to static/.
func readManifest(t *testing.T) map[string]string {
t.Helper()
f, err := os.Open(manifestPath)
require.NoError(t, err, "opening %s", manifestPath)
defer func() { require.NoError(t, f.Close()) }()
entries := make(map[string]string)
scanner := bufio.NewScanner(f)
for scanner.Scan() {
line := strings.TrimSpace(scanner.Text())
if line == "" || strings.HasPrefix(line, "#") {
continue
}
fields := strings.Fields(line)
require.Lenf(
t, fields, 2,
"%s: malformed entry %q, want \"<sha256> <path>\"",
manifestPath, line,
)
sum, path := fields[0], fields[1]
require.Lenf(t, sum, 64, "%s: %q is not a sha256", manifestPath, sum)
entries[path] = sum
}
require.NoError(t, scanner.Err(), "reading %s", manifestPath)
return entries
}
+4
View File
@@ -41,6 +41,10 @@
<dt class="w-32 text-sm font-medium text-gray-500">Username</dt>
<dd class="text-sm text-gray-900">{{.User.Username}}</dd>
</div>
<div class="flex">
<dt class="w-32 text-sm font-medium text-gray-500">Account Type</dt>
<dd class="text-sm text-gray-900">Standard User</dd>
</div>
</dl>
</div>
</div>
+7 -17
View File
@@ -24,6 +24,8 @@
</div>
</div>
{{template "webhook_stats" .}}
<div class="grid grid-cols-1 lg:grid-cols-2 gap-6">
<!-- Entrypoints -->
<div class="card">
@@ -181,30 +183,18 @@
<!-- Recent Events -->
<div class="card mt-6">
<div class="p-4 border-b border-gray-200 flex justify-between items-center">
<h2 class="text-lg font-medium text-gray-900">50 Most Recent Events</h2>
<h2 class="text-lg font-medium text-gray-900">Recent Events</h2>
<a href="/source/{{.Webhook.ID}}/logs" class="btn-text text-sm">View All</a>
</div>
<div class="divide-y divide-gray-100">
{{range .Events}}
<div class="p-4">
<div class="flex flex-wrap items-center justify-between gap-3">
<div class="flex flex-wrap items-center gap-3">
<div class="flex items-center justify-between">
<div class="flex items-center gap-3">
<span class="badge-info">{{.Method}}</span>
<span class="text-sm text-gray-500 break-all">{{.ContentType}}</span>
{{if .ResubmittedFromID}}
<span class="text-xs text-gray-500" title="This event is a copy of {{.ResubmittedFromID}}">resubmitted copy</span>
{{end}}
</div>
<div class="flex flex-wrap items-center gap-3 text-xs text-gray-400">
<span title="Body size">{{.Size}}</span>
{{if .ProcessingTime}}
<span title="Processing time: how long the slowest delivery took, from being queued to its last attempt">{{.ProcessingTime}}</span>
{{end}}
{{if .Status}}
<span class="font-medium {{.StatusClass}}" title="HTTP status from the HTTP target">{{.Status}}</span>
{{end}}
<span title="{{.ReceivedUTC}}">{{.Received}}</span>
<span class="text-sm text-gray-500">{{.ContentType}}</span>
</div>
<span class="text-xs text-gray-400">{{.CreatedAt.Format "2006-01-02 15:04:05 UTC"}}</span>
</div>
</div>
{{else}}
+87
View File
@@ -0,0 +1,87 @@
{{define "webhook_stats"}}
<!-- Statistics pane at the top of the webhook page. -->
<div class="card mb-6">
<div class="p-4 border-b border-gray-200">
<h2 class="text-lg font-medium text-gray-900">Statistics</h2>
</div>
{{with .Stats}}
<div class="p-4 flex flex-wrap gap-6 text-sm border-b border-gray-200">
<div>
<span class="text-gray-500">Entrypoints</span>
<span class="font-medium text-gray-900">{{.Entrypoints}}</span>
<span class="text-gray-500">({{.ActiveEntrypoints}} active)</span>
</div>
<div>
<span class="text-gray-500">Targets</span>
<span class="font-medium text-gray-900">{{.Targets}}</span>
<span class="text-gray-500">({{.ActiveTargets}} active)</span>
</div>
<div>
<span class="text-gray-500">Deliveries in progress</span>
<span class="font-medium text-gray-900">{{.InProgress}}</span>
</div>
<div>
<span class="text-gray-500">Last event</span>
<span class="font-medium text-gray-900">{{with .LastEventAt}}{{.Format "2006-01-02 15:04:05 UTC"}}{{else}}none{{end}}</span>
</div>
<div>
<span class="text-gray-500">Retention</span>
<span class="font-medium text-gray-900">{{$.Webhook.RetentionLabel}}</span>
</div>
</div>
<div class="p-4 grid grid-cols-1 lg:grid-cols-2 gap-6 text-sm">
<div>
<div class="flex py-2 border-b border-gray-200 text-xs text-gray-500 uppercase tracking-wide">
<span class="flex-1"></span>
<span class="w-32 text-center">Lifetime</span>
<span class="w-32 text-center">Within retention</span>
</div>
<div class="divide-y divide-gray-100">
<div class="flex py-2">
<span class="flex-1 text-gray-600">Events</span>
<span class="w-32 text-center text-gray-900">{{.Totals.Events}}</span>
<span class="w-32 text-center text-gray-900">{{.Totals.EventsWithinRetention}}</span>
</div>
<div class="flex py-2">
<span class="flex-1 text-gray-600">Deliveries</span>
<span class="w-32 text-center text-gray-900">{{.Totals.Deliveries}}</span>
<span class="w-32 text-center text-gray-900">{{.Totals.DeliveriesWithinRetention}}</span>
</div>
<div class="flex py-2">
<span class="flex-1 text-gray-600">Failures</span>
<span class="w-32 text-center text-gray-900">{{.Totals.Failures}}</span>
<span class="w-32 text-center text-gray-900">{{.Totals.FailuresWithinRetention}}</span>
</div>
</div>
</div>
<div>
<div class="flex py-2 border-b border-gray-200 text-xs text-gray-500 uppercase tracking-wide">
<span class="flex-1"></span>
<span class="w-32 text-center">Last 10 minutes</span>
<span class="w-32 text-center">Last 24 hours</span>
</div>
<div class="divide-y divide-gray-100">
<div class="flex py-2">
<span class="flex-1 text-gray-600">Events</span>
<span class="w-32 text-center text-gray-900">{{.Last10Minutes.Events}}</span>
<span class="w-32 text-center text-gray-900">{{.Last24Hours.Events}}</span>
</div>
<div class="flex py-2">
<span class="flex-1 text-gray-600">Failures</span>
<span class="w-32 text-center text-gray-900">{{.Last10Minutes.Failed}}</span>
<span class="w-32 text-center text-gray-900">{{.Last24Hours.Failed}}</span>
</div>
<div class="flex py-2">
<span class="flex-1 text-gray-600">Failure percentage</span>
<span class="w-32 text-center text-gray-900">{{.Last10Minutes.FailurePercent}}</span>
<span class="w-32 text-center text-gray-900">{{.Last24Hours.FailurePercent}}</span>
</div>
</div>
<p class="mt-2 text-xs text-gray-500">Failure percentage is the failed deliveries out of all deliveries that finished in the window. Deliveries still pending or retrying are not counted.</p>
</div>
</div>
{{else}}
<div class="p-4 text-sm text-gray-500">The statistics could not be read.</div>
{{end}}
</div>
{{end}}