sneak/webhooker
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions 23 Packages Projects Releases Wiki Activity

Compare commits

base: sneak/webhooker:prod
Branches Tags
sneak/webhooker:main sneak/webhooker:next sneak/webhooker:prod sneak/webhooker:issue-70-log-target
sneak/webhooker:v1.0.0-rc3 sneak/webhooker:v1.0.0-rc2 sneak/webhooker:v1.0.0-rc1
..
compare: sneak/webhooker:7eba0d59f58a00745b52d80e19a2ab3fbaa264a7
Branches Tags
sneak/webhooker:next sneak/webhooker:prod sneak/webhooker:main sneak/webhooker:issue-70-log-target
sneak/webhooker:v1.0.0-rc3 sneak/webhooker:v1.0.0-rc2 sneak/webhooker:v1.0.0-rc1
127 Commits
prod .. 7eba0d59f5
Author SHA1 Message Date
clawbot 7eba0d59f5 Use one recovery routine at start and on every sweep tick (closes #542)
check / check (push) Canceled after 0s
Details 
Start-up recovery and the 60-second sweep were two parallel sets of
functions doing the same job. recoverDeliveries now does it, run once at
start and then on every tick from one goroutine. A retrying delivery is
scheduled after what is left of its backoff, now if nothing is left, so
backoffElapsed is gone. The run at start still takes every pending
delivery and opens a webhook database whose file is missing; a tick
still takes only pending deliveries untouched for 15 minutes, at most
500, and skips a missing file. A retrying delivery is taken before its
event is read. The README describes one routine.

Model: opus-5-5
 2026-10-15 19:36:34 +00:00
clawbot 0cdda43e6c Move the archive files into their own package (closes #541)
check / check (push) Canceled after 0s
Details 
The SQLite archive files the database target writes are now managed by
internal/archive: file names, rotation, the writer of each target,
pruning, renames, the gzipped JSON export, the idle sweep, and the
rename lock, which was Handlers.renameMu. The database target stays in
internal/delivery and writes through archive.Writers; the engine still
closes the writers when it stops, so the stop order is unchanged. The
handlers take the rename lock from archive.Archives.RenameLock, and
rename and evict through archive.Archives.

Exported names drop their Archive prefix in the new package. The tests
moved with the code; the two that check a delivery's recorded attempt
stay in delivery.

Model: opus-5-5
 2026-10-15 21:03:43 +02:00
clawbot f10440b7cc Store a webhook's retention as a plain day count, 0 meaning forever (closes #539)
check / check (push) Canceled after 0s
Details 
retention_days loses its column default of 30, and Webhook.RetentionDays
its default tag, so a 0 is stored as 0. The 365000-day stand-in
RetentionForeverDays and the BeforeSave hook that wrote it are gone. The
reaper and the "forever" label read 0 as forever, and the forms accept 0
or 1 to MaxFiniteRetentionDays; a blank field still gives 30 on the new
webhook form and keeps the stored value on the edit form. A negative
value, which the forms refuse, is still read as forever so it cannot put
the reaper's cutoff in the future.

Model: opus-5-5
 2026-10-15 20:03:33 +02:00
clawbot 40e8ecaab4 Store a target's number of delivery attempts, at least one (closes #540)
check / check (push) Canceled after 0s
Details 
Target.MaxRetries (max_retries) becomes Target.MaxAttempts
(max_attempts), and the form field, parser and ceiling are renamed to
match. The column is NOT NULL and a check constraint refuses a value
below 1, so every target, whatever its type, is stored with at least
1. 1 is a single attempt with no circuit breaker; above 1 retries
behind the breaker up to that many attempts. Both target forms refuse
0 and less; a blank field on the add target form gives 1, and on the
edit form keeps the stored value. The target page shows the plain
number, and the README's two paragraphs on 0 against 1 become one
sentence.

Model: opus-5-5
 2026-10-15 19:38:14 +02:00
clawbot 147d58dd96 Say the query logger drops bound values (closes #535)
check / check (push) Canceled after 0s
Details 
The comment on MaxAccessLogLineBytes said the GORM adapter logs each
statement with the client's parameter written into it. Its
ParamsFilter drops every bound value, so the statement is logged with
its placeholders. The comment now says so and keeps the parts that
still hold: the widest line spends two budgets and the adapter's test
checks it against this constant.

The assertBounded doc no longer claims each line was cut, and the
postReceiver doc no longer explains itself by a log size test that is
gone.

Model: opus-5-5
 2026-10-15 18:57:58 +02:00
clawbot c7c4ea3340 Send only the Referer's scheme and host to Sentry (closes #548)
check / check (push) Canceled after 0s
Details 
A Sentry event kept the Referer header verbatim, and a Referer can be
an entrypoint URL: a sender that follows the README's http:// to
https:// redirect sends the URL it was redirected from, and a browser
sends the 404 page's URL under /h/ when it fetches that page's
stylesheet. The Sentry hook now keeps only the Referer's scheme and
host. The access log's reduction moves, unchanged, into a new package,
internal/referer, as SchemeAndHost; the access log and the Sentry hook
both call it there. The comment above sentryKeepsHeader and the
README's Sentry passage say so, and the README's Package Layout lists
the new package.

Model: opus-5-5
 2026-10-15 18:07:21 +02:00
clawbot 8c05ec4378 Build the binary without cgo (closes #545)
check / check (push) Canceled after 0s
Details 
make build now builds with CGO_ENABLED=0, and the image builds through
it. The binary's only cgo code was mattn/go-sqlite3, which
gorm.io/driver/sqlite imports and webhooker never opens: every SQLite
file is opened with modernc.org/sqlite and the connection handed to
gorm. Without cgo, mattn's package compiles to a stub nothing calls.

The Dockerfile builds the binary once. The static relink and its
netgo, osusergo and sqlite_omit_load_extension tags are gone, and with
them the Makefile's GO_LDFLAGS and GO_TAGS, which only the relink used.
The test phase keeps cgo for -race. No README or Dockerfile sentence
says the binary needs cgo any more.

Model: opus-5-5
 2026-10-15 17:36:35 +02:00
clawbot 872e4eb4a1 Drop 'unsafe-inline' from the Content-Security-Policy (closes #530)
check / check (push) Canceled after 0s
Details 
The delete prompts on the webhook page were inline onsubmit handlers.
Each delete form now carries its question in a data-confirm attribute,
which one submit listener in static/js/app.js asks. The three style
attributes became Tailwind classes, with tailwind.css regenerated, and
the x-cloak rule moved to style.css. The policy now sends
script-src 'self' and style-src 'self'.

The browser test now answers a target's delete prompt no and then yes,
and also loads the sign-in page, the webhook list, the settings and
profile pages and the 404 error page under the policy. The README
states the policy as sent.

Model: opus-5-5
 2026-10-15 16:18:26 +02:00
clawbot dfe8f3138f Bind 0.0.0.0 to IPv4 only (closes #528)
check / check (push) Canceled after 0s
Details 
Go listens on any unspecified address with one socket that answers
IPv4 and IPv6, so BIND_ADDRESS=0.0.0.0 also bound every IPv6 address,
which the README says it does not. The listener now uses tcp4 for an
IPv4 address, written plain or IPv4-mapped; :: still binds both. The
server's start hook suppresses contextcheck, as the other start hooks
do.

The image's health check asked for localhost, which wget reached over
::1 only. It now asks for the container's own address: its IPv4
address, or its IPv6 address on an IPv6-only network. A loopback bind
answers neither. The README says such a network needs ::, drops the
bindv6only condition from :: (Go sets that socket option itself) and
names an ss check.

Model: opus-5-5
 2026-10-15 15:44:42 +02:00
clawbot 0d38aea12c Keep the entrypoint UUID out of webhooker's own logs (closes #531)
check / check (push) Canceled after 0s
Details 
On the receiver route the access log and the per-entrypoint 429 line
name the route pattern whatever the status. A request chi never routed,
such as a dev-mode CORS preflight to the receiver, logs (unmatched).
The access log keeps only the scheme and host of a Referer. The
received line names the entrypoint and webhook IDs, and the
unknown-entrypoint line logs no path. The pattern is
middleware.ReceiverRoute, which routes.go registers the receiver with.

The README says the proxy's access log holds the UUID and must be
protected like the URL, and that a log target writes the request's
headers as received. Its access-log, rate-limit, Sentry and log-size
passages now match the code.

Model: opus-5-5
 2026-10-15 14:58:35 +02:00
clawbot c0f7cc29fd Leave a delivery cut off by a stop for the next start (closes #527)
check / check (push) Canceled after 0s
Details 
Stopping the engine cancels the context every http and slack request is
sent with. An attempt cut off that way was recorded as failed, which for
a target with no retries, or on a last retry, ended the delivery failed
and nothing sent it again.

The shared retry core now records nothing for a failed attempt when the
engine's context is cancelled: no result, no circuit breaker outcome, no
status change, no count in the attempt metrics. The delivery stays
pending or retrying and startup recovery sends it again. A target's own
timeout is still a failed attempt. The README says what happens to an
attempt in flight at a stop and no longer claims every attempt is
recorded.

Model: opus-5-5
 2026-10-15 12:29:17 +02:00
clawbot f14fe02011 Store delivery and archive times without the monotonic reading (closes #529)
check / check (push) Canceled after 0s
Details 
finished_at, archived_at and the updated_at stamp a re-dispatch
writes came from a bare time.Now(). The SQLite driver stores a time
as time.Time.String(), which appends the monotonic clock reading
(" m=+…") to a time read from time.Now(). All three now come from
GORM's own clock, NowFunc(), which stamps created_at, so they are
stored in the same form. The tests for each write read the stored
text back.

Model: opus-5-5
 2026-10-15 11:15:35 +02:00
clawbot 573f7f986a Build the image's static binary with netgo, osusergo and sqlite_omit_load_extension (closes #525)
check / check (push) Successful in 7m24s
Details 
The image's static relink linked glibc's getaddrinfo (from Go's cgo DNS
resolver) and dlopen (from SQLite extension loading). A static binary
still needs glibc's shared libraries at run time for both, and the
Alpine runtime has none. The relink now passes the standard build tags
netgo, osusergo and sqlite_omit_load_extension through a GO_TAGS
variable, which the Makefile's build target hands to go build as -tags
next to GO_LDFLAGS, so the version stamp is composed as before.

Model: opus-5-5
 2026-10-13 10:43:47 +02:00
clawbot 373a2f828d Describe code in its own terms, not by an earlier build's data (closes #522)
check / check (push) Successful in 8m12s
Details 
Several comments justified code by what an earlier build would have
left in a database or on disk. Each now says what the code does:
reserveSQLiteFile sets the mode on every open whoever created the
file, and expired treats a session lacking a deadline's timestamp as
expired. The comments on retention, resubmitted events, archive
pruning, target headers, the redactor and the delivery result view
drop their mentions of older rows. parseMaxRetries loses its
paragraph on targets saved before the ceiling existed, and neither
BeforeSave nor retentionCutoff cites a migration. Comments only.

Model: opus-5-5
 2026-10-13 09:57:24 +02:00
clawbot 267d5b12f2 Escape the database path in the SQLite connection string (closes #518)
check / check (push) Successful in 7m13s
Details 
SQLite reads a file: URI's path only up to the first ? or #, and
decodes %XX escapes in it. With a data directory such as
/srv/hooks#1, every database opened the one file /srv/hooks. The
connection string is now built with url.URL, which escapes the path,
and OmitHost keeps a relative path relative.

Tests open databases in a directory whose name holds #, ? and %, and
at a relative path, and check through PRAGMA database_list that each
opened the file asked for. A handlers test comment that blamed # in
the temporary directory's path is dropped, as that is no longer true.

Model: opus-5-5
 2026-10-13 09:30:24 +02:00
clawbot e73beb20df Remove upgrade and older-build text from the README (closes #520)
check / check (push) Canceled after 0s
Details 
The schema comes from 001_schema.sql, edited in place before 1.0, so the
README's upgrade procedure, downgrade note, BIND_ADDRESS upgrade note and
remarks about what older builds wrote are removed.

The description of the SQL schema files moves to a new Schema subsection
under Database Architecture, the health check command moves to Version
stamping, and the section is renamed Backup and Restore. The sentences on
session timestamps, file modes and the retention clamp now say only what
the code does.

Model: opus-5-5
 2026-10-13 08:31:51 +02:00
clawbot c8efbd41d4 Drop Completed Steps from TODO.md; the tracker is the record (closes #112)
check / check (push) Canceled after 0s
Details 
Every issue branch added an entry to the same list, so each branch
conflicted with `next` as soon as another one landed. The tracker already
records what is done, in more detail than the list did.

The Workflow section now says the tracker is that record, in place of the
rule that issue branches leave this file alone. Status points to the 1.0.0
milestone for what is open, and Next Step says @sneak tags `v1.0.0` after
running webhooker in limited production. In Future Steps, each item that
overlaps with what webhooker already does now says what exists and what is
still missing, and the delivery status and retry UI item is gone because the
event log, replay and resubmit cover it.

Model: opus-5-5
 2026-10-13 07:18:48 +02:00
clawbot 15377f792b Build every database's schema from embedded SQL files (closes #56)
check / check (push) Canceled after 0s
Details 
The main database, each webhook's event database and each archive file
get their schema from SQL files under internal/db/migrations/, one
directory per kind. Opening a file runs, in one transaction, each file
that schema_migrations does not list. GORM stays for queries.
AutoMigrate is gone, tests included.

An event database holds only its own tables now, so the sweep that
removed target rows copied into it is deleted. The association guard
stays: without it a delivery write fails on the missing targets table.

AutoMigrate also had GORM parse every model before a handle was shared.
That is now done on its own: GORM races when two goroutines first use
models that refer to each other.

Model: opus-5-5
 2026-10-13 06:34:05 +02:00
clawbot f86ab6321d Name Sentry transactions after the route they matched (closes #185)
check / check (push) Successful in 8m52s
Details 
sentryhttp names each transaction after the concrete path, which the
BeforeSendTransaction hook has to withhold, so with tracing on the
transactions would fall into one Sentry group per method, such as
POST /(redacted). A middleware installed beside every sentryhttp
handler now renames the transaction to the method and chi route
pattern once the request is routed, and sets its source to route. The
hook keeps a route-sourced name and withholds every other name as
before. A path that matches nothing at the top level keeps the
withheld name; one that matches no route inside a route group, or
that the group's middleware turns away, is named after the group's
pattern, such as GET /pages/*. Tracing stays off.

Model: opus-5-5
 2026-10-13 05:47:26 +02:00
clawbot 957de7f99f Make the HSTS max-age configurable with HSTS_MAX_AGE (closes #271)
check / check (push) Failing after 3h11m13s
Details 
HSTS_MAX_AGE, a Go duration, sets the max-age of the
Strict-Transport-Security header. Unset, it is ten years (87600h);
0 sends no header at all. A negative value, or one that is not a
whole number of seconds, stops startup like an unparseable one.
includeSubDomains and preload stay in the header. The value is shown
on /settings and in the startup log, and the README's HSTS section
now says what each part of the header commits the operator to.

Model: opus-5-5
 2026-10-13 03:27:24 +02:00
clawbot 9199544999 Document that DATA_DIR must be on a local filesystem (closes #295)
check / check (push) Successful in 12m55s
Details 
SQLite in WAL mode maps each database's -shm file into memory, and
network filesystems do not support that reliably, so the README now
says DATA_DIR must be local and names NFS and SMB/CIFS as unsupported.
It also says what a disk I/O error with code 522 points at and what to
try first. ZFS is not called unsupported: the one failure seen there
was never reproduced.

Model: opus-5-5
 2026-10-13 01:29:42 +02:00
clawbot 128eb1b644 Keep test helpers out of the shipped binary (closes #506)
check / check (push) Successful in 7m57s
Details 
The test helpers lived in ordinary `testing.go` files inside the config, database, middleware and session packages, so they were built into the binary and the shared `test-support` lint rule could not see them. The four files are gone: the session's helpers move into its own `_test.go` file, and the rest into `configtest`, `databasetest` and `middlewaretest`, which the `depguard` deny list now names, so a non-test file importing them fails lint. The test-support packages build through the production constructors.

Judgement call: the session, the middleware and the webhook database manager now take the plain logger they log through, which the application wiring provides.
Judgement call: two idle-expiry tests move the stored timestamps back instead of advancing a fake clock.

Model: opus-5-5
 2026-10-06 14:29:47 +02:00
clawbot 3de345fe6f Read the export test's heap only after pools drop their caches (closes #511)
check / check (push) Successful in 7m9s
Details 
`TestArchiveExport_Streams` read the heap after one garbage collection, but the libraries the export calls (`regexp` under GORM's table names, `encoding/json`, GORM's row scanning) keep spare buffers in a `sync.Pool`, which keeps them through one collection. Each reading therefore counted however many happened to be cached, which varied by about as much as the test's limit and failed `go test` on `next`. The test now collects twice before each reading, so a reading is the memory the export holds. The limit, the row counts and the claim are unchanged, and an export that does not stream still fails it.

Unverified: the branch's CI run waits for the runner outage to clear.

Model: opus-5-5
 2026-10-06 10:56:50 +02:00
clawbot a9d77e20d7 Make the load-sensitive tests wait for what they check (closes #507)
check / check (push) Successful in 9m48s
Details 
Three tests failed at random on a busy host. The browser test now waits for each page a click opens to load, and for Alpine.js to start on it, before reading it. The delivery tests' drain takes what is already queued instead of racing a 25 ms timer, since both dispatch paths queue before they return. The test phase keeps the tests' temporary directories on a tmpfs, because SQLite waiting for the disk made `internal/handlers` slow under load; the timeout and the parallel cap are unchanged.

Deviation: the delivery tests do not wait with a deadline; dispatch has finished when they read.
Unverified: `internal/handlers` at a host load of 260 to 290.

Model: opus-5-5
 2026-10-06 08:51:35 +02:00
clawbot faf3da9a75 Refresh the package lists before installing in script/bootstrap (closes #508)
check / check (push) Successful in 9m44s
Details 
The CI runner's image starts with empty package lists, so `script/bootstrap`, which `script/cibuild` now runs first, failed to install Go and every CI run stopped there. The apt branch of `pkg_install` now runs `apt-get update` once per run, before its first `apt-get install`, with the same code as the pending shared fix in sneak/prompts#116.

Deviation: `script/bootstrap` differs from the vendored copy until the next re-vendor.
Unverified: proven by running the CI job in the runner's image; the branch's own CI run waits for the runner outage to clear.

Model: opus-5-5
 2026-10-06 07:21:30 +02:00
clawbot fa6a9ed4dc Re-vendor the shared files from sneak/prompts at dd4027b (closes #504)
check / check (push) Failing after 5s
Details 
The shared workflow, lint config, prettier settings and policies are the copies at `sneak/prompts` commit `dd4027b`. `.gitignore`, `.editorconfig` and `.dockerignore` are the shared copy followed by this repository's own entries. Linting is the image build's lint phase on golangci-lint v2.14.0, and tests run in their own test phase. Every scripted `docker build` passes `--no-cache`, so the CI fingerprint step and the superseded-run script are gone. The binary is built with `-trimpath -s -w`, and a build that has `.git` but no version fails. The development run keeps its databases outside the checkout.

Deviation: `.dockerignore` also leaves out SQLite databases at any depth.

Model: opus-5-5
 2026-10-06 06:05:42 +02:00
clawbot 46fe7baed0 Mask a target URL's query string when the URL has no path (closes #500)
check / check (push) Successful in 5m57s
Details 
`urlSecrets` treated a target URL's request URI as a secret only when the URL had a path, so for a target URL such as `https://example.com/?token=…` a response echoing the request line showed the token in the event log and on the event's page. It now also treats the query string, and the request URI that carries it, as secrets whenever the URL has one, whatever its path. With "Pass the query string on to this target" on, only the target's own part is masked, not the event's. A URL with a path is masked as before. As for paths and userinfo, no length floor applies.

Model: opus-5-5
 2026-10-04 03:31:37 +02:00
clawbot ea8cba7264 Store and show an event's query string, and pass it on to an HTTP target when set (closes #312)
check / check (push) Successful in 4m45s
Details 
The receiver dropped the query string of every request it received, so a sender's URL parameters were silently lost. Each event now keeps it, as sent, in a new `raw_query` column of the per-webhook `events` table; a resubmitted copy carries its original's. The event log and the event's page show it in the shared request block, the event log leaving out one over 32 KiB with a link, as for headers. The archive and log targets carry it. HTTP targets gain "Pass the query string on to this target", off by default: on, deliveries, replays and resubmits append it to the target URL, joined with `&` to one already there. The access log still hides it.

Model: opus-5-5
 2026-10-04 03:00:34 +02:00
clawbot a8fc0c5d32 Merge main into next after 416 (closes #497)
check / check (push) Successful in 4m34s
Details 
#416 put a `main`-only version of fixes `next` already had onto `main`, so `next` no longer merged into `main`: `script/test` conflicted. This merge makes `main` an ancestor of `next` and keeps `next`'s files throughout, which already hold everything 416 changed, so `next`'s tree is unchanged.

Model: opus-5-5
 2026-10-03 14:29:50 +02:00
clawbot 7963188c1e Merge main into next after 416 (closes #497)
check / check (push) Successful in 6m8s
Details 
Makes main an ancestor of next, so the milestone PR merges again.
script/test conflicted and keeps next's version, which already runs
without -v, caps memory with -p 4 -parallel 8, adds coverage and reruns
failed tests with -v. password.go, password_test.go, export_test.go and
resetpw_test.go merged on their own: main's lines there are the same
as next's. The merged tree equals next's.

Model: opus-5-5
 2026-10-03 12:17:30 +00:00
clawbotandsneak 16ed356b68 main side of 414: cheaper test hashing, and failing tests visible in the build log (closes #414) (#416)
check / check (push) Successful in 3m21s
Details 
The `main` side of #414: the two changes that make `next` green, and nothing else from `next`. Each is its own commit, so it can be compared with its `next` counterpart.

- #404, as merged to `next`: a test binary hashes passwords at a 1 MB Argon2id cost instead of 64 MB, `TestHashPassword_ShippedParameters` keeps the shipped cost covered, and `script/test` runs at most four packages and eight parallel tests at once. Every test that starts a database hashed the admin password at 64 MB, which on a busy host made `internal/handlers` overrun its application start and its 90-second timeout. That is what turned `main` red.
- #415: `script/test` runs without `-v`, so the build log, which the Docker build cuts off at 2 MiB, carries one result line per package and, for a package that fails, everything its tests wrote, application log lines included, instead of only passing packages.

What the diff does not show: `script/test` differs from `next` by one line. `main` has no `script/assets` yet, so it is not called. Several packages failing at once can still reach the 2 MiB limit.

- Judgement call: both commits keep their subjects from `next`, including their `closes` references.
- Deviation and not fixed here: the same two as on #415 (no `-v` rerun on failure, #315; remaining sensitivity to extreme CPU load, #225).

Model: opus-5-5
Co-authored-by: sneak <sneak@sneak.berlin>
Reviewed-on: #416
Co-authored-by: clawbot <35+clawbot@noreply.example.org>
 2026-10-03 14:08:51 +02:00
clawbot fe5e0d4173 Install ESLint and prettier with yarn 4 through corepack (closes #493)
check / check (push) Successful in 3m48s
Details 
The `js-deps` stage installed ESLint and prettier with yarn 1, which is no longer developed and made node print a `url.parse()` deprecation warning on every install. `package.json` now pins yarn 4 by version and hash in its `packageManager` field; the stage enables it through the node image's own corepack and installs with `yarn install --immutable`. `yarn.lock` is regenerated in yarn 4's format with every package at its previously locked version, and `.yarnrc.yml` keeps the install in `node_modules/`, where the lint and Markdown stages run the tools from. The install prints no warning and stays cached until the manifests change.

Model: opus-5-5
 2026-10-03 07:03:31 +02:00
clawbot 9ccaa8ce01 Break long values on the webhook page and event log at phone width (closes #391)
check / check (push) Successful in 3m17s
Details 
At phone width, the event log cut off long event IDs and content types along with the statuses and times after them, and the webhook page ran long names past their cards and scrolled sideways. Elements that hold only a long value (a webhook, target or entrypoint name or description, an event ID, a content type) now break inside it, and the event log's title row wraps; rows themselves still wrap rather than squeeze. Wide screens are unchanged. A 390-pixel browser check of both pages, an event's attempts open, fails when anything runs past the page's or a card's edge or the page scrolls sideways.

Model: opus-5-5
 2026-10-03 06:56:43 +02:00
clawbot 935e18c6f1 Format the Markdown with prettier in make fmt and make fmt-check (closes #215)
check / check (push) Successful in 3m28s
Details 
`make fmt` formatted only Go, so the org's Markdown settings were unenforced and Markdown was wrapped by hand. prettier, pinned in `package.json` and `yarn.lock` beside ESLint, now formats the Markdown with `.prettierrc` (4-space tabs, `proseWrap: always`). It runs only in Docker: `make fmt` writes the formatted files back without a bind mount, and `make fmt-check`, `make check` and the image build fail on unformatted Markdown. The image build's lint stage now runs the Go format check directly and no longer installs `make`. `README.md` and `TODO.md` are reformatted with no word changed: the README reflows from 72 to 80 columns.

Model: opus-5-5
 2026-10-03 06:32:27 +02:00
clawbot af91d8a723 Split source_management.go along its CRUD seams (closes #274)
check / check (push) Successful in 3m22s
Details 
`internal/handlers/source_management.go` had grown to about 2,370 lines holding the webhook, event log, entrypoint and target handlers, so unrelated units had to wait on each other to touch it. It is split, as pure code movement, into files named for what they hold: `webhook_list.go`, `webhook_create.go`, `webhook_detail.go`, `webhook_edit.go`, `webhook_delete.go`, `event_log.go`, `entrypoint.go`, `target_create.go`, `target_delete.go`, `target_toggle.go` and `shared.go`. No function body, signature, comment or behaviour changed. The README's file tree and log-line caveat, and one middleware comment, name the new files.

Model: opus-5-5
 2026-10-03 06:19:46 +02:00
clawbot 7e779f7fce Event log: show only the events with a failed or a pending delivery (closes #390)
check / check (push) Successful in 3m18s
Details 
The event log had no way to list only the events whose delivery failed, and once it showed only the 50 newest, an older failure could not be found at all. It now has All, Failed (N) and Pending (N) links, carried in a `show` query parameter, so they work without the page's script library. Each filtered list keeps the 50-row limit and newest-first order, and lists an event once. It finds matching deliveries through `idx_deliveries_status` and looks their events up by ID, so its cost follows the matches, not the webhook's size. Replay returns to the list it was pressed in. The heading line says what a filter counts.

Model: opus-5-5
 2026-10-03 05:40:34 +02:00
clawbot 9079a3219d Show an event's entrypoint and request headers in the event log and on its page (closes #389)
check / check (push) Successful in 3m26s
Details 
The receiver stored each event's request headers and entrypoint, but no page showed them, so with several entrypoints the operator could not tell which sender sent an event. An expanded event in the event log, and the event's own page, now show the entrypoint by its description ("Entrypoint" without one, "deleted entrypoint" once deleted), never its URL, and the headers as one escaped block, sorted, whitespace kept. A resubmitted copy says the request it copies arrived there. The event log reads at most 32 KiB of headers per event, the body's limit, and links to the event's page beyond it. Both pages share one template, `event_request.html`.

Model: opus-5-5
 2026-10-03 05:22:14 +02:00
clawbot b9ec91c0f7 Use one name for each thing the UI shows (closes #399)
check / check (push) Successful in 3m21s
Details 
The UI gave one thing several names. The `database` type is now Archive in the type list, on its badge and on the edit page, and its settings read "Archive expiry" and "Archive rotation" everywhere. The retry field is "Delivery attempts", with help text, errors and the target list saying it counts every attempt; a stored 0 shows as one attempt. The target list uses one capitalisation. The navbar says "Sign out", the sign-in page "Sign in", and the resubmit notice "webhook" instead of "source". The README follows. Stored values, their meaning, routes and form field names are unchanged.

Model: opus-5-5
 2026-10-03 05:07:41 +02:00
clawbot 74a96b2226 Lint static/js/ with ESLint in Docker (closes #120)
check / check (push) Successful in 3m50s
Details 
`REPO_POLICIES.md` binds the repo to a JavaScript styleguide, but nothing checked `static/js/`. ESLint, pinned by `package.json` and `yarn.lock`, now lints it with the styleguide's two checkable rules, `no-var` and `prefer-const`. It runs only in Docker on a digest-pinned node image: a `js-deps` stage installs ESLint and stays cached until the manifests change, and a `js-lint` stage runs it. `script/lint` builds `js-lint`, so `make lint` and `make check` fail on a violation, and the image build depends on it as on the repo's other checks. ESLint, node and yarn are not prerequisites, and `make lint` never uses a host copy.

Model: opus-5-5
 2026-10-03 04:24:11 +02:00
clawbot 8b617efa63 Rotate a database target's archive monthly, daily or hourly (closes #379)
check / check (push) Successful in 3m27s
Details 
A database target's rotation setting (none, monthly, daily or hourly) puts the UTC period of each event's receive time in its archive file name, so each file holds exactly its period's events. It is on the new webhook page and both target forms, and shown in the target list. Renames move every one of a target's files and move them all back if one fails. The sweep prunes one file at a time under the target's lock and deletes a rotated file it leaves empty. Download opens one file at a time, oldest period first, finding each again under the target's current name. The target list names the current file and totals all of them.

Model: opus-5-5
 2026-10-03 04:18:37 +02:00
clawbot d8c60c9b67 Event log: show attempt and delivery times, label replays, zone event times (closes #386)
check / check (push) Successful in 3m21s
Details 
In the event log and on an event's page, attempts and deliveries showed no time, event times had no zone, and a replay looked like the original it repeated. Each attempt now shows when its result was recorded and each delivery when it was created, as how long ago with the full UTC time on hover, and the event log's event times read the same way. A delivery created by Replay records it in a new `replay` column and is labelled a replay in the event's summary line and its delivery list; replays made before this change are not labelled. A delivery's row is now drawn by one template, `delivery_row`, that both pages share.

Model: opus-5-5
 2026-10-03 04:12:27 +02:00
clawbot ea8384f4a2 Event log: only the newest event expanded, and only the 50 most recent (closes #349)
check / check (push) Successful in 3m17s
Details 
The event log now loads only the 50 newest events, limited in its query, and only the newest starts expanded; the rest start collapsed. Paging is removed rather than capped, since 50 events never need a second page: the Previous and Next links, the `page` query parameter and the page number Replay and Resubmit carried back are gone, and a `?page=` left in an old link shows the 50 newest. A webhook with more than 50 events reads "50 most recent of N events" beside the heading. Comments and a README line that called `page` the only query parameter the service reads now name `next` and `notice`.

Model: opus-5-5
 2026-10-03 03:43:27 +02:00
clawbot 17e6c85dd8 Name the item and what is lost in each delete prompt (closes #400)
check / check (push) Successful in 3m19s
Details 
The three delete prompts on the webhook page were generic ("Delete this target?") and named nothing. Each now names the item and says what is lost: for a webhook, its stored events, with their number (the figure the statistics pane shows), and their deliveries, while any archive files it wrote are kept; for an entrypoint, that senders using its URL get an error from now on and the URL cannot be restored; for a target, that nothing more is delivered to it while its past deliveries stay in the event log. They stay the browser's own prompts, and a name's quotes, backslashes, newlines or a closing script tag reach the prompt escaped.

Model: opus-5-5
 2026-10-03 03:33:14 +02:00
clawbot bcdd4791ec Say what a database or log target's attempt did, without a status (closes #388)
check / check (push) Successful in 3m14s
Details 
In the event log and on an event's page, every attempt by a `database` or `log` target read "success  Status: — (no response)". Those targets make no HTTP request, so there is no response to have, and "no response" reads like a failed connection, which is what it means for an `http` target. A successful `database` attempt now reads "archived" and a successful `log` attempt "written to the log", with no status shown; a failed one keeps "failure" and its error line, also with no status. `http` and `slack` attempts are unchanged. The change is in the one shared attempt template.

Model: opus-5-5
 2026-10-03 02:41:17 +02:00
clawbot f1da5e73dd Event log: an event's ID can be selected without toggling it (closes #348)
check / check (push) Successful in 3m20s
Details 
An event's row in the event log was a button element, whose text a browser will not let be selected, so an event's ID could not be copied. The row now has the button role instead: focusable, toggled by Enter and Space, with `aria-expanded` giving its state. A click on its text toggles the event after 500 ms, the usual double-click interval; the second press of a double- or triple-click cancels that, so selecting the ID leaves the event as it was. A drag over text does not toggle, and a click on the caret toggles at once. The browser test clicks as a person does, so toggling on the first click fails it.

Model: opus-5-5
 2026-10-03 02:32:09 +02:00
clawbot d2ecb83923 Offer no Replay for a delivery to a deleted target (closes #387)
check / check (push) Successful in 3m18s
Details 
In the event log, a delivery to a deleted target still offered Replay, and pressing it answered "Recreate the target, then replay", advice that cannot work: a recreated target is a new one, and the old delivery still names the deleted one. Such a delivery now has no Replay button, and its row still names the target marked "(deleted)". The refusal, which a page loaded before the delete can still reach, now tells the operator to use Resubmit to send the event to the webhook's currently active targets.

Model: opus-5-5
 2026-10-03 02:13:17 +02:00
clawbot 643077021d Show a target paused by its circuit breaker (closes #385)
check / check (push) Successful in 3m16s
Details 
A target whose circuit breaker had tripped still showed as Active, and its deliveries sat at a bare "retrying" with no attempts. Its row on the webhook page now says deliveries are paused after repeated failures and until when the cooldown ends, adding that one waiting delivery is then sent to test the target; while half-open it says deliveries are held while one tests it, with no time. Waiting deliveries show "next try no earlier than" the later of the cooldown and their own backoff, with the date when not today. The engine gains one read of a breaker's state and remaining cooldown under one lock, and shares the backoff formula.

Model: opus-5-5
 2026-10-03 02:06:19 +02:00
clawbot 22fa502638 Remove an http target's max_queue_size (closes #477)
check / check (push) Successful in 3m18s
Details 
An http target's max_queue_size was stored and shown in the target list as "Max Queue Size", but nothing in the delivery engine read it, so an operator who set it expecting deliveries to be bounded got nothing. It is removed from the target, the target list and the README's target table; neither form had a field for it. Nothing checks for a leftover value. An existing database keeps its old column, which is no longer read.

Model: opus-5-5
 2026-10-03 01:39:13 +02:00
clawbot 6395210474 Show each page's own title in the browser tab (closes #117)
check / check (push) Successful in 3m17s
Details 
Every browser tab read "Webhooker": parsePageTemplate parsed each page before htmlheader.html, whose {{block "title"}} fallback then redefined the page's {{define "title"}}. The page file is now parsed last, so its title replaces the fallback (a later definition of a template name replaces an earlier one, and an empty one never does). A test renders every page template and checks its browser tab title.

Model: opus-5-5
 2026-10-03 01:30:02 +02:00
clawbot 3489d6909a Offer archive expiry choices on the target forms, show plain units (closes #396)
check / check (push) Successful in 3m11s
Details 
A database target's archive expiry was typed by hand as never or a raw duration such as 720h, and the target list showed it back raw. Adding or editing a database target now offers the new-webhook page's list of choices (never, 1h, 12h, 24h, 30d, 90d, 365d), defined once and shared by all three forms. The edit form starts on the stored expiry, or on the submitted one after a refused save; a stored value outside the choices is listed under its own value, so saving unchanged keeps it. The target list shows the expiry in plain units: "30 days", "12 hours", "never".

Model: opus-5-5
 2026-10-03 01:16:14 +02:00
clawbot f282c6363d Keep what was typed when a target or webhook edit is refused (closes #381)
check / check (push) Successful in 3m17s
Details 
A refused save on the target edit page answered with a bare text page, losing the form and everything typed, and the webhook edit page came back with the stored values instead of the submitted ones. A refused target edit now shows the edit form again with the reason above it and every value submitted, with the same status codes as before; a refused webhook edit keeps the submitted name, description and retention. Target edits use the same validation as new targets, with no second copy; an encoding or database failure stays a logged 500. The browser test covers a refused save on both pages, and its main function is now a plain list of checks.

Model: opus-5-5
 2026-10-03 00:51:56 +02:00
clawbot 61371d388e Pin tailwindcss and check the committed stylesheet against it (closes #231)
check / check (push) Successful in 3m25s
Details 
make css ran whatever tailwindcss binary was on the host's PATH, so the committed stylesheet depended on the machine that built it, and nothing noticed when a template used a class the stylesheet lacked. make css now runs the standalone tailwindcss v4.2.1, pinned by sha256, in a Dockerfile stage, and a check stage, run by make check and required by the image build, fails when the committed static/css/tailwind.css differs from what the templates need, showing the differing rules. input.css names its sources. The unused .btn-text is removed and the stylesheet regenerated, dropping only unused rules. The README has a Stylesheet section.

Model: opus-5-5
 2026-10-03 00:30:57 +02:00
clawbot 19a6705c63 New-webhook page: optional HTTP target URL and archive with pruning (closes #373)
check / check (push) Successful in 3m19s
Details 
The new-webhook page gains an optional HTTP target URL, which creates an http target named HTTP, and an archive checkbox whose pruning choice (never, 1h, 12h, 24h, 30d, 90d, 365d) creates a database target named Archive with that expiry. Both are validated by the add target form's own validation, and the webhook, its entrypoint and its targets are created in one transaction or not at all. A refused form comes back with the reason and every value entered, retention included. The targets can be renamed on the webhook page like any other.

Model: opus-5-5
 2026-10-03 00:21:56 +02:00
clawbot 93911f28f9 Show a slack target's retry setting in the target list (closes #395)
check / check (push) Successful in 3m24s
Details 
A slack target's edit page offers Max Retries and the delivery engine honours it, but the target list showed only its masked webhook URL, so setting retries changed nothing visible. The list now shows a slack target's Max Retries line exactly as an http target's, from the one function both use, so the label and the "0 (fire-and-forget)" wording cannot drift apart. The Max Queue Size line stays on http targets only. Tests cover a slack target with retries set, and one with a queue size stored that shows no queue-size line.

Model: opus-5-5
 2026-10-03 00:19:13 +02:00
clawbot 9305af4f85 Show each target's delivered and failed deliveries in the target list (closes #372)
check / check (push) Successful in 3m17s
Details 
The target list showed nothing about how a target's deliveries were going. Each target now shows Delivered and Failed, each in total and in the last 24 hours. The totals are the per-target running totals kept for the statistics pane, so retention does not reduce them; the 24-hour figures are one count over the deliveries' final-status index, for all of the webhook's targets at once. Pending and retrying deliveries count in neither. If the event database cannot be read, each row says so instead of showing zeros. The archive details and Download button on database targets are kept.

Model: opus-5-5
 2026-10-02 23:47:16 +02:00
clawbot ff24638ba4 Show each entrypoint's last event and event count on the webhook page (closes #393)
check / check (push) Successful in 3m12s
Details 
The entrypoint list showed no sign of whether anything uses an entrypoint, so an operator with several could not tell which senders are live before deactivating or deleting one. Each entrypoint now shows when its last event arrived, relative with the UTC time on hover, or "never", and how many events arrived through it within the webhook's retention. The last-event time comes from a new entrypoint_totals row written in the transaction that stores the event and left by retention, so a sender quieter than the retention period does not read "never". The count is one grouped query over a new index. Resubmitted copies count in neither. Pre-1.0: schema changed in place.

Model: opus-5-5
 2026-10-02 23:16:07 +02:00
clawbot da75950e91 Targets section: one Add, then a target type and Next, then that type's fields (closes #370)
check / check (push) Successful in 3m18s
Details 
The targets section of the webhook page showed its add form open with every field, a URL included for types that use none. It now lists only its targets until "+ Add" is clicked; "+ Add" shows a choice of target type with Next and Cancel on one row, and Next shows the name and only that type's fields. The database and log types show no URL field and the server stores none for them; the slack form gains its retry field. A refused target brings the page back with the form open on its type, the values entered and the reason, and Cancel empties it. An encoding failure stays a logged 500. Target validation returns its message, so the new-webhook page can reuse it.

Model: opus-5-5
 2026-10-02 22:24:38 +02:00
clawbot 719d7013ee Give each event its own page and show bodies the same everywhere (closes #369)
check / check (push) Successful in 3m17s
Details 
Each event now has its own page at /hook/ID/events/EVENTID, behind the login, showing its details, its whole body and every delivery; a resubmitted copy links to its original's page. The recent events on the webhook page link there and expand to show their bodies, only the newest expanded on load. One renderer and one template show a body the same way in the recent events, the event log and the event's page: whole up to 32 KiB, cut there in the two lists with links to the event's page and the download; JSON pretty-printed unless that would grow it past four times plus 1 KiB; over 200 lines in a scrolling box; a body holding NUL or control characters treated as binary and never dumped raw.

Model: opus-5-5
 2026-10-02 22:00:42 +02:00
clawbot faf7ca1a5e Report or refuse each unusable file webhooker reads (closes #290)
check / check (push) Successful in 3m20s
Details 
An audit of every file webhooker reads configuration or required state from found cases that carried on silently. A zero-length webhooker.db, and a missing or zero-length per-webhook database, now log the "created a new, empty database" warning naming the file; restart recovery opens every live webhook's database, checking under the manager's lock that it still exists, so a missing one is reported at start. The main database's open errors name webhooker.db, for the server and webhooker resetpw; resetpw refuses a zero-length webhooker.db. A directory in place of any database file or its -wal or -shm is refused naming it. The README says how each case is treated. Also closes #459.

Model: opus-5-5
 2026-10-02 21:38:47 +02:00
clawbot 0f5f6ba6bf Let an entrypoint's description be edited in place (closes #392)
check / check (push) Successful in 3m19s
Details 
An entrypoint's description was set when it was added and could never change, so renaming one meant deleting it and adding a new one with a new URL every sender had to be given again. Each entrypoint on the webhook page now has an Edit button, in the shared secondary style, that opens its description in place with Save and Cancel and keeps its URL. The save goes through the same login, CSRF and ownership checks as the other entrypoint actions; an empty description shows as "Entrypoint". Activate and deactivate now write only the active column, so they cannot undo an edit. Tests cover each, through the router and the browser.

Model: opus-5-5
 2026-10-02 21:32:43 +02:00
clawbot 820d9391ff Index the event log's resubmit count with deleted_at (closes #325)
check / check (push) Successful in 3m22s
Details 
The event log's resubmit count, run on every page load over up to 25 event ids, read every live event in the webhook's database: GORM adds deleted_at IS NULL, and SQLite, keeping no statistics there, chose the deleted_at index over the resubmitted_from_id one. deleted_at is now the second column of idx_events_resubmitted_from_id, so the count is answered from that index for a whole page of events. A test checks SQLite's plan for the statement as GORM builds it, with a full page of ids. The README's event-tier indexes table lists the index. Pre-1.0: the index changes in the schema in place, with nothing for older databases.

Model: opus-5-5
 2026-10-02 21:23:33 +02:00
clawbot 2967c475a1 Show a database target's archive file, size and last write in the target list (closes #397)
check / check (push) Successful in 3m17s
Details 
For a database target, the target list showed only its expiry, so the archive file the README's backup and move-away advice depend on could only be found from a shell on the host. Each database target now shows its archive file's name, its size on disk (the file and its -wal together) and when it was last written, relative with the full UTC time on hover, all from the files' metadata without opening the archive. Before the first write, and after the file has been moved away, it shows the name and "not created yet". Tests cover all three states.

Model: opus-5-5
 2026-10-02 21:22:33 +02:00
clawbot 35d2f28c67 Name each embedded static file so a missing Alpine.js fails the build (closes #166)
check / check (push) Successful in 3m20s
Details 
static/static.go embedded the css and js directories, so a build without the extracted Alpine.js file compiled and produced a binary whose admin pages silently had no Alpine. It now names the four files the pages load, so such a build fails naming js/alpine.min.js; make build extracts the file first, so the failure shows when that step is skipped. Both lint stages extract Alpine.js before linting, since the static package no longer compiles without it, and the README's lint stage says so. static/css/input.css, the Tailwind source no page loads, is no longer embedded or served.

Model: opus-5-5
 2026-10-02 20:56:35 +02:00
clawbot 9526e961b5 Add a Download button that exports a database target's archive as gzipped JSON (closes #374)
check / check (push) Successful in 3m29s
Details 
Each database target on the webhook page has a Download button that streams its archive as gzipped JSON, archive-WEBHOOKNAME-TARGETNAME-TIME.json.gz, with names made safe by delivery.ArchiveFileName's function. The export reads one consistent snapshot through one cursor in a read-only transaction, so archive writes carry on, and holds the rename lock only while it reads the stored names and opens the file. It extends its write deadline as it writes, so a large archive downloads for as long as the client reads; a failure after the response has started aborts the connection so the browser marks the download failed. The request limit is now the service's own middleware, which no longer writes a 504 over a response already started.

Model: opus-5-5
 2026-10-02 20:32:09 +02:00
clawbot 4915d60d8e Route the delivery tests' gorm.Open through gormlog (closes #462)
check / check (push) Successful in 3m17s
Details 
Six test-only gorm.Open calls in internal/delivery passed a bare gorm.Config, leaving the unfiltered idiom in the tree to be copied into production code, where every gorm.Open goes through gormlog.New. They now pass gormlog.New over a logger that discards, so no gorm.Open in the tree uses a bare gorm.Config. The stale sentence saying the tree has one test-only (*gorm.DB).Scan caller is corrected in the README and in the ParamsFilter comment: only tests call Scan, and what a test binds is fixture data. Test and documentation change only.

Model: opus-5-5
 2026-10-02 20:20:49 +02:00
clawbot 0945831442 Clamp the HTTP drain by the tail-hook reserve (closes #170)
check / check (push) Successful in 3m17s
Details 
The HTTP drain at shutdown waited up to ShutdownTimeout regardless of how much of the stop budget earlier hooks had used, so a slow archive sweeper or retention reaper could eat the reserve the hooks after the server need, and the database close was skipped. The drain now waits at most the shorter of ShutdownTimeout and what is left of the budget less TailHookReserve, as the Sentry flush already does. The reserve is documented as derived from the two timeouts. Tests cover earlier hooks having spent part of the budget, on a clock that host speed cannot move, and pin that a drain on the full budget gets all of ShutdownTimeout.

Model: opus-5-5
 2026-10-02 20:11:43 +02:00
clawbot f82b730c31 Pin the HTTP target's unpinned error checks (closes #285)
check / check (push) Successful in 3m20s
Details 
Seven error checks in internal/delivery/target_http.go could be removed without any test noticing, among them withRetry's check on writing the delivery result, the branch that leaves a sent delivery retrying and recoverable when its bookkeeping write fails. Each now fails a test when removed. The "send succeeded" case starts from a tripped circuit breaker, so a probe whose send succeeds but whose result write fails must still close the breaker. The checks in remainingBackoff and backoffElapsed stay unpinned: removing them gives the same answer, and they state a rule a reader needs. Test change only.

Model: opus-5-5
 2026-10-02 19:37:36 +02:00
clawbot 1a1fee0874 Name the reaper's hard delete in the event body comments (closes #455)
check / check (push) Successful in 3m20s
Details 
The comments on eventBodyQuery and on TestHandleEventBodyDownload_ReapedEvent404s credited the soft-delete predicate for refusing a reaped event. The retention reaper deletes event rows outright and nothing soft-deletes an event, so a reaped event is simply gone. Both comments now say so; the test's "soft deleted" case is described as pinning the query's deleted_at predicate for a row no code produces today. Comments only.

Model: opus-5-5
 2026-10-02 19:36:30 +02:00
clawbot 290925f184 Fix two resubmit comments and test the resubmit route's middleware (closes #252)
check / check (push) Successful in 3m18s
Details 
Two comments named the wrong mechanism: loadResubmitSource credited soft-delete for refusing a reaped event, though the retention reaper deletes event rows outright, and createAndFanOut claimed to be the only path that creates deliveries, though per-delivery replay creates one without an event. Both now say what the code does. The resubmit route's middleware had no tests through the router; new tests drive the production router to pin the refusal without a valid CSRF token, the rate limit, signed-out requests never spending it, and another webhook's event refused by the event lookup while the user's own event is accepted. Each fails with its check removed.

Model: opus-5-5
 2026-10-02 19:20:40 +02:00
clawbot 73353bc8e5 Harden the (*gorm.DB).Scan guard test (closes #232)
check / check (push) Successful in 3m17s
Details 
The test that refuses production calls to (*gorm.DB).Scan, the one GORM path that bypasses the logger's value suppression, overstated what it checks and could pass while skipping a whole package. Its comments now say it matches receiver method names, not types, and name the evasion this leaves; GORM's Rows is dropped from the accepted names. Method values are stated as out of scope with the reason. The file-count floor is replaced by a check that every package the walk parses, static and templates included, was reached. The planted snippets are valid Go and cover each receiver form the guard claims to handle. Test change only.

Model: opus-5-5
 2026-10-02 19:19:45 +02:00
clawbot 40f59ec4d2 Try every event database file on delete and say which was left (closes #275)
check / check (push) Successful in 3m20s
Details 
DeleteDB returned on the first event database file it failed to remove, so with WAL sidecars present a failure could be reported as "the file was left behind" after the events themselves were already gone. It now tries the database file and both sidecars, and its error says which case happened and names each file left: the database file is still on disk, or it is gone (the events are lost) and only a -wal or -shm sidecar remains. The webhook delete handler logs a different message for each and never says the events survive when they do not. Tests cover a normal delete with both sidecars, and a failed removal of the database file or a sidecar, through DeleteDB and the handler.

Model: opus-5-5
 2026-10-02 18:42:29 +02:00
clawbot 806c95e305 Make every clickable control look clickable, in two shared styles (closes #375)
check / check (push) Successful in 3m16s
Details 
Many controls were plain coloured text with no sign they could be clicked: Edit, Activate, Deactivate, Delete, Replay, Resubmit, both Add controls and the copy control beside each entrypoint URL. Every clickable control is now a real button or link in one of two shared styles: the buttons in input.css, now with a pointer cursor, and btn-small, a small bordered secondary action in style.css, which the layout now loads. The site name, the footer links and each webhook list card show at rest that they open something, with focus states. The copy control shows "Copied" after a click. Rows on the webhook page and in the event log wrap at phone width. The browser test covers the copy control.

Model: opus-5-5
 2026-10-02 18:32:19 +02:00
clawbot 45bd7e9b94 Pin the close before the reopen in the archive sweep (closes #103)
check / check (push) Successful in 3m35s
Details 
The archive sweep closes a target's archive connection before it reopens the file, and no test noticed if that close was removed, which would leak one SQLite connection per target per sweep. A test now keeps the connection from before a sweep that reopens the archive and requires it closed afterwards. The sweeper's query for database targets takes the sweep's context; a sweep whose context is done returns without an error line, so stopping is not logged as a failure, and a test pins that. The comment on the sweeper's cancel function gives the true reason it needs no lock: fx calls the stop hook only after the start hook has returned.

Model: opus-5-5
 2026-10-02 18:16:31 +02:00
clawbot bf3df0312b Clear the environment in the cmd/webhooker tests that build a Config (closes #452)
check / check (push) Successful in 3m16s
Details 
The two cmd/webhooker tests that build the app graph, TestNewApp_StopTimeout and TestNewApp_SendsFxEventsToTheLogger, built a Config without clearing the environment, so a variable exported in the developer's shell changed their result: a METRICS_USERNAME without METRICS_PASSWORD failed the second. Both now call config.ClearEnvForTest before setting their own variables, as the config tests and the first-boot test already do. No other test outside internal/config builds a Config through config.New. Test change only.

Model: opus-5-5
 2026-10-02 18:03:31 +02:00
clawbot 503c57efd9 Assert the body a retry delivers, not only its status (closes #294)
check / check (push) Successful in 3m24s
Details 
TestProcessRetryTask_LargeBody_FetchFromDB and TestProcessRetryTask_SuccessfulRetry checked only that the delivery ended delivered, so deleting the event-body fetch on the retry path, the behaviour the first is named for, left both green while a retry could deliver an empty or truncated body. Both now compare the body the target received with the stored event body byte for byte, and both fail when that fetch is deleted. The other retry-path tests are not about the body and are unchanged. Test change only.

Model: opus-5-5
 2026-10-02 18:03:20 +02:00
clawbot d084f4f912 Pin the body cap's order in every page route group (closes #93)
check / check (push) Successful in 3m18s
Details 
Follow-ups from an August review of the body cap, each checked against the current tree. One route test now requires an oversized POST, with no session and no CSRF token, to be refused with 413 before CSRF runs, in every page route group with a POST route and in /settings/, so moving a group's body cap after CSRF fails it. The three test router helpers build the server through New with a lifecycle that is never started, so no field is set by hand. The middleware test comment names runMaxBodySize, and the MaxBodySize doc comment says methods other than POST, PUT and PATCH pass uncapped on purpose. The README item was already settled.

Model: opus-5-5
 2026-10-02 17:53:21 +02:00
clawbot e67fffb05d Check the log charge against every code point (closes #172)
check / check (push) Successful in 3m24s
Details 
The access log's 2,560-byte line ceiling holds only if logfield.EncodedBytes charges each code point at least what the log handlers write for it, and the test checked that on a sample. TestEncodedBytes_ChargesAtLeastWhatTheHandlersEmit now covers every code point, surrogates aside, for both handlers. Below U+1000, where the handlers' escaping varies, each code point is measured alone, both in a bare value and in a quoted one, so undercharging any of them, DEL included, fails and names it. From U+1000 up it compares batch sums, which the doc comment says can hide one JSON-only overcharge. Reverting the astral charge to 6 fails the test. It adds under 2 seconds under -race.

Model: opus-5-5
 2026-10-02 17:53:10 +02:00
clawbot 4958a6f2e4 Isolate config tests from the shell; any out-of-range PORT is ErrInvalidPort (closes #94)
check / check (push) Successful in 3m17s
Details 
The config tests unset variables without restoring them and read whatever the developer's shell exported, so results could differ from one machine to the next. config.ClearEnvForTest, in internal/config/testing.go, unsets every variable in the process environment and, when the test ends, leaves it exactly as it found it; every config test and the first-boot test call it before setting their own. TestEnvPositiveInt and TestEnvPort share one table runner, and the port errors name the bad value. A PORT of zero, below zero, or too large to parse now matches ErrInvalidPort, as one above 65535 already did. The README and the Settings page say an unparseable or non-positive RETENTION_SWEEP_INTERVAL stops startup.

Model: opus-5-5
 2026-10-02 17:44:30 +02:00
clawbot 385fbc1a6a Send fx's own events through the service's logger (closes #183)
check / check (push) Successful in 3m14s
Details 
fx printed its dependency graph and lifecycle hooks through its own console logger on standard error, so an operator shipping the JSON log to a collector got a second shape on a second stream for every start. The production app now passes fx.WithLogger with a small FxLogger in internal/logger that writes fx's events through the service's logger: graph events at debug, lifecycle at info, failures at error. Its constructor takes the configuration, so DEBUG=true applies before fx replays the events it held back. go.uber.org/fx moves from v1.20.1 to v1.24.0. Tests keep fx.NopLogger. The README says a failure before the logger exists, and the Go runtime's own output, still go to standard error as plain text.

Model: opus-5-5
 2026-10-02 17:22:05 +02:00
clawbot c22ca6218e Pin the rate-limit key for an empty RemoteAddr (closes #168)
check / check (push) Successful in 3m25s
Details 
A request whose RemoteAddr is empty has no peer identity, so the rate limiters' key falls back to the raw empty string and every such request shares one bucket: it fails closed rather than giving each its own. net/http always fills RemoteAddr for a TCP listener, so normal serving never reaches this. The behaviour is unchanged and now deliberate: a test pins the shared key, and a one-sentence comment at the fallback tells the empty case apart from a Unix-socket listener, where every peer legitimately carries the same address.

Model: opus-5-5
 2026-10-02 17:09:23 +02:00
clawbot 0ccb01cada Close the retention follow-ups from the August review (closes #99)
check / check (push) Successful in 3m17s
Details 
Follow-ups from an August review of the retention bounds, each checked against the current tree. A test now pins that a retention value above the keep-forever sentinel is stored as the sentinel. The form's retention parser returns its message directly, so the two error values that were never compared, and the function that mapped them to messages, are gone. The sweep's own keep-forever skip, which duplicated the check in retentionCutoff, is removed; the cutoff is now asked before the webhook's database is opened. The create-form refill test uses HTML-special characters and checks they come back escaped. The README item was already settled; handling for rows made by hand is declined.

Model: opus-5-5
 2026-10-02 16:50:34 +02:00
clawbot 1f22b30de3 Log the client address next to the peer address (closes #270)
check / check (push) Successful in 3m34s
Details 
Behind a trusted proxy every log line named only the proxy, so abuse could not be traced from webhooker's own logs although the rate limiters already knew the client. The access log, the rate-limit rejection lines, the CSRF warning and the receiver's request line now carry clientIP next to remoteIP. remoteIP still means the connecting peer; clientIP is the address the rate limiters key on, the forwarded client when the peer is inside TRUSTED_PROXIES, worked out once per request by the same code. The README says the field is only as trustworthy as TRUSTED_PROXIES. The access log's 2,560-byte line ceiling holds with the field charged, and a size case with an oversized X-Forwarded-For pins it.

Model: opus-5-5
 2026-10-02 16:50:22 +02:00
clawbot debe588bba Seed the retention tests 50 rows per insert, not 500 (closes #198)
check / check (push) Successful in 3m18s
Details 
Three retention tests made internal/database the slowest test package, mostly by seeding thousands of rows 500 per insert: the SQLite driver finds each parameter's value by scanning the statement's arguments from the first until it reaches that parameter's, so binding grows with the square of the parameter count. They now seed the same rows 50 per insert, about three times faster; no test case or assertion changes. The package drops from 13 to 22s to about 7s. What keeps make test above the 20s target is now mostly the cold -race compile of the tree, which moves with host load. The 90s per-package timeout stays; script/test's header records the new figures.

Model: opus-5-5
 2026-10-02 16:27:00 +02:00
clawbot e8379272ae Load Alpine's CSP build so the UI's directives run (closes #371)
check / check (push) Successful in 3m27s
Details 
Every page's Content-Security-Policy forbids eval, which the standard Alpine.js build needs, so no directive ran in a browser: both add forms on the webhook page showed open, and events in the event log could not be collapsed. The UI now loads Alpine's CSP build (@alpinejs/csp 3.14.9 in 3p/); the policy is unchanged. Each directive names a property or method of a component registered in static/js/app.js (collapsible, targetForm), and each card holds its own x-data. A browser test, built only with the browser tag, loads the webhook page and the event log under the real headers; make test-browser runs it in Docker. New test-only dependency chromedp, which raises golang.org/x/sys to 0.47.0.

Model: opus-5-5
 2026-10-02 16:09:03 +02:00
clawbot 3e209bfe4e Drive the archive reopen debounce test from a clock (closes #190)
check / check (push) Successful in 3m27s
Details 
The archive writer's reopen debounce test made two writes that had to land inside the real 2-second window, then slept 2.1 seconds to cross it, so a slow host could turn correct code red. The archive writer now reads the time for its reopen debounce from a clock field, time.Now in production, and the test moves that clock instead of sleeping: two writes at one instant open the file once, and a write one debounce later closes and reopens it once. Removing the debounce check fails the test. This was the last test whose result depended on real elapsed time.

Model: opus-5-5
 2026-10-02 16:03:01 +02:00
clawbot 88b961c115 Keep each model's parent out of its JSON (closes #177)
check / check (push) Successful in 3m16s
Details 
Every reference from a model to the record it belongs to (a target's or entrypoint's webhook, a webhook's or API key's user, an event's webhook and entrypoint, a delivery's event and target, a delivery result's delivery) is now tagged json:"-", so a preloaded model can be marshalled without the encoder recursing between parent and child. References to child records stay. Tests marshal each of the nine references set, preloaded where it matters, and check the parent's id is absent, so deleting or restoring any one tag fails a test; preloading still fills each reference.

Model: opus-5-5
 2026-10-02 15:43:06 +02:00
clawbot dc9deda173 Write a form-error page's status only after it renders (closes #128)
check / check (push) Successful in 3m33s
Details 
Six form-error paths (the login error and the webhook form's validation and name-taken branches) called WriteHeader before rendering, so if the form page's own template failed, the answer kept its 400 or 409 status with an error body instead of a 500. The new renderTemplateStatus renders into the buffer and writes the status only after the page has rendered; renderTemplate sends 200 through it, and those handlers pass their status to it. No handler calls WriteHeader before a render. Login-form tests show the 400 page still renders in full and a failing template answers 500.

Model: opus-5-5
 2026-10-02 15:34:01 +02:00
clawbot c706199389 Name the CI build step after what the image runs (closes #175)
check / check (push) Successful in 3m48s
Details 
The workflow's build step was named "Build Docker image (runs make check)", but the image has never run make check. It is now named for what the image runs: make fmt-check, golangci-lint, make test and make build. No other step name or comment in the workflow names something it does not run. In the README, the Prerequisites bullet now says Docker is needed for make lint and so for make check, and the sentence about what runs on the host is limited to the steps make check runs.

Model: opus-5-5
 2026-10-02 15:15:08 +02:00
clawbot d52cac1ec6 Toggle only a target's active state, so it cannot undo an edit (closes #431)
check / check (push) Successful in 3m27s
Details 
The target toggle loaded the target, flipped its active flag and saved the whole row, so an edit of the same target's name or settings saved in between was written back over and lost, although it reported success. The toggle now updates only the active column, so it can no longer undo an edit. A test saves an edit just after the toggle has read the target and shows the edit survives and the state flips. The entrypoint toggle is unchanged, since an entrypoint has no edit form.

Model: opus-5-5
 2026-10-02 14:43:00 +02:00
clawbot 0f9b68a0e8 Build the middleware test cookie stores with the production constructor (closes #154)
check / check (push) Successful in 3m30s
Details 
The middleware tests built their cookie stores by setting store.Options by hand, which left the securecookie codecs at the library's 30-day default instead of the 7-day cap production sets, an invisible divergence that would outlive the next change to store construction. The test store constructor moves from internal/session/export_test.go into internal/session/testing.go so other packages can reach it, and the two middleware test helpers build their stores through it. No test in the repo builds a cookie store by hand any more, and no assertion changes.

Model: opus-5-5
 2026-10-02 14:30:51 +02:00
clawbot 8cf5acaf1d Justify the handlers tests' start limit with a measurement (closes #225)
check / check (push) Successful in 3m57s
Details 
The internal/handlers tests were load-fragile because every test app hashed the admin password at 64 MB; that went with the cheaper test hashing already on next, and measuring under the host's real load found nothing left to fix in how the tests run. The comment on newTestApp now says its start limit, fx's default, is there to catch a start that hangs, and that the slowest measured start is far inside it. The header of script/test gives current figures in place of ones from before that change. Neither limit changes, and no test changes.

Model: opus-5-5
 2026-10-02 14:08:54 +02:00
clawbot 9ade217222 Show each webhook's activity in the webhook list (closes #394)
check / check (push) Successful in 3m38s
Details 
Each entry in the webhook list at /hooks now shows when its last event arrived (or "No events yet"), how many of its deliveries failed in the last 24 hours, in red when not zero, and how many entrypoints and targets are inactive, as in "4 targets, 1 inactive". The figures come from the event totals row and the statistics pane's own query, so the list no longer counts every stored event and its event count matches the pane's. A webhook whose event database cannot be read says so in its entry; the rest of the list still shows. A failed read of entrypoints or targets from the main database now returns an error page instead of showing zero.

Model: opus-5-5
 2026-10-02 13:57:43 +02:00
clawbot 5551f75251 Say at the receiver route where its 1 MB body cap lives (closes #173)
check / check (push) Successful in 3m22s
Details 
The receiver route /h/{uuid} has no MaxBodySize middleware, unlike the page route groups; its 1 MB cap is enforced in the handler, which owns the response senders see. Nothing at the route said so, so a reader could take the receiver for uncapped or remove the only bound on the one unauthenticated endpoint. The route registration and the handler's body-reading function now say where the cap lives and why. A routing test sends a body of exactly 1 MB and one a byte over through the production router and pins that only the second is refused, with the handler's 413 and message.

Model: opus-5-5
 2026-10-02 13:40:00 +02:00
clawbot fd036774f9 Name each database target's archive for its webhook and target (closes #376)
check / check (push) Successful in 3m29s
Details 
Each database target now writes its own archive file, archive-WEBHOOKNAME-TARGETNAME-TARGETID.db, named by delivery.ArchiveFileName, in place of one archive per webhook keyed on its UUID. Renaming a webhook or a target renames its archive files (with any -wal and -shm) before the new name is saved, never over an existing file, and moves every one back if a rename or the save fails. The webhook edit, the target edit and target creation share one lock so no two interleave. Deleting a webhook or target leaves its files on disk. Nothing looks for the old archive-WEBHOOKID.db files. The README gives the naming and the recovery steps.

Model: opus-5-5
 2026-10-02 13:28:49 +02:00
clawbot 21aafbf928 Remove the no-op MAINTENANCE_MODE setting (closes #317)
check / check (push) Successful in 3m25s
Details 
MAINTENANCE_MODE did nothing but make the healthcheck JSON report maintenanceMode: true; no request was ever served differently, so an operator who set it expecting requests to be refused got nothing. It is removed from the configuration, the startup configuration log line, the healthcheck JSON, the README and the Settings page, together with the uncalled Server.MaintenanceMode method and the healthcheck's dependency on the configuration. A leftover value in an environment is ignored like any other unknown variable.

Model: opus-5-5
 2026-10-02 13:27:43 +02:00
clawbot c87b469dcd Let a handler flush or set a write deadline through the access log and metrics (closes #191)
check / check (push) Successful in 3m17s
Details 
The access log's response writer and the metrics middleware's writer hid the writer beneath them, so a handler's flush, hijack or write deadline set through http.ResponseController failed with "not supported" behind them. The access log's writer now has Unwrap. The metrics middleware calls the library's public Measure with a writer of our own that has Unwrap, in place of std.Handler's writer, so the middleware order and what metrics record are unchanged. A test over a real connection sets a write deadline and flushes, metrics on and off, on a global route and in an admin page group, and fails without either Unwrap.

Model: opus-5-5
 2026-10-02 13:08:54 +02:00
clawbot b14b27b78b Serve the delivery duration histogram from boot (closes #267)
check / check (push) Successful in 3m25s
Details 
The delivery duration histogram was registered only when a delivery first ran, so until then /metrics had no series for it and a dashboard or alert on it saw nothing at all rather than zero. Each of the four target types now has its series registered at boot with zero counts, so the histogram is served from the first scrape. Tests pin that all four series are present before any delivery, on the metrics registry and through the production router.

Model: opus-5-5
 2026-10-02 12:45:05 +02:00
clawbot 287e47df7f Add a read-only Settings page for the loaded configuration (closes #402)
check / check (push) Successful in 3m19s
Details 
A new page at /settings, linked from the navigation bar and behind the login, lists every configuration field the server loaded at startup: its environment variable, the README table's description, and the value in effect. METRICS_PASSWORD and SENTRY_DSN show only as set or not set; their values are replaced before rendering and never reach the template. The route is GET only and its group is built like the other admin page groups. Tests set the credentials one at a time so each value shown is checked against its own field and a set secret never appears in the page.

Model: opus-5-5
 2026-10-02 12:30:21 +02:00
clawbot 8b5541734e Run script/test quietly with coverage, rerun failed tests verbosely (closes #315)
check / check (push) Successful in 3m20s
Details 
script/test now runs the suite once with -race -cover and no -v. On a failure it reruns only the failed top-level tests, with -v, in the packages that failed, then exits non-zero whatever the rerun shows. A green run stays quiet, and a red one ends with the failing tests' verbose output, which the Docker build's 2 MiB log limit can hold; a verbose rerun of every package would pass that limit again. A failure that names no test (a build error, a timeout) skips the rerun, since the quiet run already prints that package's output. The timeout and parallelism are unchanged.

Model: opus-5-5
 2026-10-02 11:58:43 +02:00
clawbot c513816a55 Refuse [::], 0.0.0.0, IPv6 multicast and documentation space (closes #341)
check / check (push) Successful in 3m20s
Details 
On the build host a connection to [::] reaches a listener on ::1, and one to 0.0.0.0 reaches 127.0.0.1, so a delivery target at either reached this host's loopback past the guard. 0.0.0.0/32 and ::/128 are now in alwaysBlockedNetworks, which no allowlist opens; an allowlist reaches loopback only through an entry covering a loopback address. IPv6 multicast (ff00::/8) and documentation space (2001:db8::/32) are refused by default and reopen when listed.

Every default blocklist entry has a one-line comment, each list is pinned on its own, and tests refuse each address at target creation and at delivery. The README and the rules above each list match.

Model: opus-5-5
 2026-10-02 11:22:30 +02:00
clawbot 2bb4683512 Discard fx's own log in tests that build an fx app (closes #230)
check / check (push) Successful in 3m19s
Details 
Every test that builds an fx app with fxtest.New (handlers, server, resetpw, gormlog, config) now passes fx.NopLogger, so fx's own log no longer goes to t.Logf. A hook still running after a start or stop timeout can then no longer write to a test that has already returned, which the race detector reported as a data race. What the tests assert is unchanged, and nothing about the race detector is suppressed.

Model: opus-5-5
 2026-10-02 11:08:38 +02:00
clawbot 5b1d283d06 Say what each action did in a one-line notice (closes #383)
check / check (push) Successful in 3m20s
Details 
Saving, deleting, activating or deactivating a webhook, entrypoint or target, and signing out, now land on their page with a one-line notice such as "Webhook deleted." or "Signed out.". The redirect carries a fixed code that maps to fixed text; an unknown code shows nothing, so nothing from the URL is ever echoed. One partial in the page layout shows the notice on every page, and replay and resubmit now use the same codes and partial. Error pages show no notice.

Model: opus-5-5
 2026-10-02 10:30:31 +02:00
clawbot b78abdc9da Drop TODO.md's stale docs-only cache caveat (closes #421)
check / check (push) Successful in 3m11s
Details 
Since the build context carries .git and the CI fingerprint is the hash of the commit being checked, every commit's check runs the build, docs-only commits included. TODO.md's caveat that a docs-only commit replays from the layer cache was no longer true, and the README's "CI gate honesty" section already says how a check runs, so the paragraph is removed.

Model: opus-5-5
 2026-10-02 10:14:40 +02:00
clawbot c23ffbac65 Widen the webhook page by half so an entrypoint URL fits on one line (closes #350)
check / check (push) Successful in 3m16s
Details 
The webhook page's maximum width goes from 72rem (1152 px) to 108rem (1728 px), half again as wide, so an entrypoint URL stays on one line in 1920- and 1440-pixel windows; the statistics pane and both columns widen with it. The title row now wraps, so a phone-width window no longer scrolls sideways.

The width is an inline style: static/css/style.css is linked by no page, and the committed Tailwind stylesheet has no class that wide. The webhook list, the event log, the navbar and the footer stay at 72rem.

Model: opus-5-5
 2026-10-02 09:49:33 +02:00
clawbot 2ac4d4d793 Send the build version in the outbound User-Agent (closes #313)
check / check (push) Successful in 3m25s
Details 
The http and slack targets sent the constant User-Agent webhooker/1.0. They now send webhooker/ followed by the version the build stamped, the same value the footer shows, built in one place on the delivery engine. User-Agent stays a reserved header and is still set after the target's configured headers, so a configured one cannot override it. Tests check the header each target sends against a known version, and that a configured User-Agent is replaced.

Model: opus-5-5
 2026-10-02 09:48:34 +02:00
clawbot eb4c4cc849 Serve /metrics from a registry of its own (closes #227)
check / check (push) Successful in 3m25s
Details 
A second metrics-enabled router in one process panicked on a duplicate collector registration, because every collector registered on Prometheus's global default registry. metrics.NewRegistry now builds one registry with the Go runtime and process collectors; fx provides it and the delivery metric set built on it. The middleware builds its HTTP recorder once on that registry (NewForTest on a fresh one), the engine and handlers take the metric set from fx, and nothing registers on the global default any more.

/metrics is served from the new registry with the same series names, labels and auth. A test builds two metrics-enabled routers in one process.

Model: opus-5-5
 2026-10-02 09:06:20 +02:00
clawbot cb7bafab17 Derive the image's version from the .git in the build context (closes #366)
check / check (push) Successful in 3m23s
Details 
upaas uploads its clone as a tar context, which .dockerignore does not filter, so its builds already carried .git; the unknown default of the VERSION build arg is what stamped them. The image now derives its version itself: ARG VERSION has no default, so make build falls back to script/version, which runs git describe on the copied .git; a VERSION build arg still wins.

.dockerignore sends .git without its config in a directory context and no longer leaves out tracked files, which would mark the tree -dirty. The builder installs git, trusts /build as a safe.directory, and fails when .git is present but the version comes out unknown. A shallow single-branch clone stamps its short commit.

Model: opus-5-5
 2026-10-02 08:41:20 +02:00
clawbot 2416528b77 Render admin page errors in the normal layout (closes #382)
check / check (push) Successful in 3m12s
Details 
Every 400, 403, 404 and 500 on an admin page now answers with an error page in the normal layout: the status, one fixed line explaining it, and a link back to the webhook list, or to sign-in when nobody is signed in. Unknown paths reach it through the router's not-found handler, a refused form token through the CSRF middleware, and a panic through a recoverer each admin page route group installs first. Status codes are unchanged, and the page always sends Cache-Control: no-store.

If the error page fails to render, the answer is the same status in plain text; if it panics, the answer is a 500. The receiver, the healthcheck and /metrics keep their plain answers.

Model: opus-5-5
 2026-10-02 08:15:12 +02:00
clawbot 38157d8936 Say how to allow a refused private target address (closes #398)
check / check (push) Successful in 3m16s
Details 
Refusing an http or slack target whose address is private or reserved, on add or edit, now adds one sentence: such addresses are refused by default, and the server's ALLOWED_EGRESS_CIDRS setting allows named networks, with a pointer to the README section. It suggests no value, so it never points at allowing everything.

Metadata refusals get no such sentence. To tell them apart, the default blocklist's public addresses now have their own list, blockedPublicNetworks, still checked after the allowlist; a test pins which addresses each list refuses and how listing opens them, unchanged from before.

Model: opus-5-5
 2026-10-02 07:52:50 +02:00
clawbot 7d360babed Logging in returns to the page that was asked for (closes #384)
check / check (push) Successful in 3m44s
Details 
A logged-out GET of an admin page now redirects to /pages/login with its path and query in a next parameter, when they fit in 2048 bytes. The login form carries next as a hidden field; a successful login redirects there, and a failed one shows the page again with the same next. A POST still redirects to plain /pages/login.

The value is client-chosen, so every read of it goes through one check: after percent-decoding it must start with exactly one / and contain no backslash or control character; anything else becomes /. gosec's open-redirect rule is suppressed on the two redirects that follow it. The login page's navigation bar no longer links to itself.

Model: opus-5-5
 2026-10-02 06:57:51 +02:00
clawbot 803a94be37 Add a statistics pane to the webhook page (closes #368)
check / check (push) Successful in 3m56s
Details 
The webhook page opens with a statistics pane: entrypoints and targets, deliveries in progress, the last arrival, the retention period; events, deliveries and failures, lifetime and within retention; and events, failures and failure percentage over 10 minutes and 24 hours.

Running totals, one row per target plus one for events, sit in each webhook's event database and are written in the same transaction as the rows they count; retention prunes in paused, stoppable batches and subtracts what it removes. Window figures are index-range counts on a new finished_at column. An existing event database must be recreated.

Model: opus-5-5
 2026-10-02 06:41:00 +02:00
clawbot 43ed8d4834 Move webhook pages to /hook/ID and inbound URLs to /h/UUID (closes #367)
check / check (push) Successful in 3m10s
Details 
The webhook page and everything under it move from /source/ID to /hook/ID; the list and new-webhook form from /sources to /hooks and /hooks/new; the event log from .../logs to /hook/ID/events; entrypoint URLs, the ones senders post to, from /webhook/UUID to /h/UUID. The old paths are simply gone, per the owner's ruling. Every link, redirect, form action, test, comment and README mention follows; every link to the event log, and its title and heading, read "Full Event Log".

Keeping the UUID out of logs, metrics labels and error reports, and the receiver rate limits, key on the route itself, so they cover /h/ unchanged. Routed tests follow every link and form the pages render to a moved page.

Model: opus-5-5
 2026-10-02 06:32:14 +02:00
clawbot 1c721ede41 Run the test suite without -v so a failure shows in the build log (next side of #414)
check / check (push) Successful in 3m23s
Details 
script/test now runs go test without -v. The Docker build cuts each step's log at 2 MiB, and the verbose output of the whole suite passed that limit before any failure was printed, so a red script/cibuild showed only passing packages. Without -v, go test prints one result line per package and, for a package that fails, everything its tests wrote, the application's log lines included. -race, -p 4 -parallel 8 and the 90-second per-package timeout are unchanged.

The cause of the red builds was fixed by the cheaper test hashing already on next; the same two changes go to main separately.

Model: opus-5-5
 2026-10-02 06:31:53 +02:00
clawbot bfdbc937c6 Keep make test under 2 GB of memory (closes #344)
check / check (push) Successful in 3m41s
Details 
Every test that starts a database seeds the admin account, hashing its password with Argon2id at 64 MB, about 150 MB under -race, and internal/handlers and internal/database ran dozens of those at once. That was the memory, and most of internal/handlers' run time; the product code does not leak.

HashPassword now hashes at a 1 MB cost only when testing.Testing() reports a test binary, so every test package gets it with nothing to add and a binary built by go build always hashes at the shipped parameters. TestHashPassword_ShippedParameters still hashes and verifies through HashPassword at the shipped cost. script/test adds -p 4 -parallel 8.

Model: opus-5-5
 2026-10-02 03:02:28 +02:00
clawbot 1cafaeb953 Reword the build-architecture history line in TODO.md (closes #412)
check / check (push) Successful in 5s
Details 
The architecture is read at run time, so nothing passes it in at build time. The dated history entry for issue 31 in TODO.md now says a build-architecture global was removed, without naming it, so the word no longer appears in the tree.

Model: opus-5-5
 2026-10-02 01:08:22 +02:00
clawbot 515c359e56 Trust the RFC 1918 ranges as proxies when TRUSTED_PROXIES is unset (closes #333)
check / check (push) Successful in 4m43s
Details 
Unset or empty, TRUSTED_PROXIES now defaults to 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16, so a reverse proxy reaching webhooker from one of those ranges gets per-client rate-limit buckets with nothing set. A set value replaces the default entirely; an unparseable one still fails startup. The startup warning for an empty list is gone.

The README gives the default, one rule (if any client can reach webhooker, or the proxy in front of it, from an RFC 1918 source address, set the list to the proxy's address alone), and where to find that address: the remoteIP field of the http request log line. Loopback is not in the default.

Model: opus-5-5
 2026-10-02 00:41:36 +02:00
clawbot 30e65dce53 State what the default blocklist covers (closes #244)
check / check (push) Successful in 5m9s
Details 
The README's egress section and the comment above blockedNetworks now state what the default blocklist covers: the IPv4 private and reserved ranges, IPv6 loopback, unique local and link-local addresses, and certain public addresses, each added only because it hands credentials, user data or bootstrap material to whatever can reach it without the caller presenting anything. That is the same material as the rule above alwaysBlockedNetworks, so a future candidate can be accepted or refused against one rule.

A provider's other public addresses, such as 161.26.0.0/16 and 166.8.0.0/14, are not refused. Docs and a comment only; the lists are unchanged.

Model: opus-5-5
 2026-10-02 00:21:35 +02:00
clawbot 1ac4fe0be4 Show 50 recent events with size, processing time and status (closes #347)
check / check (push) Successful in 4m32s
Details 
The webhook page's list, now headed "50 Most Recent Events", shows the 50 newest events, limited in the query. Each row adds the time received (relative, UTC on hover), the body size, the processing time (slowest delivery, queued to final outcome, retries included, "in progress" while pending), and, when the webhook has exactly one HTTP target, that target's latest HTTP status, coloured by class.

Each event now records its body size in a new body_bytes column when it is stored, so the list never reads bodies; the list loads only the delivery and attempt columns it shows, and a failed load is an error, not an empty list.

An existing event database must be recreated: the column is added in place, with no migration.

Model: opus-5-5
 2026-10-01 23:50:00 +02:00
clawbot a56f1fe0c8 Remove the fixed Account Type row from the profile page (closes #401)
check / check (push) Successful in 4m38s
Details 
The profile page no longer shows the fixed "Account Type: Standard User" row. webhooker has one account, the administrator it creates, and no account types, so the row was untrue. The Account Information section now lists the username alone, and a test checks the row stays gone.

Model: opus-5-5
 2026-10-01 23:24:41 +02:00
clawbot 9d29baaa2d Commit the Alpine.js tarball in 3p/ and extract it at build time (closes #345)
check / check (push) Successful in 6m21s
Details 
The build no longer downloads Alpine.js. Its npm package tarball is committed as 3p/alpinejs-3.14.9.tgz, byte for byte the file script/fetch-assets downloaded, with the sha256 that script pinned. script/assets (make assets) extracts package/dist/cdn.min.js to the ignored static/js/alpine.min.js; script/test runs it, so make test, make check, the pre-commit hook and the Dockerfile get the file with no network access, and make build, run and dev run it too.

Removed: script/fetch-assets, its Dockerfile step, static/vendor.sha256 and static/vendor_test.go. The README describes the new flow.

Model: opus-5-5
 2026-10-01 22:44:41 +02:00
clawbot 507980a347 Bound username length at creation (closes #184)
check / check (push) Successful in 4m8s
Details 
Usernames are limited to 1024 bytes, so no account can exist that is unable to log in. The username rides in the session cookie, which browsers and securecookie refuse past about 4 KB, leaving room for roughly 2000 bytes of username; the limit is about half that.

User.BeforeSave returns ErrUsernameTooLong when a whole User is created or saved. A byte-counting check constraint on users.username catches every other write, including a column update. The limit appears in the constant and in the struct tag; a test fails if they disagree.

Model: opus-5-5
 2026-10-01 21:38:48 +02:00
sneak f703b72ce0 Next (#364)
check / check (push) Successful in 3m59s
Details 
Reviewed-on: #364
 2026-09-29 13:05:57 +02:00
clawbot b79e4649a1 Container sets its data directory's owner and mode itself (#353)
check / check (push) Canceled after 0s
Details 
Closes #340.

The image no longer sets `USER`. Its new `ENTRYPOINT`, `deploy/docker-entrypoint.sh`, starts as root, creates `DATA_DIR` if missing, gives the directory and anything in it owned by another user to `webhooker` (UID 1000), sets the directory to `0750`, and runs the command as `webhooker` through `su-exec`. An empty root-owned bind mount, or data left by another UID, now works as mounted; the app never runs as root and is still PID 1. `CMD` is still `/app/webhooker`, so the `resetpw` commands are unchanged. Started with `--user`, the script only runs the command. It is in `/usr/local/bin`, not `/app`, which belongs to `webhooker`.

README: the UID 1000 ownership block, the upaas pre-deploy commands and the restore ownership step are gone; the upaas volume bullet names only the path.

- Judgement call: `su-exec` over `setpriv`: Alpine's small tool for this, needing only musl; busybox's `setpriv` cannot change user, and util-linux's adds `libcap-ng`.
- Deviation: `su-exec` is pinned by version (`0.2-r3`), not by hash; `ca-certificates` beside it is unpinned.
- Judgement call: each start reads every entry's owner but changes only entries owned by someone else.
- `docker exec` and the health check now run as root, since the image sets no `USER`.
- No automated test covers the script: the suite runs inside `docker build`, which cannot start a container.
- A missing host directory under upaas is sneak/upaas#235.

Model: opus-5-5
Reviewed-on: #353
Co-authored-by: clawbot <35+clawbot@noreply.example.org>
 2026-09-29 13:05:16 +02:00
clawbot 1428154bbd Let a browser with cookies from an earlier database log in (closes #359)
check / check (push) Canceled after 0s
Details 
A new database brings a new session key. A browser still holding the
old session cookie got a 500 on a correct login: Session.Get returned
the cookie's decode error and the login handler answered it with a
500. Get now treats a cookie that does not decode as absent, and
logging in replaces it. gorilla/csrf already did the same for the
CSRF cookie.

A start that creates webhooker.db now logs "created a new, empty
database" at WARN with its path, shortly before the first-boot banner,
so an unexpectedly empty DATA_DIR is noticed.

The codec tests now decode through the store, since Get no longer
reports the codec's reason.

Model: opus-5-5
 2026-09-29 12:58:44 +02:00
sneak 8ad2a86e4b Sneak/testdeploy (#356)
check / check (push) Successful in 11s
Details 
Reviewed-on: #356
 2026-09-29 12:01:33 +02:00
sneak a891b726e5 Milestone: next into main (#342)
check / check (push) Successful in 9s
Details 
Reviewed-on: #342
 2026-09-29 11:53:19 +02:00
clawbot ab63b5f777 Restrict /s/* to GET and HEAD (closes #169)
check / check (push) Successful in 3m37s
Details 
The static file server was attached with Mount, which registers every
method, so POST, PUT and DELETE on an asset were answered 200 with the
file. It is now registered for GET and HEAD only, inside a /s group
whose method-not-allowed handler answers 405 with Allow: GET, HEAD.
A method chi does not route at all, such as PROPFIND, still gets 405
from the top-level router, without Allow. The inverted test and the
README route table say the same.

Model: opus-5-5
 2026-09-29 11:10:27 +02:00
84 changed files with 2868 additions and 2510 deletions
Expand all files Collapse all files
Dockerfile
+8 -13
View File
@@ -111,16 +111,11 @@ RUN --network=none golangci-lint run --config .golangci.yml --build-tags browser
# Nothing is wanted from js-lint; the copy is what makes this phase run it.
COPY --from=js-lint /src/yarn.lock /dev/null
# Test phase. The tests run with cgo off and so without Go's race detector,
# which needs cgo on Linux; sneak ruled the race detector out in
# https://git.eeqj.de/sneak/webhooker/issues/538#issuecomment-153273.
# `make test` (script/test) builds this stage alone; the build stage below
# depends on it.
# Test phase. -race needs cgo and so a C compiler, which the Debian Go image
# ships and the alpine one does not. `make test` (script/test) builds this
# stage alone; the build stage below depends on it.
#
# golang:1.26.1-bookworm (Debian-based), 2026-03-17: the builder stage's
# image, which that stage needs for its git and make. Its test step runs
# under the image's bash, and the tests in internal/versionscript use its
# git, make and sh, and skip when they are missing.
# golang:1.26.1-bookworm (Debian-based), 2026-03-17
FROM golang:1.26.1-bookworm@sha256:4465644228bc2857a954b092167e12aa59c006a3492282a6c820bf4755fd64a4 AS test
WORKDIR /src
@@ -136,8 +131,8 @@ RUN script/assets
# -timeout applies to each package on its own, so 90s has only to clear the
# slowest one. -p 4 -parallel 8 keep the run under 2 GB of memory: at most
# four test binaries build or run at once, each with at most eight parallel
# tests. The defaults are one per core, so without these the memory the run
# needs grows with the host's core count.
# tests. Under -race every test binary and every link costs a few hundred MB,
# so the defaults (one per core) add up to several GB on a many-core host.
#
# The first run has no -v: go test then prints one result line per package,
# with its coverage, and for a package that fails, everything its tests
@@ -159,12 +154,12 @@ RUN script/assets
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
RUN --mount=type=tmpfs,target=/tmp/tests,size=512m \
export TMPDIR=/tmp/tests GOTMPDIR=/tmp; \
CGO_ENABLED=0 go test -cover -p 4 -parallel 8 -timeout 90s ./... 2>&1 | tee /tmp/go-test.log && exit 0; \
go test -race -cover -p 4 -parallel 8 -timeout 90s ./... 2>&1 | tee /tmp/go-test.log && exit 0; \
tests="$(awk '/^--- FAIL: / { print $3 }' /tmp/go-test.log | paste -s -d '|' -)"; \
packages="$(awk '/^FAIL\t/ { print $2 }' /tmp/go-test.log)"; \
if [ -n "$tests" ]; then \
echo "--- Rerunning the failed tests with -v for details ---"; \
CGO_ENABLED=0 go test -v -p 4 -parallel 8 -timeout 90s -run "^($tests)\$" $packages; \
go test -race -v -p 4 -parallel 8 -timeout 90s -run "^($tests)\$" $packages; \
fi; \
exit 1
Dockerfile.browser
+3 -3
View File
@@ -15,9 +15,9 @@ RUN go mod download
COPY . .
# The test binary embeds the templates and static files, so the browser
# stage needs nothing else. It is built with cgo off and -p 4 keeps the
# compile's memory down, both as in the test phase of Dockerfile.
RUN make assets && CGO_ENABLED=0 go test -c -p 4 -tags browser -o /browser.test ./internal/server
# stage needs nothing else. -p 4 keeps the compile's memory down, as in
# the test phase of Dockerfile.
RUN make assets && go test -c -p 4 -tags browser -o /browser.test ./internal/server
# chromedp/headless-shell:151.0.7922.109 (Debian trixie), 2026-08-11. The
# browser is on PATH as headless-shell, where the test's browser library
README.md
+61 -80
View File
@@ -60,7 +60,7 @@ make assets # Extract Alpine.js from 3p/ (build and dev run it)
make fmt # Format Go (gofmt + goimports) and Markdown (prettier, in Docker)
make fmt-check # Fail if gofmt or prettier would change anything (writes nothing)
make lint # Run golangci-lint and ESLint in Docker
make test # Run tests with cgo off, in Docker
make test # Run tests with race detection, in Docker
make test-browser # Run the browser test in Docker (Dockerfile.browser)
make check # test + lint + fmt-check + css-check (CI gate)
make build # Build binary to bin/webhooker (version-stamped)
@@ -75,11 +75,6 @@ make css-check # Fail if static/css/tailwind.css is stale (writes nothing)
make clean # Remove bin/
```
The tests run with cgo off and so without Go's race detector, which needs cgo on
Linux: an exception to the `-race` in the test phase of the vendored
`REPO_POLICIES.md`, ruled by @sneak in
https://git.eeqj.de/sneak/webhooker/issues/538#issuecomment-153273.
### Configuration
All configuration is via environment variables. For local development, you can
@@ -1585,18 +1580,13 @@ A delivery destination for events. Each target defines where and how events
should be forwarded.
| Field | Type | Description |
| ------------------ | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| -------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `id` | UUID | Primary key |
| `webhook_id` | UUID | Foreign key → Webhook |
| `name` | string | Human-readable name |
| `type` | TargetType | One of: `http`, `slack`, `database`, `log` |
| `active` | boolean | Whether deliveries are enabled (default: true) |
| `url` | string | Where an `http` or `slack` target sends each event. It is a credential: the target list shows only its scheme and host, and only the target edit form shows it whole |
| `headers` | JSON text | An `http` target's extra request headers, a JSON object of name to value |
| `timeout_seconds` | integer | An `http` target's per-request timeout; 0 uses the shared 30-second client timeout |
| `forward_query` | boolean | Whether an `http` target is sent the query string each event arrived with |
| `archive_expiry` | integer | How long a `database` target keeps each archived event, as a Go `time.Duration` (nanoseconds); 0 keeps them forever |
| `archive_rotation` | string | How often a `database` target starts a new archive file: `none`, `monthly`, `daily` or `hourly` |
| `config` | JSON text | Type-specific configuration |
| `max_attempts` | integer | How many times an `http` or `slack` target attempts each delivery in all: at least 1, and 1 by default. Ignored by `database` and `log` targets. The web UI labels it Delivery attempts |
**Relations:** Belongs to Webhook. Has many Deliveries.
@@ -1608,10 +1598,10 @@ should be forwarded.
failed delivery with exponential backoff, up to that many attempts in all,
behind a per-target circuit breaker.
- **`slack`** — Post the event as a formatted message to a Slack-compatible
incoming webhook URL, its `url`. It is built on the same HTTP core as `http`
and honours `max_attempts` identically, circuit breaker included. See the
Slack target section under "Per-Webhook Event Databases" for the message
format.
incoming webhook URL (`webhookUrl` in `config`). It is built on the same HTTP
core as `http` and honours `max_attempts` identically, circuit breaker
included. See the Slack target section under "Per-Webhook Event Databases" for
the message format.
- **`database`** — Archive the full event as a row into the target's own archive
database (`archive-{webhook_name}-{target_name}-{target_uuid}.db`) for
long-term retention, with an optional creation-validated expiry (default: keep
@@ -1627,28 +1617,29 @@ should be forwarded.
- **`log`** — Write the event to the application log (stdout). Useful for
debugging.
Each type uses only its own settings, and the ones it does not use stay empty:
`http` uses `url`, `headers`, `timeout_seconds` and `forward_query`; `slack`
uses `url`; `database` uses `archive_expiry` and `archive_rotation`; `log` uses
none. The add target form, the new webhook form and the target edit form check
every setting before it is saved, and each setting is a column of its own, so
nothing that reads a target parses its settings or has to handle one it cannot
read.
The `config` field stores type-specific configuration as JSON (e.g., destination
URL, custom headers, timeout settings).
An `http` target POSTs the event to its `url`. Its `headers` are applied last,
so they win over the event's own forwarded headers.
**`http` target configuration:**
`forward_query` is off by default, and the target URL is then sent exactly as
| Key | Type | Description |
| -------------- | ------------- | ----------------------------------------------------------------------------------------- |
| `url` | string | Destination the event is POSTed to |
| `headers` | object | Extra request headers, applied last so they win over the event's own forwarded headers |
| `timeout` | integer (sec) | Per-target request timeout; unset (or 0) uses the shared 30-second client timeout |
| `forwardQuery` | boolean | Pass the query string each event arrived with on to the target; unset (or false) does not |
`forwardQuery` is off by default, and the target URL is then sent exactly as
configured. On, each delivery appends the event's query string to the target
URL, joined with `&` when the URL already has a query string of its own; a
replayed delivery and a resubmitted event's deliveries do the same. Both target
forms offer it as "Pass the query string on to this target", and the target list
shows it when it is on.
`timeout_seconds` is capped at **300 seconds**, and the form rejects anything
above it rather than substituting the cap. A delivery attempt holds one of the
bounded pool's workers for its whole duration, so an unbounded timeout would let
a single unresponsive destination stall the queue.
`timeout` is capped at **300 seconds**, and the form rejects anything above it
rather than substituting the cap. A delivery attempt holds one of the bounded
pool's workers for its whole duration, so an unbounded timeout would let a
single unresponsive destination stall the queue.
`headers` rejects the names the delivery path or `net/http` writes regardless of
what is configured: `Host`, `Content-Length`, `Transfer-Encoding`, `Connection`,
@@ -1722,7 +1713,7 @@ The event log and the event's own page show the query string with the rest of
the request. The event log leaves out one larger than 32 KiB, as it does request
headers, and links to the event's page, which shows it whole. The `database` and
`log` targets carry it with the rest of the event. An `http` target receives it
only when its `forward_query` setting is on.
only when its `forwardQuery` setting is on.
#### Delivery
@@ -2001,10 +1992,11 @@ target UUID keeps the file name unique. A webhook named `Orders (EU)` with a
target named `Long-term archive` archives into
`archive-orders-eu-long-term-archive-{target_uuid}.db`.
A target's `archive_rotation` is `none`, the default, which keeps the one file,
or `monthly`, `daily` or `hourly`. A target that rotates writes each event to a
file named for the period of the event's receive time, in UTC, put before the
`.db`: `archive-orders-eu-long-term-archive-{target_uuid}-2026-10.db` monthly,
An optional `rotation` in the target's config JSON (e.g. `{"rotation":"daily"}`)
is `none`, the default, which keeps the one file, or `monthly`, `daily` or
`hourly`. A target that rotates writes each event to a file named for the period
of the event's receive time, in UTC, put before the `.db`:
`archive-orders-eu-long-term-archive-{target_uuid}-2026-10.db` monthly,
`…-2026-10-01.db` daily and `…-2026-10-01-19.db` hourly. Each file holds exactly
its period's events, and the first event of a new period starts the next file,
so a finished period's file can be moved away like any archive. A changed
@@ -2039,11 +2031,12 @@ archiving without stopping the service; a moved or removed archive file is
recreated automatically on the next write, from the SQL files in
`internal/db/migrations/archive/`. A zero-length archive file is written to as a
new archive: SQLite opens it as an empty database, so it holds nothing to lose.
A target's `archive_expiry` is checked by the form that saves it — the default,
`never` on the forms and 0 in the column, keeps rows forever — and rows older
than the expiry are pruned each time the archive is (re)opened. An archive write
failure is never silent success: the delivery records a failed attempt with the
error and is marked failed.
An optional `expiry` in the target's config JSON (e.g. `{"expiry":"720h"}`) is
validated when the target is created — the default (unset or the literal
`never`) keeps rows forever — and rows older than the expiry are pruned each
time the archive is (re)opened. An archive write failure is never silent
success: the delivery records a failed attempt with the error and is marked
failed.
Because reopens only happen on writes, an archive whose target has stopped
receiving events would never be pruned. A background **archive sweeper** closes
@@ -2054,7 +2047,8 @@ traffic. The sweep never creates an archive — a target whose archive file does
not yet exist is skipped, not initialised — it takes the same per-target lock
the write path uses, so it can never interleave with a write, and it leaves the
archive closed afterwards so the move-the-file-away workflow keeps working.
Archives with the expiry `never` are not touched by the sweep at all.
Archives with no expiry, or the expiry `never`, are not touched by the sweep at
all.
For a target with several files, the write path prunes only the file it writes
to, and the sweep prunes every one of them, taking the target's lock for one
@@ -2122,8 +2116,9 @@ compatible services). Each message includes event metadata (HTTP method, content
type, timestamp, body size) and the payload pretty-printed in a code block. JSON
payloads are automatically formatted with indentation for readability; non-JSON
payloads are shown as raw text. Large payloads are truncated to keep messages
reasonable. The target's `url` is the Slack/Mattermost incoming webhook
endpoint.
reasonable. Config stores `webhookUrl` — the Slack/Mattermost incoming webhook
endpoint. That is the JSON key; the error text for a missing one reads
`webhook_url is required`, which is the message, not the key.
The database uses the
[modernc.org/sqlite](https://pkg.go.dev/modernc.org/sqlite) driver.
@@ -2158,8 +2153,12 @@ External Service
1. Look up Entrypoint by UUID — 404 if unknown,
410 if inactive
2. Read the body under the 1 MB cap
3. Hand the full request, as an Event, to
Engine.StoreEvent
3. Capture full request as Event
4. Create Delivery records for each active Target
5. Build self-contained delivery.Task structs
(target config + event data inline for
bodies < 16 KiB)
6. Notify Engine via channel (no DB read needed)
│
▼
┌──────────────┐
@@ -2168,13 +2167,6 @@ External Service
│ (worker │
│ pool) │
└──────┬───────┘
4. Store the Event, a pending Delivery for each
active Target, and the totals, in one
transaction
5. Build a delivery.Task for each Delivery
(target settings + event data, the body
inline when it is under 16 KiB)
6. Queue the tasks on the delivery channel
│
┌── bounded worker pool (N workers) ──┐
▼ ▼ ▼
@@ -2188,12 +2180,6 @@ External Service
└────────────┘
```
A **Resubmit** stores its copy of an event through the same `Engine.StoreEvent`,
and a **Replay** adds one delivery of a stored event through
`Engine.CreateReplay`, which writes the delivery and its target's totals in one
transaction and queues it. Those two methods are the only code that creates a
delivery; the handlers read and check the request, then make the one call.
### Bounded Worker Pool
The delivery engine uses a **fixed-size worker pool** (default: 10 workers) to
@@ -2203,9 +2189,8 @@ preventing goroutine explosions regardless of queue depth.
**Architecture:**
- **Channels as queues:** Two buffered channels serve as bounded queues: a
delivery channel (the tasks of newly stored deliveries and of the `pending`
deliveries the recovery routine hands back) and a retry channel (tasks from
backoff timers). Both are buffered to 10,000.
delivery channel (new tasks from the webhook handler) and a retry channel
(tasks from backoff timers). Both are buffered to 10,000.
- **Fan-out via channel, not goroutines:** When an event arrives with multiple
targets, each `delivery.Task` is sent to the delivery channel. Workers pick
them up and process them — no goroutine-per-target.
@@ -3069,21 +3054,20 @@ webhooker/
│ │ └── logfield.go # Encoded-byte budget for client-supplied log values
│ ├── delivery/
│ │ ├── engine.go # Event-driven delivery engine (channel + timer based)
│ │ ├── store.go # Storing an event with its deliveries, and a replay, then queueing them
│ │ ├── circuit_breaker.go # Per-target circuit breaker for http/slack targets with retries
│ │ ├── target.go # Target interface, Task, Scheduler
│ │ ├── target_http.go # HTTP target (retries, circuit breaker)
│ │ ├── target_slack.go # Slack/Mattermost incoming-webhook target
│ │ ├── target_database.go # Database archive target, writing through internal/archive
│ │ ├── target_log.go # Log target (stdout)
│ │ ├── target_config_view.go # Masked target settings for templates
│ │ ├── target_config_view.go # Masked target config for templates
│ │ ├── queue_depth.go # Periodic sampler behind the queue-depth gauges
│ │ ├── url_mask.go # Strips credentials from *url.Error
│ │ └── ssrf.go # SSRF prevention (IP validation, safe HTTP transport)
│ ├── handlers/
│ │ ├── handlers.go # Base handler struct, JSON helpers, template rendering
│ │ ├── auth.go # Login, logout handlers
│ │ ├── delivery_replay.go # Per-delivery replay: new delivery, current target settings
│ │ ├── delivery_replay.go # Per-delivery replay: new delivery, current target config
│ │ ├── event_resubmit.go # Event resubmit: new event, all currently active targets
│ │ ├── entrypoint_view.go # Masked entrypoint view for templates
│ │ ├── event_log_view.go # Event log projection, byte-capped in SQL
@@ -3098,7 +3082,7 @@ webhooker/
│ │ ├── webhook_delete.go # Webhook delete, event database and archive writer removal
│ │ ├── event_log.go # Event log page: loaders, filters, delivery views
│ │ ├── entrypoint.go # Entrypoint create, edit, delete and toggle
│ │ ├── target_create.go # Target create, each type's settings from the form
│ │ ├── target_create.go # Target create, per-type config builders
│ │ ├── target_delete.go # Target delete
│ │ ├── target_toggle.go # Target toggle
│ │ ├── shared.go # Helpers shared by several handlers
@@ -3166,26 +3150,23 @@ Components are wired via Uber fx in this order:
10. `metrics.NewRegistry` — The registry `/metrics` serves
11. `metrics.New` — The delivery collectors, registered on that registry
12. `middleware.New` — HTTP middleware
13. `delivery.New` — Event-driven delivery engine, which also stores each event
with its deliveries and creates replay deliveries
13. `delivery.New` — Event-driven delivery engine
14. `archive.NewWriters` — The database targets' archive writers, and the lock
every rename of their files runs under
15. `archive.NewSweeper` — Periodic pruning of idle archives
16. `archive.Writers` → `archive.Archives` — interface bridge so deleting or
16. `delivery.Engine` → `delivery.Notifier` — interface bridge
17. `archive.Writers` → `archive.Archives` — interface bridge so deleting or
renaming a webhook or target reaches its archive files
17. `delivery.Engine` → `delivery.CircuitBreakers` — interface bridge so the
pages can show a target whose deliveries are paused
18. `server.New` — HTTP server and router
The server starts via
`fx.Invoke(func(*server.Server, *delivery.Engine, *database.RetentionReaper, *archive.Sweeper) {})`,
which triggers the fx lifecycle hooks in dependency order. The handlers take
`*delivery.Engine` itself: the receiver and resubmit handlers store an event
through `StoreEvent`, and the replay handler adds a delivery through
`CreateReplay`. Each task the engine queues carries the target's settings and
the event's data, and the body too when it is under 16 KiB
(`delivery.MaxInlineBodySize`); a larger body is read from the event row when
the attempt runs.
which triggers the fx lifecycle hooks in dependency order. The
`delivery.Notifier` interface allows the webhook handler to send self-contained
`delivery.Task` slices to the engine without a direct package dependency. Each
task carries all target config and event data inline (for bodies under 16 KiB,
`delivery.MaxInlineBodySize`), so the engine can deliver without reading from
any database — it only writes to record results.
### Middleware Stack
@@ -3485,8 +3466,8 @@ fixed independently of the compiler's:
`golangci-lint config verify` and `golangci-lint run`, both with
`--network=none`, and depends on `js-lint` (it copies a file from it).
`make lint` builds this stage alone.
5. **Test phase** (`test`, `golang:1.26.1-bookworm`, the builder stage's image)
— extracts Alpine.js, then runs `go test -cover` with cgo off, at most four
5. **Test phase** (`test`, `golang:1.26.1-bookworm`, whose C compiler `-race`
needs) — extracts Alpine.js, then runs `go test -race -cover` at most four
packages and eight tests at a time, with a 90-second timeout per package. On
a failure it runs only the failed tests again with `-v`, since verbose output
from the whole suite would pass the 2 MiB at which the Docker build cuts off
cmd/webhooker/main.go
+3
View File
@@ -211,6 +211,9 @@ func newApp() *fx.App {
delivery.New,
archive.NewWriters,
archive.NewSweeper,
// Wire *delivery.Engine as delivery.Notifier so the
// webhook handler can notify the engine of new deliveries.
func(e *delivery.Engine) delivery.Notifier { return e },
// Wire *archive.Writers as archive.Archives so deleting
// or renaming a webhook or target reaches its archive
// files.
internal/archive/archive.go
+15 -7
View File
@@ -143,15 +143,26 @@ func archiveNamePart(name string) string {
}
// Write writes the full event of a delivery to a database target as
// a row into the target's archive database, honouring the target's
// ArchiveExpiry and ArchiveRotation. With rotation, the event goes to
// the file for the period of its receive time.
// a row into the target's archive database, honouring the optional
// per-target expiry and rotation parsed from the target config JSON.
// With rotation, the event goes to the file for the period of its
// receive time.
func (ws *Writers) Write(d *database.Delivery) error {
webhookID := d.Event.WebhookID
if webhookID == "" {
return errArchiveMissingWebhookID
}
expiry, err := ParseExpiry(d.Target.Config)
if err != nil {
return err
}
rotation, err := ParseRotation(d.Target.Config)
if err != nil {
return err
}
// An event whose stored row was gone before its delivery ran has
// no receive time (see delivery's Engine.hydrateEvent), and goes
// to the file for now.
@@ -176,10 +187,7 @@ func (ws *Writers) Write(d *database.Delivery) error {
ContentType: d.Event.ContentType,
}
return w.write(
row, d.Target.ArchiveExpiry,
archivePeriod(d.Target.ArchiveRotation, receivedAt),
)
return w.write(row, expiry, archivePeriod(rotation, receivedAt))
}
// Rename implements Archives. It renames every one of a database
internal/archive/archive_test.go
+34 -18
View File
@@ -125,8 +125,7 @@ func databaseTargetDelivery(
Target: database.Target{
Name: tgt.Name,
Type: database.TargetTypeDatabase,
ArchiveExpiry: tgt.ArchiveExpiry,
ArchiveRotation: tgt.ArchiveRotation,
Config: tgt.Config,
},
}
}
@@ -356,16 +355,13 @@ func TestParseArchiveExpiry(t *testing.T) {
want time.Duration
wantErr bool
}{
{"empty", "", 0, false},
{"explicit never", "never", 0, false},
{"hour", "1h", time.Hour, false},
{"thirty days", "720h", 720 * time.Hour, false},
{"minutes", "30m", 30 * time.Minute, false},
{"unparseable", "nonsense", 0, true},
{"days are not a Go unit", "7d", 0, true},
{"zero duration", "0s", 0, true},
{"bare zero", "0", 0, true},
{"negative duration", "-5h", 0, true},
{"empty config", "", 0, false},
{"explicit never", `{"expiry":"never"}`, 0, false},
{"empty expiry", `{"expiry":""}`, 0, false},
{"duration", `{"expiry":"1h"}`, time.Hour, false},
{"unparseable", `{"expiry":"nonsense"}`, 0, true},
{"zero duration", `{"expiry":"0s"}`, 0, true},
{"negative duration", `{"expiry":"-5h"}`, 0, true},
}
for _, tc := range cases {
@@ -385,6 +381,26 @@ func TestParseArchiveExpiry(t *testing.T) {
}
}
func TestValidateArchiveExpiry(t *testing.T) {
t.Parallel()
valid := []string{"", "never", "1h", "720h", "30m"}
for _, in := range valid {
require.NoError(t,
archive.ValidateExpiry(in),
"expiry %q should be accepted", in,
)
}
invalid := []string{"nonsense", "7d", "-5h", "0s", "0"}
for _, in := range invalid {
require.Error(t,
archive.ValidateExpiry(in),
"expiry %q should be rejected", in,
)
}
}
// TestArchiveFileName pins the archive file name and the rules
// that make a webhook or target name safe to put in it.
func TestArchiveFileName(t *testing.T) {
@@ -457,8 +473,8 @@ func TestWrite_EachTargetHasItsOwnArchive(t *testing.T) {
t.Parallel()
env := setupArchiveTest(t)
first := env.seedDatabaseTarget(t, 0, "")
second := env.addDatabaseTarget(t, first.WebhookID, 0, "")
first := env.seedDatabaseTarget(t, "")
second := env.addDatabaseTarget(t, first.WebhookID, "")
event := newEvent(`{"n":1}`)
@@ -487,7 +503,7 @@ func TestRename_MovesTheFile(t *testing.T) {
t.Parallel()
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, 0, "")
tgt := env.seedDatabaseTarget(t, "")
oldPath := env.archivePath(tgt)
first := newEvent(`{"n":1}`)
@@ -529,7 +545,7 @@ func TestRename_NeverReplacesAFile(t *testing.T) {
t.Parallel()
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, 0, "")
tgt := env.seedDatabaseTarget(t, "")
oldPath := env.archivePath(tgt)
first := newEvent(`{"n":1}`)
@@ -577,7 +593,7 @@ func TestRename_BeforeTheNameIsSaved(t *testing.T) {
t.Parallel()
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, 0, "")
tgt := env.seedDatabaseTarget(t, "")
require.NoError(
t, env.writers.Rename(tgt.ID, "Orders", "Archive"),
@@ -705,7 +721,7 @@ func TestRenameLock_HeldByOneEditAtATime(t *testing.T) {
t.Parallel()
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, 0, "")
tgt := env.seedDatabaseTarget(t, "")
event := newEvent(`{"n":1}`)
env.writers.RenameLock().Lock()
internal/archive/evict_test.go
+9 -9
View File
@@ -37,9 +37,9 @@ func TestEvictWebhook_ClosesAndRemovesWriter(t *testing.T) {
t.Parallel()
env := setupArchiveTest(t)
first := env.seedDatabaseTarget(t, 0, "")
second := env.addDatabaseTarget(t, first.WebhookID, 0, "")
other := env.seedDatabaseTarget(t, 0, "")
first := env.seedDatabaseTarget(t, "")
second := env.addDatabaseTarget(t, first.WebhookID, "")
other := env.seedDatabaseTarget(t, "")
for _, tgt := range []*database.Target{first, second, other} {
deliverTo(t, env, tgt)
@@ -80,8 +80,8 @@ func TestEvictTarget_LeavesOtherTargets(t *testing.T) {
t.Parallel()
env := setupArchiveTest(t)
doomed := env.seedDatabaseTarget(t, 0, "")
kept := env.addDatabaseTarget(t, doomed.WebhookID, 0, "")
doomed := env.seedDatabaseTarget(t, "")
kept := env.addDatabaseTarget(t, doomed.WebhookID, "")
deliverTo(t, env, doomed)
deliverTo(t, env, kept)
@@ -303,14 +303,14 @@ func (r *racingWrites) wait() (bool, error) {
// contending for the writer's mutex when the eviction takes it.
// The write that wins the mutex after the eviction must abandon
// its work rather than reopen the archive, leaving the writer
// permanently handle-free.
// permanently handle-free. Run under -race.
func TestEvictWebhook_RacingWriteDoesNotReopenHandle(
t *testing.T,
) {
t.Parallel()
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, 0, "")
tgt := env.seedDatabaseTarget(t, "")
// Prime the registry so the test can hold the very writer the
// eviction is about to detach.
@@ -355,7 +355,7 @@ func TestEvictWebhook_LaterDeliveryRecreatesWriter(t *testing.T) {
t.Parallel()
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, 0, "")
tgt := env.seedDatabaseTarget(t, "")
deliverTo(t, env, tgt)
require.True(t, env.writers.ExportHasArchiveWriter(tgt.ID))
@@ -380,7 +380,7 @@ func TestEvictAll_WriteAfterIsRefused(t *testing.T) {
t.Parallel()
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, 0, "")
tgt := env.seedDatabaseTarget(t, "")
deliverTo(t, env, tgt)
internal/archive/rotation.go
+36 -5
View File
@@ -1,6 +1,7 @@
package archive
import (
"encoding/json"
"errors"
"fmt"
"os"
@@ -53,10 +54,35 @@ func ValidateRotation(rotation string) error {
}
}
// ParseRotation reads the rotation from a database target's
// config JSON. An empty config or an empty rotation is none.
func ParseRotation(configJSON string) (string, error) {
if configJSON == "" {
return archiveRotationNone, nil
}
var cfg DatabaseTargetConfig
err := json.Unmarshal([]byte(configJSON), &cfg)
if err != nil {
return "", fmt.Errorf("parsing database target config: %w", err)
}
err = ValidateRotation(cfg.Rotation)
if err != nil {
return "", err
}
if cfg.Rotation == "" {
return archiveRotationNone, nil
}
return cfg.Rotation, nil
}
// archivePeriod returns the period, in UTC, that a rotation puts an
// event received at receivedAt in: "2026-10" for monthly,
// "2026-10-01" for daily, "2026-10-01-19" for hourly, and "" for none
// or an empty rotation.
// "2026-10-01" for daily, "2026-10-01-19" for hourly, and "" for none.
func archivePeriod(rotation string, receivedAt time.Time) string {
switch rotation {
case archiveRotationMonthly:
@@ -89,11 +115,16 @@ func PathAt(
webhook *database.Webhook,
target *database.Target,
receivedAt time.Time,
) string {
) (string, error) {
rotation, err := ParseRotation(target.Config)
if err != nil {
return "", err
}
return archivePeriodPath(
Path(dbMgr, webhook, target),
archivePeriod(target.ArchiveRotation, receivedAt),
)
archivePeriod(rotation, receivedAt),
), nil
}
// archiveFiles lists the database target's archive files that exist,
internal/archive/rotation_test.go
+30 -17
View File
@@ -14,12 +14,16 @@ import (
"sneak.berlin/go/webhooker/internal/database/databasetest"
)
// The archive rotations.
// The archive rotations, and the configs of a daily target and of one
// whose rotation is not one of the four.
const (
rotationNone = "none"
rotationMonthly = "monthly"
rotationDaily = "daily"
rotationHourly = "hourly"
dailyConfig = `{"rotation":"daily"}`
weeklyConfig = `{"rotation":"weekly"}`
)
// The periods the tests archive into most: two days, and an hour of
@@ -104,7 +108,7 @@ func TestWrite_RotatesAtEachPeriodBoundary(t *testing.T) {
t.Parallel()
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, 0, tc.rotation)
tgt := env.seedDatabaseTarget(t, `{"rotation":"`+tc.rotation+`"}`)
path := env.archivePath(tgt)
first := env.deliverReceivedAt(t, tgt, tc.before)
@@ -123,7 +127,7 @@ func TestWrite_RotatesAtEachPeriodBoundary(t *testing.T) {
t.Parallel()
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, 0, rotationNone)
tgt := env.seedDatabaseTarget(t, `{"rotation":"`+rotationNone+`"}`)
first := env.deliverReceivedAt(t, tgt, cases[0].before)
second := env.deliverReceivedAt(t, tgt, cases[0].after)
@@ -140,7 +144,7 @@ func TestWrite_EventWithoutReceiveTime(t *testing.T) {
t.Parallel()
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, 0, rotationMonthly)
tgt := env.seedDatabaseTarget(t, `{"rotation":"`+rotationMonthly+`"}`)
path := env.archivePath(tgt)
before := time.Now().UTC().Format("2006-01")
@@ -165,13 +169,13 @@ func TestWrite_RotationChangeKeepsOldFiles(t *testing.T) {
t.Parallel()
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, 0, rotationNone)
tgt := env.seedDatabaseTarget(t, "")
path := env.archivePath(tgt)
at := time.Date(2026, 3, 4, 12, 0, 0, 0, time.UTC)
first := env.deliverReceivedAt(t, tgt, at)
tgt.ArchiveRotation = rotationDaily
tgt.Config = dailyConfig
second := env.deliverReceivedAt(t, tgt, at)
assert.Equal(t, []string{first}, archivedEventIDs(t, path))
@@ -189,7 +193,9 @@ func TestArchiveSweep_PrunesEveryFile(t *testing.T) {
t.Parallel()
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, time.Hour, rotationDaily)
tgt := env.seedDatabaseTarget(
t, `{"expiry":"1h","rotation":"`+rotationDaily+`"}`,
)
path := env.archivePath(tgt)
emptied := periodPath(path, dayPeriod)
kept := periodPath(path, nextDayPeriod)
@@ -219,13 +225,13 @@ func TestRename_MovesEveryFile(t *testing.T) {
t.Parallel()
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, 0, rotationNone)
tgt := env.seedDatabaseTarget(t, "")
oldPath := env.archivePath(tgt)
day := time.Date(2026, 3, 4, 12, 0, 0, 0, time.UTC)
unrotated := env.deliverReceivedAt(t, tgt, day)
tgt.ArchiveRotation = rotationDaily
tgt.Config = dailyConfig
first := env.deliverReceivedAt(t, tgt, day)
second := env.deliverReceivedAt(t, tgt, day.Add(24*time.Hour))
@@ -261,7 +267,7 @@ func TestRename_NeverReplacesARotatedFile(t *testing.T) {
t.Parallel()
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, 0, rotationDaily)
tgt := env.seedDatabaseTarget(t, dailyConfig)
oldPath := env.archivePath(tgt)
day := time.Date(2026, 3, 4, 12, 0, 0, 0, time.UTC)
@@ -336,26 +342,33 @@ func TestArchivePathAt(t *testing.T) {
cases := map[string]string{
"": "",
rotationNone: "",
rotationMonthly: "-2026-10",
rotationDaily: "-2026-10-01",
rotationHourly: "-2026-10-01-19",
`{"rotation":"` + rotationNone + `"}`: "",
`{"rotation":"` + rotationMonthly + `"}`: "-2026-10",
dailyConfig: "-2026-10-01",
`{"rotation":"` + rotationHourly + `"}`: "-2026-10-01-19",
}
for rotation, period := range cases {
for config, period := range cases {
target := &database.Target{
BaseModel: database.BaseModel{ID: "tgt-id"},
Name: "Archive",
ArchiveRotation: rotation,
Config: config,
}
got, err := archive.PathAt(dbMgr, webhook, target, at)
require.NoError(t, err, config)
assert.Equal(t,
filepath.Join(
dataDir, "archive-orders-archive-tgt-id"+period+".db",
),
archive.PathAt(dbMgr, webhook, target, at), rotation,
got, config,
)
}
_, err := archive.PathAt(dbMgr, webhook, &database.Target{
Config: weeklyConfig,
}, at)
require.Error(t, err)
}
// TestValidateArchiveRotation accepts the four rotations, and empty,
internal/archive/sweeper.go
+16 -5
View File
@@ -199,11 +199,22 @@ func (s *Sweeper) sweep(ctx context.Context) {
}
}
// sweepTarget prunes the archive of a single database target. A
// target with an ArchiveExpiry of 0 keeps its archive forever and is
// skipped entirely, so that archive is never pruned.
// sweepTarget prunes the archive of a single database target.
// A missing, empty, or "never" expiry parses as a zero duration
// and is skipped entirely, so those archives are never pruned.
func (s *Sweeper) sweepTarget(target *database.Target) {
expiry := target.ArchiveExpiry
expiry, err := ParseExpiry(target.Config)
if err != nil {
s.log.Error(
"archive sweep: invalid database target config",
"webhook_id", target.WebhookID,
"target_id", target.ID,
"error", err,
)
return
}
if expiry <= 0 {
return
}
@@ -212,7 +223,7 @@ func (s *Sweeper) sweepTarget(target *database.Target) {
return
}
err := s.writers.sweepArchive(target.ID, expiry)
err = s.writers.sweepArchive(target.ID, expiry)
if err == nil {
return
}
internal/archive/sweeper_test.go
+45 -39
View File
@@ -106,10 +106,10 @@ func (env *archiveEnv) archivePath(tgt *database.Target) string {
)
}
// seedDatabaseTarget creates a webhook with one database target with
// the given archive expiry and rotation, and returns the target.
// seedDatabaseTarget creates a webhook with one database target
// carrying the given target config JSON, and returns the target.
func (env *archiveEnv) seedDatabaseTarget(
t *testing.T, expiry time.Duration, rotation string,
t *testing.T, configJSON string,
) *database.Target {
t.Helper()
@@ -124,16 +124,13 @@ func (env *archiveEnv) seedDatabaseTarget(
Create(wh).Error,
)
return env.addDatabaseTarget(t, wh.ID, expiry, rotation)
return env.addDatabaseTarget(t, wh.ID, configJSON)
}
// addDatabaseTarget creates one more database target on an
// existing webhook and returns it.
func (env *archiveEnv) addDatabaseTarget(
t *testing.T,
webhookID string,
expiry time.Duration,
rotation string,
t *testing.T, webhookID, configJSON string,
) *database.Target {
t.Helper()
@@ -142,8 +139,7 @@ func (env *archiveEnv) addDatabaseTarget(
Name: "Archive",
Type: database.TargetTypeDatabase,
Active: true,
ArchiveExpiry: expiry,
ArchiveRotation: rotation,
Config: configJSON,
MaxAttempts: 1,
}
require.NoError(
@@ -327,7 +323,7 @@ func TestArchiveSweeper_LoopOutlivesStartHookContext(
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, time.Hour, "")
tgt := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
now := time.Now()
path := env.seedArchiveRows(
@@ -381,7 +377,7 @@ func TestArchiveSweep_DoesNotResurrectEvictedWriter(
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, time.Hour, "")
tgt := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
env.seedArchiveRows(
t, tgt, time.Now().Add(-48*time.Hour),
)
@@ -415,7 +411,7 @@ func TestArchiveSweep_LeavesNoRegistryEntry(t *testing.T) {
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, time.Hour, "")
tgt := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
path := env.seedArchiveRows(
t, tgt,
time.Now().Add(-48*time.Hour),
@@ -448,7 +444,7 @@ func TestArchiveSweep_KeepsWriterAdoptedByDelivery(
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, time.Hour, "")
tgt := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
env.seedArchiveRows(
t, tgt, time.Now().Add(-48*time.Hour),
)
@@ -502,7 +498,7 @@ func TestArchiveSweep_KeepsWriterAdoptedDuringSweep(
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, time.Hour, "")
tgt := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
env.seedArchiveRows(
t, tgt, time.Now().Add(-48*time.Hour),
)
@@ -555,8 +551,8 @@ func TestArchiveSweep_KeepsWriterAdoptedDuringSweep(
// TestArchiveSweep_ContinuesAfterPerTargetFailure proves a
// failure for one target does not abort the sweep for the
// others: an unreadable archive has to be logged and stepped
// over.
// others: an unparseable expiry and an unreadable archive both
// have to be logged and stepped over.
func TestArchiveSweep_ContinuesAfterPerTargetFailure(
t *testing.T,
) {
@@ -564,16 +560,21 @@ func TestArchiveSweep_ContinuesAfterPerTargetFailure(
env := setupArchiveTest(t)
// Seeded first so the sweep reaches it before the healthy
// Seeded first so the sweep reaches them before the healthy
// target: targets come back in insertion order.
corrupt := env.seedDatabaseTarget(t, time.Hour, "")
badConfig := env.seedDatabaseTarget(t, `{"expiry":"!!!"}`)
env.seedArchiveRows(
t, badConfig, time.Now().Add(-48*time.Hour),
)
corrupt := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
require.NoError(t, os.WriteFile(
env.archivePath(corrupt),
[]byte("this is not a sqlite database"),
0o600,
))
healthy := env.seedDatabaseTarget(t, time.Hour, "")
healthy := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
healthyPath := env.seedArchiveRows(
t, healthy,
time.Now().Add(-48*time.Hour),
@@ -630,7 +631,7 @@ func TestArchiveSweep_PrunesIdleArchive(t *testing.T) {
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, time.Hour, "")
tgt := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
now := time.Now()
path := env.seedArchiveRows(
@@ -669,7 +670,7 @@ func TestArchiveSweep_LeavesArchiveClosed(t *testing.T) {
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, time.Hour, "")
tgt := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
path := env.seedArchiveRows(
t, tgt, time.Now().Add(-48*time.Hour),
)
@@ -709,7 +710,7 @@ func TestArchiveSweep_ClosesHandleOfRegisteredWriter(
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, time.Hour, "")
tgt := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
env.seedArchiveRows(
t, tgt, time.Now().Add(-48*time.Hour),
)
@@ -794,14 +795,19 @@ func TestArchiveSweep_CancelledSweepLogsNoError(t *testing.T) {
}
// TestArchiveSweep_NeverExpiryUntouched proves the sweep is a
// no-op for the default retention policy, so an archive with an
// expiry of 0, which keeps it forever, is left alone.
// no-op for the default retention policy, so archives with no
// expiry (or the literal "never") behave exactly as before.
func TestArchiveSweep_NeverExpiryUntouched(t *testing.T) {
t.Parallel()
for _, configJSON := range []string{
`{"expiry":"never"}`,
`{"expiry":""}`,
"",
} {
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, 0, "")
tgt := env.seedDatabaseTarget(t, configJSON)
path := env.seedArchiveRows(
t, tgt,
time.Now().Add(-10000*time.Hour),
@@ -811,13 +817,15 @@ func TestArchiveSweep_NeverExpiryUntouched(t *testing.T) {
assert.Equal(
t, []string{sweepRowOld}, archivedEventIDs(t, path),
"an expiry of 0 must keep rows forever",
"config %q must keep rows forever", configJSON,
)
assert.False(
t, env.writers.ExportHasArchiveWriter(tgt.ID),
"an expiry of 0 must leave no registry entry behind",
"config %q must leave no registry entry behind",
configJSON,
)
}
}
// TestArchiveSweep_NeverExpirySkipsBeforeOpening pins the
// expiry <= 0 boundary in sweepTarget, which the row assertions
@@ -837,7 +845,7 @@ func TestArchiveSweep_NeverExpirySkipsBeforeOpening(
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, 0, "")
tgt := env.seedDatabaseTarget(t, `{"expiry":"never"}`)
path := env.archivePath(tgt)
seedUnmigratedArchive(t, path)
@@ -889,7 +897,7 @@ func TestArchiveSweep_DoesNotCreateArchiveFile(t *testing.T) {
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, time.Hour, "")
tgt := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
path := env.archivePath(tgt)
require.NoFileExists(t, path)
@@ -920,7 +928,7 @@ func TestArchiveSweep_DoesNotCreateAfterWriterExists(
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, time.Hour, "")
tgt := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
path, err := env.writers.ExportEnsureArchiveWriter(tgt.ID)
require.NoError(t, err)
@@ -939,7 +947,7 @@ func TestArchiveSweep_SkipsDeletedWebhookTargets(t *testing.T) {
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, time.Hour, "")
tgt := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
path := env.seedArchiveRows(
t, tgt, time.Now().Add(-48*time.Hour),
)
@@ -959,17 +967,15 @@ func TestArchiveSweep_SkipsDeletedWebhookTargets(t *testing.T) {
)
}
// TestArchiveSweep_ConcurrentWrites runs the sweep while writes to
// the same target are under way. The sweep serialises against them
// through the target's writer mutex; the suite runs without Go's race
// detector, so a sweep that did not would fail this test only by
// panicking, hanging or leaving no archive file.
// TestArchiveSweep_ConcurrentWrites proves the sweep serialises
// against writes through the target's writer mutex. Run under
// -race, an unsynchronised sweep would be caught here.
func TestArchiveSweep_ConcurrentWrites(t *testing.T) {
t.Parallel()
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, time.Hour, "")
tgt := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
deliveries := make(
[]*database.Delivery, 0, sweepConcurrentWrites,
@@ -1014,7 +1020,7 @@ func TestArchiveSweeper_StopsCleanly(t *testing.T) {
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t, time.Hour, "")
tgt := env.seedDatabaseTarget(t, `{"expiry":"1h"}`)
env.seedArchiveRows(
t, tgt, time.Now().Add(-48*time.Hour),
)
internal/archive/writer.go
+70 -14
View File
@@ -1,6 +1,7 @@
package archive
import (
"encoding/json"
"errors"
"fmt"
"io/fs"
@@ -17,9 +18,8 @@ import (
"sneak.berlin/go/webhooker/internal/gormlog"
)
// ExpiryNever is the archive expiry, as the target forms take it,
// that disables pruning so archived rows are kept forever. It is
// stored as an ArchiveExpiry of 0.
// ExpiryNever is the expiry sentinel (and default) that
// disables pruning so archived rows are kept forever.
const ExpiryNever = "never"
// archiveReopenDebounce bounds how often an archive file is
@@ -79,6 +79,19 @@ var (
)
)
// DatabaseTargetConfig is the optional per-target JSON config
// for a database (archive) target.
type DatabaseTargetConfig struct {
// Expiry is a Go duration (e.g. "720h") after which
// archived rows are pruned, or "never" (the default) to
// keep them forever.
Expiry string `json:"expiry"`
// Rotation is none (the default), monthly, daily or hourly: see
// archivePeriod.
Rotation string `json:"rotation"`
}
// archivedEvent is one fully captured webhook event stored in a
// database target's archive for long-term retention. It is a
// self-contained copy — independent of the per-webhook event
@@ -99,20 +112,63 @@ type archivedEvent struct {
ArchivedAt time.Time `gorm:"index"`
}
// ParseExpiry reads a user-supplied archive expiry for a database
// target, as the target forms submit it, into the ArchiveExpiry the
// target stores. Empty and "never" both mean keep forever, which is
// 0; anything else must be a positive Go duration such as "720h". Any
// other value is an error, so a bad expiry is refused at the form
// rather than stored.
func ParseExpiry(expiry string) (time.Duration, error) {
if expiry == "" || expiry == ExpiryNever {
// ParseExpiry reads the optional expiry from a database
// target's config JSON. An empty config, an empty expiry, or
// the literal "never" all mean keep forever, returned as a zero
// duration. Any other value must parse as a positive Go
// duration; a set-but-invalid value (unparseable, zero, or
// negative) is an error rather than a silent default, matching
// ValidateExpiry at target creation.
func ParseExpiry(
configJSON string,
) (time.Duration, error) {
if configJSON == "" {
return 0, nil
}
var cfg DatabaseTargetConfig
err := json.Unmarshal([]byte(configJSON), &cfg)
if err != nil {
return 0, fmt.Errorf(
"parsing database target config: %w", err,
)
}
if cfg.Expiry == "" || cfg.Expiry == ExpiryNever {
return 0, nil
}
dur, err := time.ParseDuration(cfg.Expiry)
if err != nil {
return 0, fmt.Errorf(
"parsing archive expiry %q: %w", cfg.Expiry, err,
)
}
if dur <= 0 {
return 0, fmt.Errorf(
"%w: %q", errArchiveExpiryNotPositive, cfg.Expiry,
)
}
return dur, nil
}
// ValidateExpiry checks a user-supplied archive expiry
// for a database target at configuration time. Valid values are
// empty, "never" (both meaning keep forever), or a positive Go
// duration such as "720h". Anything else is an error, so a bad
// expiry is rejected when the target is created rather than
// failing every subsequent delivery.
func ValidateExpiry(expiry string) error {
if expiry == "" || expiry == ExpiryNever {
return nil
}
dur, err := time.ParseDuration(expiry)
if err != nil {
return 0, fmt.Errorf(
return fmt.Errorf(
"expiry must be %q or a Go duration "+
"such as \"720h\": %w",
ExpiryNever, err,
@@ -120,12 +176,12 @@ func ParseExpiry(expiry string) (time.Duration, error) {
}
if dur <= 0 {
return 0, fmt.Errorf(
return fmt.Errorf(
"%w: %q", errArchiveExpiryNotPositive, expiry,
)
}
return dur, nil
return nil
}
// archiveWriter owns one database target's archive SQLite files.
internal/database/event_db_isolation_test.go
+1 -1
View File
@@ -28,7 +28,7 @@ func TestEventDBCreateOmitsAssociations(t *testing.T) {
WebhookID: webhookID,
Name: "leaky-target",
Type: database.TargetTypeSlack,
URL: "https://hooks.example/secret",
Config: `{"webhookUrl":"https://hooks.example/secret"}`,
MaxAttempts: 1,
}
target.ID = uuid.New().String()
internal/database/model_secrets_test.go
+7 -18
View File
@@ -29,10 +29,8 @@ func marshalModel(t *testing.T, v any) string {
// these tags the first handler that marshals a model serialises the
// secret with it. Each field below is a live credential:
//
// - Target.URL is an incoming-webhook URL whose path segments are
// the bearer token.
// - Target.Headers holds header values, routinely authorization
// tokens.
// - Target.Config holds an incoming-webhook URL whose path segments
// are the bearer token.
// - APIKey.Key is a bearer token outright.
// - Setting.Value holds the session encryption key.
// - User.Password holds the Argon2 hash, and was already tagged.
@@ -46,20 +44,11 @@ func TestModelsDoNotMarshalTheirSecrets(t *testing.T) {
model any
}{
{
name: "target url",
name: "target config",
model: database.Target{
Name: keptField,
Type: database.TargetTypeSlack,
URL: "https://h/s/" + marker,
MaxAttempts: 1,
},
},
{
name: "target headers",
model: database.Target{
Name: keptField,
Type: database.TargetTypeHTTP,
Headers: map[string]string{"Authorization": marker},
Config: `{"webhookUrl":"https://h/s/` + marker + `"}`,
MaxAttempts: 1,
},
},
@@ -98,10 +87,10 @@ func TestModelsDoNotMarshalTheirSecrets(t *testing.T) {
}
}
// TestWebhookMarshalsNoTargetURL covers the nested case: a webhook
// TestWebhookMarshalsNoTargetConfig covers the nested case: a webhook
// marshalled with its targets preloaded must not carry the credential
// through the association either.
func TestWebhookMarshalsNoTargetURL(t *testing.T) {
func TestWebhookMarshalsNoTargetConfig(t *testing.T) {
t.Parallel()
const marker = "QQNESTEDMARKERQQ"
@@ -110,7 +99,7 @@ func TestWebhookMarshalsNoTargetURL(t *testing.T) {
Name: keptField,
Targets: []database.Target{{
Name: "slack",
URL: "https://h/s/" + marker,
Config: `{"webhookUrl":"https://h/s/` + marker + `"}`,
MaxAttempts: 1,
}},
})
internal/database/model_target.go
+8 -33
View File
@@ -1,7 +1,5 @@
package database
import "time"
// TargetType represents the type of delivery target
type TargetType string
@@ -13,9 +11,7 @@ const (
TargetTypeSlack TargetType = "slack"
)
// Target represents a delivery target for a webhook. Its settings
// are the fields from URL to ArchiveRotation; each type uses some of
// them, and the ones a type does not use stay empty.
// Target represents a delivery target for a webhook
type Target struct {
BaseModel
@@ -24,35 +20,14 @@ type Target struct {
Type TargetType `gorm:"not null" json:"type"`
Active bool `gorm:"default:true" json:"active"`
// URL is where an http or slack target sends each event.
// Configuration fields (JSON stored based on type).
//
// json:"-" because the URL is the target's credential: a Slack
// incoming-webhook URL, or an http destination whose path segments
// are the secret. delivery.TargetView is the masking barrier for
// the HTML path; this tag is the barrier for any handler that
// marshals the model itself.
URL string `json:"-"`
// Headers is an http target's extra request headers, name to
// value, stored as JSON text. json:"-" for the same reason as URL:
// a header value is routinely an authorization token.
Headers map[string]string `gorm:"serializer:json" json:"-"`
// TimeoutSeconds is an http target's per-request timeout; 0 uses
// the delivery engine's shared client timeout.
TimeoutSeconds int `json:"timeoutSeconds,omitempty"`
// ForwardQuery passes each event's query string on to an http
// target, appended to URL. Off, URL is sent exactly as stored.
ForwardQuery bool `json:"forwardQuery,omitempty"`
// ArchiveExpiry is how long a database target keeps each archived
// event; 0 keeps them forever.
ArchiveExpiry time.Duration `json:"archiveExpiry,omitempty"`
// ArchiveRotation is how often a database target starts a new
// archive file: none, monthly, daily or hourly.
ArchiveRotation string `json:"archiveRotation,omitempty"`
// json:"-" because the blob holds the target's credential — a
// Slack incoming-webhook URL, or an http destination whose path
// segments are the secret. delivery.TargetView is the masking
// barrier for the HTML path; this tag is the barrier for any
// handler that marshals the model itself.
Config string `gorm:"type:text" json:"-"` // JSON configuration
// MaxAttempts is how many times an HTTP or Slack target attempts
// each delivery in all, at least 1. 1 is a single attempt with no
internal/database/model_target_test.go
+1 -20
View File
@@ -16,7 +16,7 @@ func newHTTPTarget(maxAttempts int) *database.Target {
WebhookID: uuid.New().String(),
Name: "target",
Type: database.TargetTypeHTTP,
URL: "https://example.com/",
Config: `{"url":"https://example.com/"}`,
MaxAttempts: maxAttempts,
}
}
@@ -42,22 +42,3 @@ func TestTargetCreate_AcceptsOneAttempt(t *testing.T) {
require.NoError(t, db.Create(newHTTPTarget(1)).Error)
}
// TestTargetHeaders_StoredAsJSON reads the headers column back as
// text, to show it holds the header map as JSON.
func TestTargetHeaders_StoredAsJSON(t *testing.T) {
t.Parallel()
db := startedTestDB(t)
target := newHTTPTarget(1)
target.Headers = map[string]string{"Authorization": "Bearer x"}
require.NoError(t, db.Create(target).Error)
var headers string
require.NoError(t, db.Raw(
"SELECT headers FROM targets WHERE id = ?", target.ID,
).Scan(&headers).Error)
assert.JSONEq(t, `{"Authorization":"Bearer x"}`, headers)
}
internal/database/password.go
+3 -4
View File
@@ -67,10 +67,9 @@ func DefaultPasswordConfig() *PasswordConfig {
// testArgon2Memory is the Argon2id memory cost, in KiB, that a test
// binary hashes with: 1 MB instead of the shipped 64 MB. Every test
// that starts a database hashes the bootstrap admin password, dozens
// of them run in parallel, and each 64 MB hash holds at least 64 MB
// while it runs (about 150 MB when the suite ran under the race
// detector). VerifyPassword reads the cost from the hash it checks, so
// verification follows.
// of them run in parallel, and under the race detector each 64 MB hash
// holds about 150 MB. VerifyPassword reads the cost from the hash it
// checks, so verification follows.
const testArgon2Memory = 1024
// hashAtShippedCostInTest makes a test binary hash at the shipped
internal/database/schema_test.go
+1 -6
View File
@@ -92,12 +92,7 @@ func TestMainSchemaHoldsEveryModel(t *testing.T) {
Name: "target",
Type: database.TargetTypeHTTP,
Active: true,
URL: "https://example.com/",
Headers: map[string]string{"X-Api-Key": "key"},
TimeoutSeconds: 30,
ForwardQuery: true,
ArchiveExpiry: 720 * time.Hour,
ArchiveRotation: "daily",
Config: `{"url":"https://example.com/"}`,
MaxAttempts: 3,
})
}
internal/db/migrations/main/001_schema.sql
+1 -6
View File
@@ -79,12 +79,7 @@ CREATE TABLE targets (
name text NOT NULL,
type text NOT NULL,
active numeric DEFAULT true,
url text,
headers text,
timeout_seconds integer,
forward_query numeric,
archive_expiry integer,
archive_rotation text,
config text,
max_attempts integer NOT NULL,
PRIMARY KEY (id),
CONSTRAINT fk_webhooks_targets
internal/delivery/client_ssrf_test.go
+19 -3
View File
@@ -46,7 +46,12 @@ func TestClientForRequest_TimeoutKeepsSSRFGuard(t *testing.T) {
t.Run(target, func(t *testing.T) {
t.Parallel()
client := engine.ExportClientForRequest(5, nil)
cfg := &delivery.HTTPTargetConfig{
URL: target,
Timeout: 5,
}
client := engine.ExportClientForRequest(cfg, nil)
require.NotSame(t, engine.ExportClient(), client,
"a per-target timeout must yield a "+
@@ -96,7 +101,11 @@ func TestClientForRequest_NoTimeoutUnchanged(t *testing.T) {
engine := newSSRFTestEngine()
client := engine.ExportClientForRequest(0, nil)
cfg := &delivery.HTTPTargetConfig{
URL: "https://example.com/hook",
}
client := engine.ExportClientForRequest(cfg, nil)
assert.Same(t, engine.ExportClient(), client,
"without a per-target timeout the shared client "+
@@ -114,8 +123,15 @@ func TestClientForRequest_HeadersKeepSSRFGuard(t *testing.T) {
engine := newSSRFTestEngine()
cfg := &delivery.HTTPTargetConfig{
URL: "https://example.com/with-headers",
Headers: map[string]string{
"X-Api-Key": "configured",
},
}
client := engine.ExportClientForRequest(
0, []string{probeHeaderName},
cfg, []string{"X-Api-Key"},
)
require.NotNil(t, client.CheckRedirect,
internal/delivery/engine.go
+63 -80
View File
@@ -1,5 +1,5 @@
// Package delivery stores each event with its deliveries and delivers
// them, asynchronously, to configured targets.
// Package delivery manages asynchronous event delivery
// to configured targets.
package delivery
import (
@@ -23,11 +23,10 @@ import (
const (
// deliveryChannelSize is the buffer size for the delivery
// channel. The Tasks of newly stored deliveries, and of the
// pending deliveries the recovery routine hands back, are
// sent here. Workers drain this channel. Sized large enough
// that no task should be dropped under normal load: one that
// does not fit is left for the sweep.
// channel. New Tasks from the webhook handler are sent
// here. Workers drain this channel. Sized large enough
// that the webhook handler should never block under
// normal load.
deliveryChannelSize = 10000
// retryChannelSize is the buffer size for the retry
@@ -106,19 +105,12 @@ type Task struct {
WebhookID string
EntrypointID string
// The target's fields, as they were when the task was made.
TargetID string
TargetName string
TargetType database.TargetType
TargetURL string
TargetHeaders map[string]string
TimeoutSeconds int
ForwardQuery bool
ArchiveExpiry time.Duration
ArchiveRotation string
TargetConfig string
MaxAttempts int
// The event's fields.
Method string
RawQuery string
Headers string
@@ -128,11 +120,17 @@ type Task struct {
AttemptNum int
}
// Notifier is the interface for notifying the delivery
// engine about new deliveries.
type Notifier interface {
Notify(tasks []Task)
}
// CircuitBreakers is how the handlers read a target's circuit
// breaker, so the webhook page and the event log can say that
// deliveries to the target are paused and until when. The handlers
// hold the engine itself as well; they read breakers through this
// interface so that a test can make one read as open.
// deliveries to the target are paused and until when. It keeps the
// handlers free of the engine's internals and is trivially faked in
// tests.
type CircuitBreakers interface {
StateAndCooldown(targetID string) (CircuitState, time.Duration)
}
@@ -152,13 +150,12 @@ type EngineParams struct {
}
// Engine processes queued deliveries in the background
// using a bounded worker pool architecture. It also stores each
// event with its deliveries and creates replay deliveries (see
// store.go). Beyond that it owns only the cross-target machinery:
// the worker pool, the delivery and retry channels, the recovery
// routine, and the persistence helpers and Scheduler that
// individual targets rely on. Each target type owns its own
// delivery, including retries, backoff, and circuit breaking.
// using a bounded worker pool architecture. It owns only
// the cross-target machinery: the worker pool, the queue
// and retry channels, the recovery routine, and the persistence
// helpers and Scheduler that individual targets rely on.
// Each target type owns its own delivery, including retries,
// backoff, and circuit breaking.
type Engine struct {
database *database.Database
dbManager *database.WebhookDBManager
@@ -228,6 +225,38 @@ func New(
return e
}
// Notify signals the delivery engine that new deliveries
// are ready.
func (e *Engine) Notify(tasks []Task) {
for i := range tasks {
// Owned before it is queued, and until the worker that runs
// it returns. A task can sit in a 10000-deep channel for a
// long time on a healthy system, and nothing may re-send it
// while it waits. See inflight.go.
if !e.inflight.retainIdle(tasks[i].DeliveryID) {
e.log.Warn(
"delivery already in flight, not queued again",
"delivery_id", tasks[i].DeliveryID,
"event_id", tasks[i].EventID,
)
continue
}
select {
case e.deliveryCh <- tasks[i]:
default:
e.inflight.release(tasks[i].DeliveryID)
e.log.Warn(
"delivery channel full, "+
"task will be recovered by the sweep",
"delivery_id", tasks[i].DeliveryID,
"event_id", tasks[i].EventID,
)
}
}
}
// StateAndCooldown implements CircuitBreakers. It returns the state of
// the target's circuit breaker and, while the breaker is open, what is
// left of its cooldown; the cooldown is zero once that has passed and
@@ -288,38 +317,6 @@ func (e *Engine) ScheduleRetry(
})
}
// notify queues the tasks of deliveries already stored. StoreEvent and
// CreateReplay call it once their transaction has committed.
func (e *Engine) notify(tasks []Task) {
for i := range tasks {
// Owned before it is queued, and until the worker that runs
// it returns. A task can sit in a 10000-deep channel for a
// long time on a healthy system, and nothing may re-send it
// while it waits. See inflight.go.
if !e.inflight.retainIdle(tasks[i].DeliveryID) {
e.log.Warn(
"delivery already in flight, not queued again",
"delivery_id", tasks[i].DeliveryID,
"event_id", tasks[i].EventID,
)
continue
}
select {
case e.deliveryCh <- tasks[i]:
default:
e.inflight.release(tasks[i].DeliveryID)
e.log.Warn(
"delivery channel full, "+
"task will be recovered by the sweep",
"delivery_id", tasks[i].DeliveryID,
"event_id", tasks[i].EventID,
)
}
}
}
// userAgent is the User-Agent header of every http and slack
// delivery request: the program name and the running build's
// version.
@@ -468,9 +465,9 @@ func (e *Engine) processNewTask(
}
// The recovery routine's run at start can send and release this
// delivery before StoreEvent or CreateReplay queues it. Ownership
// cannot refuse a delivery nobody holds, so the row decides whether
// it still needs sending.
// delivery before the receiver's Notify queues it. Ownership cannot
// refuse a delivery nobody holds, so the row decides whether it
// still needs sending.
row, err := e.loadDelivery(webhookDB, task.DeliveryID)
if err != nil {
e.log.Error(
@@ -895,7 +892,7 @@ func (e *Engine) recoverSingleRetry(
return
}
e.ScheduleRetry(buildTask(
e.ScheduleRetry(buildRecoveryTask(
d, webhookID, &event, &target, attemptNum+1,
), remaining)
@@ -917,7 +914,7 @@ func (e *Engine) recoverSingleRetry(
// it stopped working on before then: for example one whose bookkeeping
// write failed, whose attempt a stop cut off, that was still queued
// when the engine last stopped, whose task was dropped because the
// delivery channel was full (by notify or by queueRecovered), or whose
// delivery channel was full (by Notify or by queueRecovered), or whose
// worker could not open the webhook's database.
//
// At start every pending delivery is taken, however recent, so one a
@@ -1548,12 +1545,7 @@ func buildTargetFromTask(task *Task) database.Target {
target := database.Target{
Name: task.TargetName,
Type: task.TargetType,
URL: task.TargetURL,
Headers: task.TargetHeaders,
TimeoutSeconds: task.TimeoutSeconds,
ForwardQuery: task.ForwardQuery,
ArchiveExpiry: task.ArchiveExpiry,
ArchiveRotation: task.ArchiveRotation,
Config: task.TargetConfig,
MaxAttempts: task.MaxAttempts,
}
@@ -1818,11 +1810,7 @@ func (e *Engine) loadTarget(
return target, nil
}
// buildTask builds the task that carries delivery d of event to target
// through the engine's channels, as attempt attemptNum. The body is
// carried only when it is under MaxInlineBodySize; a larger one is read
// from the event row when the attempt runs (see hydrateEvent).
func buildTask(
func buildRecoveryTask(
d *database.Delivery,
webhookID string,
event *database.Event,
@@ -1844,12 +1832,7 @@ func buildTask(
TargetID: target.ID,
TargetName: target.Name,
TargetType: target.Type,
TargetURL: target.URL,
TargetHeaders: target.Headers,
TimeoutSeconds: target.TimeoutSeconds,
ForwardQuery: target.ForwardQuery,
ArchiveExpiry: target.ArchiveExpiry,
ArchiveRotation: target.ArchiveRotation,
TargetConfig: target.Config,
MaxAttempts: target.MaxAttempts,
Method: event.Method,
RawQuery: event.RawQuery,
@@ -1998,7 +1981,7 @@ func (e *Engine) sendRecoveredDeliveries(
continue
}
task := buildTask(
task := buildRecoveryTask(
&deliveries[i], webhookID, &event, &target,
attempts[deliveries[i].ID]+1,
)
internal/delivery/engine_integration_test.go
+111 -59
View File
@@ -3,6 +3,7 @@ package delivery_test
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"log/slog"
@@ -96,6 +97,17 @@ func iSeedWebhookDB(
return db
}
func iHTTPConfig(url string) string {
cfg := delivery.HTTPTargetConfig{URL: url}
data, err := json.Marshal(cfg)
if err != nil {
panic("failed to marshal HTTPTargetConfig")
}
return string(data)
}
func iEngine(
t *testing.T, workers int,
) *delivery.Engine {
@@ -157,7 +169,7 @@ func iSeedDelivery(
func iTask(
d database.Delivery,
event database.Event,
webhookID, targetID, name, targetURL string,
webhookID, targetID, name, config string,
maxAttempts, attemptNum int,
body *string,
) delivery.Task {
@@ -168,7 +180,7 @@ func iTask(
TargetID: targetID,
TargetName: name,
TargetType: database.TargetTypeHTTP,
TargetURL: targetURL,
TargetConfig: config,
MaxAttempts: maxAttempts,
Method: event.Method,
Headers: event.Headers,
@@ -227,9 +239,10 @@ func TestProcessNewTask_InlineBody(t *testing.T) {
)
bodyStr := event.Body
cfg := iHTTPConfig(ts.URL)
task := iTask(
d, event, s.WebhookID, targetID,
"test-target", ts.URL, 1, 1, &bodyStr,
"test-target", cfg, 1, 1, &bodyStr,
)
s.Engine.ExportProcessNewTask(
@@ -277,9 +290,10 @@ func TestProcessNewTask_LargeBody_FetchFromDB(
database.DeliveryStatusPending,
)
cfg := iHTTPConfig(ts.URL)
task := iTask(
d, event, s.WebhookID, targetID,
"test-large", ts.URL, 1, 1, nil,
"test-large", cfg, 1, 1, nil,
)
s.Engine.ExportProcessNewTask(
@@ -305,7 +319,7 @@ func TestProcessNewTask_InvalidWebhookID(t *testing.T) {
TargetID: uuid.New().String(),
TargetName: "test",
TargetType: database.TargetTypeHTTP,
TargetURL: "http://localhost:9999",
TargetConfig: iHTTPConfig("http://localhost:9999"),
MaxAttempts: 1,
Body: nil,
AttemptNum: 1,
@@ -349,6 +363,7 @@ func TestProcessRetryTask_SuccessfulRetry(t *testing.T) {
)
bodyStr := event.Body
cfg := iHTTPConfig(ts.URL)
// The target row exists because the engine confirms a scheduled
// retry's target has not been deleted before it runs it. A retry
@@ -357,12 +372,12 @@ func TestProcessRetryTask_SuccessfulRetry(t *testing.T) {
// task. See https://git.eeqj.de/sneak/webhooker/issues/107.
iCreateTarget(
t, s.MainDB, targetID, s.WebhookID, "retry-target",
database.TargetTypeHTTP, ts.URL, 5,
database.TargetTypeHTTP, cfg, 5,
)
task := iTask(
d, event, s.WebhookID, targetID,
"retry-target", ts.URL, 5, 2, &bodyStr,
"retry-target", cfg, 5, 2, &bodyStr,
)
s.Engine.ExportProcessRetryTask(
@@ -395,9 +410,10 @@ func TestProcessRetryTask_SkipsNonRetryingDelivery(
)
bodyStr := event.Body
cfg := iHTTPConfig("http://localhost:1")
task := iTask(
d, event, s.WebhookID, targetID,
"skip-target", "http://localhost:1", 5, 2, &bodyStr,
"skip-target", cfg, 5, 2, &bodyStr,
)
s.Engine.ExportProcessRetryTask(
@@ -444,14 +460,16 @@ func TestProcessRetryTask_LargeBody_FetchFromDB(
database.DeliveryStatusRetrying,
)
cfg := iHTTPConfig(ts.URL)
iCreateTarget(
t, s.MainDB, targetID, s.WebhookID, "retry-large",
database.TargetTypeHTTP, ts.URL, 5,
database.TargetTypeHTTP, cfg, 5,
)
task := iTask(
d, event, s.WebhookID, targetID,
"retry-large", ts.URL, 5, 2, nil,
"retry-large", cfg, 5, 2, nil,
)
s.Engine.ExportProcessRetryTask(
@@ -492,7 +510,7 @@ func TestWorkerLifecycle_StartStop(t *testing.T) {
)
task.TargetType = database.TargetTypeLog
s.Engine.ExportNotify([]delivery.Task{task})
s.Engine.Notify([]delivery.Task{task})
iWaitForDelivered(t, s.WebhookDB, d.ID)
@@ -552,15 +570,16 @@ func TestWorkerLifecycle_ProcessesRetryChannel(
s.Engine.ExportStart()
bodyStr := event.Body
cfg := iHTTPConfig(ts.URL)
iCreateTarget(
t, s.MainDB, targetID, s.WebhookID, "retry-chan-test",
database.TargetTypeHTTP, ts.URL, 5,
database.TargetTypeHTTP, cfg, 5,
)
task := iTask(
d, event, s.WebhookID, targetID,
"retry-chan-test", ts.URL, 5, 2, &bodyStr,
"retry-chan-test", cfg, 5, 2, &bodyStr,
)
s.Engine.ExportRetryCh() <- task
@@ -666,7 +685,7 @@ func iCreateTarget(
mainDB *gorm.DB,
targetID, webhookID, name string,
targetType database.TargetType,
targetURL string,
config string,
maxAttempts int,
) {
t.Helper()
@@ -676,7 +695,7 @@ func iCreateTarget(
Name: name,
Type: targetType,
Active: true,
URL: targetURL,
Config: config,
MaxAttempts: maxAttempts,
}
target.ID = targetID
@@ -717,7 +736,7 @@ func TestRecoverWebhookDeliveries_RetryingDeliveries(
iCreateTarget(t, s.MainDB, targetID,
s.WebhookID, "retry-recovery",
database.TargetTypeHTTP,
"http://example.com/hook", 5,
iHTTPConfig("http://example.com/hook"), 5,
)
event := iSeedEvent(
@@ -777,7 +796,7 @@ func iSeedRetryingWithType(
iCreateTarget(t, s.MainDB, targetID,
s.WebhookID, "mutated-target", targetType,
"http://example.com/hook", 5,
iHTTPConfig("http://example.com/hook"), 5,
)
event := iSeedEvent(
@@ -915,14 +934,14 @@ func TestFailUnretryableRetry_WritesNoTargetRow(
targetID := uuid.New().String()
// A Slack incoming-webhook URL: the target's URL IS the
// A Slack incoming-webhook URL: the target config IS the
// credential, which is what makes a leaked target row a
// disclosure rather than a curiosity.
hookURL := "https://hooks.slack.com/services/T00/B00/x"
iCreateTarget(t, s.MainDB, targetID,
s.WebhookID, "credential-bearing",
database.TargetTypeLog, hookURL, 5,
database.TargetTypeLog, iHTTPConfig(hookURL), 5,
)
event := iSeedEvent(
@@ -1167,7 +1186,7 @@ func TestRecoverDeliveries_SkipsAWebhookDeletedAfterTheListIsRead(
assert.False(t, dbMgr.DBExists(webhookID))
}
// --- HTTP target with custom headers ---
// --- HTTP Config with custom headers ---
func TestDeliverHTTP_CustomTargetHeaders(t *testing.T) {
t.Parallel()
@@ -1189,6 +1208,16 @@ func TestDeliverHTTP_CustomTargetHeaders(t *testing.T) {
)
defer ts.Close()
cfg := delivery.HTTPTargetConfig{
URL: ts.URL,
Headers: map[string]string{
"Authorization": "Bearer secret-token",
},
}
cfgJSON, err := json.Marshal(cfg)
require.NoError(t, err)
event := iSeedEvent(
t, s.WebhookDB, s.WebhookID,
`{"auth":"test"}`,
@@ -1203,12 +1232,9 @@ func TestDeliverHTTP_CustomTargetHeaders(t *testing.T) {
bodyStr := event.Body
task := iTask(
d, event, s.WebhookID, targetID,
"auth-target", ts.URL,
"auth-target", string(cfgJSON),
1, 1, &bodyStr,
)
task.TargetHeaders = map[string]string{
headerAuthorization: "Bearer secret-token",
}
s.Engine.ExportProcessNewTask(
context.TODO(), &task,
@@ -1221,14 +1247,11 @@ func TestDeliverHTTP_CustomTargetHeaders(t *testing.T) {
// --- HTTP delivery with custom timeout ---
// TestDeliverHTTP_TargetTimeout proves a task's TimeoutSeconds reaches
// the delivery through the engine. The destination answers after two
// seconds, inside the engine's own five-second client timeout, so only
// the task's one second can fail the delivery.
func TestDeliverHTTP_TargetTimeout(t *testing.T) {
t.Parallel()
s := newISetup(t)
db := testWebhookDB(t)
e := iEngine(t, 1)
ts := httptest.NewServer(
http.HandlerFunc(
@@ -1240,30 +1263,31 @@ func TestDeliverHTTP_TargetTimeout(t *testing.T) {
)
defer ts.Close()
event := iSeedEvent(
t, s.WebhookDB, s.WebhookID, `{"timeout":"test"}`,
)
targetID := uuid.New().String()
cfg := delivery.HTTPTargetConfig{
URL: ts.URL,
Timeout: 1,
}
d := iSeedDelivery(
t, s.WebhookDB, event.ID, targetID,
database.DeliveryStatusPending,
cfgJSON, err := json.Marshal(cfg)
require.NoError(t, err)
event, del := iSeedEventAndDelivery(
t, db, `{"timeout":"test"}`,
string(cfgJSON),
)
bodyStr := event.Body
task := iTask(
d, event, s.WebhookID, targetID,
"timeout-target", ts.URL, 1, 1, &bodyStr,
task, d := iHTTPTaskAndDelivery(
event, del, "timeout-target",
string(cfgJSON), 1, 1,
)
task.TimeoutSeconds = 1
s.Engine.ExportProcessNewTask(context.TODO(), &task)
e.ExportDeliverHTTP(context.TODO(), db, d, task)
iAssertStatus(t, s.WebhookDB, d.ID,
iAssertStatus(t, db, del.ID,
database.DeliveryStatusFailed,
)
iAssertResultFailed(t, s.WebhookDB, d.ID)
iAssertResultFailed(t, db, del.ID)
}
// TestDeliverHTTP_CutsStoredResponseAtMaxBodyLog pins the size
@@ -1301,6 +1325,8 @@ func TestDeliverHTTP_CutsStoredResponseAtMaxBodyLog(
))
defer ts.Close()
cfgJSON := iHTTPConfig(ts.URL)
event := iSeedEvent(
t, s.WebhookDB, s.WebhookID, `{"cut":"test"}`,
)
@@ -1314,7 +1340,7 @@ func TestDeliverHTTP_CutsStoredResponseAtMaxBodyLog(
bodyStr := event.Body
task := iTask(
del, event, s.WebhookID, targetID,
"cut-target", ts.URL, 1, 1, &bodyStr,
"cut-target", cfgJSON, 1, 1, &bodyStr,
)
s.Engine.ExportProcessNewTask(context.TODO(), &task)
@@ -1346,7 +1372,7 @@ func TestDeliverHTTP_CutsStoredResponseAtMaxBodyLog(
func iSeedEventAndDelivery(
t *testing.T,
db *gorm.DB,
body string,
body, _ string,
) (database.Event, database.Delivery) {
t.Helper()
@@ -1377,7 +1403,7 @@ func iSeedEventAndDelivery(
func iHTTPTaskAndDelivery(
event database.Event,
del database.Delivery,
name, targetURL string,
name, config string,
maxAttempts, attemptNum int,
) (*delivery.Task, *database.Delivery) {
task := &delivery.Task{
@@ -1387,7 +1413,7 @@ func iHTTPTaskAndDelivery(
TargetID: del.TargetID,
TargetName: name,
TargetType: database.TargetTypeHTTP,
TargetURL: targetURL,
TargetConfig: config,
MaxAttempts: maxAttempts,
AttemptNum: attemptNum,
}
@@ -1400,7 +1426,7 @@ func iHTTPTaskAndDelivery(
Target: database.Target{
Name: name,
Type: database.TargetTypeHTTP,
URL: targetURL,
Config: config,
MaxAttempts: maxAttempts,
},
}
@@ -1429,30 +1455,56 @@ func iAssertResultFailed(
assert.NotEmpty(t, result.Error)
}
// --- HTTP request with no URL ---
// --- HTTP request with invalid config ---
// TestDeliverHTTP_NoURL proves an http target without a URL, which
// the forms never store, fails its delivery and records why.
func TestDeliverHTTP_NoURL(t *testing.T) {
func TestDeliverHTTP_InvalidConfig(t *testing.T) {
t.Parallel()
db := testWebhookDB(t)
e := iEngine(t, 1)
event, del := iSeedEventAndDelivery(t, db, `{"url":"none"}`)
event, del := iSeedEventAndDelivery(
t, db, `{"config":"invalid"}`, "",
)
task, d := iHTTPTaskAndDelivery(event, del, "no-url", "", 1, 1)
task, d := iHTTPTaskAndDelivery(
event, del, "bad-config", `not-json`, 1, 1,
)
e.ExportDeliverHTTP(context.TODO(), db, d, task)
iAssertStatus(t, db, del.ID,
database.DeliveryStatusFailed,
)
iAssertResultFailed(t, db, del.ID)
}
// --- notify batching ---
// TestDeliverHTTP_InvalidConfigUnrecordedStaysPending: a delivery is
// failed for an invalid config only once the reason is recorded.
// Unrecorded, it stays pending, where the sweep finds it again.
func TestDeliverHTTP_InvalidConfigUnrecordedStaysPending(t *testing.T) {
t.Parallel()
db := testWebhookDB(t)
e := testEngine(t, 1)
event, del := iSeedEventAndDelivery(
t, db, `{"config":"invalid"}`, "",
)
task, d := iHTTPTaskAndDelivery(
event, del, "bad-config", `not-json`, 1, 1,
)
require.NoError(t, db.Exec("drop table delivery_results").Error)
e.ExportDeliverHTTP(context.TODO(), db, d, task)
iAssertStatus(t, db, del.ID,
database.DeliveryStatusPending,
)
}
// --- Notify batching ---
func TestNotify_MultipleTasks(t *testing.T) {
t.Parallel()
@@ -1467,7 +1519,7 @@ func TestNotify_MultipleTasks(t *testing.T) {
}
}
e.ExportNotify(tasks)
e.Notify(tasks)
for i := range 5 {
select {
internal/delivery/engine_lifecycle_test.go
+9 -8
View File
@@ -29,7 +29,7 @@ const (
// OnStart before the caller may enqueue work. A worker pool
// wrongly rooted in the already-done hook context has nothing
// but ctx.Done() ready in its select, so it is deterministically
// gone by the end of this window. Without the wait, notify would
// gone by the end of this window. Without the wait, Notify would
// race the pool's very first select, in which a ready ctx.Done()
// and a ready deliveryCh are chosen between at random and a
// doomed pool still delivers.
@@ -182,7 +182,7 @@ func TestEngine_WorkersOutliveStartHookContext(t *testing.T) {
// routine's run at start cannot enqueue it.
deliveryID, task := seedLogTask(t, s)
s.Engine.ExportNotify([]delivery.Task{task})
s.Engine.Notify([]delivery.Task{task})
iWaitForDelivered(t, s.WebhookDB, deliveryID)
}
@@ -202,7 +202,7 @@ func TestEngine_StopHookStopsWorkers(t *testing.T) {
// Let the pool prove it is running before stopping it, so a
// fast OnStop cannot pass by stopping something already dead.
firstID, firstTask := seedLogTask(t, s)
s.Engine.ExportNotify([]delivery.Task{firstTask})
s.Engine.Notify([]delivery.Task{firstTask})
iWaitForDelivered(t, s.WebhookDB, firstID)
var stopErr error
@@ -232,7 +232,7 @@ func TestEngine_StopHookStopsWorkers(t *testing.T) {
// With every worker gone, a freshly notified task must sit
// untouched in the queue rather than being delivered.
secondID, secondTask := seedLogTask(t, s)
s.Engine.ExportNotify([]delivery.Task{secondTask})
s.Engine.Notify([]delivery.Task{secondTask})
time.Sleep(200 * time.Millisecond)
@@ -309,12 +309,13 @@ func seedHeldDelivery(
))
t.Cleanup(ts.Close)
cfg := iHTTPConfig(ts.URL)
targetID := uuid.New().String()
iCreateWebhook(t, s.MainDB, s.WebhookID, "hook")
iCreateTarget(
t, s.MainDB, targetID, s.WebhookID, "held",
database.TargetTypeHTTP, ts.URL, maxAttempts,
database.TargetTypeHTTP, cfg, maxAttempts,
)
event := iSeedEvent(t, s.WebhookDB, s.WebhookID, `{"held":true}`)
@@ -323,7 +324,7 @@ func seedHeldDelivery(
bodyStr := event.Body
return iTask(
d, event, s.WebhookID, targetID, "held", ts.URL,
d, event, s.WebhookID, targetID, "held", cfg,
maxAttempts, attemptNum, &bodyStr,
), arrived
}
@@ -383,7 +384,7 @@ func TestEngine_StopLeavesFireAndForgetAttemptPending(t *testing.T) {
t, s, 1, database.DeliveryStatusPending, 1,
)
s.Engine.ExportNotify([]delivery.Task{task})
s.Engine.Notify([]delivery.Task{task})
stopDuringAttempt(t, lc, arrived)
iAssertStatus(
@@ -448,7 +449,7 @@ func deliverToArchive(t *testing.T, s iSetup) string {
task.TargetID = tgt.ID
task.TargetType = database.TargetTypeDatabase
s.Engine.ExportNotify([]delivery.Task{task})
s.Engine.Notify([]delivery.Task{task})
iWaitForDelivered(t, s.WebhookDB, deliveryID)
internal/delivery/engine_test.go
+196 -44
View File
@@ -65,6 +65,20 @@ func testEngine(
)
}
func newHTTPTargetConfig(url string) string {
cfg := delivery.HTTPTargetConfig{URL: url}
data, err := json.Marshal(cfg)
if err != nil {
panic(
"failed to marshal HTTPTargetConfig: " +
err.Error(),
)
}
return string(data)
}
func seedEvent(
t *testing.T, db *gorm.DB, body string,
) database.Event {
@@ -127,7 +141,7 @@ type httpDeliveryFixture struct {
func buildHTTPFixture(
dlv database.Delivery,
event database.Event,
targetID, name, targetURL string,
targetID, name, config string,
maxAttempts, attemptNum int,
) httpDeliveryFixture {
task := &delivery.Task{
@@ -137,7 +151,7 @@ func buildHTTPFixture(
TargetID: targetID,
TargetName: name,
TargetType: database.TargetTypeHTTP,
TargetURL: targetURL,
TargetConfig: config,
MaxAttempts: maxAttempts,
AttemptNum: attemptNum,
}
@@ -150,7 +164,7 @@ func buildHTTPFixture(
Target: database.Target{
Name: name,
Type: database.TargetTypeHTTP,
URL: targetURL,
Config: config,
MaxAttempts: maxAttempts,
},
}
@@ -218,7 +232,7 @@ func TestNotify_NonBlocking(t *testing.T) {
done := make(chan struct{})
go func() {
e.ExportNotify([]delivery.Task{
e.Notify([]delivery.Task{
{DeliveryID: "overflow-1"},
{DeliveryID: "overflow-2"},
})
@@ -229,7 +243,7 @@ func TestNotify_NonBlocking(t *testing.T) {
case <-done:
case <-time.After(2 * time.Second):
t.Fatal(
"notify blocked when delivery channel was full",
"Notify blocked when delivery channel was full",
)
}
}
@@ -261,8 +275,9 @@ func TestDeliverHTTP_Success(t *testing.T) {
database.DeliveryStatusPending,
)
cfg := newHTTPTargetConfig(ts.URL)
fix := buildHTTPFixture(
dlv, event, targetID, "test-http", ts.URL, 1, 1,
dlv, event, targetID, "test-http", cfg, 1, 1,
)
e.ExportDeliverHTTP(
@@ -306,9 +321,10 @@ func TestDeliverHTTP_Failure(t *testing.T) {
database.DeliveryStatusPending,
)
cfg := newHTTPTargetConfig(ts.URL)
fix := buildHTTPFixture(
dlv, event, targetID,
"test-http-fail", ts.URL, 1, 1,
"test-http-fail", cfg, 1, 1,
)
e.ExportDeliverHTTP(
@@ -335,7 +351,7 @@ func TestDeliverDatabase_ImmediateSuccess(
// The database target archives for real, so the engine needs
// the target in the main database and a data directory.
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t)
tgt := env.seedDatabaseTarget(t, "")
event := seedEvent(t, db, `{"db":"target"}`)
d := seedDatabaseTargetDelivery(t, db, event, tgt)
@@ -436,9 +452,10 @@ func TestDeliverHTTP_WithRetries_Success(t *testing.T) {
database.DeliveryStatusPending,
)
cfg := newHTTPTargetConfig(ts.URL)
fix := buildHTTPFixture(
dlv, event, targetID,
"test-http-retry", ts.URL, 5, 1,
"test-http-retry", cfg, 5, 1,
)
e.ExportDeliverHTTP(
@@ -493,12 +510,13 @@ func TestDeliverHTTP_MaxAttemptsExhausted(t *testing.T) {
database.DeliveryStatusPending,
)
cfg := newHTTPTargetConfig(ts.URL)
sched := &recordingScheduler{}
for attempt := 1; attempt <= maxAttempts; attempt++ {
fix := buildHTTPFixture(
dlv, event, targetID,
"test-http-exhaust", ts.URL, maxAttempts, attempt,
"test-http-exhaust", cfg, maxAttempts, attempt,
)
e.ExportDeliverHTTPWithScheduler(
@@ -556,9 +574,10 @@ func TestDeliverHTTP_SingleAttemptSkipsCircuitBreaker(t *testing.T) {
database.DeliveryStatusPending,
)
cfg := newHTTPTargetConfig(ts.URL)
fix := buildHTTPFixture(
dlv, event, targetID,
"test-http-single", ts.URL, 1, 1,
"test-http-single", cfg, 1, 1,
)
sched := &recordingScheduler{}
@@ -605,9 +624,10 @@ func TestDeliverHTTP_SchedulesRetryOnFailure(
database.DeliveryStatusPending,
)
cfg := newHTTPTargetConfig(ts.URL)
fix := buildHTTPFixture(
dlv, event, targetID,
"test-http-schedule", ts.URL, 5, 1,
"test-http-schedule", cfg, 5, 1,
)
e.ExportDeliverHTTP(
@@ -769,7 +789,7 @@ func seedConcurrencyTasks(
t *testing.T,
db *gorm.DB,
n int,
targetURL string,
targetCfg string,
) ([]database.Delivery, []delivery.Task) {
t.Helper()
@@ -795,7 +815,7 @@ func seedConcurrencyTasks(
Target: database.Target{
Name: fmt.Sprintf("task-%d", i),
Type: database.TargetTypeHTTP,
URL: targetURL,
Config: targetCfg,
MaxAttempts: 1,
},
}
@@ -807,7 +827,7 @@ func seedConcurrencyTasks(
TargetID: tasks[i].TargetID,
TargetName: tasks[i].Target.Name,
TargetType: tasks[i].Target.Type,
TargetURL: tasks[i].Target.URL,
TargetConfig: tasks[i].Target.Config,
MaxAttempts: 1,
AttemptNum: 1,
}
@@ -845,8 +865,9 @@ func TestWorkerPool_BoundedConcurrency(t *testing.T) {
defer ts.Close()
e := testEngine(t, numWorkers)
cfg := newHTTPTargetConfig(ts.URL)
tasks, dtasks := seedConcurrencyTasks(
t, db, numTasks, ts.URL,
t, db, numTasks, cfg,
)
runConcurrencyWorkers(
@@ -927,9 +948,12 @@ func TestDeliverHTTP_CircuitBreakerBlocks(t *testing.T) {
database.DeliveryStatusPending,
)
cfg := newHTTPTargetConfig(
"http://will-not-be-called.invalid",
)
fix := buildHTTPFixture(
dlv, event, targetID,
"test-cb-block", "http://will-not-be-called.invalid", 5, 1,
"test-cb-block", cfg, 5, 1,
)
e.ExportDeliverHTTP(
@@ -992,8 +1016,9 @@ func TestDeliverHTTP_HalfOpenBreakerDelaysQueuedTasks(t *testing.T) {
require.True(t, cb.Allow(), "the probe delivery should go through")
require.Equal(t, delivery.CircuitHalfOpen, cb.State())
const targetURL = "http://will-not-be-called.invalid"
cfg := newHTTPTargetConfig(
"http://will-not-be-called.invalid",
)
sched := &recordingScheduler{}
const queued, passes = 3, 4
@@ -1015,7 +1040,7 @@ func TestDeliverHTTP_HalfOpenBreakerDelaysQueuedTasks(t *testing.T) {
fix := buildHTTPFixture(
row, event, targetID,
"test-cb-half-open", targetURL, 5, 1,
"test-cb-half-open", cfg, 5, 1,
)
e.ExportDeliverHTTPWithScheduler(
@@ -1130,6 +1155,62 @@ func TestStateAndCooldown_ReadsHTTPAndSlackBreakers(t *testing.T) {
assert.Zero(t, cooldown, "half-open")
}
func TestParseHTTPConfig_Valid(t *testing.T) {
t.Parallel()
e := testEngine(t, 1)
cfg, err := e.ExportParseHTTPConfig(
`{"url":"https://example.com/hook",` +
`"headers":{"X-Token":"secret"}}`,
)
require.NoError(t, err)
assert.Equal(t, "https://example.com/hook", cfg.URL)
assert.Equal(t, "secret", cfg.Headers["X-Token"])
}
func TestParseHTTPConfig_Empty(t *testing.T) {
t.Parallel()
e := testEngine(t, 1)
_, err := e.ExportParseHTTPConfig("")
assert.Error(t, err,
"empty config should return error",
)
}
func TestParseHTTPConfig_MissingURL(t *testing.T) {
t.Parallel()
e := testEngine(t, 1)
_, err := e.ExportParseHTTPConfig(
`{"headers":{"X-Token":"secret"}}`,
)
assert.Error(t, err,
"config without URL should return error",
)
}
func TestParseHTTPConfig_Undecodable(t *testing.T) {
t.Parallel()
e := testEngine(t, 1)
_, err := e.ExportParseHTTPConfig(
`{"url":"https://example.com/hook","timeout":"soon"}`,
)
assert.Error(t, err,
"config that does not decode should return error, "+
"even when the part that did names a URL",
)
}
func TestScheduleRetry_SendsToRetryChannel(
t *testing.T,
) {
@@ -1277,7 +1358,7 @@ func TestDoHTTPRequest_ForwardsHeaders(t *testing.T) {
e := testEngine(t, 1)
target := &database.Target{
cfg := &delivery.HTTPTargetConfig{
URL: ts.URL,
Headers: map[string]string{
"X-Target-Auth": "bearer xyz",
@@ -1292,7 +1373,7 @@ func TestDoHTTPRequest_ForwardsHeaders(t *testing.T) {
}
statusCode, _, _, err := e.ExportDoHTTPRequest(
context.TODO(), target, event,
context.TODO(), cfg, event,
)
require.NoError(t, err)
@@ -1334,7 +1415,7 @@ func TestDoHTTPRequest_CutShortResponseIsAnError(t *testing.T) {
_, body, _, err := e.ExportDoHTTPRequest(
context.TODO(),
&database.Target{URL: ts.URL},
&delivery.HTTPTargetConfig{URL: ts.URL},
&database.Event{},
)
@@ -1386,9 +1467,9 @@ func TestApplyRequestHeaders_SendsOneContentType(t *testing.T) {
})
require.NoError(t, err)
var configured map[string]string
cfg := &delivery.HTTPTargetConfig{}
if tc.configured != "" {
configured = map[string]string{
cfg.Headers = map[string]string{
headerContentType: tc.configured,
}
}
@@ -1407,7 +1488,7 @@ func TestApplyRequestHeaders_SendsOneContentType(t *testing.T) {
Headers: string(inbound),
ContentType: tc.event,
},
configured,
cfg,
"webhooker/dev",
)
@@ -1438,7 +1519,7 @@ func TestApplyRequestHeaders_UndecodableInboundForwardsNothing(
&database.Event{
Headers: `{"X-Custom":["value1"],"X-Broken":"not a list"}`,
},
nil,
&delivery.HTTPTargetConfig{},
"webhooker/dev",
)
@@ -1456,7 +1537,7 @@ func TestProcessDelivery_RoutesToCorrectHandler(
// The database target archives for real, so the engine needs
// the target in the main database and a data directory.
env := setupArchiveTest(t)
archive := env.seedDatabaseTarget(t)
archive := env.seedDatabaseTarget(t, "")
tests := []struct {
name string
@@ -1541,6 +1622,51 @@ func TestMaxInlineBodySize_Constant(t *testing.T) {
)
}
func TestParseSlackConfig_Valid(t *testing.T) {
t.Parallel()
e := testEngine(t, 1)
cfg, err := e.ExportParseSlackConfig(
`{"webhookUrl":"https://hooks.slack.com/services/T00/B00/xxx"}`,
)
require.NoError(t, err)
assert.Equal(t,
"https://hooks.slack.com/services/T00/B00/xxx",
cfg.WebhookURL,
)
}
func TestParseSlackConfig_Empty(t *testing.T) {
t.Parallel()
e := testEngine(t, 1)
_, err := e.ExportParseSlackConfig("")
assert.Error(t, err,
"empty config should return error",
)
}
func TestParseSlackConfig_MissingWebhookURL(
t *testing.T,
) {
t.Parallel()
e := testEngine(t, 1)
_, err := e.ExportParseSlackConfig(
`{"other":"field"}`,
)
assert.Error(t, err,
"config without webhook_url should return error",
)
}
func TestFormatSlackMessage_JSONBody(t *testing.T) {
t.Parallel()
@@ -1637,11 +1763,11 @@ func TestFormatSlackMessage_LargeJSONTruncated(
}
// buildSlackDelivery creates a database.Delivery
// to a Slack target with the given webhook URL for testing.
// with a Slack target config for testing.
func buildSlackDelivery(
dlv database.Delivery,
event database.Event,
targetID, name, webhookURL string,
targetID, name, slackCfg string,
) *database.Delivery {
d := &database.Delivery{
EventID: event.ID,
@@ -1651,7 +1777,7 @@ func buildSlackDelivery(
Target: database.Target{
Name: name,
Type: database.TargetTypeSlack,
URL: webhookURL,
Config: slackCfg,
MaxAttempts: 1,
},
}
@@ -1689,10 +1815,11 @@ func TestDeliverSlack_Success(t *testing.T) {
)
defer ts.Close()
e, targetID, event, dlv := setupSlackTest(t, db)
e, targetID, slackCfg, event, dlv :=
setupSlackTest(t, db, ts.URL)
d := buildSlackDelivery(
dlv, event, targetID, "test-slack", ts.URL,
dlv, event, targetID, "test-slack", slackCfg,
)
e.ExportDeliverSlack(context.TODO(), db, d)
@@ -1711,8 +1838,9 @@ func TestDeliverSlack_Success(t *testing.T) {
func setupSlackTest(
t *testing.T,
db *gorm.DB,
serverURL string,
) (
*delivery.Engine, string,
*delivery.Engine, string, string,
database.Event, database.Delivery,
) {
t.Helper()
@@ -1720,6 +1848,13 @@ func setupSlackTest(
e := testEngine(t, 1)
targetID := uuid.New().String()
cfgBytes, err := json.Marshal(
delivery.SlackTargetConfig{
WebhookURL: serverURL,
},
)
require.NoError(t, err)
event := seedEvent(
t, db, `{"action":"test","data":"value"}`,
)
@@ -1729,7 +1864,7 @@ func setupSlackTest(
database.DeliveryStatusPending,
)
return e, targetID, event, dlv
return e, targetID, string(cfgBytes), event, dlv
}
func assertSlackPayload(
@@ -1778,6 +1913,13 @@ func TestDeliverSlack_Failure(t *testing.T) {
e := testEngine(t, 1)
targetID := uuid.New().String()
slackCfg, err := json.Marshal(
delivery.SlackTargetConfig{
WebhookURL: ts.URL,
},
)
require.NoError(t, err)
event := seedEvent(t, db, `{"test":true}`)
dlv := seedDelivery(
@@ -1787,7 +1929,7 @@ func TestDeliverSlack_Failure(t *testing.T) {
d := buildSlackDelivery(
dlv, event, targetID,
"test-slack-fail", ts.URL,
"test-slack-fail", string(slackCfg),
)
e.ExportDeliverSlack(context.TODO(), db, d)
@@ -1801,10 +1943,7 @@ func TestDeliverSlack_Failure(t *testing.T) {
)
}
// TestDeliverSlack_NoURL proves a Slack target without a URL, which
// the forms never store, fails its delivery rather than reporting it
// delivered.
func TestDeliverSlack_NoURL(t *testing.T) {
func TestDeliverSlack_InvalidConfig(t *testing.T) {
t.Parallel()
db := testWebhookDB(t)
@@ -1825,6 +1964,7 @@ func TestDeliverSlack_NoURL(t *testing.T) {
Target: database.Target{
Name: "test-slack-bad",
Type: database.TargetTypeSlack,
Config: `{"not_webhook_url":"missing"}`,
MaxAttempts: 1,
},
}
@@ -1862,6 +2002,13 @@ func TestProcessDelivery_RoutesToSlack(t *testing.T) {
e := testEngine(t, 1)
slackCfg, err := json.Marshal(
delivery.SlackTargetConfig{
WebhookURL: ts.URL,
},
)
require.NoError(t, err)
event := seedEvent(t, db, `{"route":"slack"}`)
dlv := seedDelivery(
@@ -1871,7 +2018,7 @@ func TestProcessDelivery_RoutesToSlack(t *testing.T) {
d := buildSlackDelivery(
dlv, event, dlv.TargetID,
"test-slack-route", ts.URL,
"test-slack-route", string(slackCfg),
)
task := &delivery.Task{
@@ -1986,7 +2133,7 @@ func TestDeliverLog_LogsFullContent(t *testing.T) {
func buildSlackRetryDelivery(
dlv database.Delivery,
event database.Event,
targetID, webhookURL string,
targetID, cfg string,
) *database.Delivery {
d := &database.Delivery{
EventID: event.ID,
@@ -1996,7 +2143,7 @@ func buildSlackRetryDelivery(
Target: database.Target{
Name: "test-slack-retry",
Type: database.TargetTypeSlack,
URL: webhookURL,
Config: cfg,
MaxAttempts: 5,
},
}
@@ -2015,6 +2162,11 @@ func TestDeliverSlack_WithRetries_SchedulesRetry(
e := testEngine(t, 1)
targetID := uuid.New().String()
slackCfg, err := json.Marshal(
delivery.SlackTargetConfig{WebhookURL: ts.URL},
)
require.NoError(t, err)
event := seedEvent(t, db, `{"slack":"retry"}`)
dlv := seedDelivery(
@@ -2023,7 +2175,7 @@ func TestDeliverSlack_WithRetries_SchedulesRetry(
)
d := buildSlackRetryDelivery(
dlv, event, targetID, ts.URL,
dlv, event, targetID, string(slackCfg),
)
task := &delivery.Task{
internal/delivery/event_db_isolation_test.go
+6 -4
View File
@@ -69,6 +69,7 @@ func TestEventDBHoldsNoTargetRows(t *testing.T) {
))
defer ts.Close()
cfg := iHTTPConfig(ts.URL)
targetID := uuid.New().String()
dbPath := s.DBMgr.DBPath(s.WebhookID)
@@ -84,7 +85,7 @@ func TestEventDBHoldsNoTargetRows(t *testing.T) {
)
task := iTask(
d, event, s.WebhookID, targetID,
"leaky-target", ts.URL, 5, 1, &body,
"leaky-target", cfg, 5, 1, &body,
)
s.Engine.ExportProcessNewTask(context.TODO(), &task)
@@ -101,7 +102,7 @@ func TestEventDBHoldsNoTargetRows(t *testing.T) {
// https://git.eeqj.de/sneak/webhooker/issues/107.
iCreateTarget(
t, s.MainDB, targetID, s.WebhookID, "leaky-target",
database.TargetTypeHTTP, ts.URL, 5,
database.TargetTypeHTTP, cfg, 5,
)
rd := iSeedDelivery(
@@ -110,7 +111,7 @@ func TestEventDBHoldsNoTargetRows(t *testing.T) {
)
rTask := iTask(
rd, event, s.WebhookID, targetID,
"leaky-target", ts.URL, 5, 2, &body,
"leaky-target", cfg, 5, 2, &body,
)
s.Engine.ExportProcessRetryTask(context.TODO(), &rTask)
@@ -137,6 +138,7 @@ func TestEventDBHoldsNoTargetRowsOnFailedDelivery(t *testing.T) {
))
defer ts.Close()
cfg := iHTTPConfig(ts.URL)
targetID := uuid.New().String()
event := iSeedEvent(
@@ -150,7 +152,7 @@ func TestEventDBHoldsNoTargetRowsOnFailedDelivery(t *testing.T) {
)
task := iTask(
d, event, s.WebhookID, targetID,
"leaky-target", ts.URL, 1, 1, &body,
"leaky-target", cfg, 1, 1, &body,
)
s.Engine.ExportProcessNewTask(context.TODO(), &task)
internal/delivery/event_timestamp_test.go
+20 -6
View File
@@ -93,6 +93,17 @@ func (s *tsSink) text(t *testing.T) string {
}
}
func tsSlackConfig(t *testing.T, url string) string {
t.Helper()
data, err := json.Marshal(
delivery.SlackTargetConfig{WebhookURL: url},
)
require.NoError(t, err)
return string(data)
}
// tsSeedEvent writes an event whose CreatedAt is tsEventCreatedAt
// rather than the write time, so an assertion on the rendered
// timestamp cannot pass by accident against "roughly now".
@@ -131,7 +142,7 @@ func tsSeedEvent(
// tsSeedTarget writes the slack target row into the main database.
// The retry path confirms the target still exists before sending.
func tsSeedTarget(
t *testing.T, mainDB *gorm.DB, webhookID, webhookURL string,
t *testing.T, mainDB *gorm.DB, webhookID, config string,
) database.Target {
t.Helper()
@@ -139,7 +150,7 @@ func tsSeedTarget(
WebhookID: webhookID,
Name: "slack-sink",
Type: database.TargetTypeSlack,
URL: webhookURL,
Config: config,
Active: true,
MaxAttempts: 1,
}
@@ -165,7 +176,7 @@ func tsTask(
TargetID: target.ID,
TargetName: target.Name,
TargetType: database.TargetTypeSlack,
TargetURL: target.URL,
TargetConfig: target.Config,
MaxAttempts: 1,
Method: event.Method,
Headers: event.Headers,
@@ -215,7 +226,8 @@ func (c tsCase) run(t *testing.T) (iSetup, database.Delivery, string) {
s := newISetup(t)
sink := newTSSink(t)
target := tsSeedTarget(t, s.MainDB, s.WebhookID, sink.URL)
cfg := tsSlackConfig(t, sink.URL)
target := tsSeedTarget(t, s.MainDB, s.WebhookID, cfg)
event := tsSeedEvent(t, s.WebhookDB, s.WebhookID)
d := iSeedDelivery(
@@ -326,7 +338,8 @@ func TestFormatSlackMessageOverTaskReconstructedEvent(
s := newISetup(t)
target := tsSeedTarget(t, s.MainDB, s.WebhookID, tsUndeliverableHook)
cfg := tsSlackConfig(t, tsUndeliverableHook)
target := tsSeedTarget(t, s.MainDB, s.WebhookID, cfg)
event := tsSeedEvent(t, s.WebhookDB, s.WebhookID)
d := iSeedDelivery(
@@ -397,7 +410,8 @@ func TestEventReconstructionSurvivesAReapedRow(t *testing.T) {
s := newISetup(t)
target := tsSeedTarget(t, s.MainDB, s.WebhookID, tsUndeliverableHook)
cfg := tsSlackConfig(t, tsUndeliverableHook)
target := tsSeedTarget(t, s.MainDB, s.WebhookID, cfg)
event := tsSeedEvent(t, s.WebhookDB, s.WebhookID)
d := iSeedDelivery(
internal/delivery/export_test.go
+20 -11
View File
@@ -77,10 +77,10 @@ func ExportIsForwardableHeader(name string) bool {
func ExportApplyRequestHeaders(
req *http.Request,
event *database.Event,
configured map[string]string,
cfg *HTTPTargetConfig,
userAgent string,
) []string {
return applyRequestHeaders(req, event, configured, userAgent)
return applyRequestHeaders(req, event, cfg, userAgent)
}
// ExportTruncate exposes truncate for testing.
@@ -158,11 +158,6 @@ func (e *Engine) ExportUpdateDeliveryStatus(
return e.updateDeliveryStatus(webhookDB, d, "", status)
}
// ExportNotify exposes notify.
func (e *Engine) ExportNotify(tasks []Task) {
e.notify(tasks)
}
// ExportProcessNewTask exposes processNewTask.
func (e *Engine) ExportProcessNewTask(
ctx context.Context, task *Task,
@@ -221,23 +216,37 @@ func (e *Engine) ExportSetSlackCircuitBreaker(
e.slackTarget.circuitBreakers.Store(targetID, cb)
}
// ExportParseHTTPConfig exposes parseHTTPConfig.
func (e *Engine) ExportParseHTTPConfig(
configJSON string,
) (*HTTPTargetConfig, error) {
return parseHTTPConfig(configJSON)
}
// ExportParseSlackConfig exposes parseSlackConfig.
func (e *Engine) ExportParseSlackConfig(
configJSON string,
) (*SlackTargetConfig, error) {
return parseSlackConfig(configJSON)
}
// ExportDoHTTPRequest exposes the http target's
// doHTTPRequest.
func (e *Engine) ExportDoHTTPRequest(
ctx context.Context,
target *database.Target,
cfg *HTTPTargetConfig,
event *database.Event,
) (int, string, int64, error) {
return e.httpTarget.doHTTPRequest(ctx, target, event)
return e.httpTarget.doHTTPRequest(ctx, cfg, event)
}
// ExportClientForRequest exposes the http target's
// clientForRequest.
func (e *Engine) ExportClientForRequest(
timeoutSeconds int,
cfg *HTTPTargetConfig,
originScoped []string,
) *http.Client {
return e.httpTarget.clientForRequest(timeoutSeconds, originScoped)
return e.httpTarget.clientForRequest(cfg, originScoped)
}
// ErrExportTooManyRedirects exposes the sentinel the redirect
internal/delivery/inflight.go
+1 -1
View File
@@ -61,7 +61,7 @@ func (s *inflightSet) retain(deliveryID string) {
// retainIdle takes the first reference on a delivery, and reports
// whether it got it. It fails when the engine already owns the
// delivery, which is what makes two claimants — notify and the
// delivery, which is what makes two claimants — Notify and the
// recovery routine, say — mutually exclusive rather than merely
// atomic.
func (s *inflightSet) retainIdle(deliveryID string) bool {
internal/delivery/inflight_test.go
+9 -9
View File
@@ -84,9 +84,9 @@ func TestSweepLeavesAQueuedDeliveryAlone(t *testing.T) {
)
rAgePending(t, s.WebhookDB, d.ID)
// Queued exactly as StoreEvent queues it, and never dequeued:
// Queued exactly as the receiver queues it, and never dequeued:
// no workers are running in this engine.
s.Engine.ExportNotify([]delivery.Task{{
s.Engine.Notify([]delivery.Task{{
DeliveryID: d.ID,
EventID: event.ID,
WebhookID: s.WebhookID,
@@ -109,7 +109,7 @@ func TestSweepLeavesAQueuedDeliveryAlone(t *testing.T) {
// TestRecoveryAndSweepDoNotDoubleDispatch drives the recovery routine's
// run at start and a tick of the sweep at once against one aged pending
// row. The engine runs them one after the other, but notify can race
// row. The engine runs them one after the other, but Notify can race
// either, and ownership is what keeps any two claimants from both
// sending. Before ownership the two runs both dispatched the row.
func TestRecoveryAndSweepDoNotDoubleDispatch(t *testing.T) {
@@ -224,7 +224,7 @@ func TestOwnershipIsReleasedAfterDelivery(t *testing.T) {
body := `{"released":true}`
s.Engine.ExportNotify([]delivery.Task{{
s.Engine.Notify([]delivery.Task{{
DeliveryID: d.ID,
EventID: event.ID,
WebhookID: s.WebhookID,
@@ -248,10 +248,10 @@ func TestOwnershipIsReleasedAfterDelivery(t *testing.T) {
}
// TestNotifyAfterRecoveryDoesNotSendAgain is the startup race of
// https://git.eeqj.de/sneak/webhooker/issues/299. StoreEvent has
// https://git.eeqj.de/sneak/webhooker/issues/299. The receiver has
// written a delivery, the recovery routine's run at start finds it
// pending, sends it and releases it, and only then does StoreEvent's
// notify for it arrive. Nothing owns the delivery by then, so notify
// pending, sends it and releases it, and only then does the receiver's
// Notify for it arrive. Nothing owns the delivery by then, so Notify
// takes it.
func TestNotifyAfterRecoveryDoesNotSendAgain(t *testing.T) {
t.Parallel()
@@ -289,7 +289,7 @@ func TestNotifyAfterRecoveryDoesNotSendAgain(t *testing.T) {
body := event.Body
s.Engine.ExportNotify([]delivery.Task{{
s.Engine.Notify([]delivery.Task{{
DeliveryID: d.ID,
EventID: event.ID,
WebhookID: s.WebhookID,
@@ -300,7 +300,7 @@ func TestNotifyAfterRecoveryDoesNotSendAgain(t *testing.T) {
EntrypointID: event.EntrypointID,
}})
// notify took the delivery, and a worker releases it once it has
// Notify took the delivery, and a worker releases it once it has
// run the task.
require.Eventually(
t,
internal/delivery/metrics_test.go
+15 -12
View File
@@ -192,7 +192,7 @@ func mDeliverOK(t *testing.T, s iSetup) {
body := event.Body
task := iTask(
d, event, s.WebhookID, targetID,
"metrics-ok", ts.URL, 3, 1, &body,
"metrics-ok", iHTTPConfig(ts.URL), 3, 1, &body,
)
s.Engine.ExportProcessNewTask(context.TODO(), &task)
@@ -226,17 +226,18 @@ func mExhaustRetries(t *testing.T, s iSetup) {
)
body := event.Body
cfg := iHTTPConfig(ts.URL)
// The retry below is only run if its target still exists; see
// https://git.eeqj.de/sneak/webhooker/issues/107.
iCreateTarget(
t, s.MainDB, targetID, s.WebhookID, "metrics-fail",
database.TargetTypeHTTP, ts.URL, 2,
database.TargetTypeHTTP, cfg, 2,
)
first := iTask(
d, event, s.WebhookID, targetID,
"metrics-fail", ts.URL, 2, 1, &body,
"metrics-fail", cfg, 2, 1, &body,
)
s.Engine.ExportProcessNewTask(context.TODO(), &first)
@@ -250,7 +251,7 @@ func mExhaustRetries(t *testing.T, s iSetup) {
// deterministic and off the wall clock.
second := iTask(
d, event, s.WebhookID, targetID,
"metrics-fail", ts.URL, 2, 2, &body,
"metrics-fail", cfg, 2, 2, &body,
)
s.Engine.ExportProcessRetryTask(
@@ -288,6 +289,7 @@ func TestDeliveryMetrics_CircuitBreakerGauge(t *testing.T) {
)
body := event.Body
cfg := iHTTPConfig(ts.URL)
// An attempt budget above the failure threshold, so the breaker
// rather than the budget is what stops the delivery.
@@ -297,12 +299,12 @@ func TestDeliveryMetrics_CircuitBreakerGauge(t *testing.T) {
// see https://git.eeqj.de/sneak/webhooker/issues/107.
iCreateTarget(
t, s.MainDB, targetID, s.WebhookID, "metrics-trip",
database.TargetTypeHTTP, ts.URL, maxAttempts,
database.TargetTypeHTTP, cfg, maxAttempts,
)
first := iTask(
d, event, s.WebhookID, targetID,
"metrics-trip", ts.URL, maxAttempts, 1, &body,
"metrics-trip", cfg, maxAttempts, 1, &body,
)
s.Engine.ExportProcessNewTask(context.TODO(), &first)
@@ -314,7 +316,7 @@ func TestDeliveryMetrics_CircuitBreakerGauge(t *testing.T) {
ExportDefaultFailureThreshold; attempt++ {
task := iTask(
d, event, s.WebhookID, targetID,
"metrics-trip", ts.URL, maxAttempts, attempt, &body,
"metrics-trip", cfg, maxAttempts, attempt, &body,
)
s.Engine.ExportProcessRetryTask(
@@ -361,16 +363,17 @@ func TestDeliveryMetrics_BreakerBlockedIsNotAnAttempt(
)
body := event.Body
cfg := iHTTPConfig(ts.URL)
maxAttempts := delivery.ExportDefaultFailureThreshold + 5
iCreateTarget(
t, s.MainDB, targetID, s.WebhookID, "metrics-blocked",
database.TargetTypeHTTP, ts.URL, maxAttempts,
database.TargetTypeHTTP, cfg, maxAttempts,
)
first := iTask(
d, event, s.WebhookID, targetID,
"metrics-blocked", ts.URL, maxAttempts, 1, &body,
"metrics-blocked", cfg, maxAttempts, 1, &body,
)
s.Engine.ExportProcessNewTask(context.TODO(), &first)
@@ -379,7 +382,7 @@ func TestDeliveryMetrics_BreakerBlockedIsNotAnAttempt(
ExportDefaultFailureThreshold; attempt++ {
task := iTask(
d, event, s.WebhookID, targetID,
"metrics-blocked", ts.URL, maxAttempts, attempt, &body,
"metrics-blocked", cfg, maxAttempts, attempt, &body,
)
s.Engine.ExportProcessRetryTask(context.TODO(), &task)
@@ -402,7 +405,7 @@ func TestDeliveryMetrics_BreakerBlockedIsNotAnAttempt(
blocked := iTask(
d, event, s.WebhookID, targetID,
"metrics-blocked", ts.URL, maxAttempts,
"metrics-blocked", cfg, maxAttempts,
delivery.ExportDefaultFailureThreshold+1, &body,
)
@@ -471,7 +474,7 @@ func TestDeliveryMetrics_QueueDepthGauges(t *testing.T) {
iCreateTarget(t, s.MainDB, targetID, s.WebhookID,
"queue-depth-target", database.TargetTypeHTTP,
"https://example.com/hook", 3,
iHTTPConfig("https://example.com/hook"), 3,
)
event := iSeedEvent(
internal/delivery/queue_depth_gormlog_test.go
+1 -1
View File
@@ -102,7 +102,7 @@ func TestQueueDepthSample_LogsNoBoundValue(t *testing.T) {
iCreateTarget(t, mainDB.DB(), targetID, webhookID,
"queue-depth-gormlog-target", database.TargetTypeHTTP,
"https://example.com/hook", 3,
iHTTPConfig("https://example.com/hook"), 3,
)
event := iSeedEvent(
internal/delivery/recovery_durability_test.go
+2 -2
View File
@@ -357,7 +357,7 @@ func TestFailedResultWriteLeavesDeliveryRecoverable(
Target: database.Target{
Name: "unwritable",
Type: database.TargetTypeHTTP,
URL: ts.URL,
Config: iHTTPConfig(ts.URL),
MaxAttempts: 1,
},
}
@@ -448,7 +448,7 @@ func TestFailedResultWriteWithRetriesLeavesDeliveryRecoverable(
Target: database.Target{
Name: "unwritable",
Type: database.TargetTypeHTTP,
URL: ts.URL,
Config: iHTTPConfig(ts.URL),
MaxAttempts: 3,
},
}
internal/delivery/recovery_test.go
+1 -50
View File
@@ -56,7 +56,7 @@ func recoverySetup(t *testing.T) (iSetup, string) {
iCreateTarget(
t, s.MainDB, targetID, s.WebhookID, "recovery",
database.TargetTypeHTTP,
"http://example.com/hook", 20,
iHTTPConfig("http://example.com/hook"), 20,
)
return s, targetID
@@ -205,55 +205,6 @@ func TestRecovery_PendingOlderThanTheAgeBound(t *testing.T) {
}
}
// TestRecovery_TaskCarriesEveryTargetSetting: the task the recovery
// routine queues carries every one of the target's settings, all six
// set here.
// They are copied into the task one field at a time, and a field left
// out is sent empty without any error.
func TestRecovery_TaskCarriesEveryTargetSetting(t *testing.T) {
t.Parallel()
for _, run := range recoveryRuns() {
t.Run(run.name, func(t *testing.T) {
t.Parallel()
s := newISetup(t)
iCreateWebhook(t, s.MainDB, s.WebhookID, "recovery")
target := database.Target{
WebhookID: s.WebhookID,
Name: "recovery",
Type: database.TargetTypeHTTP,
Active: true,
URL: "http://example.com/hook",
Headers: map[string]string{headerAuthorization: "Bearer t"},
TimeoutSeconds: 7,
ForwardQuery: true,
ArchiveExpiry: time.Hour,
ArchiveRotation: "daily",
MaxAttempts: 1,
}
require.NoError(t, s.MainDB.Create(&target).Error)
id := seedRecoveryDelivery(
t, s, target.ID, database.DeliveryStatusPending, 0, false,
)
rAgePending(t, s.WebhookDB, id)
s.Engine.ExportRecoverDeliveries(context.Background(), run.atStart)
task := requireQueued(t, s.Engine.ExportDeliveryCh())
assert.Equal(t, target.ID, task.TargetID)
assert.Equal(t, target.URL, task.TargetURL)
assert.Equal(t, target.Headers, task.TargetHeaders)
assert.Equal(t, target.TimeoutSeconds, task.TimeoutSeconds)
assert.Equal(t, target.ForwardQuery, task.ForwardQuery)
assert.Equal(t, target.ArchiveExpiry, task.ArchiveExpiry)
assert.Equal(t, target.ArchiveRotation, task.ArchiveRotation)
})
}
}
// TestRecovery_PendingYoungerThanTheAgeBound: a pending delivery touched
// a moment ago, as one a stop cut off is, is sent at start, which is how
// it reaches its target on the next start
internal/delivery/redirect_test.go
+15 -7
View File
@@ -85,14 +85,19 @@ func deliverWithProbeHeaders(
database.DeliveryStatusPending,
)
cfg, err := json.Marshal(delivery.HTTPTargetConfig{
URL: targetURL,
Headers: map[string]string{
probeHeaderName: probeHeaderValue,
},
})
require.NoError(t, err)
body := event.Body
task := iTask(
d, event, s.WebhookID, targetID,
"redirect-target", targetURL, 1, 1, &body,
"redirect-target", string(cfg), 1, 1, &body,
)
task.TargetHeaders = map[string]string{
probeHeaderName: probeHeaderValue,
}
s.Engine.ExportProcessNewTask(context.TODO(), &task)
@@ -304,7 +309,8 @@ func TestRedirectPolicy_StopsAtHopCap(t *testing.T) {
)
client := engine.ExportClientForRequest(
0, []string{probeHeaderName},
&delivery.HTTPTargetConfig{URL: srv.URL},
[]string{probeHeaderName},
)
require.NotNil(t, client.CheckRedirect)
@@ -344,7 +350,7 @@ func TestApplyRequestHeaders_ReportsOriginScopedNames(t *testing.T) {
inbound, err := json.Marshal(map[string][]string{
inboundHeaderName: {inboundHeaderValue},
"Content-Type": {testContentType},
headerUserAgent: {"curl/8.7.1"},
"User-Agent": {"curl/8.7.1"},
"Host": {"inbound.example.com"},
})
require.NoError(t, err)
@@ -363,10 +369,12 @@ func TestApplyRequestHeaders_ReportsOriginScopedNames(t *testing.T) {
Headers: string(inbound),
ContentType: testContentType,
},
map[string]string{
&delivery.HTTPTargetConfig{
Headers: map[string]string{
probeHeaderName: probeHeaderValue,
"Content-Type": testContentType,
},
},
"webhooker/dev",
)
internal/delivery/store.go
-164
View File
@@ -1,164 +0,0 @@
package delivery
import (
"fmt"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
)
// StoreEvent stores event with one pending delivery for each of its
// webhook's active targets, and adds them to the totals, in one
// transaction, then queues the deliveries. It returns how many
// deliveries it created.
//
// The receiver stores every event it accepts through here, and a
// resubmit stores its copy of a stored event, so every event is
// stored, retried, SSRF-guarded and circuit-broken the same way however
// it entered the system. CreateReplay is the only other place a
// delivery is created.
//
// The targets are the webhook's active ones as they stand now: a
// target created after a resubmitted event first arrived is included,
// and an inactive one is skipped. An event with no active target is
// stored with no deliveries.
func (e *Engine) StoreEvent(event *database.Event) (int, error) {
var targets []database.Target
err := e.database.DB().Where(
"webhook_id = ? AND active = ?", event.WebhookID, true,
).Find(&targets).Error
if err != nil {
return 0, fmt.Errorf("loading active targets: %w", err)
}
webhookDB, err := e.dbManager.GetDB(event.WebhookID)
if err != nil {
return 0, fmt.Errorf("getting webhook database: %w", err)
}
event.BodyBytes = int64(len(event.Body))
tasks := make([]Task, 0, len(targets))
err = webhookDB.Transaction(func(tx *gorm.DB) error {
err := tx.Create(event).Error
if err != nil {
return fmt.Errorf("creating event: %w", err)
}
for i := range targets {
dlv, err := createDelivery(tx, event.ID, targets[i].ID, false)
if err != nil {
return err
}
tasks = append(tasks, buildTask(
dlv, event.WebhookID, event, &targets[i], 1,
))
}
err = database.AddEventTotals(tx, database.EventTotals{
Events: 1, LastEventAt: &event.CreatedAt,
})
if err != nil {
return err
}
// A resubmitted copy did not arrive on its entrypoint's URL, so
// it leaves the entrypoint's last event as it is.
if event.ResubmittedFromID != nil {
return nil
}
return database.AddEntrypointTotals(tx, database.EntrypointTotals{
EntrypointID: event.EntrypointID, LastEventAt: event.CreatedAt,
})
})
if err != nil {
return 0, err
}
// Counted here, after the commit: an event exists once it is
// durably stored, which is what the delivery counters are compared
// against on a dashboard. A resubmitted event counts too, because
// it produces deliveries that the delivery side counts; the
// resubmit counter is what separates the two.
e.mtr.EventReceived()
e.notify(tasks)
return len(tasks), nil
}
// CreateReplay creates one more pending delivery of the stored event
// eventID to target, marked as a replay, adds it to the target's totals
// in the same transaction, and queues it. It returns the new
// delivery's id.
//
// The caller passes the target as it stands now and has already
// decided the replay may go ahead. The delivery being replayed is not
// touched.
func (e *Engine) CreateReplay(
webhookID, eventID string, target *database.Target,
) (string, error) {
webhookDB, err := e.dbManager.GetDB(webhookID)
if err != nil {
return "", fmt.Errorf("getting webhook database: %w", err)
}
event, err := e.loadEvent(webhookDB, eventID)
if err != nil {
return "", err
}
var dlv *database.Delivery
err = webhookDB.Transaction(func(tx *gorm.DB) error {
var err error
dlv, err = createDelivery(tx, event.ID, target.ID, true)
return err
})
if err != nil {
return "", err
}
e.notify([]Task{buildTask(dlv, webhookID, &event, target, 1)})
return dlv.ID, nil
}
// createDelivery writes a pending delivery of the event to the target
// on the caller's transaction, and adds it to the target's totals.
//
// Neither Event nor Target is set on the row, and the per-webhook
// database omits associations on every write in any case (see
// internal/database/event_db_isolation.go), so the target's settings
// are never copied into the event database.
func createDelivery(
tx *gorm.DB, eventID, targetID string, replay bool,
) (*database.Delivery, error) {
dlv := &database.Delivery{
EventID: eventID,
TargetID: targetID,
Status: database.DeliveryStatusPending,
Replay: replay,
}
err := tx.Create(dlv).Error
if err != nil {
return nil, fmt.Errorf(
"creating delivery for target %s: %w", targetID, err,
)
}
err = database.AddTargetTotals(tx, database.TargetTotals{
TargetID: targetID, Deliveries: 1,
})
if err != nil {
return nil, err
}
return dlv, nil
}
internal/delivery/store_test.go
-386
View File
@@ -1,386 +0,0 @@
package delivery_test
import (
"net/http"
"strings"
"testing"
"time"
"github.com/google/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
)
// storeTestTarget creates an active http target of the setup's webhook
// with every one of a target's settings set, so a task that drops one
// of them differs from the target.
func storeTestTarget(
t *testing.T, s iSetup, name string,
) database.Target {
t.Helper()
target := database.Target{
WebhookID: s.WebhookID,
Name: name,
Type: database.TargetTypeHTTP,
Active: true,
URL: "http://93.184.216.34/" + name,
Headers: map[string]string{"X-Api-Key": "k"},
TimeoutSeconds: 7,
ForwardQuery: true,
ArchiveExpiry: time.Hour,
ArchiveRotation: rotationDaily,
MaxAttempts: 3,
}
require.NoError(t, s.MainDB.Create(&target).Error)
return target
}
// newStoreTestEvent returns an event of the setup's webhook as the
// receiver builds one, not yet stored.
func newStoreTestEvent(s iSetup, body string) *database.Event {
return &database.Event{
WebhookID: s.WebhookID,
EntrypointID: uuid.New().String(),
Method: http.MethodPost,
RawQuery: "a=1&b=2",
Headers: `{"X-Test":["yes"]}`,
ContentType: testContentType,
Body: body,
}
}
// queuedTasks takes every task the engine has queued so far. The
// engine in these tests is never started, so nothing else takes them.
func queuedTasks(e *delivery.Engine) []delivery.Task {
var tasks []delivery.Task
for {
select {
case task := <-e.ExportDeliveryCh():
tasks = append(tasks, task)
default:
return tasks
}
}
}
// assertTaskFor proves task carries delivery deliveryID of event to
// target: every one of the target's settings and every event field.
// They are copied into the task one field at a time, and a field left
// out is sent empty without any error.
func assertTaskFor(
t *testing.T,
task delivery.Task,
deliveryID string,
event *database.Event,
target *database.Target,
) {
t.Helper()
assert.Equal(t, deliveryID, task.DeliveryID)
assert.Equal(t, event.ID, task.EventID)
assert.Equal(t, event.WebhookID, task.WebhookID)
assert.Equal(t, event.EntrypointID, task.EntrypointID)
assert.Equal(t, target.ID, task.TargetID)
assert.Equal(t, target.Name, task.TargetName)
assert.Equal(t, target.Type, task.TargetType)
assert.Equal(t, target.URL, task.TargetURL)
assert.Equal(t, target.Headers, task.TargetHeaders)
assert.Equal(t, target.TimeoutSeconds, task.TimeoutSeconds)
assert.Equal(t, target.ForwardQuery, task.ForwardQuery)
assert.Equal(t, target.ArchiveExpiry, task.ArchiveExpiry)
assert.Equal(t, target.ArchiveRotation, task.ArchiveRotation)
assert.Equal(t, target.MaxAttempts, task.MaxAttempts)
assert.Equal(t, event.Method, task.Method)
assert.Equal(t, event.RawQuery, task.RawQuery)
assert.Equal(t, event.Headers, task.Headers)
assert.Equal(t, event.ContentType, task.ContentType)
require.NotNil(t, task.Body)
assert.Equal(t, event.Body, *task.Body)
assert.Equal(t, 1, task.AttemptNum)
}
// storedDeliveries reads every delivery of an event.
func storedDeliveries(
t *testing.T, db *gorm.DB, eventID string,
) []database.Delivery {
t.Helper()
var deliveries []database.Delivery
require.NoError(t, db.Where("event_id = ?", eventID).
Find(&deliveries).Error)
return deliveries
}
// storedEventTotals reads the webhook's event totals.
func storedEventTotals(t *testing.T, db *gorm.DB) database.EventTotals {
t.Helper()
var totals database.EventTotals
require.NoError(t, db.First(&totals).Error)
return totals
}
// countRows counts the rows of model in db.
func countRows(t *testing.T, db *gorm.DB, model any) int64 {
t.Helper()
var n int64
require.NoError(t, db.Model(model).Count(&n).Error)
return n
}
// TestStoreEvent_StoresDeliveriesToActiveTargets proves StoreEvent
// stores the event with one pending delivery to each active target and
// none to an inactive one, adds them to the totals, and queues one task
// per delivery.
func TestStoreEvent_StoresDeliveriesToActiveTargets(t *testing.T) {
t.Parallel()
s := newISetup(t)
targets := []database.Target{
storeTestTarget(t, s, "first"),
storeTestTarget(t, s, "second"),
}
off := storeTestTarget(t, s, "off")
require.NoError(t, s.MainDB.Model(&database.Target{}).
Where("id = ?", off.ID).Update("active", false).Error)
event := newStoreTestEvent(s, `{"hello":"world"}`)
created, err := s.Engine.StoreEvent(event)
require.NoError(t, err)
assert.Equal(t, 2, created)
var stored database.Event
require.NoError(t, s.WebhookDB.First(&stored, "id = ?", event.ID).Error)
assert.Equal(t, event.Body, stored.Body)
assert.Equal(t, int64(len(event.Body)), stored.BodyBytes)
rows := storedDeliveries(t, s.WebhookDB, event.ID)
queued := queuedTasks(s.Engine)
require.Len(t, rows, 2)
require.Len(t, queued, 2)
deliveries := map[string]database.Delivery{}
for _, dlv := range rows {
deliveries[dlv.TargetID] = dlv
}
tasks := map[string]delivery.Task{}
for _, task := range queued {
tasks[task.TargetID] = task
}
for i := range targets {
dlv, ok := deliveries[targets[i].ID]
require.True(t, ok, "no delivery to %s", targets[i].Name)
assert.Equal(t, database.DeliveryStatusPending, dlv.Status)
assert.False(t, dlv.Replay)
assert.Equal(t, int64(1),
targetTotals(t, s.WebhookDB, targets[i].ID).Deliveries)
assertTaskFor(t, tasks[targets[i].ID], dlv.ID, event, &targets[i])
}
totals := storedEventTotals(t, s.WebhookDB)
assert.Equal(t, int64(1), totals.Events)
require.NotNil(t, totals.LastEventAt)
assert.True(t, totals.LastEventAt.Equal(stored.CreatedAt))
var entrypoint database.EntrypointTotals
require.NoError(t, s.WebhookDB.First(
&entrypoint, "entrypoint_id = ?", event.EntrypointID,
).Error)
assert.True(t, entrypoint.LastEventAt.Equal(stored.CreatedAt))
}
// TestStoreEvent_ResubmittedCopyLeavesTheEntrypoint proves a
// resubmitted copy is counted among the webhook's events but leaves its
// entrypoint's last event as it is: it did not arrive on that URL.
func TestStoreEvent_ResubmittedCopyLeavesTheEntrypoint(t *testing.T) {
t.Parallel()
s := newISetup(t)
storeTestTarget(t, s, "only")
sourceID := uuid.New().String()
event := newStoreTestEvent(s, `{"copy":true}`)
event.ResubmittedFromID = &sourceID
created, err := s.Engine.StoreEvent(event)
require.NoError(t, err)
assert.Equal(t, 1, created)
assert.Len(t, queuedTasks(s.Engine), 1)
assert.Equal(t, int64(1), storedEventTotals(t, s.WebhookDB).Events)
assert.Zero(t, countRows(t, s.WebhookDB, &database.EntrypointTotals{}))
}
// TestStoreEvent_NoActiveTarget proves an event of a webhook with no
// active target is stored with no delivery and nothing queued.
func TestStoreEvent_NoActiveTarget(t *testing.T) {
t.Parallel()
s := newISetup(t)
event := newStoreTestEvent(s, `{"to":"nobody"}`)
created, err := s.Engine.StoreEvent(event)
require.NoError(t, err)
assert.Zero(t, created)
assert.Equal(t, int64(1), countRows(t, s.WebhookDB, &database.Event{}))
assert.Empty(t, storedDeliveries(t, s.WebhookDB, event.ID))
assert.Empty(t, queuedTasks(s.Engine))
}
// TestStoreEvent_LargeBodyIsNotCarried proves a body at the inline
// limit is stored whole but left off the task, for the attempt to read
// from the event row.
func TestStoreEvent_LargeBodyIsNotCarried(t *testing.T) {
t.Parallel()
s := newISetup(t)
storeTestTarget(t, s, "only")
event := newStoreTestEvent(
s, strings.Repeat("x", delivery.MaxInlineBodySize),
)
_, err := s.Engine.StoreEvent(event)
require.NoError(t, err)
tasks := queuedTasks(s.Engine)
require.Len(t, tasks, 1)
assert.Nil(t, tasks[0].Body)
var stored database.Event
require.NoError(t, s.WebhookDB.First(&stored, "id = ?", event.ID).Error)
assert.Equal(t, event.Body, stored.Body)
}
// TestStoreEvent_RollsBackWhenAWriteFails proves the event, its
// deliveries and its totals are one transaction: when the entrypoint
// totals write fails, the last one, after the event, its delivery, the
// target totals and the event totals were written, none of them is left
// and nothing is queued.
func TestStoreEvent_RollsBackWhenAWriteFails(t *testing.T) {
t.Parallel()
s := newISetup(t)
storeTestTarget(t, s, "only")
require.NoError(t,
s.WebhookDB.Migrator().DropTable(&database.EntrypointTotals{}))
_, err := s.Engine.StoreEvent(newStoreTestEvent(s, `{"lost":true}`))
require.Error(t, err)
assert.Zero(t, countRows(t, s.WebhookDB, &database.Event{}))
assert.Zero(t, countRows(t, s.WebhookDB, &database.Delivery{}))
assert.Zero(t, countRows(t, s.WebhookDB, &database.TargetTotals{}))
totals := storedEventTotals(t, s.WebhookDB)
assert.Zero(t, totals.Events)
assert.Nil(t, totals.LastEventAt)
assert.Empty(t, queuedTasks(s.Engine))
assert.Zero(t, s.Engine.ExportInflightHeld())
}
// TestCreateReplay_CreatesAndQueuesAReplay proves CreateReplay adds a
// new pending delivery marked as a replay, counts it in the target's
// totals, queues its task built from the target it was given and the
// stored event, and leaves the delivery it repeats as it was. The
// stored event has every event field set, so a task that drops one of
// them differs from the event.
func TestCreateReplay_CreatesAndQueuesAReplay(t *testing.T) {
t.Parallel()
s := newISetup(t)
target := storeTestTarget(t, s, "replayed")
event := newStoreTestEvent(s, `{"replay":"me"}`)
require.NoError(t, s.WebhookDB.Create(event).Error)
original := iSeedDelivery(
t, s.WebhookDB, event.ID, target.ID,
database.DeliveryStatusFailed,
)
deliveryID, err := s.Engine.CreateReplay(s.WebhookID, event.ID, &target)
require.NoError(t, err)
replay := storedDelivery(t, s.WebhookDB, deliveryID)
assert.Equal(t, event.ID, replay.EventID)
assert.Equal(t, target.ID, replay.TargetID)
assert.Equal(t, database.DeliveryStatusPending, replay.Status)
assert.True(t, replay.Replay)
assert.Equal(t, database.DeliveryStatusFailed,
storedDelivery(t, s.WebhookDB, original.ID).Status)
assert.Equal(t, int64(1),
targetTotals(t, s.WebhookDB, target.ID).Deliveries)
tasks := queuedTasks(s.Engine)
require.Len(t, tasks, 1)
assertTaskFor(t, tasks[0], deliveryID, event, &target)
}
// TestCreateReplay_RollsBackWhenAWriteFails proves the replay delivery
// and its target totals are one transaction: when the totals write
// fails, after the delivery was written, no new delivery is left and
// nothing is queued.
func TestCreateReplay_RollsBackWhenAWriteFails(t *testing.T) {
t.Parallel()
s := newISetup(t)
target := storeTestTarget(t, s, "replayed")
event := iSeedEvent(t, s.WebhookDB, s.WebhookID, `{"replay":"me"}`)
iSeedDelivery(
t, s.WebhookDB, event.ID, target.ID,
database.DeliveryStatusFailed,
)
require.NoError(t,
s.WebhookDB.Migrator().DropTable(&database.TargetTotals{}))
_, err := s.Engine.CreateReplay(s.WebhookID, event.ID, &target)
require.Error(t, err)
assert.Len(t, storedDeliveries(t, s.WebhookDB, event.ID), 1)
assert.Empty(t, queuedTasks(s.Engine))
assert.Zero(t, s.Engine.ExportInflightHeld())
}
// storedDelivery reads one delivery back.
func storedDelivery(
t *testing.T, db *gorm.DB, deliveryID string,
) database.Delivery {
t.Helper()
var dlv database.Delivery
require.NoError(t, db.First(&dlv, "id = ?", deliveryID).Error)
return dlv
}
internal/delivery/target_config_edit.go
+137
View File
@@ -0,0 +1,137 @@
package delivery
import (
"encoding/json"
"errors"
"fmt"
"sneak.berlin/go/webhooker/internal/archive"
"sneak.berlin/go/webhooker/internal/database"
)
// errUnknownTargetTypeForEdit is returned when a stored target has a
// type the edit form has no field set for.
var errUnknownTargetTypeForEdit = errors.New(
"unknown target type",
)
// TargetConfigForm is the UNMASKED projection of a target's stored
// configuration, for pre-filling the target edit form.
//
// It is the deliberate exception to the rule TargetView enforces
// everywhere else: TargetView exists so that no template can render
// a target's stored blob, because a destination URL's path segments
// and a header value are both routinely the credential. An operator
// cannot correct a value they cannot see, so the edit form — and
// only the edit form — is shown the full value.
//
// Everything that keeps that exception narrow lives at the call
// site: the route is behind RequireAuth and the webhook's ownership
// check, and its group sets NoCache so the rendered secret is not
// written to a shared cache. Do not reach for this type from any
// other page.
type TargetConfigForm struct {
// URL is the destination for an HTTP target and the webhook
// URL for a Slack target.
URL string
// Headers is the HTTP target's configured headers in the
// textarea representation, one "Name: value" per line.
Headers string
// Timeout is the HTTP target's per-request timeout in seconds,
// empty when unset.
Timeout string
// ForwardQuery is the HTTP target's setting that passes each
// event's query string on to it.
ForwardQuery bool
// Expiry is the database (archive) target's row expiry.
Expiry string
// Rotation is the database (archive) target's rotation.
Rotation string
}
// NewTargetConfigForm parses a target's stored configuration into
// the edit form's fields.
//
// A configuration that does not parse is an error rather than a
// zero-valued form that silently looks like a target with no
// settings. The caller shows the operator that the stored value
// could not be read, so that saving the form is understood as
// replacing it rather than preserving it.
func NewTargetConfigForm(
t *database.Target,
) (TargetConfigForm, error) {
switch t.Type {
case database.TargetTypeHTTP:
cfg, err := parseHTTPConfig(t.Config)
if err != nil {
return TargetConfigForm{}, err
}
return TargetConfigForm{
URL: cfg.URL,
Headers: FormatTargetHeaders(cfg.Headers),
Timeout: FormatTargetTimeout(cfg.Timeout),
ForwardQuery: cfg.ForwardQuery,
}, nil
case database.TargetTypeSlack:
cfg, err := parseSlackConfig(t.Config)
if err != nil {
return TargetConfigForm{}, err
}
return TargetConfigForm{URL: cfg.WebhookURL}, nil
case database.TargetTypeDatabase:
return databaseConfigForm(t.Config)
case database.TargetTypeLog:
// The log target takes no configuration.
return TargetConfigForm{}, nil
default:
return TargetConfigForm{}, fmt.Errorf(
"%w: %q", errUnknownTargetTypeForEdit, t.Type,
)
}
}
// databaseConfigForm parses an archive target's optional expiry and
// rotation. An absent, empty or never expiry yields an empty expiry,
// on which the edit form starts at never; saving it unchanged stores
// never, which means the same as an empty expiry. An absent rotation
// is empty too, and the form starts at none. An expiry that is set
// but not a valid duration, or a rotation that is not one of the
// four, is an error, not a blank field.
func databaseConfigForm(
configJSON string,
) (TargetConfigForm, error) {
if configJSON == "" {
return TargetConfigForm{}, nil
}
var cfg archive.DatabaseTargetConfig
err := json.Unmarshal([]byte(configJSON), &cfg)
if err != nil {
return TargetConfigForm{}, fmt.Errorf(
"parsing config JSON: %w", err,
)
}
err = archive.ValidateRotation(cfg.Rotation)
if err != nil {
return TargetConfigForm{}, err
}
form := TargetConfigForm{Rotation: cfg.Rotation}
if cfg.Expiry == "" || cfg.Expiry == archive.ExpiryNever {
return form, nil
}
err = archive.ValidateExpiry(cfg.Expiry)
if err != nil {
return TargetConfigForm{}, err
}
form.Expiry = cfg.Expiry
return form, nil
}
internal/delivery/target_config_view.go
+71 -30
View File
@@ -9,17 +9,16 @@ import (
"sneak.berlin/go/webhooker/internal/database"
)
// configUnavailable is what a target's settings render as when
// the target is of an unknown type, and what its URL renders as
// when it does not parse into a scheme and host. The stored URL is
// never shown as a fallback: it can be a credential (a Slack
// incoming webhook URL is a bearer token) and a UI that prints it
// leaks that credential into browser history, screenshots and
// screen shares.
// configUnavailable is what a target's configuration renders
// as when it is absent, of an unknown type, or does not
// parse. The stored blob is never shown as a fallback: it can
// hold a credential (a Slack incoming webhook URL is a bearer
// token) and a UI that prints it leaks that credential into
// browser history, screenshots and screen shares.
const configUnavailable = "(unavailable)"
// ConfigField is one labelled, display-safe value derived
// from a target's settings.
// from a target's stored configuration.
type ConfigField struct {
Label string
Value string
@@ -33,9 +32,9 @@ type ConfigField struct {
const deletedNameSuffix = " (deleted)"
// TargetView is the display-safe projection of a target for
// the UI. It deliberately has no URL or header field, so no
// template — present or future — can render the stored
// credential.
// the UI. It deliberately has no raw configuration field, so
// no template — present or future — can render the stored
// blob.
type TargetView struct {
ID string
Name string
@@ -62,7 +61,8 @@ func (v TargetView) DisplayName() string {
}
// NewTargetViews projects targets for rendering, replacing
// each target's settings with named, display-safe fields.
// each stored configuration blob with named, display-safe
// fields.
//
// A soft-deleted row projects exactly as a live one does, minus
// the deleted marker on its name: masking is a property of the
@@ -91,8 +91,8 @@ func NewTargetViews(
}
// targetConfigFields returns the display-safe fields for a
// target's settings. A target of an unknown type gets the
// neutral placeholder.
// target's configuration. Anything it cannot parse becomes
// the neutral placeholder.
func targetConfigFields(
t *database.Target,
) []ConfigField {
@@ -102,25 +102,36 @@ func targetConfigFields(
case database.TargetTypeHTTP:
return httpConfigFields(t)
case database.TargetTypeDatabase:
return databaseConfigFields(t)
return databaseConfigFields(t.Config)
case database.TargetTypeLog:
// The log target takes no settings.
// The log target takes no configuration.
return nil
default:
return unavailableConfigFields()
}
}
// unavailableConfigFields is the neutral placeholder shown
// for a configuration that could not be presented.
func unavailableConfigFields() []ConfigField {
return []ConfigField{{
Label: "Configuration",
Value: configUnavailable,
}}
}
}
// slackConfigFields describes a Slack target: its masked
// webhook URL and its number of delivery attempts. Only the
// masked URL is shown; the full URL is the credential.
func slackConfigFields(t *database.Target) []ConfigField {
cfg, err := parseSlackConfig(t.Config)
if err != nil {
return unavailableConfigFields()
}
return []ConfigField{{
Label: "Webhook URL",
Value: MaskURL(t.URL),
Value: cfg.MaskedWebhookURL(),
}, maxAttemptsField(t)}
}
@@ -135,28 +146,33 @@ func slackConfigFields(t *database.Target) []ConfigField {
// whose path segments are the credential, and the field takes
// an arbitrary URL, so no segment can be assumed non-secret.
func httpConfigFields(t *database.Target) []ConfigField {
cfg, err := parseHTTPConfig(t.Config)
if err != nil {
return unavailableConfigFields()
}
fields := []ConfigField{{
Label: "Destination URL",
Value: MaskURL(t.URL),
Value: MaskURL(cfg.URL),
}}
if t.TimeoutSeconds > 0 {
if cfg.Timeout > 0 {
fields = append(fields, ConfigField{
Label: "Timeout",
Value: strconv.Itoa(t.TimeoutSeconds) + "s",
Value: strconv.Itoa(cfg.Timeout) + "s",
})
}
if len(t.Headers) > 0 {
if len(cfg.Headers) > 0 {
fields = append(fields, ConfigField{
Label: "Headers",
Value: fmt.Sprintf(
"%d configured", len(t.Headers),
"%d configured", len(cfg.Headers),
),
})
}
if t.ForwardQuery {
if cfg.ForwardQuery {
fields = append(fields, ConfigField{
Label: "Query string",
Value: "passed on to this target",
@@ -169,7 +185,8 @@ func httpConfigFields(t *database.Target) []ConfigField {
}
// maxAttemptsField describes a target's number of delivery
// attempts.
// attempts, which lives on the target row rather than in its
// configuration blob.
func maxAttemptsField(t *database.Target) ConfigField {
return ConfigField{
Label: "Delivery attempts",
@@ -179,11 +196,24 @@ func maxAttemptsField(t *database.Target) ConfigField {
// databaseConfigFields describes an archive target by its
// expiry in plain units, such as "30 days", or "never" when
// the archive is kept forever, and by its rotation.
func databaseConfigFields(t *database.Target) []ConfigField {
// the archive is kept forever, and by its rotation. An expiry
// that is set but not a valid duration, or a rotation that is
// not one of the four, is reported as unavailable rather than
// echoed back.
func databaseConfigFields(configJSON string) []ConfigField {
expiry, err := archive.ParseExpiry(configJSON)
if err != nil {
return unavailableConfigFields()
}
rotation, err := archive.ParseRotation(configJSON)
if err != nil {
return unavailableConfigFields()
}
value := archive.ExpiryNever
if t.ArchiveExpiry > 0 {
value = plainDuration(t.ArchiveExpiry)
if expiry > 0 {
value = plainDuration(expiry)
}
return []ConfigField{{
@@ -191,7 +221,7 @@ func databaseConfigFields(t *database.Target) []ConfigField {
Value: value,
}, {
Label: "Archive rotation",
Value: t.ArchiveRotation,
Value: rotation,
}}
}
@@ -227,3 +257,14 @@ func plainDuration(d time.Duration) string {
return d.String()
}
// MaskedWebhookURL returns the Slack webhook URL reduced to
// its scheme and host, with the path, query and any userinfo
// elided. The path segments are the credential, so none of
// them is shown: the field accepts an arbitrary URL, so no
// segment can be assumed non-secret. A URL that does not
// parse into a scheme and host yields the neutral
// placeholder, never the raw string.
func (c *SlackTargetConfig) MaskedWebhookURL() string {
return MaskURL(c.WebhookURL)
}
internal/delivery/target_config_view_test.go
+107 -43
View File
@@ -35,17 +35,18 @@ const (
viewMaxAttempts = "Delivery attempts"
)
// The archive rotations a database target's settings can name.
// The archive rotations a database target's settings can name, and
// the config of one whose rotation is not one of the four.
const (
rotationNone = "none"
rotationMonthly = "monthly"
rotationDaily = "daily"
rotationHourly = "hourly"
weeklyConfig = `{"rotation":"weekly"}`
)
// TestNewTargetViews_SlackMasksWebhookURL shows what a Slack target's
// URL renders as on the target list.
func TestNewTargetViews_SlackMasksWebhookURL(t *testing.T) {
func TestMaskedWebhookURL(t *testing.T) {
t.Parallel()
tests := map[string]struct {
@@ -89,24 +90,28 @@ func TestNewTargetViews_SlackMasksWebhookURL(t *testing.T) {
t.Run(name, func(t *testing.T) {
t.Parallel()
target := slackTarget()
target.URL = tc.url
cfg := &delivery.SlackTargetConfig{
WebhookURL: tc.url,
}
assert.Equal(
t, tc.want,
fieldMap(viewFor(t, target).Config)["Webhook URL"],
t, tc.want, cfg.MaskedWebhookURL(),
)
})
}
}
// TestNewTargetViews_SlackNeverLeaksPath is the direct
// TestMaskedWebhookURL_NeverLeaksPath is the direct
// expression of the rule: whatever the input, the masked
// value never contains a path segment of it.
func TestNewTargetViews_SlackNeverLeaksPath(t *testing.T) {
func TestMaskedWebhookURL_NeverLeaksPath(t *testing.T) {
t.Parallel()
masked := fieldMap(viewFor(t, slackTarget()).Config)["Webhook URL"]
cfg := &delivery.SlackTargetConfig{
WebhookURL: slackWebhookURL,
}
masked := cfg.MaskedWebhookURL()
assert.NotContains(t, masked, "T00000000")
assert.NotContains(t, masked, "B00000000")
@@ -175,7 +180,8 @@ func slackTarget() database.Target {
Name: slackTargetName,
Type: database.TargetTypeSlack,
Active: true,
URL: slackWebhookURL,
Config: `{"webhookUrl":"` +
slackWebhookURL + `"}`,
MaxAttempts: 1,
}
}
@@ -228,10 +234,10 @@ func TestNewTargetViews_HTTP(t *testing.T) {
view := viewFor(t, database.Target{
Type: database.TargetTypeHTTP,
URL: viewExampleHook,
TimeoutSeconds: 30,
Headers: map[string]string{headerAuthorization: "Bearer sekrit"},
ForwardQuery: true,
Config: `{"url":"` + viewExampleHook + `",` +
`"timeout":30,` +
`"headers":{"Authorization":"Bearer sekrit"},` +
`"forwardQuery":true}`,
MaxAttempts: 5,
})
@@ -262,7 +268,7 @@ func TestNewTargetViews_HTTPSingleAttempt(t *testing.T) {
view := viewFor(t, database.Target{
Type: database.TargetTypeHTTP,
URL: viewExampleHook,
Config: `{"url":"` + viewExampleHook + `"}`,
MaxAttempts: 1,
})
@@ -285,7 +291,7 @@ func TestNewTargetViews_HTTPMasksDestinationURL(t *testing.T) {
view := viewFor(t, database.Target{
Type: database.TargetTypeHTTP,
URL: slackWebhookURL,
Config: `{"url":"` + slackWebhookURL + `"}`,
MaxAttempts: 1,
})
@@ -309,20 +315,25 @@ func TestNewTargetViews_Database(t *testing.T) {
t.Parallel()
tests := map[string]struct {
expiry time.Duration
config string
want string
}{
"never": {expiry: 0, want: viewExpiryNever},
"1h": {expiry: time.Hour, want: "1 hour"},
"12h": {expiry: 12 * time.Hour, want: "12 hours"},
"24h": {expiry: 24 * time.Hour, want: "1 day"},
"720h": {expiry: 720 * time.Hour, want: "30 days"},
"2160h": {expiry: 2160 * time.Hour, want: "90 days"},
"8760h": {expiry: 8760 * time.Hour, want: "365 days"},
"36h": {expiry: 36 * time.Hour, want: "36 hours"},
"1h30m": {expiry: 90 * time.Minute, want: "90 minutes"},
"45s": {expiry: 45 * time.Second, want: "45 seconds"},
"1.5s": {expiry: 1500 * time.Millisecond, want: "1.5s"},
"empty config": {config: "", want: viewExpiryNever},
"empty expiry": {config: `{}`, want: viewExpiryNever},
"never literal": {
config: `{"expiry":"` + viewExpiryNever + `"}`,
want: viewExpiryNever,
},
"1h": {config: `{"expiry":"1h"}`, want: "1 hour"},
"12h": {config: `{"expiry":"12h"}`, want: "12 hours"},
"24h": {config: `{"expiry":"24h"}`, want: "1 day"},
"720h": {config: `{"expiry":"720h"}`, want: "30 days"},
"2160h": {config: `{"expiry":"2160h"}`, want: "90 days"},
"8760h": {config: `{"expiry":"8760h"}`, want: "365 days"},
"36h": {config: `{"expiry":"36h"}`, want: "36 hours"},
"1h30m": {config: `{"expiry":"1h30m"}`, want: "90 minutes"},
"45s": {config: `{"expiry":"45s"}`, want: "45 seconds"},
"1.5s": {config: `{"expiry":"1.5s"}`, want: "1.5s"},
}
for name, tc := range tests {
@@ -331,8 +342,7 @@ func TestNewTargetViews_Database(t *testing.T) {
view := viewFor(t, database.Target{
Type: database.TargetTypeDatabase,
ArchiveExpiry: tc.expiry,
ArchiveRotation: rotationNone,
Config: tc.config,
})
assert.Equal(
@@ -348,23 +358,33 @@ func TestNewTargetViews_Database(t *testing.T) {
}
// TestNewTargetViews_DatabaseRotation proves the target list shows a
// database target's rotation.
// database target's rotation, none when it has none stored.
func TestNewTargetViews_DatabaseRotation(t *testing.T) {
t.Parallel()
// Each stored config, and the rotation the list shows for it.
tests := map[string]string{
"": rotationNone,
`{"rotation":""}`: rotationNone,
}
for _, rotation := range []string{
rotationNone, rotationMonthly, rotationDaily, rotationHourly,
} {
t.Run(rotation, func(t *testing.T) {
tests[`{"rotation":"`+rotation+`"}`] = rotation
}
for config, want := range tests {
t.Run(config, func(t *testing.T) {
t.Parallel()
view := viewFor(t, database.Target{
Type: database.TargetTypeDatabase,
ArchiveRotation: rotation,
Config: config,
})
assert.Equal(
t, rotation, fieldMap(view.Config)["Archive rotation"],
t, want, fieldMap(view.Config)["Archive rotation"],
)
})
}
@@ -375,21 +395,63 @@ func TestNewTargetViews_Log(t *testing.T) {
view := viewFor(t, database.Target{
Type: database.TargetTypeLog,
Config: "",
})
assert.Empty(t, view.Config)
}
// TestNewTargetViews_UnknownType proves that a target of a type the
// view does not know shows the neutral placeholder, and never its
// URL.
func TestNewTargetViews_UnknownType(t *testing.T) {
// TestNewTargetViews_Unpresentable proves that no config the
// view cannot present falls back to the stored blob.
func TestNewTargetViews_Unpresentable(t *testing.T) {
t.Parallel()
view := viewFor(t, database.Target{
const blob = `{"webhookUrl":"https://hooks.slack.com` +
slackSecretPath + `"`
tests := map[string]database.Target{
"unknown target type": {
Type: database.TargetType("carrier-pigeon"),
URL: slackWebhookURL,
})
Config: blob,
},
"unparseable json": {
Type: database.TargetTypeSlack,
Config: blob,
MaxAttempts: 1,
},
"empty slack config": {
Type: database.TargetTypeSlack,
MaxAttempts: 1,
},
"slack config without url": {
Type: database.TargetTypeSlack,
Config: `{}`,
MaxAttempts: 1,
},
"unparseable http json": {
Type: database.TargetTypeHTTP,
Config: `{"url":`,
MaxAttempts: 1,
},
"unparseable archive json": {
Type: database.TargetTypeDatabase,
Config: `{"expiry":`,
},
"invalid archive expiry": {
Type: database.TargetTypeDatabase,
Config: `{"expiry":"a fortnight"}`,
},
"invalid archive rotation": {
Type: database.TargetTypeDatabase,
Config: weeklyConfig,
},
}
for name, target := range tests {
t.Run(name, func(t *testing.T) {
t.Parallel()
view := viewFor(t, target)
assert.Equal(
t,
@@ -398,4 +460,6 @@ func TestNewTargetViews_UnknownType(t *testing.T) {
},
fieldMap(view.Config),
)
})
}
}
internal/delivery/target_database_test.go
+12 -101
View File
@@ -1,7 +1,6 @@
package delivery_test
import (
"context"
"database/sql"
"fmt"
"log/slog"
@@ -21,7 +20,6 @@ import (
"sneak.berlin/go/webhooker/internal/archive"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/database/databasetest"
"sneak.berlin/go/webhooker/internal/db/migrations"
"sneak.berlin/go/webhooker/internal/delivery"
"sneak.berlin/go/webhooker/internal/gormlog"
)
@@ -80,11 +78,10 @@ func (env *archiveEnv) archivePath(tgt *database.Target) string {
)
}
// seedDatabaseTarget creates a webhook with one database target,
// which keeps its archive forever in one file, and returns the
// target.
// seedDatabaseTarget creates a webhook with one database target
// carrying the given target config JSON, and returns the target.
func (env *archiveEnv) seedDatabaseTarget(
t *testing.T,
t *testing.T, configJSON string,
) *database.Target {
t.Helper()
@@ -104,7 +101,7 @@ func (env *archiveEnv) seedDatabaseTarget(
Name: "Archive",
Type: database.TargetTypeDatabase,
Active: true,
ArchiveRotation: "none",
Config: configJSON,
MaxAttempts: 1,
}
require.NoError(
@@ -141,8 +138,7 @@ func seedDatabaseTargetDelivery(
Target: database.Target{
Name: tgt.Name,
Type: database.TargetTypeDatabase,
ArchiveExpiry: tgt.ArchiveExpiry,
ArchiveRotation: tgt.ArchiveRotation,
Config: tgt.Config,
},
}
d.ID = dlv.ID
@@ -222,7 +218,7 @@ func TestDeliverDatabase_ArchivesEvent(t *testing.T) {
t.Parallel()
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t)
tgt := env.seedDatabaseTarget(t, "")
webhookDB := testWebhookDB(t)
event := seedEvent(t, webhookDB, `{"archived":true}`)
@@ -260,9 +256,9 @@ func TestDeliverDatabase_ArchivesEvent(t *testing.T) {
}
// TestDeliverDatabase_ArchiveFailureFailsDelivery verifies that
// an archive error (here: an event with no webhook id to record)
// fails the delivery loudly: the attempt is recorded as failed
// with the error and the delivery is marked failed, not
// an archive error (here: an unparseable expiry in the target
// config) fails the delivery loudly: the attempt is recorded as
// failed with the error and the delivery is marked failed, not
// delivered.
func TestDeliverDatabase_ArchiveFailureFailsDelivery(
t *testing.T,
@@ -270,12 +266,11 @@ func TestDeliverDatabase_ArchiveFailureFailsDelivery(
t.Parallel()
env := setupArchiveTest(t)
tgt := env.seedDatabaseTarget(t)
tgt := env.seedDatabaseTarget(t, `{"expiry":"nonsense"}`)
webhookDB := testWebhookDB(t)
event := seedEvent(t, webhookDB, `{"archived":false}`)
d := seedDatabaseTargetDelivery(t, webhookDB, event, tgt)
d.Event.WebhookID = ""
env.eng.ExportDeliverDatabase(webhookDB, d)
@@ -300,96 +295,12 @@ func TestDeliverDatabase_ArchiveFailureFailsDelivery(
"the attempt must be recorded as failed",
)
assert.Contains(t,
results[0].Error, "without a webhook id",
results[0].Error, "nonsense",
"the archive error must be recorded on the attempt",
)
assert.NoFileExists(t,
env.archivePath(tgt),
"no archive file should exist for a failed write",
)
}
// seedArchivedRow creates the archive file at path holding one row, for
// eventID, archived at archivedAt.
func seedArchivedRow(
t *testing.T, path, eventID string, archivedAt time.Time,
) {
t.Helper()
sqlDB, err := sql.Open(
"sqlite", fmt.Sprintf("file:%s?mode=rwc", path),
)
require.NoError(t, err)
defer func() { _ = sqlDB.Close() }()
gdb, err := gorm.Open(
sqlite.Dialector{Conn: sqlDB},
&gorm.Config{Logger: gormlog.New(slog.New(slog.DiscardHandler))},
)
require.NoError(t, err)
require.NoError(t, migrations.Apply(gdb, migrations.Archive))
require.NoError(t, gdb.Exec(
"INSERT INTO archived_events (event_id, archived_at) VALUES (?, ?)",
eventID, archivedAt,
).Error)
}
// TestDeliverDatabase_TaskArchiveSettings proves a task's ArchiveExpiry
// and ArchiveRotation reach the archive through the engine. The
// target's row has neither, so only the task can bring them: with daily
// rotation the event goes to the file for its day, and opening that
// file with a one-hour expiry prunes the row it already held, archived
// two hours ago.
func TestDeliverDatabase_TaskArchiveSettings(t *testing.T) {
t.Parallel()
s := newISetup(t)
webhook := database.Webhook{UserID: uuid.NewString(), Name: "settings"}
webhook.ID = s.WebhookID
require.NoError(t, s.MainDB.Create(&webhook).Error)
target := database.Target{
WebhookID: s.WebhookID,
Name: "archive",
Type: database.TargetTypeDatabase,
Active: true,
MaxAttempts: 1,
}
require.NoError(t, s.MainDB.Create(&target).Error)
event := iSeedEvent(t, s.WebhookDB, s.WebhookID, `{"archived":true}`)
d := iSeedDelivery(
t, s.WebhookDB, event.ID, target.ID,
database.DeliveryStatusPending,
)
task := iTask(
d, event, s.WebhookID, target.ID, target.Name, "", 1, 1,
&event.Body,
)
task.TargetType = database.TargetTypeDatabase
task.ArchiveExpiry = time.Hour
task.ArchiveRotation = "daily"
daily := target
daily.ArchiveRotation = task.ArchiveRotation
dailyFile := archive.PathAt(s.DBMgr, &webhook, &daily, event.CreatedAt)
seedArchivedRow(t, dailyFile, "expired", time.Now().Add(-2*time.Hour))
s.Engine.ExportProcessNewTask(context.TODO(), &task)
iAssertStatus(t, s.WebhookDB, d.ID, database.DeliveryStatusDelivered)
var rows []archivedRow
require.NoError(t, openArchiveDBForRead(t, dailyFile).
Table("archived_events").Find(&rows).Error)
require.Len(t, rows, 1)
assert.Equal(t, event.ID, rows[0].EventID,
"the event goes to its day's file and the expired row is pruned",
"no archive file should exist for a failed config",
)
}
internal/delivery/target_headers.go
+9 -7
View File
@@ -17,7 +17,7 @@ import (
const MaxTargetTimeoutSeconds = 300
// Errors returned when a target's header or timeout form input
// cannot be turned into the target's setting.
// cannot be turned into a configuration.
//
// None of these ever quotes a header VALUE. A target header value
// is routinely an authorization token, and these messages are shown
@@ -48,8 +48,8 @@ var (
)
// isReservedTargetHeader reports whether name (canonicalised) is a
// header a target may not set, because the delivery path or
// net/http itself writes it regardless.
// header a target configuration may not set, because the delivery
// path or net/http itself writes it regardless.
//
// These are rejected rather than accepted-and-ignored. Storing a
// header that provably never reaches the wire tells the operator
@@ -77,11 +77,13 @@ func isReservedTargetHeader(name string) bool {
// ParseTargetHeaders turns the target form's headers field — one
// "Name: value" pair per line, blank lines ignored — into the map
// stored in database.Target.Headers. Names are canonicalised, so a
// stored in HTTPTargetConfig.Headers. Names are canonicalised, so a
// name repeated in a different case is still a duplicate rather than
// one pair silently overwriting the other.
//
// An input with no pairs yields an empty map.
// An input with no pairs yields an empty map, which omitempty drops
// from the stored config, so a target configured with no headers
// stores no headers key at all.
func ParseTargetHeaders(raw string) (map[string]string, error) {
headers := make(map[string]string)
@@ -221,8 +223,8 @@ func FormatTargetHeaders(headers map[string]string) string {
// ParseTargetTimeout interprets the target form's timeout field as
// a whole number of seconds. An empty field means "unset" and yields
// 0, which the delivery path reads as "use the shared client's
// timeout".
// 0, which omitempty drops from the stored config and which the
// delivery path reads as "use the shared client's timeout".
//
// Anything else that is not a whole number in range is an error, not
// a silently substituted default: a target whose timeout was typed
internal/delivery/target_headers_test.go
+99 -4
View File
@@ -1,21 +1,23 @@
package delivery_test
import (
"encoding/json"
"strconv"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
)
// Literals these tests repeat, named so that each header name has one
// definition.
// Literals these tests repeat, named so that the header names and the
// keep-forever archive config each have one definition.
const (
headerAuthorization = "Authorization"
headerContentType = "Content-Type"
headerUserAgent = "User-Agent"
bearerValue = "Bearer abc"
archiveConfigNever = "{\"expiry\":\"never\"}"
)
func TestParseTargetHeaders_AcceptsPairs(t *testing.T) {
@@ -39,13 +41,23 @@ func TestParseTargetHeaders_AcceptsPairs(t *testing.T) {
)
}
// An input of blank lines is no headers, not an error.
// A configuration with no headers must stay indistinguishable from
// one written before the field existed, so omitempty drops the key.
func TestParseTargetHeaders_EmptyInputYieldsNoHeaders(t *testing.T) {
t.Parallel()
got, err := delivery.ParseTargetHeaders("\n \n")
require.NoError(t, err)
assert.Empty(t, got)
encoded, err := json.Marshal(delivery.HTTPTargetConfig{
URL: "https://example.com/h",
Headers: got,
})
require.NoError(t, err)
assert.JSONEq(
t, `{"url":"https://example.com/h"}`, string(encoded),
)
}
func TestParseTargetHeaders_Rejects(t *testing.T) {
@@ -169,6 +181,89 @@ func TestFormatTargetTimeout(t *testing.T) {
assert.Empty(t, delivery.FormatTargetTimeout(-1))
}
func TestNewTargetConfigForm(t *testing.T) {
t.Parallel()
form, err := delivery.NewTargetConfigForm(&database.Target{
Type: database.TargetTypeHTTP,
Config: `{"url":"https://example.com/h",` +
`"headers":{"Authorization":"Bearer abc"},` +
`"timeout":9}`,
MaxAttempts: 1,
})
require.NoError(t, err)
assert.Equal(t, "https://example.com/h", form.URL)
assert.Equal(t, "Authorization: Bearer abc\n", form.Headers)
assert.Equal(t, "9", form.Timeout)
form, err = delivery.NewTargetConfigForm(&database.Target{
Type: database.TargetTypeSlack,
Config: `{"webhookUrl":"https://hooks.example/s"}`,
MaxAttempts: 1,
})
require.NoError(t, err)
assert.Equal(t, "https://hooks.example/s", form.URL)
form, err = delivery.NewTargetConfigForm(&database.Target{
Type: database.TargetTypeDatabase,
Config: `{"expiry":"720h","rotation":"daily"}`,
})
require.NoError(t, err)
assert.Equal(t, "720h", form.Expiry)
assert.Equal(t, rotationDaily, form.Rotation)
form, err = delivery.NewTargetConfigForm(&database.Target{
Type: database.TargetTypeLog,
})
require.NoError(t, err)
assert.Empty(t, form.URL)
}
// A keep-forever archive target yields an empty expiry, so the edit
// form starts on never; saving it unchanged stores never, which means
// the same as an empty expiry.
func TestNewTargetConfigForm_DatabaseNeverIsBlank(t *testing.T) {
t.Parallel()
for _, cfg := range []string{"", `{}`, archiveConfigNever} {
form, err := delivery.NewTargetConfigForm(
&database.Target{
Type: database.TargetTypeDatabase,
Config: cfg,
},
)
require.NoError(t, err, cfg)
assert.Empty(t, form.Expiry, cfg)
}
}
// An unreadable stored config is an error rather than a blank form
// that looks like a target with no settings, so the caller can tell
// the operator that saving replaces the stored value.
func TestNewTargetConfigForm_UnreadableConfigErrors(t *testing.T) {
t.Parallel()
cases := []*database.Target{
{Type: database.TargetTypeHTTP, Config: "not json", MaxAttempts: 1},
{Type: database.TargetTypeHTTP, Config: `{}`, MaxAttempts: 1},
{Type: database.TargetTypeSlack, Config: "", MaxAttempts: 1},
{
Type: database.TargetTypeDatabase,
Config: `{"expiry":"soon"}`,
},
{
Type: database.TargetTypeDatabase,
Config: weeklyConfig,
},
{Type: database.TargetType("nope")},
}
for _, target := range cases {
_, err := delivery.NewTargetConfigForm(target)
require.Error(t, err, target.Type)
}
}
// The ceiling exists so one misconfigured target cannot hold a
// delivery worker indefinitely, and it is inclusive.
func TestParseTargetTimeout_CeilingIsInclusive(t *testing.T) {
internal/delivery/target_http.go
+97 -23
View File
@@ -4,6 +4,7 @@ import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
@@ -16,6 +17,29 @@ import (
"sneak.berlin/go/webhooker/internal/database"
)
// Sentinel errors returned by the config parsers.
var (
errEmptyTargetConfig = errors.New(
"empty target config",
)
errMissingTargetURL = errors.New(
"target URL is required",
)
)
// HTTPTargetConfig holds configuration for http target
// types.
type HTTPTargetConfig struct {
URL string `json:"url"`
Headers map[string]string `json:"headers,omitempty"`
Timeout int `json:"timeout,omitempty"`
// ForwardQuery passes each event's query string on to the target,
// appended to URL. Off, the target URL is sent exactly as
// configured.
ForwardQuery bool `json:"forwardQuery,omitempty"`
}
// httpCore holds the retry, backoff, and circuit-breaker
// machinery shared by the HTTP and Slack targets. Each of
// those targets owns its own httpCore instance (and thus its
@@ -361,8 +385,34 @@ func (t *httpTarget) Deliver(
task *Task,
sched Scheduler,
) {
cfg, err := parseHTTPConfig(d.Target.Config)
if err != nil {
t.eng.log.Error(
"invalid HTTP target config",
"target_id", d.TargetID,
"error", err,
)
recErr := t.eng.recordResult(
webhookDB, d, task.AttemptNum,
false, 0, "", err.Error(), 0,
)
if recErr != nil {
t.eng.bookkeepingFailed(d, recErr)
return
}
t.eng.settleStatus(
webhookDB, d, d.Target.Type,
database.DeliveryStatusFailed,
)
return
}
attempt := func() attemptResult {
return t.attempt(ctx, &d.Target, &d.Event)
return t.attempt(ctx, cfg, &d.Event)
}
t.deliver(
@@ -377,11 +427,11 @@ func (t *httpTarget) Deliver(
// string; only a transport-level error does.
func (t *httpTarget) attempt(
ctx context.Context,
target *database.Target,
cfg *HTTPTargetConfig,
event *database.Event,
) attemptResult {
statusCode, respBody, duration, reqErr :=
t.doHTTPRequest(ctx, target, event)
t.doHTTPRequest(ctx, cfg, event)
success := reqErr == nil &&
statusCode >= httpSuccessMin &&
@@ -403,7 +453,7 @@ func (t *httpTarget) attempt(
func (t *httpTarget) doHTTPRequest(
ctx context.Context,
target *database.Target,
cfg *HTTPTargetConfig,
event *database.Event,
) (int, string, int64, error) {
start := time.Now()
@@ -411,7 +461,7 @@ func (t *httpTarget) doHTTPRequest(
req, reqErr := http.NewRequestWithContext(
ctx,
http.MethodPost,
target.URL,
cfg.URL,
bytes.NewReader([]byte(event.Body)),
)
if reqErr != nil {
@@ -421,15 +471,15 @@ func (t *httpTarget) doHTTPRequest(
)
}
if target.ForwardQuery {
if cfg.ForwardQuery {
appendQuery(req.URL, event.RawQuery)
}
originScoped := applyRequestHeaders(
req, event, target.Headers, t.eng.userAgent(),
req, event, cfg, t.eng.userAgent(),
)
client := t.clientForRequest(target.TimeoutSeconds, originScoped)
client := t.clientForRequest(cfg, originScoped)
resp, doErr := executeHTTPRequest(client, req)
@@ -469,17 +519,17 @@ func appendQuery(u *url.URL, rawQuery string) {
}
// clientForRequest returns the client for one delivery attempt.
// timeoutSeconds is the target's TimeoutSeconds, and originScoped is
// the header set applyRequestHeaders built for that attempt; a
// request with neither a per-target timeout nor an origin-scoped
// header gets the shared client, because there is then nothing for
// the redirect policy to strip and net/http's default policy already
// withholds Authorization and Cookie across hosts.
// originScoped is the header set applyRequestHeaders built for that
// attempt; a request with neither a per-target timeout nor an
// origin-scoped header gets the shared client, because there is
// then nothing for the redirect policy to strip and net/http's
// default policy already withholds Authorization and Cookie across
// hosts.
func (t *httpTarget) clientForRequest(
timeoutSeconds int,
cfg *HTTPTargetConfig,
originScoped []string,
) *http.Client {
if timeoutSeconds <= 0 && len(originScoped) == 0 {
if cfg.Timeout <= 0 && len(originScoped) == 0 {
return t.client
}
@@ -492,9 +542,9 @@ func (t *httpTarget) clientForRequest(
Transport: t.client.Transport,
}
if timeoutSeconds > 0 {
if cfg.Timeout > 0 {
client.Timeout = time.Duration(
timeoutSeconds,
cfg.Timeout,
) * time.Second
}
@@ -505,6 +555,31 @@ func (t *httpTarget) clientForRequest(
return client
}
func parseHTTPConfig(
configJSON string,
) (*HTTPTargetConfig, error) {
if configJSON == "" {
return nil, errEmptyTargetConfig
}
var cfg HTTPTargetConfig
err := json.Unmarshal(
[]byte(configJSON), &cfg,
)
if err != nil {
return nil, fmt.Errorf(
"parsing config JSON: %w", err,
)
}
if cfg.URL == "" {
return nil, errMissingTargetURL
}
return &cfg, nil
}
// isForwardableHeader returns true if the header should
// be forwarded to targets.
func isForwardableHeader(name string) bool {
@@ -536,12 +611,11 @@ func isForwardableHeader(name string) bool {
// wins, otherwise the event's ContentType, otherwise none. The inbound
// Content-Type in the event's headers is never forwarded.
//
// configured is the target's Headers. userAgent is set last, over
// any configured or inbound User-Agent.
// userAgent is set last, over any configured or inbound User-Agent.
func applyRequestHeaders(
req *http.Request,
event *database.Event,
configured map[string]string,
cfg *HTTPTargetConfig,
userAgent string,
) []string {
if event.ContentType != "" {
@@ -552,7 +626,7 @@ func applyRequestHeaders(
originScoped := forwardEventHeaders(req, event)
for k, v := range configured {
for k, v := range cfg.Headers {
req.Header.Set(k, v)
originScoped[http.CanonicalHeaderKey(k)] = struct{}{}
}
@@ -617,7 +691,7 @@ func forwardEventHeaders(
}
// executeHTTPRequest sends an HTTP request using the provided
// client. URLs are validated by the target forms and the
// client. URLs are validated by the config parsers and the
// SSRF-safe transport before reaching here.
//
// Transport failures are masked here, at the single point
internal/delivery/target_http_query_test.go
+26 -16
View File
@@ -2,6 +2,7 @@ package delivery_test
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
@@ -17,13 +18,25 @@ import (
// with.
const eventQuery = "a=1&b=2"
// deliverWithQuery sends one event that arrived with eventQuery to an
// HTTP target at targetURL, with its ForwardQuery setting as given,
// through the path a received event's delivery takes, and returns the
// attempt it recorded.
func deliverWithQuery(
// httpTargetConfig is the stored configuration of an HTTP target at
// targetURL.
func httpTargetConfig(
t *testing.T, targetURL string, forwardQuery bool,
) database.DeliveryResult {
) string {
t.Helper()
cfg, err := json.Marshal(delivery.HTTPTargetConfig{
URL: targetURL, ForwardQuery: forwardQuery,
})
require.NoError(t, err)
return string(cfg)
}
// deliverWithQuery sends one event that arrived with eventQuery to an
// HTTP target configured with cfg, through the path a received event's
// delivery takes, and returns the attempt it recorded.
func deliverWithQuery(t *testing.T, cfg string) database.DeliveryResult {
t.Helper()
s := newISetup(t)
@@ -34,10 +47,8 @@ func deliverWithQuery(
database.DeliveryStatusPending,
)
task := iTask(
d, event, s.WebhookID, d.TargetID, "query", targetURL, 1, 1,
&event.Body,
d, event, s.WebhookID, d.TargetID, "query", cfg, 1, 1, &event.Body,
)
task.ForwardQuery = forwardQuery
task.RawQuery = eventQuery
s.Engine.ExportProcessNewTask(context.TODO(), &task)
@@ -100,9 +111,9 @@ func TestDeliverHTTP_ForwardQuery(t *testing.T) {
))
t.Cleanup(ts.Close)
result := deliverWithQuery(
result := deliverWithQuery(t, httpTargetConfig(
t, ts.URL+tc.path, tc.forwardQuery,
)
))
assert.True(t, result.Success)
require.Len(t, received, 1)
@@ -134,12 +145,11 @@ func TestDeliverHTTP_ForwardedQueryKeepsTheTargetURLMasked(t *testing.T) {
target := &database.Target{
Type: database.TargetTypeHTTP,
URL: ts.URL + "/in?token=" + secret,
ForwardQuery: true,
Config: httpTargetConfig(t, ts.URL+"/in?token="+secret, true),
MaxAttempts: 1,
}
deliverWithQuery(t, target.URL, target.ForwardQuery)
deliverWithQuery(t, target.Config)
require.Len(t, received, 1)
sent := <-received
@@ -154,9 +164,9 @@ func TestDeliverHTTP_ForwardedQueryKeepsTheTargetURLMasked(t *testing.T) {
}
// Nothing listens on port 1.
failed := deliverWithQuery(
failed := deliverWithQuery(t, httpTargetConfig(
t, "http://127.0.0.1:1/in?token="+secret, true,
)
))
require.NotEmpty(t, failed.Error)
assert.NotContains(t, failed.Error, secret)
assert.NotContains(t, failed.Error, eventQuery)
internal/delivery/target_redact.go
+27 -6
View File
@@ -18,7 +18,7 @@ const RedactionMarker = "(redacted)"
// line behind the delivery path's own URL masking.
//
// It removes byte-identical echoes of strings taken from the
// target's URL and headers, and nothing else. Anything
// target's stored configuration, and nothing else. Anything
// the remote re-encodes survives: JSON "\/" escaping (what
// PHP's json_encode emits by default), percent-encoding, HTML
// entities, and an echo of only part of a path. It cannot
@@ -115,7 +115,7 @@ func (r Redactor) secretPrefixSuffix(s string) int {
}
// targetSecrets returns the credential-bearing strings a
// target's settings carry.
// target's configuration carries.
//
// The destination URL contributes. Its path, query and
// userinfo are the credential for both target types that have
@@ -129,15 +129,36 @@ func (r Redactor) secretPrefixSuffix(s string) int {
// picks out. That is the same class-based rule applied to
// URLs: an echoed Accept or User-Agent still renders, an
// echoed Authorization does not.
//
// A setting a type does not use is empty and contributes
// nothing, so a database or log target has nothing to redact.
func targetSecrets(t *database.Target) []string {
if t == nil {
return nil
}
return append(urlSecrets(t.URL), headerSecrets(t.Headers)...)
switch t.Type {
case database.TargetTypeSlack:
cfg, err := parseSlackConfig(t.Config)
if err != nil {
return nil
}
return urlSecrets(cfg.WebhookURL)
case database.TargetTypeHTTP:
cfg, err := parseHTTPConfig(t.Config)
if err != nil {
return nil
}
return append(
urlSecrets(cfg.URL),
headerSecrets(cfg.Headers)...,
)
case database.TargetTypeDatabase, database.TargetTypeLog:
// Neither has a destination URL, so neither has
// anything to redact.
return nil
default:
return nil
}
}
// urlSecrets returns the substrings of a destination URL that
internal/delivery/target_redact_test.go
+28 -28
View File
@@ -23,7 +23,7 @@ func TestRedactor_RemovesSlackWebhookURL(t *testing.T) {
r := delivery.NewRedactor(&database.Target{
Type: database.TargetTypeSlack,
URL: redactWebhookURL,
Config: `{"webhookUrl":"` + redactWebhookURL + `"}`,
MaxAttempts: 1,
})
@@ -54,7 +54,7 @@ func TestRedactor_RemovesSecretSeveredByACut(t *testing.T) {
r := delivery.NewRedactor(&database.Target{
Type: database.TargetTypeSlack,
URL: redactWebhookURL,
Config: `{"webhookUrl":"` + redactWebhookURL + `"}`,
MaxAttempts: 1,
})
@@ -86,19 +86,18 @@ func TestRedactor_RedactsCredentialShapedHeaderValues(
r := delivery.NewRedactor(&database.Target{
Type: database.TargetTypeHTTP,
URL: "https://example.com/in",
Headers: map[string]string{
headerAuthorization: "Bearer AAAAAAAAAAAA",
"Cookie": "session=BBBBBBBBBBBB",
probeHeaderName: "CCCCCCCCCCCC",
"X-Hub-Signature": "sha256=DDDDDDDDDDDD",
"X-Sig": "EEEEEEEEEEEE",
"X-Pass": "FFFFFFFFFFFF",
"X-HMAC": "GGGGGGGGGGGG",
"X-Credential": "HHHHHHHHHHHH",
"Accept": "application/json",
headerUserAgent: "webhooker/1.0",
},
Config: `{"url":"https://example.com/in",` +
`"headers":{` +
`"Authorization":"Bearer AAAAAAAAAAAA",` +
`"Cookie":"session=BBBBBBBBBBBB",` +
`"X-Api-Key":"CCCCCCCCCCCC",` +
`"X-Hub-Signature":"sha256=DDDDDDDDDDDD",` +
`"X-Sig":"EEEEEEEEEEEE",` +
`"X-Pass":"FFFFFFFFFFFF",` +
`"X-HMAC":"GGGGGGGGGGGG",` +
`"X-Credential":"HHHHHHHHHHHH",` +
`"Accept":"application/json",` +
`"User-Agent":"webhooker/1.0"}}`,
MaxAttempts: 1,
})
@@ -138,8 +137,8 @@ func TestRedactor_IgnoresVeryShortHeaderValues(t *testing.T) {
r := delivery.NewRedactor(&database.Target{
Type: database.TargetTypeHTTP,
URL: "https://example.com/in",
Headers: map[string]string{probeHeaderName: "ab"},
Config: `{"url":"https://example.com/in",` +
`"headers":{"X-Api-Key":"ab"}}`,
MaxAttempts: 1,
})
@@ -156,7 +155,7 @@ func TestRedactor_RemovesBarePath(t *testing.T) {
r := delivery.NewRedactor(&database.Target{
Type: database.TargetTypeSlack,
URL: redactWebhookURL,
Config: `{"webhookUrl":"` + redactWebhookURL + `"}`,
MaxAttempts: 1,
})
@@ -190,7 +189,7 @@ func TestRedactor_RemovesHTTPURLQueryAndUserinfo(t *testing.T) {
r := delivery.NewRedactor(&database.Target{
Type: database.TargetTypeHTTP,
URL: raw,
Config: `{"url":"` + raw + `"}`,
MaxAttempts: 1,
})
@@ -240,7 +239,7 @@ func TestRedactor_RemovesEchoedQueryOfURLWithoutPath(t *testing.T) {
} {
r := delivery.NewRedactor(&database.Target{
Type: database.TargetTypeHTTP,
URL: dest,
Config: `{"url":"` + dest + `"}`,
MaxAttempts: 1,
})
@@ -262,7 +261,7 @@ func TestRedactor_LeavesUnrelatedTextAlone(t *testing.T) {
r := delivery.NewRedactor(&database.Target{
Type: database.TargetTypeSlack,
URL: redactWebhookURL,
Config: `{"webhookUrl":"` + redactWebhookURL + `"}`,
MaxAttempts: 1,
})
@@ -289,7 +288,7 @@ func TestRedactor_EmptyUserinfoDoesNotShredTheBody(t *testing.T) {
r := delivery.NewRedactor(&database.Target{
Type: database.TargetTypeHTTP,
URL: dest,
Config: `{"url":"` + dest + `"}`,
MaxAttempts: 1,
})
@@ -307,10 +306,11 @@ func TestRedactor_EmptyUserinfoDoesNotShredTheBody(t *testing.T) {
)
}
// TestRedactor_ZeroValueAndURLlessTargets pins that a caller
// with no target, or a target with no destination URL, gets a
// redactor that changes nothing rather than one that panics.
func TestRedactor_ZeroValueAndURLlessTargets(t *testing.T) {
// TestRedactor_ZeroValueAndConfiglessTargets pins that a
// caller with no target, an unparseable config, or a target
// type with no destination URL gets a redactor that changes
// nothing rather than one that panics.
func TestRedactor_ZeroValueAndConfiglessTargets(t *testing.T) {
t.Parallel()
const text = "some response body"
@@ -323,8 +323,8 @@ func TestRedactor_ZeroValueAndURLlessTargets(t *testing.T) {
for _, tgt := range []database.Target{
{Type: database.TargetTypeLog},
{Type: database.TargetTypeDatabase},
{Type: database.TargetTypeSlack, MaxAttempts: 1},
{Type: database.TargetTypeHTTP, MaxAttempts: 1},
{Type: database.TargetTypeSlack, Config: "not json", MaxAttempts: 1},
{Type: database.TargetTypeHTTP, Config: "", MaxAttempts: 1},
} {
assert.Equal(
t, text,
internal/delivery/target_slack.go
+78 -16
View File
@@ -4,6 +4,7 @@ import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
@@ -14,6 +15,18 @@ import (
"sneak.berlin/go/webhooker/internal/database"
)
// errMissingWebhookURL is returned when a Slack target config
// omits its webhook URL.
var errMissingWebhookURL = errors.New(
"webhook_url is required",
)
// SlackTargetConfig holds configuration for slack target
// types.
type SlackTargetConfig struct {
WebhookURL string `json:"webhookUrl"`
}
// slackTarget delivers events to Slack incoming webhooks. It
// formats the event into a Slack message and posts it as
// JSON. It shares the retry core with the HTTP target: a
@@ -33,6 +46,19 @@ func (t *slackTarget) Deliver(
task *Task,
sched Scheduler,
) {
cfg, err := parseSlackConfig(d.Target.Config)
if err != nil {
t.eng.log.Error(
"invalid Slack target config",
"target_id", d.TargetID,
"error", err,
)
t.failConfig(webhookDB, d, err)
return
}
msg := FormatSlackMessage(&d.Event)
payload, err := json.Marshal(
@@ -45,6 +71,29 @@ func (t *slackTarget) Deliver(
"error", err,
)
t.failConfig(webhookDB, d, err)
return
}
attempt := func() attemptResult {
return t.attempt(ctx, cfg, payload)
}
t.deliver(
ctx, webhookDB, d, task, sched,
d.Target.MaxAttempts, attempt,
)
}
// failConfig records a first-attempt failure for a delivery
// that could not be prepared (bad config or unmarshalable
// payload) and marks it failed.
func (t *slackTarget) failConfig(
webhookDB *gorm.DB,
d *database.Delivery,
err error,
) {
recErr := t.eng.recordResult(
webhookDB, d, 1,
false, 0, "", err.Error(), 0,
@@ -59,27 +108,15 @@ func (t *slackTarget) Deliver(
webhookDB, d, d.Target.Type,
database.DeliveryStatusFailed,
)
return
}
attempt := func() attemptResult {
return t.attempt(ctx, d.Target.URL, payload)
}
t.deliver(
ctx, webhookDB, d, task, sched,
d.Target.MaxAttempts, attempt,
)
}
// attempt performs a single Slack POST to targetURL and derives
// its outcome, preserving the engine's original semantics: a
// attempt performs a single Slack POST and derives its
// outcome, preserving the engine's original semantics: a
// non-2xx response records an "HTTP <code>" error string and
// a transport error records a "sending request" error.
func (t *slackTarget) attempt(
ctx context.Context,
targetURL string,
cfg *SlackTargetConfig,
payload []byte,
) attemptResult {
start := time.Now()
@@ -87,7 +124,7 @@ func (t *slackTarget) attempt(
req, err := http.NewRequestWithContext(
ctx,
http.MethodPost,
targetURL,
cfg.WebhookURL,
bytes.NewReader(payload),
)
if err != nil {
@@ -149,6 +186,31 @@ func (t *slackTarget) readSlackResponse(
}
}
func parseSlackConfig(
configJSON string,
) (*SlackTargetConfig, error) {
if configJSON == "" {
return nil, errEmptyTargetConfig
}
var cfg SlackTargetConfig
err := json.Unmarshal(
[]byte(configJSON), &cfg,
)
if err != nil {
return nil, fmt.Errorf(
"parsing config JSON: %w", err,
)
}
if cfg.WebhookURL == "" {
return nil, errMissingWebhookURL
}
return &cfg, nil
}
// FormatSlackMessage builds a Slack-compatible message
// string from a webhook event.
func FormatSlackMessage(
internal/delivery/terminal_state_test.go
+7 -6
View File
@@ -45,7 +45,7 @@ func tSeedDeletedTarget(
iCreateTarget(
t, s.MainDB, targetID, s.WebhookID, name,
database.TargetTypeHTTP, url, 5,
database.TargetTypeHTTP, iHTTPConfig(url), 5,
)
event := iSeedEvent(
@@ -124,7 +124,7 @@ func TestProcessDelivery_UnknownTargetType_RecordsWhy(
target := database.Target{
Name: "mystery",
Type: tUnknownType,
URL: "http://example.com/hook",
Config: iHTTPConfig("http://example.com/hook"),
}
target.ID = targetID
@@ -140,7 +140,7 @@ func TestProcessDelivery_UnknownTargetType_RecordsWhy(
body := event.Body
task := iTask(
seeded, event, s.WebhookID, targetID, "mystery",
target.URL, 0, 1, &body,
target.Config, 0, 1, &body,
)
task.TargetType = tUnknownType
@@ -346,10 +346,11 @@ func tRetryChainSetup(
iCreateWebhook(t, s.MainDB, s.WebhookID, name)
targetID := uuid.New().String()
cfg := iHTTPConfig(ts.URL)
iCreateTarget(
t, s.MainDB, targetID, s.WebhookID, name,
database.TargetTypeHTTP, ts.URL, 5,
database.TargetTypeHTTP, cfg, 5,
)
event := iSeedEvent(
@@ -366,7 +367,7 @@ func tRetryChainSetup(
body := event.Body
return iTask(
d, event, s.WebhookID, targetID, name, ts.URL, 5, 2, &body,
d, event, s.WebhookID, targetID, name, cfg, 5, 2, &body,
), targetID
}
@@ -527,7 +528,7 @@ func TestRecoverSingleRetry_TargetUnreadable_LeavesDeliveryAlone(
iCreateTarget(
t, s.MainDB, targetID, s.WebhookID, "healthy",
database.TargetTypeHTTP,
"http://example.com/hook", 5,
iHTTPConfig("http://example.com/hook"), 5,
)
event := iSeedEvent(
internal/delivery/url_mask_test.go
+16 -4
View File
@@ -2,6 +2,7 @@ package delivery_test
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
@@ -70,6 +71,13 @@ func deliverSlackTo(
e := testEngine(t, 1)
targetID := uuid.New().String()
slackCfg, err := json.Marshal(
delivery.SlackTargetConfig{
WebhookURL: webhookURL,
},
)
require.NoError(t, err)
event := seedEvent(t, db, `{"test":true}`)
dlv := seedDelivery(
@@ -79,7 +87,7 @@ func deliverSlackTo(
d := buildSlackDelivery(
dlv, event, targetID,
"test-slack-mask", webhookURL,
"test-slack-mask", string(slackCfg),
)
e.ExportDeliverSlack(context.TODO(), db, d)
@@ -152,9 +160,13 @@ func TestDoHTTPRequest_TransportErrorMasksURL(t *testing.T) {
e := testEngine(t, 1)
cfg, err := e.ExportParseHTTPConfig(
newHTTPTargetConfig(host + maskSecretPath),
)
require.NoError(t, err)
statusCode, _, _, reqErr := e.ExportDoHTTPRequest(
context.TODO(),
&database.Target{URL: host + maskSecretPath},
context.TODO(), cfg,
&database.Event{Body: `{"test":true}`},
)
require.Error(t, reqErr)
@@ -176,7 +188,7 @@ func TestDoHTTPRequest_UnparsableURLIsMasked(t *testing.T) {
statusCode, _, _, reqErr := e.ExportDoHTTPRequest(
context.TODO(),
&database.Target{
&delivery.HTTPTargetConfig{
URL: "https://hooks.example.com" + maskSecretPath + "\n",
},
&database.Event{},
internal/delivery/user_agent_test.go
+9 -3
View File
@@ -2,6 +2,7 @@ package delivery_test
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"net/netip"
@@ -58,9 +59,9 @@ func TestUserAgent_IsTheBuildVersion(t *testing.T) {
statusCode, _, _, err := e.ExportDoHTTPRequest(
context.Background(),
&database.Target{
&delivery.HTTPTargetConfig{
URL: ts.URL,
Headers: map[string]string{headerUserAgent: "configured/1"},
Headers: map[string]string{"User-Agent": "configured/1"},
},
&database.Event{Headers: `{"User-Agent":["curl/8"]}`},
)
@@ -72,13 +73,18 @@ func TestUserAgent_IsTheBuildVersion(t *testing.T) {
db := testWebhookDB(t)
targetID := uuid.New().String()
slackCfg, err := json.Marshal(
delivery.SlackTargetConfig{WebhookURL: ts.URL},
)
require.NoError(t, err)
event := seedEvent(t, db, `{"action":"test"}`)
dlv := seedDelivery(
t, db, event.ID, targetID, database.DeliveryStatusPending,
)
e.ExportDeliverSlack(context.Background(), db, buildSlackDelivery(
dlv, event, targetID, "test-slack", ts.URL,
dlv, event, targetID, "test-slack", string(slackCfg),
))
require.Len(t, userAgents, 1, "the slack target sent no request")
assert.Equal(t, want, <-userAgents, "slack target")
internal/handlers/archive_expiry.go
+1 -23
View File
@@ -1,14 +1,8 @@
package handlers
import (
"strconv"
"time"
)
const (
// archiveExpiryNever is the archive expiry that keeps archived
// events forever. It is stored as an ArchiveExpiry of 0, and an
// empty expiry on a form means the same.
// events forever. A stored empty expiry means the same.
archiveExpiryNever = "never"
// tmplKeyArchiveExpiryChoices is the template data key for the
@@ -62,19 +56,3 @@ func archiveExpiryOptions(expiry string) []archiveChoice {
return append([]archiveChoice{own}, options...)
}
// formatArchiveExpiry writes a stored ArchiveExpiry the way the
// expiry select spells its values, so the target edit form starts on
// the stored choice: never for 0, a whole number of hours as "720h",
// and anything else as Go writes a duration, which the form accepts
// back unchanged.
func formatArchiveExpiry(expiry time.Duration) string {
switch {
case expiry == 0:
return archiveExpiryNever
case expiry%time.Hour == 0:
return strconv.FormatInt(int64(expiry/time.Hour), 10) + "h"
default:
return expiry.String()
}
}
internal/handlers/archive_expiry_test.go
+16 -20
View File
@@ -7,7 +7,6 @@ import (
"regexp"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -95,18 +94,14 @@ func TestArchiveExpiryChoices(t *testing.T) {
env := setupSourceTest(t)
choices := []struct {
value string
stored time.Duration
shown string
}{
{expiryNever, 0, expiryNever},
{"1h", time.Hour, "1 hour"},
{"12h", 12 * time.Hour, "12 hours"},
{"24h", 24 * time.Hour, "1 day"},
{"720h", 720 * time.Hour, "30 days"},
{"2160h", 2160 * time.Hour, "90 days"},
{"8760h", 8760 * time.Hour, "365 days"},
choices := []struct{ value, shown string }{
{expiryNever, expiryNever},
{"1h", "1 hour"},
{"12h", "12 hours"},
{"24h", "1 day"},
{"720h", "30 days"},
{"2160h", "90 days"},
{"8760h", "365 days"},
}
for _, choice := range choices {
@@ -127,7 +122,9 @@ func TestArchiveExpiryChoices(t *testing.T) {
targets := targetsForWebhook(t, env.db, webhook.ID)
require.Len(t, targets, 1)
assert.Equal(t, choice.stored, targets[0].ArchiveExpiry)
assert.JSONEq(
t, `{"expiry":"`+choice.value+`"}`, targets[0].Config,
)
assert.Equal(
t, []string{choice.shown},
@@ -141,7 +138,7 @@ func TestArchiveExpiryChoices(t *testing.T) {
}
// TestArchiveExpiryEditStartsOnStoredValue checks the edit form of a
// database target whose stored expiry is 0, which selects never,
// database target whose stored expiry is empty, which selects never,
// and of one whose expiry is not one of the choices, which is listed
// first as its own selected entry and saved unchanged.
func TestArchiveExpiryEditStartsOnStoredValue(t *testing.T) {
@@ -151,8 +148,7 @@ func TestArchiveExpiryEditStartsOnStoredValue(t *testing.T) {
webhook := seedWebhookWithRetention(t, env.db, 30)
empty := seedConfiguredTarget(
t, env.db, webhook.ID, database.TargetTypeDatabase,
database.Target{},
t, env.db, webhook.ID, database.TargetTypeDatabase, "",
)
_, selected := expirySelected(t, env, webhook.ID, empty.ID)
@@ -161,7 +157,7 @@ func TestArchiveExpiryEditStartsOnStoredValue(t *testing.T) {
webhook = seedWebhookWithRetention(t, env.db, 30)
unlisted := seedConfiguredTarget(
t, env.db, webhook.ID, database.TargetTypeDatabase,
database.Target{ArchiveExpiry: 36 * time.Hour},
`{"expiry":"36h"}`,
)
assert.Equal(t, []string{"36 hours"}, expiryShown(t, env, webhook.ID))
@@ -183,7 +179,7 @@ func TestArchiveExpiryEditStartsOnStoredValue(t *testing.T) {
w := submitTargetEdit(env, webhook.ID, unlisted.ID, form)
require.Equal(t, http.StatusSeeOther, w.Code, w.Body.String())
assert.Equal(
t, 36*time.Hour, storedTarget(t, env, unlisted.ID).ArchiveExpiry,
assert.JSONEq(
t, `{"expiry":"36h"}`, storedTarget(t, env, unlisted.ID).Config,
)
}
internal/handlers/archive_rotation_test.go
+23 -37
View File
@@ -16,12 +16,8 @@ import (
"sneak.berlin/go/webhooker/internal/database"
)
// The archive rotations these tests store: none keeps one file.
const (
rotationNone = "none"
rotationDaily = "daily"
rotationHourly = "hourly"
)
// rotationNone is the archive rotation that keeps one file.
const rotationNone = "none"
// rotationShown returns the archive rotations the webhook page's
// target list shows.
@@ -45,9 +41,7 @@ func TestArchiveRotationChoices(t *testing.T) {
env := setupSourceTest(t)
for _, rotation := range []string{
rotationNone, "monthly", rotationDaily, rotationHourly,
} {
for _, rotation := range []string{rotationNone, "monthly", "daily", "hourly"} {
t.Run(rotation, func(t *testing.T) {
t.Parallel()
@@ -66,8 +60,10 @@ func TestArchiveRotationChoices(t *testing.T) {
targets := targetsForWebhook(t, env.db, webhook.ID)
require.Len(t, targets, 1)
assert.Equal(t, 720*time.Hour, targets[0].ArchiveExpiry)
assert.Equal(t, rotation, targets[0].ArchiveRotation)
assert.JSONEq(t,
`{"expiry":"720h","rotation":"`+rotation+`"}`,
targets[0].Config,
)
assert.Equal(t,
[]string{rotation}, rotationShown(t, env, webhook.ID))
@@ -82,37 +78,26 @@ func TestArchiveRotationChoices(t *testing.T) {
w = submitTargetEdit(env, webhook.ID, targets[0].ID, form)
require.Equal(t, http.StatusSeeOther, w.Code, w.Body.String())
stored := storedTarget(t, env, targets[0].ID)
assert.Equal(t, 720*time.Hour, stored.ArchiveExpiry)
assert.Equal(t, rotationHourly, stored.ArchiveRotation)
assert.JSONEq(t,
`{"expiry":"720h","rotation":"hourly"}`,
storedTarget(t, env, targets[0].ID).Config,
)
})
}
}
// TestArchiveRotationEditStartsOnNone checks that a database target
// added with no rotation is stored with none and shown with none, and
// that its edit form starts on none.
// TestArchiveRotationEditStartsOnNone checks the edit form of a
// database target with no rotation stored starts on none.
func TestArchiveRotationEditStartsOnNone(t *testing.T) {
t.Parallel()
env := setupSourceTest(t)
webhook := seedWebhookWithRetention(t, env.db, 30)
form := url.Values{}
form.Set("name", "archive")
form.Set("type", string(database.TargetTypeDatabase))
w := serveTarget(
env, http.MethodPost, "/hook/"+webhook.ID+"/targets", form,
target := seedConfiguredTarget(
t, env.db, webhook.ID, database.TargetTypeDatabase, "",
)
require.Equal(t, http.StatusSeeOther, w.Code, w.Body.String())
targets := targetsForWebhook(t, env.db, webhook.ID)
require.Len(t, targets, 1)
assert.Equal(t, rotationNone, targets[0].ArchiveRotation)
page := targetEditPage(t, env, webhook.ID, targets[0].ID)
page := targetEditPage(t, env, webhook.ID, target.ID)
assert.Equal(t, []string{rotationNone}, selectedIn(page, "rotation"))
assert.Equal(t, []string{rotationNone}, rotationShown(t, env, webhook.ID))
}
@@ -140,7 +125,7 @@ func TestArchiveRotationRefused(t *testing.T) {
target := seedConfiguredTarget(
t, env.db, webhook.ID, database.TargetTypeDatabase,
database.Target{ArchiveRotation: rotationDaily},
`{"rotation":"daily"}`,
)
form.Del("type")
@@ -148,8 +133,8 @@ func TestArchiveRotationRefused(t *testing.T) {
w = submitTargetEdit(env, webhook.ID, target.ID, form)
assert.Equal(t, http.StatusBadRequest, w.Code)
assert.Contains(t, w.Body.String(), "Invalid archive rotation")
assert.Equal(
t, rotationDaily, storedTarget(t, env, target.ID).ArchiveRotation,
assert.JSONEq(t,
`{"rotation":"daily"}`, storedTarget(t, env, target.ID).Config,
)
}
@@ -176,8 +161,9 @@ func TestHandleSourceCreateSubmit_ArchiveRotation(t *testing.T) {
targets := targetsForWebhook(t, env.db, webhook.ID)
require.Len(t, targets, 1)
assert.Equal(t, 720*time.Hour, targets[0].ArchiveExpiry)
assert.Equal(t, rotationDaily, targets[0].ArchiveRotation)
assert.JSONEq(t,
`{"expiry":"720h","rotation":"daily"}`, targets[0].Config,
)
}
// TestArchiveFileView_Rotated describes a daily target's archive files
@@ -192,7 +178,7 @@ func TestArchiveFileView_Rotated(t *testing.T) {
webhook := seedWebhookWithRetention(t, env.db, 30)
target := seedConfiguredTarget(
t, env.db, webhook.ID, database.TargetTypeDatabase,
database.Target{ArchiveRotation: rotationDaily},
`{"rotation":"daily"}`,
)
path := archive.Path(env.dbMgr, &webhook, target)
internal/handlers/delivery_replay.go
+90 -12
View File
@@ -5,7 +5,9 @@ import (
"github.com/go-chi/chi"
"gorm.io/gorm"
"gorm.io/gorm/clause"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
)
// The outcomes of a replay POST, as the notice codes its redirect
@@ -44,12 +46,12 @@ const (
// HandleDeliveryReplay re-sends a finished delivery's event to its
// target.
//
// A replay never touches the delivery it repeats. The delivery engine
// creates a NEW pending delivery row for the same event and target and
// queues it as it queues the receiver's, so the original's status,
// attempts and timestamps stand as the record of what actually
// happened, and the replay is retried, SSRF-guarded and circuit-broken
// exactly as a first attempt is.
// A replay never touches the delivery it repeats. It creates a NEW
// pending delivery row for the same event and target and hands it to
// the delivery engine through the same Notifier the receiver uses, so
// the original's status, attempts and timestamps stand as the record
// of what actually happened, and the replay is retried, SSRF-guarded
// and circuit-broken exactly as a first attempt is.
//
// What is re-sent is the stored EVENT body, never the response the
// original delivery received.
@@ -143,8 +145,7 @@ func (h *Handlers) loadReplaySource(
return &original, true
}
// queueReplay has the delivery engine create and queue the new
// delivery, unless an earlier replay is still in flight.
// queueReplay writes the new delivery and hands it to the engine.
func (h *Handlers) queueReplay(
w http.ResponseWriter,
r *http.Request,
@@ -170,8 +171,18 @@ func (h *Handlers) queueReplay(
return
}
deliveryID, err := h.engine.CreateReplay(
webhook.ID, original.EventID, target,
var event database.Event
err = webhookDB.
First(&event, "id = ?", original.EventID).Error
if err != nil {
h.serverError(w, r, "failed to load event for replay", err)
return
}
task, err := createReplayDelivery(
webhookDB, webhook.ID, &event, target,
)
if err != nil {
h.serverError(
@@ -182,14 +193,15 @@ func (h *Handlers) queueReplay(
}
h.mtr.DeliveryReplayed(target.Type)
h.notifier.Notify([]delivery.Task{task})
h.log.Info(
"delivery replay queued",
"webhook_id", webhook.ID,
"event_id", original.EventID,
"event_id", event.ID,
"target_id", target.ID,
"replayed_delivery_id", original.ID,
"delivery_id", deliveryID,
"delivery_id", task.DeliveryID,
)
redirectToEventLog(w, r, webhook, replayQueued)
@@ -251,6 +263,72 @@ func countInFlightDeliveries(
return count, err
}
// createReplayDelivery writes the new pending delivery row, adds it to
// its target's totals in the same transaction, and returns the task
// that carries it to the delivery engine.
//
// The row is written with associations omitted, and neither Event nor
// Target is populated on it: GORM's SaveBeforeAssociations would
// otherwise upsert the whole target row — plaintext config, which for a
// Slack target is the credential — into the per-webhook event database.
// See https://git.eeqj.de/sneak/webhooker/issues/206.
func createReplayDelivery(
webhookDB *gorm.DB,
webhookID string,
event *database.Event,
target *database.Target,
) (delivery.Task, error) {
dlv := &database.Delivery{
EventID: event.ID,
TargetID: target.ID,
Status: database.DeliveryStatusPending,
Replay: true,
}
err := webhookDB.Transaction(func(tx *gorm.DB) error {
err := tx.Omit(clause.Associations).Create(dlv).Error
if err != nil {
return err
}
return database.AddTargetTotals(tx, database.TargetTotals{
TargetID: dlv.TargetID, Deliveries: 1,
})
})
if err != nil {
return delivery.Task{}, err
}
return delivery.Task{
DeliveryID: dlv.ID,
EventID: event.ID,
WebhookID: webhookID,
EntrypointID: event.EntrypointID,
TargetID: target.ID,
TargetName: target.Name,
TargetType: target.Type,
TargetConfig: target.Config,
MaxAttempts: target.MaxAttempts,
Method: event.Method,
RawQuery: event.RawQuery,
Headers: event.Headers,
ContentType: event.ContentType,
Body: replayBody(event.Body),
AttemptNum: 1,
}, nil
}
// replayBody returns the stored event body for a replay task to carry
// inline, or nil when it is large enough that the engine should fetch
// it from the per-webhook database instead.
func replayBody(body string) *string {
if len(body) >= delivery.MaxInlineBodySize {
return nil
}
return &body
}
// redirectToEventLog redirects a replay or resubmit back to the event
// log it was triggered from, carrying the outcome as its notice. A
// Replay form carries the list it was pressed in as show, so a replay
internal/handlers/delivery_replay_test.go
+85 -54
View File
@@ -12,6 +12,7 @@ import (
"gorm.io/gorm"
"gorm.io/gorm/clause"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/session"
)
@@ -22,7 +23,7 @@ const paramDeliveryID = "deliveryID"
// replayTargetURL is a public destination, so a target configured with
// it is one the SSRF guard would accept. Nothing in these tests
// dispatches to it: the delivery engine is never started.
// dispatches to it: the notifier is recorded, not run.
const replayTargetURL = "http://93.184.216.34/hook"
// replayEventQuery is the query string a seeded event arrived with.
@@ -162,8 +163,14 @@ func postReplay(
// TestHandleDeliveryReplay_AppendsDeliveryAndLeavesOriginal is the
// core requirement: replaying a failed delivery succeeds, appends a
// new delivery marked as a replay, and leaves the original row and its
// recorded attempt exactly as they were.
// new delivery, and leaves the original row and its recorded attempt
// exactly as they were.
//
// It also pins the two things a replay would be wrong to get from the
// original: the task carries the target's CURRENT configuration, which
// this test changes between the failure and the replay, and it carries
// the stored EVENT body rather than anything the failed attempt
// received back.
func TestHandleDeliveryReplay_AppendsDeliveryAndLeavesOriginal(
t *testing.T,
) {
@@ -174,9 +181,10 @@ func TestHandleDeliveryReplay_AppendsDeliveryAndLeavesOriginal(
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
notif *recordingNotifier
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app := newTestApp(t, &h, &sess, &db, &dbMgr, &notif)
app.RequireStart()
t.Cleanup(app.RequireStop)
@@ -184,7 +192,7 @@ func TestHandleDeliveryReplay_AppendsDeliveryAndLeavesOriginal(
wh := seedWebhook(t, db)
tgt := seedConfiguredTarget(
t, db, wh.ID, database.TargetTypeHTTP,
database.Target{URL: replayTargetURL},
`{"url":"`+replayTargetURL+`"}`,
)
event, original := seedFailedDelivery(
@@ -196,6 +204,15 @@ func TestHandleDeliveryReplay_AppendsDeliveryAndLeavesOriginal(
before := loadDelivery(t, webhookDB, original.ID)
// The operator fixes the destination, which is the whole reason
// to replay. The replay must use this, not the config the
// original delivery ran against.
const fixedConfig = `{"url":"http://93.184.216.34/fixed"}`
require.NoError(t, db.DB().Model(&database.Target{}).
Where("id = ?", tgt.ID).
Update("config", fixedConfig).Error)
w := postReplay(t, h, sess, wh.ID, original.ID)
require.Equal(t, http.StatusSeeOther, w.Code)
@@ -217,9 +234,14 @@ func TestHandleDeliveryReplay_AppendsDeliveryAndLeavesOriginal(
assert.Equal(
t, database.DeliveryStatusPending, replayed.Status,
)
assert.True(t, replayed.Replay)
assertDeliveryUntouched(t, webhookDB, before)
tasks := notif.Tasks()
require.Len(t, tasks, 1)
assertReplayTask(
t, tasks[0], wh.ID, event, tgt, replayed.ID, fixedConfig,
)
assertNoLeakedTarget(t, webhookDB)
}
@@ -253,6 +275,45 @@ func assertDeliveryUntouched(
)
}
// assertReplayTask proves the task handed to the delivery engine is
// the one the receiver would build for this event and this target, and
// that it carries wantConfig — the target's configuration as it stands
// now rather than as the original delivery ran against it.
func assertReplayTask(
t *testing.T,
task delivery.Task,
webhookID string,
event *database.Event,
target *database.Target,
wantDeliveryID, wantConfig string,
) {
t.Helper()
assert.Equal(t, wantDeliveryID, task.DeliveryID)
assert.Equal(t, event.ID, task.EventID)
assert.Equal(t, webhookID, task.WebhookID)
assert.Equal(t, event.EntrypointID, task.EntrypointID)
assert.Equal(t, target.ID, task.TargetID)
assert.Equal(t, target.Type, task.TargetType)
assert.JSONEq(
t, wantConfig, task.TargetConfig,
"replay must use the target's current configuration",
)
assert.Equal(t, event.Method, task.Method)
assert.Equal(
t, replayEventQuery, task.RawQuery,
"replay re-sends the stored query string",
)
assert.Equal(t, event.Headers, task.Headers)
assert.Equal(t, event.ContentType, task.ContentType)
assert.Equal(t, 1, task.AttemptNum)
require.NotNil(t, task.Body)
assert.Equal(
t, event.Body, *task.Body,
"replay re-sends the stored event body",
)
}
// assertNoLeakedTarget proves the per-webhook database holds no
// target: an association write would have upserted the whole target,
// plaintext config and all, into the event database. See
@@ -266,49 +327,9 @@ func assertNoLeakedTarget(t *testing.T, webhookDB *gorm.DB) {
)
}
// TestHandleDeliveryReplay_FailedStoreIsAnError proves a replay the
// engine cannot store answers with the 500 error page and adds no
// delivery.
func TestHandleDeliveryReplay_FailedStoreIsAnError(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
tgt := seedConfiguredTarget(
t, db, wh.ID, database.TargetTypeHTTP,
database.Target{URL: replayTargetURL},
)
event, original := seedFailedDelivery(t, dbMgr, wh.ID, tgt.ID)
// Without its target totals table the webhook's database refuses
// the totals write, which comes after the delivery row.
webhookDB, err := dbMgr.GetDB(wh.ID)
require.NoError(t, err)
require.NoError(t,
webhookDB.Migrator().DropTable(&database.TargetTotals{}))
w := postReplay(t, h, sess, wh.ID, original.ID)
assert.Equal(t, http.StatusInternalServerError, w.Code)
assert.Contains(t, w.Body.String(), "500 Internal Server Error")
assert.Len(t, listDeliveries(t, webhookDB, event.ID), 1)
}
// TestHandleDeliveryReplay_RefusesDeletedTarget proves the required
// refusal: a target deleted since the delivery ran is reported as
// deleted rather than erroring, and nothing is created.
// deleted rather than erroring, and nothing is created or queued.
func TestHandleDeliveryReplay_RefusesDeletedTarget(t *testing.T) {
t.Parallel()
@@ -317,9 +338,10 @@ func TestHandleDeliveryReplay_RefusesDeletedTarget(t *testing.T) {
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
notif *recordingNotifier
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app := newTestApp(t, &h, &sess, &db, &dbMgr, &notif)
app.RequireStart()
t.Cleanup(app.RequireStop)
@@ -327,7 +349,7 @@ func TestHandleDeliveryReplay_RefusesDeletedTarget(t *testing.T) {
wh := seedWebhook(t, db)
tgt := seedConfiguredTarget(
t, db, wh.ID, database.TargetTypeHTTP,
database.Target{URL: replayTargetURL},
`{"url":"`+replayTargetURL+`"}`,
)
event, original := seedFailedDelivery(
@@ -355,6 +377,10 @@ func TestHandleDeliveryReplay_RefusesDeletedTarget(t *testing.T) {
t, listDeliveries(t, webhookDB, event.ID), 1,
"a refused replay must create no delivery",
)
assert.Empty(
t, notif.Tasks(),
"a refused replay must queue nothing",
)
// The refusal is specific, which is why the target is looked up
// including soft-deleted rows: an id that never named a target
@@ -387,9 +413,10 @@ func TestHandleDeliveryReplay_RefusesWhileEarlierReplayInFlight(
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
notif *recordingNotifier
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app := newTestApp(t, &h, &sess, &db, &dbMgr, &notif)
app.RequireStart()
t.Cleanup(app.RequireStop)
@@ -397,7 +424,7 @@ func TestHandleDeliveryReplay_RefusesWhileEarlierReplayInFlight(
wh := seedWebhook(t, db)
tgt := seedConfiguredTarget(
t, db, wh.ID, database.TargetTypeHTTP,
database.Target{URL: replayTargetURL},
`{"url":"`+replayTargetURL+`"}`,
)
event, original := seedFailedDelivery(
@@ -427,6 +454,10 @@ func TestHandleDeliveryReplay_RefusesWhileEarlierReplayInFlight(
t, listDeliveries(t, webhookDB, event.ID), 2,
"the refused second replay must add nothing",
)
assert.Len(
t, notif.Tasks(), 1,
"only the first replay reaches the delivery engine",
)
// A delivery the engine has not finished is not replayable
// either, which is the same rule seen from the other side.
@@ -467,7 +498,7 @@ func TestHandleDeliveryReplay_ReturnsToTheListItWasPressedIn(
wh := seedWebhook(t, db)
tgt := seedConfiguredTarget(
t, db, wh.ID, database.TargetTypeHTTP,
database.Target{URL: replayTargetURL},
`{"url":"`+replayTargetURL+`"}`,
)
_, original := seedFailedDelivery(t, dbMgr, wh.ID, tgt.ID)
@@ -525,7 +556,7 @@ func TestHandleSourceLogs_RendersReplayControlAndBanner(t *testing.T) {
wh := seedWebhook(t, db)
tgt := seedConfiguredTarget(
t, db, wh.ID, database.TargetTypeHTTP,
database.Target{URL: replayTargetURL},
`{"url":"`+replayTargetURL+`"}`,
)
_, original := seedFailedDelivery(t, dbMgr, wh.ID, tgt.ID)
@@ -584,7 +615,7 @@ func TestHandleDeliveryReplay_LabelsTheReplay(t *testing.T) {
wh := seedWebhook(t, db)
tgt := seedConfiguredTarget(
t, db, wh.ID, database.TargetTypeHTTP,
database.Target{URL: replayTargetURL},
`{"url":"`+replayTargetURL+`"}`,
)
event, original := seedFailedDelivery(t, dbMgr, wh.ID, tgt.ID)
internal/handlers/delivery_result_view_test.go
+12 -12
View File
@@ -86,12 +86,12 @@ func seedFailedDeliveryWithResponse(
}
// seedFailureAndRender seeds a failed delivery against a
// target of the given type and URL, and returns the
// target of the given type and config, and returns the
// rendered event log page.
func seedFailureAndRender(
t *testing.T,
targetType database.TargetType,
targetURL, responseBody string,
config, responseBody string,
) string {
t.Helper()
@@ -109,7 +109,7 @@ func seedFailureAndRender(
wh := seedWebhook(t, db)
tgt := seedConfiguredTarget(
t, db, wh.ID, targetType, database.Target{URL: targetURL},
t, db, wh.ID, targetType, config,
)
seedFailedDeliveryWithResponse(
@@ -129,7 +129,7 @@ func TestHandleSourceLogs_RendersFailedAttempt(t *testing.T) {
body := seedFailureAndRender(
t,
database.TargetTypeHTTP,
"https://example.com/hook/abc",
`{"url":"https://example.com/hook/abc"}`,
"upstream exploded",
)
@@ -167,7 +167,7 @@ func TestHandleSourceLogs_EscapesResponseBody(t *testing.T) {
body := seedFailureAndRender(
t,
database.TargetTypeHTTP,
"https://example.com/hook/abc",
`{"url":"https://example.com/hook/abc"}`,
payload,
)
@@ -189,7 +189,7 @@ func TestHandleSourceLogs_RedactsCredentialEchoedInResponse(
body := seedFailureAndRender(
t,
database.TargetTypeSlack,
slackWebhookURL,
`{"webhookUrl":"`+slackWebhookURL+`"}`,
"no_service: "+slackWebhookURL,
)
@@ -228,7 +228,7 @@ func TestHandleSourceLogs_RedactsCredentialEchoedInError(
tgt := seedConfiguredTarget(
t, db, wh.ID,
database.TargetTypeSlack,
database.Target{URL: slackWebhookURL},
`{"webhookUrl":"`+slackWebhookURL+`"}`,
)
dlv := seedFailedDeliveryWithResponse(t, dbMgr, wh.ID, tgt.ID, "")
@@ -300,7 +300,7 @@ func TestHandleSourceLogs_RedactsCredentialSeveredByTheEngineCut(
body := seedFailureAndRender(
t,
database.TargetTypeSlack,
slackWebhookURL,
`{"webhookUrl":"`+slackWebhookURL+`"}`,
stored,
)
@@ -335,7 +335,7 @@ func TestHandleSourceLogs_RedactsCredentialSeveredBySQLCut(
body := seedFailureAndRender(
t,
database.TargetTypeSlack,
slackWebhookURL,
`{"webhookUrl":"`+slackWebhookURL+`"}`,
stored,
)
@@ -373,7 +373,7 @@ func TestHandleSourceLogs_RedactsForSoftDeletedTarget(
tgt := seedConfiguredTarget(
t, db, wh.ID,
database.TargetTypeSlack,
database.Target{URL: slackWebhookURL},
`{"webhookUrl":"`+slackWebhookURL+`"}`,
)
seedFailedDeliveryWithResponse(
@@ -413,7 +413,7 @@ func TestHandleSourceLogs_BoundsRenderedAttempts(t *testing.T) {
wh := seedWebhook(t, db)
tgt := seedConfiguredTarget(
t, db, wh.ID, database.TargetTypeLog, database.Target{},
t, db, wh.ID, database.TargetTypeLog, "",
)
dlv := seedFailedDeliveryWithResponse(t, dbMgr, wh.ID, tgt.ID, "")
@@ -480,7 +480,7 @@ func TestHandleSourceLogs_BoundsOversizeResponse(t *testing.T) {
wh := seedWebhook(t, db)
tgt := seedConfiguredTarget(
t, db, wh.ID, database.TargetTypeLog, database.Target{},
t, db, wh.ID, database.TargetTypeLog, "",
)
stored := strings.Repeat("A", responseCap*4) + tail
internal/handlers/event_log.go
+3 -3
View File
@@ -13,8 +13,8 @@ import (
// DeliveryView is the display-safe projection of a delivery
// for the event log page. Its target is a TargetView, so the
// target's URL and headers — which hold the target's
// credential — have no path to the template.
// stored configuration blob — which holds the target's
// credential — has no path to the template.
type DeliveryView struct {
ID string
Status database.DeliveryStatus
@@ -126,7 +126,7 @@ func (h *Handlers) HandleSourceLogs() http.HandlerFunc {
// loadTargetMap loads targets into a map of display-safe
// views keyed by target ID, each paired with its redactor.
// The projection happens here so that no caller can hand a
// raw target, URL and headers and all, to a template: the
// raw target, configuration blob and all, to a template: the
// raw rows do not leave this function.
//
// The load is Unscoped because deleting a target only soft
internal/handlers/event_resubmit.go
+21 -13
View File
@@ -171,29 +171,37 @@ func loadResubmitSource(
return src, true, nil
}
// queueResubmit has the delivery engine store the copy, with a delivery
// for each of the webhook's active targets as they stand now, through
// the same StoreEvent the receiver uses. The one difference is that the
// copy leaves its entrypoint's last event as it is, since it did not
// arrive on that URL.
// queueResubmit stores the copy and fans it out to the webhook's
// active targets.
func (h *Handlers) queueResubmit(
w http.ResponseWriter,
r *http.Request,
webhook database.Webhook,
src resubmitSource,
) {
event := &database.Event{
// The receiver's own query, run now: an active target created
// after the original event arrived is included, and an
// inactive one is skipped rather than refused.
targets, err := h.loadActiveTargets(webhook.ID)
if err != nil {
h.serverError(w, r, "failed to query targets", err)
return
}
event, tasks, err := h.createAndFanOut(
eventSource{
WebhookID: webhook.ID,
EntrypointID: src.EntrypointID,
Method: src.Method,
RawQuery: src.RawQuery,
Headers: src.Headers,
HeadersJSON: src.Headers,
ContentType: src.ContentType,
Body: string(src.Body),
Body: src.Body,
ResubmittedFromID: &src.ID,
}
deliveries, err := h.engine.StoreEvent(event)
},
targets,
)
if err != nil {
h.serverError(w, r, "failed to store resubmitted event", err)
@@ -207,11 +215,11 @@ func (h *Handlers) queueResubmit(
"webhook_id", webhook.ID,
"event_id", event.ID,
"resubmitted_from_id", src.ID,
"target_count", deliveries,
"target_count", len(tasks),
)
code := resubmitQueued
if deliveries == 0 {
if len(tasks) == 0 {
code = resubmitNoTargets
}
internal/handlers/event_resubmit_test.go
+100 -79
View File
@@ -19,7 +19,7 @@ import (
// resubmitTargetURL is a public destination, so a target configured
// with it is one the SSRF guard would accept. Nothing in these tests
// dispatches to it: the delivery engine is never started.
// dispatches to it: the notifier is recorded, not run.
const resubmitTargetURL = "http://93.184.216.34/hook"
// resubmitEventHeaders and resubmitEventQuery are the stored header
@@ -134,9 +134,10 @@ func TestHandleEventResubmit_DeliversToTargetCreatedAfterTheEvent(
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
notif *recordingNotifier
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app := newTestApp(t, &h, &sess, &db, &dbMgr, &notif)
app.RequireStart()
t.Cleanup(app.RequireStop)
@@ -150,7 +151,7 @@ func TestHandleEventResubmit_DeliversToTargetCreatedAfterTheEvent(
// captured. It has no prior delivery of anything.
tgt := seedConfiguredTarget(
t, db, wh.ID, database.TargetTypeHTTP,
database.Target{URL: resubmitTargetURL},
`{"url":"`+resubmitTargetURL+`"}`,
)
w := postResubmit(t, h, sess, wh.ID, original.ID)
@@ -188,6 +189,11 @@ func TestHandleEventResubmit_DeliversToTargetCreatedAfterTheEvent(
t, database.DeliveryStatusPending, deliveries[0].Status,
)
tasks := notif.Tasks()
require.Len(t, tasks, 1)
assert.Equal(t, deliveries[0].ID, tasks[0].DeliveryID)
assertResubmitTask(t, tasks[0], wh.ID, &fresh, tgt)
assertNoLeakedTarget(t, webhookDB)
}
@@ -219,6 +225,32 @@ func assertEventCopy(
)
}
// assertResubmitTask proves the task handed to the delivery engine is
// the one the receiver would build for the NEW event and this target.
func assertResubmitTask(
t *testing.T,
task delivery.Task,
webhookID string,
fresh *database.Event,
target *database.Target,
) {
t.Helper()
assert.Equal(t, fresh.ID, task.EventID)
assert.Equal(t, webhookID, task.WebhookID)
assert.Equal(t, fresh.EntrypointID, task.EntrypointID)
assert.Equal(t, target.ID, task.TargetID)
assert.Equal(t, target.Type, task.TargetType)
assert.Equal(t, fresh.Method, task.Method)
assert.Equal(t, fresh.RawQuery, task.RawQuery)
assert.Equal(t, fresh.Headers, task.Headers)
assert.Equal(t, fresh.ContentType, task.ContentType)
assert.Equal(t, 1, task.AttemptNum)
require.NotNil(t, task.Body)
assert.Equal(t, fresh.Body, *task.Body)
}
// TestHandleEventResubmit_IsRepeatable proves the requirement replay
// deliberately does not meet: firing the same captured event at a
// backend over and over must work, with no in-flight refusal, and each
@@ -231,9 +263,10 @@ func TestHandleEventResubmit_IsRepeatable(t *testing.T) {
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
notif *recordingNotifier
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app := newTestApp(t, &h, &sess, &db, &dbMgr, &notif)
app.RequireStart()
t.Cleanup(app.RequireStop)
@@ -241,7 +274,7 @@ func TestHandleEventResubmit_IsRepeatable(t *testing.T) {
wh := seedWebhook(t, db)
tgt := seedConfiguredTarget(
t, db, wh.ID, database.TargetTypeHTTP,
database.Target{URL: resubmitTargetURL},
`{"url":"`+resubmitTargetURL+`"}`,
)
original := seedStoredEvent(t, dbMgr, wh.ID, `{"fire":"again"}`)
@@ -267,18 +300,30 @@ func TestHandleEventResubmit_IsRepeatable(t *testing.T) {
require.NoError(t, err)
events := listEvents(t, webhookDB)
require.Len(
t, events, presses+1, "each resubmit creates its own event",
)
require.Len(t, events, presses+1)
assert.Empty(
t, listDeliveries(t, webhookDB, original.ID),
tasks := notif.Tasks()
require.Len(t, tasks, presses)
seen := make(map[string]struct{}, presses)
for _, task := range tasks {
assert.Equal(t, tgt.ID, task.TargetID)
assert.NotEqual(
t, original.ID, task.EventID,
"each resubmit delivers its own new event",
)
// Every copy has its own delivery, and names the same source
// event, so twenty presses stay traceable to the one captured
// request.
_, dup := seen[task.EventID]
assert.False(t, dup, "each resubmit creates its own event")
seen[task.EventID] = struct{}{}
require.Len(t, listDeliveries(t, webhookDB, task.EventID), 1)
}
// Every copy names the same source event, so twenty presses stay
// traceable to the one captured request.
for _, e := range events {
if e.ID == original.ID {
continue
@@ -286,10 +331,6 @@ func TestHandleEventResubmit_IsRepeatable(t *testing.T) {
require.NotNil(t, e.ResubmittedFromID)
assert.Equal(t, original.ID, *e.ResubmittedFromID)
deliveries := listDeliveries(t, webhookDB, e.ID)
require.Len(t, deliveries, 1)
assert.Equal(t, tgt.ID, deliveries[0].TargetID)
}
}
@@ -305,9 +346,10 @@ func TestHandleEventResubmit_OversizeBodySurvivesIntact(t *testing.T) {
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
notif *recordingNotifier
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app := newTestApp(t, &h, &sess, &db, &dbMgr, &notif)
app.RequireStart()
t.Cleanup(app.RequireStop)
@@ -326,7 +368,7 @@ func TestHandleEventResubmit_OversizeBodySurvivesIntact(t *testing.T) {
wh := seedWebhook(t, db)
seedConfiguredTarget(
t, db, wh.ID, database.TargetTypeHTTP,
database.Target{URL: resubmitTargetURL},
`{"url":"`+resubmitTargetURL+`"}`,
)
original := seedStoredEvent(t, dbMgr, wh.ID, stored)
@@ -343,7 +385,22 @@ func TestHandleEventResubmit_OversizeBodySurvivesIntact(t *testing.T) {
t, stored, fresh.Body,
"the stored body must be copied byte for byte",
)
assert.Len(t, listDeliveries(t, webhookDB, fresh.ID), 1)
tasks := notif.Tasks()
require.Len(t, tasks, 1)
assert.Nil(
t, tasks[0].Body,
"a body over the inline limit is fetched from the new "+
"event row rather than carried on the task",
)
// The engine's own read of the body, against the new event id:
// what it would send is what was stored.
var loaded database.Event
require.NoError(t, webhookDB.Select("body").
First(&loaded, "id = ?", tasks[0].EventID).Error)
assert.Equal(t, stored, loaded.Body)
}
// TestHandleEventResubmit_SkipsInactiveTarget proves a deactivated
@@ -357,9 +414,10 @@ func TestHandleEventResubmit_SkipsInactiveTarget(t *testing.T) {
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
notif *recordingNotifier
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app := newTestApp(t, &h, &sess, &db, &dbMgr, &notif)
app.RequireStart()
t.Cleanup(app.RequireStop)
@@ -367,11 +425,11 @@ func TestHandleEventResubmit_SkipsInactiveTarget(t *testing.T) {
wh := seedWebhook(t, db)
active := seedConfiguredTarget(
t, db, wh.ID, database.TargetTypeHTTP,
database.Target{URL: resubmitTargetURL},
`{"url":"`+resubmitTargetURL+`"}`,
)
off := seedConfiguredTarget(
t, db, wh.ID, database.TargetTypeHTTP,
database.Target{URL: resubmitTargetURL + "/off"},
`{"url":"`+resubmitTargetURL+`/off"}`,
)
require.NoError(t, db.DB().Model(&database.Target{}).
@@ -390,19 +448,21 @@ func TestHandleEventResubmit_SkipsInactiveTarget(t *testing.T) {
"an inactive target is skipped, not an error",
)
tasks := notif.Tasks()
require.Len(t, tasks, 1)
assert.Equal(t, active.ID, tasks[0].TargetID)
webhookDB, err := dbMgr.GetDB(wh.ID)
require.NoError(t, err)
fresh := theOtherEvent(t, listEvents(t, webhookDB), original.ID)
deliveries := listDeliveries(t, webhookDB, fresh.ID)
require.Len(t, deliveries, 1)
assert.Equal(t, active.ID, deliveries[0].TargetID)
require.Len(t, listDeliveries(t, webhookDB, fresh.ID), 1)
}
// TestHandleEventResubmit_NoActiveTargetsStillStoresEvent proves a
// source with nothing to deliver to behaves as the receiver does: the
// event is stored with no delivery, and the operator is told so rather
// than being shown an error.
// event is stored, nothing is queued, and the operator is told so
// rather than being shown an error.
func TestHandleEventResubmit_NoActiveTargetsStillStoresEvent(
t *testing.T,
) {
@@ -413,9 +473,10 @@ func TestHandleEventResubmit_NoActiveTargetsStillStoresEvent(
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
notif *recordingNotifier
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app := newTestApp(t, &h, &sess, &db, &dbMgr, &notif)
app.RequireStart()
t.Cleanup(app.RequireStop)
@@ -432,14 +493,12 @@ func TestHandleEventResubmit_NoActiveTargetsStillStoresEvent(
w.Header().Get("Location"),
)
assert.Empty(t, notif.Tasks())
webhookDB, err := dbMgr.GetDB(wh.ID)
require.NoError(t, err)
events := listEvents(t, webhookDB)
assert.Len(t, events, 2)
fresh := theOtherEvent(t, events, original.ID)
assert.Empty(t, listDeliveries(t, webhookDB, fresh.ID))
assert.Len(t, listEvents(t, webhookDB), 2)
}
// TestHandleEventResubmit_RefusesEventOfAnotherWebhook proves the
@@ -455,9 +514,10 @@ func TestHandleEventResubmit_RefusesEventOfAnotherWebhook(
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
notif *recordingNotifier
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app := newTestApp(t, &h, &sess, &db, &dbMgr, &notif)
app.RequireStart()
t.Cleanup(app.RequireStop)
@@ -468,7 +528,7 @@ func TestHandleEventResubmit_RefusesEventOfAnotherWebhook(
mine := seedWebhook(t, db)
seedConfiguredTarget(
t, db, mine.ID, database.TargetTypeHTTP,
database.Target{URL: resubmitTargetURL},
`{"url":"`+resubmitTargetURL+`"}`,
)
seedStoredEvent(t, dbMgr, mine.ID, `{"mine":true}`)
@@ -488,50 +548,11 @@ func TestHandleEventResubmit_RefusesEventOfAnotherWebhook(
w = postResubmit(t, h, sess, mine.ID, "not-a-uuid")
assert.Equal(t, http.StatusNotFound, w.Code)
for _, webhookID := range []string{theirs.ID, mine.ID} {
webhookDB, err := dbMgr.GetDB(webhookID)
require.NoError(t, err)
assert.Len(
t, listEvents(t, webhookDB), 1,
"a refused resubmit must store nothing",
assert.Empty(
t, notif.Tasks(),
"a refused resubmit must queue nothing",
)
}
}
// TestHandleEventResubmit_FailedStoreIsAnError proves a copy the engine
// cannot store answers with the 500 error page and leaves no new event.
func TestHandleEventResubmit_FailedStoreIsAnError(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
original := seedStoredEvent(t, dbMgr, wh.ID, `{"not":"stored"}`)
// Without its event totals table the webhook's database refuses
// the totals write, which comes after the event row.
webhookDB, err := dbMgr.GetDB(wh.ID)
require.NoError(t, err)
require.NoError(t,
webhookDB.Migrator().DropTable(&database.EventTotals{}))
w := postResubmit(t, h, sess, wh.ID, original.ID)
assert.Equal(t, http.StatusInternalServerError, w.Code)
assert.Contains(t, w.Body.String(), "500 Internal Server Error")
assert.Len(t, listEvents(t, webhookDB), 1)
}
// TestHandleSourceLogs_ShowsResubmitProvenance proves the event log
// reports the relationship in both directions, which is what keeps it
@@ -554,7 +575,7 @@ func TestHandleSourceLogs_ShowsResubmitProvenance(t *testing.T) {
wh := seedWebhook(t, db)
seedConfiguredTarget(
t, db, wh.ID, database.TargetTypeHTTP,
database.Target{URL: resubmitTargetURL},
`{"url":"`+resubmitTargetURL+`"}`,
)
original := seedStoredEvent(t, dbMgr, wh.ID, `{"trace":"me"}`)
internal/handlers/export_test.go
+25 -17
View File
@@ -133,28 +133,36 @@ func (s *Handlers) RenderTemplateForTest(
s.renderTemplate(w, r, pageTemplate, data)
}
// SetSlackTargetSettingsForTest exposes setSlackTargetSettings for use
// in the handlers_test package.
func (s *Handlers) SetSlackTargetSettingsForTest(
// BuildSlackTargetConfigForTest exposes
// buildSlackTargetConfig for use in the handlers_test package.
func (s *Handlers) BuildSlackTargetConfigForTest(
ctx context.Context,
target *database.Target,
targetURL string,
) string {
return s.setSlackTargetSettings(ctx, target, targetURL)
) (string, string, error) {
return s.buildSlackTargetConfig(ctx, targetURL)
}
// SetDatabaseTargetSettingsForTest exposes setDatabaseTargetSettings
// for use in the handlers_test package.
func SetDatabaseTargetSettingsForTest(
target *database.Target,
// BuildHTTPTargetConfigForTest exposes buildHTTPTargetConfig
// for use in the handlers_test package, taking the form fields
// an HTTP target's configuration is built from.
func (s *Handlers) BuildHTTPTargetConfigForTest(
ctx context.Context,
targetURL, headers, timeout string,
) (string, string, error) {
return s.buildHTTPTargetConfig(ctx, targetFormInput{
URL: targetURL,
Headers: headers,
Timeout: timeout,
})
}
// BuildDatabaseTargetConfigForTest exposes
// buildDatabaseTargetConfig for use in the handlers_test
// package.
func BuildDatabaseTargetConfigForTest(
expiry, rotation string,
) string {
return setDatabaseTargetSettings(target, expiry, rotation)
}
// FormatArchiveExpiryForTest exposes formatArchiveExpiry.
func FormatArchiveExpiryForTest(expiry time.Duration) string {
return formatArchiveExpiry(expiry)
) (string, string, error) {
return buildDatabaseTargetConfig(expiry, rotation)
}
// ArchiveFileViewForTest exposes archiveFileView, which describes a
internal/handlers/handlers.go
+3 -3
View File
@@ -64,7 +64,7 @@ type HandlersParams struct {
Healthcheck *healthcheck.Healthcheck
Session *session.Session
Middleware *middleware.Middleware
Engine *delivery.Engine
Notifier delivery.Notifier
Archives archive.Archives
CircuitBreakers delivery.CircuitBreakers
SSRFGuard *delivery.Guard
@@ -82,7 +82,7 @@ type Handlers struct {
dbMgr *database.WebhookDBManager
session *session.Session
mw *middleware.Middleware
engine *delivery.Engine
notifier delivery.Notifier
archives archive.Archives
breakers delivery.CircuitBreakers
mtr *metrics.Set
@@ -138,7 +138,7 @@ func New(
s.dbMgr = params.WebhookDBMgr
s.session = params.Session
s.mw = params.Middleware
s.engine = params.Engine
s.notifier = params.Notifier
s.archives = params.Archives
s.breakers = params.CircuitBreakers
s.mtr = params.Metrics
internal/handlers/handlers_test.go
+100 -105
View File
@@ -8,7 +8,6 @@ import (
"log/slog"
"net/http"
"net/http/httptest"
"strconv"
"sync"
"testing"
"time"
@@ -30,6 +29,33 @@ import (
"sneak.berlin/go/webhooker/internal/session"
)
// recordingNotifier is a delivery.Notifier that records the tasks it
// was handed, so a test can prove a handler queued the delivery it
// claims to have queued — and, on the refusal paths, that it queued
// nothing.
type recordingNotifier struct {
mu sync.Mutex
tasks []delivery.Task
}
func (n *recordingNotifier) Notify(tasks []delivery.Task) {
n.mu.Lock()
defer n.mu.Unlock()
n.tasks = append(n.tasks, tasks...)
}
// Tasks returns a copy of the recorded tasks.
func (n *recordingNotifier) Tasks() []delivery.Task {
n.mu.Lock()
defer n.mu.Unlock()
out := make([]delivery.Task, len(n.tasks))
copy(out, n.tasks)
return out
}
// recordingArchives is an archive.Archives that records what it
// was asked to do, so a test can prove that a deletion or rename
// path reached the archive writers. After FailRenames, every
@@ -237,12 +263,11 @@ func newTestAppWithConfig(
database.NewWebhookDBManager,
healthcheck.New,
session.New,
archive.NewWriters,
// The engine is built on a lifecycle that is never started,
// so no worker runs: every delivery the engine creates for a
// handler stays pending, where the test can read it.
func(p delivery.EngineParams) *delivery.Engine {
return delivery.New(fxtest.NewLifecycle(t), p)
func() *recordingNotifier {
return &recordingNotifier{}
},
func(n *recordingNotifier) delivery.Notifier {
return n
},
func() *recordingArchives {
return &recordingArchives{}
@@ -329,7 +354,7 @@ func TestHandleIndex_Authenticated(t *testing.T) {
)
}
func TestSetSlackTargetSettings_AcceptsPublicURL(t *testing.T) {
func TestBuildSlackTargetConfig_AcceptsPublicURL(t *testing.T) {
t.Parallel()
var h *handlers.Handlers
@@ -339,34 +364,32 @@ func TestSetSlackTargetSettings_AcceptsPublicURL(t *testing.T) {
t.Cleanup(app.RequireStop)
const hookURL = "http://93.184.216.34/services/T00/B00/xxx"
var target database.Target
errMsg := h.SetSlackTargetSettingsForTest(t.Context(), &target, hookURL)
assert.Empty(t, errMsg)
assert.Equal(t, hookURL, target.URL)
}
func TestSetSlackTargetSettings_RejectsReservedURL(t *testing.T) {
t.Parallel()
var h *handlers.Handlers
app := newTestApp(t, &h)
app.RequireStart()
t.Cleanup(app.RequireStop)
var target database.Target
errMsg := h.SetSlackTargetSettingsForTest(
t.Context(), &target, "http://169.254.169.254/latest/meta-data/",
cfg, errMsg, err := h.BuildSlackTargetConfigForTest(
t.Context(), "http://93.184.216.34/services/T00/B00/xxx",
)
require.NoError(t, err)
assert.Empty(t, errMsg)
assert.Contains(t, cfg, "webhookUrl")
}
func TestBuildSlackTargetConfig_RejectsReservedURL(t *testing.T) {
t.Parallel()
var h *handlers.Handlers
app := newTestApp(t, &h)
app.RequireStart()
t.Cleanup(app.RequireStop)
cfg, errMsg, err := h.BuildSlackTargetConfigForTest(
t.Context(), "http://169.254.169.254/latest/meta-data/",
)
require.NoError(t, err)
assert.Contains(t, errMsg, "Invalid target URL")
assert.Empty(t, target.URL)
assert.Empty(t, cfg)
}
func TestRenderTemplate(t *testing.T) {
@@ -460,100 +483,72 @@ func TestRenderTemplateMidRenderErrorSendsNoPartialBody(t *testing.T) {
)
}
func TestSetDatabaseTargetSettings_Valid(t *testing.T) {
func TestBuildDatabaseTargetConfig_Valid(t *testing.T) {
t.Parallel()
cases := []struct {
expiry, rotation string
wantExpiry time.Duration
wantRotation string
}{
// Empty expiry and rotation: the keep-forever, one-file
// default.
{"", "", 0, rotationNone},
{expiryNever, "", 0, rotationNone},
{" 36h ", "", 36 * time.Hour, rotationNone},
{"", rotationDaily, 0, rotationDaily},
{"90m", rotationHourly, 90 * time.Minute, rotationHourly},
}
// Empty expiry and rotation: the keep-forever, one-file default,
// empty config.
cfg, errMsg, err := handlers.BuildDatabaseTargetConfigForTest("", "")
require.NoError(t, err)
assert.Empty(t, errMsg)
assert.Empty(t, cfg)
for _, tc := range cases {
var target database.Target
// Explicit never is stored as config.
cfg, errMsg, err = handlers.BuildDatabaseTargetConfigForTest("never", "")
require.NoError(t, err)
assert.Empty(t, errMsg)
assert.JSONEq(t, `{"expiry":"never"}`, cfg)
errMsg := handlers.SetDatabaseTargetSettingsForTest(
&target, tc.expiry, tc.rotation,
// A positive duration is stored as config.
cfg, errMsg, err = handlers.BuildDatabaseTargetConfigForTest("720h", "")
require.NoError(t, err)
assert.Empty(t, errMsg)
assert.JSONEq(t, `{"expiry":"720h"}`, cfg)
// A rotation is stored as config, with or without an expiry.
cfg, errMsg, err = handlers.BuildDatabaseTargetConfigForTest("", "daily")
require.NoError(t, err)
assert.Empty(t, errMsg)
assert.JSONEq(t, `{"rotation":"daily"}`, cfg)
cfg, errMsg, err = handlers.BuildDatabaseTargetConfigForTest(
"720h", "hourly",
)
assert.Empty(t, errMsg, tc)
assert.Equal(t, tc.wantExpiry, target.ArchiveExpiry, tc)
assert.Equal(t, tc.wantRotation, target.ArchiveRotation, tc)
}
require.NoError(t, err)
assert.Empty(t, errMsg)
assert.JSONEq(t, `{"expiry":"720h","rotation":"hourly"}`, cfg)
}
// TestSetDatabaseTargetSettings_RejectsBadExpiry pins each refusal's
// whole message, as the form shows it.
func TestSetDatabaseTargetSettings_RejectsBadExpiry(
func TestBuildDatabaseTargetConfig_RejectsBadExpiry(
t *testing.T,
) {
t.Parallel()
cases := map[string]string{
"nonsense": `Invalid archive expiry: expiry must be "never" or ` +
`a Go duration such as "720h": time: invalid duration ` +
`"nonsense"`,
"7d": `Invalid archive expiry: expiry must be "never" or a Go ` +
`duration such as "720h": time: unknown unit "d" in ` +
`duration "7d"`,
"-5h": `Invalid archive expiry: expiry must be a positive ` +
`duration or "never": "-5h"`,
}
for _, bad := range []string{"nonsense", "7d", "-5h"} {
cfg, errMsg, err := handlers.BuildDatabaseTargetConfigForTest(bad, "")
for bad, want := range cases {
var target database.Target
errMsg := handlers.SetDatabaseTargetSettingsForTest(
&target, bad, "",
require.NoError(t, err)
assert.Contains(
t, errMsg, "Invalid archive expiry",
"expiry %q should be refused", bad,
)
assert.Equal(t, want, errMsg, "expiry %q", bad)
assert.Empty(t, cfg)
}
}
func TestSetDatabaseTargetSettings_RejectsBadRotation(
func TestBuildDatabaseTargetConfig_RejectsBadRotation(
t *testing.T,
) {
t.Parallel()
for _, bad := range []string{"weekly", "Daily", " none"} {
var target database.Target
cfg, errMsg, err := handlers.BuildDatabaseTargetConfigForTest("", bad)
errMsg := handlers.SetDatabaseTargetSettingsForTest(
&target, "", bad,
)
assert.Equal(
t,
"Invalid archive rotation: rotation must be none, "+
"monthly, daily or hourly: "+strconv.Quote(bad),
errMsg,
require.NoError(t, err)
assert.Contains(
t, errMsg, "Invalid archive rotation",
"rotation %q should be refused", bad,
)
}
}
// TestFormatArchiveExpiry covers the expiries the expiry choices do
// not: TestArchiveExpiryChoices shows the edit form starting on each
// choice. An expiry in whole hours is written in hours, and any other
// as Go writes it, which the form accepts back unchanged.
func TestFormatArchiveExpiry(t *testing.T) {
t.Parallel()
cases := map[time.Duration]string{
0: expiryNever,
36 * time.Hour: "36h",
90 * time.Minute: "1h30m0s",
}
for expiry, want := range cases {
assert.Equal(t, want, handlers.FormatArchiveExpiryForTest(expiry))
assert.Empty(t, cfg)
}
}
internal/handlers/source_create_targets_test.go
+12 -23
View File
@@ -7,7 +7,6 @@ import (
"net/http/httptest"
"net/url"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -55,26 +54,15 @@ func TestHandleSourceCreateSubmit_CreatesRequestedTargets(t *testing.T) {
env := setupSourceTest(t)
// Each value the archive pruning choice submits and the expiry it
// stores, after an empty one that stands for the archive checkbox
// left off.
expiries := []struct {
value string
stored time.Duration
}{
{"", 0},
{expiryNever, 0},
{"1h", time.Hour},
{"12h", 12 * time.Hour},
{"24h", 24 * time.Hour},
{"720h", 720 * time.Hour},
{"2160h", 2160 * time.Hour},
{"8760h", 8760 * time.Hour},
// Each value the archive pruning choice submits, after an empty
// one that stands for the archive checkbox left off.
expiries := []string{
"", "never", "1h", "12h", "24h", "720h", "2160h", "8760h",
}
for _, httpURL := range []string{"", editOriginalURL} {
for _, expiry := range expiries {
name := "url=" + httpURL + " archive=" + expiry.value
name := "url=" + httpURL + " archive=" + expiry
t.Run(name, func(t *testing.T) {
t.Parallel()
@@ -84,9 +72,9 @@ func TestHandleSourceCreateSubmit_CreatesRequestedTargets(t *testing.T) {
form.Set("http_url", httpURL)
form.Set("archive_expiry", "720h")
if expiry.value != "" {
if expiry != "" {
form.Set("archive", "on")
form.Set("archive_expiry", expiry.value)
form.Set("archive_expiry", expiry)
}
w := submitCreateForm(env, form)
@@ -108,16 +96,17 @@ func TestHandleSourceCreateSubmit_CreatesRequestedTargets(t *testing.T) {
wantCount++
assert.Equal(t, "HTTP", byType[database.TargetTypeHTTP].Name)
assert.Equal(t, httpURL, byType[database.TargetTypeHTTP].URL)
assert.JSONEq(t, `{"url":"`+httpURL+`"}`,
byType[database.TargetTypeHTTP].Config)
}
if expiry.value != "" {
if expiry != "" {
wantCount++
assert.Equal(t, "Archive",
byType[database.TargetTypeDatabase].Name)
assert.Equal(t, expiry.stored,
byType[database.TargetTypeDatabase].ArchiveExpiry)
assert.JSONEq(t, `{"expiry":"`+expiry+`"}`,
byType[database.TargetTypeDatabase].Config)
}
assert.Len(t, byType, wantCount)
internal/handlers/source_detail_test.go
+69 -45
View File
@@ -5,7 +5,6 @@ import (
"net/http"
"net/http/httptest"
"testing"
"time"
"github.com/go-chi/chi"
"github.com/stretchr/testify/assert"
@@ -26,24 +25,25 @@ const (
slackSecretPath
)
// seedConfiguredTarget inserts an active target of targetType with
// the settings (URL to ArchiveRotation) that settings carries, and
// returns it.
// seedConfiguredTarget inserts a target with a stored config
// blob and returns it.
func seedConfiguredTarget(
t *testing.T,
db *database.Database,
webhookID string,
targetType database.TargetType,
settings database.Target,
config string,
) *database.Target {
t.Helper()
tgt := &settings
tgt.WebhookID = webhookID
tgt.Name = "t-" + string(targetType)
tgt.Type = targetType
tgt.Active = true
tgt.MaxAttempts = 1
tgt := &database.Target{
WebhookID: webhookID,
Name: "t-" + string(targetType),
Type: targetType,
Active: true,
Config: config,
MaxAttempts: 1,
}
require.NoError(
t,
@@ -108,22 +108,11 @@ func serveSourceDetailPage(
return w
}
// TestHandleSourceDetail_MasksTargetURL is the load-bearing
// regression test for the credential leak: the rendered page must
// show a Slack target, and an http target, without any of the secret
// path segments of its URL. An http target's destination is routinely
// an incoming-webhook endpoint too, so the same Slack URL serves for
// both.
func TestHandleSourceDetail_MasksTargetURL(t *testing.T) {
t.Parallel()
labels := map[database.TargetType]string{
database.TargetTypeSlack: "Webhook URL",
database.TargetTypeHTTP: "Destination URL",
}
for targetType, label := range labels {
t.Run(string(targetType), func(t *testing.T) {
// TestHandleSourceDetail_MasksSlackWebhookURL is the
// load-bearing regression test for the credential leak: the
// rendered page must show the Slack target without any of the
// secret path segments of its webhook URL.
func TestHandleSourceDetail_MasksSlackWebhookURL(t *testing.T) {
t.Parallel()
var (
@@ -139,8 +128,51 @@ func TestHandleSourceDetail_MasksTargetURL(t *testing.T) {
wh := seedWebhook(t, db)
seedConfiguredTarget(
t, db, wh.ID, targetType,
database.Target{URL: slackWebhookURL},
t, db, wh.ID,
database.TargetTypeSlack,
`{"webhookUrl":"`+slackWebhookURL+`"}`,
)
body := renderSourceDetailPage(t, h, sess, wh.ID)
assert.NotContains(t, body, slackSecretPath)
assert.NotContains(t, body, "T00000000")
assert.NotContains(t, body, "B00000000")
assert.NotContains(
t, body, "XXXXXXXXXXXXXXXXXXXXXXXX",
)
assert.NotContains(t, body, "webhookUrl")
assert.Contains(t, body, "Webhook URL")
assert.Contains(t, body, "https://hooks.slack.com/...")
}
// TestHandleSourceDetail_MasksHTTPDestinationURL is the
// regression test for the same leak reached through the http
// target: its destination is routinely an incoming-webhook
// endpoint whose path segments are the credential, so the
// rendered page must not contain them.
func TestHandleSourceDetail_MasksHTTPDestinationURL(
t *testing.T,
) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
)
app := newTestApp(t, &h, &sess, &db)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
seedConfiguredTarget(
t, db, wh.ID,
database.TargetTypeHTTP,
`{"url":"`+slackWebhookURL+`"}`,
)
body := renderSourceDetailPage(t, h, sess, wh.ID)
@@ -152,15 +184,13 @@ func TestHandleSourceDetail_MasksTargetURL(t *testing.T) {
t, body, "XXXXXXXXXXXXXXXXXXXXXXXX",
)
assert.Contains(t, body, label)
assert.Contains(t, body, "Destination URL")
assert.Contains(t, body, "https://hooks.slack.com/...")
})
}
}
// TestHandleSourceDetail_RendersNamedTargetFields proves the
// other target types render labelled fields rather than the
// stored values.
// stored blob.
func TestHandleSourceDetail_RendersNamedTargetFields(
t *testing.T,
) {
@@ -182,24 +212,18 @@ func TestHandleSourceDetail_RendersNamedTargetFields(
seedConfiguredTarget(
t, db, wh.ID,
database.TargetTypeHTTP,
database.Target{
URL: "https://example.com/hook",
TimeoutSeconds: 30,
Headers: map[string]string{"X-Api-Key": "sekrit"},
},
`{"url":"https://example.com/hook","timeout":30,`+
`"headers":{"Authorization":"Bearer sekrit"}}`,
)
seedConfiguredTarget(
t, db, wh.ID,
database.TargetTypeDatabase,
database.Target{
ArchiveExpiry: 720 * time.Hour,
ArchiveRotation: rotationNone,
},
`{"expiry":"720h"}`,
)
seedConfiguredTarget(
t, db, wh.ID,
database.TargetType("carrier-pigeon"),
database.Target{URL: "https://pigeon.example/beak"},
`{"beak":"sharp"}`,
)
body := renderSourceDetailPage(t, h, sess, wh.ID)
@@ -218,8 +242,8 @@ func TestHandleSourceDetail_RendersNamedTargetFields(
assert.Contains(t, list, "Archive rotation: none")
assert.Contains(t, body, `<option value="database">Archive</option>`)
// An unknown type gets the neutral placeholder, never its
// stored URL.
// An unknown type gets the neutral placeholder, never the
// stored blob.
assert.Contains(t, body, "(unavailable)")
assert.NotContains(t, body, "beak")
}
internal/handlers/source_logs_deleted_target_test.go
+2 -1
View File
@@ -161,7 +161,7 @@ func TestHandleSourceLogs_MasksDeletedTargetConfig(t *testing.T) {
tgt := seedConfiguredTarget(
t, db, wh.ID,
database.TargetTypeSlack,
database.Target{URL: slackWebhookURL},
`{"webhookUrl":"`+slackWebhookURL+`"}`,
)
seedDeliveredEvent(t, dbMgr, wh.ID, tgt.ID)
@@ -175,6 +175,7 @@ func TestHandleSourceLogs_MasksDeletedTargetConfig(t *testing.T) {
assert.NotContains(
t, body, "XXXXXXXXXXXXXXXXXXXXXXXX",
)
assert.NotContains(t, body, "webhookUrl")
// The name is there; only the credential is not.
assert.Contains(t, body, tgt.Name+deletedMarker)
internal/handlers/source_logs_test.go
+2 -1
View File
@@ -138,7 +138,7 @@ func TestHandleSourceLogs_MasksSlackWebhookURL(t *testing.T) {
tgt := seedConfiguredTarget(
t, db, wh.ID,
database.TargetTypeSlack,
database.Target{URL: slackWebhookURL},
`{"webhookUrl":"`+slackWebhookURL+`"}`,
)
seedDeliveredEvent(t, dbMgr, wh.ID, tgt.ID)
@@ -151,6 +151,7 @@ func TestHandleSourceLogs_MasksSlackWebhookURL(t *testing.T) {
assert.NotContains(
t, body, "XXXXXXXXXXXXXXXXXXXXXXXX",
)
assert.NotContains(t, body, "webhookUrl")
// The page still identifies the delivery's target.
assert.Contains(t, body, tgt.Name)
internal/handlers/target_create.go
+108 -73
View File
@@ -2,6 +2,7 @@ package handlers
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/http"
@@ -64,14 +65,20 @@ func (h *Handlers) processTargetCreate(
) {
in := targetFormInputFrom(r)
target, errMsg := h.newTarget(r.Context(), webhook.ID, in)
target, errMsg, err := h.newTarget(r.Context(), webhook.ID, in)
if err != nil {
h.serverError(w, r, "failed to encode target config", err)
return
}
if errMsg != "" {
h.renderSourceDetail(w, r, webhook, in, errMsg)
return
}
err := h.db.DB().Create(target).Error
err = h.db.DB().Create(target).Error
if err != nil {
h.serverError(w, r, "failed to create target", err)
@@ -86,13 +93,15 @@ func (h *Handlers) processTargetCreate(
// newTarget validates a new target for a webhook and returns the row
// to create, or, when it refuses the target, the message the form
// shows. Every form that creates a target goes through here, so they
// all accept and refuse the same things.
// shows. An error is the server's fault, not a refusal: the accepted
// configuration could not be encoded. Every form that creates a
// target goes through here, so they all accept and refuse the same
// things.
func (h *Handlers) newTarget(
ctx context.Context,
webhookID string,
in targetFormInput,
) (*database.Target, string) {
) (*database.Target, string, error) {
target := &database.Target{
WebhookID: webhookID,
Type: in.Type,
@@ -100,33 +109,34 @@ func (h *Handlers) newTarget(
MaxAttempts: 1,
}
errMsg := h.setTargetFromForm(ctx, target, in)
if errMsg != "" {
return nil, errMsg
errMsg, err := h.setTargetFromForm(ctx, target, in)
if err != nil || errMsg != "" {
return nil, errMsg, err
}
return target, ""
return target, "", nil
}
// setTargetFromForm validates a target form against the target's type
// and sets the target's name, settings and number of delivery attempts
// from it. It returns the message the form shows for anything it
// refuses, an unknown type among them; the caller then discards the
// target, which may be partly set.
// and, when it accepts it, sets the target's name, configuration and
// number of delivery attempts from it. It returns the message the form
// shows for anything it refuses, an unknown type among them, and then
// leaves the target unchanged; an error is the server's fault, as for
// newTarget.
// The add target form and the target edit form both go through here,
// so the two cannot come to disagree about what a target may be.
func (h *Handlers) setTargetFromForm(
ctx context.Context,
target *database.Target,
in targetFormInput,
) string {
) (string, error) {
if in.Name == "" {
return "Name is required"
return "Name is required", nil
}
errMsg := h.setTargetSettings(ctx, target, in)
if errMsg != "" {
return errMsg
configJSON, errMsg, err := h.buildTargetConfig(ctx, target.Type, in)
if err != nil || errMsg != "" {
return errMsg, err
}
// An empty max_attempts keeps the target's count: the default of
@@ -137,13 +147,14 @@ func (h *Handlers) setTargetFromForm(
// target is delivering with.
maxAttempts, err := parseMaxAttempts(in.MaxAttempts, target.MaxAttempts)
if err != nil {
return "Invalid delivery attempts: " + attemptsErrorMessage(err)
return "Invalid delivery attempts: " + attemptsErrorMessage(err), nil
}
target.Name = in.Name
target.Config = configJSON
target.MaxAttempts = maxAttempts
return ""
return "", nil
}
// targetFormInput carries the raw values of a target form. Both the
@@ -202,81 +213,84 @@ func targetFormInputFrom(r *http.Request) targetFormInput {
}
}
// setTargetSettings sets a target's settings from the submitted form
// values, or returns the message the form shows for a value it
// refuses. Which fields of in apply depends on the target type; a
// type without a URL ignores any URL submitted, and the settings a
// type does not use are left empty.
func (h *Handlers) setTargetSettings(
// buildTargetConfig builds the JSON config string for a target from
// the submitted form values, or returns the message the form shows
// for a value it refuses. An error is the server's fault, not a
// refusal: the accepted configuration could not be encoded. Which
// fields of in apply depends on the target type; a type without a URL
// ignores any URL submitted.
func (h *Handlers) buildTargetConfig(
ctx context.Context,
target *database.Target,
targetType database.TargetType,
in targetFormInput,
) string {
switch target.Type {
) (string, string, error) {
switch targetType {
case database.TargetTypeHTTP:
return h.setHTTPTargetSettings(ctx, target, in)
return h.buildHTTPTargetConfig(ctx, in)
case database.TargetTypeSlack:
return h.setSlackTargetSettings(ctx, target, in.URL)
return h.buildSlackTargetConfig(ctx, in.URL)
case database.TargetTypeDatabase:
return setDatabaseTargetSettings(target, in.Expiry, in.Rotation)
return buildDatabaseTargetConfig(in.Expiry, in.Rotation)
case database.TargetTypeLog:
return ""
return "", "", nil
default:
return "Invalid target type"
return "", "Invalid target type", nil
}
}
// setHTTPTargetSettings sets an HTTP target's SSRF-validated
// destination plus the optional headers, timeout and query string
// setting the delivery path honours.
func (h *Handlers) setHTTPTargetSettings(
// buildHTTPTargetConfig builds config JSON for an HTTP target: an
// SSRF-validated destination plus the optional headers, timeout and
// query string setting the delivery path honours.
func (h *Handlers) buildHTTPTargetConfig(
ctx context.Context,
target *database.Target,
in targetFormInput,
) string {
) (string, string, error) {
errMsg := h.validateTargetURL(
ctx, in.URL, "URL is required for HTTP targets",
)
if errMsg != "" {
return errMsg
return "", errMsg, nil
}
headers, err := delivery.ParseTargetHeaders(in.Headers)
if err != nil {
return fmt.Sprintf("Invalid headers: %v", err)
return "", fmt.Sprintf("Invalid headers: %v", err), nil
}
timeout, err := delivery.ParseTargetTimeout(in.Timeout)
if err != nil {
return fmt.Sprintf("Invalid timeout: %v", err)
return "", fmt.Sprintf("Invalid timeout: %v", err), nil
}
target.URL = in.URL
target.Headers = headers
target.TimeoutSeconds = timeout
target.ForwardQuery = in.ForwardQuery
configJSON, err := marshalTargetConfig(delivery.HTTPTargetConfig{
URL: in.URL,
Headers: headers,
Timeout: timeout,
ForwardQuery: in.ForwardQuery,
})
return ""
return configJSON, "", err
}
// setSlackTargetSettings sets a Slack target's one setting, its
// SSRF-validated webhook URL.
func (h *Handlers) setSlackTargetSettings(
// buildSlackTargetConfig builds config JSON for a Slack target,
// whose whole configuration is one SSRF-validated webhook URL.
func (h *Handlers) buildSlackTargetConfig(
ctx context.Context,
target *database.Target,
targetURL string,
) string {
) (string, string, error) {
errMsg := h.validateTargetURL(
ctx, targetURL,
"Webhook URL is required for Slack targets",
)
if errMsg != "" {
return errMsg
return "", errMsg, nil
}
target.URL = targetURL
configJSON, err := marshalTargetConfig(delivery.SlackTargetConfig{
WebhookURL: targetURL,
})
return ""
return configJSON, "", err
}
// validateTargetURL refuses an empty or SSRF-blocked destination,
@@ -327,31 +341,52 @@ func (h *Handlers) validateTargetURL(
return ""
}
// setDatabaseTargetSettings sets a database (archive) target's
// expiry and rotation. Both are optional: an empty or never expiry
// keeps the archive forever, and an empty rotation is stored as none,
// one file. A bad value is refused here, at the form, instead of
// being stored.
func setDatabaseTargetSettings(
target *database.Target,
expiry, rotation string,
) string {
archiveExpiry, err := archive.ParseExpiry(strings.TrimSpace(expiry))
// marshalTargetConfig serialises a target configuration for storage.
func marshalTargetConfig(cfg any) (string, error) {
configBytes, err := json.Marshal(cfg)
if err != nil {
return fmt.Sprintf("Invalid archive expiry: %v", err)
return "", err
}
return string(configBytes), nil
}
// buildDatabaseTargetConfig builds config JSON for a database
// (archive) target. The optional expiry and rotation are validated
// here, at creation time, so a bad value is refused instead of
// failing every subsequent delivery. Each is stored only when set,
// and with neither the config is empty (the keep-forever, one-file
// default).
func buildDatabaseTargetConfig(
expiry, rotation string,
) (string, string, error) {
expiry = strings.TrimSpace(expiry)
err := archive.ValidateExpiry(expiry)
if err != nil {
return "", fmt.Sprintf("Invalid archive expiry: %v", err), nil
}
err = archive.ValidateRotation(rotation)
if err != nil {
return fmt.Sprintf("Invalid archive rotation: %v", err)
return "", fmt.Sprintf("Invalid archive rotation: %v", err), nil
}
if rotation == "" {
rotation = archiveRotationNone
cfg := map[string]any{}
if expiry != "" {
cfg["expiry"] = expiry
}
target.ArchiveExpiry = archiveExpiry
target.ArchiveRotation = rotation
if rotation != "" {
cfg["rotation"] = rotation
}
return ""
if len(cfg) == 0 {
return "", "", nil
}
configJSON, err := marshalTargetConfig(cfg)
return configJSON, "", err
}
internal/handlers/target_create_query_test.go
+1 -1
View File
@@ -176,7 +176,7 @@ func TestHandleTargetCreate_BodyURLStillCreatesTheTarget(t *testing.T) {
targets := targetsForWebhook(t, env.db, webhook.ID)
require.Len(t, targets, 1)
assert.Equal(t, targetSecretURL, targets[0].URL)
assert.Contains(t, targets[0].Config, targetSecretSegments)
// The body carried the credential, so the access log must still
// not have it: the log records the request line only.
internal/handlers/target_create_test.go
+22 -63
View File
@@ -6,111 +6,70 @@ import (
"net/url"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/database"
)
// TestHandleTargetCreate_EveryType adds a target of each type and
// checks the settings it stores. Each submission carries a url and an
// expiry: only the http and slack types store the url, only the
// database type the expiry, and every setting a type does not use
// stays empty.
// TestHandleTargetCreate_EveryType adds a target of each type. Each
// submission carries a url: only the http and slack types store one.
func TestHandleTargetCreate_EveryType(t *testing.T) {
t.Parallel()
env := setupSourceTest(t)
// fields is the rest of each submission, as a query string, and
// want holds the settings stored.
// fields is the rest of each submission, as a query string.
cases := []struct {
targetType database.TargetType
fields string
want database.Target
wantConfig string
wantAttempts int
}{
{
database.TargetTypeHTTP,
"timeout=12&max_attempts=3&forward_query=on",
database.Target{
URL: editOriginalURL,
Headers: map[string]string{},
TimeoutSeconds: 12,
ForwardQuery: true,
}, 3,
database.TargetTypeHTTP, "timeout=12&max_attempts=3",
`{"url":"` + editOriginalURL + `","timeout":12}`, 3,
},
{
database.TargetTypeSlack, "max_attempts=4",
database.Target{URL: editOriginalURL}, 4,
`{"webhookUrl":"` + editOriginalURL + `"}`, 4,
},
{
database.TargetTypeDatabase, "rotation=daily",
database.Target{
ArchiveExpiry: 720 * time.Hour,
ArchiveRotation: rotationDaily,
}, 1,
database.TargetTypeDatabase, "expiry=720h",
`{"expiry":"720h"}`, 1,
},
{database.TargetTypeLog, "", database.Target{}, 1},
{database.TargetTypeLog, "", "", 1},
}
for _, tc := range cases {
t.Run(string(tc.targetType), func(t *testing.T) {
t.Parallel()
_, got := addTargetThroughForm(
t, env, tc.targetType,
"url="+editOriginalURL+"&expiry=720h&"+tc.fields,
)
assert.Equal(t, tc.targetType, got.Type)
assert.Equal(t, tc.wantAttempts, got.MaxAttempts)
assert.Equal(t, tc.want, targetSettingsOf(got))
})
}
}
// addTargetThroughForm adds a target of targetType named every-type to
// a new webhook through the add target form, with the rest of the form
// given as a query string, and returns the webhook and the target the
// form stored.
func addTargetThroughForm(
t *testing.T,
env *sourceTestEnv,
targetType database.TargetType,
fields string,
) (database.Webhook, database.Target) {
t.Helper()
webhook := seedWebhookWithRetention(t, env.db, 30)
form, err := url.ParseQuery(fields)
form, err := url.ParseQuery(tc.fields)
require.NoError(t, err)
form.Set("name", "every-type")
form.Set("type", string(targetType))
form.Set("type", string(tc.targetType))
form.Set("url", editOriginalURL)
w := serveTarget(
env, http.MethodPost, "/hook/"+webhook.ID+"/targets", form,
env, http.MethodPost,
"/hook/"+webhook.ID+"/targets", form,
)
require.Equal(t, http.StatusSeeOther, w.Code, w.Body.String())
targets := targetsForWebhook(t, env.db, webhook.ID)
require.Len(t, targets, 1)
assert.Equal(t, tc.targetType, targets[0].Type)
assert.Equal(t, tc.wantAttempts, targets[0].MaxAttempts)
return webhook, targets[0]
if tc.wantConfig == "" {
assert.Empty(t, targets[0].Config)
} else {
assert.JSONEq(t, tc.wantConfig, targets[0].Config)
}
// targetSettingsOf returns a target's settings, URL to
// ArchiveRotation, and nothing else, for comparing with the settings a
// test expects.
func targetSettingsOf(target database.Target) database.Target {
return database.Target{
URL: target.URL,
Headers: target.Headers,
TimeoutSeconds: target.TimeoutSeconds,
ForwardQuery: target.ForwardQuery,
ArchiveExpiry: target.ArchiveExpiry,
ArchiveRotation: target.ArchiveRotation,
})
}
}
internal/handlers/target_edit.go
+52 -27
View File
@@ -25,15 +25,22 @@ const (
tmplKeyMaxTimeout = "MaxTimeout"
)
// configUnreadableMessage is shown when a target's stored
// configuration does not parse. It says plainly that saving replaces
// the stored value rather than preserving it, because the form
// cannot pre-fill what it could not read.
const configUnreadableMessage = "The stored configuration for this " +
"target could not be read. Enter the values below; saving " +
"replaces the stored configuration."
// targetEditView is the display model for the target edit page: the
// target's row fields as stored. The values the form shows, the
// UNMASKED URL and headers among them, come separately, as a
// UNMASKED configuration among them, come separately, as a
// targetFormInput.
//
// It deliberately omits database.Target's settings: the form renders
// them from the targetFormInput, and giving the template the target's
// URL and headers as well would put an unreviewed second path to the
// credential on the page.
// It deliberately omits database.Target's raw Config blob: the form
// renders named fields, and giving the template the blob as well
// would put an unreviewed second path to the credential on the page.
type targetEditView struct {
ID string
Name string
@@ -41,16 +48,13 @@ type targetEditView struct {
Active bool
}
// HandleTargetEdit shows the form to edit a target, filled with the
// target's settings as stored.
// HandleTargetEdit shows the form to edit a target.
//
// This page is the one place the full destination URL and header
// values are shown: an operator cannot correct a value they cannot
// see. It is reachable only through the /hook/{sourceID} route
// group, which supplies RequireAuth and NoCache, and only for a
// target of a webhook the session's user owns; masking
// (delivery.TargetView) is unchanged everywhere else. Do not fill a
// form like this one on any other page.
// values are shown. It is reachable only through the
// /hook/{sourceID} route group, which supplies RequireAuth and
// NoCache, and only for a target of a webhook the session's user
// owns; masking (delivery.TargetView) is unchanged everywhere else.
func (h *Handlers) HandleTargetEdit() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
webhook, target, ok := h.ownedTarget(w, r)
@@ -58,19 +62,34 @@ func (h *Handlers) HandleTargetEdit() http.HandlerFunc {
return
}
cfg, err := delivery.NewTargetConfigForm(target)
msg := ""
if err != nil {
// The error carries the parse failure, never the
// blob, so it is safe to log against the target id.
h.log.Warn(
"stored target config could not be read for editing",
"target_id", target.ID,
"error", err,
)
msg = configUnreadableMessage
}
form := targetFormInput{
Name: target.Name,
URL: target.URL,
Headers: delivery.FormatTargetHeaders(target.Headers),
Timeout: delivery.FormatTargetTimeout(target.TimeoutSeconds),
ForwardQuery: target.ForwardQuery,
URL: cfg.URL,
Headers: cfg.Headers,
Timeout: cfg.Timeout,
ForwardQuery: cfg.ForwardQuery,
MaxAttempts: strconv.Itoa(target.MaxAttempts),
Expiry: formatArchiveExpiry(target.ArchiveExpiry),
Rotation: target.ArchiveRotation,
Expiry: cfg.Expiry,
Rotation: cfg.Rotation,
}
h.renderTargetEdit(
w, r, webhook, target, form, "", http.StatusOK,
w, r, webhook, target, form, msg, http.StatusOK,
)
}
}
@@ -106,11 +125,11 @@ func (h *Handlers) HandleTargetEditSubmit() http.HandlerFunc {
// does, so an edited destination is SSRF-validated exactly as a new
// one is.
//
// The target's type is not editable. Each type uses different
// settings and its delivery history is recorded against the target
// row, so changing the type of an existing target is really the
// creation of a different one. The stored type decides which fields
// the form offers and which settings are set.
// The target's type is not editable. Each type stores a different
// configuration shape and its delivery history is recorded against
// the target row, so changing the type of an existing target is
// really the creation of a different one. The stored type decides
// which fields the form offers and which builder runs.
func (h *Handlers) applyTargetEdit(
w http.ResponseWriter,
r *http.Request,
@@ -123,7 +142,13 @@ func (h *Handlers) applyTargetEdit(
// as stored, for the page shown again when the save is refused.
edited := *target
errMsg := h.setTargetFromForm(r.Context(), &edited, in)
errMsg, err := h.setTargetFromForm(r.Context(), &edited, in)
if err != nil {
h.serverError(w, r, "failed to encode target config", err)
return
}
if errMsg != "" {
h.renderTargetEdit(
w, r, webhook, target, in, errMsg, http.StatusBadRequest,
@@ -135,7 +160,7 @@ func (h *Handlers) applyTargetEdit(
// A new name renames the archive file before it is saved (see
// archive.Writers.Rename). If either step fails, it goes back to
// the name that is still stored.
err := h.renameTargetArchive(
err = h.renameTargetArchive(
target, webhook.Name, target.Name, edited.Name,
)
if err == nil {
internal/handlers/target_edit_test.go
+64 -116
View File
@@ -2,18 +2,19 @@ package handlers_test
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"time"
"github.com/go-chi/chi"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm/clause"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
)
// The destinations the target edit tests configure. Both are literal
@@ -149,6 +150,27 @@ func storedTarget(
return target
}
// storedHTTPConfig reloads a target and parses its stored HTTP
// configuration.
func storedHTTPConfig(
t *testing.T,
env *sourceTestEnv,
targetID string,
) delivery.HTTPTargetConfig {
t.Helper()
var cfg delivery.HTTPTargetConfig
require.NoError(
t,
json.Unmarshal(
[]byte(storedTarget(t, env, targetID).Config), &cfg,
),
)
return cfg
}
// editForm is the fully populated edit submission for an HTTP
// target.
func editForm(targetURL, headers, timeout string) url.Values {
@@ -175,9 +197,12 @@ func submitTargetEdit(
)
}
// TestHandleTargetCreate_Configuration proves the add target form
// stores an HTTP target's headers and timeout, and leaves them empty
// when the form leaves them empty.
// TestHandleTargetCreate_Configuration covers the half of the gap
// that is not about editing at all: HTTPTargetConfig has carried
// Headers and Timeout, and the delivery path has honoured them, but
// the create form wrote {"url":...} and nothing else, so a
// destination needing an Authorization header could not be
// configured through the UI at all.
func TestHandleTargetCreate_Configuration(t *testing.T) {
t.Parallel()
@@ -188,9 +213,9 @@ func TestHandleTargetCreate_Configuration(t *testing.T) {
assertCreateStoresHeadersAndTimeout(t, env)
})
t.Run("without them stores only the url", func(t *testing.T) {
t.Run("without them keeps a url-only config", func(t *testing.T) {
t.Parallel()
assertCreateStoresOnlyTheURL(t, env)
assertCreateKeepsURLOnlyConfig(t, env)
})
}
@@ -203,31 +228,33 @@ func assertCreateStoresHeadersAndTimeout(
t, env, editAuthHeader+"\nX-Tenant: acme\n", "12",
)
stored := storedTarget(t, env, target.ID)
cfg := storedHTTPConfig(t, env, target.ID)
assert.Equal(t, editOriginalURL, stored.URL)
assert.Equal(t, 12, stored.TimeoutSeconds)
assert.Equal(t, editOriginalURL, cfg.URL)
assert.Equal(t, 12, cfg.Timeout)
assert.Equal(
t,
map[string]string{
"Authorization": "Bearer " + editBearerSecret,
"X-Tenant": "acme",
},
stored.Headers,
cfg.Headers,
)
}
func assertCreateStoresOnlyTheURL(
// Without the new fields the stored shape must be the same
// {"url":...} the create form wrote before they existed, so no
// existing target's configuration is rewritten by this change.
func assertCreateKeepsURLOnlyConfig(
t *testing.T, env *sourceTestEnv,
) {
t.Helper()
_, target := seedHTTPTarget(t, env, "", "")
assert.Equal(t, editOriginalURL, target.URL)
assert.Empty(t, target.Headers)
assert.Zero(t, target.TimeoutSeconds)
assert.False(t, target.ForwardQuery)
assert.JSONEq(
t, `{"url":"`+editOriginalURL+`"}`, target.Config,
)
}
// TestHandleTargetEditSubmit_Saves is the round trip the issue asks
@@ -272,8 +299,13 @@ func assertEditChangesDestination(
)
require.Equal(t, http.StatusSeeOther, w.Code, w.Body.String())
assert.Equal(
t,
editReplacedURL,
storedHTTPConfig(t, env, target.ID).URL,
)
reloaded := storedTarget(t, env, target.ID)
assert.Equal(t, editReplacedURL, reloaded.URL)
assert.Equal(t, "edited-name", reloaded.Name)
assert.Equal(t, 5, reloaded.MaxAttempts)
assert.Equal(
@@ -301,16 +333,16 @@ func assertEditRoundTripsHeadersAndTimeout(
)
require.Equal(t, http.StatusSeeOther, w.Code, w.Body.String())
stored := storedTarget(t, env, target.ID)
cfg := storedHTTPConfig(t, env, target.ID)
assert.Equal(t, 21, stored.TimeoutSeconds)
assert.Equal(t, 21, cfg.Timeout)
assert.Equal(
t,
map[string]string{
"Authorization": "Bearer rotated",
"X-Trace": "on",
},
stored.Headers,
cfg.Headers,
)
}
@@ -330,10 +362,10 @@ func assertEditClearingRemovesThem(
)
require.Equal(t, http.StatusSeeOther, w.Code, w.Body.String())
stored := storedTarget(t, env, target.ID)
cfg := storedHTTPConfig(t, env, target.ID)
assert.Empty(t, stored.Headers)
assert.Zero(t, stored.TimeoutSeconds)
assert.Empty(t, cfg.Headers)
assert.Zero(t, cfg.Timeout)
}
// Delivery attempts are offered only by the forms for target types
@@ -410,90 +442,6 @@ func TestHandleTargetEdit_CallsTheDatabaseTypeArchive(t *testing.T) {
assert.Contains(t, page, `class="label">Archive rotation</label>`)
}
// TestHandleTargetEdit_EveryType edits a target of each type: the edit
// form starts on the settings the add target form stored, and saving
// it stores the settings submitted.
func TestHandleTargetEdit_EveryType(t *testing.T) {
t.Parallel()
env := setupSourceTest(t)
// created and edited are the add target form's and the edit
// form's fields, as query strings; shown is what the edit form
// must show of created; want is the settings edited stores.
cases := []struct {
targetType database.TargetType
created, edited string
shown []string
want database.Target
}{
{
database.TargetTypeHTTP,
"url=" + editOriginalURL +
"&headers=X-Tenant:+acme&timeout=7&forward_query=on",
"url=" + editReplacedURL + "&timeout=9",
[]string{
`value="` + editOriginalURL + `"`,
"X-Tenant: acme",
`value="7"`,
`name="forward_query" value="on" checked`,
},
database.Target{
URL: editReplacedURL,
Headers: map[string]string{},
TimeoutSeconds: 9,
},
},
{
database.TargetTypeSlack,
"url=" + editOriginalURL,
"url=" + editReplacedURL,
[]string{`value="` + editOriginalURL + `"`},
database.Target{URL: editReplacedURL},
},
{
database.TargetTypeDatabase,
"expiry=720h&rotation=daily",
"expiry=24h&rotation=hourly",
[]string{
`<option value="720h" selected>`,
`<option value="daily" selected>`,
},
database.Target{
ArchiveExpiry: 24 * time.Hour,
ArchiveRotation: rotationHourly,
},
},
{database.TargetTypeLog, "", "", nil, database.Target{}},
}
for _, tc := range cases {
t.Run(string(tc.targetType), func(t *testing.T) {
t.Parallel()
webhook, target := addTargetThroughForm(
t, env, tc.targetType, tc.created,
)
page := targetEditPage(t, env, webhook.ID, target.ID)
for _, shown := range tc.shown {
assert.Contains(t, page, shown)
}
form, err := url.ParseQuery(tc.edited)
require.NoError(t, err)
form.Set("name", "edited")
w := submitTargetEdit(env, webhook.ID, target.ID, form)
require.Equal(t, http.StatusSeeOther, w.Code, w.Body.String())
got := storedTarget(t, env, target.ID)
assert.Equal(t, "edited", got.Name)
assert.Equal(t, tc.want, targetSettingsOf(got))
})
}
}
// TestHandleTarget_ForwardQuery covers the HTTP target's setting that
// passes each event's query string on to it: the add target form
// stores it checked, the edit form starts with it checked and turns it
@@ -519,7 +467,7 @@ func TestHandleTarget_ForwardQuery(t *testing.T) {
targets := targetsForWebhook(t, env.db, webhook.ID)
require.Len(t, targets, 1)
assert.True(t, storedTarget(t, env, targets[0].ID).ForwardQuery)
assert.True(t, storedHTTPConfig(t, env, targets[0].ID).ForwardQuery)
w = serveTarget(
env, http.MethodGet, targetsPath+"/"+targets[0].ID+"/edit", nil,
@@ -531,7 +479,7 @@ func TestHandleTarget_ForwardQuery(t *testing.T) {
w = submitTargetEdit(env, webhook.ID, targets[0].ID, edit)
require.Equal(t, http.StatusSeeOther, w.Code, w.Body.String())
assert.False(t, storedTarget(t, env, targets[0].ID).ForwardQuery)
assert.False(t, storedHTTPConfig(t, env, targets[0].ID).ForwardQuery)
edit.Set("url", editBlockedURL)
edit.Set("forward_query", "on")
@@ -620,7 +568,7 @@ func assertEditRejectsBlockedDestination(
assert.Contains(t, w.Body.String(), "Invalid target URL")
assert.Equal(
t, editOriginalURL,
storedTarget(t, env, target.ID).URL,
storedHTTPConfig(t, env, target.ID).URL,
"a rejected edit must leave the stored config alone",
)
}
@@ -649,9 +597,9 @@ func assertEditIgnoresQueryString(
assert.Equal(t, http.StatusBadRequest, w.Code)
stored := storedTarget(t, env, target.ID)
assert.Equal(t, editOriginalURL, stored.URL)
assert.Empty(t, stored.Headers)
cfg := storedHTTPConfig(t, env, target.ID)
assert.Equal(t, editOriginalURL, cfg.URL)
assert.Empty(t, cfg.Headers)
}
func assertEditRejectsHeaders(
@@ -669,7 +617,7 @@ func assertEditRejectsHeaders(
assert.Equal(t, http.StatusBadRequest, w.Code)
assert.Contains(t, w.Body.String(), "Invalid headers")
assert.Empty(
t, storedTarget(t, env, target.ID).Headers,
t, storedHTTPConfig(t, env, target.ID).Headers,
"a rejected header must not be stored",
)
}
@@ -689,7 +637,7 @@ func assertEditRejectsTimeout(
assert.Equal(t, http.StatusBadRequest, w.Code)
assert.Contains(t, w.Body.String(), "Invalid timeout")
assert.Equal(
t, 9, storedTarget(t, env, target.ID).TimeoutSeconds,
t, 9, storedHTTPConfig(t, env, target.ID).Timeout,
"a rejected timeout must leave the stored one alone",
)
}
@@ -809,7 +757,7 @@ func assertTargetOfAnotherWebhook404s(
assert.Equal(
t, editOriginalURL,
storedTarget(t, env, target.ID).URL,
storedHTTPConfig(t, env, target.ID).URL,
)
}
@@ -830,7 +778,7 @@ func assertWebhookOfAnotherUser404s(
target := seedConfiguredTarget(
t, env.db, other.ID, database.TargetTypeHTTP,
database.Target{URL: editOriginalURL},
`{"url":"`+editOriginalURL+`"}`,
)
w := serveTarget(
internal/handlers/target_list.go
+4 -1
View File
@@ -243,7 +243,10 @@ func readTargetDeliveries(
func (h *Handlers) archiveFileView(
webhook *database.Webhook, target *database.Target, now time.Time,
) *ArchiveFileView {
current := archive.PathAt(h.dbMgr, webhook, target, now)
current, err := archive.PathAt(h.dbMgr, webhook, target, now)
if err != nil {
return h.archiveUnreadable(&ArchiveFileView{}, target, err)
}
view := &ArchiveFileView{Name: filepath.Base(current)}
internal/handlers/target_toggle_test.go
+3 -1
View File
@@ -60,5 +60,7 @@ func TestHandleTargetToggle_DoesNotUndoAnEdit(t *testing.T) {
assert.False(t, stored.Active)
assert.Equal(t, "edited-name", stored.Name)
assert.Equal(t, 5, stored.MaxAttempts)
assert.Equal(t, editReplacedURL, stored.URL)
assert.Equal(
t, editReplacedURL, storedHTTPConfig(t, env, tgt.ID).URL,
)
}
internal/handlers/webhook.go
+259 -20
View File
@@ -2,11 +2,14 @@ package handlers
import (
"encoding/json"
"fmt"
"io"
"net/http"
"github.com/go-chi/chi"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
"sneak.berlin/go/webhooker/internal/middleware"
)
@@ -67,9 +70,8 @@ func (h *Handlers) HandleWebhook() http.HandlerFunc {
}
}
// processWebhookRequest reads the request into an event and has the
// delivery engine store it, with a delivery for each of the webhook's
// active targets, and answers the sender.
// processWebhookRequest reads the body, serializes headers, loads
// targets, and delivers the event.
//
// Nothing about the request itself is authenticated: the entrypoint
// UUID in the path is the credential, and reaching here means it
@@ -91,24 +93,30 @@ func (h *Handlers) processWebhookRequest(
return
}
event := &database.Event{
WebhookID: entrypoint.WebhookID,
EntrypointID: entrypoint.ID,
Method: r.Method,
RawQuery: r.URL.RawQuery,
Headers: string(headersJSON),
ContentType: r.Header.Get("Content-Type"),
Body: string(body),
}
deliveries, err := h.engine.StoreEvent(event)
targets, err := h.loadActiveTargets(entrypoint.WebhookID)
if err != nil {
h.receiverError(w, "failed to store webhook event", err)
h.receiverError(w, "failed to query targets", err)
return
}
h.finishWebhookResponse(w, event, entrypoint, deliveries)
h.createAndDeliverEvent(
w, r, entrypoint, body, headersJSON, targets,
)
}
// loadActiveTargets returns all active targets for a webhook.
func (h *Handlers) loadActiveTargets(
webhookID string,
) ([]database.Target, error) {
var targets []database.Target
err := h.db.DB().Where(
"webhook_id = ? AND active = ?",
webhookID, true,
).Find(&targets).Error
return targets, err
}
// lookupEntrypoint finds an entrypoint by UUID path.
@@ -169,6 +177,28 @@ func (h *Handlers) readWebhookBody(
return body, true
}
// createAndDeliverEvent stores the received event, fans it out to the
// webhook's targets, and answers the sender.
func (h *Handlers) createAndDeliverEvent(
w http.ResponseWriter,
r *http.Request,
entrypoint database.Entrypoint,
body, headersJSON []byte,
targets []database.Target,
) {
event, tasks, err := h.createAndFanOut(
requestEventSource(r, entrypoint, headersJSON, body),
targets,
)
if err != nil {
h.receiverError(w, "failed to store webhook event", err)
return
}
h.finishWebhookResponse(w, event, entrypoint, tasks)
}
// receiverError logs an error and answers the sender with a plain-text
// 500. The receiver's answers are for programs, so it never sends the
// error page the web UI uses.
@@ -182,19 +212,172 @@ func (h *Handlers) receiverError(
)
}
// finishWebhookResponse logs the stored event, with how many
// deliveries it has, and writes the HTTP response.
// eventSource carries the fields a new event is built from. The
// receiver fills it from the live request; the resubmit handler fills
// it from a stored event. Both then go through createAndFanOut, so an
// event is constructed and fanned out in one place however it entered
// the system.
type eventSource struct {
WebhookID string
EntrypointID string
Method string
RawQuery string
HeadersJSON string
ContentType string
Body []byte
// ResubmittedFromID names the event this one copies. Only the
// resubmit path sets it.
ResubmittedFromID *string
}
// event builds the row this source stores.
func (s eventSource) event() *database.Event {
return &database.Event{
WebhookID: s.WebhookID,
EntrypointID: s.EntrypointID,
Method: s.Method,
RawQuery: s.RawQuery,
Headers: s.HeadersJSON,
Body: string(s.Body),
BodyBytes: int64(len(s.Body)),
ContentType: s.ContentType,
ResubmittedFromID: s.ResubmittedFromID,
}
}
// requestEventSource describes the event a live receiver request
// stores.
func requestEventSource(
r *http.Request,
entrypoint database.Entrypoint,
headersJSON, body []byte,
) eventSource {
return eventSource{
WebhookID: entrypoint.WebhookID,
EntrypointID: entrypoint.ID,
Method: r.Method,
RawQuery: r.URL.RawQuery,
HeadersJSON: string(headersJSON),
ContentType: r.Header.Get("Content-Type"),
Body: body,
}
}
// createAndFanOut writes the event and one pending delivery per target,
// and adds them to the webhook's running totals, in a single
// transaction, then hands the tasks to the delivery engine. Every
// event is created here, received or resubmitted, so a resubmitted
// event is retried, SSRF-guarded and circuit-broken exactly as a
// received one is. Per-delivery replay is the one other path that
// creates a delivery: it adds one to an existing event without
// coming through here.
//
// The tasks are returned as well as queued, so a caller can report how
// many targets the event went to.
func (h *Handlers) createAndFanOut(
src eventSource,
targets []database.Target,
) (*database.Event, []delivery.Task, error) {
webhookDB, err := h.dbMgr.GetDB(src.WebhookID)
if err != nil {
return nil, nil, fmt.Errorf(
"getting webhook database: %w", err,
)
}
tx := webhookDB.Begin()
if tx.Error != nil {
return nil, nil, fmt.Errorf(
"beginning transaction: %w", tx.Error,
)
}
event := src.event()
err = tx.Create(event).Error
if err != nil {
tx.Rollback()
return nil, nil, fmt.Errorf("creating event: %w", err)
}
tasks, err := buildDeliveryTasks(
tx, event, targets, inlineBody(src.Body),
)
if err != nil {
tx.Rollback()
return nil, nil, err
}
err = database.AddEventTotals(tx, database.EventTotals{
Events: 1, LastEventAt: &event.CreatedAt,
})
if err != nil {
tx.Rollback()
return nil, nil, err
}
// A resubmitted copy did not arrive on its entrypoint's URL, so it
// leaves the entrypoint's last event as it is.
if src.ResubmittedFromID == nil {
err = database.AddEntrypointTotals(tx, database.EntrypointTotals{
EntrypointID: event.EntrypointID, LastEventAt: event.CreatedAt,
})
if err != nil {
tx.Rollback()
return nil, nil, err
}
}
err = tx.Commit().Error
if err != nil {
return nil, nil, fmt.Errorf(
"committing transaction: %w", err,
)
}
// Counted here, after the commit: an event exists once it is
// durably stored, which is what the delivery counters are
// compared against on a dashboard. A resubmitted event counts
// too, because it produces deliveries that the delivery side
// counts; the resubmit counter is what separates the two.
h.mtr.EventReceived()
if len(tasks) > 0 {
h.notifier.Notify(tasks)
}
return event, tasks, nil
}
// inlineBody returns a pointer to body as a string if it fits
// within the inline size limit, or nil otherwise.
func inlineBody(body []byte) *string {
if len(body) < delivery.MaxInlineBodySize {
s := string(body)
return &s
}
return nil
}
// finishWebhookResponse logs the event and writes the HTTP response.
func (h *Handlers) finishWebhookResponse(
w http.ResponseWriter,
event *database.Event,
entrypoint database.Entrypoint,
deliveries int,
tasks []delivery.Task,
) {
h.log.Info("webhook event created",
"event_id", event.ID,
"webhook_id", entrypoint.WebhookID,
"entrypoint_id", entrypoint.ID,
"target_count", deliveries,
"target_count", len(tasks),
)
w.WriteHeader(http.StatusOK)
@@ -206,3 +389,59 @@ func (h *Handlers) finishWebhookResponse(
)
}
}
// buildDeliveryTasks creates one pending delivery per target in the
// transaction, adds each to its target's totals, and returns the tasks
// for the delivery engine. The caller owns the transaction and rolls
// it back on error.
func buildDeliveryTasks(
tx *gorm.DB,
event *database.Event,
targets []database.Target,
bodyPtr *string,
) ([]delivery.Task, error) {
tasks := make([]delivery.Task, 0, len(targets))
for i := range targets {
dlv := &database.Delivery{
EventID: event.ID,
TargetID: targets[i].ID,
Status: database.DeliveryStatusPending,
}
err := tx.Create(dlv).Error
if err != nil {
return nil, fmt.Errorf(
"creating delivery for target %s: %w",
targets[i].ID, err,
)
}
err = database.AddTargetTotals(tx, database.TargetTotals{
TargetID: targets[i].ID, Deliveries: 1,
})
if err != nil {
return nil, err
}
tasks = append(tasks, delivery.Task{
DeliveryID: dlv.ID,
EventID: event.ID,
WebhookID: event.WebhookID,
EntrypointID: event.EntrypointID,
TargetID: targets[i].ID,
TargetName: targets[i].Name,
TargetType: targets[i].Type,
TargetConfig: targets[i].Config,
MaxAttempts: targets[i].MaxAttempts,
Method: event.Method,
RawQuery: event.RawQuery,
Headers: event.Headers,
ContentType: event.ContentType,
Body: bodyPtr,
AttemptNum: 1,
})
}
return tasks, nil
}
internal/handlers/webhook_create.go
+12 -6
View File
@@ -115,7 +115,13 @@ func (h *Handlers) HandleSourceCreateSubmit() http.HandlerFunc {
return
}
targets, errMsg := h.newWebhookTargets(r.Context(), in)
targets, errMsg, err := h.newWebhookTargets(r.Context(), in)
if err != nil {
h.serverError(w, r, "failed to encode target config", err)
return
}
if errMsg != "" {
refuse(errMsg)
@@ -141,7 +147,7 @@ func (h *Handlers) HandleSourceCreateSubmit() http.HandlerFunc {
func (h *Handlers) newWebhookTargets(
ctx context.Context,
in sourceFormInput,
) ([]*database.Target, string) {
) ([]*database.Target, string, error) {
var requested []targetFormInput
if in.HTTPURL != "" {
@@ -164,15 +170,15 @@ func (h *Handlers) newWebhookTargets(
targets := make([]*database.Target, 0, len(requested))
for _, form := range requested {
target, errMsg := h.newTarget(ctx, "", form)
if errMsg != "" {
return nil, errMsg
target, errMsg, err := h.newTarget(ctx, "", form)
if err != nil || errMsg != "" {
return nil, errMsg, err
}
targets = append(targets, target)
}
return targets, ""
return targets, "", nil
}
// createWebhookWithEntrypoint creates a webhook, its default
internal/handlers/webhook_detail.go
+1 -1
View File
@@ -109,7 +109,7 @@ func (h *Handlers) renderSourceDetail(
data := map[string]any{
tmplKeyWebhook: &webhook,
// Targets are projected to a display-safe view: a
// target's URL and headers hold a credential, and they
// target's stored config blob holds a credential, and it
// must never reach a template.
"Entrypoints": entrypointViews,
"Targets": h.targetRows(&webhook, targets),
internal/handlers/webhook_stats_test.go
+1 -1
View File
@@ -153,7 +153,7 @@ func seedStatsHistory(
first := seedConfiguredTarget(
t, db, wh.ID, database.TargetTypeHTTP,
database.Target{URL: replayTargetURL},
`{"url":"`+replayTargetURL+`"}`,
)
second := seedTarget(t, db, wh.ID, database.TargetTypeLog)
internal/handlers/webhook_test.go
-84
View File
@@ -206,87 +206,3 @@ func receivedLine(t *testing.T, buf *bytes.Buffer) map[string]any {
return found[0]
}
// TestHandleWebhook_StoresTheEvent proves an accepted request is
// answered with the receiver's 200 and body, and that the event is
// stored with a pending delivery to the webhook's active target.
func TestHandleWebhook_StoresTheEvent(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
db *database.Database
dbMgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &db, &dbMgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
webhook := seedWebhook(t, db)
entrypoint := seedEntrypoint(t, db, webhook.ID)
target := seedConfiguredTarget(
t, db, webhook.ID, database.TargetTypeHTTP,
database.Target{URL: replayTargetURL},
)
w := httptest.NewRecorder()
h.HandleWebhook().ServeHTTP(w, postRequest(
"/h/"+entrypoint.Path, nil,
map[string]string{"uuid": entrypoint.Path},
))
require.Equal(t, http.StatusOK, w.Code)
assert.JSONEq(t, `{"status":"ok"}`, w.Body.String())
webhookDB, err := dbMgr.GetDB(webhook.ID)
require.NoError(t, err)
events := listEvents(t, webhookDB)
require.Len(t, events, 1)
assert.Equal(t, entrypoint.ID, events[0].EntrypointID)
deliveries := listDeliveries(t, webhookDB, events[0].ID)
require.Len(t, deliveries, 1)
assert.Equal(t, target.ID, deliveries[0].TargetID)
assert.Equal(t, database.DeliveryStatusPending, deliveries[0].Status)
}
// TestHandleWebhook_FailedStoreAnswersPlainError proves that an event
// the engine cannot store is answered with the receiver's plain-text
// 500, and that nothing of it is left stored.
func TestHandleWebhook_FailedStoreAnswersPlainError(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
db *database.Database
dbMgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &db, &dbMgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
webhook := seedWebhook(t, db)
entrypoint := seedEntrypoint(t, db, webhook.ID)
// Without its event totals table the webhook's database refuses
// the totals write, which comes after the event row.
webhookDB, err := dbMgr.GetDB(webhook.ID)
require.NoError(t, err)
require.NoError(t,
webhookDB.Migrator().DropTable(&database.EventTotals{}))
w := httptest.NewRecorder()
h.HandleWebhook().ServeHTTP(w, postRequest(
"/h/"+entrypoint.Path, nil,
map[string]string{"uuid": entrypoint.Path},
))
assert.Equal(t, http.StatusInternalServerError, w.Code)
assert.Equal(t, "Internal server error\n", w.Body.String())
assert.Empty(t, listEvents(t, webhookDB))
}
internal/logfield/logfield_test.go
+6 -7
View File
@@ -21,9 +21,8 @@ const budget = 64
// batchRunes is how many consecutive code points the charge test logs
// in one value from U+1000 up. Logging each of those on its own line
// was too slow for the suite when it ran under the race detector;
// 4,096 at a time is 271 batches, each logged on two lines, so 542
// lines per handler.
// is too slow for the suite under the race detector; 4,096 at a time
// is 271 batches, each logged on two lines, so 542 lines per handler.
const batchRunes = 4096
// newHandlers are the two handlers internal/logger can install. Time
@@ -153,10 +152,10 @@ func TestEncodedBytes_ChargesAtLeastWhatTheHandlersEmit(t *testing.T) {
batches = append(batches, batch.String())
}
// What EncodedBytes charges for each batch. When the suite ran
// under -race -cover this took longer than logging the batches, so
// it is worked out once, by whichever handler finishes logging
// first, while the other is still logging.
// What EncodedBytes charges for each batch. Under -race -cover this
// takes longer than logging the batches, so it is worked out once,
// by whichever handler finishes logging first, while the other is
// still logging.
charged := sync.OnceValue(func() []int {
costs := make([]int, len(batches))
internal/resetpw/resetpw_test.go
+5 -6
View File
@@ -131,6 +131,10 @@ func run(
return code, stdout.String(), stderr.String()
}
type noopNotifier struct{}
func (n *noopNotifier) Notify([]delivery.Task) {}
type noopArchives struct {
renameMu sync.Mutex
}
@@ -186,12 +190,7 @@ func newServerApp(
database.NewWebhookDBManager,
healthcheck.New,
session.New,
archive.NewWriters,
// The engine is built on a lifecycle that is never started,
// so no worker runs.
func(p delivery.EngineParams) *delivery.Engine {
return delivery.New(fxtest.NewLifecycle(t), p)
},
func() delivery.Notifier { return &noopNotifier{} },
func() archive.Archives { return &noopArchives{} },
func() delivery.CircuitBreakers {
return &noopCircuitBreakers{}
internal/server/alpine_browser_test.go
+19 -39
View File
@@ -1183,26 +1183,16 @@ func checkNewWebhookTargets(
) {
t.Helper()
// Each value the pruning choice submits and the expiry it stores,
// after an empty one that stands for archive left off.
expiries := []struct {
value string
stored time.Duration
}{
{"", 0},
{"never", 0},
{"1h", time.Hour},
{"12h", 12 * time.Hour},
{"24h", 24 * time.Hour},
{"720h", 720 * time.Hour},
{"2160h", 2160 * time.Hour},
{"8760h", 8760 * time.Hour},
// Each value the pruning choice submits, after an empty one that
// stands for archive left off.
expiries := []string{
"", "never", "1h", "12h", "24h", "720h", "2160h", "8760h",
}
for _, httpURL := range []string{"", publicTargetURL} {
for _, expiry := range expiries {
name := "url=" + httpURL + " archive=" + expiry.value
want := map[database.TargetType]database.Target{}
name := "url=" + httpURL + " archive=" + expiry
want := map[database.TargetType]string{}
require.NoError(t, chromedp.Run(
ctx,
@@ -1215,12 +1205,10 @@ func checkNewWebhookTargets(
"#http_url", httpURL, chromedp.ByQuery,
)))
want[database.TargetTypeHTTP] = database.Target{
URL: httpURL, Headers: map[string]string{},
}
want[database.TargetTypeHTTP] = `{"url":"` + httpURL + `"}`
}
if expiry.value != "" {
if expiry != "" {
// The choice showing moves Create down, so it is
// waited for before Create is clicked.
click(ctx, t, archiveBox)
@@ -1228,30 +1216,29 @@ func checkNewWebhookTargets(
"%s: checking archive does not show the pruning choice",
name)
require.NoError(t, chromedp.Run(ctx, chromedp.SetValue(
pruningChoice, expiry.value, chromedp.BySearch,
pruningChoice, expiry, chromedp.BySearch,
)))
// The rotation choice is left on none.
want[database.TargetTypeDatabase] = database.Target{
ArchiveExpiry: expiry.stored, ArchiveRotation: "none",
}
want[database.TargetTypeDatabase] = `{"expiry":"` + expiry +
`","rotation":"none"}`
}
clickAndLoad(ctx, t, createButton)
require.Truef(t, shown(ctx, `//h1[text()="`+name+`"]`),
"%s: the new webhook's page does not open", name)
assert.Equalf(t, want, targetSettings(t, env, name),
assert.Equalf(t, want, targetConfigs(t, env, name),
"%s: the webhook does not have the targets asked for", name)
}
}
}
// targetSettings reads the targets of the webhook named name, and
// returns each one's stored settings by its type.
func targetSettings(
// targetConfigs reads the targets of the webhook named name, and
// returns each one's stored configuration by its type.
func targetConfigs(
t *testing.T, env *testEnv, name string,
) map[database.TargetType]database.Target {
) map[database.TargetType]string {
t.Helper()
var (
@@ -1264,19 +1251,12 @@ func targetSettings(
require.NoError(t, env.db.DB().
Where("webhook_id = ?", webhook.ID).Find(&targets).Error)
settings := map[database.TargetType]database.Target{}
configs := map[database.TargetType]string{}
for _, target := range targets {
settings[target.Type] = database.Target{
URL: target.URL,
Headers: target.Headers,
TimeoutSeconds: target.TimeoutSeconds,
ForwardQuery: target.ForwardQuery,
ArchiveExpiry: target.ArchiveExpiry,
ArchiveRotation: target.ArchiveRotation,
}
configs[target.Type] = target.Config
}
return settings
return configs
}
// checkRefusedNewWebhook submits the new webhook page with archive
internal/server/early_shutdown_test.go
+8 -7
View File
@@ -10,12 +10,14 @@ import (
"sneak.berlin/go/webhooker/internal/server"
)
// earlyStopIterations is how many start/stop cycles the test below
// earlyStopIterations is how many start/stop cycles the race test
// runs. The window it aims at is the gap between the OnStart hook
// returning and the serving goroutine reaching its first field
// access, which is microseconds wide, so a single cycle can miss it
// purely on scheduling. Repetition makes hitting it reliable; the
// collaborators are built once, so the cycles themselves are cheap.
// access, which is microseconds wide. The race detector reports an
// unsynchronised pair whenever it observes one, but it has to observe
// one, so a single cycle can miss purely on scheduling. Repetition
// makes the observation reliable; the collaborators are built once,
// so the cycles themselves are cheap.
const earlyStopIterations = 25
// TestEarlyShutdown_NoPanicAndNoRace stops the application
@@ -36,9 +38,8 @@ const earlyStopIterations = 25
//
// httpServer is now built in New, on the constructing goroutine, so
// it is written before any hook exists and can never be nil.
// sentryEnabled is atomic. This test catches the nil dereference
// coming back, as a panic. The suite runs without Go's race
// detector, so the data race coming back would go unreported here.
// sentryEnabled is atomic. This test is what catches either one
// coming back — under -race, which is how the suite runs.
func TestEarlyShutdown_NoPanicAndNoRace(t *testing.T) {
t.Parallel()
internal/server/routes.go
+1 -1
View File
@@ -311,7 +311,7 @@ func (s *Server) setupSourceRoutes() {
r.Post("/targets", s.h.HandleTargetCreate())
// The edit form is the one page that renders a target's
// destination URL and header values in full; see
// Handlers.HandleTargetEdit. It belongs to this group for
// delivery.TargetConfigForm. It belongs to this group for
// its RequireAuth and NoCache, which are what keep that
// exception from reaching an unauthenticated request or a
// shared cache.
internal/server/routes_test.go
+6 -7
View File
@@ -48,6 +48,10 @@ const (
metricsAuthValue = "s3cret"
)
type noopNotifier struct{}
func (n *noopNotifier) Notify([]delivery.Task) {}
// noopArchives satisfies handlers.New's archive.Archives
// dependency. No test here checks what gets evicted or renamed, so
// it records nothing. Its rename lock is a plain mutex.
@@ -141,12 +145,7 @@ func newTestEnvWithConfig(
database.NewWebhookDBManager,
healthcheck.New,
session.New,
archive.NewWriters,
// The engine is built on a lifecycle that is never started,
// so no worker runs and nothing is sent anywhere.
func(p delivery.EngineParams) *delivery.Engine {
return delivery.New(fxtest.NewLifecycle(t), p)
},
func() delivery.Notifier { return &noopNotifier{} },
func() archive.Archives { return &noopArchives{} },
func() delivery.CircuitBreakers {
return &noopCircuitBreakers{}
@@ -415,7 +414,7 @@ func (e *testEnv) seedTarget(
Name: "routed-target",
Type: database.TargetTypeHTTP,
Active: true,
URL: "http://93.184.216.34/hook",
Config: `{"url":"http://93.184.216.34/hook"}`,
MaxAttempts: 1,
}