Author SHA1 Message Date
sneak 370fad06fc Rewrite TODO.md sentence by sentence
check / check (push) Waiting to run
Rewritten from scratch against the code on next; the sections and
their contents are kept.

Model: fable-5-1
2026-10-25 05:18:45 +00:00
sneak 54b643044f Merge branch 'main' into next
check / check (push) Waiting to run
2026-10-25 02:51:56 +01:00
clawbot 55ccb28270 Run the tests with cgo off and without the race detector (closes #538)
check / check (push) Canceled after 0s
sneak ruled the race detector out
(#538 (comment)), and on
Linux it was the only reason the tests needed cgo. The test phase and the
browser test binary now build with CGO_ENABLED=0 and without -race, so no
test binary compiles mattn's C SQLite. The test phase keeps the builder
stage's Debian Go image. Its comments, the README and the code comments that
said the suite runs under -race now say what is true, and the README notes
the exception to the vendored REPO_POLICIES.md.

Model: opus-5-5
2026-10-21 17:28:50 +02:00
sneak 766f33a956 Milestone: next into main (#380)
check / check (push) Canceled after 0s
Reviewed-on: #380
2026-10-15 11:36:37 +02:00
9 changed files with 99 additions and 81 deletions
+13 -8
View File
@@ -111,11 +111,16 @@ RUN --network=none golangci-lint run --config .golangci.yml --build-tags browser
# Nothing is wanted from js-lint; the copy is what makes this phase run it.
COPY --from=js-lint /src/yarn.lock /dev/null
# Test phase. -race needs cgo and so a C compiler, which the Debian Go image
# ships and the alpine one does not. `make test` (script/test) builds this
# stage alone; the build stage below depends on it.
# Test phase. The tests run with cgo off and so without Go's race detector,
# which needs cgo on Linux; sneak ruled the race detector out in
# https://git.eeqj.de/sneak/webhooker/issues/538#issuecomment-153273.
# `make test` (script/test) builds this stage alone; the build stage below
# depends on it.
#
# golang:1.26.1-bookworm (Debian-based), 2026-03-17
# golang:1.26.1-bookworm (Debian-based), 2026-03-17: the builder stage's
# image, which that stage needs for its git and make. Its test step runs
# under the image's bash, and the tests in internal/versionscript use its
# git, make and sh, and skip when they are missing.
FROM golang:1.26.1-bookworm@sha256:4465644228bc2857a954b092167e12aa59c006a3492282a6c820bf4755fd64a4 AS test
WORKDIR /src
@@ -131,8 +136,8 @@ RUN script/assets
# -timeout applies to each package on its own, so 90s has only to clear the
# slowest one. -p 4 -parallel 8 keep the run under 2 GB of memory: at most
# four test binaries build or run at once, each with at most eight parallel
# tests. Under -race every test binary and every link costs a few hundred MB,
# so the defaults (one per core) add up to several GB on a many-core host.
# tests. The defaults are one per core, so without these the memory the run
# needs grows with the host's core count.
#
# The first run has no -v: go test then prints one result line per package,
# with its coverage, and for a package that fails, everything its tests
@@ -154,12 +159,12 @@ RUN script/assets
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
RUN --mount=type=tmpfs,target=/tmp/tests,size=512m \
export TMPDIR=/tmp/tests GOTMPDIR=/tmp; \
go test -race -cover -p 4 -parallel 8 -timeout 90s ./... 2>&1 | tee /tmp/go-test.log && exit 0; \
CGO_ENABLED=0 go test -cover -p 4 -parallel 8 -timeout 90s ./... 2>&1 | tee /tmp/go-test.log && exit 0; \
tests="$(awk '/^--- FAIL: / { print $3 }' /tmp/go-test.log | paste -s -d '|' -)"; \
packages="$(awk '/^FAIL\t/ { print $2 }' /tmp/go-test.log)"; \
if [ -n "$tests" ]; then \
echo "--- Rerunning the failed tests with -v for details ---"; \
go test -race -v -p 4 -parallel 8 -timeout 90s -run "^($tests)\$" $packages; \
CGO_ENABLED=0 go test -v -p 4 -parallel 8 -timeout 90s -run "^($tests)\$" $packages; \
fi; \
exit 1
+3 -3
View File
@@ -15,9 +15,9 @@ RUN go mod download
COPY . .
# The test binary embeds the templates and static files, so the browser
# stage needs nothing else. -p 4 keeps the compile's memory down, as in
# the test phase of Dockerfile.
RUN make assets && go test -c -p 4 -tags browser -o /browser.test ./internal/server
# stage needs nothing else. It is built with cgo off and -p 4 keeps the
# compile's memory down, both as in the test phase of Dockerfile.
RUN make assets && CGO_ENABLED=0 go test -c -p 4 -tags browser -o /browser.test ./internal/server
# chromedp/headless-shell:151.0.7922.109 (Debian trixie), 2026-08-11. The
# browser is on PATH as headless-shell, where the test's browser library
+8 -3
View File
@@ -60,7 +60,7 @@ make assets # Extract Alpine.js from 3p/ (build and dev run it)
make fmt # Format Go (gofmt + goimports) and Markdown (prettier, in Docker)
make fmt-check # Fail if gofmt or prettier would change anything (writes nothing)
make lint # Run golangci-lint and ESLint in Docker
make test # Run tests with race detection, in Docker
make test # Run tests with cgo off, in Docker
make test-browser # Run the browser test in Docker (Dockerfile.browser)
make check # test + lint + fmt-check + css-check (CI gate)
make build # Build binary to bin/webhooker (version-stamped)
@@ -75,6 +75,11 @@ make css-check # Fail if static/css/tailwind.css is stale (writes nothing)
make clean # Remove bin/
```
The tests run with cgo off and so without Go's race detector, which needs cgo on
Linux: an exception to the `-race` in the test phase of the vendored
`REPO_POLICIES.md`, ruled by @sneak in
https://git.eeqj.de/sneak/webhooker/issues/538#issuecomment-153273.
### Configuration
All configuration is via environment variables. For local development, you can
@@ -3480,8 +3485,8 @@ fixed independently of the compiler's:
`golangci-lint config verify` and `golangci-lint run`, both with
`--network=none`, and depends on `js-lint` (it copies a file from it).
`make lint` builds this stage alone.
5. **Test phase** (`test`, `golang:1.26.1-bookworm`, whose C compiler `-race`
needs) — extracts Alpine.js, then runs `go test -race -cover` at most four
5. **Test phase** (`test`, `golang:1.26.1-bookworm`, the builder stage's image)
— extracts Alpine.js, then runs `go test -cover` with cgo off, at most four
packages and eight tests at a time, with a 90-second timeout per package. On
a failure it runs only the failed tests again with `-v`, since verbose output
from the whole suite would pass the 2 MiB at which the Docker build cuts off
+51 -46
View File
@@ -1,72 +1,77 @@
# Workflow
One issue per unit of work, one branch and one PR per issue:
One issue per unit of work, and one branch and one PR per issue:
- ensure a tracked issue exists with a definition of done
- branch from `next` (never from `main`)
- do the work; open a PR based on `next` (never on `main`)
- pass an independent review, then the manager squash-merges into `next`
- push; nothing stays local-only
- make sure a tracked issue exists with a definition of done
- branch from `next`, never from `main`
- do the work and push the branch
- open a PR with base `next`, never `main`
- pass an independent review
- the repository manager squash-merges it into `next`
`next` is the branch for the next milestone and must stay green and mergeable to
`main` without notice. One `next` -> `main` PR accumulates the milestone;
releases are cut from `main` separately.
The repository manager is the role that merges each issue's PR into `next`.
Nothing stays local-only: every commit is pushed.
The tracker (https://git.eeqj.de/sneak/webhooker/issues) is the record of what
is done and what is under way. This file keeps no list of either.
`next` is the branch for the next milestone. It must stay green and mergeable
into `main` at all times. A milestone's work reaches `main` through one PR from
`next`. The owner, @sneak, merges that PR. Cutting a release from `main` is a
separate step after that.
The tracker (https://git.eeqj.de/sneak/webhooker/issues) records what is done
and what is under way. This file keeps no list of either.
# Status
What is still open for 1.0.0 is in its milestone
(https://git.eeqj.de/sneak/webhooker/milestone/9). `v1.0.0` is not tagged, and
only @sneak tags it.
(https://git.eeqj.de/sneak/webhooker/milestone/9). `v1.0.0` is not tagged. Only
@sneak tags it.
Delivery is at-least-once by design, not by accident: a send whose result row
does not land is attempted again, so a receiver can see a duplicate. That is
deliberate — the alternative is a silent lost delivery — and the README says so
under Rationale. It is not a defect to re-file.
Delivery is at least once by design. When webhooker cannot record the outcome of
a delivery, it sends that delivery again. The target can then receive the same
event twice. The alternative is a lost delivery that nothing reports. The README
says so. Such a duplicate is not a defect to re-file.
# Next Step
@sneak runs webhooker in limited production, then tags `v1.0.0`. Merging `next`
into `main` is a separate act from tagging and waits on neither of those: `next`
is kept mergeable at all times, which is the point of the branch.
into `main` is a separate act. It waits on neither the production run nor the
tag.
# Future Steps
- Per-webhook rate limits, set in the web UI and enforced in the receiver
handler, on top of the receiver's existing limits, which `RECEIVER_RATE_LIMIT`
sets for the whole server (see Rate Limiting in the README)
- API key authentication for programmatic access (the `APIKey` model exists, but
nothing creates a key and no middleware checks one)
- REST API v1
- CRUD for webhooks, entrypoints, targets
- Per-webhook rate limits, set in the web UI and enforced at the entrypoint.
Today `RECEIVER_RATE_LIMIT` is the only rate-limit setting. It is how many
requests per minute one client address may send to one entrypoint. The same
address is also capped at ten times that across all entrypoints together.
- API key authentication for programmatic access. The `APIKey` model and its
`api_keys` table exist. Nothing creates a key and nothing checks one.
- REST API v1. `/api/v1` is mounted with no routes today.
- CRUD for webhooks, entrypoints and targets
- event viewing and filtering endpoints
- event redelivery endpoint
- OpenAPI specification
- Analytics in the web UI: response times summarised per webhook or per target,
and figures for all webhooks together. The webhook page already shows event,
delivery and failure counts and failure percentages, the target list counts
each target's deliveries, the event log and each event's page show every
attempt's duration, and `/metrics` carries server-wide counts and delivery
durations
- A remember-me option at login. Every session already survives a browser
restart: it lasts up to 7 days, and ends sooner after `SESSION_IDLE_TIMEOUT`
(default 24 hours) without activity
- Password reset flow for a forgotten password over the web. The authenticated
password _change_ flow already landed, and a lost password is recoverable from
the console with `webhooker resetpw`
(https://git.eeqj.de/sneak/webhooker/issues/208)
and figures for all webhooks together. Today the webhook page shows event,
delivery and failure counts and failure percentages. The target list counts
each target's deliveries. The event log shows every attempt's duration.
`/metrics` carries server-wide counts and delivery durations.
- A remember-me option at login. Today every session survives a browser restart.
It lasts up to 7 days. It ends sooner after `SESSION_IDLE_TIMEOUT` (default 24
hours) without use.
- Password reset over the web for a forgotten password. Today a signed-in user
can change their password. A lost password is reset from the command line with
`webhooker resetpw` (https://git.eeqj.de/sneak/webhooker/issues/208).
- Later, nice to have
- email delivery target type
- SNS and S3 delivery targets
- data transformations (e.g. a configurable Slack message format; the Slack
target sends one fixed format today)
- JSONL file delivery with periodic S3 upload (the `database` target already
archives events to SQLite files, rotated hourly, daily or monthly if set,
and the webhook page offers them as one gzipped JSON download)
- data transformations, such as a configurable Slack message format. The
Slack target sends one fixed format today.
- JSONL file delivery with periodic S3 upload. Today an archive target
writes events to SQLite files. An archive target can rotate those files
hourly, daily or monthly. The webhook page offers them as one gzipped JSON
download.
- webhook event search, and filtering beyond the event log's failed and
pending views
- multi-user with role-based access control (each webhook already belongs to
one account, but there is only the account made at first boot and no way
to add another)
- multi-user with role-based access control. Today each webhook belongs to
one account. Only the account made at first boot exists. There is no way
to add another.
+1 -1
View File
@@ -303,7 +303,7 @@ func (r *racingWrites) wait() (bool, error) {
// contending for the writer's mutex when the eviction takes it.
// The write that wins the mutex after the eviction must abandon
// its work rather than reopen the archive, leaving the writer
// permanently handle-free. Run under -race.
// permanently handle-free.
func TestEvictWebhook_RacingWriteDoesNotReopenHandle(
t *testing.T,
) {
+5 -3
View File
@@ -959,9 +959,11 @@ func TestArchiveSweep_SkipsDeletedWebhookTargets(t *testing.T) {
)
}
// TestArchiveSweep_ConcurrentWrites proves the sweep serialises
// against writes through the target's writer mutex. Run under
// -race, an unsynchronised sweep would be caught here.
// TestArchiveSweep_ConcurrentWrites runs the sweep while writes to
// the same target are under way. The sweep serialises against them
// through the target's writer mutex; the suite runs without Go's race
// detector, so a sweep that did not would fail this test only by
// panicking, hanging or leaving no archive file.
func TestArchiveSweep_ConcurrentWrites(t *testing.T) {
t.Parallel()
+4 -3
View File
@@ -67,9 +67,10 @@ func DefaultPasswordConfig() *PasswordConfig {
// testArgon2Memory is the Argon2id memory cost, in KiB, that a test
// binary hashes with: 1 MB instead of the shipped 64 MB. Every test
// that starts a database hashes the bootstrap admin password, dozens
// of them run in parallel, and under the race detector each 64 MB hash
// holds about 150 MB. VerifyPassword reads the cost from the hash it
// checks, so verification follows.
// of them run in parallel, and each 64 MB hash holds at least 64 MB
// while it runs (about 150 MB when the suite ran under the race
// detector). VerifyPassword reads the cost from the hash it checks, so
// verification follows.
const testArgon2Memory = 1024
// hashAtShippedCostInTest makes a test binary hash at the shipped
+7 -6
View File
@@ -21,8 +21,9 @@ const budget = 64
// batchRunes is how many consecutive code points the charge test logs
// in one value from U+1000 up. Logging each of those on its own line
// is too slow for the suite under the race detector; 4,096 at a time
// is 271 batches, each logged on two lines, so 542 lines per handler.
// was too slow for the suite when it ran under the race detector;
// 4,096 at a time is 271 batches, each logged on two lines, so 542
// lines per handler.
const batchRunes = 4096
// newHandlers are the two handlers internal/logger can install. Time
@@ -152,10 +153,10 @@ func TestEncodedBytes_ChargesAtLeastWhatTheHandlersEmit(t *testing.T) {
batches = append(batches, batch.String())
}
// What EncodedBytes charges for each batch. Under -race -cover this
// takes longer than logging the batches, so it is worked out once,
// by whichever handler finishes logging first, while the other is
// still logging.
// What EncodedBytes charges for each batch. When the suite ran
// under -race -cover this took longer than logging the batches, so
// it is worked out once, by whichever handler finishes logging
// first, while the other is still logging.
charged := sync.OnceValue(func() []int {
costs := make([]int, len(batches))
+7 -8
View File
@@ -10,14 +10,12 @@ import (
"sneak.berlin/go/webhooker/internal/server"
)
// earlyStopIterations is how many start/stop cycles the race test
// earlyStopIterations is how many start/stop cycles the test below
// runs. The window it aims at is the gap between the OnStart hook
// returning and the serving goroutine reaching its first field
// access, which is microseconds wide. The race detector reports an
// unsynchronised pair whenever it observes one, but it has to observe
// one, so a single cycle can miss purely on scheduling. Repetition
// makes the observation reliable; the collaborators are built once,
// so the cycles themselves are cheap.
// access, which is microseconds wide, so a single cycle can miss it
// purely on scheduling. Repetition makes hitting it reliable; the
// collaborators are built once, so the cycles themselves are cheap.
const earlyStopIterations = 25
// TestEarlyShutdown_NoPanicAndNoRace stops the application
@@ -38,8 +36,9 @@ const earlyStopIterations = 25
//
// httpServer is now built in New, on the constructing goroutine, so
// it is written before any hook exists and can never be nil.
// sentryEnabled is atomic. This test is what catches either one
// coming back — under -race, which is how the suite runs.
// sentryEnabled is atomic. This test catches the nil dereference
// coming back, as a panic. The suite runs without Go's race
// detector, so the data race coming back would go unreported here.
func TestEarlyShutdown_NoPanicAndNoRace(t *testing.T) {
t.Parallel()