Compare commits

1 Commits
Author SHA1 Message Date
clawbot f01e5c45ad Give each event its own page and show bodies the same everywhere (closes #369)
check / check (push) Successful in 3m20s
Each row of the recent events on the webhook page links to the
event's own page, /hook/{id}/events/{eventID}, and expands to show its
body; only the newest starts expanded. The event's page shows its
details, its whole body and every delivery with its attempts.

One renderer, newBodyView with templates/event_body.html, shows a body
in all three places: whole up to 32 KiB, cut there in the lists with a
link to the event's page, JSON pretty-printed, a body of more than 200
lines or 32 KiB in a scrolling box, and a body that is not text left
out beside its download link. A resubmitted copy links to its
original's page.

Model: opus-5-5
2026-10-02 18:33:27 +00:00
14 changed files with 22 additions and 382 deletions
-1
View File
@@ -25,7 +25,6 @@ COPY . .
# would need a docker daemon inside the build. Keep these steps in step with
# Dockerfile.lint, including --network=none (see its header for why).
RUN make fmt-check
RUN script/assets
RUN --network=none golangci-lint config verify --config .golangci.yml
RUN --network=none golangci-lint run --config .golangci.yml --build-tags browser ./...
-4
View File
@@ -31,10 +31,6 @@ FROM deps AS lint
COPY . .
# static/static.go embeds the Alpine.js file this extracts from 3p/; without
# it the static package does not compile and cannot be linted.
RUN script/assets
# `run` silently ignores config keys it does not recognize, so a typo would
# disable a setting without a word. `config verify` is what catches that.
RUN --network=none golangci-lint config verify --config .golangci.yml
+8 -23
View File
@@ -1050,8 +1050,7 @@ Archive databases are the one exception the service is built for: the
archive writer closes and reopens its handle around writes (debounced
to at most one reopen per second), so an operator can move an
`archive-….db` away for offline retention while the service runs,
and it is recreated on the next write. The webhook page names each
`database` target's archive file. See
and it is recreated on the next write. See
[Database Architecture](#database-architecture). That is a
move-the-file-away workflow, not a substitute for the backup procedures
above.
@@ -1351,9 +1350,7 @@ apply. The directory is `3p/` rather than `vendor/` because Go treats a root
where `go:embed` picks it up. `script/test`, `make build` and `make dev` run
it first, and the Dockerfile builds through `make test` and `make build`, so
nothing downloads Alpine.js. The extracted file is not committed, and
`.dockerignore` keeps any host copy out of the build context. `static/static.go`
names every file it embeds, so a build that skips the extraction, such as a
bare `go build`, fails with an error naming `js/alpine.min.js`.
`.dockerignore` keeps any host copy out of the build context.
To move to a new version: download
`https://registry.npmjs.org/@alpinejs/csp/-/csp-<version>.tgz`, check it against
@@ -1881,19 +1878,16 @@ tags, so `AutoMigrate` creates them on a fresh database:
| `deliveries` | `event_id`, `deleted_at` | The event log, which loads each event's deliveries, and retention, which counts and deletes the deliveries of expired events |
| `delivery_results` | `delivery_id`, `deleted_at` | The event log, which loads the attempts of a page's deliveries, and retention, which deletes the attempts of expired events |
| `events` | `deleted_at`, `created_at` | The webhook page's statistics, which count recent events |
| `events` | `resubmitted_from_id`, `deleted_at` | The event log, which counts the events resubmitted from each event on a page |
| `events` | `created_at` | Retention, which selects expired events by age |
GORM's soft delete adds `deleted_at IS NULL` to these queries; retention
leaves it out. SQLite keeps no statistics on these tables, and without them it
rates the `deleted_at` index, which every live row matches, above an index on
a column matched against several values or compared with a range. So every
index but the last also covers `deleted_at`. It comes second in the `event_id`
and `delivery_id` indexes, so that retention can use them without it. The event
log's count, the one query on the `resubmitted_from_id` index, always carries
`deleted_at IS NULL` and uses both columns. In the statistics' `events` index
`deleted_at` comes first, because they compare `created_at` with a range (`>=`)
and SQLite narrows by a range only on the last column it uses.
index but the last also covers `deleted_at`. It comes second, so that
retention can use the index without it, except in `events`, where the
statistics compare `created_at` with a range (`>=`) and SQLite narrows by a
range only on the last column it uses.
#### Common Fields
@@ -2039,14 +2033,6 @@ Because each `database` target has its own archive file, a target's
webhook with different expiries keep two archives, each pruned on its
own schedule.
The webhook page shows, for each `database` target, its archive file's
name, its size on disk and when it was last written. The size counts
the `.db` and its `-wal` together, and the last write is the later of
their two modification times, since a write lands in the `-wal` first.
Both are read from the files' metadata; the archive is never opened.
Before the first write, and after the file has been moved away, the page
shows `not created yet` beside the name.
Each `database` target on the webhook page has a **Download** button,
which returns its archive as one gzipped JSON file,
`archive-{webhook_name}-{target_name}-{YYYYMMDDTHHMMSSZ}.json.gz`, the
@@ -3396,9 +3382,8 @@ version is fixed independently of the compiler's:
1. **Lint stage** (`golangci/golangci-lint:v2.12.2`, Debian-based) —
installs `make`, downloads dependencies, copies the source, and runs
`make fmt-check`, then `script/assets` to extract Alpine.js from
`3p/`, then `golangci-lint config verify` and `golangci-lint run`,
both with `--network=none`.
`make fmt-check`, then `golangci-lint config verify` and
`golangci-lint run`, both with `--network=none`.
2. **Builder stage** (`golang:1.26.1-bookworm`) — depends on the lint
stage passing (it copies a file from it), runs `make test` and
`make build` (both extract Alpine.js from `3p/` first), and finally
@@ -199,40 +199,6 @@ func TestStatisticsQueriesUseTheirIndexes(t *testing.T) {
"(deleted_at=? AND created_at>?)")
}
// TestResubmitCountUsesItsIndex does the same for the event log's count
// of the events resubmitted from each of a page's events (resubmitCounts
// in the handlers). It passes a full page of 25 ids: with an index on
// resubmitted_from_id alone, SQLite uses it for three ids and turns to
// the deleted_at index from five.
func TestResubmitCountUsesItsIndex(t *testing.T) {
t.Parallel()
mgr, lc := setupTestWebhookDBManager(t)
ctx := context.Background()
require.NoError(t, lc.Start(ctx))
defer func() { require.NoError(t, lc.Stop(ctx)) }()
db, err := mgr.GetDB(uuid.New().String())
require.NoError(t, err)
dry := db.Session(&gorm.Session{DryRun: true})
page := make([]string, 25)
for i := range page {
page[i] = uuid.New().String()
}
var counts []struct{ Total int }
assertPlanUses(t, db, dry.Model(&database.Event{}).
Select("resubmitted_from_id, count(*) AS total").
Where("resubmitted_from_id IN ?", page).
Group("resubmitted_from_id").Find(&counts),
"idx_events_resubmitted_from_id "+
"(resubmitted_from_id=? AND deleted_at=?)")
}
// assertPlanUses asserts that SQLite's plan for a statement GORM built
// in a dry run, run with the same SQL and arguments GORM would send,
// names each of the given indexes.
+3 -5
View File
@@ -19,10 +19,8 @@ type Event struct {
// narrows by a < only on the last column it uses. Its final delete
// has no deleted_at condition and uses the index on created_at
// alone. The other tables keep the unindexed BaseModel created_at.
// DeletedAt is also the second column of the resubmitted_from_id
// index, for the reason DeliveryResult gives.
CreatedAt time.Time `gorm:"index;index:idx_events_deleted_at_created_at,priority:2" json:"createdAt"`
DeletedAt gorm.DeletedAt `gorm:"index:idx_events_deleted_at_created_at,priority:1;index:idx_events_resubmitted_from_id,priority:2" json:"deletedAt,omitzero"`
CreatedAt time.Time `gorm:"index;index:idx_events_deleted_at_created_at,priority:2" json:"createdAt"`
DeletedAt gorm.DeletedAt `gorm:"index:idx_events_deleted_at_created_at,priority:1" json:"deletedAt,omitzero"`
WebhookID string `gorm:"type:uuid;not null" json:"webhookId"`
EntrypointID string `gorm:"type:uuid;not null" json:"entrypointId"`
@@ -44,7 +42,7 @@ type Event struct {
// existed. It is not a foreign key: the source event can be
// reaped by retention while its copies remain, and the id is
// kept as the record of where the copy came from either way.
ResubmittedFromID *string `gorm:"type:uuid;index:idx_events_resubmitted_from_id,priority:1" json:"resubmittedFromId,omitempty"`
ResubmittedFromID *string `gorm:"type:uuid;index" json:"resubmittedFromId,omitempty"`
// Relations. No model marshals the record it belongs to, so
// Webhook and Entrypoint are left out of the JSON.
@@ -515,47 +515,6 @@ func (w *archiveWriter) prune(expiry time.Duration) {
}
}
// ArchiveFileInfo is what the metadata of a database target's archive
// file says about it.
type ArchiveFileInfo struct {
// Size is the bytes on disk of the file and its -wal together.
Size int64
// Written is when the file or its -wal was last modified, whichever
// is later: a write lands in the -wal first.
Written time.Time
}
// StatArchive reads the metadata of the archive file at path and of
// its -wal, without opening the archive. With no file at path, which is
// so before the first write and after the operator moved it away, the
// error wraps fs.ErrNotExist.
func StatArchive(path string) (ArchiveFileInfo, error) {
file, err := os.Stat(path)
if err != nil {
return ArchiveFileInfo{}, err
}
info := ArchiveFileInfo{Size: file.Size(), Written: file.ModTime()}
wal, err := os.Stat(path + "-wal")
if errors.Is(err, fs.ErrNotExist) {
return info, nil
}
if err != nil {
return ArchiveFileInfo{}, err
}
info.Size += wal.Size()
if wal.ModTime().After(info.Written) {
info.Written = wal.ModTime()
}
return info, nil
}
// fileExists reports whether a path currently exists.
func fileExists(path string) bool {
_, err := os.Stat(path)
-44
View File
@@ -3,7 +3,6 @@ package delivery_test
import (
"database/sql"
"fmt"
"io/fs"
"log/slog"
"os"
"path/filepath"
@@ -184,49 +183,6 @@ func TestArchiveWriter_RecreatesAfterRemoval(
assert.Equal(t, "b", got[0].EventID)
}
// TestStatArchive proves StatArchive finds no file before the first
// write; after a write still held in the -wal, counts the -wal in the
// size and takes its later time as the last write; and finds no file
// again once the file has been moved away.
func TestStatArchive(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "archive-wh.db")
_, err := delivery.StatArchive(path)
require.ErrorIs(t, err, fs.ErrNotExist)
// With the clock stopped, the reopen debounce never passes, so
// the handle stays open after the write.
stopped := time.Now()
w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
w.SetNow(func() time.Time { return stopped })
require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: "a"}, 0))
written := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
earlier := written.Add(-time.Hour)
require.NoError(t, os.Chtimes(path, earlier, earlier))
require.NoError(t, os.Chtimes(path+"-wal", written, written))
file, err := os.Stat(path)
require.NoError(t, err)
wal, err := os.Stat(path + "-wal")
require.NoError(t, err)
require.Positive(t, wal.Size())
got, err := delivery.StatArchive(path)
require.NoError(t, err)
assert.Equal(t, file.Size()+wal.Size(), got.Size)
assert.True(t, written.Equal(got.Written), got.Written)
removeArchiveFiles(t, path)
_, err = delivery.StatArchive(path)
require.ErrorIs(t, err, fs.ErrNotExist)
}
func TestArchiveWriter_ReopenDebounce(t *testing.T) {
t.Parallel()
+5 -22
View File
@@ -24,22 +24,12 @@ const maxRenderedBodyBytes = 32 << 10
const maxInlineBodyLines = 200
// maxIndentDepth is how deeply a JSON body's objects and arrays may
// nest for it to be indented at all; a deeper one is shown as received.
// nest for it to be pretty-printed; a deeper one is shown as received.
// Each level indents every line inside it two more spaces, so 10 KB of
// nested brackets would indent to some 50 MB; within this depth a body
// grows at most 35 times.
const maxIndentDepth = 16
// A JSON body is shown pretty-printed only when that makes it at most
// maxIndentGrowth times its size plus indentAllowance bytes, and
// otherwise as received, so that indenting does not undo
// maxRenderedBodyBytes. The allowance keeps a small nested body
// pretty-printed.
const (
maxIndentGrowth = 4
indentAllowance = 1 << 10
)
// jsonIndent is the indent of a pretty-printed JSON body.
const jsonIndent = " "
@@ -101,14 +91,8 @@ func newBodyView(eventURL string, body []byte, size int64) BodyView {
body = indentJSON(body)
}
// The page shows a carriage return, a line feed, or the two
// together as one line break. A final one ends the last line
// rather than starting another.
text := bytes.TrimSuffix(body, []byte("\n"))
text = bytes.TrimSuffix(text, []byte("\r"))
breaks := bytes.Count(text, []byte("\n")) + bytes.Count(text, []byte("\r")) -
bytes.Count(text, []byte("\r\n"))
lines := breaks + 1
// A final newline ends the last line rather than starting another.
lines := bytes.Count(bytes.TrimSuffix(body, []byte("\n")), []byte("\n")) + 1
v.Text = string(body)
v.Scroll = lines > maxInlineBodyLines || size > maxRenderedBodyBytes
@@ -117,8 +101,7 @@ func newBodyView(eventURL string, body []byte, size int64) BodyView {
}
// indentJSON returns body pretty-printed when it is a JSON document,
// and unchanged when it is not, nests deeper than maxIndentDepth, or
// would grow past maxIndentGrowth times its size.
// and unchanged when it is not or nests deeper than maxIndentDepth.
func indentJSON(body []byte) []byte {
if !json.Valid(body) || !indentFits(body) {
return body
@@ -127,7 +110,7 @@ func indentJSON(body []byte) []byte {
var out bytes.Buffer
err := json.Indent(&out, body, "", jsonIndent)
if err != nil || out.Len() > maxIndentGrowth*len(body)+indentAllowance {
if err != nil {
return body
}
+3 -24
View File
@@ -40,8 +40,9 @@ func TestNewBodyView_FormatsValidJSON(t *testing.T) {
assert.False(t, v.Scroll)
}
// TestNewBodyView_FormatsNestedJSON proves a small document with a
// few levels of nesting is pretty-printed.
// TestNewBodyView_FormatsNestedJSON proves a document with a few
// levels of nesting is pretty-printed: only deep nesting is shown
// as received.
func TestNewBodyView_FormatsNestedJSON(t *testing.T) {
t.Parallel()
@@ -97,23 +98,6 @@ func TestNewBodyView_DeepJSONAsReceived(t *testing.T) {
assert.Equal(t, body, bodyView(body).Text)
}
// TestNewBodyView_GrowingJSONAsReceived proves a JSON body that
// pretty-printing would make more than four times its size is shown
// as it arrived, however shallow: each short element eight levels
// deep gets a line indented sixteen spaces.
func TestNewBodyView_GrowingJSONAsReceived(t *testing.T) {
t.Parallel()
numbers := func(n int) string {
return strings.Repeat("[", 8) +
strings.TrimSuffix(strings.Repeat("1,", n), ",") +
strings.Repeat("]", 8)
}
assert.NotEqual(t, numbers(10), bodyView(numbers(10)).Text)
assert.Equal(t, numbers(1000), bodyView(numbers(1000)).Text)
}
// TestNewBodyView_ScrollsPast200Lines proves a body is shown at
// its full height up to 200 lines and in the scrolling box past
// them, counting the lines after formatting.
@@ -127,11 +111,6 @@ func TestNewBodyView_ScrollsPast200Lines(t *testing.T) {
assert.False(t, bodyView(lines(200)+"\n").Scroll)
assert.True(t, bodyView(lines(201)+"\n").Scroll)
// The page shows a carriage return, a line feed, or the two
// together as one line break.
assert.True(t, bodyView(strings.Repeat("line\r", 400)).Scroll)
assert.False(t, bodyView(strings.Repeat("line\r\n", 200)).Scroll)
// One line as received, 201 once formatted: the brackets and
// 199 elements.
numbers := "[" + strings.TrimSuffix(strings.Repeat("1,", 199), ",") + "]"
+1 -1
View File
@@ -521,7 +521,7 @@ func (h *Handlers) renderSourceDetail(
// target's stored config blob holds a credential, and it
// must never reach a template.
"Entrypoints": NewEntrypointViews(entrypoints),
"Targets": h.targetRows(&webhook, targets),
"Targets": delivery.NewTargetViews(targets),
"Events": events,
"BaseURL": baseURL,
"Stats": h.loadWebhookStats(webhook.ID, entrypoints, targets),
-90
View File
@@ -1,90 +0,0 @@
package handlers
import (
"errors"
"io/fs"
"path/filepath"
"time"
"github.com/dustin/go-humanize"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
)
// TargetRowView is one row of the target list on a webhook's page.
type TargetRowView struct {
delivery.TargetView
// Archive is a database target's archive file, and nil for a target
// of any other type.
Archive *ArchiveFileView
}
// ArchiveFileView is what a database target's row shows about its
// archive file.
type ArchiveFileView struct {
Name string
// Note stands in for the size and the last write when there are
// none to show, and is empty when there are.
Note string
// Size is the size on disk. Written is how long ago the file was
// last written, and WrittenUTC the full time the page shows on
// hover.
Size string
Written string
WrittenUTC string
}
// targetRows projects a webhook's targets for the target list on its
// page.
func (h *Handlers) targetRows(
webhook *database.Webhook, targets []database.Target,
) []TargetRowView {
views := delivery.NewTargetViews(targets)
rows := make([]TargetRowView, len(views))
// NewTargetViews returns one view per target, in order.
for i := range views {
rows[i].TargetView = views[i]
if targets[i].Type == database.TargetTypeDatabase {
rows[i].Archive = h.archiveFileView(webhook, &targets[i])
}
}
return rows
}
// archiveFileView describes a database target's archive file from the
// file's metadata alone; the archive is never opened. The file is found
// by the name the archive writer uses, so it follows a rename of the
// webhook or the target.
func (h *Handlers) archiveFileView(
webhook *database.Webhook, target *database.Target,
) *ArchiveFileView {
path := delivery.ArchivePath(h.dbMgr, webhook, target)
view := &ArchiveFileView{Name: filepath.Base(path)}
file, err := delivery.StatArchive(path)
switch {
case errors.Is(err, fs.ErrNotExist):
view.Note = "not created yet"
case err != nil:
h.log.Error(
"failed to read archive file metadata",
"target_id", target.ID,
"error", err,
)
view.Note = "could not be read"
default:
view.Size = humanize.Bytes(uint64(file.Size)) //nolint:gosec // never negative
view.Written = humanize.Time(file.Written)
view.WrittenUTC = file.Written.UTC().Format(time.DateTime) + " UTC"
}
return view
}
-71
View File
@@ -1,71 +0,0 @@
package handlers_test
import (
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/session"
)
// TestHandleSourceDetail_ShowsArchiveFile proves a database target's
// row names its archive file and says "not created yet" before the
// first write, adds the file's size and last write once it has one row,
// and says "not created yet" again once the file has been moved away.
// A target of another type shows no archive file.
func TestHandleSourceDetail_ShowsArchiveFile(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := seedWebhook(t, db)
archive := seedTarget(t, db, wh.ID, database.TargetTypeDatabase)
seedTarget(t, db, wh.ID, database.TargetTypeLog)
path := delivery.ArchivePath(dbMgr, wh, archive)
body := renderSourceDetailPage(t, h, sess, wh.ID)
assert.Equal(t, 1, strings.Count(body, "Archive File:"))
assert.Contains(t, body, filepath.Base(path))
assert.Contains(t, body, "not created yet")
assert.NotContains(t, body, "Archive Size:")
seedArchive(t, path, 1, 100)
file, err := os.Stat(path)
require.NoError(t, err)
body = renderSourceDetailPage(t, h, sess, wh.ID)
assert.Contains(t, body, filepath.Base(path))
assert.NotContains(t, body, "not created yet")
assert.Regexp(t,
`Archive Size:</span>\s*<span>[1-9][0-9.]* [kM]?B</span>`, body,
)
assert.Contains(t, body,
`title="`+file.ModTime().UTC().Format(time.DateTime)+` UTC"`,
)
require.NoError(t, os.Rename(path, filepath.Join(t.TempDir(), "moved.db")))
body = renderSourceDetailPage(t, h, sess, wh.ID)
assert.Contains(t, body, filepath.Base(path))
assert.Contains(t, body, "not created yet")
assert.NotContains(t, body, "Archive Size:")
}
+2 -5
View File
@@ -5,10 +5,7 @@ import (
"embed"
)
// Static holds the CSS and JavaScript files the web UI's pages load. They
// are named one by one so that a missing js/alpine.min.js, which make
// assets extracts and git does not track, fails the build instead of
// leaving the pages without Alpine.js.
// Static holds the embedded CSS and JavaScript files for the web UI.
//
//go:embed css/tailwind.css css/style.css js/app.js js/alpine.min.js
//go:embed css js
var Static embed.FS
-17
View File
@@ -179,23 +179,6 @@
<span>{{.Value}}</span>
</div>
{{end}}
{{with .Archive}}
<div class="text-xs text-gray-500 mt-1">
<span class="font-medium text-gray-700">Archive File:</span>
<span class="break-all">{{.Name}}</span>
{{with .Note}}<span>({{.}})</span>{{end}}
</div>
{{if .Size}}
<div class="text-xs text-gray-500 mt-1">
<span class="font-medium text-gray-700">Archive Size:</span>
<span>{{.Size}}</span>
</div>
<div class="text-xs text-gray-500 mt-1">
<span class="font-medium text-gray-700">Last Written:</span>
<span title="{{.WrittenUTC}}">{{.Written}}</span>
</div>
{{end}}
{{end}}
</div>
{{else}}
<div class="p-4 text-sm text-gray-500">No targets configured.</div>