4 Commits
Author SHA1 Message Date
clawbot db42815c4c Add a Download button that exports a database target's archive as gzipped JSON (closes #374)
check / check (push) Successful in 3m25s
Each database target on the webhook page links to
/hook/ID/targets/TARGETID/download, which streams the target's archive
as archive-WEBHOOK-TARGET-YYYYMMDDTHHMMSSZ.json.gz: the webhook, the
target, exported_at, and archived_events, one object per row keyed by
column, a body that is not valid UTF-8 in base64 with body_encoding.
The export reads one row at a time on its own connection inside a
read-only transaction: one snapshot, and no write lock. The handler
holds the rename lock only while it reads the names and opens the file.
A missing archive exports empty and is not created.

Model: opus-5-5
2026-10-02 12:13:14 +00:00
clawbot 8cf5acaf1d Justify the handlers tests' start limit with a measurement (closes #225)
check / check (push) Successful in 3m57s
The internal/handlers tests were load-fragile because every test app hashed the admin password at 64 MB; that went with the cheaper test hashing already on next, and measuring under the host's real load found nothing left to fix in how the tests run. The comment on newTestApp now says its start limit, fx's default, is there to catch a start that hangs, and that the slowest measured start is far inside it. The header of script/test gives current figures in place of ones from before that change. Neither limit changes, and no test changes.

Model: opus-5-5
2026-10-02 14:08:54 +02:00
clawbot 9ade217222 Show each webhook's activity in the webhook list (closes #394)
check / check (push) Successful in 3m38s
Each entry in the webhook list at /hooks now shows when its last event arrived (or "No events yet"), how many of its deliveries failed in the last 24 hours, in red when not zero, and how many entrypoints and targets are inactive, as in "4 targets, 1 inactive". The figures come from the event totals row and the statistics pane's own query, so the list no longer counts every stored event and its event count matches the pane's. A webhook whose event database cannot be read says so in its entry; the rest of the list still shows. A failed read of entrypoints or targets from the main database now returns an error page instead of showing zero.

Model: opus-5-5
2026-10-02 13:57:43 +02:00
clawbot 5551f75251 Say at the receiver route where its 1 MB body cap lives (closes #173)
check / check (push) Successful in 3m22s
The receiver route /h/{uuid} has no MaxBodySize middleware, unlike the page route groups; its 1 MB cap is enforced in the handler, which owns the response senders see. Nothing at the route said so, so a reader could take the receiver for uncapped or remove the only bound on the one unauthenticated endpoint. The route registration and the handler's body-reading function now say where the cap lives and why. A routing test sends a body of exactly 1 MB and one a byte over through the production router and pins that only the second is refused, with the handler's 413 and message.

Model: opus-5-5
2026-10-02 13:40:00 +02:00
18 changed files with 1702 additions and 45 deletions

No files matched your search

+36 -2
View File
@@ -1786,7 +1786,7 @@ retries) is individually logged for full observability.
#### EventTotals and TargetTotals
Running counts in each event database, read by the statistics pane at the
top of the webhook page. `EventTotals` is one row:
top of the webhook page and by the webhook list. `EventTotals` is one row:
| Field | Type | Description |
| ---------------- | --------- | ----------- |
@@ -1818,6 +1818,14 @@ target. Its failure percentage for a window is the deliveries that became
`failed` in it out of all that became `delivered` or `failed` in it, and
a dash when none did.
The webhook list at `/hooks` shows three of the pane's figures for each
webhook: its events within retention and its last event, both from
`EventTotals`, and its deliveries that failed in the last 24 hours,
counted with the pane's query. It opens each webhook's event database once
(the handle stays open) and runs those two reads there, so its cost grows
with the number of webhooks and, for each, with the deliveries that
finished in the last 24 hours, never with the events stored.
#### Event-tier indexes
These indexes on the per-webhook event databases are declared in the model
@@ -1825,7 +1833,7 @@ tags, so `AutoMigrate` creates them on a fresh database:
| Table | Columns | Serves |
| ------------------ | --------------------------- | ------ |
| `deliveries` | `status`, `deleted_at`, `finished_at`, `target_id` | Startup recovery, the retry and pending sweeps every 60 seconds and the queue-depth sampler every 30 seconds, which select deliveries by status, and the webhook page's statistics, which count each target's deliveries by status and when they finished |
| `deliveries` | `status`, `deleted_at`, `finished_at`, `target_id` | Startup recovery, the retry and pending sweeps every 60 seconds and the queue-depth sampler every 30 seconds, which select deliveries by status, and the webhook page's statistics and the webhook list, which count each target's deliveries by status and when they finished |
| `deliveries` | `event_id`, `deleted_at` | The event log, which loads each event's deliveries, and retention, which counts and deletes the deliveries of expired events |
| `delivery_results` | `delivery_id`, `deleted_at` | The event log, which loads the attempts of a page's deliveries, and retention, which deletes the attempts of expired events |
| `events` | `deleted_at`, `created_at` | The webhook page's statistics, which count recent events |
@@ -1984,6 +1992,30 @@ Because each `database` target has its own archive file, a target's
webhook with different expiries keep two archives, each pruned on its
own schedule.
Each `database` target on the webhook page has a **Download** button,
which returns its archive as one gzipped JSON file,
`archive-{webhook_name}-{target_name}-{YYYYMMDDTHHMMSSZ}.json.gz`, the
names made safe as above and the time in UTC. The file holds one
object: `webhook` and `target`, each an `id` and a `name`;
`exported_at`; and `archived_events`, one object per archived row with
every column, keyed by column name. A body that is not valid UTF-8 is
written in base64, with `"body_encoding": "base64"` beside it. An
archive that does not exist yet, or was moved away, downloads with an
empty `archived_events`; the download never creates the file.
The download streams: each row is read and written out compressed
before the next is read, so neither the archive nor the JSON is held in
memory. It reads on a connection of its own, inside one read-only
transaction, so the file holds the archive as it stood when the
download started, and archive writes go on meanwhile, since under WAL a
reader never blocks a writer. While it runs, the `-wal` cannot be
checkpointed past what it reads, so a long download lets the `-wal`
grow. It finds the file by the stored names under the lock that webhook
edits, target edits and target creation hold, and lets go once the file
is open: a rename during the download moves the file without affecting
it. Like every request it is cut off after 60 seconds, which leaves the
file incomplete and failing to decompress.
Deleting a webhook releases its archives: the delivery engine's cached
archive writers are dropped and their file handles closed, so nothing
lingers after the webhook is gone. The archive **files themselves are
@@ -2846,6 +2878,7 @@ returns to the page that was asked for.
| `POST` | `/hook/{id}/targets` | Add target to webhook |
| `GET` | `/hook/{id}/targets/{targetID}/edit` | Edit target form. The one page that renders a target's destination URL and header values in full, rather than masked |
| `POST` | `/hook/{id}/targets/{targetID}/edit` | Edit target submission |
| `GET` | `/hook/{id}/targets/{targetID}/download` | Download a `database` target's archive as one gzipped JSON file. See [Database Architecture](#database-architecture) |
| `POST` | `/hook/{id}/targets/{targetID}/delete` | Delete a target |
| `POST` | `/hook/{id}/targets/{targetID}/toggle` | Enable or disable a target |
@@ -2926,6 +2959,7 @@ webhooker/
│ │ ├── target_slack.go # Slack/Mattermost incoming-webhook target
│ │ ├── target_database.go # Database archive target
│ │ ├── target_database_archive.go # Archive file lifecycle and pruning
│ │ ├── target_database_export.go # Archive download as gzipped JSON
│ │ ├── target_log.go # Log target (stdout)
│ │ ├── target_config_view.go # Masked target config for templates
│ │ ├── archive_sweeper.go # Periodic pruning of idle archives
+6 -10
View File
@@ -3,7 +3,6 @@ package delivery
import (
"context"
"fmt"
"path/filepath"
"strings"
"sync"
"time"
@@ -277,10 +276,9 @@ func (t *databaseTarget) releaseSweepWriter(
}
// newWriter builds the writer for a database target's archive. The
// file lives beside the webhook's event database in the data
// directory and is named for the webhook and the target as the main
// database has them now; from then on only rename changes the name
// the writer uses. It does not touch the archive file.
// file is the one ArchivePath gives for the webhook and the target as
// the main database names them now; from then on only rename changes
// the name the writer uses. It does not touch the archive file.
func (t *databaseTarget) newWriter(
targetID string,
) (*archiveWriter, error) {
@@ -299,12 +297,10 @@ func (t *databaseTarget) newWriter(
)
}
dir := filepath.Dir(t.eng.dbManager.DBPath(target.WebhookID))
name := ArchiveFileName(
target.Webhook.Name, target.Name, target.ID,
w := newArchiveWriter(
ArchivePath(t.eng.dbManager, &target.Webhook, &target),
t.eng.log,
)
w := newArchiveWriter(filepath.Join(dir, name), t.eng.log)
w.webhookID = target.WebhookID
return w, nil
+270
View File
@@ -0,0 +1,270 @@
package delivery
import (
"compress/gzip"
"context"
"database/sql"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"io"
"path/filepath"
"time"
"unicode/utf8"
"sneak.berlin/go/webhooker/internal/database"
)
// archiveTableQuery counts the archive's table: 0 when the archive
// writer has created the file but not yet the table in it.
const archiveTableQuery = "SELECT count(*) FROM sqlite_master " +
"WHERE type = 'table' AND name = 'archived_events'"
// archiveNextRowQuery reads every column of the first archived row
// after a given id. An export reads the archive a row at a time this
// way rather than through one cursor, because the Scan check in
// internal/gormlog accepts only a Scan straight on QueryRowContext's
// result.
const archiveNextRowQuery = "SELECT id, event_id, webhook_id, " +
"entrypoint_id, method, headers, body, content_type, archived_at " +
"FROM archived_events WHERE id > ? ORDER BY id LIMIT 1"
// ArchivePath returns where a database target's archive file is: in
// the data directory, beside the webhook's event database, under the
// name ArchiveFileName gives it.
func ArchivePath(
dbMgr *database.WebhookDBManager,
webhook *database.Webhook,
target *database.Target,
) string {
return filepath.Join(
filepath.Dir(dbMgr.DBPath(webhook.ID)),
ArchiveFileName(webhook.Name, target.Name, target.ID),
)
}
// ArchiveExportFileName returns the name a database target's archive
// downloads under:
// archive-WEBHOOKNAME-TARGETNAME-YYYYMMDDTHHMMSSZ.json.gz, the names
// made safe as in ArchiveFileName and the time in UTC.
func ArchiveExportFileName(
webhookName, targetName string, at time.Time,
) string {
return "archive-" + archiveNamePart(webhookName) + "-" +
archiveNamePart(targetName) + "-" +
at.UTC().Format("20060102T150405Z") + ".json.gz"
}
// ArchiveExport is a database target's archive opened for download.
// It reads the file on its own connection, inside one read-only
// transaction, so it writes out the archive as it stood when
// OpenArchiveExport returned.
//
// Archives are in WAL mode, where a reader works from a snapshot and
// never blocks a writer: archive writes go on while an export is open,
// and the export does not see them. SQLite cannot checkpoint the -wal
// past an open snapshot, so the -wal grows until the export is closed.
type ArchiveExport struct {
db *sql.DB
tx *sql.Tx
// empty is true when there is nothing to read: no file, or a file
// without the archive's table yet.
empty bool
}
// exportedName is how an export names its webhook and its target.
type exportedName struct {
ID string `json:"id"`
Name string `json:"name"`
}
// OpenArchiveExport opens the archive file at path for export and
// takes the snapshot the export reads. It never creates the file: with
// no file at path, the export has no rows.
//
// Once it has returned, the file is open, so a rename or a move of it
// does not affect the export, which reads the same file under its new
// name.
func OpenArchiveExport(
ctx context.Context, path string,
) (*ArchiveExport, error) {
if !fileExists(path) {
return &ArchiveExport{empty: true}, nil
}
db, err := database.OpenSQLite(path, archiveModeExisting)
if err != nil {
return nil, fmt.Errorf("opening archive %s: %w", path, err)
}
// ReadOnly makes the driver begin a deferred transaction in place
// of the BEGIN IMMEDIATE the connection string asks for, so the
// export never takes the archive's write lock.
tx, err := db.BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
if err != nil {
_ = db.Close()
return nil, fmt.Errorf("reading archive %s: %w", path, err)
}
// The transaction's first read is what takes the snapshot.
var tables int
err = tx.QueryRowContext(ctx, archiveTableQuery).Scan(&tables)
if err != nil {
_ = tx.Rollback()
_ = db.Close()
return nil, fmt.Errorf("reading archive %s: %w", path, err)
}
return &ArchiveExport{db: db, tx: tx, empty: tables == 0}, nil
}
// WriteGzipJSON writes the export to w as one gzipped JSON object:
// webhook and target, each an id and a name; exported_at; and
// archived_events, one object per archived row, keyed by column name.
// A body that is not valid UTF-8 cannot be a JSON string, so it is
// written in base64, with "body_encoding": "base64" beside it.
//
// Each row is written out before the next is read, so neither the
// archive nor its JSON is ever held in memory whole. After an error
// the gzip stream is left unfinished, so what was written does not
// decompress as a whole file.
func (x *ArchiveExport) WriteGzipJSON(
ctx context.Context,
w io.Writer,
webhook *database.Webhook,
target *database.Target,
exportedAt time.Time,
) error {
head, err := json.Marshal(map[string]any{
"webhook": exportedName{ID: webhook.ID, Name: webhook.Name},
"target": exportedName{ID: target.ID, Name: target.Name},
"exported_at": exportedAt.UTC(),
})
if err != nil {
return fmt.Errorf("encoding archive export: %w", err)
}
zw := gzip.NewWriter(w)
err = x.writeJSON(ctx, zw, head)
if err != nil {
return fmt.Errorf("writing archive export: %w", err)
}
return zw.Close()
}
// Close ends the export's transaction and closes its connection.
func (x *ArchiveExport) Close() error {
if x.db == nil {
return nil
}
_ = x.tx.Rollback()
return x.db.Close()
}
// writeJSON writes head with archived_events added as its last key,
// the rows going into it one at a time.
func (x *ArchiveExport) writeJSON(
ctx context.Context, w io.Writer, head []byte,
) error {
// head goes out without its closing brace, so that
// archived_events can follow it.
_, err := w.Write(head[:len(head)-1])
if err != nil {
return err
}
_, err = io.WriteString(w, `,"archived_events":[`)
if err != nil {
return err
}
err = x.writeRows(ctx, w)
if err != nil {
return err
}
_, err = io.WriteString(w, "\n]}\n")
return err
}
// writeRows writes each archived row to w, oldest first, one per line,
// separated by commas.
func (x *ArchiveExport) writeRows(ctx context.Context, w io.Writer) error {
if x.empty {
return nil
}
next, err := x.tx.PrepareContext(ctx, archiveNextRowQuery)
if err != nil {
return err
}
defer func() { _ = next.Close() }()
var ev archivedEvent
for sep := "\n"; ; sep = ",\n" {
err = next.QueryRowContext(ctx, ev.ID).Scan(
&ev.ID, &ev.EventID, &ev.WebhookID, &ev.EntrypointID,
&ev.Method, &ev.Headers, &ev.Body, &ev.ContentType,
&ev.ArchivedAt,
)
if errors.Is(err, sql.ErrNoRows) {
return nil
}
if err != nil {
return err
}
_, err = io.WriteString(w, sep)
if err != nil {
return err
}
err = writeRow(w, &ev)
if err != nil {
return err
}
}
}
// writeRow writes an archived row to w as a JSON object keyed by
// column name, its body in base64 when it is not valid UTF-8.
func writeRow(w io.Writer, ev *archivedEvent) error {
row := map[string]any{
"id": ev.ID,
"event_id": ev.EventID,
"webhook_id": ev.WebhookID,
"entrypoint_id": ev.EntrypointID,
"method": ev.Method,
"headers": ev.Headers,
"body": ev.Body,
"content_type": ev.ContentType,
"archived_at": ev.ArchivedAt.UTC(),
}
if !utf8.ValidString(ev.Body) {
row["body"] = base64.StdEncoding.EncodeToString([]byte(ev.Body))
row["body_encoding"] = "base64"
}
line, err := json.Marshal(row)
if err != nil {
return err
}
_, err = w.Write(line)
return err
}
@@ -0,0 +1,373 @@
package delivery_test
import (
"bytes"
"compress/gzip"
"encoding/base64"
"encoding/json"
"fmt"
"io"
"os"
"path/filepath"
"runtime"
"runtime/debug"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
)
// The webhook and the target the export tests' archives belong to.
const (
exportWebhookID = "wh-export"
exportWebhookName = "Orders (EU)"
exportTargetID = "tgt-export"
exportTargetName = "Long-term archive"
)
const (
// binaryBody is a body that is not valid UTF-8.
binaryBody = "\xff\xfe\x00\x01binary\x80"
// openedEventID is the event the snapshot tests archive before
// they open the export.
openedEventID = "opened"
)
// writeExportTo writes export to w as the archive of the export tests'
// webhook and target, exported at 2026-10-02T12:03:04Z.
func writeExportTo(
t *testing.T, export *delivery.ArchiveExport, w io.Writer,
) error {
t.Helper()
return export.WriteGzipJSON(
t.Context(), w,
&database.Webhook{
BaseModel: database.BaseModel{ID: exportWebhookID},
Name: exportWebhookName,
},
&database.Target{
BaseModel: database.BaseModel{ID: exportTargetID},
Name: exportTargetName,
},
time.Date(2026, 10, 2, 12, 3, 4, 0, time.UTC),
)
}
// exportArchive runs a whole export of the archive at path and returns
// its JSON, decompressed and parsed.
func exportArchive(t *testing.T, path string) map[string]any {
t.Helper()
export, err := delivery.OpenArchiveExport(t.Context(), path)
require.NoError(t, err)
defer func() { require.NoError(t, export.Close()) }()
return writeExport(t, export)
}
// writeExport writes an opened export and returns its JSON,
// decompressed and parsed. Reading to the end makes the gzip reader
// check that the stream was finished.
func writeExport(
t *testing.T, export *delivery.ArchiveExport,
) map[string]any {
t.Helper()
var buf bytes.Buffer
require.NoError(t, writeExportTo(t, export, &buf))
zr, err := gzip.NewReader(&buf)
require.NoError(t, err)
raw, err := io.ReadAll(zr)
require.NoError(t, err)
var got map[string]any
require.NoError(t, json.Unmarshal(raw, &got))
return got
}
// exportedEvents returns an export's archived_events.
func exportedEvents(t *testing.T, got map[string]any) []map[string]any {
t.Helper()
list, ok := got["archived_events"].([]any)
require.True(t, ok, "archived_events must be an array: %v", got)
events := make([]map[string]any, len(list))
for i, v := range list {
events[i], ok = v.(map[string]any)
require.True(t, ok, "an archived event must be an object: %v", v)
}
return events
}
// exportedEventIDs returns the event_id of each of an export's
// archived_events.
func exportedEventIDs(t *testing.T, got map[string]any) []string {
t.Helper()
events := exportedEvents(t, got)
ids := make([]string, 0, len(events))
for _, ev := range events {
ids = append(ids, fmt.Sprint(ev["event_id"]))
}
return ids
}
// TestArchiveExport_MatchesStoredRows proves an export holds the
// webhook, the target, the time, and every column of every stored
// row: a body that is valid UTF-8 as a string, and one that is not in
// base64, marked as such.
func TestArchiveExport_MatchesStoredRows(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "archive.db")
w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
bodies := []string{`{"order":1}`, "plain text", "", binaryBody}
for i, body := range bodies {
require.NoError(t, w.Write(delivery.ExportArchivedEvent{
EventID: fmt.Sprintf("ev-%d", i),
WebhookID: exportWebhookID,
EntrypointID: "ep-1",
Method: "POST",
Headers: `{"X-Test":["yes"]}`,
Body: body,
ContentType: testContentType,
}, 0))
}
var stored []delivery.ExportArchivedEvent
require.NoError(t, openArchiveDBForRead(t, path).
Order("id").Find(&stored).Error)
got := exportArchive(t, path)
assert.Equal(t,
map[string]any{"id": exportWebhookID, "name": exportWebhookName},
got["webhook"],
)
assert.Equal(t,
map[string]any{"id": exportTargetID, "name": exportTargetName},
got["target"],
)
assert.Equal(t, "2026-10-02T12:03:04Z", got["exported_at"])
events := exportedEvents(t, got)
require.Len(t, events, len(bodies))
for i, row := range stored {
assertExportedRow(t, row, events[i])
}
}
// assertExportedRow checks that ev, from an export, holds every column
// of the stored row.
func assertExportedRow(
t *testing.T, row delivery.ExportArchivedEvent, ev map[string]any,
) {
t.Helper()
archivedAt, err := time.Parse(
time.RFC3339Nano, fmt.Sprint(ev["archived_at"]),
)
require.NoError(t, err)
assert.True(t, archivedAt.Equal(row.ArchivedAt))
assert.EqualValues(t, row.ID, ev["id"])
assert.Equal(t, row.EventID, ev["event_id"])
assert.Equal(t, row.WebhookID, ev["webhook_id"])
assert.Equal(t, row.EntrypointID, ev["entrypoint_id"])
assert.Equal(t, row.Method, ev["method"])
assert.Equal(t, row.Headers, ev["headers"])
assert.Equal(t, row.ContentType, ev["content_type"])
if row.Body != binaryBody {
assert.Equal(t, row.Body, ev["body"])
assert.Len(t, ev, 9, "the nine columns and nothing else: %v", ev)
return
}
body, err := base64.StdEncoding.DecodeString(fmt.Sprint(ev["body"]))
require.NoError(t, err)
assert.Equal(t, binaryBody, string(body))
assert.Equal(t, "base64", ev["body_encoding"])
assert.Len(t, ev, 10, "the nine columns and body_encoding: %v", ev)
}
// TestArchiveExport_Empty proves an archive with nothing in it exports
// as an empty archived_events: no file, which the export must not
// create; a file the archive writer has not yet put its table in; and
// a table with no rows.
func TestArchiveExport_Empty(t *testing.T) {
t.Parallel()
dir := t.TempDir()
missing := filepath.Join(dir, "missing.db")
noTable := filepath.Join(dir, "no-table.db")
noRows := filepath.Join(dir, "no-rows.db")
require.NoError(t, os.WriteFile(noTable, nil, 0o600))
require.NoError(t,
delivery.NewExportArchiveWriter(noRows, archiveTestLogger(), 0).
Open(0),
)
for _, path := range []string{missing, noTable, noRows} {
assert.Empty(t, exportedEvents(t, exportArchive(t, path)), path)
}
for _, suffix := range archiveFileSuffixes() {
assert.NoFileExists(t, missing+suffix)
}
}
// TestArchiveExport_ReadsOneSnapshot proves an export writes the
// archive as it was when it was opened, and holds up no archive
// write: a row written while the export is open is stored, and is not
// in the export. A write held up for the whole busy timeout would
// fail.
func TestArchiveExport_ReadsOneSnapshot(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "archive.db")
w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: openedEventID}, 0))
export, err := delivery.OpenArchiveExport(t.Context(), path)
require.NoError(t, err)
defer func() { require.NoError(t, export.Close()) }()
require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: "during"}, 0))
assert.Equal(t,
[]string{openedEventID}, exportedEventIDs(t, writeExport(t, export)),
)
var stored int64
require.NoError(t, openArchiveDBForRead(t, path).
Model(&delivery.ExportArchivedEvent{}).Count(&stored).Error)
assert.Equal(t, int64(2), stored)
}
// TestArchiveExport_SurvivesRename proves that renaming the archive
// while an export of it is open, as renaming its webhook or target
// does, leaves the export reading the same file.
func TestArchiveExport_SurvivesRename(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "archive-old.db")
w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: openedEventID}, 0))
export, err := delivery.OpenArchiveExport(t.Context(), path)
require.NoError(t, err)
defer func() { require.NoError(t, export.Close()) }()
require.NoError(t, w.Rename("archive-new.db"))
require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: "after"}, 0))
require.NoFileExists(t, path)
assert.Equal(t,
[]string{openedEventID}, exportedEventIDs(t, writeExport(t, export)),
)
}
// heapPeak is an io.Writer that discards what it is given and records
// the largest heap it saw at a write.
type heapPeak struct {
max uint64
}
func (p *heapPeak) Write(b []byte) (int, error) {
var m runtime.MemStats
runtime.ReadMemStats(&m)
p.max = max(p.max, m.HeapAlloc)
return len(b), nil
}
// TestArchiveExport_Streams proves an export holds neither the archive
// nor its JSON in memory whole: exporting a 16 MiB archive grows the
// heap by less than half of that. An export that read every row before
// writing, or built the JSON before writing it, would hold all 16 MiB
// at a write.
//
//nolint:paralleltest // It measures the heap, which tests share.
func TestArchiveExport_Streams(t *testing.T) {
const (
rows = 64
bodySize = 256 << 10
limit = rows * bodySize / 2
)
path := filepath.Join(t.TempDir(), "archive.db")
w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
body := strings.Repeat("x", bodySize)
for range rows {
require.NoError(t, w.Write(delivery.ExportArchivedEvent{Body: body}, 0))
}
export, err := delivery.OpenArchiveExport(t.Context(), path)
require.NoError(t, err)
defer func() { require.NoError(t, export.Close()) }()
// A low GC target collects garbage soon after it is made, so the
// heap at each write is close to what the export is holding.
defer debug.SetGCPercent(debug.SetGCPercent(10))
runtime.GC()
var start runtime.MemStats
runtime.ReadMemStats(&start)
peak := &heapPeak{}
require.NoError(t, writeExportTo(t, export, peak))
assert.Less(t, peak.max, start.HeapAlloc+limit,
"heap at the start %d, at its peak %d", start.HeapAlloc, peak.max,
)
}
// TestArchiveExportFileName proves the download is named for the
// webhook and the target, with the names made safe as for the archive
// file, and the export time in UTC.
func TestArchiveExportFileName(t *testing.T) {
t.Parallel()
cest := time.FixedZone("CEST", int((2 * time.Hour).Seconds()))
assert.Equal(t,
"archive-orders-eu-long-term-archive-20261002T120304Z.json.gz",
delivery.ArchiveExportFileName(
exportWebhookName, exportTargetName,
time.Date(2026, 10, 2, 14, 3, 4, 0, cest),
),
)
}
+2 -1
View File
@@ -97,7 +97,8 @@ type Handlers struct {
// names through the archive rename, the save and any move back.
// Interleaved, one could rename an archive between another's
// rename and save, leaving the file named for one edit and the
// stored names from the other.
// stored names from the other. An archive download holds it while
// it reads the stored names and opens the file they give.
renameMu sync.Mutex
// dummyVerifications counts the equivalent-cost verifications
+5
View File
@@ -181,6 +181,11 @@ func (r *recordingArchives) Renames() []archiveRename {
return out
}
// newTestApp returns an app whose RequireStart fails the test when
// starting takes longer than fx's default start timeout of 15s. That
// limit catches a start that hangs, not a busy host: measured with make
// test on 2026-10-02 at host load 58-69 on 48 cores, the slowest of this
// package's starts took 0.49s.
func newTestApp(
t *testing.T,
targets ...any,
+467
View File
@@ -0,0 +1,467 @@
package handlers_test
import (
"net/http"
"net/http/httptest"
"regexp"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm"
"gorm.io/gorm/clause"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/handlers"
"sneak.berlin/go/webhooker/internal/logger"
"sneak.berlin/go/webhooker/internal/session"
)
// failedHighlight is how the list marks a number of failed deliveries
// that is not zero.
const failedHighlight = `class="font-medium text-red-600"`
// listWebhook adds a webhook with the given name, owned by the test
// user.
func listWebhook(
t *testing.T, db *database.Database, name string,
) *database.Webhook {
t.Helper()
wh := &database.Webhook{UserID: deleteTestUserID, Name: name}
require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
return wh
}
// addEntrypoints adds the given number of entrypoints, all active or
// all inactive, to a webhook and returns their paths.
func addEntrypoints(
t *testing.T, db *database.Database, webhookID string,
count int, active bool,
) []string {
t.Helper()
paths := make([]string, count)
for i := range paths {
paths[i] = statsEntrypoint(t, db, webhookID, active)
}
return paths
}
// addTargets adds the given number of targets, all active or all
// inactive, to a webhook and returns them.
func addTargets(
t *testing.T, db *database.Database, webhookID string,
count int, active bool,
) []*database.Target {
t.Helper()
targets := make([]*database.Target, count)
for i := range targets {
targets[i] = seedTarget(t, db, webhookID, database.TargetTypeLog)
require.NoError(t, db.DB().Model(targets[i]).
Update("active", active).Error)
}
return targets
}
// renderWebhookList runs the real webhook list handler as the test user
// and returns the rendered page.
func renderWebhookList(
t *testing.T, h *handlers.Handlers, sess *session.Session,
) string {
t.Helper()
cookies := authenticatedCookies(
t, sess, deleteTestUserID, deleteTestUsername,
)
w := httptest.NewRecorder()
h.HandleSourceList().ServeHTTP(
w, getRequest(t, "/hooks", cookies, nil),
)
require.Equal(t, http.StatusOK, w.Code)
return w.Body.String()
}
// listCard returns one webhook's entry in a rendered webhook list, its
// markup as rendered and its text with the markup taken out and each
// run of space made one space.
func listCard(t *testing.T, page, webhookID string) (string, string) {
t.Helper()
_, card, found := strings.Cut(page, `href="/hook/`+webhookID+`"`)
require.True(t, found, "the list has no entry for %s", webhookID)
card, _, _ = strings.Cut(card, "</a>")
text := regexp.MustCompile(`<[^>]*>`).ReplaceAllString(card, " ")
return card, strings.Join(strings.Fields(text), " ")
}
// receiveEvents posts the given number of events to an entrypoint
// through the real receiver, and returns the webhook's event database
// and its events, oldest first.
func receiveEvents(
t *testing.T,
h *handlers.Handlers,
dbMgr *database.WebhookDBManager,
webhookID, path string,
count int,
) (*gorm.DB, []database.Event) {
t.Helper()
router := receiverRouter(h)
for range count {
require.Equal(t, http.StatusOK, postReceiver(t, router, path))
}
webhookDB, err := dbMgr.GetDB(webhookID)
require.NoError(t, err)
events := listEvents(t, webhookDB)
require.Len(t, events, count)
return webhookDB, events
}
// seedFailingWebhook adds a webhook with six entrypoints, two of them
// inactive, and seven targets, five of them inactive. Four events reach
// its two active targets, arriving 31, 5, 4 and 3 hours ago, and its
// event totals row records the last one. Three deliveries failed in the
// last 24 hours, two to the first target and one to the second, one
// failed 30 hours ago, two were delivered, and two are still pending.
// It returns the webhook and when its last event arrived.
func seedFailingWebhook(
t *testing.T,
h *handlers.Handlers,
db *database.Database,
dbMgr *database.WebhookDBManager,
) (*database.Webhook, time.Time) {
t.Helper()
wh := listWebhook(t, db, "failing")
paths := addEntrypoints(t, db, wh.ID, 4, true)
addEntrypoints(t, db, wh.ID, 2, false)
active := addTargets(t, db, wh.ID, 2, true)
first, second := active[0], active[1]
addTargets(t, db, wh.ID, 5, false)
webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, paths[0], 4)
now := time.Now()
lastEventAt := now.Add(-3 * time.Hour)
statsAge(t, webhookDB, events[0].ID, now.Add(-31*time.Hour))
statsAge(t, webhookDB, events[1].ID, now.Add(-5*time.Hour))
statsAge(t, webhookDB, events[2].ID, now.Add(-4*time.Hour))
statsAge(t, webhookDB, events[3].ID, lastEventAt)
require.NoError(t, database.AddEventTotals(webhookDB,
database.EventTotals{LastEventAt: &lastEventAt}))
statsFinish(t, webhookDB,
statsDelivery(t, webhookDB, events[0].ID, first.ID),
database.DeliveryStatusFailed, now.Add(-30*time.Hour))
statsFinish(t, webhookDB,
statsDelivery(t, webhookDB, events[0].ID, second.ID),
database.DeliveryStatusDelivered, now.Add(-30*time.Hour))
statsFinish(t, webhookDB,
statsDelivery(t, webhookDB, events[1].ID, first.ID),
database.DeliveryStatusFailed, now.Add(-time.Hour))
statsFinish(t, webhookDB,
statsDelivery(t, webhookDB, events[2].ID, first.ID),
database.DeliveryStatusFailed, now.Add(-time.Minute))
statsFinish(t, webhookDB,
statsDelivery(t, webhookDB, events[2].ID, second.ID),
database.DeliveryStatusFailed, now.Add(-time.Minute))
statsFinish(t, webhookDB,
statsDelivery(t, webhookDB, events[3].ID, second.ID),
database.DeliveryStatusDelivered, now.Add(-time.Minute))
return wh, lastEventAt
}
// seedHealthyWebhook adds a webhook with four entrypoints and two
// targets, all active, and three events, arriving 8, 7 and 6 hours ago
// and each delivered to both targets. Its event totals row records the
// last event. It returns the webhook and when its last event arrived.
func seedHealthyWebhook(
t *testing.T,
h *handlers.Handlers,
db *database.Database,
dbMgr *database.WebhookDBManager,
) (*database.Webhook, time.Time) {
t.Helper()
wh := listWebhook(t, db, "healthy")
paths := addEntrypoints(t, db, wh.ID, 4, true)
targets := addTargets(t, db, wh.ID, 2, true)
webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, paths[0], 3)
now := time.Now()
lastEventAt := now.Add(-6 * time.Hour)
statsAge(t, webhookDB, events[0].ID, now.Add(-8*time.Hour))
statsAge(t, webhookDB, events[1].ID, now.Add(-7*time.Hour))
statsAge(t, webhookDB, events[2].ID, lastEventAt)
require.NoError(t, database.AddEventTotals(webhookDB,
database.EventTotals{LastEventAt: &lastEventAt}))
for _, ev := range events {
for _, target := range targets {
statsFinish(t, webhookDB,
statsDelivery(t, webhookDB, ev.ID, target.ID),
database.DeliveryStatusDelivered, now)
}
}
return wh, lastEventAt
}
// lastEventText is how the list shows when the last event arrived.
func lastEventText(at time.Time) string {
return at.UTC().Format("2006-01-02 15:04:05 UTC")
}
// TestSourceList_ShowsActivityOfEachWebhook checks the figures the list
// shows for a webhook with recent failures, a healthy one, a new one
// that has received no event, and one without an event database.
func TestSourceList_ShowsActivityOfEachWebhook(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
failing, failingLastEvent := seedFailingWebhook(t, h, db, dbMgr)
healthy, healthyLastEvent := seedHealthyWebhook(t, h, db, dbMgr)
// Creating a webhook creates its event database.
fresh := listWebhook(t, db, "fresh")
require.NoError(t, dbMgr.CreateDB(fresh.ID))
addEntrypoints(t, db, fresh.ID, 2, true)
addTargets(t, db, fresh.ID, 3, true)
quiet := listWebhook(t, db, "quiet")
addEntrypoints(t, db, quiet.ID, 2, true)
addTargets(t, db, quiet.ID, 3, true)
page := renderWebhookList(t, h, sess)
card, text := listCard(t, page, failing.ID)
assert.Contains(t, text, "6 entrypoints, 2 inactive")
assert.Contains(t, text, "7 targets, 5 inactive")
assert.Contains(t, text, "4 events within retention")
assert.Contains(t, text, "Last event "+lastEventText(failingLastEvent))
assert.Contains(t, card,
failedHighlight+">3 failed deliveries in the last 24 hours<")
card, text = listCard(t, page, healthy.ID)
assert.Contains(t, text, "4 entrypoints")
assert.Contains(t, text, "2 targets")
assert.Contains(t, text, "3 events within retention")
assert.Contains(t, text, "Last event "+lastEventText(healthyLastEvent))
assert.Contains(t, text, "0 failed deliveries in the last 24 hours")
assert.NotContains(t, text, "inactive")
assert.NotContains(t, card, failedHighlight)
card, text = listCard(t, page, fresh.ID)
assert.Contains(t, text, "2 entrypoints")
assert.Contains(t, text, "3 targets")
assert.Contains(t, text, "0 events within retention")
assert.Contains(t, text, "No events yet")
assert.Contains(t, text, "0 failed deliveries in the last 24 hours")
assert.NotContains(t, card, failedHighlight)
card, text = listCard(t, page, quiet.ID)
assert.Contains(t, text, "2 entrypoints")
assert.Contains(t, text, "3 targets")
assert.Contains(t, text, "0 events within retention")
assert.Contains(t, text, "No events yet")
assert.Contains(t, text, "0 failed deliveries in the last 24 hours")
assert.NotContains(t, card, failedHighlight)
assert.False(t, dbMgr.DBExists(quiet.ID),
"showing the list must not create an event database")
}
// TestSourceList_CountsOnlyEventsWithinRetention checks that once
// retention has removed one of a webhook's three events, the list
// counts the two still stored.
func TestSourceList_CountsOnlyEventsWithinRetention(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
log *logger.Logger
)
app := newTestApp(t, &h, &sess, &db, &dbMgr, &log)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := &database.Webhook{
UserID: deleteTestUserID, Name: "pruned", RetentionDays: 14,
}
require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
paths := addEntrypoints(t, db, wh.ID, 3, true)
addTargets(t, db, wh.ID, 4, true)
webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, paths[0], 3)
statsAge(t, webhookDB, events[0].ID, time.Now().Add(-15*24*time.Hour))
statsPrune(t, db, dbMgr, log, webhookDB)
require.Len(t, listEvents(t, webhookDB), 2)
_, text := listCard(t, renderWebhookList(t, h, sess), wh.ID)
assert.Contains(t, text, "3 entrypoints")
assert.Contains(t, text, "4 targets")
assert.Contains(t, text, "2 events within retention")
}
// TestSourceList_LastEventSurvivesPruningEveryEvent checks that once
// retention has removed every event of a webhook, the list still shows
// when the last one arrived rather than "No events yet".
func TestSourceList_LastEventSurvivesPruningEveryEvent(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
log *logger.Logger
)
app := newTestApp(t, &h, &sess, &db, &dbMgr, &log)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := &database.Webhook{
UserID: deleteTestUserID, Name: "emptied", RetentionDays: 1,
}
require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
paths := addEntrypoints(t, db, wh.ID, 2, true)
addTargets(t, db, wh.ID, 3, true)
webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, paths[0], 1)
lastEventAt := time.Now().Add(-50 * time.Hour)
statsAge(t, webhookDB, events[0].ID, lastEventAt)
require.NoError(t, database.AddEventTotals(webhookDB,
database.EventTotals{LastEventAt: &lastEventAt}))
statsPrune(t, db, dbMgr, log, webhookDB)
require.Empty(t, listEvents(t, webhookDB))
_, text := listCard(t, renderWebhookList(t, h, sess), wh.ID)
assert.Contains(t, text, "0 events within retention")
assert.Contains(t, text, "Last event "+lastEventText(lastEventAt))
assert.NotContains(t, text, "No events yet")
}
// TestSourceList_CountsOfOneInSingular checks that a webhook with one
// entrypoint, one target, one event within retention and one failed
// delivery in the last 24 hours has each written in the singular.
func TestSourceList_CountsOfOneInSingular(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
wh := listWebhook(t, db, "single")
paths := addEntrypoints(t, db, wh.ID, 1, true)
targets := addTargets(t, db, wh.ID, 1, true)
webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, paths[0], 1)
now := time.Now()
lastEventAt := now.Add(-9 * time.Hour)
statsAge(t, webhookDB, events[0].ID, lastEventAt)
require.NoError(t, database.AddEventTotals(webhookDB,
database.EventTotals{LastEventAt: &lastEventAt}))
statsFinish(t, webhookDB,
statsDelivery(t, webhookDB, events[0].ID, targets[0].ID),
database.DeliveryStatusFailed, now.Add(-time.Hour))
card, text := listCard(t, renderWebhookList(t, h, sess), wh.ID)
assert.Contains(t, card, ">1 entrypoint<")
assert.Contains(t, card, ">1 target<")
assert.Contains(t, card, ">1 event within retention<")
assert.Contains(t, text, "Last event "+lastEventText(lastEventAt))
assert.Contains(t, card,
failedHighlight+">1 failed delivery in the last 24 hours<")
}
// TestSourceList_UnreadableEventDatabase checks that a webhook whose
// event database cannot be read says so in its entry instead of
// showing zeros, and that the rest of the list is still shown.
func TestSourceList_UnreadableEventDatabase(t *testing.T) {
t.Parallel()
var (
h *handlers.Handlers
sess *session.Session
db *database.Database
dbMgr *database.WebhookDBManager
)
app := newTestApp(t, &h, &sess, &db, &dbMgr)
app.RequireStart()
t.Cleanup(app.RequireStop)
broken := listWebhook(t, db, "broken")
addEntrypoints(t, db, broken.ID, 2, true)
addTargets(t, db, broken.ID, 3, true)
brokenDB, err := dbMgr.GetDB(broken.ID)
require.NoError(t, err)
require.NoError(t,
brokenDB.Migrator().DropTable(&database.EventTotals{}))
quiet := listWebhook(t, db, "quiet")
addEntrypoints(t, db, quiet.ID, 2, true)
addTargets(t, db, quiet.ID, 3, true)
page := renderWebhookList(t, h, sess)
_, text := listCard(t, page, broken.ID)
assert.Contains(t, text, "2 entrypoints")
assert.Contains(t, text, "3 targets")
assert.Contains(t, text, "The event figures could not be read.")
assert.NotContains(t, text, "events")
assert.NotContains(t, text, "failed")
_, text = listCard(t, page, quiet.ID)
assert.Contains(t, text, "No events yet")
}
+119 -22
View File
@@ -3,10 +3,12 @@ package handlers
import (
"encoding/json"
"errors"
"fmt"
"net/http"
"slices"
"strconv"
"strings"
"time"
"github.com/go-chi/chi"
"github.com/google/uuid"
@@ -20,9 +22,20 @@ import (
type WebhookListItem struct {
database.Webhook
EntrypointCount int64
TargetCount int64
EventCount int64
EntrypointCount int
InactiveEntrypointCount int
TargetCount int
InactiveTargetCount int
// EventCount is how many events the webhook holds, LastEventAt
// when the newest arrived (nil before the first), and
// FailedLast24Hours how many of its deliveries failed in the last
// 24 hours. When the webhook's event database could not be read,
// EventsUnreadable is set and these three are not known.
EventCount int64
LastEventAt *time.Time
FailedLast24Hours int64
EventsUnreadable bool
}
// errMissingURL signals that a required URL was not provided.
@@ -154,7 +167,12 @@ func (h *Handlers) HandleSourceList() http.HandlerFunc {
return
}
items := h.buildWebhookListItems(webhooks)
items, err := h.buildWebhookListItems(webhooks)
if err != nil {
h.serverError(w, r, "failed to list webhooks", err)
return
}
data := map[string]any{
"Webhooks": items,
@@ -164,36 +182,115 @@ func (h *Handlers) HandleSourceList() http.HandlerFunc {
}
}
// buildWebhookListItems builds list items with counts.
// buildWebhookListItems builds the list's entry for each webhook. It
// fails when the main database cannot be read. A webhook whose event
// database cannot be read is marked on its own entry, and the error is
// logged.
func (h *Handlers) buildWebhookListItems(
webhooks []database.Webhook,
) []WebhookListItem {
) ([]WebhookListItem, error) {
items := make([]WebhookListItem, len(webhooks))
since := time.Now().Add(-longWindow)
for i := range webhooks {
items[i].Webhook = webhooks[i]
item := &items[i]
item.Webhook = webhooks[i]
h.db.DB().Model(&database.Entrypoint{}).Where(
"webhook_id = ?", webhooks[i].ID,
).Count(&items[i].EntrypointCount)
var err error
h.db.DB().Model(&database.Target{}).Where(
"webhook_id = ?", webhooks[i].ID,
).Count(&items[i].TargetCount)
item.EntrypointCount, item.InactiveEntrypointCount, err =
h.countWithInactive(&database.Entrypoint{}, item.ID)
if err != nil {
return nil, err
}
if h.dbMgr.DBExists(webhooks[i].ID) {
webhookDB, err := h.dbMgr.GetDB(
webhooks[i].ID,
item.TargetCount, item.InactiveTargetCount, err =
h.countWithInactive(&database.Target{}, item.ID)
if err != nil {
return nil, err
}
// Opening an event database that does not exist would create
// it, and it would hold nothing to count.
if !h.dbMgr.DBExists(item.ID) {
continue
}
err = h.readListEventFigures(item, since)
if err != nil {
h.log.Error(
"failed to read webhook list figures",
"webhook_id", item.ID,
"error", err,
)
if err == nil {
webhookDB.Model(
&database.Event{},
).Count(&items[i].EventCount)
}
item.EventsUnreadable = true
}
}
return items
return items, nil
}
// countWithInactive returns how many entrypoints or targets, as model
// says, a webhook has, and how many of them are inactive.
func (h *Handlers) countWithInactive(
model any, webhookID string,
) (int, int, error) {
var active []bool
err := h.db.DB().Model(model).
Where("webhook_id = ?", webhookID).
Pluck("active", &active).Error
if err != nil {
return 0, 0, fmt.Errorf(
"reading active flags of webhook %s: %w", webhookID, err,
)
}
inactive := 0
for _, a := range active {
if !a {
inactive++
}
}
return len(active), inactive, nil
}
// readListEventFigures fills in the figures the list shows from the
// webhook's event database, with the statistics pane's own queries:
// the event count and last arrival from the event totals row, and the
// deliveries that failed since the given time from the deliveries'
// status index.
func (h *Handlers) readListEventFigures(
item *WebhookListItem, since time.Time,
) error {
webhookDB, err := h.dbMgr.GetDB(item.ID)
if err != nil {
return err
}
var totals database.EventTotals
err = webhookDB.Take(&totals).Error
if err != nil {
return fmt.Errorf("reading event totals: %w", err)
}
item.EventCount = totals.Events - totals.EventsRemoved
item.LastEventAt = totals.LastEventAt
byTarget, err := finishedByTarget(webhookDB, since)
if err != nil {
return err
}
for _, f := range byTarget {
item.FailedLast24Hours += f.Failed
}
return nil
}
// HandleSourceCreate shows the form to create a new webhook.
+82
View File
@@ -0,0 +1,82 @@
package handlers
import (
"net/http"
"time"
"sneak.berlin/go/webhooker/internal/database"
"sneak.berlin/go/webhooker/internal/delivery"
)
// HandleTargetDownload serves a database target's archive as one
// gzipped JSON file, named for the webhook, the target and the time;
// see delivery.ArchiveExport.WriteGzipJSON for what it holds. Other
// target types have no archive and are a 404.
func (h *Handlers) HandleTargetDownload() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
webhook, target, export, ok := h.openTargetArchive(w, r)
if !ok {
return
}
defer func() { _ = export.Close() }()
now := time.Now()
w.Header().Set("Content-Type", "application/gzip")
w.Header().Set(
"Content-Disposition",
`attachment; filename="`+delivery.ArchiveExportFileName(
webhook.Name, target.Name, now,
)+`"`,
)
err := export.WriteGzipJSON(r.Context(), w, &webhook, target, now)
if err != nil {
// The 200 has gone out. The client is left with a file
// that does not decompress; the log is the record.
h.log.Error(
"failed to export archive",
"target_id", target.ID,
"error", err,
)
}
}
}
// openTargetArchive opens the archive of the request's database target
// for export. It reports false once it has written the response.
//
// It holds renameMu, which every archive rename runs under, while it
// reads the stored names and opens the file, so the file it opens is
// the one those names give. It lets go before the export is streamed:
// once the file is open, a rename does not affect the export.
func (h *Handlers) openTargetArchive(
w http.ResponseWriter,
r *http.Request,
) (database.Webhook, *database.Target, *delivery.ArchiveExport, bool) {
h.renameMu.Lock()
defer h.renameMu.Unlock()
webhook, target, ok := h.ownedTarget(w, r)
if !ok {
return database.Webhook{}, nil, nil, false
}
if target.Type != database.TargetTypeDatabase {
h.renderError(w, r, http.StatusNotFound)
return database.Webhook{}, nil, nil, false
}
export, err := delivery.OpenArchiveExport(
r.Context(), delivery.ArchivePath(h.dbMgr, &webhook, target),
)
if err != nil {
h.serverError(w, r, "failed to open archive for export", err)
return database.Webhook{}, nil, nil, false
}
return webhook, target, export, true
}
+185
View File
@@ -0,0 +1,185 @@
package handlers_test
import (
"compress/gzip"
"encoding/json"
"net/http"
"net/http/httptest"
"net/url"
"sync"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"sneak.berlin/go/webhooker/internal/database"
)
// downloadPath is the archive download route of a target.
func downloadPath(webhookID, targetID string) string {
return "/hook/" + webhookID + "/targets/" + targetID + "/download"
}
// renameTarget submits the edit form renaming a target to Renamed.
func renameTarget(
env *sourceTestEnv, webhookID, targetID string,
) *httptest.ResponseRecorder {
form := url.Values{}
form.Set("name", "Renamed")
return submitTargetEdit(env, webhookID, targetID, form)
}
// TestHandleTargetDownload proves a database target's archive
// downloads as a gzipped JSON attachment named for the webhook, the
// target and the time, here with no archive file yet, so with no
// rows; and that a target of another type has no download.
func TestHandleTargetDownload(t *testing.T) {
t.Parallel()
env := setupSourceTest(t)
wh := seedWebhookWithRetention(t, env.db, 7)
archive := seedTarget(t, env.db, wh.ID, database.TargetTypeDatabase)
logTarget := seedTarget(t, env.db, wh.ID, database.TargetTypeLog)
w := serveTarget(
env, http.MethodGet, downloadPath(wh.ID, archive.ID), nil,
)
require.Equal(t, http.StatusOK, w.Code, w.Body.String())
assert.Equal(t, "application/gzip", w.Header().Get("Content-Type"))
assert.Regexp(t,
`^attachment; filename="archive-seeded-t-database-`+
`\d{8}T\d{6}Z\.json\.gz"$`,
w.Header().Get("Content-Disposition"),
)
zr, err := gzip.NewReader(w.Body)
require.NoError(t, err)
var got map[string]json.RawMessage
require.NoError(t, json.NewDecoder(zr).Decode(&got))
assert.JSONEq(t,
`{"id":"`+archive.ID+`","name":"t-database"}`,
string(got["target"]),
)
assert.JSONEq(t, `[]`, string(got["archived_events"]))
w = serveTarget(
env, http.MethodGet, downloadPath(wh.ID, logTarget.ID), nil,
)
assert.Equal(t, http.StatusNotFound, w.Code)
}
// TestHandleTargetDownload_WaitsForRename proves a download reads the
// target's names and opens its archive under the lock a rename holds:
// started while an edit is renaming the archive, it waits, and is
// named for the target's new name.
func TestHandleTargetDownload_WaitsForRename(t *testing.T) {
t.Parallel()
env := setupSourceTest(t)
wh := seedWebhookWithRetention(t, env.db, 7)
archive := seedTarget(t, env.db, wh.ID, database.TargetTypeDatabase)
renaming, release := env.archives.BlockNextRename()
edited := make(chan *httptest.ResponseRecorder, 1)
go func() {
edited <- renameTarget(env, wh.ID, archive.ID)
}()
<-renaming
downloaded := make(chan *httptest.ResponseRecorder, 1)
go func() {
downloaded <- serveTarget(
env, http.MethodGet, downloadPath(wh.ID, archive.ID), nil,
)
}()
select {
case <-downloaded:
release()
t.Fatal("the download did not wait for the rename")
case <-time.After(100 * time.Millisecond):
}
release()
require.Equal(t, http.StatusSeeOther, (<-edited).Code)
w := <-downloaded
require.Equal(t, http.StatusOK, w.Code)
assert.Contains(t,
w.Header().Get("Content-Disposition"), "archive-seeded-renamed-",
)
}
// stalledWriter is a response writer whose first write waits until
// resume is closed, closing writing when it starts to wait.
type stalledWriter struct {
*httptest.ResponseRecorder
once sync.Once
writing chan struct{}
resume chan struct{}
}
func (s *stalledWriter) Write(b []byte) (int, error) {
s.once.Do(func() {
close(s.writing)
<-s.resume
})
return s.ResponseRecorder.Write(b)
}
// TestHandleTargetDownload_StreamsWithoutTheLock proves a download
// lets go of the rename lock once its archive is open: while the
// download is stalled writing, an edit can still rename the target.
func TestHandleTargetDownload_StreamsWithoutTheLock(t *testing.T) {
t.Parallel()
env := setupSourceTest(t)
wh := seedWebhookWithRetention(t, env.db, 7)
archive := seedTarget(t, env.db, wh.ID, database.TargetTypeDatabase)
req := httptest.NewRequestWithContext(
t.Context(), http.MethodGet, downloadPath(wh.ID, archive.ID), nil,
)
for _, c := range env.cookies {
req.AddCookie(c)
}
sw := &stalledWriter{
ResponseRecorder: httptest.NewRecorder(),
writing: make(chan struct{}),
resume: make(chan struct{}),
}
downloaded := make(chan struct{})
go func() {
targetRouter(env).ServeHTTP(sw, req)
close(downloaded)
}()
<-sw.writing
edited := make(chan *httptest.ResponseRecorder, 1)
go func() {
edited <- renameTarget(env, wh.ID, archive.ID)
}()
select {
case w := <-edited:
assert.Equal(t, http.StatusSeeOther, w.Code)
case <-time.After(10 * time.Second):
t.Error("the rename waited for the download")
}
close(sw.resume)
<-downloaded
assert.Equal(t, http.StatusOK, sw.Code)
}
+6 -2
View File
@@ -37,14 +37,18 @@ const (
editAuthHeader = "Authorization: Bearer " + editBearerSecret
)
// targetRouter mounts the target create and edit routes on a chi
// router so the handlers see the URL parameters they read.
// targetRouter mounts the target create, edit and download routes on
// a chi router so the handlers see the URL parameters they read.
func targetRouter(env *sourceTestEnv) *chi.Mux {
router := chi.NewRouter()
router.Post(
"/hook/{sourceID}/targets",
env.handlers.HandleTargetCreate(),
)
router.Get(
"/hook/{sourceID}/targets/{targetID}/download",
env.handlers.HandleTargetDownload(),
)
router.Get(
"/hook/{sourceID}/targets/{targetID}/edit",
env.handlers.HandleTargetEdit(),
+3 -1
View File
@@ -150,7 +150,9 @@ func (h *Handlers) lookupEntrypoint(
return entrypoint, true
}
// readWebhookBody reads and validates the request body size.
// readWebhookBody reads and validates the request body size. This is
// the receiver's only body cap: /h/{uuid} has no MaxBodySize
// middleware (see Server.setupWebhookRoutes).
func (h *Handlers) readWebhookBody(
w http.ResponseWriter,
r *http.Request,
+69
View File
@@ -0,0 +1,69 @@
package server_test
import (
"compress/gzip"
"encoding/json"
"net/http"
"regexp"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm/clause"
"sneak.berlin/go/webhooker/internal/database"
)
// TestHook_DownloadArchive follows the Download link the webhook page
// shows for a database target, and only for it, and gets the archive
// as a gzipped JSON file. Signed out, the link leads to the login page.
func TestHook_DownloadArchive(t *testing.T) {
t.Parallel()
env := newTestEnv(t)
userID, _ := env.seedUser(t, "archivist", "somepassword")
cookies := env.authCookies(t, userID, "archivist")
wh := env.seedWebhook(t, userID)
env.seedTarget(t, wh.ID)
archive := &database.Target{
WebhookID: wh.ID,
Name: "kept",
Type: database.TargetTypeDatabase,
Active: true,
}
require.NoError(t,
env.db.DB().Omit(clause.Associations).Create(archive).Error,
)
page := env.get("/hook/"+wh.ID, cookies)
require.Equal(t, http.StatusOK, page.Code)
links := regexp.MustCompile(
`href="(/hook/[^/"]+/targets/[^/"]+/download)"`,
).FindAllStringSubmatch(page.Body.String(), -1)
require.Len(t, links, 1, "only the database target has a Download")
link := links[0][1]
assert.Equal(t,
"/hook/"+wh.ID+"/targets/"+archive.ID+"/download", link,
)
w := env.get(link, cookies)
require.Equal(t, http.StatusOK, w.Code)
assert.Equal(t, "application/gzip", w.Header().Get("Content-Type"))
zr, err := gzip.NewReader(w.Body)
require.NoError(t, err)
var got map[string]json.RawMessage
require.NoError(t, json.NewDecoder(zr).Decode(&got))
assert.JSONEq(t,
`{"id":"`+wh.ID+`","name":"routed"}`, string(got["webhook"]),
)
w = env.get(link, nil)
assert.Equal(t, http.StatusSeeOther, w.Code)
assert.Contains(t, w.Header().Get("Location"), "/pages/login")
}
+10
View File
@@ -312,6 +312,10 @@ func (s *Server) setupSourceRoutes() {
"/targets/{targetID}/edit",
s.h.HandleTargetEditSubmit(),
)
r.Get(
"/targets/{targetID}/download",
s.h.HandleTargetDownload(),
)
r.Post(
"/targets/{targetID}/delete",
s.h.HandleTargetDelete(),
@@ -324,6 +328,12 @@ func (s *Server) setupSourceRoutes() {
}
func (s *Server) setupWebhookRoutes() {
// No MaxBodySize here, unlike the page groups. The receiver's 1 MB
// body cap is in Handlers.readWebhookBody, because the handler
// owns the response a sender gets for an oversized body and
// MaxBodySize would change it. That cap is the only bound on this
// unauthenticated endpoint's body; TestReceiver_OversizeBodyRefused
// pins it.
s.router.With(s.mw.ReceiverRateLimit()).HandleFunc(
"/h/{uuid}",
s.h.HandleWebhook(),
+47
View File
@@ -1428,6 +1428,53 @@ func TestReceiver_EntrypointURLIsRateLimited(t *testing.T) {
)
}
// TestReceiver_OversizeBodyRefused pins the receiver's 1 MB body
// cap, which lives in the handler rather than in a MaxBodySize
// middleware. A body exactly at the cap is accepted; one byte over is
// refused with the handler's own 413.
func TestReceiver_OversizeBodyRefused(t *testing.T) {
t.Parallel()
const bodyCap = 1 << 20 // 1 MB
env := newTestEnvWithConfig(t, &config.Config{
DataDir: t.TempDir(),
Environment: config.EnvironmentDev,
ReceiverRateLimit: 10,
})
userID, _ := env.seedUser(t, "receiver", "somepassword")
wh := env.seedWebhook(t, userID)
require.NoError(t, env.db.DB().Omit(clause.Associations).Create(
&database.Entrypoint{
WebhookID: wh.ID,
Path: "0b7c3e5a-2d9f-4a61-8e4b-7c1d6f2a9e35",
Active: true,
},
).Error)
send := func(size int) *httptest.ResponseRecorder {
req := httptest.NewRequestWithContext(
context.Background(), http.MethodPost,
"/h/0b7c3e5a-2d9f-4a61-8e4b-7c1d6f2a9e35",
strings.NewReader(strings.Repeat("a", size)),
)
w := httptest.NewRecorder()
env.router.ServeHTTP(w, req)
return w
}
assert.Equal(
t, http.StatusOK, send(bodyCap).Code,
"a body exactly at the cap must be accepted",
)
w := send(bodyCap + 1)
assert.Equal(t, http.StatusRequestEntityTooLarge, w.Code)
assert.Equal(t, "Request body too large\n", w.Body.String())
}
// metricsConfig is a Config differing from the routing default only
// in the two /metrics credentials.
func metricsConfig(
+9 -3
View File
@@ -2,9 +2,10 @@
# script/test: run the test suite.
#
# -timeout is applied by `go test` per package, not to the run as a whole, so
# it only has to clear the slowest single package. That is internal/handlers,
# measured in a cache-defeated builder stage on the 48-core shared build host
# (2026-08-18); load- and host-dependent, not invariants:
# it only has to clear the slowest single package. When this budget was set
# that was internal/handlers, measured in a cache-defeated builder stage on the
# 48-core shared build host (2026-08-18); load- and host-dependent, not
# invariants:
#
# 16.9s host load 5-20, GOMAXPROCS 48
# 45.9s / 47.3s / 49.0s three runs at deliberate host load 31-73
@@ -23,6 +24,11 @@
# a condition CI runs under. If a CPU-limited runner ever puts a real run near
# 67s, that is the datum to revisit the org figure with.
#
# Those figures predate tests hashing the admin password at 1 MB instead of
# 64 MB (https://git.eeqj.de/sneak/webhooker/pulls/404). After that change, in
# a cache-defeated build at host load 44-109 (2026-10-02), internal/handlers
# took 8.5s and the slowest package was internal/database at 15.8s.
#
# -p 4 -parallel 8 keep the run under 2 GB of memory: at most four test
# binaries build or run at once, each with at most eight parallel tests. Under
# -race every test binary and every link costs a few hundred MB, so the
+3
View File
@@ -157,6 +157,9 @@
{{else}}
<span class="badge-error">Inactive</span>
{{end}}
{{if eq .Type "database"}}
<a href="/hook/{{$.Webhook.ID}}/targets/{{.ID}}/download" class="text-xs text-gray-500 hover:text-primary-600" title="Download the archive as gzipped JSON">Download</a>
{{end}}
<a href="/hook/{{$.Webhook.ID}}/targets/{{.ID}}/edit" class="text-xs text-gray-500 hover:text-primary-600" title="Edit">Edit</a>
<form method="POST" action="/hook/{{$.Webhook.ID}}/targets/{{.ID}}/toggle" class="inline">
<input type="hidden" name="csrf_token" value="{{$.CSRFToken}}">
+10 -4
View File
@@ -27,10 +27,16 @@
</div>
<span class="badge-info">Retention: {{.RetentionLabel}}</span>
</div>
<div class="flex gap-6 mt-4 text-sm text-gray-500">
<span>{{.EntrypointCount}} entrypoint{{if ne .EntrypointCount 1}}s{{end}}</span>
<span>{{.TargetCount}} target{{if ne .TargetCount 1}}s{{end}}</span>
<span>{{.EventCount}} event{{if ne .EventCount 1}}s{{end}}</span>
<div class="flex flex-wrap gap-6 mt-4 text-sm text-gray-500">
<span>{{.EntrypointCount}} entrypoint{{if ne .EntrypointCount 1}}s{{end}}{{if .InactiveEntrypointCount}}, {{.InactiveEntrypointCount}} inactive{{end}}</span>
<span>{{.TargetCount}} target{{if ne .TargetCount 1}}s{{end}}{{if .InactiveTargetCount}}, {{.InactiveTargetCount}} inactive{{end}}</span>
{{if .EventsUnreadable}}
<span class="text-red-600">The event figures could not be read.</span>
{{else}}
<span>{{.EventCount}} event{{if ne .EventCount 1}}s{{end}} within retention</span>
<span>{{with .LastEventAt}}Last event {{.UTC.Format "2006-01-02 15:04:05 UTC"}}{{else}}No events yet{{end}}</span>
<span class="{{if .FailedLast24Hours}}font-medium text-red-600{{end}}">{{.FailedLast24Hours}} failed deliver{{if eq .FailedLast24Hours 1}}y{{else}}ies{{end}} in the last 24 hours</span>
{{end}}
</div>
</a>
{{end}}