Compare commits
4
Commits
e37fe7d350
...
db42815c4c
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
db42815c4c | ||
|
|
8cf5acaf1d | ||
|
|
9ade217222 | ||
|
|
5551f75251 |
No files matched your search
@@ -1786,7 +1786,7 @@ retries) is individually logged for full observability.
|
|||||||
#### EventTotals and TargetTotals
|
#### EventTotals and TargetTotals
|
||||||
|
|
||||||
Running counts in each event database, read by the statistics pane at the
|
Running counts in each event database, read by the statistics pane at the
|
||||||
top of the webhook page. `EventTotals` is one row:
|
top of the webhook page and by the webhook list. `EventTotals` is one row:
|
||||||
|
|
||||||
| Field | Type | Description |
|
| Field | Type | Description |
|
||||||
| ---------------- | --------- | ----------- |
|
| ---------------- | --------- | ----------- |
|
||||||
@@ -1818,6 +1818,14 @@ target. Its failure percentage for a window is the deliveries that became
|
|||||||
`failed` in it out of all that became `delivered` or `failed` in it, and
|
`failed` in it out of all that became `delivered` or `failed` in it, and
|
||||||
a dash when none did.
|
a dash when none did.
|
||||||
|
|
||||||
|
The webhook list at `/hooks` shows three of the pane's figures for each
|
||||||
|
webhook: its events within retention and its last event, both from
|
||||||
|
`EventTotals`, and its deliveries that failed in the last 24 hours,
|
||||||
|
counted with the pane's query. It opens each webhook's event database once
|
||||||
|
(the handle stays open) and runs those two reads there, so its cost grows
|
||||||
|
with the number of webhooks and, for each, with the deliveries that
|
||||||
|
finished in the last 24 hours, never with the events stored.
|
||||||
|
|
||||||
#### Event-tier indexes
|
#### Event-tier indexes
|
||||||
|
|
||||||
These indexes on the per-webhook event databases are declared in the model
|
These indexes on the per-webhook event databases are declared in the model
|
||||||
@@ -1825,7 +1833,7 @@ tags, so `AutoMigrate` creates them on a fresh database:
|
|||||||
|
|
||||||
| Table | Columns | Serves |
|
| Table | Columns | Serves |
|
||||||
| ------------------ | --------------------------- | ------ |
|
| ------------------ | --------------------------- | ------ |
|
||||||
| `deliveries` | `status`, `deleted_at`, `finished_at`, `target_id` | Startup recovery, the retry and pending sweeps every 60 seconds and the queue-depth sampler every 30 seconds, which select deliveries by status, and the webhook page's statistics, which count each target's deliveries by status and when they finished |
|
| `deliveries` | `status`, `deleted_at`, `finished_at`, `target_id` | Startup recovery, the retry and pending sweeps every 60 seconds and the queue-depth sampler every 30 seconds, which select deliveries by status, and the webhook page's statistics and the webhook list, which count each target's deliveries by status and when they finished |
|
||||||
| `deliveries` | `event_id`, `deleted_at` | The event log, which loads each event's deliveries, and retention, which counts and deletes the deliveries of expired events |
|
| `deliveries` | `event_id`, `deleted_at` | The event log, which loads each event's deliveries, and retention, which counts and deletes the deliveries of expired events |
|
||||||
| `delivery_results` | `delivery_id`, `deleted_at` | The event log, which loads the attempts of a page's deliveries, and retention, which deletes the attempts of expired events |
|
| `delivery_results` | `delivery_id`, `deleted_at` | The event log, which loads the attempts of a page's deliveries, and retention, which deletes the attempts of expired events |
|
||||||
| `events` | `deleted_at`, `created_at` | The webhook page's statistics, which count recent events |
|
| `events` | `deleted_at`, `created_at` | The webhook page's statistics, which count recent events |
|
||||||
@@ -1984,6 +1992,30 @@ Because each `database` target has its own archive file, a target's
|
|||||||
webhook with different expiries keep two archives, each pruned on its
|
webhook with different expiries keep two archives, each pruned on its
|
||||||
own schedule.
|
own schedule.
|
||||||
|
|
||||||
|
Each `database` target on the webhook page has a **Download** button,
|
||||||
|
which returns its archive as one gzipped JSON file,
|
||||||
|
`archive-{webhook_name}-{target_name}-{YYYYMMDDTHHMMSSZ}.json.gz`, the
|
||||||
|
names made safe as above and the time in UTC. The file holds one
|
||||||
|
object: `webhook` and `target`, each an `id` and a `name`;
|
||||||
|
`exported_at`; and `archived_events`, one object per archived row with
|
||||||
|
every column, keyed by column name. A body that is not valid UTF-8 is
|
||||||
|
written in base64, with `"body_encoding": "base64"` beside it. An
|
||||||
|
archive that does not exist yet, or was moved away, downloads with an
|
||||||
|
empty `archived_events`; the download never creates the file.
|
||||||
|
|
||||||
|
The download streams: each row is read and written out compressed
|
||||||
|
before the next is read, so neither the archive nor the JSON is held in
|
||||||
|
memory. It reads on a connection of its own, inside one read-only
|
||||||
|
transaction, so the file holds the archive as it stood when the
|
||||||
|
download started, and archive writes go on meanwhile, since under WAL a
|
||||||
|
reader never blocks a writer. While it runs, the `-wal` cannot be
|
||||||
|
checkpointed past what it reads, so a long download lets the `-wal`
|
||||||
|
grow. It finds the file by the stored names under the lock that webhook
|
||||||
|
edits, target edits and target creation hold, and lets go once the file
|
||||||
|
is open: a rename during the download moves the file without affecting
|
||||||
|
it. Like every request it is cut off after 60 seconds, which leaves the
|
||||||
|
file incomplete and failing to decompress.
|
||||||
|
|
||||||
Deleting a webhook releases its archives: the delivery engine's cached
|
Deleting a webhook releases its archives: the delivery engine's cached
|
||||||
archive writers are dropped and their file handles closed, so nothing
|
archive writers are dropped and their file handles closed, so nothing
|
||||||
lingers after the webhook is gone. The archive **files themselves are
|
lingers after the webhook is gone. The archive **files themselves are
|
||||||
@@ -2846,6 +2878,7 @@ returns to the page that was asked for.
|
|||||||
| `POST` | `/hook/{id}/targets` | Add target to webhook |
|
| `POST` | `/hook/{id}/targets` | Add target to webhook |
|
||||||
| `GET` | `/hook/{id}/targets/{targetID}/edit` | Edit target form. The one page that renders a target's destination URL and header values in full, rather than masked |
|
| `GET` | `/hook/{id}/targets/{targetID}/edit` | Edit target form. The one page that renders a target's destination URL and header values in full, rather than masked |
|
||||||
| `POST` | `/hook/{id}/targets/{targetID}/edit` | Edit target submission |
|
| `POST` | `/hook/{id}/targets/{targetID}/edit` | Edit target submission |
|
||||||
|
| `GET` | `/hook/{id}/targets/{targetID}/download` | Download a `database` target's archive as one gzipped JSON file. See [Database Architecture](#database-architecture) |
|
||||||
| `POST` | `/hook/{id}/targets/{targetID}/delete` | Delete a target |
|
| `POST` | `/hook/{id}/targets/{targetID}/delete` | Delete a target |
|
||||||
| `POST` | `/hook/{id}/targets/{targetID}/toggle` | Enable or disable a target |
|
| `POST` | `/hook/{id}/targets/{targetID}/toggle` | Enable or disable a target |
|
||||||
|
|
||||||
@@ -2926,6 +2959,7 @@ webhooker/
|
|||||||
│ │ ├── target_slack.go # Slack/Mattermost incoming-webhook target
|
│ │ ├── target_slack.go # Slack/Mattermost incoming-webhook target
|
||||||
│ │ ├── target_database.go # Database archive target
|
│ │ ├── target_database.go # Database archive target
|
||||||
│ │ ├── target_database_archive.go # Archive file lifecycle and pruning
|
│ │ ├── target_database_archive.go # Archive file lifecycle and pruning
|
||||||
|
│ │ ├── target_database_export.go # Archive download as gzipped JSON
|
||||||
│ │ ├── target_log.go # Log target (stdout)
|
│ │ ├── target_log.go # Log target (stdout)
|
||||||
│ │ ├── target_config_view.go # Masked target config for templates
|
│ │ ├── target_config_view.go # Masked target config for templates
|
||||||
│ │ ├── archive_sweeper.go # Periodic pruning of idle archives
|
│ │ ├── archive_sweeper.go # Periodic pruning of idle archives
|
||||||
|
|||||||
@@ -3,7 +3,6 @@ package delivery
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
"path/filepath"
|
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
@@ -277,10 +276,9 @@ func (t *databaseTarget) releaseSweepWriter(
|
|||||||
}
|
}
|
||||||
|
|
||||||
// newWriter builds the writer for a database target's archive. The
|
// newWriter builds the writer for a database target's archive. The
|
||||||
// file lives beside the webhook's event database in the data
|
// file is the one ArchivePath gives for the webhook and the target as
|
||||||
// directory and is named for the webhook and the target as the main
|
// the main database names them now; from then on only rename changes
|
||||||
// database has them now; from then on only rename changes the name
|
// the name the writer uses. It does not touch the archive file.
|
||||||
// the writer uses. It does not touch the archive file.
|
|
||||||
func (t *databaseTarget) newWriter(
|
func (t *databaseTarget) newWriter(
|
||||||
targetID string,
|
targetID string,
|
||||||
) (*archiveWriter, error) {
|
) (*archiveWriter, error) {
|
||||||
@@ -299,12 +297,10 @@ func (t *databaseTarget) newWriter(
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
dir := filepath.Dir(t.eng.dbManager.DBPath(target.WebhookID))
|
w := newArchiveWriter(
|
||||||
name := ArchiveFileName(
|
ArchivePath(t.eng.dbManager, &target.Webhook, &target),
|
||||||
target.Webhook.Name, target.Name, target.ID,
|
t.eng.log,
|
||||||
)
|
)
|
||||||
|
|
||||||
w := newArchiveWriter(filepath.Join(dir, name), t.eng.log)
|
|
||||||
w.webhookID = target.WebhookID
|
w.webhookID = target.WebhookID
|
||||||
|
|
||||||
return w, nil
|
return w, nil
|
||||||
|
|||||||
@@ -0,0 +1,270 @@
|
|||||||
|
package delivery
|
||||||
|
|
||||||
|
import (
|
||||||
|
"compress/gzip"
|
||||||
|
"context"
|
||||||
|
"database/sql"
|
||||||
|
"encoding/base64"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"path/filepath"
|
||||||
|
"time"
|
||||||
|
"unicode/utf8"
|
||||||
|
|
||||||
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
|
)
|
||||||
|
|
||||||
|
// archiveTableQuery counts the archive's table: 0 when the archive
|
||||||
|
// writer has created the file but not yet the table in it.
|
||||||
|
const archiveTableQuery = "SELECT count(*) FROM sqlite_master " +
|
||||||
|
"WHERE type = 'table' AND name = 'archived_events'"
|
||||||
|
|
||||||
|
// archiveNextRowQuery reads every column of the first archived row
|
||||||
|
// after a given id. An export reads the archive a row at a time this
|
||||||
|
// way rather than through one cursor, because the Scan check in
|
||||||
|
// internal/gormlog accepts only a Scan straight on QueryRowContext's
|
||||||
|
// result.
|
||||||
|
const archiveNextRowQuery = "SELECT id, event_id, webhook_id, " +
|
||||||
|
"entrypoint_id, method, headers, body, content_type, archived_at " +
|
||||||
|
"FROM archived_events WHERE id > ? ORDER BY id LIMIT 1"
|
||||||
|
|
||||||
|
// ArchivePath returns where a database target's archive file is: in
|
||||||
|
// the data directory, beside the webhook's event database, under the
|
||||||
|
// name ArchiveFileName gives it.
|
||||||
|
func ArchivePath(
|
||||||
|
dbMgr *database.WebhookDBManager,
|
||||||
|
webhook *database.Webhook,
|
||||||
|
target *database.Target,
|
||||||
|
) string {
|
||||||
|
return filepath.Join(
|
||||||
|
filepath.Dir(dbMgr.DBPath(webhook.ID)),
|
||||||
|
ArchiveFileName(webhook.Name, target.Name, target.ID),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ArchiveExportFileName returns the name a database target's archive
|
||||||
|
// downloads under:
|
||||||
|
// archive-WEBHOOKNAME-TARGETNAME-YYYYMMDDTHHMMSSZ.json.gz, the names
|
||||||
|
// made safe as in ArchiveFileName and the time in UTC.
|
||||||
|
func ArchiveExportFileName(
|
||||||
|
webhookName, targetName string, at time.Time,
|
||||||
|
) string {
|
||||||
|
return "archive-" + archiveNamePart(webhookName) + "-" +
|
||||||
|
archiveNamePart(targetName) + "-" +
|
||||||
|
at.UTC().Format("20060102T150405Z") + ".json.gz"
|
||||||
|
}
|
||||||
|
|
||||||
|
// ArchiveExport is a database target's archive opened for download.
|
||||||
|
// It reads the file on its own connection, inside one read-only
|
||||||
|
// transaction, so it writes out the archive as it stood when
|
||||||
|
// OpenArchiveExport returned.
|
||||||
|
//
|
||||||
|
// Archives are in WAL mode, where a reader works from a snapshot and
|
||||||
|
// never blocks a writer: archive writes go on while an export is open,
|
||||||
|
// and the export does not see them. SQLite cannot checkpoint the -wal
|
||||||
|
// past an open snapshot, so the -wal grows until the export is closed.
|
||||||
|
type ArchiveExport struct {
|
||||||
|
db *sql.DB
|
||||||
|
tx *sql.Tx
|
||||||
|
|
||||||
|
// empty is true when there is nothing to read: no file, or a file
|
||||||
|
// without the archive's table yet.
|
||||||
|
empty bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// exportedName is how an export names its webhook and its target.
|
||||||
|
type exportedName struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
Name string `json:"name"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// OpenArchiveExport opens the archive file at path for export and
|
||||||
|
// takes the snapshot the export reads. It never creates the file: with
|
||||||
|
// no file at path, the export has no rows.
|
||||||
|
//
|
||||||
|
// Once it has returned, the file is open, so a rename or a move of it
|
||||||
|
// does not affect the export, which reads the same file under its new
|
||||||
|
// name.
|
||||||
|
func OpenArchiveExport(
|
||||||
|
ctx context.Context, path string,
|
||||||
|
) (*ArchiveExport, error) {
|
||||||
|
if !fileExists(path) {
|
||||||
|
return &ArchiveExport{empty: true}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
db, err := database.OpenSQLite(path, archiveModeExisting)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("opening archive %s: %w", path, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ReadOnly makes the driver begin a deferred transaction in place
|
||||||
|
// of the BEGIN IMMEDIATE the connection string asks for, so the
|
||||||
|
// export never takes the archive's write lock.
|
||||||
|
tx, err := db.BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
|
||||||
|
if err != nil {
|
||||||
|
_ = db.Close()
|
||||||
|
|
||||||
|
return nil, fmt.Errorf("reading archive %s: %w", path, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The transaction's first read is what takes the snapshot.
|
||||||
|
var tables int
|
||||||
|
|
||||||
|
err = tx.QueryRowContext(ctx, archiveTableQuery).Scan(&tables)
|
||||||
|
if err != nil {
|
||||||
|
_ = tx.Rollback()
|
||||||
|
_ = db.Close()
|
||||||
|
|
||||||
|
return nil, fmt.Errorf("reading archive %s: %w", path, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return &ArchiveExport{db: db, tx: tx, empty: tables == 0}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// WriteGzipJSON writes the export to w as one gzipped JSON object:
|
||||||
|
// webhook and target, each an id and a name; exported_at; and
|
||||||
|
// archived_events, one object per archived row, keyed by column name.
|
||||||
|
// A body that is not valid UTF-8 cannot be a JSON string, so it is
|
||||||
|
// written in base64, with "body_encoding": "base64" beside it.
|
||||||
|
//
|
||||||
|
// Each row is written out before the next is read, so neither the
|
||||||
|
// archive nor its JSON is ever held in memory whole. After an error
|
||||||
|
// the gzip stream is left unfinished, so what was written does not
|
||||||
|
// decompress as a whole file.
|
||||||
|
func (x *ArchiveExport) WriteGzipJSON(
|
||||||
|
ctx context.Context,
|
||||||
|
w io.Writer,
|
||||||
|
webhook *database.Webhook,
|
||||||
|
target *database.Target,
|
||||||
|
exportedAt time.Time,
|
||||||
|
) error {
|
||||||
|
head, err := json.Marshal(map[string]any{
|
||||||
|
"webhook": exportedName{ID: webhook.ID, Name: webhook.Name},
|
||||||
|
"target": exportedName{ID: target.ID, Name: target.Name},
|
||||||
|
"exported_at": exportedAt.UTC(),
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("encoding archive export: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
zw := gzip.NewWriter(w)
|
||||||
|
|
||||||
|
err = x.writeJSON(ctx, zw, head)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("writing archive export: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return zw.Close()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Close ends the export's transaction and closes its connection.
|
||||||
|
func (x *ArchiveExport) Close() error {
|
||||||
|
if x.db == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
_ = x.tx.Rollback()
|
||||||
|
|
||||||
|
return x.db.Close()
|
||||||
|
}
|
||||||
|
|
||||||
|
// writeJSON writes head with archived_events added as its last key,
|
||||||
|
// the rows going into it one at a time.
|
||||||
|
func (x *ArchiveExport) writeJSON(
|
||||||
|
ctx context.Context, w io.Writer, head []byte,
|
||||||
|
) error {
|
||||||
|
// head goes out without its closing brace, so that
|
||||||
|
// archived_events can follow it.
|
||||||
|
_, err := w.Write(head[:len(head)-1])
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = io.WriteString(w, `,"archived_events":[`)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
err = x.writeRows(ctx, w)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = io.WriteString(w, "\n]}\n")
|
||||||
|
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
// writeRows writes each archived row to w, oldest first, one per line,
|
||||||
|
// separated by commas.
|
||||||
|
func (x *ArchiveExport) writeRows(ctx context.Context, w io.Writer) error {
|
||||||
|
if x.empty {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
next, err := x.tx.PrepareContext(ctx, archiveNextRowQuery)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
defer func() { _ = next.Close() }()
|
||||||
|
|
||||||
|
var ev archivedEvent
|
||||||
|
|
||||||
|
for sep := "\n"; ; sep = ",\n" {
|
||||||
|
err = next.QueryRowContext(ctx, ev.ID).Scan(
|
||||||
|
&ev.ID, &ev.EventID, &ev.WebhookID, &ev.EntrypointID,
|
||||||
|
&ev.Method, &ev.Headers, &ev.Body, &ev.ContentType,
|
||||||
|
&ev.ArchivedAt,
|
||||||
|
)
|
||||||
|
if errors.Is(err, sql.ErrNoRows) {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = io.WriteString(w, sep)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
err = writeRow(w, &ev)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// writeRow writes an archived row to w as a JSON object keyed by
|
||||||
|
// column name, its body in base64 when it is not valid UTF-8.
|
||||||
|
func writeRow(w io.Writer, ev *archivedEvent) error {
|
||||||
|
row := map[string]any{
|
||||||
|
"id": ev.ID,
|
||||||
|
"event_id": ev.EventID,
|
||||||
|
"webhook_id": ev.WebhookID,
|
||||||
|
"entrypoint_id": ev.EntrypointID,
|
||||||
|
"method": ev.Method,
|
||||||
|
"headers": ev.Headers,
|
||||||
|
"body": ev.Body,
|
||||||
|
"content_type": ev.ContentType,
|
||||||
|
"archived_at": ev.ArchivedAt.UTC(),
|
||||||
|
}
|
||||||
|
|
||||||
|
if !utf8.ValidString(ev.Body) {
|
||||||
|
row["body"] = base64.StdEncoding.EncodeToString([]byte(ev.Body))
|
||||||
|
row["body_encoding"] = "base64"
|
||||||
|
}
|
||||||
|
|
||||||
|
line, err := json.Marshal(row)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = w.Write(line)
|
||||||
|
|
||||||
|
return err
|
||||||
|
}
|
||||||
@@ -0,0 +1,373 @@
|
|||||||
|
package delivery_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"compress/gzip"
|
||||||
|
"encoding/base64"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"runtime"
|
||||||
|
"runtime/debug"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
|
"sneak.berlin/go/webhooker/internal/delivery"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The webhook and the target the export tests' archives belong to.
|
||||||
|
const (
|
||||||
|
exportWebhookID = "wh-export"
|
||||||
|
exportWebhookName = "Orders (EU)"
|
||||||
|
exportTargetID = "tgt-export"
|
||||||
|
exportTargetName = "Long-term archive"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
// binaryBody is a body that is not valid UTF-8.
|
||||||
|
binaryBody = "\xff\xfe\x00\x01binary\x80"
|
||||||
|
|
||||||
|
// openedEventID is the event the snapshot tests archive before
|
||||||
|
// they open the export.
|
||||||
|
openedEventID = "opened"
|
||||||
|
)
|
||||||
|
|
||||||
|
// writeExportTo writes export to w as the archive of the export tests'
|
||||||
|
// webhook and target, exported at 2026-10-02T12:03:04Z.
|
||||||
|
func writeExportTo(
|
||||||
|
t *testing.T, export *delivery.ArchiveExport, w io.Writer,
|
||||||
|
) error {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
return export.WriteGzipJSON(
|
||||||
|
t.Context(), w,
|
||||||
|
&database.Webhook{
|
||||||
|
BaseModel: database.BaseModel{ID: exportWebhookID},
|
||||||
|
Name: exportWebhookName,
|
||||||
|
},
|
||||||
|
&database.Target{
|
||||||
|
BaseModel: database.BaseModel{ID: exportTargetID},
|
||||||
|
Name: exportTargetName,
|
||||||
|
},
|
||||||
|
time.Date(2026, 10, 2, 12, 3, 4, 0, time.UTC),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// exportArchive runs a whole export of the archive at path and returns
|
||||||
|
// its JSON, decompressed and parsed.
|
||||||
|
func exportArchive(t *testing.T, path string) map[string]any {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
export, err := delivery.OpenArchiveExport(t.Context(), path)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
defer func() { require.NoError(t, export.Close()) }()
|
||||||
|
|
||||||
|
return writeExport(t, export)
|
||||||
|
}
|
||||||
|
|
||||||
|
// writeExport writes an opened export and returns its JSON,
|
||||||
|
// decompressed and parsed. Reading to the end makes the gzip reader
|
||||||
|
// check that the stream was finished.
|
||||||
|
func writeExport(
|
||||||
|
t *testing.T, export *delivery.ArchiveExport,
|
||||||
|
) map[string]any {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
var buf bytes.Buffer
|
||||||
|
|
||||||
|
require.NoError(t, writeExportTo(t, export, &buf))
|
||||||
|
|
||||||
|
zr, err := gzip.NewReader(&buf)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
raw, err := io.ReadAll(zr)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
var got map[string]any
|
||||||
|
|
||||||
|
require.NoError(t, json.Unmarshal(raw, &got))
|
||||||
|
|
||||||
|
return got
|
||||||
|
}
|
||||||
|
|
||||||
|
// exportedEvents returns an export's archived_events.
|
||||||
|
func exportedEvents(t *testing.T, got map[string]any) []map[string]any {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
list, ok := got["archived_events"].([]any)
|
||||||
|
require.True(t, ok, "archived_events must be an array: %v", got)
|
||||||
|
|
||||||
|
events := make([]map[string]any, len(list))
|
||||||
|
|
||||||
|
for i, v := range list {
|
||||||
|
events[i], ok = v.(map[string]any)
|
||||||
|
require.True(t, ok, "an archived event must be an object: %v", v)
|
||||||
|
}
|
||||||
|
|
||||||
|
return events
|
||||||
|
}
|
||||||
|
|
||||||
|
// exportedEventIDs returns the event_id of each of an export's
|
||||||
|
// archived_events.
|
||||||
|
func exportedEventIDs(t *testing.T, got map[string]any) []string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
events := exportedEvents(t, got)
|
||||||
|
ids := make([]string, 0, len(events))
|
||||||
|
|
||||||
|
for _, ev := range events {
|
||||||
|
ids = append(ids, fmt.Sprint(ev["event_id"]))
|
||||||
|
}
|
||||||
|
|
||||||
|
return ids
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestArchiveExport_MatchesStoredRows proves an export holds the
|
||||||
|
// webhook, the target, the time, and every column of every stored
|
||||||
|
// row: a body that is valid UTF-8 as a string, and one that is not in
|
||||||
|
// base64, marked as such.
|
||||||
|
func TestArchiveExport_MatchesStoredRows(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
path := filepath.Join(t.TempDir(), "archive.db")
|
||||||
|
w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
|
||||||
|
bodies := []string{`{"order":1}`, "plain text", "", binaryBody}
|
||||||
|
|
||||||
|
for i, body := range bodies {
|
||||||
|
require.NoError(t, w.Write(delivery.ExportArchivedEvent{
|
||||||
|
EventID: fmt.Sprintf("ev-%d", i),
|
||||||
|
WebhookID: exportWebhookID,
|
||||||
|
EntrypointID: "ep-1",
|
||||||
|
Method: "POST",
|
||||||
|
Headers: `{"X-Test":["yes"]}`,
|
||||||
|
Body: body,
|
||||||
|
ContentType: testContentType,
|
||||||
|
}, 0))
|
||||||
|
}
|
||||||
|
|
||||||
|
var stored []delivery.ExportArchivedEvent
|
||||||
|
|
||||||
|
require.NoError(t, openArchiveDBForRead(t, path).
|
||||||
|
Order("id").Find(&stored).Error)
|
||||||
|
|
||||||
|
got := exportArchive(t, path)
|
||||||
|
|
||||||
|
assert.Equal(t,
|
||||||
|
map[string]any{"id": exportWebhookID, "name": exportWebhookName},
|
||||||
|
got["webhook"],
|
||||||
|
)
|
||||||
|
assert.Equal(t,
|
||||||
|
map[string]any{"id": exportTargetID, "name": exportTargetName},
|
||||||
|
got["target"],
|
||||||
|
)
|
||||||
|
assert.Equal(t, "2026-10-02T12:03:04Z", got["exported_at"])
|
||||||
|
|
||||||
|
events := exportedEvents(t, got)
|
||||||
|
require.Len(t, events, len(bodies))
|
||||||
|
|
||||||
|
for i, row := range stored {
|
||||||
|
assertExportedRow(t, row, events[i])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// assertExportedRow checks that ev, from an export, holds every column
|
||||||
|
// of the stored row.
|
||||||
|
func assertExportedRow(
|
||||||
|
t *testing.T, row delivery.ExportArchivedEvent, ev map[string]any,
|
||||||
|
) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
archivedAt, err := time.Parse(
|
||||||
|
time.RFC3339Nano, fmt.Sprint(ev["archived_at"]),
|
||||||
|
)
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.True(t, archivedAt.Equal(row.ArchivedAt))
|
||||||
|
|
||||||
|
assert.EqualValues(t, row.ID, ev["id"])
|
||||||
|
assert.Equal(t, row.EventID, ev["event_id"])
|
||||||
|
assert.Equal(t, row.WebhookID, ev["webhook_id"])
|
||||||
|
assert.Equal(t, row.EntrypointID, ev["entrypoint_id"])
|
||||||
|
assert.Equal(t, row.Method, ev["method"])
|
||||||
|
assert.Equal(t, row.Headers, ev["headers"])
|
||||||
|
assert.Equal(t, row.ContentType, ev["content_type"])
|
||||||
|
|
||||||
|
if row.Body != binaryBody {
|
||||||
|
assert.Equal(t, row.Body, ev["body"])
|
||||||
|
assert.Len(t, ev, 9, "the nine columns and nothing else: %v", ev)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
body, err := base64.StdEncoding.DecodeString(fmt.Sprint(ev["body"]))
|
||||||
|
require.NoError(t, err)
|
||||||
|
assert.Equal(t, binaryBody, string(body))
|
||||||
|
assert.Equal(t, "base64", ev["body_encoding"])
|
||||||
|
assert.Len(t, ev, 10, "the nine columns and body_encoding: %v", ev)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestArchiveExport_Empty proves an archive with nothing in it exports
|
||||||
|
// as an empty archived_events: no file, which the export must not
|
||||||
|
// create; a file the archive writer has not yet put its table in; and
|
||||||
|
// a table with no rows.
|
||||||
|
func TestArchiveExport_Empty(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
missing := filepath.Join(dir, "missing.db")
|
||||||
|
noTable := filepath.Join(dir, "no-table.db")
|
||||||
|
noRows := filepath.Join(dir, "no-rows.db")
|
||||||
|
|
||||||
|
require.NoError(t, os.WriteFile(noTable, nil, 0o600))
|
||||||
|
require.NoError(t,
|
||||||
|
delivery.NewExportArchiveWriter(noRows, archiveTestLogger(), 0).
|
||||||
|
Open(0),
|
||||||
|
)
|
||||||
|
|
||||||
|
for _, path := range []string{missing, noTable, noRows} {
|
||||||
|
assert.Empty(t, exportedEvents(t, exportArchive(t, path)), path)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, suffix := range archiveFileSuffixes() {
|
||||||
|
assert.NoFileExists(t, missing+suffix)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestArchiveExport_ReadsOneSnapshot proves an export writes the
|
||||||
|
// archive as it was when it was opened, and holds up no archive
|
||||||
|
// write: a row written while the export is open is stored, and is not
|
||||||
|
// in the export. A write held up for the whole busy timeout would
|
||||||
|
// fail.
|
||||||
|
func TestArchiveExport_ReadsOneSnapshot(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
path := filepath.Join(t.TempDir(), "archive.db")
|
||||||
|
w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
|
||||||
|
|
||||||
|
require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: openedEventID}, 0))
|
||||||
|
|
||||||
|
export, err := delivery.OpenArchiveExport(t.Context(), path)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
defer func() { require.NoError(t, export.Close()) }()
|
||||||
|
|
||||||
|
require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: "during"}, 0))
|
||||||
|
|
||||||
|
assert.Equal(t,
|
||||||
|
[]string{openedEventID}, exportedEventIDs(t, writeExport(t, export)),
|
||||||
|
)
|
||||||
|
|
||||||
|
var stored int64
|
||||||
|
|
||||||
|
require.NoError(t, openArchiveDBForRead(t, path).
|
||||||
|
Model(&delivery.ExportArchivedEvent{}).Count(&stored).Error)
|
||||||
|
assert.Equal(t, int64(2), stored)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestArchiveExport_SurvivesRename proves that renaming the archive
|
||||||
|
// while an export of it is open, as renaming its webhook or target
|
||||||
|
// does, leaves the export reading the same file.
|
||||||
|
func TestArchiveExport_SurvivesRename(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
path := filepath.Join(t.TempDir(), "archive-old.db")
|
||||||
|
w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
|
||||||
|
|
||||||
|
require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: openedEventID}, 0))
|
||||||
|
|
||||||
|
export, err := delivery.OpenArchiveExport(t.Context(), path)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
defer func() { require.NoError(t, export.Close()) }()
|
||||||
|
|
||||||
|
require.NoError(t, w.Rename("archive-new.db"))
|
||||||
|
require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: "after"}, 0))
|
||||||
|
require.NoFileExists(t, path)
|
||||||
|
|
||||||
|
assert.Equal(t,
|
||||||
|
[]string{openedEventID}, exportedEventIDs(t, writeExport(t, export)),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// heapPeak is an io.Writer that discards what it is given and records
|
||||||
|
// the largest heap it saw at a write.
|
||||||
|
type heapPeak struct {
|
||||||
|
max uint64
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *heapPeak) Write(b []byte) (int, error) {
|
||||||
|
var m runtime.MemStats
|
||||||
|
|
||||||
|
runtime.ReadMemStats(&m)
|
||||||
|
p.max = max(p.max, m.HeapAlloc)
|
||||||
|
|
||||||
|
return len(b), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestArchiveExport_Streams proves an export holds neither the archive
|
||||||
|
// nor its JSON in memory whole: exporting a 16 MiB archive grows the
|
||||||
|
// heap by less than half of that. An export that read every row before
|
||||||
|
// writing, or built the JSON before writing it, would hold all 16 MiB
|
||||||
|
// at a write.
|
||||||
|
//
|
||||||
|
//nolint:paralleltest // It measures the heap, which tests share.
|
||||||
|
func TestArchiveExport_Streams(t *testing.T) {
|
||||||
|
const (
|
||||||
|
rows = 64
|
||||||
|
bodySize = 256 << 10
|
||||||
|
limit = rows * bodySize / 2
|
||||||
|
)
|
||||||
|
|
||||||
|
path := filepath.Join(t.TempDir(), "archive.db")
|
||||||
|
w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
|
||||||
|
body := strings.Repeat("x", bodySize)
|
||||||
|
|
||||||
|
for range rows {
|
||||||
|
require.NoError(t, w.Write(delivery.ExportArchivedEvent{Body: body}, 0))
|
||||||
|
}
|
||||||
|
|
||||||
|
export, err := delivery.OpenArchiveExport(t.Context(), path)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
defer func() { require.NoError(t, export.Close()) }()
|
||||||
|
|
||||||
|
// A low GC target collects garbage soon after it is made, so the
|
||||||
|
// heap at each write is close to what the export is holding.
|
||||||
|
defer debug.SetGCPercent(debug.SetGCPercent(10))
|
||||||
|
|
||||||
|
runtime.GC()
|
||||||
|
|
||||||
|
var start runtime.MemStats
|
||||||
|
|
||||||
|
runtime.ReadMemStats(&start)
|
||||||
|
|
||||||
|
peak := &heapPeak{}
|
||||||
|
|
||||||
|
require.NoError(t, writeExportTo(t, export, peak))
|
||||||
|
assert.Less(t, peak.max, start.HeapAlloc+limit,
|
||||||
|
"heap at the start %d, at its peak %d", start.HeapAlloc, peak.max,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestArchiveExportFileName proves the download is named for the
|
||||||
|
// webhook and the target, with the names made safe as for the archive
|
||||||
|
// file, and the export time in UTC.
|
||||||
|
func TestArchiveExportFileName(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
cest := time.FixedZone("CEST", int((2 * time.Hour).Seconds()))
|
||||||
|
|
||||||
|
assert.Equal(t,
|
||||||
|
"archive-orders-eu-long-term-archive-20261002T120304Z.json.gz",
|
||||||
|
delivery.ArchiveExportFileName(
|
||||||
|
exportWebhookName, exportTargetName,
|
||||||
|
time.Date(2026, 10, 2, 14, 3, 4, 0, cest),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
}
|
||||||
@@ -97,7 +97,8 @@ type Handlers struct {
|
|||||||
// names through the archive rename, the save and any move back.
|
// names through the archive rename, the save and any move back.
|
||||||
// Interleaved, one could rename an archive between another's
|
// Interleaved, one could rename an archive between another's
|
||||||
// rename and save, leaving the file named for one edit and the
|
// rename and save, leaving the file named for one edit and the
|
||||||
// stored names from the other.
|
// stored names from the other. An archive download holds it while
|
||||||
|
// it reads the stored names and opens the file they give.
|
||||||
renameMu sync.Mutex
|
renameMu sync.Mutex
|
||||||
|
|
||||||
// dummyVerifications counts the equivalent-cost verifications
|
// dummyVerifications counts the equivalent-cost verifications
|
||||||
|
|||||||
@@ -181,6 +181,11 @@ func (r *recordingArchives) Renames() []archiveRename {
|
|||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// newTestApp returns an app whose RequireStart fails the test when
|
||||||
|
// starting takes longer than fx's default start timeout of 15s. That
|
||||||
|
// limit catches a start that hangs, not a busy host: measured with make
|
||||||
|
// test on 2026-10-02 at host load 58-69 on 48 cores, the slowest of this
|
||||||
|
// package's starts took 0.49s.
|
||||||
func newTestApp(
|
func newTestApp(
|
||||||
t *testing.T,
|
t *testing.T,
|
||||||
targets ...any,
|
targets ...any,
|
||||||
|
|||||||
@@ -0,0 +1,467 @@
|
|||||||
|
package handlers_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"regexp"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"gorm.io/gorm"
|
||||||
|
"gorm.io/gorm/clause"
|
||||||
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
|
"sneak.berlin/go/webhooker/internal/handlers"
|
||||||
|
"sneak.berlin/go/webhooker/internal/logger"
|
||||||
|
"sneak.berlin/go/webhooker/internal/session"
|
||||||
|
)
|
||||||
|
|
||||||
|
// failedHighlight is how the list marks a number of failed deliveries
|
||||||
|
// that is not zero.
|
||||||
|
const failedHighlight = `class="font-medium text-red-600"`
|
||||||
|
|
||||||
|
// listWebhook adds a webhook with the given name, owned by the test
|
||||||
|
// user.
|
||||||
|
func listWebhook(
|
||||||
|
t *testing.T, db *database.Database, name string,
|
||||||
|
) *database.Webhook {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
wh := &database.Webhook{UserID: deleteTestUserID, Name: name}
|
||||||
|
require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
|
||||||
|
|
||||||
|
return wh
|
||||||
|
}
|
||||||
|
|
||||||
|
// addEntrypoints adds the given number of entrypoints, all active or
|
||||||
|
// all inactive, to a webhook and returns their paths.
|
||||||
|
func addEntrypoints(
|
||||||
|
t *testing.T, db *database.Database, webhookID string,
|
||||||
|
count int, active bool,
|
||||||
|
) []string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
paths := make([]string, count)
|
||||||
|
for i := range paths {
|
||||||
|
paths[i] = statsEntrypoint(t, db, webhookID, active)
|
||||||
|
}
|
||||||
|
|
||||||
|
return paths
|
||||||
|
}
|
||||||
|
|
||||||
|
// addTargets adds the given number of targets, all active or all
|
||||||
|
// inactive, to a webhook and returns them.
|
||||||
|
func addTargets(
|
||||||
|
t *testing.T, db *database.Database, webhookID string,
|
||||||
|
count int, active bool,
|
||||||
|
) []*database.Target {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
targets := make([]*database.Target, count)
|
||||||
|
for i := range targets {
|
||||||
|
targets[i] = seedTarget(t, db, webhookID, database.TargetTypeLog)
|
||||||
|
require.NoError(t, db.DB().Model(targets[i]).
|
||||||
|
Update("active", active).Error)
|
||||||
|
}
|
||||||
|
|
||||||
|
return targets
|
||||||
|
}
|
||||||
|
|
||||||
|
// renderWebhookList runs the real webhook list handler as the test user
|
||||||
|
// and returns the rendered page.
|
||||||
|
func renderWebhookList(
|
||||||
|
t *testing.T, h *handlers.Handlers, sess *session.Session,
|
||||||
|
) string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
cookies := authenticatedCookies(
|
||||||
|
t, sess, deleteTestUserID, deleteTestUsername,
|
||||||
|
)
|
||||||
|
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
h.HandleSourceList().ServeHTTP(
|
||||||
|
w, getRequest(t, "/hooks", cookies, nil),
|
||||||
|
)
|
||||||
|
require.Equal(t, http.StatusOK, w.Code)
|
||||||
|
|
||||||
|
return w.Body.String()
|
||||||
|
}
|
||||||
|
|
||||||
|
// listCard returns one webhook's entry in a rendered webhook list, its
|
||||||
|
// markup as rendered and its text with the markup taken out and each
|
||||||
|
// run of space made one space.
|
||||||
|
func listCard(t *testing.T, page, webhookID string) (string, string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
_, card, found := strings.Cut(page, `href="/hook/`+webhookID+`"`)
|
||||||
|
require.True(t, found, "the list has no entry for %s", webhookID)
|
||||||
|
|
||||||
|
card, _, _ = strings.Cut(card, "</a>")
|
||||||
|
text := regexp.MustCompile(`<[^>]*>`).ReplaceAllString(card, " ")
|
||||||
|
|
||||||
|
return card, strings.Join(strings.Fields(text), " ")
|
||||||
|
}
|
||||||
|
|
||||||
|
// receiveEvents posts the given number of events to an entrypoint
|
||||||
|
// through the real receiver, and returns the webhook's event database
|
||||||
|
// and its events, oldest first.
|
||||||
|
func receiveEvents(
|
||||||
|
t *testing.T,
|
||||||
|
h *handlers.Handlers,
|
||||||
|
dbMgr *database.WebhookDBManager,
|
||||||
|
webhookID, path string,
|
||||||
|
count int,
|
||||||
|
) (*gorm.DB, []database.Event) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
router := receiverRouter(h)
|
||||||
|
|
||||||
|
for range count {
|
||||||
|
require.Equal(t, http.StatusOK, postReceiver(t, router, path))
|
||||||
|
}
|
||||||
|
|
||||||
|
webhookDB, err := dbMgr.GetDB(webhookID)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
events := listEvents(t, webhookDB)
|
||||||
|
require.Len(t, events, count)
|
||||||
|
|
||||||
|
return webhookDB, events
|
||||||
|
}
|
||||||
|
|
||||||
|
// seedFailingWebhook adds a webhook with six entrypoints, two of them
|
||||||
|
// inactive, and seven targets, five of them inactive. Four events reach
|
||||||
|
// its two active targets, arriving 31, 5, 4 and 3 hours ago, and its
|
||||||
|
// event totals row records the last one. Three deliveries failed in the
|
||||||
|
// last 24 hours, two to the first target and one to the second, one
|
||||||
|
// failed 30 hours ago, two were delivered, and two are still pending.
|
||||||
|
// It returns the webhook and when its last event arrived.
|
||||||
|
func seedFailingWebhook(
|
||||||
|
t *testing.T,
|
||||||
|
h *handlers.Handlers,
|
||||||
|
db *database.Database,
|
||||||
|
dbMgr *database.WebhookDBManager,
|
||||||
|
) (*database.Webhook, time.Time) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
wh := listWebhook(t, db, "failing")
|
||||||
|
paths := addEntrypoints(t, db, wh.ID, 4, true)
|
||||||
|
addEntrypoints(t, db, wh.ID, 2, false)
|
||||||
|
|
||||||
|
active := addTargets(t, db, wh.ID, 2, true)
|
||||||
|
first, second := active[0], active[1]
|
||||||
|
|
||||||
|
addTargets(t, db, wh.ID, 5, false)
|
||||||
|
|
||||||
|
webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, paths[0], 4)
|
||||||
|
now := time.Now()
|
||||||
|
lastEventAt := now.Add(-3 * time.Hour)
|
||||||
|
|
||||||
|
statsAge(t, webhookDB, events[0].ID, now.Add(-31*time.Hour))
|
||||||
|
statsAge(t, webhookDB, events[1].ID, now.Add(-5*time.Hour))
|
||||||
|
statsAge(t, webhookDB, events[2].ID, now.Add(-4*time.Hour))
|
||||||
|
statsAge(t, webhookDB, events[3].ID, lastEventAt)
|
||||||
|
require.NoError(t, database.AddEventTotals(webhookDB,
|
||||||
|
database.EventTotals{LastEventAt: &lastEventAt}))
|
||||||
|
|
||||||
|
statsFinish(t, webhookDB,
|
||||||
|
statsDelivery(t, webhookDB, events[0].ID, first.ID),
|
||||||
|
database.DeliveryStatusFailed, now.Add(-30*time.Hour))
|
||||||
|
statsFinish(t, webhookDB,
|
||||||
|
statsDelivery(t, webhookDB, events[0].ID, second.ID),
|
||||||
|
database.DeliveryStatusDelivered, now.Add(-30*time.Hour))
|
||||||
|
statsFinish(t, webhookDB,
|
||||||
|
statsDelivery(t, webhookDB, events[1].ID, first.ID),
|
||||||
|
database.DeliveryStatusFailed, now.Add(-time.Hour))
|
||||||
|
statsFinish(t, webhookDB,
|
||||||
|
statsDelivery(t, webhookDB, events[2].ID, first.ID),
|
||||||
|
database.DeliveryStatusFailed, now.Add(-time.Minute))
|
||||||
|
statsFinish(t, webhookDB,
|
||||||
|
statsDelivery(t, webhookDB, events[2].ID, second.ID),
|
||||||
|
database.DeliveryStatusFailed, now.Add(-time.Minute))
|
||||||
|
statsFinish(t, webhookDB,
|
||||||
|
statsDelivery(t, webhookDB, events[3].ID, second.ID),
|
||||||
|
database.DeliveryStatusDelivered, now.Add(-time.Minute))
|
||||||
|
|
||||||
|
return wh, lastEventAt
|
||||||
|
}
|
||||||
|
|
||||||
|
// seedHealthyWebhook adds a webhook with four entrypoints and two
|
||||||
|
// targets, all active, and three events, arriving 8, 7 and 6 hours ago
|
||||||
|
// and each delivered to both targets. Its event totals row records the
|
||||||
|
// last event. It returns the webhook and when its last event arrived.
|
||||||
|
func seedHealthyWebhook(
|
||||||
|
t *testing.T,
|
||||||
|
h *handlers.Handlers,
|
||||||
|
db *database.Database,
|
||||||
|
dbMgr *database.WebhookDBManager,
|
||||||
|
) (*database.Webhook, time.Time) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
wh := listWebhook(t, db, "healthy")
|
||||||
|
paths := addEntrypoints(t, db, wh.ID, 4, true)
|
||||||
|
targets := addTargets(t, db, wh.ID, 2, true)
|
||||||
|
|
||||||
|
webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, paths[0], 3)
|
||||||
|
now := time.Now()
|
||||||
|
lastEventAt := now.Add(-6 * time.Hour)
|
||||||
|
|
||||||
|
statsAge(t, webhookDB, events[0].ID, now.Add(-8*time.Hour))
|
||||||
|
statsAge(t, webhookDB, events[1].ID, now.Add(-7*time.Hour))
|
||||||
|
statsAge(t, webhookDB, events[2].ID, lastEventAt)
|
||||||
|
require.NoError(t, database.AddEventTotals(webhookDB,
|
||||||
|
database.EventTotals{LastEventAt: &lastEventAt}))
|
||||||
|
|
||||||
|
for _, ev := range events {
|
||||||
|
for _, target := range targets {
|
||||||
|
statsFinish(t, webhookDB,
|
||||||
|
statsDelivery(t, webhookDB, ev.ID, target.ID),
|
||||||
|
database.DeliveryStatusDelivered, now)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return wh, lastEventAt
|
||||||
|
}
|
||||||
|
|
||||||
|
// lastEventText is how the list shows when the last event arrived.
|
||||||
|
func lastEventText(at time.Time) string {
|
||||||
|
return at.UTC().Format("2006-01-02 15:04:05 UTC")
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestSourceList_ShowsActivityOfEachWebhook checks the figures the list
|
||||||
|
// shows for a webhook with recent failures, a healthy one, a new one
|
||||||
|
// that has received no event, and one without an event database.
|
||||||
|
func TestSourceList_ShowsActivityOfEachWebhook(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
var (
|
||||||
|
h *handlers.Handlers
|
||||||
|
sess *session.Session
|
||||||
|
db *database.Database
|
||||||
|
dbMgr *database.WebhookDBManager
|
||||||
|
)
|
||||||
|
|
||||||
|
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||||
|
app.RequireStart()
|
||||||
|
|
||||||
|
t.Cleanup(app.RequireStop)
|
||||||
|
|
||||||
|
failing, failingLastEvent := seedFailingWebhook(t, h, db, dbMgr)
|
||||||
|
healthy, healthyLastEvent := seedHealthyWebhook(t, h, db, dbMgr)
|
||||||
|
|
||||||
|
// Creating a webhook creates its event database.
|
||||||
|
fresh := listWebhook(t, db, "fresh")
|
||||||
|
require.NoError(t, dbMgr.CreateDB(fresh.ID))
|
||||||
|
addEntrypoints(t, db, fresh.ID, 2, true)
|
||||||
|
addTargets(t, db, fresh.ID, 3, true)
|
||||||
|
|
||||||
|
quiet := listWebhook(t, db, "quiet")
|
||||||
|
addEntrypoints(t, db, quiet.ID, 2, true)
|
||||||
|
addTargets(t, db, quiet.ID, 3, true)
|
||||||
|
|
||||||
|
page := renderWebhookList(t, h, sess)
|
||||||
|
|
||||||
|
card, text := listCard(t, page, failing.ID)
|
||||||
|
assert.Contains(t, text, "6 entrypoints, 2 inactive")
|
||||||
|
assert.Contains(t, text, "7 targets, 5 inactive")
|
||||||
|
assert.Contains(t, text, "4 events within retention")
|
||||||
|
assert.Contains(t, text, "Last event "+lastEventText(failingLastEvent))
|
||||||
|
assert.Contains(t, card,
|
||||||
|
failedHighlight+">3 failed deliveries in the last 24 hours<")
|
||||||
|
|
||||||
|
card, text = listCard(t, page, healthy.ID)
|
||||||
|
assert.Contains(t, text, "4 entrypoints")
|
||||||
|
assert.Contains(t, text, "2 targets")
|
||||||
|
assert.Contains(t, text, "3 events within retention")
|
||||||
|
assert.Contains(t, text, "Last event "+lastEventText(healthyLastEvent))
|
||||||
|
assert.Contains(t, text, "0 failed deliveries in the last 24 hours")
|
||||||
|
assert.NotContains(t, text, "inactive")
|
||||||
|
assert.NotContains(t, card, failedHighlight)
|
||||||
|
|
||||||
|
card, text = listCard(t, page, fresh.ID)
|
||||||
|
assert.Contains(t, text, "2 entrypoints")
|
||||||
|
assert.Contains(t, text, "3 targets")
|
||||||
|
assert.Contains(t, text, "0 events within retention")
|
||||||
|
assert.Contains(t, text, "No events yet")
|
||||||
|
assert.Contains(t, text, "0 failed deliveries in the last 24 hours")
|
||||||
|
assert.NotContains(t, card, failedHighlight)
|
||||||
|
|
||||||
|
card, text = listCard(t, page, quiet.ID)
|
||||||
|
assert.Contains(t, text, "2 entrypoints")
|
||||||
|
assert.Contains(t, text, "3 targets")
|
||||||
|
assert.Contains(t, text, "0 events within retention")
|
||||||
|
assert.Contains(t, text, "No events yet")
|
||||||
|
assert.Contains(t, text, "0 failed deliveries in the last 24 hours")
|
||||||
|
assert.NotContains(t, card, failedHighlight)
|
||||||
|
assert.False(t, dbMgr.DBExists(quiet.ID),
|
||||||
|
"showing the list must not create an event database")
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestSourceList_CountsOnlyEventsWithinRetention checks that once
|
||||||
|
// retention has removed one of a webhook's three events, the list
|
||||||
|
// counts the two still stored.
|
||||||
|
func TestSourceList_CountsOnlyEventsWithinRetention(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
var (
|
||||||
|
h *handlers.Handlers
|
||||||
|
sess *session.Session
|
||||||
|
db *database.Database
|
||||||
|
dbMgr *database.WebhookDBManager
|
||||||
|
log *logger.Logger
|
||||||
|
)
|
||||||
|
|
||||||
|
app := newTestApp(t, &h, &sess, &db, &dbMgr, &log)
|
||||||
|
app.RequireStart()
|
||||||
|
|
||||||
|
t.Cleanup(app.RequireStop)
|
||||||
|
|
||||||
|
wh := &database.Webhook{
|
||||||
|
UserID: deleteTestUserID, Name: "pruned", RetentionDays: 14,
|
||||||
|
}
|
||||||
|
require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
|
||||||
|
|
||||||
|
paths := addEntrypoints(t, db, wh.ID, 3, true)
|
||||||
|
addTargets(t, db, wh.ID, 4, true)
|
||||||
|
|
||||||
|
webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, paths[0], 3)
|
||||||
|
|
||||||
|
statsAge(t, webhookDB, events[0].ID, time.Now().Add(-15*24*time.Hour))
|
||||||
|
statsPrune(t, db, dbMgr, log, webhookDB)
|
||||||
|
require.Len(t, listEvents(t, webhookDB), 2)
|
||||||
|
|
||||||
|
_, text := listCard(t, renderWebhookList(t, h, sess), wh.ID)
|
||||||
|
assert.Contains(t, text, "3 entrypoints")
|
||||||
|
assert.Contains(t, text, "4 targets")
|
||||||
|
assert.Contains(t, text, "2 events within retention")
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestSourceList_LastEventSurvivesPruningEveryEvent checks that once
|
||||||
|
// retention has removed every event of a webhook, the list still shows
|
||||||
|
// when the last one arrived rather than "No events yet".
|
||||||
|
func TestSourceList_LastEventSurvivesPruningEveryEvent(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
var (
|
||||||
|
h *handlers.Handlers
|
||||||
|
sess *session.Session
|
||||||
|
db *database.Database
|
||||||
|
dbMgr *database.WebhookDBManager
|
||||||
|
log *logger.Logger
|
||||||
|
)
|
||||||
|
|
||||||
|
app := newTestApp(t, &h, &sess, &db, &dbMgr, &log)
|
||||||
|
app.RequireStart()
|
||||||
|
|
||||||
|
t.Cleanup(app.RequireStop)
|
||||||
|
|
||||||
|
wh := &database.Webhook{
|
||||||
|
UserID: deleteTestUserID, Name: "emptied", RetentionDays: 1,
|
||||||
|
}
|
||||||
|
require.NoError(t, db.DB().Omit(clause.Associations).Create(wh).Error)
|
||||||
|
|
||||||
|
paths := addEntrypoints(t, db, wh.ID, 2, true)
|
||||||
|
addTargets(t, db, wh.ID, 3, true)
|
||||||
|
|
||||||
|
webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, paths[0], 1)
|
||||||
|
lastEventAt := time.Now().Add(-50 * time.Hour)
|
||||||
|
|
||||||
|
statsAge(t, webhookDB, events[0].ID, lastEventAt)
|
||||||
|
require.NoError(t, database.AddEventTotals(webhookDB,
|
||||||
|
database.EventTotals{LastEventAt: &lastEventAt}))
|
||||||
|
statsPrune(t, db, dbMgr, log, webhookDB)
|
||||||
|
require.Empty(t, listEvents(t, webhookDB))
|
||||||
|
|
||||||
|
_, text := listCard(t, renderWebhookList(t, h, sess), wh.ID)
|
||||||
|
assert.Contains(t, text, "0 events within retention")
|
||||||
|
assert.Contains(t, text, "Last event "+lastEventText(lastEventAt))
|
||||||
|
assert.NotContains(t, text, "No events yet")
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestSourceList_CountsOfOneInSingular checks that a webhook with one
|
||||||
|
// entrypoint, one target, one event within retention and one failed
|
||||||
|
// delivery in the last 24 hours has each written in the singular.
|
||||||
|
func TestSourceList_CountsOfOneInSingular(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
var (
|
||||||
|
h *handlers.Handlers
|
||||||
|
sess *session.Session
|
||||||
|
db *database.Database
|
||||||
|
dbMgr *database.WebhookDBManager
|
||||||
|
)
|
||||||
|
|
||||||
|
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||||
|
app.RequireStart()
|
||||||
|
|
||||||
|
t.Cleanup(app.RequireStop)
|
||||||
|
|
||||||
|
wh := listWebhook(t, db, "single")
|
||||||
|
paths := addEntrypoints(t, db, wh.ID, 1, true)
|
||||||
|
targets := addTargets(t, db, wh.ID, 1, true)
|
||||||
|
|
||||||
|
webhookDB, events := receiveEvents(t, h, dbMgr, wh.ID, paths[0], 1)
|
||||||
|
now := time.Now()
|
||||||
|
lastEventAt := now.Add(-9 * time.Hour)
|
||||||
|
|
||||||
|
statsAge(t, webhookDB, events[0].ID, lastEventAt)
|
||||||
|
require.NoError(t, database.AddEventTotals(webhookDB,
|
||||||
|
database.EventTotals{LastEventAt: &lastEventAt}))
|
||||||
|
statsFinish(t, webhookDB,
|
||||||
|
statsDelivery(t, webhookDB, events[0].ID, targets[0].ID),
|
||||||
|
database.DeliveryStatusFailed, now.Add(-time.Hour))
|
||||||
|
|
||||||
|
card, text := listCard(t, renderWebhookList(t, h, sess), wh.ID)
|
||||||
|
assert.Contains(t, card, ">1 entrypoint<")
|
||||||
|
assert.Contains(t, card, ">1 target<")
|
||||||
|
assert.Contains(t, card, ">1 event within retention<")
|
||||||
|
assert.Contains(t, text, "Last event "+lastEventText(lastEventAt))
|
||||||
|
assert.Contains(t, card,
|
||||||
|
failedHighlight+">1 failed delivery in the last 24 hours<")
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestSourceList_UnreadableEventDatabase checks that a webhook whose
|
||||||
|
// event database cannot be read says so in its entry instead of
|
||||||
|
// showing zeros, and that the rest of the list is still shown.
|
||||||
|
func TestSourceList_UnreadableEventDatabase(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
var (
|
||||||
|
h *handlers.Handlers
|
||||||
|
sess *session.Session
|
||||||
|
db *database.Database
|
||||||
|
dbMgr *database.WebhookDBManager
|
||||||
|
)
|
||||||
|
|
||||||
|
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||||
|
app.RequireStart()
|
||||||
|
|
||||||
|
t.Cleanup(app.RequireStop)
|
||||||
|
|
||||||
|
broken := listWebhook(t, db, "broken")
|
||||||
|
addEntrypoints(t, db, broken.ID, 2, true)
|
||||||
|
addTargets(t, db, broken.ID, 3, true)
|
||||||
|
|
||||||
|
brokenDB, err := dbMgr.GetDB(broken.ID)
|
||||||
|
require.NoError(t, err)
|
||||||
|
require.NoError(t,
|
||||||
|
brokenDB.Migrator().DropTable(&database.EventTotals{}))
|
||||||
|
|
||||||
|
quiet := listWebhook(t, db, "quiet")
|
||||||
|
addEntrypoints(t, db, quiet.ID, 2, true)
|
||||||
|
addTargets(t, db, quiet.ID, 3, true)
|
||||||
|
|
||||||
|
page := renderWebhookList(t, h, sess)
|
||||||
|
|
||||||
|
_, text := listCard(t, page, broken.ID)
|
||||||
|
assert.Contains(t, text, "2 entrypoints")
|
||||||
|
assert.Contains(t, text, "3 targets")
|
||||||
|
assert.Contains(t, text, "The event figures could not be read.")
|
||||||
|
assert.NotContains(t, text, "events")
|
||||||
|
assert.NotContains(t, text, "failed")
|
||||||
|
|
||||||
|
_, text = listCard(t, page, quiet.ID)
|
||||||
|
assert.Contains(t, text, "No events yet")
|
||||||
|
}
|
||||||
@@ -3,10 +3,12 @@ package handlers
|
|||||||
import (
|
import (
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"errors"
|
"errors"
|
||||||
|
"fmt"
|
||||||
"net/http"
|
"net/http"
|
||||||
"slices"
|
"slices"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
"github.com/go-chi/chi"
|
"github.com/go-chi/chi"
|
||||||
"github.com/google/uuid"
|
"github.com/google/uuid"
|
||||||
@@ -20,9 +22,20 @@ import (
|
|||||||
type WebhookListItem struct {
|
type WebhookListItem struct {
|
||||||
database.Webhook
|
database.Webhook
|
||||||
|
|
||||||
EntrypointCount int64
|
EntrypointCount int
|
||||||
TargetCount int64
|
InactiveEntrypointCount int
|
||||||
EventCount int64
|
TargetCount int
|
||||||
|
InactiveTargetCount int
|
||||||
|
|
||||||
|
// EventCount is how many events the webhook holds, LastEventAt
|
||||||
|
// when the newest arrived (nil before the first), and
|
||||||
|
// FailedLast24Hours how many of its deliveries failed in the last
|
||||||
|
// 24 hours. When the webhook's event database could not be read,
|
||||||
|
// EventsUnreadable is set and these three are not known.
|
||||||
|
EventCount int64
|
||||||
|
LastEventAt *time.Time
|
||||||
|
FailedLast24Hours int64
|
||||||
|
EventsUnreadable bool
|
||||||
}
|
}
|
||||||
|
|
||||||
// errMissingURL signals that a required URL was not provided.
|
// errMissingURL signals that a required URL was not provided.
|
||||||
@@ -154,7 +167,12 @@ func (h *Handlers) HandleSourceList() http.HandlerFunc {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
items := h.buildWebhookListItems(webhooks)
|
items, err := h.buildWebhookListItems(webhooks)
|
||||||
|
if err != nil {
|
||||||
|
h.serverError(w, r, "failed to list webhooks", err)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
data := map[string]any{
|
data := map[string]any{
|
||||||
"Webhooks": items,
|
"Webhooks": items,
|
||||||
@@ -164,36 +182,115 @@ func (h *Handlers) HandleSourceList() http.HandlerFunc {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// buildWebhookListItems builds list items with counts.
|
// buildWebhookListItems builds the list's entry for each webhook. It
|
||||||
|
// fails when the main database cannot be read. A webhook whose event
|
||||||
|
// database cannot be read is marked on its own entry, and the error is
|
||||||
|
// logged.
|
||||||
func (h *Handlers) buildWebhookListItems(
|
func (h *Handlers) buildWebhookListItems(
|
||||||
webhooks []database.Webhook,
|
webhooks []database.Webhook,
|
||||||
) []WebhookListItem {
|
) ([]WebhookListItem, error) {
|
||||||
items := make([]WebhookListItem, len(webhooks))
|
items := make([]WebhookListItem, len(webhooks))
|
||||||
|
since := time.Now().Add(-longWindow)
|
||||||
|
|
||||||
for i := range webhooks {
|
for i := range webhooks {
|
||||||
items[i].Webhook = webhooks[i]
|
item := &items[i]
|
||||||
|
item.Webhook = webhooks[i]
|
||||||
|
|
||||||
h.db.DB().Model(&database.Entrypoint{}).Where(
|
var err error
|
||||||
"webhook_id = ?", webhooks[i].ID,
|
|
||||||
).Count(&items[i].EntrypointCount)
|
|
||||||
|
|
||||||
h.db.DB().Model(&database.Target{}).Where(
|
item.EntrypointCount, item.InactiveEntrypointCount, err =
|
||||||
"webhook_id = ?", webhooks[i].ID,
|
h.countWithInactive(&database.Entrypoint{}, item.ID)
|
||||||
).Count(&items[i].TargetCount)
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
if h.dbMgr.DBExists(webhooks[i].ID) {
|
item.TargetCount, item.InactiveTargetCount, err =
|
||||||
webhookDB, err := h.dbMgr.GetDB(
|
h.countWithInactive(&database.Target{}, item.ID)
|
||||||
webhooks[i].ID,
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// Opening an event database that does not exist would create
|
||||||
|
// it, and it would hold nothing to count.
|
||||||
|
if !h.dbMgr.DBExists(item.ID) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
err = h.readListEventFigures(item, since)
|
||||||
|
if err != nil {
|
||||||
|
h.log.Error(
|
||||||
|
"failed to read webhook list figures",
|
||||||
|
"webhook_id", item.ID,
|
||||||
|
"error", err,
|
||||||
)
|
)
|
||||||
if err == nil {
|
|
||||||
webhookDB.Model(
|
item.EventsUnreadable = true
|
||||||
&database.Event{},
|
|
||||||
).Count(&items[i].EventCount)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return items
|
return items, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// countWithInactive returns how many entrypoints or targets, as model
|
||||||
|
// says, a webhook has, and how many of them are inactive.
|
||||||
|
func (h *Handlers) countWithInactive(
|
||||||
|
model any, webhookID string,
|
||||||
|
) (int, int, error) {
|
||||||
|
var active []bool
|
||||||
|
|
||||||
|
err := h.db.DB().Model(model).
|
||||||
|
Where("webhook_id = ?", webhookID).
|
||||||
|
Pluck("active", &active).Error
|
||||||
|
if err != nil {
|
||||||
|
return 0, 0, fmt.Errorf(
|
||||||
|
"reading active flags of webhook %s: %w", webhookID, err,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
inactive := 0
|
||||||
|
|
||||||
|
for _, a := range active {
|
||||||
|
if !a {
|
||||||
|
inactive++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return len(active), inactive, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// readListEventFigures fills in the figures the list shows from the
|
||||||
|
// webhook's event database, with the statistics pane's own queries:
|
||||||
|
// the event count and last arrival from the event totals row, and the
|
||||||
|
// deliveries that failed since the given time from the deliveries'
|
||||||
|
// status index.
|
||||||
|
func (h *Handlers) readListEventFigures(
|
||||||
|
item *WebhookListItem, since time.Time,
|
||||||
|
) error {
|
||||||
|
webhookDB, err := h.dbMgr.GetDB(item.ID)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
var totals database.EventTotals
|
||||||
|
|
||||||
|
err = webhookDB.Take(&totals).Error
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("reading event totals: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
item.EventCount = totals.Events - totals.EventsRemoved
|
||||||
|
item.LastEventAt = totals.LastEventAt
|
||||||
|
|
||||||
|
byTarget, err := finishedByTarget(webhookDB, since)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, f := range byTarget {
|
||||||
|
item.FailedLast24Hours += f.Failed
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// HandleSourceCreate shows the form to create a new webhook.
|
// HandleSourceCreate shows the form to create a new webhook.
|
||||||
|
|||||||
@@ -0,0 +1,82 @@
|
|||||||
|
package handlers
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/http"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
|
"sneak.berlin/go/webhooker/internal/delivery"
|
||||||
|
)
|
||||||
|
|
||||||
|
// HandleTargetDownload serves a database target's archive as one
|
||||||
|
// gzipped JSON file, named for the webhook, the target and the time;
|
||||||
|
// see delivery.ArchiveExport.WriteGzipJSON for what it holds. Other
|
||||||
|
// target types have no archive and are a 404.
|
||||||
|
func (h *Handlers) HandleTargetDownload() http.HandlerFunc {
|
||||||
|
return func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
webhook, target, export, ok := h.openTargetArchive(w, r)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
defer func() { _ = export.Close() }()
|
||||||
|
|
||||||
|
now := time.Now()
|
||||||
|
|
||||||
|
w.Header().Set("Content-Type", "application/gzip")
|
||||||
|
w.Header().Set(
|
||||||
|
"Content-Disposition",
|
||||||
|
`attachment; filename="`+delivery.ArchiveExportFileName(
|
||||||
|
webhook.Name, target.Name, now,
|
||||||
|
)+`"`,
|
||||||
|
)
|
||||||
|
|
||||||
|
err := export.WriteGzipJSON(r.Context(), w, &webhook, target, now)
|
||||||
|
if err != nil {
|
||||||
|
// The 200 has gone out. The client is left with a file
|
||||||
|
// that does not decompress; the log is the record.
|
||||||
|
h.log.Error(
|
||||||
|
"failed to export archive",
|
||||||
|
"target_id", target.ID,
|
||||||
|
"error", err,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// openTargetArchive opens the archive of the request's database target
|
||||||
|
// for export. It reports false once it has written the response.
|
||||||
|
//
|
||||||
|
// It holds renameMu, which every archive rename runs under, while it
|
||||||
|
// reads the stored names and opens the file, so the file it opens is
|
||||||
|
// the one those names give. It lets go before the export is streamed:
|
||||||
|
// once the file is open, a rename does not affect the export.
|
||||||
|
func (h *Handlers) openTargetArchive(
|
||||||
|
w http.ResponseWriter,
|
||||||
|
r *http.Request,
|
||||||
|
) (database.Webhook, *database.Target, *delivery.ArchiveExport, bool) {
|
||||||
|
h.renameMu.Lock()
|
||||||
|
defer h.renameMu.Unlock()
|
||||||
|
|
||||||
|
webhook, target, ok := h.ownedTarget(w, r)
|
||||||
|
if !ok {
|
||||||
|
return database.Webhook{}, nil, nil, false
|
||||||
|
}
|
||||||
|
|
||||||
|
if target.Type != database.TargetTypeDatabase {
|
||||||
|
h.renderError(w, r, http.StatusNotFound)
|
||||||
|
|
||||||
|
return database.Webhook{}, nil, nil, false
|
||||||
|
}
|
||||||
|
|
||||||
|
export, err := delivery.OpenArchiveExport(
|
||||||
|
r.Context(), delivery.ArchivePath(h.dbMgr, &webhook, target),
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
h.serverError(w, r, "failed to open archive for export", err)
|
||||||
|
|
||||||
|
return database.Webhook{}, nil, nil, false
|
||||||
|
}
|
||||||
|
|
||||||
|
return webhook, target, export, true
|
||||||
|
}
|
||||||
@@ -0,0 +1,185 @@
|
|||||||
|
package handlers_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"compress/gzip"
|
||||||
|
"encoding/json"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"net/url"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
|
)
|
||||||
|
|
||||||
|
// downloadPath is the archive download route of a target.
|
||||||
|
func downloadPath(webhookID, targetID string) string {
|
||||||
|
return "/hook/" + webhookID + "/targets/" + targetID + "/download"
|
||||||
|
}
|
||||||
|
|
||||||
|
// renameTarget submits the edit form renaming a target to Renamed.
|
||||||
|
func renameTarget(
|
||||||
|
env *sourceTestEnv, webhookID, targetID string,
|
||||||
|
) *httptest.ResponseRecorder {
|
||||||
|
form := url.Values{}
|
||||||
|
form.Set("name", "Renamed")
|
||||||
|
|
||||||
|
return submitTargetEdit(env, webhookID, targetID, form)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleTargetDownload proves a database target's archive
|
||||||
|
// downloads as a gzipped JSON attachment named for the webhook, the
|
||||||
|
// target and the time, here with no archive file yet, so with no
|
||||||
|
// rows; and that a target of another type has no download.
|
||||||
|
func TestHandleTargetDownload(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
env := setupSourceTest(t)
|
||||||
|
wh := seedWebhookWithRetention(t, env.db, 7)
|
||||||
|
archive := seedTarget(t, env.db, wh.ID, database.TargetTypeDatabase)
|
||||||
|
logTarget := seedTarget(t, env.db, wh.ID, database.TargetTypeLog)
|
||||||
|
|
||||||
|
w := serveTarget(
|
||||||
|
env, http.MethodGet, downloadPath(wh.ID, archive.ID), nil,
|
||||||
|
)
|
||||||
|
require.Equal(t, http.StatusOK, w.Code, w.Body.String())
|
||||||
|
assert.Equal(t, "application/gzip", w.Header().Get("Content-Type"))
|
||||||
|
assert.Regexp(t,
|
||||||
|
`^attachment; filename="archive-seeded-t-database-`+
|
||||||
|
`\d{8}T\d{6}Z\.json\.gz"$`,
|
||||||
|
w.Header().Get("Content-Disposition"),
|
||||||
|
)
|
||||||
|
|
||||||
|
zr, err := gzip.NewReader(w.Body)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
var got map[string]json.RawMessage
|
||||||
|
|
||||||
|
require.NoError(t, json.NewDecoder(zr).Decode(&got))
|
||||||
|
assert.JSONEq(t,
|
||||||
|
`{"id":"`+archive.ID+`","name":"t-database"}`,
|
||||||
|
string(got["target"]),
|
||||||
|
)
|
||||||
|
assert.JSONEq(t, `[]`, string(got["archived_events"]))
|
||||||
|
|
||||||
|
w = serveTarget(
|
||||||
|
env, http.MethodGet, downloadPath(wh.ID, logTarget.ID), nil,
|
||||||
|
)
|
||||||
|
assert.Equal(t, http.StatusNotFound, w.Code)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleTargetDownload_WaitsForRename proves a download reads the
|
||||||
|
// target's names and opens its archive under the lock a rename holds:
|
||||||
|
// started while an edit is renaming the archive, it waits, and is
|
||||||
|
// named for the target's new name.
|
||||||
|
func TestHandleTargetDownload_WaitsForRename(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
env := setupSourceTest(t)
|
||||||
|
wh := seedWebhookWithRetention(t, env.db, 7)
|
||||||
|
archive := seedTarget(t, env.db, wh.ID, database.TargetTypeDatabase)
|
||||||
|
|
||||||
|
renaming, release := env.archives.BlockNextRename()
|
||||||
|
edited := make(chan *httptest.ResponseRecorder, 1)
|
||||||
|
|
||||||
|
go func() {
|
||||||
|
edited <- renameTarget(env, wh.ID, archive.ID)
|
||||||
|
}()
|
||||||
|
|
||||||
|
<-renaming
|
||||||
|
|
||||||
|
downloaded := make(chan *httptest.ResponseRecorder, 1)
|
||||||
|
|
||||||
|
go func() {
|
||||||
|
downloaded <- serveTarget(
|
||||||
|
env, http.MethodGet, downloadPath(wh.ID, archive.ID), nil,
|
||||||
|
)
|
||||||
|
}()
|
||||||
|
|
||||||
|
select {
|
||||||
|
case <-downloaded:
|
||||||
|
release()
|
||||||
|
t.Fatal("the download did not wait for the rename")
|
||||||
|
case <-time.After(100 * time.Millisecond):
|
||||||
|
}
|
||||||
|
|
||||||
|
release()
|
||||||
|
require.Equal(t, http.StatusSeeOther, (<-edited).Code)
|
||||||
|
|
||||||
|
w := <-downloaded
|
||||||
|
require.Equal(t, http.StatusOK, w.Code)
|
||||||
|
assert.Contains(t,
|
||||||
|
w.Header().Get("Content-Disposition"), "archive-seeded-renamed-",
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// stalledWriter is a response writer whose first write waits until
|
||||||
|
// resume is closed, closing writing when it starts to wait.
|
||||||
|
type stalledWriter struct {
|
||||||
|
*httptest.ResponseRecorder
|
||||||
|
|
||||||
|
once sync.Once
|
||||||
|
writing chan struct{}
|
||||||
|
resume chan struct{}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *stalledWriter) Write(b []byte) (int, error) {
|
||||||
|
s.once.Do(func() {
|
||||||
|
close(s.writing)
|
||||||
|
<-s.resume
|
||||||
|
})
|
||||||
|
|
||||||
|
return s.ResponseRecorder.Write(b)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleTargetDownload_StreamsWithoutTheLock proves a download
|
||||||
|
// lets go of the rename lock once its archive is open: while the
|
||||||
|
// download is stalled writing, an edit can still rename the target.
|
||||||
|
func TestHandleTargetDownload_StreamsWithoutTheLock(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
env := setupSourceTest(t)
|
||||||
|
wh := seedWebhookWithRetention(t, env.db, 7)
|
||||||
|
archive := seedTarget(t, env.db, wh.ID, database.TargetTypeDatabase)
|
||||||
|
|
||||||
|
req := httptest.NewRequestWithContext(
|
||||||
|
t.Context(), http.MethodGet, downloadPath(wh.ID, archive.ID), nil,
|
||||||
|
)
|
||||||
|
for _, c := range env.cookies {
|
||||||
|
req.AddCookie(c)
|
||||||
|
}
|
||||||
|
|
||||||
|
sw := &stalledWriter{
|
||||||
|
ResponseRecorder: httptest.NewRecorder(),
|
||||||
|
writing: make(chan struct{}),
|
||||||
|
resume: make(chan struct{}),
|
||||||
|
}
|
||||||
|
downloaded := make(chan struct{})
|
||||||
|
|
||||||
|
go func() {
|
||||||
|
targetRouter(env).ServeHTTP(sw, req)
|
||||||
|
close(downloaded)
|
||||||
|
}()
|
||||||
|
|
||||||
|
<-sw.writing
|
||||||
|
|
||||||
|
edited := make(chan *httptest.ResponseRecorder, 1)
|
||||||
|
|
||||||
|
go func() {
|
||||||
|
edited <- renameTarget(env, wh.ID, archive.ID)
|
||||||
|
}()
|
||||||
|
|
||||||
|
select {
|
||||||
|
case w := <-edited:
|
||||||
|
assert.Equal(t, http.StatusSeeOther, w.Code)
|
||||||
|
case <-time.After(10 * time.Second):
|
||||||
|
t.Error("the rename waited for the download")
|
||||||
|
}
|
||||||
|
|
||||||
|
close(sw.resume)
|
||||||
|
<-downloaded
|
||||||
|
assert.Equal(t, http.StatusOK, sw.Code)
|
||||||
|
}
|
||||||
@@ -37,14 +37,18 @@ const (
|
|||||||
editAuthHeader = "Authorization: Bearer " + editBearerSecret
|
editAuthHeader = "Authorization: Bearer " + editBearerSecret
|
||||||
)
|
)
|
||||||
|
|
||||||
// targetRouter mounts the target create and edit routes on a chi
|
// targetRouter mounts the target create, edit and download routes on
|
||||||
// router so the handlers see the URL parameters they read.
|
// a chi router so the handlers see the URL parameters they read.
|
||||||
func targetRouter(env *sourceTestEnv) *chi.Mux {
|
func targetRouter(env *sourceTestEnv) *chi.Mux {
|
||||||
router := chi.NewRouter()
|
router := chi.NewRouter()
|
||||||
router.Post(
|
router.Post(
|
||||||
"/hook/{sourceID}/targets",
|
"/hook/{sourceID}/targets",
|
||||||
env.handlers.HandleTargetCreate(),
|
env.handlers.HandleTargetCreate(),
|
||||||
)
|
)
|
||||||
|
router.Get(
|
||||||
|
"/hook/{sourceID}/targets/{targetID}/download",
|
||||||
|
env.handlers.HandleTargetDownload(),
|
||||||
|
)
|
||||||
router.Get(
|
router.Get(
|
||||||
"/hook/{sourceID}/targets/{targetID}/edit",
|
"/hook/{sourceID}/targets/{targetID}/edit",
|
||||||
env.handlers.HandleTargetEdit(),
|
env.handlers.HandleTargetEdit(),
|
||||||
|
|||||||
@@ -150,7 +150,9 @@ func (h *Handlers) lookupEntrypoint(
|
|||||||
return entrypoint, true
|
return entrypoint, true
|
||||||
}
|
}
|
||||||
|
|
||||||
// readWebhookBody reads and validates the request body size.
|
// readWebhookBody reads and validates the request body size. This is
|
||||||
|
// the receiver's only body cap: /h/{uuid} has no MaxBodySize
|
||||||
|
// middleware (see Server.setupWebhookRoutes).
|
||||||
func (h *Handlers) readWebhookBody(
|
func (h *Handlers) readWebhookBody(
|
||||||
w http.ResponseWriter,
|
w http.ResponseWriter,
|
||||||
r *http.Request,
|
r *http.Request,
|
||||||
|
|||||||
@@ -0,0 +1,69 @@
|
|||||||
|
package server_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"compress/gzip"
|
||||||
|
"encoding/json"
|
||||||
|
"net/http"
|
||||||
|
"regexp"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"gorm.io/gorm/clause"
|
||||||
|
"sneak.berlin/go/webhooker/internal/database"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestHook_DownloadArchive follows the Download link the webhook page
|
||||||
|
// shows for a database target, and only for it, and gets the archive
|
||||||
|
// as a gzipped JSON file. Signed out, the link leads to the login page.
|
||||||
|
func TestHook_DownloadArchive(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
env := newTestEnv(t)
|
||||||
|
|
||||||
|
userID, _ := env.seedUser(t, "archivist", "somepassword")
|
||||||
|
cookies := env.authCookies(t, userID, "archivist")
|
||||||
|
wh := env.seedWebhook(t, userID)
|
||||||
|
env.seedTarget(t, wh.ID)
|
||||||
|
|
||||||
|
archive := &database.Target{
|
||||||
|
WebhookID: wh.ID,
|
||||||
|
Name: "kept",
|
||||||
|
Type: database.TargetTypeDatabase,
|
||||||
|
Active: true,
|
||||||
|
}
|
||||||
|
require.NoError(t,
|
||||||
|
env.db.DB().Omit(clause.Associations).Create(archive).Error,
|
||||||
|
)
|
||||||
|
|
||||||
|
page := env.get("/hook/"+wh.ID, cookies)
|
||||||
|
require.Equal(t, http.StatusOK, page.Code)
|
||||||
|
|
||||||
|
links := regexp.MustCompile(
|
||||||
|
`href="(/hook/[^/"]+/targets/[^/"]+/download)"`,
|
||||||
|
).FindAllStringSubmatch(page.Body.String(), -1)
|
||||||
|
require.Len(t, links, 1, "only the database target has a Download")
|
||||||
|
|
||||||
|
link := links[0][1]
|
||||||
|
assert.Equal(t,
|
||||||
|
"/hook/"+wh.ID+"/targets/"+archive.ID+"/download", link,
|
||||||
|
)
|
||||||
|
|
||||||
|
w := env.get(link, cookies)
|
||||||
|
require.Equal(t, http.StatusOK, w.Code)
|
||||||
|
assert.Equal(t, "application/gzip", w.Header().Get("Content-Type"))
|
||||||
|
|
||||||
|
zr, err := gzip.NewReader(w.Body)
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
var got map[string]json.RawMessage
|
||||||
|
|
||||||
|
require.NoError(t, json.NewDecoder(zr).Decode(&got))
|
||||||
|
assert.JSONEq(t,
|
||||||
|
`{"id":"`+wh.ID+`","name":"routed"}`, string(got["webhook"]),
|
||||||
|
)
|
||||||
|
|
||||||
|
w = env.get(link, nil)
|
||||||
|
assert.Equal(t, http.StatusSeeOther, w.Code)
|
||||||
|
assert.Contains(t, w.Header().Get("Location"), "/pages/login")
|
||||||
|
}
|
||||||
@@ -312,6 +312,10 @@ func (s *Server) setupSourceRoutes() {
|
|||||||
"/targets/{targetID}/edit",
|
"/targets/{targetID}/edit",
|
||||||
s.h.HandleTargetEditSubmit(),
|
s.h.HandleTargetEditSubmit(),
|
||||||
)
|
)
|
||||||
|
r.Get(
|
||||||
|
"/targets/{targetID}/download",
|
||||||
|
s.h.HandleTargetDownload(),
|
||||||
|
)
|
||||||
r.Post(
|
r.Post(
|
||||||
"/targets/{targetID}/delete",
|
"/targets/{targetID}/delete",
|
||||||
s.h.HandleTargetDelete(),
|
s.h.HandleTargetDelete(),
|
||||||
@@ -324,6 +328,12 @@ func (s *Server) setupSourceRoutes() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (s *Server) setupWebhookRoutes() {
|
func (s *Server) setupWebhookRoutes() {
|
||||||
|
// No MaxBodySize here, unlike the page groups. The receiver's 1 MB
|
||||||
|
// body cap is in Handlers.readWebhookBody, because the handler
|
||||||
|
// owns the response a sender gets for an oversized body and
|
||||||
|
// MaxBodySize would change it. That cap is the only bound on this
|
||||||
|
// unauthenticated endpoint's body; TestReceiver_OversizeBodyRefused
|
||||||
|
// pins it.
|
||||||
s.router.With(s.mw.ReceiverRateLimit()).HandleFunc(
|
s.router.With(s.mw.ReceiverRateLimit()).HandleFunc(
|
||||||
"/h/{uuid}",
|
"/h/{uuid}",
|
||||||
s.h.HandleWebhook(),
|
s.h.HandleWebhook(),
|
||||||
|
|||||||
@@ -1428,6 +1428,53 @@ func TestReceiver_EntrypointURLIsRateLimited(t *testing.T) {
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestReceiver_OversizeBodyRefused pins the receiver's 1 MB body
|
||||||
|
// cap, which lives in the handler rather than in a MaxBodySize
|
||||||
|
// middleware. A body exactly at the cap is accepted; one byte over is
|
||||||
|
// refused with the handler's own 413.
|
||||||
|
func TestReceiver_OversizeBodyRefused(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
const bodyCap = 1 << 20 // 1 MB
|
||||||
|
|
||||||
|
env := newTestEnvWithConfig(t, &config.Config{
|
||||||
|
DataDir: t.TempDir(),
|
||||||
|
Environment: config.EnvironmentDev,
|
||||||
|
ReceiverRateLimit: 10,
|
||||||
|
})
|
||||||
|
|
||||||
|
userID, _ := env.seedUser(t, "receiver", "somepassword")
|
||||||
|
wh := env.seedWebhook(t, userID)
|
||||||
|
require.NoError(t, env.db.DB().Omit(clause.Associations).Create(
|
||||||
|
&database.Entrypoint{
|
||||||
|
WebhookID: wh.ID,
|
||||||
|
Path: "0b7c3e5a-2d9f-4a61-8e4b-7c1d6f2a9e35",
|
||||||
|
Active: true,
|
||||||
|
},
|
||||||
|
).Error)
|
||||||
|
|
||||||
|
send := func(size int) *httptest.ResponseRecorder {
|
||||||
|
req := httptest.NewRequestWithContext(
|
||||||
|
context.Background(), http.MethodPost,
|
||||||
|
"/h/0b7c3e5a-2d9f-4a61-8e4b-7c1d6f2a9e35",
|
||||||
|
strings.NewReader(strings.Repeat("a", size)),
|
||||||
|
)
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
env.router.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
return w
|
||||||
|
}
|
||||||
|
|
||||||
|
assert.Equal(
|
||||||
|
t, http.StatusOK, send(bodyCap).Code,
|
||||||
|
"a body exactly at the cap must be accepted",
|
||||||
|
)
|
||||||
|
|
||||||
|
w := send(bodyCap + 1)
|
||||||
|
assert.Equal(t, http.StatusRequestEntityTooLarge, w.Code)
|
||||||
|
assert.Equal(t, "Request body too large\n", w.Body.String())
|
||||||
|
}
|
||||||
|
|
||||||
// metricsConfig is a Config differing from the routing default only
|
// metricsConfig is a Config differing from the routing default only
|
||||||
// in the two /metrics credentials.
|
// in the two /metrics credentials.
|
||||||
func metricsConfig(
|
func metricsConfig(
|
||||||
|
|||||||
+9
-3
@@ -2,9 +2,10 @@
|
|||||||
# script/test: run the test suite.
|
# script/test: run the test suite.
|
||||||
#
|
#
|
||||||
# -timeout is applied by `go test` per package, not to the run as a whole, so
|
# -timeout is applied by `go test` per package, not to the run as a whole, so
|
||||||
# it only has to clear the slowest single package. That is internal/handlers,
|
# it only has to clear the slowest single package. When this budget was set
|
||||||
# measured in a cache-defeated builder stage on the 48-core shared build host
|
# that was internal/handlers, measured in a cache-defeated builder stage on the
|
||||||
# (2026-08-18); load- and host-dependent, not invariants:
|
# 48-core shared build host (2026-08-18); load- and host-dependent, not
|
||||||
|
# invariants:
|
||||||
#
|
#
|
||||||
# 16.9s host load 5-20, GOMAXPROCS 48
|
# 16.9s host load 5-20, GOMAXPROCS 48
|
||||||
# 45.9s / 47.3s / 49.0s three runs at deliberate host load 31-73
|
# 45.9s / 47.3s / 49.0s three runs at deliberate host load 31-73
|
||||||
@@ -23,6 +24,11 @@
|
|||||||
# a condition CI runs under. If a CPU-limited runner ever puts a real run near
|
# a condition CI runs under. If a CPU-limited runner ever puts a real run near
|
||||||
# 67s, that is the datum to revisit the org figure with.
|
# 67s, that is the datum to revisit the org figure with.
|
||||||
#
|
#
|
||||||
|
# Those figures predate tests hashing the admin password at 1 MB instead of
|
||||||
|
# 64 MB (https://git.eeqj.de/sneak/webhooker/pulls/404). After that change, in
|
||||||
|
# a cache-defeated build at host load 44-109 (2026-10-02), internal/handlers
|
||||||
|
# took 8.5s and the slowest package was internal/database at 15.8s.
|
||||||
|
#
|
||||||
# -p 4 -parallel 8 keep the run under 2 GB of memory: at most four test
|
# -p 4 -parallel 8 keep the run under 2 GB of memory: at most four test
|
||||||
# binaries build or run at once, each with at most eight parallel tests. Under
|
# binaries build or run at once, each with at most eight parallel tests. Under
|
||||||
# -race every test binary and every link costs a few hundred MB, so the
|
# -race every test binary and every link costs a few hundred MB, so the
|
||||||
|
|||||||
@@ -157,6 +157,9 @@
|
|||||||
{{else}}
|
{{else}}
|
||||||
<span class="badge-error">Inactive</span>
|
<span class="badge-error">Inactive</span>
|
||||||
{{end}}
|
{{end}}
|
||||||
|
{{if eq .Type "database"}}
|
||||||
|
<a href="/hook/{{$.Webhook.ID}}/targets/{{.ID}}/download" class="text-xs text-gray-500 hover:text-primary-600" title="Download the archive as gzipped JSON">Download</a>
|
||||||
|
{{end}}
|
||||||
<a href="/hook/{{$.Webhook.ID}}/targets/{{.ID}}/edit" class="text-xs text-gray-500 hover:text-primary-600" title="Edit">Edit</a>
|
<a href="/hook/{{$.Webhook.ID}}/targets/{{.ID}}/edit" class="text-xs text-gray-500 hover:text-primary-600" title="Edit">Edit</a>
|
||||||
<form method="POST" action="/hook/{{$.Webhook.ID}}/targets/{{.ID}}/toggle" class="inline">
|
<form method="POST" action="/hook/{{$.Webhook.ID}}/targets/{{.ID}}/toggle" class="inline">
|
||||||
<input type="hidden" name="csrf_token" value="{{$.CSRFToken}}">
|
<input type="hidden" name="csrf_token" value="{{$.CSRFToken}}">
|
||||||
|
|||||||
@@ -27,10 +27,16 @@
|
|||||||
</div>
|
</div>
|
||||||
<span class="badge-info">Retention: {{.RetentionLabel}}</span>
|
<span class="badge-info">Retention: {{.RetentionLabel}}</span>
|
||||||
</div>
|
</div>
|
||||||
<div class="flex gap-6 mt-4 text-sm text-gray-500">
|
<div class="flex flex-wrap gap-6 mt-4 text-sm text-gray-500">
|
||||||
<span>{{.EntrypointCount}} entrypoint{{if ne .EntrypointCount 1}}s{{end}}</span>
|
<span>{{.EntrypointCount}} entrypoint{{if ne .EntrypointCount 1}}s{{end}}{{if .InactiveEntrypointCount}}, {{.InactiveEntrypointCount}} inactive{{end}}</span>
|
||||||
<span>{{.TargetCount}} target{{if ne .TargetCount 1}}s{{end}}</span>
|
<span>{{.TargetCount}} target{{if ne .TargetCount 1}}s{{end}}{{if .InactiveTargetCount}}, {{.InactiveTargetCount}} inactive{{end}}</span>
|
||||||
<span>{{.EventCount}} event{{if ne .EventCount 1}}s{{end}}</span>
|
{{if .EventsUnreadable}}
|
||||||
|
<span class="text-red-600">The event figures could not be read.</span>
|
||||||
|
{{else}}
|
||||||
|
<span>{{.EventCount}} event{{if ne .EventCount 1}}s{{end}} within retention</span>
|
||||||
|
<span>{{with .LastEventAt}}Last event {{.UTC.Format "2006-01-02 15:04:05 UTC"}}{{else}}No events yet{{end}}</span>
|
||||||
|
<span class="{{if .FailedLast24Hours}}font-medium text-red-600{{end}}">{{.FailedLast24Hours}} failed deliver{{if eq .FailedLast24Hours 1}}y{{else}}ies{{end}} in the last 24 hours</span>
|
||||||
|
{{end}}
|
||||||
</div>
|
</div>
|
||||||
</a>
|
</a>
|
||||||
{{end}}
|
{{end}}
|
||||||
|
|||||||
Reference in new issue
Block a user