Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f01e5c45ad | ||
|
|
9526e961b5 | ||
|
|
4915d60d8e |
@@ -1327,17 +1327,17 @@ under the real policy and checks that: both add forms stay hidden until Add is
|
||||
clicked; choosing Slack in the add target form leaves the HTTP fields out of
|
||||
what it submits, also after leaving the page and going back to it, when the
|
||||
browser restores the choice; the Copy button beside an entrypoint URL reads
|
||||
"Copied" once clicked; an entrypoint's Edit button shows its edit form in place
|
||||
of its description and hides until the form closes, Cancel hides the form and
|
||||
drops what was typed, and Save changes the description; an event expands and
|
||||
collapses, and so do a delivery's attempts inside it; and at phone width the
|
||||
menu button opens and closes the mobile menu. It also fails if the browser
|
||||
reports a console warning or error, an uncaught exception, or anything the
|
||||
policy refused. `make check` and the image build lint it but do not run it, and
|
||||
`make test` leaves it out (its file is built only with the `browser` build tag).
|
||||
Run it with `make test-browser` after changing `templates/` or `static/js/`:
|
||||
that builds `Dockerfile.browser`, which runs the test in a digest-pinned
|
||||
headless browser image, so the host needs no browser.
|
||||
"Copied" once clicked; of the recent events on the webhook page only the newest
|
||||
starts expanded, each expands and collapses, and Open leads to the event's own
|
||||
page; an event in the event log expands and collapses, and so do a delivery's
|
||||
attempts inside it; and at phone width the menu button opens and closes the
|
||||
mobile menu. It also fails if the browser reports a console warning or error,
|
||||
an uncaught exception, or anything the policy refused. `make check` and the
|
||||
image build lint it but do not run it, and `make test` leaves it out (its file
|
||||
is built only with the `browser` build tag). Run it with `make test-browser`
|
||||
after changing `templates/` or `static/js/`: that builds `Dockerfile.browser`,
|
||||
which runs the test in a digest-pinned headless browser image, so the host
|
||||
needs no browser.
|
||||
|
||||
The package's tarball is committed as `3p/alpinejs-csp-3.14.9.tgz`, byte for
|
||||
byte as the npm registry publishes it. It is a dependency, not this repo's build
|
||||
@@ -2033,6 +2033,29 @@ Because each `database` target has its own archive file, a target's
|
||||
webhook with different expiries keep two archives, each pruned on its
|
||||
own schedule.
|
||||
|
||||
Each `database` target on the webhook page has a **Download** button,
|
||||
which returns its archive as one gzipped JSON file,
|
||||
`archive-{webhook_name}-{target_name}-{YYYYMMDDTHHMMSSZ}.json.gz`, the
|
||||
names made safe as above and the time in UTC. The file holds one
|
||||
object: `webhook` and `target`, each an `id` and a `name`;
|
||||
`exported_at`; and `archived_events`, one object per archived row with
|
||||
every column, keyed by column name. A body that is not valid UTF-8 is
|
||||
written in base64, with `"body_encoding": "base64"` beside it. An
|
||||
archive that does not exist yet, or was moved away, downloads with an
|
||||
empty `archived_events`; the download never creates the file.
|
||||
|
||||
The download streams: each row is read and written out compressed
|
||||
before the next is read, so neither the archive nor the JSON is held in
|
||||
memory. It reads on a connection of its own, inside one read-only
|
||||
transaction, so the file holds the archive as it stood when the
|
||||
download started, and archive writes go on meanwhile, since under WAL a
|
||||
reader never blocks a writer. While it runs, the `-wal` cannot be
|
||||
checkpointed past what it reads, so a long download lets the `-wal`
|
||||
grow. It finds the file by the stored names under the lock that webhook
|
||||
edits, target edits and target creation hold, and lets go once the file
|
||||
is open: a rename during the download moves the file without affecting
|
||||
it.
|
||||
|
||||
Deleting a webhook releases its archives: the delivery engine's cached
|
||||
archive writers are dropped and their file handles closed, so nothing
|
||||
lingers after the webhook is gone. The archive **files themselves are
|
||||
@@ -2611,11 +2634,10 @@ on all three arms of `Trace`, including the routine one an operator
|
||||
reaches at `DEBUG`, which is the only level at which a successful
|
||||
`INSERT` is written at all. One GORM path does not consult the filter —
|
||||
`(*gorm.DB).Scan`, which records the statement through GORM's own trace
|
||||
recorder. No production code path calls it; its one caller is
|
||||
`internal/database/database_test.go:91`, whose `SELECT 1` binds
|
||||
nothing, and `internal/gormlog/scan_guard_test.go` fails if a non-test
|
||||
file calls it. `Pluck`, `Row` and `Raw` all run through the normal
|
||||
callback processor and are filtered.
|
||||
recorder. No production code path calls it; only tests do, and what a
|
||||
test binds is fixture data. `internal/gormlog/scan_guard_test.go` fails
|
||||
if a non-test file calls it. `Pluck`, `Row` and `Raw` all run through
|
||||
the normal callback processor and are filtered.
|
||||
See `#### What DEBUG=true exposes` under Configuration.
|
||||
|
||||
What that ceiling does **not** cover, stated here so the figure is not
|
||||
@@ -2891,16 +2913,17 @@ returns to the page that was asked for.
|
||||
| `POST` | `/hook/{id}/edit` | Edit webhook submission |
|
||||
| `POST` | `/hook/{id}/delete` | Delete webhook |
|
||||
| `GET` | `/hook/{id}/events` | Full Event Log |
|
||||
| `GET` | `/hook/{id}/events/{eventID}/body` | Download an event's full stored body. The log page renders each body only up to its cap, so this is the only route that serves a whole one; it is offered wherever a body is shown truncated |
|
||||
| `GET` | `/hook/{id}/events/{eventID}` | One event's own page: its details, its whole body and every delivery of it |
|
||||
| `GET` | `/hook/{id}/events/{eventID}/body` | Download an event's stored body. The pages show a body as text, cut at 32 KiB in the recent events and the event log, and leave a binary one out, so this is the only route that serves the stored bytes; it is offered wherever a body is cut or binary |
|
||||
| `POST` | `/hook/{id}/deliveries/{deliveryID}/replay` | Replay a finished delivery: creates a new delivery for the same event against the target's current configuration (30 per minute per bucket, then `429`) |
|
||||
| `POST` | `/hook/{id}/events/{eventID}/resubmit` | Resubmit a stored event: creates a new event copying it and fans that out to every currently active target (30 per minute per bucket, then `429`) |
|
||||
| `POST` | `/hook/{id}/entrypoints` | Add entrypoint to webhook |
|
||||
| `POST` | `/hook/{id}/entrypoints/{entrypointID}/edit` | Change an entrypoint's description; its URL stays the same |
|
||||
| `POST` | `/hook/{id}/entrypoints/{entrypointID}/delete` | Delete an entrypoint |
|
||||
| `POST` | `/hook/{id}/entrypoints/{entrypointID}/toggle` | Enable or disable an entrypoint |
|
||||
| `POST` | `/hook/{id}/targets` | Add target to webhook |
|
||||
| `GET` | `/hook/{id}/targets/{targetID}/edit` | Edit target form. The one page that renders a target's destination URL and header values in full, rather than masked |
|
||||
| `POST` | `/hook/{id}/targets/{targetID}/edit` | Edit target submission |
|
||||
| `GET` | `/hook/{id}/targets/{targetID}/download` | Download a `database` target's archive as one gzipped JSON file. See [Database Architecture](#database-architecture) |
|
||||
| `POST` | `/hook/{id}/targets/{targetID}/delete` | Delete a target |
|
||||
| `POST` | `/hook/{id}/targets/{targetID}/toggle` | Enable or disable a target |
|
||||
|
||||
@@ -2982,6 +3005,7 @@ webhooker/
|
||||
│ │ ├── target_slack.go # Slack/Mattermost incoming-webhook target
|
||||
│ │ ├── target_database.go # Database archive target
|
||||
│ │ ├── target_database_archive.go # Archive file lifecycle and pruning
|
||||
│ │ ├── target_database_export.go # Archive download as gzipped JSON
|
||||
│ │ ├── target_log.go # Log target (stdout)
|
||||
│ │ ├── target_config_view.go # Masked target config for templates
|
||||
│ │ ├── archive_sweeper.go # Periodic pruning of idle archives
|
||||
|
||||
@@ -32,8 +32,8 @@ type Event struct {
|
||||
ContentType string `json:"contentType"`
|
||||
|
||||
// BodyBytes is the size of Body in bytes, recorded when the event
|
||||
// is stored so the recent events list can show it without reading
|
||||
// the body.
|
||||
// is stored, so that the recent events list, which reads only the
|
||||
// start of each body, knows the whole body's size.
|
||||
BodyBytes int64 `gorm:"not null" json:"bodyBytes"`
|
||||
|
||||
// ResubmittedFromID names the event this one was copied from by
|
||||
|
||||
@@ -22,6 +22,7 @@ import (
|
||||
_ "modernc.org/sqlite" // Pure Go SQLite driver.
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
||||
)
|
||||
|
||||
const (
|
||||
@@ -70,7 +71,8 @@ func setupArchiveTest(t *testing.T) *archiveEnv {
|
||||
t.Cleanup(func() { _ = sqlDB.Close() })
|
||||
|
||||
gdb, err := gorm.Open(
|
||||
sqlite.Dialector{Conn: sqlDB}, &gorm.Config{},
|
||||
sqlite.Dialector{Conn: sqlDB},
|
||||
&gorm.Config{Logger: gormlog.New(slog.New(slog.DiscardHandler))},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
@@ -168,7 +170,8 @@ func (env *archiveEnv) seedArchiveRows(
|
||||
require.NoError(t, err)
|
||||
|
||||
gdb, err := gorm.Open(
|
||||
sqlite.Dialector{Conn: sqlDB}, &gorm.Config{},
|
||||
sqlite.Dialector{Conn: sqlDB},
|
||||
&gorm.Config{Logger: gormlog.New(slog.New(slog.DiscardHandler))},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
@@ -227,7 +230,8 @@ func countArchivedRows(path string) (int64, error) {
|
||||
defer func() { _ = sqlDB.Close() }()
|
||||
|
||||
gdb, err := gorm.Open(
|
||||
sqlite.Dialector{Conn: sqlDB}, &gorm.Config{},
|
||||
sqlite.Dialector{Conn: sqlDB},
|
||||
&gorm.Config{Logger: gormlog.New(slog.New(slog.DiscardHandler))},
|
||||
)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
|
||||
@@ -23,6 +23,7 @@ import (
|
||||
_ "modernc.org/sqlite"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
||||
)
|
||||
|
||||
// iSetup holds common integration test dependencies.
|
||||
@@ -80,7 +81,8 @@ func iMainDB(t *testing.T) *gorm.DB {
|
||||
t.Cleanup(func() { _ = sqlDB.Close() })
|
||||
|
||||
db, err := gorm.Open(
|
||||
sqlite.Dialector{Conn: sqlDB}, &gorm.Config{},
|
||||
sqlite.Dialector{Conn: sqlDB},
|
||||
&gorm.Config{Logger: gormlog.New(slog.New(slog.DiscardHandler))},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
|
||||
@@ -26,6 +26,7 @@ import (
|
||||
_ "modernc.org/sqlite"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
||||
"sneak.berlin/go/webhooker/internal/metrics"
|
||||
)
|
||||
|
||||
@@ -50,7 +51,8 @@ func testWebhookDB(t *testing.T) *gorm.DB {
|
||||
t.Cleanup(func() { _ = sqlDB.Close() })
|
||||
|
||||
db, err := gorm.Open(
|
||||
sqlite.Dialector{Conn: sqlDB}, &gorm.Config{},
|
||||
sqlite.Dialector{Conn: sqlDB},
|
||||
&gorm.Config{Logger: gormlog.New(slog.New(slog.DiscardHandler))},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
|
||||
@@ -3,7 +3,6 @@ package delivery
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
@@ -277,10 +276,9 @@ func (t *databaseTarget) releaseSweepWriter(
|
||||
}
|
||||
|
||||
// newWriter builds the writer for a database target's archive. The
|
||||
// file lives beside the webhook's event database in the data
|
||||
// directory and is named for the webhook and the target as the main
|
||||
// database has them now; from then on only rename changes the name
|
||||
// the writer uses. It does not touch the archive file.
|
||||
// file is the one ArchivePath gives for the webhook and the target as
|
||||
// the main database names them now; from then on only rename changes
|
||||
// the name the writer uses. It does not touch the archive file.
|
||||
func (t *databaseTarget) newWriter(
|
||||
targetID string,
|
||||
) (*archiveWriter, error) {
|
||||
@@ -299,12 +297,10 @@ func (t *databaseTarget) newWriter(
|
||||
)
|
||||
}
|
||||
|
||||
dir := filepath.Dir(t.eng.dbManager.DBPath(target.WebhookID))
|
||||
name := ArchiveFileName(
|
||||
target.Webhook.Name, target.Name, target.ID,
|
||||
w := newArchiveWriter(
|
||||
ArchivePath(t.eng.dbManager, &target.Webhook, &target),
|
||||
t.eng.log,
|
||||
)
|
||||
|
||||
w := newArchiveWriter(filepath.Join(dir, name), t.eng.log)
|
||||
w.webhookID = target.WebhookID
|
||||
|
||||
return w, nil
|
||||
|
||||
@@ -0,0 +1,275 @@
|
||||
package delivery
|
||||
|
||||
import (
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"path/filepath"
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
|
||||
"gorm.io/driver/sqlite"
|
||||
"gorm.io/gorm"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
||||
)
|
||||
|
||||
// archiveTableQuery counts the archive's table: 0 when the archive
|
||||
// writer has created the file but not yet the table in it.
|
||||
const archiveTableQuery = "SELECT count(*) FROM sqlite_master " +
|
||||
"WHERE type = 'table' AND name = 'archived_events'"
|
||||
|
||||
// ArchivePath returns where a database target's archive file is: in
|
||||
// the data directory, beside the webhook's event database, under the
|
||||
// name ArchiveFileName gives it.
|
||||
func ArchivePath(
|
||||
dbMgr *database.WebhookDBManager,
|
||||
webhook *database.Webhook,
|
||||
target *database.Target,
|
||||
) string {
|
||||
return filepath.Join(
|
||||
filepath.Dir(dbMgr.DBPath(webhook.ID)),
|
||||
ArchiveFileName(webhook.Name, target.Name, target.ID),
|
||||
)
|
||||
}
|
||||
|
||||
// ArchiveExportFileName returns the name a database target's archive
|
||||
// downloads under:
|
||||
// archive-WEBHOOKNAME-TARGETNAME-YYYYMMDDTHHMMSSZ.json.gz, the names
|
||||
// made safe as in ArchiveFileName and the time in UTC.
|
||||
func ArchiveExportFileName(
|
||||
webhookName, targetName string, at time.Time,
|
||||
) string {
|
||||
return "archive-" + archiveNamePart(webhookName) + "-" +
|
||||
archiveNamePart(targetName) + "-" +
|
||||
at.UTC().Format("20060102T150405Z") + ".json.gz"
|
||||
}
|
||||
|
||||
// ArchiveExport is a database target's archive opened for download.
|
||||
// It reads the file on its own connection, inside one read-only
|
||||
// transaction, so it writes out the archive as it stood when
|
||||
// OpenArchiveExport returned.
|
||||
//
|
||||
// Archives are in WAL mode, where a reader works from a snapshot and
|
||||
// never blocks a writer: archive writes go on while an export is open,
|
||||
// and the export does not see them. SQLite cannot checkpoint the -wal
|
||||
// past an open snapshot, so the -wal grows until the export is closed.
|
||||
type ArchiveExport struct {
|
||||
db *sql.DB
|
||||
tx *gorm.DB
|
||||
|
||||
// empty is true when there is nothing to read: no file, or a file
|
||||
// without the archive's table yet.
|
||||
empty bool
|
||||
}
|
||||
|
||||
// exportedName is how an export names its webhook and its target.
|
||||
type exportedName struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
}
|
||||
|
||||
// OpenArchiveExport opens the archive file at path for export and
|
||||
// takes the snapshot the export reads. It never creates the file: with
|
||||
// no file at path, the export has no rows.
|
||||
//
|
||||
// Once it has returned, the file is open, so a rename or a move of it
|
||||
// does not affect the export, which reads the same file under its new
|
||||
// name.
|
||||
//
|
||||
// The transaction lasts as long as ctx does, so ctx must last for the
|
||||
// whole export.
|
||||
func OpenArchiveExport(
|
||||
ctx context.Context, path string, log *slog.Logger,
|
||||
) (*ArchiveExport, error) {
|
||||
if !fileExists(path) {
|
||||
return &ArchiveExport{empty: true}, nil
|
||||
}
|
||||
|
||||
db, err := database.OpenSQLite(path, archiveModeExisting)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("opening archive %s: %w", path, err)
|
||||
}
|
||||
|
||||
gdb, err := gorm.Open(
|
||||
sqlite.Dialector{Conn: db}, &gorm.Config{
|
||||
// Never leave this at GORM's default. See
|
||||
// internal/gormlog.
|
||||
Logger: gormlog.New(log),
|
||||
},
|
||||
)
|
||||
if err != nil {
|
||||
_ = db.Close()
|
||||
|
||||
return nil, fmt.Errorf("opening archive %s: %w", path, err)
|
||||
}
|
||||
|
||||
// ReadOnly makes the driver begin a deferred transaction in place
|
||||
// of the BEGIN IMMEDIATE the connection string asks for, so the
|
||||
// export never takes the archive's write lock.
|
||||
tx := gdb.WithContext(ctx).Begin(&sql.TxOptions{ReadOnly: true})
|
||||
if tx.Error != nil {
|
||||
_ = db.Close()
|
||||
|
||||
return nil, fmt.Errorf("reading archive %s: %w", path, tx.Error)
|
||||
}
|
||||
|
||||
// The transaction's first read is what takes the snapshot.
|
||||
var tables int
|
||||
|
||||
err = tx.Raw(archiveTableQuery).Row().Scan(&tables)
|
||||
if err != nil {
|
||||
_ = tx.Rollback()
|
||||
_ = db.Close()
|
||||
|
||||
return nil, fmt.Errorf("reading archive %s: %w", path, err)
|
||||
}
|
||||
|
||||
return &ArchiveExport{db: db, tx: tx, empty: tables == 0}, nil
|
||||
}
|
||||
|
||||
// WriteGzipJSON writes the export to w as one gzipped JSON object:
|
||||
// webhook and target, each an id and a name; exported_at; and
|
||||
// archived_events, one object per archived row, keyed by column name.
|
||||
// A body that is not valid UTF-8 cannot be a JSON string, so it is
|
||||
// written in base64, with "body_encoding": "base64" beside it.
|
||||
//
|
||||
// Each row is written out before the next is read, so neither the
|
||||
// archive nor its JSON is ever held in memory whole. After an error
|
||||
// the gzip stream is left unfinished, so what was written does not
|
||||
// decompress as a whole file.
|
||||
func (x *ArchiveExport) WriteGzipJSON(
|
||||
ctx context.Context,
|
||||
w io.Writer,
|
||||
webhook *database.Webhook,
|
||||
target *database.Target,
|
||||
exportedAt time.Time,
|
||||
) error {
|
||||
head, err := json.Marshal(map[string]any{
|
||||
"webhook": exportedName{ID: webhook.ID, Name: webhook.Name},
|
||||
"target": exportedName{ID: target.ID, Name: target.Name},
|
||||
"exported_at": exportedAt.UTC(),
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("encoding archive export: %w", err)
|
||||
}
|
||||
|
||||
zw := gzip.NewWriter(w)
|
||||
|
||||
err = x.writeJSON(ctx, zw, head)
|
||||
if err != nil {
|
||||
return fmt.Errorf("writing archive export: %w", err)
|
||||
}
|
||||
|
||||
return zw.Close()
|
||||
}
|
||||
|
||||
// Close ends the export's transaction and closes its connection.
|
||||
func (x *ArchiveExport) Close() error {
|
||||
if x.db == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
_ = x.tx.Rollback()
|
||||
|
||||
return x.db.Close()
|
||||
}
|
||||
|
||||
// writeJSON writes head with archived_events added as its last key,
|
||||
// the rows going into it one at a time.
|
||||
func (x *ArchiveExport) writeJSON(
|
||||
ctx context.Context, w io.Writer, head []byte,
|
||||
) error {
|
||||
// head goes out without its closing brace, so that
|
||||
// archived_events can follow it.
|
||||
_, err := w.Write(head[:len(head)-1])
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
_, err = io.WriteString(w, `,"archived_events":[`)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
err = x.writeRows(ctx, w)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
_, err = io.WriteString(w, "\n]}\n")
|
||||
|
||||
return err
|
||||
}
|
||||
|
||||
// writeRows writes each archived row to w, oldest first, one per line,
|
||||
// separated by commas.
|
||||
func (x *ArchiveExport) writeRows(ctx context.Context, w io.Writer) error {
|
||||
if x.empty {
|
||||
return nil
|
||||
}
|
||||
|
||||
rows, err := x.tx.WithContext(ctx).
|
||||
Model(&archivedEvent{}).Order("id").Rows()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
for sep := "\n"; rows.Next(); sep = ",\n" {
|
||||
var ev archivedEvent
|
||||
|
||||
err = x.tx.ScanRows(rows, &ev)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
_, err = io.WriteString(w, sep)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
err = writeRow(w, &ev)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return rows.Err()
|
||||
}
|
||||
|
||||
// writeRow writes an archived row to w as a JSON object keyed by
|
||||
// column name, its body in base64 when it is not valid UTF-8.
|
||||
func writeRow(w io.Writer, ev *archivedEvent) error {
|
||||
row := map[string]any{
|
||||
"id": ev.ID,
|
||||
"event_id": ev.EventID,
|
||||
"webhook_id": ev.WebhookID,
|
||||
"entrypoint_id": ev.EntrypointID,
|
||||
"method": ev.Method,
|
||||
"headers": ev.Headers,
|
||||
"body": ev.Body,
|
||||
"content_type": ev.ContentType,
|
||||
"archived_at": ev.ArchivedAt.UTC(),
|
||||
}
|
||||
|
||||
if !utf8.ValidString(ev.Body) {
|
||||
row["body"] = base64.StdEncoding.EncodeToString([]byte(ev.Body))
|
||||
row["body_encoding"] = "base64"
|
||||
}
|
||||
|
||||
line, err := json.Marshal(row)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
_, err = w.Write(line)
|
||||
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,412 @@
|
||||
package delivery_test
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"crypto/rand"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
)
|
||||
|
||||
// The webhook and the target the export tests' archives belong to.
|
||||
const (
|
||||
exportWebhookID = "wh-export"
|
||||
exportWebhookName = "Orders (EU)"
|
||||
exportTargetID = "tgt-export"
|
||||
exportTargetName = "Long-term archive"
|
||||
)
|
||||
|
||||
const (
|
||||
// binaryBody is a body that is not valid UTF-8.
|
||||
binaryBody = "\xff\xfe\x00\x01binary\x80"
|
||||
|
||||
// openedEventID is the event the snapshot tests archive before
|
||||
// they open the export.
|
||||
openedEventID = "opened"
|
||||
)
|
||||
|
||||
// writeExportTo writes export to w as the archive of the export tests'
|
||||
// webhook and target, exported at 2026-10-02T12:03:04Z.
|
||||
func writeExportTo(
|
||||
t *testing.T, export *delivery.ArchiveExport, w io.Writer,
|
||||
) error {
|
||||
t.Helper()
|
||||
|
||||
return export.WriteGzipJSON(
|
||||
t.Context(), w,
|
||||
&database.Webhook{
|
||||
BaseModel: database.BaseModel{ID: exportWebhookID},
|
||||
Name: exportWebhookName,
|
||||
},
|
||||
&database.Target{
|
||||
BaseModel: database.BaseModel{ID: exportTargetID},
|
||||
Name: exportTargetName,
|
||||
},
|
||||
time.Date(2026, 10, 2, 12, 3, 4, 0, time.UTC),
|
||||
)
|
||||
}
|
||||
|
||||
// exportArchive runs a whole export of the archive at path and returns
|
||||
// its JSON, decompressed and parsed.
|
||||
func exportArchive(t *testing.T, path string) map[string]any {
|
||||
t.Helper()
|
||||
|
||||
export, err := delivery.OpenArchiveExport(
|
||||
t.Context(), path, archiveTestLogger(),
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
defer func() { require.NoError(t, export.Close()) }()
|
||||
|
||||
return writeExport(t, export)
|
||||
}
|
||||
|
||||
// writeExport writes an opened export and returns its JSON,
|
||||
// decompressed and parsed. Reading to the end makes the gzip reader
|
||||
// check that the stream was finished.
|
||||
func writeExport(
|
||||
t *testing.T, export *delivery.ArchiveExport,
|
||||
) map[string]any {
|
||||
t.Helper()
|
||||
|
||||
var buf bytes.Buffer
|
||||
|
||||
require.NoError(t, writeExportTo(t, export, &buf))
|
||||
|
||||
zr, err := gzip.NewReader(&buf)
|
||||
require.NoError(t, err)
|
||||
|
||||
raw, err := io.ReadAll(zr)
|
||||
require.NoError(t, err)
|
||||
|
||||
var got map[string]any
|
||||
|
||||
require.NoError(t, json.Unmarshal(raw, &got))
|
||||
|
||||
return got
|
||||
}
|
||||
|
||||
// exportedEvents returns an export's archived_events.
|
||||
func exportedEvents(t *testing.T, got map[string]any) []map[string]any {
|
||||
t.Helper()
|
||||
|
||||
list, ok := got["archived_events"].([]any)
|
||||
require.True(t, ok, "archived_events must be an array: %v", got)
|
||||
|
||||
events := make([]map[string]any, len(list))
|
||||
|
||||
for i, v := range list {
|
||||
events[i], ok = v.(map[string]any)
|
||||
require.True(t, ok, "an archived event must be an object: %v", v)
|
||||
}
|
||||
|
||||
return events
|
||||
}
|
||||
|
||||
// exportedEventIDs returns the event_id of each of an export's
|
||||
// archived_events.
|
||||
func exportedEventIDs(t *testing.T, got map[string]any) []string {
|
||||
t.Helper()
|
||||
|
||||
events := exportedEvents(t, got)
|
||||
ids := make([]string, 0, len(events))
|
||||
|
||||
for _, ev := range events {
|
||||
ids = append(ids, fmt.Sprint(ev["event_id"]))
|
||||
}
|
||||
|
||||
return ids
|
||||
}
|
||||
|
||||
// TestArchiveExport_MatchesStoredRows proves an export holds the
|
||||
// webhook, the target, the time, and every column of every stored
|
||||
// row: a body that is valid UTF-8 as a string, and one that is not in
|
||||
// base64, marked as such.
|
||||
func TestArchiveExport_MatchesStoredRows(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "archive.db")
|
||||
w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
|
||||
bodies := []string{`{"order":1}`, "plain text", "", binaryBody}
|
||||
|
||||
for i, body := range bodies {
|
||||
require.NoError(t, w.Write(delivery.ExportArchivedEvent{
|
||||
EventID: fmt.Sprintf("ev-%d", i),
|
||||
WebhookID: exportWebhookID,
|
||||
EntrypointID: "ep-1",
|
||||
Method: "POST",
|
||||
Headers: `{"X-Test":["yes"]}`,
|
||||
Body: body,
|
||||
ContentType: testContentType,
|
||||
}, 0))
|
||||
}
|
||||
|
||||
var stored []delivery.ExportArchivedEvent
|
||||
|
||||
require.NoError(t, openArchiveDBForRead(t, path).
|
||||
Order("id").Find(&stored).Error)
|
||||
|
||||
got := exportArchive(t, path)
|
||||
|
||||
assert.Equal(t,
|
||||
map[string]any{"id": exportWebhookID, "name": exportWebhookName},
|
||||
got["webhook"],
|
||||
)
|
||||
assert.Equal(t,
|
||||
map[string]any{"id": exportTargetID, "name": exportTargetName},
|
||||
got["target"],
|
||||
)
|
||||
assert.Equal(t, "2026-10-02T12:03:04Z", got["exported_at"])
|
||||
|
||||
events := exportedEvents(t, got)
|
||||
require.Len(t, events, len(bodies))
|
||||
|
||||
for i, row := range stored {
|
||||
assertExportedRow(t, row, events[i])
|
||||
}
|
||||
}
|
||||
|
||||
// assertExportedRow checks that ev, from an export, holds every column
|
||||
// of the stored row.
|
||||
func assertExportedRow(
|
||||
t *testing.T, row delivery.ExportArchivedEvent, ev map[string]any,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
archivedAt, err := time.Parse(
|
||||
time.RFC3339Nano, fmt.Sprint(ev["archived_at"]),
|
||||
)
|
||||
require.NoError(t, err)
|
||||
assert.True(t, archivedAt.Equal(row.ArchivedAt))
|
||||
|
||||
assert.EqualValues(t, row.ID, ev["id"])
|
||||
assert.Equal(t, row.EventID, ev["event_id"])
|
||||
assert.Equal(t, row.WebhookID, ev["webhook_id"])
|
||||
assert.Equal(t, row.EntrypointID, ev["entrypoint_id"])
|
||||
assert.Equal(t, row.Method, ev["method"])
|
||||
assert.Equal(t, row.Headers, ev["headers"])
|
||||
assert.Equal(t, row.ContentType, ev["content_type"])
|
||||
|
||||
if row.Body != binaryBody {
|
||||
assert.Equal(t, row.Body, ev["body"])
|
||||
assert.Len(t, ev, 9, "the nine columns and nothing else: %v", ev)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
body, err := base64.StdEncoding.DecodeString(fmt.Sprint(ev["body"]))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, binaryBody, string(body))
|
||||
assert.Equal(t, "base64", ev["body_encoding"])
|
||||
assert.Len(t, ev, 10, "the nine columns and body_encoding: %v", ev)
|
||||
}
|
||||
|
||||
// TestArchiveExport_Empty proves an archive with nothing in it exports
|
||||
// as an empty archived_events: no file, which the export must not
|
||||
// create; a file the archive writer has not yet put its table in; and
|
||||
// a table with no rows.
|
||||
func TestArchiveExport_Empty(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dir := t.TempDir()
|
||||
missing := filepath.Join(dir, "missing.db")
|
||||
noTable := filepath.Join(dir, "no-table.db")
|
||||
noRows := filepath.Join(dir, "no-rows.db")
|
||||
|
||||
require.NoError(t, os.WriteFile(noTable, nil, 0o600))
|
||||
require.NoError(t,
|
||||
delivery.NewExportArchiveWriter(noRows, archiveTestLogger(), 0).
|
||||
Open(0),
|
||||
)
|
||||
|
||||
for _, path := range []string{missing, noTable, noRows} {
|
||||
assert.Empty(t, exportedEvents(t, exportArchive(t, path)), path)
|
||||
}
|
||||
|
||||
for _, suffix := range archiveFileSuffixes() {
|
||||
assert.NoFileExists(t, missing+suffix)
|
||||
}
|
||||
}
|
||||
|
||||
// TestArchiveExport_ReadsOneSnapshot proves an export writes the
|
||||
// archive as it was when it was opened, and holds up no archive
|
||||
// write: a row written while the export is open is stored, and is not
|
||||
// in the export. A write held up for the whole busy timeout would
|
||||
// fail.
|
||||
func TestArchiveExport_ReadsOneSnapshot(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "archive.db")
|
||||
w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
|
||||
|
||||
require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: openedEventID}, 0))
|
||||
|
||||
export, err := delivery.OpenArchiveExport(
|
||||
t.Context(), path, archiveTestLogger(),
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
defer func() { require.NoError(t, export.Close()) }()
|
||||
|
||||
require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: "during"}, 0))
|
||||
|
||||
assert.Equal(t,
|
||||
[]string{openedEventID}, exportedEventIDs(t, writeExport(t, export)),
|
||||
)
|
||||
|
||||
var stored int64
|
||||
|
||||
require.NoError(t, openArchiveDBForRead(t, path).
|
||||
Model(&delivery.ExportArchivedEvent{}).Count(&stored).Error)
|
||||
assert.Equal(t, int64(2), stored)
|
||||
}
|
||||
|
||||
// TestArchiveExport_SurvivesRename proves that renaming the archive
|
||||
// while an export of it is open, as renaming its webhook or target
|
||||
// does, leaves the export reading the same file.
|
||||
func TestArchiveExport_SurvivesRename(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "archive-old.db")
|
||||
w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
|
||||
|
||||
require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: openedEventID}, 0))
|
||||
|
||||
export, err := delivery.OpenArchiveExport(
|
||||
t.Context(), path, archiveTestLogger(),
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
defer func() { require.NoError(t, export.Close()) }()
|
||||
|
||||
require.NoError(t, w.Rename("archive-new.db"))
|
||||
require.NoError(t, w.Write(delivery.ExportArchivedEvent{EventID: "after"}, 0))
|
||||
require.NoFileExists(t, path)
|
||||
|
||||
assert.Equal(t,
|
||||
[]string{openedEventID}, exportedEventIDs(t, writeExport(t, export)),
|
||||
)
|
||||
}
|
||||
|
||||
// heapPeak is an io.Writer that discards what it is given and records
|
||||
// the largest heap it saw at a write. It collects garbage before each
|
||||
// reading, so the heap it reads is what is still held.
|
||||
type heapPeak struct {
|
||||
max uint64
|
||||
}
|
||||
|
||||
func (p *heapPeak) Write(b []byte) (int, error) {
|
||||
var m runtime.MemStats
|
||||
|
||||
runtime.GC()
|
||||
runtime.ReadMemStats(&m)
|
||||
p.max = max(p.max, m.HeapAlloc)
|
||||
|
||||
return len(b), nil
|
||||
}
|
||||
|
||||
// exportHeapGrowth exports an archive of rows random bodies, each
|
||||
// bodySize bytes of base64, and returns how far the heap rose above
|
||||
// where it stood when the export began, at its highest.
|
||||
func exportHeapGrowth(t *testing.T, rows, bodySize int) uint64 {
|
||||
t.Helper()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "archive.db")
|
||||
w := delivery.NewExportArchiveWriter(path, archiveTestLogger(), 0)
|
||||
|
||||
// Base64 makes four characters of every three bytes.
|
||||
random := make([]byte, bodySize/4*3)
|
||||
|
||||
for range rows {
|
||||
_, _ = rand.Read(random)
|
||||
|
||||
require.NoError(t, w.Write(delivery.ExportArchivedEvent{
|
||||
Body: base64.StdEncoding.EncodeToString(random),
|
||||
}, 0))
|
||||
}
|
||||
|
||||
export, err := delivery.OpenArchiveExport(
|
||||
t.Context(), path, archiveTestLogger(),
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
defer func() { require.NoError(t, export.Close()) }()
|
||||
|
||||
runtime.GC()
|
||||
|
||||
var start runtime.MemStats
|
||||
|
||||
runtime.ReadMemStats(&start)
|
||||
|
||||
// Through a buffer, the heap is read once per 8 KiB of output
|
||||
// rather than at each of gzip's small writes, which takes far
|
||||
// longer.
|
||||
peak := &heapPeak{max: start.HeapAlloc}
|
||||
buffered := bufio.NewWriterSize(peak, 8<<10)
|
||||
|
||||
require.NoError(t, writeExportTo(t, export, buffered))
|
||||
require.NoError(t, buffered.Flush())
|
||||
|
||||
return peak.max - start.HeapAlloc
|
||||
}
|
||||
|
||||
// TestArchiveExport_Streams proves an export holds neither the archive
|
||||
// nor its output in memory whole: exporting 384 KiB more of archive
|
||||
// raises the heap's peak by less than half of that. The export's own
|
||||
// memory, mostly gzip's compressor, is the same for both archives, so
|
||||
// it cancels out. The bodies are random bytes in base64, which gzip
|
||||
// shrinks by only a quarter, so an export that read every row before
|
||||
// writing, or built the JSON or the gzipped file before writing it,
|
||||
// would raise the peak by at least three quarters of the difference.
|
||||
//
|
||||
// The smaller archive has two rows so that its export, too, writes
|
||||
// out more than the 8 KiB buffer in exportHeapGrowth before it ends:
|
||||
// the heap must be read while the export's own memory is held.
|
||||
//
|
||||
//nolint:paralleltest // It measures the heap, which tests share.
|
||||
func TestArchiveExport_Streams(t *testing.T) {
|
||||
const (
|
||||
bodySize = 16 << 10
|
||||
smallRows = 2
|
||||
largeRows = smallRows + 24
|
||||
limit = (largeRows - smallRows) * bodySize / 2
|
||||
)
|
||||
|
||||
small := exportHeapGrowth(t, smallRows, bodySize)
|
||||
large := exportHeapGrowth(t, largeRows, bodySize)
|
||||
|
||||
assert.Less(t, large, small+limit,
|
||||
"the heap rose by %d for %d rows and by %d for %d rows",
|
||||
small, smallRows, large, largeRows,
|
||||
)
|
||||
}
|
||||
|
||||
// TestArchiveExportFileName proves the download is named for the
|
||||
// webhook and the target, with the names made safe as for the archive
|
||||
// file, and the export time in UTC.
|
||||
func TestArchiveExportFileName(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cest := time.FixedZone("CEST", int((2 * time.Hour).Seconds()))
|
||||
|
||||
assert.Equal(t,
|
||||
"archive-orders-eu-long-term-archive-20261002T120304Z.json.gz",
|
||||
delivery.ArchiveExportFileName(
|
||||
exportWebhookName, exportTargetName,
|
||||
time.Date(2026, 10, 2, 14, 3, 4, 0, cest),
|
||||
),
|
||||
)
|
||||
}
|
||||
@@ -17,6 +17,7 @@ import (
|
||||
_ "modernc.org/sqlite" // Pure Go SQLite driver.
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
||||
)
|
||||
|
||||
func archiveTestLogger() *slog.Logger {
|
||||
@@ -42,7 +43,8 @@ func openArchiveDBForRead(
|
||||
t.Cleanup(func() { _ = sqlDB.Close() })
|
||||
|
||||
gdb, err := gorm.Open(
|
||||
sqlite.Dialector{Conn: sqlDB}, &gorm.Config{},
|
||||
sqlite.Dialector{Conn: sqlDB},
|
||||
&gorm.Config{Logger: gormlog.New(slog.New(slog.DiscardHandler))},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
|
||||
@@ -111,9 +111,9 @@ func (l *Logger) LogMode(gormlogger.LogLevel) gormlogger.Interface {
|
||||
//
|
||||
// One GORM path does not consult this: (*gorm.DB).Scan records the
|
||||
// statement through gorm's own traceRecorder, which does not implement
|
||||
// this interface. No production code path calls it; its one caller is
|
||||
// internal/database/database_test.go:91, whose SELECT 1 binds nothing.
|
||||
// scan_guard_test.go fails if a non-test file calls it.
|
||||
// this interface. No production code path calls it; only tests do, and
|
||||
// what a test binds is fixture data. scan_guard_test.go fails if a
|
||||
// non-test file calls it.
|
||||
// (*gorm.DB).Pluck, Row and Raw all run through the normal callback
|
||||
// processor and are filtered.
|
||||
func (l *Logger) ParamsFilter(
|
||||
|
||||
@@ -1,95 +0,0 @@
|
||||
package handlers_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/go-chi/chi"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/gorm"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
)
|
||||
|
||||
// TestHandleEntrypointToggle_DoesNotUndoAnEdit proves that a toggle
|
||||
// which loaded the entrypoint before an edit of its description was
|
||||
// saved does not write the old description back over the edit. The
|
||||
// edit is submitted from a callback on the toggle's own read of the
|
||||
// entrypoint, so it is saved after that read and before the toggle
|
||||
// writes.
|
||||
func TestHandleEntrypointToggle_DoesNotUndoAnEdit(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupSourceTest(t)
|
||||
wh := seedWebhookWithRetention(t, env.db, 30)
|
||||
ep := seedEntrypoint(t, env.db, wh.ID)
|
||||
require.True(t, ep.Active)
|
||||
|
||||
router := chi.NewRouter()
|
||||
router.Post(
|
||||
"/hook/{sourceID}/entrypoints/{entrypointID}/edit",
|
||||
env.handlers.HandleEntrypointEdit(),
|
||||
)
|
||||
router.Post(
|
||||
"/hook/{sourceID}/entrypoints/{entrypointID}/toggle",
|
||||
env.handlers.HandleEntrypointToggle(),
|
||||
)
|
||||
|
||||
// post submits one of the entrypoint's forms as the test user and
|
||||
// returns the response's status code.
|
||||
post := func(action string, form url.Values) int {
|
||||
req := httptest.NewRequestWithContext(
|
||||
context.Background(), http.MethodPost,
|
||||
"/hook/"+wh.ID+"/entrypoints/"+ep.ID+"/"+action,
|
||||
strings.NewReader(form.Encode()),
|
||||
)
|
||||
req.Header.Set(
|
||||
"Content-Type", "application/x-www-form-urlencoded",
|
||||
)
|
||||
|
||||
for _, c := range env.cookies {
|
||||
req.AddCookie(c)
|
||||
}
|
||||
|
||||
w := httptest.NewRecorder()
|
||||
router.ServeHTTP(w, req)
|
||||
|
||||
return w.Code
|
||||
}
|
||||
|
||||
var (
|
||||
edited bool
|
||||
editCode int
|
||||
)
|
||||
|
||||
require.NoError(t, env.db.DB().Callback().Query().
|
||||
After("gorm:query").
|
||||
Register("test:edit_after_toggle_read", func(tx *gorm.DB) {
|
||||
// Only the first read of an entrypoint, the toggle's,
|
||||
// submits the edit.
|
||||
if tx.Statement.Table != "entrypoints" || edited {
|
||||
return
|
||||
}
|
||||
|
||||
edited = true
|
||||
editCode = post(
|
||||
"edit", url.Values{"description": {"Billing sender"}},
|
||||
)
|
||||
}),
|
||||
)
|
||||
|
||||
require.Equal(t, http.StatusSeeOther, post("toggle", nil))
|
||||
require.Equal(t, http.StatusSeeOther, editCode)
|
||||
|
||||
var stored database.Entrypoint
|
||||
|
||||
require.NoError(
|
||||
t, env.db.DB().First(&stored, "id = ?", ep.ID).Error,
|
||||
)
|
||||
assert.False(t, stored.Active)
|
||||
assert.Equal(t, "Billing sender", stored.Description)
|
||||
}
|
||||
@@ -24,9 +24,10 @@ import (
|
||||
const eventBodyQuery = "SELECT cast(body as blob) " +
|
||||
"FROM events WHERE id = ? AND webhook_id = ? AND deleted_at IS NULL"
|
||||
|
||||
// HandleEventBodyDownload serves one event's stored body in
|
||||
// full, which the event log page cannot: it caps each rendered
|
||||
// body at maxRenderedBodyBytes.
|
||||
// HandleEventBodyDownload serves one event's stored body byte
|
||||
// for byte, which the pages do not: they show it as escaped
|
||||
// text, cut at maxRenderedBodyBytes in the lists of events, and
|
||||
// leave a binary one out.
|
||||
//
|
||||
// The bytes are attacker-supplied — anyone who can reach the
|
||||
// public receiver chooses them — and this route hands them back
|
||||
|
||||
@@ -0,0 +1,150 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
// maxRenderedBodyBytes is the most of one event's body that the
|
||||
// recent events on a webhook's page and the event log show; a larger
|
||||
// body is cut there and shown whole only on the event's own page.
|
||||
// Bodies come from the unauthenticated receiver under its 1 MB cap,
|
||||
// and renderTemplate buffers a whole page before writing it, so a list
|
||||
// of events cannot show every body whole.
|
||||
const maxRenderedBodyBytes = 32 << 10
|
||||
|
||||
// maxInlineBodyLines is the most lines a body is shown at its full
|
||||
// height with. A body with more lines, or larger than
|
||||
// maxRenderedBodyBytes, is shown in a box of fixed height that
|
||||
// scrolls, so that it does not make the page huge.
|
||||
const maxInlineBodyLines = 200
|
||||
|
||||
// maxIndentDepth is how deeply a JSON body's objects and arrays may
|
||||
// nest for it to be pretty-printed; a deeper one is shown as received.
|
||||
// Each level indents every line inside it two more spaces, so 10 KB of
|
||||
// nested brackets would indent to some 50 MB; within this depth a body
|
||||
// grows at most 35 times.
|
||||
const maxIndentDepth = 16
|
||||
|
||||
// jsonIndent is the indent of a pretty-printed JSON body.
|
||||
const jsonIndent = " "
|
||||
|
||||
// BodyView is an event's body as the pages show it. newBodyView
|
||||
// decides it and templates/event_body.html shows it, the same way in
|
||||
// the recent events on a webhook's page, in the event log and on the
|
||||
// event's own page.
|
||||
type BodyView struct {
|
||||
// EventURL is the event's own page. The stored body downloads
|
||||
// from EventURL/body.
|
||||
EventURL string
|
||||
|
||||
// Text is the body as shown, pretty-printed when it is JSON.
|
||||
Text string
|
||||
|
||||
// Size is the stored body's size in bytes, and ShownBytes how
|
||||
// many of them Text holds when Cut.
|
||||
Size int64
|
||||
ShownBytes int
|
||||
|
||||
// Cut reports that Text is only the start of the body.
|
||||
Cut bool
|
||||
|
||||
// Binary reports a body that is not text. It is not shown.
|
||||
Binary bool
|
||||
|
||||
// Scroll reports a body to show in a box that scrolls.
|
||||
Scroll bool
|
||||
}
|
||||
|
||||
// newBodyView decides how to show an event's body. body is the
|
||||
// stored body, or its first maxRenderedBodyBytes when only those were
|
||||
// read, and size is the stored body's size.
|
||||
func newBodyView(eventURL string, body []byte, size int64) BodyView {
|
||||
v := BodyView{EventURL: eventURL, Size: size}
|
||||
|
||||
if size > int64(len(body)) {
|
||||
v.Cut = true
|
||||
body = trimPartialRune(body)
|
||||
v.ShownBytes = len(body)
|
||||
}
|
||||
|
||||
// html/template shows invalid UTF-8 as replacement characters,
|
||||
// and a browser shows a control character other than tab, line
|
||||
// feed and carriage return as a box or not at all, so a body
|
||||
// holding either is not text.
|
||||
isControl := func(r rune) bool {
|
||||
return unicode.IsControl(r) && r != '\t' && r != '\n' && r != '\r'
|
||||
}
|
||||
|
||||
if !utf8.Valid(body) || bytes.IndexFunc(body, isControl) >= 0 {
|
||||
v.Binary = true
|
||||
|
||||
return v
|
||||
}
|
||||
|
||||
// A cut JSON document is no longer valid JSON.
|
||||
if !v.Cut {
|
||||
body = indentJSON(body)
|
||||
}
|
||||
|
||||
// A final newline ends the last line rather than starting another.
|
||||
lines := bytes.Count(bytes.TrimSuffix(body, []byte("\n")), []byte("\n")) + 1
|
||||
|
||||
v.Text = string(body)
|
||||
v.Scroll = lines > maxInlineBodyLines || size > maxRenderedBodyBytes
|
||||
|
||||
return v
|
||||
}
|
||||
|
||||
// indentJSON returns body pretty-printed when it is a JSON document,
|
||||
// and unchanged when it is not or nests deeper than maxIndentDepth.
|
||||
func indentJSON(body []byte) []byte {
|
||||
if !json.Valid(body) || !indentFits(body) {
|
||||
return body
|
||||
}
|
||||
|
||||
var out bytes.Buffer
|
||||
|
||||
err := json.Indent(&out, body, "", jsonIndent)
|
||||
if err != nil {
|
||||
return body
|
||||
}
|
||||
|
||||
return out.Bytes()
|
||||
}
|
||||
|
||||
// indentFits reports whether the objects and arrays of the JSON
|
||||
// document body nest at most maxIndentDepth deep.
|
||||
func indentFits(body []byte) bool {
|
||||
depth := 0
|
||||
|
||||
dec := json.NewDecoder(bytes.NewReader(body))
|
||||
|
||||
// A number too large for a float64 is still valid JSON.
|
||||
dec.UseNumber()
|
||||
|
||||
for {
|
||||
tok, err := dec.Token()
|
||||
if errors.Is(err, io.EOF) {
|
||||
return true
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
|
||||
switch tok {
|
||||
case json.Delim('{'), json.Delim('['):
|
||||
depth++
|
||||
if depth > maxIndentDepth {
|
||||
return false
|
||||
}
|
||||
case json.Delim('}'), json.Delim(']'):
|
||||
depth--
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,155 @@
|
||||
package handlers_test
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"sneak.berlin/go/webhooker/internal/handlers"
|
||||
)
|
||||
|
||||
// bodyView is how the pages would show body, stored whole.
|
||||
func bodyView(body string) handlers.BodyView {
|
||||
return handlers.NewBodyViewForTest([]byte(body), int64(len(body)))
|
||||
}
|
||||
|
||||
// lines is n lines of text, without a newline after the last.
|
||||
func lines(n int) string {
|
||||
return strings.TrimSuffix(strings.Repeat("line\n", n), "\n")
|
||||
}
|
||||
|
||||
// TestNewBodyView_FormatsValidJSON proves a JSON body is shown
|
||||
// pretty-printed, whatever its content type, with its keys in
|
||||
// the order they arrived.
|
||||
func TestNewBodyView_FormatsValidJSON(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
v := bodyView(`{"b":1,"a":[true,null,"x"],"c":{}}`)
|
||||
|
||||
assert.Equal(t, []string{
|
||||
`{`,
|
||||
` "b": 1,`,
|
||||
` "a": [`,
|
||||
` true,`,
|
||||
` null,`,
|
||||
` "x"`,
|
||||
` ],`,
|
||||
` "c": {}`,
|
||||
`}`,
|
||||
}, strings.Split(v.Text, "\n"))
|
||||
assert.False(t, v.Scroll)
|
||||
}
|
||||
|
||||
// TestNewBodyView_FormatsNestedJSON proves a document with a few
|
||||
// levels of nesting is pretty-printed: only deep nesting is shown
|
||||
// as received.
|
||||
func TestNewBodyView_FormatsNestedJSON(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
v := bodyView(`{"data":[[1,2,3],[4,5,6]]}`)
|
||||
|
||||
assert.Equal(t, []string{
|
||||
`{`,
|
||||
` "data": [`,
|
||||
` [`,
|
||||
` 1,`,
|
||||
` 2,`,
|
||||
` 3`,
|
||||
` ],`,
|
||||
` [`,
|
||||
` 4,`,
|
||||
` 5,`,
|
||||
` 6`,
|
||||
` ]`,
|
||||
` ]`,
|
||||
`}`,
|
||||
}, strings.Split(v.Text, "\n"))
|
||||
}
|
||||
|
||||
// TestNewBodyView_InvalidJSONAsReceived proves a body that is not
|
||||
// a JSON document is shown exactly as it arrived.
|
||||
func TestNewBodyView_InvalidJSONAsReceived(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, body := range []string{
|
||||
`{"a":1,`,
|
||||
`{"a":1} {"b":2}`,
|
||||
"plain text\n indented",
|
||||
} {
|
||||
assert.Equal(t, body, bodyView(body).Text)
|
||||
}
|
||||
}
|
||||
|
||||
// TestNewBodyView_DeepJSONAsReceived proves a JSON body nested
|
||||
// more than 16 levels deep is shown as it arrived. 10 KB of nested
|
||||
// arrays would indent to some 50 MB.
|
||||
func TestNewBodyView_DeepJSONAsReceived(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
nested := func(depth int) string {
|
||||
return strings.Repeat("[", depth) + "1" + strings.Repeat("]", depth)
|
||||
}
|
||||
|
||||
assert.NotEqual(t, nested(16), bodyView(nested(16)).Text)
|
||||
assert.Equal(t, nested(17), bodyView(nested(17)).Text)
|
||||
|
||||
body := strings.Repeat("[", 5000) + strings.Repeat("]", 5000)
|
||||
|
||||
assert.Equal(t, body, bodyView(body).Text)
|
||||
}
|
||||
|
||||
// TestNewBodyView_ScrollsPast200Lines proves a body is shown at
|
||||
// its full height up to 200 lines and in the scrolling box past
|
||||
// them, counting the lines after formatting.
|
||||
func TestNewBodyView_ScrollsPast200Lines(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
assert.False(t, bodyView(lines(200)).Scroll)
|
||||
assert.True(t, bodyView(lines(201)).Scroll)
|
||||
|
||||
// A final newline ends the last line rather than starting another.
|
||||
assert.False(t, bodyView(lines(200)+"\n").Scroll)
|
||||
assert.True(t, bodyView(lines(201)+"\n").Scroll)
|
||||
|
||||
// One line as received, 201 once formatted: the brackets and
|
||||
// 199 elements.
|
||||
numbers := "[" + strings.TrimSuffix(strings.Repeat("1,", 199), ",") + "]"
|
||||
|
||||
assert.NotContains(t, numbers, "\n")
|
||||
assert.True(t, bodyView(numbers).Scroll)
|
||||
}
|
||||
|
||||
// TestNewBodyView_LargeBodyScrolls proves a body larger than the
|
||||
// cap of the lists of events is shown in the scrolling box
|
||||
// however few lines it has, on the event's own page as in the
|
||||
// lists.
|
||||
func TestNewBodyView_LargeBodyScrolls(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
assert.False(t, bodyView(strings.Repeat("x", bodyCap)).Scroll)
|
||||
assert.True(t, bodyView(strings.Repeat("x", bodyCap+1)).Scroll)
|
||||
}
|
||||
|
||||
// TestNewBodyView_BinaryNotShown proves a body that is not text
|
||||
// is never shown: one that is not valid UTF-8, or that holds a
|
||||
// control character other than tab, line feed and carriage return.
|
||||
func TestNewBodyView_BinaryNotShown(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, body := range []string{
|
||||
"\xff\xfe\xfd",
|
||||
"a\x00b",
|
||||
// A small protobuf message: valid UTF-8, but control bytes.
|
||||
"\x08\x01\x12\x03abc",
|
||||
"\x1b[31mred\x1b[0m",
|
||||
"a\x7fb",
|
||||
} {
|
||||
v := bodyView(body)
|
||||
|
||||
assert.True(t, v.Binary, "%q", body)
|
||||
assert.Empty(t, v.Text)
|
||||
}
|
||||
|
||||
assert.False(t, bodyView("snow "+snowman).Binary)
|
||||
assert.False(t, bodyView("a\tb\r\nc\n").Binary)
|
||||
}
|
||||
@@ -0,0 +1,74 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
|
||||
"github.com/go-chi/chi"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
)
|
||||
|
||||
// HandleEventDetail shows one event on its own page: its details,
|
||||
// its whole body and every delivery of it. The page reads the
|
||||
// event's body whole, which the receiver caps at 1 MB.
|
||||
func (h *Handlers) HandleEventDetail() http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
webhook, ok := h.ownedWebhook(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
if !h.dbMgr.DBExists(webhook.ID) {
|
||||
h.renderError(w, r, http.StatusNotFound)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
webhookDB, err := h.dbMgr.GetDB(webhook.ID)
|
||||
if err != nil {
|
||||
h.serverError(w, r, "failed to get webhook database", err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
var rows []eventLogRow
|
||||
|
||||
err = webhookDB.Model(&database.Event{}).
|
||||
Select(eventColumns).
|
||||
Where(
|
||||
"id = ? AND webhook_id = ?",
|
||||
chi.URLParam(r, "eventID"), webhook.ID,
|
||||
).
|
||||
Limit(1).
|
||||
Find(&rows).Error
|
||||
if err != nil {
|
||||
h.serverError(w, r, "failed to load event", err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
if len(rows) == 0 {
|
||||
h.renderError(w, r, http.StatusNotFound)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
targets, err := h.loadTargetMap(webhook.ID)
|
||||
if err != nil {
|
||||
h.serverError(w, r, "failed to load targets", err)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
views, ok := h.eventLogViews(
|
||||
w, r, webhookDB, webhook.ID, rows, targets,
|
||||
)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
h.renderTemplate(w, r, "event_detail.html", map[string]any{
|
||||
tmplKeyWebhook: &webhook,
|
||||
"Event": views[0],
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,152 @@
|
||||
package handlers_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/go-chi/chi"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/gorm/clause"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/handlers"
|
||||
"sneak.berlin/go/webhooker/internal/session"
|
||||
)
|
||||
|
||||
// serveEventPage runs the real event page handler as the test user
|
||||
// for the given webhook and event ids.
|
||||
func serveEventPage(
|
||||
t *testing.T,
|
||||
h *handlers.Handlers,
|
||||
sess *session.Session,
|
||||
webhookID, eventID string,
|
||||
) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
|
||||
req := httptest.NewRequestWithContext(
|
||||
context.Background(),
|
||||
http.MethodGet,
|
||||
"/hook/"+webhookID+"/events/"+eventID,
|
||||
nil,
|
||||
)
|
||||
|
||||
for _, c := range authenticatedCookies(
|
||||
t, sess, deleteTestUserID, deleteTestUsername,
|
||||
) {
|
||||
req.AddCookie(c)
|
||||
}
|
||||
|
||||
rctx := chi.NewRouteContext()
|
||||
rctx.URLParams.Add(paramSourceID, webhookID)
|
||||
rctx.URLParams.Add(paramEventID, eventID)
|
||||
|
||||
req = req.WithContext(
|
||||
context.WithValue(req.Context(), chi.RouteCtxKey, rctx),
|
||||
)
|
||||
|
||||
w := httptest.NewRecorder()
|
||||
h.HandleEventDetail().ServeHTTP(w, req)
|
||||
|
||||
return w
|
||||
}
|
||||
|
||||
// TestHandleEventDetail_ShowsEventWholeWithDeliveries proves the
|
||||
// event's page shows its details, its whole body even past the cap
|
||||
// of the lists of events, pretty-printed and in the scrolling box,
|
||||
// and each delivery with its status and attempts.
|
||||
func TestHandleEventDetail_ShowsEventWholeWithDeliveries(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f := newRecentEventsFixture(t)
|
||||
target := seedTarget(t, f.db, f.webhook.ID, database.TargetTypeHTTP)
|
||||
|
||||
const sentinel = "TAIL-SENTINEL-5b2e"
|
||||
|
||||
body := `{"pad":"` + strings.Repeat("x", 2*bodyCap) +
|
||||
`","tail":"` + sentinel + `"}`
|
||||
event := f.event(t, contentTypeJSON, body, time.Now())
|
||||
f.attempt(t, f.delivery(
|
||||
t, event, target.ID, database.DeliveryStatusFailed,
|
||||
), http.StatusBadGateway, time.Second)
|
||||
|
||||
w := serveEventPage(t, f.h, f.sess, f.webhook.ID, event.ID)
|
||||
require.Equal(t, http.StatusOK, w.Code)
|
||||
|
||||
page := w.Body.String()
|
||||
|
||||
assert.Contains(t, page, event.ID)
|
||||
assert.Contains(t, page, contentTypeJSON)
|
||||
assert.Contains(t, page, strconv.Itoa(len(body))+" bytes")
|
||||
assert.Contains(t, page, "{\n "pad": "xxx")
|
||||
assert.Contains(t, page, ""tail": ""+sentinel+""\n}")
|
||||
assert.Contains(t, page, `style="max-height: 32rem; overflow-y: auto"`)
|
||||
assert.NotContains(t, page, "Showing the first")
|
||||
assert.Contains(t, page, target.Name)
|
||||
assert.Contains(t, page, ">failed</span>")
|
||||
assert.Contains(t, page, "Status: 502")
|
||||
}
|
||||
|
||||
// TestHandleEventDetail_ResubmitLinks proves a resubmitted copy's
|
||||
// page links to its original's page, and the original's page says
|
||||
// it was resubmitted.
|
||||
func TestHandleEventDetail_ResubmitLinks(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f := newRecentEventsFixture(t)
|
||||
original := f.event(t, contentTypeJSON, "{}", time.Now())
|
||||
|
||||
cp := &database.Event{
|
||||
WebhookID: f.webhook.ID,
|
||||
Method: http.MethodPost,
|
||||
Body: "{}",
|
||||
ContentType: contentTypeJSON,
|
||||
ResubmittedFromID: &original.ID,
|
||||
}
|
||||
require.NoError(t, f.webhookDB.Omit(clause.Associations).Create(cp).Error)
|
||||
|
||||
w := serveEventPage(t, f.h, f.sess, f.webhook.ID, cp.ID)
|
||||
require.Equal(t, http.StatusOK, w.Code)
|
||||
assert.Contains(
|
||||
t, w.Body.String(),
|
||||
`href="/hook/`+f.webhook.ID+`/events/`+original.ID+`"`,
|
||||
)
|
||||
|
||||
w = serveEventPage(t, f.h, f.sess, f.webhook.ID, original.ID)
|
||||
require.Equal(t, http.StatusOK, w.Code)
|
||||
assert.Contains(t, w.Body.String(), "as 1 new event<")
|
||||
}
|
||||
|
||||
// TestHandleEventDetail_UnknownEventNotFound proves the page is a
|
||||
// 404 for an event that does not exist and for one that belongs to
|
||||
// another webhook.
|
||||
func TestHandleEventDetail_UnknownEventNotFound(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
sess *session.Session
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
)
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
mine := seedWebhook(t, db)
|
||||
theirs := seedWebhook(t, db)
|
||||
|
||||
seedEventWithBody(t, dbMgr, mine.ID, "{}")
|
||||
elsewhere := seedEventWithBody(t, dbMgr, theirs.ID, "{}")
|
||||
|
||||
for _, id := range []string{"no-such-event", elsewhere.ID} {
|
||||
w := serveEventPage(t, h, sess, mine.ID, id)
|
||||
assert.Equal(t, http.StatusNotFound, w.Code, id)
|
||||
}
|
||||
}
|
||||
@@ -5,14 +5,6 @@ import (
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
// maxRenderedBodyBytes caps how many bytes of a stored event
|
||||
// body reach the event log page. Bodies come from the
|
||||
// unauthenticated receiver under the 1 MB ingest cap and
|
||||
// renderTemplate buffers a whole page before writing it, so
|
||||
// an uncapped page of paginationPerPage events is tens of
|
||||
// megabytes of resident memory per concurrent viewer.
|
||||
const maxRenderedBodyBytes = 8192
|
||||
|
||||
// eventLogColumns is the event log's projection. The casts to
|
||||
// blob are load-bearing: they make substr and length count
|
||||
// bytes rather than characters, so the cap bounds the page in
|
||||
@@ -24,27 +16,23 @@ const eventLogColumns = "id, created_at, method, content_type, " +
|
||||
"substr(cast(body as blob), 1, ?) AS body, " +
|
||||
"length(cast(body as blob)) AS body_bytes"
|
||||
|
||||
// eventColumns is eventLogColumns for the event's own page, which
|
||||
// shows the whole body.
|
||||
const eventColumns = "id, created_at, method, content_type, " +
|
||||
"resubmitted_from_id, " +
|
||||
"cast(body as blob) AS body, " +
|
||||
"length(cast(body as blob)) AS body_bytes"
|
||||
|
||||
// EventLogView is the display-safe projection of an event for
|
||||
// the event log page, alongside DeliveryView and TargetView.
|
||||
// It carries a capped body plus the true stored size, so the
|
||||
// page can mark a body as truncated without ever holding the
|
||||
// whole thing.
|
||||
// the event log page and the event's own page, alongside
|
||||
// DeliveryView and TargetView.
|
||||
type EventLogView struct {
|
||||
ID string
|
||||
CreatedAt time.Time
|
||||
Method string
|
||||
ContentType string
|
||||
|
||||
// Body holds at most maxRenderedBodyBytes bytes of the
|
||||
// stored body.
|
||||
Body string
|
||||
|
||||
// BodyBytes is the true size of the stored body.
|
||||
BodyBytes int64
|
||||
|
||||
// BodyTruncated reports that the stored body was larger
|
||||
// than the cap, so the page owes the reader a marker.
|
||||
BodyTruncated bool
|
||||
Body BodyView
|
||||
|
||||
// ResubmittedFromID names the event this one was copied
|
||||
// from, empty for an event that arrived on the receiver.
|
||||
@@ -65,16 +53,10 @@ func (v EventLogView) ResubmittedFrom() bool {
|
||||
return v.ResubmittedFromID != ""
|
||||
}
|
||||
|
||||
// BodyShownBytes is how many body bytes the page is actually
|
||||
// rendering, which the truncation marker reports beside the
|
||||
// true size.
|
||||
func (v EventLogView) BodyShownBytes() int {
|
||||
return len(v.Body)
|
||||
}
|
||||
|
||||
// eventLogRow is one row of the event log projection. Its
|
||||
// body column arrives already cut to the cap by SQLite, with
|
||||
// the true size beside it.
|
||||
// eventLogRow is one row of the event log projection, or of
|
||||
// eventColumns. In the event log its body column arrives
|
||||
// already cut to the cap by SQLite, with the true size beside
|
||||
// it.
|
||||
type eventLogRow struct {
|
||||
ID string
|
||||
CreatedAt time.Time
|
||||
@@ -85,31 +67,22 @@ type eventLogRow struct {
|
||||
BodyBytes int64
|
||||
}
|
||||
|
||||
// view projects a loaded row for rendering.
|
||||
func (r *eventLogRow) view() EventLogView {
|
||||
body := r.Body
|
||||
truncated := r.BodyBytes > int64(len(body))
|
||||
|
||||
// Only a cut body can have been left mid-sequence by
|
||||
// this query. A whole body is passed through exactly as
|
||||
// stored, however malformed.
|
||||
if truncated {
|
||||
body = trimPartialRune(body)
|
||||
}
|
||||
|
||||
// view projects a loaded row of the webhook's events for
|
||||
// rendering.
|
||||
func (r *eventLogRow) view(webhookID string) EventLogView {
|
||||
var from string
|
||||
if r.ResubmittedFromID != nil {
|
||||
from = *r.ResubmittedFromID
|
||||
}
|
||||
|
||||
return EventLogView{
|
||||
ID: r.ID,
|
||||
CreatedAt: r.CreatedAt,
|
||||
Method: r.Method,
|
||||
ContentType: r.ContentType,
|
||||
Body: string(body),
|
||||
BodyBytes: r.BodyBytes,
|
||||
BodyTruncated: truncated,
|
||||
ID: r.ID,
|
||||
CreatedAt: r.CreatedAt,
|
||||
Method: r.Method,
|
||||
ContentType: r.ContentType,
|
||||
Body: newBodyView(
|
||||
"/hook/"+webhookID+"/events/"+r.ID, r.Body, r.BodyBytes,
|
||||
),
|
||||
ResubmittedFromID: from,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -16,7 +16,7 @@ import (
|
||||
"sneak.berlin/go/webhooker/internal/session"
|
||||
)
|
||||
|
||||
// bodyCap is the number of body bytes the event log page is
|
||||
// bodyCap is the number of body bytes the lists of events are
|
||||
// allowed to render for one event.
|
||||
const bodyCap = handlers.MaxRenderedBodyBytesForTest
|
||||
|
||||
@@ -85,7 +85,7 @@ func seedAndProject(
|
||||
|
||||
// TestHandleSourceLogs_BoundsOversizeBody proves the rendered
|
||||
// page is bounded by the cap rather than by the stored payload:
|
||||
// the body here is 64 times the cap, and the ingest path would
|
||||
// the body here is 16 times the cap, and the ingest path would
|
||||
// accept twice as much again.
|
||||
func TestHandleSourceLogs_BoundsOversizeBody(t *testing.T) {
|
||||
t.Parallel()
|
||||
@@ -123,7 +123,7 @@ func TestHandleSourceLogs_BoundsOversizeBody(t *testing.T) {
|
||||
// The marker states the true stored size, not the cut one.
|
||||
assert.Contains(
|
||||
t, page,
|
||||
"showing "+strconv.Itoa(bodyCap)+
|
||||
"Showing the first "+strconv.Itoa(bodyCap)+
|
||||
" of "+strconv.Itoa(storedBytes)+" bytes",
|
||||
)
|
||||
}
|
||||
@@ -152,34 +152,35 @@ func TestHandleSourceLogs_SmallBodyRendersWhole(t *testing.T) {
|
||||
page := renderSourceLogsPage(t, h, sess, wh.ID)
|
||||
|
||||
assert.Contains(t, page, ""kept"")
|
||||
assert.NotContains(t, page, "Body truncated for display")
|
||||
assert.NotContains(t, page, "Showing the first")
|
||||
}
|
||||
|
||||
// TestEventLogView_CutMidRune proves a multi-byte rune severed
|
||||
// by the byte-wise cut is dropped rather than surfaced as a
|
||||
// mojibake tail.
|
||||
// mojibake tail, which would also make the text look binary.
|
||||
func TestEventLogView_CutMidRune(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
body := strings.Repeat(snowman, 4096)
|
||||
body := strings.Repeat(snowman, bodyCap)
|
||||
view := seedAndProject(t, body)
|
||||
|
||||
// bodyCap bytes hold bodyCap/3 whole snowmen and two bytes
|
||||
// of the next one; those two are dropped.
|
||||
whole := bodyCap / len(snowman)
|
||||
|
||||
assert.True(t, view.BodyTruncated)
|
||||
assert.Equal(t, int64(len(body)), view.BodyBytes)
|
||||
assert.Equal(t, strings.Repeat(snowman, whole), view.Body)
|
||||
assert.True(t, utf8.ValidString(view.Body))
|
||||
assert.LessOrEqual(t, len(view.Body), bodyCap)
|
||||
assert.True(t, view.Body.Cut)
|
||||
assert.False(t, view.Body.Binary)
|
||||
assert.Equal(t, int64(len(body)), view.Body.Size)
|
||||
assert.Equal(t, strings.Repeat(snowman, whole), view.Body.Text)
|
||||
assert.True(t, utf8.ValidString(view.Body.Text))
|
||||
assert.Equal(t, len(view.Body.Text), view.Body.ShownBytes)
|
||||
assert.LessOrEqual(t, view.Body.ShownBytes, bodyCap)
|
||||
}
|
||||
|
||||
// TestEventLogView_BinaryBodyLeftAsStored proves a binary
|
||||
// payload is passed through byte for byte. Its tail is invalid
|
||||
// UTF-8 however the cut falls, so repairing it would misreport
|
||||
// what the sender delivered.
|
||||
func TestEventLogView_BinaryBodyLeftAsStored(t *testing.T) {
|
||||
// TestEventLogView_BinaryBodyNotShown proves a body that is not
|
||||
// text is left out rather than shown as replacement characters,
|
||||
// whether it is cut or not.
|
||||
func TestEventLogView_BinaryBodyNotShown(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
raw := make([]byte, bodyCap+808)
|
||||
@@ -188,12 +189,21 @@ func TestEventLogView_BinaryBodyLeftAsStored(t *testing.T) {
|
||||
raw[i] = 0x80 | byte(i%0x40)
|
||||
}
|
||||
|
||||
view := seedAndProject(t, string(raw))
|
||||
for name, body := range map[string][]byte{
|
||||
"cut": raw,
|
||||
"whole": raw[:2048],
|
||||
"NUL": []byte("text\x00text"),
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
assert.True(t, view.BodyTruncated)
|
||||
assert.Equal(t, int64(len(raw)), view.BodyBytes)
|
||||
assert.Equal(t, string(raw[:bodyCap]), view.Body)
|
||||
assert.False(t, utf8.ValidString(view.Body))
|
||||
view := seedAndProject(t, string(body))
|
||||
|
||||
assert.True(t, view.Body.Binary)
|
||||
assert.Empty(t, view.Body.Text)
|
||||
assert.Equal(t, int64(len(body)), view.Body.Size)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestTrimPartialRune covers the distinction the cut repair
|
||||
|
||||
@@ -19,10 +19,16 @@ func (s *Handlers) SetLogForTest(log *slog.Logger) {
|
||||
s.log = log
|
||||
}
|
||||
|
||||
// MaxRenderedBodyBytesForTest exposes the event log's body cap
|
||||
// to the handlers_test package.
|
||||
// MaxRenderedBodyBytesForTest exposes the body cap of the lists
|
||||
// of events to the handlers_test package.
|
||||
const MaxRenderedBodyBytesForTest = maxRenderedBodyBytes
|
||||
|
||||
// NewBodyViewForTest exposes newBodyView for use in the
|
||||
// handlers_test package.
|
||||
func NewBodyViewForTest(body []byte, size int64) BodyView {
|
||||
return newBodyView("/hook/w/events/e", body, size)
|
||||
}
|
||||
|
||||
// MaxRenderedResponseBytesForTest exposes the event log's
|
||||
// delivery response cap to the handlers_test package.
|
||||
const MaxRenderedResponseBytesForTest = maxRenderedResponseBytes
|
||||
|
||||
@@ -97,7 +97,8 @@ type Handlers struct {
|
||||
// names through the archive rename, the save and any move back.
|
||||
// Interleaved, one could rename an archive between another's
|
||||
// rename and save, leaving the file named for one edit and the
|
||||
// stored names from the other.
|
||||
// stored names from the other. An archive download holds it while
|
||||
// it reads the stored names and opens the file they give.
|
||||
renameMu sync.Mutex
|
||||
|
||||
// dummyVerifications counts the equivalent-cost verifications
|
||||
@@ -149,16 +150,23 @@ func New(
|
||||
|
||||
// Parse all page templates once at startup
|
||||
s.templates = map[string]*template.Template{
|
||||
"login.html": parsePageTemplate("login.html"),
|
||||
"profile.html": parsePageTemplate("profile.html"),
|
||||
"settings.html": parsePageTemplate("settings.html"),
|
||||
"sources_list.html": parsePageTemplate("sources_list.html"),
|
||||
"sources_new.html": parsePageTemplate("sources_new.html"),
|
||||
"source_detail.html": parsePageTemplate("source_detail.html", "webhook_stats.html"),
|
||||
"source_edit.html": parsePageTemplate("source_edit.html"),
|
||||
"source_logs.html": parsePageTemplate("source_logs.html"),
|
||||
"target_edit.html": parsePageTemplate("target_edit.html"),
|
||||
"error.html": parsePageTemplate("error.html"),
|
||||
"login.html": parsePageTemplate("login.html"),
|
||||
"profile.html": parsePageTemplate("profile.html"),
|
||||
"settings.html": parsePageTemplate("settings.html"),
|
||||
"sources_list.html": parsePageTemplate("sources_list.html"),
|
||||
"sources_new.html": parsePageTemplate("sources_new.html"),
|
||||
"source_detail.html": parsePageTemplate(
|
||||
"source_detail.html", "webhook_stats.html", "event_body.html",
|
||||
),
|
||||
"source_edit.html": parsePageTemplate("source_edit.html"),
|
||||
"source_logs.html": parsePageTemplate(
|
||||
"source_logs.html", "event_body.html", "delivery_attempts.html",
|
||||
),
|
||||
"event_detail.html": parsePageTemplate(
|
||||
"event_detail.html", "event_body.html", "delivery_attempts.html",
|
||||
),
|
||||
"target_edit.html": parsePageTemplate("target_edit.html"),
|
||||
"error.html": parsePageTemplate("error.html"),
|
||||
}
|
||||
|
||||
lc.Append(fx.Hook{
|
||||
@@ -385,7 +393,7 @@ func (s *Handlers) pageData(
|
||||
// partial body and the status before a mid-render error can be
|
||||
// reported, leaving no way to serve a 500. Buffering makes a page's
|
||||
// rendered size resident memory per concurrent viewer, so every page
|
||||
// owes it a bound: the event log caps each stored body at
|
||||
// owes it a bound: the lists of events cap each stored body at
|
||||
// maxRenderedBodyBytes for exactly this reason.
|
||||
func (s *Handlers) executeTemplate(
|
||||
w http.ResponseWriter,
|
||||
|
||||
@@ -20,7 +20,6 @@ const (
|
||||
webhookSaved noticeCode = "webhook-saved"
|
||||
webhookDeleted noticeCode = "webhook-deleted"
|
||||
entrypointAdded noticeCode = "entrypoint-added"
|
||||
entrypointSaved noticeCode = "entrypoint-saved"
|
||||
entrypointDeleted noticeCode = "entrypoint-deleted"
|
||||
entrypointActivated noticeCode = "entrypoint-activated"
|
||||
entrypointDeactivated noticeCode = "entrypoint-deactivated"
|
||||
@@ -49,7 +48,6 @@ func noticeFor(r *http.Request) *notice {
|
||||
webhookSaved: {Text: "Webhook saved."},
|
||||
webhookDeleted: {Text: "Webhook deleted."},
|
||||
entrypointAdded: {Text: "Entrypoint added."},
|
||||
entrypointSaved: {Text: "Entrypoint description saved."},
|
||||
entrypointDeleted: {Text: "Entrypoint deleted."},
|
||||
entrypointActivated: {Text: "Entrypoint activated."},
|
||||
entrypointDeactivated: {Text: "Entrypoint deactivated."},
|
||||
|
||||
@@ -12,11 +12,12 @@ import (
|
||||
)
|
||||
|
||||
// recentEventColumns is the recent events list's projection. It
|
||||
// leaves out the body, for the reason maxRenderedBodyBytes gives,
|
||||
// and reads its size from body_bytes, recorded when the event was
|
||||
// reads the body cut to maxRenderedBodyBytes, as eventLogColumns
|
||||
// does, and its size from body_bytes, recorded when the event was
|
||||
// stored.
|
||||
const recentEventColumns = "id, created_at, method, content_type, " +
|
||||
"resubmitted_from_id, body_bytes"
|
||||
"resubmitted_from_id, body_bytes, " +
|
||||
"substr(cast(body as blob), 1, ?) AS body"
|
||||
|
||||
// recentAttemptColumns is the part of a recorded attempt the list
|
||||
// uses. The event log's deliveryResultColumns also reads response
|
||||
@@ -50,6 +51,9 @@ type RecentEventView struct {
|
||||
// unless the webhook has exactly one HTTP target.
|
||||
Status string
|
||||
StatusClass string
|
||||
|
||||
// Body is what the row shows when it is expanded.
|
||||
Body BodyView
|
||||
}
|
||||
|
||||
// recentEventRow is one row of recentEventColumns.
|
||||
@@ -60,6 +64,7 @@ type recentEventRow struct {
|
||||
ContentType string
|
||||
ResubmittedFromID *string
|
||||
BodyBytes uint64
|
||||
Body []byte
|
||||
}
|
||||
|
||||
// recentAttemptRow is one row of recentAttemptColumns. CreatedAt is
|
||||
@@ -100,7 +105,7 @@ func loadRecentEvents(
|
||||
var rows []recentEventRow
|
||||
|
||||
err := webhookDB.Model(&database.Event{}).
|
||||
Select(recentEventColumns).
|
||||
Select(recentEventColumns, maxRenderedBodyBytes).
|
||||
Where("webhook_id = ?", webhookID).
|
||||
Order("created_at DESC").
|
||||
Limit(recentEventLimit).
|
||||
@@ -145,7 +150,7 @@ func loadRecentEvents(
|
||||
views := make([]RecentEventView, len(rows))
|
||||
for i := range rows {
|
||||
views[i] = rows[i].view(
|
||||
byEvent[rows[i].ID], attempts, statusTargetID,
|
||||
webhookID, byEvent[rows[i].ID], attempts, statusTargetID,
|
||||
)
|
||||
}
|
||||
|
||||
@@ -182,14 +187,20 @@ func loadRecentAttempts(
|
||||
return byDelivery, nil
|
||||
}
|
||||
|
||||
// view projects a loaded row for rendering. deliveries is the
|
||||
// event's deliveries, oldest first, and attempts their recorded
|
||||
// attempts keyed by delivery ID.
|
||||
// view projects a loaded row of the webhook's events for
|
||||
// rendering. deliveries is the event's deliveries, oldest first,
|
||||
// and attempts their recorded attempts keyed by delivery ID.
|
||||
func (r *recentEventRow) view(
|
||||
webhookID string,
|
||||
deliveries []database.Delivery,
|
||||
attempts map[string][]recentAttemptRow,
|
||||
statusTargetID string,
|
||||
) RecentEventView {
|
||||
//nolint:gosec // body_bytes is at most the receiver's 1 MB cap
|
||||
body := newBodyView(
|
||||
"/hook/"+webhookID+"/events/"+r.ID, r.Body, int64(r.BodyBytes),
|
||||
)
|
||||
|
||||
v := RecentEventView{
|
||||
Method: r.Method,
|
||||
ContentType: r.ContentType,
|
||||
@@ -197,6 +208,7 @@ func (r *recentEventRow) view(
|
||||
ReceivedUTC: r.CreatedAt.UTC().Format(time.DateTime) + " UTC",
|
||||
Size: humanize.Bytes(r.BodyBytes),
|
||||
ProcessingTime: processingTime(deliveries, attempts),
|
||||
Body: body,
|
||||
}
|
||||
|
||||
if r.ResubmittedFromID != nil {
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -301,6 +302,73 @@ func TestHandleSourceDetail_NoStatusWithoutSingleHTTPTarget(
|
||||
}
|
||||
}
|
||||
|
||||
// TestHandleSourceDetail_RecentEventsLinkAndExpand proves each row
|
||||
// links to its event's own page and expands to show its body, and
|
||||
// that only the newest row starts expanded.
|
||||
func TestHandleSourceDetail_RecentEventsLinkAndExpand(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f := newRecentEventsFixture(t)
|
||||
now := time.Now()
|
||||
|
||||
older := f.event(
|
||||
t, contentTypeJSON, `{"which":"older"}`, now.Add(-time.Minute),
|
||||
)
|
||||
newer := f.event(t, contentTypeJSON, `{"which":"newer"}`, now)
|
||||
|
||||
body := f.render(t)
|
||||
|
||||
for _, e := range []*database.Event{older, newer} {
|
||||
assert.Contains(
|
||||
t, body, `href="/hook/`+f.webhook.ID+`/events/`+e.ID+`"`,
|
||||
)
|
||||
}
|
||||
|
||||
assert.Equal(t, 2, strings.Count(body, `<div x-show="open" x-cloak class="mt-3">`))
|
||||
assert.Equal(t, 1, strings.Count(body, " data-open>"))
|
||||
|
||||
open := strings.Index(body, " data-open>")
|
||||
newerBody := strings.Index(body, ""which": "newer"")
|
||||
olderBody := strings.Index(body, ""which": "older"")
|
||||
|
||||
assert.Less(t, open, newerBody, "the newest row is not the open one")
|
||||
assert.Less(t, newerBody, olderBody)
|
||||
}
|
||||
|
||||
// TestHandleSourceDetail_RecentEventBodyCut proves a body up to
|
||||
// the cap is shown whole and pretty-printed, and a larger one only
|
||||
// its first bodyCap bytes, as received, with links to the whole
|
||||
// body on the event's page and to the download.
|
||||
func TestHandleSourceDetail_RecentEventBodyCut(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f := newRecentEventsFixture(t)
|
||||
now := time.Now()
|
||||
|
||||
// A JSON document of n bytes.
|
||||
document := func(n int) string {
|
||||
return `{"pad":"` + strings.Repeat("x", n-len(`{"pad":""}`)) + `"}`
|
||||
}
|
||||
|
||||
whole := f.event(
|
||||
t, contentTypeJSON, document(bodyCap), now.Add(-time.Minute),
|
||||
)
|
||||
cut := f.event(t, contentTypeJSON, document(bodyCap+1), now)
|
||||
|
||||
body := f.render(t)
|
||||
eventURL := `href="/hook/` + f.webhook.ID + `/events/`
|
||||
|
||||
assert.Equal(t, 1, strings.Count(body, "{\n "pad": "))
|
||||
assert.Contains(t, body, "{"pad":"xxx")
|
||||
assert.Contains(
|
||||
t, body,
|
||||
"Showing the first "+strconv.Itoa(bodyCap)+" of "+
|
||||
strconv.Itoa(bodyCap+1)+" bytes, unformatted.",
|
||||
)
|
||||
assert.Contains(t, body, eventURL+cut.ID+`/body"`)
|
||||
assert.NotContains(t, body, eventURL+whole.ID+`/body"`)
|
||||
}
|
||||
|
||||
// TestHandleWebhook_RecordsBodySize proves the receiver records the
|
||||
// body's size in bytes, not characters, with the event it stores.
|
||||
func TestHandleWebhook_RecordsBodySize(t *testing.T) {
|
||||
|
||||
@@ -86,13 +86,12 @@ var errInjectedDelete = errors.New("injected delete failure")
|
||||
// save of an existing row.
|
||||
var errInjectedSave = errors.New("injected save failure")
|
||||
|
||||
// seedEntrypoint inserts an active entrypoint for a webhook and
|
||||
// returns it.
|
||||
// seedEntrypoint inserts an entrypoint for a webhook.
|
||||
func seedEntrypoint(
|
||||
t *testing.T,
|
||||
db *database.Database,
|
||||
webhookID string,
|
||||
) *database.Entrypoint {
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
ep := &database.Entrypoint{
|
||||
@@ -105,8 +104,6 @@ func seedEntrypoint(
|
||||
t,
|
||||
db.DB().Omit(clause.Associations).Create(ep).Error,
|
||||
)
|
||||
|
||||
return ep
|
||||
}
|
||||
|
||||
// countRows counts the live (not soft-deleted) rows of a model
|
||||
|
||||
@@ -1022,10 +1022,10 @@ func (h *Handlers) HandleSourceLogs() http.HandlerFunc {
|
||||
// view, which renders its target as a blank name.
|
||||
//
|
||||
// This map is historical display only. It is built for the event
|
||||
// log page and reaches nothing but DeliveryView.Target: the
|
||||
// target list on the source detail page, the edit form and the
|
||||
// replay path each resolve targets themselves, and a deleted row
|
||||
// is refused there as before.
|
||||
// log and an event's own page, and reaches nothing but
|
||||
// DeliveryView.Target: the target list on the source detail page,
|
||||
// the edit form and the replay path each resolve targets
|
||||
// themselves, and a deleted row is refused there as before.
|
||||
func (h *Handlers) loadTargetMap(
|
||||
webhookID string,
|
||||
) (map[string]eventLogTarget, error) {
|
||||
@@ -1081,10 +1081,8 @@ func (h *Handlers) loadEventsWithDeliveries(
|
||||
targetMap map[string]eventLogTarget,
|
||||
page int,
|
||||
) ([]EventLogView, int64, bool) {
|
||||
var result []EventLogView
|
||||
|
||||
if !h.dbMgr.DBExists(webhook.ID) {
|
||||
return result, 0, true
|
||||
return nil, 0, true
|
||||
}
|
||||
|
||||
webhookDB, err := h.dbMgr.GetDB(webhook.ID)
|
||||
@@ -1100,7 +1098,26 @@ func (h *Handlers) loadEventsWithDeliveries(
|
||||
webhookDB, webhook.ID, page,
|
||||
)
|
||||
|
||||
result = make([]EventLogView, len(rows))
|
||||
result, ok := h.eventLogViews(
|
||||
w, r, webhookDB, webhook.ID, rows, targetMap,
|
||||
)
|
||||
|
||||
return result, totalEvents, ok
|
||||
}
|
||||
|
||||
// eventLogViews projects loaded events for rendering, each with
|
||||
// its deliveries and how many times it has been resubmitted. Like
|
||||
// loadEventsWithDeliveries, it reports false once it has answered
|
||||
// the request with an error.
|
||||
func (h *Handlers) eventLogViews(
|
||||
w http.ResponseWriter,
|
||||
r *http.Request,
|
||||
webhookDB *gorm.DB,
|
||||
webhookID string,
|
||||
rows []eventLogRow,
|
||||
targetMap map[string]eventLogTarget,
|
||||
) ([]EventLogView, bool) {
|
||||
result := make([]EventLogView, len(rows))
|
||||
eventDeliveries := make([][]database.Delivery, len(rows))
|
||||
|
||||
var deliveryIDs []string
|
||||
@@ -1108,7 +1125,7 @@ func (h *Handlers) loadEventsWithDeliveries(
|
||||
eventIDs := make([]string, len(rows))
|
||||
|
||||
for i := range rows {
|
||||
result[i] = rows[i].view()
|
||||
result[i] = rows[i].view(webhookID)
|
||||
eventIDs[i] = rows[i].ID
|
||||
|
||||
webhookDB.Where(
|
||||
@@ -1130,7 +1147,7 @@ func (h *Handlers) loadEventsWithDeliveries(
|
||||
w, r, "failed to load delivery attempts", err,
|
||||
)
|
||||
|
||||
return nil, 0, false
|
||||
return nil, false
|
||||
}
|
||||
|
||||
resubmits, err := resubmitCounts(webhookDB, eventIDs)
|
||||
@@ -1139,7 +1156,7 @@ func (h *Handlers) loadEventsWithDeliveries(
|
||||
w, r, "failed to count event resubmissions", err,
|
||||
)
|
||||
|
||||
return nil, 0, false
|
||||
return nil, false
|
||||
}
|
||||
|
||||
for i := range rows {
|
||||
@@ -1149,7 +1166,7 @@ func (h *Handlers) loadEventsWithDeliveries(
|
||||
result[i].ResubmitCount = resubmits[rows[i].ID]
|
||||
}
|
||||
|
||||
return result, totalEvents, true
|
||||
return result, true
|
||||
}
|
||||
|
||||
// loadEventLogRows reads one page of the event log projection, newest
|
||||
@@ -1396,70 +1413,6 @@ func (h *Handlers) HandleEntrypointCreate() http.HandlerFunc {
|
||||
}
|
||||
}
|
||||
|
||||
// HandleEntrypointEdit handles changing an entrypoint's description.
|
||||
// It writes only the description column, so the entrypoint keeps its
|
||||
// URL, and an activate or deactivate saved since the page was shown
|
||||
// is not undone.
|
||||
func (h *Handlers) HandleEntrypointEdit() http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
userID, ok := h.getUserID(r)
|
||||
if !ok {
|
||||
http.Redirect(
|
||||
w, r, "/pages/login", http.StatusSeeOther,
|
||||
)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
sourceID := chi.URLParam(r, "sourceID")
|
||||
entrypointID := chi.URLParam(r, "entrypointID")
|
||||
|
||||
var webhook database.Webhook
|
||||
|
||||
err := h.db.DB().Where(
|
||||
"id = ? AND user_id = ?", sourceID, userID,
|
||||
).First(&webhook).Error
|
||||
if err != nil {
|
||||
h.renderError(w, r, http.StatusNotFound)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
// The body size cap is enforced by the MaxBodySize
|
||||
// middleware, which runs before CSRF parses the form.
|
||||
err = r.ParseForm()
|
||||
if err != nil {
|
||||
h.renderError(w, r, http.StatusBadRequest)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
result := h.db.DB().Model(&database.Entrypoint{}).Where(
|
||||
"id = ? AND webhook_id = ?", entrypointID, webhook.ID,
|
||||
).Update("description", r.PostFormValue("description"))
|
||||
if result.Error != nil {
|
||||
h.serverError(
|
||||
w, r, "failed to edit entrypoint", result.Error,
|
||||
)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
// The id came from the URL and may name another webhook's
|
||||
// entrypoint, which this webhook does not have.
|
||||
if result.RowsAffected == 0 {
|
||||
h.renderError(w, r, http.StatusNotFound)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
http.Redirect(
|
||||
w, r, withNotice("/hook/"+webhook.ID, entrypointSaved),
|
||||
http.StatusSeeOther,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// HandleTargetCreate handles adding a new target to a webhook.
|
||||
func (h *Handlers) HandleTargetCreate() http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
@@ -1941,13 +1894,9 @@ func (h *Handlers) HandleEntrypointToggle() http.HandlerFunc {
|
||||
return false, err
|
||||
}
|
||||
|
||||
// Only the active column: saving the whole row would
|
||||
// write back the description read above over an edit
|
||||
// saved since.
|
||||
active := !ep.Active
|
||||
ep.Active = !ep.Active
|
||||
|
||||
return active, h.db.DB().Model(&ep).
|
||||
Update("active", active).Error
|
||||
return ep.Active, h.db.DB().Save(&ep).Error
|
||||
},
|
||||
"failed to toggle entrypoint",
|
||||
entrypointActivated, entrypointDeactivated,
|
||||
|
||||
@@ -191,6 +191,7 @@ func storedRetentionDays(
|
||||
type sourceTestEnv struct {
|
||||
handlers *handlers.Handlers
|
||||
db *database.Database
|
||||
dbMgr *database.WebhookDBManager
|
||||
archives *recordingArchives
|
||||
cookies []*http.Cookie
|
||||
}
|
||||
@@ -204,9 +205,11 @@ func setupSourceTest(t *testing.T) *sourceTestEnv {
|
||||
|
||||
var db *database.Database
|
||||
|
||||
var dbMgr *database.WebhookDBManager
|
||||
|
||||
var archives *recordingArchives
|
||||
|
||||
app := newTestApp(t, &h, &sess, &db, &archives)
|
||||
app := newTestApp(t, &h, &sess, &db, &dbMgr, &archives)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
@@ -214,6 +217,7 @@ func setupSourceTest(t *testing.T) *sourceTestEnv {
|
||||
return &sourceTestEnv{
|
||||
handlers: h,
|
||||
db: db,
|
||||
dbMgr: dbMgr,
|
||||
archives: archives,
|
||||
cookies: authenticatedCookies(
|
||||
t, sess, sourceTestUserID, "sourceuser",
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
)
|
||||
|
||||
// downloadWriteTimeout is how long one write of a download may wait
|
||||
// for a client that has stopped reading.
|
||||
const downloadWriteTimeout = 60 * time.Second
|
||||
|
||||
// HandleTargetDownload serves a database target's archive as one
|
||||
// gzipped JSON file, named for the webhook, the target and the time;
|
||||
// see delivery.ArchiveExport.WriteGzipJSON for what it holds. Other
|
||||
// target types have no archive and are a 404.
|
||||
//
|
||||
// A download runs for as long as the client keeps reading: it reads
|
||||
// under a context the request limit does not cancel, and gives each
|
||||
// write its own deadline in place of the server's write timeout. It
|
||||
// stops when a write fails.
|
||||
func (h *Handlers) HandleTargetDownload() http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
ctx := context.WithoutCancel(r.Context())
|
||||
|
||||
webhook, target, export, ok := h.openTargetArchive(ctx, w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
defer func() { _ = export.Close() }()
|
||||
|
||||
now := time.Now()
|
||||
|
||||
w.Header().Set("Content-Type", "application/gzip")
|
||||
w.Header().Set(
|
||||
"Content-Disposition",
|
||||
`attachment; filename="`+delivery.ArchiveExportFileName(
|
||||
webhook.Name, target.Name, now,
|
||||
)+`"`,
|
||||
)
|
||||
|
||||
err := export.WriteGzipJSON(
|
||||
ctx,
|
||||
downloadWriter{w: w, rc: http.NewResponseController(w)},
|
||||
&webhook, target, now,
|
||||
)
|
||||
if err != nil {
|
||||
h.log.Error(
|
||||
"failed to export archive",
|
||||
"target_id", target.ID,
|
||||
"error", err,
|
||||
)
|
||||
|
||||
// The 200 has gone out. Aborting the connection is what
|
||||
// tells the client the file is incomplete.
|
||||
panic(http.ErrAbortHandler)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// downloadWriter writes a download to the client, giving each write
|
||||
// downloadWriteTimeout to finish.
|
||||
type downloadWriter struct {
|
||||
w http.ResponseWriter
|
||||
rc *http.ResponseController
|
||||
}
|
||||
|
||||
func (d downloadWriter) Write(b []byte) (int, error) {
|
||||
// A writer that has no write deadline, such as a test's recorder,
|
||||
// answers http.ErrNotSupported and needs none extended.
|
||||
err := d.rc.SetWriteDeadline(time.Now().Add(downloadWriteTimeout))
|
||||
if err != nil && !errors.Is(err, http.ErrNotSupported) {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
return d.w.Write(b)
|
||||
}
|
||||
|
||||
// openTargetArchive opens the archive of the request's database target
|
||||
// for export, with its reads under ctx. It reports false once it has
|
||||
// written the response.
|
||||
//
|
||||
// It holds renameMu, which every archive rename runs under, while it
|
||||
// reads the stored names and opens the file, so the file it opens is
|
||||
// the one those names give. It lets go before the export is streamed:
|
||||
// once the file is open, a rename does not affect the export.
|
||||
func (h *Handlers) openTargetArchive(
|
||||
ctx context.Context,
|
||||
w http.ResponseWriter,
|
||||
r *http.Request,
|
||||
) (database.Webhook, *database.Target, *delivery.ArchiveExport, bool) {
|
||||
h.renameMu.Lock()
|
||||
defer h.renameMu.Unlock()
|
||||
|
||||
webhook, target, ok := h.ownedTarget(w, r)
|
||||
if !ok {
|
||||
return database.Webhook{}, nil, nil, false
|
||||
}
|
||||
|
||||
if target.Type != database.TargetTypeDatabase {
|
||||
h.renderError(w, r, http.StatusNotFound)
|
||||
|
||||
return database.Webhook{}, nil, nil, false
|
||||
}
|
||||
|
||||
export, err := delivery.OpenArchiveExport(
|
||||
ctx, delivery.ArchivePath(h.dbMgr, &webhook, target), h.log,
|
||||
)
|
||||
if err != nil {
|
||||
h.serverError(w, r, "failed to open archive for export", err)
|
||||
|
||||
return database.Webhook{}, nil, nil, false
|
||||
}
|
||||
|
||||
return webhook, target, export, true
|
||||
}
|
||||
@@ -0,0 +1,379 @@
|
||||
package handlers_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"sneak.berlin/go/webhooker/internal/config"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
"sneak.berlin/go/webhooker/internal/delivery"
|
||||
"sneak.berlin/go/webhooker/internal/middleware"
|
||||
)
|
||||
|
||||
// errClientGone is the write failure of a client that has gone away.
|
||||
var errClientGone = errors.New("client gone")
|
||||
|
||||
// downloadPath is the archive download route of a target.
|
||||
func downloadPath(webhookID, targetID string) string {
|
||||
return "/hook/" + webhookID + "/targets/" + targetID + "/download"
|
||||
}
|
||||
|
||||
// renameTarget submits the edit form renaming a target to Renamed.
|
||||
func renameTarget(
|
||||
env *sourceTestEnv, webhookID, targetID string,
|
||||
) *httptest.ResponseRecorder {
|
||||
form := url.Values{}
|
||||
form.Set("name", "Renamed")
|
||||
|
||||
return submitTargetEdit(env, webhookID, targetID, form)
|
||||
}
|
||||
|
||||
// TestHandleTargetDownload proves a database target's archive
|
||||
// downloads as a gzipped JSON attachment named for the webhook, the
|
||||
// target and the time, here with no archive file yet, so with no
|
||||
// rows; and that a target of another type has no download.
|
||||
func TestHandleTargetDownload(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupSourceTest(t)
|
||||
wh := seedWebhookWithRetention(t, env.db, 7)
|
||||
archive := seedTarget(t, env.db, wh.ID, database.TargetTypeDatabase)
|
||||
logTarget := seedTarget(t, env.db, wh.ID, database.TargetTypeLog)
|
||||
|
||||
w := serveTarget(
|
||||
env, http.MethodGet, downloadPath(wh.ID, archive.ID), nil,
|
||||
)
|
||||
require.Equal(t, http.StatusOK, w.Code, w.Body.String())
|
||||
assert.Equal(t, "application/gzip", w.Header().Get("Content-Type"))
|
||||
assert.Regexp(t,
|
||||
`^attachment; filename="archive-seeded-t-database-`+
|
||||
`\d{8}T\d{6}Z\.json\.gz"$`,
|
||||
w.Header().Get("Content-Disposition"),
|
||||
)
|
||||
|
||||
zr, err := gzip.NewReader(w.Body)
|
||||
require.NoError(t, err)
|
||||
|
||||
var got map[string]json.RawMessage
|
||||
|
||||
require.NoError(t, json.NewDecoder(zr).Decode(&got))
|
||||
assert.JSONEq(t,
|
||||
`{"id":"`+archive.ID+`","name":"t-database"}`,
|
||||
string(got["target"]),
|
||||
)
|
||||
assert.JSONEq(t, `[]`, string(got["archived_events"]))
|
||||
|
||||
w = serveTarget(
|
||||
env, http.MethodGet, downloadPath(wh.ID, logTarget.ID), nil,
|
||||
)
|
||||
assert.Equal(t, http.StatusNotFound, w.Code)
|
||||
}
|
||||
|
||||
// TestHandleTargetDownload_WaitsForRename proves a download reads the
|
||||
// target's names and opens its archive under the lock a rename holds:
|
||||
// started while an edit is renaming the archive, it waits, and is
|
||||
// named for the target's new name.
|
||||
func TestHandleTargetDownload_WaitsForRename(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupSourceTest(t)
|
||||
wh := seedWebhookWithRetention(t, env.db, 7)
|
||||
archive := seedTarget(t, env.db, wh.ID, database.TargetTypeDatabase)
|
||||
|
||||
renaming, release := env.archives.BlockNextRename()
|
||||
edited := make(chan *httptest.ResponseRecorder, 1)
|
||||
|
||||
go func() {
|
||||
edited <- renameTarget(env, wh.ID, archive.ID)
|
||||
}()
|
||||
|
||||
<-renaming
|
||||
|
||||
downloaded := make(chan *httptest.ResponseRecorder, 1)
|
||||
|
||||
go func() {
|
||||
downloaded <- serveTarget(
|
||||
env, http.MethodGet, downloadPath(wh.ID, archive.ID), nil,
|
||||
)
|
||||
}()
|
||||
|
||||
select {
|
||||
case <-downloaded:
|
||||
release()
|
||||
t.Fatal("the download did not wait for the rename")
|
||||
case <-time.After(100 * time.Millisecond):
|
||||
}
|
||||
|
||||
release()
|
||||
require.Equal(t, http.StatusSeeOther, (<-edited).Code)
|
||||
|
||||
w := <-downloaded
|
||||
require.Equal(t, http.StatusOK, w.Code)
|
||||
assert.Contains(t,
|
||||
w.Header().Get("Content-Disposition"), "archive-seeded-renamed-",
|
||||
)
|
||||
}
|
||||
|
||||
// stalledWriter is a response writer whose first write waits until
|
||||
// resume is closed, closing writing when it starts to wait.
|
||||
type stalledWriter struct {
|
||||
*httptest.ResponseRecorder
|
||||
|
||||
once sync.Once
|
||||
writing chan struct{}
|
||||
resume chan struct{}
|
||||
}
|
||||
|
||||
func (s *stalledWriter) Write(b []byte) (int, error) {
|
||||
s.once.Do(func() {
|
||||
close(s.writing)
|
||||
<-s.resume
|
||||
})
|
||||
|
||||
return s.ResponseRecorder.Write(b)
|
||||
}
|
||||
|
||||
// TestHandleTargetDownload_StreamsWithoutTheLock proves a download
|
||||
// lets go of the rename lock once its archive is open: while the
|
||||
// download is stalled writing, an edit can still rename the target.
|
||||
func TestHandleTargetDownload_StreamsWithoutTheLock(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupSourceTest(t)
|
||||
wh := seedWebhookWithRetention(t, env.db, 7)
|
||||
archive := seedTarget(t, env.db, wh.ID, database.TargetTypeDatabase)
|
||||
|
||||
req := httptest.NewRequestWithContext(
|
||||
t.Context(), http.MethodGet, downloadPath(wh.ID, archive.ID), nil,
|
||||
)
|
||||
for _, c := range env.cookies {
|
||||
req.AddCookie(c)
|
||||
}
|
||||
|
||||
sw := &stalledWriter{
|
||||
ResponseRecorder: httptest.NewRecorder(),
|
||||
writing: make(chan struct{}),
|
||||
resume: make(chan struct{}),
|
||||
}
|
||||
downloaded := make(chan struct{})
|
||||
|
||||
go func() {
|
||||
targetRouter(env).ServeHTTP(sw, req)
|
||||
close(downloaded)
|
||||
}()
|
||||
|
||||
<-sw.writing
|
||||
|
||||
edited := make(chan *httptest.ResponseRecorder, 1)
|
||||
|
||||
go func() {
|
||||
edited <- renameTarget(env, wh.ID, archive.ID)
|
||||
}()
|
||||
|
||||
select {
|
||||
case w := <-edited:
|
||||
assert.Equal(t, http.StatusSeeOther, w.Code)
|
||||
case <-time.After(10 * time.Second):
|
||||
t.Error("the rename waited for the download")
|
||||
}
|
||||
|
||||
close(sw.resume)
|
||||
<-downloaded
|
||||
assert.Equal(t, http.StatusOK, sw.Code)
|
||||
}
|
||||
|
||||
// seedArchive writes rows to the archive file at path, each with a
|
||||
// body of bodySize random bytes, which do not compress. Its table has
|
||||
// only the columns the test fills; an export writes the others empty.
|
||||
func seedArchive(t *testing.T, path string, rows, bodySize int) {
|
||||
t.Helper()
|
||||
|
||||
db, err := database.OpenSQLite(path, database.SQLiteModeCreate)
|
||||
require.NoError(t, err)
|
||||
|
||||
defer func() { require.NoError(t, db.Close()) }()
|
||||
|
||||
_, err = db.ExecContext(t.Context(),
|
||||
"CREATE TABLE archived_events (id INTEGER PRIMARY KEY, body TEXT)",
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
body := make([]byte, bodySize)
|
||||
|
||||
for range rows {
|
||||
_, _ = rand.Read(body)
|
||||
|
||||
_, err = db.ExecContext(t.Context(),
|
||||
"INSERT INTO archived_events (body) VALUES (?)", string(body),
|
||||
)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
}
|
||||
|
||||
// limitedServer serves the target routes as the server does, behind the
|
||||
// access log, whose lines it returns, and the request limit, here
|
||||
// limit, which is also its write timeout. Each connection's send buffer
|
||||
// is a few KiB, so a larger response is still being written while its
|
||||
// client is not reading.
|
||||
func limitedServer(
|
||||
t *testing.T, env *sourceTestEnv, limit time.Duration,
|
||||
) (*httptest.Server, *bytes.Buffer) {
|
||||
t.Helper()
|
||||
|
||||
const sendBuffer = 4 << 10
|
||||
|
||||
logBuf := new(bytes.Buffer)
|
||||
mw := middleware.NewForTest(
|
||||
slog.New(slog.NewJSONHandler(logBuf, nil)),
|
||||
&config.Config{Environment: config.EnvironmentDev},
|
||||
nil,
|
||||
)
|
||||
|
||||
srv := httptest.NewUnstartedServer(
|
||||
mw.Logging()(mw.Timeout(limit)(targetRouter(env))),
|
||||
)
|
||||
srv.Config.WriteTimeout = limit
|
||||
srv.Config.ConnContext = func(
|
||||
ctx context.Context, c net.Conn,
|
||||
) context.Context {
|
||||
tcp, ok := c.(*net.TCPConn)
|
||||
if assert.True(t, ok) {
|
||||
assert.NoError(t, tcp.SetWriteBuffer(sendBuffer))
|
||||
}
|
||||
|
||||
return ctx
|
||||
}
|
||||
srv.Start()
|
||||
t.Cleanup(srv.Close)
|
||||
|
||||
return srv, logBuf
|
||||
}
|
||||
|
||||
// TestHandleTargetDownload_OutlastsTheRequestLimit proves a download
|
||||
// runs for as long as the client keeps reading, and is logged as the
|
||||
// 200 it was. Behind a request limit and a server write timeout of a
|
||||
// tenth of a second, the client stops reading once the response has
|
||||
// started, waits three times as long, and still gets the whole file.
|
||||
// The archive is larger than the connection holds, so the download is
|
||||
// still being written while the client waits.
|
||||
func TestHandleTargetDownload_OutlastsTheRequestLimit(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
limit = 100 * time.Millisecond
|
||||
rows = 8
|
||||
bodySize = 64 << 10
|
||||
)
|
||||
|
||||
env := setupSourceTest(t)
|
||||
wh := seedWebhookWithRetention(t, env.db, 7)
|
||||
archive := seedTarget(t, env.db, wh.ID, database.TargetTypeDatabase)
|
||||
seedArchive(
|
||||
t, delivery.ArchivePath(env.dbMgr, &wh, archive), rows, bodySize,
|
||||
)
|
||||
|
||||
srv, accessLog := limitedServer(t, env, limit)
|
||||
|
||||
req, err := http.NewRequestWithContext(
|
||||
t.Context(), http.MethodGet,
|
||||
srv.URL+downloadPath(wh.ID, archive.ID), nil,
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
||||
for _, c := range env.cookies {
|
||||
req.AddCookie(c)
|
||||
}
|
||||
|
||||
resp, err := srv.Client().Do(req)
|
||||
require.NoError(t, err)
|
||||
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
|
||||
require.Equal(t, http.StatusOK, resp.StatusCode)
|
||||
|
||||
time.Sleep(3 * limit)
|
||||
|
||||
zr, err := gzip.NewReader(resp.Body)
|
||||
require.NoError(t, err)
|
||||
|
||||
var (
|
||||
got map[string]json.RawMessage
|
||||
events []json.RawMessage
|
||||
)
|
||||
|
||||
require.NoError(t, json.NewDecoder(zr).Decode(&got))
|
||||
require.NoError(t, json.Unmarshal(got["archived_events"], &events))
|
||||
assert.Len(t, events, rows)
|
||||
|
||||
// Reading to the end makes the gzip reader check that the file was
|
||||
// finished.
|
||||
_, err = io.ReadAll(zr)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Close waits for the handler, so the access log line is written.
|
||||
srv.Close()
|
||||
|
||||
var access map[string]any
|
||||
|
||||
require.NoError(t, json.Unmarshal(accessLog.Bytes(), &access))
|
||||
assert.EqualValues(t, http.StatusOK, access["status"])
|
||||
assert.GreaterOrEqual(t,
|
||||
access["latency_ms"], float64(limit.Milliseconds()),
|
||||
"the download must outlast the request limit",
|
||||
)
|
||||
}
|
||||
|
||||
// brokenWriter is a response writer whose writes fail once the
|
||||
// response has started, as they do when the client goes away.
|
||||
type brokenWriter struct {
|
||||
*httptest.ResponseRecorder
|
||||
}
|
||||
|
||||
func (b brokenWriter) Write(p []byte) (int, error) {
|
||||
if b.Body.Len() > 0 {
|
||||
return 0, errClientGone
|
||||
}
|
||||
|
||||
return b.ResponseRecorder.Write(p)
|
||||
}
|
||||
|
||||
// TestHandleTargetDownload_AbortsWhenItFails proves a download that
|
||||
// fails after its response has started aborts the connection, so the
|
||||
// client sees a failed download rather than a file that looks
|
||||
// complete and does not decompress.
|
||||
func TestHandleTargetDownload_AbortsWhenItFails(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := setupSourceTest(t)
|
||||
wh := seedWebhookWithRetention(t, env.db, 7)
|
||||
archive := seedTarget(t, env.db, wh.ID, database.TargetTypeDatabase)
|
||||
|
||||
req := httptest.NewRequestWithContext(
|
||||
t.Context(), http.MethodGet, downloadPath(wh.ID, archive.ID), nil,
|
||||
)
|
||||
for _, c := range env.cookies {
|
||||
req.AddCookie(c)
|
||||
}
|
||||
|
||||
w := brokenWriter{ResponseRecorder: httptest.NewRecorder()}
|
||||
|
||||
assert.PanicsWithValue(t, http.ErrAbortHandler, func() {
|
||||
targetRouter(env).ServeHTTP(w, req)
|
||||
})
|
||||
assert.Equal(t, http.StatusOK, w.Code)
|
||||
}
|
||||
@@ -37,14 +37,18 @@ const (
|
||||
editAuthHeader = "Authorization: Bearer " + editBearerSecret
|
||||
)
|
||||
|
||||
// targetRouter mounts the target create and edit routes on a chi
|
||||
// router so the handlers see the URL parameters they read.
|
||||
// targetRouter mounts the target create, edit and download routes on
|
||||
// a chi router so the handlers see the URL parameters they read.
|
||||
func targetRouter(env *sourceTestEnv) *chi.Mux {
|
||||
router := chi.NewRouter()
|
||||
router.Post(
|
||||
"/hook/{sourceID}/targets",
|
||||
env.handlers.HandleTargetCreate(),
|
||||
)
|
||||
router.Get(
|
||||
"/hook/{sourceID}/targets/{targetID}/download",
|
||||
env.handlers.HandleTargetDownload(),
|
||||
)
|
||||
router.Get(
|
||||
"/hook/{sourceID}/targets/{targetID}/edit",
|
||||
env.handlers.HandleTargetEdit(),
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
package middleware
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Timeout returns middleware that gives each request limit to finish:
|
||||
// it cancels the request's context once limit has passed, and answers
|
||||
// 504 when the handler then returns without having started its
|
||||
// response.
|
||||
//
|
||||
// It replaces chi's middleware.Timeout, which writes that 504 even
|
||||
// after the handler has sent its own status. A download that outlasts
|
||||
// the limit has already sent its 200 and the whole file, so the late
|
||||
// 504 changes nothing for the client: the access log and the metrics
|
||||
// would record it in place of the 200, and net/http would complain of
|
||||
// a superfluous WriteHeader.
|
||||
func (s *Middleware) Timeout(
|
||||
limit time.Duration,
|
||||
) func(http.Handler) http.Handler {
|
||||
return func(next http.Handler) http.Handler {
|
||||
return http.HandlerFunc(func(
|
||||
w http.ResponseWriter,
|
||||
r *http.Request,
|
||||
) {
|
||||
ctx, cancel := context.WithTimeout(r.Context(), limit)
|
||||
defer cancel()
|
||||
|
||||
tw := &timeoutResponseWriter{ResponseWriter: w}
|
||||
|
||||
next.ServeHTTP(tw, r.WithContext(ctx))
|
||||
|
||||
if !tw.started &&
|
||||
errors.Is(ctx.Err(), context.DeadlineExceeded) {
|
||||
w.WriteHeader(http.StatusGatewayTimeout)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// timeoutResponseWriter records whether the handler has started its
|
||||
// response.
|
||||
type timeoutResponseWriter struct {
|
||||
http.ResponseWriter
|
||||
|
||||
started bool
|
||||
}
|
||||
|
||||
func (w *timeoutResponseWriter) WriteHeader(code int) {
|
||||
w.started = true
|
||||
|
||||
w.ResponseWriter.WriteHeader(code)
|
||||
}
|
||||
|
||||
func (w *timeoutResponseWriter) Write(b []byte) (int, error) {
|
||||
// A Write without a WriteHeader starts the response too: net/http
|
||||
// sends 200 in front of it.
|
||||
w.started = true
|
||||
|
||||
//nolint:wrapcheck // Pass the writer's own error through unchanged.
|
||||
return w.ResponseWriter.Write(b)
|
||||
}
|
||||
|
||||
// Unwrap lets http.ResponseController reach the writer underneath, so
|
||||
// a handler can still set a write deadline through this wrapper.
|
||||
func (w *timeoutResponseWriter) Unwrap() http.ResponseWriter {
|
||||
return w.ResponseWriter
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
package middleware_test
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// TestTimeout proves the request limit answers 504 to a handler that
|
||||
// outlasts it without starting its response, and leaves a response the
|
||||
// handler has started with the status it sent. Both are what the
|
||||
// access log records.
|
||||
func TestTimeout(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const limit = 10 * time.Millisecond
|
||||
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
sent int // the status the handler sends, or 0 for none
|
||||
want int
|
||||
}{
|
||||
{name: "not started", sent: 0, want: http.StatusGatewayTimeout},
|
||||
{name: "started", sent: http.StatusOK, want: http.StatusOK},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
m, buf := capturingMiddleware(t)
|
||||
handler := m.Logging()(m.Timeout(limit)(http.HandlerFunc(
|
||||
func(w http.ResponseWriter, r *http.Request) {
|
||||
if tc.sent != 0 {
|
||||
w.WriteHeader(tc.sent)
|
||||
}
|
||||
|
||||
<-r.Context().Done()
|
||||
},
|
||||
)))
|
||||
|
||||
w := httptest.NewRecorder()
|
||||
handler.ServeHTTP(w, httptest.NewRequestWithContext(
|
||||
t.Context(), http.MethodGet, "/", nil,
|
||||
))
|
||||
|
||||
assert.Equal(t, tc.want, w.Code)
|
||||
|
||||
entries := accessLogEntries(t, buf)
|
||||
require.Len(t, entries, 1)
|
||||
assert.EqualValues(t, tc.want, entries[0]["status"])
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -39,6 +39,9 @@ const (
|
||||
// the mobile menu button instead of the navigation links.
|
||||
phoneWidth = 390
|
||||
phoneHeight = 844
|
||||
|
||||
// olderBody is the body of the event received before the newest.
|
||||
olderBody = "the older event"
|
||||
)
|
||||
|
||||
// TestAlpineRunsUnderTheSecurityPolicy loads the webhook page and the
|
||||
@@ -63,6 +66,7 @@ func TestAlpineRunsUnderTheSecurityPolicy(t *testing.T) {
|
||||
Active: true,
|
||||
},
|
||||
).Error)
|
||||
env.seedEvent(t, webhook.ID, olderBody)
|
||||
event := env.seedEvent(t, webhook.ID, `{"hello":"browser"}`)
|
||||
target := env.seedTarget(t, webhook.ID)
|
||||
dlv := env.seedFailedDelivery(t, webhook.ID, event.ID, target.ID)
|
||||
@@ -86,7 +90,7 @@ func TestAlpineRunsUnderTheSecurityPolicy(t *testing.T) {
|
||||
checkAddForms(ctx, t, page)
|
||||
checkTargetType(ctx, t, page+"/events")
|
||||
checkCopy(ctx, t, page)
|
||||
checkEntrypointEdit(ctx, t, page)
|
||||
checkRecentEvents(ctx, t, page)
|
||||
checkEventLog(ctx, t, page+"/events", event.ID, target.Name)
|
||||
checkMobileMenu(ctx, t, page)
|
||||
|
||||
@@ -364,60 +368,44 @@ func checkCopy(ctx context.Context, t *testing.T, url string) {
|
||||
`clicking Copy does not show "Copied"`)
|
||||
}
|
||||
|
||||
// checkEntrypointEdit loads a webhook page whose entrypoint has no
|
||||
// description, and checks that Edit shows the edit form in place of
|
||||
// the description and hides until the form closes, so the form always
|
||||
// opens on the saved description; that Cancel hides it and drops what
|
||||
// was typed; and that Save changes the description the page shows.
|
||||
func checkEntrypointEdit(ctx context.Context, t *testing.T, url string) {
|
||||
// checkRecentEvents loads a webhook page and checks that of its recent
|
||||
// events only the newest starts expanded, showing its body, that
|
||||
// clicking the older one's row expands it and clicking again collapses
|
||||
// it, and that clicking the newest one's row collapses it. It then
|
||||
// follows the newest one's Open link to the event's own page, which
|
||||
// shows the body.
|
||||
func checkRecentEvents(ctx context.Context, t *testing.T, url string) {
|
||||
t.Helper()
|
||||
|
||||
const (
|
||||
editForm = `form[action$="/edit"]`
|
||||
input = editForm + ` input[name="description"]`
|
||||
description = `//span[text()="Entrypoint"]`
|
||||
edit = `//button[text()="Edit"]`
|
||||
)
|
||||
// The newest event's body is pretty-printed JSON. Each row's
|
||||
// toggle is the button in the element that holds its state.
|
||||
newest := `//pre[contains(., '"hello": "browser"')]`
|
||||
older := `//pre[text()="` + olderBody + `"]`
|
||||
toggle := `/ancestor::div[@x-data][1]//button`
|
||||
|
||||
require.NoError(t, chromedp.Run(ctx, loadPage(url)))
|
||||
|
||||
assert.True(t, hidden(ctx, editForm),
|
||||
"the edit form shows before Edit is clicked")
|
||||
assert.True(t, shown(ctx, newest), "the newest event starts collapsed")
|
||||
assert.True(t, hidden(ctx, older), "an older event starts expanded")
|
||||
|
||||
click(ctx, t, edit)
|
||||
assert.True(t, shown(ctx, editForm),
|
||||
"clicking Edit does not show the edit form")
|
||||
assert.True(t, hidden(ctx, description),
|
||||
"the description stays shown beside the edit form")
|
||||
assert.True(t, hidden(ctx, edit),
|
||||
"Edit stays shown while the edit form is open")
|
||||
click(ctx, t, older+toggle)
|
||||
assert.True(t, shown(ctx, older), "clicking an event does not expand it")
|
||||
|
||||
require.NoError(t, chromedp.Run(
|
||||
ctx, chromedp.SendKeys(input, "draft", chromedp.ByQuery),
|
||||
))
|
||||
click(ctx, t, `//button[text()="Cancel"]`)
|
||||
assert.True(t, hidden(ctx, editForm),
|
||||
"clicking Cancel does not hide the edit form")
|
||||
assert.True(t, shown(ctx, description),
|
||||
"clicking Cancel does not show the description again")
|
||||
assert.True(t, shown(ctx, edit),
|
||||
"clicking Cancel does not show Edit again")
|
||||
click(ctx, t, older+toggle)
|
||||
assert.True(t, hidden(ctx, older),
|
||||
"clicking an event again does not collapse it")
|
||||
|
||||
var typed string
|
||||
click(ctx, t, newest+toggle)
|
||||
assert.True(t, hidden(ctx, newest),
|
||||
"clicking the newest event does not collapse it")
|
||||
|
||||
click(ctx, t, edit)
|
||||
require.NoError(t, chromedp.Run(
|
||||
ctx, chromedp.Value(input, &typed, chromedp.ByQuery),
|
||||
))
|
||||
assert.Empty(t, typed, "Cancel keeps what was typed")
|
||||
require.NoError(t, chromedp.Run(ctx, loadPage(url)))
|
||||
click(ctx, t, newest+`/ancestor::div[@x-data][1]//a[text()="Open"]`)
|
||||
|
||||
require.NoError(t, chromedp.Run(
|
||||
ctx, chromedp.SendKeys(input, "Billing sender", chromedp.ByQuery),
|
||||
))
|
||||
click(ctx, t, `//button[text()="Save"]`)
|
||||
|
||||
assert.True(t, shown(ctx, `//span[text()="Billing sender"]`),
|
||||
"saving the edit form does not change the description")
|
||||
assert.True(t, shown(ctx, `//h2[text()="Body"]`),
|
||||
"Open does not lead to the event's own page")
|
||||
assert.True(t, shown(ctx, newest),
|
||||
"the event's own page does not show its body")
|
||||
}
|
||||
|
||||
// checkEventLog loads the event log and checks that clicking an event's
|
||||
@@ -432,7 +420,7 @@ func checkEventLog(
|
||||
// The event's row shows its ID, and its Resubmit form is in the part
|
||||
// that expands. The delivery's row there shows the target's name.
|
||||
eventRow := `//span[text()="` + eventID + `"]`
|
||||
expanded := `form[action$="/resubmit"]`
|
||||
expanded := `form[action$="/` + eventID + `/resubmit"]`
|
||||
deliveryRow := `//span[text()="` + targetName + `"]`
|
||||
attempt := `//span[text()="Attempt 1"]`
|
||||
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
package server_test
|
||||
|
||||
import (
|
||||
"compress/gzip"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"regexp"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/gorm/clause"
|
||||
"sneak.berlin/go/webhooker/internal/database"
|
||||
)
|
||||
|
||||
// TestHook_DownloadArchive follows the Download link the webhook page
|
||||
// shows for a database target, and only for it, and gets the archive
|
||||
// as a gzipped JSON file. Signed out, the link leads to the login page.
|
||||
func TestHook_DownloadArchive(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := newTestEnv(t)
|
||||
|
||||
userID, _ := env.seedUser(t, "archivist", "somepassword")
|
||||
cookies := env.authCookies(t, userID, "archivist")
|
||||
wh := env.seedWebhook(t, userID)
|
||||
env.seedTarget(t, wh.ID)
|
||||
|
||||
archive := &database.Target{
|
||||
WebhookID: wh.ID,
|
||||
Name: "kept",
|
||||
Type: database.TargetTypeDatabase,
|
||||
Active: true,
|
||||
}
|
||||
require.NoError(t,
|
||||
env.db.DB().Omit(clause.Associations).Create(archive).Error,
|
||||
)
|
||||
|
||||
page := env.get("/hook/"+wh.ID, cookies)
|
||||
require.Equal(t, http.StatusOK, page.Code)
|
||||
|
||||
links := regexp.MustCompile(
|
||||
`href="(/hook/[^/"]+/targets/[^/"]+/download)"`,
|
||||
).FindAllStringSubmatch(page.Body.String(), -1)
|
||||
require.Len(t, links, 1, "only the database target has a Download")
|
||||
|
||||
link := links[0][1]
|
||||
assert.Equal(t,
|
||||
"/hook/"+wh.ID+"/targets/"+archive.ID+"/download", link,
|
||||
)
|
||||
|
||||
w := env.get(link, cookies)
|
||||
require.Equal(t, http.StatusOK, w.Code)
|
||||
assert.Equal(t, "application/gzip", w.Header().Get("Content-Type"))
|
||||
|
||||
zr, err := gzip.NewReader(w.Body)
|
||||
require.NoError(t, err)
|
||||
|
||||
var got map[string]json.RawMessage
|
||||
|
||||
require.NoError(t, json.NewDecoder(zr).Decode(&got))
|
||||
assert.JSONEq(t,
|
||||
`{"id":"`+wh.ID+`","name":"routed"}`, string(got["webhook"]),
|
||||
)
|
||||
|
||||
w = env.get(link, nil)
|
||||
assert.Equal(t, http.StatusSeeOther, w.Code)
|
||||
assert.Contains(t, w.Header().Get("Location"), "/pages/login")
|
||||
}
|
||||
+11
-10
@@ -73,7 +73,7 @@ func (s *Server) setupGlobalMiddleware() {
|
||||
}
|
||||
|
||||
s.router.Use(s.mw.CORS())
|
||||
s.router.Use(middleware.Timeout(requestTimeout))
|
||||
s.router.Use(s.mw.Timeout(requestTimeout))
|
||||
|
||||
// Panic recovery, deliberately here rather than first. It has to
|
||||
// run inside every middleware that observes the response, so the
|
||||
@@ -252,11 +252,12 @@ func (s *Server) setupSourceRoutes() {
|
||||
r.Post("/edit", s.h.HandleSourceEditSubmit())
|
||||
r.Post("/delete", s.h.HandleSourceDelete())
|
||||
r.Get("/events", s.h.HandleSourceLogs())
|
||||
// The log page renders each body only up to its cap, so
|
||||
// this is the only route that serves a whole one. It
|
||||
// belongs to this group for its RequireAuth and
|
||||
// NoCache; see HandleEventBodyDownload for the headers
|
||||
// that keep the bytes it returns inert.
|
||||
r.Get("/events/{eventID}", s.h.HandleEventDetail())
|
||||
// The pages show a body as escaped text and leave a
|
||||
// binary one out, so this is the only route that serves
|
||||
// the stored bytes. It belongs to this group for its
|
||||
// RequireAuth and NoCache; see HandleEventBodyDownload for
|
||||
// the headers that keep the bytes it returns inert.
|
||||
r.Get(
|
||||
"/events/{eventID}/body",
|
||||
s.h.HandleEventBodyDownload(),
|
||||
@@ -289,10 +290,6 @@ func (s *Server) setupSourceRoutes() {
|
||||
"/entrypoints",
|
||||
s.h.HandleEntrypointCreate(),
|
||||
)
|
||||
r.Post(
|
||||
"/entrypoints/{entrypointID}/edit",
|
||||
s.h.HandleEntrypointEdit(),
|
||||
)
|
||||
r.Post(
|
||||
"/entrypoints/{entrypointID}/delete",
|
||||
s.h.HandleEntrypointDelete(),
|
||||
@@ -316,6 +313,10 @@ func (s *Server) setupSourceRoutes() {
|
||||
"/targets/{targetID}/edit",
|
||||
s.h.HandleTargetEditSubmit(),
|
||||
)
|
||||
r.Get(
|
||||
"/targets/{targetID}/download",
|
||||
s.h.HandleTargetDownload(),
|
||||
)
|
||||
r.Post(
|
||||
"/targets/{targetID}/delete",
|
||||
s.h.HandleTargetDelete(),
|
||||
|
||||
+43
-152
@@ -12,7 +12,6 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
"go.uber.org/fx"
|
||||
@@ -365,6 +364,7 @@ func (e *testEnv) seedEvent(
|
||||
WebhookID: webhookID,
|
||||
Method: http.MethodPost,
|
||||
Body: body,
|
||||
BodyBytes: int64(len(body)),
|
||||
ContentType: "application/octet-stream",
|
||||
}
|
||||
|
||||
@@ -399,44 +399,6 @@ func (e *testEnv) seedTarget(
|
||||
return tgt
|
||||
}
|
||||
|
||||
// seedEntrypoint creates an active entrypoint for a webhook.
|
||||
func (e *testEnv) seedEntrypoint(
|
||||
t *testing.T,
|
||||
webhookID string,
|
||||
) *database.Entrypoint {
|
||||
t.Helper()
|
||||
|
||||
ep := &database.Entrypoint{
|
||||
WebhookID: webhookID,
|
||||
Path: uuid.New().String(),
|
||||
Description: "Default entrypoint",
|
||||
Active: true,
|
||||
}
|
||||
|
||||
require.NoError(
|
||||
t,
|
||||
e.db.DB().Omit(clause.Associations).Create(ep).Error,
|
||||
)
|
||||
|
||||
return ep
|
||||
}
|
||||
|
||||
// storedEntrypoint reloads an entrypoint row.
|
||||
func (e *testEnv) storedEntrypoint(
|
||||
t *testing.T,
|
||||
entrypointID string,
|
||||
) database.Entrypoint {
|
||||
t.Helper()
|
||||
|
||||
var ep database.Entrypoint
|
||||
|
||||
require.NoError(
|
||||
t, e.db.DB().First(&ep, "id = ?", entrypointID).Error,
|
||||
)
|
||||
|
||||
return ep
|
||||
}
|
||||
|
||||
// seedFailedDelivery records a terminally failed delivery of an event
|
||||
// to a target in the webhook's own database.
|
||||
func (e *testEnv) seedFailedDelivery(
|
||||
@@ -1126,119 +1088,6 @@ func TestHook_EntrypointActions(t *testing.T) {
|
||||
assert.Zero(t, left, "the delete should remove the entrypoint")
|
||||
}
|
||||
|
||||
// TestHook_EntrypointEdit changes an entrypoint's description with the
|
||||
// edit form on the webhook page, then empties it. The entrypoint keeps
|
||||
// its URL, and with no description it shows as "Entrypoint". Without
|
||||
// the CSRF token, or without a session, the edit is refused.
|
||||
func TestHook_EntrypointEdit(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := newTestEnv(t)
|
||||
|
||||
userID, _ := env.seedUser(t, "epeditor", "somepassword")
|
||||
cookies := env.authCookies(t, userID, "epeditor")
|
||||
wh := env.seedWebhook(t, userID)
|
||||
ep := env.seedEntrypoint(t, wh.ID)
|
||||
page := "/hook/" + wh.ID
|
||||
|
||||
token, cookies := env.csrfFrom(t, page, cookies)
|
||||
action := env.urlFrom(
|
||||
t, page, `action="(/hook/[^/"]+/entrypoints/[^/"]+/edit)"`,
|
||||
cookies,
|
||||
)
|
||||
|
||||
assert.Equal(
|
||||
t, http.StatusForbidden,
|
||||
env.post(action, entrypointEditForm("", "no token"), cookies).Code,
|
||||
"an edit without a CSRF token must be refused",
|
||||
)
|
||||
|
||||
// The request without a session carries a valid CSRF token from
|
||||
// the login page, so only the session check can refuse it.
|
||||
anonToken, anon := env.csrfFrom(t, "/pages/login", nil)
|
||||
refused := env.post(
|
||||
action, entrypointEditForm(anonToken, "no session"), anon,
|
||||
)
|
||||
assert.Equal(t, http.StatusSeeOther, refused.Code)
|
||||
assert.Equal(
|
||||
t, "/pages/login", refused.Header().Get("Location"),
|
||||
"an edit without a session must be refused",
|
||||
)
|
||||
|
||||
assert.Equal(
|
||||
t, ep.Description, env.storedEntrypoint(t, ep.ID).Description,
|
||||
"a refused edit must not change the description",
|
||||
)
|
||||
|
||||
// edit submits the form with description and requires the
|
||||
// redirect back to the webhook page with the notice.
|
||||
edit := func(description string) {
|
||||
t.Helper()
|
||||
|
||||
w := env.post(
|
||||
action, entrypointEditForm(token, description), cookies,
|
||||
)
|
||||
env.requireNotice(t, w, page, "entrypoint-saved",
|
||||
"Entrypoint description saved.", cookies)
|
||||
}
|
||||
|
||||
edit("Billing sender")
|
||||
|
||||
stored := env.storedEntrypoint(t, ep.ID)
|
||||
assert.Equal(t, "Billing sender", stored.Description)
|
||||
assert.Equal(t, ep.Path, stored.Path,
|
||||
"the edit must keep the entrypoint's URL")
|
||||
|
||||
body := env.get(page, cookies).Body.String()
|
||||
assert.Contains(t, body, ">Billing sender</span>")
|
||||
assert.Contains(t, body, "/h/"+ep.Path+"</code>")
|
||||
|
||||
edit("")
|
||||
|
||||
assert.Empty(t, env.storedEntrypoint(t, ep.ID).Description)
|
||||
assert.Contains(t, env.get(page, cookies).Body.String(),
|
||||
">Entrypoint</span>", "no description shows as Entrypoint")
|
||||
}
|
||||
|
||||
// TestHook_EntrypointEdit_OtherUser404s has another logged-in user, with
|
||||
// a CSRF token of their own, try to edit an entrypoint: through the
|
||||
// owner's webhook, and through a webhook of their own. Both are 404s
|
||||
// and the description stays as it was.
|
||||
func TestHook_EntrypointEdit_OtherUser404s(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := newTestEnv(t)
|
||||
|
||||
ownerID, _ := env.seedUser(t, "epowner", "somepassword")
|
||||
wh := env.seedWebhook(t, ownerID)
|
||||
ep := env.seedEntrypoint(t, wh.ID)
|
||||
|
||||
otherID, _ := env.seedUser(t, "epother", "somepassword")
|
||||
other := env.authCookies(t, otherID, "epother")
|
||||
theirs := env.seedWebhook(t, otherID)
|
||||
token, other := env.csrfFrom(t, "/hook/"+theirs.ID, other)
|
||||
|
||||
for _, webhookID := range []string{wh.ID, theirs.ID} {
|
||||
path := "/hook/" + webhookID + "/entrypoints/" + ep.ID + "/edit"
|
||||
|
||||
w := env.post(path, entrypointEditForm(token, "not theirs"), other)
|
||||
assert.Equal(t, http.StatusNotFound, w.Code, path)
|
||||
}
|
||||
|
||||
assert.Equal(
|
||||
t, ep.Description, env.storedEntrypoint(t, ep.ID).Description,
|
||||
"another user must not change the description",
|
||||
)
|
||||
}
|
||||
|
||||
// entrypointEditForm fills in the webhook page's entrypoint edit form.
|
||||
func entrypointEditForm(token, description string) url.Values {
|
||||
return url.Values{
|
||||
"csrf_token": {token},
|
||||
"description": {description},
|
||||
}
|
||||
}
|
||||
|
||||
// TestHook_TargetActions adds a target with the form on the webhook
|
||||
// page, follows its Edit link to the target edit form and submits
|
||||
// it, then deactivates, activates and deletes it, every URL and token
|
||||
@@ -1416,6 +1265,48 @@ func TestHook_LinksBetweenPages(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestEventPage_OpenedFromRecentEvents follows the Open link of a
|
||||
// row in the recent events on the webhook page through the
|
||||
// production router to the event's own page, which shows the body
|
||||
// and links back. Another user gets a 404 at the same URL, and a
|
||||
// logged-out request is sent to log in.
|
||||
func TestEventPage_OpenedFromRecentEvents(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
env := newTestEnv(t)
|
||||
|
||||
ownerID, _ := env.seedUser(t, "owner", "somepassword")
|
||||
cookies := env.authCookies(t, ownerID, "owner")
|
||||
wh := env.seedWebhook(t, ownerID)
|
||||
evt := env.seedEvent(t, wh.ID, "OWNERS-PAYLOAD-3e9d")
|
||||
|
||||
page := "/hook/" + wh.ID
|
||||
path := env.urlFrom(t, page, `href="([^"]+)"[^>]*>Open<`, cookies)
|
||||
|
||||
require.Equal(t, page+"/events/"+evt.ID, path)
|
||||
|
||||
w := env.get(path, cookies)
|
||||
require.Equal(t, http.StatusOK, w.Code)
|
||||
assert.Contains(t, w.Body.String(), "OWNERS-PAYLOAD-3e9d")
|
||||
assert.Equal(
|
||||
t, page,
|
||||
env.urlFrom(t, path, `href="([^"]+)"[^>]*>← Back to `, cookies),
|
||||
)
|
||||
|
||||
intruderID, _ := env.seedUser(t, "intruder", "somepassword")
|
||||
|
||||
w = env.get(path, env.authCookies(t, intruderID, "intruder"))
|
||||
assert.Equal(t, http.StatusNotFound, w.Code)
|
||||
assert.NotContains(t, w.Body.String(), "OWNERS-PAYLOAD-3e9d")
|
||||
|
||||
anon := env.get(path, nil)
|
||||
assert.Equal(t, http.StatusSeeOther, anon.Code)
|
||||
assert.Equal(
|
||||
t, "/pages/login?next="+url.QueryEscape(path),
|
||||
anon.Header().Get("Location"),
|
||||
)
|
||||
}
|
||||
|
||||
// TestSourceLogs_TruncationLinkDownloadsTheBody walks the whole
|
||||
// feature the way a user does: render the event log page through
|
||||
// the production router, take the download URL out of the markup
|
||||
|
||||
+6
-2
@@ -70,11 +70,15 @@ document.addEventListener("alpine:init", function () {
|
||||
"use strict";
|
||||
|
||||
// Something a click shows and hides: the mobile menu, an add form,
|
||||
// an entrypoint's edit form, an event in the event log, a delivery's
|
||||
// attempts.
|
||||
// an event in the event log or in the recent events, a delivery's
|
||||
// attempts. It starts hidden, or shown when its element has the
|
||||
// data-open attribute.
|
||||
window.Alpine.data("collapsible", function () {
|
||||
return {
|
||||
open: false,
|
||||
init() {
|
||||
this.open = this.$root.hasAttribute("data-open");
|
||||
},
|
||||
toggle() {
|
||||
this.open = !this.open;
|
||||
},
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
{{define "delivery_attempts"}}
|
||||
<!-- A delivery's recorded attempts, as handlers.DeliveryView holds
|
||||
them: in the event log and on an event's own page. -->
|
||||
{{if .AttemptsOmitted}}
|
||||
<p class="text-xs text-gray-500">{{.AttemptsOmitted}} attempt{{if ne .AttemptsOmitted 1}}s{{end}} omitted between the first and last shown.</p>
|
||||
{{end}}
|
||||
{{range .Results}}
|
||||
<div class="rounded-md bg-white border border-gray-200 p-2">
|
||||
<div class="flex flex-wrap items-center gap-3 text-xs">
|
||||
<span class="text-gray-500">Attempt {{.AttemptNum}}</span>
|
||||
<span class="{{if .Success}}text-green-600{{else}}text-red-600{{end}}">{{if .Success}}success{{else}}failure{{end}}</span>
|
||||
<span class="text-gray-500">Status: {{if .HasStatusCode}}{{.StatusCode}}{{else}}— (no response){{end}}</span>
|
||||
<span class="text-gray-500">Duration: {{.DurationMS}} ms</span>
|
||||
</div>
|
||||
{{if .Error}}
|
||||
<p class="mt-2 text-xs text-red-700 break-all">Error: {{.Error}}</p>
|
||||
{{end}}
|
||||
{{if .ResponseBody}}
|
||||
<pre class="mt-2 text-xs text-gray-700 overflow-x-auto whitespace-pre-wrap break-all">{{.ResponseBody}}</pre>
|
||||
{{end}}
|
||||
{{if .ResponseTruncated}}
|
||||
{{if .ResponseSizeKnown}}
|
||||
<p class="mt-1 text-xs text-gray-500">Response truncated for display: showing {{.ResponseShownBytes}} of {{.ResponseBytes}} bytes.</p>
|
||||
{{else}}
|
||||
<p class="mt-1 text-xs text-gray-500">Showing {{.ResponseShownBytes}} of the {{.ResponseBytes}} recorded bytes. The response reached the recording limit, so the remote may have sent more that was never stored.</p>
|
||||
{{end}}
|
||||
{{end}}
|
||||
</div>
|
||||
{{else}}
|
||||
<p class="text-xs text-gray-500">No attempts recorded yet.</p>
|
||||
{{end}}
|
||||
{{end}}
|
||||
@@ -0,0 +1,15 @@
|
||||
{{define "event_body"}}
|
||||
<!-- An event's body, as handlers.BodyView describes it: the same in
|
||||
the recent events on the webhook page, the event log and the
|
||||
event's own page. -->
|
||||
{{if .Binary}}
|
||||
<p class="text-xs text-gray-500">This body is binary ({{.Size}} bytes) and is not shown. <a href="{{.EventURL}}/body" class="btn-small">Download the body</a></p>
|
||||
{{else if .Text}}
|
||||
<pre class="rounded-md border border-gray-200 bg-white p-2 text-xs text-gray-700 overflow-x-auto whitespace-pre-wrap break-all"{{if .Scroll}} style="max-height: 32rem; overflow-y: auto"{{end}}>{{.Text}}</pre>
|
||||
{{if .Cut}}
|
||||
<p class="mt-2 text-xs text-gray-500">Showing the first {{.ShownBytes}} of {{.Size}} bytes, unformatted. <a href="{{.EventURL}}" class="btn-small">Show the whole body</a> <a href="{{.EventURL}}/body" class="btn-small">Download the body</a></p>
|
||||
{{end}}
|
||||
{{else}}
|
||||
<p class="text-xs text-gray-500">No body.</p>
|
||||
{{end}}
|
||||
{{end}}
|
||||
@@ -0,0 +1,87 @@
|
||||
{{template "base" .}}
|
||||
|
||||
{{define "title"}}Event - {{.Webhook.Name}} - Webhooker{{end}}
|
||||
|
||||
{{define "content"}}
|
||||
<div class="max-w-6xl mx-auto px-6 py-8">
|
||||
<div class="mb-6">
|
||||
<div class="flex flex-wrap gap-2">
|
||||
<a href="/hook/{{.Webhook.ID}}" class="btn-small">← Back to {{.Webhook.Name}}</a>
|
||||
<a href="/hook/{{.Webhook.ID}}/events" class="btn-small">Full Event Log</a>
|
||||
</div>
|
||||
<h1 class="text-2xl font-medium text-gray-900 mt-2">Event</h1>
|
||||
</div>
|
||||
|
||||
{{with .Event}}
|
||||
<div class="card p-4">
|
||||
<dl class="space-y-2 text-sm">
|
||||
<div class="flex flex-wrap gap-2">
|
||||
<dt class="w-32 flex-shrink-0 text-gray-500">ID</dt>
|
||||
<dd class="font-mono text-gray-900 break-all">{{.ID}}</dd>
|
||||
</div>
|
||||
<div class="flex flex-wrap gap-2">
|
||||
<dt class="w-32 flex-shrink-0 text-gray-500">Received</dt>
|
||||
<dd class="text-gray-900">{{.CreatedAt.UTC.Format "2006-01-02 15:04:05"}} UTC</dd>
|
||||
</div>
|
||||
<div class="flex flex-wrap gap-2">
|
||||
<dt class="w-32 flex-shrink-0 text-gray-500">Method</dt>
|
||||
<dd><span class="badge-info">{{.Method}}</span></dd>
|
||||
</div>
|
||||
<div class="flex flex-wrap gap-2">
|
||||
<dt class="w-32 flex-shrink-0 text-gray-500">Content type</dt>
|
||||
<dd class="text-gray-900 break-all">{{.ContentType}}</dd>
|
||||
</div>
|
||||
<div class="flex flex-wrap gap-2">
|
||||
<dt class="w-32 flex-shrink-0 text-gray-500">Body size</dt>
|
||||
<dd class="text-gray-900">{{.Body.Size}} bytes</dd>
|
||||
</div>
|
||||
{{if .ResubmittedFrom}}
|
||||
<div class="flex flex-wrap gap-2">
|
||||
<dt class="w-32 flex-shrink-0 text-gray-500">Resubmitted from</dt>
|
||||
<dd><a href="/hook/{{$.Webhook.ID}}/events/{{.ResubmittedFromID}}" class="btn-small font-mono">{{.ResubmittedFromID}}</a></dd>
|
||||
</div>
|
||||
{{end}}
|
||||
{{if .ResubmitCount}}
|
||||
<div class="flex flex-wrap gap-2">
|
||||
<dt class="w-32 flex-shrink-0 text-gray-500">Resubmitted</dt>
|
||||
<dd class="text-gray-900">as {{.ResubmitCount}} new event{{if ne .ResubmitCount 1}}s{{end}}</dd>
|
||||
</div>
|
||||
{{end}}
|
||||
</dl>
|
||||
</div>
|
||||
|
||||
<div class="card mt-6">
|
||||
<div class="p-4 border-b border-gray-200">
|
||||
<h2 class="text-lg font-medium text-gray-900">Body</h2>
|
||||
</div>
|
||||
<div class="p-4">
|
||||
{{template "event_body" .Body}}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="card mt-6">
|
||||
<div class="p-4 border-b border-gray-200">
|
||||
<h2 class="text-lg font-medium text-gray-900">Deliveries</h2>
|
||||
</div>
|
||||
<div class="divide-y divide-gray-100">
|
||||
{{range .Deliveries}}
|
||||
<div class="p-4">
|
||||
<div class="flex flex-wrap items-center justify-between gap-3">
|
||||
<span class="text-sm text-gray-700">{{.Target.DisplayName}}</span>
|
||||
<span class="flex flex-wrap items-center gap-3">
|
||||
<span class="text-xs {{if eq .Status "delivered"}}text-green-600{{else if eq .Status "failed"}}text-red-600{{else if eq .Status "retrying"}}text-yellow-600{{else}}text-gray-400{{end}}">{{.Status}}</span>
|
||||
<span class="text-xs text-gray-400">{{.AttemptCount}} attempt{{if ne .AttemptCount 1}}s{{end}}</span>
|
||||
</span>
|
||||
</div>
|
||||
<div class="mt-2 space-y-2">
|
||||
{{template "delivery_attempts" .}}
|
||||
</div>
|
||||
</div>
|
||||
{{else}}
|
||||
<div class="p-4 text-sm text-gray-500">No deliveries.</div>
|
||||
{{end}}
|
||||
</div>
|
||||
</div>
|
||||
{{end}}
|
||||
</div>
|
||||
{{end}}
|
||||
@@ -54,26 +54,15 @@
|
||||
|
||||
<div class="divide-y divide-gray-100">
|
||||
{{range .Entrypoints}}
|
||||
<div class="p-4" x-data="collapsible">
|
||||
<div class="p-4">
|
||||
<div class="flex flex-wrap items-center justify-between gap-2 mb-1">
|
||||
<span x-show="closed" class="text-sm font-medium text-gray-900">{{if .Description}}{{.Description}}{{else}}Entrypoint{{end}}</span>
|
||||
<!-- Edit shows this form in place of the
|
||||
description and hides until it closes, so
|
||||
the form always opens on the saved
|
||||
description. Cancel resets what was typed. -->
|
||||
<form x-show="open" x-cloak method="POST" action="/hook/{{$.Webhook.ID}}/entrypoints/{{.ID}}/edit" class="flex w-full gap-2">
|
||||
<input type="hidden" name="csrf_token" value="{{$.CSRFToken}}">
|
||||
<input type="text" name="description" value="{{.Description}}" placeholder="Description (optional)" class="input text-sm flex-1">
|
||||
<button type="submit" class="btn-primary text-sm">Save</button>
|
||||
<button type="reset" @click="toggle" class="btn-secondary text-sm">Cancel</button>
|
||||
</form>
|
||||
<span class="text-sm font-medium text-gray-900">{{if .Description}}{{.Description}}{{else}}Entrypoint{{end}}</span>
|
||||
<div class="flex flex-wrap items-center gap-2">
|
||||
{{if .Active}}
|
||||
<span class="badge-success">Active</span>
|
||||
{{else}}
|
||||
<span class="badge-error">Inactive</span>
|
||||
{{end}}
|
||||
<button type="button" x-show="closed" @click="toggle" class="btn-small" title="Edit">Edit</button>
|
||||
<form method="POST" action="/hook/{{$.Webhook.ID}}/entrypoints/{{.ID}}/toggle" class="inline">
|
||||
<input type="hidden" name="csrf_token" value="{{$.CSRFToken}}">
|
||||
<button type="submit" class="btn-small" title="{{if .Active}}Deactivate{{else}}Activate{{end}}">
|
||||
@@ -168,6 +157,9 @@
|
||||
{{else}}
|
||||
<span class="badge-error">Inactive</span>
|
||||
{{end}}
|
||||
{{if eq .Type "database"}}
|
||||
<a href="/hook/{{$.Webhook.ID}}/targets/{{.ID}}/download" class="btn-small" title="Download the archive as gzipped JSON">Download</a>
|
||||
{{end}}
|
||||
<a href="/hook/{{$.Webhook.ID}}/targets/{{.ID}}/edit" class="btn-small" title="Edit">Edit</a>
|
||||
<form method="POST" action="/hook/{{$.Webhook.ID}}/targets/{{.ID}}/toggle" class="inline">
|
||||
<input type="hidden" name="csrf_token" value="{{$.CSRFToken}}">
|
||||
@@ -202,26 +194,37 @@
|
||||
<a href="/hook/{{.Webhook.ID}}/events" class="btn-small">Full Event Log</a>
|
||||
</div>
|
||||
<div class="divide-y divide-gray-100">
|
||||
{{range .Events}}
|
||||
<div class="p-4">
|
||||
<div class="flex flex-wrap items-center justify-between gap-3">
|
||||
<div class="flex flex-wrap items-center gap-3">
|
||||
<span class="badge-info">{{.Method}}</span>
|
||||
<span class="text-sm text-gray-500 break-all">{{.ContentType}}</span>
|
||||
{{if .ResubmittedFromID}}
|
||||
<span class="text-xs text-gray-500" title="This event is a copy of {{.ResubmittedFromID}}">resubmitted copy</span>
|
||||
{{end}}
|
||||
</div>
|
||||
<div class="flex flex-wrap items-center gap-3 text-xs text-gray-400">
|
||||
<span title="Body size">{{.Size}}</span>
|
||||
{{if .ProcessingTime}}
|
||||
<span title="Processing time: how long the slowest delivery took, from being queued to its last attempt">{{.ProcessingTime}}</span>
|
||||
{{end}}
|
||||
{{if .Status}}
|
||||
<span class="font-medium {{.StatusClass}}" title="HTTP status from the HTTP target">{{.Status}}</span>
|
||||
{{end}}
|
||||
<span title="{{.ReceivedUTC}}">{{.Received}}</span>
|
||||
</div>
|
||||
<!-- Each row expands to show its body; only the newest
|
||||
starts expanded. -->
|
||||
{{range $i, $event := .Events}}
|
||||
<div class="p-4" x-data="collapsible"{{if eq $i 0}} data-open{{end}}>
|
||||
<div class="flex flex-wrap items-center gap-3">
|
||||
<button type="button" class="btn-small flex-1 flex-wrap justify-between gap-3 text-left" @click="toggle">
|
||||
<span class="flex flex-wrap items-center gap-3">
|
||||
<span class="badge-info">{{.Method}}</span>
|
||||
<span class="text-sm text-gray-500 break-all">{{.ContentType}}</span>
|
||||
{{if .ResubmittedFromID}}
|
||||
<span class="text-xs text-gray-500" title="This event is a copy of {{.ResubmittedFromID}}">resubmitted copy</span>
|
||||
{{end}}
|
||||
</span>
|
||||
<span class="flex flex-wrap items-center gap-3 text-xs text-gray-400">
|
||||
<span title="Body size">{{.Size}}</span>
|
||||
{{if .ProcessingTime}}
|
||||
<span title="Processing time: how long the slowest delivery took, from being queued to its last attempt">{{.ProcessingTime}}</span>
|
||||
{{end}}
|
||||
{{if .Status}}
|
||||
<span class="font-medium {{.StatusClass}}" title="HTTP status from the HTTP target">{{.Status}}</span>
|
||||
{{end}}
|
||||
<span title="{{.ReceivedUTC}}">{{.Received}}</span>
|
||||
<svg class="w-4 h-4 text-gray-400 transition-transform" :class="caretClass" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
||||
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M19 9l-7 7-7-7"/>
|
||||
</svg>
|
||||
</span>
|
||||
</button>
|
||||
<a href="{{.Body.EventURL}}" class="btn-small">Open</a>
|
||||
</div>
|
||||
<div x-show="open" x-cloak class="mt-3">
|
||||
{{template "event_body" .Body}}
|
||||
</div>
|
||||
</div>
|
||||
{{else}}
|
||||
|
||||
@@ -44,7 +44,7 @@
|
||||
<div x-show="open" x-cloak class="mt-3 p-3 bg-gray-50 rounded-md">
|
||||
<div class="mb-3 flex flex-wrap items-center justify-between gap-2">
|
||||
<div class="text-xs text-gray-500">
|
||||
{{if .ResubmittedFrom}}Resubmitted from event <span class="font-mono">{{.ResubmittedFromID}}</span>.{{end}}
|
||||
{{if .ResubmittedFrom}}Resubmitted from event <a href="/hook/{{$.Webhook.ID}}/events/{{.ResubmittedFromID}}" class="btn-small font-mono">{{.ResubmittedFromID}}</a>.{{end}}
|
||||
{{if .ResubmitCount}}Resubmitted as {{.ResubmitCount}} new event{{if ne .ResubmitCount 1}}s{{end}}.{{end}}
|
||||
</div>
|
||||
<form method="POST" action="/hook/{{$.Webhook.ID}}/events/{{.ID}}/resubmit" class="inline">
|
||||
@@ -53,10 +53,7 @@
|
||||
<button type="submit" class="btn-small" title="Submit this event again as a new event, to every currently active target">Resubmit</button>
|
||||
</form>
|
||||
</div>
|
||||
<pre class="text-xs text-gray-700 overflow-x-auto whitespace-pre-wrap break-all">{{.Body}}</pre>
|
||||
{{if .BodyTruncated}}
|
||||
<p class="mt-2 text-xs text-gray-500">Body truncated for display: showing {{.BodyShownBytes}} of {{.BodyBytes}} bytes. The stored body is unchanged — <a href="/hook/{{$.Webhook.ID}}/events/{{.ID}}/body" class="btn-small">download the full body</a>.</p>
|
||||
{{end}}
|
||||
{{template "event_body" .Body}}
|
||||
|
||||
{{if .Deliveries}}
|
||||
<div class="mt-4 border-t border-gray-200 pt-3">
|
||||
@@ -87,34 +84,7 @@
|
||||
</div>
|
||||
|
||||
<div x-show="open" x-cloak class="mt-2 space-y-2">
|
||||
{{if .AttemptsOmitted}}
|
||||
<p class="text-xs text-gray-500">{{.AttemptsOmitted}} attempt{{if ne .AttemptsOmitted 1}}s{{end}} omitted between the first and last shown.</p>
|
||||
{{end}}
|
||||
{{range .Results}}
|
||||
<div class="rounded-md bg-white border border-gray-200 p-2">
|
||||
<div class="flex flex-wrap items-center gap-3 text-xs">
|
||||
<span class="text-gray-500">Attempt {{.AttemptNum}}</span>
|
||||
<span class="{{if .Success}}text-green-600{{else}}text-red-600{{end}}">{{if .Success}}success{{else}}failure{{end}}</span>
|
||||
<span class="text-gray-500">Status: {{if .HasStatusCode}}{{.StatusCode}}{{else}}— (no response){{end}}</span>
|
||||
<span class="text-gray-500">Duration: {{.DurationMS}} ms</span>
|
||||
</div>
|
||||
{{if .Error}}
|
||||
<p class="mt-2 text-xs text-red-700 break-all">Error: {{.Error}}</p>
|
||||
{{end}}
|
||||
{{if .ResponseBody}}
|
||||
<pre class="mt-2 text-xs text-gray-700 overflow-x-auto whitespace-pre-wrap break-all">{{.ResponseBody}}</pre>
|
||||
{{end}}
|
||||
{{if .ResponseTruncated}}
|
||||
{{if .ResponseSizeKnown}}
|
||||
<p class="mt-1 text-xs text-gray-500">Response truncated for display: showing {{.ResponseShownBytes}} of {{.ResponseBytes}} bytes.</p>
|
||||
{{else}}
|
||||
<p class="mt-1 text-xs text-gray-500">Showing {{.ResponseShownBytes}} of the {{.ResponseBytes}} recorded bytes. The response reached the recording limit, so the remote may have sent more that was never stored.</p>
|
||||
{{end}}
|
||||
{{end}}
|
||||
</div>
|
||||
{{else}}
|
||||
<p class="text-xs text-gray-500">No attempts recorded yet.</p>
|
||||
{{end}}
|
||||
{{template "delivery_attempts" .}}
|
||||
</div>
|
||||
</div>
|
||||
{{end}}
|
||||
|
||||
Reference in New Issue
Block a user