All checks were successful
check / check (push) Successful in 2m44s
The 8 KB render cap from #135 left storage untouched but no route served the rest, so a body over the cap was reachable only with filesystem access to the SQLite files — in a product whose purpose is storing webhooks so they can be inspected. GET /source/{sourceID}/logs/{eventID}/body serves the whole body to the webhook's owner, as application/octet-stream with an attachment disposition and nosniff. Those are a security control, not formatting: the bytes come from the public receiver and are handed back inside the operator's authenticated origin, and the existing CSP would not stop a stored HTML payload executing there. The truncation marker links to it only when a body was actually cut. Accepted deviation, documented rather than glossed: #157's definition of done asks the route to stream from the row. It buffers whole instead, because database/sql exposes no incremental handle on a SQLite BLOB and substr range reads re-materialise the entire column per call — an earlier revision chunked at 64 KiB and was 11-15x slower for a worse bound. Three independent reviewers confirmed no streaming path exists. Independently reviewed three times. Two earlier revisions each asserted a memory bound the code did not have; the final reviewer measured 2.057x at the ingest cap and pinned the two overlapping allocations from source — the driver's column buffer and database/sql's convertAssign clone — confirming the stated "roughly two bodies, and 2x is a floor not a ceiling" is now accurate, since SQLite's own materialisation sits outside the Go heap.
200 lines
6.5 KiB
Go
200 lines
6.5 KiB
Go
package handlers
|
|
|
|
import (
|
|
"database/sql"
|
|
"errors"
|
|
"net/http"
|
|
"strconv"
|
|
|
|
"github.com/go-chi/chi"
|
|
"github.com/google/uuid"
|
|
"gorm.io/gorm"
|
|
"sneak.berlin/go/webhooker/internal/database"
|
|
)
|
|
|
|
// eventBodyQuery reads one event's stored body as bytes. The cast
|
|
// to blob is what makes the driver hand back the stored bytes
|
|
// rather than a string conversion, so Content-Length taken from
|
|
// the result matches what goes on the wire. The soft-delete
|
|
// predicate is spelled out because Raw bypasses GORM's default
|
|
// scope, and it is what stops a reaped event still being
|
|
// downloadable.
|
|
const eventBodyQuery = "SELECT cast(body as blob) " +
|
|
"FROM events WHERE id = ? AND webhook_id = ? AND deleted_at IS NULL"
|
|
|
|
// HandleEventBodyDownload serves one event's stored body in
|
|
// full, which the event log page cannot: it caps each rendered
|
|
// body at maxRenderedBodyBytes.
|
|
//
|
|
// The bytes are attacker-supplied — anyone who can reach the
|
|
// public receiver chooses them — and this route hands them back
|
|
// inside the operator's own authenticated origin, so the
|
|
// response is deliberately not renderable. Content-Disposition
|
|
// makes the browser download rather than display it, and the
|
|
// octet-stream type plus nosniff stop it being interpreted as
|
|
// HTML or script. Without those a stored payload would execute
|
|
// as the logged-in operator. The application's CSP does not
|
|
// help here: script-src allows 'unsafe-inline' from 'self', so
|
|
// a document served from this origin could run its own inline
|
|
// script.
|
|
func (h *Handlers) HandleEventBodyDownload() http.HandlerFunc {
|
|
return func(w http.ResponseWriter, r *http.Request) {
|
|
webhook, ok := h.ownedWebhook(w, r)
|
|
if !ok {
|
|
return
|
|
}
|
|
|
|
// Parsing the id before use serves two purposes: a
|
|
// malformed id can never reach the SQL or the response
|
|
// header, and the canonical form below is drawn from
|
|
// uuid's own fixed alphabet rather than from the
|
|
// request, so the Content-Disposition value cannot be
|
|
// steered by a client.
|
|
eventID, err := uuid.Parse(chi.URLParam(r, "eventID"))
|
|
if err != nil {
|
|
http.NotFound(w, r)
|
|
|
|
return
|
|
}
|
|
|
|
h.serveEventBody(w, r, webhook, eventID.String())
|
|
}
|
|
}
|
|
|
|
// serveEventBody writes the named event's stored body to w.
|
|
//
|
|
// The event must belong to webhook, which is what keeps this
|
|
// route from reading any event in the system by id alone. Two
|
|
// things enforce that and they are not equally strong. The
|
|
// operative one is that events live in a per-webhook SQLite
|
|
// file, so a sibling webhook's event is not in the database
|
|
// being queried at all. The webhook_id predicate on the query
|
|
// below is the second guard, and it is currently redundant
|
|
// against that isolation; it is there so the scoping survives
|
|
// any future change that puts more than one webhook's events in
|
|
// one file.
|
|
//
|
|
// The body is read in one query and held whole in memory while
|
|
// it is written. That costs roughly two body-sized allocations
|
|
// per concurrent download, not one: the driver's column buffer
|
|
// and the copy database/sql makes in convertAssign when a
|
|
// []byte column is scanned into a *[]byte are live at the same
|
|
// time. Measured allocation is ~2x the body plus ~45 KB, so at
|
|
// the 1 MB ingest cap a download costs ~2 MB of Go heap. On
|
|
// top of that, SQLite's own materialisation of the column
|
|
// value sits in the driver's allocator outside the Go heap, so
|
|
// process peak is higher again: 2x is a floor, not a ceiling.
|
|
// There is no cheaper bound available — database/sql exposes
|
|
// no incremental handle on a SQLite BLOB, and reading byte
|
|
// ranges with substr does not avoid the cost either, because
|
|
// SQLite materialises the whole column value to evaluate each
|
|
// substr call. Range reads only pay for that materialisation
|
|
// once per range.
|
|
//
|
|
// One consequence is worth keeping in view: the read finishes
|
|
// before the client is written to, so no read lock is held for
|
|
// the length of a slow download. These per-webhook databases
|
|
// run in SQLite's default journal mode rather than WAL, so a
|
|
// lock held that long would block the receiver from recording
|
|
// new events.
|
|
func (h *Handlers) serveEventBody(
|
|
w http.ResponseWriter,
|
|
r *http.Request,
|
|
webhook database.Webhook,
|
|
eventID string,
|
|
) {
|
|
if !h.dbMgr.DBExists(webhook.ID) {
|
|
http.NotFound(w, r)
|
|
|
|
return
|
|
}
|
|
|
|
webhookDB, err := h.dbMgr.GetDB(webhook.ID)
|
|
if err != nil {
|
|
h.serverError(w, "failed to get webhook database", err)
|
|
|
|
return
|
|
}
|
|
|
|
body, found, err := eventBody(webhookDB, webhook.ID, eventID)
|
|
if err != nil {
|
|
h.serverError(w, "failed to read event body", err)
|
|
|
|
return
|
|
}
|
|
|
|
// A miss is a 404 whether the event belongs to another
|
|
// webhook or does not exist at all, so the response does
|
|
// not report which. Reading the body before any header is
|
|
// written is also what keeps an event reaped mid-request
|
|
// from producing a torn response: either the read finds the
|
|
// row and the whole body is served, or it does not and the
|
|
// response is a clean 404.
|
|
if !found {
|
|
http.NotFound(w, r)
|
|
|
|
return
|
|
}
|
|
|
|
setEventBodyHeaders(w, eventID, int64(len(body)))
|
|
|
|
_, err = w.Write(body)
|
|
if err != nil {
|
|
// The status and Content-Length are already committed,
|
|
// so the client sees a short download. There is no way
|
|
// to report a 500 from here; the log is the record.
|
|
h.log.Error(
|
|
"failed to write event body",
|
|
"webhook_id", webhook.ID,
|
|
"event_id", eventID,
|
|
"error", err,
|
|
)
|
|
}
|
|
}
|
|
|
|
// eventBody returns an event's stored body and whether the event
|
|
// exists within the webhook.
|
|
func eventBody(
|
|
webhookDB *gorm.DB,
|
|
webhookID, eventID string,
|
|
) ([]byte, bool, error) {
|
|
var body []byte
|
|
|
|
err := webhookDB.Raw(
|
|
eventBodyQuery, eventID, webhookID,
|
|
).Row().Scan(&body)
|
|
|
|
if errors.Is(err, sql.ErrNoRows) {
|
|
return nil, false, nil
|
|
}
|
|
|
|
if err != nil {
|
|
return nil, false, err
|
|
}
|
|
|
|
return body, true, nil
|
|
}
|
|
|
|
// setEventBodyHeaders applies the response headers that make
|
|
// this route safe to hand attacker-supplied bytes through. See
|
|
// HandleEventBodyDownload for why they are a security control
|
|
// and not a formatting choice.
|
|
//
|
|
// nosniff is also set by the global SecurityHeaders middleware.
|
|
// It is repeated here so the guarantee belongs to the route
|
|
// that needs it rather than to a middleware someone could
|
|
// reorder or scope away.
|
|
func setEventBodyHeaders(
|
|
w http.ResponseWriter,
|
|
eventID string,
|
|
size int64,
|
|
) {
|
|
w.Header().Set("Content-Type", "application/octet-stream")
|
|
w.Header().Set("X-Content-Type-Options", "nosniff")
|
|
w.Header().Set(
|
|
"Content-Disposition",
|
|
`attachment; filename="webhooker-event-`+eventID+`.bin"`,
|
|
)
|
|
w.Header().Set("Content-Length", strconv.FormatInt(size, 10))
|
|
}
|