Route GORM's logger through slog and bound it (closes #178)
All checks were successful
check / check (push) Successful in 2m50s

GORM's default logger printed the fully interpolated SQL to standard
output on every statement that returned an error, including a plain
record-not-found. On /webhook/{uuid} and on the login form the
interpolated parameter is client-chosen and unbounded, so an
unauthenticated client sized the operator's log, one line per request,
at no level the operator could turn down.

Every gorm.Open in the service now installs internal/gormlog, a
gormlogger.Interface over the service's *slog.Logger. Its lines take
the level the operator set and the handler internal/logger selected; a
record-not-found is not logged as an error, since it is the expected
outcome on both of those paths and each handler already records its
own miss at DEBUG without the SQL; slow statements are kept at WARN
above the same 200ms threshold GORM used; and every value it emits is
spent through an encoded-byte budget.

Trace orders its cases exactly as GORM's own Trace orders them --
error-that-is-not-a-miss, then slow, then routine -- so a statement
that both missed and ran slow is still reported as slow. Ordering the
drop first would have made this adapter strictly less observant than
the IgnoreRecordNotFoundError option it was chosen over, on the two
lookups the issue is about, and a miss is the statement most likely to
be slow.

That budget is internal/middleware's truncateLogField, moved to a new
internal/logfield package now that a second writer needs it. The move
is unchanged logic. MaxAccessLogLineBytes bounds a GORM line too, and
internal/gormlog asserts each line against the constant directly.

The third gorm.Open, in the archive writer, was not named in the issue
and had the same default. All three sites are pinned independently:
internal/handlers covers the main and per-webhook databases,
internal/delivery covers the archive writer, whose type is unexported.
Reverting any one of the three to a bare &gorm.Config{} fails the
suite.

The flood test's per-line and volume assertions were vacuous, because
the replaced default logger wrote only to a buffer while everything
else went to the captured stdout. It now tees to stdout as GORM's real
default does, so a reverted call site lands in the same capture and
those assertions measure the whole writer set.

README: the ceiling now covers GORM, and the writers it does not cover
are re-derived by measuring rather than by reading. fx's console
logger and the Go runtime write to standard error. net/http's nil
ErrorLog is not a separate writer at all -- slog.SetDefault redirects
the log package's default logger into internal/logger's handler, so
those lines arrive on standard output at INFO. A handler panic reaches
that same path because chi's Recoverer crashes before writing, which
is filed as #187 and is also the widest line the service can write, at
2,772 bytes against the stated 2,560.
This commit is contained in:
2026-08-18 00:22:25 +00:00
parent b573959a26
commit f9d9a2c8d7
12 changed files with 1708 additions and 139 deletions

View File

@@ -6,11 +6,8 @@ import (
"log/slog"
"net"
"net/http"
"strings"
"sync"
"time"
"unicode"
"unicode/utf8"
basicauth "github.com/99designs/basicauth-go"
"github.com/go-chi/chi"
@@ -22,6 +19,7 @@ import (
"go.uber.org/fx"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/globals"
"sneak.berlin/go/webhooker/internal/logfield"
"sneak.berlin/go/webhooker/internal/logger"
"sneak.berlin/go/webhooker/internal/session"
)
@@ -44,16 +42,6 @@ const (
// pick the size of the line it writes.
redactedQuery = "?(redacted)"
// maxLogFieldBytes bounds each access log field whose value the
// client supplies outright: the URL, the User-Agent and the
// Referer. The budget is spent in ENCODED bytes (see
// truncateLogField), so 512 still holds a real browser's User-Agent
// whole — those are plain ASCII, which encodes one byte for one —
// while a value built from characters the encoder escapes keeps a
// shorter prefix. That is the intended trade: 500 quotation marks
// are not a debugging asset.
maxLogFieldBytes = 512
// maxLogRequestIDBytes bounds the request id, which is also
// client-supplied: chi's RequestID middleware passes an inbound
// X-Request-Id header through verbatim. Its generated form is an
@@ -66,15 +54,10 @@ const (
// is half this.
maxLogMethodBytes = 32
// truncationMarker is appended to any field the access log cut, so
// a short value and a truncated one cannot be confused. It is
// charged on top of the budget, not inside it.
truncationMarker = "[truncated]"
// MaxAccessLogLineBytes is the ceiling on one JSON access log line,
// and the number an operator multiplies by the request rate to size
// log storage. It is not an observation of a sample: it is the sum
// of the budgets above, each of which truncateLogField enforces in
// of the field budgets, each of which logfield.Truncate enforces in
// ENCODED bytes, plus the part of the line no client can influence.
//
// url, useragent, referer 3*(512+11) = 1569
@@ -84,6 +67,9 @@ const (
// ----
// 2087
//
// The 512 is logfield.MaxBytes; the 11 is the truncation marker,
// charged on top of each budget rather than inside it.
//
// The fixed portion is the JSON punctuation, the field names, the
// level and the message, both timestamps at their longest, an IPv6
// remoteIP with a zone, a three-digit status and a full-width int64
@@ -91,13 +77,29 @@ const (
// than sitting on the arithmetic.
//
// The tty text handler in internal/logger is covered by the same
// figure. encodedLogFieldBytes charges every rune at least what
// figure. logfield.EncodedBytes charges every rune at least what
// the wider of the two handlers emits for it — including the ten
// bytes strconv.Quote spends on a non-printable rune at or above
// U+10000, which is four more than the JSON handler ever spends —
// so each budget bounds the encoded field under either handler.
// The text handler's fixed portion is 286, the smaller of the two,
// which puts its worst case at 2037.
//
// The access log is the widest line this service writes that
// carries client-chosen text, so the figure is also the ceiling on
// the other writer that does: the GORM adapter in
// internal/gormlog, whose widest line spends two logfield.MaxBytes
// budgets (the interpolated SQL and the driver error) against a
// fixed portion smaller than this one's.
// internal/gormlog/gormlog_test.go asserts that against this
// constant directly rather than leaving it as arithmetic.
//
// It is not the widest line the service can write. A handler panic
// arrives through net/http's nil ErrorLog as one record carrying a
// whole goroutine stack, measured at 2,772 bytes. That value is
// the runtime's, not a client's, so it is a carve-out this ceiling
// states rather than covers; see README.md and
// https://git.eeqj.de/sneak/webhooker/issues/187.
MaxAccessLogLineBytes = 2560
)
@@ -174,114 +176,6 @@ func (lrw *loggingResponseWriter) WriteHeader(code int) {
lrw.ResponseWriter.WriteHeader(code)
}
// encodedLogFieldBytes is what r costs on the line once the log
// handler has escaped it, taking the worse of the two handlers
// internal/logger configures.
//
// slog's JSON handler escapes quote, backslash, newline, carriage
// return and tab to two bytes each, and every other C0 control plus
// LINE SEPARATOR and PARAGRAPH SEPARATOR to a six-byte \u escape; it
// passes every other rune through as its own UTF-8. Its text handler
// quotes with strconv.Quote, which spells a non-printable rune below
// U+10000 as \uXXXX but one at or above U+10000 as \UXXXXXXXX — ten
// bytes, not six. The text handler is therefore the worse of the two
// for every non-printable rune, and by four bytes apiece for the
// 955,086 unassigned, private-use and format code points on planes 1
// to 16.
//
// Charging ten there is what makes MaxAccessLogLineBytes hold for the
// tty handler as well: U+1000C encodes as F0 90 80 8C, every byte
// >= 0x80, which httpguts.ValidHeaderFieldValue accepts and
// net/textproto does not strip, so a header can be filled with them.
//
// Both handlers pass printable runes through as their own UTF-8, so
// unicode.IsPrint separates the escaped cases from the plain ones for
// either handler.
func encodedLogFieldBytes(r rune) int {
const (
// A backslash and the character itself.
shortEscapeBytes = 2
// \uXXXX, which is also the width of \u00XX.
escapedRuneBytes = 6
// \UXXXXXXXX, strconv.Quote's spelling of a non-printable
// rune outside the basic multilingual plane.
escapedAstralRuneBytes = 10
// The first code point strconv.Quote spells with \U.
firstAstralRune = 0x10000
)
switch {
case r == '"' || r == '\\' || r == '\n' || r == '\r' || r == '\t':
return shortEscapeBytes
case !unicode.IsPrint(r) && r >= firstAstralRune:
return escapedAstralRuneBytes
case !unicode.IsPrint(r):
return escapedRuneBytes
default:
return utf8.RuneLen(r)
}
}
// truncateLogField caps s at maxBytes of ENCODED output, marking the
// value when it cuts.
//
// Budgeting raw bytes would not bound the line. Escaping only ever
// grows a value, so a raw budget spent on characters the encoder
// escapes buys a field several times its nominal size — and the line
// is the thing an operator is told to multiply by their request rate.
// Charging each rune what it will actually cost is what makes
// MaxAccessLogLineBytes true rather than merely larger. The visible
// consequence is that an escape-heavy value keeps a shorter prefix
// than a plain one, which is the correct trade.
//
// The result is always valid UTF-8. A cut on a byte boundary can split
// a multi-byte rune, and a header can carry bytes that were never
// valid UTF-8 to begin with; both are dropped rather than kept, since
// an encoder would otherwise spend six bytes replacing each one.
func truncateLogField(s string, maxBytes int) string {
// No rune encodes to fewer bytes than it occupies, so nothing past
// maxBytes raw can fit the budget. Slicing first bounds the scan
// below to the budget rather than to the size of the header the
// client sent.
window, cut := s, false
if len(window) > maxBytes {
window, cut = window[:maxBytes], true
}
var (
kept strings.Builder
spent int
)
for i := 0; i < len(window); {
r, size := utf8.DecodeRuneInString(window[i:])
if r == utf8.RuneError && size == 1 {
i += size
continue
}
cost := encodedLogFieldBytes(r)
if spent+cost > maxBytes {
cut = true
break
}
spent += cost
kept.WriteString(window[i : i+size])
i += size
}
if !cut {
return kept.String()
}
return kept.String() + truncationMarker
}
// concreteLogURL renders the request's own URL for the access log
// branches that keep it, with the query string replaced by a fixed
// marker.
@@ -375,21 +269,21 @@ func (s *Middleware) Logging() func(http.Handler) http.Handler {
// line does not track the size of the request.
s.log.Info("http request",
"request_start", start,
"method", truncateLogField(
"method", logfield.Truncate(
r.Method, maxLogMethodBytes,
),
"url", truncateLogField(
"url", logfield.Truncate(
accessLogURL(r, lrw.statusCode),
maxLogFieldBytes,
logfield.MaxBytes,
),
"useragent", truncateLogField(
r.UserAgent(), maxLogFieldBytes,
"useragent", logfield.Truncate(
r.UserAgent(), logfield.MaxBytes,
),
"request_id", truncateLogField(
"request_id", logfield.Truncate(
requestID, maxLogRequestIDBytes,
),
"referer", truncateLogField(
r.Referer(), maxLogFieldBytes,
"referer", logfield.Truncate(
r.Referer(), logfield.MaxBytes,
),
"proto", r.Proto,
"remoteIP", ipFromHostPort(r.RemoteAddr),