sneak/webhooker
1
0
Fork 0
Code Issues 34 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Route GORM's logger through slog and bound it (closes #178)
All checks were successful
check / check (push) Successful in 2m54s
Details

Browse Source 
GORM's default logger printed the fully interpolated SQL to standard
output on every statement that returned an error, including a plain
record-not-found. On /webhook/{uuid} and on the login form the
interpolated parameter is client-chosen and unbounded, so an
unauthenticated client sized the operator's log, one line per request,
at no level the operator could turn down.

Every gorm.Open in the service now installs internal/gormlog, a
gormlogger.Interface over the service's *slog.Logger. Its lines take
the level the operator set and the handler internal/logger selected; a
record-not-found is not logged at all, since it is the expected
outcome on both of those paths and each handler already records its
own miss at DEBUG without the SQL; slow statements are kept at WARN
above the same 200ms threshold GORM used; and every value it emits is
spent through an encoded-byte budget.

That budget is internal/middleware's truncateLogField, moved to a new
internal/logfield package now that a second writer needs it. The move
is unchanged logic. MaxAccessLogLineBytes bounds a GORM line too, and
internal/gormlog asserts each line against the constant directly.

The third gorm.Open, in the archive writer, was not named in the issue
and had the same default.

README: the ceiling now covers GORM; the writers it does not cover are
named, including net/http's nil ErrorLog, fx's console logger and the
Go runtime, none of which carry a client-chosen value.
This commit is contained in:
clawbot
2026-08-18 00:22:25 +00:00
parent 76725cffc4
commit ce36f430fb
11 changed files with 1371 additions and 139 deletions
Download Patch File Download Diff File Expand all files Collapse all files

159
internal/middleware/middleware.go
View File

@@ -6,11 +6,8 @@ import (
"log/slog"
"net"
"net/http"
"strings"
"sync"
"time"
"unicode"
"unicode/utf8"
basicauth "github.com/99designs/basicauth-go"
"github.com/go-chi/chi"
@@ -22,6 +19,7 @@ import (
"go.uber.org/fx"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/globals"
"sneak.berlin/go/webhooker/internal/logfield"
"sneak.berlin/go/webhooker/internal/logger"
"sneak.berlin/go/webhooker/internal/session"
)
@@ -44,16 +42,6 @@ const (
// pick the size of the line it writes.
redactedQuery = "?(redacted)"
// maxLogFieldBytes bounds each access log field whose value the
// client supplies outright: the URL, the User-Agent and the
// Referer. The budget is spent in ENCODED bytes (see
// truncateLogField), so 512 still holds a real browser's User-Agent
// whole — those are plain ASCII, which encodes one byte for one —
// while a value built from characters the encoder escapes keeps a
// shorter prefix. That is the intended trade: 500 quotation marks
// are not a debugging asset.
maxLogFieldBytes = 512
// maxLogRequestIDBytes bounds the request id, which is also
// client-supplied: chi's RequestID middleware passes an inbound
// X-Request-Id header through verbatim. Its generated form is an
@@ -66,15 +54,10 @@ const (
// is half this.
maxLogMethodBytes = 32
// truncationMarker is appended to any field the access log cut, so
// a short value and a truncated one cannot be confused. It is
// charged on top of the budget, not inside it.
truncationMarker = "[truncated]"
// MaxAccessLogLineBytes is the ceiling on one JSON access log line,
// and the number an operator multiplies by the request rate to size
// log storage. It is not an observation of a sample: it is the sum
// of the budgets above, each of which truncateLogField enforces in
// of the field budgets, each of which logfield.Truncate enforces in
// ENCODED bytes, plus the part of the line no client can influence.
//
// url, useragent, referer 3*(512+11) = 1569
@@ -84,6 +67,9 @@ const (
// ----
// 2087
//
// The 512 is logfield.MaxBytes; the 11 is the truncation marker,
// charged on top of each budget rather than inside it.
//
// The fixed portion is the JSON punctuation, the field names, the
// level and the message, both timestamps at their longest, an IPv6
// remoteIP with a zone, a three-digit status and a full-width int64
@@ -91,13 +77,22 @@ const (
// than sitting on the arithmetic.
//
// The tty text handler in internal/logger is covered by the same
// figure. encodedLogFieldBytes charges every rune at least what
// figure. logfield.EncodedBytes charges every rune at least what
// the wider of the two handlers emits for it — including the ten
// bytes strconv.Quote spends on a non-printable rune at or above
// U+10000, which is four more than the JSON handler ever spends —
// so each budget bounds the encoded field under either handler.
// The text handler's fixed portion is 286, the smaller of the two,
// which puts its worst case at 2037.
//
// The access log is the widest line this service writes, so the
// figure is also the ceiling on the other writer that carries
// client-chosen text: the GORM adapter in internal/gormlog, whose
// widest line spends two logfield.MaxBytes budgets (the
// interpolated SQL and the driver error) against a fixed portion
// smaller than this one's. internal/gormlog/gormlog_test.go
// asserts that against this constant directly rather than leaving
// it as arithmetic.
MaxAccessLogLineBytes = 2560
)
@@ -174,114 +169,6 @@ func (lrw *loggingResponseWriter) WriteHeader(code int) {
lrw.ResponseWriter.WriteHeader(code)
}
// encodedLogFieldBytes is what r costs on the line once the log
// handler has escaped it, taking the worse of the two handlers
// internal/logger configures.
//
// slog's JSON handler escapes quote, backslash, newline, carriage
// return and tab to two bytes each, and every other C0 control plus
// LINE SEPARATOR and PARAGRAPH SEPARATOR to a six-byte \u escape; it
// passes every other rune through as its own UTF-8. Its text handler
// quotes with strconv.Quote, which spells a non-printable rune below
// U+10000 as \uXXXX but one at or above U+10000 as \UXXXXXXXX — ten
// bytes, not six. The text handler is therefore the worse of the two
// for every non-printable rune, and by four bytes apiece for the
// 955,086 unassigned, private-use and format code points on planes 1
// to 16.
//
// Charging ten there is what makes MaxAccessLogLineBytes hold for the
// tty handler as well: U+1000C encodes as F0 90 80 8C, every byte
// >= 0x80, which httpguts.ValidHeaderFieldValue accepts and
// net/textproto does not strip, so a header can be filled with them.
//
// Both handlers pass printable runes through as their own UTF-8, so
// unicode.IsPrint separates the escaped cases from the plain ones for
// either handler.
func encodedLogFieldBytes(r rune) int {
const (
// A backslash and the character itself.
shortEscapeBytes = 2
// \uXXXX, which is also the width of \u00XX.
escapedRuneBytes = 6
// \UXXXXXXXX, strconv.Quote's spelling of a non-printable
// rune outside the basic multilingual plane.
escapedAstralRuneBytes = 10
// The first code point strconv.Quote spells with \U.
firstAstralRune = 0x10000
)
switch {
case r == '"' || r == '\\' || r == '\n' || r == '\r' || r == '\t':
return shortEscapeBytes
case !unicode.IsPrint(r) && r >= firstAstralRune:
return escapedAstralRuneBytes
case !unicode.IsPrint(r):
return escapedRuneBytes
default:
return utf8.RuneLen(r)
}
}
// truncateLogField caps s at maxBytes of ENCODED output, marking the
// value when it cuts.
//
// Budgeting raw bytes would not bound the line. Escaping only ever
// grows a value, so a raw budget spent on characters the encoder
// escapes buys a field several times its nominal size — and the line
// is the thing an operator is told to multiply by their request rate.
// Charging each rune what it will actually cost is what makes
// MaxAccessLogLineBytes true rather than merely larger. The visible
// consequence is that an escape-heavy value keeps a shorter prefix
// than a plain one, which is the correct trade.
//
// The result is always valid UTF-8. A cut on a byte boundary can split
// a multi-byte rune, and a header can carry bytes that were never
// valid UTF-8 to begin with; both are dropped rather than kept, since
// an encoder would otherwise spend six bytes replacing each one.
func truncateLogField(s string, maxBytes int) string {
// No rune encodes to fewer bytes than it occupies, so nothing past
// maxBytes raw can fit the budget. Slicing first bounds the scan
// below to the budget rather than to the size of the header the
// client sent.
window, cut := s, false
if len(window) > maxBytes {
window, cut = window[:maxBytes], true
}
var (
kept strings.Builder
spent int
)
for i := 0; i < len(window); {
r, size := utf8.DecodeRuneInString(window[i:])
if r == utf8.RuneError && size == 1 {
i += size
continue
}
cost := encodedLogFieldBytes(r)
if spent+cost > maxBytes {
cut = true
break
}
spent += cost
kept.WriteString(window[i : i+size])
i += size
}
if !cut {
return kept.String()
}
return kept.String() + truncationMarker
}
// concreteLogURL renders the request's own URL for the access log
// branches that keep it, with the query string replaced by a fixed
// marker.
@@ -375,21 +262,21 @@ func (s *Middleware) Logging() func(http.Handler) http.Handler {
// line does not track the size of the request.
s.log.Info("http request",
"request_start", start,
"method", truncateLogField(
"method", logfield.Truncate(
r.Method, maxLogMethodBytes,
),
"url", truncateLogField(
"url", logfield.Truncate(
accessLogURL(r, lrw.statusCode),
maxLogFieldBytes,
logfield.MaxBytes,
),
"useragent", truncateLogField(
r.UserAgent(), maxLogFieldBytes,
"useragent", logfield.Truncate(
r.UserAgent(), logfield.MaxBytes,
),
"request_id", truncateLogField(
"request_id", logfield.Truncate(
requestID, maxLogRequestIDBytes,
),
"referer", truncateLogField(
r.Referer(), maxLogFieldBytes,
"referer", logfield.Truncate(
r.Referer(), logfield.MaxBytes,
),
"proto", r.Proto,
"remoteIP", ipFromHostPort(r.RemoteAddr),