3 Commits

Author	SHA1	Message	Date
clawbot	04678d07e3	Route GORM's logger through slog and bound it (closes #178) ... GORM's default logger printed the fully interpolated SQL to standard output on every statement that returned an error, including a plain record-not-found. On /webhook/{uuid} and on the login form the interpolated parameter is client-chosen and unbounded, so an unauthenticated client sized the operator's log, one line per request, at no level the operator could turn down. Every gorm.Open in the service now installs internal/gormlog, a gormlogger.Interface over the service's *slog.Logger. Its lines take the level the operator set and the handler internal/logger selected; a record-not-found is not logged as an error, since it is the expected outcome on both of those paths and each handler already records its own miss at DEBUG without the SQL; slow statements are kept at WARN above the same 200ms threshold GORM used; and every value it emits is spent through internal/logfield, the same encoded-byte budget the access log spends. MaxAccessLogLineBytes bounds a GORM line too, and internal/gormlog asserts each line against the constant directly. Trace orders its cases exactly as GORM's own Trace orders them -- error-that-is-not-a-miss, then slow, then routine -- so a statement that both missed and ran slow is still reported as slow. Ordering the drop first would have made this adapter strictly less observant than the IgnoreRecordNotFoundError option it was chosen over, on the two lookups the issue is about, and a miss is the statement most likely to be slow. The third gorm.Open, in the archive writer, was not named in the issue and had the same default. All three sites are pinned independently: internal/handlers covers the main and per-webhook databases, internal/delivery covers the archive writer, whose type is unexported. Reverting any one of the three to a bare &gorm.Config{} fails the suite. The flood test's per-line and volume assertions were vacuous, because the replaced default logger wrote only to a buffer while everything else went to the captured stdout. It now tees to stdout as GORM's real default does, so a reverted call site lands in the same capture and those assertions measure the whole writer set. internal/logfield gains the zero-headroom budget assertion and the rune-splitting case: a value built from one rune must keep exactly MaxBytes/EncodedBytes(r) of them, which a raw-byte budget fails and a LessOrEqual on the budget cannot catch. README: the ceiling now covers GORM, and the writers it does not cover are re-derived by measuring rather than by reading. fx's console logger and the Go runtime write to standard error. net/http's nil ErrorLog is not a separate writer at all -- slog.SetDefault redirects the log package's default logger into internal/logger's handler, so those lines arrive on standard output at INFO. A handler panic reaches that same path because chi's Recoverer crashes before writing, filed as #187, and is the widest line the service can write: measured at roughly 2,770 bytes against the stated 2,560, a width that moves with the goroutine number and the source paths in the stack, so only the fact that it exceeds the ceiling is stated as invariant.	2026-08-18 04:18:36 +00:00
clawbot	3e261d2f01	Evict archive writers on deletion and sweep idle archives (closes #89) (#95) ... Per-webhook archive writers are now evicted when the webhook or its last database target is deleted, and a background sweeper prunes expired rows from idle archives that no longer receive writes. Archive files themselves are never deleted.	2026-08-10 15:52:20 +02:00
clawbot	ee7c626071	Implement the database archiving target (closes #43) (#84) ... Implements the `databaseTarget` as a real archiving target, replacing the always-successful stub. Delivering to a `database` target now writes the full event into a per-webhook archive SQLite file for long-term storage. ## Archive-writer semantics - **Separate file:** each webhook's full events are written as rows into `archive-{webhookID}.db` under the data dir, distinct from the per-webhook event DB (`events-{webhookID}.db`). The file and its schema are created on first write if missing. Each row carries the full event: body, headers, method, content type, webhook id, entrypoint id, event id, and an archived-at timestamp. - **Close/reopen with debounce:** after each write the archive handle is closed and reopened, unless the last (re)open was less than one second ago. This lets an operator move the archive file away for offline archiving while bounding file churn under load. A per-webhook `archiveWriter` owns this debounce state and serialises writes. - **Auto-recreate:** the file is opened create-if-missing (`mode=rwc`) and its schema re-migrated on every open, so if the archive was moved or removed since the last open, the next write recreates it. The writer also detects a missing file before writing and reopens first, so a moved-away file is recreated rather than lost. - **Optional expiry, validated at creation:** an optional `expiry` in the target's config JSON (e.g. `{"expiry":"720h"}`) is validated when the target is created (`ValidateArchiveExpiry`; bad values are rejected with a 400 at the add-target form, the Slack URL precedent). The default (missing, empty, or `"never"`) keeps rows forever with no pruning. When a positive duration is set, rows older than it (measured from each row's archived-at time) are pruned on every (re)open; because the file is reopened after writes, prune-on-open keeps the archive swept without a separate background sweeper. A set-but-invalid expiry in a stored config (unparseable, zero, or negative) is an error at delivery time too — never a silent default. - **No-retry, fail-loud:** the target performs a single attempt with no retries. On success it records one successful attempt and marks the delivery delivered. If the archive write fails, the attempt is recorded as failed with the error and the delivery is marked failed — archiving errors never report success. ## Scope - `internal/delivery/target_database.go` — the `databaseTarget` (no-retry) archives via a per-webhook writer registry; an archive error records a failed attempt and marks the delivery failed. - `internal/delivery/target_database_archive.go` (new) — the `archiveWriter`, the archived-row model, config/expiry parsing (fail-loud on set-but-invalid values), `ValidateArchiveExpiry`, and prune-on-open. - `internal/handlers/source_management.go` — database targets get a creation-validated `expiry` config (`buildDatabaseTargetConfig`); the expiry form value is read where the request body is bounded and bad values are rejected with a 400 at target creation. - `templates/source_detail.html` — the add-target form shows an expiry field for database targets. - `README.md` — the database-target documentation describes the archiving semantics. - `internal/delivery/export_test.go`, `internal/delivery/target_database_test.go`, `internal/handlers` tests — tests and their exported shims. No changes to the `Target` interface or other targets. ## Tests - a row is archived (both at the writer level and end-to-end through `Deliver`) - a forced archive failure (bad stored expiry config) yields a `Failed` delivery with a non-success `DeliveryResult` carrying the error and no archive file created - the file is recreated after removal, with only the post-removal row - the one-second reopen debounce (rapid writes reopen once; a write after the window reopens again) - expiry pruning removes rows older than the configured expiry - expiry config parsing (empty / `never` / duration accepted; unparseable, zero, and negative values error) - expiry validation at target creation (`TestValidateArchiveExpiry`; valid values build the config, bad values get a 400) ## Validation `docker build .` exits 0 (fmt-check, lint, test, build all pass). Closes #43 Co-authored-by: sneak <sneak@sneak.berlin> Reviewed-on: #84 Co-authored-by: clawbot <clawbot@noreply.example.org> Co-committed-by: clawbot <clawbot@noreply.example.org>	2026-08-07 22:50:08 +02:00