ee7c62607104a7dacba5082cc46098f57049271c
4 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| ee7c626071 |
Implement the database archiving target (closes #43) (#84)
All checks were successful
check / check (push) Successful in 4s
Implements the `databaseTarget` as a real archiving target, replacing the always-successful stub. Delivering to a `database` target now writes the full event into a per-webhook archive SQLite file for long-term storage.
## Archive-writer semantics
- **Separate file:** each webhook's full events are written as rows into `archive-{webhookID}.db` under the data dir, distinct from the per-webhook event DB (`events-{webhookID}.db`). The file and its schema are created on first write if missing. Each row carries the full event: body, headers, method, content type, webhook id, entrypoint id, event id, and an archived-at timestamp.
- **Close/reopen with debounce:** after each write the archive handle is closed and reopened, unless the last (re)open was less than one second ago. This lets an operator move the archive file away for offline archiving while bounding file churn under load. A per-webhook `archiveWriter` owns this debounce state and serialises writes.
- **Auto-recreate:** the file is opened create-if-missing (`mode=rwc`) and its schema re-migrated on every open, so if the archive was moved or removed since the last open, the next write recreates it. The writer also detects a missing file before writing and reopens first, so a moved-away file is recreated rather than lost.
- **Optional expiry, validated at creation:** an optional `expiry` in the target's config JSON (e.g. `{"expiry":"720h"}`) is validated when the target is created (`ValidateArchiveExpiry`; bad values are rejected with a 400 at the add-target form, the Slack URL precedent). The default (missing, empty, or `"never"`) keeps rows forever with no pruning. When a positive duration is set, rows older than it (measured from each row's archived-at time) are pruned on every (re)open; because the file is reopened after writes, prune-on-open keeps the archive swept without a separate background sweeper. A set-but-invalid expiry in a stored config (unparseable, zero, or negative) is an error at delivery time too — never a silent default.
- **No-retry, fail-loud:** the target performs a single attempt with no retries. On success it records one successful attempt and marks the delivery delivered. If the archive write fails, the attempt is recorded as failed with the error and the delivery is marked failed — archiving errors never report success.
## Scope
- `internal/delivery/target_database.go` — the `databaseTarget` (no-retry) archives via a per-webhook writer registry; an archive error records a failed attempt and marks the delivery failed.
- `internal/delivery/target_database_archive.go` (new) — the `archiveWriter`, the archived-row model, config/expiry parsing (fail-loud on set-but-invalid values), `ValidateArchiveExpiry`, and prune-on-open.
- `internal/handlers/source_management.go` — database targets get a creation-validated `expiry` config (`buildDatabaseTargetConfig`); the expiry form value is read where the request body is bounded and bad values are rejected with a 400 at target creation.
- `templates/source_detail.html` — the add-target form shows an expiry field for database targets.
- `README.md` — the database-target documentation describes the archiving semantics.
- `internal/delivery/export_test.go`, `internal/delivery/target_database_test.go`, `internal/handlers` tests — tests and their exported shims.
No changes to the `Target` interface or other targets.
## Tests
- a row is archived (both at the writer level and end-to-end through `Deliver`)
- a forced archive failure (bad stored expiry config) yields a `Failed` delivery with a non-success `DeliveryResult` carrying the error and no archive file created
- the file is recreated after removal, with only the post-removal row
- the one-second reopen debounce (rapid writes reopen once; a write after the window reopens again)
- expiry pruning removes rows older than the configured expiry
- expiry config parsing (empty / `never` / duration accepted; unparseable, zero, and negative values error)
- expiry validation at target creation (`TestValidateArchiveExpiry`; valid values build the config, bad values get a 400)
## Validation
`docker build .` exits 0 (fmt-check, lint, test, build all pass).
Closes #43
Co-authored-by: sneak <sneak@sneak.berlin>
Reviewed-on: #84
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
|
|||
| 81413c56e9 |
Refactor delivery targets to a Target interface (closes #77) (#81)
All checks were successful
check / check (push) Successful in 2m42s
Refactors the delivery engine so each target TYPE is an implementation of a `Target` interface, dispatched from a registry, with each target owning its full delivery including durable retries. Implements the authoritative design from issue #77 (the corrected "hand the DB + Scheduler to the target" design). ## The new interface ```go type Scheduler interface { ScheduleRetry(task Task, delay time.Duration) } type Target interface { Deliver(ctx context.Context, webhookDB *gorm.DB, d *database.Delivery, task *Task, sched Scheduler) } ``` `Deliver` receives everything a target needs to be autonomous and durable: the request context, the per-webhook `*gorm.DB`, the `*database.Delivery`, the attempt `*Task`, and a `Scheduler` (the engine) for durable re-enqueue. The target makes one attempt, writes the `DeliveryResult`, updates `DeliveryStatus`, and — for retry targets — decides whether to retry, computes its own backoff, gates with its own circuit breaker, and reschedules via the injected `Scheduler`. `processDelivery` collapses to a registry lookup (`map[database.TargetType]Target`) and a `Deliver` call; an unknown target type still fails the delivery as before. ## Per-target ownership - `httpTarget` and `slackTarget` share a retry core (`httpCore`) that owns retry, exponential backoff, and the per-target circuit breaker. The core is fire-and-forget when `MaxRetries == 0` and adds breaker-gated backed-off retries when `MaxRetries > 0`. The per-attempt request differs (HTTP forwards the body + filtered headers; Slack posts a formatted message) and is supplied as a closure, so each keeps its exact recording semantics (e.g. HTTP records no error string for a non-2xx, Slack records `HTTP <code>`). - `databaseTarget` and `logTarget` are fire-and-forget: they record a single successful attempt. Moved wholesale into the http/slack targets: `deliverHTTP*`, `handleHTTPRetry`, `circuitBreakerBlock`, `calcBackoff` / `calcRemainingBackoff` / `backoffElapsed`, the circuit-breaker `sync.Map` + `getCircuitBreaker`, `clientForConfig`, `doHTTPRequest`, `applyRequestHeaders`, and the config parsers. The engine keeps `recordResult`, `updateDeliveryStatus`, and `ScheduleRetry`. ## Slack MaxRetries gating Slack is now on the same shared core as HTTP, with retry + breaker gated on `MaxRetries`. A `MaxRetries` of 0 stays single-attempt fire-and-forget, so **every existing Slack target is unchanged**; a Slack target configured with retries gets backoff + circuit breaker. ## Log-target full content `logTarget` now logs the ENTIRE inbound webhook — full request body and full request headers, plus method, content type, and the webhook id and entrypoint id — rather than a summary line. This supersedes the smaller log-summary work (#70). ## `Task.EntrypointID` To carry the entrypoint id to the log target, `Task` gains an `EntrypointID` field, populated in the webhook handler's `buildDeliveryTasks`, the engine's recovery-task builder, and `buildEventFromTask`. ## Durability / recovery The crash-durable async retry model is preserved unchanged: one attempt per worker turn; on failure the status is set `retrying`, backoff is computed, and the task is re-enqueued via `ScheduleRetry` (a `time.AfterFunc` onto the retry channel). On restart, `recoverRetryingDeliveries` and the 60s sweep hand each orphaned `retrying` delivery back to its target to recompute the remaining backoff and reschedule (targets that own retries implement an internal `rescheduler`; fire-and-forget targets, which never produce `retrying` deliveries, are skipped). ## How behaviour is preserved No external behaviour changes except the two called out above (log target full content; Slack gaining `MaxRetries`-gated retries). All existing delivery tests pass with only their `export_test.go` wrappers re-pointed at the new structure — `ExportDeliverHTTP/Slack/Database/Log` now call the targets, `ExportGetCircuitBreaker` / `ExportClient` / `ExportClientForConfig` / `ExportDoHTTPRequest` resolve against the HTTP target's shared client and breaker map, and `ExportParseHTTPConfig` / `ExportParseSlackConfig` call the relocated free functions. Added: a `logTarget` test asserting the log line contains the full body, headers, and ids, and a Slack `MaxRetries`-gated retry test. `docker build .` is green (fmt-check, lint, test, static build all pass). Closes #77 Co-authored-by: sneak <sneak@sneak.berlin> Reviewed-on: #81 Co-authored-by: clawbot <clawbot@noreply.example.org> Co-committed-by: clawbot <clawbot@noreply.example.org> |
|||
| b1f43c9520 |
Keep the SSRF-safe transport in clientForConfig (closes #69) (#74)
Some checks failed
check / check (push) Has been cancelled
`clientForConfig()` in `internal/delivery/engine.go` built a fresh `http.Client` without a Transport when a per-target timeout was configured, dropping the request-time private-IP guard for that path. It now reuses the shared client's SSRF-safe transport (`e.client.Transport`, the same `NewSSRFSafeTransport` instance), overriding only the `Timeout`. Behaviour is unchanged when no per-target timeout is set (the shared client is returned as before), so no engine code path makes an outbound target request with a client lacking the SSRF-safe transport. Adds a delivery-package test proving a client from `clientForConfig()` with a per-target timeout still refuses private/reserved/link-local destinations, that the timeout is applied, that the SSRF-safe transport is reused (not duplicated), and that the no-timeout path returns the shared client unchanged. Confined to `internal/delivery/` only; handlers and server code untouched. Closes #69 Co-authored-by: sneak <sneak@sneak.berlin> Co-authored-by: Jeffrey Paul <sneak@noreply.example.org> Reviewed-on: #74 Co-authored-by: clawbot <clawbot@noreply.example.org> Co-committed-by: clawbot <clawbot@noreply.example.org> |
|||
| afe88c601a |
refactor: use pinned golangci-lint Docker image for linting (#55)
All checks were successful
check / check (push) Successful in 5s
Closes [issue #50](#50) ## Summary Refactors the Dockerfile to use a separate lint stage with a pinned golangci-lint Docker image, following the pattern used by [sneak/pixa](https://git.eeqj.de/sneak/pixa). This replaces the previous approach of installing golangci-lint via curl in the builder stage. ## Changes ### Dockerfile - **New `lint` stage** using `golangci/golangci-lint:v2.11.3` (Debian-based, pinned by sha256 digest) as a separate build stage - **Builder stage** depends on lint via `COPY --from=lint /src/go.sum /dev/null` — build won't proceed unless linting passes - **Go bumped** from 1.24 to 1.26.1 (`golang:1.26.1-bookworm`, pinned by sha256) - **golangci-lint bumped** from v1.64.8 to v2.11.3 - All three Docker images (golangci-lint, golang, alpine) pinned by sha256 digest - Debian-based golangci-lint image used (not Alpine) because mattn/go-sqlite3 CGO does not compile on musl (off64_t) ### Linter Config (.golangci.yml) - Migrated from v1 to v2 format (`version: "2"` added) - Removed linters no longer available in v2: `gofmt` (handled by `make fmt-check`), `gosimple` (merged into `staticcheck`), `typecheck` (always-on in v2) - Same set of linters enabled — no rules weakened ### Code Fixes (all lint issues from v2 upgrade) - Added package comments to all packages - Added doc comments to all exported types, functions, and methods - Fixed unchecked errors flagged by `errcheck` (sqlDB.Close, os.Setenv in tests, resp.Body.Close, fmt.Fprint) - Fixed unused parameters flagged by `revive` (renamed to `_`) - Fixed `gosec` G120 warnings: added `http.MaxBytesReader` before `r.ParseForm()` calls - Fixed `staticcheck` QF1012: replaced `WriteString(fmt.Sprintf(...))` with `fmt.Fprintf` - Fixed `staticcheck` QF1003: converted if/else chain to tagged switch - Renamed `DeliveryTask` → `Task` to avoid package stutter (`delivery.Task` instead of `delivery.DeliveryTask`) - Renamed shadowed builtin `max` parameter to `upperBound` in `cryptoRandInt` - Used `t.Setenv` instead of `os.Setenv` in tests (auto-restores) ### README.md - Updated version requirements: Go 1.26+, golangci-lint v2.11+ - Updated Dockerfile description in project structure ## Verification `docker build .` passes cleanly — formatting check, linting, all tests, and build all succeed. Co-authored-by: clawbot <clawbot@noreply.git.eeqj.de> Reviewed-on: #55 Co-authored-by: clawbot <clawbot@noreply.example.org> Co-committed-by: clawbot <clawbot@noreply.example.org> |
