3084ed545b8ff2b869d732d561a3dfc105bee594
17
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  clawbotandsneak
|
81413c56e9 |
Refactor delivery targets to a Target interface (closes #77) (#81)
check / check (push) Successful in 2m42s
Refactors the delivery engine so each target TYPE is an implementation of a `Target` interface, dispatched from a registry, with each target owning its full delivery including durable retries. Implements the authoritative design from issue #77 (the corrected "hand the DB + Scheduler to the target" design). ## The new interface ```go type Scheduler interface { ScheduleRetry(task Task, delay time.Duration) } type Target interface { Deliver(ctx context.Context, webhookDB *gorm.DB, d *database.Delivery, task *Task, sched Scheduler) } ``` `Deliver` receives everything a target needs to be autonomous and durable: the request context, the per-webhook `*gorm.DB`, the `*database.Delivery`, the attempt `*Task`, and a `Scheduler` (the engine) for durable re-enqueue. The target makes one attempt, writes the `DeliveryResult`, updates `DeliveryStatus`, and — for retry targets — decides whether to retry, computes its own backoff, gates with its own circuit breaker, and reschedules via the injected `Scheduler`. `processDelivery` collapses to a registry lookup (`map[database.TargetType]Target`) and a `Deliver` call; an unknown target type still fails the delivery as before. ## Per-target ownership - `httpTarget` and `slackTarget` share a retry core (`httpCore`) that owns retry, exponential backoff, and the per-target circuit breaker. The core is fire-and-forget when `MaxRetries == 0` and adds breaker-gated backed-off retries when `MaxRetries > 0`. The per-attempt request differs (HTTP forwards the body + filtered headers; Slack posts a formatted message) and is supplied as a closure, so each keeps its exact recording semantics (e.g. HTTP records no error string for a non-2xx, Slack records `HTTP <code>`). - `databaseTarget` and `logTarget` are fire-and-forget: they record a single successful attempt. Moved wholesale into the http/slack targets: `deliverHTTP*`, `handleHTTPRetry`, `circuitBreakerBlock`, `calcBackoff` / `calcRemainingBackoff` / `backoffElapsed`, the circuit-breaker `sync.Map` + `getCircuitBreaker`, `clientForConfig`, `doHTTPRequest`, `applyRequestHeaders`, and the config parsers. The engine keeps `recordResult`, `updateDeliveryStatus`, and `ScheduleRetry`. ## Slack MaxRetries gating Slack is now on the same shared core as HTTP, with retry + breaker gated on `MaxRetries`. A `MaxRetries` of 0 stays single-attempt fire-and-forget, so **every existing Slack target is unchanged**; a Slack target configured with retries gets backoff + circuit breaker. ## Log-target full content `logTarget` now logs the ENTIRE inbound webhook — full request body and full request headers, plus method, content type, and the webhook id and entrypoint id — rather than a summary line. This supersedes the smaller log-summary work (#70). ## `Task.EntrypointID` To carry the entrypoint id to the log target, `Task` gains an `EntrypointID` field, populated in the webhook handler's `buildDeliveryTasks`, the engine's recovery-task builder, and `buildEventFromTask`. ## Durability / recovery The crash-durable async retry model is preserved unchanged: one attempt per worker turn; on failure the status is set `retrying`, backoff is computed, and the task is re-enqueued via `ScheduleRetry` (a `time.AfterFunc` onto the retry channel). On restart, `recoverRetryingDeliveries` and the 60s sweep hand each orphaned `retrying` delivery back to its target to recompute the remaining backoff and reschedule (targets that own retries implement an internal `rescheduler`; fire-and-forget targets, which never produce `retrying` deliveries, are skipped). ## How behaviour is preserved No external behaviour changes except the two called out above (log target full content; Slack gaining `MaxRetries`-gated retries). All existing delivery tests pass with only their `export_test.go` wrappers re-pointed at the new structure — `ExportDeliverHTTP/Slack/Database/Log` now call the targets, `ExportGetCircuitBreaker` / `ExportClient` / `ExportClientForConfig` / `ExportDoHTTPRequest` resolve against the HTTP target's shared client and breaker map, and `ExportParseHTTPConfig` / `ExportParseSlackConfig` call the relocated free functions. Added: a `logTarget` test asserting the log line contains the full body, headers, and ids, and a Slack `MaxRetries`-gated retry test. `docker build .` is green (fmt-check, lint, test, static build all pass). Closes #77 Co-authored-by: sneak <sneak@sneak.berlin> Reviewed-on: #81 Co-authored-by: clawbot <clawbot@noreply.example.org> Co-committed-by: clawbot <clawbot@noreply.example.org> |
||
|
clawbotandsneak
|
b1f43c9520 |
Keep the SSRF-safe transport in clientForConfig (closes #69) (#74)
check / check (push) Superseded by a newer commit; never tested
`clientForConfig()` in `internal/delivery/engine.go` built a fresh `http.Client` without a Transport when a per-target timeout was configured, dropping the request-time private-IP guard for that path. It now reuses the shared client's SSRF-safe transport (`e.client.Transport`, the same `NewSSRFSafeTransport` instance), overriding only the `Timeout`. Behaviour is unchanged when no per-target timeout is set (the shared client is returned as before), so no engine code path makes an outbound target request with a client lacking the SSRF-safe transport. Adds a delivery-package test proving a client from `clientForConfig()` with a per-target timeout still refuses private/reserved/link-local destinations, that the timeout is applied, that the SSRF-safe transport is reused (not duplicated), and that the no-timeout path returns the shared client unchanged. Confined to `internal/delivery/` only; handlers and server code untouched. Closes #69 Co-authored-by: sneak <sneak@sneak.berlin> Co-authored-by: Jeffrey Paul <sneak@noreply.example.org> Reviewed-on: #74 Co-authored-by: clawbot <clawbot@noreply.example.org> Co-committed-by: clawbot <clawbot@noreply.example.org> |
||
 clawbotandclawbot
|
afe88c601a |
refactor: use pinned golangci-lint Docker image for linting (#55)
check / check (push) Successful in 5s
Closes [issue #50](#50) ## Summary Refactors the Dockerfile to use a separate lint stage with a pinned golangci-lint Docker image, following the pattern used by [sneak/pixa](https://git.eeqj.de/sneak/pixa). This replaces the previous approach of installing golangci-lint via curl in the builder stage. ## Changes ### Dockerfile - **New `lint` stage** using `golangci/golangci-lint:v2.11.3` (Debian-based, pinned by sha256 digest) as a separate build stage - **Builder stage** depends on lint via `COPY --from=lint /src/go.sum /dev/null` — build won't proceed unless linting passes - **Go bumped** from 1.24 to 1.26.1 (`golang:1.26.1-bookworm`, pinned by sha256) - **golangci-lint bumped** from v1.64.8 to v2.11.3 - All three Docker images (golangci-lint, golang, alpine) pinned by sha256 digest - Debian-based golangci-lint image used (not Alpine) because mattn/go-sqlite3 CGO does not compile on musl (off64_t) ### Linter Config (.golangci.yml) - Migrated from v1 to v2 format (`version: "2"` added) - Removed linters no longer available in v2: `gofmt` (handled by `make fmt-check`), `gosimple` (merged into `staticcheck`), `typecheck` (always-on in v2) - Same set of linters enabled — no rules weakened ### Code Fixes (all lint issues from v2 upgrade) - Added package comments to all packages - Added doc comments to all exported types, functions, and methods - Fixed unchecked errors flagged by `errcheck` (sqlDB.Close, os.Setenv in tests, resp.Body.Close, fmt.Fprint) - Fixed unused parameters flagged by `revive` (renamed to `_`) - Fixed `gosec` G120 warnings: added `http.MaxBytesReader` before `r.ParseForm()` calls - Fixed `staticcheck` QF1012: replaced `WriteString(fmt.Sprintf(...))` with `fmt.Fprintf` - Fixed `staticcheck` QF1003: converted if/else chain to tagged switch - Renamed `DeliveryTask` → `Task` to avoid package stutter (`delivery.Task` instead of `delivery.DeliveryTask`) - Renamed shadowed builtin `max` parameter to `upperBound` in `cryptoRandInt` - Used `t.Setenv` instead of `os.Setenv` in tests (auto-restores) ### README.md - Updated version requirements: Go 1.26+, golangci-lint v2.11+ - Updated Dockerfile description in project structure ## Verification `docker build .` passes cleanly — formatting check, linting, all tests, and build all succeed. Co-authored-by: clawbot <clawbot@noreply.git.eeqj.de> Reviewed-on: #55 Co-authored-by: clawbot <clawbot@noreply.example.org> Co-committed-by: clawbot <clawbot@noreply.example.org> |
||
|
|
60786c5019 |
feat: add CSRF protection, SSRF prevention, and login rate limiting (#42)
check / check (push) Successful in 4s
## Security Hardening This PR implements three security hardening issues: ### CSRF Protection (closes #35) - Session-based CSRF tokens with cryptographically random 256-bit generation - Constant-time token comparison to prevent timing attacks - CSRF middleware applied to `/pages`, `/sources`, `/source`, and `/user` routes - Hidden `csrf_token` field added to all 12+ POST forms in templates - Excluded from `/webhook` (inbound webhook POSTs) and `/api` (stateless API) ### SSRF Prevention (closes #36) - `ValidateTargetURL()` blocks private/reserved IP ranges at target creation time - Blocked ranges: `127.0.0.0/8`, `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `169.254.0.0/16`, `::1`, `fc00::/7`, `fe80::/10`, plus multicast, reserved, test-net, and CGN ranges - SSRF-safe HTTP transport with custom `DialContext` in the delivery engine for defense-in-depth (prevents DNS rebinding attacks) - Only `http` and `https` schemes allowed ### Login Rate Limiting (closes #37) - Per-IP rate limiter using `golang.org/x/time/rate` - 5 attempts per minute per IP on `POST /pages/login` - GET requests (form rendering) pass through unaffected - Automatic cleanup of stale per-IP limiter entries every 5 minutes - `X-Forwarded-For` and `X-Real-IP` header support for reverse proxies ### Files Changed **New files:** - `internal/middleware/csrf.go` + tests — CSRF middleware - `internal/middleware/ratelimit.go` + tests — Login rate limiter - `internal/delivery/ssrf.go` + tests — SSRF validation + safe transport **Modified files:** - `internal/server/routes.go` — Wire CSRF and rate limit middleware - `internal/handlers/handlers.go` — Inject CSRF token into template data - `internal/handlers/source_management.go` — SSRF validation on target creation - `internal/delivery/engine.go` — SSRF-safe HTTP transport for production - All form templates — Added hidden `csrf_token` fields - `README.md` — Updated Security section and TODO checklist `docker build .` passes (lint + tests + build). Co-authored-by: clawbot <clawbot@noreply.git.eeqj.de> Co-authored-by: clawbot <clawbot@eeqj.de> Co-authored-by: Jeffrey Paul <sneak@noreply.example.org> Reviewed-on: #42 Co-authored-by: clawbot <clawbot@noreply.example.org> Co-committed-by: clawbot <clawbot@noreply.example.org> |
||
|
clawbotanduser
|
8d702a16c6 |
feat: add Slack target type for incoming webhook notifications (#47)
check / check (push) Successful in 4s
## Summary Adds a new `slack` target type that sends webhook events as formatted messages to any Slack-compatible incoming webhook URL (Slack, Mattermost, and other compatible services). closes #44 ## What it does When a webhook event is received, the Slack target: 1. Formats a human-readable message with event metadata (HTTP method, content type, timestamp, body size) 2. Pretty-prints the payload in a code block — JSON payloads get indented formatting, non-JSON payloads are shown as raw text 3. Truncates large payloads at 3500 characters to keep Slack messages reasonable 4. POSTs the message as a `{"text": "..."}` JSON payload to the configured webhook URL ## Changes - **`internal/database/model_target.go`** — Add `TargetTypeSlack` constant - **`internal/delivery/engine.go`** — Add `SlackTargetConfig` struct, `deliverSlack` method, `FormatSlackMessage` function (exported), `parseSlackConfig` helper. Route slack targets in `processDelivery` switch. - **`internal/handlers/source_management.go`** — Handle `slack` type in `HandleTargetCreate`, building `webhook_url` config from the URL form field - **`templates/source_detail.html`** — Add "Slack" option to target type dropdown with URL field and helper text - **`README.md`** — Document the new target type, update roadmap ## Tests - `TestParseSlackConfig_Valid` / `_Empty` / `_MissingWebhookURL` — Config parsing - `TestFormatSlackMessage_JSONBody` / `_NonJSONBody` / `_EmptyBody` / `_LargeJSONTruncated` — Message formatting - `TestDeliverSlack_Success` / `_Failure` / `_InvalidConfig` — End-to-end delivery - `TestProcessDelivery_RoutesToSlack` — Routing from processDelivery switch All existing tests continue to pass. `docker build .` (which runs `make check`) passes clean. Co-authored-by: user <user@Mac.lan guest wan> Reviewed-on: #47 Co-authored-by: clawbot <clawbot@noreply.example.org> Co-committed-by: clawbot <clawbot@noreply.example.org> |
||
|
clawbotandclawbot
|
289f479772 |
test: add tests for delivery, middleware, and session packages (#32)
check / check (push) Has been cancelled
## Summary Add comprehensive test coverage for three previously-untested packages, addressing [issue #28](#28). ## Coverage Improvements | Package | Before | After | |---------|--------|-------| | `internal/delivery` | 37.1% | 74.5% | | `internal/middleware` | 0.0% | 70.2% | | `internal/session` | 0.0% | 51.5% | ## What's Tested ### delivery (37% → 75%) - `processNewTask` with inline and large (DB-fetched) bodies - `processRetryTask` success, skip non-retrying, large body fetch - Worker lifecycle start/stop, retry channel processing - `processDelivery` unknown target type handling - `recoverPendingDeliveries`, `recoverWebhookDeliveries`, `recoverInFlight` - HTTP delivery with custom headers, timeout, invalid config - `Notify` batching ### middleware (0% → 70%) - Logging middleware status code capture and pass-through - `LoggingResponseWriter` delegation - CORS dev mode (allow-all) and prod mode (no-op) - `RequireAuth` redirect for unauthenticated, pass-through for authenticated - `MetricsAuth` basic auth validation - `ipFromHostPort` helper ### session (0% → 52%) - `Get`/`Save` round-trip with real cookie store - `SetUser`, `GetUserID`, `GetUsername`, `IsAuthenticated` - `ClearUser` removes all keys - `Destroy` invalidates session (MaxAge -1) - Session persistence across requests - Edge cases: overwrite user, wrong type, constants ## Test Helpers Added - `database.NewTestDatabase` / `NewTestWebhookDBManager` — cross-package test helpers for delivery integration tests - `session.NewForTest` — creates session manager without fx lifecycle for middleware tests ## Notes - No production code modified - All tests use `httptest`, SQLite in-memory, and real cookie stores — no external network calls - Full test suite completes in ~3.5s within the 30s timeout - `docker build .` passes (lint + test + build) closes #28 Co-authored-by: clawbot <clawbot@noreply.git.eeqj.de> Reviewed-on: #32 Co-authored-by: clawbot <clawbot@noreply.example.org> Co-committed-by: clawbot <clawbot@noreply.example.org> |
||
|
clawbot
|
8e00e40008 |
docs: fix stale references to development mode and retry target type
check / check (push) Successful in 5s
- README.md: remove 'in development mode' from admin user creation description (admin user creation is unconditional) - internal/delivery/engine.go: remove 'and retry' from HTTPTargetConfig comment (retry was merged into http target type) - internal/delivery/engine_test.go: remove '/retry' from newHTTPTargetConfig comment for consistency |
||
|
clawbot
|
25e27cc57f |
refactor: merge retry target type into http (max_retries=0 = fire-and-forget)
check / check (push) Successful in 1m46s
|
||
|
clawbot
|
536e5682d6 |
test: add comprehensive delivery engine and circuit breaker tests
check / check (push) Successful in 1m48s
Add unit tests for internal/delivery/ package covering: Circuit breaker tests (circuit_breaker_test.go): - Closed state allows deliveries - Failure counting below threshold - Open transition after threshold failures - Cooldown blocks during cooldown period - Half-open transition after cooldown expires - Probe success closes circuit - Probe failure reopens circuit - Success resets failure counter - Concurrent access safety (race-safe) - CooldownRemaining for all states - CircuitState String() output Engine tests (engine_test.go): - Non-blocking Notify when channel is full - HTTP target success and failure delivery - Database target immediate success - Log target immediate success - Retry target success with circuit breaker - Max retries exhausted marks delivery failed - Retry scheduling on failure - Exponential backoff duration verification - Backoff cap at shift 30 - Body pointer semantics (inline <16KB, nil >=16KB) - Worker pool bounded concurrency - Circuit breaker blocks delivery attempts - Circuit breaker per-target creation - HTTP config parsing (valid, empty, missing URL) - scheduleRetry sends to retry channel - scheduleRetry drops when channel full - Header forwarding (forwardable vs hop-by-hop) - processDelivery routing to correct handler - Truncate helper function All tests use real SQLite databases and httptest servers. All tests pass with -race flag. |
||
|
clawbot
|
10db6c5b84 |
refactor: bounded worker pool with DB-mediated retry fallback
check / check (push) Successful in 58s
Replace unbounded goroutine-per-delivery fan-out with a fixed-size worker pool (10 workers). Channels serve as bounded queues (10,000 buffer). Workers are the only goroutines doing HTTP delivery. When retry channel overflows, timers are dropped instead of re-armed. The delivery stays in 'retrying' status in the DB and a periodic sweep (every 60s) recovers orphaned retries. The database is the durable fallback — same path used on startup recovery. Addresses owner feedback on circuit breaker recovery goroutine flood. |
||
|
clawbot
|
9b4ae41c44 |
feat: parallel fan-out delivery + circuit breaker for retry targets
check / check (push) Successful in 1m52s
- Fan out all targets for an event in parallel goroutines (fire-and-forget) - Add per-target circuit breaker for retry targets (closed/open/half-open) - Circuit breaker trips after 5 consecutive failures, 30s cooldown - Open circuit skips delivery and reschedules after cooldown - Half-open allows one probe delivery to test recovery - HTTP/database/log targets unaffected (no circuit breaker) - Recovery path also fans out in parallel - Update README with parallel delivery and circuit breaker docs |
||
|
clawbot
|
32bd40b313 |
refactor: self-contained delivery tasks — engine delivers without DB reads in happy path
check / check (push) Successful in 58s
The webhook handler now builds DeliveryTask structs carrying all target config and event data inline (for bodies ≤16KB) and sends them through the delivery channel. In the happy path, the engine delivers without reading from any database — it only writes to record delivery results. For large bodies (≥16KB), Body is nil and the engine fetches it from the per-webhook database on demand. Retry timers also carry the full DeliveryTask, so retries avoid unnecessary DB reads. The database is used for crash recovery only: on startup the engine scans for interrupted pending/retrying deliveries and re-queues them. Implements owner feedback from issue #15: > the message in the <=16KB case should have everything it needs to do > its delivery. it shouldn't touch the db until it has a success or > failure to record. |
||
|
clawbot
|
5e683af2a4 |
refactor: event-driven delivery engine with channel notifications and timer-based retries
check / check (push) Successful in 58s
Replace the polling-based delivery engine with a fully event-driven architecture using Go channels and goroutines: - Webhook handler notifies engine via buffered channel after creating delivery records, with inline event data for payloads < 16KB - Large payloads (>= 16KB) use pointer semantics (Body *string = nil) and are fetched from DB on demand, keeping channel memory bounded - Failed retry-target deliveries schedule Go timers with exponential backoff; timers fire into a separate retry channel when ready - On startup, engine scans DB once to recover interrupted deliveries (pending processed immediately, retrying get timers for remaining backoff) - DB stores delivery status for crash recovery only, not for inter-component communication during normal operation - delivery.Notifier interface decouples handlers from engine; fx wires *Engine as Notifier No more periodic polling. No more wasted cycles when idle. |
||
|
clawbot
|
43c22a9e9a |
feat: implement per-webhook event databases
check / check (push) Successful in 1m50s
Split data storage into main application DB (config only) and
per-webhook event databases (one SQLite file per webhook).
Architecture changes:
- New WebhookDBManager component manages per-webhook DB lifecycle
(create, open, cache, delete) with lazy connection pooling via sync.Map
- Main DB (DBURL) stores only config: Users, Webhooks, Entrypoints,
Targets, APIKeys
- Per-webhook DBs (DATA_DIR) store Events, Deliveries, DeliveryResults
in files named events-{webhook_uuid}.db
- New DATA_DIR env var (default: ./data dev, /data/events prod)
Behavioral changes:
- Webhook creation creates per-webhook DB file
- Webhook deletion hard-deletes per-webhook DB file (config soft-deleted)
- Event ingestion writes to per-webhook DB, not main DB
- Delivery engine polls all per-webhook DBs for pending deliveries
- Database target type marks delivery as immediately successful (events
are already in the dedicated per-webhook DB)
- Event log UI reads from per-webhook DBs with targets from main DB
- Existing webhooks without DB files get them created lazily
Removed:
- ArchivedEvent model (was a half-measure, replaced by per-webhook DBs)
- Event/Delivery/DeliveryResult removed from main DB migrations
Added:
- Comprehensive tests for WebhookDBManager (create, delete, lazy
creation, delivery workflow, multiple webhooks, close all)
- Dockerfile creates /data/events directory
README updates:
- Per-webhook event databases documented as implemented (was Phase 2)
- DATA_DIR added to configuration table
- Docker instructions updated with data volume mount
- Data model diagram updated
- TODO updated (database separation moved to completed)
Closes #15
|
||
|
clawbot
|
6c393ccb78 |
fix: database target writes to dedicated archive table
check / check (push) Successful in 1m43s
The "database" target type now writes events to a separate archived_events table instead of just marking the delivery as done. This table persists independently of internal event retention/pruning, allowing the data to be consumed by external systems or preserved indefinitely. New ArchivedEvent model copies the full event payload (method, headers, body, content_type) along with webhook/entrypoint/event/target IDs. |
||
|
clawbot
|
d4fbd6c110 |
fix: delivery engine nil pointer crash on startup (closes #17)
Store the *database.Database wrapper instead of calling .DB() eagerly at construction time. The GORM *gorm.DB is only available after the database's OnStart hook runs, but the engine constructor runs during fx resolution (before OnStart). Accessing .DB() lazily via the wrapper avoids the nil pointer panic. |
||
|
clawbot
|
7f8469a0f2 |
feat: implement core webhook engine, delivery system, and management UI (Phase 2)
check / check (push) Successful in 1m49s
- Webhook reception handler: look up entrypoint by UUID, verify active,
capture full HTTP request (method, headers, body, content-type), create
Event record, queue Delivery records for each active Target, return 200 OK.
Handles edge cases: unknown UUID → 404, inactive → 410, oversized → 413.
- Delivery engine (internal/delivery): fx-managed background goroutine that
polls for pending/retrying deliveries and dispatches to target type handlers.
Graceful shutdown via context cancellation.
- Target type implementations:
- HTTP: fire-and-forget POST with original headers forwarding
- Retry: exponential backoff (1s, 2s, 4s...) up to max_retries
- Database: immediate success (event already stored)
- Log: slog output with event details
- Webhook management pages with Tailwind CSS + Alpine.js:
- List (/sources): webhooks with entrypoint/target/event counts
- Create (/sources/new): form with auto-created default entrypoint
- Detail (/source/{id}): config, entrypoints, targets, recent events
- Edit (/source/{id}/edit): name, description, retention_days
- Delete (/source/{id}/delete): soft-delete with child records
- Add Entrypoint (/source/{id}/entrypoints): inline form
- Add Target (/source/{id}/targets): type-aware form
- Event Log (/source/{id}/logs): paginated with delivery status
- Updated README: marked completed items, updated naming conventions
table, added delivery engine to package layout and DI docs, updated
column names to reflect entity rename.
- Rebuilt Tailwind CSS for new template classes.
Part of: #15
|