86224d49829d904fd59537b19762b2fa0d2669e0
74
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  clawbotandsneak
|
ee7c626071 |
Implement the database archiving target (closes #43) (#84)
check / check (push) Successful in 4s
Implements the `databaseTarget` as a real archiving target, replacing the always-successful stub. Delivering to a `database` target now writes the full event into a per-webhook archive SQLite file for long-term storage.
## Archive-writer semantics
- **Separate file:** each webhook's full events are written as rows into `archive-{webhookID}.db` under the data dir, distinct from the per-webhook event DB (`events-{webhookID}.db`). The file and its schema are created on first write if missing. Each row carries the full event: body, headers, method, content type, webhook id, entrypoint id, event id, and an archived-at timestamp.
- **Close/reopen with debounce:** after each write the archive handle is closed and reopened, unless the last (re)open was less than one second ago. This lets an operator move the archive file away for offline archiving while bounding file churn under load. A per-webhook `archiveWriter` owns this debounce state and serialises writes.
- **Auto-recreate:** the file is opened create-if-missing (`mode=rwc`) and its schema re-migrated on every open, so if the archive was moved or removed since the last open, the next write recreates it. The writer also detects a missing file before writing and reopens first, so a moved-away file is recreated rather than lost.
- **Optional expiry, validated at creation:** an optional `expiry` in the target's config JSON (e.g. `{"expiry":"720h"}`) is validated when the target is created (`ValidateArchiveExpiry`; bad values are rejected with a 400 at the add-target form, the Slack URL precedent). The default (missing, empty, or `"never"`) keeps rows forever with no pruning. When a positive duration is set, rows older than it (measured from each row's archived-at time) are pruned on every (re)open; because the file is reopened after writes, prune-on-open keeps the archive swept without a separate background sweeper. A set-but-invalid expiry in a stored config (unparseable, zero, or negative) is an error at delivery time too — never a silent default.
- **No-retry, fail-loud:** the target performs a single attempt with no retries. On success it records one successful attempt and marks the delivery delivered. If the archive write fails, the attempt is recorded as failed with the error and the delivery is marked failed — archiving errors never report success.
## Scope
- `internal/delivery/target_database.go` — the `databaseTarget` (no-retry) archives via a per-webhook writer registry; an archive error records a failed attempt and marks the delivery failed.
- `internal/delivery/target_database_archive.go` (new) — the `archiveWriter`, the archived-row model, config/expiry parsing (fail-loud on set-but-invalid values), `ValidateArchiveExpiry`, and prune-on-open.
- `internal/handlers/source_management.go` — database targets get a creation-validated `expiry` config (`buildDatabaseTargetConfig`); the expiry form value is read where the request body is bounded and bad values are rejected with a 400 at target creation.
- `templates/source_detail.html` — the add-target form shows an expiry field for database targets.
- `README.md` — the database-target documentation describes the archiving semantics.
- `internal/delivery/export_test.go`, `internal/delivery/target_database_test.go`, `internal/handlers` tests — tests and their exported shims.
No changes to the `Target` interface or other targets.
## Tests
- a row is archived (both at the writer level and end-to-end through `Deliver`)
- a forced archive failure (bad stored expiry config) yields a `Failed` delivery with a non-success `DeliveryResult` carrying the error and no archive file created
- the file is recreated after removal, with only the post-removal row
- the one-second reopen debounce (rapid writes reopen once; a write after the window reopens again)
- expiry pruning removes rows older than the configured expiry
- expiry config parsing (empty / `never` / duration accepted; unparseable, zero, and negative values error)
- expiry validation at target creation (`TestValidateArchiveExpiry`; valid values build the config, bad values get a 400)
## Validation
`docker build .` exits 0 (fmt-check, lint, test, build all pass).
Closes #43
Co-authored-by: sneak <sneak@sneak.berlin>
Reviewed-on: #84
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
|
||
|
sneak
|
2cc8723997 |
scripts-to-rule-them-all (#59)
check / check (push) Successful in 4s
Reviewed-on: #59 Co-authored-by: sneak <sneak@sneak.berlin> Co-committed-by: sneak <sneak@sneak.berlin> |
||
|
sneak
|
e0b1e7cf54 |
Restore TODO.md and move TODO content out of README (#58)
check / check (push) Successful in 4s
Reviewed-on: #58 Co-authored-by: sneak <sneak@sneak.berlin> Co-committed-by: sneak <sneak@sneak.berlin> |
||
 clawbotandclawbot
|
afe88c601a |
refactor: use pinned golangci-lint Docker image for linting (#55)
check / check (push) Successful in 5s
Closes [issue #50](#50) ## Summary Refactors the Dockerfile to use a separate lint stage with a pinned golangci-lint Docker image, following the pattern used by [sneak/pixa](https://git.eeqj.de/sneak/pixa). This replaces the previous approach of installing golangci-lint via curl in the builder stage. ## Changes ### Dockerfile - **New `lint` stage** using `golangci/golangci-lint:v2.11.3` (Debian-based, pinned by sha256 digest) as a separate build stage - **Builder stage** depends on lint via `COPY --from=lint /src/go.sum /dev/null` — build won't proceed unless linting passes - **Go bumped** from 1.24 to 1.26.1 (`golang:1.26.1-bookworm`, pinned by sha256) - **golangci-lint bumped** from v1.64.8 to v2.11.3 - All three Docker images (golangci-lint, golang, alpine) pinned by sha256 digest - Debian-based golangci-lint image used (not Alpine) because mattn/go-sqlite3 CGO does not compile on musl (off64_t) ### Linter Config (.golangci.yml) - Migrated from v1 to v2 format (`version: "2"` added) - Removed linters no longer available in v2: `gofmt` (handled by `make fmt-check`), `gosimple` (merged into `staticcheck`), `typecheck` (always-on in v2) - Same set of linters enabled — no rules weakened ### Code Fixes (all lint issues from v2 upgrade) - Added package comments to all packages - Added doc comments to all exported types, functions, and methods - Fixed unchecked errors flagged by `errcheck` (sqlDB.Close, os.Setenv in tests, resp.Body.Close, fmt.Fprint) - Fixed unused parameters flagged by `revive` (renamed to `_`) - Fixed `gosec` G120 warnings: added `http.MaxBytesReader` before `r.ParseForm()` calls - Fixed `staticcheck` QF1012: replaced `WriteString(fmt.Sprintf(...))` with `fmt.Fprintf` - Fixed `staticcheck` QF1003: converted if/else chain to tagged switch - Renamed `DeliveryTask` → `Task` to avoid package stutter (`delivery.Task` instead of `delivery.DeliveryTask`) - Renamed shadowed builtin `max` parameter to `upperBound` in `cryptoRandInt` - Used `t.Setenv` instead of `os.Setenv` in tests (auto-restores) ### README.md - Updated version requirements: Go 1.26+, golangci-lint v2.11+ - Updated Dockerfile description in project structure ## Verification `docker build .` passes cleanly — formatting check, linting, all tests, and build all succeed. Co-authored-by: clawbot <clawbot@noreply.git.eeqj.de> Reviewed-on: #55 Co-authored-by: clawbot <clawbot@noreply.example.org> Co-committed-by: clawbot <clawbot@noreply.example.org> |
||
|
clawbotanduser
|
d771fe14df |
fix: detect TLS per-request in CSRF middleware to fix login (#54)
check / check (push) Successful in 1m55s
## Problem After the security hardening in PR #42, login fails with `Forbidden - invalid CSRF token` in production deployments. The CSRF middleware tied its `PlaintextHTTPRequest` wrapping and cookie `Secure` flag to the `IsDev()` environment check. This meant production mode always assumed HTTPS via gorilla/csrf's strict mode, which broke login in common deployment scenarios: 1. **Production behind a TLS-terminating reverse proxy**: gorilla/csrf assumed HTTPS but `r.TLS` was nil (the Go server receives HTTP from the proxy). Origin/Referer scheme mismatches caused `referer not supplied` or `origin invalid` errors. 2. **Production over direct HTTP** (testing/staging with prod config): the `Secure` cookie flag prevented the browser from sending the CSRF cookie back over HTTP, causing `CSRF token invalid` errors. ## Root Cause gorilla/csrf v1.7.3 defaults to HTTPS-strict mode unless `PlaintextHTTPRequest()` is called. In strict mode it: - Forces `requestURL.Scheme = "https"` for Origin/Referer comparisons - Requires a `Referer` header on POST and rejects `http://` Referer schemes - The `csrf.Secure(true)` option makes the browser refuse to send the CSRF cookie over HTTP The old code only called `PlaintextHTTPRequest()` in dev mode, leaving prod mode permanently stuck in HTTPS-strict mode regardless of the actual transport. ## Fix Detect the actual transport protocol **per-request** using: - `r.TLS != nil` — direct TLS connection to the Go server - `X-Forwarded-Proto: https` header — TLS-terminating reverse proxy Two gorilla/csrf middleware instances are maintained (one with `Secure: true`, one with `Secure: false`) since `csrf.Secure()` is a creation-time option. Both use the same signing key, so cookies are interchangeable. | Scenario | Cookie Secure | Origin/Referer Mode | |---|---|---| | Direct TLS (`r.TLS != nil`) | ✅ Secure | Strict (HTTPS scheme) | | Behind TLS proxy (`X-Forwarded-Proto: https`) | ✅ Secure | Strict (HTTPS scheme) | | Plaintext HTTP | ❌ Non-Secure | Relaxed (PlaintextHTTPRequest) | CSRF token validation (cookie + form double-submit) is always enforced regardless of mode. ## Testing - Added `TestCSRF_ProdMode_PlaintextHTTP_POSTWithValidToken` — prod mode over plaintext HTTP - Added `TestCSRF_ProdMode_BehindProxy_POSTWithValidToken` — prod mode behind TLS proxy - Added `TestCSRF_ProdMode_DirectTLS_POSTWithValidToken` — prod mode with direct TLS - Added `TestCSRF_ProdMode_PlaintextHTTP_POSTWithoutToken` — token still required - Added `TestIsClientTLS_*` — TLS detection unit tests - All existing CSRF tests pass unchanged - `docker build .` passes (includes `make check`) - Manual verification: built and ran the container in both `dev` and `prod` modes, confirmed login succeeds in both Closes #53 Co-authored-by: user <user@Mac.lan guest wan> Reviewed-on: #54 Co-authored-by: clawbot <clawbot@noreply.example.org> Co-committed-by: clawbot <clawbot@noreply.example.org> |
||
|
|
33e2140a5a |
feat: redirect root path based on auth state (#52)
check / check (push) Successful in 1m54s
Closes #51 The root path `/` now checks for an authenticated session and redirects accordingly: - **Authenticated users** → `303 See Other` redirect to `/sources` (the webhook dashboard) - **Unauthenticated users** → `303 See Other` redirect to `/pages/login` ### Changes - **`internal/handlers/index.go`** — Replaced the template-rendering `HandleIndex()` with a session-checking redirect handler. Removed `formatUptime` helper (dead code after this change). - **`internal/handlers/handlers.go`** — Removed `index.html` from the template map (no longer rendered). - **`internal/handlers/handlers_test.go`** — Replaced the old "handler is not nil" test with two proper redirect tests: - `unauthenticated redirects to login` — verifies 303 to `/pages/login` - `authenticated redirects to sources` — sets up an authenticated session cookie, verifies 303 to `/sources` - Removed `TestFormatUptime` (tested dead code). - **`README.md`** — Updated the API endpoints table to describe the new redirect behavior. ### How it works The handler calls `session.Get(r)` and `session.IsAuthenticated(sess)` — the same pattern used by the `RequireAuth` middleware and `HandleLoginPage`. No new dependencies or session logic introduced. The login flow is unaffected: `HandleLoginSubmit` redirects to `/` after successful login, which now forwards to `/sources` (one extra redirect hop, but correct and clean). Co-authored-by: clawbot <clawbot@noreply.git.eeqj.de> Co-authored-by: clawbot <clawbot@eeqj.de> Reviewed-on: #52 Co-authored-by: clawbot <clawbot@noreply.example.org> Co-committed-by: clawbot <clawbot@noreply.example.org> |
||
|
|
f003ec7141 |
fix: use absolute path for dev DATA_DIR default, clarify env docs (#46)
check / check (push) Successful in 1m9s
Closes #45. ## Problem 1. The README didn't clearly explain what `WEBHOOKER_ENVIRONMENT=dev` vs `prod` actually changes. 2. The dev-mode default for `DATA_DIR` was `./data` — a relative path whose meaning depends on the working directory. There's no reason to use a relative path even in development. ## Changes ### Code (`internal/config/config.go`) - Replace the dev default `DATA_DIR` from `./data` to `$XDG_DATA_HOME/webhooker` (falling back to `$HOME/.local/share/webhooker`). This follows the XDG Base Directory Specification and ensures the data directory is always an absolute path regardless of the working directory. - Add `devDataDir()` helper that resolves the XDG path, with a `/tmp/webhooker` last-resort fallback if `$HOME` can't be determined. ### Tests (`internal/config/config_test.go`) - `TestDevDataDir`: verifies XDG_DATA_HOME is respected, HOME fallback works, and the result is always absolute. - `TestDevDefaultDataDirIsAbsolute`: integration test that creates a full Config via fx and asserts the dev default DataDir is absolute. ### README - Add a table documenting exactly what `dev` vs `prod` changes: DATA_DIR default, CORS policy, and session cookie Secure flag. - Clarify that log format and security headers are independent of the environment setting. - Update the DATA_DIR default in the configuration variable table. Co-authored-by: clawbot <clawbot@eeqj.de> Co-authored-by: user <user@Mac.lan guest wan> Reviewed-on: #46 Co-authored-by: clawbot <clawbot@noreply.example.org> Co-committed-by: clawbot <clawbot@noreply.example.org> |
||
|
|
60786c5019 |
feat: add CSRF protection, SSRF prevention, and login rate limiting (#42)
check / check (push) Successful in 4s
## Security Hardening This PR implements three security hardening issues: ### CSRF Protection (closes #35) - Session-based CSRF tokens with cryptographically random 256-bit generation - Constant-time token comparison to prevent timing attacks - CSRF middleware applied to `/pages`, `/sources`, `/source`, and `/user` routes - Hidden `csrf_token` field added to all 12+ POST forms in templates - Excluded from `/webhook` (inbound webhook POSTs) and `/api` (stateless API) ### SSRF Prevention (closes #36) - `ValidateTargetURL()` blocks private/reserved IP ranges at target creation time - Blocked ranges: `127.0.0.0/8`, `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `169.254.0.0/16`, `::1`, `fc00::/7`, `fe80::/10`, plus multicast, reserved, test-net, and CGN ranges - SSRF-safe HTTP transport with custom `DialContext` in the delivery engine for defense-in-depth (prevents DNS rebinding attacks) - Only `http` and `https` schemes allowed ### Login Rate Limiting (closes #37) - Per-IP rate limiter using `golang.org/x/time/rate` - 5 attempts per minute per IP on `POST /pages/login` - GET requests (form rendering) pass through unaffected - Automatic cleanup of stale per-IP limiter entries every 5 minutes - `X-Forwarded-For` and `X-Real-IP` header support for reverse proxies ### Files Changed **New files:** - `internal/middleware/csrf.go` + tests — CSRF middleware - `internal/middleware/ratelimit.go` + tests — Login rate limiter - `internal/delivery/ssrf.go` + tests — SSRF validation + safe transport **Modified files:** - `internal/server/routes.go` — Wire CSRF and rate limit middleware - `internal/handlers/handlers.go` — Inject CSRF token into template data - `internal/handlers/source_management.go` — SSRF validation on target creation - `internal/delivery/engine.go` — SSRF-safe HTTP transport for production - All form templates — Added hidden `csrf_token` fields - `README.md` — Updated Security section and TODO checklist `docker build .` passes (lint + tests + build). Co-authored-by: clawbot <clawbot@noreply.git.eeqj.de> Co-authored-by: clawbot <clawbot@eeqj.de> Co-authored-by: Jeffrey Paul <sneak@noreply.example.org> Reviewed-on: #42 Co-authored-by: clawbot <clawbot@noreply.example.org> Co-committed-by: clawbot <clawbot@noreply.example.org> |
||
|
clawbotanduser
|
8d702a16c6 |
feat: add Slack target type for incoming webhook notifications (#47)
check / check (push) Successful in 4s
## Summary Adds a new `slack` target type that sends webhook events as formatted messages to any Slack-compatible incoming webhook URL (Slack, Mattermost, and other compatible services). closes #44 ## What it does When a webhook event is received, the Slack target: 1. Formats a human-readable message with event metadata (HTTP method, content type, timestamp, body size) 2. Pretty-prints the payload in a code block — JSON payloads get indented formatting, non-JSON payloads are shown as raw text 3. Truncates large payloads at 3500 characters to keep Slack messages reasonable 4. POSTs the message as a `{"text": "..."}` JSON payload to the configured webhook URL ## Changes - **`internal/database/model_target.go`** — Add `TargetTypeSlack` constant - **`internal/delivery/engine.go`** — Add `SlackTargetConfig` struct, `deliverSlack` method, `FormatSlackMessage` function (exported), `parseSlackConfig` helper. Route slack targets in `processDelivery` switch. - **`internal/handlers/source_management.go`** — Handle `slack` type in `HandleTargetCreate`, building `webhook_url` config from the URL form field - **`templates/source_detail.html`** — Add "Slack" option to target type dropdown with URL field and helper text - **`README.md`** — Document the new target type, update roadmap ## Tests - `TestParseSlackConfig_Valid` / `_Empty` / `_MissingWebhookURL` — Config parsing - `TestFormatSlackMessage_JSONBody` / `_NonJSONBody` / `_EmptyBody` / `_LargeJSONTruncated` — Message formatting - `TestDeliverSlack_Success` / `_Failure` / `_InvalidConfig` — End-to-end delivery - `TestProcessDelivery_RoutesToSlack` — Routing from processDelivery switch All existing tests continue to pass. `docker build .` (which runs `make check`) passes clean. Co-authored-by: user <user@Mac.lan guest wan> Reviewed-on: #47 Co-authored-by: clawbot <clawbot@noreply.example.org> Co-committed-by: clawbot <clawbot@noreply.example.org> |
||
|
clawbotandclawbot
|
1fbcf96581 |
security: add headers middleware, session regeneration, and body size limits (#41)
check / check (push) Successful in 1m47s
## Summary This PR implements three security hardening measures: ### Security Headers Middleware (closes #34) Adds a `SecurityHeaders()` middleware applied globally to all routes. Every response now includes: - `Strict-Transport-Security: max-age=63072000; includeSubDomains; preload` - `X-Content-Type-Options: nosniff` - `X-Frame-Options: DENY` - `Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'` - `Referrer-Policy: strict-origin-when-cross-origin` - `Permissions-Policy: camera=(), microphone=(), geolocation=()` ### Session Fixation Prevention (closes #38) Adds a `Regenerate()` method to the session manager that destroys the old session and creates a new one with a fresh ID, copying all session values. Called after successful login to prevent session fixation attacks. ### Request Body Size Limits (closes #39) Adds a `MaxBodySize()` middleware using `http.MaxBytesReader` to limit POST/PUT/PATCH request bodies to 1 MB. Applied to all form endpoints (`/pages`, `/sources`, `/source/*`). ## Files Changed - `internal/middleware/middleware.go` — Added `SecurityHeaders()` and `MaxBodySize()` middleware - `internal/session/session.go` — Added `Regenerate()` method for session fixation prevention - `internal/handlers/auth.go` — Updated login handler to regenerate session after authentication - `internal/server/routes.go` — Added SecurityHeaders globally, MaxBodySize to form route groups - `README.md` — Documented new middleware in stack, updated Security section, moved items to completed TODO closes #34, closes #38, closes #39 Co-authored-by: clawbot <clawbot@noreply.git.eeqj.de> Reviewed-on: #41 Co-authored-by: clawbot <clawbot@noreply.example.org> Co-committed-by: clawbot <clawbot@noreply.example.org> |
||
|
clawbot
|
8e00e40008 |
docs: fix stale references to development mode and retry target type
check / check (push) Successful in 5s
- README.md: remove 'in development mode' from admin user creation description (admin user creation is unconditional) - internal/delivery/engine.go: remove 'and retry' from HTTPTargetConfig comment (retry was merged into http target type) - internal/delivery/engine_test.go: remove '/retry' from newHTTPTargetConfig comment for consistency |
||
|
clawbot
|
25e27cc57f |
refactor: merge retry target type into http (max_retries=0 = fire-and-forget)
check / check (push) Successful in 1m46s
|
||
|
clawbot
|
4dd4dfa5eb |
chore: consolidate DBURL into DATA_DIR, codebase audit for 1.0.0
check / check (push) Successful in 56s
DBURL → DATA_DIR consolidation:
- Remove DBURL env var entirely; main DB now lives at {DATA_DIR}/webhooker.db
- database.go constructs DB path from config.DataDir, ensures dir exists
- Update DATA_DIR prod default from /data/events to /data
- Update all tests to use DataDir instead of DBURL
- Update Dockerfile: /data (not /data/events) for all SQLite databases
- Update README configuration table, Docker examples, architecture docs
Dead code removal:
- Remove unused IndexResponse struct (handlers/index.go)
- Remove unused TemplateData struct (handlers/handlers.go)
Stale comment cleanup:
- Remove TODO in server.go (DB cleanup handled by fx lifecycle)
- Fix nolint:golint → nolint:revive on ServerParams for consistency
- Clean up verbose middleware/routing comments in routes.go
- Fix TODO fan-out description (worker pool, not goroutine-per-target)
.gitignore fixes:
- Add data/ directory to gitignore
- Remove stale config.yaml entry (env-only config since rework)
|
||
|
clawbot
|
49852e7506 |
refactor: remove file-based configuration, use env vars only
check / check (push) Successful in 1m0s
Remove the entire pkg/config package (Viper-based YAML config file loader) and simplify internal/config to read all settings directly from environment variables via os.Getenv(). This eliminates the spurious "Failed to load config" log messages that appeared when no config.yaml file was present. - Delete pkg/config/ (YAML loader, resolver, manager, tests) - Delete configs/config.yaml.example - Simplify internal/config helper functions to use os.Getenv() with defaults instead of falling back to pkgconfig - Update tests to set env vars directly instead of creating in-memory YAML config files via afero - Remove afero, cloud.google.com/*, aws-sdk-go dependencies from go.mod - Update README: document env-var-only configuration, remove YAML/Viper references - Keep godotenv/autoload for .env file convenience in local development closes #27 |
||
|
clawbot
|
10db6c5b84 |
refactor: bounded worker pool with DB-mediated retry fallback
check / check (push) Successful in 58s
Replace unbounded goroutine-per-delivery fan-out with a fixed-size worker pool (10 workers). Channels serve as bounded queues (10,000 buffer). Workers are the only goroutines doing HTTP delivery. When retry channel overflows, timers are dropped instead of re-armed. The delivery stays in 'retrying' status in the DB and a periodic sweep (every 60s) recovers orphaned retries. The database is the durable fallback — same path used on startup recovery. Addresses owner feedback on circuit breaker recovery goroutine flood. |
||
|
clawbot
|
9b4ae41c44 |
feat: parallel fan-out delivery + circuit breaker for retry targets
check / check (push) Successful in 1m52s
- Fan out all targets for an event in parallel goroutines (fire-and-forget) - Add per-target circuit breaker for retry targets (closed/open/half-open) - Circuit breaker trips after 5 consecutive failures, 30s cooldown - Open circuit skips delivery and reschedules after cooldown - Half-open allows one probe delivery to test recovery - HTTP/database/log targets unaffected (no circuit breaker) - Recovery path also fans out in parallel - Update README with parallel delivery and circuit breaker docs |
||
|
clawbot
|
32bd40b313 |
refactor: self-contained delivery tasks — engine delivers without DB reads in happy path
check / check (push) Successful in 58s
The webhook handler now builds DeliveryTask structs carrying all target config and event data inline (for bodies ≤16KB) and sends them through the delivery channel. In the happy path, the engine delivers without reading from any database — it only writes to record delivery results. For large bodies (≥16KB), Body is nil and the engine fetches it from the per-webhook database on demand. Retry timers also carry the full DeliveryTask, so retries avoid unnecessary DB reads. The database is used for crash recovery only: on startup the engine scans for interrupted pending/retrying deliveries and re-queues them. Implements owner feedback from issue #15: > the message in the <=16KB case should have everything it needs to do > its delivery. it shouldn't touch the db until it has a success or > failure to record. |
||
|
clawbot
|
9b9ee1718a |
refactor: auto-generate session key and store in database
check / check (push) Successful in 57s
Remove SESSION_KEY env var requirement. On first startup, a cryptographically secure 32-byte key is generated and stored in a new settings table. Subsequent startups load the key from the database. - Add Setting model (key-value table) for application config - Add Database.GetOrCreateSessionKey() method - Session manager initializes in OnStart after database is connected - Remove DevSessionKey constant and SESSION_KEY env var handling - Remove prod validation requiring SESSION_KEY - Update README: config table, Docker instructions, security notes - Update config.yaml.example - Update all tests to remove SessionKey references Addresses owner feedback on issue #15. |
||
|
clawbot
|
5e683af2a4 |
refactor: event-driven delivery engine with channel notifications and timer-based retries
check / check (push) Successful in 58s
Replace the polling-based delivery engine with a fully event-driven architecture using Go channels and goroutines: - Webhook handler notifies engine via buffered channel after creating delivery records, with inline event data for payloads < 16KB - Large payloads (>= 16KB) use pointer semantics (Body *string = nil) and are fetched from DB on demand, keeping channel memory bounded - Failed retry-target deliveries schedule Go timers with exponential backoff; timers fire into a separate retry channel when ready - On startup, engine scans DB once to recover interrupted deliveries (pending processed immediately, retrying get timers for remaining backoff) - DB stores delivery status for crash recovery only, not for inter-component communication during normal operation - delivery.Notifier interface decouples handlers from engine; fx wires *Engine as Notifier No more periodic polling. No more wasted cycles when idle. |
||
|
clawbot
|
43c22a9e9a |
feat: implement per-webhook event databases
check / check (push) Successful in 1m50s
Split data storage into main application DB (config only) and
per-webhook event databases (one SQLite file per webhook).
Architecture changes:
- New WebhookDBManager component manages per-webhook DB lifecycle
(create, open, cache, delete) with lazy connection pooling via sync.Map
- Main DB (DBURL) stores only config: Users, Webhooks, Entrypoints,
Targets, APIKeys
- Per-webhook DBs (DATA_DIR) store Events, Deliveries, DeliveryResults
in files named events-{webhook_uuid}.db
- New DATA_DIR env var (default: ./data dev, /data/events prod)
Behavioral changes:
- Webhook creation creates per-webhook DB file
- Webhook deletion hard-deletes per-webhook DB file (config soft-deleted)
- Event ingestion writes to per-webhook DB, not main DB
- Delivery engine polls all per-webhook DBs for pending deliveries
- Database target type marks delivery as immediately successful (events
are already in the dedicated per-webhook DB)
- Event log UI reads from per-webhook DBs with targets from main DB
- Existing webhooks without DB files get them created lazily
Removed:
- ArchivedEvent model (was a half-measure, replaced by per-webhook DBs)
- Event/Delivery/DeliveryResult removed from main DB migrations
Added:
- Comprehensive tests for WebhookDBManager (create, delete, lazy
creation, delivery workflow, multiple webhooks, close all)
- Dockerfile creates /data/events directory
README updates:
- Per-webhook event databases documented as implemented (was Phase 2)
- DATA_DIR added to configuration table
- Docker instructions updated with data volume mount
- Data model diagram updated
- TODO updated (database separation moved to completed)
Closes #15
|
||
|
clawbot
|
7f8469a0f2 |
feat: implement core webhook engine, delivery system, and management UI (Phase 2)
check / check (push) Successful in 1m49s
- Webhook reception handler: look up entrypoint by UUID, verify active,
capture full HTTP request (method, headers, body, content-type), create
Event record, queue Delivery records for each active Target, return 200 OK.
Handles edge cases: unknown UUID → 404, inactive → 410, oversized → 413.
- Delivery engine (internal/delivery): fx-managed background goroutine that
polls for pending/retrying deliveries and dispatches to target type handlers.
Graceful shutdown via context cancellation.
- Target type implementations:
- HTTP: fire-and-forget POST with original headers forwarding
- Retry: exponential backoff (1s, 2s, 4s...) up to max_retries
- Database: immediate success (event already stored)
- Log: slog output with event details
- Webhook management pages with Tailwind CSS + Alpine.js:
- List (/sources): webhooks with entrypoint/target/event counts
- Create (/sources/new): form with auto-created default entrypoint
- Detail (/source/{id}): config, entrypoints, targets, recent events
- Edit (/source/{id}/edit): name, description, retention_days
- Delete (/source/{id}/delete): soft-delete with child records
- Add Entrypoint (/source/{id}/entrypoints): inline form
- Add Target (/source/{id}/targets): type-aware form
- Event Log (/source/{id}/logs): paginated with delivery status
- Updated README: marked completed items, updated naming conventions
table, added delivery engine to package layout and DI docs, updated
column names to reflect entity rename.
- Rebuilt Tailwind CSS for new template classes.
Part of: #15
|
||
|
|
b5cf4c3d2f |
docs: comprehensive README rewrite with complete service specification (#13)
check / check (push) Successful in 4s
## Summary Rewrites README.md from a basic scaffold into a comprehensive service description and specification that documents the entire webhooker application. closes #3 ## What Changed ### Naming Scheme Proposes a clear naming scheme for the data model entities: - **Processor → Webhook**: The top-level configuration entity that groups entrypoints and targets - **Webhook → Entrypoint**: The receiver URL (`/hooks/<uuid>`) where external services POST events - **Target**: Unchanged — delivery destinations for events This is documented as the target architecture in the README. The actual code rename is tracked in [issue #12](#12). ### Data Model Documentation Documents all 8 entities with: - Complete field tables (name, type, description) for every entity - Relationship descriptions (belongs-to, has-many) - Enum values for TargetType and DeliveryStatus - Entity relationship diagram (ASCII) - Common fields from BaseModel ### Database Architecture Documents the separate database architecture: - Main application DB: users, webhook configs, entrypoints, targets, API keys - Per-webhook event DBs: events, deliveries, delivery results - Rationale for separation (isolation, lifecycle, clean deletion, per-webhook retention, performance) ### Other Sections - Complete API endpoint tables (current + planned) - Package layout with file descriptions - Request flow diagram - Middleware stack documentation - Authentication design (web sessions + planned API keys) - Security measures - Rate limiting design - Dependency injection order - Docker build pipeline description - Phased TODO roadmap with links to filed issues - License set to MIT ### Code Style Divergence Issues Filed As part of reviewing the code against sneak/prompts standards: - [#7](#7) — Templates should use go:embed - [#8](#8) — Logger should use slog.LevelVar - [#9](#9) — Source management routes lack auth middleware - [#10](#10) — Config should prefer environment variables - [#11](#11) — Redundant godotenv/autoload import - [#12](#12) — Rename Processor → Webhook, Webhook → Entrypoint ## Verification - `make fmt` — ✅ passes - `docker build .` — ✅ passes (README-only change, no code modifications) Co-authored-by: clawbot <clawbot@noreply.git.eeqj.de> Co-authored-by: Jeffrey Paul <sneak@noreply.example.org> Reviewed-on: #13 Co-authored-by: clawbot <clawbot@noreply.example.org> Co-committed-by: clawbot <clawbot@noreply.example.org> |
||
|
clawbotandclawbot
|
f9a9569015 |
feat: bring repo up to REPO_POLICIES standards (#6)
check / check (push) Successful in 8s
## Summary This PR brings the webhooker repo into full REPO_POLICIES compliance, addressing both [issue #1](#1) and [issue #2](#2). ## Changes ### New files - **`cmd/webhooker/main.go`** — The missing application entry point. Uses Uber fx to wire together all internal packages (config, database, logger, server, handlers, middleware, healthcheck, globals, session). Minimal glue code. - **`REPO_POLICIES.md`** — Fetched from authoritative source (`sneak/prompts`) - **`.editorconfig`** — Fetched from authoritative source - **`.dockerignore`** — Sensible Go project exclusions - **`.gitea/workflows/check.yml`** — CI workflow that runs `docker build .` on push to any branch (Gitea Actions format, actions/checkout pinned by sha256) - **`configs/config.yaml.example`** — Moved from root `config.yaml` ### Modified files - **`Makefile`** — Complete rewrite with all REPO_POLICIES required targets: `test`, `lint`, `fmt`, `fmt-check`, `check`, `build`, `hooks`, `docker`, `clean`, plus `dev`, `run`, `deps` - **`Dockerfile`** — Complete rewrite: - Builder: `golang:1.24` (Debian-based, pinned by `sha256:d2d2bc1c84f7...`). Debian needed because `gorm.io/driver/sqlite` pulls `mattn/go-sqlite3` (CGO) which fails on Alpine musl. - golangci-lint v1.64.8 installed from GitHub release archive with sha256 verification (v1.x because `.golangci.yml` uses v1 config format) - Runs `make check` (fmt-check + lint + test + build) as build step - Final stage: `alpine:3.21` (pinned by `sha256:c3f8e73fdb79...`) with non-root user, healthcheck, port 8080 - **`README.md`** — Rewritten with all required REPO_POLICIES sections: description line with name/purpose/category/license/author, Getting Started, Rationale, Design, TODO (integrated from TODO.md), License, Author - **`.gitignore`** — Fixed `webhooker` pattern to `/webhooker` (was blocking `cmd/webhooker/`), added `config.yaml` to prevent committing runtime config with secrets - **`static/static.go`** — Removed `vendor` from embed directive (directory was empty/missing) - **`internal/database/database_test.go`** — Fixed to use in-memory config via `afero.MemMapFs` instead of depending on `config.yaml` on disk. Test is now properly isolated. - **`go.mod`/`go.sum`** — `go mod tidy` ### Removed files - **`TODO.md`** — Content integrated into README.md TODO section - **`config.yaml`** — Moved to `configs/config.yaml.example` ## Verification - `docker build .` passes (lint ✅, test ✅, build ✅) - All existing tests pass with no modifications to assertions or test logic - `.golangci.yml` untouched closes #1 closes #2 Co-authored-by: clawbot <clawbot@noreply.git.eeqj.de> Reviewed-on: #6 Co-authored-by: clawbot <clawbot@noreply.example.org> Co-committed-by: clawbot <clawbot@noreply.example.org> |
||
|
sneak
|
1244f3e2d5 | initial |