From b5cf4c3d2ff9a3850cc55d46ece350d6c520d06f Mon Sep 17 00:00:00 2001 From: clawbot Date: Mon, 2 Mar 2026 00:43:55 +0100 Subject: [PATCH] docs: comprehensive README rewrite with complete service specification (#13) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary Rewrites README.md from a basic scaffold into a comprehensive service description and specification that documents the entire webhooker application. closes https://git.eeqj.de/sneak/webhooker/issues/3 ## What Changed ### Naming Scheme Proposes a clear naming scheme for the data model entities: - **Processor → Webhook**: The top-level configuration entity that groups entrypoints and targets - **Webhook → Entrypoint**: The receiver URL (`/hooks/`) where external services POST events - **Target**: Unchanged — delivery destinations for events This is documented as the target architecture in the README. The actual code rename is tracked in [issue #12](https://git.eeqj.de/sneak/webhooker/issues/12). ### Data Model Documentation Documents all 8 entities with: - Complete field tables (name, type, description) for every entity - Relationship descriptions (belongs-to, has-many) - Enum values for TargetType and DeliveryStatus - Entity relationship diagram (ASCII) - Common fields from BaseModel ### Database Architecture Documents the separate database architecture: - Main application DB: users, webhook configs, entrypoints, targets, API keys - Per-webhook event DBs: events, deliveries, delivery results - Rationale for separation (isolation, lifecycle, clean deletion, per-webhook retention, performance) ### Other Sections - Complete API endpoint tables (current + planned) - Package layout with file descriptions - Request flow diagram - Middleware stack documentation - Authentication design (web sessions + planned API keys) - Security measures - Rate limiting design - Dependency injection order - Docker build pipeline description - Phased TODO roadmap with links to filed issues - License set to MIT ### Code Style Divergence Issues Filed As part of reviewing the code against sneak/prompts standards: - [#7](https://git.eeqj.de/sneak/webhooker/issues/7) — Templates should use go:embed - [#8](https://git.eeqj.de/sneak/webhooker/issues/8) — Logger should use slog.LevelVar - [#9](https://git.eeqj.de/sneak/webhooker/issues/9) — Source management routes lack auth middleware - [#10](https://git.eeqj.de/sneak/webhooker/issues/10) — Config should prefer environment variables - [#11](https://git.eeqj.de/sneak/webhooker/issues/11) — Redundant godotenv/autoload import - [#12](https://git.eeqj.de/sneak/webhooker/issues/12) — Rename Processor → Webhook, Webhook → Entrypoint ## Verification - `make fmt` — ✅ passes - `docker build .` — ✅ passes (README-only change, no code modifications) Co-authored-by: clawbot Co-authored-by: Jeffrey Paul Reviewed-on: https://git.eeqj.de/sneak/webhooker/pulls/13 Co-authored-by: clawbot Co-committed-by: clawbot --- README.md | 801 +++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 675 insertions(+), 126 deletions(-) diff --git a/README.md b/README.md index 7b793dc..a088e35 100644 --- a/README.md +++ b/README.md @@ -1,188 +1,737 @@ # webhooker -webhooker is a Go web application by [@sneak](https://sneak.berlin) that -receives, stores, and proxies webhooks to configured targets with retry -support, observability, and a management web UI. License: pending. +webhooker is a self-hosted webhook proxy and store-and-forward service +written in [Go](https://golang.org) by +[@sneak](https://sneak.berlin). It receives webhooks from external +services, durably stores them, and delivers them to configured targets +with retry support, logging, and observability. Category: infrastructure +/ web service. License: MIT. ## Getting Started +### Prerequisites + +- Go 1.24+ +- golangci-lint v1.64+ +- Docker (for containerized deployment) + +### Quick Start + ```bash # Clone the repo git clone https://git.eeqj.de/sneak/webhooker.git cd webhooker -# Install dependencies +# Install Go dependencies make deps -# Copy example config -cp configs/config.yaml.example config.yaml - -# Run in development mode -make dev - # Run all checks (format, lint, test, build) make check +# Run in development mode (uses SQLite in current directory) +make dev + # Build Docker image make docker ``` -### Environment Variables +### Development Commands -- `WEBHOOKER_ENVIRONMENT` — `dev` or `prod` (default: `dev`) -- `DEBUG` — Enable debug logging -- `PORT` — Server port (default: `8080`) -- `DBURL` — Database connection string -- `SESSION_KEY` — Base64-encoded 32-byte session key (required in prod) -- `METRICS_USERNAME` — Username for metrics endpoint -- `METRICS_PASSWORD` — Password for metrics endpoint -- `SENTRY_DSN` — Sentry error reporting (optional) +```bash +make fmt # Format code (gofmt + goimports) +make lint # Run golangci-lint +make test # Run tests with race detection +make check # fmt-check + lint + test + build (CI gate) +make build # Build binary to bin/webhooker +make dev # go run ./cmd/webhooker +make docker # Build Docker image +make hooks # Install git pre-commit hook that runs make check +``` + +### Configuration + +webhooker uses a YAML configuration file with environment-specific +overrides, loaded via the `pkg/config` library (Viper-based). The +environment is selected by setting `WEBHOOKER_ENVIRONMENT` to `dev` or +`prod` (default: `dev`). + +Configuration is resolved in this order (highest priority first): + +1. Environment variables +2. `.env` file (loaded via `godotenv/autoload`) +3. Config file values for the active environment +4. Config file defaults + +| Variable | Description | Default | +| ----------------------- | ----------------------------------- | -------- | +| `WEBHOOKER_ENVIRONMENT` | `dev` or `prod` | `dev` | +| `PORT` | HTTP listen port | `8080` | +| `DBURL` | SQLite database connection string | *(required)* | +| `SESSION_KEY` | Base64-encoded 32-byte session key | *(required in prod)* | +| `DEBUG` | Enable debug logging | `false` | +| `METRICS_USERNAME` | Basic auth username for `/metrics` | `""` | +| `METRICS_PASSWORD` | Basic auth password for `/metrics` | `""` | +| `SENTRY_DSN` | Sentry error reporting DSN | `""` | + +On first startup in development mode, webhooker creates an `admin` user +with a randomly generated password and logs it to stdout. This password +is only displayed once. + +### Running with Docker + +```bash +docker run -d \ + -p 8080:8080 \ + -v /path/to/data:/data \ + -e DBURL="file:/data/webhooker.db?cache=shared&mode=rwc" \ + -e SESSION_KEY="" \ + -e WEBHOOKER_ENVIRONMENT=prod \ + webhooker:latest +``` + +The container runs as a non-root user (`webhooker`, UID 1000), exposes +port 8080, and includes a health check against +`/.well-known/healthcheck`. ## Rationale -Webhook integrations between services are fragile: the receiving service -must be up when the webhook fires, there is no built-in retry for most -webhook senders, and there is no visibility into what was sent or when. -webhooker solves this by acting as a reliable intermediary that receives -webhooks, stores them, and delivers them to configured targets — with -optional retries, logging, and Prometheus metrics for observability. +Webhook integrations between services are inherently fragile. The +receiving service must be online when the webhook fires, most webhook +senders provide no built-in retry mechanism, and there is no standard +way to inspect what was sent, when it was sent, or whether delivery +succeeded. -Use cases include: +webhooker solves this by acting as a durable intermediary: -- Store-and-forward with unlimited retries for unreliable receivers -- Prometheus/Grafana metric analysis of webhook frequency, size, and - handler performance -- Introspection and debugging of webhook payloads -- Redelivery of webhook events for application testing -- Fan-out delivery of webhooks to multiple targets -- HA ingestion endpoint for delivery to less reliable systems +1. **Reliable ingestion** — webhooker is always ready to accept incoming + webhooks. It stores every received event before attempting any + delivery, so nothing is lost if downstream targets are unavailable. + +2. **Guaranteed delivery** — Events are queued for delivery to each + configured target. Failed deliveries are retried with configurable + backoff. Every delivery attempt is logged with status codes, response + bodies, and timing. + +3. **Observability** — Full request/response logging for every webhook + received and every delivery attempted. Prometheus metrics expose + volume, latency, and error rates. The web UI provides real-time + visibility into event flow. + +4. **Fan-out** — A single incoming webhook can be delivered to multiple + targets simultaneously. This enables patterns like forwarding a + GitHub webhook to both a deployment service and a Slack channel. + +5. **Replay** — Stored events can be manually redelivered for debugging + or testing, without requiring the original sender to fire the webhook + again. + +### Use Cases + +- **Store-and-forward** with configurable retries for unreliable + receivers +- **Observability** via Prometheus metrics on webhook frequency, payload + size, and delivery performance +- **Debugging** and introspection of webhook payloads in the web UI +- **Replay** of webhook events for application testing and development +- **Fan-out** delivery of a single webhook to multiple downstream + targets +- **High-availability ingestion** for delivery to less reliable backend + systems ## Design -### Architecture +### Architecture Overview -webhooker uses Uber's fx dependency injection library for managing -application lifecycle. It uses `log/slog` for structured logging, GORM -for database access, and SQLite (via `modernc.org/sqlite`, pure Go, no -CGO) for storage. HTTP routing uses chi. +webhooker is structured as a standard Go HTTP server following the +[sneak/prompts GO_HTTP_SERVER_CONVENTIONS](https://git.eeqj.de/sneak/prompts/src/branch/main/prompts/GO_HTTP_SERVER_CONVENTIONS.md). +It uses: -### Rate Limiting +- **[Uber fx](https://go.uber.org/fx)** for dependency injection and + lifecycle management +- **[go-chi](https://github.com/go-chi/chi)** for HTTP routing +- **[GORM](https://gorm.io)** for database access with + **[modernc.org/sqlite](https://pkg.go.dev/modernc.org/sqlite)** as + the runtime SQLite driver. Note: `gorm.io/driver/sqlite` transitively + depends on `mattn/go-sqlite3`, which requires CGO at build time (see + [Docker](#docker) section) +- **[slog](https://pkg.go.dev/log/slog)** (stdlib) for structured + logging with TTY detection (text for dev, JSON for prod) +- **[gorilla/sessions](https://github.com/gorilla/sessions)** for + encrypted cookie-based session management +- **[Prometheus](https://prometheus.io)** for metrics, served at + `/metrics` behind basic auth +- **[Sentry](https://sentry.io)** for optional error reporting -Global rate limiting middleware (e.g. per-IP throttling applied at the -router level) must **not** apply to webhook receiver endpoints -(`/webhook/{uuid}`). Webhook endpoints receive automated traffic from -external services at unpredictable rates, and blanket rate limits would -cause legitimate webhook deliveries to be dropped. +### Naming Conventions -Instead, each webhook endpoint has its own individually configurable rate -limit, applied within the webhook handler itself. By default, no rate -limit is applied — webhook endpoints accept traffic as fast as it arrives. -Rate limits can be configured on a per-webhook basis in the application -when needed (e.g. to protect against a misbehaving sender). +This README uses the target naming scheme for the application's core +entities. The current codebase uses older names that will be updated in +a future refactor (see +[issue #12](https://git.eeqj.de/sneak/webhooker/issues/12)): -### Database Architecture +| README (target name) | Current code name | Description | +| --------------------- | ----------------- | ----------- | +| **Webhook** | `Processor` | Top-level configuration entity grouping entrypoints and targets | +| **Entrypoint** | `Webhook` | A receiver URL where external services POST events | +| **Target** | `Target` | A delivery destination for events | -webhooker uses separate SQLite database files rather than a single -monolithic database: - -- **Main application database** — Stores application configuration and - all standard webapp data: users, sessions, API keys, and global - settings. -- **Per-processor databases** — Each processor (working name — a better - term is needed) gets its own dedicated SQLite database file containing: - input logs, processor logs, and all output queues for that specific - processor. - -This separation provides several benefits: processor databases can be -independently backed up, rotated, or archived; a high-volume processor -won't cause lock contention or bloat affecting the main application; and -individual processor data can be cleanly deleted when a processor is -removed. - -### Package Layout - -All application code lives under `internal/` to prevent external imports. -The main entry point is `cmd/webhooker/main.go`. - -- `internal/config` — Configuration management via `pkg/config` (Viper-based) -- `internal/database` — GORM database connection, migrations, and models -- `internal/globals` — Global application metadata (version, build info) -- `internal/handlers` — HTTP handlers using the closure pattern -- `internal/healthcheck` — Health check endpoint logic -- `internal/logger` — Structured logging setup (`log/slog`) -- `internal/middleware` — HTTP middleware (auth, CORS, logging, metrics) -- `internal/server` — HTTP server setup and routing -- `internal/session` — Session management (gorilla/sessions) -- `pkg/config` — Reusable multi-environment configuration library -- `static/` — Embedded static assets (CSS, JS) -- `templates/` — Go HTML templates +Throughout this document, the target names are used. The code rename is +tracked separately. ### Data Model -- **Users** — Service users with username/password (Argon2id hashing) -- **Processors** — Webhook processing units, many-to-one with users -- **Webhooks** — Inbound URL endpoints feeding into processors -- **Targets** — Delivery destinations per processor (HTTP, retry, database, log) -- **Events** — Captured webhook payloads -- **Deliveries** — Pairing of events with targets -- **Delivery Results** — Outcome of each delivery attempt -- **API Keys** — Programmatic access credentials per user +webhooker's data model has eight entities organized into two tiers: the +**application tier** (user and webhook configuration) and the **event +tier** (event ingestion, delivery, and logging). + +``` +┌─────────────────────────────────────────────────────────────┐ +│ APPLICATION TIER │ +│ (main application database) │ +│ │ +│ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │ +│ │ User │──1:N──│ Webhook │──1:N──│ Entrypoint │ │ +│ │ │ │ │ │ │ │ +│ │ │ │ │──1:N──│ Target │ │ +│ │ │ └──────────┘ └──────────────┘ │ +│ │ │──1:N──│ APIKey │ │ +│ └──────────┘ └──────────┘ │ +└─────────────────────────────────────────────────────────────┘ + +┌─────────────────────────────────────────────────────────────┐ +│ EVENT TIER │ +│ (planned: per-webhook dedicated database) │ +│ │ +│ ┌──────────┐ ┌──────────┐ ┌─────────────────┐ │ +│ │ Event │──1:N──│ Delivery │──1:N──│ DeliveryResult │ │ +│ └──────────┘ └──────────┘ └─────────────────┘ │ +└─────────────────────────────────────────────────────────────┘ +``` + +#### User + +A registered user of the webhooker service. + +| Field | Type | Description | +| ---------- | -------- | ----------- | +| `id` | UUID | Primary key | +| `username` | string | Unique login name | +| `password` | string | Argon2id hash (never exposed via API) | + +**Relations:** Has many Webhooks. Has many APIKeys. + +Passwords are hashed with Argon2id using secure defaults (64 MB memory, +1 iteration, 4 threads, 32-byte key, 16-byte salt). On first startup, +an `admin` user is created with a randomly generated 16-character +password logged to stdout. + +#### Webhook + +The top-level configuration entity (currently called "Processor" in +code). A webhook groups together one or more entrypoints (receiver URLs) +and one or more targets (delivery destinations) into a logical unit. A +user creates a webhook to set up event routing. + +| Field | Type | Description | +| ---------------- | ------- | ----------- | +| `id` | UUID | Primary key | +| `user_id` | UUID | Foreign key → User | +| `name` | string | Human-readable name | +| `description` | string | Optional description | +| `retention_days` | integer | Days to retain events (default: 30) | + +**Relations:** Belongs to User. Has many Entrypoints. Has many Targets. + +The `retention_days` field controls how long event data is kept in the +webhook's dedicated database before automatic cleanup. + +#### Entrypoint + +A receiver URL where external services POST webhook events (currently +called "Webhook" in code). Each entrypoint has a unique UUID-based path. +When an HTTP request arrives at an entrypoint's path, webhooker captures +the full request and creates an Event. + +| Field | Type | Description | +| -------------- | ------- | ----------- | +| `id` | UUID | Primary key | +| `processor_id` | UUID | Foreign key → Webhook | +| `path` | string | Unique URL path (UUID-based, e.g. `/webhook/{uuid}`) | +| `description` | string | Optional description | +| `active` | boolean | Whether this entrypoint accepts events (default: true) | + +**Relations:** Belongs to Webhook. + +A webhook can have multiple entrypoints. This allows separate URLs for +different event sources that all feed into the same processing pipeline +(e.g., one entrypoint for GitHub, another for Stripe, both routing to +the same targets). + +#### Target + +A delivery destination for events. Each target defines where and how +events should be forwarded. + +| Field | Type | Description | +| ---------------- | ---------- | ----------- | +| `id` | UUID | Primary key | +| `processor_id` | UUID | Foreign key → Webhook | +| `name` | string | Human-readable name | +| `type` | TargetType | One of: `http`, `retry`, `database`, `log` | +| `active` | boolean | Whether deliveries are enabled (default: true) | +| `config` | JSON text | Type-specific configuration | +| `max_retries` | integer | Maximum retry attempts (for retry targets) | +| `max_queue_size` | integer | Maximum queued deliveries (for retry targets) | + +**Relations:** Belongs to Webhook. Has many Deliveries. + +**Target types:** + +- **`http`** — Forward the event as an HTTP POST to a configured URL. + Fire-and-forget: a single attempt with no retries. +- **`retry`** — Forward the event via HTTP POST with automatic retry on + failure. Uses exponential backoff up to `max_retries` attempts. +- **`database`** — Store the event in the webhook's database only (no + external delivery). Useful for pure logging/archival. +- **`log`** — Write the event to the application log (stdout). Useful + for debugging. + +The `config` field stores type-specific configuration as JSON (e.g., +destination URL, custom headers, timeout settings). + +#### APIKey + +A programmatic access credential for API authentication. + +| Field | Type | Description | +| -------------- | --------- | ----------- | +| `id` | UUID | Primary key | +| `user_id` | UUID | Foreign key → User | +| `key` | string | Unique API key value | +| `description` | string | Optional description | +| `last_used_at` | timestamp | Last time this key was used (nullable) | + +**Relations:** Belongs to User. + +#### Event + +A captured incoming webhook request. Stores the complete HTTP request +data for replay and auditing. + +| Field | Type | Description | +| -------------- | ------ | ----------- | +| `id` | UUID | Primary key | +| `processor_id` | UUID | Foreign key → Webhook | +| `webhook_id` | UUID | Foreign key → Entrypoint | +| `method` | string | HTTP method (POST, PUT, etc.) | +| `headers` | JSON | Complete request headers | +| `body` | text | Raw request body | +| `content_type` | string | Content-Type header value | + +**Relations:** Belongs to Webhook. Belongs to Entrypoint. Has many +Deliveries. + +When a request arrives at an entrypoint, the full request (method, +headers, body) is captured as an Event. The event is then queued for +delivery to every active target configured on the parent webhook. + +#### Delivery + +The pairing of an event with a target. Tracks the overall delivery +status across potentially multiple attempts. + +| Field | Type | Description | +| ---------- | -------------- | ----------- | +| `id` | UUID | Primary key | +| `event_id` | UUID | Foreign key → Event | +| `target_id`| UUID | Foreign key → Target | +| `status` | DeliveryStatus | One of: `pending`, `delivered`, `failed`, `retrying` | + +**Relations:** Belongs to Event. Belongs to Target. Has many +DeliveryResults. + +**Delivery statuses:** + +- **`pending`** — Created but not yet attempted. +- **`retrying`** — At least one attempt failed; more attempts remain. +- **`delivered`** — Successfully delivered (at least one attempt + succeeded). +- **`failed`** — All retry attempts exhausted without success. + +#### DeliveryResult + +The result of a single delivery attempt. Every attempt (including +retries) is individually logged for full observability. + +| Field | Type | Description | +| --------------- | ------- | ----------- | +| `id` | UUID | Primary key | +| `delivery_id` | UUID | Foreign key → Delivery | +| `attempt_num` | integer | Attempt number (1-based) | +| `success` | boolean | Whether this attempt succeeded | +| `status_code` | integer | HTTP response status code (if applicable) | +| `response_body` | text | Response body (if applicable) | +| `error` | string | Error message (on failure) | +| `duration` | integer | Request duration in milliseconds | + +**Relations:** Belongs to Delivery. + +#### Common Fields + +All entities include these fields from `BaseModel`: + +| Field | Type | Description | +| ------------ | --------- | ----------- | +| `id` | UUID | Auto-generated UUIDv4 primary key | +| `created_at` | timestamp | Record creation time | +| `updated_at` | timestamp | Last modification time | +| `deleted_at` | timestamp | Soft-delete timestamp (nullable; GORM soft deletes) | + +### Database Architecture + +#### Current Implementation + +webhooker currently uses a **single SQLite database** for all data — +application configuration, user accounts, and (once implemented) event +storage. The database connection is managed by GORM with a single +connection string configured via `DBURL`. On first startup the database +is auto-migrated and an `admin` user is created. + +#### Planned: Per-Webhook Event Databases (Phase 2) + +In a future phase (see TODO Phase 2 below), webhooker will split into +**separate SQLite database files**: a main application database for +configuration data and per-webhook databases for event storage. + +**Main Application Database** — will store: + +- **Users** — accounts and Argon2id password hashes +- **Webhooks** (Processors) — webhook configurations +- **Entrypoints** (Webhooks) — receiver URL definitions +- **Targets** — delivery destination configurations +- **APIKeys** — programmatic access credentials + +**Per-Webhook Event Databases** — each webhook will get its own +dedicated SQLite file containing: + +- **Events** — captured incoming webhook payloads +- **Deliveries** — event-to-target pairings and their status +- **DeliveryResults** — individual delivery attempt logs + +This planned separation will provide: + +- **Isolation** — a high-volume webhook won't cause lock contention or + WAL bloat affecting the main application or other webhooks. +- **Independent lifecycle** — event databases can be independently + backed up, archived, rotated, or size-limited without impacting the + application. +- **Clean deletion** — removing a webhook and all its history is as + simple as deleting one file. +- **Per-webhook retention** — the `retention_days` field on each webhook + will control automatic cleanup of old events in that webhook's + database only. +- **Performance** — each webhook's database will have its own WAL, its + own page cache, and its own lock, so concurrent event ingestion across + webhooks won't contend. + +The database uses the +[modernc.org/sqlite](https://pkg.go.dev/modernc.org/sqlite) driver at +runtime, though CGO is required at build time due to the transitive +`mattn/go-sqlite3` dependency from `gorm.io/driver/sqlite`. + +### Request Flow + +``` +External Service + │ + │ POST /webhook/{uuid} + ▼ +┌─────────────┐ ┌──────────────┐ ┌──────────────┐ +│ chi Router │────►│ Middleware │────►│ Webhook │ +│ │ │ Stack │ │ Handler │ +└─────────────┘ └──────────────┘ └──────┬───────┘ + │ + 1. Look up Entrypoint by UUID + 2. Capture full request as Event + 3. Queue Delivery to each active Target + │ + ▼ + ┌──────────────┐ + │ Delivery │ + │ Engine │ + └──────┬───────┘ + │ + ┌────────────────────┼────────────────────┐ + ▼ ▼ ▼ + ┌────────────┐ ┌────────────┐ ┌────────────┐ + │ HTTP Target│ │Retry Target│ │ Log Target │ + │ (1 attempt)│ │ (backoff) │ │ (stdout) │ + └────────────┘ └────────────┘ └────────────┘ +``` + +### Rate Limiting + +Global rate limiting middleware (e.g., per-IP throttling applied at the +router level) **must not** apply to webhook receiver endpoints. Webhook +endpoints receive automated traffic from external services at +unpredictable rates, and blanket rate limits would cause legitimate +deliveries to be dropped. + +Instead, each webhook has its own individually configurable rate limit, +applied within the webhook handler itself. By default, no rate limit is +applied — webhook endpoints accept traffic as fast as it arrives. Rate +limits can be configured per-webhook when needed (e.g., to protect +against a misbehaving sender). ### API Endpoints -- `GET /` — Web UI index page -- `GET /.well-known/healthcheck` — Health check with uptime, version -- `GET /s/*` — Static file serving (CSS, JS) -- `GET /metrics` — Prometheus metrics (requires basic auth) -- `POST /webhook/{uuid}` — Webhook receiver endpoint -- `/pages/login`, `/pages/logout` — Authentication -- `/user/{username}` — User profile -- `/sources/*` — Webhook source management +#### Public Endpoints + +| Method | Path | Description | +| ------ | --------------------------- | ----------- | +| `GET` | `/` | Web UI index page (server-rendered) | +| `GET` | `/.well-known/healthcheck` | Health check (JSON: status, uptime, version) | +| `GET` | `/s/*` | Static file serving (embedded CSS, JS) | +| `ANY` | `/webhook/{uuid}` | Webhook receiver endpoint (accepts all methods) | + +#### Authentication Endpoints + +| Method | Path | Description | +| ------ | --------------- | ----------- | +| `GET` | `/pages/login` | Login page | +| `POST` | `/pages/login` | Login form submission | +| `POST` | `/pages/logout` | Logout (destroys session) | + +#### Authenticated Endpoints + +| Method | Path | Description | +| ------ | ------------------------ | ----------- | +| `GET` | `/user/{username}` | User profile page | +| `GET` | `/sources` | List user's webhooks | +| `GET` | `/sources/new` | Create webhook form | +| `POST` | `/sources/new` | Create webhook submission | +| `GET` | `/source/{id}` | Webhook detail view | +| `GET` | `/source/{id}/edit` | Edit webhook form | +| `POST` | `/source/{id}/edit` | Edit webhook submission | +| `POST` | `/source/{id}/delete` | Delete webhook | +| `GET` | `/source/{id}/logs` | Webhook event logs | + +#### Infrastructure Endpoints + +| Method | Path | Description | +| ------ | ---------- | ----------- | +| `GET` | `/metrics` | Prometheus metrics (requires basic auth) | + +#### API (Planned) + +| Method | Path | Description | +| -------- | ------------------------------ | ----------- | +| `GET` | `/api/v1/webhooks` | List webhooks | +| `POST` | `/api/v1/webhooks` | Create webhook | +| `GET` | `/api/v1/webhooks/{id}` | Get webhook details | +| `PUT` | `/api/v1/webhooks/{id}` | Update webhook | +| `DELETE` | `/api/v1/webhooks/{id}` | Delete webhook | +| `GET` | `/api/v1/webhooks/{id}/events` | List events for webhook | +| `POST` | `/api/v1/events/{id}/redeliver`| Redeliver an event | + +API authentication will use API keys passed via `Authorization: Bearer +` header. + +### Package Layout + +All application code lives under `internal/` to prevent external +imports. The entry point is `cmd/webhooker/main.go`. + +``` +webhooker/ +├── cmd/webhooker/ +│ └── main.go # Entry point: sets globals, wires fx +├── internal/ +│ ├── config/ +│ │ └── config.go # Configuration loading via pkg/config +│ ├── database/ +│ │ ├── base_model.go # BaseModel with UUID primary keys +│ │ ├── database.go # GORM connection, migrations, admin seed +│ │ ├── models.go # AutoMigrate for all models +│ │ ├── model_user.go # User entity +│ │ ├── model_processor.go # Webhook entity (to be renamed) +│ │ ├── model_webhook.go # Entrypoint entity (to be renamed) +│ │ ├── model_target.go # Target entity and TargetType enum +│ │ ├── model_event.go # Event entity +│ │ ├── model_delivery.go # Delivery entity and DeliveryStatus enum +│ │ ├── model_delivery_result.go # DeliveryResult entity +│ │ ├── model_apikey.go # APIKey entity +│ │ └── password.go # Argon2id hashing and verification +│ ├── globals/ +│ │ └── globals.go # Build-time variables (appname, version, arch) +│ ├── handlers/ +│ │ ├── handlers.go # Base handler struct, JSON helpers, template rendering +│ │ ├── auth.go # Login, logout handlers +│ │ ├── healthcheck.go # Health check handler +│ │ ├── index.go # Index page handler +│ │ ├── profile.go # User profile handler +│ │ ├── source_management.go # Webhook CRUD handlers (stubs) +│ │ └── webhook.go # Webhook receiver handler +│ ├── healthcheck/ +│ │ └── healthcheck.go # Health check service (uptime, version) +│ ├── logger/ +│ │ └── logger.go # slog setup with TTY detection +│ ├── middleware/ +│ │ └── middleware.go # Logging, CORS, Auth, Metrics, MetricsAuth +│ ├── server/ +│ │ ├── server.go # Server struct, fx lifecycle, signal handling +│ │ ├── http.go # HTTP server setup with timeouts +│ │ └── routes.go # All route definitions +│ └── session/ +│ └── session.go # Cookie-based session management +├── pkg/config/ # Reusable multi-environment config library +├── static/ +│ ├── static.go # //go:embed directive +│ ├── css/style.css # Custom stylesheet (system font stack, card effects, layout) +│ └── js/app.js # Client-side JavaScript (minimal bootstrap) +├── templates/ # Go HTML templates (base, index, login, etc.) +├── configs/ +│ └── config.yaml.example # Example configuration file +├── Dockerfile # Multi-stage: build + check, then Alpine runtime +├── Makefile # fmt, lint, test, check, build, docker targets +├── go.mod / go.sum +└── .golangci.yml # Linter configuration +``` + +### Dependency Injection + +Components are wired via Uber fx in this order: + +1. `globals.New` — Build-time variables (appname, version, arch) +2. `logger.New` — Structured logging (slog with TTY detection) +3. `config.New` — Configuration loading (pkg/config + environment) +4. `database.New` — SQLite connection, migrations, admin user seed +5. `healthcheck.New` — Health check service +6. `session.New` — Cookie-based session manager +7. `handlers.New` — HTTP handlers +8. `middleware.New` — HTTP middleware +9. `server.New` — HTTP server and router + +The server starts via `fx.Invoke(func(*server.Server) {})` which +triggers the fx lifecycle hooks in dependency order. + +### Middleware Stack + +Applied to all routes in this order: + +1. **Recoverer** — Panic recovery (chi built-in) +2. **RequestID** — Generate unique request IDs (chi built-in) +3. **Logging** — Structured request logging (method, URL, status, + latency, remote IP, user agent, request ID) +4. **Metrics** — Prometheus HTTP metrics (if `METRICS_USERNAME` is set) +5. **CORS** — Cross-origin resource sharing headers +6. **Timeout** — 60-second request timeout +7. **Sentry** — Error reporting to Sentry (if `SENTRY_DSN` is set; + configured with `Repanic: true` so panics still reach Recoverer) + +### Authentication + +- **Web UI:** Cookie-based sessions using gorilla/sessions with + encrypted cookies. Sessions are configured with HttpOnly, SameSite + Lax, and Secure (in production). Session lifetime is 7 days. +- **API (planned):** API key authentication via `Authorization: Bearer` + header. API keys are stored per-user with usage tracking + (`last_used_at`). +- **Metrics:** Basic authentication protecting the `/metrics` endpoint. + +### Security + +- Passwords hashed with Argon2id (64 MB memory cost) +- Session cookies are HttpOnly, SameSite Lax, Secure (prod only) +- Session key must be a 32-byte base64-encoded value +- Prometheus metrics behind basic auth +- Static assets embedded in binary (no filesystem access needed at + runtime) +- Container runs as non-root user (UID 1000) +- GORM soft deletes on all entities (data preserved for audit) + +### Docker + +The Dockerfile uses a multi-stage build: + +1. **Builder stage** (Debian-based `golang:1.24`) — installs + golangci-lint, downloads dependencies, copies source, runs `make + check` (format verification, linting, tests, compilation). +2. **Runtime stage** (`alpine:3.21`) — copies the binary, runs as + non-root user, exposes port 8080, includes a health check. + +The builder uses Debian rather than Alpine because GORM's SQLite +dialect pulls in CGO-dependent headers at compile time. The runtime +binary is statically linked and runs on Alpine. + +`docker build .` is the CI gate — if it passes, the code is formatted, +linted, tested, and compiled. ## TODO -### Phase 1: Security & Infrastructure -- [ ] Security headers (HSTS, CSP, X-Frame-Options) -- [ ] Rate limiting middleware -- [ ] CSRF protection for forms -- [ ] Request ID tracking through entire lifecycle - -### Phase 2: Authentication & Authorization -- [ ] Authentication middleware for protected routes -- [ ] Session expiration and "remember me" -- [ ] Password reset flow -- [ ] API key authentication for programmatic access - -### Phase 3: Core Webhook Features -- [ ] Webhook reception and event storage at `/webhook/{uuid}` -- [ ] Event processing and target delivery engine -- [ ] HTTP target type (fire-and-forget POST) -- [ ] Retry target type (exponential backoff) -- [ ] Database target type (store only) -- [ ] Log target type (console output) +### Phase 1: Core Webhook Engine +- [ ] Implement webhook reception and event storage at `/webhook/{uuid}` +- [ ] Build event processing and target delivery engine +- [ ] Implement HTTP target type (fire-and-forget POST) +- [ ] Implement retry target type (exponential backoff) +- [ ] Implement database target type (store only) +- [ ] Implement log target type (console output) +- [ ] Per-webhook rate limiting in the receiver handler - [ ] Webhook signature verification (GitHub, Stripe formats) +### Phase 2: Database Separation +- [ ] Split into main application DB + per-webhook event DBs +- [ ] Automatic event retention cleanup based on `retention_days` +- [ ] Per-webhook database lifecycle management (create on webhook + creation, delete on webhook removal) + +### Phase 3: Security & Infrastructure +- [ ] Implement authentication middleware for protected routes + ([#9](https://git.eeqj.de/sneak/webhooker/issues/9)) +- [ ] Security headers (HSTS, CSP, X-Frame-Options) +- [ ] CSRF protection for forms +- [ ] Session expiration and "remember me" +- [ ] Password change/reset flow +- [ ] API key authentication for programmatic access + ### Phase 4: Web UI -- [ ] Webhook source management pages (list, create, edit, delete) +- [ ] Webhook management pages (list, create, edit, delete) - [ ] Webhook request log viewer with filtering - [ ] Delivery status and retry management UI - [ ] Manual event redelivery - [ ] Analytics dashboard (success rates, response times) +- [ ] Replace Bootstrap with Tailwind CSS + Alpine.js + ([#4](https://git.eeqj.de/sneak/webhooker/issues/4)) -### Phase 5: API -- [ ] RESTful CRUD for processors, webhooks, targets +### Phase 5: REST API +- [ ] RESTful CRUD for webhooks, entrypoints, targets - [ ] Event viewing and filtering endpoints -- [ ] API documentation (OpenAPI) +- [ ] Event redelivery endpoint +- [ ] OpenAPI specification + +### Phase 6: Code Quality +- [ ] Rename Processor → Webhook, Webhook → Entrypoint in code + ([#12](https://git.eeqj.de/sneak/webhooker/issues/12)) +- [ ] Embed templates via `//go:embed` + ([#7](https://git.eeqj.de/sneak/webhooker/issues/7)) +- [ ] Use `slog.LevelVar` for dynamic log level switching + ([#8](https://git.eeqj.de/sneak/webhooker/issues/8)) +- [ ] Simplify configuration to prefer environment variables + ([#10](https://git.eeqj.de/sneak/webhooker/issues/10)) +- [ ] Remove redundant `godotenv/autoload` import + ([#11](https://git.eeqj.de/sneak/webhooker/issues/11)) ### Future -- [ ] Email source and delivery target types +- [ ] Email delivery target type - [ ] SNS, S3, Slack delivery targets -- [ ] Data transformations (e.g. webhook-to-Slack message) +- [ ] Data transformations (e.g., webhook-to-Slack message formatting) - [ ] JSONL file delivery with periodic S3 upload +- [ ] Webhook event search and filtering +- [ ] Multi-user with role-based access ## License -Pending — to be determined by the author (MIT, GPL, or WTFPL). +MIT ## Author