sneak/webhooker
1
0
Fork 0
Code Issues 4 Pull Requests 1 Actions Packages Projects Releases Wiki Activity
Files
a51e86301725bd94edb2face0862bce756b5e3be
webhooker/README.md
clawbot 8e00e40008
All checks were successful
check / check (push) Successful in 5s
Details
docs: fix stale references to development mode and retry target type 
- README.md: remove 'in development mode' from admin user creation
  description (admin user creation is unconditional)
- internal/delivery/engine.go: remove 'and retry' from HTTPTargetConfig
  comment (retry was merged into http target type)
- internal/delivery/engine_test.go: remove '/retry' from
  newHTTPTargetConfig comment for consistency
2026-03-03 16:12:43 -08:00

910 lines
40 KiB
Markdown
Raw Blame History

# webhooker
webhooker is a self-hosted webhook proxy and store-and-forward service
written in [Go](https://golang.org) by
[@sneak](https://sneak.berlin). It receives webhooks from external
services, durably stores them, and delivers them to configured targets
with retry support, logging, and observability. Category: infrastructure
/ web service. License: MIT.
## Getting Started
### Prerequisites
- Go 1.24+
- golangci-lint v1.64+
- Docker (for containerized deployment)
### Quick Start
```bash
# Clone the repo
git clone https://git.eeqj.de/sneak/webhooker.git
cd webhooker
# Install Go dependencies
make deps
# Run all checks (format, lint, test, build)
make check
# Run in development mode (uses SQLite in current directory)
make dev
# Build Docker image
make docker
```
### Development Commands
```bash
make fmt # Format code (gofmt + goimports)
make lint # Run golangci-lint
make test # Run tests with race detection
make check # fmt-check + lint + test + build (CI gate)
make build # Build binary to bin/webhooker
make dev # go run ./cmd/webhooker
make docker # Build Docker image
make hooks # Install git pre-commit hook that runs make check
```
### Configuration
All configuration is via environment variables. For local development,
you can place variables in a `.env` file in the project root (loaded
automatically via `godotenv/autoload`).
The environment is selected by setting `WEBHOOKER_ENVIRONMENT` to `dev`
or `prod` (default: `dev`).
| Variable | Description | Default |
| ----------------------- | ----------------------------------- | -------- |
| `WEBHOOKER_ENVIRONMENT` | `dev` or `prod` | `dev` |
| `PORT` | HTTP listen port | `8080` |
| `DATA_DIR` | Directory for all SQLite databases | `./data` (dev) / `/data` (prod) |
| `DEBUG` | Enable debug logging | `false` |
| `METRICS_USERNAME` | Basic auth username for `/metrics` | `""` |
| `METRICS_PASSWORD` | Basic auth password for `/metrics` | `""` |
| `SENTRY_DSN` | Sentry error reporting DSN | `""` |
On first startup, webhooker automatically generates a cryptographically
secure session encryption key and stores it in the database. This key
persists across restarts — no manual key management is needed.
On first startup, webhooker creates an `admin` user
with a randomly generated password and logs it to stdout. This password
is only displayed once.
### Running with Docker
```bash
docker run -d \
-p 8080:8080 \
-v /path/to/data:/data \
-e WEBHOOKER_ENVIRONMENT=prod \
webhooker:latest
```
The container runs as a non-root user (`webhooker`, UID 1000), exposes
port 8080, and includes a health check against
`/.well-known/healthcheck`. The `/data` volume holds all SQLite
databases: the main application database (`webhooker.db`) and the
per-webhook event databases (`events-{uuid}.db`). Mount this as a
persistent volume to preserve data across container restarts.
## Rationale
Webhook integrations between services are inherently fragile. The
receiving service must be online when the webhook fires, most webhook
senders provide no built-in retry mechanism, and there is no standard
way to inspect what was sent, when it was sent, or whether delivery
succeeded.
webhooker solves this by acting as a durable intermediary:
1. **Reliable ingestion** — webhooker is always ready to accept incoming
webhooks. It stores every received event before attempting any
delivery, so nothing is lost if downstream targets are unavailable.
2. **Guaranteed delivery** — Events are queued for delivery to each
configured target. Failed deliveries are retried with configurable
backoff. Every delivery attempt is logged with status codes, response
bodies, and timing.
3. **Observability** — Full request/response logging for every webhook
received and every delivery attempted. Prometheus metrics expose
volume, latency, and error rates. The web UI provides real-time
visibility into event flow.
4. **Fan-out** — A single incoming webhook can be delivered to multiple
targets simultaneously. This enables patterns like forwarding a
GitHub webhook to both a deployment service and a Slack channel.
5. **Replay** — Stored events can be manually redelivered for debugging
or testing, without requiring the original sender to fire the webhook
again.
### Use Cases
- **Store-and-forward** with configurable retries for unreliable
receivers
- **Observability** via Prometheus metrics on webhook frequency, payload
size, and delivery performance
- **Debugging** and introspection of webhook payloads in the web UI
- **Replay** of webhook events for application testing and development
- **Fan-out** delivery of a single webhook to multiple downstream
targets
- **High-availability ingestion** for delivery to less reliable backend
systems
## Design
### Architecture Overview
webhooker is structured as a standard Go HTTP server following the
[sneak/prompts GO_HTTP_SERVER_CONVENTIONS](https://git.eeqj.de/sneak/prompts/src/branch/main/prompts/GO_HTTP_SERVER_CONVENTIONS.md).
It uses:
- **[Uber fx](https://go.uber.org/fx)** for dependency injection and
lifecycle management
- **[go-chi](https://github.com/go-chi/chi)** for HTTP routing
- **[GORM](https://gorm.io)** for database access with
**[modernc.org/sqlite](https://pkg.go.dev/modernc.org/sqlite)** as
the runtime SQLite driver. Note: `gorm.io/driver/sqlite` transitively
depends on `mattn/go-sqlite3`, which requires CGO at build time (see
[Docker](#docker) section)
- **[slog](https://pkg.go.dev/log/slog)** (stdlib) for structured
logging with TTY detection (text for dev, JSON for prod)
- **[gorilla/sessions](https://github.com/gorilla/sessions)** for
encrypted cookie-based session management
- **[Prometheus](https://prometheus.io)** for metrics, served at
`/metrics` behind basic auth
- **[Sentry](https://sentry.io)** for optional error reporting
### Naming Conventions
The codebase uses consistent naming throughout (rename completed in
[issue #12](https://git.eeqj.de/sneak/webhooker/issues/12)):
| Entity | Description |
| ---------------- | ----------- |
| **Webhook** | Top-level configuration entity grouping entrypoints and targets |
| **Entrypoint** | A receiver URL where external services POST events |
| **Target** | A delivery destination for events |
### Data Model
webhooker's data model has eight entities organized into two tiers: the
**application tier** (user and webhook configuration) and the **event
tier** (event ingestion, delivery, and logging).
```
┌─────────────────────────────────────────────────────────────┐
│ APPLICATION TIER │
│ (main application database) │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │
│ │ User │──1:N──│ Webhook │──1:N──│ Entrypoint │ │
│ │ │ │ │ │ │ │
│ │ │ │ │──1:N──│ Target │ │
│ │ │ └──────────┘ └──────────────┘ │
│ │ │──1:N──│ APIKey │ │
│ └──────────┘ └──────────┘ │
│ │
│ ┌──────────┐ │
│ │ Setting │ (key-value application config) │
│ └──────────┘ │
└─────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ EVENT TIER │
│ (per-webhook dedicated databases) │
│ │
│ ┌──────────┐ ┌──────────┐ ┌─────────────────┐ │
│ │ Event │──1:N──│ Delivery │──1:N──│ DeliveryResult │ │
│ └──────────┘ └──────────┘ └─────────────────┘ │
└─────────────────────────────────────────────────────────────┘
```
#### Setting
A key-value pair for application-level configuration that is
auto-managed rather than user-provided. Used to store the session
encryption key and any future auto-generated settings.
| Field | Type | Description |
| ------- | ------ | ----------- |
| `key` | string | Primary key (setting name) |
| `value` | text | Setting value |
Currently stored settings:
- **`session_key`** — Base64-encoded 32-byte session encryption key,
auto-generated on first startup.
#### User
A registered user of the webhooker service.
| Field | Type | Description |
| ---------- | -------- | ----------- |
| `id` | UUID | Primary key |
| `username` | string | Unique login name |
| `password` | string | Argon2id hash (never exposed via API) |
**Relations:** Has many Webhooks. Has many APIKeys.
Passwords are hashed with Argon2id using secure defaults (64 MB memory,
1 iteration, 4 threads, 32-byte key, 16-byte salt). On first startup,
an `admin` user is created with a randomly generated 16-character
password logged to stdout.
#### Webhook
The top-level configuration entity. A webhook groups together one or
more entrypoints (receiver URLs) and one or more targets (delivery
destinations) into a logical unit. A user creates a webhook to set up
event routing.
| Field | Type | Description |
| ---------------- | ------- | ----------- |
| `id` | UUID | Primary key |
| `user_id` | UUID | Foreign key → User |
| `name` | string | Human-readable name |
| `description` | string | Optional description |
| `retention_days` | integer | Days to retain events (default: 30) |
**Relations:** Belongs to User. Has many Entrypoints. Has many Targets.
The `retention_days` field controls how long event data is kept in the
webhook's dedicated database before automatic cleanup.
#### Entrypoint
A receiver URL where external services POST webhook events. Each
entrypoint has a unique UUID-based path.
When an HTTP request arrives at an entrypoint's path, webhooker captures
the full request and creates an Event.
| Field | Type | Description |
| -------------- | ------- | ----------- |
| `id` | UUID | Primary key |
| `webhook_id` | UUID | Foreign key → Webhook |
| `path` | string | Unique URL path (UUID-based, e.g. `/webhook/{uuid}`) |
| `description` | string | Optional description |
| `active` | boolean | Whether this entrypoint accepts events (default: true) |
**Relations:** Belongs to Webhook.
A webhook can have multiple entrypoints. This allows separate URLs for
different event sources that all feed into the same processing pipeline
(e.g., one entrypoint for GitHub, another for Stripe, both routing to
the same targets).
#### Target
A delivery destination for events. Each target defines where and how
events should be forwarded.
| Field | Type | Description |
| ---------------- | ---------- | ----------- |
| `id` | UUID | Primary key |
| `webhook_id` | UUID | Foreign key → Webhook |
| `name` | string | Human-readable name |
| `type` | TargetType | One of: `http`, `database`, `log` |
| `active` | boolean | Whether deliveries are enabled (default: true) |
| `config` | JSON text | Type-specific configuration |
| `max_retries` | integer | Maximum retry attempts for HTTP targets (0 = fire-and-forget, >0 = retries with backoff) |
| `max_queue_size` | integer | Maximum queued deliveries (for HTTP targets with retries) |
**Relations:** Belongs to Webhook. Has many Deliveries.
**Target types:**
- **`http`** — Forward the event as an HTTP POST to a configured URL.
Behavior depends on `max_retries`: when `max_retries` is 0 (the
default), the target operates in fire-and-forget mode — a single
attempt with no retries and no circuit breaker. When `max_retries` is
greater than 0, failed deliveries are retried with exponential backoff
up to `max_retries` attempts, protected by a per-target circuit
breaker.
- **`database`** — Confirm the event is stored in the webhook's
per-webhook database (no external delivery). Since events are always
written to the per-webhook DB on ingestion, this target marks delivery
as immediately successful. Useful for ensuring durable event archival.
- **`log`** — Write the event to the application log (stdout). Useful
for debugging.
The `config` field stores type-specific configuration as JSON (e.g.,
destination URL, custom headers, timeout settings).
#### APIKey
A programmatic access credential for API authentication.
| Field | Type | Description |
| -------------- | --------- | ----------- |
| `id` | UUID | Primary key |
| `user_id` | UUID | Foreign key → User |
| `key` | string | Unique API key value |
| `description` | string | Optional description |
| `last_used_at` | timestamp | Last time this key was used (nullable) |
**Relations:** Belongs to User.
#### Event
A captured incoming webhook request. Stores the complete HTTP request
data for replay and auditing.
| Field | Type | Description |
| -------------- | ------ | ----------- |
| `id` | UUID | Primary key |
| `webhook_id` | UUID | Foreign key → Webhook |
| `entrypoint_id` | UUID | Foreign key → Entrypoint |
| `method` | string | HTTP method (POST, PUT, etc.) |
| `headers` | JSON | Complete request headers |
| `body` | text | Raw request body |
| `content_type` | string | Content-Type header value |
**Relations:** Belongs to Webhook. Belongs to Entrypoint. Has many
Deliveries.
When a request arrives at an entrypoint, the full request (method,
headers, body) is captured as an Event. The event is then queued for
delivery to every active target configured on the parent webhook.
#### Delivery
The pairing of an event with a target. Tracks the overall delivery
status across potentially multiple attempts.
| Field | Type | Description |
| ---------- | -------------- | ----------- |
| `id` | UUID | Primary key |
| `event_id` | UUID | Foreign key → Event |
| `target_id`| UUID | Foreign key → Target |
| `status` | DeliveryStatus | One of: `pending`, `delivered`, `failed`, `retrying` |
**Relations:** Belongs to Event. Belongs to Target. Has many
DeliveryResults.
**Delivery statuses:**
- **`pending`** — Created but not yet attempted.
- **`retrying`** — At least one attempt failed; more attempts remain.
- **`delivered`** — Successfully delivered (at least one attempt
succeeded).
- **`failed`** — All retry attempts exhausted without success.
#### DeliveryResult
The result of a single delivery attempt. Every attempt (including
retries) is individually logged for full observability.
| Field | Type | Description |
| --------------- | ------- | ----------- |
| `id` | UUID | Primary key |
| `delivery_id` | UUID | Foreign key → Delivery |
| `attempt_num` | integer | Attempt number (1-based) |
| `success` | boolean | Whether this attempt succeeded |
| `status_code` | integer | HTTP response status code (if applicable) |
| `response_body` | text | Response body (if applicable) |
| `error` | string | Error message (on failure) |
| `duration` | integer | Request duration in milliseconds |
**Relations:** Belongs to Delivery.
#### Common Fields
All entities include these fields from `BaseModel`:
| Field | Type | Description |
| ------------ | --------- | ----------- |
| `id` | UUID | Auto-generated UUIDv4 primary key |
| `created_at` | timestamp | Record creation time |
| `updated_at` | timestamp | Last modification time |
| `deleted_at` | timestamp | Soft-delete timestamp (nullable; GORM soft deletes) |
### Database Architecture
#### Per-Webhook Event Databases
webhooker uses **separate SQLite database files**: a main application
database for configuration data and per-webhook databases for event
storage. All database files live in the `DATA_DIR` directory.
**Main Application Database** (`{DATA_DIR}/webhooker.db`) — stores
configuration and application state:
- **Settings** — auto-managed key-value config (e.g. session encryption
key)
- **Users** — accounts and Argon2id password hashes
- **Webhooks** — webhook configurations
- **Entrypoints** — receiver URL definitions
- **Targets** — delivery destination configurations
- **APIKeys** — programmatic access credentials
On first startup the main database is auto-migrated, a session
encryption key is generated and stored, and an `admin` user is created.
**Per-Webhook Event Databases** (`{DATA_DIR}/events-{webhook_uuid}.db`)
— each webhook gets its own dedicated SQLite file containing:
- **Events** — captured incoming webhook payloads
- **Deliveries** — event-to-target pairings and their status
- **DeliveryResults** — individual delivery attempt logs
Per-webhook databases are created automatically when a webhook is
created (and lazily on first access for webhooks that predate this
feature). They are managed by the `WebhookDBManager` component, which
handles connection pooling, lazy opening, migrations, and cleanup.
This separation provides:
- **Isolation** — a high-volume webhook won't cause lock contention or
WAL bloat affecting the main application or other webhooks.
- **Independent lifecycle** — event databases can be independently
backed up, archived, rotated, or size-limited without impacting the
application.
- **Clean deletion** — removing a webhook and all its history is as
simple as deleting one file. Configuration is soft-deleted in the main
DB; the event database file is hard-deleted (permanently removed).
- **Per-webhook retention** — the `retention_days` field on each webhook
controls automatic cleanup of old events in that webhook's database
only.
- **Performance** — each webhook's database has its own WAL, its own
page cache, and its own lock, so concurrent event ingestion across
webhooks won't contend.
The **database target type** leverages this architecture: since events
are already stored in the per-webhook database by design, the database
target simply marks the delivery as immediately successful. The
per-webhook DB IS the dedicated event database — that's the whole point
of the database target type.
The database uses the
[modernc.org/sqlite](https://pkg.go.dev/modernc.org/sqlite) driver at
runtime, though CGO is required at build time due to the transitive
`mattn/go-sqlite3` dependency from `gorm.io/driver/sqlite`.
### Request Flow
```
External Service
│
│ POST /webhook/{uuid}
▼
┌─────────────┐ ┌──────────────┐ ┌──────────────┐
│ chi Router │────►│ Middleware │────►│ Webhook │
│ │ │ Stack │ │ Handler │
└─────────────┘ └──────────────┘ └──────┬───────┘
│
1. Look up Entrypoint by UUID
2. Capture full request as Event
3. Create Delivery records for each active Target
4. Build self-contained DeliveryTask structs
(target config + event data inline for ≤16KB)
5. Notify Engine via channel (no DB read needed)
│
▼
┌──────────────┐
│ Delivery │◄── retry timers
│ Engine │ (backoff)
│ (worker │
│ pool) │
└──────┬───────┘
│
┌── bounded worker pool (N workers) ──┐
▼ ▼ ▼
┌────────────┐ ┌────────────┐ ┌────────────┐
│ HTTP Target│ │ HTTP Target│ │ Log Target │
│(max_retries│ │(max_retries│ │ (stdout) │
│ == 0) │ │ > 0, │ └────────────┘
│ fire+forget│ │ backoff + │
└────────────┘ │ circuit │
│ breaker) │
└────────────┘
```
### Bounded Worker Pool
The delivery engine uses a **fixed-size worker pool** (default: 10
workers) to process all deliveries. At most N deliveries are in-flight
at any time, preventing goroutine explosions regardless of queue depth.
**Architecture:**
- **Channels as queues:** Two buffered channels serve as bounded queues:
a delivery channel (new tasks from the webhook handler) and a retry
channel (tasks from backoff timers). Both are buffered to 10,000.
- **Fan-out via channel, not goroutines:** When an event arrives with
multiple targets, each `DeliveryTask` is sent to the delivery channel.
Workers pick them up and process them — no goroutine-per-target.
- **Worker goroutines:** A fixed number of worker goroutines select from
both channels. Each worker processes one task at a time, then picks up
the next. Workers are the ONLY goroutines doing actual HTTP delivery.
- **Retry backpressure with DB fallback:** When a retry timer fires and
the retry channel is full, the timer is dropped — the delivery stays
in `retrying` status in the database. A periodic sweep (every 60s)
scans for these "orphaned" retries and re-queues them. No blocked
goroutines, no unbounded timer chains.
- **Bounded concurrency:** At most N deliveries (N = number of workers)
are in-flight simultaneously. Even if a circuit breaker is open for
hours and thousands of retries queue up in the channels, the workers
drain them at a controlled rate when the circuit closes.
This means:
- **No goroutine explosion** — even with 10,000 queued retries, only
N worker goroutines exist.
- **Natural backpressure** — if workers are busy, new tasks wait in the
channel buffer rather than spawning more goroutines.
- **Independent results** — each worker records its own delivery result
in the per-webhook database without coordination.
- **Graceful shutdown** — cancel the context, workers finish their
current task and exit. `WaitGroup.Wait()` ensures clean shutdown.
**Recovery paths:**
1. **Startup recovery:** When the engine starts, it scans all per-webhook
databases for `pending` and `retrying` deliveries. Pending deliveries
are sent to the delivery channel; retrying deliveries get backoff
timers scheduled.
2. **Periodic retry sweep (DB-mediated fallback):** Every 60 seconds the
engine scans for `retrying` deliveries whose backoff period has
elapsed. This catches "orphaned" retries — ones whose in-memory timer
was dropped because the retry channel was full. The database is the
durable fallback that ensures no retry is permanently lost, even under
extreme backpressure.
### Circuit Breaker (HTTP Targets with Retries)
HTTP targets with `max_retries` > 0 are protected by a **per-target circuit breaker** that
prevents hammering a down target with repeated failed delivery attempts.
The circuit breaker is in-memory only and resets on restart (which is
fine — startup recovery rescans the database anyway).
**States:**
| State | Behavior |
| ----------- | -------- |
| **Closed** | Normal operation. Deliveries flow through. Consecutive failures are counted. |
| **Open** | Target appears down. Deliveries are skipped and rescheduled for after the cooldown. |
| **Half-Open** | Cooldown expired. One probe delivery is allowed to test if the target has recovered. |
**Transitions:**
```
success ┌──────────┐
┌────────────────────► │ Closed │ ◄─── probe succeeds
│ │ (normal) │
│ └────┬─────┘
│ │ N consecutive failures
│ ▼
│ ┌──────────┐
│ │ Open │ ◄─── probe fails
│ │(tripped) │
│ └────┬─────┘
│ │ cooldown expires
│ ▼
│ ┌──────────┐
└──────────────────────│Half-Open │
│ (probe) │
└──────────┘
```
**Defaults:**
- **Failure threshold:** 5 consecutive failures before opening
- **Cooldown:** 30 seconds in open state before probing
**Scope:** Circuit breakers only apply to **HTTP targets with
`max_retries` > 0**. Fire-and-forget HTTP targets (`max_retries` == 0),
database targets (local operations), and log targets (stdout) do not use
circuit breakers.
When a circuit is open and a new delivery arrives, the engine marks the
delivery as `retrying` and schedules a retry timer for after the
remaining cooldown period. This ensures no deliveries are lost — they're
just delayed until the target is healthy again.
### Rate Limiting
Global rate limiting middleware (e.g., per-IP throttling applied at the
router level) **must not** apply to webhook receiver endpoints. Webhook
endpoints receive automated traffic from external services at
unpredictable rates, and blanket rate limits would cause legitimate
deliveries to be dropped.
Instead, each webhook has its own individually configurable rate limit,
applied within the webhook handler itself. By default, no rate limit is
applied — webhook endpoints accept traffic as fast as it arrives. Rate
limits can be configured per-webhook when needed (e.g., to protect
against a misbehaving sender).
### API Endpoints
#### Public Endpoints
| Method | Path | Description |
| ------ | --------------------------- | ----------- |
| `GET` | `/` | Web UI index page (server-rendered) |
| `GET` | `/.well-known/healthcheck` | Health check (JSON: status, uptime, version) |
| `GET` | `/s/*` | Static file serving (embedded CSS, JS) |
| `ANY` | `/webhook/{uuid}` | Webhook receiver endpoint (accepts all methods) |
#### Authentication Endpoints
| Method | Path | Description |
| ------ | --------------- | ----------- |
| `GET` | `/pages/login` | Login page |
| `POST` | `/pages/login` | Login form submission |
| `POST` | `/pages/logout` | Logout (destroys session) |
#### Authenticated Endpoints
| Method | Path | Description |
| ------ | ------------------------ | ----------- |
| `GET` | `/user/{username}` | User profile page |
| `GET` | `/sources` | List user's webhooks |
| `GET` | `/sources/new` | Create webhook form |
| `POST` | `/sources/new` | Create webhook submission |
| `GET` | `/source/{id}` | Webhook detail view |
| `GET` | `/source/{id}/edit` | Edit webhook form |
| `POST` | `/source/{id}/edit` | Edit webhook submission |
| `POST` | `/source/{id}/delete` | Delete webhook |
| `GET` | `/source/{id}/logs` | Webhook event logs |
| `POST` | `/source/{id}/entrypoints` | Add entrypoint to webhook |
| `POST` | `/source/{id}/targets` | Add target to webhook |
#### Infrastructure Endpoints
| Method | Path | Description |
| ------ | ---------- | ----------- |
| `GET` | `/metrics` | Prometheus metrics (requires basic auth) |
#### API (Planned)
| Method | Path | Description |
| -------- | ------------------------------ | ----------- |
| `GET` | `/api/v1/webhooks` | List webhooks |
| `POST` | `/api/v1/webhooks` | Create webhook |
| `GET` | `/api/v1/webhooks/{id}` | Get webhook details |
| `PUT` | `/api/v1/webhooks/{id}` | Update webhook |
| `DELETE` | `/api/v1/webhooks/{id}` | Delete webhook |
| `GET` | `/api/v1/webhooks/{id}/events` | List events for webhook |
| `POST` | `/api/v1/events/{id}/redeliver`| Redeliver an event |
API authentication will use API keys passed via `Authorization: Bearer
<key>` header.
### Package Layout
All application code lives under `internal/` to prevent external
imports. The entry point is `cmd/webhooker/main.go`.
```
webhooker/
├── cmd/webhooker/
│ └── main.go # Entry point: sets globals, wires fx
├── internal/
│ ├── config/
│ │ └── config.go # Configuration loading from environment variables
│ ├── database/
│ │ ├── base_model.go # BaseModel with UUID primary keys
│ │ ├── database.go # GORM connection, migrations, admin seed
│ │ ├── models.go # AutoMigrate for config-tier models
│ │ ├── model_setting.go # Setting entity (key-value app config)
│ │ ├── model_user.go # User entity
│ │ ├── model_webhook.go # Webhook entity
│ │ ├── model_entrypoint.go # Entrypoint entity
│ │ ├── model_target.go # Target entity and TargetType enum
│ │ ├── model_event.go # Event entity (per-webhook DB)
│ │ ├── model_delivery.go # Delivery entity (per-webhook DB)
│ │ ├── model_delivery_result.go # DeliveryResult entity (per-webhook DB)
│ │ ├── model_apikey.go # APIKey entity
│ │ ├── password.go # Argon2id hashing and verification
│ │ └── webhook_db_manager.go # Per-webhook DB lifecycle manager
│ ├── globals/
│ │ └── globals.go # Build-time variables (appname, version, arch)
│ ├── delivery/
│ │ ├── engine.go # Event-driven delivery engine (channel + timer based)
│ │ └── circuit_breaker.go # Per-target circuit breaker for HTTP targets with retries
│ ├── handlers/
│ │ ├── handlers.go # Base handler struct, JSON helpers, template rendering
│ │ ├── auth.go # Login, logout handlers
│ │ ├── healthcheck.go # Health check handler
│ │ ├── index.go # Index page handler
│ │ ├── profile.go # User profile handler
│ │ ├── source_management.go # Webhook CRUD handlers
│ │ └── webhook.go # Webhook receiver handler
│ ├── healthcheck/
│ │ └── healthcheck.go # Health check service (uptime, version)
│ ├── logger/
│ │ └── logger.go # slog setup with TTY detection
│ ├── middleware/
│ │ └── middleware.go # Logging, CORS, Auth, Metrics, MetricsAuth
│ ├── server/
│ │ ├── server.go # Server struct, fx lifecycle, signal handling
│ │ ├── http.go # HTTP server setup with timeouts
│ │ └── routes.go # All route definitions
│ └── session/
│ └── session.go # Cookie-based session management
├── static/
│ ├── static.go # //go:embed directive
│ ├── css/style.css # Custom stylesheet (system font stack, card effects, layout)
│ └── js/app.js # Client-side JavaScript (minimal bootstrap)
├── templates/ # Go HTML templates (base, index, login, etc.)
├── Dockerfile # Multi-stage: build + check, then Alpine runtime
├── Makefile # fmt, lint, test, check, build, docker targets
├── go.mod / go.sum
└── .golangci.yml # Linter configuration
```
### Dependency Injection
Components are wired via Uber fx in this order:
1. `globals.New` — Build-time variables (appname, version, arch)
2. `logger.New` — Structured logging (slog with TTY detection)
3. `config.New` — Configuration loading (environment variables)
4. `database.New` — Main SQLite connection, config migrations, admin
user seed
5. `database.NewWebhookDBManager` — Per-webhook event database
lifecycle manager
6. `healthcheck.New` — Health check service
7. `session.New` — Cookie-based session manager (key from database)
8. `handlers.New` — HTTP handlers
9. `middleware.New` — HTTP middleware
10. `delivery.New` — Event-driven delivery engine
11. `delivery.Engine` → `handlers.DeliveryNotifier` — interface bridge
12. `server.New` — HTTP server and router
The server starts via `fx.Invoke(func(*server.Server, *delivery.Engine)
{})` which triggers the fx lifecycle hooks in dependency order. The
`DeliveryNotifier` interface allows the webhook handler to send
self-contained `DeliveryTask` slices to the engine without a direct
package dependency. Each task carries all target config and event data
inline (for bodies ≤16KB), so the engine can deliver without reading
from any database — it only writes to record results.
### Middleware Stack
Applied to all routes in this order:
1. **Recoverer** — Panic recovery (chi built-in)
2. **RequestID** — Generate unique request IDs (chi built-in)
3. **Logging** — Structured request logging (method, URL, status,
latency, remote IP, user agent, request ID)
4. **Metrics** — Prometheus HTTP metrics (if `METRICS_USERNAME` is set)
5. **CORS** — Cross-origin resource sharing headers
6. **Timeout** — 60-second request timeout
7. **Sentry** — Error reporting to Sentry (if `SENTRY_DSN` is set;
configured with `Repanic: true` so panics still reach Recoverer)
### Authentication
- **Web UI:** Cookie-based sessions using gorilla/sessions with
encrypted cookies. Sessions are configured with HttpOnly, SameSite
Lax, and Secure (in production). Session lifetime is 7 days.
- **API (planned):** API key authentication via `Authorization: Bearer`
header. API keys are stored per-user with usage tracking
(`last_used_at`).
- **Metrics:** Basic authentication protecting the `/metrics` endpoint.
### Security
- Passwords hashed with Argon2id (64 MB memory cost)
- Session cookies are HttpOnly, SameSite Lax, Secure (prod only)
- Session key is a 32-byte value auto-generated on first startup and
stored in the database
- Prometheus metrics behind basic auth
- Static assets embedded in binary (no filesystem access needed at
runtime)
- Container runs as non-root user (UID 1000)
- GORM soft deletes on all entities (data preserved for audit)
### Docker
The Dockerfile uses a multi-stage build:
1. **Builder stage** (Debian-based `golang:1.24`) — installs
golangci-lint, downloads dependencies, copies source, runs `make
check` (format verification, linting, tests, compilation).
2. **Runtime stage** (`alpine:3.21`) — copies the binary, creates the
`/data` directory for all SQLite databases, runs as non-root user,
exposes port 8080, includes a health check.
The builder uses Debian rather than Alpine because GORM's SQLite
dialect pulls in CGO-dependent headers at compile time. The runtime
binary is statically linked and runs on Alpine.
`docker build .` is the CI gate — if it passes, the code is formatted,
linted, tested, and compiled.
## TODO
### Completed: Code Quality (Phase 1 of MVP)
- [x] Rename Processor → Webhook, Webhook → Entrypoint in code
([#12](https://git.eeqj.de/sneak/webhooker/issues/12))
- [x] Embed templates via `//go:embed`
([#7](https://git.eeqj.de/sneak/webhooker/issues/7))
- [x] Use `slog.LevelVar` for dynamic log level switching
([#8](https://git.eeqj.de/sneak/webhooker/issues/8))
- [x] Simplify configuration to prefer environment variables
([#10](https://git.eeqj.de/sneak/webhooker/issues/10))
- [x] Remove redundant `godotenv/autoload` import
([#11](https://git.eeqj.de/sneak/webhooker/issues/11))
- [x] Implement authentication middleware for protected routes
([#9](https://git.eeqj.de/sneak/webhooker/issues/9))
- [x] Replace Bootstrap with Tailwind CSS + Alpine.js
([#4](https://git.eeqj.de/sneak/webhooker/issues/4))
### Completed: Core Webhook Engine (Phase 2 of MVP)
- [x] Implement webhook reception and event storage at `/webhook/{uuid}`
- [x] Build event processing and target delivery engine
- [x] Implement HTTP target type (fire-and-forget with max_retries=0,
retries with exponential backoff when max_retries>0)
- [x] Implement database target type (store events in per-webhook DB)
- [x] Implement log target type (console output)
- [x] Webhook management pages (list, create, edit, delete)
- [x] Webhook request log viewer with pagination
- [x] Entrypoint and target management UI
### Completed: Per-Webhook Event Databases
- [x] Split into main application DB + per-webhook event DBs
- [x] Per-webhook database lifecycle management (create on webhook
creation, delete on webhook removal)
- [x] `WebhookDBManager` component with lazy connection pooling
- [x] Event-driven delivery engine (channel notifications + timer-based retries)
- [x] Self-contained delivery tasks: in the ≤16KB happy path, the engine
delivers without reading from any database — target config, event
headers, and body are all carried inline in the channel notification.
The engine only touches the DB to record results (success/failure).
Large bodies (≥16KB) are fetched from the per-webhook DB on demand.
- [x] Database target type marks delivery as immediately successful
(events are already in the per-webhook DB)
- [x] Parallel fan-out: all targets for an event are delivered via
the bounded worker pool (no goroutine-per-target)
- [x] Circuit breaker for HTTP targets with retries: tracks consecutive
failures per target, opens after 5 failures (30s cooldown),
half-open probe to test recovery
### Remaining: Core Features
- [ ] Per-webhook rate limiting in the receiver handler
- [ ] Webhook signature verification (GitHub, Stripe formats)
- [ ] Security headers (HSTS, CSP, X-Frame-Options)
- [ ] CSRF protection for forms
- [ ] Session expiration and "remember me"
- [ ] Password change/reset flow
- [ ] API key authentication for programmatic access
- [ ] Manual event redelivery
- [ ] Analytics dashboard (success rates, response times)
- [ ] Delivery status and retry management UI
### Remaining: Event Maintenance
- [ ] Automatic event retention cleanup based on `retention_days`
### Remaining: REST API
- [ ] RESTful CRUD for webhooks, entrypoints, targets
- [ ] Event viewing and filtering endpoints
- [ ] Event redelivery endpoint
- [ ] OpenAPI specification
### Future
- [ ] Email delivery target type
- [ ] SNS, S3, Slack delivery targets
- [ ] Data transformations (e.g., webhook-to-Slack message formatting)
- [ ] JSONL file delivery with periodic S3 upload
- [ ] Webhook event search and filtering
- [ ] Multi-user with role-based access
## License
MIT
## Author
[@sneak](https://sneak.berlin)
Reference in New Issue View Git Blame Copy Permalink