webhooker/README.md

# webhooker

webhooker is a Go web application by [@sneak](https://sneak.berlin) that
receives, stores, and proxies webhooks to configured targets with retry
support, observability, and a management web UI. License: MIT.

## Getting Started

```bash
# Clone the repo
git clone https://git.eeqj.de/sneak/webhooker.git
cd webhooker

# Install dependencies
make deps

# Copy example config
cp configs/config.yaml.example config.yaml

# Run in development mode
make dev

# Run all checks (format, lint, test, build)
make check

# Build Docker image
make docker
```

### Environment Variables

- `WEBHOOKER_ENVIRONMENT` — `dev` or `prod` (default: `dev`)
- `DEBUG` — Enable debug logging
- `PORT` — Server port (default: `8080`)
- `DBURL` — Database connection string
- `SESSION_KEY` — Base64-encoded 32-byte session key (required in prod)
- `METRICS_USERNAME` — Username for metrics endpoint
- `METRICS_PASSWORD` — Password for metrics endpoint
- `SENTRY_DSN` — Sentry error reporting (optional)

## Rationale

Webhook integrations between services are fragile: the receiving service
must be up when the webhook fires, there is no built-in retry for most
webhook senders, and there is no visibility into what was sent or when.
webhooker solves this by acting as a reliable intermediary that receives
webhooks, stores them, and delivers them to configured targets — with
optional retries, logging, and Prometheus metrics for observability.

Use cases include:

- Store-and-forward with unlimited retries for unreliable receivers
- Prometheus/Grafana metric analysis of webhook frequency, size, and
  handler performance
- Introspection and debugging of webhook payloads
- Redelivery of webhook events for application testing
- Fan-out delivery of webhooks to multiple targets
- HA ingestion endpoint for delivery to less reliable systems

## Design

### Architecture

webhooker uses Uber's fx dependency injection library for managing
application lifecycle. It uses `log/slog` for structured logging, GORM
for database access, and SQLite (via `modernc.org/sqlite`, pure Go, no
CGO) for storage. HTTP routing uses chi.

### Rate Limiting

Global rate limiting middleware (e.g. per-IP throttling applied at the
router level) must **not** apply to webhook receiver endpoints
(`/webhook/{uuid}`). Webhook endpoints receive automated traffic from
external services at unpredictable rates, and blanket rate limits would
cause legitimate webhook deliveries to be dropped.

Instead, each webhook endpoint has its own individually configurable rate
limit, applied within the webhook handler itself. By default, no rate
limit is applied — webhook endpoints accept traffic as fast as it arrives.
Rate limits can be configured on a per-webhook basis in the application
when needed (e.g. to protect against a misbehaving sender).

### Database Architecture

webhooker uses separate SQLite database files rather than a single
monolithic database:

- **Main application database** — Stores application configuration and
  all standard webapp data: users, sessions, API keys, and global
  settings.
- **Per-webhook databases** — Each webhook gets its own dedicated SQLite
  database file containing: input logs, webhook logs, and all output
  queues for that specific webhook.

This separation provides several benefits: webhook databases can be
independently backed up, rotated, or archived; a high-volume webhook
won't cause lock contention or bloat affecting the main application; and
individual webhook data can be cleanly deleted when a webhook is
removed.

### Package Layout

All application code lives under `internal/` to prevent external imports.
The main entry point is `cmd/webhooker/main.go`.

- `internal/config` — Configuration management via `pkg/config` (Viper-based)
- `internal/database` — GORM database connection, migrations, and models
- `internal/globals` — Global application metadata (version, build info)
- `internal/handlers` — HTTP handlers using the closure pattern
- `internal/healthcheck` — Health check endpoint logic
- `internal/logger` — Structured logging setup (`log/slog`)
- `internal/middleware` — HTTP middleware (auth, CORS, logging, metrics)
- `internal/server` — HTTP server setup and routing
- `internal/session` — Session management (gorilla/sessions)
- `pkg/config` — Reusable multi-environment configuration library
- `static/` — Embedded static assets (CSS, JS)
- `templates/` — Go HTML templates

### Data Model

- **Users** — Service users with username/password (Argon2id hashing)
- **Webhooks** — Webhook processing units, many-to-one with users
- **Entrypoints** — Inbound URL endpoints feeding into webhooks
- **Targets** — Delivery destinations per webhook (HTTP, retry, database, log)
- **Events** — Captured webhook payloads
- **Deliveries** — Pairing of events with targets
- **Delivery Results** — Outcome of each delivery attempt
- **API Keys** — Programmatic access credentials per user

### API Endpoints

- `GET /` — Web UI index page
- `GET /.well-known/healthcheck` — Health check with uptime, version
- `GET /s/*` — Static file serving (CSS, JS)
- `GET /metrics` — Prometheus metrics (requires basic auth)
- `POST /webhook/{uuid}` — Webhook receiver endpoint
- `/pages/login`, `/pages/logout` — Authentication
- `/user/{username}` — User profile
- `/sources/*` — Webhook source management

## TODO

### Phase 1: Security & Infrastructure
- [ ] Security headers (HSTS, CSP, X-Frame-Options)
- [ ] Rate limiting middleware
- [ ] CSRF protection for forms
- [ ] Request ID tracking through entire lifecycle

### Phase 2: Authentication & Authorization
- [ ] Authentication middleware for protected routes
- [ ] Session expiration and "remember me"
- [ ] Password reset flow
- [ ] API key authentication for programmatic access

### Phase 3: Core Webhook Features
- [ ] Webhook reception and event storage at `/webhook/{uuid}`
- [ ] Event processing and target delivery engine
- [ ] HTTP target type (fire-and-forget POST)
- [ ] Retry target type (exponential backoff)
- [ ] Database target type (store only)
- [ ] Log target type (console output)
- [ ] Webhook signature verification (GitHub, Stripe formats)

### Phase 4: Web UI
- [ ] Webhook source management pages (list, create, edit, delete)
- [ ] Webhook request log viewer with filtering
- [ ] Delivery status and retry management UI
- [ ] Manual event redelivery
- [ ] Analytics dashboard (success rates, response times)

### Phase 5: API
- [ ] RESTful CRUD for webhooks, entrypoints, targets
- [ ] Event viewing and filtering endpoints
- [ ] API documentation (OpenAPI)

### Future
- [ ] Email source and delivery target types
- [ ] SNS, S3, Slack delivery targets
- [ ] Data transformations (e.g. webhook-to-Slack message)
- [ ] JSONL file delivery with periodic S3 upload

## License

MIT License. See [LICENSE](LICENSE) for details.

## Author

[@sneak](https://sneak.berlin)