webhooker

webhooker is, at its core, a webhook proxy. it receives webhooks and stores them in a database, then delivers them (either fire-and-forget, or with retries until successful) to a configured endpoint.

webhook endpoints are inputs to an event processor. there can be multiple webhook endpoint urls configured as inputs to any given processor. each processor can have 0 or more targets. all events delivered to the event processor will be delivered to all targets. such an object is called a 'delivery'. (this is not related to whether or not the target was successful in delivering the event, just that it was proxied.)

each target type can deliver the webhook in a different way. the first target type simply POSTs the webhook to a configured URL, doing a direct fire-and-forget proxy. the duration, result, status code, and any other metadata is logged in the database associated with the 'delivery' object (which is the pairing of that event with that target).

the second target type is a 'retry' target. this target will retry the delivery of the event to the target until it is successful, or until a per-target maximum retry count is reached. the retry target will exponentially back off the retry attempts, starting at 1 second and increasing 10x each time.

the third target type is a database target. this target will store the event in the database, but will not attempt to deliver it to any endpoint.

the fourth target type is a log target. this target will log the event to the console, but will not attempt to deliver it to any endpoint.

features

• store and forward (retries can be set to unlimited)
• multiple webhook endpoints per processor
• metrics availability in prometheus format for observability
• web interface for managing processors, webhooks, and targets
• web interface for viewing deliveries, events, and delivery results
• web interface for manually redelivering events to targets

database tables

• users - these are the users of the webhooker service. each user has a unique username and password, and can have multiple API keys.
• processors - many to one with users.
  • each processor has a retention period, which is the number of days that events will be retained in the database before being deleted.
• webhooks - these are the URLs that feed into processors. many to one.
• targets - these are the delivery targets for each processor. many to one with processors.
  • each target has a type, which can be one of the following:
    • http - this target will deliver the event to a configured URL via HTTP POST.
    • retry - this target will retry the delivery of the event until it is successful or a maximum retry count is reached.
      • each 'retry' target can have a maximum queue size, which is the maximum number of events that can be queued for retry delivery. when the queue size is reached, the oldest events will be purged from the queue.
    • database - this target will store the event in a database, but will not attempt to deliver it to any endpoint.
    • log - this target will log the event to the console, but will not attempt to deliver it to any endpoint.
• deliveries - these are the delivery attempts for each target. many to one with targets.
• events - these are the events that are delivered to targets. one to 0 or more with deliveries, based on targets
• delivery_results - these are the results of each delivery attempt. many to one with deliveries, but 1-to-1 in the usual case.

Web Application

webhooker is a go application that provides a standard non-SPA web interface for managing one's processors, webhooks, targets, and API keys. it allows you to view your deliveries, events, and delivery results, and to optionally manually redeliver events to targets.

the web application is implemented primarily without javascript, using bootstrap for styling and layout. it uses go templates for rendering HTML, and those templates are embedded in the binary using go:embed.

architecture

webhooker uses Uber's fx dependency injection library for managing application lifecycle. it uses log/slog for structured logging. it uses gorm for database access, and initially uses sqlite for the database.

Development Policies and Practices

Code Structure and Organization

Package Layout

• All application code lives under internal/ to prevent external imports
• Main entry point is in cmd/webhooker/
• Package structure:
  • internal/config - Configuration management using Viper
  • internal/database - Database connections and migrations
  • internal/globals - Global application metadata (version, build info)
  • internal/handlers - HTTP handlers using closure pattern
  • internal/healthcheck - Health check endpoint logic
  • internal/logger - Structured logging setup
  • internal/middleware - HTTP middleware (auth, CORS, logging, metrics)
  • internal/server - HTTP server setup and routing
  • static/ - Static assets (CSS, JS)
  • templates/ - Go HTML templates

Dependency Injection

• Use Uber's fx for dependency injection and lifecycle management
• All components should be provided via fx.Provide in main.go
• Use parameter structs with fx.In for dependency injection
• Follow fx lifecycle hooks (OnStart, OnStop) for initialization

HTTP Routing and Handlers

• Use chi router for HTTP routing
• Handlers are methods on a Handlers struct that return http.HandlerFunc
• Use closure pattern for handlers:

  func (h *Handlers) HandleIndex() http.HandlerFunc {
      return func(w http.ResponseWriter, r *http.Request) {
          // handler logic here
      }
  }
  

• Group related routes using chi's Route method
• Apply middleware in correct order: recovery, request ID, logging, metrics, CORS, timeout

Configuration

• Use Viper for configuration management
• Support environment variables with automatic binding
• Use .env files for local development (via godotenv/autoload)
• Set sensible defaults for all configuration values
• Configuration precedence: env vars > config file > defaults

Logging

• Use zerolog for structured JSON logging
• Always log in UTC
• Use console writer for TTY, JSON for production
• Include request ID in all request-scoped logs
• Log levels: Debug (development only), Info, Warn, Error, Fatal
• Include relevant context in all log messages

Error Handling

• Use Sentry for error reporting in production (if SENTRY_DSN is set)
• Implement panic recovery middleware
• Return appropriate HTTP status codes
• Log all errors with context

Database

• Use GORM for database operations
• Support SQLite for development, PostgreSQL for production
• Run migrations on startup
• Use UTC for all timestamps
• Implement proper connection pooling

Testing and Quality

• Write tests for all new functionality
• Use table-driven tests where appropriate
• Mock external dependencies in tests
• Achieve reasonable test coverage (not 100% required)
• Use testify for assertions

Development Workflow

• Use Makefile with standard targets:
  • make test - Run all tests with go test -v ./...
  • make fmt - Format code with go fmt
  • make lint - Run golangci-lint
  • make build - Build the binary
  • Default target is test
• Pre-commit checks: make test && make fmt && make lint
• Use golangci-lint for linting with appropriate configuration

Code Style

• Follow standard Go idioms and effective Go guidelines
• Use constants for repeated values (no magic numbers/strings)
• Define constants at package level or top of file
• Keep functions small and focused
• Use meaningful variable names
• Comment exported functions and types
• Use structured types for request/response near usage

Security

• Implement proper authentication for admin endpoints
• Use basic auth for metrics endpoint
• Sanitize all user input
• Use prepared statements for database queries
• Don't log sensitive information (passwords, API keys)

Observability

• Expose Prometheus metrics on /metrics (behind auth)
• Include standard HTTP metrics (request count, duration, size)
• Add custom metrics for business logic
• Implement health check endpoint at /.well-known/healthcheck.json
• Include version info in health check response

Static Assets

• Embed static files and templates in binary using go:embed
• Serve static files from /s/ path
• Use Bootstrap for UI without heavy JavaScript
• Prefer server-side rendering over SPA

Docker

• Multi-stage Dockerfile for minimal image size
• Run as non-root user in container
• Include health check in Dockerfile
• Use Alpine Linux for smaller images

Current Implementation Status

What's Working

1. Core Infrastructure

   • Dependency injection using Uber's fx
   • Structured logging with Go's standard log/slog (replaced zerolog)
   • Configuration management with Viper
   • Environment variable support with godotenv
   • Pure Go SQLite database driver (modernc.org/sqlite) - no CGO required

2. Web Server

   • HTTP server with chi router
   • Middleware stack (recovery, request ID, logging, CORS, metrics, timeout)
   • Static file serving from embedded filesystem
   • JSON API endpoints
   • Health check endpoint at /.well-known/healthcheck.json
   • Metrics endpoint at /metrics (with basic auth)

3. Database Models

   • Full GORM models for all entities (users, processors, webhooks, targets, deliveries, events, delivery_results)
   • Argon2id password hashing with secure defaults
   • Auto-creation of admin user on first startup (password logged once)

4. Testing

   • Unit tests for globals, logger, and database packages
   • Database connection test that verifies SQLite works without CGO
   • All tests pass

5. Development Tools

   • Makefile with test, fmt, lint, build targets
   • golangci-lint configuration (some deprecated linters need updating)
   • Docker multi-stage build (updated to support CGO if needed)
   • gitignore file

API Endpoints

• GET / - Returns JSON welcome message
• GET /.well-known/healthcheck.json - Health check with uptime, version info
• GET /s/* - Static file serving (CSS, JS, Bootstrap)
• GET /metrics - Prometheus metrics (requires auth)
• GET /api/v1/* - API route group (not implemented yet)

Next Steps

1. Authentication

   • Implement user login/logout
   • API key generation and validation
   • Session management
   • Protected routes

2. Core Features

   • Webhook receiver endpoints
   • Event processing logic
   • Target delivery system (HTTP, retry, database, log)
   • Web UI templates for management

3. API Implementation

   • CRUD endpoints for processors, webhooks, targets
   • Event viewing and filtering
   • Delivery status and retry management

4. Cleanup

   • Update golangci-lint config to remove deprecated linters
   • Add more comprehensive tests
   • Implement proper error handling

Running the Application

# Development
make dev

# Build and run
make build
./bin/webhooker

# Run tests
make test

# Docker
make docker-build
make docker-run

Environment Variables

• DEBUG - Enable debug logging
• DEVELOPMENT_MODE - Enable development features
• PORT - Server port (default: 8080)
• DBURL - Database connection string
• METRICS_USERNAME - Username for metrics endpoint
• METRICS_PASSWORD - Password for metrics endpoint
• SENTRY_DSN - Sentry error reporting (optional)

purpose

• prometheus/grafana metric analysis of webhook frequency, size, and handler performance
• introspection of webhook data
• debugging of webhook delivery issues
• redelivery of webhook events for application testing and development
• storage of webhook events for later analysis
• webhook delivery to multiple targets
• webhook delivery with retries for not-always-on applications
• can serve as an HA endpoint for ingestion to deliver events to less reliable systems

future plans

• email source type
• email delivery target
• sns delivery target
• s3 delivery target
• data transformations (e.g. to convert event notification data to slack messages)
• slack delivery target
• slack source type
• text file or jsonl delivery target, with periodic uploading to s3