feat: implement per-webhook event databases
All checks were successful
check / check (push) Successful in 1m50s

Split data storage into main application DB (config only) and
per-webhook event databases (one SQLite file per webhook).

Architecture changes:
- New WebhookDBManager component manages per-webhook DB lifecycle
  (create, open, cache, delete) with lazy connection pooling via sync.Map
- Main DB (DBURL) stores only config: Users, Webhooks, Entrypoints,
  Targets, APIKeys
- Per-webhook DBs (DATA_DIR) store Events, Deliveries, DeliveryResults
  in files named events-{webhook_uuid}.db
- New DATA_DIR env var (default: ./data dev, /data/events prod)

Behavioral changes:
- Webhook creation creates per-webhook DB file
- Webhook deletion hard-deletes per-webhook DB file (config soft-deleted)
- Event ingestion writes to per-webhook DB, not main DB
- Delivery engine polls all per-webhook DBs for pending deliveries
- Database target type marks delivery as immediately successful (events
  are already in the dedicated per-webhook DB)
- Event log UI reads from per-webhook DBs with targets from main DB
- Existing webhooks without DB files get them created lazily

Removed:
- ArchivedEvent model (was a half-measure, replaced by per-webhook DBs)
- Event/Delivery/DeliveryResult removed from main DB migrations

Added:
- Comprehensive tests for WebhookDBManager (create, delete, lazy
  creation, delivery workflow, multiple webhooks, close all)
- Dockerfile creates /data/events directory

README updates:
- Per-webhook event databases documented as implemented (was Phase 2)
- DATA_DIR added to configuration table
- Docker instructions updated with data volume mount
- Data model diagram updated
- TODO updated (database separation moved to completed)

Closes #15
This commit is contained in:
clawbot
2026-03-01 17:06:43 -08:00
parent 6c393ccb78
commit 43c22a9e9a
13 changed files with 814 additions and 198 deletions

View File

@@ -37,6 +37,7 @@ type ConfigParams struct {
type Config struct {
DBURL string
DataDir string
Debug bool
MaintenanceMode bool
DevelopmentMode bool
@@ -116,6 +117,7 @@ func New(lc fx.Lifecycle, params ConfigParams) (*Config, error) {
// Load configuration values — env vars take precedence over config.yaml
s := &Config{
DBURL: envString("DBURL", "dburl"),
DataDir: envString("DATA_DIR", "dataDir"),
Debug: envBool("DEBUG", "debug"),
MaintenanceMode: envBool("MAINTENANCE_MODE", "maintenanceMode"),
DevelopmentMode: envBool("DEVELOPMENT_MODE", "developmentMode"),
@@ -129,6 +131,15 @@ func New(lc fx.Lifecycle, params ConfigParams) (*Config, error) {
params: &params,
}
// Set default DataDir based on environment
if s.DataDir == "" {
if s.IsProd() {
s.DataDir = "/data/events"
} else {
s.DataDir = "./data"
}
}
// Validate database URL
if s.DBURL == "" {
return nil, fmt.Errorf("database URL (DBURL) is required")
@@ -156,6 +167,7 @@ func New(lc fx.Lifecycle, params ConfigParams) (*Config, error) {
"debug", s.Debug,
"maintenanceMode", s.MaintenanceMode,
"developmentMode", s.DevelopmentMode,
"dataDir", s.DataDir,
"hasSessionKey", s.SessionKey != "",
"hasSentryDSN", s.SentryDSN != "",
"hasMetricsAuth", s.MetricsUsername != "" && s.MetricsPassword != "",