feat: implement per-webhook event databases
All checks were successful
check / check (push) Successful in 1m50s
All checks were successful
check / check (push) Successful in 1m50s
Split data storage into main application DB (config only) and
per-webhook event databases (one SQLite file per webhook).
Architecture changes:
- New WebhookDBManager component manages per-webhook DB lifecycle
(create, open, cache, delete) with lazy connection pooling via sync.Map
- Main DB (DBURL) stores only config: Users, Webhooks, Entrypoints,
Targets, APIKeys
- Per-webhook DBs (DATA_DIR) store Events, Deliveries, DeliveryResults
in files named events-{webhook_uuid}.db
- New DATA_DIR env var (default: ./data dev, /data/events prod)
Behavioral changes:
- Webhook creation creates per-webhook DB file
- Webhook deletion hard-deletes per-webhook DB file (config soft-deleted)
- Event ingestion writes to per-webhook DB, not main DB
- Delivery engine polls all per-webhook DBs for pending deliveries
- Database target type marks delivery as immediately successful (events
are already in the dedicated per-webhook DB)
- Event log UI reads from per-webhook DBs with targets from main DB
- Existing webhooks without DB files get them created lazily
Removed:
- ArchivedEvent model (was a half-measure, replaced by per-webhook DBs)
- Event/Delivery/DeliveryResult removed from main DB migrations
Added:
- Comprehensive tests for WebhookDBManager (create, delete, lazy
creation, delivery workflow, multiple webhooks, close all)
- Dockerfile creates /data/events directory
README updates:
- Per-webhook event databases documented as implemented (was Phase 2)
- DATA_DIR added to configuration table
- Docker instructions updated with data volume mount
- Data model diagram updated
- TODO updated (database separation moved to completed)
Closes #15
This commit is contained in:
@@ -37,6 +37,7 @@ type ConfigParams struct {
|
||||
|
||||
type Config struct {
|
||||
DBURL string
|
||||
DataDir string
|
||||
Debug bool
|
||||
MaintenanceMode bool
|
||||
DevelopmentMode bool
|
||||
@@ -116,6 +117,7 @@ func New(lc fx.Lifecycle, params ConfigParams) (*Config, error) {
|
||||
// Load configuration values — env vars take precedence over config.yaml
|
||||
s := &Config{
|
||||
DBURL: envString("DBURL", "dburl"),
|
||||
DataDir: envString("DATA_DIR", "dataDir"),
|
||||
Debug: envBool("DEBUG", "debug"),
|
||||
MaintenanceMode: envBool("MAINTENANCE_MODE", "maintenanceMode"),
|
||||
DevelopmentMode: envBool("DEVELOPMENT_MODE", "developmentMode"),
|
||||
@@ -129,6 +131,15 @@ func New(lc fx.Lifecycle, params ConfigParams) (*Config, error) {
|
||||
params: ¶ms,
|
||||
}
|
||||
|
||||
// Set default DataDir based on environment
|
||||
if s.DataDir == "" {
|
||||
if s.IsProd() {
|
||||
s.DataDir = "/data/events"
|
||||
} else {
|
||||
s.DataDir = "./data"
|
||||
}
|
||||
}
|
||||
|
||||
// Validate database URL
|
||||
if s.DBURL == "" {
|
||||
return nil, fmt.Errorf("database URL (DBURL) is required")
|
||||
@@ -156,6 +167,7 @@ func New(lc fx.Lifecycle, params ConfigParams) (*Config, error) {
|
||||
"debug", s.Debug,
|
||||
"maintenanceMode", s.MaintenanceMode,
|
||||
"developmentMode", s.DevelopmentMode,
|
||||
"dataDir", s.DataDir,
|
||||
"hasSessionKey", s.SessionKey != "",
|
||||
"hasSentryDSN", s.SentryDSN != "",
|
||||
"hasMetricsAuth", s.MetricsUsername != "" && s.MetricsPassword != "",
|
||||
|
||||
Reference in New Issue
Block a user