7ca62664d0d82de91c17a7de977e19ef68586032
7 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| f6b929f2d7 |
Add per-webhook event retention reaper (closes #63) (#78)
All checks were successful
check / check (push) Successful in 2m42s
Enforces each webhook's `RetentionDays` so per-webhook SQLite files no longer grow without bound. ## Reaper New `RetentionReaper` in `internal/database/retention.go`. A background ticker runs each sweep: it lists all webhooks from the main DB and, for each webhook with a positive `RetentionDays`, opens its per-webhook DB via `WebhookDBManager.GetDB` and deletes every `Event` (and its dependent `Delivery` and `DeliveryResult` rows) whose `CreatedAt` is older than `RetentionDays` days. - Deletions run in foreign-key-safe order: delivery results, then deliveries, then events. - Deletes are unscoped (hard deletes) so rows are physically removed and disk is reclaimed, rather than GORM soft-deleting them. - `RetentionDays <= 0` means retain forever; those webhooks are skipped. - Webhooks whose per-webhook DB does not yet exist are skipped. ## Config `internal/config/config.go` gains `RetentionSweepInterval` (env `RETENTION_SWEEP_INTERVAL`, parsed as a Go duration, default `1h`) via a new `envDuration` helper, following the existing env-helper conventions. ## Wiring `cmd/webhooker/main.go` registers `database.NewRetentionReaper` as an fx provider and forces its construction in `fx.Invoke`. The reaper starts its sweep loop on an fx `OnStart` hook and stops cleanly on `OnStop` via context cancellation, matching the existing lifecycle components. ## Test `internal/database/retention_test.go` seeds an old event chain (event + delivery + result, 40 days old) and a recent one (1 day old) in a real per-webhook DB and asserts a single sweep removes only the expired chain while keeping the recent one. A second test forces a non-positive `RetentionDays` and asserts an ancient event is retained. Note: the `Webhook.RetentionDays` column carries `gorm:"default:30"`, so a `0` passed to a GORM `Create` is replaced by the default; the test forces the value with an explicit column update to exercise the retain-forever path. No model changes were made. Validated with `docker build .` (fmt-check, lint, test, build) exit 0. Closes #63 Co-authored-by: sneak <sneak@sneak.berlin> Reviewed-on: #78 Co-authored-by: clawbot <clawbot@noreply.example.org> Co-committed-by: clawbot <clawbot@noreply.example.org> |
|||
| afe88c601a |
refactor: use pinned golangci-lint Docker image for linting (#55)
All checks were successful
check / check (push) Successful in 5s
Closes [issue #50](#50) ## Summary Refactors the Dockerfile to use a separate lint stage with a pinned golangci-lint Docker image, following the pattern used by [sneak/pixa](https://git.eeqj.de/sneak/pixa). This replaces the previous approach of installing golangci-lint via curl in the builder stage. ## Changes ### Dockerfile - **New `lint` stage** using `golangci/golangci-lint:v2.11.3` (Debian-based, pinned by sha256 digest) as a separate build stage - **Builder stage** depends on lint via `COPY --from=lint /src/go.sum /dev/null` — build won't proceed unless linting passes - **Go bumped** from 1.24 to 1.26.1 (`golang:1.26.1-bookworm`, pinned by sha256) - **golangci-lint bumped** from v1.64.8 to v2.11.3 - All three Docker images (golangci-lint, golang, alpine) pinned by sha256 digest - Debian-based golangci-lint image used (not Alpine) because mattn/go-sqlite3 CGO does not compile on musl (off64_t) ### Linter Config (.golangci.yml) - Migrated from v1 to v2 format (`version: "2"` added) - Removed linters no longer available in v2: `gofmt` (handled by `make fmt-check`), `gosimple` (merged into `staticcheck`), `typecheck` (always-on in v2) - Same set of linters enabled — no rules weakened ### Code Fixes (all lint issues from v2 upgrade) - Added package comments to all packages - Added doc comments to all exported types, functions, and methods - Fixed unchecked errors flagged by `errcheck` (sqlDB.Close, os.Setenv in tests, resp.Body.Close, fmt.Fprint) - Fixed unused parameters flagged by `revive` (renamed to `_`) - Fixed `gosec` G120 warnings: added `http.MaxBytesReader` before `r.ParseForm()` calls - Fixed `staticcheck` QF1012: replaced `WriteString(fmt.Sprintf(...))` with `fmt.Fprintf` - Fixed `staticcheck` QF1003: converted if/else chain to tagged switch - Renamed `DeliveryTask` → `Task` to avoid package stutter (`delivery.Task` instead of `delivery.DeliveryTask`) - Renamed shadowed builtin `max` parameter to `upperBound` in `cryptoRandInt` - Used `t.Setenv` instead of `os.Setenv` in tests (auto-restores) ### README.md - Updated version requirements: Go 1.26+, golangci-lint v2.11+ - Updated Dockerfile description in project structure ## Verification `docker build .` passes cleanly — formatting check, linting, all tests, and build all succeed. Co-authored-by: clawbot <clawbot@noreply.git.eeqj.de> Reviewed-on: #55 Co-authored-by: clawbot <clawbot@noreply.example.org> Co-committed-by: clawbot <clawbot@noreply.example.org> |
|||
| a51e863017 |
Remove globals.Buildarch from codebase (#31)
All checks were successful
check / check (push) Successful in 1m0s
Remove the `Buildarch` field from the globals package and all references throughout the codebase. **Changes:** - Removed `Buildarch` package-level var and struct field from `internal/globals/globals.go` - Removed `Buildarch` from the `New()` constructor - Removed `globals.Buildarch = runtime.GOARCH` and unused `runtime` import from `cmd/webhooker/main.go` - Removed `buildarch` from logger startup output in `internal/logger/logger.go` - Removed all `Buildarch` test setup and assertions from globals, logger, database, and webhook_db_manager tests All tests pass, `make check` passes, `docker build .` succeeds. closes [issue #30](#30) <!-- session: agent:sdlc-manager:subagent:5cae6803-6bdf-467d-9a56-43f135521e5f --> Co-authored-by: clawbot <clawbot@noreply.git.eeqj.de> Reviewed-on: #31 Co-authored-by: clawbot <clawbot@noreply.example.org> Co-committed-by: clawbot <clawbot@noreply.example.org> |
|||
clawbot
|
5e683af2a4 |
refactor: event-driven delivery engine with channel notifications and timer-based retries
All checks were successful
check / check (push) Successful in 58s
Replace the polling-based delivery engine with a fully event-driven architecture using Go channels and goroutines: - Webhook handler notifies engine via buffered channel after creating delivery records, with inline event data for payloads < 16KB - Large payloads (>= 16KB) use pointer semantics (Body *string = nil) and are fetched from DB on demand, keeping channel memory bounded - Failed retry-target deliveries schedule Go timers with exponential backoff; timers fire into a separate retry channel when ready - On startup, engine scans DB once to recover interrupted deliveries (pending processed immediately, retrying get timers for remaining backoff) - DB stores delivery status for crash recovery only, not for inter-component communication during normal operation - delivery.Notifier interface decouples handlers from engine; fx wires *Engine as Notifier No more periodic polling. No more wasted cycles when idle. |
||
clawbot
|
43c22a9e9a |
feat: implement per-webhook event databases
All checks were successful
check / check (push) Successful in 1m50s
Split data storage into main application DB (config only) and
per-webhook event databases (one SQLite file per webhook).
Architecture changes:
- New WebhookDBManager component manages per-webhook DB lifecycle
(create, open, cache, delete) with lazy connection pooling via sync.Map
- Main DB (DBURL) stores only config: Users, Webhooks, Entrypoints,
Targets, APIKeys
- Per-webhook DBs (DATA_DIR) store Events, Deliveries, DeliveryResults
in files named events-{webhook_uuid}.db
- New DATA_DIR env var (default: ./data dev, /data/events prod)
Behavioral changes:
- Webhook creation creates per-webhook DB file
- Webhook deletion hard-deletes per-webhook DB file (config soft-deleted)
- Event ingestion writes to per-webhook DB, not main DB
- Delivery engine polls all per-webhook DBs for pending deliveries
- Database target type marks delivery as immediately successful (events
are already in the dedicated per-webhook DB)
- Event log UI reads from per-webhook DBs with targets from main DB
- Existing webhooks without DB files get them created lazily
Removed:
- ArchivedEvent model (was a half-measure, replaced by per-webhook DBs)
- Event/Delivery/DeliveryResult removed from main DB migrations
Added:
- Comprehensive tests for WebhookDBManager (create, delete, lazy
creation, delivery workflow, multiple webhooks, close all)
- Dockerfile creates /data/events directory
README updates:
- Per-webhook event databases documented as implemented (was Phase 2)
- DATA_DIR added to configuration table
- Docker instructions updated with data volume mount
- Data model diagram updated
- TODO updated (database separation moved to completed)
Closes #15
|
||
|
clawbot
|
7f8469a0f2 |
feat: implement core webhook engine, delivery system, and management UI (Phase 2)
All checks were successful
check / check (push) Successful in 1m49s
- Webhook reception handler: look up entrypoint by UUID, verify active,
capture full HTTP request (method, headers, body, content-type), create
Event record, queue Delivery records for each active Target, return 200 OK.
Handles edge cases: unknown UUID → 404, inactive → 410, oversized → 413.
- Delivery engine (internal/delivery): fx-managed background goroutine that
polls for pending/retrying deliveries and dispatches to target type handlers.
Graceful shutdown via context cancellation.
- Target type implementations:
- HTTP: fire-and-forget POST with original headers forwarding
- Retry: exponential backoff (1s, 2s, 4s...) up to max_retries
- Database: immediate success (event already stored)
- Log: slog output with event details
- Webhook management pages with Tailwind CSS + Alpine.js:
- List (/sources): webhooks with entrypoint/target/event counts
- Create (/sources/new): form with auto-created default entrypoint
- Detail (/source/{id}): config, entrypoints, targets, recent events
- Edit (/source/{id}/edit): name, description, retention_days
- Delete (/source/{id}/delete): soft-delete with child records
- Add Entrypoint (/source/{id}/entrypoints): inline form
- Add Target (/source/{id}/targets): type-aware form
- Event Log (/source/{id}/logs): paginated with delivery status
- Updated README: marked completed items, updated naming conventions
table, added delivery engine to package layout and DI docs, updated
column names to reflect entity rename.
- Rebuilt Tailwind CSS for new template classes.
Part of: #15
|
||
| f9a9569015 |
feat: bring repo up to REPO_POLICIES standards (#6)
All checks were successful
check / check (push) Successful in 8s
## Summary This PR brings the webhooker repo into full REPO_POLICIES compliance, addressing both [issue #1](#1) and [issue #2](#2). ## Changes ### New files - **`cmd/webhooker/main.go`** — The missing application entry point. Uses Uber fx to wire together all internal packages (config, database, logger, server, handlers, middleware, healthcheck, globals, session). Minimal glue code. - **`REPO_POLICIES.md`** — Fetched from authoritative source (`sneak/prompts`) - **`.editorconfig`** — Fetched from authoritative source - **`.dockerignore`** — Sensible Go project exclusions - **`.gitea/workflows/check.yml`** — CI workflow that runs `docker build .` on push to any branch (Gitea Actions format, actions/checkout pinned by sha256) - **`configs/config.yaml.example`** — Moved from root `config.yaml` ### Modified files - **`Makefile`** — Complete rewrite with all REPO_POLICIES required targets: `test`, `lint`, `fmt`, `fmt-check`, `check`, `build`, `hooks`, `docker`, `clean`, plus `dev`, `run`, `deps` - **`Dockerfile`** — Complete rewrite: - Builder: `golang:1.24` (Debian-based, pinned by `sha256:d2d2bc1c84f7...`). Debian needed because `gorm.io/driver/sqlite` pulls `mattn/go-sqlite3` (CGO) which fails on Alpine musl. - golangci-lint v1.64.8 installed from GitHub release archive with sha256 verification (v1.x because `.golangci.yml` uses v1 config format) - Runs `make check` (fmt-check + lint + test + build) as build step - Final stage: `alpine:3.21` (pinned by `sha256:c3f8e73fdb79...`) with non-root user, healthcheck, port 8080 - **`README.md`** — Rewritten with all required REPO_POLICIES sections: description line with name/purpose/category/license/author, Getting Started, Rationale, Design, TODO (integrated from TODO.md), License, Author - **`.gitignore`** — Fixed `webhooker` pattern to `/webhooker` (was blocking `cmd/webhooker/`), added `config.yaml` to prevent committing runtime config with secrets - **`static/static.go`** — Removed `vendor` from embed directive (directory was empty/missing) - **`internal/database/database_test.go`** — Fixed to use in-memory config via `afero.MemMapFs` instead of depending on `config.yaml` on disk. Test is now properly isolated. - **`go.mod`/`go.sum`** — `go mod tidy` ### Removed files - **`TODO.md`** — Content integrated into README.md TODO section - **`config.yaml`** — Moved to `configs/config.yaml.example` ## Verification - `docker build .` passes (lint ✅, test ✅, build ✅) - All existing tests pass with no modifications to assertions or test logic - `.golangci.yml` untouched closes #1 closes #2 Co-authored-by: clawbot <clawbot@noreply.git.eeqj.de> Reviewed-on: #6 Co-authored-by: clawbot <clawbot@noreply.example.org> Co-committed-by: clawbot <clawbot@noreply.example.org> |
