5 Commits
Author SHA1 Message Date
sneak 45d541526f Align DeliveryResult tags and make the index table test-local
check / check (push) Successful in 2m52s
Re-align the json tag column in DeliveryResult after the wider
DeliveryID gorm tag, and move the test's index table out of package
scope into the test function.

Model: opus-5-5
2026-09-28 10:32:00 +00:00
sneak 06f14315fa Index the event-tier columns the sweeps, event log and retention scan (closes #314)
The per-webhook tables declared no secondary indexes, so the recovery
and sweep queries (by delivery status, every minute), the event log
(deliveries by event, results by delivery) and retention (events by
age) each scanned a whole table. Add indexes through GORM model tags so
AutoMigrate creates them on a fresh and on an existing per-webhook
database. events.created_at is indexed by overriding the embedded
BaseModel field on Event alone, leaving the other tables' created_at
unindexed. A test drops the indexes from an opened database, reopens it,
and asserts the open recreated them. The README Data Model section lists
the indexes.

Model: opus-4-8
2026-09-28 10:31:20 +00:00
clawbot 7ed1588443 Document running webhooker under upaas (closes #323)
check / check (push) Successful in 8s
Adds a "Running under upaas" section to the README: add no port
mapping, since upaas publishes mapped ports on every host interface,
and put the app on the reverse proxy's Docker network instead; one
data volume at /var/lib/webhooker, created owned by UID 1000 before
the first deploy; WEBHOOKER_ENVIRONMENT and TRUSTED_PROXIES; the
health check upaas reads 60 seconds after a deploy; and where the
first-run admin password appears and how to reset it.

upaas bind-mounts a host directory it never creates, and one made by
root stops the container at its data directory lock. The documented
creation step removes that; the image is unchanged.

Model: opus-5-5
2026-09-28 12:30:32 +02:00
clawbot b051821370 Say max_retries is the total attempt count, not a retry count (closes #316)
check / check (push) Successful in 3m54s
The help text under the field on both target forms and the max_retries rows in the README now say the number is the total number of delivery attempts: 0 is a single attempt with no retries and no circuit breaker, and N is N attempts in total. The delivery code already worked this way; only the wording was wrong, so an operator wanting one try plus two retries would have entered 2 instead of 3. A UI copy test renders both forms and pins the wording. Delivery behaviour is unchanged.

Model: opus-4-8 (implementation); fable-5-1 (merge)
2026-09-21 18:33:09 +02:00
clawbot 39afa69bfc Consolidate the data directory mode into one owner (closes #288)
check / check (push) Successful in 6m34s
Two packages each declared the 0o750 mode for DATA_DIR and both created the directory. internal/datadir now exports DirPerm as the single definition, and internal/database uses it in both places it creates the directory. The value is unchanged, so existing deployments see no permission change. datadir owns it because guarding and creating DATA_DIR is that package's whole purpose and it imports nothing that would form a cycle.

Model: opus-4-8 (implementation and review); fable-5-1 (merge)
2026-09-21 10:01:51 +02:00
9 changed files with 180 additions and 34 deletions
+67 -7
View File
@@ -731,6 +731,66 @@ listing the directory and learning your webhook UUIDs from the
`events-{uuid}.db` filenames — not the barrier protecting the
credentials.
### Running under upaas
[upaas](https://git.eeqj.de/sneak/upaas) builds the image from this
repository's `Dockerfile` and runs it. The app needs:
- **Network and port:** add no port mapping in upaas. upaas publishes
every mapped port on all interfaces of the host
([upaas issue 113](https://git.eeqj.de/sneak/upaas/issues/113)),
which would put the plain-HTTP admin UI and receiver there. Instead,
set the app's Docker Network in upaas to your reverse proxy's Docker
network; the proxy then reaches the app at `upaas-` followed by the
app name, port `8080`. Leave `PORT` unset: the image's health check
probes `8080`.
- **Volume:** one host directory mounted at `/var/lib/webhooker`.
upaas bind-mounts the host path it is given and does not create it,
and the container does not start unless UID 1000 owns it (see
[Running with Docker](#running-with-docker)). Create it before the
first deploy:
```bash
mkdir -p /path/to/data
chown 1000:1000 /path/to/data
chmod 750 /path/to/data
```
- **Environment variables:**
- `WEBHOOKER_ENVIRONMENT=prod`
- `TRUSTED_PROXIES`: your reverse proxy's address on that Docker
network. The `remoteIP` field of the `http request` log line for a
request that came through the proxy shows it; the health check's
own lines show `::1`. See [Trusted proxies](#trusted-proxies).
- Leave `BIND_ADDRESS` and `DATA_DIR` unset: the image sets
`BIND_ADDRESS` to `0.0.0.0`, and `DATA_DIR` defaults to
`/var/lib/webhooker`.
- Everything else is optional; see [Configuration](#configuration).
- **Health check:** the image's own, which requests
`/.well-known/healthcheck`. upaas reads the container's health 60
seconds after a deploy and marks the deploy failed unless it is
`healthy`.
- **First run:** the first start prints the `admin` password once, in
the banner described under [The admin account](#the-admin-account),
to the container's log. upaas names the container `upaas-` followed
by the app name, so for an app named `webhooker`:
```bash
docker logs upaas-webhooker
```
If the password is lost, stop the container, set a new password with
the app's own image and volume, and start it again (see
[Recovering a lost admin password](#recovering-a-lost-admin-password)):
```bash
docker stop upaas-webhooker
docker run --rm --volumes-from upaas-webhooker \
"$(docker inspect -f '{{.Image}}' upaas-webhooker)" \
/app/webhooker resetpw -generate admin
docker start upaas-webhooker
```
## Deployment behind a reverse proxy
webhooker terminates no TLS of its own. It serves plaintext HTTP and
@@ -1507,7 +1567,7 @@ events should be forwarded.
| `type` | TargetType | One of: `http`, `slack`, `database`, `log` |
| `active` | boolean | Whether deliveries are enabled (default: true) |
| `config` | JSON text | Type-specific configuration |
| `max_retries` | integer | Maximum retry attempts for `http` and `slack` targets (0 = fire-and-forget, >0 = retries with backoff and a circuit breaker). Ignored by `database` and `log` targets |
| `max_retries` | integer | Total delivery attempts for `http` and `slack` targets, not retries on top of the first: 0 is a single fire-and-forget attempt with no retries and no circuit breaker, and a value of N makes N attempts in all, with exponential backoff and a per-target circuit breaker. Ignored by `database` and `log` targets |
| `max_queue_size` | integer | Stored and shown on the target's detail view, but not enforced anywhere yet: nothing in the delivery engine consults it. Queue depth is set by the two fixed 10,000-entry channels |
**Relations:** Belongs to Webhook. Has many Deliveries.
@@ -1515,12 +1575,12 @@ events should be forwarded.
**Target types:**
- **`http`** — Forward the event as an HTTP POST to a configured URL.
Behavior depends on `max_retries`: when `max_retries` is 0 (the
default), the target operates in fire-and-forget mode — a single
attempt with no retries and no circuit breaker. When `max_retries` is
greater than 0, failed deliveries are retried with exponential backoff
up to `max_retries` attempts, protected by a per-target circuit
breaker.
`max_retries` is the total number of delivery attempts, not retries on
top of the first: when `max_retries` is 0 (the default), the target
operates in fire-and-forget mode, a single attempt with no retries and
no circuit breaker; a value of N makes up to N attempts in all,
retrying failed deliveries with exponential backoff and protecting them
with a per-target circuit breaker.
- **`slack`** — Post the event as a formatted message to a
Slack-compatible incoming webhook URL (`webhookUrl` in `config`). It
is built on the same HTTP core as `http` and honours `max_retries`
+4 -2
View File
@@ -17,12 +17,12 @@ import (
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/banner"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/datadir"
"sneak.berlin/go/webhooker/internal/gormlog"
"sneak.berlin/go/webhooker/internal/logger"
)
const (
dataDirPerm = 0750
randomPasswordLen = 16
sessionKeyLen = 32
)
@@ -185,7 +185,9 @@ func (d *Database) connect() error {
// caller's decision.
func (d *Database) connectTo(dataDir string) error {
// Ensure the data directory exists before opening the database.
err := os.MkdirAll(dataDir, dataDirPerm)
// datadir.DirPerm is the single source of the directory mode; this
// package creates the directory too, since either may run first.
err := os.MkdirAll(dataDir, datadir.DirPerm)
if err != nil {
return fmt.Errorf(
"creating data directory %s: %w",
+13 -13
View File
@@ -17,24 +17,24 @@ type indexedColumn struct {
field string
}
// eventTierIndexes are the columns the background work reads by: the
// recovery and sweep queries (status), the event log (event_id and
// delivery_id) and retention (created_at).
var eventTierIndexes = []indexedColumn{
{&database.Delivery{}, "Status"},
{&database.Delivery{}, "EventID"},
{&database.DeliveryResult{}, "DeliveryID"},
{&database.Event{}, "CreatedAt"},
}
// TestWebhookDBManager_OpenAddsEventTierIndexes verifies that opening a
// per-webhook database that predates these indexes creates them, so the
// queries above stop scanning whole tables. It stands in for an older
// database file by dropping the indexes AutoMigrate just created, then
// reopening the same file.
// queries that read by those columns stop scanning whole tables. It
// stands in for an older database file by dropping the indexes
// AutoMigrate just created, then reopening the same file.
func TestWebhookDBManager_OpenAddsEventTierIndexes(t *testing.T) {
t.Parallel()
// The columns the background work reads by: the recovery and sweep
// queries (status), the event log (event_id and delivery_id) and
// retention (created_at).
eventTierIndexes := []indexedColumn{
{&database.Delivery{}, "Status"},
{&database.Delivery{}, "EventID"},
{&database.DeliveryResult{}, "DeliveryID"},
{&database.Event{}, "CreatedAt"},
}
mgr, lc := setupTestWebhookDBManager(t)
ctx := context.Background()
require.NoError(t, lc.Start(ctx))
+2 -2
View File
@@ -5,10 +5,10 @@ type DeliveryResult struct {
BaseModel
DeliveryID string `gorm:"type:uuid;not null;index" json:"deliveryId"`
AttemptNum int `gorm:"not null" json:"attemptNum"`
AttemptNum int `gorm:"not null" json:"attemptNum"`
Success bool `json:"success"`
StatusCode int `json:"statusCode,omitempty"`
ResponseBody string `gorm:"type:text" json:"responseBody,omitempty"`
ResponseBody string `gorm:"type:text" json:"responseBody,omitempty"`
Error string `json:"error,omitempty"`
Duration int64 `json:"durationMs"` // Duration in milliseconds
+4 -2
View File
@@ -13,6 +13,7 @@ import (
"gorm.io/driver/sqlite"
"gorm.io/gorm"
"sneak.berlin/go/webhooker/internal/config"
"sneak.berlin/go/webhooker/internal/datadir"
"sneak.berlin/go/webhooker/internal/gormlog"
"sneak.berlin/go/webhooker/internal/logger"
)
@@ -53,8 +54,9 @@ func NewWebhookDBManager(
log: params.Logger.Get(),
}
// Create data directory if it doesn't exist
err := os.MkdirAll(m.dataDir, dataDirPerm)
// Create data directory if it doesn't exist. datadir.DirPerm is the
// single source of the directory mode; either package may run first.
err := os.MkdirAll(m.dataDir, datadir.DirPerm)
if err != nil {
return nil, fmt.Errorf(
"creating data directory %s: %w",
+6 -4
View File
@@ -29,9 +29,11 @@ import (
// process that was killed with SIGKILL blocks nothing.
const LockFileName = "webhooker.lock"
// dirPerm is the mode Acquire creates DATA_DIR with. It matches what
// internal/database uses, since whichever runs first creates it.
const dirPerm = 0o750
// DirPerm is the mode DATA_DIR is created with. It is the single
// source of that mode: internal/database consumes it rather than
// keeping its own copy, so the two packages that both create the
// directory cannot drift into disagreeing about its permissions.
const DirPerm = 0o750
// ErrLocked reports that another live process holds the data
// directory. Callers that need to know whether a deployment is running
@@ -64,7 +66,7 @@ func Acquire(dir string) (*Lock, error) {
return nil, ErrNoDir
}
err := os.MkdirAll(dir, dirPerm)
err := os.MkdirAll(dir, DirPerm)
if err != nil {
return nil, fmt.Errorf(
"creating data directory %s: %w", dir, err,
+77
View File
@@ -300,3 +300,80 @@ func TestEntrypointCopyButtonIsProgressiveEnhancement(t *testing.T) {
"the page must render to completion, not abort partway",
)
}
// maxRetriesHelp is the wording both target forms must carry. The
// delivery core makes max_retries attempts in total, not that many
// retries on top of a first try (a fresh delivery starts at attempt 1
// and target_http gives up once the attempt number reaches
// max_retries), and 0 is special-cased to a single fire-and-forget
// attempt with no circuit breaker.
const maxRetriesHelp = "This is the total number of delivery attempts, " +
"not retries on top of the first: a value of 3 makes three attempts " +
"in all. 0 means a single attempt with no retries and no circuit " +
"breaker."
// TestTargetFormMaxRetriesCopyMatchesBehaviour pins the max_retries
// help text on both the create form (the add-target form on the webhook
// detail page) and the edit form, so the copy cannot drift back to
// calling the number a retry count.
func TestTargetFormMaxRetriesCopyMatchesBehaviour(t *testing.T) {
t.Parallel()
var h *handlers.Handlers
var sess *session.Session
app := newTestApp(t, &h, &sess)
app.RequireStart()
t.Cleanup(app.RequireStop)
webhook := &database.Webhook{Name: "wh", RetentionDays: 14}
webhook.ID = testWebhookID
entrypoint := database.Entrypoint{Path: "abc123"}
entrypoint.ID = "ep-1"
createBody := renderPage(
t, h, sess, "source_detail.html", map[string]any{
dataKeyWebhook: webhook,
"Entrypoints": handlers.NewEntrypointViews(
[]database.Entrypoint{entrypoint},
),
"Targets": delivery.NewTargetViews(nil),
"Events": []database.Event{},
"BaseURL": "https://hooks.example.com",
},
)
assert.Contains(
t, createBody, maxRetriesHelp,
"the add-target form must explain max_retries as total attempts",
)
// A slack target exercises the same max_retries field while needing
// only Config.URL from the edit template, so the test data stays
// minimal. The Target key mirrors the field names the template reads
// off the handler's view value.
editBody := renderPage(
t, h, sess, "target_edit.html", map[string]any{
dataKeyWebhook: webhook,
"Target": map[string]any{
"ID": "tg-1",
"Name": "t",
"Type": "slack",
"Active": true,
"MaxRetries": 3,
"Config": map[string]any{
"URL": "https://hooks.slack.com/services/x",
},
},
dataKeyError: "",
},
)
assert.Contains(
t, editBody, maxRetriesHelp,
"the target edit form must explain max_retries as total attempts",
)
}
+6 -3
View File
@@ -120,9 +120,12 @@
<label class="text-sm text-gray-700">Timeout (seconds, blank = default):</label>
<input type="number" name="timeout" min="0" max="300" :disabled="targetType !== 'http'" class="input text-sm w-24">
</div>
<div x-show="targetType === 'http'" class="flex gap-2 items-center">
<label class="text-sm text-gray-700">Max retries (0 = fire-and-forget):</label>
<input type="number" name="max_retries" value="0" min="0" max="20" class="input text-sm w-24">
<div x-show="targetType === 'http'">
<div class="flex gap-2 items-center">
<label class="text-sm text-gray-700">Max retries:</label>
<input type="number" name="max_retries" value="0" min="0" max="20" class="input text-sm w-24">
</div>
<p class="text-xs text-gray-500 mt-1">This is the total number of delivery attempts, not retries on top of the first: a value of 3 makes three attempts in all. 0 means a single attempt with no retries and no circuit breaker.</p>
</div>
<div x-show="targetType === 'slack'">
<input type="url" name="url" placeholder="https://hooks.slack.com/services/..." :disabled="targetType !== 'slack'" class="input text-sm">
+1 -1
View File
@@ -69,7 +69,7 @@
<div class="form-group">
<label for="max_retries" class="label">Max retries</label>
<input type="number" id="max_retries" name="max_retries" value="{{.Target.MaxRetries}}" min="0" max="20" class="input">
<p class="text-xs text-gray-500 mt-1">0 is fire-and-forget: one attempt, no circuit breaker.</p>
<p class="text-xs text-gray-500 mt-1">This is the total number of delivery attempts, not retries on top of the first: a value of 3 makes three attempts in all. 0 means a single attempt with no retries and no circuit breaker.</p>
</div>
{{end}}