Compare commits
5
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
45d541526f | ||
|
|
06f14315fa | ||
|
|
7ed1588443 | ||
|
|
b051821370 | ||
|
|
39afa69bfc |
@@ -731,6 +731,66 @@ listing the directory and learning your webhook UUIDs from the
|
||||
`events-{uuid}.db` filenames — not the barrier protecting the
|
||||
credentials.
|
||||
|
||||
### Running under upaas
|
||||
|
||||
[upaas](https://git.eeqj.de/sneak/upaas) builds the image from this
|
||||
repository's `Dockerfile` and runs it. The app needs:
|
||||
|
||||
- **Network and port:** add no port mapping in upaas. upaas publishes
|
||||
every mapped port on all interfaces of the host
|
||||
([upaas issue 113](https://git.eeqj.de/sneak/upaas/issues/113)),
|
||||
which would put the plain-HTTP admin UI and receiver there. Instead,
|
||||
set the app's Docker Network in upaas to your reverse proxy's Docker
|
||||
network; the proxy then reaches the app at `upaas-` followed by the
|
||||
app name, port `8080`. Leave `PORT` unset: the image's health check
|
||||
probes `8080`.
|
||||
- **Volume:** one host directory mounted at `/var/lib/webhooker`.
|
||||
upaas bind-mounts the host path it is given and does not create it,
|
||||
and the container does not start unless UID 1000 owns it (see
|
||||
[Running with Docker](#running-with-docker)). Create it before the
|
||||
first deploy:
|
||||
|
||||
```bash
|
||||
mkdir -p /path/to/data
|
||||
chown 1000:1000 /path/to/data
|
||||
chmod 750 /path/to/data
|
||||
```
|
||||
|
||||
- **Environment variables:**
|
||||
- `WEBHOOKER_ENVIRONMENT=prod`
|
||||
- `TRUSTED_PROXIES`: your reverse proxy's address on that Docker
|
||||
network. The `remoteIP` field of the `http request` log line for a
|
||||
request that came through the proxy shows it; the health check's
|
||||
own lines show `::1`. See [Trusted proxies](#trusted-proxies).
|
||||
- Leave `BIND_ADDRESS` and `DATA_DIR` unset: the image sets
|
||||
`BIND_ADDRESS` to `0.0.0.0`, and `DATA_DIR` defaults to
|
||||
`/var/lib/webhooker`.
|
||||
- Everything else is optional; see [Configuration](#configuration).
|
||||
- **Health check:** the image's own, which requests
|
||||
`/.well-known/healthcheck`. upaas reads the container's health 60
|
||||
seconds after a deploy and marks the deploy failed unless it is
|
||||
`healthy`.
|
||||
- **First run:** the first start prints the `admin` password once, in
|
||||
the banner described under [The admin account](#the-admin-account),
|
||||
to the container's log. upaas names the container `upaas-` followed
|
||||
by the app name, so for an app named `webhooker`:
|
||||
|
||||
```bash
|
||||
docker logs upaas-webhooker
|
||||
```
|
||||
|
||||
If the password is lost, stop the container, set a new password with
|
||||
the app's own image and volume, and start it again (see
|
||||
[Recovering a lost admin password](#recovering-a-lost-admin-password)):
|
||||
|
||||
```bash
|
||||
docker stop upaas-webhooker
|
||||
docker run --rm --volumes-from upaas-webhooker \
|
||||
"$(docker inspect -f '{{.Image}}' upaas-webhooker)" \
|
||||
/app/webhooker resetpw -generate admin
|
||||
docker start upaas-webhooker
|
||||
```
|
||||
|
||||
## Deployment behind a reverse proxy
|
||||
|
||||
webhooker terminates no TLS of its own. It serves plaintext HTTP and
|
||||
@@ -1507,7 +1567,7 @@ events should be forwarded.
|
||||
| `type` | TargetType | One of: `http`, `slack`, `database`, `log` |
|
||||
| `active` | boolean | Whether deliveries are enabled (default: true) |
|
||||
| `config` | JSON text | Type-specific configuration |
|
||||
| `max_retries` | integer | Maximum retry attempts for `http` and `slack` targets (0 = fire-and-forget, >0 = retries with backoff and a circuit breaker). Ignored by `database` and `log` targets |
|
||||
| `max_retries` | integer | Total delivery attempts for `http` and `slack` targets, not retries on top of the first: 0 is a single fire-and-forget attempt with no retries and no circuit breaker, and a value of N makes N attempts in all, with exponential backoff and a per-target circuit breaker. Ignored by `database` and `log` targets |
|
||||
| `max_queue_size` | integer | Stored and shown on the target's detail view, but not enforced anywhere yet: nothing in the delivery engine consults it. Queue depth is set by the two fixed 10,000-entry channels |
|
||||
|
||||
**Relations:** Belongs to Webhook. Has many Deliveries.
|
||||
@@ -1515,12 +1575,12 @@ events should be forwarded.
|
||||
**Target types:**
|
||||
|
||||
- **`http`** — Forward the event as an HTTP POST to a configured URL.
|
||||
Behavior depends on `max_retries`: when `max_retries` is 0 (the
|
||||
default), the target operates in fire-and-forget mode — a single
|
||||
attempt with no retries and no circuit breaker. When `max_retries` is
|
||||
greater than 0, failed deliveries are retried with exponential backoff
|
||||
up to `max_retries` attempts, protected by a per-target circuit
|
||||
breaker.
|
||||
`max_retries` is the total number of delivery attempts, not retries on
|
||||
top of the first: when `max_retries` is 0 (the default), the target
|
||||
operates in fire-and-forget mode, a single attempt with no retries and
|
||||
no circuit breaker; a value of N makes up to N attempts in all,
|
||||
retrying failed deliveries with exponential backoff and protecting them
|
||||
with a per-target circuit breaker.
|
||||
- **`slack`** — Post the event as a formatted message to a
|
||||
Slack-compatible incoming webhook URL (`webhookUrl` in `config`). It
|
||||
is built on the same HTTP core as `http` and honours `max_retries`
|
||||
|
||||
@@ -17,12 +17,12 @@ import (
|
||||
"gorm.io/gorm"
|
||||
"sneak.berlin/go/webhooker/internal/banner"
|
||||
"sneak.berlin/go/webhooker/internal/config"
|
||||
"sneak.berlin/go/webhooker/internal/datadir"
|
||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
||||
"sneak.berlin/go/webhooker/internal/logger"
|
||||
)
|
||||
|
||||
const (
|
||||
dataDirPerm = 0750
|
||||
randomPasswordLen = 16
|
||||
sessionKeyLen = 32
|
||||
)
|
||||
@@ -185,7 +185,9 @@ func (d *Database) connect() error {
|
||||
// caller's decision.
|
||||
func (d *Database) connectTo(dataDir string) error {
|
||||
// Ensure the data directory exists before opening the database.
|
||||
err := os.MkdirAll(dataDir, dataDirPerm)
|
||||
// datadir.DirPerm is the single source of the directory mode; this
|
||||
// package creates the directory too, since either may run first.
|
||||
err := os.MkdirAll(dataDir, datadir.DirPerm)
|
||||
if err != nil {
|
||||
return fmt.Errorf(
|
||||
"creating data directory %s: %w",
|
||||
|
||||
@@ -17,24 +17,24 @@ type indexedColumn struct {
|
||||
field string
|
||||
}
|
||||
|
||||
// eventTierIndexes are the columns the background work reads by: the
|
||||
// recovery and sweep queries (status), the event log (event_id and
|
||||
// delivery_id) and retention (created_at).
|
||||
var eventTierIndexes = []indexedColumn{
|
||||
{&database.Delivery{}, "Status"},
|
||||
{&database.Delivery{}, "EventID"},
|
||||
{&database.DeliveryResult{}, "DeliveryID"},
|
||||
{&database.Event{}, "CreatedAt"},
|
||||
}
|
||||
|
||||
// TestWebhookDBManager_OpenAddsEventTierIndexes verifies that opening a
|
||||
// per-webhook database that predates these indexes creates them, so the
|
||||
// queries above stop scanning whole tables. It stands in for an older
|
||||
// database file by dropping the indexes AutoMigrate just created, then
|
||||
// reopening the same file.
|
||||
// queries that read by those columns stop scanning whole tables. It
|
||||
// stands in for an older database file by dropping the indexes
|
||||
// AutoMigrate just created, then reopening the same file.
|
||||
func TestWebhookDBManager_OpenAddsEventTierIndexes(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// The columns the background work reads by: the recovery and sweep
|
||||
// queries (status), the event log (event_id and delivery_id) and
|
||||
// retention (created_at).
|
||||
eventTierIndexes := []indexedColumn{
|
||||
{&database.Delivery{}, "Status"},
|
||||
{&database.Delivery{}, "EventID"},
|
||||
{&database.DeliveryResult{}, "DeliveryID"},
|
||||
{&database.Event{}, "CreatedAt"},
|
||||
}
|
||||
|
||||
mgr, lc := setupTestWebhookDBManager(t)
|
||||
ctx := context.Background()
|
||||
require.NoError(t, lc.Start(ctx))
|
||||
|
||||
@@ -5,10 +5,10 @@ type DeliveryResult struct {
|
||||
BaseModel
|
||||
|
||||
DeliveryID string `gorm:"type:uuid;not null;index" json:"deliveryId"`
|
||||
AttemptNum int `gorm:"not null" json:"attemptNum"`
|
||||
AttemptNum int `gorm:"not null" json:"attemptNum"`
|
||||
Success bool `json:"success"`
|
||||
StatusCode int `json:"statusCode,omitempty"`
|
||||
ResponseBody string `gorm:"type:text" json:"responseBody,omitempty"`
|
||||
ResponseBody string `gorm:"type:text" json:"responseBody,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
Duration int64 `json:"durationMs"` // Duration in milliseconds
|
||||
|
||||
|
||||
@@ -13,6 +13,7 @@ import (
|
||||
"gorm.io/driver/sqlite"
|
||||
"gorm.io/gorm"
|
||||
"sneak.berlin/go/webhooker/internal/config"
|
||||
"sneak.berlin/go/webhooker/internal/datadir"
|
||||
"sneak.berlin/go/webhooker/internal/gormlog"
|
||||
"sneak.berlin/go/webhooker/internal/logger"
|
||||
)
|
||||
@@ -53,8 +54,9 @@ func NewWebhookDBManager(
|
||||
log: params.Logger.Get(),
|
||||
}
|
||||
|
||||
// Create data directory if it doesn't exist
|
||||
err := os.MkdirAll(m.dataDir, dataDirPerm)
|
||||
// Create data directory if it doesn't exist. datadir.DirPerm is the
|
||||
// single source of the directory mode; either package may run first.
|
||||
err := os.MkdirAll(m.dataDir, datadir.DirPerm)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf(
|
||||
"creating data directory %s: %w",
|
||||
|
||||
@@ -29,9 +29,11 @@ import (
|
||||
// process that was killed with SIGKILL blocks nothing.
|
||||
const LockFileName = "webhooker.lock"
|
||||
|
||||
// dirPerm is the mode Acquire creates DATA_DIR with. It matches what
|
||||
// internal/database uses, since whichever runs first creates it.
|
||||
const dirPerm = 0o750
|
||||
// DirPerm is the mode DATA_DIR is created with. It is the single
|
||||
// source of that mode: internal/database consumes it rather than
|
||||
// keeping its own copy, so the two packages that both create the
|
||||
// directory cannot drift into disagreeing about its permissions.
|
||||
const DirPerm = 0o750
|
||||
|
||||
// ErrLocked reports that another live process holds the data
|
||||
// directory. Callers that need to know whether a deployment is running
|
||||
@@ -64,7 +66,7 @@ func Acquire(dir string) (*Lock, error) {
|
||||
return nil, ErrNoDir
|
||||
}
|
||||
|
||||
err := os.MkdirAll(dir, dirPerm)
|
||||
err := os.MkdirAll(dir, DirPerm)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf(
|
||||
"creating data directory %s: %w", dir, err,
|
||||
|
||||
@@ -300,3 +300,80 @@ func TestEntrypointCopyButtonIsProgressiveEnhancement(t *testing.T) {
|
||||
"the page must render to completion, not abort partway",
|
||||
)
|
||||
}
|
||||
|
||||
// maxRetriesHelp is the wording both target forms must carry. The
|
||||
// delivery core makes max_retries attempts in total, not that many
|
||||
// retries on top of a first try (a fresh delivery starts at attempt 1
|
||||
// and target_http gives up once the attempt number reaches
|
||||
// max_retries), and 0 is special-cased to a single fire-and-forget
|
||||
// attempt with no circuit breaker.
|
||||
const maxRetriesHelp = "This is the total number of delivery attempts, " +
|
||||
"not retries on top of the first: a value of 3 makes three attempts " +
|
||||
"in all. 0 means a single attempt with no retries and no circuit " +
|
||||
"breaker."
|
||||
|
||||
// TestTargetFormMaxRetriesCopyMatchesBehaviour pins the max_retries
|
||||
// help text on both the create form (the add-target form on the webhook
|
||||
// detail page) and the edit form, so the copy cannot drift back to
|
||||
// calling the number a retry count.
|
||||
func TestTargetFormMaxRetriesCopyMatchesBehaviour(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var h *handlers.Handlers
|
||||
|
||||
var sess *session.Session
|
||||
|
||||
app := newTestApp(t, &h, &sess)
|
||||
app.RequireStart()
|
||||
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
webhook := &database.Webhook{Name: "wh", RetentionDays: 14}
|
||||
webhook.ID = testWebhookID
|
||||
|
||||
entrypoint := database.Entrypoint{Path: "abc123"}
|
||||
entrypoint.ID = "ep-1"
|
||||
|
||||
createBody := renderPage(
|
||||
t, h, sess, "source_detail.html", map[string]any{
|
||||
dataKeyWebhook: webhook,
|
||||
"Entrypoints": handlers.NewEntrypointViews(
|
||||
[]database.Entrypoint{entrypoint},
|
||||
),
|
||||
"Targets": delivery.NewTargetViews(nil),
|
||||
"Events": []database.Event{},
|
||||
"BaseURL": "https://hooks.example.com",
|
||||
},
|
||||
)
|
||||
|
||||
assert.Contains(
|
||||
t, createBody, maxRetriesHelp,
|
||||
"the add-target form must explain max_retries as total attempts",
|
||||
)
|
||||
|
||||
// A slack target exercises the same max_retries field while needing
|
||||
// only Config.URL from the edit template, so the test data stays
|
||||
// minimal. The Target key mirrors the field names the template reads
|
||||
// off the handler's view value.
|
||||
editBody := renderPage(
|
||||
t, h, sess, "target_edit.html", map[string]any{
|
||||
dataKeyWebhook: webhook,
|
||||
"Target": map[string]any{
|
||||
"ID": "tg-1",
|
||||
"Name": "t",
|
||||
"Type": "slack",
|
||||
"Active": true,
|
||||
"MaxRetries": 3,
|
||||
"Config": map[string]any{
|
||||
"URL": "https://hooks.slack.com/services/x",
|
||||
},
|
||||
},
|
||||
dataKeyError: "",
|
||||
},
|
||||
)
|
||||
|
||||
assert.Contains(
|
||||
t, editBody, maxRetriesHelp,
|
||||
"the target edit form must explain max_retries as total attempts",
|
||||
)
|
||||
}
|
||||
|
||||
@@ -120,9 +120,12 @@
|
||||
<label class="text-sm text-gray-700">Timeout (seconds, blank = default):</label>
|
||||
<input type="number" name="timeout" min="0" max="300" :disabled="targetType !== 'http'" class="input text-sm w-24">
|
||||
</div>
|
||||
<div x-show="targetType === 'http'" class="flex gap-2 items-center">
|
||||
<label class="text-sm text-gray-700">Max retries (0 = fire-and-forget):</label>
|
||||
<input type="number" name="max_retries" value="0" min="0" max="20" class="input text-sm w-24">
|
||||
<div x-show="targetType === 'http'">
|
||||
<div class="flex gap-2 items-center">
|
||||
<label class="text-sm text-gray-700">Max retries:</label>
|
||||
<input type="number" name="max_retries" value="0" min="0" max="20" class="input text-sm w-24">
|
||||
</div>
|
||||
<p class="text-xs text-gray-500 mt-1">This is the total number of delivery attempts, not retries on top of the first: a value of 3 makes three attempts in all. 0 means a single attempt with no retries and no circuit breaker.</p>
|
||||
</div>
|
||||
<div x-show="targetType === 'slack'">
|
||||
<input type="url" name="url" placeholder="https://hooks.slack.com/services/..." :disabled="targetType !== 'slack'" class="input text-sm">
|
||||
|
||||
@@ -69,7 +69,7 @@
|
||||
<div class="form-group">
|
||||
<label for="max_retries" class="label">Max retries</label>
|
||||
<input type="number" id="max_retries" name="max_retries" value="{{.Target.MaxRetries}}" min="0" max="20" class="input">
|
||||
<p class="text-xs text-gray-500 mt-1">0 is fire-and-forget: one attempt, no circuit breaker.</p>
|
||||
<p class="text-xs text-gray-500 mt-1">This is the total number of delivery attempts, not retries on top of the first: a value of 3 makes three attempts in all. 0 means a single attempt with no retries and no circuit breaker.</p>
|
||||
</div>
|
||||
{{end}}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user